Instance Crashed due to Deadlock_问答_开发者_运维开发者技术经验分享

We hit the instance crash again and this time around using Samurai tool, i see several areas where deadlock is been noticed. One of the code snippets we found on deadlock is below. The corresponding application code referred at line no. 898 is just a simple statement that is trying to close a statement withing a catch block.Not sure why this is causing a deadlock.Need assistance to troubleshoot on this.

    at weblogic.jdbc.wrapper.Statement.close(Statement.java:293)
    at com.crimsonlogic.trade.sdec.sau.util.CommonUtil.getHijri(CommonUtil.java:898)
    at com.crimsonlogic.trade.sdec.sau.model.SAUSDECVehicleTransaction.setHijrifromGregorian_trnDate(SAUSDECVehicleTransaction.java:261)
    at com.crimsonlogic.trade.sdec.sau.dao.SAUSDECVehicleTransactionDAOImpl.search(SAUSDECVehicleTransactionDAOImpl.java:175)
    at com.crimsonlogic.trade.sdec.sau.dao.SAUSDECMasterDAOImpl.searchSDECMsg(SAUSDECMasterDAOImpl.java:310)
    at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean.search(SAUSDECSvcBean.java:452)
    at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean_bxozws_EOImpl.search(SAUSDECSvcBean_bxozws_EOImpl.java:1290)



Found one Java-level deadlock:
=============================
"asyncDelivery42":
  waiting to lock monitor 0xdfb820 (object 0x87351698, a oracle.jdbc.driver.OracleConnection),
  which is held by "ExecuteThread: '93' for queue: 'weblogic.kernel.Default'"
"ExecuteThread: '93' for queue: 'weblogic.kernel.Default'":
  waiting to lock monitor 0xdbf550 (object 0x873782b0, a oracle.jdbc.driver.OracleCallableStatement),
  which is held by "asyncDelivery42"

Java stack information for the threads listed above:
===================================================
"asyncDelivery42":
        at oracle.jdbc.driver.OracleConnection.getDefaultRowPrefetch(OracleConnection.java:1866)
        - waiting to lock <87351698> (a oracle.jdbc.driver.OracleConnection)
        at oracle.jdbc.driver.OracleStatement.setPrefetchInternal(OracleStatement.java:1635)
        at oracle.jdbc.driver.OracleStatement.setFetchSize(OracleStatement.java:6505)
        - locked <873782b0> (a oracle.jdbc.driver.OracleCallableStatement)
        at weblogic.jdbc.common.internal.ConnectionEnv.cleanUpStatementForReUse(ConnectionEnv.java:1091)
        at weblogic.jdbc.common.internal.ConnectionEnv.returnCachedStatement(ConnectionEnv.java:883)
        at weblogic.jdbc.wrapper.Statement.internalClose(Statement.java:271)
        at weblogic.jdbc.wrapper.Statement.close(Statement.java:293)
        at com.crimsonlogic.trade.sdec.sau.util.CommonUtil.getHijri(CommonUtil.java:898)
        at com.crimsonlogic.trade.sdec.sau.model.SAUSDECVehicleTransaction.setHijrifromGregorian_trnDate(SAUSDECVehicleTransaction.java:261)
        at com.crimsonlogic.trade.sdec.sau.dao.SAUSDECVehicleTransactionDAOImpl.search(SAUSDECVehicleTransactionDAOImpl.java:175)
        at com.crimsonlogic.trade.sdec.sau.dao.SAUSDECMasterDAOImpl.searchSDECMsg(SAUSDECMasterDAOImpl.java:310)
        at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean.search(SAUSDECSvcBean.java:452)
        at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean_bxozws_EOImpl.search(SAUSDECSvcBean_bxozws_EOImpl.java:1290)
        at com.crimsonlogic.trade.sdec.sau.ejb.SAUSDECSvcBean_bxozws_EOImpl_CBV.search(Unknown Source)
        at com.crimsonlogic.trade.sdec.sau.business.SAUSDECDelegate.search(SAUSDECDelegate.java:278)
        at com.crimsonlogic.trade.sdec.sau.business.SDECMsgProcessor.process(SDECMsgProcessor.java:72)
        at com.crimsonlogic.trade.sau.msg.ejb.MsgSvcBean.processMsg(MsgSvcBean.java:201)
        at com.crimsonlogic.trade.sau.msg.ejb.MsgSvcBean_5z2zq6_EOImpl.processMsg(MsgSvcBean_5z2zq6_EOImpl.java:1340)
        at com.crimsonlogic.trade.sau.msg.util.MsgDelegate.processMsg(MsgDelegate.java:196)
        at com.crimsonlogic.tra开发者_如何学JAVAde.sau.msg.ejb.ProcessPortMsgIDMDB.onMessage(ProcessPortMsgIDMDB.java:72)
        at weblogic.ejb20.internal.MDListener.execute(MDListener.java:382)
        at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:316)
        at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:281)
        at com.ibm.mq.jms.MQMessageConsumer.receiveAsyncQ(MQMessageConsumer.java:2431)
        - locked <7bc2f570> (a java.lang.Object)
        - locked <7bc2f4e8> (a com.ibm.mq.jms.MQQueueReceiver)
        at com.ibm.mq.jms.MQMessageConsumer.receiveAsync(MQMessageConsumer.java:3816)
        - locked <7bc2f4e8> (a com.ibm.mq.jms.MQQueueReceiver)
        at com.ibm.mq.jms.SessionAsyncHelper.run(SessionAsyncHelper.java:420)
        at java.lang.Thread.run(Thread.java:536)
"ExecuteThread: '93' for queue: 'weblogic.kernel.Default'":
        at oracle.jdbc.driver.OracleCallableStatement.sendBatch(OracleCallableStatement.java:446)
        - waiting to lock <873782b0> (a oracle.jdbc.driver.OracleCallableStatement)
        at oracle.jdbc.driver.OracleConnection.commit(OracleConnection.java:1336)
        - locked <87351698> (a oracle.jdbc.driver.OracleConnection)
        at weblogic.jdbc.wrapper.JTSConnection.internalCommit(JTSConnection.java:362)
        at weblogic.jdbc.wrapper.JTSXAResourceImpl.commit(JTSXAResourceImpl.java:56)
        at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:1223)
        at weblogic.transaction.internal.XAServerResourceInfo.commit(XAServerResourceInfo.java:472)
        at weblogic.transaction.internal.ServerSCInfo.startCommit(ServerSCInfo.java:421)
        at weblogic.transaction.internal.ServerTransactionImpl.localCommit(ServerTransactionImpl.java:1761)
        at weblogic.transaction.internal.ServerTransactionImpl.globalRetryCommit(ServerTransactionImpl.java:2384)
        at weblogic.transaction.internal.ServerTransactionImpl.globalCommit(ServerTransactionImpl.java:2315)
        at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:255)
        at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:221)
        at weblogic.ejb20.internal.BaseEJBObject.postInvoke(BaseEJBObject.java:289)
        at weblogic.ejb20.internal.StatelessEJBObject.postInvoke(StatelessEJBObject.java:141)
        at com.crimsonlogic.trade.sau.msg.ejb.MsgSvcBean_5z2zq6_EOImpl.checkLinkedRefIDStatus(MsgSvcBean_5z2zq6_EOImpl.java:110)
        at com.crimsonlogic.trade.sau.msg.util.MsgDelegate.checkLinkedRefIDStatus(MsgDelegate.java:824)
        at com.crimsonlogic.trade.sau.msg.action.ProcessMsg.getProcessMsgIDList(ProcessMsg.java:260)
        at com.crimsonlogic.trade.sau.msg.action.ProcessMsg._process(ProcessMsg.java:123)
        at com.crimsonlogic.trade.sau.msg.action.AbstractAction.process(AbstractAction.java:61)
        at com.crimsonlogic.trade.sau.msg.servlet.SAUMsgProcessServlet.doPost(SAUMsgProcessServlet.java:102)
        at com.crimsonlogic.trade.sau.msg.servlet.SAUMsgProcessServlet.doGet(SAUMsgProcessServlet.java:52)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:971)
        at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:402)
        at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6354)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
        at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3635)
        at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2585)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

Found 2 deadlocks.

Please examine checkLinkedRefIDStatus() method in MsgDelegate. It is being called from both SAUMsgProcessServlet (servlet container thread) and from ProcessPortMsgIDMDB.onMessage (MQ Series MDB listener thread).

My guess is that this method uses the same JDBC connection all the time instead of borrowing from the pool/opening new one. I don't have Oracle JDBC driver sources in front of my eyes, but I bet oracle.jdbc.driver.OracleConnection.commit is synchronized and apparently multiple threads hitting checkLinkedRefIDStatus are sharing the same connection.