Uploaded image for project: 'camunda BPM'
  1. camunda BPM
  2. CAM-10576

DatabaseFlushTest is not stable on MariaDB Galera

    Details

      Description

      The DatabaseFlushTest is expecting OptimisticLockingException but in case of MariaDb Galera the exception is deadlock. From three runs on 7.12, two are failing with:

      Expecting:
        <org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation 'INSERT VariableInstanceEntity[1197913]' with message '
      ### Error flushing statements.  Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #6) failed. 5 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: (conn:5) Deadlock found when trying to get lock; try restarting transaction
      ### Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #6) failed. 5 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: (conn:5) Deadlock found when trying to get lock; try restarting transaction'. Flush summary: 
       [
        INSERT HistoricVariableInstanceEntity[1197913]
        INSERT HistoricVariableInstanceEntity[1197915]
        INSERT HistoricVariableInstanceEntity[1197917]
        INSERT HistoricVariableUpdateEventEntity[1197914]
        INSERT HistoricVariableUpdateEventEntity[1197916]
        INSERT HistoricVariableUpdateEventEntity[1197918]
        INSERT HistoricTaskInstanceEventEntity[1197927]
        INSERT HistoricActivityInstanceEventEntity[task2-2:1197925]
        INSERT TaskEntity[1197927]
        INSERT VariableInstanceEntity[1197913]
        INSERT VariableInstanceEntity[1197915]
        INSERT VariableInstanceEntity[1197917]
        DELETE TaskEntity[1197912]
        UPDATE ExecutionEntity[1197908]
        UPDATE ExecutionEntity[1197904]
        UPDATE HistoricActivityInstanceEventEntity[task2-1:1197911]
        UPDATE HistoricTaskInstanceEventEntity[1197912]
      ]>
      to be an instance of:
        <org.camunda.bpm.engine.OptimisticLockingException>
      but was:
        <"org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation 'INSERT VariableInstanceEntity[1197913]' with message '
      ### Error flushing statements.  Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #6) failed. 5 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: (conn:5) Deadlock found when trying to get lock; try restarting transaction
      ### Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #6) failed. 5 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: (conn:5) Deadlock found when trying to get lock; try restarting transaction'. Flush summary: 
       [
        INSERT HistoricVariableInstanceEntity[1197913]
        INSERT HistoricVariableInstanceEntity[1197915]
        INSERT HistoricVariableInstanceEntity[1197917]
        INSERT HistoricVariableUpdateEventEntity[1197914]
        INSERT HistoricVariableUpdateEventEntity[1197916]
        INSERT HistoricVariableUpdateEventEntity[1197918]
        INSERT HistoricTaskInstanceEventEntity[1197927]
        INSERT HistoricActivityInstanceEventEntity[task2-2:1197925]
        INSERT TaskEntity[1197927]
        INSERT VariableInstanceEntity[1197913]
        INSERT VariableInstanceEntity[1197915]
        INSERT VariableInstanceEntity[1197917]
        DELETE TaskEntity[1197912]
        UPDATE ExecutionEntity[1197908]
        UPDATE ExecutionEntity[1197904]
        UPDATE HistoricActivityInstanceEventEntity[task2-1:1197911]
        UPDATE HistoricTaskInstanceEventEntity[1197912]
      ]
       at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:131)
       at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:355)
       at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:322)
       at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:294)
       at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:268)
       at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:186)
       at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:116)
       at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
       at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
       at org.camunda.bpm.engine.test.concurrency.ConcurrencyTestCase$1.run(ConcurrencyTestCase.java:62)
       at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.ibatis.exceptions.PersistenceException: 
      ### Error flushing statements.  Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #6) failed. 5 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: (conn:5) Deadlock found when trying to get lock; try restarting transaction
      ### Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #6) failed. 5 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: (conn:5) Deadlock found when trying to get lock; try restarting transaction
       at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
       at org.apache.ibatis.session.defaults.DefaultSqlSession.flushStatements(DefaultSqlSession.java:255)
       at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.flushBatchOperations(DbSqlSession.java:320)
       at org.camunda.bpm.engine.impl.db.sql.BatchDbSqlSession.executeDbOperations(BatchDbSqlSession.java:67)
       at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:339)
       ... 9 more
      Caused by: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #6) failed. 5 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: (conn:5) Deadlock found when trying to get lock; try restarting transaction
       at org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:146)
       at org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:129)
       at org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:122)
       at org.apache.ibatis.executor.CachingExecutor.flushStatements(CachingExecutor.java:114)
       at org.apache.ibatis.session.defaults.DefaultSqlSession.flushStatements(DefaultSqlSession.java:253)
       ... 12 more
      Caused by: java.sql.BatchUpdateException: (conn:5) Deadlock found when trying to get lock; try restarting transaction
       at org.mariadb.jdbc.MariaDbStatement.executeBatchExceptionEpilogue(MariaDbStatement.java:298)
       at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeBatch(MariaDbPreparedStatementClient.java:290)
       at org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:122)
       ... 16 more
      Caused by: java.sql.SQLTransactionRollbackException: (conn:5) Deadlock found when trying to get lock; try restarting transaction
       at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:160)
       at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:118)
       at org.mariadb.jdbc.MariaDbStatement.executeBatchExceptionEpilogue(MariaDbStatement.java:295)
       ... 18 more
      Caused by: java.sql.SQLException: Deadlock found when trying to get lock; try restarting transaction
      Query is: insert into ACT_RU_VARIABLE
          (
            ID_,
            TYPE_,
            NAME_,
            PROC_INST_ID_,
            EXECUTION_ID_,
            CASE_INST_ID_,
            CASE_EXECUTION_ID_,
            TASK_ID_,
            BYTEARRAY_ID_,
            DOUBLE_,
            LONG_,
            TEXT_,
            TEXT2_,
            VAR_SCOPE_,
            SEQUENCE_COUNTER_,
            IS_CONCURRENT_LOCAL_,
            TENANT_ID_,
            REV_
          )
          values (
            '1197913',
            'string',
            'key3',
            '1197904',
            '1197904',
            <null>,
            <null>,
            <null>,
            <null>,
            <null>,
            <null>,
            'val3',
            <null>,
            '1197904',
            1,
            0,
            <null>,
            1
          )
       at org.mariadb.jdbc.internal.util.LogQueryTool.exceptionWithQuery(LogQueryTool.java:112)
       at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol$1.handleResultException(AbstractQueryProtocol.java:306)
       at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:139)
       at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:67)
       at java.util.concurrent.FutureTask.run(FutureTask.java:262)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       ... 1 more
      ">
      

      Job:
      https://ci.cambpm.camunda.cloud/job/7.12/view/Broken/job/7.12-engine-UNIT-mariadb-galera/
      Test:
      https://github.com/camunda/camunda-bpm-platform/blob/master/engine/src/test/java/org/camunda/bpm/engine/test/persistence/DatabaseFlushTest.java

        Issue Links

          Activity

          Hide
          thorben.lindhauer Thorben Lindhauer added a comment -

          According to https://galeracluster.com/library/training/tutorials/differences.html#transaction-commits and https://github.com/codership/galera/issues/491, deadlocks can generally indicate write conflicts in concurrency scenarios. However, the first link talks about issues on commit, whereas in the failing tests the deadlocks occurs when flushing the batch statements.

          I decided to exclude the test in the galera profile. The failure does not indicate a problem with the code implemented in CAM-10409.

          Show
          thorben.lindhauer Thorben Lindhauer added a comment - According to https://galeracluster.com/library/training/tutorials/differences.html#transaction-commits and https://github.com/codership/galera/issues/491 , deadlocks can generally indicate write conflicts in concurrency scenarios. However, the first link talks about issues on commit, whereas in the failing tests the deadlocks occurs when flushing the batch statements. I decided to exclude the test in the galera profile. The failure does not indicate a problem with the code implemented in CAM-10409 .
          Show
          yana.vasileva Yana Vasileva added a comment - the test must be excluded for upgrade tests as well: https://github.com/camunda/camunda-bpm-platform/blob/7bffe6f66bc79d538602414bd405e0365075456c/qa/test-db-upgrade/pom.xml#L369-L389

            People

            • Assignee:
              Unassigned
              Reporter:
              yana.vasileva Yana Vasileva
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development