Uploaded image for project: 'camunda BPM'
  1. camunda BPM
  2. CAM-6322

Deployment of EAR with multiple WARs fails with NPE in Camunda JBoss Subsystem

    Details

    • Title Keywords:
      NullPointerException JBoss Deployment

      Description

      Deploying an EAR file containing Skinny-WARs (multiple WARs with shared libs in the EAR) to JBoss (AS7), leads sometimes to a NullPointerException.
      The problem is, that the class org.camunda.bpm.container.impl.jboss.deployment.processor.ModuleDependencyProcessor.java adds system dependencies to OTHER sibling subdeployment-units. JBoss starts the subdeployment for each WAR file inside the EAR as subdeployment in parallel. This causes concurrency problems, because each WAR subdeployment adds the camunda libs as system dependencies to all other WARs.
      The access to ModuleSpecification.addSystemDependency(), which adds the camunda libraries, is not synchronized and finally leads to an inconsistent system-dependency-list containing null values. As a result JBoss throws some times later a NPE:

      08:07:58,214 ERROR [MSC service thread 1-47] org.jboss.msc.service.ServiceControllerImpl$StartTask.startFailed:1987 - MSC000001: Failed to start service jboss.deployment.subunit."GUIWF.ear"."CCP_ConfCalc.jar".CONFIGURE_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.subunit."GUIWF.ear"."CCP_ConfCalc.jar".CONFIGURE_MODULE: JBAS018733: Verarbeitung von Phase CONFIGURE_MODULE von subdeployment "CCP_ConfCalc.jar" of deployment "GUIWF.ear" fehlgeschlagen
      at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:166)
      at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980)
      at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NullPointerException
      at org.jboss.as.server.deployment.module.ModuleSpecProcessor.createDependencies(ModuleSpecProcessor.java:237)
      at org.jboss.as.server.deployment.module.ModuleSpecProcessor.createModuleService(ModuleSpecProcessor.java:182)
      at org.jboss.as.server.deployment.module.ModuleSpecProcessor.deployModuleSpec(ModuleSpecProcessor.java:116)
      at org.jboss.as.server.deployment.module.ModuleSpecProcessor.deploy(ModuleSpecProcessor.java:79)
      at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:159)
      ... 5 more

      The same problem was reported and fixed for the JBoss JSFDependencyProcessor: https://issues.jboss.org/browse/WFLY-2496

      We are currently working with camunda release 7.3, but as I looked in the sources, it should still be open.

      Solution?
      In general, it seems not to be expected, that one Subdeployment unit adds system dependencies to another Subdeployment unit. So I can not present a clean solution, but adding a "synchronized" to the method ModuleDependencyProcessor.addSystemDependency() should decrease the concurrency possibility.

      1. example-EAR-pom.xml
        5 kB
        Ferenc Hechler
      2. example-WAR-pom.xml
        3 kB
        Ferenc Hechler
      3. WFGUI.ear
        508 kB
        Ferenc Hechler
      1. green-old-logic_orange-inverse-new-logic.png
        12 kB

        Activity

        Hide
        feri Ferenc Hechler added a comment -

        It is not easy to reproduce the error, because the concurrency depends on memory, cpu speed, number of deployed application, and much more ...

        But you can check, whether there are unallowed concurrent accesses.

        1. Start JBoss in debug mode
        2. Create a method Breakpoint in org.camunda.bpm.container.impl.jboss.deployment.processor.ModuleDependencyProcessor.deploy(DeploymentPhaseContext phaseContext)
        3. Deploy WFGUI.ear

        There are four concurrent calls to deploy() for each subdeployment.
        And each of this calls adds the system dependencies to the same instances of module specification (parent depoyment and all sibling deployments). This produces the error.

        Show
        feri Ferenc Hechler added a comment - It is not easy to reproduce the error, because the concurrency depends on memory, cpu speed, number of deployed application, and much more ... But you can check, whether there are unallowed concurrent accesses. 1. Start JBoss in debug mode 2. Create a method Breakpoint in org.camunda.bpm.container.impl.jboss.deployment.processor.ModuleDependencyProcessor.deploy(DeploymentPhaseContext phaseContext) 3. Deploy WFGUI.ear There are four concurrent calls to deploy() for each subdeployment. And each of this calls adds the system dependencies to the same instances of module specification (parent depoyment and all sibling deployments). This produces the error.
        Hide
        meyer Daniel Meyer added a comment -

        Hi Ferenc,

        Christopher Zell and I looked into the issue today and we can see what is going wrong and understand your pull request. Christopher is going to simplify the code a bit and put the fix on master.

        Best Regards,
        Daniel

        Show
        meyer Daniel Meyer added a comment - Hi Ferenc, Christopher Zell and I looked into the issue today and we can see what is going wrong and understand your pull request. Christopher is going to simplify the code a bit and put the fix on master. Best Regards, Daniel
        Hide
        feri Ferenc Hechler added a comment -

        Hi Daniel,
        Hi Christopher,

        that are good News.
        For your Information: We have installed the patched version for 7.3.0 at our development server and it runs stable for two weeks now.

        Best regards,

        feri

        Show
        feri Ferenc Hechler added a comment - Hi Daniel, Hi Christopher, that are good News. For your Information: We have installed the patched version for 7.3.0 at our development server and it runs stable for two weeks now. Best regards, feri
        Hide
        feri Ferenc Hechler added a comment -

        I see you found a simpler solution, which adds all system depenencies from the root deployment (EAR or WAR) and such avoids concurrency between the subdeployments.
        I have tested your changes in the master and it worked well.
        Best regards,
        feri

        Show
        feri Ferenc Hechler added a comment - I see you found a simpler solution, which adds all system depenencies from the root deployment (EAR or WAR) and such avoids concurrency between the subdeployments. I have tested your changes in the master and it worked well. Best regards, feri
        Hide
        christopher.zell Christopher Zell added a comment -

        Hey Ferenc,

        thanks for your feedback and also for your pull request.
        It helps us to identify the problem and to code a simpler solution.
        I'm glad that our solution works well for you.

        Best regards,

        Chris

        Show
        christopher.zell Christopher Zell added a comment - Hey Ferenc, thanks for your feedback and also for your pull request. It helps us to identify the problem and to code a simpler solution. I'm glad that our solution works well for you. Best regards, Chris

          People

          • Assignee:
            meyer Daniel Meyer
            Reporter:
            feri Ferenc Hechler
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development