Uploaded image for project: 'camunda BPM'
  1. camunda BPM
  2. CAM-9811

Engine fails on WebSphere 8.5 with certain log levels

    Details

    • Type: Bug Report
    • Status: Ready
    • Priority: L3 - Default
    • Resolution: Unresolved
    • Affects Version/s: 7.10.3, 7.12.0-alpha4
    • Fix Version/s: 7.11.x, 7.12.0
    • Component/s: None
    • Labels:
      None

      Description

      What are the steps to reproduce your problem?

      1. Perform a manual installation of the engine on WebSphere 8.5. Testing environment was WebSphere 8.5, JDK7 and Oracle 12.1 (ojdbc7).
      2. Confirm that the engine starts successfully
      3. Set the log levels to the following:
        *=info:jmsApi=all:Messaging=all:com.ibm.mq.*=all:WAS.j2c=all:Transaction=all
      4. Restart WebSphere
      5. Attempt to start the engine application, you should get an exception
      [2/21/19 17:38:29:375 CET] 00000065 RALifeCycleMa E   J2CA0052E: The lookup of the Activation Specification with JNDI Name  eis/PlatformJobExecutorActivation failed due to the following exception: javax.nam
      ing.NameNotFoundException: Context: localhostNode01Cell/nodes/localhostNode01/servers/server1, name: eis/PlatformJobExecutorActivation: First component in name PlatformJobExecutorActivation not found. [Root
       exception is org.omg.CosNaming.NamingContextPackage.NotFound: IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]
              at com.ibm.ws.naming.jndicos.CNContextImpl.mapNotFoundException(CNContextImpl.java:4564)
              at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1822)
              at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1777)
              at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1434)
              at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:616)
              at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:165)
              at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179)
              at org.apache.aries.jndi.DelegateContext.lookup(DelegateContext.java:161)
              at javax.naming.InitialContext.lookup(InitialContext.java:423)
              at com.ibm.ejs.j2c.RALifeCycleManagerImpl.activateEndpoint(RALifeCycleManagerImpl.java:1770)
              at com.ibm.ejs.container.MessageEndpointFactoryImpl.activateEndpoint(MessageEndpointFactoryImpl.java:168)
              at com.ibm.ws.runtime.component.EJBContainerImpl.startMDBs(EJBContainerImpl.java:4726)
              at com.ibm.ws.runtime.component.WASEJBRuntimeImpl.startMDBs(WASEJBRuntimeImpl.java:686)
              at com.ibm.ws.ejbcontainer.runtime.AbstractEJBRuntime.startModule(AbstractEJBRuntime.java:701)
              at com.ibm.ws.ejbcontainer.runtime.SharedEJBRuntimeImpl.startModule(SharedEJBRuntimeImpl.java:338)
              at com.ibm.ws.runtime.component.EJBContainerImpl.start(EJBContainerImpl.java:3588)
              at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1211)
              at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1390)
              at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
              at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:979)
              at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:795)
              at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplicationDynamically(ApplicationMgrImpl.java:1413)
              at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2273)
              at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436)
              at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
              at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379)
              at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127)
              at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$1.run(CompositionUnitMgrImpl.java:654)
              at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5464)
              at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5680)
              at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
              at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:668)
              at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.startCompositionUnit(CompositionUnitMgrImpl.java:612)
              at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:1303)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
              at java.lang.reflect.Method.invoke(Method.java:620)
              at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:88)
              at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
              at java.lang.reflect.Method.invoke(Method.java:620)
      
      

      What is the problem?
      Engine is crashing.

      What would be the expected behavior:
      Engine should start successfully.

      Additional info:

      1. SystemOut-7.10.3.log
        345 kB
        Nikola Koevski
      2. SystemOut-7.10.3-extract.log
        130 kB
        Nikola Koevski
      3. SystemOut -extract.log
        116 kB
        Nikola Koevski

        Issue Links

          Activity

          Hide
          thorben.lindhauer Thorben Lindhauer added a comment - - edited

          Analysis result: This is a bug in Websphere, but we can implement a workaround.

          Bug in Websphere:

          • The job execution resource adapter fails to be activated. The reason is a NullPointerException in the #toString method of the class com.ibm.ejs.j2c.ActivationSpecBindingInfo. ActivationSpecBindingInfo seems to be a Websphere internal wrapper class around activation spec classes, in our case org.camunda.bpm.container.impl.threading.ra.inflow.JobExecutionHandlerActivationSpec.
          • The #toString method is only called if trace logging is activated. I found this by debugging the corresponding websphere jar
          • The problem occurs when an activation spec is defined that defines no properties. ActivationSpecBindingInfo then wrongfully does not initialize a properties hashmap, but in #toString tries to iterate it

          Stacktrace of the NPE (cannot be found in the logs as is, maybe it can be by activating another logger; I obtained it from the debugger as a variable in the stack):

          [com.ibm.ejs.j2c.ActivationSpecBindingInfo.toString(ActivationSpecBindingInfo.java:202), java.lang.String.valueOf(String.java:1862), java.lang.StringBuilder.append(StringBuilder.java:218), com.ibm.ejs.j2c.ActivationSpecBindingInfo.initialize(ActivationSpecBindingInfo.java:188), com.ibm.ejs.j2c.RALifeCycleManagerImpl.startRA(RALifeCycleManagerImpl.java:1309), com.ibm.ejs.j2c.RALifeCycleManagerImpl.resourceProviderEvent(RALifeCycleManagerImpl.java:923), com.ibm.ws.runtime.component.ResourceMgrImpl$ResourceEventListenerFilterProxy.resourceProviderEvent(ResourceMgrImpl.java:2208), com.ibm.ws.runtime.component.ResourceMgrImpl.notifyResourceEventListeners(ResourceMgrImpl.java:1622), com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1579), com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1462), com.ibm.ejs.j2c.RALifeCycleManagerImpl.start(RALifeCycleManagerImpl.java:720), com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1246), com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1514), com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:704), com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1096), com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:798), com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2314), com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5398), com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5614), com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255), com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2319), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436), com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127), com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:985), com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502), com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)]
          

          Workaround:

          • declare a dummy POJO property in JobExecutionHandlerActivationSpec, e.g. as follows:
            private String foo;
            
            public void setFoo(String foo) {
              this.foo = foo;
            }
            
            public String getFoo() {
              return foo;
            }
          
          Show
          thorben.lindhauer Thorben Lindhauer added a comment - - edited Analysis result: This is a bug in Websphere, but we can implement a workaround. Bug in Websphere: The job execution resource adapter fails to be activated. The reason is a NullPointerException in the #toString method of the class com.ibm.ejs.j2c.ActivationSpecBindingInfo . ActivationSpecBindingInfo seems to be a Websphere internal wrapper class around activation spec classes, in our case org.camunda.bpm.container.impl.threading.ra.inflow.JobExecutionHandlerActivationSpec . The #toString method is only called if trace logging is activated. I found this by debugging the corresponding websphere jar The problem occurs when an activation spec is defined that defines no properties. ActivationSpecBindingInfo then wrongfully does not initialize a properties hashmap, but in #toString tries to iterate it Stacktrace of the NPE (cannot be found in the logs as is, maybe it can be by activating another logger; I obtained it from the debugger as a variable in the stack): [com.ibm.ejs.j2c.ActivationSpecBindingInfo.toString(ActivationSpecBindingInfo.java:202), java.lang. String .valueOf( String .java:1862), java.lang.StringBuilder.append(StringBuilder.java:218), com.ibm.ejs.j2c.ActivationSpecBindingInfo.initialize(ActivationSpecBindingInfo.java:188), com.ibm.ejs.j2c.RALifeCycleManagerImpl.startRA(RALifeCycleManagerImpl.java:1309), com.ibm.ejs.j2c.RALifeCycleManagerImpl.resourceProviderEvent(RALifeCycleManagerImpl.java:923), com.ibm.ws.runtime.component.ResourceMgrImpl$ResourceEventListenerFilterProxy.resourceProviderEvent(ResourceMgrImpl.java:2208), com.ibm.ws.runtime.component.ResourceMgrImpl.notifyResourceEventListeners(ResourceMgrImpl.java:1622), com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1579), com.ibm.ws.runtime.component.ResourceMgrImpl.installResourceProvider(ResourceMgrImpl.java:1462), com.ibm.ejs.j2c.RALifeCycleManagerImpl.start(RALifeCycleManagerImpl.java:720), com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1246), com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1514), com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:704), com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1096), com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:798), com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2314), com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5398), com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5614), com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255), com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2319), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436), com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379), com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127), com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:985), com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502), com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)] Workaround: declare a dummy POJO property in JobExecutionHandlerActivationSpec , e.g. as follows: private String foo; public void setFoo( String foo) { this .foo = foo; } public String getFoo() { return foo; }

            People

            • Assignee:
              yana.vasileva Yana Vasileva
              Reporter:
              nikola.koevski Nikola Koevski
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development