Java core dump with openj9

[zjingjbj]

Java -version output

openjdk version "11.0.4" 2019-07-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.4+11)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.15.1, JRE 11 Linux amd64-64-Bit Compressed References 20190717_286 (JIT enabled, AOT enabled)
OpenJ9 - 0f66c64
OMR - ec782f26
JCL - fa49279450 based on jdk-11.0.4+11)

Summary of problem

Java crashed with "gpf" error when running our liberty application. And it caused by VM class table lock. This happens inside docker container and standalone liberty. This crash will happen every time with openj9 but no error with openjdk with version like 11, 11.0.6 and 13.0.2. For docker container environment, the os is:
Linux 2e6aaa8a2cd2 3.10.0-862.14.4.el7.x86_64 #1 SMP Fri Sep 21 09:07:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

For standalone environment, the os is:
3.10.0-862.14.4.el7.x86_64 #1 SMP Fri Sep 21 09:07:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Diagnostic files

Below are logs extracted from javacore.txt which shows the VM class lock when such problem happens every time.

2LKREGMON          VM class table lock (0x00007F271C006948): Flat locked by "Default Executor-thread-13" (J9VMThread:0x0000000002DAA600), entry count 1
3LKWAITERQ            Waiting to enter:
3LKWAITER                "JIT Compilation Thread-0" (J9VMThread:0x000000000226C600)
3LKWAITER                "JIT Compilation Thread-1" (J9VMThread:0x0000000002270200)
3LKWAITER                "JIT-SamplerThread" (J9VMThread:0x000000000228A500)
3LKWAITER                "Active Thread: Equinox Container: 9c152571-8e06-46b5-9f41-5b57d1100206" (J9VMThread:0x00000000029BF300)
3LKWAITER                "Config-Thread-12" (J9VMThread:0x0000000002BAF100)
3LKWAITER                "Executor Service Control Timer" (J9VMThread:0x0000000002C04200)
3LKWAITER                "Default Executor-thread-2" (J9VMThread:0x0000000002C47600)
3LKWAITER                "Default Executor-thread-7" (J9VMThread:0x0000000002CA2600)
3LKWAITER                "ITCAM_GPE_Pooled_Thread : 3" (J9VMThread:0x0000000002CCE400)
3LKWAITER                "Default Executor-thread-10" (J9VMThread:0x0000000002D30400)
3LKWAITER                "Default Executor-thread-11" (J9VMThread:0x0000000002D6CB00)
3LKWAITER                "Default Executor-thread-16" (J9VMThread:0x0000000002E1FA00)
3LKWAITER                "Default Executor-thread-20" (J9VMThread:0x0000000003721600)

VM monitor table lock (0x00007F271C0A3588): Flat locked by "Default Executor-thread-13" (J9VMThread:0x0000000002DAA600), entry count 1

3XMTHREADINFO      "Default Executor-thread-13" J9VMThread:0x0000000002DAA600, omrthread_t:0x00007F261809D8C8, java/lang/Thread:0x00000000D1F2F760, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x4E, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x235F, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000020)
3XMTHREADINFO2            (native stack address range from:0x00007F26F56BA000, to:0x00007F26F56FA000, size:0x40000)
3XMCPUTIME               CPU usage total: 1.116342349 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=66160 (0x10270)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ClassLoader.defineClassImpl(Native Method)
4XESTACKTRACE                at java/lang/ClassLoader.defineClassInternal(Bytecode PC:107(Compiled Code))
4XESTACKTRACE                at java/lang/ClassLoader.defineClass(Bytecode PC:9(Compiled Code))
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl$TransletClassLoader.defineClass(Bytecode PC:7)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.defineTransletClasses(Bytecode PC:262)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.getTransletInstance(Bytecode PC:17)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.newTransformer(Bytecode PC:5)
5XESTACKTRACE                   (entered lock: com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl@0x00000000FF54E510, entry count: 1)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TransformerFactoryImpl.newTransformer(Bytecode PC:9)
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/mbeans/PluginGenerator.generateXML(PluginGenerator.java:783)
5XESTACKTRACE                   (entered lock: com/ibm/ws/webcontainer/osgi/mbeans/PluginGenerator@0x00000000FF4E4FF0, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/mbeans/GeneratePluginConfigMBean.generatePluginConfig(GeneratePluginConfigMBean.java:160)
5XESTACKTRACE                   (entered lock: com/ibm/ws/webcontainer/osgi/mbeans/GeneratePluginConfigMBean@0x00000000D25676E8, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/mbeans/GeneratePluginConfigMBean.generatePluginConfig(GeneratePluginConfigMBean.java:262)
4XESTACKTRACE                at com/ibm/ws/webserver/plugin/runtime/listeners/GeneratePluginConfigListener$1.run(GeneratePluginConfigListener.java:281)
4XESTACKTRACE                at com/ibm/ws/threading/internal/ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)
4XESTACKTRACE                at java/util/concurrent/Executors$RunnableAdapter.call(Bytecode PC:6(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/FutureTask.run(Bytecode PC:41(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor.runWorker(Bytecode PC:94(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$Worker.run(Bytecode PC:5)
4XESTACKTRACE                at java/lang/Thread.run(Bytecode PC:13)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               (0x00007F271BBD09D2 [libj9prt29.so+0x3a9d2])
4XENATIVESTACK               (0x00007F271BBB5283 [libj9prt29.so+0x1f283])
4XENATIVESTACK               (0x00007F271BBD0A4E [libj9prt29.so+0x3aa4e])
4XENATIVESTACK               (0x00007F271BBD0B44 [libj9prt29.so+0x3ab44])
4XENATIVESTACK               (0x00007F271BBB5283 [libj9prt29.so+0x1f283])
4XENATIVESTACK               (0x00007F271BBD08AB [libj9prt29.so+0x3a8ab])
4XENATIVESTACK               (0x00007F271BBCCF32 [libj9prt29.so+0x36f32])
4XENATIVESTACK               (0x00007F271BBCDCD4 [libj9prt29.so+0x37cd4])
4XENATIVESTACK               (0x00007F271BBB5283 [libj9prt29.so+0x1f283])
4XENATIVESTACK               (0x00007F271523CFF6 [libj9dmp29.so+0x19ff6])
4XENATIVESTACK               (0x00007F271523D18D [libj9dmp29.so+0x1a18d])
4XENATIVESTACK               (0x00007F271BBB5283 [libj9prt29.so+0x1f283])
4XENATIVESTACK               (0x00007F27152398DD [libj9dmp29.so+0x168dd])
4XENATIVESTACK               (0x00007F27152353BD [libj9dmp29.so+0x123bd])
4XENATIVESTACK               (0x00007F271BBB5283 [libj9prt29.so+0x1f283])
4XENATIVESTACK               (0x00007F27152361DE [libj9dmp29.so+0x131de])
4XENATIVESTACK               (0x00007F271523F53C [libj9dmp29.so+0x1c53c])
4XENATIVESTACK               (0x00007F2715227C8D [libj9dmp29.so+0x4c8d])
4XENATIVESTACK               (0x00007F27152272A5 [libj9dmp29.so+0x42a5])
4XENATIVESTACK               (0x00007F271BBB5283 [libj9prt29.so+0x1f283])
4XENATIVESTACK               (0x00007F271522A89B [libj9dmp29.so+0x789b])
4XENATIVESTACK               (0x00007F271522AA1C [libj9dmp29.so+0x7a1c])
4XENATIVESTACK               (0x00007F271524115B [libj9dmp29.so+0x1e15b])
4XENATIVESTACK               (0x00007F2720312ED2 [libj9vm29.so+0x9eed2])
4XENATIVESTACK               (0x00007F271BBB5283 [libj9prt29.so+0x1f283])
4XENATIVESTACK               (0x00007F27203130C8 [libj9vm29.so+0x9f0c8])
4XENATIVESTACK               (0x00007F271BBB474A [libj9prt29.so+0x1e74a])
4XENATIVESTACK               (0x00007F2721F7C5D0 [libpthread.so.0+0xf5d0])
4XENATIVESTACK               (0x00007F27218E68B0 [libc.so.6+0x15b8b0])
4XENATIVESTACK               (0x00007F27202F14A5 [libj9vm29.so+0x7d4a5])
4XENATIVESTACK               (0x00007F272036F6AD [libj9vm29.so+0xfb6ad])
4XENATIVESTACK               (0x00007F27203702BE [libj9vm29.so+0xfc2be])
4XENATIVESTACK               (0x00007F2712E55A7D [libjclse29.so+0x1da7d])
4XENATIVESTACK               Java_java_lang_ClassLoader_defineClassImpl+0x187 (0x00007F2712E491B7 [libjclse29.so+0x111b7])
4XENATIVESTACK               (0x00007F26FD3D049B [<unknown>+0x0])
NULL
NULL           

[DanHeidinga]

@zjingjbj Can you try with the most recent JDK11 release?

The java -version output shows the issue occurs with the openj9-0.15.1 release and the latest is the jdk-11.0.6+10 build with openj9 version openj9-0.18.1.

The latest build can be downloaded from https://adoptopenjdk.net/?variant=openjdk11&jvmVariant=openj9

[zjingjbj]

@DanHeidinga I've tried JDK11 but the same error happened in java core dump:

java -version
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.18.1, JRE 11 Linux amd64-64-Bit Compressed References 20200122_441 (JIT enabled, AOT enabled)
OpenJ9 - 51a5857
OMR - 7a1b0239a
JCL - da35e0c380 based on jdk-11.0.6+10)

In jitdump.20200303.173208.27991.0004.dmp, there is

<jitDump>
#INFO: Crashed in application thread 00000000021A0800.
#INFO: Found 4 JITed methods on Java stack.
<logRecompilation>
<compile
        method="java/lang/ClassLoader.defineClass(Ljava/lang/String;[BIILjava/security/ProtectionDomain;)Ljava/lang/Class;"
        hotness="warm"
        isProfilingCompile=0>
</compile>

In javacore.20200303.173208.27991.0002.txt, there is

2LKREGMON          VM class table lock (0x00007FBE040068E8): Flat locked by "Default Executor-thread-7" (J9VMThread:0x00000000021A0800), entry count 1
3LKWAITERQ            Waiting to enter:
3LKWAITER                "JIT Compilation Thread-000" (J9VMThread:0x0000000001685600)
3LKWAITER                "JIT Compilation Thread-001" (J9VMThread:0x0000000001689200)
3LKWAITER                "JIT Compilation Thread-002" (J9VMThread:0x000000000168CE00)
3LKWAITER                "JIT-SamplerThread" (J9VMThread:0x00000000016A3500)
3LKWAITER                "ITCAM_GPE_Pooled_Thread : 0" (J9VMThread:0x0000000001B4C500)
3LKWAITER                "Active Thread: Equinox Container: ec98237b-fe80-4551-94fd-c141235db0e9" (J9VMThread:0x0000000001DE2E00)
3LKWAITER                "Timer-3" (J9VMThread:0x0000000001F70C00)
3LKWAITER                "Config-Thread-10" (J9VMThread:0x0000000001FC2C00)
3LKWAITER                "Executor Service Control Timer" (J9VMThread:0x0000000002012D00)
3LKWAITER                "Default Executor-thread-10" (J9VMThread:0x00000000021ABC00)
3LKWAITER                "Default Executor-thread-11" (J9VMThread:0x00000000021D2D00)
3LKWAITER                "Default Executor-thread-12" (J9VMThread:0x000000000224A400)
3LKWAITER                "Default Executor-thread-15" (J9VMThread:0x000000000235D800)
3LKWAITER                "Default Executor-thread-16" (J9VMThread:0x00000000023C0500)
...
2LKREGMON          VM monitor table lock (0x00007FBE040A72C8): Flat locked by "Default Executor-thread-7" (J9VMThread:0x00000000021A0800), entry count 1
...
1XMCURTHDINFO  Current thread
3XMTHREADINFO      "Default Executor-thread-7" J9VMThread:0x00000000021A0800, omrthread_t:0x00007FBD3401A5D0, java/lang/Thread:0x00000000D0A4C9A0, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x45, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x70A6, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000020)
3XMTHREADINFO2            (native stack address range from:0x00007FBDE0173000, to:0x00007FBDE01B3000, size:0x40000)
3XMCPUTIME               CPU usage total: 1.447732976 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=975248 (0xEE190)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ClassLoader.defineClassImpl(Native Method)
4XESTACKTRACE                at java/lang/ClassLoader.defineClassInternal(ClassLoader.java:476(Compiled Code))
4XESTACKTRACE                at java/lang/ClassLoader.defineClass(ClassLoader.java:437(Compiled Code))
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl$TransletClassLoader.defineClass(TemplatesImpl.java:207)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.defineTransletClasses(TemplatesImpl.java:514)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.getTransletInstance(TemplatesImpl.java:551)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.newTransformer(TemplatesImpl.java:584)
5XESTACKTRACE                   (entered lock: com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl@0x00000000FD5EF7F8, entry count: 1)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:818)
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/mbeans/PluginGenerator.generateXML(PluginGenerator.java:783)
5XESTACKTRACE                   (entered lock: com/ibm/ws/webcontainer/osgi/mbeans/PluginGenerator@0x00000000FFA90650, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/mbeans/GeneratePluginConfigMBean.generatePluginConfig(GeneratePluginConfigMBean.java:160)
5XESTACKTRACE                   (entered lock: com/ibm/ws/webcontainer/osgi/mbeans/GeneratePluginConfigMBean@0x00000000D2896670, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/mbeans/GeneratePluginConfigMBean.generatePluginConfig(GeneratePluginConfigMBean.java:262)
4XESTACKTRACE                at com/ibm/ws/webserver/plugin/runtime/listeners/GeneratePluginConfigListener$1.run(GeneratePluginConfigListener.java:281)
4XESTACKTRACE                at com/ibm/ws/threading/internal/ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)
4XESTACKTRACE                at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:515(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/FutureTask.run(FutureTask.java:264(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:834)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               (0x00007FBE096CAF22 [libj9prt29.so+0x4ef22])
4XENATIVESTACK               (0x00007FBE0969CDF3 [libj9prt29.so+0x20df3])
4XENATIVESTACK               (0x00007FBE096CAF9E [libj9prt29.so+0x4ef9e])
4XENATIVESTACK               (0x00007FBE096CB094 [libj9prt29.so+0x4f094])
4XENATIVESTACK               (0x00007FBE0969CDF3 [libj9prt29.so+0x20df3])
4XENATIVESTACK               (0x00007FBE096CADFB [libj9prt29.so+0x4edfb])
4XENATIVESTACK               (0x00007FBE096C7422 [libj9prt29.so+0x4b422])
4XENATIVESTACK               (0x00007FBE096C81F4 [libj9prt29.so+0x4c1f4])
4XENATIVESTACK               (0x00007FBE0969CDF3 [libj9prt29.so+0x20df3])
4XENATIVESTACK               (0x00007FBE0924BD16 [libj9dmp29.so+0x1ad16])
4XENATIVESTACK               (0x00007FBE0924BEAD [libj9dmp29.so+0x1aead])
4XENATIVESTACK               (0x00007FBE0969CDF3 [libj9prt29.so+0x20df3])
4XENATIVESTACK               (0x00007FBE0924859D [libj9dmp29.so+0x1759d])
4XENATIVESTACK               (0x00007FBE0924387D [libj9dmp29.so+0x1287d])
4XENATIVESTACK               (0x00007FBE0969CDF3 [libj9prt29.so+0x20df3])
4XENATIVESTACK               (0x00007FBE09244C20 [libj9dmp29.so+0x13c20])
4XENATIVESTACK               (0x00007FBE0924E2BC [libj9dmp29.so+0x1d2bc])
4XENATIVESTACK               (0x00007FBE09235E7D [libj9dmp29.so+0x4e7d])
4XENATIVESTACK               (0x00007FBE09235475 [libj9dmp29.so+0x4475])
4XENATIVESTACK               (0x00007FBE0969CDF3 [libj9prt29.so+0x20df3])
4XENATIVESTACK               (0x00007FBE09238B8B [libj9dmp29.so+0x7b8b])
4XENATIVESTACK               (0x00007FBE09238D1F [libj9dmp29.so+0x7d1f])
4XENATIVESTACK               (0x00007FBE0925002B [libj9dmp29.so+0x1f02b])
4XENATIVESTACK               (0x00007FBE09DB0C52 [libj9vm29.so+0x98c52])
4XENATIVESTACK               (0x00007FBE0969CDF3 [libj9prt29.so+0x20df3])
4XENATIVESTACK               (0x00007FBE09DB0E65 [libj9vm29.so+0x98e65])
4XENATIVESTACK               (0x00007FBE0969C2BA [libj9prt29.so+0x202ba])
4XENATIVESTACK               (0x00007FBE0B9F6630 [libpthread.so.0+0xf630])
4XENATIVESTACK               (0x00007FBE0B3604B0 [libc.so.6+0x15c4b0])
4XENATIVESTACK               (0x00007FBE09D8F015 [libj9vm29.so+0x77015])
4XENATIVESTACK               (0x00007FBE09E0EB26 [libj9vm29.so+0xf6b26])
4XENATIVESTACK               (0x00007FBE09E0F81E [libj9vm29.so+0xf781e])
4XENATIVESTACK               (0x00007FBDFC3C20AF [libjclse29.so+0x1e0af])
4XENATIVESTACK               Java_java_lang_ClassLoader_defineClassImpl+0x187 (0x00007FBDFC3B5307 [libjclse29.so+0x11307])
4XENATIVESTACK               (0x00007FBDE3F12137 [<unknown>+0x0])
NULL
NULL

[pshipton]

Do you have the crash output from the console? Also, please zip up and attach the entire diagnostic files.

[zjingjbj]

@pshipton Thanks for the feedback! Can I send the log your private mail box? I've sent the core dump logs to your mail box. Please help to investigate.

[pshipton]

Thanks, I have received the javacore/jitdump/Snap and put them into triage/openj9-8724.

Are you able to reproduce the problem with the following build?
https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Nightly/307/OpenJ9-JDK11-x86-64_linux-20200304-030700.tar.gz
https://ci.eclipse.org/openj9/job/Build_JDK11_x86-64_linux_Nightly/307/

If so, pls compress the core file and other diagnostic files and make this available using a file sharing service.

[zjingjbj]

@pshipton I've tried the build you provided but same error occurred.

Here is the java version:

./java -version
openjdk version "11.0.7-internal" 2020-04-14
OpenJDK Runtime Environment (build 11.0.7-internal+0-adhoc.jenkins.BuildJDK11x86-64linuxNightly)
Eclipse OpenJ9 VM (build master-4e9fab784, JRE 11 Linux amd64-64-Bit Compressed References 20200303_307 (JIT enabled, AOT enabled)
OpenJ9 - 4e9fab7
OMR - f955e91
JCL - e0e94184e0b based on jdk-11.0.7+5)

In jitdump.20200304.173251.10435.0004.dmp, there is:
#INFO: Crashed in application thread 000000000150AD00.

And in javacore.20200304.173251.10435.0002.txt, there is:

2LKREGMON          VM class table lock (0x00007FDB4C006B08): Flat locked by "Default Executor-thread-14" (J9VMThread:0x000000000150AD00), entry count 1
3LKWAITERQ            Waiting to enter:
3LKWAITER                "JIT Compilation Thread-000" (J9VMThread:0x00000000008D7600)
3LKWAITER                "JIT Compilation Thread-001" (J9VMThread:0x00000000008DB200)
3LKWAITER                "JIT Compilation Thread-002" (J9VMThread:0x00000000008DEE00)
3LKWAITER                "JIT Compilation Thread-003" (J9VMThread:0x00000000008E2900)
3LKWAITER                "JIT Compilation Thread-004" (J9VMThread:0x00000000008E6500)
3LKWAITER                "JIT-SamplerThread" (J9VMThread:0x00000000008F5500)
3LKWAITER                "Active Thread: Equinox Container: fb9de15b-5aad-4e30-9b79-d73f45c7948e" (J9VMThread:0x0000000001045100)
3LKWAITER                "Config-Thread-10" (J9VMThread:0x000000000120C800)
3LKWAITER                "Executor Service Control Timer" (J9VMThread:0x0000000001265B00)
3LKWAITER                "Default Executor-thread-8" (J9VMThread:0x00000000013FEB00)
3LKWAITER                "Default Executor-thread-10" (J9VMThread:0x0000000001406300)
3LKWAITER                "Default Executor-thread-17" (J9VMThread:0x00000000013FA000)
3LKWAITER                "Default Executor-thread-18" (J9VMThread:0x0000000001C01F00)
3LKWAITER                "Default Executor-thread-21" (J9VMThread:0x0000000001C21400)
...
2LKREGMON          VM monitor table lock (0x00007FDB4C0A2918): Flat locked by "Default Executor-thread-14" (J9VMThread:0x000000000150AD00), entry count 1
...
1XMCURTHDINFO  Current thread
3XMTHREADINFO      "Default Executor-thread-14" J9VMThread:0x000000000150AD00, omrthread_t:0x00007FDA80092578, java/lang/Thread:0x00000000D1E11C90, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x51, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x3062, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000020)
3XMTHREADINFO2            (native stack address range from:0x00007FDAFDE37000, to:0x00007FDAFDE77000, size:0x40000)
3XMCPUTIME               CPU usage total: 1.154377493 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=588648 (0x8FB68)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ClassLoader.defineClassImpl(Native Method)
4XESTACKTRACE                at java/lang/ClassLoader.defineClassInternal(ClassLoader.java:476(Compiled Code))
4XESTACKTRACE                at java/lang/ClassLoader.defineClass(ClassLoader.java:437(Compiled Code))
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl$TransletClassLoader.defineClass(TemplatesImpl.java:207)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.defineTransletClasses(TemplatesImpl.java:514)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.getTransletInstance(TemplatesImpl.java:551)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl.newTransformer(TemplatesImpl.java:584)
5XESTACKTRACE                   (entered lock: com/sun/org/apache/xalan/internal/xsltc/trax/TemplatesImpl@0x00000000FFB3DA18, entry count: 1)
4XESTACKTRACE                at com/sun/org/apache/xalan/internal/xsltc/trax/TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:818)
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/mbeans/PluginGenerator.generateXML(PluginGenerator.java:783)
5XESTACKTRACE                   (entered lock: com/ibm/ws/webcontainer/osgi/mbeans/PluginGenerator@0x00000000FFB24A78, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/mbeans/GeneratePluginConfigMBean.generatePluginConfig(GeneratePluginConfigMBean.java:160)
5XESTACKTRACE                   (entered lock: com/ibm/ws/webcontainer/osgi/mbeans/GeneratePluginConfigMBean@0x00000000D28E47F0, entry count: 1)
4XESTACKTRACE                at com/ibm/ws/webcontainer/osgi/mbeans/GeneratePluginConfigMBean.generatePluginConfig(GeneratePluginConfigMBean.java:262)
4XESTACKTRACE                at com/ibm/ws/webserver/plugin/runtime/listeners/GeneratePluginConfigListener$1.run(GeneratePluginConfigListener.java:281)
4XESTACKTRACE                at com/ibm/ws/threading/internal/ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)
4XESTACKTRACE                at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:515(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/FutureTask.run(FutureTask.java:264(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:834)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               (0x00007FDB5024FB52 [libj9prt29.so+0x4fb52])
4XENATIVESTACK               (0x00007FDB50221573 [libj9prt29.so+0x21573])
4XENATIVESTACK               (0x00007FDB5024FBCE [libj9prt29.so+0x4fbce])
4XENATIVESTACK               (0x00007FDB5024FCC4 [libj9prt29.so+0x4fcc4])
4XENATIVESTACK               (0x00007FDB50221573 [libj9prt29.so+0x21573])
4XENATIVESTACK               (0x00007FDB5024FA2B [libj9prt29.so+0x4fa2b])
4XENATIVESTACK               (0x00007FDB5024C052 [libj9prt29.so+0x4c052])
4XENATIVESTACK               (0x00007FDB5024CE24 [libj9prt29.so+0x4ce24])
4XENATIVESTACK               (0x00007FDB50221573 [libj9prt29.so+0x21573])
4XENATIVESTACK               (0x00007FDB456A5F86 [libj9dmp29.so+0x1af86])
4XENATIVESTACK               (0x00007FDB456A611D [libj9dmp29.so+0x1b11d])
4XENATIVESTACK               (0x00007FDB50221573 [libj9prt29.so+0x21573])
4XENATIVESTACK               (0x00007FDB456A280D [libj9dmp29.so+0x1780d])
4XENATIVESTACK               (0x00007FDB4569DAED [libj9dmp29.so+0x12aed])
4XENATIVESTACK               (0x00007FDB50221573 [libj9prt29.so+0x21573])
4XENATIVESTACK               (0x00007FDB4569EE90 [libj9dmp29.so+0x13e90])
4XENATIVESTACK               (0x00007FDB456A85BC [libj9dmp29.so+0x1d5bc])
4XENATIVESTACK               (0x00007FDB4569004D [libj9dmp29.so+0x504d])
4XENATIVESTACK               (0x00007FDB4568F645 [libj9dmp29.so+0x4645])
4XENATIVESTACK               (0x00007FDB50221573 [libj9prt29.so+0x21573])
4XENATIVESTACK               (0x00007FDB45692D8B [libj9dmp29.so+0x7d8b])
4XENATIVESTACK               (0x00007FDB45692F1F [libj9dmp29.so+0x7f1f])
4XENATIVESTACK               (0x00007FDB456AA33B [libj9dmp29.so+0x1f33b])
4XENATIVESTACK               (0x00007FDB5092F222 [libj9vm29.so+0x92222])
4XENATIVESTACK               (0x00007FDB50221573 [libj9prt29.so+0x21573])
4XENATIVESTACK               (0x00007FDB5092F435 [libj9vm29.so+0x92435])
4XENATIVESTACK               (0x00007FDB50220A3A [libj9prt29.so+0x20a3a])
4XENATIVESTACK               (0x00007FDB52577630 [libpthread.so.0+0xf630])
4XENATIVESTACK               (0x00007FDB51EE14B0 [libc.so.6+0x15c4b0])
4XENATIVESTACK               (0x00007FDB5090D4D5 [libj9vm29.so+0x704d5])
4XENATIVESTACK               (0x00007FDB5098D7FD [libj9vm29.so+0xf07fd])
4XENATIVESTACK               (0x00007FDB5098E51E [libj9vm29.so+0xf151e])
4XENATIVESTACK               (0x00007FDB42F61525 [libjclse29.so+0x1e525])
4XENATIVESTACK               Java_java_lang_ClassLoader_defineClassImpl+0x175 (0x00007FDB42F545E5 [libjclse29.so+0x115e5])
4XENATIVESTACK               (0x00007FDB2B4467A0 [<unknown>+0x0])

Since the core dump file is too big, I cannot attach it here directly.

[pshipton]

Ok, you can compress the files and make them available using a file sharing service, such as Box.

[pshipton]

diagnostics in triage/openj9-8724

#12 <signal handler called>
#13 0x00007fdb51ee14b0 in __mempcpy_ssse3 () from /team/triage/openj9-8724/libs/libc.so.6
#14 0x00007fdb5090d4d5 in findModuleForPackage (currentThread=currentThread@entry=0x150ad00, 
    classLoader=0x7fdb4c549e48, packageName=0x0, packageNameLen=15) at ModularityHashTables.c:187
#15 0x00007fdb5098d7fd in callDynamicLoader (localBuffer=0x7fdafde73900, classFileBytesReplacedByRCA=0, 
    classFileBytesReplacedByRIA=1, translationFlags=923140608, 
    intermediateClassDataLength=<optimized out>, intermediateClassData=0x7fda28300900 "\312\376\272\276", 
    loadData=0x7fdafde73790, vmThread=0x150ad00) at defineclass.c:769
#16 internalLoadROMClass (vmThread=vmThread@entry=0x150ad00, loadData=loadData@entry=0x7fdafde73790, 
    localBuffer=localBuffer@entry=0x7fdafde73900) at defineclass.c:561
#17 0x00007fdb5098e51e in createROMClassFromClassFile (localBuffer=0x7fdafde73900, 
    loadData=0x7fdafde73790, currentThread=0x150ad00) at defineclass.c:824
#18 internalDefineClass (vmThread=0x150ad00, className=0x0, classNameLength=0, classData=<optimized out>, 
    classDataLength=<optimized out>, classDataObject=<optimized out>, classLoader=0x7fdb4c549e48, 
    protectionDomain=0xfdbe8d50, options=4097, existingROMClass=0x0, hostClass=0x0, 
    localBuffer=0x7fdafde73900) at defineclass.c:141
#19 0x00007fdb42f61525 in defineClassCommon (env=env@entry=0x150ad00, 
    classLoaderObject=classLoaderObject@entry=0x1c3ced0, className=className@entry=0x0, 
    classRep=classRep@entry=0x1c3cec0, offset=offset@entry=0, length=4619, protectionDomain=0x1c3cd98, 
    options=0, hostClass=0x0, patchMap=0x0) at common/jcldefine.c:196
#20 0x00007fdb42f545e5 in Java_java_lang_ClassLoader_defineClassImpl (env=0x150ad00, receiver=0x1c3ced0, 
    className=<optimized out>, classRep=0x1c3cec0, offset=0, length=<optimized out>, 
    protectionDomain=0x1c3cd98) at common/clsldr.cpp:73

[pshipton]

@zjingjbj can you please try the following JVM and let me know what happens. I think it will fix the crash, although there may be another problem as well.
https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK11_x86-64_linux_Personal/706/OpenJ9-JDK11-x86-64_linux-20200306-030341.tar.gz
https://ci.eclipse.org/openj9/job/Build_JDK11_x86-64_linux_Personal/706/

[zjingjbj]

@pshipton I've tried the new build but the core dump happened again. I'll send the collected log to box.

[pshipton]

It's the same crash, my change didn't make any difference.

#12 <signal handler called>
#13 0x00007fbb494464b0 in __mempcpy_ssse3 () from /team/triage/openj9-8724/crash3/libs/libc.so.6
#14 0x00007fbb43ce84d5 in findModuleForPackage (currentThread=currentThread@entry=0x19e3a00, classLoader=0x7fba1c161188,
    packageName=0x0, packageNameLen=15) at ModularityHashTables.c:187
#15 0x00007fbb43d68cbb in callDynamicLoader (localBuffer=0x7fbaf520c900, classFileBytesReplacedByRCA=0,
    classFileBytesReplacedByRIA=1, translationFlags=923140608, intermediateClassDataLength=<optimized out>,
    intermediateClassData=0x7fba18335c20 "\312\376\272\276", loadData=0x7fbaf520c790, vmThread=0x19e3a00) at defineclass.c:769
#16 internalLoadROMClass (vmThread=vmThread@entry=0x19e3a00, loadData=loadData@entry=0x7fbaf520c790,
    localBuffer=localBuffer@entry=0x7fbaf520c900) at defineclass.c:561
#17 0x00007fbb43d694fe in createROMClassFromClassFile (localBuffer=0x7fbaf520c900, loadData=0x7fbaf520c790,
    currentThread=0x19e3a00) at defineclass.c:823
#18 internalDefineClass (vmThread=0x19e3a00, className=0x0, classNameLength=0, classData=<optimized out>,
    classDataLength=<optimized out>, classDataObject=<optimized out>, classLoader=0x7fba1c161188, protectionDomain=0xff7925d0,
    options=4097, existingROMClass=0x0, hostClass=0x0, localBuffer=0x7fbaf520c900) at defineclass.c:141
#19 0x00007fbb3a542525 in defineClassCommon (env=env@entry=0x19e3a00, classLoaderObject=classLoaderObject@entry=0x1ce3320,
---Type <return> to continue, or q <return> to quit---
    className=className@entry=0x0, classRep=classRep@entry=0x1ce3310, offset=offset@entry=0, length=4619,
    protectionDomain=0x1ce31d0, options=0, hostClass=0x0, patchMap=0x0) at common/jcldefine.c:196
#20 0x00007fbb3a5355e5 in Java_java_lang_ClassLoader_defineClassImpl (env=0x19e3a00, receiver=0x1ce3320,
    className=<optimized out>, classRep=0x1ce3310, offset=0, length=<optimized out>, protectionDomain=0x1ce31d0)
    at common/clsldr.cpp:73