DaaLoadTest_all_special_5m_17 segfault during message handling at JITServer client

[cjjdespres]

Failure link

https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal/1070/

# DETECTED_JAVA_VERSION=openjdk version "11.0.24-internal" 2024-07-16
# OpenJDK Runtime Environment (build 11.0.24-internal+0-adhoc..BuildJDK11x86-64linuxjitPersonal)
# Eclipse OpenJ9 VM (build master-9544e92, JRE 11 Linux amd64-64-Bit Compressed References 20240704_1897 (JIT enabled, AOT enabled)
# OpenJ9   - 9544e92
# OMR      - 47a9d24
# JCL      - 045117e based on jdk-11.0.24+7)
# DETECTED_RELEASE_INFO=IMPLEMENTOR="Eclipse OpenJ9"
# JAVA_RUNTIME_VERSION="11.0.24-internal+0-adhoc..BuildJDK11x86-64linuxjitPersonal"
# JAVA_VERSION="11.0.24"
# JAVA_VERSION_DATE="2024-07-16"

Failure output (captured from console output)

        STF 03:07:10.915 - +------ Step 1 - Run daa load test

        STF 03:07:10.932 - Monitoring processes: DLT
        DLT stderr #0: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x92c9d5) [0x7f41e212c9d5]
        DLT stderr #1: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x938d80) [0x7f41e2138d80]
        DLT stderr #2: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x162919) [0x7f41e1962919]
        DLT stderr #3: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x29978) [0x7f41e309d978]
        DLT stderr #4: /lib64/libc.so.6(+0x54db0) [0x7f41e3254db0]
        DLT stderr #5: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1aeba7) [0x7f41e19aeba7]
        DLT stderr #6: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1b964a) [0x7f41e19b964a]
        DLT stderr #7: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x178ec9) [0x7f41e1978ec9]
        DLT stderr #8: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x179c71) [0x7f41e1979c71]
        DLT stderr #9: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x2a3c9) [0x7f41e309e3c9]
        DLT stderr #10: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1777d5) [0x7f41e19777d5]
        DLT stderr #11: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x177af8) [0x7f41e1977af8]
        DLT stderr #12: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176a27) [0x7f41e1976a27]
        DLT stderr #13: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176d92) [0x7f41e1976d92]
        DLT stderr #14: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176e42) [0x7f41e1976e42]
        DLT stderr #15: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x2a3c9) [0x7f41e309e3c9]
        DLT stderr #16: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x17720b) [0x7f41e197720b]
        DLT stderr #17: /home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0xb3a3) [0x7f41e30663a3]
        DLT stderr #18: /lib64/libc.so.6(+0x9f802) [0x7f41e329f802]
        DLT stderr #19: /lib64/libc.so.6(+0x3f450) [0x7f41e323f450]
        DLT stderr Unhandled exception
        DLT stderr Type=Segmentation error vmState=0x0005ffff
        DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000080
        DLT stderr Handler1=00007F41E2A41FB0 Handler2=00007F41E309D740 InaccessibleAddress=0000000000000000
        DLT stderr RDI=00007F41DC1088B8 RSI=00007F41CD575000 RAX=00007F41CECF8D92 RBX=00007F41DC268590
        DLT stderr RCX=00007F41CECF8DB8 RDX=00000000CD5810D0 R8=00007F41CECF8DBC R9=00007F41CECF8D91
        DLT stderr R10=000000008875CD08 R11=00000000CD5810D0 R12=00007F41D7AF85D0 R13=00007F41D7B13118
        DLT stderr R14=00007F41D7B13100 R15=000000000001E900
        DLT stderr RIP=00007F41E19AEBA7 GS=0000 FS=0000 RSP=00007F41CECF8CB0
        DLT stderr EFlags=0000000000010216 CS=0033 RBP=00007F41CECF9970 ERR=0000000000000000
        DLT stderr TRAPNO=000000000000000D OLDMASK=0000000000000000 CR2=0000000000000000
        DLT stderr xmm0=00007f41cd581c60 (f: 3445103616.000000, d: 6.912996e-310)
        DLT stderr xmm1=0000000000143000 (f: 1323008.000000, d: 6.536528e-318)
        DLT stderr xmm2=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr xmm3=00000387d7010001 (f: 3607166976.000000, d: 1.917944e-311)
        DLT stderr xmm4=00007f41dc10c718 (f: 3692087040.000000, d: 6.913008e-310)
        DLT stderr xmm5=00007f41cecf9800 (f: 3469711360.000000, d: 6.912997e-310)
        DLT stderr xmm6=00007f4100000387 (f: 903.000000, d: 6.912826e-310)
        DLT stderr xmm7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr xmm8=0000320000044c00 (f: 281600.000000, d: 2.716155e-310)
        DLT stderr xmm9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr xmm10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr xmm11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr xmm12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr xmm13=00007fbe1e49e734 (f: 508159808.000000, d: 6.939376e-310)
        DLT stderr xmm14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr xmm15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        DLT stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal_testList_3/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
        DLT stderr Module_base_address=00007F41E1800000
        DLT stderr 
        DLT stderr Method_being_compiled=java/lang/invoke/FoldNonvoidHandle.invokeExact_thunkArchetype_X(I)I
        DLT stderr Target=2_90_20240704_1897 (Linux 5.14.0-350.el9.x86_64)
        DLT stderr CPU=amd64 (4 logical CPUs) (0x1df96c000 RAM)
        DLT stderr ----------- Stack Backtrace -----------

        DLT stderr _ZL19handleServerMessagePN9JITServer12ClientStreamEP7TR_J9VMRNS_11MessageTypeE+0xadd7 (0x00007F41E19AEBA7 [libj9jit29.so+0x1aeba7])
        DLT stderr _Z13remoteCompileP10J9VMThreadPN2TR11CompilationEP17TR_ResolvedMethodP8J9MethodRNS1_24IlGeneratorMethodDetailsEPNS1_28CompilationInfoPerThreadBaseE.localalias+0x131a (0x00007F41E19B964A [libj9jit29.so+0x1b964a])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x749 (0x00007F41E1978EC9 [libj9jit29.so+0x178ec9])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x381 (0x00007F41E1979C71 [libj9jit29.so+0x179c71])
        DLT stderr omrsig_protect+0x239 (0x00007F41E309E3C9 [libj9prt29.so+0x2a3c9])
        DLT stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x385 (0x00007F41E19777D5 [libj9jit29.so+0x1777d5])
        DLT stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x128 (0x00007F41E1977AF8 [libj9jit29.so+0x177af8])
        DLT stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x377 (0x00007F41E1976A27 [libj9jit29.so+0x176a27])
        DLT stderr _ZN2TR24CompilationInfoPerThread3runEv+0x42 (0x00007F41E1976D92 [libj9jit29.so+0x176d92])
        DLT stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x82 (0x00007F41E1976E42 [libj9jit29.so+0x176e42])
        DLT stderr omrsig_protect+0x239 (0x00007F41E309E3C9 [libj9prt29.so+0x2a3c9])
        DLT stderr _Z21compilationThreadProcPv+0x17b (0x00007F41E197720B [libj9jit29.so+0x17720b])
        DLT stderr thread_wrapper+0x163 (0x00007F41E30663A3 [libj9thr29.so+0xb3a3])
        DLT stderr start_thread+0x2d2 (0x00007F41E329F802 [libc.so.6+0x9f802])
        DLT stderr __clone3+0x30 (0x00007F41E323F450 [libc.so.6+0x3f450])

[cjjdespres]

Attn @mpirvu. A new failure I saw today as I started going through the test failures from over the long weekend.

[cjjdespres]

Some other segfaults that could be related:

SharedClasses.SCM01.SingleCL_1 and SharedClassesAPI_1 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal/1079/. Console log for the first failure (second is similar):

        SharedClasses.SCM01.SingleCL_1 Start Time: Thu Jul  4 01:47:07 2024 Epoch Time (ms): 1720082827314
        variation: Mode610
        JVM_OPTIONS: -XX:+UseJITServer -Xcompressedrefs -Xjit -Xgcpolicy:gencon 

        SCL5 stderr Unhandled exception
        SCL5 stderr Type=Segmentation error vmState=0x0005ffff
        SCL5 stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
        SCL5 stderr Handler1=00007887DFD86A80 Handler2=00007887E40CD540
        SCL5 stderr R0=00007887DE49F3BC R1=00007887C26A7D50 R2=00007887DF1DE800 R3=0000000000000000
        SCL5 stderr R4=000000000009A200 R5=0000000000000000 R6=00007887DF212058 R7=0000000000000001
        SCL5 stderr R8=00007887D27D5148 R9=0000000000000001 R10=0000000000000000 R11=00007887D284F980
        SCL5 stderr R12=0000000028002482 R13=00007887C26B68F0 R14=00007887E0269D50 R15=00000000004059A0
        SCL5 stderr R16=00007887C26A9B20 R17=0000000000000000 R18=00007887DF2151D8 R19=00007887C26A9EA0
        SCL5 stderr R20=00007887D27E0360 R21=00007887C26AAA70 R22=00007887C26A9C88 R23=00007887C12D5BC0
        SCL5 stderr R24=00007887C12D0000 R25=00007887E0245BA0 R26=00007887C26A9C70 R27=00000000000000DD
        SCL5 stderr R28=00007887DF2152D0 R29=00007D1F6C0AE550 R30=00007887C12D5BC0 R31=00007887E0072AA5
        SCL5 stderr NIP=00007887DE49F3C8 MSR=800000000280F033 ORIG_GPR3=00007887E40C0C58 CTR=00007887DE41D010
        SCL5 stderr LINK=00007887DE49F3BC XER=0000000000000000 CCR=0000000048002482 SOFTE=0000000000000001
        SCL5 stderr TRAP=0000000000000300 DAR=00007D1F6C0AE550 dsisr=0000000040000000 RESULT=0000000000000000
        SCL5 stderr FPR0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR1=4064e00000000000 (f: 0.000000, d: 1.670000e+02)
        SCL5 stderr FPR2=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR3=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR4=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR5=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR7=ffffffff00000002 (f: 2.000000, d: -nan)
        SCL5 stderr FPR8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR10=0000000500000004 (f: 4.000000, d: 1.060998e-313)
        SCL5 stderr FPR11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR13=00007887df2797f0 (f: 3743913984.000000, d: 6.547603e-310)
        SCL5 stderr FPR14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR16=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR17=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR18=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR19=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR20=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR21=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR22=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR23=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR24=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR25=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR26=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR27=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR28=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR29=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR30=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr FPR31=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        SCL5 stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_extended.system_ppc64le_linux_jit_Personal_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
        SCL5 stderr Module_base_address=00007887DE260000
        SCL5 stderr 
        SCL5 stderr Method_being_compiled=java/lang/invoke/FoldNonvoidHandle.invokeExact_thunkArchetype_X(I)I
        SCL5 stderr Target=2_90_20240704_1791 (Linux 5.4.0-182-generic)
        SCL5 stderr CPU=ppc64le (4 logical CPUs) (0x1fcc00000 RAM)
        SCL5 stderr ----------- Stack Backtrace -----------

        SCL5 stderr _ZN2J916KnownObjectTable18getExistingIndexAtEPm+0x268 (0x00007887DE49F3C8 [libj9jit29.so+0x23f3c8])
        SCL5 stderr _ZL19handleServerMessagePN9JITServer12ClientStreamEP7TR_J9VMRNS_11MessageTypeE+0x16d4 (0x00007887DE41D034 [libj9jit29.so+0x1bd034])
        SCL5 stderr _Z13remoteCompileP10J9VMThreadPN2TR11CompilationEP17TR_ResolvedMethodP8J9MethodRNS1_24IlGeneratorMethodDetailsEPNS1_28CompilationInfoPerThreadBaseE+0x9fc (0x00007887DE42CCBC [libj9jit29.so+0x1cccbc])
        SCL5 stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x60c (0x00007887DE3DCEEC [libj9jit29.so+0x17ceec])
        SCL5 stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x37c (0x00007887DE3DE0EC [libj9jit29.so+0x17e0ec])
        SCL5 stderr omrsig_protect+0x358 (0x00007887E40CE948 [libj9prt29.so+0x3e948])
        SCL5 stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x2fc (0x00007887DE3DB12C [libj9jit29.so+0x17b12c])
        SCL5 stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x264 (0x00007887DE3DB8F4 [libj9jit29.so+0x17b8f4])
        SCL5 stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x38c (0x00007887DE3D9D0C [libj9jit29.so+0x179d0c])
        SCL5 stderr _ZN2TR24CompilationInfoPerThread3runEv+0x50 (0x00007887DE3DA470 [libj9jit29.so+0x17a470])
        SCL5 stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x9c (0x00007887DE3DA57C [libj9jit29.so+0x17a57c])
        SCL5 stderr omrsig_protect+0x358 (0x00007887E40CE948 [libj9prt29.so+0x3e948])
        SCL5 stderr _Z21compilationThreadProcPv+0x234 (0x00007887DE3DAB94 [libj9jit29.so+0x17ab94])
        SCL5 stderr thread_wrapper+0x1a8 (0x00007887E40613E8 [libj9thr29.so+0x113e8])
        SCL5 stderr start_thread+0xe8 (0x00007887E4988838 [libpthread.so.0+0x8838])
        SCL5 stderr clone+0x74 (0x00007887E481BA44 [libc.so.6+0x14ba44])

The following tests have similar segfaults to the one in the log above:

• testSCCacheManagement_0 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_extended.functional_ppc64le_linux_jit_Personal/1084/
• TestIBMJlmRemoteClassAuth_1 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_sanity.system_x86-64_linux_jit_Personal/1075/
• ClassLoadingTest_special_5m_22, LambdaLoadTest_special_J9_5m_20, ClassLoadingTest_special_5m_29 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_special.system_ppc64le_linux_jit_Personal/1073/

There may be others.

[cjjdespres]

The test MiniMix_5m_0 in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal/1070/ crashes during the compilation of java/lang/invoke/DirectHandle.invokeExact_thunkArchetype_D(I)D, but in a slightly different place:

        LT  stderr Type=Segmentation error vmState=0x0005ffff
        LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000080
        LT  stderr Handler1=00007FDE0B302EF0 Handler2=00007FDE10D00740 InaccessibleAddress=0000000000000000
        LT  stderr RDI=00007FDE0C077664 RSI=0400060000000000 RAX=0400060000000000 RBX=00007FDE0C135660
        LT  stderr RCX=00007FDE07234770 RDX=0000000000000000 R8=0000000000000001 R9=0000000000000000
        LT  stderr R10=0000000000000000 R11=0000000000000000 R12=00007FDE0C135660 R13=00007FDE07233730
        LT  stderr R14=00007FDE0755D090 R15=0400060000000000
        LT  stderr RIP=00007FDE0A39683A GS=0000 FS=0000 RSP=00007FDE07231B20
        LT  stderr EFlags=0000000000010202 CS=0033 RBP=00007FDE07233A10 ERR=0000000000000000
        LT  stderr TRAPNO=000000000000000D OLDMASK=0000000000000000 CR2=0000000000000000
        LT  stderr xmm0=00007fde04d2e470 (f: 80929904.000000, d: 6.946145e-310)
        LT  stderr xmm1=0000000000076e00 (f: 486912.000000, d: 2.405665e-318)
        LT  stderr xmm2=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm3=0000036f07010001 (f: 117506048.000000, d: 1.865292e-311)
        LT  stderr xmm4=00007fde0c0a74a8 (f: 202011808.000000, d: 6.946151e-310)
        LT  stderr xmm5=00007fde07233800 (f: 119748608.000000, d: 6.946147e-310)
        LT  stderr xmm6=00007fde0000036f (f: 879.000000, d: 6.946141e-310)
        LT  stderr xmm7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm8=0000000000040900 (f: 264448.000000, d: 1.306547e-318)
        LT  stderr xmm9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm10=3f3f3f3f3f3f3f3f (f: 1061109568.000000, d: 4.767923e-04)
        LT  stderr xmm11=9999999999999999 (f: 2576980480.000000, d: -2.353437e-185)
        LT  stderr xmm12=2020202020202020 (f: 538976256.000000, d: 6.013470e-154)
        LT  stderr xmm13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr Module=/home/jenkins/workspace/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/jre/lib/amd64/default/libj9jit29.so
        LT  stderr Module_base_address=00007FDE0A157000
        LT  stderr 
        LT  stderr Method_being_compiled=java/lang/invoke/DirectHandle.invokeExact_thunkArchetype_D(I)D
        LT  stderr Target=2_90_20240703_1816 (Linux 4.18.0-553.5.1.el8_10.x86_64)
        LT  stderr CPU=amd64 (4 logical CPUs) (0x1e112e000 RAM)
        LT  stderr ----------- Stack Backtrace -----------
        LT  stderr _ZN11TR_J9VMBase10getMethodsEP19TR_OpaqueClassBlock.localalias+0x1a (0x00007FDE0A39683A [libj9jit29.so+0x23f83a])
        LT  stderr _ZN16JITServerHelpers22packRemoteROMClassInfoB5cxx11EP7J9ClassP10J9VMThreadP9TR_Memoryb+0x61 (0x00007FDE0A351D91 [libj9jit29.so+0x1fad91])
        LT  stderr _ZL19handleServerMessagePN9JITServer12ClientStreamEP7TR_J9VMRNS_11MessageTypeE+0x2416 (0x00007FDE0A2FAC46 [libj9jit29.so+0x1a3c46])
        LT  stderr _Z13remoteCompileP10J9VMThreadPN2TR11CompilationEP17TR_ResolvedMethodP8J9MethodRNS1_24IlGeneratorMethodDetailsEPNS1_28CompilationInfoPerThreadBaseE.localalias+0x131a (0x00007FDE0A30E07A [libj9jit29.so+0x1b707a])
        LT  stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x749 (0x00007FDE0A2CEFB9 [libj9jit29.so+0x177fb9])
        LT  stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x381 (0x00007FDE0A2CFD61 [libj9jit29.so+0x178d61])
        LT  stderr omrsig_protect+0x239 (0x00007FDE10D013C9 [libj9prt29.so+0x2a3c9])
        LT  stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x385 (0x00007FDE0A2CD8C5 [libj9jit29.so+0x1768c5])
        LT  stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x128 (0x00007FDE0A2CDBE8 [libj9jit29.so+0x176be8])
        LT  stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x35b (0x00007FDE0A2CCB1B [libj9jit29.so+0x175b1b])
        LT  stderr _ZN2TR24CompilationInfoPerThread3runEv+0x42 (0x00007FDE0A2CCE82 [libj9jit29.so+0x175e82])
        LT  stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x82 (0x00007FDE0A2CCF32 [libj9jit29.so+0x175f32])
        LT  stderr omrsig_protect+0x239 (0x00007FDE10D013C9 [libj9prt29.so+0x2a3c9])
        LT  stderr _Z21compilationThreadProcPv+0x17b (0x00007FDE0A2CD2FB [libj9jit29.so+0x1762fb])
        LT  stderr thread_wrapper+0x163 (0x00007FDE10CC93A3 [libj9thr29.so+0xb3a3])
        LT  stderr start_thread+0xea (0x00007FDE109D21CA [libpthread.so.0+0x81ca])
        LT  stderr clone+0x43 (0x00007FDE102118D3 [libc.so.6+0x398d3])

[mpirvu]

Method_being_compiled=java/lang/invoke/FoldNonvoidHandle.invokeExact_thunkArchetype_X(I)I

You may want to check for any recently commits that deal with method handles.

[cjjdespres]

I can't see anything that's obviously related to method handles. I checked the nightly test channel, and the first day these sorts of failures appear is June 28th, with openj9 22eaa4d and omr eclipse-omr/omr@47a9d24, in https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal/1065/. That one didn't crash in a java/lang/invoke method compilation, however:

        MiniMix_10m_0 Start Time: Fri Jun 28 01:20:24 2024 Epoch Time (ms): 1719562824005
        variation: Mode150
        JVM_OPTIONS: -XX:+UseJITServer -XX:+UseCompressedOops 

        LT  stderr Type=Segmentation error vmState=0x0005ffff
        LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
        LT  stderr Handler1=00007F3C7A15BEE0 Handler2=00007F3C7A0B9740 InaccessibleAddress=0000000000000008
        LT  stderr RDI=0000000000000000 RSI=00007F3BE4110420 RAX=0000000000000000 RBX=00007F3C6800E0D0
        LT  stderr RCX=0000000000000000 RDX=0000000000000000 R8=00007F3C7407B4F8 R9=0000000000000000
        LT  stderr R10=0000000000000000 R11=0000000000000000 R12=00007F3BE4110420 R13=00007F3C78097D18
        LT  stderr R14=00007F3C78097D00 R15=00007F3C79BAAC40
        LT  stderr RIP=00007F3C790E345C GS=0000 FS=0000 RSP=00007F3C71E1CD10
        LT  stderr EFlags=0000000000010246 CS=0033 RBP=0000000000000000 ERR=0000000000000004
        LT  stderr TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000008
        LT  stderr xmm0=0000000001038b78 (f: 17009528.000000, d: 8.403823e-317)
        LT  stderr xmm1=0000000001038b78 (f: 17009528.000000, d: 8.403823e-317)
        LT  stderr xmm2=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm3=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm4=00000000000cdc18 (f: 842776.000000, d: 4.163867e-318)
        LT  stderr xmm5=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm13=00007f290b500034 (f: 189792304.000000, d: 6.907742e-310)
        LT  stderr xmm14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
        LT  stderr xmm15=00007f3c68059d50 (f: 1745198464.000000, d: 6.911851e-310)
        LT  stderr Module=/home/jenkins/workspace/Test_openjdk8_j9_extended.system_x86-64_linux_jit_Personal_testList_1/jdkbinary/j2sdk-image/jre/lib/amd64/default/libj9jit29.so
        LT  stderr Module_base_address=00007F3C78ED3000
        LT  stderr 
        LT  stderr Method_being_compiled=java/io/FileDescriptor.closeAll(Ljava/io/Closeable;)V
        LT  stderr Target=2_90_20240628_1811 (Linux 5.14.21-150500.55.65-default)
        LT  stderr CPU=amd64 (4 logical CPUs) (0x1dda70000 RAM)
        LT  stderr ----------- Stack Backtrace -----------
        LT  stderr _ZN2J916KnownObjectTable18getExistingIndexAtEPm+0x17c (0x00007F3C790E345C [libj9jit29.so+0x21045c])
        LT  stderr _ZL19handleServerMessagePN9JITServer12ClientStreamEP7TR_J9VMRNS_11MessageTypeE+0x307c (0x00007F3C790778AC [libj9jit29.so+0x1a48ac])
        LT  stderr _Z13remoteCompileP10J9VMThreadPN2TR11CompilationEP17TR_ResolvedMethodP8J9MethodRNS1_24IlGeneratorMethodDetailsEPNS1_28CompilationInfoPerThreadBaseE.localalias+0x131a (0x00007F3C7908A07A [libj9jit29.so+0x1b707a])
        LT  stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x749 (0x00007F3C7904AFB9 [libj9jit29.so+0x177fb9])
        LT  stderr _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x381 (0x00007F3C7904BD61 [libj9jit29.so+0x178d61])
        LT  stderr omrsig_protect+0x239 (0x00007F3C7A0BA3C9 [libj9prt29.so+0x2a3c9])
        LT  stderr _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x385 (0x00007F3C790498C5 [libj9jit29.so+0x1768c5])
        LT  stderr _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x128 (0x00007F3C79049BE8 [libj9jit29.so+0x176be8])
        LT  stderr _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x35b (0x00007F3C79048B1B [libj9jit29.so+0x175b1b])
        LT  stderr _ZN2TR24CompilationInfoPerThread3runEv+0x42 (0x00007F3C79048E82 [libj9jit29.so+0x175e82])
        LT  stderr _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x82 (0x00007F3C79048F32 [libj9jit29.so+0x175f32])
        LT  stderr omrsig_protect+0x239 (0x00007F3C7A0BA3C9 [libj9prt29.so+0x2a3c9])
        LT  stderr _Z21compilationThreadProcPv+0x17b (0x00007F3C790492FB [libj9jit29.so+0x1762fb])
        LT  stderr thread_wrapper+0x163 (0x00007F3C7A0823A3 [libj9thr29.so+0xb3a3])
        LT  stderr start_thread+0xdc (0x00007F3C7A9AE6EA [libpthread.so.0+0xa6ea])
        LT  stderr clone+0x41 (0x00007F3C7A88F58F [libc.so.6+0x11858f])

There are other failures from that day that also crash during the compilation of various different methods.
Since the tests on the 27th seem unaffected and they used the same omr commit, it might be an openj9 merge that happened on the 27th. That's the range 979ce92...22eaa4d.

There were three merges that day. The PR #14931 touches JITServer code but doesn't seem that controversial. The PR #19699 adds a new JITServer message type. The third PR was #19752 and does touch a bit of JITServer code.

I haven't finished going through all the failures, but the ones I've seen so far are all JDK11 or JDK8.

[cjjdespres]

Well, #14931 does touch some code related to the known object table and messages that might need ROM class packing (both things in the backtraces above), but it's hard to see how that PR makes any functional difference.

[cjjdespres]

Some of them are InaccessibleAddress=0000000000000000, so I can try to reproduce some of these locally to see what's actually being accessed.

[mpirvu]

You could try to revert #14931 and see if the problem still shows.

[cjjdespres]

I tried reproducing the DaaLoadTest_all_special_5m_17 failure manually and instead of crashes at the client I got strange crashes at the server (malloc(): unaligned tcache chunk detected among others). Reverting these commits individually will likely be faster.

[cjjdespres]

It could be #14931.

Initially I tried a 25x grinder of DaaLoadTest_all_special_5m_17 with that PR reverted and without it reverted, and both succeeded. I think that might be because the server was crashing before the client, as I saw in #19809 (comment).

I tried SharedClassesWorkload_1 next, and a 25x grinder without the revert had 5/25 failures, and a 25x grinder with the revert had 0/25 failures. It might be safest to revert #14931 and see if the nightly failures go away - there were still a lot last night.

[mpirvu]

I created #19823 to revert #14931.

One solution is to grind/redeliver this in small pieces.

[cjjdespres]

It looks like the problem has gone away with the reversion, though it's a little harder to tell than normal - a lot of yesterday's tests were aborted partway through, and a lot of today's tests failed because artifactory is down for maintenance.

[cjjdespres]

I took #14931 as a base and added fatal asserts around every line, to make sure what was substituted was equal to what was there before. I ran SharedClassesAPI_1 and got an assert corresponding to this part of that PR:

https://github.com/eclipse-openj9/openj9/pull/14931/files#diff-dfeb9a2e63d99bca244af0f98eaf513a5fac03b294385feb4f30ea9322f4ff4bL172-R172

Apparently self()->comp()->getStream() and TR::CompilationInfo::getStream() are both non-null and not equal to each other. Since the original expression was TR::CompilationInfo::getStream(), I'm assuming that it is correct and self()->comp()->getStream() is the one that's wrong.

Actually, you can see that in this bit of the constructor J9::Compilation::Compilation():

openj9/runtime/compiler/compile/J9Compilation.cpp

Lines 233 to 246 in d2b02cf

	TR::KnownObjectTable *knot = self()->getOrCreateKnownObjectTable();
	TR::IlGeneratorMethodDetails & details = ilGenRequest.details();
	if (knot && details.isMethodHandleThunk())
	{
	J9::MethodHandleThunkDetails & thunkDetails = static_cast<J9::MethodHandleThunkDetails &>(details);
	if (thunkDetails.isCustom())
	{
	TR::KnownObjectTable::Index index = knot->getOrCreateIndexAt(thunkDetails.getHandleRef());
	ListIterator<TR::ParameterSymbol> parms(&_methodSymbol->getParameterList());
	TR::ParameterSymbol* parm0 = parms.getFirst();
	parm0->setKnownObjectIndex(index);
	}
	}
	}

we call TR::KnownObjectTable::getOrCreateIndexAt(), but it's here:

openj9/runtime/compiler/control/CompilationThread.cpp

Line 9160 in d2b02cf

compiler->setStream(that->_methodBeingCompiled->_stream);

after the compilation object is created, that we set the stream of the compilation.

[cjjdespres]

Sorry, it was TR::KnownObjectTable::getExistingIndexAt() that had the fatal assert trigger in it. In that case, I'm assuming the compilation object is being reused, and some incorrect stream is being returned from self()->comp()->getStream().