Jep454Tests_testLinkerFfi_DownCall_0_FAILED Segmentation error vmState=0x00000000

[JasonFengJ9]

Failure link

From an internal build(win19x86-svl-rt5-1):

openjdk version "23-beta" 2024-09-17
IBM Semeru Runtime Open Edition 23+37-202408220132 (build 23-beta+37-202408220132)
Eclipse OpenJ9 VM 23+37-202408220132 (build master-18eaa15717, JRE 23 Windows Server 2019 amd64-64-Bit Compressed References 20240822_19 (JIT enabled, AOT enabled)
OpenJ9   - 18eaa15717
OMR      - e0dde2287
JCL      - 4f8b339af based on jdk-23+37)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2024-08-22T07:57:34.343Z] variation: NoOptions
[2024-08-22T07:57:34.721Z] JVM_OPTIONS:  

[2024-08-22T07:58:00.623Z] Unhandled exception
[2024-08-22T07:58:00.623Z] Type=Segmentation error vmState=0x00000000
[2024-08-22T07:58:00.623Z] Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFEDAF33F20 ContextFlags=0010005f
[2024-08-22T07:58:00.623Z] Handler1=00007FFEDB525560 Handler2=00007FFEDD90AC60 InaccessibleReadAddress=0000000000000003
[2024-08-22T07:58:00.623Z] RDI=FFFFFFFFFFFFFA88 RSI=0000000000000000 RAX=00007FFEDB5F8073 RBX=000000000000000D
[2024-08-22T07:58:00.623Z] RCX=0000021E9FD42650 RDX=0000000000000001 R8=0000021E9FD42710 R9=00007FFEDDDA0000
[2024-08-22T07:58:00.623Z] R10=0000000000000000 R11=00007FFEDDDB1A37 R12=00000079555BEFE0 R13=0000000000000000
[2024-08-22T07:58:00.623Z] R14=0000021E9FC9A2A8 R15=00007FFEDAF33F20
[2024-08-22T07:58:00.624Z] RIP=00007FFEDAF33F20 RSP=00000079555BEA58 RBP=00000079555BEA80 EFLAGS=0000000000010246
[2024-08-22T07:58:00.624Z] FS=0053 ES=002B DS=002B
[2024-08-22T07:58:00.624Z] XMM0=0000021e9fd42650 (f: 2681480704.000000, d: 1.151447e-311)
[2024-08-22T07:58:00.624Z] XMM1=0000000000000001 (f: 1.000000, d: 4.940656e-324)
[2024-08-22T07:58:00.624Z] XMM2=0000021e9fd42710 (f: 2681480960.000000, d: 1.151447e-311)
[2024-08-22T07:58:00.624Z] XMM3=00007ffeddda0000 (f: 3722051584.000000, d: 6.953115e-310)
[2024-08-22T07:58:00.624Z] XMM4=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-22T07:58:00.624Z] XMM5=00000000ffdb06e0 (f: 4292544256.000000, d: 2.120799e-314)
[2024-08-22T07:58:00.624Z] XMM6=00000000001256c8 (f: 1201864.000000, d: 5.937997e-318)
[2024-08-22T07:58:00.624Z] XMM7=00000000001256c8 (f: 1201864.000000, d: 5.937997e-318)
[2024-08-22T07:58:00.624Z] XMM8=4030000000000000 (f: 0.000000, d: 1.600000e+01)
[2024-08-22T07:58:00.624Z] XMM9=4030000000000000 (f: 0.000000, d: 1.600000e+01)
[2024-08-22T07:58:00.624Z] XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-22T07:58:00.624Z] XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-22T07:58:00.624Z] XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-22T07:58:00.624Z] XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-22T07:58:00.624Z] XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-22T07:58:00.624Z] XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-08-22T07:58:00.624Z] Module=C:\Users\jenkins\workspace\Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3\jdkbinary\openjdk-test-image\openj9\clinkerffitests.dll
[2024-08-22T07:58:00.624Z] Module_base_address=00007FFEDAF30000 Offset_in_DLL=0000000000003f20
[2024-08-22T07:58:00.624Z] Target=2_90_20240822_19 (Windows Server 2019 10.0 build 17763)
[2024-08-22T07:58:00.624Z] CPU=amd64 (4 logical CPUs) (0x1fff78000 RAM)
[2024-08-22T07:58:00.624Z] ----------- Stack Backtrace -----------
[2024-08-22T07:58:00.624Z] addBoolStruct1AndNestedBoolArrayStruct2WithXor_returnStruct1_dupStruct+0x0 (0x00007FFEDAF33F20 [clinkerffitests+0x3f20])
[2024-08-22T07:58:00.624Z] ffi_call_go+0x3c1 (0x00007FFEDB5F8221 [j9vm29+0x1a8221])
[2024-08-22T07:58:00.624Z] ffi_call_go+0x213 (0x00007FFEDB5F8073 [j9vm29+0x1a8073])
[2024-08-22T07:58:00.624Z] ffi_call_go+0x22 (0x00007FFEDB5F7E82 [j9vm29+0x1a7e82])
[2024-08-22T07:58:00.624Z] J9_GetInterface+0x3c4a3 (0x00007FFEDB56F4F3 [j9vm29+0x11f4f3])
[2024-08-22T07:58:00.624Z] (0x00007FFEDB46A656 [j9vm29+0x1a656])
[2024-08-22T07:58:00.624Z] (0x00000000FFDB1A04)
[2024-08-22T07:58:00.624Z] (0x00000000FFDB1A00)
[2024-08-22T07:58:00.624Z] (0x00000079555BED80)
[2024-08-22T07:58:00.624Z] (0x1FFFFFFFFFFFFFFF)
[2024-08-22T07:58:00.624Z] (0x00000079555BEFE0)
[2024-08-22T07:58:00.624Z] (0x0000021E00000001)
[2024-08-22T07:58:00.624Z] (0x00000000004AB028)
[2024-08-22T07:58:00.624Z] J9_GetInterface+0x1f934 (0x00007FFEDB552984 [j9vm29+0x102984])
[2024-08-22T07:58:00.624Z] J9VMDllMain+0x1edaa3 (0x00007FFED91DEAB3 [j9gc29+0x1eeab3])
[2024-08-22T07:58:00.624Z] J9VMDllMain+0x17d9 (0x00007FFED8FF27E9 [j9gc29+0x27e9])
[2024-08-22T07:58:00.624Z] Java_java_lang_invoke_MethodHandleNatives_resolve+0x132c (0x00007FFEDAD9974C [jclse29+0x4974c])
[2024-08-22T07:58:00.624Z] (0x0000000000000005)
[2024-08-22T07:58:00.624Z] (0x0000000000000001)
[2024-08-22T07:58:00.624Z] ---------------------------------------
[2024-08-22T07:58:00.624Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/08/22 00:58:00 - please wait.
[2024-08-22T07:58:00.624Z] JVMDUMP032I JVM requested System dump using 'C:\Users\jenkins\workspace\Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3\aqa-tests\TKG\output_17243101749888\Jep454Tests_testLinkerFfi_DownCall_0\core.20240822.005800.796.0001.dmp' in response to an event
[2024-08-22T07:58:10.982Z] JVMDUMP010I System dump written to C:\Users\jenkins\workspace\Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3\aqa-tests\TKG\output_17243101749888\Jep454Tests_testLinkerFfi_DownCall_0\core.20240822.005800.796.0001.dmp
[2024-08-22T07:58:10.982Z] JVMDUMP032I JVM requested Java dump using 'C:\Users\jenkins\workspace\Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3\aqa-tests\TKG\output_17243101749888\Jep454Tests_testLinkerFfi_DownCall_0\javacore.20240822.005800.796.0002.txt' in response to an event
[2024-08-22T07:58:12.930Z] JVMDUMP010I Java dump written to C:\Users\jenkins\workspace\Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3\aqa-tests\TKG\output_17243101749888\Jep454Tests_testLinkerFfi_DownCall_0\javacore.20240822.005800.796.0002.txt
[2024-08-22T07:58:12.930Z] JVMDUMP032I JVM requested Snap dump using 'C:\Users\jenkins\workspace\Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3\aqa-tests\TKG\output_17243101749888\Jep454Tests_testLinkerFfi_DownCall_0\Snap.20240822.005800.796.0003.trc' in response to an event
[2024-08-22T07:58:12.930Z] JVMDUMP010I Snap dump written to C:\Users\jenkins\workspace\Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3\aqa-tests\TKG\output_17243101749888\Jep454Tests_testLinkerFfi_DownCall_0\Snap.20240822.005800.796.0003.trc
[2024-08-22T07:58:12.930Z] JVMDUMP032I JVM requested JIT dump using 'C:\Users\jenkins\workspace\Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3\aqa-tests\TKG\output_17243101749888\Jep454Tests_testLinkerFfi_DownCall_0\jitdump.20240822.005800.796.0004.dmp' in response to an event
[2024-08-22T07:58:12.930Z] JVMDUMP051I JIT dump occurred in 'main' thread 0x0000000000017000
[2024-08-22T07:58:12.930Z] JVMDUMP053I JIT dump is recompiling jdk/internal/reflect/DirectMethodHandleAccessor.invokeImpl(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
[2024-08-22T07:58:13.353Z] JVMDUMP010I JIT dump written to C:\Users\jenkins\workspace\Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3\aqa-tests\TKG\output_17243101749888\Jep454Tests_testLinkerFfi_DownCall_0\jitdump.20240822.005800.796.0004.dmp
[2024-08-22T07:58:13.353Z] JVMDUMP013I Processed dump event "gpf", detail "".
[2024-08-22T07:58:13.752Z] -----------------------------------
[2024-08-22T07:58:13.752Z] Jep454Tests_testLinkerFfi_DownCall_0_FAILED

50x internal grinder

[pshipton]

It's failing 100%.
@ChengJin01 @tajila

[ChengJin01]

It looks like the crash was related to #19988 which was merged 3 days ago (Aug 25th)
But the failing build indicates it ran on Aug 22th at https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk23_j9_sanity.functional_x86-64_windows_testList_3/3/

Build #3 (Aug 22, 2024, 1:29:10 AM)

So there must be some mismatch in test for #19988 and the code.

[ChengJin01]

I tried with the latest changes on Linux locally which works good without any issue.

[ChengJin01]

Launched an internal build with the latest JDK to see how it goes on Windows.

[ChengJin01]

The crash related tests (including addBoolStruct1AndNestedBoolArrayStruct2WithXor_returnStruct1_dupStruct) at #19922 were merged 3 weeks ago to ensure all added test suites work good with the existing implementation (verified on all supported platforms) before the code changes at #19988. So it doesn't make any sense to me that it still failed in one of the added tests on Windows.

[ChengJin01]

Launched an internal build with the latest JDK to see how it goes on Windows.

The internal builds still failed on the same crash for which I figured out that test case invoked the wrong native function in downcall as follows:

1XMCURTHDINFO  Current thread
3XMTHREADINFO      "main" J9VMThread:0x0000000000017000, omrthread_t:0x0000021C235EB4F0, java/lang/Thread:0x0000000700087890, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x2, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x0000000700069EF8)
3XMTHREADINFO1            (native thread ID:0x624, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000020)
3XMCPUTIME               CPU usage total: 5.937500000 secs, user: 3.078125000 secs, system: 2.859375000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=720224 (0xAFD60)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at openj9/internal/foreign/abi/InternalDowncallHandler.invokeNative(Native Method)
4XESTACKTRACE                at openj9/internal/foreign/abi/InternalDowncallHandler.runNativeMethod(InternalDowncallHandler.java:671)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$DMH/0x0000000044342840.invokeSpecial(LambdaForm$DMH)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$MH/0x00000000443701a0.invoke(LambdaForm$MH)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$MH/0x000000004434db10.invokeExact_MT(LambdaForm$MH)
4XESTACKTRACE    ---->    at org/openj9/test/jep454/downcall/DuplicateStructTests.
test_addBoolStruct1AndNestedBoolArrayStruct2WithXor_returnStruct2_dupStruct(DuplicateStructTests.java:239)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$DMH/0x0000000044368750.invokeVirtual(LambdaForm$DMH)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$MH/0x0000000044365730.invoke(LambdaForm$MH)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$MH/0x0000000030019a10.invokeExact_MT(LambdaForm$MH(Compiled Code))
4XESTACKTRACE                at 

against the core at

[2024-08-22T07:58:00.624Z] addBoolStruct1AndNestedBoolArrayStruct2WithXor_returnStruct1_dupStruct+0x0 (0x00007FFEDAF33F20 [clinkerffitests+0x3f20])
[2024-08-22T07:58:00.624Z] ffi_call_go+0x3c1 (0x00007FFEDB5F8221 [j9vm29+0x1a8221])
[2024-08-22T07:58:00.624Z] ffi_call_go+0x213 (0x00007FFEDB5F8073 [j9vm29+0x1a8073])
[2024-08-22T07:58:00.624Z] ffi_call_go+0x22 (0x00007FFEDB5F7E82 [j9vm29+0x1a7e82])

in which case DuplicateStructTests.test_addBoolStruct1AndNestedBoolArrayStruct2WithXor_returnStruct2_dupStruct( was supposed to call addBoolStruct1AndNestedBoolArrayStruct2WithXor_returnStruct2_dupStruct instead of addBoolStruct1AndNestedBoolArrayStruct2WithXor_returnStruct1_dupStruct.

So I will need to create a test fix for the issue and verify on JDK17/22+ on all supported platforms to see how it goes (It's weird the problem was only captured on Windows)

[ChengJin01]

With my test fix at ChengJin01@021366a, the failing test passed on all supported platforms at https://hyc-runtimes-jenkins.swg-devops.com/job/Pipeline-Build-Test-Personal/23657/, including Windows in JDK17/21/23 at
https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk17_j9_sanity.functional_x86-64_windows_Personal_testList_0/373/tapResults/
https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk21_j9_sanity.functional_x86-64_windows_Personal_testList_1/137/consoleFull
https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk23_j9_sanity.functional_x86-64_windows_Personal_testList_0/14/tapTestReport/