[BUG] OpenSearch 3.1.0 fails to start with error message java.lang.NoClassDefFoundError: org/opensearch/javaagent/bootstrap/AgentPolicy$AnyCanExit

[faxm0dem]

Describe the bug

Since 3.0.0 (RPM version for x64) OpenSearch fails to start (systemd) with the following stack trace in journald :

[ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [ccsiem02.in2p3.fr] fatal error in thread [main], exiting
    java.lang.NoClassDefFoundError: org/opensearch/javaagent/bootstrap/AgentPolicy$AnyCanExit
    java.lang.NoClassDefFoundError: org/opensearch/javaagent/bootstrap/AgentPolicy$AnyCanExit
            at org.opensearch.bootstrap.Security.configure(Security.java:163) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.Security.configure(Security.java:163) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:244) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:244) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:411) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:411) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.1.0.jar:3.1.0]
            at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.1.0.jar:3.1.0]
            at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) ~[opensearch-3.1.0.jar:3.1.0]
            at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.1.0.jar:3.1.0]
            at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) ~[opensearch-3.1.0.jar:3.1.0]
    Caused by: java.lang.ClassNotFoundException: org.opensearch.javaagent.bootstrap.AgentPolicy$AnyCanExit
            at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) ~[opensearch-3.1.0.jar:3.1.0]
            at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?]
            at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) ~[opensearch-3.1.0.jar:3.1.0]
            at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?]
    Caused by: java.lang.ClassNotFoundException: org.opensearch.javaagent.bootstrap.AgentPolicy$AnyCanExit
            at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
            at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?]
            ... 10 more

Related component

Other

To Reproduce

1. Install opensearch rpm
2. apply config (from 2.9.x)
3. start using systemctl start opensearch
4. check logs for error message journalctl -rt opensearch

Expected behavior

Should start

Additional Details

RHEL9

[andrross]

This is the symptom we see when the java agent is not attached, but the RPM packaging should ensure that the JVM is launched with the agent. @cwperks any idea here?

[cwperks]

@faxm0dem What version of Java are you running with? OpenSearch 3.0.0 requires Java >= 21. The java agent is attached via JVM options from https://github.com/opensearch-project/OpenSearch/blob/main/distribution/src/config/jvm.options#L87-L88

By any chance is the config/jvm.options file modified in some way?

[faxm0dem]

I checked the jvm.options and I didnt't modify it.
I tried with system java (21, same version in rel9), and with Opensearch shipped one.
Process accounting confirms java21 is being used :

117300 systemctl start opensearch
117305 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,CLDR -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/opensearch-953927102777907329 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/opensearch -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.security.manager=allow -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=file:///etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED -XX:MaxDirectMemorySize=536870912 -Dopensearch.path.home=/usr/share/opensearch -Dopensearch.path.conf=/etc/opensearch -Dopensearch.distribution.type=rpm -Dopensearch.bundled_jdk=true -cp /usr/share/opensearch/lib/* org.opensearch.bootstrap.OpenSearch -p /var/run/opensearch/opensearch.pid
# Job for opensearch.service failed because the control process exited with error code.
See "systemctl status opensearch.service" and "journalctl -xeu opensearch.service" for details.