Classloader Confusion

[BrianSchnitzer]

Hello, I'm trying to instrument part of the Datastax Cassadra CQL Driver, and I'm running into what looks like a ClassLoader issue. As I've seen in many similar issues, I'm running a Spring app and it works fine locally when I run it with --javaagent, but throws an error at runtime once deployed. I am trying to avoid shading in the entirety of the Cassandra Driver as this agent's size needs to remain as small as possible.

Agent:

public class CassandraCaptureAgent {
    public static void premain(String agentArgs, Instrumentation inst) {
         transformMethod("com.datastax.oss.driver.internal.core.cql.CqlRequestHandler", "buildExecutionInfo", TargetBuildExecutionInfo.class, inst);
    }

    public static void transformMethod(String className,  String methodName, Class<?> adviceClass, Instrumentation inst) {
        try {
            new AgentBuilder.Default()
                .with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION)
                .type(ElementMatchers.named(className))
                .transform(
                    new AgentBuilder.Transformer.ForAdvice().advice(
                        ElementMatchers.named(methodName),
                        adviceClass.getName()
                    )
                )
                .ignore(ElementMatchers.none())
                .installOn(inst);
        } catch(Throwable t) {
            t.printStackTrace();
        }
    }
}

Advice Class:

public class TargetBuildExecutionInfo {

    public static ScheduledExecutorService executor = null;
    public static ConcurrentLimitedQueue captures = null;
    public static Logger LOGGER = null;
    static {
        try {
            executor = Executors.newSingleThreadScheduledExecutor();
            captures = new ConcurrentLimitedQueue(500);

            executor.scheduleAtFixedRate(
                    new SpanCaptureReporterThread(captures, new AtomicBoolean(true)),
                    //TODO change back to 20 after testing
                    TimeUnit.SECONDS.toMillis(5),
                    TimeUnit.SECONDS.toMillis(3),
                    TimeUnit.MILLISECONDS);
        } catch (Throwable t) {
            if (LOGGER != null) {
                LOGGER.log(Level.SEVERE, "Failed to initialize capture agent.", t);
            } else {
                t.printStackTrace();
            }
        }
    }

    @Advice.OnMethodExit
    public static void onExit(@Advice.Return(readOnly = false, typing = Assigner.Typing.DYNAMIC) ExecutionInfo executionInfo,
                              @Advice.Argument(1) Result result,
                              @Advice.FieldValue("context") InternalDriverContext context,
                              @Advice.FieldValue("startTimeNanos") Long startTimeNanos,
                              @Advice.FieldValue("initialStatement") Statement<?> statement,
                              @Advice.FieldValue("keyspace") CqlIdentifier keyspace) {
        try {
            captures.add(SpanCapture.buildCapture(
                    statement,
                    result,
                    context,
                    GlobalTracer.get().getTraceId(),
                    executionInfo.getSafePagingState(),
                    startTimeNanos,
                    System.currentTimeMillis(),
                    keyspace
            ));
        } catch (Throwable t) {
            if (LOGGER != null) {
                LOGGER.log(Level.SEVERE, "Failed to add capture to queue.", t);
            } else {
                t.printStackTrace();
            }
        }
    }
}

Runtime Error:

com.datastax.oss.driver.internal.core.cql.CqlRequestHandler.buildExecutionInfo Failed to add capture to queue.
	java.lang.NoClassDefFoundError: com/datastax/oss/driver/api/core/cql/PagingState
		at com.datastax.oss.driver.internal.core.cql.CqlRequestHandler.buildExecutionInfo(CqlRequestHandler.java:408)
		at com.datastax.oss.driver.internal.core.cql.CqlRequestHandler.setFinalResult(CqlRequestHandler.java:321)
		at com.datastax.oss.driver.internal.core.cql.CqlRequestHandler.access$1500(CqlRequestHandler.java:95)
		at com.datastax.oss.driver.internal.core.cql.CqlRequestHandler$NodeResponseCallback.onResponse(CqlRequestHandler.java:655)
		at com.datastax.oss.driver.internal.core.channel.InFlightHandler.channelRead(InFlightHandler.java:257)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
		at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
		at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1519)
		at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377)
		at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428)
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
		at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:833)
	Caused by: java.lang.ClassNotFoundException: com.datastax.oss.driver.api.core.cql.PagingState
		at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
		at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
		at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
		... 39 more

Pom:

<build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.3.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <manifestEntries>
                                        <Premain-Class>net.name.runtimeinstrumentation.cql.capture.CassandraCaptureAgent</Premain-Class>
                                        <Can-Redefine-Classes>true</Can-Redefine-Classes>
                                        <Can-Retransform-Classes>true</Can-Retransform-Classes>
                                    </manifestEntries>
                                </transformer>
                            </transformers>
                            <createDependencyReducedPom>true</createDependencyReducedPom>
                            <useDependencyReducedPomInJar>true</useDependencyReducedPomInJar>
                            <shadedArtifactAttached>false</shadedArtifactAttached>
                            <artifactSet>
                                <includes>
                                    <include>net.name.runtimeinstrumentation:cql-capture-java</include>
                                    <include>net.bytebuddy:byte-buddy</include>
                                    <include>com.fasterxml.jackson.core:jackson-annotations</include>
                                    <include>com.fasterxml.jackson.core:jackson-databind</include>
                                    <include>com.fasterxml.jackson.core:jackson-core</include>
                                </includes>
                            </artifactSet>
                            <relocations>
                                <relocation>
                                    <pattern>com.fasterxml.jackson.databind</pattern>
                                    <shadedPattern>net.name.shaded.jackson.databind</shadedPattern>
                                </relocation>
                                <relocation>
                                    <pattern>com.fasterxml.jackson.annotations</pattern>
                                    <shadedPattern>net.name.shaded.jackson.annotations</shadedPattern>
                                </relocation>
                                <relocation>
                                    <pattern>com.fasterxml.jackson.core</pattern>
                                    <shadedPattern>net.name.shaded.jackson.core</shadedPattern>
                                </relocation>
                            </relocations>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/MANIFEST.MF</exclude>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>net.bytebuddy</groupId>
            <artifactId>byte-buddy</artifactId>
            <version>1.15.11</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.13.4.2</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.15.3</version>
        </dependency>
        <dependency>
            <groupId>com.datadoghq</groupId>
            <artifactId>dd-trace-api</artifactId>
            <version>1.38.1</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.datastax.oss</groupId>
            <artifactId>java-driver-core</artifactId>
            <version>4.15.0</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

I've been tinkering with including different classloaders in the advice but nothing seems to be getting me any closer to having access to that class. I believe this is something simple but I'm just missing the right way to go about it. Thanks for any and all help in advance!

[raphw]

The problem is probably that you are calling adviceClass.getName(). For this, the class must be present.

Rather add a string of the package and name, and make sure that the Datastax classes are never mentioned in your agent otherwise.

You should also move:

public static ScheduledExecutorService executor = null;
public static ConcurrentLimitedQueue captures = null;
public static Logger LOGGER = null;

to a different class. Those classes must be available to your app loaders, so likely they need to be on the boot loader. The Logger class might not be available, so maybe rather add a Consumer<String>? Then you can still inject this class into the boot loader (using ClassInjection) and set the fields from your agents as regular code.

[BrianSchnitzer]

Hey @raphw thanks for the reply!

I'm able to see adviceClass.getName() is resolving to the correct string name at runtime, and I am able to log in the inline advice if I remove the references to our advice class code, so those changes don't seem to have an effect at the moment.

make sure that the Datastax classes are never mentioned in your agent

I need to disect the Datastax types during my instrumentation so the drivers are needed by the advice class, but the agent definition has no knowledge of the drivers, I'm not sure if that's exactly what you meant.

Another potentially useful piece of info is that I can see the Datastax dependencies present on ParallelWebAppClassLoader, and likely need those to be visible elsewhere. Is this somewhere ClassInjection would come into play? I'm not exactly sure where to handle that injection, as premain cannot see the Datastax classes and you mentioned keeping them out of the Agent.

Thanks again!

[raphw]

The advice code is inlined into the targeted class, so the Datastax classes should already be available there. You do of course need to include them during compilation, but at runtime, they should be provided. This way, you should interact with the targeted classes in your advice and then extract a description that you can safely pass ti your agents in the format of built-in types that any class loader understands.