munit and junit test reporting does not work when sbt runs tests in fork mode

[mwisnicki]

Tracer Version(s)

1.47.3~4a85e1fff5

Java Version(s)

21.0.2

JVM Vendor

Eclipse Adoptium / Temurin

Bug Report

When sbt is configured to run tests in fork mode, neither munit nor junit tests are reported to datadog.
Forking mode works for scalatest.

Expected Behavior

munit with Test/fork := true reports tests to datadog.

Reproduction Code

build.sbt

scalaVersion := "3.3.5"
Test / fork := true // comment it out to make this work
libraryDependencies ++= Seq(
  "junit" % "junit" % "4.13.2" % Test,
  "com.github.sbt" % "junit-interface" % "0.13.3" % Test,
  "org.scalameta" %% "munit" % "1.1.0" % Test,
)

src/test/scala/ExampleTest.scala

class ExampleJunitTest {
  @org.junit.Test
  def hello(): Unit = {
    println("Hello from JUnit")
  }
}

class ExampleMunitTest extends munit.FunSuite {
  test("hello") {
    println("Hello from MUnit")
  }
}

project/build.properties

sbt.version=1.10.11

Simplified snippet from .github/workflows/test.yaml

jobs:
  tests:
    steps:
      - uses: sbt/setup-sbt@v1
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 21
      - uses: datadog/test-visibility-github-action@v2
        with:
          languages: java
          api_key: ${{ secrets.DATADOG_API_KEY }}
          service: foo
      - run: sbt test

[nikita-tkachenko-datadog]

Hi @mwisnicki, thank you for reporting this!
The reason tests are not reported to Datadog is that the DD tracer is only injected into the parent SBT process, and not into the forked JVMs that execute the tests.
Could you please try using the java-instrumented-build-system: all GitHub Action setting? That will ensure the tracer is injected into every JVM.

- uses: datadog/test-visibility-github-action@v2
  with:
    languages: java
    api_key: ${{ secrets.DATADOG_API_KEY }}
    service: foo
    java-instrumented-build-system: all

[mwisnicki]

java-instrumented-build-system seems to have helped getting test results in fork mode however I've noticed that regardless of this setting and whether forking is enabled, test runs and test suites are disconnected from test session and module. Both for munit and junit. I'm surprised that session didn't propagate even in non-fork mode.

---

Additional minor problem is that I have warning Warning: dd-java-agent is being initialized more than once but that's understandable, it's injected by github action and by sbt-datadog plugin. I'll have to figure out how to disable injection from the latter in tests but for now I disabled it completely to eliminate any interference.

Finally there's I think unrelated issue that DataDog sometimes stops updating github comment with reports. I think I saw it before so it's likely not caused by this.

[mwisnicki]

It's interesting that even with Test/fork := false I see two OpenJDK and dd.trace startup messages as well as

Warning: dd-java-agent is being initialized more than once. 

[mwisnicki]

So an example test run has following tag:

test_session:
  fingerprint_v2: 7db8218e3ff2e7d4
  name: test-codegen

while disconnected but probably related test session looks like this:

test_session:
  fingerprint: 89df8bdf5ab3d8eb
  fingerprint_v2: 7db8218e3ff2e7d4
  name: test-codegen

Month ago with library_version: 1.46.1~8e0205904b (now 1.47.3) it looked similar but used to work.

[mwisnicki]

Another strange problem - I have 2 jobs in workflow triggered by PR but now only see reports from one of them.
They are set up identically except 2nd one uses branch instead of branch merged to main.
The logs from both seem correct.

[mwisnicki]

I'll try to narrow down which version introduced the break.

[nikita-tkachenko-datadog]

test runs and test suites are disconnected from test session and module

Could you provide more context here please?

[mwisnicki]

test runs and test suites are disconnected from test session and module

Could you provide more context here please?

All test runs are present but around half of individual test runs have their Session Trace tab empty (it never arrives):

It's even weirder when I go see sessions.

My workflow has 2 jobs so I expect 2 sessions.

Instead there are 2-3 sessions. Sometimes one is green but most of them grey and marked as skipped although they took more or less same time.
Their test suites and test runs tabs are empty (no matching results found).

When I downgrade to 1.46.1 things start to work:

[mwisnicki]

1.47.0 introduced the break

[mwisnicki]

Which is a shame since it also introduced much needed fix for #8445

[nikita-tkachenko-datadog]

Hi @mwisnicki, thank you for these details, they helped a lot with reproducing the problem! The fix is included in the Java tracer release v1.48.0 that was rolled out yesterday. Could you please try it out and let me know if it fixes the issue?