Add testing for latest stable version (JDK 24) #8875

sarahchen6 · 2025-05-23T16:57:26Z

What Does This Do

Add testing for stable image. As of this PR, it is JDK 24.

Motivation

Add Java 24 tests to CI

Additional Notes

Waiting until post-1.50.0 release to merge in order to not introduce more potential CI issues.

Ignoring a few smoke-tests for now after checking with the CI-Vis team. Will check in with them again afterwards about un-skipping these tests.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: https://datadoghq.atlassian.net/browse/LANGPLAT-421

pr-commenter · 2025-05-23T17:35:51Z

Benchmarks

Startup

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/add-jdk-24-testing
git_commit_date	1750857432	1750860896
git_commit_sha	`2f749a9`	`1be99fd`
release_version	1.51.0-SNAPSHOT~2f749a97c7	1.51.0-SNAPSHOT~1be99fd70a

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750862464	1750862464
ci_job_id	998106956	998106956
ci_pipeline_id	68737778	68737778
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-vaym2f8s-project-304-concurrent-0-zn2f0o0k 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-vaym2f8s-project-304-concurrent-0-zn2f0o0k 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 1 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:no_agent:high_load	better [-220.755µs; -110.609µs] or [-4.863%; -2.437%]	unstable [-75.736op/s; +150.174op/s] or [-7.485%; +14.842%]	4.374ms	1049.062op/s	4.540ms	1011.844op/s
scenario:load:petclinic:profiling:high_load	worse [+1.550ms; +2.484ms] or [+3.313%; +5.312%]	unstable [-11.324op/s; +3.049op/s] or [-11.320%; +3.048%]	48.785ms	95.900op/s	46.768ms	100.037op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~1be99fd70a, baseline=1.51.0-SNAPSHOT~2f749a97c7
    dateFormat X
    axisFormat %s
section baseline
no_agent (37.289 ms) : 36987, 37590
.   : milestone, 37289,
appsec (47.401 ms) : 46997, 47805
.   : milestone, 47401,
code_origins (45.731 ms) : 45341, 46120
.   : milestone, 45731,
iast (44.257 ms) : 43861, 44654
.   : milestone, 44257,
profiling (46.768 ms) : 46336, 47200
.   : milestone, 46768,
tracing (44.144 ms) : 43777, 44511
.   : milestone, 44144,
section candidate
no_agent (38.304 ms) : 37992, 38616
.   : milestone, 38304,
appsec (47.843 ms) : 47431, 48255
.   : milestone, 47843,
code_origins (44.78 ms) : 44416, 45144
.   : milestone, 44780,
iast (44.671 ms) : 44279, 45063
.   : milestone, 44671,
profiling (48.785 ms) : 48348, 49222
.   : milestone, 48785,
tracing (44.103 ms) : 43743, 44463
.   : milestone, 44103,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.289 ms [36.987 ms, 37.59 ms]	-
appsec	47.401 ms [46.997 ms, 47.805 ms]	10.112 ms (27.1%)
code_origins	45.731 ms [45.341 ms, 46.12 ms]	8.442 ms (22.6%)
iast	44.257 ms [43.861 ms, 44.654 ms]	6.969 ms (18.7%)
profiling	46.768 ms [46.336 ms, 47.2 ms]	9.479 ms (25.4%)
tracing	44.144 ms [43.777 ms, 44.511 ms]	6.855 ms (18.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	38.304 ms [37.992 ms, 38.616 ms]	-
appsec	47.843 ms [47.431 ms, 48.255 ms]	9.539 ms (24.9%)
code_origins	44.78 ms [44.416 ms, 45.144 ms]	6.476 ms (16.9%)
iast	44.671 ms [44.279 ms, 45.063 ms]	6.367 ms (16.6%)
profiling	48.785 ms [48.348 ms, 49.222 ms]	10.481 ms (27.4%)
tracing	44.103 ms [43.743 ms, 44.463 ms]	5.799 ms (15.1%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~1be99fd70a, baseline=1.51.0-SNAPSHOT~2f749a97c7
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.54 ms) : 4486, 4593
.   : milestone, 4540,
iast (9.303 ms) : 9151, 9454
.   : milestone, 9303,
iast_FULL (13.738 ms) : 13470, 14006
.   : milestone, 13738,
iast_GLOBAL (9.93 ms) : 9759, 10102
.   : milestone, 9930,
profiling (9.021 ms) : 8881, 9161
.   : milestone, 9021,
tracing (7.635 ms) : 7529, 7741
.   : milestone, 7635,
section candidate
no_agent (4.374 ms) : 4325, 4423
.   : milestone, 4374,
iast (9.437 ms) : 9281, 9593
.   : milestone, 9437,
iast_FULL (13.986 ms) : 13709, 14263
.   : milestone, 13986,
iast_GLOBAL (10.022 ms) : 9826, 10218
.   : milestone, 10022,
profiling (8.842 ms) : 8705, 8979
.   : milestone, 8842,
tracing (7.478 ms) : 7373, 7583
.   : milestone, 7478,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.54 ms [4.486 ms, 4.593 ms]	-
iast	9.303 ms [9.151 ms, 9.454 ms]	4.763 ms (104.9%)
iast_FULL	13.738 ms [13.47 ms, 14.006 ms]	9.199 ms (202.6%)
iast_GLOBAL	9.93 ms [9.759 ms, 10.102 ms]	5.391 ms (118.7%)
profiling	9.021 ms [8.881 ms, 9.161 ms]	4.482 ms (98.7%)
tracing	7.635 ms [7.529 ms, 7.741 ms]	3.095 ms (68.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.374 ms [4.325 ms, 4.423 ms]	-
iast	9.437 ms [9.281 ms, 9.593 ms]	5.063 ms (115.7%)
iast_FULL	13.986 ms [13.709 ms, 14.263 ms]	9.612 ms (219.8%)
iast_GLOBAL	10.022 ms [9.826 ms, 10.218 ms]	5.648 ms (129.1%)
profiling	8.842 ms [8.705 ms, 8.979 ms]	4.468 ms (102.2%)
tracing	7.478 ms [7.373 ms, 7.583 ms]	3.104 ms (71.0%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/add-jdk-24-testing
git_commit_date	1750857432	1750860896
git_commit_sha	`2f749a9`	`1be99fd`
release_version	1.51.0-SNAPSHOT~2f749a97c7	1.51.0-SNAPSHOT~1be99fd70a

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1750863025	1750863025
ci_job_id	998106957	998106957
ci_pipeline_id	68737778	68737778
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-6z7wq9aq-project-304-concurrent-0-i1dajmi2 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-6z7wq9aq-project-304-concurrent-0-i1dajmi2 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~1be99fd70a, baseline=1.51.0-SNAPSHOT~2f749a97c7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.483 ms) : 1471, 1494
.   : milestone, 1483,
appsec (2.413 ms) : 2364, 2462
.   : milestone, 2413,
iast (2.205 ms) : 2144, 2267
.   : milestone, 2205,
iast_GLOBAL (2.235 ms) : 2173, 2297
.   : milestone, 2235,
profiling (2.038 ms) : 1988, 2087
.   : milestone, 2038,
tracing (2.007 ms) : 1960, 2055
.   : milestone, 2007,
section candidate
no_agent (1.481 ms) : 1469, 1492
.   : milestone, 1481,
appsec (2.405 ms) : 2356, 2454
.   : milestone, 2405,
iast (2.195 ms) : 2134, 2257
.   : milestone, 2195,
iast_GLOBAL (2.242 ms) : 2180, 2304
.   : milestone, 2242,
profiling (2.059 ms) : 2009, 2110
.   : milestone, 2059,
tracing (2.015 ms) : 1968, 2063
.   : milestone, 2015,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.483 ms [1.471 ms, 1.494 ms]	-
appsec	2.413 ms [2.364 ms, 2.462 ms]	930.468 µs (62.8%)
iast	2.205 ms [2.144 ms, 2.267 ms]	722.655 µs (48.7%)
iast_GLOBAL	2.235 ms [2.173 ms, 2.297 ms]	752.419 µs (50.7%)
profiling	2.038 ms [1.988 ms, 2.087 ms]	555.147 µs (37.4%)
tracing	2.007 ms [1.96 ms, 2.055 ms]	524.567 µs (35.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.481 ms [1.469 ms, 1.492 ms]	-
appsec	2.405 ms [2.356 ms, 2.454 ms]	924.408 µs (62.4%)
iast	2.195 ms [2.134 ms, 2.257 ms]	714.711 µs (48.3%)
iast_GLOBAL	2.242 ms [2.18 ms, 2.304 ms]	761.524 µs (51.4%)
profiling	2.059 ms [2.009 ms, 2.11 ms]	578.691 µs (39.1%)
tracing	2.015 ms [1.968 ms, 2.063 ms]	534.501 µs (36.1%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~1be99fd70a, baseline=1.51.0-SNAPSHOT~2f749a97c7
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.043 s) : 15043000, 15043000
.   : milestone, 15043000,
appsec (14.817 s) : 14817000, 14817000
.   : milestone, 14817000,
iast (18.306 s) : 18306000, 18306000
.   : milestone, 18306000,
iast_GLOBAL (18.257 s) : 18257000, 18257000
.   : milestone, 18257000,
profiling (15.511 s) : 15511000, 15511000
.   : milestone, 15511000,
tracing (15.084 s) : 15084000, 15084000
.   : milestone, 15084000,
section candidate
no_agent (14.996 s) : 14996000, 14996000
.   : milestone, 14996000,
appsec (14.828 s) : 14828000, 14828000
.   : milestone, 14828000,
iast (18.114 s) : 18114000, 18114000
.   : milestone, 18114000,
iast_GLOBAL (17.838 s) : 17838000, 17838000
.   : milestone, 17838000,
profiling (15.85 s) : 15850000, 15850000
.   : milestone, 15850000,
tracing (14.906 s) : 14906000, 14906000
.   : milestone, 14906000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.043 s [15.043 s, 15.043 s]	-
appsec	14.817 s [14.817 s, 14.817 s]	-226.0 ms (-1.5%)
iast	18.306 s [18.306 s, 18.306 s]	3.263 s (21.7%)
iast_GLOBAL	18.257 s [18.257 s, 18.257 s]	3.214 s (21.4%)
profiling	15.511 s [15.511 s, 15.511 s]	468.0 ms (3.1%)
tracing	15.084 s [15.084 s, 15.084 s]	41.0 ms (0.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.996 s [14.996 s, 14.996 s]	-
appsec	14.828 s [14.828 s, 14.828 s]	-168.0 ms (-1.1%)
iast	18.114 s [18.114 s, 18.114 s]	3.118 s (20.8%)
iast_GLOBAL	17.838 s [17.838 s, 17.838 s]	2.842 s (19.0%)
profiling	15.85 s [15.85 s, 15.85 s]	854.0 ms (5.7%)
tracing	14.906 s [14.906 s, 14.906 s]	-90.0 ms (-0.6%)

gradle/java_no_deps.gradle

.gitlab-ci.yml

gradle/java_no_deps.gradle

AlexeyKuznetsov-DD

Just curious, do we need to change tests that are using jvmbootstraptest.CustomSecurityManager?

sarahchen6 · 2025-06-16T17:26:06Z

do we need to change tests that are using jvmbootstraptest.CustomSecurityManager?

I briefly talked to @dougqh about this, and we decided to skip them for Java 24+ because other tests cover the positive case.

dd-smoke-tests/log-injection/build.gradle

gradle/java_no_deps.gradle

dd-java-agent/instrumentation/java-concurrent/src/test/groovy/QueueTimingForkedTest.groovy

dd-smoke-tests/log-injection/build.gradle

sarahchen6 added type: enhancement Enhancements and improvements comp: testing Testing labels May 23, 2025

sarahchen6 force-pushed the sarahchen6/add-jdk-24-testing branch 3 times, most recently from bfd5082 to c7ee38f Compare June 8, 2025 17:39

sarahchen6 mentioned this pull request Jun 9, 2025

Upgrade gradle to v8.14.2 #8950

Draft

sarahchen6 force-pushed the sarahchen6/add-jdk-24-testing branch 3 times, most recently from 8110a45 to 0472d2a Compare June 13, 2025 16:50

bric3 reviewed Jun 13, 2025

View reviewed changes

gradle/java_no_deps.gradle Outdated Show resolved Hide resolved

sarahchen6 force-pushed the sarahchen6/add-jdk-24-testing branch 3 times, most recently from 2cb7960 to 0722aa1 Compare June 16, 2025 13:49

AlexeyKuznetsov-DD requested changes Jun 16, 2025

View reviewed changes

.gitlab-ci.yml Show resolved Hide resolved

gradle/java_no_deps.gradle Outdated Show resolved Hide resolved

AlexeyKuznetsov-DD requested changes Jun 16, 2025

View reviewed changes

sarahchen6 force-pushed the sarahchen6/add-jdk-24-testing branch 2 times, most recently from 06808a5 to e476a61 Compare June 17, 2025 19:29

sarahchen6 requested review from AlexeyKuznetsov-DD and bric3 June 18, 2025 16:32

sarahchen6 marked this pull request as ready for review June 18, 2025 16:32

sarahchen6 requested review from a team as code owners June 18, 2025 16:32

sarahchen6 requested review from Mariovido and removed request for a team June 18, 2025 16:32

sarahchen6 requested review from erikayasuda and smola June 18, 2025 16:32

bric3 approved these changes Jun 18, 2025

View reviewed changes

dd-smoke-tests/log-injection/build.gradle Show resolved Hide resolved

AlexeyKuznetsov-DD requested changes Jun 18, 2025

View reviewed changes

gradle/java_no_deps.gradle Outdated Show resolved Hide resolved

dd-java-agent/instrumentation/java-concurrent/src/test/groovy/QueueTimingForkedTest.groovy Outdated Show resolved Hide resolved

dd-smoke-tests/log-injection/build.gradle Show resolved Hide resolved

AlexeyKuznetsov-DD requested changes Jun 18, 2025

View reviewed changes

dd-smoke-tests/log-injection/build.gradle Show resolved Hide resolved

sarahchen6 requested a review from AlexeyKuznetsov-DD June 18, 2025 19:39

sarahchen6 added 18 commits June 25, 2025 10:14

Add stable image

18286ff

Adjust matcher

bd33925

Add smoke test gradle compatibility for java 24

d5eb7e1

Get latest java home version for stable image

8cd5545

Fix latest version finding logic

c5b65b5

Skip security manager tests for JDK 24+

677410f

Add JDK 24 to setup instructions

c5553b2

Comment out QueueTimingForkedTest for now due to mysterious behavior

14aef72

Fix QueueTimingForkedTest for JDK 24

992e7d1

Update vertx-pg-client version

2b297b0

Instead skip vertx-pg-client-4.0 entirely for JDK 22+

99ecfd5

Skip WildFlyForkedTest

ef2fd28

Ignore karate tests for Java 24 until fix is found

a980705

Fix comment

5330154

Upgrade groovy version for log injection smoke tests

7be83af

Skip failing smoke tests for Java 24 until we have a fix

0686562

Address review comments

fdace89

Improve comment for groovy upgrade

1be99fd

sarahchen6 force-pushed the sarahchen6/add-jdk-24-testing branch from 502607e to 1be99fd Compare June 25, 2025 14:15

AlexeyKuznetsov-DD approved these changes Jun 25, 2025

View reviewed changes

sarahchen6 merged commit 8c372e3 into master Jun 25, 2025
508 checks passed

sarahchen6 deleted the sarahchen6/add-jdk-24-testing branch June 25, 2025 15:15

github-actions bot added this to the 1.51.0 milestone Jun 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add testing for latest stable version (JDK 24) #8875

Add testing for latest stable version (JDK 24) #8875

Uh oh!

sarahchen6 commented May 23, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented May 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexeyKuznetsov-DD left a comment

Uh oh!

sarahchen6 commented Jun 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Add testing for latest stable version (JDK 24) #8875

Add testing for latest stable version (JDK 24) #8875

Uh oh!

Conversation

sarahchen6 commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexeyKuznetsov-DD left a comment

Choose a reason for hiding this comment

Uh oh!

sarahchen6 commented Jun 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sarahchen6 commented May 23, 2025 •

edited

Loading

pr-commenter bot commented May 23, 2025 •

edited

Loading