Update image publish jobs for mutable tags to new method #37720
+163
−95
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kacper-murzyn
added
changelog/no-changelog
team/agent-delivery
qa/rc-required
Gitlab CI Configuration Changes
|
| Removed | Modified | Added | Renamed |
|---|---|---|---|
| 4 | 20 | 8 | 0 |
ℹ️ Diff available in the job log.
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 887a28c Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +0.72 | [-2.38, +3.82] | 1 | Logs |
| ➖ | otlp_ingest_logs | memory utilization | +0.52 | [+0.40, +0.64] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | +0.21 | [+0.15, +0.27] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | +0.07 | [+0.02, +0.13] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_metrics | memory utilization | +0.06 | [-0.06, +0.18] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | +0.04 | [-0.60, +0.67] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.02 | [-0.59, +0.63] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency_http1 | egress throughput | +0.01 | [-0.60, +0.62] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.01 | [-0.62, +0.64] | 1 | Logs |
| ➖ | file_to_blackhole_300ms_latency | egress throughput | +0.00 | [-0.62, +0.62] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.00 | [-0.28, +0.28] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.00 | [-0.61, +0.61] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | -0.01 | [-0.03, +0.02] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency_linear_load | egress throughput | -0.01 | [-0.24, +0.22] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency_http2 | egress throughput | -0.04 | [-0.66, +0.57] | 1 | Logs |
| ➖ | otlp_ingest_metrics | memory utilization | -0.15 | [-0.31, +0.02] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.57 | [-0.70, -0.43] | 1 | Logs bounds checks dashboard |
| ➖ | docker_containers_memory | memory utilization | -0.57 | [-0.64, -0.50] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.58 | [-1.44, +0.28] | 1 | Logs |
| ➖ | uds_dogstatsd_20mb_12k_contexts_20_senders | memory utilization | -0.88 | [-0.94, -0.83] | 1 | Logs |
| ➖ | ddot_logs | memory utilization | -1.11 | [-1.25, -0.97] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | -2.06 | [-4.80, +0.69] | 1 | Logs bounds checks dashboard |
| ➖ | file_tree | memory utilization | -3.63 | [-3.82, -3.44] | 1 | Logs |
Bounds Checks: ❌ Failed
| perf | experiment | bounds_check_name | replicates_passed | links |
|---|---|---|---|---|
| ❌ | docker_containers_memory | memory_usage | 0/10 | |
| ✅ | docker_containers_cpu | simple_check_run | 10/10 | |
| ✅ | docker_containers_memory | simple_check_run | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency_linear_load | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | lost_bytes | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
Static quality checks✅ Please find below the results from static quality gates Successful checksInfo
|
FlorentClarret
approved these changes
Jun 6, 2025
| !reference [.manual_on_deploy_auto_on_rc] | ||
| dependencies: [] | ||
| before_script: | ||
| - VERSION="$(dda inv agent.version --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $? |
There was a problem hiding this comment.
Suggested change
| - VERSION="$(dda inv agent.version --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $? | |
| - VERSION="$(dda inv -- agent.version --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $? |
nit, to stay on the safe side even if not strictly required in this case for now
| stage: deploy_mutable_image_tags | ||
| dependencies: [] | ||
| before_script: | ||
| - VERSION="$(dda inv agent.version --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $? |
There was a problem hiding this comment.
Suggested change
| - VERSION="$(dda inv agent.version --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $? | |
| - VERSION="$(dda inv -- agent.version --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $? |
kacper-murzyn
changed the title
AliDatadog
reviewed
Jun 6, 2025
AliDatadog
reviewed
Jun 6, 2025
AliDatadog
reviewed
Jun 6, 2025
AliDatadog
reviewed
Jun 6, 2025
| dependencies: [] | ||
| before_script: | ||
| - VERSION="$(dda inv agent.version --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $? | ||
| - export IMG_TAG_REFERENCE=${AGENT_REPOSITORY}:${VERSION}${IMG_TAG_REFERENCE_FLAVOR}${IMG_TAG_REFERENCE_WIN_OS}${IMG_TAG_REFERENCE_JMX} |
There was a problem hiding this comment.
nit: can we try to further simplify that with maybe something like ${AGENT_REPOSITORY}:${VERSION}${SUFFIX} ?
I also think IMG_TAG_* variable names can be shortened
There was a problem hiding this comment.
IMG_TAG_REFERENCE has to stay, as it is passed to public-images. I wanted to make those names explicit, but I guess we can use the shortening here.
There was a problem hiding this comment.
I just think that this might be difficult to read and change for example:
- IMG_TAG_REFERENCE_FLAVOR:
- ""
- "-servercore"
- "-linux"
IMG_TAG_REFERENCE_JMX:
- ""
- "-jmx"
Where as this would be easier to maintain:
- SUFFIX: ""
- SUFFIX: "-jmx"
- SUFFIX: "-servercore"
- SUFFIX: "-servercore-jmx"
- SUFFIX: "-linux"
- SUFFIX: "-linux-jmx"
There was a problem hiding this comment.
Good idea, I tried to change it to look more like that and it should work and is much easier to read.
github-actions
bot
added
long review
There was a problem hiding this comment.
Pull Request Overview
This PR updates the image publish jobs during the Agent RC build to use a new method that only creates a tag rather than pushing a whole new index. Key changes include:
- Defining new job templates for mutable image tag deployments with parallel matrix configurations.
- Updating and reorganizing job definitions across container deployment and image tag deployment files.
- Removing obsolete job definitions to simplify the RC build flow.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| .gitlab/deploy_containers/deploy_mutable_image_tags.yml | Introduces new job templates and parallel matrix configurations for mutable image tags; removes an internal RC job definition. |
| .gitlab/deploy_containers/deploy_containers_a7.yml | Refactors job definitions for container deployments and adds a new dogstatsd deployment job with updated rules and before_script. |
Comments suppressed due to low confidence (1)
.gitlab/deploy_containers/deploy_mutable_image_tags.yml:59
- There is an inconsistency where some jobs extend the base template using '.deploy_mutable_image_tags_base' while the defined base template is '.deploy_mutable_image_tags-base_rc'. Please verify and update the template reference for consistency.
deploy_mutable_image_tags-a7-full-rc:
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR updates the jobs run during the Agent RC build to update mutable tags based on the image indexes published with the immutable tag, instead of pushing whole new index again.
We have a new method in public-images which can just create a tag if we provide
IMG_TAG_REFERENCEandIMG_NEW_TAGSMotivation
Simplify and improve image publish jobs.
Describe how you validated your changes
Changes validated locally, there will be still a need to check in the actual RC build which is not easy to simulate without poisoning the git repo and repositories.