[BUG] APM remote configuration not work on 7.65.0

[DingGGu]

Agent Environment

Describe what happened:

I setup APM with Datadog Agent, but after updating to the latest (7.65.0) version, an error related to Remote Configuration is occurring from the application that configured APM.

Getting error on dd-trace-rb:

Datadog::Core::Remote::Component#initialize') remote worker error: Datadog::Core::Remote::Client::TransportError #<Datadog::Core::Transport::InternalErrorResponse:0x0000>, error_type:Datadog::Core::Remote::Transport::HTTP::Config::Response::ParseError error:could not parse key :roots: #<JSON::ParserError:"Empty input (after ) at line 1, column 1 [parse.c:1074] in 'rpc error: code = Unknown desc = empty targets meta in director local store\n">

Getting error on dd-trace-java:

[dd-remote-config] WARN datadog.remoteconfig.ConfigurationPoller - Failed to retrieve remote configuration: unexpected response code Internal Server Error 500 rpc error: code = Unknown desc = empty targets meta in director local store
 (Will not log warnings for 5 minutes)

Describe what you expected:

Steps to reproduce the issue:

Upgrade to 7.65.0 from 7.61.0

Additional environment details (Operating System, Cloud provider, etc):
Kubernetes with Helm installation, Helm: datadog-3.112.0

[ajgajg1134]

👋 Hello! Thanks for opening an issue here, to better investigate this can you please open a Datadog Support Ticket? That will let us better investigate your exact setup and collect more information to understand the scope of the issue here.

[cbistv2]

We we experienced the same issue, using datadog-3.113.0 helm chart, but in our case the issue was because even tho remote config was enabled, no API keys was actually authorized to connect to the remote config service, once we enabled a api key, the errors disappeared along with tons of warning logs like these:
CORE | WARN | (pkg/config/remote/service/service.go:872 in ClientGetConfigs) | Timestamp expired at 0001-01-01T00:00:00Z, flushing cache

we upgraded from helm chart datadog-3.90.0 so quite a few versions, but something defiantly changed in regards to remote config, as we didint see this error before we upgraded, and remote config was enabled in the agent, but not in the backend before aswell without causing this error.

[janverhoeckx]

We have the exact same issue.

[DingGGu]

Rollback Datadog Agent to 7.64.3 will mitigate issue.

[ianchudson-caribou]

Rollback Datadog Agent to 7.64.3 will mitigate issue.

Confirming this worked for us as well.
We were upgrading and saw the huge warning spike, opened support ticket and they escalated. Then this search hit on Google.

[BinaryPaean]

The Helm charts just list the agent app version as "7". Are there release notes or other practical system to know which is the latest Helm chart version that has Agent v7.64.3?

[mikutas]

@BinaryPaean Here you are https://github.com/DataDog/helm-charts/blob/main/charts/datadog/CHANGELOG.md#311015

[BinaryPaean]

@BinaryPaean Here you are https://github.com/DataDog/helm-charts/blob/main/charts/datadog/CHANGELOG.md#311015

Thanks!

[JaggedJax]

Are there rollback instructions somewhere?

[BinaryPaean]

Are there rollback instructions somewhere?

If you installed via Helm you can re-install the chart at a target previous version. The changelog link @mikutas provided will let you find the helm chart version you want.

In our particular case the error appeared to be an intersection of using an API token issued well before the new permissions related to remote agent control existed. We explicitly disabled remote control in our agent config to resolve it for now. If/when we want the feature, we'll try with a new API token.

[miskr-instructure]

We explicitly disabled remote control in our agent config to resolve it for now. If/when we want the feature, we'll try with a new API token.

How was this done? The operator helm chart has remoteConfiguration.enabled but it is off by default, so unless the description is lying, it shouldn't be on unless you explicitly enabled it in the first place?

Looks like it is on by default in values.yaml at https://github.com/DataDog/helm-charts/blob/ca18b23d09f353de91d7bb765cdb540f19097c1b/charts/datadog/values.yaml#L2486-L2490 so it does need to be disabled explicitly.

[cynepco3hahue]

Hi folks, I unsure if it the same issue but I have error

E, [2025-05-24T09:43:50.163761 #1141] ERROR -- datadog: [datadog] (/var/www/ownbackup/current/vendor/gems/datadog-2.16.0/lib/datadog/core/remote/component.rb:52:in `rescue in block in initialize') remote worker error: Datadog::Core::Remote::Client::TransportError #<Datadog::Core::Transport::InternalErrorResponse:0x00007f7cc71e8110>, error_type:Datadog::Core::Remote::Transport::HTTP::Config::Response::ParseError error:could not parse key :roots: #<JSON::ParserError:"unexpected token at 'rpc error: code = Unimplemented desc = remote configuration service not initialized\n'"> location: /var/www/ownbackup/current/vendor/gems/datadog-2.16.0/lib/datadog/core/remote/client.rb:35:in `sync'. reseting client state

it is happening for me for version v7.64.3 I do not have anything special regarding to the configuration, just followed-up https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/ruby/#integration-instrumentation

Once I set DD_REMOTE_CONFIGURATION_ENABLED to false under datadog daemon set, it is starting to work.

[crice88]

We are receiving a similar error that's absolutely obliterating our logging in DD. The agent is running latest as a sidecar in our ECS task sets.

[ddtrace] (/usr/local/bundle/gems/ddtrace-1.20.0/lib/datadog/core/remote/component.rb:51:in `rescue in block in initialize') remote worker error: Datadog::Core::Remote::Client::TransportError #<Datadog::Core::Transport::InternalErrorResponse:0x00007f21c357dd98> location: /usr/local/bundle/gems/ddtrace-1.20.0/lib/datadog/core/remote/client.rb:112:in `sync'. reseting client state

A patch update would be nice 👍🏻

[krazylegz]

@crice88 I'm curious, do you have DD_REMOTE_CONFIGURATION_ENABLED set on your agent sidecar task?

[crice88]

@krazylegz waiting till midweek to set it. We have so many tasks that use the DD agent, we're hopeful a patch hits soon to mitigate the errors.

[griley]

Looks like version 2.17.0 of the datadog gem took care of the issue for me (I was previously at 2.14.0).