Duplicate OtlpMeterRegistry thread running in addition to my custom created one

[ashok2709]

i have created my custom OtlpMeterRegistry object and running that in my app, It is pushing metrics to the endpoint which i mentioned in ENV value correctly.

But I am seeing some error logs in which some other thread named 'otlp-metrics-publisher' is trying to export metrics to the url "http://localhost:4318/v1/metrics" and getting failed, how to remove/disable that?

Note: I have added my OtlpMeterRegistry object to a compositeMeterRegistry and using that.
otel collector endpoint is set in the ENV as "OTEL_EXPORTER_OTLP_ENDPOINT".

---

MY CODE:

        OtlpConfig otlpConfig = new OtlpConfig() {
            @Override
            public String get(@NonNull String key) {
                return null;
            }

            @Override
            @NonNull
            public Duration step() {
                return Duration.ofSeconds(10);
            }
        };

        OtlpMeterRegistry otlpMeterRegistry = new OtlpMeterRegistry(otlpConfig, Clock.SYSTEM, new NamedThreadFactory("otlp.metrics.publisher.service-name"));

---

ERROR LOG:

2025-02-26 11:27:26.448 WARN [my-service-name : otlp-metrics-publisher : {}] : java.net.ConnectException: Connection refused at java.base/sun.nio.ch.Net.pollConnect(Native Method) at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:535) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:585) at java.base/java.net.Socket.connect(Socket.java:666) at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636) at java.base/sun.net.www.http.HttpClient.(HttpClient.java:279) at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:384) at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:406) at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1308) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127) at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1056) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1429) at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1400) at io.micrometer.core.ipc.http.HttpUrlConnectionSender.send(HttpUrlConnectionSender.java:98) at io.micrometer.core.ipc.http.HttpSender$Request$Builder.send(HttpSender.java:306) at io.micrometer.registry.otlp.OtlpMeterRegistry.publish(OtlpMeterRegistry.java:185) at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafelyOrSkipIfInProgress(PushMeterRegistry.java:64) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Unknown Source) Failed to publish metrics to OTLP receiver (context: url=http://localhost:4318/v1/metrics, resource-attributes={service.name=my-service-name})

---

OTLP METER REGISTRY INTIALISATION LOGS FROM MICROMETER:

Publishing metrics for OtlpMeterRegistry every 1m to http://localhost:4318/v1/metrics with resource attributes {service.name=service-name, environment=int}

Publishing metrics for OtlpMeterRegistry every 10s to http://:4318/v1/metrics with resource attributes {environment=int, service.name=service-name}

[shakuzen]

It sounds like something else is creating a registry separate from the one you are making. You should check the documentation for the framework you are using to see why it would be creating a registry and what you should do to customize that registry. I can't help with that without more info or a minimal sample application that reproduces the issue.

[ashok2709]

HI @shakuzen the error log contains the url "localhost:4318/v1/metrics", this url will be returned ny OtlpConfig.url() method

OtlpConfig.url() method code of micrometer:

default String url() {
        return (String)PropertyValidator.getUrlString(this, "url").orElseGet(() -> {
            Map<String, String> env = System.getenv();
            String endpoint = (String)env.get("OTEL_EXPORTER_OTLP_METRICS_ENDPOINT");
            if (endpoint == null) {
                endpoint = (String)env.get("OTEL_EXPORTER_OTLP_ENDPOINT");
            }

            if (endpoint == null) {
                endpoint = "http://localhost:4318/v1/metrics";
            } else if (!endpoint.endsWith("/v1/metrics")) {
                endpoint = endpoint + "/v1/metrics";
            }

            return endpoint;
        });
    }

My application is having the ENV key "OTEL_EXPORTER_OTLP_ENDPOINT"
So by this code, the config url of the duplicate registry should be the endpoint mentioned in the ENV "OTEL_EXPORTER_OTLP_ENDPOINT"

My doubt is, even though another OtlpMeterRegistry is created by something and running, it should try to send it to endpoint mentioned in my ENV right? if ENV value is not there only it should try with default url "localhost:4318/v1/metrics"(according to the OtlpConfig.url() code)

[shakuzen]

OtlpConfig is an interface and what you've shared is the default implementation of the url method. Whatever is creating the other registry could be using its own implementation of the method. Even if it were using the value from the environment variable, you would still have two registries publishing to that endpoint, which presumably isn't what you want either. All of that is a bit beside the point I tried to make in my previous comment. So let me ask specifically: are you using Spring Boot? If so, have you read the corresponding documentation? If you are making your instance of OtlpMeterRegistry in a way that Spring does not manage it (i.e. it is not a @Bean), then Spring Boot will auto-configure its own instance of OtlpMeterRegistry separate from the one you are making. If instead you made your OtlpMeterRegistry a bean so Spring Boot knows about it, its auto-configuration would back off and it would not create its own OtlpMeterRegistry.

[ashok2709]

@shakuzen the issue is fixed now
management.otlp.metrics.export.enabled=false
This application property disabled the creation of that default meter registry
Because of this, micrometer created another meter registry with default configurations on its own.

[shakuzen]

Because of this, micrometer created another meter registry with default configurations on its won.

That is not correct. Micrometer does not ever create a MeterRegistry on its own. Micrometer knows nothing about management.otlp.metrics.export.enabled. That is a Spring Boot configuration property, and it is Spring Boot that has auto-configuration as a feature. It would have saved us both time for you to mention in the beginning that you are using Spring Boot and to check the Spring Boot documentation. It's also unnecessary to explicitly disable the auto-configuration if you make your custom OtlpMeterRegistry a bean, as I stated in my last comment. You haven't mentioned why you are creating a custom OtlpMeterRegistry , but you can probably achieve the same thing using the Spring Boot auto-configuration rather than disabling it and creating an object separately that won't be managed (e.g. won't be closed when the application shuts down). In any case, there's no issue to be fixed in Micrometer, so I'm closing this issue and marking it as invalid.

[izeye]

My doubt is, even though another OtlpMeterRegistry is created by something and running, it should try to send it to endpoint mentioned in my ENV right? if ENV value is not there only it should try with default url "localhost:4318/v1/metrics"(according to the OtlpConfig.url() code)

I created spring-projects/spring-boot#44493 to see if it was intentional or just an oversight.