Kong 3.9.0 - Upstream virtual host (Load balancer) DNS resolution failed

[pseralat]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

Kong 3.9.0

Current Behavior

I am encountering DNS resolution failure when I create upstream (load balancer) with routes and target after Kong is up. DNS is not resolving the upstream host (virtual).

    {
        "ca_certificates": null,
        "created_at": 1738815123,
        "updated_at": 1738815123,
        "id": "58dd5b75-3bdf-406e-a547-912073ec7a20",
        "retries": 5,
        "name": "SampleReadService",
        "enabled": true,
        "path": null,
        "write_timeout": 3600000,
        "client_certificate": null,
        "protocol": "https",
        "connect_timeout": 60000,
        **"host": "sample_read_upstream",**
        "read_timeout": 3600000,
        "tls_verify": null,
        "port": 443,
        "tls_verify_depth": null,
        "tags": null
    },

2025/02/06 04:15:54 [error] 1417#0: *2077 [lua] init.lua:381: execute(): DNS resolution failed: dns server error: 3 name error. Tried: ["(short)sample_read_upstream:(na) - cache-miss","sample_read_upstream.dns.podman:33 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream:33 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream.dns.podman:1 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream:1 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream.dns.podman:5 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream:5 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream.dns.podman:28 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream:28 - cache-miss/querying/dns server error: 3 name error"], client: 10.25.130.124, server: kong, request: "GET /ers/sdk HTTP/1.1", host: "172.23.160.98:44331", referrer: "https://172.23.160.98:44331/admin/", request_id: "cb4c634e45bc330f634ece2731f46c2f"

If iI restart the kong, then DNS is reloaded and it works fine. The same config used to work in Kong 2.0.3. I already tried reducing dns_stale_ttl but no help.

Expected Behavior

The upstream should be resolvable and the request should go through

Steps To Reproduce

1. Start Kong
2. Create upstream service with loadbalancer in round robin pointing to backend target IP's
3. Try sending the request and DNS failure happens

Anything else?

No response

[jeremyjpj0916]

Looks like you are seeing DNS related res issues too, see my issue, I think there is some DNS res problems in Kong right now.

[chobits]

hi ,
It's so strange behaviour, from your error, kong indeed found a dns error, and it reported it as

["(short)sample_read_upstream:(na) - cache-miss","sample_read_upstream.dns.podman:33 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream:33 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream.dns.podman:1 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream:1 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream.dns.podman:5 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream:5 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream.dns.podman:28 - cache-miss/querying/dns server error: 3 name error","sample_read_upstream:28 - cache-miss/querying/dns server error: 3 name error"

it seems that after the dns records cache expired, kong tried to resolute it then all these dns resolving chains failed.

[chobits]

hi in kong 3.9.0, we provide a new dns library which is implemented easier and has some improved debugability, you could try this new one, see this doc: https://docs.konghq.com/gateway/latest/migrate-to-new-dns-client/.

Then let us see whether there is still this problem.