Skip to content

Deconfigure istio proxy tracer explicit to avoid DNS lookups #17811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 18, 2023

Conversation

a-thaler
Copy link
Contributor

@a-thaler a-thaler commented Jul 7, 2023

Description

Debugging a kyma cluster using tcpdump on a privileged pod reveals that constantly DNS lookups for zipkin.istio-system are performed. Analyzing the configuration further revealed that the extensionProvider configuration mechanism seems to enable the zipkin tracer additionally to the configured providers. An upstream ticket got created for now istio/istio#45890, we still need to see if we can contribute a fix.
As the main problem is the additional stress to CoreDNS with negative DNS lookups, this PR disabled explicitly the tracer in the proxy config as described as a workaround in the upstream bug report.

To validate that the change will not cause any additional DNS entries, deploy a privileged pod (for example using the Gardener terminal) and execute tcpdump -i any -vvv -s0 -w capture.pcap.
Then copy the captured file using

kubectl -n term-host-11327686835437096980 cp term-11327686835437096980:/capture.pcap capture.pcap

and analyze the file using wireshark. Enter "dns" as the filter criteria.

Changes proposed in this pull request:

  • disable proxy tracer explicitly to avoid negative DNS lookups

Related issue(s)

istio/istio#45890

@a-thaler a-thaler added kind/bug Categorizes issue or PR as related to a bug. area/service-mesh Issues or PRs related to service-mesh area/tracing Issues or PRs related to the tracing module (deprecated) area/telemetry Issues or PRs related to the telemetry module labels Jul 7, 2023
@a-thaler a-thaler added this to the 2.17 milestone Jul 7, 2023
@a-thaler a-thaler requested a review from a team as a code owner July 7, 2023 15:19
@netlify
Copy link

netlify bot commented Jul 7, 2023

Deploy Preview for kyma-project-docs-preview ready!

Name Link
🔨 Latest commit f6e54da
🔍 Latest deploy log https://app.netlify.com/sites/kyma-project-docs-preview/deploys/64b154f7fd4d1d00081821a4
😎 Deploy Preview https://deploy-preview-17811--kyma-project-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@kyma-bot kyma-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jul 7, 2023
@skhalash skhalash self-assigned this Jul 10, 2023
@a-thaler a-thaler changed the title set default trace url of istio to a local IP to avoid DNS lookups Deconfigure istio proxy tracer explicit to avoid DNS lookups Jul 12, 2023
@shorim
Copy link
Contributor

shorim commented Jul 13, 2023

/retest

@kyma-bot
Copy link
Contributor

@a-thaler: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kyma-governance c89c3ae link true /test pull-kyma-governance

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kyma-bot kyma-bot added the lgtm Looks good to me! label Jul 14, 2023
@kyma-bot kyma-bot merged commit 0d97103 into kyma-project:main Jul 18, 2023
@a-thaler a-thaler deleted the istio branch July 18, 2023 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/service-mesh Issues or PRs related to service-mesh area/telemetry Issues or PRs related to the telemetry module area/tracing Issues or PRs related to the tracing module (deprecated) kind/bug Categorizes issue or PR as related to a bug. lgtm Looks good to me! size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants