Add .NET network + HTTP connection spans #1192

lmolkova · 2024-06-27T23:20:31Z

This PR introduces additional network-level spans to-be emitted by .NET 9 🤞

Related to #454, #1226

In particular:

DNS, TLS, and socket connect spans that track individual stages of connection establishment
wrapper HTTP connection setup span for above spans
wait-for-connection span that signals when a connection is not immediately available for the request
- link that connects wait-for-connection to the connection setup
- is expected to be off by default

On .NET every ActivitySource (aka tracer) is opt-in, so users have full control over what they are enabling.

The volume of new telemetry is expected to be low (with caveats):

The connection-level spans (groups 1 and 2 above) are rare - connections are created rarely.
Wait-for-connection spans are only reported when connection is not readily available and would be present when connection pool is overloaded.
Links to connection setup are most concerning since we don't know how backends are handling incoming links. See Link visualizations concerns: too many incoming links, showing transitive links opentelemetry-specification#4130 for the context. This is off by default docs)?

Overall, the feature is intended to help debug issues like:

misconfigured connection pool
DNS/TLS
abandoned response streams (or slow response stream reading) that result in connection staying busy
...

docs/dotnet/dotnet-connection-traces.md

docs/attributes-registry/dns.md

docs/dotnet/dotnet-connection-traces.md

docs/dotnet/dotnet-network-traces.md

.NET 9 introduced additions to `System.Net` distributed tracing which define a contract following OTel recommendations: - dotnet/runtime#104251 adjusted the HTTP client request activity so it natively complies with OTel recommendations. This included adding standard tags. - dotnet/runtime#103922 introduced activities breaking down the HTTP connection setup. This PR documents the activities together with their attributes in a manner similar to the metrics docs added by #37213. The matching semconv PR is open-telemetry/semantic-conventions#1192. Conceptual docs are being introduced in #42830. Moreover, the PR also adjusts certain parts of the metrics doc in order to make it more accurate, synchronize it with the activities doc and incorporate the changes from #42810. --------- Co-authored-by: Genevieve Warren <[email protected]> Co-authored-by: Miha Zupan <[email protected]>

jsuereth · 2025-03-04T18:30:42Z

model/dns/registry.yaml

@@ -16,3 +16,9 @@ groups:
          as escaped base 10 integers (\DDD). Back slashes and quotes should be escaped.
          Tabs, carriage returns, and line feeds should be converted to \t, \r, and
          \n respectively.
+      - id: dns.answers


This feels like a request/response logging mechanism.

Is there a reason .NET prefers these as span attributes vs. Events?

Note: I don't want to open the giant can of worms of Events vs. Span Attributes. I'm just trying to understand rationale here and from the non-.NET portion of semantic conventions, I don't think this impacts choice of span-attribute vs. event-attribute for dns.answers.

I just noticed your comment @jsuereth

what would be the reason to report dns.answers in an event?

Here's an example of this PR in action

DNS lookup takes 30ms and can fail, it's a valid span. And then there are a few addresses lookup returns, they make sense on the span that captures DNS lookup.

Am I missing something?

docs/dotnet/dotnet-network-traces.md

…smatch between yanml and md

lmolkova · 2025-04-21T15:39:11Z

docs/dotnet/dotnet-network-traces.md

+It's reported as a child of *HTTP client request* span.
+
+The span ends when the connection is obtained - it could happen when an existing connection becomes available or once
+a new connection is established, so the duration of *Wait For Connection* span is different from duration of the


let's try to generalize it and try to address similar problems - concurrency, mutexes, etc, look at absl

lmolkova commented Jun 27, 2024

View reviewed changes

docs/dotnet/dotnet-connection-traces.md Outdated Show resolved Hide resolved

antonfirsov reviewed Jun 28, 2024

View reviewed changes

docs/dotnet/dotnet-connection-traces.md Outdated Show resolved Hide resolved

docs/dotnet/dotnet-connection-traces.md Outdated Show resolved Hide resolved

docs/dotnet/dotnet-connection-traces.md Outdated Show resolved Hide resolved

lmolkova force-pushed the dotnet-http-connection-spans branch from 7d7c232 to 9a4ba14 Compare July 3, 2024 00:59