Link visualizations concerns: too many incoming links, showing transitive links

[lmolkova]

When working on representing connections as spans (e.g. open-telemetry/semantic-conventions#756 or open-telemetry/semantic-conventions#1192), we'd like to be able to

• record connection (setup and/or lifetime) as a span
• record link from HTTP request to the connection

As a result, it could be possible to detect connection pool misconfiguration, leaked connection streams, see when request is slow because it's waiting for connection to be initiated or becomes available to it, etc.

Problem

There are a couple visualization concerns here:

1. The single connection serves 100-10000+ requests. When I open a visualization for a trace for specific request, I don't really want to see all other requests that were served on this connection.
2. Visualizing connection span (with a lot of incoming links) is useful when there are a few, for example, in dev/test environment, but overwhelming in common case. Having the data to analyze is still useful.
3. Connection initiation and request could be minutes apart, spans become tiny on a scale that shows time between operations.

Note:
This does not affect backends that don't show incoming links (such as Jaeger).

Questions:

How do backends visualize links? Are they ready to show tons of incoming links and potentially huge graphs?

• Do they show incoming links? Azure Monitor does, are there others?
• Do they show the whole graph? E.g. if A and C are both linked to B, and I open trace A, should I see A->B or all three? Is there a depth limit?
• What if linked operations are minutes apart? That's common in messaging scenarios. Do backends show it nicely? (e.g. add a break in the timeline)

And a couple of fundamental questions:

• Is links the right tool for the job? It seems the answer is yes - we want to record relationships between spans in different traces.
• If not links then what? Events + artificial connection-id to correlate on and give up on visualization?

[danielo515]

I have a very similar problem. At first, I thought the solution was to create custom traces out of our correlation IDs and force them on every span handling a message. But given how hard OTEL sdl fights back to prevent us from setting our own trace IDs I star to think it is not the right solution. Then someone suggested to use links, and I have the same exact concerns: will I be able to see connected traces? That is the whole reason I want to use open telemetry, so if the answer is no, I will be quite disappointed

[lailabougria]

I believe it's essential to inline the linked traces, especially in asynchronous contexts like messaging. In these cases, the conventions recommend using span links for each message, and if no inlining happens at all, the visualization becomes useless. As a user, you're back to the per-service call stack, which doesn't provide the distributed context needed to understand what's happening, and the cognitive load explodes by having to look at all the trees side by side. In building a message-based system, I appreciate Azure Monitor's ability to inline span links.

Note: With inlining, I'm refering to showing the linked span within the tree, and even allowing you to expand those details within the tree without having to navigate to another view.

There are specific scenarios in which it's beneficial to have an inlined view:

• Retries => Let's say there is a retry back-off mechanism creating span links due to the time between operations. These are typically things you would always like to see inlined as they are part of the original context but are split due to the asynchronous nature of the system. In addition, you don't have unlimited retries, so it shouldn't cause an explosion of links unless your system is massively struggling and is executing retries across the entire tree.
• Message conversations => In an asynchronous system, a single business operation consists of multiple messages exchanged back and forth. It would be helpful to allow users to define the boundaries of the conversation, allowing backends to build visualizations based on the conversation.

However, I've also seen the flipside be true, as you mention here: when you open a trace that has span links across the entire message conversation, the context explodes, and you lose the benefits of inlining because the cognitive load within a single trace is too large.

Going into solution mode, would it be possible to differentiate the types of span links to allow for different decisions for inlining?
For example, if you set the LinkType to "Inline", the observability backend would know to inline the linked span; if it's set to "Default", the observability backend could show it as a linked span without inlining it. This way, it becomes the user's decision where the boundary of the inlined spans starts and ends, so to speak. Would that be a possibility?

[lmolkova]

Thanks for the input!

In OTel, we've been trying to avoid having "how backends should visualize this" on the telemetry data in a sense that the data should contain enough information to build useful visualizations, but we expect different backends to provide different visualizations for the same data.

It seems the ask is to let users express group-of-service boundaries and then visualization tools can use this information to focus on specific parts of the trace to reduce noise - is this correct?

I'm specifically separating it from the link type - it could be a part of the solution, but maybe there are some other ways and parts.
If we talk about link types, we'll need to define a set of possible types, pick a default and a configuration mechanism, document a principle on where to use which.

Speaking of the Azure Monitor UX above, it's extremely noisy since when I open trace A, it opens every other trace linked to it and also all links to those links recursively. I'd be a better default if it expanded trace A and showed all direct links to/from it but didn't show indirect links (links to links and so on). This is something that can be done purely on the UX without any modification on the data itself.

I have limited exposure into other backends that support multi-directional links, but I wonder if we can do much better wrt focusing on specific part of the trace based on the existing concepts and data + providing group-of-services boundaries in the UX rather than inside the application code/telemetry.

[lailabougria]

It seems the ask is to let users express group-of-service boundaries and then visualization tools can use this information to focus on specific parts of the trace to reduce noise - is this correct?

Exactly, that would address both scenarios:

• this would allow users to define the meaningful bits and pieces that they want to see together in a single tree because it makes sense from their point of view (where we can never have that insight from the otel-perspective)
• any retries would be part of the group-of-service-boundaries as defined by the user which is desirable too

For example in a simplified retail scenario, if you place an order, charge the credit card, and later ship it and adjust stock, it may be useful to define one boundary encompassing the order placement and payment and another one that defines the boundary for shipment and stock update. By allowing the user to somehow control that boundary, they can decide what makes sense in their context to consider as part of a single "business operation" if you will. One user may want to see all these actions in a single boundary and some may want a boundary for each part, depending on their needs. For the Azure Monitor UX case, the UI would be as noisy as the user chooses to make it as they control that boundary, without the tooling having to make hard decisions about what to add or exclude.

Link type was just the first idea that came to mind but other solutions are absolutely welcome.