[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

djaglowski · 2025-04-08T19:08:38Z

Depends on #12856

Resolves #12676

This is a reboot of #11311, incorporating metrics defined in the component telemetry RFC and attributes added in #12617.

The basic pattern is:

When building any pipeline component which produces data, wrap the "next consumer" with instrumentation to measure the number of items being passed. This wrapped consumer is then passed into the constructor of the component.
When building any pipeline component which consumes data, wrap the component itself. This wrapped consumer is saved onto the graph node so that it can be retrieved during graph assembly.

TODO:

There are no tests which directly validate the presence or content of the metrics. I've done some manual validation and believe things are working correctly. I'll work on unit tests soon but would appreciate feedback on the PR regardless.

Next Steps:

This defines but does not implement the "size" metrics. (It both defines and implements the item count metrics.)

codecov · 2025-04-08T19:30:13Z

Codecov Report

Attention: Patch coverage is 86.89655% with 57 lines in your changes missing coverage. Please review.

Project coverage is 91.56%. Comparing base (d020c90) to head (d882a04).

Files with missing lines	Patch %	Lines
service/internal/graph/connector.go	63.90%	32 Missing and 16 partials ⚠️
service/internal/graph/exporter.go	85.71%	2 Missing and 1 partial ⚠️
service/internal/graph/processor.go	89.65%	2 Missing and 1 partial ⚠️
service/internal/graph/receiver.go	75.00%	2 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (86.89%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12812      +/-   ##
==========================================
- Coverage   91.65%   91.56%   -0.10%     
==========================================
  Files         499      499              
  Lines       27426    27801     +375     
==========================================
+ Hits        25138    25456     +318     
- Misses       1809     1847      +38     
- Partials      479      498      +19

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jaronoff97

some initial thoughts, overall implementation seems very sane to me though. Thank you!

service/internal/graph/connector.go

service/internal/graph/exporter.go

service/metadata.yaml

service/documentation.md

docs/rfcs/component-universal-telemetry.md

service/metadata.yaml

codeboten

Thanks for moving this work forward @djaglowski, just a few questions

codeboten · 2025-04-11T15:25:15Z

service/internal/graph/receiver.go

+		obsConsumer := obsconsumer.NewTraces(fanoutconsumer.NewTraces(consumers), tb.ReceiverProducedItems)
+		n.Component, err = builder.CreateTraces(ctx, set, obsConsumer)


Not a blocking comment:

Suggested change

obsConsumer := obsconsumer.NewTraces(fanoutconsumer.NewTraces(consumers), tb.ReceiverProducedItems)

n.Component, err = builder.CreateTraces(ctx, set, obsConsumer)

n.Component, err = builder.CreateTraces(ctx, set, obsconsumer.NewTraces(fanoutconsumer.NewTraces(consumers), tb.ReceiverProducedItems))

djaglowski · 2025-04-11T16:25:16Z

Thanks for the reviews.

#12817 implements a subset of this PR. If we can get that merged in first, I'll rebase to reduce the scope of this one substantially.

MikeGoldsmith

I think this looks really good 👍🏻

As shared in the other PR, I'd really like to see bytes counters too but this is a great start.

Subset of #12812 This internal package defines wrappers around consumers. These are useful for instrumenting the component graph, so that we can generate telemetry describing data as it is passed in between components. Currently, this supports only a single counter metric, but in the near future it can be enhanced to automatically capture multiple metrics (e.g. item count & size), and potentially spans and/or logs as well.

… data point

This reverts commit 836b2b3.

djaglowski · 2025-04-23T17:04:05Z

I am pretty stumped on the datadog exporter integration test failure. I don't see anything explicitly indicating that it is related to this change but it does appear to fail consistently on this PR. The panic indicates a relation to prometheus, which made me suspect that the new metric naming format (using . delimiter`) could be the culprit. However, I tested this by temporarily switching to the old metric name format and observed that it failed in the same way. (836b2b3)

@mx-psi is there anyone at DataDog that might be able to provide more insight into this test?

codeboten · 2025-04-23T21:18:41Z

service/internal/graph/processor.go

+		if err != nil {
+			return fmt.Errorf("failed to create %q processor, in pipeline %q: %w", set.ID, n.pipelineID.String(), err)
+		}
+		n.consumer = obsconsumer.NewProfiles(n.Component.(xconsumer.Profiles), tb.ProcessorConsumedItems)


this could be done once after the switch statement

codeboten · 2025-04-23T22:15:56Z

service/metadata.yaml

+    receiver.produced.size:
+      prefix: otelcol.
+      enabled: false
+      description: Size of items emitted from the receiver.
+      unit: "By"
+      sum:
+        value_type: int
+        monotonic: true
+    processor.consumed.size:
+      prefix: otelcol.
+      enabled: false
+      description: Size of items passed to the processor.
+      unit: "By"
+      sum:
+        value_type: int
+        monotonic: true
+    processor.produced.size:
+      prefix: otelcol.
+      enabled: false
+      description: Size of items emitted from the processor.
+      unit: "By"
+      sum:
+        value_type: int
+        monotonic: true
+    connector.consumed.size:
+      prefix: otelcol.
+      enabled: false
+      description: Size of items passed to the connector.
+      unit: "By"
+      sum:
+        value_type: int
+        monotonic: true
+    connector.produced.size:
+      prefix: otelcol.
+      enabled: false
+      description: Size of items emitted from the connector.
+      unit: "By"
+      sum:
+        value_type: int
+        monotonic: true
+    exporter.consumed.size:
+      prefix: otelcol.
+      enabled: false
+      description: Size of items passed to the exporter.
+      unit: "By"
+      sum:
+        value_type: int
+        monotonic: true


if the size metrics aren't used yet, I'd suggest removing these from the PR

codeboten · 2025-04-23T22:19:52Z

I am pretty stumped on the datadog exporter integration test failure. I don't see anything explicitly indicating that it is related to this change but it does appear to fail consistently on this PR. The panic indicates a relation to prometheus, which made me suspect that the new metric naming format (using . delimiter`) could be the culprit. However, I tested this by temporarily switching to the old metric name format and observed that it failed in the same way. (836b2b3)

@djaglowski not sure if it's helpful, but I ran into a similar issue back in the PR that was updating the collector to use the otel config package from otel-go and @songy23 commented on the failure: #11611 (comment)

codeboten · 2025-04-23T22:24:25Z

I built the collector using the branch and when running it, this is what I see at localhost:8888/metrics:

An error has occurred while serving metrics:

2 error(s) occurred:
* collected metric "otelcol_processor_batch_metadata_cardinality" { label:{name:"otel_scope_name" value:"go.opentelemetry.io/collector/processor/batchprocessor"} label:{name:"otel_scope_version" value:""} label:{name:"processor" value:"batch"} gauge:{value:1}} was collected before with the same name and label values
* collected metric "otelcol_processor_batch_metadata_cardinality" { label:{name:"otel_scope_name" value:"go.opentelemetry.io/collector/processor/batchprocessor"} label:{name:"otel_scope_version" value:""} label:{name:"processor" value:"batch"} gauge:{value:1}} was collected before with the same name and label values

codeboten · 2025-04-23T22:40:46Z

testing this further, this appears to be the current behaviour in main, it looks like something broke in the last 20 commits

codeboten · 2025-04-23T22:44:06Z

Looks like things have been unhappy since this change: dc8e2dd

dmitryax

The PR LGTM. However, I'd like to get an agreement on #12916 before releasing this

songy23 · 2025-04-24T01:29:32Z

I built the collector using the branch and when running it, this is what I see at localhost:8888/metrics:

I just tested and saw the same, internal metrics are broken at mainline head.

songy23 · 2025-04-24T01:40:32Z

Looks like things have been unhappy since this change: dc8e2dd

Yes, reverting that commit fixes it http://github.com/open-telemetry/opentelemetry-collector/pull/12917

github-actions bot requested review from bogdandrutu, codeboten, dmitryax and mx-psi April 8, 2025 19:09

djaglowski changed the title ~~Compiling and tests passing~~ [service/internal/graph] Record normalized telemetry as it is passed between pipeline components Apr 8, 2025

djaglowski changed the title ~~[service/internal/graph] Record normalized telemetry as it is passed between pipeline components~~ [service/internal/graph] Measure telemetry as it is passed between pipeline components Apr 8, 2025

djaglowski force-pushed the pipeline-component-metrics branch 2 times, most recently from b6bb02d to 2f83f2b Compare April 8, 2025 19:20

jaronoff97 reviewed Apr 8, 2025

View reviewed changes

service/internal/graph/connector.go Outdated Show resolved Hide resolved

service/internal/graph/connector.go Show resolved Hide resolved

service/internal/graph/exporter.go Show resolved Hide resolved

service/metadata.yaml Show resolved Hide resolved

djaglowski force-pushed the pipeline-component-metrics branch from 2f83f2b to 052cffc Compare April 8, 2025 20:36

mx-psi reviewed Apr 9, 2025

View reviewed changes

service/documentation.md Outdated Show resolved Hide resolved

djaglowski commented Apr 9, 2025

View reviewed changes

docs/rfcs/component-universal-telemetry.md Outdated Show resolved Hide resolved

djaglowski force-pushed the pipeline-component-metrics branch from 052cffc to 362a610 Compare April 9, 2025 16:39

djaglowski mentioned this pull request Apr 9, 2025

[chore] Create service/internal/obsconsumer package #12817

Merged

djaglowski force-pushed the pipeline-component-metrics branch 2 times, most recently from fbed573 to b789109 Compare April 9, 2025 20:24

codeboten reviewed Apr 11, 2025

View reviewed changes

service/metadata.yaml Show resolved Hide resolved

service/metadata.yaml Show resolved Hide resolved

codeboten reviewed Apr 11, 2025

View reviewed changes

MikeGoldsmith approved these changes Apr 14, 2025

View reviewed changes

djaglowski force-pushed the pipeline-component-metrics branch from b789109 to 4afe6cd Compare April 15, 2025 13:13

djaglowski mentioned this pull request Apr 15, 2025

Permanently enable 'telemetry.newPipelineTelemetry' feature gate #12856

Merged

djaglowski force-pushed the pipeline-component-metrics branch from 4afe6cd to 5bdd398 Compare April 15, 2025 14:29

github-actions bot requested a review from dmathieu April 15, 2025 14:29

djaglowski force-pushed the pipeline-component-metrics branch 4 times, most recently from adec160 to b66a5a8 Compare April 17, 2025 14:47

djaglowski added 8 commits April 22, 2025 14:10

Move gate to service/telemetry

433c162

Add telemetry.ownMetricsUsePeriodPrefix feature gate

87f6eb2

Remove 'telemetry.newPipelineTelemetry' feature gate

9196c81

Readd gate and set to stable

a52f5ba

Remove 'telemetry.newPipelineTelemetry' feature gate

06a530b

[chore] Create service/internal/obsconsumer package

b5d8f1b

Compiling and tests passing

19fa989

Create obsconsumer package and switch connector produced attribute to…

29df66a

… data point

djaglowski force-pushed the pipeline-component-metrics branch 2 times, most recently from 4bfa6c4 to c03ad2c Compare April 22, 2025 22:22

Add test which validates expected metrics

1a1d65a

djaglowski force-pushed the pipeline-component-metrics branch from c03ad2c to 1a1d65a Compare April 22, 2025 22:32

djaglowski closed this Apr 22, 2025

djaglowski reopened this Apr 22, 2025

djaglowski added 2 commits April 23, 2025 08:14

Run integration tests with snake case metric names

836b2b3

Revert "Run integration tests with snake case metric names"

5489cb4

This reverts commit 836b2b3.

djaglowski marked this pull request as ready for review April 23, 2025 17:00

djaglowski requested a review from a team as a code owner April 23, 2025 17:00

djaglowski added 2 commits April 23, 2025 11:20

Remove outdated approach to replace gate

9477c7c

Merge branch 'main' into pipeline-component-metrics

d882a04

codeboten reviewed Apr 23, 2025

View reviewed changes

dmitryax reviewed Apr 24, 2025

View reviewed changes

songy23 mentioned this pull request Apr 24, 2025

[chore] Revert dc8e2dd #12917

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

djaglowski commented Apr 8, 2025 •

edited

Loading

codecov bot commented Apr 8, 2025 •

edited

Loading

jaronoff97 left a comment

codeboten left a comment

codeboten Apr 11, 2025

djaglowski commented Apr 11, 2025

MikeGoldsmith left a comment

djaglowski commented Apr 23, 2025

codeboten Apr 23, 2025

codeboten Apr 23, 2025

codeboten commented Apr 23, 2025

codeboten commented Apr 23, 2025

codeboten commented Apr 23, 2025

codeboten commented Apr 23, 2025

dmitryax left a comment

songy23 commented Apr 24, 2025

songy23 commented Apr 24, 2025

		obsConsumer := obsconsumer.NewTraces(fanoutconsumer.NewTraces(consumers), tb.ReceiverProducedItems)
		n.Component, err = builder.CreateTraces(ctx, set, obsConsumer)

[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

Are you sure you want to change the base?

[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

Conversation

djaglowski commented Apr 8, 2025 • edited Loading

codecov bot commented Apr 8, 2025 • edited Loading

Codecov Report

jaronoff97 left a comment

Choose a reason for hiding this comment

codeboten left a comment

Choose a reason for hiding this comment

codeboten Apr 11, 2025

Choose a reason for hiding this comment

djaglowski commented Apr 11, 2025

MikeGoldsmith left a comment

Choose a reason for hiding this comment

djaglowski commented Apr 23, 2025

codeboten Apr 23, 2025

Choose a reason for hiding this comment

codeboten Apr 23, 2025

Choose a reason for hiding this comment

codeboten commented Apr 23, 2025

codeboten commented Apr 23, 2025

codeboten commented Apr 23, 2025

codeboten commented Apr 23, 2025

dmitryax left a comment

Choose a reason for hiding this comment

songy23 commented Apr 24, 2025

songy23 commented Apr 24, 2025

djaglowski commented Apr 8, 2025 •

edited

Loading

codecov bot commented Apr 8, 2025 •

edited

Loading