Skip to content

TraceQL metrics count_over_time yields different results for different time boundaries (and instant vs. range queries) #5164

@alexbikfalvi

Description

@alexbikfalvi

Describe the bug

Results are different for count_over_time() when running an range or instant queries with different time boundaries

An example is the following, where the query is {}|count_over_time()with(exemplars=0) and the time range is:

  • for the range query a 10 min (1748254200..1748254800) with steps of 1 min
  • for the instant query a 1 min range:
min start end query_range (step=1m) query (in 1 min steps)
0..1 1748254200 1748254260 11538 7102
1..2 1748254260 1748254320 12343 9403
2..3 1748254320 1748254380 13408 8722
3..4 1748254380 1748254440 11959 8314
4..5 1748254440 1748254500 11840 11327
5..6 1748254500 1748254560 13210 5145
6..7 1748254560 1748254620 12422 8473
7..8 1748254620 1748254680 12126 7296
8..9 1748254680 1748254740 12338 8419
9..10 1748254740 1748254800 12037 5294
sum 123221 79495

Note 1: For the above the range query yields a different result for the whole range with a step of 10min

  • query_range: sum(count_over_time()) for 200..800 step=1m: 123221
  • query_range: count_over_time() for 200..800 step=10m: 118635
  • query: count_over_time() for 200..800: 118635

Note 2: Other than this difference, the results are deterministic, where Tempo always returns the same results for the same input parameters.

Note 3: Extending the range query doesn't alter it's result for the overlapping range.

min start end query_range (1748254200..1748254200) query_range (1748254020..1748254980)
-3..-2 1748254020 1748254080 - 11942
-2..-1 1748254080 1748254140 - 13169
-1..0 1748254140 1748254200 - 12079
0..1 1748254200 1748254260 11538 11538
1..2 1748254260 1748254320 12343 12343
2..3 1748254320 1748254380 13408 13408
3..4 1748254380 1748254440 11959 11959
4..5 1748254440 1748254500 11840 11840
5..6 1748254500 1748254560 13210 13210
6..7 1748254560 1748254620 12422 12422
7..8 1748254620 1748254680 12126 12126
8..9 1748254680 1748254740 12338 12338
9..10 1748254740 1748254800 12037 12037
10..11 1748254800 1748254860 - 13218
11..12 1748254860 1748254920 - 12496
12..13 1748254920 1748254980 - 12364

Note 4: A range query and instant query that span the same time range yield the same results:

  • query_range: count_over_time() for 200..260 step=1m: 7102

  • query: count_over_time() for 200..260: 7102

  • query_range: count_over_time() for 200..800 step=10m: 118635

  • query: count_over_time() for 200..800: 118635

To Reproduce

Run a TraceQL metrics range query and instant query over the same time range with steps different than the time range, as described above.

Expected behavior

An instant query will produce the same result as a range query over the same time span.

Environment

  • Infrastructure: Grafana Cloud
{
    "version": "weekly-r198-ff5a82300",
    "revision": "ff5a82300",
    "branch": "weekly-r198",
    "buildUser": "",
    "buildDate": "",
    "goVersion": "go1.24.1"
}

Metadata

Metadata

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions