Dicedb Opentelemetry integration with prometheus scraping #1462

gsarmaonline · 2025-02-07T06:08:41Z

First PR for integrating prometheus scraping using Opentelemetry exporter.

DiceDB Observability with Opentelemetry (Otel)

This module implements the observability layer required for DiceDB to emit metrics.
The current implementation integrates Opentelemetry with Prometheus in scraping mode.
The current Prometheus exporter also exposes a HTTP client which Prometheus can leverage
to scrape data from.

Refer to the README.md file in the PR for more information.

gsarmaonline · 2025-02-07T06:08:54Z

cc @JyotinderSingh @arpitbbhayani

arpitbbhayani · 2025-02-08T11:52:43Z

internal/diceotel/README.md

+The observability module should be available from all other modules. There should always
+be one instance of the observability module running.
+To use any metric in the repo, first register it with the `diceotel` module [here](metrics.go).
+```go
+func (dotel *DiceOtel) register() (err error) {
+	DiceStartCounter, _ = dotel.Meter.Int64Counter("dicedb_start", api.WithDescription("A counter for the start of the DiceDB server"))
+	// Add other metrics here
+	return
+}


THis could just be a global variable that gets initialized during the bootup.

The metrics variable should be available globally but they should also be defined only after the DiceotelSrv object is defined.
I am thinking we can follow this approach:

Expose the DiceotelSrv globally to the dicedb package

Initialise the metrics inside the diceotel object and register them when the diceotel object is being initialised

This allows us to ensure that the list of global variables don't start piling up for metrics and can be accessed
through the DiceotelSrv global reference.

arpitbbhayani · 2025-02-08T11:53:11Z

internal/diceotel/README.md

+
+Access it anywhere in the repo and call the required functions:
+```go
+	diceotel.DiceStartCounter.Add(ctx, 1)


I am not keen on prefixing Dice for all metrics.

Yes, removing it.

arpitbbhayani · 2025-02-08T11:53:42Z

internal/diceotel/README.md

+be one instance of the observability module running.
+To use any metric in the repo, first register it with the `diceotel` module [here](metrics.go).
+```go
+func (dotel *DiceOtel) register() (err error) {


not a huge fan of classes and objects here given this can just be a global variable.

arpitbbhayani · 2025-02-08T11:54:41Z

internal/diceotel/diceotel.go

+type (
+	DiceOtel struct {
+		ctx context.Context
+
+		Meter api.Meter
+	}
+)
+
+var (
+	DiceotelSrv *DiceOtel
+)


this is typical java. I do not see a value add. If we need it, we can always add it later.

arpitbbhayani · 2025-02-08T11:55:18Z

internal/diceotel/diceotel.go

+	if err = dotel.setup(); err != nil {
+		return
+	}
+	if err = dotel.register(); err != nil {
+		return
+	}


why can't both of these be in one function, why have two unnecessarity.

Makes sense.

arpitbbhayani · 2025-02-08T11:56:05Z

internal/diceotel/diceotel.go

+	// The exporter embeds a default OpenTelemetry Reader and
+	// implements prometheus.Collector, allowing it to be used as
+	// both a Reader and Collector.
+	exporter, err := prometheus.New()


are we taking a strong dependency on prometheus here? given this is otel, this should be pluggable.

In case there is a requirement to move the metrics from one provider to another, changing the Exporters should be sufficient without having to changing the existing application code where the metrics are emitted.

So the prometheus exporter is pluggable and can be changed by adding another exporter type

arpitbbhayani · 2025-02-08T12:00:18Z

internal/server/ironhawk/iothread.go

@@ -42,10 +46,20 @@ func (t *IOThread) StartSync(ctx context.Context, shardManager *ShardManager, wa
 		_c.ClientID = t.ClientID
 		_c.Mode = t.Mode

+		execStartTime := time.Now()


I recommend we call this variable __telXXXTime, this way we can follow the same variable name through the codebase. Calling this execStartTime here and something else at another place will create confusion. Open for other shorter variable names.

We can also go for the convention that __telXXXTime is all things telemetry. So, any and every variable that starts with this can be easily identified as a telemetry variable.

Sounds good.

arpitbbhayani · 2025-02-08T12:02:44Z

internal/server/ironhawk/iothread.go

+		diceotel.DiceCmdLatencyInMsHistogram.Record(
+			ctx,
+			time.Since(execStartTime).Milliseconds(),
+			metric.WithAttributeSet(
+				attribute.NewSet(
+					attribute.String("cmd", c.Cmd),
+				),
+			),
+		)


this is a great UX as a database engineer. Liked it.

Concerns are only around the naming convention. Two things

We should also add a non-docker tutorial on seeing these metrics in action, checked in the repository.

why do we have histogram in the name? this seems pretty weird. The metric registering should be about what we are registering and not how it would be displayed or processed.

For point 1, yes, let me add the non-docker tutorial as well for this.

For point 2, defining a metric as a histogram also adds more percentile wise information as well.
For example, the above command is represented in the following buckets:

➜ dice git:(gsarma/opentel-prometheus) curl http://localhost:8050/metrics | grep latency % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 9966 0 9966 0 0 14.0M 0 --:--:-- --:--:-- --:--:-- 9732k # HELP dicedb_command_exec_latency_milliseconds # TYPE dicedb_command_exec_latency_milliseconds histogram dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="0"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="5"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="10"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="25"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="50"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="75"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="100"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="250"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="500"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="750"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="1000"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="2500"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="5000"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="7500"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="10000"} 1 dicedb_command_exec_latency_milliseconds_bucket{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version="",le="+Inf"} 1 dicedb_command_exec_latency_milliseconds_sum{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version=""} 0 dicedb_command_exec_latency_milliseconds_count{cmd="GET",otel_scope_name="dicedb-otel",otel_scope_version=""} 1

So we primarily have counter, gauge and histograms as different metric types.

gsarmaonline · 2025-02-11T03:37:39Z

@arpitbbhayani Thanks for the thorough review here.
I have tried to answer most of your comments. Please take a look whenever convenient.

gsarmaonline · 2025-02-16T14:12:09Z

@arpitbbhayani I have updated the PR with the discussed changes. PTAL. Thanks!

CLAassistant · 2025-03-20T11:37:33Z

All committers have signed the CLA.

gsarmaonline added 6 commits February 8, 2025 14:25

Dicedb Opentelemetry integration

c08fa2f

Added README for Diceotel

8afad5e

Added prometheus-grafana dockerfiles

4fa1a43

Fix readme link

1bee1c7

Opentel change port number

e198479

Updated README

c45d01b

gsarmaonline force-pushed the gsarma/opentel-prometheus branch from ebc770b to c45d01b Compare February 8, 2025 06:27

arpitbbhayani requested changes Feb 8, 2025

View reviewed changes

Addressing review comments

d572221

gsarmaonline added 2 commits February 16, 2025 22:09

Addressing review comments

a953540

Merge branch 'master' into gsarma/opentel-prometheus

73fa9bc

Uh oh!

Dicedb Opentelemetry integration with prometheus scraping #1462

Are you sure you want to change the base?

Dicedb Opentelemetry integration with prometheus scraping #1462

Uh oh!

Conversation

gsarmaonline commented Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DiceDB Observability with Opentelemetry (Otel)

Uh oh!

gsarmaonline commented Feb 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gsarmaonline commented Feb 11, 2025

Uh oh!

gsarmaonline commented Feb 16, 2025

Uh oh!

CLAassistant commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

gsarmaonline commented Feb 7, 2025 •

edited

Loading

CLAassistant commented Mar 20, 2025 •

edited

Loading