Replace telegraf prometheus plugin

[dricross]

Description of the issue

The CloudWatch Agent does not support pushing Prometheus exponential histograms to CloudWatch Logs via EMF. Any exponential histograms are simply dropped by the agent. This is because the telegraf-based prometheus receiver doesn't support them. The OpenTelemetry one does support exponential histograms, so we are working on migrating all prometheus receiver uses to the OpenTelemetry plugin.

Description of changes

Staging some changes into a non-personal feature branch for replacing the telegraf-based Prometheus receiver with the OpenTelemetry-based Prometheus receciver.

• Create prometheus adapter processor
• [Translator] Update Prometheus -> EMF pipeline to use OpenTelemetry plugin + new prometheus adapter receiver. Remove TOML translation for prometheus receiver

The prometheus adapter processor will be used to translate the metrics emitted by the OpenTelemetry prometheus receiver to match what the Telegraf prometheus receiver would have output to maintain backwards compatibility.

This change "breaks" the current Prometheus -> EMF pipeline in the sense that the metrics do not show up as they used to, so these changes are not yet ready to merge to main.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Run Prometheus server to vend prometheus metrics. Configure agent to collect prometheus metrics and push to CW Logs. Metrics show up in CloudWatch, e.g.

{
  "agent": {
    "metrics_collection_interval": 10,
    "run_as_user": "root",
    "debug": true,
    "logfile": ""
  },
  "logs": {
    "metrics_collected": {
      "prometheus": {
        "prometheus_config_path": "/home/ec2-user/prometheus/prometheus.yaml",
        "log_group_name": "prometheusadapter",
        "emf_processor": {
          "metric_namespace": "prometheusadapter",
          "metric_declaration": [
            {
              "source_labels": [
                "include"
              ],
              "label_matcher": "^yes$",
              "dimensions": [
                [
                  "prom_type"
                ],
                [
                  "prom_type",
                  "quantile"
                ]
              ],
              "metric_selectors": [
                "^prometheus_test*"
              ]
            }
          ]
        }
      }
    },
    "force_flush_interval": 5
  }
}

Requirements

Before commit the code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

[jefchien]

nit: I'm guessing this was copied from somewhere. Consider removing it for now.

[dricross]

yeah, copied from somewhere else. I'll remove the comments

[jefchien]

nit: Is there a reason we want to embed the Config?

[dricross]

No particular reason. Following convention from other processors (kueueattributes, gpuattributes, ec2tagger)

[jefchien]

nit: typo

Suggested change

	type prometheusAdapaterProcessor struct {
	type prometheusAdapterProcessor struct {

[jefchien]

nit: Looks like this was copied from the kueueattributes processor?

[dricross]

yeah it was. I'll fix the name

[jefchien]

We'll want to add a name to this receiver (e.g. prometheus/cloudwatchlogs) so we can differentiate it from the one configured in the metrics section. These can and should be distinct prometheus receivers unless we find a way to dedup them.

[dricross]

gotcha, thanks. will update