Remove metrics from compat package (#3714)

This commit removes the metrics from the compat package
in favour of the existing logging and the additional tools
at hand, such as amtool, to validate Alertmanager configurations.

Due to the global nature of the compat package, a consequence
of config.Load, these metrics have proven to be less useful
in practice than expected, both in Alertmanager and other projects
such as Mimir.

There are a number of reasons for this:

1. Because the compat package is global, these metrics cannot be
   reset each time config.Load is called, as in multi-tenant
   projects like Mimir loading a config for one tenant would reset
   the metrics for all tenants. This is also the reason the metrics
   are counters and not gauges.

2. Since the metrics are counters, it is difficult to create
   meaningful dashboards for Alertmanager as, unlike in Mimir,
   configurations are not reloaded at fixed intervals, and as such,
   operators cannot use rate to track configuration changes
   over time.

In Alertmanager, there are much better tools available to validate
that an Alertmanager configuration is compatible with the UTF-8
parser, including both the existing logging from Alertmanager
server and amtool check-config.

In other projects like Mimir, we can track configurations for
individual tenants using log aggregation and storage systems
such as Loki. This gives operators far more information than
what is possible with the metrics, including the timestamp,
input and ID of tenant configurations that are incompatible
or have disagreement.

Signed-off-by: George Robinson <[email protected]>
Signed-off-by: Gokhan Sari <[email protected]>

Author: grobinson-grafana

cli/root.go

@@ -61,7 +61,7 @@ func initMatchersCompat(_ *kingpin.ParseContext) error {
 	if err != nil {
 		kingpin.Fatalf("error parsing the feature flag list: %v\n", err)
 	}
-	compat.InitFromFlags(logger, compat.NewMetrics(nil), featureConfig)
+	compat.InitFromFlags(logger, featureConfig)
 	return nil
 }
 

cmd/alertmanager/main.go

@@ -189,7 +189,7 @@ func run() int {
 		level.Error(logger).Log("msg", "error parsing the feature flag list", "err", err)
 		return 1
 	}
-	compat.InitFromFlags(logger, compat.RegisteredMetrics, ff)
+	compat.InitFromFlags(logger, ff)
 
 	err = os.MkdirAll(*dataDir, 0o777)
 	if err != nil {