alertmanager <> ruler integration not working in multi-tenancy mode

[sbaier1]

Describe the bug

it seems like the alertmanager_url configured in the ruler config within the template is incorrect:

alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.{{ template "mimir.fullname" . }}-alertmanager-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}/alertmanager

alertmanager=http://<cluster-ip>.mimir-alertmanager-headless.monitoring.svc.cluster.local.:8080/alertmanager/api/v2/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found"

To Reproduce

Steps to reproduce the behavior:

1. Deploy mimir-distributed chart with alertmanager + ruler in multi tenant mode
2. Deploy a rule that will fire immediately
3. ruler will print the above error with the default configuration template, as it cannot invoke an alert on alertmanager's metrics endpoint.

Expected behavior

i'd expect the mimir chart to properly connect the ruler deployment to the alertmanager instance(s) out of the box

not 100% sure what the correct template would be. i'm also not sure why ruler is querying api/v2/alerts when the mimir docs state the correct path is api/v1/alerts https://grafana.com/docs/mimir/latest/operators-guide/reference-http-api/

Environment

• Infrastructure: Kubernetes
• Deployment tool: helm

Additional Context

from my preliminary tests, when i forward to alertmanager and query the API using postman, POSTing to /api/v1/alerts with an OrgID works and is properly sent to the upstream alertmanager api, but POSTing to /api/v2/alerts (which is what ruler does) doesn't work and returns a 404 error.

[sbaier1]

from my initial look at the code it would seem like in the multi-tenant configuration, this simply cannot work:

• ruler seems to statically use alertmanager V2 API https://github.com/grafana/mimir/blob/mimir-2.1.0/pkg/ruler/notifier.go#L130
• alertmanager in multitenant mode seems to only expose the V1 APIs https://github.com/grafana/mimir/blob/mimir-2.1.0/pkg/api/api.go#L208-L210

[sbaier1]

alright, i figured this out now:

• ruler's error err="bad response status 404 Not Found" is a bit confusing, it intuitively led me to believe that something's wrong with service routing. this was not the case. the actual response body was what gave me the hint: the Alertmanager is not configured (this isn't printed anywhere in the logs though, i figured it out by running some queries against alertmanager with postman)
• i hadn't created any configs in alertmanager because i didn't know how mimir's multi-tenant alertmanager works yet
• i "ported" add rules and alertmanager sidecar to cortex-helm-chart cortexproject/cortex-helm-chart#150 to the mimir chart and added the alertmanager configs via configmap injection
• alertmanager instantiates the alertmanagers and ruler can push alerts

so in the end it was a user error 😓

not 100% sure what the correct template would be. i'm also not sure why ruler is querying api/v2/alerts when the mimir docs state the correct path is api/v1/alerts https://grafana.com/docs/mimir/latest/operators-guide/reference-http-api/

Looks like documentation is not correct. We moved Alertmanager to API v2 some time ago.

it's more like it's incomplete:
the v1 API is for POSTing alertmanager configurations, the prom alertmanager APIs are under alertmanager_prefix/api/v2/alerts etc., which isn't listed or explained anywhere, which definitely reinforced my belief that something was wrong with service routing.

[pracucci]

from my initial look at the code it would seem like in the multi-tenant configuration, this simply cannot work:

• ruler seems to statically use alertmanager V2 API https://github.com/grafana/mimir/blob/mimir-2.1.0/pkg/ruler/notifier.go#L130
• alertmanager in multitenant mode seems to only expose the V1 APIs https://github.com/grafana/mimir/blob/mimir-2.1.0/pkg/api/api.go#L208-L210

The ruler and alertmanager APIs are a bit complex. There are 2 different set of APIs involved:

• APIs exposed by Mimir ruler and Alertmanager used to configure them
• APIs exposed by Mimir Alertmanager, which are inherited by Prometheus Alertmanager, used to query alerts

APIs exposed by Mimir ruler and Alertmanager used to configure them

These are the v1 APIs exposed here:
https://github.com/grafana/mimir/blob/mimir-2.1.0/pkg/api/api.go#L208-L210

And here:

mimir/pkg/api/api.go

Lines 307 to 333 in 840c686

	if configAPIEnabled {
	// Ruler API Routes
	// TODO remove the /api/v1/rules/** endpoints in Mimir 2.2.0 as agreed in https://github.com/grafana/mimir/pull/763#discussion_r808270581
	a.RegisterDeprecatedRoute("/api/v1/rules", http.HandlerFunc(r.ListRules), true, true, "GET")
	a.RegisterDeprecatedRoute("/api/v1/rules/{namespace}", http.HandlerFunc(r.ListRules), true, true, "GET")
	a.RegisterDeprecatedRoute("/api/v1/rules/{namespace}/{groupName}", http.HandlerFunc(r.GetRuleGroup), true, true, "GET")
	a.RegisterDeprecatedRoute("/api/v1/rules/{namespace}", http.HandlerFunc(r.CreateRuleGroup), true, true, "POST")
	a.RegisterDeprecatedRoute("/api/v1/rules/{namespace}/{groupName}", http.HandlerFunc(r.DeleteRuleGroup), true, true, "DELETE")
	a.RegisterDeprecatedRoute("/api/v1/rules/{namespace}", http.HandlerFunc(r.DeleteNamespace), true, true, "DELETE")
	// Configuration endpoints with Prometheus prefix, so we keep Prometheus-compatible EPs and config EPs under the same prefix.
	// TODO remove the <prometheus-http-prefix>/v1/rules/** endpoints in Mimir 2.2.0 as agreed in https://github.com/grafana/mimir/pull/1222#issuecomment-1046759965
	a.RegisterDeprecatedRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/rules"), http.HandlerFunc(r.ListRules), true, true, "GET")
	a.RegisterDeprecatedRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/rules/{namespace}"), http.HandlerFunc(r.ListRules), true, true, "GET")
	a.RegisterDeprecatedRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/rules/{namespace}/{groupName}"), http.HandlerFunc(r.GetRuleGroup), true, true, "GET")
	a.RegisterDeprecatedRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/rules/{namespace}"), http.HandlerFunc(r.CreateRuleGroup), true, true, "POST")
	a.RegisterDeprecatedRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/rules/{namespace}/{groupName}"), http.HandlerFunc(r.DeleteRuleGroup), true, true, "DELETE")
	a.RegisterDeprecatedRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/rules/{namespace}"), http.HandlerFunc(r.DeleteNamespace), true, true, "DELETE")
	// Long-term maintained configuration API routes
	a.RegisterRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/config/v1/rules"), http.HandlerFunc(r.ListRules), true, true, "GET")
	a.RegisterRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/config/v1/rules/{namespace}"), http.HandlerFunc(r.ListRules), true, true, "GET")
	a.RegisterRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/config/v1/rules/{namespace}/{groupName}"), http.HandlerFunc(r.GetRuleGroup), true, true, "GET")
	a.RegisterRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/config/v1/rules/{namespace}"), http.HandlerFunc(r.CreateRuleGroup), true, true, "POST")
	a.RegisterRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/config/v1/rules/{namespace}/{groupName}"), http.HandlerFunc(r.DeleteRuleGroup), true, true, "DELETE")
	a.RegisterRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/config/v1/rules/{namespace}"), http.HandlerFunc(r.DeleteNamespace), true, true, "DELETE")
	}

For example, these APIs are used by mimirtool to let you upload the ruler and/or alertmanager YAML configuration to Mimir.

The alertmanager config API is exposed at /api/v1/alerts (with no /alertmanager prefix).

APIs exposed by Mimir Alertmanager, which are inherited by Prometheus Alertmanager, used to query alerts

These are the APIs used by the ruler, and they're the API v2. They're exposed under the /alertmanager prefix, so for example /alertmanager/api/v2/alerts. These APIs are registered here:

mimir/pkg/api/api.go

Line 203 in 840c686

a.RegisterRoutesWithPrefix(a.cfg.AlertmanagerHTTPPrefix, am, true, true)

from my preliminary tests, when i forward to alertmanager and query the API using postman, POSTing to /api/v1/alerts with an OrgID works and is properly sent to the upstream alertmanager api, but POSTing to /api/v2/alerts (which is what ruler does) doesn't work and returns a 404 error.

If the test you're doing is about POSTing an alert notification, then try to send it to /alertmanager/api/v2/alerts. Does it work?

alertmanager=http://.mimir-alertmanager-headless.monitoring.svc.cluster.local.:8080/alertmanager/api/v2/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found"

Can you confirm the computed hostname is correct and alertmanager respond at that host, please?

[pracucci]

so in the end it was a user error

I don't think it was! Mimir should be improved too.

ruler's error err="bad response status 404 Not Found" is a bit confusing, it intuitively led me to believe that something's wrong with service routing. this was not the case. the actual response body was what gave me the hint: the Alertmanager is not configured (this isn't printed anywhere in the logs though, i figured it out by running some queries against alertmanager with postman)

I'm going to open an issue about it. We need to fix the logs, to make it more clear if the problem is that there's no alertmanager configured for that tenant.

Thanks for raising this good issue!

[sbaier1]

If the test you're doing is about POSTing an alert notification, then try to send it to /alertmanager/api/v2/alerts. Does it work?

yes, this works as expected. i tried POSTing on the wrong path because of my misconceptions about how the API paths work

alertmanager=http://.mimir-alertmanager-headless.monitoring.svc.cluster.local.:8080/alertmanager/api/v2/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found"
Can you confirm the computed hostname is correct and alertmanager respond at that host, please?

yes everything is working with all default settings from the helm chart now. all i had to do was configure the alertmanagers for my tenants.