Send only watchdog resolved notification

[DFurt]

I have alertmanager set in a way that notifications are sent to pagerduty and then to the respective channels.

In order to not have a watchdog alert and incident in pagerduty, I've set the receiver to null. But this way we also don't get the resolved notification that indicates there's an issue.

What I would like to know is if there is a way to have the triggered notification silenced/not create an incident but receive the notification when it's resolved, meaning there is an issue.

An alternative route would be to invert watchdog alert, having it in resolved state instead of triggered if everything is fine and vice-versa.

Is this possible?

[grobinson-grafana]

I think this might be what you want https://training.promlabs.com/training/monitoring-and-debugging-prometheus/metrics-based-meta-monitoring/end-to-end-watchdog-alerts/?

[DFurt]

hello @grobinson-grafana , the watchdog alert is already set up but is always firing, thus creating an incident in pagerduty, in this case. What I wanted to do was to not have the incident notification sent to pagerduty, or other tool like slack, so there isn't an incident displaying. But to be notified, trough pagerduty or other tool, of alertmanager resolving the watchdog alert in case it happens.

[grobinson-grafana]

That sounds correct to me. I understand you need to set up a Dead Mans Snitch in Pagerduty, and then Pagerduty will create an incident if it stops receiving the alert from Alertmanager (https://www.pagerduty.com/docs/guides/dead-mans-snitch-integration-guide/) (mentioned at the bottom of the link I shared earlier).

[DFurt]

Yes, I would like something like that but without relying on a pagerduty integration as we might set this up for slack aswell directly, without pagerduty integrations

[grobinson-grafana]

I think it might be possible with inhibition rules, but the reason to do this with a service like Pagerduty is that it can also tell you if your Alertmanager is down, otherwise it could be down and you might not know.

[DFurt]

I'm not being able to find a solution using inhibition rules. My understanding reading this is that I can't have the alert inhibit itself but would have to use a different alert to do it. However, Watchdog is the alert that is always firing, while others only do so on certain conditions as intended.

Do you have any ideia of how this could be set up?

[grobinson-grafana]

Ah I didn't consider the case where no alerts are firing. In that case I don't think its possible and you'll have to use something like Pagerduty's Dead Mans Snitch. It's still not clear to me what it is your watchdog alert is doing, but in the case your Alertmanager is unavailable you won't get a notification in Slack, regardless of whether the watchdog alert fired or resolved. That's why, if your Watchdog alert is making sure your alerting system is fully working end-to-end, you'll want to set this up in an IRM like Pagerduty.