Support incoming Prometheus Alert Manager Alerts

[ShadowJonathan]

Is your feature request related to a problem? Please describe.

We're using a mix of Prometheus and Icinga2 at work: Icinga2 for status checks and service state, and Prometheus for metrics and metrics-based alerting.

We'd like to view Prometheus alerts in an accessible pane, and we see Icinga2 as a prime candidate, with how it is possible to preserve status checks, which appears to not entirely be trivial in Prometheus.

Describe the solution you'd like

Support receiving Prometheus Alert Manager style alerts via webhook, either natively, or via a pluggable component.

Display these alerts as service checks, with their associated hosts (e.g. the instance label on these alerts)

Alert Manager signals when alerts are resolved, though possibly also some polling on an interval would be a good backup, if the alert manager is overloaded, or icinga2 was down, during the period the resolution message was supposed to be sent.

Describe alternatives you've considered

signalio, which has this functionality (together with the dynamic hosts PR), but they haven't updated it for 2 years now, and appears to have dropped support for it.

Additional context

We've also considered https://alerta.io/ as an alerting viewing frontend, but the state and quality of purely open source tools in the alerting space (including that frontend) has been disappointing, somewhat, so we'd like to see if Prometheus + Icinga2 could be possible.

[oxzi]

Thanks for creating this issue. I transferred it to the Icinga Notifications issue tracker because I think it is better fit there. For context, Icinga Notifications is our approach to a dedicated alerting system, being only loosely tied to Icinga 2 and allowing other input sources as well.

In the past, I have demonstrated a PoC to integrate Prometheus Alertmanager into Icinga Notifications. Details are available in a (currently) unpublished blog post, still available in the Internet Archive. Reasons for unpublishing were changes in Icinga Notifications, rendering the examples from the blog post as useless.

Since Icinga Notifications is still in an early state, things are going to change, but the Prometheus Alertmanager or Prometheus itself are definitely on our radar.

[ShadowJonathan]

@oxzi one critical aspect to this, that I want to know, is that it is possible to visibly display all currently firing alerts, next to status checks, in the services column (or equivalent), would that be possible with Icinga Notifications?

[oxzi]

The Icinga Notifications project tries to uncouple itself from Icinga's strict host and service architecture, which works well when you can organize your checks into a "host and service" hierarchy, but gets frustrating if your setup does not fit into this model.

For example, some Prometheus alerts might fit into the model - i.e. when using the node_exporter -, but might also don't - i.e. for the domain_exporter checking whois information. Staying with this example, to which host do you assign the whois check?

So, to answer your question: There is an event overview, showing currently open incidents.

Please feel free to check the docs - still WIP at some points - or visit the demo instance.

[ShadowJonathan]

@oxzi I've taken a look at the demo instance; I had to adjust the max_attempts of some services for them to show up in the current incidents, but this looks exactly like what we want :)

I agree that some alerts don't fit a host -> services hierarchy, but is it still possible to (loosely) associate an alert with a particular host, or will that also stay decoupled?

---

And FWIW; we did encounter that PoC, and that's the reason why we initially started looking into this, as we're trying to unify our old nagios-based status checks environment, and a new initiative to (also) start monitoring things with prometheus-based metrics.

[oxzi]

I agree that some alerts don't fit a host -> services hierarchy, but is it still possible to (loosely) associate an alert with a particular host, or will that also stay decoupled?

At the moment, Icinga Notifications is independent of Icinga DB or the deprecated IDO backend. Thus, there is no linkage between Icinga 2 Host objects and host identifiers from other sources. But I like this idea and would suggest we look into this. Thanks for your suggestion.