Let sources affect incidents in multiple ways with a single event

[nilmerg]

We now want to allow events to have multiple effects. We need to prevent inconsistencies caused by an unreliable processing order. Therefore, effects should be atomic. For this reason, the type key is no longer used. Instead, alternative keys that can coexist in combination will be used. The following requirements must be met in this regard:

• Incidents should be able to be closed for multiple reasons. There are problems that cannot be resolved. An example of this are Director Deployments, which may cause an incident, but which is closed immediately after escalation.
• It should be possible to update incidents without triggering notifications. For example, a plugin’s output must be able to reflect the results of the last check, including in Icinga Notifications.
• In addition to the previous point, however, it should also be possible to explicitly trigger notifications even without changing the severity level. This is derived from so-called “volatile” hosts and services in Icinga.

The current distinction of events is therefore obsolete. The key type is replaced by the following ones:

• incident: true To open an incident or to trigger escalations
• close: true To close an incident irrespective of the severity
• muted: bool To mute/unmute an incident (Always in combination with muted_reason: string)
• force: true To bypass checks that may prevent notifications

Valid Combinations and their expected outcome

Any key marked with 🚫 cannot occur when any of the other keys are set or absent as described. The endpoint should respond with an error if they occur.

incident	close	muted	force	Outcome
true			🚫	An incident is opened or escalated
true	true	🚫	🚫	Escalates and closes an incident, no matter if just opened or already open
true	🚫	true	🚫	Escalates and mutes an incident, no matter if just opened or already open
true	🚫	false	🚫	Escalates and unmutes an incident, no matter if just opened or already open
	true	🚫	🚫	Closes an open incident prematurely, irrespective of the severity, no effect otherwise
	🚫	true	🚫	Mutes an open incident, notifications are sent, no effect otherwise
	🚫	false	🚫	Unmutes an open incident, notifications are sent, no effect otherwise
				An incident is updated with a new message, severity doesn't change, no notification is sent
			true	An incident is updated with a new message, severity doesn't change, notifications are sent

Legend:

What	Meaning
Empty cell	no value
🚫	not applicable

[julianbrost]

For this reason, the type key is no longer used. Instead, alternative keys that can coexist in combination will be used.

Note that this actually isn't much of a change of the current implementation, the primary mechanism is already a mute flag on the event, some type values are just automatically mapped to that flag:

icinga-notifications/internal/listener/listener.go

Lines 157 to 163 in 3025283

	if ev.Type == baseEv.TypeUnknown {
	ev.Type = baseEv.TypeState
	} else if !ev.Mute.Valid && ev.Type == baseEv.TypeMute {
	ev.SetMute(true, ev.MuteReason)
	} else if !ev.Mute.Valid && ev.Type == baseEv.TypeUnmute {
	ev.SetMute(false, ev.MuteReason)
	}

icinga-notifications/internal/event/event.go

Lines 99 to 103 in 3025283

	// SetMute alters the event mute and mute reason.
	func (e *Event) SetMute(muted bool, reason string) {
	e.Mute = types.Bool{Valid: true, Bool: muted}
	e.MuteReason = reason
	}

So it's basically removing that type handling and adding more flags like the already existing mute (and possibly renaming that one).

force: true To bypass checks that may prevent escalations or notifications

That's not supposed to mean that escalation conditions are to be bypassed, primarily it's about the following check that should be possible to be bypassed:

icinga-notifications/internal/incident/incident.go

Lines 300 to 303 in 3025283

	if oldSeverity == newSeverity {
	err := fmt.Errorf("%w: %s state event from source %d", event.ErrSuperfluousStateChange, ev.Severity.String(), ev.SourceId)
	return err
	}

Valid Combinations and their expected outcome

I'm not entirely sure how to read the table. What exactly do empty cells and 🚫 mean? Is it intentional that in one row, all first four columns are empty?

[nilmerg]

That's not supposed to mean that escalation conditions are to be bypassed

Sure, the table already didn't mention this. Removed.

I'm not entirely sure how to read the table. What exactly do empty cells and 🚫 mean?

It means, not applicable. I expected this to be clear, given the table is about combinations and that e.g. closing an incident and muting it is nonsense.

[julianbrost]

I'm not entirely sure how to read the table. What exactly do empty cells and 🚫 mean?

It means, not applicable. I expected this to be clear, given the table is about combinations and that e.g. closing an incident and muting it is nonsense.

I'm sorry, but I still don't get it. I was hoping for an answer like "empty cell means [please describe], 🚫 means [please describe]".

My initial guess would have been that empty cell means that the flag may have any value and 🚫 means that it must not be set. But that doesn't work out. The second to last row would imply that every combination is valid, conflicting with rows containing 🚫.

[nilmerg]

Sorry, forgot to mention empty cell meaning, but any value? In what world? 😢 Anyway…

Empty cell == no value
🚫 == not applicable

[julianbrost]

Not applicable as in don't care? So 🚫 (forbidden sign) does not mean that it's forbidden to set the value (if yes, that's the root of my confusion), but 🚫 = any of true, false, or not presend?

[nilmerg]

We're both confused. I'd recommend you to take a step back and look at my table with an open mindset. I can't fathom right now, that you're really asking me whether 🚫 may mean not present while I showed you that an empty cell means exactly that.

[julianbrost]

Can you please just make a clear statement what 🚫 is supposed to mean? I'm interpreting not applicable as this input is irrelevant in that case. So that would include all possible values and not being set at all.

So what does mean now? My next guess: 🚫 = must be either true or false?

[nilmerg]

Disclaimer: Generated using claude.ai

Prompt:

Please generate json examples that reflect what is described in the table in the following github issue: #407

Result:

// Row 1: incident=true (no close, no muted, no force)
// Outcome: An incident is opened or escalated
{
"name": "disk-usage-high",
"severity": "crit",
"message": "Disk usage at 95% on /dev/sda1",
"incident": true
}

// ─────────────────────────────────────────────────────────────

// Row 2: incident=true, close=true (no muted, no force)
// Outcome: Escalates and closes an incident, no matter if just opened or already open
{
"name": "director-deployment-failed",
"severity": "crit",
"message": "Deployment #42 failed, closing immediately after escalation",
"incident": true,
"close": true
}

// ─────────────────────────────────────────────────────────────

// Row 3: incident=true, muted=true (no close, no force)
// Outcome: Escalates and mutes an incident, no matter if just opened or already open
{
"name": "backup-job-running",
"severity": "warn",
"message": "Backup job causing elevated I/O — muting notifications during window",
"incident": true,
"muted": true,
"muted_reason": "Scheduled maintenance window for nightly backup"
}

// ─────────────────────────────────────────────────────────────

// Row 4: incident=true, muted=false (no close, no force)
// Outcome: Escalates and unmutes an incident, no matter if just opened or already open
{
"name": "backup-job-complete",
"severity": "warn",
"message": "Backup job still elevated post-window — unmuting and escalating",
"incident": true,
"muted": false,
"muted_reason": "Maintenance window ended, issue persists"
}

// ─────────────────────────────────────────────────────────────

// Row 5: close=true only (no incident, no muted, no force)
// Outcome: Closes an open incident prematurely, irrespective of severity; no effect otherwise
{
"name": "disk-usage-high",
"severity": "ok",
"message": "Disk usage recovered to 60% after cleanup",
"close": true
}

// ─────────────────────────────────────────────────────────────

// Row 6: muted=true only (no incident, no close, no force)
// Outcome: Mutes an open incident, notifications are sent; no effect otherwise
{
"name": "cpu-usage-high",
"severity": "warn",
"message": "CPU spike expected during batch job execution",
"muted": true,
"muted_reason": "Planned batch job running between 02:00–04:00 UTC"
}

// ─────────────────────────────────────────────────────────────

// Row 7: muted=false only (no incident, no close, no force)
// Outcome: Unmutes an open incident, notifications are sent; no effect otherwise
{
"name": "cpu-usage-high",
"severity": "warn",
"message": "Batch job finished, CPU still elevated — unmuting",
"muted": false,
"muted_reason": "Batch job completed, issue is no longer expected"
}

// ─────────────────────────────────────────────────────────────

// Row 8: no keys set (no incident, no close, no muted, no force)
// Outcome: Incident updated with a new message, severity unchanged, no notification sent
{
"name": "disk-usage-high",
"severity": "crit",
"message": "Check re-run: disk usage still at 95%, no change in state"
}

// ─────────────────────────────────────────────────────────────

// Row 9: force=true only (no incident, no close, no muted)
// Outcome: Incident updated with a new message, severity unchanged, notifications ARE sent
// (models "volatile" hosts/services in Icinga — notify on every check regardless of state change)
{
"name": "ssl-cert-expiry",
"severity": "warn",
"message": "SSL certificate expires in 14 days — re-notifying contacts",
"force": true
}

[julianbrost]

Unless I've missed something, both empty cell and 🚫 result in the key not being present in the JSON. So unless you want to say that it's actually the same, that doesn't really help.

[oxzi]

I am experiencing the same confusion as @julianbrost. And this does not only root in the JSON, but the valid combinations¹ from the table above.

In the table, there are no two rows identical but only differing between a single no value and not applicable. For example, what would differ when changing not applicable to no value in the first row?

Furthermore, not all types have both boolean characteristics defined. For example, incident is only defined as true or not applicable. What would a false value represent? I am similar puzzled about force, which is only defined for true and both non-boolean no-values.

---

Based on the table, all types except muted are flags, not boolean values. Did I understood this right?

Could you please show in pseudo code how these types should be parsed? I hope reducing text to a formalized language would increase common understanding.

Footnotes

1. As the table is labeled as "valid combinations", I would expect that every other combination is invalid by definition. ↩

[nilmerg]

Clarified once again the meaning in the OP:

Any key marked with 🚫 cannot occur when any of the other keys are set or absent as described. The endpoint should respond with an error if they occur.

Could you please show in pseudo code how these types should be parsed?

Of course, using claude.ai:

FUNCTION processEvent(event):
    // Extract keys from event (all default to absent/null if not provided)
    incident   = event.get("incident")   // true | absent
    close      = event.get("close")      // true | absent
    muted      = event.get("muted")      // true | false | absent
    force      = event.get("force")      // true | absent

    // --- Validate combinations ---

    // `force` is never applicable when `incident` is present
    IF incident == true AND force == true:
        RETURN error("'force' is not applicable when 'incident' is set")

    // `muted` is never applicable when both `incident` and `close` are true
    IF incident == true AND close == true AND muted is NOT absent:
        RETURN error("'muted' is not applicable when both 'incident' and 'close' are set")

    // `force` is not applicable when `close` is present
    IF close == true AND force == true:
        RETURN error("'force' is not applicable when 'close' is set")

    // `force` is not applicable when `muted` is present
    IF muted is NOT absent AND force == true:
        RETURN error("'force' is not applicable when 'muted' is set")

    // --- Process combinations ---

    // Row 1: incident=true, close=absent, muted=absent, force=N/A
    IF incident == true AND close is absent AND muted is absent:
        openOrEscalateIncident(event)
        RETURN

    // Row 2: incident=true, close=true, muted=N/A, force=N/A
    IF incident == true AND close == true:
        openOrEscalateIncident(event)
        closeIncident(event)
        RETURN

    // Row 3: incident=true, close=absent, muted=true, force=N/A
    IF incident == true AND close is absent AND muted == true:
        openOrEscalateIncident(event)
        muteIncident(event)
        RETURN

    // Row 4: incident=true, close=absent, muted=false, force=N/A
    IF incident == true AND close is absent AND muted == false:
        openOrEscalateIncident(event)
        unmuteIncident(event)
        RETURN

    // Row 5: incident=absent, close=true, muted=N/A, force=N/A
    IF incident is absent AND close == true:
        IF incidentIsOpen():
            closeIncident(event)  // premature close, regardless of severity
        ELSE:
            noEffect()
        RETURN

    // Row 6: incident=absent, close=absent, muted=true, force=N/A
    IF incident is absent AND close is absent AND muted == true:
        IF incidentIsOpen():
            muteIncident(event)
            sendNotifications(event)
        ELSE:
            noEffect()
        RETURN

    // Row 7: incident=absent, close=absent, muted=false, force=N/A
    IF incident is absent AND close is absent AND muted == false:
        IF incidentIsOpen():
            unmuteIncident(event)
            sendNotifications(event)
        ELSE:
            noEffect()
        RETURN

    // Row 8: all absent (plain update, no notification)
    IF incident is absent AND close is absent AND muted is absent AND force is absent:
        updateIncidentMessage(event)
        // severity unchanged, no notification sent
        RETURN

    // Row 9: force=true only (plain update WITH notification)
    IF incident is absent AND close is absent AND muted is absent AND force == true:
        updateIncidentMessage(event)
        // severity unchanged
        sendNotifications(event)
        RETURN

    // Fallback for any unhandled/invalid combination
    RETURN error("Invalid combination of event keys")