Heartbeat recovery does not send resolved notifications

[Artesfrost]

Fix missing resolved notifications for recovered heartbeats

Summary

During testing of the Heartbeats functionality in IncidentRelay, a notification gap was identified:

• heartbeat overdue alerts were created correctly;
• the initial notification was sent;
• escalation and matcher-based channel selection worked;
• the alert was resolved when the heartbeat recovered and Auto Resolve was enabled;
• however, no resolved notification was sent to the configured notification channels.

The standard alert lifecycle already sends resolved notifications through the notification-policy pipeline. The heartbeat recovery flow resolved the alert but did not call that pipeline.

---

Target use case

1. A heartbeat ping is received by IncidentRelay.
2. The heartbeat stops sending pings.
3. After the configured interval and grace period:
   • the heartbeat becomes overdue;
   • an alert is created;
   • the initial notification is sent;
   • escalation continues according to the escalation policy;
   • matcher-based notification rules select the required Teams channel.
4. The heartbeat starts sending pings again.
5. When Auto Resolve is enabled:
   • the heartbeat returns to ok;
   • the related alert is resolved;
   • a resolved notification is sent to the matching notification channels.
6. When Auto Resolve is disabled:
   • the heartbeat returns to ok;
   • the related alert remains open;
   • no resolved notification is sent.

Example heartbeat labels:

{
  "Application": "DB",
  "source_type": "heartbeat"
}

---

Changed files

One application file was modified:

app/services/heartbeats/service.py

In addition, the existing notification-policy rule configuration was updated to include the resolved event type.

---

Change 1: send resolved notifications after heartbeat recovery

Modified file

app/services/heartbeats/service.py

Affected functions

_resolve_current_overdue_alert()
_resolve_current_instance_overdue_alert()

Previous behavior

The heartbeat recovery flow called:

resolved = resolve_alert(group.id, user_id=None)

This correctly:

• changed the alert group status to resolved;
• created the standard resolved timeline event;
• refreshed service and business-impact state.

After that, the heartbeat-specific recovery event was created:

alerts_repo.create_alert_event(
    group_id=resolved.id,
    event_type="heartbeat_recovered",
    message=f"Heartbeat recovered: {heartbeat.name}",
)

However, the resolved notification pipeline was not called.

As a result, the alert was resolved in IncidentRelay, but Microsoft Teams did not receive a recovery/resolved message.

Standard lifecycle behavior

The standard incoming-alert lifecycle in:

app/services/alerts/lifecycle.py

sends resolved notifications using:

if group.last_notification_at:
    notify_alert(
        group,
        event_type="resolved",
    )

The heartbeat recovery flow did not use the same logic.

Updated behavior

The heartbeat recovery functions now call the existing notification delivery pipeline after resolving the alert:

resolved = resolve_alert(group.id, user_id=None)

if resolved.last_notification_at:
    notify_alert(
        resolved,
        event_type="resolved",
    )

The heartbeat-specific recovery event is then recorded as before.

---

Required import

Modified file

app/services/heartbeats/service.py

The following import was added:

 from app.services.integrations.auth import create_raw_token, hash_token
+from app.services.notifications.delivery import notify_alert

---

Change 2: resolved notification for a regular heartbeat

Modified function

_resolve_current_overdue_alert()

Previous code

resolved = resolve_alert(group.id, user_id=None)

alerts_repo.create_alert_event(
    group_id=resolved.id,
    event_type="heartbeat_recovered",
    message=f"Heartbeat recovered: {heartbeat.name}",
)

return resolved

Updated code

resolved = resolve_alert(group.id, user_id=None)

if resolved.last_notification_at:
    notify_alert(
        resolved,
        event_type="resolved",
    )

alerts_repo.create_alert_event(
    group_id=resolved.id,
    event_type="heartbeat_recovered",
    message=f"Heartbeat recovered: {heartbeat.name}",
)

return resolved

Conceptual diff — app/services/heartbeats/service.py

 resolved = resolve_alert(group.id, user_id=None)

+if resolved.last_notification_at:
+    notify_alert(
+        resolved,
+        event_type="resolved",
+    )
+
 alerts_repo.create_alert_event(
     group_id=resolved.id,
     event_type="heartbeat_recovered",
     message=f"Heartbeat recovered: {heartbeat.name}",
 )

---

Change 3: resolved notification for a heartbeat instance

Modified function

_resolve_current_instance_overdue_alert()

Previous code

resolved = resolve_alert(group.id, user_id=None)

alerts_repo.create_alert_event(
    group_id=resolved.id,
    event_type="heartbeat_instance_recovered",
    message=(
        f"Heartbeat recovered: "
        f"{instance.heartbeat.name} / {instance.instance_key}"
    ),
)

return resolved

Updated code

resolved = resolve_alert(group.id, user_id=None)

if resolved.last_notification_at:
    notify_alert(
        resolved,
        event_type="resolved",
    )

alerts_repo.create_alert_event(
    group_id=resolved.id,
    event_type="heartbeat_instance_recovered",
    message=(
        f"Heartbeat recovered: "
        f"{instance.heartbeat.name} / {instance.instance_key}"
    ),
)

return resolved

Conceptual diff — app/services/heartbeats/service.py

 resolved = resolve_alert(group.id, user_id=None)

+if resolved.last_notification_at:
+    notify_alert(
+        resolved,
+        event_type="resolved",
+    )
+
 alerts_repo.create_alert_event(
     group_id=resolved.id,
     event_type="heartbeat_instance_recovered",
     message=(
         f"Heartbeat recovered: "
         f"{instance.heartbeat.name} / {instance.instance_key}"
     ),
 )

---

Why last_notification_at is checked

The standard IncidentRelay lifecycle sends a resolved notification only when the alert group had already produced an outbound notification.

The same guard was retained:

if resolved.last_notification_at:

This avoids sending a resolved message for an alert that never had an initial notification.

It also keeps the heartbeat recovery behavior consistent with:

app/services/alerts/lifecycle.py

---

Notification Policy configuration

The notification-policy rules used by the heartbeat service also need to allow the resolved event type.

Example event-type configuration:

[
  "notification",
  "reminder",
  "resolved"
]

For the Database Layer rule:

[
  "escalation",
  "resolved"
]

The exact event-type assignment depends on which channels should receive recovery notifications.

No direct Teams webhook call was added. Resolved notifications continue to pass through:

Heartbeat recovery
→ resolve_alert()
→ notify_alert(event_type="resolved")
→ Notification Policy resolver
→ matcher evaluation
→ configured notification channel

---

Expected result

When Auto Resolve is enabled:

• the heartbeat returns to ok;
• the related alert is resolved;
• the standard resolved event is created;
• the matching notification-policy rule is selected;
• the configured Teams channel receives a resolved notification;
• the heartbeat-specific heartbeat_recovered event is recorded.

When Auto Resolve is disabled:

• the heartbeat returns to ok;
• the existing alert remains open;
• no resolved notification is sent.

---

Validation

Test configuration

Heartbeat interval: 60 seconds
Grace period: 30 seconds
Severity: critical
Priority: p1
Application label: DB
Notification channel: Microsoft Teams
Notification mode: service notification policy

Auto Resolve enabled

The end-to-end test completed successfully:

Heartbeat stops sending pings
→ heartbeat becomes overdue
→ alert is created
→ initial Duty Engineer notification
→ escalation after configured delay
→ Database Layer rotation
→ Database on-call engineer assignment
→ dedicated Database Teams notification
→ heartbeat ping is received
→ heartbeat status changes to ok
→ alert is resolved
→ resolved Teams notification is sent

Expected timeline events were observed:

heartbeat_overdue
notification_sent
escalated
notification_sent
resolved
heartbeat_recovered

The notification delivery also created the corresponding Teams delivery event.

Auto Resolve disabled

The second scenario also completed successfully:

Heartbeat stops sending pings
→ heartbeat becomes overdue
→ alert is created
→ notifications and escalation work
→ heartbeat ping is received
→ heartbeat status changes to ok
→ alert remains open
→ no resolved notification is sent

This behavior matches the Auto Resolve setting.

---

Request for review

Please review and confirm:

1. Heartbeat recovery should use the same resolved-notification pipeline as the standard incoming-alert lifecycle.

2. The update in:

   app/services/heartbeats/service.py
   

   is consistent with the intended heartbeat lifecycle.

3. Checking last_notification_at is the correct condition before sending a resolved notification.

4. Both regular heartbeats and heartbeat instances should use the same behavior.

5. Notification-policy rules should explicitly include the resolved event type for channels that must receive recovery messages.

6. Automated tests should cover:

   • heartbeat recovery with Auto Resolve enabled;
   • heartbeat recovery with Auto Resolve disabled;
   • recovery after the initial notification;
   • recovery before any notification was sent;
   • matcher-based resolved channel selection;
   • heartbeat-instance recovery;
   • prevention of duplicate resolved notifications.

---

Deployment details

The solution is deployed on-premises on Oracle Linux 9.

Architecture

Heartbeat client
   │
   │ HTTP ping using a heartbeat token
   ▼
Nginx
   │
   ▼
Gunicorn :8081
   │
   ▼
IncidentRelay Flask application
   │
   ├── PostgreSQL
   ├── Microsoft Teams via Power Automate webhook
   └── IncidentRelay scheduler worker

Components

Application version directory:
/var/www/incidentrelay-1.2.1

Active application path:
/var/www/incidentrelay

Modified application file:
/var/www/incidentrelay/app/services/heartbeats/service.py

Python virtual environment:
/var/www/incidentrelay/venv

Python executable:
/var/www/incidentrelay/venv/bin/python

Configuration file:
/etc/incidentrelay/incidentrelay.conf

Web application service:
incidentrelay.service

Scheduler service:
incidentrelay-scheduler.service

Database:
PostgreSQL

Database name:
incidentrelay

Application server:
Gunicorn

Application port:
8081

Reverse proxy:
Nginx

Local deployment procedure

A timestamped backup of the modified Python file was created before applying the change.

The notification-policy rule configuration was also backed up before adding the resolved event type.

Syntax validation:

/var/www/incidentrelay/venv/bin/python -m py_compile \
  /var/www/incidentrelay/app/services/heartbeats/service.py

Import validation:

cd /var/www/incidentrelay

sudo -u incidentrelay env \
  PYTHONPATH=/var/www/incidentrelay \
  INCIDENTRELAY_CONFIG_FILE=/etc/incidentrelay/incidentrelay.conf \
  /var/www/incidentrelay/venv/bin/python -c \
  'from app.services.heartbeats import service'

Service restart:

systemctl restart incidentrelay.service
systemctl restart incidentrelay-scheduler.service

Service verification:

systemctl status incidentrelay.service
systemctl status incidentrelay-scheduler.service

The final behavior was validated with new heartbeat overdue and recovery cycles.

[Aidaho12]

Quite a detailed description. If everything is ready, why don't you create a pull request?

[Artesfrost]

@Aidaho12 Thanks for the suggestion. I created the issue first because I wanted to confirm that I had correctly understood the intended logic and wasn’t overlooking an existing design decision.

I’m also deploying and testing IncidentRelay in a restricted internal environment, where I unfortunately can’t use all the development and testing tools I would normally rely on. Because this isn’t my project, I’m trying to be especially careful and respectful with proposed changes.

Now that the expected approach is clearer, I’ll try to submit similar fixes as pull requests going forward.

[Aidaho12]

The reported problem is valid, and the general direction is correct: heartbeat recovery resolves the alert group, but it does not currently complete the same notification flow used by the standard alert lifecycle.

However, the proposed implementation needs a few architectural adjustments before it should be merged.

What is correct

The heartbeat recovery flow currently calls:

resolved = resolve_alert(group.id, user_id=None)

This correctly:

• changes the alert group status to resolved;
• creates the standard resolved timeline event;
• refreshes service and business-impact state;
• updates existing messages for notification providers that support message editing.

The missing part is delivery to channels such as Microsoft Teams, where a separate resolved message must be sent.

It is also correct to use the existing notification delivery pipeline rather than adding a direct Teams webhook call.

The last_notification_at guard is appropriate:

if resolved.last_notification_at:

A resolved notification should not be sent when the alert group never produced an initial outbound notification.

Issue 1: the proposed implementation may process resolved notifications twice

resolve_alert() already calls:

update_alert_messages(
    group,
    event_type="resolved",
)

This updates existing messages for providers that support editable notifications, including Slack, Telegram, and Mattermost.

Calling this immediately afterwards:

notify_alert(
    resolved,
    event_type="resolved",
)

may update the same messages for a second time.

Possible consequences include:

• duplicate notifier calls;
• duplicate resolved_message_updated timeline events;
• repeated updates to Slack, Telegram, or Mattermost messages.

Microsoft Teams would receive the required resolved message, but editable notification providers could be processed twice.

Issue 2: resolved is not currently a Notification Policy event type

The proposed configuration includes:

[
  "notification",
  "reminder",
  "resolved"
]

However, Notification Policy rules currently support only:

NOTIFICATION_POLICY_EVENT_TYPES = {
    "notification",
    "reminder",
    "escalation",
}

The API schema intentionally rejects resolved as an unsupported Notification Policy event type.

This should not be confused with User Notification Rules, which do support resolved.

A resolved notification should normally be delivered to the same channels that received the original alert. It should not require adding resolved to the Notification Policy matcher configuration.

Issue 3: resolved delivery does not repeat Notification Policy matching

For editable events such as acknowledged and resolved, the notification delivery layer uses existing delivery records:

if event_type in EDITABLE_EVENTS:
    return [
        (delivery.channel, delivery)
        for delivery in notifications_repo.list_notifications_for_group(
            group.id
        )
    ]

Therefore, the actual flow is:

resolved event
→ existing notification delivery records
→ update the original message when supported
→ otherwise send a separate resolved message

The Notification Policy resolver and matchers are not evaluated again.

This is desirable because the recovery message should go to the same channels that received the original alert. Re-evaluating current matchers could send a recovery notification to a new channel that never received the firing alert, while failing to notify the original channel if its configuration changed.

Issue 4: pending notification state should be cleared

The standard lifecycle clears any pending group notification when the alert is resolved:

alerts_repo.clear_alert_group_notification(group)

Heartbeat recovery should do the same.

Otherwise, a delayed notification record may remain until the notification worker processes it and notices that the group is already resolved. It will probably not produce an incorrect message, but the state remains temporarily inconsistent.

Recommended implementation

resolve_alert() should support resolving an alert without immediately updating existing notification messages.

For example:

def resolve_alert(
    alert_id,
    user_id=None,
    *,
    update_messages=True,
):
    """Resolve an alert group."""
    group_before = alerts_repo.get_alert_group(alert_id)
    old_status = getattr(group_before, "status", None)

    if old_status == "resolved":
        return group_before

    group = alerts_repo.resolve_alert_group(
        alert_id,
        user_id=user_id,
    )

    alerts_repo.create_alert_event(
        group_id=group.id,
        event_type="resolved",
        message="Alert group resolved",
        user_id=user_id,
    )

    refresh_alert_group_correlations_safely(
        group,
        reason="manual_resolve",
    )

    if getattr(group, "service_id", None):
        refresh_business_services_safely_for_technical_service(
            group.service_id,
            reason="manual_resolve",
        )

    refresh_business_impacts_safely_for_group(
        group,
        reason="manual_resolve",
    )

    if update_messages:
        update_alert_messages(
            group,
            event_type="resolved",
        )

    if old_status != group.status:
        notify_stakeholders(
            group,
            "resolved",
            old_value=old_status,
        )

    return group

The default value preserves the current behavior for all existing callers.

Heartbeat recovery can then use a shared helper:

def _resolve_heartbeat_alert(
    group,
    *,
    recovery_event_type,
    recovery_message,
):
    had_notification = bool(group.last_notification_at)

    resolved = resolve_alert(
        group.id,
        user_id=None,
        update_messages=False,
    )

    alerts_repo.clear_alert_group_notification(resolved)

    alerts_repo.create_alert_event(
        group_id=resolved.id,
        event_type=recovery_event_type,
        message=recovery_message,
    )

    if had_notification:
        notify_alert(
            resolved,
            event_type="resolved",
        )

    return resolved

The regular heartbeat recovery function can use it as follows:

return _resolve_heartbeat_alert(
    group,
    recovery_event_type="heartbeat_recovered",
    recovery_message=f"Heartbeat recovered: {heartbeat.name}",
)

The heartbeat instance recovery function can use:

return _resolve_heartbeat_alert(
    group,
    recovery_event_type="heartbeat_instance_recovered",
    recovery_message=(
        f"Heartbeat recovered: "
        f"{instance.heartbeat.name} / {instance.instance_key}"
    ),
)

This produces the intended flow:

Heartbeat recovered
→ resolve_alert(update_messages=False)
→ clear pending notification state
→ record the heartbeat recovery event
→ notify_alert(event_type="resolved")
→ use existing delivery records
→ update editable messages or send a separate recovery message

Notification Policy configuration

No resolved event type needs to be added to Notification Policy rules.

For example, the Teams policy may remain:

[
  "notification",
  "escalation"
]

When the heartbeat recovers, the resolved notification will be delivered using the existing notification records created for the original alert.

User Notification Rules may still include resolved when users want personal recovery notifications.

Required tests

The change should include tests covering:

1. A regular heartbeat with auto_resolve=True sends a resolved notification.
2. A heartbeat instance with auto_resolve=True sends a resolved notification.
3. No resolved notification is sent when last_notification_at is empty.
4. With auto_resolve=False, the alert remains open and no resolved notification is sent.
5. Pending group notification state is cleared on recovery.
6. resolve_alert(update_messages=False) does not call update_alert_messages().
7. The default resolve_alert() behavior still updates existing messages.
8. Slack, Telegram, and Mattermost are not updated twice.
9. Microsoft Teams receives a separate resolved message.
10. Repeated heartbeat pings after recovery do not send duplicate resolved notifications.
11. Notification Policy rules continue to reject resolved.
12. User Notification Rules continue to support resolved.

Conclusion

The root cause and the intention of the proposed fix are correct, but directly calling notify_alert(..., event_type="resolved") after the existing resolve_alert() call does not fully match the current IncidentRelay notification architecture.

The implementation should:

• prevent duplicate message updates;
• continue using existing delivery records for resolved notifications;
• avoid adding resolved to Notification Policy event types;
• clear pending notification state;
• share the recovery logic between regular and instance heartbeats.

I can fix it on my self ltle bit latter