Skip to content

fix(push): make notification delivery observable - #18

Open
github-actions[bot] wants to merge 1 commit into
mainfrom
ci/push-observability
Open

github-actions[bot] wants to merge 1 commit into
mainfrom
ci/push-observability

Conversation

@github-actions

Copy link
Copy Markdown

Mobile notifications stopped on Sep 13 and went unnoticed for two days. Apple
returned 410 Gone on the only endpoint at 07:00:20 during the scheduled-action
window, the gateway correctly pruned it, and every notification after that
silently went nowhere. Nothing we shipped caused it - the gateway had run
continuously since Sep 7 with push enabled and VAPID intact - but the system
gave no way to notice.

The sender logged ONLY failures. A healthy send and a send with zero
subscriptions produced identical output: nothing. So "push is working" and
"push has been dead for days" looked the same in the journal, and the only
symptom was a human noticing their phone was quiet while the desktop filled up.

Two lines fix that:

  • Zero subscriptions now logs a warning naming the user and the dropped
    notification title, plus what to do about it. That is the exact state this
    outage left the system in, and it was completely silent.
  • Every send logs delivered/total. Push volume is a few notifications a day, so
    a line per send is cheap and makes the question answerable by grep.

Not done here: push_subscriptions.last_used exists but is never written, so
"when did push last actually work" is unanswerable. Maintaining it means
extending SubscriptionStore and its implementations, which is a wider change
than this warrants; worth doing if this recurs.


Opened automatically after the tiers check passed on c24b170c.

The check is attached to this branch head, so this is mergeable now.

Mobile notifications stopped on Sep 13 and went unnoticed for two days. Apple
returned 410 Gone on the only endpoint at 07:00:20 during the scheduled-action
window, the gateway correctly pruned it, and every notification after that
silently went nowhere. Nothing we shipped caused it - the gateway had run
continuously since Sep 7 with push enabled and VAPID intact - but the system
gave no way to notice.

The sender logged ONLY failures. A healthy send and a send with zero
subscriptions produced identical output: nothing. So "push is working" and
"push has been dead for days" looked the same in the journal, and the only
symptom was a human noticing their phone was quiet while the desktop filled up.

Two lines fix that:

- Zero subscriptions now logs a warning naming the user and the dropped
  notification title, plus what to do about it. That is the exact state this
  outage left the system in, and it was completely silent.
- Every send logs delivered/total. Push volume is a few notifications a day, so
  a line per send is cheap and makes the question answerable by grep.

Not done here: push_subscriptions.last_used exists but is never written, so
"when did push last actually work" is unanswerable. Maintaining it means
extending SubscriptionStore and its implementations, which is a wider change
than this warrants; worth doing if this recurs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant