fix(push): make notification delivery observable - #18
Open
github-actions[bot] wants to merge 1 commit into
Open
github-actions[bot] wants to merge 1 commit into
github-actions[bot] wants to merge 1 commit into
Conversation
Mobile notifications stopped on Sep 13 and went unnoticed for two days. Apple returned 410 Gone on the only endpoint at 07:00:20 during the scheduled-action window, the gateway correctly pruned it, and every notification after that silently went nowhere. Nothing we shipped caused it - the gateway had run continuously since Sep 7 with push enabled and VAPID intact - but the system gave no way to notice. The sender logged ONLY failures. A healthy send and a send with zero subscriptions produced identical output: nothing. So "push is working" and "push has been dead for days" looked the same in the journal, and the only symptom was a human noticing their phone was quiet while the desktop filled up. Two lines fix that: - Zero subscriptions now logs a warning naming the user and the dropped notification title, plus what to do about it. That is the exact state this outage left the system in, and it was completely silent. - Every send logs delivered/total. Push volume is a few notifications a day, so a line per send is cheap and makes the question answerable by grep. Not done here: push_subscriptions.last_used exists but is never written, so "when did push last actually work" is unanswerable. Maintaining it means extending SubscriptionStore and its implementations, which is a wider change than this warrants; worth doing if this recurs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mobile notifications stopped on Sep 13 and went unnoticed for two days. Apple
returned 410 Gone on the only endpoint at 07:00:20 during the scheduled-action
window, the gateway correctly pruned it, and every notification after that
silently went nowhere. Nothing we shipped caused it - the gateway had run
continuously since Sep 7 with push enabled and VAPID intact - but the system
gave no way to notice.
The sender logged ONLY failures. A healthy send and a send with zero
subscriptions produced identical output: nothing. So "push is working" and
"push has been dead for days" looked the same in the journal, and the only
symptom was a human noticing their phone was quiet while the desktop filled up.
Two lines fix that:
notification title, plus what to do about it. That is the exact state this
outage left the system in, and it was completely silent.
a line per send is cheap and makes the question answerable by grep.
Not done here: push_subscriptions.last_used exists but is never written, so
"when did push last actually work" is unanswerable. Maintaining it means
extending SubscriptionStore and its implementations, which is a wider change
than this warrants; worth doing if this recurs.
Opened automatically after the
tierscheck passed onc24b170c.The check is attached to this branch head, so this is mergeable now.