Fix Live Activity fanout stalls from APNs timeouts and ExpiredToken cleanup

[mithileshchellappan]

Summary

During the Canada qualifying live activity window, Pushboy accepted admin-api job requests quickly, but live activity fanout fell 20-40 minutes behind. The main observed contributors were APNs live activity send timeouts/retries and stale APNs live activity tokens that were not being invalidated on ExpiredToken.

Production evidence

Window inspected: 2026-05-23 19:45:00 UTC to 2026-05-23 21:45:00 UTC.

Live activity job:

• activity_id: 5_montreal_qualifying_2026
• topic_id: broadcast
• Pushboy HTTP acceptance for /v1/live-activity/jobs: 154 successful 202 responses, p50 24ms, p95 260ms
• This means admin-api submission was not the slow part.

Dispatch latency:

• Start dispatch: 30,156 tokens, completed in 1,270s (~21 minutes)
• Update dispatches: 153
• Update avg fanout size: 12,139 tokens
• Update latency: p50 1,338s, p95 2,315s, max 2,342s (~39 minutes)

Provider/error mix from Pushboy logs:

• 29,278 APNs live activity failures
• 22,912 APNs live activity failures classified as timeout/client-header waits
• 4,268 APNs ExpiredToken
• FCM live activity failures were not material in this window (4)

Suspected causes

1. ExpiredToken is not treated as an invalid live activity token.

   Current invalid-token handling covers BadDeviceToken, Unregistered, and FCM registration-token-not-registered, but not APNs ExpiredToken. Those stale LA tokens keep participating in fanout until passive expiry or another invalidation path removes them.

2. Timeout retries can monopolize sender workers.

   APNs sends use a 10s HTTP client timeout and retry retryable timeout-like errors up to 3 times. A single bad/stalled APNs send can therefore occupy a sender for roughly 40s. With ~7k-13k APNs/LA update tokens per update and updates arriving around every 30s, dispatches can queue faster than workers complete them.

3. Observability makes this hard to detect early.

   /v1/live-activity/jobs returning 202 quickly can look healthy while dispatch completion is badly delayed. We need dispatch/fanout metrics to separate acceptance latency, enqueue latency, and provider-send completion latency.

Proposed fixes

• Treat APNs ExpiredToken as invalid for live activity tokens and mark those tokens invalid in ApplyLAOutcomeBatch.
• Revisit APNs LA timeout/retry behavior so timeout storms cannot hold the whole sender pool for tens of minutes.
• Add/log metrics for:
  • LA dispatch created to completed duration
  • total/success/failure/unresolved counts by dispatch
  • provider error counts by reason
  • current in-flight/backlog count for LA dispatches
• Consider provider-specific concurrency/backpressure so APNs timeout storms do not starve other live activity work.

Acceptance criteria

• ExpiredToken LA failures invalidate the corresponding live activity token.
• A timeout-heavy APNs window no longer causes 20-40 minute LA update completion delays.
• Operators can see whether slowness is in job acceptance, token fanout/enqueue, or provider sends.

[mithileshchellappan]

Additional finding from follow-up investigation:

The fanout is not reading the normal tokens table. Production source confirms GetLATokenBatchForDispatch() joins live_activity_dispatches -> live_activity_jobs -> live_activity_tokens.

However, update fanout is still far too broad because update tokens are selected by the job scope/topic (broadcast) instead of the specific activityId.

Evidence for 5_montreal_qualifying_2026:

• First quali update dispatch at 2026-05-23 19:50:39 UTC had 7,293 tokens.
• Of those, only 2 APNs update tokens were created after the qualifying job started.
• 7,152 APNs update tokens and 139 FCM update tokens were created before qualifying started.
• Last quali update dispatch at 2026-05-23 21:06:39 UTC still included 7,147 APNs update tokens and 138 FCM update tokens created before qualifying started, plus 6,088 APNs tokens created after qualifying started.

So the main correctness issue is likely that live activity update tokens are not associated with the activity/job they came from. Pushboy currently sends every update for an active broadcast LA job to all active broadcast LA update tokens, including update tokens from previous activities that have not expired yet.

This explains the unexpectedly high aggregate fanout (1,857,332 update sends = 153 updates * ~12k tokens/update) even though the current activity should only have had a few thousand real update tokens.

Likely required fix:

• Store activity_id or live_activity_job_id on update-token registration, or maintain an activity-token association table.
• For action=update / action=end, fan out only to update tokens attached to that activityId.
• Keep topicId=broadcast fanout only for action=start, where start tokens are intentionally global/topic-scoped.
• Do not rely on passive expiry to separate sessions; overlapping F1 sessions make that too broad.