Skip to content

fix(fcm): survive an undecodable push frame instead of dying on it (#373) - #374

Merged
bvis merged 1 commit into
mainfrom
fix/373-fcm-poison-pill
Aug 2, 2026
Merged

bvis merged 1 commit into
mainfrom
fix/373-fcm-poison-pill

Conversation

@bvis

@bvis bvis commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Fixes the failure @wip3out3r documented in #359 and that #373 tracks: a single push message the FCM library cannot decrypt ends real-time push permanently, and silently.

The mechanism, verified against the library source

_decrypt_raw_data decodes the crypto-key and encryption header values without padding, while padding the two stored key values it decodes in the same function. Those headers are URL-safe base64 that may legitimately arrive with the trailing = stripped, so an unpadded one raises binascii.Error.

That is a ValueError, so the listen loop's inner except (OSError, EOFError) does not catch it. It reaches the outer except Exception, which logs Unknown error: Incorrect padding, shutting down FcmPushClient and terminates the client.

The severe part is where it raises. In _handle_message the decrypt runs before the two statements that acknowledge the message, so neither the persistent-id bookkeeping nor the selective ack ever executes. The message is never acknowledged, so it is redelivered on the next connection, and the supervision added in #285 faithfully restarts straight back into it. Measured by the reporter: the same message killed the client 16 times over 3.5 hours, each death 3–9 ms after receiving it, and a host reboot did not clear it because the queue lives server-side.

What this changes

1. Pad both header values, and contain any remaining failure. install_fcm_decrypt_guard patches _decrypt_raw_data to pad crypto-key / encryption before delegating — the same + "========" treatment the library already applies to the private key and secret — and returns b"" if the decrypt still fails. The handler then continues on its normal path, logs its own "Failed to decrypt data" line, hands the callback an empty payload (no ENCODED_DATA, so our notification handler ignores it), and reaches the acknowledgement. One event is lost instead of every future one.

2. Surface repeated deaths as a Repair. Three consecutive terminations sharing the same last-received persistent_id raise fcm_push_stuck. Deaths with no push received at all carry no id and deliberately do not accumulate, so an ordinary network outage cannot be misreported as a poisoned message. This exists because the failure is otherwise invisible: alarm state comes from polling and the HTS path and never from push, so nothing looks broken while real-time events are gone — the reporter only found it while investigating something unrelated.

3. Renew the registration once at that threshold. The replayed frame is queued server-side against a specific registration, so a new identity is the only exit. That is what the reporter did by hand; without this the only recovery available to a user is editing .storage. It runs at most once per streak — an unbounded series of registrations against the Firebase project is exactly what #227 exists to prevent — and re-arms after a healthy run, which also clears the Repair.

Tests

  • _pad_urlsafe_b64 across all four length remainders, including a test that pins the bug itself (unpadded input raises without the fix) so the premise is checked rather than assumed.
  • The guard against a per-test fake: unpadded decodes, padded still decodes, undecodable returns empty rather than raising, non-base64 garbage is contained, install is idempotent, first failure logs WARNING and repeats drop to DEBUG.
  • Against the real FcmPushClient, not only a double: the unpatched class genuinely raises binascii.Error on unpadded input, and the guard contains it. A fixture snapshots and restores the class so this holds regardless of suite order.
  • The Repair: raised on the third same-id death, not raised for distinct ids, not raised when no push ever arrived, streak broken by an intervening message, cleared after a healthy run, and no registry churn when there was no streak.
  • The recovery: registration discarded and renewed at the threshold, only once across seven deaths, the replacement client survives the supervisor's teardown (ordering guard), a failed re-registration is swallowed back onto the retry path, and a failed store removal skips re-registration rather than burning a registration with the poisoned queue still in place.

fcm_push_stuck is translated in all 14 shipped locales.

Two incidental fixes found on the way

  • test_start_without_firebase_messaging simulated an absent package by nulling only the parent firebase_messaging key, but the code imports the submodule firebase_messaging.fcmregister, and a cached submodule stays importable when the parent is None. The test silently stopped simulating anything as soon as another test imported the library, and fell through into the real Store. It now nulls both.
  • strings.json was missing the fcm_not_configured entry that translations/en.json already had.

Upstream

The root cause is fixed properly in sdb9696/firebase-messaging#37, reported as sdb9696/firebase-messaging#40, and independently hit by the Home Assistant Ring integration and Fermax Blue. That PR has been open and mergeable since June with no maintainer movement, and the newest release predates it. The guard is removable once a release ships the fix — same standing upstream wait as #297.

Verified: make check green on 3.12 (2100 passed, 90% coverage) and the suite green on 3.13, plus three randomised orderings to confirm no cross-test pollution remains.

Refs #373, #359

)

One push message the library cannot decrypt was enough to end real-time
push permanently, and silently.

`_decrypt_raw_data` decodes the `crypto-key` and `encryption` header values
without padding, while padding the two stored key values in the same
function. Those headers are URL-safe base64 that may legitimately arrive
without trailing `=`, so an unpadded one raises `binascii.Error`. That is a
`ValueError`, so the listen loop's `except (OSError, EOFError)` misses it, it
reaches the outer `except Exception`, and the client shuts down.

The severe part is where it raises: before the library appends the persistent
id and sends the selective ack. The message is therefore never acknowledged,
so it is redelivered on the next connection — and the supervision from #285
faithfully restarts straight back into it. The reporter measured the same
message killing the client 16 times over 3.5 hours, each death 3-9 ms after
receiving it, surviving a host reboot because the queue is server-side.

Three changes:

- `install_fcm_decrypt_guard` pads both header values before delegating, and
  contains any remaining decode failure by returning empty bytes. The handler
  then stays on its normal path and reaches the acknowledgement, so one event
  is lost instead of every future one.
- Repeated deaths on the same persistent_id raise a Repair. The failure is
  otherwise invisible: alarm state comes from polling and HTS, never from
  push, so nothing looks wrong from the outside.
- At that same threshold the stored FCM registration is discarded and renewed
  once, which is what the reporter did by hand. Without it the only recovery
  available to a user is editing `.storage`.

Upstream fixes the root cause in sdb9696/firebase-messaging#37, open and
mergeable since June with no release carrying it; the guard goes away when a
release ships it.

Also fixes two things found on the way: `test_start_without_firebase_messaging`
simulated an absent package by nulling only the parent module, which stops
working once anything imports a submodule, and `strings.json` was missing the
`fcm_not_configured` entry that `translations/en.json` already had.

Refs #373, #359
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant