Skip to content

Recover from transient read failures instead of aborting the client - #38

Open
MultivitaminJuice wants to merge 1 commit into
sdb9696:mainfrom
MultivitaminJuice:fix/transient-read-recovery
Open

MultivitaminJuice wants to merge 1 commit into
sdb9696:mainfrom
MultivitaminJuice:fix/transient-read-recovery

Conversation

@MultivitaminJuice

@MultivitaminJuice MultivitaminJuice commented Jun 3, 2026

Copy link
Copy Markdown

Addresses #33. That issue's traceback shows TimeoutError: SSL shutdown timed out being counted as ErrorType.CONNECTION, so after 3 occurrences the client hits abort_on_sequential_error_count and shuts down for good. This PR treats that timeout (and IncompleteReadError) as transient and reconnects without advancing the abort counter. Complementary to #36 — the two merge cleanly. Reported downstream in home-assistant/core#157764 and home-assistant/core#134431 (the SSL shutdown timed out → 3× ErrorType.CONNECTION → push-receiver shutdown that users have to reload the Ring integration to recover from).


Summary

Makes the MCS read loop recover from transient stream failures instead of counting them toward the fatal abort threshold and terminating the client.

Previously asyncio.IncompleteReadError and TimeoutError (when not already in a reset) fell into the generic "unexpected exception" branch, advanced the CONNECTION error counter, and after abort_on_sequential_error_count failures terminated the client for good. In practice these are transient stream hiccups — a 0-byte read, or "SSL shutdown timed out" after a stuck writer close — from which a simple reconnect recovers. On a long-running Home Assistant / Ring setup this manifested as push silently stopping until a full restart.

Changes

  • Handle asyncio.IncompleteReadError and TimeoutError as transient: log a warning and _reset() the connection without advancing the CONNECTION abort counter.
  • Catch an otherwise-unexpected ValueError in the read loop and _reset() (subject to the NOTIFY sequential-error limit) instead of letting it bubble up to the outer handler that immediately terminates the client.

The existing reset-state handling for these exception types is unchanged; the new branches only apply when the client is not already resetting.

Testing

  • uv run pytest tests/test_fcmpushclient.py – all green, with new tests asserting that an IncompleteReadError / TimeoutError triggers a reconnect (LoginRequest re-sent) without incrementing the CONNECTION counter and without terminating.
  • ruff check / ruff format --check clean.
  • Running on a live Home Assistant install (Ring integration) for several weeks; push stays alive across transient network blips that previously killed it.

Notes

This is the second of two independent PRs; the first (#37) hardens encrypted-payload decoding. They touch different code paths and can be reviewed/merged separately.

🤖 Generated with Claude Code

@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 79.379% (-0.08%) from 79.454% — MultivitaminJuice:fix/transient-read-recovery into sdb9696:main

@sdb9696 sdb9696 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @MultivitaminJuice thanks for the PR! I have some questions and change requests inline. Also this needs some kind of exponential backoff and a separate counter as it can't just keep trying every three seconds forever. Probably some extra config as well to control the behaviour.

Comment thread CHANGELOG.md Outdated
Comment on lines +3 to +8
## [Unreleased]

**Fixed bugs:**

- Treat `asyncio.IncompleteReadError` as a transient read failure: log at warning level, reconnect, and do not advance the connection error counter toward abort.
- Recover from unexpected `ValueError` in the MCS read loop via reset when the sequential NOTIFY error limit allows, instead of always terminating the client immediately.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHANGELOG is auto generated

Comment thread firebase_messaging/fcmpushclient.py Outdated
Comment on lines +743 to +752

except ValueError as ex:
_logger.error(
"Recoverable value error in FcmPushClient: %s\n%s",
ex,
traceback.format_exc(),
)
if self._try_increment_error_count(ErrorType.NOTIFY):
await self._reset()

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its not clear how this relates to this PR

Comment thread firebase_messaging/fcmpushclient.py Outdated
Comment on lines +737 to +738
if self.do_listen:
await self._reset()

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why check for self.do_listen?

Comment thread firebase_messaging/fcmpushclient.py Outdated
)
if self.do_listen:
await self._reset()
elif isinstance(osex, TimeoutError):

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be combined with above as the only difference is the log message

Comment thread firebase_messaging/fcmpushclient.py Outdated
# Transient stream read (e.g. 0 bytes of an
# expected 1): reconnect without advancing the
# CONNECTION abort counter.
_logger.warning(

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look at _log_warn_with_limit() to avoid spamming the log.

@MultivitaminJuice

Copy link
Copy Markdown
Author

Thanks for the detailed review! I'll address the inline points — combine the two branches, switch to _log_warn_with_limit, drop the redundant do_listen check, and leave the CHANGELOG alone. On the ValueError handler: it was an earlier, broader catch for decrypt errors that #37 now handles locally in _handle_data_message, so it doesn't belong in this (transient-read) PR — I'll remove it here.

On the backoff, before I implement it — does this match what you have in mind?

  • a dedicated counter for these transient resets, separate from the ErrorType ones;
  • exponential delay before each reconnect, min(base * 2**n, cap);
  • new FcmPushClientConfig fields to control it (base interval, cap, and a retry limit).

Two things I'd like your steer on:

  1. When the retry limit is reached, should the client terminate (like abort_on_sequential_error_count), or just keep retrying at the capped max interval indefinitely?
  2. Reuse the existing start_seconds_before_retry_connect / reset_interval for the base, or add dedicated fields?

Addresses review feedback on sdb9696#38.

- Merge the IncompleteReadError / TimeoutError read-loop branches into one
  and reconnect with exponential backoff (capped at reconnect_backoff_max),
  driven by a dedicated _transient_error_count, instead of warning + resetting
  on every iteration. Uses _log_warn_with_limit and drops the redundant
  do_listen guard (_reset checks it). The counter resets on the next
  successful read.
- Make _reset() self-heal: when _connect_with_retry() exhausts its tries
  during a sustained outage it no longer terminates the client; it keeps
  retrying with backoff so push recovers automatically once connectivity
  returns. The new abort_on_connection_failure flag restores the legacy
  shut-down behaviour.
- Tests for transient-read reconnect (no CONNECTION counter, no terminate),
  self-heal on repeated connect failure, and the abort_on_connection_failure
  opt-out.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@MultivitaminJuice
MultivitaminJuice force-pushed the fix/transient-read-recovery branch from 5578487 to f722b83 Compare June 13, 2026 06:47
@MultivitaminJuice

Copy link
Copy Markdown
Author

Thanks for the detailed review — and apologies in advance: I'm coming at this as a downstream Home Assistant / Ring user who ran into the push receiver dying, not as a firebase-messaging expert. So please reshape, rename, re-default, split, or drop anything that doesn't fit the library's direction — I trust your judgement on the right form far more than my own here.

Rebased onto main.

Your inline points:

  • Combined the IncompleteReadError / TimeoutError branches into one.
  • Switched to _log_warn_with_limit (no more log spam).
  • Dropped the redundant do_listen check — _reset() already guards on it.
  • Removed the out-of-scope ValueError handler.
  • Left the CHANGELOG alone.

Backoff + counter: transient read failures now reconnect with exponential backoff min(reset_interval * 2**n, reconnect_backoff_max), driven by a dedicated _transient_error_count that resets on the next successful read.

The part I'm least sure about (very much your call): in my setup the "push dies and never comes back" actually happens one level down — when _reset()_connect_with_retry() exhausts its tries during an outage and calls _terminate(), the listener exits and stays dead until the integration is reloaded. I changed _reset() to keep retrying with backoff instead (self-heal), behind a new abort_on_connection_failure flag (currently defaulting to the new behaviour).

I'm genuinely unsure this is the right layer, shape, or default for the library — it's simply what reliably keeps push alive on my HA/Ring box. I'd completely understand if you'd rather keep the terminate and leave recovery to the caller, flip the default the other way, or split the self-heal into its own PR. Happy to do whatever makes it easiest for you to review (or to drop the self-heal entirely and keep just the read-loop changes).

Tests cover the transient reconnect (no CONNECTION counter, no terminate), the self-heal on repeated connect failure, and the abort_on_connection_failure opt-out.

@Bu66as

Bu66as commented Aug 3, 2026

Copy link
Copy Markdown

Downstream data point from a Home Assistant / Ring user, and a small release request.

The gap: the last PyPI release is 0.4.5 (2025-05-10). ring-doorbell pins that release, and Home Assistant installs it from the pin — so #36 (merged 2026-06-10) currently reaches no downstream user at all. Every HA Ring user on the current stable (2026.7.4 → ring-doorbell==0.9.14) is still running the pre-#36 code.

What that looks like in practice, from my own install:

  • Doorbell push events stopped on 2026-07-08, motion push events on 2026-07-30 16:06 — nothing has arrived since.
  • The polling path is completely healthy: the Ring cloud sensors still update within ~60 s, so the account, token and network path are fine. Only the FCM listener is gone.
  • Notably, netstat still shows the MCS socket as ESTABLISHED to 74.125.71.188:5228 long after the client has stopped delivering anything. So from the outside the connection looks alive while the listener has already shut itself down.

That last point is why the workarounds circulating in the HA issue don't help: they reload the config entry when the event entity goes unavailable, but the entity never does — it just silently stops updating.

The request: would you consider cutting a 0.4.6 with #36 as-is? That alone would get the fix to downstream users, independent of when #38 lands. If you'd rather ship both together, that's of course fine too — this is really just a note that the fix is currently stuck behind the release, not behind the code.

Related HA Core issue with 21 comments from affected users: home-assistant/core#157764.

Happy to test a pre-release or a git ref against a live Ring doorbell if that's useful — I have a reproducible dead-listener install right now.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants