Conversation
…tion asyncio streams store a failed reader's exception and re-raise the SAME object on every subsequent read, so its traceback grows by a few frames per raise. The _listen loop logged the full (ever-growing) traceback on every iteration via logging.exception; combined with Python 3.13+ fine-grained traceback formatting (ast.parse per frame for caret anchors) this becomes quadratically expensive inside the event loop and was observed pinning a CPU core and starving a Home Assistant instance until its watchdog killed it. Track the last exception seen by identity: the first occurrence keeps today's full logging.exception, repeats of the same object are summarised with the existing rate-limited warning helper and the loop yields for a second so it can never spin hot on a poisoned reader. The FakeReader gains StreamReader-faithful sticky-exception semantics (set_exception) to reproduce the failure mode in tests.
This was referenced Jun 11, 2026
bvis
added a commit
to bvis/aegis-hass
that referenced
this pull request
Jun 14, 2026
When the upstream reconnect-storm fix (sdb9696/firebase-messaging#39) ships and the requirement is bumped, the local hardening should be re-evaluated. Add code anchors at the dependency pin (pyproject.toml) and the guard module so the reminder fires at the point of action; manifest.json is JSON (no comments) and stays the version source of truth. Tracked in #297.
Author
|
Hi, is anything I could do? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #23 and #33; complementary to #36 and #38 (different code path: this covers the generic "unexpected exception" branch that those PRs leave untouched).
Summary
When the MCS stream fails persistently, the
_listenloop can re-format and log the same exception object on every iteration — and that exception's traceback grows on each raise, making the logging quadratically expensive inside the event loop.Mechanism:
StreamReaderstores a failed stream's exception and re-raises the SAME object on every subsequent read (asyncio/streams.py,raise self._exception), appending a few frames to its__traceback__each time.ConnectionResetErroroutsideRESETTINGstate hits the genericelsebranch, which calls_logger.exception(...)— full traceback, every iteration._reset()early-returns (its lock is held, e.g. by the monitor task), the loop re-reads the poisoned reader with no suspending await → tight loop.ast.parseper frame (caret anchors), so formatting the growing chain dominates: observed in production as a CPU core pinned at 100% for >10 minutes inside the event loop.Observed on a Home Assistant 2026.6.1 / Python 3.14 installation during a period when Google persistently reset the MCS session right after login: HA's event loop starved, the HTTP API died, and the OS watchdog kill-looped the process. py-spy dump of the hung process:
Changes
logging.exception). Repeats of the same object are summarised through the existing_log_warn_with_limithelper (one line, no traceback) and the loopawait asyncio.sleep(1)s so it can never spin hot on a poisoned reader. Error accounting (_try_increment_error_count/_reset) is unchanged in both paths.tests/fakes.py:FakeReader.set_exception()mimicsStreamReader's sticky-exception semantics so the failure mode is reproducible in tests.mainand passes with this change.Testing
uv run pytest tests/test_fcmpushclient.py— 14 passed (13 existing + 1 new).fcmpushclient.pychange.ruff check/ruff format --checkclean.