fix: close the platform events websocket iterator on shutdown - #1077
Merged
Conversation
vdusek
added
adhoc
github-actions
Bot
added
the
tested
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1077 +/- ##
==========================================
- Coverage 91.92% 91.90% -0.03%
==========================================
Files 51 51
Lines 3232 3235 +3
==========================================
+ Hits 2971 2973 +2
- Misses 261 262 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vdusek
marked this pull request as ready for review
August 4, 2026 11:51
B4nan
approved these changes
Aug 4, 2026
vdusek
added a commit
that referenced
this pull request
Aug 6, 2026
`ApifyEventManager` ignored the re-entrancy contract of Crawlee's `EventManager`, which tracks active contexts with `_active_ref_count`. `BasicCrawler._run_crawler` always enters the *global* event manager, and `Actor.init` registers the `ApifyEventManager` as exactly that and has already entered it — so on the platform every crawler run re-entered it and opened a second websocket. Measured on master, with a local events server and a `BasicCrawler.run()` inside an entered `ApifyEventManager`: - Two connections stay open for as long as the run lasts, so every platform event arrives twice. One `Migrating` became two `Migrating` listener calls and two `PersistState(is_migrating=True)` — state persisted twice per migration. The extra connection also counts against the platform limit of 10 per run. - The crawler's exit tears down that newer connection, which leaves `__aexit__` bookkeeping pointing at an already-closed one. The Actor's own connection and its message-processing task therefore survive `Actor.exit()`, re-breaking the iterator shutdown from #1077. The websocket is now owned by the outermost context only, mirroring `LocalEventManager`. On top of that: - The shutdown moved into `_teardown_platform_websocket()` and the parent context is left in a `finally`, so a failed shutdown can no longer keep the manager active for good — which would mean never emitting `PersistState` again. - Task, connection and future are reset on exit, so the context can be entered again. - A cancelled entry cleans up as well. Without it, a later entry mistook itself for a nested context and silently returned a manager that received no platform events at all. - The error that prevented the first connection is raised as the `__cause__` of the `RuntimeError`, not only logged. - `Actor.reboot()` reads `_listeners_to_wrappers` with `get`, so the lookup no longer inserts entries for events nobody listens to. Adopts the patterns from apify/crawlee-python#2100 (Crawlee 1.9.1). *✍️ Drafted by Claude Code*
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since v4.0.0, an Actor logs
RuntimeError: aclose(): asynchronous generator is already runningright afterActor.exit(). Cosmetic - the exit code stays 0 and no data is lost - but it shows up in every run and looks like a failure.ApifyEventManager._process_platform_messagesiterateswebsocketsconnect(...)withasync for. That iterator is an async generator, and neitherbreaknor cancelling the task closes it, so the garbage collector does - duringasyncio.runteardown, where theaclose()it schedules lands after_cancel_all_taskstook its task snapshot and is never awaited.loop.shutdown_asyncgens()then callsaclose()on a generator that is still running. Up to v3.4.1 the code usedasync with connect(...), which unwound synchronously on cancellation.contextlib.aclosingcloses the iterator on every exit path -break, exception and cancellation alike. The rest of the diff is the loop body re-indenting, a unit test that asserts the iterator is closed, and an E2E test that asserts the error is absent from a run log.Reported on Discord.
✍️ Drafted by Claude Code