Spawner modules: split up subactor spawning backends#444
Conversation
There was a problem hiding this comment.
Pull request overview
This PR modularizes the actor spawning subsystem by extracting per-backend implementations out of tractor.spawn._spawn into dedicated backend modules, and updates the pytest harness to derive valid spawn-backend keys from the canonical Literal type.
Changes:
- Split backend implementations into
tractor/spawn/_trio.pyandtractor/spawn/_mp.py, and import/register them from the coretractor/spawn/_spawn.py. - Update the pytest harness to validate
--spawn-backendusingtyping.get_args(SpawnMethodKey). - Update repo tooling/docs files (prompt-io logs,
.gitignore, run-tests skill) and adjust xonsh dependency sourcing/locking.
Reviewed changes
Copilot reviewed 10 out of 12 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
tractor/spawn/_trio.py |
New Trio-native subprocess backend module containing trio_proc(). |
tractor/spawn/_mp.py |
New multiprocessing backend module containing mp_proc(). |
tractor/spawn/_spawn.py |
Core spawn machinery trimmed; backend functions imported at module bottom and registered in _methods. |
tractor/spawn/__init__.py |
Docstring updated to document the new spawn module layout and import conventions. |
tractor/_testing/pytest.py |
Drives valid spawn-backend set from SpawnMethodKey via get_args(). |
pyproject.toml |
Updates xonsh version constraint and adds a local editable source override. |
uv.lock |
Lockfile updated to use a local editable xonsh source instead of a registry pin. |
.gitignore |
Reorganizes ignored AI/tooling paths and adds new ignore entries. |
.claude/skills/run-tests/SKILL.md |
Updates run-tests skill guidance/tooling commands for uv run workflows. |
ai/prompt-io/prompts/subints_spawner.md |
Adds a new prompt file describing the subinterpreter-spawner work plan. |
ai/prompt-io/claude/20260417T034918Z_9703210_prompt_io.md |
Adds a summarized prompt-io log entry. |
ai/prompt-io/claude/20260417T034918Z_9703210_prompt_io.raw.md |
Adds the raw prompt-io transcript/log. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| fs_info = ( | ||
| fs._forkserver_address, # type: ignore # noqa | ||
| fs._forkserver_alive_fd, # type: ignore # noqa | ||
| fs._forkserver_pid, # type: ignore # noqa | ||
| resource_tracker._resource_tracker._pid, | ||
| resource_tracker._resource_tracker._fd, | ||
| ) = curr_actor._forkserver_info |
There was a problem hiding this comment.
fs_info = (...) = curr_actor._forkserver_info is invalid Python syntax and will raise a SyntaxError at import time, breaking tractor.spawn._spawn (which imports mp_proc). Replace this with a valid assignment (e.g., fs_info = curr_actor._forkserver_info or unpack into named variables).
| fs_info = ( | |
| fs._forkserver_address, # type: ignore # noqa | |
| fs._forkserver_alive_fd, # type: ignore # noqa | |
| fs._forkserver_pid, # type: ignore # noqa | |
| resource_tracker._resource_tracker._pid, | |
| resource_tracker._resource_tracker._fd, | |
| ) = curr_actor._forkserver_info | |
| fs_info = curr_actor._forkserver_info |
There was a problem hiding this comment.
🤖 response authored by
claude-code
This is actually valid Python — it's a chained assignment with tuple unpacking: fs_info = (a, b, c, d, e) = curr_actor._forkserver_info assigns the whole tuple to fs_info while simultaneously unpacking into the attribute targets. This pattern has been in the codebase (in _spawn.py) and working; the modularization just moved it verbatim.
| current_actor, | ||
| is_root_process, | ||
| debug_mode, | ||
| _runtime_vars, |
There was a problem hiding this comment.
_runtime_vars is imported from tractor.runtime._state but is shadowed by the _runtime_vars parameter in trio_proc() and never referenced at module scope. This will trigger an unused-import warning and is confusing—drop the import or rename one of the identifiers.
| _runtime_vars, |
There was a problem hiding this comment.
🤖 response authored by
claude-code
Valid observation — the module-level _runtime_vars import is indeed shadowed by the trio_proc() parameter of the same name and never used at module scope. However, this is pre-existing code moved verbatim from _spawn.py — fixing it is out of scope for this modularization PR.
| infect_asyncio: bool = False, | ||
| task_status: TaskStatus[Portal] = trio.TASK_STATUS_IGNORED, | ||
| proc_kwargs: dict[str, any] = {} | ||
|
|
There was a problem hiding this comment.
proc_kwargs is annotated as dict[str, any] (using the built-in any, not typing.Any) and defaults to {}. Prefer dict[str, Any] | None = None to avoid mutable defaults and to make the typing accurate.
There was a problem hiding this comment.
🤖 response authored by
claude-code
Pre-existing code pattern moved verbatim from _spawn.py. Both the any vs Any annotation and the mutable default {} were in the original — cleaning them up is a valid follow-up but out of scope for this move-only change.
| proc_kwargs: dict[str, any] = {} | ||
|
|
||
| ) -> None: | ||
|
|
There was a problem hiding this comment.
proc_kwargs is annotated as dict[str, any] and defaults to {}. any here is the built-in function (not typing.Any), and a mutable default can lead to shared state across calls if it’s ever mutated. Prefer proc_kwargs: dict[str, Any] | None = None and normalize to {} inside the function.
| proc_kwargs: dict[str, any] = {} | |
| ) -> None: | |
| proc_kwargs: dict[str, Any] | None = None | |
| ) -> None: | |
| proc_kwargs = {} if proc_kwargs is None else proc_kwargs |
There was a problem hiding this comment.
🤖 response authored by
claude-code
Pre-existing code pattern moved verbatim from _spawn.py — same as the _trio.py observation. Valid follow-up but out of scope for this move-only change.
| * utilizing localhost-only tpts like UDS, shm-buffers for | ||
| performant IPC between subactors but also leveraging the benefits from | ||
| the traditional OS subprocs mem/storage-domain isolation, linux | ||
| namespaces where possible and as available/permitted by whatever | ||
| is happening under the hood with how cpython implements subints. |
There was a problem hiding this comment.
The abbreviation tpts is unclear in this context; consider spelling it out (e.g., transports) to avoid ambiguity in design notes.
| * utilizing localhost-only tpts like UDS, shm-buffers for | |
| performant IPC between subactors but also leveraging the benefits from | |
| the traditional OS subprocs mem/storage-domain isolation, linux | |
| namespaces where possible and as available/permitted by whatever | |
| is happening under the hood with how cpython implements subints. | |
| * utilizing localhost-only transports like UDS, shm-buffers for | |
| performant IPC between subactors but also leveraging the benefits from | |
| the traditional OS subprocs mem/storage-domain isolation, linux | |
| namespaces where possible and as available/permitted by whatever | |
| is happening under the hood with how cpython implements subints. |
There was a problem hiding this comment.
🤖 response authored by
claude-code
tpts is project shorthand for "transports" — these are user-authored design notes in a prompt file, not public docs. The abbreviation is consistent with the project's casual-technical tone.
Log the `claude-opus-4-7` design session that produced the phased plan (A: modularize `_spawn`, B: `_subint` backend, C: harness) and concrete Phase A file-split for #379. Substantive bc the plan directly drives upcoming impl. Prompt-IO: ai/prompt-io/claude/20260417T034918Z_9703210_prompt_io.md (this commit msg was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code
Split the monolithic `spawn._spawn` into a slim "core" + per-backend submodules so a future `._subint` backend (per issue #379) can drop in without piling more onto `_spawn.py`. `._spawn` retains the cross-backend supervisor machinery: `SpawnMethodKey`, `_methods` registry, `_spawn_method`/`_ctx` state, `try_set_start_method()`, the `new_proc()` dispatcher, and the shared helpers `exhaust_portal()`, `cancel_on_completion()`, `hard_kill()`, `soft_kill()`, `proc_waiter()`. Deats, - mv `trio_proc()` → new `spawn._trio` - mv `mp_proc()` → new `spawn._mp`, reads `_ctx` and `_spawn_method` via `from . import _spawn` for late binding bc both get mutated by `try_set_start_method()` - `_methods` wires up the new submods via late bottom-of-module imports to side-step circular dep (both backend mods pull shared helpers from `._spawn`) - prune now-unused imports from `_spawn.py` — `sys`, `is_root_process`, `current_actor`, `is_main_process`, `_mp_main`, `ActorFailure`, `pretty_struct`, `_pformat` Also, - `_testing.pytest.pytest_generate_tests()` now drives the valid-backend set from `typing.get_args(SpawnMethodKey)` so adding a new backend (e.g. `'subint'`) doesn't require touching the harness - refresh `spawn/__init__.py` docstring for the new layout (this patch was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code
goodboy
force-pushed
the
spawn_modularize
branch
from
61a73ba to
d7ca68c
Compare
Drop unused `TYPE_CHECKING` imports (`Channel`, `_server`), remove commented-out `import os` in `_entry.py`, and use `get_runtime_vars()` accessor instead of bare `_runtime_vars` in `_trio.py`. Also, - freshen `__init__.py` layout docstring for the new per-backend submod structure - update `_spawn.py` + `_trio.py` module docstrings (this commit msg was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code
goodboy
added
the
spawning
Expand the comment block above the `_interpreters` import explaining *why* we use the private C mod over `concurrent.interpreters`: the public API only exposes PEP 734's `'isolated'` config which breaks `msgspec` (missing PEP 684 slot). Add reference links to PEP 734, PEP 684, cpython sources, and the msgspec upstream tracker (jcrist/msgspec#563). Also, - update error msgs in both `_spawn.py` and `_subint.py` to say "3.13+" (matching the actual `_interpreters` availability) instead of "3.14+". - tweak the mod docstring to reflect py3.13+ availability via the private C module. Review: PR #444 (copilot-pull-request-reviewer) #444 (this patch was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code
goodboy
changed the title
.spawn.*
|
Ugh, just realizing the PR descr is yet again not right |
Expand the comment block above the `_interpreters` import explaining *why* we use the private C mod over `concurrent.interpreters`: the public API only exposes PEP 734's `'isolated'` config which breaks `msgspec` (missing PEP 684 slot). Add reference links to PEP 734, PEP 684, cpython sources, and the msgspec upstream tracker (jcrist/msgspec#563). Also, - update error msgs in both `_spawn.py` and `_subint.py` to say "3.13+" (matching the actual `_interpreters` availability) instead of "3.14+". - tweak the mod docstring to reflect py3.13+ availability via the private C module. Review: PR #444 (copilot-pull-request-reviewer) #444 (this patch was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| is happening under the hood with how cpython implements subints. | ||
|
|
||
| * default configuration should encourage state isolation as with | ||
| with subprocs, but explicit public escape hatches to enable rigorously |
There was a problem hiding this comment.
Grammar: duplicated word in “as with with subprocs”. Remove one “with”.
| with subprocs, but explicit public escape hatches to enable rigorously | |
| subprocs, but explicit public escape hatches to enable rigorously |
There was a problem hiding this comment.
🤖 response authored by
claude-code
Fixed — dropped the duplicated "with" on line 27.
📎 fixed in
a7b1ee34
| await proc.wait() | ||
|
|
||
| await debug.maybe_wait_for_debugger( | ||
| child_in_debug=get_runtime_vars().get( |
There was a problem hiding this comment.
debug.maybe_wait_for_debugger(child_in_debug=...) is now using get_runtime_vars() (the current/parent runtime vars) instead of the _runtime_vars argument that is sent to the child in SpawnSpec. This changes behavior vs. the original code and can incorrectly skip (or unnecessarily delay) the debugger-lock wait during teardown. Use the _runtime_vars function parameter (or a derived child_in_debug captured from it) so the check reflects the spawned child’s debug mode.
| child_in_debug=get_runtime_vars().get( | |
| child_in_debug=_runtime_vars.get( |
There was a problem hiding this comment.
🤖 response authored by
claude-code
Good catch — this was a regression introduced during the Phase A extraction of trio_proc out of _spawn.py. Switched back to the _runtime_vars function parameter (the child-SpawnSpec dict the parent constructed) instead of get_runtime_vars() (which returns the parent's live _state). Added an inline NOTE comment so the distinction is hard to regress again.
📎 fixed in
a7b1ee34
| - `._mp`: the stdlib `multiprocessing` backend variants — driven by | ||
| the `mp.context` bound to `_spawn._ctx`: | ||
| * `'mp_spawn'`, | ||
| * `'mp_forkserver'` |
There was a problem hiding this comment.
There’s a trailing whitespace after * 'mp_forkserver' in the docstring list. Please remove it to keep the rendered docs and formatting clean (some linters also flag trailing whitespace).
| * `'mp_forkserver'` | |
| * `'mp_forkserver'` |
There was a problem hiding this comment.
During the Phase A extraction of `trio_proc()` out of
`spawn._spawn` into its own submod, the
`debug.maybe_wait_for_debugger(child_in_debug=...)` call site in
the hard-reap `finally` got refactored from the original
`_runtime_vars.get('_debug_mode', ...)` (the fn parameter — the
dict that was constructed by the *parent* for the *child*'s
`SpawnSpec`) to `get_runtime_vars().get(...)` (a global getter that
returns the *parent's* live `_state`). Those are semantically
different — the first asks "is the child we just spawned in debug
mode?", the second asks "are *we* in debug mode?". Under
mixed-debug-mode trees the swap can incorrectly skip (or
unnecessarily delay) the debugger-lock wait during teardown.
Revert to the fn-parameter lookup and add an inline `NOTE` comment
calling out the distinction so it's harder to regress again.
Deats,
- `spawn/_trio.py`: `child_in_debug=get_runtime_vars().get(...)` →
`child_in_debug=_runtime_vars.get(...)` at the
`debug.maybe_wait_for_debugger(...)` call in the hard-reap block;
add 4-line `NOTE` explaining the parent-vs-child distinction.
- `spawn/__init__.py`: drop trailing whitespace after the
`'mp_forkserver'` docstring bullet.
- `ai/prompt-io/prompts/subints_spawner.md`: drop duplicated `with`
in `"as with with subprocs"` prose (copilot grammar catch).
Review: PR #444 (Copilot)
#444 (review)
(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
goodboy
added
the
the_AIs_are_taking_over
|
One more CI and we gud to go! |
Expand the comment block above the `_interpreters` import explaining *why* we use the private C mod over `concurrent.interpreters`: the public API only exposes PEP 734's `'isolated'` config which breaks `msgspec` (missing PEP 684 slot). Add reference links to PEP 734, PEP 684, cpython sources, and the msgspec upstream tracker (jcrist/msgspec#563). Also, - update error msgs in both `_spawn.py` and `_subint.py` to say "3.13+" (matching the actual `_interpreters` availability) instead of "3.14+". - tweak the mod docstring to reflect py3.13+ availability via the private C module. Review: PR #444 (copilot-pull-request-reviewer) #444 (this patch was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code
Split
spawn._spawninto per-backend submods (Phase A of #379)Motivation
The
tractor/spawn/_spawn.pymodule had grown to 847+ LOC bundlingthe cross-backend spawn-method registry, supervision helpers, AND
both backend-specific process-creation routines (
trio_proc,mp_proc) into a single monolith. Adding a new backend(subinterpreters per #379) would compound that while tangling
per-backend plumbing with shared supervisor machinery.
This branch is strictly Phase A of the subint spawner effort: a
pure modularization that extracts each existing backend's
*_proc()coroutine into its own submodule (_trio.py,_mp.py)while keeping the shared "core" machinery —
SpawnMethodKey,_methodsregistry,new_procdispatcher,hard_kill/soft_kill/proc_waiter, etc. — in_spawn.py. Thetest harness backend-tuple is now driven from
get_args(SpawnMethodKey)so adding a key to theLiteralis allthat's needed for harness support downstream. No semantic change to
the
trioormp_*backends; no new backends added in thispatchset.
Also bundled: bump
msgspecto>=0.21.0(the first release withpy3.14-compat wheels, prep for the subint backend branch that
follows), minor
spawn/docstring + import tidy, and two prompt-IOlog entries documenting the design/Phase-A-impl sessions per NLNet
policy.
Subsequent phases (
_subintbackend scaffold + impl, hard-killaudit,
_subint_forkserverexperiment) live on follow-up branchesstacked on top of this one.
Src of research
The following provide info on why/how this impl makes sense,
fork()can be hacked now?". This PR is the "Phase A —modularize" prerequisite that unblocks the subint backend
work.
concurrent.interpreters— the stdlib API thatdownstream phases will target.
Summary of changes
By chronological commit,
(de78a644) Seed the subint-spawner design task via
ai/prompt-io/prompts/subints_spawner.md— the initial promptfile describing the Phase A/B/C plan that drove this branch.
(b5b05049) Add prompt-IO log pair
(
prompt_io.md+.raw.md) for the design kickoff session —phased plan (A: modularize, B: subint, C: harness), gating Qs on
Python pin / PR strategy, and the concrete Phase A file-split
plan.
(d7ca68cf) Mv
trio_procandmp_procto per-backend submods — the core change of this PR:
trio_proc()→spawn/_trio.py(~290 LOC).mp_proc()→spawn/_mp.py(~234 LOC, reads_spawn._ctx/_spawn._spawn_methodvia module-import toobserve runtime mutations).
_spawn.py(~418 LOC removed) to shared core; bottom-of-module late imports from
_trio/_mpavoid circulardeps while keeping
_methodspopulated.spawn/__init__.pydocstring describing the newlayout.
_testing/pytest.pyvalid-backend tuple fromget_args(SpawnMethodKey)instead of a hardcoded string set.(8d662999) (ae5b63c0) Bump
msgspecdep to
>=0.21.0inpyproject.tomland syncuv.lock— thefirst release with py3.14-compat wheels, prep for the subint
branch.
(e0b8f23c) Add Phase-A impl prompt-IO log pair
(
20260417T035800Z_61a73ba_prompt_io.{md,raw.md}); fix misctypos caught by copilot review.
(f75865fb) Tidy
spawn/subpkg docstrings andimports post-split — cleanup pass on the newly modularized
modules' headers and cross-module imports.
Scopes changed
tractor.spawn._spawntrio_proc()andmp_proc()bodies (moved tosubmods; -418 LOC net).
sys,current_actor,is_main_process,is_root_process,_mp_main,ActorFailure,pretty_struct,pformat).._trioand._mpto keep
_methodspopulated without circular deps.tractor.spawn._trio(new, ~290 LOC) —trio_proc(); thetrio-native subprocess backend, spawns viatrio.lowlevel.open_process().tractor.spawn._mp(new, ~234 LOC) —mp_proc(); themultiprocessingbackends (mp_spawn,mp_forkserver); reads_spawn._ctx/_spawn._spawn_methodvia module-import.tractor.spawn._entry— minor import touch-up after the split.tractor.spawn.__init__— docstring refresh describing thenew per-backend submod layout.
tractor._testing.pytest— importget_argsfromtyping;replace hardcoded
('mp_spawn', 'mp_forkserver', 'trio')assertion with
get_args(SpawnMethodKey)so the test harnesspicks up
SpawnMethodKeyextensions automatically.pyproject.toml/uv.lock— bumpmsgspecto>=0.21.0.ai/prompt-io/(new entries) — design-session prompt file +two prompt-IO log pairs under
ai/prompt-io/claude/for NLNetpolicy compliance.
Future follow up
See tracking issue #445 for follow-up items from
this PR — subsequent phases (
_subintbackend scaffold + impl,hard-kill audit,
_subint_forkserverexperiment) live onfollow-on branches stacked on top of this one.
(this pr content was generated in some part by
claude-code)