Spawner modules: split up subactor spawning backends

ai/prompt-io/claude/20260417T034918Z_9703210_prompt_io.md

@@ -0,0 +1,101 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+session: subints-spawner-design-kickoff
+timestamp: 2026-04-17T03:49:18Z
+git_ref: 9703210
+scope: docs
+substantive: true
+raw_file: 20260417T034918Z_9703210_prompt_io.raw.md
+---
+
+## Prompt
+
+Drive the "first big boi, from GH issue" task seeded by
+`ai/prompt-io/prompts/subints_spawner.md`: design, plan
+and implement sub-interpreter (subint) spawn-backend
+support per issue #379, including (1) modularizing
+`tractor.spawn._spawn` into per-backend submods, (2) a new
+`._subint` backend, and (3) harness parametrization via the
+existing `--spawn-backend` / `start_method` pytest fixture
+in `tractor._testing.pytest`.
+
+Follow-up clarifications from the user (this turn):
+1. Pin `<3.15` on this dev branch and feature-gate subint
+   tests — chose option (a).
+2. Split Phase A (modularization) into its own PR first.
+3. Defer the `fork()`-via-subint hack to a follow-up.
+4. Harness flag is `pytest --spawn-backend <key>` CLI →
+   `start_method` fixture (prompt file updated to match).
+
+## Response summary
+
+Produced a three-phase plan and a concrete Phase A (pure
+modularization) file-split plan for user review; no code
+written yet — the green-light to start Phase A was given
+in this same turn conditional on logging this prompt-io
+entry first.
+
+Phases:
+- **A — modularize** `tractor/spawn/_spawn.py` (847 LOC):
+  keep generic machinery in `_spawn.py`, extract
+  `trio_proc` → `spawn/_trio.py`, `mp_proc` →
+  `spawn/_mp.py`. No pin bump.
+- **B — `_subint` backend**: bump `pyproject.toml`
+  `requires-python` upper to `<3.15`; add `'subint'` to
+  `SpawnMethodKey`; reuse existing UDS transport; shm
+  escape-hatch deferred.
+- **C — harness**: drive the valid-backend tuple in
+  `tractor/_testing/pytest.py:345-349` from
+  `typing.get_args(SpawnMethodKey)`; skip subint tests on
+  Python < 3.14.
+
+Key findings surfaced to the user:
+- `pyproject.toml:12` currently pins `<3.14`; PEP 734
+  `concurrent.interpreters` only ships in 3.14 — the
+  load-bearing constraint.
+- `_testing/pytest.py:345-349` hardcodes valid backends
+  as a string tuple (`'mp_spawn'`, `'mp_forkserver'`,
+  `'trio'`) — should be `get_args(SpawnMethodKey)`.
+- `_testing/pytest.py:228` already imports
+  `try_set_start_method` from `tractor.spawn._spawn` —
+  keeping the `_spawn.py` path as the "core" module
+  avoids breaking external refs during Phase A.
+
+Phase A commit breakdown (each green via `/run-tests`):
+1. Drive `_testing/pytest.py` backend-tuple from
+   `get_args(SpawnMethodKey)` — prep.
+2. Extract `trio_proc` → `spawn/_trio.py`; `_methods`
+   registry imports it.
+3. Extract `mp_proc` → `spawn/_mp.py`; `_methods` imports
+   it.
+4. Tidy `spawn/__init__.py` docstring for the new layout.
+
+Workflow: Phase A runs in a new worktree via `/open-wkt
+spawn_modularize` (branch `wkt/spawn_modularize`) and
+lands as its own PR before Phase B/C returns to the
+current `claude_bun_opus4.7_subint_spawner_backend`
+branch.
+
+## Files changed
+
+None this turn — design/planning only. The user updated
+`ai/prompt-io/prompts/subints_spawner.md` themselves to
+complete the truncated sentence on line 32 (harness-flag
+reference).
+
+Upcoming (Phase A, next turn):
+- `tractor/spawn/_spawn.py` — trimmed to "core"
+- `tractor/spawn/_trio.py` — new; holds `trio_proc()`
+- `tractor/spawn/_mp.py` — new; holds `mp_proc()`
+- `tractor/spawn/__init__.py` — docstring refresh
+- `tractor/_testing/pytest.py` — backend tuple from
+  `get_args(SpawnMethodKey)`
+
+## Human edits
+
+None — committed as generated. User completed the
+truncated sentence on line 32 of
+`ai/prompt-io/prompts/subints_spawner.md` themselves
+*before* this prompt-io log was written; that edit was
+the user's authoring, not an edit to AI output.

ai/prompt-io/claude/20260417T034918Z_9703210_prompt_io.raw.md

@@ -0,0 +1,117 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+timestamp: 2026-04-17T03:49:18Z
+git_ref: 9703210
+diff_cmd: git diff HEAD~1..HEAD
+---
+
+No code was produced in this turn — this is a design/planning
+exchange for subint spawner support per issue #379. Captured
+verbatim below are the two substantive AI responses, which
+together fix the phased plan (A: modularize, B: subint, C:
+harness) and the concrete Phase A file split before any
+implementation begins.
+
+## Turn 1 — initial triage + gating questions
+
+> Read: issue #379, current `spawn/_spawn.py` (847 lines),
+> and pyproject pins.
+>
+> ## Task summary
+> Design/plan/implement subint backend per #379, plus
+> modularize `spawn/_spawn.py` into per-backend submods, plus
+> harness flag for test parametrization.
+>
+> ## Key finding
+> `pyproject.toml:12` pins `requires-python = ">=3.12, <3.14"`
+> — but `concurrent.interpreters` / PEP 734 **only lands in
+> 3.14**. This is the load-bearing design question before
+> coding.
+>
+> ## Proposed phasing (incremental commits)
+> - **Phase A — modularize** `_spawn.py`: extract `trio_proc`
+>   → `spawn/_trio.py`, `mp_proc` → `spawn/_mp.py`, keep
+>   generic helpers (`hard_kill`/`soft_kill`/`new_proc`
+>   dispatcher/`SpawnMethodKey`) in a slim core. Tests green
+>   at each step.
+> - **Phase B — `_subint` backend**: extend `SpawnMethodKey`
+>   with `'subint'`; reuse UDS transport by default; shm is
+>   escape-hatch follow-up.
+> - **Phase C — harness**: add backend-parametrize flag, skip
+>   subint when Python < 3.14.
+>
+> ## Gating questions before I start
+> 1. **Python pin**: bump `<3.15` and feature-gate subint
+>    tests (a), require 3.14 outright (b), or backport shim
+>    (c)? I'd recommend (a).
+> 2. **PR strategy**: one branch, commits per phase (current
+>    `claude_bun_opus4.7_subint_spawner_backend`)? Or split
+>    Phase A as its own PR first?
+> 3. **`fork()`-via-subint hack** from the issue's "Our own
+>    thoughts" section — v1 or follow-up? I'd defer to
+>    follow-up.
+> 4. **Prompt file line 32 is truncated** — what's the
+>    existing harness flag name you had in mind ("using the
+>    existing ___")?
+>
+> Which of these are already decided vs. open?
+
+## User answers (for the raw record)
+
+1. bump `<3.15`, feature-gate — option (a).
+2. split Phase A as its own PR first.
+3. defer `fork()`-via-subint hack.
+4. harness flag is `--spawn-backend` CLI → `start_method`
+   fixture, per `tractor._testing._pytest`. Prompt file
+   updated accordingly.
+
+## Turn 2 — concrete Phase A plan
+
+> Good — prompt file now ends cleanly; harness flag
+> confirmed as `--spawn-backend` CLI → `start_method`
+> fixture (via `_testing/pytest.py:193,355`). Also noted
+> `pytest.py:345-349` hardcodes valid backends as a string
+> tuple — `typing.get_args(SpawnMethodKey)` should drive
+> that list.
+>
+> ## Phase A — concrete plan (separate PR)
+>
+> **File split** in `tractor/spawn/`:
+> - `_spawn.py` stays as "core": keeps `SpawnMethodKey`,
+>   `_spawn_method`, `proc_waiter`, `try_set_start_method`,
+>   `exhaust_portal`, `cancel_on_completion`, `hard_kill`,
+>   `soft_kill`, `new_proc` dispatcher, `_methods`
+>   registry. Kept at this path so `_testing/pytest.py:228`
+>   and other external imports don't break.
+> - `_trio.py` ← `trio_proc()` (lines 448–671, ~224 lines)
+> - `_mp.py` ← `mp_proc()` (lines 672–842, ~171 lines)
+> - `_methods` dict imports the two backend fns from the
+>   new modules.
+>
+> **Commits** (each green via `/run-tests`):
+> 1. Drive `_testing/pytest.py` backend-tuple from
+>    `get_args(SpawnMethodKey)` — tiny prep.
+> 2. Extract `trio_proc` → `spawn/_trio.py`; `_methods`
+>    imports it.
+> 3. Extract `mp_proc` → `spawn/_mp.py`; `_methods`
+>    imports it.
+> 4. Tidy `spawn/__init__.py` docstring to describe new
+>    layout.
+>
+> No pyproject pin bump in Phase A (that's Phase B).
+>
+> **Workflow**: I'd run this in a new worktree via
+> `/open-wkt spawn_modularize` (branch
+> `wkt/spawn_modularize`), then open a PR before returning
+> to the current `claude_bun_opus4.7_subint_spawner_backend`
+> branch for Phase B/C.
+>
+> Green-light to `/open-wkt` and start Phase A?
+
+## User final response
+
+> all this first then yes, green light.
+
+— log this `/prompt-io` entry first, then proceed with
+Phase A in a worktree.

ai/prompt-io/claude/20260417T035800Z_61a73ba_prompt_io.md

@@ -0,0 +1,86 @@
+---
+model: claude-opus-4-7[1m]
+service: claude
+session: subints-spawner-phase-a-impl
+timestamp: 2026-04-17T03:58:05Z
+git_ref: 61a73ba
+scope: code
+substantive: true
+raw_file: 20260417T035800Z_61a73ba_prompt_io.raw.md
+---
+
+## Prompt
+
+Follow-up from the design session logged in
+`20260417T034918Z_9703210_prompt_io.md`: the user
+greenlit Phase A after `/prompt-io` tracking, so
+execute the Phase A "modularize `spawn._spawn`" plan
+end-to-end in a new worktree — four logical steps
+(pytest harness prep, `trio_proc` extraction,
+`mp_proc` extraction, docstring refresh), each green
+via `/run-tests`.
+
+User directives during execution:
+- **Option 3** for the test cadence ("continue A.2–A.4
+  first, then run the full suite once at the end of
+  Phase A").
+- **One commit** for the whole phase ("can't we just
+  commit the whole patch in one commit?") instead of
+  the 3/4-commit split I initially proposed.
+- **Don't pre-draft** commit messages — wait for the
+  user to invoke `/commit-msg` (captured as feedback
+  memory `feedback_no_auto_draft_commit_msgs.md`).
+
+## Response summary
+
+Produced the cohesive Phase A modularization patch,
+landed as commit `61a73bae` (subject: `Mv
+trio_proc`/`mp_proc` to per-backend submods`). Five
+files changed, +565 / -418 lines.
+
+Key pieces of the patch (generated by claude,
+reviewed by the human before commit):
+- `tractor/spawn/_trio.py` — **new**; receives
+  `trio_proc()` verbatim from `_spawn.py`; imports
+  cross-backend helpers back from `._spawn`.
+- `tractor/spawn/_mp.py` — **new**; receives
+  `mp_proc()` verbatim; uses `from . import _spawn`
+  for late-binding access to the mutable `_ctx` /
+  `_spawn_method` globals (design decision made
+  during impl, not the original plan).
+- `tractor/spawn/_spawn.py` — shrunk 847 → 448 LOC;
+  import pruning; bottom-of-module late imports for
+  `trio_proc` / `mp_proc` with a one-line comment
+  explaining the circular-dep reason.
+- `tractor/spawn/__init__.py` — docstring refresh
+  describing the new layout.
+- `tractor/_testing/pytest.py` — the valid-backend
+  set now comes from `typing.get_args(SpawnMethodKey)`
+  so future additions (`'subint'`) don't need harness
+  edits.
+
+## Files changed
+
+See `git diff 61a73ba~1..61a73ba --stat`:
+
+```
+ tractor/_testing/pytest.py |  12 +-
+ tractor/spawn/__init__.py  |  31 +++-
+ tractor/spawn/_mp.py       | 235 ++++++++++++++++++++++++
+ tractor/spawn/_spawn.py    | 413 +-------------------------------
+ tractor/spawn/_trio.py     | 292 ++++++++++++++++++++++++++++
+ 5 files changed, 565 insertions(+), 418 deletions(-)
+```
+
+Validation:
+- import probe + `_methods` wiring check — OK
+- spawn-relevant test subset — 37 passed, 1 skipped
+- full suite — 350 passed, 14 skipped, 7 xfailed, 1
+  xpassed
+
+## Human edits
+
+None — committed as generated by claude (no diff
+between `.claude/git_commit_msg_LATEST.md` and the
+committed body, as far as the assistant could
+observe).