doctor finds and cleans up leaked kernels

[snimu]

prime-agent doctor now finds the IPython kernels and kernel temp dirs that earlier sessions leaked, and doctor --fix cleans them up. On one machine we found hundreds of leaked ipykernel processes and ~1,900 stale prime-agent-kernel-* dirs in $TMPDIR. A separate PR (ENG-5310) prevents new leaks with a parent-death watchdog; this PR is the cleanup and observability layer for what already leaked or still slips through.

What doctor reports

• Stray kernels: processes running <python> -m ipykernel_launcher -f .../prime-agent-kernel-*/connection.json that have been reparented to init (ppid 1). Kernels with a live parent are never flagged — a live parent means the kernel is owned.
• Stale kernel temp dirs: prime-agent-kernel-* dirs in the current $TMPDIR whose connection.json is not referenced by any live process and whose mtime is older than 1 hour (so a kernel mid-startup is never raced).
• Leaked test daemons (eng-4600 fixture processes parented to init): reported as a warning only, never auto-killed — killing arbitrary node test fixtures is risky.

What --fix does

Kills confirmed strays (SIGTERM, then SIGKILL) and removes their temp dirs plus the stale dirs. Safety measures, in order:

• classification requires an anchored full-argv match (a real python binary running ipykernel_launcher with a prime-agent-kernel-* connection path) and exactly ppid 1
• immediately before killing, the process is re-checked (same command, same connection path, still orphaned) to guard against pid reuse
• a temp dir is only removed after the kill is confirmed (process actually gone; EPERM etc. are recorded as skips), and only if it is a direct prime-agent-kernel-* child of the current temp root
• if ps fails, the whole scan fails closed: nothing is classified stale, nothing is removed
• win32 is a no-op

doctor --json gains a kernels section (strays, staleTempDirs, leakedTestDaemons, psUnavailable); human output only prints kernel lines when something was found. Exit codes are unchanged.

Forkserver-backed kernels (bot review round)

Kernels forked from the kernel forkserver keep the template argv (python -c <script> <forkserver-socket>) — os.fork() never rewrites the command line — so their connection paths are invisible in ps and the argv-reference protection above cannot see them. Verified empirically by running the real embedded fork-server script and forking a real kernel: the child's argv contains the prime-agent-forkserver-* socket path but not its own connection dir.

Doctor therefore fails closed around the fork path:

• if ANY process row contains the prime-agent-forkserver- marker (template or forked child — the marker is inherited across fork, reparenting, and forkserver death), the entire stale-temp-dir sweep is skipped and doctor prints why. With no such row (macOS always, since the forkserver is Linux-only) behavior is unchanged.
• orphaned forked kernels (marker row with ppid 1) are reported but never killed. This is a deliberate v1 gap: PR Kernels exit when their owner dies #1559 makes forked kernels watch the forkserver via parent_handle=getppid(), so once it ships, dead-forkserver orphans exit on their own and no kill machinery is needed here.
• before escalating SIGTERM to SIGKILL, the process identity is re-verified with the same ps check used pre-kill, so a recycled pid is never SIGKILLed.
• the ps scans use a 10 MiB buffer so a huge process table doesn't degrade doctor into a silent no-op.

All kernel logic lives in a new file (src/cli/doctor-kernel-reap.ts) with minimal wiring into daemon-ps.ts/public-command.ts, deliberately avoiding the regions PR #1523 touches.

Tested with unit tests (parsers, classifiers, and reap behavior with injected hooks — including declining to kill on failed recheck, refusing to remove foreign-root dirs, the forkserver fail-closed rule, and the pre-SIGKILL identity recheck) and end-to-end runs on macOS: a real orphaned ipykernel + a backdated stale dir were the only things reported and reaped while 21 live session kernels stayed untouched, and a real forkserver-forked kernel with a 2h-old connection dir survived doctor --fix (sweep disabled, kernel alive, dir intact) both while owned and after being orphaned.

LOC: +879 / -10 (feature 483/-10, tests 396, changelog 1).

Fixes ENG-5311 (https://linear.app/primeintellect/issue/ENG-5311/doctor-report-and-reap-stray-kernels-stale-kernel-temp-dirs-and-leaked)

Note

Add stray IPython kernel detection and cleanup to prime-agent doctor

• Adds a new doctor-kernel-reap.ts module that scans ps output for agent-launched IPython kernels orphaned by their parent (ppid=1), classifies stale kernel temp dirs, and detects forkserver-backed kernel processes.
• prime-agent doctor now reports stray kernels and stale temp dirs; doctor --fix kills confirmed orphaned kernels (SIGTERM → SIGKILL with identity recheck) and removes stale temp dirs.
• doctor --json output gains a kernels section with counts and flags alongside the existing daemon listing.
• Kill escalation uses a re-confirm step before SIGKILL to guard against PID reuse between signal and escalation.
• Risk: relies on ps availability and strict command-line pattern matching; on Windows or when ps is unavailable, findings are empty and no cleanup occurs.

^{Macroscope summarized 92bd76a.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f11069c. Configure here.}

[macroscopeapp]

🟡 Medium cli/doctor-kernel-reap.ts:32

Forkserver-backed kernels are never classified as strays, so doctor --fix leaves leaked/orphaned Linux kernel processes running. After os.fork(), these processes retain the forkserver's python -c <FORK_SERVER_SCRIPT> <socket> argv, which does not match KERNEL_COMMAND_PATTERN; extend process parsing to recognize that forkserver argv and associate it with the kernel temp directory.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/coding-agent/src/cli/doctor-kernel-reap.ts around line 32:

Forkserver-backed kernels are never classified as strays, so `doctor --fix` leaves leaked/orphaned Linux kernel processes running. After `os.fork()`, these processes retain the forkserver's `python -c <FORK_SERVER_SCRIPT> <socket>` argv, which does not match `KERNEL_COMMAND_PATTERN`; extend process parsing to recognize that forkserver argv and associate it with the kernel temp directory.

Evidence trail:
packages/coding-agent/src/cli/doctor-kernel-reap.ts:30-66, 84-91, 251-283 @ f11069c1ff1e0cbf04267f35d6eea670b4a0492e
packages/coding-agent/src/core/kernel/fork-server.ts:175-178 @ f11069c1ff1e0cbf04267f35d6eea670b4a0492e
packages/coding-agent/src/core/kernel/fork-server-script.ts:53-79, 117-139 @ f11069c1ff1e0cbf04267f35d6eea670b4a0492e

[snimu]

Closing without merging, by choice rather than because anything is wrong with the code.

Reasoning: PR #1559 fixes the root cause — kernels now exit within seconds when their owner dies (including the forkserver path), so there is nothing left for a reaper to kill; leaked-kernel cleanup is only needed for pre-#1559 leaks, which have already been cleaned up by hand. The stale temp-dir sweep is likewise covered: macOS's dirhelper auto-reaps the files inside those dirs after 3 days, and with #1559 the dirs no longer signal leaked processes.

Meanwhile, safely killing processes and deleting directories is intrinsically expensive code (identity revalidation, fail-closed scans, TOCTOU handling — this PR grew to +879 lines doing it right), and we'd rather not carry that complexity for a problem the prevention layer already solves.

If kernel processes ever accumulate again despite #1559, the right v2 is probably just a small report-only section in doctor — the classifier work here (especially the forkserver-argv findings) is documented in the PR and its review threads for that day. Thanks to the reviewers; the fail-closed analysis here was genuinely good work.