doctor: detect and repair daemons that lost their ownership record

[snimu]

During a real incident, a daemon supervisor got fully wedged: its socket was still alive and answering the version handshake, but its ownership record had been deleted out from under it. Every actual command it received failed with an ownership-lost error, yet prime-agent doctor reported it as status "current" — and the only way out was hand-replicating internal spawn code or prime-agent shutdown, which kills ALL daemons on the machine.

What doctor now says and does

Detection (doctor, ps): doctor cross-checks each healthy-looking daemon's hello identity (generation + pid + process start id) against the supervisor ownership registry, using purely local reads — no daemon commands, since a wedged daemon can't serve them anyway. A daemon whose owner record is missing or was replaced is reported with the new status ownership-lost (human table + --json), plus a remedy footer:

! /path/to/daemon.sock: supervisor lost its ownership record — run "prime-agent doctor --fix" to restart it (sessions are preserved)

No false positives on daemons that are starting up: the owner record is written before listen(), so a daemon that hasn't written its record yet can never answer a probe. Stopping daemons fail a final pid/start-id liveness re-check; replaced daemons fail the generation match; old/foreign daemons without a hello generation are skipped.

Repair (doctor --fix): restarts only the affected daemon, not the whole fleet:

1. proves the daemon belongs to the doctor's agent dir via its per-generation snapshot-cache state dir (declines safely otherwise — repairing a foreign-agent-dir daemon would restart it under the wrong agent dir and orphan its workers);
2. takes the daemon shutdown admission and re-verifies the exact same wedged instance under it (no TOCTOU kill of a replacement);
3. kills the supervisor, confirms process exit after SIGKILL before touching the socket, holds admission across kill/socket cleanup/startup-fence wait so no concurrent autostart can bind the socket mid-kill;
4. relaunches on the same socket via the standard launch path; workers and their sessions are re-adopted by the existing adoption machinery.

If the kill or relaunch fails, doctor reports honestly and leaves the system no worse (nothing is unlinked or spawned after a failed kill).

Messages: ownership-lost errors now name the cause (released vs record missing on disk vs record replaced by another owner), and the raw proper-lockfile Lock file is already being held on the socket lease is wrapped to name the incumbent supervisor (pid + generation).

Not included (deliberately): self-heal in assertCurrent (rewriting a deleted owner record). Post-#1449 the registry lives in ~/.prime/supervisor-owners and is not OS-reaped, so external deletion is implausible; self-heal would also fight doctor --fix (a supervisor rewriting its record between detection and repair) and let a supervisor resurrect ownership an operator deliberately revoked. Both the plan and the reviewer signed off on leaving it out.

Checks

• npm run check: pass
• Focused tests: daemon-ps, daemon-supervisor-ownership, public-command, daemon-ps-format, daemon-socket — 5 files, 85 tests, all pass (new coverage: detection true/false cases incl. young-daemon and shutdown-race guards, repair call order under admission, foreign-agent-dir decline, survive-SIGKILL abort, ELOCKED contract, ownership-reason texts)
• E2E on a real isolated daemon (isolated agent dir + --daemon-socket): started daemon + session, deleted its owner record, doctor reported ownership-lost with remedy, doctor --fix killed pid 11323 and relaunched on the same socket (new pid 12570), owner record re-created, the session survived with the same worker pid (re-adoption confirmed), and a second healthy control daemon was untouched (same pid before/after). Evidence bundle: /tmp/prime-agent-features/eng-5302-e2e/

LOC: +660/−29 (feature code +290/−26 in daemon-ps, daemon-ps-format, daemon-supervisor, daemon-supervisor-ownership; tests +340/−3; changelog +1).

Fixes ENG-5302 (https://linear.app/primeintellect/issue/ENG-5302/doctor-must-detect-and-repair-daemon-ownership-loss)

---

Note

Medium Risk
Repair force-kills a live supervisor and unlinks its socket. Guards (admission, pid/start-id recheck, agent-dir match) reduce TOCTOU risk, but a failed relaunch still leaves the daemon down until autostart.

Overview
doctor / ps now flag daemons whose supervisor is still answering hello but whose on-disk ownership record is missing or replaced (ownership-lost), instead of reporting them as healthy. doctor --fix restarts only those daemons on the same socket so workers/sessions can be re-adopted.

Detection compares hello generation + pid + start id against the local ownership registry (no daemon commands). Repair holds shutdown admission across kill, socket cleanup, and startup-fence wait, then relaunches; it declines daemons whose snapshot-cache lives under a different agent dir, and will not unlink the socket if SIGKILL does not confirm exit. Ownership-lost and socket-lock errors now name the cause and incumbent owner.

^{Reviewed by Cursor Bugbot for commit b4c652d. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Detect and repair daemons that lost their supervisor ownership record in doctor

• Adds a new ownership-lost daemon status, detected by cross-checking the daemon's reported supervisor generation against the on-disk ownership registry via detectDaemonOwnershipLost in daemon-ps.ts.
• Introduces repairOwnershipLostDaemon, a session-preserving restart flow that force-kills the wedged supervisor, removes the socket, and relaunches — with PID-reuse fencing and admission scoping throughout.
• planReap now schedules a restart action for ownership-lost daemons, including the default daemon, instead of skipping them.
• forceKillDaemon now returns a boolean confirming the target is gone and accepts an optional expectedProcessStartId to avoid signaling a reused PID.
• ps output appends a per-daemon repair hint (prime-agent doctor --fix) for each ownership-lost entry, rendered in red.
• Risk: the repair path force-kills the wedged supervisor process; if autostart does not recover it, the operator must restart manually.

^{Macroscope summarized b4c652d.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 676e577. Configure here.}