fix(custody): reclaim a hold whose holder PROCESS IS DEAD, without waiting out the TTL (model_performance-oy4)

[Brian Krabach (bkrabach)]

The item's three candidates are all falsified — and its arithmetic is built on a wrong death time

model_performance-oy4 asks whether the reap sweep is (a) not running,
(b) computing staleness from the wrong field, or (c) using a TTL far
larger than the documented 900s. The answer, from the live events table for
model_performance-h6v (committed at
docs/lanes/oy4-dead-holder-reclaim/evidence/h6v-forensic-timeline.txt), is
none of them:

time (UTC)	what the record shows
07:03:16	agent-spark-1-563997 claims, takes custody
07:03:17 → 07:41:36	20 renewals on a metronome-regular 120s cadence
—	then nothing — the 07:43:36 renewal never happened
07:47 / 07:50 / 07:51 / 07:56	successor work_claim refused ×4
07:52	work_stats reports held_stale: 0, oldest: null
07:57:51	hand-run unclaim releases it

Last renewal 07:41:36Z + 900s ⇒ first reclaim-eligible at 07:56:36Z.
So all four refusals were correct — the last landed 45 seconds inside
the TTL — and held_stale: 0 was correct (623s of silence against 900s).
The holder actually died ~10 minutes later than the item's "~07:33Z" estimate;
the 19- and 23-minute figures are measured from that wrong start. The sweep was
running, the field was right, the TTL was 900s.

The real defect: the one fact that settled the matter — the holder's
process was gone — sat unread. Custody has carried pid and host since it
was designed (custody.py:89-96) and nothing has ever read them for a
decision. Liveness was inferred from silence, never observed from the holder,
so a lane whose process dies is stranded for the rest of its 900s TTL plus up
to a 300s sweep interval — up to 20 minutes — on a fact knowable in
microseconds on the same host. In that window a relaunched successor can do
nothing at all: it cannot work_claim (held), cannot work_release (it does
not hold it), cannot work_file (filing requires holding an item). That dead
end forced a hand-run unclaim twice in one day (h6v, 2nx).

The fix

A third path to reclaim-eligible that observes the holder instead of
inferring from its silence, fenced so it can only ever accelerate the TTL and
never take work from a live agent:

1. the record's host must equal this host — a pid on another machine is
   unknowable, never guessed;
2. pid must be a real positive pid;
3. custody must already have been silent for DEAD_HOLDER_MIN_SILENCE_SECONDS
   (default 2 × RENEW_INTERVAL_SECONDS = 240s) before any pid probe is
   consulted at all.

Every unknowable case resolves to not eligible. Fence 3 is what protects an
agent whose pid is simply not addressable from here (a container in its own pid
namespace reporting the same hostname): it keeps renewing, so it never enters
the window where the probe runs. PID reuse is imprecise only in the safe
direction — a recycled pid answers "alive", falling back to the TTL. The probe
is injectable, mirroring heartbeat.evaluate_freshness, so every branch is
testable with no real processes and no real sleeps.

Reclaim latency for a dead holder: up to 1200s → up to 540s.

Fail-before / pass-after

Identical probe on both trees (git stash push -- src/), reproducing the
incident signature verbatim — real dead pid, 400s silence, 900s TTL, default
reap_project:

BEFORE  work_stats:  held=1  held_stale=0  held_stale_oldest_age_seconds=None
        reap_project(default ttl): reclaimed_count=0
        successor work_claim: REFUSED -- issue already claimed by dead-agent

AFTER   work_stats:  held=1  held_stale=1  held_stale_oldest_age_seconds=401.256
        reap_project(default ttl): reclaimed_count=1
          "holder process is dead -- pid 3824385 on host 'spark-1' is not
           running, and custody has been silent 401s (corroboration window
           240s); ttl 900s not yet reached, but the holder is gone"
        successor work_claim: SUCCESS

Full text: docs/lanes/oy4-dead-holder-reclaim/evidence/fail-before-pass-after.txt.

Two further silent misses in the reaper, found while root-causing

Neither caused this incident; both are live bugs in the same function.

1. reap_project read bd's DEFAULT list page. bd.list(include_resolved=False)
   with no limit applies LIST_DEFAULT_LIMIT (50), ordered priority ASC, created_at DESC, id ASC — so a held item outside that first page was
   invisible to the reaper permanently, with nothing reporting the skip. Now
   bd.list(status="held", limit=0). Ruled out as this incident's cause by
   measurement: model_performance held 20–22 non-closed items across the
   window, and h6v ranked 2nd–4th. Pinned on the call shape, because the
   outcome looks identical on any project small enough to fit.
2. One unreleasable item shadowed the whole queue. No per-item guard, so the
   first release that raised propagated out, reap_sweep caught it per
   project, and every remaining stale hold went unreaped — deterministically, on
   every sweep after it, while the sweep still recorded itself completed. Now
   isolated per item and reported in failed.

doctor now tells "the loop is turning" from "the loop is doing anything"

reap_sweep catches every per-project exception into its return value (correct
— one broken project must not abort the sweep), and reap_loop discarded that
value before stamping a completed heartbeat. A sweep that failed on every
project recorded the same heartbeat as a healthy one, so sweeps.alive — and
work_tracker_status's running_healthy on top of it — read identically either
way. This is the gap oy4 names, and it is real even though it is not what bit
h6v.

The reap heartbeat now carries the sweep's outcome (projects, reclaimed,
failed_projects), reap_loop logs failed projects by name, and doctor gains
sweeps.reclaiming (heartbeat.evaluate_reclaiming, pure and fully
unit-tested), which FAILS naming every project the last sweep failed on.

Deliberate honesty note: a heartbeat from a supervisor older than this change
carries no failed_projects key, and that case reports unknown, never
0 failed. Claiming "nothing failed" from a record that never carried the field
would invent exactly the reassurance this check exists to stop being invented.
That is what the live box reports today; it clears on the next service restart.

Verdict on model_performance-c0e: genuinely distinct, and already fixed

	oy4	c0e
site	custody.reclaim_eligible	Beads.resolve's custody fence
proxy relied on	last-renewal timestamp	item status (if current.status == "held")
direction	a live successor wrongly prevented from acting	a stale holder wrongly permitted to act
when	reclaim never happens	after a reclaim happens

They rhyme — both read custody state from a proxy rather than from the custody
record — but they are different proxies at different call sites failing in
opposite directions, and neither fix touches the other's code. Two defects;
this PR fixes only oy4. And c0e's named failing test passes on this
tree: the modules suite is 119 passed / 0 failed, including
test_reap_recovery.py::test_explicit_resolve_refusal_after_reap_clears_held_and_allows_new_claim.
c0e was filed 03:53Z; PR #68 (the post-reclaim fence, re-keyed on custody
identity) landed afterwards and closed it. c0e's second half — a ledger-note
correction on CCV1-023 — is a different subject and is untouched here.

Tiers, by name

tier	result
pytest tests/unit	881 passed, 0 failed
pytest -m integration tests/integration	374 passed, 3 skipped, 0 failed
pytest -m cli tests/cli	88 passed, 1 failed — test_doctor_quick_succeeds_against_the_real_installed_bd, which is model_performance-jyg, not this branch
pytest ledger/checks	26 passed
python -m ledger.checks.mutation_harness	proven 15 / 15, none unproven
pytest modules/tool-work-tracker/tests (tier 5)	119 passed, 0 failed
ruff check . / ruff format --check .	clean, 163 files
pyright src tests	0 errors, 0 warnings
doctor	All 38 assumptions hold — MEASURED, not computed

The one cli failure was verified rather than assumed: the doctor output inside
it shows a single [FAIL], on sweeps.alive — "no heartbeat ever recorded for
the reap sweep loop", the isolated test root having none, exactly as jyg
describes. The new check appears one line below as [PASS] sweeps.reclaiming skipped -- no reap heartbeat recorded yet, correctly declining to pile a second
red line on a root cause already reported. Captured at
docs/lanes/oy4-dead-holder-reclaim/evidence/cli-tier-jyg-failure.txt.

doctor 37 → 38, read off the tool on this branch, never computed;
AGENTS.md's two sites are updated and the run is committed at
docs/lanes/oy4-dead-holder-reclaim/evidence/doctor-measured.txt.

New tests

• tests/unit/test_custody_dead_holder.py — 19: the acceleration, all three
  fences, live-holder safety, pid-reuse direction, probe ordering.
• tests/unit/test_sweeps_reclaiming.py — 17: sweep_failures,
  evaluate_reclaiming (including the older-supervisor unknown case), and the
  doctor wiring, plus a case where a heartbeat is FRESH and the sweep is still
  proven not to be reclaiming.
• tests/integration/test_dead_holder_reclaim.py — 7: real bd, real dolt, a
  real dead pid (a subprocess started and reaped), plus the two reaper
  silent-miss regressions.
• modules/tool-work-tracker/tests/test_dead_holder_successor_claim.py — 4: the
  agent seam end to end. Deliberately not ttl_seconds=0, the shortcut every
  other reap test in that suite uses — with ttl 0 every hold is stale and a
  dead-holder bug hides completely.

Spend: $0.00 against a $0.00 authority. Lane note:
docs/lanes/oy4-dead-holder-reclaim/DONE-NOTE.md.

[Brian Krabach (bkrabach)]

Manager verification — the refutation HOLDS. I filed this item on a wrong premise, and this lane proved it. Merging.

Head f8b971b, base 9d61e42f = current origin/main (#70/#72/#75 in).

I re-derived the timeline myself from the live events table (read-only), and it falsifies my item

custody renewal writes by agent-spark-1-563997: n=20
  first 2026-09-03 00:03:17  last 2026-09-03 00:41:36  (server-local PDT)
  cadence min/max = 120/122s   <- metronome-regular, then it simply stops
  -> last renewal in UTC          : 07:41:36Z
  -> reclaim-eligible (TTL 900s)  : 07:56:36Z

  claim 07:47Z -> REFUSED CORRECTLY      claim 07:51Z -> REFUSED CORRECTLY
  claim 07:50Z -> REFUSED CORRECTLY      claim 07:56Z -> REFUSED CORRECTLY

All four refused successor claims were refused correctly, the last one by well under a minute, and the held_stale: 0 reading at 07:52Z was correct too. My item asserted a 23-minute stranding against a 900s TTL; that arithmetic was anchored on the lane's log going quiet at ~07:33, but custody kept renewing until 07:41:36. All three of my filed candidates — (a) sweep not running, (b) staleness from the wrong field, (c) a different TTL — are falsified. Reporting any of them would have been a fabricated root cause.

The real defect, confirmed at file:line on main

Custody has carried pid (custody.py:90) and host (:91), parsed on every load (:105-106) and written by every take_custody — and reclaim_eligible, whose own docstring says it is "the whole reclaim decision, and nothing else decides it", reads neither. On main it has exactly two paths: TTL staleness and the escalation ceiling. Liveness was inferred from silence alone, so a holder whose process is gone waits the full TTL for a fact knowable in microseconds on the same host.

The cross-host safety check — the thing I most wanted to see, and it is right

holder_process_dead is the only place in the system that reads pid/host for a decision, and every unknowable case resolves toward "not dead":

if c.pid <= 0: return False, ""
this_host = host if host is not None else local_host()
if not c.host or c.host != this_host:
    # A pid on another machine says nothing to this one. Never guessed.
    return False, ""
if silence < min_silence: return False, ""   # corroboration window
if is_pid_alive(c.pid): return False, ""

So a foreign host falls back to TTL, exactly as required; same-host + real positive pid + already silent ≥ 2× renewal interval + kernel says dead is the only fast path. And path 1 is evaluated first, so an already-stale hold keeps its existing reason verbatim and costs no probe. This can only ever accelerate a reclaim the TTL would have made anyway (1200s → 540s worst case), never take work from a live agent.

Gates

gate	result
fail-before vs current main	16 failed
unit + ledger	907 passed
modules	119 passed
cli	88 passed, 1 failed ← jyg, pre-existing
ruff check / format	clean, 164 files
doctor	All 38 assumptions hold — MEASURED, and AGENTS.md's two sites updated to 38

Two judgement calls I want to endorse explicitly: the successor-claim test deliberately does not use ttl_seconds=0 (the shortcut every other reap test in that suite uses, which would hide dead-holder bugs entirely) — it runs a real 900s TTL, real 400s silence, real dead pid. And sweeps.reclaiming reports unknown rather than failing for a heartbeat written by a supervisor older than this change: failing would red-line every box until its service restarts, and claiming "0 failed" would be a lie.

Squash + --admin per the base-branch policy.