fix(tmux-restore): REFUSE when there is no tmux server, instead of manufacturing one systemd then kills - #1351
Conversation
MEASURED on the reboot of 2026-09-06 — the first actual reboot in 32 days, and
it REFUTED the hypothesis this arc was built on.
WHAT THE DOC PREDICTED, AND WHAT HAPPENED
Predicted: the boot unit races continuum's replay; damage is duplicated or
misplaced windows. Actual: continuum's replay was PERFECT — 54/54 windows,
correct (session, index) set, zero extras, zero missing. There is no
duplication race.
What actually happened is that all 43 `claude --resume` sends were silently
discarded:
18:01:46 boot
18:02:48 tmux-session-restore.service starts boot+62s (OnActiveSec=45s)
18:02:49 43 x "claude --resume" sent boot+63s
18:03:13 resurrect creates ALL 54 pane shells boot+87s <- 24s later
The journal names it outright: `Started tmux child pane <pid> launched by
process 25957`, x54, in one batch, AFTER the sends. Keys delivered to a pane
that is not ready are discarded — zero occurrences of the sent text in any
pane's scrollback, not even as unexecuted text. The unit exited 0 with
Result=success and the operator found 54 bare shells.
🔴 THE DOC'S OWN PROPOSED PROBE WOULD HAVE MISSED THIS. It said "a count well
above the plan's entry count is the race". The count was 54 vs 54.
Proof it is timing and not logic: re-running the identical `restore` seven
minutes later against the same plan relaunched 42 of 43.
THE FIX — TWO LAYERS, BECAUSE EITHER ALONE IS INSUFFICIENT
1. PREVENTION. `wait_for_workspace_to_settle()` blocks until the pane
fingerprint (pane_pid + pane_current_command for every pane) stops changing.
It waits for the OBSERVABLE rather than guessing a duration — a fixed delay
cannot be right, because the gap scales with pane count, disk speed and boot
load, and 45s was wrong by 24s here. An empty fingerprint does NOT count as
settled, or a dead tmux server would read as ready. It returns rather than
raising, so a workspace that never settles still gets a best-effort restore
— but the caller SAYS the wait timed out instead of reporting a clean run.
2. DETECTION. `_verify_sends()` polls each target until it is running claude.
`tmux send-keys` exits 0 for keystrokes that go nowhere, so "sent" was only
ever a claim about this process, never about the workspace — which is
exactly how the unit reported success having started nothing. It polls
rather than sleeping once, because claude's startup scales with the
transcript being resumed. The unit now exits 1 when the resumes did not land,
and exits 1 even when they DID land if the workspace never settled ("lucky,
not correct").
ALSO: THE INSTRUMENT COULD NOT SEE ITS OWN SUBJECT
`tmux-restore-observe.sh` reported rc 1 for 5 misplaced windows and pointed at
the journal, but had no arm for the actual failure: windows and ids were
perfect, so every comparison in it read CLEAN while the workspace was empty.
It now captures `sends_logged` and `claude_panes_live` and reports 🔴 THE
RESUMES DID NOT LAND. Restoring the windows is continuum's job and resuming the
conversations is the unit's; they fail independently and now report
independently. An UNMEASURED pane count is INCONCLUSIVE, never a confident zero.
Verified against the REAL post-capture from this boot: 43 sends / 1 live fires
the arm; 43/43 stays quiet.
The handoff's ANSWERED block, its rank 1 and its rank 3 are rewritten — rank 3
said to order the unit against continuum, which the reboot showed is NOT the
fix, and would have sent the next session to rebuild the wrong thing.
VERIFICATION
118 tests. Mutation-swept, every mutant killed by the test named for it,
control green either side, both files restored byte-identical:
settle-wait treats an empty fingerprint as settled; settle-wait never times
out; verify counts a lost send as landed; verify polls once instead of
polling; the observe resumes-arm neutered; an UNMEASURED pane count treated
as a real zero.
🔴 One mutant was scored INVALID and redone: `if cmd == "claude":` occurs twice
and a single-occurrence replace hit the skip-guard in cmd_restore rather than
the one in _verify_sends, so it was killed by two unrelated tests. Re-targeted
at the correct site it is killed by
test_verify_reports_a_send_that_never_started_claude.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019o7RBCqvjQgnYyVTFm8LTB
Claude-Session-Id: 74c5937e-be33-4cf8-aeaf-349ce46050f2
…by the gate The +7 tests pinning the boot-race fix took `scripts/session-analysis/tests` to 552 collected, which is ABOVE the drift ceiling for a floor of 440: more than 110 of slack, i.e. a whole suite could vanish underneath it with the gate still green. That is the ceiling arm working as designed — zero tests failed. 525 is copied verbatim from the gate's own message (this run's count through the documented rule), never arithmetic done by hand. 🔴 The first attempt at this edit REFUSED and was right to: the string `"scripts/session-analysis/tests|440"` occurs TWICE — once as the array entry, once inside the comment recording the PREVIOUS raise (367 -> 440). A single-occurrence replace would have rewritten the history and left the live floor untouched. Patched by line index with both the target content and the neighbouring comment asserted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019o7RBCqvjQgnYyVTFm8LTB Claude-Session-Id: 74c5937e-be33-4cf8-aeaf-349ce46050f2
Caught by the SANDBOX tier, which passed on the dev host — the two-tier trap exactly as documented. The dev host has a live tmux server; the nix build sandbox has none, and that difference decided the outcome. With no tmux, `pane_fingerprint()` returns empty forever, so `wait_for_workspace_to_settle` burned its FULL 120s timeout (`test_staleness_check_cli_parsing` took 121.25s) and then the "never settled" penalty turned rc 0 into rc 1 — for a plan with ZERO entries. Two real defects, both mine, both in the fix from the previous commit: 1. NO PANES AT ALL is not "not settled yet". There is no workspace to wait for, and blocking a boot for two minutes to learn that is a defect rather than caution. The wait now bails after `no_server_after` (10s) with settled=False — still honest, no longer slow. This matters in production too: a boot where tmux never came up would have hung the unit for 120s. 2. NOTHING TO SEND => NOTHING TO WAIT FOR, AND NOTHING THAT CAN BE LOST. An empty plan now skips the wait entirely, and the unsettled penalty applies only when sends were actually attempted. A restore with no work to do reporting failure is simply wrong. Verified under the sandbox's own condition rather than reasoned about: with a stub `tmux` that always exits 1, the target runs 79 passed in 0.06s — against 121.25s and one failure before the fix. Both defects have their own regression test naming the measurement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019o7RBCqvjQgnYyVTFm8LTB Claude-Session-Id: 74c5937e-be33-4cf8-aeaf-349ce46050f2
…-race Claude-Session-Id: 74c5937e-be33-4cf8-aeaf-349ce46050f2
…-race # Conflicts: # claudedocs/handoff-tmux-restore-chain.md Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
…turing one that dies The 2026-09-06 reboot lost 43 conversations. This branch's first diagnosis -- "the sends landed in panes that were not ready and were DISCARDED" -- was refuted by the journal, and the code, comments and tests built on it are corrected here. What actually happened: on a cold boot nothing else starts tmux, so `cmd_restore`'s own `tmux new-session` created the server, INSIDE the unit's cgroup. The 43 sends were delivered successfully. ExecStart returned, and with `Type=oneshot` / `RemainAfterExit=no` / `KillMode=control-group` systemd tore the cgroup down, taking the server and every claude process with it. The unit reported `Result=success`. The discriminator was `Started tmux child pane N launched by process <pid>` arriving in TWO cohorts -- 17 lines naming the unit's own pid, then 56 lines 24s later naming a different one. Two servers, not one unready one. Confirmed by isolated experiment on a private `-L` socket: without `RemainAfterExit` a oneshot's tmux server is gone once ExecStart returns; with it, it survives. So a duration was never the variable, and the settle-wait could not have fixed it. It is kept as a SECONDARY guard, re-described honestly. - `no_tmux_server_to_restore_into()` + a refusal ahead of the send loop. - The refusal exits 0, deliberately: `OnFailure=notify-failure@%n` bypasses DND, and until the unit triggers on the tmux socket rather than a fixed `OnActiveSec=45s`, "no server" is the STANDING cold-boot state. A non-zero exit there would toast on every boot forever. - Corrects the refuted mechanism prose at all four sites (two in tmux-session-restore.py, two in tmux-restore-observe.sh). - Drops this branch's rewrite of the handoff doc; main's corrected ANSWERED block stands. Audit follow-ups from round 1, all fixed: - `tmux-restore-observe.sh` tested `[ "$live" = UNMEASURED ]` while the emitter writes `UNMEASURED reason=...`, so the arm fell through to an integer comparison against a sentence. Now a `case` prefix match, pinned by a test using the value production actually emits (the existing test used a bare token production never writes, and could not see the bug). - Both exit-1 branches in `cmd_restore` were untested -- mutants survived. Pinned, with a settled/landed control so an unconditional `return 1` fails. - The verdict now prints that `claude_panes_live` is a WHOLE-HOST count. - Four tests assigned `tsr.pane_fingerprint` with no teardown; now monkeypatch. Scope: this stops the destruction and makes the failure legible. It does NOT make cold-boot restore work -- that needs the socket trigger, tracked separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApQvw3A9KbFEUXVjtTAk4j Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
…AN pass, not just an error Measured by reverting the case-prefix fix and reading the output: the failed `[ "$live" -lt "$sends" ]` does not abort. The shell prints `integer expected`, the test evaluates FALSE, and control lands in the `else`, which printed resumes: UNMEASURED reason=no-tmux-server-responding pane(s) running claude vs 43 send(s) logged and returned RC_CLEAN. So the arm whose entire job is to say "I do not know" returned a confident PASS -- worse than the "integer comparison error" the round-1 audit described, and the reason a reassuring zero here was indistinguishable from a working check. Corrects the code comment to state that outcome, and pins the specific shape (the reason string rendered AS a pane count) rather than only the exit code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApQvw3A9KbFEUXVjtTAk4j Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
… REFUSAL exiting 0, main-confli Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
…-race Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
Gate results — naming the tier and the base shaMerged tree:
All four tier/tree combinations are green on 🔴 Read out of ⚠ ⚠ A merged-tree gate is a claim about the tree it ran on. Mutation battery6 mutants, positive control green, each killed by its own named assertion rather than a
The |
… gated green on all four tier/ Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
…-race Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
…ed by content, all three tiers Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
What changed and why
This PR was opened on a refuted diagnosis. A round-1 audit caught it, the
mechanism was re-derived from the journal and confirmed by an isolated
experiment, and this revision replaces the fix, the prose and the tests
accordingly. The old title ("the resumes were sent 24s before the panes
existed") described the wrong mechanism and has been changed.
The refuted story
The only evidence for it was an absence — the sent text appears nowhere in
any pane's scrollback. That observation is equally consistent with a second
story, so it cannot choose between them.
What actually happened
Started tmux child pane N launched by process <pid>arrived in twocohorts: 17 lines at the unit's own timestamp naming the pid the unit
started, then 56 lines 24s later naming a different pid. Two servers, not
one unready one.
On a cold boot nothing else starts tmux, so
cmd_restore's owntmux new-sessioncreated the server — inside the unit's cgroup. The 43sends were delivered successfully. Then
ExecStartreturned and, withType=oneshot/RemainAfterExit=no/KillMode=control-group, systemd torethe cgroup down, taking the server and every claude process with it. The unit
reported
Result=success.Confirmed by experiment on a private
-Lsocket, real server untouched:Type=oneshot, noRemainAfterExitno server running on …Type=oneshot+RemainAfterExit=yesSo a duration was never the variable, and no amount of waiting could have
fixed it.
The fix
no_tmux_server_to_restore_into(), checked before the send loop — becausethe loop's own
tmux new-sessionis the destructive step. With no server,refuse: a server this process creates cannot outlive it.
The refusal exits 0, and that is deliberate. The unit is
OnFailure=notify-failure@%n, and that toast bypasses DND. Until the unittriggers on the tmux socket appearing rather than a fixed
OnActiveSec=45s,"no server" is the standing cold-boot state — a non-zero exit would fire
the alarm on every boot forever, which
nix/home.nixexplicitly calls out asre-breaching the bypass. It logs loudly instead, and says what to re-run.
The settle-wait and
_verify_sendsfrom the original revision are kept,re-described honestly as a secondary guard and a detector rather than as the
fix.
🔴 Scope — what this does NOT do
This stops the destruction and makes the failure legible. It does not make
cold-boot restore work: after this, a cold boot refuses and the operator still
has to re-run the restore once attached. Making it actually restore needs the
socket trigger in
nix/home.nix, which is deliberately not in this PR.RemainAfterExit=yesalone is measured to work and is not the answer —with
KillMode=control-groupthe unit would then own the operator's server,and stopping the unit would kill the workspace.
Audit round-1 follow-ups, all fixed
[ "$live" = UNMEASURED ]never matched production. The emitter writesUNMEASURED reason=no-tmux-server-respondingandgetreturns everythingafter
key=. Measured by reverting the fix: the failed-ltdoes not abort— the shell prints
integer expected, the test evaluates false, and controlfalls into the
else, which printedresumes: UNMEASURED reason=... pane(s) running claude vs 43 send(s)andreturned RC_CLEAN. The path whose whole job is to say "I do not know"
returned a confident pass. Now a
caseprefix match.UNMEASURED, a value production never writes. Added a test using the realemitted string; kept the old one.
cmd_restorewere untested (mutants survived).Pinned, plus a settled-and-landed control so an unconditional
return 1fails.
claude_panes_liveis a whole-host count — includes hand-started panesand ones the unit skipped as already running. The verdict now prints that
caveat, so
live >= sendsis not misread as "every send landed".tsr.pane_fingerprintwith no teardown, leakingmodule state into whatever ran next. Now
monkeypatch.Prose corrections
The refuted mechanism was asserted at four sites, all corrected:
tmux-session-restore.py(settle docstring,_verify_sendsstderr) andtmux-restore-observe.sh(arm comment,RC_RACEmessage). The failuremessage now names the observation ("these panes are not running claude")
rather than a cause it cannot determine.
This branch's rewrite of the handoff doc is dropped.
main's correctedANSWERED block stands; the merge conflict was resolved by taking main's copy
wholesale.
Verification
own named assertion (not a neighbour's): refusal removed; predicate
inverted; refusal exits 1; unsettled penalty dropped;
UNMEASURED*revertedto equality; whole-host caveat removed. Run under
PYTHONDONTWRITEBYTECODE=1.cmd_restoretest monkeypatches the server predicate rather thanreading the real tmux, because the dev-host tier has a server and the nix
sandbox has none — a test that reads the real thing asserts a different
branch in each tier and cannot fail in one of them. This arc has already
shipped two defects through exactly that gap.
and results.