Skip to content

fix(tmux-restore): REFUSE when there is no tmux server, instead of manufacturing one systemd then kills - #1351

Merged
ZacxDev merged 9 commits into
mainfrom
fix/tmux-restore-boot-race
Sep 7, 2026
Merged

fix(tmux-restore): REFUSE when there is no tmux server, instead of manufacturing one systemd then kills#1351
ZacxDev merged 9 commits into
mainfrom
fix/tmux-restore-boot-race

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Sep 7, 2026

Copy link
Copy Markdown
Member

What changed and why

This PR was opened on a refuted diagnosis. A round-1 audit caught it, the
mechanism was re-derived from the journal and confirmed by an isolated
experiment, and this revision replaces the fix, the prose and the tests
accordingly. The old title ("the resumes were sent 24s before the panes
existed") described the wrong mechanism and has been changed.

The refuted story

The unit fired at boot+63s and sent 43 claude --resume lines; resurrect
created every pane shell at boot+87s, so the sends landed in panes that were
not ready and were DISCARDED.

The only evidence for it was an absence — the sent text appears nowhere in
any pane's scrollback. That observation is equally consistent with a second
story, so it cannot choose between them.

What actually happened

Started tmux child pane N launched by process <pid> arrived in two
cohorts
: 17 lines at the unit's own timestamp naming the pid the unit
started, then 56 lines 24s later naming a different pid. Two servers, not
one unready one.

On a cold boot nothing else starts tmux, so cmd_restore's own
tmux new-session created the server — inside the unit's cgroup. The 43
sends were delivered successfully. Then ExecStart returned and, with
Type=oneshot / RemainAfterExit=no / KillMode=control-group, systemd tore
the cgroup down, taking the server and every claude process with it. The unit
reported Result=success.

Confirmed by experiment on a private -L socket, real server untouched:

unit config server after ExecStart returns
Type=oneshot, no RemainAfterExit goneno server running on …
Type=oneshot + RemainAfterExit=yes survives

So a duration was never the variable, and no amount of waiting could have
fixed it.

The fix

no_tmux_server_to_restore_into(), checked before the send loop — because
the loop's own tmux new-session is the destructive step. With no server,
refuse: a server this process creates cannot outlive it.

The refusal exits 0, and that is deliberate. The unit is
OnFailure=notify-failure@%n, and that toast bypasses DND. Until the unit
triggers on the tmux socket appearing rather than a fixed OnActiveSec=45s,
"no server" is the standing cold-boot state — a non-zero exit would fire
the alarm on every boot forever, which nix/home.nix explicitly calls out as
re-breaching the bypass. It logs loudly instead, and says what to re-run.

The settle-wait and _verify_sends from the original revision are kept,
re-described honestly as a secondary guard and a detector rather than as the
fix.

🔴 Scope — what this does NOT do

This stops the destruction and makes the failure legible. It does not make
cold-boot restore work: after this, a cold boot refuses and the operator still
has to re-run the restore once attached. Making it actually restore needs the
socket trigger in nix/home.nix, which is deliberately not in this PR.

RemainAfterExit=yes alone is measured to work and is not the answer —
with KillMode=control-group the unit would then own the operator's server,
and stopping the unit would kill the workspace.

Audit round-1 follow-ups, all fixed

  • [ "$live" = UNMEASURED ] never matched production. The emitter writes
    UNMEASURED reason=no-tmux-server-responding and get returns everything
    after key=. Measured by reverting the fix: the failed -lt does not abort
    — the shell prints integer expected, the test evaluates false, and control
    falls into the else, which printed
    resumes: UNMEASURED reason=... pane(s) running claude vs 43 send(s) and
    returned RC_CLEAN. The path whose whole job is to say "I do not know"
    returned a confident pass. Now a case prefix match.
  • The existing test could not see that bug — its fixture used a bare
    UNMEASURED, a value production never writes. Added a test using the real
    emitted string; kept the old one.
  • Both exit-1 branches in cmd_restore were untested (mutants survived).
    Pinned, plus a settled-and-landed control so an unconditional return 1
    fails.
  • claude_panes_live is a whole-host count — includes hand-started panes
    and ones the unit skipped as already running. The verdict now prints that
    caveat, so live >= sends is not misread as "every send landed".
  • Four tests assigned tsr.pane_fingerprint with no teardown, leaking
    module state into whatever ran next. Now monkeypatch.

Prose corrections

The refuted mechanism was asserted at four sites, all corrected:
tmux-session-restore.py (settle docstring, _verify_sends stderr) and
tmux-restore-observe.sh (arm comment, RC_RACE message). The failure
message now names the observation ("these panes are not running claude")
rather than a cause it cannot determine.

This branch's rewrite of the handoff doc is dropped. main's corrected
ANSWERED block stands; the merge conflict was resolved by taking main's copy
wholesale.

Verification

  • Mutation battery, 6 mutants, positive control green, each killed by its
    own
    named assertion (not a neighbour's): refusal removed; predicate
    inverted; refusal exits 1; unsettled penalty dropped; UNMEASURED* reverted
    to equality; whole-host caveat removed. Run under
    PYTHONDONTWRITEBYTECODE=1.
  • Every new cmd_restore test monkeypatches the server predicate rather than
    reading the real tmux, because the dev-host tier has a server and the nix
    sandbox has none — a test that reads the real thing asserts a different
    branch in each tier and cannot fail in one of them. This arc has already
    shipped two defects through exactly that gap.
  • Both tiers run on the merged tree; see the comment below for the base sha
    and results.

ZacxDev and others added 7 commits September 6, 2026 18:59
MEASURED on the reboot of 2026-09-06 — the first actual reboot in 32 days, and
it REFUTED the hypothesis this arc was built on.

WHAT THE DOC PREDICTED, AND WHAT HAPPENED

Predicted: the boot unit races continuum's replay; damage is duplicated or
misplaced windows. Actual: continuum's replay was PERFECT — 54/54 windows,
correct (session, index) set, zero extras, zero missing. There is no
duplication race.

What actually happened is that all 43 `claude --resume` sends were silently
discarded:

  18:01:46  boot
  18:02:48  tmux-session-restore.service starts    boot+62s (OnActiveSec=45s)
  18:02:49  43 x "claude --resume" sent            boot+63s
  18:03:13  resurrect creates ALL 54 pane shells   boot+87s   <- 24s later

The journal names it outright: `Started tmux child pane <pid> launched by
process 25957`, x54, in one batch, AFTER the sends. Keys delivered to a pane
that is not ready are discarded — zero occurrences of the sent text in any
pane's scrollback, not even as unexecuted text. The unit exited 0 with
Result=success and the operator found 54 bare shells.

🔴 THE DOC'S OWN PROPOSED PROBE WOULD HAVE MISSED THIS. It said "a count well
above the plan's entry count is the race". The count was 54 vs 54.

Proof it is timing and not logic: re-running the identical `restore` seven
minutes later against the same plan relaunched 42 of 43.

THE FIX — TWO LAYERS, BECAUSE EITHER ALONE IS INSUFFICIENT

1. PREVENTION. `wait_for_workspace_to_settle()` blocks until the pane
   fingerprint (pane_pid + pane_current_command for every pane) stops changing.
   It waits for the OBSERVABLE rather than guessing a duration — a fixed delay
   cannot be right, because the gap scales with pane count, disk speed and boot
   load, and 45s was wrong by 24s here. An empty fingerprint does NOT count as
   settled, or a dead tmux server would read as ready. It returns rather than
   raising, so a workspace that never settles still gets a best-effort restore
   — but the caller SAYS the wait timed out instead of reporting a clean run.

2. DETECTION. `_verify_sends()` polls each target until it is running claude.
   `tmux send-keys` exits 0 for keystrokes that go nowhere, so "sent" was only
   ever a claim about this process, never about the workspace — which is
   exactly how the unit reported success having started nothing. It polls
   rather than sleeping once, because claude's startup scales with the
   transcript being resumed. The unit now exits 1 when the resumes did not land,
   and exits 1 even when they DID land if the workspace never settled ("lucky,
   not correct").

ALSO: THE INSTRUMENT COULD NOT SEE ITS OWN SUBJECT

`tmux-restore-observe.sh` reported rc 1 for 5 misplaced windows and pointed at
the journal, but had no arm for the actual failure: windows and ids were
perfect, so every comparison in it read CLEAN while the workspace was empty.
It now captures `sends_logged` and `claude_panes_live` and reports 🔴 THE
RESUMES DID NOT LAND. Restoring the windows is continuum's job and resuming the
conversations is the unit's; they fail independently and now report
independently. An UNMEASURED pane count is INCONCLUSIVE, never a confident zero.

Verified against the REAL post-capture from this boot: 43 sends / 1 live fires
the arm; 43/43 stays quiet.

The handoff's ANSWERED block, its rank 1 and its rank 3 are rewritten — rank 3
said to order the unit against continuum, which the reboot showed is NOT the
fix, and would have sent the next session to rebuild the wrong thing.

VERIFICATION

118 tests. Mutation-swept, every mutant killed by the test named for it,
control green either side, both files restored byte-identical:
  settle-wait treats an empty fingerprint as settled; settle-wait never times
  out; verify counts a lost send as landed; verify polls once instead of
  polling; the observe resumes-arm neutered; an UNMEASURED pane count treated
  as a real zero.

🔴 One mutant was scored INVALID and redone: `if cmd == "claude":` occurs twice
and a single-occurrence replace hit the skip-guard in cmd_restore rather than
the one in _verify_sends, so it was killed by two unrelated tests. Re-targeted
at the correct site it is killed by
test_verify_reports_a_send_that_never_started_claude.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019o7RBCqvjQgnYyVTFm8LTB
Claude-Session-Id: 74c5937e-be33-4cf8-aeaf-349ce46050f2
…by the gate

The +7 tests pinning the boot-race fix took `scripts/session-analysis/tests` to
552 collected, which is ABOVE the drift ceiling for a floor of 440: more than
110 of slack, i.e. a whole suite could vanish underneath it with the gate still
green. That is the ceiling arm working as designed — zero tests failed.

525 is copied verbatim from the gate's own message (this run's count through the
documented rule), never arithmetic done by hand.

🔴 The first attempt at this edit REFUSED and was right to: the string
`"scripts/session-analysis/tests|440"` occurs TWICE — once as the array entry,
once inside the comment recording the PREVIOUS raise (367 -> 440). A
single-occurrence replace would have rewritten the history and left the live
floor untouched. Patched by line index with both the target content and the
neighbouring comment asserted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019o7RBCqvjQgnYyVTFm8LTB
Claude-Session-Id: 74c5937e-be33-4cf8-aeaf-349ce46050f2
Caught by the SANDBOX tier, which passed on the dev host — the two-tier trap
exactly as documented. The dev host has a live tmux server; the nix build
sandbox has none, and that difference decided the outcome.

With no tmux, `pane_fingerprint()` returns empty forever, so
`wait_for_workspace_to_settle` burned its FULL 120s timeout
(`test_staleness_check_cli_parsing` took 121.25s) and then the
"never settled" penalty turned rc 0 into rc 1 — for a plan with ZERO entries.

Two real defects, both mine, both in the fix from the previous commit:

1. NO PANES AT ALL is not "not settled yet". There is no workspace to wait
   for, and blocking a boot for two minutes to learn that is a defect rather
   than caution. The wait now bails after `no_server_after` (10s) with
   settled=False — still honest, no longer slow. This matters in production
   too: a boot where tmux never came up would have hung the unit for 120s.

2. NOTHING TO SEND => NOTHING TO WAIT FOR, AND NOTHING THAT CAN BE LOST. An
   empty plan now skips the wait entirely, and the unsettled penalty applies
   only when sends were actually attempted. A restore with no work to do
   reporting failure is simply wrong.

Verified under the sandbox's own condition rather than reasoned about: with a
stub `tmux` that always exits 1, the target runs 79 passed in 0.06s — against
121.25s and one failure before the fix. Both defects have their own regression
test naming the measurement.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019o7RBCqvjQgnYyVTFm8LTB
Claude-Session-Id: 74c5937e-be33-4cf8-aeaf-349ce46050f2
…-race

Claude-Session-Id: 74c5937e-be33-4cf8-aeaf-349ce46050f2
…-race

# Conflicts:
#	claudedocs/handoff-tmux-restore-chain.md

Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
…turing one that dies

The 2026-09-06 reboot lost 43 conversations. This branch's first diagnosis --
"the sends landed in panes that were not ready and were DISCARDED" -- was
refuted by the journal, and the code, comments and tests built on it are
corrected here.

What actually happened: on a cold boot nothing else starts tmux, so
`cmd_restore`'s own `tmux new-session` created the server, INSIDE the unit's
cgroup. The 43 sends were delivered successfully. ExecStart returned, and with
`Type=oneshot` / `RemainAfterExit=no` / `KillMode=control-group` systemd tore
the cgroup down, taking the server and every claude process with it. The unit
reported `Result=success`.

The discriminator was `Started tmux child pane N launched by process <pid>`
arriving in TWO cohorts -- 17 lines naming the unit's own pid, then 56 lines
24s later naming a different one. Two servers, not one unready one. Confirmed
by isolated experiment on a private `-L` socket: without `RemainAfterExit` a
oneshot's tmux server is gone once ExecStart returns; with it, it survives.

So a duration was never the variable, and the settle-wait could not have
fixed it. It is kept as a SECONDARY guard, re-described honestly.

- `no_tmux_server_to_restore_into()` + a refusal ahead of the send loop.
- The refusal exits 0, deliberately: `OnFailure=notify-failure@%n` bypasses
  DND, and until the unit triggers on the tmux socket rather than a fixed
  `OnActiveSec=45s`, "no server" is the STANDING cold-boot state. A non-zero
  exit there would toast on every boot forever.
- Corrects the refuted mechanism prose at all four sites (two in
  tmux-session-restore.py, two in tmux-restore-observe.sh).
- Drops this branch's rewrite of the handoff doc; main's corrected ANSWERED
  block stands.

Audit follow-ups from round 1, all fixed:
- `tmux-restore-observe.sh` tested `[ "$live" = UNMEASURED ]` while the
  emitter writes `UNMEASURED reason=...`, so the arm fell through to an
  integer comparison against a sentence. Now a `case` prefix match, pinned by
  a test using the value production actually emits (the existing test used a
  bare token production never writes, and could not see the bug).
- Both exit-1 branches in `cmd_restore` were untested -- mutants survived.
  Pinned, with a settled/landed control so an unconditional `return 1` fails.
- The verdict now prints that `claude_panes_live` is a WHOLE-HOST count.
- Four tests assigned `tsr.pane_fingerprint` with no teardown; now monkeypatch.

Scope: this stops the destruction and makes the failure legible. It does NOT
make cold-boot restore work -- that needs the socket trigger, tracked
separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ApQvw3A9KbFEUXVjtTAk4j
Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
…AN pass, not just an error

Measured by reverting the case-prefix fix and reading the output: the failed
`[ "$live" -lt "$sends" ]` does not abort. The shell prints `integer expected`,
the test evaluates FALSE, and control lands in the `else`, which printed

  resumes: UNMEASURED reason=no-tmux-server-responding pane(s) running claude vs 43 send(s) logged

and returned RC_CLEAN. So the arm whose entire job is to say "I do not know"
returned a confident PASS -- worse than the "integer comparison error" the
round-1 audit described, and the reason a reassuring zero here was
indistinguishable from a working check.

Corrects the code comment to state that outcome, and pins the specific shape
(the reason string rendered AS a pane count) rather than only the exit code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ApQvw3A9KbFEUXVjtTAk4j
Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
ZacxDev added a commit that referenced this pull request Sep 7, 2026
… REFUSAL exiting 0, main-confli

Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
…-race

Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
@ZacxDev ZacxDev changed the title fix(tmux-restore): the resumes were sent 24s before the panes existed fix(tmux-restore): REFUSE when there is no tmux server, instead of manufacturing one systemd then kills Sep 7, 2026
@ZacxDev

ZacxDev commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

Gate results — naming the tier and the base sha

Merged tree: 83697d30 (branch merged with origin/main at 969f0581). Both nix
derivations were built one at a time — a combined invocation produces false failures
through store contention, and a combined red is not trustworthy.

Tier Derivation / suite Verdict
nix sandbox checks.x86_64-linux.pytests PASSRESULT: PASS, collected 22026 / passed 22023 / failed 0
nix sandbox checks.x86_64-linux.nodetests PASSRESULT: PASS, 5 suites / 41 files / 1449 tests / 0 fail
dev host gate.sh --tier both on 3b348542 PASS — pytest 22023 passed / 0 failed; node 1449 / 0 fail
dev host gate.sh --tier both on 83697d30 PASSGATE: RESULT=PASS, pytest 22023 passed / 0 failed; node 1449 / 0 fail

scripts/session-analysis/tests collected 562 against the floor of 525 this PR
raises.

All four tier/tree combinations are green on 83697d30. gate.sh's exit status
agreed with both runners' own RESULT: lines, so this is not an rc-90 could-not-vouch.

🔴 Read out of nix log, not the exit code. A nix build sends its test output to the
daemon and prints none of it; a derivation that ran zero tests exits 0 identically. Both
verdicts above are the runners' own RESULT: and TOTAL lines.

nix build path:<a worktree> does not work here and fails exit=2 before any test
runs
: a worktree's .git is a FILE pointing at the real git dir, which defeats the
sandbox's git-config isolation. Extract with git archive HEAD | tar -x -C <dir> and build
from there.

A merged-tree gate is a claim about the tree it ran on. main moved again after
83697d30 — including real test changes, not just docs — so whoever merges this should
re-gate the tree the merge actually creates.

Mutation battery

6 mutants, positive control green, each killed by its own named assertion rather than a
neighbour's, run under PYTHONDONTWRITEBYTECODE=1 (a same-length edit landing in the same
second as the last import is invisible to CPython's mtime-in-whole-seconds cache, and the
mutant is scored SURVIVED without ever executing).

Mutant Killed by
refusal block removed test_a_restore_with_no_tmux_server_REFUSES — "waited instead of refusing"
has-session predicate inverted test_the_server_guard_reads_has_session_and_treats_failure_as_absent
refusal returns 1 instead of 0 test_a_restore_with_no_tmux_server_REFUSES — "a standing cold-boot state must not fire the OnFailure toast"
unsettled penalty dropped test_an_unsettled_workspace_exits_1_even_when_every_send_landed
UNMEASURED* reverted to equality test_the_unmeasured_marker_AS_PRODUCTION_SPELLS_IT_is_not_an_integer_comparison
whole-host caveat removed test_the_verdict_says_the_live_pane_count_is_WHOLE_HOST

The UNMEASURED mutant is worth reading: reverting the fix does not produce a loud
error. The shell prints integer expected, the -lt test evaluates false, and control
falls into the else, which rendered the reason string as a pane count and returned
RC_CLEAN. The arm whose entire job is to say "I do not know" returned a confident pass —
worse than the "integer-expected error" the round-1 audit described.

ZacxDev added a commit that referenced this pull request Sep 7, 2026
… gated green on all four tier/

Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
…-race

Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
@ZacxDev
ZacxDev merged commit 9353d95 into main Sep 7, 2026
2 checks passed
@ZacxDev
ZacxDev deleted the fix/tmux-restore-boot-race branch September 7, 2026 23:55
ZacxDev added a commit that referenced this pull request Sep 7, 2026
…ed by content, all three tiers

Claude-Session-Id: 5542cd95-4967-4463-8fe2-0f0a75194e9d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant