chore(nix/system): retire the staged tmux OOM protection — its premise was refuted - #1464
Merged
Conversation
…e was refuted Operator decision, 2026-09-09, after a round-0 (requirements & deletion) audit of PR #1459. #1459 is closed by this change: it was a real data-loss fix to a script that should not exist, and fixing it was work on something to be deleted. WHY IT EXISTED, AND WHY THAT REASON DID NOT SURVIVE The requirement is attributed to the operator verbatim — "dispatch to fix the defect, dispatch to protect tmux from oomkiller" — at 2026-09-07 22:17 CDT, TWENTY-THREE MINUTES after the incident that destroyed 47 live conversations, while memory exhaustion was still the working diagnosis. That diagnosis was REFUTED the next day. The cause was an agent running a wide tmux kill against the shared socket; zero kernel OOM lines for the incident boot, with a positive control showing the same pattern matches real OOM kills from 2026-08-28; systemd-oomd not installed; no tmux core. The scary memory figures originally cited (57.7G/38.2G/29.8G) were retracted — they are systemd per-scope LIFETIME high-water marks printed at teardown over 7-23h wall clocks, not concurrent usage. The audit swept every transcript from 2026-09-07 onward for an operator message re-confirming the ask against the corrected diagnosis: ONE match, the original. Positive control on the same scanner: "tmux" matched 40 of 679 short operator messages in devrc alone, so it can fire. No message anywhere asks for the script to be applied. MEASURED: THE PAYLOAD IS INERT AGAINST EVERY OOM THIS HOST HAS RECORDED OOM census across all five retained boots (2026-07-05 -> now): four events, all in boot -3, all `constraint=CONSTRAINT_MEMCG` inside one kubepods burstable pod, ZERO global (`CONSTRAINT_NONE`). A memcg OOM selects its victim from inside the offending cgroup; the tmux server's cgroup is /user.slice/user-1000.slice/session-3.scope. `oom_score_adj` on tmux can only change the outcome of a GLOBAL OOM, so the script could not have altered 4 of 4 observed kills. Never run, four ways: no `tmux-oom-protection` import in /etc/nixos/configuration.nix (rc 1); /etc/nixos/tmux-oom-protection.nix does not exist; no configuration.nix.bak.tmux-oom, so the wiring branch never executed even partially; and the live server's oom_score_adj still reads 0. THE COUNTER-ARGUMENT, RECORDED SO IT IS NOT RE-DERIVED AS A DISCOVERY The audit found one fact pointing the other way, and it is stronger than any case the script made for itself. The script's header claimed Claude panes at 4-15 GB "already outrank" tmux, so the change merely formalises the existing order. That is FALSE as measured 2026-09-09 (load 55, 53 live claude processes): nothing on the box is at 4-15 GB, the tmux server is 776 MB and the #2 process by RSS behind k3s-server, and /proc/<pid>/oom_score reads 668 for the tmux server and 668 for each of three sampled Claude panes — they TIE. So in a global OOM `oom_score_adj` would be the only lever separating them, and tmux dying takes every pane with it while a pane dying costs one conversation. The operator weighed that against zero global OOMs in ~66 days of retained journal, 86.4 GiB of zram, a refuted premise, and 840 lines of never-executed code, and chose deletion. Recorded here so a future session finds the argument already made rather than rediscovering it as a reason to rebuild this. WHAT IS NOT DELETED The measured cause is fixed and LIVE on both hosts: `check_tmux_kill_shared_server` (#1415) denies a wide tmux kill on the shared socket, verified by reproducing the incident command against the deployed hook. The cgroup-teardown cause is closed by #1351/#1376. Nothing about this deletion touches either. THE LEDGER MOVES IN THE SAME COMMIT, BECAUSE IT IS TWO-WAY `_KILL_MENTION_LEDGER` and the shell-text scanner's allowlist in scripts/claude-hooks/tests/test_guard_core.py both named the deleted script. That ledger fails when its set GROWS *or* SHRINKS, so leaving the entries behind would turn the suite red on a file that no longer exists. Verified both directions: after removing the two entries the three corpus tests pass (including the scanner's own positive control), and re-injecting a ledger entry for the deleted path fails with `removed: ['nix/system/apply-tmux-oom-protection.sh']` — so the pass is not a check wired to nothing. Collected-test floor checked before deleting: scripts/tests collects 13756 against a floor of 13026, and this removes 20, leaving ~710 of headroom. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013jdbmhCKa6edhTmiADsziR Claude-Session-Id: 097b404c-db17-4472-bd37-dc90cf8fa675
…dger Belongs in the previous commit; a cleanup `git checkout --` reverted the edit after the negative control ran, so that commit deleted the script while the ledger still named it — which the ledger correctly fails on (it is two-way and fails on a SHRINK as well as a growth). Re-verified on this tree: the three corpus tests pass, including the scanner's own positive control, and re-injecting an entry for the deleted path fails with `removed: [...]`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013jdbmhCKa6edhTmiADsziR Claude-Session-Id: 097b404c-db17-4472-bd37-dc90cf8fa675
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
chore(nix/system): retire the staged tmux OOM protection — its premise was refuted
Operator decision, 2026-09-09, after a round-0 (requirements & deletion) audit of
PR #1459. #1459 is closed by this change: it was a real data-loss fix to a script
that should not exist, and fixing it was work on something to be deleted.
WHY IT EXISTED, AND WHY THAT REASON DID NOT SURVIVE
The requirement is attributed to the operator verbatim — "dispatch to fix the
defect, dispatch to protect tmux from oomkiller" — at 2026-09-07 22:17 CDT,
TWENTY-THREE MINUTES after the incident that destroyed 47 live conversations,
while memory exhaustion was still the working diagnosis.
That diagnosis was REFUTED the next day. The cause was an agent running a
wide tmux kill against the shared socket; zero kernel OOM lines for the incident
boot, with a positive control showing the same pattern matches real OOM kills
from 2026-08-28; systemd-oomd not installed; no tmux core. The scary memory
figures originally cited (57.7G/38.2G/29.8G) were retracted — they are systemd
per-scope LIFETIME high-water marks printed at teardown over 7-23h wall clocks,
not concurrent usage.
The audit swept every transcript from 2026-09-07 onward for an operator message
re-confirming the ask against the corrected diagnosis: ONE match, the original.
Positive control on the same scanner: "tmux" matched 40 of 679 short operator
messages in devrc alone, so it can fire. No message anywhere asks for the script
to be applied.
MEASURED: THE PAYLOAD IS INERT AGAINST EVERY OOM THIS HOST HAS RECORDED
OOM census across all five retained boots (2026-07-05 -> now): four events, all
in boot -3, all
constraint=CONSTRAINT_MEMCGinside one kubepods burstable pod,ZERO global (
CONSTRAINT_NONE). A memcg OOM selects its victim from inside theoffending cgroup; the tmux server's cgroup is
/user.slice/user-1000.slice/session-3.scope.
oom_score_adjon tmux can onlychange the outcome of a GLOBAL OOM, so the script could not have altered 4 of 4
observed kills.
Never run, four ways: no
tmux-oom-protectionimport in/etc/nixos/configuration.nix (rc 1); /etc/nixos/tmux-oom-protection.nix does not
exist; no configuration.nix.bak.tmux-oom, so the wiring branch never executed
even partially; and the live server's oom_score_adj still reads 0.
THE COUNTER-ARGUMENT, RECORDED SO IT IS NOT RE-DERIVED AS A DISCOVERY
The audit found one fact pointing the other way, and it is stronger than any
case the script made for itself. The script's header claimed Claude panes at
4-15 GB "already outrank" tmux, so the change merely formalises the existing
order. That is FALSE as measured 2026-09-09 (load 55, 53 live claude processes):
nothing on the box is at 4-15 GB, the tmux server is 776 MB and the #2 process
by RSS behind k3s-server, and /proc//oom_score reads 668 for the tmux server
and 668 for each of three sampled Claude panes — they TIE. So in a global OOM
oom_score_adjwould be the only lever separating them, and tmux dying takesevery pane with it while a pane dying costs one conversation.
The operator weighed that against zero global OOMs in ~66 days of retained
journal, 86.4 GiB of zram, a refuted premise, and 840 lines of never-executed
code, and chose deletion. Recorded here so a future session finds the argument
already made rather than rediscovering it as a reason to rebuild this.
WHAT IS NOT DELETED
The measured cause is fixed and LIVE on both hosts:
check_tmux_kill_shared_server(#1415) denies a wide tmux kill on the shared socket, verified by reproducing the
incident command against the deployed hook. The cgroup-teardown cause is closed
by #1351/#1376. Nothing about this deletion touches either.
THE LEDGER MOVES IN THE SAME COMMIT, BECAUSE IT IS TWO-WAY
_KILL_MENTION_LEDGERand the shell-text scanner's allowlist inscripts/claude-hooks/tests/test_guard_core.py both named the deleted script. That
ledger fails when its set GROWS or SHRINKS, so leaving the entries behind would
turn the suite red on a file that no longer exists. Verified both directions:
after removing the two entries the three corpus tests pass (including the
scanner's own positive control), and re-injecting a ledger entry for the deleted
path fails with
removed: ['nix/system/apply-tmux-oom-protection.sh']— so thepass is not a check wired to nothing.
Collected-test floor checked before deleting: scripts/tests collects 13756
against a floor of 13026, and this removes 20, leaving ~710 of headroom.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_013jdbmhCKa6edhTmiADsziR