Skip to content

chore(nix/system): retire the staged tmux OOM protection — its premise was refuted - #1464

Merged
ZacxDev merged 2 commits into
mainfrom
chore/retire-tmux-oom-protection
Sep 10, 2026
Merged

chore(nix/system): retire the staged tmux OOM protection — its premise was refuted#1464
ZacxDev merged 2 commits into
mainfrom
chore/retire-tmux-oom-protection

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Sep 9, 2026

Copy link
Copy Markdown
Member

chore(nix/system): retire the staged tmux OOM protection — its premise was refuted

Operator decision, 2026-09-09, after a round-0 (requirements & deletion) audit of
PR #1459. #1459 is closed by this change: it was a real data-loss fix to a script
that should not exist, and fixing it was work on something to be deleted.

WHY IT EXISTED, AND WHY THAT REASON DID NOT SURVIVE

The requirement is attributed to the operator verbatim — "dispatch to fix the
defect, dispatch to protect tmux from oomkiller" — at 2026-09-07 22:17 CDT,
TWENTY-THREE MINUTES after the incident that destroyed 47 live conversations,
while memory exhaustion was still the working diagnosis.

That diagnosis was REFUTED the next day. The cause was an agent running a
wide tmux kill against the shared socket; zero kernel OOM lines for the incident
boot, with a positive control showing the same pattern matches real OOM kills
from 2026-08-28; systemd-oomd not installed; no tmux core. The scary memory
figures originally cited (57.7G/38.2G/29.8G) were retracted — they are systemd
per-scope LIFETIME high-water marks printed at teardown over 7-23h wall clocks,
not concurrent usage.

The audit swept every transcript from 2026-09-07 onward for an operator message
re-confirming the ask against the corrected diagnosis: ONE match, the original.
Positive control on the same scanner: "tmux" matched 40 of 679 short operator
messages in devrc alone, so it can fire. No message anywhere asks for the script
to be applied.

MEASURED: THE PAYLOAD IS INERT AGAINST EVERY OOM THIS HOST HAS RECORDED

OOM census across all five retained boots (2026-07-05 -> now): four events, all
in boot -3, all constraint=CONSTRAINT_MEMCG inside one kubepods burstable pod,
ZERO global (CONSTRAINT_NONE). A memcg OOM selects its victim from inside the
offending cgroup; the tmux server's cgroup is
/user.slice/user-1000.slice/session-3.scope. oom_score_adj on tmux can only
change the outcome of a GLOBAL OOM, so the script could not have altered 4 of 4
observed kills.

Never run, four ways: no tmux-oom-protection import in
/etc/nixos/configuration.nix (rc 1); /etc/nixos/tmux-oom-protection.nix does not
exist; no configuration.nix.bak.tmux-oom, so the wiring branch never executed
even partially; and the live server's oom_score_adj still reads 0.

THE COUNTER-ARGUMENT, RECORDED SO IT IS NOT RE-DERIVED AS A DISCOVERY

The audit found one fact pointing the other way, and it is stronger than any
case the script made for itself. The script's header claimed Claude panes at
4-15 GB "already outrank" tmux, so the change merely formalises the existing
order. That is FALSE as measured 2026-09-09 (load 55, 53 live claude processes):
nothing on the box is at 4-15 GB, the tmux server is 776 MB and the #2 process
by RSS behind k3s-server, and /proc//oom_score reads 668 for the tmux server
and 668 for each of three sampled Claude panes — they TIE. So in a global OOM
oom_score_adj would be the only lever separating them, and tmux dying takes
every pane with it while a pane dying costs one conversation.

The operator weighed that against zero global OOMs in ~66 days of retained
journal, 86.4 GiB of zram, a refuted premise, and 840 lines of never-executed
code, and chose deletion. Recorded here so a future session finds the argument
already made rather than rediscovering it as a reason to rebuild this.

WHAT IS NOT DELETED

The measured cause is fixed and LIVE on both hosts: check_tmux_kill_shared_server
(#1415) denies a wide tmux kill on the shared socket, verified by reproducing the
incident command against the deployed hook. The cgroup-teardown cause is closed
by #1351/#1376. Nothing about this deletion touches either.

THE LEDGER MOVES IN THE SAME COMMIT, BECAUSE IT IS TWO-WAY

_KILL_MENTION_LEDGER and the shell-text scanner's allowlist in
scripts/claude-hooks/tests/test_guard_core.py both named the deleted script. That
ledger fails when its set GROWS or SHRINKS, so leaving the entries behind would
turn the suite red on a file that no longer exists. Verified both directions:
after removing the two entries the three corpus tests pass (including the
scanner's own positive control), and re-injecting a ledger entry for the deleted
path fails with removed: ['nix/system/apply-tmux-oom-protection.sh'] — so the
pass is not a check wired to nothing.

Collected-test floor checked before deleting: scripts/tests collects 13756
against a floor of 13026, and this removes 20, leaving ~710 of headroom.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_013jdbmhCKa6edhTmiADsziR

ZacxDev and others added 2 commits September 9, 2026 15:02
…e was refuted

Operator decision, 2026-09-09, after a round-0 (requirements & deletion) audit of
PR #1459. #1459 is closed by this change: it was a real data-loss fix to a script
that should not exist, and fixing it was work on something to be deleted.

WHY IT EXISTED, AND WHY THAT REASON DID NOT SURVIVE

The requirement is attributed to the operator verbatim — "dispatch to fix the
defect, dispatch to protect tmux from oomkiller" — at 2026-09-07 22:17 CDT,
TWENTY-THREE MINUTES after the incident that destroyed 47 live conversations,
while memory exhaustion was still the working diagnosis.

That diagnosis was REFUTED the next day. The cause was an agent running a
wide tmux kill against the shared socket; zero kernel OOM lines for the incident
boot, with a positive control showing the same pattern matches real OOM kills
from 2026-08-28; systemd-oomd not installed; no tmux core. The scary memory
figures originally cited (57.7G/38.2G/29.8G) were retracted — they are systemd
per-scope LIFETIME high-water marks printed at teardown over 7-23h wall clocks,
not concurrent usage.

The audit swept every transcript from 2026-09-07 onward for an operator message
re-confirming the ask against the corrected diagnosis: ONE match, the original.
Positive control on the same scanner: "tmux" matched 40 of 679 short operator
messages in devrc alone, so it can fire. No message anywhere asks for the script
to be applied.

MEASURED: THE PAYLOAD IS INERT AGAINST EVERY OOM THIS HOST HAS RECORDED

OOM census across all five retained boots (2026-07-05 -> now): four events, all
in boot -3, all `constraint=CONSTRAINT_MEMCG` inside one kubepods burstable pod,
ZERO global (`CONSTRAINT_NONE`). A memcg OOM selects its victim from inside the
offending cgroup; the tmux server's cgroup is
/user.slice/user-1000.slice/session-3.scope. `oom_score_adj` on tmux can only
change the outcome of a GLOBAL OOM, so the script could not have altered 4 of 4
observed kills.

Never run, four ways: no `tmux-oom-protection` import in
/etc/nixos/configuration.nix (rc 1); /etc/nixos/tmux-oom-protection.nix does not
exist; no configuration.nix.bak.tmux-oom, so the wiring branch never executed
even partially; and the live server's oom_score_adj still reads 0.

THE COUNTER-ARGUMENT, RECORDED SO IT IS NOT RE-DERIVED AS A DISCOVERY

The audit found one fact pointing the other way, and it is stronger than any
case the script made for itself. The script's header claimed Claude panes at
4-15 GB "already outrank" tmux, so the change merely formalises the existing
order. That is FALSE as measured 2026-09-09 (load 55, 53 live claude processes):
nothing on the box is at 4-15 GB, the tmux server is 776 MB and the #2 process
by RSS behind k3s-server, and /proc/<pid>/oom_score reads 668 for the tmux server
and 668 for each of three sampled Claude panes — they TIE. So in a global OOM
`oom_score_adj` would be the only lever separating them, and tmux dying takes
every pane with it while a pane dying costs one conversation.

The operator weighed that against zero global OOMs in ~66 days of retained
journal, 86.4 GiB of zram, a refuted premise, and 840 lines of never-executed
code, and chose deletion. Recorded here so a future session finds the argument
already made rather than rediscovering it as a reason to rebuild this.

WHAT IS NOT DELETED

The measured cause is fixed and LIVE on both hosts: `check_tmux_kill_shared_server`
(#1415) denies a wide tmux kill on the shared socket, verified by reproducing the
incident command against the deployed hook. The cgroup-teardown cause is closed
by #1351/#1376. Nothing about this deletion touches either.

THE LEDGER MOVES IN THE SAME COMMIT, BECAUSE IT IS TWO-WAY

`_KILL_MENTION_LEDGER` and the shell-text scanner's allowlist in
scripts/claude-hooks/tests/test_guard_core.py both named the deleted script. That
ledger fails when its set GROWS *or* SHRINKS, so leaving the entries behind would
turn the suite red on a file that no longer exists. Verified both directions:
after removing the two entries the three corpus tests pass (including the
scanner's own positive control), and re-injecting a ledger entry for the deleted
path fails with `removed: ['nix/system/apply-tmux-oom-protection.sh']` — so the
pass is not a check wired to nothing.

Collected-test floor checked before deleting: scripts/tests collects 13756
against a floor of 13026, and this removes 20, leaving ~710 of headroom.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jdbmhCKa6edhTmiADsziR
Claude-Session-Id: 097b404c-db17-4472-bd37-dc90cf8fa675
…dger

Belongs in the previous commit; a cleanup `git checkout --` reverted the edit
after the negative control ran, so that commit deleted the script while the
ledger still named it — which the ledger correctly fails on (it is two-way and
fails on a SHRINK as well as a growth).

Re-verified on this tree: the three corpus tests pass, including the scanner's
own positive control, and re-injecting an entry for the deleted path fails with
`removed: [...]`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jdbmhCKa6edhTmiADsziR
Claude-Session-Id: 097b404c-db17-4472-bd37-dc90cf8fa675
@ZacxDev
ZacxDev merged commit 8221926 into main Sep 10, 2026
1 of 2 checks passed
@ZacxDev
ZacxDev deleted the chore/retire-tmux-oom-protection branch September 10, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant