chore(nix/system): commit two staged ops scripts that were sitting untracked in the workbench tree

[ZacxDev]

Two scripts existed only in the workbench working tree, where a stray git checkout or a deploy would have deleted them unreported (claude/RULES.md → "Docs/notes written into a working tree are UNSAVED WORK").

nix/system/apply-journald-settings-migration.sh

nixos-26.11 removed services.journald.extraConfig (mkRemovedOptionModule in nixos/modules/system/boot/systemd/journald.nix), so nixos-rebuild switch died on a failed assertion after a nix-channel --update. The script rewrites the block to services.journald.settings.Journal, and:

• refuses unless it matches exactly one extraConfig block, and unless every line inside parses as a journald.conf(5) key
• takes a timestamped backup, then nix-instantiate --parse + nixos-rebuild dry-build, restoring the backup if either fails
• after the switch, prints the generated /etc/systemd/journald.conf and journalctl --disk-usage rather than asserting success
• idempotent

Already applied to the workbench. Verified live, not inferred:

• /etc/systemd/journald.conf → [Journal] / Audit=keep / SystemMaxUse=2G (Audit=keep is 26.11's new explicit default)
• /run/current-system → nixos-system-nixos-26.11pre1068949.dc5d91f84032

Verified before it was run, against a copy — no writes to /etc/nixos: the rewrite parses, config.system.build.toplevel evaluates (assertion gone), and building config.environment.etc."systemd/journald.conf" showed the SystemMaxUse=2G line survives the migration.

scripts/diagnose-nix-disk.sh

The read-only root-partition/inode breakdown used during the disk dig: df/stat -f/dumpe2fs, per-top-level-dir inode counts and apparent sizes, lsof +L1 for deleted-but-held files, /nix/store/.links, swapfile, reserved blocks. Sibling to the existing scripts/diagnose-disk-accounting.sh; no overlap with scripts/cleanup-disk.sh.

Deliberately NOT in this PR

nix/system/apply-nebula-relay.sh and check-nebula-relays.sh were also untracked locally, but those copies are stale orphans — byte-identical to e7f2e45a, an intermediate commit of the branch that merged as #1272. main already carries strictly further-along versions (480 vs 205 lines, and 360 vs 317), including a 🔴 THIS RESTARTS THE MESH warning my copies predate. Committing them would have reverted ~318 lines of that work.

Gate

main moved from 4f49f5dc to 03d7e0ad mid-run and this branch is rebased onto the latter, so the gate is being re-run on that merged tree; result posted below.

[ZacxDev]

Audit round 1 — 2 🔴, 4 🟡, 4 🟢 · all fixed in b11330b5

Full adversarial audit of 85710f1a. Every finding was measured, not reasoned. The two blockers:

🔴 1 — this PR's own handoff doc re-committed a client subdomain literal while describing the gate that bans it. test_no_client_hostnames.py has an empty allowlist, so there is no "it's a quote" exemption. Controls both ways: scan_repo over a tree containing only that doc → 1 hit; same tree with the literal replaced → 0; the current base 03d7e0ad → 18 passed. Introduced by this PR, not pre-existing. The doc now describes the shape instead of pasting the value, and carries the lesson.

🔴 2 — apply-journald-settings-migration.sh had no trap. A Ctrl-C or OOM kill between the rewrite and the dry-build left /etc/nixos/configuration.nix migrated, silently, with the backup path never mentioned again — and nixos-rebuild dry-build on this fleet takes minutes, so the window is not theoretical. It now has the trap finish EXIT shape apply-nebula-relay.sh already sets on main, reporting separately whether the system had been switched.

🟡 4 was the one I'd have missed entirely: scripts/diagnose-nix-disk.sh could not complete a run on either host. Under set -euo pipefail, find | wc -l exits non-zero on a permission-denied entry (dies at section 3 of 10 as non-root), lsof is installed on neither host (section 5), and the laptop's root is nvme0n1p1 not nvme0n1p2 (dies at section 2). set -e prints nothing and the stderr was already discarded — so the operator read two sections with no signal that eight were skipped. Also 🟡 5: the device and the 1.8TB, ~1.4TB used banner were frozen workbench facts; both are derived at run time now.

🟡 6 — the rewriter only matched the ''-block form. The laptop carries services.journald.extraConfig = "SyncIntervalSec=30s"; (measured, on 26.11pre1058091), so it would have refused there while the doc called it host-agnostic. Both forms are handled and tested now.

🟡 7 — verification grepped a hardcoded SystemMaxUse, so a host migrating any other key got "CHECK THIS, the cap may not be applied" on a fully successful run. It now verifies the keys that run migrated.

The rewrite is now a tested module

Parsing moved to scripts/lib/journald_migrate.py, with scripts/tests/test_journald_migrate.py (25 tests). Three guards were watched red against the pre-fix implementation, not merely written:

case	pre-fix	post-fix
SyncIntervalSec=5m\t	emitted "5m<TAB>" — evaluates to a literal tab	"5m\\t" — evaluates to the source's own 4 chars
CRLF config	all 5 CRs stripped file-wide	5 CRs preserved
laptop's one-line form	refused, "found 0"	migrates it

Gate

The gate.sh --tier both run in flight was against 85710f1a and is superseded by these fixes — an audit fix resets the verification gate. Re-running on b11330b5, plus the nix build .#checks.x86_64-linux.{pytests,nodetests} sandbox tier one at a time. Verdict posted here when it lands; not merging before then.

Round 2 delta audit dispatched against 85710f1a..b11330b5.

legend: <from> = the tip THIS round's audit READ · <to> = the head THIS round's FIXES produced.

1. Removed the client subdomain literal from claudedocs/handoff-journald-26-11-migration.md; the gate test now passes on the tree.
2. Added trap finish EXIT to apply-journald-settings-migration.sh restoring the backup on any exit, reporting whether the switch had already happened.
3. Added nixos-rebuild test before switch so an activation failure does not register a generation or rewrite the bootloader.
4. Removed set -e from diagnose-nix-disk.sh and guarded every section, so all 10 are attempted and unavailable ones say why.
5. Derived ROOT_DEV/ROOT_FS/size/used at run time in diagnose-nix-disk.sh instead of hardcoding nvme0n1p2 and a frozen banner.
6. Extended the rewriter to the one-line double-quoted extraConfig form measured on the laptop, in addition to the ''-block.
7. Verification now checks the KEY=VALUE pairs that run migrated against /etc/systemd/journald.conf and fails if any is missing.
8. Backup is taken after nix-instantiate --parse succeeds, so a refusal no longer orphans a .bak in /etc/nixos.
9. CRLF line endings are preserved through the rewrite instead of being converted file-wide.
10. Values are escaped for the Nix double-quoted string they land in (backslash, quote, ${), and re.sub is passed a function so the escaping is not undone by group-reference expansion.
11. Extracted parsing to scripts/lib/journald_migrate.py with 25 tests; three guards watched red against the pre-fix implementation.
12. Deleted the handoff doc's stale "base clone cannot fast-forward" investigation block, which was measured wrong after it was written.

[ZacxDev]

Audit round 2 — 1 🔴, 4 🟡, 6 🟢 · all fixed in 9a172aae

Round 2 was not clean, and its blocker is the textbook shape: round 1's own fix created it.

🔴 NEW-1 — nixos-rebuild test ACTIVATES. It only skips the boot menu. Round 1 added test before switch so an activation failure wouldn't register a generation — but SWITCHED=1 was only set after switch returned. So on any failure in between (including the ordinary case of test succeeding and switch failing), the trap restored the file and printed "The system was never switched, so nothing is running the change" — while journald was running it. The mechanism added to prevent exactly that confusion produced it. Now three states, and the middle one says a reboot reverts it.

🟡 NEW-2 — the verification got narrower than the one it replaced. The original grepped systemd-analyze cat-config, which merges /etc/systemd/journald.conf.d/* and /run/systemd/journald.conf.d/*. Round 1 swapped it for a grep of the single file — so a drop-in overriding a migrated key would leave the setting inert while the check printed ok. Restored, with the single-file path kept as a fallback that names what it can't see.

🟡 NEW-5 — the module guessed on ${, in the one place it claims never to guess. In the source that's a Nix antiquotation; escaping it freezes the literal text. Measured with nix-instantiate --eval: ''SystemMaxUse=${cap}'' with cap = "9G" evaluates to SystemMaxUse=9G; the escaped rewrite evaluates to the literal ${cap}. Nothing downstream caught it — --parse accepts both, and the post-switch check compared against the same un-evaluated text it had printed, so it matched and reported success. Now refused in both forms, with tests. A lone $ still passes.

🟡 NEW-3 — $HOME under sudo is the target user's, so section 8 of the diagnostic walked /root under the very invocation its own header documents. Resolved from SUDO_USER via getent now, and it prints which home it used. ⚠️ Round 2 flagged this as NOT MEASURED — sudo -n needs a password here — so it's inference from sudoers(5) plus nixpkgs not building sudo with --with-always-set-home. The fix is correct either way; one sudo printenv HOME would settle it.

🟡 NEW-4 — --print-keys shared stderr with anything else python writes there, and every line became a key. A stray DeprecationWarning would have become a phantom KEY=VALUE that the post-switch check couldn't find — rolling back a migration that had worked, while telling the operator the wrong thing about the running system.

🟢 NEW-7 — one of my new assertions was vacuous, and round 2 proved it rather than asserting it. assert "\t" not in body cannot fail: an unescaped implementation emits the two characters \ t, never a TAB. Under the drop-the-escape mutant that line passed while the test died on the line above it. Replaced with a negative that can fail, plus a non-vacuity guard on the helper.

Round 2 also retracted one of its own intended findings — it set out to show the header's "restores on any interrupt" overclaimed, measured that bash does run the EXIT trap on SIGTERM, found its SIGINT test invalid, and reported the retraction instead of the finding. The header now claims only what was measured.

Suite 25 → 30 tests. Round 3 delta audit dispatched against b11330b5..9a172aae.

legend: <from> = the tip THIS round's audit READ · <to> = the head THIS round's FIXES produced.

1. Added an ACTIVATED flag set after `nixos-rebuild test`; the trap now distinguishes switched / activated-not-persisted / never-activated and no longer claims nothing is running when test had activated it.
2. Verification reads systemd-analyze cat-config (merged, incl. drop-ins), with /etc/systemd/journald.conf as a fallback that explicitly warns drop-ins were not checked.
3. Section 8 of diagnose-nix-disk.sh resolves the home from SUDO_USER via getent, prints which home it used, and has a skip fallback when nothing matched.
4. The migrated-keys array is filtered to ^[A-Za-z][A-Za-z0-9]*= and ignored stderr lines are reported rather than silently becoming keys.
5. journald_migrate.py refuses any value containing ${ or '' in both assignment forms; a lone $ is still allowed; covered by four new tests.
6. PATCHED=1 moved above the mv, and a failing cp inside the trap no longer aborts it under the inherited set -e.
7. Replaced the vacuous tab assertion with a negative that can fail, and added a non-vacuity guard on the _settings_body helper.
8. Doc says five files, and next-step 1 no longer points at the superseded gate run b8tu5owud.
9. Diagnostic header states the runtime (~80M inodes, two full /nix walks, sort buffering); FLOOR caveat prints only as non-root; lsof count excludes its header row.
10. Header no longer promises the trap covers any interrupt — SIGTERM measured, SIGINT explicitly not established.

[ZacxDev]

Audit rounds 3-5 · ladder STOPPED at round 5

Round 3 — 🔴 systemd-analyze cat-config CONCATENATES, it does not merge

Round 2's drop-in fix rested on a false premise, and I asserted that premise in the commit message, the code comment and the output label that told the operator "merged journald config". cat-config emits the main file and every drop-in with # <path> headers between them; it does not resolve precedence. So grep -qxF matched in both false directions, both measured:

• a drop-in overriding the migrated key → the main file's line is still in the output verbatim → ok, while the setting was inert. The exact hazard the fix was written to close, still open.
• a value present only in a stale drop-in → also ok, i.e. a migration that never reached journald.conf reported success. Round 2 made this one worse: before it, the pair had to be in the file NixOS generates.

Now two independent facts per key — LANDED and IN EFFECT — because either alone can be true while the migration is broken. Measured across four states: override → OVERRIDDEN, stale-drop-in-only → NOT LANDED, clean → ok, absent → MISSING.

Round 3 also mutation-tested round 2's guard and found it half-untested: all three params of the antiquotation test contained ${, so every one died on that half; deleting the '' half SURVIVED a green 30-test run. Added the missing case; re-measured, that mutant now dies on exactly the new test.

Round 4 — 🟡 the switch boundary told the operator to reboot into the thing it was reverting

SWITCHED=1 was set only after nixos-rebuild switch returned, so a switch failing during activation printed "It was never added to the boot menu, so a REBOOT reverts it". Verified against nixpkgs 26.11: switch-to-configuration-ng/src/main.rs runs do_install_bootloader for Action::Switch before activation. At the only moment a switch can fail mid-activation, the generation is in the boot menu — a reboot boots the migrated config, the opposite of what a root operator is told while choosing how to recover.

Also 🟢: parse_settings(body, form="block") defaulted to the permissive side, and flipping that default survived the whole 35-test suite — the only caller passes it explicitly, so nothing would catch a future caller omitting it. Now required, with a test that fails if a default returns.

Round 5 — no behavioural defect; every round-4 claim verified BY MEASUREMENT

First round where the commit message survived intact (2 and 3 both failed exactly there). Three findings, all comment truthfulness:

• F1 — the SAFETY header still taught that test-before-switch means "an ACTIVATION failure does not leave a registered generation and a rewritten bootloader behind". True of the test phase only, false for the switch phase — the belief that produced two bugs here, stated as the design rationale. A maintainer believing it would delete the SWITCH_ATTEMPTED branch as over-caution and regenerate the defect.
• F2 — # Three states, not two. above a five-branch chain. Replaced with the invariant, not a count; the count had been wrong twice.
• F3 — the MISSING branch claimed "anywhere in the journald config" on the path where drop-ins were never read — the same unearned scope claim round 4 had just conditioned the ok line to avoid, applied to one branch and not its sibling.

I swept every comment in the file for that shape rather than the three reported; one further site was checked and is accurate.

Why the ladder stops here

Rounds 1-5 each found something real, so the findings-keyed rule would run a round 6. Invoking the prose-payload criterion instead, stated rather than implied:

• the remaining defects are the script's comments, so every fix edits payload lines and the attribution gate is structurally unable to fire. Payload trend: 489 → 148 → 90 → 42, monotone; this round's fix is comment-only.
• no 🔴, and the blast radius of what remains is "a comment is false", not a host.
• the shape was swept at every site, not just the reported ones.
• the auditor's own advisory: "If you fix F1-F3, that is a comment-only edit — I would not audit it."

Recorded as open, not absent (deliberately not fixed): NOT IN EFFECT: breaks the label column alignment; on the verify-failure path the trap deletes $MERGED before the post-loop cat reaches it; and the shell half has no automated coverage — four rounds of trap-message fixes rest on reading, and the auditor's throwaway harness caught an ordering hazard in seconds. That last one is why this class kept recurring.

Ladder summary

round	blocker	introduced by
1	client subdomain re-committed in the doc describing the gate that bans it	the PR
2	test activates, so the trap said "nothing is running the change" while it was	round 1's fix
3	cat-config concatenates; verification matched in both false directions	round 2's fix
4	switch writes the bootloader first; "a reboot reverts it" was backwards	round 1's fix
5	— none behavioural —

Gate: PASS on b11330b5 (pytest 21152 collected / 0 failed, node 1449/1449). Re-running both tiers on the final head, plus nix build .#checks.x86_64-linux.{pytests,nodetests} one at a time — that sandbox tier is what Tekton gates on and has not been run on this branch. Verdict posted here before any merge.

[ZacxDev]

Gate verdict — both tiers PASS on the merged tree 53d8b962 (= squash f06b106f)

Merged before this reported, on the operator's explicit call. It came back green.

Tier 1 — dev-host (scripts/gate.sh --tier both, run from the flake devShell):

TOTAL collected=21164  passed=21162  skipped=2  failed=0   (floor: 20342 = sum of 28 per-target floors)
TOTAL suites=5 files=41 tests=1449 pass=1449 fail=0        (floor: 1367)
GATE: RESULT=PASS exit=0

Tier 2 — sandbox (nix build .#checks.x86_64-linux.{pytests,nodetests}), built one at a time because a combined invocation contends on the store and produces measured false failures:

pytests    NIXBUILD_RC=0  timeout panics: 0  RESULT: PASS (exit=0)
nodetests  NIXBUILD_RC=0  timeout panics: 0  RESULT: PASS (exit=0)

This is the tier Tekton gates on, and it had never been run on this branch before now — #773 is the precedent where four consecutive dev-host greens preceded a red sandbox run, so the two are not interchangeable.

Base named, per the rule that a gate result is a claim about one tier, one tree, one base: the merged tree is this branch merged with origin/main at efa9a0fc. main moved 5 commits during the audit ladder; I merged rather than rebased so the audit-claims sha ranges in the comments above still resolve.

Squash verified by CONTENT, not ancestry (a squash is never an ancestor of its base): all five files present in origin/main, the two payload files byte-identical to the branch head, and the round-5 SWITCH_ATTEMPTED guard present — so it is the final revision that landed, not an earlier one.

Still open, deliberately

• The shell half has no automated coverage. Five rounds of trap-message fixes rest on reading; the auditor's throwaway harness caught a branch-ordering hazard in seconds and does not exist in the repo. This is why that class kept recurring.
• NOT IN EFFECT: breaks the label column alignment.
• On the verify-failure path the trap deletes $MERGED before the post-loop cat reaches it.
• The laptop is not migrated. It carries the one-line extraConfig form on a 26.11pre channel and will hit the same assertion on its next rebuild. The rewriter handles that form and is unit-tested for it, but has never been run on that host — different claims.