feat(nebula): advertise a nebula relay — a read-only check, and the sudo apply beside it - #1272
Merged
Conversation
/etc/nixos/configuration.nix on workbench renders
`relay: {am_relay: false, relays: [], use_relays: true}` — an empty `relays:`
means no peer can relay TO workbench, so laptop->workbench is direct
hole-punch only. That is the blocker behind clawgate #497.
Adds an idempotent apply script in the existing nix/system/apply-*.sh style:
backs up the config, asserts exactly one match of the target block before
patching, rebuilds, then verifies.
The verification parses the rendered YAML's relay.relays rather than grepping
for the relay IP. The obvious grep is a false positive: lighthouse.hosts lists
the same two mesh addresses, so `grep -E '^\s+- 10\.42\.0\.2$'` MATCHES the
pre-change config and would certify the fix while relays: was still empty.
The script runs the pre-change file through the check on every run as a
negative control, so a PASS is only printed beside a fired control.
Both relay targets were confirmed `am_relay: true` in the LIVE
nebula-lighthouse-config on the homelab and production clusters — nebula
requires that of anything listed in a peer's relays:.
Not yet applied: sudo on workbench is password-gated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RMYjPVhL74So5kqc6XTzCW
Claude-Session-Id: 8ad54f57-207e-4fe2-9cec-b9d2fcf014f3
Audit of #1272 found three blockers in the apply script, all downstream of it being a stateful text-patcher that edits /etc/nixos and rebuilds: - It committed the production lighthouse's public IP into a PUBLIC repo, turning scripts/tests/test_no_public_ips.py red and adding a fourth site to a ratchet built only to shrink. apply-nebula-443.sh:4 already states the convention this violated. - Its verdict could not tell "already correct" from "broken": on a second run RENDERED_BEFORE is the patched config, so the negative control read as not-firing and a correctly-configured host got a hard FAIL. - Its `"relays = [" in s` idempotency guard is a whole-file substring, so on a host where apply-travel-prep.sh already inserted a relay list it silently skipped and reported success. That is the laptop's live state. Also: `import yaml` is not satisfiable from the python3 on the PATH that `sudo bash …` actually gets here, so the script could not run as documented. Rather than fix six findings on that design, this drops the auto-patching. The config edit is four lines the operator applies and can read in full; what is worth shipping is the check, which is the part that was subtle. check-nebula-relays.sh is read-only, needs no sudo, and: - parses relay.relays with the STDLIB only (no PyYAML dependency); - never greps the rendered YAML for the relay address -- lighthouse.hosts carries the same mesh addresses, so that pattern matches a config whose relays: is empty. The trap is documented in the header and asserted in the self-test; - gates its own verdict on a --self-test that must read BOTH a relays: [] and a populated fixture correctly, so a PASS is never printed by a parser that has not been shown able to say either answer; - prints the exact nix snippet to add when the check fails. Verified on the workbench node: live run FAIL rc1 (relays is empty today), shimmed PASS rc0, unit-missing rc2, unresolvable -config rc2, drop-in override picks the last ExecStart rc0, wrong-expected-relay rc1, self-test rc0. Two parser mutants (drop block-style items; report a relay for []) each fail the self-test rc1 and take the live run to rc2. The public-IP test was negative-controlled: green here, and red at the old file:line once that file is staged again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RMYjPVhL74So5kqc6XTzCW Claude-Session-Id: 8ad54f57-207e-4fe2-9cec-b9d2fcf014f3
ZacxDev
changed the title
Member
Author
Audit round 1 — findings and what the fix round didFull adversarial audit at Findings F2, F3, F4, F6 and F7 are closed by deletion: the apply script no longer exists, so there is no second-run verdict, no substring idempotency guard, no timestamped backup, no host assumption and no half-patched-config failure mode. Not fixed, recorded as open: F8 (relayed workbench traffic bills Hetzner egress). This is inherent to advertising the production lighthouse as a relay and is a decision for the operator, not a defect in the check. The verifier's own evidence — seven exit-code cases and two parser mutants — is in the PR body and the commit message. The relay is still not applied; that needs the operator's password. |
… findings designed out
check-nebula-relays.sh tells you the relay is missing; this applies it. It is a
second attempt at the script the audit rejected, built so each of that round's
findings is structurally absent rather than patched over:
- F1 (public IP in a PUBLIC repo): no public-IP literal. Both defaults are mesh
addresses; the relay's public underlay appears nowhere.
- F2 (could not tell "already correct" from "broken"): the pre-state comes from
the VERIFIER, not from the file, so a stale edit that was never switched does
not read as satisfied. Already-satisfied prints SATISFIED and exits 0 without
touching the config or taking a backup.
- F3 (whole-file substring guard silently half-applying): the anchor is the pair
` settings = {` + ` punchy = {` and must match EXACTLY once — the bare
`settings = {` line occurs twice in the live file, the pair once. A config that
already carries `relays = [` fails that check and aborts naming the line and
telling the operator to merge by hand; it never merges lists itself.
- F4 (same-second re-run clobbering the pristine backup): backup name carries
timestamp AND pid, an existing path is refused rather than overwritten, and no
backup is taken on any path that does not write.
- F5 (PyYAML/python3 unavailable on the sudo PATH): the patch is awk; python3 is
needed only by the verifier and is checked in preflight with the exact
`sudo env "PATH=$PATH"` remedy. python3 is NOT in /run/current-system/sw/bin
here, so this is a real failure mode, not a hypothetical.
- F6 (no host assertion; `hostname` is `nixos` on more than one machine): the
guard is this host's own nebula mesh address, which is unique by construction.
- F7 (no failure atomicity): the temp copy is syntax-checked with
`nix-instantiate --parse` and line-counted BEFORE the backup is taken and
before anything is moved into place; from the move onward an EXIT trap restores
the backup, and it distinguishes "never switched" from "already switched, the
running system still needs a rebuild to revert".
Verification is delegated to check-nebula-relays.sh rather than reimplemented.
Exercised end-to-end against a copy of the live configuration.nix with shimmed
id/ip/systemctl/nixos-rebuild (no sudo, nothing real touched): happy path rc0 and
the 4 lines land in the right block; re-run when satisfied rc0, config unchanged,
0 new backups; wrong host rc1 before any write; config already carrying a relays
list rc1 with the merge-by-hand message; rebuild failure rc1 with the config
restored and "never switched"; verifier failure after a successful switch rc1
with the config restored and the "already switched" warning. Two mutants — insert
invalid Nix, insert 5 lines — each abort with THAT gate's own error, before the
backup, config untouched, and the unmutated script goes green on the same fixture.
The awk `n != 1` guard is defence in depth behind the preflight anchor count and
is not independently reachable; it is not claimed as tested.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RMYjPVhL74So5kqc6XTzCW
Claude-Session-Id: 8ad54f57-207e-4fe2-9cec-b9d2fcf014f3
ZacxDev
changed the title
…rse as "no relay"
Round 2 of the audit found three defects, all in the verifier, all verified
before fixing:
- 🟡-1 the file CLAIMED "a PASS means the running service loaded the new value"
while the code read only `systemctl cat` -- the unit file. Measured: that
exits 0 for an INACTIVE unit, so a config that built but never took effect
read as applied. A comment is a claim, and this one was false. Now the config
comes from /proc/<MainPID>/cmdline, the unit is required to be active, and a
disagreement between the unit file and the running process is its own reported
outcome ("a rebuild has landed but nebula has not restarted onto it").
- 🟡-2 a one-line flow mapping parsed to NOTHING and was reported as
"relays is EMPTY" -- the exact opposite of the truth -- and a python traceback
inside the command substitution did the same, because `set -e` does not abort
there and `read` succeeds on the empty here-string. Both are now exit 2,
"cannot determine", which is a different claim from "the list is empty".
- 🟡-3 the -config extraction matched only `-config X`. Measured against the
other spellings Go's flag package accepts: with a drop-in using `--config` or
`-config=`, the old sed silently fell back to the BASE unit's config and
reported on the wrong file. One extractor now feeds from both the process
cmdline and the unit's last ExecStart, so the two cannot drift.
- 🟢-1 the header claimed the generator never emits flow style. It does:
`relays: []` -- the exact state this script exists to detect -- IS flow style.
A maintainer trusting that comment could have deleted the branch that sees it.
- 🟢-2 --self-test failure now exits 2, not 1; 1 means "relay absent", a
different claim, and a CI job could not tell them apart.
- 🟢-4 the remediation block now says relayed traffic egresses the relay in both
directions, so a billed host puts every relayed byte on that bill (F8, which
was declared open rather than fixed).
Exit codes are now 0 advertised / 1 not advertised / 2 cannot determine.
Self-test grows from 3 controls to 8: the empty flow list, a block sequence, a
populated flow list, a one-line flow mapping, a terminator case, an absent
relay key that must be rc 2 rather than a false "empty", all four -config
spellings plus absent, and the naive-grep trap. Round 2's surviving mutants M1
and M3 are killed, plus a new one for the flow-mapping branch; positive control
green. M5 (terminator `break` -> `in_block = False`) is NOT claimed as killed:
measured, it differs only on a config with a duplicate top-level `relay:` key,
which a JSON-derived generator cannot emit. It is an equivalent mutant and
adding a fixture for an impossible input would be manufacturing coverage.
apply-nebula-relay.sh: since the verifier now returns 2 when the running process
has not picked up a new config, the post-switch check retries ONCE after
restarting the unit before rolling back -- `nixos-rebuild switch` normally
restarts a changed unit, and when it has not, restarting it is the completion of
the change we just made rather than an escalation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RMYjPVhL74So5kqc6XTzCW
Claude-Session-Id: 8ad54f57-207e-4fe2-9cec-b9d2fcf014f3
Member
Author
Audit round 2 — findings and what the fix round didDelta audit of Field report from the first real run. The operator ran 🔴 It also exposed a design defect not in either audit round: this machine has a pending 26.05 → 26.11 release jump, so Separately and pre-existing: nothing has activated on that host since it booted 30 days ago ( |
… and cover both scripts Round 3 of the audit found eight issues in the root-privileged half. All eight are fixed, and both scripts now have repo-gate coverage they had none of. F-A — the rollback was not a rollback. `nixos-rebuild switch` registers the built system into /nix/var/nix/profiles/system and installs the bootloader BEFORE activation, so a failed activation still moved the machine while the trap printed "the system was never switched, so nothing is running the change". Measured on this host: generation 387 held `relays: - 10.42.0.2` after a reported-clean rollback. The order is now test -> verify -> switch -> verify; `nixos-rebuild test` activates without registering a generation or touching the bootloader, so a failure there really does persist nothing. The trap tracks five states and distinguishes never-activated, test-failed, activated-not-persisted, switch-attempted (profile MAY have moved, with the command to check) and persisted. It never claims more than it knows. F-B — the anchor was tied to nothing. Reproduced: NEBULA_NET=travel reported success naming nebula@travel.service while the four lines landed in the mesh block. The services.nebula.networks.<NET> block is now located by brace depth first and the anchor is only accepted inside it; every ambiguity aborts. F-C — root wrote to /tmp/nebula-relay-pre.$$. /tmp is 1777 and `>` follows symlinks. Now mktemp -d (0700), cleaned by the trap; the patch temp is a mktemp sibling of $CFG so the mv stays atomic. F-D — a missing backup skipped the rollback silently. There is an else now, cp's status is checked, and failure prints ROLLBACK FAILED with the manual fix. F-E — `[ -f "$CFG" ]` accepted a symlink and mv destroyed it. Symlinks are REFUSED, not followed: writing through a caller-chosen path as root is the same hazard as the /tmp one. Compared via readlink -f, so a symlinked directory component is caught too. F-F — a fixture comment claimed coverage it did not provide. Replacing the relay block's terminator with `if False:` left --self-test green. Fixture 5 keeps an honest comment; new fixture 5b kills the mutant with its own message, and says plainly that pkgs.formats.yaml cannot emit that shape, so it pins the parser's structure and not a reachable production config. F-G — the egress-cost warning never reached the operator. The verifier's rc-1 output is now printed in full before proceeding. F-I — the rc-2 retry was wider than its comment. check-nebula-relays.sh now emits a machine-readable `REASON: <token>` on every rc-2 path and the single restart fires only on unit-process-disagree. Also: unused `indent` and `local spellings` dropped; the preflight tool list names every binary the script execs; both scripts warn that applying a relay restarts the mesh the operator may be connected over. F-H — coverage. scripts/tests/test_nebula_relay_apply.py (29 tests, in the existing hermetic scripts/tests target) drives the real scripts against a fixture configuration.nix via NEBULA_CFG with id/ip/systemctl/nixos-rebuild/ nix-instantiate shimmed on PATH. The verifier under test is the real one and it reads the running process, so the rig spawns a real child carrying `-config <rendered.yml>` and the rebuild shim re-renders that yml from the .nix just patched. Nothing can reach a real rebuild, the profile, or /etc/nixos. scripts/tests/mutants-nebula-relay.sh is the battery, in the same shape as the other mutants-*.sh: three controls, applied-verified mutations, and each mutant naming the test that must kill it AND the message it must die with. 20/20 in the battery. Two results are recorded rather than rounded up: M-X-1 (the +4-lines guard) is a DECLARED SURVIVOR -- unreachable behind the awk `n != 1` guard, kept as defence in depth, not counted as coverage; and M-FB-2 is killed by that awk guard rather than by the assertion naming the wrong block, so the patch pass's range guard is defence in depth behind the scoped anchor count. Nothing was applied for real: /etc/nixos/configuration.nix is unmodified, nixos-rebuild was never run in any form, and the profile was not touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RMYjPVhL74So5kqc6XTzCW Claude-Session-Id: 8ad54f57-207e-4fe2-9cec-b9d2fcf014f3
…d-rolled shebang
The repo's own gate caught this before the sandbox did:
`test_runtime_shebangs.py::test_no_test_writes_a_usr_bin_env_shebang_at_runtime`
went red on two sites in the new harness. A test that writes a stub at RUNTIME
and then execs it cannot use `#!/usr/bin/env bash` — /usr/bin/env exists on this
NixOS host and does NOT exist in the nix build sandbox, which is the
authoritative tier, so the defect is structurally invisible here.
`testlib.mockbin.write_exec` owns that decision for the whole repo, so the shims
go through it. Consequence: the bodies are now POSIX sh, which required two real
changes rather than a shebang swap:
* the `ip` shim used `${@: -1}` (bash-only) to take the last argument; it now
walks "$@".
* the `nixos-rebuild` shim had a dead `cat …/rebuild_{test,switch}_rc` line
using brace expansion; removed (the case below already sets rc).
* the `id` shim no longer execs the real `id` through /usr/bin/env for
unexpected args — it fails loudly instead, which is the right answer for a
shim anyway.
Controlled two ways, because "it parses" and "it behaves" are different claims:
`dash -n` on all five generated shims, and the rig re-run end to end with the
shims written under dash (happy / deferred-restart / unreadable-config /
wrong-host all identical to the bash-backed /bin/sh run). This host's /bin/sh is
bash-interactive, so without that second control a bashism would have stayed
invisible until the sandbox.
Mutation battery re-run after the change: 20/20, same one declared survivor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RMYjPVhL74So5kqc6XTzCW
Claude-Session-Id: 8ad54f57-207e-4fe2-9cec-b9d2fcf014f3
… exists The patch temp is a mktemp SIBLING of $CFG rather than a file in the scratch dir, because the last step is `mv "$TMP" "$CFG"` and that is only atomic within one filesystem. Nothing asserted that it gets cleaned up, and only one path fails while it still exists: `nix-instantiate --parse` rejecting the patched file, which happens after the temp is written and before the backup is taken. test_an_invalid_nix_result_aborts_before_the_backup_and_leaves_no_temp pins all four claims on that path — the abort, its own message, no temp sibling left, and NO backup taken (the backup must not precede the parse check). The existing scratch-on-failure test now also asserts the sibling is gone. Two mutants added, both killed with this test's own message: M-X-3 drops `rm -f "$TMP"` from the cleanup -> "temp sibling leaked" M-X-4 turns the parse check into `|| true` -> "not valid Nix" Battery: 22/22, one declared survivor (unchanged). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RMYjPVhL74So5kqc6XTzCW Claude-Session-Id: 8ad54f57-207e-4fe2-9cec-b9d2fcf014f3
ZacxDev
added a commit
that referenced
this pull request
Sep 8, 2026
…ght #1272 left open (#1378) * feat(tailscale): the independent second path — plus the closure preflight that stops "add a few lines" becoming an OS upgrade Nebula is currently the ONLY remote path to the workbench, and the operator leaves the LAN for months in ~3 days. Tailscale is the second door: different control plane, different relays, no shared component. nix/system/apply-tailscale.sh Adds `services.tailscale` to /etc/nixos/configuration.nix and switches. Two roles, discriminated by the nebula mesh address because `hostname` CANNOT tell these machines apart (both answer `nixos`): 10.42.0.30 -> subnet router (useRoutingFeatures = "server", forwarding, advertises 192.168.50.0/24), 10.42.0.100 -> plain client. Anything else aborts rather than guessing. 🔴 THE CLOSURE PREFLIGHT is the point of this PR. `nixos-rebuild switch` applies EVERYTHING PENDING, not just your delta — a 4-line nebula edit once triggered a 26.05 -> 26.11 jump that took 2h16m and rebuilt the world. That was recorded as an OPEN defect on #1272 and never fixed. This script measures the change three independent ways before switching and REFUSES by default: 1. dry-build run TWICE — against the CURRENT config and the patched one — so PENDING work is reported separately from the tailscale DELTA. Blaming tailscale for a channel bump, or hiding one behind it, both become impossible. 2. the nixpkgs release string, extracted by ONE implementation from both sides and cross-checked against /run/current-system/nixos-version so a broken extractor cannot silently agree with itself. A release change is refused unconditionally. 3. the closure compared as SETS (`nix-store -qR | comm -3`), which cannot return the reassuring zero that a parsed `diff-closures` can. MEASURED on the workbench 2026-09-07: pending 40 builds / 24 fetches (159.4 MiB) with NO change at all; 47 / 25 (177.0 MiB) with tailscale. So tailscale's true cost is +7 derivations and +1 fetched path (`tailscale-1.102.3`). The thresholds are calibrated from those numbers, and the gate therefore REFUSES on this host today — correctly, since 40 queued derivations are not what anyone asked for. `--allow-world-rebuild` overrides. 🔴 `readlink -f` on BOTH sides always: /nix/var/nix/profiles/system is a symlink TO ANOTHER SYMLINK (it reads `system-389-link`), so single-level readlink returns a NAME on one side and a store path on the other and they can never compare equal. `_store_path` resolves fully and refuses anything outside /nix/store, so an empty result can never read as agreement. nix/system/check-tailscale.sh Read-only verifier, no sudo, runnable before and after. Reads LIVE RUNTIME STATE — `tailscale status --json`, `tailscale debug prefs`, /proc/sys, `ip route`. NOT the nix config and NOT the unit file: `systemctl cat` exits 0 for a dead unit, so a config that built but never activated reads as applied, which is why check-nebula-relays.sh was rewritten. 🔴 ADVERTISED and APPROVED are reported as two separate claims. An advertised route that has not been approved in the admin console carries NO traffic and is INDISTINGUISHABLE from success on the node. Approval is read from Self.PrimaryRoutes, which only the control plane populates. 🔴 Node key expiry is reported with its date and days remaining, and being ENABLED at all is an outstanding action: the 180-day default is shorter than the trip, and the lapse is silent. Exit 0 / 3 (node correct, admin-console step outstanding) / 1 / 2 are four different claims. 3 exists so apply-tailscale.sh does not roll back a correct switch because a browser tab has not been clicked. Testing * Both --self-tests green; 8/8 mutants killed on the verifier's parser and predicates, 12/12 on the apply script's parsers and BOTH branches of the gate, each by its own named case, with an unmutated positive control. * The verifier driven end-to-end against a mock tailscale: rc 0, 1, 2 and 3 all watched, including the forwarding and lanroute FAIL branches. * The public-IP ratchet watched go RED naming check-tailscale.sh with a planted realistic IP, then green again byte-identical — the green is earned, not assumed. Both scans confirmed to enumerate the new files. * shellcheck 0.11.0 clean; 290 tests green across the four content ratchets and the adjacent structural gates. NOT verified: nothing has been applied to a live host. sudo here is password-gated, so `nixos-rebuild build/switch`, the real `tailscale status` schema, and the post-switch verify path are all UNEXERCISED. Fixtures are built from the documented shapes, not from a running daemon. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WDQfnd1tsBNKSsGTD7jL7Y * fix(tailscale): the first apply run was guaranteed to roll back — plus ten more audit findings 🔴 THE BLOCKER. `apply-tailscale.sh` deliberately does not run `tailscale up` (it needs a browser), so straight after its switch the node has never authenticated: BackendState `NeedsLogin`, `Online` false, no TailscaleIPs, nothing advertised. `check-tailscale.sh` turned that into FOUR entries in `fails[]` and exited **1**, while the post-switch verify accepted only rc 0/2/3 and treated anything else as a node-side failure -> `die` -> the EXIT trap restored the backup. Driven end to end against the real scripts with a shimmed `nixos-rebuild`: the run printed `ABORT ... ROLLED BACK` and left `grep -c 'services.tailscale' = 0`. Every first run, on every host, guaranteed. The comment at the case claimed rc 3 was "the EXPECTED state straight after a switch" — it never was. FIX, and deliberately not a wider rc allow-list. "Never authenticated" is now a FIRST-CLASS STATE in the checker (`_authenticated`, two signals that must agree) with its own exit code **4 = INCOMPLETE, no defect found**, and the apply script's post-switch `case` enumerates all five codes with what each MEANS. Same fixture, new code: `=== SWITCHED ===`, block kept. rc 1 still rolls back — driven with forwarding off, and pinned by a test, because a guard that can no longer go red was removed rather than fixed. Also fixed, each with an old-vs-new control: * KEY EXPIRY (F7). An absent `Self.KeyExpiry` was resolved to "expiry disabled -> PASS" — printed on a node that had never logged in, for the single item most likely to kill this path silently mid-trip. Absence is ambiguous; "disabled" is now claimed only when the node IS authenticated, otherwise UNKNOWN. * PARTIAL `cp` (F8). `PATCHED=1` was set AFTER `cp -p "$TMP" "$CFG"`, so a copy that failed part way (ENOSPC; `/` is at 77%) aborted with PATCHED=0, the trap's rollback branch was skipped, and `/etc/nixos/configuration.nix` was left truncated with a good backup unused beside it and NOTHING printed. Measured with a `cp` that writes 40 bytes and exits 1: OLD left a 40-byte fragment ending mid-word, NEW rolls back to a valid file. The auditor judged this correct; it is not. * ROLLBACK MESSAGE (F3). It asserted "The system was never switched" from a flag that only records whether `nixos-rebuild switch` RETURNED ZERO — false in exactly the case it printed in, since a switch can activate and still exit non-zero and writes the bootloader entry first. Three states now, and only the establishable one is asserted. * THE `--allow-world-rebuild` DIAGNOSTIC (F4). "Largest pending items" was alphabetical and cut at `head -15`, so on the real 38-derivation queue every `steam-*` was hidden and `wine-wow-11.0` — the culprit in the 2h16m incident this script exists for — sorted last of all. It now shows every one, says it is a name list and not a size ranking, and when it caps it says how many it omitted. * THE GATE WAS BLIND TO DOWNLOADS (F5). `b_fetch`/`c_fetch`/`b_mib`/`c_mib` were printed in the summary table and read by no gate, so a fully substitutable pending change — 0 to build, 4 GiB to fetch — passed untouched. Measured: OLD proceeded to `== nixos-rebuild switch ==`, NEW refuses. Download volume is now gated in its own right (`_over_mib`, floating point — `-gt` cannot compare "159.4"), and the header's "three independent ways" is now four, each stated with what it CANNOT see. The release gate's major.minor blind spot is called out where it reports, since `26.11pre1066106 -> 26.11pre1066425` does not fire it. * IDEMPOTENCY (F6). `grep -q 'services\.tailscale'` meant a config containing only `# TODO: consider services.tailscale one day` exited 0 "Nothing to do" — telling the operator the backup path was done on a host where nothing was applied. Comments (`#` and `/* */`) are stripped first and an actual SETTING is required. * `--accept-dns=false` ON BOTH ROLES (F9). The reasoning is identical on the two machines; the asymmetric version handed MagicDNS the resolver of the ONE machine that leaves the LAN. The cost is stated rather than hidden, and a test pins the two scripts' `tailscale up` strings EQUAL so they cannot drift apart again. * SIGNAL-KILLED SWITCH (F2). The status is captured (`rc=0; cmd || rc=$?`) and any non-zero, signals included, is a failure that names the signal. ⚠ MEASURED CORRECTION: the reported SIGINT mechanism did NOT reproduce on bash 5.3.15 here — with the signal confirmed to have landed, bash propagates the child's SIGINT death and terminates the script, both child-only and process-group. The first "it continued" observation was a `kill` against a process with SIGINT ignored (background from a non-interactive shell), which is indistinguishable from continuing. The check IS reachable on SIGTERM/SIGHUP, driven with both. * MINORS (F10). `--role` as the final argument exited 1 printing nothing (both scripts); `--help` was a hardcoded line range that truncated one file's SAFETY paragraph and ran the other into `set -euo pipefail` (now read from the header itself); a config with no trailing newline aborted naming the wrong problem (`wc -l` counts newlines, `awk END{print NR}` counts records); a negative build delta rendered "Only -3 of the 37 are tailscale's"; and `"$CHECK"` is now `bash "$CHECK"` — measured, a copy without its exec bit made the verifier exit 126 and the config get rolled back one line after a successful switch. * AIRVPN KILLSWITCH (F11, documented not changed). `scripts/airvpn-updown`'s degraded/fallback rulesets allow a LITERAL interface list — lo, the airvpn tun, nebula.mesh, cni0, flannel.1, docker0 — then `drop`. `tailscale0` is not on it and has no DERP carve-out while nebula has three, so if that killswitch ever arms fail-closed NEBULA SURVIVES AND TAILSCALE DIES, inverting the redundancy this change is for. AirVPN is default-OFF, so this is latent; widening a killswitch is its own change with its own review. Noted in the generated Nix block for both roles and in the script header. `scripts/tests/test_tailscale_scripts.py` is new: it pins the two scripts' `up` commands equal, pins the exit-code vocabularies against each other, drives the freshly-switched state against a fake `tailscale`, and runs both scripts' own `--self-test` suites so their controls are part of the gate. All 17 mutants in a sweep were killed by their intended test (two survived the first round — the fixtures could not see comment-stripping or the `[.={]` anchor — and the discriminating cases were added). NOT VERIFIED: nothing was applied to a live host. sudo is password-gated, so the apply path is exercised only against shimmed `nixos-rebuild`/`tailscale`/`ip`/ `systemctl`. `tailscale up`, subnet-route approval and key expiry are untouched admin-console steps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFTzoM7GrDvw4WX3m7Kjfp Claude-Session-Id: 30ea5cd2-e238-4450-bf7b-dfc505be6fbe * fix(tailscale-tests): the new test wrote its own #!/usr/bin/env shebang `test_runtime_shebangs.py` caught it on the first full gate run — 1 failed, 12876 passed, and that one failure was this. A test-written stub carrying `#!/usr/bin/env bash` execs fine on this NixOS host and ENOENTs inside the nix build sandbox, so the defect is structurally invisible in the tier most people run. `testlib.mockbin.write_exec` owns the shebang (`/bin/sh`) for exactly that reason; both stub bodies here are POSIX sh. The guard working, not a reason to widen it. Re-ran the targeted ratchets (349 passed) and the 17-mutant sweep (17/17 killed by their intended test) after the change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFTzoM7GrDvw4WX3m7Kjfp Claude-Session-Id: 30ea5cd2-e238-4450-bf7b-dfc505be6fbe * fix(tailscale-tests): the fixture let the HOST answer the LAN-route question Tekton `devrc-pytests` went red on `3a090bb3`: 20918 passed, **2 failed**, both mine — `test_a_freshly_switched_node_is_rc_4_not_a_failure` and `test_absent_key_expiry_on_an_authenticated_node_is_disabled`. Both are the same defect and it is the two-tier hazard, not a code failure. `_fake_tailscale` stubbed `tailscale` and `/proc` and then let the REAL `ip -4 -o route show` answer the checker's "is this subnet reachable from here" question. That passes on this workbench, which genuinely has a route to 192.168.50.0/24, and fails in the Tekton sandbox, which has neither the route nor `ip` — so the checker adds a lanroute FAIL and returns 1 where the test asserts 4 (and 1 where it asserts 0). The claim under test had nothing to do with routing; the fixture simply left one host input un-pinned, and the tier that disagreed is the one nobody runs locally. Every host input the checker reads is now stubbed — `ip` (route table and the nebula probe) and `systemctl` alongside `tailscale` and `/proc` — with the route a fixture parameter. 🔴 And the stub gets its own control. `test_a_subnet_router_with_no_lan_route_ still_fails` runs with `lan_route=False` and asserts rc 1 plus the checker's own "no non-tailscale route" message. It passes ON A HOST THAT HAS THAT ROUTE, which is what proves the stub is actually the `ip` being consulted — without it, `lan_route=True` would be indistinguishable from not reading `ip route` at all. 24 tests green locally (23 + 1 skipped shellcheck). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFTzoM7GrDvw4WX3m7Kjfp Claude-Session-Id: 30ea5cd2-e238-4450-bf7b-dfc505be6fbe * fix(tailscale-tests): drop the shellcheck test — it could only ever SKIP, and a skip fails the gate `6f09b1b9` was red in Tekton and locally with **`failed=0`**: 20921 passed, 0 failed, `RESULT: FAIL (exit=1)`. The reason was not a test failing, it was `run-tests.sh`'s unpinned-skip guard — ERROR: 1 UNPINNED skip group(s) — coverage silently collapsed: test_tailscale_scripts.py: shellcheck not on PATH ERROR: 3 test(s) skipped, but 2 of 3 pinned entries apply here. `shellcheck` is not in this repo's `gateTools`, so it is absent in EVERY gate run: `test_shellcheck_is_clean_at_warning_level` could only ever skip. The guard is right — a skip is a test that did not run. Both remedies were worse than removing it, and the reasoning is written into the file where the test used to be: * pinning it in `EXPECTED_SKIPS` ships a test that never executes anywhere, which is exactly the vacuous-guard shape this suite exists to prevent; * adding `shellcheck` to `gateTools` changes the toolchain for every target and every developer in order to gate two files, inside a PR about something else. So shellcheck stays a manual step, and it was run for this change: `nix-shell -p shellcheck` (0.11.0), clean at `-S warning` on both scripts, with the scanner negative-controlled against a known-bad script first — exit 1 on the control, exit 0 under test. 23 tests, **zero skips**. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFTzoM7GrDvw4WX3m7Kjfp Claude-Session-Id: 30ea5cd2-e238-4450-bf7b-dfc505be6fbe * fix(tailscale): an EXPIRED node key is a FAIL, and an unmeasurable download refuses Round-3 audit fixes. The two P1s are both cases where the script answered the REASSURING question instead of the one it was asked. R1 — an expired or revoked node key exited 4, "NOT YET DETERMINABLE (no defect found)". `_authenticated` required BackendState and the address list to AGREE and resolved every disagreement to "not authenticated", so a node whose key had lapsed printed "THIS NODE HAS NEVER AUTHENTICATED ... That is the EXPECTED state immediately after apply-tailscale.sh" on the same line as the 100.x address it had retained. Keys expire at 180 days by default -- SHORTER THAN THE TRIP -- so this is the single most likely way the backup path dies while it is being relied on, and the checker called it "no defect". Replaced by `_identity_state`, printing none | ok | lost | incoherent: none no address AND a logged-out backend -> rc 4 (as before) ok an address, a live backend, not Expired -> judged as before lost an address RETAINED under NeedsLogin/NoState, OR Self.Expired -> rc 1 incoherent a live backend holding NO address -> rc 1, its own message `Self.Expired` was parsed-adjacent and discarded; it is now a record field and is read. Both documented expiry shapes are handled, since neither has been captured from a live daemon here. A KeyExpiry date in the PAST is a third, independent detector reading a different field again. Verified: NeedsLogin+addr, NoState+addr, Running+Expired and Running+empty-netmap all went 4 -> 1; NeedsLogin+no-addr is still 4; healthy 0; unapproved 3; future expiry 3; missing LAN route 1. Five mutants, each killed with its guard's own message ("exited 4, not 1"), the decisive one isolated so the self-test stayed green. R2 — the download gate failed OPEN on any size string it could not parse. `re.search(...) -> mib = 0.0` on no match, indistinguishable from a genuine zero. Measured: `these 2400 paths will be fetched:`, `2.5 TiB`, `900000000 B` and `4096,0 MiB` ALL returned `0|2400|0.0`, so a 2400-path substitutable world rebuild -- exactly what the gate exists to stop -- passed all four gates in silence. An unreadable size is now the literal `UNKNOWN` and `_gate_reasons` REFUSES on it (`_is_num` keeps it away from awk, which would have scored it 0). TiB/TB/B are understood and scaled. The FETCH COUNT is now gated too -- it was parsed correctly and printed in the summary table and read by no gate, the same "decorative column" defect, and it is the axis that still has a number when the size does not. `_gate_reasons` now requires all eight arguments instead of defaulting them. The defect lived in the SEAM -- the parser was tested on well-formed text, the gate on hand-typed numbers -- so the join is now driven directly, in the bash self-test and in pytest, with the expected size PINNED (a first version asserting only "not 0.0" let a default-the-unit mutant survive by reading 4 EiB as 4.0). Also: R3 `_cfg_tailscale_decls` blanked comments but not STRINGS, so `warnings = [ "you should run services.tailscale.enable = true; here" ]` and a `''...''` block both reported "Nothing to do. Exiting 0" on an unapplied host. Replaced the three regexes with a left-to-right scanner (# / * */ / "..." / ''...'' with their escapes), because the constructs nest. Fixtures that can actually see the mutation, plus a control that a real declaration after a closed string is still found. R4 one run printed both "what this node ADVERTISES is UNKNOWN" and "<subnet> is advertised but NOT APPROVED", the second asserting -- and sending the operator to the admin console over -- the fact the first had declared unknowable. The approval check now reads whether the advertisement claim was EVALUATED, and says plainly that it cannot choose between an unticked box and a node that never advertised. Both say the route carries no traffic either way. R5 `_header_comment_lines` used the SAME "stop at the first non-# line" rule as the --help awk, so it could not see the truncation its docstring claimed to prevent: one inserted blank line cost 56 of 75 help lines with both tests green. The boundary is now `set -euo pipefail`, a different fact, plus a line-count floor and an assertion that every exit code reaches the printed output. R6 the comment said `-gt` on "159.4" "aborts the shell ... under set -e". Measured in the shape actually used: bash prints `[: 159.4: integer expected`, returns 2, and EXECUTION CONTINUES with the gate silent. It fails OPEN. Describing a loud abort where the truth is a silent fail-open is how the guard gets deleted. The same correction applies to the `[ "$rc" = 0 ] && rc=3` note. R7 `rc 4 -- EXPECTED. The switch succeeded and the node is NOT YET AUTHENTICATED` asserted one cause for a code with several, and is reachable two lines below `PASS authed`. It now names what was observed and points at the verifier's own NOT YET DETERMINABLE list. Unchanged on purpose: SWITCH_ATTEMPTED's three states, rc 1 still rolling back, the F7 both-directions expiry cases, the F11 AirVPN note. Verified: both --self-test suites green; 40 tests in scripts/tests/test_tailscale_scripts.py (was 23); full devrc gate PASS twice, the second run on the exact committed tree (pytest 12895 in scripts/tests, node 1449); shellcheck -S warning clean on both scripts, negative-controlled against a known-bad script; 16-mutant battery under PYTHONDONTWRITEBYTECODE=1, every mutant confirmed present in the file before scoring, all killed, with a positive and a negative control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFTzoM7GrDvw4WX3m7Kjfp Claude-Session-Id: 30ea5cd2-e238-4450-bf7b-dfc505be6fbe * fix(tailscale): an expired key that survived a REBOOT is still a FAIL, and a daemon that contradicts itself is not a rollback All three expiry detectors read NETMAP fields -- `TailscaleIPs`, `Self.Expired`, `Self.KeyExpiry` -- and the netmap is in-memory only: it is fed by control-plane map responses and has no restore-from-disk path. So ONE reboot, power cut or `nixos-rebuild switch` on a node whose key had already lapsed wiped all three at once, and the checker printed "THIS NODE HAS NEVER AUTHENTICATED ... It is NOT a node-side defect" and exited 4 about a backup path that was dead. `lost` only ever survived while the daemon had run CONTINUOUSLY since before the lapse -- not an assumption worth making across a months-long absence. The durable evidence is on DISK: `ipn.Prefs.Persist` (marshalled as `Config`), reloaded from `tailscaled.state` on every daemon start. A non-empty NodeID or LoginName there plus a logged-out backend is a LOST identity. Absence still reads as `none`, and an EMPTY persisted profile -- what a never-logged-in daemon carries -- must too, or every first run would be rc 1 and apply would roll back the config it had just installed. Both directions are pinned. When prefs cannot be read at all there is a narrow on-disk fallback (`"profile-` in the state file, NOT `_current-profile`, which the empty profile also writes). MEASURED, same fixture, old vs new: NeedsLogin + no address + a persisted profile -> rc 4 "HAS NEVER AUTHENTICATED" before, rc 1 "HAD A TAILNET IDENTITY AND NO LONGER HAS A VALID ONE" now; both fresh-install controls unchanged at 4. `incoherent` was rc 1, which is apply's `die` -> EXIT trap -> restore $CFG and report that the RUNNING system was not restored -- for a state whose own message says "re-run this check before doing anything else". The cost is not a wasted run: configuration.nix loses the block while the running system keeps it, so the next `nixos-rebuild switch` by anyone silently deletes the backup path. It is now rc 4, and the checker takes the second sample itself: on `incoherent` it waits TS_SETTLE_SECS (default 3) and re-reads once. rc 1 is untouched for the states that are actually measured -- expired/revoked, not Running/Online, not advertising, forwarding off, no LAN route (all re-verified red). Also, claims the code contradicted: * the `none` message asserted "`tailscale up` has never run on this host" -- one cause out of several, from a run that cannot see a command's history. It now names the evidence it actually has (no address, no persisted profile). * `config.services.tailscale.enable` in a `mkIf` or an assertion is a READ, not a declaration -- and both were MEASURED printing "already DECLARES ... Nothing to do. Exiting 0" on a host with no tailscale at all. Guarded with a one-character lookbehind, and the "every way Nix can spell a setting and no way it can spell a mention" claim above the regex is retracted. * "every occurrence is inside a comment" was made false by this branch's own widening to string literals, and again by the read case. * the `lost` FAIL printed "the netmap addresses it was issued are still present: <none> ... would have NEITHER" whenever it was reached via the Expired flag. The evidence clause is built from what was observed. * "<subnet> carries NO traffic right now" is derived from `Self.PrimaryRoutes`, which is a fact about THIS node. Scoped. * the header claimed a 100.64.0.0/10 range test the code has never performed. Reworded rather than implemented -- a range test could only turn a real but unexpected address into "no identity", the reassuring direction. Two test-sensitivity gaps closed: the world-sized-fetch cases now assert WHICH gate refused (`_is_num` accepting everything survived them, because the fetch-count gate fired anyway), and the two fetch gates are driven with distinct pending/total values instead of masking each other. 56 pytest (was 40), 14/14 mutants killed with a survive-control, shellcheck -S warning clean on both scripts (negative-controlled). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFTzoM7GrDvw4WX3m7Kjfp Claude-Session-Id: 30ea5cd2-e238-4450-bf7b-dfc505be6fbe * docs(tailscale): the LoggedOut / Config / state-file shapes are MEASURED now, not inferred Ran a real unprivileged tailscaled 1.102.3 (the version this nixpkgs pins) with its own statedir, never authenticated, and read it back: * `tailscale debug prefs` -> "Config": null (so `Config` IS the key `ipn.Prefs.Persist` marshals to, and it is empty on a fresh node -- the control the whole R1 fix rests on) * "LoggedOut": true ON A NODE THAT HAS NEVER LOGGED IN. Reading LoggedOut as evidence of a lost identity -- which was the tempting second signal -- would have made EVERY first run rc 1 and rolled back the config apply-tailscale.sh had just installed. It is deliberately not used, and now for a measured reason rather than an unverified one. * `tailscaled.state` is the two bytes `{}`: neither `profile-` nor `_current-profile`, so the on-disk fallback finds nothing on a fresh host. Both key names do exist in the 1.102.3 binary, which is why the marker is the per-profile entry and not the pointer. Comments only -- no behaviour change. 56 pytest, self-test green, shellcheck -S warning clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFTzoM7GrDvw4WX3m7Kjfp Claude-Session-Id: 30ea5cd2-e238-4450-bf7b-dfc505be6fbe * fix(tailscale): a STOPPED tailscaled is a FAIL — and a host that never had the unit is still rc 4 A crashed or stopped `tailscaled` exited 4: "NOT YET DETERMINABLE (no defect found)". With the daemon down there is no `tailscale status` and no `tailscale debug prefs` to read, and the on-disk fallback needs root, so `_identity_state` lands on `none` and every claim comes out UNEVALUATED rather than wrong — the checker calls a dead backup path healthy, on the one fact the operator most needs told while he is 5000 km away. Measured two ways: a real tailscaled 1.102.3 killed mid-run did not move the exit code, and the new fixture control is RED against dca21bf for the same reason. `is-active` is now read for exactly ONE verdict — liveness — and no others. Nothing about identity, advertisement, approval or forwarding is keyed on it; the `:826` comment that justified ignoring the unit entirely is corrected rather than left contradicting the code. The gate is `LoadState`, NEVER `is-active`, and it is the whole risk of this change. MEASURED on systemd 261: for a service that does not exist at all, `systemctl is-active` prints `inactive` — byte-identical to a unit that exists and is stopped — while `systemctl show -p LoadState --value` prints `not-found` vs `loaded` (checked against fstrim.service / emergency.service). Keying the FAIL on `is-active` alone would make every never-applied host a node-side FAILURE, which is rc 1, which is `die` + rollback in apply-tailscale.sh: the first-run blocker this script has been fixed for twice already. Only `loaded` counts; `not-found`, `masked`, an error and an empty answer (no systemd, or a systemctl too old for `--value`) all leave the run exactly as it was. Controls, all run: * OLD code + unit exists and `inactive` -> rc 4 (the bug reproduced) * NEW code + unit exists and `inactive`/`failed` -> rc 1, with its own finding * NEW code + no unit (`LoadState=not-found`) -> rc 4, "no defect found" * NEW code + nothing answers systemctl -> rc 4 * NEW code + loaded and active, healthy node -> rc 0 still reachable * rc 1 still reachable for every pre-existing reason (expiry, backend not Running, forwarding off, no LAN route) Mutation battery on the new guard (PYTHONDONTWRITEBYTECODE=1, each mutant verified PRESENT): widening the existence gate is killed by the no-unit guard; inverting the liveness test, downgrading `fails` to `actions`, and rewording the message are each killed by the dead-unit guard's own assertion; the survive control (journalctl tail depth) survives. Claude-Session-Id: 30ea5cd2-e238-4450-bf7b-dfc505be6fbe * fix(tailscale): the client route check read the MAIN table; tailscale installs accepted routes in table 52 Found on the FIRST real deployment, not in review. With workbench advertising 192.168.50.0/24, approved in the admin console, and the laptop joined with --accept-routes, the client check reported: ACTION REQUIRED: 192.168.50.0/24 is not in this host's routing table via tailscale. Either the route is not approved in the admin console, or --accept-routes is off here, or the workbench is not advertising. All three suggested causes were false. The route was installed and in use. Measured on the laptop: ip route show table 52 -> 192.168.50.0/24 dev tailscale0 ip route show table main -> 192.168.50.0/24 dev wlp170s0 (the native LAN route) ip route get 192.168.50.250 -> dev tailscale0 table 52 src 100.71.230.83 ip rule -> 5270: from all lookup 52 (BEFORE 32766: main) `ip route show` with no table argument reads MAIN. Tailscale puts accepted subnet routes in table 52 with a policy rule ahead of main, so the predicate at :1167 could never see one -- it fired on every correctly-working client, which is the state a traveller is in every single time they run this. Fix is `table all` on that one predicate. :1148 is deliberately NOT changed: the server-side `lanroute` check wants a NON-tailscale route in main, and table all would let the tailscale route satisfy it. Controls, both live, same script: laptop --role client : EXIT 3 -> 0 (the false ACTION is gone) workbench --role server: EXIT 0 -> 0 (server role unregressed) --self-test green before and after. An earlier attempt to patch this by string-replace silently matched 0 lines; the assertion that the target line changed is what caught it, so the "fixed" run I nearly read was the unpatched script. Direction of the bug was safe -- it under-reported, never claimed a broken path worked -- but it would have sent the operator to the admin console chasing nothing, off-LAN, with no way to check the other end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WDQfnd1tsBNKSsGTD7jL7Y Claude-Session-Id: 30ea5cd2-e238-4450-bf7b-dfc505be6fbe --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 8, 2026
ZacxDev
added a commit
that referenced
this pull request
Sep 8, 2026
…s became 81 Re-anchored c5e425c -> 01956bf while round 2 was in flight. Not drift in the sense the appendix disclaims: two mutation batteries LANDED, so the inventory was missing rows, which is a coverage gap. D11 mutants-diagnose-disk-accounting.sh (ffac18f, #1366) — a gate for root-only bash nothing had ever run; THREE live defects found while writing it, all fixed. 19 mutants, all killed. D12 mutants-nebula-relay.sh (01956bf, #1272) — the audit found three blockers in the apply script, all downstream of rc 0. Both have a named catch, so: 12/12 batteries, 81 rows, KEEP 51 / TIER 30 / DROP 0. pytest floor-sum 20,214 -> 20,348; always-run share 69.5% -> 69.1%; cap on the proposed tiering saving ~30.9%. Both machine-read markers updated in the same commit, which is what they are for — `check-gate-inventory.py` fails on a stale marker rather than passing quietly. Verified: 81 rows checked, declared 81, tally counted 51/30/0, problems 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXxTANDijZTVxj398NrWBU Claude-Session-Id: 45e79797-ec0e-45c7-ad21-5d623afb297b
ZacxDev
added a commit
that referenced
this pull request
Sep 8, 2026
…env (#1407) All 20 tests in scripts/tests/test_nebula_relay_apply.py fail on the `nix build .#checks.x86_64-linux.pytests` tier, while the dev-host tier (scripts/gate.sh) is green. #1272 landed on that split. apply-nebula-relay.sh executed the verifier directly: "$CHECK" "$RELAY" >"$PRE" 2>&1 which dispatches on check-nebula-relays.sh's `#!/usr/bin/env bash` shebang and so makes /usr/bin/env a hard runtime dependency. The nix build sandbox has no /usr/bin, so every one of those execs died with /usr/bin/env: bad interpreter: No such file or directory and the script reported ABORT: the verifier could not read the current config (rc=126) rc 126 is "found but not executable" — it is NOT one of the verifier's own exit codes, so that abort blamed the CONFIG for an interpreter fault, which is why the failure did not read as a shebang problem. Fixed by invoking the verifier through `$BASH` — the absolute path of the interpreter already running this script. No PATH lookup, no /usr/bin/env, and the verifier provably runs under the same shell. Both scripts keep their shebangs: they are still executed directly by the operator, where /usr/bin/env exists, and that path is unchanged. Consolidated into one `run_check` helper rather than patched at each of the three call sites (preflight, post-test verify, post-restart retry). A predicate open-coded at N sites regrows the bug at whichever site a later fix misses. Verification is on the SANDBOX tier only, deliberately: the dev host has /usr/bin/env, so a local run cannot reproduce the failure and its green says nothing about this fix. The dev-host suite is reported for completeness (30 passed) but the load-bearing evidence is the sandbox control/fixed pair. Claude-Session: https://claude.ai/code/session_01NNxEE4ktaHsXxNsxeWXWC5 Claude-Session-Id: 4b14058c-f3f5-4988-ae42-90910f7dbe43 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev
added a commit
that referenced
this pull request
Sep 8, 2026
…ce — 67 have a named catch, 0 earn a DROP (#1375) * docs(gates): inventory all 81 gates/tiers/batteries/hooks with evidence — 67 have a named catch, 0 earn a DROP clawgate task #525. Every gate, tier, battery and hook devrc runs, each row carrying either a named catch (session id, CI run, or commit sha) or the explicit string `no evidence found` with the search that was run, plus a KEEP/DROP/TIER verdict. Nothing that runs is changed. No gate, target, battery or hook is dropped, disabled or made conditional, and no per-path selection rule is implemented — both are the task's explicit non-goals, and the verdict column is a recommendation the operator decides on. Enumeration is DERIVED at scan time, not taken from the task's own list: 31 pytest targets (TARGET_FLOORS), 5 node suites (SUITES), 2 nix check derivations, 10 mutation batteries, 16 claude-hooks modules, 13 drift-check arms, the githooks pre-push gate and 2 Tekton checks. 81 rows. Base sha 112a522. METHOD — four transcript scanners over 5,836 .jsonl (6.8 GB), each with a negative AND a positive control, all reported in §2. This session's own transcript is excluded from every count. 🔴 One control FAILED and the table was not trusted until it was explained. scan_drift.py's rc=0 positive control returned 0. Cause read out of the source rather than assumed: drift-check.sh:3554 guards the `(rc=$rc)` line behind `if [ "$rc" != 0 ]`, so a clean run is structurally incapable of emitting that format. Re-run against the clean-path sentence: 34 sessions. Control passes. The co-requirement in scanner 1 is doing real work, and this is the measurement that shows it: on the same string, text-only finds 97 sessions and text+`toolDenialKind` finds 75. The 22-session gap is sessions that DISCUSSED a guard without being blocked by it. WHAT THE EVIDENCE SHOWS - 67 of 81 rows carry a named catch. All 10 mutation batteries; 11 of 14 bash-guard arms; 11 of 13 drift-check arms; 28 of 31 pytest targets red at least once; 4 of 5 node suites; 8 of 8 hooks that are actually guards. - 14 read `no evidence found`, each with its search recorded, and none of them on idleness grounds. Six are modules that are not gates and emit nothing to find; two are drift-check arms structurally unable to fire yet; three are never-red targets covering guards that HAVE fired; three have the evidence in a sibling tier. - 0 rows earn a DROP. THREE FINDINGS, none of them a drop: 1. `scripts/devhost-tests` is in TARGET_FLOORS (31 entries) and NOT in HERMETIC_TARGETS (30). It runs in 4 sessions where its neighbours run in 200-590, while its floor still counts toward the derived global floor. This is already the failure mode the task warns about — a target that silently stopped running — arrived at by omission rather than by decision. Decide it explicitly, either way. 2. No mutation battery is wired into any gate. `grep -rn "mutants-"` over gate.sh, run-tests.sh, run-node-tests.sh, flake.nix and githooks/ returns nothing. Defensible as manual one-shot instruments, but currently undocumented — say so in their headers so the next reader does not assume they run. 3. The proposed tiering's saving is UNMEASURED and smaller than it looks: the always-run set is 14,053 of the 21,202 pytest floor-sum (66%), capping the benefit at ~34% and only on single-subsystem diffs. Floors are not wall-clock; measure per-target time before building the selection layer. Also lands scripts/check-gate-inventory.py, which asserts criteria 3 and 4 over the table: 81 rows checked, 0 problems. It carries its own controls (`--self-test`: four malformed row shapes each rejected, a well-formed row accepted, and a scoping control), and it errors rather than passing when it matches zero rows — a vacuous green there would certify an empty table. It is deliberately NOT wired into any gate and NOT placed under scripts/tests/, since a file there would change that target's collected count and force a TARGET_FLOORS re-pin. Follow-on filed as clawgate task #528: add emit_invocation to the gates and guard hooks recording the OUTCOME (and, for bash-guard, WHICH ARM), so this is answerable from activity.events directly next time rather than by transcript proxy. An event recording only "the gate ran" would reproduce the exact gap this document had to work around, so #528 makes the outcome field a criterion and requires the emitter to be fail-open and demonstrated so. Note for the reader: criterion 1 names `ZacxDev/devrc`; this repo's origin is `innovation-upstream/devrc`. Read as "the devrc repo" and targeted at origin. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXxTANDijZTVxj398NrWBU Claude-Session-Id: 45e79797-ec0e-45c7-ad21-5d623afb297b * fix(gates): round 2 — the audit refuted two headline claims, and both were a SET inferred from behaviour instead of read from its registry Adversarial audit of #1375 returned 2 deploy-blocking and 9 should-fix findings. Both 🔴 are retracted in the document rather than quietly patched, because the shape they share is the transferable part. 🔴 RETRACTED 1 — the bash-guard arm table was wrong in BOTH directions. It was assembled from the guard's deny MESSAGES rather than from `guard_core._CLAUDE_CODE_CHECKS`, which is the list `bash-guard.py:60` (POLICY = "claude-code") actually runs. * `check_git_commit_to_main` was ABSENT from the table. Re-measured independently of the audit: 435 blocks across 270 sessions — the THIRD most-fired arm in the repo, and the one enforcing this repo's own 🔴 never-commit-to-main rule. * `rm -r $HOME` was listed as a bash-guard arm. It is not one: it lives in `_IRREVERSIBLE_CHECKS`, an opencode-only policy, under a 20-line comment arguing the exclusion and ending "Do not 'finish the job'." Measured 3 blocks / 1 session — from opencode's guard, not bash-guard. (The audit reported 0/1; re-measuring found 3/1, so the attribution was wrong, not the firing.) Why it mattered: round 1's keep-without-evidence argument read "a regex on a string already being matched for eleven other arms". For `rm -r $HOME` under claude-code that is false — it is not matched at all, so its zero was a POLICY fact, not a rarity fact. An operator persuaded by that argument might have moved `check_rm_rf_critical` into the claude-code policy, reversing a deliberate and argued decision. Corrected: 12 of 14 arms have fired; the zero-firing set is two. 🔴 RETRACTED 2 — row A31's finding (`scripts/devhost-tests`) was false on all three of its claims, and its proposed remedy would have redded the sandbox tier. * "arrived by omission" — `run-tests.sh:876-898` is a dedicated `DEVHOST_TARGETS` array under a 🔴 comment explaining the choice. * "silently stopped running" — `test_nvim_clipboard_osc52.py` parses that array and asserts the registration exists. * "its floor is counted into the global floor regardless" — false; `run-tests.sh:900` sets `TARGETS=("${HERMETIC_TARGETS[@]}")`. Round 1's OWN gate output said `floor: 21281 = sum of 30 per-target floors` against 31 targets. The refuting evidence was in hand and was misread. The tests need `nvim`, which the nix sandbox lacks — which is why they were moved out. No action; the design is correct as it stands. RE-ANCHORED c5e425c -> 01956bf. The audit found the base had moved: #1377 retired `initiatives` and `repo-cos`, so round 1 carried rows for two DELETED targets and proposed selection rules for two retired subsystems. TARGET_FLOORS 31 -> 29, floor-sum 21,202 -> 20,214, always-run share 66.3% -> 69.5%, cap on the proposed saving ~34% -> ~30.5%. 81 rows -> 79. Other corrections, each measured: * The documented search needle was WRONG. Runners emit TWO spaces (`FAIL <target> (`); the doc said one. Round 1's regex used `\s+` so its COUNTS were right, but a reader following the appendix got 0 for nearly every row and would have read that as confirmation — the positive-control failure the document preaches about, in its own appendix. * Row C3 counted READERS, not firings. `gate.sh:250` emits `GATE: RESULT=UNVOUCHED`; it emits `could-not-vouch` nowhere — that string lives in CLAUDE.md, which carries both of round 1's needles in one sentence. Re-measured: 18 sessions, not 241. The KEEP survives; the figure overstated ~13x. A fabricated-name negative control CANNOT catch this, and that limit is now stated in §2. * Instrument 3 cannot separate the two tiers: both runners emit an identical `PASS <dir> (` and three dirs are both a pytest target and a node suite. Those six rows are marked AMBIGUOUS and C2's reason no longer leans on a separation the instrument cannot provide. * §10's always-run triggers omitted `scripts/testlib/**` — referenced by 122 tracked files across 8+ targets, so the omission was a live instance of the risk §11 states. Added. Dropped `pytest.ini`/`pyproject.toml`: neither exists. * "8 of 8 hooks that are actually guards" matched no partition of the rows and contradicted the next paragraph. Replaced with the full accounting: of 16 E rows, 6 are not gates, 10 are, 5 of those can block, 5 are advisory. * Two rows were contaminated by the audit itself — it ran the gate with `--set all`, putting devhost verdict lines in the corpus. The audit session is now excluded from every count alongside the authoring session; citing it would make the document evidence for itself. CHECKER — three findings, all mutation-verified after fixing: * The scoping control was VACUOUS. Its fixture had no `## 3.` line, so `in_scope` was False from the start regardless of SECTION_END; deleting the branch it claims to cover left the self-test fully green. New fixture opens the scope first. Mutation-checked: with SECTION_END deleted the control now fails with ITS OWN error (rows=2, problems=1), i.e. it is reachable. * Evidence acceptance was vocabulary, not evidence — the bare word `CONFIRMED` and a bare `×N` count both passed. Decisive control from the audit: replacing all 81 evidence cells with `CONFIRMED` yielded RESULT: PASS. Now requires a lookup-able identifier (UUID session id, `agent-<12+ hex>`, a >=7-hex sha in backticks, or `#<digits>`). Same control now fails all 79 rows. * The zero-row guard was narrower than its docstring: a document truncated to ONE row passed. The document now declares its own size and tally in two machine-read markers; a mismatch, or a missing marker, exits 2. 🔴 And the tally marker immediately earned its place: the hand-written tally was wrong AGAIN this round (47/32 written, 49/30 counted) — as it was in round 1 (51/30 written, 49/32 counted) — while the row LISTS beside it were correct both times. It is now machine-checked rather than proofread. Negative control: feeding the wrong tally back in makes the checker go red. Gate: both tiers on the merged tree — named in the PR comment with the base sha. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXxTANDijZTVxj398NrWBU Claude-Session-Id: 45e79797-ec0e-45c7-ad21-5d623afb297b * fix(gates): main moved again mid-round — two NEW batteries, so 79 rows became 81 Re-anchored c5e425c -> 01956bf while round 2 was in flight. Not drift in the sense the appendix disclaims: two mutation batteries LANDED, so the inventory was missing rows, which is a coverage gap. D11 mutants-diagnose-disk-accounting.sh (ffac18f, #1366) — a gate for root-only bash nothing had ever run; THREE live defects found while writing it, all fixed. 19 mutants, all killed. D12 mutants-nebula-relay.sh (01956bf, #1272) — the audit found three blockers in the apply script, all downstream of rc 0. Both have a named catch, so: 12/12 batteries, 81 rows, KEEP 51 / TIER 30 / DROP 0. pytest floor-sum 20,214 -> 20,348; always-run share 69.5% -> 69.1%; cap on the proposed tiering saving ~30.9%. Both machine-read markers updated in the same commit, which is what they are for — `check-gate-inventory.py` fails on a stale marker rather than passing quietly. Verified: 81 rows checked, declared 81, tally counted 51/30/0, problems 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXxTANDijZTVxj398NrWBU Claude-Session-Id: 45e79797-ec0e-45c7-ad21-5d623afb297b --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev
added a commit
that referenced
this pull request
Sep 8, 2026
…ol check, and a guard passed while deploying a dangling symlink Round-1 adversarial audit of #1406. One 🔴, one 🟡, six 🟢; the two that ship behaviour are below, each with the measurement that made it a finding rather than a reading. 🔴 **`cairn validate` is no longer the mandated post-write check, silently and at exit 0.** After this PR `~/.local/bin/cairn` is the pinned OSS package, which reimplements `validate` on the READER's resolver instead of shelling `subsystem_touch.py`. Measured on the built package against the live cache: **76 bytes and rc 0** against the writer's **5,765 bytes** carrying `entry shape:`, `marker reachability:` and `dropped lines:` — the last meaning content is ALREADY LOST. Both are "green"; only one looked. So both skills now name the writer's own invocation, and `test_subsystem_touch.py`'s pinned-sentence ledger moves with it — it went RED on the old spelling, which is the mechanism that caught this. Exit codes differ too and both skills say so: the writer exits 3 on a malformed entry, the packaged client 5 (`EXIT_CORRUPT`; its own 3 is `EXIT_UNREACHABLE_NO_CACHE`). 🟡 **`test_the_package_is_passed_to_the_home_module_through_extraSpecialArgs` ran its two checks INDEPENDENTLY over the whole region**, so a decoy satisfied both while the deploy pointed elsewhere. MEASURED SURVIVED at f98be26: cairnPackage = pkgs.hello; cairnUnused = cairn.packages.${system}.cairn; 8 passed, and `home.file.".local/bin/cairn".source` became `${pkgs.hello}/bin/cairn` — home-manager's `insertFileEntry` does an unconditional `ln -s`, so that BUILDS and deploys a DANGLING symlink, the exact outcome the required-argument design claims to prevent. `cairnPackage = pkgs.hello;` ALONE was already killed, so the guard was narrower than its docstring rather than inert. Now the value assertion is bound to that name's own right-hand side, plus a `count == 1` binding check so the asserted binding and the threaded one cannot be different lines. Also `test_home_nix_REQUIRES_the_package_argument_with_no_default` read the header as `split("\n", 1)[0]`. A nix module header may legally span lines, so on a CORRECT tree it failed with "does not accept `cairnPackage`" — a red naming a cause the tree does not have. It now walks to the `:` that closes the argument set. `run-tests.sh`'s baseline comment claimed "main is red on its own" as a durable property. An audit round measured the OPPOSITE at the same commit (30 passed / 0 failed in `test_nebula_relay_apply.py` at 01956bf, hours later). Both readings are real: that suite reads LIVE HOST STATE, and #1272 shipped the sudo apply beside the check, so applying the relay flips it with no commit involved. Its result is keyed to the HOST, not the tree — the comment now says so and says re-derive, never quote. Regression matrix, measured both ways at f98be26 (pre-fix) and HEAD: | mutant | pre-fix | post-fix | |---|---|---| | the decoy above | **SURVIVED** (8 passed) | KILLED, on the rhs assertion | | `cairnPackage = pkgs.hello;` alone | KILLED | KILLED | | a second `cairnPackage =` binding | n/a (no such check) | KILLED, on `count == 1` | | home.nix drops the argument | KILLED | KILLED | | a legal MULTI-LINE header (a CORRECT tree) | **RED — false diagnosis** | green | | the skill reverts to `cairn validate` | n/a | KILLED, on the pinned sentence | 6/6 killed by their intended guard's own message, `PYTHONDONTWRITEBYTECODE=1`, every anchor required to match exactly once, tree diffed byte-identical against the battery's snapshot afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTRPN6dhkdf8kZJch3mne5 Claude-Session-Id: 50c86165-b6c3-47d8-9fd7-14e70bb637a9
ZacxDev
added a commit
that referenced
this pull request
Sep 9, 2026
…orting an exec fault as a config fault Round-1 audit of #1407 left two findings. Both are here. FINDING 1 — the fix it shipped had no regression guard, and its only coverage was on the tier that is structurally blind to a revert. MEASURED: revert #1407's `run_check` to a direct `"$CHECK"` exec and the dev-host tier still reports 30 passed, because the dev host HAS /usr/bin/env. Only `nix build .#checks.x86_64-linux.pytests` goes red. So `scripts/gate.sh --tier both` — the command most sessions actually run — went fully green on the exact defect #1407 existed to remove. An agent "simplifying" run_check back to what #1272 wrote would have seen green and shipped it. `test_the_verifier_is_never_execed_via_its_own_shebang` is a source assertion, so it fails wherever the suite runs. Pinned as a PAIR — the helper must exist AND no direct call site may reappear — because presence alone passes if someone adds a second, direct site beside the helper, and absence alone passes if the helper is deleted outright. `test_the_verifiers_exit_codes_are_a_closed_set` pins its load-bearing precondition: the {0,1,2} set is asserted on BOTH sides, so the verifier growing an `exit 3` fails rather than being silently reclassified as an exec fault. FINDING 2 — the misdiagnosing abort was unchanged. check-nebula-relays.sh only ever exits 0, 1 or 2. Every other code fell into the same `*)` arm as a genuine rc 2 and was reported as ABORT: the verifier could not read the current config (rc=126); fix that first asserting a fact about $CFG that is false in every one of those cases — and that is precisely how the /usr/bin/env fault presented, which is why it read as a config problem instead of an exec one. Split into `2)` (a real verifier refusal — unchanged message) and `*)` (it did not run). One `verifier_answered` classifier, used at both sites rather than open-coded twice. 🔴 The consequential site is the POST-REBUILD verify, not the preflight. There a non-verifier rc reaches `die` AFTER `nixos-rebuild test` has activated, so the EXIT trap rolls back a change that WORKED and prints the PERSISTED paragraphs for what was really a signal or exec fault. VERIFICATION Red -> green, and every guard watched failing: whole suite at HEAD 38 passed new guards vs pre-#1407 nix/system 4 of 6 fail (the shebang guard correctly passes — #1407 is now in main — and the rc-2 pin passes both sides by design, being a guard against WIDENING, not a regression test) Mutation sweep, PYTHONDONTWRITEBYTECODE=1, every mutation applied-verified (the count is asserted before substituting, so an unapplied edit cannot be scored as a caught mutant): CONTROL unmutated 8 passed revert $BASH -> direct exec killed by the shebang guard verifier grows `exit 3` killed by the closed-set test widen verifier_answered to 126 killed by post-rebuild[126] + closed-set drop site-B classification killed by post-rebuild[126,137] revert the preflight split killed by preflight[126,127,137] reword the die message killed by all 5 behavioural cases 🔴 The post-rebuild test exists BECAUSE of that sweep. Widening verifier_answered was first caught by the STRUCTURAL test alone: every behavioural case aborted at the preflight, so verifier_answered — used only at the post-rebuild site — never executed. A guard that is never reached is not a guard. `apply_beside_sequenced_verifier` (answer 1, then fail) is the only way to reach that site. Four mutants added to scripts/tests/mutants-nebula-relay.sh so this is re-derivable rather than believed. ⚠ The stub verifier goes through `write_exec`, NOT a hand-written shebang. A `#!/usr/bin/env bash` stub execs on the dev host and ENOENTs in the sandbox, which would make the test fabricate rc 126 from its own stub rather than from the code under test — circular, and passing for the wrong reason on one tier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNxEE4ktaHsXxNsxeWXWC5 Claude-Session-Id: 4b14058c-f3f5-4988-ae42-90910f7dbe43
ZacxDev
added a commit
that referenced
this pull request
Sep 9, 2026
…n this checkout (#1406) * feat(nix): deploy `cairn` from the PINNED flake package, not a symlink into this checkout `~/.local/bin/cairn` was an `mkOutOfStoreSymlink` to `devrc/scripts/cairn`, so this repo carried a second copy of a reader that now lives in its own extracted OSS repo. It is now `${cairnPackage}/bin/cairn` from a locked `github:ZacxDev/cairn` input (rev 9213726), threaded input -> outputs -> extraSpecialArgs -> nix/home.nix, which takes the argument WITHOUT a default so a cut thread is an eval error rather than a symlink to `/bin/cairn`. `cairn-who` deliberately does NOT move. It is devrc-only, absent from the OSS package, and resolves `scripts/lib/cairn_who.py` through its own `__file__` — so it keeps `mkOutOfStoreSymlink`. The two deploy modes now differ ON PURPOSE and every comment saying otherwise was rewritten; `scripts/cairn` and `scripts/lib/` stay in the tree (the writer and `cairn-who` import them) and are simply no longer what lands on PATH. NOT `inputs.cairn.inputs.nixpkgs.follows = "nixpkgs"`, and that is a decision, not an omission. cairn pins `pkgs.python312` because its Dockerfile is `python:3.12-slim` and its CI pins 3.12; a bare `pkgs.python3` there once followed nixpkgs to 3.14 and shipped an interpreter nothing in that repo had run its suite under. Following would rebuild the client against devrc's `nixpkgs-unstable`, which cairn's CI has never tested — the thing deployed would stop being the thing that was tested. The cost is a second nixpkgs node in the lock. CAIRN_MIRROR_ROOT is exported, and it is a bug fix rather than configuration. devrc's fork had the frozen mirror's path hardcoded; the extracted client reads the env var, and UNSET makes `doctor` report `frozen-mirror NOT-OBSERVABLE` — a status that contributes nothing, i.e. a check that was PASSING silently becomes a check that is not RUN. Measured on this host against the real 225-entry cache root: devrc scripts/cairn frozen-mirror OK packaged client, var unset frozen-mirror NOT-OBSERVABLE packaged client, var set frozen-mirror OK (byte-identical doctor output to the fork) `NIX_DEPLOY_WHY` in test_cairn_split.py pins that comment paragraph verbatim and went red on this commit by design; it is updated to the rewritten prose, not weakened. `test_cairn_is_deployed_out_of_store_not_as_a_store_copy` in test_cairn_cli.py was the second guard asserting the old truth — it now pins the package spelling exactly, rather than being loosened to "any deploy mode". New: scripts/tests/test_cairn_flake_pin.py (8 tests) pins the input, the lock entry, the absence of a nixpkgs follow (flake.nix AND the lock's own shape), the thread end to end, the two deploy modes AS ONE RELATIONSHIP, and the CAIRN_MIRROR_ROOT export. Red -> green matrix (all 8 red at 65d8bfb, green at HEAD), plus a mutation kill per guard with each guard's OWN message. Two instrument findings recorded in the file: the first `follows` guard matched only the dotted spelling and a block-form mutant walked past it, and `nix develop -c pytest` REWRITES flake.lock, silently reverting a lock-only mutant before pytest reads it — scored SURVIVED without ever running. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session-Id: 9662077e-18a9-405d-b4ce-f6697e6fa1bb * fix(nix): the required `cairnPackage` arg broke a check that evaluates home.nix alone Both defects were caught by the authoritative gate, not by review, and neither was visible from the diff. 🔴 `skills_mapping.py` EVALUATES `nix/home.nix` STANDALONE, and its stub set did not know about the new argument. Making `cairnPackage` REQUIRED (no default) is deliberate — a broken thread should be an eval error, not a symlink to `/bin/cairn` — but nix then refuses to apply the function at all, and the check reported `nix cannot evaluate /build/src/nix/home.nix`. That message reads as a verdict on `home.nix` when it is a verdict on the STUB SET, which is the expensive half: the guard's own text says FIX THE CHECK, do NOT delete it. `cairnPackage = {}` added, `{}` rather than a string because forcing it in an interpolation errors instead of silently yielding a plausible path. ⚠ THE COMMENT EXPLAINING THAT DID NOT FIT. `skills_mapping.py` carries a MAX_MODULE_BYTES = 7_400 ceiling whose own message says an overage is a question about which ambition crept back, never about raising the number. The file was 7,292 B — 108 of headroom — and the first draft of the comment added 742. It is now three terse lines (7,361 B, 39 of headroom) and the reasoning lives here instead. That ceiling is doing its job; do not raise it for prose. 🔴 THE SKILL DESCRIPTION GREW 64 CHARS AND BREACHED A ZERO-HEADROOM RATCHET — `assert 11256 <= 11192`, plus `MEASURED_ALL_TIER_A_CHARS 11366 != 11430` in test_skill_tiers.py and a second arm in test_skill_descriptions.py: four failures, one cause. devrc #1381 already recorded that this ratchet has exactly zero headroom and kept its edit length-neutral; this one did not. Fixed length-NEUTRALLY rather than by re-pinning the constants: the fragment `client `scripts/cairn`` (22 chars) becomes `client `cairn` on PATH` (22 chars), which is both accurate after the pin and the same length, so every pinned tier constant stays valid. The deploy detail it wanted to add belongs in the BODY, where it already is — a description's job is deciding whether the skill loads. MEASURED after: description 303 chars, unchanged from main; the four guards and both cairn suites 116 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E3KEUsTLLUhS37an52rn28 Claude-Session-Id: 9662077e-18a9-405d-b4ce-f6697e6fa1bb * test(gate): ratchet the scripts/tests floor onto the rebased tree, with the +8 attributed 12927 carried 149 of SLACK against this tree's 13076. Re-pinned to 13076 - min(50, max(1, 13076/20)) = 13026 — the AUTHORITATIVE gate's own printed count through the documented rule, AFTER rebasing onto origin/main (01956bf), per the ORDER note: a floor is a claim about the tree you measured. THE +8 IS ATTRIBUTED, NOT ASSUMED. The same gate ran on PLAIN origin/main in a separate detached worktree as the control: control (plain main 01956bf) collected=13068 passed=13048 failed=20 branch collected=13076 passed=13056 failed=20 The delta is exactly `test_cairn_flake_pin.py`'s 8 tests, and all 8 pass. 🔴 BOTH SIDES REPORT failed=20, every one in `test_nebula_relay_apply.py` (arrived with #1272). Identical on both sides, so main is red on its own and this branch adds no failure. That control is what makes the collected count trustworthy despite a red gate — without it, "the reds are someone else's" is a theory, and a theory that explains a failure is not evidence for it. ⚠ THE KNOWN-RED BASELINE WENT STALE TWICE IN ONE DAY. The `age` escrow/backup and `opencode` engine 7 that were red this morning are GONE — main fixed them — and 20 new ones arrived from a different commit. The floor comment says to re-derive this baseline rather than quote it, for that reason. Floor sum 20342 -> 20441 (+99); no FATAL from TARGET_FLOORS' two-way pin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E3KEUsTLLUhS37an52rn28 Claude-Session-Id: 9662077e-18a9-405d-b4ce-f6697e6fa1bb * fix(audit): round 1 — `cairn validate` stopped being the write-protocol check, and a guard passed while deploying a dangling symlink Round-1 adversarial audit of #1406. One 🔴, one 🟡, six 🟢; the two that ship behaviour are below, each with the measurement that made it a finding rather than a reading. 🔴 **`cairn validate` is no longer the mandated post-write check, silently and at exit 0.** After this PR `~/.local/bin/cairn` is the pinned OSS package, which reimplements `validate` on the READER's resolver instead of shelling `subsystem_touch.py`. Measured on the built package against the live cache: **76 bytes and rc 0** against the writer's **5,765 bytes** carrying `entry shape:`, `marker reachability:` and `dropped lines:` — the last meaning content is ALREADY LOST. Both are "green"; only one looked. So both skills now name the writer's own invocation, and `test_subsystem_touch.py`'s pinned-sentence ledger moves with it — it went RED on the old spelling, which is the mechanism that caught this. Exit codes differ too and both skills say so: the writer exits 3 on a malformed entry, the packaged client 5 (`EXIT_CORRUPT`; its own 3 is `EXIT_UNREACHABLE_NO_CACHE`). 🟡 **`test_the_package_is_passed_to_the_home_module_through_extraSpecialArgs` ran its two checks INDEPENDENTLY over the whole region**, so a decoy satisfied both while the deploy pointed elsewhere. MEASURED SURVIVED at f98be26: cairnPackage = pkgs.hello; cairnUnused = cairn.packages.${system}.cairn; 8 passed, and `home.file.".local/bin/cairn".source` became `${pkgs.hello}/bin/cairn` — home-manager's `insertFileEntry` does an unconditional `ln -s`, so that BUILDS and deploys a DANGLING symlink, the exact outcome the required-argument design claims to prevent. `cairnPackage = pkgs.hello;` ALONE was already killed, so the guard was narrower than its docstring rather than inert. Now the value assertion is bound to that name's own right-hand side, plus a `count == 1` binding check so the asserted binding and the threaded one cannot be different lines. Also `test_home_nix_REQUIRES_the_package_argument_with_no_default` read the header as `split("\n", 1)[0]`. A nix module header may legally span lines, so on a CORRECT tree it failed with "does not accept `cairnPackage`" — a red naming a cause the tree does not have. It now walks to the `:` that closes the argument set. `run-tests.sh`'s baseline comment claimed "main is red on its own" as a durable property. An audit round measured the OPPOSITE at the same commit (30 passed / 0 failed in `test_nebula_relay_apply.py` at 01956bf, hours later). Both readings are real: that suite reads LIVE HOST STATE, and #1272 shipped the sudo apply beside the check, so applying the relay flips it with no commit involved. Its result is keyed to the HOST, not the tree — the comment now says so and says re-derive, never quote. Regression matrix, measured both ways at f98be26 (pre-fix) and HEAD: | mutant | pre-fix | post-fix | |---|---|---| | the decoy above | **SURVIVED** (8 passed) | KILLED, on the rhs assertion | | `cairnPackage = pkgs.hello;` alone | KILLED | KILLED | | a second `cairnPackage =` binding | n/a (no such check) | KILLED, on `count == 1` | | home.nix drops the argument | KILLED | KILLED | | a legal MULTI-LINE header (a CORRECT tree) | **RED — false diagnosis** | green | | the skill reverts to `cairn validate` | n/a | KILLED, on the pinned sentence | 6/6 killed by their intended guard's own message, `PYTHONDONTWRITEBYTECODE=1`, every anchor required to match exactly once, tree diffed byte-identical against the battery's snapshot afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTRPN6dhkdf8kZJch3mne5 Claude-Session-Id: 50c86165-b6c3-47d8-9fd7-14e70bb637a9 * fix(audit): round 3 — a guard that went VACUOUS, a pin that was walkable, and a comment refuted by a commit in its own merged tree Round-2 delta audit of `f98be263..b79cf63` returned no 🔴 and eight 🟡. Three of them were defects in round 1's own fix round; this closes those three, plus the operator's decision on the writer-invocation fork. The other five are recorded on the PR as open-by-decision. 🔴 **The header walk fell back to the WHOLE FILE and nothing said so.** Round 1 replaced `split("\n", 1)[0]` — which false-RED a legal multi-line header — with a bracket walk that, when the depth never returned to 0, simply ran off the end leaving `header` at its initial value: the entire file. `cairnPackage` occurs twice in the module BODY, so `THREADED_NAME in header` then passed while the argument was dropped from the header entirely. MEASURED at `b79cf63a`: a legal multi-line header that drops `cairnPackage` and carries one unbalanced `(` in a prose comment leaves the suite at **8 passed**. Fixing a false red opened a path to a false green. `_module_header()` now skips `#`/`/* */` comments and `"…"`/`''…''` strings so none of them carry depth, decides "opens with an argument set" on the first CODE token rather than `lstrip()` (which strips whitespace, not comments, and false-RED any file with an explanatory line above its header), and **raises** rather than falling back. 🔴 **The moved ledger pin named FLAGS, so it was walkable back to the defect it guarded.** `--store … --validate --scope <scope>` bound nothing to the writer. MEASURED at `b79cf63a`: routing those same flags at the packaged client leaves the suite **green at 79 passed** — round 1's 🔴, fully re-opened. The pin is now the whole normalised invocation, `cairn sync && cairn-validate --scope <scope>`, with whitespace runs normalised on both sides so a rewrap is not read as a deletion. 🔴 **The `run-tests.sh` comment asserted a diagnosis that `4f12fcef` refutes — and `4f12fcef` is an ancestor of HEAD but NOT of round 1's commit**, so round 1 wrote its theory before the refutation existed. It said `test_nebula_relay_apply.py` is "keyed to the HOST, not to the tree". It was a TIER split with a TREE cause: `apply-nebula-relay.sh` dispatched on a `#!/usr/bin/env bash` shebang, the nix sandbox has no `/usr/bin`, every exec died `bad interpreter`, and the script reported `ABORT … (rc=126)` while the dev-host tier stayed green. Retracted in place, cross-referenced to this file's own tier notes. Comment-only — verified by `git diff -U0 | grep -v '^[+-]\s*#'` being empty. Re-measured: 30 passed / 0 failed on the dev-host tier at `b79cf63a`. **`cairn-validate` ships, on the `cairn-who` seam** (operator decision). The write protocol has to name the writer, and the only spelling that ran was an absolute `python3 /home/…/devrc/scripts/lib/subsystem_touch.py` — a checkout path baked into a protocol whose point is that agents work in other repos; `cairn/SKILL.md` spelled it bare, which is not on PATH and exits 127. The launcher declares no parser: two defaults are PREPENDED to the caller's argv, because argparse's last-occurrence-wins gives the caller the override for free while a scan for an existing `--store` would be a partial re-copy of argparse's rules. `mkOutOfStoreSymlink`, not a store copy — `.resolve()` must land beside `scripts/lib/`, and no package ships the writer. ⚠ **Round 1's own figure was superseded by re-measuring rather than carried.** Both clients, same scope, same moment: the packaged `validate` writes **0 bytes to stdout** (77 B of banner on stderr) and exits 0; the writer writes **5,766 B** with all three contract blocks. "76 bytes of output" understated it — a caller reading stdout gets NOTHING. Every citation of the pair now says so. Regression matrix, red at `b79cf63a` / green at HEAD, on the REAL `nix/home.nix`: header drops the argument + unbalanced `(` in a comment → **8 passed (vacuous green)** → RED with `does not accept 'cairnPackage'`; leading file comment on a correct tree → **RED (false diagnosis)** → 11 passed. Plus 8 new tests red at base with their own messages. Mutation battery, `PYTHONDONTWRITEBYTECODE=1`, every anchor asserted to match exactly once, tree restored between mutants. 11 mutants, 11 killed by their intended guard's own message. ⚠ **A-M2 (reinstate the whole-file fallback) SURVIVED on the first attempt and is reported rather than hidden**: the vacuous-case fixture closes correctly once comments stop carrying depth, so it never reached the fall-off-the-end branch — an unreachable guard. `test_an_UNCLOSED_argument_set_…` was added with a header that has no closing `}` at all, a case no earlier assertion rejects, and the mutant then died. B3 is a control: a cosmetic rewrap of the pinned command stays green, so the normalisation is doing work rather than the pin being brittle. D-M2 swaps `--validate` for `--census` and dies on the contract-block assertion — the case a structural check type-checks straight past. `claude/skills/subsystem-index/SKILL.md` 42,269 → **41,591 B** (`HARD = 40_960`). 678 B cut. ⚠ Under the cap is arithmetically unreachable from round 1's block alone: the file was already 31 B over before round 1 touched it, so deleting that block entirely lands at 40,991. The residue is pre-existing prose — 🟡7, out of scope. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTRPN6dhkdf8kZJch3mne5 Claude-Session-Id: 50c86165-b6c3-47d8-9fd7-14e70bb637a9 * fix(gate): three reds the MERGED tree found — a seam ledger doing its job, and a guard blind to the tier it runs in The merged-tree gate on `4b27c63c` came back `failed=3`, all three from round 3's own additions and none from `origin/main`. Each is a different lesson. 🔴 **`test_store_root_ledger` went red because `cairn-validate` is a NEW router through `subsystem_read_store` — that is the seam guard WORKING**, not an obstacle. Its ledger fails when the set of routers GROWS as well as when it shrinks, precisely so a new reader cannot quietly start answering "where do I read?" for itself. A row was added saying what it reads and why: the launcher routes here for the SAME reason the ledger's three historical regressions existed — `subsystem_touch`'s own `--store` default is the FROZEN pre-cutover mirror, so a launcher that inherited it would parse the PRE-write bytes. 🔴 **`test_cairn_validate_defaults_its_store_to_the_SYNCED_CACHE_not_the_mirror` was structurally incapable of passing in the sandbox tier, and green on the dev host is what hid it.** It asserted `f"store: {expected}"` on STDOUT. That holds where the cache exists and the tool takes its success path; the `nix build` tier's `$HOME` is `/build/home` and carries no `~/.cache/subsystem-store`, so the tool exits down the not-found path and names the resolved root on STDERR instead. The claim is WHICH store the launcher chose, never whether one exists — so it belongs on the combined output. Pinning it to one stream made a guard that could only ever be green in one of the two tiers this suite runs in. ⚠ Round 3 said of these tests: *"I believe they are sandbox-safe, but that is reasoning, not a measurement."* It was reasoning, and it was wrong. MEASURED AT TWO POINTS, because one is not a general claim: green with a real cache root, and green under a `$HOME` verified to have none. The mutant that drops the `--store` prepend is KILLED at BOTH, on this guard's own message — previously the negative half did not exist at all, so a launcher that inherited the writer's default could have passed whenever both paths were printed. It is now asserted explicitly: the frozen mirror's path must NOT appear. 🔴 **`test_runtime_shebangs` flagged a spelled `"#!"` in the new launcher test.** Rather than register a string in its allowlist — which that file reserves for sites solving the problem a different, VERIFIED way, not for going green — the assertion now pins the RELATIONSHIP: `cairn-validate`'s interpreter line must equal `cairn-who`'s. Both are `mkOutOfStoreSymlink` launchers invoked as bare commands from PATH, so they must agree on how they find an interpreter, and asserting one in isolation would pass while the two drifted apart. The literal disappears as a consequence rather than as the goal. 112 passed across the three affected suites; mutation control restored byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTRPN6dhkdf8kZJch3mne5 Claude-Session-Id: 50c86165-b6c3-47d8-9fd7-14e70bb637a9 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
relay.relaysin a nebula config is the list of mesh IPs peers may use to relay packets TO this host — not the list of relays this host uses to reach others. That is whyuse_relays: trueon its own changes nothing. Workbench rendersrelays: [], so no peer can relay to it and a peer that cannot hole-punch (CGNAT, symmetric NAT, off-LAN) simply never connects. That is the blocker behind clawgate #497.This PR ships two scripts:
nix/system/check-nebula-relays.shnix/system/apply-nebula-relay.sh/etc/nixos/configuration.nix, thennixos-rebuild test→ verify →switch→ verify🔴 The second one is a root-privileged, system-mutating script. An earlier version of this body said the PR "ships only the check … does NOT edit that file and does NOT rebuild … no sudo". That was true at
405ecfd4and false frome894fe10onward — the apply script has been on the branch since. Corrected here; read the apply script as what it is.Why the check exists separately
relay.relaysusing the stdlib only, so it has no PyYAML dependency (import yamlis not satisfiable from thepython3asudo bash …actually gets here).lighthouse.hostscarries the same mesh addresses, so a pattern like^\s+- <relay>$matches a config whoserelays:is empty. Measured: it reported a match on the pre-change file and would have certified a change that had not happened. The trap is in the header and asserted in the self-test.--self-test, which must read arelays: []fixture and a populated one correctly before any PASS is printed.0advertised ·1not advertised ·2cannot determine. Every rc-2 path also prints one machine-readableREASON: <token>line, so a caller can branch on the condition instead of pattern-matching prose. A--self-testfailure is rc 2, not rc 1 — 1 means "the relay is absent", a different claim. (An earlier version of this body said rc 1. Corrected.)Round 3 — the eight findings from the audit of
e894fe10..e7f2e45aswitchregisters the profile and installs the bootloader before activation, so a failed activation still moved the machine — measured: generation 387 heldrelays: - 10.42.0.2after the script reported a clean rollback. The order is nownixos-rebuild test→ verify →nixos-rebuild switch→ verify.testactivates without registering a generation or touching the bootloader, so a failure there really does persist nothing. The trap tracks five states and its message distinguishes never activated · test failed · activated-not-persisted · switch attempted (profile may have moved, withreadlink /nix/var/nix/profiles/systemto check) · persisted.NEBULA_NETNEBULA_NET=travelreportedDONE … advertised by nebula@travel.servicewhile the four lines landed in the mesh block. Theservices.nebula.networks.<NET> = {block is now located first (brace depth, comments and string bodies skipped) and the anchor pair is only accepted inside it. No unique declaration, a block that never closes, or an anchor count ≠ 1 inside the range each abort./tmpmktemp -d(0700) for the verifier capture, removed by the trap; the patch temp is amktempsibling of$CFGso the finalmvstays atomic.elseexists,cp's status is checked, and failure prints🔴 ROLLBACK FAILED — your config is still patched at <path>with the exact four lines to delete by hand.[ -f "$CFG" ]accepted a symlink andmvdestroyed it/tmpone. Compared viareadlink -f, not[ -L ], so a symlinked directory component is caught too; the error names the resolved target and the one-flag fix.if False:left--self-testgreen. Fixture 5 keeps its (real) shape with an honest comment, and a new fixture 5b — a later top-levelam_relay:— kills the mutant with its own message. 🔴 The comment states plainly thatpkgs.formats.yamlcannot emit that shape, so 5b pins the parser's structural correctness, not a reachable production config.systemctl restartfires only onREASON: unit-process-disagree. Six other rc-2 conditions (inactive unit, unreadable-config, parser failure, …) no longer drop every mesh session for nothing.Nits also fixed: unused
indentandlocal spellingsremoved; the preflight tool list now names every binary the script execs (diff,mktemp,tr,cut,wc,cp,mv,date,readlink,grep, …); both scripts now warn that applying a relay restarts the mesh the operator may be connected over.F-H — the scripts now have repo-gate coverage
They had none:
--self-testwas never invoked byscripts/gate.shand the apply script had no automated exercise at all.scripts/tests/test_nebula_relay_apply.py— 30 tests, in the existing hermeticscripts/teststarget, soscripts/gate.shruns them. Nothing in it can reach a realnixos-rebuild,/nix/var/nix/profiles/systemor/etc/nixos: the config is a fixture undertmp_pathreached viaNEBULA_CFG, andid/ip/systemctl/nixos-rebuild/nix-instantiateare shims onPATH.-config <rendered.yml>and thenixos-rebuildshim re-renders that yml from the.nixthe script just patched. Every fake process is stopped by its own PID; no pattern ever reachespkill.scripts/tests/mutants-nebula-relay.sh— the mutation battery, in the tree so "mutation-verified" can be re-derived rather than believed. Same shape as the existingmutants-*.sh: CONTROL-CLEAN / CONTROL-KILL / CONTROL-DETECTOR, each mutation applied-verified by occurrence count, and each mutant naming the test that must kill it and the message it must die with — a mutant that dies to a different test's error reportsWRONG-KILLERrather than counting as covered.Coverage matrix (all red before the fix, green at HEAD)
nixos-rebuildnever invokedrelayslistnixos-rebuild testfails (F-A)switchnever reachedtestsucceeds, verifier failsswitchfails after a goodtestreadlink …; never claims nothing was persistedCFGis a symlink (F-E)CFGunder a symlinked directoryNEBULA_NETwhose block lacks the anchor (F-B)NEBULA_NET/tmppath live while the verifier runs (F-C)EGRESSES THE RELAYpresentcheck --self-test(F-H)Mutation results — 22/22, one declared survivor
20 mutants killed by their named test, with that test's own message, plus CONTROL-CLEAN / CONTROL-KILL / CONTROL-DETECTOR green (22/22 including the controls). Two entries are recorded honestly rather than rounded up:
M-X-1-line-count-guard-off([ "$added" = "4" ]) is a declared survivor: the awk pass already exits 3 unless it made exactly one insertion, and one insertion is always four lines, so no input reachable through the script's own guards can move$added. Kept as defence in depth, not counted as coverage. The battery asserts it survives, so if the justification ever stops holding the battery says so.M-FB-2-patch-pass-unscopedis killed, but by the awkn != 1guard (two anchors ⇒ two insertions ⇒ abort), not by the assertion naming the wrong block. So the range guard inside the patch pass is defence in depth behind the scoped anchor count; the battery scores it on the message it really dies with.Both scripts also exercised end-to-end against a copy of the live
/etc/nixos/configuration.nix(not the file itself): block located at lines 702–733, anchor unique inside it, +4 lines landing in the mesh block,rebuild log == ["test", "switch"].shellcheck -S warningis clean on both.One thing the gate turned up that is NOT this PR's
A full
scripts/gate.sh --tier pyteston the branch reports two failures. One was mine and is fixed (test_no_test_writes_a_usr_bin_env_shebang_at_runtime— the harness wrote its own#!/usr/bin/env bash; it now goes throughtestlib.mockbin.write_exec, and the shim bodies are POSIX sh, controlled withdash -nand a full behavioural re-run under dash, because this host's/bin/shis bash and would hide a bashism until the sandbox).The other —
test_clawgate_task_interview_guard.py::test_a_body_file_written_by_a_heredoc_on_the_same_line_is_read— is not from this work.scripts/claude-hooksis byte-identical between this branch's tip before my commits and after (ls-treeOID2a363038both sides), andmainhas since deleted that test along with 85 lines of the guard. So it is a branch-is-behind-mainartifact, and the merge resolves it.Gating was therefore done on an integration tree (
origin/main+ this branch), not on the branch alone:scripts/gate.sh --tier pytestthere is PASS, 30/30 targets,TOTAL collected=21508 passed=21505 skipped=3 failed=0, withscripts/testsat 12477/12477.Not verified
/etc/nixos/configuration.nixis unmodified,nixos-rebuildwas never run in any form, and/nix/var/nix/profiles/systemwas not touched or rolled back. The pending generation from the earlier run is still in place, by the operator's choice.nixos-rebuild test's no-profile-no-bootloader property is taken from its documented behaviour and from the F-A measurement ofswitch; it was not re-measured on this host, because doing so means running a rebuild./tmpsymlink attack is shown closed by observing that no predictable path is live while the verifier runs (plus a source-level check). A literal symlink-planting reproduction against the fixed script was not performed — it needs the script's$$in advance.🤖 Generated with Claude Code
https://claude.ai/code/session_01RMYjPVhL74So5kqc6XTzCW