fix(nebula): drop the dead :443 lighthouse fallback, and mark the script that added it superseded

[ZacxDev]

Removes a nebula fallback that cannot ever work, and marks the script that added it superseded.

The defect

apply-nebula-443.sh appended <lighthouse>:443 to the prod lighthouse's staticHostMap as a fallback for networks blocking UDP 4242. Measured — it is inert:

fact	source
the prod nebula lighthouse listens on 4242 only	clusters/production/apps/nebula/lighthouse/lighthouse-config.yaml → listen.port
443/udp on that host is WireGuard	k0s/host-firewall/relay-firewall.sh → 443/udp (wireguard)
no 443→4242 redirect exists	grepped across homelab-infra

Nebula handshakes a socket that does not speak nebula and gets nothing back.

The harm is not the wasted handshakes — it is that the entry reads as a fallback. In the one situation it was written for, there is none, while the config asserts otherwise. That matters now: the operator is about to be off-LAN for months and nebula is the only remote path to the workbench. Tailscale is the real answer to blocked-UDP (clawgate #497); this just stops the config claiming a safety net it does not have.

Shape

Follows PR #1272 (rank 1 of the same effort): --check mode that is read-only and rootless and reads the running unit's config; a host guard on the mesh address because workbench and portable both answer nixos; temp-patch → nix-instantiate --parse → unique backup → move → rebuild, with a trap that restores on any later failure and distinguishes "file restored, system already switched"; idempotent.

🔴 The lighthouse IP is not written into this public repo. The script derives it from the :4242 entry beside the :443 one and refuses unless both name the same host — which is also a correctness check, since a :443 beside a different host is not this entry.

sed, not awk — 3-arg match() is a gawk extension. sed reports no substitution count, so the edit is verified from the result: exactly one changed line, line count unchanged, no :443 surviving.

Verified

Substitution, on a fixture: 1 line changed · :443 gone · trailing comment preserved · line count unchanged · output parses as Nix · second pass is a no-op.
🔴 Negative control: a "10.42.0.1" = [ "a:4242" "b:4242" ]; row — two addresses, neither :443 — is left byte-identical. That is the row a looser pattern would have clobbered.

--check on both hosts, because a checker that cannot go red is not a checker:

workbench (no :443)        -> PASS  rc 0
portable  (dead entry live)-> FAIL  rc 1   naming the address

Not verified

The apply path has not been run. /etc/nixos/configuration.nix on the portable is root-only, so unlike on the workbench the patch anchor could not be checked against the real file. The script therefore derives its target at runtime and aborts rather than guesses if the shape is not what apply-nebula-443.sh leaves behind — including an explicit refusal when a :443 string is present in some other form.

To land it: sudo env "PATH=$PATH" bash nix/system/apply-nebula-drop-443.sh on the portable.

Refs clawgate #497 (rank 2) · claim nebula-pre-departure-hardening-2 · follows #1272.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TcDJ4ccN8savidfXbHPoqh

[ZacxDev]

Round 1 audit — fixes pushed as c5bd8b86

Round 1 found no 🔴 and independently verified the PR's factual premise (lighthouse listens 4242-only; 443/udp is WireGuard; no 443→4242 redirect; no IP committed). All 12 🟡 and the nits are addressed.

One fix did not come from the audit — it came from writing the test: the sed matched a :4242/:443 pair regardless of whether the two addresses named the same host. Only the shell guard's ordering made that safe. The expression now carries a \2 backreference and discriminates on its own.

Verified: 14/14 new pytest cases; then the fixed script driven end-to-end with shimmed id/ip/systemctl/nixos-rebuild — pair + unrelated nginx :443 → rc 0 with the nginx line kept; already-gone + unrelated :443 → rc 0 no-op; --dry-run → rc 64 with the config untouched; mismatched hosts → rc 1 refused. The success path exercised the new /proc-reading --check tier against a real process.

1. #1/#2 both `:443` searches now go through `hostmap_443()`, scoped to staticHostMap-shaped lines instead of the whole file; the "re-running is a no-op" claim in the header is restored to true.
2. #3 argument handling is an explicit `case` with `exit 64` on anything unrecognised; `--dry-run` no longer runs the destructive path.
3. #4 post-switch verification distinguishes rc 0/1/2, retries once on rc 2, and no longer asserts ":443 still advertised" when the verifier could not read anything.
4. #5 `--check` reads `/proc/<MainPID>/cmdline` instead of `systemctl cat`, so a deferred restart cannot read as PASS.
5. #6 `--check` requires the extracted static_host_map block to contain at least one `:4242`, else rc 2 — a positive control against silent always-PASS.
6. #7 `SWITCHED=1` is set BEFORE `nixos-rebuild switch`, so a partial switch no longer prints "the system was never switched".
7. #8 preflight dry-builds the CURRENT unpatched config, and the header states that a switch applies every pending edit in configuration.nix.
8. #9 a post-switch reachability probe was added, deliberately advisory because lighthouse ICMP is measured-flaky on this fleet.
9. #10 CLAUDE.md no longer names the superseded script as the exemplar; it points at apply-nebula-drop-443.sh.
10. #11 apply-nebula-443.sh now refuses to run behind an explicit override env var, rather than only carrying a banner comment.
11. #12 scripts/tests/test_nebula_drop_443.py added (14 cases), following test_freeze_instrumentation.py; findings #1, #2 and #3 would each have been caught by it.
12. nits: `diff` and `wc` added to the tool preflight; `nix-instantiate --parse` runs on $CFG first as a control; mode 100755.
13. NOT from the audit — the sed now uses a `\2` backreference so it requires both addresses to name the same host.

Still not verified, unchanged from the PR body: the apply path has never run against the portable's real /etc/nixos/configuration.nix, which is root-only and has never been read. The script derives its target at runtime and aborts rather than guesses.

[ZacxDev]

Round 2 audit — fixes pushed as 59d8dd79 + ec19192d

Round 2 found a 🔴 I introduced in round 1: check_one; check_rc=$? is dead code under set -e. The bare call exits the shell on any non-zero return, so the entire three-outcome verification — round 1's headline fix — was unreachable for rc 1 and rc 2. Confirmed independently before fixing: stub rc 1 and rc 2 both exited with the next line never reached.

Worse than inert: a transient post-switch read failure rolled back a change that had applied, re-adding the dead :443 entry, and printed advice that would re-apply it to the running system. Fixed at both call sites — the audit named one; the retry a few lines below had the identical shape.

Resolved, an open question from the round-2 report: the target machine's /etc/nixos is neither a flake nor a git repo (checked on the host), so nixos-rebuild switch reads configuration.nix directly and the uncommitted-edit concern does not apply.

Verified: 19/19 pytest, then driven end-to-end with shims — F1: retry runs, honest COULD NOT VERIFY reached, no false ":443 still advertised", rollback still occurs (correct for an unverified state). F2: multi-line entry refused, no ALREADY GONE, entry intact. Round 1's four cases still pass.

1. F1 both `check_one` call sites now use `check_rc=0; check_one || check_rc=$?`, so the rc 0/1/2 block and the 5s retry are reachable.
2. F2 the two `:443` searches are split: `hostmap_443` (scoped, post-patch survival) and `any_443_addr` (wide, refuse-rather-than-guess); a hand-wrapped multi-line entry is now refused instead of reported ALREADY GONE.
3. F3a the preflight control claims EVALUATION only, in its output and in the header, because `dry-build` does not build.
4. F3b the control is skipped when NEBULA_CFG is not /etc/nixos/configuration.nix, rather than asserting about a file nixos-rebuild does not read.
5. F5 the control moved below the no-op checks, so a run with nothing to do pays no evaluation and cannot die on an unrelated pending edit.
6. F6 a missing `ping` is reported distinctly from a failed probe, rather than being attributed to fleet ICMP flakiness.
7. F7 the handoff-doc count corrected from 14 to 9; the unvalidated repo-wide figure dropped rather than corrected.
8. F4 the scoping guard is now behavioural — the helpers' real regexes are extracted from the script and run against fixtures — and the module docstring's test names are pinned by a test.
9. Non-findings from the round-2 report, both being claims: the SWITCHED comment's activation order (stop -> activate -> start), and CLAUDE.md's "nine of these" -> 17.
10. CLAUDE.md's exemplar sentence, which still carried the dry-build overclaim after the script's own header was corrected; the errexit rule was added to what is worth copying.

Still not verified, unchanged: the apply path has never run against the portable's real /etc/nixos/configuration.nix (root-only, never read).

[ZacxDev]

Round 3 audit — fixes pushed as 82054827

Round 3 found a 🔴 in my own test fixtures: they used a globally routable IP literal (in no documentation range), and the repo's own test_no_public_ips.py fails on it. Merging would have left main's pytest tier red. Reproduced before fixing — 1 failed at head, 15 passed at the round-2 base — and now 203.0.113.9 (TEST-NET-3), which the rest of the file already used.

🔴 The real lesson is the verification claim. Round 2's message said "VERIFIED: 19/19 pytest" — true, and structurally blind: it named one file, and the failure was in another. A green run covers only what it ran. This round runs the adjacent set: 187 passed across test_nebula_drop_443, test_no_public_ips, test_no_client_hostnames, test_nix_read_paths, test_doc_path_rot.

🔴 And my first fix for R3-2/R3-3 re-opened F2. Block-matching alone returns an empty block when staticHostMap = { cannot be located, and an empty block rendered as "no :443 anywhere" — the round-2 finding again, one round later. Caught by re-running the round-2 regression drive after the rewrite, not by reading it. The shipped version is a union of three scopes plus a positive control that refuses when the file mentions staticHostMap but the block will not brace-match.

Footnote worth recording: posting this comment was blocked twice by the repo's own pre-commit guard for quoting the offending literal while describing the finding. The guard is right, and it is the same class of check that caught R3-1.

1. R3-1 both fixtures use TEST-NET-3; test_no_public_ips passes, and the adjacent gate set (187 tests) is run rather than a single file.
2. R3-2 any_443_addr no longer matches a WireGuard peer `endpoint =` or an nginx `proxyPass`; it scopes by context, not by address shape.
3. R3-3 any_443_addr now catches DNS-name and IPv6 entries inside the staticHostMap, so it is no longer narrower than the scoped helper it was split from.
4. The R3-2/R3-3 fix carries a positive control: it refuses when $CFG mentions staticHostMap but the block cannot be brace-matched, rather than reporting a clean file it did not read.
5. R3-4 the CLAUDE.md errexit rule now states the whole form `rc=0; check_one || rc=$?` and says why each half is load-bearing under set -e and set -u respectively.
6. R3-5 the preflight control always runs; the false "nixos-rebuild never reads $CFG" skip reason is gone, replaced by an explicit warning when $CFG is not the entrypoint.
7. R3-6 test_check_pattern_does_not_match_a_longer_port lifts the pattern from the script instead of retyping it.
8. The helper guards execute the SHIPPED function definitions, extracted from the script, rather than re-typed patterns; two extraction bugs found by running it now assert loudly instead of failing obscurely.

Still not verified, unchanged across all three rounds: the apply path has never run against the portable's real /etc/nixos/configuration.nix (root-only, never read). Both regression drives — round-1's four cases and round-2's F1/F2 — remain green.