feat(tailscale): the independent second path — and the closure preflight #1272 left open

[ZacxDev]

Rank 4 of the pre-departure hardening. Nebula is currently the only remote path to the workbench, and the operator leaves the LAN for months in ~3 days. Tailscale is the second door: different control plane, different relays, no shared component with nebula.

Two files, both in the idiom of apply-nebula-relay.sh / check-nebula-relays.sh.

nix/system/apply-tailscale.sh

Adds services.tailscale to /etc/nixos/configuration.nix, then switches.

Role discrimination. 🔴 hostname CANNOT tell these machines apart — both answer nixos. The guard is the nebula mesh address, unique by construction: 10.42.0.30 → subnet router (useRoutingFeatures = "server", IP forwarding, advertises 192.168.50.0/24); 10.42.0.100 → plain client (useRoutingFeatures = "client", which is what sets checkReversePath = "loose" so accepted subnet routes actually work). Anything else aborts before any write.

🔴 The closure preflight — the actual point of this PR

nixos-rebuild switch applies everything pending, not just your delta. A 4-line nebula edit once triggered a 26.05 → 26.11 jump that took 2h16m and rebuilt the world (largely wine-wow-11.0). That was recorded as an open defect on #1272 and never fixed. This is the fix.

Three independent measurements before switching, all from a temp file — /etc/nixos is untouched until they pass, so a refusal has nothing to roll back:

1. dry-build run TWICE, against the current config and the patched one. Pending work is reported separately from the tailscale delta, so a channel bump can neither be blamed on tailscale nor hidden behind it.
2. The nixpkgs release string, extracted by one implementation from both store-path basenames and cross-checked against /run/current-system/nixos-version — without that control an extractor that is wrong on both sides agrees with itself and waves a release jump straight through. A release change is refused unconditionally.
3. The closure as SETS (nix-store -qR | comm -3), not a parse of diff-closures — that command prints nothing when closures match, so an empty result is indistinguishable from a command that never ran.

Measured on the workbench, 2026-09-07:

	build	fetch	download
PENDING, with no change at all	40	24	159.4 MiB
TOTAL with tailscale	47	25	177.0 MiB
DELTA (tailscale only)	+7	+1	+17.6 MiB

The +1 is tailscale-1.102.3; the +7 are regenerated unit/etc/activation derivations. The thresholds are calibrated from those numbers, so the gate refuses on this host today — correctly: 40 queued derivations are not what anyone asked for. It names the clean fix (run nixos-rebuild switch as its own deliberate operation first, then re-run) and prints --allow-world-rebuild.

🔴 readlink -f on both sides, always. /nix/var/nix/profiles/system is a symlink to another symlink — measured, it reads system-389-link, a bare name — while /run/current-system points straight at the store. Single-level readlink returns a name on one side and a store path on the other, so they can never compare equal and the mismatch is reported forever. _store_path resolves fully and refuses anything outside /nix/store, so an empty string cannot masquerade as agreement.

nix/system/check-tailscale.sh

Read-only, no sudo, runnable before and after.

🔴 It reads live runtime state, not the nix config and not the unit file — tailscale status --json, tailscale debug prefs, /proc/sys, ip route. systemctl cat exits 0 for a dead unit, so a config that built but never activated reads as applied; that is exactly why check-nebula-relays.sh was rewritten, and this does not repeat it. systemctl is-active is printed as context with no verdict keyed on it.

🔴 ADVERTISED and APPROVED are two different claims and both are printed. A route the node advertises but which has not been approved in the admin console carries no traffic, and from the node's side that is indistinguishable from success. Approval is read from Self.PrimaryRoutes, which only the control plane populates.

🔴 Node key expiry is reported with its date and days remaining, and being enabled at all counts as outstanding — the 180-day default is shorter than the trip and the lapse is silent.

Four distinct exit codes: 0 all good · 3 node correct, an admin-console step outstanding · 1 node-side failure · 2 cannot determine. 3 exists so apply-tailscale.sh does not roll back a correct switch because a browser tab has not been clicked yet.

Role auto-detection degrades honestly: if nebula.mesh has no address — the expected case when tailscale matters most — it refuses and prints the --role remedy rather than guessing.

What was tested

• Both --self-tests green. 8/8 mutants killed on the verifier (substring route matching, PrimaryRoutes falling back to the node's own advertisement, both spellings of disabled key expiry, days arithmetic, three refuse-vs-answer paths) and 12/12 on the apply script (dry-build singular/plural/zero/GiB, .drv stripping, dashed hostnames, readlink vs readlink -f, the store-path check, and both branches of the gate) — each by its own named case, with an unmutated positive control, and two SURVIVED results traced to a broken mutation harness and re-run rather than reported.
• The verifier driven end-to-end against a mock tailscale: rc 0, 1, 2 and 3 all watched, including the forwarding and lanroute FAIL branches (TS_PROC_ROOT exists so the forwarding check's failing branch can be driven — neither host has IPv6 forwarding on today, so the passing branch alone would be all anyone ever saw).
• The public-IP ratchet was watched go RED naming check-tailscale.sh with a planted realistic IP, then green again byte-identical. Both content scans confirmed to enumerate the new files. Fixtures use CGNAT/RFC1918/.example.test only.
• shellcheck 0.11.0 clean. 290 tests green across the four content ratchets and the adjacent structural gates.
• Validated against the real tailscale 1.102.3 binary (the exact version the closure installs): tailscale debug prefs exists ("Print prefs") and tailscale status --json exists, and every JSON field the parser reads — AdvertiseRoutes, RouteAll, WantRunning, BackendState, TailscaleIPs, PrimaryRoutes, KeyExpiry, HostName — is present in the binary's string table. That probe was itself controlled: the first attempt reported 0 for everything because it was pointed at a 1378-byte bash wrapper, and the positive control (tailscaled, expected many, got 1) is what caught it; re-pointed at the 44 MB Go binary the control reads 106 and a fabricated field name still reads 0.
• The verifier run end-to-end against that real binary with no daemon running: rc 2 CANNOT DETERMINE, surfacing tailscaled's own "it doesn't appear to be running" message — not a false FAIL and not a false PASS.
• The --include nixos-config= plumbing validated for real: a faithful copy of the live config produced the identical system derivation hash, and the patched one a different hash that evaluates cleanly — confirming the generated block causes no attribute conflict with the module's own mkOverride 97 forwarding sysctls.

🔴 NOT verified

Nothing has been applied to a live host. sudo here is password-gated, so nixos-rebuild build, nixos-rebuild switch, the post-switch verify path, and and the response of a live, authenticated tailscaled are all unexercised. The CLI surface and field names are confirmed against the real 1.102.3 binary (above), but the exact JSON nesting is not — the fixtures are built from the documented ipn.Prefs / ipnstate.Status shapes, not captured from a running daemon. Treat the first real run as the verification.

Prerequisites and manual steps

An operator needs a Tailscale account/tailnet first. Then, after the switch, three steps that cannot be scripted — the script prints all three and the verifier keeps reporting rc 3 until they are done:

1. sudo tailscale up --advertise-routes=192.168.50.0/24 --accept-dns=false (workbench) / sudo tailscale up --accept-routes (laptop) — needs a browser.
2. Approve the subnet route in the admin console. Until then it carries nothing.
3. Disable node key expiry. The 180-day default crosses the trip.

Related: #1272 (where this defect was recorded), #1287 (multi-path host reachability).

---

Audit round 2 — 11 findings fixed (65c391f6)

Two were blockers that made the script unusable as written. Every fix below was
driven old-vs-new against the real scripts, with nixos-rebuild/tailscale/ip/
systemctl shimmed, so each control shows the OLD behaviour reproducing and the NEW one
correcting it.

🔴 F1 — the first apply run was GUARANTEED to roll back

This script deliberately does not run tailscale up (it needs a browser), so straight
after its switch the node has never authenticated: NeedsLogin, Online false, no
TailscaleIPs, nothing advertised. The verifier turned that into four entries in
fails[] and exited 1 — not 3 — while the post-switch verify accepted only rc 0/2/3
and died on anything else, so the EXIT trap restored the backup. The comment claiming
"rc 3 is the EXPECTED state straight after a switch" was false, and the section above
that says 3 is one of four exit codes was describing a state the script could not
reach.

	OLD (fec530c1)	NEW (65c391f6)
same fixture, successful switch	ABORT … ROLLED BACK, grep -c services.tailscale = 0	=== SWITCHED ===, = 1
verifier rc on that state	1 (node-side FAILURE)	4 (INCOMPLETE, no defect)

Not a wider rc allow-list. "Never authenticated" is now a first-class state in the
verifier (_authenticated, two signals that must agree — an address in 100.64.0.0/10
and a BackendState that is not NeedsLogin/NoState) with its own exit code 4 =
INCOMPLETE, and the apply script's case enumerates all five codes with what each
means. rc 1 still rolls back — driven with forwarding off, and pinned by a test,
because a guard that can no longer go red was removed rather than fixed.

🔴 F8 — a partial cp left /etc/nixos/configuration.nix truncated, silently

PATCHED=1 was set after cp -p "$TMP" "$CFG". cp truncates then writes, so a
failure part way through (ENOSPC — / on this host is at 77%) aborted under set -e
with PATCHED=0, the trap's rollback branch was skipped, and the config was left
half-written with a good backup unused beside it and nothing printed. Measured with a
cp that writes 40 bytes and exits 1:

• OLD → configuration.nix is a 40-byte fragment ending mid-word, no rollback, no message.
• NEW → rollback fires, file restored, and the message correctly says the switch was never started.

The auditor judged the original sequencing correct; it is not. Moving the flag first is
safe in the other direction — a cp that fails before writing restores a byte-identical
file.

The rest

#	Was	Now
F7	absent Self.KeyExpiry → PASS keyexpiry : disabled, printed on a node that had never logged in	absence is ambiguous; "disabled" is claimed only when the node IS authenticated, otherwise UNKNOWN
F3	rollback asserted "The system was never switched" from a flag that only records whether the switch returned zero	three states; only the establishable one is asserted, with the commands to check the other
F4	"Largest pending items", alphabetical, head -15 — on the real 38-drv queue every steam-* was hidden and wine-wow-11.0 sorted last	all of them, labelled a name list and not a size ranking; a cap says how many it omitted
F5	fetch counts and MiB were printed and read by no gate — 0 to build / 4 GiB to fetch passed untouched (measured: OLD proceeded to switch)	download volume gated in its own right (_over_mib, floating point); header's "three ways" is now four, each with what it CANNOT see; the release gate's major.minor blind spot is stated where it reports
F6	grep -q 'services\.tailscale' — a config containing only # TODO: consider services.tailscale one day exited 0 "Nothing to do"	comments (# and /* */) stripped first, an actual setting required; a mention-only file is reported and then patched
F9	--accept-dns=false on the server only, so MagicDNS took the resolver of the one machine that leaves the LAN	both roles, cost stated out loud; a test pins the two scripts' up strings equal
F10	--role last → exit 1 printing nothing (both scripts) · --help a hardcoded range that truncated one file's SAFETY paragraph and ran the other into set -euo pipefail · no-trailing-newline aborted naming the wrong problem · "Only -3 of the 37 are tailscale's" · "$CHECK" invoked directly — a copy without its exec bit exits 126 and gets the config rolled back	each fixed and pinned

F2 — measured correction, reported honestly

The reported mechanism (a SIGINT-killed foreground child does not trip errexit) did
not reproduce on bash 5.3.15 here. With the signal confirmed to have landed — the
control being the killed child's wall time, since kill against a process with SIGINT
ignored returns 0 and looks identical to "the shell continued" — bash propagates the
child's SIGINT death and terminates the script, both child-only and process-group. The
first "it continued" observation was that ignored-signal case. The explicit capture
(rc=0; nixos-rebuild switch || rc=$?) is kept anyway and is reachable on
SIGTERM/SIGHUP, both driven against the real script: it names the signal instead of
aborting silently.

🔴 F11 — the AirVPN killswitch inverts the redundancy story (documented, NOT changed)

scripts/airvpn-updown's degraded and fallback rulesets allow egress on a literal
interface list — lo, the airvpn tun, nebula.mesh, cni0, flannel.1, docker0 —
and then drop. tailscale0 is not on it, and unlike nebula (three carve-outs)
tailscale has no DERP/control-plane bypass. So if that killswitch ever arms
fail-closed, nebula survives and tailscale dies — the exact inverse of the
independence this PR is for.

AirVPN is default-OFF on these hosts, so this is a latent interaction and not a live
defect. airvpn-updown is deliberately not touched here: widening a killswitch's
allow-list is its own change with its own review, and doing it inside a PR about a
different subsystem is how a killswitch quietly stops being one. The interaction is
recorded in the generated Nix block's comment for both roles and in the apply
script's header.

What was tested this round

• Both --self-tests green (apply 42 controls, check 27) and now run by pytest, so they are part of the gate rather than something a human has to remember.
• New scripts/tests/test_tailscale_scripts.py (22 tests): pins the two scripts' tailscale up strings equal, pins the exit-code vocabularies against each other, drives the freshly-switched state against a fake tailscale, and checks --help matches each file's own comment header exactly.
• Mutation sweep: 17/17 mutants killed by their intended test, run under PYTHONDONTWRITEBYTECODE=1 in an isolated cp -a of the tree with its .git pointer removed. The first round had 2 not properly killed — the F6 fixtures could see neither comment-stripping nor the [.={] anchor — and the discriminating cases (a commented-out declaration; services.tailscale named in a warning string) were added rather than the result being reported as green.
• shellcheck -S warning clean on both files, with the scanner itself negative-controlled against a known-bad script first.

🔴 NOT verified this round

• Nothing was applied to a live host. sudo is password-gated. The apply path is exercised only against shimmed nixos-rebuild, tailscale, ip and systemctl; no real dry-build, build, switch or activation ran.
• The rc 0 and rc 3 verifier paths are driven from fixtures, not from a real authenticated daemon. JSON nesting is still from the documented shapes, unchanged from round 1.
• tailscale up, subnet-route approval and disabling key expiry remain untouched manual steps.
• The F11 interaction is reasoned from airvpn-updown's source, not observed — the killswitch was never armed to watch tailscale drop.