diff --git a/docs/dev/appliance-release.md b/docs/dev/appliance-release.md index faba98a0..c777b74f 100644 --- a/docs/dev/appliance-release.md +++ b/docs/dev/appliance-release.md @@ -66,9 +66,9 @@ is the only thing standing between the hand-written boot path and a fleet. | Phase | Asserts | Count | |---|---|---| | `boot` | EFI boot to userspace; first-boot wizard announces itself with a console token; wizard serves the token gate on `:80` | 3 | -| `update` | `/data` grew to the disk and slots did not (#784); bundle installs into the spare; spare boots; **an uncommitted update reverts on reboot**; a committed update persists; an operator can roll back off a committed version | 11 | -| `provision` | a config submitted through the wizard's real HTTP flow (token read from the console, exactly as a human would) provisions the stack: validation, cosign-verified image pulls, containers running under podman, dashboard served through caddy. Then a **reboot with no hands on it** — the stack must return unaided through `pithead-boot` (render the derived layer, compose up, health-gated slot commit), the failure mode being a miner that sits dark after every power blip. The Caddyfile is corrupted before the reboot on purpose: derived files are regenerated every boot, and a stale one killed TLS on hardware. Catches an appliance whose engine cannot run the product — which happened, invisibly to every other phase | 9 | -| `install` | the image boots as **removable** media (usb bus — the gate keys on it); the inventory offers the internal disk and never the boot medium; the real installer runs; the machine then boots from the target alone with a **complete** copy (`/var/lib/dpkg` — the overlay made an incomplete copy easy and invisible), a fresh machine-id, `/data` sized to the target, and the wizard serving. Then the **reinstall leg**: a sentinel planted in `/data`, a second install over the same disk, and the sentinel required afterwards — the chain-preserving promise, tested | 17 | +| `update` | `/data` grew to the disk and slots did not (#784); bundle installs into the spare; spare boots; **an uncommitted update reverts on reboot**; a committed update persists; **after the commit, the page served comes from the NEW dashboard image** (marker baked into the image and read back over HTTP — the tag never changes, so "containers run" proves nothing about staleness, #798); an operator can roll back off a committed version | 13 | +| `provision` | a config submitted through the wizard's real HTTP flow (token read from the console, exactly as a human would) provisions the stack: validation, cosign-verified image pulls, containers running under podman, dashboard served through caddy. Then a **reboot with no hands on it** — the stack must return unaided through `pithead-boot` (load baked images, render the derived layer, compose up, health-gated slot commit), the failure mode being a miner that sits dark after every power blip. The Caddyfile is corrupted and the archive digest records dropped before the reboot on purpose: derived things are regenerated every boot, and a stale one killed TLS on hardware. Catches an appliance whose engine cannot run the product — which happened, invisibly to every other phase | 10 | +| `install` | the image boots as **removable** media (usb bus — the gate keys on it); the inventory offers the internal disk and never the boot medium; the real installer runs; the machine then boots from the target alone with a **complete** copy (`/var/lib/dpkg` — the overlay made an incomplete copy easy and invisible), a fresh machine-id, `/data` sized to the target, and the wizard serving. Then the **reinstall leg**: a sentinel planted in `/data`, a second install over the same disk, and the sentinel required afterwards — the chain-preserving promise, tested. The keep leg reinstalls from a **newer stick** over a `/data` that already holds the old dashboard image and its digest record: the image ID must change and the page served must come from the newer image (#798) | 21 | | `fault` | three power cuts mid-write; a deliberately corrupted bundle is refused without crashing and without bricking; a power cut inside the commit window; operator rollback after all of it; the box is still updatable afterwards | 11 | A **brick is disqualifying, not deducted** — any run that leaves a machine unable to boot diff --git a/docs/dev/appliance-wizard.md b/docs/dev/appliance-wizard.md index 10bdc792..d20becc7 100644 --- a/docs/dev/appliance-wizard.md +++ b/docs/dev/appliance-wizard.md @@ -110,28 +110,38 @@ certificate beats no dashboard. ## The boot contract (provisioned machines) -After provisioning, every boot runs one unit — `pithead-boot` — whose three steps each answer +After provisioning, every boot runs one unit — `pithead-boot` — whose four steps each answer a hardware-validated failure: -1. **`pithead render`** — regenerate every *derived* file (`.env`, Caddyfile, service configs, +1. **`pithead load-images`** — load the baked container-image archives when their content + changed. The archives ship in the read-only slot, the engine's storage lives on `/data`, + and every release tags its images identically — so without this step a keep-reinstall or + A/B update boots the new OS and keeps serving the old containers (a keep-reinstall did + exactly that on hardware: new slot, RC-old dashboard, every shipped fix absent). Keyed on + the archive's digest, recorded beside the store it describes; a normal boot pays one + sha256 per archive. The first-boot wizard runs the same loader before it serves, so both + boot owners converge the same way. +2. **`pithead render`** — regenerate every *derived* file (`.env`, Caddyfile, service configs, host units) from `config.json` plus the program that is actually running. Derived files are never inspected or repaired, only rebuilt: an A/B update swaps the whole program, and a bench machine once served a days-old Caddyfile whose site list predated the code around it. On the read-only root, host units render into `/run/systemd/system` (`--runtime` enablement) and are recreated here each boot. -2. **`pithead up`** — compose owns the containers' lifecycle. Its predecessor, - `podman-restart`, started the stack into its own oneshot cgroup, and systemd SIGKILLed the - containers it had just spawned. -3. **Health-gated slot commit** — `rauc status mark-good` only once the dashboard answers +3. **`pithead up`** — compose owns the containers' lifecycle, and recreates containers when + an image behind a constant tag changed identity. Its predecessor, `podman-restart`, + started the stack into its own oneshot cgroup, and systemd SIGKILLed the containers it + had just spawned. +4. **Health-gated slot commit** — `rauc status mark-good` only once the dashboard answers through caddy on a *listed* vhost (`localhost`; bare `127.0.0.1` hits Caddy's empty default site and proves nothing). A slot that boots but cannot serve stays uncommitted on purpose: that is the state A/B fallback exists for. Unprovisioned machines never commit — GRUB's clear-and-retry keeps them booting, and a bad update before provisioning reverts. **Rule for changes:** anything generated from `config.json` or the program is derived and must -be rebuilt by `render` — adding one anywhere else recreates the staleness bug. Genuine state -(`config.json`, wallets, chain data, Tor keys, generated secrets) is never regenerated; it -gets validation and a safe fallback instead. +be rebuilt by `render` — adding one anywhere else recreates the staleness bug. The container +images are derived in the same sense: functions of the running slot, converged every boot by +`load-images`. Genuine state (`config.json`, wallets, chain data, Tor keys, generated secrets) +is never regenerated; it gets validation and a safe fallback instead. The invariant, asserted by the provision phase: **corrupt any derived file, reboot, and the machine must serve again.** @@ -147,7 +157,7 @@ had a gap between it and the next one. | pure logic | `tests/frontend/configsync.test.mjs` | path access, typed coercion, address/pair guidance | | view rendering | `tests/frontend/wizard.test.mjs` (probes) | each view given its props | | **app orchestration** | `tests/frontend/wizard.test.mjs` (stubbed server) | **stage mapping, the handoff arriving through the poll, refresh-mid-provision, rejection round-trip, request bodies** | -| host logic | `tests/stack/run.sh` | cert minting + idempotence, remote-node preflight, pre-seed, install requests, reinstall pre-fill (secret strip + fail-open) | +| host logic | `tests/stack/run.sh` | cert minting + idempotence, remote-node preflight, pre-seed, install requests, the digest-keyed image loader, reinstall pre-fill (secret strip + fail-open) | | the real thing | `tests/os/run.sh --phase provision` | token from the console → submit → handoff → ack → running stack → reboot through a corrupted Caddyfile → no failed units → slot self-commit | The orchestration row is the one that was missing. pytest proved the endpoint published the diff --git a/os/build-image.sh b/os/build-image.sh index d9e557da..75fc544f 100755 --- a/os/build-image.sh +++ b/os/build-image.sh @@ -17,7 +17,6 @@ echo "==> staging wizard image $WIZARD_IMAGE" if [ "${PITHEAD_WIZARD_FROM_REGISTRY:-0}" = "1" ]; then # Release path: the published image carries the wizard module. docker pull -q "$WIZARD_IMAGE" >/dev/null - docker save "$WIZARD_IMAGE" | gzip -1 >os/rootfs/images/dashboard.tar.gz else # No mtime cache, EVER. The old test compared the archive against build/dashboard — the # DIRECTORY, whose mtime only moves when a direct child is added or removed. Editing a file @@ -28,8 +27,18 @@ else # only exists in this working tree, so a pulled image would start and immediately exit with # "No module named mining_dashboard.wizard". Build the image the appliance will actually run. docker build -q -t "$WIZARD_IMAGE" build/dashboard >/dev/null - docker save "$WIZARD_IMAGE" | gzip -1 >os/rootfs/images/dashboard.tar.gz fi +# Harness builds only: stamp the marker INTO the dashboard image, at a path the wizard serves +# (/static/os-test-marker.txt). The tag is identical across builds, so this is the only way a +# battery can tell which dashboard actually answers after an update or reinstall — the #798 +# regression was precisely "new OS, old dashboard, every check green". Release builds set no +# marker and get no extra layer. +if [ -n "${PITHEAD_TEST_MARKER:-}" ]; then + # USER root/pithead mirrors build/dashboard/Dockerfile: the runtime user cannot write /app. + printf 'FROM %s\nUSER root\nRUN printf %%s "%s" >/app/mining_dashboard/web/static/os-test-marker.txt\nUSER pithead\n' \ + "$WIZARD_IMAGE" "$PITHEAD_TEST_MARKER" | docker build -q -t "$WIZARD_IMAGE" - >/dev/null +fi +docker save "$WIZARD_IMAGE" | gzip -1 >os/rootfs/images/dashboard.tar.gz # Stamp the commit into the image. A release build once shipped a dashboard two commits stale # because it pulled from an intermediate clone, and nothing in the artifact could reveal it — diff --git a/os/overlay/pithead-boot b/os/overlay/pithead-boot index 95464050..5fb7ba66 100755 --- a/os/overlay/pithead-boot +++ b/os/overlay/pithead-boot @@ -18,16 +18,10 @@ cd /data/pithead || exit 1 # Containers are DERIVED too (#798): the baked archives ship in the slot, but podman's store # lives on /data and the image tag is constant across releases — a keep-reinstall or A/B -# update ran RC-old containers under an RC-new OS until this loaded them. Sha-keyed, so a -# normal boot pays one sha256 per archive; 'up' below recreates on the image-id change. -for f in /opt/pithead/images/*.tar.gz; do - [ -f "$f" ] || continue - sha=$(sha256sum "$f" | cut -d" " -f1) - rec="/data/pithead/data/.loaded-$(basename "$f").sha" - [ "$(cat "$rec" 2>/dev/null)" = "$sha" ] && continue - echo "pithead-boot: loading baked $(basename "$f") (new archive)" - podman load -qi "$f" >/dev/null && printf '%s' "$sha" >"$rec" -done +# update ran RC-old containers under an RC-new OS until this loaded them. ONE loader, shared +# with the first-boot wizard (load_baked_images): sha-keyed, so a normal boot pays one sha256 +# per archive, and 'up' below recreates on the image-id change. +./pithead load-images ./pithead render || exit 1 ./pithead up || exit 1 diff --git a/pithead b/pithead index e1d5a608..ba6ddfef 100755 --- a/pithead +++ b/pithead @@ -1476,6 +1476,50 @@ consume_install_request() { # return 1 } +# Baked container-image archives (/opt/pithead/images) are DERIVED, like every rendered file: +# they ship in the read-only slot, the engine's storage lives on /data — which survives +# reinstalls and A/B updates by design — and every release tags its images identically +# (pithead-dashboard:vX.Y.Z). Keying on "does the tag exist" therefore pins a machine to the +# first image it ever loaded: a keep-reinstall or an A/B update boots the new slot and keeps +# serving the old containers (#798, and a bench box served a weeks-old setup page for the same +# reason). The archive's digest is the honest key: load on change, record the digest beside +# the storage it describes, and let 'up' recreate containers on the image-id change. Both boot +# owners run this — pithead-boot on provisioned machines, the first-boot wizard before it +# serves — so every path converges on the shipped images, and a normal boot pays one sha256 +# per archive. +load_baked_images() { # $1 (optional): image ref whose absence forces a load despite a matching digest + local required="${1:-}" dir="${PITHEAD_IMAGES_DIR:-/opt/pithead/images}" + local engine archive sha rec recorded + engine=$(container_engine) + # The wizard-era record lived at a different path; retire it so an upgraded machine doesn't + # carry a stale file forever (its absence costs one reload on the first boot after upgrading). + rm -f "$PWD/.wizard-image-sha" + for archive in "$dir"/*.tar.gz "$dir"/*.tar; do + [ -f "$archive" ] || continue + sha=$(sha256sum "$archive" 2>/dev/null | cut -d' ' -f1) + rec="$PWD/data/.loaded-$(basename "$archive").sha" + # `|| true`: a missing record is the normal first-boot case, not an error to trap. + recorded=$(cat "$rec" 2>/dev/null || true) + if [ -n "$sha" ] && [ "$sha" = "$recorded" ]; then + # A digest record can outlive the storage it describes: when the caller names the + # image it needs, the record only counts if that image actually exists. + if [ -z "$required" ] || "$engine" image exists "$required" 2>/dev/null; then + continue + fi + fi + # The single slowest step of a first or post-update boot, and the one worth narrating. + _console "Loading this build's container images ($(basename "$archive")) — the slow part..." + mkdir -p "$PWD/data" 2>/dev/null || true + if "$engine" load -i "$archive" >/dev/null 2>&1; then + printf '%s' "$sha" >"$rec" 2>/dev/null || true # unrecorded -> reloads next boot + else + # No record on failure, on purpose: the next boot must retry, not skip. + warn "Could not load the baked image archive $(basename "$archive")." + fi + done + return 0 +} + firstboot_wizard() { local arg for arg in "$@"; do @@ -1559,25 +1603,9 @@ firstboot_wizard() { fi # Offline first boot: the appliance carries the wizard image as an archive (os/rootfs/images), # because at this point the operator may have no working network yet — that is what they are - # here to configure. Load it once if the engine does not already have it. - # Load the baked image when the ARCHIVE changes — never merely when the tag is missing. - # Every build tags the wizard image identically (pithead-dashboard:vX.Y.Z), and podman's - # storage lives on /data, which survives reinstalls and updates by design. Keying on - # "does the tag exist" therefore pins a machine to the first wizard it ever loaded: a - # bench box reinstalled with a newer image kept serving the old setup page, missing - # questions that had shipped weeks earlier. The archive's digest is the honest key. - local archive stamp="$PWD/.wizard-image-sha" baked="" - for archive in /opt/pithead/images/*.tar.gz /opt/pithead/images/*.tar; do - [ -f "$archive" ] || continue - baked=$(sha256sum "$archive" 2>/dev/null | cut -d' ' -f1) - if [ "$baked" != "$(cat "$stamp" 2>/dev/null)" ] || ! "$engine" image exists "$image" 2>/dev/null; then - # The single slowest step before the page exists, and the one worth narrating. - _console "Loading the setup page (this is the slow part)..." - if "$engine" load -i "$archive" >/dev/null 2>&1; then - printf '%s' "$baked" >"$stamp" - fi - fi - done + # here to configure. The loader is the boot path's own (see load_baked_images): naming the + # image forces a load when the tag is missing entirely, whatever the digest record says. + load_baked_images "$image" # Double quotes on purpose: $engine expands NOW, at trap definition. It is a local, and the # trap also fires after this function's locals are gone — where set -u would turn the trap # itself into the crash that masks whatever actually failed. @@ -2311,6 +2339,11 @@ Maintenance: wizard instead. The one-time token prints here; five wrong tries mint a fresh one. See docs/dev/dual-distribution-plan.md § phase 3. + load-images Load the baked container-image archives (/opt/pithead/images) + into the engine when their content changed since the last load. + The boot path runs this on every boot — a reinstall or update + that ships new images converges without any wizard involvement. + onion-client-key Print the Tor client-auth line for the dashboard onion — the client PRIVATE key, kept out of 'status'. Add it to your Tor client's ClientOnionAuthDir to reach a client-auth'd onion. @@ -6714,7 +6747,7 @@ apply() { # Every dispatchable subcommand, in help order. main's dispatch, the chain validator, and the # tab-completion script (pithead-completion.bash) key off this one list; tests/stack/run.sh fails # if any of the three drift apart. -readonly PITHEAD_COMMANDS="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help" +readonly PITHEAD_COMMANDS="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard load-images control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help" # The subset allowed in a chain: commands that take no positional argument and terminate on their # own. Excluded: setup (interactive first-run), logs (follows until Ctrl+C), restore (needs an # archive path), reset-dashboard (destructive — run it deliberately, alone), and the one-shot @@ -8172,6 +8205,10 @@ main() { restore) stack_restore "$@" ;; uninstall) stack_uninstall "$@" ;; firstboot-wizard) firstboot_wizard "$@" ;; + load-images) + _reject_options load-images "$@" + load_baked_images + ;; control-run-pending) _reject_options control-run-pending "$@" require_deployed diff --git a/pithead-completion.bash b/pithead-completion.bash index 90a99330..b2f1a71e 100644 --- a/pithead-completion.bash +++ b/pithead-completion.bash @@ -13,7 +13,7 @@ # Keep in sync with PITHEAD_COMMANDS in the pithead script — tests/stack/run.sh fails if the # two lists drift. -_pithead_commands="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help" +_pithead_commands="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard load-images control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help" # Service names = the top-level keys under `services:` in the docker-compose.yml next to the # pithead being completed. $1 = path to that compose file. diff --git a/tests/os/run.sh b/tests/os/run.sh index 7ee260de..9d77da0b 100755 --- a/tests/os/run.sh +++ b/tests/os/run.sh @@ -93,6 +93,22 @@ _wait_ssh() { # $1 seconds — the definition of "not bricked" } _marker() { _ssh cat /etc/pithead-test-marker 2>/dev/null | tr -d "\r\n"; } +# The marker baked INTO the dashboard image and served by whatever container actually answers +# (/static/os-test-marker.txt, stamped by os/build-image.sh on harness builds). Distinct from +# /etc/pithead-test-marker, which only names the OS slot: the image tag is identical across +# builds, so this is the one signal that separates "new OS, new containers" from the #798 +# failure — new OS, stale containers, every other check green. $1 expected, $2 seconds. +_dash_marker_served() { + local want="$1" deadline=$(($(date +%s) + ${2:-300})) got="" + while [ "$(date +%s)" -lt "$deadline" ]; do + got=$(curl -fsSk -m 5 "https://$ip/static/os-test-marker.txt" 2>/dev/null) + [ "$got" = "$want" ] && return 0 + sleep 5 + done + printf '%s' "${got:-nothing}" + return 1 +} + # Build a bootable image carrying $1 as its slot marker, for the selected updater. _build_image() { [ -f "$KEY" ] || ssh-keygen -t ed25519 -N "" -f "$KEY" -q @@ -341,6 +357,15 @@ phase_update() { bad "marker v1 missing on the initial slot" return } + # Baseline for the stale-container check below: the v1 image must serve its own marker + # BEFORE any update, or a later "v2 never served" says nothing about staleness. + local dm + if dm=$(_dash_marker_served v1 300); then + ok "the served page comes from the v1 dashboard image" + else + bad "the v1 dashboard image never served its marker (got: $dm)" + return + fi # #784: /data must fit the MACHINE, not the image. The image ships ~9 GiB with no data # partition at all; systemd-repart creates it on the target disk at first boot. The harness @@ -442,6 +467,15 @@ phase_update() { marker=$(_ssh cat /etc/pithead-test-marker) [ "$marker" = "v2" ] && ok "COMMIT: a committed update persists across reboot" || bad "expected v2 after commit, got '$marker'" + # THE stale-container assertion: the OS slot saying v2 is not enough — an A/B update that + # ships a new dashboard must end with the NEW image answering, without any wizard + # involvement. The tag never changes and podman's store survives on /data, so only the + # boot-path loader can make this true. + if dm=$(_dash_marker_served v2 360); then + ok "UPDATE REFRESHED THE CONTAINERS: the served page comes from the v2 dashboard image" + else + bad "the OS updated to v2 but the served page still comes from the old dashboard image (got: $dm)" + fi info "leg 3 — operator-initiated rollback off a committed update" _ssh "$(_rollback_cmd)" || true @@ -807,6 +841,61 @@ phase_install() { return } _ssh "umount -A /dev/vda4 2>/dev/null || true" + # A keep-reinstall must refresh the CONTAINERS too (#798). Model the machine that hit this + # live: /data already carries a dashboard image under the release tag, with its digest + # recorded beside the store — then reinstall from a NEWER stick. Only the digest-keyed + # boot loader makes the image change; a tag-exists check keeps the old containers forever. + info "keep leg prep — plant this build's dashboard image + digest record (a machine that ran it)" + local old_dash_id="" + old_dash_id=$(_ssh "T=\$(mktemp -d) && mount /dev/vda4 \"\$T\" && + mkdir -p \"\$T/pithead/data\" \"\$T/containers/storage\" && + podman --root \"\$T/containers/storage\" load -qi /opt/pithead/images/dashboard.tar.gz >/dev/null && + sha256sum /opt/pithead/images/dashboard.tar.gz | cut -d' ' -f1 | tr -d '\n' >\"\$T/pithead/data/.loaded-dashboard.tar.gz.sha\" && + podman --root \"\$T/containers/storage\" images --format '{{.Repository}} {{.ID}}' | awk '/pithead-dashboard/{print \$2; exit}' && + umount \"\$T\"") + [ -n "$old_dash_id" ] || { + bad "could not plant the old dashboard image for the keep leg" + return + } + ok "planted the old dashboard image ($old_dash_id) and its digest record on the target's /data" + _ssh "systemctl poweroff" 2>/dev/null || true + sleep 8 + vm_destroy + info "building the NEWER stick (marker v2 — its dashboard archive differs)" + img=$(_build_image v2) || { + bad "v2 stick build failed (/tmp/os-fault-build.log)" + return + } + cp "$img" "$DISK" + qemu-img resize "$DISK" 16G >/dev/null 2>&1 || true + : >"$SERIAL" + virt-install --name "$VM" --memory 16384 --vcpus 4 --cpu host-passthrough \ + --osinfo debian12 \ + --boot uefi,firmware.feature0.name=secure-boot,firmware.feature0.enabled=no \ + --import \ + --disk "path=$DISK,format=raw,bus=usb,removable=on,boot.order=1" \ + --disk "path=$target_disk,format=raw,bus=virtio,boot.order=2" \ + --network network=default,model=virtio --graphics none \ + --serial "file,path=$SERIAL" --noautoconsole >/dev/null 2>&1 || { + bad "virt-install failed for the newer-stick keep-reinstall boot" + return + } + ip="" + tries=0 + while [ -z "$ip" ] && [ "$tries" -lt 40 ]; do + ip=$(virsh domifaddr "$VM" 2>/dev/null | awk '/ipv4/{print $4}' | cut -d/ -f1 | head -1) + [ -n "$ip" ] || sleep 3 + tries=$((tries + 1)) + done + _wait_ssh 240 || { + bad "newer stick never answered SSH for the keep leg" + return + } + ok "newer stick boots as removable media" + _ssh "for i in \$(seq 36); do [ -s /data/pithead/data/firstboot/disks.tsv ] && exit 0; sleep 5; done; exit 1" || { + bad "the newer stick never published a disk inventory" + return + } # The keep path goes through the PAGE, exactly as an operator would: a bare submit with the # disk and wipe=keep — no config, because the survivor config wins. No credentials card may # appear (the machine keeps its old login; a regenerated one here was a real bench bug). @@ -821,6 +910,16 @@ phase_install() { bad "no token for the keep-reinstall leg" return } + # Fresh boot: the token prints before the wizard container finishes coming up. + tries2=0 + while ! curl -fsSk -m 5 "https://$ip/" 2>/dev/null | grep -qi "Pithead setup"; do + sleep 5 + tries2=$((tries2 + 1)) + [ "$tries2" -lt 36 ] || { + bad "the newer stick's wizard never served its gate page" + return + } + done jar=$(mktemp) curl -fsSk -c "$jar" -d "token=$token" "https://$ip/auth" -o /dev/null 2>/dev/null && grep -q "wizard_session" "$jar" || { @@ -892,6 +991,22 @@ phase_install() { else bad "no wizard on :80 after the reinstall" fi + # The keep-leg staleness assertions (#798): the dashboard image ID must have CHANGED — the + # boot-path loader keyed on the newer stick's archive digest, over a /data that already + # held the old image under the same tag — and the page actually served must come from the + # new image, not merely "some wizard answers". + local new_dash_id dm + new_dash_id=$(_ssh "podman images --format '{{.Repository}} {{.ID}}'" | awk '/pithead-dashboard/{print $2; exit}') + if [ -n "$new_dash_id" ] && [ "$new_dash_id" != "$old_dash_id" ]; then + ok "KEEP-REINSTALL REFRESHED THE DASHBOARD IMAGE ($old_dash_id -> $new_dash_id)" + else + bad "keep-reinstall left the old dashboard image in place (id: ${new_dash_id:-none}, was $old_dash_id)" + fi + if dm=$(_dash_marker_served v2 300); then + ok "the served page comes from the NEWER stick's dashboard image" + else + bad "the reinstalled machine still serves the old dashboard image (got: $dm)" + fi rm -f "$target_disk" } @@ -1051,6 +1166,11 @@ phase_provision() { info "reboot leg — the stack must come back on its own (pithead-boot)" _ssh "echo '# corrupted by the harness — a regenerated boot must not serve this' > /data/pithead/Caddyfile" 2>/dev/null || bad "could not corrupt the Caddyfile before the reboot" + # And drop the baked-archive digest records: the wizard wrote them at first boot, so their + # mere presence afterwards proves nothing. Gone, they must come back — that is + # pithead-boot's own loader running on a provisioned machine (#798). + _ssh "rm -f /data/pithead/data/.loaded-*.sha" 2>/dev/null || + bad "could not drop the digest records before the reboot" _ssh reboot 2>/dev/null || true sleep 10 _wait_ssh 300 || { @@ -1104,6 +1224,14 @@ phase_provision() { else bad "failed units after the reboot: $failed_units" fi + # The records dropped before the reboot must be BACK: on a provisioned machine only + # pithead-boot can have rewritten them, so this is the boot path running the baked-image + # loader — the mechanism a keep-reinstall or A/B update depends on (#798). + if _ssh "test -s /data/pithead/data/.loaded-dashboard.tar.gz.sha"; then + ok "pithead-boot ran the baked-image loader (digest record rewritten)" + else + bad "the digest record never came back — pithead-boot did not run the loader" + fi # The booted slot must commit ITSELF once healthy (#793) — no harness mark-good here. On a # real appliance nothing ever ran mark-good, so RAUC called both slots bad and every boot # took GRUB's degraded fallback path. A_OK=1 + A_TRY=0 is the committed state. diff --git a/tests/stack/run.sh b/tests/stack/run.sh index c34a5d84..4d1db346 100755 --- a/tests/stack/run.sh +++ b/tests/stack/run.sh @@ -7740,24 +7740,62 @@ assert_eq "other tor keys survive" "$(jq -r '.tor.data_dir' "$ADSB/config.json") rm -rf "$ADSB" unset ADSB -echo "== unit: the baked wizard image reloads when the ARCHIVE changes ==" -# Every build tags the wizard image identically and podman's storage lives on /data, which -# survives reinstalls — so "does the tag exist" pins a machine to the first wizard it ever -# loaded. A bench box served a weeks-old setup page for exactly this reason. +echo "== unit: load_baked_images — the archive digest, not the tag, decides a load (#798) ==" +# Every build tags its images identically and the engine's storage lives on /data, which +# survives reinstalls and A/B updates — so "does the tag exist" pins a machine to the first +# image it ever loaded. Both boot owners (pithead-boot and the first-boot wizard) run this ONE +# loader; the digest record beside the store is what makes a keep-reinstall or A/B update +# converge on the shipped containers. WSB=$(mktemp -d) -mkdir -p "$WSB/images" +mkdir -p "$WSB/images" "$WSB/bin" printf 'v1-archive' >"$WSB/images/dashboard.tar.gz" +cat >"$WSB/bin/podman" <<'EOF' +#!/usr/bin/env bash +echo "[podman] $*" >>"${PODMAN_LOG:-/dev/null}" +case "$1" in + image) [ -e "${PODMAN_IMAGE_PRESENT:-/nonexistent}" ] ;; # `image exists ` + load) exit "${PODMAN_LOAD_RC:-0}" ;; +esac +EOF +chmod +x "$WSB/bin/podman" +export PODMAN_LOG="$WSB/podman.log" PITHEAD_IMAGES_DIR="$WSB/images" +lbl() { PITHEAD_ENGINE=podman PATH="$WSB/bin:$PATH" run_sourced "$WSB" load_baked_images "$@"; } +WREC="$WSB/data/.loaded-dashboard.tar.gz.sha" sha_of() { sha256sum "$1" | cut -d' ' -f1; } -assert_eq "digest of an unchanged archive is stable" \ - "$(sha_of "$WSB/images/dashboard.tar.gz")" "$(sha_of "$WSB/images/dashboard.tar.gz")" -printf '%s' "$(sha_of "$WSB/images/dashboard.tar.gz")" >"$WSB/.wizard-image-sha" -[ "$(sha_of "$WSB/images/dashboard.tar.gz")" = "$(cat "$WSB/.wizard-image-sha")" ] && - ok "recorded digest matches -> no reload" || bad "recorded digest matches" "mismatch" + +lbl >/dev/null 2>&1 +grep -q "load -i" "$PODMAN_LOG" && ok "first boot loads the archive" || + bad "first boot loads the archive" "no load call" +assert_eq "the digest is recorded beside the store" \ + "$(cat "$WREC" 2>/dev/null)" "$(sha_of "$WSB/images/dashboard.tar.gz")" +: >"$PODMAN_LOG" +lbl >/dev/null 2>&1 +grep -q "load -i" "$PODMAN_LOG" && bad "an unchanged archive is not reloaded" "loaded again" || + ok "an unchanged archive is not reloaded" printf 'v2-archive-different' >"$WSB/images/dashboard.tar.gz" -[ "$(sha_of "$WSB/images/dashboard.tar.gz")" != "$(cat "$WSB/.wizard-image-sha")" ] && - ok "a rebuilt archive differs -> reload" || bad "a rebuilt archive differs" "same digest" +: >"$PODMAN_LOG" +lbl >/dev/null 2>&1 +grep -q "load -i" "$PODMAN_LOG" && + ok "a changed archive reloads — the keep-reinstall and A/B update path" || + bad "a changed archive reloads" "no load call" +# The wizard names the image it needs: a matching record must not count when the image is gone +# (the record can outlive the storage it describes). +: >"$PODMAN_LOG" +lbl ghcr.io/x/pithead-dashboard:v0 >/dev/null 2>&1 +grep -q "load -i" "$PODMAN_LOG" && + ok "a missing required image forces a load despite a matching record" || + bad "a missing required image forces a load" "no load call" +# A failed load leaves the old record: the next boot must retry, not skip. +WV2SHA=$(cat "$WREC") +printf 'v3-archive' >"$WSB/images/dashboard.tar.gz" +export PODMAN_LOAD_RC=1 +lbl >/dev/null 2>&1 +unset PODMAN_LOAD_RC +assert_eq "a failed load records nothing — the next boot retries" "$(cat "$WREC")" "$WV2SHA" +unset PODMAN_LOG PITHEAD_IMAGES_DIR +unset -f lbl sha_of rm -rf "$WSB" -unset WSB +unset WSB WREC WV2SHA echo "== unit: pre-seeding from the installation medium ==" # The ESP is FAT and anyone can write it, so both readers treat its contents as input, not truth.