fix(os): one baked-image loader for both boot owners; the battery now proves containers refresh

docs/dev/appliance-release.md

@@ -66,9 +66,9 @@ is the only thing standing between the hand-written boot path and a fleet.
 | Phase | Asserts | Count |
 |---|---|---|
 | `boot` | EFI boot to userspace; first-boot wizard announces itself with a console token; wizard serves the token gate on `:80` | 3 |
-| `update` | `/data` grew to the disk and slots did not (#784); bundle installs into the spare; spare boots; **an uncommitted update reverts on reboot**; a committed update persists; an operator can roll back off a committed version | 11 |
-| `provision` | a config submitted through the wizard's real HTTP flow (token read from the console, exactly as a human would) provisions the stack: validation, cosign-verified image pulls, containers running under podman, dashboard served through caddy. Then a **reboot with no hands on it** — the stack must return unaided through `pithead-boot` (render the derived layer, compose up, health-gated slot commit), the failure mode being a miner that sits dark after every power blip. The Caddyfile is corrupted before the reboot on purpose: derived files are regenerated every boot, and a stale one killed TLS on hardware. Catches an appliance whose engine cannot run the product — which happened, invisibly to every other phase | 9 |
-| `install` | the image boots as **removable** media (usb bus — the gate keys on it); the inventory offers the internal disk and never the boot medium; the real installer runs; the machine then boots from the target alone with a **complete** copy (`/var/lib/dpkg` — the overlay made an incomplete copy easy and invisible), a fresh machine-id, `/data` sized to the target, and the wizard serving. Then the **reinstall leg**: a sentinel planted in `/data`, a second install over the same disk, and the sentinel required afterwards — the chain-preserving promise, tested | 17 |
+| `update` | `/data` grew to the disk and slots did not (#784); bundle installs into the spare; spare boots; **an uncommitted update reverts on reboot**; a committed update persists; **after the commit, the page served comes from the NEW dashboard image** (marker baked into the image and read back over HTTP — the tag never changes, so "containers run" proves nothing about staleness, #798); an operator can roll back off a committed version | 13 |
+| `provision` | a config submitted through the wizard's real HTTP flow (token read from the console, exactly as a human would) provisions the stack: validation, cosign-verified image pulls, containers running under podman, dashboard served through caddy. Then a **reboot with no hands on it** — the stack must return unaided through `pithead-boot` (load baked images, render the derived layer, compose up, health-gated slot commit), the failure mode being a miner that sits dark after every power blip. The Caddyfile is corrupted and the archive digest records dropped before the reboot on purpose: derived things are regenerated every boot, and a stale one killed TLS on hardware. Catches an appliance whose engine cannot run the product — which happened, invisibly to every other phase | 10 |
+| `install` | the image boots as **removable** media (usb bus — the gate keys on it); the inventory offers the internal disk and never the boot medium; the real installer runs; the machine then boots from the target alone with a **complete** copy (`/var/lib/dpkg` — the overlay made an incomplete copy easy and invisible), a fresh machine-id, `/data` sized to the target, and the wizard serving. Then the **reinstall leg**: a sentinel planted in `/data`, a second install over the same disk, and the sentinel required afterwards — the chain-preserving promise, tested. The keep leg reinstalls from a **newer stick** over a `/data` that already holds the old dashboard image and its digest record: the image ID must change and the page served must come from the newer image (#798) | 21 |
 | `fault` | three power cuts mid-write; a deliberately corrupted bundle is refused without crashing and without bricking; a power cut inside the commit window; operator rollback after all of it; the box is still updatable afterwards | 11 |
 
 A **brick is disqualifying, not deducted** — any run that leaves a machine unable to boot

docs/dev/appliance-wizard.md

@@ -110,28 +110,38 @@ certificate beats no dashboard.
 
 ## The boot contract (provisioned machines)
 
-After provisioning, every boot runs one unit — `pithead-boot` — whose three steps each answer
+After provisioning, every boot runs one unit — `pithead-boot` — whose four steps each answer
 a hardware-validated failure:
 
-1. **`pithead render`** — regenerate every *derived* file (`.env`, Caddyfile, service configs,
+1. **`pithead load-images`** — load the baked container-image archives when their content
+   changed. The archives ship in the read-only slot, the engine's storage lives on `/data`,
+   and every release tags its images identically — so without this step a keep-reinstall or
+   A/B update boots the new OS and keeps serving the old containers (a keep-reinstall did
+   exactly that on hardware: new slot, RC-old dashboard, every shipped fix absent). Keyed on
+   the archive's digest, recorded beside the store it describes; a normal boot pays one
+   sha256 per archive. The first-boot wizard runs the same loader before it serves, so both
+   boot owners converge the same way.
+2. **`pithead render`** — regenerate every *derived* file (`.env`, Caddyfile, service configs,
    host units) from `config.json` plus the program that is actually running. Derived files are
    never inspected or repaired, only rebuilt: an A/B update swaps the whole program, and a
    bench machine once served a days-old Caddyfile whose site list predated the code around it.
    On the read-only root, host units render into `/run/systemd/system` (`--runtime`
    enablement) and are recreated here each boot.
-2. **`pithead up`** — compose owns the containers' lifecycle. Its predecessor,
-   `podman-restart`, started the stack into its own oneshot cgroup, and systemd SIGKILLed the
-   containers it had just spawned.
-3. **Health-gated slot commit** — `rauc status mark-good` only once the dashboard answers
+3. **`pithead up`** — compose owns the containers' lifecycle, and recreates containers when
+   an image behind a constant tag changed identity. Its predecessor, `podman-restart`,
+   started the stack into its own oneshot cgroup, and systemd SIGKILLed the containers it
+   had just spawned.
+4. **Health-gated slot commit** — `rauc status mark-good` only once the dashboard answers
    through caddy on a *listed* vhost (`localhost`; bare `127.0.0.1` hits Caddy's empty default
    site and proves nothing). A slot that boots but cannot serve stays uncommitted on purpose:
    that is the state A/B fallback exists for. Unprovisioned machines never commit — GRUB's
    clear-and-retry keeps them booting, and a bad update before provisioning reverts.
 
 **Rule for changes:** anything generated from `config.json` or the program is derived and must
-be rebuilt by `render` — adding one anywhere else recreates the staleness bug. Genuine state
-(`config.json`, wallets, chain data, Tor keys, generated secrets) is never regenerated; it
-gets validation and a safe fallback instead.
+be rebuilt by `render` — adding one anywhere else recreates the staleness bug. The container
+images are derived in the same sense: functions of the running slot, converged every boot by
+`load-images`. Genuine state (`config.json`, wallets, chain data, Tor keys, generated secrets)
+is never regenerated; it gets validation and a safe fallback instead.
 
 The invariant, asserted by the provision phase: **corrupt any derived file, reboot, and the
 machine must serve again.**
@@ -147,7 +157,7 @@ had a gap between it and the next one.
 | pure logic | `tests/frontend/configsync.test.mjs` | path access, typed coercion, address/pair guidance |
 | view rendering | `tests/frontend/wizard.test.mjs` (probes) | each view given its props |
 | **app orchestration** | `tests/frontend/wizard.test.mjs` (stubbed server) | **stage mapping, the handoff arriving through the poll, refresh-mid-provision, rejection round-trip, request bodies** |
-| host logic | `tests/stack/run.sh` | cert minting + idempotence, remote-node preflight, pre-seed, install requests, reinstall pre-fill (secret strip + fail-open) |
+| host logic | `tests/stack/run.sh` | cert minting + idempotence, remote-node preflight, pre-seed, install requests, the digest-keyed image loader, reinstall pre-fill (secret strip + fail-open) |
 | the real thing | `tests/os/run.sh --phase provision` | token from the console → submit → handoff → ack → running stack → reboot through a corrupted Caddyfile → no failed units → slot self-commit |
 
 The orchestration row is the one that was missing. pytest proved the endpoint published the

os/build-image.sh

@@ -17,7 +17,6 @@ echo "==> staging wizard image $WIZARD_IMAGE"
 if [ "${PITHEAD_WIZARD_FROM_REGISTRY:-0}" = "1" ]; then
     # Release path: the published image carries the wizard module.
     docker pull -q "$WIZARD_IMAGE" >/dev/null
-    docker save "$WIZARD_IMAGE" | gzip -1 >os/rootfs/images/dashboard.tar.gz
 else
     # No mtime cache, EVER. The old test compared the archive against build/dashboard — the
     # DIRECTORY, whose mtime only moves when a direct child is added or removed. Editing a file
@@ -28,8 +27,18 @@ else
     # only exists in this working tree, so a pulled image would start and immediately exit with
     # "No module named mining_dashboard.wizard". Build the image the appliance will actually run.
     docker build -q -t "$WIZARD_IMAGE" build/dashboard >/dev/null
-    docker save "$WIZARD_IMAGE" | gzip -1 >os/rootfs/images/dashboard.tar.gz
 fi
+# Harness builds only: stamp the marker INTO the dashboard image, at a path the wizard serves
+# (/static/os-test-marker.txt). The tag is identical across builds, so this is the only way a
+# battery can tell which dashboard actually answers after an update or reinstall — the #798
+# regression was precisely "new OS, old dashboard, every check green". Release builds set no
+# marker and get no extra layer.
+if [ -n "${PITHEAD_TEST_MARKER:-}" ]; then
+    # USER root/pithead mirrors build/dashboard/Dockerfile: the runtime user cannot write /app.
+    printf 'FROM %s\nUSER root\nRUN printf %%s "%s" >/app/mining_dashboard/web/static/os-test-marker.txt\nUSER pithead\n' \
+        "$WIZARD_IMAGE" "$PITHEAD_TEST_MARKER" | docker build -q -t "$WIZARD_IMAGE" - >/dev/null
+fi
+docker save "$WIZARD_IMAGE" | gzip -1 >os/rootfs/images/dashboard.tar.gz
 
 # Stamp the commit into the image. A release build once shipped a dashboard two commits stale
 # because it pulled from an intermediate clone, and nothing in the artifact could reveal it —

os/overlay/pithead-boot

@@ -18,16 +18,10 @@ cd /data/pithead || exit 1
 
 # Containers are DERIVED too (#798): the baked archives ship in the slot, but podman's store
 # lives on /data and the image tag is constant across releases — a keep-reinstall or A/B
-# update ran RC-old containers under an RC-new OS until this loaded them. Sha-keyed, so a
-# normal boot pays one sha256 per archive; 'up' below recreates on the image-id change.
-for f in /opt/pithead/images/*.tar.gz; do
-    [ -f "$f" ] || continue
-    sha=$(sha256sum "$f" | cut -d" " -f1)
-    rec="/data/pithead/data/.loaded-$(basename "$f").sha"
-    [ "$(cat "$rec" 2>/dev/null)" = "$sha" ] && continue
-    echo "pithead-boot: loading baked $(basename "$f") (new archive)"
-    podman load -qi "$f" >/dev/null && printf '%s' "$sha" >"$rec"
-done
+# update ran RC-old containers under an RC-new OS until this loaded them. ONE loader, shared
+# with the first-boot wizard (load_baked_images): sha-keyed, so a normal boot pays one sha256
+# per archive, and 'up' below recreates on the image-id change.
+./pithead load-images
 
 ./pithead render || exit 1
 ./pithead up || exit 1

pithead

@@ -1476,6 +1476,50 @@ consume_install_request() { # <spool-dir>
     return 1
 }
 
+# Baked container-image archives (/opt/pithead/images) are DERIVED, like every rendered file:
+# they ship in the read-only slot, the engine's storage lives on /data — which survives
+# reinstalls and A/B updates by design — and every release tags its images identically
+# (pithead-dashboard:vX.Y.Z). Keying on "does the tag exist" therefore pins a machine to the
+# first image it ever loaded: a keep-reinstall or an A/B update boots the new slot and keeps
+# serving the old containers (#798, and a bench box served a weeks-old setup page for the same
+# reason). The archive's digest is the honest key: load on change, record the digest beside
+# the storage it describes, and let 'up' recreate containers on the image-id change. Both boot
+# owners run this — pithead-boot on provisioned machines, the first-boot wizard before it
+# serves — so every path converges on the shipped images, and a normal boot pays one sha256
+# per archive.
+load_baked_images() { # $1 (optional): image ref whose absence forces a load despite a matching digest
+    local required="${1:-}" dir="${PITHEAD_IMAGES_DIR:-/opt/pithead/images}"
+    local engine archive sha rec recorded
+    engine=$(container_engine)
+    # The wizard-era record lived at a different path; retire it so an upgraded machine doesn't
+    # carry a stale file forever (its absence costs one reload on the first boot after upgrading).
+    rm -f "$PWD/.wizard-image-sha"
+    for archive in "$dir"/*.tar.gz "$dir"/*.tar; do
+        [ -f "$archive" ] || continue
+        sha=$(sha256sum "$archive" 2>/dev/null | cut -d' ' -f1)
+        rec="$PWD/data/.loaded-$(basename "$archive").sha"
+        # `|| true`: a missing record is the normal first-boot case, not an error to trap.
+        recorded=$(cat "$rec" 2>/dev/null || true)
+        if [ -n "$sha" ] && [ "$sha" = "$recorded" ]; then
+            # A digest record can outlive the storage it describes: when the caller names the
+            # image it needs, the record only counts if that image actually exists.
+            if [ -z "$required" ] || "$engine" image exists "$required" 2>/dev/null; then
+                continue
+            fi
+        fi
+        # The single slowest step of a first or post-update boot, and the one worth narrating.
+        _console "Loading this build's container images ($(basename "$archive")) — the slow part..."
+        mkdir -p "$PWD/data" 2>/dev/null || true
+        if "$engine" load -i "$archive" >/dev/null 2>&1; then
+            printf '%s' "$sha" >"$rec" 2>/dev/null || true # unrecorded -> reloads next boot
+        else
+            # No record on failure, on purpose: the next boot must retry, not skip.
+            warn "Could not load the baked image archive $(basename "$archive")."
+        fi
+    done
+    return 0
+}
+
 firstboot_wizard() {
     local arg
     for arg in "$@"; do
@@ -1559,25 +1603,9 @@ firstboot_wizard() {
     fi
     # Offline first boot: the appliance carries the wizard image as an archive (os/rootfs/images),
     # because at this point the operator may have no working network yet — that is what they are
-    # here to configure. Load it once if the engine does not already have it.
-    # Load the baked image when the ARCHIVE changes — never merely when the tag is missing.
-    # Every build tags the wizard image identically (pithead-dashboard:vX.Y.Z), and podman's
-    # storage lives on /data, which survives reinstalls and updates by design. Keying on
-    # "does the tag exist" therefore pins a machine to the first wizard it ever loaded: a
-    # bench box reinstalled with a newer image kept serving the old setup page, missing
-    # questions that had shipped weeks earlier. The archive's digest is the honest key.
-    local archive stamp="$PWD/.wizard-image-sha" baked=""
-    for archive in /opt/pithead/images/*.tar.gz /opt/pithead/images/*.tar; do
-        [ -f "$archive" ] || continue
-        baked=$(sha256sum "$archive" 2>/dev/null | cut -d' ' -f1)
-        if [ "$baked" != "$(cat "$stamp" 2>/dev/null)" ] || ! "$engine" image exists "$image" 2>/dev/null; then
-            # The single slowest step before the page exists, and the one worth narrating.
-            _console "Loading the setup page (this is the slow part)..."
-            if "$engine" load -i "$archive" >/dev/null 2>&1; then
-                printf '%s' "$baked" >"$stamp"
-            fi
-        fi
-    done
+    # here to configure. The loader is the boot path's own (see load_baked_images): naming the
+    # image forces a load when the tag is missing entirely, whatever the digest record says.
+    load_baked_images "$image"
     # Double quotes on purpose: $engine expands NOW, at trap definition. It is a local, and the
     # trap also fires after this function's locals are gone — where set -u would turn the trap
     # itself into the crash that masks whatever actually failed.
@@ -2311,6 +2339,11 @@ Maintenance:
                             wizard instead. The one-time token prints here; five wrong tries
                             mint a fresh one. See docs/dev/dual-distribution-plan.md § phase 3.
 
+  load-images               Load the baked container-image archives (/opt/pithead/images)
+                            into the engine when their content changed since the last load.
+                            The boot path runs this on every boot — a reinstall or update
+                            that ships new images converges without any wizard involvement.
+
   onion-client-key          Print the Tor client-auth line for the dashboard onion —
                             the client PRIVATE key, kept out of 'status'. Add it to your Tor
                             client's ClientOnionAuthDir to reach a client-auth'd onion.
@@ -6714,7 +6747,7 @@ apply() {
 # Every dispatchable subcommand, in help order. main's dispatch, the chain validator, and the
 # tab-completion script (pithead-completion.bash) key off this one list; tests/stack/run.sh fails
 # if any of the three drift apart.
-readonly PITHEAD_COMMANDS="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help"
+readonly PITHEAD_COMMANDS="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard load-images control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help"
 # The subset allowed in a chain: commands that take no positional argument and terminate on their
 # own. Excluded: setup (interactive first-run), logs (follows until Ctrl+C), restore (needs an
 # archive path), reset-dashboard (destructive — run it deliberately, alone), and the one-shot
@@ -8172,6 +8205,10 @@ main() {
     restore) stack_restore "$@" ;;
     uninstall) stack_uninstall "$@" ;;
     firstboot-wizard) firstboot_wizard "$@" ;;
+    load-images)
+        _reject_options load-images "$@"
+        load_baked_images
+        ;;
     control-run-pending)
         _reject_options control-run-pending "$@"
         require_deployed

pithead-completion.bash

@@ -13,7 +13,7 @@
 
 # Keep in sync with PITHEAD_COMMANDS in the pithead script — tests/stack/run.sh fails if the
 # two lists drift.
-_pithead_commands="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help"
+_pithead_commands="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard load-images control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help"
 
 # Service names = the top-level keys under `services:` in the docker-compose.yml next to the
 # pithead being completed. $1 = path to that compose file.