fix(os): guard the loop-partition race and the debug->release update trap

docs/dev/appliance-release.md

@@ -28,6 +28,34 @@ The image also carries no installer payload. `pithead-install` rebuilds the layo
 target and copies the running slot, so the artifact never contains a compressed copy of
 itself.
 
+## Build variants and the variant stamp
+
+Every build is one of two variants, decided by whether the rootfs bakes an SSH key
+(`PITHEAD_TEST_SSH_PUBKEY` at build time):
+
+- **release** — shell-less. No key, sshd disabled, the dashboard is the only management
+  surface. The only variant that ships.
+- **debug** — the bench build: a root SSH key baked, sshd enabled. Never publish one;
+  `verify-image` without `--test` refuses it.
+
+The variant is stamped into the artifacts so tooling can tell them apart after the build:
+
+| Where | What |
+|---|---|
+| `/etc/pithead-variant` | in the rootfs — the running system and every installed slot carry it |
+| `[meta.pithead] variant=` | in the update bundle's manifest — `rauc info` shows it before anything is installed |
+
+`verify-image` asserts the stamp matches the mode it runs in: `--test` expects `debug`,
+no flag expects `release`.
+
+The stamp exists because the debug→release transition is one-way and used to be silent: a
+debug box that installs a release bundle drops SSH by design — the channel that drove the
+install — and recovery needs a console. This happened live on the bench and ended in a
+stick reinstall. `pithead os-update BUNDLE` is the install path that reads both stamps and
+warns before making that transition, requiring a y/N confirmation (or `--yes`). A bare
+`rauc install` bypasses the guard; use it only when you have already decided the SSH loss
+is acceptable.
+
 ## Development loop
 
 Everything runs from the repo root on a Linux box with docker, KVM and libvirt. The
@@ -121,10 +149,12 @@ Confirm a pasted **subaddress** (`8…`) is rejected with an explanation before
 Expected: the stack provisions and the dashboard comes up.
 
 **M7 — real update.** Build a `v+1` bundle, copy it to the machine, and install it with
-`rauc install` (the test image carries SSH for exactly this). Expected: installs, reboots
-into the new version, and an uncommitted update reverts on the next reboot. The
-dashboard-driven OS update is tracked pre-GA work — until it exists, this is the honest
-mechanism, and it is the same one the KVM update phase exercises.
+`pithead os-update BUNDLE` (the test image carries SSH for exactly this; the command wraps
+`rauc install` and compares the variant stamps first — a debug box taking a release bundle
+must warn before removing its own SSH). Expected: installs, reboots into the new version,
+and an uncommitted update reverts on the next reboot. The dashboard-driven OS update is
+tracked pre-GA work — until it exists, this is the honest mechanism, and it is the same
+one the KVM update phase exercises.
 
 **M8 — pull the plug.** During the update's write phase, physically cut power. Repeat
 three times. Expected: the machine boots the old version every time. *A brick here blocks

os/rauc/mkbundle.sh

@@ -30,6 +30,10 @@ mkfs.ext4 -q -L system "$WORK/rootfs.ext4"
 mount -o loop "$WORK/rootfs.ext4" "$WORK/mnt"
 tar -xf "$TARBALL" -C "$WORK/mnt"
 populate_slot "$WORK/mnt"
+# The build variant, read from the rootfs the bundle actually ships so the stamp cannot drift
+# from the payload: debug (SSH baked) or release (shell-less). Carried as bundle metadata so
+# `pithead os-update` can warn BEFORE a debug box replaces the SSH channel driving the install.
+VARIANT=$(tr -d ' \t\r\n' 2>/dev/null <"$WORK/mnt/etc/pithead-variant" || echo release)
 umount "$WORK/mnt"
 mv "$WORK/rootfs.ext4" "$WORK/bundle/rootfs.ext4"
 
@@ -38,6 +42,9 @@ cat >"$WORK/bundle/manifest.raucm" <<EOF
 compatible=pithead-amd64
 version=$(tr -d ' \t\r\n' <VERSION)
 
+[meta.pithead]
+variant=$VARIANT
+
 [image.rootfs]
 filename=rootfs.ext4
 EOF

os/rauc/mkimage.sh

@@ -57,6 +57,21 @@ cleanup() {
 }
 trap cleanup EXIT
 
+# losetup -P returns before the kernel/udev publish the partition nodes, and the gap widens
+# under loop-device churn (a KVM battery running beside a release build). Without this wait the
+# first mkfs fails with "unable to open ${LOOP}p1: No such device or address" — and a plain
+# retry succeeds, which is exactly how it stays invisible until release day. One guarded wait:
+# settle udev, then poll briefly for both nodes.
+udevadm settle 2>/dev/null || true
+for _ in {1..25}; do
+    [ -b "${LOOP}p1" ] && [ -b "${LOOP}p2" ] && break
+    sleep 0.2
+done
+[ -b "${LOOP}p1" ] && [ -b "${LOOP}p2" ] || {
+    echo "partition nodes for $LOOP never appeared after losetup -P" >&2
+    exit 1
+}
+
 echo "==> filesystems"
 mkfs.vfat -n ESP "${LOOP}p1" >/dev/null
 mkfs.ext4 -q -L system-a "${LOOP}p2"

os/rootfs/Containerfile

@@ -192,8 +192,25 @@ RUN test -f /usr/lib/systemd/system/systemd-repart.service
 # TEST-ONLY hooks (tests/os/run.sh --phase update): both default EMPTY and change nothing in a
 # release build. A pubkey enables root-key SSH so the harness can drive rugix-ctrl in the guest;
 # the marker distinguishes v1/v2 slots across the A/B cycle.
+#
+# The SSH bake is also what DEFINES the build variant, so it is stamped here (#819): a baked key
+# means a debug image, no key means the shell-less release. /etc/pithead-variant travels with the
+# rootfs into the image, every installed slot and the update bundle's [meta.pithead] section —
+# `pithead os-update` compares the running stamp against the bundle's, because a debug box that
+# silently takes a release bundle loses the SSH channel performing the install.
 ARG PITHEAD_TEST_SSH_PUBKEY=""
 ARG PITHEAD_TEST_MARKER=""
-RUN if [ -n "$PITHEAD_TEST_SSH_PUBKEY" ]; then         mkdir -p /root/.ssh && chmod 700 /root/.ssh         && printf '%s\n' "$PITHEAD_TEST_SSH_PUBKEY" > /root/.ssh/authorized_keys         && chmod 600 /root/.ssh/authorized_keys         && systemctl enable ssh;     fi     && if [ -n "$PITHEAD_TEST_MARKER" ]; then         printf '%s\n' "$PITHEAD_TEST_MARKER" > /etc/pithead-test-marker;     fi
+RUN if [ -n "$PITHEAD_TEST_SSH_PUBKEY" ]; then \
+        mkdir -p /root/.ssh && chmod 700 /root/.ssh \
+        && printf '%s\n' "$PITHEAD_TEST_SSH_PUBKEY" > /root/.ssh/authorized_keys \
+        && chmod 600 /root/.ssh/authorized_keys \
+        && systemctl enable ssh \
+        && printf 'debug\n' > /etc/pithead-variant; \
+    else \
+        printf 'release\n' > /etc/pithead-variant; \
+    fi \
+    && if [ -n "$PITHEAD_TEST_MARKER" ]; then \
+        printf '%s\n' "$PITHEAD_TEST_MARKER" > /etc/pithead-test-marker; \
+    fi
 
 # Rugix Ctrl lands via the bakery layer's core recipes, not here — rootfs stays updater-agnostic.

pithead

@@ -1994,6 +1994,77 @@ provision_local_miner() {
     fi
 }
 
+# --- OS update (appliance A/B slots) ---
+# The build variant stamp: debug images bake an SSH key (often the box's only management
+# channel), release images are shell-less by design. The running system carries the stamp at
+# /etc/pithead-variant; an update bundle carries it in its manifest's [meta.pithead] section.
+# os-update compares the two, because the failure this guards is silent and one-way: a debug
+# box that installs a release bundle drops SSH — the very channel driving the install — and
+# recovery needs a console.
+
+os_running_variant() { # echoes debug|release|unknown
+    local v
+    v=$(tr -d ' \t\r\n' 2>/dev/null <"${PITHEAD_VARIANT_FILE:-/etc/pithead-variant}" || true)
+    case "$v" in debug | release) echo "$v" ;; *) echo unknown ;; esac
+}
+
+os_bundle_variant() { # $1: bundle path — echoes debug|release|unknown, never fails
+    # The stamp lives under [meta.pithead] in the bundle manifest; `rauc info` exposes manifest
+    # meta in its JSON output. Parsed defensively (any object carrying a `variant` key), so a
+    # RAUC that nests the manifest differently still yields the stamp rather than a false
+    # "unknown" — and anything unparseable degrades to unknown, which the gate treats as
+    # shell-less.
+    local v
+    v=$(rauc info --output-format=json "$1" 2>/dev/null |
+        jq -r '[.. | objects | .variant? // empty] | map(select(. == "debug" or . == "release")) | first // empty' 2>/dev/null)
+    case "$v" in debug | release) echo "$v" ;; *) echo unknown ;; esac
+}
+
+os_update_needs_confirmation() { # $1: running variant, $2: bundle variant — rc 0 = confirm first
+    # Only the debug->non-debug transition loses the management channel. An unstamped bundle is
+    # treated as release: assuming "debug" here is how a box gets stranded.
+    [ "$1" = "debug" ] && [ "$2" != "debug" ]
+}
+
+os_update() {
+    local bundle="" assume_yes=0 arg
+    for arg in "$@"; do
+        case "$arg" in
+        -y | --yes) assume_yes=1 ;;
+        -*) error "Unknown option for os-update: $arg. Run '$0 help'." ;;
+        *)
+            [ -z "$bundle" ] || error "os-update takes exactly one bundle path. Run '$0 help'."
+            bundle="$arg"
+            ;;
+        esac
+    done
+    [ -n "$bundle" ] || error "os-update needs the path to an update bundle (.raucb). Run '$0 help'."
+    command -v rauc >/dev/null 2>&1 ||
+        error "rauc is not available here — OS updates apply to the appliance image, not a checkout install."
+    [ -f "$bundle" ] || error "No such bundle: $bundle"
+
+    local running target
+    running=$(os_running_variant)
+    target=$(os_bundle_variant "$bundle")
+    if os_update_needs_confirmation "$running" "$target"; then
+        warn "This system is a debug build: SSH is baked in, and it is probably the channel driving this update."
+        if [ "$target" = "release" ]; then
+            warn "The bundle is a release build — shell-less by design. Installing it removes SSH; recovery then needs a console on the box."
+        else
+            warn "The bundle carries no variant stamp — treat it as a shell-less release build. Installing it can remove SSH; recovery then needs a console on the box."
+        fi
+        if [ "$assume_yes" -eq 0 ]; then
+            read -r -p "Install it anyway? (y/N): " CONFIRM || true
+            if [[ ! "$CONFIRM" =~ ^[Yy] ]]; then
+                log "OS update cancelled."
+                return 1
+            fi
+        fi
+    fi
+    log "Installing OS update bundle: $bundle (running: $running, bundle: $target)"
+    rauc install "$bundle"
+}
+
 reset_dashboard() {
     local assume_yes=0 arg
     for arg in "$@"; do
@@ -2453,6 +2524,13 @@ Maintenance:
                             stop the miner when it is off. The boot path runs this after the
                             stack is up; a no-op outside the appliance.
 
+  os-update BUNDLE [-y|--yes]
+                            Install an OS update bundle into the spare A/B slot (appliance
+                            only — runs 'rauc install'). When this system is a debug build
+                            (SSH baked in) and the bundle is not, it warns and asks first:
+                            that install removes the SSH channel driving it.
+                              -y, --yes        skip the confirmation prompt.
+
   onion-client-key          Print the Tor client-auth line for the dashboard onion —
                             the client PRIVATE key, kept out of 'status'. Add it to your Tor
                             client's ClientOnionAuthDir to reach a client-auth'd onion.
@@ -6885,7 +6963,7 @@ apply() {
 # Every dispatchable subcommand, in help order. main's dispatch, the chain validator, and the
 # tab-completion script (pithead-completion.bash) key off this one list; tests/stack/run.sh fails
 # if any of the three drift apart.
-readonly PITHEAD_COMMANDS="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard load-images local-miner control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help"
+readonly PITHEAD_COMMANDS="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard load-images local-miner os-update control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help"
 # The subset allowed in a chain: commands that take no positional argument and terminate on their
 # own. Excluded: setup (interactive first-run), logs (follows until Ctrl+C), restore (needs an
 # archive path), reset-dashboard (destructive — run it deliberately, alone), and the one-shot
@@ -8352,6 +8430,7 @@ main() {
         require_env
         provision_local_miner
         ;;
+    os-update) os_update "$@" ;;
     control-run-pending)
         _reject_options control-run-pending "$@"
         require_deployed

pithead-completion.bash

@@ -13,7 +13,7 @@
 
 # Keep in sync with PITHEAD_COMMANDS in the pithead script — tests/stack/run.sh fails if the
 # two lists drift.
-_pithead_commands="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard load-images local-miner control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help"
+_pithead_commands="setup apply render up down restart upgrade logs status doctor support-bundle reset-dashboard backup restore uninstall firstboot-wizard load-images local-miner os-update control-run-pending onion-client-key rotate-dashboard-onion rotate-secrets render-quadlet version help"
 
 # Service names = the top-level keys under `services:` in the docker-compose.yml next to the
 # pithead being completed. $1 = path to that compose file.

tests/os/verify-image.sh

@@ -47,6 +47,19 @@ cleanup() {
 }
 trap cleanup EXIT
 
+# Same race as mkimage.sh: losetup -P returns before the partition nodes exist. Wait once here
+# so the checks below read the image, not the timing of udev on a busy box.
+udevadm settle 2>/dev/null || true
+for _ in {1..25}; do
+    [ -b "${LOOP}p1" ] && [ -b "${LOOP}p2" ] && break
+    sleep 0.2
+done
+# Hard-fail like mkimage.sh does: a mount error two screens later names the wrong culprit.
+[ -b "${LOOP}p1" ] && [ -b "${LOOP}p2" ] || {
+    echo "partition nodes never appeared on $LOOP — udev timing or a broken image" >&2
+    exit 1
+}
+
 echo "==> partition table"
 # Ships ONLY the ESP and slot A — systemd-repart builds the rest on the target's real disk. A
 # third partition here means the build regressed to image-sized /data, the bug that was #784.
@@ -169,14 +182,18 @@ if [ -f ./pithead ] && [ -f build/dashboard/mining_dashboard/wizard.py ]; then
 fi
 
 echo "==> test material"
+# The variant stamp must MATCH the material, not just exist: a debug image stamped "release"
+# defeats the os-update guard that keeps a debug box from silently dropping its own SSH.
 if [ "$MODE" = "--test" ]; then
     chk "test SSH key present (harness build)" '[ -s "$ROOT/root/.ssh/authorized_keys" ]'
+    chk "variant stamp says debug" '[ "$(cat "$ROOT/etc/pithead-variant")" = "debug" ]'
 else
     # The reason this script exists in versioned form: a leaked test key on a release image is a
     # backdoor, and ad-hoc eyeballing is how one ships.
     chk "NO test marker" '[ ! -e "$ROOT/etc/pithead-test-marker" ]'
     chk "NO SSH authorized_keys" '[ ! -s "$ROOT/root/.ssh/authorized_keys" ]'
     chk "ssh service disabled" '! ls "$ROOT"/etc/systemd/system/multi-user.target.wants/ssh.service'
+    chk "variant stamp says release" '[ "$(cat "$ROOT/etc/pithead-variant")" = "release" ]'
 fi
 
 echo ""

tests/stack/run.sh

@@ -8292,6 +8292,91 @@ unset -f okrun
 rm -rf "$OKSB"
 unset OKSB
 
+echo "== unit: os-update variant gate — a debug box never silently loses its SSH =="
+# The trap this guards: a debug image's SSH key is often the only management channel, and a
+# release bundle removes it BY DESIGN. The gate must fire on debug->release, on debug->unstamped
+# (an old bundle without the stamp is shell-less too), and nowhere else.
+run_sourced "$SANDBOX" os_update_needs_confirmation debug release
+assert_rc "debug system + release bundle -> confirmation required" "$?" "0"
+run_sourced "$SANDBOX" os_update_needs_confirmation debug unknown
+assert_rc "debug system + unstamped bundle -> confirmation required" "$?" "0"
+run_sourced "$SANDBOX" os_update_needs_confirmation debug debug
+assert_rc "debug -> debug passes without ceremony" "$?" "1"
+run_sourced "$SANDBOX" os_update_needs_confirmation release release
+assert_rc "release -> release passes (the fleet's normal update)" "$?" "1"
+run_sourced "$SANDBOX" os_update_needs_confirmation unknown release
+assert_rc "unstamped running system passes — only a KNOWN debug box has a channel to lose" "$?" "1"
+
+OUSB=$(mktemp -d)
+mkdir -p "$OUSB/bin"
+# A fake rauc: logs every call, answers `info` with a canned JSON body.
+cat >"$OUSB/bin/rauc" <<'EOF'
+#!/usr/bin/env bash
+echo "[rauc] $*" >>"${RAUC_LOG:?}"
+case "$1" in
+info) [ -s "${RAUC_INFO_JSON:-}" ] && cat "$RAUC_INFO_JSON" ;;
+install) exit 0 ;;
+esac
+exit 0
+EOF
+chmod +x "$OUSB/bin/rauc"
+touch "$OUSB/bundle.raucb"
+export RAUC_LOG="$OUSB/calls"
+
+printf 'debug\n' >"$OUSB/variant-debug"
+printf 'release\n' >"$OUSB/variant-release"
+printf 'mystery\n' >"$OUSB/variant-garbage"
+assert_eq "running variant read from the stamp file" \
+    "$(PITHEAD_VARIANT_FILE=$OUSB/variant-debug run_sourced "$SANDBOX" os_running_variant)" "debug"
+assert_eq "a garbage stamp degrades to unknown, never to a variant" \
+    "$(PITHEAD_VARIANT_FILE=$OUSB/variant-garbage run_sourced "$SANDBOX" os_running_variant)" "unknown"
+assert_eq "a missing stamp file is unknown (pre-stamp images)" \
+    "$(PITHEAD_VARIANT_FILE=$OUSB/nope run_sourced "$SANDBOX" os_running_variant)" "unknown"
+
+ourun() { # <variant-file> <info-json-file or empty> [os-update args...] — stdin closed (no tty)
+    local vf="$1" ij="$2"
+    shift 2
+    (
+        cd "$OUSB" || exit
+        PATH="$OUSB/bin:$PATH"
+        # shellcheck disable=SC1090
+        source "$STACK"
+        set +e
+        RAUC_INFO_JSON="$ij" PITHEAD_VARIANT_FILE="$vf" os_update "$@" </dev/null
+    )
+}
+printf '{"manifest":{"meta":{"pithead":{"variant":"release"}}}}\n' >"$OUSB/info-release.json"
+printf '{"meta":{"pithead":{"variant":"debug"}}}\n' >"$OUSB/info-debug.json"
+assert_eq "bundle variant parsed from rauc info JSON (nested manifest)" \
+    "$(cd "$OUSB" && PATH="$OUSB/bin:$PATH" RAUC_INFO_JSON="$OUSB/info-release.json" run_sourced "$OUSB" os_bundle_variant bundle.raucb)" "release"
+assert_eq "bundle variant parse tolerates a different nesting" \
+    "$(cd "$OUSB" && PATH="$OUSB/bin:$PATH" RAUC_INFO_JSON="$OUSB/info-debug.json" run_sourced "$OUSB" os_bundle_variant bundle.raucb)" "debug"
+assert_eq "an unstamped bundle is unknown" \
+    "$(cd "$OUSB" && PATH="$OUSB/bin:$PATH" RAUC_INFO_JSON="" run_sourced "$OUSB" os_bundle_variant bundle.raucb)" "unknown"
+
+# The command end to end, with the daemon stubbed. Non-interactive stdin means the prompt reads
+# EOF -> cancelled: precisely the automation case where a silent install would strand the box.
+: >"$RAUC_LOG"
+out=$(ourun "$OUSB/variant-debug" "$OUSB/info-release.json" bundle.raucb 2>&1)
+rc=$?
+assert_rc "debug box + release bundle, no --yes -> refused" "$rc" "1"
+assert_contains "the refusal names the SSH loss" "$out" "removes SSH"
+assert_not_contains "rauc install was NOT reached" "$(cat "$RAUC_LOG")" "install"
+: >"$RAUC_LOG"
+ourun "$OUSB/variant-debug" "$OUSB/info-release.json" bundle.raucb --yes >/dev/null 2>&1
+assert_rc "--yes acknowledges the warning and proceeds" "$?" "0"
+assert_contains "rauc install ran with the bundle" "$(cat "$RAUC_LOG")" "install bundle.raucb"
+: >"$RAUC_LOG"
+ourun "$OUSB/variant-release" "$OUSB/info-release.json" bundle.raucb >/dev/null 2>&1
+assert_rc "release -> release installs with no prompt" "$?" "0"
+assert_contains "rauc install ran unprompted" "$(cat "$RAUC_LOG")" "install bundle.raucb"
+out=$(ourun "$OUSB/variant-debug" "$OUSB/info-release.json" 2>&1)
+assert_rc "a missing bundle path is an error, not an install" "$?" "1"
+unset RAUC_LOG
+unset -f ourun
+rm -rf "$OUSB"
+unset OUSB
+
 # ---------------------------------------------------------------------------
 echo ""
 printf 'pithead tests: \033[1;32m%d passed\033[0m, ' "$PASS"