Harden live updater E2E evidence finalization

[skyrocket2026]

Summary

This draft hardens the source and evidence contract for the two-host live updater E2E. It does not claim that a fresh live run has occurred.

• capture an exact 18-step canary/stable scenario, including same-boot timer reactivation, rescue preconditions, and terminal stopped-writer refusal
• integrate the shipped guided operator setup and status path into the signed evidence contract
• retain the exact bootstrap tarball and verify it through a bounded streaming parser instead of loading up to 1 GiB into memory
• stage bootstrap downloads in a root-created random directory, guard the cleanup target, eliminate optimization-sensitive Python assert checks, and preserve signature and signed-bundle validation before extraction
• bind every fixed installer, systemd, example, operator, sysusers, and runtime-key payload to reviewed checkout bytes, including the exact channel-URL and stable-floor transformations
• retain candidate-B INPUTS.json and bind the bundled requirements lock and complete wheelhouse to its declared digest map
• validate the observed immutable GitHub release and lightweight tag API records, including exact target revision and four asset names, sizes, SHA-256 digests, states, and URLs
• require GitHub attestation verification to be constrained by source repository, exact source digest, signer workflow, and GitHub-hosted runner; preflight rejects an older CLI before paid resources are created
• replace the operator-secret scan's count-only report with a domain-separated content root over the exact scanned evidence inventory; finalization recomputes it so post-scan path/content changes fail closed
• apply bootstrap artifact size caps before required-file and scenario processing, then hash scenario artifacts by streaming reads
• create and independently verify a canonical Ed25519-signed result after successful teardown, binding controller source, control, scenarios, final host/service/timer state, and every retained evidence file
• reject unsafe archive paths, type confusion, digest drift, source-asset substitution, publication detachment, incomplete cleanup, and inconsistent guided status
• require the two timer-reconfiguration observation pairs added by upstream PR fix(live): let the reactivation proof survive an unloaded timer #211 and reject missing, duplicated, or changed non-empty invocation/trigger values during independent result replay
• lint the standalone result verifier in the release-boundary CI gate

Merged current upstream main at c33aa7994ab33df47c4529b4cd2258378df26984, including PR #212's signed production release/bootstrap publication and PR #213's corrected public bootstrap wording. Current draft head: 913b159b5e1c78e8e26696c258a13c30f9d0620d.

The upstream production runtime and bootstrap releases improve public installability, but their runtime source predates this PR. They are not evidence that this PR's live-E2E contract has run successfully, nor proof of miner availability, scoring, a chain write, emissions, or launch readiness.

Safety and authority scope

• no subnet wallet, validator cycle, weight submission, chain write, cloud deployment, or production route is exercised by this PR
• runtime/bootstrap release keys, the dedicated evidence-signing key, and operator hotkey material remain distinct
• raw evidence publication and immutable anchoring remain separate owner-authorized actions
• signed evidence proves what the controller captured; it does not turn the controller into independent validator authority
• candidate dependency provenance is checked online by gh attestation verify; the result verifier does not retain and replay an offline Sigstore bundle, so external provenance review remains required
• this PR does not authorize M1 completion or mainnet launch; launch remains NO-GO pending a successful independently reviewed live run and the wider protocol gates

Validation

• 572 passed across the 11-file guided operator, bootstrap/release publisher, live controller/result, operator-docs, release finalizer, signed release builder, updater, updater-bundle, and start-gate recovery suite on the exact current head
• 212 passed in the focused live controller/result suite on the exact current head
• 36 passed in the live controller suite after random-staging hardening
• prior branch validation also retained 17 passed for relay release-install regressions
• dedicated mutation cases reject missing, extra, changed-invocation, and changed-trigger reactivation observations
• the bootstrap command regression requires root-created unpredictable staging, an exact cleanup guard, variable-derived artifact paths, verification-before-extraction ordering, and valid emitted remote Bash
• Ruff check and format check passed, including the standalone result verifier
• Python compilation passed
• Bash syntax validation passed
• Git diff/whitespace validation passed
• four independent read-only review passes found no remaining deterministic blocker in archive parsing, fixed-source binding, publication records, secret-scan roots, timer reactivation, bootstrap staging, or scenario closure

Ownership-sensitive tests were run from an exact tracked-byte temporary copy with group/other write bits removed and umask 077; the shared working checkout and /tmp ancestor are correctly refused by production owner-controlled-path checks. The owner-private run passed all 572 tests.

GitHub Actions for the current fork head may require upstream maintainer approval; hosted-job status must be checked on the exact head before review.

Still required before accepting live evidence

• all required GitHub Actions jobs must pass on exact head 913b159b5e1c78e8e26696c258a13c30f9d0620d
• the PR must be merged into canonical main; candidate B must then be rebuilt and attested for that exact canonical merge revision
• owner-authorized execution on real, separate canary and stable hosts using a GitHub CLI that supports the pinned attestation policy flags
• independently controlled operator and evidence-signing identities
• real systemd confirmation of terminal Result, ExecMainCode, ExecMainStatus, PID, and InvocationID behavior
• confirmation that installed guided assets are root-owned regular non-symlink files at mode 0755
• external repetition of candidate artifact provenance verification, or retention/replay of offline attestation bundles and trusted roots
• retention and independent verification of the signed result, result digest, and any failure logs
• approved immutable publication/anchoring of the resulting evidence package

Until those actions succeed, this remains a reviewed test-and-evidence implementation, not launch proof.

[wallscaler]

Review of the draft at head 913b159, from a local checkout. Nothing live was run; unit tests ran locally on Python 3.12.

cathedral-validator#206: Harden live updater E2E evidence finalization

Artifact / Question / Out of scope

• Artifact: PR Harden live updater E2E evidence finalization #206, head 913b159, base merge c33aa79 (PR test(updater): assert the published bootstrap wording #213), 6 files, +8332/-51. Draft. No CI has run on the fork head.
• Question: what does it change, is it redundant with or in conflict with main since Aug 28, does it weaken a check main enforces, is the tarball parser sound.
• Out of scope: running the two-host E2E; whether the 18-step scenario matches real host behavior; the PR's own claims of 572 tests and "four independent review passes".

What it actually changes

• Every line is hand-written. No scenario JSON, no wheel manifests, no evidence fixtures are checked in. No line in the diff is over 200 characters. The test file builds its synthetic evidence tree in code.
• Logic lives in two files:
  • scripts/live_update_e2e_result.py (4243 new): a five-subcommand tool (validate-key-pair, decode-capture, verify-capture, finalize, verify-result). Strict JSON, O_NOFOLLOW plus fstat-stable reads, bounded streaming tar validation, reviewed-asset binding by reading deploy/validator-update from its own checkout, GitHub release and tag record checks, Ed25519 sign and verify of a canonical result.
  • scripts/live_validator_update_e2e.sh (+564/-32): the controller. Changes are: three new required env vars for a dedicated evidence-signing key with path-containment checks and a must-differ check against the bootstrap and runtime release keys; preflight requires and uses gh attestation verify --source-digest --signer-workflow --deny-self-hosted-runners; host capture rewritten from one || true text dump into per-section base64 TSV with required/optional flags that the verifier decodes; the remote bootstrap staging moves to a root mktemp -d /var/tmp/cathedral-live.XXXXXXXXXX with a pattern guard and an EXIT trap, explicit sys.exit instead of assert, verify-before-extract order kept; the full bootstrap tarball and candidate B INPUTS.json are retained in evidence; guided-operator idempotence and stopped-writer proofs are written as JSON with installed asset metadata; cleanup() gains a whole-tree secret scan and then finalizes and re-verifies the signed result before printing the pass line; the controller git identity check now refuses when git itself fails instead of treating a failed git status as clean.
• Everything else is tests (+3406), docs (+117), and two CI lines (ruff on the verifier).

Overlap with main since Aug 28

• Base is c33aa79. Main has moved seven PRs since: fix(docs): authenticate bootstrap before extraction #214, deploy(updater): check for releases hourly #217, Re-read finalized history after broadcast instead of one immediate lookup #219, Point the operator docs at published bootstrap 2 #220, Keep bootstrap staging on failure and allow a same-shell retry #221, Require the install block to be regenerated by substitution #222, State the validator host requirements operators ask about first #223.
• Only docs/RELEASE_MAINTAINER.md is touched by both, in different hunks (PR: the live-test section around lines 186 to 330; main: README regeneration rules around lines 345 to 380). GitHub reports MERGEABLE.
• Redundant: nothing. fix(docs): authenticate bootstrap before extraction #214, Keep bootstrap staging on failure and allow a same-shell retry #221 and Require the install block to be regenerated by substitution #222 hardened the README operator install block. Harden live updater E2E evidence finalization #206 hardens the parallel copy of that flow inside the controller's configure_host. Same idea, different file, so there are now two hand-maintained copies of the staging pattern and neither test checks the other.
• Behavioral divergence, not a conflict: Keep bootstrap staging on failure and allow a same-shell retry #221 keeps staging on failure for forensics and allows a same-shell retry. The controller copy always rm -rfs on EXIT. Fine for a disposable test VM.
• deploy(updater): check for releases hourly #217 (timers 6h to 1h): no conflict. The controller overrides both timers with drop-ins to 60s. The verifier binds timer unit bytes to the verifying checkout, so evidence is revision-bound by design.
• Re-read finalized history after broadcast instead of one immediate lookup #219 (direct_writer finalized-history wait): unrelated.
• fix(live): bind the canary first install to its signed record #209, fix(operator): read unit file state when checking for conflicting units #210, fix(live): let the reactivation proof survive an unloaded timer #211 are already in the base; the PR consumes fix(live): let the reactivation proof survive an unloaded timer #211's reactivation observation pairs consistently.
• Still new: the verifier, the signed result, the capture format, the secret-scan content root, the attestation constraints, the guided transition proofs, the git-failure fix.

Verdict

• Not mergeable as-is. One deterministic failure in the paid path (F1) that the synthetic tests cannot see.
• Apart from F1, it strengthens every check it touches and weakens none. Signature verification still precedes extraction (and the controller test asserts the ordering). Attestation is tighter. Required capture sections now fail closed where main silently passes with || true. The git identity check is safer.
• The tarball parser is sound. It runs only on bytes that already matched the bootstrap-key-signed manifest digest, so a hostile tar needs the bootstrap signing key.
• Ask for a split: (a) controller hardening (staging, attestation flags, fail-closed capture, git fix) is small and reviewable now; (b) the 7k-line signed-result verifier plus tests needs its own review pass.

Findings

F1. The new whole-tree secret scan matches the disposable hotkey's public SS58 address in captured systemd output, so a fully successful live run ends TEARDOWN_NOT_PROVEN with exit 1 after the money is spent.

• Code path: the cleanup() scan heredoc in scripts/live_validator_update_e2e.sh builds needles from public_fields = ("ss58Address", "publicKey", "accountId") plus the private fields (PR file line 1110). TEST_HOTKEY is that same ss58Address (line 834) and is written to /etc/cathedral-validator/identity.env on both hosts (lines 2668, 2791). deploy/validator-update/cathedral-validator-direct.service:34 expands it into the process argv as --expected-hotkey=${CATHEDRAL_VALIDATOR_EXPECTED_HOTKEY}. capture_host runs systemctl status cathedral-validator-direct.service --full --no-pager (section direct_unit_status, optional); its CGroup block prints the main process command line unellipsized. decode_capture in scripts/live_update_e2e_result.py writes that section in plaintext to ${label}.d/direct_unit_status.txt and into ${label}.txt. The scan rglobs everything under EVIDENCE_DIR, finds the 48-character address, raises "operator key material occurred in evidence", and sets teardown_ok=0.
• User-visible: the run prints TEARDOWN_NOT_PROVEN, returns 1, produces no signed result, and the new LIVE_UPDATE_E2E_PASS line is unreachable whenever the direct service was running during any capture. It is running at the post-install and post-timer-B captures by design (assert_guided_status ... NOT_PROVEN true true).
• Relevance: this is exactly the contract the PR adds. Main's assert_no_operator_secret_in scans private fields only, and only the setup output, which is why valupd-0901h passed.
• Verified by reading the controller, the unit file, and the verifier. Not reproduced on a host. The optional journal capture is a second likely path since the direct validator holds the same address.
• Fix: scan private fields only across the tree. The public identity is not secret and is legitimately in unit state. If the author wants public fields covered, exclude the capture artifacts from those needles.

Refuted

• "8332 lines is mostly generated." No. Zero long lines, zero fixture files, tests build fixtures programmatically.
• "Retaining a 1 GiB tarball in evidence is a memory risk." No. Whole-file hash and tar walk are both streamed; per-member reads are bounded by the manifest size, manifest sizes are capped at 1 GiB each and 4 GiB total.
• "Tar path traversal, symlink, or hardlink members." No. Nothing is materialized; _bootstrap_archive_path rejects absolute, .., ., empty and non-canonical parts; member.isreg() rejects SYMTYPE and LNKTYPE; non-empty linkname is rejected; uid, gid, uname, gname, mtime, mode and size must equal the signed manifest record. Sparse members pass isreg() but the builder emits PAX without sparse and the bytes are signature-bound anyway.
• "Size cap bypass." No. _validate_retained_bootstrap_archive checks st_size against 1 GiB and against the signed size before reading, then re-checks the streamed total.
• "Conflicts with fix(docs): authenticate bootstrap before extraction #214 and Keep bootstrap staging on failure and allow a same-shell retry #221." No. Harden live updater E2E evidence finalization #206 does not touch README.md.
• "Secret-scan exclusion set drifts from files written after the scan." Checked. After the scan the controller writes only teardown-status.txt, controller-source-finalization.stderr and the three result files, all in SECRET_SCAN_EXCLUSIONS. The post-teardown-*.json and .err files are written before the scan.
• "The attestation flags do not exist." The three are real gh attestation verify flags and preflight refuses a gh without them. .github/workflows/release-candidate.yml exists on main. Not checked against a specific gh version.
• Tried to break F1: could the direct service be inactive at every capture? No, see above. Could systemctl status omit the argv? It prints the CGroup process list with full command lines under --full. Could .d artifacts escape the scan? No, rglob("*").

Observations

• Local test run: 210 passed, 2 failed in tests/thin/test_live_update_e2e_result.py (test_controller_refuses_when_all_privileged_capture_commands_fail, test_controller_accepts_required_capture_when_optional_status_fails). Cause: BSD wc -c left-pads the byte count, so the TSV field is malformed on macOS. The command runs on the Linux host in production and CI is ubuntu. Environment-only.
• handle.read(1) after reading the full member is a tautology (tarfile bounds reads to member.size). Harmless.
• The remote claims check accepts bundle.size >= 0 where the README block refuses < 1. No effect: the controller already pinned the bundle digest from its own build output.
• The verifier binds "reviewed checkout bytes" by reading deploy/validator-update from its own checkout and does not record git identity. The doc tells the reviewer to verify from a clean checkout of source_revision_b. Procedural, and the author says so.
• Evidence dir now holds the complete bootstrap tarball including the wheelhouse. Expect hundreds of MB per run.
• Commits are authored as inference2026 <299740695+inference2026@users.noreply.github.com>; the PR is opened by skyrocket2026. Same pattern on the already-merged fix(docs): authenticate bootstrap before extraction #214 (commit 0c32fdc). Not a blocker; worth knowing.
• The gh attestation preflight and the cryptography import check happen before any paid resource. Good.

Questions for the author

1. Why include the public keyfile fields in the whole-tree scan when the expected hotkey is by design in identity.env and in the service argv? Has this branch been executed against a real host?
2. Will you split the controller hardening from the signed-result verifier?
3. Who is inference2026 relative to skyrocket2026?
4. Which gh version was the flag preflight tested against?