Validate moor_update_apply rollback states on staging

[caiopizzol]

Follow-up to #80. The transient updater (#92-#101) ships rollback code with unit-test coverage (9 shell-level tests in apps/respawner/entrypoint.test.ts using stubbed docker/curl), but the rolled_back / rollback_failed paths have NOT been live-validated. Production smoke from #80 only covered:

• success (three in-place self-updates: 0.39.0 → 0.40.0 via manual, 0.40.0 → 0.40.1, 0.40.1 → 0.41.0, 0.41.0 → 0.42.0 via moor_update_apply)
• failed pre-replacement (nonexistent target_digest → pull error captured before container swap)

Production smoke CANNOT safely cover the post-replacement failure path because it requires a deliberately broken image that pulls successfully but fails at docker compose up --wait or /api/health.

Scope

Spin up a staging VM (any cheap VPS, or a local docker-compose-on-laptop install) running moor at a version that has moor_update_apply. Build and push two deliberately-broken test images to a registry (or load locally), then validate:

Acceptance

1. rolled_back happy path. Push an image whose /api/health listener binds only 127.0.0.1:3000 (not 0.0.0.0). Compose's in-container healthcheck (docker-compose.yml:17, uses localhost:3000) passes, so compose up --wait succeeds and moor is replaced. The respawner's cross-container poll (entrypoint.sh:297, hits http://moor:3000/api/health over the Docker bridge) gets connection refused, times out after 60s, and triggers the rollback path. This exercises the post-replacement health-poll branch specifically (vs. the compose-up-fail branch a health 500 image would hit, which is the same rolled_back outcome via a different code path). Run moor_update_apply({ target_digest: <broken digest> }). Expect:

   • audit row state=rolled_back
   • error_log mentions "health check did not pass" (from apply phase)
   • rollback_error null
   • moor container's resolved .Image (the sha256:... image id) back on the previous value. .Config.Image stays ghcr.io/caiopizzol/moor:latest after rollback because the respawner retags :latest to prev_image_id in place (entrypoint.sh:194); only the resolved image id reveals which content :latest now points to.
   • moor is healthy
   • No .update-rollback-<audit_id>.yml or .update-override-<audit_id>.yml is generated under working_dir or /app/data (moor_update_apply leaks compose overrides into the next container's config_files label (label pollution + local :latest drift) #105: respawner no longer appends its own -f; retag of :latest to prev_image_id is the whole rollback mechanism)
   • drain stays enabled (operator must moor_drain_disable)

2. rollback_failed (harder to provoke). One way: between the apply pull and the up step, delete the previous image locally. The rollback's docker tag <prev_image_id> :latest will fail because the image is gone. Expect:

   • state=rollback_failed
   • error_log carries the original apply failure
   • rollback_error carries "docker tag failed" with the missing-image error
   • drain stays enabled
   • operator can manually docker compose up -d moor to recover with the bad latest tag, OR pull a known-good version explicitly

3. crashed via timeout. Apply with a deliberately broken image, kill the respawner container mid-flight (e.g., docker kill moor-respawner-<id>). Wait 30+ min. Expect:

   • audit row state transitions in_progress → crashed via startup/periodic sweep
   • error_log notes the grace-window expiry
   • drain stays enabled

Out of scope

• Adding rollback validation to CI. The setup cost outweighs the value for a flow that runs on production hosts once or twice a release.
• Auto-restore of the DB backup on rollback. PR Bind project container host ports to 127.0.0.1 (or drop host publishing entirely) #5 deliberately leaves DB restore manual.

Done when

Each of the three acceptance scenarios is recorded in a comment on this issue with the exact docker/moor commands run and the resulting moor_update_audit output. Then close.

[caiopizzol]

Acceptance #1 handoff is ready to run on a native amd64 host.

Harness artifact

• Tarball: /tmp/moor-rollback-smoke.tar.gz
• SHA-256: 930f64099b54a3a37451eac6955b3c199c6569d72115e9e76b3839060d058246
• No stale run.log is included in the archive.
• Script has an x86_64 guard and exits early on non-amd64 unless ALLOW_NON_AMD64=1 is set.
• Requirements on a fresh Debian/Ubuntu host: Docker + compose plugin, curl, jq.

Target image

ghcr.io/caiopizzol/moor:0.48.0-test-broken-health

Manifest-list digest:

sha256:cdc3fbd14bdad9d489e38710853463b299bd32444a9aec919e9cb1067348ee32

Verified manifests include runnable linux/amd64 and linux/arm64 variants. This acceptance run should still use a native amd64 host because current moor/moor-respawner release images are amd64-only (#126) and update pulls are platform-unaware (#127).

Run commands

tar -xzf moor-rollback-smoke.tar.gz
cd moor-rollback-smoke/staging

docker compose up -d --wait

LOG="../run-$(date -u +%Y%m%dT%H%M%SZ).log"
TARGET_DIGEST=sha256:cdc3fbd14bdad9d489e38710853463b299bd32444a9aec919e9cb1067348ee32 \
  ./run-rollback-smoke.sh 2>&1 | tee "$LOG"

docker compose down -v

Post the captured run-<timestamp>.log here as the evidence for rolled_back acceptance #1. Do not delete the GHCR test tag until the comment is posted and no rerun is needed.

This does not close the issue by itself. rollback_failed and crashed still need separate live recordings per the Done When section.

[caiopizzol]

Operational note for the acceptance #1 handoff:

• The harness has been validated repeatedly; do not keep revising it unless the amd64 run finds a concrete failure. Further edits invalidate the checksum and make the handoff noisier.
• On Debian/Ubuntu cloud images, jq is the most likely missing dependency. Install it if needed before running.
• If the run fails at any assertion, post the run-<timestamp>.log anyway. Debug from the real output rather than guessing.