Stack backups: scheduled volume archives to SFTP/local storage (ADR-0016)

[swimmesberger]

Summary

Implements the stack backup system designed in ADR-0016: daily archives of each opted-in stack's named volumes, shipped to external storage with retention and optional encryption, plus on-demand volume downloads from the UI.

How it works

• Snapshot: a helper container is created but never started with the stack's volumes bind-mounted read-only; the Docker archive endpoint streams one tar out (no code executes in the helper, any pullable image works — default busybox:stable). A backup-manifest.json (instance, stack, volumes, timestamp) is injected so every archive self-describes.
• Consistency: per-stack stop containers during the snapshot toggle (default on). The stop window covers only the local snapshot — containers restart before the upload begins.
• Storage: IBackupStorage with two built-ins — SFTP (SSH.NET; password and/or private key; Hetzner Storage Box friendly, port 23 documented) and local directory. Layout: {base}/{instance}/{stack}/{project}_{yyyyMMddTHHmmssZ}.tar.gz[.enc]. Uploads write a .partial name and rename on completion.
• Encryption: optional passphrase → AES-256-CBC in the OpenSSL enc container format (PBKDF2-SHA256, 600k iterations) — restore needs nothing but stock openssl; an interop test decrypts a real openssl-CLI ciphertext.
• Retention: days + max-count per stack folder after each successful run; only Watchtower-named files are considered and the newest backup is never deleted.
• Scheduling/history: global daily window (Backup:Time) + per-stack opt-in; single-flight queue with per-stack coalescing; BackupEvent history rows swept to failed on restart, like deploys.
• Settings: runtime-editable Watchtower:Backup:* via the settings store with env-var pinning (ADR-0014); secrets are write-only toward the UI. New RPC surface: backups.getConfig/updateConfig/testStorage/events/run/getStackConfig/setStackConfig.
• Volume download: GET /api/volumes/{name}/download streams any named volume as .tar.gz (same helper mechanism), session-protected like the SSE streams.

UI

• Settings → Backups card: schedule, retention, encryption, provider config, Test storage probe (writes + deletes a probe file, surfacing the backend's own error text on failure).
• Stack detail → Backups tab: schedule opt-in, stop-containers toggle, Back up now, expandable run history (status, size, remote path, log).
• Download archive action on the stack Volumes tab and the Infrastructure volumes list.

Docs

• docs/backups.md: setup (incl. Storage Box specifics and SSH key guidance), encryption notes, and the step-by-step restore procedure; linked from the README.
• docs/decisions/0016-stack-backups.md records the trade-offs (file-level snapshots, CBC-without-MAC, manual restore in v1).

Verification

• Solution builds with 0 warnings; 724 tests pass (25 new: encryption round-trip + openssl interop vector, naming/retention, local storage provider).
• tsc and the production frontend build are clean; rpc-schema.json regenerated (101 methods).
• New EF migration AddStackBackups (backfills backup_stop_containers = true for existing stacks).

Restore (second commit)

• In-app restore: stack Backups tab → Restore… lists the archives actually present on the storage (backups.listRemote — the local history is not the truth, retention deletes behind it), then a typed-name confirm runs the inverse of a backup: download + decrypt, scan the tar's table of contents, stop the stack, wipe and refill only volumes present in both the archive and on the host (mismatches logged, never guessed), restart. Lands in the same history as trigger restore.
• Refused while a deploy is running or the stack already has a backup/restore queued/running; .enc archives require the configured passphrase up front. The volume wipe is the only step that executes code in the helper container (needs sh/rm — busybox has them).
• IBackupStorage gains DownloadAsync + sizes on listing; 4 more tests (archive table-of-contents scan, storage download).
• Docs: UI restore is now the primary path; the manual openssl route remains as disaster recovery.