chore(ci): make staging ephemeral — provision per deploy, destroy after

[manavgup]

Problem

The June Fly.io invoice was ~$61/mo. About $25/mo of that was the staging environment: a 24/7 clone of production (web + worker machines, Postgres cluster, volume) plus a leftover managed Upstash Redis — all to serve as a smoke-test gate that's actually exercised for about ten minutes per deploy.

Solution

Staging now only exists while a deploy is running. The pipeline provisions a fresh staging stack, smoke-tests it, promotes to production, and destroys staging — pass or fail.

• Clean-slate provisioning: deploy-staging destroys any leftovers, then creates the app, volume, and Postgres, attaches, and sets secrets. Destroy-then-create (rather than create-if-absent) is immune to partial state from a failed teardown.
• Fresh empty DB per run is safe: migrations run via the alembic release_command; the smoke-test JWT resolves user_id from its sub claim with no DB lookup (src/wikimind/api/deps.py); every checked endpoint returns a valid empty array on a fresh DB.
• New teardown-staging job with if: always() destroys wikimind-staging + wikimind-staging-db after the smoke gate.
• Redis: staging queues on the shared self-hosted wikimind-redis at DB index 1 (prod uses DB 0). No dedicated staging Redis. (Already live: I repointed the running staging at DB 1 and destroyed the Upstash instance — verified healthy, worker connected, background_mode: arq.)
• Docling: staging reuses the prod sidecar (stateless, scale-to-zero) instead of deploying its own 4GB-image copy per run. wikimind-staging-docling is no longer referenced and will be destroyed after this merges.

Changes

• .github/workflows/deploy.yml — provision/teardown jobs, staging Redis URL staged by CI, staging docling deploy removed, header docs rewritten
• fly.staging.toml — docling URL → prod sidecar; header notes the ephemeral model
• README.md, docs/adr/adr-025-docling-serve-sidecar.md — deploy-flow descriptions updated

Notes

• First deploy after merge replaces the current always-on staging stack automatically (clean-slate provision + teardown). I'll destroy wikimind-staging-docling manually once this lands.
• Trade-off: ~2 min extra per deploy for Postgres provisioning (deploy-staging timeout bumped 10 → 15 min); staging DB state never persists between runs.
• Companion PR chore(infra): scale prod web to zero when idle #831 handles prod web scale-to-zero + the uptime monitor.

🤖 Generated with Claude Code