feat(infra): run web, worker, and Redis in one scale-to-zero machine (Phase 3) - #839
Merged
Conversation
Fold the ARQ worker and Redis into the web container so the entire stack (API, worker, Redis) can scale to zero together on Fly.io. Previously three always-on machines were required; with the combined mode a single machine is sufficient. Changes: - docker/start-combined.sh: bash supervisor that starts redis-server (localhost-only, AOF persistence to Fly volume), wraps arq in a restart loop (crashed worker must not kill the container), then exec-style-waits on gunicorn whose exit code governs the container. SIGTERM forwarded to all three children so redis flushes the AOF before the container dies (Fly auto-stop safety). - Dockerfile (prod stage only): install redis-server via apt-get with --no-install-recommends; COPY + chmod start-combined.sh. Default CMD (gunicorn only) is unchanged — combined mode is opt-in via the fly.toml command (wired in a later task). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exit-path fixes applied:
(A) redis fails to start — exit 1 fires, EXIT trap calls _shutdown, redis_pid
is empty so the guard no-ops cleanly.
(B) gunicorn clean exit (code 0) — wait+capture gives gunicorn_exit=0,
_shutdown kills worker then redis (AOF flush), EXIT trap sees
_shutting_down=1 and returns immediately, script exits 0.
(C) gunicorn crash (non-zero) — was killed by set -e at the bare `wait`
call, skipping _shutdown. Fixed with `gunicorn_exit=0; wait … ||
gunicorn_exit=$?` so we always reach _shutdown and flush redis AOF.
EXIT trap is belt-and-braces for any other unexpected set -e exit.
(D) SIGTERM steady-state — trap fires _shutdown: TERM gunicorn + worker loop
(worker loop's own TERM trap forwards to arq and waits), then waits for
both writers, then TERMs redis for AOF flush. exit 143 prevents script
resuming after cleanup.
(E) SIGTERM during startup — trap was registered only after redis start +
readiness loop + worker spawn, so bash-as-PID-1 was ignoring SIGTERM
for up to ~15 s. Fixed by initialising all PIDs to "" and registering
trap _before_ starting redis; kill -0 guards tolerate empty strings.
(F) SIGTERM between trap registration and gunicorn start — trap fires, worker
loop is TERMed (forwards to arq), redis is TERMed; gunicorn_pid="" guard
skips gunicorn kill; exit 143 prevents gunicorn from being launched
against dead redis.
Additional fixes:
- _shutdown is idempotent via _shutting_down flag (required for EXIT trap +
double-SIGTERM mid-shutdown safety).
- _run_worker_loop: registers TERM trap to forward signal to $arq_pid;
uses `sleep 2 & wait $!` backoff so the loop remains signal-responsive
during the 2-second restart delay.
- redis-server: added --save "" to disable RDB snapshots (AOF already
persists; avoids double-persistence on the volume).
- Dockerfile: replaced `COPY + RUN chmod` two-layer pattern with
`COPY --chmod=755`; fixed misleading comment about --entrypoint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hine Folds the separate wikimind-redis Fly app and the worker process group into the existing wikimind web machine. docker/start-combined.sh (added in Task 1) supervises all three processes (redis-server, arq worker, gunicorn) inside one container; fly.toml and fly.staging.toml now point [processes].web at that script instead of running gunicorn directly. Key changes: - fly.toml / fly.staging.toml: single `web` process running ./docker/start-combined.sh; WIKIMIND_REDIS_URL=redis://localhost:6379/0 added to [env]; kill_timeout = "30s" (allows gunicorn graceful drain + arq wind-down + redis AOF flush before Fly kills the container); memory bumped to 2 GB because three processes now share the machine (cost only accrues while the machine is awake — scale-to-zero keeps idle cost at zero). - deploy.yml: removes the "Deploy self-hosted Redis" step and both "Ensure worker machines are running" steps (no worker process group anymore); "Validate production infrastructure" now FAILS if WIKIMIND_REDIS_URL is still set as a Fly secret (Fly secrets silently override [env] — operator must run `fly secrets unset WIKIMIND_REDIS_URL --app wikimind` before cutover); WIKIMIND_REDIS_URL removed from staging secrets (covered by [env]). - docker.yml: fly.redis.toml removed from both push + pull_request path filters. - fly.redis.toml: deleted (wikimind-redis app + its volume are destroyed at cutover; runbook in PR). - scripts/fly-setup.sh: Redis app/volume provisioning and WIKIMIND_REDIS_URL secret-staging removed; Redis is now fully in-machine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The prod Docker image copies the script to /app/start-combined.sh. The /app/docker directory does not exist in the image, so machines would crash on boot with the old path. Changed to ./start-combined.sh which resolves to /app/start-combined.sh at runtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eadroom 1. deploy.yml — replace grep -c secret check with awk+grep -qx; captures flyctl failures instead of swallowing them as "0", and avoids the "0\n0" false-pass when grep exits 1. 2. deploy.yml — add actions/checkout step as first step of rollback-production so fly.toml is present in cwd when flyctl deploy runs. 3. fly.toml + fly.staging.toml — raise kill_timeout from 30s to 60s; 30s equalled gunicorn's graceful_timeout, leaving zero headroom for the redis AOF flush that follows. 4. fly.staging.toml [[vm]] comment — staging pins min_machines_running=1, so idle cost is bounded by the CI teardown job, not scale-to-zero. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documents the decision to co-locate web, ARQ worker, and Redis in a single scale-to-zero Fly.io machine via start-combined.sh, with explicit coverage of the four accepted trade-offs (opportunistic crons, no Redis restart loop, rollback pairing, ~1s AOF durability window). Regenerates ADR index and syncs openapi.yaml (pre-existing drift from Tasks 1-2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Redis persists its AOF data directory to the volume, not a socket - ARQ while-loop is crash-recovery supervision (worker is long-running, not burst-mode) - Rollback trade-off path corrected to /app/start-combined.sh - fly-setup.sh line reduction is 27 (not ~40) - 6-hourly cron is two jobs: subscription and price reconciliation - Status changed from bold **Status:** to ## Status heading so regenerate_adr_index.py parses it correctly; regenerated docs/adr/README.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integer seconds format for kill_timeout matches ADR and Fly's documented examples. Add permissions block to rollback-production job to allow checkout action. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After the Phase 1–2 cost work (~$61 → ~$10/mo), the remaining always-on floor is three pieces: the ARQ worker machine (24/7), the self-hosted
wikimind-redisapp (24/7), and Postgres. The web machine scales to zero but the app as a whole never sleeps.Solution
Fold worker + Redis into the single web machine so everything shares one scale-to-zero lifecycle (ADR-029 documents the decision and trade-offs). Postgres deliberately stays external — its migration is a separate decision. Floor drops to ~$4–5/mo (Postgres + volumes).
docker/start-combined.sh— supervised PID-1: redis-server (localhost, 128mb, noeviction, AOF on the volume,--save "") → ARQ worker in a crash-restart loop → gunicorn as the governing process. Ordered shutdown: drain gunicorn+arq first, TERM redis last so the AOF flush covers all writers. Signal paths hardened after adversarial review (trap before redis start, EXIT-trap backstop, TERM forwarded into the worker subshell).fly.toml/fly.staging.toml— singlewebprocess running the script;WIKIMIND_REDIS_URL=redis://localhost:6379/0in[env];kill_timeout = 60; 2GB VM (3 gunicorn workers + arq + redis; cost only accrues while awake).deploy.yml— self-hosted-Redis deploy step and both ensure-worker steps removed; new guard fails the deploy if a staleWIKIMIND_REDIS_URLsecret exists (Fly secrets override[env]— a stale secret would silently repoint at the dead external Redis); rollback job gets checkout +contents: read.fly.redis.toml, redis provisioning inscripts/fly-setup.sh, redis path filters indocker.yml.openapi.yamlsync (main's copy was stale — missing billing endpoints).Verification
make verifygreen in an isolated worktree: 2134 tests, lint/typecheck/pyright/doc-sync all pass.fly config validatepasses for both tomls; deploy-guard shell logic unit-simulated (clean + stale cases).start-combined.shand the smoke suite's job-ping round-trips through the in-machine Redis before prod deploys.Cutover runbook (I'll execute post-merge)
fly secrets unset WIKIMIND_REDIS_URL -a wikimind --stage(staged — no restart; applies atomically with the deploy).--yesauto-confirms destruction of the orphanedworkerprocess-group machine).fly apps destroy wikimind-redis --yes+ verify no orphan volumes.Rollback note: across this boundary, rollback needs matching image+config pairs (old images lack
/app/start-combined.sh) —git checkout <prev> -- fly.toml && fly deploy --image <prev-image>.Accepted trade-offs (in ADR-029)
Crons run opportunistically (only while awake); Redis has no restart loop (
/health/deepreports it; next wake cycle self-heals); ~1s AOF durability window on hard kill.🤖 Generated with Claude Code