From 06ef85217277eac7f297a9b30eb54490530f39cd Mon Sep 17 00:00:00 2001 From: Matthew Demidoff Date: Sun, 6 Sep 2026 07:45:46 +0200 Subject: [PATCH] docs(runbook): move the env file out of the checkout and add the secret rotation matrix Documents the production env file's new home (~/.config/bottleneck-auth/, loaded through COMPOSE_ENV_FILES), the one-time migration, and why a .env in the checkout should not exist. Replaces the four-line "suspected leaked secret" note with a rotation matrix that names every secret's consumers, its restart set, the blast radius of the gap, and whether an overlap exists, plus step-by-step procedures for the ones that are easy to get wrong: POSTGRES_PASSWORD (alter role over the socket, then --no-deps restarts, and the db recreate that follows), the OIDC key via retired status with no downtime, the bot token, the webhook secret, and the tunnel token. Ends with a triage order for a suspected leak. --- .env.example | 3 ++ README.md | 2 +- docs/deployment.md | 44 +++++++++++++++++++++- runbooks/oncall.md | 91 ++++++++++++++++++++++++++++++++++++++++++---- 4 files changed, 129 insertions(+), 11 deletions(-) diff --git a/.env.example b/.env.example index 754623b..7082d2a 100644 --- a/.env.example +++ b/.env.example @@ -1,3 +1,6 @@ +# Production copy of this file lives at ~/.config/bottleneck-auth/prod.env, +# loaded via COMPOSE_ENV_FILES; see docs/deployment.md. Do not keep a .env in +# the checkout. AUTH_BASE_URL=https://auth.bneck.com CLOUDFLARED_TOKEN= diff --git a/README.md b/README.md index cf15a47..61fddf8 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,7 @@ the migration smoke test, the production build, and both Docker image builds on every push. The `sdk` job separately builds and type-checks the Node SDK. The activation flow can be exercised end to end with `test.py`. Set -`TEST_BEARER` to an external app's API key in `.env`, then run `python +`TEST_BEARER` to an external app's API key in a local env file, then run `python test.py`; it creates an activation request, prints an approval URL, and polls for the profile. diff --git a/docs/deployment.md b/docs/deployment.md index f81bf36..0a40e56 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -4,9 +4,49 @@ The reference deployment runs the Compose stack on a single small host behind a Cloudflare Tunnel. The app is not published on a host port; the tunnel connects to it inside the Compose network. -## Required environment +## Environment file -Copy `.env.example` to `.env` and set at least: +Production secrets live outside the checkout, in a directory only the deploying +user can read: + +```text +~/.config/bottleneck-auth/ # mode 0700 + prod.env # mode 0600, the values from .env.example + oidc-private.pem # if the OIDC key is kept as a file + oidc-public.pem +``` + +Compose finds the file through `COMPOSE_ENV_FILES`, exported once in the +deploying user's shell: + +```sh +export COMPOSE_ENV_FILES="$HOME/.config/bottleneck-auth/prod.env" +``` + +Every `docker compose` command then reads it. Without the export, the first +`${POSTGRES_PASSWORD:?...}` interpolation fails closed with "set +POSTGRES_PASSWORD" rather than silently using stale values; the per-command +form is `docker compose --env-file "$HOME/.config/bottleneck-auth/prod.env" +...`. A `.env` in the repo root is not used and should not exist there: it is +readable by anything with the checkout, and `docker compose config` prints +every value it resolves, so run that with `--quiet` on the host. + +Moving an existing deployment (no container restart needed, the values do not +change): + +```sh +mkdir -p ~/.config/bottleneck-auth && chmod 700 ~/.config/bottleneck-auth +mv .env ~/.config/bottleneck-auth/prod.env && chmod 600 ~/.config/bottleneck-auth/prod.env +mv oidc-private.pem oidc-public.pem ~/.config/bottleneck-auth/ 2>/dev/null || true +echo 'export COMPOSE_ENV_FILES="$HOME/.config/bottleneck-auth/prod.env"' >> ~/.zshrc +# new shell, then: +docker compose config --quiet && docker compose ps +``` + +Development-only values (`TEST_BEARER`, a scratch `DATABASE_URL`) belong in a +separate file passed with `--env-file`, not in `prod.env`. + +Copy `.env.example` and set at least: - `POSTGRES_PASSWORD` - `OIDC_PRIVATE_KEY_PEM` (an RSA private key; `OIDC_KEY_ID` to name it) diff --git a/runbooks/oncall.md b/runbooks/oncall.md index 233a4cc..b647b8e 100644 --- a/runbooks/oncall.md +++ b/runbooks/oncall.md @@ -30,8 +30,9 @@ docker compose logs app | grep '"level":"error"' `docker compose restart db && docker compose up -d app`. 4. If the app crash-loops at boot with "missing required environment variables", or the bot with " is required", a secret is unset — see - `.env` against `.env.example`. `docker compose` refusing to start with - "set " is the same cause. + the env file (`docs/deployment.md`, Environment file) against + `.env.example`. `docker compose` refusing to start with "set " is the + same cause, and so is a shell without `COMPOSE_ENV_FILES` exported. ### Webhook backlog or an auto-disabled endpoint - `webhook_endpoint_auto_disabled` in the worker logs (and a Telegram alert if @@ -54,12 +55,86 @@ docker compose logs app | grep '"level":"error"' - Counters live in Redis; flushing them is safe but global. Prefer waiting out the window. -### Suspected leaked secret -1. Rotate the secret in `.env` (e.g. `OAUTH_CSRF_SECRET`, `OIDC_PRIVATE_KEY_PEM`, - `TELEGRAM_BOT_TOKEN`, `POSTGRES_PASSWORD`). -2. `docker compose up -d --build app worker` to pick it up. -3. Rotating `OAUTH_CSRF_SECRET` invalidates in-flight consent/activation CSRF - tokens (users retry). Rotating the OIDC key invalidates issued tokens. +## Secrets + +### Where they live +`~/.config/bottleneck-auth/prod.env` (0600, directory 0700), loaded through +`COMPOSE_ENV_FILES`; see `docs/deployment.md`. Values still reach `docker +inspect` of each container, so a host-level compromise is a compromise of +every secret below regardless of the file's mode. + +### Rotation matrix +Restart sets use `--no-deps` on purpose: a bare `docker compose up -d` after +changing `POSTGRES_PASSWORD` also recreates `db` (its own environment +changed), which is a short outage you should schedule, not trip over. + +| Secret | Consumers | Restart set | Blast radius of the gap | Overlap | +| --- | --- | --- | --- | --- | +| `POSTGRES_PASSWORD` | `db` (initdb only), `app`, `worker` via `DATABASE_URL` | `alter role`, then `up -d --no-deps app worker` | DB errors until both restart | none; seconds | +| `OIDC_PRIVATE_KEY_PEM`, `OIDC_KEY_ID`, `OIDC_SIGNING_KEYS_JSON` | `app` | `up -d --no-deps app` | tokens signed by a dropped key stop verifying | yes, via `retired` status | +| `OAUTH_CSRF_SECRET` | `app` | `up -d --no-deps app` | in-flight consent/activation forms fail once | none | +| `TELEGRAM_BOT_TOKEN` | `app`, `worker`, `bot` | `up -d --no-deps app worker bot` | 2FA prompts, notifications, bot sign-in fail; queued jobs retry | edit env, then BotFather revoke | +| `TELEGRAM_BOT_WEBHOOK_SECRET` | `app`, `bot` | `up -d --no-deps app bot` | a tap in the gap fails; retry works | none | +| `INTERNAL_ANALYTICS_SECRET` | `app`, the external analytics caller | `up -d --no-deps app` and the caller | analytics posts 401 until the caller updates | none | +| `TURNSTILE_SECRET_KEY`, `TURNSTILE_SITE_KEY`, `NEXT_PUBLIC_TURNSTILE_SITE_KEY` | `app` (read at runtime) | `up -d --no-deps app` | forms fail closed until restart | create the new widget first | +| `CLOUDFLARED_TOKEN` | `cloudflared` | `up -d --no-deps cloudflared` | seconds of tunnel outage | second tunnel + DNS cutover; rarely worth it | +| `RESEND_API_KEY` | `app` | `up -d --no-deps app` | verification emails fail | create new, deploy, delete old | +| `OAUTH_DYNAMIC_REGISTRATION_TOKEN` | `app`, DCR clients | `up -d --no-deps app` | DCR calls 401 until clients update | none | + +`BEARER_ADMIN_TELEGRAM_ID` and `ALERT_TELEGRAM_CHAT_ID` are identifiers, not +secrets, but changing them takes the same restart sets as `TELEGRAM_BOT_TOKEN`. + +### Procedures + +#### POSTGRES_PASSWORD +1. Put the new value in the env file (`POSTGRES_PASSWORD` and the password + inside `DATABASE_URL` if it is spelled out there). +2. Change it inside Postgres over the unix socket, which does not need the old + password: + `docker compose exec -T db psql -U auth -d auth -c "alter role auth password ''"` +3. Immediately: `docker compose up -d --no-deps app worker`. Between steps 2 + and 3 every new connection fails, so keep them back to back. +4. The `db` container's environment is now stale; the next full + `docker compose up -d` recreates it (~5 s, data stays in the volume). Do + that deliberately at a quiet moment rather than as a surprise. + +#### OIDC signing key (no downtime) +Verification accepts every key that is not `revoked`, and the JWKS publishes +the same set, so a rotation is: add the new key as active, keep the old one as +retired until every token it signed has expired, then revoke it. +1. Set `OIDC_SIGNING_KEYS_JSON` to + `[{"kid":"2026-09","privateKeyPem":"","status":"active"},{"kid":"","privateKeyPem":"","status":"retired"}]`. + Once this variable is set, `OIDC_PRIVATE_KEY_PEM` is ignored. +2. `docker compose up -d --no-deps app`, then confirm both kids appear in + `https://auth.bneck.com/oauth/jwks`. +3. Wait at least 24 h (access tokens live 15 min, but relying parties cache + the JWKS), flip the old entry to `"revoked"`, redeploy, and drop it from + the list on a later change. + +#### TELEGRAM_BOT_TOKEN +1. Put the new token in the env file first. +2. `/revoke` the old one in BotFather; the old token dies instantly. +3. Within a minute: `docker compose up -d --no-deps app worker bot`. 2FA + prompts sent in the gap fail (the user retries); queued notifications retry + under bullmq. The bot re-pins a fresh status message on start. + +#### TELEGRAM_BOT_WEBHOOK_SECRET +Edit the env file and `docker compose up -d --no-deps app bot` in one go; the +two sides must agree. + +#### CLOUDFLARED_TOKEN +Refresh the token in Zero Trust, edit the env file, `docker compose up -d +--no-deps cloudflared`. Expect a few seconds of tunnel downtime and one false +alarm from any external probe. + +### Suspected leak: triage order +1. OIDC signing key (token forgery), then `TELEGRAM_BOT_TOKEN` and + `TELEGRAM_BOT_WEBHOOK_SECRET` (2FA approval forgery). Rotate these first. +2. `OAUTH_CSRF_SECRET`. +3. `POSTGRES_PASSWORD`: the database is not reachable from outside the compose + network, so real exposure needs host access, which is a bigger incident. +4. Everything else in the matrix, then review `security_events` for activity + during the exposure window (Audit trail below). ## Deploys and restarts