docs(runbook): move the env file out of the checkout and add the secret rotation matrix

.env.example

@@ -1,3 +1,6 @@
+# Production copy of this file lives at ~/.config/bottleneck-auth/prod.env,
+# loaded via COMPOSE_ENV_FILES; see docs/deployment.md. Do not keep a .env in
+# the checkout.
 AUTH_BASE_URL=https://auth.bneck.com
 CLOUDFLARED_TOKEN=
 

README.md

@@ -101,7 +101,7 @@ the migration smoke test, the production build, and both Docker image builds
 on every push. The `sdk` job separately builds and type-checks the Node SDK.
 
 The activation flow can be exercised end to end with `test.py`. Set
-`TEST_BEARER` to an external app's API key in `.env`, then run `python
+`TEST_BEARER` to an external app's API key in a local env file, then run `python
 test.py`; it creates an activation request, prints an approval URL, and polls
 for the profile.
 

docs/deployment.md

@@ -4,9 +4,49 @@ The reference deployment runs the Compose stack on a single small host behind
 a Cloudflare Tunnel. The app is not published on a host port; the tunnel
 connects to it inside the Compose network.
 
-## Required environment
+## Environment file
 
-Copy `.env.example` to `.env` and set at least:
+Production secrets live outside the checkout, in a directory only the deploying
+user can read:
+
+```text
+~/.config/bottleneck-auth/           # mode 0700
+    prod.env                         # mode 0600, the values from .env.example
+    oidc-private.pem                 # if the OIDC key is kept as a file
+    oidc-public.pem
+```
+
+Compose finds the file through `COMPOSE_ENV_FILES`, exported once in the
+deploying user's shell:
+
+```sh
+export COMPOSE_ENV_FILES="$HOME/.config/bottleneck-auth/prod.env"
+```
+
+Every `docker compose` command then reads it. Without the export, the first
+`${POSTGRES_PASSWORD:?...}` interpolation fails closed with "set
+POSTGRES_PASSWORD" rather than silently using stale values; the per-command
+form is `docker compose --env-file "$HOME/.config/bottleneck-auth/prod.env"
+...`. A `.env` in the repo root is not used and should not exist there: it is
+readable by anything with the checkout, and `docker compose config` prints
+every value it resolves, so run that with `--quiet` on the host.
+
+Moving an existing deployment (no container restart needed, the values do not
+change):
+
+```sh
+mkdir -p ~/.config/bottleneck-auth && chmod 700 ~/.config/bottleneck-auth
+mv .env ~/.config/bottleneck-auth/prod.env && chmod 600 ~/.config/bottleneck-auth/prod.env
+mv oidc-private.pem oidc-public.pem ~/.config/bottleneck-auth/ 2>/dev/null || true
+echo 'export COMPOSE_ENV_FILES="$HOME/.config/bottleneck-auth/prod.env"' >> ~/.zshrc
+# new shell, then:
+docker compose config --quiet && docker compose ps
+```
+
+Development-only values (`TEST_BEARER`, a scratch `DATABASE_URL`) belong in a
+separate file passed with `--env-file`, not in `prod.env`.
+
+Copy `.env.example` and set at least:
 
 - `POSTGRES_PASSWORD`
 - `OIDC_PRIVATE_KEY_PEM` (an RSA private key; `OIDC_KEY_ID` to name it)

runbooks/oncall.md

@@ -30,8 +30,9 @@ docker compose logs app | grep '"level":"error"'
    `docker compose restart db && docker compose up -d app`.
 4. If the app crash-loops at boot with "missing required environment
    variables", or the bot with "<NAME> is required", a secret is unset — see
-   `.env` against `.env.example`. `docker compose` refusing to start with
-   "set <NAME>" is the same cause.
+   the env file (`docs/deployment.md`, Environment file) against
+   `.env.example`. `docker compose` refusing to start with "set <NAME>" is the
+   same cause, and so is a shell without `COMPOSE_ENV_FILES` exported.
 
 ### Webhook backlog or an auto-disabled endpoint
 - `webhook_endpoint_auto_disabled` in the worker logs (and a Telegram alert if
@@ -54,12 +55,86 @@ docker compose logs app | grep '"level":"error"'
 - Counters live in Redis; flushing them is safe but global. Prefer waiting out
   the window.
 
-### Suspected leaked secret
-1. Rotate the secret in `.env` (e.g. `OAUTH_CSRF_SECRET`, `OIDC_PRIVATE_KEY_PEM`,
-   `TELEGRAM_BOT_TOKEN`, `POSTGRES_PASSWORD`).
-2. `docker compose up -d --build app worker` to pick it up.
-3. Rotating `OAUTH_CSRF_SECRET` invalidates in-flight consent/activation CSRF
-   tokens (users retry). Rotating the OIDC key invalidates issued tokens.
+## Secrets
+
+### Where they live
+`~/.config/bottleneck-auth/prod.env` (0600, directory 0700), loaded through
+`COMPOSE_ENV_FILES`; see `docs/deployment.md`. Values still reach `docker
+inspect` of each container, so a host-level compromise is a compromise of
+every secret below regardless of the file's mode.
+
+### Rotation matrix
+Restart sets use `--no-deps` on purpose: a bare `docker compose up -d` after
+changing `POSTGRES_PASSWORD` also recreates `db` (its own environment
+changed), which is a short outage you should schedule, not trip over.
+
+| Secret | Consumers | Restart set | Blast radius of the gap | Overlap |
+| --- | --- | --- | --- | --- |
+| `POSTGRES_PASSWORD` | `db` (initdb only), `app`, `worker` via `DATABASE_URL` | `alter role`, then `up -d --no-deps app worker` | DB errors until both restart | none; seconds |
+| `OIDC_PRIVATE_KEY_PEM`, `OIDC_KEY_ID`, `OIDC_SIGNING_KEYS_JSON` | `app` | `up -d --no-deps app` | tokens signed by a dropped key stop verifying | yes, via `retired` status |
+| `OAUTH_CSRF_SECRET` | `app` | `up -d --no-deps app` | in-flight consent/activation forms fail once | none |
+| `TELEGRAM_BOT_TOKEN` | `app`, `worker`, `bot` | `up -d --no-deps app worker bot` | 2FA prompts, notifications, bot sign-in fail; queued jobs retry | edit env, then BotFather revoke |
+| `TELEGRAM_BOT_WEBHOOK_SECRET` | `app`, `bot` | `up -d --no-deps app bot` | a tap in the gap fails; retry works | none |
+| `INTERNAL_ANALYTICS_SECRET` | `app`, the external analytics caller | `up -d --no-deps app` and the caller | analytics posts 401 until the caller updates | none |
+| `TURNSTILE_SECRET_KEY`, `TURNSTILE_SITE_KEY`, `NEXT_PUBLIC_TURNSTILE_SITE_KEY` | `app` (read at runtime) | `up -d --no-deps app` | forms fail closed until restart | create the new widget first |
+| `CLOUDFLARED_TOKEN` | `cloudflared` | `up -d --no-deps cloudflared` | seconds of tunnel outage | second tunnel + DNS cutover; rarely worth it |
+| `RESEND_API_KEY` | `app` | `up -d --no-deps app` | verification emails fail | create new, deploy, delete old |
+| `OAUTH_DYNAMIC_REGISTRATION_TOKEN` | `app`, DCR clients | `up -d --no-deps app` | DCR calls 401 until clients update | none |
+
+`BEARER_ADMIN_TELEGRAM_ID` and `ALERT_TELEGRAM_CHAT_ID` are identifiers, not
+secrets, but changing them takes the same restart sets as `TELEGRAM_BOT_TOKEN`.
+
+### Procedures
+
+#### POSTGRES_PASSWORD
+1. Put the new value in the env file (`POSTGRES_PASSWORD` and the password
+   inside `DATABASE_URL` if it is spelled out there).
+2. Change it inside Postgres over the unix socket, which does not need the old
+   password:
+   `docker compose exec -T db psql -U auth -d auth -c "alter role auth password '<new>'"`
+3. Immediately: `docker compose up -d --no-deps app worker`. Between steps 2
+   and 3 every new connection fails, so keep them back to back.
+4. The `db` container's environment is now stale; the next full
+   `docker compose up -d` recreates it (~5 s, data stays in the volume). Do
+   that deliberately at a quiet moment rather than as a surprise.
+
+#### OIDC signing key (no downtime)
+Verification accepts every key that is not `revoked`, and the JWKS publishes
+the same set, so a rotation is: add the new key as active, keep the old one as
+retired until every token it signed has expired, then revoke it.
+1. Set `OIDC_SIGNING_KEYS_JSON` to
+   `[{"kid":"2026-09","privateKeyPem":"<new pem>","status":"active"},{"kid":"<old kid>","privateKeyPem":"<old pem>","status":"retired"}]`.
+   Once this variable is set, `OIDC_PRIVATE_KEY_PEM` is ignored.
+2. `docker compose up -d --no-deps app`, then confirm both kids appear in
+   `https://auth.bneck.com/oauth/jwks`.
+3. Wait at least 24 h (access tokens live 15 min, but relying parties cache
+   the JWKS), flip the old entry to `"revoked"`, redeploy, and drop it from
+   the list on a later change.
+
+#### TELEGRAM_BOT_TOKEN
+1. Put the new token in the env file first.
+2. `/revoke` the old one in BotFather; the old token dies instantly.
+3. Within a minute: `docker compose up -d --no-deps app worker bot`. 2FA
+   prompts sent in the gap fail (the user retries); queued notifications retry
+   under bullmq. The bot re-pins a fresh status message on start.
+
+#### TELEGRAM_BOT_WEBHOOK_SECRET
+Edit the env file and `docker compose up -d --no-deps app bot` in one go; the
+two sides must agree.
+
+#### CLOUDFLARED_TOKEN
+Refresh the token in Zero Trust, edit the env file, `docker compose up -d
+--no-deps cloudflared`. Expect a few seconds of tunnel downtime and one false
+alarm from any external probe.
+
+### Suspected leak: triage order
+1. OIDC signing key (token forgery), then `TELEGRAM_BOT_TOKEN` and
+   `TELEGRAM_BOT_WEBHOOK_SECRET` (2FA approval forgery). Rotate these first.
+2. `OAUTH_CSRF_SECRET`.
+3. `POSTGRES_PASSWORD`: the database is not reachable from outside the compose
+   network, so real exposure needs host access, which is a bigger incident.
+4. Everything else in the matrix, then review `security_events` for activity
+   during the exposure window (Audit trail below).
 
 ## Deploys and restarts