POST /v1/tokens intermittently 500s when D1 is at its size ceiling

[willwashburn]

POST /v1/tokens on api.relayauth.dev intermittently returns the global unhandled-error envelope:

{"error":"Internal Server Error","code":"internal_error","requestId":"..."}

That envelope is produced by the app.onError fallback in packages/server/src/server.ts — the route threw and nothing classified the failure.

Root cause

The backing D1 database is at its hard size ceiling and rejecting writes.

issueTokenPair persists an access row and a refresh row per mint (packages/server/src/routes/tokens.ts:882-883). When the database cannot allocate, those INSERTs throw D1_ERROR: Exceeded maximum DB size, the throw reaches the global handler, and the caller gets an opaque 500. Reads are unaffected by a full database, so authentication and identity lookup succeed and only the write fails.

This is a recurrence of the 2026-07-22 incident recorded in cloud/docs/runbooks/relayauth-d1-retention-gc.md, where the database reached ~9,827 MiB of the 10,240 MiB limit and "token minting then failed with D1_ERROR: Exceeded maximum DB size". The same runbook states that 90-day retention projects 28.7–72.3M rows and "does not fit safely in 10 GiB long-term".

Why it flaps

Failures arrive in windows rather than continuously — on 2026-07-29, 10/10 mints failed at 18:47Z and 10/10 succeeded at 18:52Z, alternating over ~35 minutes.

The retention GC runs on a */5 * * * * cron (cloud/infra/relayauth.ts:164) and deletes expired token rows, returning SQLite pages to the freelist. Mints succeed until the reclaimed pages are consumed, then fail until the next sweep. The flap period is the cron period.

Reproduction

Reproduced locally against the real Hono app and the real GC code, with PRAGMA max_page_count standing in for the 10GB ceiling:

[1] baseline mint         -> 201
[3] database full after 12 filler rows
    raw error: database or disk is full
[4] mint on full database -> 500 {"error":"Internal Server Error","code":"internal_error","requestId":"..."}
[5] retention GC deleted 412 expired rows
[6] mint after GC tick    -> 201

Step 4 reproduces the production envelope exactly. Steps 5-6 reproduce the recovery half of the flap.

Downstream impact

cloud's workspace provisioning mints a token during create. The 500 aborts workspace creation with Failed to create workspace, which blocked relayfile provisioning.

Observed failing requests

2026-07-29, ~18:43–18:48 UTC:

288c6dea-fa13-421e-9f4e-f6584ce32b9f
cb6beea3-5828-4b42-8937-a438670c5256
085d3253-2573-4dc8-bc29-39ce09730d13
0380c4e6-0def-4519-8b18-ce9df0438a1e
3441d318-349f-4e37-b013-56326e3ffd78

On the retention-GC lead

The GC was suspected of causing the throw. It does not:

• Migration 0004_retention_gc_indexes.sql is CREATE INDEX only, and is deliberately never applied in production — cloud/packages/relayauth/scripts/defer-retention-index-migration.ts filters it out by exact SHA-256 and fails closed (cloud#2801).
• The GC's mutation is DELETE FROM tokens WHERE id IN (SELECT value FROM json_each(?)) — one bound parameter, bounded to at most 1000 scanned rows, and it removes only already-expired rows. It takes no lock and deletes nothing issueTokenPair reads.
• relayauth does not run the GC; it is scheduled from cloud's worker.

The GC is the mitigation whose cadence produces the windows, not the cause.

Scope

Two separable problems:

1. The opaque envelope — a classifiable, retryable infrastructure condition is indistinguishable from a handler defect, so callers cannot back off and operators cannot see the cause. Addressed in fix(server): typed envelope when storage cannot allocate #70.
2. The capacity itself — the GC is not keeping up with ingest, so the July 22 condition is ongoing. Owned by cloud#2801 and the archive-to-R2 workstream; not addressed by fix(server): typed envelope when storage cannot allocate #70.

Unrelated observation

The SQL comment at packages/server/src/engine/retention-gc.ts:145-146 describes using the created_at and (org_id, created_at) indexes. Those are the indexes in the deferred 0004 migration and do not exist in production. The rowid-window design does not need them, so this is stale documentation rather than a correctness bug.