Make queue refills single-flight on the caller's connection

[wasabipesto-bot]

Fixes the remaining pool-exhaustion 503s (issue #98 §5 plus a mechanism the issue didn't list). Before building this I re-verified against production that it still targets the live problem — all numbers below are from today.

The problem, measured live

With #100–#102 deployed, /claim/detailed works and the Next scans are gone, but the API still fast-fails ~20 requests/minute (up from ~10 two days ago as the fleet grew): 126 × 503 in a 6.4-minute window, every one at exactly elapsed_ms=5000 — the pool-checkout timeout. They hit /submit and both claim endpoints indiscriminately, which is the signature of the pool being the bottleneck, not any endpoint.

Sampling pg_stat_activity at 0.6 s intervals shows what holds the pool: sustained 10–20 s stretches of 5–6 concurrent eligible_chunk queries (the Thin bulk-refill/fallback shape), recurring about once a minute. Two mechanisms produce that from field_queue.rs:

1. The refill herd (issue The api times out for most "detailed" claim requests #98 §5). Every request observing a low queue launches its own bulk claim. The niceonly queue reads 953 right now against the 250 ceiling one refill can reach — ~4 concurrent refills landing blind, and it scales with client prefetch concurrency.
2. Each refill checks out a second connection while its request holds the first. A refilling request occupies 2 of the 10 connections for the duration of its bulk claim.

At the current drain rates (niceonly ~52/s, detailed-thin ~10/s) the queues cross their refill thresholds every 3–10 seconds, so these two mechanisms fire constantly.

The fix

• Per-queue refill gate (std::sync::Mutex<()> + try_lock). The winner refills; contenders skip straight to popping rather than waiting. The winner re-checks the queue depth under the gate, so a thread that observed a low queue but won the gate after someone else already refilled doesn't refill again (this also makes the tests deterministic). Poisoned gate = contended: skip.
• Refills borrow the caller's connection. claim_niceonly/claim_detailed_thin now take &mut PgPooledConnection; both call sites already hold one. The startup prefills keep their own checkout — they run before any traffic exists to contend with.

No schema changes, no client-visible changes, same bind patterns.

Why skipping the gate (not waiting) is safe now

Issue #98 evaluated exactly this try_lock shape and warned against it: at the time, an empty detailed queue fell through to an unindexed table walk, so 1.2%-of-samples-empty was expensive. That objection is obsolete — since #100/#102 every empty-queue fallback is chunk-scoped or partial-indexed and costs tens of milliseconds. A request that loses the gate mid-refill gets a fast direct claim, not a scan. Refill capacity (200 fields/~50 ms niceonly, 100/~65 ms detailed) exceeds observed drain by two orders of magnitude, so empty windows stay rare and brief.

Tests, and their red/green evidence

DB-gated unit tests in the api crate (same NICE_TEST_DATABASE_URL convention as common/tests/claim_queries.rs; diesel added as a dev-dependency for fixtures). The core test: 8 barrier-synced threads hit an empty queue with the pool sized exactly to their held connections (held across the whole window via a second barrier, like real in-flight requests). Exactly one bulk claim may run, asserted both in the database (rows stamped) and against the queue-depth ceiling.

Both pre-fix behaviors fail it, differently:

• Gate removed → herd → 8 refills stamp up to 1,600 rows (left != 200).
• Refill via a second pool checkout → starves against the exhausted pool → 0 rows stamped (left: 0), after the checkout timeout.

Plus a sequential drain test (252 pops → exactly 2 bulk claims, claim order preserved) and the detailed queue's own gate. The claim_queries suite passes unchanged.

What this should do in production

The once-a-minute saturation stretches were: refill herd (4–6 queries) + doubled checkouts + normal traffic > 10 connections. After this, the same moment costs one refill on an already-held connection. I'd expect the elapsed_ms=5000 503s to drop to zero at current traffic; /status queue sizes should stop overshooting 250/150 (any reading above threshold + refill amount after deploy would mean the herd is back).

Not addressed here, deliberately: the Random strategy's slow tail (pivots landing in the oversized-range_size cl=0 region scan up to 36 s) — separate PR, as discussed.

Deploy: build + up, nothing else.

🤖 Generated with Claude Code