Extension A completion — secondary-limit escalation and refresh-pool fairness

[batbrainy]

Closes #9

Why this PR exists

Issue #9 tracks Extension A — Rate Limiting and Fan-Out Control (IMPLEMENTATION_PLAN.md §4). Its three sub-issues — #14 (PR 4), #16 (PR 6), #19 (PR 9) — are all merged, and #17 (PR 7) delivered the fairness and effective_enrichment_time halves. Auditing all ten child items against the merged code, rather than against the issue checklists, found eight complete and two incomplete. Both were invisible to their existing tests because each test set up a single-class world.

Defect 1 — item 8's exponential backoff did not escalate

§10: "set global_blocked_until from Retry-After (or ≥ 1 minute with exponential backoff when the header is absent)".

RateLimitPolicy#fallback_instant called @backoff.retry_at(1, now:) with a literal 1. PollBackoff#delay_for computes exponent = [attempt, 1].max - 1, so attempt 1 fixed exponent = 0 and base = 60 on every call. The ≥ 1 minute floor and the jitter were both real; the exponential was not. Nothing compensated — Ingestion::PollState routes a :secondary_limited outcome to secondary_retry, which deliberately does not increment consecutive_failures, so the source-scoped ladder never advanced either. An IP GitHub was actively throttling was re-probed every ~60–75s for as long as the throttling lasted.

Fix. A consecutive_secondary_limits counter on the budget singleton, incremented as a bind on BLOCK_SQL — under the same SELECT … FOR UPDATE that writes the block it feeds. #secondary_limit reads it and passes streak + 1, producing 60 → 120 → 240 → … capped at one hour by the existing ceilings.

Four boundaries keep it minimal:

• Only the header-absent path escalates; a server-supplied Retry-After is still obeyed within honoured's clamp, which is how §10 words the alternative.
• Only :secondary_rate_limit advances it. A primary exhaustion and a reserve breach are conditions of the budget window, not evidence this IP is being throttled — #primary_limit passes a literal 1 for the same reason.
• It survives window rollover: §10 calls secondary limits IP-scoped, a different scope from the window ROLL_WINDOW_SQL resets. Zeroing it at the boundary would hand a persistently throttled IP a fresh 60s block every hour.
• One clean response ends it. #apply! calls clear_secondary_limit_streak! for any request that completed, including a 304 — a request GitHub answered and charged for is the same evidence a 200 carries. A timeout or 5xx produced no verdict, so it neither escalates nor clears.

Defect 2 — item 9's refresh pool starved a class

§10:898 scopes refreshes "within each class's share".

Enrichment::Fairness#refresh_choice selected requested.find { refresh_available? } — first in EntityType.all order, always actor — with no preference for a class still inside its guarantee, unlike #pending_choice. Its borrow test also read the pending eligibility map, so the other class's stale rows were invisible.

Concretely, with guarantees at 20/20, actor_share_used: 20, repository_share_used: 0, no pending work anywhere and stale rows in both classes: actor borrowed from 20 to 40 while repository's untouched 20-request guarantee and eligible refresh work were never selected. That is the starvation the split exists to prevent (§10: "a naive repo-first policy would starve actor enrichment to zero indefinitely"), reproduced one pool down with the order reversed.

Fix. #refresh_choice now performs #pending_choice's two steps over a refresh-scoped eligibility map: prefer a class with room_within_guarantee?, then borrow only from a class with nothing to refresh. A borrowed refresh reports borrowed_refresh, mirroring borrowed_pending.

CandidateSelector#pending_available? is unchanged and stays pending-only. Its documented reason still holds — counting refreshes in the pending borrow test would let a refresh outrank a never-enriched candidate and invert §10's ladder — and that hazard cannot arise inside #refresh_choice, which is only reached when no class has a pending candidate at all.

Also included

• Retry-After HTTP-date form. RFC 9110 permits both forms and §10 lists the header among those to process. RateLimitSnapshot now normalizes both to a delta against its own observed_at; leaving the date form unread collapsed a server-supplied "wait 45 minutes" into the 60s fallback. A past date yields a non-positive delta, which #fallback_instant already treats as absent.
• Jitter is now proven to reach global_blocked_until. Every existing example injected Random with rand: 0.0, so the assertions would have passed with jitter removed from the production path.
• spec/config/github_initializer_spec.rb. Nothing referenced config/initializers/github.rb — deleting its to_prepare registration left the whole suite green, so item 2's "computed at startup" had no test behind it.

Rejected alternatives

• Multiply an in-force block instead of counting. No migration, but it can only escalate while a block is still in the future — and §10's case is the repeat limit arriving after the previous block expired, which is exactly when it does nothing.
• Forbid borrowing in the refresh pool outright, on the strictest reading of §10:898. Rejected: a class whose guarantee rounds to zero (ACTOR_ENRICHMENT_SHARE at 0.0/1.0) could then never refresh, and capacity would sit idle whenever the other class has no stale rows — while §10:812's borrowing rule is stated generally rather than scoped to one pool.
• Computing the block instant inside the ledger to remove the policy's stale read. Rejected: it collapses ADR 0004's split between deciding and recording. The staleness is accepted and documented — it can only under-escalate, and BLOCK_SQL's GREATEST means a shorter instant can never shorten a longer one.

Verification

Both fixes are regression-proven: reverting fairness.rb fails 4 of the new examples (including "chooses the class that still has room, not the first in class order"), and reverting rate_limit_policy.rb fails 4 more (including "escalates exponentially from the minimum block").

• Full suite against a freshly built image, no bind mount: 1720 examples, 0 failures
• Stress suite (CI's second step): 10 examples, 0 failures
• CI's e2e chain in fixture mode — bin/ingest, bin/enrich --limit 6, solid_queue:check — exit 0

Deferred

PR 12 scope (#22); the RateLimitPolicy/BudgetLedger disagreement over a non-core x-ratelimit-resource on a 403/429 (unreachable today — every URL this app issues is a core endpoint); and the stale-validator case where a page-one 200 carries no ETag header (no budget or correctness impact).

🤖 Generated with Claude Code