feature: provider quota retries and resumable reviews across gateway shutdowns

[mountainowl]

Use case

Bubo may be running a long or high-volume review when either:

1. the configured model provider throttles the request (HTTP 429, quota/rate-limit response, or an equivalent provider-specific error), or
2. Bubo's gateway/process receives a shutdown for an update, restart, Spot interruption, or operator action.

Today the reviewer is executed as an opaque subprocess. A provider throttle can therefore become a generic failed/empty review, and cooperative shutdown only prevents starting the next review; it does not expose a durable checkpoint/resume contract for the in-flight task. Operators and users cannot reliably distinguish these recoverable conditions from review defects.

This must be provider-agnostic. Codex exposed the gap, but the behavior must apply to every configured model provider.

Proposed behavior

1. Classify provider throttling and retry safely

• Normalize provider-specific throttle/quota signals into a typed provider_rate_limited condition while preserving the original redacted provider code/status for diagnostics.
• Show the user: “The model provider is rate-limiting requests. Please wait a moment and try again.”
• Retry with bounded exponential backoff plus jitter, honor Retry-After when supplied, and make the retry budget/operator limits configurable.
• Do not record a throttle as a successful empty review, no_usable_report, or an undifferentiated API/subprocess failure.
• When retries are exhausted, persist an explicit recoverable/deferred terminal state with the next eligible retry time. Never create duplicate SCM comments when the task is retried.
• Apply the classifier at the model-provider/reviewer boundary, not in the existing SCM HTTP retry helper.

2. Checkpoint and resume interrupted reviews

• On gateway/process shutdown, stop accepting new reviews and show/log: “Gateway shutting down — Your current task will be interrupted.”
• Persist the in-flight task identity, review phase, completed durable artifacts, retry state, and idempotency keys before exit.
• After restart, automatically resume from the latest safe checkpoint. If a phase cannot be resumed internally, restart that phase idempotently rather than marking the whole review complete or losing it.
• Preserve the existing graceful-drain behavior for work that can finish within the shutdown grace period; checkpoint work that cannot.
• Make recovery safe across application updates, process restarts, machine replacement, and Spot interruption.
• Prevent duplicate findings/comments and make resume outcome visible to the operator and user.

3. Telemetry and audit trail

Emit low-cardinality OpenTelemetry metrics/events/spans and durable structured events for both paths:

• provider_throttle_detected — provider, model, run ID, normalized reason, provider status/code
• provider_retry_scheduled — attempt, backoff seconds, source (retry_after or policy)
• provider_retry_exhausted — attempts and elapsed retry time
• review_interrupted — run ID, phase, shutdown reason, interruption timestamp
• review_checkpoint_persisted — checkpoint phase/version and success/failure
• review_resumed — run ID, checkpoint phase, restart timestamp
• review_resume_completed / review_resume_failed — outcome and interruption duration

Provide counters for throttles, retry attempts/exhaustion, interruptions, checkpoint success/failure, resume attempts/success/failure; histograms for backoff and interruption duration; and trace events on the review span. Do not put prompts, source code, credentials, raw provider bodies, or unbounded error text in metric attributes.

Acceptance criteria

• Adapters for at least two differently shaped provider throttle responses prove the same normalized behavior; unknown providers can opt in without poller changes.
• Unit tests cover 429, provider-specific quota text/code, Retry-After, jittered backoff, retry exhaustion, and non-retryable authentication/billing errors.
• A throttled review is never reported as success/no-findings/empty output.
• A restart test interrupts a review in each durable phase, restarts Bubo, and completes exactly once.
• Repeated restart/resume does not duplicate findings or SCM comments.
• Checkpoint corruption/version mismatch fails visibly and recoverably; it is not silently discarded.
• User-facing messages are emitted exactly for their corresponding conditions.
• Metrics, trace events, and durable events above are asserted in tests with secrets and high-cardinality payloads excluded.
• Operator documentation explains retry tuning, shutdown grace, checkpoint storage/retention, recovery states, and how to query the telemetry.

Current implementation boundary

• src/bubo/_http.py retries SCM REST traffic; it is not the model-provider retry layer.
• src/bubo/signals.py and the poll loop support cooperative shutdown between reviews.
• src/bubo/poller.py records the opaque reviewer subprocess result and generic failure telemetry.
• src/bubo/telemetry/metrics.py, src/bubo/events.py, and src/bubo/db.py are the likely telemetry, durable-event, and checkpoint/state integration points.

Area

Packaging and deployment