Skip to content

fix(gmail): retry 403 rateLimitExceeded and 5xx with exponential backoff - #80

Open
jqueguiner wants to merge 2 commits into
MaximeGaudin:mainfrom
jqueguiner:jl/gmail-retry-backoff
Open

fix(gmail): retry 403 rateLimitExceeded and 5xx with exponential backoff#80
jqueguiner wants to merge 2 commits into
MaximeGaudin:mainfrom
jqueguiner:jl/gmail-retry-backoff

Conversation

@jqueguiner

@jqueguiner jqueguiner commented Sep 12, 2026

Copy link
Copy Markdown

What

Retry transient Gmail API failures instead of surfacing them as fatal errors.

  • New api/retry.rs: bounded exponential backoff with full jitter, honouring
    Retry-After (both the seconds form and the HTTP-date form).
  • Retries 429, 5xx, and connect/timeout transport errors. Default is 4 attempts,
    500 ms base, 16 s cap.
  • Retries 403 when, and only when, Google's reason says the quota was hit:
    rateLimitExceeded, userRateLimitExceeded, quotaExceeded, backendError.
  • Every other 403 still fails fast, ACCESS_TOKEN_SCOPE_INSUFFICIENT and
    permission errors included. So do 401 and 404.
  • Call sites change by one token: .send() becomes .send_retrying(&self.retry).

First of the items in #79, smallest first.

Why

Gmail's quota (Total Query Cost) is counted per user per minute and is shared by
every process authenticated as that account. One client's burst rate-limits the
others.

The status that carries it is 403, not 429. Counted on a shared account running
several agent sessions, over 2026-09-06 to 2026-09-11:

Google API error count
403 rateLimitExceeded 16
404 notFound 3
429 (any) 0

So a retry predicate keyed on 429 would have caught none of them. 429 and 5xx are
still retried, since Gmail can send either, but 403 is the case that actually
happens.

403 cannot be decided by its status: it is also what Gmail returns for a missing
scope, a denied permission or a domain policy, none of which clear by waiting. The
discriminator is Google's reason field in the response body. Retrying a scope
error burns quota and hides a real auth problem, so the predicate is a positive
list of quota reasons, not "403 unless proven otherwise".

Before this change all 14 call sites in api/client.rs ended in
.error_for_status(), so Retry-After was discarded and the read was lost.

Jitter is full jitter on purpose. Without it, several clients that hit the same
limit wake up together and collide again.

One parser, in error.rs

error.rs already knew how to read a Google reason (is_insufficient_scope_body,
matching ACCESS_TOKEN_SCOPE_INSUFFICIENT). The quota check went next to it rather
than into a second parser in retry.rs, so one module knows Google's error shapes.

It reads both shapes Google uses for the same field, error.errors[].reason and
error.details[].reason, and matches reason tokens only. Loose prose is not a
reason: "You have exceeded your daily quota of patience." is asserted not
retryable. A scope error also wins over a quota match, in case a body carries both.

Reading the body costs the response

Deciding on a 403 means consuming it, and a consumed response reaching the caller
empty would be a regression: json_or_scope_error maps InsufficientScope by
parsing the body, so an emptied 403 would degrade to a generic Api error.

So the body is buffered once and the response rebuilt from its own parts, status,
headers and URL included. buffer_body_preserves_status_headers_and_body asserts
each of those, and resolve_signature_403_scope_error_keeps_its_body asserts the
end-to-end consequence: the scope error still arrives as InsufficientScope.

Two existing tests changed, deliberately

create_draft_429_preserves_status and get_thread_500_preserves_status asserted
that a 429/500 failed on the first response. That is the behaviour this PR
changes, so leaving them untouched was not an option.

They are renamed to ..._after_retries and now assert the property that still
matters: when every attempt fails, the status reaches the caller unchanged. The
mocks answer the error status every time, so the assertion is about the final
outcome, not the attempt count.

retryable_excludes_real_answers no longer asserts !is_retryable(FORBIDDEN) as a
final word. 403 moved out of that predicate and into
forbidden_is_decided_by_the_body_not_the_status, which asserts both halves: the
status alone does not retry it, and the body is what decides.

The "does not retry" side is covered explicitly with wiremock .expect(1), on both
get_message_does_not_retry_404 and get_message_does_not_retry_403_scope_error: a
stray retry makes the mock verification fail.

Verified

cargo test -p void-gmail, run on this branch, based on main at add6674:

test error::tests::google_error_reasons_reads_both_shapes ... ok
test error::tests::insufficient_scope_body_detects_google_reason ... ok
test error::tests::retryable_quota_body_detects_the_403_gmail_actually_sends ... ok
test error::tests::retryable_quota_body_rejects_real_answers ... ok
test api::retry::tests::retryable_covers_quota_and_backend_errors ... ok
test api::retry::tests::retryable_excludes_real_answers ... ok
test api::retry::tests::forbidden_is_decided_by_the_body_not_the_status ... ok
test api::retry::tests::buffer_body_preserves_status_headers_and_body ... ok
test api::retry::tests::delay_grows_and_stays_capped ... ok
test api::tests::get_message_retries_429_then_succeeds ... ok
test api::tests::get_message_honours_retry_after_header ... ok
test api::tests::get_thread_retries_500_then_succeeds ... ok
test api::tests::get_message_does_not_retry_404 ... ok
test api::tests::get_message_retries_403_rate_limit_then_succeeds ... ok
test api::tests::get_message_does_not_retry_403_scope_error ... ok
test api::tests::resolve_signature_403_scope_error_keeps_its_body ... ok
test api::tests::get_thread_403_rate_limit_preserves_status_and_body_after_retries ... ok
test api::tests::get_thread_500_preserves_status_after_retries ... ok
test api::tests::create_draft_429_preserves_status_after_retries ... ok

test result: ok. 92 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

cargo clippy -p void-gmail --all-targets is clean.

I checked the new tests bite, in both directions. Forcing
is_retryable_quota_body to return false fails 2:

test result: FAILED. 90 passed; 2 failed
    api::tests::get_message_retries_403_rate_limit_then_succeeds
    error::tests::retryable_quota_body_detects_the_403_gmail_actually_sends

Forcing it to return true fails 3, which is the half that matters more, since a
predicate that retries every 403 is worse than one that retries none:

test result: FAILED. 89 passed; 3 failed
    api::tests::get_message_does_not_retry_403_scope_error
    api::tests::resolve_signature_403_scope_error_keeps_its_body
    error::tests::retryable_quota_body_rejects_real_answers

Forcing is_retryable to return false fails 4 (the 429 and 5xx tests).

Notes

  • One new workspace dependency, http = "1". It is already in the lockfile as a
    reqwest 0.13 dependency, and reqwest re-exports its StatusCode and header
    types, so this pins nothing new. It is needed for the
    http::Response to reqwest::Response round-trip that rebuilds a buffered body.

Not in this PR

Gmail enforces a per-user quota shared by every process on the account, so a
burst from one client can rate-limit another. The API client had no retry: all
14 call sites ended in .error_for_status(), turning a transient 429 into a fatal
error.

Adds a bounded retry with full jitter, honouring Retry-After when present.
Non-retryable statuses (401, 403, 404) still fail fast, so a scope error or a
missing thread does not burn quota waiting for an answer that cannot change.

Two existing tests asserted that a 429/500 failed on the first response. They
now assert the status survives once retries are exhausted, which is the property
that actually matters.

Refs MaximeGaudin#79
Retrying only 429 missed the failure this was meant to fix. Gmail does not
answer 429 for the per-user quota: it answers 403 with
`reason: rateLimitExceeded`. Measured over 2026-09-06 to 2026-09-11 on a
shared account: 16 `403 rateLimitExceeded`, 3 `404 notFound`, 0 Google 429.

403 is now decided by Google's reason, not the status. Retried on
`rateLimitExceeded`, `userRateLimitExceeded`, `quotaExceeded`, `backendError`.
Every other 403 still fails fast, including `ACCESS_TOKEN_SCOPE_INSUFFICIENT`
and permission errors: retrying those burns quota and hides an auth problem.

The reason parser lives in `error.rs`, next to `is_insufficient_scope_body`,
so one module knows Google's error shapes. It reads both
`error.errors[].reason` and `error.details[].reason`, and matches reason
tokens only, never loose prose like "quota".

Deciding on a 403 consumes the response, so the body is buffered once and the
response rebuilt from its own parts. A non-retryable 403 reaches the caller
with its status, headers, URL and body intact, which is what lets
`json_or_scope_error` still map a scope error to `InsufficientScope`.
@jqueguiner jqueguiner changed the title fix(gmail): retry 429 and 5xx with exponential backoff and jitter fix(gmail): retry 403 rateLimitExceeded and 5xx with exponential backoff Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant