Identity probes for direct API tokens are rate-limited at scale (~24k/h) and amplify themselves by retrying 429s

[mattzcarey]

Rewritten after looking at production (the original plan here, shortening the 30-day identity cache, would make this worse).

What production shows (cloudflare-api-mcp, last hour, Workers Observability)

	per hour
POST /mcp requests	~1.24M
api_token_identity_probe log lines	~23.6k (0 from the OAuth callback probe)
…of which 429 retries (fetchWithRetry: 429)	~18.5k
probes that failed after all retries	~5.3k
MCP requests answered 429 temporarily_unavailable	~4.1k

All-time on the Issues tab: 4.79M fetchWithRetry failed … /accounts … 429 and 1.59M … /user … 429. Both the account-token path (per_page=5, ~6.4k/h) and the user/legacy path (/user + per_page=31, ~8.9k/h each) are being rate-limited.

Why it amplifies itself

• Retrying 429s. Every identity probe goes through fetchWithRetry (up to 4 attempts with about 1 s backoff), and user tokens probe two endpoints in parallel. One rate-limited request becomes up to 8 more requests against the same quota, spending the user's own Cloudflare API budget, which their real tool calls then also run out of.
• Failures aren't cached. Only successful identities are cached (30 days), so every request from a rate-limited token probes again.
• Concurrent misses aren't coalesced. A new token's first burst of requests all miss the cache together.

Fix

1. Don't retry 429s in identity probes. Surface temporarily_unavailable with Cloudflare's Retry-After straight away.
2. Cache a 429 briefly (for the Retry-After window) so repeat requests from that token back off instead of re-probing.
3. Keep the 30-day success cache. Revoked-token staleness is a real trade-off, but reducing it by probing more often is the wrong lever while probes are the top error. Revisit once probe volume is under control.

Related: about 24k/h of the probes are for tokens in workers-oauth-provider's own format (expired MCP access tokens), which the library hands to resolveExternalToken. That's being fixed in the library (cloudflare/workers-oauth-provider, linked from here).

[mattzcarey]

Decision: keep retrying 429s. #230 is closed, and main already retries identity probes through fetchWithRetry: 3 retries at about 1, 2 and 4 s with jitter, plus Retry-After up to 30 s.

The last hour of production (cloudflare-api-mcp, Workers Observability) backs that choice, with caveats:

Identity probe	First 429	Still 429 after 4 attempts	Rescued by retrying
Account token, /accounts?per_page=5	2603	2353	~10%
User token, /user	2855	2316	~19%

• Cloudflare sends no Retry-After on these, so every delay is the backoff, 0.5–4 s. A rate-limited request fails after about 4–7 s, or succeeds after one of the retries.
• Rate limited, try again later reached clients 4712 times in the hour.
• Tool calls saw no 429s at all (caller=execute: 0). Only identity probes get rate-limited.

Options for what's left, none of which fail faster:

1. Upgrade to the next workers-oauth-provider release. fix: never hand tokens in our own format to resolveExternalToken workers-oauth-provider#342 stops expired MCP access tokens from reaching resolveExternalToken; those tokens were about 24k probes an hour, answered Access token appears malformed.
2. Coalesce concurrent probes for one token within an isolate. A new token's first burst shares one probe instead of each request spending up to 8 API calls.
3. Look at why a rate-limited token stays rate-limited. A failed probe isn't cached, so every request from that token probes again. That spends the same quota the retries are waiting on.

(1) happens with the dependency bump. (2) and (3) are for discussion.