This repository was archived by the owner on May 14, 2026. It is now read-only.
514-1516: rotate API keys across parallel runs and warn on undersized pools#91
Merged
Conversation
… pools Adds KeyPools, a per-provider round-robin pool of API keys, so a matrix run with --parallel > 1 no longer hammers a single key. Users can supply multiple keys via the new plural env vars (ANTHROPIC_API_KEYS, OPENAI_API_KEYS, CURSOR_API_KEYS, comma-separated); when only the singular *_API_KEY is set and parallelism > 1, the CLI prints a warning to stderr and proceeds. Each container gets exactly one provider key (the one assigned by the round-robin cursor), set on the singular env var the agent reads — the plural list is never forwarded into containers. Preflight validates every key in every relevant pool against the provider API in parallel before any Docker work. Out of scope and tracked for AXP: 429 retry/backoff in agent harnesses, cross-process coordination for the "two terminals, one key" case, and token-bucket rate limiting. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Preview CLI build for DEC_BENCH_INSTALL_VERSION=preview-pr-91-59b1c9a curl -fsSL https://raw.githubusercontent.com/514-labs/agent-evals/59b1c9ac4f2e7ea38e0fd2f604635f66d3792c05/install.sh | shRelease: https://github.com/514-labs/agent-evals/releases/tag/preview-pr-91-59b1c9a Version policy:
|
Anthropic rate limits are per-organization, not per-key (per the docs at docs.anthropic.com/en/api/rate-limits). Two keys from the same org share one pool, so multi-key rotation only adds capacity when the keys come from different organizations. Update the warning text and README/AGENTS/skill docs to say so plainly, including the same-org alternatives (raise tier, lower --parallel). The plumbing in the previous commit is unchanged — it remains correct for the cross-org case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
KeyPools— a per-provider round-robin pool of API keys — so a matrix run with--parallel > 1does not all hit a single key.ANTHROPIC_API_KEYS,OPENAI_API_KEYS,CURSOR_API_KEYS(comma-separated).Closes Linear 514-1516.
Honest scope
Anthropic rate limits apply at the organization level, not per key. Two keys from the same Anthropic org share one rate-limit pool, so multi-key rotation only adds real capacity when the keys come from different organizations. The warning text and README / AGENTS / skill docs say this explicitly so users do not get a false sense of safety from setting
ANTHROPIC_API_KEYS=k1,k2with two keys from the same account.For more capacity within a single organization (the common case), the actual fixes are:
--parallelThose last two are AXP follow-ups, tracked on the Linear ticket.
What this PR is good for
Test plan
cargo test— 72 unit + 11 e2e + 2 integration tests passcargo clippy --all-targetscleandec-bench run --matrix --parallel 2withANTHROPIC_API_KEYS=a,bset — confirm each container env carries one key and they alternate--parallel 2with only singularANTHROPIC_API_KEYset — see warning, run still proceedsRelease
This changes CLI runtime behavior. Per
.claude/CLAUDE.md, after merge bumpapps/cli/Cargo.tomland cut a release so the install script picks up the new binary.🤖 Generated with Claude Code