RetryLedger

Benchmark for exactly-once compliance of tool-using LLM agents under ambiguous tool failures.

When an agent charges a card and the gateway returns an ambiguous error, the charge may have committed (lost acknowledgement) or not (true transient failure) -- the error text looks the same. A blind retry double-charges; giving up may silently drop the payment. RetryLedger measures which failure actually happens, with a deterministic oracle: a mock backend counts committed side effects per idempotency key (1 = correct, >=2 = duplicate, 0 = missed).

Headline (small tier, 24 seeds/cell, 3 vendors x 2 tiers, 576 episodes):

• Failures are almost entirely missed effects, not duplicates (duplicates average 1%, peak 8%).
• Compliance is gated by state verification: agents that check status before deciding are correct in 97% of true-failure episodes; agents that do not are correct in 5%.
• The small Gemini tier checks status only 21% of the time and misses 96% of charges on true failures; the mid tier verifies and recovers (0.96 correct).

Layout

• code/ei_retryledger.py -- episode generator + deterministic oracle
• code/ei_common.py, code/direct_backend.py -- multi-turn agent loop, multi-vendor API backend
• code/ei_fullrun.py -- matrix runner (incremental JSONL, resumable)
• code/finalize_rl.py, code/ei_stats.py -- Wilson-CI stats, results table
• code/make_rl_figure.py -- paper figure
• data/RetryLedger_full.jsonl -- all 576 episode records (per-episode oracle fields included)
• figures/ -- paper figures

Reproduce

pip install -r requirements.txt
export ANTHROPIC_API_KEY=... OPENAI_API_KEY=... GEMINI_API_KEY=...
cd code
python ei_fullrun.py RetryLedger --seeds 24 --tiers small,mid --temp 0.7
python finalize_rl.py && python make_rl_figure.py

Scoring is fully programmatic (per-key effect count); no LLM judge. See REPRODUCE.md for details.

Citation

Paper under review ("Exactly Once, or Not at All: Idempotency Compliance of Tool-Using Language-Model Agents under Ambiguous Failures"). Citation will be added on acceptance.