AgentDojo: real benchmark run + paired scorecards (commit 0acc6db unblocks)

[kenithphilip]

Tracking issue: real-numbers paired scorecards

The Wave 4E paired scorecards (docs/scorecard/static/paired-{claude-sonnet-4.5,gpt-5,gemini-2.5-pro}.intoto.jsonl) currently ship as signed envelopes with "benchmarks": {} placeholders.

Commit 0acc6db wires python -m benchmarks.agentdojo_live.submit to dispatch real Anthropic trials end-to-end and write an emitter-friendly summary JSON; the scorecard emitter's --benchmark-run already accepts that JSON. So the path from "API key" to "signed scorecard with real numbers" is now one command pipeline.

Steps to populate the claude-sonnet-4.5 scorecard

export ANTHROPIC_API_KEY=sk-ant-...
pip install -e '.[dev,agentdojo]'

# 1. Run the matrix (4 suites x 2 attacks x 3 seeds = 24 cells; ~30-60 min)
python -m benchmarks.agentdojo_live.submit \
    --models claude-sonnet-4-5 \
    --suites banking slack travel workspace \
    --attacks important_instructions tool_knowledge \
    --seeds 0 1 2 \
    --max-pairs 2 \
    --out runs/agentdojo-claude-sonnet-4-5-v1.0.jsonl

# 2. Re-emit the paired scorecard with real numbers
tessera bench emit-scorecard \
    --version v1.0.2 \
    --paired-model claude-sonnet-4.5 \
    --benchmark-run runs/agentdojo-claude-sonnet-4-5-v1.0.summary.json \
    --out docs/scorecard/static/paired-claude-sonnet-4.5.intoto.jsonl \
    --sign sigstore

Repeat for claude-haiku-4-5. For gpt-5 and gemini-2.5-pro the runner is "not yet implemented" and falls outside the unblocked scope here. See docs/benchmarks/REAL_RUN_RUNBOOK.md.

Cost estimate

Roughly $5-$30 per full 4-suite Claude-Sonnet run depending on retry policy.

Pending

• Provide ANTHROPIC_API_KEY (operator only)
• Run the pipeline above for claude-sonnet-4-5
• Commit the resulting paired-claude-sonnet-4.5.intoto.jsonl over the placeholder
• Repeat for claude-haiku-4-5
• Wire the OpenAI runner (separate work, ~half-day) for gpt-5
• Wire the Gemini runner (separate work, ~half-day) for gemini-2.5-pro