test(evals): tool-surface A/B — shipped sendMessage vs messaging-primitives post - #791

Open

Poytr1 wants to merge 17 commits into

mainfrom

claude/primitives-ab-test

Commits on Aug 9, 2026

feat(evals): add the `post` façade and measure the static cost of both tool surfaces

Poytr1
and
claude
committed
8783bbb
feat(evals): add the A/B scenario matrix and its credential-free metric extraction

Poytr1
and
claude
committed
b0565aa
feat(evals): arm-parity guidance seam — each A/B arm's prompt teaches its own surface

Poytr1
and
claude
committed
1e9b930
feat(evals): A/B fixture — one environment, two surfaces, contract-proven arm parity

Poytr1
and
claude
committed
9bea1b3
feat(evals): the behavioral A/B driver — 4 scenarios x 2 arms x N trials, daemon-judged

Poytr1
and
claude
committed
aef812c
docs(design): the tool-surface A/B write-up — method, fidelity, measured static costs

Poytr1
and
claude
committed
62611a5
fix(evals): grant §6 evaluation-registry tools the system-tool permission auto-allow

Poytr1
and
claude
committed
5cde23e
fix(evals): any failed or timed-out turn invalidates an A/B trial

Poytr1
and
claude
committed
d0d2983
docs(design): record the 24-run behavioral results — identical success, 28-46% fewer tokens under post

Poytr1
and
claude
committed
49e4719

Commits on Aug 10, 2026

Commits on Aug 11, 2026

Merge remote-tracking branch 'origin/main' into claude/primitives-ab-test
Poytr1
committed
ef5f229
test(evals): add scenario 5 — in-thread turn-taking, the #801 regression gate

Poytr1
and
claude
committed
36332e0
docs(design): record scenario 5, its 6/6 baseline, and the prompt-change gate

Poytr1
and
claude
committed
db42907
fix(evals): close two false-pass paths in the in-thread judge (review)

Poytr1
and
claude
committed
cb89aae
fix(evals): match invocation-id-suffixed messaging FQNs in the in-thread judge (review)

Poytr1
and
claude
committed
87dd69c