Make the agent routing gate pinned, powered, and uncontaminated

[zackbart]

Motivation

The current agent-routing release gate runs 30 sessions against an unpinned codex-default model and requires at least 29 passes. In the 0.20 paired control, untouched 0.19 main scored 20/30 and the field-removal candidate scored 25/30 under the same host settings. A system whose true pass probability is 95% has only about a 55% chance of clearing a 29/30 gate, so one run is both drift-sensitive and underpowered as a release verdict.

The dependent-read case is also systematically failing its one-round-trip policy. Historical evidence shows correct answers often use a second execute_code call, while the served usage skill mirrors that benchmark fixture's tool names and join key. The lane must distinguish a real product contract from harness contamination before its aggregate can govern unrelated subtraction work.

Behavioral requirements

• Every comparable agent artifact names an exact model identifier rather than an unresolved default alias.
• A release comparison uses matched model, host version, repetitions, concurrency, prompts, fixtures, and scorer, and records both candidate and untouched-control commits.
• The route gate has enough statistical power that an implementation meeting its stated target reliably passes, while a materially worse implementation reliably fails.
• The dependent-read route policy is explicitly classified as either a product contract or a benchmark preference. If it is a product contract, the ordinary agent path must meet it without fixture-specific coaching.
• Served examples do not reuse the benchmark fixture's tool names, field names, or join keys.
• Historical artifacts remain readable and are not rewritten to imply they used a pinned model.

Acceptance criteria

• Two repeated matched-control runs produce the same release verdict under the selected gate.
• The gate's documented operating characteristics give at least a 90% pass probability at the target rate and a clearly lower probability at the rejected rate.
• The dependent-read case either meets its declared product route or is scored according to the documented non-product policy.
• The current-version README explains when an absolute target and when a paired non-regression comparison is authoritative.
• npm run check passes.

[zackbart]

Pinned release-candidate evidence on 2026-08-26 confirms why this issue remains necessary.

Matched settings: Codex CLI 0.149.1, gpt-5.6-sol, o200k tokenizer, five repetitions per case, concurrency five.

Case	v0.19.0	v0.20.0 candidate
single read	5/5	4/5
dependent read	1/5	1/5
dependent reduction	5/5	4/5
multi-operation discovery	4/5	4/5
ambiguous candidate	1/5	5/5
nonstandard collection root	5/5	5/5
total	21/30	23/30

Both were 30/30 advertised-surface-valid and foreign-clean. The control was 30/30 correct and safe; the candidate was 29/30. In the candidate failure, the model selected builds.rerun_failed_jobs from a safety: "readOnly" search because the eval fixture currently annotates that semantically mutating tool as read-only. It supplied an invalid numeric argument, so schema validation rejected the call before its handler executed. The model later completed the intended reads.

This exposes two separate harness problems:

1. The safety scorer treats the fixture call as unsafe while the product-facing fixture annotation tells Connecta to admit and return it from read-only search.
2. The dependent-read lane remains systematically poor in both arms and continues to be dominated by extra programs and fixture-specific selection.

Raw artifacts:

• /tmp/connecta-0.19.0-pinned-routing.json
• /tmp/connecta-0.20.0-pinned-routing.json

They are local run artifacts, not committed product files. Release PR #497 reports the failure rather than presenting either arm as an absolute pass.

[zackbart]

Closing this as superseded by #507. That PR removes the contaminated 30-session comparison and its historical snapshots. The replacement is a four-case diagnostic benchmark, pins gpt-5.6-sol, records exact whole-agent and MCP-result costs, and does not claim to be a statistical release gate. A future matched-control release gate should start from a new issue with a current decision and power target.