[PILOT-05] Generic agent checkers through a budgeted model proxy #492

On hold by the product owner (2026-10-08). No agent may start or continue implementation of this issue or any of its slices. Leave it open. The owner will return to PILOT-05, PILOT-06 and PILOT-09 after the other issues are solved; resume only on the owner's explicit instruction. Existing scope and acceptance criteria below are retained for later.

Dependency refresh (2026-10-13)

The product-owner hold above remains in force. This edit records merged dependencies only and does not authorize implementation.

PILOT-04 (#491) is closed after PR #538 registered the selected production checker capabilities and removed the superseded catalogue path. PILOT-01 (#488) and the bounded gVisor feasibility work in PILOT-00 (#500) are also closed. Those merged foundations do not deliver the budgeted model proxy, generic agent checker, project budget ledger, or protected transcript surface owned here.

The product owner has since scoped #493 to deterministic checks for the real Terminal-Bench pilot in #555. That bounded work does not resume this generic agent-checker issue, and it does not depend on this proxy. The expanded evaluator CI project #496 also remains held. Resume this issue only on the product owner’s explicit instruction.

Outcome and agreed decisions

Depends on PILOT-04 (#491). One digest-pinned generic Claude Agent SDK runner evaluates each submitted task against the locked guide/config. PM-approved configuration defines instructions, model/version, allowed tools, maximum turns, output schema and thresholds; changing it creates a new policy version. Workstream pays model costs, never contributors/project owners, with project budgets agreed before admission.

The agent reports findings/severity/scores. Workstream applies locked thresholds mechanically to normalized checker evidence; PILOT-07 (#494)/#495 own lifecycle outcomes. A valid JSON response alone is not proof that the work passes.

Implementation scope

• FDE/engineer enters agent configuration through the existing policy draft/correction path; PM approves it in a new policy generation. No separate config write route or mutable runner configuration outside policy custody.
• Setup-agent spend is an explicit scope decision before integrating guide-compilation model calls. Recommendation: use the same proxy and ledger, charged to the project when admitted or to an authorized platform budget beforehand, so all Workstream-paid agent spend is visible. Record the decision in the bounded contract/ADR; do not silently add a second meter or assume unlimited pre-admission spend. If adopted, bind setup calls to existing setup/compilation attempt identity and authority, not a fabricated CheckerRun; define platform-budget provisioning, reservation/replay and approved cost reads before exposure.
• Runner mounts only the verified submission read-only in the external sandbox. Use supported SDK language behind PILOT-04 ([PILOT-04] Register and execute external sandboxed checkers #491) protocol; Rust remains the deterministic checker/SDK choice.
• Proxy outside the sandbox injects provider credentials and grants run/attempt/model/config-scoped, expiring authorization. Only the proxy can be reached; implement/test actual network restrictions, not merely a gVisor setting.
• CHECKERS owns a bounded project-budget configuration/ledger used by the proxy, with versioned cap, currency/unit, effective status and audited adjustments. Before the first runnable project, an explicitly authorized Workstream operator provisions it using project-scoped budget authority through the existing AUTH/grant mechanism; local setup must use the same path. Record exact action/principal/grant and idempotent update contracts in the bounded ADR/PR. Absence of a usable cap blocks new agent runs; PM approval does not itself grant budget-write authority. No contributor/project-owner billing or separate general-purpose finance system.
• Record provider/requested model identity, tokens and exact costs per attempt/project. Atomically reserve/enforce budgets for concurrent runs; account for calls that fail or outlive worker crashes and avoid duplicate accounting on completion replay. Define retry and reservation release behavior.
• Retain original transcripts/results as protected immutable evidence; expose authorized, bounded/redacted views that explain findings without leaking sensitive system/tool/provider material. Non-determinism is recorded, not hidden behind exact replay promises.
• Project-cost reads are covered-PM/project-scoped. Before exposing budget administration/global reads, document and implement their explicit AUTH principal/action/grant contract; do not infer authority from a role name or hard-code an administrator bypass.

Required infrastructure-failure contract (2026-10-15)

Budget exhaustion or a provider outage must produce an infrastructure failure, never a fixable contributor-work failure. Retain the same submitted attempt and use durable bounded retry/recovery; budget restoration or provider recovery must not require a contributor resubmission. Do not consume a revision, start a revision window, emit contributor exhaustion or change a contributor outcome. Exhausted retries remain an observable operational condition; do not turn them into a work failure or retry indefinitely. Preserve unknown-effect reconciliation and exact replay rather than blindly repeating a possible effect.

• Budget exhaustion and provider-down regressions exercise normalized results through routing: no fixable-failure outcome, revision preparation/deadline or round charge; successful authorized recovery remains attached to the original attempt without duplicate effects.

This text update does not lift the product-owner implementation hold at the top of this issue.

Leave alone

No model key in sandbox/env/files, host Docker socket, ambient filesystem/other-project tools, agent-owned acceptance/rebase or contributor failure for budget/provider exhaustion. No unrestricted raw-transcript export or promise that schema validation eliminates prompt injection.

Acceptance and how to check

• Same runner image with two approved configs produces config-specific findings; tasks retain old config until an authorized rebase.
• Low score emits normalized fixable failure consumed by routing/remediation; off-schema output is infrastructure failure.
• Hostile fixture cannot recover provider key, reach other egress or override locked assessment requirements; valid controls still pass, including quoted instruction-like text.
• Concurrent runs cannot overspend a project reservation; budget exhaustion remains an inspectable recoverable project condition with no revision round spent.
• Proxy/worker crashes, duplicate calls/completion and retries have explicit tested metering/recovery semantics.
• Cost/transcript reads deny unrelated project/contributor callers and bound/redact output.

Add model-proxy and runner configuration to local Compose using PILOT-01 (#488)/#491; only proxy receives the untracked provider key.

Two-agent coordination

Agent 1 owns budget configuration/ledger migrations, provisioning/authority and backend integration. Agent 2 owns the model-proxy service and sandboxed runner using that published budget/authority contract. Service work may use a contract fixture while the backend lands; real service integration waits for the backend contract/capability. Agent 2 does not write the ledger directly or allocate migrations.

Working and verification rules

Read AGENTS.md, CONTRIBUTING.md, docs/roadmap_status.md and the affected owner contracts before coding. The baseline inspected for this plan is main c0c4fe70f77e2e84347cec979e3de00e9d7f4295; recheck current main and open PRs before selecting a bounded implementation. Open/planned work is not delivered capability.

This is a planning issue and may require several bounded PRs. Record each PR's allowed files, prohibited changes, acceptance criteria and review scope using the repository's existing Commitrail process. Extend existing owner operations/typed public ports; do not add parallel legacy/new implementations or bypass authority. Preserve retained data, locked lineage, immutable evidence and caller-owned atomicity. A planning issue does not authorize deployment or merging a PR.

Run focused positive/negative tests and applicable repository checks. Use real PostgreSQL for database/locking claims and MinIO/S3 for storage claims. Add regressions that fail on the reproduced defect; do not skip failing tests or weaken CI. Run required full CI and affected independent review tracks before requesting human approval. Update affected current specs/ADRs, README/CLI contracts and roadmap in the same PR. Report exact tested head, commands, failure/cleanup evidence and remaining limitations; passing counts alone are insufficient.