[PILOT-04] Register and execute external sandboxed checkers #491

Agreed architecture

Workstream keeps only its default checker. No generic rule engine and no built-in pre- or post-submission check catalogues remain. Every project check, pre and post, is a digest-pinned external image, including agent checks. Workstream stores results, locks policy context and governs outcomes. Adding or changing a project checker never requires a Workstream deploy; only a change to the Workstream default checker does. This is the newly agreed target, not a claim about capability already delivered on main.

The setup agent proposes missing checks. Engineering implements/registers external images; the PM approves selection/configuration in a new policy generation. Registration alone does not change existing tasks. Adding a project checker must not require redeploying Workstream once this interface exists.

Required intake flow

This is the acceptance contract for PILOT-04 and the contributor surface in PILOT-12 (#499):

1. The contributor uploads one ZIP into ART-owned private scratch. Enforce bounded upload/extraction, safe paths, no symlinks or special files, and archive-bomb limits before exposing extracted material.
2. One Workstream default checker implements exactly the four blocking behaviors listed below. Server-computed SHA-256, byte verification and the file manifest are part of custody checking and precede external execution. Missing/empty summary and attestation are warnings only.
3. Run every required project pre-submit image selected by the task's locked policy against the exact verified material, read-only and under sandbox/resource limits. Never select a latest image or substitute a mutable tag/configuration.
4. Only after the default checker and every required project pre-check pass may the existing owner operation atomically create the immutable Submission, move the task to evaluation_pending, and retain its initial durable evaluation dispatch (PILOT-03, [PILOT-03] Create Submission and initial checker dispatch atomically #490).
5. Post-submit images evaluate that immutable Submission asynchronously. They provide evidence to governed routing; they cannot accept work themselves.

A blocking default finding or required project pre-check failure returns bounded contributor findings, creates no Submission or initial evaluation dispatch, and leaves the task in_progress. A missing runner/image, crash, timeout, invalid output or capacity/provider failure is a distinct, recoverable infrastructure outcome: no Submission, no bypass, and no contributor-failure round spent. Optional absence of project pre-check bindings under a valid locked policy is distinct from an unavailable required check.

Restart, retry and concurrent finalize must preserve the exact checked bytes, manifest, actor/assignment, predecessor, policy context, image digests, configuration and optional packet values. No premature Submission, duplicate effects, partial status transition or external execution inside a database transaction. Build the public REST/CLI surface in PILOT-12 against this same flow; do not introduce another intake lifecycle.

The external checker service/SDK uses Rust as agreed; external OCI images are pinned by digest. Project images may use the runtime their check requires, including the separate agent-checker runtime, behind the same protocol. Image digest pinning identifies the executable artifact; it does not imply every project image is written in Rust.

Current owners and dependencies

Depends on PILOT-01 (#488) for local integration. checkers/api/post_submit_catalogue.py, post_submit_catalogue.py, post_submit_implementations.py and execution.py currently use code-owned structural handlers and constant workstream-structural implementation identity. A handler change can change executable behavior without changing that identifier. Existing isolation finding #133 is covered by this replacement issue, not proof of a shipped remedy.

Implementation scope

• Own the Workstream default checker and its real executable version. Its exactly four blocking defaults run on every project:
  1. Archive safety: one valid ZIP, no encryption, symlinks, special files or traversal (../) paths, within platform size limits.
  2. Exact received identity: Workstream computes SHA-256 itself, verifies stored bytes and builds the file manifest.
  3. No high-confidence secrets: block private keys, cloud credentials and API tokens.
  4. No unchanged resubmission: block the same files as the last attempt unless the task's rules changed since then, for example after a rebase.
     Record each default's positive/negative proof before implementation. Do not infer additional blocking project rules from either old catalogue. Missing summary or attestation gives warnings only and never blocks; the optional request/CLI change is owned by PILOT-12 ([PILOT-12] Upload intent and CLI submit/status #499). A behavior change must change the default checker version; locked tasks retain exact checked identities.
• ADR for default/external split, digest pinning, the Rust external checker service/SDK, deterministic checker images and hosted gVisor (runsc) isolation. Generic agent image PILOT-05 ([PILOT-05] Generic agent checkers through a budgeted model proxy #492) uses its supported SDK runtime behind the same protocol. The ADR specifies launcher composition and isolation; it must preserve this agreed service/protocol boundary.
• Immutable authorized external-image registry registration: capability/version, phase, OCI digest, config/input/output schemas and CPU/memory/time/output limits. Publish immutable registry entries; lock exact digest/config/schema identities in approved policies.
• Extend ADR 0014 through a typed execution capability and ExternalServiceAdapterFactory at the composition root. CHECKERS retains request/lease/current-result ownership; ART retains verified materials/output evidence; TASK retains routing. No second scheduler/result store or runtime plugin discovery.
• Specify pre-phase preparation identity (no Submission yet), post-phase Submission/request/lease identity, bounded verified read-only inputs and normalized findings. Missing image, crash, timeout or invalid output is infrastructure failure; never silently skip a required check.
• Execute outside database transactions. Hosted startup refuses missing gVisor. Local macOS may use explicitly recorded docker-dev; record actual isolation on every phase attempt/CheckerRun. Trusted launcher alone has Docker socket access; sandbox/submitted code never does.
• Retain digest-pinned image availability/provenance for locked tasks. New registry entries affect later compilation only. Delete both built-in pre- and post-submission catalogues, their implementation code and their catalogue-specific tests, once the default checker covers the platform invariants they enforced. Move project requirements to project-owned pre/post images. Map each removed behavior to retained default-checker/ART custody proof or to an explicit project-image replacement; retain or replace tests protecting required outcomes, authority, lineage, rollback and locking. Do not delete retained policy or execution evidence, or keep parallel catalogue runtimes.
• Trace stored policy/request consumers and define retained-data preflight/cutover without rewriting historical hashes or introducing parallel old/new runtimes. ART still verifies identity/lineage before executing and finalizing external evidence.

Required-capability activation guard (F-020)

This guard belongs to PILOT-04 alongside external checker registration and approved policy bindings. PILOT-15 owns Markdown support only; it has no dependency on this guard. Deliver the guard as its own bounded backend PR under [PILOT-04].

• Inspect the existing compilation result's requirements, pre_submit_bindings, post_submit_bindings and capability_suggestions (backend/app/interfaces/project_agents.py) and the current guide activation/approval custody. Reproduce the gap before changing the existing owner operation.
• The PM can still review and approve checks that exist. Refuse guide activation when the exact approved compilation for that guide generation has an uncovered capability suggestion for a required check. Resolve coverage through exact requirement/stage identity and a registered digest-pinned external image bound in the approved policy; neither image registration alone nor a binding from another generation/project clears the guard.
• Return a bounded actionable refusal naming every missing capability by requirement id, stage and title, under existing PM authority and safe error projections. Failed activation commits no lifecycle, activation receipt, task release or outbox effects.
• Recovery is an explicit corrected or re-compiled proposal after engineering registers the image, followed by PM approval of the binding and normal activation. Do not mutate an earlier approved compilation or historical policy to mark its suggestions resolved.
• Preserve the setup agent's proposal format, warning-only findings and their existing acknowledgment flow, caller-owned atomicity, fresh authorization and exact activation replay. No automatic activation. The guard applies regardless of human_review_required; it does not bypass the existing false-policy activation guard owned by PILOT-08 ([PILOT-08] Revision caps, guarded rebases, false-policy activation #495).

Before finalizing the runtime ADR, incorporate the offline-build feasibility result from PILOT-00 (#500). Result-contract preparation can proceed while the spike tests execution feasibility.

Leave alone

No checker-owned acceptance, guide rebase behavior, arbitrary network access, Kubernetes, required platform invariants removed without replacement proof, or rewrites of prior evidence. Required unavailable capabilities block setup/execution clearly.

Acceptance and how to check

• Register a checker without a backend deploy; next guide compilation can propose/select it, and PM approval locks it.
• Two tasks locked to different digests/configurations run their exact versions; registration/redeploy cannot silently change either.
• The default checker has exactly the four recorded blocking defaults and a real version tied to its behavior; changed behavior cannot retain the old identity. Both built-in catalogues and their obsolete code/tests are removed with replacement proof and retained historical evidence.
• Project required-file, project size and forbidden-file rules execute in the project pre-submit image; adding or changing them requires no Workstream deploy. Platform upload/capacity limits and ART integrity/custody remain enforced.
• Real PostgreSQL/MinIO proof: activation with uncovered required capability suggestions is refused and lists the exact missing requirement ids/stages/titles, with no partial effects. Warning-only findings remain acknowledgeable and a guide with no uncovered required suggestions activates under its existing guards.
• Registration alone, wrong-project/stage/requirement bindings and stale approvals cannot clear the guard. Registering the image, obtaining a corrected/re-compiled proposal and approving its exact policy binding enables normal activation. Retain existing activation/replay/concurrency regressions, including the false-policy guard.
• Prove the required intake flow through the real ART/pre-check/Submission/dispatch owners: default rejection and required external pre-check rejection each return findings, create no Submission/dispatch and leave the task in_progress. All required pre-checks passing creates one immutable Submission plus evaluation_pending and one initial durable dispatch atomically. Post failures supply evidence only.
• A required pre-check whose runner/image is unavailable, crashes, times out or returns invalid output cannot be silently skipped or create a Submission; restart/retry recovers the identical checked packet without spending a contributor-failure round. Inject failure before admission and during the root transaction to prove no partial/duplicate effects.
• Real sandbox probes establish read-only submission, no host socket/secret/other-project access, resource limits, denied default egress and cleanup.
• Hosted missing-gVisor startup fails; macOS fallback is visible, never reported as hosted-grade isolation.
• Crashes, unavailable images, bad/oversized output, stale leases and duplicate completion are bounded infrastructure outcomes with recoverable durable evidence.

Runner Compose integration belongs here; model-proxy integration belongs to PILOT-05 (#492). Close only after both pre/post contracts and clean owner cutover are proved.

Two-agent coordination

Agent 1 owns backend result/input contracts, checker registry tables, default-checker/authority composition and built-in catalogue removal and every migration; publish stable contracts/registry first. Agent 2 owns the sandbox launcher and Rust checker SDK against those contracts, with the PILOT-00 (#500) build feasibility result informing the runtime ADR. Any backend/config/composition-root change requested by the runtime lane is handed to Agent 1; avoid two branches independently editing the same contract or allocating migration revisions.

Working and verification rules

Read AGENTS.md, CONTRIBUTING.md, docs/roadmap_status.md and the affected owner contracts before coding. The baseline inspected for this plan is main c0c4fe70f77e2e84347cec979e3de00e9d7f4295; recheck current main and open PRs before selecting a bounded implementation. Open/planned work is not delivered capability.

This is a planning issue and may require several bounded PRs. Record each PR's allowed files, prohibited changes, acceptance criteria and review scope using the repository's existing Commitrail process. Extend existing owner operations/typed public ports; do not add parallel legacy/new implementations or bypass authority. Preserve retained data, locked lineage, immutable evidence and caller-owned atomicity. A planning issue does not authorize deployment or merging a PR.

Run focused positive/negative tests and applicable repository checks. Use real PostgreSQL for database/locking claims and MinIO/S3 for storage claims. Add regressions that fail on the reproduced defect; do not skip failing tests or weaken CI. Run required full CI and affected independent review tracks before requesting human approval. Update affected current specs/ADRs, README/CLI contracts and roadmap in the same PR. Report exact tested head, commands, failure/cleanup evidence and remaining limitations; passing counts alone are insufficient.