Circuit-sweep (program-plonky2) in CI vs manual pre-release — decide + implement

[TaprootFreak]

Background

The full cyclic-recursion sweep for program-plonky2 (cargo test -p zkcoins-program-plonky2 --release --lib -- --test-threads=1) is not in CI today. It was deliberately kept out of the issue #40 migration: at production parameters (MAX_IN_COINS = 8) the sweep wallclocks in hours, single-runner serialization on dfx01 means running it on every push would saturate the M3 Ultra and starve normal PR-tests + Coverage Gate jobs.

The current convention (documented in CONTRIBUTING.md § Setup and scripts/ci-runner/README.md) is: run it manually before any release-PR-to-main.

This issue tracks the decision on whether / how to formalize that.

Options

A. Stay manual

Pros: zero runner load. Status quo.
Cons: easy to forget. No record of what was verified for each release.

B. Dedicated ci:circuit-sweep PR label

Same pattern as #48 (ci:full). Authors opt in; CI runs the sweep as an extra job when the label is set. Required only on the release-PR-to-main, enforced by a checklist item or a separate gate.

Pros: explicit opt-in, leaves a CI record, can be required on release PRs only.
Cons: each release PR pays the multi-hour wall-time as a blocker.

C. Scheduled run on develop

schedule: trigger (e.g. nightly or weekly). Sweep runs out-of-band on the latest develop. Result published as a status check / commit status.

Pros: catches regressions automatically; doesn't block PRs.
Cons: scheduled runs on develop don't gate PR merges by default — drift between "develop sweep last ran at SHA X" and the PR's HEAD.

D. Path-filter on program-plonky2/**

Sweep runs only when the PR touches circuit code (program-plonky2/**). Most PRs (server/docs/CI) skip it entirely.

Pros: targeted; circuit-code PRs get the sweep automatically without manual opt-in.
Cons: still blocks circuit-code PRs for hours. Server-side changes that interact with circuit behaviour at the witness layer wouldn't trigger it.

E. Combination

E.g. D + C: path-filter on program-plonky2/** for circuit PRs + nightly scheduled on develop as backstop. Or B + C: opt-in label + nightly schedule.

Recommendation needed on

1. Which option(s)?
2. If B or D: blocking (required check) or advisory (informational)?
3. If C: cadence + alert path on failure?

Implementation surface (whichever option lands)

• .github/workflows/ci.yaml: add a circuit-sweep job behind the chosen trigger (label / path-filter / schedule).
• runs-on: [self-hosted, m3-ultra], --test-threads=1, timeout-minutes bumped to ~360 to be safe.
• scripts/ci-runner/README.md § "Operations": document the new job + its expected wallclock.
• CONTRIBUTING.md § Setup: drop the "manual sweep before release-PR" instruction (or refine it depending on option).

Refs

• Issue Move test + coverage gate from pre-push to CI via self-hosted runner on M3 Ultra #40, closing comment lists this as Open Question 4.

[TaprootFreak]

Recommendation: Option D (path-filter), required on PRs to main, with ci:circuit-sweep label escape-hatch

Add a single circuit-sweep job to .github/workflows/ci.yaml that runs on [self-hosted, m3-ultra] and is triggered by either:

1. paths: filter matching program-plonky2/** on the PR, or
2. The PR label ci:circuit-sweep (manual opt-in for the cases where server-side code changes the witness layout — see below).

Mark it required on PRs targeting main (the release PR); advisory on PRs targeting develop.

Why D over B / C / E

Witness-layer risk is real but narrow. The boundary is program-plonky2/src/inputs.rs::ProgramInputs plus the HashOut<F> ↔ [u8;32] conversions in shared/server (digest_from_bytes / digest_to_bytes, MMR_PROOF_PATH_LEN extension, limb packing for the BIP-340 pubkey). A server-side change that re-orders a field, miscounts MMR siblings, or flips endianness on a digest can break proving without touching circuit code. That's the case Option D alone misses — and it's exactly what the ci:circuit-sweep label escape-hatch is for. Reviewer applies it when a PR touches shared/src/lib.rs, server/src/state.rs, or anything that crosses the witness boundary.

Runner saturation kills E (B+C) and pure B. The M3 Ultra is single-tenant, --test-threads=1, ~50 GB RAM/thread (scripts/ci-runner/README.md § "Disk + RAM headroom"). A scheduled nightly sweep collides with the next morning's ci:full runs; a label-gated sweep on every release PR pays multi-hour wall-time as a blocker every time. Path-filtering keeps the sweep off the queue for the 90% of PRs that touch only server/ / docs / CI.

Release cadence + forgetfulness data point. develop→main has merged exactly once since this repo's lifetime (PR #16, pre-Plonky2). The manual "run sweep before release-PR" convention in CONTRIBUTING.md has therefore never been exercised in practice — the Plonky2 path has shipped to PRD zero times. Codifying it as CI before the first real release is cheap insurance against forgetting on attempt #1.

Implementation (single PR)

• .github/workflows/ci.yaml: new circuit-sweep job. runs-on: [self-hosted, m3-ultra], timeout-minutes: 360, if: guarded on paths filter OR contains(labels.*.name, 'ci:circuit-sweep') OR github.base_ref == 'main'. Step body: cd program-plonky2 && cargo test --release --lib -- --test-threads=1. PATH-prepend trick from server-tests reused.
• Branch protection (separate UI op, not in the PR): add Circuit Sweep (M3 Ultra) as required on main only.
• CONTRIBUTING.md § Setup: drop the "run circuit sweep manually before release-PR" instruction; replace with a one-liner pointing at the new job + the ci:circuit-sweep label for witness-boundary PRs.
• scripts/ci-runner/README.md § Operations: add a "Circuit Sweep" subsection — expected wallclock (multi-hour), single-runner queue impact, how to cancel via gh run cancel if a release PR needs to ship around a known-good sweep.
• program-plonky2/CONTRIBUTING.md § "CI integration": flip "NOT in CI" → "in CI behind path-filter + label".

Follow-ups not solved here

• Failure semantics on release PR: required check = hard block on merge. We'll need a documented override path for the case where the sweep is red on a non-circuit-related flake (orphan-test-binary OOM per feedback_cleanup_test_binaries.md, runner disk pressure). Proposal: re-run via gh run rerun, not bypass. If a circuit regression genuinely blocks a hotfix release, revert the offending develop commit rather than override the gate. Track as a separate issue.
• Caching strategy: shared target/ per scripts/ci-runner/README.md § Workspace cache means a green sweep can flip red on stale incremental state. Mitigation: dedicated _work directory for the circuit-sweep job, separate from server-tests. Out of scope here.
• Runtime growth: if future Stage 5d-next-N work bumps INNER_PAD_BITS further, sweep wallclock grows superlinearly. Revisit timeout-minutes: 360 when that happens.

Second-best: Option B (label-gated, required on release PR)

Considered and rejected. Pros: same blocking semantics, no paths: heuristic to maintain. Cons: requires PR authors to remember the label on every circuit-touching PR — that's exactly the forgetfulness failure mode this issue is trying to eliminate, just relocated from "remember to run the sweep" to "remember to apply the label". Path-filter automates the common case (circuit PR) and keeps the label as a deliberate, documented escape-hatch for the rare witness-boundary case where the heuristic is wrong. B becomes the right answer if paths: filtering proves too noisy in practice (e.g. if Cargo.lock-only changes start triggering the sweep); revisit then.