v0.1.7 fs-bench-pro: establish core benchmark substrate before Init pilot

[yifanxuaaa]

Current status — 2026-09-23

The four-command Core fs-bench-pro substrate was merged into main at bb6c3a7df. The spec and retained first-pass report are committed. run now skips the independent full verifier by default for the fast lane and records SKIPPED/DIAGNOSTIC; run --verify requests the full proof. Existing receipts are unchanged.

The final-source canary exercised all ten public-route witnesses and cleaned up, but one Service telemetry record was dropped, so that receipt is INCOMPLETE; earlier identities had passing canaries. Keep this issue open until its exact-identity canary/telemetry acceptance is resolved or explicitly ruled. The 100/1,000 Init rows and 10,000 failure belong to parent #231.

---

Original filing (retained requirements; implementation-status statements below are historical)

Parent: #230. Prerequisite for the #231 first pass. This harness is for fast bug fixing and algorithm iteration, not a broad evaluation framework. #179 is closed and supplies a bounded real daemon/Workspace canary; it does not supply native Init performance.

Spec: core/docs/benchmark/fs-bench-pro/issue-231/SPEC.md (working-tree proposal; commit before implementation or collection). Agent reading/do/do-not prompt: core/docs/benchmark/fs-bench-pro/issue-231/HANDOFF.md.

Minimal substrate

• Python orchestration around the existing production daemon/Service binaries and public protocol. Do not create a second Cargo workspace, lockfile, Rust benchmark driver, control/candidate arms, paired scheduler, sample ladder or empty future-mode adapters. Add a tiny Rust verifier/driver only if the public product API and the 5 s full oracle prove it necessary.
• Put Init case declarations, its public operation and full verifier child in families/init_namespace.py; focused case/fixture/oracle-refusal checks go in tests/test_init_namespace.py. The verifier receipt is the full benchmark proof, not a unit-test result.
• Four initial operations: list; run --case ID --out NEW (lazy one-case fixture acquisition, exact build/image reuse, one public performance sample, separate full verifier child, telemetry ingest and cleanup); verify --run DIR (re-derive retained evidence); report --run DIR (render only). run --family init_namespace serially runs the 100/1,000/10,000 cases once each, with one build. The 100,000 row remains NOT_RUN.
• One sample per case at one source identity. A relevant edit creates a new diagnostic identity and fresh output; it is not a statistical repeat or an excuse to hide earlier failures. Dirty-source iteration is diagnostic; admission needs a clean seal.
• No machine-global benchmark lock. One nonblocking worktree-local run flock protects same-worktree mutable fixtures/output; it never blocks another worktree. Builds take no benchmark lock. Private worktree Cargo target, prepared source, Store, scratch and results; unique per-run ports, credentials, container names and telemetry namespaces. Refuse foreign target/output paths. Record observed CPU/disk/page-cache interference rather than claiming concurrent rows were quiet.
• Build only changed production binaries using the existing core lock/target and --locked. Every invoked Cargo build, including first-use, must be <=30 s; retain BUILD_SLOW and fix the cause on a miss. Reuse full immutable executable/image IDs. Do not inherit the legacy global BuildKit Cargo cache contention, mutable tag race or broad image pruner.
• Full separate verifier child <=5 s wall per case, including reopen, complete oracle and teardown. Preparation + three runs + proofs + cleanup should finish within 30 s per family; record misses without shrinking work or extending a deadline. Keep existing 15 s complete-performance-command budget and prospectively declared small exceptions <=25 s.
• LFT1 is the literal prefix of a version-1 JSON telemetry event, not a benchmark sample or phase. Retain one exact telemetry.lft1 source file and parsed receipt per telemetry-on case, plus raw timing/verification evidence and manifest. Recycle only validated temporary stderr/owned operational segments. Preserve every failed, ineligible, incomplete and unrun row.

Acceptance

1. A committed, reviewed spec freezes the initial CLI, one-sample boundary, cache/resource scopes, build/proof/family speed limits, worktree isolation, evidence and cleanup.
2. Focused checks prove selector refusal, seal mismatch, target/path escape, concurrent separate worktrees, same-worktree run refusal, one-sample cardinality, LFT1 loss/truncation handling, failed-cleanup retention and report re-derivation.
3. A real v0.1.7 co-design pair 1: projection and runtime — FUSE against the workspace accumulator #179 public-operation canary proves deployment, telemetry capture, teardown and receipt flow at one pinned identity. It is diagnostic and cannot clear Init.
4. Report actual cold/edited/no-op Cargo and verifier walls. No unresolved >30 s Cargo build or >5 s verifier at the accepted canary identity is hidden as a PASS. A 30 s family-cycle miss is visible and investigated.

Expected production LOC delta for this harness-only prerequisite is 0. Report actual benchmark/tooling LOC separately. #231 owns its missing public native-directory importer and the first three Init numbers; the 100,000 final gate remains open.