Skip to content

[codex] add benchmark runner and baseline - #8

Draft
divo12 wants to merge 5 commits into
codex/mine-exports-ledgerfrom
codex/benchmark-runner-baseline
Draft

[codex] add benchmark runner and baseline#8
divo12 wants to merge 5 commits into
codex/mine-exports-ledgerfrom
codex/benchmark-runner-baseline

Conversation

@divo12

@divo12 divo12 commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Stack

Depends on #7. Review against codex/mine-exports-ledger.

Problem

Exported eval suites are immutable manifests, not proof that the champion is reproducible. OFW needs one runner that executes only developer-visible cases under the frozen runtime/verifiers, establishes a durable baseline, detects semantic drift, labels synthetic cases, and stops before exceeding hard work budgets.

Implementation

  • Adds BenchmarkRunner(processed_harness, export_bundle, policy).
  • Verifies harness revision and execution/lifecycle fingerprints match the benchmark manifest.
  • Rejects selection/admission cases if they are injected into the developer suite before environment preparation.
  • Requires trace family, cluster family, and partition to match the authoritative ledger, so relabelled holdouts also fail.
  • Requires the exact snapshot path and digest to match the ledger, so authorized labels cannot swap in holdout payloads.
  • Verifies every snapshot path stays under the harness artifact root and matches its content digest.
  • Runs lifecycle and all independent verifiers with reset between attempts and guaranteed cleanup.
  • Records verifier errors/abstentions as explicit failed attempts; no score is silently dropped.
  • Supports deterministic repeats and metadata-safe whitespace/noise simulation.
  • Labels every synthetic attempt and caps its benchmark weight.
  • Enforces a hard maximum attempt budget and returns an explicit partial status.
  • Computes semantic digests without runtime-duration noise.
  • Writes durable baseline and benchmark result artifacts atomically.
  • Re-runs the champion and aborts on output/verdict/policy/runtime drift.

TDD evidence

  • Stable champion baseline reproduces across repeated runs.
  • Selection/admission cases never execute through the developer runner.
  • Relabelling a holdout as frontier still fails the ledger check.
  • Swapping a holdout snapshot under authorized developer labels still fails.
  • Reset failure returns ENVIRONMENT_ERROR and persists completed attempt evidence.
  • Unstable output triggers BASELINE_DRIFT.
  • Python verifier errors remain visible failures with zero pass rate.
  • Repeats and synthetic attempts are labeled and weighted correctly.
  • Hard attempt budget stops at the exact limit with BUDGET_EXHAUSTED.
  • Baseline artifact is persisted.

Verification

  • pytest -q — 109 passed
  • Ruff and strict mypy across src and tests — clean
  • forbidden construct scan — no Any, raw dict, object, getattr, setattr, lambdas, or __import__
  • Bandit and deep security scan — zero findings
  • pip-audit — no known vulnerabilities
  • source distribution and wheel build successfully

Deliberate scope

  • Hard v0 budget is attempt count; cost/token budgets require lifecycle cost telemetry rather than invented estimates.
  • Simulation adds deterministic formatting/noise only to privacy-safe snapshots; richer user-text mutation waits for an explicit I/O read policy.
  • Candidate comparison, selection/admission execution, mutation, scheduler, and promotion are not included.

@divo12
divo12 force-pushed the codex/benchmark-runner-baseline branch from 5627f3c to 09a815e Compare August 22, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant