Skip to content

feat(benchmark): record reproducible run provenance - #815

Open
NetVar1337 wants to merge 1 commit into
mainfrom
codex/benchmark-run-provenance
Open

NetVar1337 wants to merge 1 commit into
mainfrom
codex/benchmark-run-provenance

Conversation

@NetVar1337

@NetVar1337 NetVar1337 commented Aug 30, 2026

Copy link
Copy Markdown
Member

Summary

Benchmark reports now fail closed unless they carry a versioned reproducibility record for the exact source, benchmark inputs, live model routing, stack images, and challenge images that produced the result. The same provenance is emitted in JSON evidence, with a concise identity table in Markdown.

Changes

  • Capture source commit/dirty state, submodule revisions, provider/prompt/skill/config hashes, Python/platform, run ID, filters, and context mode before execution.
  • Read credential-filtered, plugin-composed model chains from the live LangGraph container and require its explicit auth priority to match the benchmark process.
  • Inspect challenge container image IDs while containers are running and propagate provenance failures through teardown so incomplete reports are never written.
  • Redact sensitive config keys, every header value, URL user-info, and every URL query value.
  • Require provenance in BenchmarkReport, report.json, index.json, and Markdown output.

Intent

Blast radius

  • Tier-auto — tests, internal refactors, non-policy docs, lockfile-only dep bumps.
  • Tier-delegate — agent prompts, skill bodies, middleware internals, web/CLI features.
  • Tier-supply-chain — CI/workflows, package manifests and lockfiles, install script, compose / Dockerfiles, plugin contracts, .semgrep/**.

Diff budget

  • My diff fits the budget.

The diff touches exactly 10 files and adds 399 runtime-code lines for one concern. The two longer provenance orchestration functions intentionally keep their fail-closed capture sequence visible; splitting them would create single-use forwarding helpers without reducing branching or risk.

End-to-end verification

Ran uv run pytest -q packages/decepticon/tests/unit/benchmark: 128 passed and one pre-existing SDK-coupled test skipped. These tests execute the complete source/artifact/config capture against a real temporary Git repository, exercise reporter JSON/Markdown wiring, capture a running-container image ID before teardown, and verify that missing image identity propagates as ProvenanceCaptureError. Docker and the live LangGraph stack are unavailable in this workspace, so I could not execute a paid end-to-end benchmark run or make smoke; Docker/model boundaries were exercised with deterministic subprocess stand-ins instead, and that unverified live-stack surface is explicit here.

Testing

  • make quality passes (Python + CLI + Web)
  • make smoke succeeds (clean local build + OSS-style up + health checks)
  • pytest tests/ passes (run this if you touched docker-compose.yml or tests/)
  • Every new/changed test was watched to fail without the change and pass with it
  • Every new/changed code path was executed on my machine, not just unit-tested in isolation
  • Manual testing (describe): make ci-lint passed (ruff, format, basedpyright: 0 errors). env -u ALL_PROXY -u all_proxy -u HTTP_PROXY -u http_proxy -u HTTPS_PROXY -u https_proxy -u NO_PROXY -u no_proxy uv run pytest -n auto -q -m 'not slow' -k 'not test_ops_available_true_when_daemon_accepting' passed: 5,140 passed, 45 skipped. A full make ci-test was also attempted; the only failure was the excluded existing AF_UNIX socket test because this workspace rejects socket creation with EPERM.

Quality Bar self-check

  • No banned pattern from QUALITY_BAR appears in the diff.
  • No AI-slop signature survives.
  • Every changed line traces to the stated intent.
  • Every public function added/changed has explicit type annotations and every raised exception is named.
  • I would merge this PR if a stranger opened it.
  • I would still merge this at the end of a long review day.

Related Issues

Closes #814

Comment thread benchmark/provenance.py Fixed
@NetVar1337
NetVar1337 force-pushed the codex/benchmark-run-provenance branch from 016db06 to a2d5526 Compare August 30, 2026 13:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make benchmark reports reproducible with run provenance

2 participants