feat(benchmark): record reproducible run provenance - #815
Open
NetVar1337 wants to merge 1 commit into
Open
NetVar1337 wants to merge 1 commit into
NetVar1337 wants to merge 1 commit into
Conversation
NetVar1337
force-pushed
the
codex/benchmark-run-provenance
branch
from
August 30, 2026 13:22
016db06 to
a2d5526
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Benchmark reports now fail closed unless they carry a versioned reproducibility record for the exact source, benchmark inputs, live model routing, stack images, and challenge images that produced the result. The same provenance is emitted in JSON evidence, with a concise identity table in Markdown.
Changes
BenchmarkReport,report.json,index.json, and Markdown output.Intent
Blast radius
.semgrep/**.Diff budget
The diff touches exactly 10 files and adds 399 runtime-code lines for one concern. The two longer provenance orchestration functions intentionally keep their fail-closed capture sequence visible; splitting them would create single-use forwarding helpers without reducing branching or risk.
End-to-end verification
Ran
uv run pytest -q packages/decepticon/tests/unit/benchmark: 128 passed and one pre-existing SDK-coupled test skipped. These tests execute the complete source/artifact/config capture against a real temporary Git repository, exercise reporter JSON/Markdown wiring, capture a running-container image ID before teardown, and verify that missing image identity propagates asProvenanceCaptureError. Docker and the live LangGraph stack are unavailable in this workspace, so I could not execute a paid end-to-end benchmark run ormake smoke; Docker/model boundaries were exercised with deterministic subprocess stand-ins instead, and that unverified live-stack surface is explicit here.Testing
make qualitypasses (Python + CLI + Web)make smokesucceeds (clean local build + OSS-style up + health checks)pytest tests/passes (run this if you toucheddocker-compose.ymlortests/)make ci-lintpassed (ruff, format, basedpyright: 0 errors).env -u ALL_PROXY -u all_proxy -u HTTP_PROXY -u http_proxy -u HTTPS_PROXY -u https_proxy -u NO_PROXY -u no_proxy uv run pytest -n auto -q -m 'not slow' -k 'not test_ops_available_true_when_daemon_accepting'passed: 5,140 passed, 45 skipped. A fullmake ci-testwas also attempted; the only failure was the excluded existing AF_UNIX socket test because this workspace rejects socket creation withEPERM.Quality Bar self-check
Related Issues
Closes #814