feat(suite): record run provenance in the results summary and report - #48
Merged
Merged
Conversation
Every rate is stochastic — k seeded repeats plus a bootstrap — so a results file that does not say under what conditions it was produced cannot be compared against another one, and CONTRIBUTING requires results to state seeds, hardware and wall-clock. Gap-fill as the issue frames it: seeds, k and wall_clock_s were already emitted at top level. Missing were the base seed, the bootstrap iteration count that actually shaped the intervals, and any platform tag. All of it now lives in one "provenance" block so a reader does not have to know which top-level keys happen to be provenance: base_seed, k, seeds, bootstrap_iters, wall_clock_s, runtime, leakgauge_version runtime is deliberately coarse — OS, architecture, Python version. Results files are committed to the repo, so the tag must not carry a hostname, a username or a CPU model; a test asserts neither the hostname nor the user appears in it. bootstrap_iters was a bare 10_000 default inside build_summary's signature, invisible to anyone reading a results file. It is now DEFAULT_BOOTSTRAP_ITERS and recorded, so a summary states the value that produced its intervals. SCHEMA_VERSION 1 -> 2. Purely additive: every v1 key keeps its place and meaning, pinned by a test, and _provenance_line falls back to the top-level keys so a leaderboard still renders a v1 results file. Surfaced in both stdout paths — the run table prints the conditions directly under the rates, and `report` prints them per model, since a rank reorder compares rates across files and two models run at different k are not comparable. README documents the block and the one-command repro path. 18 tests. The one that matters is determinism: two runs at the same seed produce identical aggregates, per-case rows, and whole summaries once wall-clock (a measurement of the run, not an output of it) is zeroed — recording a seed is only worth anything if quoting it reproduces the numbers. A separate test runs the README's documented command twice over the full stub suite and asserts the same, so the claim in the README is guarded rather than merely written down. 439 passed; ruff check, ruff format --check and mypy src clean. Closes bamdadd#39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39.
Gap-fill as the issue frames it —
seeds,kandwall_clock_swere already emitted at top level. Missing were the base seed, the bootstrap iteration count that actually shaped the intervals, and any platform tag.All of it now lives in one
provenanceblock, so a reader does not have to know which top-level keys happen to be provenance:Two details worth review
runtimeis deliberately coarse — OS, architecture, Python version. Results files are committed to the repo, so the tag must not carry a hostname, a username or a CPU model. A test asserts neithersocket.gethostname()norgetpass.getuser()appears in it.bootstrap_iterswas invisible. It was a bare10_000default inbuild_summary's signature, so nothing in a results file said what produced its intervals. It is nowDEFAULT_BOOTSTRAP_ITERSand recorded.Compatibility
SCHEMA_VERSION1 → 2, purely additive: every v1 key keeps its place and meaning (pinned by a test), and_provenance_linefalls back to the top-level keys so a leaderboard still renders a v1 results file rather than crashing on a missing block.Surfaced in both stdout paths
and per model in
report, since a rank reorder compares rates across files and two models run at differentkare not comparable:Tests
A separate test runs the README's documented repro command twice over the full stub suite and asserts the same, so the claim in the README is guarded rather than merely written down — if the stub suite ever stops being deterministic, the docs are wrong and CI says so.
Plus: every required field present, the seeds actually used are reflected, a non-default bootstrap count is reflected, the v1 keys are unchanged, the v1 fallback renders, and the table surfaces it.
439 passed, 4 xfailed;
ruff check,ruff format --checkandmypy srcclean.