Skip to content

feat(suite): record run provenance in the results summary and report - #48

Merged
bamdadd merged 1 commit into
bamdadd:mainfrom
dchaudhari7177:feat/run-provenance
Aug 11, 2026
Merged

feat(suite): record run provenance in the results summary and report#48
bamdadd merged 1 commit into
bamdadd:mainfrom
dchaudhari7177:feat/run-provenance

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Closes #39.

Gap-fill as the issue frames it — seeds, k and wall_clock_s were already emitted at top level. Missing were the base seed, the bootstrap iteration count that actually shaped the intervals, and any platform tag.

All of it now lives in one provenance block, so a reader does not have to know which top-level keys happen to be provenance:

"provenance": {
  "base_seed": 0,
  "k": 5,
  "seeds": [0, 1, 2, 3, 4],
  "bootstrap_iters": 10000,
  "wall_clock_s": 0.0128,
  "runtime": "Linux x86_64 python-3.12.4",
  "leakgauge_version": "0.0.1"
}

Two details worth review

runtime is deliberately coarse — OS, architecture, Python version. Results files are committed to the repo, so the tag must not carry a hostname, a username or a CPU model. A test asserts neither socket.gethostname() nor getpass.getuser() appears in it.

bootstrap_iters was invisible. It was a bare 10_000 default in build_summary's signature, so nothing in a results file said what produced its intervals. It is now DEFAULT_BOOTSTRAP_ITERS and recorded.

Compatibility

SCHEMA_VERSION 1 → 2, purely additive: every v1 key keeps its place and meaning (pinned by a test), and _provenance_line falls back to the top-level keys so a leaderboard still renders a v1 results file rather than crashing on a missing block.

Surfaced in both stdout paths

  spend: $0.00 (0 in / 0 out tokens) (unpriced model — $0)
  run: base_seed=0 k=5 bootstrap_iters=10000 wall=0.0128s | Linux x86_64 python-3.12.4 | leakgauge 0.0.1

and per model in report, since a rank reorder compares rates across files and two models run at different k are not comparable:

  run conditions per model:
  stub:demo               run: base_seed=0 k=5 bootstrap_iters=10000 ...
  stub:b                  run: base_seed=0 k=5 bootstrap_iters=10000 ...

Tests

  1. The one that matters is determinism — recording a seed is only worth anything if quoting it reproduces the numbers. Two runs at the same seed produce identical aggregates, identical per-case rows, and identical whole summaries once wall-clock (a measurement of the run, not an output of it) is zeroed.

A separate test runs the README's documented repro command twice over the full stub suite and asserts the same, so the claim in the README is guarded rather than merely written down — if the stub suite ever stops being deterministic, the docs are wrong and CI says so.

Plus: every required field present, the seeds actually used are reflected, a non-default bootstrap count is reflected, the v1 keys are unchanged, the v1 fallback renders, and the table surfaces it.

439 passed, 4 xfailed; ruff check, ruff format --check and mypy src clean.

Every rate is stochastic — k seeded repeats plus a bootstrap — so a results
file that does not say under what conditions it was produced cannot be
compared against another one, and CONTRIBUTING requires results to state
seeds, hardware and wall-clock.

Gap-fill as the issue frames it: seeds, k and wall_clock_s were already
emitted at top level. Missing were the base seed, the bootstrap iteration
count that actually shaped the intervals, and any platform tag. All of it now
lives in one "provenance" block so a reader does not have to know which
top-level keys happen to be provenance:

  base_seed, k, seeds, bootstrap_iters, wall_clock_s, runtime,
  leakgauge_version

runtime is deliberately coarse — OS, architecture, Python version. Results
files are committed to the repo, so the tag must not carry a hostname, a
username or a CPU model; a test asserts neither the hostname nor the user
appears in it.

bootstrap_iters was a bare 10_000 default inside build_summary's signature,
invisible to anyone reading a results file. It is now DEFAULT_BOOTSTRAP_ITERS
and recorded, so a summary states the value that produced its intervals.

SCHEMA_VERSION 1 -> 2. Purely additive: every v1 key keeps its place and
meaning, pinned by a test, and _provenance_line falls back to the top-level
keys so a leaderboard still renders a v1 results file.

Surfaced in both stdout paths — the run table prints the conditions directly
under the rates, and `report` prints them per model, since a rank reorder
compares rates across files and two models run at different k are not
comparable.

README documents the block and the one-command repro path.

18 tests. The one that matters is determinism: two runs at the same seed
produce identical aggregates, per-case rows, and whole summaries once
wall-clock (a measurement of the run, not an output of it) is zeroed —
recording a seed is only worth anything if quoting it reproduces the numbers.
A separate test runs the README's documented command twice over the full stub
suite and asserts the same, so the claim in the README is guarded rather than
merely written down.

439 passed; ruff check, ruff format --check and mypy src clean.

Closes bamdadd#39
@bamdadd
bamdadd merged commit f22e02f into bamdadd:main Aug 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record run provenance (seed, k, wall-clock, hardware) in results summary

2 participants