Skip to content

chore(experiments): make RESULTS.md a baseline instead of a test output - #197

Merged
pdettori merged 1 commit into
rossoctl:mainfrom
pdettori:chore/e2-results-baseline
Aug 31, 2026
Merged

chore(experiments): make RESULTS.md a baseline instead of a test output#197
pdettori merged 1 commit into
rossoctl:mainfrom
pdettori:chore/e2-results-baseline

Conversation

@pdettori

Copy link
Copy Markdown
Member

experiments/RESULTS.md was both a checked-in artifact and a test output: E2 rewrote it on every run, so any pnpm -r test left the working tree dirty with machine-local wall-clock timings, and the recorded result silently became "whatever the last person to run the suite measured".

This splits the two roles:

  • Each run writes its own table to the gitignored experiments/.results/ (SH_E2_RESULTS_DIR=<dir> to redirect).
  • The committed RESULTS.md becomes a baseline E2 asserts against, so a change that moves the read counts has to be acknowledged rather than quietly rewriting the record. Refresh deliberately with SH_E2_UPDATE_BASELINE=1 pnpm -C experiments test e2-reconstruction-cost.

Which columns a baseline can actually assert

Measured rather than assumed — E2 run twice locally, diffed against the CI run captured earlier today:

Column Local CI Comparable?
backend / checkpoint entries 53, 203, 1003, 5003 / 6,6,6,6 identical yes
ratio 8.8, 33.8, 167.2, 833.8 identical yes
backendBytes 7482, 28508, 140908, 706909 7486, 28512, 140912, 706913 no — consistent +4, serialization is environment-sensitive
checkpointBytes 896, 901, 901, 906 identical no — same class as above, excluded on principle
backendMs / checkpointMs vary run to run no

So deterministicView() covers n, both entries columns, and the ratio at the 1 decimal the table carries. Had I compared bytes, this would have gone green locally and failed on CI.

Changes

  • experiments/src/report.ts — adds parseE2Table() (reads the baseline back, tolerating surrounding prose and unrelated tables) and deterministicView(), with the measurement rationale documented at the type.
  • experiments/test/e2-reconstruction-cost.test.ts — writes to the gitignored dir; compares against the baseline unless SH_E2_UPDATE_BASELINE=1.
  • experiments/test/report.test.ts — new, 8 cases: round-trip, prose/other-table tolerance, throwing rather than silently returning nothing on a malformed table, drift-insensitivity, that a real entries change is caught, and that the committed baseline itself parses and round-trips (catching a hand-edit that breaks the shape E2 reads).
  • .gitignore, experiments/README.md — the ignored path and the refresh workflow.

Verification

  • Three consecutive E2 runs leave RESULTS.md untouched (git diff --quiet clean) while each refreshes the gitignored copy — the actual point of the change.
  • Negative control: perturbing a baseline entries cell (67) fails E2 with a clear diff; restoring it passes and leaves the file byte-identical.
  • pnpm -r test: experiments 51 → 59 passed, 0 failures repo-wide. make typecheck clean.
  • The baseline refresh in this commit moves only the ms columns — every asserted column is unchanged from what was already committed, which is the evidence the baseline is genuine rather than fitted to my machine.

Note on style

experiments/test/report.test.ts uses double quotes to match the other 12 files in the package. .prettierrc declares singleQuote: true, but nothing in experiments/ follows it and prettier is gated neither in CI nor (working) pre-commit — so I matched the surrounding code rather than leave one odd file out. Worth a separate repo-wide formatting pass if you want the config actually honoured; I didn't fold that into this change.

make lint fails on this machine before and after (Type tag 'typescript' is not recognized — pre-commit's identify is too old). Pre-existing and unrelated.

E2 overwrote the committed experiments/RESULTS.md on every run, so any
`pnpm -r test` left the working tree dirty with machine-local wall-clock
timings. The file was simultaneously a checked-in artifact and a test output,
which meant the recorded result silently became "whatever the last person to
run the suite measured".

Splits the two roles:

  - each run writes its own table to the gitignored experiments/.results/
    (override with SH_E2_RESULTS_DIR);
  - the committed RESULTS.md becomes a baseline that E2 asserts against, so a
    change moving the read counts has to be acknowledged rather than quietly
    rewriting the record. Refresh deliberately with SH_E2_UPDATE_BASELINE=1.

Only the columns that actually reproduce are compared. Measured, rather than
assumed, by running E2 twice locally and diffing against the CI run captured
earlier today:

  - entries + ratio: identical on both (deterministic synthetic fixtures)
  - backendBytes:    7482/28508/140908/706909 locally vs 7486/28512/140912/
                     706913 on CI — a consistent +4, so serialization is
                     environment-sensitive and cannot be asserted
  - backendMs / checkpointMs: vary run to run even on one machine

Hence deterministicView() covers n, both entries columns, and the ratio at the
1 decimal the table carries. Adds parseE2Table() to read the baseline back,
with unit tests for the round-trip, the drift-insensitivity, and that a real
entries change is still caught. Verified end to end: three consecutive E2 runs
leave RESULTS.md untouched, and perturbing a baseline entries cell fails E2.

The baseline refresh in this commit moves only the ms columns — every asserted
column is unchanged from what was already committed.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
@pdettori
pdettori merged commit 40f6f45 into rossoctl:main Aug 31, 2026
10 checks passed
@pdettori
pdettori deleted the chore/e2-results-baseline branch August 31, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant