Skip to content

the corpus, read and run in Python - #10

Merged
tamnd merged 1 commit into
mainfrom
corpus-runner
Aug 18, 2026
Merged

tamnd merged 1 commit into
mainfrom
corpus-runner

Conversation

@tamnd

@tamnd tamnd commented Aug 18, 2026

Copy link
Copy Markdown
Owner

The engine ships 945 conformance cases and a runner in Rust. Nine clients are meant to answer the same cases, and this is the Python one. It is a reader as well as a runner, which is the part worth explaining: the corpus is written in a small corner of YAML, and the value of a second reader is that it refuses what the first refuses. PyYAML would read these files and a good deal more besides, and would hand back 9223372036854775807 as a float on the way, so the subset is written out here as it is written out in Rust and in C.

conformance/ holds four pieces: the YAML subset, the {type, value} encoding with the comparison and the report spelling, the case and suite shapes, and the runner. python -m conformance <dir> prints the same summary the Rust runner prints, and the report lines are spelled to match it character for character, because two runners over one corpus are worth much more when their output can be diffed.

That diffing is how this was checked rather than only asserted. Two sweeps over the real corpus ran both runners per suite: one corrupting every third quoted payload, which exercises every refusal, and one rewriting every row value to a valid but different value of the same type, which exercises the report formatter. The second found four places where the Python text differed from the Rust text, all now covered by a test: a float exponent written 1e+16 rather than 1e16, a fraction of six digits rather than nine, a zero offset written +00:00 rather than Z, and a duration keeping the fields that are zero.

It also found something better than a formatting bug. Five cases were passing vacuously. The engine keeps a time to the nanosecond and Python's datetime keeps one to the microsecond, so decoding the case's own expectation through this client's value mapping truncated it, and the comparison then held one truncated value against another. Those cases exist to catch exactly that. A payload finer than a microsecond now decodes to a value nothing equals, and the runner reports the case as unsupported with the value named, because a value mapping that cannot carry the value is a limit of this client rather than a wrong answer from the engine. The five are pinned by name in the test suite, so a sixth one appearing is a failure and not a shrug.

The run reads 945 cases, 940 passed, 0 failed, 5 unsupported.

The tests are the tables from crates/zu-corpus/src/*.rs ported case for case, so a reader that grew a hole would pass its own tests and fail these. The run over the cases themselves needs the engine's checkout and is skipped without ZU_CASES, so a clone with no engine beside it still tests everything that does not need one. CI gets a corpus job that takes the revision out of the pin this client already builds against, so the cases and the engine cannot drift apart.

Two things ride along because the pin moved. The engine's file_mut now returns a Result, which the appender follows, and a pattern the graph cannot satisfy now matches nothing rather than raising, so the error test that asserted the old behaviour is replaced by one for each half of what changed.

Gates run locally and green: ruff check, ruff format --check, cargo fmt --all --check, cargo clippy --all-targets -- -D warnings, both no-default-features builds, and 381 tests.

The engine ships 945 conformance cases and a runner in Rust. Nine
clients are meant to answer the same cases, and this is the Python
one. It is a reader as well as a runner, which is the part worth
explaining: the corpus is written in a small corner of YAML, and the
value of a second reader is that it refuses what the first refuses.
PyYAML would read these files and a good deal more besides, and would
hand back 9223372036854775807 as a float on the way, so the subset is
written out here as it is written out in Rust and in C.

`conformance/` holds the four pieces: the YAML subset, the {type,
value} encoding with the comparison and the report spelling, the case
and suite shapes, and the runner. `python -m conformance <dir>` prints
the same summary the Rust runner prints, and the report lines are
spelled to match it character for character, because two runners over
one corpus are worth much more when their output can be diffed.

That diffing is how this was checked rather than only asserted. Two
sweeps over the real corpus ran both runners per suite: one corrupting
every third quoted payload, which exercises every refusal, and one
rewriting every row value to a valid but different value of the same
type, which exercises the report formatter. The second found four
places where the Python text differed from the Rust text, all of them
now covered by a test: a float exponent written 1e+16 rather than 1e16,
a fraction of six digits rather than nine, a zero offset written +00:00
rather than Z, and a duration keeping the fields that are zero.

It also found something better than a formatting bug. Five cases were
passing vacuously. The engine keeps a time to the nanosecond and
Python's datetime keeps one to the microsecond, so decoding the case's
own expectation through this client's value mapping truncated it, and
the comparison then held one truncated value against another. Those
cases exist to catch exactly that. A payload finer than a microsecond
now decodes to a value nothing equals, and the runner reports the case
as unsupported with the value named, because a value mapping that
cannot carry the value is a limit of this client rather than a wrong
answer from the engine. The five are pinned by name in the test suite
so that a sixth one appearing is a failure and not a shrug.

The tests are the tables from crates/zu-corpus/src/*.rs ported case for
case, so a reader that grew a hole would pass its own tests and fail
these. The run over the cases themselves needs the engine's checkout
and is skipped without ZU_CASES, so a clone with no engine beside it
still tests everything that does not need one. CI gets a corpus job
that takes the revision out of the pin this client already builds
against, so the cases and the engine cannot drift apart.

Two things ride along because the pin moved. The engine's
`file_mut` now returns a Result, which the appender follows, and a
pattern that the graph cannot satisfy now matches nothing rather than
raising, so the error test that asserted the old behaviour is replaced
by one for each half of what changed.
@tamnd
tamnd merged commit 70a410e into main Aug 18, 2026
28 of 30 checks passed
@tamnd
tamnd deleted the corpus-runner branch August 18, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant