the corpus, read and run in Python - #10
Merged
Merged
Conversation
The engine ships 945 conformance cases and a runner in Rust. Nine
clients are meant to answer the same cases, and this is the Python
one. It is a reader as well as a runner, which is the part worth
explaining: the corpus is written in a small corner of YAML, and the
value of a second reader is that it refuses what the first refuses.
PyYAML would read these files and a good deal more besides, and would
hand back 9223372036854775807 as a float on the way, so the subset is
written out here as it is written out in Rust and in C.
`conformance/` holds the four pieces: the YAML subset, the {type,
value} encoding with the comparison and the report spelling, the case
and suite shapes, and the runner. `python -m conformance <dir>` prints
the same summary the Rust runner prints, and the report lines are
spelled to match it character for character, because two runners over
one corpus are worth much more when their output can be diffed.
That diffing is how this was checked rather than only asserted. Two
sweeps over the real corpus ran both runners per suite: one corrupting
every third quoted payload, which exercises every refusal, and one
rewriting every row value to a valid but different value of the same
type, which exercises the report formatter. The second found four
places where the Python text differed from the Rust text, all of them
now covered by a test: a float exponent written 1e+16 rather than 1e16,
a fraction of six digits rather than nine, a zero offset written +00:00
rather than Z, and a duration keeping the fields that are zero.
It also found something better than a formatting bug. Five cases were
passing vacuously. The engine keeps a time to the nanosecond and
Python's datetime keeps one to the microsecond, so decoding the case's
own expectation through this client's value mapping truncated it, and
the comparison then held one truncated value against another. Those
cases exist to catch exactly that. A payload finer than a microsecond
now decodes to a value nothing equals, and the runner reports the case
as unsupported with the value named, because a value mapping that
cannot carry the value is a limit of this client rather than a wrong
answer from the engine. The five are pinned by name in the test suite
so that a sixth one appearing is a failure and not a shrug.
The tests are the tables from crates/zu-corpus/src/*.rs ported case for
case, so a reader that grew a hole would pass its own tests and fail
these. The run over the cases themselves needs the engine's checkout
and is skipped without ZU_CASES, so a clone with no engine beside it
still tests everything that does not need one. CI gets a corpus job
that takes the revision out of the pin this client already builds
against, so the cases and the engine cannot drift apart.
Two things ride along because the pin moved. The engine's
`file_mut` now returns a Result, which the appender follows, and a
pattern that the graph cannot satisfy now matches nothing rather than
raising, so the error test that asserted the old behaviour is replaced
by one for each half of what changed.
20 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The engine ships 945 conformance cases and a runner in Rust. Nine clients are meant to answer the same cases, and this is the Python one. It is a reader as well as a runner, which is the part worth explaining: the corpus is written in a small corner of YAML, and the value of a second reader is that it refuses what the first refuses. PyYAML would read these files and a good deal more besides, and would hand back 9223372036854775807 as a float on the way, so the subset is written out here as it is written out in Rust and in C.
conformance/holds four pieces: the YAML subset, the{type, value}encoding with the comparison and the report spelling, the case and suite shapes, and the runner.python -m conformance <dir>prints the same summary the Rust runner prints, and the report lines are spelled to match it character for character, because two runners over one corpus are worth much more when their output can be diffed.That diffing is how this was checked rather than only asserted. Two sweeps over the real corpus ran both runners per suite: one corrupting every third quoted payload, which exercises every refusal, and one rewriting every row value to a valid but different value of the same type, which exercises the report formatter. The second found four places where the Python text differed from the Rust text, all now covered by a test: a float exponent written
1e+16rather than1e16, a fraction of six digits rather than nine, a zero offset written+00:00rather thanZ, and a duration keeping the fields that are zero.It also found something better than a formatting bug. Five cases were passing vacuously. The engine keeps a time to the nanosecond and Python's
datetimekeeps one to the microsecond, so decoding the case's own expectation through this client's value mapping truncated it, and the comparison then held one truncated value against another. Those cases exist to catch exactly that. A payload finer than a microsecond now decodes to a value nothing equals, and the runner reports the case as unsupported with the value named, because a value mapping that cannot carry the value is a limit of this client rather than a wrong answer from the engine. The five are pinned by name in the test suite, so a sixth one appearing is a failure and not a shrug.The run reads
945 cases, 940 passed, 0 failed, 5 unsupported.The tests are the tables from
crates/zu-corpus/src/*.rsported case for case, so a reader that grew a hole would pass its own tests and fail these. The run over the cases themselves needs the engine's checkout and is skipped withoutZU_CASES, so a clone with no engine beside it still tests everything that does not need one. CI gets a corpus job that takes the revision out of the pin this client already builds against, so the cases and the engine cannot drift apart.Two things ride along because the pin moved. The engine's
file_mutnow returns aResult, which the appender follows, and a pattern the graph cannot satisfy now matches nothing rather than raising, so the error test that asserted the old behaviour is replaced by one for each half of what changed.Gates run locally and green:
ruff check,ruff format --check,cargo fmt --all --check,cargo clippy --all-targets -- -D warnings, both no-default-features builds, and 381 tests.