Filed on the back of #908, which fixed Code Coverage being red repo-wide for a
test binary-path bug. The job now works — so its numbers will start being read,
and they are structurally misleading in a way worth writing down before anyone
acts on them.
What the numbers say
First meaningful measurement (at v0.54.0):
| file |
line cover |
functions never executed |
synth-backend-aarch64/src/backend.rs |
41.6 % |
15 of 26 |
synth-backend-riscv/src/backend.rs |
43.7 % |
12 of 25 |
synth-synthesis/src/instruction_selector.rs |
42.7 % |
92 of 260 |
synth-backend-aarch64/src/selector.rs |
89.4 % |
20 of 186 |
Read naively that says roughly half the code generator is untested. It isn't.
Why the number is wrong (both directions)
The coverage job runs cargo llvm-cov --workspace --lcov — it instruments and
executes the Rust test suite. But the strongest evidence this project has is
the execution differentials: Python harnesses that compile fixtures and
compare results bit-for-bit against wasmtime under unicorn. Those:
- spawn
./target/release/synth as a separate process, resolved from the
SYNTH env var (scripts/repro/*.py), i.e. a binary outside the instrumented
build; and
- run in different CI jobs entirely (the
repro sweep-* jobs, the
trap-semantics oracle job, …), not under cargo llvm-cov at all.
An uninstrumented subprocess emits no profile data, so every one of those
executions is invisible to the coverage figure. backend.rs is largely ELF
emission and object writing — exercised end-to-end constantly, and almost
entirely through that invisible path.
So the figure understates real testing. It is also, for the same reason,
useless as a completeness instrument: it cannot tell you what the differentials
don't reach, which is the question actually worth asking.
Why this matters now rather than before
Before #908 the job was red for an unrelated reason, so nobody read it. Now it
is green and quotable. A green-but-meaningless number is worse than a red one —
that is the whole #890 lesson, and this is the same shape: an instrument whose
output does not mean what its name implies.
Options (not obviously one right answer)
- Point the differentials at the instrumented binary when running under
coverage (SYNTH → the llvm-cov-target build) and merge the profiles.
Highest fidelity; needs the sweeps to run inside the coverage job or to
export LLVM_PROFILE_FILE compatibly.
- Leave coverage measuring unit/integration tests only, and say so — in the
job name and in any doc that quotes it — so the number is honest about its
scope rather than implying whole-system coverage.
- Stop quoting a single percentage, and report the two populations
separately: Rust-test coverage, and differential coverage as a count of
executed checks (which the oracles already emit — e.g. #846 CHECKS=75/75,
662 float-boundary checks, 109 VFP rows).
My inclination is (2) + (3): the differentials already carry non-zero check
counts as their own evidence, and pretending one percentage summarizes both
populations is what created the confusion.
Related: #890 (the newly-wired sweeps assert exit status rather than per-script
check counts — the same "what does this number actually attest" question).
Filed on the back of #908, which fixed
Code Coveragebeing red repo-wide for atest binary-path bug. The job now works — so its numbers will start being read,
and they are structurally misleading in a way worth writing down before anyone
acts on them.
What the numbers say
First meaningful measurement (at v0.54.0):
synth-backend-aarch64/src/backend.rssynth-backend-riscv/src/backend.rssynth-synthesis/src/instruction_selector.rssynth-backend-aarch64/src/selector.rsRead naively that says roughly half the code generator is untested. It isn't.
Why the number is wrong (both directions)
The coverage job runs
cargo llvm-cov --workspace --lcov— it instruments andexecutes the Rust test suite. But the strongest evidence this project has is
the execution differentials: Python harnesses that compile fixtures and
compare results bit-for-bit against wasmtime under unicorn. Those:
./target/release/synthas a separate process, resolved from theSYNTHenv var (scripts/repro/*.py), i.e. a binary outside the instrumentedbuild; and
repro sweep-*jobs, thetrap-semantics oracle job, …), not under
cargo llvm-covat all.An uninstrumented subprocess emits no profile data, so every one of those
executions is invisible to the coverage figure.
backend.rsis largely ELFemission and object writing — exercised end-to-end constantly, and almost
entirely through that invisible path.
So the figure understates real testing. It is also, for the same reason,
useless as a completeness instrument: it cannot tell you what the differentials
don't reach, which is the question actually worth asking.
Why this matters now rather than before
Before #908 the job was red for an unrelated reason, so nobody read it. Now it
is green and quotable. A green-but-meaningless number is worse than a red one —
that is the whole #890 lesson, and this is the same shape: an instrument whose
output does not mean what its name implies.
Options (not obviously one right answer)
coverage (
SYNTH→ thellvm-cov-targetbuild) and merge the profiles.Highest fidelity; needs the sweeps to run inside the coverage job or to
export
LLVM_PROFILE_FILEcompatibly.job name and in any doc that quotes it — so the number is honest about its
scope rather than implying whole-system coverage.
separately: Rust-test coverage, and differential coverage as a count of
executed checks (which the oracles already emit — e.g.
#846 CHECKS=75/75,662 float-boundary checks, 109 VFP rows).
My inclination is (2) + (3): the differentials already carry non-zero check
counts as their own evidence, and pretending one percentage summarizes both
populations is what created the confusion.
Related: #890 (the newly-wired sweeps assert exit status rather than per-script
check counts — the same "what does this number actually attest" question).