test: pin the score-scaling invariants, not one example - #34
Open
bradAGI wants to merge 2 commits into
Open
Conversation
The scanner is the whole product here — it resolves a release, verifies it, runs the engine, scales the score, and decides whether the build fails — and until now nothing checked any of it. A regression in the gate logic or in the jq that reads the engine's ScanResult would ship silently. The suite drives the real scan/trustabl-scan.sh. Each test builds an actual gzipped release tarball containing a stub engine and a real checksums.txt over it, then puts a stub curl on PATH that serves that directory by URL basename. The download, sha256 verification, extraction and invocation therefore all run unmodified; only the network and the engine binary are substituted. The fixtures are unmodified output from a real `trustabl scan` (engine v0.1.7), so the assertions pin the scanner against the ScanResult shape the engine actually emits rather than a hand-written approximation of it.
Readiness is the engine's overall_score — a float in [0,1] — scaled to an
integer percent, and risk is its complement. Those two lines decide what every
gate, every artifact and every downstream consumer sees, and until now they
were covered by a single example: 0.9588 -> 96.
One example cannot distinguish rounding from truncation, and cannot see a clamp
at all. Three tests replace it:
- scaling across the range, including both rounding directions and the
half-way boundaries (0.004 -> 0, 0.005 -> 1, 0.994 -> 99, 0.995 -> 100)
- the clamp at both ends, for inputs outside the engine's stated contract
- readiness + risk == 100 for every input, with readiness in range
Confirmed non-vacuous by mutation. Truncating instead of rounding half up:
not ok — readiness: expected '96', got '95'
Dropping the upper clamp:
not ok — score above 1 clamps to 100: expected '100', got '150'
No production change; scan/trustabl-scan.sh is untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Readiness is the engine's
overall_score— a float in[0,1]— scaled to an integer percent, and risk is its complement:Those two lines decide what every gate, every artifact and every downstream consumer sees. They were covered by exactly one example:
0.9588 → 96.One example cannot tell rounding from truncation —
0.9588truncates to 95 and rounds to 96, but plenty of values agree under both — and it cannot exercise a clamp at all.What this adds
0.004 → 0,0.005 → 1,0.994 → 99,0.995 → 100,1 → 100.[0,1]contract. The engine owns that contract, but a clamp that silently inverted or overflowed would publish a nonsense score.readiness + risk == 100for every input, with readiness asserted in range — the invariant, rather than a spot check.Confirmed non-vacuous
A test that never failed proves nothing, so both were mutation-checked against the real script.
Truncate instead of rounding half up:
Drop the upper clamp:
Restored:
all 18 test(s) passed.