Say the numbers the demo prints, not the ones it used to print - #34
Merged
Conversation
The page gave the trial-corrected pass mark as 23 of 56 rather than 21, and the random baseline as 14.4. Its own committed demo, on the same clone, prints 25 of 56 not 24, and 16.4. A fresh recapture reproduces the demo's figures exactly, so the evidence was right and the prose had been left behind. This is the defect this repository is about, in this repository. It answers a question as of a knowing-time because a number that was true when written and is false now is what costs people money, and three of its own headline figures were exactly that. The test is the point rather than the correction. It compares the figure in the demo's sentence with the figure in the page's sentence, and two earlier versions of it were wrong in opposite directions: searching the whole page for the digit passed with the wrong sentence in place, because a page this long contains any digit somewhere, and requiring the sentences to match verbatim failed on a baseline the page legitimately rephrases. Both mistakes are written down beside the pairs. Restoring either old figure now fails it by name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The page disagreed with its own committed evidence file about three headline figures.
docs/evidence/demo.txtprintsA fresh
capture_evidence.pyrun reproduces the demo's figures exactly, so the evidence was right and the prose had been left behind.This is the defect this repository exists to detect, in this repository. QUIZZ answers a question as of a knowing-time rather than as of now, because a number that was true when it was written and is false today is the failure that costs people money. Three of its own headline numbers were precisely that: computed once, typed into prose, and left there while the thing producing them moved. A reader who runs the demo, as the page invites them to, saw a different number from the sentence beside the invitation.
The test matters more than the correction, and it took three attempts:
Both wrong turns are recorded beside the code, because the first one is the same weak assertion this portfolio's audit keeps finding elsewhere.
Mutations: restoring 23 fails it; restoring 14.4 fails it with
demo prints 16.4 for the random baseline and the page says 14.4.