Skip to content

Say the numbers the demo prints, not the ones it used to print - #34

Merged
PNX89 merged 1 commit into
mainfrom
the-page-disagreed-with-its-own-evidence
Aug 30, 2026
Merged

Say the numbers the demo prints, not the ones it used to print#34
PNX89 merged 1 commit into
mainfrom
the-page-disagreed-with-its-own-evidence

Conversation

@PNX89

@PNX89 PNX89 commented Aug 30, 2026

Copy link
Copy Markdown
Owner

The page disagreed with its own committed evidence file about three headline figures.

README said docs/evidence/demo.txt prints
trial-corrected pass mark 23 of 56, rather than 21 25 of 56, not 24
random-baseline score 14.4 of 56 16.4 of 56

A fresh capture_evidence.py run reproduces the demo's figures exactly, so the evidence was right and the prose had been left behind.

This is the defect this repository exists to detect, in this repository. QUIZZ answers a question as of a knowing-time rather than as of now, because a number that was true when it was written and is false today is the failure that costs people money. Three of its own headline numbers were precisely that: computed once, typed into prose, and left there while the thing producing them moved. A reader who runs the demo, as the page invites them to, saw a different number from the sentence beside the invitation.

The test matters more than the correction, and it took three attempts:

  1. Search the whole page for each digit from the demo's sentence. Too weak. Putting the wrong sentence back left it green, because a page this long contains a "25" somewhere regardless. Searching a document for a digit is not a comparison.
  2. Require the demo's sentence to appear verbatim. Too strong. The page legitimately rephrases the baseline claim in its own words, so a correct page failed.
  3. Compare the figure in the demo's sentence with the figure in the page's corresponding sentence. The page may say it however it likes, and it may not say a different number.

Both wrong turns are recorded beside the code, because the first one is the same weak assertion this portfolio's audit keeps finding elsewhere.

Mutations: restoring 23 fails it; restoring 14.4 fails it with demo prints 16.4 for the random baseline and the page says 14.4.

The page gave the trial-corrected pass mark as 23 of 56 rather than 21, and the random baseline
as 14.4. Its own committed demo, on the same clone, prints 25 of 56 not 24, and 16.4. A fresh
recapture reproduces the demo's figures exactly, so the evidence was right and the prose had been
left behind.

This is the defect this repository is about, in this repository. It answers a question as of a
knowing-time because a number that was true when written and is false now is what costs people
money, and three of its own headline figures were exactly that.

The test is the point rather than the correction. It compares the figure in the demo's sentence
with the figure in the page's sentence, and two earlier versions of it were wrong in opposite
directions: searching the whole page for the digit passed with the wrong sentence in place,
because a page this long contains any digit somewhere, and requiring the sentences to match
verbatim failed on a baseline the page legitimately rephrases. Both mistakes are written down
beside the pairs. Restoring either old figure now fails it by name.
@PNX89
PNX89 merged commit 0e5cb39 into main Aug 30, 2026
4 checks passed
@PNX89
PNX89 deleted the the-page-disagreed-with-its-own-evidence branch August 30, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant