Recompute the figures this page states and test the guards nothing watched - #35
Merged
Conversation
…tched Twelve audit findings of one shape: a figure or a guard with no producer. The golden set asked whether the February 1988 CPI index and the December 2011 unemployment rate had been published by September 2015. Both had, by decades and by years. The eligibility filter compared all four series against the corpus-wide earliest vintage and the four were captured on four different days, so the whole back-history of the two monthly series passed a filter aimed at the quarterly pair. The honesty predicate is now per series and is applied where the claim about a release calendar is made, in _watched_arrive. Four false questions go and the set is 52 rather than 56. latest_vintage ordered four series by a per-series version counter, so it returned the last release of whichever series had had the most. It is max(vintage) now. The two disagree at 135 of 296 historical cutoffs, and a capture four days earlier would have refused every settled question in the set. retrieval.bootstrap now grants through barrier.GRANT_SQL rather than a second copy of the statement, so the constant the barrier module publishes is the one a deployment runs. Six figures on the README, two decision records and two module docstrings were recomputed and each now has a test that compares it against the sentence carrying it: the plan count, the lookup baseline, the restatement count of 2020 Q2, the CPI title lag, the null expectation and the threshold ladder. Three guards that no test could break are now covered directly, each proved by mutating it away and watching the suite go red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Twelve audit findings, all of one shape: a figure or a guard that nothing computed. The
repository exists to catch a number that was true when it was written and is false now, and it
had six of those on its own front page, three guards no test could break, and one selection rule
that made four of its questions false statements about the ONS release calendar.
The two that changed behaviour
The golden set asked whether the February 1988 CPI index had been published by September 2015.
_eligiblefiltered every series withfirst > coverage().first_vintage, the corpus-wideearliest vintage of 2015-05-27. The four series were captured on four different days: the
quarterly pair opens on 2015-05-27 and the two monthly ones a fortnight into that October, so for
D7BT and MGSX the filter admitted the entire back-history, including months the ONS first printed
in the eighties. Four not-published questions followed, and each scored an agent wrong for
answering correctly:
The honesty predicate is now per series and is applied where the claim about a release calendar is
actually made, in
_watched_arrive, rather than folded into the observation filter. The fourfalse questions are gone and the set is 52 rather than 56.
I deliberately did NOT narrow
_eligibleitself, which the finding suggested. Rebinding it to theseries' own opening vintage moves which observations MGSX contributes, and 32 recorded exchanges
against
claude-sonnet-5are keyed on the rendered question text. Re-recording is a paid act runby hand from a keychain credential, and I have no authority to spend on it. Applying the predicate
at the point of harm removes exactly the false questions, leaves every recorded exchange valid,
and is the better separation anyway: choosing what to ask about and deciding whether one question
about it is honest are two jobs, and they were one filter.
latest_vintageordered four series by a per-series counter. The query wasselect vintage from observations order by version desc limit 1. D7BT is on 131 and the quarterly pair on 46, sothis returns the last release of whichever series has had the most, which is the newest vintage
only while that series is also the last to publish. Replaying the corpus at every historical
cutoff, the two disagree at 135 of 296 of them. Deleting the rows after 2026-08-14, as a capture
four days earlier would have, moves the refusal boundary from 2026-08-12 back to 2026-07-21 and
turns every settled question in the set into a refusal, with a message that by design says nothing
about why. It is now
max(vintage), which is the concept the docstring describes and the onegolden.buildwas already using three modules over.The figures, each now with a producer
test_the_plan_count_the_documents_quote_is_the_one_this_query_producestest_the_decision_records_measurement_table_is_still_the_measurementtest_the_readme_baseline_table_states_the_score_that_baseline_getstest_the_readme_restatement_count_counts_restatements_and_not_rowstest_the_readme_base_year_lag_is_the_one_the_extract_recordstest_both_copies_of_the_null_expectation_are_the_measured_onetest_the_ladder_is_the_one_the_decision_record_printsThe restatement one is the rows-versus-events confusion in the paragraph an interviewer reads
first: the quarter has nine rows, of which the first is a publication and eight are restatements.
The base-year one mattered more than its digits, because the paragraph is an argument about how
long a publisher's metadata can lie, and the
basiscolumn of the file committed beside it showsthe title catching up five versions later, not three years later.
Each comparison is against the sentence that carries the figure, never a search of the page for a
digit, and each one asserts the sentence is present before comparing, so a rewording fails loudly
rather than passing by matching nothing.
The guards that could be deleted
clear_in_flight'sand response is null, the clause that stops the deliberate escape hatchfrom erasing a step whose paid effect already landed. The existing test only ever cleared the
genuinely in-flight step, so the clause was never asked to refuse anything.
math.min(capacity, ...). The refill test ends at 30 seconds, where a0.2/s refill has earned six tokens against a capacity of five, so the cap was never binding.
barrier.GRANT_SQL.installwas called withrole=Nonein all three call sites, so theconstant was formatted nowhere and the grant that ran was a separate f-string in
retrieval.bootstrap. Its test checked three substrings, all of which a grant on every tablein the schema satisfies.
_AS_OF'sorder by version desc, which the two longest comments in the repository defend.The only test that looked like it covered this hand-copied the SQL into the test body and
asserted the module agreed with the copy, and every value assertion was about a period with no
same-date ties.
docs/AS_OF_CONTRACT.mdsent a reader toHOLDOUT_FROMinsrc/quizz/asof.py, a constantthat file explicitly says was replaced by declared rows. The doc-drift machinery only ever
read the README.
Every mutation, and what it did
Run through
audit_mutate.py, which refuses to report unless it has proved the file changed andchanged back.
One survived and was left alone: rewording the
first-releasequestion'sreasonstring breaksnothing, because
reasonis documentation carried in the set and never reaches a model or ascore. It was corrected anyway, since for a period older than the extract "the first published
estimate" was not true.
What I checked that a reviewer cannot see from the diff
Reproduced every finding before touching it, and watched each new test fail against the old code
rather than believing the fix. The 332 same-date tie groups were counted from the corpus and the
count is pinned, so a re-capture that flattens them fails rather than leaving those cases covering
a distinction the data no longer holds. The plan count was measured six times across four full
container rebuilds and read 1 every time. All four ADR 0004 constructions were measured; three are
compared against a live plan here, and the fourth needs a publication-only index this suite
deliberately does not build, so asserting on the planner's choice of it would be asserting on a
tie break, which this file has already been burned by once.
Gates run locally, both suites:
ruff check,ruff format --check,mypy,pytest -q(172 passed),
pytest -m services -q(17 passed, against the compose containers). Evidenceregenerated with
scripts/capture_evidence.pyand the card with the shared generator, so thedemo transcript, the test total and
site/index.htmlare a live run rather than an edit.