Skip to content

Recompute the figures this page states and test the guards nothing watched - #35

Merged
PNX89 merged 1 commit into
mainfrom
claims-nothing-computes
Aug 30, 2026
Merged

Recompute the figures this page states and test the guards nothing watched#35
PNX89 merged 1 commit into
mainfrom
claims-nothing-computes

Conversation

@PNX89

@PNX89 PNX89 commented Aug 30, 2026

Copy link
Copy Markdown
Owner

What this changes

Twelve audit findings, all of one shape: a figure or a guard that nothing computed. The
repository exists to catch a number that was true when it was written and is false now, and it
had six of those on its own front page, three guards no test could break, and one selection rule
that made four of its questions false statements about the ONS release calendar.

The two that changed behaviour

The golden set asked whether the February 1988 CPI index had been published by September 2015.
_eligible filtered every series with first > coverage().first_vintage, the corpus-wide
earliest vintage of 2015-05-27. The four series were captured on four different days: the
quarterly pair opens on 2015-05-27 and the two monthly ones a fortnight into that October, so for
D7BT and MGSX the filter admitted the entire back-history, including months the ONS first printed
in the eighties. Four not-published questions followed, and each scored an agent wrong for
answering correctly:

answer_as_of(D7BT, 1988-02, 2015-09) -> NotPublished(...)
answer_as_of(MGSX, 1971-02, 2015-09) -> NotPublished(...)

The honesty predicate is now per series and is applied where the claim about a release calendar is
actually made, in _watched_arrive, rather than folded into the observation filter. The four
false questions are gone and the set is 52 rather than 56.

I deliberately did NOT narrow _eligible itself, which the finding suggested. Rebinding it to the
series' own opening vintage moves which observations MGSX contributes, and 32 recorded exchanges
against claude-sonnet-5 are keyed on the rendered question text. Re-recording is a paid act run
by hand from a keychain credential, and I have no authority to spend on it. Applying the predicate
at the point of harm removes exactly the false questions, leaves every recorded exchange valid,
and is the better separation anyway: choosing what to ask about and deciding whether one question
about it is honest are two jobs, and they were one filter.

latest_vintage ordered four series by a per-series counter. The query was select vintage from observations order by version desc limit 1. D7BT is on 131 and the quarterly pair on 46, so
this returns the last release of whichever series has had the most, which is the newest vintage
only while that series is also the last to publish. Replaying the corpus at every historical
cutoff, the two disagree at 135 of 296 of them. Deleting the rows after 2026-08-14, as a capture
four days earlier would have, moves the refusal boundary from 2026-08-12 back to 2026-07-21 and
turns every settled question in the set into a refusal, with a message that by design says nothing
about why. It is now max(vintage), which is the concept the docstring describes and the one
golden.build was already using three modules over.

The figures, each now with a producer

claim was is who computes it now
plan count the retrieval argument rests on 3 1 test_the_plan_count_the_documents_quote_is_the_one_this_query_produces
ADR 0004's measurement table three cells of 3 1 test_the_decision_records_measurement_table_is_still_the_measurement
lookup baseline accuracy 0.333 0.389 test_the_readme_baseline_table_states_the_score_that_baseline_gets
restatements of 2020 Q2 nine eight test_the_readme_restatement_count_counts_restatements_and_not_rows
CPI title lag after rebasing until 2018-12-19 until 2016-05-17 test_the_readme_base_year_lag_is_the_one_the_extract_records
null expectation 14.4 of 56 16.4 of 52 test_both_copies_of_the_null_expectation_are_the_measured_one
threshold ladder 21, 22, 23, 23, 24, 25 23, 24, 25, 25, 26, 27 test_the_ladder_is_the_one_the_decision_record_prints

The restatement one is the rows-versus-events confusion in the paragraph an interviewer reads
first: the quarter has nine rows, of which the first is a publication and eight are restatements.
The base-year one mattered more than its digits, because the paragraph is an argument about how
long a publisher's metadata can lie, and the basis column of the file committed beside it shows
the title catching up five versions later, not three years later.

Each comparison is against the sentence that carries the figure, never a search of the page for a
digit, and each one asserts the sentence is present before comparing, so a rewording fails loudly
rather than passing by matching nothing.

The guards that could be deleted

  • clear_in_flight's and response is null, the clause that stops the deliberate escape hatch
    from erasing a step whose paid effect already landed. The existing test only ever cleared the
    genuinely in-flight step, so the clause was never asked to refuse anything.
  • The token bucket's math.min(capacity, ...). The refill test ends at 30 seconds, where a
    0.2/s refill has earned six tokens against a capacity of five, so the cap was never binding.
  • barrier.GRANT_SQL. install was called with role=None in all three call sites, so the
    constant was formatted nowhere and the grant that ran was a separate f-string in
    retrieval.bootstrap. Its test checked three substrings, all of which a grant on every table
    in the schema satisfies.
  • _AS_OF's order by version desc, which the two longest comments in the repository defend.
    The only test that looked like it covered this hand-copied the SQL into the test body and
    asserted the module agreed with the copy, and every value assertion was about a period with no
    same-date ties.
  • docs/AS_OF_CONTRACT.md sent a reader to HOLDOUT_FROM in src/quizz/asof.py, a constant
    that file explicitly says was replaced by declared rows. The doc-drift machinery only ever
    read the README.

Every mutation, and what it did

Run through audit_mutate.py, which refuses to report unless it has proved the file changed and
changed back.

asof.py     order by version desc -> vintage desc          BITES  3 tests
asof.py     the as-of inequality  -> <= '9999'             BITES  4 tests
asof.py     max(vintage) -> order by version desc limit 1  BITES  the built-corpus test
golden.py   if _watched_arrive(...) -> if True             BITES
golden.py   _watched_arrive  >  ->  >=                     BITES  2 tests
durable.py  drop "and response is null"                    BITES
ratelimit.py drop math.min(capacity, ...)                  BITES  40 admitted against a cap of 5
barrier.py  GRANT_SQL -> grant on all tables in schema     BITES  unit and services both
barrier.py  if role: -> if False:                          BITES
README      Rows Removed by Filter: 1 -> 3                 BITES
README      0.389 -> 0.333                                 BITES
README      eight -> nine restatements                     BITES
README      2016-05-17 -> 2018-12-19                       BITES
README      52 questions -> 56 in one of the two places    BITES
ADR 0004    row level security cell 1 -> 3                 BITES
ADR 0006    ladder -> the stale one                        BITES
ADR 0006    how many of 52 -> 56 questions                 BITES
gate.py     16.36 of 52 -> 14.41 of 56                     BITES
AS_OF_CONTRACT  WINDOWS/barrier.py -> HOLDOUT_FROM/asof.py BITES

One survived and was left alone: rewording the first-release question's reason string breaks
nothing, because reason is documentation carried in the set and never reaches a model or a
score. It was corrected anyway, since for a period older than the extract "the first published
estimate" was not true.

What I checked that a reviewer cannot see from the diff

Reproduced every finding before touching it, and watched each new test fail against the old code
rather than believing the fix. The 332 same-date tie groups were counted from the corpus and the
count is pinned, so a re-capture that flattens them fails rather than leaving those cases covering
a distinction the data no longer holds. The plan count was measured six times across four full
container rebuilds and read 1 every time. All four ADR 0004 constructions were measured; three are
compared against a live plan here, and the fourth needs a publication-only index this suite
deliberately does not build, so asserting on the planner's choice of it would be asserting on a
tie break, which this file has already been burned by once.

Gates run locally, both suites: ruff check, ruff format --check, mypy, pytest -q
(172 passed), pytest -m services -q (17 passed, against the compose containers). Evidence
regenerated with scripts/capture_evidence.py and the card with the shared generator, so the
demo transcript, the test total and site/index.html are a live run rather than an edit.

…tched

Twelve audit findings of one shape: a figure or a guard with no producer.

The golden set asked whether the February 1988 CPI index and the December
2011 unemployment rate had been published by September 2015. Both had, by
decades and by years. The eligibility filter compared all four series against
the corpus-wide earliest vintage and the four were captured on four different
days, so the whole back-history of the two monthly series passed a filter
aimed at the quarterly pair. The honesty predicate is now per series and is
applied where the claim about a release calendar is made, in _watched_arrive.
Four false questions go and the set is 52 rather than 56.

latest_vintage ordered four series by a per-series version counter, so it
returned the last release of whichever series had had the most. It is
max(vintage) now. The two disagree at 135 of 296 historical cutoffs, and a
capture four days earlier would have refused every settled question in the
set.

retrieval.bootstrap now grants through barrier.GRANT_SQL rather than a second
copy of the statement, so the constant the barrier module publishes is the one
a deployment runs.

Six figures on the README, two decision records and two module docstrings were
recomputed and each now has a test that compares it against the sentence
carrying it: the plan count, the lookup baseline, the restatement count of
2020 Q2, the CPI title lag, the null expectation and the threshold ladder.
Three guards that no test could break are now covered directly, each proved by
mutating it away and watching the suite go red.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@PNX89
PNX89 merged commit 2014be7 into main Aug 30, 2026
4 checks passed
@PNX89
PNX89 deleted the claims-nothing-computes branch August 30, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant