Skip to content

docs(llms): the roster count the pages state, and a guard that derives it - #632

Open
yusuf-gundogdu wants to merge 1 commit into
mainfrom
docs/llms-roster-count
Open

docs(llms): the roster count the pages state, and a guard that derives it#632
yusuf-gundogdu wants to merge 1 commit into
mainfrom
docs/llms-roster-count

Conversation

@yusuf-gundogdu

Copy link
Copy Markdown
Member

docs/llms/setup.md and docs/llms/methodology.md still described a twenty-two model roster.
Both were written when that was true, and neither moved when five models landed in #572 and a sixth
in #628. Against a document now holding twenty-eight, the pages read:

Page Said
setup.md "Twenty-two models are supported"
setup.md "Twenty-one of the twenty-two run locally"
setup.md "One of the twenty-two is hosted"
methodology.md "Twenty-two models, six surfaces, five runs: 660 runs, and all 660 passed."
methodology.md "The 660 runs above were opened over HTTP"
methodology.md "Ten, not twenty-two"
tests/unit/lib/agent/tools.test.ts "which two of twenty-two shipped models carry"

Two pull requests and a review passed over all of it, and that is the part worth fixing rather
than the seven numbers. Nothing anywhere compared the sentence to the file it describes.

The guard

tests/unit/lib/agent/model-roster-docs.test.ts now does that comparison, derived on both sides:
the count comes from modelProfiles() and the total from count * 30, which is the protocol those
pages define — six surfaces, five consecutive runs each.

The first version of it did not bite, and that is recorded in the file. Asserting only that a
page contained the current word left setup.md green when its opening sentence was reverted to
twenty-two, because two later sentences in the same file still said twenty-eight and includes was
satisfied by them. Each claim is now captured by its own sentence and compared; the run total is
checked against every roster size between ten and sixty rather than against the totals this project
happens to have had.

Both halves were confirmed to bite by reverting the two sentences that actually leaked:

Mutation Result
setup.md opening sentence back to "Twenty-two models are supported" 1 fail
methodology.md back to "660 runs, and all 660 passed" 1 fail
Both restored 2 pass

What it deliberately does not do

A blanket ban on number-words would fail on sentences that are correct. These pages legitimately
say twenty-seven (the models that run locally), twenty-one (the models at the 90-second
ceiling), Ten (the browser sweep) and thirty runs (one model's own sweep). A guard that
failed on those would be deleted within a week, so it pins the claims and the total instead.

The claim patterns are hand-listed because a sentence cannot be derived; the number in each is what
the test derives. A page whose phrasing changes fails loudly rather than silently stopping being
checked — which is the failure mode this file was written after.

Gates

format · lint · typecheck · knip · test (38 groups, 0 fail) · build — all green.
Coverage: 45992/45992 lines, 100.00%.

…s it

`setup.md` and `methodology.md` still described a twenty-two model roster. Both were written when
that was true and neither moved when five models landed and then a sixth: the pages read
"Twenty-two models are supported", "Twenty-one of the twenty-two run locally", "One of the
twenty-two is hosted" and "660 runs, and all 660 passed" against a document holding twenty-eight.
`tests/unit/lib/agent/tools.test.ts` carried the same denominator in a comment about how many
shipped models set `refusalExamples`.

Two pull requests and a review passed over all of it, which is the part worth fixing rather than
the numbers. Nothing anywhere compared the sentence to the file it describes, so
`model-roster-docs.test.ts` now does: the count comes from `modelProfiles()` and the total from
`count * 30`, the protocol those pages define.

It pins the CLAIM rather than every number in the prose, and the first version of it did not.
Asserting only that a page contained the current word left `setup.md` green when its opening
sentence was reverted to twenty-two, because two later sentences in the same file still said
twenty-eight and `includes` was satisfied by them. Each claim is captured by its own sentence now,
and the run total is checked against every roster size between ten and sixty rather than against
the totals this project happens to have had. Both halves were confirmed to bite by reverting the
two sentences that leaked.

A blanket ban on number-words would be wrong and is deliberately not what this does: these pages
correctly say twenty-seven (the models that run locally), twenty-one (the models at the 90-second
ceiling), Ten (the browser sweep) and thirty runs (one model's own sweep).
@sonarqubecloud

sonarqubecloud Bot commented Sep 7, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant