docs(llms): the roster count the pages state, and a guard that derives it - #632
Open
yusuf-gundogdu wants to merge 1 commit into
Open
docs(llms): the roster count the pages state, and a guard that derives it#632yusuf-gundogdu wants to merge 1 commit into
yusuf-gundogdu wants to merge 1 commit into
Conversation
…s it `setup.md` and `methodology.md` still described a twenty-two model roster. Both were written when that was true and neither moved when five models landed and then a sixth: the pages read "Twenty-two models are supported", "Twenty-one of the twenty-two run locally", "One of the twenty-two is hosted" and "660 runs, and all 660 passed" against a document holding twenty-eight. `tests/unit/lib/agent/tools.test.ts` carried the same denominator in a comment about how many shipped models set `refusalExamples`. Two pull requests and a review passed over all of it, which is the part worth fixing rather than the numbers. Nothing anywhere compared the sentence to the file it describes, so `model-roster-docs.test.ts` now does: the count comes from `modelProfiles()` and the total from `count * 30`, the protocol those pages define. It pins the CLAIM rather than every number in the prose, and the first version of it did not. Asserting only that a page contained the current word left `setup.md` green when its opening sentence was reverted to twenty-two, because two later sentences in the same file still said twenty-eight and `includes` was satisfied by them. Each claim is captured by its own sentence now, and the run total is checked against every roster size between ten and sixty rather than against the totals this project happens to have had. Both halves were confirmed to bite by reverting the two sentences that leaked. A blanket ban on number-words would be wrong and is deliberately not what this does: these pages correctly say twenty-seven (the models that run locally), twenty-one (the models at the 90-second ceiling), Ten (the browser sweep) and thirty runs (one model's own sweep).
yusuf-gundogdu
force-pushed
the
docs/llms-roster-count
branch
from
September 7, 2026 14:56
bc63f80 to
3aaa447
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



docs/llms/setup.mdanddocs/llms/methodology.mdstill described a twenty-two model roster.Both were written when that was true, and neither moved when five models landed in #572 and a sixth
in #628. Against a document now holding twenty-eight, the pages read:
setup.mdsetup.mdsetup.mdmethodology.mdmethodology.mdmethodology.mdtests/unit/lib/agent/tools.test.tsTwo pull requests and a review passed over all of it, and that is the part worth fixing rather
than the seven numbers. Nothing anywhere compared the sentence to the file it describes.
The guard
tests/unit/lib/agent/model-roster-docs.test.tsnow does that comparison, derived on both sides:the count comes from
modelProfiles()and the total fromcount * 30, which is the protocol thosepages define — six surfaces, five consecutive runs each.
The first version of it did not bite, and that is recorded in the file. Asserting only that a
page contained the current word left
setup.mdgreen when its opening sentence was reverted totwenty-two, because two later sentences in the same file still said twenty-eight and
includeswassatisfied by them. Each claim is now captured by its own sentence and compared; the run total is
checked against every roster size between ten and sixty rather than against the totals this project
happens to have had.
Both halves were confirmed to bite by reverting the two sentences that actually leaked:
setup.mdopening sentence back to "Twenty-two models are supported"methodology.mdback to "660 runs, and all 660 passed"What it deliberately does not do
A blanket ban on number-words would fail on sentences that are correct. These pages legitimately
say twenty-seven (the models that run locally), twenty-one (the models at the 90-second
ceiling), Ten (the browser sweep) and thirty runs (one model's own sweep). A guard that
failed on those would be deleted within a week, so it pins the claims and the total instead.
The claim patterns are hand-listed because a sentence cannot be derived; the number in each is what
the test derives. A page whose phrasing changes fails loudly rather than silently stopping being
checked — which is the failure mode this file was written after.
Gates
format·lint·typecheck·knip·test(38 groups, 0 fail) ·build— all green.Coverage: 45992/45992 lines, 100.00%.