fix(agent): a stop that read nothing is answered for whoever stopped - #635
Merged
Conversation
The drive holds a sentence written for exactly one ending — "Read it yourself. Call inspect_schema for the tables and their columns, and inspect_plan for how a statement will run" — and sent it only to a model whose profile asked for it. One profile of twenty-eight does. A model nobody has measured has no profile at all, so the model most in need of the sentence was the one guaranteed not to receive it. Measured across the sweep behind 0.14.1: three hundred runs ended `model-stopped` with `no-report`, and half of them — a hundred and fifty — had called no tool at all. Ninety-four of those hundred and fifty were ended without the drive saying anything: `mistral-small3.1:24b` 51, `granite3.3:8b` 41, `mistral:7b` 40, `gpt-oss:20b` 18, none of them on the shipped roster. A hundred and ten of the hundred and fifty were on `investigation`, the first surface a model meets. The gate's own comment had already argued the cost away. It fires only where `anyToolCalled` is false, and `compose_report` is one of the tools that counts, so a run reaching it composed no report and has already earned `no-report` — "the turn cannot cost a pass". A bound that cannot protect a passing run is not protecting anything, and what this one withheld was a sentence on a run already lost. So the gate now reads `answersUnreadStop` rather than `retriesUnreadStop`. The two differ on one thing: a profile stating `false` is a measurement and is obeyed, while no profile at all is the absence of a measurement and is answered with the turn. This is the same split `planStatementAsksFor` made beside `planStatementRetriesFor`, and the fourth setting found in this shape. All twenty-eight shipped entries state the field, so no shipped model's turn count moves. Nineteen existing tests turned red on it, every one of them a run that stops having called nothing — which is the change reaching exactly what it was aimed at. Each gets the extra scripted turn the drive now spends; `tests/evals/real-model.ts` holds the shared warm-up that accounted for several. The pair that pins the behaviour lives beside the tests for the gate it belongs to, and both halves were confirmed to bite by mutation: reading an absent profile as `false` again fails one, ignoring a stated `false` fails the other. `docs/llms/model-tuning.md` gains the section this makes necessary — the last column of the settings table is the compiled default, and two settings answer differently for a model with no entry — and `docs/AGENT.md`, `docs/BACKLOG.md` B65 and the notes on `profile.ts` and `models/index.ts` are corrected where they said the absent case resolves to the defaults.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



The drive holds a sentence written for exactly one ending:
It was sent only to a model whose profile asked for it. One profile of twenty-eight does. A
model nobody has measured has no profile at all, so the model most in need of the sentence was the
one guaranteed not to receive it.
What the ledgers say
Measured across the sweep behind 0.14.1:
model-stoppedwithno-reportBy model:
mistral-small3.1:24b51 ·granite3.3:8b41 ·mistral:7b40 ·gpt-oss:20b18.None of the four is on the shipped roster — they are exactly the unmeasured models the sentence
could not reach. A hundred and ten of the hundred and fifty were on
investigation, the firstsurface a model meets.
Why the bound was protecting nothing
The gate's own comment had already argued the cost away. It fires only where
anyToolCalledisfalse, and
compose_reportis one of the tools that counts, so a run reaching it composed no reportand has already earned
no-report— in the comment's words, "the turn cannot cost a pass."A bound that cannot protect a passing run is not protecting anything. What this one withheld was a
sentence on a run already lost.
The change
investigation.tsnow readsanswersUnreadStoprather thanretriesUnreadStop. The two differ onone thing:
retryUnreadStop: trueretryUnreadStop: falseThis is the same split
planStatementAsksFormade besideplanStatementRetriesFor, and the fourthsetting found in this shape. All twenty-eight shipped entries state the field, so no shipped
model's turn count moves — verified against
measured-profiles.jsonrather than assumed.Nineteen red tests, which is the point
Nineteen existing tests turned red, and every one of them is a run that stops having called
nothing — the change reaching exactly what it was aimed at rather than spilling past it. Each gets
the extra scripted turn the drive now spends.
tests/evals/real-model.tsholds the shared warm-upthat accounted for several of them at once.
The pair that pins the new behaviour lives beside the tests for the gate it belongs to, replacing
the one that asserted the old contract. Both halves were confirmed to bite:
falseagainfalseignoredDocs
docs/llms/model-tuning.mdgains the section this makes necessary: the last column of the settingstable is the compiled default, and two settings answer differently for a model with no entry at
all.
docs/AGENT.md,docs/BACKLOG.mdB65 and the notes onprofile.tsandmodels/index.tsarecorrected where they said the absent case resolves to the defaults.
Gates
Every step of the
Lint, Typecheck and Buildjob, in order:chart:check(strict) ·channels:showcase:check·readme:check·security:check·format·lint·typecheck·knip·build·build:lib·attw— all green.test:coverageand plaintestboth exit 0.Coverage: 45995/45995 lines, 100.00%.
Rebased onto
mainafter #633 merged; no file here is touched by any other open PR.