The pair scorer reads JSON answers, and the flywheel repair is measured - #195
Merged
Merged
Conversation
parsePredictedPairs matched parentheses only, so a run answering FINAL_ANSWER as a JSON list of lists -- the notation json.dumps produces -- scored 0.0 with every pair present and none counted. Observed on a live run: the answer was correct and the report called it a reasoning failure. Both bracket forms are now accepted; the pair shape still has to match, so no new class of text is admitted, and the 15 existing scoring tests stay green. The defect is worth more than the run that surfaced it. It had nothing to do with the module under test and would have misread any future benchmark run answering in valid JSON. The measured result for the spatial-flywheel provenance repair (#194) lands in PROVENANCE_CITATION_AB_REPORT.md rather than in a pull request body, which is rule 12's fourth resting place. One cold OOLONG-Pairs query per arm, identical graph, arms differing in one file: the spatial-flywheel addendum before and after the repair. Control refused 24 of 24 cited hashes and cached 0 insights. Treatment refused none and cached 220. Rescored with the fixed parser, F1 goes 0.800 to 1.000. The control discriminated, so this is a finding rather than noise. Two figures, per rule 7: estimate $0.25 printed before the first paid call, actual $0.4591. The overage is a blind first arm -- it returned F1 1.0 with zero sub-LLM calls, because the freshly ingested corpus carried all 220 categories and the module short-circuits when q.category is set, so nothing classified, nothing wrote, and the guard never fired. Its output was noise and the record says so. Standing is stated rather than implied: n=1 per arm establishes that the pre-repair write is refused and the repaired one is not, on the same instrument, and does not establish a distribution or a claim about answer quality, where one sample separates 0.800 from 1.000 and cannot say why.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #194. Two files: a scorer defect that misread correct answers, and the measured result for the provenance repair that PR shipped.
The scorer defect
parsePredictedPairsmatched parenthesised tuples only. A run answering— a JSON list of lists, the notation
json.dumpsproduces — matched nothing and scored 0.0, with every pair present and none counted.This was observed live: a benchmark arm was recorded as F1 0.000 with 0 predicted pairs, and the answer was in fact perfect. Rescoring the same logged answer with the fix gives 6/6 and F1 1.000.
Any run answering in valid JSON was being scored zero and reported as a reasoning failure. Both bracket forms are now accepted; the pair shape still has to match, so no new class of text is admitted. The 15 existing scoring tests stay green.
The defect is worth more than the run that surfaced it — it had nothing to do with what was under test and would have misread any future benchmark run.
The measurement
#194 repaired the default
spatial-flywheelmodule, which prescribed a provenance write the default guard refuses. That claim is now measured rather than traced, and the result lands in the report that owns it rather than in a PR body.One cold OOLONG-Pairs query per arm, identical graph — 220 questions, 0 categories, no cached insights — arms differing in one file.
The control discriminated, so this is a finding rather than noise.
Two figures, per rule 7
Estimate printed before the first paid call: $0.25. Actual: $0.4591.
The overage is a blind first arm. It returned F1 1.0 with zero sub-LLM calls, because the freshly ingested corpus carried all 220 categories and the module short-circuits when
q.categoryis set — nothing classified, nothing wrote, the guard never fired. Its output was noise and the record says so.Standing
n=1 per arm. What is established is that the pre-repair write is refused and the repaired one is not, on the same instrument. Not a distribution, and no claim about answer quality — one sample separates 0.800 from 1.000 and cannot say why.
Run on an isolated stack (
trellis-tests-approved, ports 5443/7697/6389), kept clear of the containers another worktree had running on the defaults.