laguna §5d: correct the soak counts, including a turn-success figure that was wrong at source - #20
Merged
Conversation
…that was wrong at source Two separate errors, both ours, both at source rather than in this guide. 1. The ~389-session / ~2,947-turn figures are an ~11.5h checkpoint of a run that finished at 409 sessions and 3,096 turns. Same single run, later cut. 2. The 99.9% turn-success figure was wrong in our own primary. turns.jsonl has 3,099 lines, but 3 of those are kind: integrity_probe records, not turns. All 3,096 turn records returned HTTP 200. The turn log is written unconditionally with whatever status came back, so a failed turn would carry a non-200 status, and none does. Turn success is 100%. Also restated: 409 is the session count, of which 400 completed and 9 were killed at the driver's session-token cap by design, and the thinking rate is 3 of 3,096 (0.0969%) rather than 3 of 2,944, unchanged at ~0.1%. Six surfaces, not the four we first sent: §5d prose, the stability bullet, the thinking-rate denominator, the changelog entry, the card, and the verdict line in this guide's own frontmatter. Depth-collapse language is untouched; PR TheTom#19 owns that reconciliation. Primary correction: Blackwellboy/laguna-s21-lab@4507826
TheTom
added a commit
that referenced
this pull request
Jul 29, 2026
…omposition mechanism Per @Blackwellboy's review and @apollo-mg's confirmation on #19: - card: the flat negative overshot the run; scoped to the retraction itself (n=19-27 fired/arm), and the guide's axis-2 lead now matches - the 1,293/847 direct-answer comparators are n=4 cells; stated, with the larger-n cross-stack replication carrying the weight - restored task-composition weighting as the second pooled-median mechanism - patterns.md: the retraction was for cross-run comparison, not established as a serving-state effect (that lane ran one driver at conc=1) - 5h: low-cache recovery was 1 of 4, not 0; channel-1 figures pinned to the stock upstream numbers so they cannot drift - folded #20's flag: the 07-26 changelog said 3,099 turns, the turn count is 3,096 (3 lines are integrity probes) - card PNG regenerated
TheTom
added a commit
that referenced
this pull request
Jul 29, 2026
…ne new confound (#19) * reconcile with the #10 thread: two retractions at source, one new confound The thread moved a long way in two days and several things I published are no longer supported. Reconciling rather than letting the guide drift. RETRACTED, and it was mine to carry: the reasoning-depth collapse (3,536 to 745 to 282) that I merged on 07-26 as axis 2 of the two-axis model. Its author withdrew it after an in-run interleaved re-test at C7 (200 turns, five arms, seeded shuffle) found depth among fired turns is flat: medians 858/806/721/ 933/474, all pairwise Mann-Whitney p >= 0.13, with the tools arm carrying the HIGHEST median. The published collapse was a cross-run read. The real mechanism replaces it and is better: turns exiting via tool_call carry short pre-call reasoning (462 and 136 tokens) against 1,293 and 847 for turns that answer directly. Half of tool-arm turns exit to a call, so the pooled median collapses arithmetically. The episode is truncated at the tool boundary, not suppressed. Reproduced on a second stack at the opposite end of the quant axis (UD-Q2_K_XL/llama.cpp/P100: 812 vs 2,954 chars, 3.6x, inside the other stack's range). Depth columns in the curve tables now carry a do-not-compare caveat. NEW 5h, and the most portable thing in the guide: your serving layer can decide the answer. Concurrent batched decoding flips near-ties at temp 0 (4 distinct outputs of 6 at 2 concurrent, 0 of 6 serialized); prefix-cache reuse is on by default and changes output bytes; and setting the disable flag is NOT sufficient, shown on a second stack with the flag explicitly off. It moves verdicts rather than only bytes: on a third stack cache coverage tracked finish_reason with zero overlap. That lands on us. Our HumanEval+ A/B ran -np 4, which is exactly the concurrency channel, at K=1 with no seed replication to absorb it. The +-2.5 point delta is now stated as directionally consistent with the other controlled run, not as a precise magnitude. Also folded: - multi-turn firing is a harness property, not model attenuation. Stripped history 0/10 vs preserved 10/10 on identical transcripts; 0/56 vs 14/14 at larger n with a deterministic template-render basis. "Thinking ON" in a loop only means what it says if the harness resends prior reasoning. - task shape at scale: a 100% codegen cell at n=492 fired 90.4%, so "coding suppresses thinking" is not supported. Consistent with the grid once conditioned, since its pooled 60-72% is held down by summarization at 0/10. - apparatus costs score, not obviously capability, in the narrower form both the author and reviewer settled on: capability on the 106 routed samples is unmeasured, since they were selected by the model's own behaviour. patterns.md gains the serving-state entry, which is not Laguna-specific, and the cap-hit rule is corrected to "no extractable output", never finish_reason alone, after one tester found a cap-hit that passed its tests twice. Card and PNG regenerated to match. * quickstart: the single-turn codegen question is resolved, not open Three stacks now. The two that held temperature identical across arms both came back flat-to-negative (-1.02 and -2.44 on HumanEval+); the only arm showing a gain ran its two legs at different temperatures. * apply review: scope the depth non-result, carry cell n, restore the composition mechanism Per @Blackwellboy's review and @apollo-mg's confirmation on #19: - card: the flat negative overshot the run; scoped to the retraction itself (n=19-27 fired/arm), and the guide's axis-2 lead now matches - the 1,293/847 direct-answer comparators are n=4 cells; stated, with the larger-n cross-stack replication carrying the weight - restored task-composition weighting as the second pooled-median mechanism - patterns.md: the retraction was for cross-run comparison, not established as a serving-state effect (that lane ran one driver at conc=1) - 5h: low-cache recovery was 1 of 4, not 0; channel-1 figures pinned to the stock upstream numbers so they cannot drift - folded #20's flag: the 07-26 changelog said 3,099 turns, the turn count is 3,096 (3 lines are integrity probes) - card PNG regenerated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Drafted with AI assistance (Claude); reviewed and submitted by @Blackwellboy.
Counts only. No depth-collapse language is touched here; #19 owns that reconciliation and this PR deliberately does not overlap it.
All values below were re-derived from the published artifacts before editing, not transcribed from our earlier message.
Two errors, both ours
1. Checkpoint, not final cut. The
~389 sessions / ~2,947 turnsfigures come from an ~11.5h checkpoint of a run that finished at 409 sessions / 3,096 turns. Same single run, later cut. You already had the corrected turn count sitting in your own changelog, which is how the document disagreed with itself.2. The 99.9% turn-success figure was wrong in our primary, and you inherited it in good faith.
soak/logs/turns.jsonlhas 3,099 lines. Three of those arekind: integrity_proberecords, not turns: a different shape,status: 200,http_error: false, and the same three records asintegrity_probes.jsonl. Every note we published described them as incomplete in-flight turns and derived 99.9% from them.All 3,096 turn records returned HTTP 200. The turn log in
soak_driver.pyis written unconditionally with whatever status came back, so a failed turn would appear with a non-200 status. None does. Turn success is 100%.Corrected at source first, so this PR cites a primary that agrees rather than one that contradicts it: laguna-s21-lab@4507826.
The precise distinctions
turns.jsonlkind: integrity_proberecords in the same fileThe 9 killed sessions are the same 9 incidents your 2026-07-25 entry already scopes as his driver cap firing by design. They were never subtracted from our headline, which is why 409 was reported as though all of them completed.
Six surfaces, not four
Our earlier four-line correction list was incomplete. It gave you §5d prose, the stability bullet, the changelog entry and the card, and it missed the thinking-rate denominator,
~3 of 2,944 turns. That line is the one where a wrong denominator actually moves a published rate, so it was the worst one to leave out.A sixth surface turned up while making the edit and is corrected here too: the
verdict:line in this guide own frontmatter also carried99.9% turn success. It was in neither of our lists.models/laguna-s-2.1.md§5d prose:~389 sessions and ~2,947 turnsto409 sessions and 3,096 turnsmodels/laguna-s-2.1.mdstability bullet:2,944 of 2,947 turns succeeded (99.9%)toall 3,096 turns returned HTTP 200 (100%)models/laguna-s-2.1.mdthinking rate:~3 of 2,944 turns (~0.1%)to3 of 3,096 turns (0.0969%, ~0.1%)models/laguna-s-2.1.mdchangelog2026-07-24entry: turn count and99.9%to100%cards/laguna-s-2.1.html:12h / 2947 turns,99.9% turn successto12h / 3096 turns,100% turn successmodels/laguna-s-2.1.mdfrontmatterverdict::99.9% turn successto100%Plus a dated
2026-07-29changelog entry recording the correction and pointing at the primary, following your existing convention.Checks
python3 scripts/check_guides.pypasses: valid frontmatter, required keys, no em/en-dashes.git diff --checkclean.One line I did NOT touch, deliberately
The
2026-07-26changelog entry says his soaksent enable_thinking: true on all 3,099 turns. Strictly that should be 3,096, since 3,099 is the line count and 3 of those lines are probes rather than turns. It sits inside an entry that #19 is reconciling, so it is yours to fold in whichever way suits that PR rather than something to fix from two directions at once. Flagging it so it is not missed.Nothing about the thinking-gate, persona-gate or depth-collapse conclusions changes. Only counts.