Skip to content

laguna §5d: correct the soak counts, including a turn-success figure that was wrong at source - #20

Merged
TheTom merged 1 commit into
TheTom:mainfrom
Blackwellboy:soak-count-correction
Jul 29, 2026
Merged

laguna §5d: correct the soak counts, including a turn-success figure that was wrong at source#20
TheTom merged 1 commit into
TheTom:mainfrom
Blackwellboy:soak-count-correction

Conversation

@Blackwellboy

Copy link
Copy Markdown
Contributor

Drafted with AI assistance (Claude); reviewed and submitted by @Blackwellboy.

Counts only. No depth-collapse language is touched here; #19 owns that reconciliation and this PR deliberately does not overlap it.

All values below were re-derived from the published artifacts before editing, not transcribed from our earlier message.

Two errors, both ours

1. Checkpoint, not final cut. The ~389 sessions / ~2,947 turns figures come from an ~11.5h checkpoint of a run that finished at 409 sessions / 3,096 turns. Same single run, later cut. You already had the corrected turn count sitting in your own changelog, which is how the document disagreed with itself.

2. The 99.9% turn-success figure was wrong in our primary, and you inherited it in good faith. soak/logs/turns.jsonl has 3,099 lines. Three of those are kind: integrity_probe records, not turns: a different shape, status: 200, http_error: false, and the same three records as integrity_probes.jsonl. Every note we published described them as incomplete in-flight turns and derived 99.9% from them.

All 3,096 turn records returned HTTP 200. The turn log in soak_driver.py is written unconditionally with whatever status came back, so a failed turn would appear with a non-200 status. None does. Turn success is 100%.

Corrected at source first, so this PR cites a primary that agrees rather than one that contradicts it: laguna-s21-lab@4507826.

The precise distinctions

quantity value
sessions 409 (400 completed, 9 killed at the driver session-token cap)
lines in turns.jsonl 3,099
turn records 3,096, all HTTP 200
kind: integrity_probe records in the same file 3
turns with thinking fired 3
thinking rate 3 / 3,096 = 0.0969%, reasonably shown as ~0.1%

The 9 killed sessions are the same 9 incidents your 2026-07-25 entry already scopes as his driver cap firing by design. They were never subtracted from our headline, which is why 409 was reported as though all of them completed.

Six surfaces, not four

Our earlier four-line correction list was incomplete. It gave you §5d prose, the stability bullet, the changelog entry and the card, and it missed the thinking-rate denominator, ~3 of 2,944 turns. That line is the one where a wrong denominator actually moves a published rate, so it was the worst one to leave out.

A sixth surface turned up while making the edit and is corrected here too: the verdict: line in this guide own frontmatter also carried 99.9% turn success. It was in neither of our lists.

  1. models/laguna-s-2.1.md §5d prose: ~389 sessions and ~2,947 turns to 409 sessions and 3,096 turns
  2. models/laguna-s-2.1.md stability bullet: 2,944 of 2,947 turns succeeded (99.9%) to all 3,096 turns returned HTTP 200 (100%)
  3. models/laguna-s-2.1.md thinking rate: ~3 of 2,944 turns (~0.1%) to 3 of 3,096 turns (0.0969%, ~0.1%)
  4. models/laguna-s-2.1.md changelog 2026-07-24 entry: turn count and 99.9% to 100%
  5. cards/laguna-s-2.1.html: 12h / 2947 turns, 99.9% turn success to 12h / 3096 turns, 100% turn success
  6. models/laguna-s-2.1.md frontmatter verdict:: 99.9% turn success to 100%

Plus a dated 2026-07-29 changelog entry recording the correction and pointing at the primary, following your existing convention.

Checks

python3 scripts/check_guides.py passes: valid frontmatter, required keys, no em/en-dashes. git diff --check clean.

One line I did NOT touch, deliberately

The 2026-07-26 changelog entry says his soak sent enable_thinking: true on all 3,099 turns. Strictly that should be 3,096, since 3,099 is the line count and 3 of those lines are probes rather than turns. It sits inside an entry that #19 is reconciling, so it is yours to fold in whichever way suits that PR rather than something to fix from two directions at once. Flagging it so it is not missed.

Nothing about the thinking-gate, persona-gate or depth-collapse conclusions changes. Only counts.

…that was wrong at source

Two separate errors, both ours, both at source rather than in this guide.

1. The ~389-session / ~2,947-turn figures are an ~11.5h checkpoint of a run
   that finished at 409 sessions and 3,096 turns. Same single run, later cut.

2. The 99.9% turn-success figure was wrong in our own primary. turns.jsonl
   has 3,099 lines, but 3 of those are kind: integrity_probe records, not
   turns. All 3,096 turn records returned HTTP 200. The turn log is written
   unconditionally with whatever status came back, so a failed turn would
   carry a non-200 status, and none does. Turn success is 100%.

Also restated: 409 is the session count, of which 400 completed and 9 were
killed at the driver's session-token cap by design, and the thinking rate is
3 of 3,096 (0.0969%) rather than 3 of 2,944, unchanged at ~0.1%.

Six surfaces, not the four we first sent: §5d prose, the stability bullet,
the thinking-rate denominator, the changelog entry, the card, and the
verdict line in this guide's own frontmatter.

Depth-collapse language is untouched; PR TheTom#19 owns that reconciliation.

Primary correction: Blackwellboy/laguna-s21-lab@4507826
@TheTom
TheTom merged commit 9b33ca7 into TheTom:main Jul 29, 2026
1 check passed
TheTom added a commit that referenced this pull request Jul 29, 2026
…omposition mechanism

Per @Blackwellboy's review and @apollo-mg's confirmation on #19:
- card: the flat negative overshot the run; scoped to the retraction itself
  (n=19-27 fired/arm), and the guide's axis-2 lead now matches
- the 1,293/847 direct-answer comparators are n=4 cells; stated, with the
  larger-n cross-stack replication carrying the weight
- restored task-composition weighting as the second pooled-median mechanism
- patterns.md: the retraction was for cross-run comparison, not established
  as a serving-state effect (that lane ran one driver at conc=1)
- 5h: low-cache recovery was 1 of 4, not 0; channel-1 figures pinned to the
  stock upstream numbers so they cannot drift
- folded #20's flag: the 07-26 changelog said 3,099 turns, the turn count
  is 3,096 (3 lines are integrity probes)
- card PNG regenerated
TheTom added a commit that referenced this pull request Jul 29, 2026
…ne new confound (#19)

* reconcile with the #10 thread: two retractions at source, one new confound

The thread moved a long way in two days and several things I published are no
longer supported. Reconciling rather than letting the guide drift.

RETRACTED, and it was mine to carry: the reasoning-depth collapse (3,536 to
745 to 282) that I merged on 07-26 as axis 2 of the two-axis model. Its author
withdrew it after an in-run interleaved re-test at C7 (200 turns, five arms,
seeded shuffle) found depth among fired turns is flat: medians 858/806/721/
933/474, all pairwise Mann-Whitney p >= 0.13, with the tools arm carrying the
HIGHEST median. The published collapse was a cross-run read.

The real mechanism replaces it and is better: turns exiting via tool_call
carry short pre-call reasoning (462 and 136 tokens) against 1,293 and 847 for
turns that answer directly. Half of tool-arm turns exit to a call, so the
pooled median collapses arithmetically. The episode is truncated at the tool
boundary, not suppressed. Reproduced on a second stack at the opposite end of
the quant axis (UD-Q2_K_XL/llama.cpp/P100: 812 vs 2,954 chars, 3.6x, inside
the other stack's range). Depth columns in the curve tables now carry a
do-not-compare caveat.

NEW 5h, and the most portable thing in the guide: your serving layer can
decide the answer. Concurrent batched decoding flips near-ties at temp 0 (4
distinct outputs of 6 at 2 concurrent, 0 of 6 serialized); prefix-cache reuse
is on by default and changes output bytes; and setting the disable flag is
NOT sufficient, shown on a second stack with the flag explicitly off. It moves
verdicts rather than only bytes: on a third stack cache coverage tracked
finish_reason with zero overlap.

That lands on us. Our HumanEval+ A/B ran -np 4, which is exactly the
concurrency channel, at K=1 with no seed replication to absorb it. The +-2.5
point delta is now stated as directionally consistent with the other
controlled run, not as a precise magnitude.

Also folded:
- multi-turn firing is a harness property, not model attenuation. Stripped
  history 0/10 vs preserved 10/10 on identical transcripts; 0/56 vs 14/14 at
  larger n with a deterministic template-render basis. "Thinking ON" in a loop
  only means what it says if the harness resends prior reasoning.
- task shape at scale: a 100% codegen cell at n=492 fired 90.4%, so "coding
  suppresses thinking" is not supported. Consistent with the grid once
  conditioned, since its pooled 60-72% is held down by summarization at 0/10.
- apparatus costs score, not obviously capability, in the narrower form both
  the author and reviewer settled on: capability on the 106 routed samples is
  unmeasured, since they were selected by the model's own behaviour.

patterns.md gains the serving-state entry, which is not Laguna-specific, and
the cap-hit rule is corrected to "no extractable output", never finish_reason
alone, after one tester found a cap-hit that passed its tests twice.

Card and PNG regenerated to match.

* quickstart: the single-turn codegen question is resolved, not open

Three stacks now. The two that held temperature identical across arms both
came back flat-to-negative (-1.02 and -2.44 on HumanEval+); the only arm
showing a gain ran its two legs at different temperatures.

* apply review: scope the depth non-result, carry cell n, restore the composition mechanism

Per @Blackwellboy's review and @apollo-mg's confirmation on #19:
- card: the flat negative overshot the run; scoped to the retraction itself
  (n=19-27 fired/arm), and the guide's axis-2 lead now matches
- the 1,293/847 direct-answer comparators are n=4 cells; stated, with the
  larger-n cross-stack replication carrying the weight
- restored task-composition weighting as the second pooled-median mechanism
- patterns.md: the retraction was for cross-run comparison, not established
  as a serving-state effect (that lane ran one driver at conc=1)
- 5h: low-cache recovery was 1 of 4, not 0; channel-1 figures pinned to the
  stock upstream numbers so they cannot drift
- folded #20's flag: the 07-26 changelog said 3,099 turns, the turn count
  is 3,096 (3 lines are integrity probes)
- card PNG regenerated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants