diff --git a/docs/llms/README.md b/docs/llms/README.md index 3a29bbc9..002be726 100644 --- a/docs/llms/README.md +++ b/docs/llms/README.md @@ -8,11 +8,11 @@ writes beautiful prose about a database and never calls a tool answers nothing h question these pages answer is not what a model knows but what it DOES on a run, and every figure comes from a run whose ledger is on disk. -**Twenty-seven models are supported.** Each cleared all six surfaces — Investigate, Optimize, Assess, -Operate, Analyze and Plan — five consecutive times: 810 runs, 810 passes. +**Twenty-eight models are supported.** Each cleared all six surfaces — Investigate, Optimize, Assess, +Operate, Analyze and Plan — five consecutive times: 840 runs, 840 passes. -Twenty-one of the twenty-seven cleared them at the 90-second per-turn limit the product ships. Six carry a -150-second limit of their own — `qwen3.5:9b`, `gemma4:12b`, `nemotron-3.5-lightning:30b`, +Twenty-one of the twenty-eight cleared them at the 90-second per-turn limit the product ships. Seven carry a +150-second limit of their own — `qwen3.5:9b`, `gemma4:12b`, `gemma4:31b`, `nemotron-3.5-lightning:30b`, `nemotron-3-nano:30b`, `muse-glimmer:latest` and `qwen3.6:27b` — because one turn of theirs does not fit inside 90 while every other surface does. The limit stays 90 for every other model, and each page says what its model needed. @@ -45,12 +45,13 @@ each page says what its model needed. | [`qwen3:4b`](qwen/qwen3.md) | Ollama | 2.5 GB | 72s | 160s | | [`qwen3.8:latest`](qwen/qwen3.8.md) | Ollama | 19 GB | 72s | 195s | | [`qwen3.6:27b`](qwen/qwen3.6.md) | Ollama | 17 GB | 82s | 252s | +| [`gemma4:31b`](gemma/gemma4.md) | Ollama | 19 GB | 107s | 398s | | [`muse-glimmer:latest`](muse/muse-glimmer.md) | Ollama | 18 GB | 115s | 285s | The durations are from one machine and are comparable with each other rather than portable: every figure was taken the same way, on the same database, through the same six surfaces. What they are -for is choosing between these twenty-seven — the fastest reaches the same verdicts as the slowest in -a nineteenth of the time. +for is choosing between these twenty-eight — the fastest reaches the same verdicts as the slowest in +an eighteenth of the time. One page per model version. Sizes are rows inside it, because `ollama pull qwen3:4b` is how a size is chosen and because the interesting fact is usually the difference between two sizes of @@ -84,7 +85,7 @@ pass says nothing at all about the fifth. | If you want | Take | Why | | --- | --- | --- | -| the fastest local model | [`granite4.1:8b`](granite/granite4.1.md) | 11s median, 5.3 GB, and it clears everything | +| the fastest local model | [`qwen2.5:7b`](qwen/qwen2.5.md) | 6s median, 4.7 GB, and nothing it did on thirty runs took longer than 21s | | the smallest download | [`qwen3:4b`](qwen/qwen3.md) | 2.5 GB — and not the fastest: size buys memory, not speed | | the steadiest | [`cogito:14b`](cogito/cogito.md) | 20s is the shortest slowest-run on the roster, and under twice its own median | | no local hardware at all | [`gemini-3.5-flash-lite`](gemini/gemini-3.5.md) | the one hosted model, 10s median, and it needs a key | diff --git a/docs/llms/gemma/gemma4.md b/docs/llms/gemma/gemma4.md index 922b9346..d9698d20 100644 --- a/docs/llms/gemma/gemma4.md +++ b/docs/llms/gemma/gemma4.md @@ -1,11 +1,12 @@ # gemma4 -`ollama pull gemma4:` · sizes supported: 12b, 26b +`ollama pull gemma4:` · sizes supported: 12b, 26b, 31b Every size listed here runs **all six agent surfaces**, five consecutive times each: 30 of 30 runs. The 26b is the size that carries this family, and the one whose ledger taught this project what an empty completion looks like. The 12b is the smaller, faster one, and it is the model that -made the product learn to tell an agent turn to stop thinking. +made the product learn to tell an agent turn to stop thinking. The 31b is the largest, the slowest +model on the whole roster, and the one that shows what this family's growth actually buys. ## What it does, and how long it takes @@ -15,10 +16,17 @@ Seconds are the median of the runs that passed, per surface. | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | gemma4:12b | 7.6 GB | 4s | 24s | 21s | 14s | 11s | 2s | **11s** | 1:03 | | gemma4:26b | 16 GB | 15s | 1:02 | 31s | 26s | 46s | 36s | **36s** | 1:32 | +| gemma4:31b | 19 GB | 26s | 6:18 | 3:34 | 1:33 | 2:12 | 3s | **1:47** | 6:38 | Every cell is 5/5, so the table says how long rather than whether. The 12b is three times the 26b's speed at half its size, and it did not start out that way — see its note below. +**Size buys nothing here, and the 31b is the evidence.** Three more gigabytes than the 26b cost it +three times the median and six minutes on one cell, and every verdict it reaches the 12b reaches +too, in a tenth of the time. Its optimize cell alone runs longer than a whole 12b sweep. Take it +only if this exact model is what you have to run; otherwise the 12b is the same family answering +the same six questions. + ## What it needs that the defaults do not give it ### `gemma4:12b` @@ -41,6 +49,28 @@ the clock while working rather than while thinking. their length: analyze 102s to 11s, assess 118s to 21s, operate 75s to 14s, investigate 9s to 4s. It arrived at 2/6 and 15/30 and leaves at 6/6 and 30/30, three times faster than the 26b. +### `gemma4:31b` + +**no reasoning on its plan turn.** That cell read 0/5 at the defaults, losing to the clock rather +than to the plan — the model spends the turn thinking and the deliverable never arrives. Quiet, it +answers in three seconds, the fastest plan turn in this family, and the cell locked on the second +attempt. The switch reaches plan turns only (`suppressesPlanReasoning` is read where the mode is +not `agent`, in [`investigation.ts`](../../../src/lib/agent/investigation.ts)), so it is not what +carried the five agent surfaces. + +**a 150-second turn**, for optimize and only because of optimize: a passing optimize run takes six +minutes against a 26-second investigation, and at the compiled 90-second ceiling the cell would not +close. + +Those two were earned separately, which left five cells locked at the 90-second ceiling while the +profile that ships carries 150. Raising a per-call ceiling is not obviously free — a turn cut at 90 +is re-asked, while a turn allowed 150 spends that from the run's own deadline instead — so the five +were read again under the pair that ships: twenty-five runs, every surface 5/5. The figures in the +table above are from those runs. + +It does **not** carry `suppressAgentReasoning`, which its 12b sibling does. That setting was +measured for an illness this size does not have, and a setting a model did not earn is a guess. + ### `gemma4:26b` **call ceiling of 10.** It reads more thoroughly than a run has room for: eleven calls in, with nothing left to spend a twelfth on. diff --git a/scripts/agent-model-e2e.sh b/scripts/agent-model-e2e.sh index 8f176e1e..8cec2391 100755 --- a/scripts/agent-model-e2e.sh +++ b/scripts/agent-model-e2e.sh @@ -19,7 +19,7 @@ MODELS=("$@") if [ ${#MODELS[@]} -eq 0 ]; then # Fastest first, so the run reports something early, and the hosted model last because it needs # its own environment file swapped in rather than a line rewritten. - MODELS=(qwen2.5:7b cogito:14b ministral-3:8b granite4.1:8b ministral-3:14b qwen2.5:14b qwen3.5:4b qwen2.5-coder:14b qwen2.5:32b cogito:32b granite4.2:8b granite4.1:30b ornith:9b ornith:35b qwen3.5:9b qwen3:8b gemma4:12b gemma4:26b qwen3:14b nemotron3:33b nemotron-3.5-lightning:30b nemotron-3-nano:30b qwen3.6:27b qwen3.8:latest muse-glimmer:latest qwen3:4b gemini-3.5-flash-lite) + MODELS=(qwen2.5:7b cogito:14b ministral-3:8b granite4.1:8b ministral-3:14b qwen2.5:14b qwen3.5:4b qwen2.5-coder:14b qwen2.5:32b cogito:32b granite4.2:8b granite4.1:30b ornith:9b ornith:35b qwen3.5:9b qwen3:8b gemma4:12b gemma4:26b gemma4:31b qwen3:14b nemotron3:33b nemotron-3.5-lightning:30b nemotron-3-nano:30b qwen3.6:27b qwen3.8:latest muse-glimmer:latest qwen3:4b gemini-3.5-flash-lite) fi # Watchable by default: the browser opens on screen and every click is visible, and a video of diff --git a/src/lib/agent/model-tuning/measured-profiles.json b/src/lib/agent/model-tuning/measured-profiles.json index d7d0f043..8e2290a9 100644 --- a/src/lib/agent/model-tuning/measured-profiles.json +++ b/src/lib/agent/model-tuning/measured-profiles.json @@ -150,6 +150,35 @@ ] } }, + { + "id": "gemma4:31b", + "measured": "6/6 modes locked, 30/30 runs passed with the two settings below. Investigate 26s · Optimize 378s · Assess 214s · Operate 93s · Analyze 132s · Plan 3s (medians of the passing runs), slowest run 398s — the slowest model on this list, and the only one whose median run passes a minute. Its six cells did not all close on one configuration: five locked at the compiled 90-second turn ceiling and optimize needed 150, so those five were read again under the pair that ships rather than recorded as something they had not been measured as. Twenty-five runs, every surface 5/5.", + "settings": { + "sampling": { + "temperature": 0, + "topP": 1 + }, + "unreportedCallCeiling": 12, + "reportReminderLimit": 1, + "planStatementRetries": 0, + "presentReminderLimit": 1, + "retryEmptyTurn": false, + "retryUnreadStop": false, + "suppressPlanReasoning": true, + "refusalExamples": false, + "turnTimeoutMs": 150000 + }, + "rationale": { + "suppressPlanReasoning": [ + "Its plan cell read 0/5 at the defaults and the loss was the clock, not the plan: the model spends the turn thinking and the deliverable never arrives.", + "Quiet, it answers in three seconds - the fastest plan turn of any model this size here - and the cell locked on the second attempt. It reaches plan turns only (`investigation.ts` reads it where the mode is not agent), so it is not what carried the five agent surfaces." + ], + "turnTimeoutMs": [ + "Optimize is this model's wall and its clock: a passing run takes 378 seconds against a 26-second investigation, and at the compiled 90-second ceiling the cell would not close.", + "The ceiling bounds ONE call rather than the run, so raising it does not obviously help - a turn cut at 90 is now re-asked, while a turn allowed 150 spends that from the run's own deadline instead. That trade was measured rather than argued: the four agent surfaces that had locked at 90 were read again at 150 and each returned 5/5." + ] + } + }, { "id": "granite4.1:30b", "measured": "6/6 modes locked, 30/30 runs passed at these settings. Investigate 5/5 · Optimize 5/5 · Assess 5/5 · Operate 5/5 · Analyze 5/5 · Plan 5/5.", diff --git a/tests/unit/lib/agent/model-resolution-table.test.ts b/tests/unit/lib/agent/model-resolution-table.test.ts index 77667433..c54bbfb0 100644 --- a/tests/unit/lib/agent/model-resolution-table.test.ts +++ b/tests/unit/lib/agent/model-resolution-table.test.ts @@ -97,6 +97,22 @@ const RESOLVED: ResolvedRow[] = [ suppressesAgentReasoning: true, turnTimeoutMs: 150_000, }, + { + // The family's largest, and the slowest model the roster carries: a 378-second optimize cell + // against a 26-second investigation. Two settings and they were earned separately - the plan + // switch closed a plan cell losing to `thinking`, and the 150-second ceiling is what optimize + // could not close without. Deliberately NOT `suppressesAgentReasoning`, which its 12b sibling + // carries: that one was measured for an illness this size does not have. + id: "gemma4:31b", + unreportedCallCeiling: 12, + reportReminderLimit: 1, + planStatementRetries: 0, + presentReminderLimit: 1, + retriesEmptyTurn: false, + refusalExamples: false, + suppressesPlanReasoning: true, + turnTimeoutMs: 150_000, + }, { id: "gemma4:26b", unreportedCallCeiling: 10, @@ -480,7 +496,7 @@ describe("every resolver's answer, pinned before the profiles moved", () => { test("the table covers every registered model, so a new one cannot arrive unpinned", () => { const pinned = new Set(RESOLVED.map((row) => row.id)); for (const id of Object.keys(modelProfiles())) expect(pinned.has(id)).toBe(true); - expect(Object.keys(modelProfiles())).toHaveLength(27); + expect(Object.keys(modelProfiles())).toHaveLength(28); }); }); @@ -525,6 +541,7 @@ describe("what each model records about the runs that earned its settings", () = "gemini-3.5-flash-lite": "57453d009646b45dcee4bd74c46fcad9fa03ce69790e302fc948f1a60809015a", "gemma4:12b": "9a49a9323c21ffe507698ca2ca852cc1b59647a206e73c448afeea7f1a0a674b", "gemma4:26b": "d8124e9d5b0929364129274fd4f80dea2640773147fdfd834cf2c68a5a08dd76", + "gemma4:31b": "430cdba51e7b09167b2fa59971fa033830595458d54c1272f6424341923fda54", "granite4.1:30b": "57453d009646b45dcee4bd74c46fcad9fa03ce69790e302fc948f1a60809015a", "granite4.1:8b": "a3eea21447a81fbe058e3c18a0f7194c357e5d5f22db9acfe13e6139d9198874", "granite4.2:8b": "c9e47190c44d1fda45bf035831dcb17ba21621e98a455259a438710775600ae0", diff --git a/tests/unit/lib/agent/model-tuning.test.ts b/tests/unit/lib/agent/model-tuning.test.ts index 605964fe..5aae8120 100644 --- a/tests/unit/lib/agent/model-tuning.test.ts +++ b/tests/unit/lib/agent/model-tuning.test.ts @@ -93,7 +93,7 @@ afterEach(() => { describe("the document Studio ships with", () => { test("passes its own contract", () => { const tuning = parseTuning(bundled, "test"); - expect(Object.keys(tuning.models)).toHaveLength(27); + expect(Object.keys(tuning.models)).toHaveLength(28); }); test("argues for every value it changed", () => { @@ -563,7 +563,7 @@ describe("a document an operator supplies", () => { process.env[ENV] = writeDocument("{ not json"); resetTuning(); expect(ceilingFor("gemma4:26b")).toBe(10); - expect(Object.keys(activeTuning().models)).toHaveLength(27); + expect(Object.keys(activeTuning().models)).toHaveLength(28); }); test("reports that it ignored a document, naming the file and the reason", () => { @@ -717,13 +717,13 @@ describe("a document an operator supplies", () => { test("is ignored when it breaks the contract, not partially applied", () => { process.env[ENV] = writeDocument(document({ schemaVersion: 99 })); resetTuning(); - expect(Object.keys(activeTuning().models)).toHaveLength(27); + expect(Object.keys(activeTuning().models)).toHaveLength(28); }); test("is ignored when the file is not there at all", () => { process.env[ENV] = "/nonexistent/models.json"; resetTuning(); - expect(Object.keys(activeTuning().models)).toHaveLength(27); + expect(Object.keys(activeTuning().models)).toHaveLength(28); }); test("an unset or blank variable is simply no operator document", () => { @@ -731,7 +731,7 @@ describe("a document an operator supplies", () => { // reading it as a path would warn on every boot of an install that configured nothing. process.env[ENV] = " "; resetTuning(); - expect(Object.keys(activeTuning().models)).toHaveLength(27); + expect(Object.keys(activeTuning().models)).toHaveLength(28); }); test("is read once, so a run cannot see the table change under it", () => {