Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions docs/llms/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ writes beautiful prose about a database and never calls a tool answers nothing h
question these pages answer is not what a model knows but what it DOES on a run, and every figure
comes from a run whose ledger is on disk.

**Twenty-seven models are supported.** Each cleared all six surfaces — Investigate, Optimize, Assess,
Operate, Analyze and Plan — five consecutive times: 810 runs, 810 passes.
**Twenty-eight models are supported.** Each cleared all six surfaces — Investigate, Optimize, Assess,
Operate, Analyze and Plan — five consecutive times: 840 runs, 840 passes.

Twenty-one of the twenty-seven cleared them at the 90-second per-turn limit the product ships. Six carry a
150-second limit of their own — `qwen3.5:9b`, `gemma4:12b`, `nemotron-3.5-lightning:30b`,
Twenty-one of the twenty-eight cleared them at the 90-second per-turn limit the product ships. Seven carry a
150-second limit of their own — `qwen3.5:9b`, `gemma4:12b`, `gemma4:31b`, `nemotron-3.5-lightning:30b`,
`nemotron-3-nano:30b`, `muse-glimmer:latest` and `qwen3.6:27b` — because one turn of theirs does
not fit inside 90 while every other surface does. The limit stays 90 for every other model, and
each page says what its model needed.
Expand Down Expand Up @@ -45,12 +45,13 @@ each page says what its model needed.
| [`qwen3:4b`](qwen/qwen3.md) | Ollama | 2.5 GB | 72s | 160s |
| [`qwen3.8:latest`](qwen/qwen3.8.md) | Ollama | 19 GB | 72s | 195s |
| [`qwen3.6:27b`](qwen/qwen3.6.md) | Ollama | 17 GB | 82s | 252s |
| [`gemma4:31b`](gemma/gemma4.md) | Ollama | 19 GB | 107s | 398s |
| [`muse-glimmer:latest`](muse/muse-glimmer.md) | Ollama | 18 GB | 115s | 285s |

The durations are from one machine and are comparable with each other rather than portable: every
figure was taken the same way, on the same database, through the same six surfaces. What they are
for is choosing between these twenty-seven — the fastest reaches the same verdicts as the slowest in
a nineteenth of the time.
for is choosing between these twenty-eight — the fastest reaches the same verdicts as the slowest in
an eighteenth of the time.

One page per model version. Sizes are rows inside it, because `ollama pull qwen3:4b` is how a
size is chosen and because the interesting fact is usually the difference between two sizes of
Expand Down Expand Up @@ -84,7 +85,7 @@ pass says nothing at all about the fifth.

| If you want | Take | Why |
| --- | --- | --- |
| the fastest local model | [`granite4.1:8b`](granite/granite4.1.md) | 11s median, 5.3 GB, and it clears everything |
| the fastest local model | [`qwen2.5:7b`](qwen/qwen2.5.md) | 6s median, 4.7 GB, and nothing it did on thirty runs took longer than 21s |
| the smallest download | [`qwen3:4b`](qwen/qwen3.md) | 2.5 GB — and not the fastest: size buys memory, not speed |
| the steadiest | [`cogito:14b`](cogito/cogito.md) | 20s is the shortest slowest-run on the roster, and under twice its own median |
| no local hardware at all | [`gemini-3.5-flash-lite`](gemini/gemini-3.5.md) | the one hosted model, 10s median, and it needs a key |
Expand Down
34 changes: 32 additions & 2 deletions docs/llms/gemma/gemma4.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
# gemma4

`ollama pull gemma4:<size>` · sizes supported: 12b, 26b
`ollama pull gemma4:<size>` · sizes supported: 12b, 26b, 31b

Every size listed here runs **all six agent surfaces**, five consecutive times each: 30 of 30
runs. The 26b is the size that carries this family, and the one whose ledger taught this project
what an empty completion looks like. The 12b is the smaller, faster one, and it is the model that
made the product learn to tell an agent turn to stop thinking.
made the product learn to tell an agent turn to stop thinking. The 31b is the largest, the slowest
model on the whole roster, and the one that shows what this family's growth actually buys.

## What it does, and how long it takes

Expand All @@ -15,10 +16,17 @@ Seconds are the median of the runs that passed, per surface.
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| gemma4:12b | 7.6 GB | 4s | 24s | 21s | 14s | 11s | 2s | **11s** | 1:03 |
| gemma4:26b | 16 GB | 15s | 1:02 | 31s | 26s | 46s | 36s | **36s** | 1:32 |
| gemma4:31b | 19 GB | 26s | 6:18 | 3:34 | 1:33 | 2:12 | 3s | **1:47** | 6:38 |

Every cell is 5/5, so the table says how long rather than whether. The 12b is three times the
26b's speed at half its size, and it did not start out that way — see its note below.

**Size buys nothing here, and the 31b is the evidence.** Three more gigabytes than the 26b cost it
three times the median and six minutes on one cell, and every verdict it reaches the 12b reaches
too, in a tenth of the time. Its optimize cell alone runs longer than a whole 12b sweep. Take it
only if this exact model is what you have to run; otherwise the 12b is the same family answering
the same six questions.

## What it needs that the defaults do not give it

### `gemma4:12b`
Expand All @@ -41,6 +49,28 @@ the clock while working rather than while thinking.
their length: analyze 102s to 11s, assess 118s to 21s, operate 75s to 14s, investigate 9s to 4s.
It arrived at 2/6 and 15/30 and leaves at 6/6 and 30/30, three times faster than the 26b.

### `gemma4:31b`

**no reasoning on its plan turn.** That cell read 0/5 at the defaults, losing to the clock rather
than to the plan — the model spends the turn thinking and the deliverable never arrives. Quiet, it
answers in three seconds, the fastest plan turn in this family, and the cell locked on the second
attempt. The switch reaches plan turns only (`suppressesPlanReasoning` is read where the mode is
not `agent`, in [`investigation.ts`](../../../src/lib/agent/investigation.ts)), so it is not what
carried the five agent surfaces.

**a 150-second turn**, for optimize and only because of optimize: a passing optimize run takes six
minutes against a 26-second investigation, and at the compiled 90-second ceiling the cell would not
close.

Those two were earned separately, which left five cells locked at the 90-second ceiling while the
profile that ships carries 150. Raising a per-call ceiling is not obviously free — a turn cut at 90
is re-asked, while a turn allowed 150 spends that from the run's own deadline instead — so the five
were read again under the pair that ships: twenty-five runs, every surface 5/5. The figures in the
table above are from those runs.

It does **not** carry `suppressAgentReasoning`, which its 12b sibling does. That setting was
measured for an illness this size does not have, and a setting a model did not earn is a guess.

### `gemma4:26b`

**call ceiling of 10.** It reads more thoroughly than a run has room for: eleven calls in, with nothing left to spend a twelfth on.
Expand Down
2 changes: 1 addition & 1 deletion scripts/agent-model-e2e.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ MODELS=("$@")
if [ ${#MODELS[@]} -eq 0 ]; then
# Fastest first, so the run reports something early, and the hosted model last because it needs
# its own environment file swapped in rather than a line rewritten.
MODELS=(qwen2.5:7b cogito:14b ministral-3:8b granite4.1:8b ministral-3:14b qwen2.5:14b qwen3.5:4b qwen2.5-coder:14b qwen2.5:32b cogito:32b granite4.2:8b granite4.1:30b ornith:9b ornith:35b qwen3.5:9b qwen3:8b gemma4:12b gemma4:26b qwen3:14b nemotron3:33b nemotron-3.5-lightning:30b nemotron-3-nano:30b qwen3.6:27b qwen3.8:latest muse-glimmer:latest qwen3:4b gemini-3.5-flash-lite)
MODELS=(qwen2.5:7b cogito:14b ministral-3:8b granite4.1:8b ministral-3:14b qwen2.5:14b qwen3.5:4b qwen2.5-coder:14b qwen2.5:32b cogito:32b granite4.2:8b granite4.1:30b ornith:9b ornith:35b qwen3.5:9b qwen3:8b gemma4:12b gemma4:26b gemma4:31b qwen3:14b nemotron3:33b nemotron-3.5-lightning:30b nemotron-3-nano:30b qwen3.6:27b qwen3.8:latest muse-glimmer:latest qwen3:4b gemini-3.5-flash-lite)
fi

# Watchable by default: the browser opens on screen and every click is visible, and a video of
Expand Down
29 changes: 29 additions & 0 deletions src/lib/agent/model-tuning/measured-profiles.json
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,35 @@
]
}
},
{
"id": "gemma4:31b",
"measured": "6/6 modes locked, 30/30 runs passed with the two settings below. Investigate 26s · Optimize 378s · Assess 214s · Operate 93s · Analyze 132s · Plan 3s (medians of the passing runs), slowest run 398s — the slowest model on this list, and the only one whose median run passes a minute. Its six cells did not all close on one configuration: five locked at the compiled 90-second turn ceiling and optimize needed 150, so those five were read again under the pair that ships rather than recorded as something they had not been measured as. Twenty-five runs, every surface 5/5.",
"settings": {
"sampling": {
"temperature": 0,
"topP": 1
},
"unreportedCallCeiling": 12,
"reportReminderLimit": 1,
"planStatementRetries": 0,
"presentReminderLimit": 1,
"retryEmptyTurn": false,
"retryUnreadStop": false,
"suppressPlanReasoning": true,
"refusalExamples": false,
"turnTimeoutMs": 150000
},
"rationale": {
"suppressPlanReasoning": [
"Its plan cell read 0/5 at the defaults and the loss was the clock, not the plan: the model spends the turn thinking and the deliverable never arrives.",
"Quiet, it answers in three seconds - the fastest plan turn of any model this size here - and the cell locked on the second attempt. It reaches plan turns only (`investigation.ts` reads it where the mode is not agent), so it is not what carried the five agent surfaces."
],
"turnTimeoutMs": [
"Optimize is this model's wall and its clock: a passing run takes 378 seconds against a 26-second investigation, and at the compiled 90-second ceiling the cell would not close.",
"The ceiling bounds ONE call rather than the run, so raising it does not obviously help - a turn cut at 90 is now re-asked, while a turn allowed 150 spends that from the run's own deadline instead. That trade was measured rather than argued: the four agent surfaces that had locked at 90 were read again at 150 and each returned 5/5."
]
}
},
{
"id": "granite4.1:30b",
"measured": "6/6 modes locked, 30/30 runs passed at these settings. Investigate 5/5 · Optimize 5/5 · Assess 5/5 · Operate 5/5 · Analyze 5/5 · Plan 5/5.",
Expand Down
19 changes: 18 additions & 1 deletion tests/unit/lib/agent/model-resolution-table.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,22 @@ const RESOLVED: ResolvedRow[] = [
suppressesAgentReasoning: true,
turnTimeoutMs: 150_000,
},
{
// The family's largest, and the slowest model the roster carries: a 378-second optimize cell
// against a 26-second investigation. Two settings and they were earned separately - the plan
// switch closed a plan cell losing to `thinking`, and the 150-second ceiling is what optimize
// could not close without. Deliberately NOT `suppressesAgentReasoning`, which its 12b sibling
// carries: that one was measured for an illness this size does not have.
id: "gemma4:31b",
unreportedCallCeiling: 12,
reportReminderLimit: 1,
planStatementRetries: 0,
presentReminderLimit: 1,
retriesEmptyTurn: false,
refusalExamples: false,
suppressesPlanReasoning: true,
turnTimeoutMs: 150_000,
},
{
id: "gemma4:26b",
unreportedCallCeiling: 10,
Expand Down Expand Up @@ -480,7 +496,7 @@ describe("every resolver's answer, pinned before the profiles moved", () => {
test("the table covers every registered model, so a new one cannot arrive unpinned", () => {
const pinned = new Set(RESOLVED.map((row) => row.id));
for (const id of Object.keys(modelProfiles())) expect(pinned.has(id)).toBe(true);
expect(Object.keys(modelProfiles())).toHaveLength(27);
expect(Object.keys(modelProfiles())).toHaveLength(28);
});
});

Expand Down Expand Up @@ -525,6 +541,7 @@ describe("what each model records about the runs that earned its settings", () =
"gemini-3.5-flash-lite": "57453d009646b45dcee4bd74c46fcad9fa03ce69790e302fc948f1a60809015a",
"gemma4:12b": "9a49a9323c21ffe507698ca2ca852cc1b59647a206e73c448afeea7f1a0a674b",
"gemma4:26b": "d8124e9d5b0929364129274fd4f80dea2640773147fdfd834cf2c68a5a08dd76",
"gemma4:31b": "430cdba51e7b09167b2fa59971fa033830595458d54c1272f6424341923fda54",
"granite4.1:30b": "57453d009646b45dcee4bd74c46fcad9fa03ce69790e302fc948f1a60809015a",
"granite4.1:8b": "a3eea21447a81fbe058e3c18a0f7194c357e5d5f22db9acfe13e6139d9198874",
"granite4.2:8b": "c9e47190c44d1fda45bf035831dcb17ba21621e98a455259a438710775600ae0",
Expand Down
10 changes: 5 additions & 5 deletions tests/unit/lib/agent/model-tuning.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ afterEach(() => {
describe("the document Studio ships with", () => {
test("passes its own contract", () => {
const tuning = parseTuning(bundled, "test");
expect(Object.keys(tuning.models)).toHaveLength(27);
expect(Object.keys(tuning.models)).toHaveLength(28);
});

test("argues for every value it changed", () => {
Expand Down Expand Up @@ -563,7 +563,7 @@ describe("a document an operator supplies", () => {
process.env[ENV] = writeDocument("{ not json");
resetTuning();
expect(ceilingFor("gemma4:26b")).toBe(10);
expect(Object.keys(activeTuning().models)).toHaveLength(27);
expect(Object.keys(activeTuning().models)).toHaveLength(28);
});

test("reports that it ignored a document, naming the file and the reason", () => {
Expand Down Expand Up @@ -717,21 +717,21 @@ describe("a document an operator supplies", () => {
test("is ignored when it breaks the contract, not partially applied", () => {
process.env[ENV] = writeDocument(document({ schemaVersion: 99 }));
resetTuning();
expect(Object.keys(activeTuning().models)).toHaveLength(27);
expect(Object.keys(activeTuning().models)).toHaveLength(28);
});

test("is ignored when the file is not there at all", () => {
process.env[ENV] = "/nonexistent/models.json";
resetTuning();
expect(Object.keys(activeTuning().models)).toHaveLength(27);
expect(Object.keys(activeTuning().models)).toHaveLength(28);
});

test("an unset or blank variable is simply no operator document", () => {
// Blank matters: it is what a Helm template renders when nobody filled the value in, and
// reading it as a path would warn on every boot of an install that configured nothing.
process.env[ENV] = " ";
resetTuning();
expect(Object.keys(activeTuning().models)).toHaveLength(27);
expect(Object.keys(activeTuning().models)).toHaveLength(28);
});

test("is read once, so a run cannot see the table change under it", () => {
Expand Down
Loading