Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ run by asking `GET /api/agent/config`, the same way it discovers the storage mod
| `LIBREDB_AGENT_THREAD_CONTEXT` | unset (on) | Whether a run may be told about the **conversation** it belongs to. A follow-up asked on the same connection continues the previous run's thread: the earlier steps' objectives and the most recent step's report are derived server-side from those runs' own ledgers and handed to the model fenced. Set `false`/`off`/`0` where no question's context may reach another. Every run then opens on its own and the rail SAYS so — a user who asks a follow-up is told the conversation is switched off on this server, rather than being left to infer it from an answer that does not resolve. `GET /api/agent/config` reports the state to an **admin** session, beside `modelTuning` and for the same reason: an operator who switches something off must not hear silence, and `curl` is how they check. It is deliberately not sent to every session — the sentence a USER needs comes from the run's own `thread.declined` at the moment their follow-up was not read as one, which is where it means something. An unrecognized value warns and is ignored, the same two-sided rule `LIBREDB_AGENT_ENABLED` follows: a typo must neither take a working surface away nor turn one on. It is the operator's counterpart to the control the user already has — the rail names the run being continued and offers "new conversation" beside it. |
| `WORKFLOW_TARGET_WORLD` | unset (`local`) | Durable backend for run state. Exactly two values are accepted: `local` (zero-config, on-disk, **single instance**) and `@workflow/world-postgres` (opt-in, multi-replica, needs `WORKFLOW_POSTGRES_URL`). Anything else is **refused**, not defaulted. |
| `AGENT_MODEL_TURN_TIMEOUT_MS` | unset (`90000`) | How long **one** model call may take before the drive stops waiting for it. Raise it for a LOCAL model: the default was chosen against hosted APIs, where a turn lands in seconds and a 90-second wait only ever means a request that is not coming back. Measured across 25 Ollama models on six surfaces, **nine** runs ended `model-timeout` with the model still working — one of them a reasoning model in plan mode, which holds no tools at all, cut 92 s into its **first** turn with a zero-event ledger. Those runs are scored as having answered nothing, which is a fact about this ceiling and not about the model. A value that is not a positive whole number is **ignored** and the default stands; a value is capped just under half the smallest workflow deadline, because a run has to be able to take two turns to finish. |
| `AGENT_MODEL_TUNING_PATH` | unset | A JSON document of measured per-model settings, layered over the ones Studio ships with. Studio carries a document recording what specific models were measured under — turn limit, how many readings before it is asked to report, whether an empty turn is asked again — and a model not named in it is driven with the defaults, which is the honest treatment of a model nobody has measured. This is how a model Studio has never measured gets settings somebody else measured: mount a file in the same shape and restart, with no Studio release and no code change. Merged **per model and whole** — an entry here replaces the shipped entry for that model rather than contributing one field to it, because half of one measurement beside half of another is a configuration nobody has run. A file that is missing, unreadable or off-schema is **ignored** and the shipped measurements stand — which is the one setting here that fails **open**, so it is also the one that reports itself: `GET /api/agent/config` tells an **admin** session what became of the document (`{"modelTuning":{"state":"applied"\|"ignored"\|"unset",…}}`, with the path and the parser's reason), because an operator who mounts a file and is told nothing will believe it is in force. It carries numbers and switches only: the sentences the drive says to a model stay in Studio, so supplying this file cannot change what Studio tells a model. On Kubernetes the chart mounts it for you — see `agent.modelTuning.*` in [`charts/libredb-studio/README.md`](../charts/libredb-studio/README.md). The document's own contract — every setting, its bounds, what happens to a key this build does not implement, and the example to start from — is [`docs/llms/model-tuning.md`](llms/model-tuning.md). |
| `AGENT_MODEL_TUNING_PATH` | unset | A JSON document of measured per-model settings, layered over the ones Studio ships with. Studio carries a document recording what specific models were measured under — turn limit, how many readings before it is asked to report, whether an empty turn is asked again — and a model not named in it is driven with the defaults, which is the honest treatment of a model nobody has measured — bar two settings whose gates are reachable only on a run that has already fallen short, where an absent entry is read as the absence it is rather than as a value somebody wrote. This is how a model Studio has never measured gets settings somebody else measured: mount a file in the same shape and restart, with no Studio release and no code change. Merged **per model and whole** — an entry here replaces the shipped entry for that model rather than contributing one field to it, because half of one measurement beside half of another is a configuration nobody has run. A file that is missing, unreadable or off-schema is **ignored** and the shipped measurements stand — which is the one setting here that fails **open**, so it is also the one that reports itself: `GET /api/agent/config` tells an **admin** session what became of the document (`{"modelTuning":{"state":"applied"\|"ignored"\|"unset",…}}`, with the path and the parser's reason), because an operator who mounts a file and is told nothing will believe it is in force. It carries numbers and switches only: the sentences the drive says to a model stay in Studio, so supplying this file cannot change what Studio tells a model. On Kubernetes the chart mounts it for you — see `agent.modelTuning.*` in [`charts/libredb-studio/README.md`](../charts/libredb-studio/README.md). The document's own contract — every setting, its bounds, what happens to a key this build does not implement, and the example to start from — is [`docs/llms/model-tuning.md`](llms/model-tuning.md). |
| `WORKFLOW_LOCAL_DATA_DIR` | unset — but the packaged artifacts set it: `/app/data/workflow` from the Helm chart and (from an app version later than `0.11.0`) the container image, `~/.libredb-studio/workflow-data` under `npx`. The SDK's own fallback, which those replace, is `.workflow-data` relative to the working directory. | Where the `local` backend keeps run state, and therefore the second condition above. See [Deployment](#deployment) — the SDK's fallback is wrong in a container and wrong under `npx`, so no artifact leaves it in force. |

The refusal is not pedantry. The workflow runtime reads that variable itself and treats any value
Expand Down
4 changes: 3 additions & 1 deletion docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -1703,7 +1703,9 @@ switches therefore spends two extra turns rather than one, and a model carrying
its `retryEmptyTurn: false` overridden by a switch that argues for something else.

Live on `nemotron3:33b`, whose entry records `retryEmptyTurn: false` and whose empty turns are
asked again anyway. Pinned as it behaves in `tests/isolated/agent-investigation.test.ts` rather
asked again anyway — and, since the gate began reading `answersUnreadStop` rather than
`retriesUnreadStop`, on every model with no entry at all, which is the same subsumption over a
wider set. Pinned as it behaves in `tests/isolated/agent-investigation.test.ts` rather
than repaired, because the repair — narrowing the gate to a turn with text in it — changes the
behaviour the five passing query-optimization runs were measured under, and this repository does
not move a measured cell without re-measuring it.
Expand Down
19 changes: 16 additions & 3 deletions docs/llms/model-tuning.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
Studio ships a document recording what specific models were measured under — how long one of
their turns may take, how many readings they may take before being asked to report, whether an
empty turn is worth asking again. A model it does not name is driven with the compiled defaults,
which is the honest treatment of a model nobody has measured.
which is the honest treatment of a model nobody has measured — bar the two settings noted under
[the settings](#the-settings), where an absent entry is read as the absence it is rather than as a
`false` somebody wrote.

This page is how you supply your own, with no new Studio release and no code change. It is the
other half of [`testing-your-own.md`](testing-your-own.md): that page is how you measure a model,
Expand Down Expand Up @@ -113,11 +115,11 @@ Every one is optional. What you do not state resolves to the compiled default in
| `perWorkflow` | the same object, per workflow id | sampling for named surfaces only — the narrowest an override gets | — |
| `unreportedCallCeiling` | integer 1–100 | how many calls it may make without reporting before the run is narrowed to the tools that would finish it | `12` |
| `reportReminderLimit` | integer 0–5 | how many times a turn with no call and no report may be answered with the report reminder | `1` |
| `planStatementRetries` | integer 0–5 | extra turns a PLAN run gets when its prose named neither a statement nor a refusal | `0` |
| `planStatementRetries` | integer 0–5 | extra turns a PLAN run gets when its prose named neither a statement nor a refusal | `0` — but see below |
| `presentReminderLimit` | integer 0–5 | how many times a report may be held to ask for the answer that belongs beside it | `1` |
| `verdictHoldLimit` | integer 0–5 | how many times a report whose own verdict would REJECT it may be held and told why — the third of the reminder bounds. A run about to pass never reaches this hold, so raising it costs turns only on a run that has already lost | `2` |
| `retryEmptyTurn` | boolean | whether a turn that came back EMPTY is asked once more before the run is ended | `false` |
| `retryUnreadStop` | boolean | whether a run that stopped having CALLED NOTHING is told once to read the database itself, instead of being ended — it subsumes `retryEmptyTurn`, since the gate asks what was called and not what was said | `false` |
| `retryUnreadStop` | boolean | whether a run that stopped having CALLED NOTHING is told once to read the database itself, instead of being ended — it subsumes `retryEmptyTurn`, since the gate asks what was called and not what was said | `false` — but see below |
| `suppressPlanReasoning` | boolean | whether this model's PLAN turn asks the endpoint for no reasoning at all — reaches the OpenAI-compatible adapter only (`openai`, `ollama`, `custom`), so it is a no-op on `gemini` | `false` |
| `suppressAgentReasoning` | boolean | the same, for this model's AGENT turns — for a model that either answers at once or thinks until the wall, which `turnTimeoutMs` does not address because a turn spent thinking finds the new wall too. Same adapters, same no-op on `gemini` | `false` |
| `refusalExamples` | boolean | whether a refused call is handed a worked example built from this run's ledger | `false` |
Expand All @@ -136,6 +138,17 @@ your model starts losing the schema block or ending early; raise it if your mode
conversations are long enough to be truncating. Either way the answer comes from driving it, which
is what [`testing-your-own.md`](testing-your-own.md) is for.

**Two settings answer differently for a model with NO entry at all**, and the last column above is
the compiled default rather than what an unnamed model gets. `planStatementRetries` and
`retryUnreadStop` each buy a single extra turn, and each gate is reachable only on a run that has
already fallen short — one where the closing prose named no statement, one where the model stopped
having called nothing. Neither can turn a pass into a failure, so withholding them from a model
nobody has measured only guarantees that the model most in need of the sentence is the one certain
not to receive it. A model with no entry is therefore offered both: one plan ask
(`planStatementAsksFor`) and one answer to a stop that read nothing (`answersUnreadStop`), both in
[`src/lib/agent/models/index.ts`](../../src/lib/agent/models/index.ts). **A stated value still
wins** — a `0` or a `false` you write here is a measurement, and Studio does not overrule it.

## The rules

**Merged per model and WHOLE.** An entry replaces the shipped entry for that model rather than
Expand Down
10 changes: 6 additions & 4 deletions src/lib/agent/investigation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ import {
presentReminderLimitFor,
verdictHoldLimitFor,
retriesEmptyTurn,
retriesUnreadStop,
answersUnreadStop,
suppressesAgentReasoning,
suppressesPlanReasoning,
turnTimeoutMsFor,
Expand Down Expand Up @@ -3516,8 +3516,10 @@ export async function runInvestigation(

Granted only where the run has lost anyway. `compose_report` is one of the tools
`anyToolCalled` counts, so a run reaching here with it false composed no report and
has already earned `no-report`; the turn cannot cost a pass. Once, and only for a
model whose ledger asked twice.
has already earned `no-report`; the turn cannot cost a pass. Once, and — because a
bound that cannot protect a passing run protects nothing — offered to whoever needs
it: `answersUnreadStop` reads a stated `false` as the measurement it is and an absent
profile as the absence it is, so a model nobody has measured is told to read.
*/
if (
record.mode === "agent" &&
Expand All @@ -3535,7 +3537,7 @@ export async function runInvestigation(
// and spends the very turn this retry bought: the #350/#356 defect, paid for once.
holdsTool("inspect_schema") &&
holdsTool("inspect_plan") &&
retriesUnreadStop(model.modelId)
answersUnreadStop(model.modelId)
) {
unreadStopRetried = true;
messages.push(...turn.assistantMessages);
Expand Down
29 changes: 28 additions & 1 deletion src/lib/agent/models/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,10 @@
* in the document; that is a better trade than carrying it empty until then.
*
* A model nobody has measured therefore resolves exactly to the compiled defaults, which is the
* honest treatment of a model nobody has measured.
* honest treatment of a model nobody has measured. Two resolvers below — `planStatementAsksFor`
* and `answersUnreadStop` — deliberately do not, and each says why in its own note: they feed
* gates reachable only on a run that has already fallen short, where a default cannot protect a
* passing run and so protects nothing. A stated value still wins in both.
*/

import { AGENT_THREAD_CONTEXT_MAX_CHARS } from "../execution-policy";
Expand Down Expand Up @@ -157,6 +160,30 @@ export function retriesUnreadStop(modelId: string): boolean {
return resolve(modelId, "retryUnreadStop") ?? DEFAULT_RETRY_UNREAD_STOP;
}

/**
* The same question for a model NOBODY HAS MEASURED: the sentence is sent rather than withheld.
*
* `retriesUnreadStop` folds two different answers into one, exactly as `planStatementRetriesFor`
* did before `planStatementAsksFor` split them: a profile stating `false` is a measurement, and no
* profile at all is the absence of one. Reading the absence as a `false` withheld the drive's own
* sentence — "Read it yourself. Call inspect_schema for the tables and their columns, and
* inspect_plan for how a statement will run" — from precisely the models nobody had measured yet,
* which is to say from every model this project has not reached.
*
* The gate it feeds fires only where no tool was called, so the run composed no report and has
* already earned `no-report`; it provably cannot cost a pass. Measured across the sweep behind
* 0.14.1: three hundred runs ended `model-stopped` with `no-report`, a hundred and fifty of them
* having called no tool at all, and ninety-four of those hundred and fifty were ended without the
* drive saying anything — `mistral-small3.1:24b` 51, `granite3.3:8b` 41, `mistral:7b` 40,
* `gpt-oss:20b` 18, a hundred and ten of them on `investigation`, the first surface a model meets.
*
* A measured model is untouched: all twenty-eight profiles state the field, so no shipped model's
* turn count moves.
*/
export function answersUnreadStop(modelId: string): boolean {
return resolve(modelId, "retryUnreadStop") ?? true;
}

/**
* Whether this model's plan turn is told to spend nothing on reasoning.
*
Expand Down
20 changes: 14 additions & 6 deletions src/lib/agent/models/profile.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,12 @@ export interface AgentModelProfile {
* run that called nothing composed no report and has already earned `no-report`; the turn is
* spent on a run that has lost. It cannot cost a pass, only recover a failure.
*
* Off by default even so. The ten models locked at 300/300 were measured without it, and a
* drive-wide change is twice how this repository has handed back cells it had won.
* A stated `false` is obeyed even so. The ten models locked at 300/300 were measured without
* it, and overruling a measurement is twice how this repository has handed back cells it had
* won. What is NOT obeyed is an absent entry, which is not a measurement: the drive asks
* `answersUnreadStop`, not `retriesUnreadStop`, and that resolver offers the turn to a model
* nobody has measured. All twenty-eight shipped entries state this field, so no shipped model's
* turn count moved when the two were split.
*
* It SUBSUMES `retryEmptyTurn`, and that is a property of the gate rather than of the name.
* The condition is "called nothing", with no test on what was said, so an empty completion
Expand Down Expand Up @@ -299,10 +303,14 @@ export const DEFAULT_RETRY_EMPTY_TURN = false;
/**
* A run that stops having called nothing keeps its ending, unless a model's ledger asked.
*
* Off despite being free to grant — the turn is spent on a run whose verdict is already
* `no-report` — because "free" is an argument about cost, not about wording. The sentence sent
* is read by the model and acted on by it, so it is a measured value like every other, and it
* belongs to the models measured with it rather than to all of them at once.
* Off, and it stays off for every model that states it: the sentence sent is read by the model
* and acted on by it, so it is a measured value like every other and belongs to the models
* measured with it rather than to all of them at once.
*
* This constant is what a STATED field falls back to, and every shipped entry states the field,
* so nothing resolves through it today. It is not what a model with no entry gets — that is
* `answersUnreadStop`, which offers the turn, because on a run already earning `no-report` a
* default cannot protect a passing run and so protects nothing.
*/
export const DEFAULT_RETRY_UNREAD_STOP = false;

Expand Down
1 change: 1 addition & 0 deletions tests/evals/database-assessment.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ describe("the bar this workflow is judged against is stated to the model", () =>
prompts.push(promptText(turn));
return answersProse("nothing to do")(turn);
},
answersProse("nothing to do"),
]);

expect(prompts[0]).toContain("call profile_table on at least one table before you report");
Expand Down
Loading
Loading