Skip to content

feat(chat): guard the stream — idle deadline, pre-token retry, configurable ceiling - #152

Merged
mrsibe merged 1 commit into
mainfrom
feat/stream-reliability
Sep 27, 2026
Merged

mrsibe merged 1 commit into
mainfrom
feat/stream-reliability

Conversation

@mrsibe

@mrsibe mrsibe commented Sep 27, 2026

Copy link
Copy Markdown
Owner

Closes #150, the first half of #143.

Three things the turn lifecycle could not do, all of them about a turn that goes wrong rather than about what a turn is.

A stream that goes quiet is cancelled

Nothing noticed a provider that stopped producing while holding the connection open: the turn sat in streaming for as long as the socket lived, the composer stayed locked, and a hung provider was indistinguishable from a slow one — the only way out was Stop.

  • A deadline that resets on every event, reasoning deltas included, so a model that thinks for a long time is not cut off mid-thought.
  • It cancels rather than settling: whether a silent provider is worth another attempt is the retry policy's decision, not the watchdog's. That is why ChatExecution gained a cancel that decides nothing about the turn.
  • The outcome is failed(timeout) however the cancellation surfaces — a rejected read, or the SDK turning the abort into an abort chunk. Both happen in practice, and the first version of the test only covered one of them.

A transient failure before the first token is retried

Retrying is the wrong answer for a deterministic failure: same wait, same error, plus tokens spent. So the rule is narrow, and the default is not to retry:

  • only recognisably transient failures — rate limit, 429, 5xx, overloaded, timeout, and the transport's own errors (ECONNRESET, socket hang up, fetch failed, …). Anything unrecognised, and anything like a policy refusal or a bad key, is reported as it is.
  • only while nothing the reader would see has been sent. Structural events (start, start-step) do not count: verified against the pinned SDK, a second attempt's parts append to an empty message, so the reader still sees exactly one answer.
  • bounded, with backoff — three attempts, 500 ms then 1 s — so a provider that is down is not hammered.

Each attempt gets its own signal (ChatExecution.beginAttempt), so a retry after a timeout or a cancel does not inherit an aborted one.

The output ceiling stops being a hard-coded constant

DEFAULT_MAX_TOKENS = 2048 was sent on every call and was not configurable anywhere. On a reasoning model the thinking tokens share that budget with the answer, so a reader using one saw truncated early even with a perfectly healthy stream — the ceiling was the defect, not the stream.

ModelConnection now carries maxOutputTokens, and absent means no ceiling is sent at all: the model's own default applies. It is a per-connection field for the same reason the model id is — the right number belongs to the model, not to KnowNote. Settings shows Automatic (default) or a custom value, for chat connections only.

Verification

npm test 336 pass, 0 fail — 13 new
npm run typecheck clean
npm run check:design no violations
npm run build bundles
npm run build:unpack + smoke:packaged PASS — 23 checks
npx eslint <changed files> 0 errors (the 2 no-explicit-any warnings are pre-existing lines in validation.ts)

New coverage: test/retryPolicy.test.ts (what counts as transient, what does not, the attempt bound, the content rule, backoff) and five cases in test/chatStreamManager.test.ts — a retry that succeeds and delivers the second attempt's answer, a transient failure after content that is deliberately not retried, a deterministic failure that is not retried, bounded attempts against a provider that is down, and a silent stream that is cancelled and fails as a timeout while keeping the part that had arrived. test/streamChat.test.ts pins both ceiling cases against the provider call args.

What is left

#151 — Continue vs Retry, with sibling attempts and the attempt_of column — is the last child of epic #138.

…urable ceiling

Closes #150, the first half of #143.

Three things the turn lifecycle could not do, all of them about a turn that goes wrong
rather than about what a turn *is*.

**A stream that goes quiet is cancelled.** Nothing noticed a provider that stopped
producing while holding the connection open: the turn sat in `streaming` forever, the
composer stayed locked, and a hung provider looked exactly like a slow one. There is a
deadline now, reset by every event — reasoning deltas included, so a model that thinks
for a long time is not cut off mid-thought. It *cancels* rather than settling, because
whether a silent provider is worth another attempt is the retry policy's call, so
`ChatExecution` learned to cancel without deciding anything.

**A transient failure before the first token is retried.** Retrying is the wrong answer
for a deterministic failure — same wait, same error, plus tokens — so only recognisably
transient failures qualify (rate limit, 429, 5xx, overloaded, timeout, the transport's
own errors), and anything unrecognised is reported as it is. Two more conditions:
nothing the reader would see may have been sent, and the attempts are bounded with
backoff. Structural events (`start`, `start-step`) do not count as content, which is
what makes the retry reachable at all: verified against the pinned SDK, a second
attempt's parts append to an empty message, so the reader sees one answer.

**The output ceiling stops being a hard-coded constant.** `DEFAULT_MAX_TOKENS = 2048`
was sent on every call and was not configurable anywhere. On a reasoning model the
thinking tokens share that budget with the answer, so the reader saw `truncated` early
even with a perfectly healthy stream. The connection now carries `maxOutputTokens`, and
absent means no ceiling is sent at all — the model's own default applies. It is a
per-connection field for the same reason the model id is: the right value belongs to
the model, not to KnowNote. Settings shows Automatic or a custom value, chat
connections only.

Each attempt gets its own signal (`ChatExecution.beginAttempt`), so a retry after a
timeout or a cancel does not inherit an aborted one, and the outcome a cancelled
attempt reports is translated to `failed(timeout)` however the cancellation surfaced —
a rejected read, or the SDK turning it into an `abort` chunk.

Verified: `npm test` 336 pass (6 new in `test/retryPolicy.test.ts`; five in
`test/chatStreamManager.test.ts` covering a retry that succeeds, a transient failure
after content that is *not* retried, a deterministic failure that is not retried,
bounded attempts, and a silent stream that is cancelled and fails as a timeout; plus
the ceiling's two cases in `test/streamChat.test.ts`), `npm run typecheck`,
`npm run check:design`, `npm run build`, `npm run build:unpack` + `smoke:packaged`
(23 checks).

Based on #142's branch: #149 is not merged yet, and this work is independent of it.
@github-actions github-actions Bot added the enhancement New feature or request label Sep 27, 2026
@mrsibe
mrsibe merged commit fc6961b into main Sep 27, 2026
4 checks passed
@mrsibe
mrsibe deleted the feat/stream-reliability branch September 27, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat] Stream reliability guardrails: idle timeout, retry before the first token, configurable output ceiling

1 participant