fix(ollama): num_ctx never set — small local models silently truncated to Ollama's default context window

[bobbyj]

Problem

Tinycode never sends num_ctx to Ollama at any point in the request path (grep -rln "num_ctx" packages/tinycode/src/ returns nothing). This means Ollama serves every local model at its own default runtime context window, completely independent of the model's actual trained capacity.

Reproduced with qwen3.5:9b, whose real trained context length is 262,144 tokens (confirmed via ollama show//api/tags — details.context_length: 262144). With num_ctx unset, the server truncated generation at a 4096-token total window (input+output combined) — roughly 1.5% of the model's real capacity.

Evidence

A large-enough first-turn prompt (~2050 input tokens) caused qwen3.5:9b to hit step_finish reason:"length" with tokens:{"total":4096,"input":2050,"output":2046} — input+output sums exactly to 4096, identically across 3 separate runs spanning 2 different tasks and a reworded retry (rules out prompt-content as the driver; confirms a fixed, deterministic ceiling).

Tinycode's own empty-response handling in packages/tinycode/src/session/prompt.ts:1200-1218,1235 compounds this: when a turn produces no tool call and no captured text (because generation was cut off mid-stream by the context overflow, not because the model was actually silent), it injects a synthetic nudge — literally "Please analyze and respond to the tool results above." — even though no tool call ever occurred. The model's next turn faithfully reacts to that misleading nudge, producing confused meta-commentary that looks like a prompt-engineering failure but is actually a downstream symptom of the context-window bug.

Root cause detail

packages/tinycode/src/provider/local-discovery.ts:181-183 already computes, per-model, from Ollama's own reported details.context_length:

const effectiveContext = contextLimit > 0 ? Math.floor(contextLimit * 0.8) : 0
const outputLimit = contextLimit > 0 ? Math.min(4096, Math.floor(contextLimit * 0.2)) : 0

But effectiveContext is used only for tinycode's own internal token-budget bookkeeping (limit: {context, output}) — it is never transmitted as an actual num_ctx request parameter to Ollama. So tinycode's own internal accounting believes ~209,715 tokens are available for qwen3.5:9b (80% of 262,144), while the server is actually enforcing ~4096 — a ~50x mismatch between what tinycode thinks it has and what it's actually being given.

Confirmed this isn't tinycode's own requested max_tokens (also coincidentally 4096, via maxOutputTokens() in packages/tinycode/src/provider/transform.ts:1277-1279) being the binding constraint: observed output (2046) stopped short of that 4096 client-side request. If the client-side cap were binding, output would have run the full 4096 regardless of input length. It didn't — it stopped exactly where input + output = 4096, which only happens when the server's context window (not the client's requested output ceiling) is the actual limit.

No user-facing workaround exists today either — the provider options config field (packages/tinycode/src/config/provider.ts:61,84) is forwarded into provider/client construction (baseURL, headers) per provider.ts:894-1121, not into per-request generation parameters.

Fix

1. Immediate mitigation: gate the empty-response nudge (prompt.ts:1200-1218,1235) on whether a tool call occurred on a prior turn — stop injecting a "tool results above" message when no tool was ever called. This won't fix the underlying truncation but stops it from producing a confusing, misleading follow-up.
2. Root fix: pass num_ctx explicitly when constructing Ollama requests, using the already-computed effectiveContext value (currently dead-ended at bookkeeping-only use).

Not specific to any one agent persona or prompt shape — any tinycode↔Ollama request with a first-turn prompt large enough to approach the default (not actual) context window is exposed.

Related research

Full investigation writeup with additional verification: ~/projects/ollama-lab/notes/tinycode-scaffolding-benchmark-2026.md (external repo, not in this codebase).

[bobbyj]

Follow-up: architect + critic review of a proposed default (2026-09-01)

Bobby proposed a starting-point default — when a user hasn't configured num_ctx explicitly, default it to 50% of the model's advertised context length. For qwen3.5:9b (context_length 262,144) that computes to 131,072. Ran this through an architect pass (where/how to implement) and a critic pass (is the heuristic itself safe) in parallel. Combined verdict: the 50% default is currently inert (can't reach Ollama at all via the current transport) and, independently, unsafe to ship even if it could.

Architect finding: hard transport blocker

tinycode's Ollama integration routes exclusively through Ollama's OpenAI-compatible endpoint (@ai-sdk/openai-compatible, POST {baseURL}/v1/chat/completions — local-discovery.ts:191-193). Confirmed against Ollama's own docs and server source that this endpoint cannot accept num_ctx at all:

• Ollama's docs state directly: "The OpenAI API does not have a way of setting the context size for a model... create a Modelfile... PARAMETER num_ctx <context size>" — the only supported route is baking it into a new model tag ahead of time, not a per-request parameter.
• Ollama server source (openai/openai.go:105-123, ChatCompletionRequest struct) is a fixed field allowlist with no num_ctx and no generic passthrough. Go's encoding/json silently drops unrecognized keys — a num_ctx field in the request body would be accepted by the HTTP layer and then silently discarded server-side.
• Only Ollama's native /api/chat//api/generate endpoints honor num_ctx (api/types.go:133-153, ChatRequest.Options map[string]any).

tinycode does have a provider-options injection mechanism (transform.ts:1043-1219, precedent: chat_template_kwargs for Qwen) that would mechanically produce {"num_ctx": 131072} in the request body — but it doesn't matter, because the server ignores it on this endpoint regardless of how it's injected.

Real fix requires a native /api/chat request path for Ollama models — a bigger change than a transform.ts field addition: either a new native-protocol provider adapter (replacing api.npm/api.url for Ollama in local-discovery.ts:191-193) with its own request/response translation (native options schema, NDJSON streaming instead of SSE), or an actual Ollama-native AI SDK package if one exists. Medium/high effort.

Critic finding: the 50% figure is unsafe on its own terms too

Independent of the transport blocker, verified against live data from the actual lab Ollama instance (192.168.68.74:11434) that a flat 50%-of-advertised-context default overflows available VRAM — not just for qwen3.5:9b's extreme 262K context, but for an ordinary dense model:

• granite4.2:8b (currently loaded on the lab box): 50% of its 131,072-token advertised context = 65,536 tokens → ~10.24GB of KV cache alone (2 × layers(40) × kv_heads(8) × head_dim(128) × 2 bytes ≈ 160KB/token), before the ~5.3GB model weight footprint. ~15.5GB requested against an 8GB card. Even 25% (~10.5GB total) still overflows.
• No absolute ceiling exists for num_ctx anywhere, unlike the precedent outputLimit = min(4096, contextLimit × 0.2) pattern in local-discovery.ts:181-183 — though that 4096 ceiling turns out to be coincidental (introduced generically, no documented connection to this bug), so it's a reusable shape, not reusable wisdom.
• No failure-mode handling exists if a requested num_ctx doesn't fit VRAM at load time — error.ts's ~15 context-overflow regexes all match request-time "prompt too long" errors; none match a load-time VRAM allocation failure. Suspected (unverified, correctly not tested against the shared lab GPU) real Ollama behavior: silent CPU offload — the worst failure mode, since it presents as "works but is now 10-50x slower" with zero error signal.
• New problem, worse than the original bug: effectiveContext (tinycode's own internal token-budget/compaction accounting, local-discovery.ts:182, computed independently at 80% of advertised context) would silently diverge from any num_ctx fix — tinycode's own UI/compaction logic would still believe it has 209,715 tokens of room for qwen3.5:9b while the server only honors 131,072 (the proposed default), reintroducing the exact "tinycode thinks it has more room than it's actually given" bug this issue is about.

Combined recommendation

1. Ship the nudge-gating mitigation first (prompt.ts:1200-1218,1235 — stop injecting the "tool results above" message when no tool call occurred on a prior turn). Low effort, immediately reduces user-facing confusion, fully decoupled from everything below.
2. Scope the native /api/chat transport work as its own effort — this is the real fix, and it's substantial enough to warrant separate planning rather than folding into a quick default-value change.
3. Do not ship a flat 50%-of-advertised-context default even once the transport exists, without: a VRAM-derived (or explicit hardware-tier) ceiling on top of the percentage, a load-failure detection path (diffing requested vs. actually-loaded context via /api/ps, since Ollama's failure mode here is silent), and effectiveContext rebased on the same number as num_ctx rather than computed independently.

Full agent transcripts available on request; this comment is the compiled summary.

[bobbyjohnstx]

Fixed in the following commits on main:

• auto_profile config schema with max_num_ctx cap (feat(config))
• GPU memory detection for VRAM-aware num_ctx calculation (src/provider/gpu-memory.ts)
• Ollama profile manager: auto-creates derived profiles with baked num_ctx (src/provider/ollama-profile.ts)
• Profile-aware context limits in discovery (src/provider/local-discovery.ts)
• Wired into model selection path (src/provider/provider.ts)
• 36 tests covering GPU detection, profile naming, num_ctx calculation, and profile lifecycle

The fix uses derived Ollama Modelfile profiles (e.g., qwen3.5:9b-tc32k) since /v1/chat/completions ignores per-request num_ctx but respects it when baked into a profile. Profiles are created on-demand, persist across restarts, and are cleaned up when base models are removed.

[bobbyj]

Verification of the fixes that landed (2026-09-01)

Pulled and read the 14 commits that landed since the architect/critic review above (1db0a2ece). Verified both the num_ctx/GPU-profiling work and the separate wiki-skill work in full, against the actual production code paths (not just reading the diffs in isolation).

Wiki gap — fully closed ✅

• New wiki skill (packages/tinycode/src/skill/defaults/wiki/SKILL.md) gives agents an actual judgment layer for the omt_wiki_* tools that previously existed but were never used by any default agent/skill.
• remember skill extended to route codebase-level findings (debugging, decisions, environment quirks) to the wiki specifically, with a config kill switch (experimental.wiki.triage).
• Automatic query-on-start injection (session/llm/request.ts) pulls relevant wiki hints into the system prompt on the first LLM turn, with five real guards: non-subagent only, first-turn only, config-gated, model size >8B excluded (a deliberate, correct call — protects small-model tool-calling reliability, per the original design doc's own concern about this exact risk), and skipped on empty results. Capped at 2KB. Closes upstream issue chore(deps): Bump @solid-primitives/scheduled from 1.5.2 to 1.5.3 #101.

Read-side and write-side both genuinely wired, not just tools sitting unused as before.

num_ctx / this issue — mostly closed, one real gap remains

What's fixed well (gpu-memory.ts, ollama-profile.ts, wired into Provider.getModel in provider.ts):

• Real GPU-memory-aware sizing using the exact KV-cache formula the critic review specified above, budgeted from actual detected memory minus model weight footprint — not a naive percentage of advertised context.
• Implements the workaround the architect review identified as the only real option given the OpenAI-compat transport limitation: creates a Modelfile-derived profile (e.g. qwen3.5:9b-tc18k) via /api/create with a baked-in num_ctx.
• Enforces a max_num_ctx ceiling — addresses the critic's "no absolute ceiling exists" finding directly.
• local-discovery.ts's effectiveContext/outputLimit now correctly derive from the profile's actual baked-in num_ctx when one exists, rather than being computed independently at a different percentage — fixes the critic's "internal accounting will diverge from what the server actually honors" finding.
• Stale profile cleanup runs on rediscovery.
• Confirmed this is genuinely wired into the production model-resolution path (Provider.getModel), not inert.

What's still open:

1. Remote Ollama hosts are explicitly excluded: ensureOllamaProfile guards with if (!isLocalOllama(baseURL)) return modelName — auto-profiling never runs against a non-localhost Ollama instance. This is arguably the right call given gpu-memory.ts only inspects the local machine's hardware (misdetecting a client's own GPU/memory as if it were the remote inference server's would be worse than skipping), but it means the exact deployment pattern that surfaced this bug is still unfixed: this workspace points TINYCODE_OLLAMA_HOST at a remote lab server, not localhost. Extending this to remote hosts would need either a way to query the remote server's actual available memory (if Ollama exposes one) or an explicit user-configured memory budget for non-local hosts.

2. The specific misleading-nudge branch is unchanged. session/prompt.ts:1200-1218,1235 ("Please analyze and respond to the tool results above.", fired when step === 0 with no tool calls and no text) — the code the debugger traced the original confusing-response symptom to — is byte-for-byte identical after this pull. A different, unrelated nudge mechanism ("auto-continue", gated on step > 0, i.e. only after ≥1 tool round already completed) got a real fix on 2026-08-26, but that's a separate code path from the one this bug traced to. The originally recommended mitigation (gate the empty-response nudge on whether a tool call occurred on a prior turn, not just check the current turn) hasn't landed.

Net: the deeper, harder half of this issue (safe, hardware-aware num_ctx sizing via the Modelfile-profile workaround) is well-implemented for local Ollama. The remote-host case and the immediate nudge-text mitigation remain open.

[bobbyj]

Design: extending auto-profiling to remote Ollama hosts (2026-09-01)

Ran an architect review specifically on the remaining gap from the verification comment above — isLocalOllama unconditionally skips auto-profiling for non-localhost Ollama hosts, which is this project's actual primary deployment pattern (remote GPU box via TINYCODE_OLLAMA_HOST).

Why this can't be auto-detected the way local profiling is

Confirmed against Ollama's own API docs: there is no endpoint that reports a server's total/available GPU memory. /api/ps only reports per-loaded-model size/size_vram/expires_at — a post-hoc "did this specific model fit" signal, not a pre-load capacity figure, and it can't answer "how much VRAM is free right now" (multiple models can be loaded/evicted independently). gpu-memory.ts's three real probes (sysctl, nvidia-smi, AMD sysfs) are all local-machine-only by construction — none can inspect a different host's hardware without a remote execution mechanism that doesn't currently exist.

Root cause of the gap

Not a missing detection algorithm — ensureOllamaProfile conflates "not localhost" with "no memory information available," when a user-supplied budget could satisfy the sizing requirement without any hardware probing at all. The existing default_num_ctx / per-model num_ctx overrides (ollama-profile.ts:238-243) already bypass GPU detection entirely today — but they're unreachable for remote hosts because the isLocalOllama guard (ollama-profile.ts:230) bails out before that branch is ever consulted. The gate is checking the wrong condition (hostname) instead of the real one (do we have any trustworthy sizing input at all).

Recommended fix (low effort, unblocks the primary deployment pattern)

1. Extend AutoProfileConfig (ollama-profile.ts:37-43) with remote_gpu_memory_bytes?: number, mirrored in the config schema (config/provider.ts:17-23).
2. Change the guard from if (!isLocalOllama(baseURL)) return modelName to only skip when remote and no sizing input exists at all (no remote_gpu_memory_bytes, no default_num_ctx, no per-model num_ctx):

   const local = isLocalOllama(baseURL)
   const hasRemoteBudget = config?.remote_gpu_memory_bytes !== undefined || config?.default_num_ctx !== undefined ||
     config?.models?.[modelName]?.num_ctx !== undefined
   if (!local && !hasRemoteBudget) return modelName

3. When remote memory-based sizing is used, substitute the configured budget for detectGPUMemory()'s result rather than probing local hardware.

Explicitly rejected: guessing a default for remote hosts (e.g. reusing the local 8GB fallback constant) — this would fabricate a number for hardware nobody inspected, which is strictly worse than skipping, since Ollama's failure mode for an oversized num_ctx is (suspected) silent CPU offload, not a hard error the user would notice.

Optional follow-on, lower priority: SSH-based remote probing (reusing the same probe cascade with commands prefixed by ssh <host>) for convenience when the remote box is already SSH-reachable — deliberately not the primary path, since it introduces new failure modes (auth, jump hosts, firewalls) that would need to fail closed into "no budget," never into a guess.

Safety net: post-load verification via /api/ps (comparing size_vram to size after a profile is first used) as a diagnostic signal if a configured/probed budget overshot — this is the only place Ollama's silent-CPU-offload failure mode becomes observable at all, per the critic's earlier finding on this issue.

Real-world validation done today

Manually replicated exactly the "config-only budget" path (option 1 above) as a workaround, without any code changes — created a Modelfile-derived profile directly via /api/create against the remote lab GPU host (qwen3.5:9b-tc16k, num_ctx: 16384, sized conservatively given head_count_kv is unavailable for this model's hybrid SSM/attention architecture — same gap the critic flagged earlier), then re-ran the architect→executor two-stage benchmark pipeline from earlier in this issue against it.

Result: 2 of 3 tasks flipped from FAIL to PASS (LRU Cache, Binary Search Tree), and — the more direct confirmation — tool-call engagement went from 0/4 to real multi-round tool use on every task. This directly validates the debugger's original root-cause finding: the context-window overflow was driving the "disengaged, asks for tool results" failure mode, not a prompt-framing issue. Dijkstra still failed, but now on a genuine model reasoning bug in the generated code (a broken start not in dist guard) rather than the earlier infrastructure-level failure.

(Also found and fixed a real, separate local config bug in the process: a static provider.ollama.options.baseURL config entry was silently overriding TINYCODE_OLLAMA_HOST for request routing even after discovery correctly found models on the remote host — worth being aware of for anyone else hitting a similar mismatch between "model discovered" and "requests still go elsewhere.")