Issue #65 led to a useful provider policy: bin/llm remains the sole completion path, OpenAI-compatible endpoints are handled generically, and provider-specific integration does not spread through the rest of Headlong.
That solves local-provider support cleanly, but it exposes a separate limitation documented in design/providers.md: LLM_PROVIDER is process-wide. A deployment therefore cannot naturally use a cheap/local model for routine cognition while escalating selected calls to a frontier model on a different provider.
I think this should be solved with a model-routing abstraction above the provider layer, without changing the new provider policy.
Desired property
Callers should select a route (or profile), not a provider:
routine cognition / recap / retrieval -> local route -> openai-compatible -> Ollama/vLLM/LM Studio
hard reasoning / important output -> frontier route -> anthropic/openai/etc.
bin/llm would still be the only code that knows how to call providers. shellm, thinkers, recap, memory search, etc. would only optionally say which named route they want.
Minimal interface
For example:
llm --route local "summarize this"
llm --route frontier "analyze this carefully"
A route would resolve to the existing provider configuration fields:
provider
model
api_url (optional)
Credentials should continue to come from the existing environment/provider mechanisms rather than being stored in route definitions.
One possible representation would be small per-route env files under the Headlong state directory, e.g.:
~/.headlong/routes/local.env
~/.headlong/routes/frontier.env
local.env:
LLM_PROVIDER=openai-compatible
LLM_API_URL=http://localhost:11434/v1/chat/completions
LLM_MODEL=qwen3:8b
frontier.env:
LLM_PROVIDER=anthropic
LLM_MODEL=claude-sonnet-4-7
The exact storage format is less important than the boundary: route resolution happens in bin/llm; provider handling remains exactly where it is now.
Suggested precedence for a routed call:
- explicit CLI options (
--provider, --model, LLM_API_URL equivalent if exposed)
- selected route/profile
- ordinary process-wide environment/defaults
That keeps routes as defaults rather than a second provider system.
Why this matters for Headlong specifically
Persistent agency creates a very different cost profile from request/response agents. Many continuous operations do not necessarily require a frontier model:
- background thought
- retrieval
- memory search/consolidation
- recap / trajectory compression
- classification
- monitoring
- routine planning
A local model can plausibly handle much of that work at marginal hardware/electricity cost, while a frontier route can be reserved for difficult reasoning, high-impact external output, or explicit escalation.
The architectural distinction is useful even apart from cost: provider answers “how do I call this endpoint?”; route answers “which model resource should this cognitive operation use?”
Important constraint
This proposal should not:
- add provider logic outside
bin/llm
- reintroduce provider inference from model names
- create Ollama/LM Studio/vLLM-specific branches
- require a routing proxy in front of Headlong
- make every caller understand provider URLs or credentials
The new openai-compatible provider and adapter seam remain unchanged.
First-stage scope
I would keep the first implementation intentionally mechanical rather than autonomous:
bin/llm gains a named route/profile selection mechanism.- A route resolves to provider/model/API URL configuration for that call.
- Existing provider execution, streaming, retries, usage ledger and health handling run unchanged after resolution.
- Callers may opt into a route explicitly.
- The usage ledger records the selected route in addition to provider/model when one was used.
No automatic escalation policy is required initially.
Later possibility: escalation policy
Once named routes exist, a thinker could choose to escalate:
local model
-> sufficient confidence: continue
-> insufficient confidence / high-impact task: invoke frontier route
That policy should be separate from the routing primitive. The trajectory could eventually record why an escalation occurred and whether the frontier result changed the proposed action.
Acceptance criterion
A single Headlong deployment can make both of these calls in the same process/environment:
llm --route local "summarize the last trajectory block"
llm --route frontier "analyze whether this plan has a serious flaw"
where local resolves to an OpenAI-compatible local endpoint and frontier resolves to a different existing provider, with no provider-specific code added outside bin/llm.
This is intended as a complement to the provider policy established in design/providers.md, not a replacement for it.
Issue #65 led to a useful provider policy:
bin/llmremains the sole completion path, OpenAI-compatible endpoints are handled generically, and provider-specific integration does not spread through the rest of Headlong.That solves local-provider support cleanly, but it exposes a separate limitation documented in
design/providers.md:LLM_PROVIDERis process-wide. A deployment therefore cannot naturally use a cheap/local model for routine cognition while escalating selected calls to a frontier model on a different provider.I think this should be solved with a model-routing abstraction above the provider layer, without changing the new provider policy.
Desired property
Callers should select a route (or profile), not a provider:
bin/llmwould still be the only code that knows how to call providers.shellm, thinkers, recap, memory search, etc. would only optionally say which named route they want.Minimal interface
For example:
A route would resolve to the existing provider configuration fields:
Credentials should continue to come from the existing environment/provider mechanisms rather than being stored in route definitions.
One possible representation would be small per-route env files under the Headlong state directory, e.g.:
local.env:frontier.env:The exact storage format is less important than the boundary: route resolution happens in
bin/llm; provider handling remains exactly where it is now.Suggested precedence for a routed call:
--provider,--model,LLM_API_URLequivalent if exposed)That keeps routes as defaults rather than a second provider system.
Why this matters for Headlong specifically
Persistent agency creates a very different cost profile from request/response agents. Many continuous operations do not necessarily require a frontier model:
A local model can plausibly handle much of that work at marginal hardware/electricity cost, while a frontier route can be reserved for difficult reasoning, high-impact external output, or explicit escalation.
The architectural distinction is useful even apart from cost: provider answers “how do I call this endpoint?”; route answers “which model resource should this cognitive operation use?”
Important constraint
This proposal should not:
bin/llmThe new
openai-compatibleprovider and adapter seam remain unchanged.First-stage scope
I would keep the first implementation intentionally mechanical rather than autonomous:
bin/llmgains a named route/profile selection mechanism.No automatic escalation policy is required initially.
Later possibility: escalation policy
Once named routes exist, a thinker could choose to escalate:
That policy should be separate from the routing primitive. The trajectory could eventually record why an escalation occurred and whether the frontier result changed the proposed action.
Acceptance criterion
A single Headlong deployment can make both of these calls in the same process/environment:
where
localresolves to an OpenAI-compatible local endpoint andfrontierresolves to a different existing provider, with no provider-specific code added outsidebin/llm.This is intended as a complement to the provider policy established in
design/providers.md, not a replacement for it.