Simplify inference routing: introduce inference.local and remove implicit catch-all

[pimlock]

Summary

Simplify the inference routing system by removing the implicit catch-all mechanism and replacing it with an explicit inference.local hostname addressable inside every sandbox. Inference configuration moves from per-route CRUD to cluster-level config backed by the existing provider system.

Context

The current inference routing has two paths:

1. Direct allow — Network policy explicitly allows traffic to a specific endpoint (e.g., api.anthropic.com). Works for any endpoint, not inference-specific.
2. Implicit catch-all — Requests that aren't directly allowed but are detected as inference calls get silently routed through the privacy router to a configured backend.

The catch-all is confusing. A typo in a policy (e.g., api.entropics.com instead of api.anthropic.com) silently reroutes inference to the local model instead of failing visibly. As John put it: "explicit policies for allowances and then we have this implicit secret inference catch-all which breaks the mental model."

Decisions

1. Remove the implicit catch-all — No more inspect_for_inference OPA action. If a request isn't explicitly allowed, it's denied.
2. Introduce inference.local — An always-addressable hostname inside every sandbox that routes through our inference router. No credentials needed from the agent's perspective.
3. inference.local defaults to managed NVIDIA inference — If no local model is deployed (e.g., on Brev/CPU), the router points to managed NVIDIA endpoints. When a local model is available, it switches over.
4. Direct allow unchanged — Explicit network policy allows (e.g., Claude → Anthropic) continue as-is. The router is for "your custom agent" inference.
5. Single model override — Router rewrites the model name from client to whatever is configured. Client-specified model is ignored.
6. Cluster-level inference config — How inference.local routes is configured at the cluster level, not per-sandbox. Config is: provider name + model name.
7. Providers as the credential mechanism — Instead of routes carrying API keys, use the existing provider system for secure credential injection. New providers: openai, anthropic, nvidia (all API-key-only for now). Related: Inference route API keys stored in plain object store #21, Inference route API keys exposed via ListInferenceRoutes #20.
8. Credential injection at supervisor level — Still planned independently of router changes.

Router Flow

1. Request from agent hits inference.local
2. Router detects the inference API format (OpenAI, Anthropic, etc.)
3. Router fetches cluster inference config via gRPC (cached, periodically refreshed) → gets provider name + model name
4. Router fetches provider credentials (API key) via the provider system
5. Router makes the upstream request with the correct API key and model override
6. API format translation (e.g., OpenAI ↔ Anthropic) is out of scope — handled in feat(router): add inference API translation between protocols #90

User Flow

# 1. Create a provider with credentials
nemoclaw provider create --name nvidia_build --type nvidia --from-existing

# 2. Configure cluster-level inference
nemoclaw cluster inference set --provider nvidia_build --model llama-3.1-8b

# 3. Inside any sandbox, agent hits inference.local — just works
curl http://inference.local/v1/chat/completions \
  -d '{"model": "anything", "messages": [...]}'
# model is overwritten to llama-3.1-8b, routed to nvidia_build with injected API key

Implementation

Remove

• Remove nemoclaw inference create/update/delete/list CLI commands
• Remove inference route gRPC RPCs (CreateInferenceRoute, UpdateInferenceRoute, DeleteInferenceRoute, ListInferenceRoutes, GetInferenceRoute)
• Remove InferenceRoute/InferenceRouteSpec data model from proto (or deprecate)
• Remove inspect_for_inference OPA action and the implicit catch-all code path in the sandbox proxy
• Remove routing_hint concept and route-level API key storage

Add

• Add inference.local DNS/hostname resolution inside the sandbox (resolve to the router)
• Add cluster-level inference configuration (proto fields, storage, gRPC endpoint)
• Add nemoclaw cluster inference set/get CLI commands (provider name + model name)
• Create openai, anthropic, and nvidia providers
  • Note: nvidia one already exists.
• Update the router to read cluster config (cached + refreshed) and fetch provider credentials
• Default route: managed NVIDIA inference when no local model is present
• Drop a skill/instructions in the sandbox telling agents about inference.local

Update

• Update policy files (dev-sandbox-policy.yaml, policy-local.yaml, policy-frontier.yaml, etc.) to remove inference catch-all rules
• Update OPA rego rules — simplify to allow/deny (no tri-state)
• Update architecture docs (architecture/security-policy.md, etc.)

GTC Priorities

• Primary: Awesome Brev cloud experience with managed endpoints (most users won't have Spark hardware)
• Secondary: In-cluster local model on Spark (aim for it, don't let it block)

Related Issues

• Inference route API keys stored in plain object store #21 — Provider configuration
• Inference route API keys exposed via ListInferenceRoutes #20 — Provider secrets
• feat(router): add inference API translation between protocols #90 — API format translation (OpenAI ↔ Anthropic)

[pimlock]

🏗️ build-plan

Implementation Plan

Issue type: feat
Complexity: High
Confidence: Medium-High — requirements are explicit; work still spans proto/server/sandbox/CLI/tests.

Summary

Keep the existing InferenceRoute entity, gateway route CRUD RPCs, and bundle delivery RPC, while re-homing operator UX to nemoclaw cluster inference set/get. Runtime routing becomes explicit through inference.local; the bundle effectively resolves to one managed route item. As part of this simplification, support GET /v1/models (and model-discovery flow) for OpenAI/Anthropic-compatible providers.

Scope

• proto/inference.proto: retain route CRUD + bundle RPCs; evolve schema for managed singleton behavior (drop routing_hint, add version/generation metadata).
• crates/navigator-server/src/inference.rs: keep service/entity; enforce singleton managed route semantics and monotonic version updates.
• crates/navigator-sandbox/src/grpc_client.rs + crates/navigator-sandbox/src/lib.rs: continue bundle fetch path; consume single-item bundle and cache revision/version.
• crates/navigator-sandbox/src/proxy.rs: explicit inference.local handling for CONNECT and absolute-form HTTP proxy requests; no DNS/hosts injection support.
• crates/navigator-sandbox/src/l7/inference.rs (or replacement path matcher in proxy): add support for GET /v1/models routing in explicit inference flow.
• crates/navigator-sandbox/src/opa.rs + crates/navigator-sandbox/data/sandbox-policy.rego: simplify to allow/deny only and remove implicit catch-all interception path.
• crates/navigator-router/src/backend.rs (+ crates/navigator-router/src/mock.rs tests): preserve provider-backed auth and forced model override for generation calls; allow model-list passthrough semantics for GET /v1/models.
• crates/navigator-cli/src/main.rs + crates/navigator-cli/src/run.rs: deprecate/remove standalone nemoclaw inference ...; expose nemoclaw cluster inference set/get only. get returns provider + model only.
• crates/navigator-providers/src/lib.rs (+ provider plugins): support openai and anthropic (API-key providers) with existing nvidia.
• python/navigator/sandbox.py + python/navigator/__init__.py: adjust SDK surface away from standalone route CRUD exposure.
• e2e/python/test_inference_routing.py + e2e/python/conftest.py: assert inference.local flow, singleton bundle behavior, and v1/models support.
• Internal architecture/ docs + policy/example files: update implementation docs (internal-only note about managed InferenceRoute backing).

Implementation Steps

1. Update proto definitions to preserve InferenceRoute/CRUD/bundle RPCs while reshaping fields for singleton cluster config semantics (including version metadata and removal of routing_hint).
2. Implement server-side managed singleton route behavior (internal key/name, e.g. inference.local) and monotonic version increments on mutation.
3. Add/validate provider support for openai and anthropic across provider registry/CLI flows.
4. Re-home CLI to cluster inference set/get; remove/deprecate standalone inference command UX.
5. Keep bundle delivery architecture and ensure response resolves to a single route item for configured cluster inference.
6. Implement explicit inference.local proxy routing (CONNECT + absolute-form HTTP); do not add DNS alias/hosts injection for non-proxy-aware clients.
7. Add explicit GET /v1/models support in the inference routing path so model discovery works for OpenAI/Anthropic-compatible providers.
8. Keep set-time validation simple: validate config shape/provider existence synchronously; do not hard-fail on live upstream probes in this iteration (optionally add best-effort probe as future enhancement).
9. Simplify OPA/proxy decisioning to explicit allow/deny only and remove implicit inspect/routing-hint catch-all.
10. Return actionable HTTP errors when cluster inference provider/model is not configured, including exact setup command and docs link.
11. Update internal docs/policies/tests and align wording with parallel user-docs workstream (without exposing internal entity details externally).

Test Plan

• Unit tests: singleton route validation, version increment semantics, provider credential resolution, forced model override on generation endpoints, allow/deny policy behavior.
• Integration tests: route CRUD still functions internally, cluster inference set/get UX mapping, bundle resolves to one route item, proxy inference.local handling for CONNECT + non-CONNECT proxy HTTP, and GET /v1/models pass-through.
• E2E tests: configured provider/model routes through inference.local; GET /v1/models succeeds for supported providers; missing config returns actionable error; explicit direct endpoint allow behavior remains unchanged.

Risks & Open Questions

• Final wire shape for version metadata (field placement on InferenceRoute vs InferenceRouteSpec vs bundle response).
• CLI deprecation path details for legacy standalone inference commands (hard remove vs hidden alias + warning).
• Provider credential key normalization for openai/anthropic.
• Whether to add optional cluster inference set --validate probe in a follow-up (best-effort connectivity/model check without making config writes brittle).
• Ensure external docs do not leak internal managed-route implementation details.

Documentation Impact

• Internal docs: architecture/inference-routing.md, architecture/security-policy.md, architecture/sandbox.md, architecture/README.md.
• Policy/example files: policy-*.yaml, deploy/docker/sandbox/dev-sandbox-policy.yaml, examples/inference/*.
• External user docs: update commands/behavior (inference.local, cluster inference, v1/models support), but keep managed-InferenceRoute backing internal.

---

Revision 4 — added v1/models support requirement, clarified simple set-time validation stance, kept singleton managed InferenceRoute internals
Revision 3 — keep InferenceRoute + CRUD/bundle internals; deprecate standalone CLI UX; switch to singleton managed route semantics with no routing_hint and single-item bundle
Revision 2 — incorporated maintainer clarifications: hard compatibility break accepted; no automatic provider fallback; inference.local handled at proxy layer; added exploration result for versioned server-side cluster inference config entity
Revision 1 — initial plan

[pimlock]

🏗️ build-from-issue-agent

Captured implementation decisions from planning Q&A for future reference:

Q1) Backward compatibility for removing inference route CRUD RPCs/SDK

Decision: Hard break is acceptable now. No migration window is required.
Operational note: Notify internal channel about the breaking API/SDK change.

Q2) Fallback behavior when local model is unavailable

Decision: No automatic fallback based on local model presence/absence.
Behavior: Routing follows the provider configured at cluster level.
Failure mode: If no provider/config is set, return a clear actionable HTTP error with setup guidance (and docs link).

Q3) How should inference.local be made resolvable?

Decision: Handle this at the proxy layer only; no DNS/hosts injection needed.
Rationale: In proxy mode, requests are routed via HTTP_PROXY/HTTPS_PROXY and the proxy can match inference.local directly from CONNECT/absolute-form request targets.
Non-goal: Supporting non-proxy-aware clients via DNS aliasing is out of scope.

Q4) Should cluster metadata be versioned? How is metadata represented today?

Current state: Existing ClusterMetadata is local CLI/bootstrap metadata stored in user config (~/.config/nemoclaw/...) and is not server-side cluster runtime config.
Decision: Add a new server-side cluster inference config entity with explicit versioning (monotonic version increment on updates), instead of reusing bootstrap metadata.

Q5) Documentation coordination

Decision: Keep architecture/docs updates in lockstep with implementation and align with the parallel user-docs workstream.

[pimlock]

🏗️ build-from-issue-agent

Follow-up decision capture (supersedes part of prior Q&A note):

• Keep InferenceRoute as the underlying gateway entity.
• Keep gateway inference CRUD RPCs and sandbox bundle delivery RPC.
• Re-home user-facing CLI to nemoclaw cluster inference set/get; standalone nemoclaw inference ... is deprecated/removed as the UX surface.
• routing_hint is dropped from the new model; routing is explicit via inference.local.
• Managed singleton route approach is accepted (internal key/name only; not exposed externally).
• Sandbox bundle is expected to resolve to a single route item in this design.
• External docs should describe behavior/commands only; managed-route backing stays internal.

[pimlock]

🔍 codebase-exploration (Revision 2)

Post-implementation audit: leftover dead code inventory, router/provider architecture design, and phased implementation plan.

https://gist.github.com/pimlock/fca000f2caa8ad803cc20e59a83a74d9

Key design decisions (from discussion):

• Router stays provider-agnostic — auth style carried explicitly on ResolvedRoute via AuthHeader enum
• Route resolution stays in the caller (server/sandbox), not in the router
• CLI uses cluster inference via gRPC → server → router (no direct router import)
• New InferenceProviderProfile in navigator-core consolidates provider-specific inference knowledge (base URL, protocols, auth style, credential keys)
• ProviderPlugin (credential discovery) and InferenceProviderProfile (API protocol) are separate concerns

[pimlock]

🏗️ build-from-issue-agent

Additional planning decisions captured:

v1/models support

• Include GET /v1/models in the explicit inference.local routing flow.
• This is part of the simplification because model discovery is important and is supported by OpenAI/Anthropic-compatible providers.
• The sandbox bundle remains a single managed route item; this endpoint should still work through that route.

Set-time validation strategy

• Keep this iteration simple and robust:
  • Validate provider/model config shape and provider existence synchronously at set time.
  • Do not hard-fail config writes on live upstream probe failures (network/provider transient issues can make writes brittle).
• Potential follow-up: optional best-effort validation flag (for example, probing model-list endpoint) that reports warnings/errors without changing the default reliability posture.

[johntmyers]

We will want to make sure that updates to the NemoClaw CLI skill and Policy Generation skill are handled as well.