From d934a768aaf426debc733b7f32ef0cafe5d4f811 Mon Sep 17 00:00:00 2001 From: Kris Hagel Date: Mon, 10 Aug 2026 17:31:06 -0700 Subject: [PATCH 1/2] fix(nexus): exclude models with unconfigured provider credentials from routed selection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of the nondeterministic nexus-chat-pii-passthrough.functional E2E failures (diagnosed 2026-08-10): in AUTO mode the Nexus model router's configured candidate lists can contain OpenAI (or Google/Azure/Latimer) models. On machines where that provider's API key is not configured (local dev: .env.local has no provider keys and the settings table is empty), the router still selected the model, provider creation then threw "OpenAI API key not configured" deep in the streaming path, and POST /api/nexus/chat 500'd with "Failed to process chat request". The router's fallbackModelId cannot catch this because the error occurs after routing, at stream time. The failure was nondeterministic because the nova-micro classifier's tier/intent output varies per run, changing which candidate wins. Fix (option "router excludes unconfigured families", chosen over fallback-on-missing-key — which would need a retry wrapped around the entire stream-setup path — and over pinning a family in the spec, which would mask the same 500 for real local AUTO-mode users): - lib/ai/provider-credentials.ts (new): getConfiguredChatProviders() probes OpenAI/Google/Azure/Latimer credentials through the settings manager (database-first, environment fallback, existing 5-minute cache). amazon-bedrock is always treated as configured because it can authenticate through the ambient AWS credential chain (ECS/Lambda IAM roles), which cannot be probed here. Fails open on probe errors so a settings outage cannot take chat down. - lib/nexus/model-router/router.ts: routeNexusRequest() filters the model pool to configured providers before any routed selection (text, image specialist, and required-tools enforcement paths all inherit the filter), logging the excluded providers. The fallback model — the client's explicitly selected model — is deliberately looked up in the UNFILTERED list, so an explicit selection of an unconfigured provider keeps its honest configuration error instead of being silently rerouted; router-off mode and shadow-mode legacy execution are unchanged. Tests: - lib/ai/__tests__/provider-credentials.test.ts (new): bedrock-only default, full-set when all keys present, whitespace keys rejected, Azure requires key AND resource name, fail-open on probe error. - lib/nexus/model-router/__tests__/router.test.ts: AUTO routing skips an unconfigured-provider candidate; router-off keeps the explicit model without its provider key; Advanced family with unconfigured provider fails with the clear family error; image intent does not route through an unconfigured provider. Existing 19 router tests unchanged (the new credential mock defaults to all-configured). Verification: - bunx jest lib/nexus lib/ai: 35 suites, 298 tests pass. - bun run lint and bun run typecheck: clean over the entire codebase. - Authenticated E2E: nexus-chat-pii-passthrough.functional.spec.ts passed 4/4 with --repeat-each=4 and retries disabled via scripts/test/e2e-local.sh. - Environment probe on the dev machine confirms getConfiguredChatProviders() returns only amazon-bedrock, so local AUTO routing is now deterministic (Bedrock-only) instead of classifier-dependent. docs/features/nexus-model-routing.md: request-flow step 5 now documents the credential filter and the explicit-selection carve-out. --- docs/features/nexus-model-routing.md | 2 +- lib/ai/__tests__/provider-credentials.test.ts | 62 ++++++++++++++++ lib/ai/provider-credentials.ts | 39 +++++++++++ .../model-router/__tests__/router.test.ts | 70 +++++++++++++++++++ lib/nexus/model-router/router.ts | 25 ++++++- 5 files changed, 195 insertions(+), 3 deletions(-) create mode 100644 lib/ai/__tests__/provider-credentials.test.ts create mode 100644 lib/ai/provider-credentials.ts diff --git a/docs/features/nexus-model-routing.md b/docs/features/nexus-model-routing.md index deb46619a..dfd804ce5 100644 --- a/docs/features/nexus-model-routing.md +++ b/docs/features/nexus-model-routing.md @@ -8,7 +8,7 @@ Nexus defaults to **Standard** mode. Users see one Nexus experience instead of a 2. Apply the existing K-12 input guardrail before classifier traffic without rewriting allowed request text. 3. Apply deterministic capability rules for image generation, PSD-data, web search/current information, and common instructional requests. 4. Send ambiguous requests to Amazon Nova Micro on Bedrock for a provider-neutral `intent`, `tier`, `confidence`, and reason codes. -5. Resolve an accessible, active Nexus model from the configured ordered candidates. If none is configured, use `providerMetadata.nexusRouterTier` or model-name conventions. If no tier match is available, use the closest tier in the requested family. Auto may finally use the existing client model as a safe fallback; an explicit Advanced family never silently crosses into another family. +5. Resolve an accessible, active Nexus model from the configured ordered candidates. Models whose provider credentials are not configured (checked through the settings manager with environment fallback; Bedrock always counts as configured because it can use the ambient AWS credential chain) are excluded from routed selection, so routing never picks a model whose provider creation would fail at stream time. If none is configured, use `providerMetadata.nexusRouterTier` or model-name conventions. If no tier match is available, use the closest tier in the requested family. Auto may finally use the existing client model as a safe fallback — an explicitly selected client model keeps its honest provider-configuration error rather than being silently rerouted; an explicit Advanced family never silently crosses into another family. 6. Automatically select an image-capable model for image intent, attach the existing database-backed PSD-data MCP server for PSD-data intent, or select a web-search-capable Gemini model and enable its native Google Search tool for web-search intent. 7. Persist the route decision on assistant-message metadata and expose it in `X-Nexus-Routing` for evaluation. diff --git a/lib/ai/__tests__/provider-credentials.test.ts b/lib/ai/__tests__/provider-credentials.test.ts new file mode 100644 index 000000000..6d132fd38 --- /dev/null +++ b/lib/ai/__tests__/provider-credentials.test.ts @@ -0,0 +1,62 @@ +/** @jest-environment node */ + +const mockGetOpenAI = jest.fn() +const mockGetGoogleAI = jest.fn() +const mockGetAzureOpenAI = jest.fn() +const mockGetLatimer = jest.fn() + +jest.mock("@/lib/settings-manager", () => ({ + Settings: { + getOpenAI: () => mockGetOpenAI(), + getGoogleAI: () => mockGetGoogleAI(), + getAzureOpenAI: () => mockGetAzureOpenAI(), + getLatimer: () => mockGetLatimer(), + }, +})) +jest.mock("@/lib/logger", () => ({ + createLogger: () => ({ info: jest.fn(), warn: jest.fn(), error: jest.fn(), debug: jest.fn() }), +})) + +import { getConfiguredChatProviders } from "../provider-credentials" + +describe("getConfiguredChatProviders", () => { + beforeEach(() => { + jest.clearAllMocks() + mockGetOpenAI.mockResolvedValue(null) + mockGetGoogleAI.mockResolvedValue(null) + mockGetAzureOpenAI.mockResolvedValue({ key: null, endpoint: null, resourceName: null }) + mockGetLatimer.mockResolvedValue(null) + }) + + it("always includes amazon-bedrock even with no keys configured", async () => { + const configured = await getConfiguredChatProviders() + expect(configured).toEqual(new Set(["amazon-bedrock"])) + }) + + it("includes every provider whose credential is present", async () => { + mockGetOpenAI.mockResolvedValue("sk-test") + mockGetGoogleAI.mockResolvedValue("google-key") + mockGetAzureOpenAI.mockResolvedValue({ key: "azure-key", endpoint: null, resourceName: "resource" }) + mockGetLatimer.mockResolvedValue("latimer-key") + const configured = await getConfiguredChatProviders() + expect(configured).toEqual(new Set(["openai", "google", "amazon-bedrock", "azure", "latimer"])) + }) + + it("treats whitespace-only keys as not configured", async () => { + mockGetOpenAI.mockResolvedValue(" ") + const configured = await getConfiguredChatProviders() + expect(configured.has("openai")).toBe(false) + }) + + it("requires both the Azure key and resource name", async () => { + mockGetAzureOpenAI.mockResolvedValue({ key: "azure-key", endpoint: null, resourceName: null }) + const configured = await getConfiguredChatProviders() + expect(configured.has("azure")).toBe(false) + }) + + it("fails open when the credential probe throws", async () => { + mockGetOpenAI.mockRejectedValue(new Error("settings unavailable")) + const configured = await getConfiguredChatProviders() + expect(configured).toEqual(new Set(["openai", "google", "amazon-bedrock", "azure", "latimer"])) + }) +}) diff --git a/lib/ai/provider-credentials.ts b/lib/ai/provider-credentials.ts new file mode 100644 index 000000000..14172715f --- /dev/null +++ b/lib/ai/provider-credentials.ts @@ -0,0 +1,39 @@ +import { Settings } from "@/lib/settings-manager" +import { createLogger } from "@/lib/logger" + +const log = createLogger({ module: "provider-credentials" }) + +const ALL_CHAT_PROVIDERS = ["openai", "google", "amazon-bedrock", "azure", "latimer"] as const + +/** + * Returns the lowercase provider identifiers whose credentials are configured + * (database settings with environment fallback, cached by the settings + * manager), so routing can skip models that would fail provider creation. + * + * amazon-bedrock is always included: it can authenticate through the default + * AWS credential chain (ECS/Lambda IAM roles), which cannot be probed here. + * + * Fails open — a probe failure must not take chat down, so every provider is + * treated as configured and a missing key surfaces at stream time as before. + */ +export async function getConfiguredChatProviders(): Promise> { + try { + const [openAiKey, googleKey, azure, latimerKey] = await Promise.all([ + Settings.getOpenAI(), + Settings.getGoogleAI(), + Settings.getAzureOpenAI(), + Settings.getLatimer(), + ]) + const configured = new Set(["amazon-bedrock"]) + if (openAiKey?.trim()) configured.add("openai") + if (googleKey?.trim()) configured.add("google") + if (azure.key?.trim() && azure.resourceName?.trim()) configured.add("azure") + if (latimerKey?.trim()) configured.add("latimer") + return configured + } catch (error) { + log.warn("Provider credential probe failed; treating every provider as configured", { + error: error instanceof Error ? error.message : String(error), + }) + return new Set(ALL_CHAT_PROVIDERS) + } +} diff --git a/lib/nexus/model-router/__tests__/router.test.ts b/lib/nexus/model-router/__tests__/router.test.ts index 6356e0ae9..dc37592f2 100644 --- a/lib/nexus/model-router/__tests__/router.test.ts +++ b/lib/nexus/model-router/__tests__/router.test.ts @@ -5,8 +5,12 @@ const mockFilterAccessibleResourceIds = jest.fn() const mockGetConfig = jest.fn() const mockClassify = jest.fn() const mockExecuteQuery = jest.fn() +const mockGetConfiguredChatProviders = jest.fn() jest.mock("@/lib/db/drizzle", () => ({ getNexusEnabledModels: () => mockGetNexusEnabledModels() })) +jest.mock("@/lib/ai/provider-credentials", () => ({ + getConfiguredChatProviders: () => mockGetConfiguredChatProviders(), +})) jest.mock("@/lib/db/drizzle/resource-access", () => ({ filterAccessibleResourceIds: (...args: unknown[]) => mockFilterAccessibleResourceIds(...args), })) @@ -51,6 +55,9 @@ function defineNexusModelRouterSuite1Part1() { mockGetNexusEnabledModels.mockResolvedValue(models) mockFilterAccessibleResourceIds.mockResolvedValue(models.map(model => String(model.id))) mockGetConfig.mockResolvedValue({ config, mode: "active" }) + mockGetConfiguredChatProviders.mockResolvedValue( + new Set(["openai", "google", "amazon-bedrock", "azure", "latimer"]) + ) mockClassify.mockResolvedValue({ intent: "general", tier: "medium", confidence: 0.9, reasonCodes: ["normal_request"], source: "classifier", @@ -343,3 +350,66 @@ const defineNexusModelRouterSuite1 = () => { }; describe("Nexus model router", defineNexusModelRouterSuite1) + +describe("Nexus model router credential filtering", () => { + beforeEach(() => { + jest.clearAllMocks() + mockGetNexusEnabledModels.mockResolvedValue(models) + mockFilterAccessibleResourceIds.mockResolvedValue(models.map(model => String(model.id))) + mockGetConfig.mockResolvedValue({ config, mode: "active" }) + mockClassify.mockResolvedValue({ + intent: "general", tier: "medium", confidence: 0.9, + reasonCodes: ["normal_request"], source: "classifier", + }) + }) + + it("routes AUTO past a candidate whose provider credential is not configured", async () => { + const autoConfig = nexusRouterConfigSchema.parse({ + auto: { light: [], medium: ["gpt-terra", "us.anthropic.claude-sonnet"], high: [] }, + }) + mockGetConfig.mockResolvedValue({ config: autoConfig, mode: "active" }) + mockGetConfiguredChatProviders.mockResolvedValue(new Set(["amazon-bedrock", "google"])) + + const result = await routeNexusRequest({ + text: "Help", fallbackModelId: "gpt-terra", experienceMode: "standard", + requestedFamily: "auto", enabledConnectorIds: [], userId: 7, + }) + + expect(result.modelId).toBe("us.anthropic.claude-sonnet") + expect(result.metadata.selectedFamily).toBe("anthropic") + }) + + it("keeps the explicitly selected model when routing is off, even without its provider key", async () => { + mockGetConfig.mockResolvedValue({ config, mode: "off" }) + mockGetConfiguredChatProviders.mockResolvedValue(new Set(["amazon-bedrock", "google"])) + + const result = await routeNexusRequest({ + text: "Help", fallbackModelId: "gpt-terra", experienceMode: "standard", + requestedFamily: "auto", enabledConnectorIds: [], userId: 7, + }) + + expect(result.modelId).toBe("gpt-terra") + }) + + it("fails clearly when the requested Advanced family's provider is not configured", async () => { + mockGetConfiguredChatProviders.mockResolvedValue(new Set(["amazon-bedrock", "google"])) + + await expect(routeNexusRequest({ + text: "Help", fallbackModelId: "us.anthropic.claude-sonnet", experienceMode: "advanced", + requestedFamily: "openai", enabledConnectorIds: [], userId: 7, + })).rejects.toThrow("openai family") + }) + + it("does not offer image generation through an unconfigured provider", async () => { + mockClassify.mockResolvedValue({ + intent: "image", tier: "medium", confidence: 0.99, + reasonCodes: ["explicit_image_request"], source: "deterministic", + }) + mockGetConfiguredChatProviders.mockResolvedValue(new Set(["amazon-bedrock", "openai"])) + + await expect(routeNexusRequest({ + text: "Create an image", fallbackModelId: "gpt-terra", experienceMode: "standard", + requestedFamily: "auto", enabledConnectorIds: [], userId: 7, + })).rejects.toThrow("Image generation is not available") + }) +}) diff --git a/lib/nexus/model-router/router.ts b/lib/nexus/model-router/router.ts index 1b1a267d3..e91da9c4a 100644 --- a/lib/nexus/model-router/router.ts +++ b/lib/nexus/model-router/router.ts @@ -5,6 +5,7 @@ import { nexusMcpServers } from "@/lib/db/schema" import { filterAccessibleResourceIds } from "@/lib/db/drizzle/resource-access" import { createLogger } from "@/lib/logger" import { hasCapability } from "@/lib/ai/capability-utils" +import { getConfiguredChatProviders } from "@/lib/ai/provider-credentials" import { inferModelFamily, inferModelTier, @@ -422,17 +423,37 @@ export async function routeNexusRequest( "model", models.map(model => model.id) )) + // The fallback (the model the client explicitly selected) is looked up in + // the unfiltered list on purpose: an explicit selection keeps its honest + // provider-configuration error instead of being silently rerouted. const fallback = models.find(model => model.modelId === args.fallbackModelId || String(model.id) === args.fallbackModelId ) if (!fallback) throw new Error("The fallback Nexus model is unavailable") + // Routed selection must never pick a model whose provider credential is not + // configured — provider creation would throw at stream time and fail the + // whole request, which the router's fallback cannot catch. + const configuredProviders = await getConfiguredChatProviders() + const routableModels = models.filter(model => + configuredProviders.has(model.provider.toLowerCase()) + ) + if (routableModels.length < models.length) { + const excludedProviders = [...new Set( + models + .filter(model => !configuredProviders.has(model.provider.toLowerCase())) + .map(model => model.provider) + )] + log.info("Excluding models from routing; provider credentials not configured", { + excludedProviders, + }) + } const requiredTools = [...new Set(args.enabledToolNames ?? [])] if (mode === "off") { return buildRouterOffResult({ args, config, - models, + models: routableModels, fallback, accessibleIds, requiredTools, @@ -442,7 +463,7 @@ export async function routeNexusRequest( args, config, mode, - models, + models: routableModels, fallback, accessibleIds, requiredTools, From 60c8140e44e93f2bea2bb04680ecf9c29a29e566 Mon Sep 17 00:00:00 2001 From: Kris Hagel Date: Mon, 10 Aug 2026 19:22:37 -0700 Subject: [PATCH 2/2] test(nexus): lock in router-off required-tools re-route and fail-fast exhaustion behavior MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Answers the PR #1632 review asks (claude-review + Codex P2) on the credential filter's edge cases without changing behavior: - Router off + required tools + explicitly selected model on an unconfigured provider: routing selects a configured, tool-capable model (required_tools_enforced) instead of executing a model whose provider creation would throw at stream time. - No configured provider has an accessible model: routing fails fast with the clear 'No accessible Nexus model is available' error BEFORE provider creation. The Codex P2 suggestion to execute the unconfigured explicit model as a final fallback is intentionally declined — both outcomes are errors, but executing it reproduces the exact stream-time missing-key 500 this filter exists to prevent, with a misleading mid-stream failure instead of an honest routing error. --- .../model-router/__tests__/router.test.ts | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/lib/nexus/model-router/__tests__/router.test.ts b/lib/nexus/model-router/__tests__/router.test.ts index dc37592f2..699b27b23 100644 --- a/lib/nexus/model-router/__tests__/router.test.ts +++ b/lib/nexus/model-router/__tests__/router.test.ts @@ -400,6 +400,32 @@ describe("Nexus model router credential filtering", () => { })).rejects.toThrow("openai family") }) + it("re-routes required tools to a configured provider when routing is off and the explicit model's provider is not", async () => { + mockGetConfig.mockResolvedValue({ config, mode: "off" }) + mockGetConfiguredChatProviders.mockResolvedValue(new Set(["amazon-bedrock", "google"])) + + const result = await routeNexusRequest({ + text: "Summarize my attachment", fallbackModelId: "gpt-terra", experienceMode: "standard", + requestedFamily: "auto", enabledConnectorIds: [], + enabledToolNames: ["searchNexusAttachments"], userId: 7, + }) + + expect(result.modelId).toBe("us.anthropic.claude-sonnet") + expect(result.metadata.reasonCodes).toContain("required_tools_enforced") + }) + + it("fails fast before provider creation when no configured provider has an accessible model", async () => { + // The explicitly selected model's provider being unconfigured must NOT be + // executed as a last resort here — that is the stream-time missing-key 500 + // this filter exists to prevent. Exhaustion fails fast with a clear error. + mockGetConfiguredChatProviders.mockResolvedValue(new Set(["azure"])) + + await expect(routeNexusRequest({ + text: "Help", fallbackModelId: "gpt-terra", experienceMode: "standard", + requestedFamily: "auto", enabledConnectorIds: [], userId: 7, + })).rejects.toThrow("No accessible Nexus model is available") + }) + it("does not offer image generation through an unconfigured provider", async () => { mockClassify.mockResolvedValue({ intent: "image", tier: "medium", confidence: 0.99,