From 6f2d9bc0fffa06bf62590e5a0328a1abe740c8e3 Mon Sep 17 00:00:00 2001 From: NotXf1le <89696340+NotXf1le@users.noreply.github.com> Date: Sun, 20 Sep 2026 19:11:48 +0200 Subject: [PATCH 1/2] feat: add OpenRouter support to MCP --- README.md | 2 +- packages/choosekit-mcp/README.md | 42 +++++---- packages/choosekit-mcp/package-lock.json | 12 +-- packages/choosekit-mcp/package.json | 4 +- packages/choosekit-mcp/src/cli.ts | 25 +++++- packages/choosekit-mcp/src/config.ts | 78 +++++++++++++---- packages/choosekit-mcp/src/server.ts | 29 +++++-- packages/choosekit-mcp/tests/cli.test.mjs | 81 +++++++++++++++-- .../tests/fixtures/openrouter-fetch.mjs | 34 ++++++++ packages/choosekit-mcp/tests/package.test.mjs | 15 +++- packages/choosekit-mcp/tests/server.test.mjs | 86 ++++++++++++++++++- 11 files changed, 340 insertions(+), 68 deletions(-) create mode 100644 packages/choosekit-mcp/tests/fixtures/openrouter-fetch.mjs diff --git a/README.md b/README.md index 94d75fc..d3d4dd1 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ The library has no telemetry. ### MCP server -[`choosekit-mcp`](packages/choosekit-mcp/README.md) exposes the same local llama.cpp decision interface as a read-only stdio tool for Claude Code, Codex, and OpenCode. Configure the llama.cpp endpoint, model, and scoring mode with environment variables when starting the MCP server. Every `choose` call uses this configuration. +[`choosekit-mcp`](packages/choosekit-mcp/README.md) exposes choosekit through llama.cpp or OpenRouter as a read-only stdio tool for Claude Code, Codex, and OpenCode. Select the backend and configure it with environment variables when starting the MCP server. Every `choose` call uses this configuration. ## OpenRouter diff --git a/packages/choosekit-mcp/README.md b/packages/choosekit-mcp/README.md index 2a413b3..0a4f301 100644 --- a/packages/choosekit-mcp/README.md +++ b/packages/choosekit-mcp/README.md @@ -1,20 +1,25 @@ # choosekit-mcp -`choosekit-mcp` exposes [choosekit](https://github.com/NotXf1le/choosekit) as a read-only MCP tool backed by a model already running in llama.cpp. +`choosekit-mcp` exposes [choosekit](https://github.com/NotXf1le/choosekit) as a read-only MCP tool backed by llama.cpp or OpenRouter. The server uses stdio. It returns a decision and probability distribution. Execution of the selected action remains with the MCP client. ## Configuration -Set the llama.cpp connection when the MCP process starts: +Select and configure the backend when the MCP process starts. `CHOOSEKIT_BACKEND` defaults to `llama-cpp`. -| Environment variable | Default | Description | -|---|---|---| -| `CHOOSEKIT_BASE_URL` | Required | Base URL of the llama.cpp server, for example `http://127.0.0.1:8080` | -| `CHOOSEKIT_MODEL` | Not set | Model alias sent with llama.cpp requests | -| `CHOOSEKIT_MODE` | `labels` | Candidate scoring mode: `labels` or `minimal-prefix` | +| Backend | Environment variable | Default | Description | +|---|---|---|---| +| Both | `CHOOSEKIT_BACKEND` | `llama-cpp` | `llama-cpp` or `openrouter` | +| Both | `CHOOSEKIT_MODEL` | Not set | Optional for llama.cpp; required for OpenRouter | +| Both | `CHOOSEKIT_MODE` | `labels` | `labels` for either backend; `minimal-prefix` is llama.cpp only | +| llama.cpp | `CHOOSEKIT_BASE_URL` | Required | Base URL of the llama.cpp server, for example `http://127.0.0.1:8080` | +| OpenRouter | `OPENROUTER_API_KEY` | Required | OpenRouter API key | +| OpenRouter | `OPENROUTER_PROVIDER` | Not set | Pins one provider and disables fallback | -Every tool call uses the configuration set when the MCP process starts. The llama.cpp server must provide its native `/tokenize` and `/completion` endpoints. +Every tool call uses the configuration set when the MCP process starts. Backend settings are not accepted as tool arguments. The llama.cpp server must provide its native `/tokenize` and `/completion` endpoints. OpenRouter receives the context, question, and choice descriptions. + +OpenRouter supports `labels` mode with up to 20 choices. llama.cpp supports up to 26 choices in `labels` mode and has no additional MCP choice limit in `minimal-prefix` mode. ### Claude Code @@ -22,12 +27,6 @@ Every tool call uses the configuration set when the MCP process starts. The llam claude mcp add choosekit --env CHOOSEKIT_BASE_URL=http://127.0.0.1:8080 -- npx -y choosekit-mcp ``` -On native Windows, launch `npx` through `cmd`: - -```powershell -claude mcp add choosekit --env CHOOSEKIT_BASE_URL=http://127.0.0.1:8080 -- cmd /c npx -y choosekit-mcp -``` - ### Codex ```sh @@ -40,6 +39,16 @@ codex mcp add choosekit --env CHOOSEKIT_BASE_URL=http://127.0.0.1:8080 -- npx -y opencode mcp add choosekit --env CHOOSEKIT_BASE_URL=http://127.0.0.1:8080 -- npx -y choosekit-mcp ``` +### OpenRouter + +For example, with Codex: + +```sh +codex mcp add choosekit --env CHOOSEKIT_BACKEND=openrouter --env OPENROUTER_API_KEY=... --env CHOOSEKIT_MODEL=... -- npx -y choosekit-mcp +``` + +On native Windows, launch `npx` through `cmd` in any example above: replace `-- npx -y choosekit-mcp` with `-- cmd /c npx -y choosekit-mcp`. + ## Tool The server exposes one tool named `choose`: @@ -58,11 +67,12 @@ The server exposes one tool named `choose`: With `labels`, choosekit maps the supplied choice keys to A/B/C labels for scoring, so each description must contain the option's full meaning. With `minimal-prefix`, it scores the shortest token prefixes that distinguish the original keys. When no supplied option may apply, add an explicit choice such as `insufficient_information`. -The result contains the selected key, the complete normalized distribution, raw scores, margin, entropy, token-boundary rollback, and backend usage when available. Probabilities represent relative preference among the supplied choices. Estimating correctness requires separate calibration. +The result contains the selected key, the complete normalized distribution, raw scores, margin, entropy, token-boundary rollback, and backend usage when available. A score is `null` when the upstream backend did not return a log probability for that choice; its probability in the distribution is `0`. Probabilities represent relative preference among the supplied choices. Estimating correctness requires separate calibration. ## Requirements - Node.js 20 or newer. -- A reachable llama.cpp server with a compatible model already loaded. +- For llama.cpp, a reachable server with a compatible model already loaded. +- For OpenRouter, an API key and a model that returns the required log probabilities. [Apache-2.0](LICENSE). Copyright 2026 NotXf1le. diff --git a/packages/choosekit-mcp/package-lock.json b/packages/choosekit-mcp/package-lock.json index c5a15a2..1bd08ec 100644 --- a/packages/choosekit-mcp/package-lock.json +++ b/packages/choosekit-mcp/package-lock.json @@ -1,16 +1,16 @@ { "name": "choosekit-mcp", - "version": "0.1.0", + "version": "0.2.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "choosekit-mcp", - "version": "0.1.0", + "version": "0.2.0", "license": "Apache-2.0", "dependencies": { "@modelcontextprotocol/server": "2.0.0", - "choosekit": "^0.4.2", + "choosekit": "^0.5.0", "zod": "^4.6.4" }, "bin": { @@ -60,9 +60,9 @@ } }, "node_modules/choosekit": { - "version": "0.4.2", - "resolved": "https://registry.npmjs.org/choosekit/-/choosekit-0.4.2.tgz", - "integrity": "sha512-E0yzRsxLtgDt7dczBA7S1yqXA9aaKXfHnSc+bBXkiEFPNfpMxe5eyl0RPQitdaAcJBFbhDW6shulJWSvTPZijA==", + "version": "0.5.0", + "resolved": "https://registry.npmjs.org/choosekit/-/choosekit-0.5.0.tgz", + "integrity": "sha512-eTnaA5zOr1W5sN62K3ou+/GFZoRl6mhpr7m63AvgR5aQa1sljEiglfigHGVHC/BR3XLVocdq9LyYn8iqQ4XoyA==", "license": "Apache-2.0", "engines": { "node": ">=20" diff --git a/packages/choosekit-mcp/package.json b/packages/choosekit-mcp/package.json index 0114ae0..27da1dc 100644 --- a/packages/choosekit-mcp/package.json +++ b/packages/choosekit-mcp/package.json @@ -1,6 +1,6 @@ { "name": "choosekit-mcp", - "version": "0.1.0", + "version": "0.2.0", "description": "Use choosekit as a local MCP server.", "license": "Apache-2.0", "author": { @@ -37,7 +37,7 @@ }, "dependencies": { "@modelcontextprotocol/server": "2.0.0", - "choosekit": "^0.4.2", + "choosekit": "^0.5.0", "zod": "^4.6.4" }, "devDependencies": { diff --git a/packages/choosekit-mcp/src/cli.ts b/packages/choosekit-mcp/src/cli.ts index a73a105..e0224bb 100644 --- a/packages/choosekit-mcp/src/cli.ts +++ b/packages/choosekit-mcp/src/cli.ts @@ -1,14 +1,35 @@ #!/usr/bin/env node import { serveStdio } from "@modelcontextprotocol/server/stdio"; +import type { Chooser } from "choosekit"; import { fromLlamaCpp } from "choosekit/llama-cpp"; +import { fromOpenRouter } from "choosekit/openrouter"; import { loadConfig } from "./config.js"; import { buildServer } from "./server.js"; try { const config = loadConfig(); - const chooser = fromLlamaCpp(config); - serveStdio(() => buildServer(chooser, { mode: config.mode })); + let chooser: Chooser; + switch (config.backend) { + case "llama-cpp": + chooser = fromLlamaCpp({ + baseURL: config.baseURL, + ...(config.model === undefined ? {} : { model: config.model }), + mode: config.mode, + }); + break; + case "openrouter": + chooser = fromOpenRouter({ + apiKey: config.apiKey, + model: config.model, + ...(config.provider === undefined ? {} : { provider: config.provider }), + }); + break; + } + serveStdio(() => buildServer(chooser, { + backend: config.backend, + mode: config.mode, + })); } catch (error) { const message = error instanceof Error ? error.message : "Unknown configuration error."; console.error(`choosekit-mcp: ${message}`); diff --git a/packages/choosekit-mcp/src/config.ts b/packages/choosekit-mcp/src/config.ts index 879363d..8aa1649 100644 --- a/packages/choosekit-mcp/src/config.ts +++ b/packages/choosekit-mcp/src/config.ts @@ -1,29 +1,69 @@ import { z } from "zod"; -const configSchema = z.object({ - CHOOSEKIT_BASE_URL: z.string().trim().url(), - CHOOSEKIT_MODEL: z.string().trim().min(1).optional(), - CHOOSEKIT_MODE: z.enum(["labels", "minimal-prefix"]).default("labels"), -}); +const urlSchema = z.string().url(); -export interface Config { - baseURL: string; - model?: string; - mode: "labels" | "minimal-prefix"; +export type Config = + | { + readonly backend: "llama-cpp"; + readonly baseURL: string; + readonly model?: string; + readonly mode: "labels" | "minimal-prefix"; + } + | { + readonly backend: "openrouter"; + readonly apiKey: string; + readonly model: string; + readonly provider?: string; + readonly mode: "labels"; + }; + +function requiredText(value: string | undefined, name: string): string { + const text = value?.trim(); + if (!text) throw new TypeError(`${name} is required.`); + return text; +} + +function optionalText(value: string | undefined, name: string): string | undefined { + if (value === undefined) return undefined; + const text = value.trim(); + if (!text) throw new TypeError(`${name} must not be empty.`); + return text; } export function loadConfig(env: NodeJS.ProcessEnv = process.env): Config { - const parsed = configSchema.parse({ - CHOOSEKIT_BASE_URL: env.CHOOSEKIT_BASE_URL, - CHOOSEKIT_MODEL: env.CHOOSEKIT_MODEL, - CHOOSEKIT_MODE: env.CHOOSEKIT_MODE, - }); + const backend = env.CHOOSEKIT_BACKEND?.trim() ?? "llama-cpp"; + if (backend !== "llama-cpp" && backend !== "openrouter") { + throw new TypeError("CHOOSEKIT_BACKEND must be llama-cpp or openrouter."); + } + + if (backend === "openrouter") { + const mode = env.CHOOSEKIT_MODE?.trim() ?? "labels"; + if (mode !== "labels") { + throw new TypeError("CHOOSEKIT_MODE must be labels when using OpenRouter."); + } + const provider = optionalText(env.OPENROUTER_PROVIDER, "OPENROUTER_PROVIDER"); + return { + backend, + apiKey: requiredText(env.OPENROUTER_API_KEY, "OPENROUTER_API_KEY"), + model: requiredText(env.CHOOSEKIT_MODEL, "CHOOSEKIT_MODEL"), + ...(provider === undefined ? {} : { provider }), + mode, + }; + } + const baseURL = requiredText(env.CHOOSEKIT_BASE_URL, "CHOOSEKIT_BASE_URL"); + if (!urlSchema.safeParse(baseURL).success) { + throw new TypeError("CHOOSEKIT_BASE_URL must be a valid URL."); + } + const model = optionalText(env.CHOOSEKIT_MODEL, "CHOOSEKIT_MODEL"); + const mode = env.CHOOSEKIT_MODE?.trim() ?? "labels"; + if (mode !== "labels" && mode !== "minimal-prefix") { + throw new TypeError("CHOOSEKIT_MODE must be labels or minimal-prefix."); + } return { - baseURL: parsed.CHOOSEKIT_BASE_URL, - ...(parsed.CHOOSEKIT_MODEL === undefined - ? {} - : { model: parsed.CHOOSEKIT_MODEL }), - mode: parsed.CHOOSEKIT_MODE, + backend, + baseURL, + ...(model === undefined ? {} : { model }), + mode, }; } diff --git a/packages/choosekit-mcp/src/server.ts b/packages/choosekit-mcp/src/server.ts index f8d7328..101b32d 100644 --- a/packages/choosekit-mcp/src/server.ts +++ b/packages/choosekit-mcp/src/server.ts @@ -6,6 +6,7 @@ import { z } from "zod"; export type ChoiceMode = "labels" | "minimal-prefix"; export interface ServerOptions { + readonly backend?: "llama-cpp" | "openrouter"; readonly mode?: ChoiceMode; } @@ -21,14 +22,14 @@ const usageSchema = z.object({ const decisionSchema = z.object({ choice: z.string(), distribution: z.record(z.string(), z.number().finite().min(0).max(1)), - scores: z.record(z.string(), z.number().finite().max(0)), + scores: z.record(z.string(), z.union([z.number().finite().max(0), z.null()])), margin: z.number().finite().min(0).max(1), entropy: z.number().finite().nonnegative(), boundaryTokens: z.number().int().nonnegative(), usage: usageSchema.optional(), }).strict(); -function inputSchema(mode: ChoiceMode) { +function inputSchema(backend: "llama-cpp" | "openrouter", mode: ChoiceMode) { const choicesSchema = z.fromJSONSchema({ type: "object", propertyNames: { @@ -40,7 +41,9 @@ function inputSchema(mode: ChoiceMode) { pattern: "\\S", }, minProperties: 2, - ...(mode === "labels" ? { maxProperties: 26 } : {}), + ...(backend === "openrouter" + ? { maxProperties: 20 } + : mode === "labels" ? { maxProperties: 26 } : {}), }) as z.ZodType>; return z.object({ @@ -65,7 +68,7 @@ function errorResult(error: unknown, signal: AbortSignal) { if (error instanceof ScoringError) { return { isError: true as const, - content: [{ type: "text" as const, text: "llama.cpp could not score the supplied choices." }], + content: [{ type: "text" as const, text: "The model could not score the supplied choices." }], }; } return { @@ -74,8 +77,18 @@ function errorResult(error: unknown, signal: AbortSignal) { }; } +function wireScores(scores: Readonly>): +Readonly> { + return Object.fromEntries(Object.entries(scores).map(([key, score]) => + [key, score === -Infinity ? null : score])); +} + export function buildServer(chooser: Chooser, options: ServerOptions = {}): McpServer { if (typeof chooser !== "function") throw new TypeError("chooser must be a function."); + const backend = options.backend ?? "llama-cpp"; + if (backend !== "llama-cpp" && backend !== "openrouter") { + throw new TypeError("backend must be llama-cpp or openrouter."); + } const mode = options.mode ?? "labels"; if (mode !== "labels" && mode !== "minimal-prefix") { throw new TypeError("mode must be labels or minimal-prefix."); @@ -86,11 +99,11 @@ export function buildServer(chooser: Chooser, options: ServerOptions = {}): McpS "choose", { description: toolDescription(mode), - inputSchema: inputSchema(mode), + inputSchema: inputSchema(backend, mode), outputSchema: decisionSchema, annotations: { readOnlyHint: true, - openWorldHint: false, + openWorldHint: backend === "openrouter", }, }, async ({ context, question, choices }, ctx) => { @@ -104,7 +117,7 @@ export function buildServer(chooser: Chooser, options: ServerOptions = {}): McpS const structuredContent: { choice: string; distribution: Readonly>; - scores: Readonly>; + scores: Readonly>; margin: number; entropy: number; boundaryTokens: number; @@ -112,7 +125,7 @@ export function buildServer(chooser: Chooser, options: ServerOptions = {}): McpS } = { choice: decision.choice, distribution: decision.distribution, - scores: decision.scores, + scores: wireScores(decision.scores), margin: decision.margin, entropy: decision.entropy, boundaryTokens: decision.boundaryTokens, diff --git a/packages/choosekit-mcp/tests/cli.test.mjs b/packages/choosekit-mcp/tests/cli.test.mjs index 3e4c2fe..9af1e14 100644 --- a/packages/choosekit-mcp/tests/cli.test.mjs +++ b/packages/choosekit-mcp/tests/cli.test.mjs @@ -6,18 +6,38 @@ import { fileURLToPath } from "node:url"; const packageRoot = new URL("../", import.meta.url); const cli = fileURLToPath(new URL("../dist/cli.js", import.meta.url)); +const openRouterFetchFixture = new URL("./fixtures/openrouter-fetch.mjs", import.meta.url).href; +const request = Object.freeze({ + context: "A payout failed.", + question: "Which team should handle this?", + choices: Object.freeze({ + billing: "Payments, payouts, invoices, and refunds", + technical: "Bugs, outages, integrations, and API errors", + }), +}); + +const configurationKeys = [ + "CHOOSEKIT_BACKEND", + "CHOOSEKIT_BASE_URL", + "CHOOSEKIT_MODEL", + "CHOOSEKIT_MODE", + "OPENROUTER_API_KEY", + "OPENROUTER_PROVIDER", +]; function cliEnvironment(overrides = {}) { + const env = { ...process.env }; + for (const key of configurationKeys) delete env[key]; return { - ...process.env, + ...env, CHOOSEKIT_BASE_URL: "http://127.0.0.1:1", CHOOSEKIT_MODE: "labels", ...overrides, }; } -function startCli(env = cliEnvironment()) { - const child = spawn(process.execPath, [cli], { +function startCli(env = cliEnvironment(), nodeArguments = []) { + const child = spawn(process.execPath, [...nodeArguments, cli], { cwd: packageRoot, env, stdio: ["pipe", "pipe", "pipe"], @@ -104,11 +124,28 @@ async function initialize(process_) { test("reports invalid environment configuration without writing to stdout", () => { const cases = [ - ["missing base URL", { CHOOSEKIT_BASE_URL: "" }], - ["invalid base URL", { CHOOSEKIT_BASE_URL: "not a URL" }], - ["invalid mode", { CHOOSEKIT_MODE: "keys" }], + ["missing base URL", { CHOOSEKIT_BASE_URL: "" }, + "CHOOSEKIT_BASE_URL is required."], + ["invalid base URL", { CHOOSEKIT_BASE_URL: "not a URL" }, + "CHOOSEKIT_BASE_URL must be a valid URL."], + ["invalid mode", { CHOOSEKIT_MODE: "keys" }, + "CHOOSEKIT_MODE must be labels or minimal-prefix."], + ["unknown backend", { CHOOSEKIT_BACKEND: "other" }, + "CHOOSEKIT_BACKEND must be llama-cpp or openrouter."], + ["OpenRouter without API key", { + CHOOSEKIT_BACKEND: "openrouter", CHOOSEKIT_MODEL: "test/model", + }, "OPENROUTER_API_KEY is required."], + ["OpenRouter without model", { + CHOOSEKIT_BACKEND: "openrouter", OPENROUTER_API_KEY: "test-secret", + }, "CHOOSEKIT_MODEL is required."], + ["OpenRouter with minimal-prefix", { + CHOOSEKIT_BACKEND: "openrouter", + CHOOSEKIT_MODEL: "test/model", + CHOOSEKIT_MODE: "minimal-prefix", + OPENROUTER_API_KEY: "test-secret", + }, "CHOOSEKIT_MODE must be labels when using OpenRouter."], ]; - for (const [name, override] of cases) { + for (const [name, override, expectedMessage] of cases) { const result = spawnSync(process.execPath, [cli], { cwd: packageRoot, env: cliEnvironment(override), @@ -116,11 +153,11 @@ test("reports invalid environment configuration without writing to stdout", () = }); assert.equal(result.status, 1, name); assert.equal(result.stdout, "", name); - assert.match(result.stderr, /^choosekit-mcp:/, name); + assert.equal(result.stderr, `choosekit-mcp: ${expectedMessage}\n`, name); } }); -test("lists tools without contacting llama.cpp and writes only MCP messages to stdout", async (t) => { +test("defaults to llama.cpp and lists tools without contacting it", async (t) => { const process_ = startCli(); t.after(() => process_.close()); await initialize(process_); @@ -132,6 +169,32 @@ test("lists tools without contacting llama.cpp and writes only MCP messages to s for (const line of process_.lines) assert.doesNotThrow(() => JSON.parse(line)); }); +test("serves an OpenRouter choice with key, model, and provider kept process-local", async (t) => { + const secret = "openrouter-secret-that-must-not-leak"; + const process_ = startCli(cliEnvironment({ + CHOOSEKIT_BACKEND: "openrouter", + CHOOSEKIT_BASE_URL: "", + CHOOSEKIT_MODEL: "fixture/model", + OPENROUTER_API_KEY: secret, + OPENROUTER_PROVIDER: "fixture-provider", + }), ["--import", openRouterFetchFixture]); + let stderr = ""; + process_.child.stderr.setEncoding("utf8"); + process_.child.stderr.on("data", (chunk) => { stderr += chunk; }); + t.after(() => process_.close()); + await initialize(process_); + + const response = await process_.rpc("tools/call", { + name: "choose", + arguments: request, + }); + + assert.equal(response.result.structuredContent.choice, "technical"); + assert.deepEqual(JSON.parse(response.result.content[0].text), response.result.structuredContent); + assert.doesNotMatch(process_.lines.join("\n"), new RegExp(secret)); + assert.doesNotMatch(stderr, new RegExp(secret)); +}); + test("serves a minimal-prefix choice using the configured llama.cpp endpoint", async (t) => { const requests = []; const llama = createServer(async (request, response) => { diff --git a/packages/choosekit-mcp/tests/fixtures/openrouter-fetch.mjs b/packages/choosekit-mcp/tests/fixtures/openrouter-fetch.mjs new file mode 100644 index 0000000..a24b0dd --- /dev/null +++ b/packages/choosekit-mcp/tests/fixtures/openrouter-fetch.mjs @@ -0,0 +1,34 @@ +import assert from "node:assert/strict"; + +globalThis.fetch = async (url, init) => { + assert.equal(url, "https://openrouter.ai/api/v1/chat/completions"); + assert.equal(new Headers(init.headers).get("authorization"), + "Bearer openrouter-secret-that-must-not-leak"); + const body = JSON.parse(init.body); + assert.equal(body.model, "fixture/model"); + assert.deepEqual(body.provider, { + only: ["fixture-provider"], + allow_fallbacks: false, + }); + + return new Response(JSON.stringify({ + choices: [{ + finish_reason: "length", + message: { role: "assistant", content: "B" }, + logprobs: { + content: [{ + token: "B", + bytes: [66], + logprob: -0.1, + top_logprobs: [ + { token: "A", bytes: [65], logprob: -1.1 }, + { token: "B", bytes: [66], logprob: -0.1 }, + ], + }], + }, + }], + }), { + status: 200, + headers: { "content-type": "application/json" }, + }); +}; diff --git a/packages/choosekit-mcp/tests/package.test.mjs b/packages/choosekit-mcp/tests/package.test.mjs index eed6de4..fc67c5a 100644 --- a/packages/choosekit-mcp/tests/package.test.mjs +++ b/packages/choosekit-mcp/tests/package.test.mjs @@ -11,6 +11,19 @@ const npmCli = process.platform === "win32" ? join(dirname(process.execPath), "node_modules", "npm", "bin", "npm-cli.js") : undefined; +function llamaEnvironment() { + const env = { ...process.env }; + for (const key of [ + "CHOOSEKIT_BACKEND", + "CHOOSEKIT_BASE_URL", + "CHOOSEKIT_MODEL", + "CHOOSEKIT_MODE", + "OPENROUTER_API_KEY", + "OPENROUTER_PROVIDER", + ]) delete env[key]; + return { ...env, CHOOSEKIT_BACKEND: "llama-cpp", CHOOSEKIT_BASE_URL: "" }; +} + function spawnNpm(args, options = {}) { return npmCli ? spawnSync(process.execPath, [npmCli, ...args], { encoding: "utf8", ...options }) @@ -40,7 +53,7 @@ test("the packed package installs and exposes its executable", (t) => { const launched = spawnNpm(["exec", "--no", "--", "choosekit-mcp"], { cwd: temporary, - env: { ...process.env, CHOOSEKIT_BASE_URL: "" }, + env: llamaEnvironment(), timeout: 10_000, }); assert.equal(launched.error, undefined, launched.error?.message); diff --git a/packages/choosekit-mcp/tests/server.test.mjs b/packages/choosekit-mcp/tests/server.test.mjs index 619bf9f..d840dba 100644 --- a/packages/choosekit-mcp/tests/server.test.mjs +++ b/packages/choosekit-mcp/tests/server.test.mjs @@ -85,7 +85,7 @@ async function callChoose(connection, arguments_) { return connection.rpc("tools/call", { name: "choose", arguments: arguments_ }); } -test("exposes a strict, read-only choose tool", async (t) => { +test("exposes a strict, read-only llama.cpp choose tool", async (t) => { const connection = await connect(async () => assert.fail("chooser should not run")); t.after(() => connection.close()); @@ -96,9 +96,60 @@ test("exposes a strict, read-only choose tool", async (t) => { assert.equal(tool.inputSchema.additionalProperties, false); assert.equal(tool.outputSchema.additionalProperties, false); - const invalid = await callChoose(connection, { ...request, endpoint: "https://example.com" }); + const invalid = await callChoose(connection, { ...request, backend: "not allowed" }); assert.equal(invalid.result.isError, true); - assert.match(invalid.result.content[0].text, /endpoint|unrecognized|additional/i); + assert.match(invalid.result.content[0].text, /backend|unrecognized|additional/i); +}); + +test("marks the OpenRouter tool as open-world", async (t) => { + const connection = await connect(async () => assert.fail("chooser should not run"), { + backend: "openrouter", + }); + t.after(() => connection.close()); + + const listed = await connection.rpc("tools/list"); + assert.deepEqual(listed.result.tools[0].annotations, { + readOnlyHint: true, + openWorldHint: true, + }); +}); + +test("OpenRouter accepts 20 choices and preserves the selected choice key", async (t) => { + const choices = Object.fromEntries(Array.from({ length: 20 }, (_, index) => + [`choice_${index}`, `Choice ${index}`])); + let receivedChoices; + const chooser = async ({ choices: supplied }) => { + receivedChoices = supplied; + return { + choice: "choice_19", + distribution: Object.fromEntries(Object.keys(supplied).map((key) => + [key, key === "choice_19" ? 1 : 0])), + scores: Object.fromEntries(Object.keys(supplied).map((key) => [key, -1])), + margin: 1, + entropy: 0, + boundaryTokens: 1, + }; + }; + const connection = await connect(chooser, { backend: "openrouter" }); + t.after(() => connection.close()); + + const response = await callChoose(connection, { ...request, choices }); + + assert.equal(response.result.structuredContent.choice, "choice_19"); + assert.deepEqual(receivedChoices, choices); +}); + +test("OpenRouter rejects 21 choices before calling the chooser", async (t) => { + let called = false; + const connection = await connect(async () => { called = true; }, { backend: "openrouter" }); + t.after(() => connection.close()); + const choices = Object.fromEntries(Array.from({ length: 21 }, (_, index) => + [`choice_${index}`, `Choice ${index}`])); + + const response = await callChoose(connection, { ...request, choices }); + + assert.equal(response.result.isError, true); + assert.equal(called, false); }); test("returns the same decision as the direct chooser in text and structured content", async (t) => { @@ -112,10 +163,37 @@ test("returns the same decision as the direct chooser in text and structured con assert.deepEqual(JSON.parse(response.result.content[0].text), expected); }); +test("serializes unavailable scores as null", async (t) => { + const chooser = async () => ({ + choice: "billing", + distribution: { billing: 1, technical: 0 }, + scores: { billing: -0.2, technical: -Infinity }, + margin: 1, + entropy: 0, + boundaryTokens: 1, + }); + const connection = await connect(chooser, { backend: "openrouter" }); + t.after(() => connection.close()); + + const response = await callChoose(connection, request); + const expected = { + choice: "billing", + distribution: { billing: 1, technical: 0 }, + scores: { billing: -0.2, technical: null }, + margin: 1, + entropy: 0, + boundaryTokens: 1, + }; + + assert.notEqual(response.result.isError, true); + assert.deepEqual(response.result.structuredContent, expected); + assert.deepEqual(JSON.parse(response.result.content[0].text), expected); +}); + test("does not expose scorer or implementation errors", async (t) => { const cases = [ [async () => { throw new ScoringError("secret upstream response"); }, - "llama.cpp could not score the supplied choices."], + "The model could not score the supplied choices."], [async () => { throw new Error("secret implementation detail"); }, "The choice request failed unexpectedly."], ]; From 15a42c34becbd4c95b31683560086cd8f2b5d6b1 Mon Sep 17 00:00:00 2001 From: NotXf1le <89696340+NotXf1le@users.noreply.github.com> Date: Sun, 20 Sep 2026 19:19:53 +0200 Subject: [PATCH 2/2] docs: move MCP section before backends --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index d3d4dd1..3cbe7f8 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,10 @@ The project was inspired by [Jev and the System One model interface](https://typ `choosekit` is an independent project with no affiliation to TypeSafe or Jev. +## MCP server + +[`choosekit-mcp`](packages/choosekit-mcp/README.md) exposes choosekit through llama.cpp or OpenRouter as a read-only stdio tool for Claude Code, Codex, and OpenCode. Select the backend and configure it with environment variables when starting the MCP server. Every `choose` call uses this configuration. + ## llama.cpp ```ts @@ -50,10 +54,6 @@ The llama.cpp backend requires its native `/tokenize` and `/completion` endpoint The library has no telemetry. -### MCP server - -[`choosekit-mcp`](packages/choosekit-mcp/README.md) exposes choosekit through llama.cpp or OpenRouter as a read-only stdio tool for Claude Code, Codex, and OpenCode. Select the backend and configure it with environment variables when starting the MCP server. Every `choose` call uses this configuration. - ## OpenRouter ```ts