Add OpenRouter backend

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.5.0 - 2026-09-20
+
+- Added label scoring through OpenRouter with `choosekit/openrouter`.
+
 ## 0.4.2
 
 - Made the request cancellation signal available to custom prompt formatters.

README.md

@@ -1,6 +1,6 @@
 # choosekit
 
-`choosekit` scores a finite set of choices with a language model you already run and returns a typed decision with a probability distribution.
+`choosekit` scores a finite set of choices with a language model and returns a typed decision with a probability distribution. It supports local llama.cpp models and an optional OpenRouter backend.
 
 ```sh
 npm install choosekit
@@ -18,7 +18,7 @@ Agents often need to choose from known options:
 
 `choosekit` scores choices using the model's conditional log probabilities at the token branches that distinguish them.
 
-The project was inspired by [Jev and the System One model interface](https://typesafe.ai/blog/introducing-system-one-models-and-jev): application state in, typed probabilistic decisions out. Jev is a specialized hosted model. `choosekit` explores the same useful interface with a model you control. Application state stays on infrastructure you choose, and the decision path can use a model already running inside an existing deployment.
+The project was inspired by [Jev and the System One model interface](https://typesafe.ai/blog/introducing-system-one-models-and-jev): application state in, typed probabilistic decisions out. Jev is a specialized hosted model. `choosekit` explores the same useful interface with a model you control. The llama.cpp backend keeps application state on infrastructure you choose; OpenRouter is available when a hosted model is more convenient.
 
 `choosekit` is an independent project with no affiliation to TypeSafe or Jev.
 
@@ -46,20 +46,43 @@ console.log(decision.choice);       // "no"
 console.log(decision.distribution); // { yes: ..., no: ... }
 ```
 
-Compatibility requires the native llama.cpp `/tokenize` and `/completion` endpoints with raw pre-sampling log probabilities and returned token IDs. An OpenAI-compatible `/v1` endpoint alone lacks these capabilities.
+The llama.cpp backend requires its native `/tokenize` and `/completion` endpoints. `minimal-prefix` is available only with this backend.
 
 The library has no telemetry.
 
 ### MCP server
 
 [`choosekit-mcp`](packages/choosekit-mcp/README.md) exposes the same local llama.cpp decision interface as a read-only stdio tool for Claude Code, Codex, and OpenCode. Configure the llama.cpp endpoint, model, and scoring mode with environment variables when starting the MCP server. Every `choose` call uses this configuration.
 
+## OpenRouter
+
+```ts
+import { fromOpenRouter } from "choosekit/openrouter";
+
+const choose = fromOpenRouter({
+  apiKey: process.env.OPENROUTER_API_KEY!,
+  model: "qwen/qwen3.8-27b",
+});
+```
+
+The OpenRouter backend supports models and providers that return first-token `top_logprobs`, with up to 20 choices. Unlike llama.cpp, this backend sends the prompt to OpenRouter. It requests reasoning to be disabled. Choices omitted from `top_logprobs` receive zero probability. Returned probabilities are normalized across the supplied choices and are not calibrated correctness estimates.
+
+OpenRouter may route the same model through different providers. Set `provider` to an OpenRouter provider slug to use only that provider and disable fallback:
+
+```ts
+const choose = fromOpenRouter({
+  apiKey: process.env.OPENROUTER_API_KEY!,
+  model: "qwen/qwen3.8-27b",
+  provider: process.env.OPENROUTER_PROVIDER!,
+});
+```
+
 ## Scoring modes
 
 | Mode | Candidate representation | Use when |
 |---|---|---|
-| `labels` | `A`, `B`, `C`, ... | Default. The choice set has at most 26 entries. |
-| `minimal-prefix` | Original JSON-quoted keys | The key names should influence the decision, or the set has more than 26 entries. |
+| `labels` | `A`, `B`, `C`, ... | Default. Up to 26 choices with llama.cpp or 20 with OpenRouter. |
+| `minimal-prefix` | Original JSON-quoted keys | llama.cpp only. Use when key names should influence the decision. |
 
 In `labels` mode, choices are shown to the model as `A`, `B`, `C` instead of their original keys. For example, `refund: "Issue the refund"` is shown as `"A": "Issue the refund"`. Each description must therefore make the option clear. `choosekit` maps the selected label back to the original key.
 
@@ -125,18 +148,7 @@ Most distributions are similar. Some differ substantially: the systems select di
 
 `context` is copied unchanged to the start of the scoring prompt. The default formatter then appends the question, choice descriptions, and an answer marker.
 
-For chat models, `context` should be the model's normal serialized chat prefix. Use `formatPrompt` when the decision turn needs a particular template. This example uses Qwen's chat markers:
-
-```ts
-const choose = fromLlamaCpp({
-  baseURL: "http://127.0.0.1:8080/",
-  mode: "minimal-prefix",
-  formatPrompt: ({ context, instruction }) =>
-    `${context}<|im_start|>user\n${instruction}<|im_end|>\n<|im_start|>assistant\n`,
-});
-```
-
-The formatted prompt must start with `context` unchanged so an existing server-side prefix cache can still be reused.
+Use `formatPrompt` only when you need custom prompt formatting. The result must preserve `context` as an unchanged prefix so an existing server-side prefix cache can still be reused.
 
 ## Custom scorer
 

TODO.md

@@ -4,4 +4,3 @@
 
 - Tokenize the prompt and prompt-plus-candidate inputs with bounded parallelism in the `llama.cpp` adapter instead of awaiting every `/tokenize` request sequentially. Preserve input order and exact boundary-aware tokenization; make concurrency configurable and measure transport overhead separately from server processing. Consider a future native batch-tokenization endpoint returning one token array per input, because stock `/tokenize` does not provide that response shape.
 - Add native `llama.cpp` support for requesting raw log probabilities for multiple arbitrary token IDs in one request (`logprob_token_ids`). Prefer zero-token generation (`n_predict: 0`) and calculate every requested token's log probability from the full-vocabulary softmax. Keep the current top-N plus exact forced-token fallback path for servers without the extension; an explicitly selected bulk mode must fail clearly when unsupported instead of silently changing protocols.
-- Add an OpenRouter scorer behind the existing `Scorer` and `Chooser` contracts. Support label scoring only: request `logprobs` and up to 20 `top_logprobs` from a compatible provider, require every choice label to be present, and fail clearly instead of treating a missing label as zero probability. Do not require a local tokenizer or expose `minimal-prefix` for this backend.

benchmarks/run-semif.mjs

@@ -172,7 +172,7 @@ for (let index = 0; index < rows.length; index++) {
       totalRows: allRows.length,
       selectedRows: rows.length,
     },
-    runtime: { baseURL, model, mode, adapter: "choosekit/llama-cpp", packageVersion: "0.4.2" },
+    runtime: { baseURL, model, mode, adapter: "choosekit/llama-cpp", packageVersion: "0.5.0" },
     interpretation: mode === "labels"
       ? "Package A/B/C label prompt and distinguishing-token likelihoods."
       : "Package original-key prompt and minimal distinguishing-prefix likelihoods.",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "choosekit",
-  "version": "0.4.2",
+  "version": "0.5.0",
   "description": "Typed choices from your existing language model scorer.",
   "license": "Apache-2.0",
   "author": {
@@ -40,6 +40,16 @@
         "default": "./dist/cjs/llama-cpp.js"
       }
     },
+    "./openrouter": {
+      "import": {
+        "types": "./dist/esm/openrouter.d.ts",
+        "default": "./dist/esm/openrouter.js"
+      },
+      "require": {
+        "types": "./dist/cjs/openrouter.d.ts",
+        "default": "./dist/cjs/openrouter.js"
+      }
+    },
     "./package.json": "./package.json"
   },
   "files": [
@@ -69,6 +79,7 @@
     "choices",
     "logprobs",
     "llama.cpp",
+    "openrouter",
     "typescript"
   ]
 }

src/openrouter.ts

@@ -0,0 +1,178 @@
+import { createFormattedChooser } from "./internal-chooser.js";
+import type { Chooser, ChooserOptions, Scorer, Usage } from "./types.js";
+import { isCount, isRecord, requireText, ScoringError } from "./validation.js";
+
+const ENDPOINT = "https://openrouter.ai/api/v1/chat/completions";
+const MAX_CANDIDATES = 20;
+const CLAMPED_LOGPROB = -9999;
+
+export interface OpenRouterOptions extends ChooserOptions {
+  readonly apiKey: string;
+  readonly model: string;
+  readonly fetch?: typeof globalThis.fetch;
+  /** Pin one OpenRouter provider and disable provider fallback. */
+  readonly provider?: string;
+}
+
+async function post(fetchImpl: typeof globalThis.fetch, apiKey: string, body: unknown,
+  signal?: AbortSignal): Promise<unknown> {
+  signal?.throwIfAborted();
+  let response: Response;
+  try {
+    response = await fetchImpl(ENDPOINT, {
+      method: "POST",
+      headers: {
+        authorization: `Bearer ${apiKey}`,
+        "content-type": "application/json",
+      },
+      body: JSON.stringify(body),
+      ...(signal ? { signal } : {}),
+    });
+  } catch {
+    signal?.throwIfAborted();
+    throw new ScoringError("OpenRouter request failed.");
+  }
+  signal?.throwIfAborted();
+  if (!response.ok) {
+    throw new ScoringError(`OpenRouter returned HTTP ${response.status} for /chat/completions.`);
+  }
+  try {
+    const value: unknown = await response.json();
+    signal?.throwIfAborted();
+    return value;
+  } catch {
+    signal?.throwIfAborted();
+    throw new ScoringError("OpenRouter returned invalid JSON for /chat/completions.");
+  }
+}
+
+function parseUsage(value: unknown): Usage | undefined {
+  if (value === undefined || value === null) return undefined;
+  if (!isRecord(value)
+    || !isCount(value.prompt_tokens)
+    || !isCount(value.completion_tokens)) {
+    throw new ScoringError("OpenRouter returned invalid token usage.");
+  }
+  let cachedTokens: number | null = null;
+  if (value.prompt_tokens_details !== undefined && value.prompt_tokens_details !== null) {
+    if (!isRecord(value.prompt_tokens_details)) {
+      throw new ScoringError("OpenRouter returned invalid cached-token usage.");
+    }
+    const cached = value.prompt_tokens_details.cached_tokens;
+    if (cached !== undefined && cached !== null) {
+      if (!isCount(cached) || cached > value.prompt_tokens) {
+        throw new ScoringError("OpenRouter returned invalid cached-token usage.");
+      }
+      cachedTokens = cached;
+    }
+  }
+  return Object.freeze({
+    promptTokens: value.prompt_tokens,
+    cachedTokens,
+    completionTokens: value.completion_tokens,
+    requests: 1,
+  });
+}
+
+function hasRefusal(choice: Record<string, unknown>): boolean {
+  if (choice.finish_reason === "content_filter") return true;
+  const message = choice.message;
+  return isRecord(message) && message.refusal !== undefined && message.refusal !== null;
+}
+
+function parseScores(value: unknown, candidates: readonly string[]): {
+  readonly logprobs: readonly number[];
+  readonly usage?: Usage;
+} {
+  if (!isRecord(value) || !Array.isArray(value.choices) || value.choices.length === 0) {
+    throw new ScoringError("OpenRouter returned an invalid completion.");
+  }
+  const choice: unknown = value.choices[0];
+  if (!isRecord(choice)) {
+    throw new ScoringError("OpenRouter returned an invalid completion.");
+  }
+  if (hasRefusal(choice)) {
+    throw new ScoringError("OpenRouter refused or filtered the scoring request.");
+  }
+  if (!isRecord(choice.logprobs)) {
+    throw new ScoringError("OpenRouter did not return logprobs.");
+  }
+  if (Array.isArray(choice.logprobs.refusal) && choice.logprobs.refusal.length > 0) {
+    throw new ScoringError("OpenRouter refused or filtered the scoring request.");
+  }
+  const content = choice.logprobs.content;
+  if (!Array.isArray(content) || content.length !== 1 || !isRecord(content[0])) {
+    throw new ScoringError("OpenRouter did not return exactly one scored token position.");
+  }
+  const top = content[0].top_logprobs;
+  if (!Array.isArray(top) || top.length === 0) {
+    throw new ScoringError("OpenRouter did not return top logprobs.");
+  }
+
+  const expected = new Set(candidates);
+  const found = new Map<string, number>();
+  for (const entry of top) {
+    if (!isRecord(entry) || typeof entry.token !== "string") {
+      throw new ScoringError("OpenRouter returned an invalid top-logprob entry.");
+    }
+    if (!expected.has(entry.token)) continue;
+    if (found.has(entry.token)) {
+      throw new ScoringError(`OpenRouter returned duplicate logprobs for label ${entry.token}.`);
+    }
+    if (typeof entry.logprob !== "number" || !Number.isFinite(entry.logprob)
+      || entry.logprob > 0 || entry.logprob <= CLAMPED_LOGPROB) {
+      throw new ScoringError(`OpenRouter returned an invalid or clamped logprob for label ${entry.token}.`);
+    }
+    if (entry.bytes !== undefined && entry.bytes !== null) {
+      if (!Array.isArray(entry.bytes) || entry.bytes.length !== 1
+        || entry.bytes[0] !== entry.token.charCodeAt(0)) {
+        throw new ScoringError(`OpenRouter returned invalid bytes for label ${entry.token}.`);
+      }
+    }
+    found.set(entry.token, entry.logprob);
+  }
+
+  if (found.size === 0) {
+    throw new ScoringError("OpenRouter did not return logprobs for any choice label.");
+  }
+  const usage = parseUsage(value.usage);
+  return {
+    logprobs: Object.freeze(candidates.map((candidate) => found.get(candidate) ?? -Infinity)),
+    ...(usage === undefined ? {} : { usage }),
+  };
+}
+
+export function fromOpenRouter(options: OpenRouterOptions): Chooser {
+  if (!isRecord(options)) throw new TypeError("options must be an object.");
+  const { apiKey, model, provider, formatPrompt } = options;
+  requireText(apiKey, "apiKey");
+  requireText(model, "model");
+  if (provider !== undefined) requireText(provider, "provider");
+  if (options.fetch !== undefined && typeof options.fetch !== "function") {
+    throw new TypeError("fetch must be a function.");
+  }
+  const fetchImpl = options.fetch ?? globalThis.fetch;
+  if (typeof fetchImpl !== "function") throw new TypeError("A fetch implementation is required.");
+
+  const score: Scorer = async ({ prompt, candidates, signal }) => {
+    if (candidates.length > MAX_CANDIDATES) {
+      throw new TypeError(`OpenRouter supports at most ${MAX_CANDIDATES} choices.`);
+    }
+    const response = await post(fetchImpl, apiKey, {
+      model,
+      messages: [{ role: "user", content: prompt }],
+      max_tokens: 1,
+      stream: false,
+      temperature: 1,
+      top_p: 1,
+      logprobs: true,
+      top_logprobs: MAX_CANDIDATES,
+      reasoning_effort: "none",
+      ...(provider === undefined ? {} : {
+        provider: { only: [provider], allow_fallbacks: false },
+      }),
+    }, signal);
+    return parseScores(response, candidates);
+  };
+  return createFormattedChooser(score, formatPrompt === undefined ? {} : { formatPrompt }, "labels");
+}