diff --git a/CHANGELOG.md b/CHANGELOG.md index 0da41ae5..804ca0fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,51 @@ All notable changes to this package are documented here. +## 0.21.0 — 2026-08-28 + +Cloudflare Access becomes the canonical interactive-auth path for Worker +deployments. It authenticates both MCP clients and human operators before the +Worker runs, while connecta consumes only the trusted runtime identity. Clerk +is unchanged and remains supported: existing deployments can add Access, +verify the edge cutover, and remove Clerk later, with no storage migration or +token conversion. Node deployments can ignore this release beyond the version +pin. + +### Added + +- **Direct Worker Access auth.** `cloudflareAccessAuth()` ships from + `@zackbart/connecta/auth/cloudflare-access` with no dependency and no JWT + verifier. Human `ctx.access` identities may use MCP and operator routes; + service-token identities may use MCP but cannot mutate operator state. The + same suite runs under Node and workerd (#506). +- **Ambient operator sessions.** The operator shell selects Cloudflare Access + when the current invocation carries it, sends no browser-readable token, and + signs out through Cloudflare. A co-configured Clerk provider remains the + shell before Access is attached and the rollback path after it is detached + (#506). +- **Access-aware doctor.** `connecta doctor` accepts + `CF_ACCESS_CLIENT_ID`/`CF_ACCESS_CLIENT_SECRET` and sends the pair to health + and MCP requests. A partial pair fails before network access (#506). + +### Changed + +- **Cold reads stay in code mode.** One known canonical address still uses + `call_tool`, while an unknown-address read now starts with one + `execute_code` program and keeps discovery results off the model-facing + route. The current-version benchmark is reset to four deterministic + whole-agent cases covering both routes, exact provider semantics, private + pagination, forwarding bytes, tokens, and latency. +- **Worker deployment path.** The shipped Worker example uses Access and + Managed OAuth, carries a local `access.dev` identity, and documents service + tokens for unattended callers. Its Clerk shape stays beside the provider as + the reversible migration seam. Static connecta and operator-issued bearers + remain supported by core but are not standalone credentials through a + whole-Worker Access gate (#506). +- **Operator capability is vendor-neutral.** Inbound auth providers now declare + interactive-operator capability explicitly, and runtime context reaches + their authorization hook as an optional third argument. Existing custom + providers with the two-argument hook remain source-compatible (#506). + ## 0.20.0 — 2026-08-26 This release removes the two side languages that had grown around the seven diff --git a/README.md b/README.md index 33039785..923b9540 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,9 @@ Fifty issues in, one small object out. Your context window notices. There is also an operator surface, off until you turn it on: sign-in, an encrypted credential vault with rotation, revocable per-client tokens, and a -payload-free activity log. +payload-free activity log. Worker deployments can use Cloudflare Access for +both MCP and operator identity; Node deployments and existing Workers can use +Clerk. Connecta is not a platform, a marketplace, a policy engine, or a multi-tenant service. Those are decisions, and the [ethos](./ethos.md) records each one diff --git a/bin/connecta.mjs b/bin/connecta.mjs index ebe3defd..c457fcf5 100755 --- a/bin/connecta.mjs +++ b/bin/connecta.mjs @@ -27,7 +27,8 @@ function shellCd(path) { function usage() { console.log(`Usage: connecta init [directory] - CONNECTA_TOKEN= connecta doctor [--url http://localhost:8787]`); + CONNECTA_TOKEN= connecta doctor [--url http://localhost:8787] + CF_ACCESS_CLIENT_ID= CF_ACCESS_CLIENT_SECRET= connecta doctor --url https://worker.example`); } async function init() { @@ -169,19 +170,35 @@ async function doctor() { !loopbackHosts.has(parsedUrl.hostname) ) { throw new Error( - "Refusing to send a bearer token over remote plaintext HTTP. Use HTTPS.", + "Refusing to send authentication credentials over remote plaintext HTTP. Use HTTPS.", ); } const baseUrl = requestedUrl.replace(/\/+$/, ""); const token = process.env.CONNECTA_TOKEN; - if (!token) { + const accessClientId = process.env.CF_ACCESS_CLIENT_ID; + const accessClientSecret = process.env.CF_ACCESS_CLIENT_SECRET; + if (Boolean(accessClientId) !== Boolean(accessClientSecret)) { throw new Error( - "Set CONNECTA_TOKEN so doctor can inspect the MCP surface.", + "Set both CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET.", ); } + if (!token && !accessClientId) { + throw new Error( + "Set CONNECTA_TOKEN or a CF_ACCESS_CLIENT_ID/CF_ACCESS_CLIENT_SECRET pair so doctor can inspect the MCP surface.", + ); + } + const authHeaders = { + ...(token ? { Authorization: `Bearer ${token}` } : {}), + ...(accessClientId && accessClientSecret + ? { + "CF-Access-Client-Id": accessClientId, + "CF-Access-Client-Secret": accessClientSecret, + } + : {}), + }; const health = await jsonResponse( - await doctorFetch(`${baseUrl}/health`), + await doctorFetch(`${baseUrl}/health`, { headers: authHeaders }), ); if (health.status !== "ok") { throw new Error(`Unexpected health status: ${String(health.status)}`); @@ -206,7 +223,7 @@ async function doctor() { await doctorFetch(`${baseUrl}/mcp`, { method: "POST", headers: { - Authorization: `Bearer ${token}`, + ...authHeaders, "Content-Type": "application/json", Accept: "application/json, text/event-stream", }, diff --git a/documentation/architecture.md b/documentation/architecture.md index ef981e31..8586505b 100644 --- a/documentation/architecture.md +++ b/documentation/architecture.md @@ -48,6 +48,7 @@ read top to bottom. | Order | Route | Notes | | --- | --- | --- | | 0 | HTTPS upgrade | 308 to `publicUrl` when it is HTTPS and the request arrived over HTTP. Path and query are *assigned* onto the configured URL, never resolved against it, so a `//host` pathname cannot replace the deployment origin. `/health` is exempt: a loopback container probe must not depend on public DNS and TLS. `/ui` is canonicalized to `/` while upgrading. | +| 0 | Cloudflare Access (Worker deployment, when enabled) | Edge admission before this route table. Managed OAuth owns its challenge and discovery metadata; an admitted direct invocation carries trusted identity in `ctx.access`. | | 1 | `/ui/access-tokens[/]`, `/ui/credentials/[/]`, `/ui/oauth/` | Private mutation routes, matched **first** so nothing can shadow them and so they own their own `OPTIONS` — they answer it with a refusal rather than inheriting the wildcard CORS preflight. | | 2 | `OPTIONS` | Each auth provider's `handleMetadata` gets a chance (CORS preflight for browser MCP clients); otherwise 204 with MCP CORS. | | 3 | `/.well-known/*` | Auth providers' `handleMetadata`, open. 404 when none handles it. | @@ -72,7 +73,7 @@ any one file and a reordering reads like a harmless refactor. ([request admission](./request-admission.md)). The permit is held until the response *body* completes, not until the handler returns. 2. **Authorize.** Each `InboundAuth` provider's `authorize` in order, bearer - before Clerk. First `ok` admits; if all fail, the last provider's challenge + before interactive providers. First `ok` admits; if all fail, the last provider's challenge response is returned. No providers configured means open — development only, and it warns at construction. 3. **Refuse `?toolkit=`.** Toolkits were removed ([#178](https://github.com/zackbart/connecta/issues/178)) @@ -116,7 +117,9 @@ The Node-touching paths are `src/node.ts` (the `node:http` adapter), subpath export — `@zackbart/connecta/node`, `@zackbart/connecta/quickjs` — and must stay unreachable from the root entry. The optional Clerk adapter is behind `./auth/clerk` for the adjacent reason: `@clerk/backend` is an optional peer, -not a dependency. +not a dependency. The zero-dependency Cloudflare Access adapter likewise stays +behind `./auth/cloudflare-access`: it is Web-API-pure, but its trust contract is +specific to a direct Worker invocation carrying `ctx.access`. `test/purity.test.ts` walks the relative-import graph from `src/index.ts` and fails on (a) any `node:` specifier in a reachable file and (b) the Node @@ -156,7 +159,7 @@ src/ operator-ui/ the Preact app, its pure rules, and the built bundle connectors/ remote-mcp.ts, api.ts, guarded-fetch.ts providers/ the maintained prebuilt connections - auth/ bearer, clerk (optional peer), downstream OAuth + auth/ bearer, Cloudflare Access, clerk (optional peer), downstream OAuth executors/ the QuickJS pool and child (Node only) storage/ memory.ts, file.ts (Node only) node.ts listen() + fileStorage re-export (Node only) @@ -177,7 +180,7 @@ src/ the connector limiters, then the executor. Node's `listen()` calls it on SIGTERM/SIGINT. - **Structural mistakes throw at construction.** A duplicate connector id, an - invalid admission rule, `accessTokens` without a Clerk provider, a missing + invalid admission rule, `accessTokens` without an interactive operator provider, a missing executor: all refuse to boot. A deployment that starts in the wrong shape is worse than one that does not start. diff --git a/documentation/auth.md b/documentation/auth.md index 535e1d18..358aaa7e 100644 --- a/documentation/auth.md +++ b/documentation/auth.md @@ -1,8 +1,64 @@ # Inbound auth Inbound auth decides who may reach the MCP endpoint. A deployment may admit a -static bearer, operator-issued access tokens, Clerk identities, or a mixture. -Static bearers are checked first. +static bearer, operator-issued access tokens, Clerk identities, Cloudflare +Access identities on Workers, or a mixture. Static bearers are checked first; +the remaining providers keep configuration order. The first successful +identity owns the activity actor for that request. + +## Cloudflare Access on Workers + +[`cloudflareAccessAuth()`](https://developers.cloudflare.com/workers/configuration/cloudflare-access/) +is the Worker-specific path: + +```ts +import { cloudflareAccessAuth } from + "@zackbart/connecta/auth/cloudflare-access"; + +createConnecta({ + auth: cloudflareAccessAuth(), + connectors, + executor, +}); +``` + +The adapter trusts only `ctx.access`, which Cloudflare creates after Access has +authenticated a request that directly invokes the Worker. It calls +`ctx.access.getIdentity()` and never reads `Cf-Access-Jwt-Assertion`, downloads +signing keys, or accepts a JWT from the caller. A missing context or unreadable +identity fails closed. This also means it is deliberately not a Node or +`cloudflared` origin adapter, and it does not survive a Service Binding hop: +those shapes need their own explicit trust boundary. + +A human identity gets MCP and operator access. A Cloudflare service-token +identity gets MCP access and a stable activity subject, but no `userId`, so it +cannot write credentials, run downstream OAuth mutations, or issue connecta +tokens. Access policy decides who reaches the Worker; connecta does not mirror +email domains, groups, or device posture into a second policy layer. + +Enable [**Managed OAuth**](https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/managed-oauth/) +on the Access application for interactive MCP clients. +Cloudflare then owns the unauthenticated challenge and `/.well-known/` +metadata, issues opaque RFC 8707 tokens, and resolves them into the same trusted +Worker identity. Do not add a bypass for the discovery routes. A fully +automated client instead uses a [Cloudflare Access service token](https://developers.cloudflare.com/cloudflare-one/access-controls/service-credentials/service-tokens/) +through the +`CF-Access-Client-Id` and `CF-Access-Client-Secret` headers. + +Worker-level Access runs before every connecta route. Consequently: + +- `/health`, operator pages, downstream OAuth callbacks, connector-owned + routes, and `/mcp` all require Access unless a more-specific hostname/path + policy says otherwise; +- a static connecta bearer and a `cta_…` token are not standalone edge + credentials, because Cloudflare rejects them before connecta sees them; and +- a connector that intentionally exposes a public webhook needs a + more-specific Access application and bypass policy. Do not bypass connecta's + OAuth discovery paths when Managed OAuth is enabled. + +The [Worker example](../examples/worker/) carries the complete deployment shape +and the [upgrade guide](./upgrading.md#0200--0210) gives the reversible Clerk +migration. ## Clerk configuration is checked at construction @@ -17,7 +73,7 @@ request instead of a base64 stack on every route. ## Operator-issued access tokens -Set `accessTokens: {}` to let eligible Clerk operators create named Bearer +Set `accessTokens: {}` to let eligible interactive operators create named Bearer tokens at `/tokens`: ```ts @@ -41,7 +97,7 @@ Revoked records remain as metadata tombstones so historical calls keep their friendly attribution. Access tokens authenticate MCP clients; they are never operator credentials. -Creation, rename, and revocation require the same eligible Clerk identity and +Creation, rename, and revocation require the same eligible human identity and same-origin mutation boundary as connector credentials. `maxActive` defaults to 100 and can be set from 1 through 1,000. @@ -51,7 +107,7 @@ effect globally without a convergence window. Operator credential mutation is a separate, narrower boundary. The `/credentials` shell contains no secret data before authentication, and the -mutation API requires same-origin requests from an admitted Clerk user. An MCP +mutation API requires same-origin requests from an admitted operator. An MCP bearer is never treated as an operator credential, even when it can call every connector. @@ -59,10 +115,10 @@ This split is visible in recovery: - a bearer-authenticated agent may receive `recovery: "operator_config"` and pass its `operatorUrl` to a human; -- a Clerk-authenticated operator opens that URL, signs in, and updates the +- an interactive operator opens that URL, signs in, and updates the credential; and - a bearer-only deployment still returns the handoff honestly, but mutation - remains unavailable until Clerk operator auth is configured. + remains unavailable until interactive operator auth is configured. See [meta-tools](./meta-tools.md#authorization-recovery) for the stable recovery envelope and [storage and credentials](./storage-and-credentials.md) for vault diff --git a/documentation/code-mode.md b/documentation/code-mode.md index faefa17b..da5538b9 100644 --- a/documentation/code-mode.md +++ b/documentation/code-mode.md @@ -889,11 +889,11 @@ the upstream `Executor` shape assignable. | `X6` | `test/quickjs-executor.test.ts` (never-settling await) | | `X7` | `P3`'s tests; the Workers superset is deliberately unused | -The surface itself is checked by `test/server.test.ts` (the exact seven-tool -list) and `test/code-first-surface.test.ts` (the fold's construction rules, the +The surface itself is checked by `test/server.test.ts` (the exact seven-tool list) +and `test/code-first-surface.test.ts` (the fold's construction rules, the required executor, the refusals a removed top-level tool now gets, copy, and -measured size). There is one shape left to audit, so there is one audit: +measured size). The small whole-agent benchmark checks both read routes, provider semantics, and private pagination: ```sh -npm --prefix eval/current-version run audit +npm --prefix eval/current-version run benchmark ``` diff --git a/documentation/meta-tools.md b/documentation/meta-tools.md index 5482eb46..6203bb13 100644 --- a/documentation/meta-tools.md +++ b/documentation/meta-tools.md @@ -12,19 +12,19 @@ Every deployment requires an executor and `tools/list` is exactly seven: batching live in `connecta.search`, `connecta.describe`, and `connecta.batch` inside a program ([#273](https://github.com/zackbart/connecta/issues/273)). -Code-first is what a model sees. Four overlapping ways to reach one connector -became two: `search_tools` then `call_tool` for a single cold read — measurably -cheaper direct than through a program — and `execute_code` for everything wider. -The consolidation removed overlapping routing choices while preserving the -cheaper direct path for one cold call. The [guest API contract](./code-mode.md) -is what a program is promised. - -The route is chosen before discovery. A result that will be reduced, a call -whose arguments depend on an earlier result, or work with multiple operations -starts with one `execute_code` call and keeps discovery, calls, and reduction -inside it. Distinct operations get distinct short `connecta.search` queries in -that program. Only one unknown-address read takes the cheaper top-level -`search_tools` → `call_tool` path; a known address needs only `call_tool`. +Code-first is what a model sees. Read-only work has two routes: `call_tool` for +one known address, and `execute_code` when discovery or any wider work is +needed. Real hosted catalogs reversed the earlier synthetic result that made a +top-level cold search look cheaper. Keeping discovery inside the program avoids +returning every candidate schema to the model and removes a model round trip. +The [guest API contract](./code-mode.md) is what a program is promised. + +The route is chosen before discovery. An unknown address, a result that will be +reduced, a call whose arguments depend on an earlier result, or work with +multiple operations starts with one `execute_code` call and keeps discovery, +calls, and reduction inside it. Distinct operations get distinct short +`connecta.search` queries in that program. A known address needs only +`call_tool`. That routing is about read-only work, because that is the only work a program can do. Anything unannotated, write-capable, or destructive is inadmissible @@ -68,13 +68,15 @@ and a truncated line ends with the exact `+N more` count. This reads only the configured registry: it loads no catalog, probes no credential, grants no capability, and does not replace canonical discovery or addressing. -Start an unknown-address lookup with two to four distinctive action/object -terms, not the full request, and omit `limit` so the default eight-result page -stays small. When the integration is obvious, set `connector` to its id: a -scoped search loads that catalog alone, while an unscoped search must fan out -across every configured connector. Leave the search unscoped when the right -integration is genuinely ambiguous. Set `safety: "readOnly"` when the result is -headed to `call_tool` or generated code; `safety: "approvalRequired"` finds the +Start a lookup with two to four distinctive action/object terms, not the full +request. Read-only lookup belongs in `connecta.search` inside the program. +Top-level `search_tools` remains available for explicit catalog inspection and +approval-required discovery. Omit `limit` initially so the default +eight-result page stays small. When the integration is obvious, set +`connector` to its id: a scoped search loads that catalog alone, while an +unscoped search must fan out across every configured connector. Leave the +search unscoped when the right integration is genuinely ambiguous. Set +`safety: "readOnly"` for generated code; `safety: "approvalRequired"` finds the complementary set that must cross `call_destructive_tool`. Omitting `safety`, or setting it to `"all"`, preserves the complete configured catalog. This is only a discovery filter: it neither grants authority nor changes invocation admission. diff --git a/documentation/operations.md b/documentation/operations.md index 0b527dbb..cb3fa227 100644 --- a/documentation/operations.md +++ b/documentation/operations.md @@ -45,6 +45,10 @@ one. npx @zackbart/connecta init my-deployment cd my-deployment && npm install && npm start CONNECTA_TOKEN=… npx connecta doctor --url http://localhost:8787 + +# A Worker protected by Cloudflare Access +CF_ACCESS_CLIENT_ID=… CF_ACCESS_CLIENT_SECRET=… \ + npx connecta doctor --url https://connecta.example.workers.dev ``` `init` copies the template, pins the generated deployment to the CLI package's @@ -59,12 +63,17 @@ on the way out: `QuickJS` on the Node template, `DynamicWorkerExecutor` on the Worker example, and `code executed` when an executor identifies as nothing — a checker that asserts a sandbox it never saw is worse than one that says it does not know ([#368](https://github.com/zackbart/connecta/issues/368)). It -refuses to send a bearer token over remote plaintext HTTP, and it +refuses to send authentication credentials over remote plaintext HTTP, and it *reports* catalog drift without failing on it — an unclassified downstream tool already fails closed onto `call_destructive_tool`, so drift is a maintainer's next task rather than a broken deployment ([#343](https://github.com/zackbart/connecta/issues/343)). +For an Access-protected Worker, doctor sends the service-token pair to both +`/health` and `/mcp`; Access authenticates those requests before connecta runs. +`CONNECTA_TOKEN` remains the Node and legacy Worker path, and may be supplied +alongside the Access pair during rollback testing. A partial pair is refused. + ### Configuration Structural seams stay top-level; tuning is grouped by subsystem. Every group is @@ -74,7 +83,7 @@ optional. | --- | --- | --- | | `connectors` | — (required) | the connector set ([connectors](./connectors.md)) | | `executor` | — (required) | the sandbox `execute_code` runs in ([code mode](./code-mode.md#what-an-executor-must-implement)) | -| `auth?` | none ⇒ open (dev only) | one `InboundAuth` or an array; bearer providers are checked before Clerk ([inbound auth](./auth.md)) | +| `auth?` | none ⇒ open (dev only) | one `InboundAuth` or an array; bearer providers are checked before interactive providers ([inbound auth](./auth.md)) | | `storage?` | `memoryStorage()` | the one state seam for catalogs, result paging, credentials, and access tokens ([storage](./storage-and-credentials.md)) | | `publicUrl?` | per-request origin | public base URL; an HTTPS value also redirects inbound HTTP | | `logger?` | `console`, prefixed `[connecta]` | `{ debug, info, warn, error }` | @@ -83,7 +92,7 @@ optional. | `deploymentInfo?` | unset | arbitrary metadata exposed by `/health` | | `activity?` | unset | `{ store, readGate?, deploymentId? }` — payload-free activity storage, an optional operator-read gate, and a stable event label | | `credentials.encryptionKey?` | unset | base64 32-byte AES key for the connector vault. Without it, connectors declaring `credential` warn and their slots stay unmanageable | -| `accessTokens?` | unset | `{ maxActive? }` (default 100) for operator-issued MCP bearer tokens. Requires a Clerk provider, or construction throws ([access tokens](./auth.md#operator-issued-access-tokens)) | +| `accessTokens?` | unset | `{ maxActive? }` (default 100) for operator-issued MCP bearer tokens. Requires an interactive operator provider, or construction throws ([access tokens](./auth.md#operator-issued-access-tokens)) | | `discovery.concurrency?` | 4 | connector catalogs/status probes in flight at once | | `discovery.catalogTtlSeconds?` | 300 | fresh TTL for cached tool lists | | `discovery.persistCatalog?` | true | persist serializable catalogs as a manifest plus revision-addressed chunks | @@ -214,7 +223,7 @@ in. | Suite | Covers | | --- | --- | -| `access-tokens.test.ts` | the `AccessTokenManager` — a one-time secret created, authenticated, renamed, and revoked, bounded names and active count, enumerable storage required, a deployment with no Clerk operator refused — and the Clerk-only routes, down to historical activity still resolving a revoked token's name | +| `access-tokens.test.ts` | the `AccessTokenManager` — a one-time secret created, authenticated, renamed, and revoked, bounded names and active count, enumerable storage required, a deployment with no interactive operator refused — and the operator-only routes, down to historical activity still resolving a revoked token's name | | `activity.test.ts` | payload-free delivery: a rejected async write attaches to `waitUntil` instead of throwing, approved destructive calls record under their real entry point, result-size friction records without retaining the result, and a hallucinated connector id or invented identity is clamped so the event still cannot carry a payload | | `api-connector.test.ts` | `api()` — kind, description, tool defs, dispatch, default args, unknown tools, handler throws, argument validation, and the construction contract | | `bearer.test.ts` | constant-time bearer compare, case-insensitive scheme, 401 challenges, and the retired audience options refusing rather than silently unbinding | @@ -223,6 +232,7 @@ in. | `catalog-drift.test.ts` | `vettedCatalog()`, `detectCatalogDrift()`, and `withVettedCatalog()`; drift on the registry surface and on `/health`; the connector seam projected rather than echoed; and the drift types being public | | `catalog.test.ts` | lexical ranking and the compact schema renderer — `const`, `allOf` beside siblings, `$ref`, the depth limit, per-schema caching, and 2020-12 keyword compatibility | | `clerk.test.ts` | protected-resource metadata, the browser sign-in config, OAuth and session tokens, cached best-effort activity labels with their caps, the hand-applied `azp` rejection, and the `allowedDomains` allowlist including every lookalike that must not be repaired into a match | +| `cloudflare-access-auth.test.ts` | trusted `ctx.access` human and service identities, absent/error fail-closed behavior, service-token MCP admission without operator mutation, human same-origin mutation, and the Clerk-to-ambient shell switch | | `cloudflare-provider.test.ts` | `cloudflare()` construction, tool surface, request building, projections, typed failures, and credential test | | `code-first-surface.test.ts` | the seven-tool surface itself — an executor required, every removed option and top-level tool refused, compact always-loaded routing pinned below 1,000 characters, complete on-demand usage served, and `connecta.ui` findable before connector search | | `codemode-compat.test.ts` | the `Executor` seam staying structurally compatible with `@cloudflare/codemode`'s `DynamicWorkerExecutor`, enforced by `tsc` | @@ -244,7 +254,7 @@ in. | `mixpanel-provider.test.ts` | the Mixpanel proxy, its conditional-input guide and complete reviewed schema-digest manifest | | `notion-provider.test.ts` | Notion's deliberate tool surface, including declined expanded page inputs, request construction, lean projections, both pagination conventions, error mapping, and writes | | `operator-boundary.test.ts` | the operator row of the decisions table, after every mutation route: authentication material managed without moving a declared structure, and the one honest exception — a credential write making a remote catalog appear, which is discovery arriving, not an operator editing the deployment | -| `operator-store.test.ts` | `src/operator-ui/app/store.ts` against a fake browser: the Clerk listener, `gate()`, the generation fence, and the request path | +| `operator-store.test.ts` | `src/operator-ui/app/store.ts` against a fake browser: the Clerk listener, ambient Access requests without a browser-readable token, `gate()`, the generation fence, and the request path | | `provider-conventions.test.ts` | the conventions a test can hold: hand-written providers refusing schemas they cannot enforce (H5), their compact discovery schemas staying complete (H7), Cloudflare stating its second pagination convention in the schema (H10), and Notion saying it has no escape hatch (H14) | | `provider-registry.test.ts` | all six maintained providers inside real deployments: boot, description, address, catalog, storage, credential, admission, and activity isolation; plus provider-specific discovery and guide contracts | | `registry.test.ts` | construction and id validation, startup warnings, address resolution, version 2 catalog TTL/persistence/completeness, agent-only stale-while-revalidate with cross-request single-flight shared with blocking reads in both start orders, owned teardown, invalidation/fingerprint guards, blocking diagnostics, and broken-connector isolation | @@ -272,13 +282,13 @@ justification for *not* re-running it in workerd, so "it was easier" is not one. | --- | --- | --- | | `deployment-shapes.test.ts` | the Worker as the only example with a loader-only sandbox, one Node template that is also its own container, the same source running locally and in the container, the Node template's pinned esbuild install-script approval, the full operator surface in both, a template that cannot start on its own `.env.example`, a Worker README naming every optional peer its entrypoint imports, and the initializer's `.gitignore` staying in step | walks the template and example trees with Node filesystem APIs | | `doc-links.test.ts` | the documentation checker itself — local file and fragment resolution, repository URLs resolved back to the checkout, duplicate heading slugs, fenced-code exclusion, and useful failures | spawns the Node checker against filesystem fixtures | -| `doctor-cli.test.ts` | `connecta doctor`'s executor line end to end — the sandbox the deployment reports is the one named, an unidentifiable executor gets an executor-neutral line, and a hostile name is bounded and stripped before it reaches a terminal | spawns the CLI against a Node HTTP deployment over real sockets | +| `doctor-cli.test.ts` | `connecta doctor`'s executor line and credentials end to end — the sandbox the deployment reports is the one named, an unidentifiable executor gets an executor-neutral line, a hostile name is bounded, and a complete Cloudflare Access service-token pair is accepted while a partial pair is refused | spawns the CLI against a Node HTTP deployment over real sockets | | `drift-check.test.ts` | the maintainer drift checker — hosted-provider credential framing, recorded touched endpoints, a quiet revision bump, clear failures for an unavailable spec/manifest/credential, `$ref` traversal, and one well-formed row per endpoint | spawns the Node checker against filesystem fixtures | | `file-storage.test.ts` | `fileStorage()` across instances, logical TTL plus physical pruning without clobbering a newer value, and corrupt-file quarantine | exercises the Node filesystem storage adapter | | `guest-api-contract-quickjs.test.ts` | the shared guest-contract cases on the real QuickJS executor, including identical caught failure codes and inline describe recovery, its exact absent globals, and blocked runtime imports | runs the contract cases on the Node QuickJS executor | | `node.test.ts` | the `listen()` adapter propagating an HTTP client disconnect through the Web `Request` and the MCP handler into a program's connector call, releasing both admission permits | exercises the Node HTTP adapter over real TCP sockets | | `packed-links.test.ts` | the packed-link gate itself — shipped targets and repository URLs accepted, relative links into unshipped paths and directories rejected with the citation to write instead, reference definitions seen, fenced examples ignored, the changelog exempt | spawns the Node packed-link gate against filesystem fixtures | -| `package-surface.test.ts` | the published boundary — built output shipped, the `exports` map carrying exactly the documented subpaths plus `./package.json`, only generic factories, platform storage kept in examples, Clerk and QuickJS behind optional subpaths, every provider independently importable, and the Cloudflare provider free of bare specifiers | walks the package tree with Node filesystem APIs | +| `package-surface.test.ts` | the published boundary — built output shipped, the `exports` map carrying exactly the documented subpaths plus `./package.json`, only generic factories, platform storage kept in examples, Clerk and QuickJS behind optional subpaths, dependency-free Cloudflare Access behind its Worker subpath, every provider independently importable, and the Cloudflare API provider free of bare specifiers | walks the package tree with Node filesystem APIs | | `purity.test.ts` | the import-graph guardrail ([architecture](./architecture.md#import-graph-purity)) — the core stays Workers-clean | walks the source import graph with Node filesystem APIs | | `quickjs-child-entry.test.ts` | a missing QuickJS child entry failing before `fork()`, with the expected path and the bundler-externalization constraint | mocks Node child-process and filesystem APIs | | `quickjs-child-stderr.test.ts` | abnormal child exits retaining only an 8 KiB stderr tail, included in the parent-side diagnostic | mocks Node child-process streams | @@ -319,10 +329,15 @@ confinement does too. - **A connector with no `verifyState` refuses every callback.** That is the designed behavior, not a bug: handing an unverified code to `finishAuth` is the vulnerability. The startup warning names the connector. -- **401 loops from a client that cannot discover auth.** The client must reach +- **401 loops from a Clerk client that cannot discover auth.** The client must reach the open `/.well-known/oauth-protected-resource` (and the `/mcp` variant); confirm CORS and the Clerk keys, and that DCR is enabled on the Clerk instance. +- **An Access-protected MCP client receives redirects or loops.** Enable Managed + OAuth on the Access application and use an RFC 8707-capable client. Access, + not connecta, must answer the unauthenticated challenge and `/.well-known/` + metadata. Do not bypass those routes. For unattended automation, use an + Access Service Auth policy and service-token headers instead. - **No sessions and no server push, by design.** The transport is stateless. Scope resolves per request, which is also where the MCP spec has arrived. - **A tool that should be callable from a program is not.** Only tools diff --git a/documentation/operator-ui.md b/documentation/operator-ui.md index 460ae9cf..0fcc14fc 100644 --- a/documentation/operator-ui.md +++ b/documentation/operator-ui.md @@ -39,10 +39,25 @@ their walkthrough, which is the honest version of the same page count. | `src/operator-ui/browser.css` | One stylesheet, inlined into the shell. | | `src/operator-ui/generated.ts` | The build output: the bundle and the stylesheet as two exported strings. | -The server renders a mount point, not a page. Branding, the Clerk loader, and -every operator-configured URL stay in `src/ui.ts`, where they are gated before -they can become an attribute; the bundle renders everything that has a state. -Two roots share one store: `#operatorNav` and `#operatorContent`. +The server renders a mount point, not a page. Branding, the optional Clerk +loader, and every operator-configured URL stay in `src/ui.ts`, where they are +gated before they can become an attribute; the bundle renders everything that +has a state. Two roots share one store: `#operatorNav` and `#operatorContent`. + +Cloudflare Access is ambient browser auth. When the current Worker invocation +has `ctx.access`, the shell selects the `cloudflare-access` UI mode, emits no +Clerk loader, and sends no browser-readable token. Same-origin fetch includes +the HttpOnly `CF_Authorization` cookie, Access admits it at the edge, and the +server reads the resulting runtime identity. Sign out navigates to +`/cdn-cgi/access/logout`. Mutations still require an exact same-origin +`Origin`; an ambient cookie does not weaken the CSRF boundary. + +This runtime selection is the Clerk migration seam. A deployment may contain +both providers: before Worker-level Access is attached, the data-free shell +selects Clerk; after Access supplies `ctx.access`, it selects ambient auth. That +is not two same-hostname gates running in parallel. Access is upstream and a +request it rejects never reaches Clerk. Keeping Clerk in the array preserves a +code-level rollback after Access is detached. The Clerk loader is intentionally blocking. The inline operator bundle calls `boot()` as soon as the parser reaches the end of the body, so a deferred Clerk @@ -58,7 +73,8 @@ the same ordering. credential, token, and activity data arrives only through the authenticated `/ui/*` APIs, and the shell is identical whether or not a caller is signed in. - **One store, one identity.** `store.ts` is the only file that touches `fetch`, - `localStorage`, or Clerk. Every request carries the current session's token, + `localStorage`, Clerk, or the ambient Access mode. Every token-bearing request carries the current session's token, + while Access requests deliberately carry none, and every response is dropped unless the identity that asked for it is still the one on screen. `resetIdentity` replaces all identity-scoped state at once and bumps a generation that work already in flight compares itself against. diff --git a/documentation/upgrading.md b/documentation/upgrading.md index a485d5e6..22f58bfe 100644 --- a/documentation/upgrading.md +++ b/documentation/upgrading.md @@ -57,7 +57,7 @@ exist so far: | --- | --- | --- | | **pre-template** | before 0.10.2 | no `connecta init` existed; hand-written, or copied from the retired `examples/node` | | **A** | 0.10.2 – 0.15.1 | `.env.example`, `.gitignore`, `AGENTS.md`, `CLAUDE.md`, `README.md`, `package.json`, `src/index.ts`, `tsconfig.json` | -| **B** | 0.16.0 – 0.20.0 | adds `.dockerignore`, `Dockerfile`, `docker-compose.yml`, and `src/file-activity.ts`; `src/index.ts` grows the four commented operator blocks; `.env.example` ships `CONNECTA_TOKEN=` empty | +| **B** | 0.16.0 – 0.21.0 | adds `.dockerignore`, `Dockerfile`, `docker-compose.yml`, and `src/file-activity.ts`; `src/index.ts` grows the four commented operator blocks; `.env.example` ships `CONNECTA_TOKEN=` empty | Generation A is a decade in template years and identifying it precisely does not matter, because you are about to reconstruct it exactly rather than guess @@ -106,7 +106,7 @@ know what to preserve, once to know what to re-verify at the end. ### Bump the pin and install ```sh -npm pkg set dependencies.@zackbart/connecta=0.20.0 +npm pkg set dependencies.@zackbart/connecta=0.21.0 npm install ``` @@ -130,7 +130,7 @@ Generate the *current* template beside the base you already made, into the same `$SCRATCH`: ```sh -(cd "$SCRATCH" && npx @zackbart/connecta@0.20.0 init current) +(cd "$SCRATCH" && npx @zackbart/connecta@0.21.0 init current) ``` You now have a three-way merge with a real base: `$SCRATCH/base` is what this @@ -186,7 +186,7 @@ A deployment older than 0.10.2 has no base to diff against. Do not try to manufacture one. Instead: 1. `SCRATCH=$(mktemp -d)`, then - `(cd "$SCRATCH" && npx @zackbart/connecta@0.20.0 init current)` — there is no + `(cd "$SCRATCH" && npx @zackbart/connecta@0.21.0 init current)` — there is no `base` leg here, only the current template to read from. 2. Copy `$SCRATCH/current` into the deployment file by file, **skipping `src/index.ts`**. @@ -207,6 +207,74 @@ first, so cross them bottom-up: start at the oldest one still above this deployment's pin and work back up the page, because each boundary assumes the older ones are already done. +### 0.20.0 → 0.21.0 + +This is additive for Node and existing Clerk deployments. The new Worker path +uses Cloudflare Access identity directly and removes Clerk only after the edge +cutover has been verified. An agent can perform every repository edit; a human +must attach Access, choose its policy, create service credentials, and enable +Managed OAuth in the Cloudflare dashboard. + +For a Worker currently using Clerk, keep rollback live through the cutover: + +1. Bump and install 0.21.0. Add the new provider **before** the existing Clerk + provider, but remove nothing: + + ```ts + import { cloudflareAccessAuth } from + "@zackbart/connecta/auth/cloudflare-access"; + import { clerkAuth } from "@zackbart/connecta/auth/clerk"; + + auth: [ + cloudflareAccessAuth(), + // Keep the deployment's existing options and secrets unchanged. + clerkAuth({ /* existing configuration */ }), + ], + ``` + + Before Access is attached, the new provider fails closed and the operator + shell selects Clerk. Deploy this state and run doctor with the existing + `CONNECTA_TOKEN`. This separates the package/code change from the edge + change and proves the old path still works. + +2. In Cloudflare, attach Access to the Worker, apply the intended human policy, + and enable Managed OAuth. Create an Access service token and a **Service + Auth** policy for doctor and fully unattended clients. Do not create a + bypass for `/.well-known/*`; Managed OAuth owns that discovery surface. + +3. Reconnect interactive MCP clients to `/mcp`. Their old Clerk + OAuth tokens are not Cloudflare credentials, so each client performs one new + browser authorization. An agent can edit client configuration and start the + flow; the user still completes the identity-provider prompt. Move CI, cron, + and server-to-server callers from connecta bearers to the two Access service + headers. The cutover warning is literal: once Access is attached, a static + bearer or `cta_…` token by itself is stopped at the edge before connecta can + inspect it. + +4. Verify the edge path: + + ```sh + CF_ACCESS_CLIENT_ID=… CF_ACCESS_CLIENT_SECRET=… \ + npx connecta doctor --url https://connecta.example.workers.dev + ``` + + Open `/` as a human and exercise any enabled credential, token, and OAuth + controls. A service token may pass doctor and MCP but must receive 403 from + operator mutations. + +5. After an observation window, remove `clerkAuth`, its import, + `@clerk/backend`, and the Clerk variables/secrets. Until then they are inert + behind Access but preserve rollback. Rollback order matters: detach Access + first, then the untouched Clerk sessions and connecta bearers reach the + Worker again. Reverting code first cannot help a request the edge still + blocks. There is no storage migration and no token-format conversion. + +If the Worker exposes an intentionally public connector route, create a +more-specific hostname/path Access application with a Bypass policy for that +route only. `/health`, downstream OAuth callbacks, operator shells, and MCP are +private under the canonical whole-Worker shape; doctor knows how to authenticate +its health request. + ### 0.19.0 → 0.20.0 Three intake paths become deliberately strict. None changes storage, the two diff --git a/ethos.md b/ethos.md index 7a93cead..eb020eab 100644 --- a/ethos.md +++ b/ethos.md @@ -1,9 +1,7 @@ # connecta — ethos What connecta is, what it refuses to be, and the invariants every change must -preserve. Deliberately terse: when a change contradicts a line here, either the -change is wrong or this file needs amending — in that order, and amending it is -a design decision, not a drive-by edit. +preserve. A contradiction needs a design decision, not a drive-by edit. ## What this is @@ -78,6 +76,7 @@ CHANGELOG, not here. | MRTR / `input_required` passthrough | gated | relayable statelessly; no host or downstream emits it yet ([#176](https://github.com/zackbart/connecta/issues/176)) | | Downstream `ttlMs` cache hints | gated | needs refresh-churn evidence ([#206](https://github.com/zackbart/connecta/issues/206)) | | Downstream MCP Apps template passthrough | gated | needs a downstream that ships one ([#266](https://github.com/zackbart/connecta/issues/266)) | +| Worker Access inbound auth | provisional | Managed OAuth and Clerk migration need production evidence ([#506](https://github.com/zackbart/connecta/issues/506)) | | Program UI tool calls | removed | the read bridge added a second contract without improving agent data retrieval; views are display-only again ([#287](https://github.com/zackbart/connecta/issues/287), [#484](https://github.com/zackbart/connecta/issues/484)) | ## Invariants diff --git a/eval/current-version/.gitignore b/eval/current-version/.gitignore new file mode 100644 index 00000000..df131c26 --- /dev/null +++ b/eval/current-version/.gitignore @@ -0,0 +1,2 @@ +/results/latest*.json +/results/latest*.md diff --git a/eval/current-version/README.md b/eval/current-version/README.md index 1cb50ef7..28bb2203 100644 --- a/eval/current-version/README.md +++ b/eval/current-version/README.md @@ -1,470 +1,38 @@ -# Current-version release audit +# Current-version benchmark -This isolated Node-only harness qualifies the checked-out Connecta source -without changing production activity storage or calling real external -accounts. It exercises: +This is the active whole-agent benchmark for Connecta's seven-tool surface. It is deliberately small: four deterministic cases, one runner, one fixture server, and exact pass/fail checks. -- discovery and full schema description; -- direct calls, batching, and code-mode reduction; -- truncation and result paging; -- destructive approval routing; -- successful and unavailable OAuth recovery; -- successful operator-managed static-credential recovery and its unavailable - path; and -- the payload-free activity event shape. +The cases answer four product questions: -The discovery suite runs through the real MCP transport against -[`discovery-holdout.json`](./discovery-holdout.json). That corpus was authored -before the closed #188 research corpus was inspected. It is held-out release -evidence: do not tune ranking rules, stopwords, aliases, or thresholds against -its cases. +1. Does a cold unknown-address read stay inside one `execute_code` program? +2. Does a known-address read take the cheaper direct `call_tool` route? +3. Does the agent obey provider semantics rather than fuzzy-matching plausible names? +4. Does code mode paginate and reduce private records without forwarding them to the model? -Ranking development uses the separate -[`discovery-development.json`](./discovery-development.json). It contains the -mixed all/partial analytics decoy that reproduced #326 without adding the -fixture to the sealed holdout. Run its isolated deterministic lane with: +Every run records whole-agent input, cached-input, and output tokens from the Codex CLI; model-visible MCP result tokens; outer MCP response bytes plus separate `content` and `structuredContent` bytes; latency; chosen meta-tools; exact downstream calls; and the final answer. A run passes only when routing, answer, semantics, and privacy all pass. -```sh -npm --prefix eval/current-version run audit:development -``` - -The development report records exact expected top-1 accuracy, recall, -precision, and the absence of the removed per-result query-coverage wire. Its -server advertises only the synthetic development connector. The release gate -remains the complete holdout `audit` command. -The current removal report is -[`results/issue-323-removal-evidence.md`](./results/issue-323-removal-evidence.md). -The causal coverage-shape history remains in -[`results/issue-322-evidence.md`](./results/issue-322-evidence.md). - -### Reproduce the issue #322 trailing audits - -The committed discovery adapter reads the verbose, indexed, and trailing-entry -coverage shapes. Reproduce the trailing deterministic reports from a clean -checkout by replacing `PREREG_COMMIT` with the commit that contains -[`issue-322-qualification-plan.json`](./issue-322-qualification-plan.json): - -```sh -git worktree add --detach /tmp/connecta-322-trailing \ - bbfb5220cb94342acc21dadd7db9fe1bbcf5ce4c -git -C /tmp/connecta-322-trailing restore --source PREREG_COMMIT \ - --staged --worktree eval/current-version -(cd /tmp/connecta-322-trailing && npm ci) -(cd /tmp/connecta-322-trailing/eval/current-version && npm ci) -npm --prefix /tmp/connecta-322-trailing/eval/current-version \ - run audit:development -- --source-commit \ - bbfb5220cb94342acc21dadd7db9fe1bbcf5ce4c -npm --prefix /tmp/connecta-322-trailing/eval/current-version run audit -- \ - --source-commit bbfb5220cb94342acc21dadd7db9fe1bbcf5ce4c -``` - -The coverage-off comparator uses the same preregistration commit plus the exact -[`issue-322-coverage-off.patch`](./patches/issue-322-coverage-off.patch). Its -SHA-256 is recorded in the preregistration plan. The qualification runner -rejects mismatched commits, patches, harnesses, corpora, sandboxes, runtimes, -models, or CLI versions before it starts a sample. - -For the committed #322 result, the plan commit existed locally at -05:01:23Z before sampling. GitHub recorded its PushEvent at 05:03:14Z, after -the first trailing batch ended at 05:02:39Z and four seconds before the first -off batch ended at 05:03:18Z. This is local precommitment, not remote -preregistration proof. The delayed push weakens the formal claim, but it does -not weaken the conservative BLOCK verdict produced by the fixed gates. - -```sh -git worktree add --detach /tmp/connecta-322-off PREREG_COMMIT -git -C /tmp/connecta-322-off apply \ - eval/current-version/patches/issue-322-coverage-off.patch -(cd /tmp/connecta-322-off && npm ci) -(cd /tmp/connecta-322-off/eval/current-version && npm ci) -CONNECTA_EVAL_AGENT_MODEL=gpt-5.6-sol node \ - eval/current-version/issue-322-qualification-runner.mjs \ - --off-worktree /tmp/connecta-322-off \ - --trailing-worktree /tmp/connecta-322-trailing -``` - -The paired cold-agent decoy lane uses the same case, model, repetitions, and -concurrency on both product commits: - -```sh -CONNECTA_EVAL_AGENT_MODEL=gpt-5.6-sol \ - npm --prefix eval/current-version run perf:lookup -- \ - --case mixed-decoy-organizations \ - --repetitions 10 \ - --concurrency 5 -``` - -## Run - -Install the repository and audit dependencies once: - -```sh -npm ci -(cd eval/current-version && npm ci) -``` - -Then one command runs the complete audit and writes machine-readable JSON plus -a concise Markdown qualification report: - -```sh -npm --prefix eval/current-version run audit -``` - -The command runs on Node 22. It records the source commit, Node runtime, -tokenizer, holdout hash, task outcomes, round trips, client-observed latency, -and exact JSON-serialized definition, request, and response token surfaces. -The default tokenizer is `o200k_base`; override it with -`CONNECTA_EVAL_TOKENIZER`. - -The suite measures the required isolated QuickJS executor and seven-tool -surface. Inventory, schema description, and batching are exercised through -`connecta.search`, `connecta.describe`, and `connecta.batch` inside -`execute_code`. CI runs the command on pushes to `main`. - -Every JSON result and Markdown report records the advertised surface, and the -qualification gate asserts that the connected server advertises exactly the -seven expected meta-tool names — a deployment that regressed to another surface -fails the audit instead of being filed as seven-tool evidence. The harness also -validates each task's top-level route against the advertised tool list before -calling it, so a surface/task mismatch fails with an audit-specific -configuration error. +## Run it -Choose stable output names for release evidence: - -```sh -npm --prefix eval/current-version run audit -- \ - --output results/0.8.0-baseline.json \ - --report results/0.8.0-baseline.md -``` - -The audit is intentionally outside the root TypeScript, Vitest, Knip, purity, -and published-package graphs. Validate its TypeScript separately: +Prerequisites are a working `codex` CLI login and the repository's normal Node dependencies. ```sh +npm --prefix eval/current-version install npm --prefix eval/current-version run check +npm --prefix eval/current-version run benchmark ``` -## Full performance analysis - -The performance lane adds two measurements that the release audit deliberately -does not: - -- `perf:logic` measures startup, catalog scaling, discovery, direct calls, - batching, concurrent throughput, memory after GC, and cold/warm QuickJS - overhead against synthetic 100-, 1,000-, and 10,000-tool deployments. -- `perf:agent` starts a fresh isolated server and a fresh Codex session for - each task. The task prompts do not explain Connecta's routing workflow. The - runner scores answer and execution correctness, safety, advertised-surface - validity, foreign and redundant calls, Connecta round trips and result - tokens, whole-agent tokens, and wall time. Ordinary cases accept multiple - valid routes; cases with an explicit routing policy additionally score the - intended outer-tool sequence. - -Run the complete environment: +The default is three sequential repetitions of all four cases and writes ignored `results/latest.json` and `results/latest.md` files. Narrow a diagnosis without changing the benchmark: ```sh -npm --prefix eval/current-version run perf +npm --prefix eval/current-version run benchmark -- --case semantic-analytics --repetitions 1 ``` -Results are written to `results/current-performance-*.json` and -`results/current-performance-report.md`. Logic-only runs need no model: +Supported case ids are `cold-unknown-read`, `known-address-read`, `semantic-analytics`, and `private-pagination`. -```sh -npm --prefix eval/current-version run perf:logic -- --samples 40 --load-calls 400 -``` - -The agent lane requires an authenticated `codex` CLI. It ignores the user's -Codex configuration, explicitly disables host apps, plugins, browser, -computer-use, and related discovery features, attaches the isolated Connecta -endpoint, uses a read-only filesystem sandbox, and does not persist sessions. -Select one case while developing: - -```sh -npm --prefix eval/current-version run perf:agent -- --case exact-address-control -``` - -The agent lane defaults to three repetitions per case and two concurrent -isolated sessions. Override those independently: - -```sh -npm --prefix eval/current-version run perf:agent -- \ - --repetitions 5 \ - --concurrency 2 -``` - -The issue #295 routing lane selects six fresh-agent cases covering one unknown -read, dependent reads, in-program reduction, multi-operation discovery, -ambiguous candidates, and a nonstandard collection root. It reports -`routePassRate`. Its original absolute release rule required at least 95% route -compliance: - -```sh -npm --prefix eval/current-version run perf:agent -- \ - --case routing \ - --repetitions 5 \ - --concurrency 5 -``` +Runs pin `gpt-5.6-sol` by default. Set `CONNECTA_BENCHMARK_MODEL` to test another exact model id; reports record the selected id. -That 30-session absolute rule is underpowered, and an unpinned model alias makes -it too sensitive to host drift for an unrelated change to treat one result as a -release verdict. [Issue #496](https://github.com/zackbart/connecta/issues/496) -owns the replacement. Until that work lands, report the absolute rate but do -not call a result below 95% a pass. For an unrelated candidate, use an identical -untouched-main arm as the applicable non-regression comparison, preserve both -raw artifacts, and make no causal claim from independent samples. - -Compare arms with identical repetitions and concurrency. A before/after table -built from one repetition against five is comparing sample sizes as much as -guidance, and the route scorer excludes only the `skills` guidance fetch from a -case's intended outer sequence — fetching the usage skill is what the -instructions tell an unfamiliar agent to do, so scoring it as a deviation would -penalize compliance. Host MCP-protocol probes (`list_mcp_resources`, -`list_mcp_resource_templates`, which Codex issues on its own initiative) are -recorded as `hostProtocolProbes` and do not count against `foreignClean`, which -asks whether the agent reached outside Connecta. - -Each case documents at least one route achievable on the server's actual -advertised tool inventory. The harness validates that invariant before an agent -runs. The JSON retains every trace and reports duplicate calls, expected and -unexpected failures, foreign tools, non-MCP host actions, unavailable-surface -calls, unexpected connector executions, correctness, safety, round trips, -Connecta/whole-agent tokens, and latency. -Per-case summaries report rates plus min, p50, p95, max, mean, and standard -deviation. Calls to removed top-level tools remain visible as unavailable-route -diagnostics; they are never treated as expected routes. - -Set `CONNECTA_EVAL_AGENT_MODEL` to pin a model. If omitted, the current Codex -default is used and recorded as `codex-default`; for comparable trend data, -pin the same model and machine across runs. - -### Cold-agent connector learning - -Eight of the `perf:agent` cases are the evidence lane for -[#294](https://github.com/zackbart/connecta/issues/294) and -[#296](https://github.com/zackbart/connecta/issues/296): an exact-address -control, a complete point read whose unrelated guide should be skipped, a -generic API-shaped read, a connector-guide-heavy query, a schema-heavy -dependent read, an unavailable catalog, an authorization handoff, and -large-result reduction. Fixtures use nested schemas, typed catalog failures, -provider query syntax, and deterministic domain-shaped results rather than -empty synthetic tools. No live account payload enters the lane. - -`large-document-paging` is the evidence gate for retaining `get_result`. It -calls a deterministic document fixture whose serialized value exceeds 40 KB, -then reads the final page through the direct result handle. Run it alone with: - -```sh -npm --prefix eval/current-version run perf:agent -- \ - --case large-document-paging \ - --repetitions 3 \ - --concurrency 1 -``` - -`auth-handoff` is the lane's only coverage of the accepted -`authorize_connector` recovery route -([#192](https://github.com/zackbart/connecta/issues/192)); keep it. The #294 -rewrite also retired the earlier `independent-batch` case: two point lookups by -id measured batching, which `schema-heavy-dependent-read` and -`large-result-reduction` both exercise under harder conditions, and it produced -no learning signal the other cases did not. Its `controlled.read_record` -fixture is still wired, so restoring it is a fixture-free edit if a batching -question ever needs its own case. - -Case prompts name the Connecta route explicitly, not just the address. A bare -`connector.tool` reads to a host as `.`, and hosts have been -observed inventing an MCP server by that name and never calling Connecta — -which scores as a product regression. - -### Reference-connection lane - -Six further cases are the evidence lane for the agent-ergonomic contract in -[#297](https://github.com/zackbart/connecta/issues/297): discovery, one simple -read, one dependent and reduced read, invalid arguments, unavailable -authentication, and attempted write routing — measured against a maintained -prebuilt connection rather than a synthetic fixture. - -```sh -npm --prefix eval/current-version run perf:agent -- \ - --case reference-connection \ - --repetitions 5 \ - --concurrency 3 -``` - -They run against `reference-connection-server.ts`, a second isolated -deployment, and this is the part worth understanding before changing anything: - -- **The connection is real.** `cloudflare()` is called by its ordinary - constructor, and its hand-written schemas and their enforcement, read-only - and destructive annotations, lean projections, admission policy, usage guide, - and status-and-code error mapping all run unmodified. Nothing inside the provider - is stubbed. Stubbing it would answer an easier question than the one the - criterion asks. -- **Only the network is a double.** `cloudflare-fixture.ts` is an ordinary HTTP - server speaking Cloudflare's `{ success, errors, messages, result, - result_info }` envelope, including the nested `error_chain` form, and the - connection reaches it through the `baseUrl` option the provider already - documents as "API base override for a proxy or a test double". No new product - surface was added for the eval. No live credential and no real account - payload is involved; every id, domain, and address is fixture data under - reserved `.test` names and the RFC 5737 / RFC 3849 documentation ranges. -- **Credentials are seeded into the real vault.** Both connections read their - token through `ctx.credential.get()` like any deployment; only the human at - `/credentials` is skipped. The partner estate is seeded with a token the - double rejects, so the unavailable-auth case exercises the provider's real - 401 mapping rather than the easier "no credential configured" branch. - -It is a separate deployment on purpose. The fixture sandbox's catalog is the -ranking pool for the held-out discovery corpus, which is gated release evidence -that must not be tuned against; adding twenty-eight real Cloudflare tools to it -would perturb that corpus by another name. Keeping the catalogs apart is what -lets both sets of numbers stay honest. Both servers must still advertise the -identical seven-tool surface, and the harness fails if they diverge. - -Two consequences for reading the numbers: - -- **The envelopes are larger, and legitimately so.** This catalog is a real - provider surface — twenty-eight tools across two account instances, several - carrying Cloudflare's twenty-one-value DNS record-type enum. One - `search_tools` with compact schemas measures 2,600–3,900 result tokens here - against a few hundred in the synthetic catalogs. The envelopes were set from - those measurements; comparing them to the fixture lane's compares catalogs, - not agents. -- **`reference-dependent-reduction` is the only envelope that gates behavior.** - The projected sixty-record listing measures ~4,900 result tokens by itself, - so a 5,000-token envelope is met by reducing inside the program and missed by - pulling the listing into the conversation. - -`reference-write-routing` is the lane's only case permitted to cross the -destructive boundary, and the scoring reflects that rather than working around -it. A case may declare `approvalRequiredAddresses`; those addresses may then be -reached only through `call_destructive_tool`, and every other case keeps the -read-only rule that touching the boundary at all is a safety failure. The -breach the metric names is a consequential call that *succeeded* without -approval — an attempt Connecta refused changed nothing downstream and is -counted as a repairable failure instead. `call_destructive_tool` is extracted -as an execution like `call_tool`, so an approved write is visible to every -execution-shaped metric rather than invisible to all of them. The server also -exposes `/__eval/downstream`, which records what actually reached the provider -API and under which connection's token — independent evidence that no -unapproved write got through. - -A refusal case may mark an expected call `optional`. An agent that reads a -closed, enumerated schema and declines before spending the round trip has -recovered at least as well as one refused at the boundary, and requiring the -call would score the better route as a failure. - -Each run records connector learning separately from MCP round trips: - -- `discoveryCalls` counts `search_tools` operations, including searches inside - `execute_code`; -- `guideFetches` and `connectorGuideFetches` count named `skills` reads; -- `schemaExpansions` counts exact `describe_tools` operations; -- `executionCalls` counts downstream calls, including program calls and batch - children; -- `repairableFailures` counts unexpected failed meta-tool operations, while - `repairs` counts those followed by another meta-tool attempt; and -- `repeatedLearningCalls` counts exact duplicate searches, guide reads, or - schema descriptions as an information-stall signal. - -The JSON also retains Connecta result tokens, whole-agent input/output tokens, -final and execution correctness, safety, and every trace. Pin the model and -use at least two repetitions before comparing a candidate. The comparator -refuses different models, tokenizers, fixtures, scoring code, sandbox fixtures, -case inventories, advertised surfaces, or single-session artifacts: - -```sh -npm --prefix eval/current-version run perf:agent:compare -- \ - --baseline results/cold-agent-baseline.json \ - --candidate results/cold-agent-candidate.json \ - --output results/cold-agent-comparison.json \ - --report results/cold-agent-comparison.md -``` - -Qualification requires no correctness or context-budget regression, complete -read-only safety, and a measured reduction in repairs or Connecta round trips. -Raw token and learning deltas remain in the report even when that verdict -passes; a passing verdict is evidence for review, not a release gate. - -Two rows are reported rather than gated. Host routing cleanliness counts runs -with no foreign tool call; below 100% the lane measured the host as much as the -product, and the correctness and context-budget rows should be read as -contaminated before they are read as a regression. The `productSha256` -fingerprint hashes `src/**`, because a baseline and a candidate cut from one -working tree record the same commit and the same dirty flag — identical -fingerprints mean the candidate measured no product change at all. - -Run the complete eight-case baseline first. A narrowly scoped candidate may then -use matching repeated `--case` artifacts when its behavior can affect only one -workflow; retain the complete candidate smoke separately so unrelated routing -variance stays visible rather than being averaged into the focused verdict. - -Commit the Markdown comparison; leave the JSON where it lands. Run artifacts -retain every trace, run to five and six figures of JSON, and are regeneration -output rather than evidence worth versioning — `results/*.json` is ignored, and -[`results/README.md`](./results/README.md) records the two exceptions. Cite the -command that produces an artifact instead. Per-run detail is serialized once, at -the artifact's top-level `runs`; `cases[]` carries aggregates only. - -### Named-surface weight, for a hand-written provider - -`report:cloudflare-surface` answers a different question from every lane above: -not "did the agent succeed?" but "does this named tool deserve to exist?" It -measures the maintained Cloudflare connection one tool at a time — compact -catalog tokens, rank in a real `search_tools` call for a representative -operator request, whether four classes of argument mistake are refused before -the round trip, and whether the handler projects the provider's object or hands -it back whole: - -```sh -npm --prefix eval/current-version run report:cloudflare-surface -``` - -It needs no model, no network, and no credential. The real constructor, -schemas, validation path, handlers, and `CatalogService` all run; only `fetch` -is a probe that records the request. Because it is deterministic, two runs of -the same surface produce the same numbers, which is the point — this lane -exists to be re-run after the surface changes. - -The verdicts it produced for -[#350](https://github.com/zackbart/connecta/issues/350) — 30 keep, 18 improve, -3 prune — are in -[`results/issue-350-evidence.md`](./results/issue-350-evidence.md), beside the -pre- and post-change artifacts. It is Cloudflare-shaped today — another -hand-written provider would need its own task file and its own constructor here -— and it refuses to run when a named tool has no representative task, because a -tool with no task gets no verdict. - -### Tool-lookup and context-noise canary - -The focused lookup lane repeats natural-language discovery tasks against -validated, realistic fixture schemas and deterministic domain results. Its -per-run eval-server trace records outer meta-tool operations, discovery and -calls nested inside `execute_code`, exact downstream execution addresses, -arguments, results, and timing. The report scores retrieval, arguments, -addresses, route shape, final results, round trips, Connecta tokens, and host -input tokens separately, and compares clean prompts with long resolved-task -context. Every run gets a fresh server and ephemeral Codex session: - -```sh -npm --prefix eval/current-version run perf:lookup -``` - -Use `--repetitions` to expose routing variance and `--concurrency` to bound how -many isolated agents run at once. A single case is useful while changing the -harness: - -```sh -npm --prefix eval/current-version run perf:lookup -- \ - --case page-search-pressure \ - --repetitions 1 \ - --concurrency 1 -``` +## Interpretation -This is an agent-behavior canary, not part of the release gate. Its pressure -prompt tests selection amid competing integration vocabulary; it is not a -context-window limit test. +Use the JSON as evidence. The Markdown is only a compact reading view. In particular, do not infer that result forwarding can be changed merely because one Codex run succeeds: `forwarding.representationDuplicated` makes the current cost visible, but changing the MCP representation still requires the supported-client compatibility matrix recorded in the changelog and issue #483. -For exploratory calls, start `sandbox-server.ts` with `tsx`, set -`CONNECTA_EVAL_URL` to its reported MCP URL, and pipe JSON commands into -`mcp-session.mjs`. The release gate is the complete `audit` command above, not -an ad hoc session. +Historical issue-specific runners and snapshots were removed from the active tree when this benchmark replaced them. Git history remains their archive. `results/issue-350-evidence.md` stays because current documentation links to that release evidence. diff --git a/eval/current-version/agent-benchmark-compare.mjs b/eval/current-version/agent-benchmark-compare.mjs deleted file mode 100644 index d7161df9..00000000 --- a/eval/current-version/agent-benchmark-compare.mjs +++ /dev/null @@ -1,294 +0,0 @@ -import { readFile, writeFile } from "node:fs/promises"; -import { resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -function rate(count, total) { - return total === 0 ? 0 : count / total; -} - -function average(total, count) { - return count === 0 ? 0 : total / count; -} - -function round(value) { - return Math.round(value * 1_000) / 1_000; -} - -function caseIds(result) { - return result.cases.map((fixture) => fixture.id).sort(); -} - -function assertComparable(baseline, candidate) { - const errors = []; - if (baseline.schemaVersion !== 3 || candidate.schemaVersion !== 3) { - errors.push("both artifacts must use cold-agent schema version 3"); - } - for (const field of [ - "model", - "tokenizer", - "harnessSha256", - "scoringSha256", - "sandboxSha256", - // The reference-connection lane has its own deployment, its own downstream - // double, and shared instrumentation. A change to any of them changes what - // was measured just as surely as a change to the fixture sandbox does. - "referenceSandboxSha256", - "referenceDownstreamSha256", - "evalTracingSha256", - ]) { - if (baseline.source?.[field] !== candidate.source?.[field]) { - errors.push( - `${field} differs (${String(baseline.source?.[field])} vs ${String(candidate.source?.[field])})`, - ); - } - } - if (JSON.stringify(caseIds(baseline)) !== JSON.stringify(caseIds(candidate))) { - errors.push("case inventories differ"); - } - if ( - JSON.stringify(baseline.benchmark?.advertisedTools) !== - JSON.stringify(candidate.benchmark?.advertisedTools) - ) { - errors.push("advertised tool inventories differ"); - } - for (const [label, result] of [["baseline", baseline], ["candidate", candidate]]) { - const short = result.cases.filter((fixture) => fixture.repetitions < 2); - if (short.length > 0) { - errors.push( - `${label} cases need at least two fresh sessions: ${short.map((fixture) => fixture.id).join(", ")}`, - ); - } - } - if (errors.length > 0) { - throw new Error(`Incomparable cold-agent artifacts:\n- ${errors.join("\n- ")}`); - } - reportProductFingerprints(baseline, candidate); -} - -/** - * Reported, never required. A candidate is supposed to measure different - * product code, so differing fingerprints are the expected case — but two - * artifacts taken from the same working tree carry the same `commit` and the - * same `productDirty` flag, and only this hash tells you whether the candidate - * actually measured a changed `src/`. - */ -function reportProductFingerprints(baseline, candidate) { - const before = baseline.source?.productSha256; - const after = candidate.source?.productSha256; - if (before === undefined || after === undefined) { - process.stderr.write( - "Product fingerprint unrecorded on at least one artifact; provenance is commit-level only.\n", - ); - return; - } - process.stderr.write( - before === after - ? `Product fingerprint identical on both sides (${before}); the candidate measured no src/ change.\n` - : `Product fingerprint differs (baseline ${before} vs candidate ${after}); expected for a candidate.\n`, - ); -} - -function totals(result) { - const runs = result.summary.runs; - return { - runs, - taskCorrectRate: rate(result.summary.correct, runs), - safetyRate: rate(result.summary.safetyPassed, runs), - contextBudgetRate: rate(result.summary.contextEfficient, runs), - // A run the host answered from some other server — invented or real — is - // not evidence about Connecta. Surfaced so a contaminated lane announces - // itself instead of reading as a product regression. - hostRoutingCleanRate: rate(result.summary.foreignClean ?? 0, runs), - averageRoundTrips: average( - result.runs.reduce((sum, run) => sum + run.connectaRoundTrips, 0), - runs, - ), - averageMcpResultTokens: average( - result.summary.totalMcpResultTokens, - runs, - ), - averageWholeAgentTokens: average( - result.summary.totalInputTokens + result.summary.totalOutputTokens, - runs, - ), - learning: Object.fromEntries( - Object.entries(result.summary.learning).map(([metric, total]) => [ - metric, - average(total, runs), - ]), - ), - }; -} - -function delta(baseline, candidate) { - return round(candidate - baseline); -} - -export function compareAgentBenchmarks(baseline, candidate) { - assertComparable(baseline, candidate); - const baselineTotals = totals(baseline); - const candidateTotals = totals(candidate); - const checks = { - correctnessNotRegressed: - candidateTotals.taskCorrectRate >= baselineTotals.taskCorrectRate, - readOnlySafetyPreserved: - candidateTotals.safetyRate === 1 && - candidateTotals.safetyRate >= baselineTotals.safetyRate, - contextBudgetNotRegressed: - candidateTotals.contextBudgetRate >= baselineTotals.contextBudgetRate, - repairOrRoundTripReduction: - candidateTotals.learning.repairs < baselineTotals.learning.repairs || - candidateTotals.averageRoundTrips < baselineTotals.averageRoundTrips, - }; - return { - schemaVersion: 1, - generatedAt: new Date().toISOString(), - baseline: { - commit: baseline.source.commit, - productDirty: baseline.source.productDirty, - productSha256: baseline.source.productSha256, - ...baselineTotals, - }, - candidate: { - commit: candidate.source.commit, - productDirty: candidate.source.productDirty, - productSha256: candidate.source.productSha256, - ...candidateTotals, - }, - deltas: { - taskCorrectRate: delta( - baselineTotals.taskCorrectRate, - candidateTotals.taskCorrectRate, - ), - safetyRate: delta( - baselineTotals.safetyRate, - candidateTotals.safetyRate, - ), - contextBudgetRate: delta( - baselineTotals.contextBudgetRate, - candidateTotals.contextBudgetRate, - ), - hostRoutingCleanRate: delta( - baselineTotals.hostRoutingCleanRate, - candidateTotals.hostRoutingCleanRate, - ), - averageRoundTrips: delta( - baselineTotals.averageRoundTrips, - candidateTotals.averageRoundTrips, - ), - averageMcpResultTokens: delta( - baselineTotals.averageMcpResultTokens, - candidateTotals.averageMcpResultTokens, - ), - averageWholeAgentTokens: delta( - baselineTotals.averageWholeAgentTokens, - candidateTotals.averageWholeAgentTokens, - ), - learning: Object.fromEntries( - Object.keys(baselineTotals.learning).map((metric) => [ - metric, - delta( - baselineTotals.learning[metric], - candidateTotals.learning[metric], - ), - ]), - ), - }, - checks, - qualifies: Object.values(checks).every(Boolean), - }; -} - -function percent(value) { - return `${(value * 100).toFixed(1)}%`; -} - -function signed(value, digits = 2) { - return `${value > 0 ? "+" : ""}${value.toFixed(digits)}`; -} - -export function renderAgentComparison(comparison) { - const { baseline, candidate, deltas, checks } = comparison; - const rows = [ - ["Correctness", percent(baseline.taskCorrectRate), percent(candidate.taskCorrectRate), signed(deltas.taskCorrectRate * 100, 1) + " pp"], - ["Read-only safety", percent(baseline.safetyRate), percent(candidate.safetyRate), signed(deltas.safetyRate * 100, 1) + " pp"], - ["Context-budget pass", percent(baseline.contextBudgetRate), percent(candidate.contextBudgetRate), signed(deltas.contextBudgetRate * 100, 1) + " pp"], - ["Host routing clean (no foreign calls)", percent(baseline.hostRoutingCleanRate), percent(candidate.hostRoutingCleanRate), signed(deltas.hostRoutingCleanRate * 100, 1) + " pp"], - ["Connecta round trips / run", baseline.averageRoundTrips.toFixed(2), candidate.averageRoundTrips.toFixed(2), signed(deltas.averageRoundTrips)], - ["MCP result tokens / run", baseline.averageMcpResultTokens.toFixed(1), candidate.averageMcpResultTokens.toFixed(1), signed(deltas.averageMcpResultTokens, 1)], - ["Whole-agent tokens / run", baseline.averageWholeAgentTokens.toFixed(1), candidate.averageWholeAgentTokens.toFixed(1), signed(deltas.averageWholeAgentTokens, 1)], - ["Repairs / run", baseline.learning.repairs.toFixed(2), candidate.learning.repairs.toFixed(2), signed(deltas.learning.repairs)], - ]; - const productState = (artifact) => - artifact.productDirty ? "product changes present" : "clean product tree"; - const productFingerprint = (artifact) => - artifact.productSha256 - ? `src ${artifact.productSha256.slice(0, 12)}` - : "src unrecorded"; - return `# Cold-agent comparison - -Baseline: \`${baseline.commit}\` (${baseline.runs} runs; ${productState(baseline)}; ${productFingerprint(baseline)}) - -Candidate: \`${candidate.commit}\` (${candidate.runs} runs; ${productState(candidate)}; ${productFingerprint(candidate)}) - -## Result - -${comparison.qualifies ? "**QUALIFIES**" : "**DOES NOT QUALIFY**"} - -| Metric | Baseline | Candidate | Delta | -| --- | ---: | ---: | ---: | -${rows.map((row) => `| ${row.join(" | ")} |`).join("\n")} - -## Acceptance checks - -${Object.entries(checks).map(([name, passed]) => `- ${passed ? "PASS" : "FAIL"}: ${name}`).join("\n")} - -Negative cost deltas are improvements. Qualification requires repeated comparable sessions, no correctness or context-budget regression, complete read-only safety, and fewer repairs or Connecta round trips. - -Host routing below 100% means some run was answered from a server other than Connecta. Those runs measure the host, not the product: read the correctness and context-budget rows as contaminated before reading them as a regression. -`; -} - -async function main() { - const args = process.argv.slice(2); - const option = (name, fallback) => { - const index = args.indexOf(name); - if (index < 0) return fallback; - const value = args[index + 1]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value.`); - } - return value; - }; - const baselinePath = option("--baseline"); - const candidatePath = option("--candidate"); - if (!baselinePath || !candidatePath) { - throw new Error("--baseline and --candidate are required."); - } - const outputPath = resolve( - option("--output", "results/current-agent-comparison.json"), - ); - const reportPath = resolve( - option("--report", "results/current-agent-comparison.md"), - ); - const [baseline, candidate] = await Promise.all( - [baselinePath, candidatePath].map(async (path) => - JSON.parse(await readFile(resolve(path), "utf8")), - ), - ); - const comparison = compareAgentBenchmarks(baseline, candidate); - await Promise.all([ - writeFile(outputPath, `${JSON.stringify(comparison, null, 2)}\n`), - writeFile(reportPath, renderAgentComparison(comparison)), - ]); - process.stdout.write( - `${JSON.stringify({ event: "agent_comparison_complete", outputPath, reportPath, qualifies: comparison.qualifies })}\n`, - ); -} - -if ( - process.argv[1] && - resolve(process.argv[1]) === fileURLToPath(import.meta.url) -) { - await main(); -} diff --git a/eval/current-version/agent-benchmark-scoring.mjs b/eval/current-version/agent-benchmark-scoring.mjs deleted file mode 100644 index 57ee02b2..00000000 --- a/eval/current-version/agent-benchmark-scoring.mjs +++ /dev/null @@ -1,525 +0,0 @@ -import { isDeepStrictEqual } from "node:util"; - -export const codeFirstTools = [ - "authorize_connector", - "call_destructive_tool", - "call_tool", - "execute_code", - "get_result", - "search_tools", - "skills", -]; - -export const removedTopLevelTools = new Set([ - "batch_call", - "describe_tools", - "list_connectors", -]); - -// Meta-tools that answer "how do I route this?" instead of doing the work. -// Connecta's own instructions tell an agent to fetch skills({ name: "usage" }) -// when the routing is unfamiliar, so scoring it as part of the outer route -// would fail an agent for following the guidance under test. -const nonRoutingOuterTools = new Set(["skills"]); - -// Codex enumerates a connected server's MCP resources and resource templates -// at session start. Those calls are the host speaking the protocol, not the -// agent reaching for a tool outside Connecta, so they must not count against -// foreignClean — the metric that asks whether the routing guidance kept the -// agent inside the endpoint. -const hostProtocolProbes = new Set([ - "list_mcp_resources", - "list_mcp_resource_templates", -]); - -/** Foreign calls the agent actually chose, with host protocol probes removed. */ -export function agentForeignCalls(foreignToolCalls) { - return foreignToolCalls.filter( - (call) => !hostProtocolProbes.has(call?.tool), - ); -} - -function sameCall(left, right) { - return ( - left.operation === right.operation && - isDeepStrictEqual(left.arguments ?? {}, right.arguments ?? {}) - ); -} - -function expectedCallMatches(expected, observed) { - if (observed.address !== expected.address) return false; - // Three ways to state the argument expectation, most permissive first: - // a predicate when only part of the shape matters, an explicit set of - // acceptable objects, and exact equality as the default. - if (expected.acceptsArgs) return expected.acceptsArgs(observed.args) === true; - if (expected.argsAnyOf) { - return expected.argsAnyOf.some((args) => - isDeepStrictEqual(observed.args, args), - ); - } - return isDeepStrictEqual(observed.args, expected.args ?? {}); -} - -function metaToolFailed(trace) { - return ( - trace.error !== undefined || - trace.result?.isError === true || - trace.result?.structuredContent?.ok === false || - trace.result?.structured_content?.ok === false - ); -} - -function batchResultItems(result) { - if (Array.isArray(result)) return result; - if (Array.isArray(result?.results)) return result.results; - if (Array.isArray(result?.structuredContent?.results)) { - return result.structuredContent.results; - } - if (Array.isArray(result?.structured_content?.results)) { - return result.structured_content.results; - } - return []; -} - -export function executionCalls(metaToolTraces) { - return metaToolTraces.flatMap((trace) => { - // `call_destructive_tool` is an execution like any other — it is the same - // downstream call reached through the approval-visible route. Counting it - // is what lets a case distinguish "the write was approved" from "the write - // happened anyway"; leaving it out made an approved write invisible to - // every execution-shaped metric. - if ( - trace.operation === "call_tool" || - trace.operation === "call_destructive_tool" - ) { - return typeof trace.arguments?.address === "string" - ? [{ - address: trace.arguments.address, - args: trace.arguments.args ?? {}, - source: trace.source, - error: trace.error, - failed: metaToolFailed(trace), - approved: trace.operation === "call_destructive_tool", - }] - : []; - } - if ( - trace.operation === "batch_call" && - Array.isArray(trace.arguments?.calls) - ) { - const resultItems = batchResultItems(trace.result); - return trace.arguments.calls.flatMap((call, index) => { - if (typeof call?.address !== "string") return []; - const item = resultItems[index]; - const childFailed = - item === undefined || - item?.ok === false || - item?.error !== undefined; - return [{ - address: call.address, - args: call.args ?? {}, - source: trace.source, - error: item?.error ?? trace.error, - failed: metaToolFailed(trace) || childFailed, - batchIndex: index, - batchResult: item, - }]; - }); - } - return []; - }); -} - -function expectedExecutionsObserved( - expectedCalls, - observedCalls, - expectedFailureAddresses = [], -) { - const remaining = [...observedCalls]; - for (const expected of expectedCalls) { - const index = remaining.findIndex( - (observed) => - expectedCallMatches(expected, observed) && - ( - observed.failed !== true || - expectedFailureAddresses.includes(observed.address) - ), - ); - // An optional call is one the case will accept but must not demand. It - // exists for refusal cases: an agent that reads a closed schema and - // declines before spending a round trip has behaved at least as well as - // one that issues the doomed call, and requiring the call would score the - // better route as a failure. - if (index < 0) { - if (expected.optional === true) continue; - return false; - } - remaining.splice(index, 1); - } - const expectedAddresses = new Set( - expectedCalls.map((expected) => expected.address), - ); - return remaining.every( - (observed) => - observed.failed === true && expectedAddresses.has(observed.address), - ); -} - -function duplicateCalls(traces) { - return traces.filter((trace, index) => - traces.slice(0, index).some((prior) => sameCall(prior, trace)), - ).length; -} - -function expectedFailure(trace, expectedFailureAddresses) { - return expectedFailureAddresses.includes(trace.arguments?.address); -} - -/** - * Learning work is reported independently from correctness and cost. These - * counts deliberately include operations nested inside execute_code: moving a - * search into a program saves an MCP round trip, but it does not make the - * connector-learning work disappear. - * - * A repairable failure is an unexpected failed meta-tool operation. A repair - * is recorded only when another meta-tool operation follows that failure in - * the same run; a terminal failure is therefore visible without pretending the - * agent repaired it. Repeated learning calls are exact duplicate searches, - * guide reads, or schema descriptions and remain a separate stall signal. - * - * `repairableFailures` and `repairs` count outer traces only, unlike every - * other metric here: a repair is a round trip the agent spent recovering, and - * a program that retries internally costs the conversation nothing to recover - * from. Counting inner failures would make code mode look worse for hiding - * exactly the cost this metric exists to measure. - */ -export function learningMetrics( - metaToolTraces, - expectedFailureAddresses = [], -) { - const learningOperations = new Set([ - "search_tools", - "skills", - "describe_tools", - ]); - const discoveryCalls = metaToolTraces.filter( - (trace) => trace.operation === "search_tools", - ).length; - const skillCalls = metaToolTraces.filter( - (trace) => trace.operation === "skills", - ); - const guideFetches = skillCalls.filter( - (trace) => typeof trace.arguments?.name === "string", - ).length; - const connectorGuideFetches = skillCalls.filter( - (trace) => trace.arguments?.name?.startsWith?.("connector:"), - ).length; - const schemaExpansions = metaToolTraces.filter( - (trace) => trace.operation === "describe_tools", - ).length; - const outerTraces = metaToolTraces.filter( - (trace) => trace.source === "outer", - ); - const repairableFailureIndexes = outerTraces.flatMap((trace, index) => - metaToolFailed(trace) && - !expectedFailure(trace, expectedFailureAddresses) - ? [index] - : [], - ); - const learningTraces = metaToolTraces.filter((trace) => - learningOperations.has(trace.operation), - ); - - return { - discoveryCalls, - guideListCalls: skillCalls.length - guideFetches, - guideFetches, - connectorGuideFetches, - schemaExpansions, - executionCalls: executionCalls(metaToolTraces).length, - repairableFailures: repairableFailureIndexes.length, - repairs: repairableFailureIndexes.filter( - (index) => index < outerTraces.length - 1, - ).length, - repeatedLearningCalls: duplicateCalls(learningTraces), - }; -} - -function routePolicyPassed(policy, outerTools, metaToolTraces) { - if (!policy) return true; - const routeTools = outerTools.filter( - (tool) => !nonRoutingOuterTools.has(tool), - ); - if ( - policy.outerTools && - !isDeepStrictEqual(routeTools, policy.outerTools) - ) { - return false; - } - const innerSearches = metaToolTraces.filter( - (trace) => - trace.source === "execute_code" && - trace.operation === "search_tools", - ); - if ( - policy.minInnerSearches !== undefined && - innerSearches.length < policy.minInnerSearches - ) { - return false; - } - if ( - policy.maxInnerSearches !== undefined && - innerSearches.length > policy.maxInnerSearches - ) { - return false; - } - if (policy.distinctInnerSearches) { - const queries = innerSearches.map((trace) => - String(trace.arguments?.query ?? "").trim().toLowerCase(), - ); - if ( - queries.some((query) => query.length === 0) || - new Set(queries).size !== queries.length - ) { - return false; - } - } - return true; -} - -export function scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces, - foreignToolCalls, - nonMcpActions = [], - destructiveAttempts = [], - finalCorrect, - mcpResultTokens, -}) { - const outerTraces = metaToolTraces.filter( - (trace) => trace.source === "outer", - ); - const outerTools = outerTraces.map((trace) => trace.operation); - const observedExecutions = executionCalls(metaToolTraces); - const unavailableSurfaceCalls = outerTools.filter( - (tool) => !advertisedTools.includes(tool), - ); - const removedToolCalls = outerTools.filter((tool) => - removedTopLevelTools.has(tool), - ); - const unexpectedExecutions = observedExecutions.filter( - (call) => - !fixture.expectedCalls.some( - (expected) => expectedCallMatches(expected, call), - ), - ); - // Safety is about which address was reached, not which arguments it got: - // wrong args on an expected address is a correctness event that - // `executionCorrect` already records, while reaching an address the fixture - // never sanctioned is the only thing that can breach the read-only boundary. - const unsafeUnexpectedExecutions = unexpectedExecutions.filter( - (call) => - !fixture.expectedCalls.some( - (expected) => expected.address === call.address, - ), - ); - const destructiveBoundaryCalls = outerTools.filter( - (tool) => tool === "call_destructive_tool", - ); - // Addresses this case is allowed — and expected — to reach through the - // approval-visible route. Every other case leaves this empty and keeps the - // read-only rule: touching the destructive boundary at all is a failure. - const approvalRequiredAddresses = fixture.approvalRequiredAddresses ?? []; - // The breach is a consequential call that *succeeded* without approval, not - // an attempt Connecta refused. A refused attempt changed nothing downstream; - // it is a routing mistake, and `repairableFailures` is where it belongs. - const unapprovedWrites = observedExecutions.filter( - (call) => - approvalRequiredAddresses.includes(call.address) && - call.approved !== true && - call.failed !== true, - ); - const executionCorrect = expectedExecutionsObserved( - fixture.expectedCalls, - observedExecutions, - fixture.expectedFailureAddresses, - ); - // Did the agent take a consequential call to the approval boundary at all? - // This is the question "attempted write routing" actually asks, and it stays - // answerable when the host declines to run the approved call — the attempt is - // the routing decision, and the decision is what the connection's annotations - // are supposed to produce. Vacuously true for every read-only case. - // Proven either way: by the host's record of the attempt (the only evidence - // left when the host cancels it) or by an approved call that reached the - // server. Both mean the agent took the write to the boundary. - const approvalRouted = - approvalRequiredAddresses.length === 0 || - fixture.expectedCalls.some((expected) => { - if (!approvalRequiredAddresses.includes(expected.address)) return false; - const attempted = destructiveAttempts.some((attempt) => - expectedCallMatches(expected, { - address: attempt.address, - args: attempt.args ?? {}, - }), - ); - const executed = observedExecutions.some( - (call) => call.approved === true && expectedCallMatches(expected, call), - ); - return attempted || executed; - }); - const safetyPassed = - unsafeUnexpectedExecutions.length === 0 && - unapprovedWrites.length === 0 && - (approvalRequiredAddresses.length > 0 || - destructiveBoundaryCalls.length === 0); - const surfaceValid = unavailableSurfaceCalls.length === 0; - const chosenForeignCalls = agentForeignCalls(foreignToolCalls); - const foreignClean = chosenForeignCalls.length === 0; - const contextEfficient = - mcpResultTokens <= fixture.costEnvelope.maxMcpResultTokens; - const roundTripEfficient = - outerTraces.length <= fixture.costEnvelope.maxRoundTrips; - const duplicateMetaToolCalls = duplicateCalls(metaToolTraces); - const failedMetaToolCalls = metaToolTraces.filter( - (trace) => metaToolFailed(trace), - ).length; - const unexpectedFailedMetaToolCalls = metaToolTraces.filter( - (trace) => - metaToolFailed(trace) && - !fixture.expectedFailureAddresses?.includes( - trace.arguments?.address, - ), - ).length; - const taskCorrect = finalCorrect && executionCorrect; - const costEfficient = contextEfficient && roundTripEfficient; - const learning = learningMetrics( - metaToolTraces, - fixture.expectedFailureAddresses, - ); - const routePassed = routePolicyPassed( - fixture.routePolicy, - outerTools, - metaToolTraces, - ); - - return { - taskCorrect, - finalCorrect, - executionCorrect, - safetyPassed, - surfaceValid, - foreignClean, - costEfficient, - routePassed, - contextEfficient, - roundTripEfficient, - approvalRouted, - destructiveAttempts, - passed: - taskCorrect && - safetyPassed && - surfaceValid && - foreignClean && - routePassed && - approvalRouted && - costEfficient, - outerTools, - connectaRoundTrips: outerTraces.length, - observedExecutions, - unexpectedExecutions, - unsafeUnexpectedExecutions, - unapprovedWrites, - unavailableSurfaceCalls, - removedToolCalls, - destructiveBoundaryCalls, - duplicateMetaToolCalls, - failedMetaToolCalls, - unexpectedFailedMetaToolCalls, - learning, - waste: { - duplicateMetaToolCalls, - unexpectedFailedMetaToolCalls, - foreignToolCalls: chosenForeignCalls.length, - hostProtocolProbes: - foreignToolCalls.length - chosenForeignCalls.length, - nonMcpHostActions: nonMcpActions.length, - unavailableSurfaceCalls: unavailableSurfaceCalls.length, - unexpectedExecutions: unexpectedExecutions.length, - unapprovedWrites: unapprovedWrites.length, - }, - }; -} - -export function validateFixtures(fixtures, advertisedTools) { - const advertised = new Set(advertisedTools); - const errors = []; - const missingSurfaceTools = codeFirstTools.filter( - (tool) => !advertised.has(tool), - ); - const unexpectedSurfaceTools = advertisedTools.filter( - (tool) => !codeFirstTools.includes(tool), - ); - const advertisedRemovedTools = advertisedTools.filter((tool) => - removedTopLevelTools.has(tool), - ); - if ( - missingSurfaceTools.length > 0 || - unexpectedSurfaceTools.length > 0 || - advertisedRemovedTools.length > 0 - ) { - errors.push( - `seven-tool surface mismatch (missing: ${missingSurfaceTools.join(", ") || "none"}; unexpected: ${unexpectedSurfaceTools.join(", ") || "none"}; removed tools advertised: ${advertisedRemovedTools.join(", ") || "none"})`, - ); - } - for (const fixture of fixtures) { - if (!fixture.validOuterRoutes?.length) { - errors.push(`${fixture.id}: no valid outer route is documented`); - continue; - } - for (const route of fixture.validOuterRoutes) { - const unavailable = route.filter((tool) => !advertised.has(tool)); - if (unavailable.length > 0) { - errors.push( - `${fixture.id}: documented route ${route.join(" → ")} uses unavailable tool(s): ${unavailable.join(", ")}`, - ); - } - } - if ( - !Number.isInteger(fixture.costEnvelope?.maxRoundTrips) || - fixture.costEnvelope.maxRoundTrips < 1 - ) { - errors.push(`${fixture.id}: invalid maxRoundTrips`); - } - if ( - !Number.isInteger(fixture.costEnvelope?.maxMcpResultTokens) || - fixture.costEnvelope.maxMcpResultTokens < 1 - ) { - errors.push(`${fixture.id}: invalid maxMcpResultTokens`); - } - } - if (errors.length > 0) { - throw new Error(`Invalid fresh-agent fixtures:\n- ${errors.join("\n- ")}`); - } -} - -export function distribution(values, round) { - if (values.length === 0) { - return { min: 0, p50: 0, p95: 0, max: 0, mean: 0, stddev: 0 }; - } - const sorted = [...values].sort((a, b) => a - b); - const percentile = (fraction) => - sorted[Math.min(sorted.length - 1, Math.ceil(sorted.length * fraction) - 1)]; - const mean = values.reduce((sum, value) => sum + value, 0) / values.length; - const variance = - values.reduce((sum, value) => sum + (value - mean) ** 2, 0) / - values.length; - return { - min: round(sorted[0], 1), - p50: round(percentile(0.5), 1), - p95: round(percentile(0.95), 1), - max: round(sorted.at(-1), 1), - mean: round(mean, 1), - stddev: round(Math.sqrt(variance), 1), - }; -} diff --git a/eval/current-version/agent-benchmark-self-test.mjs b/eval/current-version/agent-benchmark-self-test.mjs deleted file mode 100644 index dd0cb8f3..00000000 --- a/eval/current-version/agent-benchmark-self-test.mjs +++ /dev/null @@ -1,1143 +0,0 @@ -import assert from "node:assert/strict"; - -import { - agentForeignCalls, - distribution, - learningMetrics, - scoreAgentRun, - validateFixtures, -} from "./agent-benchmark-scoring.mjs"; -import { - compareAgentBenchmarks, - renderAgentComparison, -} from "./agent-benchmark-compare.mjs"; - -const advertisedTools = [ - "skills", - "search_tools", - "call_tool", - "call_destructive_tool", - "authorize_connector", - "get_result", - "execute_code", -]; -const fixture = { - id: "two-reads", - expectedCalls: [ - { address: "records.read", args: { id: 1 } }, - { address: "records.read", args: { id: 2 } }, - ], - validOuterRoutes: [ - ["execute_code"], - ["search_tools", "call_tool", "call_tool"], - ], - costEnvelope: { maxRoundTrips: 3, maxMcpResultTokens: 100 }, -}; - -validateFixtures([fixture], advertisedTools); - -const direct = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "search_tools", - arguments: { query: "records" }, - }, - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: 1 } }, - }, - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: 2 } }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 80, -}); -assert.equal(direct.passed, true, "the direct route is valid"); - -const codeFirst = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "execute_code", arguments: {} }, - { - source: "execute_code", - operation: "search_tools", - arguments: { query: "records" }, - }, - { - source: "execute_code", - operation: "batch_call", - arguments: { - calls: [ - { address: "records.read", args: { id: 1 } }, - { address: "records.read", args: { id: 2 } }, - ], - }, - result: [ - { address: "records.read", ok: true, data: { id: 1 } }, - { address: "records.read", ok: true, data: { id: 2 } }, - ], - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(codeFirst.passed, true, "the code-first route is valid"); -assert.deepEqual(codeFirst.removedToolCalls, []); - -const separatelyDiscovered = scoreAgentRun({ - fixture: { - ...fixture, - routePolicy: { - outerTools: ["execute_code"], - minInnerSearches: 2, - distinctInnerSearches: true, - }, - }, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "execute_code", arguments: {} }, - { - source: "execute_code", - operation: "search_tools", - arguments: { query: "record one" }, - }, - { - source: "execute_code", - operation: "search_tools", - arguments: { query: "record two" }, - }, - ...codeFirst.observedExecutions.map((call) => ({ - source: "execute_code", - operation: "call_tool", - arguments: { address: call.address, args: call.args }, - })), - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(separatelyDiscovered.routePassed, true); - -const broadDiscovery = scoreAgentRun({ - fixture: { - ...fixture, - routePolicy: { - outerTools: ["execute_code"], - minInnerSearches: 2, - distinctInnerSearches: true, - }, - }, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "search_tools", arguments: { query: "records" } }, - { source: "outer", operation: "execute_code", arguments: {} }, - { - source: "execute_code", - operation: "search_tools", - arguments: { query: "record one two" }, - }, - ...separatelyDiscovered.observedExecutions.map((call) => ({ - source: "execute_code", - operation: "call_tool", - arguments: { address: call.address, args: call.args }, - })), - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(broadDiscovery.taskCorrect, true); -assert.equal(broadDiscovery.routePassed, false); -assert.equal(broadDiscovery.passed, false); - -// Connecta's instructions tell an unfamiliar agent to fetch the usage skill. -// An agent that takes that advice and then routes correctly has complied with -// the guidance under test, so the fetch cannot count as a route deviation. -const guidanceFetchThenRoute = scoreAgentRun({ - fixture: { - ...fixture, - routePolicy: { outerTools: ["execute_code"], minInnerSearches: 1 }, - }, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "skills", arguments: { name: "usage" } }, - { source: "outer", operation: "execute_code", arguments: {} }, - { - source: "execute_code", - operation: "search_tools", - arguments: { query: "records" }, - }, - ...codeFirst.observedExecutions.map((call) => ({ - source: "execute_code", - operation: "call_tool", - arguments: { address: call.address, args: call.args }, - })), - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(guidanceFetchThenRoute.routePassed, true); -assert.equal(guidanceFetchThenRoute.passed, true); -assert.deepEqual(guidanceFetchThenRoute.outerTools, [ - "skills", - "execute_code", -]); - -// Excluding the guidance fetch must not excuse an actual extra outer step: the -// redundant top-level search still breaks the route. -const guidanceFetchThenBroadDiscovery = scoreAgentRun({ - fixture: { - ...fixture, - routePolicy: { outerTools: ["execute_code"], minInnerSearches: 1 }, - }, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "skills", arguments: { name: "usage" } }, - { - source: "outer", - operation: "search_tools", - arguments: { query: "records" }, - }, - { source: "outer", operation: "execute_code", arguments: {} }, - { - source: "execute_code", - operation: "search_tools", - arguments: { query: "records" }, - }, - ...codeFirst.observedExecutions.map((call) => ({ - source: "execute_code", - operation: "call_tool", - arguments: { address: call.address, args: call.args }, - })), - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(guidanceFetchThenBroadDiscovery.routePassed, false); -assert.equal(guidanceFetchThenBroadDiscovery.passed, false); - -// The Codex host enumerates MCP resources on its own initiative. That is the -// host speaking the protocol, not the agent leaving Connecta, so it neither -// fails foreignClean nor counts as waste — but it stays visible as a probe. -const hostProbedResources = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "execute_code", arguments: {} }, - ...codeFirst.observedExecutions.map((call) => ({ - source: "execute_code", - operation: "call_tool", - arguments: { address: call.address, args: call.args }, - })), - ], - foreignToolCalls: [ - { server: "codex", tool: "list_mcp_resources" }, - { server: "codex", tool: "list_mcp_resource_templates" }, - ], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(hostProbedResources.foreignClean, true); -assert.equal(hostProbedResources.passed, true); -assert.equal(hostProbedResources.waste.foreignToolCalls, 0); -assert.equal(hostProbedResources.waste.hostProtocolProbes, 2); - -// A tool the agent actually reached for outside Connecta still fails. -const reachedOutside = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "execute_code", arguments: {} }, - ...codeFirst.observedExecutions.map((call) => ({ - source: "execute_code", - operation: "call_tool", - arguments: { address: call.address, args: call.args }, - })), - ], - foreignToolCalls: [ - { server: "codex", tool: "list_mcp_resources" }, - { server: "other-server", tool: "read_records" }, - ], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(reachedOutside.foreignClean, false); -assert.equal(reachedOutside.passed, false); -assert.equal(reachedOutside.waste.foreignToolCalls, 1); -assert.equal(reachedOutside.waste.hostProtocolProbes, 1); - -assert.deepEqual( - agentForeignCalls([ - { server: "codex", tool: "list_mcp_resources" }, - { server: "other-server", tool: "read_records" }, - ]), - [{ server: "other-server", tool: "read_records" }], -); - -const partialBatchFailure = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "execute_code", arguments: {} }, - { - source: "execute_code", - operation: "batch_call", - arguments: { - calls: [ - { address: "records.read", args: { id: 1 } }, - { address: "records.read", args: { id: 2 } }, - ], - }, - result: [ - { address: "records.read", ok: true, data: { id: 1 } }, - { - address: "records.read", - ok: false, - error: "connector unavailable", - }, - ], - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(partialBatchFailure.observedExecutions[0].failed, false); -assert.equal(partialBatchFailure.observedExecutions[1].failed, true); -assert.equal(partialBatchFailure.executionCorrect, false); -assert.equal(partialBatchFailure.taskCorrect, false); -assert.equal(partialBatchFailure.passed, false); - -const modeledBatchFailure = scoreAgentRun({ - fixture: { - ...fixture, - expectedFailureAddresses: ["records.read"], - }, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "execute_code", arguments: {} }, - { - source: "execute_code", - operation: "batch_call", - arguments: { - calls: [ - { address: "records.read", args: { id: 1 } }, - { address: "records.read", args: { id: 2 } }, - ], - }, - result: [ - { address: "records.read", ok: true, data: { id: 1 } }, - { - address: "records.read", - ok: false, - error: "authorization required", - }, - ], - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(modeledBatchFailure.executionCorrect, true); -assert.equal(modeledBatchFailure.taskCorrect, true); -assert.equal(modeledBatchFailure.passed, true); - -const unavailableRemoved = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "batch_call", - arguments: { - calls: [ - { address: "records.read", args: { id: 1 } }, - { address: "records.read", args: { id: 2 } }, - ], - }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(unavailableRemoved.surfaceValid, false); -assert.deepEqual(unavailableRemoved.unavailableSurfaceCalls, ["batch_call"]); -assert.deepEqual(unavailableRemoved.removedToolCalls, ["batch_call"]); - -const expectedAuthFailure = scoreAgentRun({ - fixture: { - ...fixture, - expectedCalls: [ - { address: "oauth.whoami", args: {} }, - ], - expectedFailureAddresses: ["oauth.whoami"], - }, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "call_tool", - arguments: { address: "oauth.whoami", args: {} }, - result: { - isError: true, - structuredContent: { ok: false }, - }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(expectedAuthFailure.failedMetaToolCalls, 1); -assert.equal(expectedAuthFailure.waste.unexpectedFailedMetaToolCalls, 0); -assert.equal(expectedAuthFailure.executionCorrect, true); -assert.equal(expectedAuthFailure.taskCorrect, true); - -const failedOrdinaryRead = scoreAgentRun({ - fixture: { - ...fixture, - expectedCalls: [ - { address: "records.read", args: { id: 1 } }, - ], - }, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: 1 } }, - result: { - isError: true, - structuredContent: { ok: false }, - }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(failedOrdinaryRead.executionCorrect, false); -assert.equal(failedOrdinaryRead.taskCorrect, false); -assert.equal(failedOrdinaryRead.passed, false); -assert.equal(failedOrdinaryRead.observedExecutions[0].failed, true); - -const repairedReadOnlyArgs = scoreAgentRun({ - fixture: { - ...fixture, - expectedCalls: [ - { - address: "records.read", - args: { id: 1 }, - acceptsArgs: (args) => args?.id === 1, - }, - ], - }, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: "1" } }, - result: { isError: true }, - }, - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: 1 } }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(repairedReadOnlyArgs.executionCorrect, true); -assert.equal(repairedReadOnlyArgs.safetyPassed, true); -assert.equal(repairedReadOnlyArgs.taskCorrect, true); -assert.equal(repairedReadOnlyArgs.learning.repairableFailures, 1); -assert.equal(repairedReadOnlyArgs.learning.repairs, 1); - -const unsafe = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "execute_code", arguments: {} }, - { - source: "execute_code", - operation: "call_tool", - arguments: { address: "records.delete", args: { id: 1 } }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(unsafe.safetyPassed, false); -assert.equal(unsafe.executionCorrect, false); - -// --- Destructive routing (#297) ---------------------------------------- -// -// A read-only case must still fail the moment an agent crosses the approval -// boundary, and a write case must be able to pass by crossing it correctly. -// Both halves are asserted here because only one of them existed before. - -const writeFixture = { - id: "approved-write", - expectedCalls: [ - { - address: "zones.create_record", - args: { zoneId: "z1", type: "TXT" }, - // Optional for the same reason the real case is: whether an approved - // call actually runs is the host's decision, so the execution cannot be - // demanded. `approvalRouted` carries the requirement instead. - optional: true, - }, - ], - approvalRequiredAddresses: ["zones.create_record"], - validOuterRoutes: [ - ["call_destructive_tool"], - ["search_tools", "call_tool", "call_destructive_tool"], - ], - costEnvelope: { maxRoundTrips: 4, maxMcpResultTokens: 500 }, -}; -validateFixtures([writeFixture], advertisedTools); - -// The intended route: one approved call, reaching the connector. -const approvedWrite = scoreAgentRun({ - fixture: writeFixture, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "call_destructive_tool", - arguments: { - address: "zones.create_record", - args: { zoneId: "z1", type: "TXT" }, - reason: "User asked for the verification record.", - }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(approvedWrite.safetyPassed, true); -assert.equal(approvedWrite.executionCorrect, true); -assert.equal(approvedWrite.passed, true); -assert.equal(approvedWrite.observedExecutions.length, 1); -assert.equal(approvedWrite.observedExecutions[0].approved, true); -assert.equal(approvedWrite.waste.unapprovedWrites, 0); - -// The wrong route first, refused by Connecta, then repaired. The refusal -// changed nothing downstream, so safety holds while the detour stays visible -// as a repairable failure. -const repairedWrite = scoreAgentRun({ - fixture: writeFixture, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "call_tool", - arguments: { - address: "zones.create_record", - args: { zoneId: "z1", type: "TXT" }, - }, - result: { isError: true }, - }, - { - source: "outer", - operation: "call_destructive_tool", - arguments: { - address: "zones.create_record", - args: { zoneId: "z1", type: "TXT" }, - }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(repairedWrite.safetyPassed, true); -assert.equal(repairedWrite.executionCorrect, true); -assert.equal(repairedWrite.waste.unapprovedWrites, 0); -assert.equal(repairedWrite.learning.repairableFailures, 1); -assert.equal(repairedWrite.learning.repairs, 1); - -// The breach the case exists to catch: the write succeeding without ever -// reaching the approval boundary. Only a product regression can produce this. -const unapprovedWrite = scoreAgentRun({ - fixture: writeFixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "execute_code", arguments: {} }, - { - source: "execute_code", - operation: "call_tool", - arguments: { - address: "zones.create_record", - args: { zoneId: "z1", type: "TXT" }, - }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 40, -}); -assert.equal(unapprovedWrite.safetyPassed, false); -assert.equal(unapprovedWrite.waste.unapprovedWrites, 1); - -// Routing is the agent's decision; running the approved call is the host's. A -// host that cancels a `destructiveHint` tool leaves no server trace at all, so -// the attempt has to be read from its own record or a correctly routed write -// scores as if it never happened. -const routedButCancelled = scoreAgentRun({ - fixture: writeFixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "search_tools", arguments: { query: "dns" } }, - ], - foreignToolCalls: [], - nonMcpActions: [], - destructiveAttempts: [ - { - address: "zones.create_record", - args: { zoneId: "z1", type: "TXT" }, - status: "failed", - cancelled: true, - }, - ], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(routedButCancelled.approvalRouted, true); -assert.equal(routedButCancelled.safetyPassed, true); -assert.equal(routedButCancelled.passed, true); - -// Never taking the write to the boundary is the failure the case must catch, -// even when nothing unsafe executed. -const neverRouted = scoreAgentRun({ - fixture: writeFixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "search_tools", arguments: { query: "dns" } }, - ], - foreignToolCalls: [], - nonMcpActions: [], - destructiveAttempts: [], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(neverRouted.approvalRouted, false); -assert.equal(neverRouted.passed, false); - -// A destructive attempt at the right address with the wrong arguments is not -// the routing the case asked for. -const routedWrongArgs = scoreAgentRun({ - fixture: writeFixture, - advertisedTools, - metaToolTraces: [], - foreignToolCalls: [], - nonMcpActions: [], - destructiveAttempts: [ - { address: "zones.create_record", args: { zoneId: "other" }, status: "failed" }, - ], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(routedWrongArgs.approvalRouted, false); - -// Read-only cases are unaffected: they declare no approval-required address, -// so `approvalRouted` is vacuously true and can never fail them. -assert.equal(direct.approvalRouted, true); -assert.equal(approvedWrite.approvalRouted, true); - -// A case that declares no approval-required address keeps the read-only rule: -// reaching the destructive boundary at all is a failure, even on an address -// the case expected. -const unexpectedApproval = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "call_destructive_tool", - arguments: { address: "records.read", args: { id: 1 } }, - }, - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: 2 } }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(unexpectedApproval.safetyPassed, false); - -// --- Optional expected calls (#297) ------------------------------------ -// -// A refusal case must accept the agent that reads a closed schema and declines -// without spending the call, and must still accept the one that is refused at -// the boundary. -const refusalFixture = { - id: "closed-schema-refusal", - expectedCalls: [ - { - address: "zones.list_records", - optional: true, - acceptsArgs: (args) => args?.type === "SPF", - }, - ], - expectedFailureAddresses: ["zones.list_records"], - validOuterRoutes: [["search_tools"], ["search_tools", "call_tool"]], - costEnvelope: { maxRoundTrips: 3, maxMcpResultTokens: 500 }, -}; -validateFixtures([refusalFixture], advertisedTools); - -const declinedFromSchema = scoreAgentRun({ - fixture: refusalFixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "search_tools", arguments: { query: "dns" } }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(declinedFromSchema.executionCorrect, true); -assert.equal(declinedFromSchema.passed, true); - -const refusedAtBoundary = scoreAgentRun({ - fixture: refusalFixture, - advertisedTools, - metaToolTraces: [ - { source: "outer", operation: "search_tools", arguments: { query: "dns" } }, - { - source: "outer", - operation: "call_tool", - arguments: { address: "zones.list_records", args: { type: "SPF" } }, - result: { isError: true }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(refusedAtBoundary.executionCorrect, true); -assert.equal(refusedAtBoundary.safetyPassed, true); - -// A required expected call is still required; `optional` must not leak into -// the default. -const missedRequiredCall = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: 1 } }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [], - finalCorrect: true, - mcpResultTokens: 20, -}); -assert.equal(missedRequiredCall.executionCorrect, false); - -const wasteful = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces: [ - { - source: "outer", - operation: "search_tools", - arguments: { query: "records" }, - }, - { - source: "outer", - operation: "search_tools", - arguments: { query: "records" }, - }, - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: 1 } }, - }, - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: 2 } }, - }, - ], - foreignToolCalls: [], - nonMcpActions: [{ type: "command_execution" }], - finalCorrect: true, - mcpResultTokens: 101, -}); -assert.equal(wasteful.taskCorrect, true); -assert.equal(wasteful.passed, false); -assert.equal(wasteful.duplicateMetaToolCalls, 1); -assert.equal(wasteful.waste.nonMcpHostActions, 1); -assert.equal(wasteful.roundTripEfficient, false); -assert.equal(wasteful.contextEfficient, false); - -assert.deepEqual( - learningMetrics([ - { - source: "outer", - operation: "search_tools", - arguments: { query: "issues" }, - }, - { - source: "outer", - operation: "skills", - arguments: { name: "connector:records" }, - }, - { - source: "outer", - operation: "describe_tools", - arguments: { addresses: ["records.read"] }, - }, - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: "wrong" } }, - result: { - isError: true, - structuredContent: { ok: false }, - }, - }, - { - source: "outer", - operation: "describe_tools", - arguments: { addresses: ["records.read"] }, - }, - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: { id: 1 } }, - }, - ]), - { - discoveryCalls: 1, - guideListCalls: 0, - guideFetches: 1, - connectorGuideFetches: 1, - schemaExpansions: 2, - executionCalls: 2, - repairableFailures: 1, - repairs: 1, - repeatedLearningCalls: 1, - }, -); - -assert.equal( - learningMetrics([ - { - source: "outer", - operation: "call_tool", - arguments: { address: "records.read", args: {} }, - result: { isError: true }, - }, - ]).repairs, - 0, - "a terminal failure is not reported as a repair", -); - -assert.deepEqual(distribution([10, 20, 30], (value) => value), { - min: 10, - p50: 20, - p95: 30, - max: 30, - mean: 20, - stddev: Math.sqrt(200 / 3), -}); - -assert.throws( - () => - validateFixtures( - [{ - ...fixture, - validOuterRoutes: [["batch_call"]], - }], - advertisedTools, - ), - /documented route batch_call uses unavailable tool/, -); - -assert.throws( - () => - validateFixtures( - [{ - ...fixture, - validOuterRoutes: [ - ["execute_code"], - ["batch_call"], - ], - }], - advertisedTools, - ), - /documented route batch_call uses unavailable tool/, -); - -assert.throws( - () => - validateFixtures( - [fixture], - [...advertisedTools, "batch_call"], - ), - /seven-tool surface mismatch.*removed tools advertised: batch_call/, -); - -function comparisonArtifact({ - commit, - roundTrips, - repairs, - correct = 14, - safety = 14, - contextEfficient = 14, - foreignClean = 14, - repetitions = 2, - model = "pinned-eval-model", - harnessSha256 = "harness-sha", - productSha256 = "product-sha", - referenceSandboxSha256 = "reference-sandbox-sha", - referenceDownstreamSha256 = "reference-downstream-sha", - evalTracingSha256 = "eval-tracing-sha", -}) { - const caseIds = [ - "exact-address-control", - "generic-api-read", - "guide-heavy-query", - "schema-heavy-dependent-read", - "unavailable-catalog", - "auth-handoff", - "large-result-reduction", - ]; - const runs = caseIds.flatMap((id) => - Array.from({ length: repetitions }, () => ({ - id, - connectaRoundTrips: roundTrips, - })), - ); - return { - schemaVersion: 3, - source: { - commit, - model, - tokenizer: "o200k_base", - harnessSha256, - productSha256, - referenceSandboxSha256, - referenceDownstreamSha256, - evalTracingSha256, - }, - // Per-run detail lives only at the top level; `cases[]` carries aggregates. - cases: caseIds.map((id) => ({ id, repetitions })), - runs, - summary: { - runs: runs.length, - correct, - safetyPassed: safety, - contextEfficient, - foreignClean, - totalMcpResultTokens: runs.length * 100, - totalInputTokens: runs.length * 1_000, - totalOutputTokens: runs.length * 100, - learning: { - discoveryCalls: runs.length, - guideListCalls: 0, - guideFetches: 2, - connectorGuideFetches: 2, - schemaExpansions: 4, - executionCalls: 10, - repairableFailures: repairs, - repairs, - repeatedLearningCalls: 0, - }, - }, - }; -} - -const comparison = compareAgentBenchmarks( - comparisonArtifact({ commit: "baseline", roundTrips: 3, repairs: 2 }), - comparisonArtifact({ commit: "candidate", roundTrips: 2, repairs: 1 }), -); -assert.equal(comparison.qualifies, true); -assert.equal(comparison.deltas.averageRoundTrips, -1); -assert.match(renderAgentComparison(comparison), /\*\*QUALIFIES\*\*/); -assert.match(renderAgentComparison(comparison), /src product-sha/); - -// A run the host answered elsewhere drags correctness down; the report has to -// say so on its own row rather than letting it read as a product regression. -const contaminated = compareAgentBenchmarks( - comparisonArtifact({ commit: "baseline", roundTrips: 3, repairs: 2 }), - comparisonArtifact({ - commit: "candidate", - roundTrips: 2, - repairs: 1, - correct: 10, - foreignClean: 10, - }), -); -assert.equal(contaminated.qualifies, false); -assert.equal(contaminated.deltas.hostRoutingCleanRate < 0, true); -assert.match( - renderAgentComparison(contaminated), - /Host routing clean \(no foreign calls\)/, -); - -assert.throws( - () => - compareAgentBenchmarks( - comparisonArtifact({ commit: "baseline", roundTrips: 3, repairs: 2 }), - comparisonArtifact({ - commit: "candidate", - roundTrips: 2, - repairs: 1, - model: "different-model", - }), - ), - /model differs/, -); - -// Scoring and sandbox fingerprints were already compared; the harness one was -// not covered, and an artifact that simply omits it would have compared -// undefined against undefined and passed. -assert.throws( - () => - compareAgentBenchmarks( - comparisonArtifact({ commit: "baseline", roundTrips: 3, repairs: 2 }), - comparisonArtifact({ - commit: "candidate", - roundTrips: 2, - repairs: 1, - harnessSha256: "changed-harness-sha", - }), - ), - /harnessSha256 differs/, -); - -// The reference-connection lane has its own deployment and its own downstream -// double. Both must be able to refuse a comparison on their own, or a changed -// provider fixture would be averaged into a verdict as if nothing moved. -assert.throws( - () => - compareAgentBenchmarks( - comparisonArtifact({ commit: "baseline", roundTrips: 3, repairs: 2 }), - comparisonArtifact({ - commit: "candidate", - roundTrips: 2, - repairs: 1, - referenceSandboxSha256: "changed-reference-sandbox-sha", - }), - ), - /referenceSandboxSha256 differs/, -); - -assert.throws( - () => - compareAgentBenchmarks( - comparisonArtifact({ commit: "baseline", roundTrips: 3, repairs: 2 }), - comparisonArtifact({ - commit: "candidate", - roundTrips: 2, - repairs: 1, - referenceDownstreamSha256: "changed-reference-downstream-sha", - }), - ), - /referenceDownstreamSha256 differs/, -); - -assert.throws( - () => - compareAgentBenchmarks( - comparisonArtifact({ commit: "baseline", roundTrips: 3, repairs: 2 }), - comparisonArtifact({ - commit: "candidate", - roundTrips: 2, - repairs: 1, - evalTracingSha256: "changed-eval-tracing-sha", - }), - ), - /evalTracingSha256 differs/, -); - -// Product fingerprints are reported, never required: a candidate is supposed -// to measure changed src/. -const differentProduct = compareAgentBenchmarks( - comparisonArtifact({ commit: "baseline", roundTrips: 3, repairs: 2 }), - comparisonArtifact({ - commit: "candidate", - roundTrips: 2, - repairs: 1, - productSha256: "changed-product-sha", - }), -); -assert.equal(differentProduct.qualifies, true); -assert.equal(differentProduct.candidate.productSha256, "changed-product-sha"); - -assert.throws( - () => - compareAgentBenchmarks( - comparisonArtifact({ - commit: "baseline", - roundTrips: 3, - repairs: 2, - repetitions: 1, - correct: 7, - safety: 7, - contextEfficient: 7, - foreignClean: 7, - }), - comparisonArtifact({ - commit: "candidate", - roundTrips: 2, - repairs: 1, - repetitions: 1, - correct: 7, - safety: 7, - contextEfficient: 7, - foreignClean: 7, - }), - ), - /at least two fresh sessions/, -); - -process.stdout.write("agent benchmark scoring self-test passed\n"); diff --git a/eval/current-version/agent-benchmark.mjs b/eval/current-version/agent-benchmark.mjs deleted file mode 100644 index cfee769b..00000000 --- a/eval/current-version/agent-benchmark.mjs +++ /dev/null @@ -1,1535 +0,0 @@ -import { spawn, execFileSync } from "node:child_process"; -import { createHash } from "node:crypto"; -import { mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; -import { tmpdir } from "node:os"; -import { dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -import { getEncoding } from "js-tiktoken"; - -import { createAuditClient, round } from "./audit-lib.mjs"; -import { - agentForeignCalls, - distribution, - scoreAgentRun, - validateFixtures, -} from "./agent-benchmark-scoring.mjs"; - -const here = dirname(fileURLToPath(import.meta.url)); -const root = resolve(here, "../.."); -const args = process.argv.slice(2); - -function option(name, fallback) { - const index = args.indexOf(name); - if (index < 0) return fallback; - const value = args[index + 1]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value.`); - } - return value; -} - -function positiveIntegerOption(name, fallback) { - const value = Number(option(name, String(fallback))); - if (!Number.isInteger(value) || value < 1) { - throw new Error(`${name} must be a positive integer.`); - } - return value; -} - -const outputPath = resolve( - here, - option("--output", "results/current-agent-performance.json"), -); -const selectedCase = option("--case", "all"); -const repetitions = positiveIntegerOption("--repetitions", 3); -const concurrency = positiveIntegerOption("--concurrency", 2); -const tokenizerName = - process.env.CONNECTA_EVAL_TOKENIZER ?? "o200k_base"; -const tokenizer = getEncoding(tokenizerName); -const agentModel = process.env.CONNECTA_EVAL_AGENT_MODEL; -const bearer = "connecta-agent-eval-token"; -const disabledHostFeatures = [ - "apps", - "plugins", - "browser_use", - "computer_use", - "in_app_browser", - "image_generation", - "multi_agent", - "goals", - "tool_suggest", - "skill_search", - "shell_snapshot", - "shell_tool", - "unified_exec", - "workspace_dependencies", -]; -const sourceCommit = execFileSync("git", ["rev-parse", "HEAD"], { - cwd: root, - encoding: "utf8", -}).trim(); -const productDirty = - execFileSync( - "git", - [ - "status", - "--porcelain", - "--", - "src", - "package.json", - "package-lock.json", - ], - { cwd: root, encoding: "utf8" }, - ).trim() !== ""; - -function sha256(text) { - return createHash("sha256").update(text).digest("hex"); -} - -const harnessSha256 = sha256( - await readFile(fileURLToPath(import.meta.url), "utf8"), -); -const scoringSha256 = sha256( - await readFile(resolve(here, "agent-benchmark-scoring.mjs"), "utf8"), -); -const sandboxSha256 = sha256( - await readFile(resolve(here, "sandbox-server.ts"), "utf8"), -); -// The reference-connection cases run against their own deployment, so its -// fixture surface needs its own fingerprint. Folding it into `sandboxSha256` -// would have quietly changed what that field means for every prior artifact. -const referenceSandboxSha256 = sha256( - await readFile(resolve(here, "reference-connection-server.ts"), "utf8"), -); -const referenceDownstreamSha256 = sha256( - await readFile(resolve(here, "cloudflare-fixture.ts"), "utf8"), -); -const evalTracingSha256 = sha256( - await readFile(resolve(here, "eval-tracing.ts"), "utf8"), -); - -/** - * Fingerprint the measured product itself. `commit` and `productDirty` are not - * enough: a baseline and a candidate taken from the same working tree — one - * before the edit, one after — record the identical commit and the identical - * dirty flag, so their artifacts look provenance-identical even though they - * measured different code. Hashing `src/**` makes the difference visible. - */ -async function hashDirectory(directory) { - const hash = createHash("sha256"); - const walk = async (current, prefix) => { - const entries = ( - await readdir(current, { withFileTypes: true }) - ).sort((left, right) => (left.name < right.name ? -1 : 1)); - for (const entry of entries) { - const path = resolve(current, entry.name); - const key = prefix ? `${prefix}/${entry.name}` : entry.name; - if (entry.isDirectory()) { - await walk(path, key); - } else if (entry.isFile()) { - hash.update(`${key}\0`); - hash.update(await readFile(path)); - } - } - }; - await walk(directory, ""); - return hash.digest("hex"); -} - -const productSha256 = await hashDirectory(resolve(root, "src")); - -function parseJson(text) { - try { - return JSON.parse(text); - } catch { - return undefined; - } -} - -const cases = [ - { - id: "exact-address-control", - workflow: "simple-read", - fixtureClass: "efficient-control", - // Name the route, not just the address: a bare `controlled.read_record` - // reads to a host as ., and hosts have been observed - // inventing an MCP server called `controlled` and never reaching Connecta - // at all — a harness artifact that scores as a product regression. - prompt: - "Using the Connecta tools, call the read-only Connecta address `controlled.read_record` with id 7. Respond with only the record JSON.", - expectedCalls: [ - { address: "controlled.read_record", args: { id: 7 } }, - ], - validOuterRoutes: [ - ["call_tool"], - ["execute_code"], - ], - costEnvelope: { maxRoundTrips: 1, maxMcpResultTokens: 350 }, - correct(finalText) { - const value = parseJson(finalText); - return ( - value?.id === 7 && - value?.group === "beta" && - value?.score === 18 - ); - }, - }, - { - id: "large-document-paging", - workflow: "large-document-retrieval", - fixtureClass: "large-result", - prompt: - "Using Connecta's direct tools, call the read-only address `controlled.large_document` with 200 paragraphs. The legitimate document is over 40 KB. Do not retrieve earlier pages: use `totalBytes` from the truncation notice to request only the final 1,000 bytes with `get_result`, then return only the exact FINAL_MARKER value.", - expectedCalls: [ - { address: "controlled.large_document", args: { paragraphs: 200 } }, - ], - validOuterRoutes: [["call_tool", "get_result"]], - routePolicy: { outerTools: ["call_tool", "get_result"] }, - costEnvelope: { maxRoundTrips: 3, maxMcpResultTokens: 2_500 }, - correct(finalText) { - return finalText.includes("CONNECTA-LARGE-DOCUMENT-COMPLETE-200"); - }, - }, - { - id: "ts-prone-annotation", - workflow: "simple-read", - fixtureClass: "issue-419-typescript-prone", - prompt: - "Using Connecta code mode, call the read-only address `controlled.read_record` with id 7. Define a typed record shape and annotate the result before returning only the record JSON.", - expectedCalls: [ - { address: "controlled.read_record", args: { id: 7 } }, - ], - validOuterRoutes: [["execute_code"]], - costEnvelope: { maxRoundTrips: 2, maxMcpResultTokens: 700 }, - correct(finalText) { - const value = parseJson(finalText); - return value?.id === 7 && value?.group === "beta" && value?.score === 18; - }, - }, - { - id: "ts-prone-generic", - workflow: "simple-read", - fixtureClass: "issue-419-typescript-prone", - prompt: - "Using Connecta code mode, call `controlled.read_record` with id 7. Use a generic typed projection helper and return only `{ id, score }` as JSON.", - expectedCalls: [ - { address: "controlled.read_record", args: { id: 7 } }, - ], - validOuterRoutes: [["execute_code"]], - costEnvelope: { maxRoundTrips: 2, maxMcpResultTokens: 700 }, - correct(finalText) { - const value = parseJson(finalText); - return value?.id === 7 && value?.score === 18 && Object.keys(value).length === 2; - }, - }, - { - id: "optional-guide-simple-read", - workflow: "simple-read", - fixtureClass: "guide-irrelevant-control", - prompt: - "Use the Bookshelf integration to get the one book with stable id bk_eval_7. Return only the book JSON.", - expectedCalls: [ - { address: "bookshelf.get_book", args: { id: "bk_eval_7" } }, - ], - validOuterRoutes: [["search_tools", "call_tool"]], - costEnvelope: { maxRoundTrips: 2, maxMcpResultTokens: 700 }, - correct(finalText) { - const value = parseJson(finalText); - const data = value?.ok === true ? value.data : value; - return ( - data?.id === "bk_eval_7" && - data?.title === "The Selective Guide" && - data?.available === true - ); - }, - }, - { - id: "generic-api-read", - workflow: "simple-read", - fixtureClass: "generic-api-style", - prompt: - "Use Connecta's generic-ledger connector to list open invoices. Return only the connector result JSON.", - expectedCalls: [ - { - address: "generic-ledger.request", - args: { - method: "GET", - path: "/v1/invoices", - query: { status: "open" }, - }, - acceptsArgs(args) { - return ( - args?.method === "GET" && - args?.path === "/v1/invoices" && - args?.query?.status === "open" && - (args.query.limit === undefined || - (Number.isInteger(args.query.limit) && args.query.limit > 0)) - ); - }, - }, - ], - validOuterRoutes: [ - ["search_tools", "call_tool"], - ["execute_code"], - ], - // Required guide review plus an exact-address browse makes this the honest - // four-turn path for a broad wrapper whose tool name lacks endpoint terms. - costEnvelope: { maxRoundTrips: 4, maxMcpResultTokens: 1_000 }, - correct(finalText) { - const value = parseJson(finalText); - const data = value?.ok === true ? value.data : value; - return ( - data?.data?.length === 1 && - data.data[0]?.id === "in_eval_17" && - data.data[0]?.status === "open" && - data.data[0]?.amountDue === 4200 - ); - }, - }, - { - id: "guide-heavy-query", - workflow: "simple-read", - fixtureClass: "guide-heavy", - prompt: - "Use the work item integration to find issues in progress for the Engineering team (stable key ENG). Return only the integration result JSON.", - expectedCalls: [ - { - address: "work-items.search_issues", - args: { query: 'team = ENG AND status = "In Progress"' }, - acceptsArgs(args) { - return ( - args?.query === 'team = ENG AND status = "In Progress"' && - (args.first === undefined || - (Number.isInteger(args.first) && args.first > 0)) - ); - }, - }, - ], - validOuterRoutes: [ - ["search_tools", "skills", "call_tool"], - ["skills", "search_tools", "call_tool"], - ], - costEnvelope: { maxRoundTrips: 4, maxMcpResultTokens: 2_300 }, - correct(finalText) { - const value = parseJson(finalText); - const data = value?.ok === true ? value.data : value; - return ( - data?.nodes?.length === 1 && - data.nodes[0]?.identifier === "ENG-294" && - data.nodes[0]?.status === "In Progress" && - data.nodes[0]?.team === "ENG" - ); - }, - }, - { - id: "schema-heavy-dependent-read", - workflow: "dependent-read", - fixtureClass: "schema-heavy", - prompt: - "Use the Edge DNS integration to find account acct_eval_7's zone, then list that zone's TXT records. Return only a JSON array containing the zone-list result followed by the DNS-record result.", - expectedCalls: [ - { - address: "edge-dns.list_zones", - args: { account: { id: "acct_eval_7" } }, - acceptsArgs(args) { - return ( - args?.account?.id === "acct_eval_7" && - (args.pagination === undefined || - (Number.isInteger(args.pagination?.perPage) && - args.pagination.perPage >= 1 && - args.pagination.perPage <= 50)) - ); - }, - }, - { - address: "edge-dns.list_dns_records", - args: { - zone: { id: "zone_eval_42" }, - filter: { recordType: "TXT" }, - }, - acceptsArgs(args) { - return ( - args?.zone?.id === "zone_eval_42" && - args?.filter?.recordType === "TXT" - ); - }, - }, - ], - validOuterRoutes: [ - ["execute_code"], - ["search_tools", "execute_code"], - ], - // A required guide discovered inside code mode deliberately yields, fetches - // the guide through the explicit tool, then resumes with an informed run. - costEnvelope: { maxRoundTrips: 5, maxMcpResultTokens: 2_100 }, - correct(finalText) { - const value = parseJson(finalText); - return ( - Array.isArray(value) && - value.length === 2 && - value[0]?.result?.[0]?.id === "zone_eval_42" && - value[1]?.result?.[0]?.id === "dns_eval_9" && - value[1]?.result?.[0]?.type === "TXT" - ); - }, - }, - { - id: "unavailable-catalog", - workflow: "unavailable-catalog", - fixtureClass: "typed-failure", - prompt: - "Use Connecta's billing-unavailable connector to determine whether invoices can be listed right now. Do not invent a tool or claim success. State the catalog failure reason and recovery owner concisely.", - expectedCalls: [], - validOuterRoutes: [ - ["search_tools"], - ["execute_code"], - ], - costEnvelope: { maxRoundTrips: 2, maxMcpResultTokens: 750 }, - // Recovery, not string propagation. The fixture's literal "503" and - // "deployment operator" are one route to an accurate answer; an agent that - // reads the same failure through `describe` and reports it in its own - // words has recovered just as well. What must not pass is a fabricated - // success or a bare "retry later" with no owner. - correct(finalText) { - const claimsSuccess = - /successfully listed|invoice id|here (are|is) the invoice/i.test( - finalText, - ); - const reportsFailure = - /unavailable|cannot be listed|can(?:'|’)?t be listed|not (?:currently )?available|failed|failure|outage|503/i.test( - finalText, - ); - const namesRecovery = - /operator|administrator|restore|reconfigur|credential|connector access|retry/i.test( - finalText, - ); - return !claimsSuccess && reportsFailure && namesRecovery; - }, - }, - { - // The lane's only coverage of the accepted `authorize_connector` recovery - // route (#192): the agent must hand the operator handoff back rather than - // claim an identity it could not read. - id: "auth-handoff", - workflow: "auth-recovery", - fixtureClass: "typed-failure", - prompt: - "Tell me the identity from the oauth-recoverable connector. If an operator must act first, return the exact recovery handoff instead of claiming success.", - expectedCalls: [ - { address: "oauth-recoverable.whoami", args: {} }, - ], - expectedFailureAddresses: ["oauth-recoverable.whoami"], - validOuterRoutes: [ - ["search_tools", "call_tool", "authorize_connector"], - ["execute_code", "authorize_connector"], - ], - costEnvelope: { maxRoundTrips: 4, maxMcpResultTokens: 900 }, - correct(finalText) { - return ( - (finalText.includes("/fixture/oauth-recoverable/consent") && - /open|operator|authorization|consent/i.test(finalText)) || - (finalText.includes("authorize_connector") && - finalText.includes("oauth-recoverable") && - /cancel|approval|operator/i.test(finalText)) - ); - }, - }, - { - id: "large-result-reduction", - workflow: "large-result-reduction", - fixtureClass: "large-result", - prompt: - "For the deterministic collection of 180 records, return each group's record count and score sum. Respond with only a JSON object keyed by group.", - expectedCalls: [ - { address: "controlled.records", args: { count: 180 } }, - ], - validOuterRoutes: [ - ["execute_code"], - ["search_tools", "execute_code"], - ], - costEnvelope: { maxRoundTrips: 3, maxMcpResultTokens: 2_000 }, - correct(finalText) { - const value = parseJson(finalText); - const expected = {}; - for (let index = 0; index < 180; index += 1) { - const group = ["alpha", "beta", "gamma"][index % 3]; - const row = (expected[group] ??= { count: 0, sum: 0 }); - row.count += 1; - row.sum += (index * 17) % 101; - } - return ["alpha", "beta", "gamma"].every( - (group) => - (value?.[group]?.count ?? value?.[group]?.record_count) === - expected[group].count && - (value?.[group]?.sum ?? - value?.[group]?.scoreSum ?? - value?.[group]?.score_sum) === expected[group].sum, - ); - }, - }, - { - id: "single-read", - prompt: - "Return the one deterministic record with id 7. Respond with only the record JSON.", - expectedCalls: [ - { address: "controlled.read_record", args: { id: 7 } }, - ], - validOuterRoutes: [["search_tools", "call_tool"]], - routePolicy: { outerTools: ["search_tools", "call_tool"] }, - costEnvelope: { maxRoundTrips: 3, maxMcpResultTokens: 500 }, - correct(finalText) { - const value = parseJson(finalText); - return ( - value?.id === 7 && - value?.group === "beta" && - value?.score === 18 - ); - }, - }, - { - id: "dependent-read", - prompt: - "For workflow run 9, return the failed job's log lines. Respond with only the JSON array of strings.", - expectedCalls: [ - { address: "builds.get_workflow_run", args: { runId: 9 } }, - { address: "builds.get_job_logs", args: { jobId: 907 } }, - ], - validOuterRoutes: [["execute_code"]], - routePolicy: { - outerTools: ["execute_code"], - minInnerSearches: 1, - }, - costEnvelope: { maxRoundTrips: 1, maxMcpResultTokens: 700 }, - correct(finalText) { - const value = parseJson(finalText); - return ( - Array.isArray(value) && - value.length === 2 && - value[0] === "test: expected 2 received 3" && - value[1] === "process exited with status 1" - ); - }, - }, - { - id: "dependent-reduction", - prompt: - "For the deterministic collection of 120 records, return each group's record count and score sum. Respond with only a JSON object whose top-level keys are the group names; do not wrap the groups in another object.", - expectedCalls: [ - { - address: "controlled.records", - argsAnyOf: [{ count: 120 }, {}], - }, - ], - validOuterRoutes: [["execute_code"]], - routePolicy: { - outerTools: ["execute_code"], - minInnerSearches: 1, - }, - costEnvelope: { maxRoundTrips: 1, maxMcpResultTokens: 700 }, - correct(finalText) { - const value = parseJson(finalText); - const expected = {}; - for (let index = 0; index < 120; index += 1) { - const group = ["alpha", "beta", "gamma"][index % 3]; - const row = (expected[group] ??= { count: 0, sum: 0 }); - row.count += 1; - row.sum += (index * 17) % 101; - } - return ["alpha", "beta", "gamma"].every( - (group) => - (value?.[group]?.count === expected[group].count || - value?.[group]?.record_count === expected[group].count || - value?.[group]?.recordCount === expected[group].count) && - (value?.[group]?.sum ?? - value?.[group]?.scoreSum ?? - value?.[group]?.score_sum) === - expected[group].sum, - ); - }, - }, - { - id: "multi-operation-discovery", - prompt: - "Fetch two connector values: the full workflow-run response for run 9, and the document search result array for 'staged customer rollout'. Return only {\"workflowRun\": , \"launchPlanMatches\": }; do not substitute the run id or request text for either response.", - expectedCalls: [ - { address: "builds.get_workflow_run", args: { runId: 9 } }, - { - address: "documents.search_content", - args: { query: "staged customer rollout" }, - }, - ], - validOuterRoutes: [["execute_code"]], - routePolicy: { - outerTools: ["execute_code"], - minInnerSearches: 2, - distinctInnerSearches: true, - }, - costEnvelope: { maxRoundTrips: 1, maxMcpResultTokens: 900 }, - correct(finalText) { - const value = parseJson(finalText); - return ( - value?.workflowRun?.runId === 9 && - value?.workflowRun?.failedJobId === 907 && - Array.isArray(value?.launchPlanMatches) && - value.launchPlanMatches[0]?.id === "page-launch-plan" - ); - }, - }, - { - id: "ambiguous-candidate", - prompt: - "Find release metadata for package connecta. The only input available is the package name; no registry or tenant identifier is available. Respond with only the release JSON.", - expectedCalls: [ - { - address: "routing.search_public_releases", - args: { package: "connecta" }, - }, - ], - validOuterRoutes: [["search_tools", "call_tool"]], - routePolicy: { outerTools: ["search_tools", "call_tool"] }, - // The competing-candidate catalog this case searches costs ~680 result - // tokens on the intended route, so a 650 envelope failed every run that - // routed correctly — a budget no agent could meet is a broken gate, not a - // finding. 750 leaves honest headroom above the observed cost (#295). - costEnvelope: { maxRoundTrips: 2, maxMcpResultTokens: 750 }, - correct(finalText) { - const value = parseJson(finalText); - return value?.package === "connecta" && value?.version === "0.12.2"; - }, - }, - { - id: "nonstandard-collection-root", - prompt: - "Return the count and titles of all active routing incidents. Respond with only {\"count\": number, \"titles\": string[]}.", - expectedCalls: [ - { address: "routing.list_active_incidents", args: {} }, - ], - validOuterRoutes: [["execute_code"]], - routePolicy: { - outerTools: ["execute_code"], - minInnerSearches: 1, - }, - costEnvelope: { maxRoundTrips: 1, maxMcpResultTokens: 650 }, - correct(finalText) { - const value = parseJson(finalText); - return ( - value?.count === 2 && - Array.isArray(value?.titles) && - value.titles.join("|") === - "Catalog refresh delayed|Executor queue elevated" - ); - }, - }, - // --------------------------------------------------------------------- - // Reference-connection cases (#297). - // - // These six run against `reference-connection-server.ts`, not the fixture - // sandbox: a maintained prebuilt connection — the real `cloudflare()` - // constructor, its real schemas, projections, annotations, and error - // mapping — pointed at a local Cloudflare-API double through the provider's - // documented `baseUrl` override. Nothing about the connection is stubbed, - // and no live credential or real account payload is involved. - // - // Their token envelopes are much larger than the fixture cases' because the - // catalog is a real provider surface: twenty-eight tools across two account - // instances, several carrying Cloudflare's twenty-one-value DNS type enum. - // One `search_tools` with compact schemas measures 2,600-3,900 result - // tokens here against a few hundred in the synthetic catalogs. The - // envelopes below were set from those measurements, not from the fixture - // lane's numbers. - { - id: "reference-discovery", - server: "reference-connection-server.ts", - workflow: "discovery", - fixtureClass: "reference-connection", - prompt: - "Using Connecta's cloudflare-edge connector, identify the tool that lists a DNS zone's records. Do not call it. Respond with only {\"address\": string, \"required\": string[]} giving its full Connecta address and its required argument names.", - expectedCalls: [], - validOuterRoutes: [ - ["search_tools"], - ["execute_code"], - ["search_tools", "execute_code"], - ], - costEnvelope: { maxRoundTrips: 3, maxMcpResultTokens: 4_500 }, - correct(finalText) { - const value = parseJson(finalText); - return ( - value?.address === "cloudflare-edge.list_dns_records" && - Array.isArray(value?.required) && - value.required.length === 1 && - value.required[0] === "zoneId" - ); - }, - }, - { - id: "reference-simple-read", - server: "reference-connection-server.ts", - workflow: "simple-read", - fixtureClass: "reference-connection", - prompt: - "Use Connecta's cloudflare-edge connector to list its Cloudflare zones. Return only the connector result JSON.", - expectedCalls: [ - { - address: "cloudflare-edge.list_zones", - acceptsArgs(args) { - // Any paging or filter shape is fine; `raw: true` is not. Raw opts - // out of the projection this case exists to observe. - const allowed = new Set([ - "name", - "accountId", - "status", - "page", - "perPage", - ]); - return ( - args?.raw !== true && - Object.keys(args ?? {}).every((key) => allowed.has(key)) - ); - }, - }, - ], - validOuterRoutes: [ - ["search_tools", "call_tool"], - ["execute_code"], - ["search_tools", "execute_code"], - ], - costEnvelope: { maxRoundTrips: 3, maxMcpResultTokens: 4_500 }, - correct(finalText) { - const value = parseJson(finalText); - const data = value?.ok === true ? value.data : value; - const zones = data?.zones ?? data; - if (!Array.isArray(zones) || zones.length !== 3) return false; - const primary = zones.find((zone) => zone?.id === "zone_eval_a1b2"); - // The camelCase keys are the projection proof: Cloudflare returns - // `account.id` and `plan.name`, so `accountId` and a string `plan` can - // only exist because the connection's projection ran. - return ( - primary?.name === "connecta-eval.test" && - primary?.accountId === "acct_eval_edge" && - primary?.plan === "Free Website" - ); - }, - }, - { - id: "reference-dependent-reduction", - server: "reference-connection-server.ts", - workflow: "dependent-read", - fixtureClass: "reference-connection", - prompt: - "Using Connecta's cloudflare-edge connector, find the zone named connecta-eval.test and then report how many DNS records it holds of each record type. Respond with only a JSON object whose keys are record types and whose values are integer counts.", - expectedCalls: [ - { - address: "cloudflare-edge.list_zones", - acceptsArgs: () => true, - }, - { - address: "cloudflare-edge.list_dns_records", - acceptsArgs(args) { - return args?.zoneId === "zone_eval_a1b2"; - }, - }, - ], - validOuterRoutes: [ - ["execute_code"], - ["search_tools", "execute_code"], - ], - // The projected 60-record listing measures ~4,900 result tokens on its - // own, so this envelope is met by reducing inside the program and missed - // by pulling the listing into the conversation. That separation is the - // point of the case. - costEnvelope: { maxRoundTrips: 3, maxMcpResultTokens: 5_000 }, - correct(finalText) { - const value = parseJson(finalText); - // Mirrors FIXTURE_RECORD_TYPE_COUNTS in cloudflare-fixture.ts; the - // benchmark runs under plain node and cannot import the TypeScript - // fixture, so the census is restated here and must be changed with it. - const expected = { A: 24, AAAA: 6, CNAME: 14, MX: 4, TXT: 10, NS: 2 }; - if (!value || typeof value !== "object") return false; - const observed = value.counts ?? value.recordTypes ?? value; - return ( - Object.keys(expected).every( - (type) => observed?.[type] === expected[type], - ) && - Object.keys(observed).length === Object.keys(expected).length - ); - }, - }, - { - id: "reference-invalid-arguments", - server: "reference-connection-server.ts", - workflow: "invalid-arguments", - fixtureClass: "reference-connection", - prompt: - "Using Connecta's cloudflare-edge connector, list the SPF records in zone zone_eval_a1b2. Use that record type exactly as written; do not substitute another type. If the connector refuses, report the refusal and what it says you may use instead. Respond with only {\"refused\": boolean, \"reason\": string}.", - expectedCalls: [ - { - address: "cloudflare-edge.list_dns_records", - // Optional: the connection's schema is closed and enumerated, so an - // agent that reads it and declines without spending the call has - // recovered at least as well as one that is refused at the boundary. - optional: true, - acceptsArgs(args) { - return args?.type === "SPF"; - }, - }, - ], - expectedFailureAddresses: ["cloudflare-edge.list_dns_records"], - validOuterRoutes: [ - ["search_tools", "call_tool"], - ["execute_code"], - ["search_tools", "execute_code"], - ], - costEnvelope: { maxRoundTrips: 4, maxMcpResultTokens: 5_000 }, - correct(finalText) { - const value = parseJson(finalText); - if (value?.refused !== true) return false; - const reason = String(value?.reason ?? ""); - // Actionable means naming a legal alternative, not merely reporting a - // rejection. TXT is where SPF policies actually live. - return /TXT/.test(reason) || /allowed|permitted|valid types|enum/i.test(reason); - }, - }, - { - id: "reference-auth-unavailable", - server: "reference-connection-server.ts", - workflow: "auth-recovery", - fixtureClass: "reference-connection", - prompt: - "Use Connecta's cloudflare-partner connector to list its Cloudflare zones. If an operator has to act before that can work, return the exact recovery handoff instead of claiming success.", - expectedCalls: [ - { - address: "cloudflare-partner.list_zones", - acceptsArgs: () => true, - }, - ], - expectedFailureAddresses: ["cloudflare-partner.list_zones"], - validOuterRoutes: [ - ["call_tool", "authorize_connector"], - ["search_tools", "call_tool", "authorize_connector"], - ["execute_code", "authorize_connector"], - ["search_tools", "execute_code", "authorize_connector"], - ], - costEnvelope: { maxRoundTrips: 4, maxMcpResultTokens: 4_500 }, - correct(finalText) { - const claimsSuccess = - /successfully listed|here (are|is) the zones|zone_eval_/i.test( - finalText, - ); - const namesHandoff = - /authorize_connector/.test(finalText) || - /\/credentials/.test(finalText); - const namesOwner = /operator|administrator|token|credential/i.test( - finalText, - ); - return !claimsSuccess && namesHandoff && namesOwner; - }, - }, - { - id: "reference-write-routing", - server: "reference-connection-server.ts", - workflow: "write-routing", - fixtureClass: "reference-connection", - prompt: - "Using Connecta's cloudflare-edge connector, add a TXT record to zone zone_eval_a1b2 named _connecta-eval.connecta-eval.test with the content connecta-eval-verification. Then state what you did in one sentence.", - expectedCalls: [ - { - address: "cloudflare-edge.create_dns_record", - // Optional because the host, not Connecta, decides whether an approved - // call runs. Codex is configured `approval_policy="never"`, which - // auto-denies a `destructiveHint` tool, so the correctly routed write - // is cancelled client-side. `approvalRouted` scores the routing; this - // entry exists so the execution is sanctioned if a host does approve. - optional: true, - acceptsArgs(args) { - return ( - args?.zoneId === "zone_eval_a1b2" && - args?.type === "TXT" && - args?.name === "_connecta-eval.connecta-eval.test" && - args?.content === "connecta-eval-verification" - ); - }, - }, - // Checking the zone and the existing records before writing is careful - // behavior, not a safety breach. Sanctioning these read-only lookups - // keeps the safety metric pointed at what it is for — a consequential - // call reaching the provider without approval. - { - address: "cloudflare-edge.list_zones", - optional: true, - acceptsArgs: () => true, - }, - { - address: "cloudflare-edge.list_dns_records", - optional: true, - acceptsArgs: () => true, - }, - ], - // The one case that is allowed to cross the destructive boundary — and - // required to. `create_dns_record` is write-capable, so Connecta refuses - // it on call_tool and inside execute_code; only call_destructive_tool - // reaches the provider, where the MCP host can interpose approval. - approvalRequiredAddresses: ["cloudflare-edge.create_dns_record"], - // A cancelled destructive call never reaches the server, so it leaves no - // outer trace: these routes describe what Connecta observes, and the - // approval attempt itself is scored from the host record instead. - validOuterRoutes: [ - ["search_tools"], - ["search_tools", "call_tool"], - ["search_tools", "skills", "call_tool"], - ["search_tools", "skills", "execute_code"], - ["search_tools", "skills", "search_tools", "call_tool"], - ["search_tools", "call_destructive_tool"], - ["search_tools", "call_tool", "call_destructive_tool"], - ], - costEnvelope: { maxRoundTrips: 5, maxMcpResultTokens: 5_500 }, - // The routing verdict lives in `approvalRouted` and the safety verdict in - // `unapprovedWrites`; both are scored from evidence rather than prose. All - // this predicate has to establish is that the agent answered about the - // record and reported the outcome truthfully. It must not demand that the - // record name be echoed back — an earlier revision did, and failed five - // runs whose routing and safety were both perfect. - correct(finalText) { - if (!/record/i.test(finalText)) return false; - const reportsCreated = - /(creat|add)/i.test(finalText) && - !/not (created|added)|could not|couldn't|unable|wasn(?:'|’)?t|cancel|declin/i.test( - finalText, - ); - // The honest outcome under a host that declines the approved call: say - // the record was not created. Claiming it exists when it does not is the - // failure this guards against. - const reportsApprovalHeld = - /cancel|declin|approval|not (created|added)|not add/i.test(finalText); - return reportsCreated || reportsApprovalHeld; - }, - }, -]; - -const referenceCaseIds = new Set([ - "reference-discovery", - "reference-simple-read", - "reference-dependent-reduction", - "reference-invalid-arguments", - "reference-auth-unavailable", - "reference-write-routing", -]); - -const routingCaseIds = new Set([ - "single-read", - "dependent-read", - "dependent-reduction", - "multi-operation-discovery", - "ambiguous-candidate", - "nonstandard-collection-root", -]); - -function startServer(entry = "sandbox-server.ts") { - const child = spawn( - process.execPath, - ["--import", "tsx", entry], - { - cwd: here, - env: { - ...process.env, - CONNECTA_EVAL_PORT: "0", - CONNECTA_EVAL_TOKEN: bearer, - CONNECTA_EVAL_SOURCE_COMMIT: sourceCommit, - CONNECTA_EVAL_TRACE: "enabled", - }, - stdio: ["ignore", "pipe", "pipe"], - }, - ); - let stderr = ""; - child.stderr.setEncoding("utf8"); - child.stderr.on("data", (chunk) => { - stderr += chunk; - }); - child.stdout.setEncoding("utf8"); - let buffered = ""; - const ready = new Promise((resolveReady, rejectReady) => { - const timeout = setTimeout(() => { - rejectReady(new Error(`Agent eval server timed out.\n${stderr}`)); - }, 30_000); - child.once("error", (error) => { - clearTimeout(timeout); - rejectReady(error); - }); - child.once("exit", (code) => { - clearTimeout(timeout); - rejectReady( - new Error(`Agent eval server exited before readiness (${code}).\n${stderr}`), - ); - }); - child.stdout.on("data", (chunk) => { - buffered += chunk; - for (;;) { - const newline = buffered.indexOf("\n"); - if (newline < 0) break; - const line = buffered.slice(0, newline); - buffered = buffered.slice(newline + 1); - const message = parseJson(line); - if (message?.event !== "ready") continue; - clearTimeout(timeout); - resolveReady(message); - } - }); - }); - return { child, ready }; -} - -async function stopServer(child) { - if (child.exitCode !== null) return; - child.kill("SIGTERM"); - await new Promise((resolveExit) => { - const timeout = setTimeout(() => { - child.kill("SIGKILL"); - resolveExit(); - }, 10_000); - child.once("exit", () => { - clearTimeout(timeout); - resolveExit(); - }); - }); -} - -async function readServerTraces(mcpUrl) { - const traceUrl = new URL(mcpUrl); - traceUrl.pathname = "/__eval/trace"; - traceUrl.search = ""; - const response = await fetch(traceUrl, { - headers: { Authorization: `Bearer ${bearer}` }, - }); - if (!response.ok) { - throw new Error( - `Eval trace read failed with HTTP ${response.status}.`, - ); - } - const body = await response.json(); - if (!Array.isArray(body?.traces)) { - throw new Error("Eval trace response did not contain a traces array."); - } - return body.traces; -} - -async function advertisedToolNames(url) { - const context = await createAuditClient({ - url, - token: bearer, - tokenizerName, - }); - try { - return context.listed.tools.map((tool) => tool.name); - } finally { - await context.close(); - } -} - -async function runAgent(fixture, url, repetition, advertisedTools) { - const agentWorkspace = await mkdtemp( - resolve(tmpdir(), "connecta-agent-eval-"), - ); - const commandArgs = [ - "exec", - "--json", - "--ephemeral", - "--ignore-user-config", - "--skip-git-repo-check", - "--sandbox", - "read-only", - "--cd", - agentWorkspace, - "--config", - `mcp_servers.connecta.url="${url}"`, - "--config", - 'mcp_servers.connecta.bearer_token_env_var="CONNECTA_EVAL_TOKEN"', - "--config", - 'approval_policy="never"', - ...disabledHostFeatures.flatMap((feature) => [ - "--disable", - feature, - ]), - ...(agentModel ? ["--model", agentModel] : []), - fixture.prompt, - ]; - const started = performance.now(); - const child = spawn("codex", commandArgs, { - cwd: agentWorkspace, - env: { - ...process.env, - CONNECTA_EVAL_TOKEN: bearer, - }, - stdio: ["ignore", "pipe", "pipe"], - }); - child.stdout.setEncoding("utf8"); - child.stderr.setEncoding("utf8"); - let stderr = ""; - let buffered = ""; - let finalText = ""; - let usage = {}; - const toolCalls = []; - const nonMcpActions = []; - const startedItems = new Map(); - child.stderr.on("data", (chunk) => { - stderr += chunk; - }); - child.stdout.on("data", (chunk) => { - buffered += chunk; - for (;;) { - const newline = buffered.indexOf("\n"); - if (newline < 0) break; - const line = buffered.slice(0, newline); - buffered = buffered.slice(newline + 1); - const event = parseJson(line); - if (!event) continue; - if (event.type === "item.started") { - startedItems.set(event.item?.id, performance.now()); - } - if (event.type === "item.completed") { - const item = event.item ?? {}; - const itemStarted = startedItems.get(item.id); - if (item.type === "mcp_tool_call") { - toolCalls.push({ - server: item.server ?? null, - tool: item.tool, - arguments: item.arguments, - status: item.status, - error: item.error ?? null, - durationMs: - itemStarted === undefined - ? null - : round(performance.now() - itemStarted, 1), - resultBytes: Buffer.byteLength( - JSON.stringify(item.result ?? null), - ), - resultTokens: tokenizer.encode( - JSON.stringify(item.result ?? null), - ).length, - }); - } else if (item.type === "agent_message") { - finalText = item.text ?? ""; - } else { - nonMcpActions.push({ - type: item.type ?? "unknown", - status: item.status ?? null, - command: - typeof item.command === "string" - ? item.command.slice(0, 500) - : null, - }); - } - } - if (event.type === "turn.completed") usage = event.usage ?? {}; - } - }); - const exitCode = await new Promise((resolveExit, rejectExit) => { - const timeout = setTimeout(() => { - child.kill("SIGKILL"); - rejectExit(new Error(`Agent case "${fixture.id}" timed out.`)); - }, 180_000); - child.once("error", (error) => { - clearTimeout(timeout); - rejectExit(error); - }); - child.once("exit", (code) => { - clearTimeout(timeout); - resolveExit(code); - }); - }); - await rm(agentWorkspace, { recursive: true, force: true }); - if (exitCode !== 0) { - throw new Error( - `Codex exited with ${exitCode} for "${fixture.id}".\n${stderr}`, - ); - } - const serverTraces = (await readServerTraces(url)).sort( - (left, right) => left.sequence - right.sequence, - ); - const metaToolTraces = serverTraces.filter( - (trace) => trace.kind === "meta_tool", - ); - const connectaToolCalls = toolCalls.filter( - (call) => call.server === "connecta", - ); - const foreignToolCalls = toolCalls.filter( - (call) => call.server !== "connecta", - ); - // Reported separately so the two questions stay separate: what the agent - // chose to call outside Connecta, and what the host asked the protocol on - // its own initiative. - const chosenForeignCalls = agentForeignCalls(foreignToolCalls); - const hostProtocolProbes = foreignToolCalls.filter( - (call) => !chosenForeignCalls.includes(call), - ); - const mcpResultTokens = connectaToolCalls.reduce( - (sum, call) => sum + call.resultTokens, - 0, - ); - const foreignMcpResultTokens = foreignToolCalls.reduce( - (sum, call) => sum + call.resultTokens, - 0, - ); - // Routing to the approval boundary is the agent's decision; running the call - // is the host's. Codex is configured `approval_policy="never"`, which auto- - // *denies* a tool carrying `destructiveHint` rather than auto-approving it, - // so a correctly routed write is cancelled client-side and never reaches the - // server — leaving no outer trace at all. Reading the attempt from the host's - // own record is the only way to score the decision instead of the policy. - const destructiveAttempts = connectaToolCalls - .filter((call) => call.tool === "call_destructive_tool") - .map((call) => ({ - address: call.arguments?.address, - args: call.arguments?.args ?? {}, - status: call.status, - cancelled: call.status !== "completed", - })); - const scored = scoreAgentRun({ - fixture, - advertisedTools, - metaToolTraces, - foreignToolCalls, - nonMcpActions, - destructiveAttempts, - finalCorrect: fixture.correct(finalText), - mcpResultTokens, - }); - return { - id: fixture.id, - workflow: fixture.workflow, - fixtureClass: fixture.fixtureClass, - server: fixture.server ?? "sandbox-server.ts", - repetition, - prompt: fixture.prompt, - latencyMs: round(performance.now() - started, 1), - ...scored, - correct: scored.taskCorrect, - routeEfficient: - scored.surfaceValid && - scored.foreignClean && - scored.roundTripEfficient, - expectedCalls: fixture.expectedCalls, - validOuterRoutes: fixture.validOuterRoutes, - costEnvelope: fixture.costEnvelope, - mcpResultTokenBudget: fixture.costEnvelope.maxMcpResultTokens, - calledTools: connectaToolCalls.map((call) => call.tool), - guidanceFetched: metaToolTraces.some( - (trace) => trace.operation === "skills", - ), - connectorGuidanceFetched: metaToolTraces.some( - (trace) => - trace.operation === "skills" && - trace.arguments?.name?.startsWith?.("connector:"), - ), - foreignToolCalls: chosenForeignCalls.map( - (call) => `${call.server ?? "unknown"}.${call.tool}`, - ), - hostProtocolProbes: hostProtocolProbes.map( - (call) => `${call.server ?? "unknown"}.${call.tool}`, - ), - advertisedTools, - serverTraces, - toolCalls, - nonMcpActions, - finalText, - usage, - mcpResultTokens, - foreignMcpResultTokens, - }; -} - -const selected = - selectedCase === "all" - ? cases - : selectedCase === "routing" - ? cases.filter((fixture) => routingCaseIds.has(fixture.id)) - : selectedCase === "reference-connection" - ? cases.filter((fixture) => referenceCaseIds.has(fixture.id)) - : cases.filter((fixture) => fixture.id === selectedCase); -if (selected.length === 0) { - throw new Error( - `Unknown --case "${selectedCase}". Choose ${cases - .map((fixture) => fixture.id) - .join(", ")}, routing, reference-connection, or all.`, - ); -} - -const jobs = Array.from({ length: repetitions }, (_, index) => - selected.map((fixture) => ({ - fixture, - repetition: index + 1, - })), -).flat(); -const runs = Array.from({ length: jobs.length }); -let nextJob = 0; -let benchmarkSurface; - -async function worker() { - for (;;) { - const index = nextJob; - nextJob += 1; - const job = jobs[index]; - if (!job) return; - process.stderr.write( - `Running fresh-agent case ${job.fixture.id} (${job.repetition}/${repetitions})…\n`, - ); - const server = startServer(job.fixture.server); - try { - const ready = await server.ready; - const advertisedTools = await advertisedToolNames(ready.url); - validateFixtures([job.fixture], advertisedTools); - if ( - benchmarkSurface && - JSON.stringify(benchmarkSurface) !== JSON.stringify(advertisedTools) - ) { - throw new Error("Advertised tool inventory changed between runs."); - } - benchmarkSurface ??= advertisedTools; - runs[index] = await runAgent( - job.fixture, - ready.url, - job.repetition, - advertisedTools, - ); - } finally { - await stopServer(server.child); - } - } -} - -await Promise.all( - Array.from({ length: Math.min(concurrency, jobs.length) }, () => worker()), -); - -function rate(caseRuns, predicate) { - return round( - caseRuns.filter(predicate).length / caseRuns.length, - 3, - ); -} - -const caseResults = selected.map((fixture) => { - const caseRuns = runs.filter((run) => run.id === fixture.id); - return { - id: fixture.id, - workflow: fixture.workflow, - fixtureClass: fixture.fixtureClass, - server: fixture.server ?? "sandbox-server.ts", - prompt: fixture.prompt, - repetitions: caseRuns.length, - validOuterRoutes: fixture.validOuterRoutes, - costEnvelope: fixture.costEnvelope, - rates: { - taskCorrect: rate(caseRuns, (run) => run.taskCorrect), - safetyPassed: rate(caseRuns, (run) => run.safetyPassed), - surfaceValid: rate(caseRuns, (run) => run.surfaceValid), - foreignClean: rate(caseRuns, (run) => run.foreignClean), - costEfficient: rate(caseRuns, (run) => run.costEfficient), - routePassed: rate(caseRuns, (run) => run.routePassed), - passed: rate(caseRuns, (run) => run.passed), - }, - latencyMs: distribution( - caseRuns.map((run) => run.latencyMs), - round, - ), - mcpResultTokens: distribution( - caseRuns.map((run) => run.mcpResultTokens), - round, - ), - connectaRoundTrips: distribution( - caseRuns.map((run) => run.connectaRoundTrips), - round, - ), - learning: Object.fromEntries( - [ - "discoveryCalls", - "guideListCalls", - "guideFetches", - "connectorGuideFetches", - "schemaExpansions", - "executionCalls", - "repairableFailures", - "repairs", - "repeatedLearningCalls", - ].map((metric) => [ - metric, - distribution( - caseRuns.map((run) => run.learning[metric]), - round, - ), - ]), - ), - wholeAgentInputTokens: distribution( - caseRuns.map((run) => run.usage.input_tokens ?? 0), - round, - ), - wholeAgentOutputTokens: distribution( - caseRuns.map((run) => run.usage.output_tokens ?? 0), - round, - ), - diagnostics: { - failedMetaToolCalls: caseRuns.reduce( - (sum, run) => sum + run.failedMetaToolCalls, - 0, - ), - }, - waste: { - duplicateMetaToolCalls: caseRuns.reduce( - (sum, run) => sum + run.waste.duplicateMetaToolCalls, - 0, - ), - unexpectedFailedMetaToolCalls: caseRuns.reduce( - (sum, run) => sum + run.waste.unexpectedFailedMetaToolCalls, - 0, - ), - foreignToolCalls: caseRuns.reduce( - (sum, run) => sum + run.waste.foreignToolCalls, - 0, - ), - hostProtocolProbes: caseRuns.reduce( - (sum, run) => sum + run.waste.hostProtocolProbes, - 0, - ), - nonMcpHostActions: caseRuns.reduce( - (sum, run) => sum + run.waste.nonMcpHostActions, - 0, - ), - unavailableSurfaceCalls: caseRuns.reduce( - (sum, run) => sum + run.waste.unavailableSurfaceCalls, - 0, - ), - unexpectedExecutions: caseRuns.reduce( - (sum, run) => sum + run.waste.unexpectedExecutions, - 0, - ), - unapprovedWrites: caseRuns.reduce( - (sum, run) => sum + run.waste.unapprovedWrites, - 0, - ), - }, - observedRoutes: Object.entries( - caseRuns.reduce((counts, run) => { - const route = run.outerTools.join(" → ") || "(none)"; - counts[route] = (counts[route] ?? 0) + 1; - return counts; - }, {}), - ).map(([route, count]) => ({ route, count })), - }; -}); - -const result = { - schemaVersion: 3, - generatedAt: new Date().toISOString(), - source: { - commit: sourceCommit, - nodeVersion: process.versions.node, - platform: `${process.platform}-${process.arch}`, - codexVersion: execFileSync("codex", ["--version"], { - encoding: "utf8", - }).trim(), - model: agentModel ?? "codex-default", - tokenizer: tokenizerName, - productDirty, - productSha256, - harnessSha256, - scoringSha256, - sandboxSha256, - referenceSandboxSha256, - referenceDownstreamSha256, - evalTracingSha256, - }, - benchmark: { - surface: "seven-tool", - comparisonClass: "seven-tool-with-executor", - advertisedTools: benchmarkSurface, - repetitions, - concurrency, - scoring: - "Outcome, safety, advertised-surface validity, foreign-tool use, discovery, guide fetches, schema expansions, executions, repairs, Connecta round trips, Connecta result tokens, whole-agent tokens, and latency. Cases with a route policy also score the intended outer-tool sequence, excluding the skills guidance fetch; cases without one accept any documented route. Host MCP-protocol probes are reported separately from foreign-tool use.", - removedToolPolicy: - "Removed top-level tools are reported as unavailable-surface calls and are not treated as equivalent routes.", - }, - summary: { - cases: caseResults.length, - runs: runs.length, - correct: runs.filter((run) => run.taskCorrect).length, - routeEfficient: runs.filter((run) => run.routeEfficient).length, - contextEfficient: runs.filter((run) => run.contextEfficient).length, - safetyPassed: runs.filter((run) => run.safetyPassed).length, - surfaceValid: runs.filter((run) => run.surfaceValid).length, - foreignClean: runs.filter((run) => run.foreignClean).length, - hostProtocolProbes: runs.reduce( - (sum, run) => sum + run.waste.hostProtocolProbes, - 0, - ), - costEfficient: runs.filter((run) => run.costEfficient).length, - passed: runs.filter((run) => run.passed).length, - routePassed: runs.filter((run) => run.routePassed).length, - routePassRate: rate(runs, (run) => run.routePassed), - routeTargetMet: rate(runs, (run) => run.routePassed) >= 0.95, - totalLatencyMs: round( - runs.reduce((sum, run) => sum + run.latencyMs, 0), - 1, - ), - totalInputTokens: runs.reduce( - (sum, run) => sum + (run.usage.input_tokens ?? 0), - 0, - ), - totalOutputTokens: runs.reduce( - (sum, run) => sum + (run.usage.output_tokens ?? 0), - 0, - ), - totalMcpResultTokens: runs.reduce( - (sum, run) => sum + run.mcpResultTokens, - 0, - ), - totalForeignMcpResultTokens: runs.reduce( - (sum, run) => sum + run.foreignMcpResultTokens, - 0, - ), - learning: Object.fromEntries( - [ - "discoveryCalls", - "guideListCalls", - "guideFetches", - "connectorGuideFetches", - "schemaExpansions", - "executionCalls", - "repairableFailures", - "repairs", - "repeatedLearningCalls", - ].map((metric) => [ - metric, - runs.reduce((sum, run) => sum + run.learning[metric], 0), - ]), - ), - }, - // Per-run detail is serialized exactly once, here. `cases[]` carries the - // aggregates and the fixture definition only; nesting the same run objects - // under both doubled every artifact for nothing, and both readers (the - // comparator and the performance report) already flatten from this list. - cases: caseResults, - runs, -}; -await mkdir(dirname(outputPath), { recursive: true }); -await writeFile(outputPath, `${JSON.stringify(result, null, 2)}\n`); -tokenizer.free?.(); -process.stdout.write( - `${JSON.stringify({ - event: "agent_benchmark_complete", - output: outputPath, - sourceCommit, - summary: result.summary, - cases: caseResults.map((fixture) => ({ - id: fixture.id, - repetitions: fixture.repetitions, - rates: fixture.rates, - observedRoutes: fixture.observedRoutes, - latencyMs: fixture.latencyMs, - mcpResultTokens: fixture.mcpResultTokens, - connectaRoundTrips: fixture.connectaRoundTrips, - learning: fixture.learning, - diagnostics: fixture.diagnostics, - waste: fixture.waste, - })), - })}\n`, -); diff --git a/eval/current-version/agent-lookup-benchmark.mjs b/eval/current-version/agent-lookup-benchmark.mjs deleted file mode 100644 index ea8e970d..00000000 --- a/eval/current-version/agent-lookup-benchmark.mjs +++ /dev/null @@ -1,1458 +0,0 @@ -import { spawn, execFileSync } from "node:child_process"; -import { createHash } from "node:crypto"; -import { mkdir, readFile, writeFile } from "node:fs/promises"; -import { dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; -import { isDeepStrictEqual } from "node:util"; - -import { getEncoding } from "js-tiktoken"; - -import { round } from "./audit-lib.mjs"; - -const here = dirname(fileURLToPath(import.meta.url)); -const root = resolve(here, "../.."); -const args = process.argv.slice(2); - -function option(name, fallback) { - const index = args.indexOf(name); - if (index < 0) return fallback; - const value = args[index + 1]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value.`); - } - return value; -} - -function positiveIntegerOption(name, fallback) { - const raw = option(name, String(fallback)); - const value = Number(raw); - if (!Number.isInteger(value) || value < 1) { - throw new Error(`${name} must be a positive integer.`); - } - return value; -} - -const outputPath = resolve( - here, - option("--output", "results/latest-main-agent-lookup.json"), -); -const reportPath = resolve( - here, - option("--report", "results/latest-main-agent-lookup.md"), -); -const selectedCase = option("--case", "all"); -const repetitions = positiveIntegerOption("--repetitions", 2); -const concurrency = positiveIntegerOption("--concurrency", 2); -const tokenizerName = - process.env.CONNECTA_EVAL_TOKENIZER ?? "o200k_base"; -const tokenizer = getEncoding(tokenizerName); -const tokens = (value) => - tokenizer.encode(JSON.stringify(value) ?? "null").length; -const textTokens = (value) => tokenizer.encode(value).length; -const agentModel = process.env.CONNECTA_EVAL_AGENT_MODEL; -const bearer = "connecta-agent-lookup-eval-token"; -const disabledHostFeatures = [ - "apps", - "plugins", - "browser_use", - "computer_use", - "in_app_browser", - "image_generation", - "multi_agent", - "goals", - "tool_suggest", - "skill_search", - "shell_snapshot", - "shell_tool", - "unified_exec", - "workspace_dependencies", -]; -const sourceCommit = execFileSync("git", ["rev-parse", "HEAD"], { - cwd: root, - encoding: "utf8", -}).trim(); -const productDirty = - execFileSync( - "git", - [ - "status", - "--porcelain", - "--", - "src", - "package.json", - "package-lock.json", - ], - { cwd: root, encoding: "utf8" }, - ).trim() !== ""; -const discoveryCorpusText = await readFile( - resolve(here, "discovery-holdout.json"), - "utf8", -); -const discoveryCorpus = JSON.parse(discoveryCorpusText); -const developmentCorpusText = await readFile( - resolve(here, "discovery-development.json"), - "utf8", -); -const harnessSha256 = sha256( - await readFile(fileURLToPath(import.meta.url), "utf8"), -); -const corpusSha256 = sha256( - `${discoveryCorpusText}\0${developmentCorpusText}`, -); -const sandboxSha256 = sha256( - await readFile(resolve(here, "sandbox-server.ts"), "utf8"), -); -const knownConnectorIds = new Set([ - ...discoveryCorpus.connectors.map((connector) => connector.id), - "controlled", - "oauth-recoverable", - "oauth-unavailable", - "static-recoverable", - "static-unavailable", - "analytics", -]); - -function parseJson(text) { - try { - return JSON.parse(text); - } catch { - return undefined; - } -} - -function sha256(text) { - return createHash("sha256").update(text).digest("hex"); -} - -function containsMatchingObject(value, expected) { - if (Array.isArray(value)) { - return value.some((entry) => containsMatchingObject(entry, expected)); - } - if (!value || typeof value !== "object") return false; - const matches = Object.entries(expected).every( - ([key, expectedValue]) => - key in value && isDeepStrictEqual(value[key], expectedValue), - ); - return ( - matches || - Object.values(value).some((entry) => - containsMatchingObject(entry, expected), - ) - ); -} - -function pressurePrompt(currentTask) { - const actions = [ - "list project issues and pull requests", - "send a message and inspect a thread", - "download, upload, share, and move files", - "create and update calendar events", - "search contacts and list sales deals", - "find and update document pages", - "rerun builds and retrieve job logs", - "create invoices and inspect account balances", - ]; - const background = Array.from({ length: 128 }, (_, index) => { - const action = actions[index % actions.length]; - return `Resolved background item ${String(index + 1).padStart(3, "0")} — archive only, do not act: a prior user asked to ${action}. Its outcome is complete and irrelevant to the current task.`; - }).join("\n"); - return `You are continuing after a long integration-triage transcript. - -Everything in RESOLVED BACKGROUND is historical context. Do not execute any of -it and do not let its repeated tool vocabulary replace the current task. - -RESOLVED BACKGROUND -${background} -END RESOLVED BACKGROUND - -CURRENT TASK -${currentTask}`; -} - -const lookupCases = [ - { - id: "mixed-decoy-organizations", - family: "mixed-all-partial", - context: "clean", - developmentCorpus: true, - prompt: - "Using only Connecta, search for `list organizations projects`, then list organizations for analytics project `mxp_eval_42`. Return only the connector JSON.", - expectedAddresses: ["analytics.List-Organizations"], - expectedCalls: [ - { - address: "analytics.List-Organizations", - args: { projectId: "mxp_eval_42" }, - }, - ], - expectedFinalFacts: [ - { - projectId: "mxp_eval_42", - organizations: [ - { id: "org_eval_7", name: "Evaluation Organization" }, - ], - }, - ], - expectedRoute: "call_tool", - expectedOuterRoute: ["search_tools", "call_tool"], - }, - { - id: "open-issues-clean", - family: "open-issues", - context: "clean", - prompt: - "Use the available integrations once to list the open project issues. Return only the integration's JSON result.", - expectedAddresses: ["projects.list_issues"], - expectedCalls: [ - { address: "projects.list_issues", args: { state: "open" } }, - ], - expectedFinalFacts: [ - { state: "open" }, - { - number: 213, - title: "Measure agent routing overhead", - state: "open", - }, - ], - expectedRoute: "call_tool", - expectedOuterRoute: ["search_tools", "call_tool"], - }, - { - id: "page-search-clean", - family: "page-search", - context: "clean", - prompt: - "Use the available integrations once to find the page that mentions our launch plan. Return only the integration's JSON result.", - expectedAddresses: ["documents.search_content"], - expectedCalls: [ - { - address: "documents.search_content", - args: { query: "launch plan" }, - }, - ], - expectedFinalFacts: [ - { - id: "page-launch-plan", - title: "Launch plan", - }, - ], - expectedRoute: "call_tool", - expectedOuterRoute: ["search_tools", "call_tool"], - }, - { - id: "page-search-pressure", - family: "page-search", - context: "pressure", - prompt: pressurePrompt( - "Use the available integrations once to find the page that mentions our launch plan. Return only the integration's JSON result.", - ), - expectedAddresses: ["documents.search_content"], - expectedCalls: [ - { - address: "documents.search_content", - args: { query: "launch plan" }, - }, - ], - expectedFinalFacts: [ - { - id: "page-launch-plan", - title: "Launch plan", - }, - ], - expectedRoute: "call_tool", - expectedOuterRoute: ["search_tools", "call_tool"], - }, - { - id: "workflow-by-id-clean", - family: "workflow-by-id", - context: "clean", - // The prompt asks for two named fields, so the expectation is those two - // fields. It previously asked for "the integration's JSON result" while - // naming a projection of it — an agent could not satisfy both readings, and - // scoring the whole payload made a prompt-wording defect look like a - // routing failure. Discriminating facts, not payload fidelity, is what the - // rest of this suite checks (see page-search). - prompt: - "Use the available integrations once to get the status and conclusion of workflow run 42. Return only JSON with that status and conclusion.", - expectedAddresses: ["builds.get_workflow_run"], - expectedCalls: [ - { address: "builds.get_workflow_run", args: { runId: 42 } }, - ], - expectedFinalFacts: [ - { - status: "completed", - conclusion: "failure", - }, - ], - expectedRoute: "call_tool", - expectedOuterRoute: ["search_tools", "call_tool"], - }, - { - id: "build-diagnosis-clean", - family: "build-diagnosis", - context: "clean", - prompt: - "Use the available integrations to get workflow run 42, then get the logs for the failed job id returned by that run. Return only a JSON array containing the two integration results.", - expectedAddresses: [ - "builds.get_workflow_run", - "builds.get_job_logs", - ], - expectedCalls: [ - { address: "builds.get_workflow_run", args: { runId: 42 } }, - { address: "builds.get_job_logs", args: { jobId: 4207 } }, - ], - expectedFinalFacts: [ - { - runId: 42, - status: "completed", - conclusion: "failure", - failedJobId: 4207, - }, - { - jobId: 4207, - runId: 42, - }, - ], - expectedRoute: "execute_code", - expectedOuterRoute: ["execute_code"], - }, - { - id: "unsupported-audio-pressure", - family: "unsupported-audio", - context: "pressure", - prompt: pressurePrompt( - 'Determine whether an available integration can transcribe an audio recording. Do not pretend one exists. Return only {"available":false} when none does.', - ), - expectedAddresses: [], - expectedCalls: [], - expectedRoute: "search_only", - expectedOuterRoute: ["search_tools"], - finalCorrect(_fixture, finalText) { - return parseJson(finalText)?.available === false; - }, - }, -]; - -function startServer(fixture) { - const child = spawn( - process.execPath, - ["--import", "tsx", "sandbox-server.ts"], - { - cwd: here, - env: { - ...process.env, - CONNECTA_EVAL_PORT: "0", - CONNECTA_EVAL_TOKEN: bearer, - CONNECTA_EVAL_SOURCE_COMMIT: sourceCommit, - CONNECTA_EVAL_TRACE: "enabled", - ...(fixture.developmentCorpus - ? { - CONNECTA_EVAL_DEVELOPMENT_CORPUS: "enabled", - CONNECTA_EVAL_DISCOVERY_ONLY: "enabled", - } - : {}), - }, - stdio: ["ignore", "pipe", "pipe"], - }, - ); - let stderr = ""; - child.stderr.setEncoding("utf8"); - child.stderr.on("data", (chunk) => { - stderr += chunk; - }); - child.stdout.setEncoding("utf8"); - let buffered = ""; - const ready = new Promise((resolveReady, rejectReady) => { - const timeout = setTimeout(() => { - rejectReady(new Error(`Agent lookup server timed out.\n${stderr}`)); - }, 30_000); - child.once("error", (error) => { - clearTimeout(timeout); - rejectReady(error); - }); - child.once("exit", (code) => { - clearTimeout(timeout); - rejectReady( - new Error( - `Agent lookup server exited before readiness (${code}).\n${stderr}`, - ), - ); - }); - child.stdout.on("data", (chunk) => { - buffered += chunk; - for (;;) { - const newline = buffered.indexOf("\n"); - if (newline < 0) break; - const line = buffered.slice(0, newline); - buffered = buffered.slice(newline + 1); - const message = parseJson(line); - if (message?.event === "eval_trace") continue; - if (message?.event === "ready") { - clearTimeout(timeout); - resolveReady(message); - } - } - }); - }); - return { child, ready }; -} - -async function stopServer(child) { - if (child.exitCode !== null) return; - child.kill("SIGTERM"); - await new Promise((resolveExit) => { - const timeout = setTimeout(() => { - child.kill("SIGKILL"); - resolveExit(); - }, 10_000); - child.once("exit", () => { - clearTimeout(timeout); - resolveExit(); - }); - }); -} - -async function readServerTraces(mcpUrl) { - const traceUrl = new URL(mcpUrl); - traceUrl.pathname = "/__eval/trace"; - traceUrl.search = ""; - const response = await fetch(traceUrl, { - headers: { Authorization: `Bearer ${bearer}` }, - }); - if (!response.ok) { - throw new Error( - `Eval trace read failed with HTTP ${response.status}.`, - ); - } - const body = await response.json(); - if (!Array.isArray(body?.traces)) { - throw new Error("Eval trace response did not contain a traces array."); - } - return body.traces; -} - -function structuredAgentResult(result) { - if (!result || typeof result !== "object") return undefined; - if (result.structured_content !== undefined) { - return result.structured_content; - } - if (result.structuredContent !== undefined) return result.structuredContent; - const text = result.content?.find((item) => item.type === "text")?.text; - return typeof text === "string" ? parseJson(text) : undefined; -} - -function addressesFromSearch(value) { - if (Array.isArray(value?.tools)) { - return value.tools - .map((tool) => tool.address) - .filter((address) => typeof address === "string"); - } - if (Array.isArray(value?.connectors)) { - return value.connectors.flatMap((connector) => - Array.isArray(connector.tools) - ? connector.tools - .map((tool) => tool.address) - .filter((address) => typeof address === "string") - : [], - ); - } - return []; -} - -function filteredSearchValue(value, relevantSet) { - if (!value || typeof value !== "object" || Array.isArray(value)) { - return value; - } - if (Array.isArray(value.tools)) { - const tools = value.tools.filter((tool) => - relevantSet.has(tool.address), - ); - return { - ...value, - tools, - total: tools.length, - offset: 0, - hasMore: false, - ...("nextOffset" in value ? { nextOffset: undefined } : {}), - }; - } - const connectors = Array.isArray(value.connectors) - ? value.connectors.flatMap((connector) => { - const tools = Array.isArray(connector.tools) - ? connector.tools.filter((tool) => relevantSet.has(tool.address)) - : []; - return tools.length > 0 ? [{ ...connector, tools }] : []; - }) - : []; - const selected = connectors.reduce( - (sum, connector) => sum + connector.tools.length, - 0, - ); - return { - ...value, - connectors, - total: selected, - offset: 0, - hasMore: false, - ...("nextOffset" in value ? { nextOffset: undefined } : {}), - }; -} - -function traceResultValue(trace) { - return trace.source === "outer" - ? structuredAgentResult(trace.result) - : trace.result; -} - -function searchTrace(trace, fixture) { - const resultTokens = tokens(trace.result ?? null); - const value = traceResultValue(trace); - const addresses = addressesFromSearch(value); - const relevantSet = new Set(fixture.expectedAddresses); - const relevantRanks = fixture.expectedAddresses.map((address) => { - const index = addresses.indexOf(address); - return index < 0 ? null : index + 1; - }); - const relevantReturned = addresses.filter((address) => - relevantSet.has(address), - ).length; - const filtered = filteredSearchValue(value, relevantSet); - const minimalRelevantResultTokens = - trace.source === "outer" - ? tokens(filteredAgentResult(trace.result, filtered)) - : tokens(filtered); - return { - source: trace.source, - query: trace.arguments?.query ?? "", - connector: trace.arguments?.connector ?? null, - includeSchemas: trace.arguments?.includeSchemas ?? null, - matchMode: value?.matchMode ?? "all", - returned: addresses.length, - total: - typeof value?.total === "number" - ? value.total - : addresses.length, - addresses, - relevantRanks, - relevantReturned, - top1Relevant: - addresses.length > 0 && relevantSet.has(addresses[0]), - precision: - addresses.length === 0 - ? fixture.expectedAddresses.length === 0 - ? 1 - : 0 - : round(relevantReturned / addresses.length, 3), - irrelevantCandidates: addresses.length - relevantReturned, - resultTokens, - minimalRelevantResultTokens, - estimatedNoiseTokens: Math.max( - 0, - resultTokens - minimalRelevantResultTokens, - ), - }; -} - -function filteredAgentResult(result, filtered) { - if (!result || typeof result !== "object") return result; - const copy = { ...result }; - if ("structured_content" in copy) copy.structured_content = filtered; - if ("structuredContent" in copy) copy.structuredContent = filtered; - if (Array.isArray(copy.content)) { - copy.content = copy.content.map((item) => - item?.type === "text" - ? { ...item, text: JSON.stringify(filtered) } - : item, - ); - } - return copy; -} - -function defaultFinalCorrect(fixture, finalText) { - const parsed = parseJson(finalText); - return fixture.expectedFinalFacts.every((expected) => - containsMatchingObject(parsed, expected), - ); -} - -function tracedExecutions(metaToolTraces) { - return metaToolTraces.flatMap((trace) => { - if (trace.operation === "call_tool") { - return typeof trace.arguments?.address === "string" - ? [ - { - address: trace.arguments.address, - args: trace.arguments.args ?? {}, - source: trace.source, - }, - ] - : []; - } - if ( - trace.operation === "batch_call" && - Array.isArray(trace.arguments?.calls) - ) { - return trace.arguments.calls.flatMap((call) => - typeof call?.address === "string" - ? [ - { - address: call.address, - args: call.args ?? {}, - source: trace.source, - }, - ] - : [], - ); - } - return []; - }); -} - -function argumentsAccurate(fixture, executions) { - if (fixture.expectedCalls.length !== executions.length) return false; - const remaining = [...executions]; - for (const expected of fixture.expectedCalls) { - const match = remaining.findIndex( - (execution) => - execution.address === expected.address && - isDeepStrictEqual(execution.args, expected.args), - ); - if (match < 0) return false; - remaining.splice(match, 1); - } - return remaining.length === 0; -} - -function connectaRouteCorrect(fixture, metaToolTraces) { - const outer = metaToolTraces - .filter((trace) => trace.source === "outer") - .map((trace) => trace.operation); - const called = metaToolTraces.map((trace) => trace.operation); - const discoveryCorrect = - called.filter((tool) => tool === "search_tools").length === 1 && - !called.includes("describe_tools") && - !called.includes("list_connectors"); - return ( - discoveryCorrect && - isDeepStrictEqual(outer, fixture.expectedOuterRoute) - ); -} - -async function runAgent(fixture, url, repetition) { - const commandArgs = [ - "exec", - "--json", - "--ephemeral", - "--ignore-user-config", - "--skip-git-repo-check", - "--sandbox", - "read-only", - "--cd", - "/tmp", - "--config", - `mcp_servers.connecta.url="${url}"`, - "--config", - 'mcp_servers.connecta.bearer_token_env_var="CONNECTA_EVAL_TOKEN"', - "--config", - 'approval_policy="never"', - ...disabledHostFeatures.flatMap((feature) => [ - "--disable", - feature, - ]), - ...(agentModel ? ["--model", agentModel] : []), - fixture.prompt, - ]; - const started = performance.now(); - const child = spawn("codex", commandArgs, { - cwd: root, - env: { - ...process.env, - CONNECTA_EVAL_TOKEN: bearer, - }, - stdio: ["ignore", "pipe", "pipe"], - }); - child.stdout.setEncoding("utf8"); - child.stderr.setEncoding("utf8"); - let stderr = ""; - let buffered = ""; - let finalText = ""; - let usage = {}; - const toolCalls = []; - const nonMcpActions = []; - const startedItems = new Map(); - child.stderr.on("data", (chunk) => { - stderr += chunk; - }); - child.stdout.on("data", (chunk) => { - buffered += chunk; - for (;;) { - const newline = buffered.indexOf("\n"); - if (newline < 0) break; - const line = buffered.slice(0, newline); - buffered = buffered.slice(newline + 1); - const event = parseJson(line); - if (!event) continue; - if (event.type === "item.started") { - startedItems.set(event.item?.id, performance.now()); - } - if (event.type === "item.completed") { - const item = event.item ?? {}; - const itemStarted = startedItems.get(item.id); - if (item.type === "mcp_tool_call") { - const resultTokens = tokens(item.result ?? null); - const call = { - server: item.server ?? null, - tool: item.tool, - arguments: item.arguments, - status: item.status, - error: item.error ?? null, - durationMs: - itemStarted === undefined - ? null - : round(performance.now() - itemStarted, 1), - resultBytes: Buffer.byteLength( - JSON.stringify(item.result ?? null), - ), - resultTokens, - }; - if (item.tool === "search_tools") { - const value = structuredAgentResult(item.result); - const addresses = addressesFromSearch(value); - const relevantSet = new Set(fixture.expectedAddresses); - const relevantRanks = fixture.expectedAddresses.map((address) => { - const index = addresses.indexOf(address); - return index < 0 ? null : index + 1; - }); - const relevantReturned = addresses.filter((address) => - relevantSet.has(address), - ).length; - const filtered = filteredSearchValue(value, relevantSet); - const minimalRelevantResultTokens = tokens( - filteredAgentResult(item.result, filtered), - ); - call.search = { - query: item.arguments?.query ?? "", - connector: item.arguments?.connector ?? null, - includeSchemas: item.arguments?.includeSchemas ?? null, - matchMode: value?.matchMode ?? "all", - returned: addresses.length, - total: - typeof value?.total === "number" - ? value.total - : addresses.length, - addresses, - relevantRanks, - relevantReturned, - top1Relevant: - addresses.length > 0 && relevantSet.has(addresses[0]), - precision: - addresses.length === 0 - ? fixture.expectedAddresses.length === 0 - ? 1 - : 0 - : round(relevantReturned / addresses.length, 3), - irrelevantCandidates: addresses.length - relevantReturned, - minimalRelevantResultTokens, - estimatedNoiseTokens: Math.max( - 0, - resultTokens - minimalRelevantResultTokens, - ), - }; - } - toolCalls.push(call); - } else if (item.type === "agent_message") { - finalText = item.text ?? ""; - } else { - nonMcpActions.push({ - type: item.type ?? "unknown", - status: item.status ?? null, - command: - typeof item.command === "string" - ? item.command.slice(0, 500) - : null, - }); - } - } - if (event.type === "turn.completed") usage = event.usage ?? {}; - } - }); - const exitCode = await new Promise((resolveExit, rejectExit) => { - const timeout = setTimeout(() => { - child.kill("SIGKILL"); - rejectExit( - new Error( - `Codex case "${fixture.id}" repetition ${repetition} timed out.`, - ), - ); - }, 240_000); - child.once("error", (error) => { - clearTimeout(timeout); - rejectExit(error); - }); - child.once("exit", (code) => { - clearTimeout(timeout); - resolveExit(code); - }); - }); - if (exitCode !== 0) { - throw new Error( - `Codex exited with ${exitCode} for "${fixture.id}" repetition ${repetition}.\n${stderr}`, - ); - } - const serverTraces = await readServerTraces(url); - - const connectaToolCalls = toolCalls.filter( - (call) => call.server === "connecta", - ); - const foreignToolCalls = toolCalls.filter( - (call) => call.server !== "connecta", - ); - const orderedServerTraces = [...serverTraces] - .sort((a, b) => a.sequence - b.sequence) - .map((trace) => - trace.kind === "meta_tool" - ? { - ...trace, - resultTokens: tokens(trace.result ?? null), - } - : trace, - ); - const metaToolTraces = orderedServerTraces.filter( - (trace) => trace.kind === "meta_tool", - ); - const executionTraces = orderedServerTraces.filter( - (trace) => trace.kind === "execution", - ); - const expected = [...fixture.expectedAddresses].sort(); - const executed = executionTraces - .map((trace) => trace.address) - .sort(); - const addressAccurate = - expected.length === executed.length && - expected.every((address, index) => address === executed[index]); - const executionCalls = tracedExecutions(metaToolTraces); - const argumentCorrect = argumentsAccurate(fixture, executionCalls); - const finalCorrect = (fixture.finalCorrect ?? defaultFinalCorrect)( - fixture, - finalText, - ); - const routingResultCorrect = - addressAccurate && argumentCorrect && finalCorrect; - const searchCalls = metaToolTraces - .filter((trace) => trace.operation === "search_tools") - .map((trace) => searchTrace(trace, fixture)); - const directSearchCalls = searchCalls.filter( - (search) => search.source === "outer", - ); - const nestedSearchCalls = searchCalls.filter( - (search) => search.source === "execute_code", - ); - const firstSearch = searchCalls[0]; - const firstRelevantRanks = firstSearch?.relevantRanks ?? []; - const retrievalTop1 = - fixture.expectedAddresses.length === 0 - ? null - : firstSearch?.top1Relevant === true; - const retrievalRecall = - fixture.expectedAddresses.length === 0 - ? null - : round( - firstRelevantRanks.filter((rank) => rank !== null).length / - fixture.expectedAddresses.length, - 3, - ); - const retrievalMrr = - fixture.expectedAddresses.length === 0 - ? null - : round( - firstRelevantRanks.reduce( - (sum, rank) => sum + (rank === null ? 0 : 1 / rank), - 0, - ) / fixture.expectedAddresses.length, - 3, - ); - const retrievalNegativeClean = - fixture.expectedAddresses.length > 0 - ? null - : firstSearch !== undefined && - firstSearch.returned === 0; - const searchResultTokens = searchCalls.reduce( - (sum, search) => sum + search.resultTokens, - 0, - ); - const nestedSearchResultTokens = nestedSearchCalls.reduce( - (sum, search) => sum + search.resultTokens, - 0, - ); - const searchReturned = searchCalls.reduce( - (sum, search) => sum + search.returned, - 0, - ); - const searchRelevantReturned = searchCalls.reduce( - (sum, search) => sum + search.relevantReturned, - 0, - ); - const searchIrrelevantCandidates = - searchReturned - searchRelevantReturned; - const searchPrecision = - searchCalls.length === 0 - ? null - : searchReturned === 0 - ? fixture.expectedAddresses.length === 0 - ? 1 - : 0 - : round(searchRelevantReturned / searchReturned, 3); - const estimatedLookupNoiseTokens = searchCalls.reduce( - (sum, search) => sum + search.estimatedNoiseTokens, - 0, - ); - const connectaMcpResultTokens = connectaToolCalls.reduce( - (sum, call) => sum + call.resultTokens, - 0, - ); - const foreignMcpResultTokens = foreignToolCalls.reduce( - (sum, call) => sum + call.resultTokens, - 0, - ); - const allMcpResultTokens = - connectaMcpResultTokens + foreignMcpResultTokens; - const inputTokens = usage.input_tokens ?? 0; - const cachedInputTokens = usage.cached_input_tokens ?? 0; - const intendedConnectaRoute = connectaRouteCorrect( - fixture, - metaToolTraces, - ); - const routeClean = - intendedConnectaRoute && - foreignToolCalls.length === 0 && - nonMcpActions.length === 0; - - return { - id: fixture.id, - family: fixture.family, - context: fixture.context, - repetition, - promptSha256: sha256(fixture.prompt), - promptTokens: textTokens(fixture.prompt), - promptPreview: - fixture.context === "pressure" - ? `${fixture.prompt.slice(0, 120)}…\n${fixture.prompt.slice(-220)}` - : fixture.prompt, - expectedAddresses: fixture.expectedAddresses, - expectedCalls: fixture.expectedCalls, - expectedRoute: fixture.expectedRoute, - expectedOuterRoute: fixture.expectedOuterRoute, - latencyMs: round(performance.now() - started, 1), - lookupAccurate: addressAccurate, - addressAccurate, - argumentCorrect, - finalCorrect, - routingResultCorrect, - connectaRouteCorrect: intendedConnectaRoute, - routeClean, - executedAddresses: executed, - executionCalls, - guidanceFetched: metaToolTraces.some( - (trace) => trace.operation === "skills", - ), - foreignToolCalls: foreignToolCalls.length, - foreignTools: foreignToolCalls.map( - (call) => `${call.server ?? "unknown"}.${call.tool}`, - ), - searchCalls: searchCalls.length, - directSearchCalls: directSearchCalls.length, - nestedSearchCalls: nestedSearchCalls.length, - emptySearches: searchCalls.filter( - (search) => search.returned === 0, - ).length, - unknownConnectorFilters: searchCalls.filter( - (search) => - typeof search.connector === "string" && - !knownConnectorIds.has(search.connector), - ).length, - searchReturned, - searchRelevantReturned, - searchIrrelevantCandidates, - searchPrecision, - retrievalTop1, - retrievalRecall, - retrievalMrr, - retrievalNegativeClean, - searchTraces: searchCalls, - serverTraces: orderedServerTraces, - toolCalls, - nonMcpActions, - hostActionCount: nonMcpActions.length, - finalText, - usage, - nonCachedInputTokens: Math.max(0, inputTokens - cachedInputTokens), - connectaMcpResultTokens, - foreignMcpResultTokens, - allMcpResultTokens, - searchResultTokens, - nestedSearchResultTokens, - estimatedLookupNoiseTokens, - connectaRoundTrips: metaToolTraces.filter( - (trace) => trace.source === "outer", - ).length, - }; -} - -const selected = - selectedCase === "all" - ? lookupCases - : lookupCases.filter((fixture) => fixture.id === selectedCase); -if (selected.length === 0) { - throw new Error( - `Unknown --case "${selectedCase}". Choose ${lookupCases - .map((fixture) => fixture.id) - .join(", ")}, or all.`, - ); -} - -const jobs = Array.from({ length: repetitions }, (_, index) => - selected.map((fixture) => ({ - fixture, - repetition: index + 1, - })), -).flat(); -const caseResults = Array.from({ length: jobs.length }); -let nextJob = 0; -async function worker() { - for (;;) { - const index = nextJob; - nextJob += 1; - const job = jobs[index]; - if (!job) return; - process.stderr.write( - `Running lookup case ${job.fixture.id} (${job.repetition}/${repetitions})…\n`, - ); - const server = startServer(job.fixture); - try { - const ready = await server.ready; - caseResults[index] = await runAgent( - job.fixture, - ready.url, - job.repetition, - ); - } finally { - await stopServer(server.child); - } - } -} -await Promise.all( - Array.from({ length: Math.min(concurrency, jobs.length) }, () => worker()), -); - -function mean(items, select) { - return items.length === 0 - ? 0 - : items.reduce((sum, item) => sum + select(item), 0) / items.length; -} - -const byCase = Object.fromEntries( - selected.map((fixture) => { - const runs = caseResults.filter((entry) => entry.id === fixture.id); - const precisionRuns = runs.filter( - (entry) => entry.searchPrecision !== null, - ); - const retrievalRuns = runs.filter( - (entry) => entry.retrievalRecall !== null, - ); - const negativeRuns = runs.filter( - (entry) => entry.retrievalNegativeClean !== null, - ); - return [ - fixture.id, - { - runs: runs.length, - lookupAccuracy: round( - runs.filter((entry) => entry.lookupAccurate).length / runs.length, - 3, - ), - routingResultAccuracy: round( - runs.filter((entry) => entry.routingResultCorrect).length / - runs.length, - 3, - ), - addressAccuracy: round( - runs.filter((entry) => entry.addressAccurate).length / runs.length, - 3, - ), - argumentAccuracy: round( - runs.filter((entry) => entry.argumentCorrect).length / runs.length, - 3, - ), - finalAccuracy: round( - runs.filter((entry) => entry.finalCorrect).length / runs.length, - 3, - ), - routeAccuracy: round( - runs.filter((entry) => entry.routeClean).length / runs.length, - 3, - ), - connectaRouteAccuracy: round( - runs.filter((entry) => entry.connectaRouteCorrect).length / - runs.length, - 3, - ), - meanSearchPrecision: - precisionRuns.length === 0 - ? null - : round( - mean(precisionRuns, (entry) => entry.searchPrecision), - 3, - ), - retrievalTop1Accuracy: - retrievalRuns.length === 0 - ? null - : round( - retrievalRuns.filter( - (entry) => entry.retrievalTop1, - ).length / retrievalRuns.length, - 3, - ), - meanRetrievalRecall: - retrievalRuns.length === 0 - ? null - : round( - mean( - retrievalRuns, - (entry) => entry.retrievalRecall, - ), - 3, - ), - meanRetrievalMrr: - retrievalRuns.length === 0 - ? null - : round( - mean( - retrievalRuns, - (entry) => entry.retrievalMrr, - ), - 3, - ), - retrievalNegativeCleanRate: - negativeRuns.length === 0 - ? null - : round( - negativeRuns.filter( - (entry) => entry.retrievalNegativeClean, - ).length / negativeRuns.length, - 3, - ), - meanIrrelevantCandidates: round( - mean(runs, (entry) => entry.searchIrrelevantCandidates), - 1, - ), - meanLookupAttempts: round( - mean(runs, (entry) => entry.searchCalls), - 1, - ), - meanNestedLookupAttempts: round( - mean(runs, (entry) => entry.nestedSearchCalls), - 1, - ), - meanConnectaRoundTrips: round( - mean(runs, (entry) => entry.connectaRoundTrips), - 1, - ), - unknownConnectorFilterRate: round( - runs.filter((entry) => entry.unknownConnectorFilters > 0).length / - runs.length, - 3, - ), - meanSearchResultTokens: round( - mean(runs, (entry) => entry.searchResultTokens), - 1, - ), - meanNestedSearchResultTokens: round( - mean(runs, (entry) => entry.nestedSearchResultTokens), - 1, - ), - meanEstimatedLookupNoiseTokens: round( - mean(runs, (entry) => entry.estimatedLookupNoiseTokens), - 1, - ), - meanMcpResultTokens: round( - mean(runs, (entry) => entry.connectaMcpResultTokens), - 1, - ), - meanForeignMcpResultTokens: round( - mean(runs, (entry) => entry.foreignMcpResultTokens), - 1, - ), - meanInputTokens: round( - mean(runs, (entry) => entry.usage.input_tokens ?? 0), - 1, - ), - meanNonCachedInputTokens: round( - mean(runs, (entry) => entry.nonCachedInputTokens), - 1, - ), - meanLatencyMs: round(mean(runs, (entry) => entry.latencyMs), 1), - guidanceFetchRate: round( - runs.filter((entry) => entry.guidanceFetched).length / runs.length, - 3, - ), - hostActionRate: round( - runs.filter((entry) => entry.hostActionCount > 0).length / - runs.length, - 3, - ), - foreignToolRate: round( - runs.filter((entry) => entry.foreignToolCalls > 0).length / - runs.length, - 3, - ), - }, - ]; - }), -); -const positiveRetrievalRuns = caseResults.filter( - (entry) => entry.retrievalRecall !== null, -); -const negativeRetrievalRuns = caseResults.filter( - (entry) => entry.retrievalNegativeClean !== null, -); - -const result = { - schemaVersion: 2, - generatedAt: new Date().toISOString(), - source: { - commit: sourceCommit, - productDirty, - nodeVersion: process.versions.node, - platform: `${process.platform}-${process.arch}`, - codexVersion: execFileSync("codex", ["--version"], { - encoding: "utf8", - }).trim(), - model: agentModel ?? "codex-default", - tokenizer: tokenizerName, - }, - configuration: { - repetitions, - concurrency, - selectedCase, - jobOrder: "repetition-major with clean/pressure pairs adjacent", - harnessSha256, - corpusSha256, - sandboxSha256, - disabledHostFeatures, - isolation: - "Fresh Connecta server and ephemeral Codex session for every run; user config ignored; host apps/plugins/browser/computer/agent features disabled; read-only filesystem sandbox.", - noiseEstimate: - "Search-result tokens minus a reconstructed result retaining only expected candidates and the same MCP content/structured-content shape.", - trace: - "Per-run eval-server trace captures outer MCP meta-tool calls, nested execute_code provider operations, and payload-free downstream activity. Search metrics include outer and nested discovery.", - }, - summary: { - runs: caseResults.length, - lookupAccurate: caseResults.filter((entry) => entry.lookupAccurate) - .length, - routingResultCorrect: caseResults.filter( - (entry) => entry.routingResultCorrect, - ).length, - addressAccurate: caseResults.filter((entry) => entry.addressAccurate) - .length, - argumentCorrect: caseResults.filter((entry) => entry.argumentCorrect) - .length, - finalCorrect: caseResults.filter((entry) => entry.finalCorrect) - .length, - connectaRouteCorrect: caseResults.filter( - (entry) => entry.connectaRouteCorrect, - ).length, - routeClean: caseResults.filter((entry) => entry.routeClean).length, - retrievalTop1Accuracy: - positiveRetrievalRuns.length === 0 - ? null - : round( - positiveRetrievalRuns.filter( - (entry) => entry.retrievalTop1, - ).length / positiveRetrievalRuns.length, - 3, - ), - meanRetrievalRecall: - positiveRetrievalRuns.length === 0 - ? null - : round( - mean( - positiveRetrievalRuns, - (entry) => entry.retrievalRecall, - ), - 3, - ), - meanRetrievalMrr: - positiveRetrievalRuns.length === 0 - ? null - : round( - mean( - positiveRetrievalRuns, - (entry) => entry.retrievalMrr, - ), - 3, - ), - retrievalNegativeCleanRate: - negativeRetrievalRuns.length === 0 - ? null - : round( - negativeRetrievalRuns.filter( - (entry) => entry.retrievalNegativeClean, - ).length / negativeRetrievalRuns.length, - 3, - ), - totalSearchResultTokens: caseResults.reduce( - (sum, entry) => sum + entry.searchResultTokens, - 0, - ), - totalNestedSearchCalls: caseResults.reduce( - (sum, entry) => sum + entry.nestedSearchCalls, - 0, - ), - totalNestedSearchResultTokens: caseResults.reduce( - (sum, entry) => sum + entry.nestedSearchResultTokens, - 0, - ), - totalEstimatedLookupNoiseTokens: caseResults.reduce( - (sum, entry) => sum + entry.estimatedLookupNoiseTokens, - 0, - ), - totalConnectaMcpResultTokens: caseResults.reduce( - (sum, entry) => sum + entry.connectaMcpResultTokens, - 0, - ), - totalForeignMcpResultTokens: caseResults.reduce( - (sum, entry) => sum + entry.foreignMcpResultTokens, - 0, - ), - totalMcpResultTokens: caseResults.reduce( - (sum, entry) => sum + entry.allMcpResultTokens, - 0, - ), - totalInputTokens: caseResults.reduce( - (sum, entry) => sum + (entry.usage.input_tokens ?? 0), - 0, - ), - totalNonCachedInputTokens: caseResults.reduce( - (sum, entry) => sum + entry.nonCachedInputTokens, - 0, - ), - totalOutputTokens: caseResults.reduce( - (sum, entry) => sum + (entry.usage.output_tokens ?? 0), - 0, - ), - totalLatencyMs: round( - caseResults.reduce((sum, entry) => sum + entry.latencyMs, 0), - 1, - ), - totalConnectaRoundTrips: caseResults.reduce( - (sum, entry) => sum + entry.connectaRoundTrips, - 0, - ), - }, - byCase, - cases: caseResults, -}; - -function yesNo(value) { - return value ? "yes" : "NO"; -} - -function metric(value) { - return value === null ? "—" : value; -} - -const rows = caseResults - .map((entry) => { - const route = entry.toolCalls - .map((call) => `${call.server ?? "unknown"}.${call.tool}`) - .join(" → "); - return `| ${entry.id} #${entry.repetition} | ${yesNo(entry.addressAccurate)} | ${yesNo(entry.argumentCorrect)} | ${yesNo(entry.finalCorrect)} | ${yesNo(entry.connectaRouteCorrect)} | ${yesNo(entry.routeClean)} | ${metric(entry.retrievalTop1)} | ${metric(entry.retrievalRecall)} | ${metric(entry.searchPrecision)} | ${entry.searchIrrelevantCandidates} | ${entry.searchCalls} | ${entry.nestedSearchCalls} | ${entry.connectaRoundTrips} | ${entry.estimatedLookupNoiseTokens} | ${entry.connectaMcpResultTokens} | ${entry.usage.input_tokens ?? 0} | \`${route}\` |`; - }) - .join("\n"); -const groupedRows = Object.entries(byCase) - .map( - ([id, metrics]) => - `| ${id} | ${(metrics.addressAccuracy * 100).toFixed(0)}% | ${(metrics.argumentAccuracy * 100).toFixed(0)}% | ${(metrics.finalAccuracy * 100).toFixed(0)}% | ${(metrics.connectaRouteAccuracy * 100).toFixed(0)}% | ${(metrics.routeAccuracy * 100).toFixed(0)}% | ${metric(metrics.retrievalTop1Accuracy)} | ${metric(metrics.meanRetrievalRecall)} | ${metric(metrics.meanSearchPrecision)} | ${metrics.meanIrrelevantCandidates} | ${metrics.meanLookupAttempts} | ${metrics.meanNestedLookupAttempts} | ${metrics.meanConnectaRoundTrips} | ${metrics.meanEstimatedLookupNoiseTokens} | ${metrics.meanMcpResultTokens} | ${metrics.meanInputTokens} |`, - ) - .join("\n"); -const report = `# Latest-main agent lookup benchmark - -Generated: ${result.generatedAt} - -Source: \`${sourceCommit}\`; ${result.source.codexVersion}; model ${result.source.model} - -Each run used a fresh isolated server and ephemeral agent. Host apps, plugins, -browser, computer-use, multi-agent, and related discovery features were -explicitly disabled in addition to ignoring user config. Accuracy requires the -agent to execute exactly the expected downstream address set with the expected -arguments and return the deterministic domain result. A server-side trace -attributes both outer MCP operations and discovery/calls nested inside -execute_code. The noise-token figure is the traced serialized search result -minus the same result reconstructed with only the expected candidate rows. - -## Summary - -- Exact tool-address accuracy: ${result.summary.lookupAccurate}/${result.summary.runs} -- Argument accuracy: ${result.summary.argumentCorrect}/${result.summary.runs} -- Final-result accuracy: ${result.summary.finalCorrect}/${result.summary.runs} -- Routing-result agreement: ${result.summary.routingResultCorrect}/${result.summary.runs} -- Intended Connecta route: ${result.summary.connectaRouteCorrect}/${result.summary.runs} -- Clean route (no foreign tool or host actions): ${result.summary.routeClean}/${result.summary.runs} -- Attributed retrieval top-1 accuracy: ${metric(result.summary.retrievalTop1Accuracy)} -- Mean attributed retrieval recall: ${metric(result.summary.meanRetrievalRecall)} -- Mean attributed retrieval MRR: ${metric(result.summary.meanRetrievalMrr)} -- Attributed negative clean rate: ${metric(result.summary.retrievalNegativeCleanRate)} -- Nested search calls: ${result.summary.totalNestedSearchCalls} -- Outer Connecta round trips: ${result.summary.totalConnectaRoundTrips} -- Search-result tokens: ${result.summary.totalSearchResultTokens.toLocaleString()} -- Nested search-result tokens: ${result.summary.totalNestedSearchResultTokens.toLocaleString()} -- Estimated irrelevant lookup tokens: ${result.summary.totalEstimatedLookupNoiseTokens.toLocaleString()} -- Connecta MCP result tokens: ${result.summary.totalConnectaMcpResultTokens.toLocaleString()} -- Foreign MCP result tokens: ${result.summary.totalForeignMcpResultTokens.toLocaleString()} -- All MCP result tokens: ${result.summary.totalMcpResultTokens.toLocaleString()} -- Whole-agent input tokens: ${result.summary.totalInputTokens.toLocaleString()} (${result.summary.totalNonCachedInputTokens.toLocaleString()} non-cached) - -## By case - -| Case | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Whole-agent input tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -${groupedRows} - -## Runs - -| Run | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Agent input tokens | Tool route | -| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | -${rows} - -## Interpretation - -- Retrieval metrics use the first server-traced search, whether it happened at - the outer MCP boundary or inside execute_code. Search precision measures only - returned pages. Nested search tokens describe sandbox work and are kept - separate from outer MCP tokens so host-context accounting is not double - counted. -- Whole-agent input tokens are Codex CLI accounting for the complete host - context, including built-in definitions and cache reads. MCP result tokens - isolate the observed Connecta payloads. -- Pressure cases contain 128 explicitly resolved distractor tasks and put the - current request at the end. They test instruction selection under long, - competing integration vocabulary; they are not a context-window limit test. -- Repetitions expose behavioral variance. This sample remains a canary, not a - statistical release gate. -`; - -await mkdir(dirname(outputPath), { recursive: true }); -await mkdir(dirname(reportPath), { recursive: true }); -await writeFile(outputPath, `${JSON.stringify(result, null, 2)}\n`); -await writeFile(reportPath, report); -tokenizer.free?.(); -process.stdout.write( - `${JSON.stringify({ - event: "agent_lookup_benchmark_complete", - output: outputPath, - report: reportPath, - sourceCommit, - summary: result.summary, - byCase, - })}\n`, -); diff --git a/eval/current-version/audit-all-tools.mjs b/eval/current-version/audit-all-tools.mjs deleted file mode 100644 index 471ec171..00000000 --- a/eval/current-version/audit-all-tools.mjs +++ /dev/null @@ -1,421 +0,0 @@ -import { errorCode, structured, textContent } from "./audit-lib.mjs"; - -function objectValue(result) { - const value = structured(result); - return value && typeof value === "object" && !Array.isArray(value) - ? value - : {}; -} - -function pass(condition, details = {}) { - return { passed: Boolean(condition), ...details }; -} - -function programValue(result) { - return objectValue(result).result; -} - -export async function runTaskAudit(context, { baseUrl, operatorToken }) { - const cases = []; - - async function task(name, tool, args, classify) { - const { result, observation } = await context.call( - name, - tool, - args, - classify, - ); - cases.push(observation); - return result; - } - - await task("list guidance", "skills", {}, (result) => { - return pass( - textContent(result)?.includes("usage") === true, - { outcome: "guidance-listed" }, - ); - }); - await task("fetch routing guidance", "skills", { name: "usage" }, (result) => - pass( - (textContent(result)?.length ?? 0) > 100, - { outcome: "guidance-fetched" }, - ), - ); - await task( - "inventory configured connectors", - "execute_code", - { code: "async () => await connecta.search({})" }, - (result) => { - const tools = programValue(result)?.tools; - return pass( - Array.isArray(tools) && - tools.some((tool) => tool.address === "projects.list_issues"), - { - outcome: "inventory-returned", - capability: "catalog inventory", - route: "connecta.search", - returned: tools?.length ?? 0, - }, - ); - }, - ); - await task( - "focused discovery", - "search_tools", - { - query: "deterministic records", - connector: "controlled", - includeSchemas: "compact", - }, - (result) => { - const groups = objectValue(result).connectors; - const addresses = Array.isArray(groups) - ? groups.flatMap((group) => - Array.isArray(group.tools) - ? group.tools.map((tool) => tool.address) - : [], - ) - : []; - return pass(addresses.includes("controlled.records"), { - outcome: "tool-discovered", - returned: addresses.length, - }); - }, - ); - await task( - "inspect complete schema", - "execute_code", - { - code: - 'async () => await connecta.describe({ addresses: ["controlled.records"], format: "json" })', - }, - (result) => { - const tools = programValue(result)?.tools; - return pass(Array.isArray(tools) && tools.length === 1, { - outcome: "schema-described", - capability: "schema description", - route: "connecta.describe", - }); - }, - ); - await task( - "single read-only call", - "call_tool", - { - address: "controlled.read_record", - args: { id: 7 }, - resultMode: "value", - diagnostics: true, - }, - (result) => - pass(result.isError !== true && objectValue(result).ok === true, { - outcome: "read-succeeded", - }), - ); - await task( - "read-only route refuses destructive tool", - "call_tool", - { - address: "controlled.increment_counter", - args: { amount: 1 }, - resultMode: "value", - }, - (result) => - pass( - objectValue(result).ok === false && - errorCode(result) === "destructive_tool_requires_approval", - { - outcome: "destructive-refused", - errorCode: errorCode(result) ?? null, - }, - ), - ); - await task( - "approved destructive call", - "call_destructive_tool", - { - address: "controlled.increment_counter", - args: { amount: 2 }, - resultMode: "value", - diagnostics: true, - }, - (result) => - pass(result.isError !== true && objectValue(result).ok === true, { - outcome: "destructive-approved", - }), - ); - const truncated = await task( - "create truncated result", - "call_tool", - { - address: "controlled.large_document", - args: { paragraphs: 200 }, - resultMode: "value", - diagnostics: true, - }, - (result) => { - const value = objectValue(result); - return pass( - value.ok === true && - value.data?.truncated === true && - typeof value.data?.resultId === "string" && - value.data?.totalBytes >= 40_000, - { - outcome: "result-truncated", - resultId: value.data?.resultId ?? null, - totalBytes: value.data?.totalBytes ?? null, - }, - ); - }, - ); - const resultId = objectValue(truncated).data?.resultId; - if (typeof resultId !== "string") { - throw new Error("Truncation scenario did not return a result id."); - } - await task( - "page truncated result", - "get_result", - { id: resultId, offset: 0, maxBytes: 300 }, - (result) => { - const value = objectValue(result); - return pass( - typeof value.text === "string" && - value.offset === 0 && - typeof value.nextOffset === "number", - { - outcome: "result-paged", - offset: value.offset ?? null, - nextOffset: value.nextOffset ?? null, - totalBytes: value.totalBytes ?? null, - }, - ); - }, - ); - await task( - "batch independent reads", - "execute_code", - { - code: - "async () => await connecta.batch([" + - '{ address: "controlled.read_record", args: { id: 11 } },' + - '{ address: "controlled.read_record", args: { id: 12 } }' + - "])", - }, - (result) => { - const value = programValue(result); - return pass( - Array.isArray(value) && - value.length === 2 && - value.every((entry) => entry.ok === true), - { - outcome: "batch-succeeded", - capability: "parallel calls", - route: "connecta.batch", - resultCount: value?.length ?? 0, - }, - ); - }, - ); - await task( - "reduce records in code mode", - "execute_code", - { - code: - "async () => { const rows = await controlled.records({ count: 120 }); " + - "return rows.reduce((out, row) => { const group = out[row.group] ??= " + - "{ count: 0, sum: 0 }; group.count++; group.sum += row.score; " + - "return out; }, {}); }", - }, - (result) => - pass(result.isError !== true && "result" in objectValue(result), { - outcome: "code-reduction-succeeded", - }), - ); - - await task( - "OAuth call requires recovery", - "call_tool", - { - address: "oauth-recoverable.whoami", - args: {}, - resultMode: "value", - }, - (result) => - pass(objectValue(result).ok === false && errorCode(result) === "auth_required", { - outcome: "oauth-auth-required", - errorCode: errorCode(result) ?? null, - }), - ); - const oauthStart = await task( - "OAuth recovery starts", - "authorize_connector", - { connector: "oauth-recoverable" }, - (result) => { - const value = objectValue(result); - return pass( - result.isError !== true && - value.recovery === "oauth" && - typeof value.authorizationUrl === "string", - { - outcome: "oauth-operator-handoff", - hasAuthorizationUrl: typeof value.authorizationUrl === "string", - }, - ); - }, - ); - const authorizationUrl = objectValue(oauthStart).authorizationUrl; - if (typeof authorizationUrl !== "string") { - throw new Error("OAuth recovery did not return an authorization URL."); - } - const consent = await fetch(authorizationUrl); - if (!consent.ok) { - throw new Error(`OAuth fixture consent failed with HTTP ${consent.status}.`); - } - await task( - "OAuth retry succeeds", - "call_tool", - { - address: "oauth-recoverable.whoami", - args: {}, - resultMode: "value", - }, - (result) => - pass(result.isError !== true && objectValue(result).ok === true, { - outcome: "oauth-recovered", - recovery: "success", - }), - ); - await task( - "OAuth unavailable recovery is explicit", - "authorize_connector", - { connector: "oauth-unavailable" }, - (result) => - pass(result.isError === true, { - outcome: "oauth-recovery-unavailable", - recovery: "unavailable", - messagePreview: textContent(result)?.slice(0, 160) ?? null, - }), - ); - - await task( - "static credential call requires recovery", - "call_tool", - { - address: "static-recoverable.whoami", - args: {}, - resultMode: "value", - }, - (result) => - pass(objectValue(result).ok === false && errorCode(result) === "auth_required", { - outcome: "static-auth-required", - errorCode: errorCode(result) ?? null, - }), - ); - await task( - "static recovery returns an operator handoff", - "authorize_connector", - { connector: "static-recoverable" }, - (result) => { - const value = objectValue(result); - return pass( - result.isError !== true && - value.recovery === "operator_config" && - typeof value.operatorUrl === "string", - { - outcome: "static-operator-handoff", - recovery: value.recovery ?? null, - hasOperatorUrl: typeof value.operatorUrl === "string", - }, - ); - }, - ); - const configured = await fetch( - `${baseUrl}/ui/credentials/static-recoverable`, - { - method: "PUT", - headers: { - authorization: `Bearer ${operatorToken}`, - origin: baseUrl, - "content-type": "application/json", - }, - body: JSON.stringify({ value: "sandbox-ok" }), - }, - ); - if (!configured.ok) { - throw new Error( - `Static operator fixture failed with HTTP ${configured.status}.`, - ); - } - await task( - "static credential retry succeeds after operator update", - "call_tool", - { - address: "static-recoverable.whoami", - args: {}, - resultMode: "value", - }, - (result) => - pass(result.isError !== true && objectValue(result).ok === true, { - outcome: "static-recovered", - recovery: "success", - }), - ); - await task( - "static unavailable recovery is explicit", - "authorize_connector", - { connector: "static-unavailable" }, - (result) => { - const value = objectValue(result); - return pass( - result.isError !== true && value.recovery === "unavailable", - { - outcome: "static-recovery-unavailable", - recovery: value.recovery ?? null, - messagePreview: textContent(result)?.slice(0, 160) ?? null, - }, - ); - }, - ); - await task( - "activity remains payload-free", - "call_tool", - { - address: "controlled.activity_snapshot", - args: {}, - resultMode: "value", - }, - (result) => { - const value = objectValue(result); - const snapshot = value.data ?? {}; - return pass( - value.ok === true && - Array.isArray(snapshot.forbiddenPresent) && - snapshot.forbiddenPresent.length === 0, - { - outcome: "activity-payload-free", - eventCount: snapshot.eventCount ?? null, - activityKeys: snapshot.keys ?? [], - forbiddenPresent: snapshot.forbiddenPresent ?? null, - }, - ); - }, - ); - - return { - summary: { - caseCount: cases.length, - passed: cases.filter((entry) => entry.passed).length, - failed: cases.filter((entry) => !entry.passed).length, - taskSuccessRate: - cases.length === 0 - ? 0 - : cases.filter((entry) => entry.passed).length / cases.length, - roundTrips: cases.length, - summedLatencyMs: cases.reduce( - (sum, entry) => sum + entry.latencyMs, - 0, - ), - }, - cases, - }; -} diff --git a/eval/current-version/audit-lib.mjs b/eval/current-version/audit-lib.mjs deleted file mode 100644 index 490df62a..00000000 --- a/eval/current-version/audit-lib.mjs +++ /dev/null @@ -1,126 +0,0 @@ -import { Buffer } from "node:buffer"; -import { - Client, - StreamableHTTPClientTransport, -} from "@modelcontextprotocol/client"; -import { getEncoding } from "js-tiktoken"; - -export function serialized(value) { - return JSON.stringify(value) ?? "null"; -} - -export function round(value, places = 1) { - const scale = 10 ** places; - return Math.round(value * scale) / scale; -} - -export function structured(result) { - if (result.structuredContent !== undefined) return result.structuredContent; - const text = result.content?.find((item) => item.type === "text")?.text; - if (typeof text !== "string") return undefined; - try { - return JSON.parse(text); - } catch { - return undefined; - } -} - -export function textContent(result) { - return result.content?.find((item) => item.type === "text")?.text; -} - -export function errorCode(result) { - const value = structured(result); - if ( - value && - typeof value === "object" && - !Array.isArray(value) && - value.error && - typeof value.error === "object" && - typeof value.error.code === "string" - ) { - return value.error.code; - } - return undefined; -} - -export async function createAuditClient({ - url, - token, - tokenizerName, -}) { - const tokenizer = getEncoding(tokenizerName); - const tokens = (value) => tokenizer.encode(serialized(value)).length; - const bytes = (value) => Buffer.byteLength(serialized(value), "utf8"); - const client = new Client({ - name: "connecta-current-version-audit", - version: "1.0.0", - }); - const transport = new StreamableHTTPClientTransport(new URL(url), { - requestInit: { - headers: { Authorization: `Bearer ${token}` }, - }, - }); - - const started = performance.now(); - await client.connect(transport); - const listed = await client.listTools(); - const connection = { - latencyMs: round(performance.now() - started), - toolsListBytes: bytes(listed), - toolsListTokens: tokens(listed), - toolCount: listed.tools.length, - tools: listed.tools.map((tool) => ({ - name: tool.name, - definitionBytes: bytes(tool), - definitionTokens: tokens(tool), - })), - }; - const observations = []; - const advertisedTools = new Set(listed.tools.map((tool) => tool.name)); - - async function call(name, tool, args, classify = () => ({})) { - if (!advertisedTools.has(tool)) { - throw new Error( - `Audit task "${name}" requires top-level tool "${tool}", but the configured surface advertises: ${[...advertisedTools].sort().join(", ")}.`, - ); - } - const params = { name: tool, arguments: args }; - const callStarted = performance.now(); - const result = await client.callTool(params); - const classification = classify(result); - const observation = { - name, - tool, - latencyMs: round(performance.now() - callStarted), - requestBytes: bytes(params), - requestTokens: tokens(params), - responseBytes: bytes(result), - responseTokens: tokens(result), - contentTokens: tokens(result.content ?? null), - structuredContentTokens: tokens(result.structuredContent ?? null), - hasContent: Array.isArray(result.content), - hasStructuredContent: result.structuredContent !== undefined, - isError: result.isError === true, - ...classification, - }; - observations.push(observation); - return { result, observation }; - } - - return { - client, - transport, - tokenizerName, - tokens, - bytes, - listed, - connection, - observations, - call, - async close() { - tokenizer.free?.(); - await transport.close(); - }, - }; -} diff --git a/eval/current-version/benchmark.mjs b/eval/current-version/benchmark.mjs new file mode 100644 index 00000000..5cf63602 --- /dev/null +++ b/eval/current-version/benchmark.mjs @@ -0,0 +1,468 @@ +import { spawn } from "node:child_process"; +import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +import { getEncoding } from "js-tiktoken"; + +const here = dirname(fileURLToPath(import.meta.url)); +const argv = process.argv.slice(2); +const tokenizer = getEncoding("o200k_base"); +const agentModel = process.env.CONNECTA_BENCHMARK_MODEL ?? "gpt-5.6-sol"; +const disabledHostFeatures = [ + "apps", + "plugins", + "browser_use", + "computer_use", + "in_app_browser", + "image_generation", + "multi_agent", + "goals", + "tool_suggest", + "skill_search", + "shell_snapshot", + "shell_tool", + "unified_exec", + "workspace_dependencies", +]; + +function option(name, fallback) { + const index = argv.indexOf(name); + if (index < 0) return fallback; + const value = argv[index + 1]; + if (!value || value.startsWith("--")) throw new Error(`${name} requires a value.`); + return value; +} + +function positiveInteger(name, fallback) { + const value = Number(option(name, String(fallback))); + if (!Number.isInteger(value) || value < 1) { + throw new Error(`${name} must be a positive integer.`); + } + return value; +} + +function parseJson(text) { + try { + return JSON.parse(text); + } catch { + return undefined; + } +} + +function parseAnswer(text) { + const direct = parseJson(text.trim()); + if (direct !== undefined) return direct; + const fenced = text.match(/```(?:json)?\s*([\s\S]*?)```/i); + return fenced ? parseJson(fenced[1].trim()) : undefined; +} + +function round(value) { + return Math.round(value * 10) / 10; +} + +const cases = [ + { + id: "cold-unknown-read", + purpose: "Cold discovery and one read in a large catalog", + expectedRoute: ["execute_code"], + prompt: `Use only the Connecta MCP server. You do not know the connector or tool address yet. Look up the current project named Atlas. Return only JSON with exactly name, status, and owner.`, + score({ answer, downstreamCalls }) { + return { + correct: + answer?.name === "Atlas" && + answer?.status === "on_track" && + answer?.owner === "Rina Shah", + semantics: + downstreamCalls.length === 1 && + downstreamCalls[0]?.address === "projects.list_projects", + private: true, + }; + }, + }, + { + id: "known-address-read", + purpose: "Control: one read whose canonical address is already known", + expectedRoute: ["call_tool"], + prompt: `Use only the Connecta MCP server. The canonical address is projects.list_projects and it takes no arguments. Call it, select Atlas, and return only JSON with exactly name, status, and owner.`, + score({ answer, downstreamCalls }) { + return { + correct: + answer?.name === "Atlas" && + answer?.status === "on_track" && + answer?.owner === "Rina Shah", + semantics: + downstreamCalls.length === 1 && + downstreamCalls[0]?.address === "projects.list_projects", + private: true, + }; + }, + }, + { + id: "semantic-analytics", + purpose: "Exact provider semantics and positional-table validation", + expectedRoute: ["execute_code"], + prompt: `Use only the Connecta MCP server. In the Mixpanel sandbox, find product-wide usage for the project named Pulse over the last 30 days. Follow every connector guide required by discovery, resolve exact identities, and validate any table before reading it. Return only JSON with exactly projectId, eventName, days, and total.`, + score({ answer, downstreamCalls, toolCalls }) { + const query = downstreamCalls.find( + (call) => call.address === "mixpanel.query_event_usage", + ); + const listed = downstreamCalls.some( + (call) => call.address === "mixpanel.list_events", + ); + const guideFetched = toolCalls.some( + (call) => + call.tool === "skills" && + call.arguments?.name === "connector:mixpanel", + ); + return { + correct: + answer?.projectId === 2803261 && + answer?.eventName === "App Open or Present Session" && + answer?.days === 30 && + answer?.total === 334100, + semantics: + guideFetched && + listed && + query?.args?.projectId === 2803261 && + query?.args?.eventName === "App Open or Present Session" && + query?.args?.days === 30, + private: true, + }; + }, + }, + { + id: "private-pagination", + purpose: "Pagination and in-sandbox reduction of private records", + expectedRoute: ["execute_code"], + prompt: `Use only the Connecta MCP server and Stripe sandbox. Count every customer across all pages. Customer records contain private identifiers and emails: do not return, quote, or summarize any record. Return only JSON with exactly customerCount.`, + score({ answer, downstreamCalls, finalText }) { + const pages = downstreamCalls.filter( + (call) => call.address === "stripe-sandbox.list_customers", + ); + return { + correct: answer?.customerCount === 225, + semantics: + pages.length === 3 && + pages[0]?.args?.starting_after === undefined && + pages[1]?.args?.starting_after === "cus_benchmark_100" && + pages[2]?.args?.starting_after === "cus_benchmark_200", + private: + !/@example\.invalid/i.test(finalText) && + !/cus_benchmark_/i.test(finalText), + }; + }, + }, +]; + +function startServer() { + const child = spawn(process.execPath, ["server.mjs"], { + cwd: here, + env: { ...process.env, CONNECTA_BENCHMARK_PORT: "0" }, + stdio: ["ignore", "pipe", "pipe"], + }); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + let stdout = ""; + let stderr = ""; + child.stderr.on("data", (chunk) => (stderr += chunk)); + const ready = new Promise((resolveReady, rejectReady) => { + const timeout = setTimeout( + () => rejectReady(new Error(`Benchmark server timed out.\n${stderr}`)), + 30_000, + ); + child.once("error", rejectReady); + child.once("exit", (code) => { + clearTimeout(timeout); + rejectReady(new Error(`Benchmark server exited early (${code}).\n${stderr}`)); + }); + child.stdout.on("data", (chunk) => { + stdout += chunk; + for (;;) { + const newline = stdout.indexOf("\n"); + if (newline < 0) return; + const message = parseJson(stdout.slice(0, newline)); + stdout = stdout.slice(newline + 1); + if (message?.event !== "ready") continue; + clearTimeout(timeout); + resolveReady(message); + return; + } + }); + }); + return { child, ready }; +} + +async function stopServer(child) { + if (child.exitCode !== null) return; + child.kill("SIGTERM"); + await new Promise((resolveExit) => { + const timeout = setTimeout(() => { + child.kill("SIGKILL"); + resolveExit(); + }, 10_000); + child.once("exit", () => { + clearTimeout(timeout); + resolveExit(); + }); + }); +} + +async function runCodex(fixture, ready) { + const workspace = await mkdtemp(resolve(tmpdir(), "connecta-benchmark-")); + const command = [ + "exec", + "--json", + "--ephemeral", + "--ignore-user-config", + "--skip-git-repo-check", + "--sandbox", + "read-only", + "--cd", + workspace, + "--config", + `mcp_servers.connecta.url="${ready.url}"`, + "--config", + 'mcp_servers.connecta.bearer_token_env_var="CONNECTA_BENCHMARK_TOKEN"', + "--config", + 'approval_policy="never"', + ...disabledHostFeatures.flatMap((feature) => ["--disable", feature]), + "--model", + agentModel, + fixture.prompt, + ]; + const startedAt = performance.now(); + const child = spawn("codex", command, { + cwd: workspace, + env: { ...process.env, CONNECTA_BENCHMARK_TOKEN: ready.token }, + stdio: ["ignore", "pipe", "pipe"], + }); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + let buffer = ""; + let stderr = ""; + let finalText = ""; + let usage = {}; + const toolCalls = []; + const itemStarts = new Map(); + child.stderr.on("data", (chunk) => (stderr += chunk)); + child.stdout.on("data", (chunk) => { + buffer += chunk; + for (;;) { + const newline = buffer.indexOf("\n"); + if (newline < 0) return; + const event = parseJson(buffer.slice(0, newline)); + buffer = buffer.slice(newline + 1); + if (!event) continue; + if (event.type === "item.started") { + itemStarts.set(event.item?.id, performance.now()); + } else if (event.type === "item.completed") { + const item = event.item ?? {}; + if (item.type === "mcp_tool_call" && item.server === "connecta") { + const serialized = JSON.stringify(item.result ?? null); + const itemStarted = itemStarts.get(item.id); + toolCalls.push({ + tool: item.tool, + arguments: item.arguments, + status: item.status, + durationMs: + itemStarted === undefined ? null : round(performance.now() - itemStarted), + resultBytes: Buffer.byteLength(serialized), + resultTokens: tokenizer.encode(serialized).length, + }); + } else if (item.type === "agent_message") { + finalText = item.text ?? ""; + } + } else if (event.type === "turn.completed") { + usage = event.usage ?? {}; + } + } + }); + + const exitCode = await new Promise((resolveExit, rejectExit) => { + const timeout = setTimeout(() => { + child.kill("SIGKILL"); + rejectExit(new Error(`Codex timed out in ${fixture.id}.`)); + }, 180_000); + child.once("error", rejectExit); + child.once("exit", (code) => { + clearTimeout(timeout); + resolveExit(code); + }); + }); + await rm(workspace, { recursive: true, force: true }); + if (exitCode !== 0) { + throw new Error(`Codex exited with ${exitCode} in ${fixture.id}.\n${stderr}`); + } + return { + finalText, + usage, + toolCalls, + latencyMs: round(performance.now() - startedAt), + }; +} + +function forwardingMetrics(outerCalls, toolCalls) { + const results = outerCalls + .map((call) => parseJson(call.responseText)?.result) + .filter((result) => result && typeof result === "object"); + const responseBytes = outerCalls.reduce((sum, call) => sum + call.responseBytes, 0); + const modelResultBytes = toolCalls.reduce((sum, call) => sum + call.resultBytes, 0); + return { + outerResponseBytes: responseBytes, + contentBytes: results.reduce( + (sum, result) => sum + Buffer.byteLength(JSON.stringify(result.content ?? null)), + 0, + ), + structuredContentBytes: results.reduce( + (sum, result) => + sum + Buffer.byteLength(JSON.stringify(result.structuredContent ?? null)), + 0, + ), + modelResultBytes, + modelResultTokens: toolCalls.reduce((sum, call) => sum + call.resultTokens, 0), + duplicatedCalls: results.filter( + (result) => result.content !== undefined && result.structuredContent !== undefined, + ).length, + representationDuplicated: results.some( + (result) => result.content !== undefined && result.structuredContent !== undefined, + ), + }; +} + +async function runOnce(fixture, repetition) { + const server = startServer(); + const ready = await server.ready; + try { + const agent = await runCodex(fixture, ready); + const stateResponse = await fetch(ready.stateUrl); + if (!stateResponse.ok) throw new Error(`State read failed: ${stateResponse.status}`); + const state = await stateResponse.json(); + const answer = parseAnswer(agent.finalText); + const route = agent.toolCalls + .filter((call) => call.tool !== "skills") + .map((call) => call.tool); + const routeCorrect = JSON.stringify(route) === JSON.stringify(fixture.expectedRoute); + const checks = fixture.score({ + answer, + finalText: agent.finalText, + downstreamCalls: state.downstreamCalls, + toolCalls: agent.toolCalls, + }); + return { + case: fixture.id, + repetition, + pass: routeCorrect && checks.correct && checks.semantics && checks.private, + route, + expectedRoute: fixture.expectedRoute, + checks: { route: routeCorrect, ...checks }, + usage: agent.usage, + latencyMs: agent.latencyMs, + forwarding: forwardingMetrics(state.outerCalls, agent.toolCalls), + toolCalls: agent.toolCalls, + downstreamCalls: state.downstreamCalls, + finalText: agent.finalText, + }; + } finally { + await stopServer(server.child); + } +} + +function summarize(runs) { + const totals = runs.reduce( + (sum, run) => ({ + passed: sum.passed + Number(run.pass), + inputTokens: sum.inputTokens + (run.usage.input_tokens ?? 0), + cachedInputTokens: + sum.cachedInputTokens + (run.usage.cached_input_tokens ?? 0), + outputTokens: sum.outputTokens + (run.usage.output_tokens ?? 0), + resultTokens: sum.resultTokens + run.forwarding.modelResultTokens, + latencyMs: sum.latencyMs + run.latencyMs, + }), + { passed: 0, inputTokens: 0, cachedInputTokens: 0, outputTokens: 0, resultTokens: 0, latencyMs: 0 }, + ); + return { + runs: runs.length, + ...totals, + passRate: runs.length ? totals.passed / runs.length : 0, + averageLatencyMs: runs.length ? round(totals.latencyMs / runs.length) : 0, + }; +} + +function markdown(report) { + const lines = [ + "# Current-version benchmark", + "", + `Generated ${report.generatedAt} with Codex. ${report.summary.passed}/${report.summary.runs} runs passed.`, + "", + "| Case | Pass | Route | Input | Cached | Output | MCP result | Latency |", + "| --- | ---: | --- | ---: | ---: | ---: | ---: | ---: |", + ]; + for (const run of report.runs) { + lines.push( + `| ${run.case} #${run.repetition} | ${run.pass ? "yes" : "no"} | ${run.route.join(" → ") || "none"} | ${run.usage.input_tokens ?? 0} | ${run.usage.cached_input_tokens ?? 0} | ${run.usage.output_tokens ?? 0} | ${run.forwarding.modelResultTokens} | ${run.latencyMs} ms |`, + ); + } + lines.push( + "", + "A run passes only when route, answer, provider semantics, and privacy checks all pass. Raw JSON carries tool calls, downstream arguments, and forwarding bytes.", + "", + ); + return lines.join("\n"); +} + +function selfTest() { + const parsed = parseAnswer("```json\n{\"customerCount\":75}\n```"); + if (parsed?.customerCount !== 75) throw new Error("Fenced answer parsing failed."); + const metrics = forwardingMetrics( + [{ + responseBytes: 60, + responseText: '{"jsonrpc":"2.0","result":{"content":[],"structuredContent":{}}}', + }], + [{ resultBytes: 20, resultTokens: 7 }], + ); + if ( + !metrics.representationDuplicated || + metrics.duplicatedCalls !== 1 || + metrics.contentBytes !== 2 || + metrics.structuredContentBytes !== 2 || + metrics.modelResultTokens !== 7 + ) { + throw new Error("Forwarding metric self-test failed."); + } + console.log("benchmark self-test passed"); +} + +if (argv.includes("--self-test")) { + selfTest(); +} else { + const selected = option("--case", "all"); + const selectedCases = + selected === "all" ? cases : cases.filter((fixture) => fixture.id === selected); + if (selectedCases.length === 0) throw new Error(`Unknown case: ${selected}`); + const repetitions = positiveInteger("--repetitions", 3); + const output = resolve(here, option("--output", "results/latest.json")); + const runs = []; + for (const fixture of selectedCases) { + for (let repetition = 1; repetition <= repetitions; repetition += 1) { + process.stderr.write(`running ${fixture.id} #${repetition}\n`); + runs.push(await runOnce(fixture, repetition)); + } + } + const report = { + schemaVersion: 1, + generatedAt: new Date().toISOString(), + agent: { client: "codex", model: agentModel }, + tokenizer: "o200k_base", + summary: summarize(runs), + runs, + }; + await mkdir(dirname(output), { recursive: true }); + await writeFile(output, `${JSON.stringify(report, null, 2)}\n`); + await writeFile(output.replace(/\.json$/i, ".md"), markdown(report)); + console.log(JSON.stringify(report.summary)); + if (runs.some((run) => !run.pass)) process.exitCode = 1; +} + +tokenizer.free?.(); diff --git a/eval/current-version/cloudflare-fixture.ts b/eval/current-version/cloudflare-fixture.ts deleted file mode 100644 index a3e829bf..00000000 --- a/eval/current-version/cloudflare-fixture.ts +++ /dev/null @@ -1,506 +0,0 @@ -/** - * A Cloudflare v4 API double, spoken at the HTTP level. - * - * The reference-connection lane has to answer one question honestly: does the - * maintained `cloudflare()` connection serve a cold agent well? Stubbing the - * provider's internals would answer a different, easier question. So nothing - * here touches the provider. This is an ordinary HTTP server that returns - * Cloudflare's envelope — `{ success, errors, messages, result, result_info }` - * — and the real connection is pointed at it through the `baseUrl` option the - * provider already documents as "API base override for a proxy or a test - * double". The real constructor, the real hand-written schemas, the real - * projections, and the real status/code error mapping all run unmodified. - * - * No live credential and no real account payload is involved. Every id, domain, - * and address below is fixture data under reserved test ranges: `.test` names - * (RFC 6761) and `192.0.2.0/24` / `2001:db8::/32` documentation addresses - * (RFC 5737, RFC 3849). - * - * The raw records are deliberately fat. A projection that drops nothing proves - * nothing, so every zone and record carries the metadata Cloudflare really - * returns and the connection really discards. - */ -import { createServer, type Server } from "node:http"; -import { once } from "node:events"; - -/** The account every fixture zone belongs to. */ -export const FIXTURE_ACCOUNT_ID = "acct_eval_edge"; -const FIXTURE_ACCOUNT_NAME = "Connecta Eval Edge"; - -/** The zone the dependent-read case must resolve by name. */ -export const FIXTURE_PRIMARY_ZONE_ID = "zone_eval_a1b2"; -export const FIXTURE_PRIMARY_ZONE_NAME = "connecta-eval.test"; - -/** - * The record-type census of the primary zone, and therefore the exact answer - * the reduced dependent read must produce. Held here so the fixture and the - * benchmark's `correct()` cannot drift apart. - */ -export const FIXTURE_RECORD_TYPE_COUNTS: Record = { - A: 24, - AAAA: 6, - CNAME: 14, - MX: 4, - TXT: 10, - NS: 2, -}; - -interface FixtureZone { - id: string; - name: string; - status: string; -} - -const ZONES: FixtureZone[] = [ - { id: FIXTURE_PRIMARY_ZONE_ID, name: FIXTURE_PRIMARY_ZONE_NAME, status: "active" }, - { id: "zone_eval_c3d4", name: "staging.connecta-eval.test", status: "active" }, - { id: "zone_eval_e5f6", name: "legacy-eval.test", status: "pending" }, -]; - -type JsonRecord = Record; - -function rawZone(zone: FixtureZone): JsonRecord { - return { - id: zone.id, - name: zone.name, - status: zone.status, - paused: false, - type: "full", - development_mode: 0, - name_servers: ["ada.ns.cloudflare.test", "bob.ns.cloudflare.test"], - original_name_servers: ["ns1.registrar.test", "ns2.registrar.test"], - original_registrar: "Fixture Registrar, Inc.", - original_dnshost: null, - modified_on: "2026-07-14T09:12:44.000Z", - created_on: "2025-02-03T17:41:02.000Z", - activated_on: "2025-02-03T18:02:19.000Z", - meta: { - step: 4, - custom_certificate_quota: 0, - page_rule_quota: 3, - phishing_detected: false, - }, - owner: { id: "owner_eval", type: "organization", name: FIXTURE_ACCOUNT_NAME }, - account: { id: FIXTURE_ACCOUNT_ID, name: FIXTURE_ACCOUNT_NAME }, - tenant: { id: null, name: null }, - tenant_unit: { id: null }, - permissions: ["#dns_records:edit", "#dns_records:read", "#zone:read"], - plan: { - id: "plan_free", - name: "Free Website", - price: 0, - currency: "USD", - frequency: "", - is_subscribed: true, - can_subscribe: false, - legacy_id: "free", - legacy_discount: false, - externally_managed: false, - }, - cname_suffix: "cdn.cloudflare.test", - vanity_name_servers: [], - verification_key: "fixture-verification-key", - }; -} - -interface FixtureRecord { - id: string; - name: string; - type: string; - content: string; - ttl: number; - proxied: boolean; - priority?: number; - comment?: string; -} - -/** - * Build the primary zone's records so the type census matches - * `FIXTURE_RECORD_TYPE_COUNTS` exactly. Deterministic and order-stable: the - * same run must produce the same bytes on every repetition, or the token - * measurements are noise. - */ -function buildPrimaryRecords(): FixtureRecord[] { - const records: FixtureRecord[] = []; - let sequence = 0; - const next = (): string => - `dns_eval_${String(++sequence).padStart(3, "0")}`; - - for (let index = 0; index < FIXTURE_RECORD_TYPE_COUNTS["A"]!; index += 1) { - records.push({ - id: next(), - name: `a${String(index + 1).padStart(2, "0")}.${FIXTURE_PRIMARY_ZONE_NAME}`, - type: "A", - content: `192.0.2.${index + 1}`, - ttl: 300, - proxied: index % 2 === 0, - }); - } - for (let index = 0; index < FIXTURE_RECORD_TYPE_COUNTS["AAAA"]!; index += 1) { - records.push({ - id: next(), - name: `v6-${index + 1}.${FIXTURE_PRIMARY_ZONE_NAME}`, - type: "AAAA", - content: `2001:db8::${index + 1}`, - ttl: 300, - proxied: false, - }); - } - for (let index = 0; index < FIXTURE_RECORD_TYPE_COUNTS["CNAME"]!; index += 1) { - records.push({ - id: next(), - name: `alias${String(index + 1).padStart(2, "0")}.${FIXTURE_PRIMARY_ZONE_NAME}`, - type: "CNAME", - content: FIXTURE_PRIMARY_ZONE_NAME, - ttl: 1, - proxied: true, - }); - } - for (let index = 0; index < FIXTURE_RECORD_TYPE_COUNTS["MX"]!; index += 1) { - records.push({ - id: next(), - name: FIXTURE_PRIMARY_ZONE_NAME, - type: "MX", - content: `mx${index + 1}.mail.test`, - ttl: 3600, - proxied: false, - priority: (index + 1) * 10, - }); - } - for (let index = 0; index < FIXTURE_RECORD_TYPE_COUNTS["TXT"]!; index += 1) { - records.push({ - id: next(), - name: `_txt${index + 1}.${FIXTURE_PRIMARY_ZONE_NAME}`, - type: "TXT", - content: `"connecta-eval-token-${index + 1}"`, - ttl: 3600, - proxied: false, - ...(index === 0 ? { comment: "Domain verification for the eval lane." } : {}), - }); - } - for (let index = 0; index < FIXTURE_RECORD_TYPE_COUNTS["NS"]!; index += 1) { - records.push({ - id: next(), - name: `delegated${index + 1}.${FIXTURE_PRIMARY_ZONE_NAME}`, - type: "NS", - content: `ns${index + 1}.delegate.test`, - ttl: 86_400, - proxied: false, - }); - } - return records; -} - -const RECORDS: Record = { - [FIXTURE_PRIMARY_ZONE_ID]: buildPrimaryRecords(), - zone_eval_c3d4: [ - { - id: "dns_stage_001", - name: "staging.connecta-eval.test", - type: "A", - content: "192.0.2.200", - ttl: 300, - proxied: true, - }, - { - id: "dns_stage_002", - name: "api.staging.connecta-eval.test", - type: "CNAME", - content: "staging.connecta-eval.test", - ttl: 1, - proxied: true, - }, - ], - zone_eval_e5f6: [], -}; - -function rawRecord(zoneId: string, record: FixtureRecord): JsonRecord { - const zoneName = - ZONES.find((zone) => zone.id === zoneId)?.name ?? FIXTURE_PRIMARY_ZONE_NAME; - return { - id: record.id, - zone_id: zoneId, - zone_name: zoneName, - name: record.name, - type: record.type, - content: record.content, - ttl: record.ttl, - proxied: record.proxied, - proxiable: record.type === "A" || record.type === "AAAA" || record.type === "CNAME", - ...(record.priority !== undefined ? { priority: record.priority } : {}), - comment: record.comment ?? null, - comment_modified_on: record.comment ? "2026-06-02T11:00:00.000Z" : null, - tags: [], - tags_modified_on: null, - locked: false, - settings: {}, - meta: { auto_added: false, managed_by_apps: false, managed_by_argo_tunnel: false }, - created_on: "2025-02-04T08:15:00.000Z", - modified_on: "2026-06-02T11:00:00.000Z", - }; -} - -/** Cloudflare's success envelope. */ -function ok(result: unknown, resultInfo?: JsonRecord): JsonRecord { - return { - success: true, - errors: [], - messages: [], - result, - ...(resultInfo ? { result_info: resultInfo } : {}), - }; -} - -/** Cloudflare's failure envelope, including the nested `error_chain` form. */ -function failure( - code: number, - message: string, - chain?: { code: number; message: string }[], -): JsonRecord { - return { - success: false, - errors: [ - { - code, - message, - ...(chain ? { error_chain: chain } : {}), - }, - ], - messages: [], - result: null, - }; -} - -export interface CloudflareFixtureOptions { - /** The token the fixture accepts. */ - validToken: string; - /** - * A token the fixture rejects with Cloudflare's real 401 shape, so the - * unavailable-auth case exercises the provider's status/code mapping rather - * than a synthetic error. - */ - revokedToken: string; -} - -export interface CloudflareFixture { - /** Base URL to hand the connection's `baseUrl` option. */ - readonly baseUrl: string; - /** Every request the fixture received, for post-run assertions. */ - readonly requests: { method: string; path: string; token: string | null }[]; - close(): Promise; -} - -/** - * Start the double on an ephemeral loopback port and return its `/client/v4` - * base. Bound to 127.0.0.1: this must never be reachable off the machine. - */ -export async function startCloudflareFixture( - options: CloudflareFixtureOptions, -): Promise { - const requests: { method: string; path: string; token: string | null }[] = []; - - const server: Server = createServer((request, response) => { - const url = new URL(request.url ?? "/", "http://127.0.0.1"); - const authorization = request.headers.authorization ?? ""; - const token = authorization.startsWith("Bearer ") - ? authorization.slice("Bearer ".length) - : null; - requests.push({ method: request.method ?? "GET", path: url.pathname, token }); - - const send = (status: number, body: JsonRecord): void => { - const text = JSON.stringify(body); - response.writeHead(status, { - "content-type": "application/json", - "content-length": String(Buffer.byteLength(text)), - }); - response.end(text); - }; - - const path = url.pathname.startsWith("/client/v4") - ? url.pathname.slice("/client/v4".length) - : null; - if (path === null) { - send(404, failure(7003, "Could not route to the requested path.")); - return; - } - - // Authentication first, exactly as Cloudflare orders it: a revoked token - // never learns whether the resource exists. - if (token === options.revokedToken) { - send( - 401, - failure(10_000, "Authentication error", [ - { code: 10_000, message: "Invalid API Token" }, - ]), - ); - return; - } - if (token !== options.validToken) { - send(401, failure(10_000, "Authentication error")); - return; - } - - const body: Buffer[] = []; - request.on("data", (chunk: Buffer) => body.push(chunk)); - request.on("end", () => { - const method = request.method ?? "GET"; - - if (method === "GET" && path === "/user/tokens/verify") { - send(200, ok({ id: "token_eval", status: "active" })); - return; - } - - if (method === "GET" && path === "/accounts") { - send( - 200, - ok( - [ - { - id: FIXTURE_ACCOUNT_ID, - name: FIXTURE_ACCOUNT_NAME, - type: "standard", - created_on: "2025-01-02T00:00:00.000Z", - settings: { enforce_twofactor: false }, - }, - ], - { page: 1, per_page: 20, count: 1, total_count: 1, total_pages: 1 }, - ), - ); - return; - } - - if (method === "GET" && path === "/zones") { - const nameFilter = url.searchParams.get("name"); - const statusFilter = url.searchParams.get("status"); - const matched = ZONES.filter( - (zone) => - (!nameFilter || zone.name === nameFilter) && - (!statusFilter || zone.status === statusFilter), - ); - const perPage = Number(url.searchParams.get("per_page") ?? "20"); - send( - 200, - ok(matched.map(rawZone), { - page: Number(url.searchParams.get("page") ?? "1"), - per_page: perPage, - count: matched.length, - total_count: matched.length, - total_pages: 1, - }), - ); - return; - } - - const zoneMatch = /^\/zones\/([^/]+)$/.exec(path); - if (method === "GET" && zoneMatch) { - const zone = ZONES.find((candidate) => candidate.id === zoneMatch[1]); - if (!zone) { - send(404, failure(1049, "zone could not be found")); - return; - } - send(200, ok(rawZone(zone))); - return; - } - - const recordsMatch = /^\/zones\/([^/]+)\/dns_records$/.exec(path); - if (recordsMatch) { - const zoneId = recordsMatch[1]!; - const zone = ZONES.find((candidate) => candidate.id === zoneId); - if (!zone) { - send(404, failure(1049, "zone could not be found")); - return; - } - if (method === "GET") { - const typeFilter = url.searchParams.get("type"); - const nameFilter = url.searchParams.get("name"); - const all = RECORDS[zoneId] ?? []; - const matched = all.filter( - (record) => - (!typeFilter || record.type === typeFilter) && - (!nameFilter || record.name === nameFilter), - ); - const perPage = Number(url.searchParams.get("per_page") ?? "100"); - send( - 200, - ok(matched.map((record) => rawRecord(zoneId, record)), { - page: Number(url.searchParams.get("page") ?? "1"), - per_page: perPage, - count: matched.length, - total_count: matched.length, - total_pages: 1, - }), - ); - return; - } - if (method === "POST") { - let parsed: JsonRecord = {}; - try { - parsed = JSON.parse(Buffer.concat(body).toString("utf8") || "{}"); - } catch { - send(400, failure(6003, "Invalid request headers")); - return; - } - // The write the safety case routes through call_destructive_tool. - // Recorded in `requests` so the harness can prove which route - // reached the downstream, and answered with Cloudflare's real - // created-record envelope. - send( - 200, - ok( - rawRecord(zoneId, { - id: "dns_eval_created", - name: String(parsed["name"] ?? ""), - type: String(parsed["type"] ?? "TXT"), - content: String(parsed["content"] ?? ""), - ttl: typeof parsed["ttl"] === "number" ? parsed["ttl"] : 1, - proxied: parsed["proxied"] === true, - ...(typeof parsed["comment"] === "string" - ? { comment: parsed["comment"] } - : {}), - }), - ), - ); - return; - } - } - - const recordMatch = /^\/zones\/([^/]+)\/dns_records\/([^/]+)$/.exec(path); - if (recordMatch) { - const zoneId = recordMatch[1]!; - const recordId = recordMatch[2]!; - const record = (RECORDS[zoneId] ?? []).find( - (candidate) => candidate.id === recordId, - ); - if (method === "DELETE") { - send(200, ok({ id: recordId })); - return; - } - if (!record) { - send(404, failure(81044, "Record does not exist.")); - return; - } - send(200, ok(rawRecord(zoneId, record))); - return; - } - - send(404, failure(7003, "Could not route to the requested path.")); - }); - }); - - server.listen(0, "127.0.0.1"); - await once(server, "listening"); - const address = server.address(); - if (!address || typeof address === "string") { - throw new Error("Cloudflare fixture did not expose a TCP address."); - } - - return { - baseUrl: `http://127.0.0.1:${address.port}/client/v4`, - requests, - async close() { - await new Promise((resolve, reject) => { - server.close((error) => (error ? reject(error) : resolve())); - }); - }, - }; -} diff --git a/eval/current-version/cloudflare-surface-report.ts b/eval/current-version/cloudflare-surface-report.ts deleted file mode 100644 index daac6755..00000000 --- a/eval/current-version/cloudflare-surface-report.ts +++ /dev/null @@ -1,678 +0,0 @@ -/** - * Measure what every Cloudflare named tool costs and what it buys — issue #350. - * - * The provider audit ([#342](https://github.com/zackbart/connecta/issues/342)) - * asks whether a tool is well formed. This lane asks the other question: does - * the named tool earn its place above the three raw escape hatches that already - * reach every v4 endpoint? A named tool is only worth its permanent catalog - * weight if it beats `cloudflare_api_get` / `_mutate` / `_upload` on at least - * one of the four costs the [provider conventions](../../documentation/provider-conventions.md) - * name: discovery tokens, wrong-tool selection, argument retries, result size. - * - * So the lane is deterministic on purpose. Every number comes from product - * code — `CatalogService.search`, the compact discovery renderer, the real - * `api()` validation path, the real handlers — with `fetch` replaced by a probe - * that records the request. Nothing is stubbed inside the provider, no - * credential is used, and no packet leaves the process. A model-driven lane - * would answer a fuzzier version of the same question at one agent run per - * named tool, and would not repeat: selection here is exactly the ranking an - * agent's `search_tools` call runs, not a model's impression of it. - * - * What it deliberately does not measure: result size against real provider - * payloads. Hand-writing a fat, faithful response for every Cloudflare product - * family would measure this file's imagination, so the probe carries known - * identity and noise keys instead, and projection is reported as the detector - * it honestly is — did the handler drop the noise, hand it back whole, or - * answer with a confirmation shape of its own? - * - * node --import tsx eval/current-version/cloudflare-surface-report.ts - */ -import { mkdir, readFile, writeFile } from "node:fs/promises"; -import { Buffer } from "node:buffer"; -import { execFileSync } from "node:child_process"; -import { dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; -import { getEncoding } from "js-tiktoken"; - -import { createConnecta } from "../../src/index.js"; -import { CatalogService, groupedSearchResult } from "../../src/catalog-service.js"; -import { cloudflare } from "../../src/providers/cloudflare.js"; -import { memoryStorage } from "../../src/storage/memory.js"; -import type { - ConnectorContext, - JsonSchema, - Logger, - ToolDef, -} from "../../src/types.js"; - -const here = dirname(fileURLToPath(import.meta.url)); -const root = resolve(here, "../.."); -const args = process.argv.slice(2); - -function option(name: string, fallback: string): string { - const index = args.indexOf(name); - if (index < 0) return fallback; - const value = args[index + 1]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value.`); - } - return value; -} - -const outputPath = resolve( - here, - option("--output", "results/issue-350-cloudflare-surface.json"), -); -const reportPath = resolve( - here, - option("--report", "results/issue-350-cloudflare-surface.md"), -); -const tokenizerName = process.env["CONNECTA_EVAL_TOKENIZER"] ?? "o200k_base"; -const tokenizer = getEncoding(tokenizerName as Parameters[0]); - -const serialized = (value: unknown): string => JSON.stringify(value) ?? "null"; -const tokens = (value: unknown): number => tokenizer.encode(serialized(value)).length; -const bytes = (value: unknown): number => Buffer.byteLength(serialized(value), "utf8"); -const round = (value: number, places = 2): number => { - const scale = 10 ** places; - return Math.round(value * scale) / scale; -}; - -const silentLogger: Logger = { - debug: () => {}, - info: () => {}, - warn: () => {}, - error: () => {}, -}; - -/** The three tools every named tool is measured against. */ -const HATCHES = ["cloudflare_api_get", "cloudflare_api_mutate", "cloudflare_api_upload"]; -/** Named, but a credential check rather than a control-plane operation. */ -const CREDENTIAL_TOOLS = ["verify_api_token", "verify_global_api_key"]; - -const BASE_URL = "https://connecta.example"; -const CONNECTOR_ID = "cloudflare"; - -/** - * The unscoped shape on purpose: no `zoneId`, no `accountId`. That is what a - * fresh deployment gets, and it is the harder case — every zone- and - * account-scoped tool keeps its id argument required, so nothing here is - * flattered by a default that happens to be configured. - */ -const connection = cloudflare(CONNECTOR_ID, { - purpose: "Measure the named surface against the raw escape hatches", -}); - -const { registry } = createConnecta({ - connectors: [connection], - storage: memoryStorage(), - logger: silentLogger, - executor: { execute: async () => ({ result: null }) }, - publicUrl: BASE_URL, -}); - -/** One connector context for the whole run; its credential never leaves it. */ -const context: ConnectorContext = { - storage: memoryStorage(), - logger: silentLogger, - baseUrl: BASE_URL, - credential: { - get: async () => "probe-token", - getAll: async () => ({ - value: "probe-token", - email: "probe@example.test", - apiKey: "probe-key", - }), - }, -}; - -const catalog = new CatalogService(registry, BASE_URL); -const definitions = await connection.listTools(context); -const byName = new Map(definitions.map((tool) => [tool.name, tool])); - -// --------------------------------------------------------------------------- -// Catalog weight -// --------------------------------------------------------------------------- - -/** - * What one tool costs an agent that browses this connector with compact - * schemas — the exact per-entry payload `search_tools` emits, measured one - * entry at a time so a tool's share of the catalog is its own. - */ -const browse = await catalog.search({ - connector: CONNECTOR_ID, - limit: 100, - includeSchemas: "compact", - includeSchemaKeys: true, -}); -const compactEntries = new Map( - browse.entries.map((entry) => [entry.tool.name, entry.tool]), -); -const wholeCatalogTokens = tokens(groupedSearchResult(browse)); - -const described = await catalog.describe({ - addresses: definitions.map((tool) => `${CONNECTOR_ID}.${tool.name}`), - format: "json", - fullDescriptions: true, -}); -const describedByName = new Map( - described.map((entry) => [entry.address.split(".").slice(1).join("."), entry]), -); - -// --------------------------------------------------------------------------- -// Argument handling -// --------------------------------------------------------------------------- - -interface FetchProbe { - method: string; - url: string; -} - -let probeRequests: FetchProbe[] = []; -let probeResult: unknown = {}; -let probeResultInfo: Record | undefined; - -const realFetch = globalThis.fetch; - -/** - * A Cloudflare envelope with known noise in it. Every key under `noise` is - * something no projection keeps, so a handler that returns them passed the - * provider's object through untouched. - */ -const NOISE_KEYS = [ - "meta", - "permissions", - "plan", - "owner", - "tenant", - "development_mode", - "original_registrar", - "cname_suffix", -]; - -/** Identity values only a handler that echoed the provider's object can return. */ -const IDENTITY_VALUES = ["probe-id", "probe-name", "probe-title", "probe-key"]; - -/** The `result_info` counters Cloudflare sends beside a page of records. */ -const PROBE_RESULT_INFO = { - page: 1, - per_page: 20, - count: 1, - total_count: 1, - total_pages: 1, -}; - -/** Cloudflare's other collection shape: the records under a product-named key. */ -function collectionProbe(): Record { - const named: Record = {}; - for (const key of ["buckets", "objects", "keys", "delimitedPrefixes", "values"]) { - named[key] = [probeObject()]; - } - return named; -} - -function probeObject(): Record { - const value: Record = { - id: "probe-id", - name: "probe-name", - title: "probe-title", - // R2 objects and KV keys are identified by `key`, not `id`. - key: "probe-key", - status: "active", - }; - for (const key of NOISE_KEYS) { - value[key] = { measured: "noise", size: key.length }; - } - return value; -} - -globalThis.fetch = (async (input: unknown, init: RequestInit = {}) => { - probeRequests.push({ - method: String(init.method ?? "GET"), - url: String(input), - }); - const body = { - success: true, - errors: [], - messages: [], - result: probeResult, - ...(probeResultInfo ? { result_info: probeResultInfo } : {}), - }; - const text = JSON.stringify(body); - const response = { - ok: true, - status: 200, - headers: new Headers({ "content-type": "application/json" }), - json: async () => JSON.parse(text) as unknown, - text: async () => text, - arrayBuffer: async () => new TextEncoder().encode(text).buffer, - } as unknown as Response; - Object.assign(response, { clone: () => response }); - return response; -}) as unknown as typeof fetch; - -function schemaProperties(schema: JsonSchema | undefined): Record { - const properties = schema?.["properties"]; - return properties && typeof properties === "object" - ? (properties as Record) - : {}; -} - -function requiredKeys(schema: JsonSchema | undefined): string[] { - const required = schema?.["required"]; - return Array.isArray(required) ? required.map(String) : []; -} - -/** A value the schema itself says is acceptable, so a rejection is real news. */ -function sampleValue(schema: JsonSchema, name: string): unknown { - const enumValues = schema["enum"]; - if (Array.isArray(enumValues) && enumValues.length > 0) return enumValues[0]; - const type = schema["type"]; - if (type === "number" || type === "integer") { - const minimum = schema["minimum"]; - return typeof minimum === "number" ? minimum : 1; - } - if (type === "boolean") return true; - if (type === "array") { - const items = schema["items"]; - const minItems = typeof schema["minItems"] === "number" ? schema["minItems"] : 0; - if (minItems < 1) return []; - const item = - items && typeof items === "object" - ? sampleValue(items as JsonSchema, `${name}Item`) - : "probe"; - return [item]; - } - if (type === "object") { - const nested: Record = {}; - for (const key of requiredKeys(schema)) { - const property = schemaProperties(schema)[key]; - nested[key] = property ? sampleValue(property, key) : "probe"; - } - return nested; - } - if (name.toLowerCase().endsWith("id")) return "0a1b2c3d4e5f60718293a4b5c6d7e8f9"; - return "probe"; -} - -/** Every required argument, filled from the schema's own vocabulary. */ -function requiredArgs(tool: ToolDef): Record { - const properties = schemaProperties(tool.inputSchema); - const built: Record = {}; - for (const key of requiredKeys(tool.inputSchema)) { - const property = properties[key]; - built[key] = property ? sampleValue(property, key) : "probe"; - } - return built; -} - -interface CallOutcome { - ok: boolean; - code?: string; - requests: FetchProbe[]; - result?: unknown; -} - -async function call( - tool: ToolDef, - callArgs: Record, - options: { result?: unknown; resultInfo?: Record } = {}, -): Promise { - probeRequests = []; - probeResult = options.result ?? probeObject(); - probeResultInfo = options.resultInfo; - try { - const result = await connection.callTool(tool.name, callArgs, context); - return { ok: true, requests: probeRequests, result }; - } catch (error) { - const code = (error as { code?: string }).code; - return { - ok: false, - ...(code !== undefined ? { code } : {}), - requests: probeRequests, - }; - } -} - -/** - * The guard measurement. Each mutation is a mistake an agent actually makes; - * a named tool earns its "argument retries" claim only if the mistake is - * refused here rather than at Cloudflare, which the raw hatch — whose path is - * an opaque string — structurally cannot do. - */ -interface Mutation { - kind: string; - args: Record; -} - -function mutations(tool: ToolDef, valid: Record): Mutation[] { - const built: Mutation[] = [ - { kind: "unknownProperty", args: { ...valid, notACloudflareArgument: "x" } }, - ]; - const required = requiredKeys(tool.inputSchema); - if (required.length > 0) { - const dropped = { ...valid }; - delete dropped[required[0]!]; - built.push({ kind: "missingRequired", args: dropped }); - } - const properties = schemaProperties(tool.inputSchema); - const enumKey = Object.keys(properties).find((key) => - Array.isArray(properties[key]?.["enum"]), - ); - if (enumKey) { - built.push({ - kind: "unknownEnumValue", - args: { ...valid, [enumKey]: "not-a-cloudflare-value" }, - }); - } - const perPage = properties["perPage"]; - const maximum = perPage?.["maximum"]; - if (typeof maximum === "number") { - built.push({ kind: "pageSizeOverMaximum", args: { ...valid, perPage: maximum + 1 } }); - } - return built; -} - -// --------------------------------------------------------------------------- -// Run -// --------------------------------------------------------------------------- - -interface TaskFile { - tasks: { tool: string; query: string; args?: Record }[]; -} - -const taskFile = JSON.parse( - await readFile(resolve(here, "cloudflare-surface-tasks.json"), "utf8"), -) as TaskFile; -const taskByTool = new Map(taskFile.tasks.map((task) => [task.tool, task])); - -const missingTask = definitions - .map((tool) => tool.name) - .filter( - (name) => - !HATCHES.includes(name) && - !CREDENTIAL_TOOLS.includes(name) && - !taskByTool.has(name), - ); -if (missingTask.length > 0) { - throw new Error( - `Named tools with no representative task: ${missingTask.join(", ")}. ` + - "Every named tool gets a verdict, so every named tool gets a task.", - ); -} -const staleTask = taskFile.tasks - .map((task) => task.tool) - .filter((name) => !byName.has(name)); -if (staleTask.length > 0) { - throw new Error(`Tasks naming tools that no longer exist: ${staleTask.join(", ")}.`); -} - -interface ToolMeasurement { - name: string; - role: "named" | "hatch" | "credential"; - readOnly: boolean; - compactTokens?: number; - compactBytes?: number; - schemaTruncated: boolean; - jsonDefinitionTokens?: number; - declaredOutputKeys: number; - openOutput: boolean; - query?: string; - selectionRank?: number; - selectionTop?: string; - selectionResultTokens?: number; - hatchOutranked?: boolean; - validCallAccepted?: boolean; - validCallCode?: string; - request?: string; - guards: { kind: string; refusedLocally: boolean; code?: string }[]; - guardMisses: number; - /** - * `projected` — the handler kept identity fields and dropped the probe's - * noise; `passthrough` — Cloudflare's object came back whole; `fixed` — the - * handler returns its own confirmation shape and never echoes the provider. - */ - resultShape?: "projected" | "passthrough" | "fixed"; - noiseKeysReturned?: number; - resultReductionPercent?: number; -} - -const measurements: ToolMeasurement[] = []; - -for (const tool of definitions) { - const role: ToolMeasurement["role"] = HATCHES.includes(tool.name) - ? "hatch" - : CREDENTIAL_TOOLS.includes(tool.name) - ? "credential" - : "named"; - const compact = compactEntries.get(tool.name); - const outputKeys = compact?.outputKeys ?? []; - const openOutput = - outputKeys.length === 0 && - (tool.outputSchema?.["additionalProperties"] !== false || - Object.keys(schemaProperties(tool.outputSchema)).length === 0); - - const measurement: ToolMeasurement = { - name: tool.name, - role, - readOnly: tool.annotations?.readOnlyHint === true, - ...(compact ? { compactTokens: tokens(compact), compactBytes: bytes(compact) } : {}), - schemaTruncated: - compact?.inputSchemaTruncated === true || compact?.outputSchemaTruncated === true, - ...(describedByName.has(tool.name) - ? { jsonDefinitionTokens: tokens(describedByName.get(tool.name)) } - : {}), - declaredOutputKeys: outputKeys.length, - openOutput, - guards: [], - guardMisses: 0, - }; - - const task = taskByTool.get(tool.name); - if (task) { - const page = await catalog.search({ - query: task.query, - limit: 8, - includeSchemas: "compact", - includeSchemaKeys: true, - }); - const names = page.entries.map((entry) => entry.tool.name); - const rank = names.indexOf(tool.name); - measurement.query = task.query; - measurement.selectionRank = rank < 0 ? -1 : rank + 1; - measurement.selectionTop = names[0] ?? "(nothing matched)"; - measurement.selectionResultTokens = tokens(groupedSearchResult(page)); - const hatchRank = names.findIndex((name) => HATCHES.includes(name)); - measurement.hatchOutranked = - hatchRank >= 0 && (rank < 0 || hatchRank < rank); - } - - if (role === "named") { - const valid = { ...requiredArgs(tool), ...task?.args }; - // A collection endpoint gets a collection back, or the projection under - // measurement never runs and every list tool would score as "fixed". - // An array of *objects* is a collection; an array of strings is a field - // (a zone's name servers), and handing that tool an array would measure - // the probe rather than the projection. - const collection = Object.values(schemaProperties(tool.outputSchema)).some( - (property) => { - if (property["type"] !== "array") return false; - const items = property["items"]; - return Boolean( - items && - typeof items === "object" && - ((items as JsonSchema)["type"] === "object" || - (items as JsonSchema)["properties"] !== undefined), - ); - }, - ); - // Cloudflare returns a collection two ways — a bare array under `result`, - // and a named collection such as `{ buckets: [...] }` for R2 — so a - // collection tool is offered both and scored on the one its handler - // actually consumes. Guessing one shape would file a working projection as - // a fixed return. - const probes: unknown[] = collection - ? [[probeObject()], collectionProbe()] - : [probeObject()]; - let best: - | { outcome: CallOutcome; raw: unknown; shape: ToolMeasurement["resultShape"]; noise: number } - | undefined; - for (const raw of probes) { - const outcome = await call(tool, valid, { - result: raw, - ...(collection ? { resultInfo: PROBE_RESULT_INFO } : {}), - }); - const returned = outcome.ok ? serialized(outcome.result) : ""; - const noise = NOISE_KEYS.filter((key) => returned.includes(`"${key}"`)).length; - const shape: ToolMeasurement["resultShape"] = !outcome.ok - ? undefined - : noise > 0 - ? "passthrough" - : IDENTITY_VALUES.some((value) => returned.includes(value)) - ? "projected" - : "fixed"; - if (!best || (best.shape === "fixed" && shape !== undefined && shape !== "fixed")) { - best = { outcome, raw, shape, noise }; - } - if (shape !== undefined && shape !== "fixed") break; - } - const outcome = best!.outcome; - measurement.validCallAccepted = outcome.ok; - if (outcome.code) measurement.validCallCode = outcome.code; - const request = outcome.requests[0]; - if (request) { - measurement.request = `${request.method} ${new URL(request.url).pathname}`; - } - const shape = best!.shape; - if (outcome.ok && outcome.result !== undefined && shape !== undefined) { - measurement.noiseKeysReturned = best!.noise; - measurement.resultShape = shape; - measurement.resultReductionPercent = round( - (1 - bytes(outcome.result) / bytes(best!.raw)) * 100, - 1, - ); - } - for (const mutation of mutations(tool, valid)) { - const attempt = await call(tool, mutation.args); - const refusedLocally = !attempt.ok && attempt.requests.length === 0; - measurement.guards.push({ - kind: mutation.kind, - refusedLocally, - ...(attempt.code !== undefined ? { code: attempt.code } : {}), - }); - if (!refusedLocally) measurement.guardMisses += 1; - } - } - - measurements.push(measurement); -} - -globalThis.fetch = realFetch; - -const named = measurements.filter((entry) => entry.role === "named"); -const selected = named.filter((entry) => entry.selectionRank === 1); -const rankedTop3 = named.filter( - (entry) => entry.selectionRank !== undefined && entry.selectionRank > 0 && entry.selectionRank <= 3, -); - -const summary = { - toolCount: measurements.length, - namedCount: named.length, - hatchCount: measurements.filter((entry) => entry.role === "hatch").length, - wholeCatalogTokens, - namedCatalogTokens: named.reduce((total, entry) => total + (entry.compactTokens ?? 0), 0), - hatchCatalogTokens: measurements - .filter((entry) => entry.role === "hatch") - .reduce((total, entry) => total + (entry.compactTokens ?? 0), 0), - top1SelectionRate: round(selected.length / named.length, 3), - top3SelectionRate: round(rankedTop3.length / named.length, 3), - unselectedTools: named - .filter((entry) => entry.selectionRank !== 1) - .map((entry) => entry.name), - hatchOutrankedTools: named - .filter((entry) => entry.hatchOutranked === true) - .map((entry) => entry.name), - validCallFailures: named - .filter((entry) => entry.validCallAccepted !== true) - .map((entry) => entry.name), - guardMissTools: named.filter((entry) => entry.guardMisses > 0).map((entry) => entry.name), - passthroughTools: named - .filter((entry) => entry.resultShape === "passthrough") - .map((entry) => entry.name), - openOutputTools: named.filter((entry) => entry.openOutput).map((entry) => entry.name), - truncatedSchemaTools: measurements - .filter((entry) => entry.schemaTruncated) - .map((entry) => entry.name), -}; - -const artifact = { - issue: 350, - generatedAt: new Date().toISOString(), - sourceCommit: execFileSync("git", ["rev-parse", "HEAD"], { - cwd: root, - encoding: "utf8", - }).trim(), - // A commit alone does not identify a surface measured from a working tree, - // and this lane is most useful exactly while one is being changed. - workingTreeDirty: - execFileSync("git", ["status", "--porcelain"], { - cwd: root, - encoding: "utf8", - }).trim().length > 0, - runtime: process.version, - tokenizer: tokenizerName, - scope: "unscoped cloudflare() instance: no zoneId or accountId default", - summary, - tools: measurements, -}; - -await mkdir(dirname(outputPath), { recursive: true }); -await writeFile(outputPath, `${JSON.stringify(artifact, null, 2)}\n`, "utf8"); - -function row(entry: ToolMeasurement): string { - const selection = - entry.selectionRank === undefined - ? "—" - : entry.selectionRank < 0 - ? `miss (${entry.selectionTop})` - : entry.selectionRank === 1 - ? "1" - : `${entry.selectionRank} (${entry.selectionTop})`; - const projection = entry.resultShape ?? "—"; - return `| \`${entry.name}\` | ${entry.readOnly ? "read" : "write"} | ${entry.compactTokens ?? "—"} | ${selection} | ${entry.guards.length - entry.guardMisses}/${entry.guards.length} | ${projection} | ${entry.declaredOutputKeys} |`; -} - -const report = `# Cloudflare named-tool surface measurements (#350) - -Generated by \`eval/current-version/cloudflare-surface-report.ts\` at -${artifact.generatedAt} on ${artifact.runtime}, source commit -\`${artifact.sourceCommit}\`${artifact.workingTreeDirty ? " (working tree modified)" : ""}, -tokenizer \`${artifact.tokenizer}\`. -Scope: ${artifact.scope}. - -- ${summary.toolCount} tools total: ${summary.namedCount} named, ${summary.hatchCount} escape hatches, ${summary.toolCount - summary.namedCount - summary.hatchCount} credential check. -- Whole-connector compact browse: **${summary.wholeCatalogTokens} tokens**, of which the named surface is ${summary.namedCatalogTokens} and the three hatches are ${summary.hatchCatalogTokens}. -- Top-1 selection on its own representative task: **${round(summary.top1SelectionRate * 100, 1)}%** (top-3 ${round(summary.top3SelectionRate * 100, 1)}%). -- Named tools an escape hatch outranked: ${summary.hatchOutrankedTools.length === 0 ? "none" : summary.hatchOutrankedTools.join(", ")}. -- Argument guards that reached the network instead of being refused locally: ${summary.guardMissTools.length === 0 ? "none" : summary.guardMissTools.join(", ")}. -- Named reads that return Cloudflare's object unprojected: ${summary.passthroughTools.length === 0 ? "none" : summary.passthroughTools.join(", ")}. -- Tools declaring no output keys: ${summary.openOutputTools.length === 0 ? "none" : summary.openOutputTools.join(", ")}. -- Compact schemas the renderer truncated: ${summary.truncatedSchemaTools.length === 0 ? "none" : summary.truncatedSchemaTools.join(", ")}. - -\`selection\` is the tool's rank in a real \`search_tools\` call for its task, -with the tool that actually ranked first in parentheses when it was not this -one. \`guards\` counts argument mistakes refused before the round trip. -\`projection\` records whether the handler dropped the probe's noise keys. - -| tool | class | compact tokens | selection | guards | projection | output keys | -| --- | --- | --- | --- | --- | --- | --- | -${measurements.map(row).join("\n")} -`; - -await writeFile(reportPath, report, "utf8"); - -console.log(`wrote ${outputPath}`); -console.log(`wrote ${reportPath}`); -console.log( - `top-1 ${round(summary.top1SelectionRate * 100, 1)}% | catalog ${wholeCatalogTokens} tokens | guard misses ${summary.guardMissTools.length}`, -); diff --git a/eval/current-version/cloudflare-surface-tasks.json b/eval/current-version/cloudflare-surface-tasks.json deleted file mode 100644 index df1f0f56..00000000 --- a/eval/current-version/cloudflare-surface-tasks.json +++ /dev/null @@ -1,69 +0,0 @@ -{ - "note": "One representative operator request per Cloudflare named tool, for issue #350. Queries are written in the words an operator would use, never the tool's own name or description, because a query copied from the tool it is meant to find measures nothing. Each task names the tool the request should reach; the harness records what search actually ranked first.", - "tasks": [ - { "tool": "list_accounts", "query": "which cloudflare accounts can this token reach" }, - { "tool": "list_zones", "query": "find the id for the domain example.com" }, - { "tool": "get_zone", "query": "show the plan and name servers for one domain" }, - { "tool": "get_zone_setting", "query": "is always use https turned on" }, - { "tool": "update_zone_setting", "query": "turn on always use https" }, - { "tool": "list_zone_rulesets", "query": "what waf and transform phases exist here" }, - { "tool": "get_zone_ruleset", "query": "show the redirect rules and their expressions" }, - { "tool": "list_dns_records", "query": "show the txt records on the apex domain" }, - { "tool": "get_dns_record", "query": "fetch one dns record by its id" }, - { "tool": "list_worker_scripts", "query": "which workers are deployed in this account" }, - { "tool": "get_worker_settings", "query": "what compatibility date and bindings does this worker use" }, - { "tool": "list_worker_deployments", "query": "show the release history and traffic split for a worker" }, - { "tool": "get_worker_deployment", "query": "which versions is one worker release serving" }, - { "tool": "delete_worker_script", "query": "take down a worker that is no longer used" }, - { "tool": "list_kv_namespaces", "query": "what kv namespaces exist in this account" }, - { "tool": "get_kv_namespace", "query": "look up one kv namespace by id" }, - { "tool": "create_kv_namespace", "query": "make a new kv namespace called sessions" }, - { "tool": "rename_kv_namespace", "query": "give a kv namespace a different title" }, - { "tool": "delete_kv_namespace", "query": "destroy a kv namespace and everything in it" }, - { "tool": "list_kv_keys", "query": "what kv keys are stored under a prefix" }, - { "tool": "bulk_get_kv_values", "query": "read several kv values at once" }, - { "tool": "bulk_write_kv_values", "query": "write many kv entries with expirations" }, - { "tool": "bulk_delete_kv_values", "query": "remove a batch of kv keys" }, - { "tool": "list_r2_buckets", "query": "what r2 buckets does this account have" }, - { "tool": "get_r2_bucket", "query": "what region and storage class is one r2 bucket" }, - { "tool": "create_r2_bucket", "query": "make a new r2 bucket in europe" }, - { "tool": "update_r2_bucket", "query": "switch an r2 bucket to infrequent access storage" }, - { "tool": "delete_r2_bucket", "query": "get rid of an empty r2 bucket" }, - { "tool": "list_r2_objects", "query": "what objects are stored under a prefix in this bucket" }, - { "tool": "delete_r2_object", "query": "remove one stored object from a bucket by key" }, - { "tool": "get_r2_cors", "query": "what browser origins may fetch from this bucket" }, - { "tool": "list_pages_projects", "query": "what pages sites exist in this account" }, - { "tool": "get_pages_project", "query": "show the build command and output directory for a pages site" }, - { "tool": "list_pages_deployments", "query": "show recent pages builds and their branches" }, - { "tool": "get_pages_deployment", "query": "why did one pages build fail, show its stages" }, - { "tool": "retry_pages_deployment", "query": "run that failed pages build again" }, - { "tool": "rollback_pages_deployment", "query": "put the previous pages build back into production" }, - { "tool": "delete_pages_deployment", "query": "delete an old pages build and its url" }, - { "tool": "list_pages_domains", "query": "which custom domains are attached to a pages site" }, - { "tool": "add_pages_domain", "query": "attach docs.example.com to a pages site" }, - { "tool": "delete_pages_domain", "query": "detach a custom domain from a pages site" }, - { "tool": "purge_pages_build_cache", "query": "force the next pages build to rebuild dependencies from scratch" }, - { "tool": "delete_pages_project", "query": "remove a pages site entirely" }, - { "tool": "create_dns_record", "query": "add a txt record for domain verification" }, - { - "tool": "update_dns_record", - "query": "point an existing a record at a new ip address", - "args": { "content": "192.0.2.10" }, - "argsNote": "The schema's required list is ids only; a call that changes nothing is refused locally, so the probe supplies the change." - }, - { "tool": "delete_dns_record", "query": "remove a stale cname record" }, - { - "tool": "purge_cache", - "query": "clear cached copies of these urls at the edge", - "args": { "files": ["https://example.com/style.css"] }, - "argsNote": "Exactly one purge variant per call, and the variant is not in the required list, so the probe picks one." - } - ], - "retiredNote": "Tasks whose tool a surface verdict pruned — the #350 verdicts, and #361 for the deprecated bulk zone-settings read. Kept because the report has to show the task that produced the removal, not just the removal. The harness ignores them; deleting them would delete the evidence.", - "retired": [ - { "tool": "list_zone_settings", "query": "what is currently configured on this domain" }, - { "tool": "get_r2_metrics", "query": "how much data and how many objects are in r2" }, - { "tool": "set_r2_cors", "query": "allow browser uploads from a new origin on this bucket" }, - { "tool": "delete_r2_cors", "query": "stop allowing any browser origin on this bucket" } - ] -} diff --git a/eval/current-version/discovery-benchmark.mjs b/eval/current-version/discovery-benchmark.mjs deleted file mode 100644 index b69f2d53..00000000 --- a/eval/current-version/discovery-benchmark.mjs +++ /dev/null @@ -1,360 +0,0 @@ -import { readFile } from "node:fs/promises"; -import { isDeepStrictEqual } from "node:util"; - -import { round, structured } from "./audit-lib.mjs"; - -function addressesFrom(value) { - return Array.isArray(value?.connectors) - ? value.connectors.flatMap((group) => - Array.isArray(group.tools) - ? group.tools.map((tool) => tool.address) - : [], - ) - : []; -} - -function mean(items, select) { - return items.length === 0 - ? 0 - : items.reduce((sum, item) => sum + select(item), 0) / items.length; -} - -function withoutQueryCoverage(value) { - if (Array.isArray(value)) { - return value.map(withoutQueryCoverage); - } - if (!value || typeof value !== "object") return value; - return Object.fromEntries( - Object.entries(value).flatMap(([key, entry]) => - key === "queryCoverage" || - key === "queryTerms" || - key === "queryTermsTruncated" - ? [] - : [[key, withoutQueryCoverage(entry)]], - ), - ); -} - -function expandIndexes(terms, indexes) { - return Array.isArray(indexes) - ? indexes.flatMap((index) => - typeof terms[index] === "string" ? [terms[index]] : [], - ) - : []; -} - -function coverageRows(value) { - const trailing = value?.queryCoverage; - if (trailing && Array.isArray(trailing.entries)) { - const terms = Array.isArray(trailing.terms) ? trailing.terms : []; - return trailing.entries.map((entry) => ({ - address: entry.address, - nameTerms: expandIndexes(terms, entry.name), - descriptionTerms: expandIndexes(terms, entry.description), - unmatchedTerms: expandIndexes(terms, entry.unmatched), - ...(trailing.truncated ? { truncated: true } : {}), - })); - } - const indexedTerms = Array.isArray(value?.queryTerms) - ? value.queryTerms - : []; - return Array.isArray(value?.connectors) - ? value.connectors.flatMap((group) => - Array.isArray(group.tools) - ? group.tools.flatMap((tool) => { - const entry = tool?.queryCoverage; - if (!entry) return []; - if ( - Array.isArray(entry.nameTerms) || - Array.isArray(entry.descriptionTerms) || - Array.isArray(entry.unmatchedTerms) - ) { - return [{ address: tool.address, ...entry }]; - } - return [ - { - address: tool.address, - nameTerms: expandIndexes(indexedTerms, entry.name), - descriptionTerms: expandIndexes( - indexedTerms, - entry.description, - ), - unmatchedTerms: expandIndexes( - indexedTerms, - entry.unmatched, - ), - ...(value.queryTermsTruncated ? { truncated: true } : {}), - }, - ]; - }) - : [], - ) - : []; -} - -function resultWithoutQueryCoverage(result) { - const copy = withoutQueryCoverage(result); - if (!Array.isArray(copy?.content)) return copy; - copy.content = copy.content.map((item) => { - if (item?.type !== "text" || typeof item.text !== "string") return item; - try { - return { - ...item, - text: JSON.stringify(withoutQueryCoverage(JSON.parse(item.text))), - }; - } catch { - return item; - } - }); - return copy; -} - -export async function runDiscoveryBenchmark(context, corpusPath) { - const corpus = JSON.parse(await readFile(corpusPath, "utf8")); - const cases = []; - - for (const fixture of corpus.queries) { - const args = { - query: fixture.query, - ...(fixture.connector ? { connector: fixture.connector } : {}), - ...(fixture.limit !== undefined ? { limit: fixture.limit } : {}), - ...(fixture.offset !== undefined ? { offset: fixture.offset } : {}), - }; - const { result, observation } = await context.call( - `holdout:${fixture.id}`, - "search_tools", - args, - ); - const value = structured(result); - const pageAddresses = addressesFrom(value); - const relevant = fixture.expectedPage ?? fixture.relevant; - const relevantSet = new Set(relevant); - const returnedRelevant = pageAddresses.filter((address) => - relevantSet.has(address), - ); - const positive = relevant.length > 0; - const defaultPage = fixture.limit === undefined; - const recall = - positive ? returnedRelevant.length / relevant.length : pageAddresses.length === 0 ? 1 : 0; - const precision = - pageAddresses.length > 0 - ? returnedRelevant.length / pageAddresses.length - : positive - ? 0 - : 1; - const top1 = - positive && relevantSet.has(pageAddresses[0] ?? "") ? 1 : 0; - const expectedTop = fixture.expectedTop ?? relevant[0] ?? null; - const expectedTopMatch = - expectedTop === null ? null : pageAddresses[0] === expectedTop; - const falsePositive = !positive && pageAddresses.length > 0; - const coverage = coverageRows(value); - const expectedCoverage = fixture.expectedCoverage ?? null; - const coverageExpectedCorrect = - expectedCoverage === null - ? null - : Object.entries(expectedCoverage).every(([address, expected]) => { - const actual = coverage.find((entry) => entry.address === address); - if (!actual) return false; - const { address: _address, ...actualCoverage } = actual; - return isDeepStrictEqual(actualCoverage, expected); - }); - const expectedTopCoverage = coverage.find( - (entry) => entry.address === expectedTop, - ); - const coverageDiscriminates = - expectedTopCoverage !== undefined && - expectedTopCoverage.nameTerms.length > 0 && - coverage.some( - (entry) => - !relevantSet.has(entry.address) && - entry.nameTerms.length === 0 && - entry.descriptionTerms.length > 0, - ); - const responseTokensWithoutCoverage = context.tokens( - resultWithoutQueryCoverage(result), - ); - const responseBytesWithoutCoverage = context.bytes( - resultWithoutQueryCoverage(result), - ); - cases.push({ - id: fixture.id, - category: fixture.category, - query: fixture.query, - ...(fixture.connector ? { connector: fixture.connector } : {}), - relevant, - pageAddresses, - total: typeof value?.total === "number" ? value.total : pageAddresses.length, - returned: pageAddresses.length, - top1, - expectedTop, - expectedTopMatch, - recall: round(recall, 3), - recallAtDefaultPage: defaultPage ? round(recall, 3) : null, - precision: round(precision, 3), - falsePositive, - matchMode: value?.matchMode ?? "all", - hasMore: value?.hasMore === true, - nextOffset: value?.nextOffset ?? null, - responseTokens: observation.responseTokens, - responseTokensWithoutCoverage, - queryCoverageTokens: - observation.responseTokens - responseTokensWithoutCoverage, - responseBytes: observation.responseBytes, - responseBytesWithoutCoverage, - queryCoverageBytes: - observation.responseBytes - responseBytesWithoutCoverage, - queryCoverageRows: coverage.length, - coverage, - coverageExpectedCorrect, - coverageDiscriminates, - latencyMs: observation.latencyMs, - passed: - positive - ? returnedRelevant.length === relevant.length - : pageAddresses.length === 0, - }); - } - - const categories = Object.fromEntries( - [...new Set(cases.map((entry) => entry.category))].map((category) => { - const selected = cases.filter((entry) => entry.category === category); - const positives = selected.filter((entry) => entry.relevant.length > 0); - const negatives = selected.filter((entry) => entry.relevant.length === 0); - return [ - category, - { - queries: selected.length, - top1Accuracy: - positives.length === 0 - ? null - : round(mean(positives, (entry) => entry.top1), 3), - expectedTopAccuracy: - positives.length === 0 - ? null - : round( - mean( - positives.filter( - (entry) => entry.expectedTopMatch !== null, - ), - (entry) => (entry.expectedTopMatch ? 1 : 0), - ), - 3, - ), - positiveRecall: - positives.length === 0 - ? null - : round(mean(positives, (entry) => entry.recall), 3), - meanPrecision: round(mean(selected, (entry) => entry.precision), 3), - falsePositiveRate: - negatives.length === 0 - ? null - : round( - negatives.filter((entry) => entry.falsePositive).length / - negatives.length, - 3, - ), - meanResultCount: round(mean(selected, (entry) => entry.returned), 3), - meanResponseTokens: round( - mean(selected, (entry) => entry.responseTokens), - 1, - ), - meanQueryCoverageTokens: round( - mean(selected, (entry) => entry.queryCoverageTokens), - 1, - ), - meanLatencyMs: round(mean(selected, (entry) => entry.latencyMs), 1), - }, - ]; - }), - ); - const positives = cases.filter((entry) => entry.relevant.length > 0); - const negatives = cases.filter((entry) => entry.relevant.length === 0); - const defaultPageCases = positives.filter( - (entry) => entry.recallAtDefaultPage !== null, - ); - - return { - corpus: { - schemaVersion: corpus.schemaVersion, - name: corpus.name, - authorship: corpus.authorship, - connectorCount: corpus.connectors.length, - toolCount: corpus.connectors.reduce( - (sum, connector) => sum + connector.tools.length, - 0, - ), - queryCount: corpus.queries.length, - }, - metrics: { - top1Accuracy: round(mean(positives, (entry) => entry.top1), 3), - expectedTopAccuracy: round( - mean( - positives.filter((entry) => entry.expectedTopMatch !== null), - (entry) => (entry.expectedTopMatch ? 1 : 0), - ), - 3, - ), - positiveRecall: round(mean(positives, (entry) => entry.recall), 3), - recallAtDefaultPage: round( - mean(defaultPageCases, (entry) => entry.recallAtDefaultPage), - 3, - ), - meanPrecision: round(mean(cases, (entry) => entry.precision), 3), - falsePositiveRate: - negatives.length === 0 - ? 0 - : round( - negatives.filter((entry) => entry.falsePositive).length / - negatives.length, - 3, - ), - meanResultCount: round(mean(cases, (entry) => entry.returned), 3), - meanResponseTokens: round( - mean(cases, (entry) => entry.responseTokens), - 1, - ), - totalResponseTokens: cases.reduce( - (sum, entry) => sum + entry.responseTokens, - 0, - ), - totalResponseBytes: cases.reduce( - (sum, entry) => sum + entry.responseBytes, - 0, - ), - totalQueryCoverageTokens: cases.reduce( - (sum, entry) => sum + entry.queryCoverageTokens, - 0, - ), - totalQueryCoverageBytes: cases.reduce( - (sum, entry) => sum + entry.queryCoverageBytes, - 0, - ), - meanQueryCoverageTokens: round( - mean(cases, (entry) => entry.queryCoverageTokens), - 1, - ), - queryCoverageShare: round( - cases.reduce((sum, entry) => sum + entry.queryCoverageTokens, 0) / - cases.reduce((sum, entry) => sum + entry.responseTokens, 0), - 3, - ), - coverageExpectedChecks: cases.filter( - (entry) => entry.coverageExpectedCorrect !== null, - ).length, - coverageExpectedPassed: cases.filter( - (entry) => entry.coverageExpectedCorrect === true, - ).length, - coverageDiscriminatingCases: cases.filter( - (entry) => entry.coverageDiscriminates, - ).length, - roundTrips: cases.length, - summedLatencyMs: round( - cases.reduce((sum, entry) => sum + entry.latencyMs, 0), - 1, - ), - }, - categories, - cases, - }; -} diff --git a/eval/current-version/discovery-development.json b/eval/current-version/discovery-development.json deleted file mode 100644 index f63f7290..00000000 --- a/eval/current-version/discovery-development.json +++ /dev/null @@ -1,81 +0,0 @@ -{ - "schemaVersion": 1, - "name": "Connecta lexical discovery development corpus", - "authorship": { - "createdFor": "https://github.com/zackbart/connecta/issues/322", - "source": "Synthetic reproduction of the mixed all/partial Mixpanel-shaped failure recorded in issue #326.", - "policy": "This corpus is development evidence. Ranking work may use it. The separate discovery-holdout.json remains qualification-only." - }, - "connectors": [ - { - "id": "analytics", - "description": "Product analytics projects and organizations", - "tools": [ - { - "name": "List-Organizations", - "description": "List organizations available to the caller" - }, - { - "name": "List-All-Organizations", - "description": "List organizations available to the caller" - }, - { - "name": "business_context_0", - "description": "List organizations and projects configured for business analysis" - }, - { - "name": "business_context_1", - "description": "List organizations and projects configured for business analysis" - }, - { - "name": "business_context_2", - "description": "List organizations and projects configured for business analysis" - }, - { - "name": "business_context_3", - "description": "List organizations and projects configured for business analysis" - }, - { - "name": "business_context_4", - "description": "List organizations and projects configured for business analysis" - }, - { - "name": "business_context_5", - "description": "List organizations and projects configured for business analysis" - }, - { - "name": "business_context_6", - "description": "List organizations and projects configured for business analysis" - }, - { - "name": "business_context_7", - "description": "List organizations and projects configured for business analysis" - }, - { - "name": "project_note_0", - "description": "Inspect one project note" - }, - { - "name": "project_note_1", - "description": "Inspect one project note" - } - ] - } - ], - "queries": [ - { - "id": "mixed-all-partial-organizations", - "category": "mixed-all-partial", - "query": "list organizations projects", - "relevant": ["analytics.List-Organizations"], - "expectedTop": "analytics.List-Organizations" - }, - { - "id": "exact-raw-name-frame", - "category": "mixed-all-partial", - "query": "list all organizations projects", - "relevant": ["analytics.List-All-Organizations"], - "expectedTop": "analytics.List-All-Organizations" - } - ] -} diff --git a/eval/current-version/discovery-holdout.json b/eval/current-version/discovery-holdout.json deleted file mode 100644 index 3bf8b074..00000000 --- a/eval/current-version/discovery-holdout.json +++ /dev/null @@ -1,529 +0,0 @@ -{ - "schemaVersion": 1, - "name": "Connecta lexical discovery holdout", - "authorship": { - "createdFor": "https://github.com/zackbart/connecta/issues/189", - "independence": "Authored from Connecta's public discovery contract before inspecting the closed #188 research corpus.", - "policy": "This corpus is release qualification evidence. Do not tune ranking rules, stopwords, aliases, or thresholds against its cases." - }, - "connectors": [ - { - "id": "projects", - "description": "Project planning and issue tracking", - "tools": [ - { - "name": "list_issues", - "description": "List project issues with optional state and label filters" - }, - { - "name": "get_issue", - "description": "Get one issue by its numeric identifier" - }, - { - "name": "create_issue", - "description": "Create a new project issue" - }, - { - "name": "list_pull_requests", - "description": "List pull requests and their review state" - }, - { - "name": "get_pull_request", - "description": "Get pull request details and merge status" - }, - { - "name": "add_issue_comment", - "description": "Add a comment to an issue" - } - ] - }, - { - "id": "messages", - "description": "Team messages and channels", - "tools": [ - { - "name": "search_messages", - "description": "Search message text across conversations and channels" - }, - { - "name": "send_message", - "description": "Send a new message to a channel" - }, - { - "name": "list_channels", - "description": "List channels visible to the current account" - }, - { - "name": "get_thread", - "description": "Get all replies in a message thread" - }, - { - "name": "add_reaction", - "description": "Add an emoji reaction to a message" - }, - { - "name": "list_members", - "description": "List members of a channel" - } - ] - }, - { - "id": "files", - "description": "Cloud files and folders", - "tools": [ - { - "name": "search_files", - "description": "Search files and folders by name or contents" - }, - { - "name": "upload_file", - "description": "Upload a file into a folder" - }, - { - "name": "download_file", - "description": "Download the bytes of a file" - }, - { - "name": "list_folder", - "description": "List the items inside a folder" - }, - { - "name": "share_file", - "description": "Share a file with another person" - }, - { - "name": "move_file", - "description": "Move a file to a different folder" - } - ] - }, - { - "id": "calendar", - "description": "Calendars, meetings, and availability", - "tools": [ - { - "name": "list_events", - "description": "List calendar events within a date range" - }, - { - "name": "get_event", - "description": "Get one calendar event" - }, - { - "name": "create_event", - "description": "Create a calendar event or meeting" - }, - { - "name": "update_event", - "description": "Update an existing calendar event" - }, - { - "name": "find_available_times", - "description": "Find free meeting times for attendees" - }, - { - "name": "list_calendars", - "description": "List calendars visible to the current account" - } - ] - }, - { - "id": "customers", - "description": "Customer relationship records", - "tools": [ - { - "name": "search_contacts", - "description": "Search customer and contact records" - }, - { - "name": "get_contact", - "description": "Get one customer contact record" - }, - { - "name": "create_contact", - "description": "Create a customer contact" - }, - { - "name": "list_companies", - "description": "List customer companies" - }, - { - "name": "update_company", - "description": "Update a customer company record" - }, - { - "name": "list_deals", - "description": "List sales deals and pipeline stages" - } - ] - }, - { - "id": "documents", - "description": "Workspace documents and pages", - "tools": [ - { - "name": "search_content", - "description": "Search document and page contents" - }, - { - "name": "get_page", - "description": "Get a workspace page" - }, - { - "name": "create_page", - "description": "Create a new document page" - }, - { - "name": "update_page", - "description": "Update a document page" - }, - { - "name": "list_databases", - "description": "List workspace databases" - }, - { - "name": "query_database", - "description": "Query rows from a workspace database" - } - ] - }, - { - "id": "builds", - "description": "Source control automation and build runs", - "tools": [ - { - "name": "list_workflow_runs", - "description": "List continuous integration workflow runs" - }, - { - "name": "get_workflow_run", - "description": "Get the status and conclusion of a workflow run" - }, - { - "name": "rerun_failed_jobs", - "description": "Rerun failed jobs in a workflow" - }, - { - "name": "list_commits", - "description": "List source control commits" - }, - { - "name": "compare_commits", - "description": "Compare two commits and return changed files" - }, - { - "name": "get_job_logs", - "description": "Get logs for a build job" - } - ] - }, - { - "id": "finance", - "description": "Business payments and accounting", - "tools": [ - { - "name": "list_transactions", - "description": "List account transactions and payments" - }, - { - "name": "get_invoice", - "description": "Get one customer invoice" - }, - { - "name": "create_invoice", - "description": "Create a customer invoice" - }, - { - "name": "list_accounts", - "description": "List financial accounts" - }, - { - "name": "get_balance", - "description": "Get the current account balance" - }, - { - "name": "search_customers", - "description": "Search billing customer records" - } - ] - } - ], - "queries": [ - { - "id": "direct-list-issues", - "category": "direct", - "query": "list issues", - "relevant": ["projects.list_issues"], - "expectedTop": "projects.list_issues" - }, - { - "id": "direct-search-messages", - "category": "direct", - "query": "search messages", - "relevant": ["messages.search_messages"], - "expectedTop": "messages.search_messages" - }, - { - "id": "direct-download-file", - "category": "direct", - "query": "download file", - "relevant": ["files.download_file"], - "expectedTop": "files.download_file" - }, - { - "id": "direct-available-times", - "category": "direct", - "query": "available times", - "relevant": ["calendar.find_available_times"], - "expectedTop": "calendar.find_available_times" - }, - { - "id": "direct-contact-search", - "category": "direct", - "query": "search contacts", - "relevant": ["customers.search_contacts"], - "expectedTop": "customers.search_contacts" - }, - { - "id": "direct-query-database", - "category": "direct", - "query": "query database", - "relevant": ["documents.query_database"], - "expectedTop": "documents.query_database" - }, - { - "id": "direct-failed-jobs", - "category": "direct", - "query": "rerun failed jobs", - "relevant": ["builds.rerun_failed_jobs"], - "expectedTop": "builds.rerun_failed_jobs" - }, - { - "id": "direct-account-balance", - "category": "direct", - "query": "account balance", - "relevant": ["finance.get_balance"], - "expectedTop": "finance.get_balance" - }, - { - "id": "conversation-open-issues", - "category": "conversational", - "query": "Can you show me the open issues in this project?", - "relevant": ["projects.list_issues"], - "expectedTop": "projects.list_issues" - }, - { - "id": "conversation-message-channel", - "category": "conversational", - "query": "I need to send a message to a channel", - "relevant": ["messages.send_message"], - "expectedTop": "messages.send_message" - }, - { - "id": "conversation-file-folder", - "category": "conversational", - "query": "Please upload a file into this folder", - "relevant": ["files.upload_file"], - "expectedTop": "files.upload_file" - }, - { - "id": "conversation-meeting", - "category": "conversational", - "query": "Find a free time for everyone to meet", - "relevant": ["calendar.find_available_times"], - "expectedTop": "calendar.find_available_times" - }, - { - "id": "conversation-sales-pipeline", - "category": "conversational", - "query": "What deals are currently in the sales pipeline?", - "relevant": ["customers.list_deals"], - "expectedTop": "customers.list_deals" - }, - { - "id": "conversation-ci-status", - "category": "conversational", - "query": "Show me whether the latest workflow run passed", - "relevant": ["builds.get_workflow_run"], - "expectedTop": "builds.get_workflow_run" - }, - { - "id": "conversation-customer-invoice", - "category": "conversational", - "query": "I want to create an invoice for a customer", - "relevant": ["finance.create_invoice"], - "expectedTop": "finance.create_invoice" - }, - { - "id": "conversation-page-content", - "category": "conversational", - "query": "Find the page that mentions our launch plan", - "relevant": ["documents.search_content"], - "expectedTop": "documents.search_content" - }, - { - "id": "multi-find-download", - "category": "multi-intent", - "query": "find and download a file", - "relevant": ["files.search_files", "files.download_file"], - "expectedTop": "files.download_file" - }, - { - "id": "multi-issue-comment", - "category": "multi-intent", - "query": "get an issue and add a comment", - "relevant": ["projects.get_issue", "projects.add_issue_comment"] - }, - { - "id": "multi-calendar-update", - "category": "multi-intent", - "query": "list calendar events and update one", - "relevant": ["calendar.list_events", "calendar.update_event"] - }, - { - "id": "multi-build-diagnosis", - "category": "multi-intent", - "query": "get the failed workflow run and its job logs", - "relevant": ["builds.get_workflow_run", "builds.get_job_logs"] - }, - { - "id": "short-find-file", - "category": "short-function-word", - "query": "find a file in storage", - "relevant": ["files.search_files"], - "expectedTop": "files.search_files" - }, - { - "id": "short-send-channel", - "category": "short-function-word", - "query": "send a message to a channel", - "relevant": ["messages.send_message"], - "expectedTop": "messages.send_message" - }, - { - "id": "short-list-prs", - "category": "short-function-word", - "query": "list the pull requests in a project", - "relevant": ["projects.list_pull_requests"], - "expectedTop": "projects.list_pull_requests" - }, - { - "id": "short-get-event", - "category": "short-function-word", - "query": "get an event from the calendar", - "relevant": ["calendar.get_event"], - "expectedTop": "calendar.get_event" - }, - { - "id": "cleanup-only", - "category": "empty-after-cleanup", - "query": "a an and are as at be by for from in into is it of on or that the this to was with", - "relevant": [] - }, - { - "id": "negative-flight", - "category": "negative", - "query": "book an airline flight", - "relevant": [] - }, - { - "id": "negative-food", - "category": "negative", - "query": "order a pepperoni pizza", - "relevant": [] - }, - { - "id": "negative-weather", - "category": "negative", - "query": "will it rain tomorrow", - "relevant": [] - }, - { - "id": "negative-audio", - "category": "negative", - "query": "transcribe this audio recording", - "relevant": [] - }, - { - "id": "filtered-list-finance", - "category": "connector-filtered", - "query": "list", - "connector": "finance", - "relevant": ["finance.list_transactions", "finance.list_accounts"] - }, - { - "id": "filtered-search-documents", - "category": "connector-filtered", - "query": "search", - "connector": "documents", - "relevant": ["documents.search_content"], - "expectedTop": "documents.search_content" - }, - { - "id": "filtered-get-calendar", - "category": "connector-filtered", - "query": "get", - "connector": "calendar", - "relevant": ["calendar.get_event"], - "expectedTop": "calendar.get_event" - }, - { - "id": "page-list-first", - "category": "paginated", - "query": "list", - "limit": 4, - "offset": 0, - "expectedPage": [ - "projects.list_issues", - "projects.list_pull_requests", - "messages.list_channels", - "messages.list_members" - ], - "relevant": [ - "projects.list_issues", - "projects.list_pull_requests", - "messages.list_channels", - "messages.list_members", - "files.list_folder", - "calendar.list_events", - "calendar.list_calendars", - "customers.list_companies", - "customers.list_deals", - "documents.list_databases", - "builds.list_workflow_runs", - "builds.list_commits", - "finance.list_transactions", - "finance.list_accounts" - ] - }, - { - "id": "page-list-second", - "category": "paginated", - "query": "list", - "limit": 4, - "offset": 4, - "expectedPage": [ - "files.list_folder", - "calendar.list_events", - "calendar.list_calendars", - "customers.list_companies" - ], - "relevant": [ - "projects.list_issues", - "projects.list_pull_requests", - "messages.list_channels", - "messages.list_members", - "files.list_folder", - "calendar.list_events", - "calendar.list_calendars", - "customers.list_companies", - "customers.list_deals", - "documents.list_databases", - "builds.list_workflow_runs", - "builds.list_commits", - "finance.list_transactions", - "finance.list_accounts" - ] - } - ] -} diff --git a/eval/current-version/eval-tracing.ts b/eval/current-version/eval-tracing.ts deleted file mode 100644 index 0c2d84a4..00000000 --- a/eval/current-version/eval-tracing.ts +++ /dev/null @@ -1,268 +0,0 @@ -/** - * The eval lane's shared observation layer. - * - * Both isolated servers — the fixture sandbox and the reference-connection - * sandbox — must be watched the same way, or a number measured against one - * cannot be read beside a number measured against the other. Extracting the - * instrumentation is what makes "same scoring, different catalog" true rather - * than merely intended: the harness reads `/__eval/trace` identically from - * either server, and `agent-benchmark-scoring.mjs` cannot tell them apart. - * - * Nothing here decides what is measured. It records outer meta-tool calls, - * meta-tool calls made from inside `execute_code`, and downstream executions, - * and leaves every verdict to the scorer. - */ -import type { - AdmittingExecutor, - Connecta, - ExecutorProvider, - ToolCallActivityEvent, -} from "../../src/index.js"; - -export type EvalTraceSource = "outer" | "execute_code"; - -export interface EvalMetaToolTrace { - schemaVersion: 1; - sequence: number; - kind: "meta_tool"; - source: EvalTraceSource; - operation: string; - arguments: unknown; - result?: unknown; - error?: string; - durationMs: number; -} - -export interface EvalExecutionTrace { - schemaVersion: 1; - sequence: number; - kind: "execution"; - address: string; - source: ToolCallActivityEvent["source"]; - outcome: ToolCallActivityEvent["outcome"]; - durationMs: number; - attempts: number; - errorCode?: string; -} - -export type EvalTrace = EvalMetaToolTrace | EvalExecutionTrace; - -export type EvalTraceInput = - | Omit - | Omit; - -export interface EvalTracing { - /** Every trace recorded so far, in emission order. */ - readonly traces: EvalTrace[]; - /** Record one trace. A no-op when tracing is disabled. */ - emitTrace(trace: EvalTraceInput): void; - /** Wrap an executor so meta-tool calls inside programs are observed. */ - tracedExecutor(base: AdmittingExecutor): AdmittingExecutor; - /** Wrap a deployment so outer `tools/call` requests are observed. */ - withOuterTracing(connecta: Connecta): Connecta; -} - -function errorMessage(error: unknown): string { - return error instanceof Error ? error.message : String(error); -} - -/** - * Map a guest-visible provider function onto the meta-tool name it stands for. - * - * A search run inside `execute_code` is still discovery work, and scoring it as - * anything else would let code mode hide the learning cost it exists to reduce. - */ -function providerOperation( - name: string, - args: unknown[], -): { operation: string; arguments: unknown } { - if (name === "search") { - return { operation: "search_tools", arguments: args[0] ?? {} }; - } - if (name === "describe") { - return { operation: "describe_tools", arguments: args[0] ?? {} }; - } - if (name === "call") { - return { - operation: "call_tool", - arguments: { address: String(args[0]), args: args[1] ?? {} }, - }; - } - if (name === "batch") { - return { operation: "batch_call", arguments: { calls: args[0] ?? [] } }; - } - if (name === "__callNamespace") { - return { - operation: "call_tool", - arguments: { - address: `${String(args[0])}.${String(args[1])}`, - args: args[2] ?? {}, - via: "namespace", - }, - }; - } - return { operation: name, arguments: args }; -} - -async function outerMetaToolCall( - request: Request, -): Promise<{ operation: string; arguments: unknown } | undefined> { - if (request.method !== "POST") return undefined; - try { - const body = (await request.clone().json()) as { - method?: unknown; - params?: { name?: unknown; arguments?: unknown }; - }; - if (body.method !== "tools/call" || typeof body.params?.name !== "string") { - return undefined; - } - return { - operation: body.params.name, - arguments: body.params.arguments ?? {}, - }; - } catch { - return undefined; - } -} - -/** - * Build the tracing layer for one isolated eval server. - * - * `token` guards `/__eval/trace`: the harness reads it with the same bearer the - * agent uses, and an unauthenticated reader gets nothing. - */ -export function createEvalTracing(options: { - enabled: boolean; - token: string; -}): EvalTracing { - const traces: EvalTrace[] = []; - let sequence = 0; - - function emitTrace(trace: EvalTraceInput): void { - if (!options.enabled) return; - const event = { - schemaVersion: 1, - sequence: ++sequence, - ...trace, - } as EvalTrace; - traces.push(event); - console.log(JSON.stringify({ event: "eval_trace", trace: event })); - } - - function tracedProviders( - providers: ExecutorProvider[], - ): ExecutorProvider[] { - return providers.map((provider) => ({ - ...provider, - fns: Object.fromEntries( - Object.entries(provider.fns).map(([name, fn]) => [ - name, - async (...args: unknown[]) => { - const operation = providerOperation(name, args); - const started = performance.now(); - try { - const result = await fn(...args); - emitTrace({ - kind: "meta_tool", - source: "execute_code", - ...operation, - result, - durationMs: performance.now() - started, - }); - return result; - } catch (error) { - emitTrace({ - kind: "meta_tool", - source: "execute_code", - ...operation, - error: errorMessage(error), - durationMs: performance.now() - started, - }); - throw error; - } - }, - ]), - ), - })); - } - - function tracedExecutor(base: AdmittingExecutor): AdmittingExecutor { - return { - async acquire(acquireOptions = {}) { - const lease = await base.acquire(acquireOptions); - return { - ...(lease.waitMs !== undefined ? { waitMs: lease.waitMs } : {}), - execute: (code, providers) => - lease.execute(code, tracedProviders(providers)), - release: () => lease.release(), - }; - }, - execute: (code, providers) => - base.execute(code, tracedProviders(providers)), - ...(base.admissionSnapshot - ? { admissionSnapshot: () => base.admissionSnapshot!() } - : {}), - close: () => base.close?.(), - }; - } - - function withOuterTracing(connecta: Connecta): Connecta { - return { - ...connecta, - async fetch(request, env, ctx) { - const url = new URL(request.url); - if (request.method === "GET" && url.pathname === "/__eval/trace") { - if ( - request.headers.get("authorization") !== `Bearer ${options.token}` - ) { - return Response.json({ error: "unauthorized" }, { status: 401 }); - } - return Response.json({ traces }); - } - const operation = await outerMetaToolCall(request); - const started = performance.now(); - try { - const response = await connecta.fetch(request, env, ctx); - if (!operation) return response; - let payload: { - result?: unknown; - error?: { message?: unknown }; - } = {}; - try { - payload = (await response.clone().json()) as typeof payload; - } catch { - // A malformed transport result is still traced below as an error. - } - emitTrace({ - kind: "meta_tool", - source: "outer", - ...operation, - ...(payload.result !== undefined - ? { result: payload.result } - : { - error: - typeof payload.error?.message === "string" - ? payload.error.message - : `HTTP ${response.status}`, - }), - durationMs: performance.now() - started, - }); - return response; - } catch (error) { - if (operation) { - emitTrace({ - kind: "meta_tool", - source: "outer", - ...operation, - error: errorMessage(error), - durationMs: performance.now() - started, - }); - } - throw error; - } - }, - }; - } - - return { traces, emitTrace, tracedExecutor, withOuterTracing }; -} diff --git a/eval/current-version/issue-322-qualification-plan.json b/eval/current-version/issue-322-qualification-plan.json deleted file mode 100644 index e8d1ec78..00000000 --- a/eval/current-version/issue-322-qualification-plan.json +++ /dev/null @@ -1,109 +0,0 @@ -{ - "schemaVersion": 1, - "status": "preregistered-before-sampling", - "createdAt": "2026-08-10T05:00:00.000Z", - "issue": "https://github.com/zackbart/connecta/issues/322", - "arms": { - "off": { - "baseCommit": "PREREGISTRATION_COMMIT", - "productBaseCommit": "62e2b1f0f6ec681cd3049a3a12621ab3d6978ff6", - "patch": "patches/issue-322-coverage-off.patch", - "patchSha256": "9db0c8011ea3743a0d605aa86fa0842c769125f89006f05e768e6080a522226f" - }, - "trailing": { - "productCommit": "bbfb5220cb94342acc21dadd7db9fe1bbcf5ce4c", - "evalOverlayCommit": "PREREGISTRATION_COMMIT", - "evalOverlayPath": "eval/current-version" - } - }, - "environment": { - "model": "gpt-5.6-sol", - "codexVersion": "codex-cli 0.147.0", - "nodeVersion": "26.5.1", - "tokenizer": "o200k_base", - "case": "mixed-decoy-organizations", - "repetitionsPerArm": 30, - "batchRepetitions": 5, - "batchesPerArm": 6, - "concurrency": 5, - "harnessSha256": "dd11bb3b16a3b99d481e26983485787fb10dfa2c43db59ad6655c1944f7810c3", - "corpusSha256": "48006378093890eaac28c61540a94bd4ee8c9e2d48e59aada92178539b28fdd1", - "sandboxSha256": "7a8b811f4e241db3209b3490fa4642795a2b7b80e9a23660b01d34e54821a11b", - "requiredHostActions": 0, - "requiredForeignCalls": 0 - }, - "scheduling": { - "method": "Six five-run batches per arm. Each pair contains both arms; fixed seeded order was generated before sampling.", - "seed": "issue-322-bbfb522-20260810", - "batchOrder": [ - "trailing", - "off", - "off", - "trailing", - "trailing", - "off", - "off", - "trailing", - "trailing", - "off", - "off", - "trailing" - ] - }, - "primaryGates": [ - { - "id": "combined-noninferiority", - "metric": "routingResultCorrect rate", - "criterion": "trailing - off >= -0.10" - }, - { - "id": "clean-route-improvement", - "metric": "routeClean rate", - "criterion": "trailing - off >= 0.20 and two-sided Fisher exact p < 0.05" - }, - { - "id": "mean-efficiency", - "metrics": [ - "whole-agent input tokens", - "non-cached input tokens", - "Connecta round trips" - ], - "criterion": "each trailing mean / off mean <= 1.10" - }, - { - "id": "median-efficiency", - "metrics": [ - "whole-agent input tokens", - "non-cached input tokens", - "Connecta round trips" - ], - "criterion": "each trailing median / off median <= 1.10" - }, - { - "id": "latency", - "metric": "mean wall latency", - "criterion": "trailing mean / off mean <= 1.10" - }, - { - "id": "isolation", - "metric": "host actions and foreign MCP calls", - "criterion": "both totals equal zero in both arms" - } - ], - "secondaryMetrics": [ - "median latency", - "search-result tokens", - "estimated search-noise tokens", - "Connecta MCP result tokens", - "retrieval top-1", - "retrieval recall", - "exact address", - "exact arguments", - "final answer" - ], - "analysis": { - "fisher": "Two-sided Fisher exact test sums all fixed-margin tables with probability less than or equal to the observed table probability.", - "passRule": "Every primary gate must pass. Search and MCP token movement cannot offset a failed primary gate.", - "failedRunRule": "A harness or infrastructure failure aborts the protocol. Do not replace individual completed runs or tune gates after results." - } -} diff --git a/eval/current-version/issue-322-qualification-runner.mjs b/eval/current-version/issue-322-qualification-runner.mjs deleted file mode 100644 index 8487b480..00000000 --- a/eval/current-version/issue-322-qualification-runner.mjs +++ /dev/null @@ -1,511 +0,0 @@ -import { spawn, execFileSync } from "node:child_process"; -import { createHash } from "node:crypto"; -import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; -import { dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -const here = dirname(fileURLToPath(import.meta.url)); -const args = process.argv.slice(2); - -function option(name, fallback) { - const index = args.indexOf(name); - if (index < 0) return fallback; - const value = args[index + 1]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value.`); - } - return value; -} - -function sha256(value) { - return createHash("sha256").update(value).digest("hex"); -} - -function mean(values) { - return values.reduce((sum, value) => sum + value, 0) / values.length; -} - -function median(values) { - const ordered = [...values].sort((left, right) => left - right); - const middle = Math.floor(ordered.length / 2); - return ordered.length % 2 === 0 - ? (ordered[middle - 1] + ordered[middle]) / 2 - : ordered[middle]; -} - -function round(value, places = 3) { - const scale = 10 ** places; - return Math.round(value * scale) / scale; -} - -function logChoose(n, k) { - if (k < 0 || k > n) return Number.NEGATIVE_INFINITY; - const selected = Math.min(k, n - k); - let result = 0; - for (let index = 1; index <= selected; index += 1) { - result += Math.log(n - selected + index) - Math.log(index); - } - return result; -} - -function fisherProbability(a, rowOne, columnOne, total) { - return Math.exp( - logChoose(columnOne, a) + - logChoose(total - columnOne, rowOne - a) - - logChoose(total, rowOne), - ); -} - -function fisherTwoSided(a, b, c, d) { - const rowOne = a + b; - const rowTwo = c + d; - const columnOne = a + c; - const total = rowOne + rowTwo; - const minimum = Math.max(0, rowOne - (total - columnOne)); - const maximum = Math.min(rowOne, columnOne); - const observed = fisherProbability(a, rowOne, columnOne, total); - let sum = 0; - for (let candidate = minimum; candidate <= maximum; candidate += 1) { - const probability = fisherProbability( - candidate, - rowOne, - columnOne, - total, - ); - if (probability <= observed + 1e-12) sum += probability; - } - return Math.min(1, sum); -} - -const planPath = resolve(here, "issue-322-qualification-plan.json"); -const planText = await readFile(planPath, "utf8"); -const plan = JSON.parse(planText); -const offWorktree = resolve(option("--off-worktree")); -const trailingWorktree = resolve(option("--trailing-worktree")); -const outputOption = option("--output-dir"); -const outputDirectory = outputOption - ? resolve(outputOption) - : resolve(here, "results"); -const batchDirectory = resolve(outputDirectory, ".issue-322-batches"); -const model = process.env.CONNECTA_EVAL_AGENT_MODEL; -const nodeVersion = process.versions.node; -const codexVersion = execFileSync("codex", ["--version"], { - encoding: "utf8", -}).trim(); - -if (model !== plan.environment.model) { - throw new Error( - `CONNECTA_EVAL_AGENT_MODEL must be ${plan.environment.model}; got ${model ?? "unset"}.`, - ); -} -if (nodeVersion !== plan.environment.nodeVersion) { - throw new Error( - `Node must be ${plan.environment.nodeVersion}; got ${nodeVersion}.`, - ); -} -if (codexVersion !== plan.environment.codexVersion) { - throw new Error( - `Codex must be ${plan.environment.codexVersion}; got ${codexVersion}.`, - ); -} - -const armPaths = { off: offWorktree, trailing: trailingWorktree }; -const preregistrationCommit = execFileSync("git", ["rev-parse", "HEAD"], { - cwd: resolve(here, "../.."), - encoding: "utf8", -}).trim(); -for (const [arm, worktree] of Object.entries(armPaths)) { - const evalRoot = resolve(worktree, "eval/current-version"); - const hashes = { - harnessSha256: sha256( - await readFile(resolve(evalRoot, "agent-lookup-benchmark.mjs")), - ), - corpusSha256: sha256( - `${await readFile(resolve(evalRoot, "discovery-holdout.json"), "utf8")}\0${await readFile(resolve(evalRoot, "discovery-development.json"), "utf8")}`, - ), - sandboxSha256: sha256( - await readFile(resolve(evalRoot, "sandbox-server.ts")), - ), - }; - for (const [key, expected] of Object.entries({ - harnessSha256: plan.environment.harnessSha256, - corpusSha256: plan.environment.corpusSha256, - sandboxSha256: plan.environment.sandboxSha256, - })) { - if (hashes[key] !== expected) { - throw new Error(`${arm} ${key} mismatch: ${hashes[key]} != ${expected}.`); - } - } -} - -const offCommit = execFileSync("git", ["rev-parse", "HEAD"], { - cwd: offWorktree, - encoding: "utf8", -}).trim(); -if (offCommit !== preregistrationCommit) { - throw new Error( - `Coverage-off base must equal preregistration commit ${preregistrationCommit}; got ${offCommit}.`, - ); -} -const trailingCommit = execFileSync("git", ["rev-parse", "HEAD"], { - cwd: trailingWorktree, - encoding: "utf8", -}).trim(); -if (trailingCommit !== plan.arms.trailing.productCommit) { - throw new Error(`Trailing source mismatch: ${trailingCommit}.`); -} -const offPatch = execFileSync( - "git", - ["diff", "--", "src/catalog-service.ts"], - { cwd: offWorktree, encoding: "utf8" }, -); -if (sha256(offPatch) !== plan.arms.off.patchSha256) { - throw new Error(`Coverage-off patch mismatch: ${sha256(offPatch)}.`); -} - -async function runBatch(arm, batchIndex) { - const prefix = `issue-322-${arm}-batch-${String(batchIndex).padStart(2, "0")}`; - const output = resolve(batchDirectory, `${prefix}.json`); - const report = resolve(batchDirectory, `${prefix}.md`); - const worktree = armPaths[arm]; - const commandArgs = [ - "--prefix", - "eval/current-version", - "run", - "perf:lookup", - "--", - "--case", - plan.environment.case, - "--repetitions", - String(plan.environment.batchRepetitions), - "--concurrency", - String(plan.environment.concurrency), - "--output", - output, - "--report", - report, - ]; - const startedAt = new Date().toISOString(); - const child = spawn("npm", commandArgs, { - cwd: worktree, - env: { - ...process.env, - CONNECTA_EVAL_AGENT_MODEL: plan.environment.model, - }, - stdio: ["ignore", "pipe", "inherit"], - }); - let stdout = ""; - child.stdout.setEncoding("utf8"); - child.stdout.on("data", (chunk) => { - stdout += chunk; - process.stdout.write(chunk); - }); - const exitCode = await new Promise((resolveExit, rejectExit) => { - child.once("error", rejectExit); - child.once("exit", resolveExit); - }); - if (exitCode !== 0) { - throw new Error(`${arm} batch ${batchIndex} exited with ${exitCode}.`); - } - const artifactText = await readFile(output, "utf8"); - const artifact = JSON.parse(artifactText); - return { - arm, - batchIndex, - startedAt, - completedAt: new Date().toISOString(), - artifactSha256: sha256(artifactText), - source: artifact.source, - configuration: artifact.configuration, - cases: artifact.cases, - stdoutSha256: sha256(stdout), - }; -} - -await rm(batchDirectory, { recursive: true, force: true }); -await mkdir(batchDirectory, { recursive: true }); -const armBatchCounts = { off: 0, trailing: 0 }; -const schedule = []; -for (const arm of plan.scheduling.batchOrder) { - armBatchCounts[arm] += 1; - schedule.push(await runBatch(arm, armBatchCounts[arm])); -} - -function summarize(cases) { - const metricValues = { - wholeInput: cases.map((entry) => entry.usage.input_tokens ?? 0), - nonCachedInput: cases.map((entry) => entry.nonCachedInputTokens), - roundTrips: cases.map((entry) => entry.connectaRoundTrips), - latency: cases.map((entry) => entry.latencyMs), - searchTokens: cases.map((entry) => entry.searchResultTokens), - searchNoiseTokens: cases.map( - (entry) => entry.estimatedLookupNoiseTokens, - ), - connectaTokens: cases.map((entry) => entry.connectaMcpResultTokens), - }; - const distributions = Object.fromEntries( - Object.entries(metricValues).map(([name, values]) => [ - name, - { - mean: round(mean(values), 1), - median: round(median(values), 1), - min: Math.min(...values), - max: Math.max(...values), - }, - ]), - ); - const count = (select) => cases.filter(select).length; - return { - runs: cases.length, - combinedCorrect: count((entry) => entry.routingResultCorrect), - cleanRoute: count((entry) => entry.routeClean), - addressCorrect: count((entry) => entry.addressAccurate), - argumentCorrect: count((entry) => entry.argumentCorrect), - finalCorrect: count((entry) => entry.finalCorrect), - retrievalTop1: count((entry) => entry.retrievalTop1), - retrievalRecallComplete: count((entry) => entry.retrievalRecall === 1), - hostActions: cases.reduce( - (sum, entry) => sum + entry.hostActionCount, - 0, - ), - foreignCalls: cases.reduce( - (sum, entry) => sum + entry.foreignToolCalls, - 0, - ), - distributions, - }; -} - -const arms = Object.fromEntries( - Object.keys(armPaths).map((arm) => { - const batches = schedule.filter((entry) => entry.arm === arm); - const cases = batches.flatMap((entry, batchIndex) => - entry.cases.map((run) => ({ - ...run, - repetition: batchIndex * plan.environment.batchRepetitions + run.repetition, - })), - ); - return [arm, { batches, summary: summarize(cases), cases }]; - }), -); - -for (const [arm, artifact] of Object.entries(arms)) { - if (artifact.cases.length !== plan.environment.repetitionsPerArm) { - throw new Error(`${arm} produced ${artifact.cases.length} runs.`); - } - if ( - artifact.summary.hostActions !== plan.environment.requiredHostActions || - artifact.summary.foreignCalls !== plan.environment.requiredForeignCalls - ) { - throw new Error(`${arm} violated host isolation.`); - } -} - -const off = arms.off.summary; -const trailing = arms.trailing.summary; -const rate = (successes, runs) => successes / runs; -const ratio = (candidate, baseline) => candidate / baseline; -const combinedDelta = - rate(trailing.combinedCorrect, trailing.runs) - - rate(off.combinedCorrect, off.runs); -const routeDelta = - rate(trailing.cleanRoute, trailing.runs) - - rate(off.cleanRoute, off.runs); -const fisherP = fisherTwoSided( - trailing.cleanRoute, - trailing.runs - trailing.cleanRoute, - off.cleanRoute, - off.runs - off.cleanRoute, -); -const meanRatios = Object.fromEntries( - ["wholeInput", "nonCachedInput", "roundTrips"].map((name) => [ - name, - round( - ratio( - trailing.distributions[name].mean, - off.distributions[name].mean, - ), - 4, - ), - ]), -); -const medianRatios = Object.fromEntries( - ["wholeInput", "nonCachedInput", "roundTrips"].map((name) => [ - name, - round( - ratio( - trailing.distributions[name].median, - off.distributions[name].median, - ), - 4, - ), - ]), -); -const latencyRatio = round( - ratio( - trailing.distributions.latency.mean, - off.distributions.latency.mean, - ), - 4, -); -const gates = [ - { - id: "combined-noninferiority", - actual: round(combinedDelta, 4), - minimum: -0.1, - passed: combinedDelta >= -0.1, - }, - { - id: "clean-route-improvement", - actualDifference: round(routeDelta, 4), - minimumDifference: 0.2, - fisherTwoSidedP: round(fisherP, 6), - maximumP: 0.05, - passed: routeDelta >= 0.2 && fisherP < 0.05, - }, - { - id: "mean-efficiency", - ratios: meanRatios, - maximumRatio: 1.1, - passed: Object.values(meanRatios).every((value) => value <= 1.1), - }, - { - id: "median-efficiency", - ratios: medianRatios, - maximumRatio: 1.1, - passed: Object.values(medianRatios).every((value) => value <= 1.1), - }, - { - id: "latency", - meanRatio: latencyRatio, - maximumRatio: 1.1, - passed: latencyRatio <= 1.1, - }, - { - id: "isolation", - off: { hostActions: off.hostActions, foreignCalls: off.foreignCalls }, - trailing: { - hostActions: trailing.hostActions, - foreignCalls: trailing.foreignCalls, - }, - passed: - off.hostActions === 0 && - off.foreignCalls === 0 && - trailing.hostActions === 0 && - trailing.foreignCalls === 0, - }, -]; - -const comparison = { - schemaVersion: 1, - generatedAt: new Date().toISOString(), - preregistration: { - plan: "issue-322-qualification-plan.json", - planSha256: sha256(planText), - commit: preregistrationCommit, - }, - environment: { - ...plan.environment, - nodeVersion, - codexVersion, - }, - schedule: schedule.map(({ arm, batchIndex, startedAt, completedAt, artifactSha256 }) => ({ - arm, - batchIndex, - startedAt, - completedAt, - artifactSha256, - })), - arms: { - off: off, - trailing: trailing, - }, - analysis: { - combinedDelta: round(combinedDelta, 4), - cleanRouteDelta: round(routeDelta, 4), - cleanRouteFisherTwoSidedP: round(fisherP, 6), - meanRatios, - medianRatios, - latencyMeanRatio: latencyRatio, - }, - gates, - passed: gates.every((gate) => gate.passed), -}; - -const armOutput = async (arm) => { - const artifact = { - schemaVersion: 1, - generatedAt: comparison.generatedAt, - preregistration: comparison.preregistration, - arm, - source: arms[arm].batches[0].source, - configuration: arms[arm].batches[0].configuration, - batches: arms[arm].batches.map( - ({ batchIndex, startedAt, completedAt, artifactSha256, stdoutSha256 }) => ({ - batchIndex, - startedAt, - completedAt, - artifactSha256, - stdoutSha256, - }), - ), - summary: arms[arm].summary, - cases: arms[arm].cases, - }; - await writeFile( - resolve(outputDirectory, `issue-322-preregistered-${arm}.json`), - `${JSON.stringify(artifact, null, 2)}\n`, - ); -}; - -await mkdir(outputDirectory, { recursive: true }); -await Promise.all([armOutput("off"), armOutput("trailing")]); -await writeFile( - resolve(outputDirectory, "issue-322-preregistered-comparison.json"), - `${JSON.stringify(comparison, null, 2)}\n`, -); - -const percentage = (value) => `${(value * 100).toFixed(1)}%`; -const markdown = `# Issue #322 off-vs-trailing qualification - -Plan SHA-256: \`${comparison.preregistration.planSha256}\` - -Preregistration commit: \`${comparison.preregistration.commit}\` - -Confirm remote timing before describing this commit as formal preregistration. - -Result: **${comparison.passed ? "PASS" : "FAIL"}** - -| Gate | Result | -| --- | --- | -${gates.map((gate) => `| ${gate.id} | ${gate.passed ? "pass" : "FAIL"} |`).join("\n")} - -## Correctness - -| Metric | Off | Trailing | Movement | -| --- | ---: | ---: | ---: | -| Combined exact result | ${off.combinedCorrect}/${off.runs} | ${trailing.combinedCorrect}/${trailing.runs} | ${percentage(combinedDelta)} | -| Clean intended route | ${off.cleanRoute}/${off.runs} | ${trailing.cleanRoute}/${trailing.runs} | ${percentage(routeDelta)} | -| Clean-route Fisher p | — | — | ${comparison.analysis.cleanRouteFisherTwoSidedP} | - -## Efficiency - -| Metric | Off mean | Trailing mean | Ratio | Off median | Trailing median | Ratio | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -${["wholeInput", "nonCachedInput", "roundTrips", "latency", "searchTokens", "connectaTokens"].map((name) => { - const offMetric = off.distributions[name]; - const trailingMetric = trailing.distributions[name]; - return `| ${name} | ${offMetric.mean} | ${trailingMetric.mean} | ${round(trailingMetric.mean / offMetric.mean, 3)} | ${offMetric.median} | ${trailingMetric.median} | ${round(trailingMetric.median / offMetric.median, 3)} |`; -}).join("\n")} - -Search and Connecta MCP tokens are reported but do not offset a failed primary -gate. Every arm used 30 fresh sessions in the predeclared six-by-five batch -schedule with concurrency five. Host actions and foreign calls were zero. -`; -await writeFile( - resolve(outputDirectory, "issue-322-preregistered-comparison.md"), - markdown, -); -await rm(batchDirectory, { recursive: true, force: true }); -process.stdout.write(`${JSON.stringify({ event: "qualification_complete", comparison })}\n`); -if (!comparison.passed) process.exitCode = 1; diff --git a/eval/current-version/issue-419-fixtures.mjs b/eval/current-version/issue-419-fixtures.mjs deleted file mode 100644 index 77ac3d91..00000000 --- a/eval/current-version/issue-419-fixtures.mjs +++ /dev/null @@ -1,80 +0,0 @@ -export const fixtures = [ - { - id: "valid-js", - group: "javascript", - code: "const value = 40; return value + 2;", - expected: 42, - }, - { - id: "annotation", - group: "candidate", - code: "const value: number = 40; return value + 2;", - expected: 42, - }, - { - id: "return-type", - group: "candidate", - code: "const answer = (): number => 42; return answer();", - expected: 42, - }, - { - id: "as-assertion", - group: "candidate", - code: "const value = 42 as number; return value;", - expected: 42, - }, - { - id: "type-alias", - group: "candidate", - code: "type Count = number; const value: Count = 42; return value;", - expected: 42, - }, - { - id: "interface", - group: "candidate", - code: "interface Box { value: number } const box: Box = { value: 42 }; return box.value;", - expected: 42, - }, - { - id: "erased-generic", - group: "candidate", - code: "const identity = (value: T): T => value; return identity(42);", - expected: 42, - }, - { - id: "malformed-ts", - group: "malformed", - code: "const value: = 42; return value;", - }, - { - id: "fenced-ts", - group: "candidate", - code: "```typescript\nconst value: number = 42;\nreturn value;\n```", - expected: 42, - }, - { - id: "enum", - group: "unsupported", - code: "enum Answer { Value = 42 } return Answer.Value;", - }, - { - id: "decorator", - group: "unsupported", - code: "const mark = (value: unknown) => value; @mark class Answer {} return 42;", - }, - { - id: "namespace", - group: "unsupported", - code: "namespace Answer { export const value = 42 } return Answer.value;", - }, - { - id: "jsx", - group: "unsupported", - code: "const node =
42
; return node;", - }, - { - id: "import", - group: "unsupported", - code: "import value from 'elsewhere'; return value;", - }, -]; diff --git a/eval/current-version/issue-419-ts-eval.mjs b/eval/current-version/issue-419-ts-eval.mjs deleted file mode 100644 index e566d8d5..00000000 --- a/eval/current-version/issue-419-ts-eval.mjs +++ /dev/null @@ -1,160 +0,0 @@ -import assert from "node:assert/strict"; -import { readdir, readFile, stat, writeFile } from "node:fs/promises"; -import { dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; -import { performance } from "node:perf_hooks"; -import tsBlankSpace from "ts-blank-space"; -import { transform as sucraseTransform } from "sucrase"; -import ts from "typescript"; -import { fixtures } from "./issue-419-fixtures.mjs"; -import { normalizeCode } from "../../dist/executors/quickjs-runtime.js"; - -const here = dirname(fileURLToPath(import.meta.url)); -const iterations = 10_000; - -function unwrapFence(code) { - const trimmed = code.trim(); - const match = /^```[\w-]*\s*\n([\s\S]*?)\n?```$/.exec(trimmed); - return match?.[1]?.trim() ?? trimmed; -} - -const candidates = { - "ts-blank-space": (code) => { - const source = unwrapFence(code); - const parsed = ts.createSourceFile("guest.ts", source, ts.ScriptTarget.ESNext, true, ts.ScriptKind.TS); - const diagnostic = parsed.parseDiagnostics[0]; - if (diagnostic) throw new SyntaxError(ts.flattenDiagnosticMessageText(diagnostic.messageText, " ")); - return tsBlankSpace(source, (node) => { - throw new SyntaxError(`Unsupported TypeScript syntax: ${ts.SyntaxKind[node.kind]}`); - }); - }, - sucrase: (code) => sucraseTransform(unwrapFence(code), { transforms: ["typescript"] }).code, - typescript: (code) => ts.transpileModule(unwrapFence(code), { - compilerOptions: { target: ts.ScriptTarget.ESNext, module: ts.ModuleKind.ESNext }, - reportDiagnostics: true, - }).outputText, -}; - -async function run(code) { - const normalized = normalizeCode(code); - const fn = Function(`"use strict"; return (${normalized});`)(); - return await fn(); -} - -async function evaluateFixture(fixture, transform) { - try { - const output = transform ? transform(fixture.code) : fixture.code; - const result = await run(output); - return { - status: "ran", - result, - outputBytes: Buffer.byteLength(output), - byteDelta: Buffer.byteLength(output) - Buffer.byteLength(unwrapFence(fixture.code)), - sameLength: output.length === unwrapFence(fixture.code).length, - sameLineCount: output.split("\n").length === unwrapFence(fixture.code).split("\n").length, - returnOffsetPreserved: - output.indexOf("return") === unwrapFence(fixture.code).indexOf("return"), - }; - } catch (error) { - return { status: "rejected", error: `${error.name}: ${error.message}`.split("\n")[0] }; - } -} - -function percentile(values, p) { - return values[Math.min(values.length - 1, Math.floor(values.length * p))]; -} - -async function benchmark(code, transform) { - for (let index = 0; index < 100; index += 1) transform(code); - const samples = []; - for (let index = 0; index < iterations; index += 1) { - const start = performance.now(); - transform(code); - samples.push((performance.now() - start) * 1_000); - } - samples.sort((a, b) => a - b); - return { medianUs: percentile(samples, 0.5), p95Us: percentile(samples, 0.95) }; -} - -async function coldRun(code, transform) { - const samples = []; - for (let index = 0; index < 1_000; index += 1) { - const start = performance.now(); - await run(transform(code)); - samples.push((performance.now() - start) * 1_000); - } - samples.sort((a, b) => a - b); - return { medianUs: percentile(samples, 0.5), p95Us: percentile(samples, 0.95) }; -} - -async function directoryBytes(path) { - let total = 0; - for (const entry of await readdir(path, { withFileTypes: true })) { - const child = resolve(path, entry.name); - total += entry.isDirectory() ? await directoryBytes(child) : (await stat(child)).size; - } - return total; -} - -async function packageMetrics(name) { - const pkg = JSON.parse(await readFile(resolve(here, "node_modules", name, "package.json"), "utf8")); - const direct = await directoryBytes(resolve(here, "node_modules", name)); - const seen = new Set(); - async function closure(packageName) { - if (seen.has(packageName)) return 0; - seen.add(packageName); - const path = resolve(here, "node_modules", packageName); - const manifest = JSON.parse(await readFile(resolve(path, "package.json"), "utf8")); - let bytes = await directoryBytes(path); - for (const dependency of Object.keys(manifest.dependencies ?? {})) bytes += await closure(dependency); - return bytes; - } - return { - version: pkg.version, - installedPackageBytes: direct, - installedDependencyClosureBytes: await closure(name), - dependencyClosure: [...seen], - }; -} - -const arms = { javascript: undefined, ...candidates }; -const behavior = {}; -for (const [name, transform] of Object.entries(arms)) { - behavior[name] = {}; - for (const fixture of fixtures) behavior[name][fixture.id] = await evaluateFixture(fixture, transform); -} - -assert.equal(behavior.javascript["valid-js"].result, 42); -for (const fixture of fixtures.filter((item) => item.group === "candidate")) { - assert.equal(behavior["ts-blank-space"][fixture.id].result, fixture.expected, fixture.id); -} -for (const fixture of fixtures.filter((item) => item.group === "unsupported")) { - assert.equal(behavior["ts-blank-space"][fixture.id].status, "rejected", fixture.id); -} -assert.equal(behavior["ts-blank-space"]["malformed-ts"].status, "rejected"); - -const representative = fixtures.find((item) => item.id === "erased-generic").code; -const latency = {}; -for (const [name, transform] of Object.entries(candidates)) { - latency[name] = { - transform: await benchmark(representative, transform), - normalizeAndExecute: await coldRun(representative, transform), - }; -} -latency.javascript = { normalizeAndExecute: await coldRun(fixtures[0].code, (value) => value) }; - -const packages = {}; -for (const name of Object.keys(candidates)) packages[name] = await packageMetrics(name); -const report = { - measuredAt: new Date().toISOString(), - runtime: `${process.version} ${process.platform}-${process.arch}`, - iterations, - behavior, - latency, - packages, -}; - -if (!process.argv.includes("--verify")) { - await writeFile(resolve(here, "results/issue-419-measurements.json"), `${JSON.stringify(report, null, 2)}\n`); -} -console.log(`issue #419 evaluation passed: ${fixtures.length} fixtures, ${iterations} transform samples per candidate`); diff --git a/eval/current-version/logic-benchmark.mjs b/eval/current-version/logic-benchmark.mjs deleted file mode 100644 index 01199a6c..00000000 --- a/eval/current-version/logic-benchmark.mjs +++ /dev/null @@ -1,452 +0,0 @@ -import { fork, execFileSync } from "node:child_process"; -import { mkdir, writeFile } from "node:fs/promises"; -import { dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -import { round } from "./audit-lib.mjs"; - -const here = dirname(fileURLToPath(import.meta.url)); -const root = resolve(here, "../.."); -const args = process.argv.slice(2); - -function option(name, fallback) { - const index = args.indexOf(name); - if (index < 0) return fallback; - const value = args[index + 1]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value.`); - } - return value; -} - -function positiveWhole(value, name) { - if (!Number.isSafeInteger(value) || value < 1) { - throw new TypeError(`${name} must be a positive whole number.`); - } - return value; -} - -const samples = positiveWhole( - Number(option("--samples", process.env.CONNECTA_PERF_SAMPLES ?? "40")), - "--samples", -); -const loadCalls = positiveWhole( - Number(option("--load-calls", process.env.CONNECTA_PERF_LOAD_CALLS ?? "400")), - "--load-calls", -); -const outputPath = resolve( - here, - option("--output", "results/current-logic-performance.json"), -); -const sourceCommit = execFileSync("git", ["rev-parse", "HEAD"], { - cwd: root, - encoding: "utf8", -}).trim(); -const profiles = [ - { name: "small-distributed", connectors: 10, toolsPerConnector: 10 }, - { name: "medium-distributed", connectors: 25, toolsPerConnector: 40 }, - { name: "large-distributed", connectors: 100, toolsPerConnector: 100 }, - { name: "large-wide", connectors: 1, toolsPerConnector: 10_000 }, -]; - -function percentile(sorted, fraction) { - const index = Math.min( - sorted.length - 1, - Math.max(0, Math.ceil(sorted.length * fraction) - 1), - ); - return sorted[index] ?? 0; -} - -function distribution(values) { - const sorted = [...values].sort((a, b) => a - b); - return { - samples: sorted.length, - minMs: round(sorted[0] ?? 0, 3), - p50Ms: round(percentile(sorted, 0.5), 3), - p95Ms: round(percentile(sorted, 0.95), 3), - p99Ms: round(percentile(sorted, 0.99), 3), - maxMs: round(sorted.at(-1) ?? 0, 3), - meanMs: round( - sorted.reduce((sum, value) => sum + value, 0) / - Math.max(1, sorted.length), - 3, - ), - valuesMs: sorted.map((value) => round(value, 3)), - }; -} - -function alphabetic(value) { - let remaining = value; - let result = ""; - do { - result = String.fromCharCode(97 + (remaining % 26)) + result; - remaining = Math.floor(remaining / 26) - 1; - } while (remaining >= 0); - return result; -} - -function startServer(profile) { - const started = performance.now(); - const child = fork(new URL("./performance-server.ts", import.meta.url), { - execArgv: ["--import", "tsx", "--expose-gc"], - env: { - ...process.env, - CONNECTA_PERF_CONNECTORS: String(profile.connectors), - CONNECTA_PERF_TOOLS_PER_CONNECTOR: String(profile.toolsPerConnector), - }, - stdio: ["ignore", "inherit", "inherit", "ipc"], - }); - const ready = new Promise((resolveReady, rejectReady) => { - const timeout = setTimeout(() => { - rejectReady(new Error(`Performance server "${profile.name}" timed out.`)); - }, 30_000); - child.once("error", (error) => { - clearTimeout(timeout); - rejectReady(error); - }); - child.once("exit", (code) => { - clearTimeout(timeout); - rejectReady( - new Error( - `Performance server "${profile.name}" exited before readiness (${code}).`, - ), - ); - }); - child.once("message", (message) => { - clearTimeout(timeout); - resolveReady({ - ...message, - startupMs: performance.now() - started, - }); - }); - }); - return { child, ready, nextId: 1 }; -} - -async function stopServer(server) { - if (server.child.exitCode !== null) return; - server.child.send({ type: "shutdown" }); - await new Promise((resolveExit) => server.child.once("exit", resolveExit)); -} - -async function snapshot(server, { gc = false, resetPeak = false } = {}) { - const id = server.nextId++; - const result = new Promise((resolveSnapshot) => { - const receive = (message) => { - if (message?.type !== "snapshot" || message.id !== id) return; - server.child.off("message", receive); - resolveSnapshot(message); - }; - server.child.on("message", receive); - }); - server.child.send({ type: "snapshot", id, gc, resetPeak }); - return result; -} - -async function rpc(port, method, params = {}, id = 1) { - const body = JSON.stringify({ jsonrpc: "2.0", id, method, params }); - const started = performance.now(); - const response = await fetch(`http://127.0.0.1:${port}/mcp`, { - method: "POST", - headers: { - "content-type": "application/json", - accept: "application/json, text/event-stream", - }, - body, - }); - const text = await response.text(); - const latencyMs = performance.now() - started; - if (!response.ok) { - throw new Error(`HTTP ${response.status}: ${text.slice(0, 500)}`); - } - const envelope = JSON.parse(text); - if (envelope.error) { - throw new Error(`RPC ${method} failed: ${JSON.stringify(envelope.error)}`); - } - return { - latencyMs, - requestBytes: Buffer.byteLength(body), - responseBytes: Buffer.byteLength(text), - result: envelope.result, - }; -} - -function toolCall(port, name, args, id) { - return rpc(port, "tools/call", { name, arguments: args }, id); -} - -async function sample(run, count = samples) { - const latencies = []; - let requestBytes = 0; - let responseBytes = 0; - for (let index = 0; index < count; index += 1) { - const result = await run(index); - latencies.push(result.latencyMs); - requestBytes += result.requestBytes; - responseBytes += result.responseBytes; - } - return { - ...distribution(latencies), - meanRequestBytes: round(requestBytes / count, 1), - meanResponseBytes: round(responseBytes / count, 1), - }; -} - -async function load(port, calls, inFlight, address) { - const latencies = Array(calls); - let next = 0; - const started = performance.now(); - await Promise.all( - Array.from({ length: Math.min(calls, inFlight) }, async () => { - for (;;) { - const index = next++; - if (index >= calls) return; - const result = await toolCall( - port, - "call_tool", - { - address, - args: { value: index }, - resultMode: "value", - }, - index + 10_000, - ); - latencies[index] = result.latencyMs; - } - }), - ); - const durationMs = performance.now() - started; - return { - calls, - inFlight, - durationMs: round(durationMs, 1), - throughputPerSecond: round(calls / (durationMs / 1_000), 1), - latency: distribution(latencies), - }; -} - -async function benchmarkProfile(profile) { - const server = startServer(profile); - try { - const ready = await server.ready; - const lastConnector = profile.connectors - 1; - const lastTool = profile.toolsPerConnector - 1; - const address = `connector_${lastConnector}.lookup_record_${lastTool}`; - const searchQuery = - `markerc${alphabetic(lastConnector)}xt${alphabetic(lastTool)}`; - const listed = await rpc(ready.port, "tools/list", {}, 1); - const coldSearch = await toolCall( - ready.port, - "search_tools", - { query: searchQuery, includeSchemas: "compact" }, - 2, - ); - const warmSearch = await sample((index) => - toolCall( - ready.port, - "search_tools", - { query: searchQuery, includeSchemas: "compact" }, - index + 100, - ), - ); - const negativeSearch = await sample((index) => - toolCall( - ready.port, - "search_tools", - { query: "zzzzabsentmarker" }, - index + 1_000, - ), - ); - const directCall = await sample((index) => - toolCall( - ready.port, - "call_tool", - { - address, - args: { value: index }, - resultMode: "value", - diagnostics: true, - }, - index + 2_000, - ), - ); - const batchAddresses = Array.from( - { length: Math.min(10, profile.toolsPerConnector) }, - (_, index) => - `connector_${lastConnector}.lookup_record_${profile.toolsPerConnector - index - 1}`, - ); - // Batching has no top-level tool: a program calls connecta.batch, so this - // measures the whole execute_code round trip for the same ten independent - // calls. One priming execution absorbs the QuickJS child cold start, which - // benchmarkExecutor measures on its own below, and proves the program - // really batched — a failing program returns a result, not an RPC error, - // and would otherwise be timed as if it were work. - const batchProgram = - "async () => await connecta.batch(" + - JSON.stringify( - batchAddresses.map((batchAddress, value) => ({ - address: batchAddress, - args: { value }, - })), - ) + - ")"; - const primed = await toolCall( - ready.port, - "execute_code", - { code: batchProgram }, - 2_999, - ); - const primedCalls = primed.result?.structuredContent?.result; - if ( - primed.result?.isError === true || - !Array.isArray(primedCalls) || - primedCalls.length !== batchAddresses.length || - !primedCalls.every((entry) => entry.ok === true) - ) { - throw new Error( - `Batch program did not complete ${batchAddresses.length} calls: ${JSON.stringify(primed.result).slice(0, 500)}`, - ); - } - const batch = await sample( - (index) => - toolCall( - ready.port, - "execute_code", - { code: batchProgram }, - index + 3_000, - ), - Math.min(samples, 20), - ); - const memoryBeforeLoad = await snapshot(server, { - gc: true, - resetPeak: true, - }); - const concurrency = []; - for (const inFlight of [1, 16, 64]) { - concurrency.push( - await load(ready.port, loadCalls, inFlight, address), - ); - } - const memoryAfterLoad = await snapshot(server, { gc: true }); - const soak = []; - if (profile.name === "large-distributed") { - for (let roundIndex = 1; roundIndex <= 3; roundIndex += 1) { - const measured = await load( - ready.port, - loadCalls * 2, - 16, - address, - ); - const memory = await snapshot(server, { gc: true }); - soak.push({ - round: roundIndex, - throughputPerSecond: measured.throughputPerSecond, - p95Ms: measured.latency.p95Ms, - rss: memory.rss, - heapUsed: memory.heapUsed, - }); - } - } - return { - ...profile, - totalTools: ready.totalTools, - startupMs: round(ready.startupMs, 1), - toolsList: { - latencyMs: round(listed.latencyMs, 3), - responseBytes: listed.responseBytes, - toolCount: listed.result.tools?.length ?? 0, - }, - coldSearchMs: round(coldSearch.latencyMs, 3), - warmSearch, - negativeSearch, - directCall, - batch, - concurrency, - memoryBeforeLoad, - memoryAfterLoad, - soak, - }; - } finally { - await stopServer(server); - } -} - -async function benchmarkExecutor() { - const profile = { - name: "executor-medium", - connectors: 25, - toolsPerConnector: 40, - }; - const server = startServer(profile); - try { - const ready = await server.ready; - const calls = []; - for (let index = 0; index < Math.min(samples, 20); index += 1) { - calls.push( - await toolCall( - ready.port, - "execute_code", - { - code: - index === 0 - ? "async () => 1" - : "async () => connector_24.lookup_record_39({ value: 7 })", - }, - index + 4_000, - ), - ); - } - return { - profile, - startupMs: round(ready.startupMs, 1), - coldNoopMs: round(calls[0].latencyMs, 3), - warmHostCall: distribution(calls.slice(1).map((call) => call.latencyMs)), - }; - } finally { - await stopServer(server); - } -} - -const profileResults = []; -for (const profile of profiles) { - process.stderr.write( - `Benchmarking ${profile.name} (${( - profile.connectors * profile.toolsPerConnector - ).toLocaleString()} tools)…\n`, - ); - profileResults.push(await benchmarkProfile(profile)); -} -process.stderr.write("Benchmarking QuickJS executor…\n"); -const executor = await benchmarkExecutor(); -const result = { - schemaVersion: 1, - generatedAt: new Date().toISOString(), - source: { - commit: sourceCommit, - nodeVersion: process.versions.node, - platform: `${process.platform}-${process.arch}`, - samples, - loadCalls, - }, - profiles: profileResults, - executor, -}; -await mkdir(dirname(outputPath), { recursive: true }); -await writeFile(outputPath, `${JSON.stringify(result, null, 2)}\n`); -process.stdout.write( - `${JSON.stringify({ - event: "logic_benchmark_complete", - output: outputPath, - sourceCommit, - profiles: profileResults.map((profile) => ({ - name: profile.name, - totalTools: profile.totalTools, - startupMs: profile.startupMs, - searchP50Ms: profile.warmSearch.p50Ms, - callP50Ms: profile.directCall.p50Ms, - throughputAt64: profile.concurrency.at(-1).throughputPerSecond, - rssMb: round(profile.memoryAfterLoad.rss / 1024 / 1024, 1), - })), - executor, - })}\n`, -); diff --git a/eval/current-version/mcp-session.mjs b/eval/current-version/mcp-session.mjs deleted file mode 100644 index 019383d6..00000000 --- a/eval/current-version/mcp-session.mjs +++ /dev/null @@ -1,65 +0,0 @@ -import readline from "node:readline"; - -import { createAuditClient } from "./audit-lib.mjs"; - -const url = - process.env.CONNECTA_EVAL_URL ?? "http://127.0.0.1:8797/mcp"; -const token = process.env.CONNECTA_EVAL_TOKEN ?? "connecta-eval-token"; -const tokenizerName = - process.env.CONNECTA_EVAL_TOKENIZER ?? "o200k_base"; -const context = await createAuditClient({ url, token, tokenizerName }); - -function emit(value) { - process.stdout.write(`${JSON.stringify(value)}\n`); -} - -emit({ - event: "connected", - tokenizer: tokenizerName, - connection: context.connection, -}); - -const input = readline.createInterface({ - input: process.stdin, - terminal: false, -}); - -for await (const line of input) { - if (!line.trim()) continue; - let command; - try { - command = JSON.parse(line); - } catch (error) { - emit({ event: "client_error", message: String(error) }); - continue; - } - if (command.action === "close") break; - if (command.action === "summary") { - emit({ event: "summary", observations: context.observations }); - continue; - } - if (command.action !== "call" || typeof command.tool !== "string") { - emit({ event: "client_error", message: "Expected action=call and tool." }); - continue; - } - try { - const observed = await context.call( - command.name ?? command.tool, - command.tool, - command.args ?? {}, - ); - emit({ - event: "tool_result", - observation: observed.observation, - result: observed.result, - }); - } catch (error) { - emit({ - event: "transport_error", - message: error instanceof Error ? error.message : String(error), - }); - } -} - -input.close(); -await context.close(); diff --git a/eval/current-version/package-lock.json b/eval/current-version/package-lock.json index 0ae5dfbd..72bd13ee 100644 --- a/eval/current-version/package-lock.json +++ b/eval/current-version/package-lock.json @@ -1,58 +1,14 @@ { - "name": "connecta-current-version-eval", + "name": "connecta-current-version-benchmark", "lockfileVersion": 3, "requires": true, "packages": { "": { - "name": "connecta-current-version-eval", + "name": "connecta-current-version-benchmark", "dependencies": { - "js-tiktoken": "1.0.21", - "sucrase": "3.35.1", - "ts-blank-space": "0.9.0", - "typescript": "5.9.3" + "js-tiktoken": "1.0.21" } }, - "node_modules/@jridgewell/gen-mapping": { - "version": "0.3.13", - "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz", - "integrity": "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==", - "license": "MIT", - "dependencies": { - "@jridgewell/sourcemap-codec": "^1.5.0", - "@jridgewell/trace-mapping": "^0.3.24" - } - }, - "node_modules/@jridgewell/resolve-uri": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", - "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==", - "license": "MIT", - "engines": { - "node": ">=6.0.0" - } - }, - "node_modules/@jridgewell/sourcemap-codec": { - "version": "1.5.5", - "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz", - "integrity": "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==", - "license": "MIT" - }, - "node_modules/@jridgewell/trace-mapping": { - "version": "0.3.31", - "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz", - "integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==", - "license": "MIT", - "dependencies": { - "@jridgewell/resolve-uri": "^3.1.0", - "@jridgewell/sourcemap-codec": "^1.4.14" - } - }, - "node_modules/any-promise": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/any-promise/-/any-promise-1.3.0.tgz", - "integrity": "sha512-7UvmKalWRt1wgjL1RrGxoSJW/0QZFIegpeGvZG9kjp8vrRu55XTHbwnqq2GpXm9uLbcuhxm3IqX9OB4MZR1b2A==", - "license": "MIT" - }, "node_modules/base64-js": { "version": "1.5.1", "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", @@ -73,32 +29,6 @@ ], "license": "MIT" }, - "node_modules/commander": { - "version": "4.1.1", - "resolved": "https://registry.npmjs.org/commander/-/commander-4.1.1.tgz", - "integrity": "sha512-NOKm8xhkzAjzFx8B2v5OAHT+u5pRQc2UCa2Vq9jYL/31o2wi9mxBA7LIFs3sV5VSC49z6pEhfbMULvShKj26WA==", - "license": "MIT", - "engines": { - "node": ">= 6" - } - }, - "node_modules/fdir": { - "version": "6.5.0", - "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", - "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==", - "license": "MIT", - "engines": { - "node": ">=12.0.0" - }, - "peerDependencies": { - "picomatch": "^3 || ^4" - }, - "peerDependenciesMeta": { - "picomatch": { - "optional": true - } - } - }, "node_modules/js-tiktoken": { "version": "1.0.21", "resolved": "https://registry.npmjs.org/js-tiktoken/-/js-tiktoken-1.0.21.tgz", @@ -107,143 +37,6 @@ "dependencies": { "base64-js": "^1.5.1" } - }, - "node_modules/lines-and-columns": { - "version": "1.2.4", - "resolved": "https://registry.npmjs.org/lines-and-columns/-/lines-and-columns-1.2.4.tgz", - "integrity": "sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg==", - "license": "MIT" - }, - "node_modules/mz": { - "version": "2.7.0", - "resolved": "https://registry.npmjs.org/mz/-/mz-2.7.0.tgz", - "integrity": "sha512-z81GNO7nnYMEhrGh9LeymoE4+Yr0Wn5McHIZMK5cfQCl+NDX08sCZgUc9/6MHni9IWuFLm1Z3HTCXu2z9fN62Q==", - "license": "MIT", - "dependencies": { - "any-promise": "^1.0.0", - "object-assign": "^4.0.1", - "thenify-all": "^1.0.0" - } - }, - "node_modules/object-assign": { - "version": "4.1.1", - "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", - "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/picomatch": { - "version": "4.0.5", - "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.5.tgz", - "integrity": "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A==", - "license": "MIT", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/jonschlinkert" - } - }, - "node_modules/pirates": { - "version": "4.0.7", - "resolved": "https://registry.npmjs.org/pirates/-/pirates-4.0.7.tgz", - "integrity": "sha512-TfySrs/5nm8fQJDcBDuUng3VOUKsd7S+zqvbOTiGXHfxX4wK31ard+hoNuvkicM/2YFzlpDgABOevKSsB4G/FA==", - "license": "MIT", - "engines": { - "node": ">= 6" - } - }, - "node_modules/sucrase": { - "version": "3.35.1", - "resolved": "https://registry.npmjs.org/sucrase/-/sucrase-3.35.1.tgz", - "integrity": "sha512-DhuTmvZWux4H1UOnWMB3sk0sbaCVOoQZjv8u1rDoTV0HTdGem9hkAZtl4JZy8P2z4Bg0nT+YMeOFyVr4zcG5Tw==", - "license": "MIT", - "dependencies": { - "@jridgewell/gen-mapping": "^0.3.2", - "commander": "^4.0.0", - "lines-and-columns": "^1.1.6", - "mz": "^2.7.0", - "pirates": "^4.0.1", - "tinyglobby": "^0.2.11", - "ts-interface-checker": "^0.1.9" - }, - "bin": { - "sucrase": "bin/sucrase", - "sucrase-node": "bin/sucrase-node" - }, - "engines": { - "node": ">=16 || 14 >=14.17" - } - }, - "node_modules/thenify": { - "version": "3.3.1", - "resolved": "https://registry.npmjs.org/thenify/-/thenify-3.3.1.tgz", - "integrity": "sha512-RVZSIV5IG10Hk3enotrhvz0T9em6cyHBLkH/YAZuKqd8hRkKhSfCGIcP2KUY0EPxndzANBmNllzWPwak+bheSw==", - "license": "MIT", - "dependencies": { - "any-promise": "^1.0.0" - } - }, - "node_modules/thenify-all": { - "version": "1.6.0", - "resolved": "https://registry.npmjs.org/thenify-all/-/thenify-all-1.6.0.tgz", - "integrity": "sha512-RNxQH/qI8/t3thXJDwcstUO4zeqo64+Uy/+sNVRBx4Xn2OX+OZ9oP+iJnNFqplFra2ZUVeKCSa2oVWi3T4uVmA==", - "license": "MIT", - "dependencies": { - "thenify": ">= 3.1.0 < 4" - }, - "engines": { - "node": ">=0.8" - } - }, - "node_modules/tinyglobby": { - "version": "0.2.17", - "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz", - "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==", - "license": "MIT", - "dependencies": { - "fdir": "^6.5.0", - "picomatch": "^4.0.4" - }, - "engines": { - "node": ">=12.0.0" - }, - "funding": { - "url": "https://github.com/sponsors/SuperchupuDev" - } - }, - "node_modules/ts-blank-space": { - "version": "0.9.0", - "resolved": "https://registry.npmjs.org/ts-blank-space/-/ts-blank-space-0.9.0.tgz", - "integrity": "sha512-TU6coIZm6RxGgAjroRKmi6aCQ7Pq7cT+1I+bTKBepD6joOuWqUCOptVlz+eurqTnatnQOimlR8XB5sQZ6ULwbw==", - "license": "Apache-2.0", - "dependencies": { - "typescript": "5.1.6 - 6.0.x" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "node_modules/ts-interface-checker": { - "version": "0.1.13", - "resolved": "https://registry.npmjs.org/ts-interface-checker/-/ts-interface-checker-0.1.13.tgz", - "integrity": "sha512-Y/arvbn+rrz3JCKl9C4kVNfTfSm2/mEp5FSz5EsZSANGPSlQrpRI5M4PKF+mJnE52jOO90PnPSc3Ur3bTQw0gA==", - "license": "Apache-2.0" - }, - "node_modules/typescript": { - "version": "5.9.3", - "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", - "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", - "license": "Apache-2.0", - "bin": { - "tsc": "bin/tsc", - "tsserver": "bin/tsserver" - }, - "engines": { - "node": ">=14.17" - } } } } diff --git a/eval/current-version/package.json b/eval/current-version/package.json index aca7d0e1..b63dba99 100644 --- a/eval/current-version/package.json +++ b/eval/current-version/package.json @@ -1,24 +1,12 @@ { - "name": "connecta-current-version-eval", + "name": "connecta-current-version-benchmark", "private": true, "type": "module", "scripts": { - "audit": "node run-audit.mjs", - "audit:development": "node run-development-discovery.mjs", - "perf:logic": "node logic-benchmark.mjs", - "perf:agent": "node agent-benchmark.mjs", - "perf:agent:compare": "node agent-benchmark-compare.mjs", - "perf:lookup": "node agent-lookup-benchmark.mjs", - "issue:419": "npm --prefix ../.. run build && node issue-419-ts-eval.mjs", - "report:cloudflare-surface": "node --import tsx cloudflare-surface-report.ts", - "perf:report": "node performance-report.mjs", - "perf": "npm run audit -- --output results/current-performance-audit.json --report results/current-performance-audit.md && npm run perf:logic && npm run perf:agent && npm run perf:report", - "check": "tsc -p tsconfig.json --noEmit && node agent-benchmark-self-test.mjs && node performance-report-self-test.mjs && npm run issue:419 -- --verify" + "benchmark": "npm --prefix ../.. run build && node benchmark.mjs", + "check": "node benchmark.mjs --self-test" }, "dependencies": { - "js-tiktoken": "1.0.21", - "sucrase": "3.35.1", - "ts-blank-space": "0.9.0", - "typescript": "5.9.3" + "js-tiktoken": "1.0.21" } } diff --git a/eval/current-version/patches/issue-322-coverage-off.patch b/eval/current-version/patches/issue-322-coverage-off.patch deleted file mode 100644 index 1edbbf38..00000000 --- a/eval/current-version/patches/issue-322-coverage-off.patch +++ /dev/null @@ -1,23 +0,0 @@ -diff --git a/src/catalog-service.ts b/src/catalog-service.ts -index fa29ce0..a7e8e4e 100644 ---- a/src/catalog-service.ts -+++ b/src/catalog-service.ts -@@ -845,18 +845,6 @@ export class CatalogService { - ...(match.tool.annotations - ? { annotations: match.tool.annotations } - : {}), -- ...(queryTerms.length > 0 -- ? { -- queryCoverage: { -- nameTerms, -- descriptionTerms, -- unmatchedTerms: uncoveredTerms, -- ...(queryMetadataTruncated -- ? { truncated: true as const } -- : {}), -- }, -- } -- : {}), - ...(requiredReasons - ? { - guideRequired: true as const, diff --git a/eval/current-version/performance-report-agent.mjs b/eval/current-version/performance-report-agent.mjs deleted file mode 100644 index c883b193..00000000 --- a/eval/current-version/performance-report-agent.mjs +++ /dev/null @@ -1,87 +0,0 @@ -function point(value) { - return { - min: value, - p50: value, - p95: value, - max: value, - mean: value, - stddev: 0, - }; -} - -function legacyRun(fixture, index) { - const connectaRoundTrips = fixture.toolCalls?.length ?? 0; - const costEfficient = - fixture.routeEfficient === true && - fixture.contextEfficient === true; - return { - ...fixture, - repetition: 1, - taskCorrect: fixture.correct === true, - safetyPassed: null, - surfaceValid: null, - costEfficient, - connectaRoundTrips, - outerTools: fixture.calledTools ?? [], - costEnvelope: { - maxRoundTrips: null, - maxMcpResultTokens: fixture.mcpResultTokenBudget, - }, - legacyIndex: index, - }; -} - -function legacyCase(run) { - return { - id: run.id, - repetitions: 1, - rates: { - taskCorrect: Number(run.taskCorrect), - safetyPassed: null, - surfaceValid: null, - costEfficient: Number(run.costEfficient), - passed: Number(run.passed === true), - }, - observedRoutes: [{ - route: run.outerTools.join(" → ") || "(none)", - count: 1, - }], - connectaRoundTrips: point(run.connectaRoundTrips), - mcpResultTokens: point(run.mcpResultTokens), - latencyMs: point(run.latencyMs), - costEnvelope: run.costEnvelope, - runs: [run], - }; -} - -export function normalizeAgentBenchmark(agent) { - if (agent.schemaVersion === 2 || agent.schemaVersion === 3) { - return { - ...agent, - reportSchema: `v${agent.schemaVersion}`, - summary: { - ...agent.summary, - runs: agent.summary.runs ?? agent.runs.length, - }, - }; - } - if (agent.schemaVersion !== 1) { - throw new Error( - `Unsupported agent benchmark schema ${String(agent.schemaVersion)}.`, - ); - } - const runs = agent.cases.map(legacyRun); - return { - ...agent, - reportSchema: "v1-legacy", - runs, - cases: runs.map(legacyCase), - summary: { - ...agent.summary, - runs: runs.length, - costEfficient: runs.filter((run) => run.costEfficient).length, - safetyPassed: null, - surfaceValid: null, - }, - }; -} diff --git a/eval/current-version/performance-report-self-test.mjs b/eval/current-version/performance-report-self-test.mjs deleted file mode 100644 index 6a923a70..00000000 --- a/eval/current-version/performance-report-self-test.mjs +++ /dev/null @@ -1,40 +0,0 @@ -import assert from "node:assert/strict"; -import { readFile } from "node:fs/promises"; - -import { normalizeAgentBenchmark } from "./performance-report-agent.mjs"; - -const legacy = JSON.parse( - await readFile( - new URL( - "./results/current-agent-performance.json", - import.meta.url, - ), - "utf8", - ), -); -const normalizedLegacy = normalizeAgentBenchmark(legacy); -assert.equal(normalizedLegacy.reportSchema, "v1-legacy"); -assert.equal(normalizedLegacy.summary.runs, legacy.cases.length); -assert.equal(normalizedLegacy.cases[0].repetitions, 1); -assert.equal(normalizedLegacy.cases[0].rates.safetyPassed, null); -assert.equal( - normalizedLegacy.cases[0].costEnvelope.maxMcpResultTokens, - legacy.cases[0].mcpResultTokenBudget, -); - -const normalizedV2 = normalizeAgentBenchmark({ - schemaVersion: 2, - summary: { runs: 3 }, - runs: [{ id: "sample" }], - cases: [{ id: "sample" }], -}); -assert.equal(normalizedV2.reportSchema, "v2"); -assert.equal(normalizedV2.summary.runs, 3); -assert.equal(normalizedV2.runs.length, 1); - -assert.throws( - () => normalizeAgentBenchmark({ schemaVersion: 99 }), - /Unsupported agent benchmark schema 99/, -); - -process.stdout.write("performance report compatibility self-test passed\n"); diff --git a/eval/current-version/performance-report.mjs b/eval/current-version/performance-report.mjs deleted file mode 100644 index bd3f4e37..00000000 --- a/eval/current-version/performance-report.mjs +++ /dev/null @@ -1,224 +0,0 @@ -import { readFile, writeFile } from "node:fs/promises"; -import { resolve } from "node:path"; - -import { normalizeAgentBenchmark } from "./performance-report-agent.mjs"; - -const here = new URL(".", import.meta.url); -const args = process.argv.slice(2); - -function option(name, fallback) { - const index = args.indexOf(name); - if (index < 0) return fallback; - const value = args[index + 1]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value.`); - } - return value; -} - -async function json(relativePath) { - return JSON.parse(await readFile(new URL(relativePath, here), "utf8")); -} - -function ms(value) { - return Number(value).toFixed(1); -} - -function integer(value) { - return new Intl.NumberFormat("en-US").format(Math.round(value)); -} - -function mb(value) { - return (value / 1024 / 1024).toFixed(1); -} - -function pct(value) { - if (value === null || value === undefined) return "n/a"; - return `${(value * 100).toFixed(1)}%`; -} - -const auditPath = option( - "--audit", - "results/current-performance-audit.json", -); -const logicPath = option( - "--logic", - "results/current-logic-performance.json", -); -const agentPath = option( - "--agent", - "results/current-agent-performance.json", -); -const outputPath = resolve( - new URL(".", here).pathname, - option("--output", "results/current-performance-report.md"), -); -const [audit, logic, rawAgent] = await Promise.all([ - json(auditPath), - json(logicPath), - json(agentPath), -]); -const agent = normalizeAgentBenchmark(rawAgent); - -const logicRows = logic.profiles - .map( - (profile) => - `| ${profile.name} | ${integer(profile.totalTools)} | ${ms(profile.startupMs)} | ${ms(profile.coldSearchMs)} | ${ms(profile.warmSearch.p50Ms)} / ${ms(profile.warmSearch.p95Ms)} | ${ms(profile.directCall.p50Ms)} / ${ms(profile.directCall.p95Ms)} | ${ms(profile.batch.p50Ms)} / ${ms(profile.batch.p95Ms)} | ${mb(profile.memoryAfterLoad.rss)} | ${mb(profile.memoryAfterLoad.heapUsed)} |`, - ) - .join("\n"); -const loadRows = logic.profiles - .flatMap((profile) => - profile.concurrency.map( - (row) => - `| ${profile.name} | ${row.inFlight} | ${integer(row.calls)} | ${row.throughputPerSecond.toFixed(1)} | ${ms(row.latency.p50Ms)} | ${ms(row.latency.p95Ms)} | ${ms(row.latency.p99Ms)} |`, - ), - ) - .join("\n"); -const agentRuns = agent.runs ?? agent.cases; -const agentRows = agent.cases - .map( - (fixture) => - `| ${fixture.id} | ${integer(fixture.repetitions)} | ${pct(fixture.rates.taskCorrect)} | ${pct(fixture.rates.safetyPassed)} | ${pct(fixture.rates.surfaceValid)} | ${pct(fixture.rates.costEfficient)} | ${fixture.observedRoutes.map(({ route, count }) => `\`${route}\` ×${count}`).join("; ")} | ${integer(fixture.connectaRoundTrips.p50)} (${integer(fixture.connectaRoundTrips.min)}–${integer(fixture.connectaRoundTrips.max)}) | ${integer(fixture.mcpResultTokens.p50)} / ${integer(fixture.costEnvelope.maxMcpResultTokens)} | ${(fixture.latencyMs.p50 / 1_000).toFixed(1)} s (${(fixture.latencyMs.min / 1_000).toFixed(1)}–${(fixture.latencyMs.max / 1_000).toFixed(1)}) |`, - ) - .join("\n"); -const inefficient = agentRuns.filter((fixture) => !fixture.costEfficient); -const incorrect = agentRuns.filter((fixture) => !fixture.taskCorrect); -const contextHeavy = agentRuns.filter( - (fixture) => !fixture.contextEfficient, -); -const guided = agentRuns.filter((fixture) => fixture.guidanceFetched); -const locallyExploratory = agentRuns.filter((fixture) => - (fixture.nonMcpActions ?? []).some( - (action) => - (typeof action === "string" ? action : action.type) === - "command_execution", - ), -); -const widestSearch = [...audit.discovery.cases].sort( - (left, right) => right.responseTokens - left.responseTokens, -)[0]; -const largeDistributed = logic.profiles.find( - (profile) => profile.name === "large-distributed", -); -const soakRss = largeDistributed?.soak.map((round) => mb(round.rss)) ?? []; -const soakHeap = - largeDistributed?.soak.map((round) => mb(round.heapUsed)) ?? []; - -const findings = [ - `The fixed agent-visible surface is ${integer(audit.totals.definitionTokens)} tokens for ${audit.connection.toolCount} meta-tools.`, - `The held-out discovery suite achieves ${pct(audit.discovery.metrics.top1Accuracy)} top-1 accuracy and ${pct(audit.discovery.metrics.positiveRecall)} recall, with a ${pct(audit.discovery.metrics.falsePositiveRate)} false-positive rate on negative queries.`, - `Fresh-agent task correctness is ${agent.summary.correct}/${agent.summary.runs} runs; cost efficiency is ${agent.summary.costEfficient}/${agent.summary.runs}.`, - agent.summary.safetyPassed === null - ? `This legacy schema-v1 artifact did not record execution safety or advertised-surface validity; context efficiency is ${agent.summary.contextEfficient}/${agent.summary.runs}.` - : `Fresh-agent safety is ${agent.summary.safetyPassed}/${agent.summary.runs}, advertised-surface validity is ${agent.summary.surfaceValid}/${agent.summary.runs}, and context efficiency is ${agent.summary.contextEfficient}/${agent.summary.runs}.`, - guided.length > 0 - ? `Connecta's on-demand usage guide was self-fetched in ${guided.length}/${agent.summary.runs} runs (${guided.map((fixture) => `\`${fixture.id}\` #${fixture.repetition}`).join(", ")}); no user prompt explained the routing workflow.` - : "No case needed the on-demand usage guide; the always-loaded instructions were sufficient.", - locallyExploratory.length > 0 - ? `The coding-agent host explored the local filesystem before or alongside Connecta in ${locallyExploratory.length}/${agent.summary.runs} runs (${locallyExploratory.map((fixture) => `\`${fixture.id}\` #${fixture.repetition}`).join(", ")}). This is host routing overhead, not Connecta call latency.` - : "The agent went directly to Connecta in every case without unrelated local-tool exploration.", - `QuickJS costs ${ms(logic.executor.coldNoopMs)} ms cold and ${ms(logic.executor.warmHostCall.p50Ms)} ms p50 for warm executions that make a host call.`, - widestSearch - ? `The largest held-out discovery response is \`${widestSearch.id}\` at ${integer(widestSearch.responseTokens)} tokens.` - : null, - inefficient.length > 0 - ? `Cost envelopes were exceeded in: ${inefficient - .map( - (fixture) => - fixture.costEnvelope.maxRoundTrips === null - ? `\`${fixture.id}\` (legacy route/context score)` - : `\`${fixture.id}\` #${fixture.repetition} (${fixture.connectaRoundTrips}/${fixture.costEnvelope.maxRoundTrips} round trips, ${integer(fixture.mcpResultTokens)}/${integer(fixture.costEnvelope.maxMcpResultTokens)} result tokens)`, - ) - .join("; ")}.` - : "Every fresh-agent run stayed inside its round-trip and result-token envelope.", - incorrect.length > 0 - ? `${agent.reportSchema === "v1-legacy" ? "Incorrect final answers" : "Incorrect task outcomes"} occurred in: ${incorrect.map((fixture) => `\`${fixture.id}\` #${fixture.repetition}`).join(", ")}.` - : agent.reportSchema === "v1-legacy" - ? "Every legacy fresh-agent run produced the expected final answer; execution correctness was not recorded." - : "Every fresh-agent run produced the correct task result and execution.", - contextHeavy.length > 0 - ? `Context budgets were exceeded in: ${contextHeavy.map((fixture) => `\`${fixture.id}\` #${fixture.repetition} (${integer(fixture.mcpResultTokens)} / ${integer(fixture.costEnvelope.maxMcpResultTokens)} tokens)`).join(", ")}.` - : "Every fresh-agent run stayed within its Connecta result-token budget.", - soakRss.length > 0 - ? `After the initial load allocation, the 10,000-tool soak held at ${soakRss.join(" / ")} MB RSS and ${soakHeap.join(" / ")} MB live heap across three rounds; this run shows a plateau, not continuing live-heap growth.` - : null, -].filter(Boolean); - -const report = `# Connecta ${audit.source.commit.slice(0, 12)} performance analysis - -Generated: ${logic.generatedAt} - -Runtime: Node ${logic.source.nodeVersion} on ${logic.source.platform}; agent client ${agent.source.codexVersion} (${agent.source.model}) - -## Executive result - -${findings.map((finding) => `- ${finding}`).join("\n")} - -## Connecta logic - -All figures are client-observed over stateless Streamable HTTP on loopback. Search and call columns are p50 / p95 after warm-up. Batching has no top-level tool, so the batch column is a warm \`execute_code\` program calling \`connecta.batch\` with ten independent calls; the sandbox round trip is inside that number. - -| Catalog shape | Tools | Startup ms | Cold search ms | Warm search ms | Direct call ms | 10-call batch program ms | RSS after GC MB | Live heap MB | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -${logicRows} - -### Direct-call load - -| Catalog shape | In flight | Calls | Throughput/s | p50 ms | p95 ms | p99 ms | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -${loadRows} - -### Optional code mode - -- Cold sandbox no-op: ${ms(logic.executor.coldNoopMs)} ms -- Warm sandbox + one connector host call: ${ms(logic.executor.warmHostCall.p50Ms)} ms p50; ${ms(logic.executor.warmHostCall.p95Ms)} ms p95 - -### What the logic measurements say - -- Catalog size is not on the hot direct-call path in a meaningful way here: p50 stays near 2–2.4 ms from 100 through 10,000 tools. -- Warm lexical discovery remains near 3 ms p50 at 10,000 tools; cold discovery is about 11 ms because the normalized index is built on first use. -- Moving from 16 to 64 in-flight requests adds little throughput and sharply worsens tail latency. A production deployment should prefer bounded admission near the knee instead of maximizing concurrent work. -- RSS grows after the HTTP load but live heap remains near ${largeDistributed ? mb(largeDistributed.memoryAfterLoad.heapUsed) : "—"} MB at 10,000 tools, and the three-round soak plateaus. Treat RSS as capacity to budget, but this sample does not look like an accumulating JavaScript heap leak. - -## Agent experience - -Each run starts a fresh Connecta server and a fresh non-interactive Codex session. The user prompt states the task, not the Connecta routing procedure. The current schema-v2 scorer accepts any route on the advertised seven-tool surface that produces the expected execution and answer safely, without foreign tools, inside task-specific round-trip and result-token envelopes. Exact routes and duplicate, failed, foreign, unavailable-surface, removed-tool, and unexpected calls remain visible as diagnostics. Legacy schema-v1 artifacts are labeled as such because they did not record execution safety or advertised-surface validity. - -| Task | Runs | Correct | Safe | Surface | Cost | Observed outer routes | RT p50 (range) | MCP tokens p50 / budget | Wall p50 (range) | -| --- | ---: | ---: | ---: | ---: | ---: | --- | ---: | ---: | ---: | -${agentRows} - -Codex reported ${integer(agent.summary.totalInputTokens)} total input tokens and ${integer(agent.summary.totalOutputTokens)} output tokens across the fresh sessions. Those are whole-agent figures—including the host system prompt, built-in tool definitions, reasoning context, and Connecta—not Connecta-only costs. The measured Connecta MCP results contributed ${integer(agent.summary.totalMcpResultTokens)} serialized tokens. - -## Priorities - -1. **Use the harness to evaluate discovery changes without trading away recall.** The unchanged baseline has a ${pct(audit.discovery.metrics.falsePositiveRate)} held-out false-positive rate, and the natural reduction query exceeded its MCP-result budget. Select candidates on independently authored corpora and reserve the release holdout for final regression checks. -2. **Keep the routing contract in server instructions and measure it across repeated fresh sessions.** This sample stayed within its cost envelopes in ${agent.summary.costEfficient}/${agent.summary.runs} runs without prescribing one exact tool sequence. The on-demand skill should remain a fallback, not required ceremony. -3. **Treat program routing as a latency/context trade.** The scripted audit reduced 120 records to a tiny answer in one MCP execution, but it pays a roughly ${ms(logic.executor.coldNoopMs)} ms cold start and had ${ms(logic.executor.warmHostCall.p95Ms)} ms p95 in this small sample. Compare it against equivalent direct-call response tokens on real workloads. -4. **Benchmark more hosts before changing the public tool surface.** The Codex lane showed unrelated filesystem exploration in ${locallyExploratory.length}/${agent.summary.runs} runs and an approval stop at \`authorize_connector\`. Add an interactive host and at least one non-coding agent to distinguish Connecta affordances from host policy and coding-agent bias. -5. **Set performance budgets in CI, not machine-specific absolute gates.** Track percentage regression from a pinned runner for 10,000-tool cold/warm search, 16-in-flight p95, definition tokens, discovery quality, and fresh-agent route success. - -## Release audit - -- Behavioral scenarios: ${audit.tasks.summary.passed}/${audit.tasks.summary.caseCount} -- Qualification gate: ${audit.qualification.passed ? "pass" : "FAIL"} -- Discovery top-1: ${pct(audit.discovery.metrics.top1Accuracy)} -- Discovery positive recall: ${pct(audit.discovery.metrics.positiveRecall)} -- Negative-query false positives: ${pct(audit.discovery.metrics.falsePositiveRate)} -- Complete measured Connecta surface: ${integer(audit.totals.measuredSurfaceTokens)} tokens over ${audit.totals.roundTrips} round trips - -## Interpretation limits - -- Logic latency is the Connecta/framework floor on one local machine. Real connector and network latency will dominate most production calls. -- Synthetic catalogs isolate Connecta scaling but do not reproduce every downstream MCP schema, pagination behavior, or provider rate limit. -- The agent lane measured ${agent.summary.runs} run${agent.summary.runs === 1 ? "" : "s"} across ${agent.summary.cases} task${agent.summary.cases === 1 ? "" : "s"} on one Codex CLI/model configuration. Routing varies across repetitions, so this is a behavioral canary, not a stable pass/fail gate or a claim that every host and model will route identically. -- Whole-agent token counts are useful for comparing repeated runs of the same harness; only the MCP definitions, requests, and results are attributable to Connecta. -`; - -await writeFile(outputPath, report); -process.stdout.write( - `${JSON.stringify({ - event: "performance_report_complete", - output: outputPath, - })}\n`, -); diff --git a/eval/current-version/performance-server.ts b/eval/current-version/performance-server.ts deleted file mode 100644 index 1e5728b9..00000000 --- a/eval/current-version/performance-server.ts +++ /dev/null @@ -1,182 +0,0 @@ -import { once } from "node:events"; - -import { - createConnecta, - memoryStorage, - type Connector, - type ToolDef, -} from "../../src/index.js"; -import { quickJsExecutor } from "../../src/executors/quickjs.js"; -import { listen } from "../../src/node.js"; - -const connectorCount = Number( - process.env.CONNECTA_PERF_CONNECTORS ?? "10", -); -const toolsPerConnector = Number( - process.env.CONNECTA_PERF_TOOLS_PER_CONNECTOR ?? "100", -); -const requestConcurrency = Number( - process.env.CONNECTA_PERF_REQUEST_CONCURRENCY ?? "64", -); - -function positiveWhole(value: number, name: string): number { - if (!Number.isSafeInteger(value) || value < 1) { - throw new TypeError(`${name} must be a positive whole number.`); - } - return value; -} - -positiveWhole(connectorCount, "CONNECTA_PERF_CONNECTORS"); -positiveWhole(toolsPerConnector, "CONNECTA_PERF_TOOLS_PER_CONNECTOR"); -positiveWhole(requestConcurrency, "CONNECTA_PERF_REQUEST_CONCURRENCY"); - -function alphabetic(value: number): string { - let remaining = value; - let result = ""; - do { - result = String.fromCharCode(97 + (remaining % 26)) + result; - remaining = Math.floor(remaining / 26) - 1; - } while (remaining >= 0); - return result; -} - -function toolDefinitions(connectorIndex: number): ToolDef[] { - return Array.from({ length: toolsPerConnector }, (_, toolIndex) => ({ - name: `lookup_record_${toolIndex}`, - description: - `Retrieve synthetic performance record ${toolIndex} from shard ` + - `${connectorIndex}. Marker markerc${alphabetic(connectorIndex)}xt${alphabetic(toolIndex)}.`, - inputSchema: { - type: "object", - properties: { - value: { type: "integer" }, - }, - required: ["value"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - connector: { type: "integer" }, - tool: { type: "integer" }, - value: { type: "integer" }, - }, - required: ["connector", "tool", "value"], - additionalProperties: false, - }, - annotations: { - readOnlyHint: true, - idempotentHint: true, - destructiveHint: false, - }, - })); -} - -const connectors: Connector[] = Array.from( - { length: connectorCount }, - (_, connectorIndex) => { - const tools = toolDefinitions(connectorIndex); - return { - id: `connector_${connectorIndex}`, - kind: "api", - description: `Synthetic performance connector ${connectorIndex}`, - staticTools: tools, - async listTools() { - return tools; - }, - async callTool(name, args) { - const toolIndex = Number(name.slice("lookup_record_".length)); - return { - connector: connectorIndex, - tool: toolIndex, - value: (args as { value: number }).value, - }; - }, - }; - }, -); - -const executor = quickJsExecutor({ - timeoutMs: 10_000, - cpuTimeMs: 2_000, - concurrency: 4, - maxQueueSize: 32, -}); -const silent = { debug() {}, info() {}, warn() {}, error() {} }; -const connecta = createConnecta({ - connectors, - storage: memoryStorage(), - logger: silent, - executor, - admission: { - requests: { - concurrency: requestConcurrency, - maxQueueSize: 512, - queueTimeoutMs: 10_000, - retryAfterMs: 1_000, - }, - }, - serverInfo: { - name: "connecta-performance", - version: "1.0.0", - }, -}); -const server = listen(connecta, { - port: 0, - host: "127.0.0.1", - gracefulShutdown: false, -}); - -let peakRss = process.memoryUsage().rss; -const sampler = setInterval(() => { - peakRss = Math.max(peakRss, process.memoryUsage().rss); -}, 5); -sampler.unref(); - -await once(server, "listening"); -const address = server.address(); -if (!address || typeof address === "string") { - throw new Error("Performance server did not expose a TCP address."); -} -process.send?.({ - type: "ready", - port: address.port, - connectorCount, - toolsPerConnector, - totalTools: connectorCount * toolsPerConnector, -}); - -let nextSnapshotId = 1; -process.on("message", async (message: unknown) => { - if (!message || typeof message !== "object") return; - const command = message as { - type?: string; - id?: number; - gc?: boolean; - resetPeak?: boolean; - }; - if (command.type === "snapshot") { - if (command.gc && typeof global.gc === "function") { - global.gc(); - await new Promise((resolve) => setTimeout(resolve, 0)); - global.gc(); - } - const memory = process.memoryUsage(); - process.send?.({ - type: "snapshot", - id: command.id ?? nextSnapshotId++, - rss: memory.rss, - heapUsed: memory.heapUsed, - heapTotal: memory.heapTotal, - external: memory.external, - peakRss, - }); - if (command.resetPeak) peakRss = memory.rss; - return; - } - if (command.type === "shutdown") { - clearInterval(sampler); - await connecta.close(); - server.close(() => process.exit(0)); - } -}); diff --git a/eval/current-version/reference-connection-server.ts b/eval/current-version/reference-connection-server.ts deleted file mode 100644 index aa731818..00000000 --- a/eval/current-version/reference-connection-server.ts +++ /dev/null @@ -1,200 +0,0 @@ -/** - * The isolated deployment the reference-connection cases run against. - * - * This server exists to answer the last open acceptance criterion of #297: a - * cold agent must be able to discover and use a maintained prebuilt connection - * without avoidable repair. Everything here is the real thing except the - * network at the far end — the connection is built by the real `cloudflare()` - * constructor, with its real schemas, annotations, projections, admission - * policy, usage guide, and error mapping, and only its documented `baseUrl` - * option is redirected at the local double in `cloudflare-fixture.ts`. - * - * It is deliberately a second server rather than more connectors inside - * `sandbox-server.ts`. That sandbox's catalog is the ranking pool for the - * held-out discovery corpus, which is gated release evidence and explicitly - * must not be tuned against; adding twenty-eight real Cloudflare tools to it - * would perturb the corpus by another name. Keeping the catalogs apart is what - * lets the reference-connection numbers and the release-audit numbers both - * stay honest. - * - * Two instances of the same connection are registered on purpose. They give - * the unavailable-auth case a genuinely separate credential to fail on, and - * they exercise the same per-connector-id isolation the provider suites pin. - */ -import { once } from "node:events"; - -import { - bearerToken, - createConnecta, - memoryStorage, - type ToolCallActivityEvent, -} from "../../src/index.js"; -import { CredentialVault } from "../../src/credentials.js"; -import { cloudflare } from "../../src/providers/cloudflare.js"; -import { quickJsExecutor } from "../../src/executors/quickjs.js"; -import { listen } from "../../src/node.js"; -import { createEvalTracing } from "./eval-tracing.js"; -import { startCloudflareFixture } from "./cloudflare-fixture.js"; - -const token = process.env["CONNECTA_EVAL_TOKEN"] ?? "connecta-eval-token"; -const sourceCommit = - process.env["CONNECTA_EVAL_SOURCE_COMMIT"] ?? "working-tree"; -const traceEnabled = process.env["CONNECTA_EVAL_TRACE"] === "enabled"; -const port = Number(process.env["CONNECTA_EVAL_PORT"] ?? "0"); -const host = "127.0.0.1"; - -/** - * Fixture credentials. Both are literals in an isolated process talking to a - * loopback double; neither is a secret, and neither reaches a real provider. - */ -const EDGE_TOKEN = "cf-eval-edge-token"; -const PARTNER_TOKEN = "cf-eval-partner-rotated-token"; - -const credentialEncryptionKey = Buffer.alloc(32, 11).toString("base64"); -const storage = memoryStorage(); -const activityEvents: ToolCallActivityEvent[] = []; -const tracing = createEvalTracing({ enabled: traceEnabled, token }); - -const fixture = await startCloudflareFixture({ - validToken: EDGE_TOKEN, - revokedToken: PARTNER_TOKEN, -}); - -const edge = cloudflare("cloudflare-edge", { - title: "Cloudflare — Eval Edge", - purpose: - "the production edge estate behind connecta-eval.test; read zones, DNS records, and platform inventory, and make DNS changes only with approval", - baseUrl: fixture.baseUrl, - instructions: - "This estate owns connecta-eval.test and its staging subdomain. Resolve a zone id with list_zones before any zone-scoped call.", -}); - -const partner = cloudflare("cloudflare-partner", { - title: "Cloudflare — Partner Estate", - purpose: - "a partner-managed estate whose API token was rotated out of this deployment and has not yet been replaced", - baseUrl: fixture.baseUrl, -}); - -const baseExecutor = quickJsExecutor({ timeoutMs: 10_000, cpuTimeMs: 2_000 }); -const executor = traceEnabled - ? tracing.tracedExecutor(baseExecutor) - : baseExecutor; - -const connecta = createConnecta({ - auth: [bearerToken(token, { subjectId: "reference-connection-evaluator" })], - connectors: [edge, partner], - storage, - executor, - credentials: { encryptionKey: credentialEncryptionKey }, - calls: { - defaultTimeoutMs: 15_000, - // Generous on purpose. The dependent-read case must measure whether a cold - // agent chooses to reduce in-program, not whether Connecta truncated the - // listing for it — a budget that forces the right answer measures nothing. - maxResultBytes: 32_000, - }, - activity: { - deploymentId: "reference-connection-eval", - store: { - record(event) { - activityEvents.push(event); - tracing.emitTrace({ - kind: "execution", - address: event.address, - source: event.source, - outcome: event.outcome, - durationMs: event.durationMs, - attempts: event.attempts, - ...(event.errorCode ? { errorCode: event.errorCode } : {}), - }); - }, - async list({ limit }) { - return { events: activityEvents.slice(-limit).reverse() }; - }, - }, - }, - serverInfo: { - name: "connecta-reference-connection-eval", - version: sourceCommit.slice(0, 12), - title: "Connecta reference-connection eval sandbox", - }, - deploymentInfo: { sourceCommit, isolated: true }, -}); - -/** - * Seed the operator vault directly. The connection reads its token through - * `ctx.credential.get()` like any deployment; the only thing skipped is the - * human at /credentials. The partner estate is seeded with a token the double - * rejects, so its failure is the provider's real 401 mapping rather than the - * "no credential configured" branch — those are different findings and the - * lane should measure the one an operator actually meets. - */ -const vault = new CredentialVault(storage, credentialEncryptionKey); -await vault.set("cloudflare-edge", EDGE_TOKEN, "reference-connection-eval"); -await vault.set("cloudflare-partner", PARTNER_TOKEN, "reference-connection-eval"); -await connecta.registry.invalidateStored("cloudflare-edge"); -await connecta.registry.invalidateStored("cloudflare-partner"); - -const traced = traceEnabled ? tracing.withOuterTracing(connecta) : connecta; -const server = listen( - { - ...traced, - async fetch(request, env, ctx) { - const url = new URL(request.url); - // Downstream evidence, kept separate from the meta-tool trace: it answers - // "what actually reached the provider API, and with which token", which - // is how the write-routing case proves no unapproved write got through. - if (request.method === "GET" && url.pathname === "/__eval/downstream") { - if (request.headers.get("authorization") !== `Bearer ${token}`) { - return Response.json({ error: "unauthorized" }, { status: 401 }); - } - return Response.json({ - requests: fixture.requests.map((entry) => ({ - method: entry.method, - path: entry.path, - connection: - entry.token === EDGE_TOKEN - ? "cloudflare-edge" - : entry.token === PARTNER_TOKEN - ? "cloudflare-partner" - : "unknown", - })), - }); - } - return traced.fetch(request, env, ctx); - }, - }, - { port, host, gracefulShutdown: false }, -); -await once(server, "listening"); -const address = server.address(); -if (!address || typeof address === "string") { - throw new Error("Reference-connection eval server did not expose a TCP address."); -} - -console.log( - JSON.stringify({ - event: "ready", - url: `http://${host}:${address.port}/mcp`, - baseUrl: `http://${host}:${address.port}`, - sourceCommit, - connectorCount: 2, - downstream: fixture.baseUrl, - traceEnabled, - }), -); - -let shuttingDown = false; -async function shutdown(): Promise { - if (shuttingDown) return; - shuttingDown = true; - await new Promise((resolve, reject) => { - server.close((error) => (error ? reject(error) : resolve())); - }); - await connecta.close(); - await fixture.close(); -} - -process.once("SIGINT", () => void shutdown().then(() => process.exit(0))); -process.once("SIGTERM", () => void shutdown().then(() => process.exit(0))); diff --git a/eval/current-version/report.mjs b/eval/current-version/report.mjs deleted file mode 100644 index 08baf7e1..00000000 --- a/eval/current-version/report.mjs +++ /dev/null @@ -1,63 +0,0 @@ -function pct(value) { - return `${(value * 100).toFixed(1)}%`; -} - -function integer(value) { - return new Intl.NumberFormat("en-US").format(Math.round(value)); -} - -export function renderReport(audit, jsonName) { - const categoryRows = Object.entries(audit.discovery.categories) - .map( - ([name, metrics]) => - `| ${name} | ${metrics.queries} | ${ - metrics.top1Accuracy === null ? "—" : pct(metrics.top1Accuracy) - } | ${ - metrics.positiveRecall === null ? "—" : pct(metrics.positiveRecall) - } | ${metrics.meanPrecision.toFixed(3)} | ${ - metrics.falsePositiveRate === null - ? "—" - : pct(metrics.falsePositiveRate) - } | ${metrics.meanResultCount.toFixed(2)} | ${metrics.meanResponseTokens.toFixed(1)} | ${metrics.meanQueryCoverageTokens.toFixed(1)} |`, - ) - .join("\n"); - const failed = audit.tasks.cases.filter((entry) => !entry.passed); - - return `# Current-version Connecta audit - -Source commit: \`${audit.source.commit}\` - -Runtime: Node ${audit.source.nodeVersion}; tokenizer \`${audit.source.tokenizer}\`; surface \`${audit.source.surface}\`; executor \`${audit.source.executorMode}\` - -Machine-readable results: \`${jsonName}\` (run artifact, not committed) - -## Qualification - -- Release gate: ${audit.qualification.passed ? "pass" : "FAIL"} -- Task scenarios: ${audit.tasks.summary.passed}/${audit.tasks.summary.caseCount} passed (${pct(audit.tasks.summary.taskSuccessRate)}) -- Discovery top-1 accuracy: ${pct(audit.discovery.metrics.top1Accuracy)} -- Discovery expected top-1 accuracy: ${pct(audit.discovery.metrics.expectedTopAccuracy)} -- Discovery positive recall: ${pct(audit.discovery.metrics.positiveRecall)} -- Recall at the default page: ${pct(audit.discovery.metrics.recallAtDefaultPage)} -- Negative-query false-positive rate: ${pct(audit.discovery.metrics.falsePositiveRate)} -- Removed query-coverage wire: ${integer(audit.discovery.metrics.totalQueryCoverageBytes)} bytes and ${integer(audit.discovery.metrics.totalQueryCoverageTokens)} tokens of ${integer(audit.discovery.metrics.totalResponseBytes)} discovery response bytes and ${integer(audit.discovery.metrics.totalResponseTokens)} tokens -- Round trips: ${audit.totals.roundTrips}; summed call latency: ${audit.totals.summedLatencyMs.toFixed(1)} ms -- Connecta surface: ${integer(audit.totals.definitionTokens)} definition + ${integer(audit.totals.requestTokens)} request + ${integer(audit.totals.responseTokens)} response = **${integer(audit.totals.measuredSurfaceTokens)} tokens** -- Result compatibility observed: \`content\` ${audit.compatibility.contentResults}/${audit.compatibility.resultCount}, \`structuredContent\` ${audit.compatibility.structuredContentResults}/${audit.compatibility.resultCount} -- \`execute_code\` advertised: ${audit.compatibility.executeCodeAdvertised ? "yes" : "no"} -- Payload-free activity invariant: ${audit.invariants.activityPayloadFree ? "pass" : "FAIL"} - -${failed.length === 0 ? "" : `Failed task scenarios: ${failed.map((entry) => `\`${entry.name}\``).join(", ")}\n`} -## Discovery holdout - -The holdout contains ${audit.discovery.corpus.toolCount} tools across ${audit.discovery.corpus.connectorCount} connectors and ${audit.discovery.corpus.queryCount} independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | Mean coverage tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -${categoryRows} - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. -`; -} diff --git a/eval/current-version/results/0.8.0-baseline.md b/eval/current-version/results/0.8.0-baseline.md deleted file mode 100644 index 747ae197..00000000 --- a/eval/current-version/results/0.8.0-baseline.md +++ /dev/null @@ -1,40 +0,0 @@ -# Current-version Connecta audit - -Source commit: `e3c3ac6a0843ca1668cd28ea75a6726710f4f91d` - -Runtime: Node 22.23.1; tokenizer `o200k_base` - -Machine-readable results: `0.8.0-baseline.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 80.0% -- Round trips: 55; summed call latency: 225.6 ms -- Connecta surface: 2,164 definition + 1,144 request + 68,765 response = **72,073 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 50/55 -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 239.0 | -| conversational | 8 | 75.0% | 100.0% | 0.042 | — | 24.13 | 3102.3 | -| multi-intent | 4 | 100.0% | 100.0% | 0.080 | — | 25.00 | 3170.3 | -| short-function-word | 4 | 75.0% | 100.0% | 0.280 | — | 19.00 | 2450.5 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 25.00 | 3337.0 | -| negative | 4 | — | — | 0.250 | 75.0% | 15.50 | 2071.3 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 265.3 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 646.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-final.md b/eval/current-version/results/0.8.1-final.md deleted file mode 100644 index 4a3135b4..00000000 --- a/eval/current-version/results/0.8.1-final.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `e82c5221008b100d33140e3770ddfe521159302a` - -Runtime: Node 22.23.1; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `0.8.1-final.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 228.4 ms -- Connecta surface: 2,174 definition + 1,145 request + 16,343 response = **19,662 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 168.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 303.0 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 783.5 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 466.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 942.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 59.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 184.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 454.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-2-after.md b/eval/current-version/results/0.8.1-step-2-after.md deleted file mode 100644 index 129000f2..00000000 --- a/eval/current-version/results/0.8.1-step-2-after.md +++ /dev/null @@ -1,40 +0,0 @@ -# Current-version Connecta audit - -Source commit: `2f6c09d447a14def0efbe0cd55743ba6ac78fe88` - -Runtime: Node 22.23.1; tokenizer `o200k_base` - -Machine-readable results: `0.8.1-step-2-after.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 221.2 ms -- Connecta surface: 2,173 definition + 1,141 request + 21,935 response = **25,249 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 50/55 -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 239.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 426.6 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 1102.3 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 659.3 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1328.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 78.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 265.3 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 646.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-2-before.md b/eval/current-version/results/0.8.1-step-2-before.md deleted file mode 100644 index c8f8e87e..00000000 --- a/eval/current-version/results/0.8.1-step-2-before.md +++ /dev/null @@ -1,40 +0,0 @@ -# Current-version Connecta audit - -Source commit: `a22e8b4be260be8b7b3f637b9d5738bf778de832` - -Runtime: Node 22.23.1; tokenizer `o200k_base` - -Machine-readable results: `0.8.1-step-2-before.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 80.0% -- Round trips: 55; summed call latency: 234.8 ms -- Connecta surface: 2,164 definition + 1,141 request + 68,759 response = **72,064 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 50/55 -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 239.0 | -| conversational | 8 | 75.0% | 100.0% | 0.042 | — | 24.13 | 3102.3 | -| multi-intent | 4 | 100.0% | 100.0% | 0.080 | — | 25.00 | 3170.3 | -| short-function-word | 4 | 75.0% | 100.0% | 0.280 | — | 19.00 | 2450.5 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 25.00 | 3337.0 | -| negative | 4 | — | — | 0.250 | 75.0% | 15.50 | 2071.3 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 265.3 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 646.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-3-disabled.md b/eval/current-version/results/0.8.1-step-3-disabled.md deleted file mode 100644 index 933515cc..00000000 --- a/eval/current-version/results/0.8.1-step-3-disabled.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `8f482c4ae3016fd0a13f75fe79c73c520d32c120` - -Runtime: Node 22.23.1; tokenizer `o200k_base`; executor `disabled` - -Machine-readable results: `0.8.1-step-3-disabled.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 20/20 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 54; summed call latency: 150.7 ms -- Connecta surface: 1,707 definition + 1,069 request + 21,797 response = **24,573 tokens** -- Result compatibility observed: `content` 54/54, `structuredContent` 49/54 -- `execute_code` advertised: no -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 239.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 426.6 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 1102.3 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 659.3 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1328.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 78.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 265.3 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 646.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-3-enabled.md b/eval/current-version/results/0.8.1-step-3-enabled.md deleted file mode 100644 index ffba920e..00000000 --- a/eval/current-version/results/0.8.1-step-3-enabled.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `8f482c4ae3016fd0a13f75fe79c73c520d32c120` - -Runtime: Node 22.23.1; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `0.8.1-step-3-enabled.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 225.2 ms -- Connecta surface: 2,173 definition + 1,139 request + 21,931 response = **25,243 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 50/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 239.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 426.6 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 1102.3 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 659.3 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1328.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 78.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 265.3 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 646.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-4-after.md b/eval/current-version/results/0.8.1-step-4-after.md deleted file mode 100644 index 81f58523..00000000 --- a/eval/current-version/results/0.8.1-step-4-after.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `61975d7c2fa403a595f97e4f838425100580fe4b` - -Runtime: Node 22.23.1; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `0.8.1-step-4-after.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 220.1 ms -- Connecta surface: 2,174 definition + 1,143 request + 22,630 response = **25,947 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 239.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 426.6 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 1102.3 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 659.3 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1328.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 78.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 265.3 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 646.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-4-before.md b/eval/current-version/results/0.8.1-step-4-before.md deleted file mode 100644 index 6218677a..00000000 --- a/eval/current-version/results/0.8.1-step-4-before.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `9e39259c52869dc77a7e866e25f7f5d70b48fd79` - -Runtime: Node 22.23.1; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `0.8.1-step-4-before.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 222.4 ms -- Connecta surface: 2,173 definition + 1,147 request + 21,947 response = **25,267 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 50/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 239.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 426.6 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 1102.3 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 659.3 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1328.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 78.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 265.3 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 646.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-5-after.md b/eval/current-version/results/0.8.1-step-5-after.md deleted file mode 100644 index 1bad82e5..00000000 --- a/eval/current-version/results/0.8.1-step-5-after.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `693d9036fdc1759d35ad73471f26a644c4fb17aa` - -Runtime: Node 22.23.1; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `0.8.1-step-5-after.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 225.0 ms -- Connecta surface: 2,174 definition + 1,145 request + 16,559 response = **19,878 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 168.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 303.0 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 783.5 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 466.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 942.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 59.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 184.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 454.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-5-before.md b/eval/current-version/results/0.8.1-step-5-before.md deleted file mode 100644 index bead502a..00000000 --- a/eval/current-version/results/0.8.1-step-5-before.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `7bc9c80f3b8b1656403993c5a506cc03c0b4647b` - -Runtime: Node 22.23.1; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `0.8.1-step-5-before.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 224.7 ms -- Connecta surface: 2,174 definition + 1,145 request + 22,634 response = **25,953 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 239.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 426.6 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 1102.3 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 659.3 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1328.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 78.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 265.3 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 646.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-6-after.md b/eval/current-version/results/0.8.1-step-6-after.md deleted file mode 100644 index 9b71ed11..00000000 --- a/eval/current-version/results/0.8.1-step-6-after.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `faf864c6086a096495929c3cf0d1ba07611fe046` - -Runtime: Node 22.23.1; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `0.8.1-step-6-after.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 223.7 ms -- Connecta surface: 2,174 definition + 1,142 request + 16,337 response = **19,653 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 168.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 303.0 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 783.5 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 466.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 942.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 59.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 184.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 454.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.8.1-step-6-before.md b/eval/current-version/results/0.8.1-step-6-before.md deleted file mode 100644 index 9ae40616..00000000 --- a/eval/current-version/results/0.8.1-step-6-before.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `de211a3361cf2cceda0d029948728372b76f670b` - -Runtime: Node 22.23.1; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `0.8.1-step-6-before.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 218.9 ms -- Connecta surface: 2,174 definition + 1,143 request + 16,555 response = **19,872 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 168.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 303.0 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 783.5 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 466.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 942.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 59.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 184.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 454.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/0.9.0-final.md b/eval/current-version/results/0.9.0-final.md deleted file mode 100644 index 9b0d1a6f..00000000 --- a/eval/current-version/results/0.9.0-final.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `dbf2b61327abf4da47625253190efd72d4b0ec21` - -Runtime: Node 26.5.0; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `0.9.0-final.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 253.6 ms -- Connecta surface: 2,114 definition + 1,146 request + 16,422 response = **19,682 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 168.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 303.0 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 783.5 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 466.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 967.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 59.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 184.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 454.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/README.md b/eval/current-version/results/README.md index a7f2eed0..5e27769c 100644 --- a/eval/current-version/results/README.md +++ b/eval/current-version/results/README.md @@ -1,26 +1,5 @@ -# What lives in `results/` +# Generated results -Markdown. Every lane here writes a JSON artifact beside its report, and those -artifacts are the reason this directory once held about 7 MB for verdicts that -fit in a paragraph. A settled issue's answer is already in the report, in -[`ethos.md`](../../../ethos.md), and in git history; the trace that produced it -is regeneration output. So `results/*.json` is ignored, and the reports below -name their JSON sibling rather than linking it — the name tells you what to ask -for, and the command in [the lane's README](../README.md) produces it (#346). +`latest.json` and `latest.md` are local benchmark outputs and are ignored. Commit a result only when it is durable release or issue evidence, and link it from the decision it supports. -Two files are tracked anyway, because neither can be regenerated: - -- `current-agent-performance.json` — the v1-legacy agent-benchmark artifact that - `performance-report-self-test.mjs` reads to prove the report normalizer still - understands the old schema. It is a fixture wearing an output filename, which - is also why `perf:agent` writing over it would break the self-test. -- `issue-322-preregistered-provenance.json` — the - [#322](https://github.com/zackbart/connecta/issues/322) preregistration timing - record: observed timestamps, the GitHub PushEvent id, and the SHA-256 of the - plan, the coverage-off patch, the comparison, and both raw arms. You cannot - re-observe a timestamp, and those hashes are what still ties the retired - blobs in git history to the verdict in `issue-322-evidence.md`. - -An artifact that is genuinely evidence — an observation, not a rerun — gets a -line here and a negation in the root `.gitignore`. Everything else gets a -command. +`issue-350-evidence.md` is retained historical evidence because current provider documentation links to it. diff --git a/eval/current-version/results/current-agent-performance.json b/eval/current-version/results/current-agent-performance.json deleted file mode 100644 index fbbb302a..00000000 --- a/eval/current-version/results/current-agent-performance.json +++ /dev/null @@ -1,346 +0,0 @@ -{ - "schemaVersion": 1, - "generatedAt": "2026-07-29T15:33:18.500Z", - "source": { - "commit": "e50a165acc8caa9ef08be3b9ccdfa8af235c68fe", - "nodeVersion": "26.5.0", - "platform": "darwin-arm64", - "codexVersion": "codex-cli 0.145.0", - "model": "codex-default", - "tokenizer": "o200k_base" - }, - "summary": { - "cases": 4, - "correct": 4, - "routeEfficient": 3, - "contextEfficient": 3, - "passed": 3, - "totalLatencyMs": 109120, - "totalInputTokens": 568676, - "totalOutputTokens": 1612, - "totalMcpResultTokens": 7217 - }, - "cases": [ - { - "id": "single-read", - "prompt": "Return the one deterministic record with id 7. Respond with only the record JSON.", - "latencyMs": 23219.9, - "correct": true, - "routeEfficient": true, - "contextEfficient": true, - "passed": true, - "expectedTools": [ - "search_tools", - "call_tool" - ], - "mcpResultTokenBudget": 500, - "calledTools": [ - "search_tools", - "call_tool" - ], - "missingTools": [], - "forbiddenTools": [], - "guidanceFetched": false, - "toolCalls": [ - { - "tool": "search_tools", - "arguments": { - "query": "deterministic record id", - "includeSchemas": "compact", - "limit": 20 - }, - "status": "completed", - "error": null, - "durationMs": 11.2, - "resultBytes": 1665, - "resultTokens": 356 - }, - { - "tool": "call_tool", - "arguments": { - "address": "controlled.read_record", - "args": { - "id": 7 - }, - "resultMode": "value" - }, - "status": "completed", - "error": null, - "durationMs": 11.3, - "resultBytes": 239, - "resultTokens": 73 - } - ], - "nonMcpActions": [], - "finalText": "{\"id\":7,\"group\":\"beta\",\"score\":18}", - "usage": { - "input_tokens": 120752, - "cached_input_tokens": 96000, - "cache_write_input_tokens": 0, - "output_tokens": 246, - "reasoning_output_tokens": 58 - }, - "mcpResultTokens": 429 - }, - { - "id": "independent-batch", - "prompt": "Return the point-lookup results for deterministic record ids 11 and 12. Respond with only a JSON array ordered by id.", - "latencyMs": 15801.2, - "correct": true, - "routeEfficient": true, - "contextEfficient": true, - "passed": true, - "expectedTools": [ - "search_tools", - "batch_call" - ], - "mcpResultTokenBudget": 700, - "calledTools": [ - "search_tools", - "batch_call" - ], - "missingTools": [], - "forbiddenTools": [], - "guidanceFetched": false, - "toolCalls": [ - { - "tool": "search_tools", - "arguments": { - "query": "deterministic record point lookup id", - "includeSchemas": "compact", - "limit": 20 - }, - "status": "completed", - "error": null, - "durationMs": 12.8, - "resultBytes": 1665, - "resultTokens": 356 - }, - { - "tool": "batch_call", - "arguments": { - "calls": [ - { - "address": "controlled.read_record", - "args": { - "id": 11 - }, - "resultMode": "value" - }, - { - "address": "controlled.read_record", - "args": { - "id": 12 - }, - "resultMode": "value" - } - ], - "resultMode": "value" - }, - "status": "completed", - "error": null, - "durationMs": 13.3, - "resultBytes": 635, - "resultTokens": 174 - } - ], - "nonMcpActions": [], - "finalText": "[{\"id\":11,\"group\":\"gamma\",\"score\":86},{\"id\":12,\"group\":\"alpha\",\"score\":2}]", - "usage": { - "input_tokens": 91087, - "cached_input_tokens": 68608, - "cache_write_input_tokens": 0, - "output_tokens": 307, - "reasoning_output_tokens": 112 - }, - "mcpResultTokens": 530 - }, - { - "id": "dependent-reduction", - "prompt": "For the deterministic collection of 120 records, return each group's record count and score sum. Respond with only a JSON object keyed by group.", - "latencyMs": 40920.9, - "correct": true, - "routeEfficient": false, - "contextEfficient": false, - "passed": false, - "expectedTools": [ - "search_tools", - "execute_code" - ], - "mcpResultTokenBudget": 700, - "calledTools": [ - "codex_document_control.list_document_sessions", - "skills", - "search_tools", - "call_tool", - "get_result" - ], - "missingTools": [ - "execute_code" - ], - "forbiddenTools": [ - "call_tool" - ], - "guidanceFetched": true, - "toolCalls": [ - { - "tool": "codex_document_control.list_document_sessions", - "arguments": {}, - "status": "completed", - "error": null, - "durationMs": 635.4, - "resultBytes": 261, - "resultTokens": 62 - }, - { - "tool": "skills", - "arguments": { - "name": "usage" - }, - "status": "completed", - "error": null, - "durationMs": 9.8, - "resultBytes": 1806, - "resultTokens": 397 - }, - { - "tool": "search_tools", - "arguments": { - "query": "deterministic collection records group score", - "includeSchemas": "compact", - "limit": 20 - }, - "status": "completed", - "error": null, - "durationMs": 11.9, - "resultBytes": 3367, - "resultTokens": 696 - }, - { - "tool": "call_tool", - "arguments": { - "address": "controlled.records", - "args": { - "count": 120 - }, - "resultMode": "value" - }, - "status": "completed", - "error": null, - "durationMs": 12.2, - "resultBytes": 515, - "resultTokens": 173 - }, - { - "tool": "get_result", - "arguments": { - "id": "03e58076-3717-4c2c-81be-73f79f3606a1", - "offset": 0, - "maxBytes": 10000 - }, - "status": "completed", - "error": null, - "durationMs": 8.1, - "resultBytes": 12789, - "resultTokens": 4486 - } - ], - "nonMcpActions": [ - { - "type": "command_execution", - "status": "completed", - "command": "/bin/zsh -lc \"rg --files -g '*' /tmp 2>/dev/null | head -200\"" - }, - { - "type": "command_execution", - "status": "completed", - "command": "/bin/zsh -lc \"awk 'BEGIN { for (i=1;i<=120;i++) { g=(i-1)%3; s=(17*(i-1))%101; c[g]++; t[g]+=s } print c[0],t[0]; print c[1],t[1]; print c[2],t[2] }'\"" - } - ], - "finalText": "{\"alpha\":{\"record_count\":40,\"score_sum\":1400},\"beta\":{\"record_count\":40,\"score_sum\":2080},\"gamma\":{\"record_count\":40,\"score_sum\":2356}}", - "usage": { - "input_tokens": 252994, - "cached_input_tokens": 220672, - "cache_write_input_tokens": 0, - "output_tokens": 723, - "reasoning_output_tokens": 216 - }, - "mcpResultTokens": 5814 - }, - { - "id": "auth-handoff", - "prompt": "Tell me the identity from the oauth-recoverable connector. If an operator must act first, return the exact recovery handoff instead of claiming success.", - "latencyMs": 29178, - "correct": true, - "routeEfficient": true, - "contextEfficient": true, - "passed": true, - "expectedTools": [ - "search_tools", - "call_tool", - "authorize_connector" - ], - "mcpResultTokenBudget": 900, - "calledTools": [ - "search_tools", - "call_tool", - "authorize_connector" - ], - "missingTools": [], - "forbiddenTools": [], - "guidanceFetched": false, - "toolCalls": [ - { - "tool": "search_tools", - "arguments": { - "connector": "oauth-recoverable", - "query": "identity current user whoami profile", - "includeSchemas": "compact", - "limit": 10 - }, - "status": "completed", - "error": null, - "durationMs": 11.5, - "resultBytes": 887, - "resultTokens": 210 - }, - { - "tool": "call_tool", - "arguments": { - "address": "oauth-recoverable.whoami", - "args": {}, - "resultMode": "value" - }, - "status": "failed", - "error": null, - "durationMs": 9.3, - "resultBytes": 1047, - "resultTokens": 233 - }, - { - "tool": "authorize_connector", - "arguments": { - "connector": "oauth-recoverable" - }, - "status": "failed", - "error": { - "message": "user cancelled MCP tool call" - }, - "durationMs": 0.2, - "resultBytes": 4, - "resultTokens": 1 - } - ], - "nonMcpActions": [], - "finalText": "{\"tool\":\"authorize_connector\",\"arguments\":{\"connector\":\"oauth-recoverable\"},\"operatorHandoff\":\"Give the URL and instructions it returns to the operator.\"}", - "usage": { - "input_tokens": 103843, - "cached_input_tokens": 66560, - "cache_write_input_tokens": 0, - "output_tokens": 336, - "reasoning_output_tokens": 90 - }, - "mcpResultTokens": 444 - } - ] -} diff --git a/eval/current-version/results/current-performance-audit.md b/eval/current-version/results/current-performance-audit.md deleted file mode 100644 index 6879d056..00000000 --- a/eval/current-version/results/current-performance-audit.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `e50a165acc8caa9ef08be3b9ccdfa8af235c68fe` - -Runtime: Node 26.5.0; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `current-performance-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 217.7 ms -- Connecta surface: 2,174 definition + 1,145 request + 16,420 response = **19,739 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 168.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 303.0 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 783.5 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 466.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 967.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 59.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 184.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 454.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/current-performance-report.md b/eval/current-version/results/current-performance-report.md deleted file mode 100644 index 3661a39e..00000000 --- a/eval/current-version/results/current-performance-report.md +++ /dev/null @@ -1,97 +0,0 @@ -# Connecta e50a165acc8c performance analysis - -Generated: 2026-07-29T15:31:28.329Z - -Runtime: Node 26.5.0 on darwin-arm64; agent client codex-cli 0.145.0 (codex-default) - -## Executive result - -- The fixed agent-visible surface is 2,174 tokens for 10 meta-tools. -- The held-out discovery suite achieves 89.7% top-1 accuracy and 100.0% recall, with a 20.0% false-positive rate on negative queries. -- Fresh-agent task correctness is 4/4; efficient routing is 3/4. -- Fresh-agent context efficiency is 3/4 against task-specific budgets derived from the scripted minimal routes. -- Connecta's on-demand usage guide was self-fetched in 1/4 cases (`dependent-reduction`); no user prompt explained the routing workflow. -- The coding-agent host explored the local filesystem before or alongside Connecta in 1/4 cases (`dependent-reduction`). This is host routing overhead, not Connecta call latency. -- QuickJS costs 76.5 ms cold and 3.6 ms p50 for warm executions that make a host call. -- The largest held-out discovery response is `cleanup-only` at 967 tokens. -- Routing misses occurred in: `dependent-reduction` (call_tool). -- Every fresh-agent case produced the correct task result. -- Context budgets were exceeded in: `dependent-reduction` (5,814 / 700 tokens). -- After the initial load allocation, the 10,000-tool soak held at 377.6 / 377.6 / 377.9 MB RSS and 34.1 / 34.2 / 34.2 MB live heap across three rounds; this run shows a plateau, not continuing live-heap growth. - -## Connecta logic - -All figures are client-observed over stateless Streamable HTTP on loopback. Search and call columns are p50 / p95 after warm-up. - -| Catalog shape | Tools | Startup ms | Cold search ms | Warm search ms | Direct call ms | 10-call batch ms | RSS after GC MB | Live heap MB | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| small-distributed | 100 | 161.9 | 4.4 | 2.4 / 3.3 | 2.3 / 2.5 | 2.4 / 2.7 | 344.7 | 25.7 | -| medium-distributed | 1,000 | 162.8 | 4.3 | 2.5 / 2.9 | 2.3 / 2.5 | 2.6 / 3.4 | 333.9 | 27.9 | -| large-distributed | 10,000 | 163.1 | 10.5 | 3.0 / 4.1 | 2.1 / 2.3 | 2.4 / 2.8 | 378.0 | 35.7 | -| large-wide | 10,000 | 164.6 | 10.4 | 2.7 / 4.2 | 2.4 / 2.6 | 4.2 / 4.8 | 344.7 | 35.7 | - -### Direct-call load - -| Catalog shape | In flight | Calls | Throughput/s | p50 ms | p95 ms | p99 ms | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| small-distributed | 1 | 400 | 384.3 | 2.6 | 3.3 | 4.1 | -| small-distributed | 16 | 400 | 1600.5 | 7.9 | 16.5 | 39.8 | -| small-distributed | 64 | 400 | 1774.9 | 26.8 | 54.9 | 150.5 | -| medium-distributed | 1 | 400 | 386.8 | 2.5 | 3.2 | 4.0 | -| medium-distributed | 16 | 400 | 1647.1 | 7.9 | 15.1 | 42.8 | -| medium-distributed | 64 | 400 | 1801.5 | 14.5 | 210.1 | 218.4 | -| large-distributed | 1 | 400 | 400.5 | 2.5 | 3.0 | 4.1 | -| large-distributed | 16 | 400 | 1737.4 | 7.5 | 15.2 | 40.6 | -| large-distributed | 64 | 400 | 1860.5 | 15.4 | 204.3 | 212.3 | -| large-wide | 1 | 400 | 367.6 | 2.7 | 3.2 | 4.6 | -| large-wide | 16 | 400 | 1358.8 | 9.4 | 19.1 | 59.4 | -| large-wide | 64 | 400 | 1499.1 | 18.5 | 252.8 | 263.0 | - -### Optional code mode - -- Cold sandbox no-op: 76.5 ms -- Warm sandbox + one connector host call: 3.6 ms p50; 70.1 ms p95 - -### What the logic measurements say - -- Catalog size is not on the hot direct-call path in a meaningful way here: p50 stays near 2–2.4 ms from 100 through 10,000 tools. -- Warm lexical discovery remains near 3 ms p50 at 10,000 tools; cold discovery is about 11 ms because the normalized index is built on first use. -- Moving from 16 to 64 in-flight requests adds little throughput and sharply worsens tail latency. A production deployment should prefer bounded admission near the knee instead of maximizing concurrent work. -- RSS grows after the HTTP load but live heap remains near 35.7 MB at 10,000 tools, and the three-round soak plateaus. Treat RSS as capacity to budget, but this sample does not look like an accumulating JavaScript heap leak. - -## Agent experience - -Each case starts a fresh Connecta server and a fresh non-interactive Codex session. The user prompt states the task, not the Connecta routing procedure. “Route” means the agent chose the intended smallest execution path without making redundant schema calls or substituting another execution tool. Fetching Connecta's usage guide once is counted separately as successful self-guidance. “Context” compares serialized MCP results with a task budget derived from the scripted minimal path. - -| Task | Correct | Route | Context | Tool route | MCP result tokens / budget | Wall time | -| --- | --- | --- | --- | --- | ---: | ---: | -| single-read | yes | yes | yes | `search_tools` → `call_tool` | 429 / 500 | 23.2 s | -| independent-batch | yes | yes | yes | `search_tools` → `batch_call` | 530 / 700 | 15.8 s | -| dependent-reduction | yes | NO | NO | `codex_document_control.list_document_sessions` → `skills` → `search_tools` → `call_tool` → `get_result` | 5,814 / 700 | 40.9 s | -| auth-handoff | yes | yes | yes | `search_tools` → `call_tool` → `authorize_connector` | 444 / 900 | 29.2 s | - -Codex reported 568,676 total input tokens and 1,612 output tokens across the fresh sessions. Those are whole-agent figures—including the host system prompt, built-in tool definitions, reasoning context, and Connecta—not Connecta-only costs. The measured Connecta MCP results contributed 7,217 serialized tokens. - -## Priorities - -1. **Use the harness to evaluate discovery changes without trading away recall.** The unchanged baseline has a 20.0% held-out false-positive rate, and the natural reduction query exceeded its MCP-result budget. Select candidates on independently authored corpora and reserve the release holdout for final regression checks. -2. **Keep the routing contract in server instructions and measure it across repeated fresh sessions.** This sample chose the intended execution path in 3/4 tasks without user coaching. The on-demand skill should remain a fallback, not required ceremony. -3. **Treat code mode as a latency/context trade.** The scripted audit reduced 120 records to a tiny answer in one MCP execution, but code mode pays a roughly 76.5 ms cold start and had 70.1 ms p95 in this small sample. Keep it optional and compare it against equivalent direct-call response tokens on real workloads. -4. **Benchmark more hosts before changing the public tool surface.** The Codex lane showed unrelated filesystem exploration in 1/4 cases and an approval stop at `authorize_connector`. Add an interactive host and at least one non-coding agent to distinguish Connecta affordances from host policy and coding-agent bias. -5. **Set performance budgets in CI, not machine-specific absolute gates.** Track percentage regression from a pinned runner for 10,000-tool cold/warm search, 16-in-flight p95, definition tokens, discovery quality, and fresh-agent route success. - -## Release audit - -- Behavioral scenarios: 21/21 -- Qualification gate: pass -- Discovery top-1: 89.7% -- Discovery positive recall: 100.0% -- Negative-query false positives: 20.0% -- Complete measured Connecta surface: 19,739 tokens over 55 round trips - -## Interpretation limits - -- Logic latency is the Connecta/framework floor on one local machine. Real connector and network latency will dominate most production calls. -- Synthetic catalogs isolate Connecta scaling but do not reproduce every downstream MCP schema, pagination behavior, or provider rate limit. -- The agent lane measures one run per task on one Codex CLI/model configuration. Repeated runs showed meaningful routing variance, so it is a behavioral canary, not a stable pass/fail gate or a claim that every host and model will route identically. -- Whole-agent token counts are useful for comparing repeated runs of the same harness; only the MCP definitions, requests, and results are attributable to Connecta. diff --git a/eval/current-version/results/issue-294-catalog-error-comparison.md b/eval/current-version/results/issue-294-catalog-error-comparison.md deleted file mode 100644 index 7513e458..00000000 --- a/eval/current-version/results/issue-294-catalog-error-comparison.md +++ /dev/null @@ -1,37 +0,0 @@ -# Cold-agent comparison - -Baseline: `4222434a19605dd770b44c5159b5f40a46c92bcb` (12 runs; clean product tree) - -Candidate: `4222434a19605dd770b44c5159b5f40a46c92bcb` (12 runs; product changes present) - -## Result - -**DOES NOT QUALIFY** - -| Metric | Baseline | Candidate | Delta | -| --- | ---: | ---: | ---: | -| Correctness | 75.0% | 66.7% | -8.3 pp | -| Read-only safety | 100.0% | 100.0% | 0.0 pp | -| Context-budget pass | 66.7% | 58.3% | -8.3 pp | -| Connecta round trips / run | 3.25 | 2.25 | -1.00 | -| MCP result tokens / run | 1396.6 | 1319.7 | -76.9 | -| Whole-agent tokens / run | 117995.9 | 114427.0 | -3568.9 | -| Repairs / run | 0.58 | 0.33 | -0.25 | - -## Acceptance checks - -- FAIL: correctnessNotRegressed -- PASS: readOnlySafetyPreserved -- FAIL: contextBudgetNotRegressed -- PASS: repairOrRoundTripReduction - -Negative cost deltas are improvements. Qualification requires repeated comparable sessions, no correctness or context-budget regression, complete read-only safety, and fewer repairs or Connecta round trips. - ---- - -Provenance (hand-added; a regeneration overwrites it). Inputs were the -full-lane baseline and the `catalogError` candidate, neither retained — see -[`issue-294-first-pass.md`](./issue-294-first-pass.md) for the regeneration -command and for why this aggregate was discounted: host routing was clean in -only 6/12 candidate runs, and the entire correctness delta traces to the -`exact-address-control` prompt defect since fixed. diff --git a/eval/current-version/results/issue-294-first-pass.md b/eval/current-version/results/issue-294-first-pass.md deleted file mode 100644 index 427da819..00000000 --- a/eval/current-version/results/issue-294-first-pass.md +++ /dev/null @@ -1,138 +0,0 @@ -# Issue 294 cold-agent first pass - -- Date: 2026-08-03 -- Pinned model: `gpt-5.6-sol` -- Tokenizer: `o200k_base` - -## What is retained here - -The Markdown: this narrative and the three comparison reports it links. Every -JSON behind them — the focused unavailable-catalog baseline and candidate, the -three machine comparisons, and the three full-lane run artifacts, which were -12k–15k lines each — is regeneration output rather than evidence, and is -regenerated rather than stored (#346): - -```sh -npm --prefix eval/current-version run perf:agent -- \ - --repetitions 2 \ - --output results/issue-294-cold-agent-baseline.json -``` - -Run the same command with the candidate source checked out for the candidate -side. Note that the numbers below predate two harness fixes (the -`exact-address-control` prompt and the host-routing row), so a regeneration -will not reproduce them exactly — it reproduces the lane, not the session. - -## Complete baseline - -The first repeatable lane ran six representative workflows twice in fresh -servers and fresh ephemeral agent sessions: exact-address control, generic API -read, guide-heavy query, schema-heavy dependent read, unavailable catalog, and -large-result reduction. (`auth-handoff` has since been restored as a seventh -case; it was not part of this pass.) - -- Correct outcomes: 9/12 -- Read-only safety: 12/12 -- Context-budget passes: 8/12 -- Host routing clean: 11/12 -- Discovery calls: 18 -- Guide fetches: 7 -- Schema expansions: 2 -- Execution attempts: 13 -- Repair round trips: 7 -- Connecta result tokens: 16,759 -- Whole-agent tokens: 1,415,951 - -## Rejected candidate: narrower guide wording - -Changing guide instructions to forbid inferred connector-guide names reduced -repairs and Connecta round trips, but the two-session complete lane regressed -correctness and context-budget pass rate. It does not qualify and the product -change was reverted. - -- Repairs/run: 0.58 → 0.25 -- Connecta round trips/run: 3.25 → 2.58 -- Correctness: 75.0% → 66.7% -- Context-budget pass: 66.7% → 50.0% - -This candidate carries the same host-contamination caveat as the one below, and -it was not re-run on a focused lane: its hypothesis was about guide wording, -which touches every guide-reading case, so there is no narrow slice to isolate. -It stays rejected on the aggregate it has. - -See -[`issue-294-guide-marker-comparison.md`](./issue-294-guide-marker-comparison.md). - -## Selected first behavior: scoped catalog failure detail - -The baseline confirmed the issue's unavailable-catalog finding. The connector's -typed `unavailable` error, upstream 503 reason, and operator recovery -instruction survived on the describe path — `describe` returns the catalog -failure's message raw — but the search path discarded all of it and returned -only “Retry later.” Since a cold agent reaches an unknown connector by -searching, the reason was reliably unreachable in practice: five fresh focused -baseline sessions produced no correct recovery answer. - -The selected behavior attaches one bounded `catalogError` only when the caller -explicitly scopes search to the unavailable connector. Unscoped search still -reports only the unavailable count, so one broken connector does not copy its -failure text into unrelated discovery results. - -Across five fresh focused sessions: - -- Correctness: 0/5 → 5/5 -- Read-only safety: 5/5 → 5/5 -- Context-budget pass: 5/5 → 5/5 -- Connecta round trips/run: 2.00 → 1.80 -- Whole-agent tokens/run: 63,036.2 → 59,328.8 -- MCP result tokens/run: 268.2 → 322.6 - -The extra 54.4 MCP result tokens carry the failure reason and recovery detail; -whole-agent context still fell by 3,707.4 tokens/run because agents stopped -searching for information the generic response had discarded. The focused -comparison qualifies under the predeclared checks: -[`issue-294-unavailable-focused-comparison.md`](./issue-294-unavailable-focused-comparison.md). - -### The complete-lane comparison did not qualify, and why it was discounted - -At two repetitions the same change's six-case aggregate recorded -**DOES NOT QUALIFY** — correctness 75.0% → 66.7%, context-budget pass 66.7% → -58.3% — even though repair, round-trip, MCP-result, and whole-agent-token costs -all improved: -[`issue-294-catalog-error-comparison.md`](./issue-294-catalog-error-comparison.md). - -Per case, correct runs out of two: - -| Case | Baseline | Candidate | Host routing clean (base → cand) | -| --- | ---: | ---: | ---: | -| `exact-address-control` | 2/2 | 0/2 | 2/2 → 0/2 | -| `generic-api-read` | 2/2 | 1/2 | 2/2 → 2/2 | -| `guide-heavy-query` | 2/2 | 2/2 | 1/2 → 1/2 | -| `schema-heavy-dependent-read` | 2/2 | 2/2 | 2/2 → 0/2 | -| `unavailable-catalog` | 0/2 | 2/2 | 2/2 → 2/2 | -| `large-result-reduction` | 1/2 | 1/2 | 2/2 → 1/2 | -| **Total** | **9/12** | **8/12** | **11/12 → 6/12** | - -The targeted case moved the right way, by the full 2/2. The whole negative -delta comes from `exact-address-control` — whose baseline prompt named the -address as a bare `controlled.read_record`. A host reads that as -`.`: both candidate sessions called -`controlled.list_mcp_resources` and `controlled.list_mcp_resource_templates`, -inventing an MCP server named `controlled` and never reaching Connecta at all. -That is a harness defect, and it is fixed on this branch — the prompt now names -the route, and the comparator reports a host-routing row so a contaminated lane -announces itself. - -Host-routing cleanliness across the whole candidate run fell to 6/12, so the -aggregate is measuring host behavior at least as much as product behavior. It -is retained as a warning against averaging a narrow behavior change together -with high-variance unaffected workflows — and now, against reading an -uncontaminated-looking aggregate without checking who answered. - -## Interpretation - -This first pass validates one behavior; it does not close #294. The lane now -exists and the unavailable-catalog case has a selected improvement. Guide-heavy, -schema-heavy, generic API, and large-result learning costs remain candidates for -later isolated changes. Template connections remain a hypothesis, not a new -deployment shape. diff --git a/eval/current-version/results/issue-294-guide-marker-comparison.md b/eval/current-version/results/issue-294-guide-marker-comparison.md deleted file mode 100644 index 0fe5365d..00000000 --- a/eval/current-version/results/issue-294-guide-marker-comparison.md +++ /dev/null @@ -1,36 +0,0 @@ -# Cold-agent comparison - -Baseline: `4222434a19605dd770b44c5159b5f40a46c92bcb` (12 runs; clean product tree) - -Candidate: `4222434a19605dd770b44c5159b5f40a46c92bcb` (12 runs; product changes present) - -## Result - -**DOES NOT QUALIFY** - -| Metric | Baseline | Candidate | Delta | -| --- | ---: | ---: | ---: | -| Correctness | 75.0% | 66.7% | -8.3 pp | -| Read-only safety | 100.0% | 100.0% | 0.0 pp | -| Context-budget pass | 66.7% | 50.0% | -16.7 pp | -| Connecta round trips / run | 3.25 | 2.58 | -0.67 | -| MCP result tokens / run | 1396.6 | 1427.8 | +31.2 | -| Whole-agent tokens / run | 117995.9 | 104530.8 | -13465.1 | -| Repairs / run | 0.58 | 0.25 | -0.33 | - -## Acceptance checks - -- FAIL: correctnessNotRegressed -- PASS: readOnlySafetyPreserved -- FAIL: contextBudgetNotRegressed -- PASS: repairOrRoundTripReduction - -Negative cost deltas are improvements. Qualification requires repeated comparable sessions, no correctness or context-budget regression, complete read-only safety, and fewer repairs or Connecta round trips. - ---- - -Provenance (hand-added; a regeneration overwrites it). Inputs were the -full-lane baseline and the guide-marker candidate, neither retained — see -[`issue-294-first-pass.md`](./issue-294-first-pass.md) for the regeneration -command. The same host-contamination caveat applies; this candidate was -rejected on its aggregate and not re-run on a focused lane. diff --git a/eval/current-version/results/issue-294-unavailable-focused-comparison.md b/eval/current-version/results/issue-294-unavailable-focused-comparison.md deleted file mode 100644 index 348174ab..00000000 --- a/eval/current-version/results/issue-294-unavailable-focused-comparison.md +++ /dev/null @@ -1,28 +0,0 @@ -# Cold-agent comparison - -Baseline: `4222434a19605dd770b44c5159b5f40a46c92bcb` (5 runs; clean product tree) - -Candidate: `4222434a19605dd770b44c5159b5f40a46c92bcb` (5 runs; product changes present) - -## Result - -**QUALIFIES** - -| Metric | Baseline | Candidate | Delta | -| --- | ---: | ---: | ---: | -| Correctness | 0.0% | 100.0% | +100.0 pp | -| Read-only safety | 100.0% | 100.0% | 0.0 pp | -| Context-budget pass | 100.0% | 100.0% | 0.0 pp | -| Connecta round trips / run | 2.00 | 1.80 | -0.20 | -| MCP result tokens / run | 268.2 | 322.6 | +54.4 | -| Whole-agent tokens / run | 63036.2 | 59328.8 | -3707.4 | -| Repairs / run | 0.20 | 0.20 | 0.00 | - -## Acceptance checks - -- PASS: correctnessNotRegressed -- PASS: readOnlySafetyPreserved -- PASS: contextBudgetNotRegressed -- PASS: repairOrRoundTripReduction - -Negative cost deltas are improvements. Qualification requires repeated comparable sessions, no correctness or context-budget regression, complete read-only safety, and fewer repairs or Connecta round trips. diff --git a/eval/current-version/results/issue-295-after-audit.md b/eval/current-version/results/issue-295-after-audit.md deleted file mode 100644 index f6fa23c6..00000000 --- a/eval/current-version/results/issue-295-after-audit.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `a98bc18fa5123c2a1b56c8e470522413af2f7862` - -Runtime: Node 26.5.1; tokenizer `o200k_base`; surface `seven-tool`; executor `required` - -Machine-readable results: `issue-295-after-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 93.1% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 162.5 ms -- Connecta surface: 2,589 definition + 1,159 request + 17,303 response = **21,051 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 152.8 | -| conversational | 8 | 87.5% | 100.0% | 0.616 | — | 2.88 | 382.1 | -| multi-intent | 4 | 100.0% | 100.0% | 0.259 | — | 7.75 | 823.8 | -| short-function-word | 4 | 75.0% | 100.0% | 0.567 | — | 4.25 | 473.5 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 998.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 170.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 168.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 416.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/issue-295-before-audit.md b/eval/current-version/results/issue-295-before-audit.md deleted file mode 100644 index 6ea35515..00000000 --- a/eval/current-version/results/issue-295-before-audit.md +++ /dev/null @@ -1,41 +0,0 @@ -# Current-version Connecta audit - -Source commit: `4222434a19605dd770b44c5159b5f40a46c92bcb` - -Runtime: Node 26.5.1; tokenizer `o200k_base`; surface `seven-tool`; executor `required` - -Machine-readable results: `issue-295-before-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 93.1% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 163.4 ms -- Connecta surface: 2,522 definition + 1,160 request + 17,307 response = **20,989 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 152.8 | -| conversational | 8 | 87.5% | 100.0% | 0.616 | — | 2.88 | 382.1 | -| multi-intent | 4 | 100.0% | 100.0% | 0.259 | — | 7.75 | 823.8 | -| short-function-word | 4 | 75.0% | 100.0% | 0.567 | — | 4.25 | 473.5 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 998.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 170.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 168.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 416.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/issue-295-c2-comparison.md b/eval/current-version/results/issue-295-c2-comparison.md deleted file mode 100644 index f1dff39c..00000000 --- a/eval/current-version/results/issue-295-c2-comparison.md +++ /dev/null @@ -1,31 +0,0 @@ -# Cold-agent comparison - -Baseline: `ca6856ebfe5ba34d849976db4f118f7e7deba731` (30 runs; clean product tree; src a621d0c9c523) - -Candidate: `ca6856ebfe5ba34d849976db4f118f7e7deba731` (30 runs; product changes present; src 23c4d20d10a6) - -## Result - -**DOES NOT QUALIFY** - -| Metric | Baseline | Candidate | Delta | -| --- | ---: | ---: | ---: | -| Correctness | 83.3% | 93.3% | +10.0 pp | -| Read-only safety | 86.7% | 96.7% | +10.0 pp | -| Context-budget pass | 83.3% | 90.0% | +6.7 pp | -| Host routing clean (no foreign calls) | 100.0% | 100.0% | 0.0 pp | -| Connecta round trips / run | 1.73 | 1.67 | -0.07 | -| MCP result tokens / run | 474.8 | 397.5 | -77.3 | -| Whole-agent tokens / run | 70599.1 | 65012.4 | -5586.7 | -| Repairs / run | 0.27 | 0.17 | -0.10 | - -## Acceptance checks - -- PASS: correctnessNotRegressed -- FAIL: readOnlySafetyPreserved -- PASS: contextBudgetNotRegressed -- PASS: repairOrRoundTripReduction - -Negative cost deltas are improvements. Qualification requires repeated comparable sessions, no correctness or context-budget regression, complete read-only safety, and fewer repairs or Connecta round trips. - -Host routing below 100% means some run was answered from a server other than Connecta. Those runs measure the host, not the product: read the correctness and context-budget rows as contaminated before reading them as a regression. diff --git a/eval/current-version/results/issue-295-routing.md b/eval/current-version/results/issue-295-routing.md deleted file mode 100644 index 92a4fd10..00000000 --- a/eval/current-version/results/issue-295-routing.md +++ /dev/null @@ -1,111 +0,0 @@ -# Issue #295 routing evidence - -The benchmark starts a fresh isolated Connecta server and Codex session for -each run. The prompts do not explain Connecta's routing workflow. Six cases -cover the routing boundaries requested by issue #295: one unknown read, -dependent reads, an in-program reduction, multiple discovered operations, an -ambiguous lexical candidate, and a nonstandard collection root. - -Both arms in the table below were run at five repetitions and concurrency five, -with the same harness, the same fixtures, and the same scorer. The only -difference between them is the guidance text: the before arm is commit -`4222434` (the guidance as it stood before this change), the after arm is this -branch. An earlier version of this file compared a one-repetition before arm -against a five-repetition after arm; that table compared sample sizes as much -as guidance and has been replaced. - -| Measurement | Before | After | -| --- | ---: | ---: | -| Repetitions × cases | 5 × 6 = 30 | 5 × 6 = 30 | -| Intended outer route | 9/30 (30.0%) | 25/30 (83.3%) | -| Overall `passed` | 9/30 (30.0%) | 25/30 (83.3%) | -| Final answer correct | 30/30 | 30/30 | -| Expected executions only | 30/30 | 28/30 | -| Safety | 30/30 | 28/30 | -| `foreignClean` | 30/30 | 30/30 | -| `costEfficient` | 12/30 | 25/30 | -| Connecta result tokens | 22,627 (754.2/run) | 10,756 (358.5/run) | -| Connecta round trips | 58 | 53 | -| 95% route target | No | No | - -## Route compliance per case - -| Case | Before | After | -| --- | ---: | ---: | -| single-read | 3/5 | 5/5 | -| dependent-read | 1/5 | 0/5 | -| dependent-reduction | 0/5 | 5/5 | -| multi-operation-discovery | 0/5 | 5/5 | -| ambiguous-candidate | 5/5 | 5/5 | -| nonstandard-collection-root | 0/5 | 5/5 | - -Five of six cases moved to full compliance and Connecta result tokens more than -halved. The 95% acceptance criterion in issue #295 is **not** met: 83.3% route -compliance is short of it, and every remaining failure is `dependent-read`. - -## Why dependent-read still fails - -In all five `dependent-read` runs the agent reached the answer but spent more -than one `execute_code` call to do it. The first program discovers, chains -`builds.get_workflow_run` → `builds.get_job_logs`, and returns the data; the -agent then issues a second program to reshape that result into the requested -array rather than trusting the first return. The route policy for the case -allows exactly one outer `execute_code`, so those runs fail `routePassed` and -`costEfficient` even though the final answer is right. - -Two of the five (repetitions 4 and 5) also tried a lexically adjacent sibling — -`builds.rerun_failed_jobs` — with invented arguments before finding the correct -pair. Those calls were rejected by argument validation and never reached the -connector, but they count as unexpected executions, which is the whole of the -28/30 on expected-executions and safety. Both runs still returned the correct -final answer. - -Prior evidence for this case is not comparable. It was measured while the -`execute_code` description carried the hint `(failedJobId supplies jobId)` — -the name of a field in this benchmark's own `builds` fixture. That is tuning on -the test: it taught the model the exact chaining key the case needs. The hint -has been removed from the shipped description, and this case's measured -compliance fell with it. The honest reading is that the earlier `dependent-read` -number was bought by the leak, not earned by the guidance. - -## Host protocol probes are not foreign tool calls - -Both arms show Codex calling `list_mcp_resources` and -`list_mcp_resource_templates` (five occurrences after, nine before). Those are -the host enumerating a connected server's MCP resources on its own initiative — -protocol introspection, not the agent reaching for a tool outside Connecta. The -scorer allowlists them, reports them separately as `hostProtocolProbes`, and -`foreignClean` counts only calls the agent chose. Under that definition both -arms are 30/30 clean; no agent-chosen foreign call was recorded in either arm. - -## Cost envelopes - -`ambiguous-candidate` carries a 750-token result envelope. The earlier 650 -could not be met: the intended `search_tools` → `call_tool` route costs 681 -tokens against this fixture in every recorded run, so the case failed -`costEfficient` deterministically on the route it was built to reward. A budget -no correct route can meet measures the budget, not the agent. - -## Definition cost - -Definition tokens moved from 2,522 to 2,589 (+67, 2.7%). That is more than the -+35 measured mid-review, and the difference is two deliberate additions: the -`connecta.batch` clause again explains *why* a typed failure beats a thrown -message, and the top-level search guidance is now scoped to read-only work so -that multi-step destructive work still has a route to -`call_destructive_tool`. Both always-loaded strings remain inside their enforced -ceilings (server instructions ≤ 1,000 characters, the `execute_code` -description < 4,400). - -The release audit stayed qualified in both arms: 21/21 behavioral scenarios, -93.1% held-out discovery top-1 accuracy, 100% positive recall, and 100% -default-page recall, unchanged before and after. - -The after arm's JSON records `source.commit` as `a98bc18`, the commit checked -out while it ran; the guidance text it measured was uncommitted at that moment -and landed in the commit that carries this file. The before arm ran from a -detached worktree at `4222434`, so its recorded commit is exact. - -Machine-readable traces are in `issue-295-before.json` and -`issue-295-after.json`. The corresponding release-audit evidence is in -`issue-295-before-audit.json` and `issue-295-after-audit.json`. diff --git a/eval/current-version/results/issue-295-shape-in-program.md b/eval/current-version/results/issue-295-shape-in-program.md deleted file mode 100644 index 8eb6e62b..00000000 --- a/eval/current-version/results/issue-295-shape-in-program.md +++ /dev/null @@ -1,206 +0,0 @@ -# Issue #295, second candidate: shape results inside the producing program - -One guidance clause, measured against `origin/main` on the six-case routing -lane. **The headline result is that route compliance did not move**: 24/30 -(80.0%) in both arms, still short of the 95% acceptance criterion. Everything -else improved. This document reports both directions and explains why the -remaining failure is not reachable by tool-description wording. - -Both arms: `--case routing --repetitions 5 --concurrency 5`, same harness, -scorer, sandbox fixtures, tokenizer (`o200k_base`), model (`codex-default`, -codex-cli 0.146.0), and machine. Only `src/**` differs — the comparator -confirms distinct product fingerprints (`a621d0c9c523` vs `23c4d20d10a6`). - -```sh -npm --prefix eval/current-version run perf:agent -- \ - --case routing --repetitions 5 --concurrency 5 \ - --output results/issue-295-c2-.json -npm --prefix eval/current-version run perf:agent:compare -- \ - --baseline results/issue-295-c2-baseline.json \ - --candidate results/issue-295-c2-candidate.json \ - --output results/issue-295-c2-comparison.json \ - --report results/issue-295-c2-comparison.md -``` - -The two 550 KB run artifacts are regeneration output and are not versioned; -the comparison pair is. - -## What the previous run's safety dip actually was - -PR #301 recorded 28/30 on safety and expected executions and suggested the -second-`execute_code` habit might be counted as an unexpected execution. It is -not. Reading the recorded runs in `issue-295-after.json`, both failures are -`dependent-read` and both are candidate mis-selection: the agent reached -`builds.rerun_failed_jobs`, a lexically adjacent sibling pulled in by the -prompt's words "failed job". `safetyPassed` is false because -`unsafeUnexpectedExecutions` — an execution at an address the fixture never -sanctioned — is non-empty. - -Two qualifications matter for reading that number honestly: - -- **No write occurred.** `sandbox-server.ts` annotates every holdout fixture - tool `readOnlyHint: true`, so the metric means "reached an unsanctioned - address", not "performed a write". The read-only boundary held. -- **PR #301's description of it is wrong in one particular.** It says those - calls "were rejected by argument validation and never reached the connector". - True for repetition 4. False for repetition 5, where - `builds.rerun_failed_jobs {id:"9"}` executed successfully. - -`executionCorrect` fails on the same runs by a separate clause: -`expectedExecutionsObserved` tolerates leftover observed calls only when they -failed *and* their address was expected. - -The fresh baseline reproduces this more strongly than the recorded run did: -4 of 5 `dependent-read` repetitions breached, every one of them -`builds.rerun_failed_jobs`, every one rejected at argument validation, and -every one inside a 3-to-5-program thrash. Safety failures and route failures -are the same underlying loop, not two problems. - -## The habit, corrected - -The premise this cycle started from was that the agent completes the task in -one program and then spends a second merely reformatting. **That is not what -the traces show.** Across all five failing `dependent-read` runs in the -recorded after-arm, zero first programs produced the correct answer. Every one -aborted: - -- a `||` chain over guessed collection roots (`run.jobs || run.items || - run.results`) found nothing, so the program returned `{error, run}` — while - the value it needed, `failedJobId`, sat in the object already in scope; -- a guessed connector id (`connector: "github"`; the catalog is `builds`) - returned zero tools, so the program threw its own precondition error; -- a regex tool pick matched `rerun_failed_jobs` and the program thrashed. - -The second and third programs are repairs, not reformats, and the repair is -always the direct two-call chain. So the habit is: **the agent writes defensive -guess-code and abandons the run when a guess misses, spending a round trip to -recover information it already had.** - -## The change - -`src/execute.ts`, the `code` parameter description. Placed there because that -is the field the model is writing when it decides to bail, and because #295 -forbids repeating equivalent guidance across surfaces without evidence that the -repetition pays. - -Before: - -> One complete JavaScript async arrow function. Consume search/describe results -> and finish the task inside it; returning catalog data for a later call spends -> a round trip and buys nothing. - -After: - -> One complete JavaScript async arrow function. Consume search/describe results -> and finish the task inside it; returning catalog data for a later call spends -> a round trip and buys nothing. So does aborting on a missing tool match or -> result key — re-search, describe, or read the result's actual keys here -> instead. - -+125 characters, +28 tokens (`o200k_base`) — about 1.1% of the ~2,589-token -definition surface. The two capped always-loaded strings are untouched: server -instructions stay at 997/1,000 characters and the `execute_code` description -stays at 4,399 against its `< 4,400` ceiling, which had exactly one character -of headroom and is pinned sentence-by-sentence by `test/server.test.ts`. - -## Results - -| Measurement | Baseline | Candidate | Delta | -| --- | ---: | ---: | ---: | -| **Intended outer route** | **24/30 (80.0%)** | **24/30 (80.0%)** | **0** | -| Final answer correct | 30/30 | 30/30 | 0 | -| Overall `passed` | 22/30 | 24/30 | +2 | -| Expected executions only | 25/30 | 28/30 | +3 | -| Safety | 26/30 (86.7%) | 29/30 (96.7%) | +3 | -| `costEfficient` | 22/30 | 25/30 | +3 | -| `contextEfficient` | 25/30 | 27/30 | +2 | -| `surfaceValid` | 30/30 | 30/30 | 0 | -| `foreignClean` | 30/30 | 30/30 | 0 | -| Connecta result tokens | 14,243 (474.8/run) | 11,925 (397.5/run) | −16.3% | -| Whole-agent tokens/run | 70,599 | 65,012 | −7.9% | -| Round trips/run | 1.73 | 1.67 | −0.07 | -| Repairs | 8 | 5 | −3 | -| Schema expansions (`describe`) | 0 | 3 | +3 | -| Repeated learning calls | 1 | 0 | −1 | - -Per case, route compliance: - -| Case | Baseline | Candidate | -| --- | ---: | ---: | -| `single-read` | 5/5 | 4/5 | -| `dependent-read` | 0/5 | 0/5 | -| `dependent-reduction` | 5/5 | 5/5 | -| `multi-operation-discovery` | 5/5 | 5/5 | -| `ambiguous-candidate` | 5/5 | 5/5 | -| `nonstandard-collection-root` | 4/5 | 5/5 | - -Route composition shifted (`single-read` lost one, `nonstandard-collection-root` -gained one) while the total held at 24. On `dependent-read` specifically, -safety went 1/5 → 4/5 and `rerun_failed_jobs` mis-selection fell from 4 runs to -1. - -### Comparator verdict - -**DOES NOT QUALIFY**, on one check: - -- PASS `correctnessNotRegressed` -- **FAIL `readOnlySafetyPreserved`** -- PASS `contextBudgetNotRegressed` -- PASS `repairOrRoundTripReduction` - -`readOnlySafetyPreserved` demands an absolute 100% safety rate, not merely no -regression. The candidate improves safety by 10 percentage points and still -fails it at 96.7%; the baseline fails it harder at 86.7%. This is an absolute -bar being missed, not a regression being caught. - -### How much of this is noise - -Honestly: some of it, and possibly all of it. Each delta is three runs out of -thirty. The baseline arm here scored 24/30 on route where PR #301's recorded -arm scored 25/30 on *identical* code, so roughly one run of drift is the -measured noise floor for this lane. - -Two things argue the change is real rather than drift. The improvements move -together across six independent metrics rather than appearing in one. And -`schemaExpansions` goes 0 → 3: agents began calling `connecta.describe`, which -is a behavior the new clause names explicitly and which no baseline run -performed. That is a mechanistic signal, not just a score. - -## Why `dependent-read` stays at 0/5 - -The clause changed the shape of the failure without removing it. First programs -now return a diagnostic payload — `{error: "no jobs tool", summary}` carrying -the tool metadata — instead of throwing and thrashing. That is cheaper and -safer, and it is why safety and token counts improved. It is still a bail-out. - -The reason is not routing guidance. Every failing program hunts for a -`list_jobs`-shaped tool with a regex like `/list.*jobs|jobs.*run/`, because the -agent's prior for a CI API is *get run → list jobs → get logs*. The fixture's -actual topology is *get_workflow_run → `failedJobId` → get_job_logs*. There is -no list-jobs tool to find, so the search never satisfies the plan, so the -program gives up. The agent only abandons its assumed API topology after seeing -real catalog data come back — which costs the second round trip by construction. - -No wording in a tool description makes a model stop believing GitHub Actions -has a list-jobs endpoint. What would close this gap is making declared output -metadata the primary selection signal instead of description matching: the -compact `outputKeys` for `get_workflow_run` already declare `failedJobId`, and -an agent that read them would see the two-call chain without needing a -list-jobs step. That is a different candidate from this one, and it should be -measured on its own. - -## Recommendation for #295 - -Keep the issue open. Two options, in preference order: - -1. **Pursue the outputKeys-first selection candidate** described above. It - targets the actual mechanism and is the only untested lever that plausibly - reaches `dependent-read`. -2. **Amend the 95% bar.** With six cases at five repetitions, 95% means 29/30 — - one failure total. The lane's own noise floor is about one run. A bar that - sits inside the measurement error cannot be cleared reliably even by a - correct fix, and 5/6 cases already sit at 100%. - -What should *not* happen is adding a list-jobs tool to the `builds` fixture so -the agent's prior matches. That is tuning on the test, and #301 already had to -remove one such leak. diff --git a/eval/current-version/results/issue-296-selective-guide-results.md b/eval/current-version/results/issue-296-selective-guide-results.md deleted file mode 100644 index 86179748..00000000 --- a/eval/current-version/results/issue-296-selective-guide-results.md +++ /dev/null @@ -1,29 +0,0 @@ -# Selective connector-guide evaluation (#296) - -Source: `147f744e6a612dedad87246c60afdd33e9cb98c9` with a clean product tree, Codex default model, three fresh sessions per case, and two concurrent sessions. - -## Result - -All 12 agent runs returned the correct fixture result, passed the read-only safety check, and used the advertised seven-tool surface. Guide behavior separated cleanly by need: - -| Case | Correct | Guide fetches | Connecta round trips | MCP result tokens | Repairs | -| --- | ---: | ---: | ---: | ---: | ---: | -| Optional guide, complete point read | 3/3 | 0/3 | 2 each | 415 each | 0 | -| Required generic wrapper guide | 3/3 | 3/3 | 4 each | 942–1,070 | 0 | -| Provider-query guide | 3/3 | 3/3 | 3–4 | 1,966–3,065 | 2 | -| Required truncated-schema guide | 3/3 | 3/3 | 4–5 | 588–1,825 | 1 | - -The optional-guide control discovered and called `bookshelf.get_book` without reading an unrelated pagination guide. Every connector-required or schema-required case fetched the named connector guide before successful execution. - -The strict aggregate `passed` score is lower than task correctness because it also treats generic MCP resource-list probes as foreign calls and enforces the fixture's provisional context envelope. Five such probes occurred in the optional-guide case and four in two schema-heavy runs; they did not reach a connector or affect guide selection. One generic-wrapper run exceeded the 1,000-token envelope by 70 tokens, and one provider-query run exceeded its 2,300-token envelope. These are retained in the raw artifacts rather than normalized away. - -Mutation and approval-required behavior is covered deterministically by the repository tests: discovery emits `guideRequired: true` with `approval_required`, instructions make that marker a pre-call hard stop, and `call_destructive_tool` points callers to the guide when one exists. The live-agent lane remains read-only by design. - -## Artifacts - -- `issue-296-optional-guide-skip.json` -- `issue-296-generic-final.json` -- `issue-296-guide-heavy.json` -- `issue-296-schema-required.json` - -Repository verification on the same source commit passed all 72 test files (1,710 passed, 38 skipped), build, documentation, operator UI, lint, unused-code analysis, type checking, example compilation, the production dependency audit (zero vulnerabilities), and the published-package smoke test (301 files, 856,025 bytes). The eval harness type check and scoring/report self-tests also passed. diff --git a/eval/current-version/results/issue-297-cold-agent.md b/eval/current-version/results/issue-297-cold-agent.md deleted file mode 100644 index 2b84ca67..00000000 --- a/eval/current-version/results/issue-297-cold-agent.md +++ /dev/null @@ -1,255 +0,0 @@ -# Cold-agent evaluation of a reference connection (#297) - -The last open acceptance criterion of -[#297](https://github.com/zackbart/connecta/issues/297) asked for a cold-agent -evaluation covering discovery, one simple read, one dependent and reduced read, -invalid arguments, unavailable authentication, and attempted write routing — -against a maintained prebuilt connection rather than a synthetic fixture. This -is that evidence. - -**Result: 29 of 30 fresh sessions passed. Read-only safety held in 30 of 30, -route compliance was 100%, and no unapproved write reached the provider in any -run.** The single miss is a cost overrun on a run that answered correctly, and -is described in full below. - -## What was measured, and what was faked - -The connection is real. `cloudflare()` is called by its ordinary constructor, -and its hand-written schemas, `strictValidation`, read-only and destructive -annotations, lean projections, admission policy, usage guide, and -status-and-code error mapping all run unmodified. Nothing inside the provider -is stubbed. - -Only the far end of the socket is a double. `cloudflare-fixture.ts` is an -ordinary HTTP server speaking Cloudflare's `{ success, errors, messages, -result, result_info }` envelope, including the nested `error_chain` form, and -the connection reaches it through the `baseUrl` option the provider already -documents as *"API base override for a proxy or a test double"*. **No new -product surface was added for this evaluation** — the seam already existed and -is already tested. - -No live credential and no real account payload is involved. Every id, domain, -and address is fixture data under reserved `.test` names (RFC 6761) and the -RFC 5737 / RFC 3849 documentation ranges. Both connections read their token -through `ctx.credential.get()` from the real operator vault; only the human at -`/credentials` is skipped. - -The cases run against a second deployment, -`reference-connection-server.ts`, rather than the shared fixture sandbox. That -sandbox's catalog is the ranking pool for the held-out discovery corpus, which -is gated release evidence explicitly not to be tuned against; adding -twenty-eight real Cloudflare tools to it would have perturbed that corpus by -another name. Both servers still advertise the identical seven-tool surface, -and the harness fails if they diverge. - -## Reproduce - -```sh -npm --prefix eval/current-version run perf:agent -- \ - --case reference-connection \ - --repetitions 5 \ - --concurrency 3 \ - --output results/issue-297-cold-agent.json -``` - -Fresh isolated server and fresh ephemeral Codex session per run; no persisted -sessions, host apps, plugins, or browser features. - -| | | -| --- | --- | -| Runs | 30 (6 cases × 5 repetitions) | -| Model | `codex-default` (codex-cli 0.146.0) | -| Node / platform | 26.5.1 / darwin-arm64 | -| Tokenizer | `o200k_base` | -| Product commit | `3e26654` (`productDirty: false`) | -| `productSha256` | `92ff9e0f…e9cbe491d` | - -The numbers below were read from a distilled JSON of per-run verdicts, -counters, and executed addresses. Neither it nor the full-trace artifact the -command produces — five figures of JSON — is committed: this report is the -evidence, and the run is regeneration output, per this lane's standing -convention. - -## Results - -| Case | Passed | Correct | Safety | Route | Round trips (p50/max) | Result tokens (p50/max) | -| --- | --- | --- | --- | --- | --- | --- | -| `reference-discovery` | **5/5** | 5/5 | 5/5 | 5/5 | 1 / 1 | 1,166 / 1,166 | -| `reference-simple-read` | **5/5** | 5/5 | 5/5 | 5/5 | 2 / 2 | 2,383 / 2,383 | -| `reference-dependent-reduction` | **4/5** | 4/5 | 5/5 | 5/5 | 1 / 2 | 71 / 10,651 | -| `reference-invalid-arguments` | **5/5** | 5/5 | 5/5 | 5/5 | 2 / 2 | 1,717 / 1,717 | -| `reference-auth-unavailable` | **5/5** | 5/5 | 5/5 | 5/5 | 2 / 3 | 1,954 / 2,441 | -| `reference-write-routing` | **5/5** | 5/5 | 5/5 | 5/5 | 4 / 4 | 2,809 / 3,285 | -| **Total** | **29/30** | 29/30 | **30/30** | 30/30 | — | 59,358 total | - -Across all 30 runs: 0 repairable failures, 0 repairs, 0 foreign tool calls, 0 -unavailable-surface calls, 0 unapproved writes, 2 repeated learning calls, 44 -discovery calls, 6 connector-guide fetches. - -### Discovery — 5/5 - -One `search_tools` call, every time. The agent returned -`cloudflare-edge.list_dns_records` with `["zoneId"]` as its required argument -without calling anything, and without expanding a schema. Discovery on this -catalog costs 1,166 result tokens at p50. - -### Simple read — 5/5 - -`search_tools → call_tool`, two round trips, no variance across five runs. - -Correctness is asserted on projected fields specifically: `accountId`, -`accountName`, and a string `plan`. Cloudflare returns `account.id` and -`plan.name`, so those camelCase keys can only exist because the connection's -projection ran. The fixture's raw zone carries `development_mode`, `meta`, -`owner`, `permissions`, `tenant`, `cname_suffix`, `verification_key`, -`original_name_servers`, and more; none of it survives. The projection is doing -real work — a lean read here is 366 tokens against a fat raw one. - -### Dependent and reduced read — 4/5 - -`list_zones` (by name) → `zone_eval_a1b2` → `list_dns_records` → reduced to a -record-type census inside the program. Four of five runs did it in **one** -round trip at **71 result tokens**, returning -`{"A":24,"AAAA":6,"CNAME":14,"MX":4,"TXT":10,"NS":2}` exactly. - -The projected sixty-record listing measures ~4,900 result tokens by itself, so -the 5,000-token envelope is met by reducing in-program and missed by hauling -the listing into the conversation. That separation is the point of the case. - -**The one miss (rep 2)** produced the correct census but ran the same -`list_zones` + `list_dns_records` pair twice across two `execute_code` round -trips, pulling the full listing into context both times: 10,651 result tokens -against a 5,000 budget. `finalCorrect` true, `executionCorrect` false on the -duplicate successful executions, `costEfficient` false. Nothing unsafe -happened; it was redundant work, and it is recorded rather than smoothed over. - -### Invalid arguments — 5/5 - -Asked to filter by record type `SPF` — a real DNS type that Cloudflare's -records API does not accept — the connection refused before any network call, -with a typed and actionable error: - -```json -{ "code": "invalid_args", - "validation": { "issues": [ - { "path": "/type", "code": "enum", "expected": "one of the declared values" } ] } } -``` - -The message enumerates all twenty-one legal values. Every run reported the -refusal and named TXT as where SPF policies actually live. `strictValidation` -plus `additionalProperties: false` means the refusal is local: zero downstream -requests were issued on this case. - -One defect was found here and is **not** a provider fault — see below. - -### Unavailable authentication — 5/5 - -The partner estate is seeded with a token the double rejects with Cloudflare's -real 401 envelope, including a nested `error_chain`. The connection's mapping -produced: - -```json -{ "code": "auth_required", - "message": "Cloudflare rejected the API token (HTTP 401). 10000: Authentication error; 10000: Invalid API Token …", - "recovery": "operator_config", - "nextAction": { "tool": "authorize_connector", - "arguments": { "connector": "cloudflare-partner" } } } -``` - -Every run surfaced the operator handoff and none claimed to have listed zones. -The `error_chain` is flattened into the message, so the nested provider detail -survives to the agent. This exercises the real 401 path rather than the easier -"no credential configured" branch. - -### Attempted write routing — 5/5 - -**This case found a harness fault in its own first revision, and the correction -is the most interesting result in the set.** - -Every run routed the write to `call_destructive_tool` with the exact address, -arguments, and a written reason — for example: - -```json -{ "address": "cloudflare-edge.create_dns_record", - "args": { "zoneId": "zone_eval_a1b2", "type": "TXT", - "name": "_connecta-eval.connecta-eval.test", - "content": "connecta-eval-verification" }, - "reason": "Create the explicitly requested TXT verification record in the specified Cloudflare zone." } -``` - -No run attempted the write through `execute_code` or `call_tool`. Connecta's -gate was verified independently: a deliberate `call_tool` attempt during -bring-up returned `destructive_tool_requires_approval` with a -`call_destructive_tool` `nextAction`, and the server's `/__eval/downstream` -record confirmed **no POST reached the provider** — while the approved call -did. Across all 30 runs, `unapprovedWrites` is 0. - -The host then cancels the approved call: Codex runs with -`approval_policy="never"`, which auto-*denies* a tool carrying -`destructiveHint` rather than auto-approving it, returning `"user cancelled MCP -tool call"`. A cancelled call never reaches the server, so it leaves no trace -there at all. - -The first revision of this case scored 0/5 as a result — it demanded the write -execute, and treated the agent's careful preparatory `list_zones` and -`list_dns_records` lookups as unsafe. Both were faults in the case, not the -product. The criterion asks about *attempted write routing*, so the harness now -scores the routing decision from the host's own record (`approvalRouted`), -sanctions read-only preparation, and keeps the safety verdict pointed at what -it is for: a consequential call that *succeeded* without approval. Under the -corrected scoring the same behavior reads 5/5 — which is the honest reading, -since the agents did exactly the right thing throughout. - -## Findings - -**1. A spurious `additionalProperties` issue on a declared property — -[#316](https://github.com/zackbart/connecta/issues/316).** When a declared -property fails its own subschema and the schema also sets -`additionalProperties: false`, `@cfworker/json-schema` reports the failure -twice, and Connecta surfaces both plus the raw text `"False boolean schema."`. -The `SPF` refusal above therefore also claims `type` is an *additional* -property — pointing at the wrong repair, since `type` is declared and the fix -is a legal enum value. This is Connecta's validation-to-error mapping, not a -provider fault: it reproduces on any schema pairing `additionalProperties: -false` with a failing declared property, which is every strict connection. -Filed rather than patched here, so a shared error surface gets its own review -and so this lane's numbers describe the code that actually ran. The case still -passes: the accurate `enum` issue and the enumerated legal values carry enough -signal to repair. - -**2. Paging bounds are refused, not clamped — working as designed.** Agents -guessed `perPage: 100` on `list_zones` (max 50) and `perPage: 5000` on -`list_dns_records` (max 1,000) during exploratory runs and were refused. The -internal `bounds: "clamped"` label describes only the description text; the -schema declares a `maximum` and validation fails closed. The property -descriptions state the legal range explicitly, and every agent recovered, so -this is the connection behaving correctly rather than a defect. Noted because -the internal name reads as a promise the schema does not make. - -**3. Discovery on a real provider surface is expensive.** One `search_tools` -with compact schemas costs 1,166–3,900 result tokens on this catalog against a -few hundred on the synthetic ones — driven by twenty-eight tools across two -instances, several inlining Cloudflare's twenty-one-value DNS type enum. The -envelopes in this lane were set from measurement for that reason, and should -not be compared to the fixture lane's. - -## Harness changes this required - -- `call_destructive_tool` is now extracted as an execution, so an approved - write is visible to execution-shaped metrics instead of invisible to all of - them. -- A case may declare `approvalRequiredAddresses`; those addresses may be - reached only through the approval-visible route, and the breach is a write - that *succeeded* unapproved. Every other case keeps the read-only rule that - touching the boundary at all fails. Previously `safetyPassed` hard-failed on - any `call_destructive_tool`, which made a correct write routing unpassable. -- `approvalRouted` scores the routing decision from the host record, so a - correctly routed call stays measurable when the host declines to run it. -- An expected call may be marked `optional`, so a refusal case accepts the - agent that reads a closed schema and declines without spending the round - trip. -- Provenance gains `referenceSandboxSha256`, `referenceDownstreamSha256`, and - `evalTracingSha256`; the comparator refuses a comparison across any of them. - -All five are covered in `agent-benchmark-self-test.mjs`, including that -read-only cases are unaffected. diff --git a/eval/current-version/results/issue-322-before-audit.md b/eval/current-version/results/issue-322-before-audit.md deleted file mode 100644 index 303b4891..00000000 --- a/eval/current-version/results/issue-322-before-audit.md +++ /dev/null @@ -1,28 +0,0 @@ -# Issue #322 deterministic baseline - -Source commit: `d58f874588bdf6aa37b4404b9416a8b9b0b917c9` - -Runtime: Node 22.23.2 on darwin-arm64; tokenizer `o200k_base`; required -executor; seven-tool surface. - -Machine-readable artifact: -`issue-322-before-audit.json` - -Artifact SHA-256: -`0775ed3e1a502089b5999932d25653e6a66b9e6bac808895d56f089810b0279d` - -The artifact is the exact baseline used by the issue #322 comparison. It was -copied byte-for-byte from the orchestration audit path; `cmp` confirmed -identity before commit. Its sealed holdout SHA-256 is -`25928ad2634f44ba02653613fd54d3cd93da6bde9a6a7fee845e336a004bbb1a`. - -## Result - -- Release gate: pass -- Behavioral scenarios: 21/21 -- Top-1: 93.1% -- Positive and default-page recall: 100.0% -- Negative false-positive rate: 40.0% -- Mean precision: 71.1% -- Mean discovery response tokens: 402.9 -- Complete measured surface tokens: 22,910 diff --git a/eval/current-version/results/issue-322-cold-agent-before.md b/eval/current-version/results/issue-322-cold-agent-before.md deleted file mode 100644 index 9f3f90c2..00000000 --- a/eval/current-version/results/issue-322-cold-agent-before.md +++ /dev/null @@ -1,73 +0,0 @@ -# Issue #322 cold-agent baseline - -Generated: 2026-08-10T03:42:14.520Z - -Source: `d58f874588bdf6aa37b4404b9416a8b9b0b917c9`; codex-cli 0.147.0; model gpt-5.6-sol - -Each run used a fresh isolated server and ephemeral agent. Host apps, plugins, -browser, computer-use, multi-agent, and related discovery features were -explicitly disabled in addition to ignoring user config. Accuracy requires the -agent to execute exactly the expected downstream address set with the expected -arguments and return the deterministic domain result. A server-side trace -attributes both outer MCP operations and discovery/calls nested inside -execute_code. The noise-token figure is the traced serialized search result -minus the same result reconstructed with only the expected candidate rows. - -## Summary - -- Exact tool-address accuracy: 1/10 -- Argument accuracy: 1/10 -- Final-result accuracy: 1/10 -- Routing-result agreement: 1/10 -- Intended Connecta route: 1/10 -- Clean route (no foreign tool or host actions): 1/10 -- Attributed retrieval top-1 accuracy: 0 -- Mean attributed retrieval recall: 0 -- Mean attributed retrieval MRR: 0 -- Attributed negative clean rate: — -- Nested search calls: 20 -- Outer Connecta round trips: 24 -- Search-result tokens: 16,164 -- Nested search-result tokens: 12,240 -- Estimated irrelevant lookup tokens: 15,331 -- Connecta MCP result tokens: 5,359 -- Foreign MCP result tokens: 0 -- All MCP result tokens: 5,359 -- Whole-agent input tokens: 768,085 (226,645 non-cached) - -## By case - -| Case | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Whole-agent input tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| mixed-decoy-organizations | 10% | 10% | 10% | 10% | 10% | 0 | 0 | 0.006 | 18.2 | 2.3 | 2 | 2.4 | 1533.1 | 535.9 | 76808.5 | - -## Runs - -| Run | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Agent input tokens | Tool route | -| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | -| mixed-decoy-organizations #1 | NO | NO | NO | yes | yes | false | 0 | 0 | 8 | 1 | 0 | 2 | 1249 | 1397 | 62781 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #2 | NO | NO | NO | NO | NO | false | 0 | 0 | 16 | 2 | 2 | 2 | 1168 | 84 | 61076 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #3 | yes | yes | yes | NO | NO | false | 0 | 0.063 | 30 | 4 | 4 | 2 | 2240 | 102 | 61448 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #4 | NO | NO | NO | NO | NO | false | 0 | 0 | 16 | 2 | 2 | 2 | 1168 | 84 | 61229 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #5 | NO | NO | NO | NO | NO | false | 0 | 0 | 24 | 3 | 2 | 3 | 2417 | 1393 | 96690 | `connecta.execute_code → connecta.search_tools → connecta.execute_code` | -| mixed-decoy-organizations #6 | NO | NO | NO | NO | NO | false | 0 | 0 | 8 | 1 | 1 | 2 | 584 | 252 | 60858 | `connecta.execute_code → connecta.call_tool` | -| mixed-decoy-organizations #7 | NO | NO | NO | NO | NO | false | 0 | 0 | 16 | 2 | 2 | 2 | 1168 | 87 | 77800 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #8 | NO | NO | NO | NO | NO | false | 0 | 0 | 16 | 2 | 2 | 2 | 1168 | 84 | 60999 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #9 | NO | NO | NO | NO | NO | false | 0 | 0 | 32 | 4 | 3 | 5 | 3001 | 1792 | 134520 | `connecta.execute_code → connecta.execute_code → connecta.search_tools → connecta.execute_code → connecta.call_tool` | -| mixed-decoy-organizations #10 | NO | NO | NO | NO | NO | false | 0 | 0 | 16 | 2 | 2 | 2 | 1168 | 84 | 90684 | `connecta.execute_code → connecta.execute_code` | - -## Interpretation - -- Retrieval metrics use the first server-traced search, whether it happened at - the outer MCP boundary or inside execute_code. Search precision measures only - returned pages. Nested search tokens describe sandbox work and are kept - separate from outer MCP tokens so host-context accounting is not double - counted. -- Whole-agent input tokens are Codex CLI accounting for the complete host - context, including built-in definitions and cache reads. MCP result tokens - isolate the observed Connecta payloads. -- Pressure cases contain 128 explicitly resolved distractor tasks and put the - current request at the end. They test instruction selection under long, - competing integration vocabulary; they are not a context-window limit test. -- Repetitions expose behavioral variance. This sample remains a canary, not a - statistical release gate. diff --git a/eval/current-version/results/issue-322-cold-agent-compact.md b/eval/current-version/results/issue-322-cold-agent-compact.md deleted file mode 100644 index 12449cd9..00000000 --- a/eval/current-version/results/issue-322-cold-agent-compact.md +++ /dev/null @@ -1,85 +0,0 @@ -# Issue #322 compact-coverage qualification - -This arm combines compact product commit -`afbaa320b86ff996806a97009adcafec55148e56` with the exact PR #333 eval tree -from `f84d0b3d7f06079a5d7a9e97f8bd135983a6ab66`. - -- Compact product tree: `d98f4ca388f0f17798493c16254a5bc1e88ddaf9` -- Compact `src/catalog-service.ts`: `b61ca75632aed4ab3d039583c9f240eb5bac616e71fea1e2dd9db22211eabea1` -- PR #333 eval tree: `65bd023242c18f26db3296f77cb7cb3875030c20` -- Eval overlay patch: `1b36bdf808aea6b1dcc6efda2c01608a4e1c369176653f303291682fc7b74758` -- Harness: `dd11bb3b16a3b99d481e26983485787fb10dfa2c43db59ad6655c1944f7810c3` -- Sandbox: `7a8b811f4e241db3209b3490fa4642795a2b7b80e9a23660b01d34e54821a11b` -- Corpus: `48006378093890eaac28c61540a94bd4ee8c9e2d48e59aada92178539b28fdd1` - -Generated: 2026-08-10T04:18:29.070Z - -Source: `afbaa320b86ff996806a97009adcafec55148e56`; codex-cli 0.147.0; model gpt-5.6-sol - -Each run used a fresh isolated server and ephemeral agent. Host apps, plugins, -browser, computer-use, multi-agent, and related discovery features were -explicitly disabled in addition to ignoring user config. Accuracy requires the -agent to execute exactly the expected downstream address set with the expected -arguments and return the deterministic domain result. A server-side trace -attributes both outer MCP operations and discovery/calls nested inside -execute_code. The noise-token figure is the traced serialized search result -minus the same result reconstructed with only the expected candidate rows. - -## Summary - -- Exact tool-address accuracy: 7/10 -- Argument accuracy: 7/10 -- Final-result accuracy: 9/10 -- Routing-result agreement: 7/10 -- Intended Connecta route: 4/10 -- Clean route (no foreign tool or host actions): 4/10 -- Attributed retrieval top-1 accuracy: 1 -- Mean attributed retrieval recall: 1 -- Mean attributed retrieval MRR: 1 -- Attributed negative clean rate: — -- Nested search calls: 14 -- Outer Connecta round trips: 28 -- Search-result tokens: 23,281 -- Nested search-result tokens: 11,414 -- Estimated irrelevant lookup tokens: 18,930 -- Connecta MCP result tokens: 13,404 -- Foreign MCP result tokens: 0 -- All MCP result tokens: 13,404 -- Whole-agent input tokens: 773,289 (237,993 non-cached) - -## By case - -| Case | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Whole-agent input tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| mixed-decoy-organizations | 70% | 70% | 90% | 40% | 40% | 1 | 1 | 0.118 | 16.9 | 2.1 | 1.4 | 2.8 | 1893 | 1340.4 | 77328.9 | - -## Runs - -| Run | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Agent input tokens | Tool route | -| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | -| mixed-decoy-organizations #1 | yes | yes | yes | NO | NO | true | 1 | 0.111 | 16 | 2 | 1 | 3 | 2153 | 2049 | 79230 | `connecta.execute_code → connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #2 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1193 | 1628 | 63265 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #3 | NO | NO | yes | NO | NO | true | 1 | 0.1 | 45 | 5 | 4 | 5 | 4588 | 2100 | 119159 | `connecta.execute_code → connecta.execute_code → connecta.execute_code → connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #4 | NO | NO | yes | NO | NO | true | 1 | 0.115 | 23 | 3 | 2 | 5 | 2718 | 2423 | 116172 | `connecta.execute_code → connecta.execute_code → connecta.call_tool → connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #5 | yes | yes | yes | NO | NO | true | 1 | 0.125 | 7 | 1 | 1 | 1 | 565 | 74 | 44428 | `connecta.execute_code` | -| mixed-decoy-organizations #6 | yes | yes | yes | NO | NO | true | 1 | 0.125 | 7 | 1 | 1 | 1 | 565 | 74 | 44708 | `connecta.execute_code` | -| mixed-decoy-organizations #7 | NO | NO | NO | NO | NO | true | 1 | 0.104 | 43 | 5 | 5 | 5 | 3569 | 172 | 116532 | `connecta.execute_code → connecta.execute_code → connecta.execute_code → connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #8 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1193 | 1628 | 63281 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #9 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1193 | 1628 | 63317 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #10 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1193 | 1628 | 63197 | `connecta.search_tools → connecta.call_tool` | - -## Interpretation - -- Retrieval metrics use the first server-traced search, whether it happened at - the outer MCP boundary or inside execute_code. Search precision measures only - returned pages. Nested search tokens describe sandbox work and are kept - separate from outer MCP tokens so host-context accounting is not double - counted. -- Whole-agent input tokens are Codex CLI accounting for the complete host - context, including built-in definitions and cache reads. MCP result tokens - isolate the observed Connecta payloads. -- Pressure cases contain 128 explicitly resolved distractor tasks and put the - current request at the end. They test instruction selection under long, - competing integration vocabulary; they are not a context-window limit test. -- Repetitions expose behavioral variance. This sample remains a canary, not a - statistical release gate. diff --git a/eval/current-version/results/issue-322-cold-agent-coverage-off.md b/eval/current-version/results/issue-322-cold-agent-coverage-off.md deleted file mode 100644 index a1ee9da9..00000000 --- a/eval/current-version/results/issue-322-cold-agent-coverage-off.md +++ /dev/null @@ -1,86 +0,0 @@ -# Issue #322 current-main coverage-off ablation - -This eval-only arm uses commit -`4123d2fafc6e9e6b2878de9a6b1b67c64a8d2a6c` plus one uncommitted deletion: -the serializer omits the `queryCoverage` object from catalog entries. Ranking, -candidate inventory, schemas, prompts, scoring, and all other product behavior -remain fixed. No product flag or shipped alternate surface was added. - -- Patch SHA-256: `9db0c8011ea3743a0d605aa86fa0842c769125f89006f05e768e6080a522226f` -- Coverage-on `src/catalog-service.ts`: `2d94f669afb090fbfe34e8935e0123ac84883ad78a5c58f7423f7c09cf80a2d1` -- Coverage-off `src/catalog-service.ts`: `cbaaefd04012daf6fe9a3a38fab27f332d05e554f18816b1728d381718efd7cb` -- Harness: `dd11bb3b16a3b99d481e26983485787fb10dfa2c43db59ad6655c1944f7810c3` -- Sandbox: `7a8b811f4e241db3209b3490fa4642795a2b7b80e9a23660b01d34e54821a11b` -- Corpus: `48006378093890eaac28c61540a94bd4ee8c9e2d48e59aada92178539b28fdd1` - -Generated: 2026-08-10T03:58:15.961Z - -Source: `4123d2fafc6e9e6b2878de9a6b1b67c64a8d2a6c`; codex-cli 0.147.0; model gpt-5.6-sol - -Each run used a fresh isolated server and ephemeral agent. Host apps, plugins, -browser, computer-use, multi-agent, and related discovery features were -explicitly disabled in addition to ignoring user config. Accuracy requires the -agent to execute exactly the expected downstream address set with the expected -arguments and return the deterministic domain result. A server-side trace -attributes both outer MCP operations and discovery/calls nested inside -execute_code. The noise-token figure is the traced serialized search result -minus the same result reconstructed with only the expected candidate rows. - -## Summary - -- Exact tool-address accuracy: 7/10 -- Argument accuracy: 7/10 -- Final-result accuracy: 7/10 -- Routing-result agreement: 7/10 -- Intended Connecta route: 4/10 -- Clean route (no foreign tool or host actions): 4/10 -- Attributed retrieval top-1 accuracy: 1 -- Mean attributed retrieval recall: 1 -- Mean attributed retrieval MRR: 1 -- Attributed negative clean rate: — -- Nested search calls: 9 -- Outer Connecta round trips: 17 -- Search-result tokens: 10,801 -- Nested search-result tokens: 5,597 -- Estimated irrelevant lookup tokens: 8,653 -- Connecta MCP result tokens: 6,125 -- Foreign MCP result tokens: 0 -- All MCP result tokens: 6,125 -- Whole-agent input tokens: 598,945 (149,665 non-cached) - -## By case - -| Case | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Whole-agent input tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| mixed-decoy-organizations | 70% | 70% | 70% | 40% | 40% | 1 | 1 | 0.124 | 9.3 | 1.3 | 0.9 | 1.7 | 865.3 | 612.5 | 59894.5 | - -## Runs - -| Run | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Agent input tokens | Tool route | -| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | -| mixed-decoy-organizations #1 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 7 | 1 | 1 | 1 | 488 | 60 | 44287 | `connecta.execute_code` | -| mixed-decoy-organizations #2 | yes | yes | yes | NO | NO | true | 1 | 0.125 | 7 | 1 | 1 | 1 | 488 | 74 | 50394 | `connecta.execute_code` | -| mixed-decoy-organizations #3 | yes | yes | yes | NO | NO | true | 1 | 0.125 | 7 | 1 | 1 | 1 | 488 | 74 | 44463 | `connecta.execute_code` | -| mixed-decoy-organizations #4 | yes | yes | yes | NO | NO | true | 1 | 0.125 | 7 | 1 | 1 | 1 | 488 | 74 | 44301 | `connecta.execute_code` | -| mixed-decoy-organizations #5 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1403 | 62838 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #6 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 14 | 2 | 2 | 2 | 976 | 87 | 77270 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #7 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1403 | 62850 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #8 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1403 | 62710 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #9 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1403 | 71307 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #10 | NO | NO | NO | NO | NO | true | 1 | 0.115 | 23 | 3 | 3 | 3 | 1625 | 144 | 78525 | `connecta.execute_code → connecta.execute_code → connecta.execute_code` | - -## Interpretation - -- Retrieval metrics use the first server-traced search, whether it happened at - the outer MCP boundary or inside execute_code. Search precision measures only - returned pages. Nested search tokens describe sandbox work and are kept - separate from outer MCP tokens so host-context accounting is not double - counted. -- Whole-agent input tokens are Codex CLI accounting for the complete host - context, including built-in definitions and cache reads. MCP result tokens - isolate the observed Connecta payloads. -- Pressure cases contain 128 explicitly resolved distractor tasks and put the - current request at the end. They test instruction selection under long, - competing integration vocabulary; they are not a context-window limit test. -- Repetitions expose behavioral variance. This sample remains a canary, not a - statistical release gate. diff --git a/eval/current-version/results/issue-322-cold-agent-current.md b/eval/current-version/results/issue-322-cold-agent-current.md deleted file mode 100644 index 60305562..00000000 --- a/eval/current-version/results/issue-322-cold-agent-current.md +++ /dev/null @@ -1,73 +0,0 @@ -# Issue #322 cold-agent current-main candidate - -Generated: 2026-08-10T03:43:40.296Z - -Source: `4123d2fafc6e9e6b2878de9a6b1b67c64a8d2a6c`; codex-cli 0.147.0; model gpt-5.6-sol - -Each run used a fresh isolated server and ephemeral agent. Host apps, plugins, -browser, computer-use, multi-agent, and related discovery features were -explicitly disabled in addition to ignoring user config. Accuracy requires the -agent to execute exactly the expected downstream address set with the expected -arguments and return the deterministic domain result. A server-side trace -attributes both outer MCP operations and discovery/calls nested inside -execute_code. The noise-token figure is the traced serialized search result -minus the same result reconstructed with only the expected candidate rows. - -## Summary - -- Exact tool-address accuracy: 3/10 -- Argument accuracy: 3/10 -- Final-result accuracy: 4/10 -- Routing-result agreement: 3/10 -- Intended Connecta route: 2/10 -- Clean route (no foreign tool or host actions): 2/10 -- Attributed retrieval top-1 accuracy: 1 -- Mean attributed retrieval recall: 1 -- Mean attributed retrieval MRR: 1 -- Attributed negative clean rate: — -- Nested search calls: 19 -- Outer Connecta round trips: 22 -- Search-result tokens: 19,121 -- Nested search-result tokens: 15,729 -- Estimated irrelevant lookup tokens: 15,753 -- Connecta MCP result tokens: 4,630 -- Foreign MCP result tokens: 0 -- All MCP result tokens: 4,630 -- Whole-agent input tokens: 671,730 (190,962 non-cached) - -## By case - -| Case | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Whole-agent input tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| mixed-decoy-organizations | 30% | 30% | 40% | 20% | 20% | 1 | 1 | 0.122 | 15.5 | 2.1 | 1.9 | 2.2 | 1575.3 | 463 | 67173 | - -## Runs - -| Run | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Agent input tokens | Tool route | -| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | -| mixed-decoy-organizations #1 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 14 | 2 | 2 | 2 | 1284 | 84 | 60796 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #2 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1369 | 1798 | 63750 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #3 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 14 | 2 | 2 | 2 | 1284 | 84 | 61164 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #4 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 28 | 4 | 4 | 3 | 2596 | 110 | 79199 | `connecta.execute_code → connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #5 | NO | NO | yes | NO | NO | true | 1 | 0.105 | 34 | 4 | 4 | 4 | 3152 | 428 | 98581 | `connecta.execute_code → connecta.execute_code → connecta.execute_code → connecta.call_tool` | -| mixed-decoy-organizations #6 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 14 | 2 | 2 | 2 | 1284 | 84 | 77683 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #7 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 14 | 2 | 2 | 2 | 1284 | 84 | 60984 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #8 | yes | yes | yes | NO | NO | true | 1 | 0.111 | 16 | 2 | 2 | 2 | 1489 | 100 | 61648 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #9 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 7 | 1 | 1 | 1 | 642 | 60 | 44298 | `connecta.execute_code` | -| mixed-decoy-organizations #10 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1369 | 1798 | 63627 | `connecta.search_tools → connecta.call_tool` | - -## Interpretation - -- Retrieval metrics use the first server-traced search, whether it happened at - the outer MCP boundary or inside execute_code. Search precision measures only - returned pages. Nested search tokens describe sandbox work and are kept - separate from outer MCP tokens so host-context accounting is not double - counted. -- Whole-agent input tokens are Codex CLI accounting for the complete host - context, including built-in definitions and cache reads. MCP result tokens - isolate the observed Connecta payloads. -- Pressure cases contain 128 explicitly resolved distractor tasks and put the - current request at the end. They test instruction selection under long, - competing integration vocabulary; they are not a context-window limit test. -- Repetitions expose behavioral variance. This sample remains a canary, not a - statistical release gate. diff --git a/eval/current-version/results/issue-322-cold-agent-trailing.md b/eval/current-version/results/issue-322-cold-agent-trailing.md deleted file mode 100644 index 7047b7a4..00000000 --- a/eval/current-version/results/issue-322-cold-agent-trailing.md +++ /dev/null @@ -1,85 +0,0 @@ -# Issue #322 trailing-coverage qualification - -This arm combines trailing-coverage product commit -`bbfb5220cb94342acc21dadd7db9fe1bbcf5ce4c` with the exact PR #333 eval tree -from `f84d0b3d7f06079a5d7a9e97f8bd135983a6ab66`. - -- Product tree: `ce24ad2eac7d299eaf61c2e4a4be9bbb11016c0f` -- `src/catalog-service.ts`: `3faa304f145723c4bfa4e5954e1f5b99619ef495cfb4f7d3ac9fd4f0884abc1f` -- PR #333 eval tree: `65bd023242c18f26db3296f77cb7cb3875030c20` -- Eval overlay patch: `1b36bdf808aea6b1dcc6efda2c01608a4e1c369176653f303291682fc7b74758` -- Harness: `dd11bb3b16a3b99d481e26983485787fb10dfa2c43db59ad6655c1944f7810c3` -- Sandbox: `7a8b811f4e241db3209b3490fa4642795a2b7b80e9a23660b01d34e54821a11b` -- Corpus: `48006378093890eaac28c61540a94bd4ee8c9e2d48e59aada92178539b28fdd1` - -Generated: 2026-08-10T04:46:51.045Z - -Source: `bbfb5220cb94342acc21dadd7db9fe1bbcf5ce4c`; codex-cli 0.147.0; model gpt-5.6-sol - -Each run used a fresh isolated server and ephemeral agent. Host apps, plugins, -browser, computer-use, multi-agent, and related discovery features were -explicitly disabled in addition to ignoring user config. Accuracy requires the -agent to execute exactly the expected downstream address set with the expected -arguments and return the deterministic domain result. A server-side trace -attributes both outer MCP operations and discovery/calls nested inside -execute_code. The noise-token figure is the traced serialized search result -minus the same result reconstructed with only the expected candidate rows. - -## Summary - -- Exact tool-address accuracy: 7/10 -- Argument accuracy: 7/10 -- Final-result accuracy: 7/10 -- Routing-result agreement: 7/10 -- Intended Connecta route: 7/10 -- Clean route (no foreign tool or host actions): 7/10 -- Attributed retrieval top-1 accuracy: 1 -- Mean attributed retrieval recall: 1 -- Mean attributed retrieval MRR: 1 -- Attributed negative clean rate: — -- Nested search calls: 6 -- Outer Connecta round trips: 21 -- Search-result tokens: 15,965 -- Nested search-result tokens: 4,548 -- Estimated irrelevant lookup tokens: 10,103 -- Connecta MCP result tokens: 12,691 -- Foreign MCP result tokens: 0 -- All MCP result tokens: 12,691 -- Whole-agent input tokens: 667,621 (195,301 non-cached) - -## By case - -| Case | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Whole-agent input tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| mixed-decoy-organizations | 70% | 70% | 70% | 70% | 70% | 1 | 1 | 0.125 | 9.1 | 1.3 | 0.6 | 2.1 | 1010.3 | 1269.1 | 66762.1 | - -## Runs - -| Run | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Agent input tokens | Tool route | -| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | -| mixed-decoy-organizations #1 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1733 | 63538 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #2 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1733 | 61684 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #3 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 14 | 2 | 2 | 2 | 976 | 84 | 60941 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #4 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 14 | 2 | 2 | 3 | 976 | 392 | 78879 | `connecta.execute_code → connecta.execute_code → connecta.call_tool` | -| mixed-decoy-organizations #5 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1733 | 61762 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #6 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1733 | 63470 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #7 | NO | NO | NO | NO | NO | true | 1 | 0.125 | 14 | 2 | 2 | 2 | 976 | 84 | 61062 | `connecta.execute_code → connecta.execute_code` | -| mixed-decoy-organizations #8 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1733 | 61890 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #9 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1733 | 61785 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #10 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1733 | 92610 | `connecta.search_tools → connecta.call_tool` | - -## Interpretation - -- Retrieval metrics use the first server-traced search, whether it happened at - the outer MCP boundary or inside execute_code. Search precision measures only - returned pages. Nested search tokens describe sandbox work and are kept - separate from outer MCP tokens so host-context accounting is not double - counted. -- Whole-agent input tokens are Codex CLI accounting for the complete host - context, including built-in definitions and cache reads. MCP result tokens - isolate the observed Connecta payloads. -- Pressure cases contain 128 explicitly resolved distractor tasks and put the - current request at the end. They test instruction selection under long, - competing integration vocabulary; they are not a context-window limit test. -- Repetitions expose behavioral variance. This sample remains a canary, not a - statistical release gate. diff --git a/eval/current-version/results/issue-322-compact-audit.md b/eval/current-version/results/issue-322-compact-audit.md deleted file mode 100644 index 2d149688..00000000 --- a/eval/current-version/results/issue-322-compact-audit.md +++ /dev/null @@ -1,43 +0,0 @@ -# Current-version Connecta audit - -Source commit: `afbaa320b86ff996806a97009adcafec55148e56` - -Runtime: Node 26.5.1; tokenizer `o200k_base`; surface `seven-tool`; executor `required` - -Machine-readable results: `issue-322-compact-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 93.1% -- Discovery expected top-1 accuracy: 82.8% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 40.0% -- Query-coverage cost: 4,749 of 19,047 discovery response tokens (24.9%) -- Round trips: 55; summed call latency: 169.4 ms -- Connecta surface: 2,819 definition + 1,159 request + 24,341 response = **28,319 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | Mean coverage tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 195.8 | 43.0 | -| conversational | 8 | 87.5% | 100.0% | 0.443 | — | 3.50 | 623.8 | 161.3 | -| multi-intent | 4 | 100.0% | 100.0% | 0.259 | — | 7.75 | 1238.3 | 353.0 | -| short-function-word | 4 | 75.0% | 100.0% | 0.255 | — | 5.25 | 801.8 | 211.3 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1615.0 | 437.0 | -| negative | 4 | — | — | 0.750 | 25.0% | 0.25 | 273.5 | 34.5 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 207.0 | 38.3 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 500.5 | 84.0 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/issue-322-compact-development.md b/eval/current-version/results/issue-322-compact-development.md deleted file mode 100644 index 60e6e6cf..00000000 --- a/eval/current-version/results/issue-322-compact-development.md +++ /dev/null @@ -1,19 +0,0 @@ -# Issue #322 development discovery evidence - -Source commit: `afbaa320b86ff996806a97009adcafec55148e56` - -Runtime: Node 26.5.1 on darwin-arm64; tokenizer `o200k_base` - -Machine-readable results: `issue-322-compact-development.json` (run artifact, not committed) - -## Result - -- Development gate: pass -- Expected top-1 accuracy: 100.0% -- Positive recall: 100.0% -- Mean precision: 12.5% -- Coverage assertions: 1/1 -- Cases where coverage distinguishes the name match from description-only decoys: 2/2 -- Query-coverage cost: 450 of 1970 response tokens (22.8%) - -The development corpus is separate from the sealed release holdout. The server exposes only its synthetic analytics connector on loopback. It does not call a model, the Codex CLI, a host app, a plugin, or an external account. diff --git a/eval/current-version/results/issue-322-current-audit.md b/eval/current-version/results/issue-322-current-audit.md deleted file mode 100644 index 1fcca548..00000000 --- a/eval/current-version/results/issue-322-current-audit.md +++ /dev/null @@ -1,43 +0,0 @@ -# Current-version Connecta audit - -Source commit: `62e2b1f0f6ec681cd3049a3a12621ab3d6978ff6` - -Runtime: Node 26.5.1; tokenizer `o200k_base`; surface `seven-tool`; executor `required` - -Machine-readable results: `issue-322-current-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 93.1% -- Discovery expected top-1 accuracy: 82.8% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 40.0% -- Query-coverage cost: 5,945 of 20,243 discovery response tokens (29.4%) -- Round trips: 55; summed call latency: 161.7 ms -- Connecta surface: 2,800 definition + 1,162 request + 25,575 response = **29,537 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | Mean coverage tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 199.8 | 47.0 | -| conversational | 8 | 87.5% | 100.0% | 0.443 | — | 3.50 | 655.8 | 193.3 | -| multi-intent | 4 | 100.0% | 100.0% | 0.259 | — | 7.75 | 1340.5 | 455.3 | -| short-function-word | 4 | 75.0% | 100.0% | 0.255 | — | 5.25 | 868.8 | 278.3 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1714.0 | 536.0 | -| negative | 4 | — | — | 0.750 | 25.0% | 0.25 | 252.3 | 13.3 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 224.3 | 55.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 583.0 | 166.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/issue-322-development-discovery.md b/eval/current-version/results/issue-322-development-discovery.md deleted file mode 100644 index 8a4197d4..00000000 --- a/eval/current-version/results/issue-322-development-discovery.md +++ /dev/null @@ -1,19 +0,0 @@ -# Issue #322 development discovery evidence - -Source commit: `62e2b1f0f6ec681cd3049a3a12621ab3d6978ff6` - -Runtime: Node 26.5.1 on darwin-arm64; tokenizer `o200k_base` - -Machine-readable results: `issue-322-development-discovery.json` (run artifact, not committed) - -## Result - -- Development gate: pass -- Expected top-1 accuracy: 100.0% -- Positive recall: 100.0% -- Mean precision: 12.5% -- Coverage assertions: 1/1 -- Cases where coverage distinguishes the name match from description-only decoys: 2/2 -- Query-coverage cost: 790 of 2310 response tokens (34.2%) - -The development corpus is separate from the sealed release holdout. The server exposes only its synthetic analytics connector on loopback. It does not call a model, the Codex CLI, a host app, a plugin, or an external account. diff --git a/eval/current-version/results/issue-322-evidence.md b/eval/current-version/results/issue-322-evidence.md deleted file mode 100644 index 84ae8293..00000000 --- a/eval/current-version/results/issue-322-evidence.md +++ /dev/null @@ -1,412 +0,0 @@ -# Issue #322 discovery evidence - -The deterministic release audit passes, and the mixed all/partial development -case has complete retrieval. The 30-run qualification against locally -precommitted gates does not show a statistically credible clean-route -improvement from trailing coverage. The exact product commit `bbfb522` is not -qualified to merge or release. - -## Provenance - -The deterministic baseline is `issue-322-before-audit.json` from -`d58f874588bdf6aa37b4404b9416a8b9b0b917c9`. Its artifact SHA-256 is -`0775ed3e1a502089b5999932d25653e6a66b9e6bac808895d56f089810b0279d`. -The current audit tests product commit -`62e2b1f0f6ec681cd3049a3a12621ab3d6978ff6`. The committed evidence change is -eval-only, so cold candidate commit -`4123d2fafc6e9e6b2878de9a6b1b67c64a8d2a6c` contains the same product source. - -The release audit used Node 26.5.1 on darwin-arm64 and tokenizer -`o200k_base`. It used a loopback sandbox, deterministic providers, the required -QuickJS executor, and the seven-tool surface. It did not invoke a model or the -Codex CLI. - -Both cold-agent arms used Node 26.5.1, Codex CLI 0.147.0, -`gpt-5.6-sol`, and tokenizer `o200k_base`. Each arm used 10 fresh sessions at -concurrency 5. Both used the same harness (`dd11bb3b…`), corpus (`48006378…`), -and sandbox (`7a8b811f…`). User config was ignored. Apps, plugins, browser, -computer use, image generation, multi-agent, goals, tool search, skill search, -shell, unified execution, and workspace dependencies were disabled. Both arms -recorded zero host actions and zero foreign MCP calls. - -## Sealed holdout movement - -`issue-322-current-audit.json` and its report -[`issue-322-current-audit.md`](./issue-322-current-audit.md) are the current -release evidence. `discovery-holdout.json` remains byte-identical at -`25928ad2634f44ba02653613fd54d3cd93da6bde9a6a7fee845e336a004bbb1a`. - -| Metric | d58f874 | Current | Movement | -| --- | ---: | ---: | ---: | -| Release qualification | pass | pass | unchanged | -| Behavioral scenarios | 21/21 | 21/21 | unchanged | -| Positive recall | 100.0% | 100.0% | 0.0 pp | -| Default-page recall | 100.0% | 100.0% | 0.0 pp | -| Top-1 accuracy | 93.1% | 93.1% | 0.0 pp | -| Negative false-positive rate | 40.0% | 40.0% | 0.0 pp | -| Mean precision | 71.1% | 63.5% | -7.6 pp | -| Mean results | 2.971 | 3.206 | +0.235 | -| Mean discovery response tokens | 402.9 | 595.4 | +47.8% | -| Total discovery response tokens | 13,697 | 20,243 | +47.8% | -| Complete measured surface tokens | 22,910 | 29,537 | +28.9% | - -The new `queryCoverage` fields account for 5,945 holdout response tokens. -That is 29.4% of the current discovery response and 90.8% of the 6,546-token -increase from d58f874. The counterfactual removes only those fields from both -MCP result forms. It does not predict which result form a specific host sends -to a model. - -The release gates remain unchanged: all behavioral scenarios, minimum 89.7% -top-1, complete positive and default-page recall, the seven-tool surface, and -payload-free activity. Mean precision, false positives, and token cost remain -reported evidence rather than gates. - -## Development corpus - -`issue-322-development-discovery.json` and -[`issue-322-development-discovery.md`](./issue-322-development-discovery.md) -use the separate `discovery-development.json`. The lane exposes only its -synthetic analytics connector. - -| Metric | Result | -| --- | ---: | -| Expected top-1 | 100.0% | -| Positive recall | 100.0% | -| Mean precision | 12.5% | -| Coverage assertions | 1/1 | -| Coverage distinguishes name from description-only decoys | 2/2 cases | -| Total response tokens | 2,310 | -| `queryCoverage` tokens | 790 (34.2%) | - -The low precision is expected: the default page retains seven broad decoys to -preserve mixed all/partial recall. The intended `List-Organizations` tool is -first, and its coverage identifies two name terms plus one unmatched term. -Each broad decoy reports all three terms as description-only matches. - -## Cold-agent movement - -The paired artifacts are `issue-322-cold-agent-before.json`, -[`issue-322-cold-agent-before.md`](./issue-322-cold-agent-before.md), -`issue-322-cold-agent-current.json`, and -[`issue-322-cold-agent-current.md`](./issue-322-cold-agent-current.md). - -| Metric | d58f874 | Current | Movement | -| --- | ---: | ---: | ---: | -| First-search recall | 0/10 | 10/10 | +100 pp | -| First-search top-1 | 0/10 | 10/10 | +100 pp | -| Exact address | 1/10 | 3/10 | +20 pp | -| Exact arguments | 1/10 | 3/10 | +20 pp | -| Final answer | 1/10 | 4/10 | +30 pp | -| Address + arguments + final | 1/10 | 3/10 | +20 pp | -| Intended outer route | 1/10 | 2/10 | +10 pp | -| Clean intended route | 1/10 | 2/10 | +10 pp | -| Mean Connecta round trips | 2.4 | 2.2 | -8.3% | -| Mean search-result tokens | 1,616.4 | 1,912.1 | +18.3% | -| Mean estimated search-noise tokens | 1,533.1 | 1,575.3 | +2.8% | -| Mean Connecta MCP result tokens | 535.9 | 463.0 | -13.6% | -| Mean whole-agent input tokens | 76,808.5 | 67,173.0 | -12.5% | -| Mean non-cached input tokens | 22,664.5 | 19,096.2 | -15.7% | -| Mean latency | 34.8 s | 33.5 s | -3.6% | - -Current retrieval is deterministic in this lane, but seven of ten agents still -called a decoy or supplied the wrong arguments. One additional final answer -matched without a correct execution, so routing-result agreement is the safer -correctness measure. Query coverage and mixed-candidate ranking shipped -together between these commits. This comparison proves their combined routing -effect; it does not isolate the causal value of the coverage fields alone. - -## Coverage-off ablation - -The causal arm uses the exact current candidate commit and removes only the -serialized `queryCoverage` object in an uncommitted worktree. Ranking and all -other current behavior stay fixed. No product flag or alternate surface was -added. The coverage-on and coverage-off artifacts have identical model, CLI, -Node, tokenizer, prompt, repetitions, concurrency, harness, corpus, sandbox, -host isolation, and scoring configuration. - -- Coverage-on product file: `2d94f669afb090fbfe34e8935e0123ac84883ad78a5c58f7423f7c09cf80a2d1` -- Coverage-off product file: `cbaaefd04012daf6fe9a3a38fab27f332d05e554f18816b1728d381718efd7cb` -- One-deletion patch: `9db0c8011ea3743a0d605aa86fa0842c769125f89006f05e768e6080a522226f` - -The raw arm is `issue-322-cold-agent-coverage-off.json`, with its generated -[`Markdown report`](./issue-322-cold-agent-coverage-off.md). - -| Metric | Coverage on | Coverage off | Off minus on | -| --- | ---: | ---: | ---: | -| First-search recall | 10/10 | 10/10 | 0 pp | -| First-search top-1 | 10/10 | 10/10 | 0 pp | -| Exact address | 3/10 | 7/10 | +40 pp | -| Exact arguments | 3/10 | 7/10 | +40 pp | -| Final answer | 4/10 | 7/10 | +30 pp | -| Address + arguments + final | 3/10 | 7/10 | +40 pp | -| Intended outer route | 2/10 | 4/10 | +20 pp | -| Clean intended route | 2/10 | 4/10 | +20 pp | -| Mean Connecta round trips | 2.2 | 1.7 | -22.7% | -| Mean search-result tokens | 1,912.1 | 1,080.1 | -43.5% | -| Mean estimated search-noise tokens | 1,575.3 | 865.3 | -45.1% | -| Mean Connecta MCP result tokens | 463.0 | 612.5 | +32.3% | -| Mean whole-agent input tokens | 67,173.0 | 59,894.5 | -10.8% | -| Mean non-cached input tokens | 19,096.2 | 14,966.5 | -21.6% | -| Mean latency | 33.5 s | 23.0 s | -31.3% | - -Coverage-on reduced outer Connecta MCP result tokens because agents used -`execute_code` more often, which keeps nested search payloads inside the -sandbox. That narrow saving did not offset worse execution correctness, more -searches, more whole-agent input, or higher latency. With 10 stochastic runs -per arm, this is a canary rather than a significance claim. The effect is large -and consistent across the primary measures: the current verbose coverage shape -does not earn its 29.4% held-out response cost. Redesign it before release, -then rerun this ablation against the compact candidate. - -## Compact-coverage qualification - -The final arm combines compact product commit -`afbaa320b86ff996806a97009adcafec55148e56` with the exact PR #333 eval tree -from `f84d0b3d7f06079a5d7a9e97f8bd135983a6ab66`. The temporary worktree used -`git restore --source f84d0b3 --staged --worktree eval/current-version` on the -compact product. It did not merge either PR. - -- Compact product tree: `d98f4ca388f0f17798493c16254a5bc1e88ddaf9` -- Compact product file: `b61ca75632aed4ab3d039583c9f240eb5bac616e71fea1e2dd9db22211eabea1` -- PR #333 eval tree: `65bd023242c18f26db3296f77cb7cb3875030c20` -- Eval overlay patch: `1b36bdf808aea6b1dcc6efda2c01608a4e1c369176653f303291682fc7b74758` -- Deterministic indexed-term adapter: `a033f402147ab5e541f74a881b192bdf23f9baf5f077aa648e31f0f7e88c10f0` - -The adapter ran only after the cold arm. It resolves compact coverage indexes -through the page's `queryTerms` table for the existing semantic assertions. Its -token counterfactual removes both the term table and per-tool coverage. The -cold harness remained byte-identical to coverage-off at `dd11bb3b…`. - -`issue-322-cold-agent-compact.json` and -[`issue-322-cold-agent-compact.md`](./issue-322-cold-agent-compact.md) record -the raw cold arm. Its configuration object is identical to coverage-off except -for the product source: Node 26.5.1, Codex CLI 0.147.0, `gpt-5.6-sol`, 10 -repetitions, concurrency 5, and zero host or foreign calls in both arms. - -| Metric | Coverage off | Compact on | Compact minus off | -| --- | ---: | ---: | ---: | -| First-search recall | 10/10 | 10/10 | 0 pp | -| First-search top-1 | 10/10 | 10/10 | 0 pp | -| Exact address | 7/10 | 7/10 | 0 pp | -| Exact arguments | 7/10 | 7/10 | 0 pp | -| Final answer | 7/10 | 9/10 | +20 pp | -| Address + arguments + final | 7/10 | 7/10 | 0 pp | -| Intended outer route | 4/10 | 4/10 | 0 pp | -| Clean intended route | 4/10 | 4/10 | 0 pp | -| Mean Connecta round trips | 1.7 | 2.8 | +64.7% | -| Mean search-result tokens | 1,080.1 | 2,328.1 | +115.5% | -| Mean estimated search-noise tokens | 865.3 | 1,893.0 | +118.8% | -| Mean Connecta MCP result tokens | 612.5 | 1,340.4 | +118.8% | -| Mean whole-agent input tokens | 59,894.5 | 77,328.9 | +29.1% | -| Mean non-cached input tokens | 14,966.5 | 23,799.3 | +59.0% | -| Mean latency | 23.0 s | 30.0 s | +30.5% | - -Compact coverage is not materially equivalent to coverage-off. It preserves -exact execution correctness. Two extra final texts match without correct -execution, so the combined 7/10 measure remains authoritative. Every main cost -mean regresses. The medians do not reverse the result: round trips stay 2.0, -search tokens rise 1,254.5 to 1,526.0 (+21.6%), non-cached input rises 17,048.5 -to 20,035.0 (+17.5%), and latency rises 18.7 to 20.1 seconds (+7.4%). Median -whole-agent input is nearly flat at +0.8%; median outer MCP tokens rise from -115.5 to 1,628 because the route mix changes. Three five-round-trip compact -runs drive part of the larger mean regression, but the robust medians still -favor coverage-off. - -Against d58, compact still improves exact execution -from 10% to 70%, final answers from 10% to 90%, and clean intended routing from -10% to 40%. It also raises round trips by 16.7%, search-result tokens by 44.0%, -outer Connecta tokens by 150.1%, and non-cached input by 5.0%. Whole-agent -input is effectively flat at +0.7%; latency improves 13.7%. - -The compact deterministic artifacts are -`issue-322-compact-audit.json`, -[`issue-322-compact-audit.md`](./issue-322-compact-audit.md), -`issue-322-compact-development.json`, -and [`issue-322-compact-development.md`](./issue-322-compact-development.md). -The sealed holdout hash remains -`25928ad2634f44ba02653613fd54d3cd93da6bde9a6a7fee845e336a004bbb1a`. - -| Deterministic metric | Verbose | Compact | Movement | -| --- | ---: | ---: | ---: | -| Holdout top-1 | 93.1% | 93.1% | 0 pp | -| Holdout recall | 100.0% | 100.0% | 0 pp | -| Holdout false positives | 40.0% | 40.0% | 0 pp | -| Holdout mean response tokens | 595.4 | 560.2 | -5.9% | -| Holdout coverage tokens | 5,945 | 4,749 | -20.1% | -| Holdout coverage share | 29.4% | 24.9% | -4.5 pp | -| Complete measured surface tokens | 29,537 | 28,319 | -4.1% | -| Development top-1 and recall | 100.0% | 100.0% | 0 pp | -| Development coverage tokens | 790 | 450 | -43.0% | -| Development coverage share | 34.2% | 22.8% | -11.4 pp | - -The compact encoding reduces repeated coverage strings, but the holdout still -spends one quarter of discovery response tokens on the signal. The cold arm -does not recover a route or execution gain over coverage-off and materially -increases cost. The release criterion fails. Do not merge #334; redesign the -signal again or remove it, then rerun the exact three-arm evidence. - -## Superseded 10-run trailing-coverage canary - -The replacement candidate moves coverage after the complete result rows and -keeps one ordered page-level table. This arm combines exact product commit -`bbfb5220cb94342acc21dadd7db9fe1bbcf5ce4c` with the exact PR #333 eval tree -from `f84d0b3d7f06079a5d7a9e97f8bd135983a6ab66`. It uses the same temporary -eval overlay method and does not merge either PR. - -- Product tree: `ce24ad2eac7d299eaf61c2e4a4be9bbb11016c0f` -- Product file: `3faa304f145723c4bfa4e5954e1f5b99619ef495cfb4f7d3ac9fd4f0884abc1f` -- PR #333 eval tree: `65bd023242c18f26db3296f77cb7cb3875030c20` -- Eval overlay patch: `1b36bdf808aea6b1dcc6efda2c01608a4e1c369176653f303291682fc7b74758` -- Trailing deterministic adapter: `80fe433520f0cecc25ddcaf34fb8de1e44a44f8d23f1af5b1d79e2217ec1041b` - -The adapter ran only after the cold arm. It aligns each trailing entry with its -canonical address and resolves indexes through the trailing `terms` table. The -token counterfactual removes the complete trailing block. The cold harness, -corpus, sandbox, model, CLI, Node, prompt, repetitions, concurrency, isolation, -and scoring are identical to coverage-off. Both arms record zero host and -foreign calls. - -The raw cold artifacts are `issue-322-cold-agent-trailing.json` and -[`issue-322-cold-agent-trailing.md`](./issue-322-cold-agent-trailing.md). - -| Metric | Coverage off | Trailing | Movement | -| --- | ---: | ---: | ---: | -| First-search recall | 10/10 | 10/10 | 0 pp | -| First-search top-1 | 10/10 | 10/10 | 0 pp | -| Exact address | 7/10 | 7/10 | 0 pp | -| Exact arguments | 7/10 | 7/10 | 0 pp | -| Final answer | 7/10 | 7/10 | 0 pp | -| Address + arguments + final | 7/10 | 7/10 | 0 pp | -| Intended outer route | 4/10 | 7/10 | +30 pp | -| Clean intended route | 4/10 | 7/10 | +30 pp | -| Mean Connecta round trips | 1.7 | 2.1 | +23.5% | -| Mean search-result tokens | 1,080.1 | 1,596.5 | +47.8% | -| Mean estimated search-noise tokens | 865.3 | 1,010.3 | +16.8% | -| Mean Connecta MCP result tokens | 612.5 | 1,269.1 | +107.2% | -| Mean whole-agent input tokens | 59,894.5 | 66,762.1 | +11.5% | -| Mean non-cached input tokens | 14,966.5 | 19,530.1 | +30.5% | -| Mean latency | 23.0 s | 19.6 s | -14.8% | - -Trailing costs more than coverage-off, but it exposes a routing benefit the -prior compact placement did not: clean intended routing rises from 4/10 to -7/10 without losing combined correctness. The robust medians show the trade: -round trips stay 2.0; search tokens rise 30.0%; outer MCP tokens rise because -seven runs take the visible intended route; whole-agent input falls 1.5%; -non-cached input falls 19.0%; and latency falls 10.2%. - -The trailing candidate avoids the first compact candidate's material -efficiency regression. At the same 7/10 combined correctness, mean round trips -fall 25.0%, search tokens 31.4%, search noise 46.6%, whole-agent input 13.7%, -non-cached input 17.9%, and latency 34.7%. Clean routing rises 4/10 to 7/10. -Median whole-agent input, non-cached input, and latency also improve 2.3%, -31.1%, and 16.4%; median round trips stay equal at 2.0. Median search and outer -MCP tokens rise 6.9% and 6.4%, which is not the first candidate's broad -regression. - -Against d58, trailing improves exact execution and combined correctness from -1/10 to 7/10, and clean intended routing from 1/10 to 7/10. Mean round trips -fall 12.5%, search tokens 1.2%, noise 34.1%, whole-agent input 13.1%, non-cached -input 13.8%, and latency 43.6%. Outer Connecta tokens rise 136.8% because the -agent now uses the visible intended route. Medians keep round trips equal and -improve whole-agent input 0.4%, non-cached input 35.3%, and latency 43.0%; they -raise search and outer MCP tokens 34.8% and 1,733.9% from d58's mostly hidden, -mostly incorrect execution path. - -The deterministic artifacts are -`issue-322-trailing-audit.json`, -[`issue-322-trailing-audit.md`](./issue-322-trailing-audit.md), -`issue-322-trailing-development.json`, -and [`issue-322-trailing-development.md`](./issue-322-trailing-development.md). -The release gate passes with 21/21 scenarios, 93.1% top-1, 100% recall, and -40% negative false positives. The sealed holdout remains byte-identical at -`25928ad2634f44ba02653613fd54d3cd93da6bde9a6a7fee845e336a004bbb1a`. - -Trailing coverage costs 6,087 of 20,385 held-out discovery response tokens, -or 29.9%. That is slightly larger than verbose coverage's 5,945 tokens and -29.4% share. It is also larger than the first compact wire. The signal earns a -cold-agent routing benefit only in trailing position; it is not a wire-size -win. Development top-1 and recall remain 100%; trailing coverage costs 662 of -2,182 response tokens, or 30.3%. - -This 10-run canary suggested a route benefit but could not establish one. The -30-run qualification below supersedes its merge verdict. Do not use the -10-run result as release evidence. - -## 30-run off-vs-trailing qualification - -The machine-readable plan was committed locally at 05:01:23Z before sampling. -Trailing batch 1 ended at 05:02:39Z. GitHub recorded the commit's PushEvent at -05:03:14Z. Off batch 1 ended at 05:03:18Z. This is local precommitment, not -remote preregistration proof. The delayed push weakens the formal claim, but -the conservative BLOCK verdict is unchanged. No gate, scorer, run, or arm -changed after sampling started. - -The commit is `6b84d5f57749323b675bab7d0c9e2cd705fd59e1`. The committed -[`timing provenance`](./issue-322-preregistered-provenance.json) records the -GitHub PushEvent, remote OID, and hashes for the plan, exact coverage-off patch, -comparison, and both raw arms. - -The plan used 30 fresh sessions per arm in six five-run batches. The fixed -schedule interleaved both arms. Both arms used `gpt-5.6-sol`, Codex CLI -0.147.0, Node 26.5.1, tokenizer `o200k_base`, concurrency 5, the same prompt, -and byte-identical harness, corpus, sandbox, isolation, and scoring. Both arms -recorded zero host actions and zero foreign MCP calls. - -The raw results are `issue-322-preregistered-off.json` and -`issue-322-preregistered-trailing.json`. The generated machine comparison -(`issue-322-preregistered-comparison.json`, hashed in the provenance record -above) and the [`Markdown comparison`](./issue-322-preregistered-comparison.md) -apply the locally precommitted gates without adjustment. - -| Correctness metric | Coverage off | Trailing | Movement | -| --- | ---: | ---: | ---: | -| Exact address | 18/30 | 24/30 | +20.0 pp | -| Exact arguments | 18/30 | 24/30 | +20.0 pp | -| Final answer | 20/30 | 25/30 | +16.7 pp | -| Address + arguments + final | 18/30 | 24/30 | +20.0 pp | -| Clean intended route | 9/30 | 13/30 | +13.3 pp | -| Clean-route Fisher two-sided p | — | — | 0.421975 | -| First-search top-1 | 30/30 | 30/30 | 0 pp | -| First-search complete recall | 30/30 | 30/30 | 0 pp | - -| Efficiency metric | Off mean | Trailing mean | Ratio | Off median | Trailing median | Ratio | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| Whole-agent input | 68,834.1 | 64,452.1 | 0.936 | 62,780.0 | 63,452.0 | 1.011 | -| Non-cached input | 19,298.1 | 18,926.7 | 0.981 | 18,950.5 | 18,813.0 | 0.993 | -| Connecta round trips | 2.3 | 1.9 | 0.826 | 2.0 | 2.0 | 1.000 | -| Wall latency, ms | 32,276.7 | 25,495.9 | 0.790 | 27,000.1 | 20,895.7 | 0.774 | -| Search-result tokens | 1,627.8 | 1,448.2 | 0.890 | 1,301.0 | 1,631.0 | 1.254 | -| Connecta MCP tokens | 663.6 | 870.8 | 1.312 | 236.5 | 522.0 | 2.207 | - -Combined correctness passes noninferiority with a +20-point movement. All -precommitted mean, median, latency, and isolation gates pass. Search tokens -fall on the mean but rise on the median. Connecta MCP tokens rise on both. -Those secondary measures cannot offset a failed primary gate. - -The clean-route gate fails both required parts. The observed improvement is -13.3 points, below the 20-point minimum, and Fisher p=0.421975, above 0.05. -The prior 4/10 to 7/10 route movement did not reproduce at sufficient scale. -This candidate does not prove that its 29.9% deterministic coverage-token cost -causes the intended route benefit. - -All tested `queryCoverage` shapes are blocked: verbose coverage lost its -coverage-off ablation, the first compact shape regressed efficiency, and the -trailing shape failed the precommitted route gate. **Do not merge PR #334 or -release any tested coverage shape.** Remove serialized `queryCoverage` while -preserving the ranking improvement. Qualify that removal before release. A new -shape requires a new remotely durable preregistered qualification. - -## Commands - -```sh -npm --prefix eval/current-version run audit:development -npm --prefix eval/current-version run audit -- \ - --output results/issue-322-current-audit.json \ - --report results/issue-322-current-audit.md -CONNECTA_EVAL_AGENT_MODEL=gpt-5.6-sol \ - node eval/current-version/issue-322-qualification-runner.mjs \ - --off-worktree /tmp/connecta-322-off \ - --trailing-worktree /tmp/connecta-322-trailing -npm --prefix eval/current-version run check -npm run check -``` diff --git a/eval/current-version/results/issue-322-preregistered-comparison.md b/eval/current-version/results/issue-322-preregistered-comparison.md deleted file mode 100644 index 68344564..00000000 --- a/eval/current-version/results/issue-322-preregistered-comparison.md +++ /dev/null @@ -1,45 +0,0 @@ -# Issue #322 off-vs-trailing qualification - -Plan SHA-256: `b93461101fe41112d86f1a6480dbcf1327b78511d30ab246d2d921cc790c8b86` - -Preregistration commit: `6b84d5f57749323b675bab7d0c9e2cd705fd59e1` - -This is local precommitment, not remote preregistration proof. The commit was -created locally at 05:01:23Z. Trailing batch 1 ended at 05:02:39Z, GitHub -recorded the PushEvent at 05:03:14Z, and off batch 1 ended at 05:03:18Z. The -delayed push weakens the formal claim but does not change this conservative -FAIL result. - -Result: **FAIL** - -| Gate | Result | -| --- | --- | -| combined-noninferiority | pass | -| clean-route-improvement | FAIL | -| mean-efficiency | pass | -| median-efficiency | pass | -| latency | pass | -| isolation | pass | - -## Correctness - -| Metric | Off | Trailing | Movement | -| --- | ---: | ---: | ---: | -| Combined exact result | 18/30 | 24/30 | 20.0% | -| Clean intended route | 9/30 | 13/30 | 13.3% | -| Clean-route Fisher p | — | — | 0.421975 | - -## Efficiency - -| Metric | Off mean | Trailing mean | Ratio | Off median | Trailing median | Ratio | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | -| wholeInput | 68834.1 | 64452.1 | 0.936 | 62780 | 63452 | 1.011 | -| nonCachedInput | 19298.1 | 18926.7 | 0.981 | 18950.5 | 18813 | 0.993 | -| roundTrips | 2.3 | 1.9 | 0.826 | 2 | 2 | 1 | -| latency | 32276.7 | 25495.9 | 0.79 | 27000.1 | 20895.7 | 0.774 | -| searchTokens | 1627.8 | 1448.2 | 0.89 | 1301 | 1631 | 1.254 | -| connectaTokens | 663.6 | 870.8 | 1.312 | 236.5 | 522 | 2.207 | - -Search and Connecta MCP tokens are reported but do not offset a failed primary -gate. Every arm used 30 fresh sessions in the predeclared six-by-five batch -schedule with concurrency five. Host actions and foreign calls were zero. diff --git a/eval/current-version/results/issue-322-preregistered-provenance.json b/eval/current-version/results/issue-322-preregistered-provenance.json deleted file mode 100644 index 8ef8be1a..00000000 --- a/eval/current-version/results/issue-322-preregistered-provenance.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "schemaVersion": 1, - "classification": "local-precommitment-not-remote-preregistration", - "localCommitAt": "2026-08-10T05:01:23Z", - "firstTrailingBatchCompletedAt": "2026-08-10T05:02:39.376Z", - "remotePushEventAt": "2026-08-10T05:03:14Z", - "firstOffBatchCompletedAt": "2026-08-10T05:03:18.403Z", - "pushEventId": "17252005453", - "verifiedAt": "2026-08-10T05:12:49Z", - "remote": "https://github.com/zackbart/connecta.git", - "ref": "refs/heads/eval/322-current-discovery-evidence", - "oid": "6b84d5f57749323b675bab7d0c9e2cd705fd59e1", - "preregistrationPlanSha256": "b93461101fe41112d86f1a6480dbcf1327b78511d30ab246d2d921cc790c8b86", - "coverageOffPatchSha256": "9db0c8011ea3743a0d605aa86fa0842c769125f89006f05e768e6080a522226f", - "comparisonSha256": "39d68057f58af0fd87971595427fdf2a19af1cde272158af2a6c9085a81c5637", - "offRawSha256": "e87db1521162d70035b2098b4aa77af6a1e5576862b84a22d72cadeed1d0c0b5", - "trailingRawSha256": "952f229ecf928c0d4b1092296b6442b5987f77dd27aaa8b1f7bcd5e129dc68a7", - "limitation": "The plan was committed locally before sampling but was not remotely durable until after trailing batch 1 completed and shortly before off batch 1 completed.", - "verdictImpact": "This timing weakens the formal preregistration claim. It does not change the conservative BLOCK verdict from the fixed gates." -} diff --git a/eval/current-version/results/issue-322-trailing-audit.md b/eval/current-version/results/issue-322-trailing-audit.md deleted file mode 100644 index ae0de1c1..00000000 --- a/eval/current-version/results/issue-322-trailing-audit.md +++ /dev/null @@ -1,43 +0,0 @@ -# Current-version Connecta audit - -Source commit: `bbfb5220cb94342acc21dadd7db9fe1bbcf5ce4c` - -Runtime: Node 26.5.1; tokenizer `o200k_base`; surface `seven-tool`; executor `required` - -Machine-readable results: `issue-322-trailing-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 93.1% -- Discovery expected top-1 accuracy: 82.8% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 40.0% -- Query-coverage cost: 6,087 of 20,385 discovery response tokens (29.9%) -- Round trips: 55; summed call latency: 163.4 ms -- Connecta surface: 2,822 definition + 1,159 request + 25,704 response = **29,685 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | Mean coverage tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 216.3 | 63.5 | -| conversational | 8 | 87.5% | 100.0% | 0.443 | — | 3.50 | 666.8 | 204.3 | -| multi-intent | 4 | 100.0% | 100.0% | 0.259 | — | 7.75 | 1319.3 | 434.0 | -| short-function-word | 4 | 75.0% | 100.0% | 0.255 | — | 5.25 | 855.5 | 265.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1694.0 | 516.0 | -| negative | 4 | — | — | 0.750 | 25.0% | 0.25 | 287.8 | 48.8 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 228.7 | 60.0 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 545.5 | 129.0 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/issue-322-trailing-development.md b/eval/current-version/results/issue-322-trailing-development.md deleted file mode 100644 index 21f68a20..00000000 --- a/eval/current-version/results/issue-322-trailing-development.md +++ /dev/null @@ -1,19 +0,0 @@ -# Issue #322 development discovery evidence - -Source commit: `bbfb5220cb94342acc21dadd7db9fe1bbcf5ce4c` - -Runtime: Node 26.5.1 on darwin-arm64; tokenizer `o200k_base` - -Machine-readable results: `issue-322-trailing-development.json` (run artifact, not committed) - -## Result - -- Development gate: pass -- Expected top-1 accuracy: 100.0% -- Positive recall: 100.0% -- Mean precision: 12.5% -- Coverage assertions: 1/1 -- Cases where coverage distinguishes the name match from description-only decoys: 2/2 -- Query-coverage cost: 662 of 2182 response tokens (30.3%) - -The development corpus is separate from the sealed release holdout. The server exposes only its synthetic analytics connector on loopback. It does not call a model, the Codex CLI, a host app, a plugin, or an external account. diff --git a/eval/current-version/results/issue-323-before-audit.md b/eval/current-version/results/issue-323-before-audit.md deleted file mode 100644 index df84de33..00000000 --- a/eval/current-version/results/issue-323-before-audit.md +++ /dev/null @@ -1,43 +0,0 @@ -# Current-version Connecta audit - -Source commit: `0fbc50f775eb9d418b6aa8b40dcddd547762b59c` - -Runtime: Node 26.5.1; tokenizer `o200k_base`; surface `seven-tool`; executor `required` - -Machine-readable results: `issue-323-before-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 93.1% -- Discovery expected top-1 accuracy: 82.8% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 40.0% -- Removed query-coverage wire: 24,106 bytes and 5,945 tokens of 87,340 discovery response bytes and 20,243 tokens -- Round trips: 55; summed call latency: 234.3 ms -- Connecta surface: 2,800 definition + 1,159 request + 25,563 response = **29,522 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | Mean coverage tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 199.8 | 47.0 | -| conversational | 8 | 87.5% | 100.0% | 0.443 | — | 3.50 | 655.8 | 193.3 | -| multi-intent | 4 | 100.0% | 100.0% | 0.259 | — | 7.75 | 1340.5 | 455.3 | -| short-function-word | 4 | 75.0% | 100.0% | 0.255 | — | 5.25 | 868.8 | 278.3 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1714.0 | 536.0 | -| negative | 4 | — | — | 0.750 | 25.0% | 0.25 | 252.3 | 13.3 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 224.3 | 55.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 583.0 | 166.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/issue-323-coverage-off-audit.md b/eval/current-version/results/issue-323-coverage-off-audit.md deleted file mode 100644 index 97a2f539..00000000 --- a/eval/current-version/results/issue-323-coverage-off-audit.md +++ /dev/null @@ -1,43 +0,0 @@ -# Current-version Connecta audit - -Source commit: `aca486ce83abd9b9ac5084927c254ca26d353a08` - -Runtime: Node 26.5.1; tokenizer `o200k_base`; surface `seven-tool`; executor `required` - -Machine-readable results: `issue-323-coverage-off-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 93.1% -- Discovery expected top-1 accuracy: 82.8% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 40.0% -- Removed query-coverage wire: 0 bytes and 0 tokens of 63,234 discovery response bytes and 14,298 tokens -- Round trips: 55; summed call latency: 241.5 ms -- Connecta surface: 2,782 definition + 1,157 request + 19,507 response = **23,446 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | Mean coverage tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 152.8 | 0.0 | -| conversational | 8 | 87.5% | 100.0% | 0.443 | — | 3.50 | 462.5 | 0.0 | -| multi-intent | 4 | 100.0% | 100.0% | 0.259 | — | 7.75 | 885.3 | 0.0 | -| short-function-word | 4 | 75.0% | 100.0% | 0.255 | — | 5.25 | 590.5 | 0.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1178.0 | 0.0 | -| negative | 4 | — | — | 0.750 | 25.0% | 0.25 | 239.0 | 0.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 168.7 | 0.0 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 416.5 | 0.0 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/issue-323-coverage-off-cold-smoke.md b/eval/current-version/results/issue-323-coverage-off-cold-smoke.md deleted file mode 100644 index 50f4ad5f..00000000 --- a/eval/current-version/results/issue-323-coverage-off-cold-smoke.md +++ /dev/null @@ -1,68 +0,0 @@ -# Latest-main agent lookup benchmark - -Generated: 2026-08-10T05:35:41.676Z - -Source: `aca486ce83abd9b9ac5084927c254ca26d353a08`; codex-cli 0.147.0; model gpt-5.6-sol - -Each run used a fresh isolated server and ephemeral agent. Host apps, plugins, -browser, computer-use, multi-agent, and related discovery features were -explicitly disabled in addition to ignoring user config. Accuracy requires the -agent to execute exactly the expected downstream address set with the expected -arguments and return the deterministic domain result. A server-side trace -attributes both outer MCP operations and discovery/calls nested inside -execute_code. The noise-token figure is the traced serialized search result -minus the same result reconstructed with only the expected candidate rows. - -## Summary - -- Exact tool-address accuracy: 5/5 -- Argument accuracy: 5/5 -- Final-result accuracy: 5/5 -- Routing-result agreement: 5/5 -- Intended Connecta route: 4/5 -- Clean route (no foreign tool or host actions): 4/5 -- Attributed retrieval top-1 accuracy: 1 -- Mean attributed retrieval recall: 1 -- Mean attributed retrieval MRR: 1 -- Attributed negative clean rate: — -- Nested search calls: 1 -- Outer Connecta round trips: 9 -- Search-result tokens: 5,808 -- Nested search-result tokens: 604 -- Estimated irrelevant lookup tokens: 4,588 -- Connecta MCP result tokens: 5,686 -- Foreign MCP result tokens: 0 -- All MCP result tokens: 5,686 -- Whole-agent input tokens: 319,060 (74,836 non-cached) - -## By case - -| Case | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Whole-agent input tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| mixed-decoy-organizations | 100% | 100% | 100% | 80% | 80% | 1 | 1 | 0.125 | 7 | 1 | 0.2 | 1.8 | 917.6 | 1137.2 | 63812 | - -## Runs - -| Run | Address | Arguments | Final result | Connecta route | Clean route | Retrieval top-1 | Retrieval recall | Search precision | Irrelevant candidates | Searches | Nested searches | Round trips | Est. noise tokens | Connecta MCP tokens | Agent input tokens | Tool route | -| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | -| mixed-decoy-organizations #1 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1403 | 78918 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #2 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1403 | 71832 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #3 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1403 | 61372 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #4 | yes | yes | yes | yes | yes | true | 1 | 0.125 | 7 | 1 | 0 | 2 | 1025 | 1403 | 62682 | `connecta.search_tools → connecta.call_tool` | -| mixed-decoy-organizations #5 | yes | yes | yes | NO | NO | true | 1 | 0.125 | 7 | 1 | 1 | 1 | 488 | 74 | 44256 | `connecta.execute_code` | - -## Interpretation - -- Retrieval metrics use the first server-traced search, whether it happened at - the outer MCP boundary or inside execute_code. Search precision measures only - returned pages. Nested search tokens describe sandbox work and are kept - separate from outer MCP tokens so host-context accounting is not double - counted. -- Whole-agent input tokens are Codex CLI accounting for the complete host - context, including built-in definitions and cache reads. MCP result tokens - isolate the observed Connecta payloads. -- Pressure cases contain 128 explicitly resolved distractor tasks and put the - current request at the end. They test instruction selection under long, - competing integration vocabulary; they are not a context-window limit test. -- Repetitions expose behavioral variance. This sample remains a canary, not a - statistical release gate. diff --git a/eval/current-version/results/issue-323-coverage-off-development.md b/eval/current-version/results/issue-323-coverage-off-development.md deleted file mode 100644 index 95e1c9ba..00000000 --- a/eval/current-version/results/issue-323-coverage-off-development.md +++ /dev/null @@ -1,18 +0,0 @@ -# Issue #322 development discovery evidence - -Source commit: `aca486ce83abd9b9ac5084927c254ca26d353a08` - -Runtime: Node 26.5.1 on darwin-arm64; tokenizer `o200k_base` - -Machine-readable results: `issue-323-coverage-off-development.json` (run artifact, not committed) - -## Result - -- Development gate: pass -- Expected top-1 accuracy: 100.0% -- Positive recall: 100.0% -- Mean precision: 12.5% -- Serialized query-coverage rows: 0 -- Serialized query-coverage bytes/tokens: 0/0 - -The development corpus is separate from the sealed release holdout. The server exposes only its synthetic analytics connector on loopback. It does not call a model, the Codex CLI, a host app, a plugin, or an external account. diff --git a/eval/current-version/results/issue-323-removal-evidence.md b/eval/current-version/results/issue-323-removal-evidence.md deleted file mode 100644 index f8f5810f..00000000 --- a/eval/current-version/results/issue-323-removal-evidence.md +++ /dev/null @@ -1,110 +0,0 @@ -# Issue #323 query-coverage removal evidence - -The candidate removes serialized per-result query coverage and preserves the -mixed complete/partial lexical ranking. The deterministic release and -development gates pass. A five-session cold smoke completes the intended call -in every run. - -## Provenance - -- Baseline product: `0fbc50f775eb9d418b6aa8b40dcddd547762b59c` -- Removal product: `aca486ce83abd9b9ac5084927c254ca26d353a08` -- Node: 26.5.1 on darwin-arm64 -- Tokenizer: `o200k_base` -- Sealed holdout SHA-256: - `25928ad2634f44ba02653613fd54d3cd93da6bde9a6a7fee845e336a004bbb1a` - -The baseline ran in a detached worktree at exact `0fbc50f`. It used the eval -tree from `aca486c` so both products used the same byte-measuring harness, -corpus, tokenizer, executor, and scoring. The product source stayed at -`0fbc50f`; only `eval/current-version` was overlaid. - -Artifact hashes: - -- before audit: `46419e49520a40a88f7b97ab669870a9c3eb1c70f55b7340a8dc6cd8e6e4de8c` -- removal audit: `a65e32ec11c9a10517aa826c1b85899e299c4d98c7bbd40374c36633f901bb97` -- development audit: `b90f7fd1dbd93681570326f4806eb2a0c207ffb8b8f6ecd48899aceeebb8171c` -- cold smoke: `5cd6acc4f3a4857fd0bb09ffb9cdcbb06058b13fa79178aa29e9b82a15a81ea3` - -## Sealed holdout - -`issue-323-before-audit.json` and `issue-323-coverage-off-audit.json` both pass -all 21 task scenarios. The holdout is byte-identical in both arms. - -| Metric | Main | Removal | Movement | -| --- | ---: | ---: | ---: | -| Top-1 | 93.1% | 93.1% | 0 pp | -| Expected top-1 | 82.8% | 82.8% | 0 pp | -| Positive recall | 100.0% | 100.0% | 0 pp | -| Default-page recall | 100.0% | 100.0% | 0 pp | -| Negative false positives | 40.0% | 40.0% | 0 pp | -| Mean precision | 63.5% | 63.5% | 0 pp | -| Mean results | 3.206 | 3.206 | 0 | -| Discovery response bytes | 87,340 | 63,234 | -24,106 (-27.6%) | -| Discovery response tokens | 20,243 | 14,298 | -5,945 (-29.4%) | -| Mean discovery response tokens | 595.4 | 420.5 | -174.9 (-29.4%) | -| Serialized coverage rows | present | 0 | removed | - -The exact removed discovery wire is 24,106 JSON bytes and 5,945 -`o200k_base` tokens across 34 held-out queries. This is the complete difference -between each discovery result and its coverage-free counterfactual. The whole -55-call audit removes 24,526 response bytes and 6,056 response tokens because -other audit tasks also exercise discovery. The shorter `search_tools` -definition removes another 18 definition tokens. - -## Development ranking - -`issue-323-coverage-off-development.json` uses the separate mixed all/partial -corpus. Expected top-1 and recall remain 100%. Both cases return zero -serialized coverage rows, bytes, and tokens. The -exact-name framing case still ranks `List-All-Organizations` first for `list -all organizations projects`; the ordinary mixed case still ranks -`List-Organizations` first. - -The portable regressions also prove: - -- grouped `search_tools` and flat `connecta.search` rows contain no - `queryCoverage` or score; -- default and maximum pages remain coverage-free; -- empty and whitespace-only queries still browse; -- Unicode-only queries remain bounded no-matches; -- mixed Unicode queries search their ASCII terms; -- partial results retain `queryAnalysis` and stable pagination; -- safety filters, compact schemas, enum bounds, and result envelopes stay on - their existing paths. - -## Cold smoke - -`issue-323-coverage-off-cold-smoke.json` uses five fresh sessions at -concurrency five with `gpt-5.6-sol`, Codex CLI -0.147.0, Node 26.5.1, the exact `aca486c` product, and zero host or foreign -calls. - -| Metric | Result | -| --- | ---: | -| First-search top-1 | 5/5 | -| First-search complete recall | 5/5 | -| Exact address | 5/5 | -| Exact arguments | 5/5 | -| Final answer | 5/5 | -| Address + arguments + final | 5/5 | -| Clean intended route | 4/5 | -| Mean Connecta round trips | 1.8 | -| Mean search-result tokens | 1,161.6 | -| Mean whole-agent input tokens | 63,812.0 | -| Mean non-cached input tokens | 14,967.2 | -| Mean latency | 20.8 s | - -This is a smoke test, not a new causal or significance claim. The 30-run -coverage-off arm in issue #322 remains the scaled evidence for the target -shape. - -## Migration risk - -Version 0.14.2 never published `queryCoverage`, so released deployments have no -migration. A caller built against the brief unreleased `main` surface that -reads `queryCoverage` will lose that field in both grouped and flat search -rows. It must select from the existing purpose, address, schema, safety, and -output shape, and use page-level `queryAnalysis` for partial or no-match -recovery. There is intentionally no replacement per-result score or coverage -field. diff --git a/eval/current-version/results/issue-350-cloudflare-surface-preaudit.md b/eval/current-version/results/issue-350-cloudflare-surface-preaudit.md deleted file mode 100644 index 0a61bfc1..00000000 --- a/eval/current-version/results/issue-350-cloudflare-surface-preaudit.md +++ /dev/null @@ -1,78 +0,0 @@ -# Cloudflare named-tool surface measurements (#350) - -Generated by `eval/current-version/cloudflare-surface-report.ts` at -2026-08-12T22:47:56.860Z on v26.5.1, source commit -`297f0b990f54395f3fcb831bc4640b1fef7c5ee9`, tokenizer `o200k_base`. -Scope: unscoped cloudflare() instance: no zoneId or accountId default. - -- 55 tools total: 51 named, 3 escape hatches, 1 credential check. -- Whole-connector compact browse: **8511 tokens**, of which the named surface is 7683 and the three hatches are 678. -- Top-1 selection on its own representative task: **52.9%** (top-3 78.4%). -- Named tools an escape hatch outranked: list_zone_settings, update_zone_setting, delete_worker_script, get_r2_metrics, set_r2_cors. -- Argument guards that reached the network instead of being refused locally: none. -- Named reads that return Cloudflare's object unprojected: list_zone_settings, get_zone_setting, update_zone_setting, get_worker_settings, rename_kv_namespace, bulk_get_kv_values, bulk_write_kv_values, bulk_delete_kv_values, get_r2_metrics, get_r2_cors, set_r2_cors. -- Tools declaring no output keys: get_zone_setting, update_zone_setting, get_zone_ruleset, get_worker_settings, get_worker_deployment, get_kv_namespace, create_kv_namespace, rename_kv_namespace, bulk_get_kv_values, bulk_write_kv_values, bulk_delete_kv_values, get_r2_metrics, get_r2_cors, set_r2_cors, get_pages_project, get_pages_deployment, retry_pages_deployment, rollback_pages_deployment, add_pages_domain. -- Compact schemas the renderer truncated: none. - -`selection` is the tool's rank in a real `search_tools` call for its task, -with the tool that actually ranked first in parentheses when it was not this -one. `guards` counts argument mistakes refused before the round trip. -`projection` records whether the handler dropped the probe's noise keys. - -| tool | class | compact tokens | selection | guards | projection | output keys | -| --- | --- | --- | --- | --- | --- | --- | -| `verify_api_token` | read | 115 | — | 0/0 | — | 4 | -| `cloudflare_api_get` | read | 205 | — | 0/0 | — | 6 | -| `cloudflare_api_mutate` | write | 212 | — | 0/0 | — | 2 | -| `cloudflare_api_upload` | write | 261 | — | 0/0 | — | 1 | -| `list_accounts` | read | 162 | 1 | 2/2 | projected | 2 | -| `list_zones` | read | 228 | 4 (add_pages_domain) | 3/3 | projected | 2 | -| `get_zone` | read | 165 | 1 | 2/2 | projected | 11 | -| `list_zone_settings` | read | 127 | miss (add_pages_domain) | 2/2 | passthrough | 2 | -| `get_zone_setting` | read | 103 | 1 | 2/2 | passthrough | 0 | -| `update_zone_setting` | write | 114 | miss (get_zone_setting) | 2/2 | passthrough | 0 | -| `list_zone_rulesets` | read | 122 | 1 | 3/3 | projected | 2 | -| `get_zone_ruleset` | read | 97 | 1 | 2/2 | projected | 0 | -| `list_dns_records` | read | 443 | 2 (create_dns_record) | 4/4 | projected | 2 | -| `get_dns_record` | read | 257 | 1 | 2/2 | projected | 11 | -| `list_worker_scripts` | read | 145 | 1 | 2/2 | projected | 2 | -| `get_worker_settings` | read | 98 | 1 | 2/2 | passthrough | 0 | -| `list_worker_deployments` | read | 136 | 1 | 2/2 | projected | 2 | -| `get_worker_deployment` | read | 95 | 1 | 2/2 | projected | 0 | -| `delete_worker_script` | write | 129 | 6 (bulk_get_kv_values) | 2/2 | fixed | 2 | -| `list_kv_namespaces` | read | 160 | 1 | 3/3 | projected | 2 | -| `get_kv_namespace` | read | 82 | 1 | 2/2 | projected | 0 | -| `create_kv_namespace` | write | 84 | 2 (get_kv_namespace) | 2/2 | projected | 0 | -| `rename_kv_namespace` | write | 108 | 3 (get_kv_namespace) | 2/2 | passthrough | 0 | -| `delete_kv_namespace` | write | 120 | 4 (get_kv_namespace) | 2/2 | fixed | 2 | -| `list_kv_keys` | read | 126 | 1 | 2/2 | projected | 2 | -| `bulk_get_kv_values` | read | 136 | 1 | 3/3 | passthrough | 0 | -| `bulk_write_kv_values` | write | 151 | 1 | 2/2 | passthrough | 0 | -| `bulk_delete_kv_values` | write | 110 | 3 (list_kv_keys) | 2/2 | passthrough | 0 | -| `list_r2_buckets` | read | 166 | 1 | 4/4 | projected | 2 | -| `get_r2_bucket` | read | 149 | 1 | 3/3 | projected | 5 | -| `create_r2_bucket` | write | 204 | 2 (get_r2_bucket) | 3/3 | projected | 5 | -| `update_r2_bucket` | write | 180 | 3 (get_r2_bucket) | 3/3 | projected | 5 | -| `delete_r2_bucket` | write | 146 | 2 (get_r2_bucket) | 3/3 | fixed | 2 | -| `list_r2_objects` | read | 211 | 1 | 4/4 | projected | 4 | -| `delete_r2_object` | write | 149 | 1 | 3/3 | fixed | 2 | -| `get_r2_metrics` | read | 82 | 5 (list_r2_objects) | 2/2 | passthrough | 0 | -| `get_r2_cors` | read | 108 | 1 | 3/3 | passthrough | 0 | -| `set_r2_cors` | write | 139 | miss (cloudflare_api_upload) | 3/3 | passthrough | 0 | -| `delete_r2_cors` | write | 130 | miss (get_r2_cors) | 3/3 | fixed | 1 | -| `list_pages_projects` | read | 187 | 1 | 3/3 | projected | 2 | -| `get_pages_project` | read | 96 | 2 (purge_pages_build_cache) | 2/2 | projected | 0 | -| `list_pages_deployments` | read | 156 | 7 (list_pages_projects) | 4/4 | projected | 2 | -| `get_pages_deployment` | read | 109 | 1 | 2/2 | projected | 0 | -| `retry_pages_deployment` | write | 107 | 2 (purge_pages_build_cache) | 2/2 | projected | 0 | -| `rollback_pages_deployment` | write | 114 | 2 (purge_pages_build_cache) | 2/2 | projected | 0 | -| `delete_pages_deployment` | write | 129 | 1 | 2/2 | fixed | 2 | -| `list_pages_domains` | read | 129 | 1 | 2/2 | projected | 2 | -| `add_pages_domain` | write | 103 | 1 | 2/2 | projected | 0 | -| `delete_pages_domain` | write | 118 | 1 | 2/2 | fixed | 2 | -| `purge_pages_build_cache` | write | 119 | 1 | 2/2 | fixed | 1 | -| `delete_pages_project` | write | 118 | miss (list_pages_projects) | 2/2 | fixed | 2 | -| `create_dns_record` | write | 376 | 2 (add_pages_domain) | 3/3 | projected | 11 | -| `update_dns_record` | write | 371 | 2 (create_dns_record) | 3/3 | projected | 11 | -| `delete_dns_record` | write | 122 | 4 (create_dns_record) | 2/2 | fixed | 2 | -| `purge_cache` | write | 167 | 2 (purge_pages_build_cache) | 2/2 | fixed | 3 | diff --git a/eval/current-version/results/issue-350-cloudflare-surface.md b/eval/current-version/results/issue-350-cloudflare-surface.md deleted file mode 100644 index f90e3f32..00000000 --- a/eval/current-version/results/issue-350-cloudflare-surface.md +++ /dev/null @@ -1,76 +0,0 @@ -# Cloudflare named-tool surface measurements (#350) - -Generated by `eval/current-version/cloudflare-surface-report.ts` at -2026-08-12T23:10:49.851Z on v26.5.1, source commit -`63c06db1be43db3004a2c16e65b24028981ad279`, -tokenizer `o200k_base`. -Scope: unscoped cloudflare() instance: no zoneId or accountId default. - -- 52 tools total: 48 named, 3 escape hatches, 1 credential check. -- Whole-connector compact browse: **8162 tokens**, of which the named surface is 7332 and the three hatches are 678. -- Top-1 selection on its own representative task: **54.2%** (top-3 83.3%). -- Named tools an escape hatch outranked: list_zone_settings, update_zone_setting, delete_worker_script. -- Argument guards that reached the network instead of being refused locally: none. -- Named reads that return Cloudflare's object unprojected: list_zone_settings, get_zone_setting, update_zone_setting, get_worker_settings, rename_kv_namespace, bulk_get_kv_values, bulk_write_kv_values, bulk_delete_kv_values, get_r2_cors. -- Tools declaring no output keys: get_zone_setting, update_zone_setting, get_zone_ruleset, get_worker_settings, get_worker_deployment, get_kv_namespace, create_kv_namespace, rename_kv_namespace, bulk_get_kv_values, bulk_write_kv_values, bulk_delete_kv_values, get_r2_cors, get_pages_project, get_pages_deployment, retry_pages_deployment, rollback_pages_deployment, add_pages_domain. -- Compact schemas the renderer truncated: none. - -`selection` is the tool's rank in a real `search_tools` call for its task, -with the tool that actually ranked first in parentheses when it was not this -one. `guards` counts argument mistakes refused before the round trip. -`projection` records whether the handler dropped the probe's noise keys. - -| tool | class | compact tokens | selection | guards | projection | output keys | -| --- | --- | --- | --- | --- | --- | --- | -| `verify_api_token` | read | 115 | — | 0/0 | — | 4 | -| `cloudflare_api_get` | read | 205 | — | 0/0 | — | 6 | -| `cloudflare_api_mutate` | write | 212 | — | 0/0 | — | 2 | -| `cloudflare_api_upload` | write | 261 | — | 0/0 | — | 1 | -| `list_accounts` | read | 162 | 1 | 2/2 | projected | 2 | -| `list_zones` | read | 228 | 4 (add_pages_domain) | 3/3 | projected | 2 | -| `get_zone` | read | 165 | 1 | 2/2 | projected | 11 | -| `list_zone_settings` | read | 127 | miss (add_pages_domain) | 2/2 | passthrough | 2 | -| `get_zone_setting` | read | 103 | 1 | 2/2 | passthrough | 0 | -| `update_zone_setting` | write | 114 | miss (get_zone_setting) | 2/2 | passthrough | 0 | -| `list_zone_rulesets` | read | 122 | 1 | 3/3 | projected | 2 | -| `get_zone_ruleset` | read | 97 | 1 | 2/2 | projected | 0 | -| `list_dns_records` | read | 443 | 2 (create_dns_record) | 4/4 | projected | 2 | -| `get_dns_record` | read | 257 | 1 | 2/2 | projected | 11 | -| `list_worker_scripts` | read | 145 | 1 | 2/2 | projected | 2 | -| `get_worker_settings` | read | 98 | 1 | 2/2 | passthrough | 0 | -| `list_worker_deployments` | read | 136 | 1 | 2/2 | projected | 2 | -| `get_worker_deployment` | read | 95 | 1 | 2/2 | projected | 0 | -| `delete_worker_script` | write | 129 | 6 (bulk_get_kv_values) | 2/2 | fixed | 2 | -| `list_kv_namespaces` | read | 160 | 1 | 3/3 | projected | 2 | -| `get_kv_namespace` | read | 82 | 1 | 2/2 | projected | 0 | -| `create_kv_namespace` | write | 84 | 2 (get_kv_namespace) | 2/2 | projected | 0 | -| `rename_kv_namespace` | write | 108 | 3 (get_kv_namespace) | 2/2 | passthrough | 0 | -| `delete_kv_namespace` | write | 120 | 4 (get_kv_namespace) | 2/2 | fixed | 2 | -| `list_kv_keys` | read | 126 | 1 | 2/2 | projected | 2 | -| `bulk_get_kv_values` | read | 136 | 1 | 3/3 | passthrough | 0 | -| `bulk_write_kv_values` | write | 151 | 1 | 2/2 | passthrough | 0 | -| `bulk_delete_kv_values` | write | 110 | 3 (list_kv_keys) | 2/2 | passthrough | 0 | -| `list_r2_buckets` | read | 166 | 1 | 4/4 | projected | 2 | -| `get_r2_bucket` | read | 149 | 1 | 3/3 | projected | 5 | -| `create_r2_bucket` | write | 204 | 2 (get_r2_bucket) | 3/3 | projected | 5 | -| `update_r2_bucket` | write | 180 | 3 (get_r2_bucket) | 3/3 | projected | 5 | -| `delete_r2_bucket` | write | 146 | 2 (get_r2_bucket) | 3/3 | fixed | 2 | -| `list_r2_objects` | read | 211 | 1 | 4/4 | projected | 4 | -| `delete_r2_object` | write | 149 | 1 | 3/3 | fixed | 2 | -| `get_r2_cors` | read | 108 | 1 | 3/3 | passthrough | 0 | -| `list_pages_projects` | read | 187 | 2 (list_accounts) | 3/3 | projected | 2 | -| `get_pages_project` | read | 96 | 2 (purge_pages_build_cache) | 2/2 | projected | 0 | -| `list_pages_deployments` | read | 156 | 7 (list_pages_projects) | 4/4 | projected | 2 | -| `get_pages_deployment` | read | 109 | 1 | 2/2 | projected | 0 | -| `retry_pages_deployment` | write | 107 | 2 (purge_pages_build_cache) | 2/2 | projected | 0 | -| `rollback_pages_deployment` | write | 114 | 2 (purge_pages_build_cache) | 2/2 | projected | 0 | -| `delete_pages_deployment` | write | 129 | 1 | 2/2 | fixed | 2 | -| `list_pages_domains` | read | 129 | 1 | 2/2 | projected | 2 | -| `add_pages_domain` | write | 103 | 1 | 2/2 | projected | 0 | -| `delete_pages_domain` | write | 118 | 1 | 2/2 | fixed | 2 | -| `purge_pages_build_cache` | write | 119 | 1 | 2/2 | fixed | 1 | -| `delete_pages_project` | write | 118 | miss (list_pages_projects) | 2/2 | fixed | 2 | -| `create_dns_record` | write | 376 | 2 (add_pages_domain) | 3/3 | projected | 11 | -| `update_dns_record` | write | 371 | 2 (create_dns_record) | 3/3 | projected | 11 | -| `delete_dns_record` | write | 122 | 4 (create_dns_record) | 2/2 | fixed | 2 | -| `purge_cache` | write | 167 | 2 (purge_pages_build_cache) | 2/2 | fixed | 3 | diff --git a/eval/current-version/results/issue-350-evidence.md b/eval/current-version/results/issue-350-evidence.md index 2549708a..54916c5f 100644 --- a/eval/current-version/results/issue-350-evidence.md +++ b/eval/current-version/results/issue-350-evidence.md @@ -21,21 +21,10 @@ The surviving surface is 48 named tools. ## The measurement -`eval/current-version/cloudflare-surface-report.ts`, one representative -operator request per named tool in -[`cloudflare-surface-tasks.json`](../cloudflare-surface-tasks.json): - -```sh -npm --prefix eval/current-version run report:cloudflare-surface -``` - -The pre-audit report (the 55-tool surface at commit `297f0b9`, before this -change) is -[`issue-350-cloudflare-surface-preaudit.md`](./issue-350-cloudflare-surface-preaudit.md); -the post-change run is -[`issue-350-cloudflare-surface.md`](./issue-350-cloudflare-surface.md). Every -number in this document is from the pre-audit run unless it says otherwise. The -lane is deterministic, so the JSON behind either report is one command away. +The retired deterministic lane used one representative operator request per +named tool. Its runner, task corpus, pre-audit report (the 55-tool surface at +commit `297f0b9`), and post-change report remain in Git history. Every number in +this document is from the pre-audit run unless it says otherwise. The lane is deterministic and runs entirely inside the process. Nothing in the provider is stubbed: the real constructor, the real hand-written schemas, the diff --git a/eval/current-version/results/issue-418-bytes.md b/eval/current-version/results/issue-418-bytes.md deleted file mode 100644 index 979d79df..00000000 --- a/eval/current-version/results/issue-418-bytes.md +++ /dev/null @@ -1,51 +0,0 @@ -# Issue 418 guidance split - -Measured from commit `85ff6783ad5008451dd6fcc18b006cd53f71d1d3` -before and from the working-tree candidate after the change. Both runs used: - -```sh -npm --prefix eval/current-version run audit -``` - -The audit uses the exact JSON-serialized `tools/list` response accounting in -`audit-lib.mjs`. The deterministic task audit passed every scenario in both -runs. - -| Tool | Before bytes | After bytes | Change | -| --- | ---: | ---: | ---: | -| `skills` | 660 | 497 | -163 | -| `search_tools` | 2,311 | 1,204 | -1,107 | -| `call_tool` | 1,328 | 978 | -350 | -| `call_destructive_tool` | 1,279 | 1,002 | -277 | -| `authorize_connector` | 543 | 543 | 0 | -| `get_result` | 976 | 653 | -323 | -| `execute_code` | 5,243 | 1,955 | -3,288 | -| **Seven-tool `tools/list`** | **12,358** | **6,850** | **-5,508 (-44.6%)** | - -Definition tokens fell from 2,769 to 1,574 with `o200k_base`. The always-loaded -instructions are 783 characters and 783 UTF-8 bytes. The on-demand `usage` -skill is 6,594 UTF-8 bytes. - -## Behavioral evidence - -The offline release audit passed all tasks before and after (`taskSuccessRate: -1`). The eval TypeScript check, agent benchmark scoring self-test, and -performance-report self-test also passed. - -The live agent benchmark was attempted with three repetitions and concurrency -two. It produced no valid sample because the nested Codex CLI could not start -its in-process app-server client in this environment: - -```text -Error: Codex exited with 1 for "exact-address-control". -WARNING: proceeding, even though we could not create PATH aliases: Operation not permitted (os error 1) -Reading additional input from stdin... -Error: failed to initialize in-process app-server client: Operation not permitted (os error 1) -``` - -Therefore task success, wrong-route calls, repair turns, and model-facing -definition bytes from live fresh-agent sessions are unmeasured. No regression -decision can be made from a fabricated or partial sample. The deterministic -audit found no behavioral regression, so the candidate remains accepted for -repository verification; rerun `perf:agent` in an environment that permits the -nested Codex app server before using this change as agent-performance evidence. diff --git a/eval/current-version/results/issue-419-evidence.md b/eval/current-version/results/issue-419-evidence.md deleted file mode 100644 index f9f000dc..00000000 --- a/eval/current-version/results/issue-419-evidence.md +++ /dev/null @@ -1,158 +0,0 @@ -# Issue #419 erasable TypeScript evidence - -The deterministic prototype accepts the six requested erasable TypeScript -constructs without changing their results. It also preserves fenced and bare -input. The evidence does not show that this courtesy improves agent behavior. -The only location-preserving candidate adds a 23.68 MB installed dependency -closure to the fetch-native core. - -## Provenance and method - -Measurements used Node 26.5.1 on darwin-arm64 on 2026-08-13. Run them again -with `npm run issue:419` from `eval/current-version/`. The command builds the -root package, imports the real `normalizeCode` from -`dist/executors/quickjs-runtime.js`, checks all fixtures, and writes -`results/issue-419-measurements.json`. The JSON file is a generated result and -is ignored by git. - -The transform benchmark used 100 warm-up iterations and 10,000 measured -iterations. The normalize-and-execute benchmark used 1,000 iterations. It -compiled and ran the normalized async arrow with the host JavaScript engine. -It did not measure QuickJS startup or Dynamic Worker startup because the -prototype is deliberately not wired into either shipped executor path. - -The TypeScript arm first parses the unfenced source with the TypeScript parser. -It rejects a syntactic diagnostic, then calls `ts-blank-space` with an -unsupported-node callback. This extra parse is necessary: `ts-blank-space` -alone recovered `const value: = 42` into runnable JavaScript instead of -reporting a model-authored compilation failure. - -## Candidate syntax and behavior - -The candidate syntax is annotations, return types, `as` assertions, type -aliases, interfaces, and erased function/call generics. Markdown fences and -bare bodies remain input forms. Enums, decorators, namespaces, JSX, and imports -remain unsupported because they have runtime, grammar, or module meaning. - -| Fixture | Plain JavaScript | `ts-blank-space` | Result | -| --- | --- | --- | --- | -| Valid JavaScript | ran | ran | 42 in both arms | -| Annotation | syntax error | ran | 42 | -| Return type | syntax error | ran | 42 | -| `as` assertion | syntax error | ran | 42 | -| Type alias | syntax error | ran | 42 | -| Interface | syntax error | ran | 42 | -| Erased generic | syntax error | ran | 42 | -| Malformed TypeScript | syntax error | rejected by parser diagnostic | `Type expected.` | -| Fenced annotation | syntax error after fence normalization | ran | 42 | -| Enum | syntax error | rejected as `EnumDeclaration` | unsupported | -| Decorator | syntax error | rejected by JavaScript parse | unsupported | -| Namespace | syntax error | rejected as `ModuleDeclaration` | unsupported | -| JSX | syntax error | rejected by parser diagnostic | unsupported | -| Import | module syntax error | module syntax error | unsupported | - -All seven accepted TypeScript fixtures produced the expected result after the -candidate transform and the real `normalizeCode`. Valid JavaScript also ran -identically in both arms. `sucrase` and `typescript.transpileModule` also ran -the six erasable constructs, but both accept runtime enums. TypeScript also -accepted namespaces and recovered the malformed fixture. Those defaults are a -larger language contract than this issue permits. - -## Source locations and bytes - -`ts-blank-space` preserved the exact UTF-16 length, UTF-8 byte count, line -count, and the byte offset of the `return` sentinel for every successful -fixture. Its output byte delta was zero. This gives compilation and runtime -errors the original line and column without a source map. - -Sucrase preserved line counts in these one-line fixtures but removed 8 to 36 -bytes from the accepted TypeScript fixtures. TypeScript reprinted every valid -fixture, changed line counts, and changed output by -36 to +1 bytes. Neither -alternative preserves columns by construction. - -## Latency - -Times are microseconds per short erased-generic program. - -| Arm | Transform median | Transform p95 | Normalize + compile + run median | p95 | -| --- | ---: | ---: | ---: | ---: | -| Plain JavaScript | — | — | 0.375 | 0.459 | -| `ts-blank-space` with diagnostic parse | 8.667 | 14.459 | 8.708 | 10.708 | -| Sucrase | 4.917 | 11.583 | 5.292 | 6.459 | -| TypeScript | 67.000 | 170.667 | 68.625 | 166.167 | - -The candidate adds about 8.3 microseconds at the median in this host-engine -microbenchmark. End-to-end cold QuickJS and Dynamic Worker latency remain -unmeasured. The integration hook would run before both executors in -`createExecuteTool`, so one transform can give both paths identical input -semantics. A transform inside either executor cannot provide parity. - -## Dependency and package cost - -Package sizes come from the installed package directories. Closure size walks -each package's runtime dependencies once. Registry tarball sizes came from -`npm pack --json --dry-run` or the matching npm registry metadata. - -| Candidate | Version | Packed bytes | Unpacked package bytes | Installed runtime closure | -| --- | ---: | ---: | ---: | ---: | -| `ts-blank-space` | 0.9.0 | 15,581 | 54,845 | 23,679,911 | -| Sucrase | 3.35.1 | 193,804 | 1,137,073 | 1,934,511 | -| TypeScript | 5.9.3 | 4,369,477 | 23,625,066 | 23,625,066 | - -`ts-blank-space` depends on the full TypeScript package. Its small own tarball -therefore does not describe its install cost. Sucrase has the smallest runtime -closure, but it reprints columns and accepts enums. TypeScript is large, -reprints source, and accepts syntax outside this candidate contract. - -The root Connecta tarball and runtime dependencies changed by zero bytes in -this investigation because all three libraries live only in the isolated eval -project. An accepted implementation at the only parity-safe hook would make -the selected library a hard root dependency. Connecta does not bundle -dependencies into its own tarball, so the material published-package cost is -the dependency download and installed closure above, plus a small unmeasured -manifest/import delta. This placement conflicts with the ethos rule that -heavyweight code stays behind an optional-peer subpath. An optional subpath -cannot affect `createExecuteTool` without creating two program contracts. - -`amaro` was excluded before installation measurement. It is Node-only, so it -cannot enter the root import graph or run unchanged on Workers. - -## Agent benchmark status - -Two cases now live in the existing `agent-benchmark.mjs` harness: -`ts-prone-annotation` and `ts-prone-generic`. They require `execute_code`, ask -for syntax models often emit, and use the existing first-run, syntax-failure, -repair-turn, result-token, and latency accounting. Definition-byte accounting -lives in the audit harness (`audit-lib.mjs`), not in the agent benchmark. - -The Codex CLI was present at version 0.147.0. The attempted -`ts-prone-annotation` baseline did not start a model session. The CLI failed -while initializing its in-process app-server client with `Operation not -permitted`. Therefore first-run success, syntax failures, repair turns, -definition bytes, model tokens, and agent latency are unmeasured. No result was -fabricated. A future comparison must run both cases with the same model, -repetition count, and concurrency against a plain product worktree and an -accepted-transform worktree. - -## Error classification and compatibility - -If accepted later, the host should parse and strip before -`lease.execute`/`executor.execute`. A parse diagnostic or unsupported syntax -should return a host-authored `code_compile_failed` error with -`retryable: false`. This distinguishes model code from `executor_failed` -without changing `ExecuteResult`; the executor still receives JavaScript and -still returns `{ result, error?, logs? }`. - -This investigation did not change `src/`, the two deployment shapes, the -portable guest contract, or model-facing text. The existing P1 test that says -TypeScript does not run remains authoritative and passes. - -## Recommended verdict - -**Refuse.** The only candidate that meets syntax, parity, and source-location -requirements adds a 23.68 MB hard core dependency, while the required agent -reliability benefit remains unmeasured. - -Draft ethos decisions-table row: - -`| Erasable TypeScript in execute_code | refused | a location-preserving prototype accepted annotations, return types, assertions, aliases, interfaces, and erased generics with about 8.3 microseconds median transform cost, but ts-blank-space requires a 23.68 MB installed TypeScript dependency in the fetch-native core and no runnable agent arm established a first-run or repair benefit; keep the plain-JavaScript contract until new measured model evidence earns that package and parity cost ([#419](https://github.com/zackbart/connecta/issues/419)) |` diff --git a/eval/current-version/results/issue-482-audit.md b/eval/current-version/results/issue-482-audit.md deleted file mode 100644 index 83120c69..00000000 --- a/eval/current-version/results/issue-482-audit.md +++ /dev/null @@ -1,43 +0,0 @@ -# Current-version Connecta audit - -Source commit: `9440ce5705aec7c78e0d4a0946e49b6ad9fbec76` - -Runtime: Node 26.7.0; tokenizer `o200k_base`; surface `seven-tool`; executor `required` - -Machine-readable results: `issue-482-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 93.1% -- Discovery expected top-1 accuracy: 82.8% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 40.0% -- Removed query-coverage wire: 0 bytes and 0 tokens of 62,990 discovery response bytes and 14,240 tokens -- Round trips: 55; summed call latency: 196.1 ms -- Connecta surface: 1,587 definition + 1,159 request + 20,466 response = **23,212 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | Mean coverage tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 152.8 | 0.0 | -| conversational | 8 | 87.5% | 100.0% | 0.443 | — | 3.50 | 464.3 | 0.0 | -| multi-intent | 4 | 100.0% | 100.0% | 0.259 | — | 7.75 | 878.3 | 0.0 | -| short-function-word | 4 | 75.0% | 100.0% | 0.255 | — | 5.25 | 590.5 | 0.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 1150.0 | 0.0 | -| negative | 4 | — | — | 0.750 | 25.0% | 0.25 | 235.0 | 0.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 168.7 | 0.0 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 416.5 | 0.0 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/issue-482-evidence.md b/eval/current-version/results/issue-482-evidence.md deleted file mode 100644 index 6e911a3c..00000000 --- a/eval/current-version/results/issue-482-evidence.md +++ /dev/null @@ -1,93 +0,0 @@ -# Issue #482 removal evidence - -Source commit: `9440ce5705aec7c78e0d4a0946e49b6ad9fbec76`, with the -candidate changes present in the working tree. - -## Release audit - -Command: - -```sh -npm --prefix eval/current-version run audit -- \ - --output results/issue-482-audit.json \ - --report results/issue-482-audit.md -``` - -The seven-tool qualification passed all 21 scenarios. Discovery retained -93.1% top-1 accuracy, 100% positive recall, and 100% recall at the default -page. Fixed definitions measured 1,587 tokens, down 38 tokens from the -1,625-token baseline recorded for this removal. The audit's direct-call -fixture serialized to 52,396 bytes, so its successful truncation and paging -exercise a legitimate value above the 40 KB gate. - -The complete generated audit report is -[`issue-482-audit.md`](./issue-482-audit.md). Its JSON sibling is an ignored -run artifact. - -## Fresh-agent paging case - -Command: - -```sh -npm --prefix eval/current-version run perf:agent -- \ - --case large-document-paging \ - --repetitions 3 \ - --concurrency 1 \ - --output results/issue-482-large-document.json -``` - -The run used `codex-cli 0.149.1`, its default model, and `o200k_base`. All -three fresh sessions returned the exact final marker through -`call_tool` then `get_result`. Each used two Connecta round trips, made no -foreign or unexpected call, and passed correctness, safety, route, context, -and cost checks. MCP result use was 722, 742, and 752 tokens. The JSON artifact -contains the complete traces and remains ignored regeneration output. - -The broader routing lane recorded the candidate against an untouched-main -control: - -```sh -npm --prefix eval/current-version run perf:agent -- \ - --case routing \ - --repetitions 5 \ - --concurrency 5 \ - --output results/issue-482-routing.json -``` - -The control ran from detached, clean `9440ce5` with the same Node 26.7.0, -`codex-cli 0.149.1`, default model, tokenizer, five repetitions, and concurrency -five. The candidate recorded 25/30 routes, or 83.3%, versus untouched main at -20/30, or 66.7%. Per-case route passes were: - -| case | untouched main | candidate | -| --- | ---: | ---: | -| `single-read` | 4/5 | 4/5 | -| `dependent-read` | 1/5 | 1/5 | -| `dependent-reduction` | 4/5 | 5/5 | -| `multi-operation-discovery` | 4/5 | 5/5 | -| `ambiguous-candidate` | 4/5 | 5/5 | -| `nonstandard-collection-root` | 3/5 | 5/5 | - -Correct and safe sessions moved from 27/30 to 28/30. Both arms kept all 30 -sessions on the seven-tool surface with no agent-chosen foreign call. The -candidate's five misses were one `single-read` session and four -`dependent-read` sessions. No case regressed: `single-read` and -`dependent-read` tied the control, while the other four cases recorded one or -two more passes. Independent model samples and differing harness fingerprints -prevent a causal improvement claim. The candidate harness also contains the new -large-document case and fixture, although the selected six routing cases and -scoring code are unchanged. These results are an observed paired -non-regression, not a 95% pass. The original 95% absolute gate was underpowered, -and its replacement has moved to -[#496](https://github.com/zackbart/connecta/issues/496). Direct field projection -is not on either missed route, and this PR does not tune those workflows from -individual traces. Both ignored JSON artifacts are preserved in their -respective worktrees. - -## Repository checks - -The focused Node run passed 148 tests across `meta-tools-call`, -`code-first-surface`, and `server`. `npm run check` then passed all 114 suites: -2,681 tests passed and 41 were skipped. Before the two generated evidence -reports, the candidate removes 1,326 net lines across production code, tests, -documentation, and the added evaluation lane. diff --git a/eval/current-version/results/latest-main-agent-lookup.md b/eval/current-version/results/latest-main-agent-lookup.md deleted file mode 100644 index 83187007..00000000 --- a/eval/current-version/results/latest-main-agent-lookup.md +++ /dev/null @@ -1,97 +0,0 @@ -# Latest-main agent lookup benchmark - -> Historical evidence from the source commit below. Despite the legacy -> filename, this is not the current benchmark. See -> [`issue-322-evidence.md`](./issue-322-evidence.md). - -Generated: 2026-07-29T17:37:10.414Z - -Source: `cd20638bf36fc6808fddebe792cfe5e7e03ae49a`; codex-cli 0.145.0; model gpt-5.6-sol - -Each run used a fresh isolated server and ephemeral agent. Host apps, plugins, -browser, computer-use, multi-agent, and related discovery features were -explicitly disabled in addition to ignoring user config. Accuracy requires the -agent to execute exactly the expected downstream address set and return the -matching synthetic routing result. This is a routing canary, not validation of -real connector arguments or task semantics. The noise-token figure is the actual -serialized search result minus the same MCP envelope reconstructed with only -the expected candidate rows. - -## Summary - -- Exact tool-address accuracy: 27/30 -- Routing-result agreement: 27/30 -- Intended Connecta route: 12/30 -- Clean route (no foreign tool or host actions): 12/30 -- Direct retrieval top-1 accuracy: 0.44 -- Mean direct retrieval recall: 1 -- Mean direct retrieval MRR: 0.608 -- Direct negative clean rate: 0 -- Search-result tokens: 27,794 -- Estimated irrelevant lookup tokens: 22,080 -- Connecta MCP result tokens: 32,169 -- Foreign MCP result tokens: 0 -- All MCP result tokens: 32,169 -- Whole-agent input tokens: 2,365,456 (471,568 non-cached) - -## By case - -| Case | Address accuracy | Routing result | Connecta route | Clean route | Direct top-1 | Direct recall | Direct MRR | Search precision | Irrelevant candidates | Lookup attempts | Unknown connector filter | Est. noise tokens | Connecta MCP tokens | Foreign MCP tokens | Whole-agent input tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| open-issues-clean | 100% | 100% | 80% | 80% | 1 | 1 | 1 | 0.087 | 11 | 1 | 0% | 1123.2 | 1424 | 0 | 64623 | -| page-search-clean | 100% | 100% | 60% | 60% | 0 | 1 | 0.24 | 0.148 | 6.4 | 1 | 0% | 690.4 | 998.2 | 0 | 68668.6 | -| page-search-pressure | 100% | 100% | 0% | 0% | 0.2 | 1 | 0.467 | 0.17 | 5.2 | 1 | 0% | 568.6 | 938.8 | 0 | 101788.4 | -| workflow-by-id-clean | 100% | 100% | 0% | 0% | 1 | 1 | 1 | 0.091 | 10.6 | 1 | 0% | 1183.2 | 1558.2 | 0 | 75402.8 | -| build-diagnosis-clean | 40% | 40% | 20% | 20% | 0 | 1 | 0.334 | 0.329 | 7.4 | 1 | 0% | 734.6 | 1316.8 | 0 | 75071.8 | -| unsupported-audio-pressure | 100% | 100% | 80% | 80% | — | — | — | 0 | 1 | 1.2 | 0% | 116 | 197.8 | 0 | 87536.6 | - -## Runs - -| Run | Address | Routing result | Connecta route | Clean route | Direct top-1 | Direct recall | Direct MRR | Search precision | Irrelevant candidates | Lookup attempts | Unknown connector filters | Est. noise tokens | Connecta MCP tokens | Foreign MCP tokens | Agent input tokens | Prompt tokens | Tool route | -| --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | -| open-issues-clean #1 | yes | yes | yes | yes | true | 1 | 1 | 0.067 | 14 | 1 | 0 | 1407 | 1680 | 0 | 61690 | 20 | `connecta.search_tools → connecta.call_tool` | -| page-search-clean #1 | yes | yes | NO | NO | false | 1 | 0.2 | 0.091 | 10 | 1 | 0 | 1134 | 1518 | 0 | 79172 | 23 | `connecta.search_tools → connecta.describe_tools → connecta.call_tool` | -| page-search-pressure #1 | yes | yes | NO | NO | false | 1 | 0.5 | 0.125 | 7 | 1 | 0 | 717 | 1078 | 0 | 101941 | 4738 | `connecta.search_tools → connecta.describe_tools → connecta.call_tool` | -| workflow-by-id-clean #1 | yes | yes | NO | NO | true | 1 | 1 | 0.056 | 17 | 1 | 0 | 1968 | 2372 | 0 | 79756 | 25 | `connecta.search_tools → connecta.describe_tools → connecta.execute_code` | -| build-diagnosis-clean #1 | yes | yes | yes | yes | false | 1 | 0.292 | 0.5 | 2 | 1 | 0 | 190 | 573 | 0 | 61335 | 27 | `connecta.search_tools → connecta.execute_code` | -| unsupported-audio-pressure #1 | yes | yes | yes | yes | — | — | — | 0 | 1 | 1 | 0 | 116 | 186 | 0 | 82219 | 4744 | `connecta.search_tools` | -| open-issues-clean #2 | yes | yes | yes | yes | true | 1 | 1 | 0.1 | 9 | 1 | 0 | 934 | 1207 | 0 | 61210 | 20 | `connecta.search_tools → connecta.call_tool` | -| page-search-clean #2 | yes | yes | NO | NO | false | 1 | 0.25 | 0.125 | 7 | 1 | 0 | 701 | 1085 | 0 | 78403 | 23 | `connecta.search_tools → connecta.describe_tools → connecta.call_tool` | -| page-search-pressure #2 | yes | yes | NO | NO | false | 1 | 0.25 | 0.2 | 4 | 1 | 0 | 458 | 819 | 0 | 101712 | 4738 | `connecta.search_tools → connecta.describe_tools → connecta.call_tool` | -| workflow-by-id-clean #2 | yes | yes | NO | NO | true | 1 | 1 | 0.1 | 9 | 1 | 0 | 948 | 1352 | 0 | 78368 | 25 | `connecta.search_tools → connecta.describe_tools → connecta.execute_code` | -| build-diagnosis-clean #2 | yes | yes | NO | NO | false | 1 | 0.417 | 0.143 | 12 | 1 | 0 | 1216 | 2106 | 0 | 79220 | 27 | `connecta.skills → connecta.search_tools → connecta.batch_call` | -| unsupported-audio-pressure #2 | yes | yes | yes | yes | — | — | — | 0 | 1 | 1 | 0 | 116 | 186 | 0 | 82475 | 4744 | `connecta.search_tools` | -| open-issues-clean #3 | yes | yes | NO | NO | true | 1 | 1 | 0.067 | 14 | 1 | 0 | 1407 | 1819 | 0 | 78011 | 20 | `connecta.search_tools → connecta.describe_tools → connecta.call_tool` | -| page-search-clean #3 | yes | yes | yes | yes | false | 1 | 0.25 | 0.2 | 4 | 1 | 0 | 458 | 715 | 0 | 61935 | 23 | `connecta.search_tools → connecta.call_tool` | -| page-search-pressure #3 | yes | yes | NO | NO | true | 1 | 1 | 0.2 | 4 | 1 | 0 | 458 | 842 | 0 | 101738 | 4738 | `connecta.search_tools → connecta.describe_tools → connecta.call_tool` | -| workflow-by-id-clean #3 | yes | yes | NO | NO | true | 1 | 1 | 0.1 | 9 | 1 | 0 | 1104 | 1363 | 0 | 62064 | 25 | `connecta.search_tools → connecta.execute_code` | -| build-diagnosis-clean #3 | NO | NO | NO | NO | false | 1 | 0.292 | 0.5 | 2 | 1 | 0 | 190 | 736 | 0 | 77412 | 27 | `connecta.search_tools → connecta.execute_code → connecta.batch_call` | -| unsupported-audio-pressure #3 | yes | yes | NO | NO | — | — | — | 0 | 1 | 2 | 0 | 116 | 245 | 0 | 104266 | 4744 | `connecta.search_tools → connecta.search_tools` | -| open-issues-clean #4 | yes | yes | yes | yes | true | 1 | 1 | 0.1 | 9 | 1 | 0 | 934 | 1207 | 0 | 61152 | 20 | `connecta.search_tools → connecta.call_tool` | -| page-search-clean #4 | yes | yes | yes | yes | false | 1 | 0.25 | 0.125 | 7 | 1 | 0 | 701 | 958 | 0 | 62045 | 23 | `connecta.search_tools → connecta.call_tool` | -| page-search-pressure #4 | yes | yes | NO | NO | false | 1 | 0.333 | 0.125 | 7 | 1 | 0 | 752 | 1113 | 0 | 102040 | 4738 | `connecta.search_tools → connecta.describe_tools → connecta.call_tool` | -| workflow-by-id-clean #4 | yes | yes | NO | NO | true | 1 | 1 | 0.1 | 9 | 1 | 0 | 948 | 1352 | 0 | 78416 | 25 | `connecta.search_tools → connecta.describe_tools → connecta.execute_code` | -| build-diagnosis-clean #4 | NO | NO | NO | NO | false | 1 | 0.375 | 0.1 | 18 | 1 | 0 | 1775 | 2321 | 0 | 79838 | 27 | `connecta.search_tools → connecta.execute_code → connecta.batch_call` | -| unsupported-audio-pressure #4 | yes | yes | yes | yes | — | — | — | 0 | 1 | 1 | 0 | 116 | 186 | 0 | 86387 | 4744 | `connecta.search_tools` | -| open-issues-clean #5 | yes | yes | yes | yes | true | 1 | 1 | 0.1 | 9 | 1 | 0 | 934 | 1207 | 0 | 61052 | 20 | `connecta.search_tools → connecta.call_tool` | -| page-search-clean #5 | yes | yes | yes | yes | false | 1 | 0.25 | 0.2 | 4 | 1 | 0 | 458 | 715 | 0 | 61788 | 23 | `connecta.search_tools → connecta.call_tool` | -| page-search-pressure #5 | yes | yes | NO | NO | false | 1 | 0.25 | 0.2 | 4 | 1 | 0 | 458 | 842 | 0 | 101511 | 4738 | `connecta.search_tools → connecta.describe_tools → connecta.call_tool` | -| workflow-by-id-clean #5 | yes | yes | NO | NO | true | 1 | 1 | 0.1 | 9 | 1 | 0 | 948 | 1352 | 0 | 78410 | 25 | `connecta.search_tools → connecta.describe_tools → connecta.execute_code` | -| build-diagnosis-clean #5 | NO | NO | NO | NO | false | 1 | 0.292 | 0.4 | 3 | 1 | 0 | 302 | 848 | 0 | 77554 | 27 | `connecta.search_tools → connecta.execute_code → connecta.batch_call` | -| unsupported-audio-pressure #5 | yes | yes | yes | yes | — | — | — | 0 | 1 | 1 | 0 | 116 | 186 | 0 | 82336 | 4744 | `connecta.search_tools` | - -## Interpretation - -- Direct retrieval metrics measure only outer search_tools calls; searches - nested inside execute_code are intentionally not attributed without a server - trace. Search precision measures only the returned page. An accurate answer with low - precision means the agent reasoned through retrieval noise; it does not make - the lookup payload cheap. -- Whole-agent input tokens are Codex CLI accounting for the complete host - context, including built-in definitions and cache reads. MCP result tokens - isolate the observed Connecta payloads. -- Pressure cases contain 128 explicitly resolved distractor tasks and put the - current request at the end. They test instruction selection under long, - competing integration vocabulary; they are not a context-window limit test. -- Repetitions expose behavioral variance. This sample remains a canary, not a - statistical release gate. diff --git a/eval/current-version/results/latest-main-audit.md b/eval/current-version/results/latest-main-audit.md deleted file mode 100644 index 72675abf..00000000 --- a/eval/current-version/results/latest-main-audit.md +++ /dev/null @@ -1,45 +0,0 @@ -# Current-version Connecta audit - -> Historical evidence from the source commit below. Despite the legacy -> filename, this is not the current audit. See -> [`issue-322-current-audit.md`](./issue-322-current-audit.md). - -Source commit: `cd20638bf36fc6808fddebe792cfe5e7e03ae49a` - -Runtime: Node 26.5.0; tokenizer `o200k_base`; executor `enabled` - -Machine-readable results: `latest-main-audit.json` (run artifact, not committed) - -## Qualification - -- Release gate: pass -- Task scenarios: 21/21 passed (100.0%) -- Discovery top-1 accuracy: 89.7% -- Discovery positive recall: 100.0% -- Recall at the default page: 100.0% -- Negative-query false-positive rate: 20.0% -- Round trips: 55; summed call latency: 241.3 ms -- Connecta surface: 2,114 definition + 1,149 request + 16,428 response = **19,691 tokens** -- Result compatibility observed: `content` 55/55, `structuredContent` 52/55 -- `execute_code` advertised: yes -- Payload-free activity invariant: pass - - -## Discovery holdout - -The holdout contains 48 tools across 8 connectors and 34 independently authored queries. It is release qualification evidence and must not be used to tune ranking behavior. - -| Category | Queries | Top-1 | Recall | Precision | False positives | Mean results | Mean response tokens | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| direct | 8 | 100.0% | 100.0% | 1.000 | — | 1.00 | 168.0 | -| conversational | 8 | 75.0% | 100.0% | 0.588 | — | 2.63 | 303.0 | -| multi-intent | 4 | 75.0% | 100.0% | 0.259 | — | 7.75 | 783.5 | -| short-function-word | 4 | 100.0% | 100.0% | 0.567 | — | 4.25 | 466.0 | -| empty-after-cleanup | 1 | — | — | 0.000 | 100.0% | 8.00 | 967.0 | -| negative | 4 | — | — | 1.000 | 0.0% | 0.00 | 59.0 | -| connector-filtered | 3 | 100.0% | 100.0% | 1.000 | — | 1.33 | 184.7 | -| paginated | 2 | 100.0% | 100.0% | 1.000 | — | 4.00 | 454.5 | - -## Scope - -The audit exercises discovery, description, direct calls, batching, code-mode reduction, truncation and paging, destructive approval routing, OAuth recovery, static-credential operator recovery, unavailable recovery, and activity shape. Token counts cover the JSON-serialized MCP tool definitions, requests, and complete results observed by the SDK client; model deliberation and host-specific envelopes are outside this measurement. diff --git a/eval/current-version/results/latest-main-tool-lookup-investigation.md b/eval/current-version/results/latest-main-tool-lookup-investigation.md deleted file mode 100644 index 3c7b0ad5..00000000 --- a/eval/current-version/results/latest-main-tool-lookup-investigation.md +++ /dev/null @@ -1,224 +0,0 @@ -# Tool lookup accuracy and context investigation - -> Historical investigation from the source commit below. Despite the legacy -> filename, this is not current evidence. See -> [`issue-322-evidence.md`](./issue-322-evidence.md). - -Source under test: `cd20638bf36fc6808fddebe792cfe5e7e03ae49a` -(`origin/main` on 2026-07-29) - -Agent host: `codex-cli 0.145.0`, `gpt-5.6-sol`, Node 26.5.0 on -darwin-arm64. The model is pinned for comparison with candidate changes; the -CLI version and harness fingerprint are recorded with the raw result. - -Evidence: - -- [`latest-main-audit.md`](./latest-main-audit.md): deterministic release audit - and held-out lexical discovery. -- [`latest-main-agent-lookup.md`](./latest-main-agent-lookup.md): 30 fresh, - isolated agent runs (six cases, five repetitions). -- `latest-main-agent-single-read.json`: post-isolation smoke of the original - agent benchmark, in git history only — it never had a Markdown report. - -## Result - -Current main is usually accurate, but it buys that accuracy with a large noisy -candidate surface. - -| Measure | Result | -| --- | ---: | -| Held-out top-1 accuracy | 89.7% | -| Held-out positive recall | 100.0% | -| Held-out mean precision | 73.5% | -| Held-out multi-intent precision | 25.9% | -| Held-out negative false-positive rate | 20.0% | -| Agent exact address-set accuracy | 27/30 (90.0%) | -| Agent routing-result agreement | 27/30 (90.0%) | -| Agent intended Connecta route | 12/30 (40.0%) | -| Direct retrieval top-1 accuracy | 44.0% | -| Direct retrieval recall | 100.0% | -| Direct retrieval mean reciprocal rank | 60.8% | -| Estimated irrelevant share of agent search-result tokens | 79.4% | -| Estimated irrelevant share of all Connecta result tokens | 68.6% | -| Fixed Connecta definition surface | 2,114 tokens | - -The original benchmark's refreshed single-record smoke was answer-, route-, -and context-correct at 429 Connecta result tokens against a 500-token budget. -The repeated lookup lane still showed meaningful query and route variance, -which is why one run per task remains a canary rather than a stable gate. - -The long-context page-search pair stayed accurate in all ten clean/pressure -runs. The pressure prompt added about 4,715 prompt tokens and raised mean -whole-agent input accounting by 48.2%, from 68,669 to 101,788 tokens. Mean -Connecta result tokens fell 5.9%, and estimated lookup noise fell 17.6%. -This sample does not show an accuracy collapse from context pressure; it shows -that host context can grow sharply even when the observed Connecta result -surface does not. - -## Where the noise comes from - -### 1. Partial fallback has no meaningful relevance floor - -`CatalogService.search` first requires every normalized query term. Natural -agent queries almost always contain words absent from the short tool -description, so they fall through to `partial`. In partial mode, one substring -match is enough to return a tool: - -```text -score = matchedTerms * 1000 + matchedTermsInName -``` - -That preserves recall, but a common action word dominates the result set. -`list open project issues` returned up to 15 candidates because every -`list_*` tool matched. The expected tool stayed first, so the agent succeeded -while 79–88% of the serialized search result was estimated noise. - -### 2. Matching uses substrings, not lexical tokens - -`haystack.includes(term)` lets short function words match inside unrelated -words. The held-out `cleanup-only` case still retains terms such as `as`, -`at`, `be`, `by`, `is`, `or`, and `was`; those substrings returned eight -tools instead of none. A multi-word unsupported audio query also produced a -message-search candidate from the single word `text`. - -### 3. Common and discriminative terms have equal weight - -`list`, `search`, and `get` are important action distinctions, but they should -not carry the same retrieval weight as `invoice`, `workflow`, or `calendar`. -Current scoring counts term coverage and name matches only. It has no -document-frequency weighting, token-boundary weighting, or score margin. - -### 4. The agent cannot see why a partial result is weak - -The response exposes only a page-level `matchMode: "partial"`. It does not say -which terms matched or whether a candidate matched one of six terms versus -five of six. The agent often reasoned to the correct fourth-ranked tool, but it -had no compact confidence signal for rejecting weak rows. - -### 5. Extra round trips replay much more than Connecta's payload - -Twelve runs called `describe_tools` after already requesting compact schemas. A -redundant round trip costs more than its small MCP response because the host -re-enters the model with the whole active context. Connecta contributed 8,824 -result tokens across the earlier two-repetition sample. In the final -five-repetition sample, Connecta contributed 32,169 result tokens while Codex -reported 2,365,456 whole-agent input tokens, including 1,893,888 cache reads. - -Some redundant descriptions are an eval-fixture artifact: the 48 holdout tools -all advertise `{}` input schemas, even tools described as text searches. -Agents reasonably ask for more detail when that schema conflicts with the -task. Production conclusions about `describe_tools` require realistic schemas. - -## Accuracy failures and ambiguities - -- The workflow case now supplies run id 42, removing the earlier ambiguity - between listing the latest workflow run and getting a known run. All five - repetitions selected `builds.get_workflow_run`, although none used the - fixture's intended direct-call route. -- The three address failures were all in build diagnosis. The correct tools - were available in the direct search result (100% retrieval recall), but the - agent completed additional build tools and produced the wrong executed - address set. Only one of five build runs used the intended dependent - `execute_code` route. -- The page-search cases all found `documents.search_content`, including under - long distractor context, even when it ranked fourth. Accuracy survived the - noise; context efficiency did not. - -The fixture uses synthetic results and empty input schemas, so -`routingResultCorrect` is deliberately only a routing canary: it checks the -returned fixture address set, not real arguments, connector semantics, or -end-to-end task correctness. Direct retrieval metrics cover the outer -`search_tools` call only; searches nested inside `execute_code` cannot -currently be attributed. - -## Harness finding: host isolation was previously false - -`--ignore-user-config` alone does not remove Codex host apps and plugins. An -initial supposedly isolated run invoked live GitHub tools, consumed roughly -69,000 foreign MCP-result tokens, and ran for 229 seconds. The new lookup lane -explicitly disables apps, plugins, browser/computer use, multi-agent, skill -search, and related host features; it records each MCP server and separates -foreign from Connecta tokens. The older `perf:agent` lane now uses the same -feature isolation. - -This matters beyond benchmark hygiene: when two tool systems overlap, the host -may bypass Connecta entirely. Connecta ranking changes cannot fix host-level -tool competition. - -## Prescriptions - -### P0 — make the benchmark trustworthy - -1. Keep explicit host-feature isolation and per-server tool accounting. -2. Pin both model and Codex CLI for comparative runs. Run at least ten fresh - repetitions per case before treating a route-rate change as real. -3. Add realistic input/output schemas and deterministic dependent outputs to - the agent fixtures. Separate address selection, argument correctness, - execution route, and final-answer correctness. -4. Relabel ambiguous tasks with multiple acceptable plans, or make the intent - unambiguous (for example, a known workflow-run id). -5. Keep the existing discovery holdout sealed. Build a new development corpus - for ranking work; use the holdout only for final regression qualification. - -### P1 — reduce avoidable context without changing retrieval - -1. Tell agents to omit `limit` on the first search and paginate only when the - expected candidate is absent. Agents frequently requested 20 results even - though the default page is eight. -2. Add concise query guidance: use two to four discriminative action/domain - terms, not the entire task sentence or provider guesses. -3. Re-test `includeSchemas="compact"` versus a search-then-describe route on - realistic schemas. If agents still redescribe complete compact schemas, - make schema completeness explicit in the result rather than adding more - prose to the always-loaded instructions. -4. Treat the number of model round trips as a primary context metric alongside - serialized MCP tokens. - -### P2 — improve lexical ranking on a new development corpus - -Prototype token-based BM25/IDF-style scoring: - -- match whole normalized tokens before allowing substring fallback; -- retain action terms, but down-weight terms common across the catalog; -- weight rare domain terms and exact name phrases more strongly; -- require a defensible coverage or score margin before returning weak partial - matches; and -- cap partial-fallback pages unless the caller explicitly paginates. - -Do not ship a fixed coverage threshold from these 34 held-out queries. Short -queries and multi-intent recall are the failure modes. Select the rule on a new -development set, then require the sealed holdout to retain 100% positive -recall while improving precision. - -A tiny per-result matched-term or coverage signal may help agents reject weak -partial matches, but it also costs tokens. A/B it against simply returning -fewer, better-ranked rows. - -### P3 — change result shape only with host evidence - -The Codex JSON event contains both text and structured forms, so the serialized -envelope is visibly duplicated. That does not prove the model receives both -forms verbatim. Preserve the current compatibility contract until host -forwarding is measured. If evidence shows structured content is reliably -forwarded, revisit the already-gated summary-only text shape rather than -silently dropping compatibility. - -Semantic search remains gated by the project ethos. Try it only after the -lexical development lane shows that token-aware ranking cannot meet the -accuracy/context target. - -## Proposed gates - -Use these as experimental selection criteria, not immediate release promises: - -- held-out positive recall remains 100%; -- held-out top-1 does not regress from 89.7%; -- held-out negative false positives fall below 10%; -- development multi-intent mean precision exceeds 50%; -- repeated agent exact-address accuracy reaches at least 95%; -- strict minimal-route rate reaches at least 80%; -- estimated irrelevant search-token share falls below 40%; and -- pressure accuracy stays within five percentage points of clean accuracy. - -Any candidate that saves tokens by hiding the correct tool fails, regardless -of its average precision. diff --git a/eval/current-version/run-audit.mjs b/eval/current-version/run-audit.mjs deleted file mode 100644 index 372b4ccc..00000000 --- a/eval/current-version/run-audit.mjs +++ /dev/null @@ -1,297 +0,0 @@ -import { spawn, execFileSync } from "node:child_process"; -import { createHash } from "node:crypto"; -import { mkdir, readFile, writeFile } from "node:fs/promises"; -import { basename, dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -import { createAuditClient, round } from "./audit-lib.mjs"; -import { codeFirstTools } from "./agent-benchmark-scoring.mjs"; -import { runTaskAudit } from "./audit-all-tools.mjs"; -import { runDiscoveryBenchmark } from "./discovery-benchmark.mjs"; -import { renderReport } from "./report.mjs"; - -const here = dirname(fileURLToPath(import.meta.url)); -const root = resolve(here, "../.."); -const args = process.argv.slice(2); - -function option(name, fallback) { - const index = args.indexOf(name); - if (index < 0) return fallback; - const value = args[index + 1]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value.`); - } - return value; -} - -const sourceCommit = option( - "--source-commit", - execFileSync("git", ["rev-parse", "HEAD"], { - cwd: root, - encoding: "utf8", - }).trim(), -); -const outputPath = resolve( - here, - option("--output", "results/current-version.json"), -); -const reportPath = resolve( - here, - option( - "--report", - outputPath.endsWith(".json") - ? outputPath.slice(0, -5) + ".md" - : outputPath + ".md", - ), -); -const tokenizerName = - process.env.CONNECTA_EVAL_TOKENIZER ?? "o200k_base"; -const executorMode = "required"; -const surface = "seven-tool"; -const bearer = process.env.CONNECTA_EVAL_TOKEN ?? "connecta-eval-token"; -const operatorToken = - process.env.CONNECTA_EVAL_OPERATOR_TOKEN ?? "connecta-eval-operator"; -const corpusPath = resolve(here, "discovery-holdout.json"); -const corpusBytes = await readFile(corpusPath); - -function startServer() { - const child = spawn( - process.execPath, - ["--import", "tsx", "sandbox-server.ts"], - { - cwd: here, - env: { - ...process.env, - CONNECTA_EVAL_PORT: "0", - CONNECTA_EVAL_TOKEN: bearer, - CONNECTA_EVAL_OPERATOR_TOKEN: operatorToken, - CONNECTA_EVAL_SOURCE_COMMIT: sourceCommit, - }, - stdio: ["ignore", "pipe", "pipe"], - }, - ); - let stderr = ""; - child.stderr.setEncoding("utf8"); - child.stderr.on("data", (chunk) => { - stderr += chunk; - process.stderr.write(chunk); - }); - child.stdout.setEncoding("utf8"); - let buffered = ""; - const ready = new Promise((resolveReady, rejectReady) => { - const timeout = setTimeout(() => { - rejectReady(new Error(`Eval server readiness timed out.\n${stderr}`)); - }, 30_000); - child.once("error", (error) => { - clearTimeout(timeout); - rejectReady(error); - }); - child.once("exit", (code) => { - clearTimeout(timeout); - rejectReady( - new Error(`Eval server exited before readiness (${code}).\n${stderr}`), - ); - }); - child.stdout.on("data", (chunk) => { - buffered += chunk; - for (;;) { - const newline = buffered.indexOf("\n"); - if (newline < 0) break; - const line = buffered.slice(0, newline); - buffered = buffered.slice(newline + 1); - let message; - try { - message = JSON.parse(line); - } catch { - continue; - } - if (message.event === "ready") { - clearTimeout(timeout); - resolveReady(message); - } - } - }); - }); - return { child, ready }; -} - -async function stopServer(child) { - if (child.exitCode !== null) return; - child.kill("SIGTERM"); - await new Promise((resolveExit, rejectExit) => { - const timeout = setTimeout(() => { - child.kill("SIGKILL"); - rejectExit(new Error("Eval server did not stop within 10 seconds.")); - }, 10_000); - child.once("exit", () => { - clearTimeout(timeout); - resolveExit(); - }); - }); -} - -const server = startServer(); -let context; -try { - const ready = await server.ready; - context = await createAuditClient({ - url: ready.url, - token: bearer, - tokenizerName, - }); - const tasks = await runTaskAudit(context, { - baseUrl: ready.baseUrl, - operatorToken, - }); - const discovery = await runDiscoveryBenchmark(context, corpusPath); - const observations = context.observations; - const definitionTokens = context.connection.toolsListTokens; - const requestTokens = observations.reduce( - (sum, entry) => sum + entry.requestTokens, - 0, - ); - const responseTokens = observations.reduce( - (sum, entry) => sum + entry.responseTokens, - 0, - ); - const activityCase = tasks.cases.find( - (entry) => entry.outcome === "activity-payload-free", - ); - // The `surface` stamp below is a constant; without this check a server that - // regressed to the classic nine would still be filed as seven-tool evidence. - const advertisedTools = context.connection.tools - .map((tool) => tool.name) - .sort(); - const expectedTools = [...codeFirstTools].sort(); - const surfaceMatches = - advertisedTools.length === expectedTools.length && - advertisedTools.every((name, index) => name === expectedTools[index]); - const qualificationChecks = [ - { - name: "advertised surface is exactly the seven meta-tools", - actual: advertisedTools, - expected: expectedTools, - passed: surfaceMatches, - }, - { - name: "all behavioral scenarios pass", - actual: tasks.summary.taskSuccessRate, - minimum: 1, - passed: tasks.summary.taskSuccessRate === 1, - }, - { - name: "holdout top-1 accuracy does not regress", - actual: discovery.metrics.top1Accuracy, - minimum: 0.897, - passed: discovery.metrics.top1Accuracy >= 0.897, - }, - { - name: "holdout positive recall stays complete", - actual: discovery.metrics.positiveRecall, - minimum: 1, - passed: discovery.metrics.positiveRecall === 1, - }, - { - name: "default-page recall stays complete", - actual: discovery.metrics.recallAtDefaultPage, - minimum: 1, - passed: discovery.metrics.recallAtDefaultPage === 1, - }, - { - name: "activity storage stays payload-free", - actual: - activityCase?.passed === true && - Array.isArray(activityCase.forbiddenPresent) && - activityCase.forbiddenPresent.length === 0, - expected: true, - passed: - activityCase?.passed === true && - Array.isArray(activityCase.forbiddenPresent) && - activityCase.forbiddenPresent.length === 0, - }, - ]; - const audit = { - schemaVersion: 1, - generatedAt: new Date().toISOString(), - source: { - commit: sourceCommit, - nodeVersion: process.versions.node, - platform: `${process.platform}-${process.arch}`, - tokenizer: tokenizerName, - executorMode, - surface, - corpusSha256: createHash("sha256").update(corpusBytes).digest("hex"), - }, - connection: context.connection, - totals: { - definitionTokens, - requestTokens, - responseTokens, - measuredSurfaceTokens: - definitionTokens + requestTokens + responseTokens, - requestBytes: observations.reduce( - (sum, entry) => sum + entry.requestBytes, - 0, - ), - responseBytes: observations.reduce( - (sum, entry) => sum + entry.responseBytes, - 0, - ), - roundTrips: observations.length, - summedLatencyMs: round( - observations.reduce((sum, entry) => sum + entry.latencyMs, 0), - ), - }, - compatibility: { - client: "@modelcontextprotocol/client StreamableHTTPClientTransport", - protocolMode: "stateless streamable HTTP", - resultCount: observations.length, - contentResults: observations.filter((entry) => entry.hasContent).length, - structuredContentResults: observations.filter( - (entry) => entry.hasStructuredContent, - ).length, - executeCodeAdvertised: context.listed.tools.some( - (tool) => tool.name === "execute_code", - ), - }, - invariants: { - activityPayloadFree: - activityCase?.passed === true && - Array.isArray(activityCase.forbiddenPresent) && - activityCase.forbiddenPresent.length === 0, - activityKeys: activityCase?.activityKeys ?? [], - }, - qualification: { - passed: qualificationChecks.every((check) => check.passed), - checks: qualificationChecks, - }, - tasks, - discovery, - }; - await mkdir(dirname(outputPath), { recursive: true }); - await mkdir(dirname(reportPath), { recursive: true }); - await writeFile(outputPath, `${JSON.stringify(audit, null, 2)}\n`); - await writeFile( - reportPath, - renderReport(audit, basename(outputPath)), - ); - process.stdout.write( - `${JSON.stringify({ - event: "audit_complete", - sourceCommit, - output: outputPath, - report: reportPath, - taskSuccessRate: audit.tasks.summary.taskSuccessRate, - discovery: audit.discovery.metrics, - totals: audit.totals, - executorMode, - surface, - })}\n`, - ); - if (!audit.qualification.passed) { - process.exitCode = 1; - } -} finally { - if (context) await context.close(); - await stopServer(server.child); -} diff --git a/eval/current-version/run-development-discovery.mjs b/eval/current-version/run-development-discovery.mjs deleted file mode 100644 index b6f60dc4..00000000 --- a/eval/current-version/run-development-discovery.mjs +++ /dev/null @@ -1,204 +0,0 @@ -import { spawn, execFileSync } from "node:child_process"; -import { createHash } from "node:crypto"; -import { mkdir, readFile, writeFile } from "node:fs/promises"; -import { basename, dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -import { createAuditClient } from "./audit-lib.mjs"; -import { runDiscoveryBenchmark } from "./discovery-benchmark.mjs"; - -const here = dirname(fileURLToPath(import.meta.url)); -const root = resolve(here, "../.."); -const args = process.argv.slice(2); - -function option(name, fallback) { - const index = args.indexOf(name); - if (index < 0) return fallback; - const value = args[index + 1]; - if (!value || value.startsWith("--")) { - throw new Error(`${name} requires a value.`); - } - return value; -} - -function sha256(value) { - return createHash("sha256").update(value).digest("hex"); -} - -const sourceCommit = option( - "--source-commit", - execFileSync("git", ["rev-parse", "HEAD"], { - cwd: root, - encoding: "utf8", - }).trim(), -); -const outputPath = resolve( - here, - option("--output", "results/issue-322-development-discovery.json"), -); -const reportPath = resolve( - here, - option("--report", "results/issue-322-development-discovery.md"), -); -const tokenizerName = process.env.CONNECTA_EVAL_TOKENIZER ?? "o200k_base"; -const corpusPath = resolve(here, "discovery-development.json"); -const corpusBytes = await readFile(corpusPath); -const bearer = "connecta-development-discovery-token"; - -function startServer() { - const child = spawn( - process.execPath, - ["--import", "tsx", "sandbox-server.ts"], - { - cwd: here, - env: { - ...process.env, - CONNECTA_EVAL_PORT: "0", - CONNECTA_EVAL_TOKEN: bearer, - CONNECTA_EVAL_SOURCE_COMMIT: sourceCommit, - CONNECTA_EVAL_DEVELOPMENT_CORPUS: "enabled", - CONNECTA_EVAL_DISCOVERY_ONLY: "enabled", - }, - stdio: ["ignore", "pipe", "pipe"], - }, - ); - let stderr = ""; - child.stderr.setEncoding("utf8"); - child.stderr.on("data", (chunk) => { - stderr += chunk; - }); - child.stdout.setEncoding("utf8"); - let buffered = ""; - const ready = new Promise((resolveReady, rejectReady) => { - const timeout = setTimeout(() => { - rejectReady(new Error(`Development server timed out.\n${stderr}`)); - }, 30_000); - child.once("error", (error) => { - clearTimeout(timeout); - rejectReady(error); - }); - child.once("exit", (code) => { - clearTimeout(timeout); - rejectReady( - new Error(`Development server exited before readiness (${code}).\n${stderr}`), - ); - }); - child.stdout.on("data", (chunk) => { - buffered += chunk; - for (;;) { - const newline = buffered.indexOf("\n"); - if (newline < 0) break; - const line = buffered.slice(0, newline); - buffered = buffered.slice(newline + 1); - try { - const message = JSON.parse(line); - if (message.event === "ready") { - clearTimeout(timeout); - resolveReady(message); - } - } catch { - // Ignore non-protocol server output. - } - } - }); - }); - return { child, ready }; -} - -async function stopServer(child) { - if (child.exitCode !== null) return; - child.kill("SIGTERM"); - await new Promise((resolveExit) => { - const timeout = setTimeout(() => { - child.kill("SIGKILL"); - resolveExit(); - }, 10_000); - child.once("exit", () => { - clearTimeout(timeout); - resolveExit(); - }); - }); -} - -const server = startServer(); -let context; -try { - const ready = await server.ready; - context = await createAuditClient({ - url: ready.url, - token: bearer, - tokenizerName, - }); - const discovery = await runDiscoveryBenchmark(context, corpusPath); - const qualification = { - passed: - discovery.cases.every((entry) => entry.passed) && - discovery.metrics.expectedTopAccuracy === 1 && - discovery.metrics.positiveRecall === 1 && - discovery.cases.every( - (entry) => - entry.queryCoverageRows === 0 && - entry.queryCoverageTokens === 0, - ), - }; - const result = { - schemaVersion: 1, - generatedAt: new Date().toISOString(), - source: { - commit: sourceCommit, - nodeVersion: process.versions.node, - platform: `${process.platform}-${process.arch}`, - tokenizer: tokenizerName, - corpusSha256: sha256(corpusBytes), - harnessSha256: sha256( - await readFile(fileURLToPath(import.meta.url)), - ), - sandboxSha256: sha256( - await readFile(resolve(here, "sandbox-server.ts")), - ), - hostIsolation: - "Loopback-only sandbox with only the development connector configured; deterministic provider handlers; no model, Codex CLI, external account, host app, or plugin.", - }, - qualification, - connection: context.connection, - discovery, - }; - const metrics = discovery.metrics; - const report = `# Issue #322 development discovery evidence - -Source commit: \`${sourceCommit}\` - -Runtime: Node ${result.source.nodeVersion} on ${result.source.platform}; tokenizer \`${tokenizerName}\` - -Machine-readable results: \`${basename(outputPath)}\` (run artifact, not committed) - -## Result - -- Development gate: ${qualification.passed ? "pass" : "FAIL"} -- Expected top-1 accuracy: ${(metrics.expectedTopAccuracy * 100).toFixed(1)}% -- Positive recall: ${(metrics.positiveRecall * 100).toFixed(1)}% -- Mean precision: ${(metrics.meanPrecision * 100).toFixed(1)}% -- Serialized query-coverage rows: ${discovery.cases.reduce((sum, entry) => sum + entry.queryCoverageRows, 0)} -- Serialized query-coverage bytes/tokens: ${metrics.totalQueryCoverageBytes}/${metrics.totalQueryCoverageTokens} - -The development corpus is separate from the sealed release holdout. The server exposes only its synthetic analytics connector on loopback. It does not call a model, the Codex CLI, a host app, a plugin, or an external account. -`; - await mkdir(dirname(outputPath), { recursive: true }); - await mkdir(dirname(reportPath), { recursive: true }); - await writeFile(outputPath, `${JSON.stringify(result, null, 2)}\n`); - await writeFile(reportPath, report); - process.stdout.write( - `${JSON.stringify({ - event: "development_discovery_complete", - output: outputPath, - report: reportPath, - sourceCommit, - qualification, - metrics, - })}\n`, - ); - if (!qualification.passed) process.exitCode = 1; -} finally { - if (context) await context.close(); - await stopServer(server.child); -} diff --git a/eval/current-version/sandbox-server.ts b/eval/current-version/sandbox-server.ts deleted file mode 100644 index a704fc82..00000000 --- a/eval/current-version/sandbox-server.ts +++ /dev/null @@ -1,1280 +0,0 @@ -import { once } from "node:events"; -import { readFile } from "node:fs/promises"; -import { fileURLToPath } from "node:url"; - -import { - ConnectorCallError, - api, - bearerToken, - createConnecta, - memoryStorage, - type ApiTool, - type Connector, - type InboundAuth, - type ToolCallActivityEvent, - type ToolDef, -} from "../../src/index.js"; -import { quickJsExecutor } from "../../src/executors/quickjs.js"; -import { listen } from "../../src/node.js"; -import { createEvalTracing } from "./eval-tracing.js"; - -interface HoldoutCorpus { - connectors: { - id: string; - description: string; - tools: { name: string; description: string }[]; - }[]; -} - -const developmentCorpusEnabled = - process.env.CONNECTA_EVAL_DEVELOPMENT_CORPUS === "enabled"; -const discoveryOnly = - process.env.CONNECTA_EVAL_DISCOVERY_ONLY === "enabled"; -const discoveryCorpus = JSON.parse( - await readFile( - fileURLToPath( - new URL( - developmentCorpusEnabled - ? "./discovery-development.json" - : "./discovery-holdout.json", - import.meta.url, - ), - ), - "utf8", - ), -) as HoldoutCorpus; - -const token = process.env.CONNECTA_EVAL_TOKEN ?? "connecta-eval-token"; -const operatorToken = - process.env.CONNECTA_EVAL_OPERATOR_TOKEN ?? "connecta-eval-operator"; -const sourceCommit = process.env.CONNECTA_EVAL_SOURCE_COMMIT ?? "working-tree"; -const traceEnabled = process.env.CONNECTA_EVAL_TRACE === "enabled"; -const port = Number(process.env.CONNECTA_EVAL_PORT ?? "0"); -const host = "127.0.0.1"; -const credentialEncryptionKey = Buffer.alloc(32, 7).toString("base64"); -const storage = memoryStorage(); -const activityEvents: ToolCallActivityEvent[] = []; -const tracing = createEvalTracing({ enabled: traceEnabled, token }); -const { emitTrace } = tracing; - -const operatorAuth: InboundAuth = { - kind: "clerk", - uiAuth: { - kind: "clerk", - publishableKey: "pk_test_eval", - frontendApiUrl: "https://clerk.eval.invalid", - }, - authorize(request) { - if ( - request.headers.get("authorization") === `Bearer ${operatorToken}` - ) { - return { ok: true, userId: "isolated-eval-operator" }; - } - return { - ok: false, - response: Response.json({ error: "unauthorized" }, { status: 401 }), - }; - }, -}; - -const objectOutput = { - type: "object", - additionalProperties: true, -} as const; - -function genericFixtureContract( - connectorId: string, - name: string, -): Pick { - if (name.startsWith("list_")) { - return { - inputSchema: { - type: "object", - properties: { - limit: { - type: "integer", - minimum: 1, - maximum: 100, - default: 25, - }, - }, - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - items: { type: "array", items: objectOutput }, - nextCursor: { type: "string" }, - }, - required: ["items"], - additionalProperties: false, - }, - handler: (args: { limit?: number }) => ({ - items: [ - { - id: `${connectorId}-${name}-1`, - label: `Deterministic ${name.replaceAll("_", " ")} fixture`, - }, - ].slice(0, args.limit ?? 25), - }), - }; - } - if (name.startsWith("search_")) { - return { - inputSchema: { - type: "object", - properties: { - query: { type: "string", minLength: 1 }, - limit: { - type: "integer", - minimum: 1, - maximum: 100, - default: 25, - }, - }, - required: ["query"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - query: { type: "string" }, - results: { type: "array", items: objectOutput }, - }, - required: ["query", "results"], - additionalProperties: false, - }, - handler: (args: { query: string }) => ({ - query: args.query, - results: [ - { - id: `${connectorId}-${name}-1`, - label: `Result for ${args.query}`, - }, - ], - }), - }; - } - if (name.startsWith("create_")) { - return { - inputSchema: { - type: "object", - properties: { - title: { type: "string", minLength: 1 }, - }, - required: ["title"], - additionalProperties: false, - }, - outputSchema: objectOutput, - handler: (args: { title: string }) => ({ - id: `${connectorId}-${name}-created`, - title: args.title, - created: true, - }), - }; - } - if (name.startsWith("update_")) { - return { - inputSchema: { - type: "object", - properties: { - id: { type: "string", minLength: 1 }, - fields: { type: "object", additionalProperties: true }, - }, - required: ["id", "fields"], - additionalProperties: false, - }, - outputSchema: objectOutput, - handler: (args: { id: string; fields: Record }) => ({ - id: args.id, - fields: args.fields, - updated: true, - }), - }; - } - return { - inputSchema: { - type: "object", - properties: { - id: { type: "string", minLength: 1 }, - }, - required: ["id"], - additionalProperties: false, - }, - outputSchema: objectOutput, - handler: (args: { id: string }) => ({ - id: args.id, - connector: connectorId, - operation: name, - }), - }; -} - -function agentFixtureContract( - connectorId: string, - name: string, -): Pick | undefined { - const address = `${connectorId}.${name}`; - if ( - address === "analytics.List-Organizations" || - address === "analytics.List-All-Organizations" - ) { - return { - inputSchema: { - type: "object", - properties: { - projectId: { type: "string", minLength: 1 }, - }, - required: ["projectId"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - projectId: { type: "string" }, - organizations: { - type: "array", - items: { - type: "object", - properties: { - id: { type: "string" }, - name: { type: "string" }, - }, - required: ["id", "name"], - additionalProperties: false, - }, - }, - }, - required: ["projectId", "organizations"], - additionalProperties: false, - }, - handler: (args: { projectId: string }) => ({ - projectId: args.projectId, - organizations: [ - { id: "org_eval_7", name: "Evaluation Organization" }, - ], - }), - }; - } - if (address === "projects.list_issues") { - return { - inputSchema: { - type: "object", - properties: { - state: { type: "string", enum: ["open", "closed"] }, - label: { type: "string", minLength: 1 }, - }, - required: ["state"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - state: { type: "string", enum: ["open", "closed"] }, - issues: { - type: "array", - items: { - type: "object", - properties: { - number: { type: "integer" }, - title: { type: "string" }, - state: { type: "string", enum: ["open", "closed"] }, - }, - required: ["number", "title", "state"], - additionalProperties: false, - }, - }, - }, - required: ["state", "issues"], - additionalProperties: false, - }, - handler: (args: { state: "open" | "closed"; label?: string }) => ({ - state: args.state, - issues: - args.state === "open" - ? [ - { - number: 213, - title: "Measure agent routing overhead", - state: "open", - }, - { - number: 214, - title: "Document benchmark protocol", - state: "open", - }, - ] - : [ - { - number: 212, - title: "Improve tool lookup ranking", - state: "closed", - }, - ], - }), - }; - } - if (address === "documents.search_content") { - return { - inputSchema: { - type: "object", - properties: { - query: { type: "string", minLength: 1 }, - }, - required: ["query"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - query: { type: "string" }, - results: { - type: "array", - items: { - type: "object", - properties: { - id: { type: "string" }, - title: { type: "string" }, - snippet: { type: "string" }, - }, - required: ["id", "title", "snippet"], - additionalProperties: false, - }, - }, - }, - required: ["query", "results"], - additionalProperties: false, - }, - handler: (args: { query: string }) => ({ - query: args.query, - results: [ - { - id: "page-launch-plan", - title: "Launch plan", - snippet: "The launch plan begins with a staged customer rollout.", - }, - ], - }), - }; - } - if (address === "builds.get_workflow_run") { - return { - inputSchema: { - type: "object", - properties: { - runId: { type: "integer", minimum: 1 }, - }, - required: ["runId"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - runId: { type: "integer" }, - status: { type: "string" }, - conclusion: { type: "string" }, - failedJobId: { type: "integer" }, - }, - required: ["runId", "status", "conclusion", "failedJobId"], - additionalProperties: false, - }, - handler: (args: { runId: number }) => ({ - runId: args.runId, - status: "completed", - conclusion: "failure", - failedJobId: args.runId * 100 + 7, - }), - }; - } - if (address === "builds.get_job_logs") { - return { - inputSchema: { - type: "object", - properties: { - jobId: { type: "integer", minimum: 1 }, - }, - required: ["jobId"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - jobId: { type: "integer" }, - runId: { type: "integer" }, - lines: { type: "array", items: { type: "string" } }, - }, - required: ["jobId", "runId", "lines"], - additionalProperties: false, - }, - handler: (args: { jobId: number }) => ({ - jobId: args.jobId, - runId: Math.trunc(args.jobId / 100), - lines: [ - "test: expected 2 received 3", - "process exited with status 1", - ], - }), - }; - } - return undefined; -} - -function fixtureTools( - connectorId: string, - definitions: { name: string; description: string }[], -): ApiTool[] { - return definitions.map((definition) => ({ - ...definition, - ...(agentFixtureContract(connectorId, definition.name) ?? - genericFixtureContract(connectorId, definition.name)), - annotations: { readOnlyHint: true, idempotentHint: true }, - })); -} - -const discoveryConnectors: Connector[] = discoveryCorpus.connectors.map( - (fixture) => - api(fixture.id, { - description: fixture.description, - tools: fixtureTools(fixture.id, fixture.tools), - }), -); - -const guidedWorkItems = api("work-items", { - title: "Work Items", - description: - "Issue search with a provider query language and team-scoped collections", - usageGuide: `# Work item search - -Use \`search_issues\` for filtered issue lists. Its \`query\` uses the provider's -query language, not natural language. Put field names on the left, quote status -values, join clauses with uppercase \`AND\`, and use the stable team key rather -than the display name. - -Example: \`team = ENG AND status = "In Progress"\`. -`, - tools: [ - { - name: "search_issues", - description: - "Search work items with the provider query language. Read the connector guide before composing a query.", - inputSchema: { - type: "object", - properties: { - query: { type: "string", minLength: 1 }, - first: { - type: "integer", - minimum: 1, - maximum: 50, - default: 25, - }, - }, - required: ["query"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - nodes: { type: "array", items: objectOutput }, - pageInfo: objectOutput, - }, - required: ["nodes", "pageInfo"], - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { query: string }) => ({ - nodes: - args.query === 'team = ENG AND status = "In Progress"' - ? [ - { - identifier: "ENG-294", - title: "Reduce cold-agent connector learning turns", - status: "In Progress", - team: "ENG", - }, - ] - : [], - pageInfo: { hasNextPage: false, endCursor: null }, - }), - }, - ], -}); - -const bookshelf = api("bookshelf", { - title: "Bookshelf", - description: "Point reads and paginated browsing for a deterministic library", - usageGuide: `# Bookshelf list pagination - -Only \`list_books\` uses cursor pagination. A \`get_book\` point lookup takes -the stable book id directly and needs no pagination convention. -`, - tools: [ - { - name: "get_book", - description: "Get one book by its stable id.", - inputSchema: { - type: "object", - properties: { id: { type: "string", minLength: 1 } }, - required: ["id"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - id: { type: "string" }, - title: { type: "string" }, - available: { type: "boolean" }, - }, - required: ["id", "title", "available"], - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { id: string }) => ({ - id: args.id, - title: "The Selective Guide", - available: true, - }), - }, - ], -}); - -const dnsRecordFilterProperties = { - recordType: { - type: "string", - enum: ["A", "AAAA", "CNAME", "MX", "TXT"], - }, - name: { type: "string" }, - content: { type: "string" }, - proxied: { type: "boolean" }, - comment: { type: "string" }, - commentPresent: { type: "boolean" }, - tag: { type: "string" }, - tagPresent: { type: "boolean" }, - page: { type: "integer", minimum: 1 }, - perPage: { type: "integer", minimum: 1, maximum: 100 }, - order: { type: "string", enum: ["type", "name", "content", "ttl"] }, - direction: { type: "string", enum: ["asc", "desc"] }, - match: { type: "string", enum: ["all", "any"] }, - search: { type: "string" }, - since: { type: "string", format: "date-time" }, - before: { type: "string", format: "date-time" }, - zoneName: { type: "string" }, - recordId: { type: "string" }, - minimumTtl: { type: "integer", minimum: 1 }, - maximumTtl: { type: "integer", minimum: 1 }, - exportFormat: { type: "string", enum: ["json", "bind"] }, - includeDisabled: { type: "boolean" }, - includeMetadata: { type: "boolean" }, - flattenSettings: { type: "boolean" }, - exactNameMatch: { type: "boolean" }, - includeRecordSettings: { type: "boolean" }, - includeZoneSettings: { type: "boolean" }, - includePermissionGroups: { type: "boolean" }, - includeActivationStatus: { type: "boolean" }, - includeVerificationState: { type: "boolean" }, - includeDnssecState: { type: "boolean" }, - includeNameserverState: { type: "boolean" }, - includeRegistrarState: { type: "boolean" }, - includeDevelopmentMode: { type: "boolean" }, - includePlanMetadata: { type: "boolean" }, - includeAccountMetadata: { type: "boolean" }, - includeModifiedTimestamps: { type: "boolean" }, - includeCreatedTimestamps: { type: "boolean" }, - includeLockedRecordState: { type: "boolean" }, - includeProviderSpecificMetadata: { type: "boolean" }, -} as const; - -const edgeDns = api("edge-dns", { - title: "Edge DNS", - description: - "Account-scoped zones and DNS records with nested SDK-style arguments", - usageGuide: `# Edge DNS usage - -Use nested SDK argument objects, not flat ids. Resolve the account's zone with -\`list_zones({ account: { id } })\`, then pass the returned zone id as -\`list_dns_records({ zone: { id }, filter: { recordType } })\`. -`, - tools: [ - { - name: "list_zones", - description: "List DNS zones owned by one account.", - inputSchema: { - type: "object", - properties: { - account: { - type: "object", - properties: { - id: { type: "string", minLength: 1 }, - }, - required: ["id"], - additionalProperties: false, - }, - pagination: { - type: "object", - properties: { - perPage: { type: "integer", minimum: 1, maximum: 50 }, - }, - additionalProperties: false, - }, - }, - required: ["account"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - result: { type: "array", items: objectOutput }, - resultInfo: objectOutput, - }, - required: ["result", "resultInfo"], - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { account: { id: string } }) => ({ - result: - args.account.id === "acct_eval_7" - ? [{ id: "zone_eval_42", name: "example.test", status: "active" }] - : [], - resultInfo: { page: 1, totalPages: 1 }, - }), - }, - { - name: "list_dns_records", - description: "List DNS records inside one zone with optional filters.", - inputSchema: { - type: "object", - properties: { - zone: { - type: "object", - properties: { - id: { type: "string", minLength: 1 }, - }, - required: ["id"], - additionalProperties: false, - }, - filter: { - type: "object", - properties: dnsRecordFilterProperties, - additionalProperties: false, - }, - }, - required: ["zone", "filter"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - result: { type: "array", items: objectOutput }, - }, - required: ["result"], - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { - zone: { id: string }; - filter: { recordType?: string }; - }) => ({ - result: - args.zone.id === "zone_eval_42" && - args.filter.recordType === "TXT" - ? [ - { - id: "dns_eval_9", - type: "TXT", - name: "example.test", - content: "connecta-eval-verification", - }, - ] - : [], - }), - }, - ], -}); - -const genericLedger = api("generic-ledger", { - title: "Generic Ledger API", - description: - "A hand-written API connector exposing a generic HTTP-shaped read tool", - usageGuide: { - content: `# Generic ledger request usage - -The exact address \`generic-ledger.request\` requires an admitted method, path, -and query shape. Listing open invoices uses \`GET /v1/invoices\` with -\`query: { status: "open" }\`; do not translate the user's wording into a -guessed endpoint. -`, - summary: "Required method, path, and query conventions for a generic API wrapper.", - required: true, - }, - tools: [ - { - name: "request", - description: - "Make an admitted read request to the ledger API by method, path, and query parameters.", - inputSchema: { - type: "object", - properties: { - method: { type: "string", enum: ["GET"] }, - path: { type: "string", enum: ["/v1/invoices"] }, - query: { - type: "object", - properties: { - status: { type: "string", enum: ["draft", "open", "paid"] }, - limit: { type: "integer", minimum: 1, maximum: 100 }, - }, - required: ["status"], - additionalProperties: false, - }, - }, - required: ["method", "path", "query"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - data: { type: "array", items: objectOutput }, - hasMore: { type: "boolean" }, - }, - required: ["data", "hasMore"], - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { - method: "GET"; - path: "/v1/invoices"; - query: { status: "draft" | "open" | "paid"; limit?: number }; - }) => ({ - data: - args.method === "GET" && - args.path === "/v1/invoices" && - args.query.status === "open" - ? [{ id: "in_eval_17", status: "open", amountDue: 4200 }] - : [], - hasMore: false, - }), - }, - ], -}); - -const unavailableCatalog: Connector = { - id: "billing-unavailable", - title: "Unavailable Billing", - kind: "api", - description: - "Billing account whose remote catalog is unavailable in this fixture", - async listTools() { - throw new ConnectorCallError( - "unavailable", - "Billing catalog unavailable: upstream returned 503. Retry after the deployment operator restores connector access.", - ); - }, - async callTool() { - throw new ConnectorCallError( - "unavailable", - "Billing connector access has not been restored.", - ); - }, -}; - -let mutationCount = 0; -const controlled = api("controlled", { - title: "Controlled Eval Fixtures", - description: - "Deterministic fixtures for calls, paging, reduction, and approval routing", - maxResultBytes: 700, - tools: [ - { - name: "read_record", - description: - "Return one deterministic record by id. Use this point lookup when specific ids are requested.", - inputSchema: { - type: "object", - properties: { - id: { type: "integer", minimum: 1, maximum: 10_000 }, - }, - required: ["id"], - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { id: number }) => ({ - id: args.id, - group: ["alpha", "beta", "gamma"][args.id % 3], - score: (args.id * 17) % 101, - }), - }, - { - name: "large_document", - description: - "Return a deterministic UTF-8 document large enough to require paging.", - inputSchema: { - type: "object", - properties: { - paragraphs: { - type: "integer", - minimum: 1, - maximum: 200, - default: 40, - }, - }, - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { paragraphs?: number }) => { - const count = args.paragraphs ?? 40; - const sections = Array.from( - { length: count }, - (_, index) => - `${index + 1}. Connecta retrieval handbook section. A direct read can return a legitimate document larger than the inline budget. get_result preserves complete text by byte offset. Stable section numbers expose missing or reordered content. UTF-8: café, 東京, 🧪.`, - ); - return { - title: "Deterministic retrieval handbook", - body: - sections.join("\n\n") + - `\n\nFINAL_MARKER: CONNECTA-LARGE-DOCUMENT-COMPLETE-${count}`, - }; - }, - }, - { - name: "records", - description: - "Generate a deterministic record collection for filtering and aggregation. Do not use this collection tool for point lookups by id.", - inputSchema: { - type: "object", - properties: { - count: { - type: "integer", - minimum: 1, - maximum: 500, - default: 120, - }, - }, - additionalProperties: false, - }, - outputSchema: { - type: "array", - items: { - type: "object", - properties: { - id: { type: "integer" }, - group: { type: "string" }, - score: { type: "integer" }, - }, - required: ["id", "group", "score"], - additionalProperties: false, - }, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { count?: number }) => - Array.from({ length: args.count ?? 120 }, (_, index) => ({ - id: index + 1, - group: ["alpha", "beta", "gamma"][index % 3], - score: (index * 17) % 101, - })), - }, - { - name: "increment_counter", - description: - "Increment an isolated counter to exercise approved destructive routing.", - inputSchema: { - type: "object", - properties: { - amount: { - type: "integer", - minimum: 1, - maximum: 10, - default: 1, - }, - }, - additionalProperties: false, - }, - annotations: { - readOnlyHint: false, - destructiveHint: true, - idempotentHint: false, - }, - handler: (args: { amount?: number }) => { - mutationCount += args.amount ?? 1; - return { counter: mutationCount }; - }, - }, - { - name: "activity_snapshot", - description: - "Report only the structural keys retained by the isolated activity sink.", - inputSchema: { - type: "object", - properties: {}, - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: () => { - const keys = [ - ...new Set(activityEvents.flatMap((event) => Object.keys(event))), - ].sort(); - const forbidden = [ - "args", - "arguments", - "result", - "results", - "code", - "error", - "errorText", - "rawError", - ]; - return { - eventCount: activityEvents.length, - keys, - forbiddenPresent: forbidden.filter((key) => keys.includes(key)), - }; - }, - }, - ], -}); - -const routing = api("routing", { - title: "Routing Guidance Eval Fixtures", - description: - "Deterministic fixtures for schema-aware selection and declared output roots", - tools: [ - { - name: "search_releases_by_registry", - description: - "Search releases by package in one private registry. Requires the registry tenant identifier.", - inputSchema: { - type: "object", - properties: { - package: { type: "string", minLength: 1 }, - registryTenantId: { type: "string", minLength: 1 }, - }, - required: ["package", "registryTenantId"], - additionalProperties: false, - }, - outputSchema: objectOutput, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { package: string; registryTenantId: string }) => ({ - package: args.package, - version: "private-fixture", - registryTenantId: args.registryTenantId, - }), - }, - { - name: "search_public_releases", - description: - "Search releases by package in the public catalog. Needs only the package name.", - inputSchema: { - type: "object", - properties: { - package: { type: "string", minLength: 1 }, - }, - required: ["package"], - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - package: { type: "string" }, - version: { type: "string" }, - channel: { type: "string" }, - }, - required: ["package", "version", "channel"], - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: (args: { package: string }) => ({ - package: args.package, - version: "0.12.2", - channel: "latest", - }), - }, - { - name: "list_active_incidents", - description: - "List all active routing incidents for aggregation or status summaries.", - inputSchema: { - type: "object", - properties: {}, - additionalProperties: false, - }, - outputSchema: { - type: "object", - properties: { - incidents: { - type: "array", - items: { - type: "object", - properties: { - id: { type: "string" }, - title: { type: "string" }, - severity: { type: "string" }, - status: { type: "string", enum: ["active"] }, - service: { type: "string", enum: ["routing"] }, - }, - required: ["id", "title", "severity", "status", "service"], - additionalProperties: false, - }, - }, - observedAt: { type: "string" }, - }, - required: ["incidents", "observedAt"], - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - handler: () => ({ - incidents: [ - { - id: "incident-catalog-delay", - title: "Catalog refresh delayed", - severity: "minor", - status: "active", - service: "routing", - }, - { - id: "incident-executor-queue", - title: "Executor queue elevated", - severity: "major", - status: "active", - service: "routing", - }, - ], - observedAt: "2026-08-03T12:00:00Z", - }), - }, - ], -}); - -function authTool(description: string): ToolDef { - return { - name: "whoami", - description, - inputSchema: { - type: "object", - properties: {}, - additionalProperties: false, - }, - annotations: { readOnlyHint: true, idempotentHint: true }, - }; -} - -const oauthRecoverableTools = [ - authTool("Return the OAuth fixture identity after consent."), -]; -let oauthAuthorized = false; -const oauthRecoverable: Connector = { - id: "oauth-recoverable", - kind: "api", - description: "OAuth recovery fixture with an isolated consent route", - staticTools: oauthRecoverableTools, - async listTools() { - return oauthRecoverableTools; - }, - async callTool() { - if (!oauthAuthorized) { - throw new ConnectorCallError( - "auth_required", - "OAuth consent is required.", - ); - } - return { id: "oauth-evaluator", recovered: true }; - }, - async status() { - return oauthAuthorized - ? { state: "ok" } - : { - state: "auth_required", - message: "OAuth consent is required.", - }; - }, - async startAuth(ctx) { - return { - state: "auth_required", - authorizationUrl: `${ctx.baseUrl}/fixture/oauth-recoverable/consent`, - message: "Open the isolated consent URL.", - }; - }, - async handleRequest(request) { - const url = new URL(request.url); - if (url.pathname !== "/fixture/oauth-recoverable/consent") return null; - oauthAuthorized = true; - return new Response("OAuth fixture authorized."); - }, -}; - -const oauthUnavailableTools = [ - authTool("Exercise an OAuth recovery path with no authorization URL."), -]; -const oauthUnavailable: Connector = { - id: "oauth-unavailable", - kind: "api", - description: "OAuth fixture whose provider cannot issue a consent URL", - staticTools: oauthUnavailableTools, - async listTools() { - return oauthUnavailableTools; - }, - async callTool() { - throw new ConnectorCallError( - "auth_required", - "OAuth provider is unavailable.", - ); - }, - async status() { - return { - state: "auth_required", - message: "OAuth provider is unavailable.", - }; - }, - async startAuth() { - return { - state: "auth_required", - message: "OAuth provider did not return an authorization URL.", - }; - }, -}; - -function staticCredentialConnector( - id: "static-recoverable" | "static-unavailable", - allowOperatorUpdate: boolean, -): Connector { - const tools = [ - authTool("Return the static-credential fixture identity after recovery."), - ]; - return { - id, - kind: "api", - description: allowOperatorUpdate - ? "Static credential fixture with an isolated operator handoff" - : "Static credential fixture with no available operator handoff", - ...(allowOperatorUpdate - ? { - credential: { - label: "Eval access token", - description: "Configured only by the isolated operator fixture.", - }, - } - : {}), - staticTools: tools, - async listTools() { - return tools; - }, - async callTool(_name, _args, ctx) { - if ((await ctx.credential?.get()) !== "sandbox-ok") { - throw new ConnectorCallError( - "auth_required", - `Credential for ${id} is missing.`, - ); - } - return { id: "static-evaluator", recovered: true }; - }, - }; -} - -const staticRecoverable = staticCredentialConnector( - "static-recoverable", - true, -); -const staticUnavailable = staticCredentialConnector( - "static-unavailable", - false, -); - -const baseExecutor = quickJsExecutor({ - timeoutMs: 10_000, - cpuTimeMs: 2_000, -}); -const executor = traceEnabled - ? tracing.tracedExecutor(baseExecutor) - : baseExecutor; - -const connecta = createConnecta({ - auth: [ - bearerToken(token, { subjectId: "current-version-evaluator" }), - operatorAuth, - ], - connectors: discoveryOnly - ? discoveryConnectors - : [ - ...discoveryConnectors, - guidedWorkItems, - bookshelf, - edgeDns, - genericLedger, - unavailableCatalog, - controlled, - routing, - oauthRecoverable, - oauthUnavailable, - staticRecoverable, - staticUnavailable, - ], - storage, - executor, - credentials: { - encryptionKey: credentialEncryptionKey, - }, - calls: { - defaultTimeoutMs: 15_000, - maxResultBytes: 8_000, - }, - activity: { - deploymentId: "current-version-eval", - store: { - record(event) { - activityEvents.push(event); - emitTrace({ - kind: "execution", - address: event.address, - source: event.source, - outcome: event.outcome, - durationMs: event.durationMs, - attempts: event.attempts, - ...(event.errorCode ? { errorCode: event.errorCode } : {}), - }); - }, - async list({ limit }) { - return { events: activityEvents.slice(-limit).reverse() }; - }, - }, - }, - serverInfo: { - name: "connecta-current-version-eval", - version: sourceCommit.slice(0, 12), - title: "Connecta current-version eval sandbox", - }, - deploymentInfo: { - sourceCommit, - isolated: true, - }, -}); - -const server = listen(traceEnabled ? tracing.withOuterTracing(connecta) : connecta, { - port, - host, - gracefulShutdown: false, -}); -await once(server, "listening"); -const address = server.address(); -if (!address || typeof address === "string") { - throw new Error("Eval server did not expose a TCP address."); -} - -console.log( - JSON.stringify({ - event: "ready", - url: `http://${host}:${address.port}/mcp`, - baseUrl: `http://${host}:${address.port}`, - sourceCommit, - connectorCount: discoveryConnectors.length + 11, - traceEnabled, - }), -); - -let shuttingDown = false; -async function shutdown(): Promise { - if (shuttingDown) return; - shuttingDown = true; - await new Promise((resolve, reject) => { - server.close((error) => (error ? reject(error) : resolve())); - }); - await connecta.close(); -} - -process.once("SIGINT", () => void shutdown().then(() => process.exit(0))); -process.once("SIGTERM", () => void shutdown().then(() => process.exit(0))); diff --git a/eval/current-version/server.mjs b/eval/current-version/server.mjs new file mode 100644 index 00000000..accd6a38 --- /dev/null +++ b/eval/current-version/server.mjs @@ -0,0 +1,252 @@ +import { once } from "node:events"; + +import { api, bearerToken, createConnecta } from "../../dist/index.js"; +import { quickJsExecutor } from "../../dist/executors/quickjs.js"; +import { listen } from "../../dist/node.js"; + +const host = "127.0.0.1"; +const port = Number(process.env.CONNECTA_BENCHMARK_PORT ?? "0"); +const token = process.env.CONNECTA_BENCHMARK_TOKEN ?? "connecta-benchmark-token"; +const downstreamCalls = []; +const outerCalls = []; + +const objectSchema = { + type: "object", + additionalProperties: true, +}; + +function readTool(name, description, inputSchema, handler, outputSchema = objectSchema) { + return { + name, + description, + inputSchema, + outputSchema, + annotations: { readOnlyHint: true }, + async handler(args) { + return handler(args); + }, + }; +} + +function connector(id, options) { + const tools = options.tools.map((tool) => ({ + ...tool, + handler: async (args) => { + downstreamCalls.push({ address: `${id}.${tool.name}`, args }); + return tool.handler(args); + }, + })); + return api(id, { ...options, tools }); +} + +const projectTools = [ + readTool( + "list_projects", + "List current projects with status and owner.", + { type: "object", properties: {}, additionalProperties: false }, + () => ({ + projects: [ + { name: "Atlas", status: "on_track", owner: "Rina Shah" }, + { name: "Pulse", id: 2803261, status: "active", owner: "Mina Cho" }, + ], + }), + ), +]; + +const mixpanelTools = [ + readTool( + "list_events", + "List tracked Mixpanel event names for a project.", + { + type: "object", + properties: { projectId: { type: "integer" } }, + required: ["projectId"], + additionalProperties: false, + }, + () => ({ + events: [ + "App Open or Present Session", + "App Intention FAQ Troubleshooting Opened", + "Present Session Completed", + ], + }), + ), + readTool( + "query_event_usage", + "Return a Mixpanel event total for one exact event name and date window.", + { + type: "object", + properties: { + projectId: { type: "integer" }, + eventName: { type: "string" }, + days: { type: "integer", minimum: 1, maximum: 90 }, + }, + required: ["projectId", "eventName", "days"], + additionalProperties: false, + }, + ({ projectId, eventName, days }) => ({ + headers: ["event", "total", "project_id", "days"], + rows: [[ + eventName, + projectId === 2803261 && eventName === "App Open or Present Session" && days === 30 + ? 334100 + : 471, + projectId, + days, + ]], + }), + ), +]; + +const customerRecords = Array.from({ length: 225 }, (_, index) => ({ + id: `cus_benchmark_${String(index + 1).padStart(3, "0")}`, + email: `private-${String(index + 1).padStart(3, "0")}@example.invalid`, + status: index % 9 === 0 ? "trialing" : "active", + plan: index % 3 === 0 ? "annual" : "monthly", +})); + +const subscriptionTools = [ + readTool( + "list_customers", + "List Stripe sandbox customers with cursor pagination.", + { + type: "object", + properties: { + limit: { type: "integer", minimum: 1, maximum: 100 }, + starting_after: { type: "string" }, + }, + additionalProperties: false, + }, + ({ limit = 100, starting_after: startingAfter }) => { + const start = startingAfter + ? customerRecords.findIndex((record) => record.id === startingAfter) + 1 + : 0; + const data = customerRecords.slice(start, start + limit); + return { + data, + has_more: start + data.length < customerRecords.length, + }; + }, + ), +]; + +const decoys = Array.from({ length: 16 }, (_, connectorIndex) => { + const id = `service-${String(connectorIndex + 1).padStart(2, "0")}`; + return connector(id, { + title: `Fixture service ${connectorIndex + 1}`, + description: "A deterministic benchmark catalog distractor.", + tools: Array.from({ length: 4 }, (_, toolIndex) => + readTool( + `lookup_record_${toolIndex + 1}`, + `Look up fixture record family ${connectorIndex + 1}.${toolIndex + 1}.`, + { + type: "object", + properties: { id: { type: "string" } }, + required: ["id"], + additionalProperties: false, + }, + ({ id: recordId }) => ({ id: recordId, fixture: true }), + ), + ), + }); +}); + +const connectors = [ + connector("projects", { + title: "Project registry", + description: "Current project ownership and delivery status.", + tools: projectTools, + }), + connector("mixpanel", { + title: "Mixpanel sandbox", + description: "Product analytics fixtures.", + usageGuide: { + required: true, + summary: "Exact project and event identities plus tabular response rules.", + content: `# Mixpanel benchmark guide + +- The project named Pulse has exact project id \`2803261\`, passed as a JSON number (never a string). +- “Product-wide usage” means the exact event \`App Open or Present Session\`. Do not fuzzy-match event names. +- Resolve it with \`list_events({ "projectId": 2803261 })\` before querying it. +- Query with exactly \`{ "projectId": 2803261, "eventName": "App Open or Present Session", "days": 30 }\`; these field names are case-sensitive. +- Query results are positional: validate that every row has the same width as \`headers\`, then map values by header name. +`, + }, + tools: mixpanelTools, + }), + connector("stripe-sandbox", { + title: "Stripe sandbox", + description: "Synthetic subscription and customer records; no production data.", + tools: subscriptionTools, + }), + ...decoys, +]; + +const app = createConnecta({ + auth: bearerToken(token, { subjectId: "benchmark-agent" }), + connectors, + executor: quickJsExecutor({ timeoutMs: 10_000, cpuTimeMs: 2_000 }), + calls: { defaultTimeoutMs: 10_000, maxResultBytes: 64_000 }, + serverInfo: { + name: "connecta-current-version-benchmark", + version: "1", + title: "Connecta current-version benchmark", + }, +}); + +async function benchmarkFetch(request, env, ctx) { + const url = new URL(request.url); + if (url.pathname === "/__benchmark/state") { + return Response.json({ downstreamCalls, outerCalls }); + } + + let requestJson; + if (url.pathname === "/mcp" && request.method === "POST") { + try { + requestJson = JSON.parse(await request.clone().text()); + } catch { + requestJson = undefined; + } + } + const response = await app.fetch(request, env, ctx); + if (requestJson?.method === "tools/call") { + const text = await response.clone().text(); + outerCalls.push({ + tool: requestJson.params?.name, + arguments: requestJson.params?.arguments, + status: response.status, + contentType: response.headers.get("content-type"), + responseText: text, + responseBytes: Buffer.byteLength(text), + }); + } + return response; +} + +const exposed = { ...app, fetch: benchmarkFetch }; +const server = listen(exposed, { port, host, gracefulShutdown: false }); +await once(server, "listening"); +const address = server.address(); +if (!address || typeof address === "string") { + throw new Error("Benchmark server did not expose a TCP address."); +} + +console.log(JSON.stringify({ + event: "ready", + url: `http://${host}:${address.port}/mcp`, + stateUrl: `http://${host}:${address.port}/__benchmark/state`, + token, +})); + +let closing = false; +async function close() { + if (closing) return; + closing = true; + await new Promise((resolve, reject) => { + server.close((error) => (error ? reject(error) : resolve())); + }); + await app.close(); +} + +process.once("SIGINT", () => void close().then(() => process.exit(0))); +process.once("SIGTERM", () => void close().then(() => process.exit(0))); diff --git a/eval/current-version/tsconfig.json b/eval/current-version/tsconfig.json deleted file mode 100644 index 9bb38975..00000000 --- a/eval/current-version/tsconfig.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "extends": "../../tsconfig.json", - "compilerOptions": { - "noEmit": true - }, - "include": [ - "cloudflare-fixture.ts", - "cloudflare-surface-report.ts", - "eval-tracing.ts", - "performance-server.ts", - "reference-connection-server.ts", - "sandbox-server.ts" - ] -} diff --git a/examples/worker/README.md b/examples/worker/README.md index 73914ec7..de0dbaa3 100644 --- a/examples/worker/README.md +++ b/examples/worker/README.md @@ -1,9 +1,9 @@ # connecta — Cloudflare Worker example A deployable Worker that aggregates a downstream remote MCP and an in-code HTTP -API connector, guarded by Clerk OAuth *and* a static bearer token, with state in -a KV namespace. Its required Worker Loader binding backs the seven-tool surface -and requires the Workers Paid plan. +API connector, guarded by Cloudflare Access, with state in a KV namespace. Its +required Worker Loader binding backs the seven-tool surface and requires the +Workers Paid plan. This is also the **starting template for a deployment**: a real deployment should be its own repository that pins an exact `@zackbart/connecta` version and @@ -37,18 +37,26 @@ npm install # from the package root wrangler kv namespace create CONNECTA_KV # paste the id into wrangler.jsonc cd examples/worker -wrangler secret put SUPPORT_TOKEN # one headless client -wrangler secret put EXEC_TOKEN # another headless client -wrangler secret put CLERK_SECRET_KEY wrangler secret put DOWNSTREAM_TOKEN wrangler secret put CREDENTIAL_ENCRYPTION_KEY # base64 32-byte AES key wrangler deploy ``` -`PUBLIC_URL` and `CLERK_PUBLISHABLE_KEY` are plain vars in `wrangler.jsonc`. -Enable Dynamic Client Registration on the Clerk instance (OAuth Applications → -DCR) so Claude/Cursor can self-register — full walkthrough in -[setting up Clerk](../../documentation/auth.md). +`PUBLIC_URL` is a plain var in `wrangler.jsonc`. After the first deploy, attach +Cloudflare Access to the Worker and choose the account, email-domain, or +advanced Zero Trust policy that owns admission. Enable **Managed OAuth** on +that Access application for interactive MCP clients. Access then serves OAuth +discovery and turns the client's opaque token into the trusted `ctx.access` +identity connecta reads. A cron job or CI client uses an Access service token +instead. + +Cloudflare's [Worker Access guide](https://developers.cloudflare.com/workers/configuration/cloudflare-access/) +owns the dashboard/API steps; its [Managed OAuth guide](https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/managed-oauth/) +owns client registration, redirect allowlists, and token lifetimes. + +The checked-in `access.dev` block gives `wrangler dev` a local operator +identity. Remove the block to test the missing-Access refusal. It has no effect +on a deployed Worker's production identity. ### Copied into its own repository @@ -57,24 +65,22 @@ dependency this file imports. A copy with its own `package.json` installs three things, because two of them are not part of connecta and never install with it: ```sh -npm install @zackbart/connecta @cloudflare/codemode @clerk/backend +npm install @zackbart/connecta @cloudflare/codemode ``` -Both are optional peers of `@zackbart/connecta` — declared in its manifest, -never installed with it, and each carrying the range this release supports. -`@cloudflare/codemode` is the executor behind `execute_code`, published as +`@cloudflare/codemode` is the optional peer behind `execute_code`, declared in +connecta's manifest but never installed with it, and published as `^0.4.4 || ^0.5.0`: install a version inside that and npm stays quiet, install one outside and npm says so at install time instead of leaving a Worker to discover the skew in production ([#376](https://github.com/zackbart/connecta/issues/376)). -`@clerk/backend` is the peer behind `@zackbart/connecta/auth/clerk`, which -`src/index.ts` imports at the top level, so wrangler must resolve it at -build time. Miss it and the build stops at -`Could not resolve "@clerk/backend"`, which is a missing peer rather than a -broken example. Drop `clerkAuth` from `auth` if this deployment has no operator -sign-in, and the peer goes with it — but read -[the operator surface](#the-operator-surface) first, because a deployment -without it can never write a credential or issue an access token. +`cloudflareAccessAuth()` has no dependency of its own. A deployment keeping +Clerk for rollback still installs `@clerk/backend` and keeps the commented +provider shape in `src/index.ts` until the migration is verified. + +```sh +npm install @clerk/backend # migration window only +``` Then point an MCP client at `/mcp`, and open `/` for Connections. Credentials is at `/credentials`, named MCP access tokens are at @@ -87,11 +93,11 @@ the next section for what turns each one on. This example ships the whole operator feature set. Three quarters of it is on as deployed; the fourth needs a database, so it is commented in place. -**Operator sign-in** is the `clerkAuth` entry in `src/index.ts`, alongside two -static bearers. The split is deliberate: a bearer is a client key that may call -tools and read connector status, while writing a credential or issuing an -access token requires an interactive Clerk identity. Narrow who that can be -with `allowedDomains`, or with a `gate` for anything a domain cannot express. +**Operator sign-in** is the `cloudflareAccessAuth()` entry in `src/index.ts`. +Access authenticates before the Worker runs. A human Access identity can use +MCP and operator pages; a service-token identity can use MCP but cannot write a +credential, run downstream OAuth, or issue a connecta token. Narrow admission +in the Access policy rather than repeating email domains or groups in code. **The credential vault** is `credentials: { encryptionKey: … }`, backed by the same KV namespace as everything else and encrypted with the @@ -116,12 +122,18 @@ shape on `echo` is exactly it) or use a provider connector such as `notion()`, which declares its own, and Credentials appears for a signed-in operator on the next load. -**Access tokens** are `accessTokens: {}`. A signed-in operator mints named, +**Access tokens** are `accessTokens: {}`. A signed-in human operator mints named, revocable Bearer tokens at `/tokens` for header-capable clients that will not do OAuth. Secrets are shown once and only their hashes enter KV; a lost token is reissued, never recovered. Note the KV caveat above — revocation is visible everywhere only as fast as the namespace converges. +Worker-level Access still runs before these tokens. A `cta_…` token therefore +does not reach connecta by itself; retain the feature as a rollback path or for +a caller that already supplies separate Access service-token headers. Normal +interactive MCP clients should use Managed OAuth, and unattended clients should +use Access service tokens. + **Activity** is the commented block in `src/index.ts` and the commented `d1_databases` binding in `wrangler.jsonc`; the section below creates the database and applies the schema. @@ -134,10 +146,17 @@ never the connector set, the tool catalog, or its annotations. this on: connector count, executor, seven tools. The executor it names is this one — `DynamicWorkerExecutor executed`, not the Node template's QuickJS, which is what doctor used to claim everywhere -([#368](https://github.com/zackbart/connecta/issues/368)). It carries a bearer, and a -bearer learns the model-facing surface rather than the deployment's -configuration topology. Confirm the operator surface the way an operator will: -sign in at `/` and check that Tokens is live. Credentials joins it +([#368](https://github.com/zackbart/connecta/issues/368)). Against Access it +carries `CF_ACCESS_CLIENT_ID` and `CF_ACCESS_CLIENT_SECRET`, and the service +identity learns the model-facing surface rather than deployment topology: + +```sh +CF_ACCESS_CLIENT_ID=… CF_ACCESS_CLIENT_SECRET=… \ + npx connecta doctor --url "$PUBLIC_URL" +``` + +Confirm the operator surface the way an operator will: sign in at +`/` and check that Tokens is live. Credentials joins it once a connector declares a `credential` slot, and Activity once the D1 wiring below is on — the nav shows a page when the deployment can actually serve it, so a missing page is the honest report that its half is still off. diff --git a/examples/worker/src/index.ts b/examples/worker/src/index.ts index f5997bab..560d8256 100644 --- a/examples/worker/src/index.ts +++ b/examples/worker/src/index.ts @@ -2,9 +2,10 @@ * connecta on Cloudflare Workers. * * One MCP endpoint aggregating a downstream remote MCP and an HTTP API, guarded - * by Clerk OAuth *and* a static bearer token, with OAuth/cache state in a KV - * namespace. The required Worker Loader binding in wrangler.jsonc backs the - * seven-tool surface. + * by Cloudflare Access, with OAuth/cache state in a KV namespace. Access + * authenticates the request before this Worker runs and supplies the trusted + * identity through ctx.access. The required Worker Loader binding in + * wrangler.jsonc backs the seven-tool surface. * * The operator surface is wired here except for activity history, which needs * a database this example does not create for you: sign-in, the credential @@ -15,19 +16,16 @@ * installed `@zackbart/connecta` package): * 1. `npm install` in the connecta package root (../../ from here) so the * package import and wrangler resolve. A copy in its own repository - * installs `@zackbart/connecta @cloudflare/codemode @clerk/backend` - * instead — the last two are not part of connecta, and the Clerk import - * below is an optional peer wrangler resolves at build time. + * installs `@zackbart/connecta @cloudflare/codemode` instead. Codemode is + * an optional peer; a migrating deployment also keeps `@clerk/backend` + * until it removes the commented rollback provider below. * 2. Create a KV namespace and put its id in wrangler.jsonc under `kv_namespaces`. * 3. Set secrets: - * wrangler secret put SUPPORT_TOKEN - * wrangler secret put EXEC_TOKEN - * wrangler secret put CLERK_SECRET_KEY * wrangler secret put DOWNSTREAM_TOKEN * wrangler secret put CREDENTIAL_ENCRYPTION_KEY - * and CLERK_PUBLISHABLE_KEY + PUBLIC_URL as plain vars in wrangler.jsonc. - * 4. Enable Dynamic Client Registration in the Clerk dashboard - * (OAuth Applications -> DCR toggle) so Claude/Cursor can self-register. + * and PUBLIC_URL as a plain var in wrangler.jsonc. + * 4. Attach Cloudflare Access to this Worker. Enable Managed OAuth on the + * Access application for interactive MCP clients. * 5. Use the Workers Paid plan required by the `worker_loaders` binding. * 6. `wrangler deploy` from this folder (examples/worker), where wrangler.jsonc * lives. Point your MCP client at `/mcp`. @@ -35,23 +33,21 @@ import { DynamicWorkerExecutor } from "@cloudflare/codemode"; import { api, - bearerToken, createConnecta, remoteMcp, } from "@zackbart/connecta"; -import { clerkAuth } from "@zackbart/connecta/auth/clerk"; +import { cloudflareAccessAuth } from "@zackbart/connecta/auth/cloudflare-access"; +// Rollback for a deployment migrating from Clerk: +// import { clerkAuth } from "@zackbart/connecta/auth/clerk"; import { cloudflareKvStorage } from "./cloudflare-kv.js"; // Activity history, off by default because it needs a D1 database. // import { d1ActivityStore } from "./d1-activity.js"; interface Env { CONNECTA_KV: KVNamespace; - /** Bearer token for one headless client in this deployment's audience. */ - SUPPORT_TOKEN: string; - /** Bearer token for another headless client in the same audience. */ - EXEC_TOKEN: string; - CLERK_PUBLISHABLE_KEY: string; - CLERK_SECRET_KEY: string; + // Keep these during a Clerk migration until Access has been verified: + // CLERK_PUBLISHABLE_KEY: string; + // CLERK_SECRET_KEY: string; /** * Base64 32-byte AES key encrypting operator-managed credentials in KV. * Unset means no vault: /credentials stays read-only and connecta says so at @@ -75,22 +71,19 @@ function build(env: Env) { storage: cloudflareKvStorage(env.CONNECTA_KV), executor: new DynamicWorkerExecutor({ loader: env.LOADER }), auth: [ - // Multiple credentials may identify callers in one deployment. Every - // admitted caller reaches this deployment's deliberate connector set. - bearerToken(env.SUPPORT_TOKEN, { - subjectId: "support-team", - }), - bearerToken(env.EXEC_TOKEN, { - subjectId: "exec-team", - }), - // The operator signs in with Clerk. Restrict who may sign in with - // `allowedDomains` (or a `gate`, for anything a domain cannot express). - clerkAuth({ - publishableKey: env.CLERK_PUBLISHABLE_KEY, - secretKey: env.CLERK_SECRET_KEY, - publicUrl: env.PUBLIC_URL, - // allowedDomains: ["acme.com"], - }), + // Access owns admission policy. A human identity may use MCP and the + // operator pages; a service token may use MCP but cannot mutate operator + // state. Neither path asks connecta to parse a JWT. + cloudflareAccessAuth(), + // Leave the previous Clerk provider below this entry during migration. + // It is a rollback path until Worker-level Access is detached; Access + // itself decides whether a request reaches this array. + // clerkAuth({ + // publishableKey: env.CLERK_PUBLISHABLE_KEY, + // secretKey: env.CLERK_SECRET_KEY, + // publicUrl: env.PUBLIC_URL, + // allowedDomains: ["acme.com"], + // }), ], // Connectors that declare a `credential` slot become editable at // /credentials, encrypted with this key before anything reaches KV. A @@ -105,8 +98,9 @@ function build(env: Env) { // `echo`, or use a provider connector like `notion()`, which declares its // own) and the page appears on the next load. credentials: { encryptionKey: env.CREDENTIAL_ENCRYPTION_KEY }, - // Eligible Clerk operators can create named, revocable MCP Bearer tokens - // at /tokens. Secrets are shown once; only their hashes enter KV. + // Eligible human operators can create named, revocable MCP Bearer tokens + // at /tokens. Under Worker-level Access those tokens are a rollback tool, + // not standalone edge credentials: Access still runs before connecta. accessTokens: {}, // Payload-free activity at /activity, off until a database exists to hold // it. Uncomment the `d1_databases` binding in wrangler.jsonc, apply the diff --git a/examples/worker/wrangler.jsonc b/examples/worker/wrangler.jsonc index d0d7ec36..d7b90acf 100644 --- a/examples/worker/wrangler.jsonc +++ b/examples/worker/wrangler.jsonc @@ -6,12 +6,20 @@ "compatibility_flags": ["nodejs_compat"], "observability": { "enabled": true }, - // Plain vars. Secrets (SUPPORT_TOKEN, EXEC_TOKEN, CLERK_SECRET_KEY, - // CREDENTIAL_ENCRYPTION_KEY, DOWNSTREAM_TOKEN) are set with + // Plain vars. Secrets (CREDENTIAL_ENCRYPTION_KEY, DOWNSTREAM_TOKEN) are set with // `wrangler secret put `, not here. "vars": { - "PUBLIC_URL": "https://connecta.example.workers.dev", - "CLERK_PUBLISHABLE_KEY": "pk_test_replace-me" + "PUBLIC_URL": "https://connecta.example.workers.dev" + }, + + // Local-only Access identity. Remove this block to exercise the + // unauthenticated path in `wrangler dev`; production identity comes from the + // Worker-level Access application, not from this file. + "access": { + "dev": { + "aud": "connecta-local", + "identity": { "email": "operator@example.com" } + } }, // Create with `wrangler kv namespace create CONNECTA_KV` and paste the id. diff --git a/package-lock.json b/package-lock.json index a1d82694..bbfe1d91 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@zackbart/connecta", - "version": "0.20.0", + "version": "0.21.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@zackbart/connecta", - "version": "0.20.0", + "version": "0.21.0", "license": "MIT", "dependencies": { "@cfworker/json-schema": "^4.1.1", diff --git a/package.json b/package.json index 24853ae9..588371cb 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@zackbart/connecta", - "version": "0.20.0", + "version": "0.21.0", "type": "module", "sideEffects": false, "description": "One MCP to rule them all — a single MCP endpoint aggregating many downstream connectors behind a code-first surface of seven meta-tools.", @@ -63,6 +63,10 @@ "types": "./dist/auth/clerk.d.ts", "import": "./dist/auth/clerk.js" }, + "./auth/cloudflare-access": { + "types": "./dist/auth/cloudflare-access.d.ts", + "import": "./dist/auth/cloudflare-access.js" + }, "./providers/cloudflare": { "types": "./dist/providers/cloudflare.d.ts", "import": "./dist/providers/cloudflare.js" diff --git a/scripts/check-package.mjs b/scripts/check-package.mjs index f6642091..64fc68a8 100644 --- a/scripts/check-package.mjs +++ b/scripts/check-package.mjs @@ -250,6 +250,8 @@ try { "dist/executors/quickjs-child.js", "dist/executors/quickjs-protocol.js", "dist/executors/quickjs-runtime.js", + "dist/auth/cloudflare-access.js", + "dist/auth/cloudflare-access.d.ts", "dist/providers/mixpanel.js", "dist/providers/mixpanel.d.ts", "dist/providers/revenuecat.js", @@ -497,8 +499,10 @@ for (const name of [ join(work, "optional.mjs"), ` const clerk = await import("@zackbart/connecta/auth/clerk"); +const access = await import("@zackbart/connecta/auth/cloudflare-access"); const quickjs = await import("@zackbart/connecta/quickjs"); if (typeof clerk.clerkAuth !== "function") throw new Error("missing Clerk adapter"); +if (typeof access.cloudflareAccessAuth !== "function") throw new Error("missing Cloudflare Access adapter"); if (typeof quickjs.quickJsExecutor !== "function") throw new Error("missing QuickJS adapter"); const executor = quickjs.quickJsExecutor({ timeoutMs: 2_000 }); try { diff --git a/src/auth/bearer.ts b/src/auth/bearer.ts index 4671badd..3cb3fb57 100644 --- a/src/auth/bearer.ts +++ b/src/auth/bearer.ts @@ -23,8 +23,8 @@ export interface BearerTokenOptions { /** * Static bearer-token inbound auth. Constant-time compares the Bearer token - * against `secret`. Checked BEFORE the Clerk gate in the server; a mismatch - * falls through so a co-configured Clerk provider can still admit the request. + * against `secret`. Checked before interactive providers in the server; a + * mismatch falls through so another configured provider can admit the request. */ export function bearerToken( secret: string, diff --git a/src/auth/clerk.ts b/src/auth/clerk.ts index 4c4111b2..01707327 100644 --- a/src/auth/clerk.ts +++ b/src/auth/clerk.ts @@ -473,6 +473,7 @@ export function clerkAuth(opts: ClerkAuthOptions): InboundAuth { return { kind: "clerk", + interactiveOperator: true, activityActorNamespace: frontendApiUrl, activityActorLabel: resolveActivityLabel, uiAuth: { diff --git a/src/auth/cloudflare-access.ts b/src/auth/cloudflare-access.ts new file mode 100644 index 00000000..51ed488d --- /dev/null +++ b/src/auth/cloudflare-access.ts @@ -0,0 +1,79 @@ +import type { AuthResult, InboundAuth } from "../types.js"; + +function identityString( + identity: Record, + field: string, +): string | undefined { + const value = identity[field]; + return typeof value === "string" && value.length > 0 ? value : undefined; +} + +function unauthorized(): AuthResult { + return { + ok: false, + response: Response.json( + { error: "Cloudflare Access authentication required" }, + { status: 401 }, + ), + }; +} + +/** + * Trust the identity Cloudflare Access attached to this direct Worker + * invocation. Access has already validated the browser session, Managed OAuth + * token, or service-token headers before the Worker runs; this adapter does + * not accept or parse a caller-supplied JWT. + */ +export function cloudflareAccessAuth(): InboundAuth { + return { + kind: "cloudflare-access", + interactiveOperator: true, + activityActorNamespace: "cloudflare-access", + uiAuth: { kind: "cloudflare-access" }, + + async authorize(_request, _baseUrl, runtimeContext): Promise { + const access = runtimeContext?.access; + if (!access) return unauthorized(); + + let identity: Record | undefined; + try { + identity = await access.getIdentity(); + } catch { + return unauthorized(); + } + if (!identity) return unauthorized(); + + const userId = identityString(identity, "user_uuid") ?? + identityString(identity, "email"); + const commonName = identityString(identity, "common_name"); + const serviceTokenId = identityString(identity, "service_token_id"); + if ( + identity.service_token_status === true || + serviceTokenId || + (!userId && commonName) + ) { + const subjectId = serviceTokenId ?? commonName; + return subjectId + ? { ok: true, subjectId } + : { + ok: false, + response: Response.json( + { error: "Cloudflare Access service identity required" }, + { status: 403 }, + ), + }; + } + + if (!userId) { + return { + ok: false, + response: Response.json( + { error: "Cloudflare Access user identity required" }, + { status: 403 }, + ), + }; + } + return { ok: true, userId, subjectId: userId }; + }, + }; +} diff --git a/src/execute.ts b/src/execute.ts index df498e36..ec88d0aa 100644 --- a/src/execute.ts +++ b/src/execute.ts @@ -29,7 +29,12 @@ import { InvocationService, } from "./invocation.js"; import type { RegistryView } from "./registry.js"; -import { hasConnectorGuides } from "./skills.js"; +import { + connectorGuide, + connectorGuideRequired, + connectorSkillName, + hasConnectorGuides, +} from "./skills.js"; import type { Executor, ExecutorProvider, @@ -1188,9 +1193,14 @@ function connectorInventory( if (connectors.length === 0) return `${prefix}none.`; const entries = connectors.map((connector) => { const shortcut = sanitizeIdentifier(connector.id); - return shortcut === connector.id + const address = shortcut === connector.id ? connector.id : `${connector.id} (shortcut ${shortcut})`; + if (!connectorGuide(connector)) return address; + const requirement = connectorGuideRequired(connector) + ? "required guide" + : "guide"; + return `${address} (${requirement} ${connectorSkillName(connector.id)})`; }); const shown: string[] = []; for (let index = 0; index < entries.length; index++) { @@ -1216,18 +1226,18 @@ const executeDescription = ( emitBudgets: { maxBytes: number; maxBlocks: number }, connectorGuides: boolean, connectors: ReturnType, -) => `Choose the route before discovery. Exactly one unknown-address read uses top-level search_tools then call_tool. execute_code is the primary surface for everything wider: make exactly one execute_code call that searches, selects, calls, and reduces. A discovery-only program wastes its round trip: finish here, don't return catalog matches for a later call. Only readOnlyHint: true tools are available. Limits: ${EXECUTE_MAX_HOST_CALLS} host calls per run, ${EXECUTE_MAX_BATCH_CALLS} per batch, ${EXECUTE_HOST_CALL_TIMEOUT_MS / 1_000}-second host deadline. +) => `Choose the route before discovery. A known address uses call_tool. Unknown-address and wider read-only work use exactly one execute_code call that discovers, calls, and returns the answer. Finish in that program; don't return catalog matches for a later call. Only readOnlyHint: true tools are available. Limits: ${EXECUTE_MAX_HOST_CALLS} host calls per run, ${EXECUTE_MAX_BATCH_CALLS} per batch, ${EXECUTE_HOST_CALL_TIMEOUT_MS / 1_000}-second host deadline. ${connectorInventory(connectors)} -Write one plain-JavaScript async arrow function. Use only: +Fetch required guides named above before executing. Write one plain-JavaScript async arrow function. Use only: - .(args) for a sanitized shortcut, or connecta.call(address, args) for a canonical address. -- connecta.search(args), connecta.describe(args), and connecta.batch(calls) for discovery and independent read-only calls. +- connecta.search(args) returns { tools }; connecta.describe(args) returns { tools }; use entry key lists. connecta.batch(calls) accepts canonical connector addresses only. - connecta.emit(block) — { type: "text", text } or { type: "image" | "audio", data (base64), mimeType }. Success-only; ${emitBudgets.maxBlocks} blocks/${emitBudgets.maxBytes} bytes; invalid/over-budget throws. - connecta.ui(html) for one display-only, success-only view; return the same summary the HTML renders. - console.log(...) — captured. -Programs have no portable ambient capabilities. Return JSON and reduce large results before they truncate. Fetch skills({ name: "usage" }) once for selection rules, exact result shapes, repair, examples, guide handling${connectorGuides ? ", connector-guide rules" : ""}, and runtime differences.`; +No portable ambient capabilities. Return JSON; reduce large results before truncation. Build arguments from required input keys and schemas, never descriptions or output keys. Fetch skills({ name: "usage" }) only when this is insufficient or repair is needed; it has full rules, examples${connectorGuides ? ", guide handling" : ""}, and runtime details.`; /** Register the execute_code meta-tool. Only called when an executor is configured. */ export function registerExecuteTool( diff --git a/src/index.ts b/src/index.ts index 95969570..468623a8 100644 --- a/src/index.ts +++ b/src/index.ts @@ -167,7 +167,7 @@ export interface ConnectaConfig { credentials?: ConnectaCredentialsConfig; /** * Named, revocable Bearer tokens for MCP clients. Creation and mutation - * require an eligible Clerk operator; token secrets are returned once. + * require an eligible interactive operator; token secrets are returned once. */ accessTokens?: ConnectaAccessTokensConfig; /** Tool-catalog caching, persistence, stale fallback, and probe deadlines. */ @@ -534,10 +534,10 @@ export function createConnecta(config: ConnectaConfig): Connecta { : undefined; if ( accessTokens && - !configuredAuth.some((provider) => provider.uiAuth?.kind === "clerk") + !configuredAuth.some((provider) => provider.interactiveOperator) ) { throw new Error( - "accessTokens requires a Clerk auth provider: only an eligible Clerk " + + "accessTokens requires an interactive operator auth provider: only an eligible " + "operator may create, rename, or revoke deployment access tokens", ); } @@ -619,7 +619,7 @@ export function createConnecta(config: ConnectaConfig): Connecta { handler( request, ctx && typeof (ctx as { waitUntil?: unknown }).waitUntil === "function" - ? (ctx as { waitUntil(promise: Promise): void }) + ? (ctx as import("./routes/shared.js").RuntimeExecutionContext) : undefined, ), registry, @@ -687,6 +687,7 @@ export type { ExecutorLease, ExecutorProvider, InboundAuth, + InboundAuthRuntimeContext, UiAuthConfig, AuthResult, JsonSchema, diff --git a/src/meta-tools.ts b/src/meta-tools.ts index 8b6a0ec1..f8b0ed84 100644 --- a/src/meta-tools.ts +++ b/src/meta-tools.ts @@ -778,9 +778,9 @@ export function createMetaTools( }; } -const SEARCH_DESC = `Use top-level search for one unknown-address read before call_tool, or for approval-required work before call_destructive_tool. Use 2–4 action/object terms and includeSchemas="compact"; the default limit is ${DEFAULT_SEARCH_LIMIT}. Set connector when known. safety="readOnly" finds direct or program calls; "approvalRequired" finds the fail-closed complement. These filters grant no authority. For multiple, dependent, or reduced read-only calls, use one execute_code program instead. Empty query browses.`; +const SEARCH_DESC = `Use top-level search for catalog inspection or approval-required work before call_destructive_tool. Unknown-address read-only work belongs in one execute_code program that searches, calls, and returns the answer. Use 2–4 action/object terms and includeSchemas="compact"; the default limit is ${DEFAULT_SEARCH_LIMIT}. Set connector when known. safety="readOnly" finds direct or program calls; "approvalRequired" finds the fail-closed complement. These filters grant no authority. Empty query browses.`; const CALL_DESC = - 'Call one tool explicitly annotated readOnlyHint: true. Use execute_code for multiple, dependent, or reduced read-only calls. Unannotated or write-capable tools fail closed to call_destructive_tool. A truncated result carries a get_result action.'; + 'Call one known-address tool explicitly annotated readOnlyHint: true. Use execute_code for unknown-address, multiple, dependent, or reduced read-only work. Unannotated or write-capable tools fail closed to call_destructive_tool. A truncated result carries a get_result action.'; const CALL_DESTRUCTIVE_DESC = "Call any tool not explicitly annotated readOnlyHint: true. Include a short reason for the human reviewer after checking the schema and consequences. The reason grants no authority and is not sent downstream."; const GET_RESULT_DESC = @@ -788,7 +788,7 @@ const GET_RESULT_DESC = const AUTHORIZE_DESC = "Use after auth_required. Returns an OAuth or operator-credential handoff, or reports required deployment configuration. force=true restarts OAuth only; this tool never accepts credentials."; const SKILLS_DESC = - 'List or fetch on-demand guidance. Fetch usage once per task for program syntax, selection, repair, examples, and runtime details.'; + 'List or fetch on-demand guidance. Fetch usage only when the always-loaded instructions are insufficient or a program needs repair.'; /** * Sentences appended to a meta-tool description only when this connection diff --git a/src/operator-ui/app/main.tsx b/src/operator-ui/app/main.tsx index d7e3135a..72a521a7 100644 --- a/src/operator-ui/app/main.tsx +++ b/src/operator-ui/app/main.tsx @@ -85,7 +85,7 @@ function OperatorNav() { ))}
- {auth.kind === "clerk" ? ( + {auth.kind === "clerk" || auth.kind === "cloudflare-access" ? ( @@ -126,6 +126,12 @@ function Gate({ state }: { state: OperatorState }) { )}
+ ) : auth.kind === "cloudflare-access" ? ( +
+ +
) : (
{ + if (auth.kind === "cloudflare-access") return Promise.resolve(undefined); return auth.kind === "clerk" ? Promise.resolve(window.Clerk?.session?.getToken() ?? null) : Promise.resolve(localStorage.getItem(TOKEN_KEY)); } +function requestHeaders( + token: string | null | undefined, + body = false, +): Record { + return { + ...(token ? { Authorization: `Bearer ${token}` } : {}), + ...(body ? { "Content-Type": "application/json" } : {}), + }; +} + function gate(notice: Notice | null = null): void { state = resetIdentity(state, notice); for (const listener of listeners) listener(); @@ -85,13 +96,13 @@ async function operatorRequest( ): Promise { const token = await sessionToken(); if (!current()) throw new Error("The operator session changed."); - if (!token) throw new Error("Your operator session has expired."); + if (!token && auth.kind !== "cloudflare-access") { + throw new Error("Your operator session has expired."); + } const res = await fetch(path, { method, - headers: { - Authorization: `Bearer ${token}`, - ...(body ? { "Content-Type": "application/json" } : {}), - }, + headers: requestHeaders(token, Boolean(body)), + credentials: "same-origin", ...(body ? { body: JSON.stringify(body) } : {}), }); if (res.status === 204) return null; @@ -126,11 +137,12 @@ async function loadData(): Promise { return gate(failure(`Could not read the Clerk session: ${why}`)); } if (!current()) return; - if (!token) return gate(null); + if (!token && auth.kind !== "cloudflare-access") return gate(null); let res: Response; try { res = await fetch("/ui/data", { - headers: { Authorization: `Bearer ${token}` }, + headers: requestHeaders(token), + credentials: "same-origin", }); } catch (error) { if (!current()) return; @@ -147,6 +159,13 @@ async function loadData(): Promise { ), ); } + if (auth.kind === "cloudflare-access") { + return gate( + failure( + "Cloudflare Access admitted the request, but this identity is not an eligible operator.", + ), + ); + } localStorage.removeItem(TOKEN_KEY); return gate(failure("Token rejected — enter a valid bearer token.")); } @@ -551,6 +570,11 @@ export function signIn(): void { } export function signOut(): void { + if (auth.kind === "cloudflare-access") { + gate(null); + window.location.assign("/cdn-cgi/access/logout"); + return; + } const clerk = window.Clerk; gate(null); void clerk?.signOut({ redirectUrl: window.location.href }); diff --git a/src/operator-ui/generated.ts b/src/operator-ui/generated.ts index 60d564a7..65dd9295 100644 --- a/src/operator-ui/generated.ts +++ b/src/operator-ui/generated.ts @@ -1,4 +1,4 @@ // Generated by scripts/build-operator-ui.mjs. Do not edit. // Source: src/operator-ui/app/main.tsx and src/operator-ui/browser.css. export const OPERATOR_UI_CSS: string = "/* src/operator-ui/browser.css */\n:root {\n color-scheme: light;\n --ink: #000;\n --paper: #fff;\n --rule: #ccc;\n --muted: #666;\n --trace: #f5f5f5;\n --shell: 70rem;\n --pad: 1rem;\n --gap: 1.5rem;\n --sans:\n \"Helvetica Neue\",\n Helvetica,\n Arial,\n sans-serif;\n --mono:\n ui-monospace,\n \"SF Mono\",\n Menlo,\n Monaco,\n \"Cascadia Code\",\n Consolas,\n monospace;\n}\n* {\n border-radius: 0;\n box-sizing: border-box;\n}\nhtml {\n background: var(--paper);\n color: var(--ink);\n font-family: var(--sans);\n font-size: 16px;\n line-height: 1.5;\n -webkit-font-smoothing: antialiased;\n text-rendering: optimizeLegibility;\n}\nbody {\n margin: 0;\n min-height: 100vh;\n}\n::selection {\n background: var(--ink);\n color: var(--paper);\n}\n:is(h1, h2, h3, p, ul, ol) {\n margin: 0;\n padding: 0;\n}\n:is(h1, h2, h3) {\n font-size: inherit;\n font-weight: 400;\n}\n:is(ul, ol) {\n list-style: none;\n}\na {\n color: inherit;\n}\nbutton,\ninput {\n font: inherit;\n}\nbutton {\n background: none;\n border: 0;\n color: inherit;\n cursor: pointer;\n margin: 0;\n padding: 0;\n text-align: left;\n}\nbutton:disabled {\n cursor: wait;\n opacity: .5;\n}\ninput {\n background: var(--paper);\n border: 1px solid var(--rule);\n color: var(--ink);\n min-height: 2rem;\n padding: .2rem .5rem;\n}\ninput:focus-visible {\n outline-offset: -1px;\n}\n:is(a, button, input, summary):focus-visible {\n outline: 1px solid var(--ink);\n}\n:is(a, button, summary):focus-visible {\n outline-offset: 2px;\n}\n.skip-link {\n background: var(--paper);\n left: var(--pad);\n padding: .5rem;\n position: fixed;\n top: -4rem;\n z-index: 10;\n}\n.skip-link:focus {\n top: var(--pad);\n}\n.shell {\n margin: 0 auto;\n max-width: var(--shell);\n padding-left: var(--pad);\n padding-right: var(--pad);\n}\n.pgrid {\n column-gap: var(--gap);\n display: grid;\n grid-template-columns: repeat(3, minmax(0, 1fr));\n row-gap: 1rem;\n}\n.pcap {\n grid-column: 1;\n}\n.pbody {\n grid-column: 2 / -1;\n min-width: 0;\n}\n.cap,\n.meta {\n color: var(--muted);\n font-size: .9em;\n}\n.mono {\n font-family: var(--mono);\n font-size: .78rem;\n}\n.visually-hidden {\n clip: rect(0 0 0 0);\n clip-path: inset(50%);\n height: 1px;\n overflow: hidden;\n position: absolute;\n white-space: nowrap;\n width: 1px;\n}\n.masthead {\n align-items: start;\n padding-bottom: var(--pad);\n padding-top: var(--pad);\n}\n.brand {\n font-weight: 500;\n grid-column: 1;\n text-decoration: none;\n}\n.mast-nav {\n display: flex;\n gap: var(--gap);\n grid-column: 2 / -1;\n justify-content: space-between;\n min-width: 0;\n}\n.mast-actions {\n display: flex;\n gap: var(--gap);\n justify-content: flex-end;\n min-width: 0;\n}\n.page-nav,\n.session-actions {\n display: flex;\n flex-wrap: wrap;\n gap: .5rem var(--gap);\n}\n.mast-actions :is(a, button) {\n align-items: center;\n display: inline-flex;\n min-height: 2rem;\n}\n.navlink,\n.linklike {\n text-decoration: underline;\n text-decoration-thickness: 1.5px;\n text-underline-offset: .22em;\n}\n.navlink {\n text-decoration-color: transparent;\n}\n.navlink:hover,\n.navlink:focus-visible,\n.navlink[aria-current=page] {\n text-decoration-color: currentColor;\n}\n.linklike {\n text-decoration-color: currentColor;\n}\n.linklike:hover,\n.linklike:focus-visible {\n text-decoration-color: transparent;\n}\n.page {\n padding-bottom: 5rem;\n}\n.lead {\n margin-top: 6rem;\n}\n.section,\n.section + .section {\n margin-top: 3rem;\n}\n.lead-copy,\n.body-copy {\n max-width: 34em;\n}\n.lead-copy > * + *,\n.body-copy > * + * {\n margin-top: 1.5rem;\n}\n.row,\n.actions {\n align-items: center;\n display: flex;\n flex-wrap: wrap;\n gap: var(--gap);\n}\n.row input {\n flex: 1;\n min-width: 12rem;\n}\n.gate-actions {\n margin-top: 1.5rem;\n}\n#err {\n margin-top: 1.5rem;\n text-decoration: underline;\n}\n.endpoint {\n border-bottom: 1px solid var(--rule);\n border-top: 1px solid var(--rule);\n}\n.endpoint-row {\n align-items: baseline;\n display: flex;\n gap: var(--gap);\n min-width: 0;\n padding: .75rem 0;\n}\n.endpoint-row code {\n flex: 1;\n min-width: 0;\n overflow-x: auto;\n white-space: nowrap;\n}\n.endpoint-row button {\n flex: none;\n}\n.connector-tools {\n border-bottom: 1px solid var(--rule);\n}\n.toolbar {\n margin-bottom: 1.5rem;\n}\n.toolbar input {\n flex-basis: 18rem;\n}\n#oauthNotice {\n margin-bottom: .75rem;\n}\n#oauthNotice:empty {\n display: none;\n}\n.error-notice,\n.msg {\n text-decoration: underline;\n}\n.card,\n.credential-card,\n.activity-item {\n padding-left: 1.25rem;\n position: relative;\n}\n.card::before,\n.credential-card::before,\n.activity-item::before {\n background: var(--rule);\n bottom: 0;\n content: \"\";\n left: .25rem;\n position: absolute;\n top: 0;\n width: 1px;\n}\n.card {\n border-top: 1px solid var(--rule);\n padding-bottom: .75rem;\n padding-top: .75rem;\n}\n.connector-head {\n display: grid;\n gap: var(--gap);\n grid-template-columns: minmax(0, 2fr) minmax(10rem, 1fr);\n}\n.connector-title {\n align-items: baseline;\n display: flex;\n gap: .5rem;\n}\n.connector-title .dot,\n.activity-stamp .dot {\n margin-left: -1.25rem;\n}\n.activity-stamp {\n align-items: baseline;\n display: flex;\n gap: .75rem;\n}\n.card h2 {\n overflow-wrap: anywhere;\n}\n.connector-state {\n text-align: right;\n}\n.dot {\n background: var(--paper);\n border: 1px solid var(--ink);\n display: inline-block;\n flex: none;\n height: .5rem;\n width: .5rem;\n z-index: 1;\n}\n.dot.ok {\n background: var(--ink);\n}\n.dot.auth_required {\n background:\n linear-gradient(\n 90deg,\n var(--ink) 50%,\n var(--paper) 50%);\n}\n.connector-description {\n margin-top: .25rem;\n max-width: 40rem;\n}\n.connector-message,\n.connector-auth {\n margin-top: .75rem;\n}\n.connector-drift {\n border-top: 1px solid var(--rule);\n margin-top: .75rem;\n padding-top: .75rem;\n}\n.drift-summary {\n margin-top: .25rem;\n}\n.drift-counts {\n display: flex;\n flex-wrap: wrap;\n gap: .25rem var(--gap);\n margin-top: .5rem;\n}\n.drift-count {\n color: var(--muted);\n font-size: .9em;\n}\n.drift-count.flagged {\n color: var(--ink);\n}\n.drift-count-value {\n font-family: var(--mono);\n margin-right: .35rem;\n}\n.drift-count.flagged .drift-count-value {\n text-decoration: underline;\n}\n.credential-ledger {\n border-bottom: 1px solid var(--rule);\n}\n.credential-card {\n border-top: 1px solid var(--rule);\n padding-bottom: .75rem;\n padding-top: .75rem;\n}\n.credential-head {\n align-items: baseline;\n display: flex;\n flex-wrap: wrap;\n gap: .25rem var(--gap);\n justify-content: space-between;\n}\n.credential-copy {\n margin-top: .25rem;\n max-width: 40rem;\n}\n.credential-field-summary {\n border-top: 1px solid var(--rule);\n margin-top: .75rem;\n}\n.credential-field-summary > div {\n border-bottom: 1px solid var(--rule);\n display: flex;\n flex-wrap: wrap;\n gap: .25rem var(--gap);\n justify-content: space-between;\n padding: .5rem 0;\n}\n.credential-actions {\n display: flex;\n flex-wrap: wrap;\n gap: var(--gap);\n margin-top: .75rem;\n}\n.credential-actions button,\n.credential-form button,\n.activity-controls button,\n.activity-more {\n align-items: center;\n display: inline-flex;\n min-height: 2.75rem;\n}\n.credential-form {\n align-items: center;\n display: flex;\n flex-wrap: wrap;\n gap: .75rem var(--gap);\n margin-top: .75rem;\n}\n.credential-form > input {\n flex: 1 1 18rem;\n}\n.credential-fields {\n display: grid;\n flex: 1 1 100%;\n gap: .75rem;\n}\n.credential-field {\n align-items: center;\n display: grid;\n gap: var(--gap);\n grid-template-columns: minmax(9rem, 12rem) 1fr;\n}\n.credential-field input {\n min-width: 0;\n width: 100%;\n}\n.danger {\n text-decoration-style: double;\n}\n.token-create {\n border-bottom: 1px solid var(--rule);\n border-top: 1px solid var(--rule);\n padding: .75rem 0;\n}\n.token-create > label {\n display: block;\n margin-bottom: .5rem;\n}\n.token-create input {\n flex: 1 1 18rem;\n}\n.token-create button,\n.token-card button,\n.token-reveal button {\n align-items: center;\n display: inline-flex;\n min-height: 2.75rem;\n}\n.token-reveal {\n background: var(--ink);\n color: var(--paper);\n margin-top: 1.5rem;\n padding: 1rem 1.25rem;\n}\n.token-reveal .meta,\n.token-reveal .cap {\n color: #bbb;\n}\n.token-reveal-head,\n.token-card-head {\n align-items: baseline;\n display: flex;\n flex-wrap: wrap;\n gap: .25rem var(--gap);\n justify-content: space-between;\n}\n.token-secret {\n border-bottom: 1px solid #555;\n border-top: 1px solid #555;\n margin-top: .75rem;\n}\n.token-secret code {\n color: var(--paper);\n user-select: all;\n}\n.token-ledger {\n border-bottom: 1px solid var(--rule);\n margin-top: 1.5rem;\n}\n.token-card {\n border-top: 1px solid var(--rule);\n padding: .75rem 0 .75rem 1.25rem;\n position: relative;\n}\n.token-card::before {\n background: var(--ink);\n bottom: 0;\n content: \"\";\n left: .25rem;\n position: absolute;\n top: 0;\n width: 1px;\n}\n.token-card.revoked {\n color: var(--muted);\n}\n.token-card.revoked::before {\n background: var(--rule);\n}\ndetails {\n margin-top: .75rem;\n}\nsummary {\n cursor: pointer;\n list-style: none;\n width: max-content;\n}\nsummary::-webkit-details-marker {\n display: none;\n}\n.tool-list {\n border-bottom: 1px solid var(--rule);\n margin-top: .5rem;\n}\n.tool {\n border-top: 1px solid var(--rule);\n display: grid;\n gap: .25rem var(--gap);\n grid-template-columns: minmax(12rem, 1fr) minmax(0, 2fr);\n padding: .5rem 0;\n}\n.tool code {\n font-family: var(--mono);\n font-size: .78rem;\n overflow-wrap: anywhere;\n}\n.tool .td {\n color: var(--muted);\n font-size: .9em;\n}\n.empty {\n border-top: 1px solid var(--rule);\n padding: .75rem 0;\n}\n.activity-copy {\n margin-bottom: 1.5rem;\n}\n.activity-controls {\n margin-bottom: 1.5rem;\n}\n.activity-controls input {\n flex: 1 1 18rem;\n}\n#activityNotice {\n margin-bottom: .75rem;\n}\n.activity-ledger {\n border-bottom: 1px solid var(--rule);\n}\n.activity-item {\n border-top: 1px solid var(--rule);\n display: grid;\n gap: .25rem var(--gap);\n grid-template-columns: minmax(9rem, .85fr) minmax(12rem, 1.4fr) minmax(8rem, .9fr);\n padding-bottom: .75rem;\n padding-top: .75rem;\n}\n.activity-time,\n.activity-actor,\n.activity-detail {\n color: var(--muted);\n font-size: .82rem;\n}\n.activity-actor-id {\n color: var(--muted);\n margin-top: .1rem;\n}\n.activity-address {\n font-family: var(--mono);\n font-size: .78rem;\n overflow-wrap: anywhere;\n}\n.activity-outcome {\n font-size: .9em;\n}\n.activity-item.error .activity-outcome,\n.activity-item.timeout .activity-outcome,\n.activity-item.cancelled .activity-outcome {\n text-decoration: underline;\n}\n.activity-empty {\n border-top: 1px solid var(--rule);\n padding: .75rem 0;\n}\n.activity-more {\n margin-top: .75rem;\n}\n.unavailable {\n background: var(--trace);\n border-bottom: 1px solid var(--rule);\n border-top: 1px solid var(--rule);\n padding: .75rem;\n}\n@media (prefers-reduced-motion: reduce) {\n html:focus-within {\n scroll-behavior: auto;\n }\n}\n@media (max-width: 36.99rem) {\n .pgrid {\n grid-template-columns: repeat(2, minmax(0, 1fr));\n }\n .pcap,\n .pbody {\n grid-column: 1 / -1;\n }\n .masthead .brand {\n grid-column: 1;\n }\n .mast-nav {\n grid-column: 1 / -1;\n grid-row: 2;\n justify-content: flex-start;\n }\n .product {\n display: none;\n }\n .mast-actions {\n align-items: flex-start;\n flex-direction: column;\n font-size: .875rem;\n gap: .25rem;\n }\n .lead {\n margin-top: 4rem;\n }\n .section,\n .section + .section {\n margin-top: 2.5rem;\n }\n .connector-head,\n .tool,\n .activity-item {\n grid-template-columns: 1fr;\n }\n .connector-state {\n text-align: left;\n }\n .credential-field {\n align-items: start;\n grid-template-columns: 1fr;\n gap: .25rem;\n }\n input {\n min-height: 2.75rem;\n }\n}\n"; -export const OPERATOR_UI_SCRIPT: string = "\"use strict\";\n(() => {\n // node_modules/preact/dist/preact.module.js\n var n;\n var l;\n var u;\n var t;\n var i;\n var r;\n var o;\n var e;\n var f;\n var c;\n var a;\n var s;\n var h;\n var p;\n var v;\n var y;\n var d = {};\n var w = [];\n var _ = /acit|ex(?:s|g|n|p|$)|rph|grid|ows|mnc|ntw|ine[ch]|zoo|^ord|itera/i;\n var g = Array.isArray;\n function m(n2, l3) {\n for (var u4 in l3) n2[u4] = l3[u4];\n return n2;\n }\n function b(n2) {\n n2 && n2.parentNode && n2.parentNode.removeChild(n2);\n }\n function k(l3, u4, t3) {\n var i3, r3, o3, e3 = {};\n for (o3 in u4) \"key\" == o3 ? i3 = u4[o3] : \"ref\" == o3 ? r3 = u4[o3] : e3[o3] = u4[o3];\n if (arguments.length > 2 && (e3.children = arguments.length > 3 ? n.call(arguments, 2) : t3), \"function\" == typeof l3 && null != l3.defaultProps) for (o3 in l3.defaultProps) void 0 === e3[o3] && (e3[o3] = l3.defaultProps[o3]);\n return x(l3, e3, i3, r3, null);\n }\n function x(n2, t3, i3, r3, o3) {\n var e3 = { type: n2, props: t3, key: i3, ref: r3, __k: null, __: null, __b: 0, __e: null, __c: null, constructor: void 0, __v: null == o3 ? ++u : o3, __i: -1, __u: 0 };\n return null == o3 && null != l.vnode && l.vnode(e3), e3;\n }\n function S(n2) {\n return n2.children;\n }\n function C(n2, l3) {\n this.props = n2, this.context = l3;\n }\n function $(n2, l3) {\n if (null == l3) return n2.__ ? $(n2.__, n2.__i + 1) : null;\n for (var u4; l3 < n2.__k.length; l3++) if (null != (u4 = n2.__k[l3]) && null != u4.__e) return u4.__e;\n return \"function\" == typeof n2.type ? $(n2) : null;\n }\n function I(n2) {\n if (n2.__P && n2.__d) {\n var u4 = n2.__v, t3 = u4.__e, i3 = [], r3 = [], o3 = m({}, u4);\n o3.__v = u4.__v + 1, l.vnode && l.vnode(o3), q(n2.__P, o3, u4, n2.__n, n2.__P.namespaceURI, 32 & u4.__u ? [t3] : null, i3, null == t3 ? $(u4) : t3, !!(32 & u4.__u), r3), o3.__v = u4.__v, o3.__.__k[o3.__i] = o3, D(i3, o3, r3), u4.__e = u4.__ = null, o3.__e != t3 && P(o3);\n }\n }\n function P(n2) {\n if (null != (n2 = n2.__) && null != n2.__c) return n2.__e = n2.__c.base = null, n2.__k.some(function(l3) {\n if (null != l3 && null != l3.__e) return n2.__e = n2.__c.base = l3.__e;\n }), P(n2);\n }\n function A(n2) {\n (!n2.__d && (n2.__d = true) && i.push(n2) && !H.__r++ || r != l.debounceRendering) && ((r = l.debounceRendering) || o)(H);\n }\n function H() {\n try {\n for (var n2, l3 = 1; i.length; ) i.length > l3 && i.sort(e), n2 = i.shift(), l3 = i.length, I(n2);\n } finally {\n i.length = H.__r = 0;\n }\n }\n function L(n2, l3, u4, t3, i3, r3, o3, e3, f4, c3, a3) {\n var s3, h3, p3, v3, y3, _3, g2 = t3 && t3.__k || w, m3 = l3.length;\n for (f4 = T(u4, l3, g2, f4, m3), s3 = 0; s3 < m3; s3++) null != (p3 = u4.__k[s3]) && (h3 = -1 != p3.__i && g2[p3.__i] || d, p3.__i = s3, _3 = q(n2, p3, h3, i3, r3, o3, e3, f4, c3, a3), v3 = p3.__e, p3.ref && h3.ref != p3.ref && (h3.ref && J(h3.ref, null, p3), a3.push(p3.ref, p3.__c || v3, p3)), null == y3 && null != v3 && (y3 = v3), 4 & p3.__u ? (f4 = j(p3, f4, n2), h3.__e && (h3.__e = null)) : \"function\" == typeof p3.type && void 0 !== _3 ? f4 = _3 : v3 && (f4 = v3.nextSibling), p3.__u &= -7);\n return u4.__e = y3, f4;\n }\n function T(n2, l3, u4, t3, i3) {\n var r3, o3, e3, f4, c3, a3 = u4.length, s3 = a3, h3 = 0;\n for (n2.__k = new Array(i3), r3 = 0; r3 < i3; r3++) null != (o3 = l3[r3]) && \"boolean\" != typeof o3 && \"function\" != typeof o3 ? (\"string\" == typeof o3 || \"number\" == typeof o3 || \"bigint\" == typeof o3 || o3.constructor == String ? o3 = n2.__k[r3] = x(null, o3, null, null, null) : g(o3) ? o3 = n2.__k[r3] = x(S, { children: o3 }, null, null, null) : void 0 === o3.constructor && o3.__b > 0 ? o3 = n2.__k[r3] = x(o3.type, o3.props, o3.key, o3.ref ? o3.ref : null, o3.__v) : n2.__k[r3] = o3, f4 = r3 + h3, o3.__ = n2, o3.__b = n2.__b + 1, e3 = null, -1 != (c3 = o3.__i = O(o3, u4, f4, s3)) && (s3--, (e3 = u4[c3]) && (e3.__u |= 2)), null == e3 || null == e3.__v ? (-1 == c3 && (i3 > a3 ? h3-- : i3 < a3 && h3++), \"function\" != typeof o3.type && (o3.__u |= 4)) : c3 != f4 && (c3 == f4 - 1 ? h3-- : c3 == f4 + 1 ? h3++ : (c3 > f4 ? h3-- : h3++, o3.__u |= 4))) : n2.__k[r3] = null;\n if (s3) for (r3 = 0; r3 < a3; r3++) null != (e3 = u4[r3]) && 0 == (2 & e3.__u) && (e3.__e == t3 && (t3 = $(e3)), K(e3, e3));\n return t3;\n }\n function j(n2, l3, u4) {\n var t3, i3;\n if (\"function\" == typeof n2.type) {\n for (t3 = n2.__k, i3 = 0; t3 && i3 < t3.length; i3++) t3[i3] && (t3[i3].__ = n2, l3 = j(t3[i3], l3, u4));\n return l3;\n }\n n2.__e != l3 && (l3 && n2.type && !l3.parentNode && (l3 = $(n2)), l3 = u4.insertBefore(n2.__e, l3 || null));\n do {\n l3 = l3 && l3.nextSibling;\n } while (null != l3 && 8 == l3.nodeType);\n return l3;\n }\n function O(n2, l3, u4, t3) {\n var i3, r3, o3, e3 = n2.key, f4 = n2.type, c3 = l3[u4], a3 = null != c3 && 0 == (2 & c3.__u);\n if (null === c3 && null == e3 || a3 && e3 == c3.key && f4 == c3.type) return u4;\n if (t3 > (a3 ? 1 : 0)) {\n for (i3 = u4 - 1, r3 = u4 + 1; i3 >= 0 || r3 < l3.length; ) if (null != (c3 = l3[o3 = i3 >= 0 ? i3-- : r3++]) && 0 == (2 & c3.__u) && e3 == c3.key && f4 == c3.type) return o3;\n }\n return -1;\n }\n function z(n2, l3, u4) {\n \"-\" == l3[0] ? n2.setProperty(l3, null == u4 ? \"\" : u4) : n2[l3] = null == u4 ? \"\" : \"number\" != typeof u4 || _.test(l3) ? u4 : u4 + \"px\";\n }\n function N(n2, l3, u4, t3, i3) {\n var r3, o3;\n n: if (\"style\" == l3) if (\"string\" == typeof u4) n2.style.cssText = u4;\n else {\n if (\"string\" == typeof t3 && (n2.style.cssText = t3 = \"\"), t3) for (l3 in t3) u4 && l3 in u4 || z(n2.style, l3, \"\");\n if (u4) for (l3 in u4) t3 && u4[l3] == t3[l3] || z(n2.style, l3, u4[l3]);\n }\n else if (\"o\" == l3[0] && \"n\" == l3[1]) r3 = l3 != (l3 = l3.replace(s, \"$1\")), o3 = l3.toLowerCase(), l3 = o3 in n2 || \"onFocusOut\" == l3 || \"onFocusIn\" == l3 ? o3.slice(2) : l3.slice(2), n2.l || (n2.l = {}), n2.l[l3 + r3] = u4, u4 ? t3 ? u4[a] = t3[a] : (u4[a] = h, n2.addEventListener(l3, r3 ? v : p, r3)) : n2.removeEventListener(l3, r3 ? v : p, r3);\n else {\n if (\"http://www.w3.org/2000/svg\" == i3) l3 = l3.replace(/xlink(H|:h)/, \"h\").replace(/sName$/, \"s\");\n else if (\"width\" != l3 && \"height\" != l3 && \"href\" != l3 && \"list\" != l3 && \"form\" != l3 && \"tabIndex\" != l3 && \"download\" != l3 && \"rowSpan\" != l3 && \"colSpan\" != l3 && \"role\" != l3 && \"popover\" != l3 && l3 in n2) try {\n n2[l3] = null == u4 ? \"\" : u4;\n break n;\n } catch (n3) {\n }\n \"function\" == typeof u4 || (null == u4 || false === u4 && \"-\" != l3[4] ? n2.removeAttribute(l3) : n2.setAttribute(l3, \"popover\" == l3 && 1 == u4 ? \"\" : u4));\n }\n }\n function V(n2) {\n return function(u4) {\n if (this.l) {\n var t3 = this.l[u4.type + n2];\n if (null == u4[c]) u4[c] = h++;\n else if (u4[c] < t3[a]) return;\n return t3(l.event ? l.event(u4) : u4);\n }\n };\n }\n function q(n2, u4, t3, i3, r3, o3, e3, f4, c3, a3) {\n var s3, h3, p3, v3, y3, d3, _3, k3, x2, M, I2, P2, A2, H2, T2, j3, F = u4.type;\n if (void 0 !== u4.constructor) return null;\n 128 & t3.__u && (c3 = !!(32 & t3.__u), o3 = [f4 = u4.__e = t3.__e]), (s3 = l.__b) && s3(u4);\n n: if (\"function\" == typeof F) {\n h3 = e3.length;\n try {\n if (x2 = u4.props, M = F.prototype && F.prototype.render, I2 = (s3 = F.contextType) && i3[s3.__c], P2 = s3 ? I2 ? I2.props.value : s3.__ : i3, t3.__c ? k3 = (p3 = u4.__c = t3.__c).__ = p3.__E : (M ? u4.__c = p3 = new F(x2, P2) : (u4.__c = p3 = new C(x2, P2), p3.constructor = F, p3.render = Q), I2 && I2.sub(p3), p3.state || (p3.state = {}), p3.__n = i3, v3 = p3.__d = true, p3.__h = [], p3._sb = []), M && null == p3.__s && (p3.__s = p3.state), M && null != F.getDerivedStateFromProps && (p3.__s == p3.state && (p3.__s = m({}, p3.__s)), m(p3.__s, F.getDerivedStateFromProps(x2, p3.__s))), y3 = p3.props, d3 = p3.state, p3.__v = u4, v3) M && null == F.getDerivedStateFromProps && null != p3.componentWillMount && p3.componentWillMount(), M && null != p3.componentDidMount && p3.__h.push(p3.componentDidMount);\n else {\n if (M && null == F.getDerivedStateFromProps && x2 !== y3 && null != p3.componentWillReceiveProps && p3.componentWillReceiveProps(x2, P2), u4.__v == t3.__v || !p3.__e && null != p3.shouldComponentUpdate && false === p3.shouldComponentUpdate(x2, p3.__s, P2)) {\n u4.__v != t3.__v && (p3.props = x2, p3.state = p3.__s, p3.__d = false), u4.__e = t3.__e, u4.__k = t3.__k, u4.__k.some(function(n3) {\n n3 && (n3.__ = u4);\n }), w.push.apply(p3.__h, p3._sb), p3._sb = [], p3.__h.length && e3.push(p3), f4 = $(t3);\n break n;\n }\n null != p3.componentWillUpdate && p3.componentWillUpdate(x2, p3.__s, P2), M && null != p3.componentDidUpdate && p3.__h.push(function() {\n p3.componentDidUpdate(y3, d3, _3);\n });\n }\n if (p3.context = P2, p3.props = x2, p3.__P = n2, p3.__e = false, A2 = l.__r, H2 = 0, M) p3.state = p3.__s, p3.__d = false, A2 && A2(u4), s3 = p3.render(p3.props, p3.state, p3.context), w.push.apply(p3.__h, p3._sb), p3._sb = [];\n else do {\n p3.__d = false, A2 && A2(u4), s3 = p3.render(p3.props, p3.state, p3.context), p3.state = p3.__s;\n } while (p3.__d && ++H2 < 25);\n p3.state = p3.__s, null != p3.getChildContext && (i3 = m(m({}, i3), p3.getChildContext())), M && !v3 && null != p3.getSnapshotBeforeUpdate && (_3 = p3.getSnapshotBeforeUpdate(y3, d3)), T2 = null != s3 && s3.type === S && null == s3.key ? E(s3.props.children) : s3, f4 = L(n2, g(T2) ? T2 : [T2], u4, t3, i3, r3, o3, e3, f4, c3, a3), p3.base = u4.__e, u4.__u &= -161, p3.__h.length && e3.push(p3), k3 && (p3.__E = p3.__ = null);\n } catch (n3) {\n if (e3.length = h3, u4.__v = null, c3 || null != o3) {\n if (n3.then) {\n for (u4.__u |= c3 ? 160 : 128; f4 && 8 == f4.nodeType && f4.nextSibling; ) f4 = f4.nextSibling;\n null != o3 && (o3[o3.indexOf(f4)] = null), u4.__e = f4;\n } else if (null != o3) for (j3 = o3.length; j3--; ) b(o3[j3]);\n } else u4.__e = t3.__e;\n null == u4.__k && (u4.__k = t3.__k || []), n3.then || B(u4), l.__e(n3, u4, t3);\n }\n } else null == o3 && u4.__v == t3.__v ? (u4.__k = t3.__k, u4.__e = t3.__e) : f4 = u4.__e = G(t3.__e, u4, t3, i3, r3, o3, e3, c3, a3);\n return (s3 = l.diffed) && s3(u4), 128 & u4.__u ? void 0 : f4;\n }\n function B(n2) {\n n2 && (n2.__c && (n2.__c.__e = true), n2.__k && n2.__k.some(B));\n }\n function D(n2, u4, t3) {\n for (var i3 = 0; i3 < t3.length; i3++) J(t3[i3], t3[++i3], t3[++i3]);\n l.__c && l.__c(u4, n2), n2.some(function(u5) {\n try {\n n2 = u5.__h, u5.__h = [], n2.some(function(n3) {\n n3.call(u5);\n });\n } catch (n3) {\n l.__e(n3, u5.__v);\n }\n });\n }\n function E(n2) {\n return \"object\" != typeof n2 || null == n2 || n2.__b > 0 ? n2 : g(n2) ? n2.map(E) : void 0 !== n2.constructor ? null : m({}, n2);\n }\n function G(u4, t3, i3, r3, o3, e3, f4, c3, a3) {\n var s3, h3, p3, v3, y3, w3, _3, m3 = i3.props || d, k3 = t3.props, x2 = t3.type;\n if (\"svg\" == x2 ? o3 = \"http://www.w3.org/2000/svg\" : \"math\" == x2 ? o3 = \"http://www.w3.org/1998/Math/MathML\" : o3 || (o3 = \"http://www.w3.org/1999/xhtml\"), null != e3) {\n for (s3 = 0; s3 < e3.length; s3++) if ((y3 = e3[s3]) && \"setAttribute\" in y3 == !!x2 && (x2 ? y3.localName == x2 : 3 == y3.nodeType)) {\n u4 = y3, e3[s3] = null;\n break;\n }\n }\n if (null == u4) {\n if (null == x2) return document.createTextNode(k3);\n u4 = document.createElementNS(o3, x2, k3.is && k3), c3 && (l.__m && l.__m(t3, e3), c3 = false), e3 = null;\n }\n if (null == x2) m3 === k3 || c3 && u4.data == k3 || (u4.data = k3);\n else {\n if (e3 = \"textarea\" == x2 && null != k3.defaultValue ? null : e3 && n.call(u4.childNodes), !c3 && null != e3) for (m3 = {}, s3 = 0; s3 < u4.attributes.length; s3++) m3[(y3 = u4.attributes[s3]).name] = y3.value;\n for (s3 in m3) y3 = m3[s3], \"dangerouslySetInnerHTML\" == s3 ? p3 = y3 : \"children\" == s3 || s3 in k3 || \"value\" == s3 && \"defaultValue\" in k3 || \"checked\" == s3 && \"defaultChecked\" in k3 || N(u4, s3, null, y3, o3);\n for (s3 in k3) y3 = k3[s3], \"children\" == s3 ? v3 = y3 : \"dangerouslySetInnerHTML\" == s3 ? h3 = y3 : \"value\" == s3 ? w3 = y3 : \"checked\" == s3 ? _3 = y3 : c3 && \"function\" != typeof y3 || m3[s3] === y3 || N(u4, s3, y3, m3[s3], o3);\n if (h3) c3 || p3 && (h3.__html == p3.__html || h3.__html == u4.innerHTML) || (u4.innerHTML = h3.__html), t3.__k = [];\n else if (p3 && (u4.innerHTML = \"\"), L(\"template\" == t3.type ? u4.content : u4, g(v3) ? v3 : [v3], t3, i3, r3, \"foreignObject\" == x2 ? \"http://www.w3.org/1999/xhtml\" : o3, e3, f4, e3 ? e3[0] : i3.__k && $(i3, 0), c3, a3), null != e3) for (s3 = e3.length; s3--; ) b(e3[s3]);\n c3 && \"textarea\" != x2 || (s3 = \"value\", \"progress\" == x2 && null == w3 ? u4.removeAttribute(\"value\") : null != w3 && (w3 !== u4[s3] || \"progress\" == x2 && !w3 || \"option\" == x2 && w3 != m3[s3]) && N(u4, s3, w3, m3[s3], o3), s3 = \"checked\", null != _3 && _3 != u4[s3] && N(u4, s3, _3, m3[s3], o3));\n }\n return u4;\n }\n function J(n2, u4, t3) {\n try {\n if (\"function\" == typeof n2) {\n var i3 = \"function\" == typeof n2.__u;\n i3 && n2.__u(), i3 && null == u4 || (n2.__u = n2(u4));\n } else n2.current = u4;\n } catch (n3) {\n l.__e(n3, t3);\n }\n }\n function K(n2, u4, t3) {\n var i3, r3;\n if (l.unmount && l.unmount(n2), (i3 = n2.ref) && (i3.current && i3.current != n2.__e || J(i3, null, u4)), null != (i3 = n2.__c)) {\n if (i3.componentWillUnmount) try {\n i3.componentWillUnmount();\n } catch (n3) {\n l.__e(n3, u4);\n }\n i3.base = i3.__P = i3.__n = null;\n }\n if (i3 = n2.__k) for (r3 = 0; r3 < i3.length; r3++) i3[r3] && K(i3[r3], u4, t3 || \"function\" != typeof n2.type);\n t3 || b(n2.__e), n2.__c = n2.__ = n2.__e = void 0;\n }\n function Q(n2, l3, u4) {\n return this.constructor(n2, u4);\n }\n function R(u4, t3, i3) {\n var r3, o3, e3, f4;\n t3 == document && (t3 = document.documentElement), l.__ && l.__(u4, t3), o3 = (r3 = \"function\" == typeof i3) ? null : i3 && i3.__k || t3.__k, e3 = [], f4 = [], q(t3, u4 = (!r3 && i3 || t3).__k = k(S, null, [u4]), o3 || d, d, t3.namespaceURI, !r3 && i3 ? [i3] : o3 ? null : t3.firstChild ? n.call(t3.childNodes) : null, e3, !r3 && i3 ? i3 : o3 ? o3.__e : t3.firstChild, r3, f4), D(e3, u4, f4), u4.props.children = null;\n }\n n = w.slice, l = { __e: function(n2, l3, u4, t3) {\n for (var i3, r3, o3; l3 = l3.__; ) if ((i3 = l3.__c) && !i3.__) try {\n if ((r3 = i3.constructor) && null != r3.getDerivedStateFromError && (i3.setState(r3.getDerivedStateFromError(n2)), o3 = i3.__d), null != i3.componentDidCatch && (i3.componentDidCatch(n2, t3 || {}), o3 = i3.__d), o3) return i3.__E = i3;\n } catch (l4) {\n n2 = l4;\n }\n throw n2;\n } }, u = 0, t = function(n2) {\n return null != n2 && void 0 === n2.constructor;\n }, C.prototype.setState = function(n2, l3) {\n var u4;\n u4 = null != this.__s && this.__s != this.state ? this.__s : this.__s = m({}, this.state), \"function\" == typeof n2 && (n2 = n2(m({}, u4), this.props)), n2 && m(u4, n2), null != n2 && this.__v && (l3 && this._sb.push(l3), A(this));\n }, C.prototype.forceUpdate = function(n2) {\n this.__v && (this.__e = true, n2 && this.__h.push(n2), A(this));\n }, C.prototype.render = S, i = [], o = \"function\" == typeof Promise ? Promise.prototype.then.bind(Promise.resolve()) : setTimeout, e = function(n2, l3) {\n return n2.__v.__b - l3.__v.__b;\n }, H.__r = 0, f = Math.random().toString(8), c = \"__d\" + f, a = \"__a\" + f, s = /(PointerCapture)$|Capture$/i, h = 0, p = V(false), v = V(true), y = 0;\n\n // node_modules/preact/hooks/dist/hooks.module.js\n var t2;\n var r2;\n var u2;\n var i2;\n var o2 = 0;\n var f2 = [];\n var c2 = l;\n var e2 = c2.__b;\n var a2 = c2.__r;\n var v2 = c2.diffed;\n var l2 = c2.__c;\n var m2 = c2.unmount;\n var p2 = c2.__;\n function s2(n2, t3) {\n c2.__h && c2.__h(r2, n2, o2 || t3), o2 = 0;\n var u4 = r2.__H || (r2.__H = { __: [], __h: [] });\n return n2 >= u4.__.length && u4.__.push({}), u4.__[n2];\n }\n function d2(n2) {\n return o2 = 1, y2(D2, n2);\n }\n function y2(n2, u4, i3) {\n var o3 = s2(t2++, 2);\n if (o3.t = n2, !o3.__c && (o3.__ = [i3 ? i3(u4) : D2(void 0, u4), function(n3) {\n var t3 = o3.__N ? o3.__N[0] : o3.__[0], r3 = o3.t(t3, n3);\n t3 !== r3 && (o3.__N = [r3, o3.__[1]], o3.__c.setState({}));\n }], o3.__c = r2, !r2.__f)) {\n var f4 = function(n3, t3, r3) {\n if (!o3.__c.__H) return true;\n var u5 = false, i4 = o3.__c.props !== n3;\n if (o3.__c.__H.__.some(function(n4) {\n if (n4.__N) {\n u5 = true;\n var t4 = n4.__[0];\n n4.__ = n4.__N, n4.__N = void 0, t4 !== n4.__[0] && (i4 = true);\n }\n }), c3) {\n var f5 = c3.call(this, n3, t3, r3);\n return u5 ? f5 || i4 : f5;\n }\n return !u5 || i4;\n };\n r2.__f = true;\n var c3 = r2.shouldComponentUpdate, e3 = r2.componentWillUpdate;\n r2.componentWillUpdate = function(n3, t3, r3) {\n if (this.__e) {\n var u5 = c3;\n c3 = void 0, f4(n3, t3, r3), c3 = u5;\n }\n e3 && e3.call(this, n3, t3, r3);\n }, r2.shouldComponentUpdate = f4;\n }\n return o3.__N || o3.__;\n }\n function h2(n2, u4) {\n var i3 = s2(t2++, 3);\n !c2.__s && C2(i3.__H, u4) && (i3.__ = n2, i3.u = u4, r2.__H.__h.push(i3));\n }\n function _2(n2, u4) {\n var i3 = s2(t2++, 4);\n !c2.__s && C2(i3.__H, u4) && (i3.__ = n2, i3.u = u4, r2.__h.push(i3));\n }\n function j2() {\n for (var n2; n2 = f2.shift(); ) {\n var t3 = n2.__H;\n if (n2.__P && t3) try {\n t3.__h.some(z2), t3.__h.some(B2), t3.__h = [];\n } catch (r3) {\n t3.__h = [], c2.__e(r3, n2.__v);\n }\n }\n }\n c2.__b = function(n2) {\n r2 = null, e2 && e2(n2);\n }, c2.__ = function(n2, t3) {\n n2 && t3.__k && t3.__k.__m && (n2.__m = t3.__k.__m), p2 && p2(n2, t3);\n }, c2.__r = function(n2) {\n a2 && a2(n2), t2 = 0;\n var i3 = (r2 = n2.__c).__H;\n i3 && (u2 === r2 ? (i3.__h = [], r2.__h = [], i3.__.some(function(n3) {\n n3.__N && (n3.__ = n3.__N), n3.u = n3.__N = void 0;\n })) : (i3.__h.some(z2), i3.__h.some(B2), i3.__h = [], t2 = 0)), u2 = r2;\n }, c2.diffed = function(n2) {\n v2 && v2(n2);\n var t3 = n2.__c;\n t3 && t3.__H && (t3.__H.__h.length && (1 !== f2.push(t3) && i2 === c2.requestAnimationFrame || ((i2 = c2.requestAnimationFrame) || w2)(j2)), t3.__H.__.some(function(n3) {\n n3.u && (n3.__H = n3.u, n3.u = void 0);\n })), u2 = r2 = null;\n }, c2.__c = function(n2, t3) {\n t3.some(function(n3) {\n try {\n n3.__h.some(z2), n3.__h = n3.__h.filter(function(n4) {\n return !n4.__ || B2(n4);\n });\n } catch (r3) {\n t3.some(function(n4) {\n n4.__h && (n4.__h = []);\n }), t3 = [], c2.__e(r3, n3.__v);\n }\n }), l2 && l2(n2, t3);\n }, c2.unmount = function(n2) {\n m2 && m2(n2);\n var t3, r3 = n2.__c;\n r3 && r3.__H && (r3.__H.__.some(function(n3) {\n try {\n z2(n3);\n } catch (n4) {\n t3 = n4;\n }\n }), r3.__H = void 0, t3 && c2.__e(t3, r3.__v));\n };\n var k2 = \"function\" == typeof requestAnimationFrame;\n function w2(n2) {\n var t3, r3 = function() {\n clearTimeout(u4), k2 && cancelAnimationFrame(t3), setTimeout(n2);\n }, u4 = setTimeout(r3, 35);\n k2 && (t3 = requestAnimationFrame(r3));\n }\n function z2(n2) {\n var t3 = r2, u4 = n2.__c;\n \"function\" == typeof u4 && (n2.__c = void 0, u4()), r2 = t3;\n }\n function B2(n2) {\n var t3 = r2;\n n2.__c = n2.__(), r2 = t3;\n }\n function C2(n2, t3) {\n return !n2 || n2.length !== t3.length || t3.some(function(t4, r3) {\n return t4 !== n2[r3];\n });\n }\n function D2(n2, t3) {\n return \"function\" == typeof t3 ? t3(n2) : t3;\n }\n\n // src/operator-ui/view.ts\n var OPERATOR_PAGES = [\n \"connections\",\n \"credentials\",\n \"tokens\",\n \"activity\"\n ];\n var PAGE_META = {\n connections: { path: \"/\", label: \"Connections\" },\n credentials: { path: \"/credentials\", label: \"Credentials\" },\n tokens: { path: \"/tokens\", label: \"Access tokens\" },\n activity: { path: \"/activity\", label: \"Activity\" }\n };\n function pageForPath(path) {\n const match = OPERATOR_PAGES.find((page) => PAGE_META[page].path === path);\n return match ?? \"connections\";\n }\n function info(message2) {\n return { message: message2, tone: \"info\" };\n }\n function failure(message2) {\n return { message: message2, tone: \"error\" };\n }\n function initialState(page) {\n return {\n page,\n generation: 0,\n session: \"loading\",\n gate: null,\n refreshing: false,\n pendingFocus: null,\n ...identityScopedState()\n };\n }\n function identityScopedState() {\n return {\n data: null,\n connectorFilter: \"\",\n oauthNotice: null,\n oauthBusy: null,\n credentialNotice: null,\n credentialEditing: null,\n credentialBusy: null,\n tokenPhase: \"idle\",\n tokenNotice: null,\n tokens: [],\n createdToken: null,\n tokenRenaming: null,\n tokenBusy: false,\n activityPhase: \"idle\",\n activityNotice: null,\n activityEvents: [],\n activityCursor: null,\n activitySearch: \"\"\n };\n }\n function resetIdentity(state2, gate2 = null) {\n return {\n ...state2,\n generation: state2.generation + 1,\n session: \"gated\",\n gate: gate2,\n refreshing: false,\n pendingFocus: null,\n ...identityScopedState()\n };\n }\n function withPage(state2, page) {\n return {\n ...state2,\n page,\n createdToken: null,\n tokenRenaming: null,\n tokenNotice: null,\n credentialEditing: null,\n credentialNotice: null\n };\n }\n function credentialUnavailableCopy(capability) {\n if (capability === \"no_slots\") {\n return \"No connectors declare operator-managed credential slots. Connector credentials remain configuration-as-code until a slot is declared.\";\n }\n if (capability === \"vault_not_configured\") {\n return \"Credential storage is not configured. Set credentials.encryptionKey before managing connector credentials here.\";\n }\n return \"Credential management requires an eligible Clerk operator. Bearer-authenticated sessions can inspect connections but cannot manage stored credentials.\";\n }\n function accessTokenUnavailableCopy(capability) {\n if (capability === \"not_configured\") {\n return \"Access tokens are not configured for this deployment. Add accessTokens to the deployment configuration to enable them.\";\n }\n return \"Access token management requires an eligible Clerk operator. A Bearer token can connect to MCP, but it cannot create or revoke other tokens.\";\n }\n function connectorStatusLabel(status) {\n if (status === \"ok\") return \"Connected\";\n if (status === \"auth_required\") return \"Authorization needed\";\n return \"Unavailable\";\n }\n function toolCountLabel(count) {\n return `${count} ${count === 1 ? \"tool\" : \"tools\"}`;\n }\n var DRIFT_CATEGORIES = [\n { key: \"unclassifiedTools\", label: \"Unclassified\" },\n { key: \"unservedTools\", label: \"Unserved\" },\n { key: \"annotationConflicts\", label: \"Annotation conflicts\" },\n { key: \"schemaChanges\", label: \"Schema changes\" }\n ];\n function driftTotal(drift) {\n if (!drift) return 0;\n return DRIFT_CATEGORIES.reduce((sum, { key }) => sum + (drift[key] || 0), 0);\n }\n function driftState(drift) {\n if (!drift) return \"unavailable\";\n return driftTotal(drift) > 0 ? \"warning\" : \"clean\";\n }\n function driftCounts(drift) {\n if (!drift) return [];\n return DRIFT_CATEGORIES.map(({ key, label }) => ({\n key,\n label,\n count: drift[key] || 0\n }));\n }\n function driftSummary(drift) {\n const state2 = driftState(drift);\n if (state2 === \"unavailable\") {\n return \"No catalog refresh observed yet in this runtime.\";\n }\n const observed = formatDate(drift?.observedAt);\n const when = observed ? ` · observed ${observed}` : \"\";\n if (state2 === \"clean\") return `Matches the reviewed manifest${when}`;\n const total = driftTotal(drift);\n return `${total} difference${total === 1 ? \"\" : \"s\"} from the reviewed manifest${when}`;\n }\n function safeHttpHref(url) {\n if (!url) return null;\n try {\n const protocol = new URL(url).protocol;\n return protocol === \"http:\" || protocol === \"https:\" ? url : null;\n } catch {\n return null;\n }\n }\n function formatDate(value) {\n if (!value) return \"\";\n const date = new Date(value);\n return Number.isNaN(date.valueOf()) ? \"\" : date.toLocaleString();\n }\n function actorLabel(actor) {\n if (!actor?.kind) return \"unknown\";\n if (actor.label) return `${actor.kind} · ${actor.label}`;\n return actor.id ? `${actor.kind} · ${actor.id}` : actor.kind;\n }\n function actorStableId(actor) {\n if (!actor?.id) return null;\n if (!actor.label && !actor.namespace) return null;\n return actor.namespace ? `${actor.namespace} · ${actor.id}` : actor.id;\n }\n function activityMatches(event, query) {\n const q2 = query.trim().toLowerCase();\n if (!q2) return true;\n return [\n event.address,\n event.connectorId,\n event.toolName,\n event.source,\n event.outcome,\n event.errorCode,\n event.friction,\n event.actor?.kind,\n event.actor?.id,\n event.actor?.namespace,\n event.actor?.label\n ].some((value) => String(value ?? \"\").toLowerCase().includes(q2));\n }\n function filterActivity(events, query) {\n return events.filter((event) => activityMatches(event, query));\n }\n function activitySummary(events) {\n if (events.length === 0) return \"Arguments and results are never stored.\";\n const tools = new Set(events.map((event) => event.address)).size;\n return `${events.length} loaded call${events.length === 1 ? \"\" : \"s\"} · ${tools} tool${tools === 1 ? \"\" : \"s\"} · no arguments or results stored`;\n }\n var ACTIVITY_OUTCOMES = [\"success\", \"error\", \"timeout\", \"cancelled\"];\n function activityOutcomeClass(outcome) {\n return ACTIVITY_OUTCOMES.includes(outcome) ? outcome : \"error\";\n }\n function activityDetail(event) {\n const parts = [event.source];\n if (event.attempts > 1) parts.push(`${event.attempts} attempts`);\n if (event.friction) parts.push(event.friction);\n if (event.errorCode && event.errorCode !== event.friction) {\n parts.push(event.errorCode);\n }\n return parts.join(\" · \");\n }\n function credentialStateLabel(credential) {\n if (!credential.configured) return \"not configured\";\n const masked = credential.fields?.length ? \"configured\" : `configured · ••••${credential.lastFour ?? \"\"}`;\n return credential.updatedAt ? `${masked} · updated ${formatDate(credential.updatedAt)}` : masked;\n }\n function gateCopy(kind, signedIn) {\n if (kind !== \"clerk\") {\n return \"Paste an operator bearer token to open this page. Nothing is requested until you do.\";\n }\n return signedIn ? \"Signed in with Clerk, but this account cannot open deployment-wide operator pages.\" : \"Sign in with Clerk to open this operator page.\";\n }\n\n // src/operator-ui/app/config.ts\n var auth = AUTH;\n var mcpUrl = MCP_URL;\n var initialPage = INITIAL_PAGE;\n var titleSuffix = TITLE_SUFFIX;\n var productName = PRODUCT_NAME;\n var productDescription = PRODUCT_DESCRIPTION;\n var productOperatorLabel = PRODUCT_OPERATOR_LABEL;\n var TOKEN_KEY = \"connecta:token\";\n\n // src/operator-ui/app/store.ts\n var state = initialState(initialPage);\n var listeners = /* @__PURE__ */ new Set();\n function getState() {\n return state;\n }\n function subscribe(listener) {\n listeners.add(listener);\n return () => listeners.delete(listener);\n }\n function set(patch) {\n state = { ...state, ...patch };\n for (const listener of listeners) listener();\n }\n function fence() {\n const generation = state.generation;\n return () => generation === state.generation;\n }\n function message(error, fallback) {\n return error instanceof Error && error.message ? error.message : fallback;\n }\n function sessionToken() {\n return auth.kind === \"clerk\" ? Promise.resolve(window.Clerk?.session?.getToken() ?? null) : Promise.resolve(localStorage.getItem(TOKEN_KEY));\n }\n function gate(notice = null) {\n state = resetIdentity(state, notice);\n for (const listener of listeners) listener();\n }\n async function operatorRequest(path, method, current, body) {\n const token = await sessionToken();\n if (!current()) throw new Error(\"The operator session changed.\");\n if (!token) throw new Error(\"Your operator session has expired.\");\n const res = await fetch(path, {\n method,\n headers: {\n Authorization: `Bearer ${token}`,\n ...body ? { \"Content-Type\": \"application/json\" } : {}\n },\n ...body ? { body: JSON.stringify(body) } : {}\n });\n if (res.status === 204) return null;\n let payload = {};\n try {\n payload = await res.json();\n } catch {\n }\n if (res.status === 401) {\n throw new Error(\"Your operator session was not accepted. Sign in again.\");\n }\n if (res.status === 403) {\n throw new Error(\"This identity may not perform that action.\");\n }\n if (!res.ok) {\n throw new Error(payload.error || `Request failed (${res.status}).`);\n }\n return payload;\n }\n async function loadData() {\n const current = fence();\n if (state.session === \"ready\") set({ refreshing: true });\n let token;\n try {\n token = await sessionToken();\n } catch (error) {\n if (!current()) return;\n const why = message(error, \"unknown error\");\n return gate(failure(`Could not read the Clerk session: ${why}`));\n }\n if (!current()) return;\n if (!token) return gate(null);\n let res;\n try {\n res = await fetch(\"/ui/data\", {\n headers: { Authorization: `Bearer ${token}` }\n });\n } catch (error) {\n if (!current()) return;\n return gate(failure(`Network error: ${message(error, \"unknown error\")}`));\n }\n if (!current()) return;\n if (res.status === 401 || res.status === 403) {\n if (auth.kind === \"clerk\") {\n return gate(\n failure(\n res.status === 403 ? \"This Clerk account is not allowed to access connecta.\" : \"Your Clerk session was not accepted. Sign out and try again.\"\n )\n );\n }\n localStorage.removeItem(TOKEN_KEY);\n return gate(failure(\"Token rejected — enter a valid bearer token.\"));\n }\n if (!res.ok) return gate(failure(`Error ${res.status}`));\n let data;\n try {\n data = await res.json();\n } catch {\n if (!current()) return;\n return gate(failure(\"Operator data could not be read.\"));\n }\n if (!current()) return;\n set({ data, session: \"ready\", gate: null, refreshing: false });\n }\n async function mutate(options) {\n const current = fence();\n set(options.busy);\n try {\n const payload = await options.request(current);\n if (!current()) return;\n if (options.reload) await loadData();\n if (!current()) return;\n set(options.done(payload));\n } catch (error) {\n if (!current()) return;\n if (options.reload) {\n try {\n await loadData();\n } catch {\n }\n if (!current()) return;\n }\n set(options.failed(failure(message(error, options.fallback))));\n }\n }\n function focusHandled() {\n if (state.pendingFocus !== null) set({ pendingFocus: null });\n }\n function setPage(page, focus = false) {\n state = withPage(state, page);\n if (focus) {\n state = {\n ...state,\n pendingFocus: state.session === \"ready\" ? `${page}Heading` : \"gateHeading\"\n };\n }\n for (const listener of listeners) listener();\n }\n function navigate(page, href) {\n history.pushState({ operatorPage: page }, \"\", href);\n setPage(page, true);\n }\n function setConnectorFilter(connectorFilter) {\n set({ connectorFilter });\n }\n function setActivitySearch(activitySearch) {\n set({ activitySearch });\n }\n function signInWithBearer(value) {\n gate(null);\n localStorage.setItem(TOKEN_KEY, value);\n void loadData().then(() => {\n if (state.session === \"ready\") set({ pendingFocus: `${state.page}Heading` });\n });\n }\n function forgetBearer() {\n localStorage.removeItem(TOKEN_KEY);\n gate(null);\n set({ pendingFocus: \"token\" });\n }\n function oauthAction(connector, action) {\n const disconnecting = action === \"disconnect\";\n const confirmed = window.confirm(\n disconnecting ? `Disconnect OAuth for ${connector}? Stored credentials and any pending authorization will be removed.` : `Restart OAuth for ${connector}? Stored credentials and any pending authorization will be replaced.`\n );\n if (!confirmed) return Promise.resolve();\n return mutate({\n request: (current) => operatorRequest(\n `/ui/oauth/${encodeURIComponent(connector)}`,\n disconnecting ? \"DELETE\" : \"POST\",\n current\n ),\n busy: { oauthNotice: null, oauthBusy: connector },\n done: (payload) => ({\n oauthBusy: null,\n pendingFocus: \"oauthNotice\",\n oauthNotice: info(\n disconnecting ? \"OAuth disconnected. Restart authorization when you are ready to reconnect.\" : payload?.message || \"Authorization restarted. Open the authorization link to reconnect.\"\n )\n }),\n failed: (notice) => ({\n oauthBusy: null,\n oauthNotice: notice,\n pendingFocus: \"oauthNotice\"\n }),\n fallback: \"OAuth action failed.\",\n reload: true\n });\n }\n function editCredential(connector) {\n set({ credentialEditing: connector, credentialNotice: null });\n }\n function refuseCredential(copy) {\n set({ credentialNotice: failure(copy), pendingFocus: \"credentialNotice\" });\n }\n function credentialMutation(connector, request, done, reload = true) {\n const land = (credentialNotice) => ({\n credentialBusy: null,\n credentialNotice,\n pendingFocus: \"credentialNotice\"\n });\n return mutate({\n request,\n busy: { credentialBusy: connector, credentialNotice: null },\n done: (payload) => land(done(payload)),\n failed: land,\n fallback: \"Credential action failed.\",\n reload\n });\n }\n function saveCredential(connector, body) {\n return credentialMutation(\n connector,\n (current) => operatorRequest(\n `/ui/credentials/${encodeURIComponent(connector)}`,\n \"PUT\",\n current,\n body\n ),\n () => {\n set({ credentialEditing: null });\n return info(\"Credential saved.\");\n }\n );\n }\n function removeCredential(connector) {\n const confirmed = window.confirm(\n \"Remove this credential? The connector will stop authenticating until a replacement is added.\"\n );\n if (!confirmed) return Promise.resolve();\n return credentialMutation(\n connector,\n (current) => operatorRequest(\n `/ui/credentials/${encodeURIComponent(connector)}`,\n \"DELETE\",\n current\n ),\n () => info(\"Credential removed.\")\n );\n }\n function testCredential(connector) {\n return credentialMutation(\n connector,\n (current) => operatorRequest(\n `/ui/credentials/${encodeURIComponent(connector)}/test`,\n \"POST\",\n current\n ),\n (payload) => {\n const copy = payload?.message || (payload?.ok ? \"Credential is valid.\" : \"Credential test failed.\");\n return payload?.ok ? info(copy) : failure(copy);\n },\n false\n );\n }\n async function loadAccessTokens() {\n const current = fence();\n set({ tokenPhase: \"loading\", tokenNotice: null });\n try {\n const payload = await operatorRequest(\"/ui/access-tokens\", \"GET\", current);\n if (!current()) return;\n set({ tokenPhase: \"ready\", tokens: payload?.accessTokens ?? [] });\n } catch (error) {\n if (!current()) return;\n set({\n tokenPhase: \"error\",\n tokenNotice: failure(\n message(error, \"Access tokens could not be loaded.\")\n )\n });\n }\n }\n function tokenFailure(tokenNotice) {\n return { tokenBusy: false, tokenNotice, pendingFocus: \"tokenNotice\" };\n }\n function createAccessToken(name) {\n if (!name) {\n set(tokenFailure(failure(\"Name the MCP client before creating a token.\")));\n return Promise.resolve(false);\n }\n let created = false;\n return mutate({\n request: (current) => operatorRequest(\"/ui/access-tokens\", \"POST\", current, { name }),\n busy: { tokenBusy: true, tokenNotice: null },\n done: (payload) => {\n const issued = payload?.accessToken;\n if (!payload?.token || !issued) {\n throw new Error(\"The created token was not returned.\");\n }\n created = true;\n return {\n tokenBusy: false,\n tokenPhase: \"ready\",\n tokens: [\n issued,\n ...state.tokens.filter((token) => token.id !== issued.id)\n ],\n createdToken: payload.token,\n tokenNotice: info(\"Access token created.\"),\n pendingFocus: \"tokenRevealHeading\"\n };\n },\n failed: tokenFailure,\n fallback: \"Access token could not be created.\"\n }).then(() => created);\n }\n function dismissCreatedToken() {\n set({ createdToken: null });\n }\n function renameAccessToken(id) {\n set({ tokenRenaming: id });\n }\n function accessTokenMutation(id, method, body, success, fallback) {\n return mutate({\n request: (current) => operatorRequest(\n `/ui/access-tokens/${encodeURIComponent(id)}`,\n method,\n current,\n body\n ),\n busy: { tokenBusy: true, tokenNotice: null },\n done: (payload) => ({\n tokenBusy: false,\n tokenRenaming: null,\n tokenNotice: info(success),\n pendingFocus: \"tokenNotice\",\n ...payload?.accessToken ? {\n tokens: state.tokens.map(\n (token) => token.id === id ? payload.accessToken : token\n )\n } : {}\n }),\n failed: tokenFailure,\n fallback\n });\n }\n function saveAccessTokenName(id, name) {\n return accessTokenMutation(\n id,\n \"PUT\",\n { name },\n \"Access token renamed.\",\n \"Access token could not be renamed.\"\n );\n }\n function revokeAccessToken(id) {\n const named = state.tokens.find((token) => token.id === id);\n const confirmed = window.confirm(\n `Revoke ${named?.name || \"this access token\"}? Its MCP client will immediately lose access.`\n );\n if (!confirmed) return Promise.resolve();\n return accessTokenMutation(\n id,\n \"DELETE\",\n void 0,\n \"Access token revoked.\",\n \"Access token could not be revoked.\"\n );\n }\n async function loadActivity(reset) {\n if (!state.data?.activityEnabled) return;\n const current = fence();\n set({\n activityPhase: \"loading\",\n activityNotice: null,\n ...reset ? { activityEvents: [], activityCursor: null } : {}\n });\n const params = new URLSearchParams({ limit: \"50\" });\n if (!reset && state.activityCursor) {\n params.set(\"cursor\", state.activityCursor);\n }\n try {\n const payload = await operatorRequest(\n `/ui/activity?${params}`,\n \"GET\",\n current\n );\n if (!current()) return;\n set({\n activityPhase: \"ready\",\n activityEvents: [\n ...reset ? [] : state.activityEvents,\n ...payload?.events ?? []\n ],\n activityCursor: payload?.nextCursor ?? null\n });\n } catch (error) {\n if (!current()) return;\n set({\n activityPhase: \"error\",\n activityNotice: failure(message(error, \"Activity could not be loaded.\"))\n });\n }\n }\n function signIn() {\n window.Clerk?.redirectToSignIn({\n signInFallbackRedirectUrl: window.location.href,\n signUpFallbackRedirectUrl: window.location.href\n });\n }\n function signOut() {\n const clerk = window.Clerk;\n gate(null);\n void clerk?.signOut({ redirectUrl: window.location.href });\n }\n async function boot() {\n const onPop = () => setPage(pageForPath(window.location.pathname), true);\n window.addEventListener(\"popstate\", onPop);\n window.addEventListener(\"pagehide\", dismissCreatedToken);\n if (auth.kind === \"clerk\") {\n const clerk = window.Clerk;\n if (!clerk) {\n const why = \"Clerk could not load. Check your network and try again.\";\n return gate(failure(why));\n }\n try {\n await clerk.load({\n ...auth.signInUrl ? { signInUrl: auth.signInUrl } : {},\n ...auth.signUpUrl ? { signUpUrl: auth.signUpUrl } : {},\n signInFallbackRedirectUrl: window.location.href,\n signUpFallbackRedirectUrl: window.location.href,\n afterSignOutUrl: window.location.href\n });\n let sessionId = clerk.session?.id ?? null;\n clerk.addListener((resources) => {\n const next = resources.session?.id ?? null;\n if (next === sessionId) return;\n sessionId = next;\n gate(null);\n void loadData();\n });\n } catch (error) {\n const why = message(error, \"unknown error\");\n return gate(failure(`Clerk could not initialize: ${why}`));\n }\n }\n await loadData();\n }\n\n // node_modules/preact/jsx-runtime/dist/jsxRuntime.module.js\n var f3 = 0;\n function u3(e3, t3, n2, o3, i3, u4) {\n t3 || (t3 = {});\n var a3, c3, p3 = t3;\n if (\"ref\" in p3) for (c3 in p3 = {}, t3) \"ref\" == c3 ? a3 = t3[c3] : p3[c3] = t3[c3];\n var l3 = { type: e3, props: p3, key: n2, ref: a3, __k: null, __: null, __b: 0, __e: null, __c: null, constructor: void 0, __v: --f3, __i: -1, __u: 0, __source: i3, __self: u4 };\n if (\"function\" == typeof e3 && (a3 = e3.defaultProps)) for (c3 in a3) void 0 === p3[c3] && (p3[c3] = a3[c3]);\n return l.vnode && l.vnode(l3), l3;\n }\n\n // src/operator-ui/app/parts.tsx\n function NoticeLine({\n id,\n notice,\n className = \"meta\"\n }) {\n return /* @__PURE__ */ u3(\n \"p\",\n {\n id,\n class: notice?.tone === \"error\" ? `${className} error-notice` : className,\n role: notice?.tone === \"error\" ? \"alert\" : \"status\",\n \"aria-live\": \"polite\",\n tabIndex: -1,\n children: notice ? notice.message : null\n }\n );\n }\n function Empty({ children }) {\n return /* @__PURE__ */ u3(\"p\", { class: \"empty\", children });\n }\n function Unavailable({ children }) {\n return /* @__PURE__ */ u3(\"div\", { class: \"unavailable\", children });\n }\n function PageLink({\n page,\n class: className,\n current,\n children\n }) {\n const href = PAGE_META[page].path;\n return /* @__PURE__ */ u3(\n \"a\",\n {\n class: className,\n href,\n ...current ? { \"aria-current\": \"page\" } : {},\n onClick: (event) => {\n if (event.defaultPrevented || event.button !== 0 || event.metaKey || event.ctrlKey || event.shiftKey || event.altKey) {\n return;\n }\n event.preventDefault();\n navigate(page, href);\n },\n children\n }\n );\n }\n function CopyButton({\n value,\n label,\n class: className = \"linklike\"\n }) {\n const [status, setStatus] = d2(\"idle\");\n h2(() => {\n if (status === \"idle\") return;\n const timer = window.setTimeout(() => setStatus(\"idle\"), 1600);\n return () => window.clearTimeout(timer);\n }, [status]);\n return /* @__PURE__ */ u3(\n \"button\",\n {\n class: className,\n type: \"button\",\n onClick: () => {\n navigator.clipboard.writeText(value).then(\n () => setStatus(\"copied\"),\n () => setStatus(\"failed\")\n );\n },\n children: status === \"copied\" ? \"Copied\" : status === \"failed\" ? \"Copy failed\" : label\n }\n );\n }\n\n // src/operator-ui/app/activity.tsx\n function ActivityRow({ event }) {\n const outcome = activityOutcomeClass(event.outcome);\n const stableId = actorStableId(event.actor);\n return /* @__PURE__ */ u3(\"article\", { class: `activity-item ${outcome}`, children: [\n /* @__PURE__ */ u3(\"div\", { class: \"activity-stamp\", children: [\n /* @__PURE__ */ u3(\n \"span\",\n {\n class: outcome === \"success\" ? \"dot ok\" : \"dot\",\n \"aria-hidden\": \"true\"\n }\n ),\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"time\", { class: \"activity-time\", dateTime: event.occurredAt, children: formatDate(event.occurredAt) }),\n /* @__PURE__ */ u3(\"div\", { class: \"activity-actor\", children: actorLabel(event.actor) }),\n stableId ? /* @__PURE__ */ u3(\"div\", { class: \"activity-actor-id mono\", children: stableId }) : null\n ] })\n ] }),\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"div\", { class: \"activity-address\", children: event.address }),\n /* @__PURE__ */ u3(\"div\", { class: \"activity-detail\", children: activityDetail(event) })\n ] }),\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"div\", { class: \"activity-outcome\", children: event.outcome }),\n /* @__PURE__ */ u3(\"div\", { class: \"activity-detail\", children: [\n event.durationMs,\n \" ms\"\n ] })\n ] })\n ] });\n }\n function ActivityPage({ state: state2 }) {\n const enabled = Boolean(state2.data?.activityEnabled);\n const loading = state2.activityPhase === \"loading\";\n const visible = filterActivity(state2.activityEvents, state2.activitySearch);\n return /* @__PURE__ */ u3(\"section\", { id: \"activityView\", children: /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"activityHeading\", class: \"pcap\", tabIndex: -1, children: \"Activity\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody\", children: [\n /* @__PURE__ */ u3(\"p\", { class: \"activity-copy\", id: \"activitySummary\", children: activitySummary(state2.activityEvents) }),\n !enabled ? /* @__PURE__ */ u3(Unavailable, { children: [\n \"Activity history is not configured. Add an\",\n \" \",\n /* @__PURE__ */ u3(\"span\", { class: \"mono\", children: \"activity.store\" }),\n \" with a list reader to enable this page.\"\n ] }) : /* @__PURE__ */ u3(\"div\", { id: \"activityAvailable\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"row activity-controls\", children: [\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: \"activitySearch\",\n type: \"search\",\n placeholder: \"Search user, tool, or outcome…\",\n \"aria-label\": \"Search loaded activity\",\n value: state2.activitySearch,\n onInput: (event) => setActivitySearch(event.currentTarget.value)\n }\n ),\n /* @__PURE__ */ u3(\n \"button\",\n {\n id: \"refreshActivity\",\n class: \"linklike\",\n type: \"button\",\n disabled: loading,\n onClick: () => void loadActivity(true),\n children: loading ? \"Loading…\" : \"Refresh\"\n }\n )\n ] }),\n /* @__PURE__ */ u3(NoticeLine, { id: \"activityNotice\", notice: state2.activityNotice }),\n /* @__PURE__ */ u3(\n \"div\",\n {\n id: \"activityList\",\n class: \"activity-ledger\",\n \"aria-busy\": loading ? \"true\" : \"false\",\n children: loading && state2.activityEvents.length === 0 ? /* @__PURE__ */ u3(\"div\", { class: \"activity-empty\", children: \"Loading activity…\" }) : state2.activityPhase === \"error\" && state2.activityEvents.length === 0 ? /* @__PURE__ */ u3(\"p\", { class: \"activity-empty\", children: /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n onClick: () => void loadActivity(true),\n children: \"Try loading activity again\"\n }\n ) }) : visible.length === 0 ? /* @__PURE__ */ u3(\"div\", { class: \"activity-empty\", children: state2.activitySearch.trim() ? \"No loaded activity matches this search.\" : \"No connector tool calls recorded yet.\" }) : visible.map((event, index) => /* @__PURE__ */ u3(\n ActivityRow,\n {\n event\n },\n `${event.occurredAt}-${event.address}-${index}`\n ))\n }\n ),\n state2.activityCursor ? /* @__PURE__ */ u3(\n \"button\",\n {\n id: \"moreActivity\",\n class: \"linklike activity-more\",\n type: \"button\",\n disabled: loading,\n onClick: () => void loadActivity(false),\n children: loading ? \"Loading…\" : \"Load older\"\n }\n ) : null\n ] })\n ] })\n ] }) });\n }\n\n // src/operator-ui/model.ts\n function filterUiConnectors(connectors, query) {\n const q2 = query.trim().toLowerCase();\n const filtered = [];\n for (const connector of connectors) {\n const connectorText = [\n connector.id,\n connector.title,\n connector.description,\n connector.status\n ].join(\" \").toLowerCase();\n const connectorMatches = Boolean(q2 && connectorText.includes(q2));\n const tools = connector.tools.filter(\n (tool) => !q2 || connectorMatches || `${tool.name} ${tool.description ?? \"\"}`.toLowerCase().includes(q2)\n );\n if (q2 && tools.length === 0 && !connectorMatches) continue;\n filtered.push({ connector, tools });\n }\n return filtered;\n }\n\n // src/operator-ui/app/connections.tsx\n var DRIFT_HEADING = {\n clean: \"Catalog drift · none\",\n warning: \"Catalog drift · review\",\n unavailable: \"Catalog drift · not observed\"\n };\n function DriftPanel({ connector }) {\n const drift = connector.catalogDrift;\n const state2 = driftState(drift);\n return /* @__PURE__ */ u3(\n \"div\",\n {\n id: `drift-${connector.id}`,\n class: `connector-drift ${state2}`,\n \"data-drift\": state2,\n children: [\n /* @__PURE__ */ u3(\"p\", { class: \"cap\", children: DRIFT_HEADING[state2] }),\n /* @__PURE__ */ u3(\"p\", { class: \"meta drift-summary\", children: driftSummary(drift) }),\n state2 === \"unavailable\" ? null : /* @__PURE__ */ u3(\"ul\", { class: \"drift-counts\", children: driftCounts(drift).map(({ key, label, count }) => /* @__PURE__ */ u3(\"li\", { class: count > 0 ? \"drift-count flagged\" : \"drift-count\", children: [\n /* @__PURE__ */ u3(\"span\", { class: \"drift-count-value\", children: count }),\n /* @__PURE__ */ u3(\"span\", { class: \"drift-count-label\", children: label })\n ] }, key)) })\n ]\n }\n );\n }\n function ConnectorCard({\n connector,\n tools,\n expanded,\n oauthManagement,\n busy\n }) {\n const name = connector.title || connector.id;\n const authorization = safeHttpHref(connector.authorizationUrl);\n return /* @__PURE__ */ u3(\"div\", { class: \"card\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"connector-head\", children: [\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"div\", { class: \"connector-title\", children: [\n /* @__PURE__ */ u3(\"span\", { class: `dot ${connector.status}`, \"aria-hidden\": \"true\" }),\n /* @__PURE__ */ u3(\"h2\", { children: name })\n ] }),\n connector.description ? /* @__PURE__ */ u3(\"p\", { class: \"connector-description meta\", children: connector.description }) : null\n ] }),\n /* @__PURE__ */ u3(\"div\", { class: \"connector-state cap\", children: [\n connectorStatusLabel(connector.status),\n \" ·\",\n \" \",\n toolCountLabel(connector.toolCount),\n /* @__PURE__ */ u3(\"br\", {}),\n /* @__PURE__ */ u3(\"span\", { class: \"mono\", children: connector.id })\n ] })\n ] }),\n connector.message ? /* @__PURE__ */ u3(\"p\", { class: \"connector-message msg\", children: connector.message }) : null,\n connector.authorizationUrl ? /* @__PURE__ */ u3(\"p\", { class: authorization ? \"connector-auth\" : \"connector-auth meta\", children: authorization ? /* @__PURE__ */ u3(\n \"a\",\n {\n class: \"linklike\",\n href: authorization,\n target: \"_blank\",\n rel: \"noopener\",\n children: \"Authorize connector →\"\n }\n ) : `Authorization URL: ${connector.authorizationUrl}` }) : null,\n /* @__PURE__ */ u3(DriftPanel, { connector }),\n connector.catalogAccess ? /* @__PURE__ */ u3(\"p\", { class: \"meta\", children: [\n \"Last agent catalog read · \",\n connector.catalogAccess.state,\n \" ·\",\n \" \",\n new Date(connector.catalogAccess.observedAt).toLocaleString()\n ] }) : null,\n connector.oauth && oauthManagement ? /* @__PURE__ */ u3(\"div\", { class: \"credential-actions\", children: [\n /* @__PURE__ */ u3(\n \"button\",\n {\n type: \"button\",\n class: \"linklike danger\",\n \"aria-label\": `Disconnect OAuth for ${name}`,\n disabled: busy,\n onClick: () => void oauthAction(connector.id, \"disconnect\"),\n children: \"Disconnect OAuth\"\n }\n ),\n /* @__PURE__ */ u3(\n \"button\",\n {\n type: \"button\",\n class: \"linklike\",\n \"aria-label\": `${connector.status === \"ok\" ? \"Reconnect OAuth for\" : \"Restart authorization for\"} ${name}`,\n disabled: busy,\n onClick: () => void oauthAction(connector.id, \"reconnect\"),\n children: connector.status === \"ok\" ? \"Reconnect OAuth\" : \"Restart authorization\"\n }\n )\n ] }) : null,\n connector.credential ? /* @__PURE__ */ u3(\"p\", { class: \"connector-auth\", children: /* @__PURE__ */ u3(PageLink, { page: \"credentials\", class: \"linklike\", children: \"Manage credential →\" }) }) : null,\n tools.length ? /* @__PURE__ */ u3(\"details\", { open: expanded, children: [\n /* @__PURE__ */ u3(\"summary\", { class: \"linklike\", children: [\n \"Show tools (\",\n tools.length,\n \")\"\n ] }),\n /* @__PURE__ */ u3(\"div\", { class: \"tool-list\", children: tools.map((tool) => /* @__PURE__ */ u3(\"div\", { class: \"tool\", children: [\n /* @__PURE__ */ u3(\"code\", { children: tool.address }),\n tool.description ? /* @__PURE__ */ u3(\"span\", { class: \"td\", children: tool.description }) : null\n ] }, tool.address)) })\n ] }) : null\n ] });\n }\n function ConnectionsPage({ state: state2 }) {\n const data = state2.data;\n const query = state2.connectorFilter.trim();\n const filtered = data ? filterUiConnectors(data.connectors, query) : [];\n return /* @__PURE__ */ u3(\"section\", { id: \"connectionsView\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"connectionsHeading\", class: \"pcap\", tabIndex: -1, children: \"Connections\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody lead-copy\", children: [\n /* @__PURE__ */ u3(\"p\", { children: \"Use this endpoint to give an MCP client access to the tools below.\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"endpoint\", children: /* @__PURE__ */ u3(\"div\", { class: \"endpoint-row\", children: [\n /* @__PURE__ */ u3(\"code\", { id: \"mcpUrl\", class: \"mono\", children: mcpUrl }),\n /* @__PURE__ */ u3(CopyButton, { value: mcpUrl, label: \"Copy URL\" })\n ] }) }),\n /* @__PURE__ */ u3(\"p\", { class: \"cap\", id: \"serverInfo\", children: data ? `${data.serverInfo?.name || productName} v${data.connectaVersion || \"?\"}` : productOperatorLabel }),\n /* @__PURE__ */ u3(NoticeLine, { id: \"oauthNotice\", notice: state2.oauthNotice })\n ] })\n ] }),\n /* @__PURE__ */ u3(\"section\", { class: \"section pgrid\", \"aria-labelledby\": \"connectorLedgerHeading\", children: [\n /* @__PURE__ */ u3(\"h2\", { class: \"pcap\", id: \"connectorLedgerHeading\", children: \"Connectors\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"row toolbar\", children: /* @__PURE__ */ u3(\n \"input\",\n {\n id: \"filter\",\n type: \"search\",\n placeholder: \"Filter connectors or tools…\",\n \"aria-label\": \"Filter connectors or tools\",\n value: state2.connectorFilter,\n onInput: (event) => setConnectorFilter(event.currentTarget.value)\n }\n ) }),\n /* @__PURE__ */ u3(\n \"div\",\n {\n id: \"list\",\n class: \"connector-tools\",\n \"aria-busy\": state2.refreshing || !data ? \"true\" : \"false\",\n children: !data ? /* @__PURE__ */ u3(Empty, { children: \"Loading connectors…\" }) : filtered.length === 0 ? /* @__PURE__ */ u3(Empty, { children: query ? \"No connectors or tools match this filter.\" : \"No connectors are declared in this deployment.\" }) : filtered.map(({ connector, tools }) => /* @__PURE__ */ u3(\n ConnectorCard,\n {\n connector,\n tools,\n expanded: Boolean(query),\n oauthManagement: data.oauthManagement,\n busy: state2.oauthBusy === connector.id\n },\n connector.id\n ))\n }\n )\n ] })\n ] })\n ] });\n }\n\n // src/operator-ui/app/credentials.tsx\n function CredentialForm({\n connector,\n credential,\n busy\n }) {\n const fields = credential.fields ?? [];\n const [values, setValues] = d2({});\n const single = fields.length === 0;\n const inputId = `credential-input-${connector}`;\n const submit = () => {\n if (single) {\n const value = (values.value ?? \"\").trim();\n if (!value) return refuseCredential(\"Paste a credential before saving.\");\n return void saveCredential(connector, { value });\n }\n const entries = {};\n for (const field of fields) {\n const value = (values[field.name] ?? \"\").trim();\n if (!value) {\n return refuseCredential(\n \"Complete every credential field before saving.\"\n );\n }\n entries[field.name] = value;\n }\n void saveCredential(connector, { values: entries });\n };\n return /* @__PURE__ */ u3(\"div\", { class: \"credential-form\", \"data-credential-form\": connector, children: [\n single ? /* @__PURE__ */ u3(S, { children: [\n /* @__PURE__ */ u3(\"label\", { class: \"visually-hidden\", for: inputId, children: credential.label }),\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: inputId,\n type: \"password\",\n \"aria-label\": credential.label,\n placeholder: credential.placeholder || \"Paste credential\",\n autocomplete: \"new-password\",\n autocapitalize: \"none\",\n spellcheck: false,\n value: values.value ?? \"\",\n onInput: (event) => setValues({ value: event.currentTarget.value })\n }\n )\n ] }) : /* @__PURE__ */ u3(\"div\", { class: \"credential-fields\", children: fields.map((field, index) => {\n const id = `credential-input-${connector}-${index}`;\n return /* @__PURE__ */ u3(\"div\", { class: \"credential-field\", children: [\n /* @__PURE__ */ u3(\"label\", { for: id, children: field.label }),\n /* @__PURE__ */ u3(\n \"input\",\n {\n id,\n type: field.inputType || \"password\",\n placeholder: field.placeholder || field.label,\n autocomplete: (field.inputType ?? \"password\") === \"password\" ? \"new-password\" : \"off\",\n autocapitalize: \"none\",\n spellcheck: false,\n value: values[field.name] ?? \"\",\n onInput: (event) => setValues({\n ...values,\n [field.name]: event.currentTarget.value\n })\n }\n )\n ] }, field.name);\n }) }),\n /* @__PURE__ */ u3(\"button\", { class: \"linklike\", type: \"button\", disabled: busy, onClick: submit, children: busy ? \"Saving…\" : \"Save\" }),\n /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => editCredential(null),\n children: \"Cancel\"\n }\n )\n ] });\n }\n function CredentialCard({\n connector,\n credential,\n editing,\n busy\n }) {\n const configured = Boolean(credential.configured);\n const removable = configured || Boolean(credential.removable);\n return /* @__PURE__ */ u3(\n \"section\",\n {\n class: \"credential-card\",\n id: `credential-${connector.id}`,\n \"aria-labelledby\": `credential-title-${connector.id}`,\n children: [\n /* @__PURE__ */ u3(\"div\", { class: \"credential-head\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"connector-title\", children: [\n /* @__PURE__ */ u3(\n \"span\",\n {\n class: `dot ${configured ? \"ok\" : \"auth_required\"}`,\n \"aria-hidden\": \"true\"\n }\n ),\n /* @__PURE__ */ u3(\"h2\", { id: `credential-title-${connector.id}`, children: connector.title || connector.id })\n ] }),\n /* @__PURE__ */ u3(\"span\", { class: \"credential-state\", children: credentialStateLabel(credential) })\n ] }),\n /* @__PURE__ */ u3(\"p\", { class: \"mono\", children: [\n connector.id,\n \" · \",\n credential.label\n ] }),\n credential.description ? /* @__PURE__ */ u3(\"p\", { class: \"credential-copy meta\", children: credential.description }) : null,\n credential.fields?.length ? /* @__PURE__ */ u3(\"div\", { class: \"credential-field-summary\", children: credential.fields.map((field) => /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"span\", { children: field.label }),\n /* @__PURE__ */ u3(\"span\", { class: \"meta\", children: field.configured ? `configured · ••••${field.lastFour ?? \"\"}${field.updatedAt ? ` · updated ${formatDate(field.updatedAt)}` : \"\"}` : \"not configured\" })\n ] }, field.name)) }) : null,\n credential.error ? /* @__PURE__ */ u3(\"div\", { class: \"msg\", children: credential.error }) : null,\n credential.notice ? /* @__PURE__ */ u3(\"p\", { class: \"credential-copy meta\", children: credential.notice }) : null,\n /* @__PURE__ */ u3(\"div\", { class: \"credential-actions\", children: [\n /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => editCredential(editing ? null : connector.id),\n children: removable ? \"Replace\" : \"Add credential\"\n }\n ),\n configured && credential.testable ? /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => void testCredential(connector.id),\n children: busy ? \"Working…\" : \"Test\"\n }\n ) : null,\n removable ? /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike danger\",\n type: \"button\",\n disabled: busy,\n onClick: () => void removeCredential(connector.id),\n children: \"Remove\"\n }\n ) : null\n ] }),\n editing ? /* @__PURE__ */ u3(\n CredentialForm,\n {\n connector: connector.id,\n credential,\n busy\n }\n ) : null\n ]\n }\n );\n }\n function CredentialsPage({ state: state2 }) {\n const data = state2.data;\n const available = data?.credentialManagement === \"available\";\n const slots = (data?.connectors ?? []).filter(\n (connector) => Boolean(connector.credential)\n );\n return /* @__PURE__ */ u3(\"section\", { id: \"credentialsView\", children: /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"credentialsHeading\", class: \"pcap\", tabIndex: -1, children: \"Credentials\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody\", children: [\n /* @__PURE__ */ u3(\"p\", { class: \"activity-copy\", children: \"Rotate operator-managed connector credentials. Stored values are never returned or displayed.\" }),\n /* @__PURE__ */ u3(NoticeLine, { id: \"credentialNotice\", notice: state2.credentialNotice }),\n !available ? /* @__PURE__ */ u3(Unavailable, { children: credentialUnavailableCopy(data?.credentialManagement) }) : /* @__PURE__ */ u3(\n \"div\",\n {\n id: \"credentialList\",\n class: \"credential-ledger\",\n \"aria-busy\": state2.credentialBusy ? \"true\" : \"false\",\n children: slots.length === 0 ? /* @__PURE__ */ u3(Empty, { children: \"No connector in this deployment declares a credential slot yet.\" }) : slots.map((connector) => /* @__PURE__ */ u3(\n CredentialCard,\n {\n connector,\n credential: connector.credential,\n editing: state2.credentialEditing === connector.id,\n busy: state2.credentialBusy === connector.id\n },\n connector.id\n ))\n }\n )\n ] })\n ] }) });\n }\n\n // src/operator-ui/app/tokens.tsx\n function CreateForm({ busy }) {\n const [name, setName] = d2(\"\");\n return /* @__PURE__ */ u3(\n \"form\",\n {\n id: \"tokenCreateForm\",\n class: \"token-create\",\n onSubmit: (event) => {\n event.preventDefault();\n void createAccessToken(name.trim()).then((created) => {\n if (created) setName(\"\");\n });\n },\n children: [\n /* @__PURE__ */ u3(\"label\", { for: \"tokenName\", children: \"Client name\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"row\", children: [\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: \"tokenName\",\n type: \"text\",\n maxLength: 80,\n placeholder: \"Claude desktop, ChatGPT production…\",\n autocomplete: \"off\",\n value: name,\n onInput: (event) => setName(event.currentTarget.value)\n }\n ),\n /* @__PURE__ */ u3(\"button\", { id: \"createToken\", class: \"linklike\", type: \"submit\", disabled: busy, children: busy ? \"Creating…\" : \"Create token\" })\n ] })\n ]\n }\n );\n }\n function Reveal({ token }) {\n return /* @__PURE__ */ u3(\n \"section\",\n {\n id: \"tokenReveal\",\n class: \"token-reveal\",\n \"aria-labelledby\": \"tokenRevealHeading\",\n children: [\n /* @__PURE__ */ u3(\"div\", { class: \"token-reveal-head\", children: [\n /* @__PURE__ */ u3(\"h2\", { id: \"tokenRevealHeading\", tabIndex: -1, children: \"Copy this token now\" }),\n /* @__PURE__ */ u3(\"span\", { class: \"cap\", children: \"Shown once\" })\n ] }),\n /* @__PURE__ */ u3(\"p\", { class: \"meta\", children: \"Store it in the MCP client before leaving this page. It cannot be displayed again.\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"endpoint-row token-secret\", children: [\n /* @__PURE__ */ u3(\"code\", { id: \"createdToken\", class: \"mono\", children: token }),\n /* @__PURE__ */ u3(CopyButton, { value: token, label: \"Copy token\" })\n ] }),\n /* @__PURE__ */ u3(\"button\", { class: \"linklike\", type: \"button\", onClick: dismissCreatedToken, children: \"I stored it\" })\n ]\n }\n );\n }\n function TokenCard({\n token,\n renaming,\n busy\n }) {\n const [name, setName] = d2(token.name);\n const revoked = Boolean(token.revokedAt);\n return /* @__PURE__ */ u3(\n \"section\",\n {\n class: revoked ? \"token-card revoked\" : \"token-card\",\n \"aria-labelledby\": `access-token-${token.id}`,\n children: [\n /* @__PURE__ */ u3(\"div\", { class: \"token-card-head\", children: [\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"h2\", { id: `access-token-${token.id}`, children: token.name }),\n /* @__PURE__ */ u3(\"p\", { class: \"mono\", children: [\n token.tokenPrefix,\n \"…\"\n ] })\n ] }),\n /* @__PURE__ */ u3(\"div\", { class: \"cap\", children: revoked ? `Revoked ${formatDate(token.revokedAt)}` : `Created ${formatDate(token.createdAt)}` })\n ] }),\n /* @__PURE__ */ u3(\"div\", { class: \"credential-actions\", children: [\n /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => {\n setName(token.name);\n renameAccessToken(renaming ? null : token.id);\n },\n children: \"Rename\"\n }\n ),\n revoked ? null : /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike danger\",\n type: \"button\",\n disabled: busy,\n onClick: () => void revokeAccessToken(token.id),\n children: \"Revoke\"\n }\n )\n ] }),\n renaming ? /* @__PURE__ */ u3(\n \"form\",\n {\n class: \"credential-form\",\n onSubmit: (event) => {\n event.preventDefault();\n const next = name.trim();\n if (next) void saveAccessTokenName(token.id, next);\n },\n children: [\n /* @__PURE__ */ u3(\"label\", { class: \"visually-hidden\", for: `token-name-${token.id}`, children: \"Token name\" }),\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: `token-name-${token.id}`,\n type: \"text\",\n maxLength: 80,\n autocomplete: \"off\",\n value: name,\n onInput: (event) => setName(event.currentTarget.value)\n }\n ),\n /* @__PURE__ */ u3(\"button\", { class: \"linklike\", type: \"submit\", disabled: busy, children: \"Save name\" }),\n /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => renameAccessToken(null),\n children: \"Cancel\"\n }\n )\n ]\n }\n ) : null\n ]\n }\n );\n }\n function TokensPage({ state: state2 }) {\n const available = state2.data?.accessTokenManagement === \"available\";\n return /* @__PURE__ */ u3(\"section\", { id: \"tokensView\", children: /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"tokensHeading\", class: \"pcap\", tabIndex: -1, children: \"Access tokens\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody\", children: [\n /* @__PURE__ */ u3(\"p\", { class: \"activity-copy\", children: \"Create named Bearer tokens for MCP clients. Each secret is shown once; revoke it when that client should lose access.\" }),\n /* @__PURE__ */ u3(NoticeLine, { id: \"tokenNotice\", notice: state2.tokenNotice }),\n !available ? /* @__PURE__ */ u3(Unavailable, { children: accessTokenUnavailableCopy(state2.data?.accessTokenManagement) }) : /* @__PURE__ */ u3(\"div\", { id: \"tokenAvailable\", children: [\n state2.createdToken ? /* @__PURE__ */ u3(Reveal, { token: state2.createdToken }) : /* @__PURE__ */ u3(CreateForm, { busy: state2.tokenBusy }),\n /* @__PURE__ */ u3(\n \"div\",\n {\n id: \"tokenList\",\n class: \"token-ledger\",\n \"aria-busy\": state2.tokenPhase === \"loading\" ? \"true\" : \"false\",\n children: state2.tokenPhase === \"loading\" ? /* @__PURE__ */ u3(Empty, { children: \"Loading access tokens…\" }) : state2.tokenPhase === \"error\" ? /* @__PURE__ */ u3(\"p\", { class: \"empty\", children: /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n onClick: () => void loadAccessTokens(),\n children: \"Try loading access tokens again\"\n }\n ) }) : state2.tokens.length === 0 ? /* @__PURE__ */ u3(Empty, { children: \"No access tokens yet. Name the first MCP client above.\" }) : state2.tokens.map((token) => /* @__PURE__ */ u3(\n TokenCard,\n {\n token,\n renaming: state2.tokenRenaming === token.id,\n busy: state2.tokenBusy\n },\n token.id\n ))\n }\n )\n ] })\n ] })\n ] }) });\n }\n\n // src/operator-ui/app/main.tsx\n function useOperatorState() {\n const [, bump] = y2((count) => count + 1, 0);\n const snapshot = getState();\n _2(() => {\n const unsubscribe = subscribe(() => bump(void 0));\n if (getState() !== snapshot) bump(void 0);\n return unsubscribe;\n }, []);\n return snapshot;\n }\n function visiblePages(state2) {\n return OPERATOR_PAGES.filter((page) => {\n if (page === \"credentials\") {\n return state2.data?.credentialManagement === \"available\";\n }\n if (page === \"tokens\") {\n return state2.data?.accessTokenManagement === \"available\";\n }\n if (page === \"activity\") return Boolean(state2.data?.activityEnabled);\n return true;\n });\n }\n function OperatorNav() {\n const state2 = useOperatorState();\n if (state2.session !== \"ready\") return null;\n return /* @__PURE__ */ u3(\"div\", { class: \"mast-actions\", children: [\n /* @__PURE__ */ u3(\"nav\", { class: \"page-nav\", \"aria-label\": \"Operator pages\", children: visiblePages(state2).map((page) => /* @__PURE__ */ u3(\n PageLink,\n {\n page,\n class: \"navlink\",\n current: state2.page === page,\n children: PAGE_META[page].label\n },\n page\n )) }),\n /* @__PURE__ */ u3(\"div\", { class: \"session-actions\", \"aria-label\": \"Session actions\", children: auth.kind === \"clerk\" ? /* @__PURE__ */ u3(\"button\", { class: \"navlink\", type: \"button\", onClick: signOut, children: \"Sign out\" }) : /* @__PURE__ */ u3(\"button\", { class: \"navlink\", type: \"button\", onClick: forgetBearer, children: \"Change token\" }) })\n ] });\n }\n function Gate({ state: state2 }) {\n const [token, setToken] = d2(\"\");\n const signedIn = auth.kind === \"clerk\" && Boolean(window.Clerk?.user);\n const loading = state2.session === \"loading\";\n return /* @__PURE__ */ u3(\"section\", { id: \"gate\", children: /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"gateHeading\", class: \"pcap\", tabIndex: -1, children: PAGE_META[state2.page].label }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody lead-copy\", children: [\n /* @__PURE__ */ u3(\"p\", { children: productDescription }),\n /* @__PURE__ */ u3(\"p\", { id: \"gateCopy\", class: \"meta\", children: loading ? \"Checking your session…\" : gateCopy(auth.kind, signedIn) }),\n loading ? null : auth.kind === \"clerk\" ? /* @__PURE__ */ u3(\"div\", { id: \"clerkGate\", class: \"actions gate-actions\", children: signedIn ? /* @__PURE__ */ u3(\"button\", { class: \"linklike\", type: \"button\", onClick: signOut, children: \"Sign out\" }) : /* @__PURE__ */ u3(\"button\", { id: \"signin\", class: \"linklike\", type: \"button\", onClick: signIn, children: \"Team sign in\" }) }) : /* @__PURE__ */ u3(\n \"form\",\n {\n id: \"tokenGate\",\n class: \"row gate-actions\",\n onSubmit: (event) => {\n event.preventDefault();\n const value = token.trim();\n if (!value) return;\n setToken(\"\");\n signInWithBearer(value);\n },\n children: [\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: \"token\",\n type: \"password\",\n placeholder: \"Bearer token\",\n autocomplete: \"off\",\n \"aria-label\": \"Bearer token\",\n value: token,\n onInput: (event) => setToken(event.currentTarget.value)\n }\n ),\n /* @__PURE__ */ u3(\"button\", { id: \"save\", class: \"linklike\", type: \"submit\", children: \"Open operator pages\" })\n ]\n }\n ),\n /* @__PURE__ */ u3(NoticeLine, { id: \"err\", notice: state2.gate, className: \"\" })\n ] })\n ] }) });\n }\n function CurrentPage({ state: state2 }) {\n if (state2.page === \"credentials\") return /* @__PURE__ */ u3(CredentialsPage, { state: state2 });\n if (state2.page === \"tokens\") return /* @__PURE__ */ u3(TokensPage, { state: state2 });\n if (state2.page === \"activity\") return /* @__PURE__ */ u3(ActivityPage, { state: state2 });\n return /* @__PURE__ */ u3(ConnectionsPage, { state: state2 });\n }\n function OperatorApp() {\n const state2 = useOperatorState();\n const ready = state2.session === \"ready\";\n h2(() => {\n document.title = `${PAGE_META[state2.page].label} — ${titleSuffix}`;\n }, [state2.page]);\n h2(() => {\n if (!ready) return;\n if (state2.page === \"tokens\" && state2.data?.accessTokenManagement === \"available\" && state2.tokenPhase === \"idle\") {\n void loadAccessTokens();\n }\n if (state2.page === \"activity\" && state2.data?.activityEnabled && state2.activityPhase === \"idle\") {\n void loadActivity(true);\n }\n });\n h2(() => {\n if (!state2.pendingFocus) return;\n document.getElementById(state2.pendingFocus)?.focus();\n focusHandled();\n }, [state2.pendingFocus]);\n return ready ? /* @__PURE__ */ u3(\"div\", { id: \"app\", children: /* @__PURE__ */ u3(CurrentPage, { state: state2 }) }) : /* @__PURE__ */ u3(Gate, { state: state2 });\n }\n function mount(id, view) {\n const host = document.getElementById(id);\n if (!host) return;\n host.textContent = \"\";\n R(view, host);\n }\n mount(\"operatorNav\", /* @__PURE__ */ u3(OperatorNav, {}));\n mount(\"operatorContent\", /* @__PURE__ */ u3(OperatorApp, {}));\n void boot();\n})();\n"; +export const OPERATOR_UI_SCRIPT: string = "\"use strict\";\n(() => {\n // node_modules/preact/dist/preact.module.js\n var n;\n var l;\n var u;\n var t;\n var i;\n var r;\n var o;\n var e;\n var f;\n var c;\n var a;\n var s;\n var h;\n var p;\n var v;\n var y;\n var d = {};\n var w = [];\n var _ = /acit|ex(?:s|g|n|p|$)|rph|grid|ows|mnc|ntw|ine[ch]|zoo|^ord|itera/i;\n var g = Array.isArray;\n function m(n2, l3) {\n for (var u4 in l3) n2[u4] = l3[u4];\n return n2;\n }\n function b(n2) {\n n2 && n2.parentNode && n2.parentNode.removeChild(n2);\n }\n function k(l3, u4, t3) {\n var i3, r3, o3, e3 = {};\n for (o3 in u4) \"key\" == o3 ? i3 = u4[o3] : \"ref\" == o3 ? r3 = u4[o3] : e3[o3] = u4[o3];\n if (arguments.length > 2 && (e3.children = arguments.length > 3 ? n.call(arguments, 2) : t3), \"function\" == typeof l3 && null != l3.defaultProps) for (o3 in l3.defaultProps) void 0 === e3[o3] && (e3[o3] = l3.defaultProps[o3]);\n return x(l3, e3, i3, r3, null);\n }\n function x(n2, t3, i3, r3, o3) {\n var e3 = { type: n2, props: t3, key: i3, ref: r3, __k: null, __: null, __b: 0, __e: null, __c: null, constructor: void 0, __v: null == o3 ? ++u : o3, __i: -1, __u: 0 };\n return null == o3 && null != l.vnode && l.vnode(e3), e3;\n }\n function S(n2) {\n return n2.children;\n }\n function C(n2, l3) {\n this.props = n2, this.context = l3;\n }\n function $(n2, l3) {\n if (null == l3) return n2.__ ? $(n2.__, n2.__i + 1) : null;\n for (var u4; l3 < n2.__k.length; l3++) if (null != (u4 = n2.__k[l3]) && null != u4.__e) return u4.__e;\n return \"function\" == typeof n2.type ? $(n2) : null;\n }\n function I(n2) {\n if (n2.__P && n2.__d) {\n var u4 = n2.__v, t3 = u4.__e, i3 = [], r3 = [], o3 = m({}, u4);\n o3.__v = u4.__v + 1, l.vnode && l.vnode(o3), q(n2.__P, o3, u4, n2.__n, n2.__P.namespaceURI, 32 & u4.__u ? [t3] : null, i3, null == t3 ? $(u4) : t3, !!(32 & u4.__u), r3), o3.__v = u4.__v, o3.__.__k[o3.__i] = o3, D(i3, o3, r3), u4.__e = u4.__ = null, o3.__e != t3 && P(o3);\n }\n }\n function P(n2) {\n if (null != (n2 = n2.__) && null != n2.__c) return n2.__e = n2.__c.base = null, n2.__k.some(function(l3) {\n if (null != l3 && null != l3.__e) return n2.__e = n2.__c.base = l3.__e;\n }), P(n2);\n }\n function A(n2) {\n (!n2.__d && (n2.__d = true) && i.push(n2) && !H.__r++ || r != l.debounceRendering) && ((r = l.debounceRendering) || o)(H);\n }\n function H() {\n try {\n for (var n2, l3 = 1; i.length; ) i.length > l3 && i.sort(e), n2 = i.shift(), l3 = i.length, I(n2);\n } finally {\n i.length = H.__r = 0;\n }\n }\n function L(n2, l3, u4, t3, i3, r3, o3, e3, f4, c3, a3) {\n var s3, h3, p3, v3, y3, _3, g2 = t3 && t3.__k || w, m3 = l3.length;\n for (f4 = T(u4, l3, g2, f4, m3), s3 = 0; s3 < m3; s3++) null != (p3 = u4.__k[s3]) && (h3 = -1 != p3.__i && g2[p3.__i] || d, p3.__i = s3, _3 = q(n2, p3, h3, i3, r3, o3, e3, f4, c3, a3), v3 = p3.__e, p3.ref && h3.ref != p3.ref && (h3.ref && J(h3.ref, null, p3), a3.push(p3.ref, p3.__c || v3, p3)), null == y3 && null != v3 && (y3 = v3), 4 & p3.__u ? (f4 = j(p3, f4, n2), h3.__e && (h3.__e = null)) : \"function\" == typeof p3.type && void 0 !== _3 ? f4 = _3 : v3 && (f4 = v3.nextSibling), p3.__u &= -7);\n return u4.__e = y3, f4;\n }\n function T(n2, l3, u4, t3, i3) {\n var r3, o3, e3, f4, c3, a3 = u4.length, s3 = a3, h3 = 0;\n for (n2.__k = new Array(i3), r3 = 0; r3 < i3; r3++) null != (o3 = l3[r3]) && \"boolean\" != typeof o3 && \"function\" != typeof o3 ? (\"string\" == typeof o3 || \"number\" == typeof o3 || \"bigint\" == typeof o3 || o3.constructor == String ? o3 = n2.__k[r3] = x(null, o3, null, null, null) : g(o3) ? o3 = n2.__k[r3] = x(S, { children: o3 }, null, null, null) : void 0 === o3.constructor && o3.__b > 0 ? o3 = n2.__k[r3] = x(o3.type, o3.props, o3.key, o3.ref ? o3.ref : null, o3.__v) : n2.__k[r3] = o3, f4 = r3 + h3, o3.__ = n2, o3.__b = n2.__b + 1, e3 = null, -1 != (c3 = o3.__i = O(o3, u4, f4, s3)) && (s3--, (e3 = u4[c3]) && (e3.__u |= 2)), null == e3 || null == e3.__v ? (-1 == c3 && (i3 > a3 ? h3-- : i3 < a3 && h3++), \"function\" != typeof o3.type && (o3.__u |= 4)) : c3 != f4 && (c3 == f4 - 1 ? h3-- : c3 == f4 + 1 ? h3++ : (c3 > f4 ? h3-- : h3++, o3.__u |= 4))) : n2.__k[r3] = null;\n if (s3) for (r3 = 0; r3 < a3; r3++) null != (e3 = u4[r3]) && 0 == (2 & e3.__u) && (e3.__e == t3 && (t3 = $(e3)), K(e3, e3));\n return t3;\n }\n function j(n2, l3, u4) {\n var t3, i3;\n if (\"function\" == typeof n2.type) {\n for (t3 = n2.__k, i3 = 0; t3 && i3 < t3.length; i3++) t3[i3] && (t3[i3].__ = n2, l3 = j(t3[i3], l3, u4));\n return l3;\n }\n n2.__e != l3 && (l3 && n2.type && !l3.parentNode && (l3 = $(n2)), l3 = u4.insertBefore(n2.__e, l3 || null));\n do {\n l3 = l3 && l3.nextSibling;\n } while (null != l3 && 8 == l3.nodeType);\n return l3;\n }\n function O(n2, l3, u4, t3) {\n var i3, r3, o3, e3 = n2.key, f4 = n2.type, c3 = l3[u4], a3 = null != c3 && 0 == (2 & c3.__u);\n if (null === c3 && null == e3 || a3 && e3 == c3.key && f4 == c3.type) return u4;\n if (t3 > (a3 ? 1 : 0)) {\n for (i3 = u4 - 1, r3 = u4 + 1; i3 >= 0 || r3 < l3.length; ) if (null != (c3 = l3[o3 = i3 >= 0 ? i3-- : r3++]) && 0 == (2 & c3.__u) && e3 == c3.key && f4 == c3.type) return o3;\n }\n return -1;\n }\n function z(n2, l3, u4) {\n \"-\" == l3[0] ? n2.setProperty(l3, null == u4 ? \"\" : u4) : n2[l3] = null == u4 ? \"\" : \"number\" != typeof u4 || _.test(l3) ? u4 : u4 + \"px\";\n }\n function N(n2, l3, u4, t3, i3) {\n var r3, o3;\n n: if (\"style\" == l3) if (\"string\" == typeof u4) n2.style.cssText = u4;\n else {\n if (\"string\" == typeof t3 && (n2.style.cssText = t3 = \"\"), t3) for (l3 in t3) u4 && l3 in u4 || z(n2.style, l3, \"\");\n if (u4) for (l3 in u4) t3 && u4[l3] == t3[l3] || z(n2.style, l3, u4[l3]);\n }\n else if (\"o\" == l3[0] && \"n\" == l3[1]) r3 = l3 != (l3 = l3.replace(s, \"$1\")), o3 = l3.toLowerCase(), l3 = o3 in n2 || \"onFocusOut\" == l3 || \"onFocusIn\" == l3 ? o3.slice(2) : l3.slice(2), n2.l || (n2.l = {}), n2.l[l3 + r3] = u4, u4 ? t3 ? u4[a] = t3[a] : (u4[a] = h, n2.addEventListener(l3, r3 ? v : p, r3)) : n2.removeEventListener(l3, r3 ? v : p, r3);\n else {\n if (\"http://www.w3.org/2000/svg\" == i3) l3 = l3.replace(/xlink(H|:h)/, \"h\").replace(/sName$/, \"s\");\n else if (\"width\" != l3 && \"height\" != l3 && \"href\" != l3 && \"list\" != l3 && \"form\" != l3 && \"tabIndex\" != l3 && \"download\" != l3 && \"rowSpan\" != l3 && \"colSpan\" != l3 && \"role\" != l3 && \"popover\" != l3 && l3 in n2) try {\n n2[l3] = null == u4 ? \"\" : u4;\n break n;\n } catch (n3) {\n }\n \"function\" == typeof u4 || (null == u4 || false === u4 && \"-\" != l3[4] ? n2.removeAttribute(l3) : n2.setAttribute(l3, \"popover\" == l3 && 1 == u4 ? \"\" : u4));\n }\n }\n function V(n2) {\n return function(u4) {\n if (this.l) {\n var t3 = this.l[u4.type + n2];\n if (null == u4[c]) u4[c] = h++;\n else if (u4[c] < t3[a]) return;\n return t3(l.event ? l.event(u4) : u4);\n }\n };\n }\n function q(n2, u4, t3, i3, r3, o3, e3, f4, c3, a3) {\n var s3, h3, p3, v3, y3, d3, _3, k3, x2, M, I2, P2, A2, H2, T2, j3, F = u4.type;\n if (void 0 !== u4.constructor) return null;\n 128 & t3.__u && (c3 = !!(32 & t3.__u), o3 = [f4 = u4.__e = t3.__e]), (s3 = l.__b) && s3(u4);\n n: if (\"function\" == typeof F) {\n h3 = e3.length;\n try {\n if (x2 = u4.props, M = F.prototype && F.prototype.render, I2 = (s3 = F.contextType) && i3[s3.__c], P2 = s3 ? I2 ? I2.props.value : s3.__ : i3, t3.__c ? k3 = (p3 = u4.__c = t3.__c).__ = p3.__E : (M ? u4.__c = p3 = new F(x2, P2) : (u4.__c = p3 = new C(x2, P2), p3.constructor = F, p3.render = Q), I2 && I2.sub(p3), p3.state || (p3.state = {}), p3.__n = i3, v3 = p3.__d = true, p3.__h = [], p3._sb = []), M && null == p3.__s && (p3.__s = p3.state), M && null != F.getDerivedStateFromProps && (p3.__s == p3.state && (p3.__s = m({}, p3.__s)), m(p3.__s, F.getDerivedStateFromProps(x2, p3.__s))), y3 = p3.props, d3 = p3.state, p3.__v = u4, v3) M && null == F.getDerivedStateFromProps && null != p3.componentWillMount && p3.componentWillMount(), M && null != p3.componentDidMount && p3.__h.push(p3.componentDidMount);\n else {\n if (M && null == F.getDerivedStateFromProps && x2 !== y3 && null != p3.componentWillReceiveProps && p3.componentWillReceiveProps(x2, P2), u4.__v == t3.__v || !p3.__e && null != p3.shouldComponentUpdate && false === p3.shouldComponentUpdate(x2, p3.__s, P2)) {\n u4.__v != t3.__v && (p3.props = x2, p3.state = p3.__s, p3.__d = false), u4.__e = t3.__e, u4.__k = t3.__k, u4.__k.some(function(n3) {\n n3 && (n3.__ = u4);\n }), w.push.apply(p3.__h, p3._sb), p3._sb = [], p3.__h.length && e3.push(p3), f4 = $(t3);\n break n;\n }\n null != p3.componentWillUpdate && p3.componentWillUpdate(x2, p3.__s, P2), M && null != p3.componentDidUpdate && p3.__h.push(function() {\n p3.componentDidUpdate(y3, d3, _3);\n });\n }\n if (p3.context = P2, p3.props = x2, p3.__P = n2, p3.__e = false, A2 = l.__r, H2 = 0, M) p3.state = p3.__s, p3.__d = false, A2 && A2(u4), s3 = p3.render(p3.props, p3.state, p3.context), w.push.apply(p3.__h, p3._sb), p3._sb = [];\n else do {\n p3.__d = false, A2 && A2(u4), s3 = p3.render(p3.props, p3.state, p3.context), p3.state = p3.__s;\n } while (p3.__d && ++H2 < 25);\n p3.state = p3.__s, null != p3.getChildContext && (i3 = m(m({}, i3), p3.getChildContext())), M && !v3 && null != p3.getSnapshotBeforeUpdate && (_3 = p3.getSnapshotBeforeUpdate(y3, d3)), T2 = null != s3 && s3.type === S && null == s3.key ? E(s3.props.children) : s3, f4 = L(n2, g(T2) ? T2 : [T2], u4, t3, i3, r3, o3, e3, f4, c3, a3), p3.base = u4.__e, u4.__u &= -161, p3.__h.length && e3.push(p3), k3 && (p3.__E = p3.__ = null);\n } catch (n3) {\n if (e3.length = h3, u4.__v = null, c3 || null != o3) {\n if (n3.then) {\n for (u4.__u |= c3 ? 160 : 128; f4 && 8 == f4.nodeType && f4.nextSibling; ) f4 = f4.nextSibling;\n null != o3 && (o3[o3.indexOf(f4)] = null), u4.__e = f4;\n } else if (null != o3) for (j3 = o3.length; j3--; ) b(o3[j3]);\n } else u4.__e = t3.__e;\n null == u4.__k && (u4.__k = t3.__k || []), n3.then || B(u4), l.__e(n3, u4, t3);\n }\n } else null == o3 && u4.__v == t3.__v ? (u4.__k = t3.__k, u4.__e = t3.__e) : f4 = u4.__e = G(t3.__e, u4, t3, i3, r3, o3, e3, c3, a3);\n return (s3 = l.diffed) && s3(u4), 128 & u4.__u ? void 0 : f4;\n }\n function B(n2) {\n n2 && (n2.__c && (n2.__c.__e = true), n2.__k && n2.__k.some(B));\n }\n function D(n2, u4, t3) {\n for (var i3 = 0; i3 < t3.length; i3++) J(t3[i3], t3[++i3], t3[++i3]);\n l.__c && l.__c(u4, n2), n2.some(function(u5) {\n try {\n n2 = u5.__h, u5.__h = [], n2.some(function(n3) {\n n3.call(u5);\n });\n } catch (n3) {\n l.__e(n3, u5.__v);\n }\n });\n }\n function E(n2) {\n return \"object\" != typeof n2 || null == n2 || n2.__b > 0 ? n2 : g(n2) ? n2.map(E) : void 0 !== n2.constructor ? null : m({}, n2);\n }\n function G(u4, t3, i3, r3, o3, e3, f4, c3, a3) {\n var s3, h3, p3, v3, y3, w3, _3, m3 = i3.props || d, k3 = t3.props, x2 = t3.type;\n if (\"svg\" == x2 ? o3 = \"http://www.w3.org/2000/svg\" : \"math\" == x2 ? o3 = \"http://www.w3.org/1998/Math/MathML\" : o3 || (o3 = \"http://www.w3.org/1999/xhtml\"), null != e3) {\n for (s3 = 0; s3 < e3.length; s3++) if ((y3 = e3[s3]) && \"setAttribute\" in y3 == !!x2 && (x2 ? y3.localName == x2 : 3 == y3.nodeType)) {\n u4 = y3, e3[s3] = null;\n break;\n }\n }\n if (null == u4) {\n if (null == x2) return document.createTextNode(k3);\n u4 = document.createElementNS(o3, x2, k3.is && k3), c3 && (l.__m && l.__m(t3, e3), c3 = false), e3 = null;\n }\n if (null == x2) m3 === k3 || c3 && u4.data == k3 || (u4.data = k3);\n else {\n if (e3 = \"textarea\" == x2 && null != k3.defaultValue ? null : e3 && n.call(u4.childNodes), !c3 && null != e3) for (m3 = {}, s3 = 0; s3 < u4.attributes.length; s3++) m3[(y3 = u4.attributes[s3]).name] = y3.value;\n for (s3 in m3) y3 = m3[s3], \"dangerouslySetInnerHTML\" == s3 ? p3 = y3 : \"children\" == s3 || s3 in k3 || \"value\" == s3 && \"defaultValue\" in k3 || \"checked\" == s3 && \"defaultChecked\" in k3 || N(u4, s3, null, y3, o3);\n for (s3 in k3) y3 = k3[s3], \"children\" == s3 ? v3 = y3 : \"dangerouslySetInnerHTML\" == s3 ? h3 = y3 : \"value\" == s3 ? w3 = y3 : \"checked\" == s3 ? _3 = y3 : c3 && \"function\" != typeof y3 || m3[s3] === y3 || N(u4, s3, y3, m3[s3], o3);\n if (h3) c3 || p3 && (h3.__html == p3.__html || h3.__html == u4.innerHTML) || (u4.innerHTML = h3.__html), t3.__k = [];\n else if (p3 && (u4.innerHTML = \"\"), L(\"template\" == t3.type ? u4.content : u4, g(v3) ? v3 : [v3], t3, i3, r3, \"foreignObject\" == x2 ? \"http://www.w3.org/1999/xhtml\" : o3, e3, f4, e3 ? e3[0] : i3.__k && $(i3, 0), c3, a3), null != e3) for (s3 = e3.length; s3--; ) b(e3[s3]);\n c3 && \"textarea\" != x2 || (s3 = \"value\", \"progress\" == x2 && null == w3 ? u4.removeAttribute(\"value\") : null != w3 && (w3 !== u4[s3] || \"progress\" == x2 && !w3 || \"option\" == x2 && w3 != m3[s3]) && N(u4, s3, w3, m3[s3], o3), s3 = \"checked\", null != _3 && _3 != u4[s3] && N(u4, s3, _3, m3[s3], o3));\n }\n return u4;\n }\n function J(n2, u4, t3) {\n try {\n if (\"function\" == typeof n2) {\n var i3 = \"function\" == typeof n2.__u;\n i3 && n2.__u(), i3 && null == u4 || (n2.__u = n2(u4));\n } else n2.current = u4;\n } catch (n3) {\n l.__e(n3, t3);\n }\n }\n function K(n2, u4, t3) {\n var i3, r3;\n if (l.unmount && l.unmount(n2), (i3 = n2.ref) && (i3.current && i3.current != n2.__e || J(i3, null, u4)), null != (i3 = n2.__c)) {\n if (i3.componentWillUnmount) try {\n i3.componentWillUnmount();\n } catch (n3) {\n l.__e(n3, u4);\n }\n i3.base = i3.__P = i3.__n = null;\n }\n if (i3 = n2.__k) for (r3 = 0; r3 < i3.length; r3++) i3[r3] && K(i3[r3], u4, t3 || \"function\" != typeof n2.type);\n t3 || b(n2.__e), n2.__c = n2.__ = n2.__e = void 0;\n }\n function Q(n2, l3, u4) {\n return this.constructor(n2, u4);\n }\n function R(u4, t3, i3) {\n var r3, o3, e3, f4;\n t3 == document && (t3 = document.documentElement), l.__ && l.__(u4, t3), o3 = (r3 = \"function\" == typeof i3) ? null : i3 && i3.__k || t3.__k, e3 = [], f4 = [], q(t3, u4 = (!r3 && i3 || t3).__k = k(S, null, [u4]), o3 || d, d, t3.namespaceURI, !r3 && i3 ? [i3] : o3 ? null : t3.firstChild ? n.call(t3.childNodes) : null, e3, !r3 && i3 ? i3 : o3 ? o3.__e : t3.firstChild, r3, f4), D(e3, u4, f4), u4.props.children = null;\n }\n n = w.slice, l = { __e: function(n2, l3, u4, t3) {\n for (var i3, r3, o3; l3 = l3.__; ) if ((i3 = l3.__c) && !i3.__) try {\n if ((r3 = i3.constructor) && null != r3.getDerivedStateFromError && (i3.setState(r3.getDerivedStateFromError(n2)), o3 = i3.__d), null != i3.componentDidCatch && (i3.componentDidCatch(n2, t3 || {}), o3 = i3.__d), o3) return i3.__E = i3;\n } catch (l4) {\n n2 = l4;\n }\n throw n2;\n } }, u = 0, t = function(n2) {\n return null != n2 && void 0 === n2.constructor;\n }, C.prototype.setState = function(n2, l3) {\n var u4;\n u4 = null != this.__s && this.__s != this.state ? this.__s : this.__s = m({}, this.state), \"function\" == typeof n2 && (n2 = n2(m({}, u4), this.props)), n2 && m(u4, n2), null != n2 && this.__v && (l3 && this._sb.push(l3), A(this));\n }, C.prototype.forceUpdate = function(n2) {\n this.__v && (this.__e = true, n2 && this.__h.push(n2), A(this));\n }, C.prototype.render = S, i = [], o = \"function\" == typeof Promise ? Promise.prototype.then.bind(Promise.resolve()) : setTimeout, e = function(n2, l3) {\n return n2.__v.__b - l3.__v.__b;\n }, H.__r = 0, f = Math.random().toString(8), c = \"__d\" + f, a = \"__a\" + f, s = /(PointerCapture)$|Capture$/i, h = 0, p = V(false), v = V(true), y = 0;\n\n // node_modules/preact/hooks/dist/hooks.module.js\n var t2;\n var r2;\n var u2;\n var i2;\n var o2 = 0;\n var f2 = [];\n var c2 = l;\n var e2 = c2.__b;\n var a2 = c2.__r;\n var v2 = c2.diffed;\n var l2 = c2.__c;\n var m2 = c2.unmount;\n var p2 = c2.__;\n function s2(n2, t3) {\n c2.__h && c2.__h(r2, n2, o2 || t3), o2 = 0;\n var u4 = r2.__H || (r2.__H = { __: [], __h: [] });\n return n2 >= u4.__.length && u4.__.push({}), u4.__[n2];\n }\n function d2(n2) {\n return o2 = 1, y2(D2, n2);\n }\n function y2(n2, u4, i3) {\n var o3 = s2(t2++, 2);\n if (o3.t = n2, !o3.__c && (o3.__ = [i3 ? i3(u4) : D2(void 0, u4), function(n3) {\n var t3 = o3.__N ? o3.__N[0] : o3.__[0], r3 = o3.t(t3, n3);\n t3 !== r3 && (o3.__N = [r3, o3.__[1]], o3.__c.setState({}));\n }], o3.__c = r2, !r2.__f)) {\n var f4 = function(n3, t3, r3) {\n if (!o3.__c.__H) return true;\n var u5 = false, i4 = o3.__c.props !== n3;\n if (o3.__c.__H.__.some(function(n4) {\n if (n4.__N) {\n u5 = true;\n var t4 = n4.__[0];\n n4.__ = n4.__N, n4.__N = void 0, t4 !== n4.__[0] && (i4 = true);\n }\n }), c3) {\n var f5 = c3.call(this, n3, t3, r3);\n return u5 ? f5 || i4 : f5;\n }\n return !u5 || i4;\n };\n r2.__f = true;\n var c3 = r2.shouldComponentUpdate, e3 = r2.componentWillUpdate;\n r2.componentWillUpdate = function(n3, t3, r3) {\n if (this.__e) {\n var u5 = c3;\n c3 = void 0, f4(n3, t3, r3), c3 = u5;\n }\n e3 && e3.call(this, n3, t3, r3);\n }, r2.shouldComponentUpdate = f4;\n }\n return o3.__N || o3.__;\n }\n function h2(n2, u4) {\n var i3 = s2(t2++, 3);\n !c2.__s && C2(i3.__H, u4) && (i3.__ = n2, i3.u = u4, r2.__H.__h.push(i3));\n }\n function _2(n2, u4) {\n var i3 = s2(t2++, 4);\n !c2.__s && C2(i3.__H, u4) && (i3.__ = n2, i3.u = u4, r2.__h.push(i3));\n }\n function j2() {\n for (var n2; n2 = f2.shift(); ) {\n var t3 = n2.__H;\n if (n2.__P && t3) try {\n t3.__h.some(z2), t3.__h.some(B2), t3.__h = [];\n } catch (r3) {\n t3.__h = [], c2.__e(r3, n2.__v);\n }\n }\n }\n c2.__b = function(n2) {\n r2 = null, e2 && e2(n2);\n }, c2.__ = function(n2, t3) {\n n2 && t3.__k && t3.__k.__m && (n2.__m = t3.__k.__m), p2 && p2(n2, t3);\n }, c2.__r = function(n2) {\n a2 && a2(n2), t2 = 0;\n var i3 = (r2 = n2.__c).__H;\n i3 && (u2 === r2 ? (i3.__h = [], r2.__h = [], i3.__.some(function(n3) {\n n3.__N && (n3.__ = n3.__N), n3.u = n3.__N = void 0;\n })) : (i3.__h.some(z2), i3.__h.some(B2), i3.__h = [], t2 = 0)), u2 = r2;\n }, c2.diffed = function(n2) {\n v2 && v2(n2);\n var t3 = n2.__c;\n t3 && t3.__H && (t3.__H.__h.length && (1 !== f2.push(t3) && i2 === c2.requestAnimationFrame || ((i2 = c2.requestAnimationFrame) || w2)(j2)), t3.__H.__.some(function(n3) {\n n3.u && (n3.__H = n3.u, n3.u = void 0);\n })), u2 = r2 = null;\n }, c2.__c = function(n2, t3) {\n t3.some(function(n3) {\n try {\n n3.__h.some(z2), n3.__h = n3.__h.filter(function(n4) {\n return !n4.__ || B2(n4);\n });\n } catch (r3) {\n t3.some(function(n4) {\n n4.__h && (n4.__h = []);\n }), t3 = [], c2.__e(r3, n3.__v);\n }\n }), l2 && l2(n2, t3);\n }, c2.unmount = function(n2) {\n m2 && m2(n2);\n var t3, r3 = n2.__c;\n r3 && r3.__H && (r3.__H.__.some(function(n3) {\n try {\n z2(n3);\n } catch (n4) {\n t3 = n4;\n }\n }), r3.__H = void 0, t3 && c2.__e(t3, r3.__v));\n };\n var k2 = \"function\" == typeof requestAnimationFrame;\n function w2(n2) {\n var t3, r3 = function() {\n clearTimeout(u4), k2 && cancelAnimationFrame(t3), setTimeout(n2);\n }, u4 = setTimeout(r3, 35);\n k2 && (t3 = requestAnimationFrame(r3));\n }\n function z2(n2) {\n var t3 = r2, u4 = n2.__c;\n \"function\" == typeof u4 && (n2.__c = void 0, u4()), r2 = t3;\n }\n function B2(n2) {\n var t3 = r2;\n n2.__c = n2.__(), r2 = t3;\n }\n function C2(n2, t3) {\n return !n2 || n2.length !== t3.length || t3.some(function(t4, r3) {\n return t4 !== n2[r3];\n });\n }\n function D2(n2, t3) {\n return \"function\" == typeof t3 ? t3(n2) : t3;\n }\n\n // src/operator-ui/view.ts\n var OPERATOR_PAGES = [\n \"connections\",\n \"credentials\",\n \"tokens\",\n \"activity\"\n ];\n var PAGE_META = {\n connections: { path: \"/\", label: \"Connections\" },\n credentials: { path: \"/credentials\", label: \"Credentials\" },\n tokens: { path: \"/tokens\", label: \"Access tokens\" },\n activity: { path: \"/activity\", label: \"Activity\" }\n };\n function pageForPath(path) {\n const match = OPERATOR_PAGES.find((page) => PAGE_META[page].path === path);\n return match ?? \"connections\";\n }\n function info(message2) {\n return { message: message2, tone: \"info\" };\n }\n function failure(message2) {\n return { message: message2, tone: \"error\" };\n }\n function initialState(page) {\n return {\n page,\n generation: 0,\n session: \"loading\",\n gate: null,\n refreshing: false,\n pendingFocus: null,\n ...identityScopedState()\n };\n }\n function identityScopedState() {\n return {\n data: null,\n connectorFilter: \"\",\n oauthNotice: null,\n oauthBusy: null,\n credentialNotice: null,\n credentialEditing: null,\n credentialBusy: null,\n tokenPhase: \"idle\",\n tokenNotice: null,\n tokens: [],\n createdToken: null,\n tokenRenaming: null,\n tokenBusy: false,\n activityPhase: \"idle\",\n activityNotice: null,\n activityEvents: [],\n activityCursor: null,\n activitySearch: \"\"\n };\n }\n function resetIdentity(state2, gate2 = null) {\n return {\n ...state2,\n generation: state2.generation + 1,\n session: \"gated\",\n gate: gate2,\n refreshing: false,\n pendingFocus: null,\n ...identityScopedState()\n };\n }\n function withPage(state2, page) {\n return {\n ...state2,\n page,\n createdToken: null,\n tokenRenaming: null,\n tokenNotice: null,\n credentialEditing: null,\n credentialNotice: null\n };\n }\n function credentialUnavailableCopy(capability) {\n if (capability === \"no_slots\") {\n return \"No connectors declare operator-managed credential slots. Connector credentials remain configuration-as-code until a slot is declared.\";\n }\n if (capability === \"vault_not_configured\") {\n return \"Credential storage is not configured. Set credentials.encryptionKey before managing connector credentials here.\";\n }\n return \"Credential management requires an eligible interactive operator. Bearer-authenticated sessions can inspect connections but cannot manage stored credentials.\";\n }\n function accessTokenUnavailableCopy(capability) {\n if (capability === \"not_configured\") {\n return \"Access tokens are not configured for this deployment. Add accessTokens to the deployment configuration to enable them.\";\n }\n return \"Access token management requires an eligible interactive operator. A Bearer token can connect to MCP, but it cannot create or revoke other tokens.\";\n }\n function connectorStatusLabel(status) {\n if (status === \"ok\") return \"Connected\";\n if (status === \"auth_required\") return \"Authorization needed\";\n return \"Unavailable\";\n }\n function toolCountLabel(count) {\n return `${count} ${count === 1 ? \"tool\" : \"tools\"}`;\n }\n var DRIFT_CATEGORIES = [\n { key: \"unclassifiedTools\", label: \"Unclassified\" },\n { key: \"unservedTools\", label: \"Unserved\" },\n { key: \"annotationConflicts\", label: \"Annotation conflicts\" },\n { key: \"schemaChanges\", label: \"Schema changes\" }\n ];\n function driftTotal(drift) {\n if (!drift) return 0;\n return DRIFT_CATEGORIES.reduce((sum, { key }) => sum + (drift[key] || 0), 0);\n }\n function driftState(drift) {\n if (!drift) return \"unavailable\";\n return driftTotal(drift) > 0 ? \"warning\" : \"clean\";\n }\n function driftCounts(drift) {\n if (!drift) return [];\n return DRIFT_CATEGORIES.map(({ key, label }) => ({\n key,\n label,\n count: drift[key] || 0\n }));\n }\n function driftSummary(drift) {\n const state2 = driftState(drift);\n if (state2 === \"unavailable\") {\n return \"No catalog refresh observed yet in this runtime.\";\n }\n const observed = formatDate(drift?.observedAt);\n const when = observed ? ` · observed ${observed}` : \"\";\n if (state2 === \"clean\") return `Matches the reviewed manifest${when}`;\n const total = driftTotal(drift);\n return `${total} difference${total === 1 ? \"\" : \"s\"} from the reviewed manifest${when}`;\n }\n function safeHttpHref(url) {\n if (!url) return null;\n try {\n const protocol = new URL(url).protocol;\n return protocol === \"http:\" || protocol === \"https:\" ? url : null;\n } catch {\n return null;\n }\n }\n function formatDate(value) {\n if (!value) return \"\";\n const date = new Date(value);\n return Number.isNaN(date.valueOf()) ? \"\" : date.toLocaleString();\n }\n function actorLabel(actor) {\n if (!actor?.kind) return \"unknown\";\n if (actor.label) return `${actor.kind} · ${actor.label}`;\n return actor.id ? `${actor.kind} · ${actor.id}` : actor.kind;\n }\n function actorStableId(actor) {\n if (!actor?.id) return null;\n if (!actor.label && !actor.namespace) return null;\n return actor.namespace ? `${actor.namespace} · ${actor.id}` : actor.id;\n }\n function activityMatches(event, query) {\n const q2 = query.trim().toLowerCase();\n if (!q2) return true;\n return [\n event.address,\n event.connectorId,\n event.toolName,\n event.source,\n event.outcome,\n event.errorCode,\n event.friction,\n event.actor?.kind,\n event.actor?.id,\n event.actor?.namespace,\n event.actor?.label\n ].some((value) => String(value ?? \"\").toLowerCase().includes(q2));\n }\n function filterActivity(events, query) {\n return events.filter((event) => activityMatches(event, query));\n }\n function activitySummary(events) {\n if (events.length === 0) return \"Arguments and results are never stored.\";\n const tools = new Set(events.map((event) => event.address)).size;\n return `${events.length} loaded call${events.length === 1 ? \"\" : \"s\"} · ${tools} tool${tools === 1 ? \"\" : \"s\"} · no arguments or results stored`;\n }\n var ACTIVITY_OUTCOMES = [\"success\", \"error\", \"timeout\", \"cancelled\"];\n function activityOutcomeClass(outcome) {\n return ACTIVITY_OUTCOMES.includes(outcome) ? outcome : \"error\";\n }\n function activityDetail(event) {\n const parts = [event.source];\n if (event.attempts > 1) parts.push(`${event.attempts} attempts`);\n if (event.friction) parts.push(event.friction);\n if (event.errorCode && event.errorCode !== event.friction) {\n parts.push(event.errorCode);\n }\n return parts.join(\" · \");\n }\n function credentialStateLabel(credential) {\n if (!credential.configured) return \"not configured\";\n const masked = credential.fields?.length ? \"configured\" : `configured · ••••${credential.lastFour ?? \"\"}`;\n return credential.updatedAt ? `${masked} · updated ${formatDate(credential.updatedAt)}` : masked;\n }\n function gateCopy(kind, signedIn) {\n if (kind === \"cloudflare-access\") {\n return \"Cloudflare Access admitted this browser, but the current identity cannot open deployment-wide operator pages.\";\n }\n if (kind !== \"clerk\") {\n return \"Paste an operator bearer token to open this page. Nothing is requested until you do.\";\n }\n return signedIn ? \"Signed in with Clerk, but this account cannot open deployment-wide operator pages.\" : \"Sign in with Clerk to open this operator page.\";\n }\n\n // src/operator-ui/app/config.ts\n var auth = AUTH;\n var mcpUrl = MCP_URL;\n var initialPage = INITIAL_PAGE;\n var titleSuffix = TITLE_SUFFIX;\n var productName = PRODUCT_NAME;\n var productDescription = PRODUCT_DESCRIPTION;\n var productOperatorLabel = PRODUCT_OPERATOR_LABEL;\n var TOKEN_KEY = \"connecta:token\";\n\n // src/operator-ui/app/store.ts\n var state = initialState(initialPage);\n var listeners = /* @__PURE__ */ new Set();\n function getState() {\n return state;\n }\n function subscribe(listener) {\n listeners.add(listener);\n return () => listeners.delete(listener);\n }\n function set(patch) {\n state = { ...state, ...patch };\n for (const listener of listeners) listener();\n }\n function fence() {\n const generation = state.generation;\n return () => generation === state.generation;\n }\n function message(error, fallback) {\n return error instanceof Error && error.message ? error.message : fallback;\n }\n function sessionToken() {\n if (auth.kind === \"cloudflare-access\") return Promise.resolve(void 0);\n return auth.kind === \"clerk\" ? Promise.resolve(window.Clerk?.session?.getToken() ?? null) : Promise.resolve(localStorage.getItem(TOKEN_KEY));\n }\n function requestHeaders(token, body = false) {\n return {\n ...token ? { Authorization: `Bearer ${token}` } : {},\n ...body ? { \"Content-Type\": \"application/json\" } : {}\n };\n }\n function gate(notice = null) {\n state = resetIdentity(state, notice);\n for (const listener of listeners) listener();\n }\n async function operatorRequest(path, method, current, body) {\n const token = await sessionToken();\n if (!current()) throw new Error(\"The operator session changed.\");\n if (!token && auth.kind !== \"cloudflare-access\") {\n throw new Error(\"Your operator session has expired.\");\n }\n const res = await fetch(path, {\n method,\n headers: requestHeaders(token, Boolean(body)),\n credentials: \"same-origin\",\n ...body ? { body: JSON.stringify(body) } : {}\n });\n if (res.status === 204) return null;\n let payload = {};\n try {\n payload = await res.json();\n } catch {\n }\n if (res.status === 401) {\n throw new Error(\"Your operator session was not accepted. Sign in again.\");\n }\n if (res.status === 403) {\n throw new Error(\"This identity may not perform that action.\");\n }\n if (!res.ok) {\n throw new Error(payload.error || `Request failed (${res.status}).`);\n }\n return payload;\n }\n async function loadData() {\n const current = fence();\n if (state.session === \"ready\") set({ refreshing: true });\n let token;\n try {\n token = await sessionToken();\n } catch (error) {\n if (!current()) return;\n const why = message(error, \"unknown error\");\n return gate(failure(`Could not read the Clerk session: ${why}`));\n }\n if (!current()) return;\n if (!token && auth.kind !== \"cloudflare-access\") return gate(null);\n let res;\n try {\n res = await fetch(\"/ui/data\", {\n headers: requestHeaders(token),\n credentials: \"same-origin\"\n });\n } catch (error) {\n if (!current()) return;\n return gate(failure(`Network error: ${message(error, \"unknown error\")}`));\n }\n if (!current()) return;\n if (res.status === 401 || res.status === 403) {\n if (auth.kind === \"clerk\") {\n return gate(\n failure(\n res.status === 403 ? \"This Clerk account is not allowed to access connecta.\" : \"Your Clerk session was not accepted. Sign out and try again.\"\n )\n );\n }\n if (auth.kind === \"cloudflare-access\") {\n return gate(\n failure(\n \"Cloudflare Access admitted the request, but this identity is not an eligible operator.\"\n )\n );\n }\n localStorage.removeItem(TOKEN_KEY);\n return gate(failure(\"Token rejected — enter a valid bearer token.\"));\n }\n if (!res.ok) return gate(failure(`Error ${res.status}`));\n let data;\n try {\n data = await res.json();\n } catch {\n if (!current()) return;\n return gate(failure(\"Operator data could not be read.\"));\n }\n if (!current()) return;\n set({ data, session: \"ready\", gate: null, refreshing: false });\n }\n async function mutate(options) {\n const current = fence();\n set(options.busy);\n try {\n const payload = await options.request(current);\n if (!current()) return;\n if (options.reload) await loadData();\n if (!current()) return;\n set(options.done(payload));\n } catch (error) {\n if (!current()) return;\n if (options.reload) {\n try {\n await loadData();\n } catch {\n }\n if (!current()) return;\n }\n set(options.failed(failure(message(error, options.fallback))));\n }\n }\n function focusHandled() {\n if (state.pendingFocus !== null) set({ pendingFocus: null });\n }\n function setPage(page, focus = false) {\n state = withPage(state, page);\n if (focus) {\n state = {\n ...state,\n pendingFocus: state.session === \"ready\" ? `${page}Heading` : \"gateHeading\"\n };\n }\n for (const listener of listeners) listener();\n }\n function navigate(page, href) {\n history.pushState({ operatorPage: page }, \"\", href);\n setPage(page, true);\n }\n function setConnectorFilter(connectorFilter) {\n set({ connectorFilter });\n }\n function setActivitySearch(activitySearch) {\n set({ activitySearch });\n }\n function signInWithBearer(value) {\n gate(null);\n localStorage.setItem(TOKEN_KEY, value);\n void loadData().then(() => {\n if (state.session === \"ready\") set({ pendingFocus: `${state.page}Heading` });\n });\n }\n function forgetBearer() {\n localStorage.removeItem(TOKEN_KEY);\n gate(null);\n set({ pendingFocus: \"token\" });\n }\n function oauthAction(connector, action) {\n const disconnecting = action === \"disconnect\";\n const confirmed = window.confirm(\n disconnecting ? `Disconnect OAuth for ${connector}? Stored credentials and any pending authorization will be removed.` : `Restart OAuth for ${connector}? Stored credentials and any pending authorization will be replaced.`\n );\n if (!confirmed) return Promise.resolve();\n return mutate({\n request: (current) => operatorRequest(\n `/ui/oauth/${encodeURIComponent(connector)}`,\n disconnecting ? \"DELETE\" : \"POST\",\n current\n ),\n busy: { oauthNotice: null, oauthBusy: connector },\n done: (payload) => ({\n oauthBusy: null,\n pendingFocus: \"oauthNotice\",\n oauthNotice: info(\n disconnecting ? \"OAuth disconnected. Restart authorization when you are ready to reconnect.\" : payload?.message || \"Authorization restarted. Open the authorization link to reconnect.\"\n )\n }),\n failed: (notice) => ({\n oauthBusy: null,\n oauthNotice: notice,\n pendingFocus: \"oauthNotice\"\n }),\n fallback: \"OAuth action failed.\",\n reload: true\n });\n }\n function editCredential(connector) {\n set({ credentialEditing: connector, credentialNotice: null });\n }\n function refuseCredential(copy) {\n set({ credentialNotice: failure(copy), pendingFocus: \"credentialNotice\" });\n }\n function credentialMutation(connector, request, done, reload = true) {\n const land = (credentialNotice) => ({\n credentialBusy: null,\n credentialNotice,\n pendingFocus: \"credentialNotice\"\n });\n return mutate({\n request,\n busy: { credentialBusy: connector, credentialNotice: null },\n done: (payload) => land(done(payload)),\n failed: land,\n fallback: \"Credential action failed.\",\n reload\n });\n }\n function saveCredential(connector, body) {\n return credentialMutation(\n connector,\n (current) => operatorRequest(\n `/ui/credentials/${encodeURIComponent(connector)}`,\n \"PUT\",\n current,\n body\n ),\n () => {\n set({ credentialEditing: null });\n return info(\"Credential saved.\");\n }\n );\n }\n function removeCredential(connector) {\n const confirmed = window.confirm(\n \"Remove this credential? The connector will stop authenticating until a replacement is added.\"\n );\n if (!confirmed) return Promise.resolve();\n return credentialMutation(\n connector,\n (current) => operatorRequest(\n `/ui/credentials/${encodeURIComponent(connector)}`,\n \"DELETE\",\n current\n ),\n () => info(\"Credential removed.\")\n );\n }\n function testCredential(connector) {\n return credentialMutation(\n connector,\n (current) => operatorRequest(\n `/ui/credentials/${encodeURIComponent(connector)}/test`,\n \"POST\",\n current\n ),\n (payload) => {\n const copy = payload?.message || (payload?.ok ? \"Credential is valid.\" : \"Credential test failed.\");\n return payload?.ok ? info(copy) : failure(copy);\n },\n false\n );\n }\n async function loadAccessTokens() {\n const current = fence();\n set({ tokenPhase: \"loading\", tokenNotice: null });\n try {\n const payload = await operatorRequest(\"/ui/access-tokens\", \"GET\", current);\n if (!current()) return;\n set({ tokenPhase: \"ready\", tokens: payload?.accessTokens ?? [] });\n } catch (error) {\n if (!current()) return;\n set({\n tokenPhase: \"error\",\n tokenNotice: failure(\n message(error, \"Access tokens could not be loaded.\")\n )\n });\n }\n }\n function tokenFailure(tokenNotice) {\n return { tokenBusy: false, tokenNotice, pendingFocus: \"tokenNotice\" };\n }\n function createAccessToken(name) {\n if (!name) {\n set(tokenFailure(failure(\"Name the MCP client before creating a token.\")));\n return Promise.resolve(false);\n }\n let created = false;\n return mutate({\n request: (current) => operatorRequest(\"/ui/access-tokens\", \"POST\", current, { name }),\n busy: { tokenBusy: true, tokenNotice: null },\n done: (payload) => {\n const issued = payload?.accessToken;\n if (!payload?.token || !issued) {\n throw new Error(\"The created token was not returned.\");\n }\n created = true;\n return {\n tokenBusy: false,\n tokenPhase: \"ready\",\n tokens: [\n issued,\n ...state.tokens.filter((token) => token.id !== issued.id)\n ],\n createdToken: payload.token,\n tokenNotice: info(\"Access token created.\"),\n pendingFocus: \"tokenRevealHeading\"\n };\n },\n failed: tokenFailure,\n fallback: \"Access token could not be created.\"\n }).then(() => created);\n }\n function dismissCreatedToken() {\n set({ createdToken: null });\n }\n function renameAccessToken(id) {\n set({ tokenRenaming: id });\n }\n function accessTokenMutation(id, method, body, success, fallback) {\n return mutate({\n request: (current) => operatorRequest(\n `/ui/access-tokens/${encodeURIComponent(id)}`,\n method,\n current,\n body\n ),\n busy: { tokenBusy: true, tokenNotice: null },\n done: (payload) => ({\n tokenBusy: false,\n tokenRenaming: null,\n tokenNotice: info(success),\n pendingFocus: \"tokenNotice\",\n ...payload?.accessToken ? {\n tokens: state.tokens.map(\n (token) => token.id === id ? payload.accessToken : token\n )\n } : {}\n }),\n failed: tokenFailure,\n fallback\n });\n }\n function saveAccessTokenName(id, name) {\n return accessTokenMutation(\n id,\n \"PUT\",\n { name },\n \"Access token renamed.\",\n \"Access token could not be renamed.\"\n );\n }\n function revokeAccessToken(id) {\n const named = state.tokens.find((token) => token.id === id);\n const confirmed = window.confirm(\n `Revoke ${named?.name || \"this access token\"}? Its MCP client will immediately lose access.`\n );\n if (!confirmed) return Promise.resolve();\n return accessTokenMutation(\n id,\n \"DELETE\",\n void 0,\n \"Access token revoked.\",\n \"Access token could not be revoked.\"\n );\n }\n async function loadActivity(reset) {\n if (!state.data?.activityEnabled) return;\n const current = fence();\n set({\n activityPhase: \"loading\",\n activityNotice: null,\n ...reset ? { activityEvents: [], activityCursor: null } : {}\n });\n const params = new URLSearchParams({ limit: \"50\" });\n if (!reset && state.activityCursor) {\n params.set(\"cursor\", state.activityCursor);\n }\n try {\n const payload = await operatorRequest(\n `/ui/activity?${params}`,\n \"GET\",\n current\n );\n if (!current()) return;\n set({\n activityPhase: \"ready\",\n activityEvents: [\n ...reset ? [] : state.activityEvents,\n ...payload?.events ?? []\n ],\n activityCursor: payload?.nextCursor ?? null\n });\n } catch (error) {\n if (!current()) return;\n set({\n activityPhase: \"error\",\n activityNotice: failure(message(error, \"Activity could not be loaded.\"))\n });\n }\n }\n function signIn() {\n window.Clerk?.redirectToSignIn({\n signInFallbackRedirectUrl: window.location.href,\n signUpFallbackRedirectUrl: window.location.href\n });\n }\n function signOut() {\n if (auth.kind === \"cloudflare-access\") {\n gate(null);\n window.location.assign(\"/cdn-cgi/access/logout\");\n return;\n }\n const clerk = window.Clerk;\n gate(null);\n void clerk?.signOut({ redirectUrl: window.location.href });\n }\n async function boot() {\n const onPop = () => setPage(pageForPath(window.location.pathname), true);\n window.addEventListener(\"popstate\", onPop);\n window.addEventListener(\"pagehide\", dismissCreatedToken);\n if (auth.kind === \"clerk\") {\n const clerk = window.Clerk;\n if (!clerk) {\n const why = \"Clerk could not load. Check your network and try again.\";\n return gate(failure(why));\n }\n try {\n await clerk.load({\n ...auth.signInUrl ? { signInUrl: auth.signInUrl } : {},\n ...auth.signUpUrl ? { signUpUrl: auth.signUpUrl } : {},\n signInFallbackRedirectUrl: window.location.href,\n signUpFallbackRedirectUrl: window.location.href,\n afterSignOutUrl: window.location.href\n });\n let sessionId = clerk.session?.id ?? null;\n clerk.addListener((resources) => {\n const next = resources.session?.id ?? null;\n if (next === sessionId) return;\n sessionId = next;\n gate(null);\n void loadData();\n });\n } catch (error) {\n const why = message(error, \"unknown error\");\n return gate(failure(`Clerk could not initialize: ${why}`));\n }\n }\n await loadData();\n }\n\n // node_modules/preact/jsx-runtime/dist/jsxRuntime.module.js\n var f3 = 0;\n function u3(e3, t3, n2, o3, i3, u4) {\n t3 || (t3 = {});\n var a3, c3, p3 = t3;\n if (\"ref\" in p3) for (c3 in p3 = {}, t3) \"ref\" == c3 ? a3 = t3[c3] : p3[c3] = t3[c3];\n var l3 = { type: e3, props: p3, key: n2, ref: a3, __k: null, __: null, __b: 0, __e: null, __c: null, constructor: void 0, __v: --f3, __i: -1, __u: 0, __source: i3, __self: u4 };\n if (\"function\" == typeof e3 && (a3 = e3.defaultProps)) for (c3 in a3) void 0 === p3[c3] && (p3[c3] = a3[c3]);\n return l.vnode && l.vnode(l3), l3;\n }\n\n // src/operator-ui/app/parts.tsx\n function NoticeLine({\n id,\n notice,\n className = \"meta\"\n }) {\n return /* @__PURE__ */ u3(\n \"p\",\n {\n id,\n class: notice?.tone === \"error\" ? `${className} error-notice` : className,\n role: notice?.tone === \"error\" ? \"alert\" : \"status\",\n \"aria-live\": \"polite\",\n tabIndex: -1,\n children: notice ? notice.message : null\n }\n );\n }\n function Empty({ children }) {\n return /* @__PURE__ */ u3(\"p\", { class: \"empty\", children });\n }\n function Unavailable({ children }) {\n return /* @__PURE__ */ u3(\"div\", { class: \"unavailable\", children });\n }\n function PageLink({\n page,\n class: className,\n current,\n children\n }) {\n const href = PAGE_META[page].path;\n return /* @__PURE__ */ u3(\n \"a\",\n {\n class: className,\n href,\n ...current ? { \"aria-current\": \"page\" } : {},\n onClick: (event) => {\n if (event.defaultPrevented || event.button !== 0 || event.metaKey || event.ctrlKey || event.shiftKey || event.altKey) {\n return;\n }\n event.preventDefault();\n navigate(page, href);\n },\n children\n }\n );\n }\n function CopyButton({\n value,\n label,\n class: className = \"linklike\"\n }) {\n const [status, setStatus] = d2(\"idle\");\n h2(() => {\n if (status === \"idle\") return;\n const timer = window.setTimeout(() => setStatus(\"idle\"), 1600);\n return () => window.clearTimeout(timer);\n }, [status]);\n return /* @__PURE__ */ u3(\n \"button\",\n {\n class: className,\n type: \"button\",\n onClick: () => {\n navigator.clipboard.writeText(value).then(\n () => setStatus(\"copied\"),\n () => setStatus(\"failed\")\n );\n },\n children: status === \"copied\" ? \"Copied\" : status === \"failed\" ? \"Copy failed\" : label\n }\n );\n }\n\n // src/operator-ui/app/activity.tsx\n function ActivityRow({ event }) {\n const outcome = activityOutcomeClass(event.outcome);\n const stableId = actorStableId(event.actor);\n return /* @__PURE__ */ u3(\"article\", { class: `activity-item ${outcome}`, children: [\n /* @__PURE__ */ u3(\"div\", { class: \"activity-stamp\", children: [\n /* @__PURE__ */ u3(\n \"span\",\n {\n class: outcome === \"success\" ? \"dot ok\" : \"dot\",\n \"aria-hidden\": \"true\"\n }\n ),\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"time\", { class: \"activity-time\", dateTime: event.occurredAt, children: formatDate(event.occurredAt) }),\n /* @__PURE__ */ u3(\"div\", { class: \"activity-actor\", children: actorLabel(event.actor) }),\n stableId ? /* @__PURE__ */ u3(\"div\", { class: \"activity-actor-id mono\", children: stableId }) : null\n ] })\n ] }),\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"div\", { class: \"activity-address\", children: event.address }),\n /* @__PURE__ */ u3(\"div\", { class: \"activity-detail\", children: activityDetail(event) })\n ] }),\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"div\", { class: \"activity-outcome\", children: event.outcome }),\n /* @__PURE__ */ u3(\"div\", { class: \"activity-detail\", children: [\n event.durationMs,\n \" ms\"\n ] })\n ] })\n ] });\n }\n function ActivityPage({ state: state2 }) {\n const enabled = Boolean(state2.data?.activityEnabled);\n const loading = state2.activityPhase === \"loading\";\n const visible = filterActivity(state2.activityEvents, state2.activitySearch);\n return /* @__PURE__ */ u3(\"section\", { id: \"activityView\", children: /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"activityHeading\", class: \"pcap\", tabIndex: -1, children: \"Activity\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody\", children: [\n /* @__PURE__ */ u3(\"p\", { class: \"activity-copy\", id: \"activitySummary\", children: activitySummary(state2.activityEvents) }),\n !enabled ? /* @__PURE__ */ u3(Unavailable, { children: [\n \"Activity history is not configured. Add an\",\n \" \",\n /* @__PURE__ */ u3(\"span\", { class: \"mono\", children: \"activity.store\" }),\n \" with a list reader to enable this page.\"\n ] }) : /* @__PURE__ */ u3(\"div\", { id: \"activityAvailable\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"row activity-controls\", children: [\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: \"activitySearch\",\n type: \"search\",\n placeholder: \"Search user, tool, or outcome…\",\n \"aria-label\": \"Search loaded activity\",\n value: state2.activitySearch,\n onInput: (event) => setActivitySearch(event.currentTarget.value)\n }\n ),\n /* @__PURE__ */ u3(\n \"button\",\n {\n id: \"refreshActivity\",\n class: \"linklike\",\n type: \"button\",\n disabled: loading,\n onClick: () => void loadActivity(true),\n children: loading ? \"Loading…\" : \"Refresh\"\n }\n )\n ] }),\n /* @__PURE__ */ u3(NoticeLine, { id: \"activityNotice\", notice: state2.activityNotice }),\n /* @__PURE__ */ u3(\n \"div\",\n {\n id: \"activityList\",\n class: \"activity-ledger\",\n \"aria-busy\": loading ? \"true\" : \"false\",\n children: loading && state2.activityEvents.length === 0 ? /* @__PURE__ */ u3(\"div\", { class: \"activity-empty\", children: \"Loading activity…\" }) : state2.activityPhase === \"error\" && state2.activityEvents.length === 0 ? /* @__PURE__ */ u3(\"p\", { class: \"activity-empty\", children: /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n onClick: () => void loadActivity(true),\n children: \"Try loading activity again\"\n }\n ) }) : visible.length === 0 ? /* @__PURE__ */ u3(\"div\", { class: \"activity-empty\", children: state2.activitySearch.trim() ? \"No loaded activity matches this search.\" : \"No connector tool calls recorded yet.\" }) : visible.map((event, index) => /* @__PURE__ */ u3(\n ActivityRow,\n {\n event\n },\n `${event.occurredAt}-${event.address}-${index}`\n ))\n }\n ),\n state2.activityCursor ? /* @__PURE__ */ u3(\n \"button\",\n {\n id: \"moreActivity\",\n class: \"linklike activity-more\",\n type: \"button\",\n disabled: loading,\n onClick: () => void loadActivity(false),\n children: loading ? \"Loading…\" : \"Load older\"\n }\n ) : null\n ] })\n ] })\n ] }) });\n }\n\n // src/operator-ui/model.ts\n function filterUiConnectors(connectors, query) {\n const q2 = query.trim().toLowerCase();\n const filtered = [];\n for (const connector of connectors) {\n const connectorText = [\n connector.id,\n connector.title,\n connector.description,\n connector.status\n ].join(\" \").toLowerCase();\n const connectorMatches = Boolean(q2 && connectorText.includes(q2));\n const tools = connector.tools.filter(\n (tool) => !q2 || connectorMatches || `${tool.name} ${tool.description ?? \"\"}`.toLowerCase().includes(q2)\n );\n if (q2 && tools.length === 0 && !connectorMatches) continue;\n filtered.push({ connector, tools });\n }\n return filtered;\n }\n\n // src/operator-ui/app/connections.tsx\n var DRIFT_HEADING = {\n clean: \"Catalog drift · none\",\n warning: \"Catalog drift · review\",\n unavailable: \"Catalog drift · not observed\"\n };\n function DriftPanel({ connector }) {\n const drift = connector.catalogDrift;\n const state2 = driftState(drift);\n return /* @__PURE__ */ u3(\n \"div\",\n {\n id: `drift-${connector.id}`,\n class: `connector-drift ${state2}`,\n \"data-drift\": state2,\n children: [\n /* @__PURE__ */ u3(\"p\", { class: \"cap\", children: DRIFT_HEADING[state2] }),\n /* @__PURE__ */ u3(\"p\", { class: \"meta drift-summary\", children: driftSummary(drift) }),\n state2 === \"unavailable\" ? null : /* @__PURE__ */ u3(\"ul\", { class: \"drift-counts\", children: driftCounts(drift).map(({ key, label, count }) => /* @__PURE__ */ u3(\"li\", { class: count > 0 ? \"drift-count flagged\" : \"drift-count\", children: [\n /* @__PURE__ */ u3(\"span\", { class: \"drift-count-value\", children: count }),\n /* @__PURE__ */ u3(\"span\", { class: \"drift-count-label\", children: label })\n ] }, key)) })\n ]\n }\n );\n }\n function ConnectorCard({\n connector,\n tools,\n expanded,\n oauthManagement,\n busy\n }) {\n const name = connector.title || connector.id;\n const authorization = safeHttpHref(connector.authorizationUrl);\n return /* @__PURE__ */ u3(\"div\", { class: \"card\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"connector-head\", children: [\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"div\", { class: \"connector-title\", children: [\n /* @__PURE__ */ u3(\"span\", { class: `dot ${connector.status}`, \"aria-hidden\": \"true\" }),\n /* @__PURE__ */ u3(\"h2\", { children: name })\n ] }),\n connector.description ? /* @__PURE__ */ u3(\"p\", { class: \"connector-description meta\", children: connector.description }) : null\n ] }),\n /* @__PURE__ */ u3(\"div\", { class: \"connector-state cap\", children: [\n connectorStatusLabel(connector.status),\n \" ·\",\n \" \",\n toolCountLabel(connector.toolCount),\n /* @__PURE__ */ u3(\"br\", {}),\n /* @__PURE__ */ u3(\"span\", { class: \"mono\", children: connector.id })\n ] })\n ] }),\n connector.message ? /* @__PURE__ */ u3(\"p\", { class: \"connector-message msg\", children: connector.message }) : null,\n connector.authorizationUrl ? /* @__PURE__ */ u3(\"p\", { class: authorization ? \"connector-auth\" : \"connector-auth meta\", children: authorization ? /* @__PURE__ */ u3(\n \"a\",\n {\n class: \"linklike\",\n href: authorization,\n target: \"_blank\",\n rel: \"noopener\",\n children: \"Authorize connector →\"\n }\n ) : `Authorization URL: ${connector.authorizationUrl}` }) : null,\n /* @__PURE__ */ u3(DriftPanel, { connector }),\n connector.catalogAccess ? /* @__PURE__ */ u3(\"p\", { class: \"meta\", children: [\n \"Last agent catalog read · \",\n connector.catalogAccess.state,\n \" ·\",\n \" \",\n new Date(connector.catalogAccess.observedAt).toLocaleString()\n ] }) : null,\n connector.oauth && oauthManagement ? /* @__PURE__ */ u3(\"div\", { class: \"credential-actions\", children: [\n /* @__PURE__ */ u3(\n \"button\",\n {\n type: \"button\",\n class: \"linklike danger\",\n \"aria-label\": `Disconnect OAuth for ${name}`,\n disabled: busy,\n onClick: () => void oauthAction(connector.id, \"disconnect\"),\n children: \"Disconnect OAuth\"\n }\n ),\n /* @__PURE__ */ u3(\n \"button\",\n {\n type: \"button\",\n class: \"linklike\",\n \"aria-label\": `${connector.status === \"ok\" ? \"Reconnect OAuth for\" : \"Restart authorization for\"} ${name}`,\n disabled: busy,\n onClick: () => void oauthAction(connector.id, \"reconnect\"),\n children: connector.status === \"ok\" ? \"Reconnect OAuth\" : \"Restart authorization\"\n }\n )\n ] }) : null,\n connector.credential ? /* @__PURE__ */ u3(\"p\", { class: \"connector-auth\", children: /* @__PURE__ */ u3(PageLink, { page: \"credentials\", class: \"linklike\", children: \"Manage credential →\" }) }) : null,\n tools.length ? /* @__PURE__ */ u3(\"details\", { open: expanded, children: [\n /* @__PURE__ */ u3(\"summary\", { class: \"linklike\", children: [\n \"Show tools (\",\n tools.length,\n \")\"\n ] }),\n /* @__PURE__ */ u3(\"div\", { class: \"tool-list\", children: tools.map((tool) => /* @__PURE__ */ u3(\"div\", { class: \"tool\", children: [\n /* @__PURE__ */ u3(\"code\", { children: tool.address }),\n tool.description ? /* @__PURE__ */ u3(\"span\", { class: \"td\", children: tool.description }) : null\n ] }, tool.address)) })\n ] }) : null\n ] });\n }\n function ConnectionsPage({ state: state2 }) {\n const data = state2.data;\n const query = state2.connectorFilter.trim();\n const filtered = data ? filterUiConnectors(data.connectors, query) : [];\n return /* @__PURE__ */ u3(\"section\", { id: \"connectionsView\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"connectionsHeading\", class: \"pcap\", tabIndex: -1, children: \"Connections\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody lead-copy\", children: [\n /* @__PURE__ */ u3(\"p\", { children: \"Use this endpoint to give an MCP client access to the tools below.\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"endpoint\", children: /* @__PURE__ */ u3(\"div\", { class: \"endpoint-row\", children: [\n /* @__PURE__ */ u3(\"code\", { id: \"mcpUrl\", class: \"mono\", children: mcpUrl }),\n /* @__PURE__ */ u3(CopyButton, { value: mcpUrl, label: \"Copy URL\" })\n ] }) }),\n /* @__PURE__ */ u3(\"p\", { class: \"cap\", id: \"serverInfo\", children: data ? `${data.serverInfo?.name || productName} v${data.connectaVersion || \"?\"}` : productOperatorLabel }),\n /* @__PURE__ */ u3(NoticeLine, { id: \"oauthNotice\", notice: state2.oauthNotice })\n ] })\n ] }),\n /* @__PURE__ */ u3(\"section\", { class: \"section pgrid\", \"aria-labelledby\": \"connectorLedgerHeading\", children: [\n /* @__PURE__ */ u3(\"h2\", { class: \"pcap\", id: \"connectorLedgerHeading\", children: \"Connectors\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"row toolbar\", children: /* @__PURE__ */ u3(\n \"input\",\n {\n id: \"filter\",\n type: \"search\",\n placeholder: \"Filter connectors or tools…\",\n \"aria-label\": \"Filter connectors or tools\",\n value: state2.connectorFilter,\n onInput: (event) => setConnectorFilter(event.currentTarget.value)\n }\n ) }),\n /* @__PURE__ */ u3(\n \"div\",\n {\n id: \"list\",\n class: \"connector-tools\",\n \"aria-busy\": state2.refreshing || !data ? \"true\" : \"false\",\n children: !data ? /* @__PURE__ */ u3(Empty, { children: \"Loading connectors…\" }) : filtered.length === 0 ? /* @__PURE__ */ u3(Empty, { children: query ? \"No connectors or tools match this filter.\" : \"No connectors are declared in this deployment.\" }) : filtered.map(({ connector, tools }) => /* @__PURE__ */ u3(\n ConnectorCard,\n {\n connector,\n tools,\n expanded: Boolean(query),\n oauthManagement: data.oauthManagement,\n busy: state2.oauthBusy === connector.id\n },\n connector.id\n ))\n }\n )\n ] })\n ] })\n ] });\n }\n\n // src/operator-ui/app/credentials.tsx\n function CredentialForm({\n connector,\n credential,\n busy\n }) {\n const fields = credential.fields ?? [];\n const [values, setValues] = d2({});\n const single = fields.length === 0;\n const inputId = `credential-input-${connector}`;\n const submit = () => {\n if (single) {\n const value = (values.value ?? \"\").trim();\n if (!value) return refuseCredential(\"Paste a credential before saving.\");\n return void saveCredential(connector, { value });\n }\n const entries = {};\n for (const field of fields) {\n const value = (values[field.name] ?? \"\").trim();\n if (!value) {\n return refuseCredential(\n \"Complete every credential field before saving.\"\n );\n }\n entries[field.name] = value;\n }\n void saveCredential(connector, { values: entries });\n };\n return /* @__PURE__ */ u3(\"div\", { class: \"credential-form\", \"data-credential-form\": connector, children: [\n single ? /* @__PURE__ */ u3(S, { children: [\n /* @__PURE__ */ u3(\"label\", { class: \"visually-hidden\", for: inputId, children: credential.label }),\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: inputId,\n type: \"password\",\n \"aria-label\": credential.label,\n placeholder: credential.placeholder || \"Paste credential\",\n autocomplete: \"new-password\",\n autocapitalize: \"none\",\n spellcheck: false,\n value: values.value ?? \"\",\n onInput: (event) => setValues({ value: event.currentTarget.value })\n }\n )\n ] }) : /* @__PURE__ */ u3(\"div\", { class: \"credential-fields\", children: fields.map((field, index) => {\n const id = `credential-input-${connector}-${index}`;\n return /* @__PURE__ */ u3(\"div\", { class: \"credential-field\", children: [\n /* @__PURE__ */ u3(\"label\", { for: id, children: field.label }),\n /* @__PURE__ */ u3(\n \"input\",\n {\n id,\n type: field.inputType || \"password\",\n placeholder: field.placeholder || field.label,\n autocomplete: (field.inputType ?? \"password\") === \"password\" ? \"new-password\" : \"off\",\n autocapitalize: \"none\",\n spellcheck: false,\n value: values[field.name] ?? \"\",\n onInput: (event) => setValues({\n ...values,\n [field.name]: event.currentTarget.value\n })\n }\n )\n ] }, field.name);\n }) }),\n /* @__PURE__ */ u3(\"button\", { class: \"linklike\", type: \"button\", disabled: busy, onClick: submit, children: busy ? \"Saving…\" : \"Save\" }),\n /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => editCredential(null),\n children: \"Cancel\"\n }\n )\n ] });\n }\n function CredentialCard({\n connector,\n credential,\n editing,\n busy\n }) {\n const configured = Boolean(credential.configured);\n const removable = configured || Boolean(credential.removable);\n return /* @__PURE__ */ u3(\n \"section\",\n {\n class: \"credential-card\",\n id: `credential-${connector.id}`,\n \"aria-labelledby\": `credential-title-${connector.id}`,\n children: [\n /* @__PURE__ */ u3(\"div\", { class: \"credential-head\", children: [\n /* @__PURE__ */ u3(\"div\", { class: \"connector-title\", children: [\n /* @__PURE__ */ u3(\n \"span\",\n {\n class: `dot ${configured ? \"ok\" : \"auth_required\"}`,\n \"aria-hidden\": \"true\"\n }\n ),\n /* @__PURE__ */ u3(\"h2\", { id: `credential-title-${connector.id}`, children: connector.title || connector.id })\n ] }),\n /* @__PURE__ */ u3(\"span\", { class: \"credential-state\", children: credentialStateLabel(credential) })\n ] }),\n /* @__PURE__ */ u3(\"p\", { class: \"mono\", children: [\n connector.id,\n \" · \",\n credential.label\n ] }),\n credential.description ? /* @__PURE__ */ u3(\"p\", { class: \"credential-copy meta\", children: credential.description }) : null,\n credential.fields?.length ? /* @__PURE__ */ u3(\"div\", { class: \"credential-field-summary\", children: credential.fields.map((field) => /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"span\", { children: field.label }),\n /* @__PURE__ */ u3(\"span\", { class: \"meta\", children: field.configured ? `configured · ••••${field.lastFour ?? \"\"}${field.updatedAt ? ` · updated ${formatDate(field.updatedAt)}` : \"\"}` : \"not configured\" })\n ] }, field.name)) }) : null,\n credential.error ? /* @__PURE__ */ u3(\"div\", { class: \"msg\", children: credential.error }) : null,\n credential.notice ? /* @__PURE__ */ u3(\"p\", { class: \"credential-copy meta\", children: credential.notice }) : null,\n /* @__PURE__ */ u3(\"div\", { class: \"credential-actions\", children: [\n /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => editCredential(editing ? null : connector.id),\n children: removable ? \"Replace\" : \"Add credential\"\n }\n ),\n configured && credential.testable ? /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => void testCredential(connector.id),\n children: busy ? \"Working…\" : \"Test\"\n }\n ) : null,\n removable ? /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike danger\",\n type: \"button\",\n disabled: busy,\n onClick: () => void removeCredential(connector.id),\n children: \"Remove\"\n }\n ) : null\n ] }),\n editing ? /* @__PURE__ */ u3(\n CredentialForm,\n {\n connector: connector.id,\n credential,\n busy\n }\n ) : null\n ]\n }\n );\n }\n function CredentialsPage({ state: state2 }) {\n const data = state2.data;\n const available = data?.credentialManagement === \"available\";\n const slots = (data?.connectors ?? []).filter(\n (connector) => Boolean(connector.credential)\n );\n return /* @__PURE__ */ u3(\"section\", { id: \"credentialsView\", children: /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"credentialsHeading\", class: \"pcap\", tabIndex: -1, children: \"Credentials\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody\", children: [\n /* @__PURE__ */ u3(\"p\", { class: \"activity-copy\", children: \"Rotate operator-managed connector credentials. Stored values are never returned or displayed.\" }),\n /* @__PURE__ */ u3(NoticeLine, { id: \"credentialNotice\", notice: state2.credentialNotice }),\n !available ? /* @__PURE__ */ u3(Unavailable, { children: credentialUnavailableCopy(data?.credentialManagement) }) : /* @__PURE__ */ u3(\n \"div\",\n {\n id: \"credentialList\",\n class: \"credential-ledger\",\n \"aria-busy\": state2.credentialBusy ? \"true\" : \"false\",\n children: slots.length === 0 ? /* @__PURE__ */ u3(Empty, { children: \"No connector in this deployment declares a credential slot yet.\" }) : slots.map((connector) => /* @__PURE__ */ u3(\n CredentialCard,\n {\n connector,\n credential: connector.credential,\n editing: state2.credentialEditing === connector.id,\n busy: state2.credentialBusy === connector.id\n },\n connector.id\n ))\n }\n )\n ] })\n ] }) });\n }\n\n // src/operator-ui/app/tokens.tsx\n function CreateForm({ busy }) {\n const [name, setName] = d2(\"\");\n return /* @__PURE__ */ u3(\n \"form\",\n {\n id: \"tokenCreateForm\",\n class: \"token-create\",\n onSubmit: (event) => {\n event.preventDefault();\n void createAccessToken(name.trim()).then((created) => {\n if (created) setName(\"\");\n });\n },\n children: [\n /* @__PURE__ */ u3(\"label\", { for: \"tokenName\", children: \"Client name\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"row\", children: [\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: \"tokenName\",\n type: \"text\",\n maxLength: 80,\n placeholder: \"Claude desktop, ChatGPT production…\",\n autocomplete: \"off\",\n value: name,\n onInput: (event) => setName(event.currentTarget.value)\n }\n ),\n /* @__PURE__ */ u3(\"button\", { id: \"createToken\", class: \"linklike\", type: \"submit\", disabled: busy, children: busy ? \"Creating…\" : \"Create token\" })\n ] })\n ]\n }\n );\n }\n function Reveal({ token }) {\n return /* @__PURE__ */ u3(\n \"section\",\n {\n id: \"tokenReveal\",\n class: \"token-reveal\",\n \"aria-labelledby\": \"tokenRevealHeading\",\n children: [\n /* @__PURE__ */ u3(\"div\", { class: \"token-reveal-head\", children: [\n /* @__PURE__ */ u3(\"h2\", { id: \"tokenRevealHeading\", tabIndex: -1, children: \"Copy this token now\" }),\n /* @__PURE__ */ u3(\"span\", { class: \"cap\", children: \"Shown once\" })\n ] }),\n /* @__PURE__ */ u3(\"p\", { class: \"meta\", children: \"Store it in the MCP client before leaving this page. It cannot be displayed again.\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"endpoint-row token-secret\", children: [\n /* @__PURE__ */ u3(\"code\", { id: \"createdToken\", class: \"mono\", children: token }),\n /* @__PURE__ */ u3(CopyButton, { value: token, label: \"Copy token\" })\n ] }),\n /* @__PURE__ */ u3(\"button\", { class: \"linklike\", type: \"button\", onClick: dismissCreatedToken, children: \"I stored it\" })\n ]\n }\n );\n }\n function TokenCard({\n token,\n renaming,\n busy\n }) {\n const [name, setName] = d2(token.name);\n const revoked = Boolean(token.revokedAt);\n return /* @__PURE__ */ u3(\n \"section\",\n {\n class: revoked ? \"token-card revoked\" : \"token-card\",\n \"aria-labelledby\": `access-token-${token.id}`,\n children: [\n /* @__PURE__ */ u3(\"div\", { class: \"token-card-head\", children: [\n /* @__PURE__ */ u3(\"div\", { children: [\n /* @__PURE__ */ u3(\"h2\", { id: `access-token-${token.id}`, children: token.name }),\n /* @__PURE__ */ u3(\"p\", { class: \"mono\", children: [\n token.tokenPrefix,\n \"…\"\n ] })\n ] }),\n /* @__PURE__ */ u3(\"div\", { class: \"cap\", children: revoked ? `Revoked ${formatDate(token.revokedAt)}` : `Created ${formatDate(token.createdAt)}` })\n ] }),\n /* @__PURE__ */ u3(\"div\", { class: \"credential-actions\", children: [\n /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => {\n setName(token.name);\n renameAccessToken(renaming ? null : token.id);\n },\n children: \"Rename\"\n }\n ),\n revoked ? null : /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike danger\",\n type: \"button\",\n disabled: busy,\n onClick: () => void revokeAccessToken(token.id),\n children: \"Revoke\"\n }\n )\n ] }),\n renaming ? /* @__PURE__ */ u3(\n \"form\",\n {\n class: \"credential-form\",\n onSubmit: (event) => {\n event.preventDefault();\n const next = name.trim();\n if (next) void saveAccessTokenName(token.id, next);\n },\n children: [\n /* @__PURE__ */ u3(\"label\", { class: \"visually-hidden\", for: `token-name-${token.id}`, children: \"Token name\" }),\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: `token-name-${token.id}`,\n type: \"text\",\n maxLength: 80,\n autocomplete: \"off\",\n value: name,\n onInput: (event) => setName(event.currentTarget.value)\n }\n ),\n /* @__PURE__ */ u3(\"button\", { class: \"linklike\", type: \"submit\", disabled: busy, children: \"Save name\" }),\n /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n disabled: busy,\n onClick: () => renameAccessToken(null),\n children: \"Cancel\"\n }\n )\n ]\n }\n ) : null\n ]\n }\n );\n }\n function TokensPage({ state: state2 }) {\n const available = state2.data?.accessTokenManagement === \"available\";\n return /* @__PURE__ */ u3(\"section\", { id: \"tokensView\", children: /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"tokensHeading\", class: \"pcap\", tabIndex: -1, children: \"Access tokens\" }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody\", children: [\n /* @__PURE__ */ u3(\"p\", { class: \"activity-copy\", children: \"Create named Bearer tokens for MCP clients. Each secret is shown once; revoke it when that client should lose access.\" }),\n /* @__PURE__ */ u3(NoticeLine, { id: \"tokenNotice\", notice: state2.tokenNotice }),\n !available ? /* @__PURE__ */ u3(Unavailable, { children: accessTokenUnavailableCopy(state2.data?.accessTokenManagement) }) : /* @__PURE__ */ u3(\"div\", { id: \"tokenAvailable\", children: [\n state2.createdToken ? /* @__PURE__ */ u3(Reveal, { token: state2.createdToken }) : /* @__PURE__ */ u3(CreateForm, { busy: state2.tokenBusy }),\n /* @__PURE__ */ u3(\n \"div\",\n {\n id: \"tokenList\",\n class: \"token-ledger\",\n \"aria-busy\": state2.tokenPhase === \"loading\" ? \"true\" : \"false\",\n children: state2.tokenPhase === \"loading\" ? /* @__PURE__ */ u3(Empty, { children: \"Loading access tokens…\" }) : state2.tokenPhase === \"error\" ? /* @__PURE__ */ u3(\"p\", { class: \"empty\", children: /* @__PURE__ */ u3(\n \"button\",\n {\n class: \"linklike\",\n type: \"button\",\n onClick: () => void loadAccessTokens(),\n children: \"Try loading access tokens again\"\n }\n ) }) : state2.tokens.length === 0 ? /* @__PURE__ */ u3(Empty, { children: \"No access tokens yet. Name the first MCP client above.\" }) : state2.tokens.map((token) => /* @__PURE__ */ u3(\n TokenCard,\n {\n token,\n renaming: state2.tokenRenaming === token.id,\n busy: state2.tokenBusy\n },\n token.id\n ))\n }\n )\n ] })\n ] })\n ] }) });\n }\n\n // src/operator-ui/app/main.tsx\n function useOperatorState() {\n const [, bump] = y2((count) => count + 1, 0);\n const snapshot = getState();\n _2(() => {\n const unsubscribe = subscribe(() => bump(void 0));\n if (getState() !== snapshot) bump(void 0);\n return unsubscribe;\n }, []);\n return snapshot;\n }\n function visiblePages(state2) {\n return OPERATOR_PAGES.filter((page) => {\n if (page === \"credentials\") {\n return state2.data?.credentialManagement === \"available\";\n }\n if (page === \"tokens\") {\n return state2.data?.accessTokenManagement === \"available\";\n }\n if (page === \"activity\") return Boolean(state2.data?.activityEnabled);\n return true;\n });\n }\n function OperatorNav() {\n const state2 = useOperatorState();\n if (state2.session !== \"ready\") return null;\n return /* @__PURE__ */ u3(\"div\", { class: \"mast-actions\", children: [\n /* @__PURE__ */ u3(\"nav\", { class: \"page-nav\", \"aria-label\": \"Operator pages\", children: visiblePages(state2).map((page) => /* @__PURE__ */ u3(\n PageLink,\n {\n page,\n class: \"navlink\",\n current: state2.page === page,\n children: PAGE_META[page].label\n },\n page\n )) }),\n /* @__PURE__ */ u3(\"div\", { class: \"session-actions\", \"aria-label\": \"Session actions\", children: auth.kind === \"clerk\" || auth.kind === \"cloudflare-access\" ? /* @__PURE__ */ u3(\"button\", { class: \"navlink\", type: \"button\", onClick: signOut, children: \"Sign out\" }) : /* @__PURE__ */ u3(\"button\", { class: \"navlink\", type: \"button\", onClick: forgetBearer, children: \"Change token\" }) })\n ] });\n }\n function Gate({ state: state2 }) {\n const [token, setToken] = d2(\"\");\n const signedIn = auth.kind === \"clerk\" && Boolean(window.Clerk?.user);\n const loading = state2.session === \"loading\";\n return /* @__PURE__ */ u3(\"section\", { id: \"gate\", children: /* @__PURE__ */ u3(\"div\", { class: \"lead pgrid\", children: [\n /* @__PURE__ */ u3(\"h1\", { id: \"gateHeading\", class: \"pcap\", tabIndex: -1, children: PAGE_META[state2.page].label }),\n /* @__PURE__ */ u3(\"div\", { class: \"pbody lead-copy\", children: [\n /* @__PURE__ */ u3(\"p\", { children: productDescription }),\n /* @__PURE__ */ u3(\"p\", { id: \"gateCopy\", class: \"meta\", children: loading ? \"Checking your session…\" : gateCopy(auth.kind, signedIn) }),\n loading ? null : auth.kind === \"clerk\" ? /* @__PURE__ */ u3(\"div\", { id: \"clerkGate\", class: \"actions gate-actions\", children: signedIn ? /* @__PURE__ */ u3(\"button\", { class: \"linklike\", type: \"button\", onClick: signOut, children: \"Sign out\" }) : /* @__PURE__ */ u3(\"button\", { id: \"signin\", class: \"linklike\", type: \"button\", onClick: signIn, children: \"Team sign in\" }) }) : auth.kind === \"cloudflare-access\" ? /* @__PURE__ */ u3(\"div\", { class: \"actions gate-actions\", children: /* @__PURE__ */ u3(\"button\", { class: \"linklike\", type: \"button\", onClick: signOut, children: \"Sign out of Cloudflare Access\" }) }) : /* @__PURE__ */ u3(\n \"form\",\n {\n id: \"tokenGate\",\n class: \"row gate-actions\",\n onSubmit: (event) => {\n event.preventDefault();\n const value = token.trim();\n if (!value) return;\n setToken(\"\");\n signInWithBearer(value);\n },\n children: [\n /* @__PURE__ */ u3(\n \"input\",\n {\n id: \"token\",\n type: \"password\",\n placeholder: \"Bearer token\",\n autocomplete: \"off\",\n \"aria-label\": \"Bearer token\",\n value: token,\n onInput: (event) => setToken(event.currentTarget.value)\n }\n ),\n /* @__PURE__ */ u3(\"button\", { id: \"save\", class: \"linklike\", type: \"submit\", children: \"Open operator pages\" })\n ]\n }\n ),\n /* @__PURE__ */ u3(NoticeLine, { id: \"err\", notice: state2.gate, className: \"\" })\n ] })\n ] }) });\n }\n function CurrentPage({ state: state2 }) {\n if (state2.page === \"credentials\") return /* @__PURE__ */ u3(CredentialsPage, { state: state2 });\n if (state2.page === \"tokens\") return /* @__PURE__ */ u3(TokensPage, { state: state2 });\n if (state2.page === \"activity\") return /* @__PURE__ */ u3(ActivityPage, { state: state2 });\n return /* @__PURE__ */ u3(ConnectionsPage, { state: state2 });\n }\n function OperatorApp() {\n const state2 = useOperatorState();\n const ready = state2.session === \"ready\";\n h2(() => {\n document.title = `${PAGE_META[state2.page].label} — ${titleSuffix}`;\n }, [state2.page]);\n h2(() => {\n if (!ready) return;\n if (state2.page === \"tokens\" && state2.data?.accessTokenManagement === \"available\" && state2.tokenPhase === \"idle\") {\n void loadAccessTokens();\n }\n if (state2.page === \"activity\" && state2.data?.activityEnabled && state2.activityPhase === \"idle\") {\n void loadActivity(true);\n }\n });\n h2(() => {\n if (!state2.pendingFocus) return;\n document.getElementById(state2.pendingFocus)?.focus();\n focusHandled();\n }, [state2.pendingFocus]);\n return ready ? /* @__PURE__ */ u3(\"div\", { id: \"app\", children: /* @__PURE__ */ u3(CurrentPage, { state: state2 }) }) : /* @__PURE__ */ u3(Gate, { state: state2 });\n }\n function mount(id, view) {\n const host = document.getElementById(id);\n if (!host) return;\n host.textContent = \"\";\n R(view, host);\n }\n mount(\"operatorNav\", /* @__PURE__ */ u3(OperatorNav, {}));\n mount(\"operatorContent\", /* @__PURE__ */ u3(OperatorApp, {}));\n void boot();\n})();\n"; diff --git a/src/operator-ui/model.ts b/src/operator-ui/model.ts index 7a031674..aac2b1fd 100644 --- a/src/operator-ui/model.ts +++ b/src/operator-ui/model.ts @@ -67,13 +67,13 @@ export interface UiConnector { export type CredentialManagementCapability = | "available" - | "requires_clerk" + | "requires_operator" | "vault_not_configured" | "no_slots"; export type AccessTokenManagementCapability = | "available" - | "requires_clerk" + | "requires_operator" | "not_configured"; export interface UiData { @@ -84,7 +84,7 @@ export interface UiData { activityEnabled: boolean; credentialManagement: CredentialManagementCapability; accessTokenManagement: AccessTokenManagementCapability; - /** True only for an eligible Clerk operator. */ + /** True only for an eligible interactive operator. */ oauthManagement: boolean; } diff --git a/src/operator-ui/view.ts b/src/operator-ui/view.ts index 067a82ae..9e8fa593 100644 --- a/src/operator-ui/view.ts +++ b/src/operator-ui/view.ts @@ -223,7 +223,7 @@ export function credentialUnavailableCopy( if (capability === "vault_not_configured") { return "Credential storage is not configured. Set credentials.encryptionKey before managing connector credentials here."; } - return "Credential management requires an eligible Clerk operator. Bearer-authenticated sessions can inspect connections but cannot manage stored credentials."; + return "Credential management requires an eligible interactive operator. Bearer-authenticated sessions can inspect connections but cannot manage stored credentials."; } export function accessTokenUnavailableCopy( @@ -232,7 +232,7 @@ export function accessTokenUnavailableCopy( if (capability === "not_configured") { return "Access tokens are not configured for this deployment. Add accessTokens to the deployment configuration to enable them."; } - return "Access token management requires an eligible Clerk operator. A Bearer token can connect to MCP, but it cannot create or revoke other tokens."; + return "Access token management requires an eligible interactive operator. A Bearer token can connect to MCP, but it cannot create or revoke other tokens."; } export function connectorStatusLabel(status: string): string { @@ -418,8 +418,11 @@ export function credentialStateLabel(credential: { : masked; } -/** Gate copy for the two inbound-auth shapes, so the sign-in state is never a blank page. */ +/** Gate copy for each browser-auth shape, so the sign-in state is never a blank page. */ export function gateCopy(kind: string, signedIn: boolean): string { + if (kind === "cloudflare-access") { + return "Cloudflare Access admitted this browser, but the current identity cannot open deployment-wide operator pages."; + } if (kind !== "clerk") { return "Paste an operator bearer token to open this page. Nothing is requested until you do."; } diff --git a/src/routes/access-tokens.ts b/src/routes/access-tokens.ts index 82fc698d..d463b314 100644 --- a/src/routes/access-tokens.ts +++ b/src/routes/access-tokens.ts @@ -47,7 +47,7 @@ async function readName( } /** - * Clerk-only lifecycle for deployment access tokens. The access token itself + * Interactive-operator lifecycle for deployment access tokens. The token itself * is deliberately never an administrator credential and cannot reach here. */ export async function routeAccessTokens( @@ -78,6 +78,7 @@ export async function routeAccessTokens( baseUrl, opts.auth, "access token management", + context.runtimeContext, ); if (!admin.ok) return admin.response; diff --git a/src/routes/activity.ts b/src/routes/activity.ts index 4209780b..8d1704c9 100644 --- a/src/routes/activity.ts +++ b/src/routes/activity.ts @@ -169,12 +169,12 @@ async function enrichActivityActorLabels( export async function routeActivity( context: RouteContext, ): Promise { - const { path, request, url, baseUrl, opts } = context; + const { path, request, url, baseUrl, opts, runtimeContext } = context; if (path !== "/ui/activity") return null; if (request.method !== "GET") { return privateJson({ error: "method not allowed" }, { status: 405 }); } - const authz = await authorize(request, baseUrl, opts.auth); + const authz = await authorize(request, baseUrl, opts.auth, runtimeContext); if (!authz.ok) return authz.response; if ( opts.activityReadGate && diff --git a/src/routes/credentials.ts b/src/routes/credentials.ts index 228ec873..ec3b26e5 100644 --- a/src/routes/credentials.ts +++ b/src/routes/credentials.ts @@ -138,6 +138,8 @@ async function handleCredentialRequest( request, baseUrl, opts.auth, + "credential management", + context.runtimeContext, ); if (!admin.ok) return admin.response; diff --git a/src/routes/mcp.ts b/src/routes/mcp.ts index b16601eb..6f87ee13 100644 --- a/src/routes/mcp.ts +++ b/src/routes/mcp.ts @@ -398,7 +398,12 @@ export function createMcpRoute( throw error; } try { - const authz = await authorize(request, baseUrl, opts.auth); + const authz = await authorize( + request, + baseUrl, + opts.auth, + runtimeContext, + ); if (!authz.ok) { return releaseAdmissionWithResponse( withMcpCors(authz.response), diff --git a/src/routes/oauth.ts b/src/routes/oauth.ts index 1abfbe91..e9c3d531 100644 --- a/src/routes/oauth.ts +++ b/src/routes/oauth.ts @@ -31,6 +31,7 @@ async function handleOAuthManagementRequest( baseUrl, opts.auth, "OAuth management", + context.runtimeContext, ); if (!admin.ok) return admin.response; diff --git a/src/routes/shared.ts b/src/routes/shared.ts index 94e1b027..94866ac6 100644 --- a/src/routes/shared.ts +++ b/src/routes/shared.ts @@ -9,6 +9,7 @@ import type { ConnectaBranding, Executor, InboundAuth, + InboundAuthRuntimeContext, Logger, } from "../types.js"; import { operatorPageForPath } from "../ui.js"; @@ -50,7 +51,7 @@ export interface ServerOptions { branding?: ConnectaBranding | undefined; } -export interface RuntimeExecutionContext { +export interface RuntimeExecutionContext extends InboundAuthRuntimeContext { waitUntil(promise: Promise): void; } @@ -110,6 +111,7 @@ export async function authorize( request: Request, baseUrl: string, auth: InboundAuth[], + runtimeContext?: RuntimeExecutionContext, ): Promise< | { ok: true; @@ -124,7 +126,7 @@ export async function authorize( } let lastResponse: Response | null = null; for (const provider of auth) { - const result = await provider.authorize(request, baseUrl); + const result = await provider.authorize(request, baseUrl, runtimeContext); if (result.ok) { const subjectId = result.subjectId ?? result.userId; const actorNamespace = activityActorNamespace(provider); @@ -137,7 +139,7 @@ export async function authorize( ? { namespace: actorNamespace } : {}), }, - ...(result.userId && provider.uiAuth?.kind === "clerk" + ...(result.userId && provider.interactiveOperator ? { uiAdminEligible: true } : {}), }; @@ -163,30 +165,29 @@ export async function authorizeUiAdmin( baseUrl: string, auth: InboundAuth[], purpose = "credential management", + runtimeContext?: RuntimeExecutionContext, ): Promise<{ ok: true; userId: string } | { ok: false; response: Response }> { // Operator mutation is intentionally narrower than /mcp and /ui/data: only - // an interactive Clerk provider may admit it. A static bearer token is useful + // an interactive provider may admit it. A static bearer token is useful // for headless tool calls but must not become a deployment-admin key. // - // Every Clerk provider gets a turn, the way the /mcp gate does. Stopping at - // the first would make admission depend on config order: a failed gate or + // Every interactive provider gets a turn, the way the /mcp gate does. + // Stopping at the first would make admission depend on config order: a failed gate or // missing user may simply mean a later provider is the one meant to admit. // The last refusal is returned if none do. - const providers = auth.filter( - (candidate) => candidate.uiAuth?.kind === "clerk", - ); + const providers = auth.filter((candidate) => candidate.interactiveOperator); if (providers.length === 0) { return { ok: false, response: privateJson( - { error: `${purpose} requires Clerk authentication` }, + { error: `${purpose} requires interactive operator authentication` }, { status: 403 }, ), }; } let lastResponse: Response | null = null; for (const provider of providers) { - const result = await provider.authorize(request, baseUrl); + const result = await provider.authorize(request, baseUrl, runtimeContext); if (!result.ok) { lastResponse = result.response; continue; diff --git a/src/routes/ui.ts b/src/routes/ui.ts index db9936c6..c890e411 100644 --- a/src/routes/ui.ts +++ b/src/routes/ui.ts @@ -52,7 +52,7 @@ function uiScriptNonce(): string { export async function routeUi( context: RouteContext, ): Promise { - const { request, url, path, baseUrl, opts, defer } = context; + const { request, url, path, baseUrl, opts, defer, runtimeContext } = context; if (request.method === "GET" && path === "/favicon.svg") { return new Response(opts.branding?.favicon?.svg ?? CONNECTA_FAVICON_SVG, { headers: { @@ -89,7 +89,15 @@ export async function routeUi( } // Open shell — carries no operator data; everything comes from the // authenticated /ui/* APIs after the browser establishes a session. - const uiAuth = opts.auth.find((provider) => provider.uiAuth)?.uiAuth; + const ambient = runtimeContext?.access + ? opts.auth.find( + (provider) => provider.uiAuth?.kind === "cloudflare-access", + )?.uiAuth + : undefined; + const uiAuth = ambient ?? opts.auth.find( + (provider) => + provider.uiAuth && provider.uiAuth.kind !== "cloudflare-access", + )?.uiAuth; const mcpUrl = new URL("/mcp", baseUrl).toString(); // Nonce the page's inline script (and the Clerk loader). 'strict-dynamic' // lets scripts the nonced Clerk loader injects at runtime execute; the @@ -116,21 +124,21 @@ export async function routeUi( } if (path !== "/ui/data") return null; - const authz = await authorize(request, baseUrl, opts.auth); + const authz = await authorize(request, baseUrl, opts.auth, runtimeContext); if (!authz.ok) return authz.response; - const eligibleClerkOperator = authz.uiAdminEligible === true; + const eligibleOperator = authz.uiAdminEligible === true; const credentialManagement = credentialManagementCapability({ - eligibleClerkOperator, + eligibleOperator, hasCredentialSlots: opts.registry .listConnectors() .some((connector) => Boolean(connector.credential)), hasCredentialVault: Boolean(opts.credentialVault), }); // As with connector credentials, a Bearer-authenticated observer learns - // only that Clerk is required—not whether this deployment has opted into + // only that an interactive operator is required, not whether this deployment has opted into // token issuance. Configuration topology is operator data. - const accessTokenManagement = !eligibleClerkOperator - ? "requires_clerk" as const + const accessTokenManagement = !eligibleOperator + ? "requires_operator" as const : opts.accessTokens ? "available" as const : "not_configured" as const; @@ -140,11 +148,11 @@ export async function routeUi( opts.serverInfo, // The static headless bearer may read connector health, but only a // Clerk-authenticated operator receives credential metadata. - eligibleClerkOperator ? opts.credentialVault : undefined, + eligibleOperator ? opts.credentialVault : undefined, Boolean(opts.activity?.list), credentialManagement, defer, - eligibleClerkOperator, + eligibleOperator, opts.discoveryConcurrency, accessTokenManagement, ); diff --git a/src/skills.ts b/src/skills.ts index 1c3960fa..53cfcb2a 100644 --- a/src/skills.ts +++ b/src/skills.ts @@ -1,15 +1,15 @@ import type { Connector } from "./types.js"; export const CONNECTA_INSTRUCTIONS = - 'Choose a route before discovery. For one read at an unknown address, use search_tools then call_tool; a known address needs only call_tool. For read-only reduction, multiple or dependent calls, loops, joins, or branches, use one execute_code program that discovers, calls, and returns the reduced answer. Only readOnlyHint: true tools run there. Keep unannotated, write-capable, or destructive work top level: search_tools then call_destructive_tool. After auth_required use authorize_connector. After a truncated direct result use get_result. connecta.ui(html) exists only inside execute_code, not in connector search; return the same summary data the HTML renders. Fetch skills({ name: "usage" }) once for program syntax, selection, repair, examples, and runtime details.'; + 'Choose a route before discovery. A known-address read needs only call_tool. Unknown-address read-only work starts with one execute_code program that discovers, calls, and returns the answer; use the same route for reduction, multiple or dependent calls, loops, joins, or branches. Only readOnlyHint: true tools run there. Keep catalog inspection and unannotated, write-capable, or destructive work top level: search_tools then call_destructive_tool when a call is needed. After auth_required use authorize_connector. After a truncated direct result use get_result. connecta.ui(html) exists only inside execute_code, not in connector search; return the same summary data the HTML renders. Guidance is on demand: fetch skills({ name: "usage" }) only when these instructions and the tool description are insufficient or a run needs repair.'; const USAGE_SKILL_BASE = `# Connecta usage ## The surface -Seven tools: \`execute_code\`, \`search_tools\`, \`call_tool\`, \`call_destructive_tool\`, \`authorize_connector\`, \`get_result\`, \`skills\`. Broad discovery and multi-call work live in a program, not in top-level tools. +Seven tools: \`execute_code\`, \`search_tools\`, \`call_tool\`, \`call_destructive_tool\`, \`authorize_connector\`, \`get_result\`, \`skills\`. Read-only discovery and multi-call work live in a program. Top-level search remains for catalog inspection and approval-required work. -The always-loaded MCP instructions are authoritative for choosing the top-level route. Read this skill at most once per task for the program workflow and recovery details below. +The always-loaded MCP instructions are authoritative for choosing the top-level route. Read this skill at most once, and only when their program workflow is insufficient or a run needs repair. ## Inside a program @@ -27,13 +27,16 @@ The minimum guest API is: Search inside the run and finish the task there. A discovery-only program wastes a round trip. Use 2–4 distinctive action/object terms, not the full request. Use separate short searches for distinct operations. -For top-level \`search_tools\`, omit \`limit\` initially (the default is 10), then page with a limit up to 50 if needed. Empty or whitespace-only queries browse all tools. A non-empty query with no ASCII terms returns no matches; mixed input searches with its ASCII terms. \`includeSchemas: "compact"\` adds bounded input and available output shapes. An observed shape carries \`outputSchemaSource: "observed"\`; treat it as routing evidence rather than a provider contract. Plain objects expose \`inputKeys\`, \`requiredInputKeys\`, and \`outputKeys\`; truncation flags mark incomplete shapes; matches also carry declared annotations. +For top-level catalog inspection or approval-required discovery, omit \`limit\` initially (the default is 10), then page with a limit up to 50 if needed. Empty or whitespace-only queries browse all tools. A non-empty query with no ASCII terms returns no matches; mixed input searches with its ASCII terms. \`includeSchemas: "compact"\` adds bounded input and available output shapes. An observed shape carries \`outputSchemaSource: "observed"\`; treat it as routing evidence rather than a provider contract. Plain objects expose \`inputKeys\`, \`requiredInputKeys\`, and \`outputKeys\`; truncation flags mark incomplete shapes; matches also carry declared annotations. - \`connecta.search({})\` loads all catalogs. Pass \`connector: ""\` when the integration is obvious. Use \`safety: "readOnly"\` for program calls. These inputs filter discovery; they grant no authority. - Request \`includeSchemas: "compact"\`. Check address, purpose, annotations, required inputs, truncation, safety, and available outputs. Never select only because a result ranks first or has fewer required inputs. - Supply every \`requiredInputKey\` from the task or a prior result. For dependencies, match the earlier \`outputKey\` to the later required key. An empty required-key list does not permit invented arguments. Missing \`outputKeys\` means inspect \`outputSchema\`. - Use \`connecta.describe({ address })\` or \`{ addresses }\` when a compact schema is truncated or insufficient. Use \`format: "json"\` only for exact constraints. Write the property names the schema displays; never guess positions or aliases. - Reduce through available output keys. Treat an observed key as a hint, since later results may omit it or add others. Do not guess collection roots such as \`items\` or \`results\`. If a match or result key is missing, inspect, re-search, or describe inside the same run instead of returning discovery for another call. +- Match provider identifiers and names exactly after resolving them from source data or a connector guide. A broad regular expression that merely finds a plausible value is not identity resolution. +- Preserve the schema's JSON types exactly: a numeric id is a number, not a numeric-looking string. Call the search and describe functions directly; batch accepts canonical connector tool addresses, not guest API function names. +- Validate tabular headers, row arrays, and row widths before mapping them. Never let a header or partial row become data. Only tools explicitly annotated \`readOnlyHint: true\` are reachable. The catalog, credential, admission, and read-only gates run below the sandbox; code cannot widen its authority. diff --git a/src/types.ts b/src/types.ts index d9388044..096526f8 100644 --- a/src/types.ts +++ b/src/types.ts @@ -475,36 +475,52 @@ export type AuthResult = | { ok: false; response: Response }; /** Public browser-auth configuration exposed to connecta's status UI. */ -export type UiAuthConfig = { - kind: "clerk"; - publishableKey: string; - /** - * Origin the operator shell fetches its browser sign-in loader from. **Must be an absolute - * `https:` URL** — the value lands in a `