diff --git a/AGENTS.md b/AGENTS.md index 0ba8ee0b65..e6fe240464 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,7 +17,7 @@ This is a TypeScript monorepo built for agent-assisted development. Keep the roo - `apps/kimi-code`: the CLI / TUI application. It consumes core capabilities through `@moonshot-ai/kimi-code-sdk` and must not depend directly on `@moonshot-ai/agent-core`. When writing or modifying its terminal UI, use the `write-tui` skill (`.agents/skills/write-tui/SKILL.md`). - `apps/kimi-web`: the browser web UI, a peer to the TUI. Vue 3 + Vite + vue-i18n; talks to the server over REST + WebSocket under `/api/v1`. It must not depend on `@moonshot-ai/agent-core` (wire types are re-implemented locally). Debug against the two engines via the root `pnpm dev:v1` / `pnpm dev:v2` backend scripts — the dev Sidebar shows the active backend and switches it at runtime. See `apps/kimi-web/AGENTS.md`. - `apps/vis`, `apps/vis/server`, `apps/vis/web`: visual debugging tools for sessions and replays. -- `apps/kimi-inspect`: web inspector for the kap-server `/api/v1/debug` RPC surface — workspace/session browser, per-session chat, and Service panels (data + trigger buttons) for the Session and Agent scopes. A left icon rail (`src/components/NavRail.tsx`) switches top-level views: the Chat workspace, the Model Catalog (`src/components/ModelCatalogView.tsx` — every Provider with its Models and the default marker, via `IModelCatalog` / `IModelService` channel proxies), and App Services (`src/components/AppServicesView.tsx` — the app-scope Service reflection, full width; the Agent scope stays in the Chat view's right dock (`src/components/RightPanel.tsx`) across two tabs: the `Agent` tab (`Inspector`: agent switcher + a Plan lookup card — `PlanCard` in `src/components/Inspector.tsx` — querying `GET /sessions/{id}/transcript/plan` (one tool_call_id, or every plan of the agent) via `src/transcript/api.ts`'s `fetchTranscriptPlan` — plus the agent Service panels) and the `State` tab (every key an Agent Service registered into the agent-state container, polled live via `IAgentStateService.snapshot()` — the same live diff-tree view as the session State tab, sharing `StateCard` from `src/components/StateCard.tsx`), while the Session scope has its own column right next to the session-list sidebar (`src/components/SessionPane.tsx`) with two tabs: Services (the pending-interactions card — `src/components/InteractionsCard.tsx` — plus the session Service panels) and State (every key a Session Service registered into the session-state container, read on demand via `ISessionStateService.snapshot()`)). Expanding a Model opens the model inspector inside that view: provider/model config layers plus the resolved runtime view with per-value provenance (config / override / builtin / env / synthesized), served on demand by `IModelCatalog.inspect` — the same resolution pass the runtime's `get` serves, traced via `ResolutionTraceCollector` and assembled by `kosong/model/inspection.ts`. Built on its own old-klient-style channel layer (`src/channel/`: the VS Code `ProxyChannel` model — service-bound `IChannel`, HTTP `ProxyChannel` for calls routed to `/api/v1/debug`), typed by `agent-core-v2` Service interfaces; `GET /api/v1/debug/channels` loads the whole wire protocol 1:1 (every scoped Service, no whitelist). There is no Service-event push channel: panels fetch/refresh on demand (`Sidebar` polls react-query on a 15 s interval), and a connection failure shows a blocking "Debug surface unavailable" screen instead of falling back anywhere. Session-level coarse status is the one exception: `src/activity/` holds a second `/api/v1/ws` client (`GlobalEventsWs`) that subscribes to nothing and consumes the server-pushed global facts — `event.session.work_changed` updates a per-session activity map (`SessionActivityHub` + subscribe/version store, seeded on connect/reconnect from `GET /api/v1/sessions`), while `event.session.created` / `session.meta.updated` invalidate the `['sessions']` query; the `Sidebar` session rows render `running` / `approval` / `question` / `failed` badges from it via `useSessionActivities`. The Vite dev server proxies `/api` to a running kap-server (`KIMI_SERVER_URL`, default `http://127.0.0.1:58627`) and exposes `GET /__inspect/servers` (`vite/serverDiscovery.ts`), which scans the local kap-server instance registry (`~/.kimi-code/server/instances` + legacy `lock`) and the home token so the app can zero-config auto-connect and switch servers from the header dropdown at runtime. The per-session chat (`src/components/ChatView.tsx`) renders turn-granularly from the **transcript** surface instead of context memory: full state is read from `GET /api/v1/sessions/{id}/transcript` (initial load = newest page, refreshes re-read from the tail backwards), older history auto-pages with `before_turn` via an IntersectionObserver sentinel at the top of the scroll view, and each timeline item is wrapped in `content-visibility: auto` + `contain-intrinsic-size` so the browser virtualizes off-screen rendering natively (no windowing library); `/api/v1/ws` is an incremental channel (`transcript.ops`, grade `block` — the cheapest grade that still carries whole-state frame upserts, dropping per-token `append` frames; `transcript.reset` is ignored by the store, surfaced only to the audit recorder via the optional `onReset` handler). The channel tracks the op-batch watermark: a dedicated `subscribe_v2` control frame carries the per-agent grades and the `transcript_since` cursor, a seq gap / reconnect / `resync_required` / append gap triggers a point-to-point catch-up (`fetchTranscriptOps` → `GET .../transcript/ops?since_seq=`), and any legacy/incomplete answer falls back to the full REST refresh. Convergence reuses `@moonshot-ai/transcript`'s L2 reducer (`src/transcript/`: REST/WS clients + store; the data model and reducer come from the package, nothing is re-implemented locally). The Transcript audit panel (`src/components/audit/`, the `Audit` tab of the chat view's right dock — `src/components/RightPanel.tsx`, fed the trail by `ChatView`'s `onTrailChange`) replays how the visible store was built: an `AuditTrail` (`src/audit/`) records every step — each REST page (request + replace/prepend), every WS frame (`transcript.ops` live/buffered/flushed/catchup, `transcript.reset`), loss signals, and prompt/cancel actions — with the resulting immutable `AgentState` per entry; the panel offers a draggable timeline plus a Diff tab (structural diff vs the previous entry: added/modified/removed colored, long strings tail-truncated, all fields kept), a full State view, and the raw Event payload. +- `apps/kimi-inspect`: web inspector for the kap-server `/api/v1/debug` RPC surface — workspace/session browser, per-session chat, and Service panels (data + trigger buttons) for the Session and Agent scopes. A left icon rail (`src/components/NavRail.tsx`) switches top-level views: the Chat workspace, the global message search (`src/components/SearchView.tsx` — cross-session full-text search over `POST /api/v1/search`, cursor-paged via a manual Load more; an exact-match checkbox maps to the API's `mode: 'literal'` substring search, which ignores sort and orders newest-first; a `live`/`index` badge on the results shows which server route served them (in-memory session transcript vs the persisted index)), the Model Catalog (`src/components/ModelCatalogView.tsx` — every Provider with its Models and the default marker, via `IModelCatalog` / `IModelService` channel proxies), and App Services (`src/components/AppServicesView.tsx` — the app-scope Service reflection, full width; the Agent scope stays in the Chat view's right dock (`src/components/RightPanel.tsx`) across two tabs: the `Agent` tab (`Inspector`: agent switcher + a Plan lookup card — `PlanCard` in `src/components/Inspector.tsx` — querying `GET /sessions/{id}/transcript/plan` (one tool_call_id, or every plan of the agent) via `src/transcript/api.ts`'s `fetchTranscriptPlan` — plus the agent Service panels) and the `State` tab (every key an Agent Service registered into the agent-state container, polled live via `IAgentStateService.snapshot()` — the same live diff-tree view as the session State tab, sharing `StateCard` from `src/components/StateCard.tsx`), while the Session scope has its own column right next to the session-list sidebar (`src/components/SessionPane.tsx`) with two tabs: Services (the pending-interactions card — `src/components/InteractionsCard.tsx` — plus the session Service panels) and State (every key a Session Service registered into the session-state container, read on demand via `ISessionStateService.snapshot()`)). Expanding a Model opens the model inspector inside that view: provider/model config layers plus the resolved runtime view with per-value provenance (config / override / builtin / env / synthesized), served on demand by `IModelCatalog.inspect` — the same resolution pass the runtime's `get` serves, traced via `ResolutionTraceCollector` and assembled by `kosong/model/inspection.ts`. Built on its own old-klient-style channel layer (`src/channel/`: the VS Code `ProxyChannel` model — service-bound `IChannel`, HTTP `ProxyChannel` for calls routed to `/api/v1/debug`), typed by `agent-core-v2` Service interfaces; `GET /api/v1/debug/channels` loads the whole wire protocol 1:1 (every scoped Service, no whitelist). There is no Service-event push channel: panels fetch/refresh on demand (`Sidebar` polls react-query on a 15 s interval), and a connection failure shows a blocking "Debug surface unavailable" screen instead of falling back anywhere. Session-level coarse status is the one exception: `src/activity/` holds a second `/api/v1/ws` client (`GlobalEventsWs`) that subscribes to nothing and consumes the server-pushed global facts — `event.session.work_changed` updates a per-session activity map (`SessionActivityHub` + subscribe/version store, seeded on connect/reconnect from `GET /api/v1/sessions`), while `event.session.created` / `session.meta.updated` invalidate the `['sessions']` query; the `Sidebar` session rows render `running` / `approval` / `question` / `failed` badges from it via `useSessionActivities`. The Vite dev server proxies `/api` to a running kap-server (`KIMI_SERVER_URL`, default `http://127.0.0.1:58627`) and exposes `GET /__inspect/servers` (`vite/serverDiscovery.ts`), which scans the local kap-server instance registry (`~/.kimi-code/server/instances` + legacy `lock`) and the home token so the app can zero-config auto-connect and switch servers from the header dropdown at runtime. The per-session chat (`src/components/ChatView.tsx`) renders turn-granularly from the **transcript** surface instead of context memory and carries an in-chat search bar (`src/components/ChatSearchBar.tsx`): it searches the current session via `POST /api/v1/search` with `container: { session_id }` (usually served by the live route, since selecting a session resumes it), and a result click funnels through the app shell's `openSearchHit` — the same agent-switch + `ChatJump` (page-back, scroll, flash) path the global search view uses; full state is read from `GET /api/v1/sessions/{id}/transcript` (initial load = newest page, refreshes re-read from the tail backwards), older history auto-pages with `before_turn` via an IntersectionObserver sentinel at the top of the scroll view, and each timeline item is wrapped in `content-visibility: auto` + `contain-intrinsic-size` so the browser virtualizes off-screen rendering natively (no windowing library); `/api/v1/ws` is an incremental channel (`transcript.ops`, grade `block` — the cheapest grade that still carries whole-state frame upserts, dropping per-token `append` frames; `transcript.reset` is ignored by the store, surfaced only to the audit recorder via the optional `onReset` handler). The channel tracks the op-batch watermark: a dedicated `subscribe_v2` control frame carries the per-agent grades and the `transcript_since` cursor, a seq gap / reconnect / `resync_required` / append gap triggers a point-to-point catch-up (`fetchTranscriptOps` → `GET .../transcript/ops?since_seq=`), and any legacy/incomplete answer falls back to the full REST refresh. Convergence reuses `@moonshot-ai/transcript`'s L2 reducer (`src/transcript/`: REST/WS clients + store; the data model and reducer come from the package, nothing is re-implemented locally). The Transcript audit panel (`src/components/audit/`, the `Audit` tab of the chat view's right dock — `src/components/RightPanel.tsx`, fed the trail by `ChatView`'s `onTrailChange`) replays how the visible store was built: an `AuditTrail` (`src/audit/`) records every step — each REST page (request + replace/prepend), every WS frame (`transcript.ops` live/buffered/flushed/catchup, `transcript.reset`), loss signals, and prompt/cancel actions — with the resulting immutable `AgentState` per entry; the panel offers a draggable timeline plus a Diff tab (structural diff vs the previous entry: added/modified/removed colored, long strings tail-truncated, all fields kept), a full State view, and the raw Event payload. - `packages/agent-core`: the unified agent engine, including Agent, Session, profile, skills, tools, plan, permission, background, records, the in-process DI service layer (`src/services/`), and other core capabilities. - `packages/node-sdk`: the public TypeScript SDK and harness. - `packages/kosong`: the LLM / provider abstraction layer. @@ -25,10 +25,11 @@ This is a TypeScript monorepo built for agent-assisted development. Keep the roo - `packages/oauth`: Kimi OAuth and managed auth utilities. - `packages/telemetry`: shared client-side telemetry infrastructure. - `packages/transcript`: the isomorphic transcript rendering data layer — agent-granular L1 store, idempotent L2 operations, `off/turn/block/delta` L3 subscription granularity, framework-free L4 view registry, and turn-cursor pagination. Pure TypeScript (browser-safe, no engine imports) and the sole owner of all transcript contract types (`src/contract/`); consumed by `packages/kap-server` (engine events → transcript, REST + WS surface; live stores backfill history from the persisted per-agent wire records — main on first attach, any agent on demand, cold sessions rebuild any agent — with 0-based turn ordinals matching the engine's). The cold rebuild is a two-level fold over `wire.jsonl` as the single source of truth: `history/groupTurns.ts` (context messages → turn tree) plus `history/foldFacts.ts` (non-context records → tasks, interactions, todos, goal/plan/swarm meta, and end-appended markers/taskrefs; interactions left pending at shutdown fold to `cancelled`). Plan content is a recorded fact too: each ExitPlanMode review submission offloads the document to `agents//plan//v.md` and persists a reference-only `plan.revision` record (`{id, version, path, sha256, bytes}`), which projects — live and cold — to a `plan.revision` marker and the `modes.plan` badge (`{reviewPath, version}`). It also owns the op-batch sequencing contract (`transcriptSeqSchema` in `contract/schema.ts`): a per-(session, agent) monotonic batch `seq` on `transcript.ops` / `transcript.reset` / the REST transcript response, the `transcript_since` subscription cursor, and the `GET .../transcript/ops` catch-up response shape — every field optional so pre-seq peers fall back to loss-signal-driven refreshes. Beyond the timeline, the model carries wire-equivalent detail: steps carry `usage` / `finishReason` / `timing` (LLM latencies) / `retry` / interrupt reason, turns carry `durationMs` / `error` / `usage`, tool frames carry the streamed `inputText` and the latest `progress`, tasks carry subagent `resultSummary` / `error` / `stateReason` / `usage`, `meta.agent` mirrors the agent status slices (model / usage / context / permission / phase), a global `prompts` entity (op `prompt.upsert`) tracks the prompt queue, and `hook.result` lands as a `'hook'` marker. These live-projected fields are NOT backfilled by the cold rebuild (known limitation). -- `packages/kap-server`: the Kimi Code server, backed by the DI × Scope agent engine (`@moonshot-ai/agent-core-v2`). Exposes sessions over REST + WebSocket (`/api/v1` + `/api/v1/ws`); bootstrapped from `src/start.ts` and consumed by `apps/kimi-code`. The RPC surface is `/api/v1/debug/*` — a reflection dispatcher over the ENTIRE scoped DI registry (every Service callable, no whitelist; `src/transport/registerDebugRoutes.ts` + `serviceDispatcherRoutes.ts`), mounted only with `--debug-endpoints` on a loopback bind and gated by the global bearer auth; repo dev scripts pass the flag. Its transcript surface implements the op-batch sequencing contract: `TranscriptService.dispatchOps` assigns every dispatched batch a per-agent consecutive `seq` and retains it in a bounded in-memory journal (`TRANSCRIPT_OPS_JOURNAL_CAPACITY`, dies with the live store); WS `transcript.ops`/`transcript.reset` payloads carry the seq/watermark, a `transcript_since` subscription cursor (carried, with the per-agent grades, by the `subscribe_v2` control frame — the only transcript subscription channel; its agent-grained counterpart `unsubscribe_v2` detaches listed agents' streams, or the whole session's when `agent_ids` is absent, letting the detached agents' legacy events flow again) replays journaled batches instead of a baseline reset when the journal covers it, and `GET /sessions/{id}/transcript/ops?since_seq=` serves point-to-point catch-up (`complete: false` = journal can't cover or session cold → caller falls back to a full refresh). Beside the paged route, `GET /sessions/{id}/transcript/plan?agent_id=[&tool_call_id=]` projects an agent's ExitPlanMode plan info (content / path / options / review outcome; `tool_call_id` narrows to one call, omitted lists every recoverable plan) from the first available fact — the linked approval interaction's persisted request display, the live tool frame's display, or the tool result output text. The baseline `transcript.reset` itself is items-empty (`TRANSCRIPT_RESET_TAIL_TURNS = 0`): it carries only global state + the watermark + `has_more_older`, because history always pages in over REST. When a WS connection subscribes to the transcript protocol (grade ≠ `off` for an agent), the broadcaster suppresses the transcript-projected `session_event` types for that connection × agent (`TRANSCRIPT_PROJECTED_EVENT_TYPES` + `suppressedByTranscript` in `sessionEventBroadcaster.ts`; cursor replay via `getBufferedSince` applies the same filter). Suppression is only a per-connection send view — the journal still records everything, and connections without transcript grades are unaffected. The session's work aggregate behind `event.session.work_changed` (`busy` / `main_turn_active` / `pending_interaction` / `last_turn_reason`) is owned by the core's `ISessionActivityView` (`sessionActivity` domain, Session scope): the broadcaster only schedules the wire emission around turn frames (`busy:false` lands after `turn.ended`), and `resolveSessionFacts` (`src/routes/sessions.ts`) reads the same view — never fold per-agent activity at the edge. Delivery split on `/api/v1/ws`: global events (`session.meta.updated` and the `event.session.*` / `event.workspace.*` / `event.config.*` families, including every activated session's `event.session.work_changed`) fan out to EVERY established connection — `WsConnectionV1` registers itself via `broadcaster.addGlobalTarget` on construction and unregisters on close — while session/agent-grained events only reach connections subscribed to that session (subject to `agent_filter` and the transcript suppression above); transcript frames are a separate channel governed by the per-agent grades alone and bypass `agent_filter` entirely. +- `packages/kap-server`: the Kimi Code server, backed by the DI × Scope agent engine (`@moonshot-ai/agent-core-v2`). Exposes sessions over REST + WebSocket (`/api/v1` + `/api/v1/ws`); bootstrapped from `src/start.ts` and consumed by `apps/kimi-code`. The RPC surface is `/api/v1/debug/*` — a reflection dispatcher over the ENTIRE scoped DI registry (every Service callable, no whitelist; `src/transport/registerDebugRoutes.ts` + `serviceDispatcherRoutes.ts`), mounted only with `--debug-endpoints` on a loopback bind and gated by the global bearer auth; repo dev scripts pass the flag. Its transcript surface implements the op-batch sequencing contract: `TranscriptService.dispatchOps` assigns every dispatched batch a per-agent consecutive `seq` and retains it in a bounded in-memory journal (`TRANSCRIPT_OPS_JOURNAL_CAPACITY`, dies with the live store); WS `transcript.ops`/`transcript.reset` payloads carry the seq/watermark, a `transcript_since` subscription cursor (carried, with the per-agent grades, by the `subscribe_v2` control frame — the only transcript subscription channel; its agent-grained counterpart `unsubscribe_v2` detaches listed agents' streams, or the whole session's when `agent_ids` is absent, letting the detached agents' legacy events flow again) replays journaled batches instead of a baseline reset when the journal covers it, and `GET /sessions/{id}/transcript/ops?since_seq=` serves point-to-point catch-up (`complete: false` = journal can't cover or session cold → caller falls back to a full refresh). Beside the paged route, `GET /sessions/{id}/transcript/plan?agent_id=[&tool_call_id=]` projects an agent's ExitPlanMode plan info (content / path / options / review outcome; `tool_call_id` narrows to one call, omitted lists every recoverable plan) from the first available fact — the linked approval interaction's persisted request display, the live tool frame's display, or the tool result output text. The baseline `transcript.reset` itself is items-empty (`TRANSCRIPT_RESET_TAIL_TURNS = 0`): it carries only global state + the watermark + `has_more_older`, because history always pages in over REST. When a WS connection subscribes to the transcript protocol (grade ≠ `off` for an agent), the broadcaster suppresses the transcript-projected `session_event` types for that connection × agent (`TRANSCRIPT_PROJECTED_EVENT_TYPES` + `suppressedByTranscript` in `sessionEventBroadcaster.ts`; cursor replay via `getBufferedSince` applies the same filter). Suppression is only a per-connection send view — the journal still records everything, and connections without transcript grades are unaffected. The session's work aggregate behind `event.session.work_changed` (`busy` / `main_turn_active` / `pending_interaction` / `last_turn_reason`) is owned by the core's `ISessionActivityView` (`sessionActivity` domain, Session scope): the broadcaster only schedules the wire emission around turn frames (`busy:false` lands after `turn.ended`), and `resolveSessionFacts` (`src/routes/sessions.ts`) reads the same view — never fold per-agent activity at the edge. Delivery split on `/api/v1/ws`: global events (`session.meta.updated` and the `event.session.*` / `event.workspace.*` / `event.config.*` families, including every activated session's `event.session.work_changed`) fan out to EVERY established connection — `WsConnectionV1` registers itself via `broadcaster.addGlobalTarget` on construction and unregisters on close — while session/agent-grained events only reach connections subscribed to that session (subject to `agent_filter` and the transcript suppression above); transcript frames are a separate channel governed by the per-agent grades alone and bypass `agent_filter` entirely. The global search surface is `POST /api/v1/search` (`src/search/` + `src/routes/search.ts`): a cross-session full-text search over user messages, assistant text, and session titles, backed by a single minidb database at `/search-index` (`IGlobalSearchService`, App scope — the write-lock holder is the indexer, other processes open read-only and catch up via WAL). It serves two modes: `terms` (the default — minidb's inverted text index over ASCII words + CJK uni/bigrams, no positions, term-level AND) and `literal` (substring-exact search: a hashed 2/3-gram index supplies candidates, every candidate's text is then confirmed with `includes`, so hits carry zero false positives; literal ignores `sort` and returns newest-first, and a candidate set truncated at `LITERAL_CANDIDATE_CAP` is flagged `incomplete: 'candidate_cap'`). When `container.session_id` is provided and that session is live in this process (`TranscriptService.forSessionLive` returns a store, wired via `setLiveTranscriptSource` in `start.ts`), BOTH modes instead scan the in-memory transcript store (turn prompts + assistant text frames, history established via `whenReady`/`ensureAgentHistory`) — no index involved; terms-mode live hits are scored Σ log(1+tf) (comparable only within a route, per the `GlobalSearchSource` contract), live-route errors never fall back to the index, and the response's `source: 'live' | 'index'` field (also mixed into the page-token fingerprint, so a mid-pagination route flip invalidates the old token) tells the caller which route served the page. - `packages/klient`: the client SDK — a contract-driven facade over agent-core-v2 with aggregated `global.*` / `session(id).*` / `agent(id).*` methods, zod validation on every call, and klient-level typed event forwarding. Transport is chosen once at creation via subpath entry (`@moonshot-ai/klient/ipc|memory`); both return the same `Klient`. The package also hosts the e2e suites: the legacy `/api/v1` live suites (`test/e2e/legacy/`) and the docker e2e runner (`pnpm --filter @moonshot-ai/klient docker:e2e`). See `packages/klient/AGENTS.md`. - `packages/server-e2e`: live e2e tests and scenarios against a running server (`KIMI_SERVER_URL`, default `http://127.0.0.1:58627`). See `packages/server-e2e/AGENTS.md`. - `packages/tree-sitter-bash`: a pure-TypeScript bash parser (no runtime deps, no wasm) that produces a syntax tree with tree-sitter-bash 0.25.0 named-node type names and UTF-16 code-unit offsets. `parse(source, { timeoutMs, maxNodes })` runs under a deterministic budget (default 50 ms / 50k nodes, plus per-chain recursion depth caps) and returns a discriminated `ParseResult` (`{ ok, rootNode, hasError }` or `{ ok: false, reason: 'aborted' }`) — callers must treat aborted/hasError trees as "cannot analyze" and degrade. Parser only, no safety judgments; consumers (e.g. Bash tool permission matching) live elsewhere. Known deviations from the reference are tracked in the package README's "Known differences" section, pinned by differential fixtures tested against the real `tree-sitter-bash` wasm (dev-only). +- `packages/minidb`: the embedded JSON document store (`MiniDb`) behind kap-server's search index — snapshot + WAL persistence with an exclusive write lock (losers open read-only and catch up from the WAL), plus a larger-than-RAM full-text layer: `src/text-index.ts` is the inverted index (in-RAM dictionary + delta, on-disk postings in `src/text-postings.ts`, rebuilt from the Store on open and on compaction) with an injectable `tokenizer`/`queryTokenizer`; the default tokenizer keeps ASCII words and CJK uni/bigrams, while `src/trigram.ts` provides the hashed 2/3-gram tokenizer (NFKC + lowercase, code-point windows) that backs substring-exact search. Text-index definitions (including the tokenizer name) persist in `db.textindexes.json`. ## Environment Requirements diff --git a/apps/kimi-code/package.json b/apps/kimi-code/package.json index ec17e64721..a4dd123daa 100644 --- a/apps/kimi-code/package.json +++ b/apps/kimi-code/package.json @@ -63,9 +63,9 @@ "test:native:smoke": "node scripts/native/smoke.mjs", "dev": "node scripts/dev.mjs", "dev:cli-only": "tsx --import ../../build/register-raw-text-loader.mjs ./src/main.ts", - "dev:server": "tsx --tsconfig ./tsconfig.dev.json --import ../../build/register-raw-text-loader.mjs ./src/main.ts web --no-open --debug-endpoints", - "dev:kap-server": "tsx --tsconfig ./tsconfig.dev.json --import ../../build/register-raw-text-loader.mjs ./src/main.ts web --no-open --debug-endpoints", - "dev:kap-server:multi": "tsx --tsconfig ./tsconfig.dev.json --import ../../build/register-raw-text-loader.mjs ./src/main.ts web --no-open --debug-endpoints", + "dev:server": "KIMI_CODE_DEV_SERVER=1 tsx --tsconfig ./tsconfig.dev.json --import ../../build/register-raw-text-loader.mjs ./src/main.ts web --no-open --debug-endpoints", + "dev:kap-server": "KIMI_CODE_DEV_SERVER=1 tsx --tsconfig ./tsconfig.dev.json --import ../../build/register-raw-text-loader.mjs ./src/main.ts web --no-open --debug-endpoints", + "dev:kap-server:multi": "KIMI_CODE_DEV_SERVER=1 tsx --tsconfig ./tsconfig.dev.json --import ../../build/register-raw-text-loader.mjs ./src/main.ts web --no-open --debug-endpoints", "dev:server:restart": "node scripts/dev-server-restart.mjs", "dev:plugin-marketplace": "node scripts/dev-plugin-marketplace-server.mjs", "build:plugin-marketplace": "node scripts/build-plugin-marketplace-cdn.mjs", diff --git a/apps/kimi-code/scripts/dev-server-restart.mjs b/apps/kimi-code/scripts/dev-server-restart.mjs index 4beb2bc6f3..62a34029a4 100644 --- a/apps/kimi-code/scripts/dev-server-restart.mjs +++ b/apps/kimi-code/scripts/dev-server-restart.mjs @@ -45,7 +45,7 @@ function start() { console.error('[dev:server:restart] starting server…'); child = spawn(tsxBin, tsxArgs, { cwd: APP_ROOT, - env: process.env, + env: { ...process.env, KIMI_CODE_DEV_SERVER: '1' }, // Server does not read stdin; keep ours free for the Enter trigger. stdio: ['ignore', 'inherit', 'inherit'], }); diff --git a/apps/kimi-code/src/cli/sub/web/run.ts b/apps/kimi-code/src/cli/sub/web/run.ts index 540ef778cf..9318b85395 100644 --- a/apps/kimi-code/src/cli/sub/web/run.ts +++ b/apps/kimi-code/src/cli/sub/web/run.ts @@ -8,6 +8,7 @@ * `startServer`). */ +import { existsSync } from 'node:fs'; import { join } from 'node:path'; import { hostRequestHeadersSeed } from '@moonshot-ai/agent-core-v2'; @@ -266,6 +267,12 @@ async function runServerInProcess( // logger, close }`, so adapt it to the `RoutedServer` surface the rest of // this runner consumes. const logger = createServerLogger({ level: options.logLevel }); + const webAssetsDir = serverWebAssetsDir(); + if (webAssetsDir === undefined) { + logger.info( + 'dev mode: web assets not built; starting the API server without the web UI', + ); + } const v2 = await startServer({ host: options.host, port: options.port, @@ -288,7 +295,7 @@ async function runServerInProcess( // requests (model, WebSearch, FetchURL) carry the same User-Agent + // X-Msh-* identity as direct CLI runs. seeds: hostRequestHeadersSeed(buildKimiDefaultHeaders(version)), - webAssetsDir: serverWebAssetsDir(), + webAssetsDir, }); logger.info('serving the REST/WS API and the bundled web UI'); running = { @@ -315,8 +322,23 @@ async function runServerInProcess( }); } -function serverWebAssetsDir(): string { - return resolveServerWebAssetsDir(); +/** + * Resolve the web assets directory passed to kap-server. In dev mode + * (`KIMI_CODE_DEV_SERVER=1`, set by the repo's `dev:server` / `dev:kap-server*` + * scripts) a missing `dist-web` build is tolerated: the server starts API-only + * and the web UI is expected to come from the kimi-web Vite dev server. + * Outside dev mode the directory is always returned and kap-server keeps + * failing fast when the assets are missing. + */ +export function serverWebAssetsDir( + env: NodeJS.ProcessEnv = process.env, + nativeWebAssetsDir: string | null = getNativeWebAssetsDir(), +): string | undefined { + const dir = resolveServerWebAssetsDir(nativeWebAssetsDir); + if (env['KIMI_CODE_DEV_SERVER'] === '1' && !existsSync(join(dir, 'index.html'))) { + return undefined; + } + return dir; } export function resolveServerWebAssetsDir( diff --git a/apps/kimi-code/test/cli/web/web.test.ts b/apps/kimi-code/test/cli/web/web.test.ts index d23497b5d1..1b51bfc535 100644 --- a/apps/kimi-code/test/cli/web/web.test.ts +++ b/apps/kimi-code/test/cli/web/web.test.ts @@ -542,6 +542,38 @@ describe('server web asset directory resolution', () => { const { resolveServerWebAssetsDir } = await import('#/cli/sub/web/run'); expect(resolveServerWebAssetsDir(null)).toMatch(/[/\\]dist-web$/); }); + + it('returns the assets dir when it is built, dev mode or not', async () => { + const { serverWebAssetsDir } = await import('#/cli/sub/web/run'); + const dir = mkdtempSync(join(tmpdir(), 'kimi-web-assets-')); + try { + writeFileSync(join(dir, 'index.html'), ''); + expect(serverWebAssetsDir({}, dir)).toBe(dir); + expect(serverWebAssetsDir({ KIMI_CODE_DEV_SERVER: '1' }, dir)).toBe(dir); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('requires built assets outside dev mode', async () => { + const { serverWebAssetsDir } = await import('#/cli/sub/web/run'); + const dir = mkdtempSync(join(tmpdir(), 'kimi-web-assets-')); + try { + expect(serverWebAssetsDir({}, dir)).toBe(dir); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('tolerates missing assets in dev mode (API-only server)', async () => { + const { serverWebAssetsDir } = await import('#/cli/sub/web/run'); + const dir = mkdtempSync(join(tmpdir(), 'kimi-web-assets-')); + try { + expect(serverWebAssetsDir({ KIMI_CODE_DEV_SERVER: '1' }, dir)).toBeUndefined(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); }); function makeLegacyKillDeps(overrides: Partial = {}): { diff --git a/apps/kimi-inspect/src/App.tsx b/apps/kimi-inspect/src/App.tsx index b7c0b67de5..bbb8c562f6 100644 --- a/apps/kimi-inspect/src/App.tsx +++ b/apps/kimi-inspect/src/App.tsx @@ -10,24 +10,26 @@ * the `models` view is the full-width model catalog; the `services` view is * the full-width app-scope Service reflection (`AppServicesView`); the * `bash` view is the full-width `IBashParserService` playground - * (`BashParserView`). + * (`BashParserView`); the `search` view is the full-width global message + * search (`SearchView`) whose hits navigate back into the chat timeline. */ -import { useEffect, useState } from 'react'; - import { ISessionLifecycleService } from '@moonshot-ai/agent-core-v2/app/sessionLifecycle/sessionLifecycle'; +import { useEffect, useState } from 'react'; import type { AuditTrail } from './audit/trail'; import { AppServicesView } from './components/AppServicesView'; import { BashParserView } from './components/BashParserView'; -import { ChatView } from './components/ChatView'; +import { ChatView, type ChatJump } from './components/ChatView'; import { ModelCatalogView } from './components/ModelCatalogView'; import { NavRail, type AppView } from './components/NavRail'; import { RightPanel } from './components/RightPanel'; +import { SearchView } from './components/SearchView'; import { ServerSwitcher } from './components/ServerSwitcher'; import { SessionPane } from './components/SessionPane'; import { Sidebar } from './components/Sidebar'; import { useConnection } from './connection'; +import type { SearchHit } from './search/api'; import { errorMessage } from './ui'; export function App() { @@ -39,6 +41,8 @@ export function App() { const [resumeError, setResumeError] = useState(null); /** Audit trail of the chat view's transcript channel, rendered in the right dock. */ const [trail, setTrail] = useState(null); + /** Pending chat navigation requested from another view (search result click). */ + const [jump, setJump] = useState(null); // Resume (materialize) the session on the server when it is selected, so // session / agent scoped Services become reachable. @@ -66,8 +70,23 @@ export function App() { useEffect(() => { setSessionId(null); setAgentId('main'); + setJump(null); }, [baseUrl]); + // A search hit opens the chat view at its session / agent / turn / step. + // Title hits belong to the session (agent '') and carry no turn: switch + // over without a scroll target. + const openSearchHit = (hit: SearchHit): void => { + setSessionId(hit.sessionId); + setAgentId(hit.agentId === '' ? 'main' : hit.agentId); + setView('chat'); + setJump({ + turnId: hit.turn === undefined ? undefined : `t${hit.turn}`, + stepId: hit.stepId, + nonce: Date.now(), + }); + }; + return (
@@ -94,6 +113,8 @@ export function App() { setView('chat'); }} /> + ) : view === 'search' ? ( + ) : ( <> @@ -108,6 +129,9 @@ export function App() { agentId={agentId} ready={ready} onTrailChange={setTrail} + jump={jump} + onJumpHandled={() => setJump(null)} + onOpenSearchHit={openSearchHit} /> )} { expect(hub.store.get('s1')).toEqual(facts({ busy: true, mainTurnActive: true })); expect(hub.store.get('s2')?.pendingInteraction).toBe('approval'); // The hello goes out with no subscriptions — global facts flow regardless. - const hello = JSON.parse(instances[0]!.sent[0]!) as { type: string; payload: { subscriptions: string[] } }; + const hello = JSON.parse(instances[0]!.sent[0]!) as { + type: string; + payload: { subscriptions: string[] }; + }; expect(hello.type).toBe('client_hello'); expect(hello.payload.subscriptions).toEqual([]); hub.close(); diff --git a/apps/kimi-inspect/src/activity/store.ts b/apps/kimi-inspect/src/activity/store.ts index 80f67cedb9..b68fffc619 100644 --- a/apps/kimi-inspect/src/activity/store.ts +++ b/apps/kimi-inspect/src/activity/store.ts @@ -14,11 +14,8 @@ * `useSyncExternalStore`. */ -import { - GlobalEventsWs, - type SessionWorkFacts, -} from './ws'; import type { WsLikeCtor } from '../channel/wsLike'; +import { GlobalEventsWs, type SessionWorkFacts } from './ws'; export type { SessionWorkFacts }; @@ -131,8 +128,7 @@ export class SessionActivityHub { { busy: item['busy'], mainTurnActive: item['main_turn_active'] === true, - pendingInteraction: - pending === 'approval' || pending === 'question' ? pending : 'none', + pendingInteraction: pending === 'approval' || pending === 'question' ? pending : 'none', lastTurnReason: reason === 'completed' || reason === 'cancelled' || reason === 'failed' ? reason diff --git a/apps/kimi-inspect/src/activity/useSessionActivity.ts b/apps/kimi-inspect/src/activity/useSessionActivity.ts index f4b94722db..cdb83799fe 100644 --- a/apps/kimi-inspect/src/activity/useSessionActivity.ts +++ b/apps/kimi-inspect/src/activity/useSessionActivity.ts @@ -9,8 +9,8 @@ * and a memo-created hub would stay closed for the rest of the page's life. */ -import { useEffect, useState, useSyncExternalStore } from 'react'; import { useQueryClient } from '@tanstack/react-query'; +import { useEffect, useState, useSyncExternalStore } from 'react'; import { useConnection } from '../connection'; import { SessionActivityHub, SessionActivityStore, type SessionWorkFacts } from './store'; diff --git a/apps/kimi-inspect/src/activity/ws.ts b/apps/kimi-inspect/src/activity/ws.ts index 18bd2bc51a..6e9b77af30 100644 --- a/apps/kimi-inspect/src/activity/ws.ts +++ b/apps/kimi-inspect/src/activity/ws.ts @@ -206,8 +206,7 @@ function parseWorkFacts(payload: unknown): SessionWorkFacts | undefined { return { busy: p['busy'], mainTurnActive: p['main_turn_active'] === true, - pendingInteraction: - pending === 'approval' || pending === 'question' ? pending : 'none', + pendingInteraction: pending === 'approval' || pending === 'question' ? pending : 'none', lastTurnReason: reason === 'completed' || reason === 'cancelled' || reason === 'failed' ? reason : undefined, }; diff --git a/apps/kimi-inspect/src/audit/audit.test.ts b/apps/kimi-inspect/src/audit/audit.test.ts index c601ab3bc9..0043ec73ba 100644 --- a/apps/kimi-inspect/src/audit/audit.test.ts +++ b/apps/kimi-inspect/src/audit/audit.test.ts @@ -3,21 +3,23 @@ * and tail-preserving truncation used by the chat view's audit panel. */ +import { EMPTY_AGENT_STATE, type AgentState, type TranscriptTurn } from '@moonshot-ai/transcript'; import { describe, expect, it } from 'vitest'; -import { - EMPTY_AGENT_STATE, - type AgentState, - type TranscriptTurn, -} from '@moonshot-ai/transcript'; - import { diffValue, type DiffNode } from './diff'; import { serializeState } from './serialize'; import { AuditTrail, AUDIT_TRAIL_MAX_ENTRIES } from './trail'; import { tailTrunc } from './truncate'; function turnItem(n: number): TranscriptTurn { - return { kind: 'turn', turnId: `t${n}`, ordinal: n, state: 'completed', origin: { kind: 'user' }, steps: [] }; + return { + kind: 'turn', + turnId: `t${n}`, + ordinal: n, + state: 'completed', + origin: { kind: 'user' }, + steps: [], + }; } function stateWith(items: readonly TranscriptTurn[]): AgentState { @@ -35,12 +37,19 @@ describe('diffValue', () => { }); it('marks added, removed, and modified object keys', () => { - const node = diffValue({ keep: 1, gone: 'x', changed: 'a' }, { keep: 1, fresh: true, changed: 'b' }); + const node = diffValue( + { keep: 1, gone: 'x', changed: 'a' }, + { keep: 1, fresh: true, changed: 'b' }, + ); expect(node.status).toBe('modified'); expect(node.children?.get('keep')?.status).toBe('unchanged'); expect(node.children?.get('fresh')?.status).toBe('added'); expect(node.children?.get('gone')).toMatchObject({ status: 'removed', prev: 'x' }); - expect(node.children?.get('changed')).toMatchObject({ status: 'modified', prev: 'a', value: 'b' }); + expect(node.children?.get('changed')).toMatchObject({ + status: 'modified', + prev: 'a', + value: 'b', + }); }); it('matches entity arrays by id instead of index', () => { @@ -70,7 +79,7 @@ describe('diffValue', () => { [step('t1.1', 'completed'), step('t1.2', 'completed')], [step('t1.1', 'completed'), step('t1.2', 'running')], ); - expect([...node.children?.keys() ?? []]).toEqual(['t1.1', 't1.2']); + expect([...(node.children?.keys() ?? [])]).toEqual(['t1.1', 't1.2']); expect(node.children?.get('t1.1')?.status).toBe('unchanged'); expect(node.children?.get('t1.2')?.status).toBe('modified'); }); @@ -124,8 +133,14 @@ describe('serializeState', () => { const state: AgentState = { ...EMPTY_AGENT_STATE, tasks: new Map([ - ['b-task', { taskId: 'b-task', kind: 'shell', state: 'running', detached: false, outputTail: '' }], - ['a-task', { taskId: 'a-task', kind: 'tool', state: 'completed', detached: false, outputTail: '' }], + [ + 'b-task', + { taskId: 'b-task', kind: 'shell', state: 'running', detached: false, outputTail: '' }, + ], + [ + 'a-task', + { taskId: 'a-task', kind: 'tool', state: 'completed', detached: false, outputTail: '' }, + ], ]), pendingInteractions: new Set(['z', 'a']), }; @@ -188,7 +203,9 @@ describe('AuditTrail', () => { expect(entries[1]!.state).toBe(s2); expect(entries[1]).toMatchObject({ delivery: 'live', envelopeAt: '2026-01-01T00:00:00Z' }); expect(entries[2]).toMatchObject({ event: 'prompt', detail: 'hello' }); - expect(entries.every((entry) => typeof entry.at === 'string' && entry.at.length > 0)).toBe(true); + expect(entries.every((entry) => typeof entry.at === 'string' && entry.at.length > 0)).toBe( + true, + ); expect(entries.every((entry) => entry.summary.length > 0)).toBe(true); }); diff --git a/apps/kimi-inspect/src/audit/trail.ts b/apps/kimi-inspect/src/audit/trail.ts index abbe260031..efed4cfd2b 100644 --- a/apps/kimi-inspect/src/audit/trail.ts +++ b/apps/kimi-inspect/src/audit/trail.ts @@ -138,7 +138,11 @@ export class AuditTrail { }); } - recordEvent(event: EventAuditEntry['event'], detail: string | undefined, state: AgentState): void { + recordEvent( + event: EventAuditEntry['event'], + detail: string | undefined, + state: AgentState, + ): void { const label = event === 'ack-refresh' ? 'subscribe ack → REST refresh' diff --git a/apps/kimi-inspect/src/channel/channel.test.ts b/apps/kimi-inspect/src/channel/channel.test.ts index 55c0f0d3bc..1064924fea 100644 --- a/apps/kimi-inspect/src/channel/channel.test.ts +++ b/apps/kimi-inspect/src/channel/channel.test.ts @@ -88,11 +88,11 @@ describe('makeProxy', () => { it('routes methods to call and onXxx members to listen', async () => { const seen = { calls: [] as [string, unknown[]][], listens: [] as string[] }; const channel: IChannel = { - call: async (command: string, args?: unknown[]): Promise => { + call: async (command: string, args?: unknown[]): Promise => { seen.calls.push([command, args ?? []]); return 'ret' as T; }, - listen: (event: string): Event => { + listen: (event: string): Event => { seen.listens.push(event); return () => ({ dispose: () => {} }); }, @@ -137,9 +137,7 @@ describe('probeDebugSurface', () => { it('throws a --debug-endpoints hint when the surface is not mounted (HTTP 404)', async () => { stubProbeFetch(() => ({ ok: false, status: 404 })); - await expect(probeDebugSurface({ baseUrl: 'http://h:6' })).rejects.toThrow( - /--debug-endpoints/, - ); + await expect(probeDebugSurface({ baseUrl: 'http://h:6' })).rejects.toThrow(/--debug-endpoints/); }); it('throws an unreachable-server error when fetch itself fails', async () => { diff --git a/apps/kimi-inspect/src/channel/channels.ts b/apps/kimi-inspect/src/channel/channels.ts index f36c6f984e..508e6125fc 100644 --- a/apps/kimi-inspect/src/channel/channels.ts +++ b/apps/kimi-inspect/src/channel/channels.ts @@ -14,9 +14,9 @@ import { createDecorator } from '@moonshot-ai/agent-core-v2/_base/di/instantiation'; +import type { ServiceProxy } from './channel'; import { DEBUG_RPC_BASE, type InspectClient } from './client'; import { RPCError } from './errors'; -import type { ServiceProxy } from './channel'; /** Wire scope kinds reported by the channels endpoint (`app` ≡ the core route). */ export type ChannelScope = 'app' | 'session' | 'agent'; diff --git a/apps/kimi-inspect/src/channel/wsLike.ts b/apps/kimi-inspect/src/channel/wsLike.ts index f489ce76bb..440a9e8c77 100644 --- a/apps/kimi-inspect/src/channel/wsLike.ts +++ b/apps/kimi-inspect/src/channel/wsLike.ts @@ -8,7 +8,10 @@ export interface WsLike { readonly readyState: number; send(data: string): void; close(code?: number, reason?: string): void; - addEventListener(type: 'open' | 'message' | 'close' | 'error', listener: (event: never) => void): void; + addEventListener( + type: 'open' | 'message' | 'close' | 'error', + listener: (event: never) => void, + ): void; } export interface WsLikeCtor { diff --git a/apps/kimi-inspect/src/components/ChatSearchBar.tsx b/apps/kimi-inspect/src/components/ChatSearchBar.tsx new file mode 100644 index 0000000000..2d762f809c --- /dev/null +++ b/apps/kimi-inspect/src/components/ChatSearchBar.tsx @@ -0,0 +1,147 @@ +/** + * In-chat search bar — searches the messages of the CURRENT session only + * (`container: { session_id }` on `POST /api/v1/search`; the whole session, + * not just the active agent) and renders the first page as a dropdown under + * the input. Clicking a hit hands it to the app shell (`onOpenHit`), which + * switches the agent when the hit belongs to another one and jumps the + * transcript to the hit's turn / step. Esc or an outside click closes the + * panel. + */ + +import { useEffect, useRef, useState } from 'react'; + +import { useConnection } from '../connection'; +import { fetchSearchPage, type SearchHit } from '../search/api'; +import { Badge, ErrorLine, relTime } from '../ui'; + +const PAGE_SIZE = 20; + +export function ChatSearchBar({ + sessionId, + onOpenHit, +}: { + sessionId: string; + onOpenHit?: ((hit: SearchHit) => void) | undefined; +}) { + const { baseUrl, config } = useConnection(); + const [input, setInput] = useState(''); + const [open, setOpen] = useState(false); + const [hits, setHits] = useState(null); + const [searching, setSearching] = useState(false); + const [error, setError] = useState(null); + const rootRef = useRef(null); + + // A session switch drops the previous session's query and results. + useEffect(() => { + setInput(''); + setHits(null); + setError(null); + setOpen(false); + }, [sessionId]); + + // Clicking outside the bar closes the result panel. + useEffect(() => { + if (!open) return; + const onDown = (event: MouseEvent) => { + if (rootRef.current !== null && !rootRef.current.contains(event.target as Node)) { + setOpen(false); + } + }; + document.addEventListener('mousedown', onDown); + return () => { + document.removeEventListener('mousedown', onDown); + }; + }, [open]); + + const search = async () => { + const query = input.trim(); + if (query === '' || searching) return; + setSearching(true); + setError(null); + setOpen(true); + try { + const token = config.token.trim(); + const page = await fetchSearchPage({ + baseUrl, + token: token === '' ? undefined : token, + query, + // Substring semantics (Ctrl+F-like): terms mode is whole-token matching, + // which surprises in an in-session search box. + mode: 'literal', + container: { sessionId }, + pageSize: PAGE_SIZE, + }); + setHits(page.items); + } catch (error) { + setHits(null); + setError(error); + } finally { + setSearching(false); + } + }; + + return ( +
+ setInput(e.target.value)} + onKeyDown={(e) => { + if (e.key === 'Enter') { + e.preventDefault(); + void search(); + } else if (e.key === 'Escape') { + setOpen(false); + } + }} + onFocus={() => { + if (hits !== null) setOpen(true); + }} + /> + {open ? ( +
+ {searching ? ( +
Searching…
+ ) : error !== null ? ( + + ) : hits === null ? null : hits.length === 0 ? ( +
+ No hits in this session. +
+ ) : ( +
+ {hits.map((hit, i) => ( + + ))} +
+ )} +
+ ) : null} +
+ ); +} diff --git a/apps/kimi-inspect/src/components/ChatView.tsx b/apps/kimi-inspect/src/components/ChatView.tsx index 90153e01c2..2ba937a2e3 100644 --- a/apps/kimi-inspect/src/components/ChatView.tsx +++ b/apps/kimi-inspect/src/components/ChatView.tsx @@ -19,17 +19,6 @@ * derives from transcript state (`meta.activity` / running turns). */ -import { - createContext, - useCallback, - useContext, - useEffect, - useLayoutEffect, - useRef, - useState, - useSyncExternalStore, -} from 'react'; - import { IAgentRPCService } from '@moonshot-ai/agent-core-v2/agent/rpc/rpc'; import { ISessionApprovalService } from '@moonshot-ai/agent-core-v2/session/approval/approval'; import { @@ -56,24 +45,51 @@ import { type TurnOrigin, type TurnState, } from '@moonshot-ai/transcript'; +import { + createContext, + useCallback, + useContext, + useEffect, + useLayoutEffect, + useRef, + useState, + useSyncExternalStore, +} from 'react'; -import { useConnection } from '../connection'; import { AuditTrail } from '../audit/trail'; +import { useConnection } from '../connection'; +import type { SearchHit } from '../search/api'; import { fetchTranscriptOps, fetchTranscriptPage, TRANSCRIPT_PAGE_SIZE } from '../transcript/api'; import { createCoalescedRunner, + hasTurnId, oldestTurnId, recoverLoadedWindow, TranscriptChatStore, } from '../transcript/store'; import { TranscriptWs } from '../transcript/ws'; import { ActionButton, Badge, ErrorLine, JsonView, relTime } from '../ui'; +import { ChatSearchBar } from './ChatSearchBar'; const noopSubscribe = () => () => {}; /** Active session id for deeply nested interaction views (approve/answer buttons). */ const SessionContext = createContext(''); +/** + * A navigation request into the chat timeline (e.g. from a search hit). The + * channel pages backwards until the turn enters the loaded window, then + * scrolls it into view and flashes it briefly. + */ +export interface ChatJump { + /** Turn to locate (`t`); omitted = switch session/agent only, no scroll. */ + readonly turnId?: string | undefined; + /** Step within the turn (`t.`); falls back to the turn card. */ + readonly stepId?: string | undefined; + /** Changes on every request so re-clicking the same hit re-triggers. */ + readonly nonce: number; +} + interface TranscriptChannel { /** Null until the effect has created the store (pre-ready / no session). */ readonly store: TranscriptChatStore | null; @@ -326,12 +342,21 @@ export function ChatView({ agentId, ready, onTrailChange, + jump, + onJumpHandled, + onOpenSearchHit, }: { sessionId: string | null; agentId: string; ready: boolean; /** Hands the audit trail of the current channel up to the app shell (the audit panel lives in the right dock, not inside this view). */ onTrailChange?: (trail: AuditTrail | null) => void; + /** Pending navigation into the timeline (search result click). */ + jump?: ChatJump | null | undefined; + /** Called once the jump has been processed (or found un-actionable). */ + onJumpHandled?: (() => void) | undefined; + /** Hands an in-chat search hit up to the app shell (agent switch + jump). */ + onOpenSearchHit?: ((hit: SearchHit) => void) | undefined; }) { const { klient, baseUrl, config } = useConnection(); const [input, setInput] = useState(''); @@ -343,6 +368,8 @@ export function ChatView({ const anchorRef = useRef(null); /** Whether the viewport was pinned to the bottom before the last update. */ const stickBottomRef = useRef(true); + /** The jump target being flashed (cleared on a timer). */ + const [flash, setFlash] = useState<{ turnId: string; stepId?: string | undefined } | null>(null); const captureAnchor = useCallback(() => { const el = scrollRef.current; @@ -363,6 +390,82 @@ export function ChatView({ onTrailChange?.(trail); }, [onTrailChange, trail]); + // Jump navigation (search result click): once the channel has loaded, page + // backwards until the target turn enters the window, then scroll to the + // step (or the turn card) and flash it briefly. A turn that never appears + // (cut by an undo) degrades to no scroll. + useEffect(() => { + if (jump === null || jump === undefined || !loaded || store === null || sessionId === null) { + return; + } + if (jump.turnId === undefined) { + onJumpHandled?.(); + return; + } + let cancelled = false; + const turnId = jump.turnId; + const stepId = jump.stepId; + void (async () => { + stickBottomRef.current = false; + const token = config.token.trim(); + let recoverBefore: string | undefined; + await recoverLoadedWindow( + store, + turnId, + (beforeTurn) => { + recoverBefore = beforeTurn; + return fetchTranscriptPage({ + baseUrl, + token: token === '' ? undefined : token, + sessionId, + agentId, + beforeTurn, + pageSize: TRANSCRIPT_PAGE_SIZE, + }); + }, + () => cancelled, + (page) => { + trail?.recordRest( + { beforeTurn: recoverBefore, pageSize: TRANSCRIPT_PAGE_SIZE }, + 'prepend', + page, + store.getState(), + ); + }, + ); + if (cancelled) return; + if (!hasTurnId(store.getState().items, turnId)) { + onJumpHandled?.(); + return; + } + setFlash({ turnId, stepId }); + // The prepend renders asynchronously; wait two frames before scrolling. + requestAnimationFrame(() => { + requestAnimationFrame(() => { + if (cancelled) return; + const root = scrollRef.current; + const stepEl = + stepId !== undefined + ? root?.querySelector(`[data-step-id="${CSS.escape(stepId)}"]`) + : undefined; + const target = stepEl ?? root?.querySelector(`[data-turn-id="${CSS.escape(turnId)}"]`); + target?.scrollIntoView({ block: 'start' }); + }); + }); + onJumpHandled?.(); + })(); + return () => { + cancelled = true; + }; + }, [jump, loaded, store, sessionId, agentId, baseUrl, config, trail, onJumpHandled]); + + // The flash highlight clears itself after a short moment. + useEffect(() => { + if (flash === null) return; + const timer = setTimeout(() => setFlash(null), 2400); + return () => clearTimeout(timer); + }, [flash]); + useLayoutEffect(() => { const el = scrollRef.current; if (el === null) return; @@ -502,7 +605,9 @@ export function ChatView({ {state.pendingInteractions.size} pending ) : null}
- + + +
{state.hasMoreOlder ? (
@@ -545,10 +650,22 @@ export function ChatView({
todo (latest)
{latestTodo.items.map((entry, i) => (
- + {entry.status === 'done' ? '✔' : entry.status === 'in_progress' ? '◐' : '□'} - + {entry.title}
@@ -569,6 +686,7 @@ export function ChatView({ tasks={state.tasks} interactions={state.interactions} attachments={state.attachments} + flash={flash} />
))} @@ -576,7 +694,7 @@ export function ChatView({ ))}
- +
{sendError !== null ? (
@@ -618,15 +736,26 @@ function ItemView({ tasks, interactions, attachments, + flash, }: { item: TranscriptItem; tasks: ReadonlyMap; interactions: ReadonlyMap; attachments: ReadonlyMap; + /** The jump target being flashed, if any. */ + flash?: { turnId: string; stepId?: string | undefined } | null | undefined; }) { switch (item.kind) { case 'turn': - return ; + return ( + + ); case 'marker': return ; case 'taskref': @@ -674,20 +803,31 @@ function TurnView({ tasks, interactions, attachments, + flash, }: { turn: TranscriptTurn; tasks: ReadonlyMap; interactions: ReadonlyMap; attachments: ReadonlyMap; + /** The jump target being flashed, if any. */ + flash?: { turnId: string; stepId?: string | undefined } | null | undefined; }) { + const turnFlashed = flash?.turnId === turn.turnId && flash.stepId === undefined; return ( -
+
{turn.turnId} {turn.origin.kind} {turn.state} {turn.startedAt !== undefined ? ( - {relTime(Date.parse(turn.startedAt))} + + {relTime(Date.parse(turn.startedAt))} + ) : null} {turn.usage !== undefined ? ( {usageText(turn.usage)} @@ -701,7 +841,11 @@ function TurnView({ ) : null} {turn.steps.map((step) => ( -
+
{step.frames.map((frame) => (
@@ -798,15 +943,21 @@ function AttachmentChips({ {ids.map((id) => { const attachment = attachments.get(id); const label = attachment?.name ?? attachment?.mediaType ?? id; - const href = - attachment?.source?.kind === 'url' ? attachment.source.url : undefined; + const href = attachment?.source?.kind === 'url' ? attachment.source.url : undefined; return ( - 📎 {href !== undefined ? {label} : label} + 📎{' '} + {href !== undefined ? ( + + {label} + + ) : ( + label + )} ); })} @@ -892,7 +1043,9 @@ function ToolFrameView({ return (
- + tool {frame.name} @@ -906,11 +1059,15 @@ function ToolFrameView({ ))} {task !== undefined ? task: {task.state} : null} - {frame.todoId !== undefined ? todo: {frame.todoId} : null} + {frame.todoId !== undefined ? ( + todo: {frame.todoId} + ) : null}
{frame.input !== undefined ? ( typeof frame.input === 'string' ? ( - 
{frame.input}
+ 
+            {frame.input}
+
) : ( ) @@ -1009,10 +1166,12 @@ function InteractionEntityView({ if (parts.length > 0) answers[question.question] = parts.join(', '); } // Mirror the TUI adapter: no answers at all resolves with null. - const result = - Object.keys(answers).length > 0 ? { answers, method: 'enter' as const } : null; + const result = Object.keys(answers).length > 0 ? { answers, method: 'enter' as const } : null; run(() => - klient.session(sessionId).service(ISessionQuestionService).answer(interaction.interactionId, result), + klient + .session(sessionId) + .service(ISessionQuestionService) + .answer(interaction.interactionId, result), ); }; @@ -1109,7 +1268,9 @@ function NoticeFrameView({ frame }: { frame: NoticeFrame }) { : 'bg-neutral-900/60 text-neutral-400'; return (
- {frame.source !== undefined ? [{frame.source}] : null} + {frame.source !== undefined ? ( + [{frame.source}] + ) : null} {frame.message} {frame.detail !== undefined ? : null}
diff --git a/apps/kimi-inspect/src/components/Inspector.tsx b/apps/kimi-inspect/src/components/Inspector.tsx index 3ecc7d8181..0afb270a41 100644 --- a/apps/kimi-inspect/src/components/Inspector.tsx +++ b/apps/kimi-inspect/src/components/Inspector.tsx @@ -13,11 +13,10 @@ * log — was removed server-side, so there is no live push to render. */ +import { ISessionMetadata } from '@moonshot-ai/agent-core-v2/session/sessionMetadata/sessionMetadata'; import { useQuery } from '@tanstack/react-query'; import { useEffect, useMemo, useState } from 'react'; -import { ISessionMetadata } from '@moonshot-ai/agent-core-v2/session/sessionMetadata/sessionMetadata'; - import { serviceByName } from '../channel'; import { useConnection } from '../connection'; import { type AnyService } from '../panels'; @@ -40,7 +39,11 @@ export function Inspector({ const meta = useQuery({ queryKey: ['sessionMeta', sessionId], - queryFn: () => klient.session(sessionId as string).service(ISessionMetadata).read(), + queryFn: () => + klient + .session(sessionId as string) + .service(ISessionMetadata) + .read(), enabled: sessionId !== null && ready, }); @@ -76,11 +79,13 @@ export function Inspector({ // session that isn't selected/ready. const proxyFor = useMemo(() => { return (name: string): AnyService | null => { - return serviceByName(klient, name, { - scope: 'agent', - sessionId: sessionId !== null && ready ? sessionId : undefined, - agentId: effectiveAgent, - }) ?? null; + return ( + serviceByName(klient, name, { + scope: 'agent', + sessionId: sessionId !== null && ready ? sessionId : undefined, + agentId: effectiveAgent, + }) ?? null + ); }; }, [klient, sessionId, effectiveAgent, ready]); @@ -107,11 +112,15 @@ export function Inspector({ {stoppedAgents.has(effectiveAgent) ? (
- this agent is not materialized in the running server (e.g. created before a - restart) — calls will fail; its persisted records remain on disk + this agent is not materialized in the running server (e.g. created before a restart) — + calls will fail; its persisted records remain on disk +
+ ) : null} + {meta.isError ? ( +
+
) : null} - {meta.isError ?
: null}
) : null} @@ -222,7 +231,9 @@ function PlanEntryView({ entry }: { entry: TranscriptPlanInfo }) { return (
- {entry.toolCallId} + + {entry.toolCallId} + {entry.source} {review !== undefined ? ( void reload()}>Load
- {error !== null ?
: null} + {error !== null ? ( +
+ +
+ ) : null} {pending.length === 0 ? (
nothing pending (click Load to check)
) : ( pending.map((item) => ( -
+
{item.kind} {item.id} @@ -95,8 +101,12 @@ export function InteractionsCard({ sessionId }: { sessionId: string }) {
- void decide(item.id, 'approved')}>Approve - void decide(item.id, 'rejected')}>Reject + void decide(item.id, 'approved')}> + Approve + + void decide(item.id, 'rejected')}> + Reject +
) : item.kind === 'question' ? ( @@ -166,11 +176,7 @@ function QuestionView({ * numbers/booleans are stringified, anything else (or missing) falls back — * never "[object Object]". */ -function payloadField( - payload: Record, - key: string, - fallback: string, -): string { +function payloadField(payload: Record, key: string, fallback: string): string { const value = payload[key]; if (typeof value === 'string') return value; if (typeof value === 'number' || typeof value === 'boolean') return String(value); diff --git a/apps/kimi-inspect/src/components/ModelCatalogView.tsx b/apps/kimi-inspect/src/components/ModelCatalogView.tsx index cae26447ad..6b26f0b923 100644 --- a/apps/kimi-inspect/src/components/ModelCatalogView.tsx +++ b/apps/kimi-inspect/src/components/ModelCatalogView.tsx @@ -17,20 +17,19 @@ * There is no live event push; the queries refresh on a slow poll. */ -import { useQuery, useQueryClient } from '@tanstack/react-query'; -import { useEffect, useRef, useState } from 'react'; - +import { IAgentProfileService } from '@moonshot-ai/agent-core-v2/agent/profile/profile'; +import { ISessionLifecycleService } from '@moonshot-ai/agent-core-v2/app/sessionLifecycle/sessionLifecycle'; +import type { InspectionSource } from '@moonshot-ai/agent-core-v2/kosong/contract/inspection'; +import type { TokenUsage } from '@moonshot-ai/agent-core-v2/kosong/contract/usage'; import { IModelCatalog, type ModelCatalogItem, type ModelPingResult, type ProviderCatalogItem, } from '@moonshot-ai/agent-core-v2/kosong/model/catalog'; -import type { InspectionSource } from '@moonshot-ai/agent-core-v2/kosong/contract/inspection'; -import type { TokenUsage } from '@moonshot-ai/agent-core-v2/kosong/contract/usage'; import { IModelService } from '@moonshot-ai/agent-core-v2/kosong/model/model'; -import { ISessionLifecycleService } from '@moonshot-ai/agent-core-v2/app/sessionLifecycle/sessionLifecycle'; -import { IAgentProfileService } from '@moonshot-ai/agent-core-v2/agent/profile/profile'; +import { useQuery, useQueryClient } from '@tanstack/react-query'; +import { useEffect, useRef, useState } from 'react'; import { useConnection } from '../connection'; import { ActionButton, Badge, ErrorLine, JsonTree, JsonView, errorMessage } from '../ui'; @@ -207,9 +206,7 @@ export function ModelCatalogView({ ))}
- queryClient.invalidateQueries({ queryKey: ['modelCatalog'] })} - > + queryClient.invalidateQueries({ queryKey: ['modelCatalog'] })}> Refresh
@@ -220,17 +217,17 @@ export function ModelCatalogView({ ) : null} {loading ?
loading…
: null} {!loading && flatEntries.length === 0 ? ( -
no providers or models configured
+
+ no providers or models configured +
) : null}
{/* left: the model list */} -
- +
+
{/* center: one god object per model */}
{entry.item.display_name !== undefined && entry.item.display_name !== entry.item.model ? ( -
{entry.item.display_name}
+
+ {entry.item.display_name} +
) : null}
@@ -369,7 +368,8 @@ function ModelSection({ refetchInterval: 15_000, }); const [ping, setPing] = useState< - { readonly status: 'idle' | 'running' } | { readonly status: 'done'; readonly result: ModelPingResult } + | { readonly status: 'idle' | 'running' } + | { readonly status: 'done'; readonly result: ModelPingResult } >({ status: 'idle' }); const [creating, setCreating] = useState(false); const [sessionError, setSessionError] = useState(null); diff --git a/apps/kimi-inspect/src/components/NavRail.tsx b/apps/kimi-inspect/src/components/NavRail.tsx index ede78fc77a..126791c0ab 100644 --- a/apps/kimi-inspect/src/components/NavRail.tsx +++ b/apps/kimi-inspect/src/components/NavRail.tsx @@ -6,7 +6,7 @@ import type { ReactNode } from 'react'; -export type AppView = 'chat' | 'models' | 'services' | 'bash'; +export type AppView = 'chat' | 'search' | 'models' | 'services' | 'bash'; interface ViewDef { readonly id: AppView; @@ -35,6 +35,16 @@ const VIEWS: readonly ViewDef[] = [ ), }, + { + id: 'search', + title: 'Search', + icon: ( + + + + + ), + }, { id: 'models', title: 'Model Catalog', diff --git a/apps/kimi-inspect/src/components/RightPanel.tsx b/apps/kimi-inspect/src/components/RightPanel.tsx index 554a2e690d..b6d5edcd72 100644 --- a/apps/kimi-inspect/src/components/RightPanel.tsx +++ b/apps/kimi-inspect/src/components/RightPanel.tsx @@ -13,13 +13,12 @@ * switches. */ -import { useState } from 'react'; - import { IAgentStateService } from '@moonshot-ai/agent-core-v2/agent/state/agentState'; +import { useState } from 'react'; +import type { AuditTrail } from '../audit/trail'; import { useConnection } from '../connection'; import { Badge } from '../ui'; -import type { AuditTrail } from '../audit/trail'; import { AuditPanel } from './audit/AuditPanel'; import { Inspector } from './Inspector'; import { StateCard } from './StateCard'; diff --git a/apps/kimi-inspect/src/components/SearchView.tsx b/apps/kimi-inspect/src/components/SearchView.tsx new file mode 100644 index 0000000000..b76dd3da36 --- /dev/null +++ b/apps/kimi-inspect/src/components/SearchView.tsx @@ -0,0 +1,254 @@ +/** + * Search view — the global cross-session message search (`POST /api/v1/search`). + * Full width like the other non-chat views. Results are pageable (Load more); + * clicking a hit hands it to the app shell (`onOpenResult`), which switches + * to the chat view at the hit's session / agent / turn / step. + */ + +import { useState } from 'react'; + +import { useConnection } from '../connection'; +import { fetchSearchPage, type SearchHit, type SearchIndexState } from '../search/api'; +import { ActionButton, Badge, ErrorLine, relTime } from '../ui'; + +type RoleFilter = 'all' | 'user' | 'assistant' | 'title'; +type SortOrder = 'score' | 'time_desc' | 'time_asc'; +type SearchMode = 'terms' | 'literal'; + +const PAGE_SIZE = 20; + +interface ExecutedSearch { + readonly query: string; + readonly role?: 'user' | 'assistant' | 'title' | undefined; + readonly sort: SortOrder; + readonly mode: SearchMode; + readonly items: readonly SearchHit[]; + readonly hasMore: boolean; + readonly pageToken?: string | undefined; + readonly incomplete?: 'candidate_cap' | undefined; + readonly source?: 'live' | 'index' | undefined; + readonly indexState: SearchIndexState; +} + +export function SearchView({ onOpenResult }: { onOpenResult: (hit: SearchHit) => void }) { + const { baseUrl, config } = useConnection(); + const [input, setInput] = useState(''); + const [role, setRole] = useState('all'); + const [sort, setSort] = useState('score'); + const [exact, setExact] = useState(false); + const [result, setResult] = useState(null); + const [searching, setSearching] = useState(false); + const [loadingMore, setLoadingMore] = useState(false); + const [error, setError] = useState(null); + + const runSearch = async () => { + const query = input.trim(); + if (query === '' || searching) return; + setSearching(true); + setError(null); + try { + const token = config.token.trim(); + const mode: SearchMode = exact ? 'literal' : 'terms'; + const page = await fetchSearchPage({ + baseUrl, + token: token === '' ? undefined : token, + query, + role: role === 'all' ? undefined : role, + sort, + mode, + pageSize: PAGE_SIZE, + }); + setResult({ + query, + role: role === 'all' ? undefined : role, + sort, + mode, + items: page.items, + hasMore: page.hasMore, + pageToken: page.pageToken, + incomplete: page.incomplete, + source: page.source, + indexState: page.indexState, + }); + } catch (error) { + setError(error); + } finally { + setSearching(false); + } + }; + + const loadMore = async () => { + if (result === null || !result.hasMore || loadingMore) return; + setLoadingMore(true); + setError(null); + try { + const token = config.token.trim(); + const page = await fetchSearchPage({ + baseUrl, + token: token === '' ? undefined : token, + query: result.query, + role: result.role, + sort: result.sort, + mode: result.mode, + pageSize: PAGE_SIZE, + pageToken: result.pageToken, + }); + setResult({ + ...result, + items: [...result.items, ...page.items], + hasMore: page.hasMore, + pageToken: page.pageToken, + incomplete: page.incomplete, + source: page.source, + indexState: page.indexState, + }); + } catch (error) { + setError(error); + } finally { + setLoadingMore(false); + } + }; + + return ( +
+
+ setInput(e.target.value)} + onKeyDown={(e) => { + if (e.key === 'Enter') { + e.preventDefault(); + void runSearch(); + } + }} + /> + + + + void runSearch()} disabled={searching || input.trim() === ''}> + {searching ? 'Searching…' : 'Search'} + + {result !== null ? ( + + {result.indexState.state === 'building' + ? `index building (${result.indexState.indexedSessions}/${result.indexState.totalSessions} sessions) — results may be incomplete` + : `${result.indexState.documents} documents indexed (${result.indexState.state})`} + + ) : null} + {result?.source !== undefined ? ( + + + source: {result.source} + + + ) : null} +
+ +
+ {error !== null ? ( +
+ +
+ ) : null} + {result === null ? ( +
+ Search user prompts, assistant replies and session titles across every session. +
+ ) : result.items.length === 0 && !searching ? ( +
No hits for “{result.query}”.
+ ) : ( +
+ {result.incomplete === 'candidate_cap' ? ( +
+ too many matches — the candidate set was truncated, results may be incomplete +
+ ) : null} + {result.items.map((hit, i) => ( + + ))} + {result.hasMore ? ( +
+ void loadMore()} disabled={loadingMore}> + {loadingMore ? 'Loading…' : 'Load more'} + +
+ ) : null} +
+ )} +
+
+ ); +} + +function HitCard({ hit, onOpen }: { hit: SearchHit; onOpen: (hit: SearchHit) => void }) { + return ( + + ); +} diff --git a/apps/kimi-inspect/src/components/ServicePanels.tsx b/apps/kimi-inspect/src/components/ServicePanels.tsx index 0a5a690824..466a07fb19 100644 --- a/apps/kimi-inspect/src/components/ServicePanels.tsx +++ b/apps/kimi-inspect/src/components/ServicePanels.tsx @@ -25,11 +25,7 @@ import { useQuery } from '@tanstack/react-query'; import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; -import { - fetchChannelDescriptors, - type ChannelDescriptor, - type ChannelScope, -} from '../channel'; +import { fetchChannelDescriptors, type ChannelDescriptor, type ChannelScope } from '../channel'; import { useConnection } from '../connection'; import { AGENT_PANELS, @@ -123,18 +119,11 @@ export function ScopePanels({ ) : null} {entries.map(({ name, channel, def }) => { if (def !== undefined) { - return ( - - ); + return ; } if (channel === undefined) return null; return ( - + ); })} @@ -280,7 +269,11 @@ export function ScopePanelsScrollspy({ ))} -
+
{channels.isError ? (
@@ -359,7 +352,15 @@ function makeRecordingProxy( const at = Date.now(); return Promise.resolve(member(...args)).then( (result) => { - record({ service, method: String(prop), args, at, durationMs: Date.now() - at, ok: true, result }); + record({ + service, + method: String(prop), + args, + at, + durationMs: Date.now() - at, + ok: true, + result, + }); return result; }, (error: unknown) => { @@ -413,7 +414,9 @@ function HistoryPane({ className="flex cursor-pointer items-center gap-2 px-2 py-1.5 select-none" onClick={() => onToggle(r.id)} > - + ● @@ -517,12 +520,18 @@ function ServiceCard({ ) : null}
- {error !== null ?
: null} + {error !== null ? ( +
+ +
+ ) : null} {def.fetch !== undefined ? ( loaded ? ( ) : ( -
click Load to read this Service
+
+ click Load to read this Service +
) ) : null} {def.actions !== undefined && def.actions.length > 0 ? ( @@ -560,7 +569,9 @@ function ServiceCard({
) : null} {def.fetch === undefined && data !== undefined ? ( -
+
+ +
) : null}
@@ -738,7 +749,9 @@ function MethodArgInputs({ onChange(fieldKey(i), v)} diff --git a/apps/kimi-inspect/src/components/SessionPane.tsx b/apps/kimi-inspect/src/components/SessionPane.tsx index ba083ea26d..3469c08eea 100644 --- a/apps/kimi-inspect/src/components/SessionPane.tsx +++ b/apps/kimi-inspect/src/components/SessionPane.tsx @@ -10,9 +10,8 @@ * without a Refresh button. */ -import { useMemo, useState } from 'react'; - import { ISessionStateService } from '@moonshot-ai/agent-core-v2/session/state/sessionState'; +import { useMemo, useState } from 'react'; import { serviceByName } from '../channel'; import { useConnection } from '../connection'; @@ -23,13 +22,7 @@ import { StateCard } from './StateCard'; type Tab = 'services' | 'state'; -export function SessionPane({ - sessionId, - ready, -}: { - sessionId: string | null; - ready: boolean; -}) { +export function SessionPane({ sessionId, ready }: { sessionId: string | null; ready: boolean }) { const { klient } = useConnection(); const [tab, setTab] = useState('services'); @@ -77,9 +70,7 @@ export function SessionPane({ queryKey={['sessionState', sessionId]} title="Session state" label="sessionStateService" - fetchSnapshot={() => - klient.session(sessionId).service(ISessionStateService).snapshot() - } + fetchSnapshot={() => klient.session(sessionId).service(ISessionStateService).snapshot()} /> )}
diff --git a/apps/kimi-inspect/src/components/Sidebar.tsx b/apps/kimi-inspect/src/components/Sidebar.tsx index e299246582..5464cb1d27 100644 --- a/apps/kimi-inspect/src/components/Sidebar.tsx +++ b/apps/kimi-inspect/src/components/Sidebar.tsx @@ -7,20 +7,25 @@ * endpoint. */ -import { useQuery, useQueryClient } from '@tanstack/react-query'; -import { useState } from 'react'; - import { IAgentProfileService } from '@moonshot-ai/agent-core-v2/agent/profile/profile'; import { IConfigService } from '@moonshot-ai/agent-core-v2/app/config/config'; -import { ISessionIndex, type SessionSummary } from '@moonshot-ai/agent-core-v2/app/sessionIndex/sessionIndex'; +import { + ISessionIndex, + type SessionSummary, +} from '@moonshot-ai/agent-core-v2/app/sessionIndex/sessionIndex'; import { ISessionLifecycleService } from '@moonshot-ai/agent-core-v2/app/sessionLifecycle/sessionLifecycle'; -import { IWorkspaceService, type Workspace } from '@moonshot-ai/agent-core-v2/app/workspace/workspace'; +import { + IWorkspaceService, + type Workspace, +} from '@moonshot-ai/agent-core-v2/app/workspace/workspace'; import { IModelCatalog } from '@moonshot-ai/agent-core-v2/kosong/model/catalog'; +import { useQuery, useQueryClient } from '@tanstack/react-query'; +import { useState } from 'react'; +import type { SessionWorkFacts } from '../activity/store'; +import { useSessionActivities } from '../activity/useSessionActivity'; import type { InspectClient } from '../channel'; import { useConnection } from '../connection'; -import { useSessionActivities } from '../activity/useSessionActivity'; -import type { SessionWorkFacts } from '../activity/store'; import { Badge, ErrorLine, relTime } from '../ui'; /** @@ -61,11 +66,17 @@ export function Sidebar({ queryFn: () => klient .core(ISessionIndex) - .list({ workspaceIds: workspaceId === null ? undefined : [workspaceId], includeArchived: true, limit: 200 }), + .list({ + workspaceIds: workspaceId === null ? undefined : [workspaceId], + includeArchived: true, + limit: 200, + }), refetchInterval: 15_000, }); - const sortedWorkspaces = (workspaces.data ?? []).toSorted((a, b) => b.lastOpenedAt - a.lastOpenedAt); + const sortedWorkspaces = (workspaces.data ?? []).toSorted( + (a, b) => b.lastOpenedAt - a.lastOpenedAt, + ); const sortedSessions = (sessions.data?.items ?? []).toSorted((a, b) => b.updatedAt - a.updatedAt); const createSession = async (ws: Workspace | null) => { diff --git a/apps/kimi-inspect/src/components/audit/AuditPanel.tsx b/apps/kimi-inspect/src/components/audit/AuditPanel.tsx index c01e295fda..7dbdea7274 100644 --- a/apps/kimi-inspect/src/components/audit/AuditPanel.tsx +++ b/apps/kimi-inspect/src/components/audit/AuditPanel.tsx @@ -13,14 +13,13 @@ * raw REST request/response or WS payload). */ -import { useEffect, useMemo, useRef, useState, useSyncExternalStore } from 'react'; - import { EMPTY_AGENT_STATE } from '@moonshot-ai/transcript'; +import { useEffect, useMemo, useRef, useState, useSyncExternalStore } from 'react'; import { diffValue, type DiffNode } from '../../audit/diff'; import { serializeState } from '../../audit/serialize'; -import { tailTrunc } from '../../audit/truncate'; import type { AuditEntry, AuditTrail } from '../../audit/trail'; +import { tailTrunc } from '../../audit/truncate'; import { Badge } from '../../ui'; import { plainNode, StateTree } from './StateTree'; @@ -89,7 +88,8 @@ export function AuditPanel({ trail }: { trail: AuditTrail }) { const root: DiffNode | null = useMemo(() => { if (current === undefined || tab === 'event') return null; if (tab === 'state') return plainNode(serializeState(current.state)); - const prevState = currentPos > 0 ? (entries[currentPos - 1]?.state ?? EMPTY_AGENT_STATE) : EMPTY_AGENT_STATE; + const prevState = + currentPos > 0 ? (entries[currentPos - 1]?.state ?? EMPTY_AGENT_STATE) : EMPTY_AGENT_STATE; return diffValue(serializeState(prevState), serializeState(current.state)); }, [current, currentPos, entries, tab]); diff --git a/apps/kimi-inspect/src/components/audit/StateTree.test.tsx b/apps/kimi-inspect/src/components/audit/StateTree.test.tsx index 09715fe44d..8ce34226a2 100644 --- a/apps/kimi-inspect/src/components/audit/StateTree.test.tsx +++ b/apps/kimi-inspect/src/components/audit/StateTree.test.tsx @@ -7,11 +7,10 @@ * 2. Whole-subtree adds expand into fully fielded, indented tree rows. */ +import { EMPTY_AGENT_STATE, type AgentState, type TranscriptTurn } from '@moonshot-ai/transcript'; import { renderToStaticMarkup } from 'react-dom/server'; import { describe, expect, it } from 'vitest'; -import { EMPTY_AGENT_STATE, type AgentState, type TranscriptTurn } from '@moonshot-ai/transcript'; - import { diffValue } from '../../audit/diff'; import { serializeState } from '../../audit/serialize'; import { plainNode, StateTree } from './StateTree'; @@ -51,7 +50,10 @@ describe('StateTree', () => { }); it('expands whole-subtree adds into full field rows (all keys, no JSON dump)', () => { - const root = diffValue(serializeState(EMPTY_AGENT_STATE), serializeState(stateWith([turn(0, 'HELLO')]))); + const root = diffValue( + serializeState(EMPTY_AGENT_STATE), + serializeState(stateWith([turn(0, 'HELLO')])), + ); const html = renderToStaticMarkup(); expect(html).not.toContain('{"kind"'); for (const field of ['turnId', 'ordinal', 'state', 'origin', 'prompt', 'steps']) { diff --git a/apps/kimi-inspect/src/components/audit/StateTree.tsx b/apps/kimi-inspect/src/components/audit/StateTree.tsx index 31fefb2598..f85ca17189 100644 --- a/apps/kimi-inspect/src/components/audit/StateTree.tsx +++ b/apps/kimi-inspect/src/components/audit/StateTree.tsx @@ -72,7 +72,9 @@ function Leaf({ value, tone }: { value: unknown; tone: DiffStatus }) { if (value === undefined) return undefined; if (typeof value === 'string') { if (value.includes('\n')) return ; - return "{tailTrunc(value)}"; + return ( + "{tailTrunc(value)}" + ); } if (typeof value === 'number' || typeof value === 'boolean') { return {String(value)}; diff --git a/apps/kimi-inspect/src/connection.tsx b/apps/kimi-inspect/src/connection.tsx index ae727b6ea1..652d053d96 100644 --- a/apps/kimi-inspect/src/connection.tsx +++ b/apps/kimi-inspect/src/connection.tsx @@ -31,16 +31,8 @@ import { type ReactNode, } from 'react'; -import { - createInspectClient, - probeDebugSurface, - type InspectClient, -} from './channel'; -import { - fetchServerDiscovery, - pickDefaultServer, - useServerDiscovery, -} from './servers'; +import { createInspectClient, probeDebugSurface, type InspectClient } from './channel'; +import { fetchServerDiscovery, pickDefaultServer, useServerDiscovery } from './servers'; export interface ConnectionConfig { /** Server base URL; empty string means same-origin (the Vite dev proxy). */ @@ -135,8 +127,7 @@ export function ConnectionProvider({ children }: { children: ReactNode }) { readonly error: string | null; } | null>(null); const [probeNonce, setProbeNonce] = useState(0); - const configKey = - config === null ? null : `${resolveBaseUrl(config.url)}|${config.token.trim()}`; + const configKey = config === null ? null : `${resolveBaseUrl(config.url)}|${config.token.trim()}`; useEffect(() => { if (config === null || configKey === null) { diff --git a/apps/kimi-inspect/src/panels.ts b/apps/kimi-inspect/src/panels.ts index 58c3aa865d..92abed8a4f 100644 --- a/apps/kimi-inspect/src/panels.ts +++ b/apps/kimi-inspect/src/panels.ts @@ -16,17 +16,6 @@ * every Service. */ -import { IAuthSummaryService } from '@moonshot-ai/agent-core-v2/app/auth/auth'; -import { IConfigService } from '@moonshot-ai/agent-core-v2/app/config/config'; -import { IFlagService } from '@moonshot-ai/agent-core-v2/app/flag/flag'; -import { IProviderService } from '@moonshot-ai/agent-core-v2/kosong/provider/provider'; - -import { ISessionApprovalService } from '@moonshot-ai/agent-core-v2/session/approval/approval'; -import { ISessionInitService } from '@moonshot-ai/agent-core-v2/session/sessionInit/sessionInit'; -import { ISessionInteractionService } from '@moonshot-ai/agent-core-v2/session/interaction/interaction'; -import { ISessionMetadata } from '@moonshot-ai/agent-core-v2/session/sessionMetadata/sessionMetadata'; -import { ISessionQuestionService } from '@moonshot-ai/agent-core-v2/session/question/question'; -import { ISessionWorkspaceContext } from '@moonshot-ai/agent-core-v2/session/workspaceContext/workspaceContext'; import { IAgentActivityView } from '@moonshot-ai/agent-core-v2/agent/activityView/activityView'; import { IAgentContextSizeService } from '@moonshot-ai/agent-core-v2/agent/contextSize/contextSize'; import { IAgentGoalService } from '@moonshot-ai/agent-core-v2/agent/goal/goal'; @@ -40,6 +29,16 @@ import { IAgentSwarmService } from '@moonshot-ai/agent-core-v2/agent/swarm/swarm import { IAgentTaskService } from '@moonshot-ai/agent-core-v2/agent/task/task'; import { IAgentToolRegistryService } from '@moonshot-ai/agent-core-v2/agent/toolRegistry/toolRegistry'; import { IAgentUsageService } from '@moonshot-ai/agent-core-v2/agent/usage/usage'; +import { IAuthSummaryService } from '@moonshot-ai/agent-core-v2/app/auth/auth'; +import { IConfigService } from '@moonshot-ai/agent-core-v2/app/config/config'; +import { IFlagService } from '@moonshot-ai/agent-core-v2/app/flag/flag'; +import { IProviderService } from '@moonshot-ai/agent-core-v2/kosong/provider/provider'; +import { ISessionApprovalService } from '@moonshot-ai/agent-core-v2/session/approval/approval'; +import { ISessionInteractionService } from '@moonshot-ai/agent-core-v2/session/interaction/interaction'; +import { ISessionQuestionService } from '@moonshot-ai/agent-core-v2/session/question/question'; +import { ISessionInitService } from '@moonshot-ai/agent-core-v2/session/sessionInit/sessionInit'; +import { ISessionMetadata } from '@moonshot-ai/agent-core-v2/session/sessionMetadata/sessionMetadata'; +import { ISessionWorkspaceContext } from '@moonshot-ai/agent-core-v2/session/workspaceContext/workspaceContext'; /** Loosely-typed view of a scoped service proxy (every member is a remote call). */ export type AnyService = Record Promise>; @@ -229,7 +228,12 @@ export const AGENT_PANELS: readonly ServicePanelDef[] = [ scope: 'agent', fetch: (svc) => call(svc, 'list'), actions: [ - { label: 'Stop task', input: 'Task id', danger: true, run: (svc, id) => call(svc, 'stop', id) }, + { + label: 'Stop task', + input: 'Task id', + danger: true, + run: (svc, id) => call(svc, 'stop', id), + }, { label: 'stopAll', danger: true, run: (svc) => call(svc, 'stopAll') }, ], }, @@ -248,7 +252,11 @@ export const AGENT_PANELS: readonly ServicePanelDef[] = [ scope: 'agent', fetch: (svc) => call(svc, 'list'), actions: [ - { label: 'Reconnect server', input: 'Server name', run: (svc, name) => call(svc, 'reconnect', name) }, + { + label: 'Reconnect server', + input: 'Server name', + run: (svc, name) => call(svc, 'reconnect', name), + }, ], }, { diff --git a/apps/kimi-inspect/src/search/api.test.ts b/apps/kimi-inspect/src/search/api.test.ts new file mode 100644 index 0000000000..c90fe1d805 --- /dev/null +++ b/apps/kimi-inspect/src/search/api.test.ts @@ -0,0 +1,215 @@ +/** + * Tests for the global search REST client (`POST /api/v1/search`): request + * serialization, envelope unwrapping, and hit parsing. + */ + +import { describe, expect, it } from 'vitest'; + +import { fetchSearchPage } from './api'; + +function okEnvelope(data: unknown) { + return { code: 0, msg: 'success', data, request_id: 'r1' }; +} + +function fakeFetch(envelope: unknown) { + const calls: { url: string; init?: RequestInit }[] = []; + const fetchImpl = (async (url: string | URL, init?: RequestInit) => { + calls.push({ url: String(url), init }); + return { json: async () => envelope }; + }) as unknown as typeof fetch; + return { calls, fetchImpl }; +} + +const pageData = { + items: [ + { + session_id: 's1', + workspace_id: 'ws', + session_title: '搜索重构', + agent_id: 'main', + role: 'assistant', + snippet: 'Here is the apple guide.', + time: 1_700_000_000_000, + turn: 3, + step_id: 't3.2', + score: 1.5, + }, + { + session_id: 's2', + workspace_id: 'ws', + session_title: 'title doc', + agent_id: '', + role: 'title', + snippet: '苹果询价', + time: 1_700_000_100_000, + score: 0.9, + }, + // Malformed entries are dropped, not fatal. + { session_id: 42 }, + ], + has_more: true, + page_token: 'tok-next', + index_state: { state: 'ready', indexed_sessions: 2, total_sessions: 2, documents: 10 }, +}; + +describe('fetchSearchPage', () => { + it('posts the snake_case body with bearer auth and maps hits to camelCase', async () => { + const { calls, fetchImpl } = fakeFetch(okEnvelope(pageData)); + const page = await fetchSearchPage({ + baseUrl: 'http://h:1', + token: 'tok', + query: '苹果', + role: 'assistant', + sort: 'time_desc', + pageSize: 20, + pageToken: 'tok-prev', + fetchImpl, + }); + + expect(calls).toHaveLength(1); + expect(calls[0]!.url).toBe('http://h:1/api/v1/search'); + expect(calls[0]!.init?.method).toBe('POST'); + expect(calls[0]!.init?.headers).toEqual({ + 'content-type': 'application/json', + authorization: 'Bearer tok', + }); + expect(JSON.parse(calls[0]!.init?.body as string)).toEqual({ + query: '苹果', + role: 'assistant', + sort: 'time_desc', + page_size: 20, + page_token: 'tok-prev', + }); + + expect(page.items).toHaveLength(2); + const assistant = page.items[0]!; + expect(assistant.sessionId).toBe('s1'); + expect(assistant.role).toBe('assistant'); + expect(assistant.turn).toBe(3); + expect(assistant.stepId).toBe('t3.2'); + const title = page.items[1]!; + expect(title.agentId).toBe(''); + expect(title.turn).toBeUndefined(); + expect(title.stepId).toBeUndefined(); + expect(page.hasMore).toBe(true); + expect(page.pageToken).toBe('tok-next'); + expect(page.indexState).toEqual({ + state: 'ready', + indexedSessions: 2, + totalSessions: 2, + documents: 10, + }); + }); + + it('omits the authorization header when no token is configured', async () => { + const { calls, fetchImpl } = fakeFetch(okEnvelope(pageData)); + await fetchSearchPage({ baseUrl: 'http://h:1', query: '苹果', fetchImpl }); + expect(calls[0]!.init?.headers).toEqual({ 'content-type': 'application/json' }); + }); + + it('serializes mode into the body and parses incomplete: candidate_cap', async () => { + const { calls, fetchImpl } = fakeFetch( + okEnvelope({ ...pageData, has_more: false, incomplete: 'candidate_cap' }), + ); + const page = await fetchSearchPage({ + baseUrl: 'http://h:1', + query: 'C++', + mode: 'literal', + fetchImpl, + }); + + expect(JSON.parse(calls[0]!.init?.body as string)).toEqual({ + query: 'C++', + mode: 'literal', + }); + expect(page.incomplete).toBe('candidate_cap'); + }); + + it('leaves incomplete undefined for absent or unexpected values', async () => { + const { fetchImpl } = fakeFetch(okEnvelope(pageData)); + const page = await fetchSearchPage({ baseUrl: 'http://h:1', query: '苹果', fetchImpl }); + expect(page.incomplete).toBeUndefined(); + + const { fetchImpl: fetchImpl2 } = fakeFetch( + okEnvelope({ ...pageData, incomplete: 'something_else' }), + ); + const page2 = await fetchSearchPage({ + baseUrl: 'http://h:1', + query: '苹果', + fetchImpl: fetchImpl2, + }); + expect(page2.incomplete).toBeUndefined(); + }); + + it('serializes container into the body (session only, and session + agent)', async () => { + const { calls, fetchImpl } = fakeFetch(okEnvelope(pageData)); + await fetchSearchPage({ + baseUrl: 'http://h:1', + query: '苹果', + container: { sessionId: 's1' }, + fetchImpl, + }); + expect(JSON.parse(calls[0]!.init?.body as string)).toEqual({ + query: '苹果', + container: { session_id: 's1' }, + }); + + const { calls: calls2, fetchImpl: fetchImpl2 } = fakeFetch(okEnvelope(pageData)); + await fetchSearchPage({ + baseUrl: 'http://h:1', + query: '苹果', + container: { sessionId: 's1', agentId: 'main' }, + fetchImpl: fetchImpl2, + }); + expect(JSON.parse(calls2[0]!.init?.body as string)).toEqual({ + query: '苹果', + container: { session_id: 's1', agent_id: 'main' }, + }); + }); + + it('parses source: live and source: index', async () => { + const { fetchImpl } = fakeFetch(okEnvelope({ ...pageData, source: 'live' })); + const page = await fetchSearchPage({ baseUrl: 'http://h:1', query: '苹果', fetchImpl }); + expect(page.source).toBe('live'); + + const { fetchImpl: fetchImpl2 } = fakeFetch(okEnvelope({ ...pageData, source: 'index' })); + const page2 = await fetchSearchPage({ + baseUrl: 'http://h:1', + query: '苹果', + fetchImpl: fetchImpl2, + }); + expect(page2.source).toBe('index'); + }); + + it('leaves source undefined for absent or unknown values', async () => { + const { fetchImpl } = fakeFetch(okEnvelope(pageData)); + const page = await fetchSearchPage({ baseUrl: 'http://h:1', query: '苹果', fetchImpl }); + expect(page.source).toBeUndefined(); + + const { fetchImpl: fetchImpl2 } = fakeFetch(okEnvelope({ ...pageData, source: 'whatever' })); + const page2 = await fetchSearchPage({ + baseUrl: 'http://h:1', + query: '苹果', + fetchImpl: fetchImpl2, + }); + expect(page2.source).toBeUndefined(); + }); + + it('throws on a non-zero envelope code', async () => { + const { fetchImpl } = fakeFetch({ + code: 40001, + msg: 'query must be a non-empty string', + data: null, + }); + await expect(fetchSearchPage({ baseUrl: 'http://h:1', query: '', fetchImpl })).rejects.toThrow( + /40001/, + ); + }); + + it('throws on a malformed payload', async () => { + const { fetchImpl } = fakeFetch(okEnvelope({ has_more: false })); + await expect(fetchSearchPage({ baseUrl: 'http://h:1', query: 'x', fetchImpl })).rejects.toThrow( + /unexpected response shape/, + ); + }); +}); diff --git a/apps/kimi-inspect/src/search/api.ts b/apps/kimi-inspect/src/search/api.ts new file mode 100644 index 0000000000..f01651b69c --- /dev/null +++ b/apps/kimi-inspect/src/search/api.ts @@ -0,0 +1,164 @@ +/** + * REST client for the global message search endpoint: + * `POST {baseUrl}/api/v1/search`. + * + * The endpoint is a cross-session full-text search over user messages, + * assistant text and session titles. The wire shape is validated locally by + * hand (this app does not depend on zod): a malformed envelope or payload + * throws, individual hits with an unexpected shape are dropped rather than + * failing the whole page. + */ + +export interface SearchHit { + readonly sessionId: string; + readonly workspaceId: string; + readonly sessionTitle: string; + /** 'main' or a subagent id; '' for title hits (they belong to the session). */ + readonly agentId: string; + readonly role: 'user' | 'assistant' | 'title'; + readonly snippet: string; + /** Epoch ms. */ + readonly time: number; + /** 0-based turn ordinal (`t` in the transcript); absent for title hits. */ + readonly turn?: number | undefined; + /** Transcript step id (`t.`); assistant hits on step-aware servers. */ + readonly stepId?: string | undefined; + readonly score: number; +} + +export interface SearchIndexState { + readonly state: 'building' | 'ready' | 'readonly'; + readonly indexedSessions: number; + readonly totalSessions: number; + readonly documents: number; +} + +export interface SearchPage { + readonly items: readonly SearchHit[]; + readonly hasMore: boolean; + readonly pageToken?: string | undefined; + /** + * Set when the server truncated the literal-mode candidate set before + * confirmation — the page may be missing real hits. + */ + readonly incomplete?: 'candidate_cap' | undefined; + /** + * Which backend served the search: 'live' scanned the in-memory session + * transcript, 'index' queried the persisted search index. Absent when the + * server omits it or reports an unknown value. + */ + readonly source?: 'live' | 'index' | undefined; + readonly indexState: SearchIndexState; +} + +export interface FetchSearchPageOptions { + readonly baseUrl: string; + readonly token?: string | undefined; + readonly query: string; + /** Restrict to one document role; omitted searches all. */ + readonly role?: 'user' | 'assistant' | 'title' | undefined; + /** Default server-side 'score'. */ + readonly sort?: 'score' | 'time_desc' | 'time_asc' | undefined; + /** + * Match mode; default server-side 'terms'. 'literal' is a substring match + * (case-insensitive, NFKC-normalized); the server ignores `sort` and orders + * by time desc. + */ + readonly mode?: 'terms' | 'literal' | undefined; + /** + * Scope the search to one session (and optionally one agent). Wire: + * `container: { session_id, agent_id? }`. + */ + readonly container?: + | { readonly sessionId: string; readonly agentId?: string | undefined } + | undefined; + readonly pageSize?: number | undefined; + /** Opaque cursor from the previous page; the query conditions must not change. */ + readonly pageToken?: string | undefined; + /** Injectable for tests. */ + readonly fetchImpl?: typeof fetch; +} + +const ROLES = new Set(['user', 'assistant', 'title']); + +function parseHit(value: unknown): SearchHit | undefined { + if (value === null || typeof value !== 'object' || Array.isArray(value)) return undefined; + const h = value as Record; + if ( + typeof h['session_id'] !== 'string' || + typeof h['workspace_id'] !== 'string' || + typeof h['session_title'] !== 'string' || + typeof h['agent_id'] !== 'string' || + typeof h['role'] !== 'string' || + !ROLES.has(h['role']) || + typeof h['snippet'] !== 'string' || + typeof h['time'] !== 'number' || + typeof h['score'] !== 'number' + ) { + return undefined; + } + return { + sessionId: h['session_id'], + workspaceId: h['workspace_id'], + sessionTitle: h['session_title'], + agentId: h['agent_id'], + role: h['role'] as SearchHit['role'], + snippet: h['snippet'], + time: h['time'], + turn: typeof h['turn'] === 'number' ? h['turn'] : undefined, + stepId: typeof h['step_id'] === 'string' ? h['step_id'] : undefined, + score: h['score'], + }; +} + +export async function fetchSearchPage(opts: FetchSearchPageOptions): Promise { + const headers: Record = { 'content-type': 'application/json' }; + if (opts.token !== undefined && opts.token !== '') { + headers['authorization'] = `Bearer ${opts.token}`; + } + const doFetch = opts.fetchImpl ?? fetch; + const res = await doFetch(`${opts.baseUrl}/api/v1/search`, { + method: 'POST', + headers, + body: JSON.stringify({ + query: opts.query, + role: opts.role, + sort: opts.sort, + mode: opts.mode, + container: + opts.container === undefined + ? undefined + : { session_id: opts.container.sessionId, agent_id: opts.container.agentId }, + page_size: opts.pageSize, + page_token: opts.pageToken, + }), + }); + const envelope = (await res.json()) as { code: number; msg: string; data: unknown }; + if (envelope.code !== 0) { + throw new Error(`search failed (${envelope.code}): ${envelope.msg}`); + } + const data = envelope.data as Record | null; + if (data === null || typeof data !== 'object' || !Array.isArray(data['items'])) { + throw new Error('search: unexpected response shape'); + } + const rawState = (data['index_state'] ?? {}) as Record; + const items = (data['items'] as unknown[]) + .map(parseHit) + .filter((hit): hit is SearchHit => hit !== undefined); + return { + items, + hasMore: data['has_more'] === true, + pageToken: typeof data['page_token'] === 'string' ? data['page_token'] : undefined, + incomplete: data['incomplete'] === 'candidate_cap' ? 'candidate_cap' : undefined, + source: data['source'] === 'live' || data['source'] === 'index' ? data['source'] : undefined, + indexState: { + state: + rawState['state'] === 'building' || rawState['state'] === 'readonly' + ? rawState['state'] + : 'ready', + indexedSessions: Number(rawState['indexed_sessions'] ?? 0), + totalSessions: Number(rawState['total_sessions'] ?? 0), + documents: Number(rawState['documents'] ?? 0), + }, + }; +} diff --git a/apps/kimi-inspect/src/transcript/transcript.test.ts b/apps/kimi-inspect/src/transcript/transcript.test.ts index 99a0a1b7fe..8a7bab27bd 100644 --- a/apps/kimi-inspect/src/transcript/transcript.test.ts +++ b/apps/kimi-inspect/src/transcript/transcript.test.ts @@ -4,8 +4,6 @@ * test suite and are intentionally not re-tested here. */ -import { describe, expect, it, vi } from 'vitest'; - import { itemId, type StepHeader, @@ -14,6 +12,7 @@ import { type TurnHeader, type TurnState, } from '@moonshot-ai/transcript'; +import { describe, expect, it, vi } from 'vitest'; import type { WsLike } from '../channel/wsLike'; import { @@ -52,7 +51,13 @@ const textFrameUpsert = (turnId: string, stepId: string, frameId: string, text: frame: { kind: 'text' as const, frameId, role: 'assistant' as const, text }, }); -const frameAppend = (turnId: string, stepId: string, frameId: string, offset: number, text: string) => ({ +const frameAppend = ( + turnId: string, + stepId: string, + frameId: string, + offset: number, + text: string, +) => ({ op: 'append' as const, target: { type: 'frame' as const, turnId, stepId, frameId }, offset, @@ -132,7 +137,12 @@ class FakeWs implements WsLike { function makeWs(handlers: Partial[0]['handlers']> = {}) { const seen = { - ops: [] as { agentId: string; ops: readonly TranscriptOperation[]; at?: string; seq?: number }[], + ops: [] as { + agentId: string; + ops: readonly TranscriptOperation[]; + at?: string; + seq?: number; + }[], resets: [] as { agentId: string; hasMoreOlder: boolean; at?: string; seq?: number }[], resyncs: 0, reconnects: 0, @@ -172,7 +182,9 @@ describe('fetchTranscriptPage', () => { agent_id: 'main', items: [turnItem(1)], has_more: true, - tasks: [{ taskId: 'bash-1', kind: 'shell', state: 'running', detached: false, outputTail: 'x' }], + tasks: [ + { taskId: 'bash-1', kind: 'shell', state: 'running', detached: false, outputTail: 'x' }, + ], interactions: [], attachments: [], todos: [], @@ -597,7 +609,9 @@ describe('TranscriptChatStore', () => { ...emptyPage, items: [turnItem(1), turnItem(2)], hasMoreOlder: true, - tasks: [{ taskId: 'bash-1', kind: 'shell', state: 'running', detached: false, outputTail: '' }], + tasks: [ + { taskId: 'bash-1', kind: 'shell', state: 'running', detached: false, outputTail: '' }, + ], meta: { activity: 'idle' }, pendingInteractions: ['apr-1'], }, @@ -613,8 +627,16 @@ describe('TranscriptChatStore', () => { it('prepends older pages ahead of the window, dedupes, keeps live globals', () => { const store = new TranscriptChatStore(); - store.applyPage({ ...emptyPage, items: [turnItem(3)], hasMoreOlder: true, meta: { activity: 'idle' } }, { replace: true }); - store.applyPage({ ...emptyPage, items: [turnItem(1), turnItem(2)], hasMoreOlder: true, meta: {} }); + store.applyPage( + { ...emptyPage, items: [turnItem(3)], hasMoreOlder: true, meta: { activity: 'idle' } }, + { replace: true }, + ); + store.applyPage({ + ...emptyPage, + items: [turnItem(1), turnItem(2)], + hasMoreOlder: true, + meta: {}, + }); expect(store.getState().items.map((item) => itemId(item))).toEqual(['t1', 't2', 't3']); expect(store.getState().hasMoreOlder).toBe(true); // Globals from the older page do not clobber the fresher live state. diff --git a/apps/kimi-inspect/src/transcript/ws.ts b/apps/kimi-inspect/src/transcript/ws.ts index cdb3fa785d..8e5d160e73 100644 --- a/apps/kimi-inspect/src/transcript/ws.ts +++ b/apps/kimi-inspect/src/transcript/ws.ts @@ -47,11 +47,7 @@ export interface TranscriptFrameMeta { export interface TranscriptWsHandlers { /** Incremental L2 op batch for the agent (the only data frame consumed). */ - onOps: ( - agentId: string, - ops: readonly TranscriptOperation[], - meta?: TranscriptFrameMeta, - ) => void; + onOps: (agentId: string, ops: readonly TranscriptOperation[], meta?: TranscriptFrameMeta) => void; /** * Baseline snapshot frame. The chat consumer deliberately ignores these * (full state is REST-sourced) — the handler exists for observers such as @@ -213,11 +209,7 @@ export class TranscriptWs { // The subscribe_v2 ack: the server has attached the transcript stream // by now — reconcile once per socket (ops emitted between the REST // page load and this point are missed; the consumer refreshes). - if ( - !this.subscribeV2Acked && - frame.id !== undefined && - frame.id === this.subscribeV2Id - ) { + if (!this.subscribeV2Acked && frame.id !== undefined && frame.id === this.subscribeV2Id) { this.subscribeV2Acked = true; this.handlers.onReconnected(); } diff --git a/apps/kimi-inspect/src/ui.tsx b/apps/kimi-inspect/src/ui.tsx index ae2169839d..d132795472 100644 --- a/apps/kimi-inspect/src/ui.tsx +++ b/apps/kimi-inspect/src/ui.tsx @@ -41,7 +41,9 @@ export function Badge({ violet: 'bg-violet-900/60 text-violet-300', }; return ( - {children} + + {children} + ); } @@ -199,9 +201,7 @@ function TreeNode({ // undefined — skip them entirely instead of rendering source-less noise. const entries = isArray ? value.map((item, index) => [String(index), item] as const) - : Object.entries(value as Record).filter( - ([, item]) => item !== undefined, - ); + : Object.entries(value as Record).filter(([, item]) => item !== undefined); const [openBrace, closeBrace] = isArray ? ['[', ']'] : ['{', '}']; return (
@@ -300,5 +300,7 @@ function LeafValue({ value, className }: { readonly value: unknown; readonly cla if (typeof value === 'boolean') { return {String(value)}; } - return {JSON.stringify(value) ?? 'unknown'}; + return ( + {JSON.stringify(value) ?? 'unknown'} + ); } diff --git a/packages/kap-server/package.json b/packages/kap-server/package.json index e3b4e8e062..dcef4e7557 100644 --- a/packages/kap-server/package.json +++ b/packages/kap-server/package.json @@ -25,6 +25,7 @@ "@fastify/swagger": "^9.7.0", "@moonshot-ai/agent-core-v2": "workspace:^", "@moonshot-ai/kimi-code-oauth": "workspace:^", + "@moonshot-ai/minidb": "workspace:*", "@moonshot-ai/transcript": "workspace:^", "bcryptjs": "^2.4.3", "fastify": "^5.1.0", diff --git a/packages/kap-server/src/protocol/rest-search.ts b/packages/kap-server/src/protocol/rest-search.ts new file mode 100644 index 0000000000..4424d7f71a --- /dev/null +++ b/packages/kap-server/src/protocol/rest-search.ts @@ -0,0 +1,56 @@ +/** + * POST /v1/search + * + * Wire shapes for the global message search endpoint. The wire uses + * snake_case (REST convention in this repo); `routes/search.ts` maps to the + * camelCase service contract in `src/search/contract.ts`. + */ + +import { z } from 'zod'; + +export const searchMessagesBodySchema = z.object({ + query: z.string().min(1), + mode: z.enum(['terms', 'literal']).optional(), + op: z.enum(['AND', 'OR']).optional(), + container: z + .object({ + session_id: z.string().min(1).optional(), + agent_id: z.string().min(1).optional(), + }) + .optional(), + role: z.enum(['user', 'assistant', 'title']).optional(), + start_time: z.number().int().nonnegative().optional(), + end_time: z.number().int().nonnegative().optional(), + sort: z.enum(['score', 'time_desc', 'time_asc']).optional(), + page_size: z.number().int().min(1).max(50).optional(), + page_token: z.string().min(1).optional(), +}); +export type SearchMessagesBody = z.infer; + +export const searchMessageHitSchema = z.object({ + session_id: z.string(), + workspace_id: z.string(), + session_title: z.string(), + agent_id: z.string(), + role: z.enum(['user', 'assistant', 'title']), + snippet: z.string(), + time: z.number(), + turn: z.number().int().nonnegative().optional(), + step_id: z.string().optional(), + score: z.number(), +}); + +export const searchMessagesResponseSchema = z.object({ + items: z.array(searchMessageHitSchema), + has_more: z.boolean(), + page_token: z.string().optional(), + incomplete: z.enum(['candidate_cap']).optional(), + index_state: z.object({ + state: z.enum(['building', 'ready', 'readonly']), + indexed_sessions: z.number(), + total_sessions: z.number(), + documents: z.number(), + }), + source: z.enum(['live', 'index']), +}); +export type SearchMessagesResponse = z.infer; diff --git a/packages/kap-server/src/routes/registerApiV1Routes.ts b/packages/kap-server/src/routes/registerApiV1Routes.ts index 9a4d88a597..307578094c 100644 --- a/packages/kap-server/src/routes/registerApiV1Routes.ts +++ b/packages/kap-server/src/routes/registerApiV1Routes.ts @@ -32,6 +32,7 @@ import { registerModelCatalogRoutes } from './modelCatalog'; import { registerOAuthRoutes } from './oauth'; import { registerPromptsRoutes } from './prompts'; import { registerQuestionsRoutes } from './questions'; +import { registerSearchRoutes } from './search'; import { registerSessionExportRoute } from './sessionExport'; import { registerSessionsRoutes } from './sessions'; import { registerShutdownRoutes } from './shutdown'; @@ -121,6 +122,7 @@ export async function registerApiV1Routes( apiV1 as unknown as Parameters[0], core, ); + registerSearchRoutes(apiV1 as unknown as Parameters[0], core); registerTasksRoutes(apiV1 as unknown as Parameters[0], core); registerApprovalsRoutes( apiV1 as unknown as Parameters[0], diff --git a/packages/kap-server/src/routes/search.ts b/packages/kap-server/src/routes/search.ts new file mode 100644 index 0000000000..3942a2f696 --- /dev/null +++ b/packages/kap-server/src/routes/search.ts @@ -0,0 +1,125 @@ +/** + * `/search` route handler — global cross-session message search. + * + * Thin adapter over the App-scoped `IGlobalSearchService` (see + * `src/search/searchService.ts`): maps the snake_case wire body to the + * service contract, projects the result back, and maps the service's + * parameter errors to `40001`. + * + * POST /search body: SearchMessagesBody data: SearchMessagesResponse + */ + +import { type Scope } from '@moonshot-ai/agent-core-v2'; +import { z } from 'zod'; + +import { errEnvelope, okEnvelope } from '../envelope'; +import { requestLog } from '../lib/requestLog'; +import { defineRoute } from '../middleware/defineRoute'; +import { ErrorCode } from '../protocol/error-codes'; +import { + searchMessagesBodySchema, + searchMessagesResponseSchema, + type SearchMessagesBody, + type SearchMessagesResponse, +} from '../protocol/rest-search'; +import type { GlobalSearchPage, GlobalSearchQuery } from '../search/contract'; +import { GlobalSearchError, IGlobalSearchService } from '../search/searchService'; + +interface SearchRouteHost { + post( + path: string, + options: { preHandler: unknown[]; schema?: Record }, + handler: ( + req: { id: string; body: unknown }, + reply: { send(payload: unknown): unknown }, + ) => Promise | void, + ): unknown; +} + +const detailsSchema = z.array(z.object({ path: z.string(), message: z.string() })); + +function toServiceQuery(body: SearchMessagesBody): GlobalSearchQuery { + return { + query: body.query, + mode: body.mode, + op: body.op, + container: + body.container === undefined + ? undefined + : { sessionId: body.container.session_id, agentId: body.container.agent_id }, + role: body.role, + startTime: body.start_time, + endTime: body.end_time, + sort: body.sort, + pageSize: body.page_size, + pageToken: body.page_token, + }; +} + +function toWirePage(page: GlobalSearchPage): SearchMessagesResponse { + return { + items: page.items.map((hit) => ({ + session_id: hit.sessionId, + workspace_id: hit.workspaceId, + session_title: hit.sessionTitle, + agent_id: hit.agentId, + role: hit.role, + snippet: hit.snippet, + time: hit.time, + turn: hit.turn, + step_id: hit.stepId, + score: hit.score, + })), + has_more: page.hasMore, + page_token: page.pageToken, + incomplete: page.incomplete, + index_state: { + state: page.indexState.state, + indexed_sessions: page.indexState.indexedSessions, + total_sessions: page.indexState.totalSessions, + documents: page.indexState.documents, + }, + source: page.source, + }; +} + +export function registerSearchRoutes(app: SearchRouteHost, core: Scope): void { + const route = defineRoute( + { + method: 'POST', + path: '/search', + body: searchMessagesBodySchema, + success: { data: searchMessagesResponseSchema }, + errors: { + [ErrorCode.VALIDATION_FAILED]: { detailsSchema }, + }, + description: + 'Global full-text search over user messages, assistant replies and session titles across all sessions', + tags: ['search'], + }, + async (req, reply) => { + try { + const page = await core.accessor.get(IGlobalSearchService).search(toServiceQuery(req.body)); + reply.send(okEnvelope(toWirePage(page), req.id)); + } catch (error) { + if ( + error instanceof GlobalSearchError && + (error.reason === 'invalid_query' || error.reason === 'invalid_page_token') + ) { + reply.send(errEnvelope(ErrorCode.VALIDATION_FAILED, error.message, req.id, error.stack)); + return; + } + requestLog(req)?.error({ err: error }, 'global search request failed'); + reply.send( + errEnvelope( + ErrorCode.INTERNAL_ERROR, + error instanceof Error ? error.message : String(error), + req.id, + error instanceof Error ? error.stack : undefined, + ), + ); + } + }, + ); + app.post(route.path, route.options, route.handler as Parameters[2]); +} diff --git a/packages/kap-server/src/search/contract.ts b/packages/kap-server/src/search/contract.ts new file mode 100644 index 0000000000..7d7dc1693c --- /dev/null +++ b/packages/kap-server/src/search/contract.ts @@ -0,0 +1,129 @@ +/** + * `search` module — global message search contract (temporary feature, lives + * in kap-server until it graduates into agent-core-v2). + * + * The API shape borrows from Lark/Feishu's IM message endpoints: + * - a `container` concept (`container_id_type` + `container_id`) — here a + * message hangs under a session (and optionally one agent inside it); + * omitting `container` searches globally; + * - opaque cursor pagination (`pageSize` + `pageToken` + `hasMore`) where + * the query conditions may NOT change mid-pagination — the token encodes + * a fingerprint of the conditions and a mismatch is a parameter error; + * - POST + JSON body for search (a query operation, not a resource fetch); + * - an explicit sort enum (`sort_type`) and a time-range filter. + * + * This file is the single source of truth for the request/response shapes, + * shared by the Service interface (`searchService.ts`) and the REST zod + * schemas (`protocol/rest-search.ts`). + */ + +// ---- request --------------------------------------------------------------- + +export interface GlobalSearchQuery { + /** Keyword(s), required. */ + readonly query: string; + /** + * 'terms' (default) — the word-level full-text index; 'literal' — exact + * substring match over the n-gram index (case-insensitive, NFKC-folded; + * needs at least 2 normalized characters). `op`/`sort` only apply to + * 'terms'; literal hits carry score 0 and sort by time desc. + */ + readonly mode?: 'terms' | 'literal'; + /** Term combination, default AND. */ + readonly op?: 'AND' | 'OR'; + /** Omit to search across every session. */ + readonly container?: { + readonly sessionId?: string; + readonly agentId?: string; + }; + /** Restrict to one document role. */ + readonly role?: 'user' | 'assistant' | 'title'; + /** Epoch ms, inclusive bounds. */ + readonly startTime?: number; + readonly endTime?: number; + /** Default 'score' (relevance). */ + readonly sort?: 'score' | 'time_desc' | 'time_asc'; + /** Default 20, max 50. */ + readonly pageSize?: number; + /** Opaque cursor from the previous page; omit for the first page. */ + readonly pageToken?: string; +} + +// ---- response -------------------------------------------------------------- + +export interface GlobalSearchHit { + readonly sessionId: string; + readonly workspaceId: string; + readonly sessionTitle: string; + /** 'main' or a subagent id. */ + readonly agentId: string; + readonly role: 'user' | 'assistant' | 'title'; + /** ~80-char window around the first hit term, generated server-side. */ + readonly snippet: string; + /** Epoch ms of the wire record (session `updatedAt` for title docs). */ + readonly time: number; + /** + * 0-based turn ordinal in the transcript view (same numbering as the + * `before_turn` pagination cursor of `GET /sessions/{id}/transcript` — the + * turn lives at `t`). The numbering is monotonic over the wire + * journal: compaction / clear do not renumber. Absent for title hits and + * for docs indexed before turn tracking; docs whose turns were later cut by + * an undo keep their pre-undo ordinal (no longer jumpable). + */ + readonly turn?: number; + /** + * Transcript step id (`t.`, e.g. `t3.2`) of the step that + * produced this assistant text — the same id space as the transcript model + * (`packages/transcript` `model/ids.ts`), so a client can jump straight to + * the step. The ordinal is the engine's live step numbering (the wire + * record's `step` field); vacuous steps own no document, so ordinals may + * have gaps. Present only for assistant-role hits indexed after step + * tracking existed; docs whose turns were later cut by an undo keep their + * pre-undo id (no longer jumpable — same deviation as `turn`). + */ + readonly stepId?: string; + readonly score: number; +} + +export interface GlobalSearchIndexState { + /** + * building — the first full sync has not finished yet, results may be + * incomplete; ready — a full sync completed in this process; + * readonly — another process holds the index write lock, this process only + * reads (incrementally catching up from the WAL before each search). + */ + readonly state: 'building' | 'ready' | 'readonly'; + /** Progress counters behind `state`. */ + readonly indexedSessions: number; + readonly totalSessions: number; + readonly documents: number; +} + +/** + * Which backend served the page: + * - 'index' — the minidb full-text index (the default; always used when the + * container session is not live in this process); + * - 'live' — an in-memory scan of the live session's `TranscriptStore` + * (container-scoped queries on a session resumed in this process). + * Scores are only comparable within one source. + */ +export type GlobalSearchSource = 'live' | 'index'; + +export interface GlobalSearchPage { + readonly items: GlobalSearchHit[]; + readonly hasMore: boolean; + /** Present iff `hasMore`. */ + readonly pageToken?: string; + /** + * 'candidate_cap' — the literal-mode candidate set exceeded the cap, so + * confirmation was truncated and the page may miss real hits. + */ + readonly incomplete?: 'candidate_cap'; + readonly indexState: GlobalSearchIndexState; + /** + * The route that produced this page. The page token's fingerprint covers + * it: a route flip mid-pagination (e.g. the session closed) invalidates + * the token and the client must restart the search. + */ + readonly source: GlobalSearchSource; +} diff --git a/packages/kap-server/src/search/searchService.ts b/packages/kap-server/src/search/searchService.ts new file mode 100644 index 0000000000..64f5c91781 --- /dev/null +++ b/packages/kap-server/src/search/searchService.ts @@ -0,0 +1,1447 @@ +/** + * `search` module — `IGlobalSearchService` implementation (temporary feature, + * lives in kap-server until it graduates into agent-core-v2). + * + * Cross-session full-text search over user messages, assistant text and + * session titles, backed by a single minidb database at + * `/search-index`. + * + * Concurrency model — "the lock is the election": + * - `MiniDb.open({ onLockFail: 'readonly' })`: the process that grabs the + * exclusive write lock becomes the indexer (build + incremental sync); + * every other process opens read-only and never rescans wire files. + * - A read-only instance refreshes before each search via a cheap file + * fingerprint (db.wal / db.snapshot / db.textindexes.json): unchanged → + * serve the in-memory view; WAL pure-append on the same inode → + * `MiniDb.catchUpFromWal` incremental replay; anything else → close + + * full reopen. When the indexer dies, the next opener takes the lock and + * becomes the new indexer. + * - In-process, syncs are serialized behind a single-flight promise. + * + * Incremental indexing anchors on wire.jsonl byte offsets (the files are + * append-only JSONL): a `\0meta\file\` key per wire file records how + * far it has been indexed; growth re-reads only the new byte range, shrinkage + * drops the file's docs and rescans. Session title docs (`/$title`) are + * overwritten each sync; disappeared sessions are dropped by key prefix. + * + * Registration: this module is side-effect-imported by `start.ts` BEFORE + * `bootstrap()` runs, so the module-level `registerScopedService` below lands + * in the DI registry in time and the service is instantiated (App scope, + * OnScopeCreated) with the rest — which also exposes it on the `/api/v1/debug` + * reflection surface as `globalSearch` with zero extra code. + */ + +import { createHash } from 'node:crypto'; +import { open, readdir, rm, stat } from 'node:fs/promises'; +import { join, relative } from 'node:path'; + +import { + createDecorator, + IBootstrapService, + ILogService, + ISessionIndex, + LifecycleScope, + ScopeActivation, + registerScopedService, + type SessionSummary, +} from '@moonshot-ai/agent-core-v2'; +import { LockError, MiniDb, normalizeLiteral, tokenize, type BatchInputOp } from '@moonshot-ai/minidb'; +import type { TranscriptStore } from '@moonshot-ai/transcript'; + +import type { + GlobalSearchHit, + GlobalSearchIndexState, + GlobalSearchPage, + GlobalSearchQuery, + GlobalSearchSource, +} from './contract'; +import { makeSnippet } from './snippet'; +import { analyzeWireLine, type StepEffect, type TurnEffect } from './wireExtract'; + +// --------------------------------------------------------------------------- +// Constants & stored document shapes +// --------------------------------------------------------------------------- + +const INDEX_DIR_NAME = 'search-index'; +const TEXT_INDEX_NAME = 'body'; +/** n-gram substring index backing literal mode, alongside 'body'. */ +const TRI_INDEX_NAME = 'tri'; +const WIRE_FILENAME = 'wire.jsonl'; + +/** Key namespaces inside the single db. */ +const FILE_META_PREFIX = '\0meta\\file\\'; +const SESSION_META_PREFIX = '\0meta\\session\\'; +const STATS_KEY = '\0meta\\stats'; + +/** + * minidb keys are limited to 128 bytes, far shorter than an absolute wire + * path — the file meta key is a hash of the path (the path itself, and the + * owning session, live in the value). + */ +function fileMetaKey(filePath: string): string { + return FILE_META_PREFIX + createHash('sha256').update(filePath).digest('hex').slice(0, 32); +} + +/** Cap one indexed document's text so huge pastes do not bloat the index. */ +const MAX_DOC_TEXT_CHARS = 20_000; +/** Upper bound for text-index candidates handed to the scoring map / query. */ +const MAX_TEXT_HITS = 100_000; +/** + * Upper bound for literal-mode n-gram candidates handed to the confirmation + * pass (a store `get` plus a substring scan each — pure CPU). Beyond the cap + * the page is truncated and flagged `incomplete: 'candidate_cap'`. + */ +const LITERAL_CANDIDATE_CAP = 10_000; +/** Sessions are listed in pages of this size. */ +const SESSION_PAGE_SIZE = 500; + +interface MessageDoc { + readonly kind: 'message'; + readonly sessionId: string; + readonly workspaceId: string; + readonly sessionTitle: string; + readonly agentId: string; + readonly role: 'user' | 'assistant'; + readonly text: string; + readonly time: number; + /** + * 0-based turn ordinal in the transcript view (groupTurns numbering). Absent + * for docs indexed before turn tracking existed. + */ + readonly turn?: number; + /** + * Transcript step id (`t.`, engine live numbering from the wire + * record's `step` field) of the step that produced this assistant text. + * Absent for user docs and docs indexed before step tracking existed. + */ + readonly stepId?: string; +} + +interface TitleDoc { + readonly kind: 'title'; + readonly sessionId: string; + readonly workspaceId: string; + readonly sessionTitle: string; + /** Titles belong to the session, not an agent — always ''. */ + readonly agentId: ''; + readonly role: 'title'; + readonly text: string; + readonly time: number; +} + +interface FileMetaDoc { + readonly kind: 'fileMeta'; + /** Owning session, used to drop metas when a session disappears. */ + readonly sessionId: string; + /** Doc-key coordinates of this file's documents (see `docKeyPrefix`). */ + readonly agentId: string; + readonly source: 'root' | 'agents'; + /** Absolute wire path (debugging aid; the key is its hash). */ + readonly path: string; + /** Byte offset up to which the wire file has been indexed. */ + readonly offset: number; + readonly size: number; + /** + * Turn counter state at `offset` — persisted with the watermark so an + * incremental pass resumes counting instead of restarting at turn 0. + * Absent in metas written before turn tracking; treated as the initial + * state, which makes a legacy meta resume mid-file with a zeroed counter — + * an accepted one-time drift, self-healing on the next shrink/rescan. + */ + readonly turnState?: TurnCounterState; + /** + * Step tracker state at `offset` — persisted with the watermark for the + * same resume reason as `turnState`. Absent in metas written before step + * tracking: such a file is RESCANNED from scratch (docs dropped, offset + * reset) so stepIds are all-or-nothing per file instead of drifting. + */ + readonly stepState?: StepTrackerState; +} + +interface SessionMetaDoc { + readonly kind: 'sessionMeta'; +} + +// --------------------------------------------------------------------------- +// Turn counter (transcript groupTurns numbering, replayed over the wire file) +// --------------------------------------------------------------------------- + +interface TurnOpener { + readonly turn: number; + readonly anchor: boolean; +} + +interface TurnCounterState { + /** Ordinal the next opened turn will get (0-based). */ + readonly next: number; + /** Whether a turn is currently open (groupTurns' `ensureTurn` gate). */ + readonly hasTurn: boolean; + /** Turn openers, in order — the replay stack for `context.undo`. */ + readonly openers: readonly TurnOpener[]; +} + +const INITIAL_TURN_STATE: TurnCounterState = { next: 0, hasTurn: false, openers: [] }; + +function initialTurnState(): TurnCounterState { + return INITIAL_TURN_STATE; +} + +/** + * Replay `context.undo {count}`: drop the last `count` anchor-opened turns. + * The counter rewinds to the ordinal of the earliest dropped anchor, and the + * opener stack is truncated there. An undo with fewer anchors than `count` + * never reaches the wire (the engine's precheck rejects it) — left untouched. + */ +function applyUndoToTurnState(state: TurnCounterState, count: number): TurnCounterState { + let found = 0; + for (let i = state.openers.length - 1; i >= 0; i--) { + if (state.openers[i]!.anchor) { + found++; + if (found === count) { + return { + next: state.openers[i]!.turn, + hasTurn: i > 0, + openers: state.openers.slice(0, i), + }; + } + } + } + return state; +} + +/** + * Advance the counter with one record's turn effect. Returns the ordinal that + * documents extracted from the SAME record belong to: a user opener carries + * the turn it opens; assistant content carries the current turn (after the + * `ensure` gate). Undefined when the record owns no turn. + * + * The counter is monotonic except for `undo` rewinds: `apply_compaction` and + * `clear` do NOT renumber (the transcript's cold replay keeps full history + * and groupTurns numbers it continuously; the live TurnModel is monotonic + * too), so they are `none` effects by construction. Docs indexed BEFORE an + * `undo` keep their pre-undo ordinals — those messages no longer exist in the + * transcript view, so their ordinals point nowhere (known, accepted + * deviation, same class as "folded-away messages stay searchable"). + */ +function advanceTurnCounter( + state: TurnCounterState, + effect: TurnEffect, +): { docTurn: number | undefined; state: TurnCounterState } { + switch (effect.kind) { + case 'open': + return { + docTurn: state.next, + state: { + next: state.next + 1, + hasTurn: true, + openers: [...state.openers, { turn: state.next, anchor: effect.anchor }], + }, + }; + case 'ensure': { + const next = state.hasTurn ? state : { ...state, next: state.next + 1, hasTurn: true }; + return { docTurn: next.next - 1, state: next }; + } + case 'undo': + return { docTurn: undefined, state: applyUndoToTurnState(state, effect.count) }; + case 'none': + return { docTurn: undefined, state }; + } +} + +// --------------------------------------------------------------------------- +// Step tracker (transcript step ids `t.`, per-turn uuid → ordinal) +// --------------------------------------------------------------------------- + +interface StepTrackerState { + /** Current turn's step uuid → ordinal (the wire `step` field, else the fallback counter). */ + readonly byUuid: Record; + /** `step.begin` count within the current turn — the fallback ordinal source. */ + readonly begins: number; +} + +const INITIAL_STEP_STATE: StepTrackerState = { byUuid: {}, begins: 0 }; + +function initialStepState(): StepTrackerState { + return INITIAL_STEP_STATE; +} + +/** + * Advance the tracker with one record's step effect. `begin` maps the step's + * uuid to its ordinal: the wire record's own `step` field when present (the + * engine's live 1-based numbering — the same numbering transcript step ids + * use), otherwise the count of begins seen in this turn (v1 loops had no + * loop-level retries, so counting matches the surviving-step numbering). + * The mapping is never narrowed per step — it is reset wholesale at turn + * boundaries (`open`, a fallback-opening `ensure`, `undo`) by the caller. + */ +function advanceStepTracker(state: StepTrackerState, effect: StepEffect): StepTrackerState { + if (effect.kind !== 'begin') return state; + const begins = state.begins + 1; + const ordinal = effect.ordinal ?? begins; + if (state.byUuid[effect.uuid] === ordinal) return state; + return { byUuid: { ...state.byUuid, [effect.uuid]: ordinal }, begins }; +} + +interface StatsDoc { + readonly kind: 'stats'; + readonly sessions: number; + readonly documents: number; + readonly lastIndexedAt: number; +} + +type SearchDoc = MessageDoc | TitleDoc | FileMetaDoc | SessionMetaDoc | StatsDoc; + +/** + * Fire-and-forget close promises produced by `dispose()` (DI disposal is + * synchronous). The server shutdown path awaits these via + * `drainGlobalSearchDisposals()` before the homeDir is released, so a + * teardown `rm()` never races an in-flight minidb open/close. + */ +const pendingDisposals = new Set>(); + +export async function drainGlobalSearchDisposals(): Promise { + await Promise.all(pendingDisposals); +} + +// --------------------------------------------------------------------------- +// Service interface +// --------------------------------------------------------------------------- + +export type GlobalSearchErrorReason = + | 'invalid_query' + | 'invalid_page_token' + | 'readonly_index' + | 'index_unavailable'; + +export class GlobalSearchError extends Error { + constructor( + readonly reason: GlobalSearchErrorReason, + message: string, + ) { + super(message); + this.name = 'GlobalSearchError'; + } +} + +export interface IGlobalSearchService { + readonly _serviceBrand: undefined; + search(query: GlobalSearchQuery): Promise; + /** Full rebuild: wipe the index and rescan every wire file. */ + reindex(): Promise<{ sessions: number; documents: number }>; + status(): Promise<{ sessions: number; documents: number; lastIndexedAt: number | null }>; + /** + * Wire the live-transcript source for the in-memory search route. Called + * once from the composition root (start.ts) after `TranscriptService` is + * constructed; until then every search takes the index route. + */ + setLiveTranscriptSource(source: LiveTranscriptSource): void; +} + +export const IGlobalSearchService = createDecorator('globalSearch'); + +/** + * Live-transcript access behind the in-memory (live) search route. + * Implemented by `TranscriptService` (`src/services/transcript/`); declared + * here with only the three methods the route needs, so the search module + * does not import the transcript service's dependency stack. + */ +export interface LiveTranscriptSource { + /** Transcript store of a session live in this process; undefined when not in memory. */ + forSessionLive(sessionId: string): TranscriptStore | undefined; + /** Resolves when the session's initial history backfill has landed. */ + whenReady(sessionId: string): Promise; + /** Replay one agent's persisted history into the live store (idempotent per agent). */ + ensureAgentHistory(sessionId: string, agentId: string): Promise; +} + +// --------------------------------------------------------------------------- +// Query normalization & page tokens +// --------------------------------------------------------------------------- + +interface NormalizedQuery { + readonly query: string; + readonly mode: 'terms' | 'literal'; + /** + * Literal mode only: `normalizeLiteral(query)`, computed once here and + * reused by candidate confirmation and the snippet anchor. The n-gram + * index's query tokenizer applies the same normalization to the query + * terms, so index and comparison agree by construction. + */ + readonly literalQuery?: string; + /** + * Terms mode only: the query's deduplicated terms under minidb's default + * `tokenize` (the same tokenizer the 'body' text index applies to both + * sides). Computed once here so the live route's in-memory AND match agrees + * with the index route by construction. Empty when the query tokenizes to + * nothing (e.g. punctuation only) — both routes then match zero docs, + * mirroring `TextIndex.search`. + */ + readonly termsQuery?: readonly string[]; + readonly op: 'AND' | 'OR'; + readonly container?: { readonly sessionId?: string; readonly agentId?: string }; + readonly role?: 'user' | 'assistant' | 'title'; + readonly startTime?: number; + readonly endTime?: number; + readonly sort: 'score' | 'time_desc' | 'time_asc'; + readonly pageSize: number; +} + +function normalizeQuery(input: GlobalSearchQuery): NormalizedQuery { + const mode = input.mode ?? 'terms'; + // Literal matching is byte-exact (mod NFKC/case) — whitespace is part of + // the query, so it is never trimmed. + const query = mode === 'literal' ? input.query : input.query.trim(); + if (query.length === 0) { + throw new GlobalSearchError('invalid_query', 'query must be a non-empty string'); + } + const literalQuery = mode === 'literal' ? normalizeLiteral(query) : undefined; + // NOTE: the <2-code-point gate for literal queries lives in the INDEX route + // (`searchIndex`) — it is a constraint of the n-gram candidate index, not of + // literal matching itself. The live route (pure in-memory scan) accepts any + // non-empty literal query, down to a single code point. + const pageSize = input.pageSize ?? 20; + if (!Number.isInteger(pageSize) || pageSize < 1 || pageSize > 50) { + throw new GlobalSearchError('invalid_query', 'pageSize must be an integer between 1 and 50'); + } + return { + query, + mode, + literalQuery, + termsQuery: mode === 'terms' ? [...new Set(tokenize(query))] : undefined, + op: input.op ?? 'AND', + container: input.container, + role: input.role, + startTime: input.startTime, + endTime: input.endTime, + sort: input.sort ?? 'score', + pageSize, + }; +} + +/** + * The page token encodes a fingerprint of the query conditions plus the skip + * offset — changing conditions mid-pagination invalidates the token (same + * rule as Lark's search API). The serving route (`source`) is part of the + * fingerprint: a route flip mid-pagination (e.g. the container session + * closed and the live route fell away) invalidates the token too, so the + * client restarts the search instead of silently switching result sets. + */ +function tokenFingerprint(q: NormalizedQuery, source: GlobalSearchSource): string { + const basis = JSON.stringify([ + q.query, + q.mode, + q.op, + q.container?.sessionId, + q.container?.agentId, + q.role, + q.startTime, + q.endTime, + q.sort, + source, + ]); + return createHash('sha256').update(basis).digest('base64url').slice(0, 16); +} + +function encodePageToken(q: NormalizedQuery, source: GlobalSearchSource, skip: number): string { + return Buffer.from(JSON.stringify({ f: tokenFingerprint(q, source), s: skip })).toString( + 'base64url', + ); +} + +function decodePageToken( + q: NormalizedQuery, + source: GlobalSearchSource, + token: string | undefined, +): number { + if (token === undefined) return 0; + let parsed: unknown; + try { + parsed = JSON.parse(Buffer.from(token, 'base64url').toString('utf8')); + } catch { + throw new GlobalSearchError('invalid_page_token', 'pageToken is malformed'); + } + if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) { + throw new GlobalSearchError('invalid_page_token', 'pageToken is malformed'); + } + const p = parsed as { f?: unknown; s?: unknown }; + if (p.f !== tokenFingerprint(q, source)) { + throw new GlobalSearchError( + 'invalid_page_token', + 'pageToken does not match the query conditions; query conditions must not change mid-pagination', + ); + } + if (typeof p.s !== 'number' || !Number.isInteger(p.s) || p.s < 0) { + throw new GlobalSearchError('invalid_page_token', 'pageToken is malformed'); + } + return p.s; +} + +// --------------------------------------------------------------------------- +// Implementation +// --------------------------------------------------------------------------- + +export class GlobalSearchService implements IGlobalSearchService { + declare readonly _serviceBrand: undefined; + + /** Minimum interval between search-triggered sync passes (test knob). */ + syncDebounceMs = 2_000; + + /** Literal-mode candidate cap (test knob, see LITERAL_CANDIDATE_CAP). */ + literalCandidateCap = LITERAL_CANDIDATE_CAP; + + private db: MiniDb | null = null; + private openPromise: Promise | null = null; + private syncPromise: Promise | null = null; + private refreshPromise: Promise | null = null; + private lastSyncStartedAt = 0; + private fullSyncDone = false; + /** WAL watermark (bytes applied) for read-only catch-up. */ + private walOffset = 0; + private fingerprint = ''; + private summaries = new Map(); + private disposed = false; + /** Set while `reindex()` swaps the db — syncs started meanwhile are no-ops. */ + private reindexing = false; + /** Live-transcript source for the in-memory route; null until start.ts wires it. */ + private liveSource: LiveTranscriptSource | null = null; + + constructor( + @ISessionIndex private readonly sessionIndex: ISessionIndex, + @IBootstrapService private readonly bootstrap: IBootstrapService, + @ILogService private readonly log: ILogService, + ) { + // App-scope OnScopeCreated activation: kick the first full sync off in the + // background so server bootstrap never blocks on indexing. + this.kickBackgroundSync(); + } + + setLiveTranscriptSource(source: LiveTranscriptSource): void { + this.liveSource = source; + } + + // -- lifecycle --------------------------------------------------------------- + + private get indexDir(): string { + return join(this.bootstrap.homeDir, INDEX_DIR_NAME); + } + + private ensureOpen(): Promise { + this.openPromise ??= this.openDb().catch((error: unknown) => { + this.openPromise = null; + throw error; + }); + return this.openPromise; + } + + private async openDb(): Promise { + const db = await this.openSearchDb(); + // The scope may have been disposed while the (slow) open was in flight — + // close the handle immediately instead of leaking it and writing the + // text-index definition below into a directory the caller may already be + // deleting. + if (this.disposed) { + await db.close().catch(() => {}); + throw new GlobalSearchError('index_unavailable', 'search service is disposed'); + } + this.db = db; + this.walOffset = db.recoveryInfo?.walScanEnd ?? 0; + if (!db.readOnly) { + // Both indexes are created here (not at first write) so a pre-existing + // db gets the tri index built over its current documents on first open + // after the upgrade, and a read-only peer only ever reopens on the + // definitions-file fingerprint change. + for (const [name, options] of [ + [TEXT_INDEX_NAME, { fields: ['text'] }], + [TRI_INDEX_NAME, { fields: ['text'], tokenizer: 'ngram' }], + ] as const) { + try { + await db.createTextIndex(name, options); + } catch (error) { + if (!(error instanceof Error && error.message.includes('already exists'))) throw error; + } + } + } + this.fingerprint = await this.computeFingerprint(); + } + + /** + * Open the index db, rebuilding from scratch on unrecoverable corruption + * (the index is derived data — never repaired, only rebuilt). + * + * Rebuild is WRITER-ONLY: a process that fails to grab the write lock must + * never delete the directory out from under the live indexer. Lock state is + * not observable once `open` throws, so corruption is disambiguated with a + * probe open WITHOUT `onLockFail`: it throws `LockError` before recovery + * when another process holds the lock, and re-throws the corruption + * (releasing the lock) when the lock is free — in which case this process + * is the would-be writer and may rebuild. + */ + private async openSearchDb(): Promise> { + const opts = { + dir: this.indexDir, + valueCodec: 'json', + fsyncPolicy: 'everysec', + onLockFail: 'readonly', + } as const; + try { + return await MiniDb.open(opts); + } catch (error) { + if (!isRebuildableCorruption(error)) throw error; + let probeError: unknown; + try { + const probe = await MiniDb.open({ dir: opts.dir, valueCodec: opts.valueCodec }); + await probe.close().catch(() => {}); + probeError = undefined; // lock free AND data fine — cannot happen, but treat as rebuildable + } catch (error) { + probeError = error; + } + if (probeError instanceof LockError) { + // Another process holds the write lock: leave its files alone. The + // caller's open fails; the next search retries from scratch. + throw error; + } + await rm(this.indexDir, { recursive: true, force: true }); + return MiniDb.open(opts); + } + } + + dispose(): void { + this.disposed = true; + // DI disposal is synchronous, but closing a MiniDb is not: wait for any + // in-flight open to settle, then close the handle. The promise is + // registered module-level so the server shutdown path + // (`drainGlobalSearchDisposals` in start.ts) can await it before the + // homeDir is torn down — otherwise teardown rm() races the close and + // fails with ENOTEMPTY. + const pending = (async () => { + await this.openPromise?.catch(() => {}); + const db = this.db; + this.db = null; + if (db) await db.close().catch(() => {}); + })(); + pendingDisposals.add(pending); + void pending.finally(() => pendingDisposals.delete(pending)); + } + + // -- read-only freshness (fingerprint + WAL catch-up) ------------------------- + + private async computeFingerprint(): Promise { + const parts: string[] = []; + for (const name of ['db.wal', 'db.snapshot', 'db.textindexes.json']) { + try { + const s = await stat(join(this.indexDir, name)); + parts.push(`${name}:${s.dev}:${s.ino}:${s.mtimeMs}:${s.size}`); + } catch { + parts.push(`${name}:-`); + } + } + return parts.join('|'); + } + + /** + * Bring a read-only instance up to date with the indexer's committed + * writes. Unchanged fingerprint → zero IO; WAL pure-append → incremental + * `catchUpFromWal`; anything else → close + full reopen (which may also + * promote this process to indexer when the old writer's lock is gone). + */ + private refreshReadonly(): Promise { + this.refreshPromise ??= this.doRefreshReadonly() + .catch(() => { + // A failed refresh must not fail the search — serve the stale view. + }) + .finally(() => { + this.refreshPromise = null; + }); + return this.refreshPromise; + } + + private async doRefreshReadonly(): Promise { + const db = this.db; + if (!db || !db.readOnly || this.disposed) return; + const fp = await this.computeFingerprint(); + if (fp === this.fingerprint) return; + const [, snapPrev, defsPrev] = this.fingerprint.split('|'); + const [, snapNow, defsNow] = fp.split('|'); + if (snapPrev === snapNow && defsPrev === defsNow) { + const res = await db.catchUpFromWal(this.walOffset); + if (res !== null) { + this.walOffset = res.offset; + this.fingerprint = fp; + return; + } + } + // WAL rotated/truncated, snapshot or index definitions changed, or the + // watermark no longer aligns: close and reopen from scratch. + await db.close().catch(() => {}); + if (this.db === db) { + this.db = null; + this.openPromise = null; + await this.ensureOpen(); + } + } + + // -- sync (indexer only) -------------------------------------------------------- + + private kickBackgroundSync(): void { + void this.ensureSyncStarted().catch((error: unknown) => { + this.log.warn('global search: background sync failed', { + error: error instanceof Error ? error.message : String(error), + }); + }); + } + + /** Single-flight: concurrent callers share the in-flight sync. */ + private ensureSyncStarted(): Promise { + if (this.syncPromise === null) { + const p = this.runSync().finally(() => { + if (this.syncPromise === p) this.syncPromise = null; + }); + this.syncPromise = p; + } + return this.syncPromise; + } + + private async runSync(): Promise { + // `reindexing`: a rebuild is swapping the db out — this pass is a no-op; + // the rebuild itself runs the authoritative sync when done. + if (this.disposed || this.reindexing) return; + const sessions = await this.listAllSessions(); + // Nothing to index and no index on disk yet: don't even create the + // `/search-index` directory — it would show up in the fs folder + // picker and cost every server boot a pointless db open. + if (sessions.length === 0 && !(await pathExists(this.indexDir))) { + this.summaries = new Map(); + this.lastSyncStartedAt = Date.now(); + this.fullSyncDone = true; + return; + } + + await this.ensureOpen(); + const db = this.db; + if (!db || db.readOnly || this.disposed) return; + this.lastSyncStartedAt = Date.now(); + + this.summaries = new Map(sessions.map((s) => [s.id, s])); + const currentIds = new Set(sessions.map((s) => s.id)); + + // Drop sessions whose directory disappeared since the last sync. + for (const row of db.query({ key: { prefix: SESSION_META_PREFIX }, project: [] })) { + const sessionId = row.key.slice(SESSION_META_PREFIX.length); + if (!currentIds.has(sessionId)) await this.deleteSessionDocs(db, sessionId); + } + + let indexed = 0; + for (const summary of sessions) { + if (this.disposed) return; + try { + await this.syncSession(db, summary); + indexed++; + } catch (error) { + // One unreadable session must not abort the whole pass. + this.log.warn('global search: failed to index session', { + sessionId: summary.id, + error: error instanceof Error ? error.message : String(error), + }); + } + } + + const metaCount = db.query({ key: { prefix: '\0meta\\' }, project: [] }).length; + const stats: StatsDoc = { + kind: 'stats', + sessions: indexed, + documents: db.size - metaCount, + lastIndexedAt: Date.now(), + }; + await db.set(STATS_KEY, stats); + this.fullSyncDone = true; + } + + private async listAllSessions(): Promise { + const out: SessionSummary[] = []; + let cursor: string | undefined; + do { + const page = await this.sessionIndex.list({ cursor, limit: SESSION_PAGE_SIZE }); + out.push(...page.items); + cursor = page.nextCursor; + } while (cursor !== undefined); + return out; + } + + private async deleteSessionDocs(db: MiniDb, sessionId: string): Promise { + for (const row of db.query({ key: { prefix: `${sessionId}/` }, project: [] })) { + await db.del(row.key); + } + for (const row of db.query({ key: { prefix: FILE_META_PREFIX } })) { + if (row.value.kind === 'fileMeta' && row.value.sessionId === sessionId) { + await db.del(row.key); + } + } + await db.del(SESSION_META_PREFIX + sessionId); + } + + private async syncSession(db: MiniDb, summary: SessionSummary): Promise { + const sessionDir = this.bootstrap.sessionDir(summary.workspaceId, summary.id); + const wireFiles = await collectWireFiles(sessionDir); + const seenPaths = new Set(wireFiles.map((file) => file.path)); + + // A wire file that vanished on its own (e.g. one agent's log deleted + // while the session lives on): drop its docs and meta. Session-level + // disappearance is handled separately in runSync. + for (const row of db.query({ key: { prefix: FILE_META_PREFIX } })) { + const meta = row.value; + if (meta.kind !== 'fileMeta' || meta.sessionId !== summary.id) continue; + if (seenPaths.has(meta.path)) continue; + await this.deleteFileDocs(db, meta); + await db.del(row.key); + } + + for (const file of wireFiles) { + await this.syncWireFile(db, summary, file); + } + + const title = summary.title ?? ''; + const titleKey = `${summary.id}/$title`; + const existing = db.get(titleKey); + if (title.length > 0) { + if (existing?.kind !== 'title' || existing.text !== title) { + const doc: TitleDoc = { + kind: 'title', + sessionId: summary.id, + workspaceId: summary.workspaceId, + sessionTitle: title, + agentId: '', + role: 'title', + text: title, + time: summary.updatedAt, + }; + await db.set(titleKey, doc); + } + } else if (existing !== undefined) { + await db.del(titleKey); + } + // Session marker: presence is the information — write only when missing. + if (db.get(SESSION_META_PREFIX + summary.id) === undefined) { + const sessionMeta: SessionMetaDoc = { kind: 'sessionMeta' }; + await db.set(SESSION_META_PREFIX + summary.id, sessionMeta); + } + } + + private async deleteFileDocs(db: MiniDb, meta: FileMetaDoc): Promise { + const prefix = `${meta.sessionId}/${meta.agentId}/${meta.source}:`; + for (const row of db.query({ key: { prefix }, project: [] })) { + await db.del(row.key); + } + } + + private async syncWireFile( + db: MiniDb, + summary: SessionSummary, + file: WireFileRef, + ): Promise { + let size: number; + try { + size = (await stat(file.path)).size; + } catch { + return; // transiently unreadable — retry next pass + } + const metaKey = fileMetaKey(file.path); + const meta = db.get(metaKey); + let offset = meta?.kind === 'fileMeta' ? meta.offset : 0; + let turnState: TurnCounterState = + meta?.kind === 'fileMeta' ? (meta.turnState ?? initialTurnState()) : initialTurnState(); + let stepState: StepTrackerState = + meta?.kind === 'fileMeta' ? (meta.stepState ?? initialStepState()) : initialStepState(); + const fileMeta = ( + nextOffset: number, + turns: TurnCounterState, + steps: StepTrackerState, + ): FileMetaDoc => ({ + kind: 'fileMeta', + sessionId: summary.id, + agentId: file.agentId, + source: file.source, + path: file.path, + offset: nextOffset, + size, + turnState: turns, + stepState: steps, + }); + // Metas written before step tracking carry no `stepState`: rescan the + // file from scratch so stepIds are all-or-nothing per file rather than + // drifting mid-file (the shrink path does exactly this). + const legacyMeta = meta?.kind === 'fileMeta' && meta.stepState === undefined; + if (size < offset || legacyMeta) { + // File was rebuilt/truncated: drop its docs and rescan from scratch — + // the turn counter and step tracker restart with it. + await this.deleteFileDocs(db, fileMeta(0, initialTurnState(), initialStepState())); + offset = 0; + turnState = initialTurnState(); + stepState = initialStepState(); + } + if (size === offset) { + await db.set(metaKey, fileMeta(offset, turnState, stepState)); + return; + } + + // Read only the new byte range; consume up to the last complete line. A + // short read (the file was truncated between stat and read) just defers + // the remainder to the next pass — the watermark below never advances + // past bytes that were actually read. + const handle = await open(file.path, 'r'); + let buf: Buffer; + try { + buf = Buffer.allocUnsafe(size - offset); + const { bytesRead } = await handle.read(buf, 0, buf.length, offset); + buf = buf.subarray(0, bytesRead); + } finally { + await handle.close(); + } + const lastNl = buf.lastIndexOf(0x0a); + if (lastNl === -1) return; // no complete line yet + const complete = buf.subarray(0, lastNl + 1).toString('utf8'); + + const ops: BatchInputOp[] = []; + let byteCursor = offset; + for (const line of complete.split('\n')) { + const lineBytes = Buffer.byteLength(line, 'utf8') + 1; + const lineOffset = byteCursor; + byteCursor += lineBytes; + const analysis = analyzeWireLine(line); + // Turn counting runs independently of indexing: every line moves the + // counter (a text-less user message still opens a turn). + const advanced = advanceTurnCounter(turnState, analysis.turn); + // A turn boundary invalidates the step mapping: a new turn opens + // (`open`, or `ensure` opening a fallback turn from no-turn), or an + // `undo` rewinds the counter mid-turn. + if ( + analysis.turn.kind === 'open' || + analysis.turn.kind === 'undo' || + (analysis.turn.kind === 'ensure' && !turnState.hasTurn) + ) { + stepState = initialStepState(); + } + turnState = advanced.state; + stepState = advanceStepTracker(stepState, analysis.step); + const extracted = analysis.messages; + for (let i = 0; i < extracted.length; i++) { + const e = extracted[i]!; + const stepOrdinal = e.stepUuid !== undefined ? stepState.byUuid[e.stepUuid] : undefined; + const doc: MessageDoc = { + kind: 'message', + sessionId: summary.id, + workspaceId: summary.workspaceId, + sessionTitle: summary.title ?? '', + agentId: file.agentId, + role: e.role, + text: e.text.length > MAX_DOC_TEXT_CHARS ? e.text.slice(0, MAX_DOC_TEXT_CHARS) : e.text, + time: e.time ?? summary.updatedAt, + turn: advanced.docTurn, + // A doc whose step cannot be resolved (no `step.begin` seen, or a + // turn boundary invalidated the mapping) just omits the id. + stepId: + advanced.docTurn !== undefined && stepOrdinal !== undefined + ? `t${advanced.docTurn}.${stepOrdinal}` + : undefined, + }; + // A line can yield several docs — the per-line index keeps keys unique. + ops.push({ + op: 'set', + key: `${docKeyPrefix(summary.id, file)}${lineOffset}:${i}`, + value: doc, + }); + } + } + const newOffset = offset + Buffer.byteLength(complete, 'utf8'); + ops.push({ op: 'set', key: metaKey, value: fileMeta(newOffset, turnState, stepState) }); + await db.batch(ops); + } + + // -- public API --------------------------------------------------------------- + + /** + * Route: a container-scoped query on a session that is live in this process + * scans the in-memory transcript store instead of the index, in both terms + * and literal mode. Anything else takes the index route. The live route + * never falls back on error — the store being in hand means the session is + * alive, so a scan failure is a real error, not a degradation signal. + */ + async search(input: GlobalSearchQuery): Promise { + const q = normalizeQuery(input); + const sessionId = q.container?.sessionId; + const liveStore = sessionId !== undefined ? this.liveSource?.forSessionLive(sessionId) : undefined; + if (liveStore !== undefined && sessionId !== undefined) { + return this.searchLive(q, sessionId, liveStore, input.pageToken); + } + return this.searchIndex(q, input.pageToken); + } + + // -- live route (in-memory transcript scan) ------------------------------------ + + private async searchLive( + q: NormalizedQuery, + sessionId: string, + store: TranscriptStore, + pageToken: string | undefined, + ): Promise { + const skip = decodePageToken(q, 'live', pageToken); + const source = this.liveSource; + if (source === null) { + // Unreachable (the router only enters with a source-wired store), but a + // null deref here would mask a wiring bug — fail loudly instead. + throw new GlobalSearchError('index_unavailable', 'live transcript source is not wired'); + } + // Backfill gates: the main-agent history, then every agent in scope, so + // the scan covers full history rather than only post-resume content. + await source.whenReady(sessionId); + const agentIds = + q.container?.agentId !== undefined + ? [q.container.agentId] + : store.agents().map((agent) => agent.agentId); + for (const agentId of agentIds) { + await source.ensureAgentHistory(sessionId, agentId); + } + const docs = await this.collectLiveDocs(sessionId, store, agentIds); + // Literal mode needs no candidate index: every in-memory document is a + // candidate and the shared confirmation pass decides. Terms mode runs the + // in-memory AND match first, scoring each hit. + const matched = + q.mode === 'literal' + ? this.matchDocs( + q, + docs.map((value) => ({ value, score: 0 })), + ) + : this.matchDocs(q, matchLiveTerms(q.termsQuery ?? [], docs)); + return this.toPage(q, 'live', skip, matched, undefined, { + state: 'ready', + indexedSessions: 1, + totalSessions: 1, + documents: docs.length, + }); + } + + /** + * Flatten the live transcript store into the same document shape the index + * route searches (`MessageDoc` / `TitleDoc`): + * - one user doc per non-empty `turn.prompt` (turn ordinal + turn time); + * - one assistant doc per assistant-role text frame (turn ordinal + + * stepId); thinking / tool / notice frames are skipped; + * - one title doc from the session-index summary, same as the sync path. + * Text is trimmed and empty results skipped, mirroring the index side's + * `wireExtract` (which trims both user and assistant text). + */ + private async collectLiveDocs( + sessionId: string, + store: TranscriptStore, + agentIds: readonly string[], + ): Promise<(MessageDoc | TitleDoc)[]> { + const summary = await this.sessionIndex.get(sessionId); + const workspaceId = summary?.workspaceId ?? ''; + const sessionTitle = summary?.title ?? ''; + const fallbackTime = summary?.updatedAt ?? 0; + const parseTime = (iso: string | undefined): number => { + if (iso === undefined) return fallbackTime; + const ms = Date.parse(iso); + return Number.isNaN(ms) ? fallbackTime : ms; + }; + const docs: (MessageDoc | TitleDoc)[] = []; + for (const agentId of agentIds) { + const transcript = store.getAgent(agentId); + if (transcript === undefined) continue; + for (const item of transcript.snapshot().items) { + if (item.kind !== 'turn') continue; + const turnTime = parseTime(item.startedAt); + const prompt = item.prompt?.trim() ?? ''; + if (prompt.length > 0) { + docs.push({ + kind: 'message', + sessionId, + workspaceId, + sessionTitle, + agentId, + role: 'user', + text: prompt.length > MAX_DOC_TEXT_CHARS ? prompt.slice(0, MAX_DOC_TEXT_CHARS) : prompt, + time: turnTime, + turn: item.ordinal, + stepId: undefined, + }); + } + for (const step of item.steps) { + const stepTime = parseTime(step.endedAt ?? step.startedAt ?? item.startedAt); + for (const frame of step.frames) { + if (frame.kind !== 'text' || frame.role !== 'assistant') continue; + const text = frame.text.trim(); + if (text.length === 0) continue; + docs.push({ + kind: 'message', + sessionId, + workspaceId, + sessionTitle, + agentId, + role: 'assistant', + text: text.length > MAX_DOC_TEXT_CHARS ? text.slice(0, MAX_DOC_TEXT_CHARS) : text, + time: stepTime, + turn: item.ordinal, + stepId: step.stepId, + }); + } + } + } + } + if (sessionTitle.length > 0) { + docs.push({ + kind: 'title', + sessionId, + workspaceId, + sessionTitle, + agentId: '', + role: 'title', + text: sessionTitle, + time: fallbackTime, + }); + } + return docs; + } + + // -- index route (minidb) ------------------------------------------------------- + + private async searchIndex( + q: NormalizedQuery, + pageToken: string | undefined, + ): Promise { + const skip = decodePageToken(q, 'index', pageToken); + + await this.ensureOpen(); + if (this.db?.readOnly === true) { + await this.refreshReadonly(); + } + const db = this.db; + if (db === null) { + throw new GlobalSearchError('index_unavailable', 'search index is unavailable'); + } + + if (!db.readOnly) { + if (this.fullSyncDone) { + // Incremental catch-up before searching, debounced; the first full + // sync is never awaited (search serves whatever is indexed so far). + if (Date.now() - this.lastSyncStartedAt >= this.syncDebounceMs) { + await this.ensureSyncStarted().catch(() => {}); + } + } else { + this.kickBackgroundSync(); + } + } + + // One text-index pass: db.search returns every candidate with its score; + // container/role/time filters and the requested sort are applied in + // memory. (A separate db.query({text}) for pagination would scan the same + // postings a second time.) + let candidates: { key: string; value: SearchDoc | undefined; score: number }[]; + let incomplete: 'candidate_cap' | undefined; + try { + if (q.mode === 'literal') { + // The n-gram index cannot confirm queries shorter than 2 normalized + // code points. Judged AFTER normalization on purpose: NFKC can change + // the length (the ligature 'ff' folds to 'ff' and becomes legal). The + // live route has no such constraint — it never reaches this branch. + if (Array.from(q.literalQuery ?? '').length < 2) { + throw new GlobalSearchError( + 'invalid_query', + 'literal queries need at least 2 characters (after Unicode normalization)', + ); + } + // Ask for one past the cap so an over-cap candidate set is detectable. + candidates = db.search(TRI_INDEX_NAME, q.query, { + op: 'AND', + limit: this.literalCandidateCap + 1, + }); + if (candidates.length > this.literalCandidateCap) { + candidates.length = this.literalCandidateCap; + incomplete = 'candidate_cap'; + } + } else { + candidates = db.search(TEXT_INDEX_NAME, q.query, { op: q.op, limit: MAX_TEXT_HITS }); + } + } catch (error) { + // A read-only instance can open before the writer has created the text + // index — serve an empty page instead of failing the search. + if (error instanceof Error && error.message.includes('no such text index')) { + return { + items: [], + hasMore: false, + pageToken: undefined, + incomplete: undefined, + indexState: this.readIndexState(db), + source: 'index', + }; + } + throw error; + } + + const matched = this.matchDocs(q, candidates); + return this.toPage(q, 'index', skip, matched, incomplete, this.readIndexState(db)); + } + + // -- shared match & page assembly (both routes) -------------------------------- + + /** + * Container/role/time filtering plus literal confirmation — one + * implementation shared by the index route (confirming n-gram candidates) + * and the live route (scanning every in-memory document). + */ + private matchDocs( + q: NormalizedQuery, + docs: Iterable<{ value: SearchDoc | undefined; score: number }>, + ): { value: MessageDoc | TitleDoc; score: number; anchor?: number }[] { + const literalQuery = q.literalQuery; + const matched: { value: MessageDoc | TitleDoc; score: number; anchor?: number }[] = []; + for (const { value: doc, score } of docs) { + if (doc === undefined || (doc.kind !== 'message' && doc.kind !== 'title')) continue; + if (q.container?.sessionId !== undefined && doc.sessionId !== q.container.sessionId) continue; + if (q.container?.agentId !== undefined && doc.agentId !== q.container.agentId) continue; + if (q.role !== undefined && doc.role !== q.role) continue; + if (q.startTime !== undefined && doc.time < q.startTime) continue; + if (q.endTime !== undefined && doc.time > q.endTime) continue; + if (literalQuery !== undefined) { + // Two-phase execution (same model as Elasticsearch's wildcard field): + // candidates (from the n-gram index, or every in-memory doc on the + // live route) are confirmed against the document text — hash + // collisions and non-contiguous n-gram coverage can produce false + // positives. Zero false positives is the hard guarantee of literal + // mode. The match offset doubles as the snippet anchor. Deliberate + // deviation from ES: the comparison is case-insensitive (NFKC + + // lowercase), aligned with the terms tokenizer. + const at = normalizeLiteral(doc.text).indexOf(literalQuery); + if (at === -1) continue; + matched.push({ value: doc, score: 0, anchor: at }); + } else { + matched.push({ value: doc, score }); + } + } + return matched; + } + + /** Sort, paginate and project the matched docs into a page (both routes). */ + private toPage( + q: NormalizedQuery, + source: GlobalSearchSource, + skip: number, + matched: { value: MessageDoc | TitleDoc; score: number; anchor?: number }[], + incomplete: 'candidate_cap' | undefined, + indexState: GlobalSearchIndexState, + ): GlobalSearchPage { + // Literal mode: the normalized query (computed in normalizeQuery), reused + // by confirmation and the snippet anchor. + const literalQuery = q.literalQuery; + // Literal hits carry no relevance score and always order by time desc + // (`sort` is a terms-mode concept). 'score' keeps the relevance order the + // route produced: the text index's on the index route, `matchLiveTerms'` + // on the live route. + if (q.mode === 'literal' || q.sort === 'time_desc') { + matched.sort((a, b) => b.value.time - a.value.time); + } else if (q.sort === 'time_asc') { + matched.sort((a, b) => a.value.time - b.value.time); + } + + const pageRows = matched.slice(skip, skip + q.pageSize + 1); + const hasMore = pageRows.length > q.pageSize; + const items: GlobalSearchHit[] = pageRows.slice(0, q.pageSize).map((row) => { + const doc = row.value; + return { + sessionId: doc.sessionId, + workspaceId: doc.workspaceId, + sessionTitle: this.summaries.get(doc.sessionId)?.title ?? doc.sessionTitle, + agentId: doc.agentId, + role: doc.role, + snippet: + doc.kind === 'title' + ? doc.text + : row.anchor !== undefined && literalQuery !== undefined + ? makeSnippet(doc.text, q.query, 80, { at: row.anchor, len: literalQuery.length }) + : makeSnippet(doc.text, q.query), + time: doc.time, + turn: doc.kind === 'message' ? doc.turn : undefined, + stepId: doc.kind === 'message' ? doc.stepId : undefined, + score: row.score, + }; + }); + + return { + items, + hasMore, + pageToken: hasMore ? encodePageToken(q, source, skip + q.pageSize) : undefined, + incomplete, + indexState, + source, + }; + } + + async reindex(): Promise<{ sessions: number; documents: number }> { + await this.ensureOpen(); + if (this.db?.readOnly === true) { + throw new GlobalSearchError( + 'readonly_index', + 'another process holds the search-index write lock; reindex from that process', + ); + } + this.reindexing = true; + try { + // Let the in-flight sync settle before closing the db it writes into. + // Syncs triggered while we wait see `reindexing` and return as no-ops, + // so one await is sufficient — no new writer of the old db can appear. + await this.syncPromise?.catch(() => {}); + const db = this.db; + if (db) { + await db.close().catch(() => {}); + this.db = null; + } + this.openPromise = null; + this.fullSyncDone = false; + await rm(this.indexDir, { recursive: true, force: true }); + await this.ensureOpen(); + } finally { + this.reindexing = false; + } + await this.ensureSyncStarted(); + const stats = this.db?.get(STATS_KEY); + return { + sessions: stats?.kind === 'stats' ? stats.sessions : 0, + documents: stats?.kind === 'stats' ? stats.documents : 0, + }; + } + + async status(): Promise<{ sessions: number; documents: number; lastIndexedAt: number | null }> { + await this.ensureOpen(); + if (this.db?.readOnly === true) { + await this.refreshReadonly(); + } else { + this.kickBackgroundSync(); + } + const stats = this.db?.get(STATS_KEY); + return { + sessions: stats?.kind === 'stats' ? stats.sessions : 0, + documents: stats?.kind === 'stats' ? stats.documents : 0, + lastIndexedAt: stats?.kind === 'stats' ? stats.lastIndexedAt : null, + }; + } + + private readIndexState(db: MiniDb): GlobalSearchIndexState { + const stats = db.get(STATS_KEY); + const indexed = stats?.kind === 'stats' ? stats.sessions : 0; + const documents = stats?.kind === 'stats' ? stats.documents : 0; + return { + state: db.readOnly ? 'readonly' : this.fullSyncDone ? 'ready' : 'building', + indexedSessions: indexed, + totalSessions: db.readOnly ? indexed : Math.max(indexed, this.summaries.size), + documents, + }; + } +} + +// --------------------------------------------------------------------------- +// live-route terms matching +// --------------------------------------------------------------------------- + +/** + * Terms-mode matching for the live route. Both query (already tokenized and + * deduplicated in `normalizeQuery`) and documents are split with minidb's + * default `tokenize` — the same tokenizer the index route's 'body' text index + * uses — so a document matches when EVERY query term appears in its term set + * (AND). The score is Σ log(1 + tf) per query term: it is only comparable + * within the live route, since there is no corpus-wide IDF in memory (the + * `GlobalSearchSource` contract comment says the same). Hits are returned + * score-sorted, mirroring `TextIndex.search`, because the shared `toPage` + * keeps the candidate order for `sort: 'score'`. + */ +function matchLiveTerms( + terms: readonly string[], + docs: readonly (MessageDoc | TitleDoc)[], +): { value: MessageDoc | TitleDoc; score: number }[] { + // A query that tokenizes to nothing matches zero docs, same as the index. + if (terms.length === 0) return []; + const matched: { value: MessageDoc | TitleDoc; score: number }[] = []; + for (const doc of docs) { + const counts = new Map(); + for (const token of tokenize(doc.text)) counts.set(token, (counts.get(token) ?? 0) + 1); + let score = 0; + let hit = true; + for (const term of terms) { + const tf = counts.get(term) ?? 0; + if (tf === 0) { + hit = false; + break; + } + score += Math.log(1 + tf); + } + if (hit) matched.push({ value: doc, score }); + } + matched.sort((a, b) => b.score - a.score); + return matched; +} + +// --------------------------------------------------------------------------- +// wire file enumeration & doc keys +// --------------------------------------------------------------------------- + +interface WireFileRef { + readonly path: string; + /** 'main' or a subagent id, for both legacy and v2 layouts. */ + readonly agentId: string; + /** + * Key discriminator: a session can carry BOTH a legacy root wire.jsonl and + * v2 per-agent logs; without this their `/` keys collide. + */ + readonly source: 'root' | 'agents'; +} + +async function pathExists(path: string): Promise { + try { + await stat(path); + return true; + } catch { + return false; + } +} + +async function collectWireFiles(sessionDir: string): Promise { + const files: WireFileRef[] = []; + const root = join(sessionDir, WIRE_FILENAME); + try { + if ((await stat(root)).isFile()) files.push({ path: root, agentId: 'main', source: 'root' }); + } catch { + // no legacy root log + } + const agentsDir = join(sessionDir, 'agents'); + try { + const entries = await readdir(agentsDir, { recursive: true, withFileTypes: true }); + for (const entry of entries) { + if (!entry.isFile() || entry.name !== WIRE_FILENAME) continue; + const path = join(entry.parentPath, entry.name); + files.push({ path, agentId: relative(agentsDir, entry.parentPath), source: 'agents' }); + } + } catch { + // no agents dir + } + return files; +} + +function docKeyPrefix(sessionId: string, file: WireFileRef): string { + return `${sessionId}/${file.agentId}/${file.source}:`; +} + +/** Same rebuildability test as `MiniDb.openOrRebuild`. */ +function isRebuildableCorruption(error: unknown): boolean { + return ( + error instanceof SyntaxError || + (error !== null && + typeof error === 'object' && + (error as { name?: string }).name === 'CorruptFrameError') + ); +} + +registerScopedService( + LifecycleScope.App, + IGlobalSearchService, + GlobalSearchService, + ScopeActivation.OnScopeCreated, + 'globalSearch', +); diff --git a/packages/kap-server/src/search/snippet.ts b/packages/kap-server/src/search/snippet.ts new file mode 100644 index 0000000000..c71ed54ad7 --- /dev/null +++ b/packages/kap-server/src/search/snippet.ts @@ -0,0 +1,66 @@ +/** + * `search` module — hit snippet generation (pure function). + * + * minidb's text index scores documents but cannot locate the match inside the + * text, so the snippet is produced here: a ~`radius`-char window around the + * first occurrence of any query term (case-insensitive), whitespace-collapsed, + * with ellipses marking truncation. Highlighting is left to clients. + */ + +function collapseWs(s: string): string { + return s.replaceAll(/\s+/g, ' ').trim(); +} + +/** Query terms for locating: whitespace-split words plus the whole query. */ +export function snippetTerms(query: string): string[] { + const terms = query + .split(/\s+/) + .map((t) => t.trim()) + .filter((t) => t.length > 0); + const whole = query.trim(); + if (whole.length > 0 && !terms.includes(whole)) terms.push(whole); + return terms; +} + +/** + * `anchor` — a caller-known hit location (`at` = offset of the match in + * `text`, `len` = match length in code units), e.g. the confirmation offset + * from literal search. When given, the term-guessing pass is skipped. The + * window math clamps out-of-range offsets, so an anchor taken from a + * normalized copy of the text (NFKC can shift offsets) degrades to a + * slightly shifted window, never an error. + */ +export function makeSnippet( + text: string, + query: string, + radius = 80, + anchor?: { at: number; len: number }, +): string { + let hitAt = -1; + let hitLen = 0; + if (anchor !== undefined) { + hitAt = anchor.at; + hitLen = anchor.len; + } else { + const lower = text.toLowerCase(); + // Earliest occurrence across all terms wins; on ties prefer the longer term. + for (const term of snippetTerms(query)) { + const i = lower.indexOf(term.toLowerCase()); + if (i === -1) continue; + if (hitAt === -1 || i < hitAt || (i === hitAt && term.length > hitLen)) { + hitAt = i; + hitLen = term.length; + } + } + } + + if (hitAt === -1) { + const head = collapseWs(text.slice(0, radius * 2)); + return text.length > radius * 2 ? `${head}…` : head; + } + + const start = Math.max(0, hitAt - radius); + const end = Math.min(text.length, hitAt + hitLen + radius); + const window = collapseWs(text.slice(start, end)); + return `${start > 0 ? '…' : ''}${window}${end < text.length ? '…' : ''}`; +} diff --git a/packages/kap-server/src/search/wireExtract.ts b/packages/kap-server/src/search/wireExtract.ts new file mode 100644 index 0000000000..d3c7598d26 --- /dev/null +++ b/packages/kap-server/src/search/wireExtract.ts @@ -0,0 +1,325 @@ +/** + * `search` module — wire.jsonl record extraction (pure functions). + * + * One wire.jsonl line yields three independent readings: + * + * 1. Indexable messages (0..n): `context.append_message` with + * `message.role === 'user'` → the user's real input (origins that are NOT + * user-typed are filtered out); `context.append_loop_event` with + * `event.type === 'content.part'` and a text part → one assistant text + * block (thinking, tool calls and tool results are NOT indexed). + * + * 2. A turn effect — how the record moves the 0-based turn counter, aligned + * with the transcript's cold-path grouping (`packages/transcript/src/ + * history/groupTurns.ts`). Turn counting is deliberately INDEPENDENT of + * indexing: a text-less user message (pure image) is not indexed but + * still opens a turn, and a filtered origin may still open one + * (`system_trigger` names in TURN_OPENING_SYSTEM_TRIGGERS). + * + * 3. A step effect — `step.begin` contributes a (uuid → ordinal) mapping + * entry so assistant text (which carries the event's `stepUuid`) can be + * attributed to its transcript step id (`t.`). Only + * `step.begin` moves the tracker; `step.end` is a no-op because the + * mapping lives until the next turn boundary. + * + * Kept free of any service / filesystem dependency so it is unit-testable. + */ + +export interface ExtractedWireMessage { + readonly role: 'user' | 'assistant'; + readonly text: string; + /** Epoch ms; undefined when the record carries no usable time. */ + readonly time?: number; + /** + * Owning step of an assistant text (the `content.part` event's `stepUuid`); + * user messages carry no step. + */ + readonly stepUuid?: string; +} + +/** + * How one wire record moves the 0-based turn counter (transcript groupTurns + * rules): + * - `open` — a user message that starts a new turn; `anchor` marks undo + * anchors (`isUndoAnchor`: no origin / kind 'user' / user-slash skill or + * plugin command), needed to replay `context.undo` on the counter; + * - `ensure` — assistant content; attaches to the current turn, opening a + * fallback turn when none exists yet (groupTurns' `ensureTurn`). Limited + * to the loop events whose folded assistant message SURVIVES settling + * (`content.part` with non-vacuous text, or `tool.call` — a tool.result + * folds to a tool message, and a vacuous step is dropped, so neither of + * those opens a turn); + * - `undo` — `context.undo`: drop the last `count` anchor-opened turns; + * - `none` — anything else. In particular `context.apply_compaction` and + * `context.clear` do NOT renumber: the transcript's cold replay keeps the + * full history (compaction appends a `compaction_summary` marker message, + * `clear` only raises a floor) and groupTurns numbers it continuously, + * matching the live TurnModel whose turn ids are monotonic. + */ +export type TurnEffect = + | { readonly kind: 'open'; readonly anchor: boolean } + | { readonly kind: 'ensure' } + | { readonly kind: 'undo'; readonly count: number } + | { readonly kind: 'none' }; + +/** + * How one wire record moves the per-turn step tracker: + * - `begin` — `step.begin`: map `uuid` to its step ordinal. `ordinal` is the + * wire record's own `step` field (the engine's live 1-based numbering, + * which the transcript's step ids `t.` use); absent on records + * too old to carry it — the tracker then falls back to counting begins + * within the turn (v1 loops had no loop-level retries, so counting equals + * the surviving-step numbering); + * - `none` — anything else. In particular `step.end` does NOT unmap: the + * mapping is reset at turn boundaries, not per step. + */ +export type StepEffect = + | { readonly kind: 'begin'; readonly uuid: string; readonly ordinal?: number } + | { readonly kind: 'none' }; + +export interface WireLineAnalysis { + readonly messages: ExtractedWireMessage[]; + readonly turn: TurnEffect; + readonly step: StepEffect; +} + +const NONE: TurnEffect = { kind: 'none' }; +const ENSURE: TurnEffect = { kind: 'ensure' }; +const STEP_NONE: StepEffect = { kind: 'none' }; + +/** + * Message origins appended by the system rather than typed by the user. These + * ride the wire as user-role `append_message` records but are not user input. + * Aligned with the transcript's grouping (hidden origins injection / + * system_trigger / retry, plus the marker origin compaction_summary). + * `skill_activation` / `plugin_command` are handled separately: a `user-slash` + * trigger is a command the user actually typed and stays searchable; other + * triggers are system noise and are filtered. + */ +const NON_USER_ORIGIN_KINDS: ReadonlySet = new Set([ + 'injection', + 'system_trigger', + 'retry', + 'compaction_summary', +]); + +/** groupTurns: hidden user origins that are folded away without opening a turn. */ +const HIDDEN_USER_ORIGINS: ReadonlySet = new Set(['injection', 'system_trigger', 'retry']); +/** + * groupTurns: hidden origins that nonetheless OPEN a real engine turn + * (`MessageStepRequest` with `admission: 'newTurn'` — goal continuation, + * subagent run prompts). They advance the ordinal but are not undo anchors. + */ +const TURN_OPENING_SYSTEM_TRIGGERS: ReadonlySet = new Set([ + 'goal_continuation', + 'subagent', +]); +/** groupTurns: origins rendered as timeline markers rather than turns. */ +const MARKER_USER_ORIGINS: ReadonlySet = new Set([ + 'skill_activation', + 'plugin_command', + 'compaction_summary', +]); + +interface OriginLike { + readonly kind?: unknown; + readonly trigger?: unknown; + readonly name?: unknown; +} + +function isUserSlashPrompt(origin: OriginLike): boolean { + return ( + (origin.kind === 'skill_activation' || origin.kind === 'plugin_command') && + origin.trigger === 'user-slash' + ); +} + +function isUserTypedOrigin(origin: OriginLike): boolean { + if (origin.kind === 'skill_activation' || origin.kind === 'plugin_command') { + return origin.trigger === 'user-slash'; + } + if (typeof origin.kind === 'string' && NON_USER_ORIGIN_KINDS.has(origin.kind)) return false; + return true; +} + +/** Same normalization as `sessionExport/wire-scan.ts`: seconds vs epoch ms. */ +function normalizeTimestampMs(value: unknown): number | undefined { + if (typeof value !== 'number' || !Number.isFinite(value) || value <= 0) return undefined; + return value > 1e12 ? Math.floor(value) : Math.floor(value * 1000); +} + +interface ContentPartLike { + readonly type?: unknown; + readonly text?: unknown; +} + +function textOfContent(content: unknown): string { + if (!Array.isArray(content)) return ''; + let text = ''; + for (const part of content as readonly ContentPartLike[]) { + if ( + part !== null && + typeof part === 'object' && + part.type === 'text' && + typeof part.text === 'string' + ) { + text += part.text; + } + } + return text; +} + +interface ParsedWireRecord { + readonly type?: unknown; + readonly time?: unknown; + readonly message?: unknown; + readonly event?: unknown; + readonly count?: unknown; +} + +function parseWireLine(line: string): ParsedWireRecord | undefined { + const trimmed = line.trim(); + if (trimmed.length === 0) return undefined; + let record: unknown; + try { + record = JSON.parse(trimmed); + } catch { + return undefined; + } + if (record === null || typeof record !== 'object' || Array.isArray(record)) return undefined; + return record as ParsedWireRecord; +} + +function turnEffectOfAppendMessage(message: unknown): TurnEffect { + if (message === null || typeof message !== 'object') return NONE; + const m = message as { role?: unknown; origin?: unknown }; + if (m.role === 'system') return NONE; // groupTurns skips system messages + if (m.role === 'assistant') return ENSURE; + if (m.role !== 'user') return NONE; // tool-role messages attach, never open + + const origin = + m.origin !== null && typeof m.origin === 'object' ? (m.origin as OriginLike) : undefined; + const kind = origin?.kind; + if (typeof kind === 'string' && HIDDEN_USER_ORIGINS.has(kind)) { + if ( + kind === 'system_trigger' && + typeof origin?.name === 'string' && + TURN_OPENING_SYSTEM_TRIGGERS.has(origin.name) + ) { + return { kind: 'open', anchor: false }; + } + return NONE; + } + if (typeof kind === 'string' && MARKER_USER_ORIGINS.has(kind)) { + // A user-slash skill/plugin command is a real user prompt (the engine's + // `isRealUserPrompt`): marker AND a turn opener, and an undo anchor. + if (origin !== undefined && isUserSlashPrompt(origin)) return { kind: 'open', anchor: true }; + return NONE; + } + // Every other user message opens a turn (kind 'user' / undefined / cron / + // task / hook_result / shell_command / …). Only origin-less and 'user' + // ones are undo anchors (`isUndoAnchor`). + const anchor = kind === undefined || kind === 'user'; + return { kind: 'open', anchor }; +} + +/** Full reading of one wire.jsonl line; unparseable lines analyze to zero. */ +export function analyzeWireLine(line: string): WireLineAnalysis { + const r = parseWireLine(line); + if (r === undefined) return { messages: [], turn: NONE, step: STEP_NONE }; + const time = normalizeTimestampMs(r.time); + + if (r.type === 'context.append_message') { + const turn = turnEffectOfAppendMessage(r.message); + const message = r.message; + const messages: ExtractedWireMessage[] = []; + if (message !== null && typeof message === 'object') { + const m = message as { role?: unknown; content?: unknown; origin?: unknown }; + if (m.role === 'user') { + const origin = m.origin; + const userTyped = + origin === null || + origin === undefined || + (typeof origin === 'object' && isUserTypedOrigin(origin as OriginLike)); + if (userTyped) { + const text = textOfContent(m.content).trim(); + if (text.length > 0) messages.push({ role: 'user', text, time }); + } + } + } + return { messages, turn, step: STEP_NONE }; + } + + if (r.type === 'context.append_loop_event') { + const event = r.event; + if (event === null || typeof event !== 'object') { + return { messages: [], turn: NONE, step: STEP_NONE }; + } + const e = event as { + type?: unknown; + part?: unknown; + uuid?: unknown; + step?: unknown; + stepUuid?: unknown; + }; + const messages: ExtractedWireMessage[] = []; + if (e.type === 'step.begin') { + if (typeof e.uuid !== 'string' || e.uuid.length === 0) { + return { messages: [], turn: NONE, step: STEP_NONE }; + } + const ordinal = + typeof e.step === 'number' && Number.isSafeInteger(e.step) && e.step > 0 + ? e.step + : undefined; + return { messages: [], turn: NONE, step: { kind: 'begin', uuid: e.uuid, ordinal } }; + } + const stepUuid = + typeof e.stepUuid === 'string' && e.stepUuid.length > 0 ? e.stepUuid : undefined; + // `ensure` only for events whose folded assistant message survives + // settling: non-vacuous content parts and tool calls. A `tool.result` + // folds into a tool message (never opens a turn), and a step with only + // vacuous content is dropped at `step.end`. + let turn: TurnEffect = NONE; + if (e.type === 'content.part') { + const part = e.part; + if (part !== null && typeof part === 'object') { + const p = part as ContentPartLike & { think?: unknown; encrypted?: unknown }; + if (p.type === 'text' && typeof p.text === 'string') { + const text = p.text.trim(); + if (text.length > 0) { + messages.push({ role: 'assistant', text, time, stepUuid }); + turn = ENSURE; + } + } else if (p.type === 'think' && typeof p.think === 'string') { + // Vacuous thinking (empty, unsigned) is dropped at `step.end`, + // same as empty text; signed or non-empty thinking survives. + if (p.think.trim().length > 0 || p.encrypted !== undefined) turn = ENSURE; + } else { + // Non-text content (image, …) is non-vacuous: the assistant survives. + turn = ENSURE; + } + } + } else if (e.type === 'tool.call') { + turn = ENSURE; + } + return { messages, turn, step: STEP_NONE }; + } + + if (r.type === 'context.undo') { + const count = r.count; + if (typeof count === 'number' && Number.isSafeInteger(count) && count > 0) { + return { messages: [], turn: { kind: 'undo', count }, step: STEP_NONE }; + } + return { messages: [], turn: NONE, step: STEP_NONE }; + } + + return { messages: [], turn: NONE, step: STEP_NONE }; +} + +/** + * Extract indexable messages from one wire.jsonl line. Unparseable lines and + * record types outside the two indexed shapes yield an empty array. + */ +export function extractFromWireLine(line: string): ExtractedWireMessage[] { + return analyzeWireLine(line).messages; +} diff --git a/packages/kap-server/src/start.ts b/packages/kap-server/src/start.ts index 47c0ca30ab..79610a3d95 100644 --- a/packages/kap-server/src/start.ts +++ b/packages/kap-server/src/start.ts @@ -79,6 +79,12 @@ import { createCredentialValidator } from './services/auth/credentials'; import { resolvePasswordHash } from './services/auth/password'; import { createTokenStore } from './services/auth/tokenStore'; +// Temporary feature: global message search. Importing this module registers +// `IGlobalSearchService` (App scope) into the DI registry as a side effect, so +// it MUST stay above any `bootstrap()` call — registration happens at module +// evaluation time. +import { drainGlobalSearchDisposals, IGlobalSearchService } from './search/searchService'; + export interface ServerStartOptions { readonly host?: string; readonly port?: number; @@ -344,6 +350,10 @@ export async function startServer(opts: ServerStartOptions = {}): Promise matches the documented v2 route table and meta e "POST", "/api/v1/providers/{tail}", ], + [ + "POST", + "/api/v1/search", + ], [ "POST", "/api/v1/sessions", diff --git a/packages/kap-server/test/search/searchRoute.test.ts b/packages/kap-server/test/search/searchRoute.test.ts new file mode 100644 index 0000000000..9c35d50730 --- /dev/null +++ b/packages/kap-server/test/search/searchRoute.test.ts @@ -0,0 +1,209 @@ +import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { ISessionIndex, type SessionSummary } from '@moonshot-ai/agent-core-v2'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +import { type RunningServer, startServer } from '../../src/start'; +import { authedFetch } from '../helpers/auth'; + +interface Envelope { + code: number; + msg: string; + data: T; + request_id: string; + details?: { path: string; message: string }[]; +} + +interface SearchPageWire { + items: { + session_id: string; + workspace_id: string; + session_title: string; + agent_id: string; + role: string; + snippet: string; + time: number; + turn?: number; + step_id?: string; + score: number; + }[]; + has_more: boolean; + page_token?: string; + incomplete?: string; + index_state: { + state: string; + indexed_sessions: number; + total_sessions: number; + documents: number; + }; + source: string; +} + +const WS = 'ws_route'; + +function stubSessionIndex(summaries: SessionSummary[]): ISessionIndex { + return { + _serviceBrand: undefined, + list: async () => ({ items: summaries, nextCursor: undefined }), + get: async () => undefined, + countActive: async () => summaries.length, + }; +} + +describe('server-v2 /api/v1/search', () => { + let server: RunningServer | undefined; + let home: string | undefined; + let base: string; + + beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), 'kimi-server-v2-search-')); + // Fixture first: the boot-time background sync picks it up on its own. + const sessionDir = join(home, 'sessions', WS, 's1', 'agents', 'main'); + await mkdir(sessionDir, { recursive: true }); + await writeFile( + join(sessionDir, 'wire.jsonl'), + [ + JSON.stringify({ + type: 'context.append_message', + time: 1_700_000_000_000, + message: { + role: 'user', + content: [{ type: 'text', text: '帮我查一下苹果的价格' }], + origin: { kind: 'user' }, + }, + }), + JSON.stringify({ + type: 'context.append_loop_event', + time: 1_700_000_000_100, + event: { type: 'step.begin', uuid: 'u1', turnId: '0', step: 1 }, + }), + JSON.stringify({ + type: 'context.append_loop_event', + time: 1_700_000_000_200, + event: { + type: 'content.part', + stepUuid: 'u1', + part: { type: 'text', text: '苹果现价每斤九块九。' }, + }, + }), + ].join('\n') + '\n', + 'utf8', + ); + const summaries: SessionSummary[] = [ + { + id: 's1', + workspaceId: WS, + title: '苹果询价', + createdAt: 1_700_000_000_000, + updatedAt: 1_700_000_000_000, + archived: false, + }, + ]; + server = await startServer({ + host: '127.0.0.1', + port: 0, + homeDir: home, + logLevel: 'silent', + seeds: [[ISessionIndex, stubSessionIndex(summaries)]], + }); + base = `http://127.0.0.1:${server.port}`; + }); + + afterEach(async () => { + if (server !== undefined) { + await server.close(); + server = undefined; + } + if (home !== undefined) { + await rm(home, { recursive: true, force: true }); + home = undefined; + } + }); + + async function postSearch(body: unknown): Promise> { + const res = await authedFetch(server!, base, '/api/v1/search', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(body), + }); + return (await res.json()) as Envelope; + } + + it('searches across sessions and returns the wire-shaped page', { timeout: 20_000 }, async () => { + // The first sync runs in the background at boot; poll until it lands. + let body: Envelope | undefined; + for (let attempt = 0; attempt < 100; attempt++) { + body = await postSearch({ query: '苹果' }); + expect(body.code).toBe(0); + if (body.data.items.length > 0) break; + await new Promise((resolve) => setTimeout(resolve, 100)); + } + expect(body).toBeDefined(); + expect(body!.data.items.length).toBeGreaterThan(0); + + // Both the user message and the session title match '苹果'. + const hit = body!.data.items.find((h) => h.role === 'user'); + expect(hit).toBeDefined(); + expect(hit!.session_id).toBe('s1'); + expect(hit!.workspace_id).toBe(WS); + expect(hit!.session_title).toBe('苹果询价'); + expect(hit!.agent_id).toBe('main'); + expect(hit!.snippet).toContain('苹果'); + expect(hit!.step_id).toBeUndefined(); + // The assistant hit carries its transcript step id. + const assistant = body!.data.items.find((h) => h.role === 'assistant'); + expect(assistant).toBeDefined(); + expect(assistant!.turn).toBe(0); + expect(assistant!.step_id).toBe('t0.1'); + expect(body!.data.items.some((h) => h.role === 'title')).toBe(true); + expect(body!.data.has_more).toBe(false); + expect(['building', 'ready', 'readonly']).toContain(body!.data.index_state.state); + // No session is live in this server, so the page comes from the index. + expect(body!.data.source).toBe('index'); + }); + + it('rejects invalid bodies with 40001', async () => { + const emptyQuery = await postSearch({ query: '' }); + expect(emptyQuery.code).toBe(40001); + + const oversizedPage = await postSearch({ query: '苹果', page_size: 51 }); + expect(oversizedPage.code).toBe(40001); + + const badSort = await postSearch({ query: '苹果', sort: 'newest' }); + expect(badSort.code).toBe(40001); + + const badMode = await postSearch({ query: '苹果', mode: 'exact' }); + expect(badMode.code).toBe(40001); + + // A 1-character literal query is rejected by the service, not the schema. + const shortLiteral = await postSearch({ query: '苹', mode: 'literal' }); + expect(shortLiteral.code).toBe(40001); + expect(shortLiteral.msg).toContain('at least 2 characters'); + + // A page token that decodes to a non-object is a parameter error, not a 500. + const nullToken = await postSearch({ + query: '苹果', + page_token: Buffer.from('null').toString('base64url'), + }); + expect(nullToken.code).toBe(40001); + }); + + it('serves literal mode through the wire', { timeout: 20_000 }, async () => { + let body: Envelope | undefined; + for (let attempt = 0; attempt < 100; attempt++) { + body = await postSearch({ query: '的价格', mode: 'literal' }); + expect(body.code).toBe(0); + if (body.data.items.length > 0) break; + await new Promise((resolve) => setTimeout(resolve, 100)); + } + expect(body).toBeDefined(); + const hit = body!.data.items.find((h) => h.role === 'user'); + expect(hit).toBeDefined(); + expect(hit!.snippet).toContain('的价格'); + expect(hit!.score).toBe(0); + expect(body!.data.items.some((h) => h.role === 'assistant')).toBe(false); + expect(body!.data.incomplete).toBeUndefined(); + }); +}); diff --git a/packages/kap-server/test/search/searchService.test.ts b/packages/kap-server/test/search/searchService.test.ts new file mode 100644 index 0000000000..bc587b27eb --- /dev/null +++ b/packages/kap-server/test/search/searchService.test.ts @@ -0,0 +1,1527 @@ +import { appendFile, mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import type { + IBootstrapService, + ILogService, + ISessionIndex, + SessionSummary, +} from '@moonshot-ai/agent-core-v2'; +import { TranscriptStore, type TranscriptOperation } from '@moonshot-ai/transcript'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +import { + GlobalSearchError, + GlobalSearchService, + drainGlobalSearchDisposals, + type LiveTranscriptSource, +} from '../../src/search/searchService'; + +// --------------------------------------------------------------------------- +// fixtures & stubs +// --------------------------------------------------------------------------- + +const WS = 'ws_test'; + +const T1 = 1_700_000_000_000; +const T2 = 1_700_000_100_000; +const T3 = 1_700_000_200_000; + +function summary(id: string, title: string, updatedAt = T1): SessionSummary { + return { id, workspaceId: WS, title, createdAt: updatedAt, updatedAt, archived: false }; +} + +function makeBootstrap(home: string): IBootstrapService { + return { + homeDir: home, + sessionDir: (ws: string, sid: string) => join(home, 'sessions', ws, sid), + } as unknown as IBootstrapService; +} + +function makeSessionIndex(list: ISessionIndex['list']): ISessionIndex { + return { + _serviceBrand: undefined, + list, + get: async () => undefined, + countActive: async () => 0, + }; +} + +function staticIndex(summaries: SessionSummary[]): ISessionIndex { + return makeSessionIndex(async () => ({ items: summaries, nextCursor: undefined })); +} + +function userLine(text: string, time: number, origin?: unknown): string { + return JSON.stringify({ + type: 'context.append_message', + time, + message: { + role: 'user', + content: [{ type: 'text', text }], + origin: origin ?? { kind: 'user' }, + }, + }); +} + +function assistantLine(text: string, time: number): string { + return JSON.stringify({ + type: 'context.append_loop_event', + time, + event: { type: 'content.part', part: { type: 'text', text } }, + }); +} + +function stepBeginLine(uuid: string, step: number, time: number): string { + return JSON.stringify({ + type: 'context.append_loop_event', + time, + event: { type: 'step.begin', uuid, turnId: '0', step }, + }); +} + +function assistantStepLine(text: string, stepUuid: string, time: number): string { + return JSON.stringify({ + type: 'context.append_loop_event', + time, + event: { type: 'content.part', stepUuid, part: { type: 'text', text } }, + }); +} + +function rawRecord(value: unknown): string { + return JSON.stringify(value); +} + +async function writeWire( + home: string, + sessionId: string, + agentId: string, + lines: string[], +): Promise { + const dir = join(home, 'sessions', WS, sessionId, 'agents', agentId); + await mkdir(dir, { recursive: true }); + const file = join(dir, 'wire.jsonl'); + await writeFile(file, lines.map((l) => `${l}\n`).join(''), 'utf8'); + return file; +} + +const noopLog = { + error: () => {}, + warn: () => {}, + info: () => {}, + debug: () => {}, +} as unknown as ILogService; + +function makeService(home: string, index: ISessionIndex): GlobalSearchService { + const service = new GlobalSearchService(index, makeBootstrap(home), noopLog); + service.syncDebounceMs = 0; + return service; +} + +// --------------------------------------------------------------------------- +// suite +// --------------------------------------------------------------------------- + +describe('GlobalSearchService', () => { + let home: string | undefined; + const services: GlobalSearchService[] = []; + + beforeEach(async () => { + home = await mkdtemp(join(tmpdir(), 'kimi-kap-search-')); + }); + + afterEach(async () => { + for (const service of services.splice(0)) service.dispose(); + await drainGlobalSearchDisposals(); + if (home !== undefined) { + await rm(home, { recursive: true, force: true }); + home = undefined; + } + }); + + function track(service: GlobalSearchService): GlobalSearchService { + services.push(service); + return service; + } + + it('indexes user and assistant text and finds Chinese and English terms', async () => { + const s1 = summary('s1', '搜索重构讨论', T1); + await writeWire(home!, 's1', 'main', [ + userLine('帮我看看苹果怎么挑', T1), + assistantLine('Here is the apple picking guide.', T2), + userLine('忽略我', T3, { kind: 'injection', variant: 'reminder' }), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const cn = await service.search({ query: '苹果' }); + expect(cn.items.length).toBeGreaterThan(0); + const cnHit = cn.items[0]!; + expect(cnHit.sessionId).toBe('s1'); + expect(cnHit.workspaceId).toBe(WS); + expect(cnHit.sessionTitle).toBe('搜索重构讨论'); + expect(cnHit.agentId).toBe('main'); + expect(cnHit.role).toBe('user'); + expect(cnHit.snippet).toContain('苹果'); + expect(cnHit.time).toBe(T1); + expect(cnHit.score).toBeGreaterThan(0); + + const en = await service.search({ query: 'apple' }); + expect(en.items.some((h) => h.role === 'assistant')).toBe(true); + + // The injection-origin user message must NOT be indexed. + const injected = await service.search({ query: '忽略我' }); + expect(injected.items).toEqual([]); + }); + + it('hits session titles as title docs', async () => { + const s1 = summary('s1', '季度总结报告', T1); + await writeWire(home!, 's1', 'main', [userLine('随便说点什么', T1)]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page = await service.search({ query: '季度' }); + const titleHit = page.items.find((h) => h.role === 'title'); + expect(titleHit).toBeDefined(); + expect(titleHit?.sessionId).toBe('s1'); + expect(titleHit?.snippet).toContain('季度'); + }); + + it('filters by container (session and agent)', async () => { + const s1 = summary('s1', 'one', T1); + const s2 = summary('s2', 'two', T1); + await writeWire(home!, 's1', 'main', [userLine('苹果 from s1', T1)]); + await writeWire(home!, 's2', 'main', [userLine('苹果 from s2 main', T1)]); + await writeWire(home!, 's2', 'agent-1', [userLine('苹果 from s2 subagent', T1)]); + const service = track(makeService(home!, staticIndex([s1, s2]))); + await service.reindex(); + + const all = await service.search({ query: '苹果' }); + expect(all.items.length).toBe(3); + + const inS2 = await service.search({ query: '苹果', container: { sessionId: 's2' } }); + expect(inS2.items.length).toBe(2); + expect(inS2.items.every((h) => h.sessionId === 's2')).toBe(true); + + const inSub = await service.search({ + query: '苹果', + container: { sessionId: 's2', agentId: 'agent-1' }, + }); + expect(inSub.items.length).toBe(1); + expect(inSub.items[0]?.agentId).toBe('agent-1'); + }); + + it('filters by role and time range', async () => { + const s1 = summary('s1', 'roles', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 early', T1), + assistantLine('苹果 middle', T2), + userLine('苹果 late', T3), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const users = await service.search({ query: '苹果', role: 'user' }); + expect(users.items.length).toBe(2); + expect(users.items.every((h) => h.role === 'user')).toBe(true); + + const ranged = await service.search({ query: '苹果', startTime: T2, endTime: T2 }); + expect(ranged.items.length).toBe(1); + expect(ranged.items[0]?.time).toBe(T2); + }); + + it('sorts by time in both directions', async () => { + const s1 = summary('s1', 'sort', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 one', T1), + userLine('苹果 two', T2), + userLine('苹果 three', T3), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const desc = await service.search({ query: '苹果', sort: 'time_desc' }); + expect(desc.items.map((h) => h.time)).toEqual([T3, T2, T1]); + const asc = await service.search({ query: '苹果', sort: 'time_asc' }); + expect(asc.items.map((h) => h.time)).toEqual([T1, T2, T3]); + }); + + it('paginates with an opaque cursor and rejects changed conditions', async () => { + const s1 = summary('s1', 'paging', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 one', T1), + userLine('苹果 two', T2), + userLine('苹果 three', T3), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page1 = await service.search({ query: '苹果', sort: 'time_asc', pageSize: 2 }); + expect(page1.items.length).toBe(2); + expect(page1.hasMore).toBe(true); + expect(page1.pageToken).toBeDefined(); + + const page2 = await service.search({ + query: '苹果', + sort: 'time_asc', + pageSize: 2, + pageToken: page1.pageToken, + }); + expect(page2.items.length).toBe(1); + expect(page2.hasMore).toBe(false); + expect(page2.pageToken).toBeUndefined(); + + const times = [...page1.items, ...page2.items].map((h) => h.time); + expect(new Set(times).size).toBe(3); + + // Same token with a changed query condition → parameter error. + await expect( + service.search({ query: '香蕉', sort: 'time_asc', pageToken: page1.pageToken }), + ).rejects.toMatchObject({ reason: 'invalid_page_token' }); + // Malformed token. + await expect(service.search({ query: '苹果', pageToken: '!!!' })).rejects.toBeInstanceOf( + GlobalSearchError, + ); + }); + + it('picks up appended wire lines on the next search', async () => { + const s1 = summary('s1', 'incremental', T1); + const file = await writeWire(home!, 's1', 'main', [userLine('苹果 initial', T1)]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + await appendFile(file, `${userLine('苹果 appended', T2)}\n`, 'utf8'); + const page = await service.search({ query: '苹果' }); + expect(page.items.length).toBe(2); + expect(page.items.some((h) => h.snippet.includes('appended'))).toBe(true); + }); + + it('reports indexState building before the first full sync and ready after', async () => { + const s1 = summary('s1', 'state', T1); + await writeWire(home!, 's1', 'main', [userLine('苹果 state', T1)]); + + // Block the session enumeration until released, so the constructor's + // background sync cannot finish before the first search. + let release!: () => void; + const gate = new Promise((resolve) => { + release = resolve; + }); + const index = makeSessionIndex(async () => { + await gate; + return { items: [s1], nextCursor: undefined }; + }); + const service = track(makeService(home!, index)); + + const building = await service.search({ query: '苹果' }); + expect(building.indexState.state).toBe('building'); + expect(building.items).toEqual([]); + + release(); + await service.reindex(); + const ready = await service.search({ query: '苹果' }); + expect(ready.indexState.state).toBe('ready'); + expect(ready.indexState.indexedSessions).toBe(1); + expect(ready.indexState.totalSessions).toBe(1); + expect(ready.indexState.documents).toBe(2); // 1 message + 1 title doc + }); + + it('drops docs of sessions that disappear between syncs', async () => { + const s1 = summary('s1', 'gone', T1); + await writeWire(home!, 's1', 'main', [userLine('苹果 ephemeral', T1)]); + const sessions = [s1]; + const service = track( + makeService( + home!, + makeSessionIndex(async () => ({ items: sessions, nextCursor: undefined })), + ), + ); + await service.reindex(); + expect((await service.search({ query: '苹果' })).items.length).toBe(1); + + sessions.length = 0; // session directory vanished from the index + const page = await service.search({ query: '苹果' }); + expect(page.items).toEqual([]); + }); + + it('rescans a wire file that shrank between syncs', async () => { + const s1 = summary('s1', 'shrink', T1); + const file = await writeWire(home!, 's1', 'main', [ + userLine('苹果 old one', T1), + userLine('苹果 old two', T2), + userLine('苹果 old three', T3), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + expect((await service.search({ query: '苹果' })).items.length).toBe(3); + + // Rewrite with a shorter file: stale docs must be dropped and the new + // content rescanned from offset 0. + await writeFile(file, `${userLine('香蕉 fresh', T1)}\n`, 'utf8'); + const stale = await service.search({ query: '苹果' }); + expect(stale.items).toEqual([]); + const fresh = await service.search({ query: '香蕉' }); + expect(fresh.items.length).toBe(1); + expect(fresh.items[0]?.snippet).toContain('fresh'); + }); + + it('does not advance the watermark past an incomplete trailing line', async () => { + const s1 = summary('s1', 'tail', T1); + const file = await writeWire(home!, 's1', 'main', [userLine('苹果 base', T1)]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + // A partial line (no trailing newline) must not be indexed nor consumed. + await appendFile(file, userLine('苹果 partial', T2), 'utf8'); + expect((await service.search({ query: 'partial' })).items).toEqual([]); + + // Once the line is completed, the next pass picks it up. + await appendFile(file, '\n', 'utf8'); + const page = await service.search({ query: 'partial' }); + expect(page.items.length).toBe(1); + expect(page.items[0]?.role).toBe('user'); + }); + + it('indexes legacy root and v2 agents layouts of one session without key collisions', async () => { + const s1 = summary('s1', 'dual layout', T1); + // Legacy v1 layout: /wire.jsonl; v2 layout: agents/main/wire.jsonl. + await writeWire(home!, 's1', 'main', [userLine('苹果 from agents', T2)]); + await writeFile( + join(home!, 'sessions', WS, 's1', 'wire.jsonl'), + `${userLine('苹果 from root', T1)}\n`, + 'utf8', + ); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page = await service.search({ query: '苹果', sort: 'time_asc', role: 'user' }); + expect(page.items.length).toBe(2); + expect(page.items.every((h) => h.agentId === 'main')).toBe(true); + const snippets = page.items.map((h) => h.snippet); + expect(snippets.some((s) => s.includes('root'))).toBe(true); + expect(snippets.some((s) => s.includes('agents'))).toBe(true); + }); + + it('rejects a pageToken that decodes to a non-object', async () => { + const s1 = summary('s1', 'token', T1); + await writeWire(home!, 's1', 'main', [userLine('苹果 token', T1)]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + for (const payload of ['null', '42', '"str"', '[1,2]']) { + const token = Buffer.from(payload).toString('base64url'); + await expect(service.search({ query: '苹果', pageToken: token })).rejects.toMatchObject({ + reason: 'invalid_page_token', + }); + } + }); + + it('drops docs of a wire file that disappears while its session remains', async () => { + const s1 = summary('s1', 'file gone', T1); + await writeWire(home!, 's1', 'main', [userLine('苹果 main agent', T1)]); + const subFile = await writeWire(home!, 's1', 'agent-1', [userLine('苹果 sub agent', T2)]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + expect((await service.search({ query: '苹果' })).items.length).toBe(2); + + await rm(subFile); + const page = await service.search({ query: '苹果' }); + expect(page.items.length).toBe(1); + expect(page.items[0]?.agentId).toBe('main'); + }); + + it('runs a second instance read-only and catches up from the WAL', async () => { + const s1 = summary('s1', 'shared', T1); + const file = await writeWire(home!, 's1', 'main', [userLine('苹果 base', T1)]); + const index = staticIndex([s1]); + + const writer = track(makeService(home!, index)); + await writer.reindex(); + + // Same process, same homeDir: the lock is held by `writer`, so this + // instance must downgrade to read-only instead of rebuilding. + const reader = track(makeService(home!, index)); + const status = await reader.status(); + expect(status.documents).toBe(2); // replayed at open: message + title + + const first = await reader.search({ query: '苹果' }); + expect(first.indexState.state).toBe('readonly'); + expect(first.items.length).toBe(1); + + // The writer indexes a new line; the reader must see it via the + // fingerprint check + catchUpFromWal incremental replay (no full reopen). + await appendFile(file, `${userLine('苹果 delta', T2)}\n`, 'utf8'); + await writer.search({ query: '苹果' }); // writer-side incremental sync + const caughtUp = await reader.search({ query: '苹果' }); + expect(caughtUp.items.length).toBe(2); + expect(caughtUp.items.some((h) => h.snippet.includes('delta'))).toBe(true); + + // WAL rotation on the writer forces the reader's full-reopen fallback; + // results stay correct afterwards. + const writerDb = (writer as unknown as { db: { compact(): Promise } | null }).db; + await writerDb?.compact(); + const afterRotation = await reader.search({ query: '苹果' }); + expect(afterRotation.items.length).toBe(2); + }); + + it('rejects reindex on a read-only instance', async () => { + const s1 = summary('s1', 'lock', T1); + await writeWire(home!, 's1', 'main', [userLine('苹果 lock', T1)]); + const index = staticIndex([s1]); + const writer = track(makeService(home!, index)); + await writer.reindex(); + + const reader = track(makeService(home!, index)); + await expect(reader.reindex()).rejects.toMatchObject({ reason: 'readonly_index' }); + }); + + // -- turn ordinals ------------------------------------------------------------ + + it('assigns 0-based turn ordinals to user and assistant hits', async () => { + const s1 = summary('s1', 'turns', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 question zero', T1), + assistantLine('苹果 answer zero', T2), + userLine('苹果 question one', T3), + assistantLine('苹果 answer one', T3 + 1000), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const users = await service.search({ query: '苹果', role: 'user', sort: 'time_asc' }); + expect(users.items.map((h) => h.turn)).toEqual([0, 1]); + const assistants = await service.search({ query: '苹果', role: 'assistant', sort: 'time_asc' }); + expect(assistants.items.map((h) => h.turn)).toEqual([0, 1]); + }); + + it('counts turns independently of indexing (text-less prompts, hidden & marker origins)', async () => { + const s1 = summary('s1', 'counting', T1); + await writeWire(home!, 's1', 'main', [ + // Pure-image user prompt: not indexed, but opens turn 0. + rawRecord({ + type: 'context.append_message', + time: T1, + message: { role: 'user', content: [{ type: 'image', source: { kind: 'url', url: 'x' } }] }, + }), + // Injection: no turn. + userLine('苹果 injected', T1 + 100, { kind: 'injection', variant: 'reminder' }), + // Turn-opening system trigger: opens turn 2 (promptless), not indexed. + userLine('苹果 continuation', T1 + 200, { + kind: 'system_trigger', + name: 'goal_continuation', + }), + // Marker without user-slash: no turn. + userLine('苹果 skill noise', T1 + 300, { kind: 'skill_activation', trigger: 'model-tool' }), + userLine('苹果 typed', T2, { kind: 'user' }), + // user-slash skill: indexed AND opens turn 4. + userLine('/commit 苹果 ship it', T3, { kind: 'skill_activation', trigger: 'user-slash' }), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page = await service.search({ query: '苹果', sort: 'time_asc' }); + const bySnippet = (needle: string) => + page.items.find((h) => h.snippet.includes(needle) && h.role === 'user'); + expect(bySnippet('injected')).toBeUndefined(); // filtered out of the index + expect(bySnippet('continuation')).toBeUndefined(); + expect(bySnippet('skill noise')).toBeUndefined(); + expect(bySnippet('typed')?.turn).toBe(2); // image=0, injection=–, trigger=1 + expect(bySnippet('ship it')?.turn).toBe(3); + }); + + it('attaches assistant content to a fallback turn when no prompt opened one', async () => { + const s1 = summary('s1', 'fallback', T1); + await writeWire(home!, 's1', 'main', [ + assistantLine('苹果 orphan answer', T1), + userLine('苹果 later question', T2), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page = await service.search({ query: '苹果', sort: 'time_asc' }); + expect(page.items.map((h) => [h.role, h.turn])).toEqual([ + ['assistant', 0], + ['user', 1], + ]); + }); + + it('keeps the turn counter across incremental sync passes', async () => { + const s1 = summary('s1', 'resume', T1); + const file = await writeWire(home!, 's1', 'main', [ + userLine('苹果 first', T1), + assistantLine('苹果 first reply', T2), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + await appendFile( + file, + `${userLine('苹果 second', T3)}\n${assistantLine('苹果 second reply', T3 + 1000)}\n`, + 'utf8', + ); + const page = await service.search({ query: '苹果', sort: 'time_asc' }); + expect(page.items.map((h) => [h.role, h.turn])).toEqual([ + ['user', 0], + ['assistant', 0], + ['user', 1], + ['assistant', 1], + ]); + }); + + it('restarts the turn counter when a shrunk file is rescanned', async () => { + const s1 = summary('s1', 'shrink turns', T1); + const file = await writeWire(home!, 's1', 'main', [ + userLine('苹果 a', T1), + userLine('苹果 b', T2), + userLine('苹果 c', T3), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + expect( + (await service.search({ query: '苹果', sort: 'time_asc' })).items.map((h) => h.turn), + ).toEqual([0, 1, 2]); + + await writeFile(file, `${userLine('苹果 only', T1)}\n`, 'utf8'); + const page = await service.search({ query: '苹果' }); + expect(page.items.length).toBe(1); + expect(page.items[0]?.turn).toBe(0); + }); + + it('rewinds the counter on context.undo and renumbers after it', async () => { + const s1 = summary('s1', 'undo', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 before', T1), + assistantLine('苹果 before reply', T2), + userLine('苹果 undone', T3), + assistantLine('苹果 undone reply', T3 + 1000), + rawRecord({ type: 'context.undo', time: T3 + 2000, count: 1 }), + userLine('苹果 redone', T3 + 3000), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page = await service.search({ query: '苹果', sort: 'time_asc' }); + const bySnippet = (needle: string) => page.items.find((h) => h.snippet.includes(needle)); + expect(bySnippet('before')?.turn).toBe(0); + // The undone turn's docs keep their pre-undo ordinal (transcript no longer + // shows them — accepted deviation), and the redo reuses ordinal 1. + expect(bySnippet('undone reply')?.turn).toBe(1); + expect(bySnippet('redone')?.turn).toBe(1); + }); + + it('keeps numbering monotonic across context.apply_compaction', async () => { + const s1 = summary('s1', 'compaction', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 before compaction', T1), + assistantLine('苹果 old reply', T2), + // The transcript's cold replay keeps full history (the compaction becomes + // a `compaction_summary` marker message) and groupTurns numbers it + // continuously — so the indexer must NOT reset its counter either. + rawRecord({ + type: 'context.apply_compaction', + time: T3, + summary: 'condensed', + compactedCount: 2, + }), + userLine('summary', T3 + 1000, { kind: 'compaction_summary' }), + // Assistant content right after the compaction marker still attaches to + // the pre-compaction turn (the marker does not open one). + assistantLine('苹果 post-compaction reply', T3 + 1500), + userLine('苹果 after compaction', T3 + 2000), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page = await service.search({ query: '苹果', sort: 'time_asc' }); + const bySnippet = (needle: string) => page.items.find((h) => h.snippet.includes(needle)); + expect(bySnippet('before compaction')?.turn).toBe(0); + expect(bySnippet('old reply')?.turn).toBe(0); + expect(bySnippet('post-compaction reply')?.turn).toBe(0); + expect(bySnippet('after compaction')?.turn).toBe(1); + }); + + // -- step ids --------------------------------------------------------------- + + it('assigns transcript step ids to assistant hits; user and title hits carry none', async () => { + const s1 = summary('s1', '苹果 steps', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 question', T1), + stepBeginLine('u1', 1, T1 + 100), + assistantStepLine('苹果 first draft', 'u1', T1 + 200), + // A vacuous step: begins but owns no text — no document, and the next + // step keeps the wire's original ordinal (live numbering, gaps allowed). + stepBeginLine('u2', 2, T1 + 300), + stepBeginLine('u3', 3, T1 + 400), + assistantStepLine('苹果 second draft', 'u3', T1 + 500), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const assistants = await service.search({ query: '苹果', role: 'assistant', sort: 'time_asc' }); + expect(assistants.items.map((h) => [h.turn, h.stepId])).toEqual([ + [0, 't0.1'], + [0, 't0.3'], + ]); + + const users = await service.search({ query: '苹果', role: 'user' }); + expect(users.items[0]?.stepId).toBeUndefined(); + const title = await service.search({ query: '苹果', role: 'title' }); + expect(title.items[0]?.stepId).toBeUndefined(); + }); + + it('omits step ids when no matching step.begin was seen', async () => { + const s1 = summary('s1', 'orphans', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 question', T1), + // A stepUuid the tracker never saw a begin for… + assistantStepLine('苹果 orphan', 'unknown-uuid', T2), + // …and a legacy record with no stepUuid at all. + assistantLine('苹果 legacy', T3), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page = await service.search({ query: '苹果', role: 'assistant', sort: 'time_asc' }); + expect(page.items.map((h) => [h.turn, h.stepId])).toEqual([ + [0, undefined], + [0, undefined], + ]); + }); + + it('resets step numbering at turn boundaries and after an undo', async () => { + const s1 = summary('s1', 'reset', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 first', T1), + stepBeginLine('u1', 1, T1 + 100), + assistantStepLine('苹果 reply one', 'u1', T1 + 200), + userLine('苹果 second', T2), + stepBeginLine('u2', 1, T2 + 100), + assistantStepLine('苹果 reply two', 'u2', T2 + 200), + rawRecord({ type: 'context.undo', time: T2 + 300, count: 1 }), + userLine('苹果 redone', T3), + stepBeginLine('u3', 1, T3 + 100), + assistantStepLine('苹果 redone reply', 'u3', T3 + 200), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page = await service.search({ query: '苹果', role: 'assistant', sort: 'time_asc' }); + // The undone step keeps its pre-undo id (same deviation as turns); the + // redo renumbers from a fresh tracker. + expect(page.items.map((h) => h.stepId)).toEqual(['t0.1', 't1.1', 't1.1']); + }); + + it('falls back to counting step.begin records when the wire carries no ordinal', async () => { + const s1 = summary('s1', 'fallback', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 question', T1), + rawRecord({ + type: 'context.append_loop_event', + time: T1 + 100, + event: { type: 'step.begin', uuid: 'u1' }, + }), + assistantStepLine('苹果 reply', 'u1', T1 + 200), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page = await service.search({ query: '苹果', role: 'assistant' }); + expect(page.items[0]?.stepId).toBe('t0.1'); + }); + + it('keeps step attribution across incremental sync passes', async () => { + const s1 = summary('s1', 'resume steps', T1); + // The first pass indexes the turn boundary and the step.begin only; the + // text arrives later — the uuid → ordinal mapping must survive in the + // persisted stepState for the next pass to attribute the doc. + const file = await writeWire(home!, 's1', 'main', [ + userLine('苹果 question', T1), + stepBeginLine('u1', 1, T1 + 100), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + await appendFile(file, `${assistantStepLine('苹果 reply', 'u1', T2)}\n`, 'utf8'); + const page = await service.search({ query: '苹果', role: 'assistant' }); + expect(page.items.map((h) => [h.turn, h.stepId])).toEqual([[0, 't0.1']]); + }); + + it('rescans a wire file whose meta predates step tracking', async () => { + const s1 = summary('s1', 'legacy meta', T1); + const file = await writeWire(home!, 's1', 'main', [ + userLine('苹果 question', T1), + stepBeginLine('u1', 1, T1 + 100), + assistantStepLine('苹果 reply one', 'u1', T1 + 200), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + // Simulate a file meta written before step tracking existed by stripping + // stepState from the persisted meta. + const db = ( + service as unknown as { + db: { + query(criteria: { + key: { prefix: string }; + }): { key: string; value: Record }[]; + set(key: string, value: unknown): Promise; + } | null; + } + ).db; + expect(db).not.toBeNull(); + const metaRows = db!.query({ key: { prefix: '\0meta\\file\\' } }); + expect(metaRows.length).toBe(1); + for (const row of metaRows) { + const { stepState: _stripped, ...rest } = row.value; + await db!.set(row.key, rest); + } + + // Appending triggers a sync; the legacy meta must force a full rescan of + // the file, so every doc — old and new — ends up with a stepId. + await appendFile(file, `${assistantStepLine('苹果 reply two', 'u1', T2)}\n`, 'utf8'); + const page = await service.search({ query: '苹果', role: 'assistant', sort: 'time_asc' }); + expect(page.items.map((h) => h.stepId)).toEqual(['t0.1', 't0.1']); + }); + + // -- literal mode ------------------------------------------------------------- + + describe('literal mode', () => { + /** Symbol/CJK/emoji-heavy session the terms tokenizer cannot serve. */ + async function literalFixture(): Promise { + const s1 = summary('s1', 'literal 会话', T1); + await writeWire(home!, 's1', 'main', [ + userLine('modern C++ patterns', T1), + assistantLine('use foo-bar here', T1 + 100), + userLine('a foo bar without dash', T1 + 200), + userLine('检查项 **已通过** 审核', T1 + 300), + userLine('inline math $\\frac{a}{b}$ here', T1 + 400), + userLine('launch 🚀🎉 today', T1 + 500), + userLine('짧은 한국어 문구 테스트', T1 + 600), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + return service; + } + + it('finds symbol substrings exactly; distractors without the exact substring do not hit', async () => { + const service = await literalFixture(); + + const cpp = await service.search({ query: 'C++', mode: 'literal' }); + expect(cpp.items.length).toBe(1); + expect(cpp.items[0]?.snippet).toContain('C++'); + expect(cpp.items[0]?.score).toBe(0); // literal hits are unscored + expect(cpp.incomplete).toBeUndefined(); + + // 'foo-bar' must not match the 'foo bar' doc, and vice versa. + const dashed = await service.search({ query: 'foo-bar', mode: 'literal' }); + expect(dashed.items.length).toBe(1); + expect(dashed.items[0]?.snippet).toContain('foo-bar'); + const spaced = await service.search({ query: 'foo bar', mode: 'literal' }); + expect(spaced.items.length).toBe(1); + expect(spaced.items[0]?.snippet).toContain('without dash'); + + const passed = await service.search({ query: '**已通过**', mode: 'literal' }); + expect(passed.items.length).toBe(1); + expect(passed.items[0]?.snippet).toContain('已通过'); + + const latex = await service.search({ query: '$\\frac{a}{b}$', mode: 'literal' }); + expect(latex.items.length).toBe(1); + expect(latex.items[0]?.snippet).toContain('\\frac'); + + const korean = await service.search({ query: '한국어 문구', mode: 'literal' }); + expect(korean.items.length).toBe(1); + expect(korean.items[0]?.snippet).toContain('한국어'); + }); + + it('is case-insensitive and orders by time desc regardless of sort', async () => { + const service = await literalFixture(); + const page = await service.search({ query: 'c++', mode: 'literal' }); + expect(page.items.length).toBe(1); + expect(page.items[0]?.snippet).toContain('C++'); + + // Multiple hits: time desc even when sort is left at its default. + const foo = await service.search({ query: 'foo', mode: 'literal' }); + expect(foo.items.map((h) => h.time)).toEqual([T1 + 200, T1 + 100]); + }); + + it('serves 2-character queries over the 2-gram path, including emoji pairs', async () => { + const service = await literalFixture(); + const plus = await service.search({ query: '++', mode: 'literal' }); + expect(plus.items.length).toBe(1); + expect(plus.items[0]?.snippet).toContain('C++'); + + const emoji = await service.search({ query: '🚀🎉', mode: 'literal' }); + expect(emoji.items.length).toBe(1); + expect(emoji.items[0]?.snippet).toContain('🚀🎉'); + }); + + it('rejects queries shorter than 2 normalized characters', async () => { + const service = await literalFixture(); + await expect(service.search({ query: 'c', mode: 'literal' })).rejects.toMatchObject({ + reason: 'invalid_query', + }); + await expect(service.search({ query: 'c', mode: 'literal' })).rejects.toThrow( + /literal queries need at least 2 characters/, + ); + // NFKC can LEGALIZE a 1-character query: the ligature 'ff' folds to 'ff'. + const ligature = await service.search({ query: 'ff', mode: 'literal' }); + expect(ligature.items).toEqual([]); + // An untrimmed 2-space query is a legal literal query too. + const spaces = await service.search({ query: ' ', mode: 'literal' }); + expect(spaces.items).toEqual([]); + }); + + it('flags candidate-cap truncation as incomplete', async () => { + const s1 = summary('s1', 'capped', T1); + await writeWire(home!, 's1', 'main', [ + userLine('cap-target one', T1), + userLine('cap-target two', T1 + 100), + userLine('cap-target three', T1 + 200), + userLine('cap-target four', T1 + 300), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + service.literalCandidateCap = 2; + + const page = await service.search({ query: 'cap-target', mode: 'literal' }); + expect(page.items.length).toBe(2); + expect(page.incomplete).toBe('candidate_cap'); + // Every returned item is still a confirmed substring hit. + expect(page.items.every((h) => h.snippet.includes('cap-target'))).toBe(true); + + // terms mode never reports incompleteness. + const terms = await service.search({ query: 'cap-target' }); + expect(terms.incomplete).toBeUndefined(); + }); + + it('paginates and rejects page tokens after a mode change', async () => { + const s1 = summary('s1', 'paging', T1); + await writeWire(home!, 's1', 'main', [ + userLine('page-target one', T1), + userLine('page-target two', T1 + 100), + userLine('page-target three', T1 + 200), + ]); + const service = track(makeService(home!, staticIndex([s1]))); + await service.reindex(); + + const page1 = await service.search({ query: 'page-target', mode: 'literal', pageSize: 2 }); + expect(page1.items.length).toBe(2); + expect(page1.hasMore).toBe(true); + const page2 = await service.search({ + query: 'page-target', + mode: 'literal', + pageSize: 2, + pageToken: page1.pageToken, + }); + expect(page2.items.length).toBe(1); + expect(page2.hasMore).toBe(false); + const times = [...page1.items, ...page2.items].map((h) => h.time); + expect(new Set(times).size).toBe(3); + + // The token fingerprints the mode: a literal token fails under terms… + await expect( + service.search({ query: 'page-target', pageToken: page1.pageToken }), + ).rejects.toMatchObject({ reason: 'invalid_page_token' }); + // …and a terms token fails under literal. + const termsPage = await service.search({ query: 'page-target', pageSize: 2 }); + await expect( + service.search({ query: 'page-target', mode: 'literal', pageToken: termsPage.pageToken }), + ).rejects.toMatchObject({ reason: 'invalid_page_token' }); + }); + + it('keeps terms mode untouched when the query contains symbols', async () => { + const service = await literalFixture(); + // Default mode tokenizes 'C++' to the word 'c' — behavior unchanged. + const terms = await service.search({ query: 'C++' }); + expect(terms.items.length).toBeGreaterThan(0); + expect(terms.incomplete).toBeUndefined(); + }); + }); + + // -- live route (in-memory transcript scan) ------------------------------------ + + describe('live route', () => { + /** Session-index stub whose `get` resolves the fixture summaries. */ + function gettableIndex(summaries: SessionSummary[]): ISessionIndex { + const byId = new Map(summaries.map((s) => [s.id, s])); + return { + _serviceBrand: undefined, + list: async () => ({ items: summaries, nextCursor: undefined }), + get: async (id) => byId.get(id), + countActive: async () => summaries.length, + }; + } + + interface LiveSourceCalls { + whenReady: string[]; + ensureAgentHistory: [string, string][]; + } + + function fakeLiveSource( + stores: Map, + calls?: LiveSourceCalls, + ): LiveTranscriptSource { + return { + forSessionLive: (sessionId) => stores.get(sessionId), + whenReady: async (sessionId) => { + calls?.whenReady.push(sessionId); + }, + ensureAgentHistory: async (sessionId, agentId) => { + calls?.ensureAgentHistory.push([sessionId, agentId]); + }, + }; + } + + /** + * A real TranscriptStore mirroring the wire fixture of the parity test: + * turn 0 with a user prompt at T1 and one assistant text frame in step + * `t0.1` at T2, plus a thinking frame and a tool frame that must NOT be + * searchable. + */ + function makeLiveStore(sessionId: string): TranscriptStore { + const store = new TranscriptStore(sessionId); + store.ensureAgent('main', { agentId: 'main', type: 'main' }); + store.getAgent('main')!.apply([ + { + op: 'turn.upsert', + turn: { + kind: 'turn', + turnId: 't0', + ordinal: 0, + state: 'completed', + origin: { kind: 'user' }, + prompt: '帮我看看苹果怎么挑', + startedAt: new Date(T1).toISOString(), + }, + }, + { + op: 'step.upsert', + turnId: 't0', + step: { + kind: 'step', + stepId: 't0.1', + turnId: 't0', + ordinal: 1, + state: 'completed', + startedAt: new Date(T2).toISOString(), + }, + }, + { + op: 'frame.upsert', + turnId: 't0', + stepId: 't0.1', + frame: { kind: 'thinking', frameId: 't0.1.f1', text: '苹果 thinking 不可见' }, + }, + { + op: 'frame.upsert', + turnId: 't0', + stepId: 't0.1', + frame: { + kind: 'tool', + frameId: 't0.1.f2', + toolCallId: 'call-1', + name: 'Read', + state: 'done', + }, + }, + { + op: 'frame.upsert', + turnId: 't0', + stepId: 't0.1', + frame: { + kind: 'text', + frameId: 't0.1.f3', + role: 'assistant', + text: '苹果要挑红富士。', + }, + }, + ]); + return store; + } + + /** + * Generic turn builder for the terms-mode and edge-case tests: one turn + * (optional prompt, optional state) whose steps carry only assistant text + * frames. + */ + function addLiveTurn( + store: TranscriptStore, + agentId: string, + turn: { + ordinal: number; + startedAt: number; + prompt?: string; + state?: 'running' | 'completed'; + steps?: readonly { + stepId: string; + startedAt?: number; + endedAt?: number; + state?: 'running' | 'completed'; + texts?: readonly string[]; + }[]; + }, + ): void { + const turnId = `t${turn.ordinal}`; + const ops: TranscriptOperation[] = [ + { + op: 'turn.upsert', + turn: { + kind: 'turn', + turnId, + ordinal: turn.ordinal, + state: turn.state ?? 'completed', + origin: { kind: 'user' }, + prompt: turn.prompt, + startedAt: new Date(turn.startedAt).toISOString(), + }, + }, + ]; + for (const step of turn.steps ?? []) { + ops.push({ + op: 'step.upsert', + turnId, + step: { + kind: 'step', + stepId: step.stepId, + turnId, + ordinal: Number(step.stepId.split('.')[1] ?? 0), + state: step.state ?? 'completed', + startedAt: + step.startedAt !== undefined ? new Date(step.startedAt).toISOString() : undefined, + endedAt: step.endedAt !== undefined ? new Date(step.endedAt).toISOString() : undefined, + }, + }); + (step.texts ?? []).forEach((text, i) => { + ops.push({ + op: 'frame.upsert', + turnId, + stepId: step.stepId, + frame: { + kind: 'text', + frameId: `${step.stepId}.f${i}`, + role: 'assistant', + text, + }, + }); + }); + } + store.getAgent(agentId)!.apply(ops); + } + + it('serves container-scoped literal queries from the live transcript store', async () => { + const s1 = summary('s1', '苹果标题', T1); + const stores = new Map([['s1', makeLiveStore('s1')]]); + const calls: LiveSourceCalls = { whenReady: [], ensureAgentHistory: [] }; + const service = track(makeService(home!, gettableIndex([s1]))); + service.setLiveTranscriptSource(fakeLiveSource(stores, calls)); + + const page = await service.search({ + query: '苹果', + mode: 'literal', + container: { sessionId: 's1' }, + }); + expect(page.source).toBe('live'); + // 1 user doc + 1 assistant doc + 1 title doc were scanned. + expect(page.indexState).toEqual({ + state: 'ready', + indexedSessions: 1, + totalSessions: 1, + documents: 3, + }); + // Backfill gates ran for the session and its whole roster. + expect(calls.whenReady).toEqual(['s1']); + expect(calls.ensureAgentHistory).toEqual([['s1', 'main']]); + + const user = page.items.find((h) => h.role === 'user'); + expect(user).toBeDefined(); + expect(user!.sessionId).toBe('s1'); + expect(user!.workspaceId).toBe(WS); + expect(user!.sessionTitle).toBe('苹果标题'); + expect(user!.agentId).toBe('main'); + expect(user!.turn).toBe(0); + expect(user!.stepId).toBeUndefined(); + expect(user!.time).toBe(T1); + expect(user!.snippet).toContain('苹果'); + + const assistant = page.items.find((h) => h.role === 'assistant'); + expect(assistant).toBeDefined(); + expect(assistant!.turn).toBe(0); + expect(assistant!.stepId).toBe('t0.1'); + expect(assistant!.time).toBe(T2); + + const title = page.items.find((h) => h.role === 'title'); + expect(title).toBeDefined(); + expect(title!.snippet).toBe('苹果标题'); + + // Thinking and tool frames are not searchable. + const thinking = await service.search({ + query: '不可见', + mode: 'literal', + container: { sessionId: 's1' }, + }); + expect(thinking.items).toEqual([]); + }); + + it('accepts single-character literal queries on the live route', async () => { + // The <2-character gate is an n-gram index constraint; the live route's + // in-memory scan has no such limit. + const s1 = summary('s1', '苹果标题', T1); + const service = track(makeService(home!, gettableIndex([s1]))); + service.setLiveTranscriptSource(fakeLiveSource(new Map([['s1', makeLiveStore('s1')]]))); + + const page = await service.search({ + query: '苹', + mode: 'literal', + container: { sessionId: 's1' }, + }); + expect(page.source).toBe('live'); + // user prompt + assistant frame + title all contain '苹'. + expect(page.items.length).toBe(3); + expect(page.items.map((h) => h.role).sort()).toEqual(['assistant', 'title', 'user']); + + // The index route keeps rejecting the same 1-character query. + await expect(service.search({ query: '苹', mode: 'literal' })).rejects.toMatchObject({ + reason: 'invalid_query', + }); + }); + + it('falls back to the index route when no source is wired or the session is not live', async () => { + const s1 = summary('s1', 'fallback', T1); + await writeWire(home!, 's1', 'main', [userLine('苹果 from index', T1)]); + const service = track(makeService(home!, gettableIndex([s1]))); + await service.reindex(); + + // No source wired at all → always the index route. + const unwired = await service.search({ + query: '苹果', + mode: 'literal', + container: { sessionId: 's1' }, + }); + expect(unwired.source).toBe('index'); + expect(unwired.items.length).toBe(1); + + // Source wired but the session is not in memory → still the index route. + service.setLiveTranscriptSource(fakeLiveSource(new Map())); + const notLive = await service.search({ + query: '苹果', + mode: 'literal', + container: { sessionId: 's1' }, + }); + expect(notLive.source).toBe('index'); + expect(notLive.items.length).toBe(1); + }); + + it('serves terms queries from the live store and orders hits by tf score', async () => { + const s1 = summary('s1', '无关标题', T1); + const store = new TranscriptStore('s1'); + store.ensureAgent('main', { agentId: 'main', type: 'main' }); + addLiveTurn(store, 'main', { ordinal: 0, startedAt: T1, prompt: '苹果怎么挑' }); + addLiveTurn(store, 'main', { ordinal: 1, startedAt: T2, prompt: '苹果苹果都要' }); + const service = track(makeService(home!, gettableIndex([s1]))); + service.setLiveTranscriptSource(fakeLiveSource(new Map([['s1', store]]))); + + const page = await service.search({ query: '苹果', container: { sessionId: 's1' } }); + expect(page.source).toBe('live'); + expect(page.items.length).toBe(2); + // The doc repeating the term scores higher (Σ log(1 + tf)). + expect(page.items[0]!.time).toBe(T2); + expect(page.items[1]!.time).toBe(T1); + expect(page.items[0]!.score).toBeGreaterThan(page.items[1]!.score); + expect(page.items[1]!.score).toBeGreaterThan(0); + + // Duplicate query terms collapse to one (same as `TextIndex.search`). + const dup = await service.search({ query: '苹果 苹果', container: { sessionId: 's1' } }); + expect(dup.items.map((h) => h.time)).toEqual(page.items.map((h) => h.time)); + }); + + it('returns matching terms result sets on both routes for equivalent data', async () => { + const s1 = summary('s1', '无关标题', T1); + await writeWire(home!, 's1', 'main', [ + userLine('帮我看看苹果怎么挑', T1), + stepBeginLine('u1', 1, T1 + 100), + assistantStepLine('苹果要挑红富士。', 'u1', T2), + ]); + const stores = new Map([['s1', makeLiveStore('s1')]]); + const service = track(makeService(home!, gettableIndex([s1]))); + await service.reindex(); + service.setLiveTranscriptSource(fakeLiveSource(stores)); + + const query = { query: '苹果', container: { sessionId: 's1' } }; + const live = await service.search(query); + expect(live.source).toBe('live'); + expect(live.items.length).toBe(2); + + stores.delete('s1'); // the same query now falls to the index route + const index = await service.search(query); + expect(index.source).toBe('index'); + expect(index.items.length).toBe(2); + + // Scores are not comparable across routes (the live route has no + // corpus-wide IDF); compare hit identity, plus each route's own + // score-ordering property. + const identity = (page: typeof live) => + page.items + .map((h) => ({ + sessionId: h.sessionId, + agentId: h.agentId, + role: h.role, + time: h.time, + turn: h.turn, + stepId: h.stepId, + })) + .sort((a, b) => a.time - b.time); + expect(identity(live)).toEqual(identity(index)); + for (const page of [live, index]) { + for (let i = 1; i < page.items.length; i++) { + expect(page.items[i - 1]!.score).toBeGreaterThanOrEqual(page.items[i]!.score); + } + } + }); + + it('applies role, time, agent and sort filters on the live route', async () => { + const s1 = summary('s1', '', T1); + const store = new TranscriptStore('s1'); + store.ensureAgent('main', { agentId: 'main', type: 'main' }); + store.ensureAgent('sub', { agentId: 'sub', type: 'sub' }); + addLiveTurn(store, 'main', { + ordinal: 0, + startedAt: T1, + prompt: '苹果 user question', + steps: [{ stepId: 't0.1', startedAt: T2, texts: ['苹果 assistant answer'] }], + }); + addLiveTurn(store, 'sub', { ordinal: 0, startedAt: T3, prompt: '苹果 subagent prompt' }); + const service = track(makeService(home!, gettableIndex([s1]))); + service.setLiveTranscriptSource(fakeLiveSource(new Map([['s1', store]]))); + + const base = { query: '苹果', container: { sessionId: 's1' } }; + const users = await service.search({ ...base, role: 'user' }); + expect(users.items.length).toBe(2); + expect(users.items.every((h) => h.role === 'user')).toBe(true); + + const assistants = await service.search({ ...base, role: 'assistant' }); + expect(assistants.items.length).toBe(1); + expect(assistants.items[0]!.stepId).toBe('t0.1'); + + const ranged = await service.search({ ...base, startTime: T2, endTime: T2 }); + expect(ranged.items.length).toBe(1); + expect(ranged.items[0]!.time).toBe(T2); + + const subOnly = await service.search({ + ...base, + container: { sessionId: 's1', agentId: 'sub' }, + }); + expect(subOnly.items.length).toBe(1); + expect(subOnly.items[0]!.agentId).toBe('sub'); + + const asc = await service.search({ ...base, sort: 'time_asc' }); + expect(asc.items.map((h) => h.time)).toEqual([T1, T2, T3]); + const desc = await service.search({ ...base, sort: 'time_desc' }); + expect(desc.items.map((h) => h.time)).toEqual([T3, T2, T1]); + }); + + it('hits the session title doc on the live route (terms mode)', async () => { + const s1 = summary('s1', '苹果标题', T1); + const store = new TranscriptStore('s1'); + store.ensureAgent('main', { agentId: 'main', type: 'main' }); + addLiveTurn(store, 'main', { ordinal: 0, startedAt: T1, prompt: '随便聊聊' }); + const service = track(makeService(home!, gettableIndex([s1]))); + service.setLiveTranscriptSource(fakeLiveSource(new Map([['s1', store]]))); + + const page = await service.search({ query: '苹果', container: { sessionId: 's1' } }); + expect(page.source).toBe('live'); + expect(page.items.length).toBe(1); + const hit = page.items[0]!; + expect(hit.role).toBe('title'); + expect(hit.agentId).toBe(''); + expect(hit.sessionId).toBe('s1'); + expect(hit.snippet).toBe('苹果标题'); + }); + + it('scopes container.agentId queries to that agent only', async () => { + const s1 = summary('s1', '', T1); + const store = new TranscriptStore('s1'); + store.ensureAgent('main', { agentId: 'main', type: 'main' }); + store.ensureAgent('sub', { agentId: 'sub', type: 'sub' }); + addLiveTurn(store, 'main', { ordinal: 0, startedAt: T1, prompt: '苹果 from main' }); + addLiveTurn(store, 'sub', { ordinal: 0, startedAt: T2, prompt: '苹果 from sub' }); + const calls: LiveSourceCalls = { whenReady: [], ensureAgentHistory: [] }; + const service = track(makeService(home!, gettableIndex([s1]))); + service.setLiveTranscriptSource(fakeLiveSource(new Map([['s1', store]]), calls)); + + const page = await service.search({ + query: '苹果', + container: { sessionId: 's1', agentId: 'sub' }, + }); + expect(page.source).toBe('live'); + // Backfill ran for the requested agent only, and only its docs scanned. + expect(calls.whenReady).toEqual(['s1']); + expect(calls.ensureAgentHistory).toEqual([['s1', 'sub']]); + expect(page.indexState.documents).toBe(1); + expect(page.items.length).toBe(1); + expect(page.items[0]!.agentId).toBe('sub'); + }); + + it('handles an empty roster, prompt-less turns and empty text frames', async () => { + const s1 = summary('s1', '', T1); + const calls: LiveSourceCalls = { whenReady: [], ensureAgentHistory: [] }; + const service = track(makeService(home!, gettableIndex([s1]))); + const stores = new Map([['s1', new TranscriptStore('s1')]]); + service.setLiveTranscriptSource(fakeLiveSource(stores, calls)); + + // Empty roster: no agents at all — nothing to backfill or scan. + const empty = await service.search({ query: '苹果', container: { sessionId: 's1' } }); + expect(empty.source).toBe('live'); + expect(empty.items).toEqual([]); + expect(empty.indexState.documents).toBe(0); + expect(calls.whenReady).toEqual(['s1']); + expect(calls.ensureAgentHistory).toEqual([]); + + // A turn without a prompt and steps with empty/whitespace-only text + // frames produce no documents and do not break the scan. + const store = new TranscriptStore('s1'); + store.ensureAgent('main', { agentId: 'main', type: 'main' }); + addLiveTurn(store, 'main', { + ordinal: 0, + startedAt: T1, + steps: [{ stepId: 't0.1', startedAt: T1, texts: ['', ' '] }], + }); + addLiveTurn(store, 'main', { ordinal: 1, startedAt: T2, prompt: '苹果 survives' }); + stores.set('s1', store); + const page = await service.search({ query: '苹果', container: { sessionId: 's1' } }); + expect(page.items.length).toBe(1); + expect(page.items[0]!.role).toBe('user'); + expect(page.indexState.documents).toBe(1); + }); + + it('does not fall back to the index when the live route fails', async () => { + const s1 = summary('s1', 'boom', T1); + await writeWire(home!, 's1', 'main', [userLine('苹果 from index', T1)]); + const service = track(makeService(home!, gettableIndex([s1]))); + await service.reindex(); + + const store = makeLiveStore('s1'); + service.setLiveTranscriptSource({ + forSessionLive: (sessionId) => (sessionId === 's1' ? store : undefined), + whenReady: async () => { + throw new Error('backfill boom'); + }, + ensureAgentHistory: async () => {}, + }); + + // The index has a hit for this query, but a live-route failure is a + // real error and must surface instead of falling back. + await expect( + service.search({ query: '苹果', container: { sessionId: 's1' } }), + ).rejects.toThrow('backfill boom'); + }); + + it('searches the partial text of an in-flight turn', async () => { + const s1 = summary('s1', '', T1); + const store = new TranscriptStore('s1'); + store.ensureAgent('main', { agentId: 'main', type: 'main' }); + addLiveTurn(store, 'main', { + ordinal: 0, + startedAt: T1, + state: 'running', + prompt: '苹果 running prompt', + steps: [{ stepId: 't0.1', startedAt: T2, state: 'running', texts: ['苹果 partial answer'] }], + }); + const service = track(makeService(home!, gettableIndex([s1]))); + service.setLiveTranscriptSource(fakeLiveSource(new Map([['s1', store]]))); + + const page = await service.search({ query: '苹果', container: { sessionId: 's1' } }); + expect(page.source).toBe('live'); + expect(page.items.length).toBe(2); + expect(page.items.some((h) => h.role === 'user' && h.snippet.includes('running'))).toBe(true); + expect( + page.items.some((h) => h.role === 'assistant' && h.snippet.includes('partial')), + ).toBe(true); + }); + + it('matches nothing for a query that tokenizes to zero terms', async () => { + const s1 = summary('s1', '', T1); + const stores = new Map([['s1', makeLiveStore('s1')]]); + const service = track(makeService(home!, gettableIndex([s1]))); + service.setLiveTranscriptSource(fakeLiveSource(stores)); + + const page = await service.search({ query: '+++', container: { sessionId: 's1' } }); + expect(page.source).toBe('live'); + expect(page.items).toEqual([]); + expect(page.hasMore).toBe(false); + }); + + it('rejects page tokens across a route flip (the fingerprint covers the source)', async () => { + const s1 = summary('s1', 'flip', T1); + await writeWire(home!, 's1', 'main', [ + userLine('苹果 index one', T1), + assistantLine('苹果 index two', T2), + ]); + const stores = new Map([['s1', makeLiveStore('s1')]]); + const service = track(makeService(home!, gettableIndex([s1]))); + await service.reindex(); + service.setLiveTranscriptSource(fakeLiveSource(stores)); + + // Live route, page 1 of 2 (title 'flip' does not contain the query). + const query = { + query: '苹果', + mode: 'literal' as const, + container: { sessionId: 's1' }, + pageSize: 1, + }; + const livePage = await service.search(query); + expect(livePage.source).toBe('live'); + expect(livePage.hasMore).toBe(true); + + // The session closes mid-pagination: the same query now takes the index + // route and must reject the live-issued token. + stores.delete('s1'); + await expect( + service.search({ ...query, pageToken: livePage.pageToken }), + ).rejects.toMatchObject({ reason: 'invalid_page_token' }); + + // And the reverse: an index-issued token fails once the session is live. + const indexPage = await service.search(query); + expect(indexPage.source).toBe('index'); + expect(indexPage.hasMore).toBe(true); + stores.set('s1', makeLiveStore('s1')); + await expect( + service.search({ ...query, pageToken: indexPage.pageToken }), + ).rejects.toMatchObject({ reason: 'invalid_page_token' }); + }); + + it('returns identical literal results on both routes for equivalent data', async () => { + const s1 = summary('s1', '无关标题', T1); + await writeWire(home!, 's1', 'main', [ + userLine('帮我看看苹果怎么挑', T1), + stepBeginLine('u1', 1, T1 + 100), + assistantStepLine('苹果要挑红富士。', 'u1', T2), + ]); + const stores = new Map([['s1', makeLiveStore('s1')]]); + const service = track(makeService(home!, gettableIndex([s1]))); + await service.reindex(); + service.setLiveTranscriptSource(fakeLiveSource(stores)); + + const query = { query: '苹果', mode: 'literal' as const, container: { sessionId: 's1' } }; + const live = await service.search(query); + expect(live.source).toBe('live'); + + stores.delete('s1'); // the same query now falls to the index route + const index = await service.search(query); + expect(index.source).toBe('index'); + + const project = (page: typeof live) => + page.items.map((h) => ({ + sessionId: h.sessionId, + workspaceId: h.workspaceId, + sessionTitle: h.sessionTitle, + agentId: h.agentId, + role: h.role, + snippet: h.snippet, + time: h.time, + turn: h.turn, + stepId: h.stepId, + score: h.score, + })); + expect(project(live)).toEqual(project(index)); + }); + }); +}); diff --git a/packages/kap-server/test/search/snippet.test.ts b/packages/kap-server/test/search/snippet.test.ts new file mode 100644 index 0000000000..7b4dc80eee --- /dev/null +++ b/packages/kap-server/test/search/snippet.test.ts @@ -0,0 +1,44 @@ +import { describe, expect, it } from 'vitest'; + +import { makeSnippet } from '../../src/search/snippet'; + +describe('makeSnippet', () => { + it('centers the window on the first hit term', () => { + const text = `${'a'.repeat(200)} needle ${'b'.repeat(200)}`; + const snippet = makeSnippet(text, 'needle', 20); + expect(snippet).toContain('needle'); + expect(snippet.startsWith('…')).toBe(true); + expect(snippet.endsWith('…')).toBe(true); + expect(snippet.length).toBeLessThan(text.length); + }); + + it('matches case-insensitively', () => { + expect(makeSnippet('Hello World', 'hello')).toBe('Hello World'); + }); + + it('matches CJK terms', () => { + const snippet = makeSnippet('我们需要重构全局搜索模块以支持中文', '搜索'); + expect(snippet).toContain('搜索'); + }); + + it('uses the earliest occurrence across multiple terms', () => { + const text = `${'x'.repeat(100)} beta ${'x'.repeat(100)} alpha`; + const snippet = makeSnippet(text, 'alpha beta', 10); + expect(snippet).toContain('beta'); + }); + + it('falls back to the text head when no term matches', () => { + const text = `start ${'y'.repeat(500)}`; + const snippet = makeSnippet(text, 'absent', 40); + expect(snippet.startsWith('start')).toBe(true); + expect(snippet.endsWith('…')).toBe(true); + }); + + it('does not add ellipses when the whole text fits', () => { + expect(makeSnippet('short text', 'text')).toBe('short text'); + }); + + it('collapses whitespace runs', () => { + expect(makeSnippet('a\n\nb c\tneedle', 'needle', 80)).toBe('a b c needle'); + }); +}); diff --git a/packages/kap-server/test/search/wireExtract.test.ts b/packages/kap-server/test/search/wireExtract.test.ts new file mode 100644 index 0000000000..088dfff1d3 --- /dev/null +++ b/packages/kap-server/test/search/wireExtract.test.ts @@ -0,0 +1,413 @@ +import { describe, expect, it } from 'vitest'; + +import { analyzeWireLine, extractFromWireLine } from '../../src/search/wireExtract'; + +const line = (value: unknown): string => JSON.stringify(value); + +function userRecord(text: string, time: number, origin?: unknown): string { + return line({ + type: 'context.append_message', + time, + message: { role: 'user', content: [{ type: 'text', text }], origin }, + }); +} + +describe('extractFromWireLine', () => { + it('extracts user text messages', () => { + const out = extractFromWireLine( + userRecord('帮我重构搜索模块', 1_700_000_000_000, { kind: 'user' }), + ); + expect(out).toEqual([{ role: 'user', text: '帮我重构搜索模块', time: 1_700_000_000_000 }]); + }); + + it('keeps user messages without an origin', () => { + const out = extractFromWireLine(userRecord('hello world', 1_700_000_000_000)); + expect(out).toHaveLength(1); + expect(out[0]?.role).toBe('user'); + }); + + it('joins multiple text parts of one user message', () => { + const out = extractFromWireLine( + line({ + type: 'context.append_message', + time: 1_700_000_000_000, + message: { + role: 'user', + content: [ + { type: 'text', text: 'first part ' }, + { type: 'image', url: 'data:...' }, + { type: 'text', text: 'second part' }, + ], + origin: { kind: 'user' }, + }, + }), + ); + expect(out).toEqual([ + { role: 'user', text: 'first part second part', time: 1_700_000_000_000 }, + ]); + }); + + it.each(['injection', 'system_trigger', 'retry', 'compaction_summary', 'plugin_command'])( + 'filters out origin kind %s', + (kind) => { + expect( + extractFromWireLine(userRecord('not user input', 1_700_000_000_000, { kind })), + ).toEqual([]); + }, + ); + + it.each(['skill_activation', 'plugin_command'])( + 'keeps %s messages the user typed as a slash command', + (kind) => { + const out = extractFromWireLine( + userRecord('/commit 整理提交', 1_700_000_000_000, { + kind, + trigger: 'user-slash', + skillName: 'commit', + }), + ); + expect(out).toEqual([{ role: 'user', text: '/commit 整理提交', time: 1_700_000_000_000 }]); + }, + ); + + it.each(['model-tool', 'nested-skill'])( + 'filters out skill activations triggered by %s', + (trigger) => { + expect( + extractFromWireLine( + userRecord('skill body', 1_700_000_000_000, { kind: 'skill_activation', trigger }), + ), + ).toEqual([]); + }, + ); + + it('ignores non-user append_message roles', () => { + const out = extractFromWireLine( + line({ + type: 'context.append_message', + time: 1_700_000_000_000, + message: { role: 'assistant', content: [{ type: 'text', text: 'assistant as message' }] }, + }), + ); + expect(out).toEqual([]); + }); + + it('extracts assistant text content parts from loop events', () => { + const out = extractFromWireLine( + line({ + type: 'context.append_loop_event', + time: 1_700_000_000_000, + event: { type: 'content.part', part: { type: 'text', text: 'Here is the refactor plan.' } }, + }), + ); + expect(out).toEqual([ + { role: 'assistant', text: 'Here is the refactor plan.', time: 1_700_000_000_000 }, + ]); + }); + + it('ignores thinking parts and tool events', () => { + expect( + extractFromWireLine( + line({ + type: 'context.append_loop_event', + time: 1, + event: { type: 'content.part', part: { type: 'thinking', thinking: 'hmm' } }, + }), + ), + ).toEqual([]); + expect( + extractFromWireLine( + line({ + type: 'context.append_loop_event', + time: 1, + event: { type: 'tool.call', name: 'Bash', args: { command: 'ls' } }, + }), + ), + ).toEqual([]); + expect( + extractFromWireLine( + line({ + type: 'context.append_loop_event', + time: 1, + event: { type: 'tool.result', result: { output: 'file list' } }, + }), + ), + ).toEqual([]); + }); + + it('ignores other record types, blank lines and unparseable JSON', () => { + expect(extractFromWireLine(line({ type: 'metadata', protocol_version: '1.4' }))).toEqual([]); + expect(extractFromWireLine(line({ type: 'turn_begin', time: 1 }))).toEqual([]); + expect(extractFromWireLine('')).toEqual([]); + expect(extractFromWireLine(' ')).toEqual([]); + expect(extractFromWireLine('{not json')).toEqual([]); + expect(extractFromWireLine('[1,2,3]')).toEqual([]); + }); + + it('drops messages whose text is empty after trimming', () => { + expect(extractFromWireLine(userRecord(' ', 1_700_000_000_000, { kind: 'user' }))).toEqual([]); + }); + + it('normalizes second-based timestamps to epoch ms', () => { + const out = extractFromWireLine(userRecord('seconds time', 1_700_000_000, { kind: 'user' })); + expect(out[0]?.time).toBe(1_700_000_000_000); + }); + + it('omits time when the record has no usable timestamp', () => { + const out = extractFromWireLine( + line({ + type: 'context.append_message', + message: { role: 'user', content: [{ type: 'text', text: 'no time' }] }, + }), + ); + expect(out[0]?.time).toBeUndefined(); + }); +}); + +describe('analyzeWireLine turn effects', () => { + const turnOf = (jsonl: string) => analyzeWireLine(jsonl).turn; + + it('user messages without an origin open an anchor turn', () => { + expect(turnOf(userRecord('hi', 1))).toEqual({ kind: 'open', anchor: true }); + expect(turnOf(userRecord('hi', 1, { kind: 'user' }))).toEqual({ kind: 'open', anchor: true }); + }); + + it('text-less user messages still open a turn (counting is index-independent)', () => { + expect( + turnOf( + line({ + type: 'context.append_message', + message: { + role: 'user', + content: [{ type: 'image', source: { kind: 'url', url: 'x' } }], + }, + }), + ), + ).toEqual({ kind: 'open', anchor: true }); + }); + + it('hidden origins do not open turns, except turn-opening system triggers', () => { + expect(turnOf(userRecord('x', 1, { kind: 'injection', variant: 'v' }))).toEqual({ + kind: 'none', + }); + expect(turnOf(userRecord('x', 1, { kind: 'retry' }))).toEqual({ kind: 'none' }); + expect(turnOf(userRecord('x', 1, { kind: 'system_trigger', name: 'reminder' }))).toEqual({ + kind: 'none', + }); + expect( + turnOf(userRecord('x', 1, { kind: 'system_trigger', name: 'goal_continuation' })), + ).toEqual({ kind: 'open', anchor: false }); + expect(turnOf(userRecord('x', 1, { kind: 'system_trigger', name: 'subagent' }))).toEqual({ + kind: 'open', + anchor: false, + }); + }); + + it('marker origins open a turn only for user-slash prompts', () => { + expect( + turnOf(userRecord('/s', 1, { kind: 'skill_activation', trigger: 'user-slash' })), + ).toEqual({ kind: 'open', anchor: true }); + expect(turnOf(userRecord('/s', 1, { kind: 'plugin_command', trigger: 'user-slash' }))).toEqual({ + kind: 'open', + anchor: true, + }); + expect(turnOf(userRecord('s', 1, { kind: 'skill_activation', trigger: 'model-tool' }))).toEqual( + { + kind: 'none', + }, + ); + expect(turnOf(userRecord('sum', 1, { kind: 'compaction_summary' }))).toEqual({ kind: 'none' }); + }); + + it('cron / task / hook / shell_command origins open non-anchor turns', () => { + expect(turnOf(userRecord('cron', 1, { kind: 'cron_job', jobId: 'j' }))).toEqual({ + kind: 'open', + anchor: false, + }); + expect(turnOf(userRecord('t', 1, { kind: 'task', taskId: 't1' }))).toEqual({ + kind: 'open', + anchor: false, + }); + expect(turnOf(userRecord('h', 1, { kind: 'hook_result', event: 'stop' }))).toEqual({ + kind: 'open', + anchor: false, + }); + expect(turnOf(userRecord('!', 1, { kind: 'shell_command', phase: 'input' }))).toEqual({ + kind: 'open', + anchor: false, + }); + }); + + it('assistant-surviving loop events ensure a turn; tool results and bare steps do not', () => { + expect( + turnOf( + line({ + type: 'context.append_message', + message: { role: 'assistant', content: [{ type: 'text', text: 'a' }] }, + }), + ), + ).toEqual({ kind: 'ensure' }); + expect( + turnOf( + line({ + type: 'context.append_loop_event', + event: { type: 'content.part', part: { type: 'text', text: 'a' } }, + }), + ), + ).toEqual({ kind: 'ensure' }); + expect( + turnOf( + line({ + type: 'context.append_loop_event', + event: { type: 'tool.call', name: 'Bash', args: {} }, + }), + ), + ).toEqual({ kind: 'ensure' }); + // A tool result folds into a tool message — never opens a turn. + expect( + turnOf( + line({ + type: 'context.append_loop_event', + event: { type: 'tool.result', result: { output: 'ok' } }, + }), + ), + ).toEqual({ kind: 'none' }); + // Bare step markers and vacuous content: the folded assistant is dropped. + expect( + turnOf(line({ type: 'context.append_loop_event', event: { type: 'step.begin' } })), + ).toEqual({ kind: 'none' }); + expect( + turnOf( + line({ + type: 'context.append_loop_event', + event: { type: 'content.part', part: { type: 'text', text: ' ' } }, + }), + ), + ).toEqual({ kind: 'none' }); + // Thinking parts follow the same vacuous rule: empty unsigned thinking is + // dropped at step.end; non-empty or signed thinking survives. + expect( + turnOf( + line({ + type: 'context.append_loop_event', + event: { type: 'content.part', part: { type: 'think', think: 'reasoning' } }, + }), + ), + ).toEqual({ kind: 'ensure' }); + expect( + turnOf( + line({ + type: 'context.append_loop_event', + event: { type: 'content.part', part: { type: 'think', think: ' ' } }, + }), + ), + ).toEqual({ kind: 'none' }); + expect( + turnOf( + line({ + type: 'context.append_loop_event', + event: { type: 'content.part', part: { type: 'think', think: '', encrypted: 'sig' } }, + }), + ), + ).toEqual({ kind: 'ensure' }); + }); + + it('system-role and tool-role messages have no effect', () => { + expect( + turnOf(line({ type: 'context.append_message', message: { role: 'system', content: [] } })), + ).toEqual({ kind: 'none' }); + expect( + turnOf( + line({ + type: 'context.append_message', + message: { role: 'tool', content: [{ type: 'text', text: 'out' }] }, + }), + ), + ).toEqual({ kind: 'none' }); + }); + + it('compaction and clear do NOT renumber; undo carries its count; invalid undo is none', () => { + // The transcript's cold replay keeps full history and groupTurns numbers + // it continuously, so these records have no turn effect. + expect( + turnOf(line({ type: 'context.apply_compaction', summary: 's', compactedCount: 2 })), + ).toEqual({ kind: 'none' }); + expect(turnOf(line({ type: 'context.clear' }))).toEqual({ kind: 'none' }); + expect(turnOf(line({ type: 'context.undo', count: 2 }))).toEqual({ kind: 'undo', count: 2 }); + expect(turnOf(line({ type: 'context.undo', count: 0 }))).toEqual({ kind: 'none' }); + expect(turnOf(line({ type: 'context.undo' }))).toEqual({ kind: 'none' }); + }); + + it('other record types have no effect', () => { + expect(turnOf(line({ type: 'metadata', protocol_version: '1.4' }))).toEqual({ kind: 'none' }); + expect(analyzeWireLine('{not json').turn).toEqual({ kind: 'none' }); + }); +}); + +describe('analyzeWireLine step effects', () => { + const stepOf = (jsonl: string) => analyzeWireLine(jsonl).step; + + it('step.begin maps its uuid to the wire-carried ordinal', () => { + expect( + stepOf( + line({ + type: 'context.append_loop_event', + event: { type: 'step.begin', uuid: 'u1', turnId: '0', step: 2 }, + }), + ), + ).toEqual({ kind: 'begin', uuid: 'u1', ordinal: 2 }); + }); + + it('step.begin without a step field leaves the ordinal to the fallback counter', () => { + expect( + stepOf( + line({ type: 'context.append_loop_event', event: { type: 'step.begin', uuid: 'u1' } }), + ), + ).toEqual({ kind: 'begin', uuid: 'u1', ordinal: undefined }); + }); + + it('step.begin without a usable uuid or with an invalid ordinal degrades cleanly', () => { + expect( + stepOf(line({ type: 'context.append_loop_event', event: { type: 'step.begin' } })), + ).toEqual({ kind: 'none' }); + expect( + stepOf( + line({ + type: 'context.append_loop_event', + event: { type: 'step.begin', uuid: 'u1', step: -1 }, + }), + ), + ).toEqual({ kind: 'begin', uuid: 'u1', ordinal: undefined }); + }); + + it('step.end and other loop events have no step effect', () => { + expect( + stepOf(line({ type: 'context.append_loop_event', event: { type: 'step.end', uuid: 'u1' } })), + ).toEqual({ kind: 'none' }); + expect( + stepOf( + line({ + type: 'context.append_loop_event', + event: { type: 'tool.call', stepUuid: 'u1', name: 'Bash', args: {} }, + }), + ), + ).toEqual({ kind: 'none' }); + }); + + it('assistant text carries the content.part stepUuid; user text does not', () => { + const assistant = analyzeWireLine( + line({ + type: 'context.append_loop_event', + time: 1_700_000_000_000, + event: { type: 'content.part', stepUuid: 'u1', part: { type: 'text', text: 'plan' } }, + }), + ); + expect(assistant.messages).toEqual([ + { role: 'assistant', text: 'plan', time: 1_700_000_000_000, stepUuid: 'u1' }, + ]); + + const user = analyzeWireLine(userRecord('hi', 1)); + expect(user.messages[0]?.stepUuid).toBeUndefined(); + expect(user.step).toEqual({ kind: 'none' }); + }); +}); diff --git a/packages/minidb/src/index.ts b/packages/minidb/src/index.ts index a6ea101928..7458909712 100644 --- a/packages/minidb/src/index.ts +++ b/packages/minidb/src/index.ts @@ -17,7 +17,8 @@ import { recover, catchUpWal, frameToOps } from './recovery.js'; import { compact, shouldCompact } from './compaction.js'; import { IndexManager, UniqueViolationError } from './index-manager.js'; import { DtIndex } from './dt-index.js'; -import { TextIndex } from './text-index.js'; +import { TextIndex, type TextIndexOptions } from './text-index.js'; +import { createNgramTokenizer } from './trigram.js'; import { CompoundIndexManager } from './compound-index.js'; import { getPath, match, project } from './query.js'; import { LockFile, LockError } from './lockfile.js'; @@ -29,12 +30,16 @@ import type { IndexDef, IndexInfo } from './index-manager.js'; import type { CompoundIndexDef, CompoundIndexInfo } from './compound-index.js'; import type { DtRangeEntry } from './dt-index.js'; import type { RangeOptions } from './skiplist.js'; +import type { TextIndexTokenizerName } from './trigram.js'; export { UniqueViolationError } from './index-manager.js'; export { LockError } from './lockfile.js'; +export { normalizeLiteral, createNgramTokenizer } from './trigram.js'; +export { tokenize } from './text-index.js'; export type { RecoveryInfo } from './recovery.js'; export type { IndexDef, IndexInfo, IndexType } from './index-manager.js'; export type { CompoundIndexDef, CompoundIndexInfo } from './compound-index.js'; +export type { TextIndexTokenizerName } from './trigram.js'; // ClusterDb (the multi-process sharding layer) lives at the './cluster' // subpath export to keep this module free of import cycles. @@ -199,6 +204,34 @@ interface PreparedOp { valueDecoded: V | undefined; } +/** Persisted shape of one entry in `db.textindexes.json`. `tokenizer` is + * absent in definitions written before n-gram support existed, which means + * 'default'; it is also omitted for new default indexes so their definitions + * keep the legacy shape byte-for-byte. */ +interface TextIndexDef { + name: string; + fields: readonly string[] | null; + tokenizer?: TextIndexTokenizerName; +} + +/** Map a persisted tokenizer name to the TextIndex tokenizer pair. 'default' + * (or a legacy definition without the field) returns empty options, keeping + * the built-in tokenizer path untouched. The query side only diverges for + * 'ngram' (a length >= 3 query emits only its 3-grams); both sides share the + * same normalization, so candidates stay a superset of the true matches. */ +function textIndexTokenizers( + name: TextIndexTokenizerName | undefined, +): Pick { + if (name === undefined || name === 'default') return {}; + if (name === 'ngram') { + return { + tokenizer: createNgramTokenizer(), + queryTokenizer: createNgramTokenizer({ forQuery: true }), + }; + } + throw new RangeError(`unknown text index tokenizer: ${String(name)}`); +} + export class MiniDb { dir!: string; walPath!: string; @@ -213,7 +246,7 @@ export class MiniDb { readonly dt = new DtIndex(); readonly compound = new CompoundIndexManager(); private readonly text = new Map(); - private textDefs: { name: string; fields: readonly string[] | null }[] = []; + private textDefs: TextIndexDef[] = []; private codec!: ValueCodec; private codecName: ValueCodecName = 'buffer'; @@ -492,12 +525,13 @@ export class MiniDb { private async loadTextIndexDefinitions(): Promise { try { const raw = await fs.readFile(this.textIndexPath, 'utf8'); - this.textDefs = JSON.parse(raw) as { name: string; fields: readonly string[] | null }[]; + this.textDefs = JSON.parse(raw) as TextIndexDef[]; for (const d of this.textDefs) { this.text.set( d.name, new TextIndex({ fields: d.fields, + ...textIndexTokenizers(d.tokenizer), // A read-only opener must not write to a live writer's postings file; // it keeps the base postings in memory instead. postingsPath: this.readOnly ? undefined : this.textPostingsPath(d.name), @@ -1202,13 +1236,16 @@ export class MiniDb { // ---- full-text search --------------------------------------------------- - async createTextIndex(name: string, { fields }: { fields?: readonly string[] } = {}): Promise { + async createTextIndex( + name: string, + { fields, tokenizer }: { fields?: readonly string[]; tokenizer?: TextIndexTokenizerName } = {}, + ): Promise { this.ensureOpen(); this.ensureWritable(); if (this.codecName !== 'json') throw new Error('text indexes require valueCodec: "json"'); if (this.text.has(name)) throw new Error(`text index "${name}" already exists`); - const ti = new TextIndex({ fields, postingsPath: this.textPostingsPath(name) }); - const def = { name, fields: fields ?? null }; + const ti = new TextIndex({ fields, ...textIndexTokenizers(tokenizer), postingsPath: this.textPostingsPath(name) }); + const def: TextIndexDef = { name, fields: fields ?? null, tokenizer }; // Build BEFORE registering: a failed build must leave no phantom index // behind — a registered-but-unbuilt index would both poison every write // path that walks this.text and make a retry fail with "already exists". diff --git a/packages/minidb/src/text-index.ts b/packages/minidb/src/text-index.ts index bbe59d598f..5080840e1f 100644 --- a/packages/minidb/src/text-index.ts +++ b/packages/minidb/src/text-index.ts @@ -63,6 +63,15 @@ function stringLeaves(obj: unknown, acc: string[] = []): string[] { export interface TextIndexOptions { fields?: readonly string[] | null; + /** Custom tokenizer, applied to indexed documents (and to query text when + * no `queryTokenizer` is given). Defaults to the built-in word/CJK + * `tokenize`. */ + tokenizer?: (text: string) => string[]; + /** Custom tokenizer for query text in `search()`. Defaults to `tokenizer`. + * Only diverges for the n-gram index: the query side emits fewer, more + * selective terms (a length >= 3 query needs only its 3-grams), while the + * index side indexes both widths so every query shape can match. */ + queryTokenizer?: (text: string) => string[]; /** Path to the postings file. If omitted, the index keeps its base postings * in memory instead of on disk (used by read-only openers, which must not * write to a live writer's directory). */ @@ -86,6 +95,8 @@ const EMPTY_MAP: ReadonlyMap = new Map(); export class TextIndex { private readonly fields: readonly string[] | null; + private readonly tokenizer: (text: string) => string[]; + private readonly queryTokenizer: (text: string) => string[]; private readonly path: string | null; private readonly cacheTerms: number; @@ -117,6 +128,8 @@ export class TextIndex { constructor(opts: TextIndexOptions = {}) { this.fields = opts.fields ?? null; + this.tokenizer = opts.tokenizer ?? tokenize; + this.queryTokenizer = opts.queryTokenizer ?? this.tokenizer; this.path = opts.postingsPath ?? null; this.cacheTerms = opts.cacheTerms ?? 1024; if (!this.path) this.memBase = new Map(); @@ -168,7 +181,7 @@ export class TextIndex { const docID = newKeys.length; newKeys.push(key); newKeyToId.set(key, docID); - const tokens = tokenize(this.extract(value)); + const tokens = this.tokenizer(this.extract(value)); const counts = new Map(); for (const t of tokens) counts.set(t, (counts.get(t) ?? 0) + 1); for (const [t, c] of counts) { @@ -241,7 +254,7 @@ export class TextIndex { const docID = this.keys.length; this.keys.push(key); this.keyToId.set(key, docID); - const tokens = tokenize(this.extract(doc)); + const tokens = this.tokenizer(this.extract(doc)); const counts = new Map(); for (const t of tokens) counts.set(t, (counts.get(t) ?? 0) + 1); for (const [t, c] of counts) { @@ -306,7 +319,7 @@ export class TextIndex { } search(query: string, opts: SearchOptions = {}): SearchHit[] { - const qtokens = [...new Set(tokenize(query))]; + const qtokens = [...new Set(this.queryTokenizer(query))]; if (!qtokens.length) return []; const op = opts.op ?? 'AND'; const limit = opts.limit ?? 50; diff --git a/packages/minidb/src/trigram.ts b/packages/minidb/src/trigram.ts new file mode 100644 index 0000000000..537d5be71f --- /dev/null +++ b/packages/minidb/src/trigram.ts @@ -0,0 +1,80 @@ +// src/trigram.ts +// +// n-gram (2-gram + 3-gram) substring tokenizer for literal search. +// +// A TextIndex with this tokenizer indexes hashed character n-grams instead of +// words, so symbol-heavy text (`C++`, `$\frac{a}{b}$`, `**passed**`, emoji) +// becomes searchable as exact substrings. The index only yields candidates — +// zero false positives is guaranteed by a downstream confirmation step that +// re-checks `normalizeLiteral(doc).includes(normalizeLiteral(query))` against +// the original text; hash collisions and non-contiguous n-gram matches only +// cost extra confirmations. +// +// Deliberate deviations from the reference (Elasticsearch `wildcard` field): +// - Case-insensitive: NFKC + lowercase, aligned with the default tokenizer. +// - Windows are cut by Unicode code point (Array.from), not UTF-16 code +// unit, so surrogate pairs (emoji) are never split. +// - n-grams hash into 2^HASH_BITS buckets via the existing crc32 (no new +// deps); 2-grams and 3-grams carry distinct tag prefixes so their buckets +// never alias. + +import { crc32 } from './crc32.js'; + +/** Tokenizer kinds that can be persisted in a text index definition + * (`db.textindexes.json`). A definition without the field (written before + * n-gram support existed) means 'default'. */ +export type TextIndexTokenizerName = 'default' | 'ngram'; + +/** Width of the n-gram hash space in bits (4M buckets). Collisions only add + * confirmation work downstream; they never affect correctness. */ +const HASH_BITS = 22; +const HASH_MASK = (1 << HASH_BITS) - 1; + +/** Normalize text for literal matching: Unicode NFKC (fullwidth `＄` -> `$`, + * compatibility glyphs folded) + lowercase. The search layer's confirmation + * step must use this exact function so index and comparison agree. */ +export function normalizeLiteral(text: string): string { + return text.normalize('NFKC').toLowerCase(); +} + +function termFor(gram: string, width: number): string { + const hash = crc32(Buffer.from(gram, 'utf8')) & HASH_MASK; + return String(width) + hash.toString(36); +} + +/** Encode one n-gram (2 or 3 code points) as an index term: a width tag plus + * the low HASH_BITS of its crc32 in base 36. Deterministic across processes. */ +export function ngramTerm(gram: string): string { + const width = Array.from(gram).length; + if (width !== 2 && width !== 3) { + throw new RangeError(`ngramTerm expects a 2- or 3-gram, got ${width} code points`); + } + return termFor(gram, width); +} + +export interface NgramTokenizerOptions { + /** Query side: a length-2 query emits only its 2-gram, a longer query only + * its 3-grams (fewer, more selective terms). Index side (the default) + * emits every 3-gram plus every 2-gram so both query shapes can match. + * Text shorter than 2 normalized code points yields no terms — the search + * layer must reject such queries itself. */ + forQuery?: boolean; +} + +/** Build a tokenizer that maps text to hashed n-gram terms (see file header). + * Windows slide over code points, so astral characters stay whole. */ +export function createNgramTokenizer(opts: NgramTokenizerOptions = {}): (text: string) => string[] { + return (text) => { + const chars = Array.from(normalizeLiteral(text)); + const n = chars.length; + const terms: string[] = []; + if (n < 2) return terms; + const widths = opts.forQuery ? (n === 2 ? [2] : [3]) : n >= 3 ? [3, 2] : [2]; + for (const w of widths) { + for (let i = 0; i + w <= n; i++) { + terms.push(termFor(chars.slice(i, i + w).join(''), w)); + } + } + return terms; + }; +} diff --git a/packages/minidb/test/text-index.test.ts b/packages/minidb/test/text-index.test.ts index a2ae5054c7..9489d8233e 100644 --- a/packages/minidb/test/text-index.test.ts +++ b/packages/minidb/test/text-index.test.ts @@ -11,6 +11,7 @@ import os from 'node:os'; import path from 'node:path'; import { MiniDb } from '../src/index.js'; import { TextIndex } from '../src/text-index.js'; +import { normalizeLiteral, ngramTerm, createNgramTokenizer } from '../src/trigram.js'; import { encodePostingList, decodePostingList, @@ -275,3 +276,223 @@ test('MiniDb: compaction rebuilds postings (file reclaimed)', async () => { await fs.rm(dir, { recursive: true, force: true }); } }); + +// ---- trigram (n-gram literal tokenizer) ------------------------------------ + +test('trigram: normalization (case, NFKC, code points)', () => { + assert.equal(normalizeLiteral('AbC'), 'abc'); + // NFKC folds compatibility glyphs: fullwidth dollar -> ascii dollar + assert.equal(normalizeLiteral('＄100'), '$100'); + // surrogate pairs count as one code point and survive normalization + assert.equal(Array.from(normalizeLiteral('A🙂')).length, 2); + assert.equal(normalizeLiteral('🙂'), '🙂'); +}); + +test('trigram: hash terms are stable and width-tagged', () => { + // crc32 of the utf8 bytes, low 22 bits, base 36 — pinned so every process + // derives the same term for the same n-gram. + assert.equal(ngramTerm('ab'), '24m0d'); + assert.equal(ngramTerm('abc'), '31exfm'); + assert.equal(ngramTerm('🚀🎉'), '217syy'); + // 2-grams and 3-grams live in different tag namespaces and can never alias + assert.ok(ngramTerm('ab').startsWith('2')); + assert.ok(ngramTerm('abc').startsWith('3')); + assert.notEqual(ngramTerm('ab'), ngramTerm('abc')); + assert.throws(() => ngramTerm('a'), /2- or 3-gram/); +}); + +test('trigram: index vs query tokenizer shapes', () => { + const ix = createNgramTokenizer(); + const q = createNgramTokenizer({ forQuery: true }); + // shorter than 2 normalized code points -> no terms (upper layer rejects) + assert.deepEqual(q('a'), []); + assert.deepEqual(ix('a'), []); + assert.deepEqual(q(''), []); + // length 2: both sides emit exactly the one 2-gram + assert.deepEqual(q('AB'), [ngramTerm('ab')]); + assert.deepEqual(ix('AB'), [ngramTerm('ab')]); + // length >= 3: query side only 3-grams; index side 3-grams + 2-grams + assert.deepEqual(q('abcd'), [ngramTerm('abc'), ngramTerm('bcd')]); + assert.deepEqual(ix('abcd').sort(), [ngramTerm('ab'), ngramTerm('abc'), ngramTerm('bc'), ngramTerm('bcd'), ngramTerm('cd')].sort()); + // emoji are single code points: '🙂a' has length 2 -> one 2-gram, not a + // 3-gram over split UTF-16 surrogates + assert.deepEqual(q('🙂a'), [ngramTerm('🙂a')]); +}); + +test('trigram: normalization can change length (single ligature ff becomes a legal query)', () => { + // 'ff' (U+FB00) is one code point, but NFKC folds it to 'ff' — so the query + // side emits exactly one 2-gram and the search layer's >=2 check (judged + // after normalization) accepts what looks like a 1-character query. + assert.equal(normalizeLiteral('ff'), 'ff'); + assert.deepEqual(createNgramTokenizer({ forQuery: true })('ff'), [ngramTerm('ff')]); +}); + +test('trigram: query of exactly 3 code points emits its single 3-gram', () => { + const q = createNgramTokenizer({ forQuery: true }); + // the boundary where the query side switches from 2-grams to 3-grams + assert.deepEqual(q('abc'), [ngramTerm('abc')]); + assert.deepEqual(q('已通过'), [ngramTerm('已通过')]); + // the index side of a 3-char text still emits both widths + assert.deepEqual( + createNgramTokenizer()('abc').sort(), + [ngramTerm('abc'), ngramTerm('ab'), ngramTerm('bc')].sort(), + ); +}); + +test('TextIndex: injected n-gram tokenizer matches symbol substrings', async () => { + const dir = await tmpDir(); + try { + // Wired the same way MiniDb.createTextIndex(..., { tokenizer: 'ngram' }) + // does: index side both widths, query side forQuery. + const ti = new TextIndex({ + postingsPath: path.join(dir, 'tri.postings'), + tokenizer: createNgramTokenizer(), + queryTokenizer: createNgramTokenizer({ forQuery: true }), + }); + ti.add('cpp', { text: 'modern C++ patterns' }); + ti.add('arrow', { text: 'rewrite a->b safely' }); + ti.add('dash', { text: 'a-b is not an arrow' }); // shares 'a-' but has no '->' + ti.add('done', { text: '检查项 **已通过** 审核' }); + ti.add('latex', { text: 'inline math $\\frac{a}{b}$ here' }); + ti.add('emoji', { text: 'launch 🚀🎉 today' }); + + assert.deepEqual(ti.search('C++').map((h) => h.key), ['cpp']); + assert.deepEqual(ti.search('c++').map((h) => h.key), ['cpp']); // case-insensitive + assert.deepEqual(ti.search('->').map((h) => h.key), ['arrow']); // 2-char query via 2-gram + assert.deepEqual(ti.search('a->b').map((h) => h.key), ['arrow']); // distractor 'a-b' excluded + assert.deepEqual(ti.search('已通过').map((h) => h.key), ['done']); + assert.deepEqual(ti.search('\\frac{a}{b}').map((h) => h.key), ['latex']); + assert.deepEqual(ti.search('🚀🎉').map((h) => h.key), ['emoji']); + // single character yields no terms, hence no hits + assert.deepEqual(ti.search('a'), []); + ti.close(); + } finally { + await fs.rm(dir, { recursive: true, force: true }); + } +}); + +test('TextIndex: n-gram tokenizer delta add/remove/overwrite', async () => { + const dir = await tmpDir(); + try { + const ti = new TextIndex({ + postingsPath: path.join(dir, 'tri.postings'), + tokenizer: createNgramTokenizer(), + queryTokenizer: createNgramTokenizer({ forQuery: true }), + }); + ti.build([{ key: 'a', value: { text: 'C++ guide' } }]); + assert.deepEqual(ti.search('c++').map((h) => h.key), ['a']); + + // writes after build land in the delta and stay searchable + ti.add('b', { text: 'C++ cookbook' }); + assert.deepEqual(ti.search('c++').map((h) => h.key).sort(), ['a', 'b']); + + ti.remove('a'); + assert.deepEqual(ti.search('c++').map((h) => h.key), ['b']); + assert.equal(ti.N, 1); + + // overwrite tombstones the old n-grams + ti.add('b', { text: 'plain c guide' }); + assert.deepEqual(ti.search('c++').map((h) => h.key), []); + assert.deepEqual(ti.search('c guide').map((h) => h.key), ['b']); + ti.close(); + } finally { + await fs.rm(dir, { recursive: true, force: true }); + } +}); + +test('TextIndex: queryTokenizer tokenizes searches when given, falls back otherwise', async () => { + const dir = await tmpDir(); + try { + // Docs are indexed under the index tokenizer's term; a search must + // consult the QUERY tokenizer's term instead. + const ti = new TextIndex({ + postingsPath: path.join(dir, 't.postings'), + tokenizer: () => ['idx-term'], + queryTokenizer: () => ['query-term'], + }); + ti.add('a', { text: 'whatever' }); + assert.deepEqual(ti.search('anything'), []); // looks up 'query-term', never indexed + ti.close(); + + // Without a queryTokenizer, search falls back to the index tokenizer. + const fallback = new TextIndex({ + postingsPath: path.join(dir, 't2.postings'), + tokenizer: () => ['idx-term'], + }); + fallback.add('a', { text: 'whatever' }); + assert.deepEqual(fallback.search('anything').map((h) => h.key), ['a']); + fallback.close(); + } finally { + await fs.rm(dir, { recursive: true, force: true }); + } +}); + +test('MiniDb: n-gram text index tokenizes queries with the forQuery shape', async () => { + const dir = await tmpDir(); + try { + const db = await MiniDb.open({ dir, valueCodec: 'json' }); + await db.createTextIndex('tri', { fields: ['text'], tokenizer: 'ngram' }); + // 'ab only' shares the 2-gram 'ab' with the query 'abc' but none of its + // 3-grams. Under OR, an index-side query tokenizer (3-grams + 2-grams) + // would surface it via 'ab'; the forQuery side emits only the 'abc' + // 3-gram, so nothing matches. + await db.set('partial', { text: 'ab only' }); + assert.deepEqual(db.search('tri', 'abc', { op: 'OR' }), []); + await db.set('full', { text: 'abc here' }); + assert.deepEqual(db.search('tri', 'abc', { op: 'OR' }).map((r) => r.key), ['full']); + await db.close(); + } finally { + await fs.rm(dir, { recursive: true, force: true }); + } +}); + +test('MiniDb: n-gram text index persists tokenizer, survives reopen', async () => { + const dir = await tmpDir(); + try { + let db = await MiniDb.open({ dir, valueCodec: 'json' }); + await db.createTextIndex('tri', { fields: ['text'], tokenizer: 'ngram' }); + await db.createTextIndex('body', { fields: ['text'] }); + await db.set('a', { text: 'modern C++ patterns' }); + await db.set('b', { text: 'plain c plus plus' }); + await db.close(); + + // the sidecar records the tokenizer for the n-gram index; the default + // index keeps the legacy shape (no tokenizer field), which is also what + // definition files written before n-gram support look like. + const defs = JSON.parse(await fs.readFile(path.join(dir, 'db.textindexes.json'), 'utf8')) as { + name: string; + tokenizer?: string; + }[]; + assert.equal(defs.find((d) => d.name === 'tri')!.tokenizer, 'ngram'); + assert.ok(!('tokenizer' in defs.find((d) => d.name === 'body')!)); + + db = await MiniDb.open({ dir, valueCodec: 'json' }); + // n-gram index restored as n-gram: 'C++' matches only the real substring + assert.deepEqual(db.search('tri', 'C++').map((r) => r.key), ['a']); + // default index restored as default: 'C++' still tokenizes to the word 'c' + assert.deepEqual(db.search('body', 'C++').map((r) => r.key).sort(), ['a', 'b']); + + // delta writes after reopen use the restored tokenizer too + await db.set('c', { text: 'another C++ note' }); + assert.deepEqual(db.search('tri', 'c++').map((r) => r.key).sort(), ['a', 'c']); + await db.close(); + } finally { + await fs.rm(dir, { recursive: true, force: true }); + } +}); + +test('MiniDb: createTextIndex rejects an unknown tokenizer', async () => { + const dir = await tmpDir(); + try { + const db = await MiniDb.open({ dir, valueCodec: 'json' }); + await assert.rejects( + db.createTextIndex('x', { tokenizer: 'bogus' as 'ngram' }), + /unknown text index tokenizer/, + ); + // the failed creation left nothing behind + assert.throws(() => db.search('x', 'q'), /no such text index/); + await db.close(); + } finally { + await fs.rm(dir, { recursive: true, force: true }); + } +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a40f42f2d0..91c08bee32 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -841,6 +841,9 @@ importers: '@moonshot-ai/kimi-code-oauth': specifier: workspace:^ version: link:../oauth + '@moonshot-ai/minidb': + specifier: workspace:* + version: link:../minidb '@moonshot-ai/transcript': specifier: workspace:^ version: link:../transcript