feat(kap-server): add global message search with literal and live-session modes - #2321
Conversation
Cross-session full-text search over user messages, assistant text and session titles, backed by a minidb index under <home>/search-index with a single-writer lock election and read-only WAL catch-up for other processes. Hits carry transcript anchors (turn ordinal and step id) so clients can jump straight to the matching turn or step.
The left rail gains a Search view over the global search endpoint, with role and sort filters and cursor pagination. Clicking a hit switches to the chat view and navigates to its session, agent, turn and step — the channel pages the turn into the loaded window, scrolls it into view and flashes the target briefly.
The repo's dev server scripts (dev:server, dev:kap-server, dev:kap-server:multi, dev:server:restart) now set KIMI_CODE_DEV_SERVER=1. When it is set and dist-web/index.html is missing, kimi web starts the API server without the bundled web UI instead of failing at startup, so backend dev no longer requires a kimi-web build.
- minidb: text indexes accept an injectable tokenizer/queryTokenizer, and a hashed 2/3-gram tokenizer (NFKC + lowercase, code-point windows) backs substring search; tokenizer names persist in db.textindexes.json with backward-compatible defaults - /api/v1/search gains mode: 'literal' — n-gram candidates confirmed against the original text (zero false positives), with an 'candidate_cap' incomplete flag when the candidate set truncates - container.session_id queries against a session live in this process scan the in-memory transcript store instead of the index (both modes); the response's source: live|index field names the serving route and rides in the page-token fingerprint - kimi-inspect: exact-match toggle and source badge in the search view, plus an in-chat session search bar with jump-to-hit navigation
|
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 165d78eb9e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (m.role === 'user') { | ||
| const origin = m.origin; | ||
| const userTyped = | ||
| origin === null || | ||
| origin === undefined || | ||
| (typeof origin === 'object' && isUserTypedOrigin(origin as OriginLike)); | ||
| if (userTyped) { | ||
| const text = textOfContent(m.content).trim(); | ||
| if (text.length > 0) messages.push({ role: 'user', text, time }); | ||
| } | ||
| } |
There was a problem hiding this comment.
When replaying older/cold wire records where assistant replies are stored as context.append_message (the transcript cold fold handles these as assistant text frames in packages/transcript/src/history/groupTurns.ts), this branch only extracts user messages, so those assistant replies never enter the persisted search index and global search silently misses them. The search surface is repo-defined as covering assistant text, so this leaves searchable transcript content out of the index.
AGENTS.md reference: AGENTS.md:L28-L28
Useful? React with 👍 / 👎.
| const matched = | ||
| q.mode === 'literal' | ||
| ? this.matchDocs( | ||
| q, | ||
| docs.map((value) => ({ value, score: 0 })), | ||
| ) | ||
| : this.matchDocs(q, matchLiveTerms(q.termsQuery ?? [], docs)); |
There was a problem hiding this comment.
For POST /api/v1/search requests with container.session_id pointing at a live session and op: 'OR', this path still calls matchLiveTerms without the requested operator, and that helper requires every query term to be present. The same query falls back to db.search(..., { op: q.op }) when the session is not live, so OR searches return different and incorrectly narrower results depending only on whether the session is currently materialized.
Useful? React with 👍 / 👎.
| } else { | ||
| candidates = db.search(TEXT_INDEX_NAME, q.query, { op: q.op, limit: MAX_TEXT_HITS }); |
There was a problem hiding this comment.
For common term-mode queries, db.search is capped at MAX_TEXT_HITS before the container/role/time filters and requested time sort are applied in matchDocs/toPage. In a large index, a scoped search such as one session or a recent time range can have its real matches beyond the first 100k score-ranked global candidates, causing an empty or incomplete page with no incomplete flag even though matching documents exist.
Useful? React with 👍 / 👎.
1 participant
Related Issue
No linked issue — the problem is explained below.
Problem
kimi-inspect(the dev inspector for kap-server) had no way to search message content: finding "which session discussed X" meant opening sessions one by one. A first-cut global search existed on a branch, but three limitations made it weak in practice:C++,$\frac{a}{b}$, or**已通过**were unsearchable, and multi-character CJK queries produced scattered-character false positives.What changed
1. Global message search API (
POST /api/v1/search)Cross-session full-text search over user messages, assistant text, and session titles, backed by a single minidb database at
<home>/search-index(write-lock holder indexes; other processes open read-only and catch up via WAL). Cursor pagination with a query-conditions fingerprint, role/time/container filters, and an index-state report.2. Literal substring search (minidb n-gram index)
tokenizer/queryTokenizer; a hashed 2/3-gram tokenizer (NFKC + lowercase, code-point windows, crc32 buckets) powers substring search. Tokenizer names persist indb.textindexes.json, backward compatible with existing definitions.mode: 'literal'on the search API: n-gram candidates are confirmed against the original text, so hits carry zero false positives; a truncated candidate set is flaggedincomplete: 'candidate_cap'. The <2-character minimum applies only to the index route (it is an n-gram constraint, not a matching constraint).wildcardfield (two-stage candidate + confirm), with three deliberate deviations documented in code: case-insensitive matching, code-point windows, explicit error on 1-character queries.3. Live session route (in-memory transcript scan)
When
container.session_idis provided and that session is live in the process (TranscriptService.forSessionLive), bothtermsandliteralmodes scan the in-memory transcript store (turn prompts + assistant text frames, full history via the existing backfill) instead of the index — freshly streamed text is searchable immediately. The response carriessource: 'live' | 'index', also mixed into the page-token fingerprint so a mid-pagination route flip invalidates old tokens. Live-route errors never silently fall back to the index.4. kimi-inspect UI
source: live|indexbadge, candidate-truncation notice; hits navigate into the chat timeline (page-back + scroll + flash).container.session_id, usually served by the live route) with jump-to-hit navigation.5. Dev tooling (unrelated, small)
The repo's dev server scripts (
dev:server,dev:kap-server*) now setKIMI_CODE_DEV_SERVER=1; when it is set anddist-webis missing,kimi webstarts the API server without the bundled web UI instead of failing at startup, so backend dev no longer requires a web build.Checklist
gen-changesetsskill, or this PR needs no changeset. (No changeset: kap-server REST/protocol changes consumed only by dev tooling — kimi-inspect — are not user-perceivable per the changeset rules; the dev-script change is repo-internal.)gen-docsskill, or this PR needs no doc update. (Dev-tooling only; root AGENTS.md project map updated instead.)