diff --git a/README.md b/README.md index 5fec46f47..e035cf20e 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,35 @@ # codemem -[![CI](https://github.com/kunickiaj/codemem/actions/workflows/ci.yml/badge.svg)](https://github.com/kunickiaj/codemem/actions/workflows/ci.yml) [![codecov](https://codecov.io/gh/kunickiaj/codemem/branch/main/graph/badge.svg)](https://codecov.io/gh/kunickiaj/codemem) [![Release](https://img.shields.io/github/v/release/kunickiaj/codemem)](https://github.com/kunickiaj/codemem/releases) +[![CI](https://github.com/kunickiaj/codemem/actions/workflows/ci.yml/badge.svg)](https://github.com/kunickiaj/codemem/actions/workflows/ci.yml) +[![npm version](https://img.shields.io/npm/v/codemem)](https://www.npmjs.com/package/codemem) [![npm downloads per month](https://img.shields.io/npm/dm/codemem?label=npm%20downloads%2Fmonth)](https://www.npmjs.com/package/codemem) [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE) -Persistent memory for [OpenCode](https://opencode.ai) and [Claude Code](https://claude.ai/code). codemem captures what you work on across sessions, retrieves relevant context using hybrid search, and injects relevant context automatically in OpenCode 1. +**The code is still there. The reasoning usually isn’t.** -- **Local-first** — everything lives in SQLite on your machine +codemem is persistent coding memory across sessions, machines, and teammates for [OpenCode](https://opencode.ai), [Claude Code](https://claude.ai/code), and Codex. It captures decisions, dead ends, and repository-specific traps, then automatically brings relevant context into later prompts. + +- **Automatic context injection** — relevant memories reach the agent without asking it to search; unchanged memories already in OpenCode context are not repeated +- **Optional sync and sharing** — peer-to-peer sync carries selected project memory across machines; share project knowledge with a teammate or Team when it helps +- **Local-first storage** — memories live in SQLite on your machine; observer processing uses your configured model provider and can incur costs or consume plan usage - **Hybrid retrieval** — FTS5 BM25 lexical search + sqlite-vec semantic search, merged and re-ranked - **Automatic injection for OpenCode 1** — the plugin injects context into every prompt, no manual steps - **Claude Code plugin support** — install from the codemem marketplace source - **Built-in viewer** — browse memories, sessions, and observer output in a local web UI -- **Peer-to-peer sync** — replicate memories across machines without a central service +- **Remote MCP access** — advanced single-user self-hosting can expose an OAuth-protected Streamable HTTP MCP endpoint to configured remote clients; keep the localhost viewer private ([guide](docs/remote-mcp-oauth.md)) - - codemem viewer — feed tab + + codemem Feed showing invented coding decisions, discoveries, and fixes +*Synthetic project memories, not real session data. Explore the [Feed, Facts, and Projects walkthrough](docs/user-guide.md#explore-the-viewer) to see how to inspect what was captured.* + ## Quick start **Prerequisites:** Node.js 24.15+ and npm (or pnpm). Native database support covers macOS x64/arm64, Linux x64/arm64 (glibc 2.34+ or musl), and Windows x64. 32-bit targets, including Linux armv7, are not supported. -codemem keeps one minimum Node.js version across published packages and workspace tooling. +**Linux:** set `ONNXRUNTIME_NODE_INSTALL=skip` in your shell and the environment that launches OpenCode, Claude Code, or Codex **before the first package install**. This avoids downloading the unused ONNX Runtime GPU provider while keeping CPU inference. Setup-managed `npx` launchers cannot set it themselves. ### OpenCode @@ -40,7 +47,7 @@ npx -y codemem setup --opencode-only 2. Restart OpenCode. -On OpenCode 1, the plugin manages backend execution automatically — no separate global install is required. +`npx` uses a downloaded or cached package to configure the OpenCode host; it does not create a durable `codemem` CLI installation. On OpenCode 1, the configured plugin manages backend execution independently, so no global install is required for automatic capture and context injection. 3. Verify: @@ -50,51 +57,56 @@ npx -y codemem stats npx -y codemem db raw-events-status ``` -That's it. On OpenCode 1, the plugin captures activity, builds memories, and injects context from here on. +That's it. On OpenCode 1, the plugin captures activity, builds memories, and injects relevant context from here on. -If you want `codemem` available directly on your `PATH` for manual commands and semantic retrieval, install the CLI globally. The CLI installs its matching embedding runtime by default. The command differs by platform: +### Try a fresh-session recall -On Linux, skip the unused ONNX Runtime GPU provider download: +After completing a task, check that its decision appears in the local viewer at `http://localhost:38888`. Start a new OpenCode session in the same project and ask about that decision without supplying the answer. For example, if your task involved a database migration: ```text -env ONNXRUNTIME_NODE_INSTALL=skip npm install -g codemem +What decision did we make about the database migration, and why? +``` + +Verify the answer against the stored decision and the original task evidence. If capture is still pending or recall is empty, inspect the local state: + +```text +npx -y codemem status +npx -y codemem db raw-events-status ``` -On Apple silicon macOS and Windows, install normally: +### Observer access and external-model costs + +The observer is the model that turns captured activity into memories. It needs a configured runtime and usable authentication: `api_http` uses your provider credentials; sidecar runtimes use Claude or Codex authentication. Local storage does not mean local-only processing: captured context is sent to the configured model, and calls can incur charges or consume plan usage. See [configuration](#configuration) for options. + +
+Installation, upgrades, and runtime details + +codemem keeps one minimum Node.js version across published packages and workspace tooling. + +If you want `codemem` on your `PATH` for manual commands and semantic retrieval, install the CLI globally. The CLI installs its matching embedding runtime by default: ```text +# Linux: skip the unused ONNX Runtime GPU provider download +env ONNXRUNTIME_NODE_INSTALL=skip npm install -g codemem + +# Apple silicon macOS and Windows npm install -g codemem ``` -For a smaller keyword-only install, use `npm install -g codemem --omit=optional` -and set `CODEMEM_EMBEDDING_DISABLED=1` in every Codemem process. The flag is -required because npm also omits sqlite-vec's optional platform package; the CLI -then remains functional with FTS5. - -Setup-managed `npx` launchers cannot set a platform-specific install variable. -On Linux, either use the guarded global installation above or set -`ONNXRUNTIME_NODE_INSTALL=skip` in the environment that launches OpenCode, -Claude Code, or Codex before its first `npx` package resolution. This prevents -the unused ONNX Runtime GPU-provider download while retaining CPU inference. - -After upgrading an existing installation, rerun `codemem setup` (or the -app-specific `--opencode-only`, `--claude-only`, or `--codex-only` form). -Setup replaces the old managed `npx -y codemem mcp` launcher and codemem MCP -entries detected as UV/UVX-based so both packages share one runtime. Other -custom MCP commands remain unchanged. - -Generated MCP configurations use the durable global `codemem` binary when it is -available. Without a global install, setup-managed `npx` launchers request both -packages in the same temporary environment. After changing the installation, -restart the host whose config you updated — OpenCode, Claude Code, or Codex — -plus any running `codemem serve` process. Each MCP process caches runtime -availability for its lifetime, so a still-running Claude or Codex MCP host keeps -lexical-only recall until it restarts; restarting `codemem serve` alone does not -restart that MCP child. - -The semantic runtime is pinned to CPU inference on every platform. -ONNX Runtime 1.24.3 does not ship a macOS x64 binary, so Intel Macs continue -with FTS5 keyword retrieval when semantic runtime initialization fails. +For a smaller keyword-only install, use `npm install -g codemem --omit=optional` and set `CODEMEM_EMBEDDING_DISABLED=1` in every Codemem process. The flag is required because npm also omits sqlite-vec's optional platform package; the CLI then remains functional with FTS5. + +An npm-capable manager, such as mise's `npm:codemem` backend, can also provide the durable CLI. Ensure `codemem` is on your `PATH`, then run `codemem setup --opencode-only`. + +Upgrade a durable CLI with the package manager that installed it, then rerun the corresponding setup command for an existing setup-managed integration. Setup replaces its old managed `npx -y codemem mcp` launcher and codemem MCP entries detected as UV/UVX-based so both packages share one runtime; other custom MCP commands remain unchanged. The host plugin is managed independently. Claude marketplace installs use the plugin's bundled MCP configuration and do not require a separate `setup --claude-only` step. + +Generated MCP configurations use the global `codemem` binary when available. Otherwise, setup-managed `npx` launchers request both packages in one temporary environment. Restart the updated host and any `codemem serve` process after an installation change. An already-running Claude or Codex MCP host keeps lexical-only recall until it restarts; restarting `codemem serve` does not restart that child. + +The semantic runtime is pinned to CPU inference on every platform. ONNX Runtime 1.24.3 does not ship a macOS x64 binary, so Intel Macs continue with FTS5 keyword retrieval when semantic runtime initialization fails. + +
+ +
+OpenCode recall and source-checkout details OpenCode plugin and CLI are now split intentionally: @@ -122,22 +134,23 @@ missing host identity, or sibling IDs. Explicit `pack` and MCP requests keep their existing behavior. See [the requester-session contract](docs/opencode-retained-recall.md#requester-session-continuity). -### Claude Code (marketplace install) - -1. Install codemem's Claude MCP config: +
-```text -npx -y codemem setup --claude-only -``` +### Claude Code (marketplace install) -2. In [Claude Code](https://claude.ai/code), add the codemem marketplace source and install the plugin: +1. In [Claude Code](https://claude.ai/code), add the codemem marketplace source and install the plugin: ```text /plugin marketplace add kunickiaj/codemem /plugin install codemem ``` -The Claude plugin starts MCP with the TS CLI (`codemem mcp`). +2. Restart Claude Code. + +The plugin bundles its MCP configuration and capture/context-injection hooks, and starts MCP with the TS CLI (`codemem mcp`). No preliminary `codemem setup --claude-only` command or global CLI install is required. The prerequisites and observer-access requirements above still apply. + +
+Claude and Codex adapter transport details Claude and Codex plugins normalize native hooks at the plugin edge and send the resulting envelope to the canonical `POST /api/raw-events` endpoint. New ingestion requests include the intended database path and runtime identity target; Viewer rejects a mismatch before writing, and the client uses its existing identity-correct command fallback. On a retryable Viewer failure, Codex persists that exact envelope before attempting command fallbacks and removes the spool only after a fallback succeeds; Claude uses the command fallbacks without a file spool. Claude `SessionEnd` asks Viewer to finish boundary extraction best-effort inside the host's 1.5-second default exit budget, reserving command-fallback time after preprocessing and across both HTTP attempts. `Stop` flushing remains opt-in and uses a 130-second host timeout for its 125-second internal extraction budget. Transcript fallback reads at most the final 16 MiB: it preserves the first record when the tail starts immediately after a newline, but discards the first fragment when the tail starts in the middle of a record. The checked-in dependency-free normalizers are generated from the TypeScript implementations in `packages/core/src/claude-hooks.ts` and `packages/core/src/codex-hooks.ts`. Named Viewer hook routes remain compatibility aliases/callers for older packaged and plugin-free CLI paths; requests that omit targeting fields remain accepted for 0.41 compatibility. @@ -151,9 +164,11 @@ fail closed. Prompt and event HTTP reject non-loopback Viewer hosts without fetching them. Codex reserves a total 4.5-second prompt-output budget within its 5-second host timeout. -### Codex (early beta) +
+ +### Codex -Codex support is **early beta** — functional and dogfooded, but not yet promoted to a stable support tier. It installs through Codex's own plugin marketplace: +Codex installs through its own plugin marketplace: 1. Add the codemem marketplace and install the plugin: @@ -176,7 +191,7 @@ This merges `[mcp_servers.codemem]` into `~/.codex/config.toml` and writes `~/.c Codex hook ingestion shares the same raw-event pipeline as Claude and OpenCode through normalized `POST /api/raw-events`. After a retryable HTTP failure it writes the exact envelope to `~/.codemem/codex-raw-event-spool`, attempts the `codemem enqueue-raw-event` command fallbacks, and removes the spooled envelope only after success. That spool is separate from the legacy native-hook spool. `UserPromptSubmit` runs capture ingest in the background and injects memory context via `additionalContext`; disable injection with `CODEMEM_INJECT_CONTEXT=0`. See [docs/plugin-reference.md](docs/plugin-reference.md) for details and troubleshooting. -> Migrating from `opencode-mem`? See [docs/rename-migration.md](docs/rename-migration.md). +> Was this repository previously installed as `opencode-mem`? See the [rename migration guide](docs/rename-migration.md). It covers this repository's former name, not importing data from [`tickernelz/opencode-mem`](https://github.com/tickernelz/opencode-mem). ## How it works @@ -363,10 +378,11 @@ The viewer includes a grouped Settings modal (`Connection`, `Processing`, `Devic Observer runtime/auth: -- Runtime options: `api_http` and `claude_sidecar`. +- Runtime options: `api_http`, `claude_sidecar`, and `codex_sidecar`. - `api_http` defaults to `gpt-5.1-codex-mini` (OpenAI path) unless you set `observer_model`. - Anthropic direct API calls accept Anthropic model IDs/aliases. codemem maps the common Claude shorthand `claude-4.5-haiku` to Anthropic's direct API alias `claude-haiku-4-5`; you can also set a pinned snapshot like `claude-haiku-4-5-20251001` explicitly. - `claude_sidecar` defaults to `claude-4.5-haiku`; if the selected `observer_model` is unsupported by Claude CLI, codemem retries once with Claude's CLI default model. +- `codex_sidecar` uses the local Codex CLI's authentication and defaults to `gpt-5.1-codex-mini` unless `observer_model` is set. See [observer auth modes](docs/plugin-reference.md#observer-auth-modes) for configuration and automatic selection rules. - `claude_sidecar` command is configurable with `claude_command` (`CODEMEM_CLAUDE_COMMAND`) as a JSON argv array. - Config file example: `"claude_command": ["wrapper", "claude", "--"]` - Env var example: `CODEMEM_CLAUDE_COMMAND='["wrapper","claude","--"]'` @@ -490,5 +506,5 @@ The repository's root `opencode.jsonc` also enables a contributor-only lint-feed - [Coordinator deployment](docs/coordinator-deployment.md) — advanced operator deployment and discovery - [Coordinator E2E runbook](docs/coordinator-e2e-runbook.md) — advanced coordinator validation - [Plugin reference](docs/plugin-reference.md) — plugin behavior, env vars, stream reliability -- [Migration guide](docs/rename-migration.md) — migrating from `opencode-mem` +- [Rename migration guide](docs/rename-migration.md) — this repository's former `opencode-mem` name; not an importer for `tickernelz/opencode-mem` - [Contributing](CONTRIBUTING.md) — development setup, tests, linting, releases diff --git a/docs/architecture.md b/docs/architecture.md index d626e3c97..b56333f20 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -62,7 +62,7 @@ Support tiers describe operational expectations for each adapter path: | OpenCode 1 plugin | Supported | Primary reference adapter for lifecycle events and injection behavior. | | OpenCode 2 plugin | Experimental | The beta entrypoint loads as an inactive compatibility shell; capture and injection are not enabled. | | Claude hooks/plugin | Supported | Hook-first queue path with CLI/runtime fallback and parity slices tracked in adapter stack PRs. | -| Codex plugin (hooks + MCP) | Experimental (early beta) | Functional capture pipeline (`plugins/codex/`, `packages/core/src/codex-hooks.ts`) dogfooded end-to-end: edge normalization → `POST /api/raw-events` → observer → memories. Prompt-time injection present and env-gated but not fully validated on strict models. Not yet promoted to a stable support tier. | +| Codex plugin (hooks + MCP) | Supported | Functional capture pipeline (`plugins/codex/`, `packages/core/src/codex-hooks.ts`) dogfooded end-to-end: edge normalization → `POST /api/raw-events` → observer → memories. Prompt-time injection is present and env-gated but not fully validated on strict models. | | Windsurf integration | Experimental | Planned via shared adapter contract after OpenCode/Claude stabilization. | | Cursor integration | Experimental | Planned via shared adapter contract after OpenCode/Claude stabilization. | @@ -72,7 +72,7 @@ Rollout sequencing: 2. Reach OpenCode parity on ingest + retrieval quality. 3. Ship Claude MVP, then close parity gaps (injection/capture/lifecycle). 4. Keep Claude in Supported tier by enforcing reliability and review gates. -5. Add additional adapters (Codex/Windsurf/Cursor) behind the same contract. +5. Add additional adapters (Windsurf/Cursor) behind the same contract. Explicit non-goals: diff --git a/docs/docs-screenshots.md b/docs/docs-screenshots.md new file mode 100644 index 000000000..c4935e04d --- /dev/null +++ b/docs/docs-screenshots.md @@ -0,0 +1,59 @@ +# Viewer Screenshot Guide + +Use the synthetic viewer to capture the current generated UI without opening a real Codemem database or configuration. + +## Start the fixture + +Build the UI, then start the foreground fixture. It prints JSON containing a random loopback URL; open that URL in a dedicated CMux docs surface. + +```fish +pnpm --filter @codemem/ui build +pnpm exec tsx --conditions source scripts/docs-viewer.ts +``` + +Leave that command running. In a second terminal, use the printed URL to create a browser surface, then copy the returned surface reference: + +```fish +set viewer_url "http://127.0.0.1:" +cmux browser open "$viewer_url" --focus false --json +set surface "" +cmux browser --surface "$surface" viewport 1440 1050 +cmux browser --surface "$surface" wait --text "Reject duplicate watering commands" --timeout-ms 10000 +cmux browser --surface "$surface" snapshot --interactive +``` + +The launcher starts a sanitized child with a fresh temporary database containing six invented private memories for `atlas-notes` and `garden-api`. It disables embeddings, observer work, sweeper work, sync, and update checks, and blocks non-loopback server calls through `fetch`. That block does not cover other HTTP clients or raw sockets; the viewer also loads public CDN fonts and icons in the browser, so this is not a network sandbox. + +The fixture serves the current working tree, including uncommitted UI changes. It is visual evidence only—not release evidence or validation of extraction or semantic-recall quality. Stop it with `ctrl-c`; its temporary runtime remains available for inspection. After stopping the fixture, verify the printed runtime belongs to this run before removing it manually; never substitute your normal Codemem directory. + +## Capture + +Use a dedicated docs workspace/surface, wait for the page to render, and capture only the browser viewport. A background CMux workspace can return text while its paint is suspended, producing a blank image. + +Before **every** capture—and again after navigation or reload—inject this CSS. It hides only the fixture database-path line; it does not change memory text or controls. + +```fish +cmux browser --surface "$surface" addstyle '#metaLine { visibility: hidden !important; }' +``` + +Create an inspection directory and write captures there first: + +```fish +mkdir -p .tmp/docs-screenshots +cmux browser --surface "$surface" screenshot --out ".tmp/docs-screenshots/docs-feed-dark.png" --json +``` + +Set the viewport to `1440x1050`, wait for rendering, then capture these states: + +- Feed in dark theme: `docs-feed-dark.png` +- Feed in light theme: `docs-feed-light.png` +- Feed with a memory's **Facts** button active, dark theme: `docs-memory-facts-dark.png` +- Feed with a memory's **Facts** button active, light theme: `docs-memory-facts.png` +- Projects, dark theme: `docs-projects-dark.png` +- Projects, light theme: `docs-projects.png` + +CMux screenshots capture the viewport only; do not describe them as full-page captures. Do not capture the terminal or the full desktop. + +## Publish after review + +Inspect each temporary image before copying it to `docs/images/`. Confirm that it contains no private data or local paths, uses the intended theme and fixture content, has the fixed viewport, and carries the expected fixture state. Keep the Projects image framed as an informational review state: it has two Sharing review findings, no recipients, and no sync activity. diff --git a/docs/images/docs-feed-dark.png b/docs/images/docs-feed-dark.png new file mode 100644 index 000000000..53356fdde Binary files /dev/null and b/docs/images/docs-feed-dark.png differ diff --git a/docs/images/docs-feed-light.png b/docs/images/docs-feed-light.png new file mode 100644 index 000000000..5a9b5e40c Binary files /dev/null and b/docs/images/docs-feed-light.png differ diff --git a/docs/images/docs-memory-facts-dark.png b/docs/images/docs-memory-facts-dark.png new file mode 100644 index 000000000..b9b3387cc Binary files /dev/null and b/docs/images/docs-memory-facts-dark.png differ diff --git a/docs/images/docs-memory-facts.png b/docs/images/docs-memory-facts.png new file mode 100644 index 000000000..281f99446 Binary files /dev/null and b/docs/images/docs-memory-facts.png differ diff --git a/docs/images/docs-projects-dark.png b/docs/images/docs-projects-dark.png new file mode 100644 index 000000000..c28bc51ab Binary files /dev/null and b/docs/images/docs-projects-dark.png differ diff --git a/docs/images/docs-projects.png b/docs/images/docs-projects.png new file mode 100644 index 000000000..31c945015 Binary files /dev/null and b/docs/images/docs-projects.png differ diff --git a/docs/plugin-reference.md b/docs/plugin-reference.md index b756c8c91..d82ba7328 100644 --- a/docs/plugin-reference.md +++ b/docs/plugin-reference.md @@ -127,9 +127,9 @@ For Claude hooks, project resolution precedence is: `PreToolUse` is intentionally deferred in the default template. Current memory extraction uses `PostToolUse` / `PostToolUseFailure` (`tool_result`) as the shipped Claude tool signal. -## Codex integration (early beta) +## Codex integration -Codex support is early beta — functional and dogfooded end-to-end, but not yet promoted to a stable support tier. The Codex plugin uses the same shared raw-event pipeline as Claude and OpenCode. It is packaged under `plugins/codex/` with `.codex-plugin/plugin.json`, bundled `.mcp.json`, and hook scripts under `plugins/codex/scripts/`. +Codex is a supported integration. The Codex plugin uses the same shared raw-event pipeline as Claude and OpenCode. It is packaged under `plugins/codex/` with `.codex-plugin/plugin.json`, bundled `.mcp.json`, and hook scripts under `plugins/codex/scripts/`. Codex's Node/ESM wrapper adds a timestamp and nonce when the host omitted a timestamp, normalizes exactly once, and sends the exact envelope to `POST /api/raw-events`. Healthy HTTP ingestion starts no `codemem` or `npx` child. After a retryable HTTP failure, it durably spools the normalized envelope before starting this fallback chain: @@ -174,7 +174,7 @@ For Codex hooks, project resolution precedence matches the Claude hook path: `Stop` events map the inline `last_assistant_message` when present, and fall back to the last assistant message in `transcript_path` so final responses are captured even when the inline field is omitted. This fallback uses the same backward, bounded 16 MiB JSONL scan and record-boundary rules as Claude. -The packaged Codex template registers `SessionStart`, `UserPromptSubmit`, `PostToolUse`, and `Stop` in `plugins/codex/hooks/hooks.json`. Codex support is early beta; see `docs/plans/2026-05-28-codex-first-class-integration.md` for the rollout plan and validation gates. +The packaged Codex template registers `SessionStart`, `UserPromptSubmit`, `PostToolUse`, and `Stop` in `plugins/codex/hooks/hooks.json`. See `docs/plans/2026-05-28-codex-first-class-integration.md` for the historical rollout plan and validation gates. ### Install, update, and uninstall diff --git a/docs/user-guide.md b/docs/user-guide.md index 6ad18a607..8cb3da39a 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -1,5 +1,30 @@ # User Guide +## Explore the viewer + +The Feed shows captured memories; use the theme control to switch appearance and search to narrow the list. + + + + Feed showing captured memories + + +Select **Facts** on a memory to review its extracted facts without leaving the Feed. + + + + Extracted facts for a memory in the Feed + + +Optionally, use **Projects** to review project-level information. This example uses synthetic fixture data: its two Sharing review findings are informational, show no recipients, and do not show a successful sharing or sync flow. + + + + Projects view with fixture data + + +All screenshots use invented data. Maintainers can reproduce them with the [screenshot guide](docs-screenshots.md). + ## Check for updates Use the read-only release check to compare the running CLI with the latest npm release on its @@ -501,11 +526,13 @@ environment variable, which `env`/`cmd.exe`/PowerShell do not share): npm install -g codemem ``` -Rerun `codemem setup` after upgrading an existing installation. The scoped -`--opencode-only`, `--claude-only`, and `--codex-only` forms work too. Setup -replaces the old managed `npx -y codemem mcp` launcher and codemem MCP entries -detected as UV/UVX-based so both packages resolve in one runtime. Other custom -MCP commands remain unchanged. +For an existing setup-managed installation, upgrade `codemem` with the package +manager that originally installed it, then rerun its setup command. The scoped +`--opencode-only` and `--codex-only` forms work too. Setup replaces the old +managed `npx -y codemem mcp` launcher and codemem MCP entries detected as +UV/UVX-based so both packages resolve in one runtime. Other custom MCP commands +remain unchanged. Claude marketplace installs bundle MCP configuration and need +no separate setup command. For a smaller keyword-only install, use `npm install -g codemem --omit=optional` and set `CODEMEM_EMBEDDING_DISABLED=1` in every Codemem process. The flag is diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index e5a128713..2ee204e26 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -674,6 +674,15 @@ export { buildMemoryPackWithTraceAsync, estimateTokens, } from "./pack.js"; +export type { PiFlushSignal, PiHookAdapterEvent, PiHookRawEventEnvelope } from "./pi-hooks.js"; +export { + buildIngestPayloadFromPiEvent, + buildPiFlushSignalFromEvent, + buildRawEventEnvelopeFromPiEvent, + MAPPABLE_PI_EVENTS, + mapPiEventPayload, + PI_FLUSH_ONLY_EVENTS, +} from "./pi-hooks.js"; export type { BlockedPolicyTeamDeviceEligibilityResult, DerivePolicyTeamDeviceEligibilityInput, diff --git a/packages/core/src/pi-hooks.test.ts b/packages/core/src/pi-hooks.test.ts new file mode 100644 index 000000000..3e68537cd --- /dev/null +++ b/packages/core/src/pi-hooks.test.ts @@ -0,0 +1,885 @@ +/** + * Tests for pi-hooks.ts — AdapterEvent v1 mapping for pi extension events. + * + * Covers: + * - mapPiEventPayload: all mappable event types, skip cases, deterministic ids, + * tool_result isError, fork → new stream identity + * - buildPiFlushSignalFromEvent: session_before_compact flush-only contract + * - buildRawEventEnvelopeFromPiEvent: envelope shape + source "pi" + * - buildIngestPayloadFromPiEvent: session context fields + * - store attribution: pi-ingested rows carry source = "pi" (no opencode rows) + */ + +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { connect } from "./db.js"; +import { + buildIngestPayloadFromPiEvent, + buildPiFlushSignalFromEvent, + buildRawEventEnvelopeFromPiEvent, + MAPPABLE_PI_EVENTS, + mapPiEventPayload, + PI_EVENT_ID_ALGO, +} from "./pi-hooks.js"; +import { ingestRawEvents } from "./raw-event-ingest.js"; +import { MemoryStore } from "./store.js"; +import { initTestSchema } from "./test-utils.js"; + +function requireEnvelope(envelope: ReturnType) { + if (envelope === null) { + throw new Error("expected pi envelope"); + } + return envelope; +} + +const PI_EVENT_ID = /^pi_evt_[0-9a-f]{24}$/; + +// --------------------------------------------------------------------------- +// mapPiEventPayload — event type mapping +// --------------------------------------------------------------------------- + +describe("mapPiEventPayload", () => { + describe("session_start → session_start", () => { + it("maps session start with deterministic id", () => { + const event = mapPiEventPayload({ + piEvent: "session_start", + sessionId: "pi-sess-1", + cwd: "/tmp/repo", + ts: "2026-06-01T12:00:00Z", + }); + + expect(event).not.toBeNull(); + expect(event?.schema_version).toBe("1.0"); + expect(event?.source).toBe("pi"); + expect(event?.event_type).toBe("session_start"); + expect(event?.session_id).toBe("pi-sess-1"); + expect(event?.event_id).toMatch(PI_EVENT_ID); + expect(event?.cwd).toBe("/tmp/repo"); + expect(event?.ts).toBe("2026-06-01T12:00:00Z"); + }); + + it("prefers entryId for the event id suffix when present", () => { + const event = mapPiEventPayload({ + piEvent: "session_start", + sessionId: "pi-sess-1", + entryId: "entry-start-1", + ts: "2026-06-01T12:00:00Z", + }); + expect(event?.event_id).toMatch(PI_EVENT_ID); + }); + }); + + describe("session_shutdown → session_end", () => { + it("maps reason field", () => { + const event = mapPiEventPayload({ + piEvent: "session_shutdown", + sessionId: "pi-sess-end", + reason: "user_exit", + ts: "2026-06-01T13:00:00Z", + }); + + expect(event).not.toBeNull(); + expect(event?.event_type).toBe("session_end"); + expect(event?.payload.reason).toBe("user_exit"); + expect(event?.event_id).toMatch(PI_EVENT_ID); + expect(event?.source).toBe("pi"); + expect(event?.meta.event_id_algo).toBe(PI_EVENT_ID_ALGO); + expect(PI_EVENT_ID_ALGO).toBe("pi/1"); + }); + }); + + describe("message_end role=user → prompt", () => { + it("maps prompt text and meta", () => { + const event = mapPiEventPayload({ + piEvent: "message_end", + sessionId: "pi-sess-msg", + entryId: "entry-u1", + role: "user", + text: "Run tests", + cwd: "/tmp/repo", + custom_field: "keep-me", + ts: "2026-06-01T12:01:00Z", + }); + + expect(event).not.toBeNull(); + expect(event?.source).toBe("pi"); + expect(event?.event_type).toBe("prompt"); + expect(event?.payload.text).toBe("Run tests"); + expect(event?.event_id).toMatch(PI_EVENT_ID); + expect(event?.meta.pi_event).toBe("message_end"); + expect(event?.meta.entry_id).toBe("entry-u1"); + expect((event?.meta.pi_fields as Record | undefined)?.custom_field).toBe( + "keep-me", + ); + }); + + it("returns null for empty text", () => { + expect( + mapPiEventPayload({ + piEvent: "message_end", + sessionId: "pi-sess-msg", + entryId: "entry-u1", + role: "user", + text: " ", + }), + ).toBeNull(); + }); + + it("returns null without entryId", () => { + expect( + mapPiEventPayload({ + piEvent: "message_end", + sessionId: "pi-sess-msg", + role: "user", + text: "hello", + }), + ).toBeNull(); + }); + }); + + describe("message_end role=assistant → assistant", () => { + it("maps assistant text", () => { + const event = mapPiEventPayload({ + piEvent: "message_end", + sessionId: "pi-sess-msg", + entryId: "entry-a1", + role: "assistant", + text: "All done", + ts: "2026-06-01T12:02:00Z", + }); + + expect(event).not.toBeNull(); + expect(event?.event_type).toBe("assistant"); + expect(event?.payload.text).toBe("All done"); + expect(event?.event_id).toMatch(PI_EVENT_ID); + }); + }); + + describe("turn_end → assistant", () => { + it("maps turn_end text", () => { + const event = mapPiEventPayload({ + piEvent: "turn_end", + sessionId: "pi-sess-turn", + entryId: "entry-t1", + text: "Turn complete", + ts: "2026-06-01T12:03:00Z", + }); + expect(event?.event_type).toBe("assistant"); + expect(event?.payload.text).toBe("Turn complete"); + expect(event?.event_id).toMatch(PI_EVENT_ID); + }); + + it("does not map agent_end (D2: extension emits message_end only)", () => { + expect(MAPPABLE_PI_EVENTS.has("agent_end")).toBe(false); + expect( + mapPiEventPayload({ + piEvent: "agent_end", + sessionId: "pi-sess-agent", + entryId: "entry-ag1", + text: "Agent finished", + ts: "2026-06-01T12:04:00Z", + }), + ).toBeNull(); + }); + }); + + describe("tool_call → tool_call", () => { + it("maps tool name, input, and toolCallId", () => { + const event = mapPiEventPayload({ + piEvent: "tool_call", + sessionId: "pi-sess-tool", + toolCallId: "tc-1", + toolName: "bash", + toolInput: { command: "pnpm test" }, + ts: "2026-06-01T12:05:00Z", + }); + + expect(event).not.toBeNull(); + expect(event?.event_type).toBe("tool_call"); + expect(event?.payload.tool_name).toBe("bash"); + expect(event?.payload.tool_input).toEqual({ command: "pnpm test" }); + expect(event?.event_id).toMatch(PI_EVENT_ID); + expect(event?.meta.tool_call_id).toBe("tc-1"); + }); + + it("defaults tool_input to {} when missing", () => { + const event = mapPiEventPayload({ + piEvent: "tool_call", + sessionId: "pi-sess-tool", + toolCallId: "tc-2", + toolName: "read", + ts: "2026-06-01T12:05:00Z", + }); + expect(event?.payload.tool_input).toEqual({}); + }); + + it("returns null for missing toolName", () => { + expect( + mapPiEventPayload({ + piEvent: "tool_call", + sessionId: "pi-sess-tool", + toolCallId: "tc-3", + }), + ).toBeNull(); + }); + + it("returns null without toolCallId or entryId", () => { + expect( + mapPiEventPayload({ + piEvent: "tool_call", + sessionId: "pi-sess-tool", + toolName: "bash", + }), + ).toBeNull(); + }); + }); + + describe("tool_result → tool_result (isError)", () => { + it("maps ok result", () => { + const event = mapPiEventPayload({ + piEvent: "tool_result", + sessionId: "pi-sess-tool", + toolCallId: "tc-1", + toolName: "bash", + toolOutput: { exit_code: 0 }, + isError: false, + ts: "2026-06-01T12:06:00Z", + }); + + expect(event).not.toBeNull(); + expect(event?.event_type).toBe("tool_result"); + expect(event?.payload.status).toBe("ok"); + expect(event?.payload.tool_output).toEqual({ exit_code: 0 }); + expect(event?.payload.tool_error).toBeNull(); + expect(event?.event_id).toMatch(PI_EVENT_ID); + }); + + it("maps isError result", () => { + const event = mapPiEventPayload({ + piEvent: "tool_result", + sessionId: "pi-sess-tool", + toolCallId: "tc-err", + toolName: "bash", + isError: true, + error: { message: "1 failed" }, + ts: "2026-06-01T12:07:00Z", + }); + + expect(event).not.toBeNull(); + expect(event?.event_type).toBe("tool_result"); + expect(event?.payload.status).toBe("error"); + expect(event?.payload.tool_output).toBeNull(); + expect(event?.payload.error).toEqual({ message: "1 failed" }); + expect(event?.payload.tool_error).toEqual({ message: "1 failed" }); + expect(event?.event_id).toMatch(PI_EVENT_ID); + }); + }); + + describe("skip cases", () => { + it("returns null for unsupported event type", () => { + expect( + mapPiEventPayload({ + piEvent: "before_agent_start", + sessionId: "pi-sess-1", + }), + ).toBeNull(); + }); + + it("returns null for missing sessionId", () => { + expect( + mapPiEventPayload({ + piEvent: "session_start", + }), + ).toBeNull(); + }); + + it("returns null for empty sessionId", () => { + expect( + mapPiEventPayload({ + piEvent: "session_start", + sessionId: " ", + }), + ).toBeNull(); + }); + + it("returns null for session_before_compact (flush-only, not transcript)", () => { + expect( + mapPiEventPayload({ + piEvent: "session_before_compact", + sessionId: "pi-sess-1", + ts: "2026-06-01T12:00:00Z", + }), + ).toBeNull(); + }); + + it("returns null for unknown role on message_end", () => { + expect( + mapPiEventPayload({ + piEvent: "message_end", + sessionId: "pi-sess-1", + entryId: "e1", + role: "system", + text: "nope", + }), + ).toBeNull(); + }); + }); + + describe("deterministic event ids", () => { + it("produces identical ids for identical payloads", () => { + const payload = { + piEvent: "message_end", + sessionId: "pi-sess-stable", + entryId: "entry-stable-1", + role: "user", + text: "hello", + ts: "2026-06-01T12:00:00Z", + }; + const first = mapPiEventPayload(payload); + const second = mapPiEventPayload(payload); + expect(first?.event_id).toBe(second?.event_id); + expect(first?.event_id).toMatch(PI_EVENT_ID); + }); + + it("event_id is invariant to ts (different or absent)", () => { + // Dedup key must not incorporate wall-clock ts — retries with a fresh + // clock or missing ts must collapse to the same raw-event identity. + const base = { + piEvent: "message_end", + sessionId: "pi-sess-ts-invariant", + entryId: "entry-ts-1", + role: "assistant", + text: "same logical message", + }; + const withTsA = mapPiEventPayload({ ...base, ts: "2026-01-01T00:00:00Z" }); + const withTsB = mapPiEventPayload({ ...base, ts: "2026-12-31T23:59:59Z" }); + const withoutTs = mapPiEventPayload({ ...base }); + expect(withTsA?.event_id).toMatch(PI_EVENT_ID); + expect(withTsB?.event_id).toBe(withTsA?.event_id); + expect(withoutTs?.event_id).toBe(withTsA?.event_id); + // ts itself may differ; only event_id must be stable. + expect(withTsA?.ts).not.toBe(withTsB?.ts); + }); + + it("uses the pi_evt_ hashed format", () => { + const event = mapPiEventPayload({ + piEvent: "tool_call", + sessionId: "S", + toolCallId: "T", + toolName: "read", + ts: "2026-06-01T12:00:00Z", + }); + expect(event?.event_id).toMatch(PI_EVENT_ID); + }); + }); + + describe("fork id change → new stream identity", () => { + it("different sessionId yields different event_id and session_id", () => { + const base = { + piEvent: "message_end" as const, + entryId: "entry-same", + role: "user", + text: "same text", + ts: "2026-06-01T12:00:00Z", + }; + const parent = mapPiEventPayload({ ...base, sessionId: "sess-parent" }); + const fork = mapPiEventPayload({ ...base, sessionId: "sess-fork" }); + + expect(parent?.session_id).toBe("sess-parent"); + expect(fork?.session_id).toBe("sess-fork"); + expect(parent?.event_id).toMatch(PI_EVENT_ID); + expect(fork?.event_id).toMatch(PI_EVENT_ID); + expect(parent?.event_id).not.toBe(fork?.event_id); + }); + + it("envelope stream identity follows the forked session id", () => { + const parentEnv = buildRawEventEnvelopeFromPiEvent({ + piEvent: "session_start", + sessionId: "sess-parent", + ts: "2026-06-01T12:00:00Z", + }); + const forkEnv = buildRawEventEnvelopeFromPiEvent({ + piEvent: "session_start", + sessionId: "sess-fork", + ts: "2026-06-01T12:00:00Z", + }); + + expect(parentEnv?.session_stream_id).toBe("sess-parent"); + expect(forkEnv?.session_stream_id).toBe("sess-fork"); + expect(parentEnv?.session_stream_id).not.toBe(forkEnv?.session_stream_id); + expect(parentEnv?.source).toBe("pi"); + expect(forkEnv?.source).toBe("pi"); + }); + }); + + describe("snake_case field aliases", () => { + it("accepts session_id / pi_event / entry_id aliases", () => { + const event = mapPiEventPayload({ + pi_event: "message_end", + session_id: "pi-snake", + entry_id: "e-snake", + role: "user", + text: "aliased", + ts: "2026-06-01T12:00:00Z", + }); + expect(event?.session_id).toBe("pi-snake"); + expect(event?.event_id).toMatch(PI_EVENT_ID); + expect(event?.source).toBe("pi"); + }); + }); +}); + +// --------------------------------------------------------------------------- +// buildPiFlushSignalFromEvent +// --------------------------------------------------------------------------- + +describe("buildPiFlushSignalFromEvent", () => { + it("returns a flush signal for session_before_compact", () => { + const signal = buildPiFlushSignalFromEvent({ + piEvent: "session_before_compact", + sessionId: "pi-sess-compact", + cwd: "/tmp/repo", + project: "repo", + ts: "2026-06-01T14:00:00Z", + }); + + expect(signal).not.toBeNull(); + expect(signal?.kind).toBe("flush"); + expect(signal?.reason).toBe("session_before_compact"); + expect(signal?.source).toBe("pi"); + expect(signal?.session_id).toBe("pi-sess-compact"); + expect(signal?.ts).toBe("2026-06-01T14:00:00Z"); + // Must never look like a compaction object for pi to apply. + expect(signal && "compaction" in signal).toBe(false); + }); + + it("returns null for transcript events", () => { + expect( + buildPiFlushSignalFromEvent({ + piEvent: "session_start", + sessionId: "pi-sess-1", + }), + ).toBeNull(); + }); + + it("returns null without sessionId", () => { + expect( + buildPiFlushSignalFromEvent({ + piEvent: "session_before_compact", + }), + ).toBeNull(); + }); + + it("does not produce a raw envelope or ingest payload for compaction", () => { + const payload = { + piEvent: "session_before_compact", + sessionId: "pi-sess-compact", + ts: "2026-06-01T14:00:00Z", + }; + expect(mapPiEventPayload(payload)).toBeNull(); + expect(buildRawEventEnvelopeFromPiEvent(payload)).toBeNull(); + expect(buildIngestPayloadFromPiEvent(payload)).toBeNull(); + expect(buildPiFlushSignalFromEvent(payload)).not.toBeNull(); + }); +}); + +// --------------------------------------------------------------------------- +// buildRawEventEnvelopeFromPiEvent +// --------------------------------------------------------------------------- + +describe("buildRawEventEnvelopeFromPiEvent", () => { + it("returns null for unsupported event", () => { + expect( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "before_agent_start", + sessionId: "pi-sess-1", + }), + ).toBeNull(); + }); + + it("wraps adapter events for raw-event ingestion with source pi", () => { + const envelope = buildRawEventEnvelopeFromPiEvent({ + piEvent: "session_start", + sessionId: "pi-sess-env", + ts: "2026-06-01T12:00:00Z", + cwd: "/tmp/repo", + project: "repo", + }); + + expect(envelope).not.toBeNull(); + expect(envelope?.source).toBe("pi"); + expect(envelope?.event_type).toBe("pi.hook"); + expect(envelope?.session_stream_id).toBe("pi-sess-env"); + expect(envelope?.session_id).toBe("pi-sess-env"); + expect(envelope?.opencode_session_id).toBe("pi-sess-env"); + expect(envelope?.started_at).toBe("2026-06-01T12:00:00Z"); + expect(envelope?.event_id).toMatch(PI_EVENT_ID); + expect(envelope?.payload.type).toBe("pi.hook"); + const adapter = envelope?.payload._adapter as Record | undefined; + expect(adapter?.source).toBe("pi"); + expect(adapter?.schema_version).toBe("1.0"); + expect(adapter?.event_type).toBe("session_start"); + }); + + it("sets started_at only for session_start", () => { + const envelope = buildRawEventEnvelopeFromPiEvent({ + piEvent: "message_end", + sessionId: "pi-sess-env", + entryId: "e1", + role: "user", + text: "hi", + ts: "2026-06-01T12:01:00Z", + }); + expect(envelope?.started_at).toBeNull(); + }); +}); + +// --------------------------------------------------------------------------- +// buildIngestPayloadFromPiEvent +// --------------------------------------------------------------------------- + +describe("buildIngestPayloadFromPiEvent", () => { + it("returns null for unsupported event", () => { + expect( + buildIngestPayloadFromPiEvent({ + piEvent: "unknown", + sessionId: "pi-sess-1", + }), + ).toBeNull(); + }); + + it("wraps adapter event in session_context with source pi and all aliases", () => { + const ingest = buildIngestPayloadFromPiEvent({ + piEvent: "session_start", + sessionId: "pi-sess-xyz", + cwd: "/tmp/repo", + ts: "2026-06-01T12:00:00Z", + }); + + expect(ingest).not.toBeNull(); + const ctx = ingest?.session_context as Record; + expect(ctx.source).toBe("pi"); + expect(ctx.stream_id).toBe("pi-sess-xyz"); + expect(ctx.session_stream_id).toBe("pi-sess-xyz"); + expect(ctx.session_id).toBe("pi-sess-xyz"); + expect(ctx.opencode_session_id).toBe("pi-sess-xyz"); + + const events = ingest?.events as Array>; + expect(events).toHaveLength(1); + expect(events[0]?.type).toBe("pi.hook"); + const adapter = events[0]?._adapter as Record | undefined; + expect(adapter?.source).toBe("pi"); + expect(adapter?.event_type).toBe("session_start"); + }); + + it("sets cwd from pi payload", () => { + const ingest = buildIngestPayloadFromPiEvent({ + piEvent: "message_end", + sessionId: "pi-sess-cwd", + entryId: "e-cwd", + role: "user", + text: "hello", + cwd: "/home/user/myrepo", + ts: "2026-06-01T12:00:00Z", + }); + expect(ingest?.cwd).toBe("/home/user/myrepo"); + }); +}); + +// --------------------------------------------------------------------------- +// Attribution: pi-ingested rows carry source = "pi" +// --------------------------------------------------------------------------- + +describe("pi source attribution via recordRawEvent", () => { + let tmpDir: string; + let store: MemoryStore; + + beforeEach(() => { + tmpDir = mkdtempSync(join(tmpdir(), "codemem-pi-hooks-test-")); + const dbPath = join(tmpDir, "test.sqlite"); + const db = connect(dbPath); + initTestSchema(db); + db.close(); + store = new MemoryStore(dbPath); + }); + + afterEach(() => { + store.close(); + rmSync(tmpDir, { recursive: true, force: true }); + }); + + it("stores raw events with source=pi and creates no opencode rows", () => { + const envelope = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "message_end", + sessionId: "pi-attr-sess", + entryId: "entry-attr-1", + role: "user", + text: "attribute me to pi", + ts: "2026-06-01T15:00:00Z", + cwd: tmpDir, + project: "codemem", + }), + ); + // Explicit source from envelope — never rely on recordRawEvent default. + expect(envelope.source).toBe("pi"); + + const inserted = store.recordRawEvent({ + opencodeSessionId: envelope.session_stream_id, + source: envelope.source, + eventId: envelope.event_id, + eventType: envelope.event_type, + payload: envelope.payload, + tsWallMs: envelope.ts_wall_ms, + }); + expect(inserted).toBe(true); + + const piRows = store.db + .prepare(`SELECT source, stream_id, event_id, event_type FROM raw_events WHERE source = ?`) + .all("pi") as Array<{ + source: string; + stream_id: string; + event_id: string; + event_type: string; + }>; + expect(piRows).toHaveLength(1); + expect(piRows[0]?.source).toBe("pi"); + expect(piRows[0]?.stream_id).toBe("pi-attr-sess"); + expect(piRows[0]?.event_id).toMatch(PI_EVENT_ID); + expect(piRows[0]?.event_type).toBe("pi.hook"); + + const opencodeRows = store.db + .prepare(`SELECT COUNT(*) AS n FROM raw_events WHERE source = ?`) + .get("opencode") as { n: number }; + expect(Number(opencodeRows.n)).toBe(0); + + const sessionRows = store.db + .prepare(`SELECT source, stream_id FROM raw_event_sessions`) + .all() as Array<{ source: string; stream_id: string }>; + expect(sessionRows).toHaveLength(1); + expect(sessionRows[0]?.source).toBe("pi"); + expect(sessionRows[0]?.stream_id).toBe("pi-attr-sess"); + }); + + it("dedupes retries by (source, stream, event_id) for pi", () => { + const envelope = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "tool_call", + sessionId: "pi-dedupe-sess", + toolCallId: "tc-dedupe", + toolName: "read", + toolInput: { path: "README.md" }, + ts: "2026-06-01T15:01:00Z", + }), + ); + + const write = () => + store.recordRawEvent({ + opencodeSessionId: envelope.session_stream_id, + source: envelope.source, + eventId: envelope.event_id, + eventType: envelope.event_type, + payload: envelope.payload, + tsWallMs: envelope.ts_wall_ms, + }); + expect(write()).toBe(true); + expect(write()).toBe(false); + + const count = store.db + .prepare(`SELECT COUNT(*) AS n FROM raw_events WHERE source = ? AND stream_id = ?`) + .get("pi", "pi-dedupe-sess") as { n: number }; + expect(Number(count.n)).toBe(1); + }); + + it("forked session id creates a separate pi stream partition", () => { + for (const sessionId of ["sess-parent", "sess-fork"]) { + const envelope = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "message_end", + sessionId, + entryId: "entry-shared-logical", + role: "user", + text: "fork test", + ts: "2026-06-01T15:02:00Z", + }), + ); + store.recordRawEvent({ + opencodeSessionId: envelope.session_stream_id, + source: envelope.source, + eventId: envelope.event_id, + eventType: envelope.event_type, + payload: envelope.payload, + tsWallMs: envelope.ts_wall_ms, + }); + } + + const rows = store.db + .prepare( + `SELECT source, stream_id, event_id FROM raw_events WHERE source = ? ORDER BY stream_id`, + ) + .all("pi") as Array<{ source: string; stream_id: string; event_id: string }>; + expect(rows).toHaveLength(2); + expect(rows.map((r) => r.stream_id).sort()).toEqual(["sess-fork", "sess-parent"]); + expect(rows.every((r) => r.source === "pi")).toBe(true); + expect(new Set(rows.map((r) => r.event_id)).size).toBe(2); + + const opencodeCount = store.db + .prepare(`SELECT COUNT(*) AS n FROM raw_events WHERE source = ?`) + .get("opencode") as { n: number }; + expect(Number(opencodeCount.n)).toBe(0); + }); +}); + +describe("ingestRawEvents accepts pi envelopes", () => { + let tmpDir: string; + let store: MemoryStore; + + beforeEach(() => { + tmpDir = mkdtempSync(join(tmpdir(), "codemem-pi-hooks-ingest-")); + const dbPath = join(tmpDir, "test.sqlite"); + const db = connect(dbPath); + initTestSchema(db); + db.close(); + store = new MemoryStore(dbPath); + }); + + afterEach(() => { + store.close(); + rmSync(tmpDir, { recursive: true, force: true }); + }); + + it("persists source pi with no opencode rows", () => { + const envelope = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "message_end", + sessionId: "pi-canonical-1", + entryId: "entry-1", + role: "user", + text: "hello", + ts: "2026-06-01T16:00:00Z", + }), + ); + const result = ingestRawEvents(store, envelope); + expect(result.inserted).toBe(1); + expect(result.skipped).toBe(0); + + const row = store.db.prepare(`SELECT source, stream_id, event_id FROM raw_events`).get() as { + source: string; + stream_id: string; + event_id: string; + }; + expect(row.source).toBe("pi"); + expect(row.stream_id).toBe("pi-canonical-1"); + expect(row.event_id).toBe(envelope.event_id); + + const opencodeCount = store.db + .prepare(`SELECT COUNT(*) AS n FROM raw_events WHERE source = ?`) + .get("opencode") as { n: number }; + expect(Number(opencodeCount.n)).toBe(0); + }); + + it.each([ + { sessionId: "session/with/slash", entryId: "entry+1" }, + { sessionId: "s".repeat(80), entryId: "e+".repeat(40) }, + ])("hashes unconstrained identity into an ingest-safe event_id", ({ sessionId, entryId }) => { + const envelope = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "message_end", + sessionId, + entryId, + role: "user", + text: "hello", + ts: "2026-06-01T16:00:00Z", + }), + ); + expect(envelope.event_id).toMatch(/^[A-Za-z0-9._:-]+$/); + expect(envelope.event_id.length).toBeLessThanOrEqual(128); + const result = ingestRawEvents(store, envelope); + expect(result.inserted).toBe(1); + expect(result.skipped).toBe(0); + }); + + it("persists reload then exit as distinct session_end events", () => { + const sessionId = "pi-reload-exit"; + const reload = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "session_shutdown", + sessionId, + entryId: "session_end", + reason: "reload", + ts: "2026-06-01T16:00:00Z", + }), + ); + const exit = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "session_shutdown", + sessionId, + entryId: "session_end", + reason: "quit", + ts: "2026-06-01T16:01:00Z", + }), + ); + expect(reload.event_id).not.toBe(exit.event_id); + expect(ingestRawEvents(store, reload)).toMatchObject({ inserted: 1, skipped: 0 }); + expect(ingestRawEvents(store, exit)).toMatchObject({ inserted: 1, skipped: 0 }); + expect(ingestRawEvents(store, exit)).toMatchObject({ inserted: 0, skipped: 1 }); + }); + + it("persists reload then reload as distinct session_end events", () => { + const sessionId = "pi-reload-reload"; + const first = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "session_shutdown", + sessionId, + entryId: "session_end", + reason: "reload", + ts: "2026-06-01T16:00:00Z", + }), + ); + const second = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "session_shutdown", + sessionId, + entryId: "session_end", + reason: "reload", + ts: "2026-06-01T16:01:00Z", + }), + ); + expect(first.event_id).not.toBe(second.event_id); + expect(ingestRawEvents(store, first)).toMatchObject({ inserted: 1, skipped: 0 }); + expect(ingestRawEvents(store, second)).toMatchObject({ inserted: 1, skipped: 0 }); + expect(ingestRawEvents(store, first)).toMatchObject({ inserted: 0, skipped: 1 }); + expect(ingestRawEvents(store, second)).toMatchObject({ inserted: 0, skipped: 1 }); + }); + + it("distinguishes ts-less same-reason shutdowns by generated time", () => { + vi.useFakeTimers(); + try { + vi.setSystemTime(new Date("2026-06-01T17:00:00Z")); + const first = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "session_shutdown", + sessionId: "pi-no-ts-reload", + entryId: "session_end", + reason: "reload", + }), + ); + vi.setSystemTime(new Date("2026-06-01T17:00:01Z")); + const second = requireEnvelope( + buildRawEventEnvelopeFromPiEvent({ + piEvent: "session_shutdown", + sessionId: "pi-no-ts-reload", + entryId: "session_end", + reason: "reload", + }), + ); + expect(first.event_id).not.toBe(second.event_id); + expect(ingestRawEvents(store, first)).toMatchObject({ inserted: 1, skipped: 0 }); + expect(ingestRawEvents(store, second)).toMatchObject({ inserted: 1, skipped: 0 }); + } finally { + vi.useRealTimers(); + } + }); +}); diff --git a/packages/core/src/pi-hooks.ts b/packages/core/src/pi-hooks.ts new file mode 100644 index 000000000..d591f4949 --- /dev/null +++ b/packages/core/src/pi-hooks.ts @@ -0,0 +1,456 @@ +/** + * Pi extension event payload mapping. + * + * Normalizes pi coding-agent extension events into AdapterEvent v1 envelopes + * for the shared raw-event sweeper pipeline. Mirrors claude-hooks.ts / + * codex-hooks.ts structure with source hard-coded to "pi". + * + * Entry points: + * mapPiEventPayload(payload) → adapter event or null + * buildRawEventEnvelopeFromPiEvent(...) → raw event envelope or null + * buildIngestPayloadFromPiEvent(...) → ingest payload or null + * buildPiFlushSignalFromEvent(...) → flush signal (compaction only) + * + * session_before_compact is observe-only: it never becomes a transcript event + * and never returns a compaction object for pi to apply. + */ + +import { createHash } from "node:crypto"; +import { normalizeProjectLabel, resolveHookProject } from "./claude-hooks.js"; + +// --------------------------------------------------------------------------- +// Constants +// --------------------------------------------------------------------------- + +/** Pi extension events that map to AdapterEvent v1 transcript types. */ +export const MAPPABLE_PI_EVENTS = new Set([ + "session_start", + "session_shutdown", + "message_end", + "turn_end", + "tool_call", + "tool_result", +]); + +/** Events that only signal a boundary flush (never stored as transcript). */ +export const PI_FLUSH_ONLY_EVENTS = new Set(["session_before_compact"]); + +/** Frozen discriminator for Pi's derived event identity contract. */ +export const PI_EVENT_ID_ALGO = "pi/1"; +// --------------------------------------------------------------------------- +// Timestamp helpers +// --------------------------------------------------------------------------- + +function nowIso(): string { + return new Date().toISOString().replace(/\.(\d{3})\d*Z$/, ".$1Z"); +} + +function normalizeIsoTs(value: unknown): string | null { + if (typeof value !== "string") return null; + const text = value.trim(); + if (!text) return null; + const hasTimezone = + /[Zz]$/.test(text) || /[+-]\d{2}:\d{2}$/.test(text) || /[+-]\d{4}$/.test(text); + const parsed = new Date(hasTimezone ? text : `${text}Z`); + if (Number.isNaN(parsed.getTime())) return null; + const hasFractional = /\.\d+([Zz+-]|$)/.test(text); + return hasFractional + ? parsed.toISOString().replace(/\.(\d{3})Z$/, ".$1000Z") + : parsed.toISOString().replace(/\.\d{3}Z$/, "Z"); +} + +function isoToWallMs(value: string): number { + return new Date(value).getTime(); +} + +// --------------------------------------------------------------------------- +// Coercion helpers +// --------------------------------------------------------------------------- + +function coerceString(value: unknown): string { + return typeof value === "string" ? value.trim() : ""; +} + +/** Read the first defined field among camelCase / snake_case aliases. */ +function field(payload: Record, ...keys: string[]): unknown { + for (const key of keys) { + if (Object.hasOwn(payload, key) && payload[key] !== undefined) return payload[key]; + } + return undefined; +} + +function coerceSessionId(payload: Record): string | null { + const value = coerceString(field(payload, "sessionId", "session_id")); + return value || null; +} + +function coercePiEventName(payload: Record): string { + return coerceString(field(payload, "piEvent", "pi_event", "event", "type")); +} + +function objectOrEmpty(value: unknown): Record { + return value != null && typeof value === "object" && !Array.isArray(value) + ? (value as Record) + : {}; +} + +function coerceBool(value: unknown): boolean { + if (typeof value === "boolean") return value; + if (typeof value === "number") return value !== 0; + if (typeof value === "string") { + const t = value.trim().toLowerCase(); + return t === "true" || t === "1" || t === "yes"; + } + return false; +} + +/** + * Deterministic event id: `pi_evt_` + sha256(identity).hex.slice(0, 24). + * Same logical event must yield the same id across HTTP/CLI retries. + */ +function buildPiEventId(...parts: string[]): string { + const digest = createHash("sha256").update(parts.join("|"), "utf-8").digest("hex").slice(0, 24); + return `pi_evt_${digest}`; +} + +// --------------------------------------------------------------------------- +// mapPiEventPayload +// --------------------------------------------------------------------------- + +export interface PiHookAdapterEvent { + schema_version: "1.0"; + source: "pi"; + session_id: string; + event_id: string; + event_type: string; + ts: string; + ordering_confidence: "low"; + cwd: string | null; + payload: Record; + meta: Record; +} + +/** + * Map a pi extension event payload to a normalized AdapterEvent v1. + * Returns null if the event type is unsupported, flush-only, or required + * fields are missing. + */ +export function mapPiEventPayload(payload: Record): PiHookAdapterEvent | null { + const piEvent = coercePiEventName(payload); + if (!piEvent) return null; + + // Compaction is observe-only — never a transcript AdapterEvent. + if (PI_FLUSH_ONLY_EVENTS.has(piEvent)) return null; + if (!MAPPABLE_PI_EVENTS.has(piEvent)) return null; + + const sessionId = coerceSessionId(payload); + if (!sessionId) return null; + + const normalizedRawTs = normalizeIsoTs(field(payload, "ts", "timestamp")); + const ts = normalizedRawTs ?? nowIso(); + + const entryId = coerceString(field(payload, "entryId", "entry_id")); + const toolCallId = coerceString(field(payload, "toolCallId", "tool_call_id")); + const cwdRaw = field(payload, "cwd"); + const cwd = typeof cwdRaw === "string" ? cwdRaw : null; + + const consumed = new Set([ + "piEvent", + "pi_event", + "event", + "type", + "sessionId", + "session_id", + "entryId", + "entry_id", + "toolCallId", + "tool_call_id", + "cwd", + "ts", + "timestamp", + "project", + ]); + + let eventType: string; + let eventPayload: Record; + let idPart: string | null = null; + + if (piEvent === "session_start") { + eventType = "session_start"; + eventPayload = {}; + idPart = entryId || "session_start"; + } else if (piEvent === "session_shutdown") { + const reason = field(payload, "reason"); + eventType = "session_end"; + eventPayload = { reason: reason ?? null }; + // Payload ts keeps retries stable; a generated ts falls back so ts-less + // same-reason shutdowns stay distinct (mirrors claude-hooks eventIdTsSeed). + idPart = + entryId && entryId !== "session_end" ? entryId : `session_end:${coerceString(reason)}:${ts}`; + consumed.add("reason"); + } else if (piEvent === "message_end") { + const role = coerceString(field(payload, "role")).toLowerCase(); + const text = coerceString(field(payload, "text", "content", "prompt")); + if (!text) return null; + if (!entryId) return null; + if (role === "user") { + eventType = "prompt"; + eventPayload = { text }; + } else if (role === "assistant") { + eventType = "assistant"; + eventPayload = { text }; + } else { + return null; + } + idPart = entryId; + consumed.add("role"); + consumed.add("text"); + consumed.add("content"); + consumed.add("prompt"); + } else if (piEvent === "turn_end") { + // Design D2: turn_end → assistant. Extension emits message_end for completed + // assistant turns; agent_end is intentionally not in the mappable set. + const text = coerceString(field(payload, "text", "content")); + if (!text) return null; + if (!entryId && !toolCallId) return null; + eventType = "assistant"; + eventPayload = { text }; + idPart = entryId || toolCallId; + consumed.add("role"); + consumed.add("text"); + consumed.add("content"); + } else if (piEvent === "tool_call") { + const toolName = coerceString(field(payload, "toolName", "tool_name", "name")); + if (!toolName) return null; + if (!toolCallId && !entryId) return null; + const toolInput = objectOrEmpty(field(payload, "toolInput", "tool_input", "args", "input")); + eventType = "tool_call"; + eventPayload = { tool_name: toolName, tool_input: toolInput }; + idPart = toolCallId || entryId; + consumed.add("toolName"); + consumed.add("tool_name"); + consumed.add("name"); + consumed.add("toolInput"); + consumed.add("tool_input"); + consumed.add("args"); + consumed.add("input"); + } else if (piEvent === "tool_result") { + const toolName = coerceString(field(payload, "toolName", "tool_name", "name")); + if (!toolName) return null; + if (!toolCallId && !entryId) return null; + const toolInput = objectOrEmpty(field(payload, "toolInput", "tool_input", "args", "input")); + const isError = coerceBool(field(payload, "isError", "is_error")); + const toolOutput = field(payload, "toolOutput", "tool_output", "output", "result") ?? null; + const error = field(payload, "error", "tool_error") ?? null; + eventType = "tool_result"; + if (isError) { + eventPayload = { + tool_name: toolName, + status: "error", + tool_input: toolInput, + tool_output: null, + tool_error: error ?? true, + error: error ?? true, + }; + } else { + eventPayload = { + tool_name: toolName, + status: "ok", + tool_input: toolInput, + tool_output: toolOutput, + tool_error: null, + }; + } + // tool_result ids prefer toolCallId so call/result pair on the same id root. + idPart = toolCallId ? `${toolCallId}:result` : `${entryId}:result`; + consumed.add("toolName"); + consumed.add("tool_name"); + consumed.add("name"); + consumed.add("toolInput"); + consumed.add("tool_input"); + consumed.add("args"); + consumed.add("input"); + consumed.add("toolOutput"); + consumed.add("tool_output"); + consumed.add("output"); + consumed.add("result"); + consumed.add("isError"); + consumed.add("is_error"); + consumed.add("error"); + consumed.add("tool_error"); + } else { + return null; + } + + if (!idPart) return null; + + const meta: Record = { + event_id_algo: PI_EVENT_ID_ALGO, + pi_event: piEvent, + ordering_confidence: "low", + }; + if (entryId) meta.entry_id = entryId; + if (toolCallId) meta.tool_call_id = toolCallId; + if (normalizedRawTs === null) meta.ts_normalized = "generated"; + + const unknown: Record = {}; + for (const [key, value] of Object.entries(payload)) { + if (!consumed.has(key)) unknown[key] = value; + } + if (Object.keys(unknown).length > 0) meta.pi_fields = unknown; + + return { + schema_version: "1.0", + source: "pi", + session_id: sessionId, + event_id: buildPiEventId(sessionId, piEvent, idPart), + event_type: eventType, + ts, + ordering_confidence: "low", + cwd, + payload: eventPayload, + meta, + }; +} + +// --------------------------------------------------------------------------- +// Flush signal (session_before_compact — observe only) +// --------------------------------------------------------------------------- + +export interface PiFlushSignal { + kind: "flush"; + reason: "session_before_compact"; + source: "pi"; + session_id: string; + ts: string; + cwd: string | null; + project: string | null; +} + +/** + * Build a flush signal for pi compaction boundaries. + * Returns null unless the payload is a session_before_compact event with a + * session id. Never produces a compaction object for pi to apply. + */ +export function buildPiFlushSignalFromEvent( + payload: Record, +): PiFlushSignal | null { + const piEvent = coercePiEventName(payload); + if (!PI_FLUSH_ONLY_EVENTS.has(piEvent)) return null; + + const sessionId = coerceSessionId(payload); + if (!sessionId) return null; + + const normalizedRawTs = normalizeIsoTs(field(payload, "ts", "timestamp")); + const ts = normalizedRawTs ?? nowIso(); + const cwdRaw = field(payload, "cwd"); + const cwd = typeof cwdRaw === "string" ? cwdRaw : null; + const project = + resolveHookProject(cwd, field(payload, "project")) ?? + normalizeProjectLabel(field(payload, "project")); + + return { + kind: "flush", + reason: "session_before_compact", + source: "pi", + session_id: sessionId, + ts, + cwd, + project, + }; +} + +// --------------------------------------------------------------------------- +// buildRawEventEnvelopeFromPiEvent +// --------------------------------------------------------------------------- + +export interface PiHookRawEventEnvelope { + session_stream_id: string; + session_id: string; + opencode_session_id: string; + source: "pi"; + event_id: string; + event_type: "pi.hook"; + payload: Record; + ts_wall_ms: number; + cwd: string | null; + project: string | null; + started_at: string | null; +} + +/** + * Build a raw event envelope from a pi extension event payload. + * Returns null if the payload is unsupported, flush-only, or missing fields. + * Source is always the literal "pi" — never falls through to a default. + */ +export function buildRawEventEnvelopeFromPiEvent( + piPayload: Record, +): PiHookRawEventEnvelope | null { + const adapterEvent = mapPiEventPayload(piPayload); + if (adapterEvent === null) return null; + + const sessionId = adapterEvent.session_id.trim(); + if (!sessionId) return null; + const ts = adapterEvent.ts.trim(); + if (!ts) return null; + + const cwdRaw = field(piPayload, "cwd"); + const cwd = typeof cwdRaw === "string" ? cwdRaw : null; + const project = + resolveHookProject(cwd, field(piPayload, "project")) ?? + normalizeProjectLabel(field(piPayload, "project")); + const piEvent = coercePiEventName(piPayload); + + return { + session_stream_id: sessionId, + session_id: sessionId, + opencode_session_id: sessionId, + source: "pi", + event_id: adapterEvent.event_id, + event_type: "pi.hook", + payload: { + type: "pi.hook", + timestamp: ts, + _adapter: adapterEvent, + }, + ts_wall_ms: isoToWallMs(ts), + cwd, + project, + started_at: piEvent === "session_start" ? ts : null, + }; +} + +// --------------------------------------------------------------------------- +// buildIngestPayloadFromPiEvent +// --------------------------------------------------------------------------- + +/** + * Build an ingest pipeline payload from a pi extension event. + * Used by the direct-ingest path. Source is always the literal "pi". + * Returns null if the payload is unsupported or flush-only. + */ +export function buildIngestPayloadFromPiEvent( + piPayload: Record, +): Record | null { + const adapterEvent = mapPiEventPayload(piPayload); + if (adapterEvent === null) return null; + + const sessionId = adapterEvent.session_id; + return { + cwd: field(piPayload, "cwd") ?? null, + events: [ + { + type: "pi.hook", + timestamp: adapterEvent.ts, + _adapter: adapterEvent, + }, + ], + session_context: { + source: "pi", + stream_id: sessionId, + session_stream_id: sessionId, + session_id: sessionId, + opencode_session_id: sessionId, + }, + }; +} diff --git a/packages/viewer-server/src/index.ts b/packages/viewer-server/src/index.ts index 2c5dad5ef..bc677c819 100644 --- a/packages/viewer-server/src/index.ts +++ b/packages/viewer-server/src/index.ts @@ -28,6 +28,7 @@ import { configRoutes } from "./routes/config.js"; import { diagnosticsRoutes } from "./routes/diagnostics.js"; import { healthRoutes } from "./routes/health.js"; import { memoryRoutes } from "./routes/memory.js"; +import { memoryToolRoutes } from "./routes/memory-tools.js"; import { observerStatusRoutes } from "./routes/observer-status.js"; import { packTransportRoutes } from "./routes/pack.js"; import { rawEventsRoutes } from "./routes/raw-events.js"; @@ -179,6 +180,7 @@ export function createApp(opts?: AppOptions) { app.route("/", diagnosticsRoutes(storeFactory)); app.route("/", statsRoutes(storeFactory)); app.route("/", memoryRoutes(storeFactory)); + app.route("/", memoryToolRoutes(storeFactory)); app.route("/", packTransportRoutes(storeFactory)); app.route( "/", diff --git a/packages/viewer-server/src/routes/memory-tools.test.ts b/packages/viewer-server/src/routes/memory-tools.test.ts new file mode 100644 index 000000000..cc9b0a75d --- /dev/null +++ b/packages/viewer-server/src/routes/memory-tools.test.ts @@ -0,0 +1,790 @@ +/** + * Route tests for pi-hooks ingest + memory tool-support HTTP twins. + * + * Contracts mirror packages/mcp-server tool handlers against MemoryStore. + */ + +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + initTestSchema, + insertTestSession, + MemoryStore, + type RawEventSweeper, +} from "@codemem/core"; +import Database from "better-sqlite3"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { createApp } from "../index.js"; + +// Keep route tests hermetic: no embedding model downloads on the hot path. +// Save/restore so sibling suites in a shared worker are unaffected. +let savedEmbeddingDisabled: string | undefined; +beforeAll(() => { + savedEmbeddingDisabled = process.env.CODEMEM_EMBEDDING_DISABLED; + process.env.CODEMEM_EMBEDDING_DISABLED = "1"; +}); +afterAll(() => { + if (savedEmbeddingDisabled === undefined) delete process.env.CODEMEM_EMBEDDING_DISABLED; + else process.env.CODEMEM_EMBEDDING_DISABLED = savedEmbeddingDisabled; +}); +function createTestStore(): { store: MemoryStore; cleanup: () => void } { + const tmpDir = mkdtempSync(join(tmpdir(), "codemem-memory-tools-test-")); + const dbPath = join(tmpDir, "test.sqlite"); + const rawDb = new Database(dbPath); + initTestSchema(rawDb); + rawDb + .prepare( + "INSERT INTO sync_device(device_id, public_key, fingerprint, created_at) VALUES (?, ?, ?, ?)", + ) + .run("test-device-001", "test-public-key", "test-fingerprint", new Date().toISOString()); + rawDb.close(); + const store = new MemoryStore(dbPath); + return { + store, + cleanup: () => { + store.close(); + rmSync(tmpDir, { recursive: true, force: true }); + }, + }; +} + +function createTestApp(opts?: { sweeper?: Partial | null }) { + let store: MemoryStore | null = null; + let storeCleanup: (() => void) | null = null; + const staticDir = mkdtempSync(join(tmpdir(), "codemem-memory-tools-static-")); + writeFileSync(join(staticDir, "index.html"), "test"); + const previousStaticDir = process.env.CODEMEM_VIEWER_STATIC_DIR; + process.env.CODEMEM_VIEWER_STATIC_DIR = staticDir; + const storeFactory = () => { + if (!store) { + const created = createTestStore(); + store = created.store; + storeCleanup = created.cleanup; + } + return store; + }; + const app = createApp({ + storeFactory, + sweeper: (opts?.sweeper ?? null) as RawEventSweeper | null, + }); + return { + app, + getStore: () => store, + ensureStore: () => storeFactory(), + cleanup: () => { + storeCleanup?.(); + store = null; + storeCleanup = null; + if (previousStaticDir == null) delete process.env.CODEMEM_VIEWER_STATIC_DIR; + else process.env.CODEMEM_VIEWER_STATIC_DIR = previousStaticDir; + rmSync(staticDir, { recursive: true, force: true }); + }, + }; +} + +function jsonHeaders(): Record { + return { + "Content-Type": "application/json", + Origin: "http://127.0.0.1:38888", + }; +} + +function seedMemories(store: MemoryStore): { sessionId: number; ids: number[] } { + const sessionId = insertTestSession(store.db); + // Ensure project matches insertTestSession default so project filters work. + store.db.prepare("UPDATE sessions SET project = ? WHERE id = ?").run("test-project", sessionId); + const ids = [ + store.remember( + sessionId, + "discovery", + "Database migration guide", + "How to run migrations", + 0.9, + ), + store.remember(sessionId, "feature", "Auth system", "JWT tokens and refresh flow", 0.8), + store.remember(sessionId, "decision", "Use SQLite", "Pick sqlite for local store", 0.7), + store.remember(sessionId, "bugfix", "Fix race in cache", "Race on concurrent writes", 0.6), + ]; + return { sessionId, ids }; +} + +// --------------------------------------------------------------------------- +// 4.1 POST /api/pi-hooks +// --------------------------------------------------------------------------- + +describe("POST /api/pi-hooks", () => { + // hashed id format shared with @codemem/core (pi/1 algo) + const PI_EVENT_ID = /^pi_evt_[0-9a-f]{24}$/; + it("records a pi event with source=pi and nudges the sweeper with (stream, pi)", async () => { + const nudge = vi.fn(); + const { app, getStore, cleanup } = createTestApp({ + sweeper: { nudge } as Partial, + }); + try { + const payload = { + piEvent: "session_start", + sessionId: "pi-sess-route-1", + cwd: "/tmp/pi-proj", + ts: "2026-04-01T12:00:00.000Z", + }; + const res = await app.request("/api/pi-hooks", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify(payload), + }); + expect(res.status).toBe(200); + expect(await res.json()).toEqual({ inserted: 1, skipped: 0 }); + + const store = getStore(); + if (!store) throw new Error("store missing"); + + const eventRow = store.db + .prepare( + "SELECT source, stream_id, event_id, event_type FROM raw_events WHERE stream_id = ?", + ) + .get("pi-sess-route-1") as { + source: string; + stream_id: string; + event_id: string; + event_type: string; + }; + expect(eventRow.source).toBe("pi"); + expect(eventRow.stream_id).toBe("pi-sess-route-1"); + expect(eventRow.event_id).toMatch(PI_EVENT_ID); + expect(eventRow.event_type).toBe("pi.hook"); + + const sessionRow = store.db + .prepare("SELECT source, stream_id FROM raw_event_sessions WHERE stream_id = ?") + .get("pi-sess-route-1") as { source: string; stream_id: string }; + expect(sessionRow.source).toBe("pi"); + + const opencodeCount = store.db + .prepare("SELECT COUNT(*) AS n FROM raw_events WHERE source = 'opencode'") + .get() as { n: number }; + expect(opencodeCount.n).toBe(0); + + expect(nudge).toHaveBeenCalledWith("pi-sess-route-1", "pi"); + } finally { + cleanup(); + } + }); + + it("dedupes identical pi events on retry", async () => { + const nudge = vi.fn(); + const { app, getStore, cleanup } = createTestApp({ + sweeper: { nudge } as Partial, + }); + try { + const payload = { + piEvent: "message_end", + sessionId: "pi-sess-dedupe", + entryId: "entry-42", + role: "user", + text: "hello from pi", + ts: "2026-04-01T12:01:00.000Z", + }; + const first = await app.request("/api/pi-hooks", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify(payload), + }); + expect(await first.json()).toEqual({ inserted: 1, skipped: 0 }); + + const second = await app.request("/api/pi-hooks", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify(payload), + }); + expect(await second.json()).toEqual({ inserted: 0, skipped: 1 }); + + const store = getStore(); + if (!store) throw new Error("store missing"); + const count = store.db + .prepare("SELECT COUNT(*) AS n FROM raw_events WHERE source = 'pi' AND stream_id = ?") + .get("pi-sess-dedupe") as { n: number }; + expect(count.n).toBe(1); + expect(nudge).toHaveBeenCalledTimes(2); + expect(nudge).toHaveBeenNthCalledWith(1, "pi-sess-dedupe", "pi"); + expect(nudge).toHaveBeenNthCalledWith(2, "pi-sess-dedupe", "pi"); + } finally { + cleanup(); + } + }); + + it("skips unsupported / flush-only pi events without writing rows", async () => { + const nudge = vi.fn(); + const { app, getStore, ensureStore, cleanup } = createTestApp({ + sweeper: { nudge } as Partial, + }); + try { + ensureStore(); + const res = await app.request("/api/pi-hooks", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify({ + piEvent: "session_before_compact", + sessionId: "pi-sess-compact", + }), + }); + expect(res.status).toBe(200); + expect(await res.json()).toEqual({ inserted: 0, skipped: 1 }); + expect(nudge).not.toHaveBeenCalled(); + + const store = getStore(); + if (!store) throw new Error("store missing"); + const count = store.db.prepare("SELECT COUNT(*) AS n FROM raw_events").get() as { + n: number; + }; + expect(count.n).toBe(0); + } finally { + cleanup(); + } + }); + + it("does not duplicate when the same envelope is posted to /api/raw-events", async () => { + const { app, getStore, cleanup } = createTestApp(); + try { + const payload = { + piEvent: "message_end", + sessionId: "pi-sess-alias", + entryId: "entry-alias", + role: "user", + text: "alias parity", + ts: "2026-04-01T12:02:00.000Z", + }; + const first = await app.request("/api/pi-hooks", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify(payload), + }); + expect(await first.json()).toEqual({ inserted: 1, skipped: 0 }); + + const envelope = await import("@codemem/core").then((mod) => + mod.buildRawEventEnvelopeFromPiEvent(payload), + ); + expect(envelope).not.toBeNull(); + const second = await app.request("/api/raw-events", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify(envelope), + }); + const secondBody = (await second.json()) as { inserted: number; skipped: number }; + expect(secondBody.inserted).toBe(0); + expect(secondBody.skipped).toBe(1); + + const store = getStore(); + if (!store) throw new Error("store missing"); + const count = store.db + .prepare("SELECT COUNT(*) AS n FROM raw_events WHERE source = 'pi'") + .get() as { n: number }; + expect(count.n).toBe(1); + const opencode = store.db + .prepare("SELECT COUNT(*) AS n FROM raw_events WHERE source = 'opencode'") + .get() as { n: number }; + expect(opencode.n).toBe(0); + } finally { + cleanup(); + } + }); +}); +// --------------------------------------------------------------------------- +// 4.2 / 4.3 Memory tool routes vs MCP twins +// --------------------------------------------------------------------------- + +describe("memory tool routes", () => { + describe("POST /api/memories/remember (memory_remember)", () => { + it("creates a memory and returns { id }", async () => { + const { app, getStore, cleanup } = createTestApp(); + try { + const res = await app.request("/api/memories/remember", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify({ + kind: "decision", + title: "Adopt HTTP tool routes", + body: "Close the viewer gap so pi can call tools over HTTP.", + confidence: 0.85, + project: "codemem", + }), + }); + expect(res.status).toBe(200); + const body = (await res.json()) as { id: number }; + expect(typeof body.id).toBe("number"); + expect(body.id).toBeGreaterThan(0); + + const store = getStore(); + if (!store) throw new Error("store missing"); + const item = store.get(body.id); + expect(item?.title).toBe("Adopt HTTP tool routes"); + expect(item?.kind).toBe("decision"); + expect(Number(item?.active)).toBe(1); + + const session = store.db + .prepare("SELECT project, tool_version FROM sessions WHERE id = ?") + .get(item?.session_id) as { project: string; tool_version: string }; + expect(session.project).toBe("codemem"); + expect(session.tool_version).toBe("viewer-api"); + } finally { + cleanup(); + } + }); + + it("rejects invalid kind", async () => { + const { app, cleanup } = createTestApp(); + try { + const res = await app.request("/api/memories/remember", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify({ + kind: "not-a-kind", + title: "x", + body: "y", + }), + }); + expect(res.status).toBe(400); + const body = (await res.json()) as { error: string }; + expect(body.error).toMatch(/kind must be one of/); + } finally { + cleanup(); + } + }); + + it("rejects an oversized remember body", async () => { + const { app, cleanup } = createTestApp(); + try { + const res = await app.request("/api/memories/remember", { + method: "POST", + headers: { ...jsonHeaders(), "content-length": "9999999" }, + body: JSON.stringify({ kind: "decision", title: "x", body: "y" }), + }); + expect(res.status).toBe(413); + } finally { + cleanup(); + } + }); + }); + + describe("GET /api/memories/timeline (memory_timeline)", () => { + it("rejects a non-string kind filter", async () => { + const { app, cleanup } = createTestApp(); + try { + const res = await app.request( + `/api/memories/timeline?filters=${encodeURIComponent(JSON.stringify({ kind: { x: 1 } }))}`, + { headers: jsonHeaders() }, + ); + expect(res.status).toBe(400); + const body = (await res.json()) as { error: string }; + expect(body.error).toMatch(/kind must be a string/); + } finally { + cleanup(); + } + }); + + it("returns a chronological window around an anchor id", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + const { ids } = seedMemories(store); + const anchor = ids[2]; + + const res = await app.request( + `/api/memories/timeline?memory_id=${anchor}&depth_before=2&depth_after=2`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { items: Array<{ id: number }> }; + expect(Array.isArray(body.items)).toBe(true); + expect(body.items.some((item) => item.id === anchor)).toBe(true); + expect(body.items.length).toBeGreaterThanOrEqual(1); + } finally { + cleanup(); + } + }); + + it("anchors via query string like the MCP tool", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + seedMemories(store); + const res = await app.request( + "/api/memories/timeline?query=Database&depth_before=1&depth_after=1", + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { items: Array<{ title: string }> }; + expect(body.items.length).toBeGreaterThan(0); + expect(body.items.some((item) => /Database|migration/i.test(item.title))).toBe(true); + } finally { + cleanup(); + } + }); + + it("honors JSON filters.include_visibility (MCP filter surface parity)", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + const sessionId = insertTestSession(store.db); + store.db + .prepare("UPDATE sessions SET project = ? WHERE id = ?") + .run("test-project", sessionId); + const sharedId = store.remember( + sessionId, + "discovery", + "Timeline shared visibility row", + "shared body for timeline filter", + 0.9, + undefined, + { visibility: "shared" }, + ); + const privateId = store.remember( + sessionId, + "discovery", + "Timeline private visibility row", + "private body for timeline filter", + 0.9, + undefined, + { visibility: "private" }, + ); + + const filters = encodeURIComponent(JSON.stringify({ include_visibility: ["private"] })); + const res = await app.request( + `/api/memories/timeline?query=Timeline&depth_before=5&depth_after=5&filters=${filters}`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { items: Array<{ id: number; title: string }> }; + const ids = body.items.map((item) => item.id); + expect(ids).toContain(privateId); + expect(ids).not.toContain(sharedId); + } finally { + cleanup(); + } + }); + + it("rejects malformed filters JSON with 400", async () => { + const { app, cleanup } = createTestApp(); + try { + const res = await app.request("/api/memories/timeline?query=x&filters=not-json"); + expect(res.status).toBe(400); + expect(await res.json()).toEqual({ error: "filters must be valid JSON" }); + } finally { + cleanup(); + } + }); + }); + + describe("POST /api/memories/expand (memory_expand)", () => { + it("returns anchors, timeline, missing_ids, errors, metadata", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + const { ids } = seedMemories(store); + const res = await app.request("/api/memories/expand", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify({ + ids: [ids[0], ids[1], 999999], + depth_before: 1, + depth_after: 1, + include_observations: true, + }), + }); + expect(res.status).toBe(200); + const body = (await res.json()) as { + anchors: Array<{ id: number }>; + timeline: Array<{ id: number }>; + observations: Array<{ id: number }>; + missing_ids: number[]; + errors: Array<{ code: string }>; + metadata: { + requested_ids_count: number; + returned_anchor_count: number; + include_observations: boolean; + }; + }; + expect(body.anchors.map((a) => a.id).sort()).toEqual([ids[0], ids[1]].sort()); + expect(body.timeline.length).toBeGreaterThanOrEqual(2); + expect(body.observations.length).toBeGreaterThan(0); + expect(body.missing_ids).toContain(999999); + expect(body.errors.some((e) => e.code === "NOT_FOUND")).toBe(true); + expect(body.metadata.requested_ids_count).toBe(3); + expect(body.metadata.returned_anchor_count).toBe(2); + expect(body.metadata.include_observations).toBe(true); + } finally { + cleanup(); + } + }); + + it("rejects a non-string include_scope_ids filter with 400", async () => { + const { app, cleanup } = createTestApp(); + try { + const res = await app.request("/api/memories/expand", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify({ + ids: [1], + include_scope_ids: { x: 1 }, + }), + }); + expect(res.status).toBe(400); + const body = (await res.json()) as { error: string }; + expect(body.error).toMatch(/include_scope_ids/); + } finally { + cleanup(); + } + }); + }); + + describe("GET /api/memories/schema (memory_schema)", () => { + it("returns kinds, kind_descriptions, fields, and filters", async () => { + const { app, cleanup } = createTestApp(); + try { + const res = await app.request("/api/memories/schema"); + expect(res.status).toBe(200); + const body = (await res.json()) as { + kinds: string[]; + kind_descriptions: Record; + fields: Record; + filters: string[]; + }; + expect(body.kinds).toEqual( + expect.arrayContaining([ + "discovery", + "change", + "feature", + "bugfix", + "refactor", + "decision", + "exploration", + ]), + ); + expect(body.kind_descriptions.decision).toMatch(/design/i); + expect(body.fields.title).toBe("short text"); + expect(body.fields.body).toBe("long text"); + expect(body.filters).toEqual(expect.arrayContaining(["kind", "project", "scope_id"])); + // Sorted like MCP Object.keys(...).toSorted() + expect(body.filters).toEqual([...body.filters].toSorted()); + } finally { + cleanup(); + } + }); + }); + + describe("GET /api/memories/search_index (memory_search_index)", () => { + it("returns compact index entries without body text", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + seedMemories(store); + const res = await app.request("/api/memories/search_index?query=Database&limit=5"); + expect(res.status).toBe(200); + const body = (await res.json()) as { + items: Array>; + }; + expect(body.items.length).toBeGreaterThan(0); + const first = body.items[0]; + expect(first).toEqual( + expect.objectContaining({ + id: expect.any(Number), + kind: expect.any(String), + title: expect.any(String), + score: expect.any(Number), + created_at: expect.any(String), + session_id: expect.any(Number), + metadata: expect.any(Object), + }), + ); + // Compact index: no body / body_text field (MCP parity) + expect(first).not.toHaveProperty("body"); + expect(first).not.toHaveProperty("body_text"); + } finally { + cleanup(); + } + }); + + it("requires query", async () => { + const { app, cleanup } = createTestApp(); + try { + const res = await app.request("/api/memories/search_index"); + expect(res.status).toBe(400); + expect(await res.json()).toEqual({ error: "query required" }); + } finally { + cleanup(); + } + }); + + it("honors JSON filters.include_visibility so private rows can be selected", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + const sessionId = insertTestSession(store.db); + store.db + .prepare("UPDATE sessions SET project = ? WHERE id = ?") + .run("test-project", sessionId); + // Distinct titles so FTS/search ranking is unambiguous. + const sharedId = store.remember( + sessionId, + "bugfix", + "IndexAlpha shared row", + "shared body IndexAlpha", + 0.9, + undefined, + { visibility: "shared" }, + ); + const privateId = store.remember( + sessionId, + "bugfix", + "IndexAlpha private row", + "private body IndexAlpha", + 0.9, + undefined, + { visibility: "private" }, + ); + + // Without filters both (or at least shared) should be findable. + const unfiltered = await app.request( + "/api/memories/search_index?query=IndexAlpha&limit=10", + ); + expect(unfiltered.status).toBe(200); + const unfilteredBody = (await unfiltered.json()) as { + items: Array<{ id: number }>; + }; + const unfilteredIds = unfilteredBody.items.map((i) => i.id); + expect(unfilteredIds).toContain(sharedId); + + const filters = encodeURIComponent(JSON.stringify({ include_visibility: ["private"] })); + const res = await app.request( + `/api/memories/search_index?query=IndexAlpha&limit=10&filters=${filters}`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { items: Array<{ id: number; title: string }> }; + const ids = body.items.map((item) => item.id); + expect(ids).toContain(privateId); + expect(ids).not.toContain(sharedId); + } finally { + cleanup(); + } + }); + + it("honors kind inside JSON filters (not only top-level kind)", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + seedMemories(store); + const filters = encodeURIComponent(JSON.stringify({ kind: "feature" })); + const res = await app.request( + `/api/memories/search_index?query=Auth&limit=10&filters=${filters}`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { items: Array<{ kind: string; title: string }> }; + expect(body.items.length).toBeGreaterThan(0); + for (const item of body.items) { + expect(item.kind).toBe("feature"); + } + } finally { + cleanup(); + } + }); + }); + + describe("POST /api/memories/explain (memory_explain)", () => { + it("returns scored explanation payload for a query", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + seedMemories(store); + const res = await app.request("/api/memories/explain", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify({ query: "database", limit: 5 }), + }); + expect(res.status).toBe(200); + const body = (await res.json()) as { + items: Array<{ id: number }>; + errors: unknown[]; + }; + expect(Array.isArray(body.items)).toBe(true); + expect(Array.isArray(body.errors)).toBe(true); + expect(body.items.length).toBeGreaterThan(0); + } finally { + cleanup(); + } + }); + + it("explains specific ids", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + const { ids } = seedMemories(store); + const res = await app.request("/api/memories/explain", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify({ ids: [ids[0], ids[1]], limit: 10 }), + }); + expect(res.status).toBe(200); + const body = (await res.json()) as { items: Array<{ id: number }> }; + const returned = body.items.map((i) => i.id); + expect(returned).toEqual(expect.arrayContaining([ids[0], ids[1]])); + } finally { + cleanup(); + } + }); + }); + + describe("POST /api/memories/distill_candidates (memory_distill_candidates)", () => { + it("returns a distill report shape (judge off for determinism)", async () => { + const { app, ensureStore, cleanup } = createTestApp(); + try { + const store = ensureStore(); + // Seed recurring-ish content so mining has something to cluster. + const sessionId = insertTestSession(store.db); + for (let i = 0; i < 4; i++) { + store.remember( + sessionId, + "discovery", + `Prefer explicit source attribution ${i}`, + "Always pass source pi explicitly; never rely on opencode defaults.", + 0.8, + ); + } + const res = await app.request("/api/memories/distill_candidates", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify({ + limit: 5, + min_recurrence: 2, + judge: false, + all_projects: true, + }), + }); + expect(res.status).toBe(200); + const body = (await res.json()) as { + candidates: unknown[]; + metadata: Record; + }; + expect(Array.isArray(body.candidates)).toBe(true); + expect(body.metadata).toEqual(expect.any(Object)); + } finally { + cleanup(); + } + }); + + it("rejects project combined with all_projects", async () => { + const { app, cleanup } = createTestApp(); + try { + const res = await app.request("/api/memories/distill_candidates", { + method: "POST", + headers: jsonHeaders(), + body: JSON.stringify({ + all_projects: true, + project: "codemem", + judge: false, + }), + }); + expect(res.status).toBe(400); + const body = (await res.json()) as { error: string }; + expect(body.error).toMatch(/project cannot be combined with all_projects/); + } finally { + cleanup(); + } + }); + }); +}); diff --git a/packages/viewer-server/src/routes/memory-tools.ts b/packages/viewer-server/src/routes/memory-tools.ts new file mode 100644 index 000000000..1f0f75c95 --- /dev/null +++ b/packages/viewer-server/src/routes/memory-tools.ts @@ -0,0 +1,773 @@ +/** + * Memory tool-support routes — HTTP twins of MCP tools that the viewer + * previously lacked (remember, timeline, expand, schema, search_index, + * explain, distill_candidates). Used by thin clients (pi extension, CLI) + * that prefer HTTP over opening the store in-process. + * + * Behavioral contracts mirror packages/mcp-server/src/tools/* against the + * same @codemem/core MemoryStore APIs. No dependency on @codemem/mcp. + */ + +import { existsSync, readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import type { + DistillContextDocument, + MemoryFilters, + MemoryItemResponse, + MemoryResult, + MemoryStore, +} from "@codemem/core"; +import { + buildDistillReport, + dedupeOrderedIds, + judgeDistillReport, + MEMORY_KIND_DESCRIPTIONS as MEMORY_KINDS, + ObserverClient, + parseStrictInteger, + projectMatchesFilter, + REMEMBER_MEMORY_KINDS, + resolveProject, + resolveProjectRoot, + storeVectors, + toJson, +} from "@codemem/core"; +import { Hono } from "hono"; +import { parseJsonObjectBody, queryInt } from "../helpers.js"; + +type StoreFactory = () => MemoryStore; + +const ALLOWED_REMEMBER_KINDS = new Set(REMEMBER_MEMORY_KINDS); + +const MEMORY_TOOLS_MAX_BODY_BYTES = 1_048_576; +/** Filter names exposed by memory_schema (sorted, matches MCP filterSchema keys). */ +const SCHEMA_FILTER_NAMES = [ + "exclude_actor_ids", + "exclude_scope_ids", + "exclude_trust_states", + "exclude_visibility", + "exclude_workspace_ids", + "exclude_workspace_kinds", + "include_actor_ids", + "include_scope_ids", + "include_trust_states", + "include_visibility", + "include_workspace_ids", + "include_workspace_kinds", + "kind", + "ownership_scope", + "personal_first", + "project", + "scope_id", + "trust_bias", + "visibility", + "widen_shared_min_personal_results", + "widen_shared_min_personal_score", + "widen_shared_when_weak", +].toSorted(); + +const SCHEMA_FIELDS = { + title: "short text", + body: "long text", + subtitle: "short text", + facts: "list", + narrative: "long text", + concepts: "list", + files_read: "list", + files_modified: "list", + prompt_number: "int", +}; + +function cleanProject(value: string | null | undefined): string | null { + const trimmed = value?.trim(); + return trimmed ? trimmed : null; +} + +function resolveWriteProject(input: { + project?: string | null; + envProject?: string | null; +}): string | null { + return cleanProject(input.project) ?? cleanProject(input.envProject) ?? null; +} + +/** + * Build MemoryFilters from a raw args object (query or body). + * + * Project scoping matches existing viewer routes (pack/memory/forget): only an + * explicit `project` arg applies. Unlike MCP tools, the viewer does not inject + * cwd/CODEMEM_PROJECT as an implicit default — the server process cwd is not a + * reliable client project signal. Callers (pi extension, CLI) pass project when + * they want scope. + */ +type FilterParse = { ok: true; filters: MemoryFilters | undefined } | { ok: false; error: string }; + +function isFilterScalar(value: unknown): boolean { + return typeof value === "string" || typeof value === "number" || typeof value === "boolean"; +} + +function isValidFilterValue(value: unknown): boolean { + if (isFilterScalar(value)) return true; + if (Array.isArray(value)) return value.every(isFilterScalar); + return false; +} + +function buildFilters( + raw: Record, + defaultProject: string | null = null, +): FilterParse { + const filters: MemoryFilters = {}; + let hasAny = false; + + if (raw.project != null && typeof raw.project !== "string") { + return { ok: false, error: "project must be a string" }; + } + const explicitProject = typeof raw.project === "string" ? cleanProject(raw.project) : undefined; + // Only fall back to defaultProject when the caller omitted `project` entirely + // (expand uses defaultProject=null for blank-string clear; see expand route). + const resolvedProject = + explicitProject !== undefined + ? explicitProject || undefined + : cleanProject(defaultProject) || undefined; + if (resolvedProject) { + filters.project = resolvedProject; + hasAny = true; + } + + for (const key of [ + "kind", + "visibility", + "scope_id", + "include_scope_ids", + "exclude_scope_ids", + "include_visibility", + "exclude_visibility", + "include_workspace_ids", + "exclude_workspace_ids", + "include_workspace_kinds", + "exclude_workspace_kinds", + "include_actor_ids", + "exclude_actor_ids", + "include_trust_states", + "exclude_trust_states", + "ownership_scope", + "personal_first", + "trust_bias", + "widen_shared_when_weak", + "widen_shared_min_personal_results", + "widen_shared_min_personal_score", + ] as const) { + const val = raw[key]; + if (val === undefined || val === null) continue; + if (key === "kind" && typeof val !== "string") { + return { ok: false, error: "kind must be a string" }; + } + if (!isValidFilterValue(val)) { + return { ok: false, error: `${key} has an invalid type` }; + } + (filters as Record)[key] = val; + hasAny = true; + } + + return { ok: true, filters: hasAny ? filters : undefined }; +} + +/** + * Parse MemoryFilters from a GET query string. + * + * Full filter surface (arrays / booleans / numbers matching MCP filterSchema) + * is accepted via a single JSON-encoded `filters` query param so GET stays + * ergonomic without multi-value keys. Top-level `project` and `kind` remain + * as convenience aliases for existing callers and override the same keys in + * the JSON object when both are present. + * + * Example: + * /api/memories/search_index?query=foo&filters={"include_visibility":["private"]} + */ +function parseGetFilters(queryGetter: (name: string) => string | undefined): FilterParse { + const filterRaw: Record = {}; + + const filtersParam = queryGetter("filters"); + if (filtersParam != null && filtersParam.trim() !== "") { + let parsed: unknown; + try { + parsed = JSON.parse(filtersParam); + } catch { + return { ok: false, error: "filters must be valid JSON" }; + } + if (parsed == null || typeof parsed !== "object" || Array.isArray(parsed)) { + return { ok: false, error: "filters must be a JSON object" }; + } + Object.assign(filterRaw, parsed as Record); + } + + // Top-level convenience aliases (backward compatible with project+kind only). + const project = queryGetter("project"); + const kind = queryGetter("kind"); + if (project != null) filterRaw.project = project; + if (kind != null) filterRaw.kind = kind; + + return buildFilters(filterRaw); +} + +function getMemoryForAccess( + store: MemoryStore, + memoryId: number, + filters?: MemoryFilters, +): MemoryItemResponse | null { + const rows = store.timeline(null, memoryId, 0, 0, filters ?? null); + return rows.find((row) => row.id === memoryId) ?? null; +} + +function getManyForAccess( + store: MemoryStore, + ids: number[], + filters?: MemoryFilters, +): MemoryItemResponse[] { + if (ids.length === 0) return []; + const results: MemoryItemResponse[] = []; + for (const id of ids) { + const item = getMemoryForAccess(store, id, filters); + if (item) results.push(item); + } + return results; +} + +function clampInt(value: number, min: number, max: number): number { + return Math.min(max, Math.max(min, value)); +} + +function parseOptionalInt(value: unknown): number | null { + if (value == null) return null; + if (typeof value === "number" && Number.isInteger(value)) return value; + if (typeof value === "string") return parseStrictInteger(value); + return null; +} + +function parseOptionalBoolean(value: unknown): boolean | "invalid" { + if (value == null) return false; + if (typeof value === "boolean") return value; + if (typeof value === "string") { + const normalized = value.trim().toLowerCase(); + if (normalized === "true" || normalized === "1" || normalized === "yes") return true; + if (normalized === "false" || normalized === "0" || normalized === "no") return false; + } + return "invalid"; +} +function parseJsonBody( + body: unknown, +): { ok: true; value: Record } | { ok: false; error: string } { + if (body == null || typeof body !== "object" || Array.isArray(body)) { + return { ok: false, error: "payload must be an object" }; + } + return { ok: true, value: body as Record }; +} + +function rememberMemory( + store: MemoryStore, + input: { + kind: string; + title: string; + body: string; + confidence: number; + project?: string | null; + }, +): { memId: number; title: string; body: string } { + return store.db.transaction(() => { + const now = new Date().toISOString(); + const user = process.env.USER ?? "unknown"; + const cwd = process.cwd(); + const project = resolveWriteProject({ + project: input.project, + envProject: process.env.CODEMEM_PROJECT, + }); + + const sessionInfo = store.db + .prepare( + `INSERT INTO sessions(started_at, ended_at, cwd, project, user, tool_version, metadata_json) + VALUES (?, ?, ?, ?, ?, ?, ?)`, + ) + .run(now, now, cwd, project, user, "viewer-api", toJson({ viewer: true })); + const sessionId = Number(sessionInfo.lastInsertRowid); + + const memId = store.remember(sessionId, input.kind, input.title, input.body, input.confidence); + if (!getMemoryForAccess(store, memId)) { + throw new Error("unauthorized_scope"); + } + + store.db + .prepare("UPDATE sessions SET ended_at = ?, metadata_json = ? WHERE id = ?") + .run(new Date().toISOString(), toJson({ viewer: true }), sessionId); + + return { memId, title: input.title, body: input.body }; + })(); +} + +function readContextFile( + path: string, + displayPath: string, + scope: DistillContextDocument["scope"], +): DistillContextDocument | null { + if (!existsSync(path)) return null; + const text = readFileSync(path, "utf8"); + return text.trim() ? { path: displayPath, text, scope } : null; +} + +function loadDefaultContextDocuments( + includeProjectContext: boolean, + cwd = process.cwd(), +): DistillContextDocument[] { + const projectRoot = resolveProjectRoot(cwd) ?? cwd; + const documents = [ + includeProjectContext + ? readContextFile(join(projectRoot, "AGENTS.md"), "AGENTS.md", "project") + : null, + readContextFile( + join(homedir(), ".config", "opencode", "AGENTS.md"), + "~/.config/opencode/AGENTS.md", + "user", + ), + ]; + return documents.filter((document): document is DistillContextDocument => document != null); +} + +function shouldIncludeProjectContext( + args: { all_projects?: boolean; project?: unknown }, + defaultProject: string | null, +): boolean { + if (args.all_projects) return false; + const currentProject = resolveProject(process.cwd()); + if (!currentProject) return false; + const explicitProject = typeof args.project === "string" ? args.project.trim() : ""; + const targetProject = explicitProject + ? resolveProject(process.cwd(), explicitProject) + : defaultProject; + if (!targetProject) return false; + return projectMatchesFilter(targetProject, currentProject); +} + +function buildDistillFilters( + args: { all_projects?: boolean } & Record, + defaultProject: string | null, +): FilterParse { + if (args.all_projects && typeof args.project === "string" && args.project.trim()) { + return { ok: false, error: "project cannot be combined with all_projects" }; + } + return buildFilters(args, args.all_projects ? null : defaultProject); +} + +function mapSearchIndexItem(m: MemoryResult) { + return { + id: m.id, + kind: m.kind, + title: m.title, + score: m.score, + created_at: m.created_at, + session_id: m.session_id, + metadata: m.metadata, + }; +} + +function expandMemories( + store: MemoryStore, + args: { + ids: unknown[]; + depth_before: number; + depth_after: number; + include_observations: boolean; + filters?: MemoryFilters; + }, +) { + const resolvedProject = args.filters?.project ?? null; + const { ordered: orderedIds, invalid: invalidIds } = dedupeOrderedIds(args.ids); + const errors: Array> = []; + + if (invalidIds.length > 0) { + errors.push({ + code: "INVALID_ARGUMENT", + field: "ids", + message: "some ids are not valid integers", + ids: invalidIds, + }); + } + + const missingNotFound: number[] = []; + const missingProjectMismatch: number[] = []; + const missingFilterMismatch: number[] = []; + const anchors: MemoryItemResponse[] = []; + const timelineItems: MemoryItemResponse[] = []; + const timelineSeen = new Set(); + const sessionProjects = new Map(); + + for (const memoryId of orderedIds) { + const item = store.get(memoryId); + if (!item?.active) { + missingNotFound.push(memoryId); + continue; + } + + const sessionId = item.session_id; + if (resolvedProject && sessionId > 0) { + if (!sessionProjects.has(sessionId)) { + const row = store.db + .prepare("SELECT project FROM sessions WHERE id = ? LIMIT 1") + .get(sessionId) as { project: string | null } | undefined; + sessionProjects.set(sessionId, typeof row?.project === "string" ? row.project : null); + } + if (!projectMatchesFilter(resolvedProject, sessionProjects.get(sessionId) ?? null)) { + missingProjectMismatch.push(memoryId); + continue; + } + } else if (resolvedProject && sessionId <= 0) { + missingProjectMismatch.push(memoryId); + continue; + } + + const expanded = store.timeline( + null, + memoryId, + args.depth_before, + args.depth_after, + args.filters, + ); + const anchor = expanded.find((expandedItem) => expandedItem.id === memoryId); + if (!anchor) { + missingFilterMismatch.push(memoryId); + continue; + } + + anchors.push(anchor); + for (const expandedItem of expanded) { + const expandedId = expandedItem.id; + if (expandedId <= 0 || timelineSeen.has(expandedId)) continue; + timelineSeen.add(expandedId); + timelineItems.push(expandedItem); + } + } + + if (missingNotFound.length > 0) { + errors.push({ + code: "NOT_FOUND", + field: "ids", + message: "some requested ids were not found", + ids: missingNotFound, + }); + } + if (missingProjectMismatch.length > 0) { + errors.push({ + code: "PROJECT_MISMATCH", + field: "project", + message: "some requested ids are outside the requested project scope", + ids: missingProjectMismatch, + }); + } + if (missingFilterMismatch.length > 0) { + errors.push({ + code: "FILTER_MISMATCH", + field: "filters", + message: "some requested ids are outside the requested filters", + ids: missingFilterMismatch, + }); + } + + let observations: MemoryItemResponse[] = []; + if (args.include_observations) { + const observationSeen = new Set(); + const observationIds: number[] = []; + for (const item of [...anchors, ...timelineItems]) { + if (item.id > 0 && !observationSeen.has(item.id)) { + observationSeen.add(item.id); + observationIds.push(item.id); + } + } + observations = getManyForAccess(store, observationIds, args.filters); + } + + return { + anchors, + timeline: timelineItems, + observations, + missing_ids: orderedIds.filter( + (memoryId: number) => + missingNotFound.includes(memoryId) || + missingProjectMismatch.includes(memoryId) || + missingFilterMismatch.includes(memoryId), + ), + errors, + metadata: { + project: resolvedProject, + requested_ids_count: orderedIds.length, + returned_anchor_count: anchors.length, + timeline_count: timelineItems.length, + include_observations: args.include_observations, + }, + }; +} + +export function memoryToolRoutes(getStore: StoreFactory) { + const app = new Hono(); + + // POST /api/memories/remember — twin of memory_remember + app.post("/api/memories/remember", async (c) => { + const store = getStore(); + const body = await parseJsonObjectBody(c, MEMORY_TOOLS_MAX_BODY_BYTES); + if (body instanceof Response) return body; + const parsed = parseJsonBody(body); + if (!parsed.ok) return c.json({ error: parsed.error }, 400); + const args = parsed.value; + + const kind = typeof args.kind === "string" ? args.kind.trim().toLowerCase() : ""; + if (!kind || !ALLOWED_REMEMBER_KINDS.has(kind)) { + return c.json( + { + error: `kind must be one of: ${[...ALLOWED_REMEMBER_KINDS].join(", ")}`, + }, + 400, + ); + } + const title = typeof args.title === "string" ? args.title : ""; + const bodyText = typeof args.body === "string" ? args.body : ""; + if (!title.trim()) return c.json({ error: "title is required" }, 400); + if (!bodyText.trim()) return c.json({ error: "body is required" }, 400); + + let confidence = 0.5; + if (args.confidence != null) { + if (typeof args.confidence !== "number" || Number.isNaN(args.confidence)) { + return c.json({ error: "confidence must be a number" }, 400); + } + confidence = Math.min(1, Math.max(0, args.confidence)); + } + const project = typeof args.project === "string" ? args.project : undefined; + + try { + const result = rememberMemory(store, { + kind, + title, + body: bodyText, + confidence, + project, + }); + try { + await storeVectors(store.db, result.memId, result.title, result.body); + } catch { + // Memory writes should succeed even if embeddings are unavailable. + } + return c.json({ id: result.memId }); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + if (msg.includes("Invalid memory kind")) return c.json({ error: msg }, 400); + if (msg === "unauthorized_scope") return c.json({ error: msg }, 403); + return c.json({ error: msg }, 400); + } + }); + + // GET /api/memories/timeline — twin of memory_timeline + // Full filter surface via JSON `filters` query param (MCP filterSchema parity). + app.get("/api/memories/timeline", (c) => { + const store = getStore(); + const query = c.req.query("query") || undefined; + const memoryIdRaw = c.req.query("memory_id"); + const memoryId = + memoryIdRaw != null && memoryIdRaw !== "" ? parseStrictInteger(memoryIdRaw) : null; + if (memoryIdRaw != null && memoryIdRaw !== "" && memoryId == null) { + return c.json({ error: "memory_id must be int" }, 400); + } + const depthBefore = clampInt(queryInt(c.req.query("depth_before"), 3), 0, 100); + const depthAfter = clampInt(queryInt(c.req.query("depth_after"), 3), 0, 100); + + const parsedFilters = parseGetFilters((name) => c.req.query(name)); + if (!parsedFilters.ok) return c.json({ error: parsedFilters.error }, 400); + + const items = store.timeline( + query ?? null, + memoryId, + depthBefore, + depthAfter, + parsedFilters.filters, + ); + return c.json({ items }); + }); + + // POST /api/memories/expand — twin of memory_expand + app.post("/api/memories/expand", async (c) => { + const store = getStore(); + const body = await parseJsonObjectBody(c, MEMORY_TOOLS_MAX_BODY_BYTES); + if (body instanceof Response) return body; + const parsed = parseJsonBody(body); + if (!parsed.ok) return c.json({ error: parsed.error }, 400); + const args = parsed.value; + + if (!Array.isArray(args.ids)) { + return c.json({ error: "ids must be an array" }, 400); + } + if (args.ids.length > 200) { + return c.json({ error: "ids must contain at most 200 entries" }, 400); + } + + const depthBeforeRaw = parseOptionalInt(args.depth_before); + const depthAfterRaw = parseOptionalInt(args.depth_after); + const depthBefore = clampInt(depthBeforeRaw ?? 3, 0, 100); + const depthAfter = clampInt(depthAfterRaw ?? 3, 0, 100); + const includeObservations = parseOptionalBoolean(args.include_observations); + if (includeObservations === "invalid") { + return c.json({ error: "include_observations must be a boolean" }, 400); + } + // Explicit blank project clears scoping (MCP expand parity). Viewer routes + // do not inject a cwd default project; only an explicit non-blank project scopes. + const parsedFilters = buildFilters(args, null); + if (!parsedFilters.ok) return c.json({ error: parsedFilters.error }, 400); + + const value = expandMemories(store, { + ids: args.ids, + depth_before: depthBefore, + depth_after: depthAfter, + include_observations: includeObservations === true, + filters: parsedFilters.filters, + }); + return c.json(value); + }); + + // GET /api/memories/schema — twin of memory_schema + app.get("/api/memories/schema", (c) => { + return c.json({ + kinds: Object.keys(MEMORY_KINDS), + kind_descriptions: MEMORY_KINDS, + fields: SCHEMA_FIELDS, + filters: SCHEMA_FILTER_NAMES, + }); + }); + + // GET /api/memories/search_index — twin of memory_search_index + // Full filter surface via JSON `filters` query param (MCP filterSchema parity). + app.get("/api/memories/search_index", (c) => { + const store = getStore(); + const query = c.req.query("query") ?? ""; + if (!query.trim()) { + return c.json({ error: "query required" }, 400); + } + const limit = clampInt(queryInt(c.req.query("limit"), 8), 1, 50); + const parsedFilters = parseGetFilters((name) => c.req.query(name)); + if (!parsedFilters.ok) return c.json({ error: parsedFilters.error }, 400); + const items = store.search(query, limit, parsedFilters.filters).map(mapSearchIndexItem); + return c.json({ items }); + }); + + // POST /api/memories/explain — twin of memory_explain + app.post("/api/memories/explain", async (c) => { + const store = getStore(); + const body = await parseJsonObjectBody(c, MEMORY_TOOLS_MAX_BODY_BYTES); + if (body instanceof Response) return body; + const parsed = parseJsonBody(body); + if (!parsed.ok) return c.json({ error: parsed.error }, 400); + const args = parsed.value; + const query = typeof args.query === "string" ? args.query : null; + let ids: number[] | null = null; + if (args.ids != null) { + if (!Array.isArray(args.ids)) { + return c.json({ error: "ids must be an array" }, 400); + } + if (args.ids.length > 200) { + return c.json({ error: "ids must contain at most 200 entries" }, 400); + } + const { ordered, invalid } = dedupeOrderedIds(args.ids); + if (invalid.length > 0) { + return c.json({ error: "some ids are not valid integers", ids: invalid }, 400); + } + ids = ordered; + } + const limit = clampInt(parseOptionalInt(args.limit) ?? 10, 1, 50); + const includePackContext = parseOptionalBoolean(args.include_pack_context); + if (includePackContext === "invalid") { + return c.json({ error: "include_pack_context must be a boolean" }, 400); + } + const parsedFilters = buildFilters(args); + if (!parsedFilters.ok) return c.json({ error: parsedFilters.error }, 400); + + const result = store.explain(query, ids, limit, parsedFilters.filters, { + includePackContext: includePackContext === true, + }); + return c.json(result); + }); + + // POST /api/memories/distill_candidates — twin of memory_distill_candidates + app.post("/api/memories/distill_candidates", async (c) => { + const store = getStore(); + const body = await parseJsonObjectBody(c, MEMORY_TOOLS_MAX_BODY_BYTES); + if (body instanceof Response) return body; + const parsed = parseJsonBody(body); + if (!parsed.ok) return c.json({ error: parsed.error }, 400); + const args = parsed.value; + const limit = clampInt(parseOptionalInt(args.limit) ?? 10, 1, 50); + const minRecurrence = clampInt(parseOptionalInt(args.min_recurrence) ?? 2, 1, 50); + const allProjects = parseOptionalBoolean(args.all_projects); + if (allProjects === "invalid") { + return c.json({ error: "all_projects must be a boolean" }, 400); + } + const includeDocumented = parseOptionalBoolean(args.include_documented); + if (includeDocumented === "invalid") { + return c.json({ error: "include_documented must be a boolean" }, 400); + } + const maxEvidenceItems = clampInt(parseOptionalInt(args.max_evidence_items) ?? 5, 1, 20); + const judgeParsed = args.judge === undefined ? true : parseOptionalBoolean(args.judge); + if (judgeParsed === "invalid") { + return c.json({ error: "judge must be a boolean" }, 400); + } + const judge = judgeParsed === true; + try { + // Prefer explicit project / CODEMEM_PROJECT for context docs; no cwd default. + const resolvedDefaultProject = + cleanProject(typeof args.project === "string" ? args.project : null) ?? + cleanProject(process.env.CODEMEM_PROJECT); + const filterArgs = { ...args, all_projects: allProjects }; + const parsedFilters = buildDistillFilters(filterArgs, resolvedDefaultProject); + if (!parsedFilters.ok) return c.json({ error: parsedFilters.error }, 400); + const kinds = typeof args.kind === "string" && args.kind.trim() ? [args.kind] : undefined; + const fetchLimit = judge ? Math.min(limit * 3, limit + 20) : limit; + + let result = await buildDistillReport(store, { + candidate: { + includeDocumented, + maxEvidenceItems, + }, + contextDocuments: loadDefaultContextDocuments( + shouldIncludeProjectContext( + { all_projects: allProjects, project: args.project }, + resolvedDefaultProject, + ), + ), + corpus: { filters: parsedFilters.filters ?? null, kinds }, + limit: fetchLimit, + minRecurrence, + }); + + if (judge) { + try { + const client = new ObserverClient(); + result = await judgeDistillReport(result, async (system, user) => { + const response = await client.observe(system, user); + return response.raw; + }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + result = { + ...result, + metadata: { ...result.metadata, judged: false, judge_error: message }, + }; + } + if (result.candidates.length > limit) { + result = { + ...result, + candidates: result.candidates.slice(0, limit), + metadata: { ...result.metadata, candidate_count: limit }, + }; + } + } + + return c.json(result); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + return c.json({ error: msg }, 400); + } + }); + + return app; +} diff --git a/packages/viewer-server/src/routes/raw-events.ts b/packages/viewer-server/src/routes/raw-events.ts index b80016217..924bb3c32 100644 --- a/packages/viewer-server/src/routes/raw-events.ts +++ b/packages/viewer-server/src/routes/raw-events.ts @@ -1,6 +1,6 @@ /** * Raw events routes — GET & POST /api/raw-events, GET /api/raw-events/status, - * POST /api/claude-hooks, POST /api/codex-hooks. + * POST /api/claude-hooks, POST /api/codex-hooks, POST /api/pi-hooks. */ import { homedir } from "node:os"; @@ -9,6 +9,7 @@ import type { HookTranscriptOutcome, MemoryStore, RawEventSweeper } from "@codem import { buildRawEventEnvelopeFromCodexHook, buildRawEventEnvelopeFromHook, + buildRawEventEnvelopeFromPiEvent, ingestRawEvents, RawEventIngestValidationError, schema, @@ -336,5 +337,26 @@ export function rawEventsRoutes(getStore: StoreFactory, sweeper?: RawEventSweepe } }); + // POST /api/pi-hooks — ingest pi extension events (compat alias) + app.post("/api/pi-hooks", async (c) => { + const result = await parseJsonObjectBody(c, MAX_RAW_EVENTS_BODY_BYTES); + if (result instanceof Response) return result; + const payload = result; + + try { + const envelope = buildRawEventEnvelopeFromPiEvent(payload); + if (envelope === null) { + return c.json({ inserted: 0, skipped: 1 }); + } + const ingestResult = await ingestNormalizedEnvelope(getStore(), sweeper, { + ...envelope, + source: "pi", + }); + return c.json({ inserted: ingestResult.inserted, skipped: ingestResult.skipped }); + } catch (err) { + return boundedIngestErrorResponse(c, err); + } + }); + return app; } diff --git a/scripts/docs-viewer.ts b/scripts/docs-viewer.ts new file mode 100644 index 000000000..e5d2e7239 --- /dev/null +++ b/scripts/docs-viewer.ts @@ -0,0 +1,515 @@ +import { spawn } from "node:child_process"; +import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { createRequire } from "node:module"; +import { tmpdir } from "node:os"; +import { basename, dirname, join, resolve } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const CHILD_MARKER = "CODEMEM_DOCS_VIEWER_CHILD"; +const FIXTURE_ROOT_ENV = "CODEMEM_DOCS_VIEWER_ROOT"; +const FIXTURE_REVISION = "current working tree, including generated viewer assets"; +const FIXTURE_ROOT_PATTERN = /^codemem-docs-viewer-[A-Za-z0-9]{6}$/; +const HANDSHAKE_TIMEOUT_MS = 5_000; +const SERVER_CLOSE_TIMEOUT_MS = 2_000; + +interface RuntimePaths { + root: string; + runtime: string; + db: string; + config: string; + keys: string; +} + +interface FixtureMemory { + kind: "bugfix" | "decision" | "discovery"; + title: string; + subtitle: string; + body: string; + tags: string[]; + filesRead: string[]; + filesModified: string[]; + concepts: string[]; + facts: string[]; +} + +interface ViewerServer { + close: (callback: (error?: Error) => void) => void; + closeAllConnections: () => void; + closeIdleConnections: () => void; + once: (event: "error", listener: (error: Error) => void) => void; +} + +interface LaunchMessage { + type: "launch"; + root: string; + tempParent: string; +} + +type ServeViewer = ( + options: { fetch: (request: Request) => unknown; hostname: string; port: number }, + listeningListener: (info: { port: number }) => void, +) => ViewerServer; + +function runtimePaths(root: string): RuntimePaths { + return { + root, + runtime: join(root, "runtime"), + db: join(root, "runtime", "mem.sqlite"), + config: join(root, "runtime", "config", "codemem.json"), + keys: join(root, "runtime", "keys"), + }; +} + +function createRuntimePaths(): RuntimePaths { + const paths = runtimePaths(mkdtempSync(join(tmpdir(), "codemem-docs-viewer-"))); + for (const path of ["home", "tmp", "xdg/config", "xdg/cache", "xdg/data"]) { + mkdirSync(join(paths.root, path), { recursive: true }); + } + for (const path of [paths.runtime, paths.keys, dirname(paths.config)]) { + mkdirSync(path, { recursive: true }); + } + return paths; +} + +function loaderArgs(execArgv: string[]): string[] { + const loaderFlags = new Set(["--experimental-loader", "--import", "--loader", "--require", "-r"]); + const loaderPrefixes = ["--experimental-loader=", "--import=", "--loader=", "--require="]; + const inherited: string[] = []; + for (let index = 0; index < execArgv.length; index += 1) { + const arg = execArgv[index]; + if (arg === "--conditions" && execArgv[index + 1] === "source") { + inherited.push(arg, "source"); + index += 1; + continue; + } + if (arg === "--conditions=source") { + inherited.push(arg); + continue; + } + if (loaderPrefixes.some((prefix) => arg.startsWith(prefix)) && arg.includes("tsx")) { + inherited.push(arg); + continue; + } + if (!loaderFlags.has(arg)) continue; + const value = execArgv[index + 1]; + if (!value) throw new Error(`Missing value for inherited Node argument ${arg}`); + if (value.includes("tsx")) inherited.push(arg, value); + index += 1; + } + const hasSourceCondition = inherited.some( + (arg, index) => + arg === "--conditions=source" || + (arg === "--conditions" && inherited[index + 1] === "source"), + ); + if (!hasSourceCondition) { + inherited.unshift("--conditions=source"); + } + return inherited; +} + +function childEnvironment(paths: RuntimePaths): NodeJS.ProcessEnv { + return { + [CHILD_MARKER]: "1", + [FIXTURE_ROOT_ENV]: paths.root, + HOME: join(paths.root, "home"), + USER: "demo", + LOGNAME: "demo", + TMPDIR: join(paths.root, "tmp"), + XDG_CONFIG_HOME: join(paths.root, "xdg", "config"), + XDG_CACHE_HOME: join(paths.root, "xdg", "cache"), + XDG_DATA_HOME: join(paths.root, "xdg", "data"), + CODEMEM_RUNTIME_ROOT: paths.runtime, + CODEMEM_DB: paths.db, + CODEMEM_CONFIG: paths.config, + CODEMEM_KEYS_DIR: paths.keys, + CODEMEM_DEVICE_ID: "docs-fixture-device", + CODEMEM_ACTOR_ID: "docs-fixture-actor", + CODEMEM_ACTOR_DISPLAY_NAME: "Demo Developer", + CODEMEM_EMBEDDING_DISABLED: "1", + CODEMEM_EMBEDDING_OFFLINE: "1", + CODEMEM_RAW_EVENTS_SWEEPER: "0", + CODEMEM_SYNC_KEY_STORE: "file", + CODEMEM_SYNC_MDNS: "0", + }; +} + +async function launchIsolatedChild(): Promise { + const paths = createRuntimePaths(); + const scriptPath = fileURLToPath(import.meta.url); + const child = spawn(process.execPath, [...loaderArgs(process.execArgv), scriptPath], { + env: childEnvironment(paths), + stdio: ["inherit", "inherit", "inherit", "ipc"], + }); + let launched = false; + child.on("message", (message) => { + if (launched || !message || typeof message !== "object") return; + if (!("type" in message) || message.type !== "ready") return; + launched = true; + child.send({ type: "launch", root: paths.root, tempParent: dirname(paths.root) }); + }); + + for (const signal of ["SIGINT", "SIGTERM"] as const) { + process.once(signal, () => child.kill(signal)); + } + + await new Promise((resolve, reject) => { + child.once("error", reject); + child.once("exit", (code, signal) => { + if (signal) { + process.exitCode = signal === "SIGINT" ? 130 : 143; + } else { + process.exitCode = code ?? 1; + } + resolve(); + }); + }); +} + +async function receiveLaunchMessage(): Promise { + if (!process.connected || typeof process.send !== "function") { + throw new Error("Synthetic viewer child requires its parent IPC channel"); + } + return new Promise((resolveMessage, reject) => { + const timer = setTimeout( + () => reject(new Error("Synthetic viewer parent handshake timed out")), + HANDSHAKE_TIMEOUT_MS, + ); + process.once("message", (message) => { + clearTimeout(timer); + if (!message || typeof message !== "object" || !("type" in message)) { + reject(new Error("Synthetic viewer parent sent an invalid launch message")); + return; + } + const launch = message as Partial; + if (launch.type !== "launch" || typeof launch.root !== "string") { + reject(new Error("Synthetic viewer parent sent an invalid launch message")); + return; + } + if (typeof launch.tempParent !== "string") { + reject(new Error("Synthetic viewer parent omitted its temp directory")); + return; + } + resolveMessage(launch as LaunchMessage); + }); + process.send?.({ type: "ready" }); + }); +} + +function loadRuntimePaths(launch: LaunchMessage): RuntimePaths { + const root = resolve(launch.root); + const tempParent = resolve(launch.tempParent); + if (dirname(root) !== tempParent || !FIXTURE_ROOT_PATTERN.test(basename(root))) { + throw new Error("Synthetic viewer parent supplied an invalid fixture root"); + } + if (process.env[FIXTURE_ROOT_ENV] !== root) { + throw new Error("Synthetic viewer environment does not match its parent handshake"); + } + return runtimePaths(root); +} + +function validateRuntimeEnvironment(paths: RuntimePaths): void { + const expected = { + HOME: join(paths.root, "home"), + TMPDIR: join(paths.root, "tmp"), + XDG_CONFIG_HOME: join(paths.root, "xdg", "config"), + XDG_CACHE_HOME: join(paths.root, "xdg", "cache"), + XDG_DATA_HOME: join(paths.root, "xdg", "data"), + CODEMEM_RUNTIME_ROOT: paths.runtime, + CODEMEM_DB: paths.db, + CODEMEM_CONFIG: paths.config, + CODEMEM_KEYS_DIR: paths.keys, + }; + for (const [name, value] of Object.entries(expected)) { + if (process.env[name] !== value) { + throw new Error(`Synthetic viewer rejected mismatched ${name}`); + } + } +} + +function prepareRuntime(paths: RuntimePaths): void { + if (existsSync(paths.db)) { + throw new Error(`Refusing to reuse synthetic viewer database: ${paths.db}`); + } + const config = { + actor_id: "docs-fixture-actor", + actor_display_name: "Demo Developer", + observer_auth_source: "none", + observer_tier_routing_enabled: false, + sync_enabled: false, + sync_mdns: false, + }; + writeFileSync(paths.config, `${JSON.stringify(config, null, 2)}\n`, { + flag: "wx", + mode: 0o600, + }); +} + +function atlasMemories(): FixtureMemory[] { + return [ + { + kind: "decision", + title: "Keep note parsing deterministic", + subtitle: "Markdown structure is parsed before search indexing", + body: "Atlas Notes parses headings, task markers, and wiki links in one deterministic pass. The index receives normalized text plus stable section anchors, so a re-index does not reorder otherwise unchanged notes.", + tags: ["architecture", "markdown", "search"], + filesRead: ["src/notes/parse-note.ts", "src/search/index-note.ts"], + filesModified: ["docs/architecture/note-pipeline.md"], + concepts: ["deterministic parsing", "stable anchors", "search indexing"], + facts: [ + "Pipeline: parse → normalize → index", + "Anchor format: heading slug plus source offset", + ], + }, + { + kind: "discovery", + title: "Backlinks need normalized note identifiers", + subtitle: "Display titles are not stable relationship keys", + body: "Renaming a note changed its display title but not its file identity. Backlink edges remain correct when they use the normalized repository-relative note identifier and resolve the latest title only while rendering.", + tags: ["backlinks", "data-model"], + filesRead: ["src/graph/backlinks.ts", "src/notes/note-id.ts"], + filesModified: [], + concepts: ["backlinks", "note identity", "rename safety"], + facts: [ + "Stable key: repository-relative note identifier", + "Presentation: resolve title at render time", + ], + }, + { + kind: "bugfix", + title: "Preserve task state during note refresh", + subtitle: "Refresh no longer resets optimistic checkbox updates", + body: "A background refresh could replace a locally toggled task with an older server snapshot. The client now retains pending task revisions until the matching write response arrives, then reconciles against the returned revision.", + tags: ["tasks", "concurrency", "ui"], + filesRead: ["src/tasks/task-state.ts", "src/api/save-task.ts"], + filesModified: ["src/tasks/task-state.ts", "src/tasks/task-state.test.ts"], + concepts: ["optimistic update", "revision", "background refresh"], + facts: [ + "Cause: stale refresh replaced a pending local revision", + "Fix: reconcile after the matching write response", + ], + }, + ]; +} + +function gardenMemories(): FixtureMemory[] { + return [ + { + kind: "decision", + title: "Use cursor pagination for plant observations", + subtitle: "Observation time and identifier form the stable cursor", + body: "Garden API pages observations by descending observed time with the observation identifier as a deterministic tie-breaker. Clients can request the next page without skipped or duplicated rows when new observations arrive.", + tags: ["api", "pagination", "observations"], + filesRead: ["src/observations/list.ts", "src/http/cursors.ts"], + filesModified: ["docs/api/observations.md"], + concepts: ["cursor pagination", "stable ordering", "observation feed"], + facts: ["Sort: observed_at DESC, observation_id DESC", "Page size: 50 by default"], + }, + { + kind: "discovery", + title: "Sensor timestamps can arrive out of order", + subtitle: "Ingestion time cannot stand in for observation time", + body: "Offline greenhouse sensors upload buffered readings after reconnecting. Reports must group by the sensor-provided observation time while retaining ingestion time for operational diagnostics and replay analysis.", + tags: ["sensors", "timestamps", "ingestion"], + filesRead: ["src/sensors/ingest-reading.ts", "src/reports/daily-summary.ts"], + filesModified: [], + concepts: ["event time", "ingestion time", "offline sensors"], + facts: ["Reporting clock: sensor observation time", "Diagnostic clock: API ingestion time"], + }, + { + kind: "bugfix", + title: "Reject duplicate watering commands", + subtitle: "Retries now reuse an idempotency record", + body: "A timed-out client retry could schedule the same watering command twice. The endpoint now stores the request key with the first command result and returns that result for later retries within the retention window.", + tags: ["idempotency", "watering", "retries"], + filesRead: ["src/watering/create-command.ts", "src/storage/idempotency.ts"], + filesModified: ["src/watering/create-command.ts", "src/watering/create-command.test.ts"], + concepts: ["idempotency", "retry safety", "watering command"], + facts: ["Cause: timeout hid the first successful write", "Retention: 24 hours"], + }, + ]; +} + +function seedProject( + store: InstanceType, + project: string, + cwd: string, + memories: FixtureMemory[], +): number { + const sessionId = store.startSession({ + cwd, + project, + user: "demo", + toolVersion: "docs-fixture", + metadata: { fixture: "docs-viewer", synthetic: true }, + }); + for (const [index, memory] of memories.entries()) { + store.remember( + sessionId, + memory.kind, + memory.title, + memory.body, + 0.9 - index * 0.05, + memory.tags, + { + visibility: "private", + workspace_kind: "personal", + workspace_id: "personal:docs-fixture-actor", + origin_source: "docs-fixture", + subtitle: memory.subtitle, + narrative: memory.body, + facts: memory.facts, + concepts: memory.concepts, + files_read: memory.filesRead, + files_modified: memory.filesModified, + prompt_number: index + 1, + }, + ); + } + store.endSession(sessionId, { fixture: "docs-viewer", memory_count: memories.length }); + return memories.length; +} + +function denyExternalFetch(): void { + const systemFetch = globalThis.fetch; + globalThis.fetch = async (input, init) => { + const url = new URL(input instanceof Request ? input.url : String(input)); + if (!["127.0.0.1", "::1", "localhost"].includes(url.hostname)) { + throw new Error(`Synthetic docs viewer blocked outbound fetch to ${url.origin}`); + } + return systemFetch(input, init); + }; +} + +async function loadServeViewer(): Promise { + const requireFromViewer = createRequire( + new URL("../packages/viewer-server/package.json", import.meta.url), + ); + const moduleUrl = pathToFileURL(requireFromViewer.resolve("@hono/node-server")); + const loaded: unknown = await import(moduleUrl.href); + if (!loaded || typeof loaded !== "object" || !("serve" in loaded)) { + throw new Error("@hono/node-server did not export serve"); + } + const serve = (loaded as { serve?: unknown }).serve; + if (typeof serve !== "function") throw new Error("@hono/node-server serve export is invalid"); + return serve as ServeViewer; +} + +async function closeServer(server: ViewerServer): Promise { + await new Promise((resolve, reject) => { + let settled = false; + const finish = (error?: Error) => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (error) reject(error); + else resolve(); + }; + const timer = setTimeout(() => { + server.closeAllConnections(); + finish(); + }, SERVER_CLOSE_TIMEOUT_MS); + server.close((error) => { + finish(error); + }); + server.closeIdleConnections(); + }); +} + +function offlineUpdateStatus( + currentVersion: string, +): import("../packages/core/src/index.ts").UpdateStatus { + return { + current_version: currentVersion, + channel: null, + latest_version: null, + update_available: false, + first_seen_at: null, + checked_at: null, + stale: false, + install_kind: "repo-dev", + auto_update_eligible: false, + recommended_action: "Offline synthetic fixture; update checks are disabled.", + error: null, + }; +} + +function printFixture(paths: RuntimePaths, port: number, memoryCount: number): void { + console.log( + JSON.stringify( + { + url: `http://127.0.0.1:${port}`, + runtime: paths.root, + database: paths.db, + memories: memoryCount, + fixture_revision: FIXTURE_REVISION, + synthetic_only: true, + }, + null, + 2, + ), + ); +} + +async function waitForShutdown(): Promise { + await new Promise((resolve) => { + for (const signal of ["SIGINT", "SIGTERM"] as const) process.once(signal, resolve); + }); +} + +async function runFixture(): Promise { + const launch = await receiveLaunchMessage(); + const paths = loadRuntimePaths(launch); + validateRuntimeEnvironment(paths); + prepareRuntime(paths); + denyExternalFetch(); + + const [{ initDatabase, MemoryStore, VERSION }, { createApp }, serve] = await Promise.all([ + import("../packages/core/src/index.ts"), + import("../packages/viewer-server/src/index.ts"), + loadServeViewer(), + ]); + initDatabase(paths.db); + const store = new MemoryStore(paths.db); + let server: ViewerServer | null = null; + try { + const memoryCount = + seedProject(store, "atlas-notes", "/demo/atlas-notes", atlasMemories()) + + seedProject(store, "garden-api", "/demo/garden-api", gardenMemories()); + const app = createApp({ + storeFactory: () => store, + observer: null, + sweeper: null, + getSyncRuntimeStatus: () => ({ phase: "disabled", detail: "Synthetic docs fixture" }), + getUpdateStatus: async () => offlineUpdateStatus(VERSION), + }); + + const started = await new Promise<{ server: ViewerServer; port: number }>((resolve, reject) => { + const candidate = serve({ fetch: app.fetch, hostname: "127.0.0.1", port: 0 }, (info) => + resolve({ server: candidate, port: info.port }), + ); + candidate.once("error", reject); + }); + server = started.server; + printFixture(paths, started.port, memoryCount); + await waitForShutdown(); + } finally { + try { + if (server) await closeServer(server); + } finally { + store.close(); + } + } +} + +async function main(): Promise { + if (process.env[CHILD_MARKER] === "1") { + await runFixture(); + return; + } + await launchIsolatedChild(); +} + +void main().catch((error) => { + console.error(error instanceof Error ? (error.stack ?? error.message) : String(error)); + process.exitCode = 1; +});