Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ jobs:
publish_if_missing "@codemem/server" "packages/viewer-server"
publish_if_missing "codemem" "packages/cli"
publish_if_missing "@codemem/opencode-plugin" "packages/opencode-plugin"
publish_if_missing "@codemem/pi-extension" "packages/pi-extension"

- name: Verify latest dist-tag is absent or stable
# npm assigns `latest` to a package's first-ever version regardless of
Expand Down
46 changes: 43 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ codemem is persistent coding memory across sessions, machines, and teammates for
- **Hybrid retrieval** — FTS5 BM25 lexical search + sqlite-vec semantic search, merged and re-ranked
- **Automatic injection for OpenCode 1** — the plugin injects context into every prompt, no manual steps
- **Claude Code plugin support** — install from the codemem marketplace source
- **Multi-agent** — OpenCode, Claude Code, Codex, and pi share one project-scoped store
- **Built-in viewer** — browse memories, sessions, and observer output in a local web UI
- **Remote MCP access** — advanced single-user self-hosting can expose an OAuth-protected Streamable HTTP MCP endpoint to configured remote clients; keep the localhost viewer private ([guide](docs/remote-mcp-oauth.md))

Expand Down Expand Up @@ -194,14 +195,53 @@ Codex hook ingestion shares the same raw-event pipeline as Claude and OpenCode t

> Was this repository previously installed as `opencode-mem`? See the [rename migration guide](docs/rename-migration.md). It covers this repository's former name, not importing data from [`tickernelz/opencode-mem`](https://github.com/tickernelz/opencode-mem).

## How it works
### Pi

Pi support ships as the `@codemem/pi-extension` pi-package. Install the CLI, then let setup wire the extension:

```text
npm i -g codemem
codemem setup
```

`codemem setup` auto-detects pi (`pi` on PATH or the agent dir; honors `PI_CODING_AGENT_DIR`) and appends `npm:@codemem/pi-extension@<version>` to `~/.pi/agent/settings.json` `packages`. Flags:

| Flag | Purpose |
|------|---------|
| `--pi-only` | Only configure pi |
| `--pi-mcp` | Opt into MCP via third-party `pi-mcp-adapter` (writes `mcp.json` only when the adapter is detected) |
| `--pi-extension-path <path>` | Dev: write a local-path `packages` entry instead of the npm pin |

Uninstall: remove the `@codemem/pi-extension` entry from pi's `packages` list and restart pi. The shared memory store is left intact.

What you get:

- **Ingest** — extension POSTs to `POST /api/pi-hooks` (a compatibility alias that normalizes the payload once and runs it through the canonical ingest envelope with `source: "pi"`, the same event identity as `POST /api/raw-events`), with `codemem pi-hook-ingest` CLI fallback (spool when offline)
- **Injection** — turn-local `systemPrompt` append on `before_agent_start` (`## codemem memories`); never the persistent `message` channel
- **Tools** — all 14 `memory_*` tools registered natively via `pi.registerTool` (HTTP preferred, CLI fallback). No `pi-mcp-adapter` required for tools
- **Compaction** — pi-only observe-only boundary: `session_before_compact` flushes extraction before pi discards context; never replaces pi's summarizer
- **Fork/resume** — stream identity re-keys on every `session_start`
- **Project identity** — the extension resolves the project from the nearest Git root (same walk as the other adapters)
- **Dashboard** — pi rows appear in the source-agnostic feed/sessions/projects tabs with no extra setup

Cross-agent: one shared store. Memories from OpenCode/Claude/Codex sessions inject into pi (and the reverse) because packs are project-scoped, never agent-scoped.

Caveats (v1):

- Observer extraction from pi config supports **API-key providers only**. OAuth-only installs get an explicit `unconfigured (oauth-only)` status — never a silent 401. Set `observer_provider` / `observer_model` explicitly when needed. Selection is cheap-model-first.
- Preferred HTTP `GET /api/pack` is unledgered — pi injection does not write an opencode retrieval-ledger row.
- `--pi-mcp` requires the third-party `pi-mcp-adapter` package; without it setup writes nothing MCP-related and explains the prerequisite. Native tools remain the default surface (`pi.tools_mode: native`).

See [`packages/pi-extension/README.md`](packages/pi-extension/README.md) and [docs/plugin-reference.md](docs/plugin-reference.md) for config knobs and lifecycle details.

Adapters hook into runtime event systems (the OpenCode 1 plugin and Claude hooks). They capture tool calls and conversation messages, flush them through an observer pipeline that produces typed memories, and surface retrieval context for future prompts.

> The workflow below describes OpenCode 1 recall. OpenCode 2 captures activity,
> manages lifecycle cleanup, and exposes manual memory tools, but it does not inject
> automatic recall because its context hook cannot identify the request safely.

## How it works

Adapters hook into runtime event systems (OpenCode plugin, Claude/Codex hooks, and the pi extension). They capture tool calls and conversation messages, flush them through an observer pipeline that produces typed memories, and surface retrieval context for future prompts.
```mermaid
sequenceDiagram
participant OC as OpenCode 1
Expand Down Expand Up @@ -292,7 +332,7 @@ For architecture details, see [docs/architecture.md](docs/architecture.md).
| **Plumbing** | `codemem mcp` | MCP stdio server; best-effort starts the local viewer unless `CODEMEM_VIEWER=0` or `CODEMEM_VIEWER_AUTO=0` is set |
| | `codemem mcp http` | Local Streamable HTTP MCP server (`POST /mcp`, loopback-only by default) |

Run `codemem --help` for the human-facing command list. Adapter plumbing commands (`claude-hook-*`, `codex-hook-*`, `enqueue-raw-event`, and `prompt-pack-ledger`) remain executable for packaged-plugin and stale-client compatibility but are hidden from help and shell completion. `show`, `forget`, and `remember` still work as hidden top-level aliases. `export-memories` and `import-memories` remain visible but are deprecated — they warn on stderr and will be hidden from help and completion in a future release; use `codemem memory export` / `codemem memory import`.
Run `codemem --help` for the human-facing command list. Adapter plumbing commands (`claude-hook-*`, `codex-hook-*`, `pi-hook-*`, `enqueue-raw-event`, and `prompt-pack-ledger`) remain executable for packaged-plugin and stale-client compatibility but are hidden from help and shell completion. `show`, `forget`, and `remember` still work as hidden top-level aliases. `export-memories` and `import-memories` remain visible but are deprecated — they warn on stderr and will be hidden from help and completion in a future release; use `codemem memory export` / `codemem memory import`.

Use `codemem status` to answer whether the local database, viewer, sync, maintenance,
semantic index, raw-event ingestion, and observer need attention. It is observational:
Expand Down
15 changes: 13 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ codemem has five main pieces: **adapters** that capture shell/runtime activity,

| Component | What it does | Key files |
|-----------|-------------|-----------|
| Adapters | Capture and normalize agent events before enqueueing raw events | `packages/opencode-plugin/.opencode/plugins/codemem.js`, `packages/opencode-plugin/.opencode/lib/runtime.js`, `plugins/claude/scripts/ingest-hook.mjs`, `plugins/codex/scripts/ingest-hook.mjs`, `packages/core/src/claude-hooks.ts`, `packages/core/src/codex-hooks.ts` |
| Adapters | Capture and normalize agent events before enqueueing raw events | `packages/opencode-plugin/.opencode/plugins/codemem.js`, `packages/opencode-plugin/.opencode/lib/runtime.js`, `plugins/claude/scripts/ingest-hook.mjs`, `plugins/codex/scripts/ingest-hook.mjs`, `packages/core/src/claude-hooks.ts`, `packages/core/src/codex-hooks.ts`, `packages/core/src/pi-hooks.ts`, `packages/pi-extension/` |
| Ingest pipeline | Extracts tool events, builds transcripts, runs the observer | `packages/core/src/ingest-pipeline.ts`, `packages/core/src/ingest-events.ts` |
| Observer | Produces typed observations and session summaries from transcripts | `packages/core/src/observer-output.ts`, `packages/core/src/observer-output-schema.ts`, `packages/core/src/ingest-xml-parser.ts` |
| Store | SQLite persistence for sessions, memories, artifacts, embeddings | `packages/core/src/store.ts`, `packages/core/src/schema.ts` |
Expand All @@ -29,6 +29,8 @@ flowchart LR
CH -->|same envelope: enqueue-raw-event| DB
CX["Codex hooks"] -->|normalize once; POST /api/raw-events| VW
CX -->|same envelope: enqueue-raw-event/spool| DB
PI["pi extension"] -->|POST /api/pi-hooks (alias)| VW
PI -->|fallback: pi-hook-ingest direct enqueue/spool| DB
VW --> DB["SQLite"]
DB -->|flush batch claimed| IN["Ingest pipeline"]
IN --> OB["Observer"]
Expand All @@ -40,7 +42,7 @@ flowchart LR

1. Adapters capture tool/conversation lifecycle events and normalize them into raw events with optional `_adapter` envelopes.
2. OpenCode streams raw events to the viewer ingest API (`POST /api/raw-events`) with preflight checks (`GET /api/raw-events/status`) and can fall back to CLI queue enqueue when stream writes fail. Prompt-time packs and prompt-pack ledger transitions also use viewer POST APIs first, with CLI fallback only for retryable transport or version failures.
3. Claude and Codex use checked-in, dependency-free normalizers generated from their core TypeScript implementations. Each detached event wrapper normalizes once, posts the exact envelope to `POST /api/raw-events`, and reuses that serialization for `enqueue-raw-event` or durable spool fallback. The canonical endpoint accepts additive adapter metadata. Named hook routes remain compatibility aliases used by older packaged or plugin-free CLI paths; the current packaged-wrapper audit found no named-route strings, so the aliases are not primary but cannot be removed yet.
3. Claude and Codex use checked-in, dependency-free normalizers generated from their core TypeScript implementations. Each detached event wrapper normalizes once, posts the exact envelope to `POST /api/raw-events`, and reuses that serialization for `enqueue-raw-event` or durable spool fallback. The canonical endpoint accepts additive adapter metadata. Named hook routes remain compatibility aliases used by older packaged or plugin-free CLI paths; the current packaged-wrapper audit found no named-route strings, so the aliases are not primary but cannot be removed yet. The pi extension posts pi lifecycle payloads to `POST /api/pi-hooks`, a compatibility alias that maps the payload through `buildRawEventEnvelopeFromPiEvent` and runs `ingestNormalizedEnvelope` with `source: "pi"` — the same event identity as `POST /api/raw-events` with `source: "pi"` — with `codemem pi-hook-ingest` (plus spool) as the fallback.
4. The viewer/store persists raw events and queues durable flush batches.
5. Idle and sweeper workers claim batches and run them through ingest.
6. Before building session context, raw events are passed through `normalizeEventsForSessionContext` (in `ingest-transcript.ts`) which projects adapter-enveloped events (`_adapter` schema v1.0) into the flat `user_prompt` / `tool.execute.after` shapes that `buildSessionContext` scans. This is critical for Claude Code hook events which always arrive wrapped in the adapter envelope.
Expand All @@ -62,6 +64,7 @@ Support tiers describe operational expectations for each adapter path:
| OpenCode 1 plugin | Supported | Primary reference adapter for lifecycle events and injection behavior. |
| OpenCode 2 plugin | Experimental | The beta entrypoint captures conversation, tool, terminal usage, and lifecycle activity with bounded cleanup. It exposes manual `mem-status`, `mem-recent`, and `mem-stats` tools through `tool.transform` with `codemode: false`; automatic recall remains disabled because the context hook has no request kind or request ID. |
| Claude hooks/plugin | Supported | Hook-first queue path with CLI/runtime fallback and parity slices tracked in adapter stack PRs. |
| pi extension | Supported | Thin pi-package (`packages/pi-extension`, `packages/core/src/pi-hooks.ts`): extension → `POST /api/pi-hooks` alias → canonical ingest envelope (`source: "pi"`) → observer → memories; turn-local `systemPrompt` injection; 14 native `memory_*` tools; observe-only compaction boundary; fork/resume-aware streams; Git-root project identity. Observer derivation from pi config is API-key-only in v1 (OAuth → explicit `unconfigured (oauth-only)`). |
| Codex plugin (hooks + MCP) | Supported | Functional capture pipeline (`plugins/codex/`, `packages/core/src/codex-hooks.ts`) dogfooded end-to-end: edge normalization → `POST /api/raw-events` → observer → memories. Prompt-time injection is present and env-gated but not fully validated on strict models. |
| Windsurf integration | Experimental | Planned via shared adapter contract after OpenCode/Claude stabilization. |
| Cursor integration | Experimental | Planned via shared adapter contract after OpenCode/Claude stabilization. |
Expand Down Expand Up @@ -371,6 +374,14 @@ delivery reuses the exact envelope for CLI enqueue fallback. The named `POST /ap
remains a compatibility alias/caller for older packaged or plugin-free CLI paths. The queue/sweeper
behavior and `CODEMEM_CLAUDE_HOOK_FLUSH_ON_STOP=1` opt-in for `Stop` remain unchanged.

Pi extension ingest posts pi lifecycle payloads to the `POST /api/pi-hooks` compatibility alias, which
normalizes them through `buildRawEventEnvelopeFromPiEvent` and runs `ingestNormalizedEnvelope` with
`source: "pi"` (same event identity as canonical `POST /api/raw-events`); retryable delivery reuses the
payload for `codemem pi-hook-ingest` plus a pi-specific spool. Boundary flush events
(`session_before_compact`, `session_shutdown`) always go through the CLI so extraction actually runs
before pi discards context. Preferred HTTP `GET /api/pack` is unledgered (no opencode
retrieval-ledger row). The queue/sweeper behavior is shared with the other adapters.

### OpenCode session finalization triggers
- `session.idle` — finalizes current local buffer
- `session.created` — finalizes before switching to a new session
Expand Down
40 changes: 40 additions & 0 deletions docs/plugin-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,46 @@ Hooks loaded from the user config layer require a one-time trust approval in Cod
- **Normalized spool backlog drains automatically** at one envelope per successful ingest. The wrapper never reads or removes files from the legacy native-hook spool.
- **A model rejects injected context** (for example "the conversation must end with a user message"): disable prompt-time injection with `CODEMEM_INJECT_CONTEXT=0`. Capture/ingest keeps working and recall is still available through the MCP tools.

## Pi extension

Pi support is the `@codemem/pi-extension` pi-package. Install once, then restart pi:

```text
npm i -g codemem
codemem setup --pi-only
```

Setup appends `npm:@codemem/pi-extension@<version>` to `~/.pi/agent/settings.json` `packages` (JSONC-safe, idempotent). It also derives unset `observer_*` keys from pi's API-key providers (cheap-model-first) without copying secrets. Flags:

- `--pi-mcp` — opt into MCP via third-party `pi-mcp-adapter` (writes `mcp.json` only when the adapter is present; flips `pi.tools_mode` to `mcp-adapter`)
- `--pi-extension-path <path>` — dev local-path `packages` entry

Uninstall by removing the packages entry and restarting pi.

### Surfaces

| Surface | Behavior |
|---|---|
| Ingest | Extension POSTs to `POST /api/pi-hooks`, a compatibility alias that normalizes the payload once into the canonical ingest envelope with `source: "pi"` — the same event identity as `POST /api/raw-events` with `source: "pi"`. Falls back to `codemem pi-hook-ingest` + spool when HTTP is unavailable. Boundary events (`session_before_compact`, `session_shutdown`) always flush via the CLI so extraction actually runs |
| Injection | `before_agent_start` appends a turn-local `systemPrompt` block (`## codemem memories`); never returns `message` |
| Tools | 14 native `memory_*` tools via `pi.registerTool` (HTTP preferred, CLI fallback). Default `pi.tools_mode: native` — no adapter required |
| Compaction | Observe-only: `session_before_compact` flushes extraction; never returns a custom `compaction` summary |
| Fork/resume | Re-keys stream identity on every `session_start`; durable cursors via `pi.appendEntry` |
| Project identity | Nearest Git root (walks up for a directory `.git` or a `gitdir:` worktree file), same walk as the other adapters |

Prompt-time pack retrieval uses the preferred HTTP `GET /api/pack` (or `codemem pi-hook-inject` / `pack --json` fallback). That HTTP pack path is unledgered — no opencode retrieval-ledger row is written for pi injection.

Dashboard tabs are source-agnostic: pi rows appear alongside OpenCode/Claude/Codex with no extra setup. Packs are project-scoped, so memory crosses agents automatically.

### Observer derivation caveats (v1)

- API-key providers only (`openai-completions` / `openai-responses` / `anthropic-messages`). OAuth-only installs surface `unconfigured (oauth-only)` — never silent 401s.
- Explicit `observer_*` config/env always wins over pi-derived values.
- Setup never copies pi `auth.json` keys into the codemem config.
- `--pi-mcp` requires `pi-mcp-adapter`; without it setup explains the prerequisite and writes nothing MCP-related.

See [`packages/pi-extension/README.md`](../packages/pi-extension/README.md) for env knobs and lifecycle rules.

## Post-restart config sanity checklist

After restarting OpenCode or the viewer, run this quick check when behavior looks off:
Expand Down
4 changes: 2 additions & 2 deletions docs/trusted-publisher.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ Trusted publishing must be configured for every package the workflow publishes:
- `@codemem/server`
- `codemem`
- `@codemem/opencode-plugin`
- `@codemem/pi-extension`

Before the first tagged release that includes a new npm package, publish a
distinct bootstrap prerelease such as `0.0.0-alpha.0` with authenticated
maintainer credentials and a non-latest dist-tag such as `bootstrap`. Do not use
the intended release version for this bootstrap. Then configure the trusted
publisher above; npm requires the package to exist first. Do not use a release
tag until this setup is complete.

## GitHub workflow behavior

`.github/workflows/release.yml` publishes from two triggers:
Expand All @@ -45,7 +45,7 @@ dependency order:
4. `@codemem/server`
5. `codemem`
6. `@codemem/opencode-plugin`

7. `@codemem/pi-extension`
Publish command shape:

- `pnpm --filter <package> publish --provenance --access public --tag <dist-tag>`
Expand Down
1 change: 1 addition & 0 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,7 @@ codemem status --db-path ./codemem.sqlite --config ./codemem.json
## Observer auth configuration

- Runtime choices are `api_http`, `claude_sidecar`, and `codex_sidecar`.
- Pi users stay on `api_http`. `codemem setup` can derive unset `observer_provider` / `observer_model` from pi API-key providers (cheap-model-first) without copying secrets. OAuth-only pi installs stay explicitly `unconfigured (oauth-only)` in v1 — set observer settings manually.
- `claude_sidecar` runs observer calls through the local Claude runtime (subscription/session auth) and does not require `ANTHROPIC_API_KEY`.
- `claude_command` controls how `claude_sidecar` invokes Claude CLI (default `["claude"]`).
- Wrapper example: `"claude_command": ["wrapper", "claude", "--"]`
Expand Down
2 changes: 2 additions & 0 deletions docs/versioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ CodeMem uses one shared semantic version stream across its npm packages.

- npm: `codemem` (CLI)
- npm: `@codemem/opencode-plugin` (OpenCode plugin)
- npm: `@codemem/pi-extension` (pi coding-agent extension)

## Policy

Expand All @@ -21,6 +22,7 @@ Version bumps are prepared on a release branch and touch these files:
- `packages/embeddings/package.json` (`version`)
- `packages/cli/package.json` (`version`)
- `packages/opencode-plugin/package.json` (`version`)
- `packages/pi-extension/package.json` (`version`)
- `packages/mcp-server/package.json` (`version`)
- `packages/viewer-server/package.json` (`version`)
- `packages/core/src/index.ts` (`VERSION` export)
Expand Down
Loading
Loading