diff --git a/docs/convergence-integration.md b/docs/convergence-integration.md new file mode 100644 index 0000000..dfcc998 --- /dev/null +++ b/docs/convergence-integration.md @@ -0,0 +1,313 @@ +# Convergence Integration — the Trajectory Lens in Agent Relay Loop + +**How trajectory data maps into the Agent Relay Loop convergence store.** + +Agent Relay Loop converges three capture lenses into one normalized event store +(`convergence_events` in `relayhistory-cloud`): + +- **cost/usage** — `burn` +- **reasoning** — `trajectories` (this repo) +- **recall** — `ai-hist` / `relayhistory` + +This doc is the **source-of-truth mapping for the reasoning lens**: what a tool emits +when it pushes trajectory data, and how it lands in the store. It is the trajectory +slice of the ratified WS-1 contract. + +> **Authoritative schema:** `relayhistory-cloud/docs/decisions/2026-06-21-normalized-agent-event-schema.md` +> (the WS-1 ADR). This doc describes the same contract from the trajectory side; if they +> ever diverge, the ADR wins. + +--- + +## The convergence envelope + +A client emits one **envelope per event** in the `records[]` of `POST /v1/ingest`. +Tenancy (`orgId`/`workspaceId`/`machineId`/`userId`) is **never sent** — the server +derives it from the `rth_at_*` auth context. The client sends provenance + content only. + +```jsonc +{ + "v": 1, + "kind": "decision", // event class — IN THE PK + "source": "trajectories", // capture lens/harness — IN THE PK + "lens": "trajectories", // non-PK query facet + "sessionId": "traj_abc", // = the trajectory id — IN THE PK + "eventId": "decision:traj_abc:0", // deterministic, namespaced — IN THE PK + "ts": "2026-06-21T10:00:00.000Z", // ISO-8601 (convert epoch-ms → ISO before emit) + "type": "decision", + "content": "Question: …\nChose: …", // RAW readable text (no `Task:` prefix — server adds it) + "confidence": 0.9, // source-native float 0..1 (server converts) + "actorName": "planner", // from persona_id / agent + "taskTitle": "Build WS-1 schema", // structured field (server folds into content) + "record": { "decision": { "chosen": "…", "alternatives": ["…"] } } +} +``` + +Natural key (PK): `(orgId, machineId, source, sessionId, kind, eventId)`. Re-pushing the +same event upserts the same row — sync is idempotent. + +--- + +## Field mapping: trajectory schema → convergence envelope + +| Trajectory source (`src/core/schema.ts`) | Convergence field | Notes | +| --- | --- | --- | +| `Trajectory.id` (`traj_…`) | `sessionId` | the trajectory IS the session for this lens | +| `TrajectoryEvent.ts` (epoch-ms `number`) | `ts` | **convert epoch-ms → ISO-8601** before emit | +| `TrajectoryEvent.type` | `type` | reasoning vocab carried through | +| `TrajectoryEvent.content` | `content` | raw; server scrubs + prepends `Task:` | +| `TrajectoryEvent.significance` | `significance` | enum (`low\|medium\|high\|critical`) — never numeric | +| `TrajectoryEvent.confidence` / `Decision.confidence` / `Retrospective.confidence` | `confidence` | float `0..1` on the wire; server → `confidence_basis_points` (×10000) | +| `TrajectoryEvent.tags` | `tags` | carried through | +| `TrajectoryEvent.raw` | — | **dropped** by default-tier scrub; never emitted | +| `agents[].name` / `chapter.agentName` (`persona_id`) | `actorName` | "which engineer/agent" attribution | +| `Trajectory.projectId` | `projectId` | | +| `Trajectory.task.title/description/status` | `taskTitle` / `taskDescription` / `taskStatus` | structured fields; `taskTitle` feeds embedding via server `Task:` enrichment | +| `Trajectory.task.source` `{system,id}` | `taskRef` | future cross-lens correlation seed (often `null`; not persisted by `ai-hist` yet) | +| `Decision` `{question,chosen,reasoning,alternatives}` | `content` + `record.decision` | client renders readable `content`; `alternatives` is `string[] \| {option,reason}[]` | +| `Retrospective.{summary,approach,learnings[],suggestions[],challenges[]}` | fan-out events | see below | +| agent-trace `files[].path/ranges` | `filesTouched` / `codeChurn` | | + +--- + +## Event-id scheme (deterministic, kind-namespaced, collision-free) + +`TrajectoryEvent` has **no native id**, so ids are synthesized deterministically so the +same event always re-pushes to the same PK: + +| Source | `kind` | `eventId` | +| --- | --- | --- | +| chapter stream event | (event type) | `trajevent::` *(deferred — see scope)* | +| decision[i] | `decision` | `decision::` | +| retrospective `learnings[i]` | `finding` | `finding::learning:` | +| retrospective `challenges[i]` | `finding` | `finding::challenge:` | +| retrospective `suggestions[i]` | `reflection` | `reflection::suggestion:` | +| retrospective `summary` | `reflection` | `reflection::summary` | +| retrospective `approach` | `reflection` | `reflection::approach` | + +**Rules that keep this collision-free:** +- `kind` is in the PK, and the `::` segment disambiguates same-kind arrays + (so `finding:…:learning:0` ≠ `finding:…:challenge:0`). +- Retrospective events key off **`trajectoryId`, not `chapterId`** — the retrospective is + trajectory-level and has no chapter. +- Index by **natural stored array order** — never sort/dedupe/filter before indexing, or + indices shift and idempotency breaks. +- Emit explicit ids; never rely on a server fallback for retro events. + +--- + +## Server-owned transforms (do NOT do these client-side) + +The ingest server is the boundary that normalizes. Clients send raw + source-native: + +- **Confidence:** wire carries float `0..1`; server `toBasisPoints()` → `confidence_basis_points` + (int `0..10000`). One conversion point. Do not pre-convert. Confidence is **not** shadowed + in `record` — single persisted scale. +- **Task enrichment:** server prepends `Task: ` (and `Task description:`) to + `content` before embedding. The client sends **raw** content — do **not** pre-prefix, or + you double it. +- **Scrub (compliance boundary):** server redacts secrets/PII in `content` in place; drops + `raw` wholesale; minimizes `record` to a bounded allowlist; normalizes home-dir paths + (`/Users//` → `~`). Client-side scrub is defense-in-depth only, never the boundary. + +--- + +## v1 scope vs deferred + +- **v1 (shipping):** the **distilled** lens — `decisions` + `retrospective`. This is the + highest signal-per-byte (it powers Plan/WS-5) and is what `ai-hist`'s local store + persists today. +- **Deferred to WS-6/Pair:** the full **chapter event stream** (`trajevent:*` — + prompts/thinking/tool_calls). It is not in `ai-hist`'s local store and requires re-parsing + the source trajectory file via its `path`. Sequenced with Pair because the raw sequence is + what real-time failure-mode warnings need. + +--- + +## Acceptance fixture + +The trajectory→row contract is pinned end-to-end by one shared fixture (`traj_abc`), +asserted from both ends: + +- **client:** `ai-hist-core` Rust `matches_trajectory_expert_acceptance_fixture` +- **server:** `relayhistory-cloud` PGlite `ingest-integration.test.ts` + +A decision (confidence 0.9) + a retrospective (summary/approach 0.8; one learning, +suggestion, challenge) produces exactly 6 rows: + +| eventId | kind | confidence_basis_points | +| --- | --- | --- | +| `decision:traj_abc:0` | decision | 9000 | +| `reflection:traj_abc:summary` | reflection | 8000 | +| `reflection:traj_abc:approach` | reflection | 8000 | +| `finding:traj_abc:learning:0` | finding | null | +| `reflection:traj_abc:suggestion:0` | reflection | null | +| `finding:traj_abc:challenge:0` | finding | null | + +All rows: `source`/`lens = "trajectories"`, `session_id = "traj_abc"`, +`task_title = "Build WS-1 schema"`, `content` server-prefixed with `Task:`, re-push → still +6 rows (idempotent). + +--- + +## Addendum (v1.1): compacted roll-up schema — the LLM-compaction lens + +**Why this exists.** The LLM-compaction workflow writes +`~/Projects//.trajectories/compacted/compact_*.json` — roll-ups that distill many +source trajectories into the highest signal-per-byte reasoning (decisions, lessons, key +findings). Empirically these carry **more Pair-surfaceable value** than per-run `completed/` +trajectories (a 200-file sample of `completed/` was ~94% `summary`/`approach` only — 0 +decisions/suggestions; the compacted roll-ups average ~9 decisions + ~27 keyFindings + ~8 +lessons each). `parse_trajectory_file` historically **skipped** `type == "compacted"` +(treated as non-ingestable summaries) — which is why an all-`prompt` corpus reached prod and +Pair stayed silent on real data. This addendum defines how to ingest them. + +**Compacted envelope (observed shape):** +```jsonc +{ + "id": "compact_d60kxya4ert0_2026-04-20", // → sessionId / trajectoryId / eventId namespace + "type": "compacted", + "sourceTrajectories": ["traj_…", …], // provenance only — NOT mapped to events + "decisions": [{ question, chosen, reasoning, impact }, …], + "decisionGroups": [{ category, decisions:[…] }, …], // categorization view — NOT mapped (double-count) + "keyFindings": ["…", …], + "keyLearnings":["…", …], + "lessons": [{ context, lesson, recommendation }, …], + "conventions": [{ pattern, rationale, scope }, …], + "narrative": "…", + "openQuestions":["…", …] +} +``` + +**Mapping: compacted field → convergence event** + +| Compacted field | `kind` | `eventId` | value | +| --- | --- | --- | --- | +| `decisions[i]` | `decision` | `decision::` | high | +| `lessons[i]` (`recommendation` is the actionable nudge) | `reflection` | `reflection::lesson:` | **highest — the "remember to…" equivalent** | +| `keyFindings[i]` | `finding` | `finding::keyfinding:` | med | +| `keyLearnings[i]` | `finding` | `finding::learning:` | med | +| `conventions[i]` | `reflection` | `reflection::convention:` | med | +| `narrative` | `reflection` | `reflection::summary` | low | +| `openQuestions[i]` | `finding` | `finding::openquestion:` | low | +| `decisionGroups` | — | *(skip)* | categorization of `decisions[]`; emitting it double-counts. Optionally fold `.category` into the matching decision event's `tags`. | + +**Rules (consistent with the v1 contract above):** +- `sessionId` = `trajectoryId` = the compaction `id`; `source`/`lens = "trajectories"`. +- Every compacted-derived event carries `tag: "compacted"` so roll-up reasoning is + distinguishable from per-run trajectory events (and Pair/queries can filter if desired). +- Index by **natural stored array order**; never sort/dedup/filter before indexing (idempotent PK). +- `content` = raw readable text (e.g. a lesson → `"Lesson: …\nRecommendation: …"`); the + **server** scrubs + Task-enriches. Client never pre-prefixes. +- No per-item `confidence` in this schema → `confidence` omitted → `confidence_basis_points = null`. + `significance` omitted unless the workflow emits it. +- `parse_trajectory_file` must **stop skipping** `type == "compacted"` (add a compacted branch); + `map_compacted` applies the table above. Per-run trajectory mapping is unchanged. + +**Acceptance fixture (`compact_fixture`) — pinned both ends like `traj_abc`:** + +A compacted file with 2 `decisions`, 2 `keyFindings`, 1 `keyLearnings`, 2 `lessons`, +1 `conventions`, a `narrative`, and 1 `openQuestions` produces exactly **10 rows**: + +| eventId | kind | +| --- | --- | +| `decision:compact_fixture:0` | decision | +| `decision:compact_fixture:1` | decision | +| `finding:compact_fixture:keyfinding:0` | finding | +| `finding:compact_fixture:keyfinding:1` | finding | +| `finding:compact_fixture:learning:0` | finding | +| `finding:compact_fixture:openquestion:0` | finding | +| `reflection:compact_fixture:convention:0` | reflection | +| `reflection:compact_fixture:lesson:0` | reflection | +| `reflection:compact_fixture:lesson:1` | reflection | +| `reflection:compact_fixture:summary` | reflection | + +All rows: `source`/`lens = "trajectories"`, `session_id = "compact_fixture"`, `tag` contains +`"compacted"`, `content` server-prefixed with `Task:`; re-push → still 10 rows (idempotent). +Counts: `decision×2`, `finding×4` (2 keyfinding + 1 learning + 1 openquestion), +`reflection×4` (2 lesson + 1 convention + 1 summary). + +**Scrub-bait (load-bearing — pins server-scrub on the new path):** `lessons[0].recommendation` +in the fixture carries a planted secret +`ghp_FAKE0000000000000000000000000000abcd`. The resulting +`reflection:compact_fixture:lesson:0` `content` **must** contain `[REDACTED]` and **must not** +contain `ghp_`. (A clean-only fixture would make a "scrub ✅" vacuous — same self-validating-bait +principle as the per-run `finding:traj_abc:learning:0` bait.) + +### Security acceptance criteria (per reviewer — the mapper is reviewed against these) +1. **Scrub fires on the new path** — the planted bait above ingests as `[REDACTED]` (server-side + scrub, the compliance boundary; `recommendation`/`reasoning`/`narrative` text is exactly where + a real secret would hide). +2. **No raw-blob echo** — only the scrubbed, distilled `content` is stored/surfaceable; the raw + compacted JSON (`sourceTrajectories`, full blob) is **dropped wholesale** (same `raw`-drop + principle as per-run trajectories). Pair snippets never expose the un-distilled blob. +3. **Tenancy server-derived** — `orgId`/`machineId`/`workspaceId` come from the auth context + server-side; the mapper/client never asserts them from the file. +4. **Idempotent PK** — natural-order eventIds + the `compacted` tag ⇒ re-ingest is a no-op, never + duplicate citations (the dup-citation usefulness issue, avoided by construction). + +--- + +## Addendum (v1.2): the Learn lens — distill *ordinary session history* into Pair signal + +**Why this exists (the adoption ceiling).** Pair surfaces `decision`/`finding`/`reflection` +events and **excludes `prompt`**. Those high-signal kinds come from *distilled* reasoning — +which today requires either trajectory capture (per-run) or the compaction roll-ups (v1.1). +**Most users have neither** — they have ordinary Claude Code / Codex **session/prompt history**. +So Pair returns empty `allow` for them (verified: on a 24.4k all-`prompt` prod corpus Pair was +silent until distilled events existed). The **Learn lens** closes this: distill a user's own +session history into the same convergence events, so Pair works **without** pre-existing +trajectories. + +**Producer-pluggable architecture.** Pair stays source-agnostic: every producer (per-run +trajectories, compaction roll-ups, **Learn-distill**, future mining) emits the **same +convergence envelope** and flows through the **single `/v1/ingest` scrub chokepoint** → it +inherits scrub + tenancy + idempotency for free. Pair never knows the source. + +**Pipeline (maximal reuse — most of this is already shipped):** +``` +session history → [input adapter] → distiller (reuse compaction prompt+schema) → + compacted-rollup schema → [reuse the merged v1.1 map_compacted] → decision/finding/reflection → ingest +``` +- **Input adapter (new):** build distiller input from a **full session transcript** — + prompt + response + **tool-call outputs** — not bare prompts (quality collapses on prompts + alone). Window by session (or session-cluster). +- **Distiller (reuse):** `COMPACTION_SYSTEM_PROMPT` + `COMPACTED_OUTPUT_SCHEMA` + (`src/compact/prompts.ts`) already turn "work sessions" into decisions/keyFindings/lessons. + Output = the v1.1 compacted-rollup schema → **reuse `map_compacted`** unchanged. +- **eventId namespace (new):** namespace by a **stable distillation-unit id**, + `learn_` — same kind/segment scheme as v1.1 + (`reflection:learn_:lesson:`, `decision:learn_:`, …) + `tag:["learn"]`. + The hash must be **content-stable** so re-distilling the same window yields the **same + eventIds** (idempotent PK → no duplicate citations across re-distill windows). + +**🔒 LLM-locality / privacy model (the headline design decision — gates default-on):** +The distiller input is the **full transcript incl. tool-call outputs** — the *peak* +secret/file-content density in the corpus, and it is **pre-scrub** at distill time (our scrub +is at ingest, *after* distillation). Therefore: +- **Default-on ⇒ LOCAL distillation REQUIRED** — raw transcript must not leave the machine to a + cloud LLM. The compaction provider already supports a `baseUrl` override + (`OPENAI_BASE_URL`/`ANTHROPIC_BASE_URL`) → point at a local model (e.g. Ollama). +- **Cloud-LLM distillation ⇒ explicit opt-in + disclosure**, and only sound if the + distill-provider == the agent-provider the user already trusts with these sessions. +- **Invariant either way:** raw history **never reaches Agent Relay / relayhistory servers** — + only distilled + scrubbed events sync. Distillation is **client-orchestrated** (the user's own + LLM key); the server only ever sees post-distill, post-scrub events. + +**Ranking note (from prod real-data verification):** generic `reflection:summary`/`:approach` +events are abundant (on a real corpus: thousands of summaries vs. tens of decisions) and **flood +out** the high-value `:lesson`/`decision`/`:suggestion` signal. The Learn lens **must not** emit +a `:summary`/`:approach` per session (it would bury the actionable signal). Either (a) omit +generic summaries from the Learn output, or (b) the Pair ranker **down-weights** +`:summary`/`:approach` relative to `:lesson`/`decision`/`:suggestion`. Favor the *actionable* +kinds; summaries are context, not warnings. + +**Acceptance fixture (`learn_fixture`) — pinned both ends, with bait:** a single session +transcript distills to a bounded event set (e.g. 1 decision + 2 lessons + 1 keyFinding); the +transcript contains a planted `ghp_FAKE…` in a tool-call output, and the resulting +`reflection:learn_:lesson:` `content` **must** be `[REDACTED]` (scrub on the Learn path). +Re-distilling the same transcript → identical eventIds (idempotent). + +**Security acceptance (reviewed against):** (1) all Learn output through the ingest-scrub +chokepoint (bait → `[REDACTED]`); (2) distillation locality — default-on only if local; +cloud-LLM opt-in + disclosed; raw never to Agent Relay servers; (3) stable/idempotent eventIds.