From 1a944f71d35f5d09595548d405c8dec4e5a484d1 Mon Sep 17 00:00:00 2001 From: DAVID AMID Date: Sun, 30 Aug 2026 15:47:03 +0300 Subject: [PATCH 1/3] docs(analysis): evaluate a Claude Code plugin as a fourth transport MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Asks whether the `components` core can ship as a Claude Code plugin — a thin transport beside `proxy/`, the AuthBridge plugin and `adapters/bifrost`, reusing components via configuration. Not a replacement for the proxy. One fact decides the whole evaluation. No plugin surface can read or write the outbound request: across all ~34 hook events there is no field that rewrites `messages[]`, `system[]`, `tools[]` or `cache_control`, and plugin `settings.json` accepts only `agent` and `subagentStatusLine`, so a plugin cannot even point a session at the proxy. What IS interceptable is one tool result at the moment it is produced, via `PostToolUse` → `updatedToolOutput`. That draws the line between content-scoped and envelope-scoped work, and it is the line between the two component families. Offloaders distribute. `cmdfilter`, `format`, `toon`, `extract`, `collapse`, `smartcrush`, `skeleton` and `dedup` are pure functions from one tool output to a shorter one, which is exactly what the hook hands over and accepts back. Two get better than they are in the proxy: the hook supplies `tool_input.command` and `tool_input.file_path` instead of making `cmdfilter` and `skeleton` infer both from transcript text. Cache management does not distribute, and this is the finding worth arguing with. `cachesplit` restructures the top-level `system` array; `cacheinject` reasons over `messages[]` positionally and emits request metadata; the keepalive on `feat/keepalive-strategies` must originate a request that byte-exactly reproduces a prefix and price it from `CachedTokens`, which only the provider's response carries. None of that is a tool output — it is the envelope, assembled after the last hook runs and never persisted. Verified rather than assumed: a session transcript carries `messages` and `toolUseResult` and no system prompt or tool schemas (`scripts/inspect_transcript.py`). The failure modes are asymmetric too, which is the deeper reason. A wrong offload wastes one expand round-trip, bounded and type-enforced reversible. Wrong cache work inverts: a mistimed keepalive creates at 1.25x instead of refreshing at 0.1x, a breakpoint over budget is a 400, a representation flip inside a cached prefix re-writes the suffix at 11.5x. Envelope work has no merely-no-saving fail direction. The compensating result is that the defensive half of our KV-cache layer stops being necessary rather than being ported. `state.go` names its own premise — an offloader must re-emit identical bytes "otherwise the agent (which re-sends the ORIGINAL each turn)" flips the representation. A hook rewrites the output before it enters the transcript, so the agent never holds the original. Freeze/replay, `MaxCachedIdx`, `Tracker`, `frozen_flips` and sticky ids have nothing left to defend, and `extract_llm`'s sampling nondeterminism stops disqualifying it from repair. Also `PreCompact`/`SessionEnd` replace `proxy/agentcompaction.go`'s string match against Claude Code 2.1.215 internals, which has a documented reachable false positive. What is given up: `cachesplit` (-34.1% cost, 0% -> 96.7% hit, and in every preset), plus `mask` (27.5% Terminal-Bench, 12.5% SWE-bench) and `failed_run`, both of which rewrite EARLIER messages and so cannot work at a hook that fires once at birth. On implicit prefix-cache backends (vLLM/llm-d) the cache loss is zero, because `prefixsplit` is already a no-op there. Also documented: permanence cuts both ways (an expand's restored original joins the transcript for good, so the plugin wants a more conservative pipeline than the proxy), `/stats` cost tiers are unobtainable so plugin mode cannot be benchmarked the way the proxy is, and the whole thing is gated on one unverified fact — whether `updatedToolOutput` persists into the transcript. If it does not, the proposition collapses to an expand-only MCP server. That experiment is named as gate 0 and should run before any adapter code. On DAM: land the proxy in the gateway. DAM is harness-plural and its egress already matches our gateway credential model, so a Claude-Code-only plugin covers one harness of four and none of the bring-your-own-ACP case. Ship the plugin as the Claude-Code-session layer on top, never as the DAM integration. Docs only — no code, no behavior change. Signed-off-by: DAVID AMID Assisted-By: Claude Opus 5 (1M context) --- docs/analysis/claude-code-plugin.md | 442 ++++++++++++++++++++++++++++ scripts/inspect_transcript.py | 49 +++ 2 files changed, 491 insertions(+) create mode 100644 docs/analysis/claude-code-plugin.md create mode 100644 scripts/inspect_transcript.py diff --git a/docs/analysis/claude-code-plugin.md b/docs/analysis/claude-code-plugin.md new file mode 100644 index 00000000..a084cce1 --- /dev/null +++ b/docs/analysis/claude-code-plugin.md @@ -0,0 +1,442 @@ +# Claude Code plugin as a third transport for context-guru + +Scope: can the existing `components` core be deployed as a [Claude Code +plugin](https://code.claude.com/docs/en/plugins) — as a *thin* fourth host next to +`proxy/`, the AuthBridge plugin and `adapters/bifrost`, reusing components via +configuration rather than reimplementing them? And does building it help the DAM push? + +Not a replacement for the proxy. Deployment/transport only. + +## The one constraint that decides everything + +**No plugin surface can read or write the outbound Messages API request.** Confirmed +against the [hooks reference](https://code.claude.com/docs/en/hooks#decision-control): +across all ~34 hook events there is no field that rewrites `messages[]`, `system[]`, +`tools[]`, or any `cache_control`. `UserPromptSubmit` explicitly "can't replace the +prompt; it only injects `additionalContext` alongside it". Plugin `settings.json` accepts +only `agent` and `subagentStatusLine`, so a plugin cannot even set `ANTHROPIC_BASE_URL` +to point the session at the proxy. + +What *is* interceptable is **one tool result, at the moment it is produced, before it +enters context** — `PostToolUse` → `hookSpecificOutput.updatedToolOutput`, which +"replaces the tool's output with the provided value before it is sent to Claude". The +docs name this use case directly: *"For redaction or transformation use cases, intercept +at `PreToolUse` for outbound tool inputs and `PostToolUse` for inbound tool results."* + +So the dividing line through our component set is not lossy-vs-lossless. It is: + +- **per-tool-output text transform** → works, sometimes better than the proxy; +- **needs the request envelope, or needs to rewrite an *earlier* message** → impossible. + +That second clause is the expensive one. `PostToolUse` fires once, at message birth. It +can never go back. + +## The one picture: the transcript is the boundary + +Everything below follows from *where* each host sits relative to Claude Code's session +transcript. The plugin is **upstream** of it; the proxy is **downstream** of it. + +```mermaid +flowchart TB + subgraph harness["Claude Code process — PLUGIN territory (upstream of the transcript)"] + direction TB + TOOL["tool executes
(Bash / Read / MCP)"] + HOOK{"PostToolUse hook
updatedToolOutput"} + TR[("session transcript
messages + tool results")] + ASM["assemble request
tools + system + messages
(built here, never persisted)"] + TOOL --> HOOK --> TR --> ASM + end + + subgraph wire["the wire — PROXY territory (downstream of the transcript)"] + direction TB + APPLY{"apply.Body
pipeline + splice"} + UP["provider
+ KV cache"] + APPLY --> UP + end + + ASM ==>|"the full request body
exists ONLY here"| APPLY + UP -.->|"usage: cache_read /
cache_write / fresh"| APPLY + + MCP["MCP server
context_guru_expand"] + MCP -.->|"restored original
as a new tool result"| TR + + classDef plug fill:#e8f4ea,stroke:#4a7,stroke-width:1px + classDef prox fill:#eef0f8,stroke:#77a,stroke-width:1px + class TOOL,HOOK,TR,ASM,MCP plug + class APPLY,UP prox +``` + +Read off the consequences: + +| | Plugin (upstream) | Proxy (downstream) | +|---|---|---| +| Sees `tools` / `system` / `cache_control` | **No** — assembled after it, never persisted | Yes | +| Sees provider usage (cache tiers, cost) | **No** | Yes | +| Sees one tool output before it is recorded | **Yes** | No — only after, re-sent every turn | +| Effect on the transcript | **Permanent** — rewrites history at birth | Per-request — must re-derive each turn | +| Knows session lifecycle (`Stop`, `SessionEnd`, `PreCompact`) | **Yes, explicitly** | Inferred from traffic | + +So the split is not "which components are portable". It is: **anything that needs the +request body or the provider's response must be on the wire, and that is the definition of +the proxy. The plugin's entire boundary is "everything except the wire."** + +That single fact decides `cachesplit`, `cacheinject`, the keepalive ping, `/stats` cost +accounting, and the freeze/replay layer — in that order of importance. + +## Why this distribution method fits offloaders but not cache management + +The two families differ in *what they need to touch*, and the plugin boundary cuts exactly +between them. + +**An offloader needs one tool output.** `cmdfilter` filtering `terraform plan` noise, +`collapse` windowing a 2,000-line log, `skeleton` reducing a source read — each is a pure +function from one blob of text to a shorter blob of text. It needs no neighbouring message, +no `system` array, no provider response. `PostToolUse` hands it exactly that blob and +accepts a replacement. The unit of work and the unit of interception are the same size, so +the transport is a fit rather than a compromise. + +**Cache management needs the whole request, and needs to write metadata into it.** Not one +message — the *shape* of the entire prefix: + +- `cachesplit` restructures the top-level `system` array. Components never see it; only + `apply` does, from the raw body. +- `cacheinject` reasons over `messages[]` positionally — divergence point, turn-stable + anchors, the 4-slot budget counted across `system` + `tools` + `messages` together — and + its output is a `cache_control` key, which is request metadata, not content. +- the keepalive must *originate* a request that byte-exactly reproduces a prefix, and price + the decision from `CachedTokens`, which only the provider's response carries. + +None of that is a large tool output. It is the envelope, and the envelope is assembled after +the plugin's last hook has run and is never written to disk (verified: a session transcript +carries `messages` and `toolUseResult`, and no system prompt or tool schemas — see +`scripts/inspect_transcript.py`). + +So the rule is not "lossy vs lossless" or "cheap vs expensive". It is: + +> **Content-scoped work distributes to the harness. Envelope-scoped work requires being on +> the wire.** + +And there is a second, sharper asymmetry in how the two families *fail*. An offloader that +gets it wrong wastes an expand round-trip — bounded, recoverable, and reversibility is +type-enforced. Cache work that gets it wrong inverts: a mistimed keepalive creates an entry +at 1.25× instead of refreshing at 0.1×; a breakpoint over budget is a hard 400; a +representation flip inside a cached prefix re-writes the suffix at 11.5×. Envelope-scoped +work has no fail-open direction that is merely "no saving" — which is precisely why it wants +the host that can see the whole request and the provider's answer to it. + +## The KV-cache layer: the defensive half is free, the offensive half is impossible + +This is the part that matters most, and the answer is not one answer. Our KV-cache +awareness is two separate mechanisms with opposite fates under a plugin. + +### Defensive: keep the provider's cached prefix byte-stable + +`components/offload/state.go`, `Ctx.CacheAware` / `Ctx.MaxCachedIdx`, `modes.Tracker`, +`cg:frz:`, `FrozenLost` / `repairLostFreeze`, `frozen_flips`, the sticky-session set, +`cg:len:`. This is the single hardest body of code in the repo, and `state.go` states +exactly why it exists: + +> once an offloader compacts an output, it must send the SAME bytes for that output on +> every later turn — **otherwise the agent (which re-sends the ORIGINAL each turn)** makes +> the output flip compacted→full→compacted, churning the provider KV cache. + +The parenthetical is the whole load-bearing premise, and **a `PostToolUse` hook falsifies +it.** The agent cannot re-send the original because the agent never received the original. +`updatedToolOutput` replaces the output *before it enters Claude Code's transcript*, so +the reduced bytes are what the transcript holds and what gets re-sent, verbatim, forever. + +Every consequence follows mechanically: + +- **No re-derivation, so no freeze.** `freeze` / `reapplyFrozen` / `frozenKey` exist to + reproduce byte-identical output on turn *n+1*. There is no turn *n+1* decision to make. +- **No lost-freeze problem.** `frozenLost`, `repairLostFreeze`, `FrozenLoser`, + `frozen_dropped` / `frozen_repaired` / `frozen_flips`, and the "fail direction inverts + for an established compaction" reasoning in design.md all describe a store entry whose + loss corrupts a cached prefix. Nothing is replayed, so nothing can be lost. +- **No tail gate.** `MaxCachedIdx` / `TailOnly` / `modes.Tracker` restrict mutation to + messages the provider hasn't cached. A hook mutates content that has **never been sent + to the provider at all** — it is not merely in the uncached tail, it is pre-wire. This is + strictly stronger than the invariant the tracker enforces, and it removes the + read-then-write race that `Tracker.Turn` was built to fix. +- **`extract_llm`'s nondeterminism stops being disqualifying.** It is excluded from + `repairLostFreeze` because a sampled model output "may emit different bytes at depth". A + decision made exactly once cannot differ from itself. (Its *economics* still don't work + on caching backends — separate problem, unchanged.) +- **Sticky ids and the overcount correction go away.** `saved_tokens` vs + `saved_tokens_unique` exists because the agent re-sends history and the cumulative figure + double-counts. In plugin form each saving is realized once, at birth. +- **Agent self-compaction stops being inferred.** `proxy/agentcompaction.go` detects it by + string-matching the agent's compaction prompt — pinned to Claude Code 2.1.215's + `Jao()`/`GMu()` internals, with a documented reachable false positive (our own docs page + quoting the phrase). A plugin gets `PreCompact` / `PostCompact` as first-class events. A + fragile phrase match against a specific agent build is replaced by a signal. + +So the defensive half isn't ported — it's **obviated**. That is the strongest argument for +the plugin, and it is structural rather than a matter of care. + +### Offensive: make the provider cache more of the prefix + +`cachesplit` (the volatile-tail system split) and `cacheinject` (breakpoint placement). +Both write request metadata. Both are categorically unreachable — see the component table. + +**But this half only exists on explicit-breakpoint backends.** `apply/prefixsplit.go` is +explicit: + +> Only meaningful where breakpoints are EXPLICIT (Anthropic family). Under an implicit +> longest-prefix cache the match already ends at the divergence, so a block boundary buys +> nothing. + +That splits the verdict by backend, and it is the crux of the whole evaluation: + +| Backend | Offensive half | Plugin's net cache position | +|---|---|---| +| Anthropic / Bedrock / Vertex (explicit `cache_control`) | `cachesplit` = **−34.1% cost, 0% → 96.7% hit** | **Real loss.** The best-evidenced component in the repo is unreachable. | +| vLLM / llm-d / on-prem, OpenAI auto-cache (implicit longest-prefix) | `cachesplit` and `cacheinject` are **already no-ops** | **No loss at all** — and the defensive half, which is the *entire* cache story there, comes free. Strictly better than the proxy on the cache axis. | + +Given `extract_llm.go`'s "on-prem vLLM under KV-cache pressure" measurements and the llm-d +TOON config, the implicit case is a real target, not a hypothetical. On that traffic a +plugin is the better cache deployment. On Anthropic traffic — which is what Claude Code on +DAM will actually be — it is not. + +## Component verdicts + +### Reformat (lossless) + +| Component | Plugin | Why | +|---|---|---| +| `format` | **Yes** | Acts only on tool messages whose text is JSON — a pure per-output recompaction. Nothing envelope-level despite the name. | +| `toon` | **Yes** | Same shape: one tool output's uniform JSON array → TOON. | +| `cacheinject` | **No — categorically** | Four independent blockers, below. | +| `cachesplit` | **No — categorically** | Its whole effect is a rewrite of the top-level `system` array (`apply/prefixsplit.go`). There is no `system` array in plugin-land. | + +**`cacheinject` in detail** — the offensive half's second component, and the single worst +fit in the repo: + +1. Its output *is* request metadata. `cache_control` on a message content block. No hook + emits that. +2. Even granted a write channel, the algorithm is **positional over the whole + `messages[]` array**: `want[len(req.Input)-1]`, `commonPrefix(prev, now)` for the + divergence point, and the `lookbackBlocks`-strided turn-stable anchors. A + `PostToolUse` hook sees one tool result and no array at all. +3. `Ctx.ExistingBreakpoints` is computed by `apply` from the **raw wire body**, spanning + `system` + `tools` + `tool_result` blocks — the exact three places components can't + see, which is what issue #32 was. That body does not exist in a hook. +4. The v2 contract is *"keep every breakpoint the caller set, then spend the leftover + slots"* — where the caller is Claude Code itself. A plugin cannot observe Claude + Code's own breakpoints, so it cannot compute the budget, and overshooting the + provider's cap of 4 is a hard 400. + +`cacheinject` is in no preset today (placement is unmeasured), so losing it costs nothing +measured. **`cachesplit` is the real loss**: it is in *every* preset and it is the +best-evidenced component we have — −34.1% cost, 0% → 96.7% cache hit in an isolated A/B. +The plugin gives that up entirely. + +### Offload (lossy, reversible) + +| Component | Plugin | Why | +|---|---|---| +| `cmdfilter` | **Yes — better than the proxy** | Per-output DSL filter, and the hook input hands you the actual `tool_input.command`. The proxy has to infer the command from the transcript; a hook is told. Should cut `cmdfilter_selector_misses`. | +| `collapse` | **Yes** | Head/tail window over one oversized output. | +| `smartcrush` | **Yes** | Crushes one output's homogeneous array. | +| `extract` | **Yes** | Deterministic per-output noise collapse. | +| `skeleton` | **Yes — better than the proxy** | Per-output tree-sitter reduction, and the hook supplies `tool_input.file_path`, so language selection stops being content sniffing. Still needs the `cg_skeleton` build tag. | +| `dedup` | **Yes, with state** | Replaces the *later* byte-identical output — forward-only, which is exactly what a hook can do. Needs cross-turn digests, so it needs a store that outlives one hook process. | +| `extract_llm` | **Technically yes; leave it off** | The transform is per-output, but its economic gate needs the caching-backend fact and the fresh/cache-read token split, none of which a hook can see. Since #28 it declines on caching backends anyway (~8× underwater, break-even ~30,500 tokens/output). A synchronous cheap-model call inside a per-tool-call hook is also the worst place to put one. | +| `failed_run` | **No** | It keeps the newest run and collapses **earlier** runs. Retroactive by definition; at the moment run 2 is produced, run 1 is already in Claude Code's transcript and immutable. | +| `mask` | **No** | Age-based GC of *older* tool outputs. Same retroactivity. This is the top measured lever on our target traffic — 27.5% on Terminal-Bench, 12.5% on SWE-bench — and it is unreachable. | +| `summarize` | **No** | Whole transcript, and it changes the message count. | + +### Infrastructure + +| Package | Plugin | Notes | +|---|---|---| +| `expand/` | **Yes — strictly better** | Ship `context_guru_expand` as an MCP tool via `.mcp.json`. This deletes the proxy's worst machinery: the 3-round `maxExpandRounds` cap, the response parsing, the continuation builder, and above all the **SSE buffering** (today a streaming response must be buffered whole to detect an expand call — `sse_buffered_pct` / `sse_ttfb_ms_avg_buffered`). An MCP tool call is just a tool call. Nothing to buffer, no round cap. | +| `store/`, `session/` | **Yes, needs a home** | Hooks are one-shot processes; our store is in-memory with sliding TTL and pinned prefixes. Either a long-lived process holds it (the MCP server already is one) or SQLite under `${CLAUDE_PLUGIN_DATA}`. Session keying gets *simpler*: hook input carries `session_id`, so `session.Resolve`'s content-hash fallback is never needed. | +| `config/` | **Yes, unchanged** | One new preset. This is the "reuse via specific configuration" the plugin should be. | +| `metrics/`, `/stats` | **Partial — and this bites** | Content-token savings are countable. The four provider-billed tiers (`fresh_input_tokens`, `cache_read_tokens`, `cache_write_tokens`, `output_tokens`) come from the API *response*, which a plugin never sees. So `savings_pct_attempted`, `savings_pct_new_input`, every cost figure, and the `deploy/harbor/*.py` reproduction path do not exist in plugin mode. Worse, Claude Code's own OTel spans capture the **original** output — the docs are explicit that telemetry runs before the hook. **The plugin cannot be benchmarked the way the proxy is.** | +| `dash/` | **Yes** | Runs alongside; `monitors/monitors.json` can launch it, a skill can open it. Content capture stays opt-in. | +| `modes.Tracker`, `MaxCachedIdx`, `TailOnly`, freeze/repair | **Moot — and that is the upside** | See below. | +| `apply/` | **No — replaced by something much smaller** | No wire body, so `normalize`, the round-trip guard, the `metawrite` metadata exception, `wireBreakpoints` and `prefixsplit` all have nothing to do. A new `adapters/cchook` needs none of it. | +| `proxy/` control plane — `spendgate`, `tenancy`, `limits`, `refusals`, `promexport` | **No** | Multi-tenant gateway concerns. A plugin is single-user and local. | +| `adapters/bifrost` | N/A | | + +## Permanence cuts both ways + +The defensive-half argument above is the plugin's best property, but it has a symmetric +downside that has to be priced in. + +Under the proxy, an offload decision is re-made every turn and is therefore *revisable*. +Under a hook it is made once and is **irrevocable from the transcript's point of view**: + +- A good reduction is permanently good — realized once, re-sent free forever. +- A **premature offload is permanently bad.** Today a bounce (`wasted_tokens` / `bounces`) + costs one round-trip and the next turn can decide differently. In plugin form, when the + model calls `context_guru_expand`, the restored original arrives as a *new* MCP tool + result appended to the transcript — the marker stays where it was, and now both the marker + and the full original are re-sent for the rest of the session. An expand goes from a + per-turn cost to a permanent one, *plus* the marker overhead. + +Note what this does to `state.go`'s kept-verbatim machinery: `keptKey` / `isKeptVerbatim` / +`MarkKeptVerbatim` exist to stop an expand→re-compact→expand loop. That loop is impossible +here (the hook never revisits the message), so the guard is unnecessary — but the *reason* +it was needed becomes a worse problem rather than a solved one. + +The design consequence is concrete: **the plugin should be tuned more conservatively than +the proxy**, favouring high-confidence deterministic reductions (`cmdfilter`, `format`, +`toon`, `extract`) over speculative ones (`collapse`'s blind head/tail window). It also +makes `mask`'s `keep_head_chars` head-peek idea — which measurably stops needless expands — +more valuable exactly where it cannot run. + +One piece of `state.go` gets *better*, not obviated: the **global (session-independent) +result cache**. `getResultGlobal` / `extract.ResultKey` exploit cross-session content +recurrence — measured 82 of 103 unique contents recurring across sessions, ~93% of +`extract_llm`'s realized value. The proxy holds that in an in-memory store that dies on +restart. A plugin store under `${CLAUDE_PLUGIN_DATA}` is a per-user file that survives +session exit and reboot, so it can realize *more* of that reuse than the proxy does. +`markSeenContent`'s recurrence signal keeps working for the same reason. + +## What the plugin would actually contain + +``` +context-guru-plugin/ + .claude-plugin/plugin.json # userConfig: preset, ttl, dashboard, log level + .mcp.json # context-guru --mcp → context_guru_expand (+ holds the Store) + hooks/hooks.json # PostToolUse (matcher ".*") → context-guru --hook + monitors/monitors.json # optional: dashboard / stats tail + skills/ # /cg:stats, /cg:preset, /cg:explain + bin/ # the binary, if not vendored via userConfig path +``` + +New Go code, all of it thin: + +- `adapters/cchook` — read the hook JSON on stdin, wrap the tool output as a single + synthetic `role:"tool"` `ChatMessage`, run the configured pipeline, emit + `updatedToolOutput`. Fail open on everything (echo nothing → original stands). +- an MCP stdio mode on the existing binary, serving `expand/`'s tool def and owning the + store the hook talks to. +- one preset, e.g. `hook: format → toon → dedup → cmdfilter → extract → smartcrush → collapse` + — `general` minus `mask`, `failed_run`, `cachesplit`, `extract_llm`, `summarize`. + +Everything under `components/`, `components/offload/`, `components/dsl/`, `store/`, +`session/`, `config/`, `expand/`, `metrics/`, `dash/` is reused unmodified. + +## Three things to verify empirically before committing + +**0. Does `updatedToolOutput` persist into the session transcript?** This is the gate +everything above rests on, and the docs do not state it directly. The whole +defensive-half-is-free argument requires that the *replacement* is what Claude Code stores +and re-sends on every later turn. Two data points point that way — the docs say the value +is substituted "before it is sent to Claude", and for `additionalContext` they say Claude +Code "saves the injected text in the session transcript … replays the saved text rather +than re-running the hook" on resume — but neither is a statement about +`updatedToolOutput`'s persistence. The counter-hypothesis is that the original is retained +and the replacement applies to one request only. + +If persistence does **not** hold, the plugin saves tokens on exactly one request per tool +call and nothing thereafter, the entire freeze/replay layer is needed again (with no way to +run it, since the hook can't see the request), and the whole proposition collapses. **Test +this first, before anything else is built:** run a `PostToolUse` hook that replaces a Bash +output with a sentinel string, take two more turns, then inspect the session transcript +JSONL and — decisively — a captured outbound request body to confirm the sentinel is what +gets re-sent. + +1. **Is `updatedToolOutput` subject to the 10,000-character cap?** The docs cap + "`additionalContext`, `systemMessage`, and plain stdout" and do not name + `updatedToolOutput` — but if the cap applies to total hook stdout, the hook is unusable + for exactly the large outputs that matter. Test with a >10 KB reduced output. +2. **Shape validation fails silently.** "For built-in tools, a value that doesn't match the + tool's output schema is ignored and the original output is used." That is fail-open, + which suits us, but it is *invisible* — the same class of bug as #32, where a component + applied 46 breakpoints and 0 reached the wire. The hook must verify its own emission and + count rejections, or we will ship a silently inert plugin again. + +Also note `PostToolUse` does not fire for tool calls rejected before execution, and tool +*errors* route to `PostToolUseFailure` instead — a nonzero-exit `Bash` call is still a +success, so the `cmdfilter` path is covered, but a plugin wanting full coverage needs both +events. + +## Does this help the DAM push? + +[DAM](https://github.com/dam-agents/dam) is TypeScript, explicitly harness-plural (Claude +Code, Pi Agent, Bob, Codex, plus "bring your own harness — any runtime compatible with +ACP"), and runs agents in isolated pods behind **policy-enforced gateways** with zero-trust +credentials the runtime never sees. + +**Mostly no.** + +- A Claude Code plugin covers **one of four named harnesses** and none of the + bring-your-own case. The proxy covers all of them because it sits at the transport. +- DAM's egress shape *is already* our gateway credential model — provider key injected at + the gateway, agent holds a placeholder. `proxy/` drops into that path introducing no new + concept; the plugin would have to be installed and force-enabled per harness, per pod. +- Long-lived agents are DAM's headline, which makes **cache economics** the dominant cost + term. Claude Code on DAM means Anthropic/Bedrock — the explicit-breakpoint column, where + the offensive half is worth −34.1% and is unreachable from a plugin. (Were DAM routing to + an on-prem vLLM/llm-d backend, this row would flip: there the offensive half is already a + no-op and the plugin's free defensive half would be the better cache deployment. Worth + confirming which backends DAM actually targets before treating this as settled.) +- `spendgate`, `tenancy`, `limits`, `promexport` are what a platform actually needs to + enforce, and they are proxy-only. A plugin is user-installable and user-disableable; a + platform wants a policy. +- No `/stats`, so no benchmark reproduction on DAM traffic. + +**The narrow yes.** DAM already ships `.agents/skills`, so it has a skills-shaped +distribution channel. Three pieces are worth shipping there *on top of* the gateway: + +1. the **MCP expand server** — valuable regardless of transport, because it removes SSE + buffering from the gateway's hot path; +2. the **operator surface** — skills for `/cg:stats`, the dashboard monitor; +3. the **`adapters/cchook` boundary itself** — a per-tool-output adapter is reusable by any + ACP harness exposing a tool-result hook, so the refactor has cross-harness value even + where the packaging does not. + +## What the plugin is actually for + +Stated bluntly, because it is easy to lose in the detail: **the keepalive idle signal is not +a reason to build a plugin.** It is a rider on one, worth having only if a plugin exists for +other reasons. If the list below doesn't stand up, don't build it. + +Ranked by what the plugin uniquely provides — things the proxy *cannot* do, not things it +already does: + +1. **Offload at birth, permanently** *(the whole case — and it is conditional)*. Rewriting a + tool output before it enters the transcript means the reduction is realized once and + re-sent free forever, and the entire defensive KV-cache layer — `state.go`'s + freeze/replay, `MaxCachedIdx`, `Tracker`, `frozen_flips`, sticky ids — stops being + necessary rather than being ported. That is the only item here that changes the + architecture. **It rests entirely on the unverified persistence check (§0).** +2. **`context_guru_expand` as an MCP tool.** Deletes the 3-round `maxExpandRounds` cap, the + response parsing, and the whole-SSE-buffering penalty. Worth doing *even with no hook at + all*, and it works alongside the proxy rather than instead of it. +3. **Better selectors for free.** `cmdfilter` gets the actual `tool_input.command` and + `skeleton` gets `tool_input.file_path`, instead of inferring both from transcript text. +4. **Explicit lifecycle signals.** `PreCompact`/`PostCompact` replace + `proxy/agentcompaction.go`'s string match against Claude Code 2.1.215's internals — which + has a documented reachable false positive. `SessionEnd` tells a proxy-side keepalive that + a session is dead rather than thinking, which is the waste that caps `K` at 2–3. +5. **Durable cross-session result cache.** `${CLAUDE_PLUGIN_DATA}` survives restarts; the + proxy's in-memory store does not. On the measured 82-of-103 cross-session content + recurrence, that realizes more reuse than the proxy can. +6. **Distribution.** `/plugin install` and a marketplace, versus "point your agent at a + proxy". Real, but packaging, not capability. + +Items 3–6 are worth a few hundred lines. They do not justify the project on their own. **The +project is item 1, or it is item 2 alone.** Decide it with the §0 experiment before writing +adapter code. + +## Recommendation + +Build it, scoped as a convenience/transport layer, not as a strategy: + +- **Do**: MCP expand server, `PostToolUse` offload hook, operator skills, one `hook` preset. +- **Don't**: attempt `cacheinject`, `cachesplit`, `mask`, `failed_run`, or `/stats` cost + accounting in plugin form. Cache work and multi-tenant policy stay in the proxy, which + remains the only place they are possible. +- **For DAM**: land the proxy in the gateway. Ship the plugin as the Claude-Code-session + layer on top of it — never as the DAM integration. +- **Gate the work** on check 0 — transcript persistence — before writing any adapter code. + It is a half-day experiment and it decides whether the plugin is worth building at all. If + it holds, the defensive KV-cache half comes free and a large amount of our hardest code is + simply not needed in that deployment. If it doesn't, stop: build the expand MCP server + alone and leave the pipeline in the proxy. Check 1 (the 10 KB cap) is the second gate; if + it bites, same outcome. diff --git a/scripts/inspect_transcript.py b/scripts/inspect_transcript.py new file mode 100644 index 00000000..cae55b4f --- /dev/null +++ b/scripts/inspect_transcript.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Structural probe: does a Claude Code transcript carry the request prefix? + +Answers one question for the keepalive analysis — can a plugin reconstruct the +byte-exact `tools` + `system` + `messages` a cache-read ping must reproduce? +Prints only structure (keys, record types, counts), never content. +""" +import json +import sys +from collections import Counter + +path = sys.argv[1] +keys = set() +types = Counter() +roles = Counter() +sys_hits = set() +tool_hits = set() + +for line in open(path): + line = line.strip() + if not line: + continue + try: + d = json.loads(line) + except Exception: + continue + if not isinstance(d, dict): + continue + keys |= set(d.keys()) + types[d.get("type")] += 1 + msg = d.get("message") + if isinstance(msg, dict): + roles[msg.get("role")] += 1 + for k in msg: + if "system" in k.lower(): + sys_hits.add("message." + k) + if "tool" in k.lower(): + tool_hits.add("message." + k) + for k in d: + if "system" in k.lower(): + sys_hits.add(k) + if "tool" in k.lower(): + tool_hits.add(k) + +print("record types: ", dict(types)) +print("message roles: ", dict(roles)) +print("top-level keys: ", sorted(keys)) +print("system-ish keys: ", sorted(sys_hits) or "NONE") +print("tool-schema-ish: ", sorted(tool_hits) or "NONE") From 1abb0260a95124faf2cfd765b91f53f4ea4f3ba7 Mon Sep 17 00:00:00 2001 From: DAVID AMID Date: Sun, 30 Aug 2026 22:14:01 +0300 Subject: [PATCH 2/3] =?UTF-8?q?docs(analysis):=20address=20review=20?= =?UTF-8?q?=E2=80=94=20fix=20CI,=20close=20Gate=200,=20correct=20the=20cap?= =?UTF-8?q?=20figure?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review found one broken thing, settled the doc's central open question, and caught a wrong number. All four points addressed. CI was red: mkdocs build --strict rejected docs/analysis/claude-code-plugin.md because it was in no nav entry, and there is no docs/analysis convention to copy. Moved it to docs/superpowers/specs/2026-08-30-claude-code-plugin-transport-design.md, which exclude_docs already covers and whose stated purpose — "working plans/specs ... not published site content" — is what this doc is. That also matches the dated-spec naming of the keepalive design doc and of the local-distribution proposal in #130. The reviewer offered the alternative of a nav entry under Results:; exclusion is the better fit, because publishing a page that evaluates a plugin we have not built would read on the docs site as a product that exists. Verified with a real strict build (1.45s, no warnings) and a negative control: the same file under docs/analysis/ still aborts strict mode, so the move is the fix rather than something incidental. Gate 0 is closed, in the doc's favour. Review ran the experiment the doc asked for — a PostToolUse hook replacing a Bash output with a sentinel, captured through a raw-logging reverse proxy so the evidence is the literal outbound body rather than the transcript file — and the replacement persists and is resent verbatim on later turns. A working collapse plugin then measured -6,285 tokens on a real session, appearing as the same reduction on turn 1's cache-write and turn 2's cache-read, which is what separates a permanent reduction of resent context from a one-turn display trick. The "three things to verify" section becomes a resolved-gate section plus the risks that actually remain, and the recommendation stops being conditional. The predicted failure mode arrived on the first attempt, which is worth recording rather than smoothing over: updatedToolOutput must be the object tool_response shape ({stdout, stderr, interrupted, isImage, noOutputExpected} for Bash), and a bare string is silently ignored. That is now a hard requirement on adapters/cchook — emit the object shape, count your own rejections — instead of a general warning. The 10,000-character cap figure was wrong, and "cap" was the wrong frame. That number came from the additionalContext / systemMessage / plain-stdout cap, which does not govern this field. Measured: verbatim and uncapped to ~30,000 chars, real threshold in (30,000, 40,000] and most likely 32,768, and above it neither truncation nor rejection — the CLI's ordinary large-output handler produces a ~2,260-char wrapper with a 2KB preview and a disk pointer while the local record stays intact. For an oversized hook emission that is a token-cost improvement, not a hazard. Two precision fixes. The envelope claim now records that it was verified by a stronger method than the docs (11 events exercised live, plus the installed CLI's own hookSpecificOutput validation schema: 33 events, 22 with output fields, none touching the envelope) and carries the nuance that promptCacheTtl / CLAUDE_CODE_PROMPT_CACHE_TTL is a reachable cache-TTL lever via settings or env — not via any hook or manifest field, and session-wide rather than per-breakpoint — so a flat "zero cache control" reading would overstate the gap. And the component figures quoted throughout (cachesplit's -34.1%, mask's 27.5%/12.5%, the ~7,017-token system block) are now labelled as this repo's frozen historical measurements, quoted accurately but not re-verified against current traffic by this evaluation. Finally, the recommendation leads with the scope the reviewer articulated better than the doc did: a plugin can replicate the offloader half of this repo, persistently and measurably, and categorically cannot replicate the cache-management half — not "a plugin can do what the proxy does." Docs only — no code, no behavior change. Signed-off-by: DAVID AMID Assisted-By: Claude Opus 5 (1M context) --- ...30-claude-code-plugin-transport-design.md} | 136 ++++++++++++------ 1 file changed, 95 insertions(+), 41 deletions(-) rename docs/{analysis/claude-code-plugin.md => superpowers/specs/2026-08-30-claude-code-plugin-transport-design.md} (80%) diff --git a/docs/analysis/claude-code-plugin.md b/docs/superpowers/specs/2026-08-30-claude-code-plugin-transport-design.md similarity index 80% rename from docs/analysis/claude-code-plugin.md rename to docs/superpowers/specs/2026-08-30-claude-code-plugin-transport-design.md index a084cce1..220c669b 100644 --- a/docs/analysis/claude-code-plugin.md +++ b/docs/superpowers/specs/2026-08-30-claude-code-plugin-transport-design.md @@ -17,12 +17,38 @@ prompt; it only injects `additionalContext` alongside it". Plugin `settings.json only `agent` and `subagentStatusLine`, so a plugin cannot even set `ANTHROPIC_BASE_URL` to point the session at the proxy. +Verified twice over during review of this PR, and by a stronger method than the docs: 11 hook +events exercised live (including `PostToolBatch`, `PreModelSwitch`/`PostModelSwitch` and +`InstructionsLoaded`, which this doc had not covered), plus a direct read of the installed +CLI's own `hookSpecificOutput` validation schema — 33 events, only 22 carrying output fields +at all, and the validation union touches no envelope field. `cache_control` **is** present in +the captured wire body, placed by the CLI on system blocks and the trailing message; it is +simply never visible to, or writable by, any hook. + +**One precision, so this is not read as more absolute than it is.** "No plugin surface can set +`cache_control`" is true, but cache *TTL* is not entirely out of reach: `promptCacheTtl` / +`CLAUDE_CODE_PROMPT_CACHE_TTL` is a real lever (5m default on an API key, 1h on a +subscription) — reachable through `settings.json` or the environment, **not** through any hook +or plugin manifest field, and session-wide and static rather than per-breakpoint. It changes +nothing here, because `cachesplit` and `cacheinject` both turn on per-breakpoint, +per-turn reasoning that remains unreachable from plugin-land. It is worth stating because a +settings-based install — the shape proposed for local distribution — *can* set it, and a flat +claim of "zero cache control" would overstate the gap. + What *is* interceptable is **one tool result, at the moment it is produced, before it enters context** — `PostToolUse` → `hookSpecificOutput.updatedToolOutput`, which "replaces the tool's output with the provided value before it is sent to Claude". The docs name this use case directly: *"For redaction or transformation use cases, intercept at `PreToolUse` for outbound tool inputs and `PostToolUse` for inbound tool results."* +**On the figures quoted throughout.** Component measurements cited below (`cachesplit`'s +−34.1% / 0%→96.7%, `mask`'s 27.5% and 12.5%, the ~7,017-token system block in +`prefixsplit.go`) are this repo's own recorded results, read from code comments and +`docs/RESULTS.md`. They are quoted accurately but they are **frozen historical measurements, +not re-verified against current traffic** by this evaluation. The comparisons here turn on +their relative order of magnitude, which is robust to drift; anyone planning work on the +strength of a specific number should re-measure it. + So the dividing line through our component set is not lossy-vs-lossless. It is: - **per-tool-output text transform** → works, sometimes better than the proxy; @@ -320,40 +346,66 @@ New Go code, all of it thin: Everything under `components/`, `components/offload/`, `components/dsl/`, `store/`, `session/`, `config/`, `expand/`, `metrics/`, `dash/` is reused unmodified. -## Three things to verify empirically before committing - -**0. Does `updatedToolOutput` persist into the session transcript?** This is the gate -everything above rests on, and the docs do not state it directly. The whole -defensive-half-is-free argument requires that the *replacement* is what Claude Code stores -and re-sends on every later turn. Two data points point that way — the docs say the value -is substituted "before it is sent to Claude", and for `additionalContext` they say Claude -Code "saves the injected text in the session transcript … replays the saved text rather -than re-running the hook" on resume — but neither is a statement about -`updatedToolOutput`'s persistence. The counter-hypothesis is that the original is retained -and the replacement applies to one request only. - -If persistence does **not** hold, the plugin saves tokens on exactly one request per tool -call and nothing thereafter, the entire freeze/replay layer is needed again (with no way to -run it, since the hook can't see the request), and the whole proposition collapses. **Test -this first, before anything else is built:** run a `PostToolUse` hook that replaces a Bash -output with a sentinel string, take two more turns, then inspect the session transcript -JSONL and — decisively — a captured outbound request body to confirm the sentinel is what -gets re-sent. - -1. **Is `updatedToolOutput` subject to the 10,000-character cap?** The docs cap - "`additionalContext`, `systemMessage`, and plain stdout" and do not name - `updatedToolOutput` — but if the cap applies to total hook stdout, the hook is unusable - for exactly the large outputs that matter. Test with a >10 KB reduced output. -2. **Shape validation fails silently.** "For built-in tools, a value that doesn't match the - tool's output schema is ignored and the original output is used." That is fail-open, - which suits us, but it is *invisible* — the same class of bug as #32, where a component - applied 46 breakpoints and 0 reached the wire. The hook must verify its own emission and - count rejections, or we will ship a silently inert plugin again. - -Also note `PostToolUse` does not fire for tool calls rejected before execution, and tool -*errors* route to `PostToolUseFailure` instead — a nonzero-exit `Bash` call is still a -success, so the `cmdfilter` path is covered, but a plugin wanting full coverage needs both -events. +## Gate 0: persistence — RESOLVED, in this proposal's favour + +Everything above rested on one unverified fact: that the *replacement* is what Claude Code +stores and re-sends, not just what it shows the model once. The counter-hypothesis — original +retained, replacement applied to a single request — would have collapsed the whole +proposition, because the freeze/replay layer would be needed again with no way to run it. + +**Measured, 2026-08-30 (review of this PR).** A `PostToolUse` hook replaced a Bash output +with a sentinel; the outbound body was captured through a raw-logging reverse proxy — the +wire, not just the transcript file — across two turns of a real session. Turn 2, +`messages[3]`: + +```json +{"tool_use_id": "toolu_bdrk_01SC6Vg1gZAtY1WEVe7xsbsR", "type": "tool_result", + "content": "SENTINEL-OBJ-999-REPLACED", "is_error": false} +``` + +The real content appears in **no** resent `tool_result`, and a later turn asked for the exact +tool output answered with the sentinel at `input_tokens: 69` — i.e. from resent history, not +regeneration. **The replacement persists and is resent verbatim.** + +A working `PostToolUse` collapse plugin (matcher `Bash`, head/tail with an omitted-count +marker) was then A/B measured on a real session: **−6,285 tokens**, appearing as the *same* +reduction on turn 1's cache-write and turn 2's cache-read. Two independent measurements of +one session, which is what distinguishes a permanent reduction of resent context from a +one-turn display trick. + +So the defensive-half-is-free argument holds, and the recommendation below is an +empirically-grounded bet rather than a reasoned one. + +### The operational gotcha, hit exactly as predicted + +`updatedToolOutput` **must be the object shape** matching the tool's `tool_response` schema — +for `Bash`: `{stdout, stderr, interrupted, isImage, noOutputExpected}`. A **bare string is +silently ignored** and the original output stands. The first implementation attempt during +review did exactly this and read as a negative result until the shape was corrected. + +This is the failure mode flagged below as the thing to guard against, and it arrived on the +first try. The adapter must verify its own emission and count rejections; a silently inert +plugin is the #32 bug in a new costume. + +## Remaining risks + +1. **Shape validation fails silently** — see above. Now demonstrated, not hypothetical. + `adapters/cchook` needs an emission-verified counter from day one. +2. **Oversized emissions degrade rather than fail — and the earlier "10,000-char cap" figure + in this doc was wrong.** Measured by bisection with distinct-letter payload segments: + `updatedToolOutput.stdout` goes out verbatim and uncapped up to ~30,000 chars, with the + real threshold in **(30,000, 40,000]** — most likely 32,768, i.e. 3–4× the figure this doc + first cited from the `additionalContext` / `systemMessage` / plain-stdout cap, which does + not govern this field. Above the threshold there is neither truncation nor rejection: the + CLI's ordinary large-tool-output handler takes over, so the wire `tool_result` becomes a + ~2,260-char `` wrapper carrying a 2 KB preview plus a pointer to the full + content on disk, while the local `toolUseResult` record stays intact (confirmed at 100,000 + chars). "Cap" was the wrong frame — this is graceful degradation into the same + preview-plus-pointer pattern Claude Code already applies everywhere, and for an oversized + hook emission it is a token-cost *improvement*, not a hazard to design around. +3. **Event coverage.** `PostToolUse` does not fire for tool calls rejected before execution, + and tool *errors* route to `PostToolUseFailure` — a nonzero-exit `Bash` call is still a + success, so the `cmdfilter` path is covered, but full coverage needs both events. ## Does this help the DAM push? @@ -426,7 +478,11 @@ adapter code. ## Recommendation -Build it, scoped as a convenience/transport layer, not as a strategy: +The one-line version, and the honest scope of the claim: **a Claude Code plugin can replicate +the offloader half of this repo — persistently, measurably — and categorically cannot +replicate the cache-management half.** Not "a plugin can do what the proxy does." + +Build it, scoped as a transport for that half: - **Do**: MCP expand server, `PostToolUse` offload hook, operator skills, one `hook` preset. - **Don't**: attempt `cacheinject`, `cachesplit`, `mask`, `failed_run`, or `/stats` cost @@ -434,9 +490,7 @@ Build it, scoped as a convenience/transport layer, not as a strategy: remains the only place they are possible. - **For DAM**: land the proxy in the gateway. Ship the plugin as the Claude-Code-session layer on top of it — never as the DAM integration. -- **Gate the work** on check 0 — transcript persistence — before writing any adapter code. - It is a half-day experiment and it decides whether the plugin is worth building at all. If - it holds, the defensive KV-cache half comes free and a large amount of our hardest code is - simply not needed in that deployment. If it doesn't, stop: build the expand MCP server - alone and leave the pipeline in the proxy. Check 1 (the 10 KB cap) is the second gate; if - it bites, same outcome. +- **Gate 0 is now closed** (persistence confirmed on the wire, plus a −6,285-token A/B on a + real session), so the build is no longer conditional. Carry forward one hard requirement + instead: the adapter emits the **object** `tool_response` shape and counts its own + rejections, because a bare string is accepted silently and does nothing. From cdab623417974b224acc127ae96bc4d7f39a7fa9 Mon Sep 17 00:00:00 2001 From: DAVID AMID Date: Tue, 1 Sep 2026 12:50:04 +0300 Subject: [PATCH 3/3] docs(analysis): separate the two plugin candidates, price cachesplit by regime, and add rtk Addresses the three blocking items from review, plus the cheap ones. 1. Two candidates, and only one is a new transport. The document analysed plugin-as-interceptor (PostToolUse -> updatedToolOutput) without ever weighing plugin-as-installer-for-the-proxy (a skill writing env.ANTHROPIC_BASE_URL into the user's own settings.json), which is what #130 proposes and which gives up none of the component set. New section states the distinction, the component tables are scoped explicitly to (A), and a header note stops this page being read as the justification for #130. The headline blocker is rescoped: a plugin *manifest's* settings.json accepts only agent/subagentStatusLine -- that is not the same as "a plugin cannot set ANTHROPIC_BASE_URL", and the difference is the option that won. Title said "third transport" against the body's "fourth"; now fourth throughout. 2. cachesplit is priced by regime. -34.1%/0%->96.7% is a warm-regime figure -- one Terminal-Bench task x 3 trials (cacheinject.md:203, and :209 says treat it as one task, not a fleet average) with the A/B run back-to-back inside the TTL (dashboard.md:219). Cold interactive traffic is $0.0298 across 1,127 sessions / 11,361 requests (dashboard.md:204). Both are right; the document transferred the warm number onto a definitionally cold-regime user, called cachesplit "the best-evidenced component we have" where cacheinject.md:209 says otherwise, and claimed the figures were "not re-verified against current traffic" -- which for cachesplit is false. Corrected, the Anthropic row reads close to the vLLM row. The DAM recommendation stands but is now justified on spendgate/tenancy/limits and harness-plurality rather than on that figure. 3. Gate 0 is closed consistently. The ranked go/no-go section still called the case conditional and told the reader to run the experiment that already ran. Also: rtk is this architecture and we benchmarked it as a full arm -- -9.0% billed cost, reward-neutral, zero request-path latency (results/rtk.md:11) -- so expected value has a floor instead of resting on one session's -6,285 tokens, and the doc now claims the edge it was missing: rtk is a shell hook, so Read/Grep/Glob bypass it, while matcher ".*" does not. mask quoted as ~27.5-29.5%, single-task replay, never enforced in a benchmark arm. extract_llm 8x -> 82x underwater. Hook events ~34 -> 33. Dropped the guessed 32,768 threshold, keeping the measured (30,000, 40,000] bracket. Noted that inspect_transcript.py reports key names, not record types, so its `system` record type is a transcript event and not a request system array. Keepalive branch reference replaced with #126; extract_llm_sweep (#118) added to the offload table. inspect_transcript.py moved to deploy/harbor/, this repo's convention for analysis Python. Rebased onto main (3ebc65d). Docs-only; no code and no behaviour change. mkdocs is unaffected because docs/superpowers/ is excluded from the site build (mkdocs.yml:88-90); the three in-page anchors were validated against the file's own heading slugs and all other links are external. Assisted-By: Claude Opus 5 (1M context) Signed-off-by: DAVID AMID --- .../harbor}/inspect_transcript.py | 0 ...-30-claude-code-plugin-transport-design.md | 198 ++++++++++++++---- 2 files changed, 161 insertions(+), 37 deletions(-) rename {scripts => deploy/harbor}/inspect_transcript.py (100%) diff --git a/scripts/inspect_transcript.py b/deploy/harbor/inspect_transcript.py similarity index 100% rename from scripts/inspect_transcript.py rename to deploy/harbor/inspect_transcript.py diff --git a/docs/superpowers/specs/2026-08-30-claude-code-plugin-transport-design.md b/docs/superpowers/specs/2026-08-30-claude-code-plugin-transport-design.md index 220c669b..03a7d5b0 100644 --- a/docs/superpowers/specs/2026-08-30-claude-code-plugin-transport-design.md +++ b/docs/superpowers/specs/2026-08-30-claude-code-plugin-transport-design.md @@ -1,4 +1,13 @@ -# Claude Code plugin as a third transport for context-guru +# Claude Code plugin as a fourth transport for context-guru + +> **What this document is, and is not.** It evaluates **plugin-as-interceptor** — a plugin +> that rewrites tool output in-process via `PostToolUse`, and is therefore a genuinely new +> transport. That design is **unbuilt**, and this evaluation recommends building it. +> [#130](https://github.com/rossoctl/context-guru/pull/130) proposes a **different** design +> that reuses the same plugin surface — plugin-as-installer-for-the-proxy — and gives up none +> of the component set. The two are not variants of one plan; see +> [§Two candidates](#two-candidates-and-only-one-of-them-is-a-new-transport). This document is +> not the justification for #130. Scope: can the existing `components` core be deployed as a [Claude Code plugin](https://code.claude.com/docs/en/plugins) — as a *thin* fourth host next to @@ -11,11 +20,17 @@ Not a replacement for the proxy. Deployment/transport only. **No plugin surface can read or write the outbound Messages API request.** Confirmed against the [hooks reference](https://code.claude.com/docs/en/hooks#decision-control): -across all ~34 hook events there is no field that rewrites `messages[]`, `system[]`, +across all 33 hook events there is no field that rewrites `messages[]`, `system[]`, `tools[]`, or any `cache_control`. `UserPromptSubmit` explicitly "can't replace the -prompt; it only injects `additionalContext` alongside it". Plugin `settings.json` accepts -only `agent` and `subagentStatusLine`, so a plugin cannot even set `ANTHROPIC_BASE_URL` -to point the session at the proxy. +prompt; it only injects `additionalContext` alongside it". + +A plugin **manifest's** `settings.json` accepts only `agent` and `subagentStatusLine`, so a +plugin cannot point the session at the proxy *declaratively, from its own manifest*. Read as +"a plugin cannot set `ANTHROPIC_BASE_URL`" that would be wrong, and the distinction turns out +to matter more than anything else here: a plugin **can** write the key into the user's real +`settings.json` from an install skill, which is exactly the design #130 proposes and this +document does not evaluate as a transport candidate. See +[§Two candidates](#two-candidates-and-only-one-of-them-is-a-new-transport). Verified twice over during review of this PR, and by a stronger method than the docs: 11 hook events exercised live (including `PostToolBatch`, `PreModelSwitch`/`PostModelSwitch` and @@ -41,13 +56,31 @@ enters context** — `PostToolUse` → `hookSpecificOutput.updatedToolOutput`, w docs name this use case directly: *"For redaction or transformation use cases, intercept at `PreToolUse` for outbound tool inputs and `PostToolUse` for inbound tool results."* -**On the figures quoted throughout.** Component measurements cited below (`cachesplit`'s -−34.1% / 0%→96.7%, `mask`'s 27.5% and 12.5%, the ~7,017-token system block in -`prefixsplit.go`) are this repo's own recorded results, read from code comments and -`docs/RESULTS.md`. They are quoted accurately but they are **frozen historical measurements, -not re-verified against current traffic** by this evaluation. The comparisons here turn on -their relative order of magnitude, which is robust to drift; anyone planning work on the -strength of a specific number should re-measure it. +**On the figures quoted throughout — and one correction to how this document used them.** +Component measurements cited below are this repo's own recorded results, quoted accurately from +code comments and `docs/RESULTS.md`. An earlier revision of this document called them "not +re-verified against current traffic"; for `cachesplit` that is **false**, and the correction +matters to a decision made at the end of this document. + +`cachesplit`'s **−34.1% / 0%→96.7%** is a *warm-regime* figure: `docs/components/cacheinject.md:203` +records one Terminal-Bench task × 3 trials at Sonnet 5 rates, and `:209` says in terms *"treat +34.1% as one task measured three times, not a fleet average"*. `docs/dashboard.md:219` states the +A/B *"ran tasks back-to-back inside the TTL"*, which is the only regime where a volatile-tail split +can pay. Against that, `docs/dashboard.md:204` records the *cold interactive* figure on this +deployment: **$0.0298 across 1,127 sessions / 11,361 requests.** Both numbers are right. They +differ by three orders of magnitude because they measure different regimes. + +Three facts bound the cold case further, all from `docs/dashboard.md`: the environment block is +snapshotted **once per session** (a nine-turn session that created and committed four files +produced *one* volatile-tail hash), **1,105 of 1,127** first requests read **zero** tokens from +cache because the previous prefix had expired under the 5-minute TTL, and only 9 session starts had +a warm prefix at all. `cachesplit` also does nothing outside a git repo, and is a no-op on implicit +prefix-cache backends (`config/config.go:379-380`). + +So the rule for reading every figure below: **name the regime.** A plugin user is a human running +interactive sessions minutes or hours apart — definitionally the cold regime — not a benchmark +harness running tasks back-to-back. Asking the reader to trust "order of magnitude" is asking them +to trust the one property that is regime-dependent. So the dividing line through our component set is not lossy-vs-lossless. It is: @@ -133,9 +166,12 @@ message — the *shape* of the entire prefix: the decision from `CachedTokens`, which only the provider's response carries. None of that is a large tool output. It is the envelope, and the envelope is assembled after -the plugin's last hook has run and is never written to disk (verified: a session transcript -carries `messages` and `toolUseResult`, and no system prompt or tool schemas — see -`scripts/inspect_transcript.py`). +the plugin's last hook has run and is never written to disk. Verified: a session transcript carries +`messages` and `toolUseResult`, and no system prompt or tool schemas — see +`deploy/harbor/inspect_transcript.py`. Read its output carefully: it inspects transcript **key +names**, so `system-ish keys: NONE` is the finding, while the `record types` tally it also prints +does contain a `system` entry — a transcript *event* type, not a request `system` array. The wire +capture in §0 is what establishes the point conclusively. So the rule is not "lossy vs lossless" or "cheap vs expensive". It is: @@ -216,7 +252,7 @@ That splits the verdict by backend, and it is the crux of the whole evaluation: | Backend | Offensive half | Plugin's net cache position | |---|---|---| -| Anthropic / Bedrock / Vertex (explicit `cache_control`) | `cachesplit` = **−34.1% cost, 0% → 96.7% hit** | **Real loss.** The best-evidenced component in the repo is unreachable. | +| Anthropic / Bedrock / Vertex (explicit `cache_control`) | `cachesplit` is unreachable — worth **−34.1%** warm (one task, back-to-back inside the TTL) but **$0.0298 / 1,127 sessions** cold | **Loss of cents, in the regime a plugin user is actually in.** Real on a benchmark harness; close to the vLLM row for an interactive human. | | vLLM / llm-d / on-prem, OpenAI auto-cache (implicit longest-prefix) | `cachesplit` and `cacheinject` are **already no-ops** | **No loss at all** — and the defensive half, which is the *entire* cache story there, comes free. Strictly better than the proxy on the cache axis. | Given `extract_llm.go`'s "on-prem vLLM under KV-cache pressure" measurements and the llm-d @@ -224,6 +260,50 @@ TOON config, the implicit case is a real target, not a hypothetical. On that tra plugin is the better cache deployment. On Anthropic traffic — which is what Claude Code on DAM will actually be — it is not. +## Two candidates, and only one of them is a new transport + +The whole document to this point analyses **one** way to use the plugin surface. There is a +second, and it is the one #130 proposes. They share a package format and nothing else — +different interception point, different capability set, different risk. + +| | **(A) plugin-as-interceptor** | **(B) plugin-as-installer-for-the-proxy** | +|---|---|---| +| Mechanism | `PostToolUse` → `updatedToolOutput`, in-process | `SessionStart` hook + an install skill that writes `env.ANTHROPIC_BASE_URL` into the user's real `settings.json` | +| Where the work happens | inside Claude Code, upstream of the transcript | on the wire, in `proxy/` — unchanged | +| Is it a new transport? | **Yes** — a fourth one, hence this document's title | **No.** It is transport #1 (the proxy) in new packaging | +| Component set | offloaders only; the cache half is unreachable | **all of it.** Nothing is given up, because the proxy still sees the envelope | +| `/stats`, cost tiers, benchmarks | unavailable — no provider response | unchanged | +| Status | **unbuilt.** Recommended below | proposed in #130 | +| Defensive KV-cache layer | obviated (the win in §KV-cache) | still required, exactly as today | + +**Everything in this document's component tables describes (A).** Every "unreachable from a +plugin" verdict below is a statement about interception inside the harness, and is +**irrelevant to (B)**, which runs the full proxy and therefore keeps `cachesplit`, +`cacheinject`, `mask`, `failed_run` and `/stats`. A reader who takes the tables below as a +description of the plugin #130 ships will be wrong about most of the component set. + +(B) also disposes of this document's own headline blocker, and it is worth being explicit that +it does so *legitimately* rather than by a loophole: the manifest cannot carry +`ANTHROPIC_BASE_URL`, but nothing stops a skill from merging that key into the settings file +the user already owns. This document mentions settings-based installation only as a +`promptCacheTtl` aside and never weighs it as a transport candidate. It is the option that won, +and the reason it wins is that it pays none of the costs enumerated here. + +So the two are complementary rather than competing: **(B) is how you get the proxy onto a +stranger's machine; (A) is a capability the proxy does not have** — offload at birth, and the +defensive cache layer becoming unnecessary. (A) remains unbuilt and remains the live question. + +**On the transport count.** (A) would be a fourth transport beside `proxy/`, the AuthBridge +plugin and `adapters/bifrost`. (B) adds none, so README's three-transport *Integrate* table is +correct as it stands and should not change on account of #130. + +**Status, and why this document should not be read as a record of it.** (A) has never been +built. (B) is specified in #130 — three skills over three scripts plus a `SessionStart` hook that +self-gates on `$ANTHROPIC_BASE_URL` matching its own port — and its implementation is in flight, so +the authority on what (B) actually ships is #130 and its implementing PR, never this page. What +matters here is only the design distinction: **(B) does not need any of the interception this +document analyses, and therefore pays none of its costs.** + ## Component verdicts ### Reformat (lossless) @@ -253,9 +333,13 @@ fit in the repo: provider's cap of 4 is a hard 400. `cacheinject` is in no preset today (placement is unmeasured), so losing it costs nothing -measured. **`cachesplit` is the real loss**: it is in *every* preset and it is the -best-evidenced component we have — −34.1% cost, 0% → 96.7% cache hit in an isolated A/B. -The plugin gives that up entirely. +measured. **`cachesplit` is the larger loss of the two, and smaller than this document first +claimed.** It is in every preset, and an earlier revision called it "the best-evidenced component +we have" — which `docs/components/cacheinject.md:209` contradicts directly, since the −34.1% is one +task measured three times. Priced in the regime a plugin user occupies it is **$0.0298 across 1,127 +sessions**, and on Terminal-Bench it recorded `cachesplit acted=0` for a *structural* reason +(`docs/results/terminal-bench-comparison.md:155`: the SDK never appends the git/env snapshot the CLI +does). The plugin gives up cents on interactive traffic and gains the free defensive half. ### Offload (lossy, reversible) @@ -267,9 +351,10 @@ The plugin gives that up entirely. | `extract` | **Yes** | Deterministic per-output noise collapse. | | `skeleton` | **Yes — better than the proxy** | Per-output tree-sitter reduction, and the hook supplies `tool_input.file_path`, so language selection stops being content sniffing. Still needs the `cg_skeleton` build tag. | | `dedup` | **Yes, with state** | Replaces the *later* byte-identical output — forward-only, which is exactly what a hook can do. Needs cross-turn digests, so it needs a store that outlives one hook process. | -| `extract_llm` | **Technically yes; leave it off** | The transform is per-output, but its economic gate needs the caching-backend fact and the fresh/cache-read token split, none of which a hook can see. Since #28 it declines on caching backends anyway (~8× underwater, break-even ~30,500 tokens/output). A synchronous cheap-model call inside a per-tool-call hook is also the worst place to put one. | +| `extract_llm` | **Technically yes; leave it off** | The transform is per-output, but its economic gate needs the caching-backend fact and the fresh/cache-read token split, none of which a hook can see. Since #28 it declines on caching backends anyway (**82×** underwater on Terminal-Bench — 197,548 unique tokens saved, worth $0.0395 at the cache-read rate they actually bill at, for $3.26 spent; the earlier "~8×" priced those tokens as *fresh* when they sit in the cached prefix). A synchronous cheap-model call inside a per-tool-call hook is also the worst place to put one. | +| `extract_llm_sweep` | **No** | Landed in #118 after this evaluation was written. It adjudicates *spent* tool outputs over the model's already-cached transcript, so it is retroactive by construction — it decides about messages that are already in Claude Code's history. Same wall as `mask` and `failed_run`, and for the same reason. | | `failed_run` | **No** | It keeps the newest run and collapses **earlier** runs. Retroactive by definition; at the moment run 2 is produced, run 1 is already in Claude Code's transcript and immutable. | -| `mask` | **No** | Age-based GC of *older* tool outputs. Same retroactivity. This is the top measured lever on our target traffic — 27.5% on Terminal-Bench, 12.5% on SWE-bench — and it is unreachable. | +| `mask` | **No** | Age-based GC of *older* tool outputs. Same retroactivity. It is the largest *known* token lever on our target traffic — **~27.5–29.5%, a single-task replay, never enforced in a benchmark arm** (`docs/components.md:439` says 27.5% Terminal-Bench / 12.5% SWE-bench; `docs/results/terminal-bench-comparison.md:86` says ~29.5%, from an arm where `mask` was **not** enabled) — and it is unreachable. Treat it as an unenforced upper bound, not a realized saving. | | `summarize` | **No** | Whole transcript, and it changes the message count. | ### Infrastructure @@ -286,6 +371,35 @@ The plugin gives that up entirely. | `proxy/` control plane — `spendgate`, `tenancy`, `limits`, `refusals`, `promexport` | **No** | Multi-tenant gateway concerns. A plugin is single-user and local. | | `adapters/bifrost` | N/A | | +## Prior art: rtk already ships this architecture, and we already benchmarked it + +This document proposed a hook that rewrites tool output before it enters context without noting +that **we have measured exactly that design** — as a competitor, in a full benchmark arm. +[rtk](https://github.com/rtk-ai/rtk) (Rust Token Killer) is a shell-level Bash-output hook, and +`docs/results/rtk.md:11` records it on SWE-bench Verified, 50 tasks: + +| arm | billed cost | vs baseline | reward | steps | request-path latency | +|---|--:|---|--:|--:|---| +| baseline (no compaction) | $31.98 | — | 43 solved | 36.1 | — | +| **rtk** | **$29.09** | **−9.0%** | 43 solved (**neutral**) | 33.2 (−8%) | **zero** | +| context-guru (proxy) | — | −13.2% | — | — | on the wire | + +So the expected value of candidate (A) is not unknown, and it is not the "−6,285 tokens on one +session" figure quoted in §0 — that has no denominator, no percentage, no `acted` count and no +paired arm, so by this repo's own conventions it is a demonstration that the mechanism works, not a +savings measurement. **Use −9.0%, reward-neutral, at zero request-path latency as the floor.** + +**And this design should beat rtk, for a structural reason worth stating because it is the clearest +unclaimed edge here.** rtk is a *shell* hook, so it only ever sees `Bash`. Claude Code's built-in +`Read`, `Grep` and `Glob` bypass it entirely — that is rtk's ceiling, and on code traffic those tools +carry a large share of the tokens. `updatedToolOutput` has no such ceiling: the hook manifest +proposed below uses matcher `".*"`, so `skeleton` on a `Read` and `extract` on a `Grep` are reachable +where rtk cannot go. Expected value: **≥ −9.0%, because matcher `.*` reaches the built-in file tools +rtk cannot**, and because `expand/` gives back a reversibility rtk does not have. + +The honest counterweight: rtk achieves its −9.0% with no store, no MCP server and no expand path, so +it is also the argument that the *first* useful version of (A) is small. + ## Permanence cuts both ways The defensive-half argument above is the plugin's best property, but it has a symmetric @@ -394,8 +508,9 @@ plugin is the #32 bug in a new costume. 2. **Oversized emissions degrade rather than fail — and the earlier "10,000-char cap" figure in this doc was wrong.** Measured by bisection with distinct-letter payload segments: `updatedToolOutput.stdout` goes out verbatim and uncapped up to ~30,000 chars, with the - real threshold in **(30,000, 40,000]** — most likely 32,768, i.e. 3–4× the figure this doc - first cited from the `additionalContext` / `systemMessage` / plain-stdout cap, which does + real threshold somewhere in **(30,000, 40,000]** — a bracket 10,000 wide, deliberately not + narrowed to a round guess, since no governing constant appears near `updatedToolOutput` in the + binary. That is 3–4× the figure this doc first cited from the `additionalContext` / `systemMessage` / plain-stdout cap, which does not govern this field. Above the threshold there is neither truncation nor rejection: the CLI's ordinary large-tool-output handler takes over, so the wire `tool_result` becomes a ~2,260-char `` wrapper carrying a 2 KB preview plus a pointer to the full @@ -422,11 +537,14 @@ credentials the runtime never sees. the gateway, agent holds a placeholder. `proxy/` drops into that path introducing no new concept; the plugin would have to be installed and force-enabled per harness, per pod. - Long-lived agents are DAM's headline, which makes **cache economics** the dominant cost - term. Claude Code on DAM means Anthropic/Bedrock — the explicit-breakpoint column, where - the offensive half is worth −34.1% and is unreachable from a plugin. (Were DAM routing to - an on-prem vLLM/llm-d backend, this row would flip: there the offensive half is already a - no-op and the plugin's free defensive half would be the better cache deployment. Worth - confirming which backends DAM actually targets before treating this as settled.) + term. Claude Code on DAM means Anthropic/Bedrock — the explicit-breakpoint column, where the + offensive half is unreachable from a plugin. **But price this carefully rather than at −34.1%:** + long-lived agents on a platform are closer to the *warm* regime than a human's interactive + sessions are, so this is the one place the benchmark figure is the more relevant one — and it is + still a single-task measurement, not a fleet average. (Were DAM routing to an on-prem vLLM/llm-d + backend, this row would flip: there the offensive half is already a no-op and the plugin's free + defensive half would be the better cache deployment. Worth confirming which backends DAM actually + targets before treating this as settled.) - `spendgate`, `tenancy`, `limits`, `promexport` are what a platform actually needs to enforce, and they are proxy-only. A plugin is user-installable and user-disableable; a platform wants a policy. @@ -451,12 +569,12 @@ other reasons. If the list below doesn't stand up, don't build it. Ranked by what the plugin uniquely provides — things the proxy *cannot* do, not things it already does: -1. **Offload at birth, permanently** *(the whole case — and it is conditional)*. Rewriting a - tool output before it enters the transcript means the reduction is realized once and - re-sent free forever, and the entire defensive KV-cache layer — `state.go`'s - freeze/replay, `MaxCachedIdx`, `Tracker`, `frozen_flips`, sticky ids — stops being - necessary rather than being ported. That is the only item here that changes the - architecture. **It rests entirely on the unverified persistence check (§0).** +1. **Offload at birth, permanently** *(the whole case)*. Rewriting a tool output before it + enters the transcript means the reduction is realized once and re-sent free forever, and the + entire defensive KV-cache layer — `state.go`'s freeze/replay, `MaxCachedIdx`, `Tracker`, + `frozen_flips`, sticky ids — stops being necessary rather than being ported. That is the only + item here that changes the architecture. **§0 settled the persistence question it used to rest + on: the replacement is what Claude Code stores and re-sends.** 2. **`context_guru_expand` as an MCP tool.** Deletes the 3-round `maxExpandRounds` cap, the response parsing, and the whole-SSE-buffering penalty. Worth doing *even with no hook at all*, and it works alongside the proxy rather than instead of it. @@ -473,8 +591,7 @@ already does: proxy". Real, but packaging, not capability. Items 3–6 are worth a few hundred lines. They do not justify the project on their own. **The -project is item 1, or it is item 2 alone.** Decide it with the §0 experiment before writing -adapter code. +project is item 1, or it is item 2 alone.** §0 settled item 1; decide item 2 on its own merits. ## Recommendation @@ -489,7 +606,14 @@ Build it, scoped as a transport for that half: accounting in plugin form. Cache work and multi-tenant policy stay in the proxy, which remains the only place they are possible. - **For DAM**: land the proxy in the gateway. Ship the plugin as the Claude-Code-session - layer on top of it — never as the DAM integration. + layer on top of it — never as the DAM integration. Note that this call now rests on + `spendgate` / `tenancy` / `limits` / `promexport` and harness-plurality, **not** on the −34.1% + figure an earlier revision leaned on: DAM is closer to the warm regime than a human is, but the + figure is still one task, and the platform-policy argument is the one that carries weight. +- **Not a substitute for (B).** If the goal is "an evaluator tries context-guru in one command", + that is candidate (B) in [§Two candidates](#two-candidates-and-only-one-of-them-is-a-new-transport) + and #130, which keeps the whole component set. (A) is a capability argument, not a distribution + one — item 6 below is packaging, and (B) does packaging better. - **Gate 0 is now closed** (persistence confirmed on the wire, plus a −6,285-token A/B on a real session), so the build is no longer conditional. Carry forward one hard requirement instead: the adapter emits the **object** `tool_response` shape and counts its own