From 6203a47d58f2835006517be954ef39fe5d43f23b Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 14:58:15 +0200 Subject: [PATCH 01/18] spec: initialize compaction continuity preservation --- .../spec.json | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 .kiro/specs/compaction-continuity-preservation/spec.json diff --git a/.kiro/specs/compaction-continuity-preservation/spec.json b/.kiro/specs/compaction-continuity-preservation/spec.json new file mode 100644 index 00000000..b20cbf1c --- /dev/null +++ b/.kiro/specs/compaction-continuity-preservation/spec.json @@ -0,0 +1,23 @@ +{ + "feature_name": "compaction-continuity-preservation", + "created_at": "2026-08-17T14:48:00+02:00", + "updated_at": "2026-08-17T14:48:00+02:00", + "language": "en", + "phase": "initialized", + "approvals": { + "requirements": { + "generated": false, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false, + "project_description": "Preserve the latest accepted plan, explicit user decisions, constraints, rationale, meaningful rejections, and remaining work across lossy coding-agent compaction. Reuse the compaction-recognition authority specified by compaction-event-detection without weakening its metadata-only observer contract. Maintain a bounded revisioned continuity capsule, harvest structured plan state deterministically, and use an optional independently routed background auxiliary model only for unstructured semantic extraction. Auxiliary extraction is off the primary agent session, has process-owned bounded worker lifetime and narrow synchronization barriers, and is normal billable model usage attributed by default to the originating authenticated user/account. Preserve existing routing, B2BUA, secure-session, billing, usage, streaming, and provider-opaque compaction semantics; add no second transcript database, provider client path, hidden system billing account, generic task runtime, or mutation of encrypted/opaque compaction payloads." +} From 0436680a1be7b8aa31fae29286f512c886cdbdbe Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 14:59:49 +0200 Subject: [PATCH 02/18] spec: define compaction continuity requirements --- .../requirements.md | 192 ++++++++++++++++++ 1 file changed, 192 insertions(+) create mode 100644 .kiro/specs/compaction-continuity-preservation/requirements.md diff --git a/.kiro/specs/compaction-continuity-preservation/requirements.md b/.kiro/specs/compaction-continuity-preservation/requirements.md new file mode 100644 index 00000000..601d17f5 --- /dev/null +++ b/.kiro/specs/compaction-continuity-preservation/requirements.md @@ -0,0 +1,192 @@ +# Requirements Document + +## Introduction + +Go-LIP shall preserve the decision state that matters for continuing a long coding-agent session across lossy context-compaction events. The feature is not general memory and is not a replacement compactor. Its purpose is narrower: retain the latest accepted plan, explicit user product/architecture decisions, constraints, useful rationale, meaningful rejected alternatives, current plan progress, and unresolved next actions when ordinary compaction would otherwise discard them. + +The preserved state shall be represented as a bounded, versioned **Continuity Capsule** rather than an ever-growing natural-language transcript. Structured planning state already exposed by an agent shall be harvested deterministically. A separately configured auxiliary LLM may be used only for genuinely semantic extraction such as identifying conversational acceptance, user decisions, rationale, or supersession that cannot be recovered safely from structured carriers. + +Auxiliary semantic extraction is explicitly **off the primary agent session** and runs as independent background work. It may use a completely different model/provider/route from the main coding session. It is nevertheless real additional model usage: by default its usage and cost belong to the same authenticated user/account that caused the extraction and must flow through normal Go-LIP admission, usage, metering, customer billing, and provider-cost accounting. + +This specification depends on the compaction-recognition authority defined by the existing `compaction-event-detection` specification. That detector remains metadata-only and non-mutating. Continuity preservation adds a separate content-bearing preservation/interception capability and shall not turn the detector observer into a request/response mutation surface. + +## Boundary Context + +- **In scope:** continuity-capsule schema/merge semantics; deterministic structured-plan harvesting; bounded semantic-extraction eligibility; sanitized extraction input; background auxiliary execution; independent extractor routing; originating-user billing; compaction-boundary synchronization; verified plaintext augmentation and post-compaction reinjection; repeated compactions; authoritative session/A-leg scoping; trusted session overrides; privacy, observability, and TDD gates. +- **Dependency:** implementation requires the compaction detector/rule authority specified by `compaction-event-detection`. If that runtime capability has not landed, it is a prerequisite rather than a reason to duplicate compaction signatures in this feature. +- **Existing authorities preserved:** core routing/selector parsing; B2BUA A-leg/B-leg lineage; secure-session authority; generation pinning; `ProcessServices` process lifetime; existing auxiliary execution; usage/concurrency authority; BillingCallID-based post-usage billing; canonical stream/retry commitment; existing feature-extension merge/runtime snapshot model. +- **Out of scope:** general RAG/long-term memory; retaining every conversation fact; replacing an agent's own compaction implementation; inventing a new provider client; modifying encrypted/opaque native compaction blobs; billing the extractor to a hidden system account by default; general-purpose durable job orchestration; general agent identity inference; provider-specific branching in core; a second full-transcript database. + +## Requirement 1: Shared Compaction Recognition Without Weakening Detection + +1.1. Continuity preservation shall reuse the canonical protocol/signature/history recognition authority defined by `compaction-event-detection`; it shall not maintain a second independent compaction-signature matrix. +1.2. The metadata-only `compaction.Observer` contract defined by that specification shall remain non-mutating, content-free, and fail-open. +1.3. A preservation-specific content-bearing contract may be added separately, but it shall not expose request/response content through ordinary compaction observer events. +1.4. Preservation shall be keyed from the same authoritative A-leg and compaction transaction identity used by the detector whenever those values are available. +1.5. A strict start candidate shall not cause a billable semantic-extraction job merely because a signature appears in a request that never successfully opens upstream. +1.6. When a recognized compaction request successfully opens upstream, continuity preservation may start one extraction job for that logical compaction transaction while the primary compaction request continues independently. +1.7. Completion-only/history evidence shall never invent a historical start event, but it may trigger preservation/reinjection behavior needed for a compaction that was only observable after installation. +1.8. Request-preview logic needed to protect the first post-compaction turn shall use the same deterministic matcher/fingerprint authority as committed detection and shall not itself emit lifecycle events or mutate detector transaction state. +1.9. Provider/frontend wire DTOs and provider SDK types shall not enter core compaction-continuity logic. +1.10. Generic words such as `plan`, `summarize`, `continue`, or `compact` shall not by themselves establish a compaction boundary or accepted plan. + +## Requirement 2: Versioned Continuity Capsule and Decision Semantics + +2.1. Persisted/in-memory continuity state shall use a versioned schema with at least schema version, monotonic revision, source/high-watermark metadata, and a content digest. +2.2. The capsule shall represent the latest accepted plan separately from user decisions/constraints so a plan can progress without rewriting unrelated product decisions. +2.3. Plan steps shall support at least pending, in-progress, completed, and removed/superseded semantics. +2.4. User decisions shall support at least active, superseded, and rejected semantics. +2.5. Each retained semantic fact shall carry enough source provenance to distinguish explicit user text, explicit user acceptance/correction, deterministic structured-plan state, and semantic extractor inference without retaining the whole transcript. +2.6. Explicit later user intent shall supersede conflicting older active intent; two contradictory decisions shall not remain simultaneously active merely because both occurred historically. +2.7. Assistant brainstorming/proposals shall not become authoritative user decisions unless later user evidence accepts, selects, or instructs execution of them. +2.8. A structured current plan exposed authoritatively by the coding harness may be retained as current plan state even when no conversational acceptance sentence is needed. +2.9. Explicit user rejection of an alternative may be retained when it materially constrains future work; a rejected alternative shall not later be presented as the active plan absent new user intent. +2.10. Useful user-provided rationale/trade-offs may be retained when needed to avoid re-litigating a settled choice, but rationale shall remain subordinate to the associated active/superseded decision status. +2.11. Ambiguous semantic evidence shall be omitted or represented as provisional/non-authoritative; it shall not be promoted to an explicit user decision. +2.12. Merge behavior shall be deterministic given the previous capsule and a validated extraction delta: duplicate facts shall be coalesced, statuses shall not regress, and stale revisions shall not overwrite newer state. +2.13. Capsule size shall be bounded by configured byte/token-equivalent limits. Overflow handling shall deterministically prioritize active decisions, active constraints, pending/in-progress plan steps, useful rationale, then historical/completed material. +2.14. Completed plan steps and superseded history may be condensed or dropped under bounded-retention policy, while currently active decisions and unresolved work remain until explicitly superseded/removed or the continuity branch ends. +2.15. The capsule is continuity state, not an audit transcript. It shall not contain arbitrary shell logs, source-file dumps, binary content, credentials, or unrelated tool output. + +## Requirement 3: Deterministic-First Extraction and Bounded Source Preparation + +3.1. Before invoking an extractor LLM, the feature shall harvest supported machine-readable planning state mechanically from canonical calls/items/tool data. +3.2. Initial deterministic carrier coverage shall include versioned equivalents of Codex `update_plan`, OpenCode todo state, Cline-style task-progress/checklist state where observable, and other stable structured plan carriers established during implementation research. +3.3. Structured carrier matching shall use canonical item/tool shapes rather than provider wire DTOs, and each carrier rule shall have a stable versioned ID plus positive/near-miss fixtures. +3.4. Structured plan data that can be normalized deterministically shall not be sent to an LLM merely to rediscover the same facts. +3.5. Cheap local eligibility heuristics shall decide whether semantic extraction is likely to add information; those heuristics may suppress an unnecessary model call but shall not themselves declare arbitrary prose to be accepted user intent. +3.6. A compaction with no relevant plan/decision/constraint candidates and no stale/missing capsule state shall perform zero semantic-extractor LLM calls. +3.7. Extraction input shall include the previous capsule and only the bounded new decision-relevant context needed to advance its source high-watermark. +3.8. User messages shall receive highest source priority; assistant planning/proposal/clarification content may be retained when needed to interpret later user acceptance/correction. +3.9. Ordinary tool outputs, shell/compiler logs, large file/code dumps, images, video, binary/file payloads, and unrelated external content shall be dropped or heavily truncated by default. +3.10. Structured plan/TODO tool calls and small results may survive source sanitization when their contents are needed for deterministic plan recovery. +3.11. Untrusted tool-result/external text shall be excluded from semantic decision extraction by default unless a narrowly justified rule requires it; if included, it remains untrusted data rather than extractor instructions. +3.12. Source preparation shall be incremental where possible: repeated compactions should cost approximately according to new relevant context plus the bounded prior capsule, not total historical session age. +3.13. The feature may keep a bounded process-local sanitized source window for local/completion-only compactions, but that window shall not become an unbounded or durable shadow transcript. +3.14. When secure-session transcript capture is already enabled and additional historical source is required, the feature may read that existing transcript subject to its authorization/redaction policy instead of creating a second transcript store. + +## Requirement 4: Off-Session Background Extraction and Worker Lifetime + +4.1. Semantic extraction shall execute as a proxy-created **auxiliary/internal** LLM invocation, not as a visible user/assistant turn in the primary agent conversation. +4.2. The extractor response shall be consumed only by continuity logic and shall not be surfaced to the primary client as an assistant message. +4.3. The extraction call shall have its own request/B-leg execution lifecycle and shall retain parent session/A-leg/trace/principal correlation only as non-authoritative lineage/accounting metadata. +4.4. Background extraction shall run behind a bounded independent asynchronous worker boundary. An in-process worker/goroutine pool is acceptable; an external worker process may be supported later without changing the semantic contract. +4.5. The implementation shall not spawn one unbounded goroutine per candidate/event and shall enforce explicit maximum concurrent jobs and queue capacity. +4.6. Job submission shall acquire any required runtime-generation execution lifetime before the originating request loses its spawn right, so delayed worker execution cannot use a retired generation accidentally. +4.7. Parent request cancellation shall not implicitly convert background execution into inline work; cancellation/retention semantics for a successfully opened compaction shall be explicit and leak-free. +4.8. A background job shall use an independent context/deadline rooted in the worker/service lifetime rather than continuing to depend on a canceled request context for execution. +4.9. Equivalent jobs for the same authoritative continuity key, compaction transaction, and target source revision shall be coalesced/idempotent so retry/failover/lifecycle duplication cannot cause duplicate billable extraction calls. +4.10. Queue saturation, worker unavailability, shutdown, or inability to retain a valid execution generation shall fail according to preservation failure policy and shall not silently fall back to an unbounded goroutine or direct provider call. +4.11. Process shutdown shall stop new job admission, cancel/join worker-owned execution, release retained generation lifetimes exactly once, and leave no goroutines/jobs behind. +4.12. Background job results shall be revision-checked before merge; an older completed job shall not overwrite newer user intent or a newer capsule revision. +4.13. Result/job retention shall be bounded and cleaned after consumption/expiry; the worker subsystem shall not become a general durable queue or arbitrary task scheduler. + +## Requirement 5: Independently Configurable Extractor Route and Session Isolation + +5.1. The semantic extractor route/model shall be independently configurable from the main session's route/model. +5.2. The extractor may use a completely different provider and model from the primary coding session. +5.3. Changing the primary session route, model alias, or A-leg routing override shall not implicitly change the configured extractor route. +5.4. An extractor route shall still be parsed/resolved by normal Go-LIP routing and capability/admission logic; the feature shall not select/open provider clients directly. +5.5. If the configured extractor selector uses normal failover/alias syntax, its attempts shall follow the same pre-output recovery and no-retry-after-output rules as other canonical calls. +5.6. The auxiliary call shall be stamped with a stable role/origin equivalent to `compaction_continuity_extractor` / internal auxiliary for diagnostics/accounting classification. +5.7. The extractor call shall have tools disabled and shall not be capable of executing workspace tools, MCP actions, shell commands, or other side effects. +5.8. The extractor shall use strict bounded output intended for schema validation, not an unconstrained conversational continuation. +5.9. The preservation feature itself shall be suppressed on its auxiliary extraction call, and existing auxiliary-depth recursion protection shall remain effective. +5.10. Parent session IDs/A-leg IDs carried on the child request are correlation only; they shall not cause the auxiliary call to append a primary secure-session turn, change primary last-activity/turn counts, mutate primary route overrides, or become user-visible session history. +5.11. The child call may have its own private execution/B2BUA lineage as required for routing, attempts, usage, and billing, but that lineage shall remain distinguishable from the primary user's conversational turn lineage. +5.12. A missing/invalid extractor route shall not silently inherit an expensive main-session model unless an explicit `inherit` policy is configured; the default behavior shall be validation failure or semantic-extraction disablement according to configuration rules. + +## Requirement 6: Originating-User Billing, Usage, and Cost Attribution + +6.1. Every semantic extractor LLM invocation is real additional model usage and shall be treated as billable auxiliary inference unless an explicit future operator-funded policy says otherwise. +6.2. By default, the extractor shall inherit the originating authenticated principal/scope so customer/account identity resolves to the same user/account that owns the triggering session. +6.3. The extractor shall pass through normal usage/concurrency authority and, where authoritative billing is enabled, the normal cheap credit screen, route/quote, atomic operational-exposure admission, terminal usage append, customer settlement, and provider-cost processing. +6.4. The auxiliary extractor shall receive its own BillingCallID and normal B-leg attempt records; its financial state shall not be merged into the primary call's BillingCallID. +6.5. Retries/failover B-legs that actually submit extractor work shall be accounted under the auxiliary BillingCallID using existing per-leg semantics. +6.6. Account/user aggregate usage and cost totals shall include extractor usage even though the call is not visible as a primary conversational turn. +6.7. Primary protocol-visible response usage shall not be inflated by or merged with extractor input/output/cache tokens; auxiliary and primary protocol usage remain separate execution records. +6.8. Operator/user-facing accounting evidence shall make auxiliary continuity-extraction usage distinguishable from primary inference through a content-free role/origin/workload dimension. +6.9. Provider COGS for extractor B-legs shall remain attributable to those actual B-legs and shall not be charged to the provider/model used by the primary session unless that is the extractor route selected. +6.10. If admission/credit policy rejects the auxiliary call before upstream submission, semantic extraction shall be skipped/fail-open by default; the proxy shall not bypass billing policy or silently charge an unrelated system account. +6.11. If upstream extractor work has already been submitted, resulting usage remains attributable/billable according to normal accounting even if its result arrives too late, is invalid, or is discarded as stale. +6.12. Enabling continuity preservation shall be documented as potentially generating extra billed inference beyond primary turns visible to the user. +6.13. An operator-funded/system-funded extractor is outside the first implementation; if introduced later it must be explicit opt-in accounting policy rather than implicit behavior. + +## Requirement 7: Compaction Integration, Bounded Barriers, and Reinjection + +7.1. For a strict recognized compaction request, semantic extraction shall start no earlier than the point where the primary compaction B-leg has successfully opened unless an already-authoritative local completion requires pre-open recovery. +7.2. Once started, the extractor shall run concurrently with the primary compaction work where possible rather than serially blocking the main B-leg for its full inference duration. +7.3. Preservation may join/wait for a relevant background job only at a narrow explicitly bounded preservation barrier where the result is needed to protect the compaction result or first post-compaction turn. +7.4. A barrier timeout shall obey configured failure policy; default fail-open behavior continues native compaction/continuation rather than deadlocking the session. +7.5. A completed validated capsule may be mechanically added to a compaction result only when Go-LIP has a verified mutable plaintext continuation-summary carrier for that dialect/path. +7.6. `CompactionItem.EncryptedContent`, provider opaque blobs, signatures, encrypted state, and unknown native compaction payloads shall never be rewritten to inject continuity text. +7.7. If the current compaction result cannot be safely augmented, the capsule shall be marked for proxy-owned reinjection on the first eligible post-compaction request. +7.8. Before that first eligible post-compaction B-leg opens, continuity logic shall use an already-ready capsule or await the matching in-flight extraction job up to the configured barrier limit. +7.9. Reinjection shall use canonical authority-aware request mutation: legacy/message-authoritative calls and item-authoritative calls shall receive a valid bounded proxy-owned instruction/message representation without violating canonical call invariants. +7.10. The injected continuity block shall be clearly delimited/versioned, content-bounded, and distinguishable from user text; it shall instruct the model to treat the facts as continuation state rather than new user input. +7.11. A capsule revision shall not be injected more than once for the same post-compaction boundary unless an explicit repeated-injection policy is configured. +7.12. Completion-only/local/history compaction detected on the first post-compaction request may schedule semantic extraction from the bounded pre-compaction source and synchronize before opening that request when necessary. +7.13. When deterministic extraction alone provides a sufficient current capsule, reinjection/augmentation shall not wait for or create a semantic-extractor job. +7.14. Preservation hooks/errors shall not alter routing/retry/output-commit semantics other than the explicitly bounded pre-output synchronization and canonical continuity injection described here. +7.15. No second LLM round trip shall normally be used merely to rewrite/improve the agent's native summary after the continuity extractor already produced a validated capsule. + +## Requirement 8: Repeated Compactions, Scope, Concurrency, and Reload Behavior + +8.1. Continuity state shall be keyed by authoritative branch identity: authoritative SessionID when available plus A-leg/branch identity; client-supplied session hints alone shall not authorize access to another branch's capsule. +8.2. Where secure-session authority is unavailable, the proxy-owned A-leg plus principal/scope isolation shall be used rather than trusting arbitrary client hints. +8.3. Each capsule merge shall use monotonic revision/compare-and-merge semantics so concurrent turns/jobs cannot let stale extraction overwrite a newer explicit decision. +8.4. Replayed/duplicate compaction lifecycle signals and B-leg retry/failover shall be idempotent with respect to job submission, capsule merge, and reinjection. +8.5. A new unrelated session or unrelated A-leg shall begin without inherited capsule state. +8.6. Reset/branch replacement shall retire/cancel pending work for the old branch according to bounded cleanup policy; it shall not leak decisions into the new branch. +8.7. Fork/clone inheritance, if supported, shall be explicit copy-on-fork from a known parent revision. Absence of an explicit fork relationship means no inheritance. +8.8. Process-owned capsule/job/source state shall survive immutable runtime-generation replacement and configuration reload while the process remains alive. +8.9. In-flight extraction shall use the extractor configuration/route snapshot captured at job submission; a later config reload affects only newly submitted jobs. +8.10. Three or more successive compactions shall merge `previous capsule + new relevant delta -> new capsule` rather than recursively trusting only the previous lossy summary. +8.11. Repeated compaction shall not cause monotonic duplicate prompt/capsule growth; dedupe and size bounds apply at every revision. +8.12. Process restart durability shall be explicit: this feature shall not claim restart-surviving capsule state unless an existing authorized durable source can reconstruct it or a separately designed durable capsule store is configured. +8.13. If a durable secure-session transcript is available and policy allows it, a resumed session may reconstruct missing capsule state from that transcript; if not, resume fails open with no hidden transcript capture or cross-session borrowing. + +## Requirement 9: Privacy, Security, and Trusted Session Policy + +9.1. The feature shall be disabled by default unless the operator explicitly enables it. +9.2. A remote extractor route creates a new data-egress path and shall be documented/configured as such. +9.3. Extractor input shall pass through applicable redaction/secret-handling policy before remote egress; raw credentials/secrets shall not be sent merely because they appeared in the session. +9.4. Transcript text supplied to the extractor shall be framed as untrusted data. Instructions embedded in user/tool/external content shall not override the fixed extractor task/schema/system policy. +9.5. The extractor shall have no tools and no direct filesystem/network side-effect capability beyond the configured model request. +9.6. Extractor prompt/output/capsule contents shall not be logged, emitted in normal metrics, or placed in ordinary diagnostics by default. +9.7. Metrics/diagnostics may expose content-free IDs, revisions, rule IDs, counts, sizes, latency, token usage, route/backend/model identifiers, and outcome/error classes. +9.8. Existing secure-session principal/workspace/tenant isolation shall apply to any transcript/source read used for extraction. +9.9. When transcript capture is disabled, this feature shall not silently enable durable full-transcript capture solely to support extraction. +9.10. A bounded process-local sanitized source window is permitted when needed, but it shall be isolated by authoritative continuity key and cleared/expired under bounded retention policy. +9.11. Per-session enable/disable or extractor-route override shall come only from trusted proxy-owned policy/session state. Unauthenticated client headers/metadata shall not self-enable external transcript export or choose the billed extractor route. +9.12. Trusted per-session policy shall override global defaults only within operator-defined allowed bounds; it shall not increase egress/worker/token limits beyond configured maxima unless explicitly authorized. +9.13. Auxiliary lineage/correlation metadata exposed to logs/accounting shall not include raw prompt excerpts or capsule content. + +## Requirement 10: Configuration, Failure Policy, and Observability + +10.1. Global feature configuration shall support at minimum enablement, deterministic/semantic preservation categories, extractor route, extractor timeout/input/output bounds, background concurrency/queue bounds, barrier timeout, capsule bound, retention/TTL, and failure mode. +10.2. Semantic extraction shall require an explicit route or explicit inherit policy; configuration shall not silently select a main-session model by accident. +10.3. Default failure mode shall be fail-open for model traffic: extractor timeout/error, saturation, invalid output, or persistence/state failure shall not make ordinary compaction unusable. +10.4. Fail-open shall preserve any already-valid deterministic capsule state and native compaction behavior rather than injecting malformed/partial model output. +10.5. Extractor output shall be parsed against a strict versioned schema with byte/depth/count limits before any merge or injection; malformed or extra-authority output shall be rejected. +10.6. Observability shall count compaction candidates, deterministic hits, semantic jobs submitted/coalesced/skipped/saturated, job latency/outcome, extractor input/output tokens, capsule revision/size/fact counts, barrier waits/timeouts, augmentations, reinjections, and stale/conflict rejections. +10.7. Billing/usage observability shall expose auxiliary continuity cost separately from primary inference without duplicating financial truth outside existing billing/metering stores. +10.8. Configuration reload shall not mutate active job routing/budgets mid-flight; new jobs use the newly compiled generation/config. +10.9. Disabling the feature for future turns shall prevent new extraction jobs while allowing bounded cleanup/settlement of already-submitted billable jobs. +10.10. No extractor failure path shall retry indefinitely, spin, block shutdown indefinitely, or fall back to an unconfigured provider/model. + +## Requirement 11: TDD, Architecture, and Non-Interference Gates + +11.1. Implementation shall begin with RED tests for capsule merge/supersession, structured carriers, eligibility/sanitization, background job lifetime, session isolation, independent routing, billing attribution, compaction barriers, repeated compactions, and opaque payload protection. +11.2. Worker/scheduler packages shall use deterministic scheduling controls and `goleak`/race tests for queue saturation, cancellation, shutdown, duplicate submissions, stale completion, and barrier races. +11.3. Tests shall prove an extraction job is not submitted when the compaction-looking request fails before upstream Open. +11.4. Tests shall prove the auxiliary child is absent from the primary secure-session transcript/turn count and does not mutate primary A-leg route-override/session state. +11.5. Tests shall prove the extractor may route to a different backend/model than the primary call and that a primary A-leg routing override does not rewrite the extractor selector. +11.6. Billing tests shall prove the child has its own BillingCallID/B-leg usage, inherits the originating account, contributes to account totals, remains distinguishable as auxiliary continuity work, and does not alter primary protocol-visible usage. +11.7. Tests shall cover extractor rejection before upstream submission, submitted-but-invalid result, submitted-but-stale result, retry/failover cost, and fail-open behavior. +11.8. Tests shall cover at least three successive compactions with decision supersession, completed/pending plan progress, dedupe, and bounded capsule growth. +11.9. Tests shall prove encrypted/opaque compaction content is byte-identical when continuity preservation is enabled. +11.10. Architecture gates shall prevent provider/frontend DTO imports, direct provider clients, generic service locators/task runtimes, a second full-transcript store, and financial mutation outside existing billing authorities. +11.11. The implementation shall preserve canonical request/event validity, output-commit/no-retry-after-output semantics, B2BUA lineage, secure-session ownership, generation pinning, and current billing settlement authorities. +11.12. Focused and repository quality tests shall run without external model credentials; live extractor tests, if any, are optional environment-gated evidence rather than a correctness prerequisite. +11.13. Final review shall remove duplicate rule catalogs, unnecessary persistence/framework layers, unbounded queues/goroutines, provider-specific core branches, and any second LLM summary-rewrite pass not required by the accepted design. From 1616841be128c1858dfb19d85fbc24510daaee3b Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:00:52 +0200 Subject: [PATCH 03/18] spec: analyze compaction continuity brownfield gaps --- .../gap-analysis.md | 155 ++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 .kiro/specs/compaction-continuity-preservation/gap-analysis.md diff --git a/.kiro/specs/compaction-continuity-preservation/gap-analysis.md b/.kiro/specs/compaction-continuity-preservation/gap-analysis.md new file mode 100644 index 00000000..9b6aef7a --- /dev/null +++ b/.kiro/specs/compaction-continuity-preservation/gap-analysis.md @@ -0,0 +1,155 @@ +# Brownfield Requirements Gap Analysis + +## Result + +**PASS after requirements corrections.** The feature can be implemented without a new provider client, second transcript database, second billing path, or general workflow runtime, but the initial requirements needed several brownfield constraints made more explicit. The most important gaps are lifecycle ownership for real background auxiliary work, strict isolation from the primary secure session, mutation ordering around the planned compaction detector, and the fact that canonical native compaction output is often opaque/encrypted rather than a mutable summary string. + +The corrections below are normative and must be reflected in the final `requirements.md`, `design.md`, and `tasks.md`. + +## Existing Brownfield Facts + +- `compaction-event-detection` is currently a merged **specification**, not a runtime capability on `main`. It defines a process-owned detector, metadata-only `compaction.Observer`, and request start emission only after successful upstream `Open`. +- Its detector design already owns the versioned compaction signature/history matrix and authoritative A-leg transaction state. Duplicating that matrix in continuity preservation would create drift and false disagreement. +- Current `pkg/lipsdk/auxiliary.Client` is synchronous. `Collect` delegates to `Stream`, which delegates to the normal runtime `Executor`. +- Current `internal/core/auxreq.Client` preserves the parent principal/scope, marks derived origin internal, supports `DisablePlugins`, and retains `genpin.KindAsync` **when the auxiliary call actually starts**. +- `genpin.Retainer.Retain` is explicitly a spawn-right operation: post-request-lease spawn attempts fail closed. Therefore a delayed worker cannot simply keep a request context and call synchronous `Aux.Collect` later. +- The current auxiliary child executes through the normal `Executor`. Without an explicit detached-session mode, it can enter secure-session BeginTurn/session recording paths; `Visibility` and `Role` are currently lineage metadata, not session-isolation authority. +- `lipapi.Call.Route.Selector` already provides a canonical route authority for a child call. The normal core router can therefore resolve a completely independent extractor selector without a second provider client. +- The billing identity adapter resolves account identity from the authenticated principal in `scope`. Because auxiliary execution already clones that scope, originating-user billing can reuse the existing billing path if detached execution preserves principal scope. +- `ProcessServices` owns process-lifetime services; `ExtensionState` is process-owned and survives generation reload. Feature bundle lifecycles, in contrast, belong to compiled feature/generation composition and are the wrong sole owner for a worker intended to outlive one immutable generation. +- `ExtensionState` is in-memory by default. `ScopeSession` partitions by `SessionView.PartitionKey()` and therefore by authoritative SessionID when available, but branch/A-leg must remain part of the feature key to avoid cross-branch aliasing. +- Secure-session transcript recording already exists when `TranscriptEnabled` is active. Creating another durable full transcript solely for this feature would violate existing extension-state and secure-session boundaries. +- `lipapi.CompactionItem` carries `EncryptedContent` and opaque provider data; it does **not** provide a universal plaintext summary field that can safely be appended to. +- Current `FeatureBundle` has no content-bearing compaction-preservation extension point. The #312 spec proposes only non-mutating `CompactionObservers`. + +## Gaps and Required Corrections + +### 1. #312 is a prerequisite, not an implemented dependency + +The initial feature request could be read as if compaction lifecycle events already exist in runtime. + +**Correction:** implementation planning shall explicitly order the `compaction-event-detection` runtime capability first. This feature may extend/refactor that implementation, but shall not copy its rule catalog as a fallback. Tests for this spec may use a local fake/shared recognizer contract until the prerequisite lands. + +### 2. Preservation mutation cannot be smuggled into `compaction.Observer` + +The detector's observer contract is intentionally content-free and non-mutating. Preservation needs canonical content and, for verified paths, limited mutation. + +**Correction:** add a distinct additive preservation/interception contract. Keep observer payload and dispatch semantics unchanged. FeatureBundle/snapshot merge must expose preservation interceptors separately and defensively freeze them like other extension slices. + +### 3. Request preview and committed detection have different truth semantics + +A first post-compaction request may need continuity before its B-leg opens, while #312 deliberately commits request detection only after successful `Open`. + +**Correction:** if a pre-open preview is required, it must be a pure/non-committing view over the same matcher/fingerprint state. Preview may protect a preservation barrier but cannot emit lifecycle events, advance detector state, or establish a billable strict-start job. Successful Open remains the committed start boundary. + +### 4. Final-release mutation ordering must remain explicit + +The #312 design places `ResponseReleased` at the final canonical release seam and guarantees observation does not mutate the event. Continuity may need to add a validated capsule to a verified plaintext carrier before client delivery. + +**Correction:** detector derivation, preservation finalization, metadata-observer dispatch, and client release require one explicit ordering. The detector itself remains pure/non-mutating; the preservation stage is separate, and no ordinary response hook runs after it. Opaque/native content remains byte-identical. + +### 5. Synchronous `Aux.Collect` is insufficient for real background work + +Launching `Aux.Collect` in a goroutine after the parent request returns is not safe: generation spawn rights can be gone, the parent context can be canceled, and shutdown cannot reliably own the goroutine. + +**Correction:** introduce one narrow **background auxiliary collection** capability backed by a process-owned bounded scheduler. Submission synchronously captures the executor/generation right and retains `genpin.KindAsync` before returning. Worker execution uses a worker-owned context/deadline; Await/Forget use a bounded job ID/result registry. This is an auxiliary-execution primitive, not a generic task engine. + +### 6. Worker lifetime belongs to `ProcessServices`, not only a feature lifecycle + +A generation-scoped plugin lifecycle can disappear on reload while a compaction extraction is still in flight. + +**Correction:** the scheduler is process-owned and closed through existing ProcessServices ownership/closer ordering. Generation snapshots receive non-owning clients. The job itself retains exactly the generation needed to execute its captured child call. + +### 7. Normal auxiliary execution can accidentally create primary-session effects + +Current auxiliary lineage is not sufficient to prevent secure-session BeginTurn/transcript/last-activity effects. + +**Correction:** add a typed internal **detached auxiliary session mode** recognized by core execution. It preserves authenticated principal/scope and parent correlation but suppresses primary secure-session authority/turn recording, primary A-leg route overrides, and client-visible session history. This mode must be internal canonical execution metadata, not a provider-visible opaque extension. + +### 8. Independent extractor routing must not inherit primary A-leg override implicitly + +The main user's A-leg may have a runtime routing override. Applying it to the internal extractor would violate the requested independently configurable model. + +**Correction:** the detached child uses its explicitly configured canonical `Call.Route.Selector` (or explicit `inherit` policy) and a private auxiliary execution lineage. Parent A-leg ID is correlation only and cannot become route authority for the child. + +### 9. User billing can reuse current authority, but auxiliary workload identity must be projected + +Principal scope inheritance is sufficient for account identity, and routing through the normal Executor gives the child its own BillingCallID. However operators/users must be able to separate primary inference from continuity overhead. + +**Correction:** add a bounded content-free workload/origin classification (`compaction_continuity_extractor`) to the accounting/metering/diagnostic correlation path. It must not become a second money ledger or pricing engine. Primary protocol usage remains unchanged while account-level billing includes the child. + +### 10. Provider-native compaction payloads are not universal summary text + +Mechanical append to `CompactionItem.EncryptedContent` or opaque data could corrupt replay/provider state. + +**Correction:** result-side merge is permitted only for a verified mutable plaintext carrier. Otherwise the capsule is persisted as proxy state and injected into the first eligible post-compaction canonical request. Encrypted/opaque fields are byte-for-byte immutable under this feature. + +### 11. Process extension state is reload-safe but not restart-durable + +It is appropriate for revisions, sanitized windows, pending job IDs and injection watermarks within one process. It does not justify claiming durable continuity after restart. + +**Correction:** define first implementation durability honestly. Process state survives generation reload. Across process restart, reconstruction is allowed from the existing authorized secure-session transcript when available; otherwise continuity state may be absent and behavior fails open. No new durable full transcript or generic durable feature-state framework is introduced by this spec. + +### 12. `ScopeSession` alone is not a branch key + +Authoritative SessionID is a useful partition, but one session can contain/relate to multiple A-leg branches or forks. + +**Correction:** capsule/source/job records use the authoritative session partition plus an explicit A-leg/branch key. When no secure SessionID exists, principal-isolated A-leg authority is used. Client hints never select another branch's capsule. + +### 13. Durable transcript access must stay behind a narrow authorized seam + +An official feature plugin should not import secure-session Bun/store internals merely to recover historical text. + +**Correction:** optional historical recovery uses a narrow authorized transcript-source/read adapter supplied by composition/core. Normal extraction prefers the canonical compaction baseline and process-local sanitized window, so most executions need no durable transcript read. + +### 14. Raw generic background results should not become another sensitive store + +A process-owned scheduler may need to hold a completed auxiliary result until the preservation barrier consumes it. + +**Correction:** background result storage is bounded/TTL-limited and never logged. Continuity parses/validates the small extractor response promptly and stores only the normalized capsule/delta afterward. Forget/expiry releases raw collected output. + +### 15. Disabling/config reload must not orphan already-billed work + +A generation reload can disable the feature or change extractor routing while a prior job has already submitted provider work. + +**Correction:** in-flight jobs use a submission-time immutable configuration/execution snapshot and finish/settle or cancel through the normal captured generation. Disabling prevents new jobs but does not erase accounting obligations for submitted work. + +## Brownfield Compatibility Matrix + +| Existing subsystem | Required treatment | +|---|---| +| `compactiondetect` / #312 rule authority | prerequisite and shared matcher/state; not duplicated | +| `compaction.Observer` | unchanged metadata-only/non-mutating contract | +| `FeatureBundle` / request snapshot | additive separate preservation-interceptor slice | +| `internal/core/auxreq` | retain synchronous client; add narrow background collector/scheduler path | +| `genpin` | retain `KindAsync` synchronously at job submission; no post-lease spawn | +| `ProcessServices` | owns bounded background auxiliary scheduler lifetime | +| secure session | primary turn/session state untouched by detached extractor | +| B2BUA | parent IDs correlation only; child gets private auxiliary execution lineage | +| routing / route overrides | child selector parsed normally; parent A-leg override not inherited implicitly | +| billing / metering | existing authorities and separate child BillingCallID reused | +| protocol-visible usage | primary response usage unchanged; child usage separately recorded | +| `ExtensionState` | process-local capsule/source/job metadata; key additionally includes A-leg | +| secure-session transcript | optional authorized historical recovery; no second transcript DB | +| `CompactionItem` opaque/encrypted fields | immutable; never used as arbitrary text injection carrier | +| provider/frontends | no provider-specific continuity logic in core | +| generation reload | process state/scheduler survive; job uses submission-time generation/config | + +## Corrected Required Invariants + +1. Detection truth and preservation mutation are separate contracts sharing one recognition authority. +2. A strict compaction start is billable only after successful upstream Open; preview never emits/commits lifecycle truth. +3. Background extraction captures its execution generation and principal at submit time and is process-owned thereafter. +4. The extractor is detached from the primary secure-session turn while remaining financially attributable to the originating user. +5. Parent A-leg IDs are correlation, not extractor routing/session authority. +6. Every submitted extractor call uses normal Executor/BillingCallID/B-leg accounting; primary protocol usage stays separate. +7. Opaque/encrypted compaction state is never rewritten for continuity. +8. Capsule/source/job state survives generation reload but does not falsely claim process-restart durability. +9. Session partition plus explicit A-leg/branch identity prevents cross-branch capsule leakage. +10. No second provider client, transcript database, money ledger, or generic task framework is introduced. + +## Requirements Correction Status + +The final requirements must explicitly incorporate gaps 2–15, especially: additive preservation-interceptor merge/freeze semantics; process-owned background auxiliary scheduler; submit-time generation retention; detached auxiliary secure-session behavior; final-release stage ordering; workload classification for billing; branch keying; narrow transcript recovery; and honest restart durability. + +With those corrections applied, the requirements quality gate is **PASS** and design may proceed. From d3a8a3bdde5bc9df24bb84a0ff4b1e4d55b94630 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:02:11 +0200 Subject: [PATCH 04/18] spec: apply compaction continuity gap corrections --- .../requirements.md | 323 +++++++++--------- 1 file changed, 167 insertions(+), 156 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/requirements.md b/.kiro/specs/compaction-continuity-preservation/requirements.md index 601d17f5..422118f9 100644 --- a/.kiro/specs/compaction-continuity-preservation/requirements.md +++ b/.kiro/specs/compaction-continuity-preservation/requirements.md @@ -12,181 +12,192 @@ This specification depends on the compaction-recognition authority defined by th ## Boundary Context -- **In scope:** continuity-capsule schema/merge semantics; deterministic structured-plan harvesting; bounded semantic-extraction eligibility; sanitized extraction input; background auxiliary execution; independent extractor routing; originating-user billing; compaction-boundary synchronization; verified plaintext augmentation and post-compaction reinjection; repeated compactions; authoritative session/A-leg scoping; trusted session overrides; privacy, observability, and TDD gates. -- **Dependency:** implementation requires the compaction detector/rule authority specified by `compaction-event-detection`. If that runtime capability has not landed, it is a prerequisite rather than a reason to duplicate compaction signatures in this feature. -- **Existing authorities preserved:** core routing/selector parsing; B2BUA A-leg/B-leg lineage; secure-session authority; generation pinning; `ProcessServices` process lifetime; existing auxiliary execution; usage/concurrency authority; BillingCallID-based post-usage billing; canonical stream/retry commitment; existing feature-extension merge/runtime snapshot model. -- **Out of scope:** general RAG/long-term memory; retaining every conversation fact; replacing an agent's own compaction implementation; inventing a new provider client; modifying encrypted/opaque native compaction blobs; billing the extractor to a hidden system account by default; general-purpose durable job orchestration; general agent identity inference; provider-specific branching in core; a second full-transcript database. - -## Requirement 1: Shared Compaction Recognition Without Weakening Detection - -1.1. Continuity preservation shall reuse the canonical protocol/signature/history recognition authority defined by `compaction-event-detection`; it shall not maintain a second independent compaction-signature matrix. -1.2. The metadata-only `compaction.Observer` contract defined by that specification shall remain non-mutating, content-free, and fail-open. -1.3. A preservation-specific content-bearing contract may be added separately, but it shall not expose request/response content through ordinary compaction observer events. -1.4. Preservation shall be keyed from the same authoritative A-leg and compaction transaction identity used by the detector whenever those values are available. -1.5. A strict start candidate shall not cause a billable semantic-extraction job merely because a signature appears in a request that never successfully opens upstream. -1.6. When a recognized compaction request successfully opens upstream, continuity preservation may start one extraction job for that logical compaction transaction while the primary compaction request continues independently. -1.7. Completion-only/history evidence shall never invent a historical start event, but it may trigger preservation/reinjection behavior needed for a compaction that was only observable after installation. -1.8. Request-preview logic needed to protect the first post-compaction turn shall use the same deterministic matcher/fingerprint authority as committed detection and shall not itself emit lifecycle events or mutate detector transaction state. -1.9. Provider/frontend wire DTOs and provider SDK types shall not enter core compaction-continuity logic. -1.10. Generic words such as `plan`, `summarize`, `continue`, or `compact` shall not by themselves establish a compaction boundary or accepted plan. +- **In scope:** continuity-capsule schema/merge semantics; deterministic structured-plan harvesting; bounded semantic-extraction eligibility; sanitized extraction input; process-owned background auxiliary execution; independent extractor routing; originating-user billing; detached auxiliary session semantics; compaction-boundary synchronization; verified plaintext augmentation and post-compaction reinjection; repeated compactions; authoritative session/A-leg scoping; trusted session overrides; privacy, observability, and TDD gates. +- **Dependency:** implementation requires the detector/rule authority specified by `compaction-event-detection`. If that runtime capability has not landed, it is a prerequisite rather than a reason to duplicate compaction signatures in this feature. +- **Existing authorities preserved:** core routing/selector parsing; B2BUA lineage; secure-session authority; generation pinning; `ProcessServices`; auxiliary execution; usage/concurrency authority; BillingCallID-based post-usage billing; canonical stream/retry commitment; extension merge/runtime snapshots. +- **Out of scope:** general RAG/long-term memory; retaining every conversation fact; replacing an agent compactor; a new provider client; rewriting encrypted/opaque native compaction blobs; hidden system-account billing by default; general-purpose durable job orchestration; general agent identity inference; provider-specific branching in core; a second full-transcript database. + +## Requirement 1: Shared Compaction Recognition and Separate Preservation Contract + +1.1. Continuity preservation shall reuse the protocol/signature/history recognition authority defined by `compaction-event-detection`; it shall not maintain a second independent compaction-signature matrix. +1.2. The metadata-only `compaction.Observer` contract shall remain non-mutating, content-free, and fail-open. +1.3. Go-LIP may add an additive content-bearing preservation/interception contract, but it shall be distinct from `compaction.Observer` and from ordinary response hooks. +1.4. `FeatureBundle`, the single feature merge surface, and the frozen request-runtime snapshot shall expose preservation interceptors separately and defensively copy/freeze them using existing extension conventions. +1.5. Preservation shall use the same authoritative A-leg and compaction transaction identity as the detector when available. +1.6. A strict start candidate shall not cause a billable semantic-extraction job if the compaction-looking request never successfully opens upstream. +1.7. When a recognized compaction request successfully opens upstream, preservation may start one extraction job for that logical transaction while primary compaction work continues independently. +1.8. Completion-only/history evidence shall never invent a historical start, but it may trigger extraction/reinjection required for a compaction only observable after installation. +1.9. Any pre-open request preview needed to protect the first post-compaction turn shall be a pure/non-committing view over the same matcher/fingerprint authority; preview shall not emit lifecycle events, advance detector state, or establish a strict-start billing trigger. +1.10. Final-release ordering shall keep detector derivation non-mutating: selected canonical event -> derived compaction lifecycle metadata -> separate preservation finalization -> metadata-observer dispatch -> client release. No ordinary response hook may run after preservation finalization. +1.11. Provider/frontend wire DTOs and provider SDK types shall not enter core continuity logic. +1.12. Generic words such as `plan`, `summarize`, `continue`, or `compact` shall not by themselves establish a compaction boundary or accepted plan. ## Requirement 2: Versioned Continuity Capsule and Decision Semantics -2.1. Persisted/in-memory continuity state shall use a versioned schema with at least schema version, monotonic revision, source/high-watermark metadata, and a content digest. -2.2. The capsule shall represent the latest accepted plan separately from user decisions/constraints so a plan can progress without rewriting unrelated product decisions. +2.1. Continuity state shall use a versioned schema containing at least schema version, monotonic revision, source/high-watermark metadata, branch identity, and content digest. +2.2. The capsule shall represent the latest accepted/current plan separately from user decisions and constraints. 2.3. Plan steps shall support at least pending, in-progress, completed, and removed/superseded semantics. 2.4. User decisions shall support at least active, superseded, and rejected semantics. -2.5. Each retained semantic fact shall carry enough source provenance to distinguish explicit user text, explicit user acceptance/correction, deterministic structured-plan state, and semantic extractor inference without retaining the whole transcript. -2.6. Explicit later user intent shall supersede conflicting older active intent; two contradictory decisions shall not remain simultaneously active merely because both occurred historically. -2.7. Assistant brainstorming/proposals shall not become authoritative user decisions unless later user evidence accepts, selects, or instructs execution of them. -2.8. A structured current plan exposed authoritatively by the coding harness may be retained as current plan state even when no conversational acceptance sentence is needed. -2.9. Explicit user rejection of an alternative may be retained when it materially constrains future work; a rejected alternative shall not later be presented as the active plan absent new user intent. -2.10. Useful user-provided rationale/trade-offs may be retained when needed to avoid re-litigating a settled choice, but rationale shall remain subordinate to the associated active/superseded decision status. -2.11. Ambiguous semantic evidence shall be omitted or represented as provisional/non-authoritative; it shall not be promoted to an explicit user decision. -2.12. Merge behavior shall be deterministic given the previous capsule and a validated extraction delta: duplicate facts shall be coalesced, statuses shall not regress, and stale revisions shall not overwrite newer state. -2.13. Capsule size shall be bounded by configured byte/token-equivalent limits. Overflow handling shall deterministically prioritize active decisions, active constraints, pending/in-progress plan steps, useful rationale, then historical/completed material. -2.14. Completed plan steps and superseded history may be condensed or dropped under bounded-retention policy, while currently active decisions and unresolved work remain until explicitly superseded/removed or the continuity branch ends. -2.15. The capsule is continuity state, not an audit transcript. It shall not contain arbitrary shell logs, source-file dumps, binary content, credentials, or unrelated tool output. +2.5. Retained facts shall carry bounded provenance sufficient to distinguish explicit user text, user acceptance/correction, deterministic structured-plan state, and semantic extractor inference without retaining the full transcript. +2.6. Later explicit user intent shall supersede conflicting older active intent; contradictory decisions shall not remain simultaneously active merely because both occurred historically. +2.7. Assistant brainstorming/proposals shall not become authoritative user decisions unless later user evidence accepts/selects/instructs execution of them. +2.8. A structured current plan exposed authoritatively by the harness may be retained as current plan state without requiring a conversational acceptance sentence. +2.9. Explicitly rejected alternatives may be retained when they constrain future work; they shall not reappear as active absent later user reversal. +2.10. Useful rationale/trade-offs may be retained with the associated decision and shall follow that decision's active/superseded state. +2.11. Ambiguous semantic evidence shall be omitted or represented as provisional/non-authoritative, never promoted to explicit user intent. +2.12. Merge behavior shall be deterministic for `previous capsule + validated delta`: duplicate facts coalesce, statuses do not regress, and stale revisions cannot overwrite newer state. +2.13. Capsule size shall be bounded by configured byte/token-equivalent limits; overflow shall prioritize active decisions/constraints, pending/in-progress plan steps, useful rationale, then historical/completed material. +2.14. Completed steps and superseded history may be condensed/dropped under retention policy while active decisions and unresolved work remain until superseded/removed or the branch ends. +2.15. The capsule is continuity state, not an audit transcript, and shall not contain arbitrary logs, file dumps, binaries, credentials, or unrelated tool output. ## Requirement 3: Deterministic-First Extraction and Bounded Source Preparation 3.1. Before invoking an extractor LLM, the feature shall harvest supported machine-readable planning state mechanically from canonical calls/items/tool data. 3.2. Initial deterministic carrier coverage shall include versioned equivalents of Codex `update_plan`, OpenCode todo state, Cline-style task-progress/checklist state where observable, and other stable structured plan carriers established during implementation research. -3.3. Structured carrier matching shall use canonical item/tool shapes rather than provider wire DTOs, and each carrier rule shall have a stable versioned ID plus positive/near-miss fixtures. -3.4. Structured plan data that can be normalized deterministically shall not be sent to an LLM merely to rediscover the same facts. -3.5. Cheap local eligibility heuristics shall decide whether semantic extraction is likely to add information; those heuristics may suppress an unnecessary model call but shall not themselves declare arbitrary prose to be accepted user intent. -3.6. A compaction with no relevant plan/decision/constraint candidates and no stale/missing capsule state shall perform zero semantic-extractor LLM calls. -3.7. Extraction input shall include the previous capsule and only the bounded new decision-relevant context needed to advance its source high-watermark. -3.8. User messages shall receive highest source priority; assistant planning/proposal/clarification content may be retained when needed to interpret later user acceptance/correction. -3.9. Ordinary tool outputs, shell/compiler logs, large file/code dumps, images, video, binary/file payloads, and unrelated external content shall be dropped or heavily truncated by default. -3.10. Structured plan/TODO tool calls and small results may survive source sanitization when their contents are needed for deterministic plan recovery. -3.11. Untrusted tool-result/external text shall be excluded from semantic decision extraction by default unless a narrowly justified rule requires it; if included, it remains untrusted data rather than extractor instructions. -3.12. Source preparation shall be incremental where possible: repeated compactions should cost approximately according to new relevant context plus the bounded prior capsule, not total historical session age. -3.13. The feature may keep a bounded process-local sanitized source window for local/completion-only compactions, but that window shall not become an unbounded or durable shadow transcript. -3.14. When secure-session transcript capture is already enabled and additional historical source is required, the feature may read that existing transcript subject to its authorization/redaction policy instead of creating a second transcript store. - -## Requirement 4: Off-Session Background Extraction and Worker Lifetime - -4.1. Semantic extraction shall execute as a proxy-created **auxiliary/internal** LLM invocation, not as a visible user/assistant turn in the primary agent conversation. -4.2. The extractor response shall be consumed only by continuity logic and shall not be surfaced to the primary client as an assistant message. -4.3. The extraction call shall have its own request/B-leg execution lifecycle and shall retain parent session/A-leg/trace/principal correlation only as non-authoritative lineage/accounting metadata. -4.4. Background extraction shall run behind a bounded independent asynchronous worker boundary. An in-process worker/goroutine pool is acceptable; an external worker process may be supported later without changing the semantic contract. -4.5. The implementation shall not spawn one unbounded goroutine per candidate/event and shall enforce explicit maximum concurrent jobs and queue capacity. -4.6. Job submission shall acquire any required runtime-generation execution lifetime before the originating request loses its spawn right, so delayed worker execution cannot use a retired generation accidentally. -4.7. Parent request cancellation shall not implicitly convert background execution into inline work; cancellation/retention semantics for a successfully opened compaction shall be explicit and leak-free. -4.8. A background job shall use an independent context/deadline rooted in the worker/service lifetime rather than continuing to depend on a canceled request context for execution. -4.9. Equivalent jobs for the same authoritative continuity key, compaction transaction, and target source revision shall be coalesced/idempotent so retry/failover/lifecycle duplication cannot cause duplicate billable extraction calls. -4.10. Queue saturation, worker unavailability, shutdown, or inability to retain a valid execution generation shall fail according to preservation failure policy and shall not silently fall back to an unbounded goroutine or direct provider call. -4.11. Process shutdown shall stop new job admission, cancel/join worker-owned execution, release retained generation lifetimes exactly once, and leave no goroutines/jobs behind. -4.12. Background job results shall be revision-checked before merge; an older completed job shall not overwrite newer user intent or a newer capsule revision. -4.13. Result/job retention shall be bounded and cleaned after consumption/expiry; the worker subsystem shall not become a general durable queue or arbitrary task scheduler. - -## Requirement 5: Independently Configurable Extractor Route and Session Isolation - -5.1. The semantic extractor route/model shall be independently configurable from the main session's route/model. -5.2. The extractor may use a completely different provider and model from the primary coding session. -5.3. Changing the primary session route, model alias, or A-leg routing override shall not implicitly change the configured extractor route. -5.4. An extractor route shall still be parsed/resolved by normal Go-LIP routing and capability/admission logic; the feature shall not select/open provider clients directly. -5.5. If the configured extractor selector uses normal failover/alias syntax, its attempts shall follow the same pre-output recovery and no-retry-after-output rules as other canonical calls. -5.6. The auxiliary call shall be stamped with a stable role/origin equivalent to `compaction_continuity_extractor` / internal auxiliary for diagnostics/accounting classification. -5.7. The extractor call shall have tools disabled and shall not be capable of executing workspace tools, MCP actions, shell commands, or other side effects. -5.8. The extractor shall use strict bounded output intended for schema validation, not an unconstrained conversational continuation. -5.9. The preservation feature itself shall be suppressed on its auxiliary extraction call, and existing auxiliary-depth recursion protection shall remain effective. -5.10. Parent session IDs/A-leg IDs carried on the child request are correlation only; they shall not cause the auxiliary call to append a primary secure-session turn, change primary last-activity/turn counts, mutate primary route overrides, or become user-visible session history. -5.11. The child call may have its own private execution/B2BUA lineage as required for routing, attempts, usage, and billing, but that lineage shall remain distinguishable from the primary user's conversational turn lineage. -5.12. A missing/invalid extractor route shall not silently inherit an expensive main-session model unless an explicit `inherit` policy is configured; the default behavior shall be validation failure or semantic-extraction disablement according to configuration rules. +3.3. Carrier matching shall use canonical item/tool shapes rather than provider DTOs; each rule shall have a stable versioned ID and positive/near-miss fixtures. +3.4. Carrier rules identify structured plan semantics, not agent identity; compaction-family identity remains owned by the shared detector. +3.5. Structured state that can be normalized deterministically shall not be sent to an LLM merely to rediscover the same facts. +3.6. Cheap local eligibility heuristics shall decide whether semantic extraction is likely to add information; they may suppress a model call but shall not declare arbitrary prose to be accepted user intent. +3.7. A compaction with no relevant plan/decision/constraint candidates and no stale/missing capsule need shall perform zero semantic-extractor calls. +3.8. Extractor input shall include the previous capsule plus only bounded new decision-relevant context needed to advance its source high-watermark. +3.9. User messages receive highest source priority; assistant plan/proposal/clarification content may be retained only as needed to interpret user acceptance/correction. +3.10. Ordinary tool outputs, shell/compiler logs, large file/code dumps, images/video/binary payloads, and unrelated external content shall be dropped or heavily truncated by default. +3.11. Structured plan/TODO tool calls and small required results may survive sanitization. +3.12. Untrusted tool-result/external text shall be excluded from semantic decision extraction by default; any included content remains untrusted data rather than extractor instructions. +3.13. Source preparation shall be incremental where possible so repeated-compaction cost follows new relevant context plus the bounded prior capsule, not total session age. +3.14. The feature may keep a bounded process-local sanitized source window for completion-only/local compactions, but it shall not become an unbounded or durable shadow transcript. +3.15. When existing secure-session transcript capture is enabled and authorized historical recovery is needed, it shall be accessed through a narrow read/source adapter rather than by importing secure-session store/Bun internals into the feature. +3.16. When transcript capture is disabled, the feature shall not enable durable full-transcript capture solely for continuity extraction. + +## Requirement 4: Off-Session Background Auxiliary Execution + +4.1. Semantic extraction shall execute as a proxy-created auxiliary/internal LLM invocation, not as a visible user/assistant turn in the primary agent conversation. +4.2. Extractor output shall be consumed by continuity logic and never surfaced as an assistant message. +4.3. The child shall have its own execution/B-leg lifecycle; parent session/A-leg/trace values are non-authoritative correlation/accounting lineage only. +4.4. Background extraction shall run behind a bounded independent asynchronous worker boundary. A bounded in-process goroutine/worker pool is acceptable; an external worker may be supported later without changing semantics. +4.5. The implementation shall enforce explicit maximum concurrent jobs and queue capacity and shall not spawn an unbounded goroutine per candidate/event. +4.6. Background scheduling shall be process-owned through `ProcessServices` or an equivalent existing process ownership seam, not solely by a generation-scoped feature lifecycle. +4.7. Submission shall synchronously resolve/capture the executable generation and retain `genpin.KindAsync` before returning; a delayed worker shall not attempt a new spawn after the originating request lease is gone. +4.8. Worker execution shall use an independent worker-owned context/deadline carrying only cloned required principal/scope/correlation values, not the canceled parent request context as its lifetime root. +4.9. The process-owned scheduler shall expose a narrow background-auxiliary collection contract with bounded job IDs/results and await/forget behavior; it shall not become a generic arbitrary task engine. +4.10. Equivalent jobs for the same authoritative branch, compaction transaction, and target source revision shall coalesce/idempotently reuse one submission so retries/failover cannot duplicate billable extraction calls. +4.11. Queue saturation, unavailable worker, shutdown, or failed generation retention shall follow preservation failure policy and shall not fall back to an unbounded goroutine/direct provider call. +4.12. Process shutdown shall stop admission, cancel/join worker execution, release generation pins exactly once, and satisfy repository goleak/race requirements. +4.13. Job results shall be revision-checked before merge; stale jobs cannot overwrite newer intent/capsule revisions. +4.14. Completed raw auxiliary results shall be held only in bounded TTL-limited process memory until parsed/consumed, never logged, and explicitly forgotten/expired after normalized capsule/delta storage. +4.15. Disabling/reloading the feature shall prevent new jobs according to the new configuration but shall not erase accounting obligations or leak execution for provider work already submitted. + +## Requirement 5: Independently Configurable Route and Detached Session Semantics + +5.1. The extractor route/model shall be independently configurable from the main session route/model and may use a completely different provider/model. +5.2. Changing the main session model, alias, or A-leg route override shall not implicitly change the extractor route. +5.3. Extractor routing shall still use normal Go-LIP canonical selector parsing, routing, capability, failover, admission, and attempt machinery; no direct provider client may be opened by the feature. +5.4. The child shall use an explicit configured `Call.Route.Selector` or an explicit `inherit` policy. Missing/invalid route shall not silently inherit the main model. +5.5. Parent A-leg ID shall be correlation only and shall not become child route-override authority. +5.6. The child shall be stamped with stable content-free role/origin equivalent to `compaction_continuity_extractor` / internal auxiliary. +5.7. Tools shall be disabled; the child shall not execute workspace tools, MCP actions, shell commands, or other side effects. +5.8. Extractor output shall be bounded and schema-oriented rather than unconstrained conversational output. +5.9. The preservation plugin shall suppress itself on the child; existing auxiliary-depth recursion limits remain effective. +5.10. Core auxiliary execution shall provide a typed detached-session mode for this workload. It shall preserve authenticated principal/scope and parent correlation while suppressing primary secure-session BeginTurn/turn transcript/last-activity effects. +5.11. Detached execution shall not mutate the primary A-leg route override, primary session turn count, primary session transcript, or client-visible session history. +5.12. Detached-session authority shall be internal execution metadata and shall not be encoded as provider-visible opaque call content. +5.13. The child may use private B2BUA/sessionless execution lineage required for routing/usage/billing, but it shall be clearly distinguishable from the user's primary conversational turn. ## Requirement 6: Originating-User Billing, Usage, and Cost Attribution -6.1. Every semantic extractor LLM invocation is real additional model usage and shall be treated as billable auxiliary inference unless an explicit future operator-funded policy says otherwise. -6.2. By default, the extractor shall inherit the originating authenticated principal/scope so customer/account identity resolves to the same user/account that owns the triggering session. -6.3. The extractor shall pass through normal usage/concurrency authority and, where authoritative billing is enabled, the normal cheap credit screen, route/quote, atomic operational-exposure admission, terminal usage append, customer settlement, and provider-cost processing. -6.4. The auxiliary extractor shall receive its own BillingCallID and normal B-leg attempt records; its financial state shall not be merged into the primary call's BillingCallID. -6.5. Retries/failover B-legs that actually submit extractor work shall be accounted under the auxiliary BillingCallID using existing per-leg semantics. -6.6. Account/user aggregate usage and cost totals shall include extractor usage even though the call is not visible as a primary conversational turn. -6.7. Primary protocol-visible response usage shall not be inflated by or merged with extractor input/output/cache tokens; auxiliary and primary protocol usage remain separate execution records. -6.8. Operator/user-facing accounting evidence shall make auxiliary continuity-extraction usage distinguishable from primary inference through a content-free role/origin/workload dimension. -6.9. Provider COGS for extractor B-legs shall remain attributable to those actual B-legs and shall not be charged to the provider/model used by the primary session unless that is the extractor route selected. -6.10. If admission/credit policy rejects the auxiliary call before upstream submission, semantic extraction shall be skipped/fail-open by default; the proxy shall not bypass billing policy or silently charge an unrelated system account. -6.11. If upstream extractor work has already been submitted, resulting usage remains attributable/billable according to normal accounting even if its result arrives too late, is invalid, or is discarded as stale. -6.12. Enabling continuity preservation shall be documented as potentially generating extra billed inference beyond primary turns visible to the user. -6.13. An operator-funded/system-funded extractor is outside the first implementation; if introduced later it must be explicit opt-in accounting policy rather than implicit behavior. - -## Requirement 7: Compaction Integration, Bounded Barriers, and Reinjection - -7.1. For a strict recognized compaction request, semantic extraction shall start no earlier than the point where the primary compaction B-leg has successfully opened unless an already-authoritative local completion requires pre-open recovery. -7.2. Once started, the extractor shall run concurrently with the primary compaction work where possible rather than serially blocking the main B-leg for its full inference duration. -7.3. Preservation may join/wait for a relevant background job only at a narrow explicitly bounded preservation barrier where the result is needed to protect the compaction result or first post-compaction turn. -7.4. A barrier timeout shall obey configured failure policy; default fail-open behavior continues native compaction/continuation rather than deadlocking the session. -7.5. A completed validated capsule may be mechanically added to a compaction result only when Go-LIP has a verified mutable plaintext continuation-summary carrier for that dialect/path. -7.6. `CompactionItem.EncryptedContent`, provider opaque blobs, signatures, encrypted state, and unknown native compaction payloads shall never be rewritten to inject continuity text. -7.7. If the current compaction result cannot be safely augmented, the capsule shall be marked for proxy-owned reinjection on the first eligible post-compaction request. -7.8. Before that first eligible post-compaction B-leg opens, continuity logic shall use an already-ready capsule or await the matching in-flight extraction job up to the configured barrier limit. -7.9. Reinjection shall use canonical authority-aware request mutation: legacy/message-authoritative calls and item-authoritative calls shall receive a valid bounded proxy-owned instruction/message representation without violating canonical call invariants. -7.10. The injected continuity block shall be clearly delimited/versioned, content-bounded, and distinguishable from user text; it shall instruct the model to treat the facts as continuation state rather than new user input. -7.11. A capsule revision shall not be injected more than once for the same post-compaction boundary unless an explicit repeated-injection policy is configured. -7.12. Completion-only/local/history compaction detected on the first post-compaction request may schedule semantic extraction from the bounded pre-compaction source and synchronize before opening that request when necessary. -7.13. When deterministic extraction alone provides a sufficient current capsule, reinjection/augmentation shall not wait for or create a semantic-extractor job. -7.14. Preservation hooks/errors shall not alter routing/retry/output-commit semantics other than the explicitly bounded pre-output synchronization and canonical continuity injection described here. -7.15. No second LLM round trip shall normally be used merely to rewrite/improve the agent's native summary after the continuity extractor already produced a validated capsule. - -## Requirement 8: Repeated Compactions, Scope, Concurrency, and Reload Behavior - -8.1. Continuity state shall be keyed by authoritative branch identity: authoritative SessionID when available plus A-leg/branch identity; client-supplied session hints alone shall not authorize access to another branch's capsule. -8.2. Where secure-session authority is unavailable, the proxy-owned A-leg plus principal/scope isolation shall be used rather than trusting arbitrary client hints. -8.3. Each capsule merge shall use monotonic revision/compare-and-merge semantics so concurrent turns/jobs cannot let stale extraction overwrite a newer explicit decision. -8.4. Replayed/duplicate compaction lifecycle signals and B-leg retry/failover shall be idempotent with respect to job submission, capsule merge, and reinjection. -8.5. A new unrelated session or unrelated A-leg shall begin without inherited capsule state. -8.6. Reset/branch replacement shall retire/cancel pending work for the old branch according to bounded cleanup policy; it shall not leak decisions into the new branch. -8.7. Fork/clone inheritance, if supported, shall be explicit copy-on-fork from a known parent revision. Absence of an explicit fork relationship means no inheritance. -8.8. Process-owned capsule/job/source state shall survive immutable runtime-generation replacement and configuration reload while the process remains alive. -8.9. In-flight extraction shall use the extractor configuration/route snapshot captured at job submission; a later config reload affects only newly submitted jobs. -8.10. Three or more successive compactions shall merge `previous capsule + new relevant delta -> new capsule` rather than recursively trusting only the previous lossy summary. -8.11. Repeated compaction shall not cause monotonic duplicate prompt/capsule growth; dedupe and size bounds apply at every revision. -8.12. Process restart durability shall be explicit: this feature shall not claim restart-surviving capsule state unless an existing authorized durable source can reconstruct it or a separately designed durable capsule store is configured. -8.13. If a durable secure-session transcript is available and policy allows it, a resumed session may reconstruct missing capsule state from that transcript; if not, resume fails open with no hidden transcript capture or cross-session borrowing. +6.1. Every semantic extractor invocation is real additional model usage and shall be billable auxiliary inference unless a future explicit operator-funded policy says otherwise. +6.2. By default the child shall inherit the originating authenticated principal/scope so the same customer/account identity is resolved. +6.3. The child shall pass through normal usage/concurrency authority and, where enabled, normal credit screen, route/quote, operational-exposure admission, terminal usage append, customer settlement, and provider-cost processing. +6.4. The auxiliary call shall receive its own BillingCallID and normal B-leg records; it shall not share the primary call's BillingCallID. +6.5. Actual extractor retries/failover B-legs shall be accounted under that auxiliary BillingCallID using existing per-leg semantics. +6.6. User/account aggregate usage/cost totals shall include extractor usage even though the call is not a visible primary turn. +6.7. Primary protocol-visible response usage shall not be inflated by extractor tokens; auxiliary and primary protocol usage remain separate execution records. +6.8. Existing metering/billing/diagnostics shall receive a bounded content-free workload/origin classification identifying continuity extraction, without creating a second money ledger or rating engine. +6.9. Provider COGS shall remain attributed to the actual extractor B-legs/provider/model selected by the extractor route. +6.10. If credit/admission rejects the child before upstream submission, semantic extraction shall skip/fail-open by default; billing policy shall not be bypassed and an unrelated system account shall not be charged. +6.11. If upstream extractor work was submitted, resulting usage remains billable/accountable even if the result is late, invalid, or discarded as stale. +6.12. Enabling the feature shall be documented as potentially generating extra billed inference beyond visible primary turns. +6.13. Operator-funded/system-funded extraction is outside the first implementation and, if later added, must be explicit opt-in accounting policy. + +## Requirement 7: Compaction Integration, Barriers, Augmentation, and Reinjection + +7.1. For a strict recognized compaction request, semantic extraction shall start no earlier than successful primary compaction B-leg Open, unless completion-only/local evidence is first observable on a request that must be protected pre-open. +7.2. After submission, extraction shall run concurrently with primary compaction work where possible rather than serially blocking the main B-leg for its full inference duration. +7.3. Preservation may await a background job only at a narrow bounded barrier where its result is needed to protect a compaction result or first eligible post-compaction turn. +7.4. Barrier timeout obeys configured failure policy; default fail-open continues native compaction/continuation rather than deadlocking. +7.5. A completed validated capsule may be mechanically added to a compaction result only for a verified mutable plaintext continuation-summary carrier. +7.6. `CompactionItem.EncryptedContent`, opaque provider blobs, signatures, encrypted state, and unknown native compaction payloads shall remain byte-identical. +7.7. If result-side augmentation is unsafe/unavailable, the capsule shall be marked for proxy-owned reinjection on the first eligible post-compaction request. +7.8. Before that first B-leg opens, preservation shall use an already-ready capsule or await its matching in-flight job up to the configured barrier. +7.9. Reinjection shall be canonical-authority-aware: legacy/message-authoritative and item-authoritative calls shall receive a valid bounded proxy-owned instruction/message representation without violating `Call.Validate` authority rules. +7.10. The continuity block shall be versioned/delimited/bounded and distinguishable from user text; it shall state that facts are prior continuation state, not a new user request. +7.11. One capsule revision shall not be injected repeatedly for the same boundary absent an explicit repeated-injection policy. +7.12. Completion-only/local/history compaction first recognized on the post-compaction request may schedule background extraction from the bounded prior source and synchronize before opening that request. +7.13. If deterministic extraction already supplies the necessary capsule, no semantic model job/barrier is required. +7.14. Preservation errors shall not change route selection, failover, no-retry-after-output, or output commitment except for the explicitly bounded pre-output barrier and canonical continuity injection. +7.15. No second LLM pass shall normally rewrite/improve the native summary after one validated semantic extractor result already exists. + +## Requirement 8: Repeated Compactions, Branch Scope, Concurrency, Reload, and Restart + +8.1. Continuity state shall be keyed by authoritative SessionID when available plus explicit A-leg/branch identity; client session hints alone shall never select another branch's capsule. +8.2. Without secure-session authority, principal-isolated proxy-owned A-leg authority shall be used rather than arbitrary client hints. +8.3. `ScopeSession` may provide the session partition, but the feature key shall additionally include A-leg/branch identity to prevent branch aliasing. +8.4. Capsule updates shall use monotonic revision/compare-and-merge semantics so concurrent turns/jobs cannot overwrite newer explicit intent. +8.5. Duplicate lifecycle signals and B-leg retry/failover shall be idempotent for job submission, merge, and reinjection. +8.6. New unrelated sessions/A-legs start without inherited state. +8.7. Reset/branch replacement shall retire/cancel pending old-branch work under bounded cleanup and shall not leak decisions into the new branch. +8.8. Fork/clone inheritance, if supported, shall be explicit copy-on-fork from a known parent revision; no explicit parent relationship means no inheritance. +8.9. Capsule/source/job state and the background scheduler shall be process-owned so they survive immutable generation replacement/config reload. +8.10. In-flight jobs use immutable route/budget/config and generation captured at submission; later reload affects newly submitted jobs only. +8.11. Three or more successive compactions shall merge `previous capsule + new relevant delta -> new capsule`, not recursively trust only the prior lossy summary. +8.12. Repeated compaction shall not cause monotonic duplicate capsule/prompt growth. +8.13. First implementation restart durability shall be honest: process-local state is not claimed durable across process restart. +8.14. When an authorized durable secure-session transcript already exists, missing capsule state may be reconstructed through the narrow transcript-source adapter; otherwise resume/restart fails open with no hidden capture or cross-session borrowing. +8.15. This spec shall not add a generic durable feature-state/job framework merely to make the capsule restart-durable. ## Requirement 9: Privacy, Security, and Trusted Session Policy -9.1. The feature shall be disabled by default unless the operator explicitly enables it. -9.2. A remote extractor route creates a new data-egress path and shall be documented/configured as such. -9.3. Extractor input shall pass through applicable redaction/secret-handling policy before remote egress; raw credentials/secrets shall not be sent merely because they appeared in the session. -9.4. Transcript text supplied to the extractor shall be framed as untrusted data. Instructions embedded in user/tool/external content shall not override the fixed extractor task/schema/system policy. -9.5. The extractor shall have no tools and no direct filesystem/network side-effect capability beyond the configured model request. -9.6. Extractor prompt/output/capsule contents shall not be logged, emitted in normal metrics, or placed in ordinary diagnostics by default. -9.7. Metrics/diagnostics may expose content-free IDs, revisions, rule IDs, counts, sizes, latency, token usage, route/backend/model identifiers, and outcome/error classes. -9.8. Existing secure-session principal/workspace/tenant isolation shall apply to any transcript/source read used for extraction. -9.9. When transcript capture is disabled, this feature shall not silently enable durable full-transcript capture solely to support extraction. -9.10. A bounded process-local sanitized source window is permitted when needed, but it shall be isolated by authoritative continuity key and cleared/expired under bounded retention policy. -9.11. Per-session enable/disable or extractor-route override shall come only from trusted proxy-owned policy/session state. Unauthenticated client headers/metadata shall not self-enable external transcript export or choose the billed extractor route. -9.12. Trusted per-session policy shall override global defaults only within operator-defined allowed bounds; it shall not increase egress/worker/token limits beyond configured maxima unless explicitly authorized. -9.13. Auxiliary lineage/correlation metadata exposed to logs/accounting shall not include raw prompt excerpts or capsule content. +9.1. The feature shall be disabled by default and requires explicit operator enablement. +9.2. A remote extractor route is a new data-egress path and shall be documented/configured as such. +9.3. Applicable redaction/secret policy shall run before extractor egress; raw credentials/secrets shall not be exported merely because they appeared in the session. +9.4. Transcript/source text shall be framed as untrusted data. Embedded user/tool/external instructions cannot override fixed extractor task/schema/system policy. +9.5. The child has no tools and no side-effect capability beyond the configured model request. +9.6. Extractor prompt/output/capsule contents shall not appear in normal logs/metrics/diagnostics by default. +9.7. Content-free IDs, revisions, rule/carrier IDs, counts, sizes, latency, token usage, route/backend/model identifiers, and outcomes may be observed. +9.8. Existing principal/workspace/tenant authorization applies to any transcript/source read. +9.9. Transcript-disabled sessions shall not silently acquire a durable full transcript. +9.10. A bounded sanitized process-local source window is allowed but must be branch-isolated and TTL/size bounded. +9.11. Per-session enable/disable or extractor-route override shall come only from trusted proxy-owned policy/session state; unauthenticated client headers/metadata cannot self-enable egress or choose the billed route. +9.12. Trusted per-session overrides shall remain within operator-defined maxima unless separately authorized. +9.13. Auxiliary lineage/accounting metadata shall not contain raw prompt excerpts or capsule content. ## Requirement 10: Configuration, Failure Policy, and Observability -10.1. Global feature configuration shall support at minimum enablement, deterministic/semantic preservation categories, extractor route, extractor timeout/input/output bounds, background concurrency/queue bounds, barrier timeout, capsule bound, retention/TTL, and failure mode. -10.2. Semantic extraction shall require an explicit route or explicit inherit policy; configuration shall not silently select a main-session model by accident. -10.3. Default failure mode shall be fail-open for model traffic: extractor timeout/error, saturation, invalid output, or persistence/state failure shall not make ordinary compaction unusable. -10.4. Fail-open shall preserve any already-valid deterministic capsule state and native compaction behavior rather than injecting malformed/partial model output. -10.5. Extractor output shall be parsed against a strict versioned schema with byte/depth/count limits before any merge or injection; malformed or extra-authority output shall be rejected. -10.6. Observability shall count compaction candidates, deterministic hits, semantic jobs submitted/coalesced/skipped/saturated, job latency/outcome, extractor input/output tokens, capsule revision/size/fact counts, barrier waits/timeouts, augmentations, reinjections, and stale/conflict rejections. -10.7. Billing/usage observability shall expose auxiliary continuity cost separately from primary inference without duplicating financial truth outside existing billing/metering stores. -10.8. Configuration reload shall not mutate active job routing/budgets mid-flight; new jobs use the newly compiled generation/config. -10.9. Disabling the feature for future turns shall prevent new extraction jobs while allowing bounded cleanup/settlement of already-submitted billable jobs. -10.10. No extractor failure path shall retry indefinitely, spin, block shutdown indefinitely, or fall back to an unconfigured provider/model. +10.1. Global feature config shall support at least enablement, preserved categories, extractor route, timeout/input/output limits, worker concurrency/queue bounds, barrier timeout, capsule/source/result bounds/TTL, and failure mode. +10.2. Semantic extraction requires an explicit route or explicit inherit policy; no accidental main-model default is allowed. +10.3. Default failure mode shall be fail-open for model traffic: extractor timeout/error/saturation/invalid output/state failure does not make native compaction unusable. +10.4. Fail-open preserves already-valid deterministic capsule state and never injects malformed/partial extractor output. +10.5. Extractor output shall be validated against a strict versioned schema with byte/depth/count limits before merge/injection. +10.6. Observability shall count compaction candidates; deterministic hits; semantic jobs submitted/coalesced/skipped/saturated; latency/outcome; extractor input/output tokens; capsule revision/size/fact counts; barrier waits/timeouts; augmentation/reinjection; stale/conflict rejection. +10.7. Billing/usage observability shall expose auxiliary continuity cost through existing accounting stores rather than duplicating financial truth. +10.8. Config reload shall not mutate active job route/budgets mid-flight. +10.9. Disabling the feature prevents new jobs while allowing bounded cleanup/accounting of already-submitted work. +10.10. No failure path may retry indefinitely, spin, block shutdown indefinitely, or fall back to an unconfigured provider/model. +10.11. Background raw-result retention shall be bounded/TTL-limited and observable only through content-free counts/status; normalized capsule state replaces raw result after consumption. ## Requirement 11: TDD, Architecture, and Non-Interference Gates -11.1. Implementation shall begin with RED tests for capsule merge/supersession, structured carriers, eligibility/sanitization, background job lifetime, session isolation, independent routing, billing attribution, compaction barriers, repeated compactions, and opaque payload protection. -11.2. Worker/scheduler packages shall use deterministic scheduling controls and `goleak`/race tests for queue saturation, cancellation, shutdown, duplicate submissions, stale completion, and barrier races. -11.3. Tests shall prove an extraction job is not submitted when the compaction-looking request fails before upstream Open. -11.4. Tests shall prove the auxiliary child is absent from the primary secure-session transcript/turn count and does not mutate primary A-leg route-override/session state. -11.5. Tests shall prove the extractor may route to a different backend/model than the primary call and that a primary A-leg routing override does not rewrite the extractor selector. -11.6. Billing tests shall prove the child has its own BillingCallID/B-leg usage, inherits the originating account, contributes to account totals, remains distinguishable as auxiliary continuity work, and does not alter primary protocol-visible usage. -11.7. Tests shall cover extractor rejection before upstream submission, submitted-but-invalid result, submitted-but-stale result, retry/failover cost, and fail-open behavior. -11.8. Tests shall cover at least three successive compactions with decision supersession, completed/pending plan progress, dedupe, and bounded capsule growth. -11.9. Tests shall prove encrypted/opaque compaction content is byte-identical when continuity preservation is enabled. -11.10. Architecture gates shall prevent provider/frontend DTO imports, direct provider clients, generic service locators/task runtimes, a second full-transcript store, and financial mutation outside existing billing authorities. -11.11. The implementation shall preserve canonical request/event validity, output-commit/no-retry-after-output semantics, B2BUA lineage, secure-session ownership, generation pinning, and current billing settlement authorities. -11.12. Focused and repository quality tests shall run without external model credentials; live extractor tests, if any, are optional environment-gated evidence rather than a correctness prerequisite. -11.13. Final review shall remove duplicate rule catalogs, unnecessary persistence/framework layers, unbounded queues/goroutines, provider-specific core branches, and any second LLM summary-rewrite pass not required by the accepted design. +11.1. Implementation shall begin with RED tests for capsule merge/supersession, structured carriers, eligibility/sanitization, background lifetime, detached session behavior, independent routing, billing attribution, barriers, repeated compactions, and opaque payload protection. +11.2. Worker packages shall use deterministic scheduling controls plus `goleak`/race tests for saturation, cancellation, shutdown, duplicate submission, stale completion, and barrier races. +11.3. Tests shall prove no extraction job is submitted when a strict compaction-looking request fails before upstream Open. +11.4. Tests shall prove the child is absent from primary secure-session transcript/turn count/last activity and does not mutate primary A-leg route-override/session state. +11.5. Tests shall prove extractor routing can differ from primary routing and that primary A-leg overrides do not rewrite the child selector. +11.6. Billing tests shall prove separate auxiliary BillingCallID/B-leg usage, originating account attribution, aggregate inclusion, continuity workload classification, and unchanged primary protocol usage. +11.7. Tests shall cover pre-submit billing rejection, submitted-but-invalid result, submitted-but-stale result, retry/failover cost, and fail-open behavior. +11.8. Tests shall cover at least three successive compactions with decision supersession, plan progress, dedupe, and bounded growth. +11.9. Tests shall prove encrypted/opaque compaction content is byte-identical with preservation enabled. +11.10. Tests shall prove process generation reload cannot orphan jobs/state and that job submission captured `KindAsync` ownership before the request spawn right ended. +11.11. Architecture gates shall prevent provider/frontend DTO imports, direct provider clients, generic service locators/task runtimes, a second full-transcript store, feature-owned money ledgers, or generation-scoped ownership as the sole worker lifetime. +11.12. The implementation shall preserve canonical call/event validity, output commitment/no-retry-after-output, B2BUA lineage, secure-session ownership, generation pinning, and current billing settlement authorities. +11.13. Focused/repository tests shall run without external model credentials; live extractor tests are optional environment-gated evidence. +11.14. Final review shall remove duplicate rule catalogs, unnecessary persistence/framework layers, unbounded queues/goroutines, provider-specific core branches, and any redundant second LLM summary-rewrite pass. From 514bf489bcfa77adbe083c3eca329e838ec4efcc Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:02:27 +0200 Subject: [PATCH 05/18] spec: mark compaction continuity requirements generated --- .kiro/specs/compaction-continuity-preservation/spec.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/spec.json b/.kiro/specs/compaction-continuity-preservation/spec.json index b20cbf1c..0bb119d5 100644 --- a/.kiro/specs/compaction-continuity-preservation/spec.json +++ b/.kiro/specs/compaction-continuity-preservation/spec.json @@ -3,10 +3,10 @@ "created_at": "2026-08-17T14:48:00+02:00", "updated_at": "2026-08-17T14:48:00+02:00", "language": "en", - "phase": "initialized", + "phase": "requirements-generated", "approvals": { "requirements": { - "generated": false, + "generated": true, "approved": false }, "design": { From a278a796337315305957395eeb01e9ca233b01f4 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:03:46 +0200 Subject: [PATCH 06/18] spec: research compaction continuity design --- .../research.md | 460 ++++++++++++++++++ 1 file changed, 460 insertions(+) create mode 100644 .kiro/specs/compaction-continuity-preservation/research.md diff --git a/.kiro/specs/compaction-continuity-preservation/research.md b/.kiro/specs/compaction-continuity-preservation/research.md new file mode 100644 index 00000000..77b4b3a2 --- /dev/null +++ b/.kiro/specs/compaction-continuity-preservation/research.md @@ -0,0 +1,460 @@ +# Research & Design Decisions + +## Summary + +Issue #344 is valid, but the safest implementation is not “run another summarizer and append whatever it returns.” Go-LIP already has most of the right architectural pieces: canonical requests/items, secure-session/B2BUA authority, process-owned extension state, auxiliary execution with principal propagation, generation pins, and BillingCallID-scoped billing. The missing pieces are narrow: + +1. the `compaction-event-detection` runtime capability must land and remain the single compaction-recognition authority; +2. preservation needs a **separate content-bearing interceptor** rather than mutating the metadata-only observer contract; +3. true background auxiliary execution needs a process-owned bounded scheduler that captures generation ownership at submit time; +4. the child needs typed **detached session semantics** so it is financially attributable to the user without becoming another primary conversation turn; +5. continuity must use a bounded structured capsule and only patch verified plaintext carriers; opaque/encrypted compaction state is reinjection-only. + +The resulting design is a parallel-first pipeline: deterministic plan harvest and source preparation are local; one independently routed semantic extractor job may run concurrently with the agent's own compaction; the main path only waits at a bounded preservation barrier when a result is actually required. + +## External Precedent + +The feature follows patterns already used by mature coding-agent harnesses: + +| Project | Relevant behavior | Design implication | +|---|---|---| +| Pi | default compaction summary explicitly carries goal, constraints/preferences, progress, key decisions, next steps, and critical context | decision state deserves first-class preservation rather than generic prose only | +| Pi | `session_before_compact` can replace/customize compaction and examples use a cheaper secondary model | pre-/around-compaction interception and independent model choice are established patterns | +| OpenAI Codex | compact prompt asks for progress/key decisions, user preferences/constraints, next steps, critical continuation data | preservation categories align with real agent continuation needs | +| OpenAI Codex | `update_plan` exposes structured plan state | deterministic plan harvest should precede semantic LLM inference | +| OpenCode | pre-summary compaction hook and rolling structured session checkpoint | structured cumulative state is preferable to repeatedly summarizing prior summaries | +| OpenCode | todo state is structured and appears in compaction/session representations | machine-readable plan progress can survive without LLM rediscovery | +| Codex issue #14347 | repeated compaction can lose historical decisions | repeated compactions require cumulative merge semantics, not recursive summary trust | + +References: + +- https://github.com/earendil-works/pi/blob/main/packages/coding-agent/src/core/compaction/compaction.ts +- https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/extensions.md +- https://github.com/earendil-works/pi/blob/main/packages/coding-agent/examples/extensions/custom-compaction.ts +- https://github.com/openai/codex/blob/main/codex-rs/prompts/templates/compact/prompt.md +- https://github.com/openai/codex/blob/main/codex-rs/core/src/tools/handlers/plan.rs +- https://github.com/anomalyco/opencode/blob/dev/packages/web/src/content/docs/plugins.mdx +- https://github.com/anomalyco/opencode/blob/dev/specs/v2/session.md +- https://github.com/anomalyco/opencode/blob/dev/packages/sdk/js/src/gen/types.gen.ts +- https://github.com/openai/codex/issues/14347 + +## Go-LIP Brownfield Findings + +### Compaction detection is designed but not implemented yet + +`.kiro/specs/compaction-event-detection/` defines: + +- canonical protocol/signature/history recognition; +- a process-owned detector keyed by authoritative A-leg; +- strict start only after successful upstream `Open`; +- completion at the final canonical release seam; +- a metadata-only, non-mutating `pkg/lipsdk/compaction.Observer`. + +There is no runtime `pkg/lipsdk/compaction` package on current `main`. Continuity preservation therefore has a hard implementation-order dependency on that spec. The correct response is to make #312 a prerequisite, not to recreate its signature table in #344. + +### Existing auxiliary execution already preserves user identity + +`pkg/lipsdk/auxiliary.Request` carries role, visibility, parent lineage, disabled plugin IDs, and a canonical child call. `internal/core/auxreq.Client`: + +- increments auxiliary depth; +- suppresses requested plugins; +- clones the parent principal/scope and marks `scope.OriginInternal`; +- creates an auxiliary trace ID and lineage extension; +- delegates to the ordinary runtime Executor. + +This is exactly the path we want for routing, attempts, usage, billing and provider calls. A direct HTTP/provider client would duplicate too many authorities. + +### But current auxiliary execution is synchronous + +`Aux.Collect` calls `Stream` and collects immediately. The client retains `genpin.KindAsync` when the child call starts. `genpin.Retainer` explicitly says a post-lease spawn attempt fails closed. + +Therefore this is unsafe: + +```text +request hook + -> go func() { Aux.Collect(parentCtx, ...) } // wrong +request returns / generation lease ends + -> goroutine starts later and tries to retain/execute +``` + +The fix is a submit-time handoff: resolve the current executor and retain `KindAsync` synchronously, then enqueue immutable child work to a process-owned scheduler. The worker owns its own context/deadline and releases the pin exactly once after terminal collection. + +### Worker ownership belongs to ProcessServices + +`ProcessServices` already owns process-lifetime mutable services, worker-like subsystems, stores and closers. `ExtensionState` is also process-owned and survives generation replacement. A feature `Lifecycle` alone is generation-composed and is not the right sole owner for work that may overlap a hot reload. + +A narrow `auxreq` background collector/scheduler under ProcessServices matches current ownership architecture and keeps the feature plugin away from Executor/provider internals. + +### Current auxiliary execution needs a detached-session policy + +The child delegates through the ordinary Executor. The primary prepare path performs secure-session BeginTurn, transcript/activity recording, A-leg route-override snapshotting and other user-turn semantics. + +`auxiliary.Request.Visibility` is currently only lineage metadata; it does not suppress those primary session effects. A continuity extractor must therefore use a typed internal execution/session mode that says: + +- preserve authenticated principal/scope; +- preserve parent IDs only as correlation; +- do not begin/append a primary secure-session turn; +- do not update primary last activity/turn count/transcript; +- do not apply the primary A-leg routing override as child route authority. + +This is execution metadata, not provider-visible prompt content. + +### Independent routing fits the canonical child call + +`lipapi.Call.Route.Selector` already expresses route intent. The extractor can set an operator-configured selector on its child call and then use the normal core planner, aliases, failover, capability checks and admission. + +This allows, for example: + +```text +primary: anthropic:claude-frontier-coding +extractor: openai-responses:small-fast-model +``` + +without any provider-specific code. Parent route overrides remain primary-session authority only. An explicit `inherit` option can be supported, but accidental inheritance is a bad default because it defeats the cost-control purpose of a dedicated extractor route. + +### Existing billing naturally supports originating-user attribution + +The authoritative billing identity adapter derives the account from `scope.PrincipalID`. Auxiliary execution already clones scope from the parent. If detached mode preserves principal/scope, the child can pass through the normal Executor and receive: + +- normal usage/concurrency admission; +- its own BillingCallID; +- its own B-leg attempts/failover; +- normal operational exposure and post-usage customer settlement; +- normal provider COGS processing. + +No new money path is necessary. The missing detail is classification: accounting/diagnostics should project a bounded `compaction_continuity_extractor` workload/origin so users/operators can distinguish auxiliary continuity cost from primary inference. That is metadata, not a new ledger/rating authority. + +Primary response protocol usage must remain the primary call's usage only. Account totals include both independent calls. + +### Process ExtensionState is useful but not restart-durable + +`ProcessServices.ExtensionState` is an in-memory process-owned store. `ScopeSession` partitions by `SessionView.PartitionKey()`, which chooses authoritative SessionID when available. This is suitable for: + +- current capsule revision; +- high-watermarks; +- bounded sanitized source window; +- pending background job ID; +- pending/injected revision watermarks. + +The feature key must additionally include authoritative A-leg/branch identity because session partition alone is not a branch key. + +This state survives generation reload, but not process restart. The spec should say so. When secure-session transcript capture is already enabled, a narrow authorized transcript reader may reconstruct missing capsule state. Otherwise restart/resume is fail-open; this feature does not justify a second durable transcript or general durable plugin-state platform. + +### Native compaction is often opaque + +`lipapi.CompactionItem` carries: + +- `EncapsulatedID`; +- `Dialect` / `Implementor`; +- `EncryptedContent`; +- `Opaque` provider data. + +There is no universal plaintext “summary” property. Editing encrypted/opaque state could break replay or provider protocol semantics. + +Therefore: + +- result-side augmentation is allowed only when the path exposes a verified mutable plaintext summary/continuation carrier; +- otherwise the validated capsule remains proxy-owned state and is injected into the first eligible post-compaction request; +- opaque/encrypted content is an exact-preservation invariant. + +## Decision D1 — Treat #312 as a hard prerequisite and share one recognizer + +Implementation should first land `compaction-event-detection` (or implement it in chronological dependency order before this feature). The detector's rule table remains the only compaction signature authority. + +Continuity may need a pure request **preview** so the first post-compaction turn can be protected before B-leg Open. That preview should be factored from the same matcher/fingerprint logic but must not mutate detector state or emit lifecycle events. Committed start/completion still follows #312 semantics. + +## Decision D2 — Add a separate compaction preservation interceptor + +Do not mutate `compaction.Observer`. + +Add a distinct additive FeatureBundle contribution (name indicative): + +```go +type Preserver interface { + ID() string + BeforeRequest(ctx context.Context, call *lipapi.Call, meta RequestMeta, preview RequestPreview, svc Services) error + RequestOpened(ctx context.Context, call lipapi.Call, meta OpenMeta, derived []Event, svc Services) error + BeforeResponseRelease(ctx context.Context, ev *lipapi.Event, meta ResponseMeta, derived []Event, svc Services) error +} +``` + +Exact method names may differ, but the semantic stages are deliberate: + +- **BeforeRequest:** pre-open only; check pending reinjection, protect completion-only first turn, never emit detector truth. +- **RequestOpened:** successful-open only; commit sanitized source and start one background semantic job for a real strict compaction transaction. +- **BeforeResponseRelease:** final selected event; bounded join and verified plaintext augmentation/fallback marker before metadata observers/client release. + +The slice is merged/frozen separately from `CompactionObservers`. + +## Decision D3 — Add a narrow process-owned background auxiliary collector + +Add an additive auxiliary capability rather than expanding `Client` into a task framework. Conceptually: + +```go +type JobID string + +type BackgroundClient interface { + SubmitCollect(ctx context.Context, req Request, opts SubmitOptions) (JobID, error) + Await(ctx context.Context, id JobID) (lipapi.Collected, error) + Forget(id JobID) +} +``` + +`SubmitCollect` must synchronously: + +1. validate/copy the canonical child request; +2. resolve the current Executor runner; +3. retain `genpin.KindAsync` while the request still has spawn authority; +4. clone only required principal/scope/correlation context; +5. enqueue into a bounded process scheduler. + +Workers execute with a scheduler-rooted context and configured timeout. A job registry supports coalescing and a bounded await barrier. Raw results have a short TTL and are forgotten after parse/validation. ProcessServices owns shutdown/cancel/join. + +This capability remains narrowly about background auxiliary model collection. It is not a durable queue, cron service, arbitrary function executor, or external workflow engine. + +## Decision D4 — Use a typed detached auxiliary execution mode + +Extend internal auxiliary request/execution policy with a typed detached-session setting (exact enum/name may differ). It shall: + +- retain user principal/scope for billing/security; +- mark internal origin and parent correlation; +- suppress primary secure-session BeginTurn/activity/transcript effects; +- avoid primary A-leg route-override authority; +- keep the child private from client-visible history; +- still use ordinary route planning, B2BUA attempt lineage, usage, billing and streaming internally. + +Do not use a magic string in `Call.Extensions` as authority and do not create a hidden primary-session user turn. + +## Decision D5 — Make the extractor route explicit and immutable per job + +Feature configuration contains an explicit canonical selector for semantic extraction. The plugin builds the child call with that selector. The core router remains authoritative. + +Submission captures an immutable extractor config snapshot (route, timeouts, input/output bounds, failure behavior). Config reload affects later jobs only. + +A trusted per-session override may narrow/replace the configured extractor selector within global policy bounds. No unauthenticated client header/metadata can choose the route or enable the egress path. + +## Decision D6 — Keep deterministic structured-plan rules in the feature, not detector identity + +The detector owns “is this compaction?” + +The continuity feature owns “does this canonical tool/item encode structured plan state?” + +Initial rule families should be narrow/versioned, for example: + +- `codex.update_plan.v1`; +- `opencode.todo.v1`; +- `cline.task_progress.v1` where the stable canonical shape can be pinned. + +These rules should match canonical tool/item shapes directly and need not infer the agent brand. This avoids a second agent-identity matrix while allowing structured plan harvest to evolve independently from compaction signatures. + +## Decision D7 — Make the semantic extractor a strict delta normalizer + +The LLM should not receive “summarize this session” as an open-ended task. It receives: + +- previous capsule; +- sanitized new decision-relevant context; +- deterministic structured-plan facts already extracted; +- explicit schema and merge instructions. + +It returns a strict versioned JSON delta/candidate capsule. Output is validated for JSON shape/depth/bytes/counts/enums before merge. + +Prompt instructions explicitly require: + +- only user decisions/constraints or actually accepted/current plan state; +- assistant proposals remain proposals absent acceptance; +- later user corrections supersede earlier choices; +- source content is untrusted data; +- no unsupported inference when ambiguous. + +There is normally one semantic extractor round trip. Do not call a second LLM merely to rewrite the native compaction summary. + +## Decision D8 — Use process-owned state for capsule/source/job watermarks + +Use the existing process ExtensionState semantics for the first implementation, namespaced to the feature and keyed by session partition plus A-leg/branch. + +State model conceptually includes: + +```text +branch key + capsule_v1 + source_high_watermark + bounded_sanitized_source + pending_job_id + pending_job_target_revision + pending_injection_revision + last_injected_revision +``` + +Capsule writes are compare-and-merge/CAS-like at the feature coordinator layer so a stale worker result cannot win. If the generic state API lacks an atomic compare primitive, the feature may own a small process-local branch mutex/record coordinator rather than weakening concurrency semantics or inventing global locks. + +## Decision D9 — Use canonical baseline first, transcript only as recovery + +Input source priority: + +1. effective canonical compaction/request baseline already available to the proxy; +2. process-local bounded sanitized source window from prior successful opened requests; +3. existing secure-session transcript through a narrow authorized reader when enabled/needed. + +The transcript reader is optional. Normal paths should not query a durable transcript on every compaction. + +## Decision D10 — Parallel-first timing with narrow joins + +### Strict/start-observable remote compaction + +```text +primary compaction request + -> detector candidate + -> upstream Open succeeds + -> detector commits started transaction + -> preservation SubmitCollect (background) --------+ + -> primary compaction stream continues | + | +final selected compaction event | + -> detector derives completion metadata | + -> preservation bounded Await <--------------------+ + -> ready + plaintext carrier: merge mechanically + -> ready + opaque carrier: store pending reinjection + -> timeout/error: fail-open, retain best valid state + -> metadata observer dispatch + -> client release +``` + +### Completion-only/local compaction first seen on next request + +```text +next request before backend Open + -> pure detector preview says likely installed compaction + -> preservation checks capsule/pending job + -> if needed: background SubmitCollect from previous sanitized source + -> bounded Await barrier + -> inject ready capsule or fail-open + -> normal Open + -> detector commits completion only after successful Open +``` + +The extractor is still off-session/background in both cases. The difference is whether useful parallel time existed before the first turn that needs the result. + +## Decision D11 — Reinjection is the universal safe fallback + +When native compaction content is opaque, store the capsule and inject a bounded proxy-owned continuity block into the first eligible post-compaction request. + +Injection must respect canonical authority: + +- message-authoritative call -> proxy-owned instruction/developer/system message representation allowed by current canonical contract; +- item-authoritative call -> canonical message item representation, not simultaneous legacy `Instructions`/`Messages` that would violate `Call.Validate`. + +Use one helper that preserves call validity rather than protocol-specific encoders. + +A revision watermark prevents duplicate injection. + +## Decision D12 — User billing is normal billing, not special charging code + +The detached child retains the original `scope.PrincipalID`, so ordinary billing resolves the same account. The child gets a separate BillingCallID and per-B-leg records. + +Add only a content-free workload/origin projection such as: + +```text +workload = auxiliary +aux_role = compaction_continuity_extractor +parent_session_id_hash / parent_trace correlation as allowed +``` + +This lets reports distinguish continuity overhead while the money ledger/journal remains unchanged. + +If the child is denied before provider submission, preservation fails open by default. If provider work was submitted, usage remains billable even if the result is discarded later. + +## Decision D13 — Global config plus trusted per-session narrowing/override + +Conceptual feature config: + +```yaml +# exact nesting follows standard feature-plugin opaque config conventions +compaction_continuity: + enabled: false + preserve: + plan: true + user_decisions: true + constraints: true + rationale: true + rejected_alternatives: true + extractor: + route: "openai-responses:small-model" + timeout: 8s + max_input_tokens: 12000 + max_output_tokens: 2000 + max_concurrency: 2 + queue_capacity: 16 + result_ttl: 30s + barrier_timeout: 2s + max_capsule_tokens: 2500 + source_ttl: 2h + failure_mode: fail_open +``` + +Values above are examples, not normative defaults. Implementation tests should pin chosen defaults/validation. + +Per-session overrides come only from trusted proxy-owned policy/session state and cannot exceed global egress/resource maxima unless explicitly authorized. + +## Decision D14 — Do not claim restart durability in v1 + +The first implementation guarantees continuity across compaction and generation reload **within the process**. It does not create a new durable capsule database. + +If an authorized secure-session transcript exists after restart, the capsule can be reconstructed on demand. Otherwise the feature resumes without prior process-only capsule state and fails open. This is truthful and materially simpler than introducing a new durable state subsystem for a UX enhancement. + +## Rejected Alternatives + +### Run extractor inline on the main request goroutine + +Rejected. It violates the off-session/background requirement and adds full extractor latency directly before native compaction. + +### Fire-and-forget goroutine calling current `Aux.Collect` + +Rejected. It can lose generation spawn rights, inherit canceled request lifetime, leak on shutdown, and has no bounded admission/result ownership. + +### Direct provider/HTTP client for the extractor + +Rejected. It would bypass routing, principal scope, usage authority, billing, B2BUA attempts, retries and current provider abstractions. + +### Put extractor call into the primary session as another assistant/user turn + +Rejected. It pollutes conversation state, can influence agent behavior, changes transcript/turn semantics and makes auxiliary billing indistinguishable from primary inference. + +### Append text to every `CompactionItem` + +Rejected. Native compaction may be encrypted/opaque provider state and does not expose a universal safe text carrier. + +### Second LLM call to rewrite/improve native summary + +Rejected by default. The validated continuity capsule is already the semantic result; deterministic merge/reinjection is cheaper and more reliable. + +### Re-send the entire raw session on every compaction + +Rejected. It creates O(session age) cost, unnecessary privacy egress and prompt-injection surface. Use delta + prior capsule + bounded recovery sources. + +### Build a second durable transcript/capsule/job database now + +Rejected. Existing secure transcript can be used when authorized; process state is sufficient for the primary compaction UX objective. A general durable job/state system is disproportionate. + +### Use the #312 history heuristic as semantic truth about accepted plans + +Rejected. Compaction detection and user-decision semantics are different problems. Heuristics may decide when to call the semantic extractor, not manufacture user decisions. + +## Main Risks and Mitigations + +| Risk | Mitigation | +|---|---| +| semantic extractor hallucinates acceptance | strict schema/provenance, deterministic merge, explicit-user precedence, ambiguity omission | +| extra user cost | deterministic-first/eligibility gate, independently cheap route, visible auxiliary cost classification | +| worker leaks across reload/shutdown | ProcessServices ownership, submit-time KindAsync pin, bounded scheduler, race/goleak tests | +| extractor changes primary session state | typed detached-session execution mode and session/transcript regression tests | +| stale job overwrites correction | branch revision/high-watermark compare-and-merge | +| opaque compaction corrupted | exact byte-preservation invariant; reinjection fallback | +| first post-compaction turn races job | pre-open pure preview + bounded Await barrier | +| privacy egress of tool/file dumps | canonical sanitizer, redaction, excluded external/tool data by default | +| duplicate billable jobs on retry | transaction/revision coalescing key | +| config reload changes running job unexpectedly | immutable submission snapshot + retained generation | +| restart loses process capsule | truthful v1 contract; authorized transcript reconstruction only when available | + +## Research Conclusion + +The feature has high UX value and fits Go-LIP's current architecture if implemented as **structured process continuity + a narrow background auxiliary capability**, not as a generic memory subsystem or compactor rewrite. The highest-risk brownfield work is not the LLM prompt; it is lifecycle/session/accounting correctness around a detached asynchronous child call. The design and tasks should therefore put RED tests around those boundaries before implementing semantic extraction details. From 9b79f596a3eedda7a8fb47a7e551e3615472c783 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:06:36 +0200 Subject: [PATCH 07/18] spec: design compaction continuity preservation --- .../design.md | 866 ++++++++++++++++++ 1 file changed, 866 insertions(+) create mode 100644 .kiro/specs/compaction-continuity-preservation/design.md diff --git a/.kiro/specs/compaction-continuity-preservation/design.md b/.kiro/specs/compaction-continuity-preservation/design.md new file mode 100644 index 00000000..82174332 --- /dev/null +++ b/.kiro/specs/compaction-continuity-preservation/design.md @@ -0,0 +1,866 @@ +# Design Document + +## Overview + +Implement issue #344 as a bounded compaction-continuity feature that preserves decision state without replacing coding-agent compaction or creating a parallel inference/billing stack. + +The design has seven cooperating pieces: + +1. the existing `compaction-event-detection` detector/rule authority, implemented first and refactored only enough to provide a pure request preview; +2. a separate additive `pkg/lipsdk/compaction.Preserver` content-bearing extension surface; +3. a process-owned bounded background auxiliary collector under `internal/core/auxreq`; +4. a typed detached auxiliary execution/session mode that preserves principal/billing identity but not primary session-turn effects; +5. a process-owned branch coordinator that serializes revision/job/injection state across runtime generations; +6. an official `internal/plugins/features/compactioncontinuity` implementation for capsule schema, deterministic plan carriers, sanitizer, extractor prompt/schema, merge and injection; +7. existing routing, B2BUA, usage/metering and BillingCallID accounting reused unchanged as execution authorities. + +The semantic extractor runs independently from the primary coding turn and may use a completely different model/provider. It is nevertheless a normal canonical Go-LIP child call, so the originating user is billed for actual extractor usage by default. + +## Goals + +- Preserve accepted/current plan state and explicit user product/architecture decisions across repeated lossy compactions. +- Make structured planning state deterministic and LLM-free where possible. +- Run semantic extraction off-session and in background, with bounded worker/resource ownership. +- Allow a separately configured cheap/fast extractor route independent of the primary session route. +- Attribute auxiliary usage/cost to the originating authenticated user through existing billing authorities. +- Prevent the first post-compaction turn from outrunning required continuity state. +- Keep provider-native encrypted/opaque compaction content byte-identical. +- Keep the feature bounded, fail-open and observable without logging decision content. + +## Non-Goals + +- general long-term/RAG memory; +- storing every conversation fact; +- replacing native/agent compaction; +- a general durable job framework; +- a second transcript database; +- a second financial ledger/rating engine; +- direct provider clients for the extractor; +- mutating `compaction.Observer`; +- provider-specific continuity branches inside core; +- transparent restart durability when no durable authorized source exists. + +## Dependency and Implementation Order + +The runtime implementation of `.kiro/specs/compaction-event-detection/` is a prerequisite. + +Chronological implementation order is therefore: + +```text +compaction-event-detection runtime + | + +--> pure request preview extracted from same matcher/fingerprint authority + | + v +compaction-continuity-preservation +``` + +Continuity implementation must not recreate the #312 agent-signature table as a temporary fallback. + +## Existing Architecture Reused + +- `lipapi.Call`, `NormalizedItems` and canonical item/tool/message forms provide provider-neutral source data. +- `lipapi.OperationContextCompaction` / `ItemKindCompaction` provide canonical strict semantics where supported. +- `internal/core/compactiondetect` (from prerequisite) owns compaction evidence, rule IDs, A-leg transactions and history heuristic. +- `pkg/lipsdk/feature.FeatureBundle` / `internal/featurebundle` / `extensions.RequestRuntimeSnapshot` provide additive extension composition. +- `internal/core/auxreq` already delegates auxiliary calls through the normal Executor, clones principal/scope, marks internal origin and supports plugin suppression. +- `pkg/lipsdk/genpin` already defines `KindAsync` for request-spawned asynchronous generation ownership. +- `ProcessServices` owns process lifetime and `ExtensionState` across generation reloads. +- secure-session recording already provides an optional durable transcript when explicitly enabled. +- billing account identity already comes from authenticated principal scope; child calls naturally get independent BillingCallIDs. + +## Target Architecture + +```text + PRIMARY USER TURN / COMPACTION PROCESS-OWNED AUXILIARY PLANE + + canonical request + | + v + shared detector PreviewRequest (pure) -----> preservation BeforeRequest + | | pending capsule/job? + | | bounded Await if required + | +--> canonical reinjection + v + normal prepare / route / billing / Open + | + +-- Open succeeds ---------------------> detector RequestOpened / transaction + | | + | +--> Preserver.RequestOpened + | | + | +--> deterministic harvest + | +--> sanitize delta + | +--> BackgroundAux.SubmitCollect + | | + | v + | bounded worker pool + | | + | detached child Executor + | | + | independent route/model + | | + | normal usage/billing + | | + | strict JSON result + | | + | validate + branch merge + | + primary compaction stream proceeds concurrently + | + final selected event + | + detector derives completion metadata (no mutation) + | + preservation BeforeResponseRelease + |---- bounded Await if useful ----------------------------+ + |---- verified plaintext carrier? -> mechanical augment + |---- opaque/late? -> mark pending reinjection + v + compaction metadata observers + v + client release +``` + +## D1. Shared Detector Preview Versus Committed Detection + +The prerequisite detector currently has committed request semantics conceptually equivalent to: + +```go +func (d *Detector) RequestOpened(meta RequestMeta, call lipapi.Call) []compaction.Event +``` + +Add a pure preview API or internal equivalent: + +```go +type RequestPreview struct { + Evidence compaction.Evidence + RuleID string + Kind PreviewKind // none | start_candidate | completion_candidate + // Opaque deterministic fingerprint/token for revalidation; no raw content. +} + +func (d *Detector) PreviewRequest(meta RequestMeta, call lipapi.Call) RequestPreview +``` + +Properties: + +- same rule/fingerprint implementation as committed detection; +- read-only with respect to transaction/fingerprint state; +- emits no observer event; +- cannot be used as proof that an upstream compaction started; +- useful only to let `BeforeRequest` identify a completion-only/installed-compaction boundary before opening the first post-compaction B-leg. + +`RequestOpened` remains the only request-side state-commit point and remains after successful `Open`. + +## D2. Public Preservation Extension Contract + +Extend the `pkg/lipsdk/compaction` package introduced by the prerequisite with a distinct preservation surface. Exact names may change, but responsibilities shall remain separate. + +Conceptual contract: + +```go +type PreservationMeta struct { + TraceID string + SessionID string + ALegID string + BLegID string + AttemptSeq int + TransactionID string + RuleID string + Evidence Evidence +} + +type Services struct { + State ContinuityState + BackgroundAux auxiliary.BackgroundClient +} + +type Preserver interface { + ID() string + BeforeRequest( + context.Context, + *lipapi.Call, + RequestPreview, + PreservationMeta, + Services, + ) error + + RequestOpened( + context.Context, + lipapi.Call, + []Event, + PreservationMeta, + Services, + ) error + + BeforeResponseRelease( + context.Context, + *lipapi.Event, + []Event, + PreservationMeta, + Services, + ) error +} +``` + +`BeforeRequest` may mutate only the canonical request through the documented continuity injection path. `RequestOpened` is observation/source/job scheduling after successful Open. `BeforeResponseRelease` may mutate only a verified plaintext continuation carrier. + +`FeatureBundle` gains a separate optional `CompactionPreservers []compaction.Preserver` slice. The single merge surface concatenates it in registration order and the request snapshot exposes a frozen defensive copy. + +Do **not** add request/response content to `compaction.Event` or change `Observer` into a decision/mutation interface. + +## D3. Process-Owned Background Auxiliary Collector + +### SDK surface + +Keep synchronous `auxiliary.Client` source-compatible. Add an additive interface: + +```go +type JobID string + +type SubmitOptions struct { + CoalesceKey string + Timeout time.Duration +} + +type BackgroundClient interface { + SubmitCollect(ctx context.Context, req Request, opts SubmitOptions) (JobID, error) + Await(ctx context.Context, id JobID) (lipapi.Collected, error) + Forget(id JobID) +} +``` + +`DisabledBackgroundClient` provides explicit not-configured behavior. + +### Core implementation + +Create a focused scheduler under `internal/core/auxreq` (names indicative): + +```text +BackgroundScheduler + fixed/bounded worker count + bounded input queue + keyed coalescing map + bounded result map + TTL + process root context + closeOnce + WaitGroup +``` + +It executes canonical auxiliary calls only; it does not accept arbitrary Go callbacks/functions. + +### Submit-time ownership transfer + +`SubmitCollect` executes synchronously until ownership is safely transferred: + +1. validate cloneable child call and options; +2. resolve current generation's `ExecutorRunner` through the generation snapshot cell; +3. obtain request `genpin.Retainer` and `Retain(genpin.KindAsync)`; +4. clone required principal/scope/correlation context into an immutable job envelope; +5. reserve/coalesce the job in the process scheduler; +6. enqueue or fail atomically; +7. on enqueue failure, release the newly retained pin immediately. + +A worker never attempts to acquire a fresh generation spawn right later. The captured pin is released exactly once after terminal child execution/collection or cancellation. + +### Job context + +Worker context derives from the scheduler/process root plus copied safe execution attribution, not from the parent request's cancellation tree. Per-job timeout then bounds model work. + +Preserve: + +- principal/scope; +- internal origin; +- parent trace/session/A-leg correlation; +- submission-time generation identity. + +Do not preserve: + +- parent cancellation lifetime as root; +- primary secure-session authority token/turn authority; +- primary A-leg route override authority. + +### Result lifetime + +Raw `lipapi.Collected` output is retained only until awaited/parsed, with explicit count/byte/TTL limits. `Forget` or expiry deletes it. Logs/metrics expose only job ID hash/status/size/token metadata. + +## D4. Detached Auxiliary Session Mode + +Current auxiliary execution delegates through the normal Executor, whose standard path performs secure-session BeginTurn and primary request authority work. Add a typed execution policy to `auxiliary.Request` or an internal sibling type: + +```go +type SessionMode uint8 +const ( + SessionModeNormal SessionMode = iota + SessionModeDetached +) +``` + +The exact enum location may differ, but `Detached` is core-interpreted execution metadata and never provider-visible call content. + +Detached semantics: + +- authenticated principal/scope is preserved; +- parent IDs remain correlation metadata; +- no primary secure-session BeginTurn/new TurnID; +- no primary transcript/activity/turn-count mutation; +- no primary resume-token/session-authority propagation; +- no primary A-leg route-override lookup/application; +- no client-visible session headers/history; +- child still receives normal independent trace, route plan, B-legs, usage, metering and billing. + +The implementation should factor the Executor prepare path rather than copy the whole executor. For detached calls, execute only the identity/scope + ordinary execution authorities that remain relevant. + +Security invariant: `SessionModeDetached` may only originate from trusted internal auxiliary code; frontends cannot request it from wire fields. + +## D5. Independent Extractor Routing + +Feature config requires either: + +- explicit `extractor.route`, or +- explicit `extractor.route_policy: inherit`. + +Default should not accidentally inherit the main route. + +The extractor builds a canonical child call with its own `Route.Selector`. Normal routing then owns aliases, selector grammar, safe execution-composition policy, health, retry/failover and B-leg opening. + +Parent A-leg override is not consulted in detached mode. + +The child is stamped: + +```text +Role = "compaction_continuity_extractor" +Visibility = "private" +Origin = internal auxiliary +``` + +Tools are empty/disabled and tool choice is none/valid equivalent. + +## D6. Originating-User Billing and Workload Classification + +The child enters the normal Executor with cloned authenticated `scope`. Existing billing identity therefore resolves the same customer account. + +Every extraction gets: + +- separate BillingCallID; +- separate B-leg attempt sequence; +- separate usage records; +- separate operational exposure/customer settlement; +- separate provider COGS. + +No special charging function is added. + +Add a bounded classification to execution/accounting evidence, conceptually: + +```go +type WorkloadClass string +const ( + WorkloadPrimary WorkloadClass = "primary" + WorkloadAuxiliary WorkloadClass = "auxiliary" +) + +AuxRole = "compaction_continuity_extractor" +``` + +Prefer reusing existing auxiliary lineage metadata as the source and project it into metering/billing/report DTOs where needed. Do not put raw capsule/source text into money records. + +Primary frontend usage projection remains tied to the primary BillingCall/response only. Aggregate account/operator reports include both calls and can group auxiliary cost. + +If the child is rejected before upstream submission, the feature records a fail-open extraction outcome and does not bypass admission. If provider work was submitted, later invalid/stale extraction does not erase actual usage/cost. + +## D7. Process-Owned Continuity Branch Coordinator + +A per-generation feature mutex is insufficient because immutable generation reload can overlap with in-flight workers/turns. `ExtensionState` is process-owned but its current `Get` + `Put` API has no atomic compare-and-swap. + +Create a **small process-owned continuity coordinator** in `internal/core/compactioncontinuity` (name indicative) and build it in `ProcessServices` when the capability is enabled/available. It is not a general plugin framework. + +Responsibilities: + +- serialize updates per authoritative branch key; +- use process `ExtensionState` as backing storage for serializable state; +- provide revision-checked load/update operations to preservation services; +- track pending job IDs and injection watermarks; +- enforce max entries/TTL with lazy/opportunistic cleanup; +- contain no extractor prompts, provider code or billing logic. + +Conceptual branch key: + +```go +type BranchKey struct { + AuthoritativeSessionID string // when available + ALegID string + PrincipalPartition string // used when no secure session id +} +``` + +Client session hints do not participate as authority. + +Conceptual state: + +```go +type BranchState struct { + Revision uint64 + CapsuleJSON json.RawMessage + CapsuleDigest [32]byte + SourceHighWatermark string + SanitizedSourceJSON json.RawMessage + PendingJobID auxiliary.JobID + PendingJobTargetRevision uint64 + PendingInjectionRevision uint64 + LastInjectedRevision uint64 + LastCompactionTransaction string + UpdatedAt time.Time +} +``` + +The coordinator can expose opaque capsule/source JSON to the plugin so core does not own semantic fact interpretation. + +## D8. Continuity Capsule Schema + +Feature-owned v1 model (exact field names may evolve under tests): + +```json +{ + "schema_version": 1, + "revision": 12, + "source_high_watermark": "...", + "plan": { + "status": "accepted", + "source": "structured|user_acceptance|semantic", + "steps": [ + {"id":"...","text":"...","status":"pending","source_ref":"..."} + ] + }, + "decisions": [ + { + "id":"...", + "statement":"...", + "status":"active", + "authority":"user_explicit|user_acceptance|semantic", + "rationale":"...", + "source_ref":"..." + } + ], + "constraints": [], + "rejected_alternatives": [], + "open_questions": [] +} +``` + +### Merge precedence + +Highest authority / latest source wins: + +1. later explicit user correction/decision; +2. later explicit user acceptance/selection; +3. authoritative deterministic structured-plan update; +4. validated semantic inference; +5. older capsule state. + +Semantic inference may propose new facts/status transitions but cannot resurrect a superseded/rejected fact against newer explicit user evidence. + +Fact IDs are stable across revisions. Existing IDs supplied in the previous capsule should be reused by the extractor when updating a fact. New deterministic carrier facts get deterministic IDs from type + canonical normalized content/source; new semantic facts get normalized content/category IDs only after validation. + +### Bounded retention + +When over budget, deterministic pruning order is: + +1. preserve active constraints/decisions; +2. preserve pending/in-progress plan steps; +3. preserve associated rationale needed for those facts; +4. preserve meaningful current rejections; +5. condense completed plan history; +6. drop oldest superseded/irrelevant history. + +Never truncate JSON in the middle; prune complete facts and revalidate. + +## D9. Deterministic Plan Carrier Catalog + +Place semantic carrier parsing in `internal/plugins/features/compactioncontinuity`, not provider/core adapters. + +Static versioned table with explicit canonical matchers: + +```text +codex.update_plan.v1 +opencode.todo.v1 +cline.task_progress.v1 +``` + +Implementation research must pin actual canonical tool/item shapes from supported harness snapshots before enabling each rule. A rule requires table-driven positive and near-miss tests. + +No generic “markdown checklist = accepted plan” rule. + +The detector's compaction RuleID may be recorded for diagnostics, but carrier matching should work from canonical tool shapes rather than infer a provider brand. + +## D10. Sanitized Extraction Source + +Source builder walks the effective canonical call and produces a bounded structured envelope, not a raw wire dump. + +Priority: + +1. user messages/text relevant to decisions; +2. assistant plan/proposal/clarification text needed to interpret user replies; +3. recognized structured plan/TODO carriers; +4. previous capsule. + +Default drops: + +- ordinary tool results; +- command output; +- compiler/test logs; +- large source/code/file dumps; +- images/video/binary/file blobs; +- reasoning payloads not needed for explicit user decisions; +- unrelated external text. + +Tool/external content that must be included is tagged `untrusted_external` and delimited. + +Size policy should use deterministic bytes/estimated tokens and prefer tail + decision-bearing user turns rather than arbitrary truncation. + +Every successfully opened primary request may refresh the process-local **sanitized** source window/high-watermark. Failed pre-open requests never become committed source state. + +## D11. Semantic Extractor Child Call + +Build one canonical call with: + +- configured independent route; +- fixed system/developer instructions describing the extraction task; +- previous capsule JSON; +- deterministic plan facts; +- sanitized context delta; +- no tools; +- bounded max output; +- preservation plugin disabled; +- detached session mode; +- role `compaction_continuity_extractor`. + +Expected output is one strict JSON object, e.g.: + +```json +{ + "schema_version": 1, + "base_revision": 11, + "facts": [...], + "plan_updates": [...], + "remove_or_supersede": [...] +} +``` + +Validation happens before BranchCoordinator merge: + +- JSON only / one top-level object; +- schema version exact; +- base revision expected or stale-discard path; +- byte/depth/item/count/string bounds; +- enum values exact; +- no unknown authority escalation; +- no raw tool/blob fields; +- valid source references only within supplied bounded input when required. + +Malformed output is discarded fail-open. + +## D12. Semantic Eligibility Gate + +The local gate exists to save cost, not to decide truth. + +Candidate signals may include: + +- recognized structured plan carrier changed; +- assistant substantial plan followed by user affirmative/corrective turn; +- explicit user choice/correction/constraint language; +- previous capsule absent while compaction source contains planning markers; +- new relevant user turns after capsule high-watermark. + +Near misses must avoid triggering solely on ordinary words such as `plan`, `summary`, or `continue`. + +If deterministic state fully satisfies the preservation categories for the new delta, skip the LLM job. + +## D13. Strict Remote-Compaction Flow + +### Request preview + +Before normal B-leg Open: + +- detector preview may identify a strict start candidate; +- preservation may prepare/sanitize state but **must not submit a strict-start semantic job yet**; +- existing pending reinjection from a prior completed compaction is handled here. + +### Successful Open + +After primary compaction B-leg successfully opens: + +1. detector commits `started` transaction; +2. preservation commits the sanitized request source; +3. deterministic carrier extraction updates capsule if possible; +4. eligibility gate decides whether semantic extraction adds value; +5. if yes, `BackgroundAux.SubmitCollect` submits exactly one coalesced job; +6. primary compaction stream continues without waiting. + +### Completion release + +At final selected canonical event: + +1. detector derives completion metadata without mutating the event; +2. preservation checks the matching transaction/job; +3. if a bounded Await completes and output validates, merge new capsule revision; +4. if event has a verified mutable plaintext continuation carrier, append/merge a versioned continuity block mechanically; +5. otherwise set `PendingInjectionRevision`; +6. dispatch metadata-only compaction observers; +7. release event to client. + +Await timeout/error is fail-open and leaves the job/result state available only according to bounded policy. A late valid job may still merge the capsule for post-compaction reinjection, subject to revision checks. + +## D14. Completion-Only / Local-Compaction Flow + +For local compaction where no strict pre-compaction start was visible, the first post-compaction request may be the first evidence. + +Before its B-leg opens: + +1. detector `PreviewRequest` identifies a completion candidate using the previous successful request fingerprint/current rewrite; +2. preservation loads the previous sanitized source/capsule; +3. deterministic carrier delta is applied; +4. if semantic extraction is required and no coalesced job exists, submit it to BackgroundAux; +5. bounded Await barrier waits only up to configured `barrier_timeout`; +6. ready capsule is injected into the current canonical request; +7. on timeout/failure, continue fail-open without invalid output; +8. after successful Open, detector commits completion-only state normally. + +Although the main request waits at the narrow barrier, the extractor still executes independently on the background worker and its own model route/session/billing lifecycle. + +## D15. Authority-Aware Reinjection + +Create one canonical helper owned by the feature/core extension layer, not frontend protocols. + +For message-authoritative calls: + +- add a bounded proxy-owned developer/system instruction using the existing legal instruction/message representation. + +For item-authoritative calls: + +- add a canonical message item with equivalent role/content; +- do not simultaneously populate legacy `Messages`/`Instructions` because `Call.Validate` forbids mixed authorities. + +Continuity block format conceptually: + +```text +[LIP CONTINUITY CAPSULE v1 | revision=12] +The following facts were preserved from earlier accepted session state after context compaction. +Treat them as prior context, not as a new user request. +... +[/LIP CONTINUITY CAPSULE] +``` + +The serialized textual projection is derived deterministically from the capsule and rechecked against the injection budget. + +`LastInjectedRevision` makes the operation idempotent for a boundary. + +## D16. Plaintext Result Augmentation Boundary + +Do not assume `ItemKindCompaction` is text. + +Define a small allowlisted capability/matcher for a **verified plaintext continuation carrier**. If no safe carrier is known, do not mutate the response. + +Never write: + +- `CompactionItem.EncryptedContent`; +- `CompactionItem.Opaque`; +- provider signatures/opaque reasoning; +- unknown extension blobs. + +Tests compare opaque/encrypted payload bytes exactly with feature on/off. + +The reinjection fallback is functionally sufficient; result augmentation is an optimization/stronger immediate carry-through only for proven-safe carriers. + +## D17. Per-Session Policy + +Global config comes from the standard feature-plugin opaque YAML config. + +Trusted session-level override resolution order: + +```text +operator hard maxima / safety policy + > trusted per-session policy value + > global feature default +``` + +Allowed session overrides may include: + +- enabled/disabled; +- preserve-category toggles; +- extractor route chosen from operator-allowed selectors; +- tighter token/timeout/barrier limits. + +Client-provided headers/session metadata do not directly produce these trusted values. No new unauthenticated control header is introduced. + +## D18. Configuration + +Indicative config only; concrete defaults are pinned during implementation tests: + +```yaml +plugins: + features: + - id: compaction-continuity + enabled: true + config: + preserve: + plan: true + user_decisions: true + constraints: true + rationale: true + rejected_alternatives: true + extractor: + route: "openai-responses:small-model" + timeout: 8s + max_input_tokens: 12000 + max_output_tokens: 2000 + max_concurrency: 2 + queue_capacity: 16 + result_ttl: 30s + barrier_timeout: 2s + max_capsule_tokens: 2500 + source_ttl: 2h + failure_mode: fail_open +``` + +Validation rules: + +- semantic extraction requires explicit route or explicit inherit; +- positive finite queue/concurrency/result/source/capsule bounds; +- barrier <= job timeout or explicitly documented independent behavior; +- per-session values cannot exceed hard global maxima; +- invalid config fails startup/generation compile rather than silently enabling expensive defaults. + +## D19. Failure Semantics + +Default is fail-open for model traffic. + +| Failure | Behavior | +|---|---| +| no candidate state | skip semantic call | +| background queue full | record saturation; deterministic capsule/native compaction continue | +| generation retain fails | no job; fail-open | +| child billing/admission denied | no provider bypass; deterministic state/native compaction continue | +| extractor route/provider fails | child normal retry rules; then fail-open | +| extractor timeout | job fails; no malformed injection | +| schema invalid | discard raw output; account actual submitted usage | +| job stale | discard/forget result; do not overwrite capsule | +| barrier timeout | continue native request/compaction; late valid result may merge if still current | +| capsule over budget | deterministic whole-fact pruning | +| state/coordinator unavailable | skip preservation mutation; native traffic continues | +| process shutdown | stop admission, cancel/join worker, settle normal submitted usage paths | + +A future explicit fail-closed policy may be considered separately; v1 should not make a UX preservation feature a default model-traffic availability dependency. + +## D20. Privacy and Security + +- Feature default disabled. +- Remote extractor route is explicit data egress. +- Use existing redaction/secret handling before child egress. +- Do not send full raw session automatically. +- Treat all transcript/source as quoted untrusted data. +- Child has no tools and preservation plugin is suppressed. +- Never log extractor prompt/output/capsule. +- Raw completed background result lives only in bounded TTL process memory until parse/forget. +- Preserve principal/tenant/workspace authorization for any optional transcript read. +- Do not turn transcript capture on implicitly. +- Child detached-session mode is trusted internal-only, not frontend-controlled. + +## D21. Observability + +Add content-free metrics/diagnostics: + +- compaction preview/start/completion counts by evidence/rule; +- deterministic carrier hits by versioned carrier ID; +- semantic eligibility skip reasons; +- background jobs submitted/coalesced/saturated/canceled/completed/failed/stale; +- worker queue depth/in-flight count; +- extractor route/backend/model identifiers where existing policy allows; +- extractor input/output/cache token usage and child cost via existing usage/billing; +- capsule revision, serialized size, active fact/plan-step counts; +- barrier wait count/duration/timeouts; +- result augmentation versus reinjection counts; +- stale/CAS conflict rejection; +- opaque-carrier no-mutation path count. + +Normal logs expose IDs/hashes/counts, never preserved text. + +## D22. Lifetime and Concurrency Invariants + +1. ProcessServices owns scheduler and branch coordinator. +2. Generation snapshot owns only non-owning handles to process services. +3. Each background job owns exactly one retained `KindAsync` pin after successful submission. +4. Job pin is released exactly once on every terminal/cancel/enqueue-failure path. +5. A job never obtains a new generation after its submit boundary. +6. Branch coordinator serializes revision/injection/job state across overlapping old/new generations. +7. Worker callbacks never run while coordinator internal locks are held. +8. Provider/model work never runs under a global branch-state mutex. +9. Job/coordinator maps have explicit bounds/TTL and no unbounded timer goroutines. +10. Config reload changes only future job configuration. + +## D23. Process Restart and Durable Resume + +V1 guarantees: + +- continuity across compaction within a process; +- continuity across immutable generation reload; +- no cross-branch state leakage. + +V1 does **not** guarantee capsule persistence across process restart. + +If secure-session transcript capture already exists and future/implementation scope wires an authorized transcript-source adapter, the feature may reconstruct the capsule on demand after restart. The adapter must be read-only, bounded and principal/session authorized. Otherwise the missing capsule is explicit fail-open state. + +Do not add a durable job queue or full transcript store under this spec. + +## D24. Expected Change Surface + +Primary implementation packages: + +- `pkg/lipsdk/compaction` — preservation contract alongside prerequisite observer contract; +- `pkg/lipsdk/feature` — additive preserver slice; +- `pkg/lipsdk/auxiliary` — additive BackgroundClient/job types and detached session policy carrier; +- `internal/featurebundle` / `internal/core/extensions` — merge/frozen snapshot/dispatch stages; +- `internal/core/compactiondetect` — pure preview extracted from same rule authority; +- `internal/core/auxreq` — bounded background collector and detached execution handoff; +- `internal/core/compactioncontinuity` — narrow process branch coordinator; +- `internal/core/runtime` — request-open/final-release/detached prepare wiring; +- `internal/infra/runtimebundle` — ProcessServices ownership and snapshot clients; +- `internal/plugins/features/compactioncontinuity` — capsule, carrier rules, sanitizer, eligibility, extractor, merge, injection, config; +- existing metering/billing/report DTO/projection locations only as needed for content-free auxiliary workload classification; +- focused architecture/testkit fixtures. + +Packages that should **not** need provider-specific edits: + +- individual backend providers/connectors; +- individual frontend protocol codecs; +- provider SDK adapters for compaction continuity semantics. + +## Testing Strategy + +TDD order: + +1. RED capsule merge/supersession/pruning and structured carrier fixtures. +2. RED sanitizer/eligibility/extractor-schema tests. +3. RED BackgroundAux submit-time pin/coalescing/saturation/shutdown/goleak tests. +4. RED detached-session tests proving no primary secure-session/route-override effects. +5. RED separate-route + user-billing/BillingCallID/protocol-usage tests. +6. RED compaction integration tests for successful-Open trigger, completion-only preview, parallel job, bounded barrier, plaintext/opaque behavior. +7. RED three-plus repeated-compaction/concurrency/generation-reload tests. +8. GREEN minimal implementation in the same order. +9. Repository quality, race, architecture and simplification gates. + +No real model credentials are required. Extractor behavior uses deterministic fake/local backends returning fixture JSON. + +## Design Invariants + +1. One compaction recognition authority; no duplicate signature matrix. +2. Metadata observer remains non-mutating and content-free. +3. Extractor is off the primary session but billed to the originating user by default. +4. Extractor route is explicit and independent unless explicit inherit is configured. +5. Background job ownership is captured before the request spawn right ends. +6. ProcessServices, not a generation plugin lifecycle, owns worker lifetime. +7. Auxiliary child does not create/modify a primary secure-session turn. +8. Normal routing/B2BUA/usage/billing own child execution; no direct provider path. +9. Primary protocol-visible usage excludes extractor usage; account totals include it. +10. Continuity state is bounded, revisioned, branch-scoped and generation-reload-safe. +11. Opaque/encrypted compaction bytes are immutable. +12. First post-compaction turn has a bounded synchronization barrier when required. +13. No second transcript DB, money ledger, generic workflow engine, or redundant LLM summary pass. From 72042c8d30efc4cf891504464b8ba1a7506e0758 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:08:11 +0200 Subject: [PATCH 08/18] spec: validate compaction continuity brownfield design --- .../design-review.md | 302 ++++++++++++++++++ 1 file changed, 302 insertions(+) create mode 100644 .kiro/specs/compaction-continuity-preservation/design-review.md diff --git a/.kiro/specs/compaction-continuity-preservation/design-review.md b/.kiro/specs/compaction-continuity-preservation/design-review.md new file mode 100644 index 00000000..ba717879 --- /dev/null +++ b/.kiro/specs/compaction-continuity-preservation/design-review.md @@ -0,0 +1,302 @@ +# Brownfield Design Validation + +## Verdict + +**GO after correction loop.** The design fits current Go-LIP ownership boundaries and reuses existing routing, auxiliary execution, generation pins, B2BUA, secure-session, extension and billing authorities. Initial design review identified four issues that must be corrected before task generation: final-release detector ordering, private child A-leg semantics, useful late-result retention, and explicit prerequisite/fail-closed composition when the #312 detector surface is unavailable. + +No provider-specific implementation, second transcript database, second billing path, or generic workflow engine is required. + +## Validation Inputs + +Reviewed against current `main` and steering constraints, including: + +- `.kiro/specs/compaction-event-detection/{requirements,research,design,tasks}.md`; +- `pkg/lipsdk/auxiliary/client.go`; +- `internal/core/auxreq/client.go`; +- `pkg/lipsdk/genpin/pin.go`; +- `pkg/lipapi/{call,items}.go`; +- `pkg/lipsdk/feature/bundle.go` and `internal/featurebundle/merge_surface.go`; +- `internal/core/extensions` / `internal/infra/runtimebundle/build_extension.go`; +- `internal/infra/runtimebundle/{process_services,shared_mutable}.go`; +- `pkg/lipsdk/session/{view,opener}.go`; +- `internal/core/runtime/executor_prepare_secure.go`; +- `internal/core/state/{partition,mem}.go`; +- `internal/infra/billingcompose/identity.go`; +- `.kiro/steering/{structure,testing,routing-and-orchestration}.md`. + +## Finding V1 — Final detector observation must still see the actually released event + +### Initial concern + +The first design ordering said: + +```text +selected event + -> detector derives/commits completion + -> preservation mutates verified plaintext carrier + -> observer dispatch + -> client +``` + +The #312 invariant is stronger: `ResponseReleased` represents the canonical event actually released after final mutation/gating. Calling committed detector observation before preservation means it technically observes a pre-final form. + +### Correction + +Add a pure `PreviewResponse` (or equivalent shared matcher preview) analogous to request preview: + +```text +selected event + -> detector PreviewResponse (pure, no state/event emission) + -> preservation BeforeResponseRelease (bounded join/mutation) + -> detector ResponseReleased(final event) (commits completion) + -> metadata compaction observer dispatch + -> client release +``` + +`PreviewResponse` may expose the active transaction/rule candidate required to find the preservation job, but it cannot mark completion. This keeps the detector non-mutating and truthful to the final-release contract. + +### Status + +**Accepted; requirements/design must be corrected.** + +## Finding V2 — Detached child needs its own private A-leg, not no A-leg and not the parent A-leg + +### Initial concern + +Normal runtime authority/billing/attempt mechanics assume a logical A-leg. Reusing the parent A-leg would risk route-override/session-state coupling. Running completely without an A-leg would require a second execution path and weaken current B2BUA/request-authority assumptions. + +### Correction + +Detached auxiliary execution creates/touches a **private child A-leg** for the child canonical call using existing B2BUA lifecycle/store semantics. Parent IDs stay explicit lineage only: + +```text +primary SessionID / A-leg A123 + | + +-- auxiliary lineage parent_a_leg=A123 + | + +-- child A-leg AUX456 + +-- B-leg 1 extractor backend/model + +-- B-leg 2 failover if needed +``` + +Rules: + +- child A-leg gets no primary secure-session ownership/resume semantics; +- parent route override is never read as child authority; +- child gets normal request authority/BillingCallID/B-leg accounting; +- child A-leg/attempts remain distinguishable as internal auxiliary workload; +- client-visible primary transcript/turn/activity remains unchanged. + +### Status + +**Accepted; requirements/design must be corrected.** + +## Finding V3 — A short raw-result TTL can lose a valid background extraction before the next turn + +### Initial concern + +If the strict-compaction response barrier times out, the semantic job may finish later. If its raw result expires after only a short scheduler TTL and the next user turn arrives much later, the capsule never receives that valid result. + +### Correction + +Pending-job result retention must be useful for the branch, while remaining bounded: + +- raw output is still size/count bounded and never logged; +- while a BranchState references `PendingJobID`, the scheduler result may be retained up to the configured pending-continuity TTL, not an arbitrary tiny default; +- first eligible `Await` parses/validates/merges and immediately `Forget`s raw output; +- branch/job expiry clears both sides coherently; +- no result lives longer than the configured bounded continuity/source retention window; +- implementation may later add a typed completion-consumer optimization, but v1 does not need arbitrary callbacks in the scheduler. + +### Status + +**Accepted; config/result-lifetime design must be corrected.** + +## Finding V4 — Feature composition must fail clearly when prerequisite preservation/detector services are absent + +### Initial concern + +#312 is not runtime code on current `main`. A feature plugin silently receiving disabled compaction services would appear enabled while doing nothing, which is misleading and hard to operate. + +### Correction + +When `compaction-continuity` is enabled: + +- generation/process composition verifies the prerequisite detector preview/commit capability, branch coordinator and BackgroundAux client exist; +- missing prerequisite service is a startup/generation compile error with a clear message; +- feature disabled keeps zero/no-op behavior and does not require semantic extractor configuration; +- implementation tasks are chronologically ordered so #312 runtime lands first. + +This is configuration fail-closed, not request traffic fail-closed. + +### Status + +**Accepted; design/config tasks must pin it.** + +## Finding V5 — Process-owned branch coordination is justified despite feature-state guidance + +### Concern + +The extension platform generally prefers plugin state through `lipsdk/state.Store` rather than per-feature core state. However current Store `Get`/`Put` has no atomic CAS, and a per-generation plugin mutex cannot protect overlapping old/new generation callbacks and background workers. + +### Decision + +Keep the process-owned **narrow BranchCoordinator** as a synchronization facade and use process `ExtensionState` as its serialized backing where practical. The coordinator owns only: + +- branch-key lock/serialization; +- revision/high-watermark/job/injection metadata; +- bounded entry TTL/eviction; +- opaque capsule/source blobs. + +It does **not** own plan semantics, extractor prompts, provider clients, billing, generic state transactions, or arbitrary tasks. + +This is smaller and safer than adding a generic transactional plugin-store API solely for this feature. + +### Status + +**Accepted as designed.** + +## Finding V6 — Background auxiliary scheduler is a reusable primitive but must remain narrow + +### Concern + +Adding an async service can become a generic workflow engine. + +### Decision + +The scheduler accepts only canonical auxiliary model collection requests and returns bounded collection jobs. It cannot schedule arbitrary functions, timers, cron, durable tasks, webhooks, or user-defined callbacks. ProcessServices owns it; generation snapshots receive non-owning adapters that capture the current runner/pin at submission. + +### Status + +**Accepted with architecture tests.** + +## Finding V7 — Detached mode must not become a client bypass around secure session + +### Concern + +An exported/internal session mode that suppresses secure-session BeginTurn could be abused if a frontend could set it. + +### Decision + +Detached mode is carried only on the trusted auxiliary SDK/internal request object after frontend decode; no `lipapi.Call` JSON/wire field or header maps to it. Architecture/tests prove frontend inputs cannot request detached mode. Feature plugins are already trusted in-process extensions; the standard continuity feature is the initial consumer. + +### Status + +**Accepted with security tests.** + +## Finding V8 — Billing requirements align with current money architecture + +### Validation + +Current billing account identity is principal-scope-derived; detached child can preserve this scope. Normal Executor submission then naturally receives independent BillingCallID/exposure/terminal usage/provider COGS. No stream-time financial mutation is required. + +The only additive need is a content-free auxiliary workload/role projection for reports/diagnostics. That metadata must not influence pricing implicitly. + +### Status + +**PASS.** + +## Finding V9 — Primary protocol usage separation is achievable + +The child is an independent internal call and is never encoded into the primary frontend response. Existing protocol usage remains scoped to primary canonical stream. Account-level durable usage/billing includes child records separately. + +### Status + +**PASS; regression tests required.** + +## Finding V10 — Opaque compaction fallback is mandatory, not optional polish + +`CompactionItem` exposes encrypted/opaque state rather than universal plaintext summary. The result-augmentation path therefore cannot be the only preservation mechanism. + +Reinjection on the first eligible post-compaction request is the universal safe fallback and must be implemented/tested even if one or more plaintext carriers are supported. + +### Status + +**PASS after design explicitly treats reinjection as mandatory fallback.** + +## Validation Checklist + +| Check | Result | +|---|---| +| one compaction-recognition authority | PASS | +| detector observer remains content-free/non-mutating | PASS | +| actual final event observed by committed detector | PASS after V1 correction | +| strict extraction not billed before primary Open | PASS | +| real background execution with submit-time generation ownership | PASS | +| worker process-owned across generation reload | PASS | +| no unbounded goroutine/queue | PASS | +| off-primary-session semantics | PASS after V2 correction | +| independent extractor selector | PASS | +| parent route override cannot hijack child | PASS | +| same user/account billed by default | PASS | +| separate auxiliary BillingCallID/B-legs | PASS | +| primary protocol usage unchanged | PASS | +| late job result remains useful | PASS after V3 correction | +| encrypted/opaque content immutable | PASS | +| mandatory reinjection fallback | PASS | +| no second transcript DB | PASS | +| no provider-specific core branches | PASS | +| no generic workflow engine | PASS | +| prerequisite absence visible at startup | PASS after V4 correction | +| generation reload semantics explicit | PASS | +| restart durability honest | PASS | +| TDD/race/goleak coverage feasible offline | PASS | + +## Design-to-Requirement Trace + +| Design decision/component | Primary requirements | +|---|---| +| shared detector preview + prerequisite | 1.1–1.12 | +| separate Preserver FeatureBundle surface | 1.2–1.4, 7.14, 11.11–11.12 | +| Continuity Capsule + merge precedence | 2.1–2.15, 8.4, 8.11–8.12 | +| deterministic carrier catalog | 3.1–3.7 | +| sanitizer/source window | 3.8–3.16, 9.3–9.10 | +| BackgroundAux scheduler | 4.1–4.15, 11.2, 11.10–11.11 | +| detached child + private A-leg | 5.5–5.13, 11.4–11.5 | +| explicit extractor route | 5.1–5.5, 10.1–10.2 | +| normal user billing + workload class | 6.1–6.13, 10.7, 11.6–11.7 | +| remote strict-compaction flow | 7.1–7.8 | +| authority-aware reinjection | 7.7–7.15 | +| branch coordinator | 8.1–8.12, 11.8, 11.10 | +| reload/restart behavior | 8.9–8.15, 10.8–10.9 | +| trusted policy/privacy | 9.1–9.13 | +| failure/observability | 10.1–10.11 | +| architecture/testing gates | 11.1–11.14 | + +## Simplification Review + +The chosen design adds two infrastructure capabilities only because existing seams cannot safely satisfy the requirements: + +1. **BackgroundAux scheduler** — required because current Aux.Collect is synchronous and post-lease spawn is unsafe. +2. **BranchCoordinator** — required because process ExtensionState lacks atomic revision update and feature-instance locks do not span generations. + +Everything else is additive use/refactoring of existing authorities. In particular, do **not** add: + +- a second Executor/provider client; +- generic async function/task APIs; +- durable queue/database; +- event bus for preservation; +- general memory/RAG subsystem; +- new per-provider compaction adapters; +- new billing journal/rater; +- a generic transactional state platform; +- a second summary-model pass. + +## Implementation Risks to Pin With Tests + +1. `KindAsync` retained too late -> deterministic post-lease submission RED test. +2. child accidentally resumes parent secure session -> transcript/turn/activity RED tests. +3. child inherits parent route override -> explicit different-route RED test. +4. parent/child BillingCallID collision -> billing RED test. +5. primary protocol usage includes child tokens -> frontend usage RED test. +6. detector commits completion before preservation final event -> ResponsePreview/ResponseReleased ordering test. +7. opaque `EncryptedContent` altered -> exact bytes RED test. +8. job result expires before next turn -> pending-result retention/expiry RED test. +9. stale worker overwrites user correction -> revision race RED test. +10. generation reload drops job/coordinator -> retained-generation RED test. +11. queue saturation creates goroutine fallback -> goroutine-count/goleak RED test. +12. disable/reload drops already-submitted billing settlement -> terminal accounting RED test. + +## Final Gate + +After applying V1–V4 corrections to requirements/design, the architecture is **GO** for TDD task generation. The implementation should be split into phases that establish background/detached/billing correctness before semantic extractor prompting and final compaction integration. From 21f46818ab130b11474b78a59eff6b0ef840d522 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:09:04 +0200 Subject: [PATCH 09/18] spec: apply compaction continuity design validation corrections --- .../requirements.md | 49 ++++++++++--------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/requirements.md b/.kiro/specs/compaction-continuity-preservation/requirements.md index 422118f9..3b37a2f5 100644 --- a/.kiro/specs/compaction-continuity-preservation/requirements.md +++ b/.kiro/specs/compaction-continuity-preservation/requirements.md @@ -28,9 +28,10 @@ This specification depends on the compaction-recognition authority defined by th 1.7. When a recognized compaction request successfully opens upstream, preservation may start one extraction job for that logical transaction while primary compaction work continues independently. 1.8. Completion-only/history evidence shall never invent a historical start, but it may trigger extraction/reinjection required for a compaction only observable after installation. 1.9. Any pre-open request preview needed to protect the first post-compaction turn shall be a pure/non-committing view over the same matcher/fingerprint authority; preview shall not emit lifecycle events, advance detector state, or establish a strict-start billing trigger. -1.10. Final-release ordering shall keep detector derivation non-mutating: selected canonical event -> derived compaction lifecycle metadata -> separate preservation finalization -> metadata-observer dispatch -> client release. No ordinary response hook may run after preservation finalization. -1.11. Provider/frontend wire DTOs and provider SDK types shall not enter core continuity logic. -1.12. Generic words such as `plan`, `summarize`, `continue`, or `compact` shall not by themselves establish a compaction boundary or accepted plan. +1.10. Response-side preservation shall use a pure/non-committing response preview over the same detector authority to identify a potential completion before mutation; preview shall not mark a transaction complete or dispatch observers. +1.11. Final-release ordering shall be: selected canonical event -> pure response preview -> separate preservation finalization -> committed `ResponseReleased` on the resulting final event -> metadata-observer dispatch -> client release. No ordinary response hook may run after preservation finalization. +1.12. Provider/frontend wire DTOs and provider SDK types shall not enter core continuity logic. +1.13. Generic words such as `plan`, `summarize`, `continue`, or `compact` shall not by themselves establish a compaction boundary or accepted plan. ## Requirement 2: Versioned Continuity Capsule and Decision Semantics @@ -84,7 +85,7 @@ This specification depends on the compaction-recognition authority defined by th 4.11. Queue saturation, unavailable worker, shutdown, or failed generation retention shall follow preservation failure policy and shall not fall back to an unbounded goroutine/direct provider call. 4.12. Process shutdown shall stop admission, cancel/join worker execution, release generation pins exactly once, and satisfy repository goleak/race requirements. 4.13. Job results shall be revision-checked before merge; stale jobs cannot overwrite newer intent/capsule revisions. -4.14. Completed raw auxiliary results shall be held only in bounded TTL-limited process memory until parsed/consumed, never logged, and explicitly forgotten/expired after normalized capsule/delta storage. +4.14. Completed raw auxiliary results shall be bounded by count/bytes/TTL, never logged, and forgotten immediately after validation/merge. While a BranchState still references a pending useful JobID, its result retention shall be long enough for the configured bounded continuity/pending window rather than an arbitrary short cache TTL; branch/job expiry shall clear both sides coherently. 4.15. Disabling/reloading the feature shall prevent new jobs according to the new configuration but shall not erase accounting obligations or leak execution for provider work already submitted. ## Requirement 5: Independently Configurable Route and Detached Session Semantics @@ -99,9 +100,11 @@ This specification depends on the compaction-recognition authority defined by th 5.8. Extractor output shall be bounded and schema-oriented rather than unconstrained conversational output. 5.9. The preservation plugin shall suppress itself on the child; existing auxiliary-depth recursion limits remain effective. 5.10. Core auxiliary execution shall provide a typed detached-session mode for this workload. It shall preserve authenticated principal/scope and parent correlation while suppressing primary secure-session BeginTurn/turn transcript/last-activity effects. -5.11. Detached execution shall not mutate the primary A-leg route override, primary session turn count, primary session transcript, or client-visible session history. -5.12. Detached-session authority shall be internal execution metadata and shall not be encoded as provider-visible opaque call content. -5.13. The child may use private B2BUA/sessionless execution lineage required for routing/usage/billing, but it shall be clearly distinguishable from the user's primary conversational turn. +5.11. Detached execution shall create/use a **private child A-leg** for the auxiliary logical call using existing B2BUA lifecycle semantics; it shall neither be sessionless nor reuse the primary A-leg as execution authority. +5.12. The private child A-leg may own normal B-leg attempts/request authority/billing correlation, while `ParentALegID` remains explicit lineage only. +5.13. Detached execution shall not mutate the primary A-leg route override, primary session turn count, primary session transcript, or client-visible session history. +5.14. Detached-session authority shall be trusted internal execution metadata and shall not be encoded as provider-visible opaque call content or settable from frontend wire fields. +5.15. Child A-leg/attempt lineage shall remain clearly distinguishable as internal auxiliary workload from the user's primary conversational turn. ## Requirement 6: Originating-User Billing, Usage, and Cost Attribution @@ -175,29 +178,31 @@ This specification depends on the compaction-recognition authority defined by th 10.1. Global feature config shall support at least enablement, preserved categories, extractor route, timeout/input/output limits, worker concurrency/queue bounds, barrier timeout, capsule/source/result bounds/TTL, and failure mode. 10.2. Semantic extraction requires an explicit route or explicit inherit policy; no accidental main-model default is allowed. -10.3. Default failure mode shall be fail-open for model traffic: extractor timeout/error/saturation/invalid output/state failure does not make native compaction unusable. -10.4. Fail-open preserves already-valid deterministic capsule state and never injects malformed/partial extractor output. -10.5. Extractor output shall be validated against a strict versioned schema with byte/depth/count limits before merge/injection. -10.6. Observability shall count compaction candidates; deterministic hits; semantic jobs submitted/coalesced/skipped/saturated; latency/outcome; extractor input/output tokens; capsule revision/size/fact counts; barrier waits/timeouts; augmentation/reinjection; stale/conflict rejection. -10.7. Billing/usage observability shall expose auxiliary continuity cost through existing accounting stores rather than duplicating financial truth. -10.8. Config reload shall not mutate active job route/budgets mid-flight. -10.9. Disabling the feature prevents new jobs while allowing bounded cleanup/accounting of already-submitted work. -10.10. No failure path may retry indefinitely, spin, block shutdown indefinitely, or fall back to an unconfigured provider/model. -10.11. Background raw-result retention shall be bounded/TTL-limited and observable only through content-free counts/status; normalized capsule state replaces raw result after consumption. +10.3. Enabling the feature shall fail generation/startup composition clearly if the prerequisite compaction detector preview/commit service, process branch coordinator, or BackgroundAux capability is unavailable; disabled feature mode shall retain no-op compatibility. +10.4. Default request-time failure mode shall be fail-open for model traffic: extractor timeout/error/saturation/invalid output/state failure does not make native compaction unusable. +10.5. Fail-open preserves already-valid deterministic capsule state and never injects malformed/partial extractor output. +10.6. Extractor output shall be validated against a strict versioned schema with byte/depth/count limits before merge/injection. +10.7. Observability shall count compaction candidates; deterministic hits; semantic jobs submitted/coalesced/skipped/saturated; latency/outcome; extractor input/output tokens; capsule revision/size/fact counts; barrier waits/timeouts; augmentation/reinjection; stale/conflict rejection. +10.8. Billing/usage observability shall expose auxiliary continuity cost through existing accounting stores rather than duplicating financial truth. +10.9. Config reload shall not mutate active job route/budgets mid-flight. +10.10. Disabling the feature prevents new jobs while allowing bounded cleanup/accounting of already-submitted work. +10.11. No failure path may retry indefinitely, spin, block shutdown indefinitely, or fall back to an unconfigured provider/model. +10.12. Background raw-result retention shall be bounded and useful only while referenced/pending under the configured continuity window; normalized capsule state replaces raw result after consumption. ## Requirement 11: TDD, Architecture, and Non-Interference Gates 11.1. Implementation shall begin with RED tests for capsule merge/supersession, structured carriers, eligibility/sanitization, background lifetime, detached session behavior, independent routing, billing attribution, barriers, repeated compactions, and opaque payload protection. 11.2. Worker packages shall use deterministic scheduling controls plus `goleak`/race tests for saturation, cancellation, shutdown, duplicate submission, stale completion, and barrier races. 11.3. Tests shall prove no extraction job is submitted when a strict compaction-looking request fails before upstream Open. -11.4. Tests shall prove the child is absent from primary secure-session transcript/turn count/last activity and does not mutate primary A-leg route-override/session state. +11.4. Tests shall prove the child uses a private auxiliary A-leg, is absent from primary secure-session transcript/turn count/last activity, and does not mutate primary A-leg route-override/session state. 11.5. Tests shall prove extractor routing can differ from primary routing and that primary A-leg overrides do not rewrite the child selector. 11.6. Billing tests shall prove separate auxiliary BillingCallID/B-leg usage, originating account attribution, aggregate inclusion, continuity workload classification, and unchanged primary protocol usage. 11.7. Tests shall cover pre-submit billing rejection, submitted-but-invalid result, submitted-but-stale result, retry/failover cost, and fail-open behavior. 11.8. Tests shall cover at least three successive compactions with decision supersession, plan progress, dedupe, and bounded growth. 11.9. Tests shall prove encrypted/opaque compaction content is byte-identical with preservation enabled. -11.10. Tests shall prove process generation reload cannot orphan jobs/state and that job submission captured `KindAsync` ownership before the request spawn right ended. -11.11. Architecture gates shall prevent provider/frontend DTO imports, direct provider clients, generic service locators/task runtimes, a second full-transcript store, feature-owned money ledgers, or generation-scoped ownership as the sole worker lifetime. -11.12. The implementation shall preserve canonical call/event validity, output commitment/no-retry-after-output, B2BUA lineage, secure-session ownership, generation pinning, and current billing settlement authorities. -11.13. Focused/repository tests shall run without external model credentials; live extractor tests are optional environment-gated evidence. -11.14. Final review shall remove duplicate rule catalogs, unnecessary persistence/framework layers, unbounded queues/goroutines, provider-specific core branches, and any redundant second LLM summary-rewrite pass. +11.10. Tests shall prove response preview does not commit lifecycle state and committed `ResponseReleased` sees the exact final event after any permitted plaintext preservation mutation. +11.11. Tests shall prove process generation reload cannot orphan jobs/state and that job submission captured `KindAsync` ownership before the request spawn right ended. +11.12. Architecture gates shall prevent provider/frontend DTO imports, direct provider clients, generic service locators/task runtimes, a second full-transcript store, feature-owned money ledgers, or generation-scoped ownership as the sole worker lifetime. +11.13. The implementation shall preserve canonical call/event validity, output commitment/no-retry-after-output, B2BUA lineage, secure-session ownership, generation pinning, and current billing settlement authorities. +11.14. Focused/repository tests shall run without external model credentials; live extractor tests are optional environment-gated evidence. +11.15. Final review shall remove duplicate rule catalogs, unnecessary persistence/framework layers, unbounded queues/goroutines, provider-specific core branches, and any redundant second LLM summary-rewrite pass. From a74ae27ab4894867aebbf1fd5c6f6a5961319676 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:10:23 +0200 Subject: [PATCH 10/18] spec: apply compaction continuity design validation corrections --- .../design.md | 827 ++++++++---------- 1 file changed, 375 insertions(+), 452 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/design.md b/.kiro/specs/compaction-continuity-preservation/design.md index 82174332..3d428359 100644 --- a/.kiro/specs/compaction-continuity-preservation/design.md +++ b/.kiro/specs/compaction-continuity-preservation/design.md @@ -6,155 +6,166 @@ Implement issue #344 as a bounded compaction-continuity feature that preserves d The design has seven cooperating pieces: -1. the existing `compaction-event-detection` detector/rule authority, implemented first and refactored only enough to provide a pure request preview; +1. the `compaction-event-detection` detector/rule authority, implemented first and refactored only enough to provide pure request/response previews; 2. a separate additive `pkg/lipsdk/compaction.Preserver` content-bearing extension surface; 3. a process-owned bounded background auxiliary collector under `internal/core/auxreq`; -4. a typed detached auxiliary execution/session mode that preserves principal/billing identity but not primary session-turn effects; -5. a process-owned branch coordinator that serializes revision/job/injection state across runtime generations; +4. a typed detached auxiliary execution mode that preserves principal/billing identity while using a private child A-leg and no primary session-turn effects; +5. a process-owned branch coordinator serializing revision/job/injection state across runtime generations; 6. an official `internal/plugins/features/compactioncontinuity` implementation for capsule schema, deterministic plan carriers, sanitizer, extractor prompt/schema, merge and injection; -7. existing routing, B2BUA, usage/metering and BillingCallID accounting reused unchanged as execution authorities. +7. existing routing, B2BUA, usage/metering and BillingCallID accounting reused as execution authorities. -The semantic extractor runs independently from the primary coding turn and may use a completely different model/provider. It is nevertheless a normal canonical Go-LIP child call, so the originating user is billed for actual extractor usage by default. +The semantic extractor runs independently from the primary coding turn and may use a completely different model/provider. It is nevertheless a normal canonical Go-LIP child call, so actual extractor usage is billed to the originating authenticated user/account by default. ## Goals - Preserve accepted/current plan state and explicit user product/architecture decisions across repeated lossy compactions. - Make structured planning state deterministic and LLM-free where possible. -- Run semantic extraction off-session and in background, with bounded worker/resource ownership. -- Allow a separately configured cheap/fast extractor route independent of the primary session route. -- Attribute auxiliary usage/cost to the originating authenticated user through existing billing authorities. +- Run semantic extraction off-session/background with bounded process ownership. +- Allow a separately configured extractor route independent of the primary session route. +- Attribute auxiliary usage/cost to the originating user through existing billing authorities. - Prevent the first post-compaction turn from outrunning required continuity state. - Keep provider-native encrypted/opaque compaction content byte-identical. -- Keep the feature bounded, fail-open and observable without logging decision content. +- Keep the feature bounded, fail-open and content-safe in observability. ## Non-Goals - general long-term/RAG memory; - storing every conversation fact; - replacing native/agent compaction; -- a general durable job framework; +- a general durable job/workflow framework; - a second transcript database; - a second financial ledger/rating engine; - direct provider clients for the extractor; - mutating `compaction.Observer`; -- provider-specific continuity branches inside core; -- transparent restart durability when no durable authorized source exists. +- provider-specific continuity branches in core; +- claiming restart durability without an authorized durable source. -## Dependency and Implementation Order +## Dependency and Composition Gate -The runtime implementation of `.kiro/specs/compaction-event-detection/` is a prerequisite. - -Chronological implementation order is therefore: +`.kiro/specs/compaction-event-detection/` is a hard runtime prerequisite. ```text compaction-event-detection runtime | - +--> pure request preview extracted from same matcher/fingerprint authority + +--> pure request/response previews from the same matcher/fingerprint authority | v compaction-continuity-preservation ``` -Continuity implementation must not recreate the #312 agent-signature table as a temporary fallback. +When the continuity feature is **enabled**, generation/process composition verifies that detector preview/commit services, process BranchCoordinator and BackgroundAux are available. Missing prerequisites fail generation/startup composition with a clear error. When the feature is disabled, these requirements are no-op/compatible and no extractor configuration is required. + +Continuity must never recreate the #312 signature matrix as a fallback. ## Existing Architecture Reused -- `lipapi.Call`, `NormalizedItems` and canonical item/tool/message forms provide provider-neutral source data. -- `lipapi.OperationContextCompaction` / `ItemKindCompaction` provide canonical strict semantics where supported. -- `internal/core/compactiondetect` (from prerequisite) owns compaction evidence, rule IDs, A-leg transactions and history heuristic. -- `pkg/lipsdk/feature.FeatureBundle` / `internal/featurebundle` / `extensions.RequestRuntimeSnapshot` provide additive extension composition. -- `internal/core/auxreq` already delegates auxiliary calls through the normal Executor, clones principal/scope, marks internal origin and supports plugin suppression. -- `pkg/lipsdk/genpin` already defines `KindAsync` for request-spawned asynchronous generation ownership. -- `ProcessServices` owns process lifetime and `ExtensionState` across generation reloads. -- secure-session recording already provides an optional durable transcript when explicitly enabled. -- billing account identity already comes from authenticated principal scope; child calls naturally get independent BillingCallIDs. +- `lipapi.Call`, canonical messages/items/tools and normalized walkers provide provider-neutral source data. +- prerequisite `internal/core/compactiondetect` owns compaction evidence, rule IDs, A-leg transactions and history heuristic. +- `FeatureBundle` / single merge surface / request snapshot provide additive extension composition. +- `internal/core/auxreq` delegates child calls through the normal Executor, clones principal/scope, marks internal origin and suppresses plugins. +- `pkg/lipsdk/genpin.KindAsync` already models request-spawned asynchronous generation ownership. +- `ProcessServices` owns process lifetime and process `ExtensionState` across generation reloads. +- secure-session recording already provides optional durable transcript data when explicitly enabled. +- billing account identity derives from authenticated principal scope; independent child calls receive independent BillingCallIDs. ## Target Architecture ```text - PRIMARY USER TURN / COMPACTION PROCESS-OWNED AUXILIARY PLANE + PRIMARY / COMPACTION FLOW PROCESS-OWNED AUXILIARY PLANE canonical request | v - shared detector PreviewRequest (pure) -----> preservation BeforeRequest - | | pending capsule/job? - | | bounded Await if required - | +--> canonical reinjection - v - normal prepare / route / billing / Open + detector PreviewRequest (pure) | - +-- Open succeeds ---------------------> detector RequestOpened / transaction - | | - | +--> Preserver.RequestOpened - | | - | +--> deterministic harvest - | +--> sanitize delta - | +--> BackgroundAux.SubmitCollect - | | - | v - | bounded worker pool - | | - | detached child Executor - | | - | independent route/model - | | - | normal usage/billing - | | - | strict JSON result - | | - | validate + branch merge + +--> Preserver.BeforeRequest + | pending capsule/job? + | bounded Await only when needed + +--> canonical reinjection | - primary compaction stream proceeds concurrently + normal prepare / route / billing / Open + | + +-- Open succeeds + | + +--> detector RequestOpened (commit) + +--> Preserver.RequestOpened + | deterministic harvest / sanitizer + +--> BackgroundAux.SubmitCollect --------+ + | + primary compaction stream continues v + bounded workers + | + detached child call + | + private child A-leg + | + independent selector + | + normal B-legs/billing + | + strict JSON result + | + bounded result registry | final selected event | - detector derives completion metadata (no mutation) + detector PreviewResponse (pure) + | + Preserver.BeforeResponseRelease + |---- bounded Await --------------------------------------+ + |---- validate/merge capsule + |---- safe plaintext? mechanical augment + |---- opaque/late? pending reinjection + | + detector ResponseReleased(final event) (commit) + | + metadata-only compaction observers | - preservation BeforeResponseRelease - |---- bounded Await if useful ----------------------------+ - |---- verified plaintext carrier? -> mechanical augment - |---- opaque/late? -> mark pending reinjection - v - compaction metadata observers - v client release ``` -## D1. Shared Detector Preview Versus Committed Detection - -The prerequisite detector currently has committed request semantics conceptually equivalent to: +## D1. Pure Preview Versus Committed Detector State -```go -func (d *Detector) RequestOpened(meta RequestMeta, call lipapi.Call) []compaction.Event -``` +The prerequisite detector remains the one recognition authority. -Add a pure preview API or internal equivalent: +Add pure internal/public-to-core preview shapes (exact names may differ): ```go type RequestPreview struct { - Evidence compaction.Evidence + Evidence Evidence RuleID string Kind PreviewKind // none | start_candidate | completion_candidate - // Opaque deterministic fingerprint/token for revalidation; no raw content. + TransactionID string // only when safely derivable from existing active state +} + +type ResponsePreview struct { + Evidence Evidence + RuleID string + Kind PreviewKind // none | completion_candidate + TransactionID string } func (d *Detector) PreviewRequest(meta RequestMeta, call lipapi.Call) RequestPreview +func (d *Detector) PreviewResponse(meta ResponseMeta, ev lipapi.Event) ResponsePreview ``` -Properties: +Preview properties: -- same rule/fingerprint implementation as committed detection; -- read-only with respect to transaction/fingerprint state; -- emits no observer event; -- cannot be used as proof that an upstream compaction started; -- useful only to let `BeforeRequest` identify a completion-only/installed-compaction boundary before opening the first post-compaction B-leg. +- same matcher/fingerprint logic as committed detection; +- no transaction/fingerprint mutation; +- no lifecycle event emission; +- no observer dispatch; +- request preview cannot establish a billable strict start; +- response preview lets preservation identify the matching job before final response mutation. -`RequestOpened` remains the only request-side state-commit point and remains after successful `Open`. +Committed boundaries remain: -## D2. Public Preservation Extension Contract +- `RequestOpened` only after successful upstream Open; +- `ResponseReleased` only after all permitted preservation finalization, on the exact event sent to the client. -Extend the `pkg/lipsdk/compaction` package introduced by the prerequisite with a distinct preservation surface. Exact names may change, but responsibilities shall remain separate. +## D2. Preservation Extension Contract + +Extend the `pkg/lipsdk/compaction` package introduced by #312 with a distinct preservation surface. Conceptual contract: @@ -177,43 +188,27 @@ type Services struct { type Preserver interface { ID() string - BeforeRequest( - context.Context, - *lipapi.Call, - RequestPreview, - PreservationMeta, - Services, - ) error - - RequestOpened( - context.Context, - lipapi.Call, - []Event, - PreservationMeta, - Services, - ) error - - BeforeResponseRelease( - context.Context, - *lipapi.Event, - []Event, - PreservationMeta, - Services, - ) error + BeforeRequest(context.Context, *lipapi.Call, RequestPreview, PreservationMeta, Services) error + RequestOpened(context.Context, lipapi.Call, []Event, PreservationMeta, Services) error + BeforeResponseRelease(context.Context, *lipapi.Event, ResponsePreview, PreservationMeta, Services) error } ``` -`BeforeRequest` may mutate only the canonical request through the documented continuity injection path. `RequestOpened` is observation/source/job scheduling after successful Open. `BeforeResponseRelease` may mutate only a verified plaintext continuation carrier. +Semantics: + +- `BeforeRequest`: pre-open pending reinjection / completion-only barrier; may mutate only through the continuity injection helper. +- `RequestOpened`: successful-open source commit and background job scheduling; current primary request has already been sent upstream. +- `BeforeResponseRelease`: pure-preview-guided bounded join and verified result-side augmentation before committed `ResponseReleased`. -`FeatureBundle` gains a separate optional `CompactionPreservers []compaction.Preserver` slice. The single merge surface concatenates it in registration order and the request snapshot exposes a frozen defensive copy. +`FeatureBundle` gains `CompactionPreservers []compaction.Preserver`; the single merge surface concatenates in registration order and the runtime snapshot exposes a frozen defensive copy. -Do **not** add request/response content to `compaction.Event` or change `Observer` into a decision/mutation interface. +No request/response content is added to `compaction.Event` and `Observer` remains unchanged. ## D3. Process-Owned Background Auxiliary Collector -### SDK surface +### Additive SDK capability -Keep synchronous `auxiliary.Client` source-compatible. Add an additive interface: +Keep synchronous `auxiliary.Client` source-compatible. Add a narrow interface: ```go type JobID string @@ -230,62 +225,51 @@ type BackgroundClient interface { } ``` -`DisabledBackgroundClient` provides explicit not-configured behavior. +`DisabledBackgroundClient` returns explicit not-configured errors. -### Core implementation +### Scheduler implementation -Create a focused scheduler under `internal/core/auxreq` (names indicative): +Create focused `internal/core/auxreq` background collection support: ```text BackgroundScheduler - fixed/bounded worker count - bounded input queue - keyed coalescing map - bounded result map + TTL + bounded worker count + bounded queue + coalescing map + bounded result registry process root context closeOnce + WaitGroup ``` -It executes canonical auxiliary calls only; it does not accept arbitrary Go callbacks/functions. +It accepts only canonical auxiliary model collection jobs, never arbitrary Go callbacks/tasks/timers. ### Submit-time ownership transfer -`SubmitCollect` executes synchronously until ownership is safely transferred: - -1. validate cloneable child call and options; -2. resolve current generation's `ExecutorRunner` through the generation snapshot cell; -3. obtain request `genpin.Retainer` and `Retain(genpin.KindAsync)`; -4. clone required principal/scope/correlation context into an immutable job envelope; -5. reserve/coalesce the job in the process scheduler; -6. enqueue or fail atomically; -7. on enqueue failure, release the newly retained pin immediately. - -A worker never attempts to acquire a fresh generation spawn right later. The captured pin is released exactly once after terminal child execution/collection or cancellation. - -### Job context +`SubmitCollect` synchronously: -Worker context derives from the scheduler/process root plus copied safe execution attribution, not from the parent request's cancellation tree. Per-job timeout then bounds model work. +1. validates/clones the child request; +2. resolves the current generation `ExecutorRunner` from the request snapshot cell; +3. obtains `genpin.Retainer` and `Retain(KindAsync)` while spawn authority is live; +4. clones required principal/scope/correlation attribution; +5. reserves/coalesces the job; +6. enqueues or fails atomically; +7. releases a newly retained pin immediately on failed handoff. -Preserve: +A worker never attempts a later generation retain. The captured pin is released exactly once after terminal child collection/cancel. -- principal/scope; -- internal origin; -- parent trace/session/A-leg correlation; -- submission-time generation identity. +### Worker context -Do not preserve: +Worker context derives from the scheduler/process root and copied attribution, not from parent request cancellation. A per-job timeout bounds inference. -- parent cancellation lifetime as root; -- primary secure-session authority token/turn authority; -- primary A-leg route override authority. +Preserve principal/scope, internal origin, parent correlation and captured generation. Do not preserve primary resume/session authority or primary A-leg route authority. -### Result lifetime +### Result retention -Raw `lipapi.Collected` output is retained only until awaited/parsed, with explicit count/byte/TTL limits. `Forget` or expiry deletes it. Logs/metrics expose only job ID hash/status/size/token metadata. +Raw `lipapi.Collected` is count/byte bounded and never logged. A result referenced by a BranchState `PendingJobID` remains available for the configured bounded pending-continuity retention window, not merely a tiny cache TTL. First successful consume validates/merges then calls `Forget` immediately. Branch/job expiry clears both sides coherently. No result outlives the configured bounded continuity/source retention horizon. -## D4. Detached Auxiliary Session Mode +## D4. Detached Auxiliary Execution and Private Child A-Leg -Current auxiliary execution delegates through the normal Executor, whose standard path performs secure-session BeginTurn and primary request authority work. Add a typed execution policy to `auxiliary.Request` or an internal sibling type: +Add trusted auxiliary execution policy (exact type location may differ): ```go type SessionMode uint8 @@ -295,104 +279,111 @@ const ( ) ``` -The exact enum location may differ, but `Detached` is core-interpreted execution metadata and never provider-visible call content. +This is carried on the trusted auxiliary request object after frontend decode; no wire field/header maps to it. -Detached semantics: - -- authenticated principal/scope is preserved; -- parent IDs remain correlation metadata; -- no primary secure-session BeginTurn/new TurnID; -- no primary transcript/activity/turn-count mutation; -- no primary resume-token/session-authority propagation; -- no primary A-leg route-override lookup/application; -- no client-visible session headers/history; -- child still receives normal independent trace, route plan, B-legs, usage, metering and billing. +Detached child flow: -The implementation should factor the Executor prepare path rather than copy the whole executor. For detached calls, execute only the identity/scope + ordinary execution authorities that remain relevant. - -Security invariant: `SessionModeDetached` may only originate from trusted internal auxiliary code; frontends cannot request it from wire fields. +```text +originating principal/scope + | + v +private auxiliary logical call + | + +--> create/touch private child A-leg AUX456 + | parent_a_leg=A123 only in lineage + | + +--> normal route selector / request authority / BillingCallID + | + +--> one or more child B-legs + | + +--> normal terminal usage/billing +``` -## D5. Independent Extractor Routing +Detached semantics: -Feature config requires either: +- preserve authenticated principal/scope for security and billing; +- parent SessionID/A-leg/trace are correlation only; +- do not call primary secure-session BeginTurn or create primary TurnID; +- do not mutate primary transcript/activity/turn counts; +- do not propagate primary resume authority; +- do not fetch/apply primary A-leg route override; +- create/use a **private child A-leg** through existing B2BUA lifecycle/store semantics; +- keep child A-leg/B-legs private/internal but fully usable by existing request authority/billing machinery; +- no client-visible session/history/header effects. -- explicit `extractor.route`, or -- explicit `extractor.route_policy: inherit`. +Implementation should factor shared Executor prepare logic rather than clone an entire second executor. -Default should not accidentally inherit the main route. +## D5. Independent Extractor Routing -The extractor builds a canonical child call with its own `Route.Selector`. Normal routing then owns aliases, selector grammar, safe execution-composition policy, health, retry/failover and B-leg opening. +Feature config requires either explicit `extractor.route` or explicit `route_policy: inherit`. -Parent A-leg override is not consulted in detached mode. +The plugin builds a canonical child `Call.Route.Selector`; normal routing owns parsing, aliases, safe composition policy, health, failover and B-leg opening. The private child A-leg has no parent route override state, so the primary A-leg override cannot hijack the extractor. -The child is stamped: +Child attribution: ```text -Role = "compaction_continuity_extractor" -Visibility = "private" +Role = compaction_continuity_extractor +Visibility = private Origin = internal auxiliary ``` -Tools are empty/disabled and tool choice is none/valid equivalent. +Tools are absent/disabled; tool choice is none/legal equivalent. + +Submission captures route/timeouts/token bounds/failure policy immutably. Reload changes only future jobs. ## D6. Originating-User Billing and Workload Classification -The child enters the normal Executor with cloned authenticated `scope`. Existing billing identity therefore resolves the same customer account. +Detached execution retains original authenticated scope, so existing billing identity resolves the same account. -Every extraction gets: +Every extraction has: -- separate BillingCallID; -- separate B-leg attempt sequence; -- separate usage records; -- separate operational exposure/customer settlement; -- separate provider COGS. +- independent BillingCallID; +- private child A-leg and independent B-leg attempts; +- normal usage/concurrency authority; +- normal credit screen/exposure admission when authoritative billing applies; +- normal terminal usage/customer settlement/provider COGS. -No special charging function is added. +No special charging function or money ledger is added. -Add a bounded classification to execution/accounting evidence, conceptually: +Project a bounded content-free workload classification into existing accounting/metering/diagnostics, preferably from auxiliary lineage: -```go -type WorkloadClass string -const ( - WorkloadPrimary WorkloadClass = "primary" - WorkloadAuxiliary WorkloadClass = "auxiliary" -) - -AuxRole = "compaction_continuity_extractor" +```text +workload_class = auxiliary +aux_role = compaction_continuity_extractor ``` -Prefer reusing existing auxiliary lineage metadata as the source and project it into metering/billing/report DTOs where needed. Do not put raw capsule/source text into money records. +This classification does not implicitly change rates. -Primary frontend usage projection remains tied to the primary BillingCall/response only. Aggregate account/operator reports include both calls and can group auxiliary cost. +Primary frontend protocol usage stays primary-call-only. Account/operator totals include child records and can group auxiliary cost. -If the child is rejected before upstream submission, the feature records a fail-open extraction outcome and does not bypass admission. If provider work was submitted, later invalid/stale extraction does not erase actual usage/cost. +Pre-submit rejection means no provider usage and preservation fail-open. Once provider work is submitted, actual usage remains accountable even if the result is invalid/late/stale. -## D7. Process-Owned Continuity Branch Coordinator +## D7. Process-Owned Branch Coordinator -A per-generation feature mutex is insufficient because immutable generation reload can overlap with in-flight workers/turns. `ExtensionState` is process-owned but its current `Get` + `Put` API has no atomic compare-and-swap. +A per-generation feature mutex cannot protect overlapping generations/workers. Current process `ExtensionState` is reload-stable but has no atomic compare-and-swap. -Create a **small process-owned continuity coordinator** in `internal/core/compactioncontinuity` (name indicative) and build it in `ProcessServices` when the capability is enabled/available. It is not a general plugin framework. +Create a small process-owned `internal/core/compactioncontinuity` coordinator. It is a synchronization/state facade, not semantic memory logic. Responsibilities: - serialize updates per authoritative branch key; -- use process `ExtensionState` as backing storage for serializable state; -- provide revision-checked load/update operations to preservation services; -- track pending job IDs and injection watermarks; -- enforce max entries/TTL with lazy/opportunistic cleanup; -- contain no extractor prompts, provider code or billing logic. +- use process `ExtensionState` as serialized backing where practical; +- revision-checked load/update; +- pending job/injection watermarks; +- bounded max entries/TTL and lazy cleanup; +- opaque capsule/source blobs only. Conceptual branch key: ```go type BranchKey struct { - AuthoritativeSessionID string // when available + AuthoritativeSessionID string ALegID string - PrincipalPartition string // used when no secure session id + PrincipalPartition string // used when no secure SessionID } ``` -Client session hints do not participate as authority. +Client session hints never authorize lookup. Conceptual state: @@ -412,11 +403,11 @@ type BranchState struct { } ``` -The coordinator can expose opaque capsule/source JSON to the plugin so core does not own semantic fact interpretation. +Core coordinator never interprets plan/decision facts. Worker/provider work runs outside coordinator locks. -## D8. Continuity Capsule Schema +## D8. Continuity Capsule V1 -Feature-owned v1 model (exact field names may evolve under tests): +Feature-owned model: ```json { @@ -446,9 +437,7 @@ Feature-owned v1 model (exact field names may evolve under tests): } ``` -### Merge precedence - -Highest authority / latest source wins: +Merge precedence: 1. later explicit user correction/decision; 2. later explicit user acceptance/selection; @@ -456,28 +445,15 @@ Highest authority / latest source wins: 4. validated semantic inference; 5. older capsule state. -Semantic inference may propose new facts/status transitions but cannot resurrect a superseded/rejected fact against newer explicit user evidence. - -Fact IDs are stable across revisions. Existing IDs supplied in the previous capsule should be reused by the extractor when updating a fact. New deterministic carrier facts get deterministic IDs from type + canonical normalized content/source; new semantic facts get normalized content/category IDs only after validation. +Semantic output cannot resurrect a fact against newer explicit intent. IDs remain stable across revisions; existing IDs are reused, deterministic carrier facts use deterministic normalized IDs, and new semantic IDs are accepted only after validation. -### Bounded retention - -When over budget, deterministic pruning order is: - -1. preserve active constraints/decisions; -2. preserve pending/in-progress plan steps; -3. preserve associated rationale needed for those facts; -4. preserve meaningful current rejections; -5. condense completed plan history; -6. drop oldest superseded/irrelevant history. - -Never truncate JSON in the middle; prune complete facts and revalidate. +Bounded pruning preserves active decisions/constraints and pending/in-progress plan steps first, then useful rationale/current rejections, then condenses completed/superseded history. Prune whole facts and revalidate; never truncate JSON syntactically. ## D9. Deterministic Plan Carrier Catalog -Place semantic carrier parsing in `internal/plugins/features/compactioncontinuity`, not provider/core adapters. +Place structured carrier parsing in `internal/plugins/features/compactioncontinuity`, not provider adapters/core detector identity. -Static versioned table with explicit canonical matchers: +Initial versioned families, after pinning actual canonical fixtures: ```text codex.update_plan.v1 @@ -485,215 +461,168 @@ opencode.todo.v1 cline.task_progress.v1 ``` -Implementation research must pin actual canonical tool/item shapes from supported harness snapshots before enabling each rule. A rule requires table-driven positive and near-miss tests. - -No generic “markdown checklist = accepted plan” rule. - -The detector's compaction RuleID may be recorded for diagnostics, but carrier matching should work from canonical tool shapes rather than infer a provider brand. +Rules match canonical tool/item shapes, not agent brand. Each needs positive/near-miss tests. There is no generic “markdown checklist means accepted plan” rule. ## D10. Sanitized Extraction Source -Source builder walks the effective canonical call and produces a bounded structured envelope, not a raw wire dump. +Build a bounded structured envelope from effective canonical calls rather than raw wire bodies. Priority: -1. user messages/text relevant to decisions; -2. assistant plan/proposal/clarification text needed to interpret user replies; -3. recognized structured plan/TODO carriers; -4. previous capsule. +1. user decision/constraint text; +2. assistant plan/proposal/clarification needed to interpret later user replies; +3. recognized structured plan carriers; +4. prior capsule. -Default drops: +Default drop/truncate: - ordinary tool results; -- command output; -- compiler/test logs; -- large source/code/file dumps; +- command/compiler logs; +- large source/file/code dumps; - images/video/binary/file blobs; -- reasoning payloads not needed for explicit user decisions; -- unrelated external text. +- unnecessary reasoning payloads; +- unrelated external content. -Tool/external content that must be included is tagged `untrusted_external` and delimited. +Any exceptional included tool/external material is tagged untrusted and delimited. -Size policy should use deterministic bytes/estimated tokens and prefer tail + decision-bearing user turns rather than arbitrary truncation. +Every **successfully opened primary request** may refresh the process sanitized source/high-watermark. Failed pre-open calls never become committed source state. -Every successfully opened primary request may refresh the process-local **sanitized** source window/high-watermark. Failed pre-open requests never become committed source state. +Source priority is: current/effective canonical baseline -> process sanitized window -> optional existing authorized secure-session transcript via a narrow reader. No durable transcript read is required on ordinary hot path. -## D11. Semantic Extractor Child Call +## D11. Semantic Extractor Call -Build one canonical call with: +One canonical child call contains: - configured independent route; -- fixed system/developer instructions describing the extraction task; -- previous capsule JSON; +- fixed extraction instructions; +- prior capsule; - deterministic plan facts; -- sanitized context delta; +- sanitized delta; - no tools; - bounded max output; -- preservation plugin disabled; -- detached session mode; +- continuity plugin suppressed; +- detached mode/private child A-leg; - role `compaction_continuity_extractor`. -Expected output is one strict JSON object, e.g.: +Expected output is one strict JSON object such as: ```json { "schema_version": 1, "base_revision": 11, - "facts": [...], - "plan_updates": [...], - "remove_or_supersede": [...] + "facts": [], + "plan_updates": [], + "remove_or_supersede": [] } ``` -Validation happens before BranchCoordinator merge: +Validate before merge: -- JSON only / one top-level object; -- schema version exact; -- base revision expected or stale-discard path; -- byte/depth/item/count/string bounds; -- enum values exact; +- exactly one JSON object; +- exact schema version/base revision handling; +- byte/depth/item/string/count limits; +- exact enums; - no unknown authority escalation; - no raw tool/blob fields; -- valid source references only within supplied bounded input when required. +- valid/allowed source refs where used. -Malformed output is discarded fail-open. +Malformed output is discarded fail-open. There is normally no second LLM pass. ## D12. Semantic Eligibility Gate -The local gate exists to save cost, not to decide truth. +The local gate saves cost; it does not decide semantic truth. -Candidate signals may include: +Candidate signals may include structured plan change, substantial assistant plan followed by affirmative/corrective user text, explicit user choice/constraint/correction language, absent/stale capsule around planning markers, or new decision-relevant user turns after high-watermark. -- recognized structured plan carrier changed; -- assistant substantial plan followed by user affirmative/corrective turn; -- explicit user choice/correction/constraint language; -- previous capsule absent while compaction source contains planning markers; -- new relevant user turns after capsule high-watermark. - -Near misses must avoid triggering solely on ordinary words such as `plan`, `summary`, or `continue`. - -If deterministic state fully satisfies the preservation categories for the new delta, skip the LLM job. +Generic words alone never trigger. If deterministic state fully satisfies the configured preservation categories, skip the semantic job. ## D13. Strict Remote-Compaction Flow -### Request preview - -Before normal B-leg Open: - -- detector preview may identify a strict start candidate; -- preservation may prepare/sanitize state but **must not submit a strict-start semantic job yet**; -- existing pending reinjection from a prior completed compaction is handled here. +### Pre-open -### Successful Open +- `PreviewRequest` may identify a strict start candidate. +- preservation may prepare source/check old pending reinjection. +- **do not submit a strict-start semantic job yet**; a failed Open must not bill extraction. -After primary compaction B-leg successfully opens: +### Successful primary Open -1. detector commits `started` transaction; -2. preservation commits the sanitized request source; -3. deterministic carrier extraction updates capsule if possible; -4. eligibility gate decides whether semantic extraction adds value; -5. if yes, `BackgroundAux.SubmitCollect` submits exactly one coalesced job; -6. primary compaction stream continues without waiting. +1. detector `RequestOpened` commits start/transaction; +2. preservation commits sanitized source; +3. deterministic carrier extraction updates capsule; +4. eligibility decides whether semantic work adds value; +5. `BackgroundAux.SubmitCollect` submits one coalesced job if needed; +6. primary compaction stream proceeds concurrently. -### Completion release +### Final selected event -At final selected canonical event: +1. detector `PreviewResponse` identifies potential completion/transaction without committing; +2. preservation optionally `Await`s the matching job for the bounded barrier; +3. ready strict output is validated and revision-merged; +4. verified plaintext carrier may receive deterministic continuity projection; +5. unsafe opaque/late paths set `PendingInjectionRevision`; +6. detector `ResponseReleased` receives the **post-preservation final event** and commits completion; +7. metadata-only observers dispatch; +8. final event releases to client. -1. detector derives completion metadata without mutating the event; -2. preservation checks the matching transaction/job; -3. if a bounded Await completes and output validates, merge new capsule revision; -4. if event has a verified mutable plaintext continuation carrier, append/merge a versioned continuity block mechanically; -5. otherwise set `PendingInjectionRevision`; -6. dispatch metadata-only compaction observers; -7. release event to client. - -Await timeout/error is fail-open and leaves the job/result state available only according to bounded policy. A late valid job may still merge the capsule for post-compaction reinjection, subject to revision checks. +A barrier timeout is fail-open. A late result remains available while referenced by bounded pending state and can be consumed by the next eligible turn. ## D14. Completion-Only / Local-Compaction Flow -For local compaction where no strict pre-compaction start was visible, the first post-compaction request may be the first evidence. - -Before its B-leg opens: - -1. detector `PreviewRequest` identifies a completion candidate using the previous successful request fingerprint/current rewrite; -2. preservation loads the previous sanitized source/capsule; -3. deterministic carrier delta is applied; -4. if semantic extraction is required and no coalesced job exists, submit it to BackgroundAux; -5. bounded Await barrier waits only up to configured `barrier_timeout`; -6. ready capsule is injected into the current canonical request; -7. on timeout/failure, continue fail-open without invalid output; -8. after successful Open, detector commits completion-only state normally. - -Although the main request waits at the narrow barrier, the extractor still executes independently on the background worker and its own model route/session/billing lifecycle. - -## D15. Authority-Aware Reinjection +If the first evidence is the next rewritten request: -Create one canonical helper owned by the feature/core extension layer, not frontend protocols. +1. `PreviewRequest` identifies a completion candidate from shared history matcher/fingerprint state; +2. preservation loads prior sanitized source/capsule; +3. deterministic carrier delta merges; +4. if semantic work is required and no coalesced job exists, submit BackgroundAux; +5. bounded `Await` barrier runs before primary B-leg Open; +6. ready capsule is injected; timeout/failure continues fail-open; +7. normal primary Open proceeds; +8. only after successful Open does `RequestOpened` commit completion-only detector state. -For message-authoritative calls: +The extractor is still off-session/background; only the narrow synchronization point is on the main request path because no earlier observable parallel window existed. -- add a bounded proxy-owned developer/system instruction using the existing legal instruction/message representation. +## D15. Canonical Reinjection -For item-authoritative calls: +One provider-neutral helper applies a deterministic text projection of the capsule. -- add a canonical message item with equivalent role/content; -- do not simultaneously populate legacy `Messages`/`Instructions` because `Call.Validate` forbids mixed authorities. - -Continuity block format conceptually: - -```text -[LIP CONTINUITY CAPSULE v1 | revision=12] -The following facts were preserved from earlier accepted session state after context compaction. -Treat them as prior context, not as a new user request. -... -[/LIP CONTINUITY CAPSULE] -``` +- message-authoritative call -> legal proxy-owned developer/system instruction/message representation; +- item-authoritative call -> legal canonical message item; +- never populate legacy Messages/Instructions alongside item authority in violation of `Call.Validate`. -The serialized textual projection is derived deterministically from the capsule and rechecked against the injection budget. +Projection format is versioned/delimited and says it is prior continuation state, not a new user request. -`LastInjectedRevision` makes the operation idempotent for a boundary. +`LastInjectedRevision` prevents duplicate injection for one boundary. Projection is rechecked against injection budget after serialization. ## D16. Plaintext Result Augmentation Boundary -Do not assume `ItemKindCompaction` is text. +`ItemKindCompaction` is not assumed text. Maintain an allowlisted capability/matcher for verified mutable plaintext continuation carriers only. -Define a small allowlisted capability/matcher for a **verified plaintext continuation carrier**. If no safe carrier is known, do not mutate the response. - -Never write: +Never modify: - `CompactionItem.EncryptedContent`; - `CompactionItem.Opaque`; - provider signatures/opaque reasoning; - unknown extension blobs. -Tests compare opaque/encrypted payload bytes exactly with feature on/off. +Tests compare those bytes exactly with feature on/off. -The reinjection fallback is functionally sufficient; result augmentation is an optimization/stronger immediate carry-through only for proven-safe carriers. +Reinjection is mandatory universal fallback; response augmentation is only a safe-path optimization/stronger immediate carry-through. ## D17. Per-Session Policy -Global config comes from the standard feature-plugin opaque YAML config. - -Trusted session-level override resolution order: +Global feature config is standard opaque feature YAML. Trusted session resolution: ```text operator hard maxima / safety policy - > trusted per-session policy value + > trusted per-session value > global feature default ``` -Allowed session overrides may include: - -- enabled/disabled; -- preserve-category toggles; -- extractor route chosen from operator-allowed selectors; -- tighter token/timeout/barrier limits. - -Client-provided headers/session metadata do not directly produce these trusted values. No new unauthenticated control header is introduced. +Allowed trusted session controls may include enabled, preserved categories, approved extractor selector, and tighter limits. Client headers/session metadata do not directly set these values and no new unauthenticated control header is introduced. ## D18. Configuration -Indicative config only; concrete defaults are pinned during implementation tests: +Indicative only; implementation tests pin defaults: ```yaml plugins: @@ -714,153 +643,147 @@ plugins: max_output_tokens: 2000 max_concurrency: 2 queue_capacity: 16 - result_ttl: 30s barrier_timeout: 2s + pending_result_ttl: 2h max_capsule_tokens: 2500 source_ttl: 2h failure_mode: fail_open ``` -Validation rules: +Values are examples, not normative defaults. -- semantic extraction requires explicit route or explicit inherit; -- positive finite queue/concurrency/result/source/capsule bounds; -- barrier <= job timeout or explicitly documented independent behavior; -- per-session values cannot exceed hard global maxima; -- invalid config fails startup/generation compile rather than silently enabling expensive defaults. +Validation: + +- enabled feature requires prerequisite detector preview/commit + BranchCoordinator + BackgroundAux; +- semantic mode requires explicit route or explicit inherit; +- finite positive queue/concurrency/source/result/capsule bounds; +- pending result TTL must be consistent with branch/source usefulness and remains hard bounded; +- trusted session overrides cannot exceed global maxima without explicit authorization; +- invalid enabled config fails generation/startup composition instead of selecting an expensive model silently. ## D19. Failure Semantics -Default is fail-open for model traffic. +Default request-time behavior is fail-open. | Failure | Behavior | |---|---| -| no candidate state | skip semantic call | -| background queue full | record saturation; deterministic capsule/native compaction continue | +| prerequisite missing while feature enabled | generation/startup compile error | +| no decision candidate | no semantic job | +| queue full | deterministic capsule/native flow continue | | generation retain fails | no job; fail-open | -| child billing/admission denied | no provider bypass; deterministic state/native compaction continue | -| extractor route/provider fails | child normal retry rules; then fail-open | -| extractor timeout | job fails; no malformed injection | -| schema invalid | discard raw output; account actual submitted usage | -| job stale | discard/forget result; do not overwrite capsule | -| barrier timeout | continue native request/compaction; late valid result may merge if still current | -| capsule over budget | deterministic whole-fact pruning | +| child billing/admission denied | no bypass; deterministic/native flow continue | +| extractor backend/timeout fails | child normal recovery then fail-open | +| invalid schema | discard result; actual submitted usage still accountable | +| stale result | discard/forget; no state regression | +| barrier timeout | continue native flow; pending result may be consumed later while bounded/useful | +| capsule over budget | whole-fact deterministic pruning | | state/coordinator unavailable | skip preservation mutation; native traffic continues | -| process shutdown | stop admission, cancel/join worker, settle normal submitted usage paths | - -A future explicit fail-closed policy may be considered separately; v1 should not make a UX preservation feature a default model-traffic availability dependency. +| process shutdown | stop admission; cancel/join worker; normal submitted usage/accounting remains terminally owned | ## D20. Privacy and Security -- Feature default disabled. -- Remote extractor route is explicit data egress. -- Use existing redaction/secret handling before child egress. -- Do not send full raw session automatically. -- Treat all transcript/source as quoted untrusted data. -- Child has no tools and preservation plugin is suppressed. -- Never log extractor prompt/output/capsule. -- Raw completed background result lives only in bounded TTL process memory until parse/forget. -- Preserve principal/tenant/workspace authorization for any optional transcript read. -- Do not turn transcript capture on implicitly. -- Child detached-session mode is trusted internal-only, not frontend-controlled. +- disabled by default; +- remote extractor is explicit data egress; +- existing redaction/secret policy precedes egress; +- source is bounded/sanitized, not full raw session by default; +- source content is untrusted quoted data; +- no tools and preservation plugin suppressed; +- prompt/output/capsule absent from normal logs/metrics; +- pending raw result exists only in bounded process memory; +- transcript reads preserve principal/session/workspace authorization; +- no implicit durable transcript capture; +- detached mode is trusted auxiliary metadata unavailable to frontends. ## D21. Observability -Add content-free metrics/diagnostics: +Content-free metrics/diagnostics: -- compaction preview/start/completion counts by evidence/rule; -- deterministic carrier hits by versioned carrier ID; +- compaction preview/start/completion by evidence/rule; +- structured carrier hits by rule ID; - semantic eligibility skip reasons; - background jobs submitted/coalesced/saturated/canceled/completed/failed/stale; -- worker queue depth/in-flight count; -- extractor route/backend/model identifiers where existing policy allows; -- extractor input/output/cache token usage and child cost via existing usage/billing; -- capsule revision, serialized size, active fact/plan-step counts; -- barrier wait count/duration/timeouts; -- result augmentation versus reinjection counts; -- stale/CAS conflict rejection; -- opaque-carrier no-mutation path count. +- queue depth/in-flight; +- extractor route/backend/model identifiers where policy allows; +- extractor tokens/cache/cost through existing usage/billing; +- capsule revision/serialized size/fact counts; +- barrier wait duration/timeouts; +- augmentation/reinjection counts; +- stale/revision conflicts; +- opaque-carrier no-mutation count. -Normal logs expose IDs/hashes/counts, never preserved text. +Logs use hashes/IDs/counts only. ## D22. Lifetime and Concurrency Invariants -1. ProcessServices owns scheduler and branch coordinator. -2. Generation snapshot owns only non-owning handles to process services. -3. Each background job owns exactly one retained `KindAsync` pin after successful submission. -4. Job pin is released exactly once on every terminal/cancel/enqueue-failure path. -5. A job never obtains a new generation after its submit boundary. -6. Branch coordinator serializes revision/injection/job state across overlapping old/new generations. -7. Worker callbacks never run while coordinator internal locks are held. -8. Provider/model work never runs under a global branch-state mutex. -9. Job/coordinator maps have explicit bounds/TTL and no unbounded timer goroutines. -10. Config reload changes only future job configuration. - -## D23. Process Restart and Durable Resume - -V1 guarantees: +1. ProcessServices owns BackgroundScheduler and BranchCoordinator. +2. Generation snapshots hold non-owning clients only. +3. Each successfully submitted background job owns exactly one captured `KindAsync` pin. +4. Pin releases exactly once on every terminal/cancel/handoff-failure path. +5. A worker never obtains a different generation after submit. +6. BranchCoordinator serializes revision/job/injection state across overlapping generations. +7. No external model/provider call executes while a branch-coordinator lock is held. +8. No observer/plugin callback runs under scheduler/coordinator internal locks. +9. Job/result/branch maps have hard bounds/TTL and no unbounded cleanup goroutine pattern. +10. Reload changes future jobs only; already-submitted work remains accountable and terminally owned. -- continuity across compaction within a process; -- continuity across immutable generation reload; -- no cross-branch state leakage. +## D23. Restart and Durable Resume -V1 does **not** guarantee capsule persistence across process restart. +V1 guarantees continuity across compaction and immutable generation reload within one process. It does not create durable capsule/job storage. -If secure-session transcript capture already exists and future/implementation scope wires an authorized transcript-source adapter, the feature may reconstruct the capsule on demand after restart. The adapter must be read-only, bounded and principal/session authorized. Otherwise the missing capsule is explicit fail-open state. +An optional future/implementation recovery adapter may reconstruct from an already-enabled authorized secure-session transcript. It must be read-only, bounded and authorization-preserving. Without that source, missing process capsule after restart is explicit fail-open state. -Do not add a durable job queue or full transcript store under this spec. +No durable job queue or full transcript store is added here. ## D24. Expected Change Surface -Primary implementation packages: +Primary packages: -- `pkg/lipsdk/compaction` — preservation contract alongside prerequisite observer contract; +- `pkg/lipsdk/compaction` — preview-related preservation contract beside prerequisite observer; - `pkg/lipsdk/feature` — additive preserver slice; -- `pkg/lipsdk/auxiliary` — additive BackgroundClient/job types and detached session policy carrier; -- `internal/featurebundle` / `internal/core/extensions` — merge/frozen snapshot/dispatch stages; -- `internal/core/compactiondetect` — pure preview extracted from same rule authority; -- `internal/core/auxreq` — bounded background collector and detached execution handoff; -- `internal/core/compactioncontinuity` — narrow process branch coordinator; -- `internal/core/runtime` — request-open/final-release/detached prepare wiring; -- `internal/infra/runtimebundle` — ProcessServices ownership and snapshot clients; -- `internal/plugins/features/compactioncontinuity` — capsule, carrier rules, sanitizer, eligibility, extractor, merge, injection, config; -- existing metering/billing/report DTO/projection locations only as needed for content-free auxiliary workload classification; -- focused architecture/testkit fixtures. - -Packages that should **not** need provider-specific edits: - -- individual backend providers/connectors; -- individual frontend protocol codecs; -- provider SDK adapters for compaction continuity semantics. +- `pkg/lipsdk/auxiliary` — additive BackgroundClient/job types and trusted detached policy carrier; +- `internal/featurebundle` / `internal/core/extensions` — merge/snapshot/preservation stages; +- `internal/core/compactiondetect` — pure request/response previews sharing matcher authority; +- `internal/core/auxreq` — bounded scheduler, submit-time pin handoff, detached child adapter; +- `internal/core/compactioncontinuity` — narrow BranchCoordinator; +- `internal/core/runtime` — request preview/open, final-response ordering, detached prepare/private child A-leg wiring; +- `internal/infra/runtimebundle` — process ownership and generation snapshot clients; +- `internal/plugins/features/compactioncontinuity` — config/capsule/carriers/sanitizer/eligibility/extractor/merge/injection; +- existing metering/billing/report projections only for content-free auxiliary workload classification; +- focused testkit/architecture fixtures. + +Individual provider/backend and frontend protocol packages should not need compaction-continuity-specific branches. ## Testing Strategy TDD order: -1. RED capsule merge/supersession/pruning and structured carrier fixtures. +1. RED capsule merge/supersession/pruning + carrier fixtures. 2. RED sanitizer/eligibility/extractor-schema tests. -3. RED BackgroundAux submit-time pin/coalescing/saturation/shutdown/goleak tests. -4. RED detached-session tests proving no primary secure-session/route-override effects. +3. RED BackgroundAux submit-time pin/coalescing/saturation/result-lifetime/shutdown tests. +4. RED detached-session/private-child-A-leg tests. 5. RED separate-route + user-billing/BillingCallID/protocol-usage tests. -6. RED compaction integration tests for successful-Open trigger, completion-only preview, parallel job, bounded barrier, plaintext/opaque behavior. -7. RED three-plus repeated-compaction/concurrency/generation-reload tests. -8. GREEN minimal implementation in the same order. -9. Repository quality, race, architecture and simplification gates. +6. RED detector request/response preview and final-release ordering tests. +7. RED strict/remote and completion-only compaction barriers, plaintext/opaque paths. +8. RED three-plus repeated-compaction/concurrency/generation-reload tests. +9. GREEN minimal implementation in that order. +10. repository quality/race/architecture/simplification gates. -No real model credentials are required. Extractor behavior uses deterministic fake/local backends returning fixture JSON. +No external model credentials are required; deterministic fake/local backends return fixture JSON. ## Design Invariants 1. One compaction recognition authority; no duplicate signature matrix. -2. Metadata observer remains non-mutating and content-free. -3. Extractor is off the primary session but billed to the originating user by default. -4. Extractor route is explicit and independent unless explicit inherit is configured. -5. Background job ownership is captured before the request spawn right ends. -6. ProcessServices, not a generation plugin lifecycle, owns worker lifetime. -7. Auxiliary child does not create/modify a primary secure-session turn. -8. Normal routing/B2BUA/usage/billing own child execution; no direct provider path. -9. Primary protocol-visible usage excludes extractor usage; account totals include it. -10. Continuity state is bounded, revisioned, branch-scoped and generation-reload-safe. -11. Opaque/encrypted compaction bytes are immutable. -12. First post-compaction turn has a bounded synchronization barrier when required. -13. No second transcript DB, money ledger, generic workflow engine, or redundant LLM summary pass. +2. Metadata observer remains content-free/non-mutating. +3. Committed detector sees the actual final released event after permitted preservation mutation. +4. Extractor is off primary session but billed to originating user by default. +5. Extractor route is explicit/independent unless explicit inherit. +6. Background ownership is captured before request spawn right ends. +7. ProcessServices owns worker/coordinator lifetime. +8. Detached child uses a private auxiliary A-leg and never creates a primary secure-session turn. +9. Normal routing/B2BUA/usage/billing own child execution; no direct provider path. +10. Primary protocol usage excludes child usage; account totals include it. +11. Continuity state is bounded, revisioned, branch-scoped and reload-safe. +12. Opaque/encrypted compaction bytes are immutable. +13. First post-compaction turn has a bounded barrier when required. +14. Pending late results remain useful only within a bounded branch retention window. +15. No second transcript DB, money ledger, generic workflow engine or redundant summary LLM pass. From d11b190b5af540739e5929cec99e1dd9369f5ca0 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:11:05 +0200 Subject: [PATCH 11/18] spec: reconcile compaction continuity gap analysis --- .../gap-analysis.md | 213 ++++++++++-------- 1 file changed, 117 insertions(+), 96 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/gap-analysis.md b/.kiro/specs/compaction-continuity-preservation/gap-analysis.md index 9b6aef7a..d9faef46 100644 --- a/.kiro/specs/compaction-continuity-preservation/gap-analysis.md +++ b/.kiro/specs/compaction-continuity-preservation/gap-analysis.md @@ -2,154 +2,175 @@ ## Result -**PASS after requirements corrections.** The feature can be implemented without a new provider client, second transcript database, second billing path, or general workflow runtime, but the initial requirements needed several brownfield constraints made more explicit. The most important gaps are lifecycle ownership for real background auxiliary work, strict isolation from the primary secure session, mutation ordering around the planned compaction detector, and the fact that canonical native compaction output is often opaque/encrypted rather than a mutable summary string. - -The corrections below are normative and must be reflected in the final `requirements.md`, `design.md`, and `tasks.md`. +**PASS after requirements and design-validation corrections.** The feature can be implemented without a new provider client, second transcript database, second billing path, or general workflow runtime. The correction loop made the real brownfield constraints explicit: #312 is a prerequisite; preservation mutation is separate from metadata observation; background auxiliary work needs submit-time generation ownership and ProcessServices lifetime; detached extraction needs a private child A-leg; late results must remain useful within a bounded branch window; and opaque/encrypted compaction state is reinjection-only. ## Existing Brownfield Facts -- `compaction-event-detection` is currently a merged **specification**, not a runtime capability on `main`. It defines a process-owned detector, metadata-only `compaction.Observer`, and request start emission only after successful upstream `Open`. -- Its detector design already owns the versioned compaction signature/history matrix and authoritative A-leg transaction state. Duplicating that matrix in continuity preservation would create drift and false disagreement. -- Current `pkg/lipsdk/auxiliary.Client` is synchronous. `Collect` delegates to `Stream`, which delegates to the normal runtime `Executor`. -- Current `internal/core/auxreq.Client` preserves the parent principal/scope, marks derived origin internal, supports `DisablePlugins`, and retains `genpin.KindAsync` **when the auxiliary call actually starts**. -- `genpin.Retainer.Retain` is explicitly a spawn-right operation: post-request-lease spawn attempts fail closed. Therefore a delayed worker cannot simply keep a request context and call synchronous `Aux.Collect` later. -- The current auxiliary child executes through the normal `Executor`. Without an explicit detached-session mode, it can enter secure-session BeginTurn/session recording paths; `Visibility` and `Role` are currently lineage metadata, not session-isolation authority. -- `lipapi.Call.Route.Selector` already provides a canonical route authority for a child call. The normal core router can therefore resolve a completely independent extractor selector without a second provider client. -- The billing identity adapter resolves account identity from the authenticated principal in `scope`. Because auxiliary execution already clones that scope, originating-user billing can reuse the existing billing path if detached execution preserves principal scope. -- `ProcessServices` owns process-lifetime services; `ExtensionState` is process-owned and survives generation reload. Feature bundle lifecycles, in contrast, belong to compiled feature/generation composition and are the wrong sole owner for a worker intended to outlive one immutable generation. -- `ExtensionState` is in-memory by default. `ScopeSession` partitions by `SessionView.PartitionKey()` and therefore by authoritative SessionID when available, but branch/A-leg must remain part of the feature key to avoid cross-branch aliasing. -- Secure-session transcript recording already exists when `TranscriptEnabled` is active. Creating another durable full transcript solely for this feature would violate existing extension-state and secure-session boundaries. -- `lipapi.CompactionItem` carries `EncryptedContent` and opaque provider data; it does **not** provide a universal plaintext summary field that can safely be appended to. -- Current `FeatureBundle` has no content-bearing compaction-preservation extension point. The #312 spec proposes only non-mutating `CompactionObservers`. +- `compaction-event-detection` is currently a merged **specification**, not runtime code on `main`. It defines process-owned A-leg detector state, one versioned rule matrix, metadata-only `compaction.Observer`, start only after successful upstream `Open`, and committed response observation at the final release seam. +- Current `pkg/lipsdk/auxiliary.Client` is synchronous. `Collect` delegates to `Stream` and the normal runtime Executor. +- `internal/core/auxreq.Client` already clones authenticated principal/scope, marks internal origin, supports plugin suppression and retains `genpin.KindAsync` when auxiliary execution starts. +- `genpin.Retainer` makes spawn rights explicit; attempting to retain after the request lease ends fails closed. +- Current auxiliary `Role`/`Visibility` are lineage metadata only. They do not suppress secure-session BeginTurn/turn transcript/activity or parent route-override authority. +- `lipapi.Call.Route.Selector` already gives a child canonical independent route that normal core routing can parse/execute. +- Billing account identity is principal-scope-derived. Preserving principal scope lets an independent child reuse current usage/billing authorities and obtain its own BillingCallID. +- `ProcessServices` owns process lifetime. `ExtensionState` survives immutable generation reload, while feature lifecycles are generation-composed. +- `ExtensionState` is process-local/in-memory by default and `ScopeSession` partitions by authoritative SessionID when available; A-leg/branch must remain an additional continuity key. +- Secure-session transcript storage already exists when explicitly enabled; it is the only appropriate durable historical recovery source in v1. +- `lipapi.CompactionItem` carries `EncryptedContent` and opaque provider data rather than a universal plaintext summary field. +- Current `FeatureBundle` has no content-bearing compaction-preservation surface; #312 proposes only non-mutating compaction observers. + +## Gaps and Corrections + +### G1 — #312 runtime capability is absent + +**Gap:** #344 could otherwise silently duplicate compaction signatures or pretend lifecycle events already exist. + +**Correction:** implementation order makes `compaction-event-detection` runtime a hard prerequisite. Enabled continuity composition fails clearly if preview/commit capability is absent. Disabled continuity remains compatible/no-op. + +### G2 — `compaction.Observer` cannot carry preservation content/mutation + +**Gap:** observer events intentionally contain no canonical request/response body and return no replacement decision. + +**Correction:** add a distinct `compaction.Preserver`-style slice, separately merged/frozen in FeatureBundle/runtime snapshot. Observer remains unchanged. + +### G3 — pre-open protection must not commit detector truth -## Gaps and Required Corrections +**Gap:** first post-compaction request may need continuity before B-leg Open, while detector start/completion commitment is intentionally tied to successful Open/final release. -### 1. #312 is a prerequisite, not an implemented dependency +**Correction:** factor pure request preview from the same matcher/fingerprint authority. Preview can guide a barrier but cannot mutate detector state, emit lifecycle events, or start a strict billable extraction merely from an unopened signature. -The initial feature request could be read as if compaction lifecycle events already exist in runtime. +### G4 — response preservation must not make `ResponseReleased` observe a pre-final event -**Correction:** implementation planning shall explicitly order the `compaction-event-detection` runtime capability first. This feature may extend/refactor that implementation, but shall not copy its rule catalog as a fallback. Tests for this spec may use a local fake/shared recognizer contract until the prerequisite lands. +**Gap:** an initial sequence of committed detector completion -> preservation mutation would violate #312's “event actually released” meaning. -### 2. Preservation mutation cannot be smuggled into `compaction.Observer` +**Correction:** add pure response preview and use final ordering: -The detector's observer contract is intentionally content-free and non-mutating. Preservation needs canonical content and, for verified paths, limited mutation. +```text +selected event + -> PreviewResponse (pure) + -> preservation finalization / safe plaintext mutation + -> ResponseReleased(final event) (commit) + -> metadata observers + -> client +``` -**Correction:** add a distinct additive preservation/interception contract. Keep observer payload and dispatch semantics unchanged. FeatureBundle/snapshot merge must expose preservation interceptors separately and defensively freeze them like other extension slices. +No ordinary response hook runs after preservation finalization. -### 3. Request preview and committed detection have different truth semantics +### G5 — synchronous `Aux.Collect` is not a safe background worker -A first post-compaction request may need continuity before its B-leg opens, while #312 deliberately commits request detection only after successful `Open`. +**Gap:** `go Aux.Collect(parentCtx)` can start after spawn authority is gone, inherit cancellation, leak across shutdown, and lacks bounded queue/result ownership. -**Correction:** if a pre-open preview is required, it must be a pure/non-committing view over the same matcher/fingerprint state. Preview may protect a preservation barrier but cannot emit lifecycle events, advance detector state, or establish a billable strict-start job. Successful Open remains the committed start boundary. +**Correction:** add a narrow process-owned BackgroundAux collector. `SubmitCollect` synchronously resolves the current runner, retains `KindAsync`, clones safe attribution and transfers ownership to a bounded scheduler. Workers use scheduler-rooted deadlines. Await/Forget operate on bounded job IDs/results. No arbitrary callbacks/functions/tasks are accepted. -### 4. Final-release mutation ordering must remain explicit +### G6 — worker lifetime cannot be generation-only -The #312 design places `ResponseReleased` at the final canonical release seam and guarantees observation does not mutate the event. Continuity may need to add a validated capsule to a verified plaintext carrier before client delivery. +**Gap:** generation-scoped feature lifecycle may retire while a job remains in flight. -**Correction:** detector derivation, preservation finalization, metadata-observer dispatch, and client release require one explicit ordering. The detector itself remains pure/non-mutating; the preservation stage is separate, and no ordinary response hook runs after it. Opaque/native content remains byte-identical. +**Correction:** ProcessServices owns scheduler and BranchCoordinator. Generation snapshots hold non-owning adapters. Each job retains exactly the generation it needs. -### 5. Synchronous `Aux.Collect` is insufficient for real background work +### G7 — detached child cannot reuse primary secure-session turn semantics -Launching `Aux.Collect` in a goroutine after the parent request returns is not safe: generation spawn rights can be gone, the parent context can be canceled, and shutdown cannot reliably own the goroutine. +**Gap:** normal Executor preparation would otherwise enter primary BeginTurn/transcript/activity/route-override paths. -**Correction:** introduce one narrow **background auxiliary collection** capability backed by a process-owned bounded scheduler. Submission synchronously captures the executor/generation right and retains `genpin.KindAsync` before returning. Worker execution uses a worker-owned context/deadline; Await/Forget use a bounded job ID/result registry. This is an auxiliary-execution primitive, not a generic task engine. +**Correction:** add trusted internal detached auxiliary mode. It preserves principal/scope but suppresses primary secure-session turn effects and parent route authority. -### 6. Worker lifetime belongs to `ProcessServices`, not only a feature lifecycle +### G8 — detached child still needs normal B2BUA/request authority -A generation-scoped plugin lifecycle can disappear on reload while a compaction extraction is still in flight. +**Gap:** making the child completely session/A-leg-less would force a second execution/billing path; reusing the parent A-leg would contaminate primary authority. -**Correction:** the scheduler is process-owned and closed through existing ProcessServices ownership/closer ordering. Generation snapshots receive non-owning clients. The job itself retains exactly the generation needed to execute its captured child call. +**Correction:** detached child creates/touches a **private child A-leg** via existing B2BUA semantics. Parent A-leg is lineage only. The child then gets ordinary request authority, private B-legs, separate BillingCallID, usage and provider COGS. -### 7. Normal auxiliary execution can accidentally create primary-session effects +### G9 — extractor route must remain independent -Current auxiliary lineage is not sufficient to prevent secure-session BeginTurn/transcript/last-activity effects. +**Gap:** parent A-leg route override could otherwise hijack the extractor model. -**Correction:** add a typed internal **detached auxiliary session mode** recognized by core execution. It preserves authenticated principal/scope and parent correlation but suppresses primary secure-session authority/turn recording, primary A-leg route overrides, and client-visible session history. This mode must be internal canonical execution metadata, not a provider-visible opaque extension. +**Correction:** child uses explicit configured `Call.Route.Selector` (or explicit inherit policy). Detached private A-leg has no inherited parent override. Normal router remains authoritative; no provider client bypass. -### 8. Independent extractor routing must not inherit primary A-leg override implicitly +### G10 — accounting needs workload distinction, not a new money path -The main user's A-leg may have a runtime routing override. Applying it to the internal extractor would violate the requested independently configurable model. +**Gap:** principal inheritance solves account attribution but operators/users need to separate continuity overhead from primary inference. -**Correction:** the detached child uses its explicitly configured canonical `Call.Route.Selector` (or explicit `inherit` policy) and a private auxiliary execution lineage. Parent A-leg ID is correlation only and cannot become route authority for the child. +**Correction:** project bounded content-free auxiliary workload/role (`compaction_continuity_extractor`) into existing metering/billing/report correlation. Child has separate BillingCallID/B-legs; primary protocol usage remains unchanged; account totals include child usage. -### 9. User billing can reuse current authority, but auxiliary workload identity must be projected +### G11 — `CompactionItem` cannot be treated as plaintext summary -Principal scope inheritance is sufficient for account identity, and routing through the normal Executor gives the child its own BillingCallID. However operators/users must be able to separate primary inference from continuity overhead. +**Gap:** mechanical append into `EncryptedContent`/opaque state can corrupt provider replay/continuation semantics. -**Correction:** add a bounded content-free workload/origin classification (`compaction_continuity_extractor`) to the accounting/metering/diagnostic correlation path. It must not become a second money ledger or pricing engine. Primary protocol usage remains unchanged while account-level billing includes the child. +**Correction:** result augmentation is allowlisted only for verified mutable plaintext carriers. Encrypted/opaque bytes are immutable. Mandatory safe fallback is proxy-owned first-post-compaction reinjection. -### 10. Provider-native compaction payloads are not universal summary text +### G12 — process state is reload-safe, not restart-durable -Mechanical append to `CompactionItem.EncryptedContent` or opaque data could corrupt replay/provider state. +**Gap:** ExtensionState is useful for v1 but cannot support a restart-survival claim. -**Correction:** result-side merge is permitted only for a verified mutable plaintext carrier. Otherwise the capsule is persisted as proxy state and injected into the first eligible post-compaction canonical request. Encrypted/opaque fields are byte-for-byte immutable under this feature. +**Correction:** v1 promises process/generation continuity only. Authorized existing secure-session transcript may be used through a narrow bounded reader to reconstruct after restart; otherwise missing capsule is fail-open. No second durable transcript/job/state platform. -### 11. Process extension state is reload-safe but not restart-durable +### G13 — session partition alone is not a branch identity -It is appropriate for revisions, sanitized windows, pending job IDs and injection watermarks within one process. It does not justify claiming durable continuity after restart. +**Gap:** `ScopeSession` can group multiple A-leg branches/forks. -**Correction:** define first implementation durability honestly. Process state survives generation reload. Across process restart, reconstruction is allowed from the existing authorized secure-session transcript when available; otherwise continuity state may be absent and behavior fails open. No new durable full transcript or generic durable feature-state framework is introduced by this spec. +**Correction:** continuity key is authoritative SessionID partition plus explicit A-leg/branch. Without secure SessionID, principal-isolated proxy-owned A-leg is authority. Client hints never select another branch. -### 12. `ScopeSession` alone is not a branch key +### G14 — per-generation mutex cannot protect capsule revisions -Authoritative SessionID is a useful partition, but one session can contain/relate to multiple A-leg branches or forks. +**Gap:** Store Get/Put has no CAS and a feature-instance lock disappears on reload. -**Correction:** capsule/source/job records use the authoritative session partition plus an explicit A-leg/branch key. When no secure SessionID exists, principal-isolated A-leg authority is used. Client hints never select another branch's capsule. +**Correction:** add a small process-owned BranchCoordinator that serializes revision/high-watermark/job/injection updates while using ExtensionState as serialized backing where practical. It treats capsule/source blobs opaquely and is not a generic transactional state framework. -### 13. Durable transcript access must stay behind a narrow authorized seam +### G15 — raw late result must remain useful without becoming durable memory -An official feature plugin should not import secure-session Bun/store internals merely to recover historical text. +**Gap:** if the completion barrier times out and result TTL is too short, a valid extraction can disappear before the next turn. -**Correction:** optional historical recovery uses a narrow authorized transcript-source/read adapter supplied by composition/core. Normal extraction prefers the canonical compaction baseline and process-local sanitized window, so most executions need no durable transcript read. +**Correction:** while BranchState references a PendingJobID, bounded raw result retention remains useful up to the configured pending continuity window; first consumption parses/merges and Forget deletes raw output. Branch/job expiry clears both coherently. No result outlives bounded continuity/source retention. -### 14. Raw generic background results should not become another sensitive store +### G16 — durable transcript access must stay behind authorization boundary -A process-owned scheduler may need to hold a completed auxiliary result until the preservation barrier consumes it. +**Gap:** feature code importing secure-session Bun/store internals would break layering/security. -**Correction:** background result storage is bounded/TTL-limited and never logged. Continuity parses/validates the small extractor response promptly and stores only the normalized capsule/delta afterward. Forget/expiry releases raw collected output. +**Correction:** optional restart/historical recovery uses a narrow authorized read adapter. Ordinary compaction uses current canonical baseline/process sanitized window first. -### 15. Disabling/config reload must not orphan already-billed work +### G17 — config reload/disable cannot erase submitted billing obligations -A generation reload can disable the feature or change extractor routing while a prior job has already submitted provider work. +**Gap:** disabling feature after provider submission cannot retroactively make the child free or orphan its generation. -**Correction:** in-flight jobs use a submission-time immutable configuration/execution snapshot and finish/settle or cancel through the normal captured generation. Disabling prevents new jobs but does not erase accounting obligations for submitted work. +**Correction:** jobs use immutable submission-time config/generation and complete/cancel/settle through captured authorities. New config only affects future jobs; disable stops new submissions. ## Brownfield Compatibility Matrix | Existing subsystem | Required treatment | |---|---| -| `compactiondetect` / #312 rule authority | prerequisite and shared matcher/state; not duplicated | +| #312 `compactiondetect` | prerequisite; shared request/response preview + committed state; no duplicate matrix | | `compaction.Observer` | unchanged metadata-only/non-mutating contract | -| `FeatureBundle` / request snapshot | additive separate preservation-interceptor slice | -| `internal/core/auxreq` | retain synchronous client; add narrow background collector/scheduler path | -| `genpin` | retain `KindAsync` synchronously at job submission; no post-lease spawn | -| `ProcessServices` | owns bounded background auxiliary scheduler lifetime | -| secure session | primary turn/session state untouched by detached extractor | -| B2BUA | parent IDs correlation only; child gets private auxiliary execution lineage | -| routing / route overrides | child selector parsed normally; parent A-leg override not inherited implicitly | -| billing / metering | existing authorities and separate child BillingCallID reused | -| protocol-visible usage | primary response usage unchanged; child usage separately recorded | -| `ExtensionState` | process-local capsule/source/job metadata; key additionally includes A-leg | -| secure-session transcript | optional authorized historical recovery; no second transcript DB | -| `CompactionItem` opaque/encrypted fields | immutable; never used as arbitrary text injection carrier | -| provider/frontends | no provider-specific continuity logic in core | -| generation reload | process state/scheduler survive; job uses submission-time generation/config | - -## Corrected Required Invariants - -1. Detection truth and preservation mutation are separate contracts sharing one recognition authority. -2. A strict compaction start is billable only after successful upstream Open; preview never emits/commits lifecycle truth. -3. Background extraction captures its execution generation and principal at submit time and is process-owned thereafter. -4. The extractor is detached from the primary secure-session turn while remaining financially attributable to the originating user. -5. Parent A-leg IDs are correlation, not extractor routing/session authority. -6. Every submitted extractor call uses normal Executor/BillingCallID/B-leg accounting; primary protocol usage stays separate. -7. Opaque/encrypted compaction state is never rewritten for continuity. -8. Capsule/source/job state survives generation reload but does not falsely claim process-restart durability. -9. Session partition plus explicit A-leg/branch identity prevents cross-branch capsule leakage. -10. No second provider client, transcript database, money ledger, or generic task framework is introduced. - -## Requirements Correction Status - -The final requirements must explicitly incorporate gaps 2–15, especially: additive preservation-interceptor merge/freeze semantics; process-owned background auxiliary scheduler; submit-time generation retention; detached auxiliary secure-session behavior; final-release stage ordering; workload classification for billing; branch keying; narrow transcript recovery; and honest restart durability. - -With those corrections applied, the requirements quality gate is **PASS** and design may proceed. +| FeatureBundle/snapshot | additive separately frozen preservation-interceptor slice | +| synchronous `auxiliary.Client` | retained source-compatible | +| new BackgroundAux | additive narrow model-collection scheduler only | +| `genpin` | `KindAsync` retained synchronously at submit | +| ProcessServices | owns scheduler + BranchCoordinator | +| ExtensionState | process backing for bounded serializable branch state | +| secure session | primary turn/transcript/activity untouched by detached child | +| B2BUA | private child A-leg/B-legs; parent IDs lineage only | +| routing | explicit child selector; parent override not inherited implicitly | +| usage/billing | existing authorities; separate child BillingCallID and workload class | +| frontend protocol usage | remains primary-call-only | +| secure transcript | optional authorized recovery only; no second DB | +| compaction opaque/encrypted fields | exact byte preservation | +| provider/frontends | no continuity-specific core/adapter branches | +| generation reload | process state survives; jobs use captured generation/config | + +## Corrected Invariants + +1. One compaction recognition authority; previews never commit lifecycle truth. +2. `compaction.Observer` remains metadata-only and preservation mutation is separate. +3. Committed `ResponseReleased` sees the actual final event after permitted preservation mutation. +4. Strict semantic extraction is not submitted before successful primary Open. +5. Background jobs acquire generation ownership before request spawn authority ends. +6. ProcessServices owns workers/branch synchronization across reload. +7. Detached extractor uses a private child A-leg, not the parent A-leg and not a second provider path. +8. Same authenticated user/account is billed by default; primary protocol usage stays separate. +9. Parent route override cannot silently change extractor route. +10. Opaque/encrypted compaction state is immutable; reinjection is universal fallback. +11. Branch state is bounded/revisioned/SessionID+A-leg scoped and reload-safe, not falsely restart-durable. +12. No second transcript DB, financial ledger, generic task framework, or second summary LLM pass. + +## Final Gate + +All identified requirements gaps and design-validation corrections are reflected in final `requirements.md` and `design.md`. The brownfield gate is **PASS** for TDD task generation. From 8ffdbe1b3c361cbef0756d01128f85a4e2233689 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:11:15 +0200 Subject: [PATCH 12/18] spec: mark compaction continuity design generated --- .kiro/specs/compaction-continuity-preservation/spec.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/spec.json b/.kiro/specs/compaction-continuity-preservation/spec.json index 0bb119d5..6e466df6 100644 --- a/.kiro/specs/compaction-continuity-preservation/spec.json +++ b/.kiro/specs/compaction-continuity-preservation/spec.json @@ -3,14 +3,14 @@ "created_at": "2026-08-17T14:48:00+02:00", "updated_at": "2026-08-17T14:48:00+02:00", "language": "en", - "phase": "requirements-generated", + "phase": "design-generated", "approvals": { "requirements": { "generated": true, "approved": false }, "design": { - "generated": false, + "generated": true, "approved": false }, "tasks": { From 69138b2389114154bbad5b3f9bd2008d17bc7875 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:12:50 +0200 Subject: [PATCH 13/18] spec: generate compaction continuity implementation tasks --- .../tasks.md | 419 ++++++++++++++++++ 1 file changed, 419 insertions(+) create mode 100644 .kiro/specs/compaction-continuity-preservation/tasks.md diff --git a/.kiro/specs/compaction-continuity-preservation/tasks.md b/.kiro/specs/compaction-continuity-preservation/tasks.md new file mode 100644 index 00000000..4adc4740 --- /dev/null +++ b/.kiro/specs/compaction-continuity-preservation/tasks.md @@ -0,0 +1,419 @@ +# Implementation Plan + +## Execution Rules + +- Follow TDD strictly: characterization/RED tests and contract gates precede production implementation in each dependency layer. +- `compaction-event-detection` runtime is a hard prerequisite. Do not duplicate its signature matrix if it is not yet implemented; land that spec first. +- Keep every task independently reviewable with **no more than five concrete actions**. +- Preserve existing routing, B2BUA, secure-session, generation pinning, output commitment, usage authority, BillingCallID accounting, and provider-opaque compaction semantics. +- Auxiliary semantic extraction is a separate off-session background model call, independently routed and billed to the originating user/account by default. +- Do not add provider-specific continuity branches, a direct provider client, second transcript store, feature-owned money ledger, generic task/workflow engine, or second LLM summary-rewrite pass. +- No real model credentials are required for correctness tests; use deterministic local/fake backends and fixture JSON. + +## Phase 1 — Freeze Contracts and Failure Boundaries With RED Tests + +### Task 1.1 — Freeze prerequisite detector preview and final-release semantics + +- Verify the #312 runtime detector exists; if absent, stop this spec's implementation and complete `compaction-event-detection` first rather than adding fallback signatures. +- Add RED tests for pure `PreviewRequest` and `PreviewResponse` sharing the same rule/fingerprint authority as committed detection without mutating detector state or emitting lifecycle events. +- Prove a strict preview on a request that never opens cannot create a started transaction or semantic extraction submission. +- Prove committed `ResponseReleased` receives the exact event after permitted preservation finalization, not a pre-final copy. +- Add near-miss and completion-only preview fixtures from the existing detector rule matrix. + +_Requirements: 1.1–1.13, 7.1, 11.3, 11.10_ + +_Validation: focused `internal/core/compactiondetect` RED tests; no duplicated rule catalog._ + +### Task 1.2 — Freeze the separate preservation extension contract + +- Add RED SDK tests for a content-bearing `compaction.Preserver`-style contract that is distinct from metadata-only `compaction.Observer`. +- Add RED FeatureBundle/merge/runtime-snapshot tests for ordered additive preservers, defensive copies, nil validation and frozen generation semantics. +- Prove ordinary compaction observer event payloads still contain no canonical request/response content and cannot mutate traffic. +- Freeze preservation callback ordering: BeforeRequest -> successful-open callback -> response-preview finalization -> committed detector observation -> metadata observer dispatch. +- Add architecture assertions preventing preservation mutation from being implemented as an ordinary response hook after the finalization seam. + +_Requirements: 1.2–1.5, 1.9–1.11, 11.12–11.13_ + +_Validation: `go test ./pkg/lipsdk/... ./internal/featurebundle/... ./internal/core/extensions/...` is RED for the new contract._ + +### Task 1.3 — Freeze background auxiliary scheduling and generation ownership + +- Add deterministic RED scheduler tests for bounded queue/concurrency, coalescing, result retention, Await/Forget, and saturation without goroutine fallback. +- Prove `SubmitCollect` retains `genpin.KindAsync` synchronously while the parent spawn right is live and a worker never retains a replacement generation later. +- Add enqueue-failure/cancel/timeout/shutdown paths proving every retained pin is released exactly once. +- Add a delayed-start test where the parent request context is canceled before the worker runs but the worker uses its own bounded context safely. +- Add `goleak` and race tests for queue saturation, shutdown, concurrent Await/Forget and late job completion. + +_Requirements: 4.4–4.15, 8.9–8.10, 11.2, 11.11_ + +_Validation: focused `internal/core/auxreq` tests are RED and deterministic._ + +### Task 1.4 — Freeze detached child A-leg, routing, and session isolation + +- Add RED tests proving a detached child creates a private auxiliary A-leg and never reuses the parent's A-leg as execution authority. +- Prove parent SessionID/A-leg/trace survive only as lineage while primary secure-session BeginTurn, transcript, activity and turn count remain unchanged. +- Prove the child uses its explicit extractor selector and a primary A-leg runtime route override cannot rewrite it. +- Prove detached mode is trusted auxiliary metadata unavailable from frontend/wire canonical fields. +- Cover private child B-leg failover/terminal lineage without client-visible session headers/history. + +_Requirements: 5.1–5.15, 8.1–8.3, 9.11, 11.4–11.5_ + +_Validation: runtime/auxreq/secure-session/B2BUA focused tests are RED._ + +### Task 1.5 — Freeze capsule semantics, billing attribution, and repeated-compaction outcomes + +- Add RED table tests for capsule revision, precedence, decision supersession, rejection, plan-step progress, stale merge rejection and deterministic pruning. +- Add RED structured carrier fixtures for Codex update-plan, OpenCode todo and supported Cline task-progress shapes plus near misses. +- Add RED billing tests requiring the child to inherit the originating account while receiving a separate BillingCallID/B-legs and auxiliary workload classification. +- Prove primary protocol-visible usage excludes extractor tokens while account/operator totals include the independent child usage. +- Add an end-to-end RED scenario with at least three compactions, user decision correction, completed/pending plan steps, dedupe and bounded capsule growth. + +_Requirements: 2.1–2.15, 3.1–3.7, 6.1–6.13, 8.4–8.12, 11.6–11.8_ + +_Validation: feature/billing/integration target tests are RED before production feature code._ + +## Phase 2 — Implement Minimal Shared Infrastructure + +### Task 2.1 — Implement pure detector request/response previews + +- Refactor prerequisite detector match/fingerprint code so preview and committed paths call one shared internal recognition authority. +- Implement `PreviewRequest` without state mutation and preserve existing successful-Open commit behavior in `RequestOpened`. +- Implement `PreviewResponse` without completion mutation and keep `ResponseReleased` as the committed final-event boundary. +- Make Task 1.1 positives/near misses/state-snapshot assertions green without changing observer payloads. +- Keep unsupported protocol controls and provider DTOs outside the detector. + +_Requirements: 1.1, 1.5–1.13_ + +_Design: D1; D13–D14_ + +_Validation: focused detector suites green; existing #312 tests remain unchanged/green._ + +### Task 2.2 — Implement and compose the preservation SDK surface + +- Add the separate preservation types/services/interfaces beside the existing compaction observer contract. +- Add `CompactionPreservers` to FeatureBundle, one merge surface and frozen request-snapshot accessor with existing validation conventions. +- Add core extension dispatch helpers that isolate preserver panic/error under configured preservation fail-open semantics. +- Wire the semantic stage order without introducing a mutating generic StageID or moving ordinary response hooks after finalization. +- Make Task 1.2 SDK/merge/order tests green. + +_Requirements: 1.2–1.4, 1.10–1.11, 10.3–10.5_ + +_Design: D2_ + +_Validation: SDK/featurebundle/extensions tests green._ + +### Task 2.3 — Implement the process-owned BackgroundAux collector + +- Add additive BackgroundClient/JobID/options APIs while keeping synchronous `auxiliary.Client` source-compatible. +- Implement the bounded `internal/core/auxreq` scheduler with keyed coalescing, worker pool, result registry and process-root context. +- Implement submit-time runner capture/KindAsync retention and exact cleanup on enqueue failure, terminal completion, cancellation and shutdown. +- Implement useful bounded pending-result retention plus Await/Forget without arbitrary callbacks or durable task semantics. +- Register scheduler ownership/Close through ProcessServices and make Task 1.3 green under race/goleak. + +_Requirements: 4.4–4.15, 8.9–8.10, 10.11–10.12_ + +_Design: D3; D22_ + +_Validation: focused auxreq/runtimebundle tests green; scheduler has no network dependency._ + +### Task 2.4 — Implement detached auxiliary execution with a private child A-leg + +- Add trusted internal detached-session policy on auxiliary execution without adding a frontend/wire `lipapi.Call` control. +- Factor Executor preparation so detached calls preserve principal/scope but skip primary secure-session BeginTurn/activity/transcript/resume effects. +- Create/touch a private child A-leg through existing B2BUA lifecycle and keep parent A-leg/session values as lineage only. +- Ensure detached route planning does not read the parent's route override and ordinary child request authority/B-legs still execute. +- Make Task 1.4 session/B2BUA/routing tests green without copying the Executor. + +_Requirements: 5.5–5.15, 9.11, 11.4–11.5, 11.13_ + +_Design: D4–D5_ + +_Validation: focused runtime/auxreq/B2BUA/secure-session tests green._ + +### Task 2.5 — Implement the process branch coordinator + +- Add a narrow ProcessServices-owned coordinator keyed by authoritative SessionID + A-leg/branch or principal-isolated A-leg fallback. +- Serialize branch revision/high-watermark/pending-job/injection updates and use process ExtensionState as opaque serialized backing where practical. +- Enforce max entries/TTL/lazy cleanup and never call model/provider/plugin code while coordinator locks are held. +- Expose only the preservation state operations needed by the compaction services; keep capsule/source semantics opaque to core. +- Add reload/concurrency tests proving old/new generations and workers cannot overwrite branch state out of order. + +_Requirements: 8.1–8.15, 10.12, 11.11–11.12_ + +_Design: D7; D22–D23_ + +_Validation: coordinator unit/race tests green and no generic transactional state framework appears._ + +## Phase 3 — Implement the Continuity Feature Semantics + +### Task 3.1 — Implement feature configuration and prerequisite validation + +- Register the official `compaction-continuity` feature and decode bounded preserve/extractor/worker/barrier/capsule/source/result/failure settings. +- Require explicit extractor route or explicit inherit when semantic extraction is enabled; feature remains disabled by default. +- Validate finite positive maxima and consistency of pending-result/source/branch retention. +- Fail enabled generation/startup composition clearly when detector preview/commit, BranchCoordinator or BackgroundAux services are absent. +- Add config reload tests proving in-flight jobs retain immutable submission-time config while new jobs use new generation config. + +_Requirements: 9.1–9.2, 10.1–10.3, 10.9–10.12_ + +_Design: Dependency Gate; D17–D19_ + +_Validation: config/runtimebundle feature tests green._ + +### Task 3.2 — Implement Continuity Capsule v1 and deterministic carrier rules + +- Implement versioned capsule/fact/plan types, stable IDs, source authority/status enums and strict validation. +- Implement deterministic precedence, stale-revision rejection, dedupe and whole-fact bounded pruning. +- Add versioned canonical carrier rules for the researched Codex/OpenCode/Cline plan shapes without agent-brand inference. +- Normalize supported carrier updates into capsule plan state without an LLM call. +- Make Task 1.5 capsule/carrier tests green including malformed/near-miss fixtures. + +_Requirements: 2.1–2.15, 3.1–3.5, 8.4, 8.11–8.12_ + +_Design: D8–D9_ + +_Validation: feature pure-unit tests green._ + +### Task 3.3 — Implement sanitized incremental source preparation and eligibility + +- Walk canonical calls into a bounded source envelope prioritizing user decisions, relevant assistant planning and recognized plan carriers. +- Drop/truncate ordinary tool results, logs, file/code dumps, media/binaries, unnecessary reasoning and unrelated external content. +- Mark any narrowly retained external/tool material as untrusted data and apply existing redaction/secret treatment before egress. +- Implement local semantic-eligibility heuristics that only decide whether to pay for extraction and never establish accepted intent themselves. +- Commit sanitized source/high-watermark only after successfully opened primary requests and make zero-call/no-candidate tests green. + +_Requirements: 3.6–3.16, 9.3–9.10_ + +_Design: D10; D12_ + +_Validation: sanitizer/eligibility privacy fixtures green with no external calls._ + +### Task 3.4 — Implement the semantic extractor child call and strict result parser + +- Build one detached no-tools auxiliary child using the configured independent selector, prior capsule, deterministic plan facts and sanitized delta. +- Suppress the continuity plugin and stamp private role/origin/parent lineage without primary session authority. +- Define the fixed extraction prompt/schema requiring explicit-user precedence, accepted/current plan semantics, supersession and ambiguity omission. +- Parse exactly one bounded JSON result with schema/base-revision/enums/depth/count/string/source-ref validation; discard malformed/authority-escalating output. +- Add fake-backend tests proving no second LLM summary-rewrite call is generated. + +_Requirements: 3.8–3.13, 5.1–5.9, 9.4–9.6, 10.6, 11.15_ + +_Design: D5; D11–D12_ + +_Validation: deterministic child-call/parser tests green._ + +### Task 3.5 — Integrate validated deltas with branch state and late results + +- On Await success, validate base revision then merge the extractor delta through the BranchCoordinator into a new bounded capsule revision. +- Reject/forget stale results without changing active decisions or reinjection watermarks. +- Keep PendingJobID/result useful across a timed-out response barrier until bounded next-turn consumption or coherent expiry. +- Immediately Forget raw collected output after validation/merge and store only normalized capsule/source/job metadata. +- Add concurrent explicit-user-correction versus late-worker tests proving newer intent always wins. + +_Requirements: 2.12–2.15, 4.13–4.15, 8.4–8.5, 10.12_ + +_Design: D3; D7–D8_ + +_Validation: branch/feature concurrency tests green under `-race`._ + +## Phase 4 — Integrate Preservation at Compaction Boundaries + +### Task 4.1 — Schedule extraction after a real strict compaction Open + +- At successful primary request Open, obtain committed detector events and invoke preservation with the effective canonical baseline/correlation. +- Commit the sanitized source and deterministic plan updates before deciding whether semantic extraction is necessary. +- Submit at most one coalesced semantic job for the detector transaction/source revision; never submit from an unopened strict preview. +- Let the primary compaction stream proceed immediately after submission without waiting for extractor completion. +- Cover retry/failover start dedupe and failed-Open zero-billing behavior. + +_Requirements: 1.5–1.8, 3.13, 4.9–4.10, 7.1–7.2, 11.3_ + +_Design: D13_ + +_Validation: runtime integration tests green with deterministic delayed extractor backend._ + +### Task 4.2 — Implement response-preview finalization and bounded completion barrier + +- Route each final selected event through pure detector `PreviewResponse` before committed `ResponseReleased`. +- Let preservation resolve/await the matching job only for the configured bounded barrier and merge a valid ready result. +- Keep ordinary hooks/gates/finalizers before this stage and committed detector observation/metadata observers/client delivery after it. +- On timeout/error, mark the appropriate pending state and continue fail-open without blocking indefinitely. +- Prove committed detector and client both see the exact post-preservation final event. + +_Requirements: 1.10–1.11, 7.2–7.5, 10.4–10.5, 11.10_ + +_Design: D1–D2; D13_ + +_Validation: final-stream release/ordering tests green across live/gated/recovery paths._ + +### Task 4.3 — Protect completion-only/local first post-compaction turns + +- Before B-leg Open, use pure request preview to recognize installed/completion-only local compaction without committing detector state. +- Load prior sanitized source/capsule and apply deterministic plan updates; submit one background job only if semantic eligibility requires it. +- Await only up to the bounded barrier and inject a valid ready capsule before the first post-compaction B-leg opens. +- Continue fail-open on timeout/failure and commit detector completion only after the primary request successfully opens. +- Cover reset/new-A-leg/near-miss cases so an unrelated short rewrite does not trigger extraction/injection. + +_Requirements: 1.8–1.9, 7.3–7.4, 7.8, 7.12–7.14, 8.5–8.7_ + +_Design: D14_ + +_Validation: local-compaction history/integration tests green._ + +### Task 4.4 — Implement authority-aware first-turn reinjection + +- Serialize the current capsule deterministically into one versioned/delimited bounded continuity block. +- Inject through a canonical helper that uses legal message-authority versus item-authority representation and preserves `Call.Validate` invariants. +- Track PendingInjectionRevision/LastInjectedRevision so one boundary/revision is injected at most once by default. +- Ensure injected text is proxy-owned prior context, not rewritten as a user message or copied into unrelated branch state. +- Add legacy-message and item-authority tests plus duplicate/retry/reload idempotency tests. + +_Requirements: 7.7–7.11, 8.1–8.5, 11.13_ + +_Design: D15_ + +_Validation: lipapi/runtime feature tests green for both canonical authorities._ + +### Task 4.5 — Implement verified plaintext augmentation and opaque exact-preservation fallback + +- Define the minimal verified plaintext continuation-carrier capability/matcher; default unknown/native compaction paths to no response mutation. +- Mechanically merge the ready capsule projection only on an allowed plaintext carrier; never use another LLM to rewrite it. +- Mark pending first-turn reinjection whenever carrier is opaque/unsupported or extractor result is not ready. +- Add exact byte comparisons for `CompactionItem.EncryptedContent`, `Opaque`, signatures and unknown extension blobs with feature enabled/disabled. +- Prove reinjection alone preserves continuity when result augmentation is unavailable. + +_Requirements: 7.5–7.7, 7.15, 11.9, 11.15_ + +_Design: D16_ + +_Validation: opaque/native protocol fixtures green with byte identity._ + +## Phase 5 — Complete Billing, Policy, Privacy, and Observability + +### Task 5.1 — Project auxiliary workload identity through normal billing/metering + +- Reuse existing auxiliary lineage as the source of a bounded workload class/role in usage, metering, billing and report correlation. +- Keep pricing/rating behavior unchanged unless existing operator policy explicitly differentiates the selected route/model. +- Prove the child receives its own BillingCallID and per-B-leg provider COGS while account identity remains the originating principal. +- Cover child failover, pre-submit credit rejection and submitted-but-discarded result accounting. +- Make Task 1.5 billing tests green without adding a feature-owned financial store. + +_Requirements: 6.1–6.13, 10.8, 11.6–11.7_ + +_Design: D6_ + +_Validation: focused billing/metering/report tests green._ + +### Task 5.2 — Certify primary protocol-usage and secure-session isolation + +- Prove primary frontend usage events/totals contain only primary call usage even when the extractor runs concurrently. +- Prove account/operator totals include both primary and auxiliary records and can distinguish continuity workload. +- Prove detached child causes no primary session transcript entry, TurnID increment, last-activity mutation, resume effect or client session header. +- Prove private child A-leg/attempt lineage remains available for operator/accounting diagnostics. +- Cover cancellation/timeout/failover paths for both parent and child without cross-settlement. + +_Requirements: 4.1–4.3, 5.10–5.15, 6.4–6.9, 11.4, 11.6_ + +_Validation: frontend/session/B2BUA/billing integration tests green._ + +### Task 5.3 — Implement trusted per-session policy and egress controls + +- Resolve effective continuity policy as operator hard maxima > trusted session values > global feature defaults. +- Allow only explicitly approved per-session enable/category/route/tighter-limit overrides; reject/ignore unauthenticated client attempts as specified. +- Apply existing redaction/secret treatment before semantic child egress and preserve tenant/workspace authorization on any optional transcript read. +- Keep transcript-disabled sessions from acquiring a hidden durable transcript and isolate bounded source by branch key. +- Add adversarial prompt-injection/secret/tool-output fixtures proving source text cannot override extractor instructions or leak excluded payloads. + +_Requirements: 9.1–9.13, 10.1–10.2_ + +_Design: D17; D20_ + +_Validation: feature/security/session-policy tests green._ + +### Task 5.4 — Implement failure handling and content-free observability + +- Add metrics for previews/events, carrier hits, eligibility skips, job queue/outcomes, token usage, capsule sizes/revisions, barriers, augmentation/reinjection and stale conflicts. +- Log only bounded IDs/hashes/status/counts and never extractor prompt/output/capsule text. +- Implement explicit fail-open handling for queue saturation, generation retain failure, child admission denial, provider failure, invalid schema, stale result and barrier timeout. +- Ensure disable/reload stops new jobs but preserves bounded completion/accounting of already-submitted work. +- Add tests proving no failure path spins/retries indefinitely, chooses an unconfigured model, or blocks shutdown indefinitely. + +_Requirements: 9.6–9.7, 10.3–10.12_ + +_Design: D19–D21_ + +_Validation: metrics/log/failure-path tests green._ + +### Task 5.5 — Document operator behavior, billing cost, and durability limits + +- Document feature enablement, independent extractor selector/model, worker/barrier/capsule bounds and fail-open semantics in standard configuration/operator docs. +- State explicitly that extractor inference is additional billable user-attributed usage and show how auxiliary cost is distinguished from primary inference. +- Document remote history egress/privacy implications and that the extractor is off-session/no-tools. +- Document v1 process/generation durability versus process-restart limitations and optional authorized transcript reconstruction behavior. +- Document #312 prerequisite and troubleshooting for missing detector/background services or rejected extractor billing/admission. + +_Requirements: 6.12–6.13, 8.13–8.15, 9.1–9.2, 10.1–10.3_ + +_Validation: docs/config examples pass repository docs/config validation._ + +## Phase 6 — Certify Repeated Compaction, Concurrency, ROI, and Simplicity + +### Task 6.1 — Run the full repeated-compaction semantic matrix + +- Exercise at least three successive compactions with accepted plan, product decisions, constraints, rationale, rejections and open questions. +- Change a decision after the first compaction and prove the older value remains superseded through later compactions. +- Advance plan steps through pending -> in-progress -> completed and prove pending/current work survives while completed history can be pruned. +- Exercise deterministic-only, semantic-extractor and mixed paths and prove no duplicate model call when deterministic state is sufficient. +- Verify capsule/prompt size remains bounded without monotonically duplicated facts/injection blocks. + +_Requirements: 2.6–2.14, 3.5–3.7, 8.10–8.12, 11.8_ + +_Validation: deterministic multi-compaction integration suite green._ + +### Task 6.2 — Certify concurrency, generation reload, and stale-result safety + +- Race concurrent primary turns, late extractor completion and explicit user correction on one branch; newer explicit intent must win. +- Reload to a new immutable generation while an extractor job is queued/running and prove BranchCoordinator/job ownership survives correctly. +- Change/disable extractor config across reload and prove old jobs retain captured route/budgets while new jobs use new policy or do not submit. +- Exercise branch reset/new A-leg/fork-no-parent and prove no capsule/job leakage between branches. +- Verify pending result/job expiry coherently clears BranchState without stale injection. + +_Requirements: 4.13–4.15, 8.1–8.10, 10.9–10.12, 11.11_ + +_Validation: focused `-race` reload/concurrency suite green._ + +### Task 6.3 — Certify worker/resource shutdown and leak freedom + +- Run queue saturation, worker timeout, client cancellation, process shutdown and late completion under `-race` and goleak. +- Assert every successful background submission retains/releases exactly one `KindAsync` pin and no job starts after scheduler close linearizes. +- Assert no provider/model call or plugin callback runs while scheduler/BranchCoordinator internal locks are held. +- Assert bounded queue/result/branch state under sustained synthetic compaction load. +- Run existing generation pin/ProcessServices shutdown tests to ensure no ownership regression. + +_Requirements: 4.4–4.15, 8.9, 11.2, 11.11–11.13_ + +_Validation: race/goleak/process lifecycle tests green._ + +### Task 6.4 — Run architecture and security scope gates + +- Add/execute architecture tests forbidding provider/frontend DTO imports, direct provider clients and provider-specific continuity branches in core/feature packages. +- Prove detached mode cannot be set through frontend/wire fields and continuity source/capsule/raw result is absent from ordinary logs/money records. +- Prove no second transcript database, feature-owned money ledger, generic arbitrary-task scheduler/service locator or durable job framework was added. +- Prove prerequisite detector retains one signature authority and metadata observers remain content-free/non-mutating. +- Prove opaque/encrypted compaction bytes and existing no-retry-after-output/request authority semantics remain unchanged. + +_Requirements: 1.1–1.4, 7.6, 9.3–9.13, 11.9, 11.12–11.15_ + +_Validation: `go test ./internal/archtest/...` plus security/contract suites green._ + +### Task 6.5 — Run final repository gates and simplification review + +- Run focused packages, `make quality-checks`, `make test-unit`, required race/goleak suites and deterministic config/docs checks without external model credentials. +- Review implementation diff for redundant abstractions, duplicate rule/state ownership, unbounded work, hidden provider/session/billing bypasses and unnecessary public API expansion. +- Confirm normal requests with feature disabled/no candidate state incur only negligible bounded checks and no auxiliary model calls. +- Record implementation evidence separating UX/cost trade-off: preserved continuity and extra billed auxiliary usage/latency only at bounded barriers. +- If the feature requires a general workflow engine, second money path, or unsafe opaque mutation to pass tests, re-scope rather than weaken the frozen requirements. + +_Requirements: 10.3–10.12, 11.1–11.15_ + +_Validation: full repository release-quality gates green; final architecture remains within the spec boundaries._ From 536e920c1bd4b50a238e15678f109f33bb1c244c Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 15:13:02 +0200 Subject: [PATCH 14/18] spec: mark compaction continuity tasks generated --- .kiro/specs/compaction-continuity-preservation/spec.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/spec.json b/.kiro/specs/compaction-continuity-preservation/spec.json index 6e466df6..b8068c2d 100644 --- a/.kiro/specs/compaction-continuity-preservation/spec.json +++ b/.kiro/specs/compaction-continuity-preservation/spec.json @@ -3,7 +3,7 @@ "created_at": "2026-08-17T14:48:00+02:00", "updated_at": "2026-08-17T14:48:00+02:00", "language": "en", - "phase": "design-generated", + "phase": "tasks-generated", "approvals": { "requirements": { "generated": true, @@ -14,7 +14,7 @@ "approved": false }, "tasks": { - "generated": false, + "generated": true, "approved": false } }, From fe2eb6f1c58a95d8b6221cf240641347ecad6130 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Tue, 18 Aug 2026 20:31:42 +0200 Subject: [PATCH 15/18] spec: address CodeRabbit continuity requirements review --- .../requirements.md | 34 ++++++++++++------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/requirements.md b/.kiro/specs/compaction-continuity-preservation/requirements.md index 3b37a2f5..1c75e94d 100644 --- a/.kiro/specs/compaction-continuity-preservation/requirements.md +++ b/.kiro/specs/compaction-continuity-preservation/requirements.md @@ -27,7 +27,7 @@ This specification depends on the compaction-recognition authority defined by th 1.6. A strict start candidate shall not cause a billable semantic-extraction job if the compaction-looking request never successfully opens upstream. 1.7. When a recognized compaction request successfully opens upstream, preservation may start one extraction job for that logical transaction while primary compaction work continues independently. 1.8. Completion-only/history evidence shall never invent a historical start, but it may trigger extraction/reinjection required for a compaction only observable after installation. -1.9. Any pre-open request preview needed to protect the first post-compaction turn shall be a pure/non-committing view over the same matcher/fingerprint authority; preview shall not emit lifecycle events, advance detector state, or establish a strict-start billing trigger. +1.9. Any pre-open request preview needed to protect the first post-compaction turn shall be a pure/non-committing view over the same matcher/fingerprint authority; preview shall not emit lifecycle events, advance detector state, or establish a billable semantic-extraction trigger. 1.10. Response-side preservation shall use a pure/non-committing response preview over the same detector authority to identify a potential completion before mutation; preview shall not mark a transaction complete or dispatch observers. 1.11. Final-release ordering shall be: selected canonical event -> pure response preview -> separate preservation finalization -> committed `ResponseReleased` on the resulting final event -> metadata-observer dispatch -> client release. No ordinary response hook may run after preservation finalization. 1.12. Provider/frontend wire DTOs and provider SDK types shall not enter core continuity logic. @@ -50,6 +50,8 @@ This specification depends on the compaction-recognition authority defined by th 2.13. Capsule size shall be bounded by configured byte/token-equivalent limits; overflow shall prioritize active decisions/constraints, pending/in-progress plan steps, useful rationale, then historical/completed material. 2.14. Completed steps and superseded history may be condensed/dropped under retention policy while active decisions and unresolved work remain until superseded/removed or the branch ends. 2.15. The capsule is continuity state, not an audit transcript, and shall not contain arbitrary logs, file dumps, binaries, credentials, or unrelated tool output. +2.16. The serialized capsule envelope shall be self-binding and self-validating: it shall carry a stable content-free branch binding plus a digest computed over a canonical representation of the complete versioned envelope excluding only the digest field itself. The branch binding and digest shall be checked before registry transfer consumption, merge, reinjection/projection, recovery, or generation-reload reuse; raw branch/account identifiers need not be exposed to the extractor model merely to satisfy this invariant. +2.17. Every active decision shall have deterministic conflict identity independent of its extractor-generated fact ID. The implementation shall use a stable normalized conflict key and may additionally use validated `supersedes` references; at most one decision may remain active for one conflict key, and a correction that targets an existing active decision shall deactivate/supersede that decision rather than coexist under a new ID. ## Requirement 3: Deterministic-First Extraction and Bounded Source Preparation @@ -81,12 +83,13 @@ This specification depends on the compaction-recognition authority defined by th 4.7. Submission shall synchronously resolve/capture the executable generation and retain `genpin.KindAsync` before returning; a delayed worker shall not attempt a new spawn after the originating request lease is gone. 4.8. Worker execution shall use an independent worker-owned context/deadline carrying only cloned required principal/scope/correlation values, not the canceled parent request context as its lifetime root. 4.9. The process-owned scheduler shall expose a narrow background-auxiliary collection contract with bounded job IDs/results and await/forget behavior; it shall not become a generic arbitrary task engine. -4.10. Equivalent jobs for the same authoritative branch, compaction transaction, and target source revision shall coalesce/idempotently reuse one submission so retries/failover cannot duplicate billable extraction calls. +4.10. Equivalent submitted jobs shall coalesce by authoritative **parent continuity branch + committed compaction transaction + target source revision**. When a completion-only pure preview exists before any transaction is committed, the feature shall derive a stable non-billable preview intent key from the authoritative parent branch, detector-owned preview boundary/fingerprint, and target source revision; an empty transaction ID shall never be used as a billable submission key. After successful primary Open, that preview intent shall bind to the committed transaction before any new `SubmitCollect`, and retries/failover shall reuse the bound identity. 4.11. Queue saturation, unavailable worker, shutdown, or failed generation retention shall follow preservation failure policy and shall not fall back to an unbounded goroutine/direct provider call. 4.12. Process shutdown shall stop admission, cancel/join worker execution, release generation pins exactly once, and satisfy repository goleak/race requirements. 4.13. Job results shall be revision-checked before merge; stale jobs cannot overwrite newer intent/capsule revisions. 4.14. Completed raw auxiliary results shall be bounded by count/bytes/TTL, never logged, and forgotten immediately after validation/merge. While a BranchState still references a pending useful JobID, its result retention shall be long enough for the configured bounded continuity/pending window rather than an arbitrary short cache TTL; branch/job expiry shall clear both sides coherently. 4.15. Disabling/reloading the feature shall prevent new jobs according to the new configuration but shall not erase accounting obligations or leak execution for provider work already submitted. +4.16. The authoritative parent continuity BranchKey shall be captured before detached child execution begins and stored with pending job state. Await, result validation/merge, capsule revision, pending injection, and reinjection shall use that parent key; the private child A-leg exists only for auxiliary execution/routing/billing and shall never become the continuity-state branch key. ## Requirement 5: Independently Configurable Route and Detached Session Semantics @@ -121,32 +124,34 @@ This specification depends on the compaction-recognition authority defined by th 6.11. If upstream extractor work was submitted, resulting usage remains billable/accountable even if the result is late, invalid, or discarded as stale. 6.12. Enabling the feature shall be documented as potentially generating extra billed inference beyond visible primary turns. 6.13. Operator-funded/system-funded extraction is outside the first implementation and, if later added, must be explicit opt-in accounting policy. +6.14. Every auxiliary B-leg and failover leg that reaches independent terminal accounting shall carry a positive authoritative `AttemptSeq` and valid final billing evidence under the existing billing contract: usage quantities/presence, cost evidence/presence, `Source`, `Authority`, and `DedupeKey`. A record with `AttemptSeq <= 0` or otherwise invalid accounting identity/evidence shall not be treated as successfully accounted merely because the auxiliary BillingCallID is distinct. ## Requirement 7: Compaction Integration, Barriers, Augmentation, and Reinjection -7.1. For a strict recognized compaction request, semantic extraction shall start no earlier than successful primary compaction B-leg Open, unless completion-only/local evidence is first observable on a request that must be protected pre-open. +7.1. A **new billable semantic-extraction job**, whether triggered by strict start or completion-only/local evidence, shall start no earlier than successful primary B-leg Open. Pre-open completion-only handling may use deterministic state, await a previously submitted matching job, or create a non-billable preview intent, but shall not submit fresh provider work. 7.2. After submission, extraction shall run concurrently with primary compaction work where possible rather than serially blocking the main B-leg for its full inference duration. 7.3. Preservation may await a background job only at a narrow bounded barrier where its result is needed to protect a compaction result or first eligible post-compaction turn. 7.4. Barrier timeout obeys configured failure policy; default fail-open continues native compaction/continuation rather than deadlocking. 7.5. A completed validated capsule may be mechanically added to a compaction result only for a verified mutable plaintext continuation-summary carrier. 7.6. `CompactionItem.EncryptedContent`, opaque provider blobs, signatures, encrypted state, and unknown native compaction payloads shall remain byte-identical. 7.7. If result-side augmentation is unsafe/unavailable, the capsule shall be marked for proxy-owned reinjection on the first eligible post-compaction request. -7.8. Before that first B-leg opens, preservation shall use an already-ready capsule or await its matching in-flight job up to the configured barrier. +7.8. Before that first B-leg opens, preservation shall use an already-ready capsule or await a matching job that was submitted by an earlier successfully opened request up to the configured barrier; it shall not create new billable semantic provider work solely to satisfy the pre-open barrier. 7.9. Reinjection shall be canonical-authority-aware: legacy/message-authoritative and item-authoritative calls shall receive a valid bounded proxy-owned instruction/message representation without violating `Call.Validate` authority rules. 7.10. The continuity block shall be versioned/delimited/bounded and distinguishable from user text; it shall state that facts are prior continuation state, not a new user request. -7.11. One capsule revision shall not be injected repeatedly for the same boundary absent an explicit repeated-injection policy. -7.12. Completion-only/local/history compaction first recognized on the post-compaction request may schedule background extraction from the bounded prior source and synchronize before opening that request. +7.11. Reinjection deduplication shall use a compound watermark containing authoritative branch binding, compaction boundary/transaction identity, and capsule revision. The same revision may therefore be reinjected for a later distinct compaction boundary, while one boundary/revision is not duplicated by retries. +7.12. Completion-only/local/history compaction first recognized on the post-compaction request may prepare a non-billable preview intent and deterministic capsule before Open. Only after that request successfully opens and detector state commits may the intent bind to the committed transaction and submit one coalesced semantic job; a failed Open produces no billable child job. 7.13. If deterministic extraction already supplies the necessary capsule, no semantic model job/barrier is required. 7.14. Preservation errors shall not change route selection, failover, no-retry-after-output, or output commitment except for the explicitly bounded pre-output barrier and canonical continuity injection. 7.15. No second LLM pass shall normally rewrite/improve the native summary after one validated semantic extractor result already exists. +7.16. `BeforeRequest`, `RequestOpened`, and `BeforeResponseRelease` errors/panics shall be recorded as preservation failures and shall not propagate into primary request/compaction/response handling. Any canonical request/event mutation attempted by preservation shall be transactional at the preservation seam: if the callback fails, panics, or leaves an invalid canonical object, the runtime shall restore the pre-preservation object and continue native traffic fail-open. ## Requirement 8: Repeated Compactions, Branch Scope, Concurrency, Reload, and Restart -8.1. Continuity state shall be keyed by authoritative SessionID when available plus explicit A-leg/branch identity; client session hints alone shall never select another branch's capsule. +8.1. Continuity state shall be keyed by authoritative SessionID when available plus the **parent primary A-leg/branch identity** captured before any detached child A-leg is created; client session hints and child auxiliary A-leg IDs alone shall never select another branch's capsule. 8.2. Without secure-session authority, principal-isolated proxy-owned A-leg authority shall be used rather than arbitrary client hints. 8.3. `ScopeSession` may provide the session partition, but the feature key shall additionally include A-leg/branch identity to prevent branch aliasing. 8.4. Capsule updates shall use monotonic revision/compare-and-merge semantics so concurrent turns/jobs cannot overwrite newer explicit intent. -8.5. Duplicate lifecycle signals and B-leg retry/failover shall be idempotent for job submission, merge, and reinjection. +8.5. Duplicate lifecycle signals and B-leg retry/failover shall be idempotent for job submission and merge; reinjection idempotency shall be scoped to its compound branch/boundary/revision watermark rather than revision alone. 8.6. New unrelated sessions/A-legs start without inherited state. 8.7. Reset/branch replacement shall retire/cancel pending old-branch work under bounded cleanup and shall not leak decisions into the new branch. 8.8. Fork/clone inheritance, if supported, shall be explicit copy-on-fork from a known parent revision; no explicit parent relationship means no inheritance. @@ -179,8 +184,8 @@ This specification depends on the compaction-recognition authority defined by th 10.1. Global feature config shall support at least enablement, preserved categories, extractor route, timeout/input/output limits, worker concurrency/queue bounds, barrier timeout, capsule/source/result bounds/TTL, and failure mode. 10.2. Semantic extraction requires an explicit route or explicit inherit policy; no accidental main-model default is allowed. 10.3. Enabling the feature shall fail generation/startup composition clearly if the prerequisite compaction detector preview/commit service, process branch coordinator, or BackgroundAux capability is unavailable; disabled feature mode shall retain no-op compatibility. -10.4. Default request-time failure mode shall be fail-open for model traffic: extractor timeout/error/saturation/invalid output/state failure does not make native compaction unusable. -10.5. Fail-open preserves already-valid deterministic capsule state and never injects malformed/partial extractor output. +10.4. Default request-time failure mode shall be fail-open for model traffic: extractor timeout/error/saturation/invalid output/state failure **and preservation callback error/panic** do not make native compaction unusable. +10.5. Fail-open preserves already-valid deterministic capsule state, never injects malformed/partial extractor output, and restores the pre-preservation canonical request/event if a preservation mutation fails validation or the callback itself fails before the mutation is committed. 10.6. Extractor output shall be validated against a strict versioned schema with byte/depth/count limits before merge/injection. 10.7. Observability shall count compaction candidates; deterministic hits; semantic jobs submitted/coalesced/skipped/saturated; latency/outcome; extractor input/output tokens; capsule revision/size/fact counts; barrier waits/timeouts; augmentation/reinjection; stale/conflict rejection. 10.8. Billing/usage observability shall expose auxiliary continuity cost through existing accounting stores rather than duplicating financial truth. @@ -188,17 +193,18 @@ This specification depends on the compaction-recognition authority defined by th 10.10. Disabling the feature prevents new jobs while allowing bounded cleanup/accounting of already-submitted work. 10.11. No failure path may retry indefinitely, spin, block shutdown indefinitely, or fall back to an unconfigured provider/model. 10.12. Background raw-result retention shall be bounded and useful only while referenced/pending under the configured continuity window; normalized capsule state replaces raw result after consumption. +10.13. Content-free preservation-failure diagnostics shall identify callback stage and outcome without logging prompt/capsule content; callback failure shall not be mistaken for primary model/provider failure or influence retry authority. ## Requirement 11: TDD, Architecture, and Non-Interference Gates 11.1. Implementation shall begin with RED tests for capsule merge/supersession, structured carriers, eligibility/sanitization, background lifetime, detached session behavior, independent routing, billing attribution, barriers, repeated compactions, and opaque payload protection. 11.2. Worker packages shall use deterministic scheduling controls plus `goleak`/race tests for saturation, cancellation, shutdown, duplicate submission, stale completion, and barrier races. -11.3. Tests shall prove no extraction job is submitted when a strict compaction-looking request fails before upstream Open. +11.3. Tests shall prove no billable semantic extraction job is submitted when either a strict compaction-looking request **or a completion-only first post-compaction request** fails before upstream Open; completion-only pre-open intent creation/deduplication must remain non-billable. 11.4. Tests shall prove the child uses a private auxiliary A-leg, is absent from primary secure-session transcript/turn count/last activity, and does not mutate primary A-leg route-override/session state. 11.5. Tests shall prove extractor routing can differ from primary routing and that primary A-leg overrides do not rewrite the child selector. -11.6. Billing tests shall prove separate auxiliary BillingCallID/B-leg usage, originating account attribution, aggregate inclusion, continuity workload classification, and unchanged primary protocol usage. +11.6. Billing tests shall prove separate auxiliary BillingCallID/B-leg usage, originating account attribution, aggregate inclusion, continuity workload classification, unchanged primary protocol usage, positive `AttemptSeq` on every auxiliary/failover B-leg, and valid usage/cost/`Source`/`Authority`/`DedupeKey` evidence under the existing terminal-accounting contract. Coverage shall include rejection of `AttemptSeq <= 0` rather than silently treating such a record as accounted. 11.7. Tests shall cover pre-submit billing rejection, submitted-but-invalid result, submitted-but-stale result, retry/failover cost, and fail-open behavior. -11.8. Tests shall cover at least three successive compactions with decision supersession, plan progress, dedupe, and bounded growth. +11.8. Tests shall cover at least three successive compactions with deterministic decision conflict keys/validated supersession, plan progress, dedupe, capsule branch-binding/digest validation, and bounded growth. 11.9. Tests shall prove encrypted/opaque compaction content is byte-identical with preservation enabled. 11.10. Tests shall prove response preview does not commit lifecycle state and committed `ResponseReleased` sees the exact final event after any permitted plaintext preservation mutation. 11.11. Tests shall prove process generation reload cannot orphan jobs/state and that job submission captured `KindAsync` ownership before the request spawn right ended. @@ -206,3 +212,5 @@ This specification depends on the compaction-recognition authority defined by th 11.13. The implementation shall preserve canonical call/event validity, output commitment/no-retry-after-output, B2BUA lineage, secure-session ownership, generation pinning, and current billing settlement authorities. 11.14. Focused/repository tests shall run without external model credentials; live extractor tests are optional environment-gated evidence. 11.15. Final review shall remove duplicate rule catalogs, unnecessary persistence/framework layers, unbounded queues/goroutines, provider-specific core branches, and any redundant second LLM summary-rewrite pass. +11.16. Tests shall use distinct parent and child A-leg IDs and prove pending jobs, Await, merge, revision, late-result handling, and reinjection remain attached to the captured parent continuity BranchKey across concurrency and reload. +11.17. Reinjection tests shall cover two successive compaction boundaries with the same capsule revision, canonical validation failure, failed primary Open, aborted/no client release, then retry. The compound watermark shall advance only after successful final client release; pending injection shall survive earlier failures, and call-local retry/failover shall not duplicate insertion within one primary attempt lifecycle. From 11d0c0bb905fea75a99c1553e30b61316a494651 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Tue, 18 Aug 2026 20:33:28 +0200 Subject: [PATCH 16/18] spec: address CodeRabbit continuity design review --- .../design.md | 305 +++++++++++++----- 1 file changed, 222 insertions(+), 83 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/design.md b/.kiro/specs/compaction-continuity-preservation/design.md index 3d428359..be8160be 100644 --- a/.kiro/specs/compaction-continuity-preservation/design.md +++ b/.kiro/specs/compaction-continuity-preservation/design.md @@ -23,7 +23,7 @@ The semantic extractor runs independently from the primary coding turn and may u - Run semantic extraction off-session/background with bounded process ownership. - Allow a separately configured extractor route independent of the primary session route. - Attribute auxiliary usage/cost to the originating user through existing billing authorities. -- Prevent the first post-compaction turn from outrunning required continuity state. +- Prevent the first post-compaction turn from outrunning already-ready or deterministically recoverable continuity state without submitting fresh billable provider work before its primary Open. - Keep provider-native encrypted/opaque compaction content byte-identical. - Keep the feature bounded, fail-open and content-safe in observability. @@ -80,8 +80,8 @@ Continuity must never recreate the #312 signature matrix as a fallback. | +--> Preserver.BeforeRequest | pending capsule/job? - | bounded Await only when needed - +--> canonical reinjection + | bounded Await only for previously submitted work + +--> canonical reinjection (transactional helper) | normal prepare / route / billing / Open | @@ -92,7 +92,7 @@ Continuity must never recreate the #312 signature matrix as a fallback. | deterministic harvest / sanitizer +--> BackgroundAux.SubmitCollect --------+ | - primary compaction stream continues v + primary stream continues v bounded workers | detached child call @@ -122,6 +122,8 @@ Continuity must never recreate the #312 signature matrix as a fallback. metadata-only compaction observers | client release + | + successful release commits branch-level reinjection watermark ``` ## D1. Pure Preview Versus Committed Detector State @@ -136,6 +138,7 @@ type RequestPreview struct { RuleID string Kind PreviewKind // none | start_candidate | completion_candidate TransactionID string // only when safely derivable from existing active state + BoundaryFingerprint string // stable detector-owned preview identity when no transaction exists } type ResponsePreview struct { @@ -155,7 +158,8 @@ Preview properties: - no transaction/fingerprint mutation; - no lifecycle event emission; - no observer dispatch; -- request preview cannot establish a billable strict start; +- request preview cannot establish a billable semantic-extraction start; +- completion-only preview can expose a stable boundary fingerprint for a non-billable intent before a committed transaction exists; - response preview lets preservation identify the matching job before final response mutation. Committed boundaries remain: @@ -200,6 +204,16 @@ Semantics: - `RequestOpened`: successful-open source commit and background job scheduling; current primary request has already been sent upstream. - `BeforeResponseRelease`: pure-preview-guided bounded join and verified result-side augmentation before committed `ResponseReleased`. +The interface remains `error`-returning, but **preserver errors are not primary-traffic errors**. Core dispatch owns the following composition rule: + +1. invoke each preserver behind panic isolation; +2. on callback error/panic, emit only content-free preservation diagnostics and continue native traffic; +3. `BeforeRequest` and `BeforeResponseRelease` mutation runs against a transactional helper/snapshot so callback error, panic, or post-mutation canonical validation restores the exact pre-preservation `Call`/`Event` before continuing; +4. `RequestOpened` failure cannot undo the already-opened primary request and therefore only records preservation failure/cleans feature-local pending state; +5. preservation failure never becomes route/failover/no-retry authority and never masquerades as a provider failure. + +A practical implementation can clone the bounded canonical mutation target before dispatch or make the continuity helper return an undo/commit handle; do not add a general transaction framework. + `FeatureBundle` gains `CompactionPreservers []compaction.Preserver`; the single merge surface concatenates in registration order and the runtime snapshot exposes a frozen defensive copy. No request/response content is added to `compaction.Event` and `Observer` remains unchanged. @@ -243,6 +257,14 @@ BackgroundScheduler It accepts only canonical auxiliary model collection jobs, never arbitrary Go callbacks/tasks/timers. +The scheduler only sees **committed billable job identities**. A completion-only preview that has no committed transaction is represented in BranchCoordinator as a non-billable `PreviewIntentKey`; it is not inserted into the scheduler coalescing map and does not retain a generation pin. After successful primary Open, the preview intent binds to the committed transaction and produces the normal scheduler key: + +```text +coalesce_key = H(parent_branch_binding | committed_transaction_id | target_source_revision) +``` + +An empty transaction ID is invalid for a new billable continuity submission. + ### Submit-time ownership transfer `SubmitCollect` synchronously: @@ -250,18 +272,20 @@ It accepts only canonical auxiliary model collection jobs, never arbitrary Go ca 1. validates/clones the child request; 2. resolves the current generation `ExecutorRunner` from the request snapshot cell; 3. obtains `genpin.Retainer` and `Retain(KindAsync)` while spawn authority is live; -4. clones required principal/scope/correlation attribution; -5. reserves/coalesces the job; +4. clones required principal/scope/correlation attribution plus the **captured parent continuity branch binding** as opaque lineage; +5. reserves/coalesces the committed job; 6. enqueues or fails atomically; 7. releases a newly retained pin immediately on failed handoff. A worker never attempts a later generation retain. The captured pin is released exactly once after terminal child collection/cancel. +The captured parent branch binding is not execution authority for the child. It exists so feature logic can associate pending results with the primary continuity branch; the private child A-leg created later must never replace it. + ### Worker context Worker context derives from the scheduler/process root and copied attribution, not from parent request cancellation. A per-job timeout bounds inference. -Preserve principal/scope, internal origin, parent correlation and captured generation. Do not preserve primary resume/session authority or primary A-leg route authority. +Preserve principal/scope, internal origin, parent correlation, captured parent continuity branch binding and captured generation. Do not preserve primary resume/session authority or primary A-leg route authority. ### Result retention @@ -284,25 +308,32 @@ This is carried on the trusted auxiliary request object after frontend decode; n Detached child flow: ```text -originating principal/scope - | - v -private auxiliary logical call - | - +--> create/touch private child A-leg AUX456 - | parent_a_leg=A123 only in lineage +parent continuity BranchKey = Session/A-leg A123 (captured before submit) | + +---- opaque lineage only --------------------------------+ + | +originating principal/scope | + | | + v | +private auxiliary logical call | + | | + +--> create/touch private child A-leg AUX456 | + | parent_a_leg=A123 only in lineage | + | | +--> normal route selector / request authority / BillingCallID | +--> one or more child B-legs | +--> normal terminal usage/billing + +continuity Await/merge/revision/reinjection always use captured parent BranchKey, +never AUX456. ``` Detached semantics: - preserve authenticated principal/scope for security and billing; -- parent SessionID/A-leg/trace are correlation only; +- parent SessionID/A-leg/trace are correlation only for child execution but remain the separately captured authority for continuity-state lookup; - do not call primary secure-session BeginTurn or create primary TurnID; - do not mutate primary transcript/activity/turn counts; - do not propagate primary resume authority; @@ -339,9 +370,13 @@ Every extraction has: - independent BillingCallID; - private child A-leg and independent B-leg attempts; +- positive authoritative `AttemptSeq` on every independently persisted child/failover B-leg; - normal usage/concurrency authority; - normal credit screen/exposure admission when authoritative billing applies; -- normal terminal usage/customer settlement/provider COGS. +- normal terminal usage/customer settlement/provider COGS; +- final billing evidence with the existing usage quantities/presence and cost evidence/presence plus valid `Source`, `Authority`, and `DedupeKey` for each auxiliary B-leg/failover record. + +The existing independent-leg billing boundary rejects `AttemptSeq <= 0`; continuity must not treat a distinct BillingCallID as sufficient if the per-leg record is invalid or would be rejected. Tests therefore pin field-level evidence, not only call identity. No special charging function or money ledger is added. @@ -366,9 +401,10 @@ Create a small process-owned `internal/core/compactioncontinuity` coordinator. I Responsibilities: -- serialize updates per authoritative branch key; +- serialize updates per authoritative **parent** branch key; - use process `ExtensionState` as serialized backing where practical; - revision-checked load/update; +- non-billable preview intent -> committed transaction binding; - pending job/injection watermarks; - bounded max entries/TTL and lazy cleanup; - opaque capsule/source blobs only. @@ -378,42 +414,64 @@ Conceptual branch key: ```go type BranchKey struct { AuthoritativeSessionID string - ALegID string + ALegID string // parent primary A-leg; never detached child A-leg PrincipalPartition string // used when no secure SessionID } ``` -Client session hints never authorize lookup. +Client session hints never authorize lookup. The feature captures this key before any auxiliary child A-leg is created and derives a stable content-free `BranchBinding = SHA256(domain || canonical(BranchKey))` for serialized capsule/job identity. Raw principal/session identifiers need not be sent to the extractor model. Conceptual state: ```go +type PreviewIntent struct { + Key string // H(branch binding | detector preview boundary/fingerprint | target revision) + TargetSourceRevision uint64 +} + +type InjectionTarget struct { + BoundaryKey string + CapsuleRevision uint64 +} + +type InjectionWatermark struct { + BranchBinding string + BoundaryKey string + CapsuleRevision uint64 +} + type BranchState struct { Revision uint64 CapsuleJSON json.RawMessage CapsuleDigest [32]byte SourceHighWatermark string SanitizedSourceJSON json.RawMessage + PendingPreviewIntent *PreviewIntent PendingJobID auxiliary.JobID PendingJobTargetRevision uint64 - PendingInjectionRevision uint64 - LastInjectedRevision uint64 + PendingJobBranchBinding string + PendingInjection *InjectionTarget + LastReleasedInjection *InjectionWatermark LastCompactionTransaction string UpdatedAt time.Time } ``` +`PendingJobBranchBinding` must match the branch state that owns the JobID before Await/merge. Child A-leg IDs are never accepted as a replacement key. + Core coordinator never interprets plan/decision facts. Worker/provider work runs outside coordinator locks. ## D8. Continuity Capsule V1 -Feature-owned model: +Feature-owned serialized envelope: ```json { "schema_version": 1, "revision": 12, "source_high_watermark": "...", + "branch_binding": "sha256:...", + "content_digest": "sha256:...", "plan": { "status": "accepted", "source": "structured|user_acceptance|semantic", @@ -424,6 +482,8 @@ Feature-owned model: "decisions": [ { "id":"...", + "conflict_key":"architecture.billing.mode", + "supersedes":[], "statement":"...", "status":"active", "authority":"user_explicit|user_acceptance|semantic", @@ -437,6 +497,16 @@ Feature-owned model: } ``` +### Envelope binding and digest + +- `branch_binding` is the content-free digest derived from the authoritative parent `BranchKey`; the private child A-leg never contributes to it. +- `content_digest` is `SHA256(domain || canonical_json(envelope_without_content_digest))`. The digest scope includes schema version, revision, high-watermark, branch binding and the complete semantic payload; only the digest field itself is omitted to avoid recursion. +- Canonical JSON rules (object-key order, number/string representation, no insignificant whitespace) are fixed by feature tests. The implementation may use a typed canonical encoder rather than generic map serialization. +- Registry consume, merge, reinjection/projection, authorized recovery and generation-reload reuse first validate branch binding + digest. A mismatch is fail-open discard, never cross-branch adoption. +- Model-facing extractor input/projection may omit raw branch identifiers and may omit the branch binding itself after local validation; self-binding is a storage/transfer integrity contract, not a reason to expose account/session identity remotely. + +### Decision identity and merge precedence + Merge precedence: 1. later explicit user correction/decision; @@ -445,9 +515,22 @@ Merge precedence: 4. validated semantic inference; 5. older capsule state. -Semantic output cannot resurrect a fact against newer explicit intent. IDs remain stable across revisions; existing IDs are reused, deterministic carrier facts use deterministic normalized IDs, and new semantic IDs are accepted only after validation. +Each decision has two identities with different purposes: + +- `id`: stable fact identity/provenance across revisions; +- `conflict_key`: stable normalized semantic slot describing what choice the decision governs. + +Extractor-generated IDs alone never decide conflicts. Validation/merge enforces: + +- at most one `active` decision per `conflict_key`; +- a new active decision for an existing conflict key deterministically supersedes the older active decision according to precedence/revision; +- a correction whose semantic slot cannot safely reuse the prior conflict key must carry validated `supersedes` references to known active decision IDs; unknown/cross-branch references are rejected; +- non-conflicting decisions retain independent conflict keys and stable IDs; +- semantic output cannot resurrect a superseded fact against newer explicit intent. -Bounded pruning preserves active decisions/constraints and pending/in-progress plan steps first, then useful rationale/current rejections, then condenses completed/superseded history. Prune whole facts and revalidate; never truncate JSON syntactically. +Deterministic carrier facts use deterministic normalized IDs. Semantic IDs are accepted only after schema/source/conflict validation. + +Bounded pruning preserves active decisions/constraints and pending/in-progress plan steps first, then useful rationale/current rejections, then condenses completed/superseded history. Prune whole facts and revalidate/re-digest; never truncate JSON syntactically. ## D9. Deterministic Plan Carrier Catalog @@ -472,7 +555,7 @@ Priority: 1. user decision/constraint text; 2. assistant plan/proposal/clarification needed to interpret later user replies; 3. recognized structured plan carriers; -4. prior capsule. +4. prior locally validated capsule semantic payload. Default drop/truncate: @@ -495,7 +578,7 @@ One canonical child call contains: - configured independent route; - fixed extraction instructions; -- prior capsule; +- prior validated capsule semantic payload/base revision; - deterministic plan facts; - sanitized delta; - no tools; @@ -512,6 +595,13 @@ Expected output is one strict JSON object such as: "base_revision": 11, "facts": [], "plan_updates": [], + "decision_updates": [ + { + "id": "...", + "conflict_key": "architecture.billing.mode", + "supersedes": ["decision-old"] + } + ], "remove_or_supersede": [] } ``` @@ -524,7 +614,8 @@ Validate before merge: - exact enums; - no unknown authority escalation; - no raw tool/blob fields; -- valid/allowed source refs where used. +- valid/allowed source refs where used; +- decision conflict keys normalized/bounded and `supersedes` references limited to known decisions in the same validated parent capsule. Malformed output is discarded fail-open. There is normally no second LLM pass. @@ -548,18 +639,19 @@ Generic words alone never trigger. If deterministic state fully satisfies the co 1. detector `RequestOpened` commits start/transaction; 2. preservation commits sanitized source; -3. deterministic carrier extraction updates capsule; +3. deterministic carrier extraction updates/re-digests capsule; 4. eligibility decides whether semantic work adds value; -5. `BackgroundAux.SubmitCollect` submits one coalesced job if needed; -6. primary compaction stream proceeds concurrently. +5. derive committed coalescing identity `H(parent_branch_binding | transaction_id | target_source_revision)`; +6. `BackgroundAux.SubmitCollect` submits one coalesced job if needed and BranchState stores the JobID against the captured parent branch; +7. primary compaction stream proceeds concurrently. ### Final selected event 1. detector `PreviewResponse` identifies potential completion/transaction without committing; -2. preservation optionally `Await`s the matching job for the bounded barrier; -3. ready strict output is validated and revision-merged; +2. preservation resolves the matching parent-branch JobID and optionally `Await`s it for the bounded barrier; +3. ready strict output is validated and revision-merged/re-digested into the same parent branch; 4. verified plaintext carrier may receive deterministic continuity projection; -5. unsafe opaque/late paths set `PendingInjectionRevision`; +5. unsafe opaque/late paths set `PendingInjection{BoundaryKey, CapsuleRevision}`; 6. detector `ResponseReleased` receives the **post-preservation final event** and commits completion; 7. metadata-only observers dispatch; 8. final event releases to client. @@ -568,30 +660,59 @@ A barrier timeout is fail-open. A late result remains available while referenced ## D14. Completion-Only / Local-Compaction Flow -If the first evidence is the next rewritten request: +If the first evidence is the next rewritten request, v1 deliberately avoids fresh billable semantic work before the primary request is known to have opened. + +### Before primary Open -1. `PreviewRequest` identifies a completion candidate from shared history matcher/fingerprint state; -2. preservation loads prior sanitized source/capsule; -3. deterministic carrier delta merges; -4. if semantic work is required and no coalesced job exists, submit BackgroundAux; -5. bounded `Await` barrier runs before primary B-leg Open; -6. ready capsule is injected; timeout/failure continues fail-open; -7. normal primary Open proceeds; -8. only after successful Open does `RequestOpened` commit completion-only detector state. +1. `PreviewRequest` identifies a completion candidate from the shared history matcher/fingerprint state without committing it. +2. capture the authoritative parent BranchKey/branch binding and derive `PreviewIntentKey = H(parent_branch_binding | preview_boundary_fingerprint | target_source_revision)`. +3. store/coalesce only that **non-billable preview intent** in BranchCoordinator; do not call `BackgroundAux.SubmitCollect`, do not retain `KindAsync`, and do not create a child A-leg yet. +4. load/validate the prior capsule and merge deterministic carrier delta. +5. if a matching semantic job was already submitted by an earlier successfully opened request (for example a late strict job), optionally `Await` that existing JobID up to the bounded barrier; otherwise there is no pre-open semantic wait. +6. inject the best already-ready/deterministic capsule transactionally if required, then continue normal primary Open fail-open. -The extractor is still off-session/background; only the narrow synchronization point is on the main request path because no earlier observable parallel window existed. +### After successful primary Open + +7. detector `RequestOpened` commits the completion-only transaction. +8. bind the PreviewIntentKey to the committed transaction and commit source/high-watermark/deterministic state for this successfully opened request. +9. if semantic eligibility still requires additional information, derive the normal committed coalescing key and submit one background job now. +10. that newly submitted job may improve subsequent turns/continuity state, but it is **not allowed to retroactively justify pre-open billing** for this request. + +If primary Open fails, discard/expire the preview intent under bounded cleanup and produce **zero new billable continuity child work**. The extractor remains off-session/background after successful submission; v1 accepts that completion-only semantic information discovered only at this boundary may not be available for the same request, because preserving billing/lifecycle correctness is stronger than paying before Open. ## D15. Canonical Reinjection -One provider-neutral helper applies a deterministic text projection of the capsule. +One provider-neutral helper applies a deterministic text projection of a locally branch/digest-validated capsule. - message-authoritative call -> legal proxy-owned developer/system instruction/message representation; - item-authoritative call -> legal canonical message item; - never populate legacy Messages/Instructions alongside item authority in violation of `Call.Validate`. -Projection format is versioned/delimited and says it is prior continuation state, not a new user request. +Projection format is versioned/delimited and says it is prior continuation state, not a new user request. Internal branch binding/digest metadata need not be emitted into the model-facing text after local validation. + +### Boundary-scoped deduplication + +Revision alone is not an injection identity. Use: + +```go +type InjectionWatermark struct { + BranchBinding string + BoundaryKey string // committed transaction when available; detector preview boundary key while awaiting bind + CapsuleRevision uint64 +} +``` + +Rules: + +1. `PendingInjection` records the boundary + revision requiring carry-forward. +2. a successive distinct compaction boundary may inject the same capsule revision again; this is required when no new facts were learned between two opaque compactions. +3. within one primary call, a call-local ephemeral marker prevents the helper from inserting the same continuity block twice during internal retry/failover preparation; this marker is not the durable branch watermark. +4. canonical insertion must pass `Call.Validate`. Callback error/panic/validation failure restores the exact pre-injection call and leaves `PendingInjection` unchanged. +5. primary Open failure leaves `PendingInjection` unchanged. +6. `LastReleasedInjection` advances and matching `PendingInjection` clears **only after the primary turn reaches successful final client release**. If the turn aborts before release, a later eligible request may inject again. +7. after completion-only Open binds a preview boundary to a committed transaction, the branch state rewrites the pending boundary key consistently before final release bookkeeping. -`LastInjectedRevision` prevents duplicate injection for one boundary. Projection is rechecked against injection budget after serialization. +Projection is rechecked against injection budget after serialization. ## D16. Plaintext Result Augmentation Boundary @@ -622,13 +743,13 @@ Allowed trusted session controls may include enabled, preserved categories, appr ## D18. Configuration -Indicative only; implementation tests pin defaults: +Indicative only; implementation tests pin defaults. The example intentionally remains disabled so copying it cannot silently enable remote data egress or extra billed inference: ```yaml plugins: features: - id: compaction-continuity - enabled: true + enabled: false config: preserve: plan: true @@ -676,7 +797,12 @@ Default request-time behavior is fail-open. | invalid schema | discard result; actual submitted usage still accountable | | stale result | discard/forget; no state regression | | barrier timeout | continue native flow; pending result may be consumed later while bounded/useful | -| capsule over budget | whole-fact deterministic pruning | +| capsule branch-binding/digest mismatch | discard capsule/result for this branch; native flow continues | +| capsule over budget | whole-fact deterministic pruning then re-digest | +| preserver callback error/panic | record content-free stage failure; do not propagate to primary traffic | +| failed/invalid BeforeRequest mutation | restore pre-preservation `Call`; keep pending reinjection; native flow continues | +| failed/invalid BeforeResponseRelease mutation | restore pre-preservation `Event`; committed detector sees restored final event | +| completion-only primary Open fails | discard/expire non-billable preview intent; no new child submission/billing | | state/coordinator unavailable | skip preservation mutation; native traffic continues | | process shutdown | stop admission; cancel/join worker; normal submitted usage/accounting remains terminally owned | @@ -689,6 +815,7 @@ Default request-time behavior is fail-open. - source content is untrusted quoted data; - no tools and preservation plugin suppressed; - prompt/output/capsule absent from normal logs/metrics; +- branch binding is a content-free one-way binding and raw parent BranchKey identifiers need not be sent to the model; - pending raw result exists only in bounded process memory; - transcript reads preserve principal/session/workspace authorization; - no implicit durable transcript capture; @@ -701,14 +828,17 @@ Content-free metrics/diagnostics: - compaction preview/start/completion by evidence/rule; - structured carrier hits by rule ID; - semantic eligibility skip reasons; +- preview intents created/bound/expired without provider submission; - background jobs submitted/coalesced/saturated/canceled/completed/failed/stale; - queue depth/in-flight; - extractor route/backend/model identifiers where policy allows; - extractor tokens/cache/cost through existing usage/billing; -- capsule revision/serialized size/fact counts; +- auxiliary accounting rejection by invalid attempt/evidence contract; +- capsule revision/serialized size/fact counts/digest-validation outcome; - barrier wait duration/timeouts; -- augmentation/reinjection counts; -- stale/revision conflicts; +- augmentation/reinjection counts and pending/committed watermark outcomes; +- preserver callback fail-open outcomes by stage; +- stale/revision/conflict-key conflicts; - opaque-carrier no-mutation count. Logs use hashes/IDs/counts only. @@ -717,20 +847,22 @@ Logs use hashes/IDs/counts only. 1. ProcessServices owns BackgroundScheduler and BranchCoordinator. 2. Generation snapshots hold non-owning clients only. -3. Each successfully submitted background job owns exactly one captured `KindAsync` pin. +3. Each successfully submitted background job owns exactly one captured `KindAsync` pin; non-billable preview intents own none. 4. Pin releases exactly once on every terminal/cancel/handoff-failure path. 5. A worker never obtains a different generation after submit. -6. BranchCoordinator serializes revision/job/injection state across overlapping generations. -7. No external model/provider call executes while a branch-coordinator lock is held. -8. No observer/plugin callback runs under scheduler/coordinator internal locks. -9. Job/result/branch maps have hard bounds/TTL and no unbounded cleanup goroutine pattern. -10. Reload changes future jobs only; already-submitted work remains accountable and terminally owned. +6. Parent BranchKey/branch binding is captured before child A-leg creation and remains the only continuity-state key for that job. +7. BranchCoordinator serializes revision/job/injection state across overlapping generations. +8. No external model/provider call executes while a branch-coordinator lock is held. +9. No observer/plugin callback runs under scheduler/coordinator internal locks. +10. Job/result/branch/preview-intent maps have hard bounds/TTL and no unbounded cleanup goroutine pattern. +11. Reload changes future jobs only; already-submitted work remains accountable and terminally owned. +12. Branch-level reinjection watermark commits only after successful final client release; validation/Open/aborted-release failures retain pending state. ## D23. Restart and Durable Resume V1 guarantees continuity across compaction and immutable generation reload within one process. It does not create durable capsule/job storage. -An optional future/implementation recovery adapter may reconstruct from an already-enabled authorized secure-session transcript. It must be read-only, bounded and authorization-preserving. Without that source, missing process capsule after restart is explicit fail-open state. +An optional future/implementation recovery adapter may reconstruct from an already-enabled authorized secure-session transcript. It must be read-only, bounded and authorization-preserving. Reconstructed capsules receive the current authoritative branch binding and a freshly validated canonical digest before use. Without that source, missing process capsule after restart is explicit fail-open state. No durable job queue or full transcript store is added here. @@ -741,11 +873,11 @@ Primary packages: - `pkg/lipsdk/compaction` — preview-related preservation contract beside prerequisite observer; - `pkg/lipsdk/feature` — additive preserver slice; - `pkg/lipsdk/auxiliary` — additive BackgroundClient/job types and trusted detached policy carrier; -- `internal/featurebundle` / `internal/core/extensions` — merge/snapshot/preservation stages; +- `internal/featurebundle` / `internal/core/extensions` — merge/snapshot/preservation stages and fail-open transactional mutation dispatch; - `internal/core/compactiondetect` — pure request/response previews sharing matcher authority; - `internal/core/auxreq` — bounded scheduler, submit-time pin handoff, detached child adapter; -- `internal/core/compactioncontinuity` — narrow BranchCoordinator; -- `internal/core/runtime` — request preview/open, final-response ordering, detached prepare/private child A-leg wiring; +- `internal/core/compactioncontinuity` — narrow BranchCoordinator/preview-intent/injection-watermark state; +- `internal/core/runtime` — request preview/open, final-response ordering, detached prepare/private child A-leg wiring, successful-release watermark commit; - `internal/infra/runtimebundle` — process ownership and generation snapshot clients; - `internal/plugins/features/compactioncontinuity` — config/capsule/carriers/sanitizer/eligibility/extractor/merge/injection; - existing metering/billing/report projections only for content-free auxiliary workload classification; @@ -757,16 +889,18 @@ Individual provider/backend and frontend protocol packages should not need compa TDD order: -1. RED capsule merge/supersession/pruning + carrier fixtures. +1. RED capsule branch-binding/digest, decision conflict-key/supersession, merge/pruning + carrier fixtures. 2. RED sanitizer/eligibility/extractor-schema tests. -3. RED BackgroundAux submit-time pin/coalescing/saturation/result-lifetime/shutdown tests. -4. RED detached-session/private-child-A-leg tests. -5. RED separate-route + user-billing/BillingCallID/protocol-usage tests. +3. RED BackgroundAux submit-time pin/committed coalescing/saturation/result-lifetime/shutdown tests plus non-billable preview-intent binding tests. +4. RED detached-session/private-child-A-leg tests proving parent BranchKey remains continuity authority. +5. RED separate-route + user-billing/BillingCallID/per-B-leg AttemptSeq/evidence/protocol-usage tests. 6. RED detector request/response preview and final-release ordering tests. -7. RED strict/remote and completion-only compaction barriers, plaintext/opaque paths. -8. RED three-plus repeated-compaction/concurrency/generation-reload tests. -9. GREEN minimal implementation in that order. -10. repository quality/race/architecture/simplification gates. +7. RED preserver callback error/panic rollback and canonical validation fail-open tests. +8. RED strict/remote and completion-only compaction barriers including failed-Open zero-child-billing. +9. RED boundary-scoped reinjection tests: same revision across two compactions, validation/Open/release failure then retry, plaintext/opaque paths. +10. RED three-plus repeated-compaction/concurrency/generation-reload tests. +11. GREEN minimal implementation in that order. +12. repository quality/race/architecture/simplification gates. No external model credentials are required; deterministic fake/local backends return fixture JSON. @@ -775,15 +909,20 @@ No external model credentials are required; deterministic fake/local backends re 1. One compaction recognition authority; no duplicate signature matrix. 2. Metadata observer remains content-free/non-mutating. 3. Committed detector sees the actual final released event after permitted preservation mutation. -4. Extractor is off primary session but billed to originating user by default. -5. Extractor route is explicit/independent unless explicit inherit. -6. Background ownership is captured before request spawn right ends. -7. ProcessServices owns worker/coordinator lifetime. -8. Detached child uses a private auxiliary A-leg and never creates a primary secure-session turn. -9. Normal routing/B2BUA/usage/billing own child execution; no direct provider path. -10. Primary protocol usage excludes child usage; account totals include it. -11. Continuity state is bounded, revisioned, branch-scoped and reload-safe. -12. Opaque/encrypted compaction bytes are immutable. -13. First post-compaction turn has a bounded barrier when required. -14. Pending late results remain useful only within a bounded branch retention window. -15. No second transcript DB, money ledger, generic workflow engine or redundant summary LLM pass. +4. Preserver callback failure is feature-local/fail-open; failed mutation restores the prior canonical object. +5. Extractor is off primary session but billed to originating user by default. +6. No new billable semantic child is submitted before successful primary Open, including completion-only/local discovery. +7. Extractor route is explicit/independent unless explicit inherit. +8. Background ownership is captured before request spawn right ends. +9. ProcessServices owns worker/coordinator lifetime. +10. Detached child uses a private auxiliary A-leg and never creates a primary secure-session turn; continuity state remains keyed to the captured parent branch. +11. Normal routing/B2BUA/usage/billing own child execution; every persisted auxiliary B-leg has valid AttemptSeq/final evidence and there is no direct provider path. +12. Primary protocol usage excludes child usage; account totals include it. +13. Continuity capsule is bounded, revisioned, parent-branch-bound, digest-validated and reload-safe. +14. Contradictory active decisions cannot coexist solely because semantic extraction emitted new IDs; conflict keys/supersedes define deterministic replacement. +15. Opaque/encrypted compaction bytes are immutable. +16. First post-compaction request may await only already-submitted semantic work before Open; fresh semantic work waits for successful Open. +17. Reinjection dedupe is branch/boundary/revision-scoped and commits only after final client release. +18. Pending late results remain useful only within a bounded branch retention window. +19. Configuration examples/default behavior remain disabled until explicit operator opt-in. +20. No second transcript DB, money ledger, generic workflow engine or redundant summary LLM pass. From bbc81e5852f04b006e84923285ed8162583b5dcd Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Tue, 18 Aug 2026 20:34:51 +0200 Subject: [PATCH 17/18] spec: align continuity implementation tasks with review fixes --- .../tasks.md | 243 +++++++++--------- 1 file changed, 122 insertions(+), 121 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/tasks.md b/.kiro/specs/compaction-continuity-preservation/tasks.md index 4adc4740..fe15c012 100644 --- a/.kiro/specs/compaction-continuity-preservation/tasks.md +++ b/.kiro/specs/compaction-continuity-preservation/tasks.md @@ -6,7 +6,8 @@ - `compaction-event-detection` runtime is a hard prerequisite. Do not duplicate its signature matrix if it is not yet implemented; land that spec first. - Keep every task independently reviewable with **no more than five concrete actions**. - Preserve existing routing, B2BUA, secure-session, generation pinning, output commitment, usage authority, BillingCallID accounting, and provider-opaque compaction semantics. -- Auxiliary semantic extraction is a separate off-session background model call, independently routed and billed to the originating user/account by default. +- Auxiliary semantic extraction is a separate off-session background model call, independently routed and billed to the originating user/account by default; no new billable continuity child may be submitted before the triggering primary request successfully opens. +- Capture the parent continuity BranchKey before detached child A-leg creation; child A-leg identity is execution/billing lineage only and never continuity-state authority. - Do not add provider-specific continuity branches, a direct provider client, second transcript store, feature-owned money ledger, generic task/workflow engine, or second LLM summary-rewrite pass. - No real model credentials are required for correctness tests; use deterministic local/fake backends and fixture JSON. @@ -16,11 +17,11 @@ - Verify the #312 runtime detector exists; if absent, stop this spec's implementation and complete `compaction-event-detection` first rather than adding fallback signatures. - Add RED tests for pure `PreviewRequest` and `PreviewResponse` sharing the same rule/fingerprint authority as committed detection without mutating detector state or emitting lifecycle events. -- Prove a strict preview on a request that never opens cannot create a started transaction or semantic extraction submission. +- Prove a strict or completion-only preview on a request that never opens cannot create a billable semantic extraction submission; completion-only may create only a bounded non-billable preview intent. - Prove committed `ResponseReleased` receives the exact event after permitted preservation finalization, not a pre-final copy. -- Add near-miss and completion-only preview fixtures from the existing detector rule matrix. +- Add near-miss and completion-only preview fixtures, including stable preview boundary/fingerprint identity when no committed transaction exists. -_Requirements: 1.1–1.13, 7.1, 11.3, 11.10_ +_Requirements: 1.1–1.13, 4.10, 7.1, 7.12, 11.3, 11.10_ _Validation: focused `internal/core/compactiondetect` RED tests; no duplicated rule catalog._ @@ -30,16 +31,16 @@ _Validation: focused `internal/core/compactiondetect` RED tests; no duplicated r - Add RED FeatureBundle/merge/runtime-snapshot tests for ordered additive preservers, defensive copies, nil validation and frozen generation semantics. - Prove ordinary compaction observer event payloads still contain no canonical request/response content and cannot mutate traffic. - Freeze preservation callback ordering: BeforeRequest -> successful-open callback -> response-preview finalization -> committed detector observation -> metadata observer dispatch. -- Add architecture assertions preventing preservation mutation from being implemented as an ordinary response hook after the finalization seam. +- Add RED callback error/panic tests proving BeforeRequest/BeforeResponseRelease mutations roll back to the exact prior canonical object and all preserver failures remain feature-local fail-open rather than primary provider/retry errors. -_Requirements: 1.2–1.5, 1.9–1.11, 11.12–11.13_ +_Requirements: 1.2–1.5, 1.9–1.11, 7.16, 10.4–10.5, 10.13, 11.12–11.13_ _Validation: `go test ./pkg/lipsdk/... ./internal/featurebundle/... ./internal/core/extensions/...` is RED for the new contract._ ### Task 1.3 — Freeze background auxiliary scheduling and generation ownership -- Add deterministic RED scheduler tests for bounded queue/concurrency, coalescing, result retention, Await/Forget, and saturation without goroutine fallback. -- Prove `SubmitCollect` retains `genpin.KindAsync` synchronously while the parent spawn right is live and a worker never retains a replacement generation later. +- Add deterministic RED scheduler tests for bounded queue/concurrency, committed-transaction coalescing, result retention, Await/Forget, and saturation without goroutine fallback. +- Prove `SubmitCollect` rejects/never uses an empty transaction-derived continuity coalescing key, retains `genpin.KindAsync` synchronously while the parent spawn right is live, and a worker never retains a replacement generation later. - Add enqueue-failure/cancel/timeout/shutdown paths proving every retained pin is released exactly once. - Add a delayed-start test where the parent request context is canceled before the worker runs but the worker uses its own bounded context safely. - Add `goleak` and race tests for queue saturation, shutdown, concurrent Await/Forget and late job completion. @@ -51,24 +52,24 @@ _Validation: focused `internal/core/auxreq` tests are RED and deterministic._ ### Task 1.4 — Freeze detached child A-leg, routing, and session isolation - Add RED tests proving a detached child creates a private auxiliary A-leg and never reuses the parent's A-leg as execution authority. -- Prove parent SessionID/A-leg/trace survive only as lineage while primary secure-session BeginTurn, transcript, activity and turn count remain unchanged. +- Prove parent SessionID/A-leg/trace survive only as lineage for child execution while the separately captured parent BranchKey remains continuity-state authority and primary secure-session BeginTurn/transcript/activity/turn count remain unchanged. - Prove the child uses its explicit extractor selector and a primary A-leg runtime route override cannot rewrite it. - Prove detached mode is trusted auxiliary metadata unavailable from frontend/wire canonical fields. - Cover private child B-leg failover/terminal lineage without client-visible session headers/history. -_Requirements: 5.1–5.15, 8.1–8.3, 9.11, 11.4–11.5_ +_Requirements: 4.16, 5.1–5.15, 8.1–8.3, 9.11, 11.4–11.5, 11.16_ _Validation: runtime/auxreq/secure-session/B2BUA focused tests are RED._ ### Task 1.5 — Freeze capsule semantics, billing attribution, and repeated-compaction outcomes -- Add RED table tests for capsule revision, precedence, decision supersession, rejection, plan-step progress, stale merge rejection and deterministic pruning. +- Add RED table tests for capsule branch binding/digest, revision, decision conflict-key/supersession precedence, rejection, plan-step progress, stale merge rejection and deterministic pruning. - Add RED structured carrier fixtures for Codex update-plan, OpenCode todo and supported Cline task-progress shapes plus near misses. -- Add RED billing tests requiring the child to inherit the originating account while receiving a separate BillingCallID/B-legs and auxiliary workload classification. -- Prove primary protocol-visible usage excludes extractor tokens while account/operator totals include the independent child usage. -- Add an end-to-end RED scenario with at least three compactions, user decision correction, completed/pending plan steps, dedupe and bounded capsule growth. +- Add RED billing tests requiring originating-account attribution, separate BillingCallID/B-legs, positive AttemptSeq, valid usage/cost/Source/Authority/DedupeKey evidence, retry/failover accounting and auxiliary workload classification. +- Prove primary protocol-visible usage excludes extractor tokens while account/operator totals include the independent child usage and an `AttemptSeq <= 0` independent leg is rejected rather than silently counted. +- Add an end-to-end RED scenario with at least three compactions, user decision correction, completed/pending plan steps, same-revision distinct-boundary reinjection, dedupe and bounded capsule growth. -_Requirements: 2.1–2.15, 3.1–3.7, 6.1–6.13, 8.4–8.12, 11.6–11.8_ +_Requirements: 2.1–2.17, 3.1–3.7, 6.1–6.14, 7.11, 8.4–8.12, 11.6–11.8, 11.17_ _Validation: feature/billing/integration target tests are RED before production feature code._ @@ -77,12 +78,12 @@ _Validation: feature/billing/integration target tests are RED before production ### Task 2.1 — Implement pure detector request/response previews - Refactor prerequisite detector match/fingerprint code so preview and committed paths call one shared internal recognition authority. -- Implement `PreviewRequest` without state mutation and preserve existing successful-Open commit behavior in `RequestOpened`. +- Implement `PreviewRequest` without state mutation, exposing stable completion-only boundary/fingerprint identity when no transaction exists, and preserve existing successful-Open commit behavior in `RequestOpened`. - Implement `PreviewResponse` without completion mutation and keep `ResponseReleased` as the committed final-event boundary. - Make Task 1.1 positives/near misses/state-snapshot assertions green without changing observer payloads. - Keep unsupported protocol controls and provider DTOs outside the detector. -_Requirements: 1.1, 1.5–1.13_ +_Requirements: 1.1, 1.5–1.13, 4.10_ _Design: D1; D13–D14_ @@ -92,11 +93,11 @@ _Validation: focused detector suites green; existing #312 tests remain unchanged - Add the separate preservation types/services/interfaces beside the existing compaction observer contract. - Add `CompactionPreservers` to FeatureBundle, one merge surface and frozen request-snapshot accessor with existing validation conventions. -- Add core extension dispatch helpers that isolate preserver panic/error under configured preservation fail-open semantics. -- Wire the semantic stage order without introducing a mutating generic StageID or moving ordinary response hooks after finalization. -- Make Task 1.2 SDK/merge/order tests green. +- Add core extension dispatch helpers that isolate preserver panic/error and record content-free failure outcomes without propagating them into primary traffic. +- Make request/event preservation mutation transactional with a bounded clone/undo helper so callback failure, panic, or canonical validation failure restores the pre-preservation object; keep ordinary response hooks before finalization. +- Make Task 1.2 SDK/merge/order/rollback tests green without introducing a generic transaction or mutating StageID framework. -_Requirements: 1.2–1.4, 1.10–1.11, 10.3–10.5_ +_Requirements: 1.2–1.4, 1.10–1.11, 7.16, 10.3–10.5, 10.13_ _Design: D2_ @@ -105,7 +106,7 @@ _Validation: SDK/featurebundle/extensions tests green._ ### Task 2.3 — Implement the process-owned BackgroundAux collector - Add additive BackgroundClient/JobID/options APIs while keeping synchronous `auxiliary.Client` source-compatible. -- Implement the bounded `internal/core/auxreq` scheduler with keyed coalescing, worker pool, result registry and process-root context. +- Implement the bounded `internal/core/auxreq` scheduler with committed keyed coalescing, worker pool, result registry and process-root context; do not admit pre-open preview intents as jobs. - Implement submit-time runner capture/KindAsync retention and exact cleanup on enqueue failure, terminal completion, cancellation and shutdown. - Implement useful bounded pending-result retention plus Await/Forget without arbitrary callbacks or durable task semantics. - Register scheduler ownership/Close through ProcessServices and make Task 1.3 green under race/goleak. @@ -120,11 +121,11 @@ _Validation: focused auxreq/runtimebundle tests green; scheduler has no network - Add trusted internal detached-session policy on auxiliary execution without adding a frontend/wire `lipapi.Call` control. - Factor Executor preparation so detached calls preserve principal/scope but skip primary secure-session BeginTurn/activity/transcript/resume effects. -- Create/touch a private child A-leg through existing B2BUA lifecycle and keep parent A-leg/session values as lineage only. +- Create/touch a private child A-leg through existing B2BUA lifecycle, keep parent A-leg/session values as lineage only for child execution, and preserve the separately captured parent branch binding untouched. - Ensure detached route planning does not read the parent's route override and ordinary child request authority/B-legs still execute. - Make Task 1.4 session/B2BUA/routing tests green without copying the Executor. -_Requirements: 5.5–5.15, 9.11, 11.4–11.5, 11.13_ +_Requirements: 4.16, 5.5–5.15, 9.11, 11.4–11.5, 11.13, 11.16_ _Design: D4–D5_ @@ -132,15 +133,15 @@ _Validation: focused runtime/auxreq/B2BUA/secure-session tests green._ ### Task 2.5 — Implement the process branch coordinator -- Add a narrow ProcessServices-owned coordinator keyed by authoritative SessionID + A-leg/branch or principal-isolated A-leg fallback. -- Serialize branch revision/high-watermark/pending-job/injection updates and use process ExtensionState as opaque serialized backing where practical. +- Add a narrow ProcessServices-owned coordinator keyed by the authoritative **parent** SessionID + primary A-leg/branch or principal-isolated primary A-leg fallback; capture this key before detached child A-leg creation. +- Serialize branch revision/high-watermark/pending-job, non-billable preview-intent -> committed-transaction binding, and compound injection-watermark updates using process ExtensionState as opaque backing where practical. +- Require pending JobID/branch-binding consistency for Await/merge/revision/reinjection and never derive continuity keys from the private child A-leg. - Enforce max entries/TTL/lazy cleanup and never call model/provider/plugin code while coordinator locks are held. -- Expose only the preservation state operations needed by the compaction services; keep capsule/source semantics opaque to core. -- Add reload/concurrency tests proving old/new generations and workers cannot overwrite branch state out of order. +- Add reload/concurrency tests with distinct parent/child A-legs proving old/new generations/workers cannot overwrite or move capsule state to the auxiliary branch. -_Requirements: 8.1–8.15, 10.12, 11.11–11.12_ +_Requirements: 4.10, 4.16, 7.11–7.12, 8.1–8.15, 10.12, 11.11–11.12, 11.16–11.17_ -_Design: D7; D22–D23_ +_Design: D7; D14–D15; D22–D23_ _Validation: coordinator unit/race tests green and no generic transactional state framework appears._ @@ -149,7 +150,7 @@ _Validation: coordinator unit/race tests green and no generic transactional stat ### Task 3.1 — Implement feature configuration and prerequisite validation - Register the official `compaction-continuity` feature and decode bounded preserve/extractor/worker/barrier/capsule/source/result/failure settings. -- Require explicit extractor route or explicit inherit when semantic extraction is enabled; feature remains disabled by default. +- Require explicit extractor route or explicit inherit when semantic extraction is enabled; feature remains disabled by default and shipped examples stay `enabled: false`. - Validate finite positive maxima and consistency of pending-result/source/branch retention. - Fail enabled generation/startup composition clearly when detector preview/commit, BranchCoordinator or BackgroundAux services are absent. - Add config reload tests proving in-flight jobs retain immutable submission-time config while new jobs use new generation config. @@ -162,15 +163,15 @@ _Validation: config/runtimebundle feature tests green._ ### Task 3.2 — Implement Continuity Capsule v1 and deterministic carrier rules -- Implement versioned capsule/fact/plan types, stable IDs, source authority/status enums and strict validation. -- Implement deterministic precedence, stale-revision rejection, dedupe and whole-fact bounded pruning. +- Implement the versioned capsule envelope with parent `branch_binding`, canonical `content_digest`, revision/high-watermark, fact/plan types, stable IDs, decision `conflict_key`/`supersedes`, source authority/status enums and strict validation. +- Implement canonical digest encoding/verification plus deterministic conflict precedence, stale-revision rejection, at-most-one-active-decision-per-conflict-key, dedupe and whole-fact bounded pruning/re-digest. - Add versioned canonical carrier rules for the researched Codex/OpenCode/Cline plan shapes without agent-brand inference. -- Normalize supported carrier updates into capsule plan state without an LLM call. -- Make Task 1.5 capsule/carrier tests green including malformed/near-miss fixtures. +- Normalize supported carrier updates into capsule plan state without an LLM call while preserving parent branch binding. +- Make Task 1.5 capsule/carrier tests green including branch mismatch/digest mismatch/unknown supersedes/malformed/near-miss fixtures. -_Requirements: 2.1–2.15, 3.1–3.5, 8.4, 8.11–8.12_ +_Requirements: 2.1–2.17, 3.1–3.5, 8.1, 8.4, 8.11–8.12_ -_Design: D8–D9_ +_Design: D7–D9_ _Validation: feature pure-unit tests green._ @@ -190,27 +191,27 @@ _Validation: sanitizer/eligibility privacy fixtures green with no external calls ### Task 3.4 — Implement the semantic extractor child call and strict result parser -- Build one detached no-tools auxiliary child using the configured independent selector, prior capsule, deterministic plan facts and sanitized delta. -- Suppress the continuity plugin and stamp private role/origin/parent lineage without primary session authority. -- Define the fixed extraction prompt/schema requiring explicit-user precedence, accepted/current plan semantics, supersession and ambiguity omission. -- Parse exactly one bounded JSON result with schema/base-revision/enums/depth/count/string/source-ref validation; discard malformed/authority-escalating output. +- Build one detached no-tools auxiliary child using the configured independent selector, prior locally validated capsule semantic payload, deterministic plan facts and sanitized delta. +- Suppress the continuity plugin and stamp private role/origin/parent lineage without primary session authority or raw BranchKey exposure. +- Define the fixed extraction prompt/schema requiring explicit-user precedence, accepted/current plan semantics, normalized decision conflict keys, validated supersedes references and ambiguity omission. +- Parse exactly one bounded JSON result with schema/base-revision/enums/depth/count/string/source-ref/conflict-key/supersedes validation; discard malformed/authority-escalating/cross-branch output. - Add fake-backend tests proving no second LLM summary-rewrite call is generated. -_Requirements: 3.8–3.13, 5.1–5.9, 9.4–9.6, 10.6, 11.15_ +_Requirements: 2.6–2.7, 2.17, 3.8–3.13, 5.1–5.9, 9.4–9.6, 10.6, 11.15_ -_Design: D5; D11–D12_ +_Design: D5; D8; D11–D12_ _Validation: deterministic child-call/parser tests green._ -### Task 3.5 — Integrate validated deltas with branch state and late results +### Task 3.5 — Integrate validated deltas with parent branch state and late results -- On Await success, validate base revision then merge the extractor delta through the BranchCoordinator into a new bounded capsule revision. -- Reject/forget stale results without changing active decisions or reinjection watermarks. +- On Await success, verify JobID parent branch binding, capsule branch binding/digest and base revision, then merge the extractor delta through BranchCoordinator into a new bounded/re-digested capsule revision on that parent branch. +- Reject/forget stale, digest-invalid, wrong-branch or conflict-invalid results without changing active decisions or reinjection watermarks. - Keep PendingJobID/result useful across a timed-out response barrier until bounded next-turn consumption or coherent expiry. - Immediately Forget raw collected output after validation/merge and store only normalized capsule/source/job metadata. -- Add concurrent explicit-user-correction versus late-worker tests proving newer intent always wins. +- Add concurrent explicit-user-correction versus late-worker tests with different parent/child A-legs proving newer intent wins and state never migrates to the child branch. -_Requirements: 2.12–2.15, 4.13–4.15, 8.4–8.5, 10.12_ +_Requirements: 2.12–2.17, 4.13–4.16, 8.1, 8.4–8.5, 10.12, 11.16_ _Design: D3; D7–D8_ @@ -220,71 +221,71 @@ _Validation: branch/feature concurrency tests green under `-race`._ ### Task 4.1 — Schedule extraction after a real strict compaction Open -- At successful primary request Open, obtain committed detector events and invoke preservation with the effective canonical baseline/correlation. +- At successful primary request Open, obtain committed detector events and invoke preservation with the effective canonical baseline/correlation and captured parent BranchKey. - Commit the sanitized source and deterministic plan updates before deciding whether semantic extraction is necessary. -- Submit at most one coalesced semantic job for the detector transaction/source revision; never submit from an unopened strict preview. +- Derive `H(parent_branch_binding | committed transaction | target source revision)` and submit at most one coalesced semantic job; never submit from an unopened strict preview or with an empty transaction ID. - Let the primary compaction stream proceed immediately after submission without waiting for extractor completion. -- Cover retry/failover start dedupe and failed-Open zero-billing behavior. +- Cover retry/failover start dedupe, parent-vs-child branch identity, and failed-Open zero-billing behavior. -_Requirements: 1.5–1.8, 3.13, 4.9–4.10, 7.1–7.2, 11.3_ +_Requirements: 1.5–1.8, 3.13, 4.9–4.10, 4.16, 7.1–7.2, 11.3, 11.16_ -_Design: D13_ +_Design: D3; D13_ _Validation: runtime integration tests green with deterministic delayed extractor backend._ ### Task 4.2 — Implement response-preview finalization and bounded completion barrier - Route each final selected event through pure detector `PreviewResponse` before committed `ResponseReleased`. -- Let preservation resolve/await the matching job only for the configured bounded barrier and merge a valid ready result. +- Let preservation resolve/await the matching parent-branch job only for the configured bounded barrier and merge a valid ready result. - Keep ordinary hooks/gates/finalizers before this stage and committed detector observation/metadata observers/client delivery after it. -- On timeout/error, mark the appropriate pending state and continue fail-open without blocking indefinitely. -- Prove committed detector and client both see the exact post-preservation final event. +- On timeout/error/panic/invalid preservation mutation, restore the pre-preservation event where needed, mark appropriate pending state and continue fail-open without blocking indefinitely. +- Prove committed detector and client both see the exact post-preservation or correctly restored final event. -_Requirements: 1.10–1.11, 7.2–7.5, 10.4–10.5, 11.10_ +_Requirements: 1.10–1.11, 7.2–7.5, 7.16, 10.4–10.5, 11.10_ -_Design: D1–D2; D13_ +_Design: D1–D2; D13; D19_ -_Validation: final-stream release/ordering tests green across live/gated/recovery paths._ +_Validation: final-stream release/ordering/fail-open tests green across live/gated/recovery paths._ -### Task 4.3 — Protect completion-only/local first post-compaction turns +### Task 4.3 — Protect completion-only/local first post-compaction turns without pre-open billing -- Before B-leg Open, use pure request preview to recognize installed/completion-only local compaction without committing detector state. -- Load prior sanitized source/capsule and apply deterministic plan updates; submit one background job only if semantic eligibility requires it. -- Await only up to the bounded barrier and inject a valid ready capsule before the first post-compaction B-leg opens. -- Continue fail-open on timeout/failure and commit detector completion only after the primary request successfully opens. -- Cover reset/new-A-leg/near-miss cases so an unrelated short rewrite does not trigger extraction/injection. +- Before B-leg Open, use pure request preview to recognize installed/completion-only local compaction, capture the parent branch, and derive/store one non-billable preview-intent key from branch + detector preview boundary/fingerprint + target source revision without committing detector state or submitting BackgroundAux. +- Load/validate prior capsule/source, apply deterministic plan updates, and await only a matching job that was already submitted by an earlier successfully opened request; inject the best already-ready/deterministic capsule before Open when required. +- After successful primary Open, commit detector completion, bind the preview intent to the committed transaction/source revision, then submit one coalesced semantic job if eligibility still requires it; the new job improves later continuity rather than retroactively justifying pre-open provider work. +- Continue fail-open on preview/await/injection failure; if primary Open fails, retain required pending injection but discard/expire the non-billable intent and prove zero new child BillingCallID/B-leg/provider usage was created. +- Cover retries, distinct parent/child A-legs, reset/new-A-leg/near-miss cases so unrelated rewrites do not trigger extraction/injection or cross-branch state. -_Requirements: 1.8–1.9, 7.3–7.4, 7.8, 7.12–7.14, 8.5–8.7_ +_Requirements: 1.8–1.9, 4.10, 4.16, 7.1, 7.3–7.4, 7.8, 7.12–7.14, 8.1, 8.5–8.7, 11.3, 11.16_ _Design: D14_ -_Validation: local-compaction history/integration tests green._ +_Validation: local-compaction history/integration/billing tests green, including failed-Open zero-child-work._ -### Task 4.4 — Implement authority-aware first-turn reinjection +### Task 4.4 — Implement authority-aware boundary-scoped first-turn reinjection -- Serialize the current capsule deterministically into one versioned/delimited bounded continuity block. -- Inject through a canonical helper that uses legal message-authority versus item-authority representation and preserves `Call.Validate` invariants. -- Track PendingInjectionRevision/LastInjectedRevision so one boundary/revision is injected at most once by default. -- Ensure injected text is proxy-owned prior context, not rewritten as a user message or copied into unrelated branch state. -- Add legacy-message and item-authority tests plus duplicate/retry/reload idempotency tests. +- Serialize a locally branch/digest-validated capsule deterministically into one versioned/delimited bounded continuity block and inject through a canonical helper that preserves message/item authority and `Call.Validate` invariants. +- Track `PendingInjection{BoundaryKey,CapsuleRevision}` plus `LastReleasedInjection{BranchBinding,BoundaryKey,CapsuleRevision}`; use a call-local marker only to prevent duplicate insertion inside one primary retry/failover lifecycle. +- On callback error/panic, validation failure, or primary Open failure, restore the pre-injection call and keep pending injection unchanged; bind a completion-only preview boundary to its committed transaction after successful Open. +- Advance `LastReleasedInjection` and clear matching pending state only after successful final client release so a failed/aborted turn retries, while a later distinct compaction boundary may reinject the same capsule revision. +- Add legacy-message/item-authority tests plus same-revision-two-boundaries, validation-failure, failed-Open, aborted-release-then-retry, retry/failover and reload idempotency tests. -_Requirements: 7.7–7.11, 8.1–8.5, 11.13_ +_Requirements: 7.7–7.11, 7.16, 8.1–8.5, 10.5, 11.13, 11.17_ -_Design: D15_ +_Design: D2; D15_ -_Validation: lipapi/runtime feature tests green for both canonical authorities._ +_Validation: lipapi/runtime feature tests green for both canonical authorities and all reinjection commit/failure points._ ### Task 4.5 — Implement verified plaintext augmentation and opaque exact-preservation fallback - Define the minimal verified plaintext continuation-carrier capability/matcher; default unknown/native compaction paths to no response mutation. - Mechanically merge the ready capsule projection only on an allowed plaintext carrier; never use another LLM to rewrite it. -- Mark pending first-turn reinjection whenever carrier is opaque/unsupported or extractor result is not ready. +- Mark boundary/revision-scoped pending first-turn reinjection whenever carrier is opaque/unsupported or extractor result is not ready. - Add exact byte comparisons for `CompactionItem.EncryptedContent`, `Opaque`, signatures and unknown extension blobs with feature enabled/disabled. -- Prove reinjection alone preserves continuity when result augmentation is unavailable. +- Prove reinjection alone preserves continuity when result augmentation is unavailable and a later boundary can reuse the same capsule revision safely. -_Requirements: 7.5–7.7, 7.15, 11.9, 11.15_ +_Requirements: 7.5–7.7, 7.11, 7.15, 11.9, 11.15, 11.17_ -_Design: D16_ +_Design: D15–D16_ _Validation: opaque/native protocol fixtures green with byte identity._ @@ -294,11 +295,11 @@ _Validation: opaque/native protocol fixtures green with byte identity._ - Reuse existing auxiliary lineage as the source of a bounded workload class/role in usage, metering, billing and report correlation. - Keep pricing/rating behavior unchanged unless existing operator policy explicitly differentiates the selected route/model. -- Prove the child receives its own BillingCallID and per-B-leg provider COGS while account identity remains the originating principal. -- Cover child failover, pre-submit credit rejection and submitted-but-discarded result accounting. -- Make Task 1.5 billing tests green without adding a feature-owned financial store. +- Prove every auxiliary/failover B-leg has its own BillingCallID association, positive authoritative AttemptSeq, usage/cost presence evidence, Source, Authority and DedupeKey required by the existing independent terminal-accounting boundary while account identity remains the originating principal. +- Cover child failover, `AttemptSeq <= 0` rejection, pre-submit credit rejection and submitted-but-discarded result accounting without losing provider COGS/usage obligations. +- Make Task 1.5 billing tests green without adding a feature-owned financial store or alternate settlement path. -_Requirements: 6.1–6.13, 10.8, 11.6–11.7_ +_Requirements: 6.1–6.14, 10.8, 11.6–11.7_ _Design: D6_ @@ -309,10 +310,10 @@ _Validation: focused billing/metering/report tests green._ - Prove primary frontend usage events/totals contain only primary call usage even when the extractor runs concurrently. - Prove account/operator totals include both primary and auxiliary records and can distinguish continuity workload. - Prove detached child causes no primary session transcript entry, TurnID increment, last-activity mutation, resume effect or client session header. -- Prove private child A-leg/attempt lineage remains available for operator/accounting diagnostics. -- Cover cancellation/timeout/failover paths for both parent and child without cross-settlement. +- Prove private child A-leg/attempt lineage remains available for operator/accounting diagnostics while continuity state remains keyed by the captured parent BranchKey. +- Cover cancellation/timeout/failover paths for both parent and child without cross-settlement or cross-branch merge. -_Requirements: 4.1–4.3, 5.10–5.15, 6.4–6.9, 11.4, 11.6_ +_Requirements: 4.1–4.3, 4.16, 5.10–5.15, 6.4–6.9, 11.4, 11.6, 11.16_ _Validation: frontend/session/B2BUA/billing integration tests green._ @@ -321,24 +322,24 @@ _Validation: frontend/session/B2BUA/billing integration tests green._ - Resolve effective continuity policy as operator hard maxima > trusted session values > global feature defaults. - Allow only explicitly approved per-session enable/category/route/tighter-limit overrides; reject/ignore unauthenticated client attempts as specified. - Apply existing redaction/secret treatment before semantic child egress and preserve tenant/workspace authorization on any optional transcript read. -- Keep transcript-disabled sessions from acquiring a hidden durable transcript and isolate bounded source by branch key. +- Keep transcript-disabled sessions from acquiring a hidden durable transcript, isolate bounded source by parent branch key, and keep raw BranchKey identifiers out of remote extractor input unless independently required/authorized. - Add adversarial prompt-injection/secret/tool-output fixtures proving source text cannot override extractor instructions or leak excluded payloads. -_Requirements: 9.1–9.13, 10.1–10.2_ +_Requirements: 2.16, 9.1–9.13, 10.1–10.2_ -_Design: D17; D20_ +_Design: D8; D17; D20_ _Validation: feature/security/session-policy tests green._ ### Task 5.4 — Implement failure handling and content-free observability -- Add metrics for previews/events, carrier hits, eligibility skips, job queue/outcomes, token usage, capsule sizes/revisions, barriers, augmentation/reinjection and stale conflicts. -- Log only bounded IDs/hashes/status/counts and never extractor prompt/output/capsule text. -- Implement explicit fail-open handling for queue saturation, generation retain failure, child admission denial, provider failure, invalid schema, stale result and barrier timeout. -- Ensure disable/reload stops new jobs but preserves bounded completion/accounting of already-submitted work. -- Add tests proving no failure path spins/retries indefinitely, chooses an unconfigured model, or blocks shutdown indefinitely. +- Add metrics for previews/events, preview intents, carrier hits, eligibility skips, job queue/outcomes, token usage, capsule digest/size/revisions, barriers, augmentation/reinjection/watermark outcomes, accounting rejection and preserver callback failures. +- Log only bounded IDs/hashes/status/counts and never extractor prompt/output/capsule text or raw BranchKey identifiers. +- Implement explicit fail-open handling for preserver callback error/panic/rollback, queue saturation, generation retain failure, child admission denial, provider failure, invalid schema/digest/conflict, stale result and barrier timeout. +- Ensure disable/reload stops new jobs but preserves bounded completion/accounting of already-submitted work and coherently expires non-billable preview intents. +- Add tests proving no failure path spins/retries indefinitely, chooses an unconfigured model, changes primary retry authority, or blocks shutdown indefinitely. -_Requirements: 9.6–9.7, 10.3–10.12_ +_Requirements: 7.16, 9.6–9.7, 10.3–10.13_ _Design: D19–D21_ @@ -346,13 +347,13 @@ _Validation: metrics/log/failure-path tests green._ ### Task 5.5 — Document operator behavior, billing cost, and durability limits -- Document feature enablement, independent extractor selector/model, worker/barrier/capsule bounds and fail-open semantics in standard configuration/operator docs. -- State explicitly that extractor inference is additional billable user-attributed usage and show how auxiliary cost is distinguished from primary inference. +- Document disabled-by-default feature enablement, independent extractor selector/model, worker/barrier/capsule bounds and fail-open semantics in standard configuration/operator docs. +- State explicitly that extractor inference is additional billable user-attributed usage, that no fresh child work starts before successful primary Open, and show how auxiliary cost is distinguished from primary inference. - Document remote history egress/privacy implications and that the extractor is off-session/no-tools. - Document v1 process/generation durability versus process-restart limitations and optional authorized transcript reconstruction behavior. -- Document #312 prerequisite and troubleshooting for missing detector/background services or rejected extractor billing/admission. +- Document #312 prerequisite and troubleshooting for missing detector/background services, callback fail-open, or rejected extractor billing/admission. -_Requirements: 6.12–6.13, 8.13–8.15, 9.1–9.2, 10.1–10.3_ +_Requirements: 6.12–6.13, 7.1, 8.13–8.15, 9.1–9.2, 10.1–10.5_ _Validation: docs/config examples pass repository docs/config validation._ @@ -361,33 +362,33 @@ _Validation: docs/config examples pass repository docs/config validation._ ### Task 6.1 — Run the full repeated-compaction semantic matrix - Exercise at least three successive compactions with accepted plan, product decisions, constraints, rationale, rejections and open questions. -- Change a decision after the first compaction and prove the older value remains superseded through later compactions. -- Advance plan steps through pending -> in-progress -> completed and prove pending/current work survives while completed history can be pruned. +- Change a decision after the first compaction and prove the older value is deterministically superseded by conflict key/validated supersedes and remains non-active through later compactions. +- Advance plan steps through pending -> in-progress -> completed and prove pending/current work survives while completed history can be pruned and the capsule digest revalidates after each revision. - Exercise deterministic-only, semantic-extractor and mixed paths and prove no duplicate model call when deterministic state is sufficient. -- Verify capsule/prompt size remains bounded without monotonically duplicated facts/injection blocks. +- Verify capsule/prompt size remains bounded and two opaque compaction boundaries can reinject the same capsule revision once per boundary without monotonically duplicated facts/injection blocks. -_Requirements: 2.6–2.14, 3.5–3.7, 8.10–8.12, 11.8_ +_Requirements: 2.6–2.17, 3.5–3.7, 7.11, 8.10–8.12, 11.8, 11.17_ _Validation: deterministic multi-compaction integration suite green._ -### Task 6.2 — Certify concurrency, generation reload, and stale-result safety +### Task 6.2 — Certify concurrency, generation reload, parent-branch ownership, and stale-result safety -- Race concurrent primary turns, late extractor completion and explicit user correction on one branch; newer explicit intent must win. -- Reload to a new immutable generation while an extractor job is queued/running and prove BranchCoordinator/job ownership survives correctly. +- Race concurrent primary turns, late extractor completion and explicit user correction on one parent branch; newer explicit intent must win. +- Reload to a new immutable generation while an extractor job is queued/running and prove BranchCoordinator/job ownership survives with the original captured parent branch binding despite a distinct child A-leg. - Change/disable extractor config across reload and prove old jobs retain captured route/budgets while new jobs use new policy or do not submit. -- Exercise branch reset/new A-leg/fork-no-parent and prove no capsule/job leakage between branches. -- Verify pending result/job expiry coherently clears BranchState without stale injection. +- Exercise branch reset/new A-leg/fork-no-parent and prove no capsule/job/preview-intent/injection leakage between branches. +- Verify pending result/job/preview-intent expiry coherently clears BranchState without stale injection or child-branch migration. -_Requirements: 4.13–4.15, 8.1–8.10, 10.9–10.12, 11.11_ +_Requirements: 4.13–4.16, 8.1–8.10, 10.9–10.12, 11.11, 11.16_ _Validation: focused `-race` reload/concurrency suite green._ ### Task 6.3 — Certify worker/resource shutdown and leak freedom - Run queue saturation, worker timeout, client cancellation, process shutdown and late completion under `-race` and goleak. -- Assert every successful background submission retains/releases exactly one `KindAsync` pin and no job starts after scheduler close linearizes. +- Assert every successful background submission retains/releases exactly one `KindAsync` pin, non-billable preview intents retain none, and no job starts after scheduler close linearizes. - Assert no provider/model call or plugin callback runs while scheduler/BranchCoordinator internal locks are held. -- Assert bounded queue/result/branch state under sustained synthetic compaction load. +- Assert bounded queue/result/branch/preview-intent state under sustained synthetic compaction load. - Run existing generation pin/ProcessServices shutdown tests to ensure no ownership regression. _Requirements: 4.4–4.15, 8.9, 11.2, 11.11–11.13_ @@ -397,23 +398,23 @@ _Validation: race/goleak/process lifecycle tests green._ ### Task 6.4 — Run architecture and security scope gates - Add/execute architecture tests forbidding provider/frontend DTO imports, direct provider clients and provider-specific continuity branches in core/feature packages. -- Prove detached mode cannot be set through frontend/wire fields and continuity source/capsule/raw result is absent from ordinary logs/money records. +- Prove detached mode cannot be set through frontend/wire fields and continuity source/capsule/raw result/raw BranchKey is absent from ordinary logs/money records and unnecessary extractor egress. - Prove no second transcript database, feature-owned money ledger, generic arbitrary-task scheduler/service locator or durable job framework was added. -- Prove prerequisite detector retains one signature authority and metadata observers remain content-free/non-mutating. -- Prove opaque/encrypted compaction bytes and existing no-retry-after-output/request authority semantics remain unchanged. +- Prove prerequisite detector retains one signature authority, preview intents use its boundary identity, metadata observers remain content-free/non-mutating, and preserver error handling does not become generic retry authority. +- Prove opaque/encrypted compaction bytes, existing no-retry-after-output/request authority semantics, and billing independent-leg validation remain unchanged. -_Requirements: 1.1–1.4, 7.6, 9.3–9.13, 11.9, 11.12–11.15_ +_Requirements: 1.1–1.4, 2.16, 7.6, 7.16, 9.3–9.13, 11.9, 11.12–11.15_ _Validation: `go test ./internal/archtest/...` plus security/contract suites green._ ### Task 6.5 — Run final repository gates and simplification review - Run focused packages, `make quality-checks`, `make test-unit`, required race/goleak suites and deterministic config/docs checks without external model credentials. -- Review implementation diff for redundant abstractions, duplicate rule/state ownership, unbounded work, hidden provider/session/billing bypasses and unnecessary public API expansion. -- Confirm normal requests with feature disabled/no candidate state incur only negligible bounded checks and no auxiliary model calls. -- Record implementation evidence separating UX/cost trade-off: preserved continuity and extra billed auxiliary usage/latency only at bounded barriers. -- If the feature requires a general workflow engine, second money path, or unsafe opaque mutation to pass tests, re-scope rather than weaken the frozen requirements. +- Review implementation diff for redundant abstractions, duplicate rule/state ownership, unbounded work, hidden provider/session/billing bypasses, child-vs-parent branch confusion and unnecessary public API expansion. +- Confirm normal requests with feature disabled/no candidate state incur only negligible bounded checks and no auxiliary model calls or preview-intent leaks. +- Record implementation evidence separating UX/cost trade-off: preserved continuity and extra billed auxiliary usage/latency only after successful Open plus bounded barriers for already-submitted work. +- If the feature requires a general workflow engine, second money path, pre-open billable child, child-keyed continuity state, or unsafe opaque mutation to pass tests, re-scope rather than weaken the frozen requirements. -_Requirements: 10.3–10.12, 11.1–11.15_ +_Requirements: 10.3–10.13, 11.1–11.17_ _Validation: full repository release-quality gates green; final architecture remains within the spec boundaries._ From c789933befd5521db149ed7e113a22d41136aab6 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Tue, 18 Aug 2026 20:37:12 +0200 Subject: [PATCH 18/18] spec: reconcile brownfield validation with CodeRabbit review --- .../design-review.md | 254 +++++++++++++++--- 1 file changed, 217 insertions(+), 37 deletions(-) diff --git a/.kiro/specs/compaction-continuity-preservation/design-review.md b/.kiro/specs/compaction-continuity-preservation/design-review.md index ba717879..8d32952c 100644 --- a/.kiro/specs/compaction-continuity-preservation/design-review.md +++ b/.kiro/specs/compaction-continuity-preservation/design-review.md @@ -2,7 +2,9 @@ ## Verdict -**GO after correction loop.** The design fits current Go-LIP ownership boundaries and reuses existing routing, auxiliary execution, generation pins, B2BUA, secure-session, extension and billing authorities. Initial design review identified four issues that must be corrected before task generation: final-release detector ordering, private child A-leg semantics, useful late-result retention, and explicit prerequisite/fail-closed composition when the #312 detector surface is unavailable. +**GO after correction loops.** The design fits current Go-LIP ownership boundaries and reuses existing routing, auxiliary execution, generation pins, B2BUA, secure-session, extension and billing authorities. + +The initial brownfield validation identified four architectural corrections before task generation: final-release detector ordering, private child A-leg semantics, useful late-result retention, and explicit prerequisite/fail-closed composition when the #312 detector surface is unavailable. A later CodeRabbit review identified additional specification-level correctness gaps around field-level billing evidence, preservation callback failure handling, capsule self-validation, decision conflict identity, completion-only pre-open billing/coalescing, parent-branch ownership, reinjection commit identity, and the disabled-by-default example. Those gaps are now incorporated into the requirements, design, and task plan. No provider-specific implementation, second transcript database, second billing path, or generic workflow engine is required. @@ -20,6 +22,7 @@ Reviewed against current `main` and steering constraints, including: - `internal/infra/runtimebundle/{process_services,shared_mutable}.go`; - `pkg/lipsdk/session/{view,opener}.go`; - `internal/core/runtime/executor_prepare_secure.go`; +- `internal/core/runtime/billing_leg.go`; - `internal/core/state/{partition,mem}.go`; - `internal/infra/billingcompose/identity.go`; - `.kiro/steering/{structure,testing,routing-and-orchestration}.md`. @@ -57,7 +60,7 @@ selected event ### Status -**Accepted; requirements/design must be corrected.** +**PASS after correction.** ## Finding V2 — Detached child needs its own private A-leg, not no A-leg and not the parent A-leg @@ -89,7 +92,7 @@ Rules: ### Status -**Accepted; requirements/design must be corrected.** +**PASS after correction.** ## Finding V3 — A short raw-result TTL can lose a valid background extraction before the next turn @@ -110,13 +113,13 @@ Pending-job result retention must be useful for the branch, while remaining boun ### Status -**Accepted; config/result-lifetime design must be corrected.** +**PASS after correction.** ## Finding V4 — Feature composition must fail clearly when prerequisite preservation/detector services are absent ### Initial concern -#312 is not runtime code on current `main`. A feature plugin silently receiving disabled compaction services would appear enabled while doing nothing, which is misleading and hard to operate. +#312 is not runtime code on the original validation baseline. A feature plugin silently receiving disabled compaction services would appear enabled while doing nothing, which is misleading and hard to operate. ### Correction @@ -131,7 +134,7 @@ This is configuration fail-closed, not request traffic fail-closed. ### Status -**Accepted; design/config tasks must pin it.** +**PASS after correction.** ## Finding V5 — Process-owned branch coordination is justified despite feature-state guidance @@ -143,8 +146,8 @@ The extension platform generally prefers plugin state through `lipsdk/state.Stor Keep the process-owned **narrow BranchCoordinator** as a synchronization facade and use process `ExtensionState` as its serialized backing where practical. The coordinator owns only: -- branch-key lock/serialization; -- revision/high-watermark/job/injection metadata; +- parent branch-key lock/serialization; +- revision/high-watermark/job/preview-intent/injection metadata; - bounded entry TTL/eviction; - opaque capsule/source blobs. @@ -154,7 +157,7 @@ This is smaller and safer than adding a generic transactional plugin-store API s ### Status -**Accepted as designed.** +**PASS as designed.** ## Finding V6 — Background auxiliary scheduler is a reusable primitive but must remain narrow @@ -166,9 +169,11 @@ Adding an async service can become a generic workflow engine. The scheduler accepts only canonical auxiliary model collection requests and returns bounded collection jobs. It cannot schedule arbitrary functions, timers, cron, durable tasks, webhooks, or user-defined callbacks. ProcessServices owns it; generation snapshots receive non-owning adapters that capture the current runner/pin at submission. +Completion-only pre-open identity is held as a non-billable BranchCoordinator preview intent, not as a scheduler job. Only a committed post-Open transaction may create a new billable BackgroundAux submission. + ### Status -**Accepted with architecture tests.** +**PASS with architecture tests.** ## Finding V7 — Detached mode must not become a client bypass around secure session @@ -182,23 +187,29 @@ Detached mode is carried only on the trusted auxiliary SDK/internal request obje ### Status -**Accepted with security tests.** +**PASS with security tests.** -## Finding V8 — Billing requirements align with current money architecture +## Finding V8 — Billing requirements must include the current independent-leg evidence contract ### Validation -Current billing account identity is principal-scope-derived; detached child can preserve this scope. Normal Executor submission then naturally receives independent BillingCallID/exposure/terminal usage/provider COGS. No stream-time financial mutation is required. +Billing account identity is principal-scope-derived, so the detached child can preserve the originating account while receiving an independent BillingCallID and private child A-leg/B-legs. Normal Executor submission can therefore reuse existing exposure, terminal usage, settlement, and provider COGS authorities. + +The runtime contract is stricter than merely assigning a distinct BillingCallID: -The only additive need is a content-free auxiliary workload/role projection for reports/diagnostics. That metadata must not influence pricing implicitly. +- independent B-leg persistence requires an authoritative positive `AttemptSeq`; records with `AttemptSeq <= 0` are rejected rather than assigned synthetic order; +- final billing evidence carries the existing token/usage quantities and presence bits, cost evidence/presence, `Source`, `Authority`, and `DedupeKey`; +- retry/failover legs must satisfy the same field-level contract independently. + +The continuity spec therefore requires these invariants for every auxiliary/failover B-leg and requires RED coverage for rejected non-positive AttemptSeq and invalid accounting evidence. Workload/role classification remains content-free and must not implicitly change pricing. ### Status -**PASS.** +**PASS after field-level billing correction; RED tests required.** ## Finding V9 — Primary protocol usage separation is achievable -The child is an independent internal call and is never encoded into the primary frontend response. Existing protocol usage remains scoped to primary canonical stream. Account-level durable usage/billing includes child records separately. +The child is an independent internal call and is never encoded into the primary frontend response. Existing protocol usage remains scoped to the primary canonical stream. Account-level durable usage/billing includes child records separately. ### Status @@ -214,6 +225,150 @@ Reinjection on the first eligible post-compaction request is the universal safe **PASS after design explicitly treats reinjection as mandatory fallback.** +## Finding V11 — Preserver callback errors must be explicitly fail-open and mutation-safe + +### Review concern + +`BeforeRequest`, `RequestOpened`, and `BeforeResponseRelease` return `error`. Without a composition rule, a preservation-only timeout/state/sanitizer error could escape into primary handling and make native compaction unusable. `BeforeRequest`/response-finalization can also mutate canonical objects before returning an error. + +### Correction + +- core dispatch isolates callback panic/error and records content-free preservation failure rather than propagating it as primary model/provider failure; +- `BeforeRequest` and `BeforeResponseRelease` run through a bounded transactional clone/undo helper; +- callback error, panic, or post-mutation canonical validation failure restores the exact pre-preservation `Call`/`Event`; +- `RequestOpened` failure cannot undo the already-opened primary request and only affects preservation-local state; +- preservation failure never changes routing, failover or no-retry-after-output authority. + +This remains a focused mutation rollback seam, not a general transaction framework. + +### Status + +**PASS after correction; callback error/panic/rollback tests required.** + +## Finding V12 — Serialized capsule must carry branch binding and canonical content digest + +### Review concern + +The requirement already called for branch identity and content digest, but the capsule JSON example kept them only outside the serialized envelope. Such bytes could not self-validate their scope/integrity after registry transfer, reinjection, recovery or reload. + +### Correction + +Capsule V1 now carries: + +- `branch_binding`: a stable content-free digest derived from the authoritative **parent** BranchKey; +- `content_digest`: SHA-256 over the canonical versioned envelope excluding only the digest field itself. + +The digest scope includes schema version, revision, source high-watermark, branch binding and semantic payload. Registry consume, merge, reinjection/projection, recovery and reload reuse validate binding + digest before use. Raw account/session identifiers need not be exported to the extractor model. + +### Status + +**PASS after correction; canonical encoding and mismatch tests required.** + +## Finding V13 — Decision conflict identity cannot rely on extractor-generated fact IDs + +### Review concern + +A semantically conflicting decision can arrive under a fresh semantic fact ID, so ID reuse alone cannot ensure that later explicit corrections supersede older active choices. + +### Correction + +Decision facts now separate: + +- stable fact `id` for provenance/history; +- normalized `conflict_key` for the semantic decision slot; +- optional validated `supersedes` references for explicit cross-slot corrections. + +At most one decision may remain active per conflict key. A newer decision for an occupied conflict key deterministically supersedes the older active decision under the existing authority/revision precedence. Unknown/cross-branch `supersedes` references are rejected. + +### Status + +**PASS after correction; contradictory-active-decision tests required.** + +## Finding V14 — Completion-only extraction cannot submit fresh billable work before primary Open + +### Review concern + +The earlier completion-only flow called `SubmitCollect` before the first post-compaction primary B-leg opened. If that Open failed, the detached child could still produce real provider usage, contradicting failed-Open zero-billing semantics. The earlier coalescing rule also required a transaction that did not yet exist. + +### Correction + +Before Open, completion-only flow may only: + +1. use detector pure preview to derive a stable boundary/fingerprint; +2. derive/coalesce a **non-billable** preview intent from parent branch + preview boundary + target source revision; +3. merge deterministic state; +4. await a matching job that was already submitted by an earlier successfully opened request; +5. inject already-ready/deterministic continuity if needed. + +Only after successful primary Open does detector state commit, the preview intent bind to the committed transaction, and a fresh billable BackgroundAux job become eligible for submission. A failed Open creates no new child BillingCallID/B-leg/provider work. + +This intentionally accepts that semantic information first discoverable at this boundary may improve later turns rather than justify premature provider billing. + +### Status + +**PASS after correction; failed-Open zero-child-work and preview-intent binding tests required.** + +## Finding V15 — Continuity state belongs to the parent branch, never the detached child A-leg + +### Review concern + +The child must have a private A-leg for execution, but if a worker derives BranchCoordinator identity from that A-leg, late results update an auxiliary branch and cannot protect the primary conversation. + +### Correction + +The authoritative parent BranchKey is captured before auxiliary submission/child A-leg creation and stored as a content-free branch binding with pending job state. The parent key is used for: + +- preview intent binding; +- `Await` result ownership checks; +- capsule merge/revision; +- pending injection/reinjection; +- late-result and reload coordination. + +The child A-leg remains only child execution/routing/billing authority and lineage. Tests must use different parent and child A-leg IDs. + +### Status + +**PASS after correction; distinct-parent/child concurrency/reload tests required.** + +## Finding V16 — Reinjection identity and commit point must be boundary-scoped + +### Review concern + +A revision-only `LastInjectedRevision` can wrongly suppress the same capsule revision after a second opaque compaction. Advancing a durable watermark at insertion time is also too early: canonical validation, primary Open, or final release can still fail, causing a later retry to skip required continuity. + +### Correction + +Reinjection now uses a compound identity: + +```text +(parent branch binding, compaction boundary/transaction, capsule revision) +``` + +- the same revision may be injected again for a later distinct compaction boundary; +- a call-local ephemeral marker prevents duplicate insertion during one request's internal retry/failover lifecycle; +- callback/validation failure restores the pre-injection call and leaves pending state; +- failed primary Open leaves pending state; +- branch-level `LastReleasedInjection` advances and matching pending state clears only after successful final client release; +- an aborted/no-release turn therefore retries continuity on a later eligible request. + +### Status + +**PASS after correction; same-revision/two-boundary and failure-then-retry tests required.** + +## Finding V17 — Configuration example must not contradict disabled-by-default behavior + +### Review concern + +The example had `enabled: true` while the security requirement says the new egress/billed feature is disabled by default. Copying the example could unintentionally enable remote extraction. + +### Correction + +The illustrative configuration is now `enabled: false` and explicitly described as disabled unless the operator opts in. The remote route remains illustrative only. + +### Status + +**PASS after correction.** + ## Validation Checklist | Check | Result | @@ -222,18 +377,30 @@ Reinjection on the first eligible post-compaction request is the universal safe | detector observer remains content-free/non-mutating | PASS | | actual final event observed by committed detector | PASS after V1 correction | | strict extraction not billed before primary Open | PASS | +| completion-only fresh extraction not billed before primary Open | PASS after V14 correction | +| completion-only preview identity binds to committed transaction | PASS after V14 correction | | real background execution with submit-time generation ownership | PASS | | worker process-owned across generation reload | PASS | | no unbounded goroutine/queue | PASS | | off-primary-session semantics | PASS after V2 correction | +| private child A-leg never becomes continuity branch key | PASS after V15 correction | | independent extractor selector | PASS | | parent route override cannot hijack child | PASS | | same user/account billed by default | PASS | | separate auxiliary BillingCallID/B-legs | PASS | +| auxiliary/failover AttemptSeq > 0 | PASS after V8 correction | +| auxiliary usage/cost/Source/Authority/DedupeKey evidence pinned | PASS after V8 correction | | primary protocol usage unchanged | PASS | +| preserver callback failure is fail-open | PASS after V11 correction | +| failed preservation mutation rolls back canonical object | PASS after V11 correction | +| capsule carries parent branch binding and canonical digest | PASS after V12 correction | +| decision conflicts use deterministic conflict identity | PASS after V13 correction | | late job result remains useful | PASS after V3 correction | +| reinjection dedupe is branch/boundary/revision scoped | PASS after V16 correction | +| reinjection watermark commits only after final client release | PASS after V16 correction | | encrypted/opaque content immutable | PASS | | mandatory reinjection fallback | PASS | +| shipped example remains disabled by default | PASS after V17 correction | | no second transcript DB | PASS | | no provider-specific core branches | PASS | | no generic workflow engine | PASS | @@ -246,22 +413,22 @@ Reinjection on the first eligible post-compaction request is the universal safe | Design decision/component | Primary requirements | |---|---| -| shared detector preview + prerequisite | 1.1–1.12 | -| separate Preserver FeatureBundle surface | 1.2–1.4, 7.14, 11.11–11.12 | -| Continuity Capsule + merge precedence | 2.1–2.15, 8.4, 8.11–8.12 | +| shared detector preview + prerequisite | 1.1–1.13, 4.10 | +| separate Preserver FeatureBundle surface + rollback | 1.2–1.4, 7.14, 7.16, 10.4–10.5, 11.12–11.13 | +| Continuity Capsule + branch digest + decision precedence | 2.1–2.17, 8.1, 8.4, 8.11–8.12 | | deterministic carrier catalog | 3.1–3.7 | | sanitizer/source window | 3.8–3.16, 9.3–9.10 | -| BackgroundAux scheduler | 4.1–4.15, 11.2, 11.10–11.11 | -| detached child + private A-leg | 5.5–5.13, 11.4–11.5 | +| BackgroundAux scheduler + committed coalescing | 4.1–4.15, 11.2, 11.10–11.11 | +| detached child + parent branch ownership | 4.16, 5.5–5.15, 8.1–8.3, 11.4–11.5, 11.16 | | explicit extractor route | 5.1–5.5, 10.1–10.2 | -| normal user billing + workload class | 6.1–6.13, 10.7, 11.6–11.7 | -| remote strict-compaction flow | 7.1–7.8 | -| authority-aware reinjection | 7.7–7.15 | -| branch coordinator | 8.1–8.12, 11.8, 11.10 | -| reload/restart behavior | 8.9–8.15, 10.8–10.9 | +| normal user billing + field-level B-leg evidence | 6.1–6.14, 10.7–10.8, 11.6–11.7 | +| strict and completion-only compaction flows | 7.1–7.16, 11.3 | +| boundary-scoped authority-aware reinjection | 7.7–7.16, 8.5, 11.17 | +| branch coordinator / preview intents / watermarks | 4.10, 4.16, 8.1–8.12, 11.8, 11.10–11.11, 11.16–11.17 | +| reload/restart behavior | 8.9–8.15, 10.8–10.10 | | trusted policy/privacy | 9.1–9.13 | -| failure/observability | 10.1–10.11 | -| architecture/testing gates | 11.1–11.14 | +| failure/observability | 10.1–10.13 | +| architecture/testing gates | 11.1–11.17 | ## Simplification Review @@ -270,6 +437,8 @@ The chosen design adds two infrastructure capabilities only because existing sea 1. **BackgroundAux scheduler** — required because current Aux.Collect is synchronous and post-lease spawn is unsafe. 2. **BranchCoordinator** — required because process ExtensionState lacks atomic revision update and feature-instance locks do not span generations. +The CodeRabbit corrections do **not** justify further infrastructure. Preview intents, injection watermarks, parent-branch bindings and preservation rollback remain narrow state/contracts inside those existing seams; they do not require a generic transaction service, workflow scheduler or durable queue. + Everything else is additive use/refactoring of existing authorities. In particular, do **not** add: - a second Executor/provider client; @@ -288,15 +457,26 @@ Everything else is additive use/refactoring of existing authorities. In particul 2. child accidentally resumes parent secure session -> transcript/turn/activity RED tests. 3. child inherits parent route override -> explicit different-route RED test. 4. parent/child BillingCallID collision -> billing RED test. -5. primary protocol usage includes child tokens -> frontend usage RED test. -6. detector commits completion before preservation final event -> ResponsePreview/ResponseReleased ordering test. -7. opaque `EncryptedContent` altered -> exact bytes RED test. -8. job result expires before next turn -> pending-result retention/expiry RED test. -9. stale worker overwrites user correction -> revision race RED test. -10. generation reload drops job/coordinator -> retained-generation RED test. -11. queue saturation creates goroutine fallback -> goroutine-count/goleak RED test. -12. disable/reload drops already-submitted billing settlement -> terminal accounting RED test. +5. auxiliary `AttemptSeq <= 0` or invalid final evidence silently loses accounting -> field-level billing RED tests. +6. primary protocol usage includes child tokens -> frontend usage RED test. +7. detector commits completion before preservation final event -> ResponsePreview/ResponseReleased ordering test. +8. preserver error/panic aborts native traffic or leaves partial mutation -> rollback/fail-open RED tests. +9. capsule branch binding/digest mismatch is consumed -> transfer/reload/reinjection integrity RED tests. +10. semantic new ID allows contradictory active decision -> conflict-key/supersedes RED tests. +11. completion-only pre-open path submits provider work -> failed-Open zero-child-billing RED test. +12. preview intent fails to bind/coalesce after Open -> retry/failover duplicate-submission RED test. +13. child A-leg is used as continuity key -> distinct parent/child late-result RED test. +14. revision-only injection suppresses later opaque boundary -> same-revision/two-boundary RED test. +15. injection watermark advances before validation/Open/final release -> failure-then-retry RED test. +16. opaque `EncryptedContent` altered -> exact bytes RED test. +17. job result expires before next turn -> pending-result retention/expiry RED test. +18. stale worker overwrites user correction -> revision race RED test. +19. generation reload drops job/coordinator -> retained-generation RED test. +20. queue saturation creates goroutine fallback -> goroutine-count/goleak RED test. +21. disable/reload drops already-submitted billing settlement -> terminal accounting RED test. ## Final Gate -After applying V1–V4 corrections to requirements/design, the architecture is **GO** for TDD task generation. The implementation should be split into phases that establish background/detached/billing correctness before semantic extractor prompting and final compaction integration. +After the original V1–V4 correction loop and the later V8/V11–V17 CodeRabbit correction pass, the specification is **GO** for TDD implementation. Requirements, design, validation, and tasks now agree on the critical lifecycle rules: parent-branch state ownership, no fresh billable completion-only child before successful primary Open, current independent-leg billing evidence, fail-open transactional preservation callbacks, self-validating capsule bytes, deterministic decision conflict replacement, and boundary-scoped reinjection committed only after final client release. + +The implementation should establish detector/preservation/background/detached/billing/branch contracts with RED tests before semantic extractor prompting and final compaction integration. If implementation requires pre-open billable child work, child-keyed continuity state, a second billing path, a generic workflow/transaction engine, or opaque provider mutation to pass, the design must be re-scoped rather than weakening these invariants.