diff --git a/.kiro/specs/backend-connector-resource-reconciliation/design-review.md b/.kiro/specs/backend-connector-resource-reconciliation/design-review.md new file mode 100644 index 00000000..7002a32f --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/design-review.md @@ -0,0 +1,191 @@ +# Brownfield Design Validation + +## Verdict + +**GO after CodeRabbit lifecycle/concurrency hardening.** The selected design remains a focused, high-ROI extension of Go-LIP's existing generation architecture, but the first review correctly identified several places where the original prose was weaker than the safety claim. Those findings were cross-checked against `processhost.Host`, runtimebundle ownership, adapter cleanup, and the standard backend-plugin host Session and have been incorporated into normative requirements/design/tasks. + +The resulting design still targets O(N) -> O(K) physical connector reconstruction without introducing a general Cordis runtime, processhost redesign, public ABI/config surface, dynamic discovery, or request-path lookup. + +## Review Findings and Disposition + +### Detached entries survive invalidation — VALID / FIXED + +Original `current[identity]` indexing was insufficient for process shutdown: invalidation can remove a resource from the reusable map while old generations still retain leases. `Pool.Close` could not then enumerate it. + +The design now requires a process-owned `owned`/all-entry set containing every successfully constructed incarnation until entry-level physical cleanup completes. Invalidation detaches only from `current`; terminal Close snapshots both current and detached residual entries through that ownership set. + +### Pending waiter reservation race — VALID / FIXED + +Original prose allowed waiters to increment refs only after a building entry became live. A fast first claimant could therefore release to zero before a scheduled waiter formally acquired its ref. + +Every waiter now reserves its prospective lease claim under the pool mutex **before** waiting. Cancellation abandons only that claim. A deterministic scheduling test must hold a waiter after reservation and prove first-lease release cannot physically close the resource. + +### Lease once versus physical cleanup once — VALID / FIXED + +Per-lease `sync.Once` only prevents one lease from releasing twice. It does not protect physical cleanup when final Release races `Pool.Close`. + +The physical entry now owns a separate cleanup-once operation and stored result. Final lease release and fail-safe process shutdown converge on that same operation. Detached invalidation does not create another physical cleanup owner. + +### Cleanup ownership handoff — VALID / CLARIFIED + +Current construction has two existing cleanup pieces: adapter/session cleanup and `ActivateResult.Cleanup` (`processhost.CloseInstance`). The pool entry consumes one composite per-resource cleanup on a physical miss; generations receive only lease release. + +This is **not** a transfer of process supervision out of `processhost.Host`. Host keeps slot/instance state, invalidation, reaping, and `Host.Close` fail-safe authority. The pool owns semantic retention and timing of that existing per-resource cleanup capability. + +An exactly-once test must count both composite cleanup pieces and subsequent Host.Close. + +### Pool Close can hang on an unbounded builder — VALID / FIXED + +`ProcessServices.Close` is contextless/synchronous. The original “wait for builders” language was incomplete because current discovered construction can call Activate with a background lifetime. + +The pool now owns a cancelable build root. An absent Acquire starts exactly one pool-owned builder goroutine; caller contexts control only claimant waiting. `Pool.Close` linearizes closing, cancels the build root, then joins builders before physical residual cleanup/host teardown. A blocked-builder test must exit only on that cancellation and prove no late publication. + +### Identity scenario coverage — VALID IN PRINCIPLE / LITERAL SUGGESTION NARROWED + +CodeRabbit correctly requested stronger proof for artifact, secret, policy, and process-model identity dimensions. However, those dimensions are startup-fixed in the current production discovered-factory closure and are not all hot-reloadable through SIGHUP. + +The corrected plan therefore uses two evidence matrices: + +1. high-cardinality **generation reload** evidence for dimensions that actually vary during current reload plus invalidation; +2. focused **physical identity/construction** evidence for artifact digest, secret fingerprint, normalized runtime policy, process model, factory/logical identity. + +`shared_artifact` remains a non-pooled/restart-required fallback rather than being treated as a pooled replacement scenario. This covers the correctness concern without inventing unsupported hot artifact/policy/process-model reload behavior. + +### Acquire/Close linearization — VALID / FIXED + +The original requirements said only that Close rejects new acquisitions and is race-safe. That is not enough to prevent a pending builder/Acquire from handing out a resource after residual cleanup begins. + +The design now gives Close a mutex-protected terminal linearization point. After `closing=true`, no new claim is reserved and no build result is handed off as post-close success. Close cancels builders, waits builders and Acquire handoffs, then snapshots/cleans residual owned entries. + +### Shared connector operation concurrency — VALID / FIXED WITH EXISTING CONTRACT + +Cross-checking the standard production host shows: + +- `backendplugin/host.Session.Execute` already serializes Execute calls with `lifecycleMu` and serializes Execute versus Close; +- Resolve/ListModels/CountTokens/FinalizeBilling can overlap via the existing host/server instance lease model. + +The spec therefore does **not** invent a new connector concurrency flag or remove Session serialization. It now explicitly characterizes retained-old-generation Execute overlapping new-generation Execute through one pooled Session and covers metadata/auxiliary overlap under race/conformance tests. + +Sharing one Session extends existing Execute serialization across generations and can remove the incidental transient parallelism provided today by two fresh Sessions. This is now documented as an operational tradeoff rather than hidden behind unconditional observational-equivalence wording. If that measured behavior is unacceptable for target workloads, pooling is re-scoped rather than host concurrency being redesigned here. + +## Validation Checklist + +### Generation consistency — PASS + +Published generations remain immutable. Changed identity builds a replacement before publication; removed resources remain available only through old generation leases. No live substitution exists. + +### `ResourceLedger` authority — PASS + +Every generation receives one fresh lease release. The physical composite cleanup never enters multiple ledgers. `buildBackends` remains the generation cleanup transfer point. + +### Processhost authority — PASS + +The pool contains no process launch, peer authentication, process-tree cleanup, slot supervision, or request transport logic. `processhost.Host` remains the physical supervisor and terminal fail-safe. + +### Process ownership transfer — PASS + +Pool is created beside host before factory installation, captured lexically, then transferred to `ProcessServices`. Reverse close order remains pool -> host -> artifacts -> staging on successful ownership transfer and bootstrap failure. + +### Physical cleanup exactly once — PASS after correction + +Entry-level cleanup-once unifies final release and pool fail-safe shutdown. Generation lease once remains only a claim-release guard. + +### Detached ownership — PASS after correction + +All successful physical incarnations remain in process ownership until cleanup completes even if removed from semantic reuse by invalidation. + +### Acquire/waiter protocol — PASS after correction + +Claim reservation occurs before waiting. The first claimant cannot close the resource while another active waiter remains unaccounted for. + +### Acquire/Close shutdown boundary — PASS after correction + +Close has an explicit terminal linearization point, cancels pool builders, joins builders/handoffs, and prevents late publication before fail-safe cleanup. + +### Builder lifetime — PASS after correction + +Pool—not an arbitrary caller—owns physical builder cancellation/join. Caller cancellation is local to its reservation. This aligns with contextless ProcessServices shutdown without creating a permanent worker subsystem. + +### Physical identity — PASS + +Separate private identity includes/configures treatment for artifact, instance/factory, process model, opaque Configure bytes, normalized policy and secret fingerprint. DTO/input drift is fail-closed. + +### Identity evidence realism — PASS after correction + +Startup-fixed physical inputs are covered by focused identity/construction tests, not falsely presented as current hot-reload dimensions. High-cardinality reload evidence remains aligned with actual reloadability. + +### Candidate last-good isolation — PASS + +Reuse hit is query-only for candidate preparation. Candidate rollback only releases its claim and does not mutate/reconfigure/close/invalidate the shared last-good resource. + +### Standard Session concurrency — PASS with explicit operational gate + +The design preserves the current host Session concurrency implementation. Cross-generation Execute serialization and metadata/auxiliary overlap receive direct tests. No public concurrency ABI is added. + +### Request hot path — PASS + +Pool operations occur only at generation construction/retirement/process shutdown. Normal request execution uses captured backend functions. + +### `shared_artifact` and built-in exclusions — PASS + +Neither is pulled into the first implementation. Existing restart-required/shared-process behavior is unchanged. + +### Security/ABI — PASS + +Verified artifact binding, secure IPC, peer authentication before Configure/secrets, environment restrictions, process-tree cleanup, and current backend-plugin ABI remain unchanged. + +### ROI gate — PASS + +Deterministic 100-enabled-connector work counts remain the primary justification. Supporting evidence now also records the standard Session cross-generation execution-scheduling tradeoff so the optimization is evaluated as a whole. + +## Design-to-Requirement Trace + +| Requirement | Design coverage | +|---|---| +| R1 scale evidence | high-cardinality reload count matrix and re-scope gate | +| R2 narrow boundary | discovered overlap-safe per-instance only; pool above processhost | +| R3 identity | complete private key, startup-fixed/reload-varying split, drift gate | +| R4 Acquire/Close | pre-reserved claims, pool-owned builder, linearized Close, no late publish | +| R5 generation semantics | immutable projections, changed/remove/rollback, query-only candidate reuse | +| R6 invalidation | exact incarnation, detached process ownership, fresh replacement | +| R7 cleanup/shutdown | entry-level cleanup once; pool -> host -> artifacts -> staging | +| R8 concurrency/non-interference | preserve Session contract; characterize cross-generation serialization | +| R9 TDD/architecture | scheduling/race/blocked-builder/ownership/identity/ROI gates | + +## Simplification Review + +The hardened design still rejects: + +1. generic resource manager/container; +2. processhost generation awareness; +3. overloading `BackendStateIdentity`; +4. idle TTL cache; +5. public feature/concurrency flags; +6. shared model registry; +7. dynamic plugin reconciliation; +8. Cordis requires/provides graph/fibers; +9. Session concurrency redesign. + +The new `owned` entry set, builder cancellation root, and entry cleanup-once are accepted because they close concrete correctness holes introduced by resource sharing; they are not general-purpose framework concepts. + +## Implementation Risks to Pin With Tests + +- waiter reserved too late and resource reaches zero before handoff; +- invalidated detached entry disappears from process shutdown ownership; +- final release and Pool.Close double-run session/host cleanup; +- Close waits forever for a builder using the wrong lifetime context; +- builder publishes after Close linearizes; +- Acquire hands out a resource after fail-safe cleanup begins; +- stale invalidation detaches a replacement incarnation; +- physical identity omits a Configure/launch input; +- startup-fixed identity dimension is accidentally treated as hot reload without redesign; +- candidate rollback mutates/invalidates last-good shared resource; +- standard Session cross-generation Execute serialization causes hidden deadlock/cancellation regression; +- metadata/Count/Finalize overlap exposes connector race; +- pool begins duplicating processhost or appears in request/public surfaces. + +## Final Gate + +**GO.** All technically valid CodeRabbit lifecycle/concurrency findings are now represented in normative requirements, design mechanics, and TDD tasks. The identity-matrix suggestion was adopted with a brownfield correction: startup-fixed inputs receive focused identity/construction coverage rather than fictional hot-reload scenarios. + +Implementation remains approval-gated by `spec.json`. The implementation must still pass the deterministic scale gate and may be re-scoped if the existing standard Session execution-serialization tradeoff erodes the expected operational ROI. diff --git a/.kiro/specs/backend-connector-resource-reconciliation/design.md b/.kiro/specs/backend-connector-resource-reconciliation/design.md new file mode 100644 index 00000000..213559c1 --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/design.md @@ -0,0 +1,634 @@ +# Design Document + +## Overview + +This design removes redundant physical reconstruction of unchanged discovered executable `per_instance` backend connectors during material runtime-generation reload while preserving Go-LIP's immutable-generation consistency model. + +The architecture adds one package-private, process-scoped connector-resource reconciliation owner above `processhost.Host`. The owner maps a complete semantic physical-resource identity to the current reusable physical incarnation while also retaining an ownership set of every successfully constructed incarnation that has not completed physical cleanup. Generation compilation acquires a lease claim. An unchanged candidate receives the already-configured backend/session resource plus a generation-owned lease release; a changed or invalidated identity constructs a fresh physical incarnation exactly through the current host/adapter path. + +`ResourceLedger` remains the generation cleanup authority. `processhost.Host` remains the sole executable-process/IPC supervisor. The pool does **not** take over process slots, peer authentication, invalidation, reaping, or `Host.Close`; it only owns semantic reconciliation, dependent claims, and the timing of the existing per-resource composite cleanup. + +The design is intentionally not a component runtime. There is no generic dependency graph, service locator, DI container, HMR, live plugin discovery, reusable public resource framework, or request-hot-path lookup. + +### Goals + +- Reduce physical connector reconstruction on material reload from all enabled eligible connectors to only changed or unusable identities. +- Preserve immutable `GenerationRuntime`, request/async generation pinning, last-good rollback, retained old-generation work, and no-drop retirement. +- Make physical identity complete/fail-closed for all construction/configure inputs. +- Give each physical incarnation one entry-level exactly-once cleanup capability while allowing several generation leases. +- Make Acquire/Close/build shutdown behavior linearizable and cancellation-safe. +- Preserve processhost security and supervision rather than layering a second process manager. +- Preserve established standard host-session operation concurrency and explicitly measure the cross-generation effect of sharing one Session. +- Establish deterministic high-cardinality operation-count evidence before enabling production reuse. + +### Non-Goals + +- Replace `runtimehost.Manager`, `GenerationRuntime`, `ResourceLedger`, `ProcessServices`, or `processhost.Host`. +- Pool built-in/in-process backend factories. +- Change `shared_artifact` process semantics or restart-required overlap behavior. +- Share generation-local executor/model-registry/catalog/routing/feature/policy/billing state. +- Add connector watchers, rescans, HMR, dynamic install/upgrade/remove, or an idle TTL cache. +- Add a public resource-pool API, manifest capability, ABI field, config flag, or concurrency flag. +- Increase standard `backendplugin/host.Session` Execute parallelism or remove its current lifecycle serialization. +- Guarantee that an external connector can never fail while a candidate performs a query-shaped operation. + +## Boundary Commitments + +| Concern | Existing authority | Treatment | +|---|---|---| +| process lifetime | `ProcessServices` / `processResourceOwner` | owns pool shutdown | +| generation cleanup | `ResourceLedger` | owns lease release | +| request/async lifetime | runtimehost generation refs | unchanged | +| process/IPC supervision | `processhost.Host` | unchanged | +| exact executable trust | `VerifiedArtifact` | digest contributes to identity | +| connector protocol | `pkg/lipsdk/backendplugin` | unchanged | +| discovered catalog | startup-fixed discovery/trust | unchanged | +| generation projections | runtimebundle/compiler | rebuilt per generation | +| physical cleanup timing | new private entry | composite cleanup exactly once | + +Revalidate this design if a configure/launch input changes lifetime, `ConfiguredInstance` gains a new generation-preparation lifecycle action, processhost ownership semantics change, reuse expands beyond discovered `per_instance`, standard Session concurrency changes, or request execution begins consulting the pool. + +## Existing Architecture and Target + +Today an unchanged discovered `per_instance` backend is reconstructed for every material generation: + +```text +Gen17 -> logical backend A -> physical A#17 + +compile Gen18 + -> same logical backend A -> physical A#18 (Activate + Configure) + +publish Gen18 + old work -> A#17 + new work -> A#18 + +retire Gen17 -> cleanup A#17 +``` + +The unique host activation handle is deliberate and remains required whenever a **new** physical incarnation is constructed. + +Target for an unchanged identity: + +```text +ProcessServices + └─ backendResourcePool + └─ identity A -> incarnation 41 + ├─ backend/session functions + ├─ composite physical cleanup (entry-owned) + ├─ claims = 2 + └─ owned until cleanup completes + ▲ ▲ + Gen17 Gen18 +``` + +Changed identity remains two physical resources: + +```text +A-old -> incarnation 41 <- Gen17 +A-new -> incarnation 42 <- Gen18 candidate +``` + +## Selected Architecture + +```mermaid +graph TB + Install[Discovered install preparation] --> Host[processhost.Host] + Install --> Pool[backendResourcePool] + Install --> Factory[discovered factory closures] + Factory --> Pool + Pool --> Physical[configured connector incarnation] + Physical --> Host + Process[ProcessServices] --> Pool + Process --> Host + Compile[Generation compiler] --> Factory + Factory --> Lease[BackendBuildResult: backend + lease release] + Lease --> Ledger[ResourceLedger] + Compile --> Gen[GenerationRuntime] +``` + +The pool is a connector-specific reconciliation/ownership index, not a service registry. Request execution receives only backend functions already captured in an immutable generation. + +## Construction and Ownership Timing + +Discovered factory closures are installed before `ProcessServices` exists. Therefore the pool is created beside the discovered host and captured lexically: + +```text +prepareDiscoveredPluginInstall + acquire staging + verify artifacts + create processhost.Host + create backendResourcePool + install factories capturing host + pool + │ + ▼ +BuildHost + transfer host + pool + artifacts + staging + │ + ▼ +NewProcessServices + register staging cleanup + register artifact cleanup + register host cleanup + register pool cleanup + +reverse process close order: + pool -> host -> artifacts -> staging +``` + +The same relative order is required on pre-transfer/bootstrap failure. There is no global registry or setter race. + +## Private State Model + +Illustrative shapes: + +```go +type backendResourcePool struct { + mu sync.Mutex + closing bool + nextInc uint64 + + current map[backendResourceIdentity]*backendResourceEntry + owned map[*backendResourceEntry]struct{} // physical success until cleanup completes + + buildCtx context.Context + cancelBuild context.CancelFunc + buildWG sync.WaitGroup + + // Tracks Acquire calls that have entered the handoff protocol so Close can + // establish a terminal boundary before residual cleanup. + handoffWG sync.WaitGroup +} + +type backendResourceEntry struct { + identity backendResourceIdentity + incarnation uint64 + state backendResourceState + claims int + ready chan struct{} + + backend execbackend.Backend + cleanup func() error + buildErr error + + cleanupOnce sync.Once + cleanupErr error +} + +type backendResourceLease struct { + pool *backendResourcePool + entry *backendResourceEntry + once sync.Once // one claim release only +} +``` + +Exact implementation may use a condition/counter instead of `handoffWG`, and may merge fields when safe. Required semantics are more important than these names. + +There is deliberately no generic `Resource[T]`, `Scope`, `Provider`, `Component`, `Context`, `Registry`, or public `LeaseManager` abstraction. + +## Physical Resource Identity + +### Principle + +Identity answers: + +> Can two generation builds safely execute through the same already-configured connector instance without another physical Configure/build? + +False negatives cost optimization. False positives can violate correctness, so equality is conservative. + +### Inputs + +At the physical construction/configure choke point, identity treatment covers: + +1. logical configured instance ID; +2. factory kind; +3. exact `VerifiedArtifact.DigestHex`; +4. process model; +5. exact effective opaque Configure YAML bytes; +6. normalized `backendplugin.RuntimePolicy`, including host-owned normalization such as `DisableTransportRetries=true`; +7. configure-time `SecretBundle` by private digest when present; +8. any future generation-varying launch/configure input. + +Use SHA-256 with domain-separated, length-delimited fields. Runtime policy projection explicitly enumerates every field. Secret names are sorted; names/values are length-framed and hashed; plaintext does not survive identity construction or appear in logs/status/errors. + +`BackendStateIdentity` remains a separate, narrower affinity/health continuity contract and is not the physical reuse key. + +### Startup-fixed versus reload-varying inputs + +Current production discovered factory closures capture artifact/process-model and install-time runtime policy at startup; the current discovered path also does not hot-rotate an artifact through SIGHUP. These facts still participate in identity treatment because their lifetime can evolve and focused construction tests can exercise them, but the high-cardinality **reload** matrix must not pretend they are currently reloadable. + +Evidence is split: + +- generation-reload matrix: unchanged/config-changed/remove/invalidate dimensions that current reload can actually exercise; +- focused identity/construction matrix: artifact digest, secret fingerprint, normalized policy, process model, factory/instance identity. + +`shared_artifact` remains non-pooled; a process-model difference that leaves `per_instance` eligibility does not mean “build another pooled resource.” + +### Drift/fail-closed gate + +A structural contract test must force deliberate review when the Configure/physical input surface changes. Production hashing should remain explicit rather than reflection-driven. If completeness cannot be proven, the path uses current isolated construction. + +## Resource State Machine and Linearization + +Conceptual states: + +```text +building -> live -> detached -> closed + │ │ ▲ + └------> failed │ invalidation/final-release/close removes current +``` + +`current` is only the reusable semantic index. `owned` contains every successfully constructed physical entry until entry-level physical cleanup completes, even after invalidation detaches it from `current`. + +### Acquire/Close linearization + +`Acquire` and `Close` use the same pool mutex for their terminal decision: + +- successful Acquire claim reservation/handoff linearizes while `closing == false`; +- Close linearizes when it sets `closing = true` under that mutex; +- after that point no new claim may be reserved and no pending build result may be published/handed off as a post-close success. + +Close cancels pool-owned builders and waits both builder completion and Acquire handoff completion before residual cleanup. This prevents a lease from being handed a physical resource after fail-safe cleanup has already run. + +### Pool-owned builder lifetime + +An absent-key Acquire does not make the caller the physical builder owner. It: + +1. installs a building entry and reserves the caller's prospective claim under the mutex; +2. starts exactly one short-lived **pool-owned** builder goroutine using a context derived from `pool.buildCtx`; +3. increments `buildWG` before the goroutine becomes runnable; +4. the caller then waits like any other claimant on `ready` or its own context. + +The physical builder must receive the pool-owned context through `processhost.Activate`/Configure. The pooled path must not use `context.Background()` for the build lifetime. + +Caller cancellation abandons only that caller's reserved claim. It does not cancel a build that may serve other claimants. If every claim disappears before a build succeeds, the completed physical result is not published as an idle entry; it is cleaned immediately. + +Pool Close calls `cancelBuild()` before `buildWG.Wait()`. The processhost/transport stack is already context-aware; a blocked-build test proves the path exits on cancellation and cannot publish late. + +### Waiter reservation protocol + +For a building entry, every caller increments `claims` **before** it waits. The claim is already the ownership reservation that will become its lease if the build succeeds. + +```text +Acquire(building) + lock + closing? -> reject + claims++ // reserve before wait + unlock + + wait ready | caller ctx + + canceled -> abandon reserved claim + ready -> re-check state / close boundary + success -> return lease for existing reserved claim +``` + +This removes the race in which the first caller builds and releases to zero before a waiter has incremented a ref. A deterministic scheduling test holds a waiter after reservation, releases the first returned lease, then proves physical cleanup cannot occur until the waiter abandons/releases its claim. + +### Build completion + +On success, builder completion reacquires the mutex: + +- if pool is closing or the entry has zero remaining claims, do not publish it as reusable; record physical ownership long enough to clean it and wake waiters; +- otherwise attach backend/composite cleanup, add entry to `owned`, mark live, and wake waiters; +- no external cleanup/Configure/launch runs under the mutex. + +On failure: + +- remove the building entry from `current` if still exact; +- store build error and mark failed; +- wake waiters; +- do not negative-cache it; +- waiter claim abandonment eventually reaches zero without a physical cleanup because construction never completed. + +A later independent Acquire may retry. + +### Live reuse + +A live hit reserves/increments its claim under the mutex and returns the immutable backend value plus a fresh lease release. It performs no factory call, process activation, Configure, or adapter build. + +### Release + +Lease `sync.Once` only prevents one generation lease from releasing its claim twice. It is **not** the physical cleanup authority. + +On release: + +```text +lease.once: + lock + decrement exact entry claim + if claims > 0 -> unlock, return + if current[key] == entry -> remove from current + mark detached + unlock + entry.cleanupPhysical() +``` + +`entry.cleanupPhysical()` owns a separate `sync.Once` and stored result. Every path that may physically tear down the pooled resource—normal final release and pool fail-safe shutdown—calls this same method. It removes the entry from `owned` only after cleanup completes. + +Thus final Release racing Pool.Close cannot execute physical cleanup twice. + +## Physical Cleanup Ownership Handoff + +This design distinguishes **process supervision ownership** from the **per-resource cleanup capability**. + +Current physical construction produces two cleanup responsibilities: + +1. `adapter.Build(...).Cleanup()` -> closes the configured host Session / connector instance RPC-side resource; +2. `ActivateResult.Cleanup` -> calls `processhost.Host.CloseInstance(hostActivationID)`, which removes the host instance and reaps its process slot when appropriate. + +`buildDiscoveredPhysical` shall return one idempotent composite physical cleanup that preserves the current ordering/error-join behavior across those two operations. On a pool miss, the reconciliation entry consumes that composite cleanup. It is never copied into a generation ledger. + +Each generation receives only: + +```go +pluginreg.BackendBuildResult{ + Backend: entry.backend, + Cleanup: lease.Release, +} +``` + +`buildBackends` continues transferring that cleanup into `ResourceLedger`. + +`processhost.Host` retains its internal `instances`/`slots`, invalidation/reap logic, and `Host.Close` fail-safe cleanup. The pool does not own a `Process` object or replace host supervision; it owns only when the existing per-instance composite cleanup may be invoked without violating another generation's lease. + +An exactly-once regression shall count session close, `CloseInstance`/activation cleanup, entry cleanup, and later `Host.Close` to prove these paths do not become competing physical owners. + +## Physical Construction Integration + +Refactor `buildDiscoveredBackend` only enough to separate: + +1. effective input/eligibility preparation; +2. physical construction of a new unique host activation/session/adapter resource; +3. pool Acquire returning a leased generation result. + +Conceptually: + +```go +func buildDiscoveredBackend(...) (pluginreg.BackendBuildResult, error) { + input, err := prepareDiscoveredPhysicalInput(...) + if err != nil { ... } + if pool == nil || !eligible(input) { + return buildDiscoveredPhysical(ctx, input) + } + + id, shareable, err := physicalIdentity(input) + if err != nil { ... } + if !shareable { + return buildDiscoveredPhysical(ctx, input) + } + + return pool.Acquire(ctx, id, func(buildCtx context.Context, inc uint64) (physicalBackendResource, error) { + return buildDiscoveredPhysical(buildCtx, input, inc) + }) +} +``` + +Every **new** per-instance physical build retains the current unique host activation ID behavior. Pool reuse avoids entering that build path at all. + +## Candidate Preparation and Generation-Local State + +A reuse hit performs no Configure/Start/Stop/Close/mutating preflight. Candidate rollback releases only its lease and cannot invalidate the shared resource merely because unrelated candidate validation failed. + +Generation-local structures are still recreated: + +```text +leased backend values + -> BackendInventory + -> new modelregistry.Runtime / snapshot / refresh ownership + -> new executor/routing/policy/billing views + -> new handler / GenerationBundle / ResourceLedger +``` + +Query-shaped metadata operations may therefore reach the same physical Session from overlapping generations. This is acceptable only under the established standard-host concurrency behavior below. + +If future generation preparation introduces a mutating lifecycle call against an external connector, that path becomes non-shareable until separately redesigned. + +## Established Connector Operation Concurrency + +### Standard host contract today + +Production uses `backendplugin/host.Session` through the default discovered connector path. + +- `Session.Execute` holds the existing `lifecycleMu` for the full execute RPC; two Execute calls on the same Session are serialized, and `Close` cannot tear down the transport during Execute. +- `Resolve`, `ListModels`, optional `CountTokens`, and optional `FinalizeBilling` do not take that client lifecycle mutex. The gRPC server already leases a configured instance around those calls; connectors are therefore already exposed to metadata/auxiliary overlap with other operations today. + +This specification preserves those facts. It does not remove `lifecycleMu`, introduce a new semaphore, or change the public `ConfiguredInstance` ABI. + +### Cross-generation consequence + +With fresh Gen17/Gen18 Sessions today, overlap can transiently provide two independent Execute serialization domains. With one pooled Session, retained Gen17 and new Gen18 Execute calls share the same existing serialization domain. + +That is a capacity/scheduling change during overlap, not a canonical request transformation. It must be explicit rather than hidden behind an “observationally identical” claim. + +A deterministic test shall hold an old-generation Execute open, start a new-generation Execute on the same pooled Session, and prove behavior follows the existing Session serialization without deadlock, cancellation corruption, or wrong-generation cleanup. Supporting benchmark/evidence should record the overlap effect. + +If that established serialization makes connector reuse operationally unacceptable for the intended long-lived-stream workload, implementation must re-scope instead of changing Session concurrency under this spec. + +Focused race/conformance tests shall also cover overlapping-generation `Resolve`, `ListModels`, CountTokens, FinalizeBilling, and execution through the standard host. Non-standard injected/test session implementations are pooled only when they satisfy the same established host behavior; otherwise they bypass reuse. + +## Invalidation and Detached Entries + +Semantic identity and physical incarnation differ: + +```text +identity X + incarnation 7 -- fails/detaches, still leased by Gen17 + incarnation 8 -- new current resource for later generation +``` + +Invalidation flow for an entry: + +1. under pool synchronization compare the exact `(identity, entry/incarnation)`; +2. if it is the current entry, remove only that exact incarnation from `current` and mark detached; +3. leave it in `owned` while existing lease claims remain or until terminal pool cleanup invokes entry cleanup; +4. future Acquire sees no current reusable entry and may build a new incarnation; +5. delegate the physical process-generation invalidation/reap to existing `processhost.Host`; +6. do not live-substitute a replacement into published generations; +7. stale callbacks from incarnation 7 cannot remove incarnation 8. + +Final lease release later invokes the same entry-level cleanup once; it may find the process already reaped, using existing idempotent/error-normalization semantics. + +Tracking detached entries in `owned` is essential: terminal `Pool.Close` can enumerate and fail-safe clean an invalidated resource even if a retained generation leaked/failed to release its lease before process teardown. + +## Process Shutdown + +`runtimebundle.Host.Close` remains the process shutdown coordinator, and runtimehost generation drain remains the expected first stage. + +Relevant target order: + +```text +generation admission stopped / generations drain + -> lease releases + -> ProcessServices.Close + backendResourcePool.Close + 1. lock: set closing (linearization point), reject later Acquire + 2. cancel pool build context + 3. unlock + 4. wait pool-owned builders and Acquire handoffs + 5. lock: detach/snapshot all residual owned entries, including invalidated/detached + 6. unlock + 7. call entry.cleanupPhysical on residual entries + processhost.Host.Close + VerifiedArtifact.Close + staging removal +``` + +Pool Close never holds its mutex while waiting for builders/handoffs or running physical cleanup. Builders use pool-owned cancellation and cannot publish after the close boundary. Entry cleanup is once-guarded, so a concurrent final lease release and shutdown converge safely. + +Under normal successful host shutdown, generation drain should make residual claims empty before ProcessServices close. Residual cleanup remains a terminal fail-safe for broken/aborted ownership paths; after terminal process shutdown, old leases are not promised a usable connector. + +## Error Handling + +- Preserve existing runtimebundle/processhost error wrapping and public reload categories. +- Add no public `resource_reuse_failed` category. +- Failed build leaves no reusable current resource and is not permanently cached. +- Caller cancellation while waiting returns the caller context error and abandons only its reservation. +- Pool Close cancellation is pool-owned and terminates builders for process shutdown. +- Final normal lease cleanup errors flow through existing ResourceLedger aggregation. +- Residual process-shutdown cleanup errors join existing ProcessServices close errors. +- Entry cleanup stores one result so racing cleanup callers observe one physical cleanup outcome. + +## Concurrency Rules to Prove + +1. two concurrent absent Acquires -> one builder, two pre-reserved claims, one incarnation; +2. first returned lease release before second waiter wakes -> no physical cleanup until waiter abandons/releases; +3. waiter cancellation -> only that claim drops; builder/other claims survive; +4. final release racing new Acquire -> Acquire either reserves before detach or builds after detach; never receives closing entry; +5. Close linearizes against Acquire -> no post-close claim/build publication; +6. Close cancels a blocked pool-owned builder and waits it; no late publication after host teardown; +7. final release racing Close -> entry-level physical cleanup exactly once; +8. invalidation detaches current but leaves entry process-owned until cleanup; Close can enumerate it; +9. stale invalidation cannot detach newer incarnation; +10. candidate rollback racing retained old-generation release preserves correct claim count and cleanup timing. + +## Scale and Identity Evidence + +### High-cardinality generation-reload matrix + +At least 100 synthetic discovered `per_instance` connector rows, no external credentials, deterministic counters for factory physical build, activation/launch, Configure, cleanup, lease acquire/release. + +| Scenario | Expected physical construction | +|---|---| +| baseline unrelated reload before reuse | O(N), characterize current behavior | +| target unrelated reload, N unchanged | 0 new builds/activations/Configure | +| one/K backend configs changed | exactly 1/K new physical resources | +| remove/disable subset | no build for removed rows; old resource retained by old generation | +| candidate rollback after reuse hits | no physical cleanup of active resources | +| candidate builds K new then fails | K new resources cleaned | +| invalidate one then compile same config | exactly one fresh incarnation | + +### Focused identity/construction matrix + +Independent focused tests exercise physical input dimensions that are startup-fixed in current production reload but are correctness-critical identity inputs: + +| Difference | Expected | +|---|---| +| artifact digest | identity miss; fresh physical build when otherwise eligible | +| secret fingerprint | identity miss; fresh build when secrets are effective input | +| normalized RuntimePolicy | identity miss; fresh build | +| process model | no alias; `shared_artifact` follows existing non-pooled/restart-required path | +| factory kind / logical instance | no alias | + +This separation avoids claiming hot artifact/policy/process-model reload support that does not exist today. + +### Supporting benchmark + +Record candidate compile time/allocations and available synthetic/native resource observations. Primary correctness/ROI remains deterministic operation counts. No request throughput/token-latency gain is claimed. + +Also characterize retained-old-generation/new-generation Execute scheduling on one pooled standard Session so the known serialization tradeoff is visible in implementation evidence. + +## File Structure Plan + +Possible private files: + +```text +internal/infra/runtimebundle/ + backend_resource_identity.go + backend_resource_pool.go + backend_resource_identity_test.go + backend_resource_pool_test.go + discovered_factories.go + plugin_catalog.go / composition_root.go / process_services*.go + reload_backend_resource_reuse_test.go + +internal/archtest/ + backend_resource_reconciliation_test.go +``` + +No new generic `resource`, `container`, `dependency`, or lifecycle framework package is required. + +## Testing Strategy + +TDD order: + +1. baseline/high-cardinality and identity RED tests; +2. reserved-claim/Acquire-Close/detached ownership/builder cancellation RED tests; +3. exactly-once physical cleanup/ownership handoff RED tests; +4. candidate isolation and standard Session operation-concurrency RED tests; +5. private identity + pool implementation; +6. process ownership transfer and discovered-factory integration; +7. generation reload matrix, invalidation and retained-work integration; +8. race/goleak/security/conformance/no-drop regressions; +9. benchmark/evidence and final simplification gate. + +Existing ResourceLedger, processhost, discovered overlap/restart-required, backend security/conformance, retained-generation, and reload last-good/no-drop suites remain regression authorities. + +## Rejected Alternatives + +### Reconfigure an existing resource in place + +Rejected: mutates provider state under an old generation. + +### Put semantic generation reconciliation in processhost + +Rejected: conflates LIP configuration identity with physical process supervision. + +### Reuse `BackendStateIdentity` + +Rejected: does not cover artifact, policy, secret, process-model, or future physical inputs. + +### Return one `BackendBuildResult` cleanup to multiple generations + +Rejected: permits early/double physical teardown. + +### Track only `current` entries + +Rejected after review: invalidation removes an entry from `current` while retained generations can still lease it, leaving Pool.Close unable to enumerate residual owned resources. + +### Let the initiating Acquire own the builder + +Rejected after review: caller cancellation/shutdown lifetime becomes ambiguous and can leave Close waiting on an unbounded background operation. The pool owns builder context/goroutine lifetime. + +### Increment waiter refs only after build publication + +Rejected after review: the first claimant can release to zero before a scheduled waiter acquires its ref. Claims are reserved before waiting. + +### Add idle cache or TTL + +Rejected: adds eviction/resource-pressure policy with no demonstrated need. + +### Change Session Execute concurrency as part of reuse + +Rejected: this would broaden the refactor into host/ABI concurrency semantics. Preserve current serialization, measure its overlap effect, and re-scope pooling if unacceptable. + +### Pool all backend types / share whole model runtime + +Rejected: no evidence-backed ROI and would weaken clear generation ownership. + +## Design Success Criteria + +The refactor succeeds only if: + +1. unchanged eligible reloads produce zero new physical Activate/Configure work; +2. changed/unusable identities get fresh physical incarnations before publication; +3. candidate rollback cannot mutate/close the last-good shared connector; +4. every waiting Acquire reserves ownership before waiting, eliminating zero-ref handoff races; +5. Close has a terminal linearization point, cancels/joins pool builders, and prevents late publication; +6. invalidated/detached physical entries remain process-owned/enumerable until cleanup; +7. final release, invalidation aftermath, and process shutdown converge on one entry-level exactly-once physical cleanup; +8. processhost remains the sole physical supervisor and pool close precedes host/artifact/staging teardown; +9. generation-local runtime state remains separate and request execution performs no pool lookup; +10. established standard Session concurrency is preserved and its cross-generation Execute serialization is explicitly characterized; +11. identity tests cover all physical input dimensions without pretending startup-fixed inputs are hot-reloadable; +12. no public config/ABI or generic runtime framework is introduced; +13. deterministic scale evidence still justifies the implementation after accounting for concurrency/lifecycle complexity. diff --git a/.kiro/specs/backend-connector-resource-reconciliation/gap-analysis.md b/.kiro/specs/backend-connector-resource-reconciliation/gap-analysis.md new file mode 100644 index 00000000..a8fee0ea --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/gap-analysis.md @@ -0,0 +1,130 @@ +# Brownfield Requirements Gap Analysis + +## Result + +**PASS after requirements corrections.** The current Go-LIP architecture already has the ownership, immutable-generation, executable-process, and identity primitives needed for a focused reconciliation layer. The missing capability is not general dependency management: it is a process-scoped way for overlapping generations to share one unchanged configured external `per_instance` connector resource without sharing generation-local derived state. + +The initial requirements were directionally correct but needed several brownfield constraints made explicit before design. Those corrections are recorded below and must be reflected in the final `requirements.md`. + +## Existing Brownfield Facts + +- `buildBackends` constructs every enabled backend row during generation compilation, so a material reload can reconstruct unchanged backends. +- Discovered executable `per_instance` factories deliberately mint a generation-unique host activation handle so candidate and active generations can coexist safely today. +- `processhost.Host` already supervises process slots, authenticated IPC, configured instances, invalidation, and cleanup; duplicating those responsibilities would be harmful. +- `BackendStateIdentity` already demonstrates identity-sensitive reuse for affinity/health observation state, but its identity is intentionally narrower than the inputs that define a configured physical connector. +- `ProcessServices` now has the private `processResourceOwner`/owned-acquisition discipline from the earlier atomic-owned-resource-lifecycle work. +- The executable plugin host, verified artifacts, and staging directory are created before `ProcessServices`, then ownership transfers into `ProcessServices` before initial generation compilation. +- Installed connector catalog size is not itself the scaling problem: discovery is lazy with respect to process launch and already has 100-manifest no-launch coverage. + +## Gaps and Required Corrections + +### 1. Installed connector cardinality is not the target scaling dimension + +A large trusted catalog does not imply a large live resource set because discovery does not launch every connector. The expensive case is many **enabled configured process-backed instances** combined with material runtime reload. + +**Correction:** all scale requirements and benchmarks are defined in terms of enabled eligible connector instances, not installed manifest count. + +### 2. `BackendStateIdentity` is too weak for physical-resource reuse + +Its `{InstanceID, FactoryKind, ConfigDigest}` identity is appropriate for affinity/health continuity but does not represent exact executable artifact, configure-time runtime policy, secret values, or process model. + +**Correction:** define a separate private physical-resource identity. Reusing helpers is allowed, but physical reuse must never be authorized solely by `BackendStateIdentity.Compatible`. + +### 3. All configure-affecting inputs must participate in identity or reuse must fail closed + +The executable plugin receives opaque YAML, `SecretBundle`, `RuntimePolicy`, factory/instance identity, and negotiated process context. Future additions to configure-time input create a correctness hazard if the identity silently ignores them. + +**Correction:** require an explicit identity-construction choke point over the effective physical construction/configure input. If an input cannot be safely fingerprinted or declared process-stable, the resource is non-shareable and uses current generation-local construction. Add an architecture/contract test that forces identity review when configure-time DTO shape changes. + +### 4. The pool must exist before discovered lifecycle factories capture it + +Production installs discovered factory closures before `NewProcessServices`, while process ownership is transferred afterward. Creating the pool only inside `NewProcessServices` would arrive too late unless the factory used an indirection/service locator, which this project deliberately avoids. + +**Correction:** create the private reconciliation owner beside the discovered `processhost.Host` during discovered-install preparation, capture it directly in eligible factory closures, then transfer its lifetime into `ProcessServices`. No global lookup or post-construction setter is needed. + +### 5. Pool shutdown ordering is constrained by existing host/artifact/staging ownership + +Physical resource cleanup may need adapter/session cleanup followed by host instance cleanup. Therefore the resource pool must be torn down while `processhost.Host` and verified artifacts are still usable. + +**Correction:** on successful process ownership transfer, register ordering so normal reverse teardown is: generations drain → reconciliation pool/final physical cleanup → `processhost.Host` → verified artifacts → staging removal. Bootstrap-error release must preserve the same dependency order. + +### 6. Generation cleanup and physical cleanup are different ownership concepts + +Current `BackendBuildResult.Cleanup` is transferred into `ResourceLedger`. If that same cleanup were copied into two generations, either generation could close a connector still referenced by the other. + +**Correction:** the pool owns the physical cleanup exactly once. Each generation receives a fresh idempotent **lease release** as its `BackendBuildResult.Cleanup`; `ResourceLedger` continues to own that cleanup. No backend `Close`/lifecycle hook returned to the generation may bypass the lease and directly close the shared physical resource. + +### 7. Semantic identity is not physical incarnation identity + +A connector process/session may die while its desired configuration remains unchanged. Reusing by semantic key alone would hand a dead resource to the next candidate. Conversely, an old failure callback could incorrectly invalidate a newly rebuilt resource with the same semantic key. + +**Correction:** every physical entry has an incarnation token/version. Invalidation detaches only the exact incarnation, making it unavailable to future acquisitions; a later acquire creates a new incarnation. Stale invalidation cannot evict a newer incarnation. + +### 8. `shared_artifact` is a different problem + +The current shared-process model has explicit isolation/concurrency declarations and can be restart-required when overlap is unsafe. Mixing it into first-pass reconciliation would entangle process slot sharing with generation resource sharing. + +**Correction:** first implementation is `per_instance` discovered external connectors only. `shared_artifact` remains unchanged. + +### 9. Generation-local derived state must not be accidentally pooled + +A configured external session can be reused while executor maps, routing views, model-registry runtime, model catalog, feature surface, lifecycle context, and policy/accounting composition remain generation-specific. + +**Correction:** pool only the configured physical connector adapter/backend resource. Rebuild all current generation projections exactly as today from the leased backend/profile/inventory surface. + +### 10. Dynamic inventories/capabilities do not automatically make a resource non-shareable + +The adapter already models dynamic facts through runtime operations such as `Resolve` and `ListModels`. Reconstructing a connector merely because an unrelated generation changed would not make those dynamic facts more correct. + +**Correction:** reuse is allowed when **construction/configure inputs** are identical; dynamic runtime queries remain dynamic. If a connector requires generation-dependent hidden configuration not represented at configure time, it is non-shareable until that dependency is explicit in identity. + +### 11. A timing-only ROI gate would be fragile + +The existing candidate compiler is fast on ordinary fixtures, while real executable connector startup cost can vary by platform and connector. Fixed millisecond thresholds would conflate CI noise with architecture value. + +**Correction:** deterministic operation counts are the primary acceptance evidence. Wall time, allocations, process count/RSS/FD observations, and benchstat remain supporting evidence. + +### 12. The previous Cordis-inspired ownership spec does not already solve reuse + +`atomic-owned-resource-lifecycle` correctly hardened ownership locality and deliberately left backend construction unchanged because that work addressed forgotten cleanup, not cross-generation physical reuse. The new concern appears at connector-scale reload cardinality. + +**Correction:** reuse the existing ownership primitives; do not replace or generalize them. This specification adds one connector-specific lease/reconciliation owner and nothing broader. + +## Brownfield Compatibility Matrix + +| Existing subsystem | Required treatment | +|---|---| +| `runtimehost.Manager` / generation leases | unchanged | +| `GenerationRuntime` immutability | unchanged | +| `ResourceLedger` | owns per-generation lease release; unchanged authority | +| `ProcessServices` | gains lifetime ownership of one private connector reconciliation owner | +| `processResourceOwner` | reused for process teardown registration; not replaced | +| `processhost.Host` | unchanged process/IPC supervisor | +| discovered factory install | captures private pool before ProcessServices ownership transfer | +| `BackendBuildResult` | eligible factory returns leased cleanup instead of physical cleanup | +| `BackendStateIdentity` | remains affinity/health identity; not physical reuse authority | +| model registry/catalog | rebuilt per generation | +| plugin discovery/trust | startup-fixed and unchanged | +| backend-plugin ABI | unchanged | +| built-in backends | unchanged | +| `shared_artifact` connectors | unchanged | +| routing/streaming/retry/accounting | unchanged | +| public config/SDK | no new surface | + +## Corrected Required Invariants + +1. Optimization scope is enabled eligible executable `per_instance` connector resources, not catalog size. +2. Physical reuse requires complete configure/construction identity; incomplete identity falls back safely. +3. Semantic identity and physical incarnation are separate. +4. One physical resource has exactly one physical cleanup owner and many generation lease owners. +5. `ResourceLedger` owns lease release, never a shared physical closer. +6. Candidate rollback is local: releasing a reused lease cannot disturb the last-good generation. +7. Changed/removed resources preserve old-generation availability until drain. +8. Pool shutdown precedes host/artifact teardown and does not become a second process supervisor. +9. Generation-local projections remain generation-local. +10. No generic container, service locator, dynamic dependency graph, watcher, or request-time lookup is introduced. +11. Deterministic high-cardinality operation counts are the primary ROI gate. + +## Requirements Correction Status + +The final requirements must incorporate gaps 3–7 especially: a configure-input identity choke point, pre-`ProcessServices` pool construction/ownership transfer, strict physical-cleanup versus lease-cleanup separation, incarnation-safe invalidation, and explicit teardown order. Once those are present, the requirements quality gate is **PASS**. diff --git a/.kiro/specs/backend-connector-resource-reconciliation/requirements.md b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md new file mode 100644 index 00000000..7fe968f2 --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md @@ -0,0 +1,132 @@ +# Requirements Document + +## Introduction + +Go-LIP shall reduce unnecessary reconstruction of expensive executable backend connector resources across immutable runtime generations when a material configuration reload does not change those connectors. The optimization shall preserve the existing generation publication model: every request remains bound to one immutable `GenerationRuntime`; changed resources are constructed before publication; old generations and their resources remain valid until their existing work drains; candidate failure leaves the last-good generation untouched. + +This specification borrows only the Cordis-v4 ideas that fit this problem: semantic provider identity, reconciliation of unchanged desired resources, explicit physical-incarnation identity, and retention while dependent generations still hold the provider. It does **not** introduce a Cordis component runtime, reactive dependency graph, fibers, dependency injection, service location, HMR, or a generic effect/resource system. + +The first implementation is intentionally narrow. It targets discovered executable backend connectors whose declared process model is `per_instance` and whose current reload policy permits candidate/active overlap. Installed-but-disabled connectors, statically linked backends, and `shared_artifact` connectors remain outside the first implementation. + +## Boundary Context + +- In scope: deterministic scale evidence, private physical connector-resource identity, process-lifetime reconciliation, per-generation leases, candidate rollback, invalidation/incarnation behavior, Acquire/Close linearization, shutdown ordering, established host-session concurrency, and focused executable-connector integration. +- Out of scope: general component reconciliation, frontend/feature reconciliation, dynamic plugin install/uninstall, discovery watchers, new backend-plugin ABI fields, new public configuration knobs, built-in backend pooling, `shared_artifact` behavior changes, request migration between generations, live mutation of a published backend instance, or redesign of host-session execution concurrency. +- Existing authorities remain: `ProcessServices` for process-owned resources, `ResourceLedger` for generation-owned cleanup, `processhost.Host` for executable process/IPC supervision, runtimehost generation leases for request/async lifetime, and the existing backend-plugin ABI for connector behavior. +- Performance intent: eliminate redundant connector construction work during material reloads. This is not a request-hot-path optimization. + +## Requirement 1: Evidence-First Scale Justification + +1.1. Before production reuse is enabled, add a deterministic high-cardinality characterization harness that compiles overlapping generations with at least 100 enabled synthetic host-backed `per_instance` connector instances. +1.2. The harness shall count at minimum physical connector builds, `processhost` activations or launches, Configure operations, and physical cleanup operations; timing/allocation benchmarks may supplement but shall not replace these deterministic counters. +1.3. The baseline shall demonstrate current reconstruction behavior for an unrelated material reload in which connector-defining generation inputs are unchanged. +1.4. After implementation, an unrelated material reload with `N` unchanged eligible live connectors shall perform **zero** new physical connector builds, activations/launches, and Configure operations for those `N` connectors; it may perform `N` lightweight lease acquisitions and rebuild normal generation-local projections. +1.5. For a candidate with `K` changed or unusable eligible connector identities and all remaining eligible connectors unchanged, physical construction shall be proportional only to `K`. +1.6. Test acceptance shall not depend on fixed wall-clock thresholds vulnerable to CI host variance. Benchstat or equivalent measurements may be supporting evidence. +1.7. If these count-based gains cannot be achieved without changing request semantics, weakening shutdown/reload safety, or introducing a general runtime/container abstraction, implementation shall stop or re-scope rather than preserve speculative infrastructure. + +## Requirement 2: Narrow Eligibility and Ownership Boundary + +2.1. Reconciliation shall be private to runtime composition and shall not become a public SDK capability or request-time service locator. +2.2. The initial eligible set shall be discovered executable connectors using `ProcessModelPerInstance` and an overlap-safe reload policy. +2.3. Built-in/in-process backend factories shall retain current generation ownership unless a later evidence-backed specification separately justifies reuse. +2.4. `ProcessModelSharedArtifact` connectors shall retain their current explicit sharing/restart-required semantics; this specification shall not weaken their isolation or overlap gates. +2.5. Plugin discovery/trust shall remain startup-fixed. Connector installation, removal, directory rescanning, executable upgrade discovery, and automatic file watching are not added. +2.6. `processhost.Host` shall remain the sole executable process/IPC supervisor. The reconciliation layer shall not duplicate launch, peer authentication, process-tree cleanup, slot/instance supervision, or transport management. +2.7. No new public YAML field, manifest field, CLI flag, environment variable, or backend-plugin ABI field shall be required merely to enable this internal optimization. +2.8. Because discovered lifecycle factory closures are installed before `ProcessServices` construction, the private reconciliation owner shall be created beside the discovered `processhost.Host`, captured lexically by eligible factory closures, and then have its lifetime transferred into `ProcessServices`; no global registry, service locator, or mutable post-construction lookup may bridge this timing boundary. +2.9. The reconciliation owner shall remain connector-specific and package-private and shall expose no generic keyed `Get`/`Resolve` API for unrelated runtime services. + +## Requirement 3: Semantic Physical Resource Identity + +3.1. Reuse shall require an exact private identity representing the configured physical connector resource, not Go object equality and not only the logical backend instance ID. +3.2. At minimum, identity treatment shall cover logical instance ID, factory kind, exact verified executable artifact digest, process model, opaque connector configuration content, effective configure-time runtime policy, and configure-time secret material by a non-reversible fingerprint. +3.3. Identity construction shall be deterministic for semantically identical effective inputs within one process. +3.4. Secret plaintext shall never be retained in the identity, logged, emitted in diagnostics, or exposed through public status. Secret fingerprinting shall be local, length-framed, deterministic for equality, and private. +3.5. Executable artifact replacement shall produce a distinct identity even when logical instance ID and YAML configuration are unchanged. +3.6. Credential/secret change shall produce a distinct identity when secret material is part of effective Configure input. +3.7. Runtime-policy change shall produce a distinct identity when the normalized policy differs. +3.8. Process-model change shall not accidentally reuse a `per_instance` entry; unsupported/non-eligible process models shall use their existing non-pooled path. +3.9. `BackendStateIdentity` may provide precedent or low-level hashing helpers, but its current `{InstanceID, FactoryKind, ConfigDigest}` contract is insufficient proof that two physical connector resources are interchangeable. +3.10. Identity shall be derived at one explicit construction/configuration choke point. A future configure-time or launch-identity input shall require an intentional identity decision rather than being silently omitted. +3.11. If a construction/configure input varies between generations and cannot be represented safely and deterministically, that resource shall be non-shareable and use current generation-local construction. +3.12. Facts that are startup-fixed in the current production path, including discovered artifact/process-model and install-time runtime policy, need not be presented as hot-reload dimensions; focused identity/construction tests shall still prove they cannot alias if exercised directly, and drift tests shall force review if their lifetime changes. + +## Requirement 4: Process-Scoped Acquire, Waiter Reservation, and Close Linearization + +4.1. An eligible physical connector resource shall have one process-scoped reconciliation entry and zero or more generation lease claims. +4.2. Acquiring an exact live identity that is current shall reserve a claim under the pool mutex and reuse the existing configured resource rather than invoking physical construction. +4.3. For an absent identity, exactly one pool-owned physical builder shall be started. Every caller waiting on a building entry, including the initiating caller, shall reserve its prospective lease claim **before** waiting; caller cancellation releases only that reserved claim and does not cancel a builder needed by other callers. +4.4. A fast first claimant shall not be able to release the resource to zero while another uncanceled waiter for the same build has not yet completed its handoff. The reserved-claim protocol or an equivalent barrier shall make this scheduling race impossible. +4.5. The physical builder shall run under a pool-owned cancellation context, not a caller-owned/background lifetime. Pool shutdown shall cancel that build context before joining builders. +4.6. `Acquire` success and `Close` shall have an explicit linearization order under the same synchronization boundary: once `Close` marks the pool closing, no new Acquire may reserve a claim and no pending Acquire may publish/hand off a newly built resource as a post-close success. +4.7. `Close` shall reject new acquisitions, cancel pool-owned builders, wait for in-flight builders and acquisition handoffs to terminate, and only then perform residual physical cleanup. A builder finishing after closure begins shall clean its result without publishing it as reusable. +4.8. A failed physical build/configure shall not be permanently negative-cached; waiters observe the failure, reservations are released, and a later independent acquisition may retry. +4.9. Each generation-facing lease release shall be idempotent. Releasing one of several claims shall not close the physical resource; final normal release detaches the entry and invokes entry-owned physical cleanup. +4.10. The physical cleanup returned by underlying adapter/process construction shall be retained only by the reconciliation entry. Every generation-facing `BackendBuildResult.Cleanup` for a pooled resource shall be a fresh lease release, never the physical cleanup function. +4.11. Eligible pooled backend values/lifecycle hooks shall expose no alternate generation-owned `Close`/`Stop` path capable of bypassing the lease and tearing down a resource retained by another generation. + +## Requirement 5: Preserve Immutable Generation and Candidate-Isolation Semantics + +5.1. Published `GenerationRuntime` objects remain immutable; no connector resource shall be reconfigured or replaced underneath a published generation. +5.2. If connector identity changes, the candidate shall construct a distinct replacement before publication while the old generation retains its old resource until its leases drain. +5.3. If a connector is removed or disabled, the candidate shall acquire no replacement lease; old generations may continue using the removed connector until their retained work drains. +5.4. A failed candidate that leased an existing resource shall release only its candidate lease and shall not disturb the active generation's lease/resource. +5.5. A failed candidate that created a new resource shall release it through rollback; if no other claim exists, physical cleanup shall complete before rollback completes. +5.6. Generation-owned derived state—including executor maps/views, routing views, model-registry runtime, model catalog, feature composition, policy state, billing composition, and generation lifecycle context—shall remain generation-owned and shall not move into the connector pool. +5.7. Existing no-drop, retained-generation, old-stream/async, and last-good reload guarantees shall remain unchanged. +5.8. A reused configured connector may contribute the same underlying backend/session functions to multiple generation-local executor maps, but each generation shall rebuild its own normal projections and lifecycle structures. +5.9. On a reuse hit, candidate preparation shall not invoke `Configure`, `Start`, `Stop`, `Close`, mutating preflight, or another generation-local mutation on the shared physical connector. Candidate rejection/rollback shall never invalidate the shared resource merely because the candidate was rejected. +5.10. Query-shaped operations already represented by the backend-plugin contract, such as `Resolve` and `ListModels`, may remain part of generation-local preparation/refresh subject to Requirement 8 concurrency gates. +5.11. If future generation preparation or an external adapter requires a mutating lifecycle action against the configured connector, that resource path shall become non-shareable until a separate design proves safe reuse. + +## Requirement 6: Invalidation, Detached Entries, and Physical Incarnations + +6.1. Semantic resource identity and physical resource incarnation shall be distinct concepts: the same semantic identity may later require a fresh physical incarnation after failure. +6.2. When the existing connector/process invalidation path declares a physical incarnation unusable, that exact entry shall become non-acquirable for future candidates before or atomically with delegating to processhost invalidation. +6.3. Invalidation shall remove only the exact failed incarnation from the `current` semantic index. A stale callback from an older incarnation shall not detach a newer current incarnation. +6.4. Detached/invalidated entries may remain referenced by generations that already leased them and shall remain tracked by the process reconciliation owner until their physical cleanup has completed. +6.5. The pool shall maintain an ownership set or equivalent enumeration of **all successfully constructed but not-yet-physically-cleaned entries**, including detached entries, so terminal process shutdown can fail-safe clean them. +6.6. A future acquisition for the same semantic identity after invalidation shall build a fresh incarnation rather than returning the detached entry. +6.7. Invalidation shall not live-swap a replacement into existing generations or decrement their lease claims merely because the physical incarnation failed. +6.8. Existing `processhost` generation invalidation/reap behavior remains authoritative for the physical process. Reconciliation controls only future reuse eligibility, dependent retention, and the timing of the per-resource cleanup capability. + +## Requirement 7: Exactly-Once Physical Cleanup and Shutdown Ownership + +7.1. There shall be exactly one logical physical cleanup capability for each pooled configured connector resource. It shall be an idempotent composite of the existing adapter/session cleanup and `ActivateResult.Cleanup`/`processhost.CloseInstance` path. +7.2. The reconciliation entry—not each lease—shall own the physical-cleanup once-state and stored cleanup result. Final lease release, pool shutdown, and invalidation-related terminal cleanup shall all converge on that same entry-level exactly-once operation. +7.3. `processhost.Host` does **not** transfer supervisory ownership of processes/slots/instances to the pool. It retains launch, instance tables, invalidation, reaping, and `Host.Close` fail-safe authority; the pool owns only the decision of when to invoke the existing per-resource composite cleanup while dependents still exist. +7.4. The reconciliation owner shall close before `processhost.Host` during normal process teardown so residual per-resource cleanup can still call the live host/session ownership paths. +7.5. Normal successful shutdown ordering shall remain: generation admission stops and generations drain/release leases; pool close linearizes/cancels and joins any builders; pool fail-safe cleans any residual current **and detached** entries; `processhost.Host` closes; verified artifact handles close; staging removal runs. +7.6. Bootstrap/error cleanup before process ownership transfer shall preserve the same relative pool -> host -> artifacts -> staging order for resources already acquired. +7.7. Pool shutdown shall be idempotent. `Close` racing a final lease release shall still execute physical cleanup exactly once; a later lease release after fail-safe shutdown cleanup shall be harmless. +7.8. Existing cleanup-error normalization/error-join behavior shall be preserved. Final normal lease-triggered cleanup errors surface through existing generation rollback/close aggregation; process-shutdown residual cleanup errors join the existing process close aggregation. +7.9. Partial construction failure shall not leak sessions, host instances, processes, IPC connections, pool entries, reserved claims, or builder goroutines. +7.10. `ProcessServices` shall own reconciliation shutdown through its existing private process-resource ownership mechanism; no second closer stack or process shutdown coordinator shall be introduced. + +## Requirement 8: Connector Concurrency and Non-Interference + +8.1. Reuse is valid only when the complete construction/configure identity and established host-session behavior make one configured resource safe to retain across overlapping generations; otherwise use isolated construction. +8.2. The standard production `backendplugin/host.Session` concurrency behavior shall remain unchanged. In particular, its existing `lifecycleMu` serialization of `Execute` versus `Execute`/`Close` shall not be removed or bypassed by this specification. +8.3. Sharing one standard Session therefore extends the existing per-session Execute serialization across overlapping generations. The implementation shall explicitly characterize a retained old-generation Execute overlapping a new-generation Execute and shall not claim preservation of the incidental extra execution parallelism provided today by two separately constructed sessions. +8.4. Query/auxiliary RPCs that the current standard host already permits to overlap—`Resolve`, `ListModels`, optional `CountTokens`, and optional `FinalizeBilling`—shall receive race/conformance coverage when invoked through one pooled Session across overlapping generations and alongside execution. +8.5. A non-standard/injected connector session path that cannot satisfy the established standard-host operation-concurrency contract shall be non-shareable rather than gaining a new public concurrency flag in this specification. +8.6. Dynamic provider facts modeled through runtime calls continue through those calls and do not require connector reconstruction on unrelated generation changes. +8.7. The optimization shall not alter canonical requests/events, route selection, retries, failover, output-commit rules, stream ordering within an attempt, cancellation, billing finalization semantics, accounting evidence, token counting, or provider-specific translation semantics. +8.8. No resource-pool lookup or lock shall be added to normal request execution; acquisition/release occurs at generation construction/retirement boundaries. +8.9. Existing backend-plugin security invariants—verified artifact binding, secure local IPC, peer authentication before Configure/secrets, environment restrictions, and process-tree cleanup—shall remain unchanged. +8.10. Connector-specific configuration parsing remains inside the connector. The host may fingerprint opaque Configure bytes but shall not learn provider-specific schemas merely to decide reuse. +8.11. Functional/canonical behavior for unchanged resources shall remain equivalent to the current overlap model except for explicitly documented physical-resource reuse and the resulting extension of the existing per-session Execute serialization across retained generations. + +## Requirement 9: TDD, Concurrency, and Architecture Gates + +9.1. Add RED tests for high-cardinality construction counts, identity discrimination, waiter reservation, lease lifetime, detached-entry ownership, rollback, invalidation/incarnation behavior, Acquire/Close linearization, builder cancellation, exactly-once cleanup, operation concurrency, and shutdown ordering before enabling reuse. +9.2. Include scheduling-sensitive tests for: first claimant release before a waiter wakes; waiter cancellation; final release racing new Acquire; final release racing Pool.Close; invalidation followed by Pool.Close with an outstanding old-generation lease; and stale invalidation racing replacement publication. +9.3. Include a blocked physical builder that exits only on its pool-owned context cancellation; Pool.Close must cancel it, wait for it, clean any partial/success result, and prevent late publication. +9.4. Split identity evidence into two scopes: the high-cardinality generation-reload matrix covers inputs that actually vary through current reload plus invalidation; a focused physical identity/construction matrix independently proves artifact digest, secret fingerprint, process model, and normalized runtime-policy differences cannot hit the same pooled entry. `shared_artifact` remains a non-pooled fallback, not a pooled replacement case. +9.5. Add a DTO/input drift gate forcing deliberate identity review when the external configure-time physical input surface changes. +9.6. Add race/goleak coverage for the private pool/builders and overlapping standard-host metadata/auxiliary/execution operations across retained generations. +9.7. Add an exactly-once ownership regression that counts adapter/session cleanup, `ActivateResult.Cleanup`/host instance cleanup, pool fail-safe cleanup, and later `Host.Close`, proving the same pooled physical resource cannot be torn down twice through competing ownership paths. +9.8. Architecture tests shall reject a generic service registry/container API, request-time pool lookup, public reusable-resource framework, provider-specific switch, or migration of `processhost` supervision responsibility. +9.9. Repository quality, focused executable-plugin security/conformance tests, existing backend recomposition/overlap tests, ResourceLedger lifecycle tests, and reload no-drop/last-good tests shall remain green. +9.10. Final implementation review shall remove unused abstraction layers and preserve the smallest private design that satisfies deterministic scale and correctness gates; pooled reuse shall be re-scoped if the established Session concurrency model makes the measured overlap behavior operationally unacceptable. diff --git a/.kiro/specs/backend-connector-resource-reconciliation/research.md b/.kiro/specs/backend-connector-resource-reconciliation/research.md new file mode 100644 index 00000000..48e4c860 --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/research.md @@ -0,0 +1,253 @@ +# Research & Design Decisions + +## Summary + +- **Feature**: `backend-connector-resource-reconciliation` +- **Discovery Scope**: brownfield runtime optimization / connector-scale lifecycle refactor +- **Selected Cordis principle**: semantic provider identity + physical incarnation + dependent retention/reconciliation. +- **Not selected**: generic Cordis component runtime, reactive dependency graph, fibers, service locator, HMR, or generic effect/resource framework. +- **Primary target**: unchanged discovered executable `per_instance` connector resources reconstructed across overlapping immutable generations. +- **Post-review hardening**: detached-entry ownership, reserved waiter claims, entry-level exactly-once physical cleanup, pool-owned builder cancellation, explicit Acquire/Close linearization, exact cleanup handoff to processhost, identity-evidence split, and established host-session concurrency characterization. + +## Brownfield Findings + +### Generation correctness is already solved at the right coarse boundary + +Go-LIP already provides immutable `GenerationRuntime`, request/async generation retention, transactional last-good publication, `ResourceLedger` rollback/retirement, and manager-owned drain. Those mechanisms remain authoritative. A Cordis-like dynamic component graph would duplicate existing correctness machinery. + +### The scale pressure is enabled live connector count, not manifest count + +Discovery is manifest/trust oriented and does not launch every installed connector. The expensive case is many **enabled configured external connector instances** rebuilt during a material generation change. `buildBackends` constructs every enabled backend row for every candidate generation. + +### `per_instance` discovered connectors deliberately duplicate across generations today + +`buildDiscoveredBackend` mints unique host activation IDs for `per_instance` so active and candidate generations can safely coexist. This is correct for changed connectors but also means an unchanged connector is physically Activated/Configured again during unrelated material reload. + +### Reconciliation belongs above processhost + +`processhost.Host` already owns lazy process launch, secure local IPC, peer authentication, slot/instance bookkeeping, process-generation invalidation, and process-tree cleanup. It should not learn Go-LIP semantic generation/config identity. The new owner is therefore a private runtimebundle connector-resource reconciliation layer that calls the existing physical builder/host. + +### Prior Cordis-derived ownership work remains valid + +The archived `atomic-owned-resource-lifecycle` spec hardened process acquisition/cleanup locality and generation loop ownership but deliberately left backend lifecycle alone because `BackendBuildResult` already paired backend + cleanup and `buildBackends` transferred cleanup to `ResourceLedger`. This spec addresses a different problem: **one expensive physical backend resource may be retained by multiple overlapping generations**. It reuses rather than replaces those ownership authorities. + +## Physical Identity Research + +### Existing `BackendStateIdentity` is precedent, not the key + +Go-LIP already reuses process-owned affinity/health observation state across compatible generations using `{InstanceID, FactoryKind, ConfigDigest}`. That demonstrates identity-sensitive continuity is locally idiomatic, but physical connector interchangeability is stricter. + +### Physical construction inputs + +Current discovered construction uses/captures: + +- logical `InstanceID`; +- factory kind; +- exact verified artifact digest; +- process model/sharing profile; +- opaque YAML Configure bytes; +- normalized `RuntimePolicy`; +- `SecretBundle` when supplied; +- negotiation/session behavior from the fixed executable/host protocol. + +Therefore a separate private physical identity must treat artifact, process model, config bytes, policy, secrets, factory and logical instance deliberately. Secret values are locally hashed with deterministic length framing and never surfaced. + +### Startup-fixed versus reload-varying facts + +The production discovered factory closure is installed at startup. Artifact/process model and `DiscoveredInstallOptions.RuntimePolicy` are captured there. Current SIGHUP reload does not rediscover/replace the artifact. The current discovered builder also does not inject changing secret material through the shown production path. + +Consequences: + +- these fields still belong in the **identity contract** because they define physical semantics and may evolve later; +- high-cardinality **generation reload** evidence should not pretend artifact/process-model/policy changes are currently hot-reloadable; +- focused identity/construction tests exercise those dimensions directly; +- `shared_artifact` remains a non-pooled/restart-required fallback rather than a “changed pooled process model” case. + +This resolves CodeRabbit's request for broader identity coverage without inventing unsupported reload capabilities. + +## Physical Cleanup and processhost Ownership + +### Current composite cleanup + +The current discovered builder combines: + +1. adapter `BuildResult.Cleanup()` -> `session.Close(...)`, which closes the configured connector instance/host session; +2. `ActivateResult.Cleanup` -> `processhost.Host.CloseInstance(hostActivationID)`, which updates host instance ownership and reaps the slot/process when appropriate. + +The adapter BuildResult itself is once-guarded, and processhost reaping is idempotent, but pooling adds another possible caller (`Pool.Close`). A **lease-level** once guard alone is therefore insufficient. + +### Selected ownership wording + +`processhost.Host` keeps supervisory ownership: process/slot tables, peer identity, invalidation, reaping, and `Host.Close` fail-safe behavior stay there. + +The pool entry consumes the existing **per-resource composite cleanup capability** when a physical build succeeds. Generations never receive that composite; they receive only lease release. The pool therefore controls *when* the per-resource cleanup may be invoked, while processhost remains the component that actually supervises/reaps physical processes. + +An entry-level cleanup-once state is shared by normal final lease release and process shutdown. This avoids a final-release-versus-Pool.Close double cleanup race. + +## Detached Resource Ownership + +Initial design only indexed `current[semanticIdentity]`. That is insufficient after invalidation: + +```text +identity X -> incarnation 7 current +invalidate 7 +current[X] removed +Gen17 still has lease to incarnation 7 +``` + +If ProcessServices then closes before that lease is released, a Pool.Close that only walks `current` cannot enumerate incarnation 7. + +Selected correction: retain an `owned`/all-entry set containing every successfully constructed physical incarnation until entry-level physical cleanup completes. Invalidation only detaches from `current`; it does not surrender process-level ownership bookkeeping. Pool Close snapshots all residual owned entries, including detached/invalidated ones, and invokes the same once-guarded physical cleanup. + +## Acquire/Waiter Concurrency + +### Why post-publication waiter ref increments are unsafe + +The original pending-entry idea let waiters observe `building`, wait on readiness, and increment refs only after publication. A scheduler can then do: + +1. first caller builds, receives lease; +2. waiter is still sleeping/not yet incremented; +3. first caller releases, refs reach zero, physical resource closes; +4. waiter wakes and attempts to acquire the now-closed entry. + +Selected correction: **every waiter reserves its prospective claim under the pool mutex before waiting**. The initiating caller is also just a claimant. A waiter cancellation abandons only its own reservation. + +### Builder ownership + +The initiating Acquire must not own the physical builder lifetime. Otherwise caller cancellation and contextless `ProcessServices.Close()` can leave an unbounded build blocking shutdown. + +Selected correction: + +- absent Acquire creates a pending entry and reserves its claim; +- the pool starts one short-lived builder goroutine under a pool-owned cancellation context; +- all Acquires wait on entry readiness or their own context; +- Pool.Close sets closing, cancels the build root, then waits builders; +- build completion after closing may clean a result but cannot publish it as reusable. + +This uses existing processhost/transport context cancellation rather than adding a background worker subsystem. + +## Acquire/Close Linearization + +`ProcessServices.Close` is contextless and synchronous, so the pool needs a precise terminal contract. + +Selected contract: + +- `Close` linearizes by setting `closing=true` under the same mutex used for Acquire claim reservation; +- after that point no new claim reservation is accepted and no pending builder result is handed off as a post-close success; +- Close cancels pool-owned builders; +- Close waits builders and acquisition handoff activity; +- only then does it snapshot/clean residual owned entries; +- no physical build/cleanup/wait happens while the mutex is held. + +This is stronger than simply “reject new Acquire” and directly addresses the lease-after-cleanup race CodeRabbit identified. + +## Invalidation and Incarnations + +Semantic identity says the desired configuration is unchanged. Physical incarnation says which concrete process/session currently realizes it. + +Invalidation: + +- compares exact entry/incarnation; +- detaches that entry from `current` before/with host invalidation; +- leaves the detached entry in process ownership until cleanup; +- does not decrement existing generation claims or live-swap a replacement; +- allows a future candidate to build a fresh incarnation for the same semantic key; +- stale old-incarnation callbacks cannot remove the new current entry. + +## Candidate Isolation + +Fresh physical processes currently give candidate preparation strong failure-domain isolation. Reuse is allowed only for query-shaped candidate preparation: + +- no Configure/Start/Stop/Close/mutating preflight on a reuse hit; +- rollback releases only the candidate lease; +- candidate failure alone does not invalidate the resource; +- future mutating preparation makes the path non-shareable until separately designed. + +Generation-local model registry/catalog/routing/policy/billing state remains generation-owned. + +## Established Operation Concurrency + +### Standard host Session behavior + +Cross-checking `pkg/lipsdk/backendplugin/host/session.go` shows: + +- `Session.Execute` holds `lifecycleMu` across the complete Execute RPC, so two Execute calls on one Session are serialized and Close cannot race underneath an active Execute; +- `Resolve`, `ListModels`, CountTokens, and FinalizeBilling are not serialized by that client lifecycle mutex; +- server-side instance leasing already allows metadata/auxiliary calls to overlap an active configured instance operation while protecting Close. + +Therefore pooling one production Session across generations does **not** justify inventing a new concurrency contract. The implementation must preserve the existing standard-host behavior and test it. + +### Important overlap tradeoff + +Today Gen17 and Gen18 fresh physical Sessions create two independent Execute serialization domains during candidate/retirement overlap. Pooling an unchanged connector makes retained-old and new-generation Execute calls share one Session serialization domain. + +That can reduce transient overlap concurrency / create head-of-line waiting for long streams. It is not a canonical request mutation, but it is operationally material and must not be hidden behind an unconditional “observationally equivalent” claim. + +Selected treatment: + +- do not change `Session.Execute` concurrency in this spec; +- characterize a long retained old-generation Execute overlapping a new-generation Execute; +- cover metadata/Count/Finalize/execution overlap under race/conformance tests; +- if established serialization makes the optimization unacceptable for target workloads, re-scope pooling rather than widening this spec into ABI/session-concurrency redesign; +- injected/non-standard session paths are non-shareable unless they satisfy the same established host contract. + +## Scale/ROI Evidence Strategy + +Primary correctness/ROI is deterministic work count: + +```text +before: O(N enabled eligible connectors) physical build per material generation +after: O(K changed/unusable physical builds) + O(N) cheap claims/projections +``` + +Use at least 100 synthetic enabled discovered `per_instance` connectors. Count physical build, Activate/launch, Configure, physical cleanup, lease acquire/release. Wall time/allocation is supporting evidence only. + +Two matrices are required: + +1. **Reload matrix**: unchanged, config changes, remove/disable, candidate rollback, invalidation/rebuild. +2. **Physical identity matrix**: artifact, secret, normalized runtime policy, process model, factory/logical identity. + +Also record the cross-generation Session Execute serialization behavior so lifecycle savings are evaluated together with the actual overlap scheduling tradeoff. + +## Architecture Pattern Evaluation + +| Option | Decision | Reason | +|---|---|---| +| Full Cordis runtime | Reject | duplicates generation/runtime ownership machinery | +| Put semantic reconciliation in processhost | Reject | conflates physical supervision with LIP config semantics | +| Reuse `BackendStateIdentity` | Reject | identity incomplete for physical reuse | +| Return same physical cleanup to generations | Reject | early/double teardown | +| Only track current entries | Reject | detached invalidated resources disappear from shutdown ownership | +| Waiter refs after readiness | Reject | zero-ref handoff race | +| Caller-owned physical build | Reject | shutdown cancellation/lifetime ambiguity | +| Pool-owned connector entries + generation leases | **Select** | focused ROI; preserves existing authorities | +| Idle TTL cache | Reject | speculative retention/eviction policy | +| Change Session concurrency here | Reject | broadens scope; preserve/measure existing behavior | +| No change for non-shareable paths | Keep | safest fallback | + +## Risks & Mitigations + +- **Identity omission** -> one choke point, fail-closed fallback, DTO/input drift test. +- **Detached resource leak** -> process-owned all-entry set until cleanup completes. +- **Waiter zero-ref race** -> reserve claim before waiting. +- **Shutdown hangs on builder** -> pool-owned cancelable build context + joined builder tests. +- **Acquire after shutdown cleanup** -> explicit Acquire/Close linearization and handoff join. +- **Double cleanup** -> entry-level `cleanupOnce`, not only lease once. +- **Pool/processhost ownership confusion** -> pool owns timing of composite per-resource cleanup; host remains physical supervisor/fail-safe. +- **Stale invalidation** -> exact incarnation comparison. +- **Candidate mutates last-good** -> query-only reuse; mutating path non-shareable. +- **Cross-generation Execute head-of-line blocking** -> preserve standard Session semantics, characterize, re-scope if unacceptable. +- **Scope creep** -> package-private connector-specific API and architecture tests. +- **Speculative complexity** -> 100-connector count gate and final simplification/re-scope gate. + +## References + +- User-supplied paper: *A Programming Paradigm for Spatiotemporal Composability* — semantic provider identity, dependent retention, reconciliation, revertible cleanup concepts. +- `internal/infra/runtimebundle/discovered_factories.go` — current external connector construction and unique per-generation activation handles. +- `internal/infra/backendplugins/processhost/host.go` — process/instance supervision, invalidation and cleanup. +- `internal/infra/backendplugins/adapter/backend.go` and `processhost/build_result.go` — backend/session cleanup shape. +- `pkg/lipsdk/backendplugin/host/session.go` — standard host operation concurrency and Session lifecycle serialization. +- `pkg/lipsdk/backendplugin/server.go` — configured-instance leasing around RPC calls and Close. +- `internal/infra/runtimebundle/backend_state_identity.go` — existing narrower semantic identity precedent. +- `internal/infra/runtimebundle/process_services.go` / `resource_ledger.go` — process/generation cleanup authorities. +- archived `atomic-owned-resource-lifecycle` and runtime convergence specs — explicit ownership/no-container architecture constraints. diff --git a/.kiro/specs/backend-connector-resource-reconciliation/spec.json b/.kiro/specs/backend-connector-resource-reconciliation/spec.json new file mode 100644 index 00000000..0bb0cba4 --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/spec.json @@ -0,0 +1,23 @@ +{ + "feature_name": "backend-connector-resource-reconciliation", + "created_at": "2026-08-17T00:42:00+02:00", + "updated_at": "2026-08-17T11:06:00+02:00", + "language": "en", + "phase": "tasks-generated", + "approvals": { + "requirements": { + "generated": true, + "approved": false + }, + "design": { + "generated": true, + "approved": false + }, + "tasks": { + "generated": true, + "approved": false + } + }, + "ready_for_implementation": false, + "project_description": "Reduce cross-generation churn for expensive executable backend connectors by reusing unchanged configured per-instance connector resources through complete semantic physical identity, incarnation-safe invalidation, pre-reserved generation lease claims, process-owned tracking of detached resources, pool-owned cancelable builders, explicit Acquire/Close linearization, and entry-level exactly-once physical cleanup. Preserve immutable GenerationRuntime publication, ResourceLedger rollback/retirement, processhost supervision, the executable backend-plugin ABI, startup-fixed discovery, routing/streaming/accounting semantics, candidate last-good isolation, and the established backendplugin/host.Session operation-concurrency contract. Scope the first implementation to eligible discovered per_instance connectors only; candidate reuse is query-only, mutating preparation or non-conforming session behavior falls back to isolated construction, and generation-local model/routing/policy views remain generation-owned. Add no DI container, service locator, reactive component graph, live plugin watcher, generic resource framework, public lifecycle/concurrency knob, or request-hot-path lookup. Require deterministic high-cardinality operation-count evidence plus explicit cross-generation Session scheduling evidence so the optimization remains justified by connector-scale behavior rather than Cordis-inspired abstraction for its own sake." +} diff --git a/.kiro/specs/backend-connector-resource-reconciliation/tasks.md b/.kiro/specs/backend-connector-resource-reconciliation/tasks.md new file mode 100644 index 00000000..d5b73f0f --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/tasks.md @@ -0,0 +1,251 @@ +# Implementation Plan + +## Execution Rules + +- Follow TDD: characterization/RED tests and contract gates precede production reuse. +- Keep every task independently reviewable with no more than five concrete actions. +- Preserve public APIs, backend-plugin ABI, configuration schema, immutable generations, `ResourceLedger`, `ProcessServices`, and `processhost.Host` supervision. +- Prefer replacing the touched per-generation physical-cleanup path with lease cleanup rather than layering a competing ownership path. +- Do not broaden scope to builtins, `shared_artifact`, generic resource management, dynamic discovery, Session concurrency redesign, or request-time lookup. + +## Phase 1 — Freeze Scale, Identity, Lifetime, and Concurrency Contracts + +### Task 1.1 — Build the high-cardinality reload characterization harness + +- Add a deterministic runtimebundle fixture with at least 100 enabled synthetic discovered `per_instance` connector instances and no external credentials. +- Count physical build/factory invocation, processhost Activate/launch, Configure, physical cleanup, and later lease acquire/release operations. +- Characterize the current unrelated-material-reload baseline while the old generation remains retained. +- Add RED target assertions for unchanged reload (`0` new physical builds/activations/Configure) and config-changed reload (`K` changed identities -> `K` physical replacements). +- Add a supporting candidate-compilation benchmark without wall-clock correctness thresholds. + +_Requirements: 1.1–1.7, 2.2, 9.1, 9.4_ + +_Validation: deterministic current O(N) reconstruction is recorded and target count assertions are RED._ + +### Task 1.2 — Lock complete physical identity and drift behavior with RED tests + +- Add table-driven identity tests for logical instance/factory, artifact digest, process model, opaque Configure bytes, normalized RuntimePolicy, and secret fingerprint. +- Prove config/artifact/secret/policy differences cannot alias; prove `shared_artifact` is non-pooled fallback rather than a pooled replacement case. +- Prove secret plaintext never appears in identity/debug/error/status output and `BackendStateIdentity` alone cannot authorize physical reuse. +- Add a Configure/physical-input drift gate forcing intentional identity treatment when DTO/input shape changes. +- Keep reload-varying and startup-fixed evidence distinct so tests do not imply unsupported hot artifact/policy/process-model reload. + +_Requirements: 3.1–3.12, 8.1, 8.10, 9.4–9.5_ + +_Validation: focused identity/construction tests are RED and fail closed on omitted input treatment._ + +### Task 1.3 — Freeze reserved-claim and entry-ownership state machine with RED tests + +- Add first/live/concurrent Acquire tests where every building-entry waiter reserves its prospective claim before waiting; include a deterministic first-release-before-waiter-wake schedule. +- Add cancellation/failure tests proving waiter cancellation drops only its claim, failed builds are not negative-cached, and a later Acquire can retry. +- Add invalidation tests proving exact-incarnation detach, detached entry retention in process ownership, fresh same-key replacement, and stale invalidation safety. +- Add final Release versus new Acquire and final Release versus Pool.Close races, requiring one entry-level physical cleanup outcome. +- Add an invalidation + outstanding old-generation lease + Pool.Close test proving detached residual entries are enumerated and fail-safe cleaned. + +_Requirements: 4.1–4.4, 4.8–4.11, 6.1–6.8, 7.1–7.3, 7.7, 9.1–9.2, 9.7_ + +_Validation: tests are RED against the absent pool and pin zero-ref handoff/detached/double-cleanup races._ + +### Task 1.4 — Freeze Acquire/Close linearization and builder lifetime with RED tests + +- Add a terminal Close linearization test proving no new claim can be reserved and no pending build result can be handed off after `closing=true` linearizes. +- Add a physical builder that blocks until its **pool-owned** context is canceled; Pool.Close must cancel it, join it, and prevent late publication before returning. +- Add concurrent Close/Acquire/build-completion schedules proving residual cleanup starts only after builders and acquisition handoffs terminate. +- Prove a successful physical result arriving after Close begins is cleaned exactly once rather than published as reusable. +- Add goleak/race coverage for canceled waiters, pending builders, Close, and late completion. + +_Requirements: 4.5–4.7, 7.5, 7.7, 7.9, 9.1–9.3, 9.6_ + +_Validation: shutdown-race tests are RED and define the pool's terminal linearization contract._ + +### Task 1.5 — Freeze cleanup handoff, candidate isolation, and standard Session concurrency + +- Add an ownership test counting adapter/session cleanup, `ActivateResult.Cleanup`/host instance cleanup, pool cleanup, and later `Host.Close`, proving one physical resource is not torn down twice. +- Add candidate rollback with active+candidate sharing: rollback releases only the candidate claim and leaves active execution/query behavior available. +- Add a characterization gate proving pooled external backend values expose no generation-owned physical Close/Start/Stop bypass and reuse-hit preparation is query-only. +- Add standard-host overlapping-generation race/conformance for Resolve/ListModels/CountTokens/FinalizeBilling alongside execution. +- Add a retained-old-generation Execute + new-generation Execute test that explicitly observes existing `Session.Execute` serialization without changing it, plus process teardown order pool -> host -> artifacts -> staging. + +_Requirements: 5.4–5.11, 7.1–7.10, 8.2–8.5, 8.11, 9.6–9.7, 9.10_ + +_Validation: last-good isolation, cleanup ownership, established Session concurrency, and teardown-order assertions are RED._ + +## Phase 2 — Implement the Minimal Private Identity and Reconciliation Owner + +### Task 2.1 — Implement explicit fail-closed physical identity + +- Add one package-private identity builder at the discovered physical construction/configure choke point using domain-separated, length-framed SHA-256 inputs. +- Explicitly project every RuntimePolicy field and deterministic opaque Configure bytes; hash sorted length-framed secret names/values without retaining plaintext. +- Include logical instance/factory, exact artifact digest and process model; document startup-fixed inputs while still treating them in focused identity tests. +- Return an explicit shareability decision so incomplete/unsupported input uses current isolated construction. +- Make Task 1.2 identity/privacy/drift tests green without production reflection or a public identity type. + +_Requirements: 3.1–3.12, 8.1, 8.10_ + +_Validation: focused identity tests green and no secret/raw identity leakage exists._ + +### Task 2.2 — Implement reserved-claim entries and exactly-once physical cleanup + +- Add a package-private entry model with building/live/detached/failed state, exact incarnation token, pre-reserved claims, readiness signaling, and current semantic indexing. +- Add a process-owned set of every successful physical entry until its entry-level cleanup-once completes, including invalidated/detached entries. +- Make every generation lease release idempotently drop one claim; only final normal release detaches current and invokes the entry-level cleanup-once operation. +- Store physical cleanup only on the entry and use the same cleanup-once path for final release and process fail-safe shutdown. +- Make Task 1.3 tests green without performing physical cleanup while holding the pool mutex. + +_Requirements: 4.1–4.4, 4.8–4.11, 6.1–6.8, 7.1–7.3, 7.7_ + +_Validation: reserved-claim, detached ownership, invalidation and cleanup-race tests green under `-race`._ + +### Task 2.3 — Implement pool-owned physical builders and terminal Close + +- Create one pool-owned cancellable build root; absent identity starts exactly one joined builder goroutine and all callers wait as claimants rather than owning the build. +- Pass pool builder context through pooled `processhost.Activate`/Configure instead of a background lifetime; caller cancellation abandons only that caller's reservation. +- Implement Close linearization under the pool mutex, reject later Acquires, cancel builders, then wait builders and acquisition handoffs before residual cleanup. +- Prevent a build completion after Close from publishing/handoff; clean any completed physical result through the entry cleanup-once path. +- Make Task 1.4 blocked-builder/linearization/goleak/race tests green. + +_Requirements: 4.5–4.8, 7.5, 7.7, 7.9_ + +_Validation: no builder can outlive Pool.Close or publish after its terminal boundary._ + +### Task 2.4 — Transfer pool ownership through existing process construction + +- Create the pool beside `processhost.Host` during discovered-install preparation and capture it lexically in eligible discovered factory closures. +- Extend the private install/process-build ownership bundle so pool lifetime transfers into `ProcessServices` without global/setter lookup. +- Register cleanup in existing process ownership so reverse shutdown is pool -> host -> verified artifacts -> staging while preserving unrelated ProcessServices ordering. +- Mirror the same relative cleanup order on every pre-transfer/bootstrap failure and prevent double cleanup after ownership transfer. +- Keep pool API connector-specific/package-private and make Task 1.5 teardown/ownership tests green. + +_Requirements: 2.1, 2.6–2.9, 7.3–7.10_ + +_Validation: success and partial-startup ownership transfer is exactly once and dependency ordered._ + +## Phase 3 — Integrate Reuse at the Discovered `per_instance` Factory Seam + +### Task 3.1 — Split preparation, physical construction, and lease acquisition + +- Refactor discovered backend construction into effective input/identity preparation, the current physical Activate/Configure/adapter build, and pool Acquire without provider-specific branches. +- Preserve unique host activation IDs for every **new** per-instance physical incarnation and leave processhost ownership keys unchanged. +- Route only eligible overlap-safe discovered `per_instance` resources through the pool; builtins, `shared_artifact`, and non-shareable resources keep current construction. +- On a pool miss, consume the current composite adapter/session + activation cleanup into the entry; on a hit, return existing backend functions plus a fresh lease release. +- Keep `buildBackends`/`ResourceLedger` as the generation cleanup transfer authority and make unchanged/config-mixed count tests green. + +_Requirements: 1.4–1.5, 2.2–2.7, 4.2, 4.9–4.11, 7.1–7.3_ + +_Validation: unchanged reload performs zero physical reconstruction; K changed configs produce K physical builds._ + +### Task 3.2 — Bind invalidation to exact pooled incarnation + +- Wrap newly built pooled adapter invalidation so it detaches only the exact pool entry/incarnation before/atomically with delegating to `processhost.InvalidateProcessGeneration`. +- Preserve processhost physical reap/recovery and cleanup normalization; invalidation does not decrement generation claims or live-substitute a replacement. +- Keep detached invalidated entry in the pool's process ownership set until entry cleanup completes. +- Prove later same-config Acquire builds one fresh incarnation and old delayed callbacks cannot detach it. +- Keep non-pooled and `shared_artifact` invalidation behavior unchanged. + +_Requirements: 6.1–6.8, 7.3, 8.7, 8.9_ + +_Validation: invalidation/replacement tests green and existing processhost invalidation suites remain green._ + +### Task 3.3 — Preserve generation-local state and query-only candidate behavior + +- Continue building new generation-local inventories, model registry/catalog, executor/routing/policy/billing views, handler, lifecycle context, and `ResourceLedger` from leased backends. +- Ensure a reuse hit performs no Configure/Start/Stop/Close/mutating preflight and candidate rollback never invalidates because the candidate failed. +- Verify changed same-ID config, remove/disable, retained old stream/async work, and candidate failure retain current semantics. +- Verify standard-host metadata/auxiliary overlap across generations under race/conformance while preserving existing Session lifecycle locking. +- Keep canonical request/event, routing, failover, streaming, cancellation, accounting, billing and token-counting code paths unchanged. + +_Requirements: 5.1–5.11, 8.4, 8.6–8.11_ + +_Validation: backend recomposition/no-drop suites plus query-only/cross-generation tests green._ + +### Task 3.4 — Add architecture fences against scope creep and cleanup bypass + +- Prove the pool remains private to runtime composition and is absent from request execution, public SDKs, provider-specific packages, and connector authoring APIs. +- Reject generic service/container/keyed runtime registry APIs introduced for this feature. +- Lock `processhost.Host` as the only process/IPC supervisor; the pool may call existing cleanup/invalidation seams but not duplicate supervision logic. +- Lock generation cleanup to lease release for pooled resources and reject alternate physical lifecycle hooks bypassing entry ownership. +- Assert no public YAML/manifest/ABI/concurrency option was added for reconciliation. + +_Requirements: 2.1, 2.5–2.9, 4.10–4.11, 7.3, 8.2, 8.8–8.10, 9.8_ + +_Validation: representative forbidden architecture fixtures fail and intended private design passes._ + +## Phase 4 — Certify ROI, Identity, Concurrency, and Simplicity + +### Task 4.1 — Certify the high-cardinality generation-reload matrix + +- Run the 100-enabled-connector fixture for unchanged, one/K config changes, remove/disable, candidate rollback, and invalidation-then-rebuild. +- Assert physical build/Activate/Configure counts are `0` for unchanged reuse and proportional only to changed/unusable identities. +- Assert physical live-resource overlap avoids duplicating unchanged connectors and candidate rollback of reuse hits performs no physical cleanup of active resources. +- Assert candidate-only new resources clean on rollback/final release and invalidated entries can be fail-safe cleaned at process shutdown. +- Record deterministic before/after counts as implementation/PR evidence. + +_Requirements: 1.1–1.7, 4.9, 5.2–5.5, 6.4–6.6, 7.5_ + +_Validation: structural O(N) physical build -> O(K) physical build claim is green without timing thresholds._ + +### Task 4.2 — Certify the focused physical identity/construction matrix + +- Exercise distinct artifact digests, secret fingerprints, normalized RuntimePolicy values, factory/logical IDs, and process models directly at the physical identity/construction seam. +- Prove each eligible identity difference misses the existing pool entry and creates a fresh resource when construction is otherwise shareable. +- Prove `shared_artifact`/other non-eligible process model uses existing non-pooled/restart-required behavior rather than a pooled replacement. +- Verify no startup-fixed field is falsely documented/tested as current SIGHUP hot-reload support. +- Re-run the DTO/input drift gate against the final production identity projection. + +_Requirements: 3.1–3.12, 9.4–9.5_ + +_Validation: all physical identity dimensions are covered without inventing unsupported reload behavior._ + +### Task 4.3 — Run race, leak, security, conformance, and reload regression gates + +- Run targeted `-race`/goleak for reserved claims, Acquire/Close/build cancellation, invalidation, entry cleanup, and overlapping standard-host operations. +- Run processhost activation/cleanup/invalidation and executable backend-plugin security/conformance suites. +- Run ResourceLedger, backend recomposition, discovered overlap/restart-required, candidate rollback, retained-generation, and reload last-good/no-drop suites. +- Run repository formatting/vet/lint/architecture gates without weakening assertions/skips. +- Verify secret/config identity data and opaque YAML do not leak to logs, metrics, errors, statuses or public DTOs. + +_Requirements: 7.7–7.10, 8.4–8.10, 9.1–9.9_ + +_Validation: concurrency/security/reload/repository gates green._ + +### Task 4.4 — Record performance and Session overlap evidence + +- Run comparable high-cardinality candidate-build benchmarks and report timing/allocations separately from deterministic work counts. +- Deterministically hold a retained old-generation Execute and start a new-generation Execute on the same pooled standard Session; record the existing serialization behavior and cancellation/close outcome. +- Confirm normal request execution adds no pool lookup/lock and no material regression attributable to reconciliation. +- Document that the claimed gain is reload physical-resource churn/peak overlap reduction, not inference throughput or token latency. +- Re-scope pooled configured-session reuse if the established cross-generation Execute serialization is operationally unacceptable for intended long-lived-stream workloads; do not redesign Session concurrency in this spec. + +_Requirements: 1.6–1.7, 8.2–8.3, 8.8, 8.11, 9.10_ + +_Validation: ROI evidence includes both lifecycle savings and the real overlap-scheduling tradeoff._ + +### Task 4.5 — Perform final simplification and authority audit + +- Remove duplicate ownership stacks, generic wrappers, public knobs, request-path coupling, or unused lifecycle abstractions from the implementation diff. +- Confirm `ProcessServices`, `ResourceLedger`, `processhost.Host`, runtimehost generation refs, and unique activation IDs remain the same authorities. +- Confirm builtins, `shared_artifact`, dynamic discovery, shared model registry, and host Session concurrency redesign remain outside scope. +- Confirm every physical resource has one entry-level cleanup path and every generation owns only one lease release. +- If count/behavior gates do not justify the added lifecycle machinery, revert/re-scope rather than ship speculative architecture. + +_Requirements: 1.7, 2.1–2.9, 7.1–7.10, 9.8–9.10_ + +_Validation: final diff remains narrowly connector-lifecycle focused and evidence-backed._ + +## Requirement Coverage Matrix + +| Requirement | Primary tasks | +|---|---| +| R1 Evidence-first scale | 1.1, 3.1, 4.1, 4.4–4.5 | +| R2 Narrow boundary | 1.5, 2.4, 3.1, 3.4, 4.5 | +| R3 Physical identity | 1.2, 2.1, 4.2 | +| R4 Acquire/Close/claims | 1.3–1.4, 2.2–2.3, 3.1 | +| R5 Immutable generation/candidate | 1.5, 3.1, 3.3, 4.1 | +| R6 Invalidation/detached ownership | 1.3, 2.2, 3.2, 4.1 | +| R7 Cleanup/shutdown | 1.3–1.5, 2.2–2.4, 3.1, 4.3, 4.5 | +| R8 Concurrency/non-interference | 1.5, 3.3–3.4, 4.3–4.4 | +| R9 TDD/architecture | Phase 1, 3.4, Phase 4 | + +## Completion Gate + +Do not consider this specification implemented unless deterministic reconstruction counts meet the target **and** the hardened ownership/concurrency gates prove: reserved waiter claims, detached-entry shutdown ownership, terminal Acquire/Close linearization, pool-owned builder cancellation, entry-level exactly-once physical cleanup, candidate last-good isolation, and explicit preservation/measurement of standard Session operation concurrency.