From 1ee26b87c0abffddcb388819645bddff125cbebb Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:50:19 +0200 Subject: [PATCH 01/12] spec: initialize backend connector resource reconciliation --- .../spec.json | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 .kiro/specs/backend-connector-resource-reconciliation/spec.json diff --git a/.kiro/specs/backend-connector-resource-reconciliation/spec.json b/.kiro/specs/backend-connector-resource-reconciliation/spec.json new file mode 100644 index 00000000..913adc9a --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/spec.json @@ -0,0 +1,23 @@ +{ + "feature_name": "backend-connector-resource-reconciliation", + "created_at": "2026-08-17T00:42:00+02:00", + "updated_at": "2026-08-17T00:42:00+02:00", + "language": "en", + "phase": "initialized", + "approvals": { + "requirements": { + "generated": false, + "approved": false + }, + "design": { + "generated": false, + "approved": false + }, + "tasks": { + "generated": false, + "approved": false + } + }, + "ready_for_implementation": false, + "project_description": "Reduce cross-generation churn for expensive executable backend connectors by reusing unchanged configured per-instance connector resources through semantic identity and process-scoped reference-counted generation leases. Preserve immutable GenerationRuntime publication, ResourceLedger rollback/retirement, processhost supervision, the executable backend-plugin ABI, startup-fixed discovery, routing/streaming/accounting semantics, and last-good reload. Scope the first implementation to eligible discovered per_instance connectors only; add no DI container, service locator, reactive component graph, live plugin watcher, generic resource framework, or request-hot-path lookup. Require deterministic high-cardinality evidence before and after implementation so the optimization remains justified by connector-scale behavior rather than Cordis-inspired abstraction for its own sake." +} From 9cc7ff9550bc2585d7334d306af51c1c0b4fa309 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:51:03 +0200 Subject: [PATCH 02/12] spec: define backend connector reconciliation requirements --- .../requirements.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 .kiro/specs/backend-connector-resource-reconciliation/requirements.md diff --git a/.kiro/specs/backend-connector-resource-reconciliation/requirements.md b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md new file mode 100644 index 00000000..e391dd60 --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md @@ -0,0 +1,105 @@ +# Requirements Document + +## Introduction + +Go-LIP shall reduce unnecessary reconstruction of expensive executable backend connector resources across immutable runtime generations when a material configuration reload does not change those connectors. The optimization shall preserve the existing generation publication model: every request remains bound to one immutable `GenerationRuntime`; changed resources are constructed before publication; old generations and their resources remain valid until their existing work drains; candidate failure leaves the last-good generation untouched. + +This specification borrows only the Cordis-v4 ideas that fit this problem: semantic provider identity, reconciliation of unchanged desired resources, and lifetime retention while dependents still hold the provider. It does **not** introduce a Cordis component runtime, reactive dependency graph, fibers, dependency injection, service location, HMR, or generic effect system. + +The first implementation is intentionally narrow. It targets discovered executable backend connectors whose declared process model is `per_instance` and whose current reload policy permits candidate/active overlap. Installed-but-disabled connectors, statically linked backends, and `shared_artifact` connectors are not justification for this refactor and remain outside the first implementation. + +## Boundary Context + +- In scope: deterministic scale evidence, private physical connector-resource identity, process-lifetime reconciliation, per-generation leases, candidate rollback, invalidation/incarnation behavior, shutdown ordering, and focused executable-connector integration. +- Out of scope: general component reconciliation, frontend/feature reconciliation, dynamic plugin install/uninstall, discovery watchers, new backend-plugin ABI fields, new public configuration knobs, built-in backend pooling, `shared_artifact` behavior changes, request migration between generations, or live mutation of a published backend instance. +- Existing authorities remain: `ProcessServices` for process-owned resources, `ResourceLedger` for generation-owned cleanup, `processhost.Host` for executable process/IPC supervision, runtimehost generation leases for request/async lifetime, and the existing backend-plugin ABI for connector behavior. +- Performance intent: eliminate redundant connector construction work during material reloads. This is not a request-hot-path optimization. + +## Requirement 1: Evidence-First Scale Justification + +1.1. Before production reuse is enabled, add a deterministic high-cardinality characterization harness that compiles overlapping generations with at least 100 enabled synthetic host-backed `per_instance` connector instances. +1.2. The harness shall count at minimum physical connector builds, `processhost` activations or launches, configure operations, and physical cleanup operations; timing/allocation benchmarks may supplement but shall not replace these deterministic counters. +1.3. The baseline shall demonstrate the current reconstruction behavior for an unrelated material reload in which connector-defining inputs are unchanged. +1.4. After implementation, an unrelated material reload with `N` unchanged eligible live connectors shall perform **zero** new physical connector builds, activations/launches, and configure operations for those `N` connectors; it may perform `N` lightweight lease acquisitions. +1.5. For a candidate with `K` changed eligible connector identities and all remaining eligible connectors unchanged, the target is exactly the necessary replacement construction for the changed set; unchanged identities shall be leased rather than rebuilt. +1.6. Test acceptance shall not depend on fixed wall-clock thresholds that are vulnerable to CI host variance. Benchstat or equivalent measurements may be recorded as supporting evidence. +1.7. If the implementation cannot achieve the count-based reduction without changing request semantics or introducing a general runtime/container abstraction, the implementation shall be stopped or re-scoped rather than preserving the abstraction for speculative future value. + +## Requirement 2: Narrow Eligibility and Ownership Boundary + +2.1. Reconciliation shall be private to runtime composition and shall not become a public SDK capability or a request-time service locator. +2.2. The initial eligible set shall be discovered executable connectors using `ProcessModelPerInstance` and an overlap-safe reload policy. +2.3. Built-in/in-process backend factories shall retain current generation ownership unless a later evidence-backed specification separately justifies reuse. +2.4. `ProcessModelSharedArtifact` connectors shall retain their current explicit sharing/restart-required semantics; this specification shall not weaken their isolation or overlap gates. +2.5. Plugin discovery/trust shall remain startup-fixed. Connector installation, removal, directory rescanning, executable upgrade discovery, and automatic file watching are not added by this specification. +2.6. `processhost.Host` shall remain the sole executable process/IPC supervisor; the reconciliation layer shall not duplicate launch, peer authentication, process-tree cleanup, slot management, or transport supervision. +2.7. No new public YAML field, manifest field, CLI flag, environment variable, or backend-plugin ABI field shall be required merely to turn this optimization on. + +## Requirement 3: Semantic Physical Resource Identity + +3.1. Reuse shall require an exact private identity representing the configured physical connector resource, not Go object equality and not only the logical backend instance ID. +3.2. At minimum, the identity shall distinguish logical instance ID, factory kind, exact verified executable artifact digest, process model, opaque connector configuration content, effective configure-time runtime policy, and configure-time secret material by a non-reversible fingerprint. +3.3. Identity construction shall be deterministic for semantically identical effective inputs within one process. +3.4. Secret plaintext shall never be retained in the identity, logged, emitted in diagnostics, or exposed through public status. Secret fingerprinting shall be local and deterministic for equality purposes only. +3.5. Executable artifact replacement shall produce a distinct resource identity even when logical instance ID and YAML configuration are unchanged. +3.6. Credential rotation or another configure-affecting secret change shall produce a distinct resource identity. +3.7. Runtime-policy changes that affect the configured connector shall produce a distinct resource identity. +3.8. `BackendStateIdentity` may provide precedent or low-level hashing helpers, but its current `{InstanceID, FactoryKind, ConfigDigest}` contract shall not be treated as sufficient proof that two physical connector resources are interchangeable. + +## Requirement 4: Process-Scoped Acquire and Generation Lease Contract + +4.1. An eligible physical connector resource shall have one process-scoped lifetime owner and zero or more generation leases. +4.2. Acquiring an exact live identity that is already current shall reuse the existing immutable configured resource and increment its dependent lease count rather than invoking the connector factory/process activation path again. +4.3. Concurrent acquisitions for the same absent identity shall construct at most one physical current resource; successful contenders shall receive leases to that same resource. +4.4. A generation shall own only its lease release through its existing `ResourceLedger`; generation rollback/retirement shall not directly own or close a shared physical connector resource. +4.5. Releasing one of several leases shall not close the physical resource. Releasing the final lease shall detach the resource and invoke its physical cleanup exactly once. +4.6. There shall be no idle TTL/cache-retention policy in the first implementation. A valid current physical resource exists only while at least one generation lease retains it. +4.7. A failed physical build/configure shall not be cached as a permanent negative result; a later independent acquisition may retry through the existing construction path. +4.8. Lease release shall be idempotent and safe under candidate rollback, generation retirement, and process shutdown races. + +## Requirement 5: Preserve Immutable Generation Semantics + +5.1. Published `GenerationRuntime` objects remain immutable; no connector resource shall be reconfigured or replaced underneath a published generation. +5.2. If connector identity changes, the candidate shall construct a distinct replacement resource before publication while the old generation retains its old resource until its leases drain. +5.3. If a connector is removed or disabled, the candidate shall acquire no replacement lease; old generations may continue using the removed connector until their existing leases drain. +5.4. A failed candidate that leased an existing resource shall release only its candidate lease and shall not disturb the active generation's lease/resource. +5.5. A failed candidate that created a new resource shall release it through rollback; if no other lease exists, physical cleanup shall run before rollback completes. +5.6. Existing generation-owned derived state—including executor maps/views, routing views, model-registry runtime, model catalog, feature composition, policy state, and generation lifecycle context—shall remain generation-owned and shall not be moved into the connector resource pool. +5.7. Existing no-drop, retained-generation, old-stream, and last-good reload guarantees shall remain unchanged. + +## Requirement 6: Invalidation and Resource Incarnations + +6.1. Semantic resource identity and physical resource incarnation shall be distinct concepts: the same semantic identity may need a later fresh physical incarnation after failure. +6.2. When the existing connector/process invalidation path declares a physical resource generation unusable, that exact resource incarnation shall become non-acquirable for future candidates. +6.3. A future acquisition for the same semantic identity after invalidation shall create a fresh physical incarnation rather than returning the invalidated one. +6.4. Invalidation shall not live-swap a replacement resource into generations that already reference the failed incarnation. Their normal backend/process failure and recovery semantics remain authoritative. +6.5. An invalidation callback originating from an older/detached incarnation shall not evict or invalidate a newer current incarnation for the same semantic identity. +6.6. Existing `processhost` generation invalidation/process reap behavior shall remain authoritative for the physical process; reconciliation only controls future resource reuse eligibility and lease lifetime. + +## Requirement 7: Cleanup, Shutdown, and Error Preservation + +7.1. There shall be exactly one physical cleanup owner for each configured connector resource. A generation-specific closer shall never be capable of closing a physical resource still leased by another generation. +7.2. The process-scoped reconciliation owner shall close before `processhost.Host` during normal process teardown so remaining physical resource cleanup can still use the live host/session ownership path. +7.3. Process shutdown shall reject new resource acquisitions, be idempotent, and fail-safe close any residual physical resources after generation drain without creating a second process shutdown coordinator. +7.4. Verified artifacts and staging resources shall retain their existing later teardown ordering after all connector resources and `processhost.Host` are done with them. +7.5. Existing cleanup-error normalization and error-join behavior shall be preserved. Lease-triggered final physical cleanup errors shall surface through the existing generation rollback/close path rather than a new public error category. +7.6. Partial construction failure shall not leak sessions, host instances, processes, IPC connections, or leases. + +## Requirement 8: Connector Semantics and Non-Interference + +8.1. Reuse is valid only for a configured connector resource whose construction inputs are fully represented by the physical resource identity. If the implementation cannot prove identity completeness for an eligible path, that path shall fall back to current generation-local construction rather than risk unsafe reuse. +8.2. Dynamic provider facts that are already modeled through runtime calls such as `Resolve`, `ListModels`, health/readiness, or normal execution shall continue through those calls and shall not require connector reconstruction on unrelated Go-LIP generation changes. +8.3. The optimization shall not alter canonical requests/events, route selection, retries, failover, output-commit rules, streaming order, cancellation, billing finalization, accounting evidence, token counting, or provider-specific semantics. +8.4. No resource-pool lookup or lock shall be added to the normal request execution hot path; acquisition/release occurs at generation construction/retirement boundaries. +8.5. Existing backend-plugin security invariants—verified artifact binding, secure local IPC, peer authentication before secrets/configure, environment restrictions, and process-tree cleanup—shall remain unchanged. +8.6. Old and new generation behavior shall remain observationally equivalent to the current overlap model for unchanged resources except that unnecessary physical connector reconstruction is removed. + +## Requirement 9: TDD, Concurrency, and Architecture Gates + +9.1. Add RED tests for high-cardinality construction counts, identity discrimination, lease lifetime, rollback, invalidation/incarnation behavior, concurrency, and shutdown ordering before enabling production reuse. +9.2. Test concurrent acquire of one absent identity, concurrent release, release racing invalidation, candidate rollback while an old generation is retained, and a fresh acquire after invalidation. +9.3. Test unchanged reload, changed same-ID config, artifact change, secret change, policy change, remove/disable, candidate failure, and process shutdown. +9.4. Add race/goleak coverage for the private reconciliation owner and relevant executable connector lifecycle integration. +9.5. Architecture tests shall reject a generic service registry/container API, request-time lookup surface, public reusable-resource framework, provider-specific switch, or migration of `processhost` supervision responsibility. +9.6. Repository quality, focused executable-plugin conformance/security tests, and existing reload no-drop/rollback tests shall remain green. +9.7. Final implementation review shall remove unused abstraction layers and preserve the smallest private design that satisfies the deterministic scale and correctness gates. From cd7f8d6b54551d86d5f08fe7c71b92b520fd1f07 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:51:37 +0200 Subject: [PATCH 03/12] spec: analyze backend reconciliation brownfield gaps --- .../gap-analysis.md | 130 ++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 .kiro/specs/backend-connector-resource-reconciliation/gap-analysis.md diff --git a/.kiro/specs/backend-connector-resource-reconciliation/gap-analysis.md b/.kiro/specs/backend-connector-resource-reconciliation/gap-analysis.md new file mode 100644 index 00000000..a8fee0ea --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/gap-analysis.md @@ -0,0 +1,130 @@ +# Brownfield Requirements Gap Analysis + +## Result + +**PASS after requirements corrections.** The current Go-LIP architecture already has the ownership, immutable-generation, executable-process, and identity primitives needed for a focused reconciliation layer. The missing capability is not general dependency management: it is a process-scoped way for overlapping generations to share one unchanged configured external `per_instance` connector resource without sharing generation-local derived state. + +The initial requirements were directionally correct but needed several brownfield constraints made explicit before design. Those corrections are recorded below and must be reflected in the final `requirements.md`. + +## Existing Brownfield Facts + +- `buildBackends` constructs every enabled backend row during generation compilation, so a material reload can reconstruct unchanged backends. +- Discovered executable `per_instance` factories deliberately mint a generation-unique host activation handle so candidate and active generations can coexist safely today. +- `processhost.Host` already supervises process slots, authenticated IPC, configured instances, invalidation, and cleanup; duplicating those responsibilities would be harmful. +- `BackendStateIdentity` already demonstrates identity-sensitive reuse for affinity/health observation state, but its identity is intentionally narrower than the inputs that define a configured physical connector. +- `ProcessServices` now has the private `processResourceOwner`/owned-acquisition discipline from the earlier atomic-owned-resource-lifecycle work. +- The executable plugin host, verified artifacts, and staging directory are created before `ProcessServices`, then ownership transfers into `ProcessServices` before initial generation compilation. +- Installed connector catalog size is not itself the scaling problem: discovery is lazy with respect to process launch and already has 100-manifest no-launch coverage. + +## Gaps and Required Corrections + +### 1. Installed connector cardinality is not the target scaling dimension + +A large trusted catalog does not imply a large live resource set because discovery does not launch every connector. The expensive case is many **enabled configured process-backed instances** combined with material runtime reload. + +**Correction:** all scale requirements and benchmarks are defined in terms of enabled eligible connector instances, not installed manifest count. + +### 2. `BackendStateIdentity` is too weak for physical-resource reuse + +Its `{InstanceID, FactoryKind, ConfigDigest}` identity is appropriate for affinity/health continuity but does not represent exact executable artifact, configure-time runtime policy, secret values, or process model. + +**Correction:** define a separate private physical-resource identity. Reusing helpers is allowed, but physical reuse must never be authorized solely by `BackendStateIdentity.Compatible`. + +### 3. All configure-affecting inputs must participate in identity or reuse must fail closed + +The executable plugin receives opaque YAML, `SecretBundle`, `RuntimePolicy`, factory/instance identity, and negotiated process context. Future additions to configure-time input create a correctness hazard if the identity silently ignores them. + +**Correction:** require an explicit identity-construction choke point over the effective physical construction/configure input. If an input cannot be safely fingerprinted or declared process-stable, the resource is non-shareable and uses current generation-local construction. Add an architecture/contract test that forces identity review when configure-time DTO shape changes. + +### 4. The pool must exist before discovered lifecycle factories capture it + +Production installs discovered factory closures before `NewProcessServices`, while process ownership is transferred afterward. Creating the pool only inside `NewProcessServices` would arrive too late unless the factory used an indirection/service locator, which this project deliberately avoids. + +**Correction:** create the private reconciliation owner beside the discovered `processhost.Host` during discovered-install preparation, capture it directly in eligible factory closures, then transfer its lifetime into `ProcessServices`. No global lookup or post-construction setter is needed. + +### 5. Pool shutdown ordering is constrained by existing host/artifact/staging ownership + +Physical resource cleanup may need adapter/session cleanup followed by host instance cleanup. Therefore the resource pool must be torn down while `processhost.Host` and verified artifacts are still usable. + +**Correction:** on successful process ownership transfer, register ordering so normal reverse teardown is: generations drain → reconciliation pool/final physical cleanup → `processhost.Host` → verified artifacts → staging removal. Bootstrap-error release must preserve the same dependency order. + +### 6. Generation cleanup and physical cleanup are different ownership concepts + +Current `BackendBuildResult.Cleanup` is transferred into `ResourceLedger`. If that same cleanup were copied into two generations, either generation could close a connector still referenced by the other. + +**Correction:** the pool owns the physical cleanup exactly once. Each generation receives a fresh idempotent **lease release** as its `BackendBuildResult.Cleanup`; `ResourceLedger` continues to own that cleanup. No backend `Close`/lifecycle hook returned to the generation may bypass the lease and directly close the shared physical resource. + +### 7. Semantic identity is not physical incarnation identity + +A connector process/session may die while its desired configuration remains unchanged. Reusing by semantic key alone would hand a dead resource to the next candidate. Conversely, an old failure callback could incorrectly invalidate a newly rebuilt resource with the same semantic key. + +**Correction:** every physical entry has an incarnation token/version. Invalidation detaches only the exact incarnation, making it unavailable to future acquisitions; a later acquire creates a new incarnation. Stale invalidation cannot evict a newer incarnation. + +### 8. `shared_artifact` is a different problem + +The current shared-process model has explicit isolation/concurrency declarations and can be restart-required when overlap is unsafe. Mixing it into first-pass reconciliation would entangle process slot sharing with generation resource sharing. + +**Correction:** first implementation is `per_instance` discovered external connectors only. `shared_artifact` remains unchanged. + +### 9. Generation-local derived state must not be accidentally pooled + +A configured external session can be reused while executor maps, routing views, model-registry runtime, model catalog, feature surface, lifecycle context, and policy/accounting composition remain generation-specific. + +**Correction:** pool only the configured physical connector adapter/backend resource. Rebuild all current generation projections exactly as today from the leased backend/profile/inventory surface. + +### 10. Dynamic inventories/capabilities do not automatically make a resource non-shareable + +The adapter already models dynamic facts through runtime operations such as `Resolve` and `ListModels`. Reconstructing a connector merely because an unrelated generation changed would not make those dynamic facts more correct. + +**Correction:** reuse is allowed when **construction/configure inputs** are identical; dynamic runtime queries remain dynamic. If a connector requires generation-dependent hidden configuration not represented at configure time, it is non-shareable until that dependency is explicit in identity. + +### 11. A timing-only ROI gate would be fragile + +The existing candidate compiler is fast on ordinary fixtures, while real executable connector startup cost can vary by platform and connector. Fixed millisecond thresholds would conflate CI noise with architecture value. + +**Correction:** deterministic operation counts are the primary acceptance evidence. Wall time, allocations, process count/RSS/FD observations, and benchstat remain supporting evidence. + +### 12. The previous Cordis-inspired ownership spec does not already solve reuse + +`atomic-owned-resource-lifecycle` correctly hardened ownership locality and deliberately left backend construction unchanged because that work addressed forgotten cleanup, not cross-generation physical reuse. The new concern appears at connector-scale reload cardinality. + +**Correction:** reuse the existing ownership primitives; do not replace or generalize them. This specification adds one connector-specific lease/reconciliation owner and nothing broader. + +## Brownfield Compatibility Matrix + +| Existing subsystem | Required treatment | +|---|---| +| `runtimehost.Manager` / generation leases | unchanged | +| `GenerationRuntime` immutability | unchanged | +| `ResourceLedger` | owns per-generation lease release; unchanged authority | +| `ProcessServices` | gains lifetime ownership of one private connector reconciliation owner | +| `processResourceOwner` | reused for process teardown registration; not replaced | +| `processhost.Host` | unchanged process/IPC supervisor | +| discovered factory install | captures private pool before ProcessServices ownership transfer | +| `BackendBuildResult` | eligible factory returns leased cleanup instead of physical cleanup | +| `BackendStateIdentity` | remains affinity/health identity; not physical reuse authority | +| model registry/catalog | rebuilt per generation | +| plugin discovery/trust | startup-fixed and unchanged | +| backend-plugin ABI | unchanged | +| built-in backends | unchanged | +| `shared_artifact` connectors | unchanged | +| routing/streaming/retry/accounting | unchanged | +| public config/SDK | no new surface | + +## Corrected Required Invariants + +1. Optimization scope is enabled eligible executable `per_instance` connector resources, not catalog size. +2. Physical reuse requires complete configure/construction identity; incomplete identity falls back safely. +3. Semantic identity and physical incarnation are separate. +4. One physical resource has exactly one physical cleanup owner and many generation lease owners. +5. `ResourceLedger` owns lease release, never a shared physical closer. +6. Candidate rollback is local: releasing a reused lease cannot disturb the last-good generation. +7. Changed/removed resources preserve old-generation availability until drain. +8. Pool shutdown precedes host/artifact teardown and does not become a second process supervisor. +9. Generation-local projections remain generation-local. +10. No generic container, service locator, dynamic dependency graph, watcher, or request-time lookup is introduced. +11. Deterministic high-cardinality operation counts are the primary ROI gate. + +## Requirements Correction Status + +The final requirements must incorporate gaps 3–7 especially: a configure-input identity choke point, pre-`ProcessServices` pool construction/ownership transfer, strict physical-cleanup versus lease-cleanup separation, incarnation-safe invalidation, and explicit teardown order. Once those are present, the requirements quality gate is **PASS**. From fa5673364ae413867a2fca9dd49fff3fe77a18ed Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:52:13 +0200 Subject: [PATCH 04/12] spec: correct backend reconciliation requirements after gap analysis --- .../requirements.md | 33 ++++++++++++++----- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/.kiro/specs/backend-connector-resource-reconciliation/requirements.md b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md index e391dd60..7004840d 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/requirements.md +++ b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md @@ -4,7 +4,7 @@ Go-LIP shall reduce unnecessary reconstruction of expensive executable backend connector resources across immutable runtime generations when a material configuration reload does not change those connectors. The optimization shall preserve the existing generation publication model: every request remains bound to one immutable `GenerationRuntime`; changed resources are constructed before publication; old generations and their resources remain valid until their existing work drains; candidate failure leaves the last-good generation untouched. -This specification borrows only the Cordis-v4 ideas that fit this problem: semantic provider identity, reconciliation of unchanged desired resources, and lifetime retention while dependents still hold the provider. It does **not** introduce a Cordis component runtime, reactive dependency graph, fibers, dependency injection, service location, HMR, or generic effect system. +This specification borrows only the Cordis-v4 ideas that fit this problem: semantic provider identity, reconciliation of unchanged desired resources, physical incarnation identity, and lifetime retention while dependents still hold the provider. It does **not** introduce a Cordis component runtime, reactive dependency graph, fibers, dependency injection, service location, HMR, or generic effect system. The first implementation is intentionally narrow. It targets discovered executable backend connectors whose declared process model is `per_instance` and whose current reload policy permits candidate/active overlap. Installed-but-disabled connectors, statically linked backends, and `shared_artifact` connectors are not justification for this refactor and remain outside the first implementation. @@ -34,6 +34,8 @@ The first implementation is intentionally narrow. It targets discovered executab 2.5. Plugin discovery/trust shall remain startup-fixed. Connector installation, removal, directory rescanning, executable upgrade discovery, and automatic file watching are not added by this specification. 2.6. `processhost.Host` shall remain the sole executable process/IPC supervisor; the reconciliation layer shall not duplicate launch, peer authentication, process-tree cleanup, slot management, or transport supervision. 2.7. No new public YAML field, manifest field, CLI flag, environment variable, or backend-plugin ABI field shall be required merely to turn this optimization on. +2.8. Because discovered lifecycle factory closures are installed before `ProcessServices` construction, the private reconciliation owner shall be created beside the discovered `processhost.Host`, captured directly by eligible factory closures, and then have its lifetime transferred into `ProcessServices`; the implementation shall not solve this timing boundary with a global registry, service locator, or mutable post-construction dependency lookup. +2.9. The reconciliation owner shall remain connector-specific and package-private. It shall expose no generic keyed `Get`/`Resolve` API for unrelated runtime services. ## Requirement 3: Semantic Physical Resource Identity @@ -45,6 +47,9 @@ The first implementation is intentionally narrow. It targets discovered executab 3.6. Credential rotation or another configure-affecting secret change shall produce a distinct resource identity. 3.7. Runtime-policy changes that affect the configured connector shall produce a distinct resource identity. 3.8. `BackendStateIdentity` may provide precedent or low-level hashing helpers, but its current `{InstanceID, FactoryKind, ConfigDigest}` contract shall not be treated as sufficient proof that two physical connector resources are interchangeable. +3.9. Identity shall be derived at one explicit construction/configuration choke point from the effective inputs that define physical connector behavior. A future configure-time or launch-identity input shall require an intentional identity decision rather than being silently omitted. +3.10. If a construction/configure input varies between candidate generations and cannot be represented safely and deterministically in the identity, that resource shall be treated as non-shareable and shall use the existing generation-local construction path. +3.11. Process-stable facts need not be redundantly hashed, but every such omission shall be documented and protected by tests so a later change from process-stable to generation-varying cannot create unsafe reuse. ## Requirement 4: Process-Scoped Acquire and Generation Lease Contract @@ -56,6 +61,8 @@ The first implementation is intentionally narrow. It targets discovered executab 4.6. There shall be no idle TTL/cache-retention policy in the first implementation. A valid current physical resource exists only while at least one generation lease retains it. 4.7. A failed physical build/configure shall not be cached as a permanent negative result; a later independent acquisition may retry through the existing construction path. 4.8. Lease release shall be idempotent and safe under candidate rollback, generation retirement, and process shutdown races. +4.9. The physical cleanup returned by the underlying connector/process build shall be retained only by the reconciliation entry. Every generation-facing `BackendBuildResult.Cleanup` for a reused resource shall be a fresh idempotent lease release, not the physical cleanup function. +4.10. Eligible leased backend values/lifecycle hooks shall not expose an alternate generation-owned `Close` or `Stop` route that can bypass the lease and tear down a physical resource while another generation retains it. ## Requirement 5: Preserve Immutable Generation Semantics @@ -66,6 +73,7 @@ The first implementation is intentionally narrow. It targets discovered executab 5.5. A failed candidate that created a new resource shall release it through rollback; if no other lease exists, physical cleanup shall run before rollback completes. 5.6. Existing generation-owned derived state—including executor maps/views, routing views, model-registry runtime, model catalog, feature composition, policy state, and generation lifecycle context—shall remain generation-owned and shall not be moved into the connector resource pool. 5.7. Existing no-drop, retained-generation, old-stream, and last-good reload guarantees shall remain unchanged. +5.8. A reused configured connector may contribute the same underlying immutable backend/session functions to multiple generation-local executor maps, but each generation shall rebuild its own projections, inventories, routing and model views using the same existing generation compiler sequence. ## Requirement 6: Invalidation and Resource Incarnations @@ -75,15 +83,20 @@ The first implementation is intentionally narrow. It targets discovered executab 6.4. Invalidation shall not live-swap a replacement resource into generations that already reference the failed incarnation. Their normal backend/process failure and recovery semantics remain authoritative. 6.5. An invalidation callback originating from an older/detached incarnation shall not evict or invalidate a newer current incarnation for the same semantic identity. 6.6. Existing `processhost` generation invalidation/process reap behavior shall remain authoritative for the physical process; reconciliation only controls future resource reuse eligibility and lease lifetime. +6.7. Invalidation shall detach the exact failed incarnation from future semantic acquisition before or atomically with delegating to the existing processhost invalidation path, so a concurrent candidate cannot acquire a resource already known to be unusable. +6.8. Detached/invalidated entries may remain referenced by generations that already leased them until those leases release; they shall not remain discoverable as the current reusable entry. ## Requirement 7: Cleanup, Shutdown, and Error Preservation 7.1. There shall be exactly one physical cleanup owner for each configured connector resource. A generation-specific closer shall never be capable of closing a physical resource still leased by another generation. 7.2. The process-scoped reconciliation owner shall close before `processhost.Host` during normal process teardown so remaining physical resource cleanup can still use the live host/session ownership path. -7.3. Process shutdown shall reject new resource acquisitions, be idempotent, and fail-safe close any residual physical resources after generation drain without creating a second process shutdown coordinator. -7.4. Verified artifacts and staging resources shall retain their existing later teardown ordering after all connector resources and `processhost.Host` are done with them. -7.5. Existing cleanup-error normalization and error-join behavior shall be preserved. Lease-triggered final physical cleanup errors shall surface through the existing generation rollback/close path rather than a new public error category. -7.6. Partial construction failure shall not leak sessions, host instances, processes, IPC connections, or leases. +7.3. Normal successful shutdown ordering shall remain: generation admission stops and generations drain/release their leases; then reconciliation fail-safe cleanup closes any residual physical entries; then `processhost.Host` closes; then verified artifact handles close; then staging removal runs. +7.4. Bootstrap/error cleanup before process ownership transfer shall preserve the same dependency order for any reconciliation owner, host, artifacts, and staging resources already acquired. +7.5. Process shutdown shall reject new resource acquisitions, be idempotent, and fail-safe close any residual physical resources after generation drain without creating a second process shutdown coordinator. +7.6. Verified artifacts and staging resources shall retain their existing later teardown ordering after all connector resources and `processhost.Host` are done with them. +7.7. Existing cleanup-error normalization and error-join behavior shall be preserved. Lease-triggered final physical cleanup errors shall surface through the existing generation rollback/close path rather than a new public error category. +7.8. Partial construction failure shall not leak sessions, host instances, processes, IPC connections, pool entries, or leases. +7.9. `ProcessServices` shall own reconciliation shutdown through its existing private process resource ownership mechanism; no second closer stack or independent process shutdown graph shall be introduced. ## Requirement 8: Connector Semantics and Non-Interference @@ -93,13 +106,15 @@ The first implementation is intentionally narrow. It targets discovered executab 8.4. No resource-pool lookup or lock shall be added to the normal request execution hot path; acquisition/release occurs at generation construction/retirement boundaries. 8.5. Existing backend-plugin security invariants—verified artifact binding, secure local IPC, peer authentication before secrets/configure, environment restrictions, and process-tree cleanup—shall remain unchanged. 8.6. Old and new generation behavior shall remain observationally equivalent to the current overlap model for unchanged resources except that unnecessary physical connector reconstruction is removed. +8.7. Connector-specific configuration parsing remains inside the connector. The host identity may fingerprint opaque configure bytes but shall not learn provider-specific configuration schemas merely to decide reuse. ## Requirement 9: TDD, Concurrency, and Architecture Gates 9.1. Add RED tests for high-cardinality construction counts, identity discrimination, lease lifetime, rollback, invalidation/incarnation behavior, concurrency, and shutdown ordering before enabling production reuse. 9.2. Test concurrent acquire of one absent identity, concurrent release, release racing invalidation, candidate rollback while an old generation is retained, and a fresh acquire after invalidation. 9.3. Test unchanged reload, changed same-ID config, artifact change, secret change, policy change, remove/disable, candidate failure, and process shutdown. -9.4. Add race/goleak coverage for the private reconciliation owner and relevant executable connector lifecycle integration. -9.5. Architecture tests shall reject a generic service registry/container API, request-time lookup surface, public reusable-resource framework, provider-specific switch, or migration of `processhost` supervision responsibility. -9.6. Repository quality, focused executable-plugin conformance/security tests, and existing reload no-drop/rollback tests shall remain green. -9.7. Final implementation review shall remove unused abstraction layers and preserve the smallest private design that satisfies the deterministic scale and correctness gates. +9.4. Add an identity-contract test or equivalent compile-time/structural gate that forces deliberate review when the external connector configure-time DTO/input surface changes. +9.5. Add race/goleak coverage for the private reconciliation owner and relevant executable connector lifecycle integration. +9.6. Architecture tests shall reject a generic service registry/container API, request-time lookup surface, public reusable-resource framework, provider-specific switch, or migration of `processhost` supervision responsibility. +9.7. Repository quality, focused executable-plugin conformance/security tests, and existing reload no-drop/rollback tests shall remain green. +9.8. Final implementation review shall remove unused abstraction layers and preserve the smallest private design that satisfies the deterministic scale and correctness gates. From 2a5fa28bb7bafa97956eba7541885f8aafb2bc11 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:53:08 +0200 Subject: [PATCH 05/12] spec: document backend reconciliation research --- .../research.md | 273 ++++++++++++++++++ 1 file changed, 273 insertions(+) create mode 100644 .kiro/specs/backend-connector-resource-reconciliation/research.md diff --git a/.kiro/specs/backend-connector-resource-reconciliation/research.md b/.kiro/specs/backend-connector-resource-reconciliation/research.md new file mode 100644 index 00000000..1f48fa7d --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/research.md @@ -0,0 +1,273 @@ +# Research & Design Decisions + +## Summary + +- **Feature**: `backend-connector-resource-reconciliation` +- **Discovery Scope**: brownfield runtime optimization / connector-scale lifecycle refactor +- **Key Findings**: + - Go-LIP already solves the difficult correctness problem with immutable generations, request/async generation leases, `ResourceLedger`, and manager-owned retirement. Those mechanisms should remain authoritative. + - Catalog cardinality is already cheap: executable connector discovery is manifest-only and lazy with respect to process launch. The scaling pressure is the number of **enabled configured executable connector instances** rebuilt by a material generation reload. + - Current `buildBackends` constructs every enabled backend in every generation. Discovered `per_instance` factories deliberately mint unique host activation IDs so candidate and active generations can overlap, which also guarantees unchanged physical connectors are duplicated during candidate construction. + - The highest-ROI Cordis-v4 idea for this specific seam is provider identity + reconciliation + dependent lifetime retention: exact unchanged configured connector resources can be retained by more than one immutable generation and closed after the final generation releases them. + - A generic Cordis runtime remains a poor fit. The selected design is one private connector-specific process owner that returns per-generation leases and delegates all physical process supervision to existing `processhost.Host`. + - Physical reuse needs a stronger identity than `BackendStateIdentity`: exact artifact, opaque configure payload, runtime policy, secret fingerprint, process model, logical instance/factory identity, and any future configure-affecting input must be represented or reuse must fail closed. + +## Research Log + +### Cordis v4: the transferable principle + +- **Context**: Re-evaluate the user-supplied paper *A Programming Paradigm for Spatiotemporal Composability* (Yifan Shi, Wei Zhang, Tianyi Cui) against the current backend connector architecture. +- **Findings**: + - Cordis models components/providers by identity rather than only current values and reconciles a desired component graph against current runtime state. + - Dependents retain access to a withdrawing provider while their teardown runs; the provider is physically removed only after dependent cleanup. + - Revertible effects make rollback/teardown explicit and reverse effects when a component is withdrawn. + - Cordis's generic reactive coeffect graph, fibers, HMR, and dynamic service context solve a broader class of application-platform problems than Go-LIP needs here. +- **Implications**: + - Borrow semantic provider identity, physical incarnation identity, reconciliation, and reference-retained teardown. + - Map a Go-LIP generation to a dependent/lease holder, not to a Cordis component graph. + - Keep generation publication/retirement as the consistency boundary and avoid live dependency replacement under requests. + +### Prior Cordis-derived ownership work + +- **Context**: Determine whether `.kiro/specs/archive/atomic-owned-resource-lifecycle` already addressed this problem. +- **Sources Consulted**: + - archived `atomic-owned-resource-lifecycle` research/design + - `internal/infra/runtimebundle/process_owner.go` + - `internal/infra/runtimebundle/resource_ledger.go` +- **Findings**: + - The previous spec addressed acquisition/cleanup locality and worker ownership. + - It deliberately left backend lifecycle unchanged because `BackendBuildResult` already paired a backend with cleanup and `buildBackends` immediately transferred that cleanup to `ResourceLedger`. + - Main now contains `processResourceOwner` and `acquireOwnedProcess`, confirming that process cleanup locality is already hardened. +- **Implications**: + - This spec is not a correction to the prior work. It addresses a different dimension: **one expensive physical backend resource being useful to several overlapping generations**. + - Reuse existing ownership authorities; do not invent another general effect/owner framework. + +### Current generation construction and backend rebuild behavior + +- **Context**: Identify whether material reload reconstructs unchanged backend resources. +- **Sources Consulted**: + - `internal/infra/runtimebundle/build_model.go` + - `internal/pluginreg/lifecycle.go` + - `internal/infra/runtimebundle/reload_backend_recomposition_test.go` +- **Findings**: + - `buildModelRuntime` calls `buildBackends` for every candidate generation. + - `buildBackends` iterates every enabled backend row and calls `BuildBackendWithLifecycle`. + - A changed same-ID backend is intentionally constructed separately so old pinned generations retain old behavior while new generations use the replacement. + - This same whole-set construction happens for unchanged rows on unrelated material generation changes. +- **Implications**: + - Do not change changed/remove semantics. + - Introduce reuse only at the physical connector construction seam for exact unchanged identities. + - Continue rebuilding generation-local executor maps, inventories, model registry, routing projections, and policy views. + +### Discovered executable connector activation + +- **Context**: Determine where physical duplication originates and where reconciliation should sit. +- **Sources Consulted**: + - `internal/infra/runtimebundle/discovered_factories.go` + - `internal/infra/backendplugins/processhost/host.go` + - `internal/infra/backendplugins/processhost/model.go` + - `internal/infra/runtimebundle/reload_discovered_overlap_test.go` +- **Findings**: + - `InstallDiscoveredExports` registers one generic lifecycle factory per validated manifest export; there is no provider-specific switch. + - `buildDiscoveredBackend` encodes opaque YAML, activates/configures the host instance, builds an adapter backend, and returns physical cleanup. + - For `per_instance`, the runtime deliberately mints a distinct host activation handle for each candidate construction so old/new generations never collide in `Host.instances`. + - `processhost.Host` already has correct lazy launch, singleflight process slot creation, peer authentication, instance tracking, generation invalidation, and cleanup. + - `shared_artifact` has different isolation/concurrency and overlap policy semantics. +- **Implications**: + - Put reconciliation **above** `processhost`, not inside it. + - Preserve the unique host activation handle for each newly created physical incarnation. + - Reuse prevents calling physical construction at all for unchanged identity; it does not teach `processhost` about LIP config generations. + - Exclude `shared_artifact` initially. + +### Discovery cardinality versus live cardinality + +- **Context**: Validate whether hundreds of installed connectors themselves justify runtime reconciliation. +- **Sources Consulted**: + - `internal/infra/backendplugins/discovery/hundred_test.go` + - `docs/adr/0008-hybrid-backend-connector-plugins.md` + - `docs/backend-plugins/authoring.md` +- **Findings**: + - Discovery has explicit coverage for 100 synthetic manifests without launching them. + - Optional connectors are separate executable modules; installed but unconfigured plugins remain inactive. + - Activation is lazy and trusted artifacts are exact-digest bound. +- **Implications**: + - Do not optimize the manifest catalog or add dynamic discovery. + - The evidence harness must model many **enabled process-backed instances**, not merely many installed manifests. + +### Existing backend state identity precedent + +- **Context**: Check whether Go-LIP already distinguishes semantic backend identity across generations. +- **Sources Consulted**: + - `internal/infra/runtimebundle/backend_state_identity.go` + - `internal/infra/runtimebundle/shared_mutable.go` +- **Findings**: + - `BackendStateIdentity` namespaces process-owned affinity/health observations by instance ID, factory kind, and config digest. + - Compatible identities allow process-owned observation continuity across generation replacement; changed identities hide stale state from the new generation. + - This is a strong local precedent for identity-sensitive reuse but is purposefully narrower than physical connector interchangeability. +- **Implications**: + - Keep `BackendStateIdentity` unchanged. + - Introduce a separate private physical resource identity and avoid conflating observation-state compatibility with connector/session reuse compatibility. + +### Physical identity inputs + +- **Context**: Determine what makes two configured executable connector resources interchangeable. +- **Sources Consulted**: + - `pkg/lipsdk/backendplugin/types.go` + - `internal/infra/backendplugins/trust/artifact.go` + - `internal/infra/runtimebundle/discovered_factories.go` +- **Findings**: + - A connector is configured with logical `InstanceID`, `FactoryKind`, opaque YAML, secrets, `RuntimePolicy`, negotiation metadata, and an exact verified executable artifact/process model. + - `VerifiedArtifact.DigestHex` is the exact launch identity that must distinguish binary upgrades. + - `RuntimePolicy` contains execution bounds/timeouts/locality/environment policy that can affect configured behavior. + - Secret rotation can change configured credentials without changing public YAML. +- **Implications**: + - Physical identity must fingerprint every generation-varying construction/configure input. + - Secret values are hashed locally with stable sorted framing and never surfaced. + - Prefer one explicit identity builder at the same seam where effective configure input is assembled. + - Add a contract test that makes future configure DTO/input additions require deliberate identity review. + - When completeness cannot be proven, fall back to non-shared construction. + +### Adapter/backend cleanup shape + +- **Context**: Check whether the resulting backend value itself can safely be shared across executor maps. +- **Sources Consulted**: + - `internal/infra/backendplugins/adapter/backend.go` + - `internal/infra/backendplugins/processhost/build_result.go` + - `internal/infra/runtimebundle/build_model.go` +- **Findings**: + - `adapter.Build` creates an `execbackend.Backend` whose functional fields close over the configured `ExecuteSession` and resolved profile. + - Physical session cleanup is retained in `processhost.BuildResult` rather than embedded as an ordinary request-time operation. + - `buildDiscoveredBackend` combines adapter cleanup with `ActivateResult.Cleanup`, then returns it as `pluginreg.BackendBuildResult.Cleanup`. + - `buildBackends` currently registers that cleanup into the generation `ResourceLedger`. +- **Implications**: + - A reconciliation entry can own the physical `BackendBuildResult` and physical cleanup. + - Each generation must receive the same immutable backend functional value plus a **new lease-release cleanup**, never the underlying physical cleanup. + - Tests must prove no generation-local `Close`/lifecycle path can bypass the lease for pooled external resources. + +### Process ownership transfer and teardown order + +- **Context**: The discovered host exists before `ProcessServices`; determine how a pool can be captured by factories while still being process-owned. +- **Sources Consulted**: + - `internal/infra/runtimebundle/plugin_catalog.go` + - `internal/infra/runtimebundle/composition_root.go` + - `internal/infra/runtimebundle/host_build.go` + - `internal/infra/runtimebundle/process_services.go` +- **Findings**: + - `prepareDiscoveredPluginInstall` creates staging/trust artifacts and `processhost.Host` before factory registration. + - `InstallDiscoveredExports` registers factory closures before `NewProcessServices` freezes discovery. + - The host/artifacts/staging ownership bundle then transfers into `ProcessServices`. + - Existing reverse teardown intentionally closes host before artifacts and artifacts before staging. +- **Implications**: + - Create the private backend resource pool beside the discovered host during install preparation so closures capture it directly. + - Transfer the pool with the host into `ProcessServices` using a package-private construction field/seam. + - Register pool close after host close registration so reverse teardown runs pool → host → artifacts → staging. + - Error paths before transfer use the same dependency order. + +### Failure and incarnation semantics + +- **Context**: A desired identity can remain unchanged while its current process/session fails. +- **Findings**: + - Existing adapter invalidation can invalidate a `processhost` process generation. + - A semantic-key-only cache would risk returning the known-dead entry to later candidates. + - The same semantic identity must be able to acquire a fresh physical process/session after failure. +- **Implications**: + - Track physical incarnation identity independently from semantic identity. + - Invalidation detaches the exact current incarnation before/with delegating to existing host invalidation. + - New acquisition may build a fresh incarnation under the same semantic key. + - Old leases stay bound to the old incarnation; no published generation is live-mutated. + - Incarnation token comparison prevents a delayed stale invalidation from removing a newer entry. + +### ROI measurement strategy + +- **Context**: Candidate compilation is already fast on ordinary synthetic fixtures, while real connector startup is platform/provider dependent. +- **Findings**: + - Existing reload benchmarks are useful but do not model hundreds of enabled external process-backed connectors. + - Wall-clock startup varies with OS process creation, IPC, connector implementation, machine load, and CI environment. + - The architectural cost being removed has deterministic work-count semantics independent of timing noise. +- **Implications**: + - Primary gates are counts: builds, activations/launches, configure calls, physical cleanup. + - Add a 100-enabled-connector benchmark/evidence fixture and record wall time/allocations as supporting data. + - Expected unchanged reload changes from O(N) physical construction to O(N) lightweight lease acquisitions with zero physical reconstruction. + - Mixed reload with K changed connectors performs physical construction proportional to K, not N. + +## Architecture Pattern Evaluation + +| Option | Description | Strengths | Risks / Limitations | Decision | +|---|---|---|---|---| +| Full Cordis runtime | Generic provider graph, fibers, effects/coeffects, reactive reconciliation | broad composability | fights Go-LIP generation model; duplicates lifecycle machinery; high maintenance cost | Reject | +| Put generation reconciliation in `processhost` | Make host understand config-generation identity and reuse | central process knowledge | conflates physical supervision with semantic LIP generation composition | Reject | +| Reuse `BackendStateIdentity` directly | Treat existing observation-state key as physical resource key | smallest apparent change | unsafe: misses artifact, policy, secrets and future configure inputs | Reject | +| Cache `BackendBuildResult` per generation-independent key | Return same cleanup/value to multiple generations | simple lookup | double/early cleanup; no incarnation-safe invalidation | Reject | +| Process-owned connector resource leases | Private semantic identity → current physical incarnation → generation leases; host remains physical supervisor | focused ROI, preserves generations, scales with unchanged enabled connectors | identity and concurrency correctness require strong tests | **Select** | +| No change | Rebuild every enabled connector each material generation | simplest correctness model | O(N) physical activation wave and overlap resource spike at high enabled cardinality | Keep as fallback for non-shareable paths | + +## Design Decisions + +### Decision: Reconcile only configured physical external connector resources + +- Keep executor maps, model registry/catalog, routes, feature surfaces, generation lifecycle and policies generation-local. +- Share only the configured adapter/backend session resource for exact eligible identities. +- Rationale: this removes expensive reconstruction without weakening the generation consistency boundary. + +### Decision: Create the pool during discovered install, own it through ProcessServices + +- Factory closures need the pool before `ProcessServices` exists. +- Construct pool next to `processhost.Host`, capture directly, then transfer process lifetime into `ProcessServices`. +- Do not use a global map or setter/locator. + +### Decision: One semantic current entry plus explicit physical incarnation + +- Semantic identity expresses desired configured resource equality. +- Incarnation identifies the concrete live process/session created for that desire. +- Invalidation detaches an incarnation without banning the semantic identity forever. + +### Decision: Physical cleanup lives in the entry; ResourceLedger owns a lease release + +- Underlying `BackendBuildResult.Cleanup` is consumed by the pool and never copied to multiple generations. +- Each Acquire returns an independent idempotent release closure. +- Final release performs physical cleanup exactly once. + +### Decision: No idle cache + +- Reuse exists only across overlapping retained generations. +- Final lease release closes immediately. +- This avoids cache sizing, TTLs, memory/process retention, eviction races, and another operational knob. + +### Decision: No public opt-in flag + +- Exact-identity reuse is an internal semantic-preserving optimization. +- Unsafe/incomplete identities fall back automatically to current construction. +- A public feature flag would expose internal lifecycle structure and create configuration/test matrix cost without adding user capability. + +### Decision: Preserve unique host activation IDs for newly built incarnations + +- The pool prevents unnecessary build calls for unchanged resources. +- When a build is actually required, current host-instance uniqueness remains intact, preserving overlap safety and current `processhost` assumptions. + +## Risks & Mitigations + +- **Identity omission causes unsafe reuse** — one construction-input identity choke point, fail-closed fallback, DTO-shape/identity contract tests. +- **One generation closes another generation's resource** — pool owns physical cleanup; generation ledgers own only idempotent lease release; no bypass closer. +- **Dead resource reused** — exact-incarnation invalidation detaches before future acquisition. +- **Stale invalidation kills new resource** — compare incarnation token before detaching current entry. +- **Pool becomes a service locator** — package-private connector-specific API, construction-only use, architecture tests forbid generic `Get`/`Resolve` and request-path access. +- **Pool duplicates processhost** — no launch/IPC/process tree/peer auth logic in pool; physical build and invalidation delegate to existing host. +- **Shutdown ordering regression** — explicit pool → host → artifacts → staging characterization tests on success and bootstrap error. +- **Resource retained after last generation** — no idle cache; refcount zero synchronously detaches and closes. +- **Race between Acquire, Release, Invalidate, Close** — small state machine with no external cleanup under mutex; race/goleak tests and per-key build serialization. +- **Optimization hides generation-specific derived state** — explicit tests prove model/routing/policy views are rebuilt while physical activation counts remain zero for unchanged resources. +- **Speculative complexity** — deterministic 100-connector operation-count gate and final simplification review; revert/re-scope if architecture cost exceeds demonstrated gain. + +## References + +- User-supplied paper: *A Programming Paradigm for Spatiotemporal Composability* — provider identity, reconciliation, dependent teardown, revertible-effect concepts. +- `.kiro/specs/archive/atomic-owned-resource-lifecycle/` — prior focused Cordis-derived ownership hardening. +- `.kiro/specs/archive/runtime-architecture-convergence-and-shrinkage/` — canonical one-process/one-generation/one-host architecture and anti-container guardrails. +- `docs/runtime-config-reload.md` — transactional last-good generation publication/retirement contract. +- `docs/adr/0008-hybrid-backend-connector-plugins.md` — executable connector process model and lazy discovery/activation boundary. +- `internal/infra/runtimebundle/build_model.go` — all-enabled-backend candidate construction and generation-local model runtime. +- `internal/infra/runtimebundle/discovered_factories.go` — discovered lifecycle factory activation/configure/cleanup seam. +- `internal/infra/backendplugins/processhost/` — physical executable process/IPC ownership authority. +- `internal/infra/runtimebundle/backend_state_identity.go` and `shared_mutable.go` — existing semantic identity precedent for affinity/health state. +- `internal/infra/runtimebundle/process_owner.go` — current process-owned acquisition/cleanup authority facade. +- `pkg/lipsdk/backendplugin/types.go` — configure-time DTO and runtime policy identity surface. From d5a617cc4f95a3e22e8821184752f77f0316b939 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:56:09 +0200 Subject: [PATCH 06/12] spec: design backend connector resource reconciliation --- .../design.md | 736 ++++++++++++++++++ 1 file changed, 736 insertions(+) create mode 100644 .kiro/specs/backend-connector-resource-reconciliation/design.md diff --git a/.kiro/specs/backend-connector-resource-reconciliation/design.md b/.kiro/specs/backend-connector-resource-reconciliation/design.md new file mode 100644 index 00000000..462041f9 --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/design.md @@ -0,0 +1,736 @@ +# Design Document + +## Overview + +This design removes redundant physical reconstruction of unchanged discovered executable `per_instance` backend connectors during material runtime-generation reload while preserving Go-LIP's existing immutable-generation consistency model. + +The selected architecture adds one package-private, process-scoped connector resource reconciliation owner above `processhost.Host`. The owner maps an exact semantic physical-resource identity to the current live physical incarnation. Generation compilation acquires a lease. An unchanged candidate receives the already-configured immutable backend/session resource plus a generation-owned lease release; a changed identity constructs a new physical incarnation exactly as today. The existing `ResourceLedger` owns each generation's lease release, and the existing process host continues to own process launch, authenticated IPC, configured host instances, process invalidation, and process-tree teardown. + +The design is intentionally **not** a component runtime. There is no generic dependency graph, provider lookup, service locator, DI container, HMR, dynamic plugin discovery, reusable public resource framework, or request-hot-path lookup. + +### Goals + +- Reduce physical connector construction on material reload from all enabled eligible connectors to only connectors whose physical identity changed or whose current incarnation is unusable. +- Preserve immutable `GenerationRuntime` publication, old-generation drain, last-good candidate rollback, and current processhost security/supervision. +- Make physical resource identity complete and fail-closed with respect to configure/construction inputs. +- Give one physical resource exactly one physical cleanup owner while allowing several overlapping generations to hold independent idempotent leases. +- Preserve changed/remove behavior: changed resources replace before publication; removed resources remain usable only through old retained generations until drain. +- Preserve physical failure semantics without live-swapping resources under published generations. +- Establish deterministic high-cardinality operation-count evidence before and after production integration. + +### Non-Goals + +- Replace `runtimehost.Manager`, generation leases, `GenerationRuntime`, `ResourceLedger`, `ProcessServices`, or `processhost.Host`. +- Pool built-in/in-process backend factories. +- Change `shared_artifact` process semantics or restart-required overlap behavior. +- Share generation-local executor, model-registry, catalog, routing, feature, policy, billing, or lifecycle state. +- Add live connector install/upgrade/remove discovery, watchers, rescans, or HMR. +- Add a public resource-pool API or a new connector manifest/ABI/config flag. +- Change canonical request/event translation, routing, retry/failover, streaming, output commit, cancellation, token accounting, or billing behavior. +- Guarantee that an external process can never fail while a candidate performs a semantically read-only metadata query; the guarantee is that candidate rollback/rejection does not reconfigure, stop, close, or intentionally invalidate a shared last-good resource. + +## Boundary Commitments + +### Existing Authorities That Remain Authoritative + +| Concern | Existing authority | Design treatment | +|---|---|---| +| process-scoped lifetime | `ProcessServices` / `processResourceOwner` | owns reconciliation close | +| generation cleanup | `ResourceLedger` | owns lease release | +| request/async generation lifetime | `runtimehost.Manager` / generation refs | unchanged | +| executable process/IPC | `processhost.Host` | unchanged | +| exact executable trust | `VerifiedArtifact` | artifact digest enters identity | +| plugin ABI | `pkg/lipsdk/backendplugin` | unchanged | +| discovery/catalog | startup-fixed discovery/trust | unchanged | +| generation model views | current compiler/runtimebundle | rebuilt per generation | + +### Revalidation Triggers + +The implementation must be revalidated if any of these occur: + +- a new configure-time or physical-construction input is added to executable connectors; +- `ConfiguredInstance` gains a new lifecycle method used during generation preparation; +- generation compilation begins invoking a mutating lifecycle action on discovered connector backends; +- `processhost` ownership keys/process models change; +- physical resource reuse expands beyond discovered `per_instance` connectors; +- a public configuration/ABI field is proposed for reconciliation; +- request execution begins consulting the resource pool; +- pool shutdown can run before generation drain or after processhost/artifact teardown. + +## Existing Architecture Analysis + +### Current successful overlap model + +Today a material generation build follows the safe but potentially expensive pattern: + +```text +Generation 17 (published) + └── backend A -> physical connector A#17 + +compile Generation 18 + └── backend A -> physical connector A#18 // new Activate + Configure + +publish Generation 18 + ├── new admissions -> A#18 + └── retained Gen17 work -> A#17 + +retire Gen17 + └── cleanup A#17 +``` + +The fresh physical connector is deliberate. `buildDiscoveredBackend` mints a unique host activation handle for `per_instance` connectors so `processhost.Host.instances` can hold candidate and active handles simultaneously. That mechanism must remain for **new physical incarnations**. + +The inefficiency appears when A's physical defining inputs did not change. In that case the candidate reconstructs a resource whose configured behavior is intentionally identical. + +### Target overlap model + +```text +ProcessServices + └── backendResourcePool + └── semantic identity A + └── incarnation 41 + ├── configured backend/session + ├── physical cleanup + └── refs = 2 + ▲ ▲ + │ │ + Gen17 Gen18 +``` + +Generation 18 still gets its own executor map, model registry, routing views, policies, handler, and `ResourceLedger`. Only the configured external connector resource is shared. + +Changed identity remains the existing double-resource pattern: + +```text +identity A-old -> incarnation 41 <- Gen17 +identity A-new -> incarnation 42 <- Gen18 candidate +``` + +## Selected Architecture + +### Component Map + +```mermaid +graph TB + Install[Discovered install preparation] --> Host[processhost.Host] + Install --> Pool[backendResourcePool] + Pool --> Host + Install --> Factories[discovered lifecycle factory closures] + Factories --> Pool + Pool --> Physical[configured connector physical incarnation] + Physical --> Host + Process[ProcessServices] --> Pool + Process --> Host + Compile[Generation compiler] --> Factories + Factories --> Lease[BackendBuildResult with lease cleanup] + Lease --> Ledger[ResourceLedger] + Compile --> Gen[GenerationRuntime] + Gen --> Backend[leased immutable backend functions] +``` + +The pool is an **ownership/reconciliation index**, not a service registry. Only discovered factory construction calls it. Request execution sees only the backend functions already embedded in its immutable generation executor. + +### Construction and Ownership Timing + +Production currently installs discovered factory closures before `ProcessServices` exists. The pool therefore must be created at discovered-install preparation time next to `processhost.Host`: + +```text +prepareDiscoveredPluginInstall + acquire staging + verify artifacts + create processhost.Host + create backendResourcePool(host-associated lifetime) + register discovered factories capturing host + pool + │ + ▼ +BuildHost + transfer host + pool + artifacts + staging + │ + ▼ +NewProcessServices + register staging cleanup + register artifact cleanup + register host cleanup + register pool cleanup // reverse close: pool -> host -> artifacts -> staging +``` + +Exact field/helper names may adapt to package conventions. The required property is direct lexical capture plus lifetime transfer; no global registration or mutable service lookup is allowed. + +`discoveredBackendInstall.release` and early bootstrap cleanup gain the pool and release in dependency order: + +```text +pool -> host -> artifacts -> staging +``` + +### Private Types + +Illustrative package-private shapes: + +```go +type backendResourceIdentity struct { + instanceID string + factoryKind string + artifactDigest string + processModel processhost.ProcessModel + configDigest [32]byte + configureDigest [32]byte +} + +type backendResourcePool struct { + mu sync.Mutex + closing bool + nextInc uint64 + current map[backendResourceIdentity]*backendResourceEntry +} + +type backendResourceEntry struct { + identity backendResourceIdentity + incarnation uint64 + state backendResourceState + refs int + ready chan struct{} + + backend execbackend.Backend + cleanup func() error // physical cleanup; pool-only + buildErr error +} + +type backendResourceLease struct { + pool *backendResourcePool + entry *backendResourceEntry + once sync.Once + err error +} +``` + +These are design shapes, not public contracts. The implementation should use fewer fields/types if equivalent safety can be expressed more simply. + +There is deliberately no generic `Resource[T]`, `Scope`, `Provider`, `Component`, `Context`, `Registry`, or public `LeaseManager` abstraction. + +## Physical Resource Identity + +### Identity Principle + +A semantic physical identity answers only this question: + +> If two generation builds present these inputs, is it correct for both generations to execute through the same already-configured connector instance without another Configure/physical build? + +Equality must be conservative. False negatives cost performance; false positives can violate correctness. + +### Required Inputs + +Identity is derived at the `buildDiscoveredBackend` construction/configuration choke point from: + +1. logical configured `instanceID`; +2. manifest/export `factoryKind`; +3. exact `VerifiedArtifact.DigestHex`; +4. declared `ProcessModel` (`per_instance` for eligible resources); +5. effective opaque YAML bytes actually sent to Configure; +6. effective `backendplugin.RuntimePolicy` after host-owned normalization such as `DisableTransportRetries=true`; +7. configure-time secret bundle fingerprint when non-empty; +8. any future generation-varying input consumed by physical launch/configure semantics. + +`BackendStateIdentity` is not used as the reuse key. It may share a low-level canonical digest helper only if doing so does not couple the two compatibility contracts. + +### Canonical Fingerprinting + +Use SHA-256 with unambiguous length-delimited/domain-separated framing, not string concatenation with secret values. Example conceptual framing: + +```text +backend-resource/v1 +field(instance_id, bytes) +field(factory_kind, bytes) +field(artifact_digest, bytes) +field(process_model, bytes) +field(config_yaml, bytes) +field(runtime_policy_v1, canonical bytes) +field(secret_fingerprint_v1, digest bytes) +``` + +Runtime policy serialization must explicitly enumerate every field in `backendplugin.RuntimePolicy`. Slices/maps use deterministic ordering where their semantics are set/map-like. SecretBundle fingerprinting sorts secret names and hashes length-framed name/value bytes; only the resulting digest survives identity construction. + +### DTO Drift Gate + +A focused structural/contract test must make additions to the configure-time physical input surface fail until identity treatment is intentional. Acceptable techniques include: + +- a test that explicitly projects every `RuntimePolicy` field and fails when struct shape changes; +- a compile-time helper that consumes a versioned private identity DTO and a test comparing the external DTO's reflected field names in test code; +- another deterministic approach that forces review without reflection in production. + +Production identity code should remain explicit rather than generic reflection-based serialization. + +### Fail-Closed Eligibility + +If an input cannot be represented safely or is known to carry generation-specific semantics outside the identity, the factory bypasses reconciliation and runs the current physical construction path. Non-shareable fallback is correctness-preserving and requires no user-visible error. + +## Resource Pool State Machine + +### States + +A minimal entry needs these conceptual states: + +```text +building -> live -> detached/invalid -> closed + │ │ │ + └------> failed └-> final lease release -> physical cleanup +``` + +`detached` means it is no longer the current reusable entry for its semantic identity. Existing leases may still reference it. + +A `closing` pool rejects new acquisitions and waits for in-progress physical builders before host teardown. + +### Acquire Existing + +```text +Acquire(identity) + lock + current[identity] == live entry ? + yes -> refs++ + unlock + return backend + fresh lease.Release + no -> absent/building path +``` + +No connector factory, host activation, Configure RPC, or adapter build occurs on a live reuse hit. + +### Concurrent Absent Acquire + +The implementation must ensure one physical construction per semantic identity. A small per-key pending entry is preferred over a broad generic singleflight abstraction if it simplifies ref ownership: + +```text +first caller: + install building entry with reserved ref + build physical resource outside mutex + publish live entry + +other callers: + observe building entry + wait on ready/cancellation + retry state check + increment ref only after live +``` + +The design must avoid this race: the first caller builds/releases to zero before a waiting caller has formally acquired its ref. A pending-entry protocol or equivalent must make ref reservation and publication unambiguous. + +Physical build/configure and physical cleanup must never run while holding the pool mutex. + +### Failed Build + +- remove/detach the failed pending entry; +- wake waiters; +- run any partial cleanup through existing build/processhost error ownership; +- return the existing wrapped build error; +- do not negative-cache failure permanently; +- a later independent acquire may build again. + +### Release + +Each generation gets a fresh idempotent release closure. + +```text +Release(entry) + once + lock + refs-- + if refs > 0: + unlock + return nil + if current[key] == entry: + delete current[key] + mark detached/closing + capture physical cleanup + unlock + cleanup exactly once +``` + +Because there is no idle cache, final release closes immediately. + +## Physical Construction Integration + +### Current path retained as the builder + +The pool does not duplicate `buildDiscoveredBackend`. Refactor that function only enough to separate: + +1. identity/eligibility preparation; +2. physical construction of a new unique host activation/session/adapter resource; +3. pool acquisition returning a leased generation result. + +Conceptual shape: + +```go +func buildDiscoveredBackend(...) (pluginreg.BackendBuildResult, error) { + input, err := prepareDiscoveredPhysicalInput(...) + if err != nil { ... } + + if !eligible(input) || pool == nil { + return buildDiscoveredPhysical(input) + } + + id, shareable, err := physicalIdentity(input) + if err != nil { ... } + if !shareable { + return buildDiscoveredPhysical(input) + } + + return pool.Acquire(ctx, id, func(incarnation uint64) (physicalBackendResource, error) { + return buildDiscoveredPhysical(input, incarnation) + }) +} +``` + +The current unique `hostInstanceID = logicalID#sequence` behavior remains inside `buildDiscoveredPhysical` for every newly created incarnation. + +### Pool return shape + +The entry owns: + +- `execbackend.Backend` functional value; +- underlying adapter/session cleanup; +- `ActivateResult.Cleanup`/host instance cleanup; +- exact invalidation binding for that physical process generation/incarnation. + +The generation receives: + +```go +pluginreg.BackendBuildResult{ + Backend: entry.backend, + Cleanup: lease.Release, +} +``` + +`buildBackends` remains the authority that transfers `BackendBuildResult.Cleanup` into `ResourceLedger`. Therefore rollback/retirement semantics need no new generation cleanup engine. + +### No cleanup bypass + +For eligible external pooled resources, `buildBackends` must not synthesize another generation-local physical close from backend lifecycle hooks. Characterization tests should lock the current adapter contract: the pooled backend has no `Start`/`Stop`/`Close` path whose invocation would physically close the configured session outside the pool lease. + +If future external connector adapters gain such lifecycle hooks, pooling eligibility must be revisited rather than silently wrapping them twice. + +## Candidate Preparation and Last-Good Isolation + +Sharing a physical process changes one aspect of the failure domain: overlapping generations no longer own independent OS processes for an unchanged connector. This is acceptable only if candidate construction does not use the shared resource for **generation-local mutation**. + +### Allowed reused-resource preparation operations + +Current external adapter semantics expose query-shaped operations such as: + +- `Resolve` capability/profile lookup; +- `ListModels` inventory snapshot when advertised. + +These may be invoked by generation-local model registry construction/refresh. They are semantically query operations in the backend-plugin contract and may use internal connector caches, but candidate code must not call `Configure`, `Close`, `Stop`, or a mutating lifecycle/preflight operation on a reused physical resource. + +### Required last-good guarantee + +A candidate that later fails due unrelated validation/composition must only release its lease. It must not: + +- reconfigure the shared resource; +- close/stop it; +- invalidate it merely because the candidate is rejected; +- mutate processhost instance ownership; +- cancel active-generation executions. + +An external connector/process can still independently fail while serving query/execution calls; existing invalidation/recovery semantics remain authoritative. The optimization must not turn ordinary candidate rollback into such a failure. + +If generation preparation later requires a mutating backend lifecycle action, that backend/resource path becomes non-shareable until a new design proves isolation. + +## Generation-Local Derived State + +Physical reuse does **not** mean generation reuse. For every material generation compile, existing code still builds: + +```text +leased physical backends + ↓ +BackendInventory slice + ↓ +new modelregistry.Runtime + ↓ +new registry snapshot / generation-local refresh ownership + ↓ +new executor/routing/policy views + ↓ +new HTTP handler / GenerationBundle / ResourceLedger +``` + +This preserves current per-generation model/routing consistency. It also means two overlapping generations may issue concurrent metadata queries through the same physical connector. That is compatible with a configured instance already serving concurrent attempts; race/conformance tests must cover it. + +The first implementation does not attempt to deduplicate model-registry refresh loops or other derived computation. That is separate evidence work. + +## Invalidation and Incarnations + +### Why incarnation identity is required + +Configuration equality does not imply a particular process/session is still healthy. + +```text +semantic identity X + incarnation 7 -- fails + incarnation 8 -- replacement, same desired configuration +``` + +The pool therefore tracks the exact entry/incarnation in every invalidation callback. + +### Invalidation flow + +For a new physical build, adapter invalidation is wrapped conceptually as: + +```go +func() { + pool.Invalidate(identity, incarnation) + _ = host.InvalidateProcessGeneration(processGeneration) +} +``` + +Required order/properties: + +1. mark/detach only the exact entry incarnation from `current[identity]`; +2. future Acquire cannot obtain it; +3. delegate physical invalidation/reap to existing `processhost.Host`; +4. do not decrement existing generation refs merely because resource failed; +5. final lease release later runs the entry's idempotent physical cleanup, which may encounter already-gone transport and continues using existing cleanup normalization; +6. if `current[identity]` already points to a newer incarnation, a stale callback does not modify the newer entry. + +### No live substitution + +Existing generations retain backend functions closing over the failed incarnation. They observe existing failure/recovery behavior. The pool never swaps an entry pointer behind an executor or redirects an already-open attempt to the new incarnation. + +A later generation build can acquire a new incarnation for the same semantic identity. + +## Changed, Removed, and Rollback Flows + +### Unchanged unrelated material reload + +```text +Gen17 lease X refs=1 +compile Gen18 -> Acquire(X) refs=2; no physical build +candidate validates +publish Gen18 +retire Gen17 -> refs=1 +retire Gen18 later -> refs=0 -> physical cleanup +``` + +### Changed config/artifact/secret/policy + +```text +Gen17 lease X refs=1 +compile Gen18 -> identity Y != X + -> build Y incarnation +publish Gen18 +Gen17 keeps X until drain +``` + +### Remove/disable + +```text +Gen17 retains X +Gen18 does not Acquire(X) +publish Gen18 +X closes only after Gen17's final lease release +``` + +### Candidate fails after reusing X + +```text +active Gen17 refs(X)=1 +candidate Gen18 Acquire(X) -> refs=2 +later candidate failure +ResourceLedger rollback -> lease release -> refs=1 +active Gen17 unaffected +``` + +### Candidate fails after creating Y + +```text +candidate creates Y refs=1 +later failure +rollback -> refs=0 -> physical cleanup Y +active X unchanged +``` + +## Process Shutdown + +`Host.Close` remains the sole process shutdown coordinator and existing runtimehost generation drain remains first. + +Target ordering after generations have drained: + +```text +ProcessServices.Close reverse ownership + 1. backendResourcePool.Close + - reject new Acquire + - wait for any in-progress builder to terminate + - fail-safe close residual entries + 2. processhost.Host.Close + 3. VerifiedArtifact.Close handles + 4. staging directory removal + ...existing earlier/later ProcessServices resources as currently ordered... +``` + +The actual closer list contains many other process resources; the important relative ordering is pool before host before artifacts before staging. + +`backendResourcePool.Close` is idempotent. Under correct host shutdown its live refcount set should normally be empty because generations drained. Residual entries indicate a failed/aborted ownership path and are closed as a fail-safe rather than leaked. + +Pool close must not hold its mutex while waiting for builders or running physical cleanup. + +## Error Handling + +- Preserve existing `runtimebundle`/`processhost` build error wrapping and public reload categories. +- Do not add `resource_reuse_failed` or similar public error categories. +- Failed physical build leaves no reusable current entry. +- Final lease release returns underlying normalized physical cleanup error through the existing `ResourceLedger` rollback/close aggregation path. +- Non-final lease release normally returns nil because it performs no physical cleanup. +- Pool shutdown joins residual cleanup failures consistently with existing process close aggregation. +- Cancellation while waiting for another caller's build returns the caller's context error without canceling the builder on behalf of other dependents. + +## Concurrency Design + +### Locking rules + +- One small pool mutex protects map membership, entry state, refcounts, pool-closing flag, and incarnation allocation. +- No process launch, Configure RPC, metadata RPC, session close, host cleanup, channel wait, or callback is executed while holding the mutex. +- Waiters block on per-entry readiness outside the mutex and re-check state after wake. +- Lease release is `sync.Once`-guarded. +- Physical cleanup is exactly-once entry-owned even when invalidation already reaped the process. + +### Races to prove + +1. two candidates concurrently acquire the same absent identity -> one physical build, two leases; +2. waiter cancellation does not tear down another caller's resource; +3. final release races new Acquire -> either Acquire reserves before final detach or builds a new incarnation after detach; never acquires a closing resource; +4. invalidation races Acquire -> no acquire after known invalidation can receive the invalid entry; +5. stale invalidation races new incarnation publication -> newer entry survives; +6. pool Close races pending build -> builder cannot publish into a closing pool and physical result is cleaned before host close; +7. candidate rollback races old-generation release -> refcount/cleanup exactly once. + +## Scale and ROI Evidence Design + +### Deterministic 100-connector harness + +Add a focused runtimebundle/processhost-backed fixture capable of creating at least 100 enabled synthetic discovered `per_instance` rows without external credentials. The fixture exposes counters for: + +- lifecycle factory/physical build invocation; +- host activation; +- OS/fake launcher launch where the test profile supports it; +- Configure/dial session count; +- physical session/host cleanup count; +- lease acquisition/release count after implementation. + +Use a fake/in-process session or test processhost substrate where necessary so the deterministic gate remains fast and cross-platform. Separate native process smoke tests continue to protect actual process cleanup/security. + +### Required scenarios + +| Scenario | Physical construction expectation | +|---|---| +| baseline unrelated material reload before implementation | O(N) new construction, characterize current behavior | +| target unrelated material reload, N unchanged | 0 new builds/activations/configures | +| one of N configs changed | 1 replacement physical build | +| K of N identities changed | K replacement physical builds | +| disabled/removed subset | 0 builds for removed rows; old resources close after old generation drain | +| candidate fails after all N reuse hits | 0 physical cleanup of active resources | +| candidate builds K new then fails | K new resources cleaned; active unchanged resources retained | +| invalidated one of N then compile same config | exactly 1 new physical incarnation | + +### Supporting benchmark evidence + +Extend/reuse `reload_bench_test.go` or add a focused benchmark to report candidate compile time and allocations for high-cardinality external connector fixtures. If platform-test infrastructure makes useful native process metrics available, record peak launch/process/FD/RSS observations, but do not make unstable host metrics the correctness gate. + +The primary claimed improvement is structural: + +```text +physical construction work: + before ≈ O(N enabled eligible connectors) per material generation + target ≈ O(K changed/unusable connectors) + O(N) cheap lease/projection work +``` + +No request throughput or per-token latency improvement is claimed. + +## File Structure Plan + +Exact filenames may adapt during implementation, but responsibilities should remain narrow: + +```text +internal/infra/runtimebundle/ +├── backend_resource_identity.go # private identity/fingerprinting +├── backend_resource_pool.go # private entry/lease/reconciliation owner +├── backend_resource_pool_test.go # RED state/concurrency tests +├── backend_resource_identity_test.go # identity completeness/discrimination +├── discovered_factories.go # narrow integration / physical builder split +├── plugin_catalog.go # construct pool beside processhost host +├── composition_root.go # transfer/error-release ownership bundle +├── process_services*.go # process ownership transfer/ordering +├── build_model.go # preserve lease cleanup -> ResourceLedger +└── reload_backend_resource_reuse_test.go # high-cardinality + generation semantics + +internal/infra/backendplugins/processhost/ +└── existing files # no ownership redesign; focused test seams only if required + +internal/archtest/ +└── backend_resource_reconciliation_test.go # no generic registry/request-path/public framework +``` + +No new package is required unless file/package budgets make a tiny connector-lifecycle subpackage materially clearer. A new generic `resource`, `lifecycle`, `container`, or `dependency` package is explicitly disallowed by this design. + +## Testing Strategy + +### TDD order + +1. high-cardinality characterization and target count RED tests; +2. identity discrimination/completeness RED tests; +3. pool lease/refcount/build/invalidation/shutdown RED tests; +4. process ownership-order RED tests; +5. discovered factory reuse integration; +6. unchanged/changed/remove/rollback generation integration; +7. race/goleak, plugin security/conformance, no-drop reload regression; +8. supporting benchmark/benchstat evidence; +9. final simplification review. + +### Regression suites + +At minimum preserve/pass the focused equivalents of: + +- runtimebundle backend recomposition tests; +- discovered overlap/restart-required tests; +- ResourceLedger rollback/retirement tests; +- processhost activation/cleanup/invalidation tests; +- backend-plugin security and conformance gates; +- runtime reload last-good/no-drop tests; +- ownership/architecture tests from the prior resource-lifecycle refactor. + +## Rejected Alternatives + +### Reconfigure an existing physical connector in place + +Rejected. It would mutate the provider under old generations and destroy generation consistency. + +### Make processhost reuse logical instance IDs across generations + +Rejected. `processhost` should not know LIP semantic config identity, and existing unique activation handles are valuable for genuine replacement overlap. + +### Keep new physical resources alive in an idle cache + +Rejected. Cross-generation overlap is sufficient for the target optimization; idle retention introduces TTL/eviction/resource-pressure policy with no demonstrated need. + +### Pool every backend type + +Rejected. Cheap in-process builtins do not justify the ownership complexity and may have different lifecycle semantics. + +### Add a manifest `reusable_across_generations` capability immediately + +Rejected for the first implementation. The host-owned executable adapter path and complete identity/fallback rules are sufficient to prove the concept without expanding ABI. If real connectors later demonstrate incompatible semantics that cannot be inferred from the common adapter contract, a separate compatibility/capability specification can revisit this. + +### Share the entire model runtime across generations + +Rejected. Model/routing views are part of the immutable generation contract and may depend on other config fields; physical connector reuse does not justify sharing them. + +## Design Success Criteria + +The refactor is successful only if all of these are true: + +1. unchanged eligible connector rows generate zero new physical activation/configure work on unrelated material reload; +2. changed/unusable connectors still get fresh physical incarnations before publication; +3. candidate rollback cannot close or reconfigure the last-good generation's shared physical resource; +4. final generation lease owns the moment of physical cleanup exactly once; +5. stale invalidation cannot poison a replacement incarnation; +6. process shutdown remains one coordinated order with pool before host/artifacts/staging; +7. generation-local derived runtime state remains separate; +8. no request-path lookup/lock is added; +9. no public config/ABI or generic runtime framework is added; +10. the implementation diff makes the expensive lifecycle behavior simpler to reason about at high connector cardinality rather than introducing more concepts than it removes. From d968d5e4da8c556e2fc3ab8d97739c0129141a40 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:57:04 +0200 Subject: [PATCH 07/12] spec: validate backend reconciliation design --- .../design-review.md | 176 ++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 .kiro/specs/backend-connector-resource-reconciliation/design-review.md diff --git a/.kiro/specs/backend-connector-resource-reconciliation/design-review.md b/.kiro/specs/backend-connector-resource-reconciliation/design-review.md new file mode 100644 index 00000000..11d869b2 --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/design-review.md @@ -0,0 +1,176 @@ +# Brownfield Design Validation + +## Verdict + +**GO after one requirements tightening.** The selected design fits the current Go-LIP runtime and connector boundaries and provides a focused path from O(N) physical connector reconstruction toward O(K changed/unusable physical reconstruction) while retaining immutable generations. The design itself already contains the necessary candidate-isolation treatment, but that safety property must also be promoted into normative requirements before task generation. + +No broader Cordis runtime, processhost redesign, public ABI, or dynamic discovery work is justified. + +## Validation Checklist + +### Generation consistency boundary — PASS + +The design preserves `GenerationRuntime` as the immutable request-plane unit. Reconciliation occurs only while constructing the backend resources used to build a new generation. Published generations are never reconfigured, rebound, or live-migrated. + +Changed identity still creates a distinct physical resource before publication; removed/disabled identity is retained only by old generation leases. This preserves existing backend recomposition semantics. + +### `ResourceLedger` authority — PASS + +Each generation receives a fresh lease-release cleanup and existing `buildBackends` continues to transfer that cleanup into the generation `ResourceLedger`. No new generation cleanup engine exists. + +The physical connector cleanup is pool-owned and therefore cannot be copied into two generation ledgers. This is the central correctness requirement for sharing. + +### Process ownership — PASS + +The pool is process-scoped but is not a second process supervisor. `processhost.Host` remains responsible for launch, process slots, authenticated IPC, peer identity, configured host instance ownership, generation invalidation and process-tree cleanup. + +The pool only knows semantic identity, physical incarnation, backend value, physical cleanup, and dependent refcount. + +### Pre-`ProcessServices` factory capture — PASS + +Production discovered factories are installed before `ProcessServices` exists. The design correctly creates the pool beside `processhost.Host` during discovered-install preparation, captures it lexically in factory closures, then transfers pool lifetime into `ProcessServices`. + +This avoids a service locator, global runtime registry, setter race, or post-publication dependency mutation. + +### Teardown ordering — PASS + +Current staging/artifact/host ownership has strict reverse-order requirements. Registering pool ownership after host ownership yields the needed relative shutdown order: + +`pool -> processhost.Host -> verified artifacts -> staging`. + +The design also requires equivalent bootstrap-failure ordering before ownership transfer. + +### Physical identity completeness — PASS + +The design rejects direct use of the narrower `BackendStateIdentity` and derives a separate key at the physical construction/configure choke point. Artifact digest, logical instance/factory identity, opaque Configure bytes, process model, runtime policy and secret fingerprint are covered. + +The explicit DTO-drift gate is important: identity correctness must fail closed as connector configure inputs evolve. + +### Secret handling — PASS + +Secret values are only locally length-framed and hashed; no plaintext or secret-derived diagnostic value is exposed. Identity digests themselves should remain private because equality fingerprints can still be sensitive metadata. + +### Incarnation-safe invalidation — PASS + +The design correctly distinguishes desired semantic identity from a concrete process/session incarnation. Invalidation detaches the exact incarnation before/with existing processhost invalidation, future Acquire builds a new incarnation, and stale callbacks cannot remove a newer current entry. + +Existing generations are not silently rebound to the replacement. + +### `shared_artifact` exclusion — PASS + +The design does not mix two independent sharing problems. Current `shared_artifact` process sharing and restart-required overlap gates remain unchanged. First-pass reconciliation is discovered `per_instance` only. + +### Built-in backend exclusion — PASS + +There is no evidence that cheap in-process builtins justify process-level lease machinery. Keeping them generation-owned limits complexity and avoids turning the selected connector optimization into a generic backend runtime. + +### Request hot path — PASS + +Resource lookup/refcounting occurs only during generation construction/retirement. Executor request calls use already-captured backend functions and introduce no pool mutex or identity hash on normal inference traffic. + +### Dynamic inventory/model views — PASS with explicit constraint + +The design correctly keeps `modelregistry.Runtime`, inventory projection, routing/model views and refresh-loop ownership generation-local. A reused physical session may therefore receive concurrent `Resolve`/`ListModels` calls from overlapping generations. + +This is compatible with the existing configured-instance abstraction because one instance already serves runtime execution and metadata calls, but implementation tests must cover overlapping generation metadata access/race safety. Do **not** solve this by moving the model registry into the pool. + +### Candidate last-good isolation — PASS after normative tightening + +This is the most important design-validation finding. + +Fresh physical processes currently give candidate preparation strong failure-domain isolation. Sharing intentionally reduces physical failure-domain independence for an **unchanged** connector. The design handles the acceptable boundary correctly: + +- a candidate reuse hit performs no Configure/Start/Stop/Close on the shared resource; +- candidate rejection/rollback releases only its lease; +- generation-local model preparation may use query-shaped `Resolve`/`ListModels` calls; +- candidate rollback never invalidates the resource merely because the candidate failed; +- any future mutating generation-preparation lifecycle makes the resource non-shareable until separately proven safe. + +The current requirements imply this through immutability/non-interference but do not state it strongly enough. Add a normative clause requiring **query-only candidate preparation on a reused physical resource** and fallback to generation-local physical construction if a mutating lifecycle/preflight becomes necessary. + +This is not a claim that an external connector process can never crash while queried. Existing process failure remains possible. The preservation requirement is that the candidate lifecycle itself cannot intentionally mutate/close/reconfigure the last-good shared resource. + +### Adapter cleanup shape — PASS with regression lock + +Current external adapter physical session cleanup lives in the returned BuildResult cleanup rather than an ordinary backend `Close` hook. This makes the selected lease model viable. + +Implementation must add a characterization/architecture test so future external adapter lifecycle hooks cannot silently create a physical-cleanup bypass around the pool. + +### Concurrency state machine — PASS + +The proposed pending-entry protocol avoids a subtle refcount race that a naive `singleflight.Do` result handoff can create: the first caller cannot close a just-built resource before waiters have formally acquired it. + +No physical build/cleanup under the mutex, exact-incarnation invalidation, cancellation-safe waiting, and close/pending-builder synchronization are appropriate. + +### Failure caching — PASS + +Failed physical build is not permanently cached. This preserves current retry-on-later-attempt behavior and avoids introducing hidden negative-cache policy. + +### No idle cache — PASS + +Closing on final generation lease keeps the scope tightly aligned with reload overlap. TTL/eviction/process-pressure policy would be speculative and materially increase operational complexity. + +### Backend-plugin ABI/security — PASS + +The design changes no public ABI and retains verified artifact binding, secure local IPC, peer auth before Configure/secrets, environment restrictions and process-tree cleanup. The pool does not parse provider-specific YAML. + +### ROI gate — PASS + +Deterministic operation counts are a stronger primary gate than wall-clock thresholds. The 100-enabled-connector harness directly proves whether the expensive O(N) activation/configure wave exists and whether implementation removes it. Benchstat remains supporting evidence. + +## Required Requirements Correction + +Add normative requirements equivalent to: + +1. On a reuse hit, candidate preparation shall not invoke Configure, Start, Stop, Close, mutating preflight, or another generation-local mutation on the shared physical connector; query-shaped Resolve/ListModels access may remain generation-local. +2. If the candidate compiler or external adapter later requires such a mutating preparation action, the resource is non-shareable and must use current isolated physical construction until a separate design proves safe reuse. +3. Overlapping-generation metadata access on a reused physical configured instance requires race/conformance coverage. + +The current `design.md` already states these constraints; only `requirements.md` needs tightening. + +## Design-to-Requirement Trace + +| Requirement | Design coverage | +|---|---| +| R1 scale evidence | 100-instance count harness, mixed-K matrix, supporting benchmarks | +| R2 narrow boundary | discovered per-instance only; private pool above processhost | +| R3 physical identity | explicit configure-input identity and fail-closed DTO drift gate | +| R4 lease contract | pending/live entries, per-generation lease release, final physical cleanup | +| R5 generation semantics | unchanged/changed/remove/rollback flows; derived state stays generation-local | +| R6 incarnation invalidation | exact entry token, detach, fresh same-key incarnation, no live substitution | +| R7 cleanup/shutdown | pool-before-host/artifacts/staging, one physical cleanup owner | +| R8 non-interference | no request path lookup, ABI/security/routing/billing unchanged, query-only candidate rule | +| R9 TDD/architecture | RED identity/pool/scale tests, race/goleak, anti-container gates | + +## Simplification Review + +The design deliberately rejects the following tempting expansions: + +1. **No generic resource manager.** One connector-specific private pool is enough. +2. **No processhost generation awareness.** The host remains a physical supervisor. +3. **No `BackendStateIdentity` semantic overloading.** Observation-state compatibility remains independent. +4. **No idle cache.** Reuse exists only while generations overlap. +5. **No public feature flag.** Unsafe resources fall back rather than exposing lifecycle internals. +6. **No shared model registry.** Generation consistency remains coarse-grained where it provides value. +7. **No dynamic plugin reconciliation.** Startup-fixed trust/discovery remains unchanged. +8. **No Cordis requires/provides graph or fibers.** There is no current service-dependency problem requiring them. + +## Implementation Risks to Pin With Tests + +- a Configure-time DTO field added without identity treatment; +- two candidate builders constructing the same absent identity twice; +- a candidate rollback closing the active generation's resource; +- final release racing a new Acquire and handing out a closing entry; +- old incarnation invalidation deleting a newer incarnation; +- pool Close returning while a pending builder can still publish/use a closing host; +- external adapter gaining `Close`/Start/Stop semantics that bypass lease cleanup; +- dynamic inventory refresh races across overlapping generations; +- changed config/artifact/secret/policy incorrectly hitting reuse; +- removed backend closing before retained old generation drains; +- pool shutdown running after host/artifact teardown; +- request execution accidentally consulting the pool; +- architecture growing into a generic registry/container. + +## Final Gate + +Design validation is **GO** once the candidate query-only/fallback rule is copied into normative requirements. After that correction, task decomposition may proceed without further architecture changes. From 8456909a382024fd4a128cfae3a85eeb40ec5a81 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:57:46 +0200 Subject: [PATCH 08/12] spec: tighten candidate isolation after design validation --- .../requirements.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.kiro/specs/backend-connector-resource-reconciliation/requirements.md b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md index 7004840d..0a0809a6 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/requirements.md +++ b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md @@ -74,6 +74,9 @@ The first implementation is intentionally narrow. It targets discovered executab 5.6. Existing generation-owned derived state—including executor maps/views, routing views, model-registry runtime, model catalog, feature composition, policy state, and generation lifecycle context—shall remain generation-owned and shall not be moved into the connector resource pool. 5.7. Existing no-drop, retained-generation, old-stream, and last-good reload guarantees shall remain unchanged. 5.8. A reused configured connector may contribute the same underlying immutable backend/session functions to multiple generation-local executor maps, but each generation shall rebuild its own projections, inventories, routing and model views using the same existing generation compiler sequence. +5.9. On a reuse hit, candidate preparation shall not invoke `Configure`, `Start`, `Stop`, `Close`, a mutating preflight, or another generation-local mutation on the shared physical connector. Candidate rejection or rollback shall release only the candidate lease and shall never invalidate the shared resource merely because the candidate was rejected. +5.10. Query-shaped metadata operations already represented by the backend-plugin contract, such as `Resolve` and `ListModels`, may remain part of generation-local preparation/refresh against a reused configured instance. This does not permit hidden reconfiguration by the host. +5.11. If future generation preparation or an external adapter requires a mutating lifecycle action against the configured connector, that resource path shall become non-shareable and use isolated physical construction until a separate design proves safe cross-generation reuse. ## Requirement 6: Invalidation and Resource Incarnations @@ -107,6 +110,8 @@ The first implementation is intentionally narrow. It targets discovered executab 8.5. Existing backend-plugin security invariants—verified artifact binding, secure local IPC, peer authentication before secrets/configure, environment restrictions, and process-tree cleanup—shall remain unchanged. 8.6. Old and new generation behavior shall remain observationally equivalent to the current overlap model for unchanged resources except that unnecessary physical connector reconstruction is removed. 8.7. Connector-specific configuration parsing remains inside the connector. The host identity may fingerprint opaque configure bytes but shall not learn provider-specific configuration schemas merely to decide reuse. +8.8. Reusing one configured instance across overlapping generations shall not imply that all external process failures are isolated per generation; however, the optimization shall not add candidate-owned mutation that creates a new failure path for the last-good generation. Existing connector/process failure and invalidation semantics remain authoritative. +8.9. Overlapping generation access to query-shaped metadata operations on a reused configured instance shall be covered by race/conformance tests. The implementation shall not move the model registry into the pool merely to avoid concurrent metadata calls. ## Requirement 9: TDD, Concurrency, and Architecture Gates @@ -118,3 +123,4 @@ The first implementation is intentionally narrow. It targets discovered executab 9.6. Architecture tests shall reject a generic service registry/container API, request-time lookup surface, public reusable-resource framework, provider-specific switch, or migration of `processhost` supervision responsibility. 9.7. Repository quality, focused executable-plugin conformance/security tests, and existing reload no-drop/rollback tests shall remain green. 9.8. Final implementation review shall remove unused abstraction layers and preserve the smallest private design that satisfies the deterministic scale and correctness gates. +9.9. Add a regression gate proving that pooled external adapters expose no generation-owned physical `Close`/`Start`/`Stop` bypass and that candidate preparation on a reuse hit is query-only with respect to the configured physical resource. From db5b5cb9ef867605e66bb59609193d2df5ef1fd7 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:58:49 +0200 Subject: [PATCH 09/12] spec: add backend reconciliation implementation plan --- .../tasks.md | 241 ++++++++++++++++++ 1 file changed, 241 insertions(+) create mode 100644 .kiro/specs/backend-connector-resource-reconciliation/tasks.md diff --git a/.kiro/specs/backend-connector-resource-reconciliation/tasks.md b/.kiro/specs/backend-connector-resource-reconciliation/tasks.md new file mode 100644 index 00000000..7650f0b4 --- /dev/null +++ b/.kiro/specs/backend-connector-resource-reconciliation/tasks.md @@ -0,0 +1,241 @@ +# Implementation Plan + +## Execution Rules + +- Follow TDD: characterization/RED tests and contract gates precede production behavior changes. +- Keep every task independently reviewable and limited to at most five concrete actions. +- Preserve existing public APIs, backend-plugin ABI, configuration schema, immutable generation model, `ResourceLedger`, and `processhost` ownership. +- Prefer deletion/replacement of old per-generation physical-cleanup plumbing at the touched seam over layering a second cleanup path. +- Do not broaden scope to built-in backends, `shared_artifact`, generic resource management, dynamic discovery, or request-time lookup. + +## Phase 1 — Establish Scale, Identity, and Lifetime RED Gates + +### Task 1.1 — Build the high-cardinality executable-connector reload harness + +- Add a deterministic runtimebundle test fixture with at least 100 enabled synthetic discovered `per_instance` connector instances, unique logical IDs/prefixes, and no external credentials. +- Instrument physical factory/build, processhost activation/launch, Configure, physical cleanup, and later lease acquire/release counts without changing production behavior. +- Characterize the current unrelated-material-reload baseline: unchanged enabled connector rows are physically reconstructed for the candidate while the old generation remains retained. +- Add RED target assertions for unchanged reload (`0` new physical builds/activations/configures) and mixed reload (`K` changed identities -> `K` physical replacements). +- Add/extend a focused benchmark fixture for candidate compilation at representative connector cardinalities without using wall-clock thresholds as correctness assertions. + +_Requirements: 1.1–1.7, 2.2, 9.1, 9.3_ + +_Validation: current production behavior is characterized deterministically; target reuse assertions are RED before implementation._ + +_Design: Scale and ROI Evidence Design; Deterministic 100-connector harness._ + +### Task 1.2 — Lock physical resource identity requirements with RED tests + +- Add table-driven RED tests proving identity equality only when logical instance, factory, artifact digest, process model, opaque Configure bytes, effective runtime policy, and secret fingerprint are compatible. +- Prove config, artifact, credential/secret, and runtime-policy changes produce distinct identities while secret plaintext never appears in identity string/debug/error output. +- Add a contract gate that forces deliberate identity treatment when the external configure-time physical input/`RuntimePolicy` surface changes. +- Prove `BackendStateIdentity` compatibility alone cannot authorize physical connector reuse. +- Add fallback tests for an input explicitly marked non-shareable/incompletely representable. + +_Requirements: 3.1–3.11, 8.1, 8.7, 9.1, 9.4_ + +_Validation: identity tests fail before the new private identity builder exists and protect DTO drift._ + +_Design: Physical Resource Identity; DTO Drift Gate; Fail-Closed Eligibility._ + +### Task 1.3 — Specify the lease/refcount/incarnation state machine with RED tests + +- Add RED tests for first Acquire, reuse Acquire, independent idempotent releases, and physical cleanup only on final release with no idle retention. +- Add concurrent absent-Acquire tests proving one physical build and one current incarnation with multiple leases; include waiter cancellation without canceling another caller's build. +- Add invalidation tests proving exact-incarnation detach, fresh same-semantic-key rebuild, and stale old-incarnation invalidation cannot evict the replacement. +- Add races for final Release versus new Acquire and pool Close versus a pending physical build; no caller may receive a closing/invalid entry. +- Add failed-build tests proving no permanent negative cache, no leaked ref/entry, and successful later retry. + +_Requirements: 4.1–4.10, 6.1–6.8, 7.5, 7.8, 9.1–9.2_ + +_Validation: state-machine tests are RED against the absent pool and define exactly-once cleanup/concurrency behavior._ + +_Design: Resource Pool State Machine; Concurrency Design; Invalidation and Incarnations._ + +### Task 1.4 — Lock candidate isolation and process teardown ordering with RED tests + +- Add a candidate-rollback test where active and candidate generations share one synthetic physical resource; rollback must release only the candidate lease and leave active execution/query behavior available. +- Add an external-adapter characterization gate proving a pooled backend has no generation-owned physical `Close`/`Start`/`Stop` bypass around lease cleanup. +- Add overlapping-generation `Resolve`/`ListModels` metadata access coverage under race instrumentation while keeping model-registry/runtime projections generation-local. +- Add ProcessServices/bootstrap ownership-order tests requiring pool close before processhost, artifacts, and staging on success and partial startup failure. +- Add a non-shareable fallback test for any candidate path that requires mutating Configure/Start/Stop/Close/preflight preparation on an existing physical connector. + +_Requirements: 2.8–2.9, 5.4–5.11, 7.1–7.9, 8.2, 8.8–8.9, 9.9_ + +_Validation: RED tests capture the last-good isolation boundary and exact pool → host → artifacts → staging dependency order._ + +_Design: Candidate Preparation and Last-Good Isolation; Process Shutdown; No cleanup bypass._ + +## Phase 2 — Implement the Minimal Private Identity and Reconciliation Owner + +### Task 2.1 — Implement explicit fail-closed physical resource identity + +- Add one package-private identity builder at the discovered physical construction/configure choke point using domain-separated, length-framed SHA-256 input. +- Canonically project every current `RuntimePolicy` field and deterministic opaque Configure bytes; fingerprint any non-empty `SecretBundle` with sorted, length-framed names/values without retaining plaintext. +- Include logical instance/factory identity, exact verified artifact digest, and process model; document any process-stable omitted input protected by tests. +- Return a shareability decision so incomplete/unsupported identity input falls back to current physical construction rather than producing an unsafe key. +- Make all Task 1.2 identity and privacy/DTO-drift tests green without adding production reflection or a public identity type. + +_Requirements: 3.1–3.11, 8.1, 8.7_ + +_Validation: focused identity tests green; secret leak scans/errors contain no raw secret material._ + +_Design: Physical Resource Identity; Canonical Fingerprinting._ + +### Task 2.2 — Implement the connector-specific resource pool and lease state machine + +- Add one package-private `backendResourcePool`/entry/lease implementation with pending/live/detached state, exact physical incarnation tokens, refcounts, and a closing state. +- Serialize same-identity physical construction without holding the pool mutex across build/configure; make canceled waiters independent of the builder/other dependents. +- Return the immutable backend value plus a fresh idempotent generation lease-release cleanup; retain physical cleanup only on the entry and run it exactly once after final release. +- Implement exact-incarnation invalidation/detach and no-negative-cache build failure semantics, with all physical cleanup outside the lock. +- Implement idempotent Close that rejects new Acquire, joins pending builders, and fail-safe cleans residual entries before returning. + +_Requirements: 4.1–4.10, 6.1–6.8, 7.5, 7.8_ + +_Validation: Task 1.3 tests green under normal and race execution; no physical operation executes while pool mutex is held._ + +_Design: Private Types; Resource Pool State Machine; Concurrency Design._ + +### Task 2.3 — Transfer pool ownership through the existing process construction path + +- Create the pool beside `processhost.Host` during discovered-install preparation and make eligible discovered factory closures capture it directly without global lookup/setter wiring. +- Extend the private discovered-install/process-build ownership bundle so pool lifetime transfers into `ProcessServices` while direct test/install paths can remain explicit and non-global. +- Register process cleanup so reverse shutdown orders pool before processhost, verified artifacts, and staging; preserve existing ordering of unrelated ProcessServices resources. +- Mirror the same dependency order on every pre-transfer/bootstrap failure path and prevent double cleanup after successful ownership transfer. +- Keep the pool connector-specific/package-private and make Task 1.4 process ownership-order tests green. + +_Requirements: 2.1, 2.6–2.9, 7.2–7.6, 7.9_ + +_Validation: ownership tests prove exactly-once transfer and pool → host → artifacts → staging teardown on success/failure._ + +_Design: Construction and Ownership Timing; Process Shutdown._ + +## Phase 3 — Integrate Reuse Into Discovered `per_instance` Generation Construction + +### Task 3.1 — Split physical construction from lease acquisition at the discovered factory seam + +- Refactor the existing discovered backend builder into a narrow preparation/identity step plus the current physical Activate/Configure/adapter-build path without provider-specific branches. +- Preserve unique host activation IDs for every **new** `per_instance` physical incarnation; do not change `processhost.OwnershipKey` or host instance semantics. +- Route only eligible discovered overlap-safe `per_instance` resources through pool Acquire; leave builtins, `shared_artifact`, and non-shareable resources on current generation-local construction. +- Make pool hits return the existing backend functional value plus lease cleanup, while pool misses consume the underlying physical cleanup as entry ownership. +- Keep `buildBackends`/`ResourceLedger` as the generation cleanup transfer authority and make unchanged/mixed high-cardinality RED assertions green. + +_Requirements: 1.4–1.5, 2.2–2.7, 4.2–4.10, 5.1–5.8_ + +_Validation: unchanged eligible reload performs zero physical reconstruction; K changed identities produce exactly K new physical builds._ + +_Design: Physical Construction Integration; Current path retained as the builder; Pool return shape._ + +### Task 3.2 — Bind processhost failure invalidation to exact pooled incarnations + +- Wrap the existing adapter invalidation callback for newly built pooled resources so it first detaches the exact pool incarnation from future reuse and then delegates to `processhost.InvalidateProcessGeneration`. +- Preserve existing processhost physical reap/recovery behavior and existing cleanup-error normalization after an invalidated process is already gone. +- Ensure invalidation does not decrement generation leases or live-substitute a replacement under an existing generation. +- Prove a later same-config candidate builds one fresh physical incarnation and that an old delayed callback cannot invalidate the new incarnation. +- Keep non-pooled and `shared_artifact` invalidation behavior byte-for-byte/semantically equivalent to current behavior. + +_Requirements: 6.1–6.8, 7.7, 8.3, 8.5_ + +_Validation: invalidation/restart tests green; current processhost invalidation tests remain green._ + +_Design: Invalidation flow; No live substitution._ + +### Task 3.3 — Preserve generation-local model/routing/policy composition on reuse hits + +- Keep building a new generation-local `BackendInventory` slice, model-registry runtime, model/catalog snapshot, executor/routing/policy views, HTTP handler, generation lifecycle context, and `ResourceLedger` from leased backends. +- Verify unchanged pooled resources can serve overlapping generations' metadata queries/executions without moving refresh loops or model state into the pool. +- Verify changed same-ID, remove/disable, retained old stream/async work, and candidate-failure rollback retain the exact current behavioral contracts. +- Ensure candidate reuse hits perform no Configure/Start/Stop/Close/mutating preflight on the physical resource; automatically bypass reuse if that condition is not satisfied. +- Keep all canonical request/event, streaming, failover, cancellation, accounting, billing and token-counting code paths unchanged. + +_Requirements: 5.1–5.11, 8.2–8.9, 9.9_ + +_Validation: existing backend recomposition/no-drop tests plus new query-only/cross-generation tests green._ + +_Design: Generation-Local Derived State; Candidate Preparation and Last-Good Isolation._ + +### Task 3.4 — Add architecture fences against scope creep and cleanup bypass + +- Add architecture tests proving the pool remains private to runtime composition and is not imported/reached from request execution, public SDKs, connectors, or provider-specific packages. +- Reject generic service/container vocabulary or a reusable keyed runtime registry API introduced to support this feature. +- Lock `processhost.Host` as the only physical process/IPC supervisor and reject duplicate launch/peer/process-tree logic in the pool. +- Lock pooled external backend lifecycle so generation cleanup is a lease release rather than an alternate physical `Close`/Start/Stop path. +- Assert no public YAML/manifest/ABI option is added for this internal optimization. + +_Requirements: 2.1, 2.5–2.9, 4.9–4.10, 8.3–8.5, 9.6, 9.9_ + +_Validation: architecture gates fail on representative forbidden fixtures and pass the intended private design._ + +_Design: Boundary Commitments; Rejected Alternatives; No cleanup bypass._ + +## Phase 4 — Prove ROI, Correctness, and Final Simplicity + +### Task 4.1 — Certify the high-cardinality reconciliation matrix + +- Run the 100-enabled-connector harness for unchanged, one-changed, K-changed, remove/disable, candidate rollback, and one-invalidated-then-rebuild scenarios. +- Assert physical build/activation/Configure counts are `0` for unchanged reuse hits and proportional only to changed/unusable identities. +- Assert peak synthetic physical live-resource count follows active resources plus changed candidate replacements rather than duplicating every unchanged connector. +- Assert candidate rollback of reuse hits performs no physical cleanup of resources retained by the active generation and newly built candidate-only resources close at final release. +- Record the deterministic before/after operation-count evidence in implementation/PR evidence. + +_Requirements: 1.1–1.7, 4.5–4.7, 5.2–5.5, 6.3, 7.1_ + +_Validation: all count-based ROI/correctness gates green independent of wall-clock variance._ + +_Design: Required scenarios; Design Success Criteria 1–5._ + +### Task 4.2 — Run concurrency, leak, security, conformance, and reload regression gates + +- Run focused `-race` coverage for pool Acquire/Release/Invalidate/Close plus overlapping generation metadata/execution paths and run goleak coverage for pending-builder/shutdown cases. +- Run processhost activation/cleanup/invalidation and executable backend-plugin security/conformance suites on supported local/CI profiles. +- Run runtimebundle ResourceLedger, backend recomposition, discovered overlap/restart-required, candidate rollback, generation retention, and reload no-drop/last-good tests. +- Run repository formatting, vet/lint/quality checks and existing architecture boundaries; fix root causes without weakening assertions or skips. +- Verify no secret/config digest/raw YAML leakage was added to logs, metrics, statuses, errors, or public DTOs. + +_Requirements: 7.7–7.8, 8.3–8.9, 9.2–9.7, 9.9_ + +_Validation: race/goleak/security/conformance/reload/quality suites green; no weakened existing gate._ + +_Design: Testing Strategy; Risks & Mitigations._ + +### Task 4.3 — Record supporting benchmark evidence without overclaiming request performance + +- Run the high-cardinality candidate compilation benchmark before/after with stable fixture inputs and use benchstat when comparable samples are available. +- Report candidate build time/allocations and available synthetic/native process-launch/resource observations, clearly separating deterministic counts from noisy platform metrics. +- Confirm normal request execution benchmarks show no new pool lookup/lock and no material regression attributable to the feature. +- Document that the claimed gain is reload physical-resource churn/peak overlap reduction, not inference throughput or token latency. +- Keep timing data informational unless a stable repository performance budget already governs the measured path. + +_Requirements: 1.2, 1.6, 8.4_ + +_Validation: evidence supports O(N physical build) → O(K physical build) structural claim without unstable timing gates._ + +_Design: Supporting benchmark evidence; Scale and ROI Evidence Design._ + +### Task 4.4 — Perform the final simplification and scope gate + +- Review production diff for duplicate ownership stacks, generic resource/container abstractions, unnecessary wrappers, public knobs, or request-path coupling and remove them. +- Confirm builtins, `shared_artifact`, discovery/watchers, model-registry sharing, and dynamic plugin lifecycle remain outside scope. +- Confirm `ProcessServices`, `ResourceLedger`, `processhost.Host`, runtimehost generation leases, and unique host activation semantics remain the same authorities after the refactor. +- Compare implementation complexity against deterministic scale evidence; if the count target is not met or the design requires broad runtime machinery, revert/re-scope instead of shipping speculative infrastructure. +- Update implementation evidence/docs only where needed to describe the internal optimization and its preserved operational semantics. + +_Requirements: 1.7, 2.1–2.9, 5.6–5.7, 8.3–8.6, 9.6–9.8_ + +_Validation: final diff is narrowly connector-lifecycle focused, evidence-backed, and contains no new general runtime concept._ + +_Design: Simplification Review; Design Success Criteria._ + +## Requirement Coverage Matrix + +| Requirement | Primary tasks | +|---|---| +| R1 Evidence-first scale justification | 1.1, 3.1, 4.1, 4.3, 4.4 | +| R2 Narrow eligibility/ownership boundary | 1.4, 2.3, 3.1, 3.4, 4.4 | +| R3 Physical resource identity | 1.2, 2.1 | +| R4 Acquire/lease contract | 1.3, 2.2, 3.1, 4.1 | +| R5 Immutable generation semantics | 1.4, 3.1, 3.3, 4.1 | +| R6 Invalidation/incarnations | 1.3, 2.2, 3.2, 4.1 | +| R7 Cleanup/shutdown/errors | 1.4, 2.2, 2.3, 4.2 | +| R8 Connector non-interference | 1.4, 2.1, 3.3, 3.4, 4.2–4.3 | +| R9 TDD/concurrency/architecture gates | Phase 1, 3.4, 4.2, 4.4 | From 0884de38171eba63cbdf2dcb96afae9db85dceb8 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:59:37 +0200 Subject: [PATCH 10/12] spec: close backend reconciliation design validation --- .../design-review.md | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/.kiro/specs/backend-connector-resource-reconciliation/design-review.md b/.kiro/specs/backend-connector-resource-reconciliation/design-review.md index 11d869b2..7f1e1e81 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/design-review.md +++ b/.kiro/specs/backend-connector-resource-reconciliation/design-review.md @@ -2,7 +2,9 @@ ## Verdict -**GO after one requirements tightening.** The selected design fits the current Go-LIP runtime and connector boundaries and provides a focused path from O(N) physical connector reconstruction toward O(K changed/unusable physical reconstruction) while retaining immutable generations. The design itself already contains the necessary candidate-isolation treatment, but that safety property must also be promoted into normative requirements before task generation. +**GO.** The selected design fits the current Go-LIP runtime and connector boundaries and provides a focused path from O(N) physical connector reconstruction toward O(K changed/unusable physical reconstruction) while retaining immutable generations. + +The validation identified one normative gap around candidate failure-domain isolation. That correction has been applied to `requirements.md` before task generation: reuse hits are query-only with respect to the configured physical connector, mutating generation-preparation lifecycle requires isolated fallback, and overlapping metadata access is race/conformance tested. No broader Cordis runtime, processhost redesign, public ABI, or dynamic discovery work is justified. @@ -74,19 +76,17 @@ The design correctly keeps `modelregistry.Runtime`, inventory projection, routin This is compatible with the existing configured-instance abstraction because one instance already serves runtime execution and metadata calls, but implementation tests must cover overlapping generation metadata access/race safety. Do **not** solve this by moving the model registry into the pool. -### Candidate last-good isolation — PASS after normative tightening +### Candidate last-good isolation — PASS; normative correction applied -This is the most important design-validation finding. +This was the most important design-validation finding. -Fresh physical processes currently give candidate preparation strong failure-domain isolation. Sharing intentionally reduces physical failure-domain independence for an **unchanged** connector. The design handles the acceptable boundary correctly: +Fresh physical processes currently give candidate preparation strong failure-domain isolation. Sharing intentionally reduces physical failure-domain independence for an **unchanged** connector. The accepted boundary is now normative in Requirements 5.9–5.11, 8.8–8.9, and 9.9: -- a candidate reuse hit performs no Configure/Start/Stop/Close on the shared resource; -- candidate rejection/rollback releases only its lease; +- a candidate reuse hit performs no Configure/Start/Stop/Close/mutating preflight on the shared resource; +- candidate rejection/rollback releases only its lease and does not invalidate the resource merely because the candidate failed; - generation-local model preparation may use query-shaped `Resolve`/`ListModels` calls; -- candidate rollback never invalidates the resource merely because the candidate failed; -- any future mutating generation-preparation lifecycle makes the resource non-shareable until separately proven safe. - -The current requirements imply this through immutability/non-interference but do not state it strongly enough. Add a normative clause requiring **query-only candidate preparation on a reused physical resource** and fallback to generation-local physical construction if a mutating lifecycle/preflight becomes necessary. +- any future mutating generation-preparation lifecycle makes the resource non-shareable until separately proven safe; +- overlapping metadata access receives race/conformance coverage. This is not a claim that an external connector process can never crash while queried. Existing process failure remains possible. The preservation requirement is that the candidate lifecycle itself cannot intentionally mutate/close/reconfigure the last-good shared resource. @@ -118,15 +118,15 @@ The design changes no public ABI and retains verified artifact binding, secure l Deterministic operation counts are a stronger primary gate than wall-clock thresholds. The 100-enabled-connector harness directly proves whether the expensive O(N) activation/configure wave exists and whether implementation removes it. Benchstat remains supporting evidence. -## Required Requirements Correction +## Requirements Correction Applied -Add normative requirements equivalent to: +The design review required three safety clauses, all now present before task generation: -1. On a reuse hit, candidate preparation shall not invoke Configure, Start, Stop, Close, mutating preflight, or another generation-local mutation on the shared physical connector; query-shaped Resolve/ListModels access may remain generation-local. -2. If the candidate compiler or external adapter later requires such a mutating preparation action, the resource is non-shareable and must use current isolated physical construction until a separate design proves safe reuse. -3. Overlapping-generation metadata access on a reused physical configured instance requires race/conformance coverage. +1. **Query-only candidate reuse:** Requirements 5.9–5.10 prohibit Configure/Start/Stop/Close/mutating preparation on a reused physical connector while allowing query-shaped Resolve/ListModels access. +2. **Fail-closed future lifecycle:** Requirement 5.11 requires isolated construction if future candidate preparation needs a mutating connector lifecycle action. +3. **Shared metadata concurrency evidence:** Requirements 8.9 and 9.9 require overlapping-generation race/conformance coverage and lock out a generation-owned physical cleanup bypass. -The current `design.md` already states these constraints; only `requirements.md` needs tightening. +No design rewrite was required because `design.md` already expressed these constraints; the validation correction made them normative. ## Design-to-Requirement Trace @@ -136,11 +136,11 @@ The current `design.md` already states these constraints; only `requirements.md` | R2 narrow boundary | discovered per-instance only; private pool above processhost | | R3 physical identity | explicit configure-input identity and fail-closed DTO drift gate | | R4 lease contract | pending/live entries, per-generation lease release, final physical cleanup | -| R5 generation semantics | unchanged/changed/remove/rollback flows; derived state stays generation-local | +| R5 generation semantics | unchanged/changed/remove/rollback flows; derived state stays generation-local; query-only candidate reuse | | R6 incarnation invalidation | exact entry token, detach, fresh same-key incarnation, no live substitution | | R7 cleanup/shutdown | pool-before-host/artifacts/staging, one physical cleanup owner | | R8 non-interference | no request path lookup, ABI/security/routing/billing unchanged, query-only candidate rule | -| R9 TDD/architecture | RED identity/pool/scale tests, race/goleak, anti-container gates | +| R9 TDD/architecture | RED identity/pool/scale tests, race/goleak, adapter lifecycle and anti-container gates | ## Simplification Review @@ -173,4 +173,4 @@ The design deliberately rejects the following tempting expansions: ## Final Gate -Design validation is **GO** once the candidate query-only/fallback rule is copied into normative requirements. After that correction, task decomposition may proceed without further architecture changes. +**GO.** Requirements, design, and brownfield validation are aligned. Task decomposition may proceed under the explicit evidence-first and query-only candidate-reuse gates; no further architecture correction is required before implementation. From 6dc56201b54b016f3516c660523266f8c9c15ca5 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:59:52 +0200 Subject: [PATCH 11/12] spec: finalize backend connector reconciliation SDD --- .../spec.json | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.kiro/specs/backend-connector-resource-reconciliation/spec.json b/.kiro/specs/backend-connector-resource-reconciliation/spec.json index 913adc9a..6244e6e8 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/spec.json +++ b/.kiro/specs/backend-connector-resource-reconciliation/spec.json @@ -1,23 +1,23 @@ { "feature_name": "backend-connector-resource-reconciliation", "created_at": "2026-08-17T00:42:00+02:00", - "updated_at": "2026-08-17T00:42:00+02:00", + "updated_at": "2026-08-17T00:59:00+02:00", "language": "en", - "phase": "initialized", + "phase": "tasks-generated", "approvals": { "requirements": { - "generated": false, + "generated": true, "approved": false }, "design": { - "generated": false, + "generated": true, "approved": false }, "tasks": { - "generated": false, + "generated": true, "approved": false } }, "ready_for_implementation": false, - "project_description": "Reduce cross-generation churn for expensive executable backend connectors by reusing unchanged configured per-instance connector resources through semantic identity and process-scoped reference-counted generation leases. Preserve immutable GenerationRuntime publication, ResourceLedger rollback/retirement, processhost supervision, the executable backend-plugin ABI, startup-fixed discovery, routing/streaming/accounting semantics, and last-good reload. Scope the first implementation to eligible discovered per_instance connectors only; add no DI container, service locator, reactive component graph, live plugin watcher, generic resource framework, or request-hot-path lookup. Require deterministic high-cardinality evidence before and after implementation so the optimization remains justified by connector-scale behavior rather than Cordis-inspired abstraction for its own sake." + "project_description": "Reduce cross-generation churn for expensive executable backend connectors by reusing unchanged configured per-instance connector resources through complete semantic physical identity, incarnation-safe invalidation, and process-scoped reference-counted generation leases. Preserve immutable GenerationRuntime publication, ResourceLedger rollback/retirement, processhost supervision, the executable backend-plugin ABI, startup-fixed discovery, routing/streaming/accounting semantics, candidate last-good isolation, and no-drop reload. Scope the first implementation to eligible discovered per_instance connectors only; candidate reuse is query-only, mutating preparation falls back to isolated physical construction, and generation-local model/routing/policy views remain generation-owned. Add no DI container, service locator, reactive component graph, live plugin watcher, generic resource framework, public lifecycle knob, or request-hot-path lookup. Require deterministic high-cardinality operation-count evidence before and after implementation so the optimization remains justified by connector-scale behavior rather than Cordis-inspired abstraction for its own sake." } From 0ac79d6aaba1386a6319db9f28ef879e923b6de2 Mon Sep 17 00:00:00 2001 From: matdev83 <211248003+matdev83@users.noreply.github.com> Date: Mon, 17 Aug 2026 11:22:58 +0200 Subject: [PATCH 12/12] spec: harden backend resource reconciliation contracts --- .../design-review.md | 211 +++-- .../design.md | 864 ++++++++---------- .../requirements.md | 192 ++-- .../research.md | 468 +++++----- .../spec.json | 4 +- .../tasks.md | 318 +++---- 6 files changed, 983 insertions(+), 1074 deletions(-) diff --git a/.kiro/specs/backend-connector-resource-reconciliation/design-review.md b/.kiro/specs/backend-connector-resource-reconciliation/design-review.md index 7f1e1e81..7002a32f 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/design-review.md +++ b/.kiro/specs/backend-connector-resource-reconciliation/design-review.md @@ -2,175 +2,190 @@ ## Verdict -**GO.** The selected design fits the current Go-LIP runtime and connector boundaries and provides a focused path from O(N) physical connector reconstruction toward O(K changed/unusable physical reconstruction) while retaining immutable generations. +**GO after CodeRabbit lifecycle/concurrency hardening.** The selected design remains a focused, high-ROI extension of Go-LIP's existing generation architecture, but the first review correctly identified several places where the original prose was weaker than the safety claim. Those findings were cross-checked against `processhost.Host`, runtimebundle ownership, adapter cleanup, and the standard backend-plugin host Session and have been incorporated into normative requirements/design/tasks. -The validation identified one normative gap around candidate failure-domain isolation. That correction has been applied to `requirements.md` before task generation: reuse hits are query-only with respect to the configured physical connector, mutating generation-preparation lifecycle requires isolated fallback, and overlapping metadata access is race/conformance tested. +The resulting design still targets O(N) -> O(K) physical connector reconstruction without introducing a general Cordis runtime, processhost redesign, public ABI/config surface, dynamic discovery, or request-path lookup. -No broader Cordis runtime, processhost redesign, public ABI, or dynamic discovery work is justified. +## Review Findings and Disposition -## Validation Checklist +### Detached entries survive invalidation — VALID / FIXED -### Generation consistency boundary — PASS +Original `current[identity]` indexing was insufficient for process shutdown: invalidation can remove a resource from the reusable map while old generations still retain leases. `Pool.Close` could not then enumerate it. -The design preserves `GenerationRuntime` as the immutable request-plane unit. Reconciliation occurs only while constructing the backend resources used to build a new generation. Published generations are never reconfigured, rebound, or live-migrated. +The design now requires a process-owned `owned`/all-entry set containing every successfully constructed incarnation until entry-level physical cleanup completes. Invalidation detaches only from `current`; terminal Close snapshots both current and detached residual entries through that ownership set. -Changed identity still creates a distinct physical resource before publication; removed/disabled identity is retained only by old generation leases. This preserves existing backend recomposition semantics. +### Pending waiter reservation race — VALID / FIXED -### `ResourceLedger` authority — PASS +Original prose allowed waiters to increment refs only after a building entry became live. A fast first claimant could therefore release to zero before a scheduled waiter formally acquired its ref. -Each generation receives a fresh lease-release cleanup and existing `buildBackends` continues to transfer that cleanup into the generation `ResourceLedger`. No new generation cleanup engine exists. +Every waiter now reserves its prospective lease claim under the pool mutex **before** waiting. Cancellation abandons only that claim. A deterministic scheduling test must hold a waiter after reservation and prove first-lease release cannot physically close the resource. -The physical connector cleanup is pool-owned and therefore cannot be copied into two generation ledgers. This is the central correctness requirement for sharing. +### Lease once versus physical cleanup once — VALID / FIXED -### Process ownership — PASS +Per-lease `sync.Once` only prevents one lease from releasing twice. It does not protect physical cleanup when final Release races `Pool.Close`. -The pool is process-scoped but is not a second process supervisor. `processhost.Host` remains responsible for launch, process slots, authenticated IPC, peer identity, configured host instance ownership, generation invalidation and process-tree cleanup. +The physical entry now owns a separate cleanup-once operation and stored result. Final lease release and fail-safe process shutdown converge on that same operation. Detached invalidation does not create another physical cleanup owner. -The pool only knows semantic identity, physical incarnation, backend value, physical cleanup, and dependent refcount. +### Cleanup ownership handoff — VALID / CLARIFIED -### Pre-`ProcessServices` factory capture — PASS +Current construction has two existing cleanup pieces: adapter/session cleanup and `ActivateResult.Cleanup` (`processhost.CloseInstance`). The pool entry consumes one composite per-resource cleanup on a physical miss; generations receive only lease release. -Production discovered factories are installed before `ProcessServices` exists. The design correctly creates the pool beside `processhost.Host` during discovered-install preparation, captures it lexically in factory closures, then transfers pool lifetime into `ProcessServices`. +This is **not** a transfer of process supervision out of `processhost.Host`. Host keeps slot/instance state, invalidation, reaping, and `Host.Close` fail-safe authority. The pool owns semantic retention and timing of that existing per-resource cleanup capability. -This avoids a service locator, global runtime registry, setter race, or post-publication dependency mutation. +An exactly-once test must count both composite cleanup pieces and subsequent Host.Close. -### Teardown ordering — PASS +### Pool Close can hang on an unbounded builder — VALID / FIXED -Current staging/artifact/host ownership has strict reverse-order requirements. Registering pool ownership after host ownership yields the needed relative shutdown order: +`ProcessServices.Close` is contextless/synchronous. The original “wait for builders” language was incomplete because current discovered construction can call Activate with a background lifetime. -`pool -> processhost.Host -> verified artifacts -> staging`. +The pool now owns a cancelable build root. An absent Acquire starts exactly one pool-owned builder goroutine; caller contexts control only claimant waiting. `Pool.Close` linearizes closing, cancels the build root, then joins builders before physical residual cleanup/host teardown. A blocked-builder test must exit only on that cancellation and prove no late publication. -The design also requires equivalent bootstrap-failure ordering before ownership transfer. +### Identity scenario coverage — VALID IN PRINCIPLE / LITERAL SUGGESTION NARROWED -### Physical identity completeness — PASS +CodeRabbit correctly requested stronger proof for artifact, secret, policy, and process-model identity dimensions. However, those dimensions are startup-fixed in the current production discovered-factory closure and are not all hot-reloadable through SIGHUP. -The design rejects direct use of the narrower `BackendStateIdentity` and derives a separate key at the physical construction/configure choke point. Artifact digest, logical instance/factory identity, opaque Configure bytes, process model, runtime policy and secret fingerprint are covered. +The corrected plan therefore uses two evidence matrices: -The explicit DTO-drift gate is important: identity correctness must fail closed as connector configure inputs evolve. +1. high-cardinality **generation reload** evidence for dimensions that actually vary during current reload plus invalidation; +2. focused **physical identity/construction** evidence for artifact digest, secret fingerprint, normalized runtime policy, process model, factory/logical identity. -### Secret handling — PASS +`shared_artifact` remains a non-pooled/restart-required fallback rather than being treated as a pooled replacement scenario. This covers the correctness concern without inventing unsupported hot artifact/policy/process-model reload behavior. -Secret values are only locally length-framed and hashed; no plaintext or secret-derived diagnostic value is exposed. Identity digests themselves should remain private because equality fingerprints can still be sensitive metadata. +### Acquire/Close linearization — VALID / FIXED -### Incarnation-safe invalidation — PASS +The original requirements said only that Close rejects new acquisitions and is race-safe. That is not enough to prevent a pending builder/Acquire from handing out a resource after residual cleanup begins. -The design correctly distinguishes desired semantic identity from a concrete process/session incarnation. Invalidation detaches the exact incarnation before/with existing processhost invalidation, future Acquire builds a new incarnation, and stale callbacks cannot remove a newer current entry. +The design now gives Close a mutex-protected terminal linearization point. After `closing=true`, no new claim is reserved and no build result is handed off as post-close success. Close cancels builders, waits builders and Acquire handoffs, then snapshots/cleans residual owned entries. -Existing generations are not silently rebound to the replacement. +### Shared connector operation concurrency — VALID / FIXED WITH EXISTING CONTRACT -### `shared_artifact` exclusion — PASS +Cross-checking the standard production host shows: -The design does not mix two independent sharing problems. Current `shared_artifact` process sharing and restart-required overlap gates remain unchanged. First-pass reconciliation is discovered `per_instance` only. +- `backendplugin/host.Session.Execute` already serializes Execute calls with `lifecycleMu` and serializes Execute versus Close; +- Resolve/ListModels/CountTokens/FinalizeBilling can overlap via the existing host/server instance lease model. -### Built-in backend exclusion — PASS +The spec therefore does **not** invent a new connector concurrency flag or remove Session serialization. It now explicitly characterizes retained-old-generation Execute overlapping new-generation Execute through one pooled Session and covers metadata/auxiliary overlap under race/conformance tests. -There is no evidence that cheap in-process builtins justify process-level lease machinery. Keeping them generation-owned limits complexity and avoids turning the selected connector optimization into a generic backend runtime. +Sharing one Session extends existing Execute serialization across generations and can remove the incidental transient parallelism provided today by two fresh Sessions. This is now documented as an operational tradeoff rather than hidden behind unconditional observational-equivalence wording. If that measured behavior is unacceptable for target workloads, pooling is re-scoped rather than host concurrency being redesigned here. -### Request hot path — PASS +## Validation Checklist -Resource lookup/refcounting occurs only during generation construction/retirement. Executor request calls use already-captured backend functions and introduce no pool mutex or identity hash on normal inference traffic. +### Generation consistency — PASS -### Dynamic inventory/model views — PASS with explicit constraint +Published generations remain immutable. Changed identity builds a replacement before publication; removed resources remain available only through old generation leases. No live substitution exists. -The design correctly keeps `modelregistry.Runtime`, inventory projection, routing/model views and refresh-loop ownership generation-local. A reused physical session may therefore receive concurrent `Resolve`/`ListModels` calls from overlapping generations. +### `ResourceLedger` authority — PASS -This is compatible with the existing configured-instance abstraction because one instance already serves runtime execution and metadata calls, but implementation tests must cover overlapping generation metadata access/race safety. Do **not** solve this by moving the model registry into the pool. +Every generation receives one fresh lease release. The physical composite cleanup never enters multiple ledgers. `buildBackends` remains the generation cleanup transfer point. -### Candidate last-good isolation — PASS; normative correction applied +### Processhost authority — PASS -This was the most important design-validation finding. +The pool contains no process launch, peer authentication, process-tree cleanup, slot supervision, or request transport logic. `processhost.Host` remains the physical supervisor and terminal fail-safe. -Fresh physical processes currently give candidate preparation strong failure-domain isolation. Sharing intentionally reduces physical failure-domain independence for an **unchanged** connector. The accepted boundary is now normative in Requirements 5.9–5.11, 8.8–8.9, and 9.9: +### Process ownership transfer — PASS -- a candidate reuse hit performs no Configure/Start/Stop/Close/mutating preflight on the shared resource; -- candidate rejection/rollback releases only its lease and does not invalidate the resource merely because the candidate failed; -- generation-local model preparation may use query-shaped `Resolve`/`ListModels` calls; -- any future mutating generation-preparation lifecycle makes the resource non-shareable until separately proven safe; -- overlapping metadata access receives race/conformance coverage. +Pool is created beside host before factory installation, captured lexically, then transferred to `ProcessServices`. Reverse close order remains pool -> host -> artifacts -> staging on successful ownership transfer and bootstrap failure. -This is not a claim that an external connector process can never crash while queried. Existing process failure remains possible. The preservation requirement is that the candidate lifecycle itself cannot intentionally mutate/close/reconfigure the last-good shared resource. +### Physical cleanup exactly once — PASS after correction -### Adapter cleanup shape — PASS with regression lock +Entry-level cleanup-once unifies final release and pool fail-safe shutdown. Generation lease once remains only a claim-release guard. -Current external adapter physical session cleanup lives in the returned BuildResult cleanup rather than an ordinary backend `Close` hook. This makes the selected lease model viable. +### Detached ownership — PASS after correction -Implementation must add a characterization/architecture test so future external adapter lifecycle hooks cannot silently create a physical-cleanup bypass around the pool. +All successful physical incarnations remain in process ownership until cleanup completes even if removed from semantic reuse by invalidation. -### Concurrency state machine — PASS +### Acquire/waiter protocol — PASS after correction -The proposed pending-entry protocol avoids a subtle refcount race that a naive `singleflight.Do` result handoff can create: the first caller cannot close a just-built resource before waiters have formally acquired it. +Claim reservation occurs before waiting. The first claimant cannot close the resource while another active waiter remains unaccounted for. -No physical build/cleanup under the mutex, exact-incarnation invalidation, cancellation-safe waiting, and close/pending-builder synchronization are appropriate. +### Acquire/Close shutdown boundary — PASS after correction -### Failure caching — PASS +Close has an explicit terminal linearization point, cancels pool builders, joins builders/handoffs, and prevents late publication before fail-safe cleanup. -Failed physical build is not permanently cached. This preserves current retry-on-later-attempt behavior and avoids introducing hidden negative-cache policy. +### Builder lifetime — PASS after correction -### No idle cache — PASS +Pool—not an arbitrary caller—owns physical builder cancellation/join. Caller cancellation is local to its reservation. This aligns with contextless ProcessServices shutdown without creating a permanent worker subsystem. -Closing on final generation lease keeps the scope tightly aligned with reload overlap. TTL/eviction/process-pressure policy would be speculative and materially increase operational complexity. +### Physical identity — PASS -### Backend-plugin ABI/security — PASS +Separate private identity includes/configures treatment for artifact, instance/factory, process model, opaque Configure bytes, normalized policy and secret fingerprint. DTO/input drift is fail-closed. -The design changes no public ABI and retains verified artifact binding, secure local IPC, peer auth before Configure/secrets, environment restrictions and process-tree cleanup. The pool does not parse provider-specific YAML. +### Identity evidence realism — PASS after correction -### ROI gate — PASS +Startup-fixed physical inputs are covered by focused identity/construction tests, not falsely presented as current hot-reload dimensions. High-cardinality reload evidence remains aligned with actual reloadability. + +### Candidate last-good isolation — PASS -Deterministic operation counts are a stronger primary gate than wall-clock thresholds. The 100-enabled-connector harness directly proves whether the expensive O(N) activation/configure wave exists and whether implementation removes it. Benchstat remains supporting evidence. +Reuse hit is query-only for candidate preparation. Candidate rollback only releases its claim and does not mutate/reconfigure/close/invalidate the shared last-good resource. -## Requirements Correction Applied +### Standard Session concurrency — PASS with explicit operational gate -The design review required three safety clauses, all now present before task generation: +The design preserves the current host Session concurrency implementation. Cross-generation Execute serialization and metadata/auxiliary overlap receive direct tests. No public concurrency ABI is added. + +### Request hot path — PASS -1. **Query-only candidate reuse:** Requirements 5.9–5.10 prohibit Configure/Start/Stop/Close/mutating preparation on a reused physical connector while allowing query-shaped Resolve/ListModels access. -2. **Fail-closed future lifecycle:** Requirement 5.11 requires isolated construction if future candidate preparation needs a mutating connector lifecycle action. -3. **Shared metadata concurrency evidence:** Requirements 8.9 and 9.9 require overlapping-generation race/conformance coverage and lock out a generation-owned physical cleanup bypass. +Pool operations occur only at generation construction/retirement/process shutdown. Normal request execution uses captured backend functions. -No design rewrite was required because `design.md` already expressed these constraints; the validation correction made them normative. +### `shared_artifact` and built-in exclusions — PASS + +Neither is pulled into the first implementation. Existing restart-required/shared-process behavior is unchanged. + +### Security/ABI — PASS + +Verified artifact binding, secure IPC, peer authentication before Configure/secrets, environment restrictions, process-tree cleanup, and current backend-plugin ABI remain unchanged. + +### ROI gate — PASS + +Deterministic 100-enabled-connector work counts remain the primary justification. Supporting evidence now also records the standard Session cross-generation execution-scheduling tradeoff so the optimization is evaluated as a whole. ## Design-to-Requirement Trace | Requirement | Design coverage | |---|---| -| R1 scale evidence | 100-instance count harness, mixed-K matrix, supporting benchmarks | -| R2 narrow boundary | discovered per-instance only; private pool above processhost | -| R3 physical identity | explicit configure-input identity and fail-closed DTO drift gate | -| R4 lease contract | pending/live entries, per-generation lease release, final physical cleanup | -| R5 generation semantics | unchanged/changed/remove/rollback flows; derived state stays generation-local; query-only candidate reuse | -| R6 incarnation invalidation | exact entry token, detach, fresh same-key incarnation, no live substitution | -| R7 cleanup/shutdown | pool-before-host/artifacts/staging, one physical cleanup owner | -| R8 non-interference | no request path lookup, ABI/security/routing/billing unchanged, query-only candidate rule | -| R9 TDD/architecture | RED identity/pool/scale tests, race/goleak, adapter lifecycle and anti-container gates | +| R1 scale evidence | high-cardinality reload count matrix and re-scope gate | +| R2 narrow boundary | discovered overlap-safe per-instance only; pool above processhost | +| R3 identity | complete private key, startup-fixed/reload-varying split, drift gate | +| R4 Acquire/Close | pre-reserved claims, pool-owned builder, linearized Close, no late publish | +| R5 generation semantics | immutable projections, changed/remove/rollback, query-only candidate reuse | +| R6 invalidation | exact incarnation, detached process ownership, fresh replacement | +| R7 cleanup/shutdown | entry-level cleanup once; pool -> host -> artifacts -> staging | +| R8 concurrency/non-interference | preserve Session contract; characterize cross-generation serialization | +| R9 TDD/architecture | scheduling/race/blocked-builder/ownership/identity/ROI gates | ## Simplification Review -The design deliberately rejects the following tempting expansions: +The hardened design still rejects: -1. **No generic resource manager.** One connector-specific private pool is enough. -2. **No processhost generation awareness.** The host remains a physical supervisor. -3. **No `BackendStateIdentity` semantic overloading.** Observation-state compatibility remains independent. -4. **No idle cache.** Reuse exists only while generations overlap. -5. **No public feature flag.** Unsafe resources fall back rather than exposing lifecycle internals. -6. **No shared model registry.** Generation consistency remains coarse-grained where it provides value. -7. **No dynamic plugin reconciliation.** Startup-fixed trust/discovery remains unchanged. -8. **No Cordis requires/provides graph or fibers.** There is no current service-dependency problem requiring them. +1. generic resource manager/container; +2. processhost generation awareness; +3. overloading `BackendStateIdentity`; +4. idle TTL cache; +5. public feature/concurrency flags; +6. shared model registry; +7. dynamic plugin reconciliation; +8. Cordis requires/provides graph/fibers; +9. Session concurrency redesign. + +The new `owned` entry set, builder cancellation root, and entry cleanup-once are accepted because they close concrete correctness holes introduced by resource sharing; they are not general-purpose framework concepts. ## Implementation Risks to Pin With Tests -- a Configure-time DTO field added without identity treatment; -- two candidate builders constructing the same absent identity twice; -- a candidate rollback closing the active generation's resource; -- final release racing a new Acquire and handing out a closing entry; -- old incarnation invalidation deleting a newer incarnation; -- pool Close returning while a pending builder can still publish/use a closing host; -- external adapter gaining `Close`/Start/Stop semantics that bypass lease cleanup; -- dynamic inventory refresh races across overlapping generations; -- changed config/artifact/secret/policy incorrectly hitting reuse; -- removed backend closing before retained old generation drains; -- pool shutdown running after host/artifact teardown; -- request execution accidentally consulting the pool; -- architecture growing into a generic registry/container. +- waiter reserved too late and resource reaches zero before handoff; +- invalidated detached entry disappears from process shutdown ownership; +- final release and Pool.Close double-run session/host cleanup; +- Close waits forever for a builder using the wrong lifetime context; +- builder publishes after Close linearizes; +- Acquire hands out a resource after fail-safe cleanup begins; +- stale invalidation detaches a replacement incarnation; +- physical identity omits a Configure/launch input; +- startup-fixed identity dimension is accidentally treated as hot reload without redesign; +- candidate rollback mutates/invalidates last-good shared resource; +- standard Session cross-generation Execute serialization causes hidden deadlock/cancellation regression; +- metadata/Count/Finalize overlap exposes connector race; +- pool begins duplicating processhost or appears in request/public surfaces. ## Final Gate -**GO.** Requirements, design, and brownfield validation are aligned. Task decomposition may proceed under the explicit evidence-first and query-only candidate-reuse gates; no further architecture correction is required before implementation. +**GO.** All technically valid CodeRabbit lifecycle/concurrency findings are now represented in normative requirements, design mechanics, and TDD tasks. The identity-matrix suggestion was adopted with a brownfield correction: startup-fixed inputs receive focused identity/construction coverage rather than fictional hot-reload scenarios. + +Implementation remains approval-gated by `spec.json`. The implementation must still pass the deterministic scale gate and may be re-scoped if the existing standard Session execution-serialization tradeoff erodes the expected operational ROI. diff --git a/.kiro/specs/backend-connector-resource-reconciliation/design.md b/.kiro/specs/backend-connector-resource-reconciliation/design.md index 462041f9..213559c1 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/design.md +++ b/.kiro/specs/backend-connector-resource-reconciliation/design.md @@ -2,145 +2,123 @@ ## Overview -This design removes redundant physical reconstruction of unchanged discovered executable `per_instance` backend connectors during material runtime-generation reload while preserving Go-LIP's existing immutable-generation consistency model. +This design removes redundant physical reconstruction of unchanged discovered executable `per_instance` backend connectors during material runtime-generation reload while preserving Go-LIP's immutable-generation consistency model. -The selected architecture adds one package-private, process-scoped connector resource reconciliation owner above `processhost.Host`. The owner maps an exact semantic physical-resource identity to the current live physical incarnation. Generation compilation acquires a lease. An unchanged candidate receives the already-configured immutable backend/session resource plus a generation-owned lease release; a changed identity constructs a new physical incarnation exactly as today. The existing `ResourceLedger` owns each generation's lease release, and the existing process host continues to own process launch, authenticated IPC, configured host instances, process invalidation, and process-tree teardown. +The architecture adds one package-private, process-scoped connector-resource reconciliation owner above `processhost.Host`. The owner maps a complete semantic physical-resource identity to the current reusable physical incarnation while also retaining an ownership set of every successfully constructed incarnation that has not completed physical cleanup. Generation compilation acquires a lease claim. An unchanged candidate receives the already-configured backend/session resource plus a generation-owned lease release; a changed or invalidated identity constructs a fresh physical incarnation exactly through the current host/adapter path. -The design is intentionally **not** a component runtime. There is no generic dependency graph, provider lookup, service locator, DI container, HMR, dynamic plugin discovery, reusable public resource framework, or request-hot-path lookup. +`ResourceLedger` remains the generation cleanup authority. `processhost.Host` remains the sole executable-process/IPC supervisor. The pool does **not** take over process slots, peer authentication, invalidation, reaping, or `Host.Close`; it only owns semantic reconciliation, dependent claims, and the timing of the existing per-resource composite cleanup. + +The design is intentionally not a component runtime. There is no generic dependency graph, service locator, DI container, HMR, live plugin discovery, reusable public resource framework, or request-hot-path lookup. ### Goals -- Reduce physical connector construction on material reload from all enabled eligible connectors to only connectors whose physical identity changed or whose current incarnation is unusable. -- Preserve immutable `GenerationRuntime` publication, old-generation drain, last-good candidate rollback, and current processhost security/supervision. -- Make physical resource identity complete and fail-closed with respect to configure/construction inputs. -- Give one physical resource exactly one physical cleanup owner while allowing several overlapping generations to hold independent idempotent leases. -- Preserve changed/remove behavior: changed resources replace before publication; removed resources remain usable only through old retained generations until drain. -- Preserve physical failure semantics without live-swapping resources under published generations. -- Establish deterministic high-cardinality operation-count evidence before and after production integration. +- Reduce physical connector reconstruction on material reload from all enabled eligible connectors to only changed or unusable identities. +- Preserve immutable `GenerationRuntime`, request/async generation pinning, last-good rollback, retained old-generation work, and no-drop retirement. +- Make physical identity complete/fail-closed for all construction/configure inputs. +- Give each physical incarnation one entry-level exactly-once cleanup capability while allowing several generation leases. +- Make Acquire/Close/build shutdown behavior linearizable and cancellation-safe. +- Preserve processhost security and supervision rather than layering a second process manager. +- Preserve established standard host-session operation concurrency and explicitly measure the cross-generation effect of sharing one Session. +- Establish deterministic high-cardinality operation-count evidence before enabling production reuse. ### Non-Goals -- Replace `runtimehost.Manager`, generation leases, `GenerationRuntime`, `ResourceLedger`, `ProcessServices`, or `processhost.Host`. +- Replace `runtimehost.Manager`, `GenerationRuntime`, `ResourceLedger`, `ProcessServices`, or `processhost.Host`. - Pool built-in/in-process backend factories. - Change `shared_artifact` process semantics or restart-required overlap behavior. -- Share generation-local executor, model-registry, catalog, routing, feature, policy, billing, or lifecycle state. -- Add live connector install/upgrade/remove discovery, watchers, rescans, or HMR. -- Add a public resource-pool API or a new connector manifest/ABI/config flag. -- Change canonical request/event translation, routing, retry/failover, streaming, output commit, cancellation, token accounting, or billing behavior. -- Guarantee that an external process can never fail while a candidate performs a semantically read-only metadata query; the guarantee is that candidate rollback/rejection does not reconfigure, stop, close, or intentionally invalidate a shared last-good resource. +- Share generation-local executor/model-registry/catalog/routing/feature/policy/billing state. +- Add connector watchers, rescans, HMR, dynamic install/upgrade/remove, or an idle TTL cache. +- Add a public resource-pool API, manifest capability, ABI field, config flag, or concurrency flag. +- Increase standard `backendplugin/host.Session` Execute parallelism or remove its current lifecycle serialization. +- Guarantee that an external connector can never fail while a candidate performs a query-shaped operation. ## Boundary Commitments -### Existing Authorities That Remain Authoritative - -| Concern | Existing authority | Design treatment | +| Concern | Existing authority | Treatment | |---|---|---| -| process-scoped lifetime | `ProcessServices` / `processResourceOwner` | owns reconciliation close | +| process lifetime | `ProcessServices` / `processResourceOwner` | owns pool shutdown | | generation cleanup | `ResourceLedger` | owns lease release | -| request/async generation lifetime | `runtimehost.Manager` / generation refs | unchanged | -| executable process/IPC | `processhost.Host` | unchanged | -| exact executable trust | `VerifiedArtifact` | artifact digest enters identity | -| plugin ABI | `pkg/lipsdk/backendplugin` | unchanged | -| discovery/catalog | startup-fixed discovery/trust | unchanged | -| generation model views | current compiler/runtimebundle | rebuilt per generation | - -### Revalidation Triggers - -The implementation must be revalidated if any of these occur: +| request/async lifetime | runtimehost generation refs | unchanged | +| process/IPC supervision | `processhost.Host` | unchanged | +| exact executable trust | `VerifiedArtifact` | digest contributes to identity | +| connector protocol | `pkg/lipsdk/backendplugin` | unchanged | +| discovered catalog | startup-fixed discovery/trust | unchanged | +| generation projections | runtimebundle/compiler | rebuilt per generation | +| physical cleanup timing | new private entry | composite cleanup exactly once | -- a new configure-time or physical-construction input is added to executable connectors; -- `ConfiguredInstance` gains a new lifecycle method used during generation preparation; -- generation compilation begins invoking a mutating lifecycle action on discovered connector backends; -- `processhost` ownership keys/process models change; -- physical resource reuse expands beyond discovered `per_instance` connectors; -- a public configuration/ABI field is proposed for reconciliation; -- request execution begins consulting the resource pool; -- pool shutdown can run before generation drain or after processhost/artifact teardown. +Revalidate this design if a configure/launch input changes lifetime, `ConfiguredInstance` gains a new generation-preparation lifecycle action, processhost ownership semantics change, reuse expands beyond discovered `per_instance`, standard Session concurrency changes, or request execution begins consulting the pool. -## Existing Architecture Analysis +## Existing Architecture and Target -### Current successful overlap model - -Today a material generation build follows the safe but potentially expensive pattern: +Today an unchanged discovered `per_instance` backend is reconstructed for every material generation: ```text -Generation 17 (published) - └── backend A -> physical connector A#17 +Gen17 -> logical backend A -> physical A#17 -compile Generation 18 - └── backend A -> physical connector A#18 // new Activate + Configure +compile Gen18 + -> same logical backend A -> physical A#18 (Activate + Configure) -publish Generation 18 - ├── new admissions -> A#18 - └── retained Gen17 work -> A#17 +publish Gen18 + old work -> A#17 + new work -> A#18 -retire Gen17 - └── cleanup A#17 +retire Gen17 -> cleanup A#17 ``` -The fresh physical connector is deliberate. `buildDiscoveredBackend` mints a unique host activation handle for `per_instance` connectors so `processhost.Host.instances` can hold candidate and active handles simultaneously. That mechanism must remain for **new physical incarnations**. - -The inefficiency appears when A's physical defining inputs did not change. In that case the candidate reconstructs a resource whose configured behavior is intentionally identical. +The unique host activation handle is deliberate and remains required whenever a **new** physical incarnation is constructed. -### Target overlap model +Target for an unchanged identity: ```text ProcessServices - └── backendResourcePool - └── semantic identity A - └── incarnation 41 - ├── configured backend/session - ├── physical cleanup - └── refs = 2 - ▲ ▲ - │ │ - Gen17 Gen18 + └─ backendResourcePool + └─ identity A -> incarnation 41 + ├─ backend/session functions + ├─ composite physical cleanup (entry-owned) + ├─ claims = 2 + └─ owned until cleanup completes + ▲ ▲ + Gen17 Gen18 ``` -Generation 18 still gets its own executor map, model registry, routing views, policies, handler, and `ResourceLedger`. Only the configured external connector resource is shared. - -Changed identity remains the existing double-resource pattern: +Changed identity remains two physical resources: ```text -identity A-old -> incarnation 41 <- Gen17 -identity A-new -> incarnation 42 <- Gen18 candidate +A-old -> incarnation 41 <- Gen17 +A-new -> incarnation 42 <- Gen18 candidate ``` ## Selected Architecture -### Component Map - ```mermaid graph TB - Install[Discovered install preparation] --> Host[processhost.Host] - Install --> Pool[backendResourcePool] - Pool --> Host - Install --> Factories[discovered lifecycle factory closures] - Factories --> Pool - Pool --> Physical[configured connector physical incarnation] - Physical --> Host - Process[ProcessServices] --> Pool - Process --> Host - Compile[Generation compiler] --> Factories - Factories --> Lease[BackendBuildResult with lease cleanup] - Lease --> Ledger[ResourceLedger] - Compile --> Gen[GenerationRuntime] - Gen --> Backend[leased immutable backend functions] + Install[Discovered install preparation] --> Host[processhost.Host] + Install --> Pool[backendResourcePool] + Install --> Factory[discovered factory closures] + Factory --> Pool + Pool --> Physical[configured connector incarnation] + Physical --> Host + Process[ProcessServices] --> Pool + Process --> Host + Compile[Generation compiler] --> Factory + Factory --> Lease[BackendBuildResult: backend + lease release] + Lease --> Ledger[ResourceLedger] + Compile --> Gen[GenerationRuntime] ``` -The pool is an **ownership/reconciliation index**, not a service registry. Only discovered factory construction calls it. Request execution sees only the backend functions already embedded in its immutable generation executor. +The pool is a connector-specific reconciliation/ownership index, not a service registry. Request execution receives only backend functions already captured in an immutable generation. -### Construction and Ownership Timing +## Construction and Ownership Timing -Production currently installs discovered factory closures before `ProcessServices` exists. The pool therefore must be created at discovered-install preparation time next to `processhost.Host`: +Discovered factory closures are installed before `ProcessServices` exists. Therefore the pool is created beside the discovered host and captured lexically: ```text prepareDiscoveredPluginInstall acquire staging verify artifacts create processhost.Host - create backendResourcePool(host-associated lifetime) - register discovered factories capturing host + pool + create backendResourcePool + install factories capturing host + pool │ ▼ BuildHost @@ -151,586 +129,506 @@ NewProcessServices register staging cleanup register artifact cleanup register host cleanup - register pool cleanup // reverse close: pool -> host -> artifacts -> staging -``` - -Exact field/helper names may adapt to package conventions. The required property is direct lexical capture plus lifetime transfer; no global registration or mutable service lookup is allowed. + register pool cleanup -`discoveredBackendInstall.release` and early bootstrap cleanup gain the pool and release in dependency order: - -```text -pool -> host -> artifacts -> staging +reverse process close order: + pool -> host -> artifacts -> staging ``` -### Private Types +The same relative order is required on pre-transfer/bootstrap failure. There is no global registry or setter race. -Illustrative package-private shapes: +## Private State Model -```go -type backendResourceIdentity struct { - instanceID string - factoryKind string - artifactDigest string - processModel processhost.ProcessModel - configDigest [32]byte - configureDigest [32]byte -} +Illustrative shapes: +```go type backendResourcePool struct { - mu sync.Mutex - closing bool - nextInc uint64 + mu sync.Mutex + closing bool + nextInc uint64 + current map[backendResourceIdentity]*backendResourceEntry + owned map[*backendResourceEntry]struct{} // physical success until cleanup completes + + buildCtx context.Context + cancelBuild context.CancelFunc + buildWG sync.WaitGroup + + // Tracks Acquire calls that have entered the handoff protocol so Close can + // establish a terminal boundary before residual cleanup. + handoffWG sync.WaitGroup } type backendResourceEntry struct { identity backendResourceIdentity incarnation uint64 state backendResourceState - refs int + claims int ready chan struct{} backend execbackend.Backend - cleanup func() error // physical cleanup; pool-only + cleanup func() error buildErr error + + cleanupOnce sync.Once + cleanupErr error } type backendResourceLease struct { pool *backendResourcePool entry *backendResourceEntry - once sync.Once - err error + once sync.Once // one claim release only } ``` -These are design shapes, not public contracts. The implementation should use fewer fields/types if equivalent safety can be expressed more simply. +Exact implementation may use a condition/counter instead of `handoffWG`, and may merge fields when safe. Required semantics are more important than these names. There is deliberately no generic `Resource[T]`, `Scope`, `Provider`, `Component`, `Context`, `Registry`, or public `LeaseManager` abstraction. ## Physical Resource Identity -### Identity Principle +### Principle -A semantic physical identity answers only this question: +Identity answers: -> If two generation builds present these inputs, is it correct for both generations to execute through the same already-configured connector instance without another Configure/physical build? +> Can two generation builds safely execute through the same already-configured connector instance without another physical Configure/build? -Equality must be conservative. False negatives cost performance; false positives can violate correctness. +False negatives cost optimization. False positives can violate correctness, so equality is conservative. -### Required Inputs +### Inputs -Identity is derived at the `buildDiscoveredBackend` construction/configuration choke point from: +At the physical construction/configure choke point, identity treatment covers: -1. logical configured `instanceID`; -2. manifest/export `factoryKind`; +1. logical configured instance ID; +2. factory kind; 3. exact `VerifiedArtifact.DigestHex`; -4. declared `ProcessModel` (`per_instance` for eligible resources); -5. effective opaque YAML bytes actually sent to Configure; -6. effective `backendplugin.RuntimePolicy` after host-owned normalization such as `DisableTransportRetries=true`; -7. configure-time secret bundle fingerprint when non-empty; -8. any future generation-varying input consumed by physical launch/configure semantics. - -`BackendStateIdentity` is not used as the reuse key. It may share a low-level canonical digest helper only if doing so does not couple the two compatibility contracts. +4. process model; +5. exact effective opaque Configure YAML bytes; +6. normalized `backendplugin.RuntimePolicy`, including host-owned normalization such as `DisableTransportRetries=true`; +7. configure-time `SecretBundle` by private digest when present; +8. any future generation-varying launch/configure input. -### Canonical Fingerprinting +Use SHA-256 with domain-separated, length-delimited fields. Runtime policy projection explicitly enumerates every field. Secret names are sorted; names/values are length-framed and hashed; plaintext does not survive identity construction or appear in logs/status/errors. -Use SHA-256 with unambiguous length-delimited/domain-separated framing, not string concatenation with secret values. Example conceptual framing: +`BackendStateIdentity` remains a separate, narrower affinity/health continuity contract and is not the physical reuse key. -```text -backend-resource/v1 -field(instance_id, bytes) -field(factory_kind, bytes) -field(artifact_digest, bytes) -field(process_model, bytes) -field(config_yaml, bytes) -field(runtime_policy_v1, canonical bytes) -field(secret_fingerprint_v1, digest bytes) -``` - -Runtime policy serialization must explicitly enumerate every field in `backendplugin.RuntimePolicy`. Slices/maps use deterministic ordering where their semantics are set/map-like. SecretBundle fingerprinting sorts secret names and hashes length-framed name/value bytes; only the resulting digest survives identity construction. - -### DTO Drift Gate +### Startup-fixed versus reload-varying inputs -A focused structural/contract test must make additions to the configure-time physical input surface fail until identity treatment is intentional. Acceptable techniques include: +Current production discovered factory closures capture artifact/process-model and install-time runtime policy at startup; the current discovered path also does not hot-rotate an artifact through SIGHUP. These facts still participate in identity treatment because their lifetime can evolve and focused construction tests can exercise them, but the high-cardinality **reload** matrix must not pretend they are currently reloadable. -- a test that explicitly projects every `RuntimePolicy` field and fails when struct shape changes; -- a compile-time helper that consumes a versioned private identity DTO and a test comparing the external DTO's reflected field names in test code; -- another deterministic approach that forces review without reflection in production. +Evidence is split: -Production identity code should remain explicit rather than generic reflection-based serialization. +- generation-reload matrix: unchanged/config-changed/remove/invalidate dimensions that current reload can actually exercise; +- focused identity/construction matrix: artifact digest, secret fingerprint, normalized policy, process model, factory/instance identity. -### Fail-Closed Eligibility +`shared_artifact` remains non-pooled; a process-model difference that leaves `per_instance` eligibility does not mean “build another pooled resource.” -If an input cannot be represented safely or is known to carry generation-specific semantics outside the identity, the factory bypasses reconciliation and runs the current physical construction path. Non-shareable fallback is correctness-preserving and requires no user-visible error. +### Drift/fail-closed gate -## Resource Pool State Machine +A structural contract test must force deliberate review when the Configure/physical input surface changes. Production hashing should remain explicit rather than reflection-driven. If completeness cannot be proven, the path uses current isolated construction. -### States +## Resource State Machine and Linearization -A minimal entry needs these conceptual states: +Conceptual states: ```text -building -> live -> detached/invalid -> closed - │ │ │ - └------> failed └-> final lease release -> physical cleanup +building -> live -> detached -> closed + │ │ ▲ + └------> failed │ invalidation/final-release/close removes current ``` -`detached` means it is no longer the current reusable entry for its semantic identity. Existing leases may still reference it. +`current` is only the reusable semantic index. `owned` contains every successfully constructed physical entry until entry-level physical cleanup completes, even after invalidation detaches it from `current`. -A `closing` pool rejects new acquisitions and waits for in-progress physical builders before host teardown. +### Acquire/Close linearization -### Acquire Existing - -```text -Acquire(identity) - lock - current[identity] == live entry ? - yes -> refs++ - unlock - return backend + fresh lease.Release - no -> absent/building path -``` +`Acquire` and `Close` use the same pool mutex for their terminal decision: -No connector factory, host activation, Configure RPC, or adapter build occurs on a live reuse hit. +- successful Acquire claim reservation/handoff linearizes while `closing == false`; +- Close linearizes when it sets `closing = true` under that mutex; +- after that point no new claim may be reserved and no pending build result may be published/handed off as a post-close success. -### Concurrent Absent Acquire +Close cancels pool-owned builders and waits both builder completion and Acquire handoff completion before residual cleanup. This prevents a lease from being handed a physical resource after fail-safe cleanup has already run. -The implementation must ensure one physical construction per semantic identity. A small per-key pending entry is preferred over a broad generic singleflight abstraction if it simplifies ref ownership: +### Pool-owned builder lifetime -```text -first caller: - install building entry with reserved ref - build physical resource outside mutex - publish live entry - -other callers: - observe building entry - wait on ready/cancellation - retry state check - increment ref only after live -``` +An absent-key Acquire does not make the caller the physical builder owner. It: -The design must avoid this race: the first caller builds/releases to zero before a waiting caller has formally acquired its ref. A pending-entry protocol or equivalent must make ref reservation and publication unambiguous. +1. installs a building entry and reserves the caller's prospective claim under the mutex; +2. starts exactly one short-lived **pool-owned** builder goroutine using a context derived from `pool.buildCtx`; +3. increments `buildWG` before the goroutine becomes runnable; +4. the caller then waits like any other claimant on `ready` or its own context. -Physical build/configure and physical cleanup must never run while holding the pool mutex. +The physical builder must receive the pool-owned context through `processhost.Activate`/Configure. The pooled path must not use `context.Background()` for the build lifetime. -### Failed Build +Caller cancellation abandons only that caller's reserved claim. It does not cancel a build that may serve other claimants. If every claim disappears before a build succeeds, the completed physical result is not published as an idle entry; it is cleaned immediately. -- remove/detach the failed pending entry; -- wake waiters; -- run any partial cleanup through existing build/processhost error ownership; -- return the existing wrapped build error; -- do not negative-cache failure permanently; -- a later independent acquire may build again. +Pool Close calls `cancelBuild()` before `buildWG.Wait()`. The processhost/transport stack is already context-aware; a blocked-build test proves the path exits on cancellation and cannot publish late. -### Release +### Waiter reservation protocol -Each generation gets a fresh idempotent release closure. +For a building entry, every caller increments `claims` **before** it waits. The claim is already the ownership reservation that will become its lease if the build succeeds. ```text -Release(entry) - once - lock - refs-- - if refs > 0: - unlock - return nil - if current[key] == entry: - delete current[key] - mark detached/closing - capture physical cleanup - unlock - cleanup exactly once -``` - -Because there is no idle cache, final release closes immediately. +Acquire(building) + lock + closing? -> reject + claims++ // reserve before wait + unlock -## Physical Construction Integration + wait ready | caller ctx -### Current path retained as the builder + canceled -> abandon reserved claim + ready -> re-check state / close boundary + success -> return lease for existing reserved claim +``` -The pool does not duplicate `buildDiscoveredBackend`. Refactor that function only enough to separate: +This removes the race in which the first caller builds and releases to zero before a waiter has incremented a ref. A deterministic scheduling test holds a waiter after reservation, releases the first returned lease, then proves physical cleanup cannot occur until the waiter abandons/releases its claim. -1. identity/eligibility preparation; -2. physical construction of a new unique host activation/session/adapter resource; -3. pool acquisition returning a leased generation result. +### Build completion -Conceptual shape: +On success, builder completion reacquires the mutex: -```go -func buildDiscoveredBackend(...) (pluginreg.BackendBuildResult, error) { - input, err := prepareDiscoveredPhysicalInput(...) - if err != nil { ... } +- if pool is closing or the entry has zero remaining claims, do not publish it as reusable; record physical ownership long enough to clean it and wake waiters; +- otherwise attach backend/composite cleanup, add entry to `owned`, mark live, and wake waiters; +- no external cleanup/Configure/launch runs under the mutex. - if !eligible(input) || pool == nil { - return buildDiscoveredPhysical(input) - } +On failure: - id, shareable, err := physicalIdentity(input) - if err != nil { ... } - if !shareable { - return buildDiscoveredPhysical(input) - } +- remove the building entry from `current` if still exact; +- store build error and mark failed; +- wake waiters; +- do not negative-cache it; +- waiter claim abandonment eventually reaches zero without a physical cleanup because construction never completed. - return pool.Acquire(ctx, id, func(incarnation uint64) (physicalBackendResource, error) { - return buildDiscoveredPhysical(input, incarnation) - }) -} -``` +A later independent Acquire may retry. -The current unique `hostInstanceID = logicalID#sequence` behavior remains inside `buildDiscoveredPhysical` for every newly created incarnation. +### Live reuse -### Pool return shape +A live hit reserves/increments its claim under the mutex and returns the immutable backend value plus a fresh lease release. It performs no factory call, process activation, Configure, or adapter build. -The entry owns: +### Release -- `execbackend.Backend` functional value; -- underlying adapter/session cleanup; -- `ActivateResult.Cleanup`/host instance cleanup; -- exact invalidation binding for that physical process generation/incarnation. +Lease `sync.Once` only prevents one generation lease from releasing its claim twice. It is **not** the physical cleanup authority. -The generation receives: +On release: -```go -pluginreg.BackendBuildResult{ - Backend: entry.backend, - Cleanup: lease.Release, -} +```text +lease.once: + lock + decrement exact entry claim + if claims > 0 -> unlock, return + if current[key] == entry -> remove from current + mark detached + unlock + entry.cleanupPhysical() ``` -`buildBackends` remains the authority that transfers `BackendBuildResult.Cleanup` into `ResourceLedger`. Therefore rollback/retirement semantics need no new generation cleanup engine. +`entry.cleanupPhysical()` owns a separate `sync.Once` and stored result. Every path that may physically tear down the pooled resource—normal final release and pool fail-safe shutdown—calls this same method. It removes the entry from `owned` only after cleanup completes. -### No cleanup bypass +Thus final Release racing Pool.Close cannot execute physical cleanup twice. -For eligible external pooled resources, `buildBackends` must not synthesize another generation-local physical close from backend lifecycle hooks. Characterization tests should lock the current adapter contract: the pooled backend has no `Start`/`Stop`/`Close` path whose invocation would physically close the configured session outside the pool lease. +## Physical Cleanup Ownership Handoff -If future external connector adapters gain such lifecycle hooks, pooling eligibility must be revisited rather than silently wrapping them twice. +This design distinguishes **process supervision ownership** from the **per-resource cleanup capability**. -## Candidate Preparation and Last-Good Isolation +Current physical construction produces two cleanup responsibilities: -Sharing a physical process changes one aspect of the failure domain: overlapping generations no longer own independent OS processes for an unchanged connector. This is acceptable only if candidate construction does not use the shared resource for **generation-local mutation**. +1. `adapter.Build(...).Cleanup()` -> closes the configured host Session / connector instance RPC-side resource; +2. `ActivateResult.Cleanup` -> calls `processhost.Host.CloseInstance(hostActivationID)`, which removes the host instance and reaps its process slot when appropriate. -### Allowed reused-resource preparation operations +`buildDiscoveredPhysical` shall return one idempotent composite physical cleanup that preserves the current ordering/error-join behavior across those two operations. On a pool miss, the reconciliation entry consumes that composite cleanup. It is never copied into a generation ledger. -Current external adapter semantics expose query-shaped operations such as: +Each generation receives only: + +```go +pluginreg.BackendBuildResult{ + Backend: entry.backend, + Cleanup: lease.Release, +} +``` -- `Resolve` capability/profile lookup; -- `ListModels` inventory snapshot when advertised. +`buildBackends` continues transferring that cleanup into `ResourceLedger`. -These may be invoked by generation-local model registry construction/refresh. They are semantically query operations in the backend-plugin contract and may use internal connector caches, but candidate code must not call `Configure`, `Close`, `Stop`, or a mutating lifecycle/preflight operation on a reused physical resource. +`processhost.Host` retains its internal `instances`/`slots`, invalidation/reap logic, and `Host.Close` fail-safe cleanup. The pool does not own a `Process` object or replace host supervision; it owns only when the existing per-instance composite cleanup may be invoked without violating another generation's lease. -### Required last-good guarantee +An exactly-once regression shall count session close, `CloseInstance`/activation cleanup, entry cleanup, and later `Host.Close` to prove these paths do not become competing physical owners. -A candidate that later fails due unrelated validation/composition must only release its lease. It must not: +## Physical Construction Integration -- reconfigure the shared resource; -- close/stop it; -- invalidate it merely because the candidate is rejected; -- mutate processhost instance ownership; -- cancel active-generation executions. +Refactor `buildDiscoveredBackend` only enough to separate: -An external connector/process can still independently fail while serving query/execution calls; existing invalidation/recovery semantics remain authoritative. The optimization must not turn ordinary candidate rollback into such a failure. +1. effective input/eligibility preparation; +2. physical construction of a new unique host activation/session/adapter resource; +3. pool Acquire returning a leased generation result. -If generation preparation later requires a mutating backend lifecycle action, that backend/resource path becomes non-shareable until a new design proves isolation. +Conceptually: -## Generation-Local Derived State +```go +func buildDiscoveredBackend(...) (pluginreg.BackendBuildResult, error) { + input, err := prepareDiscoveredPhysicalInput(...) + if err != nil { ... } + if pool == nil || !eligible(input) { + return buildDiscoveredPhysical(ctx, input) + } -Physical reuse does **not** mean generation reuse. For every material generation compile, existing code still builds: + id, shareable, err := physicalIdentity(input) + if err != nil { ... } + if !shareable { + return buildDiscoveredPhysical(ctx, input) + } -```text -leased physical backends - ↓ -BackendInventory slice - ↓ -new modelregistry.Runtime - ↓ -new registry snapshot / generation-local refresh ownership - ↓ -new executor/routing/policy views - ↓ -new HTTP handler / GenerationBundle / ResourceLedger + return pool.Acquire(ctx, id, func(buildCtx context.Context, inc uint64) (physicalBackendResource, error) { + return buildDiscoveredPhysical(buildCtx, input, inc) + }) +} ``` -This preserves current per-generation model/routing consistency. It also means two overlapping generations may issue concurrent metadata queries through the same physical connector. That is compatible with a configured instance already serving concurrent attempts; race/conformance tests must cover it. +Every **new** per-instance physical build retains the current unique host activation ID behavior. Pool reuse avoids entering that build path at all. -The first implementation does not attempt to deduplicate model-registry refresh loops or other derived computation. That is separate evidence work. +## Candidate Preparation and Generation-Local State -## Invalidation and Incarnations +A reuse hit performs no Configure/Start/Stop/Close/mutating preflight. Candidate rollback releases only its lease and cannot invalidate the shared resource merely because unrelated candidate validation failed. -### Why incarnation identity is required - -Configuration equality does not imply a particular process/session is still healthy. +Generation-local structures are still recreated: ```text -semantic identity X - incarnation 7 -- fails - incarnation 8 -- replacement, same desired configuration +leased backend values + -> BackendInventory + -> new modelregistry.Runtime / snapshot / refresh ownership + -> new executor/routing/policy/billing views + -> new handler / GenerationBundle / ResourceLedger ``` -The pool therefore tracks the exact entry/incarnation in every invalidation callback. +Query-shaped metadata operations may therefore reach the same physical Session from overlapping generations. This is acceptable only under the established standard-host concurrency behavior below. -### Invalidation flow +If future generation preparation introduces a mutating lifecycle call against an external connector, that path becomes non-shareable until separately redesigned. -For a new physical build, adapter invalidation is wrapped conceptually as: +## Established Connector Operation Concurrency -```go -func() { - pool.Invalidate(identity, incarnation) - _ = host.InvalidateProcessGeneration(processGeneration) -} -``` +### Standard host contract today -Required order/properties: +Production uses `backendplugin/host.Session` through the default discovered connector path. -1. mark/detach only the exact entry incarnation from `current[identity]`; -2. future Acquire cannot obtain it; -3. delegate physical invalidation/reap to existing `processhost.Host`; -4. do not decrement existing generation refs merely because resource failed; -5. final lease release later runs the entry's idempotent physical cleanup, which may encounter already-gone transport and continues using existing cleanup normalization; -6. if `current[identity]` already points to a newer incarnation, a stale callback does not modify the newer entry. +- `Session.Execute` holds the existing `lifecycleMu` for the full execute RPC; two Execute calls on the same Session are serialized, and `Close` cannot tear down the transport during Execute. +- `Resolve`, `ListModels`, optional `CountTokens`, and optional `FinalizeBilling` do not take that client lifecycle mutex. The gRPC server already leases a configured instance around those calls; connectors are therefore already exposed to metadata/auxiliary overlap with other operations today. -### No live substitution +This specification preserves those facts. It does not remove `lifecycleMu`, introduce a new semaphore, or change the public `ConfiguredInstance` ABI. -Existing generations retain backend functions closing over the failed incarnation. They observe existing failure/recovery behavior. The pool never swaps an entry pointer behind an executor or redirects an already-open attempt to the new incarnation. +### Cross-generation consequence -A later generation build can acquire a new incarnation for the same semantic identity. +With fresh Gen17/Gen18 Sessions today, overlap can transiently provide two independent Execute serialization domains. With one pooled Session, retained Gen17 and new Gen18 Execute calls share the same existing serialization domain. -## Changed, Removed, and Rollback Flows +That is a capacity/scheduling change during overlap, not a canonical request transformation. It must be explicit rather than hidden behind an “observationally identical” claim. -### Unchanged unrelated material reload +A deterministic test shall hold an old-generation Execute open, start a new-generation Execute on the same pooled Session, and prove behavior follows the existing Session serialization without deadlock, cancellation corruption, or wrong-generation cleanup. Supporting benchmark/evidence should record the overlap effect. -```text -Gen17 lease X refs=1 -compile Gen18 -> Acquire(X) refs=2; no physical build -candidate validates -publish Gen18 -retire Gen17 -> refs=1 -retire Gen18 later -> refs=0 -> physical cleanup -``` +If that established serialization makes connector reuse operationally unacceptable for the intended long-lived-stream workload, implementation must re-scope instead of changing Session concurrency under this spec. -### Changed config/artifact/secret/policy +Focused race/conformance tests shall also cover overlapping-generation `Resolve`, `ListModels`, CountTokens, FinalizeBilling, and execution through the standard host. Non-standard injected/test session implementations are pooled only when they satisfy the same established host behavior; otherwise they bypass reuse. -```text -Gen17 lease X refs=1 -compile Gen18 -> identity Y != X - -> build Y incarnation -publish Gen18 -Gen17 keeps X until drain -``` +## Invalidation and Detached Entries -### Remove/disable +Semantic identity and physical incarnation differ: ```text -Gen17 retains X -Gen18 does not Acquire(X) -publish Gen18 -X closes only after Gen17's final lease release +identity X + incarnation 7 -- fails/detaches, still leased by Gen17 + incarnation 8 -- new current resource for later generation ``` -### Candidate fails after reusing X +Invalidation flow for an entry: -```text -active Gen17 refs(X)=1 -candidate Gen18 Acquire(X) -> refs=2 -later candidate failure -ResourceLedger rollback -> lease release -> refs=1 -active Gen17 unaffected -``` +1. under pool synchronization compare the exact `(identity, entry/incarnation)`; +2. if it is the current entry, remove only that exact incarnation from `current` and mark detached; +3. leave it in `owned` while existing lease claims remain or until terminal pool cleanup invokes entry cleanup; +4. future Acquire sees no current reusable entry and may build a new incarnation; +5. delegate the physical process-generation invalidation/reap to existing `processhost.Host`; +6. do not live-substitute a replacement into published generations; +7. stale callbacks from incarnation 7 cannot remove incarnation 8. -### Candidate fails after creating Y +Final lease release later invokes the same entry-level cleanup once; it may find the process already reaped, using existing idempotent/error-normalization semantics. -```text -candidate creates Y refs=1 -later failure -rollback -> refs=0 -> physical cleanup Y -active X unchanged -``` +Tracking detached entries in `owned` is essential: terminal `Pool.Close` can enumerate and fail-safe clean an invalidated resource even if a retained generation leaked/failed to release its lease before process teardown. ## Process Shutdown -`Host.Close` remains the sole process shutdown coordinator and existing runtimehost generation drain remains first. +`runtimebundle.Host.Close` remains the process shutdown coordinator, and runtimehost generation drain remains the expected first stage. -Target ordering after generations have drained: +Relevant target order: ```text -ProcessServices.Close reverse ownership - 1. backendResourcePool.Close - - reject new Acquire - - wait for any in-progress builder to terminate - - fail-safe close residual entries - 2. processhost.Host.Close - 3. VerifiedArtifact.Close handles - 4. staging directory removal - ...existing earlier/later ProcessServices resources as currently ordered... +generation admission stopped / generations drain + -> lease releases + -> ProcessServices.Close + backendResourcePool.Close + 1. lock: set closing (linearization point), reject later Acquire + 2. cancel pool build context + 3. unlock + 4. wait pool-owned builders and Acquire handoffs + 5. lock: detach/snapshot all residual owned entries, including invalidated/detached + 6. unlock + 7. call entry.cleanupPhysical on residual entries + processhost.Host.Close + VerifiedArtifact.Close + staging removal ``` -The actual closer list contains many other process resources; the important relative ordering is pool before host before artifacts before staging. - -`backendResourcePool.Close` is idempotent. Under correct host shutdown its live refcount set should normally be empty because generations drained. Residual entries indicate a failed/aborted ownership path and are closed as a fail-safe rather than leaked. +Pool Close never holds its mutex while waiting for builders/handoffs or running physical cleanup. Builders use pool-owned cancellation and cannot publish after the close boundary. Entry cleanup is once-guarded, so a concurrent final lease release and shutdown converge safely. -Pool close must not hold its mutex while waiting for builders or running physical cleanup. +Under normal successful host shutdown, generation drain should make residual claims empty before ProcessServices close. Residual cleanup remains a terminal fail-safe for broken/aborted ownership paths; after terminal process shutdown, old leases are not promised a usable connector. ## Error Handling -- Preserve existing `runtimebundle`/`processhost` build error wrapping and public reload categories. -- Do not add `resource_reuse_failed` or similar public error categories. -- Failed physical build leaves no reusable current entry. -- Final lease release returns underlying normalized physical cleanup error through the existing `ResourceLedger` rollback/close aggregation path. -- Non-final lease release normally returns nil because it performs no physical cleanup. -- Pool shutdown joins residual cleanup failures consistently with existing process close aggregation. -- Cancellation while waiting for another caller's build returns the caller's context error without canceling the builder on behalf of other dependents. - -## Concurrency Design +- Preserve existing runtimebundle/processhost error wrapping and public reload categories. +- Add no public `resource_reuse_failed` category. +- Failed build leaves no reusable current resource and is not permanently cached. +- Caller cancellation while waiting returns the caller context error and abandons only its reservation. +- Pool Close cancellation is pool-owned and terminates builders for process shutdown. +- Final normal lease cleanup errors flow through existing ResourceLedger aggregation. +- Residual process-shutdown cleanup errors join existing ProcessServices close errors. +- Entry cleanup stores one result so racing cleanup callers observe one physical cleanup outcome. -### Locking rules +## Concurrency Rules to Prove -- One small pool mutex protects map membership, entry state, refcounts, pool-closing flag, and incarnation allocation. -- No process launch, Configure RPC, metadata RPC, session close, host cleanup, channel wait, or callback is executed while holding the mutex. -- Waiters block on per-entry readiness outside the mutex and re-check state after wake. -- Lease release is `sync.Once`-guarded. -- Physical cleanup is exactly-once entry-owned even when invalidation already reaped the process. +1. two concurrent absent Acquires -> one builder, two pre-reserved claims, one incarnation; +2. first returned lease release before second waiter wakes -> no physical cleanup until waiter abandons/releases; +3. waiter cancellation -> only that claim drops; builder/other claims survive; +4. final release racing new Acquire -> Acquire either reserves before detach or builds after detach; never receives closing entry; +5. Close linearizes against Acquire -> no post-close claim/build publication; +6. Close cancels a blocked pool-owned builder and waits it; no late publication after host teardown; +7. final release racing Close -> entry-level physical cleanup exactly once; +8. invalidation detaches current but leaves entry process-owned until cleanup; Close can enumerate it; +9. stale invalidation cannot detach newer incarnation; +10. candidate rollback racing retained old-generation release preserves correct claim count and cleanup timing. -### Races to prove +## Scale and Identity Evidence -1. two candidates concurrently acquire the same absent identity -> one physical build, two leases; -2. waiter cancellation does not tear down another caller's resource; -3. final release races new Acquire -> either Acquire reserves before final detach or builds a new incarnation after detach; never acquires a closing resource; -4. invalidation races Acquire -> no acquire after known invalidation can receive the invalid entry; -5. stale invalidation races new incarnation publication -> newer entry survives; -6. pool Close races pending build -> builder cannot publish into a closing pool and physical result is cleaned before host close; -7. candidate rollback races old-generation release -> refcount/cleanup exactly once. +### High-cardinality generation-reload matrix -## Scale and ROI Evidence Design +At least 100 synthetic discovered `per_instance` connector rows, no external credentials, deterministic counters for factory physical build, activation/launch, Configure, cleanup, lease acquire/release. -### Deterministic 100-connector harness - -Add a focused runtimebundle/processhost-backed fixture capable of creating at least 100 enabled synthetic discovered `per_instance` rows without external credentials. The fixture exposes counters for: - -- lifecycle factory/physical build invocation; -- host activation; -- OS/fake launcher launch where the test profile supports it; -- Configure/dial session count; -- physical session/host cleanup count; -- lease acquisition/release count after implementation. +| Scenario | Expected physical construction | +|---|---| +| baseline unrelated reload before reuse | O(N), characterize current behavior | +| target unrelated reload, N unchanged | 0 new builds/activations/Configure | +| one/K backend configs changed | exactly 1/K new physical resources | +| remove/disable subset | no build for removed rows; old resource retained by old generation | +| candidate rollback after reuse hits | no physical cleanup of active resources | +| candidate builds K new then fails | K new resources cleaned | +| invalidate one then compile same config | exactly one fresh incarnation | -Use a fake/in-process session or test processhost substrate where necessary so the deterministic gate remains fast and cross-platform. Separate native process smoke tests continue to protect actual process cleanup/security. +### Focused identity/construction matrix -### Required scenarios +Independent focused tests exercise physical input dimensions that are startup-fixed in current production reload but are correctness-critical identity inputs: -| Scenario | Physical construction expectation | +| Difference | Expected | |---|---| -| baseline unrelated material reload before implementation | O(N) new construction, characterize current behavior | -| target unrelated material reload, N unchanged | 0 new builds/activations/configures | -| one of N configs changed | 1 replacement physical build | -| K of N identities changed | K replacement physical builds | -| disabled/removed subset | 0 builds for removed rows; old resources close after old generation drain | -| candidate fails after all N reuse hits | 0 physical cleanup of active resources | -| candidate builds K new then fails | K new resources cleaned; active unchanged resources retained | -| invalidated one of N then compile same config | exactly 1 new physical incarnation | +| artifact digest | identity miss; fresh physical build when otherwise eligible | +| secret fingerprint | identity miss; fresh build when secrets are effective input | +| normalized RuntimePolicy | identity miss; fresh build | +| process model | no alias; `shared_artifact` follows existing non-pooled/restart-required path | +| factory kind / logical instance | no alias | -### Supporting benchmark evidence +This separation avoids claiming hot artifact/policy/process-model reload support that does not exist today. -Extend/reuse `reload_bench_test.go` or add a focused benchmark to report candidate compile time and allocations for high-cardinality external connector fixtures. If platform-test infrastructure makes useful native process metrics available, record peak launch/process/FD/RSS observations, but do not make unstable host metrics the correctness gate. +### Supporting benchmark -The primary claimed improvement is structural: +Record candidate compile time/allocations and available synthetic/native resource observations. Primary correctness/ROI remains deterministic operation counts. No request throughput/token-latency gain is claimed. -```text -physical construction work: - before ≈ O(N enabled eligible connectors) per material generation - target ≈ O(K changed/unusable connectors) + O(N) cheap lease/projection work -``` - -No request throughput or per-token latency improvement is claimed. +Also characterize retained-old-generation/new-generation Execute scheduling on one pooled standard Session so the known serialization tradeoff is visible in implementation evidence. ## File Structure Plan -Exact filenames may adapt during implementation, but responsibilities should remain narrow: +Possible private files: ```text internal/infra/runtimebundle/ -├── backend_resource_identity.go # private identity/fingerprinting -├── backend_resource_pool.go # private entry/lease/reconciliation owner -├── backend_resource_pool_test.go # RED state/concurrency tests -├── backend_resource_identity_test.go # identity completeness/discrimination -├── discovered_factories.go # narrow integration / physical builder split -├── plugin_catalog.go # construct pool beside processhost host -├── composition_root.go # transfer/error-release ownership bundle -├── process_services*.go # process ownership transfer/ordering -├── build_model.go # preserve lease cleanup -> ResourceLedger -└── reload_backend_resource_reuse_test.go # high-cardinality + generation semantics - -internal/infra/backendplugins/processhost/ -└── existing files # no ownership redesign; focused test seams only if required + backend_resource_identity.go + backend_resource_pool.go + backend_resource_identity_test.go + backend_resource_pool_test.go + discovered_factories.go + plugin_catalog.go / composition_root.go / process_services*.go + reload_backend_resource_reuse_test.go internal/archtest/ -└── backend_resource_reconciliation_test.go # no generic registry/request-path/public framework + backend_resource_reconciliation_test.go ``` -No new package is required unless file/package budgets make a tiny connector-lifecycle subpackage materially clearer. A new generic `resource`, `lifecycle`, `container`, or `dependency` package is explicitly disallowed by this design. +No new generic `resource`, `container`, `dependency`, or lifecycle framework package is required. ## Testing Strategy -### TDD order +TDD order: + +1. baseline/high-cardinality and identity RED tests; +2. reserved-claim/Acquire-Close/detached ownership/builder cancellation RED tests; +3. exactly-once physical cleanup/ownership handoff RED tests; +4. candidate isolation and standard Session operation-concurrency RED tests; +5. private identity + pool implementation; +6. process ownership transfer and discovered-factory integration; +7. generation reload matrix, invalidation and retained-work integration; +8. race/goleak/security/conformance/no-drop regressions; +9. benchmark/evidence and final simplification gate. -1. high-cardinality characterization and target count RED tests; -2. identity discrimination/completeness RED tests; -3. pool lease/refcount/build/invalidation/shutdown RED tests; -4. process ownership-order RED tests; -5. discovered factory reuse integration; -6. unchanged/changed/remove/rollback generation integration; -7. race/goleak, plugin security/conformance, no-drop reload regression; -8. supporting benchmark/benchstat evidence; -9. final simplification review. +Existing ResourceLedger, processhost, discovered overlap/restart-required, backend security/conformance, retained-generation, and reload last-good/no-drop suites remain regression authorities. -### Regression suites +## Rejected Alternatives -At minimum preserve/pass the focused equivalents of: +### Reconfigure an existing resource in place -- runtimebundle backend recomposition tests; -- discovered overlap/restart-required tests; -- ResourceLedger rollback/retirement tests; -- processhost activation/cleanup/invalidation tests; -- backend-plugin security and conformance gates; -- runtime reload last-good/no-drop tests; -- ownership/architecture tests from the prior resource-lifecycle refactor. +Rejected: mutates provider state under an old generation. -## Rejected Alternatives +### Put semantic generation reconciliation in processhost + +Rejected: conflates LIP configuration identity with physical process supervision. + +### Reuse `BackendStateIdentity` + +Rejected: does not cover artifact, policy, secret, process-model, or future physical inputs. + +### Return one `BackendBuildResult` cleanup to multiple generations + +Rejected: permits early/double physical teardown. -### Reconfigure an existing physical connector in place +### Track only `current` entries -Rejected. It would mutate the provider under old generations and destroy generation consistency. +Rejected after review: invalidation removes an entry from `current` while retained generations can still lease it, leaving Pool.Close unable to enumerate residual owned resources. -### Make processhost reuse logical instance IDs across generations +### Let the initiating Acquire own the builder -Rejected. `processhost` should not know LIP semantic config identity, and existing unique activation handles are valuable for genuine replacement overlap. +Rejected after review: caller cancellation/shutdown lifetime becomes ambiguous and can leave Close waiting on an unbounded background operation. The pool owns builder context/goroutine lifetime. -### Keep new physical resources alive in an idle cache +### Increment waiter refs only after build publication -Rejected. Cross-generation overlap is sufficient for the target optimization; idle retention introduces TTL/eviction/resource-pressure policy with no demonstrated need. +Rejected after review: the first claimant can release to zero before a scheduled waiter acquires its ref. Claims are reserved before waiting. -### Pool every backend type +### Add idle cache or TTL -Rejected. Cheap in-process builtins do not justify the ownership complexity and may have different lifecycle semantics. +Rejected: adds eviction/resource-pressure policy with no demonstrated need. -### Add a manifest `reusable_across_generations` capability immediately +### Change Session Execute concurrency as part of reuse -Rejected for the first implementation. The host-owned executable adapter path and complete identity/fallback rules are sufficient to prove the concept without expanding ABI. If real connectors later demonstrate incompatible semantics that cannot be inferred from the common adapter contract, a separate compatibility/capability specification can revisit this. +Rejected: this would broaden the refactor into host/ABI concurrency semantics. Preserve current serialization, measure its overlap effect, and re-scope pooling if unacceptable. -### Share the entire model runtime across generations +### Pool all backend types / share whole model runtime -Rejected. Model/routing views are part of the immutable generation contract and may depend on other config fields; physical connector reuse does not justify sharing them. +Rejected: no evidence-backed ROI and would weaken clear generation ownership. ## Design Success Criteria -The refactor is successful only if all of these are true: - -1. unchanged eligible connector rows generate zero new physical activation/configure work on unrelated material reload; -2. changed/unusable connectors still get fresh physical incarnations before publication; -3. candidate rollback cannot close or reconfigure the last-good generation's shared physical resource; -4. final generation lease owns the moment of physical cleanup exactly once; -5. stale invalidation cannot poison a replacement incarnation; -6. process shutdown remains one coordinated order with pool before host/artifacts/staging; -7. generation-local derived runtime state remains separate; -8. no request-path lookup/lock is added; -9. no public config/ABI or generic runtime framework is added; -10. the implementation diff makes the expensive lifecycle behavior simpler to reason about at high connector cardinality rather than introducing more concepts than it removes. +The refactor succeeds only if: + +1. unchanged eligible reloads produce zero new physical Activate/Configure work; +2. changed/unusable identities get fresh physical incarnations before publication; +3. candidate rollback cannot mutate/close the last-good shared connector; +4. every waiting Acquire reserves ownership before waiting, eliminating zero-ref handoff races; +5. Close has a terminal linearization point, cancels/joins pool builders, and prevents late publication; +6. invalidated/detached physical entries remain process-owned/enumerable until cleanup; +7. final release, invalidation aftermath, and process shutdown converge on one entry-level exactly-once physical cleanup; +8. processhost remains the sole physical supervisor and pool close precedes host/artifact/staging teardown; +9. generation-local runtime state remains separate and request execution performs no pool lookup; +10. established standard Session concurrency is preserved and its cross-generation Execute serialization is explicitly characterized; +11. identity tests cover all physical input dimensions without pretending startup-fixed inputs are hot-reloadable; +12. no public config/ABI or generic runtime framework is introduced; +13. deterministic scale evidence still justifies the implementation after accounting for concurrency/lifecycle complexity. diff --git a/.kiro/specs/backend-connector-resource-reconciliation/requirements.md b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md index 0a0809a6..7fe968f2 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/requirements.md +++ b/.kiro/specs/backend-connector-resource-reconciliation/requirements.md @@ -4,123 +4,129 @@ Go-LIP shall reduce unnecessary reconstruction of expensive executable backend connector resources across immutable runtime generations when a material configuration reload does not change those connectors. The optimization shall preserve the existing generation publication model: every request remains bound to one immutable `GenerationRuntime`; changed resources are constructed before publication; old generations and their resources remain valid until their existing work drains; candidate failure leaves the last-good generation untouched. -This specification borrows only the Cordis-v4 ideas that fit this problem: semantic provider identity, reconciliation of unchanged desired resources, physical incarnation identity, and lifetime retention while dependents still hold the provider. It does **not** introduce a Cordis component runtime, reactive dependency graph, fibers, dependency injection, service location, HMR, or generic effect system. +This specification borrows only the Cordis-v4 ideas that fit this problem: semantic provider identity, reconciliation of unchanged desired resources, explicit physical-incarnation identity, and retention while dependent generations still hold the provider. It does **not** introduce a Cordis component runtime, reactive dependency graph, fibers, dependency injection, service location, HMR, or a generic effect/resource system. -The first implementation is intentionally narrow. It targets discovered executable backend connectors whose declared process model is `per_instance` and whose current reload policy permits candidate/active overlap. Installed-but-disabled connectors, statically linked backends, and `shared_artifact` connectors are not justification for this refactor and remain outside the first implementation. +The first implementation is intentionally narrow. It targets discovered executable backend connectors whose declared process model is `per_instance` and whose current reload policy permits candidate/active overlap. Installed-but-disabled connectors, statically linked backends, and `shared_artifact` connectors remain outside the first implementation. ## Boundary Context -- In scope: deterministic scale evidence, private physical connector-resource identity, process-lifetime reconciliation, per-generation leases, candidate rollback, invalidation/incarnation behavior, shutdown ordering, and focused executable-connector integration. -- Out of scope: general component reconciliation, frontend/feature reconciliation, dynamic plugin install/uninstall, discovery watchers, new backend-plugin ABI fields, new public configuration knobs, built-in backend pooling, `shared_artifact` behavior changes, request migration between generations, or live mutation of a published backend instance. +- In scope: deterministic scale evidence, private physical connector-resource identity, process-lifetime reconciliation, per-generation leases, candidate rollback, invalidation/incarnation behavior, Acquire/Close linearization, shutdown ordering, established host-session concurrency, and focused executable-connector integration. +- Out of scope: general component reconciliation, frontend/feature reconciliation, dynamic plugin install/uninstall, discovery watchers, new backend-plugin ABI fields, new public configuration knobs, built-in backend pooling, `shared_artifact` behavior changes, request migration between generations, live mutation of a published backend instance, or redesign of host-session execution concurrency. - Existing authorities remain: `ProcessServices` for process-owned resources, `ResourceLedger` for generation-owned cleanup, `processhost.Host` for executable process/IPC supervision, runtimehost generation leases for request/async lifetime, and the existing backend-plugin ABI for connector behavior. - Performance intent: eliminate redundant connector construction work during material reloads. This is not a request-hot-path optimization. ## Requirement 1: Evidence-First Scale Justification 1.1. Before production reuse is enabled, add a deterministic high-cardinality characterization harness that compiles overlapping generations with at least 100 enabled synthetic host-backed `per_instance` connector instances. -1.2. The harness shall count at minimum physical connector builds, `processhost` activations or launches, configure operations, and physical cleanup operations; timing/allocation benchmarks may supplement but shall not replace these deterministic counters. -1.3. The baseline shall demonstrate the current reconstruction behavior for an unrelated material reload in which connector-defining inputs are unchanged. -1.4. After implementation, an unrelated material reload with `N` unchanged eligible live connectors shall perform **zero** new physical connector builds, activations/launches, and configure operations for those `N` connectors; it may perform `N` lightweight lease acquisitions. -1.5. For a candidate with `K` changed eligible connector identities and all remaining eligible connectors unchanged, the target is exactly the necessary replacement construction for the changed set; unchanged identities shall be leased rather than rebuilt. -1.6. Test acceptance shall not depend on fixed wall-clock thresholds that are vulnerable to CI host variance. Benchstat or equivalent measurements may be recorded as supporting evidence. -1.7. If the implementation cannot achieve the count-based reduction without changing request semantics or introducing a general runtime/container abstraction, the implementation shall be stopped or re-scoped rather than preserving the abstraction for speculative future value. +1.2. The harness shall count at minimum physical connector builds, `processhost` activations or launches, Configure operations, and physical cleanup operations; timing/allocation benchmarks may supplement but shall not replace these deterministic counters. +1.3. The baseline shall demonstrate current reconstruction behavior for an unrelated material reload in which connector-defining generation inputs are unchanged. +1.4. After implementation, an unrelated material reload with `N` unchanged eligible live connectors shall perform **zero** new physical connector builds, activations/launches, and Configure operations for those `N` connectors; it may perform `N` lightweight lease acquisitions and rebuild normal generation-local projections. +1.5. For a candidate with `K` changed or unusable eligible connector identities and all remaining eligible connectors unchanged, physical construction shall be proportional only to `K`. +1.6. Test acceptance shall not depend on fixed wall-clock thresholds vulnerable to CI host variance. Benchstat or equivalent measurements may be supporting evidence. +1.7. If these count-based gains cannot be achieved without changing request semantics, weakening shutdown/reload safety, or introducing a general runtime/container abstraction, implementation shall stop or re-scope rather than preserve speculative infrastructure. ## Requirement 2: Narrow Eligibility and Ownership Boundary -2.1. Reconciliation shall be private to runtime composition and shall not become a public SDK capability or a request-time service locator. +2.1. Reconciliation shall be private to runtime composition and shall not become a public SDK capability or request-time service locator. 2.2. The initial eligible set shall be discovered executable connectors using `ProcessModelPerInstance` and an overlap-safe reload policy. 2.3. Built-in/in-process backend factories shall retain current generation ownership unless a later evidence-backed specification separately justifies reuse. 2.4. `ProcessModelSharedArtifact` connectors shall retain their current explicit sharing/restart-required semantics; this specification shall not weaken their isolation or overlap gates. -2.5. Plugin discovery/trust shall remain startup-fixed. Connector installation, removal, directory rescanning, executable upgrade discovery, and automatic file watching are not added by this specification. -2.6. `processhost.Host` shall remain the sole executable process/IPC supervisor; the reconciliation layer shall not duplicate launch, peer authentication, process-tree cleanup, slot management, or transport supervision. -2.7. No new public YAML field, manifest field, CLI flag, environment variable, or backend-plugin ABI field shall be required merely to turn this optimization on. -2.8. Because discovered lifecycle factory closures are installed before `ProcessServices` construction, the private reconciliation owner shall be created beside the discovered `processhost.Host`, captured directly by eligible factory closures, and then have its lifetime transferred into `ProcessServices`; the implementation shall not solve this timing boundary with a global registry, service locator, or mutable post-construction dependency lookup. -2.9. The reconciliation owner shall remain connector-specific and package-private. It shall expose no generic keyed `Get`/`Resolve` API for unrelated runtime services. +2.5. Plugin discovery/trust shall remain startup-fixed. Connector installation, removal, directory rescanning, executable upgrade discovery, and automatic file watching are not added. +2.6. `processhost.Host` shall remain the sole executable process/IPC supervisor. The reconciliation layer shall not duplicate launch, peer authentication, process-tree cleanup, slot/instance supervision, or transport management. +2.7. No new public YAML field, manifest field, CLI flag, environment variable, or backend-plugin ABI field shall be required merely to enable this internal optimization. +2.8. Because discovered lifecycle factory closures are installed before `ProcessServices` construction, the private reconciliation owner shall be created beside the discovered `processhost.Host`, captured lexically by eligible factory closures, and then have its lifetime transferred into `ProcessServices`; no global registry, service locator, or mutable post-construction lookup may bridge this timing boundary. +2.9. The reconciliation owner shall remain connector-specific and package-private and shall expose no generic keyed `Get`/`Resolve` API for unrelated runtime services. ## Requirement 3: Semantic Physical Resource Identity 3.1. Reuse shall require an exact private identity representing the configured physical connector resource, not Go object equality and not only the logical backend instance ID. -3.2. At minimum, the identity shall distinguish logical instance ID, factory kind, exact verified executable artifact digest, process model, opaque connector configuration content, effective configure-time runtime policy, and configure-time secret material by a non-reversible fingerprint. +3.2. At minimum, identity treatment shall cover logical instance ID, factory kind, exact verified executable artifact digest, process model, opaque connector configuration content, effective configure-time runtime policy, and configure-time secret material by a non-reversible fingerprint. 3.3. Identity construction shall be deterministic for semantically identical effective inputs within one process. -3.4. Secret plaintext shall never be retained in the identity, logged, emitted in diagnostics, or exposed through public status. Secret fingerprinting shall be local and deterministic for equality purposes only. -3.5. Executable artifact replacement shall produce a distinct resource identity even when logical instance ID and YAML configuration are unchanged. -3.6. Credential rotation or another configure-affecting secret change shall produce a distinct resource identity. -3.7. Runtime-policy changes that affect the configured connector shall produce a distinct resource identity. -3.8. `BackendStateIdentity` may provide precedent or low-level hashing helpers, but its current `{InstanceID, FactoryKind, ConfigDigest}` contract shall not be treated as sufficient proof that two physical connector resources are interchangeable. -3.9. Identity shall be derived at one explicit construction/configuration choke point from the effective inputs that define physical connector behavior. A future configure-time or launch-identity input shall require an intentional identity decision rather than being silently omitted. -3.10. If a construction/configure input varies between candidate generations and cannot be represented safely and deterministically in the identity, that resource shall be treated as non-shareable and shall use the existing generation-local construction path. -3.11. Process-stable facts need not be redundantly hashed, but every such omission shall be documented and protected by tests so a later change from process-stable to generation-varying cannot create unsafe reuse. - -## Requirement 4: Process-Scoped Acquire and Generation Lease Contract - -4.1. An eligible physical connector resource shall have one process-scoped lifetime owner and zero or more generation leases. -4.2. Acquiring an exact live identity that is already current shall reuse the existing immutable configured resource and increment its dependent lease count rather than invoking the connector factory/process activation path again. -4.3. Concurrent acquisitions for the same absent identity shall construct at most one physical current resource; successful contenders shall receive leases to that same resource. -4.4. A generation shall own only its lease release through its existing `ResourceLedger`; generation rollback/retirement shall not directly own or close a shared physical connector resource. -4.5. Releasing one of several leases shall not close the physical resource. Releasing the final lease shall detach the resource and invoke its physical cleanup exactly once. -4.6. There shall be no idle TTL/cache-retention policy in the first implementation. A valid current physical resource exists only while at least one generation lease retains it. -4.7. A failed physical build/configure shall not be cached as a permanent negative result; a later independent acquisition may retry through the existing construction path. -4.8. Lease release shall be idempotent and safe under candidate rollback, generation retirement, and process shutdown races. -4.9. The physical cleanup returned by the underlying connector/process build shall be retained only by the reconciliation entry. Every generation-facing `BackendBuildResult.Cleanup` for a reused resource shall be a fresh idempotent lease release, not the physical cleanup function. -4.10. Eligible leased backend values/lifecycle hooks shall not expose an alternate generation-owned `Close` or `Stop` route that can bypass the lease and tear down a physical resource while another generation retains it. - -## Requirement 5: Preserve Immutable Generation Semantics +3.4. Secret plaintext shall never be retained in the identity, logged, emitted in diagnostics, or exposed through public status. Secret fingerprinting shall be local, length-framed, deterministic for equality, and private. +3.5. Executable artifact replacement shall produce a distinct identity even when logical instance ID and YAML configuration are unchanged. +3.6. Credential/secret change shall produce a distinct identity when secret material is part of effective Configure input. +3.7. Runtime-policy change shall produce a distinct identity when the normalized policy differs. +3.8. Process-model change shall not accidentally reuse a `per_instance` entry; unsupported/non-eligible process models shall use their existing non-pooled path. +3.9. `BackendStateIdentity` may provide precedent or low-level hashing helpers, but its current `{InstanceID, FactoryKind, ConfigDigest}` contract is insufficient proof that two physical connector resources are interchangeable. +3.10. Identity shall be derived at one explicit construction/configuration choke point. A future configure-time or launch-identity input shall require an intentional identity decision rather than being silently omitted. +3.11. If a construction/configure input varies between generations and cannot be represented safely and deterministically, that resource shall be non-shareable and use current generation-local construction. +3.12. Facts that are startup-fixed in the current production path, including discovered artifact/process-model and install-time runtime policy, need not be presented as hot-reload dimensions; focused identity/construction tests shall still prove they cannot alias if exercised directly, and drift tests shall force review if their lifetime changes. + +## Requirement 4: Process-Scoped Acquire, Waiter Reservation, and Close Linearization + +4.1. An eligible physical connector resource shall have one process-scoped reconciliation entry and zero or more generation lease claims. +4.2. Acquiring an exact live identity that is current shall reserve a claim under the pool mutex and reuse the existing configured resource rather than invoking physical construction. +4.3. For an absent identity, exactly one pool-owned physical builder shall be started. Every caller waiting on a building entry, including the initiating caller, shall reserve its prospective lease claim **before** waiting; caller cancellation releases only that reserved claim and does not cancel a builder needed by other callers. +4.4. A fast first claimant shall not be able to release the resource to zero while another uncanceled waiter for the same build has not yet completed its handoff. The reserved-claim protocol or an equivalent barrier shall make this scheduling race impossible. +4.5. The physical builder shall run under a pool-owned cancellation context, not a caller-owned/background lifetime. Pool shutdown shall cancel that build context before joining builders. +4.6. `Acquire` success and `Close` shall have an explicit linearization order under the same synchronization boundary: once `Close` marks the pool closing, no new Acquire may reserve a claim and no pending Acquire may publish/hand off a newly built resource as a post-close success. +4.7. `Close` shall reject new acquisitions, cancel pool-owned builders, wait for in-flight builders and acquisition handoffs to terminate, and only then perform residual physical cleanup. A builder finishing after closure begins shall clean its result without publishing it as reusable. +4.8. A failed physical build/configure shall not be permanently negative-cached; waiters observe the failure, reservations are released, and a later independent acquisition may retry. +4.9. Each generation-facing lease release shall be idempotent. Releasing one of several claims shall not close the physical resource; final normal release detaches the entry and invokes entry-owned physical cleanup. +4.10. The physical cleanup returned by underlying adapter/process construction shall be retained only by the reconciliation entry. Every generation-facing `BackendBuildResult.Cleanup` for a pooled resource shall be a fresh lease release, never the physical cleanup function. +4.11. Eligible pooled backend values/lifecycle hooks shall expose no alternate generation-owned `Close`/`Stop` path capable of bypassing the lease and tearing down a resource retained by another generation. + +## Requirement 5: Preserve Immutable Generation and Candidate-Isolation Semantics 5.1. Published `GenerationRuntime` objects remain immutable; no connector resource shall be reconfigured or replaced underneath a published generation. -5.2. If connector identity changes, the candidate shall construct a distinct replacement resource before publication while the old generation retains its old resource until its leases drain. -5.3. If a connector is removed or disabled, the candidate shall acquire no replacement lease; old generations may continue using the removed connector until their existing leases drain. +5.2. If connector identity changes, the candidate shall construct a distinct replacement before publication while the old generation retains its old resource until its leases drain. +5.3. If a connector is removed or disabled, the candidate shall acquire no replacement lease; old generations may continue using the removed connector until their retained work drains. 5.4. A failed candidate that leased an existing resource shall release only its candidate lease and shall not disturb the active generation's lease/resource. -5.5. A failed candidate that created a new resource shall release it through rollback; if no other lease exists, physical cleanup shall run before rollback completes. -5.6. Existing generation-owned derived state—including executor maps/views, routing views, model-registry runtime, model catalog, feature composition, policy state, and generation lifecycle context—shall remain generation-owned and shall not be moved into the connector resource pool. -5.7. Existing no-drop, retained-generation, old-stream, and last-good reload guarantees shall remain unchanged. -5.8. A reused configured connector may contribute the same underlying immutable backend/session functions to multiple generation-local executor maps, but each generation shall rebuild its own projections, inventories, routing and model views using the same existing generation compiler sequence. -5.9. On a reuse hit, candidate preparation shall not invoke `Configure`, `Start`, `Stop`, `Close`, a mutating preflight, or another generation-local mutation on the shared physical connector. Candidate rejection or rollback shall release only the candidate lease and shall never invalidate the shared resource merely because the candidate was rejected. -5.10. Query-shaped metadata operations already represented by the backend-plugin contract, such as `Resolve` and `ListModels`, may remain part of generation-local preparation/refresh against a reused configured instance. This does not permit hidden reconfiguration by the host. -5.11. If future generation preparation or an external adapter requires a mutating lifecycle action against the configured connector, that resource path shall become non-shareable and use isolated physical construction until a separate design proves safe cross-generation reuse. - -## Requirement 6: Invalidation and Resource Incarnations - -6.1. Semantic resource identity and physical resource incarnation shall be distinct concepts: the same semantic identity may need a later fresh physical incarnation after failure. -6.2. When the existing connector/process invalidation path declares a physical resource generation unusable, that exact resource incarnation shall become non-acquirable for future candidates. -6.3. A future acquisition for the same semantic identity after invalidation shall create a fresh physical incarnation rather than returning the invalidated one. -6.4. Invalidation shall not live-swap a replacement resource into generations that already reference the failed incarnation. Their normal backend/process failure and recovery semantics remain authoritative. -6.5. An invalidation callback originating from an older/detached incarnation shall not evict or invalidate a newer current incarnation for the same semantic identity. -6.6. Existing `processhost` generation invalidation/process reap behavior shall remain authoritative for the physical process; reconciliation only controls future resource reuse eligibility and lease lifetime. -6.7. Invalidation shall detach the exact failed incarnation from future semantic acquisition before or atomically with delegating to the existing processhost invalidation path, so a concurrent candidate cannot acquire a resource already known to be unusable. -6.8. Detached/invalidated entries may remain referenced by generations that already leased them until those leases release; they shall not remain discoverable as the current reusable entry. - -## Requirement 7: Cleanup, Shutdown, and Error Preservation - -7.1. There shall be exactly one physical cleanup owner for each configured connector resource. A generation-specific closer shall never be capable of closing a physical resource still leased by another generation. -7.2. The process-scoped reconciliation owner shall close before `processhost.Host` during normal process teardown so remaining physical resource cleanup can still use the live host/session ownership path. -7.3. Normal successful shutdown ordering shall remain: generation admission stops and generations drain/release their leases; then reconciliation fail-safe cleanup closes any residual physical entries; then `processhost.Host` closes; then verified artifact handles close; then staging removal runs. -7.4. Bootstrap/error cleanup before process ownership transfer shall preserve the same dependency order for any reconciliation owner, host, artifacts, and staging resources already acquired. -7.5. Process shutdown shall reject new resource acquisitions, be idempotent, and fail-safe close any residual physical resources after generation drain without creating a second process shutdown coordinator. -7.6. Verified artifacts and staging resources shall retain their existing later teardown ordering after all connector resources and `processhost.Host` are done with them. -7.7. Existing cleanup-error normalization and error-join behavior shall be preserved. Lease-triggered final physical cleanup errors shall surface through the existing generation rollback/close path rather than a new public error category. -7.8. Partial construction failure shall not leak sessions, host instances, processes, IPC connections, pool entries, or leases. -7.9. `ProcessServices` shall own reconciliation shutdown through its existing private process resource ownership mechanism; no second closer stack or independent process shutdown graph shall be introduced. - -## Requirement 8: Connector Semantics and Non-Interference - -8.1. Reuse is valid only for a configured connector resource whose construction inputs are fully represented by the physical resource identity. If the implementation cannot prove identity completeness for an eligible path, that path shall fall back to current generation-local construction rather than risk unsafe reuse. -8.2. Dynamic provider facts that are already modeled through runtime calls such as `Resolve`, `ListModels`, health/readiness, or normal execution shall continue through those calls and shall not require connector reconstruction on unrelated Go-LIP generation changes. -8.3. The optimization shall not alter canonical requests/events, route selection, retries, failover, output-commit rules, streaming order, cancellation, billing finalization, accounting evidence, token counting, or provider-specific semantics. -8.4. No resource-pool lookup or lock shall be added to the normal request execution hot path; acquisition/release occurs at generation construction/retirement boundaries. -8.5. Existing backend-plugin security invariants—verified artifact binding, secure local IPC, peer authentication before secrets/configure, environment restrictions, and process-tree cleanup—shall remain unchanged. -8.6. Old and new generation behavior shall remain observationally equivalent to the current overlap model for unchanged resources except that unnecessary physical connector reconstruction is removed. -8.7. Connector-specific configuration parsing remains inside the connector. The host identity may fingerprint opaque configure bytes but shall not learn provider-specific configuration schemas merely to decide reuse. -8.8. Reusing one configured instance across overlapping generations shall not imply that all external process failures are isolated per generation; however, the optimization shall not add candidate-owned mutation that creates a new failure path for the last-good generation. Existing connector/process failure and invalidation semantics remain authoritative. -8.9. Overlapping generation access to query-shaped metadata operations on a reused configured instance shall be covered by race/conformance tests. The implementation shall not move the model registry into the pool merely to avoid concurrent metadata calls. +5.5. A failed candidate that created a new resource shall release it through rollback; if no other claim exists, physical cleanup shall complete before rollback completes. +5.6. Generation-owned derived state—including executor maps/views, routing views, model-registry runtime, model catalog, feature composition, policy state, billing composition, and generation lifecycle context—shall remain generation-owned and shall not move into the connector pool. +5.7. Existing no-drop, retained-generation, old-stream/async, and last-good reload guarantees shall remain unchanged. +5.8. A reused configured connector may contribute the same underlying backend/session functions to multiple generation-local executor maps, but each generation shall rebuild its own normal projections and lifecycle structures. +5.9. On a reuse hit, candidate preparation shall not invoke `Configure`, `Start`, `Stop`, `Close`, mutating preflight, or another generation-local mutation on the shared physical connector. Candidate rejection/rollback shall never invalidate the shared resource merely because the candidate was rejected. +5.10. Query-shaped operations already represented by the backend-plugin contract, such as `Resolve` and `ListModels`, may remain part of generation-local preparation/refresh subject to Requirement 8 concurrency gates. +5.11. If future generation preparation or an external adapter requires a mutating lifecycle action against the configured connector, that resource path shall become non-shareable until a separate design proves safe reuse. + +## Requirement 6: Invalidation, Detached Entries, and Physical Incarnations + +6.1. Semantic resource identity and physical resource incarnation shall be distinct concepts: the same semantic identity may later require a fresh physical incarnation after failure. +6.2. When the existing connector/process invalidation path declares a physical incarnation unusable, that exact entry shall become non-acquirable for future candidates before or atomically with delegating to processhost invalidation. +6.3. Invalidation shall remove only the exact failed incarnation from the `current` semantic index. A stale callback from an older incarnation shall not detach a newer current incarnation. +6.4. Detached/invalidated entries may remain referenced by generations that already leased them and shall remain tracked by the process reconciliation owner until their physical cleanup has completed. +6.5. The pool shall maintain an ownership set or equivalent enumeration of **all successfully constructed but not-yet-physically-cleaned entries**, including detached entries, so terminal process shutdown can fail-safe clean them. +6.6. A future acquisition for the same semantic identity after invalidation shall build a fresh incarnation rather than returning the detached entry. +6.7. Invalidation shall not live-swap a replacement into existing generations or decrement their lease claims merely because the physical incarnation failed. +6.8. Existing `processhost` generation invalidation/reap behavior remains authoritative for the physical process. Reconciliation controls only future reuse eligibility, dependent retention, and the timing of the per-resource cleanup capability. + +## Requirement 7: Exactly-Once Physical Cleanup and Shutdown Ownership + +7.1. There shall be exactly one logical physical cleanup capability for each pooled configured connector resource. It shall be an idempotent composite of the existing adapter/session cleanup and `ActivateResult.Cleanup`/`processhost.CloseInstance` path. +7.2. The reconciliation entry—not each lease—shall own the physical-cleanup once-state and stored cleanup result. Final lease release, pool shutdown, and invalidation-related terminal cleanup shall all converge on that same entry-level exactly-once operation. +7.3. `processhost.Host` does **not** transfer supervisory ownership of processes/slots/instances to the pool. It retains launch, instance tables, invalidation, reaping, and `Host.Close` fail-safe authority; the pool owns only the decision of when to invoke the existing per-resource composite cleanup while dependents still exist. +7.4. The reconciliation owner shall close before `processhost.Host` during normal process teardown so residual per-resource cleanup can still call the live host/session ownership paths. +7.5. Normal successful shutdown ordering shall remain: generation admission stops and generations drain/release leases; pool close linearizes/cancels and joins any builders; pool fail-safe cleans any residual current **and detached** entries; `processhost.Host` closes; verified artifact handles close; staging removal runs. +7.6. Bootstrap/error cleanup before process ownership transfer shall preserve the same relative pool -> host -> artifacts -> staging order for resources already acquired. +7.7. Pool shutdown shall be idempotent. `Close` racing a final lease release shall still execute physical cleanup exactly once; a later lease release after fail-safe shutdown cleanup shall be harmless. +7.8. Existing cleanup-error normalization/error-join behavior shall be preserved. Final normal lease-triggered cleanup errors surface through existing generation rollback/close aggregation; process-shutdown residual cleanup errors join the existing process close aggregation. +7.9. Partial construction failure shall not leak sessions, host instances, processes, IPC connections, pool entries, reserved claims, or builder goroutines. +7.10. `ProcessServices` shall own reconciliation shutdown through its existing private process-resource ownership mechanism; no second closer stack or process shutdown coordinator shall be introduced. + +## Requirement 8: Connector Concurrency and Non-Interference + +8.1. Reuse is valid only when the complete construction/configure identity and established host-session behavior make one configured resource safe to retain across overlapping generations; otherwise use isolated construction. +8.2. The standard production `backendplugin/host.Session` concurrency behavior shall remain unchanged. In particular, its existing `lifecycleMu` serialization of `Execute` versus `Execute`/`Close` shall not be removed or bypassed by this specification. +8.3. Sharing one standard Session therefore extends the existing per-session Execute serialization across overlapping generations. The implementation shall explicitly characterize a retained old-generation Execute overlapping a new-generation Execute and shall not claim preservation of the incidental extra execution parallelism provided today by two separately constructed sessions. +8.4. Query/auxiliary RPCs that the current standard host already permits to overlap—`Resolve`, `ListModels`, optional `CountTokens`, and optional `FinalizeBilling`—shall receive race/conformance coverage when invoked through one pooled Session across overlapping generations and alongside execution. +8.5. A non-standard/injected connector session path that cannot satisfy the established standard-host operation-concurrency contract shall be non-shareable rather than gaining a new public concurrency flag in this specification. +8.6. Dynamic provider facts modeled through runtime calls continue through those calls and do not require connector reconstruction on unrelated generation changes. +8.7. The optimization shall not alter canonical requests/events, route selection, retries, failover, output-commit rules, stream ordering within an attempt, cancellation, billing finalization semantics, accounting evidence, token counting, or provider-specific translation semantics. +8.8. No resource-pool lookup or lock shall be added to normal request execution; acquisition/release occurs at generation construction/retirement boundaries. +8.9. Existing backend-plugin security invariants—verified artifact binding, secure local IPC, peer authentication before Configure/secrets, environment restrictions, and process-tree cleanup—shall remain unchanged. +8.10. Connector-specific configuration parsing remains inside the connector. The host may fingerprint opaque Configure bytes but shall not learn provider-specific schemas merely to decide reuse. +8.11. Functional/canonical behavior for unchanged resources shall remain equivalent to the current overlap model except for explicitly documented physical-resource reuse and the resulting extension of the existing per-session Execute serialization across retained generations. ## Requirement 9: TDD, Concurrency, and Architecture Gates -9.1. Add RED tests for high-cardinality construction counts, identity discrimination, lease lifetime, rollback, invalidation/incarnation behavior, concurrency, and shutdown ordering before enabling production reuse. -9.2. Test concurrent acquire of one absent identity, concurrent release, release racing invalidation, candidate rollback while an old generation is retained, and a fresh acquire after invalidation. -9.3. Test unchanged reload, changed same-ID config, artifact change, secret change, policy change, remove/disable, candidate failure, and process shutdown. -9.4. Add an identity-contract test or equivalent compile-time/structural gate that forces deliberate review when the external connector configure-time DTO/input surface changes. -9.5. Add race/goleak coverage for the private reconciliation owner and relevant executable connector lifecycle integration. -9.6. Architecture tests shall reject a generic service registry/container API, request-time lookup surface, public reusable-resource framework, provider-specific switch, or migration of `processhost` supervision responsibility. -9.7. Repository quality, focused executable-plugin conformance/security tests, and existing reload no-drop/rollback tests shall remain green. -9.8. Final implementation review shall remove unused abstraction layers and preserve the smallest private design that satisfies the deterministic scale and correctness gates. -9.9. Add a regression gate proving that pooled external adapters expose no generation-owned physical `Close`/`Start`/`Stop` bypass and that candidate preparation on a reuse hit is query-only with respect to the configured physical resource. +9.1. Add RED tests for high-cardinality construction counts, identity discrimination, waiter reservation, lease lifetime, detached-entry ownership, rollback, invalidation/incarnation behavior, Acquire/Close linearization, builder cancellation, exactly-once cleanup, operation concurrency, and shutdown ordering before enabling reuse. +9.2. Include scheduling-sensitive tests for: first claimant release before a waiter wakes; waiter cancellation; final release racing new Acquire; final release racing Pool.Close; invalidation followed by Pool.Close with an outstanding old-generation lease; and stale invalidation racing replacement publication. +9.3. Include a blocked physical builder that exits only on its pool-owned context cancellation; Pool.Close must cancel it, wait for it, clean any partial/success result, and prevent late publication. +9.4. Split identity evidence into two scopes: the high-cardinality generation-reload matrix covers inputs that actually vary through current reload plus invalidation; a focused physical identity/construction matrix independently proves artifact digest, secret fingerprint, process model, and normalized runtime-policy differences cannot hit the same pooled entry. `shared_artifact` remains a non-pooled fallback, not a pooled replacement case. +9.5. Add a DTO/input drift gate forcing deliberate identity review when the external configure-time physical input surface changes. +9.6. Add race/goleak coverage for the private pool/builders and overlapping standard-host metadata/auxiliary/execution operations across retained generations. +9.7. Add an exactly-once ownership regression that counts adapter/session cleanup, `ActivateResult.Cleanup`/host instance cleanup, pool fail-safe cleanup, and later `Host.Close`, proving the same pooled physical resource cannot be torn down twice through competing ownership paths. +9.8. Architecture tests shall reject a generic service registry/container API, request-time pool lookup, public reusable-resource framework, provider-specific switch, or migration of `processhost` supervision responsibility. +9.9. Repository quality, focused executable-plugin security/conformance tests, existing backend recomposition/overlap tests, ResourceLedger lifecycle tests, and reload no-drop/last-good tests shall remain green. +9.10. Final implementation review shall remove unused abstraction layers and preserve the smallest private design that satisfies deterministic scale and correctness gates; pooled reuse shall be re-scoped if the established Session concurrency model makes the measured overlap behavior operationally unacceptable. diff --git a/.kiro/specs/backend-connector-resource-reconciliation/research.md b/.kiro/specs/backend-connector-resource-reconciliation/research.md index 1f48fa7d..48e4c860 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/research.md +++ b/.kiro/specs/backend-connector-resource-reconciliation/research.md @@ -4,270 +4,250 @@ - **Feature**: `backend-connector-resource-reconciliation` - **Discovery Scope**: brownfield runtime optimization / connector-scale lifecycle refactor -- **Key Findings**: - - Go-LIP already solves the difficult correctness problem with immutable generations, request/async generation leases, `ResourceLedger`, and manager-owned retirement. Those mechanisms should remain authoritative. - - Catalog cardinality is already cheap: executable connector discovery is manifest-only and lazy with respect to process launch. The scaling pressure is the number of **enabled configured executable connector instances** rebuilt by a material generation reload. - - Current `buildBackends` constructs every enabled backend in every generation. Discovered `per_instance` factories deliberately mint unique host activation IDs so candidate and active generations can overlap, which also guarantees unchanged physical connectors are duplicated during candidate construction. - - The highest-ROI Cordis-v4 idea for this specific seam is provider identity + reconciliation + dependent lifetime retention: exact unchanged configured connector resources can be retained by more than one immutable generation and closed after the final generation releases them. - - A generic Cordis runtime remains a poor fit. The selected design is one private connector-specific process owner that returns per-generation leases and delegates all physical process supervision to existing `processhost.Host`. - - Physical reuse needs a stronger identity than `BackendStateIdentity`: exact artifact, opaque configure payload, runtime policy, secret fingerprint, process model, logical instance/factory identity, and any future configure-affecting input must be represented or reuse must fail closed. - -## Research Log - -### Cordis v4: the transferable principle - -- **Context**: Re-evaluate the user-supplied paper *A Programming Paradigm for Spatiotemporal Composability* (Yifan Shi, Wei Zhang, Tianyi Cui) against the current backend connector architecture. -- **Findings**: - - Cordis models components/providers by identity rather than only current values and reconciles a desired component graph against current runtime state. - - Dependents retain access to a withdrawing provider while their teardown runs; the provider is physically removed only after dependent cleanup. - - Revertible effects make rollback/teardown explicit and reverse effects when a component is withdrawn. - - Cordis's generic reactive coeffect graph, fibers, HMR, and dynamic service context solve a broader class of application-platform problems than Go-LIP needs here. -- **Implications**: - - Borrow semantic provider identity, physical incarnation identity, reconciliation, and reference-retained teardown. - - Map a Go-LIP generation to a dependent/lease holder, not to a Cordis component graph. - - Keep generation publication/retirement as the consistency boundary and avoid live dependency replacement under requests. - -### Prior Cordis-derived ownership work - -- **Context**: Determine whether `.kiro/specs/archive/atomic-owned-resource-lifecycle` already addressed this problem. -- **Sources Consulted**: - - archived `atomic-owned-resource-lifecycle` research/design - - `internal/infra/runtimebundle/process_owner.go` - - `internal/infra/runtimebundle/resource_ledger.go` -- **Findings**: - - The previous spec addressed acquisition/cleanup locality and worker ownership. - - It deliberately left backend lifecycle unchanged because `BackendBuildResult` already paired a backend with cleanup and `buildBackends` immediately transferred that cleanup to `ResourceLedger`. - - Main now contains `processResourceOwner` and `acquireOwnedProcess`, confirming that process cleanup locality is already hardened. -- **Implications**: - - This spec is not a correction to the prior work. It addresses a different dimension: **one expensive physical backend resource being useful to several overlapping generations**. - - Reuse existing ownership authorities; do not invent another general effect/owner framework. - -### Current generation construction and backend rebuild behavior - -- **Context**: Identify whether material reload reconstructs unchanged backend resources. -- **Sources Consulted**: - - `internal/infra/runtimebundle/build_model.go` - - `internal/pluginreg/lifecycle.go` - - `internal/infra/runtimebundle/reload_backend_recomposition_test.go` -- **Findings**: - - `buildModelRuntime` calls `buildBackends` for every candidate generation. - - `buildBackends` iterates every enabled backend row and calls `BuildBackendWithLifecycle`. - - A changed same-ID backend is intentionally constructed separately so old pinned generations retain old behavior while new generations use the replacement. - - This same whole-set construction happens for unchanged rows on unrelated material generation changes. -- **Implications**: - - Do not change changed/remove semantics. - - Introduce reuse only at the physical connector construction seam for exact unchanged identities. - - Continue rebuilding generation-local executor maps, inventories, model registry, routing projections, and policy views. - -### Discovered executable connector activation - -- **Context**: Determine where physical duplication originates and where reconciliation should sit. -- **Sources Consulted**: - - `internal/infra/runtimebundle/discovered_factories.go` - - `internal/infra/backendplugins/processhost/host.go` - - `internal/infra/backendplugins/processhost/model.go` - - `internal/infra/runtimebundle/reload_discovered_overlap_test.go` -- **Findings**: - - `InstallDiscoveredExports` registers one generic lifecycle factory per validated manifest export; there is no provider-specific switch. - - `buildDiscoveredBackend` encodes opaque YAML, activates/configures the host instance, builds an adapter backend, and returns physical cleanup. - - For `per_instance`, the runtime deliberately mints a distinct host activation handle for each candidate construction so old/new generations never collide in `Host.instances`. - - `processhost.Host` already has correct lazy launch, singleflight process slot creation, peer authentication, instance tracking, generation invalidation, and cleanup. - - `shared_artifact` has different isolation/concurrency and overlap policy semantics. -- **Implications**: - - Put reconciliation **above** `processhost`, not inside it. - - Preserve the unique host activation handle for each newly created physical incarnation. - - Reuse prevents calling physical construction at all for unchanged identity; it does not teach `processhost` about LIP config generations. - - Exclude `shared_artifact` initially. - -### Discovery cardinality versus live cardinality - -- **Context**: Validate whether hundreds of installed connectors themselves justify runtime reconciliation. -- **Sources Consulted**: - - `internal/infra/backendplugins/discovery/hundred_test.go` - - `docs/adr/0008-hybrid-backend-connector-plugins.md` - - `docs/backend-plugins/authoring.md` -- **Findings**: - - Discovery has explicit coverage for 100 synthetic manifests without launching them. - - Optional connectors are separate executable modules; installed but unconfigured plugins remain inactive. - - Activation is lazy and trusted artifacts are exact-digest bound. -- **Implications**: - - Do not optimize the manifest catalog or add dynamic discovery. - - The evidence harness must model many **enabled process-backed instances**, not merely many installed manifests. - -### Existing backend state identity precedent - -- **Context**: Check whether Go-LIP already distinguishes semantic backend identity across generations. -- **Sources Consulted**: - - `internal/infra/runtimebundle/backend_state_identity.go` - - `internal/infra/runtimebundle/shared_mutable.go` -- **Findings**: - - `BackendStateIdentity` namespaces process-owned affinity/health observations by instance ID, factory kind, and config digest. - - Compatible identities allow process-owned observation continuity across generation replacement; changed identities hide stale state from the new generation. - - This is a strong local precedent for identity-sensitive reuse but is purposefully narrower than physical connector interchangeability. -- **Implications**: - - Keep `BackendStateIdentity` unchanged. - - Introduce a separate private physical resource identity and avoid conflating observation-state compatibility with connector/session reuse compatibility. - -### Physical identity inputs - -- **Context**: Determine what makes two configured executable connector resources interchangeable. -- **Sources Consulted**: - - `pkg/lipsdk/backendplugin/types.go` - - `internal/infra/backendplugins/trust/artifact.go` - - `internal/infra/runtimebundle/discovered_factories.go` -- **Findings**: - - A connector is configured with logical `InstanceID`, `FactoryKind`, opaque YAML, secrets, `RuntimePolicy`, negotiation metadata, and an exact verified executable artifact/process model. - - `VerifiedArtifact.DigestHex` is the exact launch identity that must distinguish binary upgrades. - - `RuntimePolicy` contains execution bounds/timeouts/locality/environment policy that can affect configured behavior. - - Secret rotation can change configured credentials without changing public YAML. -- **Implications**: - - Physical identity must fingerprint every generation-varying construction/configure input. - - Secret values are hashed locally with stable sorted framing and never surfaced. - - Prefer one explicit identity builder at the same seam where effective configure input is assembled. - - Add a contract test that makes future configure DTO/input additions require deliberate identity review. - - When completeness cannot be proven, fall back to non-shared construction. - -### Adapter/backend cleanup shape - -- **Context**: Check whether the resulting backend value itself can safely be shared across executor maps. -- **Sources Consulted**: - - `internal/infra/backendplugins/adapter/backend.go` - - `internal/infra/backendplugins/processhost/build_result.go` - - `internal/infra/runtimebundle/build_model.go` -- **Findings**: - - `adapter.Build` creates an `execbackend.Backend` whose functional fields close over the configured `ExecuteSession` and resolved profile. - - Physical session cleanup is retained in `processhost.BuildResult` rather than embedded as an ordinary request-time operation. - - `buildDiscoveredBackend` combines adapter cleanup with `ActivateResult.Cleanup`, then returns it as `pluginreg.BackendBuildResult.Cleanup`. - - `buildBackends` currently registers that cleanup into the generation `ResourceLedger`. -- **Implications**: - - A reconciliation entry can own the physical `BackendBuildResult` and physical cleanup. - - Each generation must receive the same immutable backend functional value plus a **new lease-release cleanup**, never the underlying physical cleanup. - - Tests must prove no generation-local `Close`/lifecycle path can bypass the lease for pooled external resources. - -### Process ownership transfer and teardown order - -- **Context**: The discovered host exists before `ProcessServices`; determine how a pool can be captured by factories while still being process-owned. -- **Sources Consulted**: - - `internal/infra/runtimebundle/plugin_catalog.go` - - `internal/infra/runtimebundle/composition_root.go` - - `internal/infra/runtimebundle/host_build.go` - - `internal/infra/runtimebundle/process_services.go` -- **Findings**: - - `prepareDiscoveredPluginInstall` creates staging/trust artifacts and `processhost.Host` before factory registration. - - `InstallDiscoveredExports` registers factory closures before `NewProcessServices` freezes discovery. - - The host/artifacts/staging ownership bundle then transfers into `ProcessServices`. - - Existing reverse teardown intentionally closes host before artifacts and artifacts before staging. -- **Implications**: - - Create the private backend resource pool beside the discovered host during install preparation so closures capture it directly. - - Transfer the pool with the host into `ProcessServices` using a package-private construction field/seam. - - Register pool close after host close registration so reverse teardown runs pool → host → artifacts → staging. - - Error paths before transfer use the same dependency order. - -### Failure and incarnation semantics - -- **Context**: A desired identity can remain unchanged while its current process/session fails. -- **Findings**: - - Existing adapter invalidation can invalidate a `processhost` process generation. - - A semantic-key-only cache would risk returning the known-dead entry to later candidates. - - The same semantic identity must be able to acquire a fresh physical process/session after failure. -- **Implications**: - - Track physical incarnation identity independently from semantic identity. - - Invalidation detaches the exact current incarnation before/with delegating to existing host invalidation. - - New acquisition may build a fresh incarnation under the same semantic key. - - Old leases stay bound to the old incarnation; no published generation is live-mutated. - - Incarnation token comparison prevents a delayed stale invalidation from removing a newer entry. - -### ROI measurement strategy - -- **Context**: Candidate compilation is already fast on ordinary synthetic fixtures, while real connector startup is platform/provider dependent. -- **Findings**: - - Existing reload benchmarks are useful but do not model hundreds of enabled external process-backed connectors. - - Wall-clock startup varies with OS process creation, IPC, connector implementation, machine load, and CI environment. - - The architectural cost being removed has deterministic work-count semantics independent of timing noise. -- **Implications**: - - Primary gates are counts: builds, activations/launches, configure calls, physical cleanup. - - Add a 100-enabled-connector benchmark/evidence fixture and record wall time/allocations as supporting data. - - Expected unchanged reload changes from O(N) physical construction to O(N) lightweight lease acquisitions with zero physical reconstruction. - - Mixed reload with K changed connectors performs physical construction proportional to K, not N. +- **Selected Cordis principle**: semantic provider identity + physical incarnation + dependent retention/reconciliation. +- **Not selected**: generic Cordis component runtime, reactive dependency graph, fibers, service locator, HMR, or generic effect/resource framework. +- **Primary target**: unchanged discovered executable `per_instance` connector resources reconstructed across overlapping immutable generations. +- **Post-review hardening**: detached-entry ownership, reserved waiter claims, entry-level exactly-once physical cleanup, pool-owned builder cancellation, explicit Acquire/Close linearization, exact cleanup handoff to processhost, identity-evidence split, and established host-session concurrency characterization. -## Architecture Pattern Evaluation +## Brownfield Findings + +### Generation correctness is already solved at the right coarse boundary + +Go-LIP already provides immutable `GenerationRuntime`, request/async generation retention, transactional last-good publication, `ResourceLedger` rollback/retirement, and manager-owned drain. Those mechanisms remain authoritative. A Cordis-like dynamic component graph would duplicate existing correctness machinery. + +### The scale pressure is enabled live connector count, not manifest count + +Discovery is manifest/trust oriented and does not launch every installed connector. The expensive case is many **enabled configured external connector instances** rebuilt during a material generation change. `buildBackends` constructs every enabled backend row for every candidate generation. + +### `per_instance` discovered connectors deliberately duplicate across generations today + +`buildDiscoveredBackend` mints unique host activation IDs for `per_instance` so active and candidate generations can safely coexist. This is correct for changed connectors but also means an unchanged connector is physically Activated/Configured again during unrelated material reload. + +### Reconciliation belongs above processhost + +`processhost.Host` already owns lazy process launch, secure local IPC, peer authentication, slot/instance bookkeeping, process-generation invalidation, and process-tree cleanup. It should not learn Go-LIP semantic generation/config identity. The new owner is therefore a private runtimebundle connector-resource reconciliation layer that calls the existing physical builder/host. + +### Prior Cordis-derived ownership work remains valid + +The archived `atomic-owned-resource-lifecycle` spec hardened process acquisition/cleanup locality and generation loop ownership but deliberately left backend lifecycle alone because `BackendBuildResult` already paired backend + cleanup and `buildBackends` transferred cleanup to `ResourceLedger`. This spec addresses a different problem: **one expensive physical backend resource may be retained by multiple overlapping generations**. It reuses rather than replaces those ownership authorities. + +## Physical Identity Research + +### Existing `BackendStateIdentity` is precedent, not the key + +Go-LIP already reuses process-owned affinity/health observation state across compatible generations using `{InstanceID, FactoryKind, ConfigDigest}`. That demonstrates identity-sensitive continuity is locally idiomatic, but physical connector interchangeability is stricter. + +### Physical construction inputs + +Current discovered construction uses/captures: + +- logical `InstanceID`; +- factory kind; +- exact verified artifact digest; +- process model/sharing profile; +- opaque YAML Configure bytes; +- normalized `RuntimePolicy`; +- `SecretBundle` when supplied; +- negotiation/session behavior from the fixed executable/host protocol. + +Therefore a separate private physical identity must treat artifact, process model, config bytes, policy, secrets, factory and logical instance deliberately. Secret values are locally hashed with deterministic length framing and never surfaced. + +### Startup-fixed versus reload-varying facts + +The production discovered factory closure is installed at startup. Artifact/process model and `DiscoveredInstallOptions.RuntimePolicy` are captured there. Current SIGHUP reload does not rediscover/replace the artifact. The current discovered builder also does not inject changing secret material through the shown production path. + +Consequences: + +- these fields still belong in the **identity contract** because they define physical semantics and may evolve later; +- high-cardinality **generation reload** evidence should not pretend artifact/process-model/policy changes are currently hot-reloadable; +- focused identity/construction tests exercise those dimensions directly; +- `shared_artifact` remains a non-pooled/restart-required fallback rather than a “changed pooled process model” case. + +This resolves CodeRabbit's request for broader identity coverage without inventing unsupported reload capabilities. + +## Physical Cleanup and processhost Ownership + +### Current composite cleanup + +The current discovered builder combines: + +1. adapter `BuildResult.Cleanup()` -> `session.Close(...)`, which closes the configured connector instance/host session; +2. `ActivateResult.Cleanup` -> `processhost.Host.CloseInstance(hostActivationID)`, which updates host instance ownership and reaps the slot/process when appropriate. + +The adapter BuildResult itself is once-guarded, and processhost reaping is idempotent, but pooling adds another possible caller (`Pool.Close`). A **lease-level** once guard alone is therefore insufficient. + +### Selected ownership wording + +`processhost.Host` keeps supervisory ownership: process/slot tables, peer identity, invalidation, reaping, and `Host.Close` fail-safe behavior stay there. + +The pool entry consumes the existing **per-resource composite cleanup capability** when a physical build succeeds. Generations never receive that composite; they receive only lease release. The pool therefore controls *when* the per-resource cleanup may be invoked, while processhost remains the component that actually supervises/reaps physical processes. + +An entry-level cleanup-once state is shared by normal final lease release and process shutdown. This avoids a final-release-versus-Pool.Close double cleanup race. -| Option | Description | Strengths | Risks / Limitations | Decision | -|---|---|---|---|---| -| Full Cordis runtime | Generic provider graph, fibers, effects/coeffects, reactive reconciliation | broad composability | fights Go-LIP generation model; duplicates lifecycle machinery; high maintenance cost | Reject | -| Put generation reconciliation in `processhost` | Make host understand config-generation identity and reuse | central process knowledge | conflates physical supervision with semantic LIP generation composition | Reject | -| Reuse `BackendStateIdentity` directly | Treat existing observation-state key as physical resource key | smallest apparent change | unsafe: misses artifact, policy, secrets and future configure inputs | Reject | -| Cache `BackendBuildResult` per generation-independent key | Return same cleanup/value to multiple generations | simple lookup | double/early cleanup; no incarnation-safe invalidation | Reject | -| Process-owned connector resource leases | Private semantic identity → current physical incarnation → generation leases; host remains physical supervisor | focused ROI, preserves generations, scales with unchanged enabled connectors | identity and concurrency correctness require strong tests | **Select** | -| No change | Rebuild every enabled connector each material generation | simplest correctness model | O(N) physical activation wave and overlap resource spike at high enabled cardinality | Keep as fallback for non-shareable paths | +## Detached Resource Ownership -## Design Decisions +Initial design only indexed `current[semanticIdentity]`. That is insufficient after invalidation: -### Decision: Reconcile only configured physical external connector resources +```text +identity X -> incarnation 7 current +invalidate 7 +current[X] removed +Gen17 still has lease to incarnation 7 +``` -- Keep executor maps, model registry/catalog, routes, feature surfaces, generation lifecycle and policies generation-local. -- Share only the configured adapter/backend session resource for exact eligible identities. -- Rationale: this removes expensive reconstruction without weakening the generation consistency boundary. +If ProcessServices then closes before that lease is released, a Pool.Close that only walks `current` cannot enumerate incarnation 7. -### Decision: Create the pool during discovered install, own it through ProcessServices +Selected correction: retain an `owned`/all-entry set containing every successfully constructed physical incarnation until entry-level physical cleanup completes. Invalidation only detaches from `current`; it does not surrender process-level ownership bookkeeping. Pool Close snapshots all residual owned entries, including detached/invalidated ones, and invokes the same once-guarded physical cleanup. -- Factory closures need the pool before `ProcessServices` exists. -- Construct pool next to `processhost.Host`, capture directly, then transfer process lifetime into `ProcessServices`. -- Do not use a global map or setter/locator. +## Acquire/Waiter Concurrency -### Decision: One semantic current entry plus explicit physical incarnation +### Why post-publication waiter ref increments are unsafe -- Semantic identity expresses desired configured resource equality. -- Incarnation identifies the concrete live process/session created for that desire. -- Invalidation detaches an incarnation without banning the semantic identity forever. +The original pending-entry idea let waiters observe `building`, wait on readiness, and increment refs only after publication. A scheduler can then do: -### Decision: Physical cleanup lives in the entry; ResourceLedger owns a lease release +1. first caller builds, receives lease; +2. waiter is still sleeping/not yet incremented; +3. first caller releases, refs reach zero, physical resource closes; +4. waiter wakes and attempts to acquire the now-closed entry. -- Underlying `BackendBuildResult.Cleanup` is consumed by the pool and never copied to multiple generations. -- Each Acquire returns an independent idempotent release closure. -- Final release performs physical cleanup exactly once. +Selected correction: **every waiter reserves its prospective claim under the pool mutex before waiting**. The initiating caller is also just a claimant. A waiter cancellation abandons only its own reservation. -### Decision: No idle cache +### Builder ownership -- Reuse exists only across overlapping retained generations. -- Final lease release closes immediately. -- This avoids cache sizing, TTLs, memory/process retention, eviction races, and another operational knob. +The initiating Acquire must not own the physical builder lifetime. Otherwise caller cancellation and contextless `ProcessServices.Close()` can leave an unbounded build blocking shutdown. -### Decision: No public opt-in flag +Selected correction: -- Exact-identity reuse is an internal semantic-preserving optimization. -- Unsafe/incomplete identities fall back automatically to current construction. -- A public feature flag would expose internal lifecycle structure and create configuration/test matrix cost without adding user capability. +- absent Acquire creates a pending entry and reserves its claim; +- the pool starts one short-lived builder goroutine under a pool-owned cancellation context; +- all Acquires wait on entry readiness or their own context; +- Pool.Close sets closing, cancels the build root, then waits builders; +- build completion after closing may clean a result but cannot publish it as reusable. -### Decision: Preserve unique host activation IDs for newly built incarnations +This uses existing processhost/transport context cancellation rather than adding a background worker subsystem. + +## Acquire/Close Linearization + +`ProcessServices.Close` is contextless and synchronous, so the pool needs a precise terminal contract. + +Selected contract: + +- `Close` linearizes by setting `closing=true` under the same mutex used for Acquire claim reservation; +- after that point no new claim reservation is accepted and no pending builder result is handed off as a post-close success; +- Close cancels pool-owned builders; +- Close waits builders and acquisition handoff activity; +- only then does it snapshot/clean residual owned entries; +- no physical build/cleanup/wait happens while the mutex is held. + +This is stronger than simply “reject new Acquire” and directly addresses the lease-after-cleanup race CodeRabbit identified. + +## Invalidation and Incarnations + +Semantic identity says the desired configuration is unchanged. Physical incarnation says which concrete process/session currently realizes it. + +Invalidation: + +- compares exact entry/incarnation; +- detaches that entry from `current` before/with host invalidation; +- leaves the detached entry in process ownership until cleanup; +- does not decrement existing generation claims or live-swap a replacement; +- allows a future candidate to build a fresh incarnation for the same semantic key; +- stale old-incarnation callbacks cannot remove the new current entry. + +## Candidate Isolation + +Fresh physical processes currently give candidate preparation strong failure-domain isolation. Reuse is allowed only for query-shaped candidate preparation: + +- no Configure/Start/Stop/Close/mutating preflight on a reuse hit; +- rollback releases only the candidate lease; +- candidate failure alone does not invalidate the resource; +- future mutating preparation makes the path non-shareable until separately designed. + +Generation-local model registry/catalog/routing/policy/billing state remains generation-owned. + +## Established Operation Concurrency + +### Standard host Session behavior + +Cross-checking `pkg/lipsdk/backendplugin/host/session.go` shows: + +- `Session.Execute` holds `lifecycleMu` across the complete Execute RPC, so two Execute calls on one Session are serialized and Close cannot race underneath an active Execute; +- `Resolve`, `ListModels`, CountTokens, and FinalizeBilling are not serialized by that client lifecycle mutex; +- server-side instance leasing already allows metadata/auxiliary calls to overlap an active configured instance operation while protecting Close. + +Therefore pooling one production Session across generations does **not** justify inventing a new concurrency contract. The implementation must preserve the existing standard-host behavior and test it. + +### Important overlap tradeoff + +Today Gen17 and Gen18 fresh physical Sessions create two independent Execute serialization domains during candidate/retirement overlap. Pooling an unchanged connector makes retained-old and new-generation Execute calls share one Session serialization domain. + +That can reduce transient overlap concurrency / create head-of-line waiting for long streams. It is not a canonical request mutation, but it is operationally material and must not be hidden behind an unconditional “observationally equivalent” claim. + +Selected treatment: + +- do not change `Session.Execute` concurrency in this spec; +- characterize a long retained old-generation Execute overlapping a new-generation Execute; +- cover metadata/Count/Finalize/execution overlap under race/conformance tests; +- if established serialization makes the optimization unacceptable for target workloads, re-scope pooling rather than widening this spec into ABI/session-concurrency redesign; +- injected/non-standard session paths are non-shareable unless they satisfy the same established host contract. + +## Scale/ROI Evidence Strategy + +Primary correctness/ROI is deterministic work count: + +```text +before: O(N enabled eligible connectors) physical build per material generation +after: O(K changed/unusable physical builds) + O(N) cheap claims/projections +``` + +Use at least 100 synthetic enabled discovered `per_instance` connectors. Count physical build, Activate/launch, Configure, physical cleanup, lease acquire/release. Wall time/allocation is supporting evidence only. + +Two matrices are required: + +1. **Reload matrix**: unchanged, config changes, remove/disable, candidate rollback, invalidation/rebuild. +2. **Physical identity matrix**: artifact, secret, normalized runtime policy, process model, factory/logical identity. + +Also record the cross-generation Session Execute serialization behavior so lifecycle savings are evaluated together with the actual overlap scheduling tradeoff. + +## Architecture Pattern Evaluation -- The pool prevents unnecessary build calls for unchanged resources. -- When a build is actually required, current host-instance uniqueness remains intact, preserving overlap safety and current `processhost` assumptions. +| Option | Decision | Reason | +|---|---|---| +| Full Cordis runtime | Reject | duplicates generation/runtime ownership machinery | +| Put semantic reconciliation in processhost | Reject | conflates physical supervision with LIP config semantics | +| Reuse `BackendStateIdentity` | Reject | identity incomplete for physical reuse | +| Return same physical cleanup to generations | Reject | early/double teardown | +| Only track current entries | Reject | detached invalidated resources disappear from shutdown ownership | +| Waiter refs after readiness | Reject | zero-ref handoff race | +| Caller-owned physical build | Reject | shutdown cancellation/lifetime ambiguity | +| Pool-owned connector entries + generation leases | **Select** | focused ROI; preserves existing authorities | +| Idle TTL cache | Reject | speculative retention/eviction policy | +| Change Session concurrency here | Reject | broadens scope; preserve/measure existing behavior | +| No change for non-shareable paths | Keep | safest fallback | ## Risks & Mitigations -- **Identity omission causes unsafe reuse** — one construction-input identity choke point, fail-closed fallback, DTO-shape/identity contract tests. -- **One generation closes another generation's resource** — pool owns physical cleanup; generation ledgers own only idempotent lease release; no bypass closer. -- **Dead resource reused** — exact-incarnation invalidation detaches before future acquisition. -- **Stale invalidation kills new resource** — compare incarnation token before detaching current entry. -- **Pool becomes a service locator** — package-private connector-specific API, construction-only use, architecture tests forbid generic `Get`/`Resolve` and request-path access. -- **Pool duplicates processhost** — no launch/IPC/process tree/peer auth logic in pool; physical build and invalidation delegate to existing host. -- **Shutdown ordering regression** — explicit pool → host → artifacts → staging characterization tests on success and bootstrap error. -- **Resource retained after last generation** — no idle cache; refcount zero synchronously detaches and closes. -- **Race between Acquire, Release, Invalidate, Close** — small state machine with no external cleanup under mutex; race/goleak tests and per-key build serialization. -- **Optimization hides generation-specific derived state** — explicit tests prove model/routing/policy views are rebuilt while physical activation counts remain zero for unchanged resources. -- **Speculative complexity** — deterministic 100-connector operation-count gate and final simplification review; revert/re-scope if architecture cost exceeds demonstrated gain. +- **Identity omission** -> one choke point, fail-closed fallback, DTO/input drift test. +- **Detached resource leak** -> process-owned all-entry set until cleanup completes. +- **Waiter zero-ref race** -> reserve claim before waiting. +- **Shutdown hangs on builder** -> pool-owned cancelable build context + joined builder tests. +- **Acquire after shutdown cleanup** -> explicit Acquire/Close linearization and handoff join. +- **Double cleanup** -> entry-level `cleanupOnce`, not only lease once. +- **Pool/processhost ownership confusion** -> pool owns timing of composite per-resource cleanup; host remains physical supervisor/fail-safe. +- **Stale invalidation** -> exact incarnation comparison. +- **Candidate mutates last-good** -> query-only reuse; mutating path non-shareable. +- **Cross-generation Execute head-of-line blocking** -> preserve standard Session semantics, characterize, re-scope if unacceptable. +- **Scope creep** -> package-private connector-specific API and architecture tests. +- **Speculative complexity** -> 100-connector count gate and final simplification/re-scope gate. ## References -- User-supplied paper: *A Programming Paradigm for Spatiotemporal Composability* — provider identity, reconciliation, dependent teardown, revertible-effect concepts. -- `.kiro/specs/archive/atomic-owned-resource-lifecycle/` — prior focused Cordis-derived ownership hardening. -- `.kiro/specs/archive/runtime-architecture-convergence-and-shrinkage/` — canonical one-process/one-generation/one-host architecture and anti-container guardrails. -- `docs/runtime-config-reload.md` — transactional last-good generation publication/retirement contract. -- `docs/adr/0008-hybrid-backend-connector-plugins.md` — executable connector process model and lazy discovery/activation boundary. -- `internal/infra/runtimebundle/build_model.go` — all-enabled-backend candidate construction and generation-local model runtime. -- `internal/infra/runtimebundle/discovered_factories.go` — discovered lifecycle factory activation/configure/cleanup seam. -- `internal/infra/backendplugins/processhost/` — physical executable process/IPC ownership authority. -- `internal/infra/runtimebundle/backend_state_identity.go` and `shared_mutable.go` — existing semantic identity precedent for affinity/health state. -- `internal/infra/runtimebundle/process_owner.go` — current process-owned acquisition/cleanup authority facade. -- `pkg/lipsdk/backendplugin/types.go` — configure-time DTO and runtime policy identity surface. +- User-supplied paper: *A Programming Paradigm for Spatiotemporal Composability* — semantic provider identity, dependent retention, reconciliation, revertible cleanup concepts. +- `internal/infra/runtimebundle/discovered_factories.go` — current external connector construction and unique per-generation activation handles. +- `internal/infra/backendplugins/processhost/host.go` — process/instance supervision, invalidation and cleanup. +- `internal/infra/backendplugins/adapter/backend.go` and `processhost/build_result.go` — backend/session cleanup shape. +- `pkg/lipsdk/backendplugin/host/session.go` — standard host operation concurrency and Session lifecycle serialization. +- `pkg/lipsdk/backendplugin/server.go` — configured-instance leasing around RPC calls and Close. +- `internal/infra/runtimebundle/backend_state_identity.go` — existing narrower semantic identity precedent. +- `internal/infra/runtimebundle/process_services.go` / `resource_ledger.go` — process/generation cleanup authorities. +- archived `atomic-owned-resource-lifecycle` and runtime convergence specs — explicit ownership/no-container architecture constraints. diff --git a/.kiro/specs/backend-connector-resource-reconciliation/spec.json b/.kiro/specs/backend-connector-resource-reconciliation/spec.json index 6244e6e8..0bb0cba4 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/spec.json +++ b/.kiro/specs/backend-connector-resource-reconciliation/spec.json @@ -1,7 +1,7 @@ { "feature_name": "backend-connector-resource-reconciliation", "created_at": "2026-08-17T00:42:00+02:00", - "updated_at": "2026-08-17T00:59:00+02:00", + "updated_at": "2026-08-17T11:06:00+02:00", "language": "en", "phase": "tasks-generated", "approvals": { @@ -19,5 +19,5 @@ } }, "ready_for_implementation": false, - "project_description": "Reduce cross-generation churn for expensive executable backend connectors by reusing unchanged configured per-instance connector resources through complete semantic physical identity, incarnation-safe invalidation, and process-scoped reference-counted generation leases. Preserve immutable GenerationRuntime publication, ResourceLedger rollback/retirement, processhost supervision, the executable backend-plugin ABI, startup-fixed discovery, routing/streaming/accounting semantics, candidate last-good isolation, and no-drop reload. Scope the first implementation to eligible discovered per_instance connectors only; candidate reuse is query-only, mutating preparation falls back to isolated physical construction, and generation-local model/routing/policy views remain generation-owned. Add no DI container, service locator, reactive component graph, live plugin watcher, generic resource framework, public lifecycle knob, or request-hot-path lookup. Require deterministic high-cardinality operation-count evidence before and after implementation so the optimization remains justified by connector-scale behavior rather than Cordis-inspired abstraction for its own sake." + "project_description": "Reduce cross-generation churn for expensive executable backend connectors by reusing unchanged configured per-instance connector resources through complete semantic physical identity, incarnation-safe invalidation, pre-reserved generation lease claims, process-owned tracking of detached resources, pool-owned cancelable builders, explicit Acquire/Close linearization, and entry-level exactly-once physical cleanup. Preserve immutable GenerationRuntime publication, ResourceLedger rollback/retirement, processhost supervision, the executable backend-plugin ABI, startup-fixed discovery, routing/streaming/accounting semantics, candidate last-good isolation, and the established backendplugin/host.Session operation-concurrency contract. Scope the first implementation to eligible discovered per_instance connectors only; candidate reuse is query-only, mutating preparation or non-conforming session behavior falls back to isolated construction, and generation-local model/routing/policy views remain generation-owned. Add no DI container, service locator, reactive component graph, live plugin watcher, generic resource framework, public lifecycle/concurrency knob, or request-hot-path lookup. Require deterministic high-cardinality operation-count evidence plus explicit cross-generation Session scheduling evidence so the optimization remains justified by connector-scale behavior rather than Cordis-inspired abstraction for its own sake." } diff --git a/.kiro/specs/backend-connector-resource-reconciliation/tasks.md b/.kiro/specs/backend-connector-resource-reconciliation/tasks.md index 7650f0b4..d5b73f0f 100644 --- a/.kiro/specs/backend-connector-resource-reconciliation/tasks.md +++ b/.kiro/specs/backend-connector-resource-reconciliation/tasks.md @@ -2,240 +2,250 @@ ## Execution Rules -- Follow TDD: characterization/RED tests and contract gates precede production behavior changes. -- Keep every task independently reviewable and limited to at most five concrete actions. -- Preserve existing public APIs, backend-plugin ABI, configuration schema, immutable generation model, `ResourceLedger`, and `processhost` ownership. -- Prefer deletion/replacement of old per-generation physical-cleanup plumbing at the touched seam over layering a second cleanup path. -- Do not broaden scope to built-in backends, `shared_artifact`, generic resource management, dynamic discovery, or request-time lookup. +- Follow TDD: characterization/RED tests and contract gates precede production reuse. +- Keep every task independently reviewable with no more than five concrete actions. +- Preserve public APIs, backend-plugin ABI, configuration schema, immutable generations, `ResourceLedger`, `ProcessServices`, and `processhost.Host` supervision. +- Prefer replacing the touched per-generation physical-cleanup path with lease cleanup rather than layering a competing ownership path. +- Do not broaden scope to builtins, `shared_artifact`, generic resource management, dynamic discovery, Session concurrency redesign, or request-time lookup. -## Phase 1 — Establish Scale, Identity, and Lifetime RED Gates +## Phase 1 — Freeze Scale, Identity, Lifetime, and Concurrency Contracts -### Task 1.1 — Build the high-cardinality executable-connector reload harness +### Task 1.1 — Build the high-cardinality reload characterization harness -- Add a deterministic runtimebundle test fixture with at least 100 enabled synthetic discovered `per_instance` connector instances, unique logical IDs/prefixes, and no external credentials. -- Instrument physical factory/build, processhost activation/launch, Configure, physical cleanup, and later lease acquire/release counts without changing production behavior. -- Characterize the current unrelated-material-reload baseline: unchanged enabled connector rows are physically reconstructed for the candidate while the old generation remains retained. -- Add RED target assertions for unchanged reload (`0` new physical builds/activations/configures) and mixed reload (`K` changed identities -> `K` physical replacements). -- Add/extend a focused benchmark fixture for candidate compilation at representative connector cardinalities without using wall-clock thresholds as correctness assertions. +- Add a deterministic runtimebundle fixture with at least 100 enabled synthetic discovered `per_instance` connector instances and no external credentials. +- Count physical build/factory invocation, processhost Activate/launch, Configure, physical cleanup, and later lease acquire/release operations. +- Characterize the current unrelated-material-reload baseline while the old generation remains retained. +- Add RED target assertions for unchanged reload (`0` new physical builds/activations/Configure) and config-changed reload (`K` changed identities -> `K` physical replacements). +- Add a supporting candidate-compilation benchmark without wall-clock correctness thresholds. -_Requirements: 1.1–1.7, 2.2, 9.1, 9.3_ +_Requirements: 1.1–1.7, 2.2, 9.1, 9.4_ -_Validation: current production behavior is characterized deterministically; target reuse assertions are RED before implementation._ +_Validation: deterministic current O(N) reconstruction is recorded and target count assertions are RED._ -_Design: Scale and ROI Evidence Design; Deterministic 100-connector harness._ +### Task 1.2 — Lock complete physical identity and drift behavior with RED tests -### Task 1.2 — Lock physical resource identity requirements with RED tests +- Add table-driven identity tests for logical instance/factory, artifact digest, process model, opaque Configure bytes, normalized RuntimePolicy, and secret fingerprint. +- Prove config/artifact/secret/policy differences cannot alias; prove `shared_artifact` is non-pooled fallback rather than a pooled replacement case. +- Prove secret plaintext never appears in identity/debug/error/status output and `BackendStateIdentity` alone cannot authorize physical reuse. +- Add a Configure/physical-input drift gate forcing intentional identity treatment when DTO/input shape changes. +- Keep reload-varying and startup-fixed evidence distinct so tests do not imply unsupported hot artifact/policy/process-model reload. -- Add table-driven RED tests proving identity equality only when logical instance, factory, artifact digest, process model, opaque Configure bytes, effective runtime policy, and secret fingerprint are compatible. -- Prove config, artifact, credential/secret, and runtime-policy changes produce distinct identities while secret plaintext never appears in identity string/debug/error output. -- Add a contract gate that forces deliberate identity treatment when the external configure-time physical input/`RuntimePolicy` surface changes. -- Prove `BackendStateIdentity` compatibility alone cannot authorize physical connector reuse. -- Add fallback tests for an input explicitly marked non-shareable/incompletely representable. +_Requirements: 3.1–3.12, 8.1, 8.10, 9.4–9.5_ -_Requirements: 3.1–3.11, 8.1, 8.7, 9.1, 9.4_ +_Validation: focused identity/construction tests are RED and fail closed on omitted input treatment._ -_Validation: identity tests fail before the new private identity builder exists and protect DTO drift._ +### Task 1.3 — Freeze reserved-claim and entry-ownership state machine with RED tests -_Design: Physical Resource Identity; DTO Drift Gate; Fail-Closed Eligibility._ +- Add first/live/concurrent Acquire tests where every building-entry waiter reserves its prospective claim before waiting; include a deterministic first-release-before-waiter-wake schedule. +- Add cancellation/failure tests proving waiter cancellation drops only its claim, failed builds are not negative-cached, and a later Acquire can retry. +- Add invalidation tests proving exact-incarnation detach, detached entry retention in process ownership, fresh same-key replacement, and stale invalidation safety. +- Add final Release versus new Acquire and final Release versus Pool.Close races, requiring one entry-level physical cleanup outcome. +- Add an invalidation + outstanding old-generation lease + Pool.Close test proving detached residual entries are enumerated and fail-safe cleaned. -### Task 1.3 — Specify the lease/refcount/incarnation state machine with RED tests +_Requirements: 4.1–4.4, 4.8–4.11, 6.1–6.8, 7.1–7.3, 7.7, 9.1–9.2, 9.7_ -- Add RED tests for first Acquire, reuse Acquire, independent idempotent releases, and physical cleanup only on final release with no idle retention. -- Add concurrent absent-Acquire tests proving one physical build and one current incarnation with multiple leases; include waiter cancellation without canceling another caller's build. -- Add invalidation tests proving exact-incarnation detach, fresh same-semantic-key rebuild, and stale old-incarnation invalidation cannot evict the replacement. -- Add races for final Release versus new Acquire and pool Close versus a pending physical build; no caller may receive a closing/invalid entry. -- Add failed-build tests proving no permanent negative cache, no leaked ref/entry, and successful later retry. +_Validation: tests are RED against the absent pool and pin zero-ref handoff/detached/double-cleanup races._ -_Requirements: 4.1–4.10, 6.1–6.8, 7.5, 7.8, 9.1–9.2_ +### Task 1.4 — Freeze Acquire/Close linearization and builder lifetime with RED tests -_Validation: state-machine tests are RED against the absent pool and define exactly-once cleanup/concurrency behavior._ +- Add a terminal Close linearization test proving no new claim can be reserved and no pending build result can be handed off after `closing=true` linearizes. +- Add a physical builder that blocks until its **pool-owned** context is canceled; Pool.Close must cancel it, join it, and prevent late publication before returning. +- Add concurrent Close/Acquire/build-completion schedules proving residual cleanup starts only after builders and acquisition handoffs terminate. +- Prove a successful physical result arriving after Close begins is cleaned exactly once rather than published as reusable. +- Add goleak/race coverage for canceled waiters, pending builders, Close, and late completion. -_Design: Resource Pool State Machine; Concurrency Design; Invalidation and Incarnations._ +_Requirements: 4.5–4.7, 7.5, 7.7, 7.9, 9.1–9.3, 9.6_ -### Task 1.4 — Lock candidate isolation and process teardown ordering with RED tests +_Validation: shutdown-race tests are RED and define the pool's terminal linearization contract._ -- Add a candidate-rollback test where active and candidate generations share one synthetic physical resource; rollback must release only the candidate lease and leave active execution/query behavior available. -- Add an external-adapter characterization gate proving a pooled backend has no generation-owned physical `Close`/`Start`/`Stop` bypass around lease cleanup. -- Add overlapping-generation `Resolve`/`ListModels` metadata access coverage under race instrumentation while keeping model-registry/runtime projections generation-local. -- Add ProcessServices/bootstrap ownership-order tests requiring pool close before processhost, artifacts, and staging on success and partial startup failure. -- Add a non-shareable fallback test for any candidate path that requires mutating Configure/Start/Stop/Close/preflight preparation on an existing physical connector. +### Task 1.5 — Freeze cleanup handoff, candidate isolation, and standard Session concurrency -_Requirements: 2.8–2.9, 5.4–5.11, 7.1–7.9, 8.2, 8.8–8.9, 9.9_ +- Add an ownership test counting adapter/session cleanup, `ActivateResult.Cleanup`/host instance cleanup, pool cleanup, and later `Host.Close`, proving one physical resource is not torn down twice. +- Add candidate rollback with active+candidate sharing: rollback releases only the candidate claim and leaves active execution/query behavior available. +- Add a characterization gate proving pooled external backend values expose no generation-owned physical Close/Start/Stop bypass and reuse-hit preparation is query-only. +- Add standard-host overlapping-generation race/conformance for Resolve/ListModels/CountTokens/FinalizeBilling alongside execution. +- Add a retained-old-generation Execute + new-generation Execute test that explicitly observes existing `Session.Execute` serialization without changing it, plus process teardown order pool -> host -> artifacts -> staging. -_Validation: RED tests capture the last-good isolation boundary and exact pool → host → artifacts → staging dependency order._ +_Requirements: 5.4–5.11, 7.1–7.10, 8.2–8.5, 8.11, 9.6–9.7, 9.10_ -_Design: Candidate Preparation and Last-Good Isolation; Process Shutdown; No cleanup bypass._ +_Validation: last-good isolation, cleanup ownership, established Session concurrency, and teardown-order assertions are RED._ ## Phase 2 — Implement the Minimal Private Identity and Reconciliation Owner -### Task 2.1 — Implement explicit fail-closed physical resource identity +### Task 2.1 — Implement explicit fail-closed physical identity -- Add one package-private identity builder at the discovered physical construction/configure choke point using domain-separated, length-framed SHA-256 input. -- Canonically project every current `RuntimePolicy` field and deterministic opaque Configure bytes; fingerprint any non-empty `SecretBundle` with sorted, length-framed names/values without retaining plaintext. -- Include logical instance/factory identity, exact verified artifact digest, and process model; document any process-stable omitted input protected by tests. -- Return a shareability decision so incomplete/unsupported identity input falls back to current physical construction rather than producing an unsafe key. -- Make all Task 1.2 identity and privacy/DTO-drift tests green without adding production reflection or a public identity type. +- Add one package-private identity builder at the discovered physical construction/configure choke point using domain-separated, length-framed SHA-256 inputs. +- Explicitly project every RuntimePolicy field and deterministic opaque Configure bytes; hash sorted length-framed secret names/values without retaining plaintext. +- Include logical instance/factory, exact artifact digest and process model; document startup-fixed inputs while still treating them in focused identity tests. +- Return an explicit shareability decision so incomplete/unsupported input uses current isolated construction. +- Make Task 1.2 identity/privacy/drift tests green without production reflection or a public identity type. -_Requirements: 3.1–3.11, 8.1, 8.7_ +_Requirements: 3.1–3.12, 8.1, 8.10_ -_Validation: focused identity tests green; secret leak scans/errors contain no raw secret material._ +_Validation: focused identity tests green and no secret/raw identity leakage exists._ -_Design: Physical Resource Identity; Canonical Fingerprinting._ +### Task 2.2 — Implement reserved-claim entries and exactly-once physical cleanup -### Task 2.2 — Implement the connector-specific resource pool and lease state machine +- Add a package-private entry model with building/live/detached/failed state, exact incarnation token, pre-reserved claims, readiness signaling, and current semantic indexing. +- Add a process-owned set of every successful physical entry until its entry-level cleanup-once completes, including invalidated/detached entries. +- Make every generation lease release idempotently drop one claim; only final normal release detaches current and invokes the entry-level cleanup-once operation. +- Store physical cleanup only on the entry and use the same cleanup-once path for final release and process fail-safe shutdown. +- Make Task 1.3 tests green without performing physical cleanup while holding the pool mutex. -- Add one package-private `backendResourcePool`/entry/lease implementation with pending/live/detached state, exact physical incarnation tokens, refcounts, and a closing state. -- Serialize same-identity physical construction without holding the pool mutex across build/configure; make canceled waiters independent of the builder/other dependents. -- Return the immutable backend value plus a fresh idempotent generation lease-release cleanup; retain physical cleanup only on the entry and run it exactly once after final release. -- Implement exact-incarnation invalidation/detach and no-negative-cache build failure semantics, with all physical cleanup outside the lock. -- Implement idempotent Close that rejects new Acquire, joins pending builders, and fail-safe cleans residual entries before returning. +_Requirements: 4.1–4.4, 4.8–4.11, 6.1–6.8, 7.1–7.3, 7.7_ -_Requirements: 4.1–4.10, 6.1–6.8, 7.5, 7.8_ +_Validation: reserved-claim, detached ownership, invalidation and cleanup-race tests green under `-race`._ -_Validation: Task 1.3 tests green under normal and race execution; no physical operation executes while pool mutex is held._ +### Task 2.3 — Implement pool-owned physical builders and terminal Close -_Design: Private Types; Resource Pool State Machine; Concurrency Design._ +- Create one pool-owned cancellable build root; absent identity starts exactly one joined builder goroutine and all callers wait as claimants rather than owning the build. +- Pass pool builder context through pooled `processhost.Activate`/Configure instead of a background lifetime; caller cancellation abandons only that caller's reservation. +- Implement Close linearization under the pool mutex, reject later Acquires, cancel builders, then wait builders and acquisition handoffs before residual cleanup. +- Prevent a build completion after Close from publishing/handoff; clean any completed physical result through the entry cleanup-once path. +- Make Task 1.4 blocked-builder/linearization/goleak/race tests green. -### Task 2.3 — Transfer pool ownership through the existing process construction path +_Requirements: 4.5–4.8, 7.5, 7.7, 7.9_ -- Create the pool beside `processhost.Host` during discovered-install preparation and make eligible discovered factory closures capture it directly without global lookup/setter wiring. -- Extend the private discovered-install/process-build ownership bundle so pool lifetime transfers into `ProcessServices` while direct test/install paths can remain explicit and non-global. -- Register process cleanup so reverse shutdown orders pool before processhost, verified artifacts, and staging; preserve existing ordering of unrelated ProcessServices resources. -- Mirror the same dependency order on every pre-transfer/bootstrap failure path and prevent double cleanup after successful ownership transfer. -- Keep the pool connector-specific/package-private and make Task 1.4 process ownership-order tests green. +_Validation: no builder can outlive Pool.Close or publish after its terminal boundary._ -_Requirements: 2.1, 2.6–2.9, 7.2–7.6, 7.9_ +### Task 2.4 — Transfer pool ownership through existing process construction -_Validation: ownership tests prove exactly-once transfer and pool → host → artifacts → staging teardown on success/failure._ +- Create the pool beside `processhost.Host` during discovered-install preparation and capture it lexically in eligible discovered factory closures. +- Extend the private install/process-build ownership bundle so pool lifetime transfers into `ProcessServices` without global/setter lookup. +- Register cleanup in existing process ownership so reverse shutdown is pool -> host -> verified artifacts -> staging while preserving unrelated ProcessServices ordering. +- Mirror the same relative cleanup order on every pre-transfer/bootstrap failure and prevent double cleanup after ownership transfer. +- Keep pool API connector-specific/package-private and make Task 1.5 teardown/ownership tests green. -_Design: Construction and Ownership Timing; Process Shutdown._ +_Requirements: 2.1, 2.6–2.9, 7.3–7.10_ -## Phase 3 — Integrate Reuse Into Discovered `per_instance` Generation Construction +_Validation: success and partial-startup ownership transfer is exactly once and dependency ordered._ -### Task 3.1 — Split physical construction from lease acquisition at the discovered factory seam +## Phase 3 — Integrate Reuse at the Discovered `per_instance` Factory Seam -- Refactor the existing discovered backend builder into a narrow preparation/identity step plus the current physical Activate/Configure/adapter-build path without provider-specific branches. -- Preserve unique host activation IDs for every **new** `per_instance` physical incarnation; do not change `processhost.OwnershipKey` or host instance semantics. -- Route only eligible discovered overlap-safe `per_instance` resources through pool Acquire; leave builtins, `shared_artifact`, and non-shareable resources on current generation-local construction. -- Make pool hits return the existing backend functional value plus lease cleanup, while pool misses consume the underlying physical cleanup as entry ownership. -- Keep `buildBackends`/`ResourceLedger` as the generation cleanup transfer authority and make unchanged/mixed high-cardinality RED assertions green. +### Task 3.1 — Split preparation, physical construction, and lease acquisition -_Requirements: 1.4–1.5, 2.2–2.7, 4.2–4.10, 5.1–5.8_ +- Refactor discovered backend construction into effective input/identity preparation, the current physical Activate/Configure/adapter build, and pool Acquire without provider-specific branches. +- Preserve unique host activation IDs for every **new** per-instance physical incarnation and leave processhost ownership keys unchanged. +- Route only eligible overlap-safe discovered `per_instance` resources through the pool; builtins, `shared_artifact`, and non-shareable resources keep current construction. +- On a pool miss, consume the current composite adapter/session + activation cleanup into the entry; on a hit, return existing backend functions plus a fresh lease release. +- Keep `buildBackends`/`ResourceLedger` as the generation cleanup transfer authority and make unchanged/config-mixed count tests green. -_Validation: unchanged eligible reload performs zero physical reconstruction; K changed identities produce exactly K new physical builds._ +_Requirements: 1.4–1.5, 2.2–2.7, 4.2, 4.9–4.11, 7.1–7.3_ -_Design: Physical Construction Integration; Current path retained as the builder; Pool return shape._ +_Validation: unchanged reload performs zero physical reconstruction; K changed configs produce K physical builds._ -### Task 3.2 — Bind processhost failure invalidation to exact pooled incarnations +### Task 3.2 — Bind invalidation to exact pooled incarnation -- Wrap the existing adapter invalidation callback for newly built pooled resources so it first detaches the exact pool incarnation from future reuse and then delegates to `processhost.InvalidateProcessGeneration`. -- Preserve existing processhost physical reap/recovery behavior and existing cleanup-error normalization after an invalidated process is already gone. -- Ensure invalidation does not decrement generation leases or live-substitute a replacement under an existing generation. -- Prove a later same-config candidate builds one fresh physical incarnation and that an old delayed callback cannot invalidate the new incarnation. -- Keep non-pooled and `shared_artifact` invalidation behavior byte-for-byte/semantically equivalent to current behavior. +- Wrap newly built pooled adapter invalidation so it detaches only the exact pool entry/incarnation before/atomically with delegating to `processhost.InvalidateProcessGeneration`. +- Preserve processhost physical reap/recovery and cleanup normalization; invalidation does not decrement generation claims or live-substitute a replacement. +- Keep detached invalidated entry in the pool's process ownership set until entry cleanup completes. +- Prove later same-config Acquire builds one fresh incarnation and old delayed callbacks cannot detach it. +- Keep non-pooled and `shared_artifact` invalidation behavior unchanged. -_Requirements: 6.1–6.8, 7.7, 8.3, 8.5_ +_Requirements: 6.1–6.8, 7.3, 8.7, 8.9_ -_Validation: invalidation/restart tests green; current processhost invalidation tests remain green._ +_Validation: invalidation/replacement tests green and existing processhost invalidation suites remain green._ -_Design: Invalidation flow; No live substitution._ +### Task 3.3 — Preserve generation-local state and query-only candidate behavior -### Task 3.3 — Preserve generation-local model/routing/policy composition on reuse hits +- Continue building new generation-local inventories, model registry/catalog, executor/routing/policy/billing views, handler, lifecycle context, and `ResourceLedger` from leased backends. +- Ensure a reuse hit performs no Configure/Start/Stop/Close/mutating preflight and candidate rollback never invalidates because the candidate failed. +- Verify changed same-ID config, remove/disable, retained old stream/async work, and candidate failure retain current semantics. +- Verify standard-host metadata/auxiliary overlap across generations under race/conformance while preserving existing Session lifecycle locking. +- Keep canonical request/event, routing, failover, streaming, cancellation, accounting, billing and token-counting code paths unchanged. -- Keep building a new generation-local `BackendInventory` slice, model-registry runtime, model/catalog snapshot, executor/routing/policy views, HTTP handler, generation lifecycle context, and `ResourceLedger` from leased backends. -- Verify unchanged pooled resources can serve overlapping generations' metadata queries/executions without moving refresh loops or model state into the pool. -- Verify changed same-ID, remove/disable, retained old stream/async work, and candidate-failure rollback retain the exact current behavioral contracts. -- Ensure candidate reuse hits perform no Configure/Start/Stop/Close/mutating preflight on the physical resource; automatically bypass reuse if that condition is not satisfied. -- Keep all canonical request/event, streaming, failover, cancellation, accounting, billing and token-counting code paths unchanged. +_Requirements: 5.1–5.11, 8.4, 8.6–8.11_ -_Requirements: 5.1–5.11, 8.2–8.9, 9.9_ - -_Validation: existing backend recomposition/no-drop tests plus new query-only/cross-generation tests green._ - -_Design: Generation-Local Derived State; Candidate Preparation and Last-Good Isolation._ +_Validation: backend recomposition/no-drop suites plus query-only/cross-generation tests green._ ### Task 3.4 — Add architecture fences against scope creep and cleanup bypass -- Add architecture tests proving the pool remains private to runtime composition and is not imported/reached from request execution, public SDKs, connectors, or provider-specific packages. -- Reject generic service/container vocabulary or a reusable keyed runtime registry API introduced to support this feature. -- Lock `processhost.Host` as the only physical process/IPC supervisor and reject duplicate launch/peer/process-tree logic in the pool. -- Lock pooled external backend lifecycle so generation cleanup is a lease release rather than an alternate physical `Close`/Start/Stop path. -- Assert no public YAML/manifest/ABI option is added for this internal optimization. +- Prove the pool remains private to runtime composition and is absent from request execution, public SDKs, provider-specific packages, and connector authoring APIs. +- Reject generic service/container/keyed runtime registry APIs introduced for this feature. +- Lock `processhost.Host` as the only process/IPC supervisor; the pool may call existing cleanup/invalidation seams but not duplicate supervision logic. +- Lock generation cleanup to lease release for pooled resources and reject alternate physical lifecycle hooks bypassing entry ownership. +- Assert no public YAML/manifest/ABI/concurrency option was added for reconciliation. -_Requirements: 2.1, 2.5–2.9, 4.9–4.10, 8.3–8.5, 9.6, 9.9_ +_Requirements: 2.1, 2.5–2.9, 4.10–4.11, 7.3, 8.2, 8.8–8.10, 9.8_ -_Validation: architecture gates fail on representative forbidden fixtures and pass the intended private design._ +_Validation: representative forbidden architecture fixtures fail and intended private design passes._ -_Design: Boundary Commitments; Rejected Alternatives; No cleanup bypass._ +## Phase 4 — Certify ROI, Identity, Concurrency, and Simplicity -## Phase 4 — Prove ROI, Correctness, and Final Simplicity +### Task 4.1 — Certify the high-cardinality generation-reload matrix -### Task 4.1 — Certify the high-cardinality reconciliation matrix +- Run the 100-enabled-connector fixture for unchanged, one/K config changes, remove/disable, candidate rollback, and invalidation-then-rebuild. +- Assert physical build/Activate/Configure counts are `0` for unchanged reuse and proportional only to changed/unusable identities. +- Assert physical live-resource overlap avoids duplicating unchanged connectors and candidate rollback of reuse hits performs no physical cleanup of active resources. +- Assert candidate-only new resources clean on rollback/final release and invalidated entries can be fail-safe cleaned at process shutdown. +- Record deterministic before/after counts as implementation/PR evidence. -- Run the 100-enabled-connector harness for unchanged, one-changed, K-changed, remove/disable, candidate rollback, and one-invalidated-then-rebuild scenarios. -- Assert physical build/activation/Configure counts are `0` for unchanged reuse hits and proportional only to changed/unusable identities. -- Assert peak synthetic physical live-resource count follows active resources plus changed candidate replacements rather than duplicating every unchanged connector. -- Assert candidate rollback of reuse hits performs no physical cleanup of resources retained by the active generation and newly built candidate-only resources close at final release. -- Record the deterministic before/after operation-count evidence in implementation/PR evidence. +_Requirements: 1.1–1.7, 4.9, 5.2–5.5, 6.4–6.6, 7.5_ -_Requirements: 1.1–1.7, 4.5–4.7, 5.2–5.5, 6.3, 7.1_ +_Validation: structural O(N) physical build -> O(K) physical build claim is green without timing thresholds._ -_Validation: all count-based ROI/correctness gates green independent of wall-clock variance._ +### Task 4.2 — Certify the focused physical identity/construction matrix -_Design: Required scenarios; Design Success Criteria 1–5._ +- Exercise distinct artifact digests, secret fingerprints, normalized RuntimePolicy values, factory/logical IDs, and process models directly at the physical identity/construction seam. +- Prove each eligible identity difference misses the existing pool entry and creates a fresh resource when construction is otherwise shareable. +- Prove `shared_artifact`/other non-eligible process model uses existing non-pooled/restart-required behavior rather than a pooled replacement. +- Verify no startup-fixed field is falsely documented/tested as current SIGHUP hot-reload support. +- Re-run the DTO/input drift gate against the final production identity projection. -### Task 4.2 — Run concurrency, leak, security, conformance, and reload regression gates +_Requirements: 3.1–3.12, 9.4–9.5_ -- Run focused `-race` coverage for pool Acquire/Release/Invalidate/Close plus overlapping generation metadata/execution paths and run goleak coverage for pending-builder/shutdown cases. -- Run processhost activation/cleanup/invalidation and executable backend-plugin security/conformance suites on supported local/CI profiles. -- Run runtimebundle ResourceLedger, backend recomposition, discovered overlap/restart-required, candidate rollback, generation retention, and reload no-drop/last-good tests. -- Run repository formatting, vet/lint/quality checks and existing architecture boundaries; fix root causes without weakening assertions or skips. -- Verify no secret/config digest/raw YAML leakage was added to logs, metrics, statuses, errors, or public DTOs. +_Validation: all physical identity dimensions are covered without inventing unsupported reload behavior._ -_Requirements: 7.7–7.8, 8.3–8.9, 9.2–9.7, 9.9_ +### Task 4.3 — Run race, leak, security, conformance, and reload regression gates -_Validation: race/goleak/security/conformance/reload/quality suites green; no weakened existing gate._ +- Run targeted `-race`/goleak for reserved claims, Acquire/Close/build cancellation, invalidation, entry cleanup, and overlapping standard-host operations. +- Run processhost activation/cleanup/invalidation and executable backend-plugin security/conformance suites. +- Run ResourceLedger, backend recomposition, discovered overlap/restart-required, candidate rollback, retained-generation, and reload last-good/no-drop suites. +- Run repository formatting/vet/lint/architecture gates without weakening assertions/skips. +- Verify secret/config identity data and opaque YAML do not leak to logs, metrics, errors, statuses or public DTOs. -_Design: Testing Strategy; Risks & Mitigations._ +_Requirements: 7.7–7.10, 8.4–8.10, 9.1–9.9_ -### Task 4.3 — Record supporting benchmark evidence without overclaiming request performance +_Validation: concurrency/security/reload/repository gates green._ -- Run the high-cardinality candidate compilation benchmark before/after with stable fixture inputs and use benchstat when comparable samples are available. -- Report candidate build time/allocations and available synthetic/native process-launch/resource observations, clearly separating deterministic counts from noisy platform metrics. -- Confirm normal request execution benchmarks show no new pool lookup/lock and no material regression attributable to the feature. -- Document that the claimed gain is reload physical-resource churn/peak overlap reduction, not inference throughput or token latency. -- Keep timing data informational unless a stable repository performance budget already governs the measured path. +### Task 4.4 — Record performance and Session overlap evidence -_Requirements: 1.2, 1.6, 8.4_ - -_Validation: evidence supports O(N physical build) → O(K physical build) structural claim without unstable timing gates._ +- Run comparable high-cardinality candidate-build benchmarks and report timing/allocations separately from deterministic work counts. +- Deterministically hold a retained old-generation Execute and start a new-generation Execute on the same pooled standard Session; record the existing serialization behavior and cancellation/close outcome. +- Confirm normal request execution adds no pool lookup/lock and no material regression attributable to reconciliation. +- Document that the claimed gain is reload physical-resource churn/peak overlap reduction, not inference throughput or token latency. +- Re-scope pooled configured-session reuse if the established cross-generation Execute serialization is operationally unacceptable for intended long-lived-stream workloads; do not redesign Session concurrency in this spec. -_Design: Supporting benchmark evidence; Scale and ROI Evidence Design._ +_Requirements: 1.6–1.7, 8.2–8.3, 8.8, 8.11, 9.10_ -### Task 4.4 — Perform the final simplification and scope gate +_Validation: ROI evidence includes both lifecycle savings and the real overlap-scheduling tradeoff._ -- Review production diff for duplicate ownership stacks, generic resource/container abstractions, unnecessary wrappers, public knobs, or request-path coupling and remove them. -- Confirm builtins, `shared_artifact`, discovery/watchers, model-registry sharing, and dynamic plugin lifecycle remain outside scope. -- Confirm `ProcessServices`, `ResourceLedger`, `processhost.Host`, runtimehost generation leases, and unique host activation semantics remain the same authorities after the refactor. -- Compare implementation complexity against deterministic scale evidence; if the count target is not met or the design requires broad runtime machinery, revert/re-scope instead of shipping speculative infrastructure. -- Update implementation evidence/docs only where needed to describe the internal optimization and its preserved operational semantics. +### Task 4.5 — Perform final simplification and authority audit -_Requirements: 1.7, 2.1–2.9, 5.6–5.7, 8.3–8.6, 9.6–9.8_ +- Remove duplicate ownership stacks, generic wrappers, public knobs, request-path coupling, or unused lifecycle abstractions from the implementation diff. +- Confirm `ProcessServices`, `ResourceLedger`, `processhost.Host`, runtimehost generation refs, and unique activation IDs remain the same authorities. +- Confirm builtins, `shared_artifact`, dynamic discovery, shared model registry, and host Session concurrency redesign remain outside scope. +- Confirm every physical resource has one entry-level cleanup path and every generation owns only one lease release. +- If count/behavior gates do not justify the added lifecycle machinery, revert/re-scope rather than ship speculative architecture. -_Validation: final diff is narrowly connector-lifecycle focused, evidence-backed, and contains no new general runtime concept._ +_Requirements: 1.7, 2.1–2.9, 7.1–7.10, 9.8–9.10_ -_Design: Simplification Review; Design Success Criteria._ +_Validation: final diff remains narrowly connector-lifecycle focused and evidence-backed._ ## Requirement Coverage Matrix | Requirement | Primary tasks | |---|---| -| R1 Evidence-first scale justification | 1.1, 3.1, 4.1, 4.3, 4.4 | -| R2 Narrow eligibility/ownership boundary | 1.4, 2.3, 3.1, 3.4, 4.4 | -| R3 Physical resource identity | 1.2, 2.1 | -| R4 Acquire/lease contract | 1.3, 2.2, 3.1, 4.1 | -| R5 Immutable generation semantics | 1.4, 3.1, 3.3, 4.1 | -| R6 Invalidation/incarnations | 1.3, 2.2, 3.2, 4.1 | -| R7 Cleanup/shutdown/errors | 1.4, 2.2, 2.3, 4.2 | -| R8 Connector non-interference | 1.4, 2.1, 3.3, 3.4, 4.2–4.3 | -| R9 TDD/concurrency/architecture gates | Phase 1, 3.4, 4.2, 4.4 | +| R1 Evidence-first scale | 1.1, 3.1, 4.1, 4.4–4.5 | +| R2 Narrow boundary | 1.5, 2.4, 3.1, 3.4, 4.5 | +| R3 Physical identity | 1.2, 2.1, 4.2 | +| R4 Acquire/Close/claims | 1.3–1.4, 2.2–2.3, 3.1 | +| R5 Immutable generation/candidate | 1.5, 3.1, 3.3, 4.1 | +| R6 Invalidation/detached ownership | 1.3, 2.2, 3.2, 4.1 | +| R7 Cleanup/shutdown | 1.3–1.5, 2.2–2.4, 3.1, 4.3, 4.5 | +| R8 Concurrency/non-interference | 1.5, 3.3–3.4, 4.3–4.4 | +| R9 TDD/architecture | Phase 1, 3.4, Phase 4 | + +## Completion Gate + +Do not consider this specification implemented unless deterministic reconstruction counts meet the target **and** the hardened ownership/concurrency gates prove: reserved waiter claims, detached-entry shutdown ownership, terminal Acquire/Close linearization, pool-owned builder cancellation, entry-level exactly-once physical cleanup, candidate last-good isolation, and explicit preservation/measurement of standard Session operation concurrency.