refactor(backend): separate responsibilities with a frozen client contract

[samheutmaker]

The backend mixed provider setup, native clients, host operations, startup wiring and resource ownership inside the daemon and session store. Separate those responsibilities while preserving the existing SDK, protocol, persisted data and runtime behavior.

This consolidates all 30 original commits from the 20-PR stack plus its consolidation documentation, based on development 271c0f8d2a35648d1b45056d57432590b783483c. The original reorganization implementation is unchanged. The security/toolchain repair in #329 was merged into this branch before landing in development.

Review map

Area	Result	Original focused reviews
Provider and host responsibilities	Provider implementation lives in internal/provider; host MCP import and icon state stay server-owned. Existing configuration atomicity, lock order and cancellation are preserved.	#304, #305, #306, #308
Startup and host operations	Runtime construction has its own CLI file; host directory operations are grouped within the daemon package.	#309, #310
Resource lifetimes	Startup constructs the shared process manager and workspace coordinator; daemon owns process shutdown and Store borrows resources. Creation, sharing, transaction positions and close order are preserved.	#311, #312
Native clients and dependencies	internal/client owns the existing Go client; internal/daemonconn and internal/commandpresentation hold shared primitives. CLI/TUI/ACP imports change mechanically without a forwarding service.	#315, #317
Persistence and documentation	Store lifecycle declarations are grouped; architecture and the completed plan document current ownership. Further actor/tracing extraction is deferred.	#318, #319
Compatibility guard	Fixed original SDK/source/generated-artifact checks, behavioral comparisons, mutation controls and both-engine persisted-data rollback run in CI. The reference pin never moves.	#300, #302

The combined diff also includes the separately proposed prerequisite repairs: narrow brace-expansion lockfile patches (#299), credential-isolated test fixtures (#301), UI focus readiness (#307), gateway fixture synchronization (#313), snapshot-aware performance measurement (#314), and the TUI title fixture (#316). These remain distinct commits; the PR is not solely file moves.

Preserved contract

• TypeScript SDK and protocol 6.9 source and generated artifacts; responses, events, ordering, acceptance, deduplication, cancellation, reconnect/replay and recovery behavior.
• SQLite schema 21, migrations and persisted-data compatibility, including binary rollback after candidate writes.
• Both execution engines, REPL and retained-child semantics, provider defaults/encoding/retries, permissions, tracing/OTLP/prompts and accounting.
• One runtime and existing representations, without a translation layer or frontend migration. Only internal Go import paths change for native consumers.

Validation

• Final repair head 64e37008d48b55ee520d0acbc087738fccf6bcab passed all 22 checks, including full CI, security, Linux/macOS compatibility, desktop lifecycle/SSH, and distribution checks. The refactor(backend): separate responsibilities with a frozen client contract #320 merge tree was verified byte-for-byte identical to that tested tree before landing.
• All original PR heads are ancestors of this branch. The original consolidation at d29344dcf638a9b992fe6640b74e17d832efd5d3 differs from the tested original stack tip only in docs/backend-reorganization-plan.md; subsequent security repairs are isolated in fix(deps): repair security blockers before backend reorganization merge #329. Frozen SDK/protocol/migration paths remain identical to development; git diff --check passes.
• Final integrated task acceptance passed on the unchanged implementation, including 49 SDK acceptance cases with zero failures/skips, native CLI/TUI/ACP/runtime checks and packed-consumer checks. Web check and packaging passed.
• The original final tip passed full CI and Security, including Linux/macOS compatibility/runtime jobs, Go coverage/lint, SDK/browser, mobile, distribution and desktop checks. For the current dependency/toolchain repair, the final validation below supersedes those historical runs.
• Native evidence: 47 Go tests and 20 Electron fixture checks passed, with 49 source/dependency fingerprints matching the final implementation. The user also reports an initial Desktop review working.
• Limitations remain explicit: DriverParity skipped on macOS because of Linux-only browser paths; OS compositor capture lacked macOS permission (guest capture passed). Optional integration-tag TUI lint retains an inherited noctx finding; required default lint passes. Historical canceled workflow entries on test: guard backend reorganization compatibility #302/refactor(daemon): separate host MCP import ownership #304/refactor(daemon): prepare provider operation boundary #305 remain recorded and are not represented as passes.

Architecture · Plan and detailed preservation evidence

Supersedes the review chain #299 → #300 → #301 → #307 → #302 → #304 → #305 → #306 → #308 → #309 → #310 → #311 → #312 → #313 → #314 → #315 → #316 → #317 → #318 → #319. Those PRs are retained as historical focused reviews, with their branches and evidence preserved. This does not include or merge the abandoned backend-redesign or frontend-restoration stacks.

---

Note

High Risk
Large-scale Go package moves and daemon construction changes touch auth, clients, persistence wiring, and release CI gates; behavior is intended to be preserved but regressions would affect core runtime and merge blocking.

Overview
Splits backend responsibilities out of internal/daemon while keeping protocol 6.9, schema 21, and external behavior unchanged. Native Go clients move to internal/client; shared paths, dial/launch, and socket helpers move to internal/daemonconn; provider discovery/configuration/login moves to internal/provider; stored command text formatting moves to internal/commandpresentation. CLI/TUI/ACP/gateway code updates imports and types (protocol.*, session.CreateSession) instead of daemon.* client APIs.

Daemon startup is slimmed in cmd/whip/daemon.go: runtime session construction (model resolution, MCP, compaction, tool services) lives in new cmd/whip/daemon_runtime.go, session.Open takes a workspace coordinator, and daemon.New receives a shared process manager.

CI and local gates add a matrix compatibility job (scripts/backend-compat/run.mjs) wired into the required go aggregate, extend runtime/acceptance coverage for daemonconn/client/command presentation/TUI integration, run web performance-probe unit tests, bump golangci-lint to v2.14.0, and trigger CI/security on codex/backend-reorg-* branches. Toolchain/security: Go 1.27.2, docs TanStack bumps, SDK job uses npm run audit:production instead of raw npm audit.

Web performance inlines probe logic into performance-probes.mjs (desktop + WebSocket), with tests for snapshot vs notification commit timing, deduplication, and bounds; performance.mjs imports that module.

Docs refresh architecture.md (gateway vs daemon, package map, resource lifetimes) and add dependency-security.md; backend-redesign-plan.md is marked superseded.

^{Reviewed by Cursor Bugbot for commit 7501aa5. Bugbot is set up for automated code reviews on this repo. Configure here.}