test(cuinterpose): unit, integration, and GPU lifecycle coverage

[galletas1712]

Summary

Refs #295 (approved).

Collect frontend, Rust, fake-driver, GPU, agent/operator, and pod-contract coverage for the assembled stack. Add memory-IPC and jobfile-free lifecycle regressions.

Review boundary

Remove tests for the deleted shim PageBroker backend and launch-job path. Keep this as the final test PR. The fake CUDA driver checks call/ownership behavior; physical-GPU tests check real contents and mappings.

Extend the existing identity test to check leading zeroes, byte order, maximum values, serialization round trips, and malformed input. Cleanup regressions remain in this final test PR.

Stack and compatibility

The stack starts directly on main; it does not depend on the PageBroker GPU-transfer branch or #323. The shim only saves shared creator bytes through host carriers. Never-shared allocations remain native CUDA state. The stack removes launch-job/jobfile support, and older jobfile-dependent or draft shim artifacts are rejected rather than migrated.

PageBroker and native CustomStorage changes are separate. No PR in this stack adds that implementation.

Order	PR	Scope
1	#326	C preload frontend and private ABI
2	#327	typed identities and peer transport
3	#328	process identity and peer export service
4	#329	VMM ownership and allocation lifecycle
5	#342	implement CUDA memory IPC over tracked VMM
6	#330	host-carrier storage for shared allocation bytes
7	#331	multicast tracking and reconstruction
8	#332	Rust lifecycle coordinator
9	#333	package frontend, Rust backend, and coordinator
10	#334	deliver CUDA tools and enable opt-in preload
11	#335	coordinate cuinterpose capture and restore without jobfiles
12	#336	deliver CUDA tools without rewriting workload commands
13	#337	explain host carriers, memory IPC, and restore ordering
14	#338	unit, integration, and GPU lifecycle coverage

Validation

Validation of the assembled implementation:

• Full API, agent, and operator Go tests; key agent race tests.
• Rust workspace tests, strict Clippy, GNU/musl builds, ABI/ELF checks, packaged frontend/fake-driver tests, and memory-IPC regressions.
• Helm tests/lint, Python manifest/report tests, repository lint, and make check in a clean disposable worktree.
• A small real-agent CRIU cross-node test passed on two GPU nodes, covering memory IPC, private/shared VMM, multicast/graph replay, bytes, and original addresses. It preceded only the final manifest-format guard; that guard has local regression coverage.

GLM testing without CustomStorage was cancelled at the user's request and is not a pass. GLM qualification uses a separate composition with CustomStorage; previous experimental GLM results are not qualification of this rebuilt stack. The installed test driver is not claimed to be a stock-driver qualification.

Tests above were run on the assembled implementation, not claimed independently for every source-only intermediate PR. The standard-library cleanup was validated with Rust workspace tests and strict Clippy, GNU/musl builds and packaged fake-driver tests, plus targeted pod-contract and CUDA-agent Go tests. Physical-GPU tests were not rerun for this cleanup; the earlier GPU results above remain attributed to the pre-cleanup implementation.

Cross-node framework qualification with separate CustomStorage composition

The separate schwinns/cuinterpose-customstorage-glm-matrix branch combines this stack with native CustomStorage/PageBroker changes, which are not part of this PR. Fresh GLM-5.2-NVFP4 TP8/EP8 capture/restore ran l9nsv to s2877, eight B200s per node. Source and restored responses were read for coherence, including fresh Paris/Tokyo/Berlin answers; none of these passes cold-reloaded the model.

Framework	Qualified configuration	Capture	Restore
vLLM	Default FlashInfer MoE; sleep level 1, wake weights before capture	458.247 s	63.498 s
SGLang	Official torch-memory-saver==0.0.10, no package source patches	344.194 s	68.030 s
TensorRT-LLM rc24	Explicit NCCL all-reduce; MPI ob1/tcp,self	298.533 s	112.331 s
TensorRT-LLM rc24	Explicit NCCL all-reduce; MPI UCX/TCP with UCX_TCP_KEEPIDLE=inf	305.810 s	106.641 s

The SGLang image was built on-cluster from the same pinned base with the official 0.0.10 wheel. Fifteen installed package files match wheel RECORD hashes, and all eight scheduler workers map both memory-saver and cuinterpose. This supersedes the earlier qualification requiring a local memory-saver patch.

UCX failure diagnosis: the failing checkpoint already contained closed TCP sockets. CRIU initially saw both endpoints established but entered TCP repair for them about 19 seconds apart; the reciprocal endpoint had a 10-second keepalive idle and 2-second interval. The passing full-model control disabled UCX TCP keepalive before launch and had no closed sockets in its original checkpoint. This strongly supports capture-time keepalive expiry, not restore IP/port rewriting. Small UCX cross-node Bcast tests and an additional five-minute post-capture delay also passed. The exact kernel timer sequence was not packet-traced.

This PR keeps the tested ob1/TCP guide defaults and documents the UCX alternative and its loss of normal-runtime keepalive peer-failure detection. All 37 framework-manifest tests pass.

Corrected TensorRT attribution: forced NCCL_SYMMETRIC corrupts source inference even with no cuinterpose mapped; a no-shim AUTO run also reproduces corruption with the same tactic-key set as the earlier failing shim run. Default symmetric collectives remain unqualified, but these controls do not establish an interposer defect. The exact erroneous runtime operation remains under investigation; no speculative shim change was made.

These results use the existing patched driver, not stock-driver qualification. The vLLM run preceded the separate PageBroker cancellation fix; the latest SGLang and UCX TensorRT runs used the fixed broker. Timings are agent wall times excluding model startup/image pulls and do not establish 30-second restore parity. The final documentation port/rebase did not change runtime code or the composition's tree.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}