test(agent): cuinterpose GPU tests

[galletas1712]

Summary

agent/cmd/cuinterpose/tests/gpu: a pytest suite that runs the shim and the coordinator against the real driver on two GPUs through cuda-checkpoint --launch-job. One interposed parent forks two PyTorch workers; each creates POSIX-shareable allocations (one before any CUDA context exists, one large one filled with seeded random bytes), imports a descriptor from the uninterposed test process, shares a symmetric-memory buffer with the other rank, and captures a collective into a CUDA graph. The test runs the coordinator, the native cuCheckpointProcess* sequence through cuda.bindings, and the coordinator again, then checks the state file, the coordinator's carrier counts, the copy throughput against a pinned baseline measured on the same machine, every byte, the graph replay, and a fresh raw import. A second test holds a raw import across prepare and checks the refusal. test_multicast.py runs PyTorch's multimem all-reduce with a cuMulticastBindAddr rebind and skips without NVLink.

CUINTERPOSE_BUILD_DIR points at prebuilt binaries (for example copied out of the agent image); otherwise the tests build the shim with CUDA_HOME. Dependencies are pinned in pyproject.toml against the cu130 torch index.

Origin

Re-cut of test_cucheckpoint.py from #152/#165/#166 and its multicast case from #133.

Review threads carried

Old thread	Raised	Disposition
#78 Copilot test_cucheckpoint.py:373, CodeRabbit test_cucheckpoint.py:288	GPU test failed instead of skipping without prerequisites; prerequisite check missed grep.	Fixed: conftest.py skips without torch, cuda-bindings, two GPUs, or a launch job; prebuilt binaries avoid the native build entirely.

Validation

• Fake-driver suites (GoogleTest, AddressSanitizer + UndefinedBehaviorSanitizer) run in the agent image build against CUDA 13.1 headers at every layer of the stack; at the top: proto 22, table 3, forward 12, coordinator 12, state 14, lifecycle 4, multicast 6, no sanitizer reports.
• go test ./... in api, operator, agent; make lint, make helm-lint, make verify-license-headers.
• Agent image built from the top of the stack and deployed on nscale-dev (B200, kernel driver 595.58.03) with this chart.
• GPU tests (two B200s, cuda-checkpoint --launch-job): POSIX round trip with seeded 1 GiB carriers per rank, multicast round trip with PyTorch's multimem all-reduce and a cuMulticastBindAddr rebind, refusal while a raw import is alive: 3 passed. Host-carrier restore phase 87 to 108 GB/s aggregate over two GPUs (pinned-copy baseline 55 GB/s per GPU).
• End to end: vLLM 0.27.1 AsyncLLM, Qwen3-0.6B, tensor parallel 2, FlashInfer TRT-LLM attention and fused allreduce (trtllm backend), PodSnapshot of a Deployment shaped with podcontract.ShapeCuinterposeCapture, then restore: checkpoint 52 s (CRIU dump 49 s, cuinterpose prepare 0.43 s; 4 CUDA processes, 1052 records, 382 host carriers, 2.08 GB); restore 5.7 s (cuinterpose 0.35 s: carriers 0.05 s, unicast 0.09 s, multicast 0.18 s); the restored replica answers coherently ("The capital of Italy is Rome").

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[galletas1712]

Superseded by the nine-PR re-cut (packaging → delivery → agent → forwarding → coordinator → tracking → lifecycle → multicast → GPU tests): Go and packaging split by component, tests trimmed, docs held back. The replacement PRs are listed on the packaging PR.