feat(agent): checkpoint CUDA multicast state

[galletas1712]

Summary

Multicast objects (cuMulticast*) tracked like allocations and rebuilt in four phases. Doc: docs/reference/cuinterpose.md §3.3 (multicast phases), §6 (barrier and export invariants), and the second and third pitfalls in §8.

• multicast.c: logical handles over one driver handle per object per process; tickets served from the export cache; records of attached devices, bindings (by handle or address, v1 or device-explicit ABI), and mappings with merged access; the effective extent (r615 gives more capacity than requested and NCCL uses it; the coordinator takes the largest extent any participant reports); create, add-device, bind, and map run without the state lock and re-look up the object afterwards; PREPARE_MULTICAST closes the cached descriptor before releasing the object; creators recreate and re-export, importers fetch, everyone attaches devices, then binds and maps, each behind a coordinator barrier.
• Non-POSIX objects pass through; binding memory the shim does not track to a tracked object is refused.
• Tests: multicast_preload_test (two-rank group through the real coordinator over a fake driver that now models multicast objects, effective extent, unbind then rebind by address, pass-through, refusal of untracked members, descriptor-before-object teardown).

Origin

Re-cut of #133 (both commits: c47baab's rationale for parallel PREPARE_MULTICAST and -pthread is now in the coordinator's comments) and the multicast tracking half of #79.

#133 feat(agent): checkpoint CUDA multicast state (verbatim, bot blocks removed)

Summary

• intercept and virtualize same-node POSIX CUDA multicast objects, devices, bindings, mappings, and access state
• tear down multicast topology before unicast/native CUDA checkpoint and rebuild it from fresh objects after restore
• preserve the application's original multicast creation properties while separately tracking the largest range accepted by the driver
• serialize that driver-accepted extent for topology validation, avoiding rejection when r615 rounds multicast physical capacity above the requested size
• avoid holding shared interposer state locks across collective multicast driver calls and revalidate phase after reacquiring them
• use bounded control-socket timeouts and detached control threads

The generic unicast access-range fix lives in #152. This is the multicast layer of stack #156; follow-on fixes #165, #166, #167, and #174 are stacked above it.

r615 root cause

The deployed workload loads the locally built r615.68 libcuda exactly (SHA-256 9edeb49896e9b8d8dfb45dd1c9f3930c137f2d8f358f66594bb146b35437d850). In r615.68, multicast creation retains the public requested size but memDeviceBlockAllocMulticastObjectRM rounds physical capacity to the 512 MiB FLA page size. FlashInfer consequently creates a smaller logical object and successfully binds/maps the 512 MiB driver-recommended extent.

The interposer previously serialized only the smaller creation size. Its coordinator then rejected the valid 512 MiB binding before native CUDA checkpoint with invalid multicast binding. The fix records successful map/bind ranges as the effective extent while retaining the original properties for multicast object identity and replay. No driver change, FLA disable, FABRIC disable, or device-to-device copy is involved.

Validation

• fake-driver regression: request a 2 KiB multicast object, accept an 8 KiB map, and require the serialized record to contain the 8 KiB driver-accepted extent
  • before: FAIL: multicast record uses driver-accepted extent
  • after: multicast behavior OK
• native CUDA checkpoint tests cover POSIX symmetric-memory CUDA graphs and two-GPU multicast symmetric-memory CUDA graphs
• go test ./api/... ./operator/... ./agent/...
• GLM 5.2 / DeepSeek-V4-Flash-NVFP4 SGLang TP8/EP8 on eight B200 GPUs with the fixed r615.68 userspace driver:
  • the previous interposer fails during prepare with topology validate failed: invalid multicast binding
  • the fixed stack completes checkpoint, restores active and standby engines, and returns coherent post-restore inference (17 * 23 = 391)
  • after terminating the active engine, the standby acquires the GMS lock in about 121 ms, registers in about 4.0 s, and again returns 391

Review threads carried

No review threads on #133. The #79 threads are in PR 2's table; the multicast-specific dispositions: no export_raw (cache), state v2 accepted and documented, bounds and completeness checked by the coordinator with extents taken as the largest reported.

Validation

• Fake-driver suites (GoogleTest, AddressSanitizer + UndefinedBehaviorSanitizer) run in the agent image build against CUDA 13.1 headers at every layer of the stack; at the top: proto 22, table 3, forward 12, coordinator 12, state 14, lifecycle 4, multicast 6, no sanitizer reports.
• go test ./... in api, operator, agent; make lint, make helm-lint, make verify-license-headers.
• Agent image built from the top of the stack and deployed on nscale-dev (B200, kernel driver 595.58.03) with this chart.
• GPU tests (two B200s, cuda-checkpoint --launch-job): POSIX round trip with seeded 1 GiB carriers per rank, multicast round trip with PyTorch's multimem all-reduce and a cuMulticastBindAddr rebind, refusal while a raw import is alive: 3 passed. Host-carrier restore phase 87 to 108 GB/s aggregate over two GPUs (pinned-copy baseline 55 GB/s per GPU).
• End to end: vLLM 0.27.1 AsyncLLM, Qwen3-0.6B, tensor parallel 2, FlashInfer TRT-LLM attention and fused allreduce (trtllm backend), PodSnapshot of a Deployment shaped with podcontract.ShapeCuinterposeCapture, then restore: checkpoint 52 s (CRIU dump 49 s, cuinterpose prepare 0.43 s; 4 CUDA processes, 1052 records, 382 host carriers, 2.08 GB); restore 5.7 s (cuinterpose 0.35 s: carriers 0.05 s, unicast 0.09 s, multicast 0.18 s); the restored replica answers coherently ("The capital of Italy is Rome").

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[galletas1712]

Superseded by the nine-PR re-cut (packaging → delivery → agent → forwarding → coordinator → tracking → lifecycle → multicast → GPU tests): Go and packaging split by component, tests trimmed, docs held back. The replacement PRs are listed on the packaging PR.