feat(agent): checkpoint CUDA multicast state

[galletas1712]

Summary

• intercept and virtualize same-node POSIX CUDA multicast objects, devices, bindings, mappings, and access state
• tear down multicast topology before unicast/native CUDA checkpoint and rebuild it from fresh objects after restore
• preserve the application's original multicast creation properties while separately tracking the largest range accepted by the driver
• serialize that driver-accepted extent for topology validation, avoiding rejection when r615 rounds multicast physical capacity above the requested size
• avoid holding shared interposer state locks across collective multicast driver calls and revalidate phase after reacquiring them
• use bounded control-socket timeouts and detached control threads

The generic unicast access-range fix lives in #152. This is the multicast layer of stack #156; follow-on fixes #165, #166, #167, and #174 are stacked above it.

r615 root cause

The deployed workload loads the locally built r615.68 libcuda exactly (SHA-256 9edeb49896e9b8d8dfb45dd1c9f3930c137f2d8f358f66594bb146b35437d850). In r615.68, multicast creation retains the public requested size but memDeviceBlockAllocMulticastObjectRM rounds physical capacity to the 512 MiB FLA page size. FlashInfer consequently creates a smaller logical object and successfully binds/maps the 512 MiB driver-recommended extent.

The interposer previously serialized only the smaller creation size. Its coordinator then rejected the valid 512 MiB binding before native CUDA checkpoint with invalid multicast binding. The fix records successful map/bind ranges as the effective extent while retaining the original properties for multicast object identity and replay. No driver change, FLA disable, FABRIC disable, or device-to-device copy is involved.

Validation

• fake-driver regression: request a 2 KiB multicast object, accept an 8 KiB map, and require the serialized record to contain the 8 KiB driver-accepted extent
  • before: FAIL: multicast record uses driver-accepted extent
  • after: multicast behavior OK
• native CUDA checkpoint tests cover POSIX symmetric-memory CUDA graphs and two-GPU multicast symmetric-memory CUDA graphs
• go test ./api/... ./operator/... ./agent/...
• GLM 5.2 / DeepSeek-V4-Flash-NVFP4 SGLang TP8/EP8 on eight B200 GPUs with the fixed r615.68 userspace driver:
  • the previous interposer fails during prepare with topology validate failed: invalid multicast binding
  • the fixed stack completes checkpoint, restores active and standby engines, and returns coherent post-restore inference (17 * 23 = 391)
  • after terminating the active engine, the standby acquires the GMS lock in about 121 ms, registers in about 4.0 s, and again returns 391

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 60566c92-35e0-44aa-8976-dc81af0b0d78

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[galletas1712]

Superseded by #219 (multicast state); the parallel PREPARE_MULTICAST dispatch from c47baab is covered by a coordinator test in #216. The re-cut stack (#212 → #220) supersedes this PR. Its body is quoted verbatim in the replacement's Origin section, and each review thread here has a row in the replacement's "Review threads carried" table with what was done about it.