feat(snapshot): checkpoint POSIX CUDA multicast

[galletas1712]

Summary

Add transparent checkpoint and restore for complete same-node CUDA multicast
groups backed by CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR, stacked on the
POSIX CUDA VMM interposer in #13129.

This PR covers the common PyTorch symmetric-memory and NCCL NVLS topology on one
node. It does not add FABRIC/IMEX/MNNVL, cross-node rendezvous, Redis, legacy
CUDA IPC wrappers, operator API, or Go-side CUDA topology knowledge.

Design

Resource model

• Treat multicast as a topology overlay over checkpoint-managed unicast
  allocations.
• Keep unicast and multicast identities distinct behind tagged logical CUDA
  handles.
• Keep real current-generation CUDA handles private to the shim.
• Record one deterministic multicast metadata creator, the complete device set,
  and each member process's local bindings, mappings, access, and logical-handle
  liveness.
• Allow each multicast object to be owned by a subset of the discovered
  processes. Processes outside that object are phase no-ops.
• Keep validation object-local and complete: creator, device, binding, and
  property validation covers every member and property of each multicast
  object without requiring unrelated processes to own it.
• Keep allocation bytes exclusively under native cuCheckpoint; multicast does
  not introduce another byte-saving path.

Ownership boundaries

File	Responsibility
interpose.c	LD_PRELOAD/API interception, CUDA resolver mediation, generic handle translation, and unicast state
posix.c	Sealed POSIX capability FDs and same-node creator exchange through Unix sockets plus SCM_RIGHTS
multicast.c	Multicast identities, handles, team membership, BindMem/BindAddr, mappings/access, detach, reconstruction, and validation
coordinator.c	Complete topology validation and ordered participant phases

Go transports the opaque native sidecar and orders the existing native
checkpoint lifecycle. It does not enumerate or serialize CUDA multicast
topology.

Interception surface

Track and translate:

• cuMulticastCreate
• cuMulticastAddDevice
• cuMulticastBindMem and CUDA 13.1 _v2
• cuMulticastBindAddr and CUDA 13.1 _v2
• cuMulticastUnbind
• cuMulticastGetGranularity
• generic VMM export/import, release/retain, allocation-property, map/unmap, and
  access consumers

Cover direct Driver API calls plus:

• explicit dlsym() lookups from libcuda.so and libcudart.so;
• cuGetProcAddress, cuGetProcAddress_v2, and _ptsz; and
• cudaGetDriverEntryPoint and cudaGetDriverEntryPointByVersion, including
  CUDA 13 _ptsz exports.

Checkpoint topology teardown

The coordinator first validates the complete object-local participant topology
and the existing unicast creator-anchor invariant. Every discovered process
enters the ordered phases, but a process that is not a member of any multicast
object is a multicast phase no-op.

Multicast is detached before unicast normalization:

1. create any unicast checkpoint carriers needed by the existing base flow;
2. unmap each multicast VA while preserving its reservation;
3. unbind each recorded local unicast member;
4. release creator/imported multicast handles and destroy the old object;
5. wait for every participant to report multicast detached;
6. remove managed unicast mappings without freeing VA reservations and release
   importer handles; and
7. let native cuCheckpoint save unicast allocation bytes.

The shim commits each multicast record's detached/checkpointed bookkeeping only
after the corresponding CUDA call succeeds. A failed CUDA teardown therefore
does not falsely advance native state.

Restore topology replay

After native CUDA restore and driver unlock, the application remains externally
quiesced while the shim reconstructs:

1. unicast creator handles and mappings;
2. current-generation unicast imports and importer mappings;
3. one fresh POSIX multicast object from the recorded metadata creator;
4. fresh imports of that object in its other member processes;
5. the complete device team;
6. each member process's own BindMem or BindAddr binding;
7. original multicast VAs and access;
8. logical-to-current handle translation; and
9. final object-local topology validation against the opaque sidecar.

Nonmember processes remain no-ops throughout multicast reconstruction. CUDA
multicast bind may block until the full object-local team joins. The shim
therefore releases its bookkeeping mutex only around the native bind call so
creator endpoints remain serviceable, then reacquires it before committing
local state.

The coordinator writes opaque snapshot-cuda-posix-v2 state. It contains stable
participant/resource identities and topology only—no raw FDs, authorization
tokens, creator endpoints, or allocation bytes.

Runtime contract

• VMM interposition remains explicit launch-time opt-in through
  snapshot-cuda-vmm-launch or snapshotctl --cuda-vmm-interpose.
• Single-GPU POSIX VMM checkpointing does not require a CUDA launch-job file.
• Multi-GPU workloads also use the existing driver-owned
  cuda-checkpoint --launch-job path. Manual snapshotctl users pass both
  --cuda-checkpoint-wrap and --cuda-vmm-interpose.
• Legacy CUDA IPC remains driver-owned and is not wrapped here.
• The complete process group must be externally quiesced before checkpoint
  preparation.
• A multicast object may belong to a subgroup of discovered processes;
  multicast detach, restore, and validation phases are no-ops in nonmembers.
• Creator, device, binding, and property validation remains complete within
  each multicast object.

Meaningful CUDA tests

The colocated self-contained pytest project contains two real two-GPU
cuCheckpoint scenarios:

• a non-multimem POSIX symmetric-memory regression; and
• a required multicast symmetric-memory path with no unicast fallback.

The multicast case:

• requires PyTorch multicast support and a nonzero multicast pointer;
• exercises interposed BindMem during symmetric-memory setup;
• replaces the local binding through direct interposed cuMulticastBindAddr;
• performs a real multimem all-reduce;
• captures that collective in a CUDA graph;
• verifies exact results before checkpoint;
• executes coordinator prepare, native lock/checkpoint/restore/unlock, and
  coordinator restore;
• validates fresh post-restore sharing and private allocation bytes; and
• replays the same pre-checkpoint CUDA graph and verifies exact results after
  restore.

No CRIU or PodSnapshot is required for this component gate; the tests call the
native cuCheckpoint API directly and run under
cuda-checkpoint --launch-job.

Earlier two-B200 qualification passed both component scenarios:

test_cucheckpoint_preserves_symmetric_memory_cuda_graph
# 1 passed in 8.02s

test_cucheckpoint_preserves_multicast_symmetric_memory_cuda_graph
# 1 passed in 9.75s

An earlier composed acceptance run also exercised DeepSeek V4 Flash TP4 with
multicast enabled, disjoint restore from GPUs 0-3 to GPUs 4-7, nonmember process
phase no-ops, three multicast objects, and coherent post-restore inference. That
campaign used independent composed-test fixes not included in this product PR.
GPU campaigns were not rerun for this restack.

Supported scope and limitations

Supported:

• complete same-node multicast objects, including objects owned by a subset of
  discovered processes;
• POSIX-FD-backed CUDA VMM members;
• distributed member add/bind behavior used by PyTorch symmetric memory and
  NCCL NVLS;
• BindMem and BindAddr common paths; and
• one CUDA context per multicast identity per process.

Explicitly excluded:

• FABRIC/IMEX/MNNVL and cross-node groups;
• partial-team or independent-participant restore;
• raw external POSIX imports retained across checkpoint;
• legacy CUDA IPC reconstruction in this shim;
• checkpoint during sharing or communicator construction;
• fork after the shim has tracked CUDA state; and
• rollback after destructive checkpoint preparation.

Potentially silent raw-FD alias, unshimmed broker, uncovered resolver,
raw-handle collision, and pass-through BindAddr limitations are documented in
the interposer README.

Validation

The multicast changes were replayed from only the two canonical commits:

a63baafc84 feat(snapshot): checkpoint POSIX CUDA multicast
9bfee56a85 fix(snapshot): commit multicast state after CUDA success

git range-diff shows the second patch is unchanged. The first differs only
where it inherits #13129's simplified identity model and fixed-structure
reserved-byte layout.

Validated on the final restacked tree:

make -C deploy/snapshot/cmd/cuda-vmm-interpose clean
make -C deploy/snapshot/cmd/cuda-vmm-interpose \
  CUDA_HOME="$PWD/../.cuda13/nvidia/cu13"

cd deploy/snapshot
go test ./...

Additional passing checks:

• native compilation with -std=gnu11 -O2 -Wall -Wextra -Werror;
• ELF inspection confirming no direct libcuda or libcudart dependency;
• all applicable pre-commit hooks on the multicast range;
• shell syntax;
• Ruff format and lint;
• Python compilation and pyproject.toml parsing;
• git diff --check; and
• range-path audit confirming only the snapshot Dockerfile and
  deploy/snapshot/cmd/cuda-vmm-interpose/ changed above feat(snapshot): checkpoint POSIX CUDA VMM sharing #13129.

Live composed-stack acceptance

Build on Demand run 31908730437 produced the exact composed SHA
ad5d5ebc6f4f30da9bc65ac7355a836702622328. That composition passed:

• the direct DeepSeek V4 Flash TP4 multicast/NCCL NVLS qualification, including
  three multicast objects, nonmember-process phase no-ops, restore from pinned
  GPUs 0-3 to disjoint pinned GPUs 4-7 on s2877, and three exact
  DSV4_RESTORE_OK responses; and
• broader GLM 5.2 vLLM TP8/DP1/EP8 composed-stack coverage on the same ordered
  eight-GPU set on s2877 (not a migration test), with three deterministic,
  coherent HTTP 200 responses ending in GLM52_RESTORE_OK after restore.

DeepSeek V4 is the direct live qualification of raw CUDA multicast/NCCL NVLS
replay. GLM does not make that claim: its accepted snapshot lifecycle policy
runtime-injected NCCL_CUMEM_ENABLE=0, NCCL_NVLS_ENABLE=0, and
NCCL_IB_DISABLE=1. GLM instead validates the wider composed checkpoint,
restore, GMS, wake, and inference path around this stack.

No container image was built locally. GPU tests were not rerun during this
restack.

Stack

• Base: feat(snapshot): checkpoint POSIX CUDA VMM sharing #13129 (rewrite/snapshot-cuda-vmm-posix-native)
• This PR: one GPG-signed, DCO-signed commit above the rewritten feat(snapshot): checkpoint POSIX CUDA VMM sharing #13129 tip

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a0073e-637d-7025-a936-8983f9f4767d.docs.buildwithfern.com/dynamo/dev

[datadog-official]

⚠️ Warnings

🚦 4 Pipeline jobs failed

Docs link check | lychee     

See error

Failed to access API: multiple 404 Not Found errors when trying to reach https://xgrammar.mlc.ai/docs/structural_tag/structural_tag_api.html.

PR | DGDR Deploy Test / CPU / lifecycle     

See error

Profiling job failed: Job has reached the specified backoff limit while waiting for the 'Ready' phase of the DGDR.

PR | deploy-status-check     

View all 4 failed jobs.

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 11.32%
• Overall Coverage: 47.98% (-1.28%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: cc46e5f | Docs | Datadog PR Page | Give us feedback!}

[devin-ai-integration]

Devin Review found 6 potential issues.

[galletas1712]

Closing: deploy/snapshot has moved to ai-dynamo/snapshot. Replacement is ai-dynamo/snapshot#79, stacked on ai-dynamo/snapshot#78.