feat: deliver CUDA tools and launch multi-GPU sources

[galletas1712]

Summary

Layer 2 of the eleven-PR cuinterpose stack #293. Closed PR #214 is intentionally not in the active stack.

This PR applies the CUDA launch-job requirement where Snapshot shapes source Pods and makes the required binaries available at stable paths. It does not opt a workload into cuinterpose.

ShapeCUDATools creates one snapshot-cuda emptyDir, runs an init container from the configured Snapshot agent image, copies cuda-checkpoint and libcuinterpose.so, and mounts both tools read-only at /tmp/snapshot-cuda in every target container. Tool delivery is unconditional for Snapshot targets and is independent of cuinterpose opt-in.

Only a target that may use more than one GPU has its command rewritten under:

/tmp/snapshot-cuda/cuda-checkpoint --launch-job ...

The CUDA job file is persisted in Snapshot's control volume. The operator counts nvidia.com/gpu resources and referenced DRA claims; indeterminate claim sizes are conservatively treated as multi-GPU because unnecessary wrapping is harmless while missing it causes a late capture failure. Existing source Jobs are verified for universal tool delivery and, where required, the launch wrapper before adoption.

Delivery and interposition remain separate. The mounted shim is inert until #222 puts it in LD_PRELOAD; opting into cuinterpose does not change the GPU-count-based launch rule.

The snapshot-control volume, /snapshot-control mount, and canonical and legacy control-directory variables are reserved. If a workload predefines them, source shaping accepts only one emptyDir, one exact writable subPath=<container-name> mount, and literal /snapshot-control variable values. PVC, hostPath, alternate-volume, broad-mount, read-only, duplicate, and valueFrom collisions are rejected atomically. This matches restore-side fail-closed validation and prevents the coordinator socket directory from becoming cross-Pod storage.

At capture, the manifest records cudaTools.delivered. Restore recreates /tmp/snapshot-cuda before CRIU by using the fixed ns-bind-mount role, allowing CRIU to reopen the launch wrapper and shim at their original file-backed paths.

Stack boundary

Based on #212. #222 adds explicit Pod opt-in and preloads the shim. This PR does not create control sockets, run the coordinator, or track CUDA state.

Validation

On the final stack, make test passes in api, agent, and operator. The source-shaping tests cover the exact reserved-volume contract and atomic rejection of unsafe collisions. The pinned CUDA 13.1 builder compiles the production binaries and passes all cuinterpose fake-driver suites. The final published stack head 0eae9c4f9b66093332f00a332afaf56048e1e64a passed the full make check gate, the CUDA 13.1 production build, and all 73 sanitizer-backed native cuinterpose tests. Its physical-GPU suite passed all 3 tests with no skips on two DRA-assigned NVIDIA B200 GPUs. The unicast test kept an explicit allocation-ID ticket-backed peer mapping per worker live across capture and restore; the multicast test required a nonzero multicast VA and shim-logical handle, exercised BindAddr, and passed its collective and captured-graph replay. Detailed hardware evidence and measurements are in #220.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[galletas1712]

Superseded by the replacement 13-PR C-frontend/Rust-backend cuinterpose draft stack: https://github.com/ai-dynamo/snapshot/stack/339 (#326–#338), tracking approved proposal #295. The new stack preserves one reviewable stage per PR, with tests collected in #338 and the design in #337. PageBroker transfer changes remain a separate dependency. Closing this older C implementation; retaining its branch and discussion for reference.