feat(agent): package the cuinterpose shim and coordinator

[galletas1712]

Summary

This is the first layer of the eleven-PR cuinterpose stack #293: #212 → #213 → #222 → #223 → #215 → #216 → #217 → #292 → #218 → #219 → #220. Closed PR #214 is intentionally not part of the active stack.

Cuinterpose lets Snapshot checkpoint and restore same-node CUDA memory shared between processes, including CUDA multicast objects. This PR establishes only the build, packaging, and shared wire contract. The shim and coordinator are placeholders at this layer.

agent/cmd/cuinterpose/Makefile builds libcuinterpose.so without linking CUDA, builds a static cuinterpose-coordinator, enforces hidden visibility and the glibc 2.34 ceiling, and discovers the fake-driver test suites added by later layers. The CUDA helper image installs CUDA 13.1 headers and packages both binaries under /snapshot-binaries/snapshot-cuda for restore.

protocol.h defines the fixed header and state-record layouts, participant and allocation identity, diagnostics, and the final operation vocabulary:

HANDSHAKE → INSPECT
capture: PREPARE_MULTICAST → SAVE_ALLOCATIONS → PREPARE_UNICAST
restore: LOAD_ALLOCATIONS → RESTORE_UNICAST → RESTORE_MULTICAST_*

Allocation content is represented generically by ALLOCATION_CONTENT and CONTENT_ONLY; host carriers are an implementation behind the save/load operations in #292 and #218. The protocol also reports live raw imports and unsupported exportable CUDA resource creations so #216 can refuse checkpoint before destructive work.

The bounded Unix-socket helpers support fixed messages, optional records, and one SCM_RIGHTS descriptor. They suppress SIGPIPE, reject malformed ancillary data, and close received descriptors on every error path. Generic arrays, cleanup, identity, I/O, parsing/randomness, tables/ranges, and timing are centralized under agent/cmd/cuinterpose/util/; feature modules keep domain-specific logic.

Stack boundary

Based on main. The next PR, #213, delivers cuda-checkpoint and the shim into workload containers. This layer does not inject the shim, discover participants, intercept CUDA, or execute lifecycle operations.

Validation

On the final stack, make test passes in all three Go modules. The pinned CUDA 13.1 builder compiles the production shared shim and static coordinator with -Werror, then passes 9 protocol, 9 table/cache, 6 utility, 14 coordinator, 12 forwarding, 4 lifecycle, 6 multicast, and 13 tracking tests with ASan/UBSan where configured.

The final published stack head 0eae9c4f9b66093332f00a332afaf56048e1e64a passed the full make check gate, the CUDA 13.1 production build, and all 73 sanitizer-backed native cuinterpose tests. Its physical-GPU suite passed all 3 tests with no skips on two DRA-assigned NVIDIA B200 GPUs. The unicast test kept an explicit allocation-ID ticket-backed peer mapping per worker live across capture and restore; the multicast test required a nonzero multicast VA and shim-logical handle, exercised BindAddr, and passed its collective and captured-graph replay. Detailed hardware evidence and measurements are in #220.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[galletas1712]

Stack order (bottom to top):

• feat(agent): package the cuinterpose shim and coordinator #212 feat(agent): package the cuinterpose shim and coordinator
• feat: deliver CUDA tools and launch multi-GPU sources #213 feat: deliver the cuinterpose shim to annotated Pods
• feat(agent): orchestrate the cuinterpose coordinator around checkpoint and restore #214 feat(agent): orchestrate the cuinterpose coordinator around checkpoint and restore
• feat(agent): cuinterpose shim symbol resolution and forwarding #215 feat(agent): cuinterpose shim symbol resolution and forwarding
• feat(agent): cuinterpose coordinator #216 feat(agent): cuinterpose coordinator
• feat(agent): track POSIX CUDA VMM state #217 feat(agent): track POSIX CUDA VMM state
• feat(agent): checkpoint and restore POSIX CUDA VMM state #218 feat(agent): checkpoint and restore POSIX CUDA VMM state
• feat(agent): checkpoint CUDA multicast state #219 feat(agent): checkpoint CUDA multicast state
• test(agent): exercise cuinterpose checkpoint and restore on GPUs #220 test(agent): cuinterpose GPU tests

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[dfeigin-nv]

I think, when looking at the over 1000+ LOCs in https://github.com/ai-dynamo/snapshot/blob/b657304f6001902a595ce2775c792bd470101e38/agent/cmd/cuinterpose/coordinator.c in #216 , that using C has its downsides. Let's consider using C++ since we have some complex logic here?

[galletas1712]

Superseded by the replacement 13-PR C-frontend/Rust-backend cuinterpose draft stack: https://github.com/ai-dynamo/snapshot/stack/339 (#326–#338), tracking approved proposal #295. The new stack preserves one reviewable stage per PR, with tests collected in #338 and the design in #337. PageBroker transfer changes remain a separate dependency. Closing this older C implementation; retaining its branch and discussion for reference.