test(agent): exercise cuinterpose checkpoint and restore on GPUs

[galletas1712]

Summary

Layer 11 of the eleven-PR cuinterpose stack #293, based on #219. Closed PR #214 is not in the active stack.

This PR adds real-driver two-GPU checkpoint/restore tests under agent/cmd/cuinterpose/tests/gpu. The uninterposed controller launches an interposed parent and two worker ranks through cuda-checkpoint --launch-job. It drives interposer preparation, native CUDA lock/checkpoint/restore/unlock, interposer reconstruction, and workload assertions. It does not deploy the operator or namespace-entry wrapper.

Shared and private VMM regression

The tests distinguish actual sharing from export capability:

• A ticket-backed peer allocation remains live across checkpoint and restore.
• A large seeded allocation is explicitly exported, so carrier throughput continues to measure the interposer-owned path.
• An additional POSIX-capable VMM allocation is never exported or bound into multicast. It remains native-owned through the updated feat(agent): checkpoint and restore POSIX CUDA VMM state #218/feat(agent): checkpoint CUDA multicast state #219 lifecycle.
• After restore, the private allocation's existing VA must contain its original seeded bytes; its original handle must support cuMemGetAllocationPropertiesFromHandle; retain-by-address and normal cleanup must succeed.
• Carrier byte/count assertions include only actually shared creators rather than every tracked creator.

The suite also retains raw-import preflight refusal, creator/peer content checks, PyTorch symmetric-memory sharing, captured all-reduce replay, and post-restore raw import coverage. The multicast test requires real multicast selection, exercises bindings including BindAddr, and validates its collective and captured graph after reconstruction.

The suite skips absent prerequisites: Python CUDA/PyTorch bindings, build tools or prebuilt binaries, launch-job support, two distinct GPUs, and NVLink/multicast support where required. CUINTERPOSE_BUILD_DIR selects node-built binaries; otherwise CUDA_HOME supplies build headers.

Stack transport

The ownership change is introduced in #218, merged through #219, then merged here. The GPU regression is a separate commit in this PR. Existing branch history is preserved; no force-push is needed. No merged-test-branch NIXL, CustomStorage, driver patch, or phase-batching code is included.

Validation

Preparation checks passed:

git diff --check
cc -fsyntax-only -Wall -Wextra -Werror (interposer, CUDA 13.1 headers)
.venv/bin/python -m py_compile tests/gpu/worker.py tests/gpu/test_posix.py

Candidate 21008b50b93a9879a805665e331e777bb93abf49 passes all 74 native fake-driver tests, including five lifecycle and six multicast tests, with the configured ASan/UBSan instrumentation. Tests used CUDA 13.1 headers, the system GCC/G++ toolchain, cached GoogleTest, and a short control-socket directory. Logs: stack-private-vmm/pr220-final-tests.log.

Exact PR head 21008b50b93a9879a805665e331e777bb93abf49 passed all 3 physical-GPU tests, 0 skips, 0 failures, in 19.896 s on two B200 GPUs on nscale-dev, 2026-09-12 UTC. The source commit and built-binary checksums were recorded, and JUnit confirms:

• test_checkpoint_restores_multicast_group
• test_checkpoint_restores_shared_posix_memory
• test_prepare_is_refused_while_a_raw_import_is_alive

The node compiled the exact source using the Makefile's documented ALLOW_OLD_CUDA_HEADERS=1 override with CUDA 13.0 headers. Pytest ran through the uv-managed environment at agent/cmd/cuinterpose/tests/gpu/.venv/bin/python; the three warnings concern record_property with xunit2, not skipped coverage. Evidence is saved in the private test bundle restore-optimization-e1a06b2/evidence/pr220-gpu/.

This was an exact-stack test, separate from the merged CustomStorage performance experiment.

Node test commands (with evidence set to the Job's output directory):

make -C agent/cmd/cuinterpose BUILD_DIR=/tmp/pr220-build \
  CUDA_HOME=/usr/local/cuda ALLOW_OLD_CUDA_HEADERS=1 all
export CUINTERPOSE_BUILD_DIR=/tmp/pr220-build
/snapshot-tools/cuda-checkpoint --launch-job \
  uv run --project agent/cmd/cuinterpose/tests/gpu \
  pytest agent/cmd/cuinterpose/tests/gpu -vv -m gpu \
  -o junit_family=xunit2 --junitxml="$evidence/junit.xml" -rs

Historical evidence only

Historical stack head 0eae9c4f9b66093332f00a332afaf56048e1e64a passed the full local gate, 73 native tests, and all three B200 GPU tests with no skips in 19.79 seconds. Its unicast case restored 545,259,520 bytes at 105.47 GB/s aggregate H2D copy throughput; its multicast and live raw-import refusal cases passed. Those numbers describe the previous ownership behavior, not this candidate's performance.

Contribution notes

This updates the existing test PR rather than duplicating it. AI assistance was used for the implementation, regression, and description. The human submitter must review every change and run the relevant tests.

This ownership fix is being validated in parallel with the restore-performance experiment. That experiment's NIXL and driver changes are not included here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[galletas1712]

Superseded by the replacement 13-PR C-frontend/Rust-backend cuinterpose draft stack: https://github.com/ai-dynamo/snapshot/stack/339 (#326–#338), tracking approved proposal #295. The new stack preserves one reviewable stage per PR, with tests collected in #338 and the design in #337. PageBroker transfer changes remain a separate dependency. Closing this older C implementation; retaining its branch and discussion for reference.