feat(agent): coordinate CUDA interposer lifecycle by galletas1712 · Pull Request #155 · ai-dynamo/snapshot

galletas1712 · 2026-09-01T07:58:53Z

Summary

implement the native cuinterposer-coordinator protocol and topology validation
detect live shim participants through Unix sockets and persisted restore state through cuinterposer.state
invoke prepare before native CUDA checkpointing
pre-open and invoke the coordinator across CRIU mount-namespace replacement during restore
keep orchestration dormant when the forwarding-only shim from feat(agent): forward POSIX CUDA VMM APIs #78 has no control sockets or state file

This is layer 3 of stack #156. It adds coordinator and agent orchestration, but not POSIX allocation tracking or checkpoint behavior.

Validation

all Go tests
production CUDA-builder compile
static coordinator has no dynamic dependencies
intermediate branch builds independently

coderabbitai · 2026-09-01T07:59:00Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

_{Comment @coderabbitai help to get the list of available commands.}

Implement the native coordinator protocol, socket and state-file detection, and agent prepare/restore orchestration. The forwarding-only shim does not create endpoints, so this layer remains dormant until checkpoint behavior is added above it. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>

galletas1712 · 2026-09-04T19:25:10Z

Superseded by #216 (coordinator); the agent orchestration moved to #214. The re-cut stack (#212 → #220) supersedes this PR. Its body is quoted verbatim in the replacement's Origin section, and each review thread here has a row in the replacement's "Review threads carried" table with what was done about it.

…t and restore The checkpoint manifest gains cuinterpose.requested (the source Pod's opt-in) and cuinterpose.prepared (the coordinator's prepare completed and wrote cuinterpose.state, which drives the coordinator on restore; a prepared checkpoint without its state file is refused). The shim's mount is not the shim's concern: it travels with the CUDA tools delivery (cudaTools.delivered) whether or not the Pod opted in. Detection is fail-closed: a Pod that requested the shim must show a control socket for every CUDA process and a Pod that did not must show none; anything else fails the checkpoint with the processes named. Only sockets count; procfs environ is not evidence. Stale shim sockets are removed before CRIU recreates the processes, the coordinator binary is opened before the mount namespace changes, its progress lines are logged as structured fields, and prepare and restore get their own timing phases. The Go constants are pinned to protocol.h by a test and the coordinator's argument contract is covered against a fake binary. Until the shim opens control sockets (the lifecycle change), an annotated Pod with CUDA processes is refused at checkpoint by the fail-closed rule. Compared with #110 and #155: restore is driven by the manifest rather than by the presence of a state file, so a lost state file is an error instead of a silent native restore; detection is a truth table instead of "sockets present". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>

Add cuinterpose-coordinator, the static program the snapshot agent runs once before the native CUDA checkpoint and once after the native CUDA restore. The coordinator talks to every CUDA process over the shim's control socket, checks that their descriptions of shared memory agree, refuses to prepare while any process still holds an untracked import, drives teardown and rebuild in the order the driver needs, and writes the cuinterpose.state sidecar with an atomic rename and a directory fsync. Every restore phase is dispatched to all participants at once with a barrier after it; the barrier before multicast binding is a driver requirement, since cuMulticastBindMem spins until every device of the team is attached. The coordinator prints one progress line per phase with elapsed time and, for the carrier phases, bytes and throughput, which the agent logs. Compared with the earlier coordinator layer (#155): SIGPIPE from a participant that hung up no longer kills the coordinator; a missing state file on restore is an error instead of a silent success; --control-dir replaces a hard-coded path and an environment fallback; multicast BIND_MEM records are checked against the member allocation's bounds; diagnostics no longer print stale indices; timeouts use the SNAPSHOT_ prefix; and the coordinator is covered by a GoogleTest suite against fake participants, run during the image build. This is an intermediate layer: the shim does not answer control requests until the tracking and lifecycle changes land, so an annotated Pod is still refused by the agent's fail-closed detection here. Restoring on a node whose agent image differs from the checkpoint node's is not verified and fails inside CRIU's file validation; this is deferred. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>