feat(agent): cuinterpose coordinator

[galletas1712]

Summary

coordinator.c, replacing PR 1's placeholder: --prepare|--restore --proc-root --checkpoint-dir --control-dir --process OBS NS...; IDENTIFY, INSPECT, topology validation (one creator per allocation, mappings in bounds, multicast groups complete, bind-mem member bounds), refusal while any participant holds live raw imports, phases with barriers (the multicast barrier is a driver requirement: cuMulticastBindMem spins until every device of the team is attached), cuinterpose.state (cuinterpose-state-v2) written with atomic rename and directory fsync and mandatory on restore, one machine-readable progress line per phase. Doc: docs/reference/cuinterpose.md §3.2, §3.3, §5, §6.

Intermediate layer: the shim does not answer control requests until PRs 4 and 5, so an annotated Pod is still refused here. coordinator_test runs the binary against fake participants (phase ordering, barrier, topology validation, raw import refusal, state file golden, every restore precondition).

Origin

Re-cut of #155.

#155 feat(agent): coordinate CUDA interposer lifecycle (verbatim, bot blocks removed)

Summary

• implement the native cuinterposer-coordinator protocol and topology validation
• detect live shim participants through Unix sockets and persisted restore state through cuinterposer.state
• invoke prepare before native CUDA checkpointing
• pre-open and invoke the coordinator across CRIU mount-namespace replacement during restore
• keep orchestration dormant when the forwarding-only shim from feat(agent): forward POSIX CUDA VMM APIs #78 has no control sockets or state file

This is layer 3 of stack #156. It adds coordinator and agent orchestration, but not POSIX allocation tracking or checkpoint behavior.

Validation

• all Go tests
• production CUDA-builder compile
• static coordinator has no dynamic dependencies
• intermediate branch builds independently

Review threads carried

No review threads on #155. The coordinator threads from #78 and #79 are listed in PR 2's table; all of them are fixed here.

Validation

• Fake-driver suites (GoogleTest, AddressSanitizer + UndefinedBehaviorSanitizer) run in the agent image build against CUDA 13.1 headers at every layer of the stack; at the top: proto 22, table 3, forward 12, coordinator 12, state 14, lifecycle 4, multicast 6, no sanitizer reports.
• go test ./... in api, operator, agent; make lint, make helm-lint, make verify-license-headers.
• Agent image built from the top of the stack and deployed on nscale-dev (B200, kernel driver 595.58.03) with this chart.
• GPU tests (two B200s, cuda-checkpoint --launch-job): POSIX round trip with seeded 1 GiB carriers per rank, multicast round trip with PyTorch's multimem all-reduce and a cuMulticastBindAddr rebind, refusal while a raw import is alive: 3 passed. Host-carrier restore phase 87 to 108 GB/s aggregate over two GPUs (pinned-copy baseline 55 GB/s per GPU).
• End to end: vLLM 0.27.1 AsyncLLM, Qwen3-0.6B, tensor parallel 2, FlashInfer TRT-LLM attention and fused allreduce (trtllm backend), PodSnapshot of a Deployment shaped with podcontract.ShapeCuinterposeCapture, then restore: checkpoint 52 s (CRIU dump 49 s, cuinterpose prepare 0.43 s; 4 CUDA processes, 1052 records, 382 host carriers, 2.08 GB); restore 5.7 s (cuinterpose 0.35 s: carriers 0.05 s, unicast 0.09 s, multicast 0.18 s); the restored replica answers coherently ("The capital of Italy is Rome").

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[galletas1712]

Superseded by the nine-PR re-cut (packaging → delivery → agent → forwarding → coordinator → tracking → lifecycle → multicast → GPU tests): Go and packaging split by component, tests trimmed, docs held back. The replacement PRs are listed on the packaging PR.