docs: cuinterpose design and limitations - #208
Closed
galletas1712 wants to merge 2 commits into
Closed
galletas1712 wants to merge 2 commits into
galletas1712 wants to merge 2 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
pytest suite under agent/cmd/cuinterpose/tests/gpu that runs the shim and the coordinator against the real driver on two GPUs, through `cuda-checkpoint --launch-job` so the workers are checkpointable. The test process starts one interposed parent that forks two PyTorch workers; each creates POSIX-shareable allocations of its own (one before any CUDA context exists, one large one filled with seeded random bytes), imports a descriptor from the uninterposed test process, shares a symmetric-memory buffer with the other rank, and captures a collective into a CUDA graph. The test runs the coordinator, the native cuCheckpointProcess* sequence through cuda.bindings, and the coordinator again, then checks that only the state file was written, that the coordinator's summary counts at least the workers' own allocations as carried, that the host-to-device copies reached a fraction (default 80%) of a pinned copy measured on the same machine, that every byte survived and the graph replays with the right result, and that a fresh raw import works. A second test holds a raw import across prepare and checks the refusal names it and the workload keeps working. The multicast variant runs PyTorch's multimem all-reduce with one rank rebinding its slice through cuMulticastBindAddr, and skips without NVLink. Prebuilt binaries are used through CUINTERPOSE_BUILD_DIR (for example copied out of the agent image); otherwise the tests build the shim with CUDA_HOME. Dependencies are pinned in pyproject.toml against the cu130 torch index. Run on two B200s (nscale-dev): 3 passed; carrier restore phase 87 to 108 GB/s aggregate over the two GPUs against a 55 GB/s single-GPU pinned baseline. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Completes docs/reference/cuinterpose.md for reviewers of the cuinterpose stack: the other flows (fork, short-lived processes, agent restart, failed prepare and restore), the lifecycle phases as a state diagram with what each entry point returns per phase, the on-disk and on-wire formats, the invariants, measured performance on B200 (checkpoint 52 s of which CRIU 49 s and cuinterpose 0.43 s; restore 5.7 s of which cuinterpose 0.35 s; host carriers before and after the pinned arena), the pitfalls that shaped the design, what is explicitly unsupported and why, accepted risks, knobs and log lines, the test matrix, and a per-PR reading guide. docs/limitations.md gains the user-facing list of what checkpointing shared CUDA memory does not cover. The vLLM guide's app.py takes extra engine arguments from SNAPSHOT_ENGINE_ARGS (JSON, nested configs converted to vLLM's dataclasses) and the guide explains the tensor-parallel setup: the cuinterpose annotation, an explicit command, PodSnapshot versus SnapshotJob shaping, and keeping FlashInfer's allreduce on POSIX descriptors. The shim README gains the per-suite test descriptions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
galletas1712
force-pushed
the
docs/cuinterpose
branch
from
September 4, 2026 17:52
a510fa7 to
66741f3
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/reference/cuinterpose.md: the design for reviewers (problem, cast, flows as sequence diagrams, lifecycle phases, formats, invariants, measured performance, pitfalls, unsupported cases, accepted risks, knobs and log lines, test matrix, per-PR reading guide).docs/limitations.md: what checkpointing shared CUDA memory does not cover. The vLLM guide gainsSNAPSHOT_ENGINE_ARGSand a tensor-parallel section (annotation, explicit command, PodSnapshot versus SnapshotJob shaping, keeping FlashInfer's allreduce on POSIX descriptors). The shim README shrinks to usage, build, and tests.Validation
make verify-license-headers,make helm-lint; the vLLM guide'sapp.pyis the one used in the end-to-end run described in the other PRs.🤖 Generated with Claude Code