feat(agent): forward CUDA multicast APIs - #79
Closed
galletas1712 wants to merge 1 commit into
Closed
galletas1712 wants to merge 1 commit into
galletas1712 wants to merge 1 commit into
Conversation
galletas1712
requested review from
Ronkahn21,
danbar2,
dfeigin-nv,
hutm,
julienmancuso,
oleg-kushniriov,
shayasoolin and
shmuel-runai
as code owners
August 18, 2026 22:17
This was referenced Aug 18, 2026
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
from
August 25, 2026 01:29
67bcc30 to
a1d518a
Compare
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
from
August 25, 2026 01:40
a1d518a to
dd404b7
Compare
This was referenced Aug 25, 2026
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
from
August 27, 2026 08:04
dd404b7 to
9b6fd4a
Compare
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
from
August 27, 2026 08:08
9b6fd4a to
2968358
Compare
There was a problem hiding this comment.
Pull request overview
This PR extends the agent’s POSIX CUDA VMM interposer/coordinator so it can checkpoint and restore same-node CUDA multicast groups (in addition to existing POSIX FD–backed unicast VMM allocations), and updates the native integration tests and documentation accordingly.
Changes:
- Adds a new multicast tracking/replay implementation (
multicast.c/.h) and wires it into the interposer and coordinator control flow. - Bumps the native protocol/capability versions and expands record formats to encode multicast topology (objects/devices/bindings/mappings).
- Extends
test_cucheckpoint.pywith a multicast variant and updates docs to run tests undercuda-checkpoint --launch-job.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| agent/Dockerfile | Includes new multicast source/header in the agent image build context. |
| agent/cmd/cuda-vmm-interpose/tests/test_cucheckpoint.py | Adds a multicast test path and ensures tests run via --launch-job. |
| agent/cmd/cuda-vmm-interpose/README.md | Documents multicast support and updated test invocation/contract. |
| agent/cmd/cuda-vmm-interpose/protocol.h | Protocol bump to v2; adds multicast operations/record kinds and expands record/header structs. |
| agent/cmd/cuda-vmm-interpose/posix.h | Capability bump to v2; adds resource kind and multicast metadata fields. |
| agent/cmd/cuda-vmm-interpose/posix.c | Validates v2 capability fields; propagates resource_kind in export requests/responses. |
| agent/cmd/cuda-vmm-interpose/multicast.h | New public multicast interposer API surface. |
| agent/cmd/cuda-vmm-interpose/multicast.c | New multicast lifecycle tracking, prepare (teardown), restore (recreate/import/join/rebind/remap), and validation. |
| agent/cmd/cuda-vmm-interpose/Makefile | Builds multicast implementation into the interposer library. |
| agent/cmd/cuda-vmm-interpose/interpose.c | Integrates multicast into interception, record emission, prepare/restore phases, and export/import handling. |
| agent/cmd/cuda-vmm-interpose/coordinator.c | Validates multicast topology and adds multicast prepare/restore orchestration; bumps state header to v2. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+1308
to
+1309
|
|
||
| *output = -1; |
Comment on lines
566
to
567
| if (fgets(line, sizeof(line), input) == NULL || strcmp(line, "snapshot-cuda-posix-v2\n") != 0) | ||
| return -1; |
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
from
August 27, 2026 08:10
2968358 to
6107dda
Compare
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
from
August 27, 2026 08:13
6107dda to
efac128
Compare
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
from
September 1, 2026 06:23
1cb4214 to
5e66024
Compare
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
from
September 1, 2026 07:45
5e66024 to
8a78b22
Compare
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
2 times, most recently
from
September 1, 2026 08:20
11a2ea6 to
5af639c
Compare
Add transparent wrappers for CUDA multicast entry points and register them across dlsym, cuGetProcAddress, and CUDA runtime resolver paths. Keep multicast state, coordination, and checkpoint replay outside this layer. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
from
September 1, 2026 08:23
5af639c to
d69e430
Compare
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-multicast-posix
branch
2 times, most recently
from
September 2, 2026 20:20
40fea08 to
d69e430
Compare
This was referenced Sep 4, 2026
Contributor
Author
|
Superseded by #215 (multicast forwarding and bind ABI selection) and #219 (multicast state). The re-cut stack (#212 → #220) supersedes this PR. Its body is quoted verbatim in the replacement's Origin section, and each review thread here has a row in the replacement's "Review threads carried" table with what was done about it. |
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Replace the inert placeholder with a shim that gets between an application and libcuda on every path a driver function can be reached: direct linking, dlsym() on a hand-loaded libcuda/libcudart, cuGetProcAddress and its _v2 and _v2_ptsz forms, and cudaGetDriverEntryPoint(ByVersion). Every wrapped entry point forwards the call unchanged; tracking comes in later changes. Callers asking for CUDA 13.1 or newer receive the device-explicit multicast bind ABI. The ticket format (posix.c) is included so it can be unit-tested now; nothing calls it yet. The control-socket helpers and protocol.h landed with the delivery change. Compared with the earlier forwarding layers (#78, #79): - the shim is built with hidden visibility and -z defs, and the build asserts that only the CUDA entry points, dlsym, and cuinterpose_build_info are exported (the earlier build exported twelve generic helper names such as write_all and random_bytes into every workload process); - socket writes use send(MSG_NOSIGNAL) so a peer that hung up cannot kill the workload with SIGPIPE, and a malformed control message can no longer leak a received descriptor; - the shim is built against CUDA 13.1 headers so the _v2 bind wrappers exist, with no runtime dependency on the header version; - environment variables use the SNAPSHOT_ prefix only; - the fake-driver test is a GoogleTest suite, run under AddressSanitizer and UndefinedBehaviorSanitizer, covering all resolution paths for every wrapped symbol, and it runs during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Replace the inert placeholder with a shim that gets between an application and libcuda on every path a driver function can be reached: direct linking, dlsym() on a hand-loaded libcuda/libcudart, cuGetProcAddress and its _v2 and _v2_ptsz forms, and cudaGetDriverEntryPoint(ByVersion). Every wrapped entry point forwards the call unchanged; tracking comes in later changes. Callers asking for CUDA 13.1 or newer receive the device-explicit multicast bind ABI. The ticket format (posix.c) is included so it can be unit-tested now; nothing calls it yet. The control-socket helpers and protocol.h landed with the delivery change. Compared with the earlier forwarding layers (#78, #79): - the shim is built with hidden visibility and -z defs, and the build asserts that only the CUDA entry points, dlsym, and cuinterpose_build_info are exported (the earlier build exported twelve generic helper names such as write_all and random_bytes into every workload process); - socket writes use send(MSG_NOSIGNAL) so a peer that hung up cannot kill the workload with SIGPIPE, and a malformed control message can no longer leak a received descriptor; - the shim is built against CUDA 13.1 headers so the _v2 bind wrappers exist, with no runtime dependency on the header version; - environment variables use the SNAPSHOT_ prefix only; - the fake-driver test is a GoogleTest suite, run under AddressSanitizer and UndefinedBehaviorSanitizer, covering all resolution paths for every wrapped symbol, and it runs during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Replace the inert placeholder with a shim that gets between an application and libcuda on every path a driver function can be reached: direct linking, dlsym() on a hand-loaded libcuda/libcudart, cuGetProcAddress and its _v2 and _v2_ptsz forms, and cudaGetDriverEntryPoint(ByVersion). Every wrapped entry point forwards the call unchanged; tracking comes in later changes. Callers asking for CUDA 13.1 or newer receive the device-explicit multicast bind ABI. The ticket format (posix.c) is included so it can be unit-tested now; nothing calls it yet. The control-socket helpers and protocol.h landed with the delivery change. Compared with the earlier forwarding layers (#78, #79): - the shim is built with hidden visibility and -z defs, and the build asserts that only the CUDA entry points, dlsym, and cuinterpose_build_info are exported (the earlier build exported twelve generic helper names such as write_all and random_bytes into every workload process); - socket writes use send(MSG_NOSIGNAL) so a peer that hung up cannot kill the workload with SIGPIPE, and a malformed control message can no longer leak a received descriptor; - the shim is built against CUDA 13.1 headers so the _v2 bind wrappers exist, with no runtime dependency on the header version; - environment variables use the SNAPSHOT_ prefix only; - the fake-driver test is a GoogleTest suite, run under AddressSanitizer and UndefinedBehaviorSanitizer, covering all resolution paths for every wrapped symbol, and it runs during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Replace the inert placeholder with a shim that gets between an application and libcuda on every path a driver function can be reached: direct linking, dlsym() on a hand-loaded libcuda/libcudart, cuGetProcAddress and its _v2 and _v2_ptsz forms, and cudaGetDriverEntryPoint(ByVersion). Every wrapped entry point forwards the call unchanged; tracking comes in later changes. Callers asking for CUDA 13.1 or newer receive the device-explicit multicast bind ABI. The ticket format (posix.c) is included so it can be unit-tested now; nothing calls it yet. The control-socket helpers and protocol.h landed with the delivery change. Compared with the earlier forwarding layers (#78, #79): - the shim is built with hidden visibility and -z defs, and the build asserts that only the CUDA entry points, dlsym, and cuinterpose_build_info are exported (the earlier build exported twelve generic helper names such as write_all and random_bytes into every workload process); - socket writes use send(MSG_NOSIGNAL) so a peer that hung up cannot kill the workload with SIGPIPE, and a malformed control message can no longer leak a received descriptor; - the shim is built against CUDA 13.1 headers so the _v2 bind wrappers exist, with no runtime dependency on the header version; - environment variables use the SNAPSHOT_ prefix only; - the fake-driver test is a GoogleTest suite, run under AddressSanitizer and UndefinedBehaviorSanitizer, covering all resolution paths for every wrapped symbol, and it runs during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Replace the inert placeholder with a shim that gets between an application and libcuda on every path a driver function can be reached: direct linking, dlsym() on a hand-loaded libcuda/libcudart, cuGetProcAddress and its _v2 and _v2_ptsz forms, and cudaGetDriverEntryPoint(ByVersion). Every wrapped entry point forwards the call unchanged; tracking comes in later changes. Callers asking for CUDA 13.1 or newer receive the device-explicit multicast bind ABI. The ticket format (posix.c) is included so it can be unit-tested now; nothing calls it yet. The control-socket helpers and protocol.h landed with the delivery change. Compared with the earlier forwarding layers (#78, #79): - the shim is built with hidden visibility and -z defs, and the build asserts that only the CUDA entry points, dlsym, and cuinterpose_build_info are exported (the earlier build exported twelve generic helper names such as write_all and random_bytes into every workload process); - socket writes use send(MSG_NOSIGNAL) so a peer that hung up cannot kill the workload with SIGPIPE, and a malformed control message can no longer leak a received descriptor; - the shim is built against CUDA 13.1 headers so the _v2 bind wrappers exist, with no runtime dependency on the header version; - environment variables use the SNAPSHOT_ prefix only; - the fake-driver test is a GoogleTest suite, run under AddressSanitizer and UndefinedBehaviorSanitizer, covering all resolution paths for every wrapped symbol, and it runs during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
multicast.cdlsym,cuGetProcAddress, and CUDA runtime resolversThis is layer 5 of stack #156.
Validation
go test ./api/... ./operator/... ./agent/...