feat(agent): checkpoint POSIX CUDA VMM state - #152
Closed
galletas1712 wants to merge 1 commit into
Closed
galletas1712 wants to merge 1 commit into
galletas1712 wants to merge 1 commit into
Conversation
galletas1712
requested review from
Ronkahn21,
danbar2,
dfeigin-nv,
hhzhang16,
hutm,
julienmancuso,
oleg-kushniriov,
shayasoolin and
shmuel-runai
as code owners
September 1, 2026 06:17
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-posix-checkpoint
branch
from
September 1, 2026 06:23
56520f3 to
b5d021d
Compare
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-posix-checkpoint
branch
from
September 1, 2026 07:45
b5d021d to
e5a5144
Compare
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-posix-checkpoint
branch
from
September 1, 2026 07:58
e5a5144 to
cd4e12c
Compare
galletas1712
changed the base branch from
feat/snapshot-cuda-vmm-posix-native
to
feat/snapshot-cuda-vmm-coordinator
September 1, 2026 07:59
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-posix-checkpoint
branch
from
September 1, 2026 08:20
cd4e12c to
3428083
Compare
Track POSIX-shareable CUDA VMM allocations, mappings, access ranges, and ticket imports. Add the static coordinator and order prepare and restore around native CUDA checkpointing. Keep multicast outside this layer and preserve the coordinator across CRIU mount replacement through a pre-opened file descriptor. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
force-pushed
the
feat/snapshot-cuda-vmm-posix-checkpoint
branch
2 times, most recently
from
September 2, 2026 20:20
482957c to
17d8187
Compare
This was referenced Sep 4, 2026
This was referenced Sep 4, 2026
Contributor
Author
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Turn the forwarding shim into one that knows which CUDA allocations are shared between processes. Allocations created with exactly the POSIX file-descriptor handle type are tracked behind logical handles; every other handle type passes through untouched. Exports return sealed memfd tickets instead of the driver's descriptor; an import of a ticket asks the creator process for the real descriptor over its control socket. The creator answers from an export cache that holds the one real descriptor per allocation, so its listener never calls into the driver and never takes the shim's main lock, and a creator busy inside a long collective call cannot stall its peers. Bookkeeping lives in hash tables plus a sorted range index that shrink as well as grow; records are freed when no handle and no mapping remains, so a server that maps and unmaps for days does not accumulate dead entries or slow down. One driver handle backs each allocation per process: repeated imports and cuMemRetainAllocationHandle alias it, and the last logical handle releases it. A ticket lives as long as the creator holds a handle or mapping, so freeing memory frees it. cuMemSetAccess keeps the union of access per location so one call per peer GPU survives; a call that partly overlaps a tracked mapping, or would grant more than 32 locations, is refused before the driver sees it. cuMemMap refuses overlapping a tracked mapping and cuMemUnmap may cover several whole mappings but not cut through one. Imports of descriptors that are not tickets are counted so the coordinator can refuse to checkpoint while one is alive. Fork children drop inherited records and register on their first CUDA activity, including through the driver's own symbol resolution, so a child that initializes CUDA without VMM calls is still visible. The multicast wrappers translate tracked member handles for the bind calls and otherwise still forward. The shim does not yet answer the coordinator's lifecycle requests; that is the next change. Compared with the earlier tracking layers (#152, #166): the export happens once at ticket time instead of on demand under the main lock; access sets are merged per location instead of overwritten by the last call; lists that never shrank are replaced by tables that do; the CUDA context is captured at creation rather than only at export; unmap and map are range-checked; the listener retries transient accept errors; the socket path is chmod 0600; NULL and non-zero-flag exports return CUDA_ERROR_INVALID_VALUE as the driver does. Fourteen state tests, table and export-cache unit tests, and the earlier suites run under AddressSanitizer and UndefinedBehaviorSanitizer during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Turn the forwarding shim into one that knows which CUDA allocations are shared between processes. Allocations created with exactly the POSIX file-descriptor handle type are tracked behind logical handles; every other handle type passes through untouched. Exports return sealed memfd tickets instead of the driver's descriptor; an import of a ticket asks the creator process for the real descriptor over its control socket. The creator answers from an export cache that holds the one real descriptor per allocation, so its listener never calls into the driver and never takes the shim's main lock, and a creator busy inside a long collective call cannot stall its peers. Bookkeeping lives in hash tables plus a sorted range index that shrink as well as grow; records are freed when no handle and no mapping remains, so a server that maps and unmaps for days does not accumulate dead entries or slow down. One driver handle backs each allocation per process: repeated imports and cuMemRetainAllocationHandle alias it, and the last logical handle releases it. A ticket lives as long as the creator holds a handle or mapping, so freeing memory frees it. cuMemSetAccess keeps the union of access per location so one call per peer GPU survives; a call that partly overlaps a tracked mapping, or would grant more than 32 locations, is refused before the driver sees it. cuMemMap refuses overlapping a tracked mapping and cuMemUnmap may cover several whole mappings but not cut through one. Imports of descriptors that are not tickets are counted so the coordinator can refuse to checkpoint while one is alive. Fork children drop inherited records and register on their first CUDA activity, including through the driver's own symbol resolution, so a child that initializes CUDA without VMM calls is still visible. The multicast wrappers translate tracked member handles for the bind calls and otherwise still forward. The shim does not yet answer the coordinator's lifecycle requests; that is the next change. Compared with the earlier tracking layers (#152, #166): the export happens once at ticket time instead of on demand under the main lock; access sets are merged per location instead of overwritten by the last call; lists that never shrank are replaced by tables that do; the CUDA context is captured at creation rather than only at export; unmap and map are range-checked; the listener retries transient accept errors; the socket path is chmod 0600; NULL and non-zero-flag exports return CUDA_ERROR_INVALID_VALUE as the driver does. Fourteen state tests, table and export-cache unit tests, and the earlier suites run under AddressSanitizer and UndefinedBehaviorSanitizer during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Turn the forwarding shim into one that knows which CUDA allocations are shared between processes. Allocations created with exactly the POSIX file-descriptor handle type are tracked behind logical handles; every other handle type passes through untouched. Exports return sealed memfd tickets instead of the driver's descriptor; an import of a ticket asks the creator process for the real descriptor over its control socket. The creator answers from an export cache that holds the one real descriptor per allocation, so its listener never calls into the driver and never takes the shim's main lock, and a creator busy inside a long collective call cannot stall its peers. Bookkeeping lives in hash tables plus a sorted range index that shrink as well as grow; records are freed when no handle and no mapping remains, so a server that maps and unmaps for days does not accumulate dead entries or slow down. One driver handle backs each allocation per process: repeated imports and cuMemRetainAllocationHandle alias it, and the last logical handle releases it. A ticket lives as long as the creator holds a handle or mapping, so freeing memory frees it. cuMemSetAccess keeps the union of access per location so one call per peer GPU survives; a call that partly overlaps a tracked mapping, or would grant more than 32 locations, is refused before the driver sees it. cuMemMap refuses overlapping a tracked mapping and cuMemUnmap may cover several whole mappings but not cut through one. Imports of descriptors that are not tickets are counted so the coordinator can refuse to checkpoint while one is alive. Fork children drop inherited records and register on their first CUDA activity, including through the driver's own symbol resolution, so a child that initializes CUDA without VMM calls is still visible. The multicast wrappers translate tracked member handles for the bind calls and otherwise still forward. The shim does not yet answer the coordinator's lifecycle requests; that is the next change. Compared with the earlier tracking layers (#152, #166): the export happens once at ticket time instead of on demand under the main lock; access sets are merged per location instead of overwritten by the last call; lists that never shrank are replaced by tables that do; the CUDA context is captured at creation rather than only at export; unmap and map are range-checked; the listener retries transient accept errors; the socket path is chmod 0600; NULL and non-zero-flag exports return CUDA_ERROR_INVALID_VALUE as the driver does. Fourteen state tests, table and export-cache unit tests, and the earlier suites run under AddressSanitizer and UndefinedBehaviorSanitizer during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Turn the forwarding shim into one that knows which CUDA allocations are shared between processes. Allocations created with exactly the POSIX file-descriptor handle type are tracked behind logical handles; every other handle type passes through untouched. Exports return sealed memfd tickets instead of the driver's descriptor; an import of a ticket asks the creator process for the real descriptor over its control socket. The creator answers from an export cache that holds the one real descriptor per allocation, so its listener never calls into the driver and never takes the shim's main lock, and a creator busy inside a long collective call cannot stall its peers. Bookkeeping lives in hash tables plus a sorted range index that shrink as well as grow; records are freed when no handle and no mapping remains, so a server that maps and unmaps for days does not accumulate dead entries or slow down. One driver handle backs each allocation per process: repeated imports and cuMemRetainAllocationHandle alias it, and the last logical handle releases it. A ticket lives as long as the creator holds a handle or mapping, so freeing memory frees it. cuMemSetAccess keeps the union of access per location so one call per peer GPU survives; a call that partly overlaps a tracked mapping, or would grant more than 32 locations, is refused before the driver sees it. cuMemMap refuses overlapping a tracked mapping and cuMemUnmap may cover several whole mappings but not cut through one. Imports of descriptors that are not tickets are counted so the coordinator can refuse to checkpoint while one is alive. Fork children drop inherited records and register on their first CUDA activity, including through the driver's own symbol resolution, so a child that initializes CUDA without VMM calls is still visible. The multicast wrappers translate tracked member handles for the bind calls and otherwise still forward. The shim does not yet answer the coordinator's lifecycle requests; that is the next change. Compared with the earlier tracking layers (#152, #166): the export happens once at ticket time instead of on demand under the main lock; access sets are merged per location instead of overwritten by the last call; lists that never shrank are replaced by tables that do; the CUDA context is captured at creation rather than only at export; unmap and map are range-checked; the listener retries transient accept errors; the socket path is chmod 0600; NULL and non-zero-flag exports return CUDA_ERROR_INVALID_VALUE as the driver does. Fourteen state tests, table and export-cache unit tests, and the earlier suites run under AddressSanitizer and UndefinedBehaviorSanitizer during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
galletas1712
added a commit
that referenced
this pull request
Sep 11, 2026
Turn the forwarding shim into one that knows which CUDA allocations are shared between processes. Allocations created with exactly the POSIX file-descriptor handle type are tracked behind logical handles; every other handle type passes through untouched. Exports return sealed memfd tickets instead of the driver's descriptor; an import of a ticket asks the creator process for the real descriptor over its control socket. The creator answers from an export cache that holds the one real descriptor per allocation, so its listener never calls into the driver and never takes the shim's main lock, and a creator busy inside a long collective call cannot stall its peers. Bookkeeping lives in hash tables plus a sorted range index that shrink as well as grow; records are freed when no handle and no mapping remains, so a server that maps and unmaps for days does not accumulate dead entries or slow down. One driver handle backs each allocation per process: repeated imports and cuMemRetainAllocationHandle alias it, and the last logical handle releases it. A ticket lives as long as the creator holds a handle or mapping, so freeing memory frees it. cuMemSetAccess keeps the union of access per location so one call per peer GPU survives; a call that partly overlaps a tracked mapping, or would grant more than 32 locations, is refused before the driver sees it. cuMemMap refuses overlapping a tracked mapping and cuMemUnmap may cover several whole mappings but not cut through one. Imports of descriptors that are not tickets are counted so the coordinator can refuse to checkpoint while one is alive. Fork children drop inherited records and register on their first CUDA activity, including through the driver's own symbol resolution, so a child that initializes CUDA without VMM calls is still visible. The multicast wrappers translate tracked member handles for the bind calls and otherwise still forward. The shim does not yet answer the coordinator's lifecycle requests; that is the next change. Compared with the earlier tracking layers (#152, #166): the export happens once at ticket time instead of on demand under the main lock; access sets are merged per location instead of overwritten by the last call; lists that never shrank are replaced by tables that do; the CUDA context is captured at creation rather than only at export; unmap and map are range-checked; the listener retries transient accept errors; the socket path is chmod 0600; NULL and non-zero-flag exports return CUDA_ERROR_INVALID_VALUE as the driver does. Fourteen state tests, table and export-cache unit tests, and the earlier suites run under AddressSanitizer and UndefinedBehaviorSanitizer during the agent image build. Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
interpose.candposix.cThis is layer 4 of stack #156. Coordinator/protocol and Go orchestration are isolated in #155.
Validation