Skip to content

feat(agent): track POSIX CUDA VMM state - #217

Closed
galletas1712 wants to merge 1 commit into
feat/cuinterpose-coordinatorfrom
feat/cuinterpose-posix-tracking
Closed

galletas1712 wants to merge 1 commit into
feat/cuinterpose-coordinatorfrom
feat/cuinterpose-posix-tracking

Conversation

@galletas1712

@galletas1712 galletas1712 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 7 of the eleven-PR cuinterpose stack #293. Closed PR #214 is intentionally not in the active stack.

This PR implements running-state POSIX CUDA VMM tracking without checkpoint/restore lifecycle mechanics.

cuMemCreate has an explicit three-way gate:

Requested handle type Runtime behavior Checkpoint behavior
0 Private allocation passes through natively Allowed
Exactly CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR Tracked and virtualized Supported
Any other nonzero type, including FABRIC or combinations Passes through to the driver Refused by #216 preflight

Only successful unsupported exportable creations increment the process-wide diagnostic. It is intentionally sticky for that process generation because cuinterpose did not track the resource and cannot prove that releasing one handle eliminated every derived sharing dependency. FABRIC logs once that checkpoint will be refused. A fork child discards inherited CUDA state and starts with a fresh count.

Tracked allocations receive stable random allocation IDs and tagged logical handles. Repeated imports and retained handles collapse onto one real driver handle per allocation per process. Mappings are indexed by address range; access grants merge by location; partial/ambiguous operations are refused; a possibly partial driver access failure marks state unknown so inspection fails closed.

On first POSIX export, the creator performs one real CUDA export, keeps that descriptor in the export cache under the allocation ID, and returns a sealed ticket FD to the application. An importing shim reads the ticket, connects to the original creator endpoint, requests the ticket's participant/resource/allocation identity, receives a fresh descriptor over SCM_RIGHTS, imports it, and returns another logical handle. The random allocation ID is the opaque resource identity and export-cache key. Re-export by an importer still names the original creator.

A non-ticket descriptor import passes through and is counted as live raw sharing until its driver handle is released. #216 refuses checkpoint while that count is nonzero. The export cache has independent locking, in-flight pinning, drain-on-drop, and lifecycle quiesce/resume so peer requests remain serviceable without the main state lock.

The library constructor establishes the participant ID and /snapshot-control/cuinterpose-<namespace-pid>.sock. CUINTERPOSE_PARTICIPANT_ID is the explicit identity override used by tests and controlled launchers. The socket is mode 0600; the Pod-local control emptyDir is mounted through a container-name subPath, so ordinary other Pods have no filesystem path to it and Snapshot-managed target containers receive isolated views. The workload Pod remains one trust domain if its author deliberately gives a sidecar the reserved volume. This boundary does not claim protection from node root or an equivalently privileged workload. Fork children discard inherited CUDA bookkeeping and lazily create a new identity/socket on first CUDA activity. Control-endpoint ownership uses scoped cleanup and direct returns rather than goto.

Stack boundary

Based on #216. #292 adds the isolated current allocation-content storage module; #218 invokes it from SAVE_ALLOCATIONS/LOAD_ALLOCATIONS and implements unicast teardown/rebuild.

Validation

The pinned CUDA 13.1 builder passes 13 tracking tests, 9 table/cache tests, and 9 protocol tests with ASan/UBSan where configured. Coverage includes private/POSIX/FABRIC gate behavior, handle alias collapse, range mapping/access semantics, ticket exchange through a forked child, raw import accounting, listener concurrency, descriptor exhaustion, and table churn. make test passes in all Go modules. The final published stack head 0eae9c4f9b66093332f00a332afaf56048e1e64a passed the full make check gate, the CUDA 13.1 production build, and all 73 sanitizer-backed native cuinterpose tests. Its physical-GPU suite passed all 3 tests with no skips on two DRA-assigned NVIDIA B200 GPUs. The unicast test kept an explicit allocation-ID ticket-backed peer mapping per worker live across capture and restore; the multicast test required a nonzero multicast VA and shim-logical handle, exercised BindAddr, and passed its collective and captured-graph replay. Detailed hardware evidence and measurements are in #220.

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

@copy-pr-bot

copy-pr-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Turn the forwarding shim into one that knows which CUDA allocations are shared
between processes. Allocations created with exactly the POSIX file-descriptor
handle type are tracked behind logical handles; every other handle type passes
through untouched. Exports return sealed memfd tickets instead of the driver's
descriptor; an import of a ticket asks the creator process for the real
descriptor over its control socket. The creator answers from an export cache
that holds the one real descriptor per allocation, so its listener never calls
into the driver and never takes the shim's main lock, and a creator busy inside
a long collective call cannot stall its peers.

Bookkeeping lives in hash tables plus a sorted range index that shrink as well
as grow; records are freed when no handle and no mapping remains, so a server
that maps and unmaps for days does not accumulate dead entries or slow down.
One driver handle backs each allocation per process: repeated imports and
cuMemRetainAllocationHandle alias it, and the last logical handle releases it.
A ticket lives as long as the creator holds a handle or mapping, so freeing
memory frees it.

cuMemSetAccess keeps the union of access per location so one call per peer GPU
survives; a call that partly overlaps a tracked mapping, or would grant more
than 32 locations, is refused before the driver sees it. cuMemMap refuses
overlapping a tracked mapping and cuMemUnmap may cover several whole mappings
but not cut through one. Imports of descriptors that are not tickets are
counted so the coordinator can refuse to checkpoint while one is alive. Fork
children drop inherited records and register on their first CUDA activity,
including through the driver's own symbol resolution, so a child that
initializes CUDA without VMM calls is still visible.

The multicast wrappers translate tracked member handles for the bind calls and
otherwise still forward. The shim does not yet answer the coordinator's
lifecycle requests; that is the next change.

Compared with the earlier tracking layers (#152, #166): the export happens once
at ticket time instead of on demand under the main lock; access sets are merged
per location instead of overwritten by the last call; lists that never shrank
are replaced by tables that do; the CUDA context is captured at creation rather
than only at export; unmap and map are range-checked; the listener retries
transient accept errors; the socket path is chmod 0600; NULL and non-zero-flag
exports return CUDA_ERROR_INVALID_VALUE as the driver does. Fourteen state
tests, table and export-cache unit tests, and the earlier suites run under
AddressSanitizer and UndefinedBehaviorSanitizer during the agent image build.

Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
@galletas1712

Copy link
Copy Markdown
Contributor Author

Superseded by the replacement 13-PR C-frontend/Rust-backend cuinterpose draft stack: https://github.com/ai-dynamo/snapshot/stack/339 (#326#338), tracking approved proposal #295. The new stack preserves one reviewable stage per PR, with tests collected in #338 and the design in #337. PageBroker transfer changes remain a separate dependency. Closing this older C implementation; retaining its branch and discussion for reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants