Skip to content

refactor(agent): isolate cuinterpose host-carrier storage - #292

Closed
galletas1712 wants to merge 1 commit into
feat/cuinterpose-posix-trackingfrom
feat/cuinterpose-host-carrier
Closed

galletas1712 wants to merge 1 commit into
feat/cuinterpose-posix-trackingfrom
feat/cuinterpose-host-carrier

Conversation

@galletas1712

@galletas1712 galletas1712 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 8 of the eleven-PR cuinterpose stack #293. Closed PR #214 is intentionally not in the active stack.

This PR isolates the current allocation-content transport in host_carrier.c/.h, with CUDA context scoping in context.c/.h. It does not add lifecycle operations, change allocation eligibility, introduce a backend registry, or add a save-all flag. #218 calls this concrete module from the generic SAVE_ALLOCATIONS and LOAD_ALLOCATIONS operations.

The public surface is deliberately small: save a batch, load a batch, release restored storage, and discard inherited CPU-side state after fork. The lifecycle caller supplies each allocation's context, properties, size, device-handle pointer, and host-address pointer; it remains responsible for eligibility, mappings, access grants, export caches, and importer topology.

On save, the module creates one anonymous host arena per shim process and registers it as portable pinned memory. It groups allocations by CUDA context, maps each group into a contiguous temporary device VA range, and copies D2H on one nonblocking stream with one synchronization per group. CRIU captures the arena as ordinary process memory.

On load, the module verifies or recreates host registration, creates fresh physical allocations with the original properties, stages each context group, copies H2D, and publishes fresh handles only after the complete batch succeeds. Save and load each use an explicit operation-state object with scoped cleanup and direct returns, so partial failures release fresh handles and partial host state without goto. The adjacent context helper restores the caller's prior context and can retain a device primary context when tracking never observed one.

PageBroker boundary

This is a replacement seam, not a speculative pluggable framework. PageBroker currently implements only TransferEngineType::POSIX_COPY and registers only PosixCopyEngine; its protobuf reserves DirectRestoreRequest for a future GPU engine. There is no GPU engine to integrate with today.

When PageBroker defines a GPU/direct engine and a transaction contract for allocation IDs, content, and completion barriers, the two host-carrier lifecycle calls are the intended replacement point. Tickets, participant identity, topology records, and coordinator ordering should not need to become PageBroker-aware.

Stack boundary

Based on #217. #218 owns which supported exportable allocations preserve content and composes this module with unicast teardown, creator publication, and importer reconstruction.

Validation

The pinned CUDA 13.1 builder compiles host_carrier.c into the production shim with -Werror. The #218 lifecycle suite exercises successful save/load, context grouping, missing registration recovery, failure cleanup, and fork behavior through the fake CUDA driver; the final stack passes all 4 lifecycle tests under ASan/UBSan. make test passes in all Go modules. The final published stack head 0eae9c4f9b66093332f00a332afaf56048e1e64a passed the full make check gate, the CUDA 13.1 production build, and all 73 sanitizer-backed native cuinterpose tests. Its physical-GPU suite passed all 3 tests with no skips on two DRA-assigned NVIDIA B200 GPUs. The unicast test kept an explicit allocation-ID ticket-backed peer mapping per worker live across capture and restore; the multicast test required a nonzero multicast VA and shim-logical handle, exercised BindAddr, and passed its collective and captured-graph replay. Detailed hardware evidence and measurements are in #220.

@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Move pinned host arena management, staging mappings, context batching, and D2H/H2D copies behind a small concrete module. The lifecycle remains responsible for selecting allocations and publishing restored topology, so a future PageBroker content path can replace this implementation without changing handle tracking or phase ordering.

Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
@galletas1712

Copy link
Copy Markdown
Contributor Author

Superseded by the replacement 13-PR C-frontend/Rust-backend cuinterpose draft stack: https://github.com/ai-dynamo/snapshot/stack/339 (#326#338), tracking approved proposal #295. The new stack preserves one reviewable stage per PR, with tests collected in #338 and the design in #337. PageBroker transfer changes remain a separate dependency. Closing this older C implementation; retaining its branch and discussion for reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant