[1/5] feat: NVSHMEM symmetric window for GPU AFD transport

[specture724]

First of a five-PR stack that brings up GpuAsyncAFDConnector and runs
DeepSeek-V4 on it. Review and merge in order; each PR is based on the one
before it.

1. this PR — NVSHMEM symmetric window (transport)
2. [2/5] feat: the async GPU connector on the symmetric window #333 — the async GPU connector
3. [3/5] feat: run DeepSeek-V4 on the async GPU connector #334 — both GPU runners + DeepSeek-V4
4. [4/5] feat: capture the FFN experts at bucketed padded shapes #335 — bucketed padded FFN expert graphs
5. [5/5][WIP] feat: request-aligned DBO ubatch splitting (flag off by default) #336 — request-aligned DBO ubatch splitting (flag off by default)

What this adds

A one-sided transport with no control plane: a rank writes a whole slot into a
peer's symmetric window — header, routing indices, weights, payload — and stamps
a flag last, on the same stream, so a peer that sees the flag sees the payload.

The flag protocol is what makes the window CUDA-graph-capturable, and that fixes
its shape. A captured stream wait compares against a value baked in at capture
time, so a sender cannot signal with a fresh sequence number every replay. It
signals with a constant marker instead, and the reader resets the flag in-band
once it has consumed the slot, which makes every replay identical.

cuda_rt and nvshmem_rt are thin ctypes bindings: stream memory ops for the
in-stream wait, and NVSHMEM symmetric allocation plus peer pointer lookup.

Scope

No AFD concepts appear in this layer at all — it is a wire format and the
driver calls under it, and it imports nothing from the rest of the plugin. That
is why it is a separate PR: it can be reviewed by itself.

Testing

tests/unit/connectors/gpu/test_symm_window.py pins the slot layout and the
header codec (field offsets, what a header does and does not carry, corrupt
magic, shutdown flag round trip). Runs on CPU, no GPU needed.

Full unit suite is unchanged against main: same 13 pre-existing failures,
none added. pre-commit clean over the PR range.

🤖 Generated with Claude Code

[hsliuustc0106]

Consolidated review of 6509abd (static review; pre-commit/DCO green). No blockers; five P2s:

[P2] Validate the symmetric-heap layout across ranks — afd_plugin/connectors/gpu/symm_window.py:328
__init__ never exchanges total_bytes/capacities over the group, and the receiver only checks magic/version, so config drift between role deployments produces unequal heaps (deep transport crash) or silently clamped partial reads (_view(...)[: sizes[0]] at :472 and :735-755) with no error. All-gather the capacity tuple in __init__ and raise on mismatch, naming the differing parameter.

[P2] Fence the constructor zero-reset against earlier peers — afd_plugin/connectors/gpu/symm_window.py:329
A peer returning from its own collective malloc can issue its first fabric write while this rank's zero_() is still queued behind stream work; the zero erases the arriving flag and the receiver spins forever. End __init__ with a group.barrier() after the synchronize, or state the required post-construction collective as a class-level contract that #333's handshake must provably provide.

[P2] 1,191 of 1,336 added lines are unreachable at this head — afd_plugin/connectors/gpu/symm_window.py:303
Only the test file imports symm_window; the factory registry and connectors/gpu/__init__.py are untouched. Acceptable as part 1/5 if #333/#334 land immediately, but if the stack stalls, main carries dead transport whose ctypes ABI assumptions (struct sizes, (1<<16)+sizeof versioning, libnvshmem_host.so.3 discovery) have zero executable evidence. The series should not be called complete before a GPU-gated test or E2E exercises the real window.

[P2] Justify the module-level mutable globals — afd_plugin/connectors/gpu/cuda_rt.py:31
_lib/_wait_value32/_checked_devices are new process-wide mutable state without the justification AGENTS.md requires; nvshmem_rt.py:110 already models the expected comment.

[P2] Update the connector design page — docs/design/module/connector_contracts.md
afd_plugin/connectors/**/*.py is a primary path of this page: the NVSHMEM substrate needs an evidence-table row (window/rt files ↔ test_symm_window.py) plus a limitations line for the NVSHMEM host library, NVLink-reachable PEs, and stream-mem-ops device gate. Backend-mode/topology rows can ride with #333.

Validation gaps: no Buildkite test-ready run in the rollup; no GPU-marked test anywhere in the series yet; the docstring perf numbers (hot-spin vs backoff TTFT, launch-latency percentages) need linked artifacts when #334's DSV4 runs land.

[specture724]

Thanks — all five addressed in 50c3e04. The stack was rebased end to end, so 333–336 moved with it.

Symmetric-heap layout across ranks. __init__ now all-gathers the geometry before allocating and raises naming the parameter that differs (num_regions, ring_depth, header_words, partial_cap, token_cap, shared_cap, hidden_size, payload_itemsize, slot_bytes, total_bytes). Erroring at construction beats either failure mode you named.

Constructor zero-reset fence. You were right that the collective malloc does not order this: it orders the allocation, and a peer that returns first can land a dispatch while the zero_() is still queued, erasing the arriving flag. __init__ now ends with zero_() → torch.cuda.synchronize() → dist.barrier(group) → torch.cuda.synchronize(), so nobody writes until every rank has finished resetting. I took the barrier rather than the class-level contract, since it costs one collective at startup.

Module-level mutable globals. Documented, in the shape nvshmem_rt.py:112 already models — the driver library is process-global and resolving the handle per call would repeat a dlopen on the layer path, which is the host cost the module exists to remove.

Design page. connector_contracts.md gains an evidence row (window/rt files ↔ test_symm_window.py, noted CPU-only) and a limitations paragraph covering the NVSHMEM host library, the NVLink-reachability requirement behind nvshmem_ptr, the stream-mem-ops device gate, and the fact that the ctypes ABI assumptions have no executable evidence until a GPU-gated test exists.

Unreachable at this head. No change — this is inherent to a 1/5 PR, and git grep confirms your count: test_symm_window.py is the only importer. I'd rather keep the layer reviewable on its own than merge it with its first consumer. Your condition stands: the series should not be called complete before a GPU-gated test or E2E exercises a real window, and I've recorded that in the limitations paragraph rather than leaving it implicit.

Validation gaps noted — the GPU-marked test and the artifacts behind the docstring's perf numbers are still outstanding.