[TransferEngine] Add native RDMA notifications

[anranxia]

Description

Classic TE sends notifications over TCP even when data transfers use ordinary RDMA. This adds a synchronous TCP round trip to transfers that request notifications.

This PR adds native SEND/RECV notifications to ordinary RdmaTransport:

• Add one notification QP per endpoint and exchange its connection information through the existing endpoint handshake.
• Share a notification CQ within each context, with capacity management and generation-based completion routing.
• Manage notification buffers, backpressure, reconnection, and teardown within the endpoint lifecycle.
• Reuse MC_RDMA_NOTIFY_ENABLED and MC_RDMA_NOTIFY_OOB_FALLBACK, without adding configuration variables or changing rdma_twosided.

A successful native notification send indicates local submission, not confirmed remote receipt. Unsupported peers and oversized messages may fall back to TCP when enabled; failed or uncertain sends are not replayed.

Each endpoint uses 32 MiB of registered host memory for notification buffers. The maximum combined name and msg size is 65,528 bytes.

Module

• Transfer Engine (mooncake-transfer-engine)
• Docs

Type of Change

• New feature
• Performance improvement
• Documentation update

How Has This Been Tested?

Release/CUDA builds succeeded on two RDMA hosts.

Regression tests:

MC_TEST_DEVICE_NAME=mlx5_bond_0 MC_GID_INDEX=3 \
  ./build/mooncake-transfer-engine/tests/rdma_native_notify_test
./build/mooncake-transfer-engine/tests/rdma_endpoint_state_test
./build/mooncake-transfer-engine/tests/endpoint_store_test

All 34 tests passed: 20 notification tests, 8 endpoint-state tests, and 6 endpoint-store tests.

Coverage includes concurrent sends, endpoint reuse and restart, backpressure, fallback, shared-CQ growth, stale completions, and capacity failures.

Performance testing:

Unmodified nixlbench v1.4.1, two NVIDIA H20-3e hosts, GPU-to-GPU READ/WRITE, one logical RDMA device per host, batch=1, one thread, pipeline depth=1, notifications enabled. Each block size used 100 warmup iterations and 1,000 measured iterations, with one sweep per configuration.

The same NIXL binary and plugin loaded either the baseline or modified TE library. All 60 configurations completed successfully.

Operation	Block	Avg latency before → after (µs)	Bandwidth before → after (GB/s)
READ	4 KiB	440.3 → 19.2	0.009303 → 0.213145
READ	8 KiB	430.8 → 19.5	0.019014 → 0.421139
READ	16 KiB	436.3 → 20.8	0.037550 → 0.787125
READ	32 KiB	434.8 → 21.6	0.075370 → 1.515564
READ	64 KiB	448.2 → 23.0	0.146223 → 2.855226
READ	128 KiB	451.4 → 24.6	0.290377 → 5.323153
READ	256 KiB	453.4 → 29.9	0.578225 → 8.775870
READ	512 KiB	449.1 → 37.1	1.167367 → 14.113872
READ	1 MiB	463.1 → 50.4	2.264249 → 20.796414
READ	2 MiB	489.4 → 78.3	4.285429 → 26.792789
READ	4 MiB	555.9 → 136.3	7.545327 → 30.764916
READ	8 MiB	665.9 → 250.5	12.596756 → 33.491602
READ	16 MiB	893.4 → 474.0	18.778181 → 35.395045
READ	32 MiB	1329.1 → 894.1	25.246360 → 37.529815
READ	64 MiB	2176.6 → 1728.3	30.832592 → 38.829699
WRITE	4 KiB	436.0 → 18.0	0.009395 → 0.227454
WRITE	8 KiB	436.4 → 18.0	0.018772 → 0.454001
WRITE	16 KiB	442.0 → 19.5	0.037069 → 0.839086
WRITE	32 KiB	431.1 → 20.2	0.076004 → 1.626042
WRITE	64 KiB	427.5 → 21.4	0.153284 → 3.061858
WRITE	128 KiB	423.2 → 23.1	0.309726 → 5.662102
WRITE	256 KiB	438.8 → 27.7	0.597442 → 9.453103
WRITE	512 KiB	444.6 → 35.3	1.179235 → 14.849407
WRITE	1 MiB	467.2 → 50.3	2.244614 → 20.863447
WRITE	2 MiB	482.4 → 78.7	4.347393 → 26.652839
WRITE	4 MiB	533.3 → 134.9	7.865490 → 31.091950
WRITE	8 MiB	671.5 → 245.5	12.491803 → 34.173937
WRITE	16 MiB	872.8 → 461.0	19.222402 → 36.391751
WRITE	32 MiB	1292.5 → 869.6	25.960052 → 38.584012
WRITE	64 MiB	2111.9 → 1678.9	31.776806 → 39.971258

Latency and bandwidth are taken directly from nixlbench output.

Test results:

• Unit tests pass
• Integration tests pass
• Manual performance testing completed

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• AI tools were used

OpenAI Codex assisted with implementation, tests, benchmarking, and documentation. The human submitter is responsible for reviewing every changed line before submission.

[anranxia]

The CI failure appears unrelated to this PR. Of 215 CTest targets, only fileread_worker_pool_test failed: it expected 4 process threads but observed 5, then failed to return to the original thread-count baseline.

This matches the known issue addressed by #4210: the test counts all threads under /proc/self/task, including background threads created asynchronously by the etcd Go runtime. This PR does not modify the affected test or worker pool.

The RDMA-related CTest targets, build, formatting, and documentation checks passed. CI Gate failed because of the Store test failure.