Conversation
gd-0
marked this pull request as draft
September 8, 2026 21:15
Drop the redundant Box around pooled send slots (the Vec is never resized), add a UdpSocket::ring accessor, skip the extra per-socket ACK scan on idle io_uring polls, and dedupe the bench size filter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in Linux io_uring backend for reliable UDP through
UdpConfig::io. Both peers can choose their backend independently; the wire protocol is unchanged. The default remains syscall UDP, and TCP is unchanged.The backend uses one ring per socket, multishot recvmsg with provided buffers, and pooled owned send buffers. Compatible runs retain GSO across mixed-peer batches, with datagram fallback when offload fails. Polling has explicit receive limits; thread handoff and teardown synchronously cancel kernel requests before releasing their buffers. Send slots remain individually boxed so borrowing one slot cannot invalidate kernel pointers into another. Default capacity reserves about 6 MiB per socket. Linux 6.0+ and permission to use io_uring are required; setup failures are reported through the existing socket-creation path.
The audit found and fixed two issues: send backpressure discarded the pending ACK flag, and ACK processing could replenish an intended bounded receive drain. ACKs now remain pending until accepted, and every receive pass is capped. Both UDP backends emit ACKs between receive batches, preventing slow callbacks from deferring them until the receive queue drains. Regression coverage verifies ACK retry after saturation, sustained duplex delivery with one send slot and one receive buffer, and sustained large-message delivery on both backends.
Five alternating measured loopback rounds at
afcad76against maine59fe83, after discarded warmup runs. These measurements precede the subsequent plumbing and syscall ACK-scheduling changes and have not been rerun for the current head:The broadcast gains include improved GSO grouping, so these measure the complete backend rather than io_uring alone. These are local loopback results; physical NIC/WAN performance and non-Linux builds remain unverified.
Validation:
just fmt,just clippy, andcargo test --workspace --all-features --lockedpassed in hosted CI atfd57087with default parallelism (365 tests; four existing ignored doc tests). The ACK regression was reproduced before its fix. Coverage includes the shared UDP suite on both backends, mixed peers, loss/reconnect, DCache, sustained large messages, queue exhaustion, GSO fallback, GRO validation, IPv4/IPv6, buffer-ring tail wraparound, and thread movement.Follow-up validation:
just fmt,just clippy, and the fullflux-networksuite passed (116 tests, run serially after a parallel TCP test encounteredAddrInUse). The newly shared sustained test reproduced syscall ACK starvation before the fix and passes for both backends afterward. An isolated Miri reproducer rejects the unboxed vector indexing pattern under Stacked Borrows and passes with boxed slots; this is not a full-backend Miri run.Compatibility: adding the public
UdpConfig::iofield requires updates to exhaustive struct literals. Literals using struct update syntax keep compiling. Account for this source change in the next release; this PR does not bump the workspace version. Uses the workspace's existing io-uring dependency.Follow-up syscall measurements (2026-09-09): main
e59fe83, pre-ACK-fix89b648c, and current implementationab2d295. Separate saved binaries, five measured rounds after one discarded round per size/revision, rotating revision order; no tests/builds ran concurrently. Ryzen 9 9950X, Linux 7.1.5, release/native, IPv4 loopback, the same pinned cores and requested 16 MiB socket buffers.FLUX_BENCH_TRANSPORT=udp FLUX_BENCH_SCALE=16 FLUX_BENCH_SIZE=2kor64k. Entries are medians of per-run statistics, not pooled percentiles.The ACK fix has a measured throughput cost of 2.5% for 2 KiB bursts and 2.0% for 64 KiB broadcasts in this run; 64 KiB bursts improve 2.5%. Paced throughput is unchanged. These are syscall comparisons, not refreshed io_uring speedup claims. The older io_uring table above remains attributed to its measured revision.
Each receive batch now scans all peers owned by the driver, filtering by socket, and emits at most one pending ACK per matching peer. The scan is O(total driver peers) per batch; the broadcast benchmark covers eight peers, not hundreds.
Default-parallel local validation passes all 116 network tests. The prior CI failure was a sustained-message sender disconnect, not a deadline expiry. Constraining local socket buffers to 212,992 bytes reproduced that disconnect; restoring large buffers on the same two CPUs passed the unchanged test on both backends. CI now sets
rmem_maxandwmem_maxto 16 MiB so Linux can honor the test's requested socket capacity. Message counts, deadlines, assertions, and default test parallelism are unchanged. Hosted CI atfd57087passes the full workspace suite with default parallelism: 365 passed, four existing ignored doc tests. Both shared sustained-message variants pass; lint and semver checks are green. An additional local run with CI compiler flags (RUSTFLAGS="") and two CPUs also passes all 116 network tests with default parallelism.