diff --git a/crates/windows-ioring-sys/CHECKLIST.md b/crates/windows-ioring-sys/CHECKLIST.md index 466e1a0cd..81709f9d8 100644 --- a/crates/windows-ioring-sys/CHECKLIST.md +++ b/crates/windows-ioring-sys/CHECKLIST.md @@ -10,7 +10,7 @@ own dated groups, M8-M10 and M15-M18 [here](COMPLETED-CHECKLIST.md#moved-2026-08-30----m15-through-m18-the-testing-strategy-response-to-eight-defects). -**Only `M6+` remains, and it is parked rather than pending** -- see the `M{n}+` convention: it is gated work +**`M20` is pending; `M6+` is parked rather than pending** -- see the `M{n}+` convention: it is gated work with no current obligation, not an unfinished milestone. `M19` below is complete and awaits archival with the next group. @@ -74,6 +74,61 @@ the API whose breaking change 0.2.0 is being cut for, and it is reachable with n and D-45 is added to its table of shipped defects of this shape. **Swept the count restatements too:** that file said "three defects" in four places and is now four, which is the restatement drift the repository's own conventions warn about. + +## M20 -- Repairs from the 2026-08-30 NUMA-sharding measurement + +Queued from +[DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md](../../design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md), +which measured a shipping ARM laptop and found the L3 heuristic's justification does not hold there. These +are documentation and policy repairs only; **no defect was found in `ring_copy`** -- `Policy::select` +already degrades to a whole-machine domain and reports it, which an initial reading of the session got +wrong and the code corrected. + +The design questions the session opened are deliberately **not** queued here. It is still open, and its +conclusions belong to it until it converges. + +- [ ] **M20.1** -- Correct the L3 heuristic's justification in + [DESIGN-NOTES.md](DESIGN-NOTES.md). It currently says the last-level-cache domain "is meaningful on Intel + and ARM too, where the NUMA node often is not." **Measured counter-example:** a Snapdragon X2 Elite + (X2E80100, Qualcomm Oryon; 12 cores, no SMT) reports **zero** L3 cache domains -- `L3CacheSize = 0` from + WMI, and `GetLogicalProcessorInformationEx` yields L1 and L2 only, with L2 forming two domains of six + processors that agree with the two `Module` domains. The claim that L3 is meaningful on ARM is false on a + shipping part. Keep the finding that L3 beats the NUMA node; restate the rule as **the outermost cache + level that actually partitions the machine**, and say what happens when no such level is reported. Sweep + every restatement of the L3 rule per the repository's blast-radius convention, including the README and + `ring_copy`'s `policy.rs` doc comments, not only the one sentence quoted above. + +- [ ] **M20.2** -- Record the measurement itself as a decision in + [DESIGN-NOTES.md](DESIGN-NOTES.md), so the next reader inherits the datapoint rather than re-measuring: + an ARM Windows laptop with no L3 at all, and zero `Win32_NumaNode` instances, is the *common* consumer + shape now rather than an exotic one. This is the ARM sibling of the existing zero-NUMA-node VM + observation and belongs beside it. + +- [ ] **M20.3** -- Make `ring_copy`'s degraded-fallback path observable in a test. The whole-machine + fallback in `Policy::select` is the branch every zero-relation machine takes, and this session was the + first time anyone confirmed it runs. Assert both halves on a synthetic topology: that a policy whose + relation is absent returns one whole-machine domain with `degraded = true`, and that a policy whose + relation is present is **not** flagged degraded -- the second half matters because a test of the first + alone would pass against a function that always degrades. + +- [ ] **M20.4** -- Correct "What is not reachable" in [DESIGN-NOTES.md](DESIGN-NOTES.md). It says mapping a + file handle to its backing device's NUMA node "has no clean user-mode path" and "means walking volume to + disk to device instance and reading `DEVPKEY_Device_Numa_Node`". **That is wrong on mechanism.** + `FSCTL_QUERY_VOLUME_NUMA_INFO` is documented in the IFS docs, takes a handle to a **file or directory** + directly, and returns `FSCTL_QUERY_VOLUME_NUMA_INFO_OUTPUT { ULONG NumaNode }`. No walking required. + The **conclusion survives for a better reason**, and that is the point of the rewrite: the documented + meaning is the node the *volume* resides on, not where the file's extents live, so it cannot answer + "which ring should this file's I/O go to" even when it succeeds; and it is absent whenever the device + advertised no proximity domain. Record `GetNumaNodeNumberFromHandle` as the other path -- a wrapper over + `NtQueryInformationFile` with `FileNumaNodeInformation` (class 53) -- and that PHNT and the WDK mark that + class **reserved for system use**, so this crate must not build on it. State plainly that no published + measurement of either call succeeding on an ordinary NTFS data file could be found, and cite + [file-handle-numa-spike.rs](design-sessions/spikes/file-handle-numa-spike.rs) as the unrun instrument. + **Blocked on hardware, not on a decision:** settling it needs a multi-node machine with storage whose + PDO advertises a proximity domain. Write the correction now (the documentation defect is independent of + the measurement) and leave the empirical question open. + + ## M6+ -- Model B: explicit-thread delivery and affinity Parked, not pending. Deferred by the engineer's explicit direction during the 2026-08-22 design session, diff --git a/crates/windows-ioring-sys/Cargo.toml b/crates/windows-ioring-sys/Cargo.toml index 199007e75..5a211bebb 100644 --- a/crates/windows-ioring-sys/Cargo.toml +++ b/crates/windows-ioring-sys/Cargo.toml @@ -89,7 +89,24 @@ windows-threadpool-sys = { version = "0.1.3", path = "../windows-threadpool-sys" # crate rather than raw `GetLogicalProcessorInformationEx` calls. M7's # ring-copy sample additionally needs the `serde` feature, to load a fed-in # topology description (D-10 in windows-topology-sys's DESIGN-NOTES.md). -windows-topology-sys = { version = "0.2.0", path = "../windows-topology-sys", features = [ +# +# **Path-only, with no `version`, deliberately.** Cargo omits a versionless +# dev-dependency from the published manifest entirely -- verified by packaging +# this crate and reading the result -- and that is the right outcome here. A +# `version` would have been published as a requirement nothing ever checks: +# publish verification builds the library alone, not examples or tests, so the +# pin was never exercised at the one moment it was supposed to matter. +# +# What it *did* do was break the workspace. Cargo enforces `path` + `version` +# agreement at every build regardless of dependency kind, so this pin reading +# "0.1.0" against a `windows-topology-sys` bumped to 0.2.0 failed `cargo +# metadata` for every crate here -- turning a release of an unrelated crate into +# a red `main`. Deleting it removes the last of those. +# +# The examples still build from a checkout, which is where anyone reads them. +# They are not buildable from the packaged tarball, which they were only ever +# incidentally. +windows-topology-sys = { path = "../windows-topology-sys", features = [ "serde", ] } # M7's ring-copy sample deserializes a fed-in topology description (--topology). diff --git a/crates/windows-ioring-sys/PLANS.md b/crates/windows-ioring-sys/PLANS.md index 1cb4183a4..b05ab1a68 100644 --- a/crates/windows-ioring-sys/PLANS.md +++ b/crates/windows-ioring-sys/PLANS.md @@ -6,4 +6,4 @@ contained are archived in [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md). Desi | Path to CHECKLIST.md | Status | Brief description | Design Notes | |---|---|---|---| -| [CHECKLIST.md](CHECKLIST.md) | in progress | Memory-safe Rust over the Windows 11 / Server 2022 `IoRing` submission/completion ring, as a separate crate from `windows-overlapped-io-sys` (duplicate-then-decide). Covers ring lifecycle and capability negotiation, zero-allocation token-owned buffers, the batch submission builder, threadless delivery through `ThreadpoolWait`, file/buffer registration, consumer-facing documentation, and the `ring-copy` topology-aligned sample (M1-M7 archived). The pinned-thread (Model B) architecture remains parked as `M6+` by the engineer's explicit direction. `M8` (complete) closed a PR #20 review finding: `FileRef::Raw(HANDLE)`'s lifetime gap, fixed with `unsafe fn` raw entry points plus a safe, `Arc`-backed `SharedFile` wrapper for the common case. `M9` (complete) closed further PR #20 review findings: cross-ring `Token`/`RegisteredFile`/`RegisteredBuffers` confusion (a new per-ring `RingId`, checked at claim/push time), `PendingBufferRegistration` freeing its buffers instead of leaking them on an unclaimed drop, and `Batch::do_submit` letting `Drop` silently retry an already-attempted, already-failed submit. `M10` (active) finished auditing the ring completion contract against all ten specification-gap categories (M10.1-M10.3 complete): category 3 found that `supports` answers for the kernel's op table rather than this crate's push surface and that the registration one-shot is spent by queueing rather than succeeding (D-28); categories 1, 2, 6, 8 and 9 established the load-bearing rule that **every successfully queued SQE produces exactly one completion**, plus that this crate deliberately joins nothing (D-29, D-30); and D-14's registration-index continuity assumption was **dissolved rather than measured** -- the collision it guarded against needs a second registration, which was forbidden the day after D-14 was written, leaving only the reserved-not-confirmed meaning of the public counts to state (D-31). The audit also surfaced two API gaps, now queued as work rather than left in the design notes: `M10.4` (complete) gave `FileRef::Registered` safe entry points, since a registered index carries no lifetime obligation and the `unsafe` guarding it was vacuous (D-29): the safe pushes are now generic over a sealed `FileTarget` trait whose associated `Guard` type carries the one real difference between the two targets, which also made the fully-registered (registered file *and* registered buffer) combination expressible for the first time, non-breakingly (D-33). Investigating M10's own recorded test failures then found a **live use-after-free in shipped 0.1.2** and fixed it as `M10.6`: `BuildIoRingRegisterBuffers` reads its `IORING_BUFFER_INFO` array when the op runs rather than at build time -- the opposite of its file-handle sibling, which the rustdoc had wrongly generalized across -- so the array is now owned by the `IoRing` (D-32). `M10.5` (complete) added named predicates for the conditions a consumer must branch on -- `IORING_E_SUBMISSION_QUEUE_FULL` above all, which every push's rustdoc names as the backpressure signal but which `io::Error::kind()` cannot discriminate (D-30): a complete `RingCondition` enum, predicates for the runtime-actionable conditions, and a sealed `IoRingErrorExt` that puts them on `io::Error` so the downcast is named once rather than hand-rolled per call site (D-34). **M10 is complete.** **`M11` is complete and archived** (2026-08-28): it made the completion event a ring primitive, prompted by an external consumer proposal. `IoRing::completion_event` returns an owned duplicate of the ring's own event so a caller can wait on the ring alongside other handles without surrendering it (D-20); its contract is pinned by eleven sabotage-verified tests; `EventDelivery` is re-expressed on top of it, leaving one `SetIoRingCompletionEvent` call site; `windows-threadpool-sys` moved behind a default-on `threadpool` feature with CI building both combinations (D-22); the wakeup shapes and the barrier's ring-edge limit were swept across every place that states them; and `examples/model_b_multiplexed.rs` works the multiplexed shape end to end. The spike that answered the proposal established that the completion event is **edge-triggered** on the completion queue going empty to non-empty (D-19) -- which also exposed a live bug in shipped 0.1.2, where `EventDelivery` permanently stranded completions queued before handover, fixed in M11.3 by the same change that consolidated `EventDelivery` onto the new primitive. **`M12` is complete and archived** (2026-08-28): it addressed durability, from the same exchange. A spike had established that a flush **without** `DRAIN_PRECEDING_OPS` does not cover preceding writes (D-23) while the barrier that fixes it is a full ring-wide stall spanning submissions (D-24), making `Batch::flush` with default options a silent data-loss bug rather than a missing feature. `Batch::flush`/`flush_raw` now require an explicit `FlushCoverage`, so that spelling no longer exists; a `NO_BUFFERING` integration test proves the barrier's behaviour rather than its flag, and measured that *which direction* the reordering shows in is device-dependent (amending D-23); and the parameters the crate had hardcoded away are exposed as `WriteCaching` (`FILE_WRITE_FLAGS`) and `FlushMode` (`FILE_FLUSH_MODE`, whose `NoSync` is the one mode that commits nothing). Durability had been absent from `lib.rs` and `README.md` entirely, and both now state the three facts. **`M13` is complete and archived** (2026-08-29): the `epoch_log` sample is the vehicle D-26 makes for carrying durability *policy* to consumers without this crate owning it. Its own durability contract is written down first, in its own words (Design Autonomy), then implemented -- records composed into a registered arena and appended, epochs closed by one covering flush whose completion is what makes `is_durable` answer `true`, and a multiplexed wait on the ring's completion event alongside a shutdown latch. The replay pass is what turns it from a demonstration into evidence: it holds the durable region to a strict standard, tolerates a torn tail as the contract requires, and is itself proved able to fail by a negative control. Writing it also found and fixed a gap in the crate (D-35: per-buffer outstanding accounting and `RegisteredBuffers::get_mut`). **`M14` is complete and archived** (2026-08-29): the second half of the sample, covering the two things the ring cannot do for a consumer. A non-ring `FSCTL` (`FSCTL_SET_ZERO_DATA`, reclaiming a retired segment) is ordered against ring epochs by the log itself, since `drain_preceding` orders SQEs against SQEs (D-24) and reaches across neither the ring boundary nor a second ring; a thread-pool control plane runs checkpointing as Model A on a *second* ring while the log thread keeps Model B for the data path, because D-21 forbids a ring given to `EventDelivery` also being waited on directly -- so the ordering chain crosses log thread to pool thread to reclaim worker and back with the log thread blocking for none of it. All three epoch-commit strategies are implemented behind one interface, checked both by replay and by requiring the three to be **byte-identical**, and measured on the running machine. The measurement's finding is that the three are **indistinguishable** here -- the cross-strategy spread is the size of one strategy's run-to-run spread, because every strategy pays one device flush per epoch at hundreds of microseconds while their real differences land in the tens -- and it found two harness bugs nothing else caught, including one where a barrier benchmark that awaits each commit before appending again measures the barrier as free. Both findings are promoted into "Durability on the ring" in [DESIGN-NOTES.md](DESIGN-NOTES.md), since both are about the design rather than the demonstration. **`M15`-`M18` (not started) are the testing-strategy response to the eight defects the 0.1.x line and the M11-M14 branch produced.** They are organised by *defect population* rather than by technique, because the populations need different tools and one of them needs a tool that does not exist: (A) preconditions never varied -- every `event_delivery` test handed over a fresh ring, which is why [#47](https://github.com/MikeGrier/windows-threadpool-sys/issues/47) survived; (B) failure paths never taken -- no test ever ran `completion.result()` returning `Err`, which is why the checkpoint path could authorise a reclaim after a failed write; and (C) *permissions rather than behaviour* -- `&mut Vec` permits `reserve`/`resize`/reassign though no code path performs it, which is [D-35](DESIGN-NOTES.md#d-35) and [D-36](DESIGN-NOTES.md#d-36), the two most severe findings, and **no runtime technique reaches that population at all**. M15 and M16 gate the 0.2.0 release. M15 is deterministic memory instrumentation: a guard-page global allocator, chosen over Application Verifier / PageHeap by measurement rather than assumption ([D-37](DESIGN-NOTES.md#d-37) -- PageHeap works and `reg add` alone is enough, but IFEO is keyed by image file name and one test target produced six distinct hashed names in a day), plus a tracked poison pattern covering the gap guard pages structurally cannot see ([D-38](DESIGN-NOTES.md#d-38) -- a guard page catches access to memory that should not be touched, and is blind to the kernel writing into a live, valid buffer, which is exactly what `write_registered` and `read_registered` promise it will not do). M16 makes the contract executable: a public `RingContract` rendering [the category-2 rule](DESIGN-NOTES.md#one-sqe-one-completion), "one SQE, exactly one completion", checkable rather than merely stated, plus a fault-injection seam that finally takes the failure paths. M17 covers A by generating over the operation space instead of enumerating it by hand, gated on an explicit decision about randomized sampling that this component's conventions require be approved and recorded rather than assumed. M18 covered C, where review is a *primary* technique rather than a backstop, and added `cargo-mutants` against a measured rate of vacuous tests. **M8 through M18 are now complete and archived**, leaving only `M6+`, which is parked rather than pending. The strategy as a whole -- which technique reaches which population, what each one actually found, and what none of them reach -- is recorded in [DESIGN-NOTES.md](DESIGN-NOTES.md#testing-strategy-m185); mutation coverage went from 79.7% to 95.8%, and the borrow-surface audit found one further defect of the same shape as the two that prompted it ([D-43](DESIGN-NOTES.md#d-43)). A mock `IoRing` was considered and **rejected**: both shipped defects were the kernel behaving differently from this crate's assumptions, so a mock would have encoded the same assumptions and passed both bugs green. | [DESIGN-NOTES.md](DESIGN-NOTES.md), [DESIGN-SESSION-2026-08-28-completion-event-multiplexing.md](design-sessions/DESIGN-SESSION-2026-08-28-completion-event-multiplexing.md), [DESIGN-SESSION-2026-08-28-external-consumer-correspondence.md](design-sessions/DESIGN-SESSION-2026-08-28-external-consumer-correspondence.md) | +| [CHECKLIST.md](CHECKLIST.md) | in progress | Memory-safe Rust over the Windows 11 / Server 2022 `IoRing` submission/completion ring, as a separate crate from `windows-overlapped-io-sys` (duplicate-then-decide). Covers ring lifecycle and capability negotiation, zero-allocation token-owned buffers, the batch submission builder, threadless delivery through `ThreadpoolWait`, file/buffer registration, consumer-facing documentation, and the `ring-copy` topology-aligned sample (M1-M7 archived). The pinned-thread (Model B) architecture remains parked as `M6+` by the engineer's explicit direction. `M8` (complete) closed a PR #20 review finding: `FileRef::Raw(HANDLE)`'s lifetime gap, fixed with `unsafe fn` raw entry points plus a safe, `Arc`-backed `SharedFile` wrapper for the common case. `M9` (complete) closed further PR #20 review findings: cross-ring `Token`/`RegisteredFile`/`RegisteredBuffers` confusion (a new per-ring `RingId`, checked at claim/push time), `PendingBufferRegistration` freeing its buffers instead of leaking them on an unclaimed drop, and `Batch::do_submit` letting `Drop` silently retry an already-attempted, already-failed submit. `M10` (active) finished auditing the ring completion contract against all ten specification-gap categories (M10.1-M10.3 complete): category 3 found that `supports` answers for the kernel's op table rather than this crate's push surface and that the registration one-shot is spent by queueing rather than succeeding (D-28); categories 1, 2, 6, 8 and 9 established the load-bearing rule that **every successfully queued SQE produces exactly one completion**, plus that this crate deliberately joins nothing (D-29, D-30); and D-14's registration-index continuity assumption was **dissolved rather than measured** -- the collision it guarded against needs a second registration, which was forbidden the day after D-14 was written, leaving only the reserved-not-confirmed meaning of the public counts to state (D-31). The audit also surfaced two API gaps, now queued as work rather than left in the design notes: `M10.4` (complete) gave `FileRef::Registered` safe entry points, since a registered index carries no lifetime obligation and the `unsafe` guarding it was vacuous (D-29): the safe pushes are now generic over a sealed `FileTarget` trait whose associated `Guard` type carries the one real difference between the two targets, which also made the fully-registered (registered file *and* registered buffer) combination expressible for the first time, non-breakingly (D-33). Investigating M10's own recorded test failures then found a **live use-after-free in shipped 0.1.2** and fixed it as `M10.6`: `BuildIoRingRegisterBuffers` reads its `IORING_BUFFER_INFO` array when the op runs rather than at build time -- the opposite of its file-handle sibling, which the rustdoc had wrongly generalized across -- so the array is now owned by the `IoRing` (D-32). `M10.5` (complete) added named predicates for the conditions a consumer must branch on -- `IORING_E_SUBMISSION_QUEUE_FULL` above all, which every push's rustdoc names as the backpressure signal but which `io::Error::kind()` cannot discriminate (D-30): a complete `RingCondition` enum, predicates for the runtime-actionable conditions, and a sealed `IoRingErrorExt` that puts them on `io::Error` so the downcast is named once rather than hand-rolled per call site (D-34). **M10 is complete.** **`M11` is complete and archived** (2026-08-28): it made the completion event a ring primitive, prompted by an external consumer proposal. `IoRing::completion_event` returns an owned duplicate of the ring's own event so a caller can wait on the ring alongside other handles without surrendering it (D-20); its contract is pinned by eleven sabotage-verified tests; `EventDelivery` is re-expressed on top of it, leaving one `SetIoRingCompletionEvent` call site; `windows-threadpool-sys` moved behind a default-on `threadpool` feature with CI building both combinations (D-22); the wakeup shapes and the barrier's ring-edge limit were swept across every place that states them; and `examples/model_b_multiplexed.rs` works the multiplexed shape end to end. The spike that answered the proposal established that the completion event is **edge-triggered** on the completion queue going empty to non-empty (D-19) -- which also exposed a live bug in shipped 0.1.2, where `EventDelivery` permanently stranded completions queued before handover, fixed in M11.3 by the same change that consolidated `EventDelivery` onto the new primitive. **`M12` is complete and archived** (2026-08-28): it addressed durability, from the same exchange. A spike had established that a flush **without** `DRAIN_PRECEDING_OPS` does not cover preceding writes (D-23) while the barrier that fixes it is a full ring-wide stall spanning submissions (D-24), making `Batch::flush` with default options a silent data-loss bug rather than a missing feature. `Batch::flush`/`flush_raw` now require an explicit `FlushCoverage`, so that spelling no longer exists; a `NO_BUFFERING` integration test proves the barrier's behaviour rather than its flag, and measured that *which direction* the reordering shows in is device-dependent (amending D-23); and the parameters the crate had hardcoded away are exposed as `WriteCaching` (`FILE_WRITE_FLAGS`) and `FlushMode` (`FILE_FLUSH_MODE`, whose `NoSync` is the one mode that commits nothing). Durability had been absent from `lib.rs` and `README.md` entirely, and both now state the three facts. **`M13` is complete and archived** (2026-08-29): the `epoch_log` sample is the vehicle D-26 makes for carrying durability *policy* to consumers without this crate owning it. Its own durability contract is written down first, in its own words (Design Autonomy), then implemented -- records composed into a registered arena and appended, epochs closed by one covering flush whose completion is what makes `is_durable` answer `true`, and a multiplexed wait on the ring's completion event alongside a shutdown latch. The replay pass is what turns it from a demonstration into evidence: it holds the durable region to a strict standard, tolerates a torn tail as the contract requires, and is itself proved able to fail by a negative control. Writing it also found and fixed a gap in the crate (D-35: per-buffer outstanding accounting and `RegisteredBuffers::get_mut`). **`M14` is complete and archived** (2026-08-29): the second half of the sample, covering the two things the ring cannot do for a consumer. A non-ring `FSCTL` (`FSCTL_SET_ZERO_DATA`, reclaiming a retired segment) is ordered against ring epochs by the log itself, since `drain_preceding` orders SQEs against SQEs (D-24) and reaches across neither the ring boundary nor a second ring; a thread-pool control plane runs checkpointing as Model A on a *second* ring while the log thread keeps Model B for the data path, because D-21 forbids a ring given to `EventDelivery` also being waited on directly -- so the ordering chain crosses log thread to pool thread to reclaim worker and back with the log thread blocking for none of it. All three epoch-commit strategies are implemented behind one interface, checked both by replay and by requiring the three to be **byte-identical**, and measured on the running machine. The measurement's finding is that the three are **indistinguishable** here -- the cross-strategy spread is the size of one strategy's run-to-run spread, because every strategy pays one device flush per epoch at hundreds of microseconds while their real differences land in the tens -- and it found two harness bugs nothing else caught, including one where a barrier benchmark that awaits each commit before appending again measures the barrier as free. Both findings are promoted into "Durability on the ring" in [DESIGN-NOTES.md](DESIGN-NOTES.md), since both are about the design rather than the demonstration. **`M15`-`M18` (not started) are the testing-strategy response to the eight defects the 0.1.x line and the M11-M14 branch produced.** They are organised by *defect population* rather than by technique, because the populations need different tools and one of them needs a tool that does not exist: (A) preconditions never varied -- every `event_delivery` test handed over a fresh ring, which is why [#47](https://github.com/MikeGrier/windows-threadpool-sys/issues/47) survived; (B) failure paths never taken -- no test ever ran `completion.result()` returning `Err`, which is why the checkpoint path could authorise a reclaim after a failed write; and (C) *permissions rather than behaviour* -- `&mut Vec` permits `reserve`/`resize`/reassign though no code path performs it, which is [D-35](DESIGN-NOTES.md#d-35) and [D-36](DESIGN-NOTES.md#d-36), the two most severe findings, and **no runtime technique reaches that population at all**. M15 and M16 gate the 0.2.0 release. M15 is deterministic memory instrumentation: a guard-page global allocator, chosen over Application Verifier / PageHeap by measurement rather than assumption ([D-37](DESIGN-NOTES.md#d-37) -- PageHeap works and `reg add` alone is enough, but IFEO is keyed by image file name and one test target produced six distinct hashed names in a day), plus a tracked poison pattern covering the gap guard pages structurally cannot see ([D-38](DESIGN-NOTES.md#d-38) -- a guard page catches access to memory that should not be touched, and is blind to the kernel writing into a live, valid buffer, which is exactly what `write_registered` and `read_registered` promise it will not do). M16 makes the contract executable: a public `RingContract` rendering [the category-2 rule](DESIGN-NOTES.md#one-sqe-one-completion), "one SQE, exactly one completion", checkable rather than merely stated, plus a fault-injection seam that finally takes the failure paths. M17 covers A by generating over the operation space instead of enumerating it by hand, gated on an explicit decision about randomized sampling that this component's conventions require be approved and recorded rather than assumed. M18 covered C, where review is a *primary* technique rather than a backstop, and added `cargo-mutants` against a measured rate of vacuous tests. **M8 through M18 are now complete and archived**; `M20` is pending, and `M6+` is parked rather than pending. The strategy as a whole -- which technique reaches which population, what each one actually found, and what none of them reach -- is recorded in [DESIGN-NOTES.md](DESIGN-NOTES.md#testing-strategy-m185); mutation coverage went from 79.7% to 95.8%, and the borrow-surface audit found one further defect of the same shape as the two that prompted it ([D-43](DESIGN-NOTES.md#d-43)). A mock `IoRing` was considered and **rejected**: both shipped defects were the kernel behaving differently from this crate's assumptions, so a mock would have encoded the same assumptions and passed both bugs green. **M20** queues the documentation and policy-test repairs from the 2026-08-30 NUMA-sharding measurement: a shipping ARM laptop reports no L3 cache domain at all, which falsifies the justification given for the last-level-cache heuristic (though not the heuristic's preference over the NUMA node, and not `ring_copy`, whose degraded fallback already handles it correctly). | [DESIGN-NOTES.md](DESIGN-NOTES.md), [DESIGN-SESSION-2026-08-28-completion-event-multiplexing.md](design-sessions/DESIGN-SESSION-2026-08-28-completion-event-multiplexing.md), [DESIGN-SESSION-2026-08-28-external-consumer-correspondence.md](design-sessions/DESIGN-SESSION-2026-08-28-external-consumer-correspondence.md), [DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md](../../design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md) | diff --git a/crates/windows-ioring-sys/UNRESOLVED-TEST-FAILURES.md b/crates/windows-ioring-sys/UNRESOLVED-TEST-FAILURES.md index abf45c5f5..d47693d65 100644 --- a/crates/windows-ioring-sys/UNRESOLVED-TEST-FAILURES.md +++ b/crates/windows-ioring-sys/UNRESOLVED-TEST-FAILURES.md @@ -4,4 +4,18 @@ Pre-existing failures that do not block an unrelated commit, recorded per the re checklist-execution rules. When one is resolved, move its entry into a sibling [RESOLVED-TEST-FAILURES.md](RESOLVED-TEST-FAILURES.md) (append-only) rather than deleting it. -None currently. +## `flush_barrier::a_covering_flush_waits_for_preceding_writes_and_an_unordered_one_does_not` + +**Flaky under a full-workspace run, green in isolation.** Observed once during +`cargo test --workspace --all-features` on 2026-09-03 (2899 of 2900 passed); the +same test re-run on its own with `--test flush_barrier` passes. + +Not caused by the change that observed it, which touched +`windows-topology-sys` only. The test measures real I/O ordering, so it is +sensitive to load: a full workspace run has every other suite competing for the +disk, and the window this test asserts is a timing one. + +Recorded rather than fixed because the failure mode -- a load-sensitive +assertion in a real-I/O test -- needs a decision about whether the test should +be made load-independent or marked as serial, and that is not this change's +scope. It has not been seen to fail in CI. diff --git a/crates/windows-ioring-sys/src/batch.rs b/crates/windows-ioring-sys/src/batch.rs index a8ac20f6e..b25b35dda 100644 --- a/crates/windows-ioring-sys/src/batch.rs +++ b/crates/windows-ioring-sys/src/batch.rs @@ -651,6 +651,18 @@ impl RegisteredBuffers { } /// Whether this registration holds no buffers. + /// + /// Always `false` in practice, and a mutation run reports the constant as + /// surviving for that reason rather than for want of a test. Nothing in + /// this crate rejects an empty vector, but the kernel refuses the + /// submission with `E_INVALIDARG`, so a caller never holds an empty + /// registration. `windows_refuses_an_empty_buffer_registration` pins that + /// platform behaviour; if a future Windows accepts it, that test fails and + /// this becomes reachable. + /// + /// Kept because it is half of the `len`/`is_empty` pair every Rust + /// collection surface offers, and because "cannot happen today" is a + /// weaker claim than "cannot happen". #[must_use] pub fn is_empty(&self) -> bool { self.buffers.is_empty() diff --git a/crates/windows-ioring-sys/src/batch/tests.rs b/crates/windows-ioring-sys/src/batch/tests.rs index b2fa54d86..28b59a9fd 100644 --- a/crates/windows-ioring-sys/src/batch/tests.rs +++ b/crates/windows-ioring-sys/src/batch/tests.rs @@ -446,3 +446,92 @@ fn dropping_a_registration_with_work_outstanding_is_refused() { // already in progress would abort instead of failing the test. drop(buffers); } + +#[test] +fn require_refuses_an_op_the_ring_does_not_support() { + // `Batch::require -> Ok(())` survived: every op this crate names is + // genuinely supported on any real host these tests run on, so nothing + // distinguished the real check from one that always passes. + // `set_supported_ops_for_test` constructs a ring that lacks one, so the + // refusal has something to refuse. + let mut ring = IoRing::new(8, 8).expect("create ring"); + ring.set_supported_ops_for_test(&[crate::Op::Nop]); + let batch = Batch::new(&mut ring); + + let error = batch + .require(crate::Op::Read) + .expect_err("Read was left out of the constructed capability set"); + assert_eq!(error.kind(), std::io::ErrorKind::Unsupported); + + batch + .require(crate::Op::Nop) + .expect("Nop is in the constructed capability set"); +} + +#[test] +fn the_debug_rendering_names_the_registration_and_its_identity() { + // `>::fmt -> Ok(Default::default())` + // survived: that mutation writes nothing to the formatter, so the + // rendering comes back empty regardless of what the registration holds. + let mut ring = IoRing::new(8, 8).expect("create ring"); + let mut batch = Batch::new(&mut ring); + let pending = batch + .register_buffers(vec![vec![0_u8; 64]]) + .expect("queue buffer registration"); + let rendering = format!("{pending:?}"); + assert!( + rendering.contains("PendingBufferRegistration"), + "got {rendering}" + ); + assert!( + rendering.contains(&pending.user_data().to_string()), + "the operation's identity must appear: {rendering}" + ); +} + +#[test] +fn windows_refuses_an_empty_buffer_registration() { + // Written while chasing `RegisteredBuffers::is_empty -> false`, which a + // mutation run reports as surviving. It survives because the state it would + // misreport **cannot be reached**: nothing in this crate rejects an empty + // vector -- `register_buffers` only checks that the count fits a `u32` and + // that the ring has no prior table -- but the kernel refuses the submission + // with `E_INVALIDARG`, so no caller ever holds an empty registration and + // `is_empty` never has occasion to return `true`. + // + // That makes the mutant unreachable rather than untested, and manufacturing + // an in-crate struct literal to kill it would assert a shape the API cannot + // produce. What is worth pinning is the platform behaviour itself, because + // it is undocumented, it is the reason the accessor looks untested, and a + // future version that started accepting empty registrations would change + // which states this crate can be in. + let mut ring = IoRing::new(8, 8).expect("create ring"); + let mut batch = Batch::new(&mut ring); + let pending = batch + .register_buffers(Vec::>::new()) + .expect("this crate queues it; the refusal comes from the kernel"); + let user_data = pending.user_data(); + batch.submit_and_wait(1, 5_000).expect("submit"); + + let completion = crate::ring::pop_within(&mut ring, "the registration's completion"); + assert_eq!(completion.user_data(), user_data); + + let Err(error) = pending + .claim_if(&completion) + .expect("its own completion is accepted") + else { + panic!("Windows accepted an empty buffer registration; is_empty is now reachable"); + }; + // Asserted through the crate's own downcast rather than against + // `Display`'s text: the point is which HRESULT the kernel returned, and + // matching a formatted string would also fail if the message were reworded, + // reporting a documentation change as a platform change. + let code = crate::IoRingErrorExt::as_ioring_error(&error) + .expect("a kernel refusal carries an HRESULT") + .code(); + assert_eq!( + code, + windows_sys::Win32::Foundation::E_INVALIDARG, + "the kernel must refuse an empty registration with E_INVALIDARG, got {error}" + ); +} diff --git a/crates/windows-ioring-sys/src/error/tests.rs b/crates/windows-ioring-sys/src/error/tests.rs index 1d89b856e..7de6847a1 100644 --- a/crates/windows-ioring-sys/src/error/tests.rs +++ b/crates/windows-ioring-sys/src/error/tests.rs @@ -56,6 +56,21 @@ fn code_reports_the_raw_value() { assert_eq!(error.code(), IORING_E_SUBMISSION_QUEUE_FULL); } +#[test] +fn the_debug_rendering_names_the_type_and_the_code() { + // `::fmt -> Ok(Default::default())` survived: + // that mutation writes nothing to the formatter, so `format!("{error:?}")` + // comes back empty. The real rendering names the type and prints the code + // in hex, neither of which an empty string can satisfy. + let error = IoRingError::new(IORING_E_SUBMISSION_QUEUE_FULL); + let rendering = format!("{error:?}"); + assert!(rendering.contains("IoRingError"), "got {rendering}"); + assert!( + rendering.contains(&format!("{:08X}", IORING_E_SUBMISSION_QUEUE_FULL as u32)), + "the code must appear in hex: {rendering}" + ); +} + // --- named conditions and predicates (M10.5, D-30) --- #[test] diff --git a/crates/windows-ioring-sys/src/event_delivery/tests.rs b/crates/windows-ioring-sys/src/event_delivery/tests.rs index ccb09cc16..194b62b0b 100644 --- a/crates/windows-ioring-sys/src/event_delivery/tests.rs +++ b/crates/windows-ioring-sys/src/event_delivery/tests.rs @@ -59,6 +59,25 @@ fn a_scope_reports_the_rings_static_properties() { assert_eq!(scope.info().expect("query info").submission_queue_size, 8); } +#[test] +fn a_scope_reflects_a_ring_that_genuinely_lacks_support() { + // `RingScope::supports -> true` survived: the test above only ever asks + // about an op the host genuinely supports, so the honest forwarder and + // the constant agree everywhere a real host could answer. Restricting the + // ring's capability set before wrapping it constructs the disagreement -- + // the same seam `Batch::require`'s own gap needed. + let mut ring = IoRing::new(8, 8).expect("create ring"); + ring.set_supported_ops_for_test(&[Op::Nop]); + let delivery = EventDelivery::new(ring, |_completion| {}, None).expect("wire event delivery"); + + let scope = delivery.scope(); + assert!(scope.supports(Op::Nop)); + assert!( + !scope.supports(Op::Read), + "Read was left out of the constructed capability set" + ); +} + #[test] fn a_scope_reports_registration_counts_that_change_with_registrations() { let ring = IoRing::new(8, 8).expect("create ring"); diff --git a/crates/windows-ioring-sys/src/ring.rs b/crates/windows-ioring-sys/src/ring.rs index de6868d6f..a69802227 100644 --- a/crates/windows-ioring-sys/src/ring.rs +++ b/crates/windows-ioring-sys/src/ring.rs @@ -184,6 +184,15 @@ impl InjectedFailure { Self::Ring(condition) => condition.code(), // `HRESULT_FROM_WIN32`: severity 1, facility 7 (`FACILITY_WIN32`), // and the low 16 bits of the code. + // + // The `|` here is provably equivalent to `^`, and a mutation run + // will report that mutant surviving: `0x8007_0000`'s low sixteen + // bits are zero and `code & 0xFFFF`'s high sixteen bits are zero, + // so the two operands never share a set bit and every bitwise + // combinator that agrees on disjoint inputs agrees here too. `|` + // is kept because it reads as "these are separate fields" where + // `^` would read as an arithmetic accident; no test can tell them + // apart, so none is written. Self::Win32(code) => (0x8007_0000_u32 | (code & 0xFFFF)).cast_signed(), Self::Hresult(code) => code, }; @@ -547,6 +556,31 @@ impl IoRing { self.supported_ops.contains(op) } + /// Overrides the cached capability set to exactly `ops`, for tests that + /// need a ring known to lack support for something. + /// + /// Every real host this crate has been tested against supports all seven + /// named ops, which is exactly why [`IoRing::supports`] and + /// [`Batch::require`](crate::Batch)'s use of it could not be told apart + /// from a constant `true` by any test that only ever asked a real ring: + /// the honest answer and the constant agree on every host available to + /// run the test. This seam constructs the disagreement instead of hoping + /// to find a host that has it. + /// + /// Not available outside `#[cfg(test)]`, for the same reason + /// [`Completion::synthetic`] is not: production code has no legitimate + /// reason to claim a capability the kernel did not actually report. + #[cfg(test)] + pub(crate) fn set_supported_ops_for_test(&mut self, ops: &[Op]) { + self.supported_ops = OpSupport(ops.iter().fold(0_u8, |mask, &op| { + let index = Op::ALL + .iter() + .position(|&candidate| candidate == op) + .expect("Op::ALL is exhaustive"); + mask | (1 << index) + })); + } + /// An owned duplicate of this ring's completion event, so a caller can /// wait on the ring alongside other handles without surrendering it /// (M11.1, D-20). @@ -955,6 +989,15 @@ impl IoRing { impl Drop for IoRing { fn drop(&mut self) { + // A count of how many times this body has run, so a test can confirm + // the rundown-and-close actually executes rather than trusting the + // impl exists. The counter is thread-local: a process-wide one is + // incremented by every other test's rings as they drop, which would + // let an `after > before` assertion be satisfied by somebody else's + // drop and mask the very mutation it exists to catch. + #[cfg(test)] + DROP_RUNS.with(|runs| runs.set(runs.get() + 1)); + // Best-effort rundown: a ring with an operation still outstanding at // drop time is a use bug (M3's Batch/Token are the sanctioned way to // avoid it), but Drop cannot propagate the error, so this asserts in @@ -971,5 +1014,58 @@ impl Drop for IoRing { } } +// How many times `IoRing`'s `Drop` impl has run **on the calling thread**; see +// its use there. +// +// A plain comment rather than a doc comment: rustdoc does not document items +// produced by a macro invocation, so a doc comment here is an +// `unused_doc_comments` warning, which CI escalates with `-D warnings`. +// +// Thread-local rather than a shared `static`, and that is load-bearing rather +// than tidiness. `cargo test` runs tests as threads in one process, so a +// process-wide counter is incremented by every other test's rings as they +// drop -- and an assertion of the form `after > before` is then satisfied by +// *somebody else's* drop, which is exactly the mutant it was written to catch. +// A thread-local is only touched by rings dropped on this test's own thread. +#[cfg(test)] +thread_local! { + pub(crate) static DROP_RUNS: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +/// Pop one completion, bounded, for tests that need a real one. +/// +/// `Batch::submit_and_wait` returning does **not** mean a completion is +/// poppable -- its own documentation says so, because the timeout can expire +/// first. That leaves two ways for a test to be wrong, and this exists so +/// neither is spelled out at each call site. +/// +/// A single `try_pop` flakes: under load the completion arrives just after the +/// check. A bare `loop` around `try_pop` is worse, because it converts that +/// flake into a hang -- and `cargo test` runs tests as threads in one process, +/// so a hung test stops the *whole harness* reporting and the failure arrives +/// with no test name attached. A bounded wait fails loudly instead, naming what +/// it waited for. +/// +/// Thirty seconds matches the deadline the crate's own `failure_paths` +/// integration test already uses; it is a hang bound, not a latency +/// expectation, so it is far above any real completion time. +#[cfg(test)] +pub(crate) fn pop_within(ring: &mut IoRing, what: &str) -> Completion { + // Named once so the bound and the message it reports cannot drift apart. + const BOUND: std::time::Duration = std::time::Duration::from_secs(30); + + let deadline = std::time::Instant::now() + BOUND; + loop { + if let Some(completion) = ring.try_pop().expect("pop") { + return completion; + } + assert!( + std::time::Instant::now() < deadline, + "timed out after {BOUND:?} waiting for {what}" + ); + std::thread::yield_now(); + } +} + #[cfg(test)] mod tests; diff --git a/crates/windows-ioring-sys/src/ring/tests.rs b/crates/windows-ioring-sys/src/ring/tests.rs index a22b5e1b2..06374f71c 100644 --- a/crates/windows-ioring-sys/src/ring/tests.rs +++ b/crates/windows-ioring-sys/src/ring/tests.rs @@ -72,6 +72,25 @@ fn capability_reporting_never_claims_more_than_is_io_ring_op_supported_reports() } } +#[test] +fn supports_reports_exactly_the_capability_set_it_was_given() { + // `IoRing::supports -> true` survived: every op named in this crate is + // genuinely supported on any real host these tests run on, so the honest + // answer and the constant agree everywhere a test could ask a real ring. + // `set_supported_ops_for_test` constructs the disagreement instead of + // hoping to find a host that lacks something. + let mut ring = IoRing::new(8, 8).expect("create ring"); + ring.set_supported_ops_for_test(&[Op::Nop, Op::Read]); + + assert!(ring.supports(Op::Nop)); + assert!(ring.supports(Op::Read)); + assert!( + !ring.supports(Op::Write), + "an op left out of the constructed set must read back as unsupported" + ); + assert!(!ring.supports(Op::Cancel)); +} + #[test] fn nop_read_and_write_are_supported_on_any_real_ring() { // A sanity floor: every documented IoRing version supports at least @@ -138,6 +157,33 @@ fn dropping_a_ring_with_nothing_outstanding_does_not_hang() { drop(ring); } +#[test] +fn dropping_a_ring_actually_runs_its_drop_body() { + // `::drop -> ()` survived: nothing distinguished a + // ring that ran rundown-and-close from one that silently leaked its + // kernel handle, because closing it is invisible to every test that only + // asks the ring itself. `DROP_RUNS` is incremented as the first line of + // the real body, so a mutation that replaces the whole body removes the + // increment along with everything else. + // + // The counter is thread-local, and an EXACT count is asserted. An earlier + // version used a process-wide static and asserted `after > before`, which + // is not race-free: `cargo test` runs tests as threads in one process, so + // another test dropping any ring between the two reads satisfies the + // assertion on its own -- masking precisely the mutant this exists to + // catch. Only rings dropped on this thread can move a thread-local, and + // this test drops exactly one. + let before = super::DROP_RUNS.with(std::cell::Cell::get); + let ring = IoRing::new(8, 8).expect("create ring"); + drop(ring); + let after = super::DROP_RUNS.with(std::cell::Cell::get); + assert_eq!( + after, + before + 1, + "dropping one ring must run its Drop impl exactly once (before={before}, after={after})" + ); +} + // --- The fault-injection seam (M16.3) --- /// A real completion for a real, finished operation. @@ -174,11 +220,7 @@ fn real_completion(ring: &mut IoRing) -> (usize, crate::Completion) { .expect("queue a flush"); batch.submit_and_wait(1, 30_000).expect("submit and wait"); - let completion = loop { - if let Some(completion) = ring.try_pop().expect("pop") { - break completion; - } - }; + let completion = super::pop_within(ring, "the flush's completion"); let _ = std::fs::remove_file(&path); (user_data, completion) } @@ -239,11 +281,7 @@ fn an_injected_failure_preserves_the_identity_a_token_claims_against() { unsafe { batch.read_raw(file.as_raw_handle(), vec![0_u8; 5], 0, PushOptions::new()) } .expect("queue a read"); batch.submit_and_wait(1, 30_000).expect("submit and wait"); - let completion = loop { - if let Some(completion) = ring.try_pop().expect("pop") { - break completion; - } - }; + let completion = super::pop_within(&mut ring, "the read's completion"); completion .result() .expect("the read really did succeed, or this test proves nothing"); @@ -265,13 +303,53 @@ fn an_injected_failure_preserves_the_identity_a_token_claims_against() { let _ = std::fs::remove_file(&path); } -// Deliberately not tested: that injection zeroes the transferred byte count. -// `information` is private and `result()` yields `Err` for an injected -// failure, so the zeroing is unobservable through the public API -- there is -// no assertion to write. It is still done, because modelling a state the -// kernel never produces would be wrong even where nothing can see it, but a -// test asserting only `is_err()` under that name would be coverage in -// appearance and nothing in substance. +#[test] +fn an_injected_failure_zeroes_the_transferred_byte_count() { + // The deletion of `information: 0,` from the struct-update survived: with + // it gone, `..self` supplies the *original* transfer count, so an + // injected "failure" completion silently keeps reporting real bytes + // transferred. `Completion::result` cannot show this -- it only returns + // `information` on success, and this seam only injects failure -- so the + // field is read directly. This module is `ring.rs`'s own child and can + // see it, which is exactly what an earlier version of this file's comment + // (just above) said was impossible. + use crate::{Batch, PushOptions}; + use std::os::windows::io::AsRawHandle; + + let path = std::env::temp_dir().join(format!( + "windows-ioring-sys-injection-information-{}-{:?}.tmp", + std::process::id(), + std::thread::current().id() + )); + std::fs::write(&path, b"hello").expect("create fixture"); + let file = std::fs::OpenOptions::new() + .read(true) + .open(&path) + .expect("open fixture"); + + let mut ring = IoRing::new(16, 16).expect("create ring"); + let mut batch = Batch::new(&mut ring); + // SAFETY: `file` outlives the operation, and the completion is popped + // below before it is dropped. + let _token = + unsafe { batch.read_raw(file.as_raw_handle(), vec![0_u8; 5], 0, PushOptions::new()) } + .expect("queue a read"); + batch.submit_and_wait(1, 30_000).expect("submit and wait"); + let completion = super::pop_within(&mut ring, "the fixture read's completion"); + assert_eq!( + completion.information, 5, + "the fixture must transfer five real bytes, or this test proves nothing" + ); + + let injected = completion + .with_injected_failure(crate::InjectedFailure::Ring(crate::RingCondition::Corrupt)); + assert_eq!( + injected.information, 0, + "an injected failure must report zero transferred, not the real completion's count" + ); + + let _ = std::fs::remove_file(&path); +} #[test] fn each_spelling_of_a_failure_produces_the_condition_it_names() { @@ -451,3 +529,18 @@ fn an_injected_failure_carries_the_condition_it_names() { assert_eq!(win32.user_data(), base.user_data()); assert_eq!(win32.ring_id(), base.ring_id()); } + +#[test] +fn the_debug_rendering_names_the_ring_and_its_key_fields() { + // `::fmt -> Ok(Default::default())` survived: that + // mutation writes nothing to the formatter at all, so `format!("{ring:?}")` + // comes back empty. Asserting the type name and a real field value is + // enough to tell "wrote nothing" from "wrote the real struct". + let ring = IoRing::new(8, 8).expect("create ring"); + let rendering = format!("{ring:?}"); + assert!(rendering.contains("IoRing"), "got {rendering}"); + assert!( + rendering.contains("version"), + "the version field name must appear: {rendering}" + ); +} diff --git a/design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md b/design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md new file mode 100644 index 000000000..621162af6 --- /dev/null +++ b/design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md @@ -0,0 +1,1202 @@ +# Design session -- NUMA-sharded I/O execution domains (2026-08-30) + +> Tier-3 record. [DESIGN-NOTES.md](../DESIGN-NOTES.md) is authoritative and wins +> on any conflict. This file records how the discussion went, what was measured, +> and what is still open. +> +> **API names below are as of 2026-08-30 and some have since changed.** The text +> is left as it was said, because a session record that is edited to match later +> code stops being a record of the session; the current name is given in brackets +> at each site instead. Renames so far: `Topology` -> `MachineMemoryTopology` +> (2026-09-02), and `Core`, which no longer exists as a type -- a processor's +> `efficiency_class` is now a `ProcessorFacts` field of type `Observed`, +> so what this session calls an input that "exists" is now explicitly +> maybe-absent-with-a-reason. + +**Status: OPEN. This session has only just begun.** What follows is the opening +survey and the first measurement, not a set of converged decisions. No decision +below is settled unless it says so explicitly. + +Repo-wide by scope: it touches +[windows-ioring-sys](../crates/windows-ioring-sys/DESIGN-NOTES.md), +[windows-threadpool-sys](../crates/windows-threadpool-sys/README.md) (which has no +DESIGN-NOTES.md of its own), +[windows-topology-sys](../crates/windows-topology-sys/DESIGN-NOTES.md), and the +deferred namespace facility designed in +[DESIGN-SESSION-2026-08-27-pseudo-async-namespace-operations.md](DESIGN-SESSION-2026-08-27-pseudo-async-namespace-operations.md). + +## Starting intent + +The engineer's framing: a high-performance flow of deferred `CreateFile` -> +offloaded `CreateFile` -> `IoRing` (or several), with NUMA-affined buffers +backing sector-aligned I/O, organized as Seastar-style shards over queues. + +Stated afterwards, and worth recording because it shapes how the material below +should be read: the topic was seeded to see what would grow from prior work +planted around the repository, rather than started from nothing. + +## What already exists, so it is not redesigned + +- **The namespace/data plane split is decided.** Win32 is asynchronous on the + data plane (overlapped I/O, `IoRing`) and synchronous-only on the namespace + plane (open, delete, rename, attributes). "Deferred `CreateFile` feeding an + `IoRing`" is precisely a namespace-plane operation handing to a data-plane one. +- **The Win32 ring is not the `IoRing`** (that session's decision 7): share the + ring type, not the storage, and unify at the wait. +- **The namespace facility owns no threads**: pooled, elastic, `runs_long` + mandatory, quarantine ceiling. +- **Model B is already specified** in the ioring crate's + [DESIGN-NOTES.md](../crates/windows-ioring-sys/DESIGN-NOTES.md), naming the unit + as an execution domain -- one pinned thread, its ring, its node-local + registered buffer pool, its shard of the work -- with + [D-27](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-27) arguing pinning is + what makes per-thread a proxy for per-CPU. +- **`ring_copy` already implements it end to end**: `SetThreadGroupAffinity` per + domain, `VirtualAllocExNuma` buffers, one ring per domain, policy selectable + as `ByL3` / `ByNode` / `ByPackage` / `ByCore` / `Single`. +- **[D-8](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-8) reserves the + abstraction** being discussed here: a `RingFleet`-style layer, deferred until + "there is evidence about what sharding actually helps." + +## Constraints established during the session + +- **Registration is one-shot per ring, for both buffers and file handles.** + Verified in `Batch::register_buffers` and `Batch::register_files`: a second + call is refused because `BuildIoRingRegister*` **replaces the whole table**, + invalidating every index already handed out. A freshly opened handle therefore + **cannot be added to a running ring's file table**. Any "deferred `CreateFile` + -> ring" pipeline that expects to register each new file hits this on + operation two. +- **Ring count is forced to equal pinned-thread count**, because the submission + queue is not thread-safe. Choosing a coarse policy does not give many threads + sharing few rings; it gives *few threads*. +- **Processor groups are a hard floor**: above 64 logical processors the + partition is forced whether wanted or not. +- **No library in this workspace can pin a thread.** `windows-topology-sys` is + read-only and reports `ProcessorSet` without applying it; + `windows-threadpool-sys` has no affinity, NUMA, or pinning support at all. The + only pinning in the repository is inline in the `ring_copy` example. +- **No SPSC or MPSC queue exists in this workspace.** There is a domain-specific + bounded queue in `windows-file-watcher` and `std::sync::mpsc` in a probe. + Nothing general, and no `crossbeam` dependency. + +## Crossbeam: assessed, and it does not provide the doorbell + +The engineer asked specifically about crossbeam's doorbell mechanism and its +buffer management. Checked against the published API rather than reputation: + +- **`crossbeam-queue`** (`ArrayQueue`, `SegQueue`) is lock-free MPMC. `pop` + returns `Option`; it never blocks and never signals. There is nothing to wait + on. +- **`crossbeam-channel`** blocks in `recv`, but parks on its own internal + primitive and exposes **no waitable HANDLE**. Its `Select` is built purely + from channel operations (`sel.recv`, `sel.send`); there is no method to + register a foreign OS object. + +So the mismatch is symmetric and fatal for a Model B shard: +`WaitForMultipleObjects` cannot see a crossbeam channel, and crossbeam's +`Select` cannot see the `IoRing` completion event. A shard that must park on +"my ring completed something **or** a peer sent me work" cannot express that +wait with crossbeam, and would have to poll one while blocking on the other. + +**This workspace already solved the doorbell problem** in +[queue.rs](../crates/windows-file-watcher/src/queue.rs), whose module +documentation states the general principle: on Windows a HANDLE **is** the +universal waitable currency, so an event is the native composition point rather +than a lowest common denominator. It hands out a lazily created manual-reset +event, signalled under the same lock a receiver holds while deciding there is +nothing to take, "so a wakeup cannot be lost in the gap between those two +decisions, because there is no gap" -- the same lost-wakeup hazard class as +[D-19](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-19)'s edge-triggered ring +contract. + +**Buffer management, two senses, and conflating them is a trap.** `ArrayQueue` +allocates a fixed buffer at construction and fails `push` when full -- that +failure is the backpressure signal. `SegQueue` is unbounded, allocates segments +on demand, and needs deferred reclamation, which couples shards that were +supposed to share nothing. But either way crossbeam manages **message** storage, +never **I/O buffer** storage: the I/O buffers are the registered pool, allocated +once, NUMA-affined, sector-aligned, one-shot per ring. A cross-shard queue must +carry descriptors (buffer index, handle, completion record), never bytes -- if it +carried bytes, the copy would have defeated the reason for registering. + +*Not yet decided:* whether to adopt `crossbeam-queue` for the data structure and +add a doorbell beside it, or build the queue with the doorbell integral. The +argument for integral is the file-watcher's: an external doorbell cannot be +signalled under the queue's own lock, which reintroduces the gap. + +## What "shard" means here + +Asked directly, and answered: the shard is **the execution domain -- one pinned +thread**, with its ring, its node-local registered pool, and its slice of state. +It is **not** the last-level cache domain. L3 is one *policy* for deciding how +many shards and where to pin them. + +A consequence that is easy to miss: because ring count equals thread count, +selecting `ByL3` on a 64-core, 8-CCX part does not yield 64 threads over 8 rings. +It yields **8 pinned threads in total**. That is correct for `ring_copy`, a +bandwidth-bound copy where a few threads saturate memory, and probably wrong for +a general execution substrate, where Seastar shards per logical core and uses +NUMA only to place each shard's memory. The repository currently has one answer +where the effort may need two. + +## Measurement M-1: a shipping ARM laptop reports no L3 at all + +Probed with `Topology::discover()` [now `MachineMemoryTopology::discover()`] on +the development machine. + +**Snapdragon(R) X2 Elite - X2E80100 - Qualcomm Oryon(TM) CPU**, 12 cores, 12 +logical processors, no SMT: + +``` +processor groups : 1 {0: 12} +domains by kind : Cache 26, Core 12, Group 1, Memory(NUMA) 1, Module 2, Package 1 +cache by level : L1 -> 24 domains (1 processor each) + L2 -> 2 domains (6 processors each) + L3 -> none +pinned-thread count each ring_copy policy would produce: + ByCore 12 | ByL3 0 | ByNode 1 | ByPackage 1 | Single 1 +``` + +Corroborated by WMI: `L3CacheSize = 0`, and **zero `Win32_NumaNode` instances** -- +the same signature the ioring notes already recorded for the machine they were +investigated on. + +**What this does and does not show.** An initial reading of this session claimed +`ByL3` returning zero domains would create zero rings. **That was wrong, and +checking the code corrected it**: `Policy::select` falls through to a +whole-machine domain when the preferred relation matches nothing, and reports +`degraded = true` so the sample can say so honestly. There is no defect in +`ring_copy`. + +What the measurement does falsify is narrower and is in the prose. The ioring +notes justify the L3 heuristic partly by saying it "is meaningful on Intel and +ARM too, where the NUMA node often is not." On this ARM part L3 is **not** +meaningful, because it does not exist; the natural cluster boundary is **L2**, +two domains of six, corroborated by the two `Module` domains the same probe +reported. The heuristic is still right that L3 beats the NUMA node. The durable +idea underneath it appears to be "the outermost cache level that actually +partitions the machine," not "L3 specifically" -- and on this machine the +difference is between describing two clusters and describing one whole machine. + +Whether the two-cluster structure is worth using is a separate question this +session has not answered. + +## Finding F-1: a file handle's NUMA node is reachable, and answers a coarser question + +Contributed by the engineer as research, **not measured here**. It corrects the +ioring notes on mechanism while leaving their conclusion standing for a better +reason. + +- **`FSCTL_QUERY_VOLUME_NUMA_INFO` is documented and takes a file or directory + handle directly**, returning `FSCTL_QUERY_VOLUME_NUMA_INFO_OUTPUT { ULONG + NumaNode }`. Confirmed independently against the IFS documentation during the + session. The ioring notes say this mapping "has no clean user-mode path" and + "means walking volume to disk to device instance and reading + `DEVPKEY_Device_Numa_Node`". **That is wrong**: there is one documented call, + and it accepts the handle a caller already has. +- **What it returns is the node the *volume* resides on**, not where the file's + extents live. NTFS does not expose per-file or per-extent NUMA through this + API, and nothing states that `FileNumaNodeInformation` is filled from MFT or + runlist locality. +- **`GetNumaNodeNumberFromHandle` is the other path**: a Win32 wrapper over + `NtQueryInformationFile` with `FileNumaNodeInformation` (class 53, Windows 7 + and later), yielding `FILE_NUMA_NODE_INFORMATION { USHORT NodeNumber }`. PHNT + and the WDK mark that class **reserved for system use**. Documented Win32 + behaviour when there is no node is `FALSE` with an undefined `NodeNumber`. + This crate must not build on it. +- **The volume node exists only when the device layer advertised one**: + `IoGetDeviceNumaNode` on the PDO, or user-mode + `DEVPKEY_Numa_Proximity_Domain` with `GetNumaProximityNode`. A single-node + machine, a PDO returning `STATUS_NOT_FOUND`, or a software or virtual disk + with no proximity data is precisely the "no association" case. +- **No published experiment could be found** showing either call succeeding on a + garden-variety NTFS data file and naming a node. There is also no evidence + that success depends on `FILE_FLAG_NO_BUFFERING`, on overlapped I/O, or on + which process opened the file. + +### F-1a: one weak datapoint from the vacuous machine + +The spike was smoke-run on the development machine, not for an answer but to +prove the instrument works before handing it to someone with real hardware. It +was worth doing twice over. + +**It found a defect in itself.** The first version opened the directory for Q5 +with `File::open`, which fails on a directory without +`FILE_FLAG_BACKUP_SEMANTICS`, so Q5 could never have been answered. Corrected to +`CreateFileW`. An instrument checked in unrun is one whose bugs are still in it; +running it on hardware where the *result* is vacuous still validates the +*apparatus*. + +**And it does establish one thing, narrowly.** On ARM64 Windows, single node: + +``` +regular NTFS data file : FSCTL ok, NumaNode = 0 | GetNumaNodeNumberFromHandle ok, NodeNumber = 0 +directory handle : FSCTL ok, NumaNode = 0 | GetNumaNodeNumberFromHandle ok, NodeNumber = 0 +``` + +Both calls **succeed** on a garden-variety NTFS data file, and agree. That is +directly responsive to "no published experiment shows +`GetNumaNodeNumberFromHandle` succeeding on a garden-variety NTFS data file": +here it does. It also shows the FSCTL accepting a directory handle, as the IFS +docs say. + +**What it does not establish**, and the distinction is the whole value of the +result: node `0` is the *only* node this machine has, so neither call is shown +to name a *meaningful* node. What is refined is the negative case -- the +documented "returns FALSE when the object has no node" did **not** occur here, +so "ordinary NTFS file" is not itself the absent case. Absence must come from +the device layer advertising no proximity domain, which is exactly what cannot +be reproduced on this hardware. + +**Why this matters to the seam, and it is an opportunity rather than a +problem.** The discriminating check is to call both on the same handle: if they +agree, what is being observed is volume locality. And volume locality, though +coarse, arrives at exactly the right moment -- the namespace worker has the +handle in hand at the instant it completes the open, so a routing key for +"which domain should own this file's I/O" is available **for free at the seam**, +with no extra open and no device-tree walk. That does not make automatic +placement correct, and the conclusion below stands, but it does mean the +information is cheaper than the notes imply. + +The conclusion the notes draw survives, restated: the crate should still not +offer "put this file's I/O on the right ring," because the answer is +volume-granular, frequently absent, and meaningless for spanned volumes and +Storage Spaces where one volume sits on several devices. It also still does not +pin thread-pool completions. + +**Named blocker, per the repository's deferral protocol.** Settling this +empirically needs a multi-node machine with storage whose PDO advertises a +proximity domain. The development machine has one node and reports zero +`Win32_NumaNode` instances, so any run here is vacuous: failure would prove +nothing and success could only report `0`. The instrument is therefore checked +in unrun as +[file-handle-numa-spike.rs](../crates/windows-ioring-sys/design-sessions/spikes/file-handle-numa-spike.rs), +with the hardware gap stated in the spikes +[README.md](../crates/windows-ioring-sys/design-sessions/spikes/README.md). The +documentation defect is independent of the measurement and is queued as M20.4 +regardless. + +## Converged: one uniform, tunable architecture, sized by the topology + +This is the session's first converged position, and it arrived by correcting a +framing this record had already adopted. + +**The framing that was wrong.** An earlier turn concluded that because almost +every consumer machine yields a single domain, "this entire apparatus is +server-class-only, and should be sized and justified as such." The engineer +rejected that, on the grounds that it sounds like the feature does not work on +laptops, and that it should extend up and down smoothly instead. + +That objection is right, and it is right against a **standing rule of this +repository** rather than merely on taste. PLATFORM INTEGRITY says: "do not +narrow the platform to serve the visible goal -- every platform component must +remain a *level* platform, its lower baselines first-class, not optional +trimmings to cut because the current task does not need them." Scoping the work +to server hardware is exactly that narrowing. The ioring notes already had the +correct words where this session lost them: a machine that yields one ring is +"correct", not degraded. + +**The architecture.** There is one shape at every size -- a domain is a pinned +thread, its ring, its node-local registered pool, and its shard of the work. +A laptop runs one. A server runs several. There is no laptop mode and no server +mode; there is one shape and a count. N=1 is not a fallback that lost something: +on a single-node, single-LLC machine one domain **is** the optimal partition. + +**What is genuinely additive above one domain**, and it is additive rather than +a second mode: + +- the **cross-domain** queue and its doorbell -- with one domain there is no peer + to message. **This is not the client-facing queue**, and the distinction + matters enough to state plainly, because reading "the queue" as "all queues" + would wrongly suggest no queue work is needed for the first deliverable. + There are two roles: + - **client to domain (the SQ), and domain to client (the CQ)** -- needed at + **every** size including N=1, because a foreign client thread must still + reach the single domain. This is the two-layer ring, and it is on the N=1 + critical path. + - **domain to domain** -- genuinely absent below two domains. +- the routing policy -- with one domain there is no choice to make, so the + volume-to-node key has nothing to select. + +Both are **absent** at N=1, not stubbed or bypassed. Nothing on the +single-domain path consults a router that always answers zero. + +**Consequence for build order, and it inverts what this session had implied.** +N=1 is the **first deliverable and the substrate**, not the leftover. It is the +common case, it is complete and correct on its own, and N>1 extends it without +disturbing it. That is a better sequencing argument than starting from the +fleet, and it means the first thing built is useful on every machine in the +table below rather than on none of them. + +**The mechanism that makes "smooth" concrete: affinity is a set, not a point.** +`SetThreadGroupAffinity` takes a mask, and `windows-topology-sys` already hands +out `ProcessorSet` with correct multi-group handling. A domain's affinity is +simply the `ProcessorSet` of its partition: + +| Machine | N | Each domain's affinity set | +|---|---|---| +| Uniform laptop or VM | 1 | the whole machine | +| Heterogeneous laptop (P/E, or ARM clusters) | 1 | the performance cluster only | +| 2-CCD desktop | 1-2 | each CCD's processors | +| 12-CCD EPYC | 4-12 | each CCD's processors | + +Same call, same type, different set. Nothing special-cases the small end. + +**Domain count and pinning tightness are separate knobs.** Pinning to a single +core buys locality *relative to other domains*; with one domain there is nothing +to be local relative to, and hard-pinning an I/O thread to one core of a laptop +that is also running everything else may be worse than letting it float across +the performance cores. But **heterogeneity means even N=1 wants an affinity +mask**, because an unconstrained thread can be scheduled onto an efficiency core +or an LPE island. The development machine is the case in point: two clusters of +six, and `efficiency_class` is already exposed on `Core` [now the +`ProcessorFacts::efficiency_class` field, an `Observed`] by the topology +crate. +So the small end does not want *no* affinity -- it wants a *set*, which is +exactly what the large end wants too. + +Stated once: **one mechanism, sized by the topology.** A domain is affinitized +to a `ProcessorSet`; how many domains exist, and how wide each set is, falls out +of the machine. + +## Converged: round-robin is incoherent with the model, not merely suboptimal + +The engineer's position was that round-robin assignment "seems actually +dangerous" and that a high-performance consumer might prefer a single thread on +a single processor. Agreed, and the reason is stronger than the averaging +argument that first suggested it. + +Round-robin across domains **breaks the ownership premise Model B exists for**. +If a file's I/O lands on domain 1 now and domain 3 next, that file's state -- +buffer slots, outstanding accounting, continuations -- is owned by no single +shard. That is sharing on the data path, which is the one thing shared-nothing +is defined by. It is not a worse point on the same curve; it is off the curve. + +The consequence for a latency-sensitive consumer follows directly: a single +thread on a single processor is deterministic and owned, where round-robin is +non-deterministic and shared. A known cost can be engineered around; jitter +cannot. + +**Position:** round-robin is acceptable only as an explicitly chosen default for +consumers who have expressed no preference, and it should be named to admit what +it is rather than offered beside "by node" as though it were a peer policy. + +## Converged: report, do not route + +The engineer proposed that when no useful mapping is available, the facility +should message the caller and have them respond with how to proceed. + +**One mechanism correction:** an arbitrary completion cannot be posted into an +`IoRing` completion queue. The namespace session already rejected that path -- +"the `IoRing` API has no post/user-completion entry point, so a namespace +completion cannot be placed in its CQ." (`IORING_OP_NOP` can inject a marker, +but only the ring's owning thread may submit, so a namespace worker on another +thread cannot reach in.) On the facility's **own** ring, which it owns outright, +posting is available. + +**A simpler form needs no new mechanism at all.** Rather than the facility +asking a question and awaiting an answer, the open's completion carries the hint +and the client routes: + +``` +completion = { handle, volume_node: Option, provenance: Measured | Absent | Overridden } +``` + +The client already receives that completion. No round trip, no pending-decision +state, no "what if the client never answers," and no new queue direction. It is +the principle the ioring notes already state -- "leaves the mapping to whoever +knows their storage layout" -- made concrete: the facility reports, the client +routes. + +The upcall form remains the right answer for a higher layer that owns a flow +end to end and cannot hand control back. Both are kept, with the reporting form +as the primitive and the upcall as something built on it if a consumer needs it. + +## Converged: a domain runtime is not a thread pool + +The engineer raised a fear of ending up writing a thread pool, given that the +Windows pool cannot affinitize to any of the objects of interest. + +What Model B needs is a **domain runtime**, and every distinguishing feature is +the *absence* of a thread-pool feature: + +| A thread pool does | A domain runtime does | +|---|---| +| dynamic sizing, injection, retirement | spawn N at startup, join at shutdown | +| work stealing and load balancing | nothing -- stealing would violate share-nothing | +| queue management, priorities, quarantine | one loop per thread: park, drain, run | +| grow under blocking (`runs_long`) | never blocks on namespace work by construction | + +`ring_copy` already contains a working instance of it inline: pin, allocate +node-local, register, loop. The ioring notes already committed to the shape -- +"Model B on the hot data path ... and Model A for the control plane, background, +and cold paths" -- and `M6+.4` already queues binding a ring's thread with +`SetThreadGroupAffinity`. + +The load-bearing point is the one that motivated the fear: the Windows pool +**cannot** affinitize, so this capability cannot come from it. That is not a +reason to rebuild the pool; it is the reason the two halves are separate. The +namespace plane keeps the pool because it needs quarantine and elasticity for +blocking calls; the data plane owns threads because it needs pinning. Neither +can serve the other, and that is the design rather than a compromise. + +## The target architecture: a two-layer ring + +The engineer's refined vision: an SQ/CQ against an "async API substrate". You +post a deferred `CreateFileW`; the CQ may return a request to clarify NUMA +placement (behind an option set on the SQ, so the complexity is opt-in); you +answer; you get back a token for high-performance I/O using the epoch/durability +metaphor. All ring-based, NUMA-aware, "without a lot of exotic client +programming". + +**The structural resolution is that the client-facing ring is ours, and the +`IoRing` lives inside a domain as an implementation detail.** + +``` +client thread --post--> [ our SQ: MPSC + doorbell ] --> domain thread --> IoRing SQ +client thread <-drain-- [ our CQ: MPSC + doorbell ] <-- domain thread <-- IoRing CQ +``` + +Four problems collapse into that one decision, which is the main reason to +believe the decomposition is right: + +- **Submission thread-safety.** Only the domain thread touches the real SQ. This + is a *third* answer to `M6+.2`, which is parked with "needs either a + submit-ownership handoff or an internal lock. Neither is obviously right" -- + the answer is neither: a queue only the domain drains. +- **The missing post entry point.** The namespace session rejected posting a + completion into an `IoRing` CQ on mechanism. Our CQ is ours, so the + clarification CQE the vision needs becomes possible. +- **One ring or two.** The client sees one; the namespace and data planes keep + separate storage. That is decision 7 -- "share the ring type, not the storage; + unify at the wait" -- seen from the client's side. +- **The epoch metaphor ports**, because it is already written against a CQ. + +### The client never allocates an I/O buffer + +This, rather than the ring shape, is what removes the "exotic client +programming". The facility owns the registered pool: `VirtualAllocExNuma` on the +domain's node, sector-aligned, registered once. The client acquires a **slot**, +writes into it, submits, and gets it back. + +The client cannot place a buffer wrongly because it never chooses. Every +alternative -- client allocates and we validate, client hints and we advise -- +returns the decision to where the knowledge is not. It also fits the one-shot +registration constraint exactly: the pool is the long-lived fixed thing, so +registration's principal limitation stops being one. + +### Three policy tiers, generalized beyond placement + +| Tier | Client says | Facility does | +|---|---|---| +| Default | nothing | picks; at N=1 there is no choice, so this is free | +| Informed | "tell me" | completion carries the hint and its provenance; client routes | +| Consulted | "ask me" (SQ option) | posts a clarification CQE; client answers | + +The engineer generalized this past NUMA: most things "need to just be taken care +of for people", but policy in general must be expressible either as **optional +parameters** or as **queries via CQE/SQE pairs**. So the tiers are the shape for +every policy decision the facility must make, not a placement-specific device. + +## Costs of the target architecture, and their mitigations + +### C-1 The queue hop + +Every foreign-thread submission crosses an MPSC push plus a doorbell before +reaching the ring, where a run-to-completion client would have none. + +- **The push is cheap; the doorbell is the syscall.** Signal only on the + empty-to-non-empty edge, and only when the consumer is parked. A queue that + stays non-empty needs no signal at all, so a busy domain -- the case that + matters -- pays approximately zero doorbells. +- **A brief consumer spin before parking** removes the park/unpark round trip + under load. Spin duration is another knob sized by the topology: generous when + a domain owns a core exclusively, zero when it shares one with the rest of a + laptop. +- **The hop is where batching happens.** N submissions become N pushes, one + doorbell, one drain, and **one** `SubmitIoRing`. Under load it plausibly + reduces syscalls rather than adding them. +- **It should be pay-for-what-you-use**: a client whose continuation runs on the + domain thread submits directly, with no queue and no doorbell. +### C-1 measured: batching settles it, and the eventcount can wait + +Measured on the ARM64 development machine by +`probe-doorbell-cost`, now part of the platform-probes CI job so the numbers +accumulate across the runner fleet. + +| operation | ns/op | +|---|---| +| `atomic_fetch_add` | 7.2 | +| `set_event_already_signalled` | 81.2 | +| `set_reset_event` (one doorbell cycle) | 164.9 | +| `wait_zero_signalled` | 94.8 | +| `submit_io_ring_empty` | 79.2 | +| park and wake, round trip | 2196.4 | + +**The finding is that batching, not the skip rule, is the lever.** One doorbell +per drained batch costs 164.9 ns at a batch of one, 20.6 at eight, and 5.2 at +thirty-two -- so at a batch of about **23 the doorbell already costs less per +operation than the atomic push it accompanies**. A first implementation may +always-signal and remain honest. + +So **the eventcount is deferred, not adopted.** Publish-recheck-park is the +highest-risk protocol in this design, and nothing yet shows it is worth its +lost-wakeup risk. R3 and R4 are amended accordingly. + +**Two mistakes in the probe are recorded because they nearly produced findings:** + +- **It deadlocked.** The first park-and-wake had one thread calling `SetEvent` + in a loop against another in `WaitForSingleObject(INFINITE)`. An auto-reset + event does not count signals, so two arriving before one wait collapse into + one, the waiter's count never catches up, and it blocks for ever -- it hung + for over four hundred seconds before being killed. A probe that can hang is a + probe that can hang a build. It now uses a two-event ping-pong for strict + alternation, bounds every wait, and returns nothing at all on timeout rather + than averaging a partial run. +- **Its headline ratio had a denominator that is not a syscall.** The probe was + written to divide the doorbell cost by an empty `SubmitIoRing` and report + "the doorbell is N% of a syscall". It reported **210%**, which should have + been read as a broken denominator rather than a result: 79 ns is far too cheap + for a kernel transition, so an empty submit is almost certainly + short-circuiting in user mode. The verdict logic was deleted rather than + tuned. The honest denominator is the cost of the real work a submission + carries, which this probe does not measure -- so it reports absolute costs and + the batching arithmetic and forms no ratio. + +**Still unmeasured, and worth naming:** the doorbell's cost against a submission +carrying real I/O. That is the number that would justify adopting the +eventcount, and it needs a workload rather than a microbenchmark. + +### What these microbenchmarks do not establish + +Both C-1 and the request-cost measurement below are **per-operation overheads +of single, uncontended operations**. That is not the same thing as queue +efficiency, and the distinction is worth stating because both numbers invite the +same over-reading: + +- **Per-operation overhead is not throughput.** What makes a queue good or bad + is its behaviour under contention, the cache traffic of its ring, how well + batching amortises, and how it behaves at capacity. None of that is visible in + a single push or a single `SetEvent`. +- **One operation type is not the operation mix.** A namespace open is the + heaviest payload the queue carries; a registered-buffer read is the lightest, + and it is the hot path. A conclusion drawn from either says nothing about the + other. + +So neither measurement licenses a claim of the form "the queue's mechanics do +not matter". What C-1 supports is narrow and still useful: **batching alone +makes the doorbell cheap enough that the eventcount need not be bought up +front.** + +### The request's cost, and what it says about operation mix + +Measured on the same machine by `probe-request-cost`: + +| operation | ns/op | x a doorbell | +|---|---|---| +| `prepare_short_path` | 534.7 | 3.2 | +| `prepare_long_path` | 1593.0 | 9.7 | +| `build_open_request` | 452.7 | 2.7 | +| `clone_prepared_units` | 95.3 | 0.6 | +| `capture_handle` | 282.5 | 1.7 | + +**The queue can carry an owned request, and the R7 contradiction dissolves.** +`OpenFile` is `Send` (verified by compile-time assertion), owns its +`PreparedPath`, and carries a `CapturedHandle` that is already a *duplicate* -- +so nothing borrowed crosses the hop and no lifetime outlives the submitting +thread. The design does not need the "write the path into a buffer slot" scheme +proposed earlier in this session; the namespace crate was built for exactly this +and already solves it. + +**R7 needs amending rather than satisfying.** "POD descriptors, no allocation on +push" cannot hold for a payload that owns a heap string. Split it: the queue's +*slot* stays fixed-size and POD (tag, correlation id, index), while the +*payload* is an owned request the client allocated before pushing. The property +that mattered survives -- no allocation inside the push, and no lifetime hazard. + +**Two costs that are easy to under-count:** + +- **`prepare` is a Win32 call, not an allocation.** It invokes + `GetFullPathNameW`, because the namespace session settled that the path is + resolved at submission -- the process CWD is mutable by any thread, so + resolving later would be racy. No allocator change removes that. Cloning + already-prepared units costs 95.3 ns, which *bounds* what an inline-storage or + recycling scheme could recover, and only for a caller that can reuse a + resolved path. +- **A handle must be duplicated**, at 282.5 ns -- a kernel transition, not a + memory copy. Raised by the engineer, and it is the part a "what does an SQE + hold" analysis focused on memory would miss entirely. + +**Scope:** this is a statement about **operation mix**, not about the queue. For +an open-heavy workload, doorbell tuning would be optimizing the small half of +the cost. It says nothing about the registered-buffer read path, where the +descriptor is a slot index and an offset and the queue's mechanics are the whole +per-operation cost. + +### C-1a Why the doorbell must be a HANDLE, and cannot be `WaitOnAddress` + +`WaitOnAddress` is plausibly cheaper in isolation. It is still unusable here, +and cost does not enter into it: `WaitOnAddress` waits on a **memory location**, +`WaitForMultipleObjects` waits on **kernel objects**, and no API combines them. +A domain that must wake on either "my ring completed" or "a peer sent me work" +waits on both at once, and the ring's completion event is a HANDLE. + +This is the same structural constraint reached from a different direction than +the crossbeam analysis above, which is good evidence it is a property of the +platform rather than of a library. + +The two ideas also interact decisively: **the skip-when-busy rule removes the +very cost `WaitOnAddress` would save.** They are alternatives, not complements, +and only one of them composes. + +A corollary worth stating plainly: a domain parked in the fused +`SubmitIoRing(wait_n)` cannot observe a queue at all. **A domain that accepts +foreign submissions must use [D-20](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-20)'s +multiplexed-wait row.** Accepting foreign work and using the lowest-overhead +park are mutually exclusive. + +### C-1b Which side of the lock the doorbell is touched on + +Asked directly by the engineer: why can the event not be signalled after the +lock is released, and does it have to represent the fullness of the queue? + +**It can be signalled outside the lock. Only the reset must be inside.** A late +`SetEvent` can at worst arrive after the consumer already drained that item and +parked, producing a spurious wakeup -- the consumer wakes, finds nothing, parks +again. A reset outside the lock is fatal: + +``` +Consumer: lock, drain to empty, unlock +Producer: lock, push(B), unlock, SetEvent +Consumer: ResetEvent <-- clears the signal for B +Consumer: park <-- lost wakeup; B is stranded +``` + +So the invariant is precisely: **the reset must be atomic with the observation +that there is nothing to take.** And yes -- the event represents the fullness of +the queue. It is *level* state, a function of the contents rather than a record +of edges, which is why it is manual-reset and why a redundant signal is free +while a stale reset is fatal. + +### C-2 The pending-clarification handle -- dissolved + +Deliver the handle **with** the question rather than holding it behind the +question. The client then owns it under ordinary rules; a client that never +answers has leaked its own resource, not stranded a facility-held object with no +owner and no deadline. + +Following that through: **at the primitive layer the consulted tier collapses +into the informed tier.** Ask what the facility would do with the answer. +Register the file into that domain's ring? Unavailable -- registration is +one-shot. Allocate slots on the right node? Those come from the domain when the +client asks it. There is no work the answer unlocks that the client cannot do by +submitting to the domain it chose. The round trip has no payload. + +The consulted tier survives where the facility performs I/O **on the client's +behalf** -- a higher-level "read this whole file" API that must choose placement +and has nobody to ask. That is a layer above the primitive. + +Whatever the facility still holds transiently needs a deadline and a disposal +path **allowed to block**, since closing a handle to a dead network path is +exactly the work this facility exists to keep off a caller's thread. + +### C-3 The durability model graduates to its own crate + +Recorded as the engineer's decision: the `epoch_log` sample becomes a canonical +layer, probably a separate crate. Durability groups are a natural capability +whose absence is surprising. + +It composes because the mechanism it needs was already measured: +[D-23](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-23) (an unflagged flush +does **not** cover preceding writes) and +[D-24](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-24) +(`DRAIN_PRECEDING_OPS` is a full ring-wide barrier spanning submissions). Group +commit is policy over that mechanism, which is a textbook reason for a separate +crate rather than a feature. + +**One constraint to carry from the start:** the barrier stops at the ring's +edge, so a durability epoch is **per-domain**. A client writing through two +domains and wanting one durability point needs two flushes and an explicit join. +The crate must represent that or refuse it; it must not quietly imply an epoch +spans domains. + +### C-4 The composed layer swallows the sharp edges + +The engineer's direction: primitives matter, but this is the "build layers that +compose them" phase, and the composed layer should not expose sharp edges. + +The mechanism is already recorded as the namespace session's decision 8 -- a +type-level traversal where each step offers only the legal next steps -- and +this repository already applies it (`RingScope` so no `&mut IoRing` escapes; +`get(&mut self)` so a hazard is a type error). The edges to swallow: + +- edge-triggered drain ([D-19](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-19)): + drain to empty every pass; a single `try_pop` deadlocks; +- buffer slot lifecycle: acquire, outstanding accounting, release only on an + observed completion; +- one-shot registration: fixed at construction, never named by the client; +- token claiming: `claim_if` matching both `user_data` and `ring_id`; +- flush barrier semantics (D-23); +- handle disposal, including the blocking close. + +**If the client can name any of these, the layer has leaked.** + +## Rejected: a `Ring` trait with a client-implemented `ring_doorbell()` + +Proposed during the session, and rejected -- but the first argument offered +against it was wrong and is corrected here rather than quietly dropped. + +**The withdrawn argument.** It was claimed that a client callback would run +under the queue lock and therefore risk deadlock. C-1b shows the signal side +need not be under the lock at all, so that argument does not hold. + +**The arguments that do hold:** + +1. **Set and reset are two halves of one invariant** and cannot be split across + an ownership boundary. The reset must happen under our lock, atomic with the + emptiness observation; if the client owns the signalling object, we cannot + perform it. This is the file-watcher's recorded reason: owning the doorbell + "makes the reset discipline an internal invariant rather than a client + obligation". +2. **A client callback on the producer's submit path is a cadence hazard** -- if + it blocks, it stalls the producer. +3. **Type-parameter propagation**, which is the cost the file-watcher actually + measured: a `Doorbell` trait "would have made `Monitor`, `Session`, and + `Sender` all generic over it". + +And the extension point already exists at no cost: hand out the HANDLE. The +client composes it into `WaitForMultipleObjects`, a `ThreadpoolWait`, an async +reactor, or ignores it -- outside our lock, on their own schedule, with no type +parameter reaching `Ring`, `Domain`, and every producer handle. + +On the narrower question of generics versus `dyn` should polymorphism be needed +elsewhere: static dispatch on the hot path, but the cost that actually bites is +the type parameter infecting every type that touches a queue, not the dispatch. + +## Two locality consumers, not one + +Raised by the engineer's question about what a client gets back and whether it +is affinitized to the calling thread. It exposes an incompleteness in the ioring +notes' strongest placement claim. + +There are **two** consumers of a buffer's locality: + +- the **device**, which DMAs into it; +- the **client**, which reads it after completion. + +The notes argue placement dominates because "a buffer on a node remote from the +device means every byte crosses the interconnect, on every operation, forever". +That is true of the DMA and silent about the read side. With the device on one +node and the client thread on another, both cannot be satisfied: + +| Buffer placed | DMA cost | Client read cost | +|---|---|---| +| near the device | local | remote, on every read | +| near the client | one crossing | local | + +**A DMA writes once; a consumer may read many times.** So "near the device" is +not automatically right -- it wins when the consumer barely touches the bytes, +and loses when the consumer works over them repeatedly. The notes' claim is +incomplete rather than wrong, and the completion is that the *access pattern* +decides. + +**On binding to the calling thread: no, not implicitly.** An unpinned client +thread migrates, so a binding inferred from where it happened to be is stale +before it is used. That is the namespace session's decision 9 -- "ambient state +is derived from an explicit binding, never the origin of one". + +**Proposed instead:** the client *asks* -- "which domain is nearest me?" -- and +then uses it explicitly. Topology becomes an input the client may consult, never +an authority applied behind its back. A foreign thread has in any case already +accepted a hop and probably a remote read; a client wanting true consumer +locality should be running *on* the domain. + +## Coherence assessment + +Honest state of the design at the end of this session, since it was asked +directly. + +**Solid.** The two-layer ring resolves submission thread-safety, the missing CQ +post entry point, and "one ring or two" with a single decision; three +constraints falling to one choice is usually a sign the decomposition is right. +The domain-owned pool makes NUMA invisible rather than merely easy. The uniform +architecture sizes from 1 to N without a second mode. + +**Structurally open, and not to be papered over:** + +1. **The client-facing ring is a substantial new artifact** -- MPSC, doorbell, + descriptor format, completion tagging -- existing nowhere in this workspace. + It needs a home and probably its own crate, and it is larger than anything + this session has called "the seam". +2. ~~**CQ cardinality.**~~ **Resolved** -- one CQ per domain, client chooses the + observation strategy. See "Resolved: CQ cardinality" above. +3. ~~**Whether the client ever sees an `IoRing`.**~~ **Resolved by the + engineer: it does not.** `windows-ioring-sys` becomes an implementation + detail of the higher crate. This is layering rather than absorption -- it + remains a published crate in its own right (0.2.0 shipped 2026-08-30) and + gains a dependent; direct consumers can still use it. + +## Specification: the submission and completion queues + +Requested as a specification rather than a sketch. These are the requirements +the session's conclusions imply. The two directions are **not** the same shape. + +**R1 Cardinality.** The SQ is **MPSC** -- many client threads, one domain +thread. The CQ is **SPSC** -- one domain thread, one drainer. The CQ constraint +is deliberate: "drain to empty" is ambiguous with two racing drainers, and +drain-to-empty is what +[D-19](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-19) requires. Nothing is +lost, because per-domain CQs already give a client N drainers; a client wanting +parallel processing drains on one thread and dispatches. + +**R2 Bounded.** Fixed capacity at construction. `push` on a full queue returns a +typed error; it never blocks and never grows. **That failure is the +backpressure** -- an unbounded queue has none, which is why `SegQueue` was the +wrong model. + +**R3 Lock-free producers.** No mutex on the producer path. A producer-side lock +serializes precisely what multi-producer exists to parallelize. + +**Park and notify may start as an unconditional signal.** An earlier form of +this requirement mandated an **eventcount** -- consumer publishes intent to +park, re-checks the queue, then waits -- as the way to close the lost-wakeup gap +without a lock. **C-1 measured that and the mandate does not survive**: batching +alone drives the doorbell below the cost of the atomic push it accompanies, so a +first implementation may always-signal. The eventcount stays in the design as a +*later* step, adopted against a measurement of real work rather than up front, +because publish-recheck-park is the highest-risk protocol here and its +lost-wakeup risk should be bought only once something has shown it is worth +paying for. + +**R4 Doorbell.** A queue-owned **manual-reset event**, created **lazily** so a +polling-only consumer allocates no kernel object. Level semantics: signalled +exactly when the consumer has something to observe. **The reset is atomic with +the emptiness observation; the signal may be outside any lock** (see C-1b). +Handed out as a borrowed handle plus an owned duplicate. + +Skipping the signal when the queue was already non-empty, or when the consumer +is not parked, is an **optimization rather than a requirement** -- see R3 and +C-1. The skip that costs nothing and can be taken immediately is the +already-non-empty one, which needs no knowledge of the consumer's state; the +one that needs the consumer to publish whether it is parked is the part C-1 says +to defer. + +**R5 Wakeup safety.** No lost wakeups. Spurious wakeups are permitted, and the +consumer must tolerate them. Drain to empty on every pass. + +**R6 Parking.** Optional consumer spin before parking, with the duration tunable +and **sized by the topology** -- generous when a domain owns a core exclusively, +zero when it shares one with the rest of a laptop. + +**R7 Payload.** Two parts, because an earlier single-sentence form of this +requirement ("POD descriptors only, no allocation on push") could not hold for a +namespace open, whose payload owns a heap-allocated path: + +- **The slot is fixed-size and POD**: operation, correlation id, buffer slot + index, offset, user tag. +- **The payload may be an owned value moved in** -- an `OpenFile` from + [windows-namespace-request-sys](../crates/windows-namespace-request-sys/README.md), + which is `Send`, owns its `PreparedPath`, and holds a *duplicated* handle. + +**No allocation inside the push**, and no borrowed lifetime crossing the hop: +the client allocates before pushing, on its own thread, where blocking and +failing are both acceptable. Never bytes -- carrying bytes would mean copying +out of the registered pool, defeating the reason to register. + +**R8 Shutdown.** The consumer learns when all producers are gone; producers +learn when the consumer is gone and fail with a typed error. Descriptors in +flight at teardown are **accounted, not dropped** -- some own handles, and their +disposal must be allowed to block. + +**R9 Observability.** Depth and high-water for tuning, plus **a count of +doorbells actually rung**. That makes R4's skip rule measurable rather than +assumed, and sabotage-verifiable: disabling the skip must change the number. + +**R10 No client callbacks.** No trait and no closure on the producer or consumer +path. The HANDLE is the extension point. + +**What is reused from the file-watcher, and what is not.** The *invariant* is +reused: the event is level state, signalled exactly when there is something to +observe, with the reset atomic against the emptiness decision. The +*implementation* is not: [queue.rs](../crates/windows-file-watcher/src/queue.rs) +uses `Mutex` and `Condvar`, which is right for change-notification cadence and +wrong for an I/O hot path, because it puts a lock on the producer side. Stating +this explicitly so that "reuse the queue" does not become reuse of the wrong +half. + +## Resolved: CQ cardinality, and a correction about how wide waits work + +**Correction, from the engineer.** An earlier turn in this session claimed +`ThreadpoolWait` "internally manages the 64-handle groups". That is wrong. Modern +thread-pool waits are backed by **kernel-side wait completion packets** +associated with the pool's completion port; there is no user-mode grouping and +no fan-out of waiting threads per 64 handles. + +That improves the answer rather than complicating it: wide waits cost the +dispatch hop, not a thread per group. + +**Resolution: one CQ per domain, one HANDLE each, and the client chooses how to +observe them** -- `WaitForMultipleObjects` on its own thread when the count is +within the limit and no hop is wanted, `ThreadpoolWait` when wider or when the +hop is acceptable. + +This preserves shared-nothing, since there is no single queue every domain +writes to, and it pushes the trade-off to the only party that knows which side +of it it is on. It is also the payoff from rejecting the `Ring` trait: because +the extension point is a HANDLE, this strategy did not have to be anticipated. + +## Resolved: how a client places its own threads + +Following from "two locality consumers" above and the question of whether the +facility would end up building a thread pool. There are three thread +populations, and each has a distinct justification: + +| Population | Owner | Why | +|---|---|---| +| Namespace and blocking operations | the **Windows pool** | needs quarantine and elasticity; `runs_long`; must survive a wedged network call | +| Domain I/O threads | **us**, one per domain | pinning; the Windows pool cannot affinitize | +| Client continuations | **the client** | see below | + +Three options were considered for the third row: (a) continuations run on the +domain thread, Seastar-style -- best locality, but client code must never block; +(b) placed worker threads per domain, which is the thread pool the engineer +wanted to avoid; (c) the facility reports placement and the client places its +own threads. + +**Chosen: (c)**, with (a) available for consumers who want it, and (b) only +against a real consumer that neither serves. The risk worth guarding is not +building (b) but building it *first*, before knowing whether (c) suffices. + +### Two tiers of thread construction, because binding afterwards is not equivalent + +An earlier form of this section concluded that a one-call binder was sufficient +and that a `CreateThread` wrapper would be "edging too close to the slippery +slope". **That was wrong, and the engineer corrected it**: constructing a thread +so that it has the right attributes *from the beginning* -- stack as well as +processor affinity -- is the difficult part, and it is the part a consumer +cannot easily do. + +**Why binding afterwards is strictly worse.** A thread's stack is allocated at +creation time, on whatever node the *creating* thread's policy selects. Spawn +from node 0, bind to node 1, and the stack stays on node 0 permanently -- every +local, every spill, every call frame is a remote access for the life of the +thread, and no amount of later `SetThreadGroupAffinity` moves it. There is also +a window before the bind lands in which the thread runs on the wrong processor +and warms the wrong caches. + +**Why this is a missing layer rather than a convenience.** Creation-time +affinity requires `CreateRemoteThreadEx` against one's own process with a +`PROC_THREAD_ATTRIBUTE_LIST` carrying `PROC_THREAD_ATTRIBUTE_GROUP_AFFINITY` and +`PROC_THREAD_ATTRIBUTE_IDEAL_PROCESSOR`, assembled through +`InitializeProcThreadAttributeList` and `UpdateProcThreadAttribute` -- a two-pass +sizing call, a manually managed opaque buffer, and lifetime rules requiring the +attribute values to outlive the call. `std::thread::Builder` can set a stack +size and **nothing else**; it spawns with no attribute list. So a Rust consumer +has *no path* to a correctly constructed thread without dropping to raw Win32, +and once there must also re-supply what `std` was doing for it, notably catching +unwind at the entry so a panic does not cross an `extern "system"` boundary. + +Each of those steps is simple. Collectively they are a minefield nobody crosses, +which is the [SMOP principle](../DESIGN-NOTES.md#the-value-is-existence-not-cleverness) +exactly: the value is existence, and when the correct construction is difficult, +providing the constructor *is* the feature. + +**The line is ownership, not construction.** The facility helps *construct* a +thread and never *owns* one: a builder assembles the attribute list, applies the +domain's `GROUP_AFFINITY`, sets a stack reservation, wraps the entry in +`catch_unwind`, and hands back a thread **the client owns**. No handle kept, +nothing monitored, nothing restarted. The slope is ownership; construction is +not a step down it. + +| | correct from birth | for threads you did not create | +|---|---|---| +| stack placement | follows the creation-time affinity | already fixed, possibly remote | +| pre-bind window | none | exists | +| API | domain thread builder | `bind_current_thread()` plus restore guard | + +The binder remains, honestly labelled as the degraded path, for threads the +client did not create -- a pool thread, an existing worker. Its restore guard is +required rather than decorative: a client binding a *pool* thread must restore +it, because the thread-pool contract is that a callback restores any thread +state it changes. That also places the feature, since affinity is thread-scoped +state applied and restored, exactly the family +[windows-thread-ambient-sys](../crates/windows-thread-ambient-sys/README.md) +already handles. It either belongs there or must mirror that crate's guard +discipline rather than inventing a second pattern. + +**Unverified, and worth measuring rather than assuming:** whether a thread +created with `PROC_THREAD_ATTRIBUTE_GROUP_AFFINITY` actually receives a +node-local *stack*. It is measurable -- `QueryWorkingSetEx` returns +`PSAPI_WORKING_SET_EX_BLOCK` with a `Node` field, so the address of a local in +the new thread can be asked which node its page is on -- and it is added as Q7 +to [file-handle-numa-spike.rs](../crates/windows-ioring-sys/design-sessions/spikes/file-handle-numa-spike.rs). +Same hardware blocker as F-1: on a single-node machine the answer is always 0. + +So option (c) is concrete: the domain's `ProcessorSet`, an answer to "which +domain is nearest me", a builder that constructs a correctly placed thread, and +a binder for threads that already exist. Every piece needed to process a CQ on +the right thread with the right affinity, and not one thread of ours. + +## Working position on domain counts (not a decision) + +Only the first row is measured. The rest are from published topologies and must +be treated as unverified until probed. + +Read this as **the count the one architecture takes on each machine**, not as a +boundary between supported and unsupported hardware -- see the convergence above. +Every row runs the same domain shape; the rows differ only in N and in how wide +each domain's affinity set is. + +| Configuration | Cores / LPs | Cache domains | Nodes | Groups | I/O domains | +|---|---|---|---|---|---| +| Snapdragon X2 Elite (measured) | 12 / 12 | 2 x L2, no L3 | 1 | 1 | 1 | +| Intel Core Ultra laptop (P+E+LPE) | ~16 / 22 | 1 x L3 | 1 | 1 | 1 | +| Cloud VM, 4-8 vCPU | 4-8 | 0-1 | 0-1 | 1 | 1 | +| Ryzen 7800X3D / 9800X3D | 8 / 16 | 1 x L3 | 1 | 1 | 1 | +| Ryzen 7950X | 16 / 32 | 2 x L3 (2 CCD) | 1 | 1 | 1-2 | +| Threadripper / 1-socket Xeon | 32-64 | 2-8 | 1-4 | 1-2 | 2-4 | +| EPYC 9004 96-core NPS1 | 96 / 192 | 12 x L3 | 1 | 3 forced | 4-12 | +| Dual-socket Xeon SPR + SNC | 64 / 128 | 2 x L3 | 4 | 2 | 2-4 | + +The reasoning behind the numbers matters more than the numbers: + +- **The bound is the storage device, not the CPU.** Shard-per-core is a Seastar + *application* answer, where the shard owns application state and I/O is + incidental. For an *I/O domain* count, throughput comes from queue depth, not + thread count: one consumer NVMe drive is saturated by one or two submitting + threads at moderate queue depth, and further threads add registered pools and + device-queue contention without adding throughput. +- **The notes already point here.** "Buffer placement probably dominates thread + placement": a buffer on a node remote from the device means every byte crosses + the interconnect forever, where the callback's location is a one-time + cache-warmth question. If placement relative to the *device* dominates, the + device sets the partition. +- **Registration punishes guessing high.** Registration pins pages and is + one-shot per ring, so N domains means N separately pinned pools: a 256 MiB + working set is 256 MiB pinned at one domain, or 3 GiB at twelve, or twelve + pools too small to keep a device busy. +- **The mapping needed is coarse and often absent** -- see F-1 above, which + corrects the ioring notes' stronger claim that it is unreachable. A volume's + node is one documented FSCTL away, but it is volume-granular, frequently has + no answer, and is meaningless where one volume spans several devices. Device + topology is therefore still configuration rather than detection, for a + narrower reason than the notes currently give. + +**Proposed default, not yet agreed:** one domain, adding one only when the +device it serves can be named; above 64 logical processors take the group floor +and no more. + +## Working under a hardware gap: what is blocked, and what breaks + +The engineer expects access to a genuine multi-node NUMA machine eventually, but +not soon. Until then this design rests on documentation rather than measurement, +which is a different grade of evidence than most decisions in this repository +and must not be allowed to blur into them. + +**Sorted by dependence on an unverified NUMA claim:** + +| Not blocked -- verifiable on the development machine | Blocked on NUMA hardware | +|---|---| +| the MPSC, eventcount, and doorbell (R1-R10 is pure concurrency) | whether the FSCTL names a *meaningful* volume node (F-1) | +| the two-layer ring and the client-facing API shape | Q6, whether a Storage Space reports honestly or reports a fiction | +| one-shot registration semantics (already established) | Q7, whether creation-time affinity yields a node-local stack | +| ~~the C-1 doorbell measurement~~ -- **done**, see "C-1 measured" above | the *magnitude* of the buffer-placement benefit | +| the composed layer's type-level traversal | domain-count tuning above one | +| the durability crate, whose mechanism was already measured as D-23/D-24 | | +| whether `CreateRemoteThreadEx` with an attribute list works at all | | + +**The pattern in the blocked column is the reassuring part: none of those +threaten the structure. They threaten the justification and the tuning.** If +every one came back badly, the architecture would stand and the NUMA-specific +features would be decorative rather than wrong. + +**And the first deliverable depends on none of them.** At N=1 there is no +routing, no placement choice, and the buffer goes on the only node there is. So +"build N=1 first", chosen above because it is the common case and the substrate, +is *also* the plan that needs no NUMA hardware. The whole first deliverable and +most of the second can be built before the machine exists. + +### Practices to adopt while the gap lasts + +1. **Mark documented-but-unwitnessed claims distinctly from measured ones.** + This repository's decisions are unusually well measured, which creates its own + hazard: a reader cannot tell + [D-23](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-23) -- measured, with a + control case -- from a claim taken off a documentation page. Anything + load-bearing that rests on documentation must say so *in the decision*, the + way F-1 above says "contributed by the engineer as research, not measured + here". + +2. **Quarantine each unverified claim so that a correction is surgical.** Do not + let "creation-time affinity yields a node-local stack" become load-bearing for + anything beyond the thread builder's justification. If Q7 returns false, the + response should be editing one rationale, not restructuring a design. + +3. **Pre-build the instruments now, while the context is fresh.** Time on a + borrowed machine is likely to be short and should be spent measuring, not + writing `CreateRemoteThreadEx` attribute-list code. F-1's spike is already + written *and smoke-tested*, and that smoke run found a real defect in it -- + the `File::open` directory failure that would otherwise have surfaced on the + borrowed machine. Every remaining question deserves the same treatment before + the hardware appears. + +### What to run when the machine is available + +Ordered so that a short session yields the most: + +1. **F-1 / Q1-Q5** -- [file-handle-numa-spike.rs](../crates/windows-ioring-sys/design-sessions/spikes/file-handle-numa-spike.rs), + as-is, against an ordinary volume. Establishes whether either call names a + meaningful node, and whether they agree. +2. **Q6** -- the same spike with a Storage Space directory as `argv[1]`, with the + space's layout recorded alongside. Distinguishes an honest answer from a + fiction, which is the outcome that would be worse than no answer. +3. **Q7** -- [thread-stack-numa-spike.rs](../crates/windows-ioring-sys/design-sessions/spikes/thread-stack-numa-spike.rs), + written and smoke-tested. Decides whether the thread builder's principal + justification holds. Three threads (attribute at creation, plain control, + plain-then-bound) each report a shallow and a deep stack page, which also + distinguishes creation-time placement from first-touch placement. +4. **Magnitude** -- a read benchmark with the registered pool placed local + against remote, which is the number the entire domain-count argument rests on + and which nothing in this session has measured. + +## Open questions + +- **The three structural gaps** in the coherence assessment above: where the + client-facing ring lives, CQ cardinality, and whether the client ever sees an + `IoRing`. These are now the session's principal open questions and they + supersede the framing below. +- **Which end to design from.** Three candidates were put: (a) the seam between + the pooled namespace plane and the pinned data plane, given one-shot + registration and no shared ring storage; (b) the `RingFleet` that + [D-8](../crates/windows-ioring-sys/DESIGN-NOTES.md#d-8) deferred; (c) the queue + and doorbell substrate. Not yet chosen. +- **How a newly opened handle reaches a shard at all**, given registration cannot + be extended. Candidates not yet evaluated: pre-registered fixed tables with + slot management, unregistered handles on the hot path, ring recycling. +- **Shard count for a general substrate**: per core, or per outermost + partitioning cache. Related: whether an I/O domain count and an application + shard count are the same number, which the "all three coincide" argument + assumes. +- **Heterogeneous cores.** Intel P/E/LPE and ARM performance/efficiency clusters + mean pinning an I/O shard to an efficiency core is a latency trap. + `windows-topology-sys` exposes `efficiency_class` on `Core` [now the + `ProcessorFacts::efficiency_class` field, an `Observed`, so it may now be + absent with a reason rather than a bare number], so the input exists; no + policy consumes it. +- **Whether the L2 cluster structure on ARM parts is worth sharding on**, or + whether one whole-machine domain is the right answer there anyway. + +## Corrections made during the session + +- **"`ByL3` yielding zero domains would create zero rings" was wrong.** + `Policy::select` already degrades to a whole-machine domain and flags it. The + surviving finding is about the *justification prose* in the ioring notes, not + about the sample's behaviour. +- **"The `epoch_log` example does sector-aligned I/O" was wrong**, and was part + of the session's opening premise. That example opens a plain file and gets + alignment structurally from 4096-byte slots. The actual + `FILE_FLAG_NO_BUFFERING` work lives in `tests/flush_barrier.rs` and + `tests/handover.rs`. NUMA-affined buffers (`ring_copy`) and sector-aligned + unbuffered I/O (those tests) are currently **two separate things in two + separate places**, never combined.