diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 211c59e3..6274b8d9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -243,6 +243,40 @@ jobs: run: cargo run -p windows-platform-probes --bin probe-ioring --locked - name: probe magnitudes (completion port) run: cargo run -p windows-platform-probes --bin probe-completion-port --locked + # Printed on every build on purpose. Hosted runners are a heterogeneous + # fleet, so accumulating this across builds turns ordinary CI into a slow + # survey of what real machines look like -- and the negative result, that + # cloud runners are consistently single-node, is itself evidence for how + # the execution-domain design should size itself by default. The probe + # emits one `x-probe-topology` JSON line so the results can be mined out + # of logs mechanically rather than read by eye. + - name: probe magnitudes (topology) + run: cargo run -p windows-platform-probes --bin probe-topology --locked + # `--release` on the next two, and ONLY on those two, because they are the + # only probes here that report nanoseconds. Everything above measures + # BEHAVIOUR -- what an API refuses, what a handle reports, which errors + # appear -- and that does not change with the optimisation level, so those + # keep the dev profile and the build time it saves. + # + # These two do not have that luxury. They compare operations a few tens of + # nanoseconds apart, and in an unoptimized build the loop, the closure + # indirection, and the request construction around each measured call + # carry overhead that does not shrink uniformly across them. The RATIOS + # are what the design reads -- "a doorbell is x% of a submit", "a captured + # handle costs Nx a built request" -- and a ratio of two figures each + # inflated by a different amount is not the shipping one. + # + # Decides how much machinery the two-layer ring's doorbell needs. Its + # park-and-wake handshake is bounded rather than INFINITE on purpose: the + # first version of it deadlocked, because an auto-reset event does not + # count signals and the waiter's count never caught up. A probe that can + # hang is a probe that can hang a build. + - name: probe magnitudes (doorbell cost) + run: cargo run -p windows-platform-probes --bin probe-doorbell-cost --locked --release + # Read with the doorbell probe above: together they say whether the + # queue's mechanics or the request's own cost deserves the attention. + - name: probe magnitudes (request cost) + run: cargo run -p windows-platform-probes --bin probe-request-cost --locked --release # Both halves of the long-path pair, deliberately. Either alone says # nothing: the finding is the *difference* between two executables that # differ only in whether `build.rs` embedded the `longPathAware` manifest, diff --git a/.gitignore b/.gitignore index 16ceee39..f1a2d501 100644 --- a/.gitignore +++ b/.gitignore @@ -13,5 +13,17 @@ target # Scratch / diagnostic output (git-ignored per repo instructions) .scratch/ + +# windows-placement-probe writes its backup record into the working directory, +# so running the tool from inside a checkout leaves one of these behind. Ignored +# because `git add -A` will otherwise sweep a machine's measurements into a +# commit, which has already happened once. +placement-probe-v*.json .vs .vscode/settings.json + +# tpu-mcp writes .bak beside any file it repairs. Those are transient +# backups of a repair that has already been verified and committed, so they are +# never wanted in history -- and being adjacent to the file they back up, they +# are easy to sweep up with `git add -A` without noticing. +*.bak diff --git a/CHECKLIST-io-domains.md b/CHECKLIST-io-domains.md new file mode 100644 index 00000000..43d3189d --- /dev/null +++ b/CHECKLIST-io-domains.md @@ -0,0 +1,842 @@ +# Checklist: NUMA-sharded I/O execution domains + +Feature-scoped checklist for the `mikegrier/deferred-namespace-ops` branch. It covers a new queue crate, +a domain runtime, a durability layer, and extensions to three existing crates, so it lives at the +workspace root -- their lowest common source-component -- rather than inside any one of them. Per the +naming convention for feature files, it is deleted outright once every item is complete, with the content +moved to [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md). + +Authoritative decisions are in [DESIGN-NOTES.md](DESIGN-NOTES.md); the session that produced them is +[DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md](design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md), +which is **still open**. Milestone numbers continue the workspace sequence: [CHECKLIST.md](CHECKLIST.md) +holds M19-M21 and [CHECKLIST-thread-ambient.md](CHECKLIST-thread-ambient.md) held M22-M27. + +## What is ready, and what deliberately is not + +**M30-M32 are specified.** The queue crate's requirements (R1-R10, as amended by the C-1 and +request-cost measurements) are settled, its shapes are chosen, and its boundary is decided. None of it +depends on NUMA hardware, and all of it is testable with no ring, no pool, and no I/O. + +**M33+ is parked, not pending.** The domain runtime cannot be written until M32 settles three contract +questions -- ordering, correlation, and backpressure -- that would change its shape. Those are decision +items in M32 rather than assumptions baked into M33+, because the session recorded them as open and +promoting an open question to a settled one by writing code around it is exactly the failure this +sequencing avoids. + +**The N=1 path is the whole of the first deliverable.** A single execution domain needs no routing +policy, no cross-domain queue, and no placement choice, so nothing below is blocked on the multi-node +hardware the session could not obtain. What N>1 adds is additive, not a second mode. + +## M30 -- The queue crate: name, skeleton, and the SPSC shape + +- [x] **M30.1** -- Decide the crate's name and record why, **before** anything depends on it, because + renaming a crate that has dependents is churn this repository avoids. Two things to settle together: + whether the `-sys` suffix applies (every existing `windows-*-sys` crate is thin-over-Win32, and this is + a data structure with an opinion, so it probably does not), and the name of the domain runtime crate + that will sit above it, since the pair should read as a pair. Candidates raised: `windows-io-queue`, + `windows-signalled-queue`, `windows-queue`. Record the decision in [DESIGN-NOTES.md](DESIGN-NOTES.md) + so the reasoning survives the choice. + **Decided: `windows-waitable-queues`**, no `-sys` suffix, recorded in + [DESIGN-NOTES.md](DESIGN-NOTES.md#the-waitable-queues-crate-is-named-plural-and-carries-no-sys-suffix). + The engineer proposed the plural and it is right for a reason stronger than taste: + [windows-platform-probes](crates/windows-platform-probes/README.md) is already plural, so the workspace + distinguishes singular-for-one-facility from plural-for-a-collection-of-peers, and this is the second + kind. **`windows-io-queue`, floated during the same discussion, was rejected** -- the queues have + nothing to do with I/O, and naming a general facility after its first consumer is the mistake + `windows-topology-sys` avoided. What unifies them is waitability, a word this workspace already owns + through `WaitableHandle`. + **One consequence accepted deliberately:** the plural forbids a bare `Queue` type, since a crate named + "queues" exporting one would claim a primacy the name denies. Every type is specifically named and a + consumer must say which it wants. + **The runtime crate's name is deliberately NOT fixed here**, which narrows this item as written. The + churn argument applies to a crate with dependents, and M30.2 creates the queue crate immediately while + the runtime does not exist until M33+. The rule is recorded instead -- the pair should read as a pair, + and the runtime's name may carry `io` because that crate genuinely is about I/O. + +- [x] **M30.2** -- Create the crate with `publish = true` (the engineer's decision: this is general-purpose + and worth publishing, unlike `windows-guard-alloc`), and write its [DESIGN-NOTES.md](crates/windows-waitable-queues/DESIGN-NOTES.md) with the decisions + the session already reached: the shape menu and which shapes ship now, the + concrete-types-plus-optional-trait rule, the overflow policy, and the doorbell invariant. This is the + Tier-1 transcription of Tier-3 session content -- design notes are not a work queue, so a decision that + lives only in the session record is orphaned. + + **Settle one question here that M30.3 then depends on:** whether `WaitableQueue` is a single + consumer-side trait, or a producer trait and a consumer trait. Waitability lives on the consumer -- it + waits, while the producer merely rings -- so a single trait would be consumer-side and the producer's + contract would go unnamed. Decide rather than defer, because M30.3 writes the first signatures against + whichever answer this gives. + **Answered, and the question turned out to be the wrong one.** The engineer's observation that the + shapes would be "sliced and diced by various traits as we go along" is right, and following it shows a + single `WaitableQueue` trait -- one *or* a producer/consumer pair -- is not merely inelegant but + **unimplementable by the shapes that are planned**: a poll-only queue has no doorbell to return, and an + unbounded one has no capacity to report. So the answer is **narrow capability traits** on the + `std::io` model (`Read`/`Write`/`Seek`, not one `Io`), each naming one capability, with a shape + implementing the subset it genuinely has. Recorded as [D-2](crates/windows-waitable-queues/DESIGN-NOTES.md#d-2) + with the anticipated set. + **And no trait ships until a second implementation exists to validate it** ([D-3](crates/windows-waitable-queues/DESIGN-NOTES.md#d-3)): + a trait written against one type designs in a vacuum, since every signature that type happens to have + looks like a requirement. The trait *shape* is fixed now because it constrains M30.3; the traits + themselves land with M31.1. + Crate created with [DESIGN-NOTES.md](crates/windows-waitable-queues/DESIGN-NOTES.md) (D-1..D-8), + [README.md](crates/windows-waitable-queues/README.md), [PLANS.md](PLANS.md) pointing back at this file, and + registration in the workspace members, [release-please-config.json](release-please-config.json), and + [.release-please-manifest.json](.release-please-manifest.json) + -- the last two because `publish = true` makes it release-managed, and omitting them would have left it + silently unreleasable. + **One earlier position reversed with its reason recorded** ([D-7](crates/windows-waitable-queues/DESIGN-NOTES.md#d-7)): + shapes are plain modules, not Cargo features. Two features are four configurations against a + `feature-matrix` CI job that would have to grow, and the benefit is one dead-code elimination already + provides. + +- [x] **M30.3** -- The SPSC bounded ring, with no doorbell and no Win32 at all: a pure data structure with + acquire/release head and tail and no CAS on either side. It is the CQ direction (R1), and it is first + because everything harder is a variation on it. Tests are ordinary fast unit tests -- capacity edges, + wraparound, full and empty, and that a `pop` never observes a partially written `T`. + + **This item sets the shape every later queue must match**, so it is where the trait-compatibility + constraint binds: split producer and consumer handles, with cardinality carried by whether each is + `Clone` (see + [DESIGN-NOTES.md](DESIGN-NOTES.md#the-waitable-queues-crate-is-named-plural-and-carries-no-sys-suffix)). + Getting this wrong is not a local mistake -- if the first shape ships a signature the second cannot + match, the `WaitableQueue` trait becomes a breaking change to one of them rather than an addition. + Verify it the cheap way: write the trait's method signatures down as a comment before writing the + type, and confirm the type satisfies them. + **Done, and the signatures are written down in [spsc.rs](crates/windows-waitable-queues/src/spsc.rs)'s module documentation before the type**, as + the item asked. `push`/`pop` take **`&self`**, not `&mut self`: the latter would also make + single-producer sound and is what several SPSC crates use, but it cannot generalize to a shape where + several threads push through a shared handle, and one spelling has to serve every shape. + Cardinality is carried by the auto traits instead -- the handles are `Send` but **not `Sync`** and not + `Clone`, so "single" is a fact the compiler checks. A multi-producer shape relaxes exactly one cell of + that table. + **Sabotage-verified rather than merely green.** Six deliberate defects, each confirmed to fail the + suite: a drop loop starting at zero instead of `head`, an off-by-one in the full test, `Full` reported + where `Disconnected` is owed, `pop` not advancing `head`, `push` not advancing `tail`, and a mask of + `capacity` instead of `capacity - 1`. + **One sabotage was NOT caught, and it is recorded rather than smoothed over:** weakening the producer's + `Acquire` load to `Relaxed` leaves the suite green. That is a genuine limit of stress testing, not a + missing test -- an ordering bug needs an interleaving the hardware and scheduler must be coaxed into + producing, and neither ARM64 nor x86-64 will oblige on demand. Queued as M31.6. + **A harness defect worth remembering:** the first sabotage sweep reported "not caught" for the + `pop`-does-not-advance case, because the detector matched on the string `test result: FAILED` and the + test process had instead died with `STATUS_HEAP_CORRUPTION`, which prints no such line. Nine tests had + in fact failed. A sabotage harness that recognizes only one failure shape will eventually certify a + hole that is not there -- or miss one that is. Detect by exit code. + The same defect then bit the *gate*: piping `cargo clippy` through `Select-String` makes + `$LASTEXITCODE` report the filter's status, not cargo's, so a clean-looking `exit=0` was hiding a real + `-D warnings` failure (`clippy::doc_overindented_list_items`, seven sites in `windows-platform-probes`, + actual exit 101) that CI would have caught. Fixed in the preceding commit. Redirect with `*>` and read + `$LASTEXITCODE` before any pipe. + +- [x] **M30.4** -- The doorbell, as its own reviewable unit: a queue-owned **manual-reset** event created + **lazily**, so a polling-only consumer allocates no kernel object. Level semantics -- signalled exactly + when the consumer has something to observe. **The reset must be atomic with the observation that there + is nothing to take; the signal need not be** (C-1b measured why: a late signal is a spurious wakeup, a + stale reset is a lost one). Hand it out as a borrowed handle plus an owned duplicate, per the + file-watcher's precedent. + **Landed together with M30.5 in one commit, because these two items are not independent and the + checklist was wrong to split them.** A `Doorbell` that no queue calls is dead code, and this workspace + builds with `-D warnings`, so M30.4 cannot compile on its own. Recorded as an acknowledged + structuring defect rather than worked around by widening the type's visibility to silence the lint -- + making an API public to dodge a warning is a real design decision taken for a fake reason. + Delivered as [src/doorbell.rs](crates/windows-waitable-queues/src/doorbell.rs): lazily created (a poll-only consumer allocates no kernel object, + asserted, not assumed), manual-reset, with `handle` / `owned` / `signal` / `clear`. The redundant + signal is skipped through an `AtomicBool` mirroring the event, which is sound in exactly one + direction -- see the done-note on M30.5 for the asymmetry that permits it. + +- [x] **M30.5** -- Join the two, and **sabotage-verify the lost-wakeup guard**: a test that reverses the + reset and the emptiness check must deadlock, and must stop deadlocking when the order is restored. A + wakeup invariant asserted only by a passing test is a test of nothing -- this is the same discipline + the ioring crate's `wait_then_drain` and the M17.4 calibration established. + **Done. The guard is `Consumer::arm`, which clears the doorbell and *then* checks emptiness** -- the + reverse of the order that reads naturally, which is why it needed proving rather than asserting. The + sabotage test drives the race deterministically on one thread (an interleaving that must be hit to + prove a point is not one to leave to the scheduler) and ends in a real bounded `WaitForSingleObject`: + reversed, it returns `WAIT_TIMEOUT` with an item sitting in the queue -- the lost wakeup, reproduced; + correct, the check finds the item and never waits at all. + **Corrected after review: the deterministic test named above tested a *copy* of the wrong order, + not the real `arm`.** `arm_reversed_racing` is a hand-written duplicate with the two statements + swapped, so it could only ever show that *a* reversed order is wrong -- it could not detect the real + `arm` being reversed. Measured: sabotaging the real `arm` was caught in **one run out of three**, + because detection then depended on two threads interleaving inside a window tens of nanoseconds wide. + This is the anti-pattern CONTRACT INTEGRITY rule 1 names -- a second copy of a rule checks the copy, + not the rule -- and it was found only because the sweep was re-run under a tighter bound and the + result flipped. `Consumer::arm` now carries a `#[cfg(test)]` hook that fires between the clear and the + check, so a test drives the **real** `arm` through that exact window on one thread. Caught every run + since. + **Thirteen sabotages, twelve defects and one control, all behaving as expected.** Caught: push not + signalling; producer `Drop` not signalling; `arm` checking before clearing; `arm` not creating the + doorbell before checking; the final drain returning nothing; `clear` resetting the event but not the + mirror flag; auto-reset instead of manual-reset; the event created already signalled. Three of those + are caught **as hangs rather than failures**, which is the correct shape for a lost-wakeup defect. + **The control matters as much as the defects:** removing the skip-redundant-signal optimisation must + *not* fail, and does not -- so the suite is asserting the contract rather than the implementation. + **The sweep paid for itself twice, and neither finding came from reading the code.** + (1) A test gap: the drain-after-disconnect guard sat in a race window no test could reach, so + breaking it changed nothing. Fixed by extracting it as `Consumer::finish`, a named step a test can + call directly instead of hoping to schedule the window. + (2) A harness defect: one sabotage inserted `if false { signal(); }` beside the live call instead of + deleting it, so it sabotaged nothing and the resulting pass read as a hole in the tests. A sabotage + that does not sabotage is worse than none, because it retires a question that was never asked -- + always confirm the injected defect actually changes behaviour before believing a "not caught". + +## M31 -- The MPSC shape and the queue's contract + +- [x] **M31.1** -- The bounded array MPSC: Vyukov's sequence protocol, where a producer CASes the tail + forward, writes, then publishes by storing the slot's sequence. Lock-free rather than wait-free, bounded + by construction so backpressure is free, and no allocation anywhere. Pad the head and tail onto separate + cache lines and say so in a comment, because the padding is load-bearing and looks like waste. + **Done as `src/mpsc.rs`, since renamed to + [slotwise_mpsc.rs](crates/windows-waitable-queues/src/slotwise_mpsc.rs)**, with the padding commented at *both* positions rather than once, since a + reader arriving at either field is the one who might delete it. Recorded as + [D-10](crates/windows-waitable-queues/DESIGN-NOTES.md#d-10). + **The traits landed here too, because M30.2 scheduled them here** ("the traits themselves land with + M31.1") and [D-3](crates/windows-waitable-queues/DESIGN-NOTES.md#d-3) required a second implementation + to validate them against. **The signatures `spsc` wrote down in advance held unchanged**, which is that + check actually being run rather than assumed, and the load-bearing one turned out to be `push(&self)`: + `&mut self` would have been sound for one producer and would have made the trait *unimplementable* by + this shape. Recorded as [D-11](crates/windows-waitable-queues/DESIGN-NOTES.md#d-11). `Reserving`, + `LossReporting` and `Observable` are deliberately still absent -- they belong to M31.2 and M31.4, and + shipping an empty trait now would be the design-in-a-vacuum D-3 forbids, one level up. + **The protocol refused a capacity of one, and that is reported rather than worked around.** With a + single slot, "published at `p`" and "free again at `p + capacity`" are the *same number*, so a producer + would read the sequence of the item it had just pushed, conclude the slot was free, and overwrite an + unread item. `spsc` accepts one, so the minimum is a property of the *shape*, not of the crate -- + `CapacityError` already carried a `max_valid` on exactly that argument and now carries a `min_valid` + too. Every workaround considered puts a load of the consumer's position back on the producer's hot path + for every queue, in order to serve a capacity of one that `spsc` already represents exactly. + [D-12](crates/windows-waitable-queues/DESIGN-NOTES.md#d-12). + **Two things were extracted rather than copied, and one of them is a contract.** The blocking receive + loop *is* the arming protocol (D-9), not glue around it, so a second spelling of it would have been a + second copy of a rule -- the exact mistake M30.5 already paid for, where a lost-wakeup proof exercised a + hand-written duplicate of `arm` and could not have noticed the real `arm` being reversed. It now lives + in [blocking.rs](crates/windows-waitable-queues/src/blocking.rs) with shapes binding to it, and the `ARM_RACE` hook is shared for the same reason + ([D-13](crates/windows-waitable-queues/DESIGN-NOTES.md#d-13)). The capacity rule moved to [capacity.rs](crates/windows-waitable-queues/src/capacity.rs) + on the weaker version of the same argument. + **One question the checklist did not anticipate: what "empty" means for arming.** `len` and "would `pop` + find something" disagree over a slot a producer has claimed but not published, and arming on `len` is + safe but spins until that producer is rescheduled. Arming asks the readiness question instead, which is + also what puts D-9's `SeqCst` pair on the right two locations for this shape + ([D-14](crates/windows-waitable-queues/DESIGN-NOTES.md#d-14)). + **This shape exposed a lost wakeup in the doorbell that `spsc` could not have found, and it was fixed + at the layer that owns it.** `Doorbell::clear` cleared its mirror flag and *then* reset the event; a + producer signalling between those two lines set the flag and issued a real `SetEvent`, and the + `ResetEvent` that followed erased the signal while leaving the flag set -- so the doorbell was dark + while claiming to be lit and every later signal skipped. The order had a written argument behind it + ("the caller's re-check sees the racing producer's item") that is **true for `spsc` and false for + `slotwise_mpsc`**, whose re-check asks only whether the *head* slot is published. The fix moves the guarantee + from the caller to the type: once `clear` returns the flag is false, so no future shape has to have a + re-check strong enough to cover the window. [D-15](crates/windows-waitable-queues/DESIGN-NOTES.md#d-15), + which amends D-9 rather than being filed beside it. + **It was found by the sabotage harness refusing to sweep against a red baseline** -- the baseline run, + whose only job is to prove the suite is green before any defect is injected, hung once in a suite that + passed 120 tests in 0.28s six runs running. A single unreproducible hang is the finding it is tempting + to blame on a busy machine. + 122 unit tests and 4 doctests, the whole suite in 0.30s. Twenty-three sabotages, all behaving as + declared: ten new ones for this milestone, two of them controls. **One of those controls earned its + keep immediately** -- it caught the new doorbell test asserting the signal-skip *optimisation* rather + than the contract, which is precisely what a control is for. + +- [x] **M31.2** -- Overflow policy, which is more than "return `Err`". Ship fail-fast plus a `reserve` + that guarantees a slot for a message that must not be lost, following + [queue.rs](crates/windows-file-watcher/src/queue.rs), which already carries three policies including a + **coalesced loss latch** the consumer is guaranteed to observe. **Never offer overwrite-oldest**: for + telemetry that is a lost sample, but for an I/O submission it is a lost operation, and the two must not + share a policy knob. + **Done, and the multi-producer case forced a decision the item did not anticipate.** Honouring a + reservation means knowing how many slots remain, which means reading the consumer's position -- one + line every thread touches -- on *every* push, including the pushes that never reserve anything. + `slotwise_mpsc`'s producer avoids that read by design: it asks the slot's own sequence "are you free", and those + are dispersed across the slot array. So `slotwise_mpsc` genuinely cannot answer the reservation question, and + rather than charge every caller for a capability not every caller wants, **`reserving_mpsc` ships as a + peer and `slotwise_mpsc` is untouched** ([D-16](crates/windows-waitable-queues/DESIGN-NOTES.md#d-16)). The + engineer chose this split over the alternatives when it was raised. + **The reservation count and the claim position share one 64-bit word, and that is the correctness + argument rather than tidiness** ([D-17](crates/windows-waitable-queues/DESIGN-NOTES.md#d-17)). With the + count in its own atomic, a pushing producer reads it then writes the position while a reserving one + writes it then reads the position, and each can miss the other -- granting a slot that does not exist. + **`SeqCst` fences do not close this**, unlike the superficially identical hazard in D-9: the Dekker + argument needs store-then-load on both sides and the pusher is load-then-store, so both sides missing + each other is consistent with every total order. The 32/32 split is forced by the arithmetic and caps + the shape at 2^31 items, reported through the same per-shape bound D-12 introduced for the minimum. + Two consequences worth noting: redeeming is a single exchange that moves both halves, so + `occupied + reserved` is never momentarily wrong; and the producer stops needing the slot sequence for + the "free" direction, so this shape's `pop` is one store *shorter* than `slotwise_mpsc`'s. + **A 128-bit compare-and-swap was raised, refused, and then adopted for a separate shape.** D-18 + refused it; **[D-37](crates/windows-waitable-queues/DESIGN-NOTES.md#d-37) supersedes that** -- read + D-37 rather than this paragraph, which has already restated a superseded version once. In short: + widening *this* shape's word would make its contract depend on the target, because + `i686-pc-windows-msvc` has no lock-free 128-bit exchange and the fallback is a silent global lock. + So the narrow shape keeps its 64-bit word on every target, and a wide claim ships as its own peer + where the exchange is genuinely lock-free. + **`spsc` reserves too, nearly free**, since one producer means `reserve` and `push` are the same + thread. Its reservation *borrows* the producer where `reserving_mpsc`'s is owned and `Send`, because + there the handle **is** the single-producer guarantee and an owned reservation could outlive it on + another thread. That difference is why `Reserving`'s associated type is generic over a lifetime, and it + is D-3 working: the trait was shaped by two implementations rather than around one. + **The loss latch is deliberately not generalised, and the reason is recorded rather than skipped** + ([D-19](crates/windows-waitable-queues/DESIGN-NOTES.md#d-19)). Coalescing works in the file watcher + because a desync is *idempotent*; a queue of arbitrary `T` has no such property. What generalises is a + loss *count*, which is M31.4's observability rather than an overflow policy. + **Two defects surfaced, both caught rather than reasoned about.** `Reservation::send` used + `mem::forget` to suppress its double-release, which leaks the `Arc` the reservation holds -- so the + shared state was never dropped and every item still in the ring leaked with it; found by the + drop-counting test. And the first `const` assertions guarding the packing were **tautological**, + asserting that `BOUNDS_MAX` equalled its own definition; widening the position to 40 bits sailed past + them while silently narrowing the count's field to 24, which is how the packing actually breaks. Both + rewritten, and the assertions verified by sabotage -- a too-wide and a too-narrow split each now fail + the build with the right message. + **And the sweep found a third, which is the one it exists to find: `spsc`'s reservation guarantee was + entirely untested.** Every reservation test had been written against `reserving_mpsc`, and because the + two implementations share *nothing* -- a plain counter against a packed compare-and-swap word -- + covering one left the other completely unguarded. A green suite cannot show that; only asking "would + these tests fail if the code were wrong" can. Writing the missing wakeup test then surfaced something + worth keeping: **the compiler refuses to move an `spsc` reservation to another thread at all**, because + it borrows a `!Sync` producer. That is the borrow doing its job, so it is now a `compile_fail` doctest + rather than a sentence -- itself verified by removing the attribute and confirming the error is the + `Send`/`Sync` one and not a typo. + 167 unit tests, 5 doctests and 1 `compile_fail` doctest, the whole suite in 0.31s. Thirty-one + sabotages, all behaving as declared: eight new ones for this milestone. + +- [x] **M31.3** -- Shutdown in both directions: the consumer learns when every producer is gone, and a + producer learns when the consumer is gone and fails with a typed error. Descriptors in flight at + teardown are **accounted, not dropped** -- some own handles, and their disposal must be allowed to + block, which is the hazard the namespace session flagged for undrained completions. + **The first two clauses were already shipped, and this was audited rather than assumed.** Every shape + has `is_disconnected` on both ends, `PushError::Disconnected(T)` hands the item back to a producer whose + consumer is gone, and M31.2 added `Disconnected` for a reservation redeemed into a dead queue. A + reservation also counts as a producer, so an outstanding promise holds the stream open. Nothing was + needed there; saying so is the point, since the alternative is checking a box on work done elsewhere. + **The third clause was the whole item, and the default was quietly bad.** Undrained items were destroyed + *in place*, inside the last `Arc` release -- so `T`'s destructor ran on whichever thread happened to drop + last. That thread is not knowable in advance and nobody chose it: it may be a pool callback that must not + block, and the namespace session's example is closing a handle to a dead network path, which is exactly + the blocking operation the facility exists to keep off a caller's thread. + **`Drop` cannot be made to hand them back** -- `&mut self`, no return, cannot fail, and by then every + handle is gone so there is nobody to return them *to*. That is why `Disposal` is supplied at + construction rather than requested at teardown: the last handle to drop is the only place that sees + every survivor, and it is the one place with no way to report + ([D-20](crates/windows-waitable-queues/DESIGN-NOTES.md#d-20)). The default is unchanged and still + destroys in place, because for items that own nothing that is exactly right -- what changed is that it + now has a name and an alternative. + **The claim under test is about threads, not counts.** Asserting only that the sink receives the items + would test the mechanism rather than the property, so the suite records the `ThreadId` a destructor runs + on and asserts it is *not* the thread that released the last handle -- with a control, without a sink, + showing that it is. Without that control the first test would look identical if destructors simply never + ran anywhere observable. + **Two smaller decisions recorded rather than left implicit.** A panicking sink is caught and the walk + continues ([D-21](crates/windows-waitable-queues/DESIGN-NOTES.md#d-21)), because a panic escaping a + destructor abandons every item behind it and aborts outright during an unwind. And `into_remaining` was + considered and refused ([D-22](crates/windows-waitable-queues/DESIGN-NOTES.md#d-22)): producers may push + after the consumer is consumed, so it would cover only the orderly path, and `drain` already does what + it would do. + Routing is asserted once per shape rather than once for the crate, since each walks its own layout -- + M31.2's sweep taught that lesson about the reservation guarantee, and this applies it before the sweep + had to teach it twice. + 194 unit tests, 7 doctests and 1 `compile_fail` doctest, the whole suite in 0.33s. Thirty-six sabotages, + all behaving as declared: five new ones for this milestone. + +- [x] **M31.4** -- Observability (R9): depth, high-water, and **a count of doorbells actually rung**. That + last one is what makes the skip rule measurable rather than assumed, and sabotage-verifiable -- disabling + the skip must move the number. + **The interesting thing about the three numbers is that they do not cost the same**, and each was placed + where it is already paid for. Refusals increment only on the failure path. Rings increment only when + `SetEvent` is actually called -- ~7 ns against a syscall measured at ~81 ns -- and the *skipped* signals + are deliberately not counted, because that increment would land on exactly the path the skip exists to + cheapen. Depth needed nothing new at all: `Bounded::len` already computes it from positions the queue + keeps anyway. + **High-water is the one that cannot be placed that way, and the cost is uneven in a way that lands on + D-16.** A peak must observe every change. On `spsc` that is free (the producer already reads `head` and + owns `tail`) and on `reserving_mpsc` near-free (its producer reads `head` for the room check), but + `slotwise_mpsc`'s producer **never reads `head`** -- that is the property D-16 built a separate shape to + preserve. Always-on would have imposed D-16's refused cost on every `slotwise_mpsc` user, to serve a metric most + will never read, immediately before M31.5 measures that exact path. Omitting it would have narrowed the + shape. So it is **opt-in at construction**, off by default, and `slotwise_mpsc` pays one predictable branch on a + read-only field when it is off ([D-23](crates/windows-waitable-queues/DESIGN-NOTES.md#d-23)). The + engineer chose this over the narrow-trait and always-on alternatives when it was raised. + Untracked reports `None` rather than `0`, because "nobody was counting" and "it never filled" are + different answers and only one of them should make a caller shrink a queue. + **Two independent switches across three shapes is why `Options` is now a builder**, replacing M31.3's + `bounded_with_disposal`. As constructors that is four per shape and twelve in the crate, with every + future switch doubling it. The crate is unreleased, so the replacement cost nothing. + **One consequence is worth naming because it inverts something already written down** + ([D-24](crates/windows-waitable-queues/DESIGN-NOTES.md#d-24)). [sabotage.json](crates/windows-waitable-queues/sabotage.json) carried a *control* that + removed the skip optimisation expecting `survives` -- and it had earned its place, by proving the suite + asserted the contract rather than the implementation. Counting the rings makes the skip observable, so + the same patch now has to be **caught**, and the entry changed sides. That is R9 working rather than a + regression: an optimisation nobody can measure is an assumption. What it costs is that the skip is now + part of what the queue promises, which is the right trade for a queue whose reason to exist is a wakeup + protocol -- but it is a trade. The vacated control is replaced rather than dropped, by `slotwise_mpsc`'s + tracking guard, which is genuinely an optimisation and must still survive removal. + **`Observable` deliberately does not restate depth** + ([D-25](crates/windows-waitable-queues/DESIGN-NOTES.md#d-25)), though D-2's sketch listed it: `len` + already reports it, and one number with two spellings is two places to drift. + 221 unit tests, 9 doctests and 1 `compile_fail` doctest, the whole suite in 0.30s. Thirty-nine + sabotages: three new, one converted from control to defect, and one new control replacing it. + +- [x] **M31.5** -- The contention benchmark that decides whether the deferred shapes are needed: N producer + threads pushing, throughput against N. **This is the item that either justifies or kills the linked and + sharded MPSC shapes**, and it is deliberately a measurement rather than a judgement, for the same reason + C-1 was. If the tail CAS does not contend at realistic producer counts, the array queue is the only MPSC + this crate ever needs. + + Record the result either way -- a measurement that says "the simple thing is fine" is worth as much as + one that does not, and is the cheaper outcome to lose track of. + + **Also measure `reserving_mpsc` against `slotwise_mpsc`, and decide their merge-or-delete here.** M31.2 shipped + them as two shapes because reservation costs the producer a read of the consumer's position on every + push, and *how much* that costs was a judgement rather than a measurement + ([D-16](crates/windows-waitable-queues/DESIGN-NOTES.md#d-16)). This benchmark already stands up N + producers against a tail, so measuring both under the same harness is nearly free. + The decision it forces: if the shared-line read turns out to be cheap at realistic contention, the two + shapes **merge** and the non-reserving one goes; if it is expensive, both stay and the split is + vindicated. This item exists because a duplicated path silently becoming permanent -- because nobody + circled back -- is the failure mode the duplicate-then-decide rule actually warns about, and an + intention recorded only in a design note is not scheduled work. + **Done, and both answers were surprises.** The probe is + [queue_contention.rs](crates/windows-platform-probes/src/queue_contention.rs), run by hand in release on + an AMD EPYC 7763 (8C/16T, x64), median of five with a discarded warm-up; three invocations agreed. + **Note the architecture -- every previous measurement in this workspace was ARM64**, so this fills the + x64 gap rather than extending the record, and M31.7 exists to close the other half. + **The tail claim contends, so the licence to close M-inf.1 was not granted** -- but the gate there is + now a number rather than a judgement, because contending and being the bottleneck are different things. + See M-inf.1 for the quantified trigger. + **`reserving_mpsc` is up to 4x FASTER than `slotwise_mpsc` under contention, which inverts D-16's premise** + ([D-26](crates/windows-waitable-queues/DESIGN-NOTES.md#d-26)). The split shipped on the reasoning that + reading the consumer's position made the reserving shape the expensive one; it is the cheaper one at + every producer count from two upward, and the premise survives only at a single producer against a live + consumer -- where `spsc` is the right answer anyway. + **Investigated before concluding, at the engineer's direction, and the gap is intrinsic rather than a + fixable flaw** ([D-27](crates/windows-waitable-queues/DESIGN-NOTES.md#d-27)). Both protocols do one CAS + plus one load per attempt; the difference is *which* load. `slotwise_mpsc` must read the slot's own sequence + before claiming -- an address that marches through memory as the tail advances, written by the producers + it is racing -- where `reserving_mpsc` reads one fixed `head`. The false-sharing hypothesis was tested + and rejected: padding each slot onto its own cache line recovers about a fifth at eight producers for + four times the memory, and leaves the shape 2.8x slower. The padding was reverted. + **A methodological trap worth keeping: a debug build reports the two shapes as identical** (249.7 vs + 254.0 ns at sixteen producers, against 193.5 vs 52.2 in release). That is why this probe is deliberately + *not* in the CI probe job, which runs debug -- it would produce a confident wrong answer rather than a + noisy one. + **The merge-or-delete decision is now live with data behind it and is queued as M31.8**, not taken + here: the investigation changed what the decision is *about*, from "is the extra read cheap" to "which + claim protocol should survive", and that is the engineer's call. + +- [x] **M31.7** -- Re-run `probe-queue-contention` on the ARM64 development machine and record the curve + beside the x64 one. **Not a formality.** M31.5's finding is a statement about cache-coherence + behaviour, and this workspace has already been bitten once by measuring only on ARM64 -- + [windows-platform-probes](crates/windows-platform-probes/DESIGN-NOTES.md) records that case. M31.5 + inverted a design premise on x64 evidence alone; if ARM64 disagrees, the merge decision in M31.8 changes + with it, and so does M-inf.1's threshold. + Run it in **release**: a debug build reports the two shapes as identical, which is why the probe is not + in CI. + **Done. Both of M31.5's claims hold on ARM64, and the reserving advantage is larger, not smaller.** + Host: Snapdragon X2 Elite (Qualcomm Oryon), 12 cores, no SMT, no L3, two L2 clusters of six. Release + build, median of three runs of the binary, isolated regime, ns/push: + + | producers | mpsc | reserving | atomic floor | mpsc/reserving | x64 ratio for comparison | + |---|---|---|---|---|---| + | 1 | 6.5 | 6.1 | 2.7 | 1.1x | 1.0x | + | 2 | 29.8 | 9.4 | 3.6 | 3.2x | 1.8x | + | 4 | 60.6 | 12.9 | 5.2 | 4.7x | 2.5x | + | 8 | 167.4 | 29.8 | 8.8 | 5.6x | 3.7x | + | 16 | 194.9 | 30.6 | 10.6 | 6.4x | 3.7x | + | 32 | 195.0 | 30.6 | 9.9 | 6.4x | 4.2x | + + Claim 1 (throughput falls as producers are added) holds: `slotwise_mpsc` costs 30x more per push at 32 + producers than at one. Claim 2 (`reserving_mpsc` is up to 4x faster) holds and is exceeded -- **6.4x + here against 4.2x on x64**. So M31.8's merge decision is not weakened by the second architecture; the + evidence for the head-based protocol is stronger on ARM64 than it was on x64. + Two differences worth having on the record rather than smoothing away. `slotwise_mpsc` **plateaus at ~195 ns + from 16 producers upward** where x64 kept climbing to 239.7 -- expected, since this host has 12 cores + and no SMT, so 16 and 32 are oversubscribed and the curve saturates. And **N=4 is by far the noisiest + point** (`slotwise_mpsc` ranged 49.5 to 104.1 across the three runs, against under 2% spread at N=16 and above); + with two six-core L2 clusters and no L3, whether four threads land inside one cluster or straddle both + changes the answer, and at N>=8 straddling is forced so the variance disappears. Read the N=4 row as a + range, not a point. + + > **-> CROSS-COMPONENT NOTE:** this run also contradicted D-28, which is recorded against that decision + > and against M31.8's use of it below, not here. + +- [x] **M31.8** -- **MIRRORED BY [CHECKLIST-ship-topology-and-queues.md](CHECKLIST-ship-topology-and-queues.md) + SH-1.1 -- one piece of work seen from two plans. Check both off in the same commit; neither is done + alone.** That file also records why this is *release*-blocking rather than merely design-blocking: + the decision may delete a public type, which is free before `windows-waitable-queues` 0.1.0 and a + yank-and-migrate after it. + Decide merge-or-delete for `slotwise_mpsc` and `reserving_mpsc`, now that M31.5 has measured + them and M31.7 will have checked the other architecture. + **The decision changed shape once the investigation ran.** M31.2 framed it as "if the shared-line read + is cheap, the two merge and the non-reserving one goes". The read is not merely cheap -- it is cheaper + than the read it replaces -- so the real question is **which claim protocol survives**: Vyukov's + sequence, which reads a marching slot, or the head-based one, which reads a fixed line. + The candidates, with what each costs: + - **Delete `slotwise_mpsc`, keep `reserving_mpsc`.** Simplest surface, and the faster shape under contention. + Loses the 2x advantage `slotwise_mpsc` holds at one producer with a live consumer, and lowers the maximum + capacity from 2^63 to 2^31 for every caller. + - **Keep both**, and correct their documentation, which currently states D-16's falsified premise as + the reason the split exists. The split would then be justified by *profile* -- one shape for few + producers, one for many -- which is a real distinction but a harder one to explain. + - **Change `slotwise_mpsc`'s protocol** to decide freedom from `head`, closing the gap. This makes the two + shapes genuinely "one queue with and without reservations", which is what D-16 assumed they already + were, and is the only option that removes the surprise rather than documenting it. + Whichever is chosen, D-16's and `slotwise_mpsc`'s own documentation must be corrected in the same change: they + currently assert a cost relationship the measurement reversed. That sweep is part of this item. + **An input that was written off has come back, and this paragraph previously said the opposite.** + Peer-index caching is available to the head-based protocol and structurally unavailable to Vyukov's. + This item used to record that `probe-peer-index-cache` had measured it as making our ring *slower* + (D-28), and instructed that it "must not be argued as" a differentiator. **That instruction was based + on x64 evidence alone, and ARM64 reverses it**: the same binary measures caching at **17x faster** + there (31.2 -> 1.8 ns/item), with the mechanism D-28 itself names -- batch depth -- coming out at ~150 + items per shared read instead of the ~3.6 that made it lose on x64. See D-28, now amended. + So this **is** live as a differentiator, and it points the same way M31.7's contention curve does: it + is an optimisation only the head-based protocol can adopt, and on one of our two architectures it is + worth an order of magnitude. Do not resolve M31.8 by reinstating the old "it does not matter" line. + What it is *not* is settled. The technique wins on one host and loses on the other, so adopting it + unconditionally is as unsupported as rejecting it was. The decision this item owes is about the + protocol; whether any shape then *adopts* caching is a separate question that needs a policy for a + measurement that inverts by host, and that question is M-inf.4 rather than this item. + +- [ ] **M31.6** -- **GOVERNED BY [CHECKLIST-ship-topology-and-queues.md](CHECKLIST-ship-topology-and-queues.md) + SH-1.2, which decides only whether this blocks the 0.1.0 release. SH-1.2 completing does NOT complete + this item**; it records an answer here. + **The answer, recorded 2026-08-31: this does NOT gate `windows-waitable-queues` 0.1.0. It gates + 1.0**, and the crate ships 0.1.0 disclosing the gap in its own documentation rather than leaving an + adopter to find it. See D-31. This item stays open, and the disclosure is a promise it now carries. + + **Its scope is corrected by the same decision, and this is the part worth reading before starting.** + Loom models atomics; it cannot model `SetEvent`/`ResetEvent`. So it covers the three queue shapes' + head/tail/sequence orderings -- which *is* where the demonstrated blind spot lives -- and it does + **not** cover the doorbell, whose correctness is precisely the interleaving of an `AtomicBool` mirror + with those syscalls. Stubbing them would verify a model of `SetEvent` rather than `SetEvent`, which + is the "measures the model, not the thing" trap this workspace has already been caught by once. + **D-15's lost wakeup, the only ordering bug this crate has actually had, was found by sabotage and + loom would not have found it.** Do not let completing this item be read as "the orderings are now + verified": the doorbell needs a separate answer, and this item does not supply it. + Verify the memory orderings with a model checker, because stress testing demonstrably + cannot. **Measured, not assumed:** during M30.3's sabotage sweep, weakening the producer's `Acquire` + load of `head` to `Relaxed` left all twenty tests green, while every *logic* defect injected alongside + it was caught. A stress test can only observe the interleavings the hardware and scheduler happen to + produce, and neither ARM64 nor x86-64 will produce the reordering that makes a missing acquire visible + just because a test asks nicely. + `loom` is the tool: it enumerates interleavings under a weak-memory model rather than sampling them, so + a missing `Acquire`/`Release` pair becomes a deterministic failure. It is a dev-dependency and a + `cfg(loom)` shim over the atomics, so it costs the shipped crate nothing. + **Sabotage-verify the verifier**, exactly as here: the loom test is only worth its weight if + reintroducing that same `Relaxed` makes it fail. If it does not, the model is not covering the path + and the test is decoration. + Scope it to the orderings, not the logic -- loom explores exponentially, so a loom test that also + checks FIFO order over a thousand items will not terminate. + **A second, sharper target arrived from the M30.4/M30.5 code review, and it is the more important + one.** The doorbell carries two `SeqCst` fences -- before the loads in `Doorbell::signal`, after the + stores in `Doorbell::clear` -- which defeat a store-buffer (Dekker) reordering between the producer's + decision to skip signalling and the consumer's emptiness check. Without them the item is queued, no + signal is raised, and the consumer parks forever. **Removing either fence leaves the entire suite + green**, and no sabotage can express it, because the defect is a fact about the memory model rather + than an interleaving a scheduler can be coaxed into producing. Both fences must therefore be loom's + first two subjects, and the test earns its place only if deleting each one makes it fail. + Note that loom must model the doorbell's `OnceLock` publication too, since the lazy-creation path is + one of the two sides of the hazard; a loom test that only models the steady state will pass with the + `signal` fence removed and prove nothing about the case that motivated it. + +## M32 -- Contracts the runtime cannot be written without + +These are decision items, not implementation. Each is open in the session record, and each would change +the runtime's shape, so they land before M33+ begins. (The heading said "all three" while listing four; +it now lists five, and the count is dropped rather than maintained.) + +- [ ] **M32.1** -- **The ordering guarantee.** Open since the 2026-08-27 namespace session, which + observed that `DeleteFile(X)` then `CreateFile(X)` on a pool does not execute in order and said the + contract "must state this explicitly rather than let it fall out of the implementation". A + single-consumer SQ gives per-domain FIFO *for free* -- the question is whether it is **promised**. + Promising it constrains every future implementation; withholding it makes composition harder for a + client that has an ordering requirement and no other way to express one. Decide, and state the + guarantee in the queue's own documentation rather than leaving it as an artifact. + +- [ ] **M32.2** -- **Correlation.** Who mints the tag that joins a submission to its completion, and how + it survives the two-layer translation into the ring's own `user_data`. Constraints already established: + `IoRing` mints `user_data` starting at **0** on a fresh ring, and `Token::claim_if` requires both + `user_data` **and** `RingId` to match. The client-facing tag is therefore not the ring's tag, and the + mapping between them is state the domain owns. + +- [ ] **M32.3** -- **Backpressure behaviour.** R2 says a full queue fails, and that failure is the + backpressure. But a client with nowhere to go either spins or drops, so decide whether a blocking submit + exists -- and if it does, **what it blocks on**, because a blocking submit that cannot be composed into + a `WaitForMultipleObjects` reintroduces exactly the wait-composition problem that ruled out crossbeam. + +- [ ] **M32.4** -- Transcribe the session's converged decisions from Tier 3 into Tier 1, and record the + ones this checklist rests on in [DESIGN-NOTES.md](DESIGN-NOTES.md): the uniform tunable architecture, + report-don't-route, the domain runtime not being a thread pool, the rejection of round-robin, and the + two-layer ring. **A decision recorded only in a session record steers nothing**, and this checklist is + the mechanism that makes them binding. + +- [ ] **M32.5** -- **Note that the shard plan is *not* one of these contracts, and where it went.** + M33+.1 opens with "one pinned thread, its `IoRing`, its node-local registered pool, its shard", + which presupposes a mapping naming which thread, which node and which shard -- and that mapping was + unowned: M32's other four contracts are all about the queue, and no item anywhere computed the plan. + It is now [crates/topology-planner](crates/topology-planner/COMPONENT.md), a component + of its own, because it applies **policy** over the topology's facts and reasonable clients will + choose differently. Nothing here needs to decide it; this item exists so a reader of M33+ does not + conclude the mapping is obvious, which is how it went missing. + +> **-> CROSS-COMPONENT HANDOFF:** M33+ below spans `crates/windows-thread-ambient-sys`, +> `crates/windows-namespace-request-sys`, and `crates/windows-ioring-sys`. Each has its own +> [CHECKLIST.md](CHECKLIST.md); the items are held here until M32 settles, then move to the component that owns them. +> +> **The plan M33+ executes comes from +> [crates/topology-planner/CHECKLIST.md](crates/topology-planner/CHECKLIST.md)**, which is +> itself gated on the locality-model design session. So M33+ has two prerequisites, not one. + +## M33+ -- The domain runtime (gated on M32) + +Parked, not pending. Shape recorded so it is not lost, per the `M{n}+` convention. + +- [ ] **M33+.1** -- The domain: one pinned thread, its `IoRing`, its node-local registered pool, its shard. + N=1 first and complete on its own; N>1 adds routing and a cross-domain queue without disturbing it. + +- [ ] **M33+.2** -- The thread builder, into + [windows-thread-ambient-sys](crates/windows-thread-ambient-sys/README.md): construct a thread with + `PROC_THREAD_ATTRIBUTE_GROUP_AFFINITY` set **at creation**, because a stack is allocated then and + binding afterwards cannot move it. Plus `bind_current_thread` with a restore guard for threads the + client did not create. **Its principal justification is unverified** -- see + [thread-stack-numa-spike.rs](crates/windows-ioring-sys/design-sessions/spikes/thread-stack-numa-spike.rs), + which is written and smoke-tested but needs multi-node hardware. If it comes back showing creation-time + affinity does *not* govern stack placement, this item shrinks to the binder alone. + +- [ ] **M33+.3** -- Extend [windows-namespace-request-sys](crates/windows-namespace-request-sys/README.md) + so an `Outcome` can carry the volume-node hint and its provenance alongside the handle, which is the + "report, don't route" primitive. + +- [ ] **M33+.4** -- The `threadpool` feature: a client-side helper for multiplexing more CQ doorbells than + `WaitForMultipleObjects` accepts, using `ThreadpoolWait` (kernel-side wait completion packets, so wide + waits cost the dispatch hop rather than a thread per 64). **Default-off and at the edge** -- a domain + waits on three handles and never approaches the limit, so the dependency belongs to whoever multiplexes. + +- [ ] **M33+.5** -- The durability layer as its own crate: **composition with shared vocabulary, not + derivation.** It contains a domain and submits through it; it re-exports `Op` and `Completion` where the + concept is genuinely the same, and defines `Epoch` and its own commit types where it adds meaning. + Carry one constraint from the start: the flush barrier stops at the ring's edge, so **an epoch is + per-domain** and a client spanning two domains needs two flushes and an explicit join. + +## M-inf -- Ungated + +- [ ] **M-inf.1** -- The linked and sharded MPSC shapes, if and only if M31.5 shows the array queue's tail + CAS contends at realistic producer counts. + **M31.5 has run, and the gate is now quantified rather than open.** The tail claim *does* contend, on + x64: aggregate throughput falls with every producer added, `slotwise_mpsc` from 111M to 4.2M pushes/sec and + `reserving_mpsc` from 116M to 17.6M, against a bare contended atomic that falls only to a third. So the + licence M31.5 offered to close this item outright -- "if the tail CAS does not contend, the array queue + is the only MPSC this crate ever needs" -- was **not** granted. + **But contending is not the same as being the bottleneck, and this stays parked on that distinction.** + At eight producers `reserving_mpsc` still sustains ~26M pushes/sec, or ~39 ns per push. A sharded queue + is worth building only for a consumer whose per-item work is *smaller* than the contention it would + remove, and the I/O domain this crate was written for is nowhere near that: C-1 already established + that a real request dwarfs the queue's mechanics. + So the trigger is now a number rather than a judgement: **build these when a consumer appears whose + per-item cost is on the order of the ~39 ns/push (8 producers) or ~57 ns/push (32 producers) that the + array queue's claim costs under contention.** Until then a sharded queue would optimise the small half. + Re-measuring on ARM64 is the cheap way to find out whether that threshold moves; see M31.7. + +- [ ] **M-inf.2** -- The eventcount, if and only if a measurement against real I/O shows the doorbell + costs enough to be worth its lost-wakeup risk. C-1 showed batching alone drives it below the atomic push + it accompanies, so nothing currently justifies it. + +- [ ] **M-inf.3** -- An allocation-model change to `PreparedPath` (inline storage or request recycling). + Bounded before anyone builds it: `prepare` is dominated by `GetFullPathNameW`, a Win32 call no allocator + removes, and cloning already-prepared units is 95 ns of a 453 ns request. That 95 ns is the ceiling on + the win, and only for a caller that can reuse a resolved path. + +- [ ] **M-inf.5** -- **Domain-local queue placement**, now that the cost of getting it wrong is measured. + `probe-core-affinity` finds an SPSC handoff costs **38.5 ns/item within a domain and 215.3 ns/item + across domains on the ARM64 host -- 5.6x for nothing but where the two threads run**. That is far + larger than any micro-optimisation this crate has considered, and it is a *placement* decision rather + than a code one, which puts it squarely in the runtime's remit rather than the queue's. + **This item's premise -- that a domain is a set of interchangeable processors -- is itself untested, + and is now queued.** Every measurement behind the 5.6x pins to a *single* processor (`mask = 1 << + cpu`), so "place the thread in the domain" has only ever been evaluated as "place the thread on one + chosen member of the domain". A set mask permits placements a single-processor mask forbids, + including both ends of a queue on one logical processor, which on an SMT host is the *common* case + for a same-cache set rather than a corner one. + **FED BY [CHECKLIST-placement-tool.md](CHECKLIST-placement-tool.md) M6.7, which does not check this + item off** -- this item is the domain-local placement work itself -- **but whose answer must be + written in here when it lands.** M6 measures set-wide against pinned affinity under two interference + models and reports migration and co-residency counts, so a null result can be distinguished from a + scheduler that never moved anything. **If the sets are not equivalent, + the 5.6x is a number about pinned threads and this item needs restating**, not merely re-measuring. + + The design already intends one pinned thread per domain, so the queue between two threads of the same + domain is the common case and is fine. What this measurement bounds is the **cross-domain** queue -- + the one M30's design deferred on the grounds that N=1 does not need it -- and the number to carry into + that decision is 5.6x, not zero. + Gated on the domain runtime existing (M33+.1), not on more measurement. + **Do not read the 5.6x as a cache effect or as a core-speed effect.** On the ARM64 machine the + efficiency classes and cache domains coincide exactly, so the two are perfectly confounded. + **We now do have a host that separates them.** The x64 host has one L3 domain and one efficiency + class across eight L2 domains, so its `cross cache, same class` row varies the cache domain alone, + with class, package, L3 and NUMA held constant: the isolated cache-crossing cost there is + **1.8x - 2.0x** on the unoptimised handoff. That is not the same 5.6x boundary -- it is a shallower + crossing (L2 inside a shared L3, not a cluster-to-cluster hop) -- so it bounds rather than + decomposes the ARM64 number. Do not subtract one from the other. + +- [ ] **M-inf.4** -- Peer-index caching in the head-based shapes, and more importantly **a policy for an + optimisation whose sign depends on the host.** Gated on that policy, not on more measurement -- we + already have the measurement, twice, and it disagrees with itself. + D-28 rejected the technique on x64, where the producer and consumer stayed lock-step at a batch depth + near 1 and caching cost ~1.8x. M31.7 re-ran the same binary on ARM64 and got a batch depth around 150 + and a **17x speedup**. Both are real; the variable is how the two threads interleave, which is a + property of the host (core count, SMT, cluster layout, scheduler placement) rather than of our code. + So the question this item owes is not "is it faster" but **what do we ship when a technique is a large + win on one supported machine and a loss on another.** The candidates, none of them free: + - **Ship it off**, as today. Costs ARM64 an order of magnitude on a shape that could have it. + - **Ship it on.** Costs x64 roughly 1.8x on the same shape. + - **Adapt at run time** from an observed batch depth, which is the only option that could win on both + and is also the only one that puts a heuristic in the push path -- and a mispredicting heuristic is + worse than either fixed choice. + - **Make it a construction-time option**, pushing the decision to a caller who may know their + producer/consumer coupling better than we do, at the cost of a knob nobody can set well without + running the probe themselves. + Whichever is chosen, it must be stated as a *policy* the crate owns rather than as a fact about a + processor -- see PLATFORM INTEGRITY: this is exactly a lower baseline that must not be quietly dropped + because the machine on the desk today prefers the other answer. + **Placement was tested on both hosts, and the picture is now complete.** `probe-core-affinity` was + written to check whether the host difference was really a *placement* difference. On ARM64 alone it + looked eliminated: caching wins at **both** placements there (14.4x within a domain, 3.0x across), and + threads placed together batch ~135x *deeper* than threads placed apart. Running it on x64 changed the + answer -- **the verdict flips inside that single machine**: pinned to SMT siblings, caching WINS 1.8x + at a batch depth of 116-163; pinned across cores it LOSES 2.0x at a depth of 1.7. Unpinned threads + land across cores, which is precisely the losing row, so D-28's original result was one placement + reported as though it were the machine. + The unified rule both hosts obey: caching wins when `(cost of the shared read) x (reads saved)` + exceeds the cost of idling on a stale bound. **Both terms are placement-dependent, which is why one + term alone never explained it.** ARM64 wins even at depth ~0.4 because the read it saves is genuinely + expensive (215 ns baseline); x64 loses at a similar depth because its cross-core read is cheap + (19-21 ns -- crossing L2 while staying inside one L3, one package, one NUMA node, one efficiency + class). So the sign is predictable from the two terms, and this item's policy question is unchanged + but now better posed: **the knob is placement, not architecture**, and any policy keyed to the + instruction set would be keyed to the wrong variable. + **What the x64 host contributed that ARM64 could not.** ARM64's cache domains and efficiency classes + are perfectly confounded (see M-inf.3's caution at the top of this section). The x64 host has one L3 + domain, one efficiency class, and eight L2 domains, so its `cross cache, same class` row varies only + the cache domain -- the isolated cache-crossing cost is **1.8x - 2.0x**. It cannot express + `same cache, same class` at all, because its outermost partitioning cache is L2 and is shared by + exactly the two siblings of one core. The two hosts are complementary; neither alone produces the + full table, and M-inf.3's "we do not have such a host" caution should be read against that. + + **Coverage, stated explicitly, because the two hosts turn out to be disjoint rather than + overlapping.** Not one placement is measured by both machines, so no row is a cross-check of another + and every row rests on a single host: + + | placement | ARM64 (Snapdragon X2) | x64 (EPYC 7763 slice) | measured by | + |---|---|---|---| + | SMT siblings (one core) | not expressible (no SMT) | **yes** | x64 only | + | same cache, same class | **yes** | not expressible (L2 is per-core-pair) | ARM64 only | + | same cache, cross class | not expressible (confounded) | not expressible (one class) | **neither** | + | cross cache, same class | not expressible (confounded) | **yes** | x64 only | + | cross cache, cross class | **yes** | not expressible (one class) | ARM64 only | + | cross NUMA node | not expressible (one node) | not expressible (one node) | **neither** | + + A machine cannot express a placement when its topology makes the pair impossible: ARM64 confounds + cache domain with efficiency class (crossing one crosses the other), and the x64 slice has exactly + one efficiency class and one L3 with L2 shared only by SMT siblings. + + `same cache, cross class` -- two *different* cores sharing a cache domain but differing in class -- + **is unmeasurable on either host, and no re-run of either will produce it.** It needs heterogeneous + cores inside one cache domain. + + **`cross NUMA node` is the other unmeasured row, and the probe was silently unable to report it + until now.** `ProcessorPlace` carried core, class and cache domain but *not* the NUMA node, and + `Placement` had no node dimension, so a cross-node pair would have been bucketed under a cache label + with nothing in the output saying so -- and `representative_pairs` picks whichever pair it enumerates + first, so which one you got would not have been reproducible either. On a scarce NUMA machine that + would have produced a large number attributed to the wrong cause. Fixed ahead of the machine rather + than after it: `numa_node` is now carried, `Placement::CrossNumaNode` is classified *first* (crossing + a node dominates cache and class, exactly as sharing a core does), and six tests cover it including + the precedence cases. Verified by sabotage -- removing the check fails four of them. + This is the same defect class as the omitted `SMT siblings` row, and the third time in this + investigation that an instrument's *classification or presentation*, rather than its measurement, + was the thing about to produce a wrong answer. + + **The node-crossing path is validated against synthetic multi-socket topologies, offline.** + `classify` and `representative_pairs` are pure functions of a processor list, so a mocked list + exercises them exactly as real hardware would; five fixtures cover a two-socket host with several + cache domains per node, one with a single cache domain per node, a no-SMT server, and a four-node + host. Each asserts not merely that the expected rows appear but that **the pair chosen for each row + actually satisfies that row's predicate** -- a table with right labels and wrong pairs behind them + is worse than a missing row. All nine node-related tests fail when the classifier's node check is + removed. The *timings* are deliberately not mocked and cannot be: `measure` pins to real + processors, and pinning to one that does not exist fails loudly rather than fabricating a number. + + **Inter-node distance is measured per hop, not collapsed into one row.** `CrossNumaNode` is a single + placement however many nodes exist, so on a host with three or more it would report one hop and + imply the rest were like it -- and which hop you got would depend on enumeration order. Real + multi-node hardware is not equidistant: two nodes on one package are far closer than two across a + socket link. `node_pairs` therefore selects one representative processor pair per *distinct* node + pair, and `measure` reports each hop separately in `by_node_pair`. + **Corrected 2026-09-02:** this said the selection was "keyed `(low, high)` so a link is measured + once rather than once per direction", which the code has not done for some time -- it keys + `(producer.numa_node, consumer.numa_node)` and its comment states that "both *directions* are + kept", with `by_node_pair` adding that "each hop is measured once per ring placement, so there are + two". Four measurements per undirected edge, not one. Found while stating + [EP-D-3](crates/topology-planner/DESIGN-NOTES.md#ep-d-3), whose whole subject is that + residency is directional, so a parked item asserting the opposite would have been read as evidence + against it. The probe prints the resulting table, names the + cheapest and dearest hop, and says outright whether the spread is small enough for the single + `cross NUMA node` row to be a fair summary. + **These are measured hops, not a firmware distance matrix.** Windows exposes no NUMA distance table + -- there is no Win32 equivalent of reading ACPI SLIT -- so measuring the handoff is the only way to + learn that two nodes are further apart than another two. Seven tests cover the selection on + synthetic 1-, 2-, 3-, 4- and 8-node hosts, including that the hop count is the triangular number of + the node count, that selection is stable across calls, and that non-zero-based node ids still work; + all five relevant ones fail when the canonical-ordering guard is broken. On the single-node hosts we + have, the section prints nothing rather than an empty table. + + **A prediction that will otherwise look like a bug on the real run: on a multi-socket host, + `cross cache, same class` may be absent entirely.** `cache_domain` is defined as the outermost cache + level that *partitions the machine*, so its meaning moves with the host. On the single-socket EPYC + slice that level is L2, and the row measures an L2 crossing inside one L3. On a two-socket box whose + last-level cache is per-socket, that level becomes the socket -- so two cores either share the cache + domain (same node) or sit on different nodes, the node check claims the pair first, and the + cross-cache row has no members. A synthetic fixture pins this down. **Read that absence as the + topology speaking, not as a defect**, and note the corollary: the EPYC slice's isolated 1.8x - 2.0x + cache-crossing number may have no counterpart on a multi-socket host at all. + + **A third host is planned -- an Intel cloud dev box -- and it should be expected to add no new rows.** + An earlier revision of this item predicted it would express four placements at once, on the + assumption of a *hybrid client* part (P-cores with SMT, E-cores without). That assumption is wrong + for a cloud VM: cloud Intel means Xeon, which has no efficiency cores, so `ec[...]` will almost + certainly read as a single class exactly like the EPYC slice, and the two class-crossing rows stay + inexpressible. + **All three hosts are VM slices, and a slice flattens topology.** The EPYC slice is the proof + already in hand: a 7763 is 64 cores across eight CCXs each with its own L3, and 16 of those cores + would span two of them -- yet `probe-topology` reports `L3[16]`, a single domain, and a single NUMA + node. The hypervisor presented a flat view. **So the missing rows are not merely unmeasured, they + are probably unreachable from any dev-box-sized VM slice**, and expecting a third slice to supply + them would repeat the error of expecting a third architecture to. + The Intel slice is still worth running, for a narrower and more honest reason: **it tests whether + the SMT-sibling result reproduces on Intel Hyper-Threading rather than AMD SMT.** That row currently + rests on one machine and one vendor's implementation of the feature, and it is the row carrying the + claim that sharing L1 produces deep batches. A second SMT vendor either strengthens it or breaks it. + Run `probe-topology` first regardless: whether the outermost partitioning cache is L2 or L3 decides + which rows exist at all, and on a VM slice it is not predictable from the part number. + **What would actually add rows**, if either becomes available: bare metal for `same cache, same + class` and `same cache, cross class`, or a deliberately large multi-NUMA VM SKU (not a dev box) for + a genuine node crossing. See the NUMA gap recorded below before spending time on the latter. + + **A two-socket Sapphire Rapids host is expected to become available, and it fills nearly the whole + table at once.** Sockets give a genuine node crossing and a real `cross cache, same class` row at the + L3 level; SMT gives the sibling row; two cores within a socket give `same cache, same class`. If + **Sub-NUMA Clustering** is enabled it subdivides each socket, so the machine may present four or + eight nodes -- and that would be the first host on which the node-pair matrix shows *variation* + rather than one hop, because an intra-socket SNC hop and a cross-socket hop are not the same + distance. That matrix was built for exactly this case and has never met a machine that can populate + it. + Three things to carry into that run: + - **It still cannot produce `same cache, cross class`.** The cores are homogeneous, so that row + stays unmeasurable on every host we have access to. + - **It is x86-64, so it is TSO.** It will expose weakened memory orderings no better than the EPYC + slice did, and a clean run there must not be read as ordering validation. ARM64 remains the more + revealing host for that, and per D-31 neither substitutes for a model checker. + - **It will present multiple processor groups**, which the tooling does not yet handle and would + silently collapse rather than refuse. That is + [CHECKLIST-placement-tool.md](CHECKLIST-placement-tool.md) M1B, and it must land before the machine + is used. + **Ask for `probe-topology` output first**, before any long run. Whether SNC is on, how many groups + the host presents, and where its partitioning cache sits decide what everything else means, and none + of the three is knowable from the part number. diff --git a/CHECKLIST-placement-tool.md b/CHECKLIST-placement-tool.md new file mode 100644 index 00000000..6a5169ad --- /dev/null +++ b/CHECKLIST-placement-tool.md @@ -0,0 +1,745 @@ +# Checklist: a shareable placement-cost tool + +**Goal.** A small, publishable Windows tool that a stranger can install, run once, and send back a +single structured result -- so that this workspace can collect placement and NUMA-hop measurements +from hardware it does not own. **The motivating gap is concrete: every host available here has exactly +one NUMA node**, so the entire `cross NUMA node` row and the whole inter-node hop matrix are +unmeasured, and no amount of local work will change that. + +**The gate applies to crates.io publication only, and not to the GitHub binaries.** An earlier revision +of this paragraph gated the whole file on SH-4.1 and SH-4.3, which was wrong and would have delayed the +tool by the length of the entire release sequence -- including M6's stress work -- for no reason. + +- **CI-built binaries are compiled from this repository**, so the tool's dependencies resolve through + `path` and nothing has to exist on crates.io. **PT-5.1 is therefore not gated at all**, and it is the + distribution that matters: the download is the provenance, per PT-3.2. +- **GATED BY [CHECKLIST-ship-topology-and-queues.md](CHECKLIST-ship-topology-and-queues.md) SH-4.1 + (topology 0.2.0) and SH-4.3 (queues 0.1.0): PT-5.3 only**, publishing the tool to crates.io, where a + path dependency needs a real published version behind it. **This bullet is the gate of record; when + those land, edit it to say so and name the two versions.** A gate that has silently lifted is as + harmful as one that has not. + +M1B and M6 are outside all of this and say so where they are defined. + +**Gated on shipping [crates/windows-topology-sys](crates/windows-topology-sys) and +[crates/windows-waitable-queues](crates/windows-waitable-queues) first.** Not a preference: the tool +depends on the former, and calibrates against the latter's `spsc`. Both are `0.1.0` and the topology +crate now carries an unreleased breaking change (`feat(topology)!`), so it wants a release before +anything downstream is published against it. + +**Why a new crate rather than publishing the existing probes.** +[crates/windows-platform-probes](crates/windows-platform-probes) is `publish = false`, `version = +0.0.0`, and every binary opens by saying it is "an experiment, not a component". That boundary is +deliberate and stays. It also carries ~13 probes irrelevant to this question, which would be public +surface and a maintenance obligation for no benefit. + +Related: [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M-inf.4 and M-inf.5, both of which are +waiting on numbers only other people's machines can produce. + +## M1: decisions that shape everything after + +- [x] **PT-1.1** -- **Name the crate**, and record the reasoning. It measures what a producer/consumer + handoff costs as a function of where the two threads run, which is broader than queues and narrower + than "topology". Candidates to weigh rather than a foregone answer: `windows-placement-probe`, + `windows-handoff-cost`, `windows-locality-report`. Check availability on crates.io before settling. + **Named `windows-placement-probe`.** It says what the thing is (a probe, not a library to build on) + and what it measures (placement), and it matches the `probe-` binary naming already in this + workspace. `windows-handoff-cost` was rejected as too narrow -- the tool already reports topology and + NUMA hops, which are not handoffs -- and `windows-locality-report` as understating that it *measures* + rather than summarises. All five candidates were confirmed free on crates.io before choosing. + Created as a workspace member with `publish = false`, because PT-5.3 has not decided crates.io yet + and `false` is the setting that cannot publish something by accident. + +- [x] **PT-1.2** -- **Decide what the submission record carries about the machine beyond the + fingerprint**, specifically the CPU model name. The fingerprint deliberately omits model names + because "a fingerprint that changes when the answer does not is a fingerprint nobody can compare" -- + correct for comparing placements, and a real loss when a stranger sends a result you cannot ask + follow-up questions about. + **This had to be settled before the first submission arrives, because the asymmetry is brutal.** A + record cannot be regenerated: a field the tool did not collect is missing *permanently* from every + result gathered before the omission was noticed, and the machines are other people's. + Under-collecting is unrecoverable; over-collecting is a privacy cost that can at least be corrected + going forward by collecting less. + + **Decided: collect the CPU model, the OS build, and a virtualisation hint. The canonical fingerprint + string stays clean; all three live in the record beside it.** + + The reasoning, in the order it actually holds: + - **A CPU model is not personal data.** It is a hardware characteristic shared by millions of + machines. The things that would be sensitive -- hostname, user name, file paths, domain membership, + serial numbers, installed software -- are not collected and must not be. That is the primary + argument; it stands whether or not the model could be inferred. + - **Withholding it gains nothing anyway**, because a detailed topology plus cache geometry narrows + the field to a small class of parts. This is the supporting argument, and it is deliberately *not* + treated as a principle: "it could be inferred, so collect it" would justify almost anything, and + the test remains whether the field is sensitive on its own merits. + + **Two fields ride along by the same reasoning, and both are more explanatory for this dataset than + the model is:** + - **The OS build.** Placement cost is a scheduler behaviour, and the scheduler changes between + Windows builds. Two results that disagree are otherwise indistinguishable from two builds + disagreeing, and that is unrecoverable after the fact. + - **A virtualisation hint.** This workspace has already established that **VM slices flatten + topology** -- the EPYC slice reports one L3 domain and one NUMA node for silicon that has eight and + two -- which is precisely why the interesting rows are unmeasured here. Being able to separate bare + metal from VM submissions is therefore not incidental: it is the distinction that decides whether a + submission can supply the missing rows at all. **Record it as a hint and label it as one**; + hypervisor detection is not reliably decidable from user mode, and a field that overstates its + confidence is worse than an absent one. + + **The runner can suppress the model**, with a flag, and the tool says so where it lists what it + collects. Not because the field is sensitive in general, but because the one case where it might be + is real and narrow -- an engineering sample or unreleased part would leak a name that is not yet + public -- and because "here is what I collect, and you may turn this off" is a materially stronger + thing to say to someone doing a favour than "trust me". The field is optional in the record, so a + suppressed submission stays valid rather than becoming unparseable. + **Suppression is recorded, not merely absent.** A field that is missing because the runner withheld + it and a field that is missing because the host would not answer are different facts, and a + collector that cannot tell them apart will eventually read one as the other -- the same reason an + inexpressible placement is reported rather than skipped. + + **And the flag must not be oversold, which is the more important half.** For the engineering-sample + case it addresses the *smaller* leak. A pre-release part is identified at least as well by its + **topology** -- an unusual core count, a novel cache arrangement, an unreleased NUMA layout -- and + the topology is the entire point of the submission, so it cannot be suppressed without making the + record worthless. **The tool therefore cannot make an NDA-covered machine safe to submit from, and + must not imply that it can.** Say so plainly in the README: if the hardware is confidential, the + whole output describes it, and the right answer is not to send it. That is worth more to the + audience most likely to own a multi-socket machine than any reassurance would be. + +- [x] **PT-1.3** -- **Decide the fate of the three existing probe binaries** (`probe-topology`, + `probe-core-affinity`, `probe-peer-index-cache`) once their modules move. Keeping them as thin + wrappers preserves the internal workflow; deleting them removes a second way to run the same + measurement and a second place for output to drift. **Do not decide by taste -- the risk being + weighed is two renderings of one measurement disagreeing**, which this investigation has already hit + three times. + **Decided: keep them, and move the *rendering* into the library so there is only one of it.** The + two stated worries turn out not to be in tension, because they are about different things. The + engineer's -- that a combined binary accretes flags and modes until it is the grab-bag this crate was + extracted from -- is about **entry points**. The drift worry is about **renderings**. Sharing the + render code kills the drift risk outright, after which extra entry points cost nothing. + So: the shared tool is **one binary, one run, one record**, because a stranger doing a favour must + not be asked to run three things and collate them. The internal probes stay **separate and thin**, + because running one measurement in isolation is the whole point of a development loop. Every binary + becomes an entry point only; measurement *and* rendering live in the library and are called, never + reimplemented. A binary that formats its own output is the defect, not a binary that exists. + +## M1B: processor groups, before a large machine is ever offered + +**Executes after M2, despite the number.** The code it changes lives in +[crates/windows-platform-probes](crates/windows-platform-probes) until the move, and doing this work in +its final home keeps the move a pure relocation with its provenance trail intact. The "before" in the +heading is about the *machine*, not about M2. + +**This is the blocker that would waste the opportunity.** A large multi-socket host -- the kind that is +the entire point of this tool -- has more than 64 logical processors, so Windows presents it as +**multiple processor groups**, each numbering from zero. + +- [x] **PT-1B.1** -- **Carry `(group, number)` as a processor's identity.** `ProcessorPlace` keys on a + bare `u8` number and `places_from_topology` discards the group outright (`for (_group, number)`), so + every group's processor 5 collides on one map key. **The result is not a crash.** Numbers stay below + 64 within a group, so `assert!(cpu < 64)` never fires: the tool runs, pins to whichever processor + won the collision, and prints a confident placement table describing a topology it silently + collapsed. That is the same defect class as the omitted SMT row, on the machine we would get one + attempt at. + +- [x] **PT-1B.2** -- **Pin with `SetThreadGroupAffinity`.** `SetThreadAffinityMask` takes a mask + within the caller's current group and cannot express a processor in another one, so it is not a + matter of widening the mask. Keep the existing failure discipline: pinning that does not land must + abort the run rather than fall back to an unpinned measurement. + +- [x] **PT-1B.3** -- **Verify against a synthetic multi-group topology**, since no host here has more + than one group. `places_from_topology` is a pure conversion and already testable; a fixture with two + groups whose numbers overlap must produce distinct processors, and the sabotage is to key on the + number alone and watch the count halve. + **Nine tests added, and they found two real defects rather than confirming the change.** `classify` + compared `core` without comparing `group`, so two processors in different groups whose core ids + collided were reported as **SMT siblings** -- attributing a shared L1 that cannot exist, since a core + cannot span a group. And the fallback core id was derived from the number alone, which is what made + those collisions possible. + **The planned sabotage could not be performed, which is the strongest available result.** Keying the + conversion on the number alone no longer *compiles*: the maps are keyed `(u16, u8)`, so the collapse + is unrepresentable rather than merely tested against. The sabotage that does compile -- dropping the + group comparison from `classify` -- was performed and is caught. + **One test asserted a promise the code never made** and was rewritten rather than the code bent to + fit: `representative_pairs` returns one pair per placement *category*, and "in a different group" is + not a category, so requiring a pair from every group was wrong. Its fixture also gave both groups the + same cache-domain ids, describing a cache shared across groups, which no machine does. + +- [x] **PT-1B.4** -- **Refuse loudly if groups are present and unsupported.** Whatever remains + unimplemented when a large machine is offered, the tool must say so and stop. A refusal costs one + message; a collapsed topology costs a wrong answer nobody can detect from the output, on hardware + that is not coming back. + +## M1C: direction and memory placement, before a NUMA machine is spent + +**Raised by the engineer asking why the hop count was the edge count rather than twice it.** It should +be twice it, and answering that exposed a second defect underneath. + +- [x] **M1C.1** -- **Measure both directions of a node pair.** `node_pairs` is undirected, with the + reasoning that `0 -> 1` and `1 -> 0` "traverse the same link". That conflates the *link*, which is + symmetric, with the *workload over it*, which is not: the producer **writes** slots and + release-stores `tail`, the consumer **reads** slots and release-stores `head`, and a remote write + needs exclusive ownership and invalidation where a remote read does not. Swapping the ends is a + different measurement, not a repeat. + Doubles the hop count, so state the cost plainly: `n*(n-1)` rather than `n*(n-1)/2`. On a four-node + host that is 12 hops instead of 6, and the runtime estimate must follow. + **Keep the two directions distinguishable in the record.** Reporting a mean of them would destroy + exactly the asymmetry this item exists to measure. + +- [x] **M1C.2** -- **Control and record which node the ring's memory is on.** `Ring::new` runs on the + calling thread, which is never pinned, so under first-touch the ring lands on whatever node the + *orchestrating* thread happened to occupy -- possibly neither the producer's nor the consumer's. + **On a multi-socket machine there are three positions, not two**, and the third is currently + uncontrolled and unrecorded. Two runs could differ solely because the main thread migrated, with + nothing in the output to say so. + This is not a refinement; it is what makes a NUMA number mean anything. A hop measured with the + memory on an unknown third node is not a measurement of that hop. + **Decided: measure both endpoints as separate rows.** The memory goes on the producer's node in one + row and the consumer's in another, and **the memory node is recorded beside the two processor + nodes** in every row. + This measures remote-write and remote-read cost independently, which is the pair of quantities the + asymmetry in M1C.1 is actually about: with memory on the producer's node the producer writes locally + and the consumer reads remotely, and swapping the memory reverses exactly that. + + **The cost, stated plainly.** Four configurations per undirected edge -- two directions times two + memory placements -- so `2*n*(n-1)` hop measurements rather than today's `n*(n-1)/2`. On a four-node + host that is 24 rather than 6, and at two strategies and three repetitions it is 144 timed handoffs + for the hops alone. Under two minutes at the worst per-item cost measured so far, which is + affordable for hardware this scarce. **PT-4.2's estimate must be updated with it**, or the tool will + under-promise the wait on precisely the machines that take longest. + + **The design carries its own consistency check, which is worth keeping rather than optimising + away.** Of the four configurations per edge, two are "producer-local" and two are "consumer-local", + differing only in which physical node each role sits on. On a symmetric interconnect each pair + should agree; **if they disagree, the interconnect is asymmetric, and that is a finding** rather + than noise. Averaging the pairs, or measuring only one of each, would discard it. + +- [x] **M1C.3** -- **Say what the placement label means once direction exists.** A row currently reads + as a pair of positions; it must read as producer-here, consumer-there, memory-somewhere. The + existing `Placement` names are direction-free and will quietly under-describe a directed run, which + is the "table with right labels and wrong pairs" failure in a new place. + +## M2: the move + +- [x] **PT-2.1** -- Move `fingerprint`, `core_affinity` and `peer_index_cache` into the new crate, and + make `windows-platform-probes` depend on it. This inverts today's direction deliberately: the + published crate owns the measurement, the internal grab-bag borrows it. A **pure relocation** with + the provenance trail the repository requires for a split -- commit trailers and per-file headers -- + because these modules carry a session's worth of hard-won reasoning in their comments and blame must + survive. + +- [x] **PT-2.2** -- Keep `queue_contention` and every unrelated probe where they are. The new crate is + not a home for "measurement code in general"; it is one tool with one question, and admitting a + second unrelated probe is how it becomes the grab-bag it was extracted from. + +- [x] **PT-2.3** -- Verify the move changed no behaviour: the three probe binaries (or their + replacements per PT-1.3) produce the same numbers on this host as recorded in + [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M-inf.4, and the full sabotage set still fails + where it should. + **Verified three ways.** Git recorded all five files as **100% renames**, so this was a pure + relocation and `--follow` and blame both carry through without the provenance headers a *split* + would need -- a split copies and leaves the source behind, a move does not. Test counts add up + exactly: 58 in the new crate plus 25 in the old is the 83 that existed before. And + `probe-core-affinity` reproduces its pre-move results (siblings 1.8x WINS at batch depth ~85, + cross-cache 0.54x LOSES at ~1.9), which is the check that matters, because compiling proves the + names resolved and nothing more. + **One defect surfaced, unrelated to the move and pre-existing:** `queue_contention` still imported + `windows_waitable_queues::mpsc`, stale since the `slotwise_mpsc` rename. It went unnoticed because + nothing had rebuilt that crate since, and the move is what forced the rebuild. Fixed here rather + than left for the release. + +## M3: the submission record + +Ordered so each item's prerequisites land first: the two identity fields are decided before the record +that carries them is written. + +- [x] **PT-3.1** -- **A linearly increasing integer schema version that cannot silently drift, guarded + by an archived schema rather than a hash.** The counter itself is easy for a consumer to compare + (`schema >= 2`); the hazard is forgetting to bump it when the record's shape changes, which no amount + of care reliably prevents. So derive rather than restate, per this repository's own rule -- but + derive into something that survives. + **A hash was considered first and rejected, because it does not survive its own history.** With a + table of `version -> hash`, only the *current* version's hash can ever be recomputed; every earlier + row is a frozen constant nobody can verify. The hash function then becomes an unversioned contract -- + change the traversal, the digest, or how key paths are canonicalised, and every historical row + silently becomes wrong, with nothing to detect it. A digest is also opaque: it reports *that* the + shape moved and never *what* moved, so a review cannot see whether a change was additive or breaking. + **Archive the shape itself.** One golden file per schema version, listing the record's key paths + (sorted, recursively) as text. A test generates the current shape and asserts it equals the golden + for the current `SCHEMA_VERSION`; a change fails the test and the diff *shows what changed*. Bumping + means adding the next golden, deliberately. + This buys three things a hash cannot: a stored submission can be **validated against the schema it + declares**, years later; the version-to-version diff is **reviewable**; and there is **no hash + function to keep stable**, so no way for history to rot. + **Golden files are append-only and a published version is never redefined** -- the same discipline as + [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md). Once a record exists in the wild claiming schema N, + N's meaning is fixed, because the record cannot be regenerated. Verify by sabotage: add a field, + confirm the test fails and names the difference; bump, add the golden, confirm it passes. + +- [x] **PT-3.2** -- **Stamp the exact build, and say loudly when it is not an official one.** The + record carries the git commit, whether the working tree was dirty when it was built, the crate + version, and whether it came from CI or a local build. + **This is the same problem as `Provenance` one layer up, and takes the same shape**: an official + CI-built binary from a clean tree is the trusted case, and everything else -- a local build, a dirty + tree, an unknown commit -- must be visibly marked so a result that arrives from one is not silently + pooled with the rest. Default to the untrusted reading when the answer cannot be established, for + the same reason `Provenance::Synthetic` is `Default`: forgetting must be safe. + A [build.rs](crates/windows-placement-probe/build.rs) reads the commit from an environment variable when CI sets one, falls back to `git` + when there is a repository, and records *unknown* otherwise -- which is exactly what a `cargo + install` from a crates.io tarball will produce, and is the honest answer there. + +- [x] **PT-3.3** -- Emit **one** machine-readable record per run, carrying: the schema version + (PT-3.1), the build identity (PT-3.2), the topology **provenance**, a UTC timestamp, the host + fingerprint, every placement measurement, and every node-hop measurement. + **Build identity is the load-bearing field.** Results will arrive over months from different builds, + and a measurement that does not say which build produced it is an unlabelled number -- the exact + failure this workspace spent [crates/windows-topology-sys/DESIGN-NOTES.md](crates/windows-topology-sys/DESIGN-NOTES.md) + `D-12` fixing one layer down. There is currently **no** version stamped in any probe output. + +- [x] **PT-3.4** -- Keep the human-readable report as well, and derive both from the same measured + values so they cannot disagree. The reader running the tool should be able to see, in prose, the + same conclusion the record encodes -- otherwise nobody notices when a run is nonsense. + +- [x] **PT-3.5** -- **The terminal output is the submission.** Collection happens by asking people to + paste a run into a GitHub Discussions thread on this repository, so the paste is the channel and the + whole record must survive it. + **This reverses this item's original reasoning, and the reversal is the point.** It previously said + to write a file *because* copying terminal output invites truncated and reflowed submissions. That + risk is real and does not go away by choosing a different channel -- it has to be *mitigated* rather + than avoided: + - **Everything needed is on screen.** The record prints to stdout, not only to a file. A submission + that requires the sender to find and attach a file will sometimes arrive without it. + - **A self-check the reader can run.** A short checksum over the record, printed beside it, so a + truncated or reflowed paste is *detectable* rather than silently half-ingested. This is the same + principle as the schema golden: detect corruption instead of trusting the channel. + - **Paste-safe formatting.** GitHub renders Discussions as markdown, so the output must survive a + fenced code block and must not depend on colour, cursor control, or overlong lines that wrap. + - **Tell the runner exactly what to do**, in the output itself: which thread, and to paste inside a + fenced block. An instruction that lives only in a README is an instruction half of them will not + have read. + **The target is select-all, copy, paste, done.** Every extra step is a submission that does not + arrive, so the tool emits its own markdown fences: a runner who has never thought about markdown + pastes the whole thing and it renders as a code block anyway. Instructions caught inside the fence + are trivial noise next to a paste that renders as mangled prose. + A file is still written, because it costs nothing and someone will prefer to attach one -- but it is + a backup, never a required step, and the run must be complete and submittable without it. + +- [x] **PT-3.6** -- Read the three machine-description fields PT-1.2 settled, each of which needs a + source this crate does not currently use. **Every one of them is optional in the record**, so a host + that will not answer produces a record missing a field rather than a failed run or a fabricated + value. + - **CPU model** -- the registry's `ProcessorNameString` under + `HKLM\HARDWARE\DESCRIPTION\System\CentralProcessor\0` is the pragmatic source and works on both + x64 and ARM64, unlike the CPUID brand string. + - **OS build** -- the reported version must be the real one. The Win32 compatibility shims lie to + unmanifested processes about the major version, so verify against a known build rather than + trusting the first API that returns a number. + - **Virtualisation hint** -- record confidence honestly. There is no user-mode call that decides + this, so whatever signal is used, the field says "hint" and a negative means *not detected* rather + than *bare metal*. + +## M4: the runner's experience, and their trust + +- [x] **PT-4.1** -- **One entry point.** A single binary that runs everything and produces one record. + "Run these three and send me all three outputs" is friction for someone doing a favour, and invites + partial submissions that cannot be compared. + +- [x] **PT-4.2** -- **State the runtime before doing the work**, from the discovered topology rather + than a guess: the hop matrix alone is `n*(n-1)/2` hops times two strategies times three repetitions + times two million items, on top of the placements. On a four-node machine that is a materially + longer run than on this one, and the person deserves to know before it starts. + +- [x] **PT-4.6** -- **Set up the Discussions thread people paste into, and link it from the tool.** + The tool's output names where to send a result, so that destination has to exist before the tool + ships, not after -- an instruction pointing at a thread that is not there is worse than no + instruction. Pin it, and state in the first post what is collected and what a submission is used + for, so a reader who arrives from a search rather than from the README still sees it. + **Done: [discussion 55](https://github.com/MikeGrier/windows-threadpool-sys/discussions/55), "Please + share data from `windows-placement-probe`".** The tool points at that thread rather than at the + discussions index, so a runner lands on the reply box instead of a list they have to search -- a + link that costs an extra navigation is one more place a submission stops. A test asserts the URL + still ends in a discussion number, which catches the plausible later mistake of trimming it back to + the index during a tidy-up. + +- [x] **PT-4.3** -- **Say exactly what is collected and what is not**, in the tool's own output and in + its README, and make it verifiable by reading the record. Collected, per PT-1.2: core/cache/NUMA + shape, timings, CPU model, OS build, and the virtualisation hint. **Not** collected: hostname, user + name, file paths, environment variables, serial numbers, or anything about installed software -- + and that list is a commitment, not a description of the current implementation. **The tool makes no + network connections**; the person sends the file themselves, deliberately. Mention the model + suppression flag here, where someone deciding whether to run it will actually see it -- alongside + the honest limit from PT-1.2, that the flag does not make confidential hardware safe to submit, + because the topology describes the part regardless. + +- [x] **PT-4.4** -- Pin the thread-pinning failure behaviour for a stranger's machine. It currently + panics, which is right for us (a silently unpinned thread measures the scheduler, not the placement) + but reads as a crash to someone doing a favour. It must fail with an explanation of what could not + be pinned and why the run cannot continue honestly -- **and must not fall back to an unpinned + measurement**, which would produce a plausible number that means nothing. + +- [x] **PT-4.5** -- **Let the runner see everything before sending it, and decide with the real values + rather than a promise.** This is a stronger privacy property than any suppression flag, and cheaper: + the record is a text file, so the honest instruction is "open it and read it -- if you are not happy + with something in there, do not send it." + Two things make that instruction usable rather than theatre: + - **A fast preview of the machine-description fields**, available without running the measurement. + The full run takes minutes and grows with node count; nobody should have to spend that to discover + what the tool would learn about their machine. Someone can then look, decide, and only then commit + to the run. + - **A record a human can actually read** -- field names that mean something without a schema in hand, + and no opaque blobs. A file that must be decoded to be checked cannot honestly be described as + inspectable. + +- [x] **PT-4.7** -- **Lay the JSON out for a reader rather than a parser.** `to_string_pretty` gives + every array element its own line, so eight cache domains cost eight lines saying `2` and a + sixty-four-node host would spend sixty-four lines listing its nodes one integer at a time -- on + precisely the machine whose submission matters most. Arrays holding no object now collapse onto one + line, or fill lines up to a width budget, while objects still expand one field per line. + **Field order is part of the contract, not incidental.** The first draft laid out a + `serde_json::Value`, whose object is a `BTreeMap`, and every test still passed while the output + quietly sorted `build` above `schema_version` and made each measurement row open with + `consumer_batch`. Caught by reading a run, not by the suite. Order is now preserved by walking an + order-preserving tree; `serde_json`'s `preserve_order` feature is deliberately **not** used, because + cargo unifies features and four other crates in this workspace share `serde_json`. + +## M5: distribution + +**The CI-built artifact is the canonical way to get this tool**, not `cargo install`. Two reasons, and +the second is the real one: a downloader needs no Rust toolchain, and **the download itself is the +provenance**. A binary attached to a release in this repository is traceable to the commit that built +it, in a way a locally built copy of the same source is not -- which is what makes PT-3.5's "official +build" distinction meaningful rather than decorative. + +- [x] **PT-5.1** -- CI builds the tool on tag and attaches the binary to a GitHub release, injecting + the commit into the environment variable PT-3.5 reads. **Verify the negative case**: a locally built + binary must produce a record marked as an unofficial build, and a CI-built one must not. A + distinction nobody has watched fail is a distinction that does not work. + **Done, and both directions are checked inside the workflow itself rather than trusted.** It builds + with the stamps and asserts the artifact reports itself official *and names this commit*; then + rebuilds **without** them and asserts that binary marks itself unofficial; then rebuilds with the + stamps for release, because the negative check overwrote the artifact and shipping that file would + attach an `!!UNOFFICIAL!!` binary to an official release. + A `--version` flag was added for this, printing the whole build identity rather than a version + number -- CI asserts on it, and a downloader can check the same thing before trusting a binary. + **`aarch64-pc-windows-msvc` is in the matrix and cannot be verified locally.** The cross-build fails + on this machine with `unresolved external symbol __imp_GetProcessHeap`, which is a missing local + ARM64 MSVC library rather than a code fault -- `std` itself uses that symbol, so a real defect would + break every ARM64 Rust program. + **The pull request verifies it, which is better than the dispatch this originally called for.** The + workflow now also triggers on a pull request touching the tool, building and verifying both targets + without releasing. Two things made that the right answer rather than a convenience: + - **Nothing else in this repository builds the ARM64 target.** [ci.yml](.github/workflows/ci.yml) cross-compiles only + `thumbv7em` for `wtf-string`, so without this a tag would be the first time `aarch64` was ever + attempted -- turning a build failure into a broken release. + - **`workflow_dispatch` could not have done it.** GitHub only offers dispatch for workflows already + on the **default branch**, so a workflow still on a feature branch cannot be dispatched at all -- + which is precisely when it needs verifying. The original instruction here was unusable. + The release job stays guarded on the tag ref, so a pull request publishes nothing however it runs. + +- [x] **PT-5.2** -- A README written for someone who has never seen this repository: what question the + tool answers, why their machine is interesting, where to download it, how to run it, what to send + back, and what it collects. Assume no context and no obligation. Lead with the download, not with + `cargo install`. + +- [x] **PT-5.3** -- Decide whether to publish to crates.io **as well**, and record the reasoning. It + costs a semver obligation and yields records whose commit is *unknown* by construction (a crates.io + tarball carries no repository), which is a strictly weaker submission. The case for it is reach; the + case against is that the weaker path is also the more discoverable one, and submissions will drift + towards it. + **Decided: yes, publish -- but not yet.** Timing is what answers the objection. Publishing *after* + the download path exists, is documented, and has been walked end to end means the strong path is the + one a runner meets first, and crates.io becomes the fallback it should be rather than the default. + Reasoning recorded in + [DESIGN-NOTES.md](crates/windows-placement-probe/DESIGN-NOTES.md), including the rejected + alternative of baking the commit into the packaged source -- which would let a crates.io build name + a commit while still not showing that CI built it, and so would have the record's trust section + claim something it cannot support. + The publication itself is **PT-5.6** below; it is not part of this item, which was only ever a + decision. + **REVERSED 2026-09-02: decided no, never publish to crates.io.** The GitHub release binary is the + only distribution, and `publish = false` is now permanent. Two reasons, the second of which was not + known when the above was written: + 1. **The reach premise was backwards.** A released binary needs no Rust toolchain; `cargo install` + needs a toolchain, a compiler, and a build of the whole dependency tree. crates.io therefore + reaches a *subset* of the download path's audience -- a convenience for Rust developers, bought + by making the weakest-provenance path the most discoverable. M5's own preamble had already said + the download "needs no Rust toolchain" and is "the provenance". + 2. **A published crate cannot use bare `path` dependencies, and cargo enforces the resulting + `version` pins at every build rather than at publication.** So a pin left stale by any workspace + bump breaks the entire workspace's resolution. Measured: topology at 0.2.0 against this crate's + `"0.1.0"` pin failed `cargo metadata` outright. Publishing would have made that a permanent tax; + not publishing let both pins be deleted. + Recorded in [DESIGN-NOTES.md](crates/windows-placement-probe/DESIGN-NOTES.md), which keeps the + superseded reasoning because its provenance argument is still why the record marks unofficial + builds. + +- [x] **PT-5.4** -- Package metadata and a statement of what is and is not covered by semver. The + **record's schema is a compatibility surface** the moment anyone stores one; the internal measurement + code is not. + +- [ ] **PT-5.5** -- Walk the whole path end to end on a machine without this repository checked out: + download, run, find the record, read the README's instructions for sending it. A path nobody has + walked is a path that does not work, and the person walking it will be doing a favour rather than + debugging. + **Deliberately left open: this cannot be completed from here.** It needs a real release to download + from and a machine without this checkout, and doing it against a local build would test something + else while looking like it had passed. The ARM64 development machine is the obvious first walker, + and it doubles as the check that the unverified `aarch64` artifact from PT-5.1 actually runs. + +## M5+: crates.io -- WITHDRAWN, never to be published + +**PT-5.3's decision to publish here was reversed on 2026-09-02; this milestone will not be pulled in +and numbered.** Kept as a heading rather than deleted, so that a reader who remembers a plan to +publish finds the reversal instead of a gap. The reasoning is on PT-5.3 above and in +[DESIGN-NOTES.md](crates/windows-placement-probe/DESIGN-NOTES.md): the reach premise was backwards, +and a published crate would have owed permanent dependency-pin maintenance that cargo enforces at +every build rather than at publication. + +- [x] **PT-5.6** -- **WITHDRAWN: `windows-placement-probe` is never published to crates.io.** Checked + off as *decided against*, not as done. Its cross-component prerequisites on `SH-4.1` and `SH-4.3` + are void, and the reciprocal note in + [CHECKLIST-ship-topology-and-queues.md](CHECKLIST-ship-topology-and-queues.md) has been updated to + say so -- a prerequisite that outlives the item needing it is how work gets blocked on nothing. + Two of its three "must not skip" points are void with it: the dependency pins were **deleted** + rather than corrected (the crate is path-only now), and there is no `cargo install`ed copy to run. + The third survives on its own merit and is **not** lost: the README should still say that a + locally built copy produces records marked unofficial, because a runner can still build one from + source. That is **PT-5.7** below rather than a bullet inside a withdrawn item. + +- [ ] **PT-5.7** -- **Say in the README what a locally built copy costs the data.** Rescued from + PT-5.6, whose withdrawal would otherwise have taken it. The point never depended on crates.io: a + runner who clones and `cargo build`s gets a binary that marks its records `!!UNOFFICIAL!!` with no + commit, exactly as a `cargo install`ed one would have. They should learn that from the README + rather than from their own output, and it is the negative case that makes PT-3.5's "official build" + distinction mean something to a reader rather than only to CI. +## M6: is a set of "equivalent" processors actually equivalent? + +- [ ] **PT-6.1** -- **Give the fingerprint a placement signature, or keep saying it is not canonical.** + [fingerprint.rs](crates/windows-placement-probe/src/fingerprint.rs) records each partition as a list + of *sizes* -- processors per cache domain, per efficiency class, per NUMA node -- and never how + those partitions intersect. Two eight-processor hosts can both render + `L2[4,4] ec[0:4,1:4] numa[4,4]` while one puts each efficiency class in its own cache domain and the + other splits both classes across both; only the second can express a same-cache/cross-class pair. + **The placements available to a run differ while the fingerprint agrees**, so string equality is not + placement equivalence. + The claim has been corrected in place, so nothing is currently wrong -- this item is the stronger + fix, not a bug. It needs a canonical signature of the expressible placements *in* the string, which + means a serialized field and therefore a schema bump. + **Deliberately gated on some other reason to bump the schema**, because a summary line is not worth + a version of its own when every measurement row already names the placement it was taken at, which + is what a collector needing equivalence should read. Raised by review 5073245942 on pull request + #56. + +- [ ] **PT-6.2** -- **Give the NUMA list an absence marker, so an unreported node set is not read as + one node.** [fingerprint.rs](crates/windows-placement-probe/src/fingerprint.rs) renders + `numa_node_sizes` as the nodes the topology *reported*, so a host naming no memory domains renders + `numa[]` while every processor is still counted and every placement still reports node `0` -- the + documented single-node default. The node list therefore does not sum to `processors` in that one + case. + The asymmetry with the cache list is deliberate, not an oversight: `cache_domain_sizes` fills itself + with the processor count when no level partitions the host, but it can afford to, because it renders + `L-` for "no partitioning level" and so `L-[16]` cannot be mistaken for a real single-domain level. + The NUMA list has no such marker, so `numa[16]` would be indistinguishable from a host that genuinely + reported one node of 16, and the more useful fact -- that the machine said nothing about NUMA -- + would be lost. + The behaviour is documented on the field and pinned by + `a_bare_topology_renders_its_processors_but_claims_no_numa_nodes`, so nothing is currently wrong; + this item is the stronger fix. A marker is a serialized-field change and therefore a schema bump, + so like PT-6.1 it is **deliberately gated on some other reason to bump the schema**. Found while + fixing the processor count raised by review on pull request #56. + +**Not gated on the release, unlike the rest of this file.** The work is an extension of the affinity +measurement, which today lives in [crates/windows-platform-probes](crates/windows-platform-probes) and +moves wholesale under PT-2.1. Build it there now; it travels with everything else. + +**The assumption under test.** Several designs in this workspace treat a *set* of processors as +interchangeable -- any processor in this cache domain, any processor in this NUMA node -- and place +threads by domain rather than by processor. [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M-inf.5 +rests on exactly that. **Every measurement taken so far pins to a single processor** (`mask = 1 << cpu`), +so the assumption has never been tested; it has only been assumed while being carefully avoided. + +**There is a structural reason to doubt it, before any scheduling subtlety.** A set mask permits +placements a single-processor mask forbids -- including **both threads on one logical processor**, +which turns an SPSC handoff from concurrency into time-slicing, with the spin-wait burning its quantum +before the peer can run. On an SMT host the `same cache domain` set *is* the two siblings of one core, +so this is not a corner case there, it is the common one. + +- [ ] **M6.1** -- Derive each processor's **equivalence set** from the topology -- SMT siblings, cache + domain, NUMA node, efficiency class -- and pin down which sets a given host can express, the same way + placements already are. A set with one member is not a test of anything and must be reported as + inexpressible rather than measured. + +- [ ] **M6.2** -- Add **affinity mode** as a dimension beside placement and strategy: `Pinned` (today's + single bit) and `SetWide` (each thread masked to *its own* equivalence set, which preserves the + placement relation while relaxing the choice within it). For `CrossCacheSameClass` that means the + producer may use any processor of its cache domain and the consumer any of its own; for + `SameCacheSameClass` both threads share one set, which is where co-residency becomes possible. + **In `SetWide` the placement label states intent, not outcome** -- the scheduler may do something + else entirely, and saying otherwise would be the "asserts its conclusion" defect again. + +- [ ] **M6.3** -- Measure the **mechanism**, not only the elapsed time, or the result cannot be read. + Sample `GetCurrentProcessorNumber` in both loops and report **migration count** (did the thread move + at all?) and **co-residency fraction** (how often were producer and consumer on the same processor?). + Co-residency is the killer observable and can only be non-zero in `SetWide`. + Without these, "the two modes matched" is indistinguishable from "the scheduler never moved + anything", which is precisely the false-equivalence this milestone exists to rule out -- and is the + same trap the peer-index probe's read counters were added to escape. + +- [ ] **M6.4** -- Run **long enough for the scheduler to act**. The present 2M items is roughly 40 ms + on an idle host, over which nothing migrates and both modes will look identical for want of any + reason to differ. Choose the duration from measured migration counts -- long enough that migrations + are actually observed under load -- rather than from a round number, and record the reasoning. + +- [ ] **M6.5** -- **Interference pass one: competing spinners confined to the same equivalence set.** + Adversarial and controlled: it forces the scheduler to choose *within* the class, which is the + precise claim under test. Vary the number of competitors relative to set size, since one spinner in a + four-processor set is a different question from four. Keep it reproducible -- an interference model + that varies run to run turns every comparison into noise. + +- [ ] **M6.6** -- **Interference pass two: a concurrent copy of the real workload.** A second + producer/consumer pair on the same set, which is what a domain runtime actually looks like when more + than one queue is live. Pass one establishes whether the scheduler *can* break the equivalence; this + establishes whether it *does* under load anyone would really generate. **Report both**: a difference + that appears only under adversarial spinners is a real finding with a narrower consequence, and + collapsing the two would lose exactly that distinction. + +- [ ] **M6.7** -- **FEEDS [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M-inf.5, whose premise + this tests. On completing this, edit M-inf.5 with the answer** -- it does not check M-inf.5 off (that + item is the domain-local placement work itself), but M-inf.5's 5.6x is a number about *pinned* + threads until this says otherwise, and leaving that unstated is how a measured caveat quietly becomes + an assumed fact. + Report per set kind whether the equivalence holds, in the tool's own words, derived + from the measurement rather than asserted. **A null result is + a real result here** -- "the sets behaved equivalently under both interference models, and here are + the migration counts showing the scheduler was genuinely exercised" retires a long-standing doubt, + and is worth as much as a difference would be. + +## M7: report what Windows contradicts about itself + +Opened by [D-17](crates/windows-topology-sys/DESIGN-NOTES.md) in the topology crate, which establishes +that two Win32 topology sources can be **stably** inconsistent -- and that this is expected on hardware +we do not have and on prerelease firmware, rather than being exotic. + +The division of labour is deliberate and follows the same facts-versus-policy line the rest of this +workspace uses. **`windows-topology-sys` records the disagreement**; it does not report it, because a +crate that states facts should not be in the business of producing bug reports. **This tool reports +it**, because reporting is what this tool is for, and because the provenance that makes such a report +actionable is identifying and therefore belongs behind the review this tool already applies. + +- [ ] **PT-7.1** -- **Surface the topology's recorded inconsistencies**, in the tool's output and in + the submission record. This is the only place they become visible to anyone: the topology crate keeps + what each source said, and nothing in the workspace currently looks at it. + **Partly done by `M36.4` on 2026-09-04, and what remains is narrower than the text below.** The + **record** half is complete: `topology_coherence` carries the whole `Coherence`, so a submission + names the disagreeing processors individually. The **output** half is partly done: the report has + a section that appears only on `Disagreed`, gives the counts on each side and the retry number, + and says the measurements are unaffected. Two things are still open, and both are decisions rather + than plumbing: + (a) **Name the processors in the printed text**, not only in the record -- "what each source + claimed", which is what this item asks for and what counts alone do not give. + (b) **Decide whether an inconsistent run is marked in "where this result came from".** + `is_fully_traceable` is deliberately untouched, so that section currently reads "an official build, + reading this machine's real topology" directly above the disagreement section. That is not a + contradiction -- the build *is* official and the topology *was* read -- but a reader may feel one. + This item's own two-sided framing below is the guidance for settling it: mark it plainly enough + that neither the runner nor a later reader is left guessing, without dressing up a nuisance as a + prize. + Report what disagreed and what each source claimed, not merely that something did -- "incoherent" is + not actionable, and the point of collecting from strangers' machines is to learn something specific + about hardware nobody here can buy. + An inconsistent machine is **still a valid submission**, and should be marked rather than rejected. + **Its value is genuinely two-sided, and the tool should not pretend otherwise.** In the long run it + is the more valuable submission -- evidence of something no local run can produce, and potentially a + bug report against a firmware table. In the short run it is an **annoyance**: a run whose numbers a + reader must qualify, from a machine whose description cannot be taken at face value. + So mark it plainly enough that a runner is not left wondering whether their machine is broken or + their run is wasted, and plainly enough that a reader of the submission knows which parts to trust -- + without dressing up a nuisance as a prize. + +- [ ] **PT-7.2** -- **Add the firmware provenance an inconsistency report needs to be actionable** -- + mainboard and BIOS version at minimum -- suppressible by the runner, with the suppression recorded + rather than merely absent, exactly as `MachineDescription`'s model handling already does. + **Weigh it against the existing honesty about what suppression buys.** This checklist already + establishes that the flag "must not be oversold", that a pre-release part "is identified at least as + well by its topology", and that the tool "cannot make an NDA-covered machine safe to submit from, and + must not imply that it can". Firmware provenance sits under that same caveat and arguably deepens it: + a BIOS version can pin a specific board revision more precisely than a CPU model names a part. + So the honest framing is unchanged rather than weakened -- if the hardware is confidential, the right + answer remains not to send it -- but the README's list of what is collected must grow to match, per + `PT-4.3`, and the runner must still see the real values before deciding, per `PT-4.5`. +## M36 -- Redact the secondary metadata by default + +- [x] **M36.1** -- **Floor the record's timestamp to the minute, in UTC.** Done 2026-09-04. A + second-precision stamp links two submissions from one host to each other even after every + identifying field is withheld, and nothing in the analysis needs finer -- these measure a machine's + shape, not an ordering of events. UTC with no local offset, because an offset narrows the submitter + to a band of longitudes for no gain. `recorded_at_subsecond_millis` is untouched: it is + `serde(skip)` and exists only so two runs in one second get distinct file names. + +- [x] **M36.2** -- **Redact the secondary metadata by default, with an opt-in to include it.** + Done 2026-09-04, with a single `--include-metadata` as recommended; `--no-cpu-model` survives as a + subtraction from it, because the confidential-part case it was built for is not covered by the + general opt-in and passing it alone is redundant rather than wrong. Suppression is recorded for + every newly redactable field: `os_build_suppressed` and `recorded_at_suppressed` beside their + `Option`s, and a `VirtualisationHint::Suppressed` variant rather than a flag, since that enum's + other variants are all claims about what was observed. The backup file's name drops the stamp with + the record, so the withheld minute cannot escape through a file a runner attaches. See + [DESIGN-NOTES.md](crates/windows-placement-probe/DESIGN-NOTES.md) -> "The measurement is not + redactable; the context is, and is withheld by default". + Engineer's decision, 2026-09-04. The secondary metadata is the timestamp, the OS build, and the + hypervisor name/hint -- everything in `MachineDescription` and the `recorded_at*` fields that is + *context* rather than *measurement*. The topology is excluded from this by construction: it is the + measurement, and the README already says so plainly. + **Default flips to redacted.** `--no-cpu-model` becomes one case of a general rule rather than the + only switch. Decide whether the opt-in is one flag or per-field; a single `--include-metadata` is + the smaller surface and is the recommendation unless a per-field need appears. + **Suppression must stay distinguishable from absence**, which the existing `model_suppressed` flag + already does for the model: a field withheld by the runner and a field the host would not answer are + different facts, and a collector that cannot tell them apart will read one as the other. Every newly + redactable field needs the same treatment. + **No `SCHEMA_VERSION` bump**: the freeze starts at the first release and this crate has not had one. + +- [x] **M36.3** -- **Say in the README what redaction costs.** Done 2026-09-04, as a + "What redaction costs" section. Built on the asymmetry `PT-1.2` already established -- withheld + context cannot be recovered later, while over-collection can be corrected going forward -- then + what each of the four fields buys, ordered by explanatory value rather than by sensitivity, with + the minute named as the weakest of them. Two guards against mis-reading the new default: redaction + does not make a submitter anonymous, because the topology is always sent and is the most + identifying thing in the record; and a redacted submission is still a good submission, because + sending nothing is by far the worse outcome. There is real value in correlating + metadata anomalies with specific platform versions -- a defect that shows up only on one OS build, + or only under one hypervisor, is exactly what the secondary metadata is for. A reader choosing to + include it should understand they are helping, and a reader choosing not to should understand what + they are withholding. State the trade rather than presenting redaction as free. + +- [x] **M36.4** -- **On `Coherence::Disagreed`, ask for the unredacted record privately.** Done + 2026-09-04. **The dependency below was mis-stated and is corrected here**: `Coherence` was *not* + reachable from the record. `topology_provenance` is carried and `Fingerprint` is built from the + topology, but the fingerprint carries only the provenance, so the record had no way to know its + two sources had disagreed. The record gained `topology_coherence`, carrying the whole `Coherence` + including the processor lists -- a boolean would have made the ask hollow, since the record a + maintainer is offered has to contain what they would look at. It is a field of the *record* and + deliberately not of the `Fingerprint`, which is compared for equality to catch a spliced record + and would then discard a good measurement over a difference in no shape at all. + The wording is informative rather than coercive, per the engineer's direction: it reports what was + detected, says the measurements are unaffected, names both possible causes -- inconsistent + platform metadata *or* a defect in this tool -- as undecidable from the runner's machine, offers a + way to help, and closes with "None of that is required." A test asserts the release is present and + that no pressure word appears. See + [DESIGN-NOTES.md](crates/windows-placement-probe/DESIGN-NOTES.md) -> "A disagreement is reported + where it happens, and the ask attached to it is an offer". + The report + emits extra text when the topology's two sources disagreed past the retry: say that the metadata was + inconsistent, and ask the runner to contact the `windows-threadpool-sys` maintainers through the + discussions or issues boards and share an **unredacted** probe file **privately**, so the + inconsistency can be verified -- or the probe fixed -- and a bug logged with Windows. + **This is the point of the whole design.** Redaction is the default because most records do not need + the context; the one case where the context matters most is a disagreement, which + [D-17](crates/windows-topology-sys/DESIGN-NOTES.md#d-17) attributes to prerelease hardware, + defective firmware tables, or a feature landing in one enumeration before the other -- the + bug-worthy cases. So the request is made exactly there, and privately, rather than by collecting + everything from everyone against the possibility. + Depends on M36.2 (there must be something to un-redact) and on `Coherence` being reachable from + the record, which it is: `topology_provenance` is already carried, and `Fingerprint` is built from + the topology. diff --git a/CHECKLIST-ship-topology-and-queues.md b/CHECKLIST-ship-topology-and-queues.md new file mode 100644 index 00000000..fce590b7 --- /dev/null +++ b/CHECKLIST-ship-topology-and-queues.md @@ -0,0 +1,1473 @@ +# Checklist: ship the topology and queue crates + +**Goal.** Get `windows-topology-sys` 0.2.0 and `windows-waitable-queues` 0.1.0 released, so the +placement tool in [CHECKLIST-placement-tool.md](CHECKLIST-placement-tool.md) has something to build +against and other people can run it on hardware this workspace does not own. + +## Where this stands + +The release has not happened. PR #56 has been open as a **draft** since 2026-08-31 and is 221 commits +ahead of `main`. + +**Milestone numbers are not a running order.** M7 through M15 are *review rounds on PR #56*, so they +happened -- and continue to happen -- **inside M3**, between the pull request opening and a merge +that has not occurred. Reading the file top to bottom puts the review of a pull request after the +merge that closes it, which is backwards. Only M1 through M6 are a sequence. + +| Milestone | State | What it is waiting on | +|---|---|---| +| M1 settle the public surface | **done, archived** | -- | +| M2 repair the release plumbing | 1 of 5 open | only SH-2.3, which needs the merge commit | +| M3 land the branch | 4 of 5 open | now gated on M16; SH-3.1.1 runs after the model lands | +| M4 release | open | M3 | +| M5 verify from outside | open | M4 | +| M6 long-running validation | open | gates SH-4.3, so it gates the queue crate's publication | +| M7-M13 review rounds | **done, archived** | -- | +| M14 ninth review round | 1 open | SH-14.1, the ABA defect; disclosed at SH-15.8, fix is M15 | +| M15 the claim protocol | 5 open | SH-15.6 is the decision; gated on SH-15.5.1 | +| M16 tenth review round | 7 done, 6 superseded | its own findings are fixed; the model work moved to `MMT-*` | +| M-inf parked | ungated | not scheduled, deliberately | + +**The critical path is M16's locality-model work -> SH-3.1.1 -> SH-3.4 -> M4.** M14 and M15 do not +block it: SH-14.1 ships disclosed rather than fixed +([D-36](crates/windows-waitable-queues/DESIGN-NOTES.md#d-36)) and the disclosure -- which was the +actual release blocker -- landed at SH-15.8, so both conclude after 0.1.0 provided the pull request +**says** that is deliberate. SH-3.1.1 owns saying it. + +**M16 is different, and this was decided rather than drifted into.** Its four gated items +(SH-16.5, SH-16.8, SH-16.9, SH-16.10) are one piece of work -- replacing the locality model, +consuming CPU Sets, and collapsing three restatements of one rule -- and the decision is that +**PR #56 does not merge until it lands**. They were briefly listed here as non-blocking; that is +corrected. It is gated in turn on +[DESIGN-SESSION-2026-09-02-cache-locality-model.md](design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md), +which has open questions, so **design concludes before implementation starts**. + +What blocks the queue crate specifically, and separately, is M6. + +## Before checking anything off in this file + +Items here are cross-linked to the other plans, and **a cross-reference is an instruction, not a +footnote**. Every marker names its counterpart and states what completing this item obliges, because +the reciprocal edit is the step that gets skipped: the work feels finished, the box gets ticked, and +a second plan silently keeps describing a world that no longer exists. + +The markers, and what each obliges when its item completes: + +- **MIRRORS / MIRRORED BY** -- the same work seen from two plans. **Check both boxes in the same + commit**, and cite both IDs in the message. Neither is done alone. +- **GOVERNS / GOVERNED BY** -- this item decides something *about* the other; it never completes it. + Write the decision onto the governed item, and leave its box alone. +- **UNBLOCKS / LIFTS THE GATE ON / GATED BY** -- edit the gated file's gate paragraph so it states the + new reality. A gate that has silently lifted is as harmful as one that has not. +- **FEEDS / FED BY** -- write the *answer* into the fed item. It does not check that item off. + +Every marker in this repository's root checklists is reciprocal: if you follow one and find no +counterpart at the other end, that is a defect to fix, not a link to ignore. + +**Deliberately redundant with [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md).** That file plans the +*design*; this one plans the *release*, and a release has its own failure modes that a design checklist +will not surface. Where the two overlap -- M31.8 in particular -- this file states why the item is +release-blocking rather than restating the decision itself. + +## The state this starts from, verified rather than assumed + +- `windows-topology-sys` **0.1.0 is published**. The `provenance` field added on this branch is a + breaking change to a struct with public fields, so the next release is **0.2.0**, not 0.1.1. +- `windows-waitable-queues` **is not published**. First release, and its packaging is already complete: + description, keywords, categories, README, documentation link and a workspace license are all + present. Packaging is not a blocker; do not re-investigate it. +- `windows-ioring-sys` **0.2.0 is published and pins `windows-topology-sys = "0.1.0"` -- but as a + dev-dependency**, so consumers never resolve it and the pin obliges no release. Corrected at SH-2.2, + which was written on the opposite assumption. +- This branch was **54 commits ahead of `main` with no pull request** when this file was written. + **As of 2026-09-02 it is 221 commits ahead, and PR #56 has been open (as a draft) since + 2026-08-31.** Release automation runs on `main`, so nothing ships until it merges -- but the + pull request itself is no longer the thing to create, and the review rounds in M7 onwards all + happened on it while it sat open. + +> **M1 -- settling the public surface before publication -- is complete and archived.** Moved to +> [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md) on 2026-09-02. + +## M2: repair the release plumbing before relying on it + +**Read in file order, not numeric order.** These items are in dependency order, and the numbers are +historical -- SH-2.5 was split out of SH-2.2 and sits beside it, while SH-2.3 is last because it +needs the merge commit. Renumbering would break the references already in commit messages, so the +order is the authority and the numbers are only names. + +- [x] **SH-2.1** -- **Add `windows-waitable-queues-v*` to the tag trigger list in + [.github/workflows/publish-crate.yml](.github/workflows/publish-crate.yml).** It is missing. The + crate *is* registered with release-please, so release-please will happily raise the release PR and + push the tag -- and then nothing will publish it, with no error, because no workflow matches the tag. + **This is a silent failure, which is why it is its own item**: the symptom is a tag that exists, a + changelog that looks right, and a crate that never appears on crates.io. + **Done, in both the tag trigger and the `workflow_dispatch` choices** -- the second matters because + without it the manual escape hatch could not publish the crate either, so there would have been no + way to recover from the first failure by hand. + **And the drift is now checked rather than remembered.** This defect existed because two files had + to agree and nothing compared them; it was found by a reviewer, not by CI. + [tools/check-publishable.ps1](tools/check-publishable.ps1) asserts that every crate release-please + manages has a publish trigger, and runs as its own CI job. The comparison is deliberately + one-directional -- a crate may be publishable without being release-managed, which is what + `windows-placement-probe` is today. Verified by removing the trigger again and watching the check + fail with the crate named. + +- [x] **SH-2.2** -- Update the three `windows-topology-sys = "0.1.0"` pins when 0.2.0 ships. + **RE-PLANNED: this item asked for a decision whose premise was false, and the decision it demanded + does not arise.** It said `windows-ioring-sys` is published against the old topology, "so the + breaking bump obliges updating that dependency and releasing `windows-ioring-sys` too", and that + "the first symptom is a consumer unable to resolve the two together". Neither holds. + **`windows-ioring-sys`'s topology dependency is a dev-dependency.** Cargo reports + `kind=dev`, and the crate's `src/` never names it -- only four files under `examples/` do. Cargo + does not resolve dev-dependencies *of* dependencies, so a consumer of `windows-ioring-sys` never + sees `windows-topology-sys` at all. **Verified rather than reasoned**: a scratch crate depending on + the published `windows-ioring-sys 0.2.0` resolves `windows-sys`, `windows-link`, + `windows-threadpool-sys` and `windows-overlapped-io-sys`, and no topology crate. There is no + resolution conflict to avoid and **no ioring release is obliged**, so the ordering question this + item existed to settle is empty. + **RESOLVED 2026-09-02: there are no pins left to update.** All three were deleted rather than + maintained, once it turned out none of them was doing anything a reader would want: + - `windows-placement-probe` and `windows-platform-probes` -- decided never to be published to a + registry, so **every** workspace dependency is path-only now. Eight version fields deleted + between them, not merely the two topology ones. See those crates' DESIGN-NOTES and `PT-5.3` + (reversed). + - `windows-ioring-sys` -- a `[dev-dependencies]` entry, and cargo omits a versionless + dev-dependency from the published manifest entirely. Deleted as SH-2.5, which also records why + the pin was never checked by anything. + **The whole hazard class is gone, verified rather than argued.** Bumping topology to 0.2.0 now + leaves `cargo metadata` resolving cleanly; before today it failed in three places, and + release-please could only have fixed one of them. + **The gate on SH-3.4 is therefore lifted** -- there is nothing here that had to wait for topology's + manifest to move, because nothing here needs a version at all. + **One consequence to watch for, not to pre-empt.** The `cargo-workspace` plugin bumps dependents of + a bumped package. If it treats the (now versionless) dev-dependency as grounds to bump + `windows-ioring-sys`, an ioring release will happen -- not because one is obliged (it is not; see + above) but because the tooling produced one. That is acceptable if it occurs; it is only a problem + if it is mistaken for evidence that the obligation existed after all. + +- [x] **SH-2.5** -- **Resolved by deleting the pin, and neither of the two fixes this item proposed + was needed -- because its premise was wrong.** It asserted that "at `cargo publish` the + verification build resolves the `version` requirement from crates.io", so a divergence would + surface as a failed publish. **Publish verification does not build examples or tests.** Measured by + packaging this crate: the verification step compiled the library and its real dependencies and + nothing else -- not even `serde_json`, which is a versioned dev-dependency the examples need. So + the pin was never exercised at the one moment it was supposed to matter, and no publish could ever + have failed for this reason. + **What the pin did do was break the workspace**, which is the opposite of the hazard as filed. See + SH-2.2: cargo enforces `path` + `version` agreement at every build regardless of dependency kind, + so this pin against a bumped topology failed `cargo metadata` for every crate here. + **The fix is to delete it, and cargo cooperates**: a versionless dev-dependency is omitted from the + published manifest entirely -- verified by packaging with the version removed and reading the + result, where `windows-topology-sys` appears nowhere. Full `cargo package` with verification then + succeeds. So there is no crates.io requirement left to diverge *from*, and the residual hazard the + item was really about disappears rather than being managed. + What is given up is that the examples are not buildable from the packaged tarball, which they were + only ever incidentally: publish never checked them, and anyone reading an example does it from a + checkout, where they build as before. + **This was the last version pin in the workspace that a topology bump could break.** Verified: with + it gone, `cargo metadata` against a topology bumped to 0.2.0 resolves cleanly. + **The workspace now holds an invariant worth naming, because it is what makes the hazard stay + gone:** every crate that still carries a versioned `path` dependency is one release-please manages + -- `windows-file-enumeration-sys`, `windows-file-watcher`, its example harness, + `windows-ioring-sys`, `windows-namespace-request-sys`, `windows-thread-ambient-sys` and + `windows-threadpool-sys`, thirteen pins between them. Those pins are genuinely needed (each is a + published crate depending on a published crate) *and* the `cargo-workspace` plugin exists to + rewrite exactly them. Every pin the plugin could **not** see is now gone. + A pin outside the plugin's `packages` map is the shape that breaks `main`, so if one is ever added, + it should be questioned rather than maintained. Checking that mechanically would suit + [tools/check-publishable.ps1](tools/check-publishable.ps1), which already reads both the + release-please config and the manifests -- **not done here**, since it adds CI surface and this + item was scoped to the pin. + +- [x] **SH-2.4** -- Clear the **eight rustdoc warnings** in `windows-waitable-queues` before it is + published: an unresolved link to `MIN_CAPACITY`, six links from public documentation to private + items (`Shared::len`, `Doorbell::clear`, `Doorbell`, `BOUNDS`), and one redundant explicit link + target. + Ordinarily out of scope for the item that found them, and in scope here for one reason: **docs.rs is + the face of a first release.** A link that silently resolves to nothing in a workspace build renders + as a dead or missing reference to the first person who ever reads these docs, and a link to a private + item points at a page they cannot open. + **Correction: they did not pre-date this branch, and they were not warnings.** This item said so, on + the reasonable assumption that documentation nobody had touched could not have broken. `main` is + green and the branch is red, so the branch broke them -- most likely the `mpsc` -> `slotwise_mpsc` + rename, which moved every item these links named. And CI denies `broken_intra_doc_links` and + `private_intra_doc_links`, so they were **errors failing every run on the pull request**, not + warnings deferred until publication. Nothing here was blocked on the release; the release was + blocked on this. + **Done.** `MIN_CAPACITY` never existed anywhere -- the prose promised a constant that was never + written -- so that sentence now states the rule itself. The private-item links are delinked rather + than repointed, because a public page cannot link to a page that is not generated. Fixed alongside + five more in `windows-placement-probe` and one in `windows-thread-ambient-sys` that the same job was + failing on; the whole workspace now passes `cargo doc --workspace --all-features` under CI's exact + `RUSTDOCFLAGS`. + +- [ ] **SH-2.3** -- Dry-run both publishes (`cargo publish --dry-run`) from the merge commit, and read + **Run at the branch tip on 2026-09-03 and clean for four crates** -- `windows-waitable-queues`, + `windows-topology-sys`, `windows-file-watcher`, `windows-ioring-sys`. Still open because this item + asks for the **merge commit**, which does not exist yet; the branch-tip run is evidence that nothing + structural blocks it, not a substitute. + What the run settled, so it is not re-litigated: `windows-ioring-sys` publishes while holding + *versionless* path dev-dependencies on `windows-topology-sys` and `windows-guard-alloc`. Those are + stripped from the published manifest, which is exactly what makes them legal -- a version pin there + would fail, naming a crate that is unpublished (`guard-alloc`) or not yet at that version. And no + crate anywhere pins `windows-topology-sys` or `windows-waitable-queues` to a version, so the 0.2.0 + bumps cannot break resolution. The branch adds **no third-party dependency** and changes **no + external dependency version**: the only additions to `Cargo.lock` are the two new workspace crates. + the packaged file list rather than only the exit code. A crate that builds in a workspace can still + fail to package -- excluded files, a path dependency without a version, a README that is not in the + package. + +## M3: land the branch + +**Read this milestone as interleaved with M7 onwards, not before them.** The file's linear order +implies the review rounds follow the merge, which is backwards and was noticed on 2026-09-02: PR #56 +opened on 2026-08-31, nine review rounds arrived while it sat open, and the merge has still not +happened. Review rounds are **reactive** -- they cannot be scheduled after SH-3.4, because merging +ends the pull request they are rounds *of*. + +**Which of those rounds gate the merge: M16 does, M14 and M15 do not.** SH-14.1 is a real defect +that ships **disclosed rather than fixed** +([D-36](crates/windows-waitable-queues/DESIGN-NOTES.md#d-36), delivered by SH-15.8), and everything +open in M15 is follow-on work on the fix. What *did* gate the release was the disclosure, and that +landed. So SH-3.4 may proceed with M14 and M15 still open -- but a reviewer must be told that is +deliberate, which is SH-3.1.1's job below. + +**M16 is the exception, by decision.** Its locality-model work (SH-16.5, SH-16.8, SH-16.9, +SH-16.10) is a merge blocker: the model it replaces is the one `windows-topology-sys` 0.2.0 would +publish, and shipping a public surface that is already known to be the wrong shape is what the +milestone exists to avoid. So SH-3.4 waits on it. + +**Updated 2026-09-03 -- what that work now is, and what discharges the gate.** All six of those items +are superseded into the `MMT-*` plan in +[crates/windows-topology-sys/COMPLETED-CHECKLIST.md](crates/windows-topology-sys/COMPLETED-CHECKLIST.md), so the gate is +discharged by **MMT M2 through M5 landing in this PR**, which is the engineer's direction. Two things +that previously stood in the way are gone: + +- **The design session no longer gates it.** The session's open questions were answered as `D-13` + through `D-21` in + [crates/windows-topology-sys/DESIGN-NOTES.md](crates/windows-topology-sys/DESIGN-NOTES.md); the + `MMT-*` plan is what it produced. Read "the MMT plan concludes" wherever the earlier wording said + "the session concludes". +- **The reshape no longer waits on the planner.** [D-21](crates/windows-topology-sys/DESIGN-NOTES.md#d-21) + establishes that `windows-topology-sys` publishes a refined view of what the platform publishes, + with an adapter absorbing the planner's needs -- so the reshape is self-justified. `topology-planner` + contributes only planning documents to this PR and is deliberately deferred past it. + +- [x] **SH-3.1** -- ~~Open the pull request~~ **-- already open since 2026-08-31 as a draft.** Checked + off as *superseded by events*, not as done: the item asked for something that had already happened + by the time anyone read it, and it stated "54 commits" against a branch now **221 commits** ahead. + Its surviving instruction is **SH-3.1.1** below, which is the part that was never done. + +- [x] **SH-3.1.1** -- **Review the PR as a diff rather than as a memory of having written it, then + mark it ready.** + **Done 2026-09-03 for the review and the description; the promotion is SH-3.1.2, which is the + engineer''s.** The item was right that nobody owned "who decides it is ready", so it is now assigned + rather than left implicit. + The description was rewritten against the diff and the title with it: both dated from 2026-08-31 at + 54 commits, against a branch now 282, and neither mentioned the topology reshape -- the largest + change in the branch -- or the two new crates beside the queue crate. It now states the 17 breaking + changes, the four crates release-please will bump, and what ships knowingly unfinished. + **The review found a real out-of-bounds read** in `cpu_set::decode` (`Type` is at offset 4; the + guard proved four bytes; the full-struct check sat one line *after* the read), fixed in `be64f17` + and witnessed with a guard page -- `0xC0000005` before, clean after. A test could not have caught it: + the decoded output is identical either way. It also found the same defect class unguarded in + `walk::decode`, which is unchanged on this branch and so is queued as `M6` in + [crates/windows-topology-sys/CHECKLIST.md](crates/windows-topology-sys/CHECKLIST.md) rather than + folded into a merging PR. Two release-please surprises became SH-3.4.1 and SH-3.4.2. + +- [x] **SH-3.1.2** -- **The engineer takes PR #56 out of draft**, after reading the diff. + **Done 2026-09-04: the PR reports `isDraft=false` and `mergeable=MERGEABLE`.** Recorded here from + the API rather than from having been told, because the item's whole point was that the decision has + an owner and a visible outcome. + Originally: assigned by + decision on 2026-09-03 rather than left unowned, which is the gap SH-3.1.1 named. A draft cannot be + merged, so this gates SH-3.4 whether or not anything else is outstanding. + Not blocking on SH-3.2/SH-3.3 by necessity -- those can run against an open PR -- but they were + deliberately left unrun at promotion time, so do not read "ready" as "gated". 221 commits across the topology crate, the queue crate and the probes is far more + than fits in a session's recollection, and the branch contains at least one deliberate breaking + change plus several documented reversals of earlier conclusions -- D-18 amended and then + superseded, PT-5.3 reversed, SH-14.3 absorbed, and a crate's version scheme changed from semver to + a date. + **Taking it out of draft is a step this file never named**, and it is the real gate on SH-3.4 + rather than a formality: a draft cannot be merged, and nothing above says who decides it is ready. + **The description must state what is knowingly unfinished**, so a reviewer does not read open + milestones as oversight: SH-14.1 ships disclosed per D-36, M15 is follow-on work on its fix, and + `permit_mpsc` is an experimental non-default module exempt from the crate's semver promise. + **Gated on M16's locality-model work**, which is in scope for this PR by decision. The description + cannot be written before then without being wrong twice over: it would omit the largest change in + the branch, and it would list the locality model among the deferred things when it is not deferred. + So this item now runs *after* SH-16.5/16.8/16.9/16.10, not before them. + +- [ ] **SH-3.2** -- Run the full gate on the merge result, not merely on the branch tip: `cargo fmt + --check`, `cargo clippy --all-targets`, `cargo check --all-targets` in **both** debug and release, + and the in-scope test suites including doctests. Release-mode warnings differ from debug ones, which + is why the milestone discipline names both. + +- [x] **SH-3.3** -- Run the `windows-waitable-queues` sabotage sweep on a clean tree and confirm every + entry still behaves as declared. It is the crate about to become public and the sweep is what has + caught its real defects -- including a lost wakeup that only surfaced because a *baseline* run hung + once in an otherwise green suite. + **Done 2026-09-03: 39 of 39 behave as declared** -- 37 caught, and the two `CONTROL` entries + survived as they are supposed to, which is the manifest checking itself. Baseline green, sources + restored, exit 0. + **But the first run was 36 of 39, and the three failures were the interesting part.** They were not + survivors: they came back `MANIFEST STALE`, meaning the patterns no longer matched and those + sabotages **were not run at all**. Three guards on the crate about to be published were silently + unverified, and a green sweep summary would never have said so -- the tool reports staleness + precisely because a sabotage that does not apply proves nothing. + **This branch''s own work caused the drift**, which is why it had to be caught here rather than + assumed: `slotwise_mpsc frees a slot one short of the next lap` broke when positions widened to 64 + bits and the expression gained an `as Position` cast; `slotwise_mpsc accepts a capacity of one` + broke when `WRAPPING_MAX_CAPACITY` was renamed `MAX_ADMISSIBLE_CAPACITY`; and `reserving_mpsc: + reserve does not check for room` broke when the no-room path grew its stale-`word` retry, turning a + three-line block into twelve and giving `has_room_beyond_reservations` a second call site in + `push`. The repaired pattern anchors on the comment that is unique to `reserve`, so the two sites + cannot be confused. + Each repair was re-run individually and **caught** before the full sweep, so the fix restored the + check rather than merely restoring the match. +- [x] **SH-3.3.1** -- **Finish and verify the memory-ordering discipline sweep.** + **Done 2026-09-04: 308 lib + 8 doc + 1 integration green in 0.41s, and the sabotage sweep reports + all 39 behaving as declared** with both `CONTROL` entries surviving and no `MANIFEST STALE`. + **The verification found a real defect in this work.** Waiting for a fresh `head` made + `the_high_water_mark_never_exceeds_the_capacity` hang -- it wrote an impossible `head` + (`u32::MAX - 10`) to force an over-report and restored it only *after* `send` returned, so the new + wait spun for 314s of CPU rather than proceeding. The state that test constructs is one the fixed + code correctly refuses. It was replaced by `publish_waits_for_a_head_that_has_freed_the_slot`, + which asserts the guarantee instead of the mitigation, and which was **sabotage-verified**: with + the wait reduced back to a single load it fails in 0.05s with the message written for it. The + clamp is kept as documented defence-in-depth and is now unreachable by construction, which is + recorded at the site so a mutation run's survivor there is read as unreachable code rather than a + missing test. + **The rule, from the engineer, recorded as `D-38`/`D-39`/`D-40` in + [DESIGN-NOTES.md](crates/windows-waitable-queues/DESIGN-NOTES.md):** acquire/release discipline is + never intermixed with relaxed on the same atomic. A relaxed operation is *unordered* -- unanchored + with respect to the ordered operations on that object and free to be moved by the optimizer or the + processor -- so it is not pinned to its textual site and statement-order reasoning about it is not a + weak argument but no argument. It is still fully **atomic**, and that half is load-bearing + independently (`D-40`): the claim word packs two `u32`s into a `u64`, so a torn read would yield a + state the queue was never in. **When the two repairs differ, promote the load** -- an unnecessary + acquire is a benchmark someone can bring later, an unnecessary relaxed is a defect on hardware we do + not own. + **Why no test here can see this:** on x86-64 TSO a decorative `Acquire` and a `Relaxed` load emit + nearly the same code, which is the same blindness `D-31` measured. On AArch64 they are `ldar` and + `ldr`, and CI builds `aarch64-pc-windows-msvc`. + **What changed** (audit output in `D-38`): four atomics had acquire loads with **no release write + anywhere**, so the acquires paired with nothing -- `reserving_mpsc`'s claim word, `permit_mpsc`'s + `tail` and `head`, `slotwise_mpsc`'s `tail`; those are now uniformly relaxed. Four had a real + release store with relaxed loads mixed in -- `reserving_mpsc::head`, `slotwise_mpsc::head`, + `spsc::head`, `spsc::tail`; those loads were promoted to acquire. `permit_mpsc`'s permit counter had + a relaxed operation inside a genuinely load-bearing edge and is now `Release`. `permit_mpsc`'s `Drop` + was converted to `get_mut()`, matching `reserving_mpsc`, which removes the question instead of + answering it. The `producers` refcounts are deliberately unchanged (`D-39`). + **Tooling note:** the cargo-mcp server's `cargo_test` was wedged for this item -- it accepted the + call and never spawned a cargo process (confirmed repeatedly via `Get-Process`), while + `cargo_check`, `cargo_fmt` and `cargo_clippy` worked normally on the same process, and + `cargo_test` with `no_run` also worked. So the failure is in the test-execution phase, not the + build phase, and not a stale binary: the running process, the installed extension, the VSIX and + the source repo are all `cargo-mcp` 0.12.1. The engineer authorised the terminal as a fallback + while diagnosing it. + **This also settles half of SH-4.7**, whose memory-ordering half is subsumed by the rule above. + + +- [ ] **SH-3.4** -- Merge to `main`, and confirm release-please raises a release PR proposing + **0.2.0** for the topology crate. If it proposes 0.1.1, the breaking-change marker did not take and + the version would silently understate the break -- fix the marker rather than editing the version by + hand, or the next break will do the same thing. + **This item named one crate; SH-3.1.1's diff review found four will be bumped.** The config sets + both `bump-minor-pre-major` and `bump-patch-for-minor-pre-major`, so for a 0.x crate a breaking + change bumps the **minor** and everything else the **patch**. Check all four, not just topology: + + | Crate | From | Expect | Driven by | + |---|---|---|---| + | `windows-topology-sys` | 0.1.0 | **0.2.0** | 9 breaking | + | `windows-waitable-queues` | 0.0.1 | **0.1.0** | 6 breaking, from a corrected starting point -- SH-3.4.1 | + | `windows-ioring-sys` | 0.2.0 | **0.3.0** (shipped 2026-09-05) | 2 breaking by path attribution -- SH-3.4.2 | + | `windows-file-watcher` | 0.1.3 | **0.2.0** | 1 breaking, the reopen-by-id removal | + | `windows-thread-ambient-sys` | 0.2.0 | **0.2.1** | 3 `fix:` commits scoped to other crates that changed its `src/` | + | `windows-file-watcher-example-test-harness` | 0.1.2 | **0.1.3** | not its own commits -- the `cargo-workspace` plugin, below | + + **The other six crates get no release at all**, and that is correct rather than a gap: + `wtf-string`, `windows-threadpool-sys`, `windows-overlapped-io-sys`, + `windows-impersonation-token-sys`, `windows-file-enumeration-sys` and + `windows-namespace-request-sys` have only `test:`, `docs:` and `refactor:` commits since their last + tags. Only `feat`, `fix` and breaking changes trigger a release; the rest are changelog-only. An + earlier version of this item assumed any conventional commit bumps a patch, which is wrong. + + **The harness bump is the `cargo-workspace` plugin, and it is verified rather than assumed.** The + harness has a *runtime* dependency `windows-file-watcher = "0.1.3"`, which `^0.1.3` does **not** + satisfy once that crate reaches 0.2.0 -- so something must rewrite it or the published manifest is + broken. The plugin does exactly that, as this repository''s own history shows: `8b37f9f` and + `5f7f6af` (both `chore: release main`) each bumped the harness *and* rewrote its + `windows-file-watcher` requirement in the same commit. Confirm it happened again rather than + trusting it; it is the one dependency edge in the workspace that a bump can actually break. + +- [x] **SH-3.4.1** -- **Decide `windows-waitable-queues`' first published version before the release + PR merges.** + **DECIDED 2026-09-03: the crate starts at `0.0.1`, so its first published version is `0.1.0`.** + Not the `Release-As` route this item first proposed -- that was the wrong instrument, and the two + cases differ in a way worth stating. The ioring pin *asserts* something falsifiable ("no break + happened"), which is why it needs a guard. Here nothing is asserted: the crate has **no release + tag**, so its manifest entry was never a record of a release, only a starting point -- and the + starting point was simply wrong. Setting it to `0.0.1` corrects it rather than overriding it. + Three consequences, in order of importance: with `bump-minor-pre-major` a breaking change takes + `0.0.1` to **`0.1.0`**, which is what a first release should look like; it is **robust** where a + fixed `Release-As` is not, since further breaking commits before release still yield `0.1.0`; and + it needs no footer, no pin, and nothing to remember. + **`0.0.0` would have been the wrong value** -- release-please special-cases it and the pre-major + options stop applying, jumping to `1.0.0` + ([googleapis/release-please#2087](https://github.com/googleapis/release-please/issues/2087)). + Safe to lower: both consumers are versionless path dependencies on `publish = false` crates, so no + requirement anywhere can fail to resolve. Verified by `cargo metadata` and a clean + `cargo publish --dry-run`. + The crate is not on crates.io, sits at 0.1.0 in the manifest, and carries six `!` + commits, so release-please will propose **0.2.0** and 0.1.0 will never exist. The `!` markers are + honest about the branch's history but describe an API that was never published, so nothing can + break. Either accept 0.2.0 as the first version, or force the first release with `Release-As: 0.1.0`. + Not a defect -- a naming decision that is cheap now and permanent afterwards. + +- [x] **SH-3.4.2** -- **Decide what to do about `windows-ioring-sys`' unearned breaking bump.** + **OUTCOME: the pin was never applied, and 0.3.0 shipped on 2026-09-05.** The decision recorded here + on 2026-09-03 was to pin ioring to 0.2.1 with a `Release-As: 0.2.1` footer on `cdce13b`. That commit + stayed on this branch; the release ran from `main` without it, so tag `windows-ioring-sys-v0.3.0` + exists and the crate's CHANGELOG carries the very entry the pin was written to avoid -- a + BREAKING CHANGES heading citing `**topology:** reshape the topology model around observed domains`. + The pin has been removed from both branches rather than left stating an intention that events have + overtaken: a `Release-As: 0.2.1` carried forward now would ask release-please to regress a crate + already at 0.3.0. + + **What survives is the general lesson, and it is now enforced rather than remembered.** + [tools/check-commit-scope.ps1](tools/check-commit-scope.ps1) is on `main` and flags a + release-triggering commit that spans more than one released crate, which is the mechanism that + produced this bump. The reasoning below about path attribution, the three-commit deprecation dance + for a cross-crate rename, and the measured cost of a blanket one-crate-per-commit rule is unchanged + and still applies; only the pin is gone. + + Release-please attributes a commit by the **paths it touches**, not by its Conventional Commits scope. + Two `feat(topology)!` commits (`b9e0c35`, `36e397d`) touched `crates/windows-ioring-sys/`, so it + will take a breaking **0.3.0**. Its public API did not break: the only changes there were one + doc-comment heading in `lib.rs` (`# Topology guidance` -> `# MachineMemoryTopology guidance`) and + code under `examples/ring_copy/`. A 0.3.0 whose CHANGELOG cites breaking changes would misinform + consumers who experience none. **The general lesson outlives this instance**: a breaking commit that + incidentally edits a second crate's files bumps that crate as breaking too, so either keep such + commits path-clean or expect to correct the bump. + + **This is not hypothetical -- it has already shipped.** `windows-ioring-sys`'s existing CHANGELOG + carries two `**guard-alloc:**` entries (`983afbc`, `36ecd8a`), which landed there because those + commits touched `crates/windows-ioring-sys/tests/registration.rs`; guard-alloc is a dev-dependency, + so exercising it meant editing ioring's tests. Same mechanism, one release earlier, unnoticed. + + **Splitting the commit is the right shape, and a naive two-way split is wrong.** The obvious fix -- + "rename in topology, then update the ioring example in a `chore:` commit" -- produces a commit that + **does not compile**: `examples/ring_copy/plan.rs` does `use windows_topology_sys::{..., Topology}` + and `pub fn build_plan(topology: &Topology, ...)`, so a topology-only rename breaks it until the + follow-up lands. CI would not catch it (it builds the PR head and main's tip, not each commit), but + `git bisect` across that range would. + The version that compiles at every step is **three** commits, the deprecation dance: (1) add the new + name with the old one kept as an alias -- additive, `feat(topology)`, not breaking; (2) move the + consumer to the new name -- `chore(ioring)`, no novelty; (3) delete the alias -- `feat(topology)!`, + touching only topology paths. Adopt that shape for a cross-crate rename from now on. + + **Do not rewrite these two commits.** They are 15 and 44 deep in a pushed branch with an open PR; + `git rebase -i` is forbidden by this repository's own terminal rules; and retrofitting the alias + step would mean fabricating a deprecation that never happened, for one changelog line. + + **The option that was considered and NOT taken:** a `Release-As: 0.2.1` footer on a commit touching + **only** `crates/windows-ioring-sys/`. Verified against release-please's manifest-mode documentation + that the footer is applied **per package, by the paths the commit touches**, so it would have pinned + ioring without disturbing the other five bumps. Recorded because the mechanism is worth knowing -- + but see the outcome at the head of this item: the footer never reached `main`, 0.3.0 shipped, and + the pin has since been removed rather than carried forward against a released version. + + **Measured, because the rule had to be affordable before it could be recommended.** Nine + release-triggering commits on this branch span more than one *released* crate -- and only two of + those are the ioring case. **Seven genuinely changed both crates'' source**, so a blanket + "one crate per commit" rule would have forced non-compiling commits seven times to fix a problem + that existed twice. That is why the standing rule flags rather than blocks. + The measurement also **found a hole in the first version of the check**. `983afbc` + (`feat(guard-alloc)`) touched only *one* released crate -- ioring -- because guard-alloc is + `publish = false`; a "spans more than one crate" test misses it entirely, yet it is the commit that + actually shipped a wrong entry. The rule is therefore two-pronged: flag a commit that spans several + released crates, **and** one whose `(scope)` names a crate other than the one it will be attributed + to. + Enforced going forward by [tools/check-commit-scope.ps1](tools/check-commit-scope.ps1), wired into + the pre-commit gate in [.github/copilot-instructions.md](.github/copilot-instructions.md) so an + unattended session runs it rather than being expected to remember the rule. + **The gate this used to hold over SH-2.2 is lifted** -- that item is closed, having had nothing + left to do once the pins were deleted rather than maintained. + **No longer carries a pin hazard.** An earlier version of this item warned that the PR must not be + merged until `windows-ioring-sys`'s topology pin read 0.2.0, because a `^0.1.0` requirement against + a path crate at 0.2.0 fails resolution and would have landed a red `main`. That was true when + written and is now moot: **all three such pins were deleted on 2026-09-02** (SH-2.2, SH-2.5), and a + topology bump was re-verified to resolve cleanly with none of them present. Nothing in the release + PR needs checking beyond the version number itself. + +## M4: release + +- [ ] **SH-4.1** -- Release `windows-topology-sys` 0.2.0 and confirm it appears on crates.io and builds + on docs.rs. Docs.rs builds under its own configuration, so a crate that documents locally can still + fail there. + **The obligation to [CHECKLIST-placement-tool.md](CHECKLIST-placement-tool.md) is void as of + 2026-09-02.** This item used to half-unblock PT-5.3 -- publishing the tool to crates.io -- and that + decision was reversed: the tool is never published to a registry, so there is nothing here to + unblock and no gate bullet to edit. It never gated the tool's **GitHub binaries**, which CI builds + from this repository through `path` dependencies, and those remain the only distribution. + +- [ ] **SH-4.2** -- Update `windows-ioring-sys` to depend on the published 0.2.0 and release it, per + the order settled in SH-2.2. + +- [ ] **SH-4.3** -- Release `windows-waitable-queues` at **the version SH-3.4.1 settles**, with + SH-2.1's fix in place. Confirm the tag triggered a publish rather than assuming it did. + **This item said "0.1.0" and that is not what release-please will propose.** Six breaking commits + against a manifest version of 0.1.0 yields **0.2.0** under `bump-minor-pre-major`, on a crate that + has never been published -- so 0.1.0 would be skipped entirely. SH-3.4.1 decides whether to accept + that or force 0.1.0; this item follows it rather than asserting a version of its own. + **The gate on [CHECKLIST-placement-tool.md](CHECKLIST-placement-tool.md) PT-5.3 is void as of + 2026-09-02** -- that decision was reversed and the tool is never published to a registry, so there + is no gate to lift and no bullet to edit. The tool's GitHub binaries never waited on this. + Blocked by SH-1.1, and by M31.6 as well if SH-1.2 decided that it gates. + +- [ ] **SH-4.4** -- **State the supported Windows baseline in the crates this PR did not release.** + `windows-topology-sys` was corrected in PR #56 (it claimed Vista / Server 2008 while statically + importing a Windows 10 API), and the engineer''s ruling is that a crate should claim **Windows 11 + and the matching server release** -- the floor that is *tested*, not the oldest the APIs might + work on. **Scoped deliberately to the releasing crates at the time**, so the rest are queued here + rather than changed under a PR that does not publish them. + To review, one crate at a time: `windows-threadpool-sys`, `windows-overlapped-io-sys`, + `windows-file-enumeration-sys`, `windows-impersonation-token-sys`, `windows-namespace-request-sys`, + `wtf-string`, and `windows-file-watcher`. + **Two traps found doing the first one.** A statement about *when an API appeared* is not a baseline + claim and must not be rewritten -- `windows-ioring-sys`'' "Windows 11 and Server 2022 added + `IoRing`" and `windows-file-watcher`''s "supported from Windows 10 version 1803 onward" are both + correct as API facts. And **Server 2022 is not the Windows 11 counterpart**: it is built on the + Windows 10 "Iron" codebase (build 20348), while Server 2025 shares Windows 11 24H2''s build 26100. + A crate claiming a Windows 11 floor pairs with **Server 2025**. + +- [ ] **SH-4.5** -- **Propagate enumeration incompleteness into the placement probe''s traceability + state.** + Raised in PR #56 review. `places_from_topology` ignores + `MachineMemoryTopology::enumeration_anomalies`, and `SubmissionRecord::is_fully_traceable` gates + only on `build.is_official()` and two `Provenance::Measured` checks -- which a `discover()` result + satisfies even when records were dropped. A truncated enumeration can therefore be filed as an + **unremarkable small machine**, which is the one thing a measurement tool must not do. + **Deliberately not rushed into PR #56**: the honest fix records the anomaly count *in the record* + and gates that state on it, and the record carries a **versioned schema** (see the `collapse the + record schema back to v1` commit), so this is a schema decision rather than a one-line guard. + Downgrading `Provenance` instead would be wrong -- per [D-22](crates/windows-topology-sys/DESIGN-NOTES.md#d-22) + provenance records how the object was *obtained*, not how complete it is. + The gap is newly reachable because `enumeration_anomalies` itself is new in this PR; before it, + there was nothing to propagate. + +- [ ] **SH-4.6** -- **Rule on whether `experimental-permit-claim` may be published at all.** + Raised in PR #56 review, and the reviewer is **technically right**: a public Cargo feature and its + `pub mod` are observable API even when non-default, so a consumer can enable the feature and depend + on `permit_mpsc`. Deleting it in a later compatible release would break that build, and no amount + of documentation exempts it from semver -- which is exactly what `SH-15.6` currently plans to do. + Three options, and this is the engineer''s call: commit to compatibility for the published feature; + move the experiment behind `#[doc(hidden)]` and a `cfg` that is not a Cargo feature, so it is not + observable API; or keep it out of the published crate entirely and measure it from a path + dependency. The disclosure in the module docs is honest but does not settle the semver question. + +- [x] **SH-4.7** -- **Two `permit_mpsc` findings from the PR #56 review, both now settled. The + contract half was fixed on 2026-09-04: `Reservation::send` returns `Disconnected` instead of + publishing into a ring nobody will read. Originally: one of which contradicts a + prior review.** Both are in the experimental module, so neither blocks the release, and both should + be settled before `SH-15.6` decides the module''s fate. + **Contract:** `Reservation::send` publishes unconditionally even when the consumer is already + dropped, so the caller gets no indication and the item is discarded at teardown -- where + `reserving_mpsc::Reservation::send` returns `Disconnected` with the item. The module claims only + the *admission* protocol differs, so this is a divergence it does not disclose. + **Memory ordering -- settled by SH-3.3.1, 2026-09-04.** The reviewer argued the relaxed ticket + operation gives no acquire edge from the consumer''s most recent `release_permit`; a prior round + concluded the opposite, that the release sequence supplies it. Both were reasoning about whether + the edge could be *rescued*, and the answer taken was to stop depending on the rescue: the permit + counter''s overdraw undo is now `Release` rather than `Relaxed`, so the counter carries one + discipline throughout and the edge holds without appealing to the release-sequence rule -- a rule + that was narrowed once already, when C++20 dropped same-thread relaxed stores from it. Recorded as + `D-38` in [DESIGN-NOTES.md](crates/windows-waitable-queues/DESIGN-NOTES.md). The cost is one + `stlxr` over `stxr` on AArch64, on the contended slow path. + +- [ ] **SH-4.8** -- **Six more PR #56 review findings, none blocking the release.** Replied to and + resolved on the PR; recorded here so none is lost. + **Memory safety in a test (do first).** `reopen_by_id_cannot_be_watched.rs:153` returns early on a + zero return with `ERROR_IO_PENDING` -- a *successful* overlapped submission -- dropping `buffer` + and `overlapped` while the queued IRP may still write through both. Production + `classify_submission` handles this case; the test does not. Wait for or cancel the operation + before dropping. + **`reserving_mpsc`''s `head` acquire load** (`reserving_mpsc.rs:608`), which **contradicts an + earlier review round** that called it the only acquire edge `Reservation::send`''s non-atomic slot + write has. This is a shipping shape, so settle it with a written argument naming the execution and + the edge -- do not strengthen the ordering to be safe, which hides whichever model is wrong. + **Overlapping domains resolved by iteration order**: `memory_domain_of` returns the first match and + the core map lets a later domain replace an earlier one. For hand-built and deserialized + topologies -- which the API explicitly accepts -- an overlap is an *ambiguity*, and + `memory_domain_of` is the sharper case because its value reaches `VirtualAllocExNuma`. + **A live-host test asserting cross-API agreement** (`cpu_set/tests.rs:242`) contradicts the model''s + premise that CPU Sets may disagree with the walk; it is a latent failure on untried hardware. + Assert that both observations are *recorded*, not that they agree. + **`release-placement-probe.yml` gating -- DONE 2026-09-04.** `workflow_dispatch` against an + existing release tag satisfied the tag-prefix condition, so a build-only run could create or modify + a release. All four publishing guards now require `github.event_name == ''push''`, and the trigger''s + own comment no longer claims the restriction is "inherent" -- it was not, which is precisely how + the hole survived. Raised again by Copilot at review `5117514238` before it was fixed. + **`queue_contention.rs:241`** starts its clock without ordering against workers entering their + loops, so a descheduled coordinator under-reports the baseline -- the optimistic direction, in a + probe whose numbers are quoted as evidence. + +- [ ] **SH-4.10** -- **The fingerprint collapses multi-source observations by last-write-wins, and + privileges the relationship walk.** Raised by Copilot across reviews `5117032381`, `5117514238` and + `5117911029` -- three rounds, one root, so it is recorded once here rather than three times. + `Source` has **no trust ordering** by construction + ([observation.rs](crates/windows-topology-sys/src/observation.rs)), and `fold_in_cpu_sets` + deliberately keeps differing memberships as *separate domains* so a disagreement survives into the + model. `places_from_topology` then flattens that with `HashMap::insert`, so: + (a) a processor in two core domains takes whichever domain is visited last, making core and + efficiency-class labels **iteration-order dependent**; + (b) the same for NUMA membership; + (c) a domain observed only by CPU Sets yields `MissingPlacement::NumaNode` even though it carries an + OS-reported node label, so a real observation is **discarded** for coming from the wrong source. + A reported disagreement is thereby converted into a silent arbitrary choice -- which is the defect + the topology model was reshaped to prevent, reappearing one layer up in its first consumer. + **The fix is a design decision, not a patch**: accept a sole or agreed label, and refuse the + measurement as ambiguous when the sources actually conflict. Refusing is consistent with this + seam''s existing rule that an invented value is worse than a lost one. Relates to `SH-4.8`''s + overlapping-domain finding, which is the same shape one layer down in `memory_domain_of`. + +- [ ] **SH-4.11** -- **The record''s anti-splice guard compares fingerprints, which cannot establish + what it claims.** Raised by Copilot at review `5117911029`. + [main.rs](crates/windows-placement-probe/src/bin/placement_probe/main.rs) compares + `observation.host != host` to refuse a record whose announced shape and measured rows came from two + different readings. But `Fingerprint` records only **marginal sizes** and its own documentation says + plainly that equal fingerprints may have different cache/class/NUMA *intersections*. A topology that + changed between the two discoveries while preserving every count passes the check, and the record + then combines the first topology with rows measured from the second -- the exact splice the guard + exists to prevent. + The honest fix is to compare a canonical **placement signature**, or the derived `places` + themselves, rather than the fingerprint. That signature is already queued as `PT-6.1` in + [CHECKLIST-placement-tool.md](CHECKLIST-placement-tool.md), so this item is gated on it and the two + should be done together. + **Not a reason to weaken the guard**: it still catches every change that alters a count, which is + every case observed so far. It is weaker than its own comment claims, and the comment must be + corrected even if the check is not. + +- [ ] **SH-4.12** -- **`ring_copy`''s `ByL3` policy restates the partition rule instead of asking for + it.** Raised by Copilot at reviews `5116772196` and `5116886015`. + [policy.rs](crates/windows-ioring-sys/examples/ring_copy/policy.rs) selects domains with + `matches!(domain.kind, DomainKind::Cache { level: 3, .. })`. The reshaped topology model makes + `outermost_partitioning_cache()` the one definition of which cache level partitions a machine, and + level numbering is explicitly **not** the ordering contract. So this consumer can produce + **overlapping ring domains** where two cache kinds are reported at level 3, and degrades to a single + whole-machine domain on a host whose outermost partition is at some other level -- neither of which + the policy''s own documentation admits to. + This is the consumer-side twin of the platform-integrity rule: bind to the specified primitive, not + to the level number that happens to be L3 on today''s hardware. The fix renames the policy as well + as changing it, since `byl3` is a user-facing CLI value that would no longer describe what it does. + +- [ ] **SH-4.13** -- **`ProcessorSet` cannot represent every `u8` processor id, and the public API + cannot uphold both "every processor" and "no abort".** Raised by Copilot across three unresolved + review threads -- `topology.rs:100`, `topology.rs:946` and `granularity.rs:94` -- which share one + root and are recorded once here. The reviewer''s own phrasing on the third is the clearest statement + of the problem and is quoted deliberately. + `MachineMemoryTopology` and `Processor::id` are **public and constructible**, and the derived + deserializer accepts `number: 255` under `processors` even though `ProcessorSet` rejects the same id + under `domains`. Such a topology then **panics** in `machine_processors()`, and on a 32-bit target a + processor number of 40 panics in `granularity`''s `insert` before the argument guard can answer. + **Not fixable site by site**: validating each lookup closes the path that was reported and leaves + the next one open, because the gap is between what a `Processor` may say and what a `ProcessorSet` + can hold. The decision is which of the two moves -- widen `ProcessorSet` to the full `u8` range, or + make invalid topology data *reportable* rather than fatal (the `Observed`/anomaly shape this crate + already uses for "the platform said something we cannot represent"). + **Precedent, and the reason this is not theoretical:** the proximity panic fixed in `c072a8a` was + the same family -- a panic reachable from public, constructible input -- and it was real. + Whichever way it goes, add the malformed-description case to the deserialization tests, which is the + path that currently has no coverage at all. + +- [ ] **SH-4.14** -- **`probe-long-path-aware` builds without its manifest on `windows-gnu`.** Raised + by Copilot at `crates/windows-platform-probes/build.rs:29`. The resource is embedded only for MSVC, + so on a GNU target the "aware" and "unaware" binaries measure the **same** configuration while their + names and their reports claim opposite intent. + That is worse than failing to build: the pair exists to isolate one variable, and a report stating a + conclusion the binary did not test is a wrong measurement rather than a missing one. **Fail clearly + on non-MSVC** is the preferred answer over embedding the resource for GNU, since the experiment is + meaningless unless the two halves genuinely differ. + **Gated on CI, deliberately.** This workspace builds MSVC targets only, so a GNU fix would ship + unexercised -- the honest sequencing is to add the target to CI first and then make this fail + against it, rather than writing a guard nothing runs. + +- [ ] **SH-4.9** -- **`tools/check-publishable.ps1`: three findings with one root.** Its checks are + **text searches standing in for structural facts**, which is how a check goes quietly vacuous. + An unanchored pattern is satisfied by a *commented-out* assignment, so CI would believe the + dependency registry exists while the shell never defines it -- re-creating the publish race the + check exists to prevent. The trigger/choice searches scan every YAML list rather than + `on.push.tags` and `on.workflow_dispatch.inputs.crate.options`, so an identical entry in a matrix + keeps them green after the real trigger is removed. And the description still says the placement + probe is "not on crates.io yet", implying a registry publication that `publish = false` ruled out. + Fix together by parsing the workflow structure rather than adding anchors until the next false + green. + +## M5: verify from outside the workspace + +- [ ] **SH-5.1** -- In a scratch project **outside this repository**, depend on both crates from + crates.io and build something that uses each. This is the first exercise of the crates as + *dependencies* rather than as path members, and it is where a missing `version` on a path dependency, + an unexported type, or a feature that only resolves inside the workspace will show up. + +- [ ] **SH-5.2** -- Confirm the published `windows-topology-sys` still reports `Provenance::Measured` + from `discover()` when consumed as a dependency, and that a `MachineMemoryTopology::default()` is `Synthetic`. + The provenance rules are the newest thing in the crate and the least exercised outside it. + +## M6: long-running validation + +**Numbered last among the release milestones, and it gates SH-4.3 all the same.** (It is no longer +*positioned* last: M14 and M15 were appended afterwards, as review rounds that had to go somewhere.) +`windows-waitable-queues` +0.1.0 does not publish until this milestone is done. The reasoning is in SH-1.2 / D-31: the crate ships +without machine-checked orderings, and long-running validation is part of what it owes instead. + +**What a pass here does not mean, stated once and repeated in the tool's own output.** Hours of green +stress says nothing about memory orderings. That is measured, not cautious: weakening the producer's +`Acquire` to `Relaxed` left the whole suite green. A stress tool that omits this becomes false comfort +-- someone points at a long clean run and concludes the orderings are fine, which is exactly the claim +D-31 says cannot be supported. + +- [ ] **SH-6.1** -- **The wraparound scenario, which is the one reachable correctness gap.** + `reserving_mpsc` packs its position into 32 bits, so it wraps after 2^32 pushes -- **between 37 + seconds and about four minutes** at this crate's own measured rates, and reachable in production + within hours. The range is [D-26](crates/windows-waitable-queues/DESIGN-NOTES.md#d-26)'s isolated + table read as a wrap time: 8.6 ns/push with one producer is 116M/s, so 2^32 is 37 s; 28.0 ns with + two is 35.7M/s, so 120 s; 56.9 ns with thirty-two is 17.6M/s, so 244 s. An earlier version of this + item said only "about two minutes", which is the two-producer figure quoted as though it were the + whole story. + **CORRECTION (review round nine).** An earlier version of this item said `spsc` and + `slotwise_mpsc` "use `usize` positions and cannot be driven there at all", and that is **false on a + 32-bit target**, where `usize` *is* 32 bits. The claim was written from a 64-bit reading and never + re-checked against the 32-bit support the crate otherwise takes seriously enough to have a + dedicated `BOUNDS_MAX` derivation and a `const` assertion for. See SH-14.1 and SH-14.2, which are + about the *correctness* hole this testing gap was hiding. + **SUPERSEDED IN PART, and the scope is now narrower than the paragraph above implies.** + `slotwise_mpsc` **no longer reaches the wrap on any target**: SH-14.2 widened its positions to a + named `Position = u64` everywhere, so it needs 2^64 claims. `spsc` never had a compare-exchange + claim to race. So the only shape this item still has to drive across the wrap is + `reserving_mpsc` -- and if SH-15.6 adopts the permit claim, whose ticket is likewise `u64`, that + one goes too and this item's subject disappears entirely. + **A wrap test alone cannot witness SH-14.1**, which is worth stating here because this item reads + as though it could. Crossing 2^32 exercises the arithmetic; the defect additionally needs a + producer *held* between its room check and its claim. That seam is SH-15.7's, and the two are + complementary rather than alternatives. + What exists today is *ring* wraparound (positions cycling through slots) and the packing arithmetic + checked at the boundary; what does not is the queue actually crossing 2^32 end to end. **Tracking + every item is impossible at that count**, so the invariants are the cheap ones: per-producer sequence + numbers strictly increasing in consumption order, and an exact total count. O(producers) memory + rather than O(items). + +- [ ] **SH-6.2** -- **Diagnostic history, merged by position rather than by a clock.** Unseeded is + correct here: the scheduler is the source of variation, not the PRNG, so a seed would make the inputs + reproducible while the interleaving that caused the failure stays unreproducible -- the appearance of + determinism with none of the substance. What is needed is **reconstructability**. + Each thread keeps a small lock-free ring of recent records: thread, operation, position, value, + outcome. **The merge needs no clock and no global counter**, because the queue under test already + carries a total order -- its positions -- so records sort by position after the fact. A global + sequence number would give a true order and perturb the hot path it is trying to observe; a + timestamp costs a clock read per operation. Both were considered and neither is needed. + The one case positions do not order is a *refused* push, which has no position; record the position + it attempted and mark it refused. + +- [ ] **SH-6.3** -- **Detect the failures worth detecting**, and dump the history on any of them: item + loss or duplication, per-producer order violation, a panic in any thread, and **no progress**. The + last needs a watchdog thread against a progress counter, and is the case that most needs history and + is least served by a seed -- a hang leaves no assertion behind, only a stuck process. + +- [ ] **SH-6.4** -- **Cover all three shapes and the doorbell.** The doorbell is the point: SH-1.2 + established that a model checker *cannot* cover it, because its correctness is an atomic mirror flag + interleaving with real `SetEvent`/`ResetEvent` calls. Stress is one of the few instruments that + exercises that at all. D-15's lost wakeup surfaced because a baseline run hung **once**; more hours + of running is the only lever we have on that class. + Include the parking path, not just the polling one -- a consumer that never parks never exercises + the doorbell protocol that D-9 and D-15 are about. + +- [ ] **SH-6.5** -- **Ship it as a tool, not only as a test.** A binary with duration and concurrency + knobs, so a user can stress *their* hardware. That matters concretely: x64 and ARM64 have already + disagreed once about this crate's behaviour, and no test we run here covers a machine we do not own. + Keep a short in-suite smoke run over the same engine so the code cannot rot, and keep it out of the + fast unit suite, which must stay under a second. + +> **M7 through M13 -- seven PR #56 review rounds -- are complete and archived.** Moved to +> [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md) on 2026-09-02. M14 and M15 stay below because they +> carry open work. + +**Everything from here down happened, and happens, *inside* M3 rather than after it.** These are +rounds of review on PR #56, which opened on 2026-08-31 and has not merged; they are reactive work +that arrives while a pull request is open, so their position at the end of this file is numbering +order and not a schedule. Reading it as a schedule would put the review of a pull request after the +merge that closes it. + +**M14 and M15 do not gate SH-3.4; M16 does.** For M14 and M15 that is a decision rather than an +oversight: the defect they concern ships **disclosed rather than fixed** +([D-36](crates/windows-waitable-queues/DESIGN-NOTES.md#d-36)), and the disclosure -- which *was* the +release blocker -- landed at SH-15.8. SH-3.1.1 is responsible for saying so in the pull request +description, so a reviewer does not mistake those open milestones for unfinished business. + +**M16's locality-model work is a merge blocker**, by decision rather than by drift -- an earlier +revision of this file listed it alongside the others as non-blocking, and that is corrected here. +The reason it differs: M14 and M15 concern a defect in an *implementation*, which can ship +disclosed, whereas M16 concerns the *shape of the public model* `windows-topology-sys` 0.2.0 would +publish. A disclosed implementation defect can be fixed in 0.2.1; a published model cannot be +reshaped without another break. + +## M14: PR #56 ninth review round -- an ABA hole the wrap test would not have caught + +Two findings, both verified by derivation against the source. They are **not** the wrap gap SH-6.1 +already tracked: that item is about *testing* the wrap, and its planned stress would not have found +either of these, because neither requires the wrap alone -- each requires a producer to remain +stalled **across** it, inside a window a few instructions wide. + +**The shared shape of both.** A producer decides a slot is writable, is suspended, and resumes after +the claim counter has made a full lap back to the bit pattern it read. Its compare-exchange then +succeeds against a value that is numerically equal but logically a whole generation later, and the +decision it is about to act on was made against the *earlier* generation. The exchange protects the +counter; nothing protects the decision. + +- [ ] **SH-14.1** -- **`reserving_mpsc` can overwrite a live slot after 2^32 pushes, on every + target.** `POSITION_BITS` is 32 by construction -- the claim word is a `u64` split into a 32-bit + reservation count and a 32-bit position -- so this does **not** depend on a 32-bit `usize` and is + not a 32-bit-only concern. + The sequence: a producer reads `word = (reserved, position)` and calls + `has_room_beyond_reservations`, which reads `head` and returns room. It stalls. Other producers + claim 2^32 positions, wrapping `position` back to the value it read; with `reserved` at its steady + state (commonly zero), the *whole word* recurs. The stalled producer's + `compare_exchange_weak(word, ...)` now succeeds, and it publishes into a slot whose room was + decided against a `head` that has since advanced -- so the slot may hold an item the consumer has + not taken. The SAFETY comment above `publish` ("no other producer can also have claimed [this + position]") remains true and is not the property that fails; the failing property is that the slot + was free. + 2^32 pushes is **37 seconds to about four minutes at this crate's measured rates** (SH-6.1 carries + the derivation), so the window is not exotic -- it needs an unlucky stall, not an unreachable one. + Two producers is the relevant figure at two minutes, since the hazard needs a second producer to + advance the counter while the first is held; the 37-second single-producer number is the ceiling on + how fast this counter can be driven at all, not a rate at which the bug can fire. + +- [x] **SH-14.2** -- **`slotwise_mpsc` had the same hole on a 32-bit target.** Its positions were + `AtomicUsize`, which is 32 bits there. A producer that has observed `sequence == position` -- the + slot is free -- and then stalls across a full lap resumes to find the same `tail` bits, succeeds + at `compare_exchange_weak(position, position + 1)`, and writes a slot that may now hold a live + item from the previous lap of the ring. The sequence observation that made the write safe is never + re-checked, and the exchange covers only `tail`. + This is the finding that also falsified SH-6.1's claim that this shape "cannot be driven there at + all" -- corrected in place. + **Fixed by widening the counter rather than by narrowing the platform.** Positions and slot + sequences are now a named `Position = u64` on every target, so the lap needs 2^64 claims and cannot + be reached. On 64-bit this is exactly what `usize` already was; on 32-bit the exchange becomes a + 64-bit one. Verified on a real 32-bit target rather than argued: the suite passes under + `i686-pc-windows-msvc` (290 tests), and a probe there reports `target_has_atomic = "64"` with + `AtomicU64` **lock-free** -- so the fix costs a `cmpxchg8b`, not a hidden mutex, which is the + outcome that would have made this a bad trade. + `producers` stays `AtomicUsize`: it is a handle refcount, not a position, and nothing compares it + against one. + +- [x] **SH-14.3** -- **SUPERSEDED BY SH-15.6, which is the same decision with better information.** + Checked off as *asked and answered elsewhere*, not as decided: the decision itself is still open, + and it is open in exactly one place now rather than two. + This item enumerated four options and asked for a choice. Keeping it open beside SH-15.6 meant two + live items for one decision, and worse, **this one's option list is now wrong in three ways**: + option 1 said widening is impossible for `reserving_mpsc`, which is true only of its *own* word -- + [D-37](crates/windows-waitable-queues/DESIGN-NOTES.md#d-37) widens a separate shape instead; + option 4's "drop 32-bit" turns out to be entailed by option 1 rather than an alternative to it + ([D-18](crates/windows-waitable-queues/DESIGN-NOTES.md#d-18), amended); and the list has no entry + for the claim protocol that was since built and measured + ([D-35](crates/windows-waitable-queues/DESIGN-NOTES.md#d-35)), which is the current front-runner. + A stale option list competing with a current one is how a decision gets re-litigated from the wrong + premises. The live list is SH-15.6's. + Its one surviving contribution is the observation that no test crossing the wrap can witness the + bug -- a producer must be *held* between its decision and its exchange. That is not lost: it is + SH-15.7, which owns the seam. + +- [x] **SH-14.4** -- **Every statement of the wait protocol was missing its last step.** Five findings, + one cause. `blocking::recv` has always had four steps -- pop, `arm`, **check disconnection and take + one last time**, wait -- but the trait contract, all three shapes' `arm` docs, two worked examples, + and the README each described the three-step form. A caller following any of them waits forever at + the end of the stream: the last producer's drop rings the doorbell **once**, `arm` clears precisely + that ring, and with no producer left nothing rings it again. + The root defect is a contract that overstated itself. `arm` answers exactly one question -- can a + later *push* be missed -- and on a producerless queue the answer is trivially no, so it returns + `true`. Documented flatly as "safe to wait", that is an invitation to hang. It now says what it + measures, and the four-step protocol is stated on `Waitable::arm` with the other statements + pointing at it rather than paraphrasing it again. + Pinned by `arm_reports_safe_to_wait_on_an_empty_disconnected_queue`, which asserts both halves -- + `arm` returns `true`, *and* the doorbell is dark afterwards -- so the exception is bound to + observable behaviour rather than to prose. The final `pop` is likewise not belt-and-braces: a + producer may push *and then* drop between the drain and the check, which is what `Parked::finish` + exists for. + The README is now compiled as a doctest (`cfg(doctest)`, matching three sibling crates). It carries + no code today, so this compiles nothing -- it is there so the first example somebody adds is + compiled rather than trusted, this round being the demonstration that prose nothing executes rots. + +## M15: the claim protocol, prototyped rather than argued (absorbs SH-14.3) + +**This milestone owns SH-14.1's fix, and SH-15.6 is where it is decided.** It absorbed SH-14.3, whose +four options were stale before they were chosen from. The candidates now on the table are three, not +four: the permit claim (built and measured, SH-15.3/SH-15.5), the wide claim word (planned, SH-15.9), +and doing nothing but the disclosure already shipped at SH-15.8. None could be chosen on reasoning +alone, because [D-26](crates/windows-waitable-queues/DESIGN-NOTES.md#d-26) had already measured that +the single shared line is what collapses under contention -- so an "obviously cheaper" claim protocol +that touches two shared lines instead of one might well have been slower. It was not +([D-35](crates/windows-waitable-queues/DESIGN-NOTES.md#d-35)), which is exactly why it was measured. + +**Built as a duplicated path, per the repository's platform-integrity rule.** The prototype does not +modify `reserving_mpsc`: the shipping shape keeps working and keeps its tests green while the +speculative one is proven or discarded, and merge-or-delete is SH-15.6 rather than something that +happens by drift. + +**Numbering note: there is no SH-15.4.** It was the second arm -- the per-cell cycle claim -- and it +moved to `SH-inf.1` when in-order delivery, inline storage and non-blocking progress turned out to be +over-constrained together. The number is left vacant rather than reused, so a reference to SH-15.4 in +an older commit still resolves to something. + +**The principle the prototype is an instance of**, stated once so it is not re-derived: *the atomic +operation that authorizes the write must cover everything the decision depended on.* Today's protocol +decides "there is room" from a separately-read `head` and then compare-exchanges only the claim word, +so a full recurrence of the 32-bit position field revalidates nothing. Every fix below closes that +gap; the options in SH-14.3 instead make the recurrence harder to reach. + +- [x] **SH-15.1** -- **Record the prior-art research as a design note before it is lost.** The survey + is the reason this milestone exists and none of it is currently written down. It must capture: that + `crossbeam-queue::ArrayQueue`, `concurrent-queue` and `thingbuf` all use our protocol shape and none + re-validates after its compare-exchange; that all three are saved only by putting the whole counter + in one `usize`, so all three carry the identical exposure on a 32-bit target; that Nikolaev's SCQ + (DISC 2019, open access, DOI `10.4230/LIPIcs.DISC.2019.28`, section 3 "ABA safety") states the width + assumption the field relies on and states it for **CPU-word width**, which a 32-bit subfield does not + satisfy; that DPDK's `rte_ring` is the closest published twin of our exact protocol and its published + justification (Programmer's Guide 6.5.4) covers modular *arithmetic* only, not lap recurrence; and + that SCQ and CRQ both fix it structurally by making the counter an unconditional fetch-and-add and + moving the authorizing compare-exchange onto the cell. + Also correct the exposure figure, which is currently wrong in both SH-6.1 and SH-14.1: at the crate's + own measured 8.6 ns/push the wrap is **37 seconds**, not "about two minutes". Two minutes is the + two-producer figure and roughly four the 32-producer one; since the hazard needs at least two + producers the headline is defensible, but the range and its basis belong in the text. + +- [x] **SH-15.2** -- **Amend [D-18](crates/windows-waitable-queues/DESIGN-NOTES.md#d-18), whose stated + rationale no longer holds.** It refuses a 128-bit compare-and-swap because it "would lift the 2^31 cap + and nothing else", which was written before SH-14.1 was known -- a 64-bit position would also collapse + the recurrence, so the decision denies the existence of its main benefit. + **Checked against the pinned toolchain rather than against documentation**, because two of its three + supporting costs turned out to disagree with it. `rustc 1.98.0 --print cfg` reports, per target: + `x86_64-pc-windows-msvc` emits `target_feature="cmpxchg16b"` **and** `target_has_atomic="128"`; + `aarch64-pc-windows-msvc` emits `target_has_atomic="128"` with no target feature required; + `i686-pc-windows-msvc` emits `target_has_atomic="64"` and **no** `"128"`. + So: the claim "`x86_64-pc-windows-msvc` does not enable the target feature by default" is **false** + on 1.98 -- there is no floor to raise and no runtime detection to pay. The claim "there is no usable + `AtomicU128`" is **true** and verified (still unstable, rust-lang/rust#99069), so the dependency cost + stands. And the decisive new fact D-18 never had: **a 128-bit claim word cannot work on `i686` at + all**, so that option is not "widen the word", it is "widen the word *and* drop 32-bit support" -- + which collapses SH-14.3 option 1 into option 4 and makes it the engineer's call under the + platform-integrity rule. Amend rather than reverse: the refusal may well stand, but every reason + currently given for it is either wrong or incomplete. + +- [x] **SH-15.3** -- **Arm A: the central-permit claim, as a duplicated shape.** Admission becomes a + single atomic on one `permits` counter initialised to the capacity, and the position degrades to a + pure ticket (`fetch_add`, which has no predicate and therefore cannot be revalidated wrongly). A + producer holding a permit and taking ticket `p` has `p - head <= capacity - 1` by counting, so its + slot is provably free and the position may wrap freely. This satisfies `reserving_mpsc`'s own stated + requirement -- "two independent claimants on one resource must synchronise on one location" -- with + the permit counter as that location, so it strengthens the existing argument rather than contradicting + it. Reservations map directly: a reservation is a permit held across time, still taking no position, + so an outstanding one reduces capacity without head-of-line blocking the stream. + **Not claimed to be non-blocking.** A preempted ticket-holder still stalls the consumer at its + position; this arm fixes the ABA hole and nothing about the progress condition. + +- [x] **SH-15.5** -- **Measure arm A against the shipping shape in `probe-queue-contention`.** The + probe deliberately measures the real shapes rather than stand-ins ("a stand-in would only measure + itself"), so the arm must be a real module in the queue crate for this to mean anything. Report both + regimes: isolated for the claim cost alone, drained for what the shared line costs when a consumer is + writing it. The question this answers is narrow -- does removing the room-decision race cost + throughput, given that arm A touches two shared lines on the push path where today's shape touches + one plus a read. + +- [ ] **SH-15.5.1** -- **Settle why the two shapes' refusal counts differ by orders of magnitude, + because SH-15.6 cannot be decided without it.** + **Spawned by SH-15.5, not a leftover of it.** SH-15.5 is checked because its own action -- taking + the measurement -- is finished; this is new work that the measurement revealed, which is why the + pairing of a checked parent and an open child is correct rather than a contradiction. In the drained regime `permit_mpsc` recorded + roughly 460,000 refusals at eight producers where `reserving_mpsc` recorded 0, and the counts are + unstable across runs (`reserving_mpsc` itself recorded 0 and then 2,363 for the same + configuration). Two candidate explanations, which the current harness cannot separate: the permit + shape is genuinely faster, so it attempts more pushes against a full queue and is refused more + often as a consequence; **or** its optimistic overdraw refuses near-full more readily than the + shipping shape's re-read of the claim does, in which case adopting it would change how eagerly a + caller sees backpressure. + The distinction matters and is not cosmetic. `reserving_mpsc` re-reads the claim and retries before + reporting `Full`, so it refuses only when the queue was genuinely full at an instant it observed. + If the permit shape refuses more eagerly, that is a **behavioural change to a public contract**, + and per D-34's own criterion it must be stated rather than discovered by a caller. + Measure refusals per *attempt* rather than per run, at a fixed attempt count with the consumer's + drain rate pinned, so throughput and refusal rate are separated. A test that admits exactly + `capacity` items from N concurrent producers into an initially empty queue would also settle the + narrow question of whether an overdraw can refuse while a slot is provably free. + +- [ ] **SH-15.6** -- **Decide: merge, delete, or ship as a third peer.** **RE-PLANNED: this item was + written as a binary and the binary was wrong.** "Merge or delete" presumes one protocol dominates, + and [D-35](crates/windows-waitable-queues/DESIGN-NOTES.md#d-35) measured that none does -- the + permit claim wins from four producers upward and loses at one, with no configuration-free winner. + [D-29](crates/windows-waitable-queues/DESIGN-NOTES.md#d-29) already settled how this crate answers + that question for the two existing shapes: **both ship, the crate publishes what it measured, and + the caller decides on its own hardware.** Deleting a shape because no visible consumer wants it is + what the platform-integrity rule forbids; adopting one because it won most rows would be the same + error facing the other way. + So the live outcomes are three, and the third is now the most likely: adopt arm A into + `reserving_mpsc`; delete it and take one of SH-14.3's original options; or promote it to a named + peer alongside the other two, with the measurement published so a caller can choose. The + duplicated path still may not become permanent *by inattention* -- that is what this item guards -- + but becoming permanent *by decision* is a legitimate outcome rather than a failure of the + duplication rule. + Whichever way it goes, SH-14.1's hazard must be either fixed or documented as an accepted + limitation with its exposure stated -- it may not simply stay open. **That disclosure is no longer + gated on this item**: see SH-15.8, which must land before 0.1.0 publishes regardless of what is + decided here. + **One comparison to make explicitly rather than leave implied, now that SH-15.9 adds a third + answer to SH-14.1.** On the evidence so far the permit claim dominates the wide claim on every + axis except maturity: it fixes the hazard on *all* targets where the wide claim covers only 64-bit + ones, it is 2.7x faster at 16-32 producers where the wide claim keeps the retry loop that costs, + it needs no dependency, and **it now reaches the same 2^62 ceiling on every target**: its ticket + was widened to `u64` (as `slotwise_mpsc`'s was in SH-14.2), which was the wide claim's last + remaining unique advantage. + What the wide claim has instead is **risk**: it is a width change to a shape that has been through + nine review rounds and whose behaviour is unchanged, where the permit claim is a new protocol with + 23 tests, no trait impls, no verification, and an open question about whether it reports + backpressure more eagerly (SH-15.5.1). Those are genuinely different products -- conservative fix + versus better fix -- which is the argument for shipping both rather than the argument for choosing. + Do not let this comparison be settled by whichever is finished first. + **Gated on SH-15.5.1**, not on SH-15.5: the throughput question is answered + ([D-35](crates/windows-waitable-queues/DESIGN-NOTES.md#d-35) -- 2.7x faster at 16-32 producers, + 1.45x slower at one), but adopting a claim that reports backpressure more eagerly would be a + behavioural change to a public contract, and that is not yet known either way. + Note also what the measurement did **not** cover, so adoption does not quietly assume it: the + permit shape has no `Waitable`/`Observable`/`Reserving` trait impls, no `Options`/disposal + integration, no high-water tracking, no race hooks, and no 32-bit run. Merging means writing all of + those, so the merge is a milestone rather than a rename. + +- [ ] **SH-15.7** -- **Build the stall seam that can actually witness the bug.** SH-14.3 already notes + the property is invisible to a test that merely crosses the wrap: it needs a producer *held* between + its room decision and its exchange. The crate's existing race hooks (`ARM`, `CLEAR`, `CLAIM`) are the + right shape. Without this, the prototype above is argued rather than demonstrated, and the fix that + is adopted has no regression test that would go red if it were reverted. + **COMPLEMENTS SH-6.1, and neither substitutes for the other.** That item drives the queue across + 2^32 and so exercises the arithmetic; this one supplies the stall that turns a wrap into the actual + defect. A reader who does only SH-6.1 will get a green run and conclude wrongly. + **This is the one M15 item that is worth doing whatever SH-15.6 decides** -- a fix with no test + that fails without it is a fix nobody can safely revisit. + +- [x] **SH-15.8** -- **Disclose SH-14.1 publicly, and gate 0.1.0 on the disclosure rather than on the + fix.** **RELEASE BLOCKER.** The crate is days from its first publish with a known path to *silent + data loss* -- a producer overwriting a live, unconsumed item -- documented nowhere a caller would + see. That is not acceptable to ship in silence, and it is separable from deciding the fix: the + limitation exists now, whatever SH-15.6 later concludes. + **Precedent, and the reason this is a legitimate outcome rather than a dodge.** + [D-31](crates/windows-waitable-queues/DESIGN-NOTES.md#d-31) already ships one known gap this way -- + "the disclosure, not the deferral, is the decision" -- with its own README section and crate-doc + section stating plainly what is verified and what is not. This follows that shape exactly and sits + beside it. + **But the two are not equally forgiving, and the disclosure must say so.** An unverified memory + ordering is a risk of a bug; this is a *known* bug with a computed exposure. Its failure mode is + silent: no error, no panic, no counter -- an item is overwritten and the consumer receives the + wrong one, so **a caller cannot detect it and therefore cannot mitigate it after the fact.** A + disclosure that only a careful reader finds is not a disclosure for a fault of that shape. + What it must state, in the crate docs, the README, and `reserving_mpsc`'s own module docs: + 1. **It is not a 32-bit-only concern.** `POSITION_BITS` is 32 by construction on every target, so + this reaches x86-64 and ARM64 exactly as it reaches i686. The sibling spelling "32-bit + position" invites precisely the misreading that SH-6.1 already had to be corrected for once, so + the words "on every target" belong in the first sentence. + 2. **The exposure, quantified**: 2^32 pushes, which is 37 s to about 4 minutes of sustained pushing + at this crate's own measured rates -- roughly two minutes at two producers, the smallest count + that can trigger it. Sustained, not cumulative-over-uptime. + 3. **What is required to trigger it**: the wrap *plus* a producer stalled between its room check + and its claim, a window a few instructions wide. Rare, not unreachable, and a preemption is + enough. + 4. **The alternatives a caller has**, which is what makes this a decision they can actually take: + `slotwise_mpsc` does not have this hazard (SH-14.2 widened its positions to 64 bits on every + target); `spsc` never had it; and the queue is safe at any push volume below the wrap. + Sweep for consistency when writing it, per the contract-integrity rule: this fact will end up + stated in at least four places and they must not drift. + +- [ ] **SH-15.9** -- **Ship the claim word in two widths: the narrow one everywhere, the wide one where + the hardware allows.** The engineer's decision, and it follows + [D-29](crates/windows-waitable-queues/DESIGN-NOTES.md#d-29): publish what we measured and let the + caller choose, rather than picking one tradeoff for everyone. + - **`reserving_mpsc` is unchanged and always ships**, on every target, with SH-15.8's warnings. It + is never silently swapped for the wide one on targets that could support it: a shape whose + contract changes with the target is exactly what + [PLATFORM INTEGRITY](../.github/copilot-instructions.md) rule 2 forbids, and a caller reading + "2^32" in the docs must get 2^32. + - **`reserving_mpsc_wide` is new**: the same claim protocol with a `u128` word split 64/64. The + position then needs 2^64 pushes to recur -- about 16,000 years at this crate's measured rates -- + so SH-14.1 is unreachable rather than merely unlikely. The capacity ceiling rises from 2^31 to + 2^62, which was [D-18](crates/windows-waitable-queues/DESIGN-NOTES.md#d-18)'s original point. + + **The gate is one line of `Cargo.toml`, and this was measured rather than designed.** An earlier + version of this item specified `#[cfg(target_has_atomic = "128")]` plus a `const` assertion on + `is_always_lock_free()`. Both are unnecessary. Probing `portable-atomic` 1.15 on the pinned + toolchain established: + - With **`default-features = false`**, `portable_atomic::AtomicU128` **does not exist** on + `i686-pc-windows-msvc` (`error[E0432]: unresolved import ... no AtomicU128 in the root`), nor on + `x86_64` built with `-C target-feature=-cmpxchg16b`. It exists exactly where the target has a + compile-time-guaranteed native lock-free 128-bit exchange. **The `use` statement is the gate**, + and it fails loudly, naming the missing type. + - With **default features**, the `fallback` feature compiles on i686 and silently substitutes a + **global lock**. That -- not anything intrinsic to a 128-bit exchange -- is the whole source of + the silent-degradation hazard, and it is opted out of rather than guarded against. + - `#[cfg(target_has_atomic = "128")]` is the **wrong** gate regardless: `rustc 1.98.0 --print cfg` + still emits it under `-C target-feature=-cmpxchg16b`, because it tracks the target's maximum + atomic width and not instruction availability. + - `is_always_lock_free()` **is** const-evaluable (confirmed: `const X: bool = + AtomicU128::is_always_lock_free();` compiles, yielding `true` on x86_64). A const assertion on it + is nonetheless **worse than useless** here -- redundant where the type exists, and unreachable + where it does not, because there is nothing to compile. + So: depend on `portable-atomic` with `default-features = false`, and add no cfg and no assertion. + Note the consequence plainly in the shape's docs: **`reserving_mpsc_wide` cannot be built for + i686 at all**, so a caller targeting 32-bit uses `reserving_mpsc` (with SH-15.8's warnings) or the + permit claim. A portability cliff was chosen over a performance cliff because a compile error + naming `AtomicU128` is more informative than a queue that silently stops being lock-free. + + **[D-7](crates/windows-waitable-queues/DESIGN-NOTES.md#d-7) puts the burden of proof on adding a + feature, and it is met here rather than waived.** D-7 rejected feature-gating shapes because the + only benefit was compile time, which dead-code elimination already provides. That reasoning does + not reach this case: the wide shape's cost is a **new third-party dependency** on a crate whose + only current one is `windows-sys`, and dead-code elimination removes nothing from `Cargo.lock`, + from a downstream auditor's review, or from a `cargo vet` run. A caller who does not want the + dependency must be able to not have it. Record the discharge as a decision rather than leaving it + to look like D-7 was ignored. + +- [ ] **SH-15.10** -- **Measure the wide claim beside the narrow one, and publish the difference.** + Same harness, same host, same run as SH-15.5. The question is narrow and worth an answer either + way: does a 128-bit exchange cost anything measurable against a 64-bit one on this hardware? If it + does not, the wide shape is strictly better wherever it builds, and the guidance should say so. If + it does, that number is what a caller needs to choose between reach and speed. + Note the expected shape of the result, so a surprise is recognisable: the wide claim keeps the + compare-exchange **retry loop**, which [D-35](crates/windows-waitable-queues/DESIGN-NOTES.md#d-35) + identified as what actually costs at high producer counts -- so it should track `reserving_mpsc` + closely and should **not** approach the permit claim's numbers. A wide claim that measured as fast + as the permit claim would mean D-35's explanation is wrong. + +## M16: PR #56 tenth review round -- the SH-3.1.1 diff review + +> **Six of these items are superseded.** SH-16.5, SH-16.8, SH-16.9, SH-16.11, SH-16.12 and SH-16.13 +> are all the same piece of work seen from different angles -- reshaping the machine memory topology +> -- and they now live in +> [crates/windows-topology-sys/COMPLETED-CHECKLIST.md](crates/windows-topology-sys/COMPLETED-CHECKLIST.md) as a plan of +> their own, numbered `MMT-*`. They are left here, unchecked and marked, rather than deleted: each +> records how the defect was *found*, which the new plan does not repeat. +> +> **Read the new plan for what to do; read these for why.** The six that remain live here are the +> review round's own findings, already fixed. + +**This round is the one [SH-3.1.1](#m3-land-the-branch) asked for**, and it is the first that read the +branch as a *diff* rather than reacting to a reviewer's comment. Five reviewers took non-overlapping +crate scopes across all 200 changed files; seven findings came back, listed here worst-first rather +than by crate. + +**Two of them are the reason the round was worth running.** SH-16.1 is a regression this branch +introduced *two commits ago* -- it would have failed the next `windows-ioring-sys` publish, twenty +minutes in, with an error naming the wrong cause. SH-16.2 is a soundness hole in the crate that is +about to freeze its API, in a shape whose selling point is that its ordering arguments are written +down and checked. + +**Neither was reachable from a memory of having written the code**, which is exactly what SH-3.1.1 +predicted about a 222-commit branch. + +- [x] **SH-16.1** -- **`publish-crate.yml`'s sibling-dependency wait cannot handle a `*` requirement, + so `windows-ioring-sys` can no longer publish.** The wait step derives a concrete version with + `sed -E 's/^\^//'`, which handles only a caret. Commit `f1fc4eb` on this branch made ioring's + topology dev-dependency path-only, so `cargo metadata` now reports `req=*` -- **verified, not + assumed**. `windows-topology-sys` is in `workspace_crates`, so the loop is entered, `dep_version` + becomes the literal `*`, and `select(.vers == "*")` can never match: 60 attempts x 20 s, then an + error telling the operator to re-run once the dependency is available, which will never help. + A versionless path dev-dependency is **stripped from the published manifest entirely**, so there is + nothing to wait for and the right answer is to skip it. + Note that `tools/check-publishable.ps1`, added in this same branch to catch "release-managed but + unpublishable", does **not** catch this -- ioring passes all three of its checks. + **Done:** `*` is skipped with the reason stated, and any requirement that does not reduce to a + comparable version (`~1.2`, `>=1, <2`, `=1.2.3`) now fails **immediately** naming the requirement, + rather than reaching the same twenty-minute timeout by a different route. Verified by extracting the + `run:` block and exercising it under `bash` against real `cargo metadata` output: ioring's seven + dependencies now resolve to two waits, four skips and one versionless skip, and the step exits 0. + A side benefit worth recording, found by getting the harness wrong first: the new check also + catches a returning CR corruption -- the failure the `tr -d '\r'` above was added for -- because + `0.1.3\r` is no longer a comparable version. That failure used to be a silent timeout too. + +- [x] **SH-16.2** -- **`reserving_mpsc::Reservation::send` wrote a slot with no happens-before edge to + the consumer's read of the previous occupant.** A slot is freed only by `Consumer::pop`'s + `head.store(Release)`, and the matching acquire lives in `has_room_beyond_reservations`. + `Producer::push` gets its edge from that room check; `send` deliberately has none -- the code says + so -- and its claim CAS is `Relaxed` on every path, so there was no release sequence to inherit + either. The claim proves the slot is *logically* free, which is not the same as a synchronization + edge, and the `SAFETY` comment cited "the room check that permitted the claim" on the one path + where no room check exists. + **The default configuration was the unsound one**: `Options::tracking_high_water()` accidentally + repaired it, because the metric's `head.load(Acquire)` sat just before the write. Fixed by making + that load unconditional, which is where it belonged. + +- [x] **SH-16.3** -- **`CancelIo` does not wait, so a test frees an `OVERLAPPED` and an I/O buffer the + kernel may still write to.** In + [reopen_by_id_cannot_be_watched.rs](crates/windows-file-watcher/tests/reopen_by_id_cannot_be_watched.rs), + an overlapped `ReadDirectoryChangesW` is issued into a **stack-local** `OVERLAPPED` and a heap + buffer, then `CancelIo` is called and both are dropped immediately. `CancelIo` only *requests* + cancellation; the IRP still completes asynchronously and writes `Internal`/`InternalHigh` into a + frame that has been reclaimed. Two safety comments assert the opposite of what the code guarantees. + Aggravated by the helper being called twice back-to-back, so the second call's `overlapped` likely + lands on the same stack address the first IRP will write into. + **This crate has already been bitten by this exact class of corruption** -- the + `STATUS_STACK_BUFFER_OVERRUN` history recorded on the now-removed `reopen_via_existing_handle`. + Fix by calling `GetOverlappedResult(..., bWait = TRUE)` and accepting `ERROR_OPERATION_ABORTED` + before either buffer leaves scope. + **Done, and the wait is measured to be load-bearing rather than assumed.** A probe on the control + path returned `completed=0, err=995` -- `ERROR_OPERATION_ABORTED` -- proving an IRP really was + outstanding at the moment `CancelIo` returned and completed only during the wait. Without it, that + completion landed on a reclaimed frame. The same wait is what makes `Owned`'s later `CloseHandle` + safe, since closing a handle with I/O outstanding is another cancellation request and not a wait. + +- [x] **SH-16.4** -- **`cache_partitions_at_level` counted a domain covering no processors as a + partition.** An empty `ProcessorSet` is not *equal* to any non-empty one, so deduplication kept it, + and `is_disjoint` is vacuously true on it, so the pairwise check passed it. A level with one real + cache plus one empty domain therefore reported two partitions and was treated as dividing a machine + it does not divide. `Domain` is publicly constructible and `ProcessorSet` has `empty()`, so this is + reachable by hand and by deserialization -- precisely the input the method promises not to trust. + Fixed by dropping empty domains, with the contrast against `memory_domains` (which deliberately + keeps a processor-less domain, D-5) recorded at the filter. + +- [x] **SH-16.5** -- **DISCHARGED 2026-09-03 by M5+.4 -- `cache_domain` is `Observed`, the refusal is gone, and `Slice::same_cache_domain` answers `None` rather than `same` for an unobserved participant.** SUPERSEDED by [crates/windows-topology-sys/COMPLETED-CHECKLIST.md](crates/windows-topology-sys/COMPLETED-CHECKLIST.md) (MMT-*); kept for how it was found.** **`windows-placement-probe` refuses a partially-covering cache level that + `windows-topology-sys` deliberately hands back.** `outermost_partitioning_cache` documents that + "full coverage of the online processors is deliberately *not* required"; `places_from_topology` + treats any online processor the chosen level does not name as `MissingPlacement::CacheDomain` and + fails the **entire run** with `InvalidData`. Two crates state opposite rules about the same return + value -- a [CONTRACT INTEGRITY](.github/copilot-instructions.md) defect, not merely a bug. + Decide the rule **once**, in the crate that owns the topology, and have the consumer ask rather than + restate. Note the asymmetry that makes the NUMA arm different and correct: for NUMA, `None` has no + honest value, whereas `cache_domain` is already `Option`. + **BLOCKED on + [DESIGN-SESSION-2026-09-02-cache-locality-model.md](design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md) + -- and *not* for want of a consumer.** The fix was implemented; implementing it surfaced a design + question the fix would have silently answered. The primitive it adds is a single "which cache domain + is this processor in", which **is** the single-boundary collapse that session is about, so landing it + would prejudge the outcome. The prototype compiled, and its topology-side tests passed and were + sabotage-verified; it was reverted deliberately and preserved outside the repository as + `sh-16.5-prototype.patch`. + **Unblocked 2026-09-03, and superseded rather than resumed.** The session's questions were answered + as `D-13` through `D-21`, and the answer is *not* the primitive this item prototyped: under + [D-19](crates/windows-topology-sys/DESIGN-NOTES.md#d-19) the unified relation set with its + inclusion order replaces a single per-processor cache-domain lookup, so the prototype would have + landed the collapse the session existed to remove. The contradiction is fixed by `MMT` **M2+.5** and + **M5+.4** instead. The patch is kept as the record of what was tried and why it was not taken. + +- [x] **SH-16.8** -- **DISCHARGED 2026-09-03 by M2 -- the granularity order carries all seven kinds and any depth, and `minimal_shared` is the meet rather than a single cache level.** SUPERSEDED by [crates/windows-topology-sys/COMPLETED-CHECKLIST.md](crates/windows-topology-sys/COMPLETED-CHECKLIST.md) (MMT-*); kept for how it was found.** **The locality model collapses a seven-kind, any-depth topology onto one cache + boundary, and nothing records that as a choice.** Raised by the engineer during the SH-16.5 fix, and + confirmed: `windows-topology-sys` hardcodes no level count (`level` is a `u8`, and a regression test + already guards against a consumer sweeping `1..=4`) and models `Group`, `Package`, `Die`, `Module`, + `Core`, `Cache` and `Memory` -- but `outermost_partitioning_cache` selects one level and discards the + rest, `ProcessorPlace::cache_domain` is one scalar, and `Placement` carries three tiers. + Three consequences, all verified: "same cache" denotes **a different boundary on different machines**, + so a label is not portable across records; `CrossCache` conflates "different L2, same L3" with + "different L3" on any machine with two live boundaries; and it has already cost a row in this + project's own matrix -- the x64 host's "cannot express `same cache, same class`" note in + [DESIGN-NOTES.md](crates/windows-waitable-queues/DESIGN-NOTES.md) is attributed to hardware, but + those sixteen processors do share one L3, so a per-level model would express it. + Gated on the session above, which carries the design space and the open questions. + **Scope addition from [D-13](crates/windows-topology-sys/DESIGN-NOTES.md):** the audit that decision + performed over every `Option` in the crate found exactly one site that documentation cannot fix. + `DomainKind::Memory::memory_bytes` is unambiguous from `discover`, which always sets `None`, but a + **description's** `None` conflates "the description omitted the field" with "this node's capacity is + genuinely unknown" -- the two are the same value today. Whatever representation this item lands must + cover it, since absence becoming first-class is precisely the fix. + **Direction now settled** by the engineer: presence and observation must be modeled, not + collapsed into an `Option`. "Win32 did not report it" and "it was found not to be present" are + different facts, and the representation must be built for **observed connectivity** rather than + for a ladder of levels with optional rungs. That rules out the SH-16.5 prototype's `Unknown` arm, + which merges both. Shape still open. + +- [x] **SH-16.9** -- **DISCHARGED 2026-09-03 by M5+.3 -- the rule has one implementation, which `windows-platform-probes` now asks rather than restates.** SUPERSEDED by [crates/windows-topology-sys/COMPLETED-CHECKLIST.md](crates/windows-topology-sys/COMPLETED-CHECKLIST.md) (MMT-*); kept for how it was found.** **The "outermost partitioning cache" rule is stated three times, and two of the + three disagree.** `MachineMemoryTopology::outermost_partitioning_cache` requires more than one partition **and** + pairwise disjointness. `Observation::outermost_partitioning_cache` in `windows-platform-probes` is + `caches.iter().filter(|c| c.domains > 1).max_by_key(|c| c.level)` -- **no disjointness check** -- + computed over a `CacheLevel` summary that crate builds itself, even though it already depends on + `windows-topology-sys`. On a hand-built or deserialized topology with overlapping domains the two + crates give different answers to the same question. `windows-placement-probe` restates it a third + time by rebuilding the map from the partition list, which is SH-16.5. + A [CONTRACT INTEGRITY](.github/copilot-instructions.md) defect of the exact shape the rules name: + a rule re-encoded by a consumer rather than derived from the owner. Note the ordering -- fixing + this by pointing both consumers at today's method would have to be redone once SH-16.8 lands, so + either fix it now and accept the rework, or sequence it after the design session. + +- [x] **SH-16.10** -- **`GetSystemCpuSetInformation` is not consumed anywhere, so a whole Win32 + topology model is unexposed.** The crate consumes all seven `GetLogicalProcessorInformationEx` + relations, but `SYSTEM_CPU_SET_INFORMATION` is a *second, parallel* model carrying at least + `LastLevelCacheIndex` -- Windows's own LLC grouping, which is a **different answer** from + "outermost partitioning cache" and would be directly comparable against it -- plus + `SchedulingClass`, `AllocationTag`, `EfficiencyClass`, and per-processor `Parked` / `Allocated` / + `RealTime` state. + Raised by the engineer's question of whether we expose everything a real system would reveal + through the Win32 API set. Today the answer is **no**. + Note `Parked` and `Allocated` bear directly on **thread counts and assignments**, one of the three + decisions the model exists to serve, so this is a gap already costing a named use rather than + speculative completeness. + **Ungated, and split, because the gating premise was wrong.** This said "gated on SH-16.8, since + what shape it lands in depends on the model". That conflated two things: *acquiring* the data and + *reconciling* it with what `GetLogicalProcessorInformationEx` already reports. Acquisition does + not depend on the model at all -- CPU Sets is a **cheap OS read**, in the same class as the walk + this crate already does, and nothing about reading it presumes a granularity representation. Only + reconciliation depends on the model, and that is now SH-16.13. + Field list **verified against `windows-sys 0.61.2`** rather than recalled: `Id`, `Group`, + `LogicalProcessorIndex`, `CoreIndex`, `LastLevelCacheIndex`, `NumaNodeIndex`, `EfficiencyClass`, + a union carrying `AllFlags` (`Parked` / `Allocated` / `AllocatedToTargetProcess` / `RealTime`), a + union carrying `SchedulingClass`, and `AllocationTag`. All five APIs are present + (`GetSystemCpuSetInformation`, `GetThreadSelectedCpuSets`, `SetThreadSelectedCpuSets`, + `SetThreadSelectedCpuSetMasks`, `SetProcessDefaultCpuSets`) and `Win32_System_SystemInformation` + is already an enabled feature, so there is no manifest change and no blocker. + **Done.** `src/cpu_set.rs` walks the records with the same buffer discipline the relationship walk + uses -- size first, advance by each record's own `Size`, read every field unaligned -- and + `MachineMemoryTopology::discover` now populates `MachineMemoryTopology::cpu_sets`. Carried as + `Option>` where `None` means **not observed**, which a hand-built or deserialized + topology genuinely is; that is the honest use of `Option`, one absence rather than two collapsed + together. `#[serde(default)]` so descriptions written before the field still load. + **Nothing is reconciled**, per duplicate-then-decide. SH-16.13 owns that. + **The live dump justified the caution.** On the x64 host, CPU Sets reports **one** distinct + `LastLevelCacheIndex` across all sixteen processors, while `outermost_partitioning_cache` reports + **eight** partitions at L2. Both are right -- Windows names the *last* level, the derivation names + the outermost level that *divides* -- so a merge treating `LastLevelCacheIndex` as "the cache + domain" would have collapsed eight shard groups into one on this machine. Kept as a test asserting + the *relationship* (Windows's grouping is never finer) rather than the host's numbers. + It also confirms the matrix-hole argument from + [DESIGN-SESSION-2026-09-02-cache-locality-model.md](design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md): + this is the host recorded as unable to express `same cache, same class`, and a second source now + says all sixteen share an LLC, so that row is real rather than inferred. + **One thing is verified only against the SDK's documented bitfield order, not against Windows:** + the four flag bit positions. Every processor on this host reads `parked=false, allocated=false, + allocated_to_target_process=false, real_time=false`, which is consistent with a process that has + requested no CPU-set allocation but confirms no bit position. `each_flag_is_read_from_its_own_bit` + checks the decode is self-consistent, not that it matches the OS. Confirm against a parked + processor or an explicit `SetProcessDefaultCpuSets` before relying on the flags. + +- [x] **SH-16.13** -- **DISCHARGED 2026-09-03 by M3+.1.2 -- CPU Sets are folded into the relation set and carried beside the walk, so both observers are visible per relation.** SUPERSEDED by [crates/windows-topology-sys/COMPLETED-CHECKLIST.md](crates/windows-topology-sys/COMPLETED-CHECKLIST.md) (MMT-*); kept for how it was found.** **Reconcile the CPU-set observation with the relationship walk.** `CoreIndex`, + `NumaNodeIndex` and `EfficiencyClass` **duplicate** facts `GetLogicalProcessorInformationEx` + already reports, from a different kernel path -- so this is not redundancy to remove, it is a + **second independent observer of the same relations**, and the two can disagree under a hypervisor + or where one path is stale. + This is the concrete instance of the design session's "can one relation hold several + observations?" question, which until now rested on the file-handle spike's agree/disagree + reasoning about a different subject. It is no longer speculative: two Win32 sources describe the + same processor's NUMA node and efficiency class today. + Per [PLATFORM INTEGRITY](.github/copilot-instructions.md)'s duplicate-then-decide rule, SH-16.10 + lands the CPU-set data as its **own** observation alongside the existing domains, without merging. + This item is the merge-or-delete decision, made when the model settles rather than pre-empted. + Gated on SH-16.8. + Note it also bears on SH-16.12: CPU Sets carries `EfficiencyClass` as a plain `u8` with **no + sentinel**, so it is a cleaner source for the field whose `capacity` encoding collides with + "unknown". + +- [x] **SH-16.11** -- **DISCHARGED 2026-09-03 by M5+.5 -- `distances` and the `Distances` type are deleted.** SUPERSEDED by [crates/windows-topology-sys/COMPLETED-CHECKLIST.md](crates/windows-topology-sys/COMPLETED-CHECKLIST.md) (MMT-*); kept for how it was found.** + **And now ANSWERED, in the opposite direction to what this item proposed.** + [D-20](crates/windows-topology-sys/DESIGN-NOTES.md#d-20) rules that the crate does not go below the + Win32 topology APIs, so a fact Win32 does not report is not one the crate has: `distances` is + **deleted, not filled**. The removal is `M5+.5` in + [crates/windows-topology-sys/COMPLETED-CHECKLIST.md](crates/windows-topology-sys/COMPLETED-CHECKLIST.md). Everything + below is the reasoning that led there and is kept for that; it no longer describes work. **`MachineMemoryTopology::distances` is a field for a fact Win32 cannot supply, it is never + populated, and the measurement that would fill it already exists elsewhere.** `discover()` + hardcodes `distances: None`, every other construction sets `None`, and no consumer reads the + field. Windows exposes no API for NUMA node distance -- ACPI carries SLIT, Win32 does not surface + it -- so measurement is the only source. `windows-placement-probe` **already measures the + equivalent** through `node_pairs_measured()`, producing per-node-pair handoff cost with ring + placement, and renders it as a table that goes nowhere else. + This is the canonical case for the whole model: under the bar that the model must be usable + **without further measurement**, a consumer shaping memory allocation must today either run the + probe at decision time -- forbidden -- or guess. Gated on SH-16.8, and on the open question of + which component owns the measurement phase. + **Corrected while stating [EP-D-3](crates/topology-planner/DESIGN-NOTES.md#ep-d-3): the + wording above reads as an oversight, and it is not one.** The field is documented as being for a + fed-in description, because Windows exposes no user-mode SLIT reader -- accurate, and deliberate. + Two sharper problems replace the one this item claimed. + **First, `distances` can never carry `Measured` provenance, by construction.** Its only inputs are + hand construction (defaulting to `Synthetic`) and deserialization (capped at `Restored` by + `downgraded_to`), and `discover()` hardcodes `None`. So populating it would not help: a planner on + a real machine still could not obtain trustworthy distance *for that machine*. + **Second, even populated it answers the wrong question.** The matrix is SLIT-shaped -- one + symmetric, workload-independent scalar per pair -- while the residency decision is directional, + since the producer writes and the consumer reads. `D-9` in + [crates/windows-topology-sys/DESIGN-NOTES.md](crates/windows-topology-sys/DESIGN-NOTES.md) already + anticipated exactly this and deferred it, naming an attributed edge list that "would absorb HMAT, + **asymmetry**, and multi-hop CXL fabrics", with the trigger being that "scalar distance + demonstrably mismodels a machine somebody is tuning for". `D-8` keeps the JSON schema outside + semver specifically to make that revision cheap. + **The trigger is approached but not met, and the difference is a measurement nobody here can + take.** The probe treats direction as real -- four numbers per undirected edge, and its code says + "a hop is not symmetric even though the link is" -- but no run has *shown* those numbers differ, + because both development hosts report a single NUMA node and every such run prints "VACUOUS ON + THIS MACHINE". Take that measurement on multi-node hardware before reopening D-9 on asymmetry + grounds, not after. + +- [x] **SH-16.12** -- **DISCHARGED 2026-09-03 by M5+.1, subsumed by M4+.2 -- the shard-set surface has no sentinel, so `0` is never overloaded.** SUPERSEDED by [crates/windows-topology-sys/COMPLETED-CHECKLIST.md](crates/windows-topology-sys/COMPLETED-CHECKLIST.md) (MMT-*); kept for how it was found.** **`Processor::capacity` uses `0` as both a legitimate efficiency class and a + sentinel for "not known", and the two collide on the common case.** It is computed + `online.then(|| find the owning Core domain).flatten().unwrap_or(0)`, so `0` means the processor is + offline, *or* is online but named by no `Core` domain, *or* genuinely has efficiency class zero. + The third is **every processor on every non-hybrid machine**, so the sentinel is not a rare + collision -- it is the usual value. + Found by [crates/topology-planner](crates/topology-planner/DESIGN-NOTES.md#ep-d-1) + EP-1.1 while checking what a shard planner can rely on, and it is worse for that consumer than for + most: Windows orders efficiency class with `0` as **least** performant, so on a hybrid part an + unknown processor is indistinguishable from an efficiency core. A policy excluding efficiency cores + would silently drop a processor that may be a performance core; a policy tiering them would put it + in the wrong tier. Neither shows up in a functional test. + **A third instance of the pattern SH-16.8 exists to fix**, and the one not previously swept -- the + others being `ProcessorPlace::cache_domain`'s `Option` (SH-16.5) and + `MachineDescription::cpu_model`, where the same conflation was noticed and solved with a side + boolean. Note this one is *worse* than an `Option`: a sentinel that collides with a valid value + cannot be distinguished even by a careful caller. Gated on SH-16.8, since the fix is the same + question -- how absence is represented -- and doing it twice would be doing it twice. + Note `DomainKind::Core { efficiency_class }` already carries the value without a sentinel, so the + interim guidance is to read that instead; the defect is that `capacity` exists and looks usable. + +- [x] **SH-16.6** -- **The thread-stack NUMA spike's `deep_probe` measures the shallow end of its own + filler, so the discrimination it exists to make is inert.** The stack grows down, so `filler[0]` is + the deepest address and `filler[last]` sits immediately below the caller's frame -- but the probe + takes `&raw const filler[last]`, landing very likely on the same page as the shallow probe rather + than 64 KiB away. The spike would then report "not first touch" on a machine where placement *is* + by first touch: a confident wrong answer, in a file whose whole point is avoiding those. Both ends + are already touched, so probing `filler[0]` is a one-token change. + **Done, and the defect was worse than reported.** Printing the three addresses on all three spike + threads showed the old probe was not merely *likely* on the shallow probe's page -- it was on the + **same page every time**, 209 bytes away, where the review had estimated "at worst adjacent". So + `shallow.node != deep.node` compared one page against itself and could not fire even in principle. + After the change the two probes are 16 pages apart on every thread. Measured on all three threads + (`0x...dff7c0` vs `0x...dff6ef` -> same page; vs `0x...def6f0` -> 16 pages), then the instrumentation + was removed. + +- [x] **SH-16.7** -- **A `windows-thread-ambient-sys` test claims a restore-failure it never + injects.** `release_reports_a_genuine_restore_failure_and_restores_on_drop_even_without_it` asserts + the *opposite*: it `expect`s the release to succeed and both closing assertions check that restore + worked. Its siblings in `declared/tests.rs` and `error_mode/tests.rs` do force genuine failures; this + one inherited the name without the failure-injection half. + **Done, by renaming -- and the review's stated hazard did not hold.** It reported that + `TransactionGuard::release`'s error path "reads as covered when it is not". Checked rather than + taken: the path **is** covered, by `explicit_release_reports_an_injected_restore_failure` in the + same file, via a `FaultPoint::TransactionSet` injection built for it. Verified by running both. + So the defect was only ever the name. Renamed to + `release_and_drop_each_restore_a_real_entry_transaction`, and the comment now records *why* the + sibling naming does not apply -- a transaction restore either sets a real handle or clears to + "none", and both succeed, so unlike a null WOW64 cookie or `SEM_NOALIGNMENTFAULTEXCEPT` there is no + naturally-rejecting value to provoke. Written down so the missing half is not re-attempted. + +## M-inf: parked, ungated + +- [ ] **SH-inf.1** -- **The per-cell cycle claim (SCQ's shape), which is the non-blocking one.** The + shared counter becomes an unconditional fetch-and-add authorizing nothing, and the reuse decision plus + the write are validated together by a compare-exchange on the slot's own `{cycle, safe}` word -- + removing the shared decision rather than moving it, and making the queue genuinely lock-free rather + than merely ABA-free. + **Parked deliberately, and the reason is a real constraint rather than scheduling.** In-order delivery, + inline item storage, and non-blocking progress are over-constrained together: storing `T` in the ring + forces a producer to claim a slot and then write it, so a preemption between the two necessarily stalls + in-order consumption. SCQ escapes this only because it queues *indices* -- the payload is written + outside the queue protocol -- which is a different data structure from the one this crate offers. So + this is not an improvement to today's shapes but a fourth shape with different semantics, and it is + worth having only alongside a decision that some caller wants non-blocking progress more than it wants + inline storage. + Related: the whole family this crate belongs to is **technically blocking**, which is worth stating + plainly somewhere public. wCQ (Nikolaev and Ravindran, SPAA 2022) says so of exactly this shape -- + queues that "require a thread to reserve a ring buffer slot prior to writing new data ... are + technically blocking since one stalled (e.g., preempted) thread in the middle of an operation can + adversely affect other threads" -- and names DPDK's ring as a case "erroneously dubbed as 'lock-free'". + This crate should not repeat that error by implication; see SH-15.1, which records the survey. diff --git a/CHECKLIST.md b/CHECKLIST.md index da22c021..eb4d2c21 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -106,6 +106,91 @@ be settled rather than discovered later. completion, then submits the query. A compound entry is reserved for a measured performance argument and would be a fusion of these two entries rather than a capability they lack. Depends on M21.3. +## M34 -- Tooling + +Numbered M34 rather than M22 because the three root-level checklists share one milestone space: +[CHECKLIST.md](CHECKLIST.md) holds M19-M21, [CHECKLIST-thread-ambient.md](CHECKLIST-thread-ambient.md) +M22-M29, and [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M30-M33. + +- [x] **M34.1** -- Promote the ad-hoc sabotage harness into a reusable tool. -> [completed 2026-08-31](COMPLETED-CHECKLIST.md#m341) + +- [ ] **M34.3** -- **Archive the completed bodies in + [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md)**, which holds twelve checked items still + carrying their full write-ups. The completed-item rule moves a large one to + [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md) immediately and leaves a one-line anchored stub, so + the active file stays a list of what is *left*. Raised in review 5072735803 on pull request #56, + where it was noted that the problem recurs throughout that file rather than at the one line cited. + [M34.1](COMPLETED-CHECKLIST.md#m341) is the worked example of the shape: `### ` in + the archive under a dated group, a stub with a completion link in its place. + Bookkeeping with no bearing on correctness, which is why it is queued rather than folded into a + branch already under review -- but it is 757 lines of checklist that a reader currently has to scan + past to find the open work, so it is not cosmetic either. + +- [ ] **M34.2** -- **Route every tool's output through one sink, per the repository's own rule**: never + call `println!`/`eprintln!` from more than one site in a tool; introduce a writer trait, sink or + formatter at the first occurrence and route everything through it. + **Updated 2026-09-04 (second pass): every probe now leads its report with the host line.** The + banner had reached only the seven binaries this item named plus the shared long-path renderer, + which left **eight** probes -- `cancel_io`, `completion_port`, `device_map`, `error_mode`, + `handle_state`, `ioring`, `pool_growth`, `worker_context` -- composing a report that named no + machine. `pool_growth` was the sharpest case: it printed "every number here is from this host and + this Windows build" while giving a reader no way to tell which host that was. The rest are + behavioural findings about what *this* Windows does, which is equally uninterpretable unattributed. + All eight now emit `banner_line()` as the first line of the returned text, verified by running each + binary rather than by reading the diff. Raised by Copilot at review `5118237348`. + **Updated 2026-09-04: the conversion is done; what remains is the capture test.** The item said + "seven binaries violate this today" and named them, from review 5072622803 on pull request #56. + Five had already been converted when a later review round re-checked, and the last two -- + [queue_contention.rs](crates/windows-platform-probes/src/bin/queue_contention.rs) and + [peer_index_cache.rs](crates/windows-platform-probes/src/bin/peer_index_cache.rs) -- were fixed in + that pull request, so all seven now compose their whole report as text and hand it to a sink at one + place. Verified by counting, not by reading: no probe binary contains a direct `println!`/ + `eprintln!` at all. The two survivors were the *banner*, which those two rendered by calling a + helper that wrote to stdout itself -- so a captured report was missing the one line naming the + machine that produced it, and the banner also emitted mid-`render`, ahead of the body, making the + order on a terminal luck rather than construction. + **The stdout-writing banner helpers are gone rather than documented against.** `print_banner` and + `print_banner_with` were removed and `banner_lines_with` returns the string instead, because three + call sites had each grown a comment warning about them -- a rule restated three times instead of a + hazard removed once. The defect class is now unreachable by construction: there is no + banner helper that writes to a stream. + **The PowerShell tools are NOT part of this item, because they are already done.** A later review + round on the same pull request observed that the inventory above named only Rust binaries while five + scripts emitted from many sites, so those were converted in that pull request rather than queued + here: [inject-mutant.ps1](tools/inject-mutant.ps1), + [check-publishable.ps1](tools/check-publishable.ps1), + [run-numa-spikes.ps1](tools/run-numa-spikes.ps1), [run-mutants.ps1](tools/run-mutants.ps1) and + [run-sabotage.ps1](tools/run-sabotage.ps1) each now route everything through one `Write-Report` + sink. They were small enough to convert in place, which is exactly why they did not need deferring. + (`run-sabotage.ps1`'s `Exit-WithMessage` is deliberately outside its sink: that path writes to + stderr and exits, and there the destination is part of the meaning.) + **What remains is the capture test, and it has a structural obstacle worth naming.** + The point of the rule is that output becomes testable, so this item is not checked off on the + refactor alone -- an abstraction introduced without a capture-based test spends the cost and skips + the benefit. `Captured` exists in [report.rs](crates/windows-platform-probes/src/report.rs) for + exactly that purpose, and `banner_line` is already asserted directly. + **The obstacle: each probe's `render()` lives in its own `bin` target, which nothing can import.** + That is precisely why the two banner defects survived every test -- there was no reachable seam to + assert against. Closing it means moving each `render()` into the crate's library and leaving `main` + as the one place that names the stream, which is a real refactor rather than a test to write. + Decide the seam once and apply it uniformly. + A PowerShell sink is a function whose destination can be swapped, but this workspace runs no + PowerShell test harness in which to assert against it, and inventing one to cover five diagnostic + scripts is not a cost this item is willing to spend without deciding to adopt such a harness first. + Start with `placement_probe`: its output is a published artifact that strangers paste into a + discussion thread, so "can this be captured and asserted end to end?" has real value there rather + than being architectural tidiness. + +- [x] **M34.4** -- Share the native-command guard through a dot-sourced `tools/common.ps1`, route + every capture site through it, and prove it on both PowerShell hosts. + -> [completed 2026-09-07](COMPLETED-CHECKLIST.md#m344) + +## M35 -- Measure what the long-path opt-in actually does + +- [x] **M35.1** -- Measure whether the long-path opt-in lifts `MAX_PATH` for a relative path, and + whether it does so without re-parsing it. It does both, and the regularize-then-prefix hypothesis + is falsified. -> [completed 2026-09-04](COMPLETED-CHECKLIST.md#m351) + ## M-inf -- Parked Ungated work with no identified predecessor deliverable. diff --git a/COMPLETED-CHECKLIST.md b/COMPLETED-CHECKLIST.md index 8b89f5c4..5a339696 100644 --- a/COMPLETED-CHECKLIST.md +++ b/COMPLETED-CHECKLIST.md @@ -1585,6 +1585,125 @@ with the three rejected alternatives). version in prose each produce a distinct located failure; exit 2 is reserved for configuration errors and the script is cwd-independent. +## Moved 2026-08-31 -- topology provenance: a topology now carries where it came from, and cannot pass as measured + +# Checklist: topology provenance + +**Problem.** [crates/windows-topology-sys/src/topology.rs](crates/windows-topology-sys/src/topology.rs) +documents that a `Topology` is "built either by `Topology::discover` from the running system, by hand, +or (with the `serde` feature) by deserializing a fed-in description" -- and **nothing distinguishes the +three once built**. `Topology` derives `Default`, has public fields, and derives `Deserialize`. There is +a passing test that parses a *Linux-shaped* description, complete with an ACPI SLIT-style distance +matrix, on a Windows-only crate. A consumer handed that value treats another machine's topology, or a +fabricated one, as this machine's truth. + +This is not hypothetical for the work in flight. `probe-core-affinity` needs synthetic multi-node +topologies precisely because no NUMA machine is available, and the whole point of a probe is that its +output is believed. + +**Decision.** Topology content carries its own provenance, defaulting to the *untrusted* value so that +forgetting is safe and claiming is deliberate. Persisted forms carry it visibly, and loading can only +ever downgrade -- a file cannot assert that it is this machine. + +Related: [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M-inf.4, which is what surfaced this. + +## M1: the marker, and its invariants + +- [x] **TP-1.1** -- Add `Provenance` to `windows-topology-sys` with three states ordered by trust: + `Measured` (read from the running system), `Restored` (deserialized from a description of some + machine), `Synthetic` (constructed by hand). **`Synthetic` is `Default`.** That is the load-bearing + choice: `Topology::default()`, `..Default::default()`, and any construction that omits the field all + come out tainted, so a caller must do work to claim data is real rather than work to admit it is not. + Document that the threat model is *accident*, not forgery -- a caller who writes + `provenance: Measured` over fabricated data has lied deliberately, and no type prevents that. + +- [x] **TP-1.2** -- Add the field to `Topology` and set `Measured` in `discover()`. This is a **breaking + change** for struct-literal construction, and deliberately so: every existing site is forced to state + which kind of data it holds. Update the crate's own tests and every dependent that constructs a + `Topology` by hand. + +- [x] **TP-1.3** -- Serde: serialize the marker so it is *visible* in the persisted form, and + **downgrade on load** -- `Measured` becomes `Restored`, everything else is unchanged. The rule is + **never upgrade**, so a hand-edited `"provenance": "measured"` is ignored rather than honoured. A + description absent the field loads as `Synthetic`. Test each of the four load cases, including that a + round trip of a measured topology does not come back measured. + +## M2: making it loud where it is read + +- [x] **TP-2.1** -- `Fingerprint` in [crates/windows-placement-probe/src/fingerprint.rs](crates/windows-placement-probe/src/fingerprint.rs) + carries the provenance and renders it **first and unmissably** when it is not `Measured`. The + fingerprint string is documented as canonical, so string equality is a usable comparison -- which + means the marker must be *inside* the string, or a synthetic host could compare equal to a real one. + That is the specific bug this prevents, not merely a display nicety. + +- [x] **TP-2.2** -- Every probe banner and every persisted probe line inherits it, since + `print_banner` and `Slice` are what end up pasted into checklists and design notes. A number quoted + from a synthetic run must arrive already labelled, because the label is what a reader will not think + to ask for. + **Done, and the banner inherits it by construction** -- it embeds the fingerprint's own `Display` + rather than re-rendering, so the two cannot drift. `print_banner` was split so the line is available + as a string (`banner_line`) and the marker's arrival is asserted rather than confirmed by reading a + format string. + **`Slice` deliberately carries no marker of its own, and the reason is structural rather than an + oversight.** A `Slice` records which processors a measurement was pinned to, and one can only exist + from a real `measure()` run: `measure` takes no injected topology (and + [crates/windows-placement-probe/src/core_affinity.rs](crates/windows-placement-probe/src/core_affinity.rs) + now documents why it must not), and pinning to a processor that does not exist panics. A slice is + therefore always real, and it is always printed beneath the banner that carries the host's + provenance. **If `measure` ever does gain such a seam, this reasoning collapses and `Slice` needs its + own marker** -- which is a second, independent reason not to add one. + +## M3: closing the loop with the probes + +- [x] **TP-3.1** -- Reconsider whether `probe-core-affinity`'s synthetic hosts should be expressed as + `Topology` values rather than as `Vec`. Going through `Topology` would exercise the + provenance path end to end and let a synthetic *NUMA* host drive selection through the real + `discover_places` conversion; staying at `ProcessorPlace` keeps the tests pure and fast. **Decide on + the evidence, and record the decision either way** -- this item is not "do it", it is "choose". + Note the constraint from + [crates/windows-placement-probe/src/core_affinity.rs](crates/windows-placement-probe/src/core_affinity.rs): + `measure()` must still not gain a topology-injection seam, whatever is decided here. + + **Decided: both, because they are tests of different units -- and the evidence that settled it was a + hole, not a preference.** `classify`, `representative_pairs` and `node_pairs` take `ProcessorPlace`; + that *is* their input type, so `ProcessorPlace` fixtures test them at their own boundary and stay. + What was missing is that `discover_places` -- which carries the rules for which cache level + partitions the machine, which core and class each processor belongs to, and which NUMA node -- took + no argument, called `Topology::discover()` internally, and appeared in **zero tests**. It was not + merely untested; it was untestable. + + **That hole was load-bearing and is now proven closed.** The NUMA lookup added earlier could not be + verified on a single-node host, because a correct map and a completely broken one both yield node 0. + Replacing the whole lookup with a hardcoded `0` was tried against the suite as it stood before this + item: **it passed everything.** Against the suite now, three tests fail. The `ProcessorPlace` + fixtures could never have caught it, because they encode what a test author *assumed* the conversion + produces -- the exact "depend on specified primitives, never on incidental behavior" trap. + + **A pure `places_from_topology` seam was added; `measure()` still has none.** The distinction is the + rule worth keeping: *a seam that only moves data is safe; a seam that lets fabricated labels reach + real hardware is not.* Feeding a synthetic topology to a conversion yields synthetic positions, which + is what the caller asked for and cannot be mistaken for a measurement. Feeding one to `measure()` + would produce genuine timings under fabricated node ids, because a synthetic topology's processor + *numbers* are still valid on the real host and every pin would succeed. + +## Moved 2026-08-31 -- the sabotage harness became a tool + +### M34.1 -- Promote the ad-hoc sabotage harness into a reusable tool. *(completed 2026-08-31 20:03:57 -04:00)* +- [x] **M34.1** -- Promote the ad-hoc sabotage harness into a reusable tool. **Done.** + [tools/run-sabotage.ps1](tools/run-sabotage.ps1) plus + [tools/README-sabotage.md](tools/README-sabotage.md), driven by a `sabotage.json` kept beside the + code it patches; the first is + [crates/windows-waitable-queues/sabotage.json](crates/windows-waitable-queues/sabotage.json), whose + nine entries reproduce the M30.4/M30.5 sweep exactly through the promoted tool. + Six of the tool's own guards were verified by making each one fire: a name filter matching nothing, + a missing file, a dirty target, a pattern matching 14 sites instead of 1, a patch that changes + nothing, and a deliberately red baseline. A harness whose guards are untested is the thing it exists + to warn about. + Two subtleties are recorded in [DESIGN-NOTES.md](DESIGN-NOTES.md) -> `Sabotage sweeps` rather than + left in the script: a **survived** sabotage may be a defect in the *sabotage* rather than a hole in + the tests, which is why the patch is now printed on every unexpected result; and a **too-short + timeout manufactures a false "caught"**, crediting tests with catching a defect they never ran + against, so the bound errs generous. ## Moved 2026-09-01 -- Thread ambient mutation gaps ### M23.6 -- Close mutation gaps with deterministic fault injection and exhaustive assertions. *(completed 2026-09-01 20:25:29 UTC-04:00)* @@ -1597,3 +1716,557 @@ capture-set formatting, declared emptiness, and restore reports; then rerun muta classify any survivors that are behaviorally equivalent. The final mutation run tested 233 mutants: 142 were caught, 91 were unviable, and none were missed. + +## Moved 2026-09-02 -- M1 of the topology/queues release: the public surface settled before publication + +Every item complete. The milestone existed because its decisions were free before 0.1.0 and expensive +after: a deleted public type costs a yank-and-migrate once published, and `Reserving`'s associated +type needed its bound before any caller could depend on the unbounded form. Moved from +[CHECKLIST-ship-topology-and-queues.md](CHECKLIST-ship-topology-and-queues.md). + +### M1: settle the public surface before it is public + +- [x] **SH-1.1** -- **MIRRORS [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M31.8 -- one piece of + work seen from two plans. Check both off in the same commit; neither is done alone.** + **Decide M31.8 (merge-or-delete for `slotwise_mpsc` and `reserving_mpsc`) before the first + publish, not after.** This is the highest-leverage item in the file and it is release-blocking for a + mechanical reason: the decision may *delete a public type*. Doing that before 0.1.0 costs nothing; + doing it after means a breaking release, a yank-and-migrate for anyone who adopted it, and a + permanent line in the changelog explaining why a shape existed for one version. + The measurement is already done and agrees across both architectures -- see M31.5 and M31.7 in + [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) -- so this needs a decision, not more work. + +- [x] **SH-1.2** -- **GOVERNS [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M31.6 -- this is not + that item and does not complete it.** It decides only whether M31.6 blocks SH-4.3. If the answer is + "it gates", record that on M31.6 and SH-4.3 cannot proceed until M31.6 is done; if "it does not", + record that too, so a later reader does not mistake a considered choice for an oversight. **Checking + this off never checks off M31.6.** + **Decide explicitly whether M31.6 (loom verification) gates 0.1.0**, and record the + answer either way rather than letting it drift into "not yet". + + **Decided: it does not gate 0.1.0. It gates 1.0, and the gap is disclosed in the crate's own + documentation rather than left for an adopter to discover.** Recorded as D-31. + + Three findings drove it, and the second was not expected: + + - **Loom would close the demonstrated gap.** The sabotage sweep showed a weakened `Acquire` on the + producer's load of `head` survives the whole suite. That defect lives in queue code, which is + exactly what loom models well. + - **Loom would *not* close the gap where a real bug actually occurred.** The doorbell's correctness + is the interleaving of an `AtomicBool` mirror with real `SetEvent`/`ResetEvent` syscalls. Loom + models the atomics and cannot model the syscalls; stubbing them tests a *model* of `SetEvent` + rather than `SetEvent`. D-15's lost wakeup -- the only ordering bug this crate has actually had -- + was found by sabotage, and loom would not have found it. So loom is valuable and is **not** the + thing standing between this crate and confidence about its hardest part. + - **The risk loom addresses is mostly regression risk**, and that risk is lowest now. The orderings + are believed correct and were reasoned about at the time; sabotage *introduced* the weakening to + prove the suite was blind to it. Regression risk rises with contributors, changes, and consumers + -- all of which start after publication, not before. + + Against that, gating would block 0.1.0, and through it the placement tool and the NUMA measurements + from other people's machines that this whole sequence exists to obtain. Loom is invasive work: every + atomic in the crate goes behind a `cfg` shim across four modules. + + **The disclosure is what makes this a decision rather than a punt**, and it is not optional: the + crate documentation states what is verified, states that stress testing here is *known* not to catch + ordering defects and cites the measurement showing it, and says loom is planned before 1.0. An + adopter then makes their own call with the same information we have. `0.x` carries the rest. + The reason it deserves a deliberate answer rather than a default: the sabotage sweep demonstrated + that weakening the producer's `Acquire` load of `head` to `Relaxed` left **all twenty tests green**, + while every logic defect injected beside it was caught. So this is not an untested-by-omission gap, + it is a gap this workspace has *evidence* the existing tests cannot close. Publishing a lock-free + queue with it open is a defensible choice; making it unknowingly is not. + +- [x] **SH-1.3** -- **Qualify both MPSC shapes by name.** `mpsc` beside `reserving_mpsc` made one + canonical by implication -- which contradicts this crate's own "no shape is the canonical one", and + after SH-1.1 is simply false. Renamed to `slotwise_mpsc`, which names its claim protocol: it claims + slot by slot with no shared counter. `sequence_mpsc` was considered and rejected for inviting the + reading that it alone preserves FIFO order, which both shapes do. Recorded as D-30. + **Belongs in M1 for the same reason SH-1.1 does**: it is a public-surface change, free before the + first publish and a breaking rename with a deprecation path afterwards. + +- [x] **SH-1.4** -- **State the algorithms' pedigree and why an existing crate is not used.** A public + concurrent-queue crate has to answer both questions or a reader assumes the worst: that the + algorithms are homegrown, and that the author did not look at the alternatives. + Neither is true, and the honest answers are load-bearing. The algorithms are *published designs* + chosen deliberately, because a concurrent queue is a bad place to be original -- the failure mode is + a reordering that appears on one machine, under load, months later. And the reason no channel crate + fits is structural rather than dismissive: **on Windows, waiting is a kernel-object operation**, so a + queue whose readiness is not a `HANDLE` cannot join a `WaitForMultipleObjects` alongside an I/O + completion, a process handle, or a cancellation event -- however good its own blocking receive, and + however rich its own `select`, which can only select over its own channels. + Written into both the crate docs and the README, because docs.rs shows one and crates.io the other. + +- [x] **SH-1.5** -- **Bound `Reserving::Reservation<'a>` so a generic caller can redeem what it claims.** + Done: the `Claim` trait carries `send` and `is_disconnected`, `Reservation<'a>` is bound on it, and + both reservation types implement it as forwarders. 87 lines across five files, no concrete signature + changed, `slotwise_mpsc` untouched because it does not implement `Reserving` at all. Mutation-tested + rather than assumed: 62 mutants over the whole reservation surface report 0 missed, and the first + run found a real gap -- `is_disconnected` stuck at `false` survived on `spsc`, because the connected + case was asserted there and the disconnected case only on the other shape. + The associated type is declared with no bound at all, so a caller generic over + [`Reserving`](crates/windows-waitable-queues/src/traits.rs) can call `reserve()` and then do + nothing with the result except drop it. `reserve` is `#[must_use]` precisely because a held claim + withholds capacity from every other producer -- and the one operation that discharges it, `send`, + is inherent to each shape's concrete type and unreachable through the trait. The trait cannot + express the operation it exists for. + + **The two implementors already agree exactly, so this is additive**: both + `spsc::Reservation<'a, T>` and `reserving_mpsc::Reservation` already have + `send(self, item: T) -> Result<(), Disconnected>`, `is_disconnected(&self) -> bool`, and a + `Drop` that returns the slot. No concrete signature changes; nothing to migrate. + + Add a `Reservation` trait carrying `send` and `is_disconnected`, bound the associated type on it, + and implement it for both types. `is_disconnected` is included rather than deferred for the reason + the `Reserving` docs give at length -- a caller needs to learn the stream ended *before* doing the + work the claim was taken for -- and because `reserving_mpsc`'s reservation is `Send`, so it may be + redeemed on a thread holding no producer handle to ask instead. Adding it later is the same + breaking change, merely deferred. + + **Why this blocks rather than waits.** Adding a bound to an associated type is a breaking change + to the trait: every implementor must then satisfy it. It is free while the crate is unpublished + and a major bump with a migration afterwards, and this is the milestone that exists to settle + exactly that -- see SH-1.1 and SH-1.3, both landed on the same "free before the first publish" + reasoning. D-3 already makes this argument ("the trait *shape* is fixed now so signatures stay + compatible"); this is the same reasoning applied to a piece it missed. Pull request #56 is what + puts these traits in front of consumers, so the window closes when it merges. + + **How it surfaced**, recorded because the route is the useful part: not from review and not from a + failing test, but from a `cargo mutants` run showing that nothing exercised the capability traits + at all, and then from being unable to write the obvious generic test for `Reserving` -- the test + in [traits/tests.rs](crates/windows-waitable-queues/src/traits/tests.rs) is scoped to claim-and-release + and says so. A contract gap presenting as an untestable API is a signal worth keeping. + + Extend that test to claim-and-redeem through the trait as part of this item, since it is the + check that would have caught the gap in the first place. + +## Moved 2026-09-02 -- M7 through M13: seven PR #56 review rounds, all findings resolved + +Seven consecutive automated-review rounds on the pull request that ships `windows-topology-sys` +0.2.0 and `windows-waitable-queues` 0.1.0, kept as milestones so each round's findings stayed +attributable to the round that raised them. All items complete. Moved from +[CHECKLIST-ship-topology-and-queues.md](CHECKLIST-ship-topology-and-queues.md); the two later rounds +(M14, M15) remain there because they carry open work. + +### M7: PR #56 automated-review round + +The findings an automated review raised against the pull request that lands this work, verified against +the source before being accepted. Each item names what was checked, so a later reader can tell a real +repair from a reviewer's guess that was taken on trust. + +- [x] **SH-7.1** -- **`reserving_mpsc` reports `Full` from a claim word that was never current.** + `push` and `reserve` load the claim word relaxed, then test room with + `has_room_beyond_reservations(position, reserved)`, which computes + `position.wrapping_sub(head)`. If other producers claim and publish past `position` and the consumer + drains them while this thread is between the load and the room check, `head` passes the stale + `position` and the subtraction wraps to near `u32::MAX` -- so the queue reports `Full` (and records a + refusal) at the moment it is empty, and `reserve` returns `None` for the same reason. The compare-and- + swap that would have caught the staleness is never reached, because both paths return before it. + Re-read the claim and retry when it moved; report no room only from a word still current. + +- [x] **SH-7.2** -- **The NUMA cross-check compares a count against a highest identifier.** + `windows-platform-probes`'s `Observation::cross_check` compares `numa_domains` (a count of memory + domains) with `GetNumaHighestNodeNumber() + 1`. Windows documents that value as the highest node + *number*, and does not guarantee node numbers are dense -- nodes 0 and 2 give a count of 2 and a + highest of 2, and the probe then reports a parsing regression on correct hardware. Memory domains + already carry the node number in `Domain::id`, so compare highest against highest. + +- [x] **SH-7.3** -- **A cache level is called a partition without checking that it is one.** + `cache_partitions_at_level` deduplicates by equal processor set, which is exactly right for the + measured case it was written for (L1i and L1d over identical sets). It does not establish a + *partition*: `Topology` is deliberately constructible by hand and by deserialization (D-12), so + distinct-but-overlapping sets reach `outermost_partitioning_cache`, which returns them as domains a + consumer then double-counts. Require the distinct sets to be pairwise disjoint before a level + qualifies as partitioning. + +- [x] **SH-7.4** -- **`windows-waitable-queues` cannot build its documentation on docs.rs.** The crate + is Windows-only and imports `std::os::windows::io` unconditionally, but its manifest omits the + `[package.metadata.docs.rs]` target block that every other published Windows-only crate here carries, + so docs.rs would build it for its default Linux target and fail. Add the same block. + +- [x] **SH-7.5** -- **The mutant injector replaces every occurrence on the line, not the first.** + `tools/inject-mutant.ps1` calls the *static* `[regex]::Replace(input, pattern, replacement, 1)`, whose + fourth parameter is `RegexOptions` -- `1` is `IgnoreCase`, not a replacement count, and no static + overload takes a count at all. The tool therefore does precisely what its own header comment says it + exists to avoid. Fix the replacement, refuse a line whose pattern occurs more than once unless a + column disambiguates it, verify the baseline is green before trusting a "caught", run with all + features so a feature-gated mutation is not reported as surviving, perform the mutating write inside + the guarded region so a failed write still restores, and route its output through one sink. + +- [x] **SH-7.6** -- **A spike that fails to run is reported as a finding about the machine.** + `tools/run-numa-spikes.ps1` checks the exit code of `cargo build` but not of `cargo run`, then decides + vacuity by searching the output for `VACUOUS`. A crashed spike prints no such line, so the summary + says "**NOT vacuous -- this runner has more than one NUMA node**" and the script exits 0. That is the + instrument breaking while claiming a result, which the script's own documentation says is the one + thing worth failing over. + +- [x] **SH-7.7** -- **Two tools write output from several sites, and two hazards remain in the + sabotage/mutation harness.** `tools/check-publishable.ps1` and `tools/inject-mutant.ps1` each call + `Write-Host` from several places, against the repository's one-output-sink rule. + `tools/run-sabotage.ps1` performs its patching write before entering the `try` whose `finally` + restores the file, so a write that throws part-way leaves the clean source damaged. + `tools/run-mutants.ps1` derives a deterministic output directory per package or file, so a second run + of the same scope overwrites the analysis the parameter documentation promises to preserve. + The placement probe's tests name scratch directories without the process id, so two concurrent test + processes -- which the documented `-j 2` mutation workflow creates -- delete each other's fixtures. + +- [x] **SH-7.8** -- **Reply to every thread and resolve the ones that are addressed**, including the one + finding that was checked and found not to hold: `GetSystemDirectoryW` returning exactly the buffer + length is unreachable (success excludes the terminator, failure includes it and so exceeds the + buffer), though the guard is widened anyway so the next reader need not redo the analysis. + +### M8: PR #56 third review round (suppressed findings) + +The reviewer generated no new inline comments in these rounds and instead listed **suppressed** findings in +the review body, so none of them arrived as a resolvable thread. They are recorded here because a finding +that produces no thread is otherwise invisible to the "are all comments resolved?" check that gates merge. + +- [x] **SH-8.1** -- **The contention probe times thread creation, and lets early producers run alone.** + All five timed runs in `windows-platform-probes`'s `queue_contention` start the clock *before* + `thread::scope` spawns anything, and every worker begins pushing the moment it is spawned. At 50,000 + pushes each, an early producer can finish a large uncontended prefix -- or finish outright -- while the + last threads are still being created, so a row labelled 16 or 32 producers may never have had 16 or 32 + contenders. The measured interval also includes spawn cost. This is not a cosmetic inaccuracy: the + module's own header says these numbers decide whether two speculative queue shapes get written at all + and whether the two shipped shapes merge. Hold every participant -- producers *and*, in the drained + runs, the consumer -- at a start barrier, and start the clock when it releases. + +- [x] **SH-8.2** -- **A failed backup write leaves a truncated file under the canonical name.** + `write_backup_to_new_file` reserves the name with `create_new` and then `write_all`s through `?`, so a + disk-full or quota failure returns an error while leaving a zero-length or partial `.json` behind. That + file is indistinguishable from a real record to whoever collects it, and the next run's collision + suffix steps politely around it. Publish by rename: write the bytes to an exclusively-created temporary + in the same directory, flush, and move it onto the reserved name only once the write has succeeded. + +- [x] **SH-8.3** -- **`places_from_topology` drops processors and invents NUMA membership.** + Two defects in one conversion, both reachable only through a hand-built or deserialized `Topology` -- + which is exactly the input this seam exists to accept (D-12). + It iterates `class_of`, which is populated only from `DomainKind::Core` domains, so an online processor + with no core domain is **silently absent from the result** -- and the documented core-id fallback + beneath it, written to keep group 1's cpu5 distinct from group 0's, is unreachable dead code as a + direct consequence. + It then defaults absent NUMA membership to `unwrap_or(0)`. That is the right answer only when the + topology names no memory domain at all; when it names nodes 1 and 2, it **fabricates node 0** and files + a processor under a node the machine does not have -- the precise failure this crate's own rule + ("a seam that only moves data is safe; a seam that lets fabricated labels reach real hardware is not") + exists to prevent. + Iterate the online processors so every one is placed, and refuse a topology that names memory domains + but not this processor's, rather than inventing one. + +### M9: PR #56 fourth review round + +- [x] **SH-9.1** -- **Both bounded shapes could report a length larger than their capacity.** + `len` reads the producer-side position and then `head`, which are two instants; a consumer draining + past the sampled position makes the wrapping subtraction yield a number near the integer maximum. The + comment beside it claimed the overestimate was "safe in the direction that matters for a backpressure + gauge", which is true of a *bounded* overestimate and not of `usize::MAX`. Both are now clamped to the + capacity, so the skew still resolves towards full -- the safe direction -- while the impossible value + is gone. + +- [x] **SH-9.2** -- **`reserving_mpsc` inherited a `remaining()` that counted reserved slots as room.** + `Bounded::remaining` defaults to `capacity - len`, and this shape's `len` excludes reservations by + design, so an empty queue of four holding one reservation answered four while only three items fit -- + promising room for a push guaranteed to be refused. Overridden on both handles and both trait impls, + reading the packed claim word **once** so the position and the reservation count cannot be sampled at + different instants; `is_full` is now defined in terms of it rather than restating the rule. + +- [x] **SH-9.3** -- **The pull request description described the release plumbing, not the product.** + The body framed the change as CI and provenance work and mentioned `windows-waitable-queues` only + under release tracking, while the majority of the diff is that crate's public API and its three + lock-free queue implementations. Rewritten to lead with the shipped surface. + +### M10: PR #56 fifth review round + +- [x] **SH-10.1** -- **`BOUNDS_MAX` does not compile on a 32-bit target.** `reserving_mpsc`'s maximum + was a flat `1 << 31`, derived from the packed position's width alone. On a 32-bit target the + crate-wide `WRAPPING_MAX_CAPACITY` is `usize::MAX / 2`, which is `2^31 - 1` -- *narrower* than the + packing -- so the const assertion that no shape may exceed it fails the build outright, for every + capacity including the small valid ones. Now the narrower of the two limits, kept a power of two so + the value stays one a caller could actually pass. Verified in both directions against a real + `i686-pc-windows-msvc` check: the old constant fails with `E0080`, the new one compiles. + +- [x] **SH-10.2** -- **The backup's final name was visible empty for the whole write.** The previous + round reserved the destination with `create_new` and renamed onto it, which fixed the truncated-file + case and left a worse one: an empty file under the record's own name for the duration of the write, + and permanently if the process was killed in that window -- contradicting the absent-or-complete + guarantee its own doc comment claimed. Publication is now a single atomic no-replace `MoveFileExW` + from a fully-written temporary. `std::fs::rename` cannot express this: on Windows it always passes + `MOVEFILE_REPLACE_EXISTING`, so it would clobber a record a concurrent run had placed. + +- [x] **SH-10.3** -- **The tool discovered the topology three times.** The plan used one reading, the + fingerprint another, and `core_affinity::measure` a third, so a processor going offline mid-run could + have the announced plan, the recorded host, and the measured rows describing different machines with + nothing saying which. The plan and the fingerprint now derive from one `Topology::discover`. + `measure` still discovers its own, and deliberately so: its documentation refuses a + `measure_with(places)` seam because a supplied list's processor *numbers* stay valid on the real host + while its node labels need not, so every pin would succeed and real timings would be filed under + fabricated labels. Its rows carry their own places, so each row states what it measured. + +- [x] **SH-10.4** -- **`spsc` had the same `remaining()` defect, and it was missed.** The previous round + corrected `reserving_mpsc` and stopped there, but `spsc` implements `Reserving` too -- so reserving + every slot left it reporting the full capacity as available while both `push` and `reserve` refused. + Its `Bounded` impls now override `remaining` on the producer *and* the consumer, its `len` is clamped + to the capacity like the other two shapes', and `is_full` is defined in terms of `remaining` rather + than restating the rule. The trait's default now documents that a `Reserving` shape must override it, + so the next shape to reserve does not inherit the same wrong answer silently. + +- [x] **SH-10.5** -- **The high-water depth could record a peak the queue never reached.** + `reserving_mpsc`'s `publish` sampled the depth from its own position and a relaxed load of `head`, + ungated and unclamped. `slotwise_mpsc`'s twin is bounded by construction -- its producer's acquire + load of the slot's sequence synchronizes-with the consumer freeing that slot, so `head` cannot be + older than `position - capacity + 1` -- but this shape has a second entry point with no such edge: + `Reservation::send` redeems without a room check, so the only `head` its thread is ordered against is + the one *`reserve`* read, which may be arbitrarily old by the time the reservation is redeemed. The + sample is now gated on tracking (parity with the twin), read before publication, and clamped to the + capacity. + `Observable::high_water`'s contract is corrected to match what all three shapes actually deliver: an + **upper bound** on the true peak, never below it and never above the capacity, with the reason the + cheap sample is preferred to an exact count. Counting exactly would put a read-modify-write on a line + shared by every producer and the consumer into every push and every pop -- the line this crate pads + its positions apart to keep out of the hot path. + +### M11: PR #56 sixth review round + +Three findings against `places_from_topology`, all of the same shape, plus three against the +mutation wrapper. The conversion's three silent fallbacks are replaced by one rule. + +- [x] **SH-11.1** -- **Three fallbacks each invented an answer that reads as a real one.** + `places_from_topology` accepted a topology whose domains do not cover every processor, and filled + each gap with a value indistinguishable from a measured one. A processor absent from every core + domain was given a synthetic core id derived from its group and number, which can equal a real + core domain's id -- `classify` then reports two processors as SMT siblings when one's core is + merely unknown. Its efficiency class became `0`, which is also a genuine Windows class, so + `within_class_pair` reports a same-class pair against a real class-0 core. Its cache domain became + `None`, which the type already means "no cache level partitions this machine" -- so two processors + omitted from an incomplete partition compare equal and serialize a confident same-cache + measurement. + The three share one cause: an absence was read as a value. The rule now distinguishes *uniform* + absence from a *gap*. A machine that reports no core domains at all, or no partitioning cache + level, has told us something true about itself and still converts. A machine that places every + other processor but not this one has told us nothing about this one, and the conversion refuses: + `places_from_topology` returns `Err(UnplacedProcessor)` naming the processor and, in a new + `MissingPlacement` field, which of core / cache domain / NUMA node was missing. + `MissingPlacement` is `#[non_exhaustive]`. + Core and efficiency class are two spellings of one rule -- `Topology::cores()` filters to + `DomainKind::Core`, so a processor's class is known exactly when its core is -- and an + `EfficiencyClass` variant written for the second was removed on discovering it is unreachable. + Sabotage confirms the pair behaves that way: removing either refusal alone leaves the suite green, + because the other still fires; removing both fails two tests. + +- [x] **SH-11.2** -- **The mutation wrapper's output directory could collide.** The stamp has + one-second resolution, so two runs launched in the same second -- a script starting several scopes + at once, which is exactly the case that wants separate output -- selected the same directory and + interleaved their results. A short random suffix now follows the stamp, which still sorts + chronologically. + +- [x] **SH-11.3** -- **The wrapper terminated fault handlers it did not start.** Cleanup matched + `WerFault` / `WerFaultSecure` / `vsjitdebugger` by name across the whole session, so a crash report + the user was reading or a debugger attached to an unrelated process was killed by a mutation sweep. + The wrapper now records the ones already running at startup and skips them. + +- [x] **SH-11.4** -- **The `mutants.out` nesting finding does not hold; documented in place.** + The report was that `Join-Path $OutputDirectory 'mutants.out'` doubles a path cargo-mutants already + appends. It does not: cargo-mutants treats `--output` as the parent and creates `mutants.out` + inside it. Verified on disk -- a run with `--output .scratch\mutants-encoding-` produced + `\mutants.out\caught.txt` with 22 lines, matching the 22 caught the wrapper reported. A + comment now records the evidence, since the path reads like a duplication and has been challenged + once already. + +- [x] **SH-11.5** -- **A hard-killed run could block the next one's backup entirely.** The temporary + was named for the record plus this process's id and nothing else, and created with `create_new`. A + run killed mid-write leaves that file behind, and Windows reuses process ids -- so a later run + issued the same id found the corpse under the only name it would ever try. The resulting + `AlreadyExists` left `write_temporary` *before* the caller's suffix loop was reached, so the whole + backup failed rather than landing under a next-best name. The temporary now carries its own + attempt counter, matching the final name's budget; a stale file is stepped around rather than + overwritten, since it belongs to whatever left it. + +- [x] **SH-11.6** -- **Both tools bypassed their own single-output-sink contract.** `run-mutants.ps1` + emitted its per-category summary directly to the success stream, and `run-sabotage.ps1` did the + same for the `-List` output, every blank line, the result table, and the injected patch text -- + each contradicting the `Write-Report` doc comment directly above them. All now route through the + sink, which gained pipeline binding so a formatted table can flow into it. Verified: the `-List` + path emits zero objects to the success stream. + +- [x] **SH-11.7** -- **The sabotage harness silently narrowed its own sweep.** Found while checking + SH-11.6's output: the harness prepends the `test` subcommand to a manifest's `testArgs`, but nine + of the eleven manifests already begin with `test`. The result was `cargo test test -p ...`, in + which the second word is not a subcommand but a TESTNAME filter -- a sweep claiming to run a + package's suite while running a subset of it, the same false-green this tool exists to prevent. + The vector is now normalised so either manifest spelling produces one `test`. + **No prior verification was weakened**: every crate swept so far keeps its tests under a + `mod tests`, so the accidental filter matched all of them, and all fourteen recorded baselines + report "0 filtered out". The defect was latent, and would have appeared on the first crate laid + out differently. + +### M12: PR #56 seventh review round + +- [x] **SH-12.1** -- **The banner undercounted the machine it was about to measure.** + `Fingerprint::processors` is documented as the logical-processor count but was summed over + core-domain membership, which agreed with that meaning only while every processor was guaranteed to + sit in a core domain. SH-11.1 stopped guaranteeing it: `places_from_topology` now explicitly accepts + a topology naming no cores and places every online processor. The banner consequently read + `0p/0c` for a machine the measurement was about to use four processors on -- a defect this branch + created rather than inherited. + The count is now read off `topology.processors` with the same `online` filter the placement applies, + so the summary counts exactly what the measurement will use. `cores` deliberately still counts core + domains: zero there is the honest report that the topology named none. + Sabotage-verified. Two new tests pin both directions -- an uncored processor is still counted, an + offline slot is not -- and each asserts equality against `places_from_topology`'s own output rather + than a literal, so the two cannot drift apart again. + +- [x] **SH-12.2** -- **Two consequences of that fix, found by sweeping it rather than reported.** + `cache_domain_sizes` fills itself with the processor count when no cache level partitions the host, + so the bare-machine render silently improved from `L-[0]` to `L-[4]`. + `numa_node_sizes` did not, and now does not sum to `processors` in that one case: it reports the + nodes the topology *named*, and a bare topology names none, while every placement still reports the + documented node-`0` default. Keeping it that way is deliberate -- the cache list can afford to fill + itself because `L-` marks the absence, and `numa[4]` would be indistinguishable from a host that + genuinely reported one node of four. The behaviour is now documented on the field and pinned by a + test that asserts the whole render, including the `!!SYNTHETIC!!` provenance marker. The stronger + fix needs a marker, which is a serialized field and so a schema bump, tracked as + [CHECKLIST-placement-tool.md](CHECKLIST-placement-tool.md) `PT-6.2`. + +### M13: PR #56 eighth review round + +- [x] **SH-13.1** -- **A record could splice two machines together.** The tool announced a shape read + at one instant while `core_affinity::measure` discovered again at another, so a processor going + offline -- or moving group or node -- between them produced a record whose `host` described one + machine while every row was measured on a different one. Nothing in the file said so, and the host + is precisely what a reader interprets row sets *through*. + `measure` now reports the shape it actually ran on, as `Observation::host`, which is the fix that + keeps the anti-synthetic boundary intact: the measurement still discovers for itself and no seam + accepts a fabricated shape from outside. `SubmissionRecord::new` refuses when the announced and + measured hosts differ, so the splice is unrepresentable rather than merely avoided at the one + current call site; the tool checks first anyway and reports the disagreement in terms a runner can + act on. Refusing rather than silently recording the measured shape, because the notice is what the + runner consented to. + +- [x] **SH-13.2** -- **The tool wrote from 54 independent print sites.** The repository's + one-output-sink rule requires an output abstraction at the *first* output site so the storage + target and the formatting stay separable from the call sites that compose content. This binary had + none, which is why its collection notice -- a disclosure a runner reads before agreeing to publish + facts about their machine -- could only be exercised by running the process and capturing stdout. + A `Sink` trait now carries the two streams the tool genuinely has, `print_collection_notice` and + `print_plan` became `render_*` functions returning a `String` (matching the idiom the record report + already used), and `main` is the only place that names the real streams. + Verified as a pure refactor by comparing the built binary's output before and after: `--preview` + and `--help` are **byte-identical**, and `--version` differs only by the build identity correctly + reporting the working tree as `DIRTY`. Eight new tests cover what was previously unreachable, + including that the notice shows the model rather than describing it, that a withheld model reads + differently from one the host would not report, and that the two streams cannot satisfy each + other's assertions. + +- [x] **SH-13.3** -- **The two new probes wrote from 94 independent print sites between them.** Same + rule as SH-13.2, in [core_affinity.rs](crates/windows-platform-probes/src/bin/core_affinity.rs) + (67 sites) and [doorbell_cost.rs](crates/windows-platform-probes/src/bin/doorbell_cost.rs) (27). + A `Report` sink now lives in [report.rs](crates/windows-platform-probes/src/report.rs), shared by + both. One stream, not two: unlike the placement probe these have only ever written to stdout, and + inventing a diagnostic stream they do not use would be adding a distinction the tools do not make. + Each `main` is now three lines -- measure, render, emit -- and is the only place naming the real + stream. + One find during the conversion that the mechanical part would have missed: `render` called + `fingerprint::print_banner()`, which writes to stdout *itself*. Left alone it would have put the + identifying line on the terminal while leaving it out of the returned report, so a captured report + would be missing the one line saying which machine produced it -- and the `!!SYNTHETIC!!` taint + marker with it. `banner_line()` already existed for exactly this and is now used. + Verified as a pure refactor by running both probes before and after and comparing with numerals + masked (their output is timing-dependent, so byte equality is not available): 38 lines and 50 lines + respectively, **structurally identical** both times. + +- [x] **SH-13.4** -- **The other twelve probes still print directly, and now there is a sink to + adopt.** `probe-peer-index-cache` (55 sites), `probe-request-cost` (45), `probe-topology` (32), + `probe-queue-contention` (27), `probe-ioring` (24), `probe-completion-port` (22), + `probe-worker-context` (22), `probe-device-map` (21), `probe-cancel-io` (19), + `probe-pool-growth` (16), `probe-handle-state` (14), `probe-error-mode` (10) -- 307 sites. + Deliberately **not** done in the review round that introduced the sink: those probes predate it and + are outside that round's scope, and each conversion needs its own before/after comparison against + the probe's real output, which is what makes it a refactor rather than a rewrite. + Queued rather than left as a note precisely because a half-adopted abstraction is the state most + likely to be forgotten -- the next probe author will see twelve neighbours printing directly and + reasonably conclude that is the house style. +## Moved 2026-09-07 -- M34.4: the native-command guard became shared support, proven on both hosts + +### M34.4 -- Share the native-command guard through a dot-sourced `tools/common.ps1`, route every capture site through it, and prove it on both PowerShell hosts. *(completed 2026-09-07 21:16:31 -04:00)* + +Queued as a decision rather than a fix: the remaining sites needed a third and fourth copy of +one guard, and `tools/` had no sharing convention. The decision was to adopt one, and finding +out *which* one is the substance of this item. + +**Under Windows PowerShell 5.1, a native command that writes to stderr while +`$ErrorActionPreference` is `Stop` raises a terminating error when its stderr is redirected +with `2>&1`.** PowerShell 7 does not, which is why this class survives review and CI. + +**The obvious answer -- a `.psm1` -- is wrong, and wrong invisibly.** A scriptblock carries +the session state it was created in, so `Invoke-Native { cargo build }` runs in the caller's +scope while a module copy flips the preference in the module's scope; the flip never reaches +the call. Measured with the identical body in a module: **under 5.1 seven of eight cases +failed, while all eight passed under PowerShell 7.** Dot-sourcing puts the function in the +caller's own scope, where the plain assignment does reach the call. A module *can* be made to +work via `$PSCmdlet.SessionState.PSVariable.Set(...)` -- measured working on both hosts -- and +was rejected: the guard would rest on a subtlety that looks removable, and simplifying it back +reintroduces a defect that still passes on PowerShell 7. + +Delivered: + +- **[tools/common.ps1](tools/common.ps1)** -- `Invoke-Native` and `ConvertTo-OutputLines`, with + the "why not a module" argument and its measurement at the definition site. +- **[tools/test-common.ps1](tools/test-common.ps1)** -- eight cases covering capture, stream + merging, exit-code survival, diagnostic text, record flattening, preference restoration + (including when the command throws), and that `Stop` stays armed for non-native errors. It + runs its cases in the invoking host, then **re-invokes itself in the other one**, and treats + a missing host as a FAILURE rather than a skip -- a single-host pass is not the claim the + file exists to make. +- **Every capture site routed**: `run-numa-spikes.ps1` and `soak-flush-barrier.ps1` lost their + local copies; `run-sabotage.ps1` (`check-ignore`) and `test-run-sabotage.ps1` (`init`, + four `add -A`, and its child-process harness invocation) now go through the shared guard. + `run-mutants.ps1` needs none -- it redirects nothing, confirmed by experiment on both hosts. +- **CI runs both shells.** The `sabotage harness tests` job ran `shell: pwsh` only, which is + precisely why the defect was invisible; it now runs `test-common.ps1` plus + `test-run-sabotage.ps1` under **both** `pwsh` and `powershell`. + +Verified by sabotage: delivering the identical guard as a module turns the new suite red on +5.1 (7 of 8) while staying green on 7, so the suite detects the regression it was written for. +Both consumer scripts and the full sabotage suite pass on both hosts. + +**Deliberately not done: consolidating `Write-Report`.** Six scripts define one, and they are +not duplicates -- they differ in level vocabulary (`warn`/`warning`, `bad`/`error`, plus +`good`, `note`, `detail`, `heading`) and in rendering, with two emitting GitHub Actions +annotations and four emitting console colours. Merging them would change six tools' output to +remove a duplication that is only apparent. Recorded in +[DESIGN-NOTES.md](DESIGN-NOTES.md#tools-shared-support) so it is a decision rather than an +oversight. + +## Moved 2026-09-07 -- M35.1: what the long-path opt-in actually does + +### M35.1 -- Measure whether the long-path opt-in lifts `MAX_PATH` for a relative path, and whether it does so without re-parsing it. It does both, and the regularize-then-prefix hypothesis is falsified. *(completed 2026-09-04, archived 2026-09-07 21:40:23 -04:00)* + +**Measure whether the long-path opt-in lifts `MAX_PATH` for a *relative* path, and whether it does +so without changing how the path is parsed.** + +**Done 2026-09-04, and it settles a question that had produced three wrong answers from reading.** +`probe-long-path-aware` and `probe-long-path-unaware` in +[windows-platform-probes](crates/windows-platform-probes/src/long_path.rs) are the same code +differing only in whether their manifest declares `longPathAware`; `build.rs` embeds it into that +one binary via `rustc-link-arg-bin`, so every other probe binary is unaffected. + +**Result, on a host with `LongPathsEnabled=1`.** With the opt-in, a relative path of 429 characters +opens in every shape -- plain, containing `b\..`, and forward-slash separated. Without it, all three +are refused with `ERROR_PATH_NOT_FOUND` while the same shapes at 78 characters open. The targets are +created first, so a not-found from a file that provably exists is the length refusal. + +**So the documented reading was right and the review finding was wrong**: the opt-in covers relative +paths, and `MAX_PATH` binds them only in a process that has not opted in. + +**And the regularize-then-prefix hypothesis is falsified.** If the opt-in worked by prepending +`\\?\`, that prefix would disable `.`, `..` and forward-slash translation, so those shapes would +have failed past the ceiling while working below it. Both resolve at both lengths. The opt-in lifts +the length check without re-parsing, so there is no discontinuity at `MAX_PATH` for a caller of +`windows-file-watcher` to fall into. + +The measurement is recorded where the claim lives, in `Session::subscribe`'s note. + +*(Archiving note: the original body said the embed leaves "the other thirteen probes" unaffected. +There were sixteen probe binaries by the time this was archived, so the count was already stale and +would have been written into the record as a false number. Replaced with the count-free phrasing, +which stays true as probes are added. Raised in review 5125955392 on pull request #56.)* diff --git a/COMPLETED-PLANS.md b/COMPLETED-PLANS.md index b270db53..7697e58e 100644 --- a/COMPLETED-PLANS.md +++ b/COMPLETED-PLANS.md @@ -8,6 +8,7 @@ and in [crates/windows-threadpool-sys/COMPLETED-CHECKLIST.md](crates/windows-thr | Path to CHECKLIST.md | Completion Date | Brief description | Design Notes | |---|---|---|---| +| [CHECKLIST-topology-provenance.md, archived](COMPLETED-CHECKLIST.md) | 2026-08-31 | Topology content carries its own provenance. `Topology` is documented as constructible by hand and by deserializing a description written for "a machine you do not have", and nothing distinguished either from `discover()`. `Provenance` is `Synthetic` by `Default`, so forgetting is safe and claiming is deliberate; only `discover` yields `Measured`; deserialization can only downgrade, so a file cannot assert it is this machine. The marker renders *inside* the canonical fingerprint string, because a marker beside it would let a fabricated host compare equal to a real one. A pure `places_from_topology` seam was added while `measure()` was deliberately left without one -- a seam that only moves data is safe, one that lets fabricated labels reach real hardware is not -- which closed an unverifiable NUMA mapping: hardcoding the node to 0 passed the entire suite beforehand and fails three tests now. | [crates/windows-topology-sys/DESIGN-NOTES.md](crates/windows-topology-sys/DESIGN-NOTES.md) `D-12`, [crates/windows-platform-probes/DESIGN-NOTES.md](crates/windows-platform-probes/DESIGN-NOTES.md) | | [CHECKLIST.md](CHECKLIST.md) | 2026-08-17 | Workspace metadata, release automation, name reservation, shared cross-crate invariants, generation-stamped operation identities so a retained `OperationId` cannot alias a recycled operation, and six rounds of review hardening: typed wait provenance, teardown-gated re-arming, borrow-checked callback environments, reusable cleanup groups, `stop_and_drain`, borrow-checked exclusivity for the blocking backend, a documented wait-overlap contract, and rejection of values the Win32 fields cannot honour across every adapter. | [DESIGN-NOTES.md](DESIGN-NOTES.md) | | [crates/windows-overlapped-io-sys/CHECKLIST.md](crates/windows-overlapped-io-sys/CHECKLIST.md) | 2026-08-17 | Overlapped-I/O foundation complete: endpoints/provenance, operation storage, raw IOCP and blocking backends, cancellation/rundown, submission seam, safe per-family adapters for file read/write plus scatter/gather (`fs`) and sockets on both backends (`socket`), and a buffer-owning but `unsafe` raw-control-code `DeviceIoControl` seam (`device`). | [crates/windows-overlapped-io-sys/DESIGN-NOTES.md](crates/windows-overlapped-io-sys/DESIGN-NOTES.md) | | [crates/windows-threadpool-sys/CHECKLIST.md](crates/windows-threadpool-sys/CHECKLIST.md) | 2026-08-17 | Thread pool complete: callback environment, private pools, cleanup groups, work, one-shot and periodic timers as distinct types, waits that own a handle of proven provenance, and the `TP_IO` backend over the shared seam, with examples, documentation, and an opt-in timer stress suite. | [DESIGN-NOTES.md](DESIGN-NOTES.md) | diff --git a/Cargo.lock b/Cargo.lock index 55c81d5c..fd7c3a0d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -216,9 +216,12 @@ dependencies = [ name = "windows-platform-probes" version = "0.0.0" dependencies = [ + "windows-namespace-request-sys", "windows-placement-probe", "windows-sys", "windows-threadpool-sys", + "windows-topology-sys", + "windows-waitable-queues", "wtf-string", ] diff --git a/DESIGN-NOTES.md b/DESIGN-NOTES.md index 2418f1aa..e93f677a 100644 --- a/DESIGN-NOTES.md +++ b/DESIGN-NOTES.md @@ -40,6 +40,141 @@ wants to avoid contributing to it. The Windows threadpool types are inherently m choices up to the developer. The `windows-sys` crate published by Microsoft helps with the basics of the FFI to the APIs, but does little to help turn the alphabet and phrasebook into a useful programming model. +## The value is existence, not cleverness: "it is only a SMOP" is why it is missing, not a reason to skip it + +A governing principle for the whole repository, stated because it decides +questions that otherwise get decided by an instinct to keep scope small. + +Nothing here is magic. Every layer in this repository is a Simple Matter Of +Programming -- attribute lists, guard pages, ring buffers, affinity masks, all +of it documented and none of it clever. **The reason these layers are worth +building is precisely that they do not exist**, and their absence is why +capable people muddle along with the most reachable tools: `fprintf`, +`CreateFileW` with default parameters, a thread created with no attributes at +all. Not because those are believed to be right, but because the correct +alternative was never within reach. + +Three rules follow, in decreasing order of how often they are needed: + +1. **"It is only a SMOP" is never an argument against building something.** It + is the explanation for why it is still missing. The observation that a thing + is straightforward is evidence *for* providing it, since straightforward work + nobody has done is exactly the gap a platform layer fills. Difficulty is not + what makes a layer valuable; availability is. + +2. **The measure of success is whether the correct path is easier to reach than + the obvious wrong one.** `fprintf` and a default `CreateFileW` win by being + reachable. A faster, safer, more correct facility that is harder to reach + than the wrong thing has failed, however good it is, because it will not be + reached. Ergonomics is not polish applied at the end; it is the feature. + +3. **When the correct construction is difficult, providing the constructor is + the feature.** If getting a thing right requires a two-pass sizing call, an + opaque buffer with lifetime rules, and three attributes that must be set + before creation rather than after, then assembling that correctly *is* the + deliverable. Declining it on the grounds that each step is simple leaves the + consumer exactly where they started. + +The failure mode this rule exists to prevent is an assistant or an engineer +optimizing to protect *the library* from scope, when the whole purpose is to +absorb difficulty on the *consumer's* behalf. The question is never "is this +small enough to be worth our while"; it is "is the correct thing currently +within a consumer's reach, and if not, what would put it there". + +**This principle schedules no work of its own.** It is a decision rule for +weighing future proposals, not a change to existing code, so the absence of a +checklist item for it is intentional rather than an oversight. + +## `windows-waitable-queues` is plural, and carries no `-sys` suffix + +Two naming decisions for the queue crate that +[CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M30 introduces, recorded before anything +depends on the name, because renaming a crate that has dependents is churn. + +**No `-sys` suffix.** Every `windows-*-sys` crate in this workspace is thin-over-Win32: it makes +an existing API memory-safe without adding policy. This crate is a data structure with an +opinion -- it chooses a slot protocol, an overflow policy, and a signalling discipline that Win32 +has no equivalent of. Calling it `-sys` would misdescribe the layer, and the suffix is the only +signal a reader has for how much the crate decides on their behalf. + +**Plural, because it is a collection of peers rather than one facility.** The workspace already +draws this distinction: `windows-topology-sys` is singular because it provides *the* topology, +while [windows-platform-probes](crates/windows-platform-probes/README.md) is plural because it +provides many probes with none of them canonical. The queue crate is the second kind. SPSC and +MPSC are siblings, and the shapes deferred to `M-inf` -- intrusive-linked and sharded -- are +siblings too. No single queue is the queue. + +**The consequence is accepted deliberately: there is no bare `Queue` type.** A crate named +"queues" that exported one would be claiming a primacy the name denies, so a consumer must say +which shape it wants: the shipped surface is a module per shape -- `spsc`, `slotwise_mpsc`, +`reserving_mpsc` -- each exporting its own `Producer` and `Consumer` handles. That stops a +default from accreting by accident, which is the failure the plural is chosen to prevent. + +**The single `WaitableQueue` trait anticipated here was rejected when the traits were built.** +Superseded by +[D-2](crates/windows-waitable-queues/DESIGN-NOTES.md#d-2) in the crate's own notes, and by +[traits.rs](crates/windows-waitable-queues/src/traits.rs), which states the same thing where a +reader of the code will meet it. + +The reasoning below was right about *why* a trait is not the same case as a bare type, and wrong +about the shape it would take. What forced the change is that a fat trait is not merely +inelegant but **unimplementable** by shapes this crate intends to ship: a queue that is never +waited on has no doorbell to return, and an unbounded one has no capacity to report. So the +capability is sliced the way `std::io` slices it -- `Read`, `Write`, `Seek`, rather than one +`Io` -- and what ships is `Bounded`, `Consumer`, `Drain`, `Observable`, `Producer`, `Reserving` +and `Waitable`, with each shape implementing the subset it genuinely has. + +**What survives unchanged is the constraint that motivated recording this early.** If one shape +ships `pop(&mut self) -> Option` and another ships `try_pop(&self) -> Result`, no +trait unifies them afterwards without a breaking change to one of them. Signatures must +therefore be trait-compatible from the first type -- which is exactly as binding for a set of +narrow traits as it would have been for one wide one. + +**So every shape is split into producer and consumer handles, and cardinality is expressed by +`Clone`.** This is the hard part, because the conventions differ by shape: an SPSC queue is +usually a split `Producer`/`Consumer` pair, while a shared MPMC queue is usually one `Arc` +with `&self` on both ends, and no trait spans those two structures. Making every shape +split-handle resolves it, and buys something better than uniformity: + +| Shape | Producer | Consumer | +|---|---|---| +| SPSC | not `Clone` | not `Clone` | +| MPSC | `Clone` | not `Clone` | +| MPMC | `Clone` | `Clone` | + +Cardinality becomes a **compile-time guarantee rather than a documented precondition**: an SPSC +producer that cannot be cloned cannot become a second producer. The alternative -- a shared +`&self` queue carrying an unchecked "only one consumer" contract -- is precisely the kind of +rule-you-must-remember that +[`RingScope`](crates/windows-ioring-sys/DESIGN-NOTES.md#d-43) and `get(&mut self)` were +introduced to eliminate elsewhere in this workspace. + +**The doorbell belongs on the consumer side**, since the consumer is what waits and the producer +merely rings. Whether that made the contract one consumer-side trait or a producer/consumer pair +was deferred to the crate's own design notes rather than guessed here, and has since been +answered: neither. The contract is sliced by *capability*, so waiting is its own trait -- +[`Waitable`](crates/windows-waitable-queues/src/traits.rs) -- which a shape implements only if it +has a doorbell at all. See +[D-2](crates/windows-waitable-queues/DESIGN-NOTES.md#d-2). + +**What unifies the family is waitability, not I/O.** An earlier candidate, `windows-io-queue`, +was rejected on this point: the queues themselves have nothing to do with I/O, and the domain +runtime is merely their first consumer. Naming a general facility after its first client is the +mistake `windows-topology-sys` avoided when it declined to become a partitioning policy. What +every queue here shares is that it can be waited on **alongside other kernel objects**, which is +precisely what `crossbeam-channel` structurally cannot offer: its `Select` accepts only channel +operations, so `WaitForMultipleObjects` cannot see a crossbeam channel and `Select` cannot see an +`IoRing` completion event. That gap is the reason to build rather than depend, so it belongs in +the name. + +"Waitable" is chosen over a coined word because this workspace already owns it with a precise +meaning -- `windows-threadpool-sys` exposes `WaitableHandle`, governed by the two decisions +[below](#a-safe-wait-constructor-takes-proven-wait-provenance-not-any-handle) on wait provenance +and close routines. The new crate inherits an established concept rather than inventing one. + +**"Ring" was considered and is wrong for the family.** It is accurate for the array shapes and +false for the intrusive-linked one, which is genuinely not a ring. `queues` covers both. + ## Windows SDK model and constraints This crate targets the object-based thread pool API (introduced in Windows Vista) rather than the legacy @@ -1159,6 +1294,41 @@ written into the *audit table* while the decision row it contradicted still said finding is not discharged by being recorded in the audit. It is discharged when the statement it contradicted has changed. +## Sabotage verification has an instrument, and three rules that are not obvious + +Sabotage verification is already required here in several places -- of a derived-fact binding under +[restatement drift](#restatement-drift), of the baseline-consistency checker, of the queue crate's +lost-wakeup guard. What was missing was a way to run one that did not have to be reinvented, badly, +each time. [tools/run-sabotage.ps1](tools/run-sabotage.ps1) is that instrument, documented in +[README-sabotage.md](tools/README-sabotage.md), driven by a `sabotage.json` kept beside the code it +patches. + +The principle is stated elsewhere and is not restated here. What belongs here is the three operational +rules the script encodes, because each was arrived at by getting it wrong and none is guessable: + +- **Judge by exit code, never by reading output.** A test process that dies of heap corruption prints + no `test result: FAILED` line, so a harness that greps for one reports a hole in the tests where + there is none. This cost a real hunt for a nonexistent defect. +- **A timeout counts as caught.** A missing wakeup does not fail a test, it hangs it. An unbounded + harness hangs with it, and a bounded one that treats a timeout as inconclusive throws away the + detection it just achieved. +- **The baseline must be green before anything is patched.** Against an already-red suite every + sabotage "fails" and the sweep proves nothing while reading as a clean bill of health. + +Two further points that are easy to skip and expensive to skip: + +**A survived sabotage is not automatically a hole in the tests.** It may be a defect in the sabotage. +A patch that inserts unreachable code beside a live call, instead of deleting the call, changes the +file without changing the behaviour, and the suite then passes for the honest reason that nothing was +broken. That happened here and was misread as a test hole before anyone read the patch, which is why +the script now prints the patch for every unexpected result. + +**A manifest without a control is only half an instrument.** A control is a change that is *not* a +defect -- typically removing an optimisation -- and it must leave the suite green. Without one, a +sweep can tell you the tests are sensitive but not that they are sensitive to the right things; a +control reported as caught means a test has begun asserting the implementation rather than the +contract, and that test is the thing to fix. + ## Remoting synchronous namespace operations: the measured platform A planned facility makes synchronous-only Win32 operations available diff --git a/PLANS.md b/PLANS.md index 3d9617ba..7c6e2fd4 100644 --- a/PLANS.md +++ b/PLANS.md @@ -10,17 +10,22 @@ plans tracker: [crates/windows-file-enumeration-sys/PLANS.md](crates/windows-fil [crates/windows-platform-probes/PLANS.md](crates/windows-platform-probes/PLANS.md), [crates/windows-thread-ambient-sys/PLANS.md](crates/windows-thread-ambient-sys/PLANS.md), [crates/windows-threadpool-sys/PLANS.md](crates/windows-threadpool-sys/PLANS.md), -[crates/windows-topology-sys/PLANS.md](crates/windows-topology-sys/PLANS.md), and +[crates/windows-topology-sys/PLANS.md](crates/windows-topology-sys/PLANS.md), +[crates/windows-waitable-queues/PLANS.md](crates/windows-waitable-queues/PLANS.md), and [crates/wtf-string/PLANS.md](crates/wtf-string/PLANS.md). Checklists whose work is finished move to [COMPLETED-PLANS.md](COMPLETED-PLANS.md). | Path to CHECKLIST.md | Status | Brief description | Design Notes | |---|---|---|---| -| [CHECKLIST.md](CHECKLIST.md) | not started | M19: propagate the 2026-08-27 platform measurements (IoRing registration replaces the table; the completion-port/`IoRing` fork; `runs_long` as the growth mechanism; the measured 512 default maximum) into the crates whose code or documentation currently assumes otherwise. M20: decide the session-independent path form, now that path resolution is measured to follow the impersonated token's logon session. M21: reconcile with the impersonation and enumeration crates that landed during the session. | [DESIGN-NOTES.md](DESIGN-NOTES.md#remoting-synchronous-namespace-operations) | +| [CHECKLIST-mutation-survivors.md](CHECKLIST-mutation-survivors.md) | not started | Work queued from the workspace-wide cargo-mutants sweep of 2026-09-02, whose findings are kept in [mutation-sweeps/2026-09-02/](mutation-sweeps/2026-09-02/README.md) rather than re-derived -- the run took roughly fourteen hours. 2,792 caught, 1,112 survived, 198 timed out. **The headline numbers mislead in three ways and the README says how**: a timeout in a blocking-API crate is usually a detection that lost its name rather than a gap (measured: one of `windows-waitable-queues`' 120 timeouts fails four tests in 0.00s when re-injected alone), a low score on an executable probe crate is measuring the wrong thing, and three kinds of survivor -- equivalent mutants, unreachable code, and constants that want a `const` assertion -- are not missing tests at all. M1 covers the shipping crates; M2 holds the two crates that are not libraries and whose scope is an engineer's decision; M3 re-runs and prunes rather than hand-editing the tool's output into a second source of truth. | [mutation-sweeps/2026-09-02/README.md](mutation-sweeps/2026-09-02/README.md) | +| [crates/topology-planner/CHECKLIST.md](crates/topology-planner/CHECKLIST.md) | in progress | **Planned, not built** -- the directory holds a plan and no code, and becomes a crate when M2 begins. Owns the mapping from a stated **goal** plus an abstracted idealized machine description to a set of execution domains: which processors host a domain, where each thread pins, which memory node it allocates from, what channel connects each pair, and where each channel's buffer lives. Filed because that mapping was **unowned**: [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) M32 lists the contracts "the runtime cannot be written without" and all of them concern the queue, while M33+.1 opens with "one pinned thread, its `IoRing`, its node-local registered pool, its shard" -- presupposing a plan nothing computed. Separate from `windows-topology-sys` because that crate states **facts** and this one applies **policy**; fusing them is what produced `outermost_partitioning_cache`, a policy answer sitting in the facts crate that three consumers then re-derived differently (SH-16.9). M1 was a *requirements* milestone -- it states what the topology must answer, and it fed the locality-model session, which has since concluded as `D-13`..`D-21`. **The component is deferred past PR #56 by direction**: it contributes only planning documents there, and per `D-21` the topology reshape lands without it, since `windows-topology-sys` publishes a refined view of what the platform publishes and an adapter absorbs the rest. M2+ and M3+ are parked on that session concluding, and are additionally **awaiting a re-cut**: EP-D-4 and EP-D-5 re-scoped the component into four parts (`topology-model` holding the abstract machine description, the planner's traits and the plan type; `topology-planner`; an inward Windows adapter; an outward realizer), and only M1 has been reconciled with that. EP-1.1 is done and already earned its keep: checking the shard-set query against the model found `Processor::capacity` using `0` as both a valid efficiency class and a "not known" sentinel, which collide on every non-hybrid machine (filed as SH-16.12). | [crates/topology-planner/DESIGN-NOTES.md](crates/topology-planner/DESIGN-NOTES.md), [design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md](design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md) | +| [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md) | in progress | M30-M32: a published queue crate (SPSC and bounded-array MPSC, with a lazily created manual-reset doorbell whose reset cannot be separated from the observation that there is nothing to take -- achieved by ordering plus a re-check rather than by a lock, per D-9 and D-15), then the three contract decisions -- ordering, correlation, backpressure -- the domain runtime cannot be written without. M33+ parks the runtime itself, the creation-time-affinity thread builder, the namespace `Outcome` extension, the client-side `ThreadpoolWait` fan-in helper, and the durability crate. M-inf holds three items each gated on a specific measurement rather than on taste. The N=1 path is the whole first deliverable and depends on no NUMA hardware. | [DESIGN-NOTES.md](DESIGN-NOTES.md), [DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md](design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md) | +| [CHECKLIST.md](CHECKLIST.md) | in progress | M19: propagate the 2026-08-27 platform measurements (IoRing registration replaces the table; the completion-port/`IoRing` fork; `runs_long` as the growth mechanism; the measured 512 default maximum) into the crates whose code or documentation currently assumes otherwise. M20: decide the session-independent path form, now that path resolution is measured to follow the impersonated token's logon session. M21: reconcile with the impersonation and enumeration crates that landed during the session. M34 carries the review-driven repairs raised while shipping the placement tool: M34.1 (the reusable sabotage harness) is done, and M34.2 (route the placement tool's output through a sink rather than writing to stdout from many sites) and M34.3 (archive twelve completed item bodies) are open. | [DESIGN-NOTES.md](DESIGN-NOTES.md#remoting-synchronous-namespace-operations) | +| [CHECKLIST-ship-topology-and-queues.md](CHECKLIST-ship-topology-and-queues.md) | in progress | Release `windows-topology-sys` 0.2.0 and `windows-waitable-queues` 0.1.0, which everything shorter-term depends on. Deliberately redundant with [CHECKLIST-io-domains.md](CHECKLIST-io-domains.md): that file plans the design, this one plans the release, and a release has failure modes a design checklist does not surface. Two were found while writing it -- `windows-waitable-queues-v*` is missing from the publish workflow's tag list, so release-please would tag it and nothing would publish it, silently; and `windows-ioring-sys` is published against `windows-topology-sys = "0.1.0"`. (That second finding was later **corrected at SH-2.2**: the pin is a *dev*-dependency, which consumers never resolve, so it obliges a pin update but no release.) M1 settled the public surface before it was public (done, archived); M2 repairs the plumbing; M3 lands the branch; M4 releases; M5 verifies from outside the workspace; M6 is long-running validation and gates the queue crate's release specifically. M7-M13 were seven PR #56 review rounds (done, archived); M14, M15 and M16 are the three later rounds and carry the file's open work -- M15 owns the fix for an ABA hole that ships **disclosed rather than fixed**, so it does not block the release. M16 is the SH-3.1.1 diff review, the first to read the branch as a diff rather than react to a comment: seven findings, six fixed, including a publish-workflow regression this branch had introduced two commits earlier and a soundness hole in the crate about to freeze its API. Its remaining four are blocked on [design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md](design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md), which began by asking whether collapsing a seven-kind, any-depth topology onto a single cache boundary is the right projection and has since settled that presence and observation must be modeled rather than collapsed into an `Option`. **That work now gates the merge**: unlike M14 and M15, which concern a defect in an implementation that can ship disclosed, M16 concerns the shape of the public model `windows-topology-sys` 0.2.0 would publish, and a published model cannot be reshaped without another break. So M3 waits on M16, and M16 waits on the session. The file opens with a status table. | [crates/windows-topology-sys/DESIGN-NOTES.md](crates/windows-topology-sys/DESIGN-NOTES.md), [crates/windows-waitable-queues/DESIGN-NOTES.md](crates/windows-waitable-queues/DESIGN-NOTES.md) | +| [CHECKLIST-placement-tool.md](CHECKLIST-placement-tool.md) | in progress | A small publishable Windows tool a stranger can install, run once, and send back one structured result, so this workspace can collect placement and NUMA-hop measurements from hardware it does not own. The gap is concrete: every host available here has one NUMA node, so the whole `cross NUMA node` row and the inter-node hop matrix are unmeasured and no local work will change that. Extracted rather than published in place, because [crates/windows-platform-probes](crates/windows-platform-probes) is deliberately "an experiment, not a component". **Only the crates.io publication is gated on releasing `windows-topology-sys` and `windows-waitable-queues`** -- an earlier version of this row gated the whole tool on them, which was wrong: the canonical artifact is a CI-built binary compiled from this repository through `path` dependencies, and it needs nothing published. M1 settles the name, what the record says about the machine beyond the fingerprint, and the fate of the existing probe binaries; M1B keys a processor by `(group, number)` so a machine with more than 64 of them is not silently miscounted; M1C measures each NUMA hop in both directions and places the ring deliberately; M2 moves the measurement modules; M3 defines a schema-versioned submission record; M4 covers the runner's experience and trust; M5 distributes the binary; M5+ publishes to crates.io once the download path is established. | [crates/windows-placement-probe/DESIGN-NOTES.md](crates/windows-placement-probe/DESIGN-NOTES.md), [crates/windows-platform-probes/DESIGN-NOTES.md](crates/windows-platform-probes/DESIGN-NOTES.md) | | [CHECKLIST-thread-ambient.md](CHECKLIST-thread-ambient.md) | in progress | M22-M23: extract the captured-context composite into `windows-thread-ambient-sys`, a standalone platform layer that captures a thread's ambient state and applies it on another thread. M24-M26: `windows-namespace-request-sys`, marshalable Win32 namespace call parameter sets, over a round-one entry list audited from three real consumers (this repository's watcher and enumeration crates, and `MikeGrier/Globazog-rs`) rather than guessed. M27: `windows-platform-probes`, a durable home for the measurements this workspace's designs rest on, under a three-tier scheme (asserted / ignored / binary-only) where every tier is compiled by an ordinary build. Feature-scoped and deleted when complete; it is the whole of the `mikegrier/thread-ambient` branch's work, and is deliberately separate from the deferred namespace-facility items in [CHECKLIST.md](CHECKLIST.md). | [crates/windows-thread-ambient-sys/DESIGN-NOTES.md](crates/windows-thread-ambient-sys/DESIGN-NOTES.md) | | [crates/windows-overlapped-io-sys/CHECKLIST.md](crates/windows-overlapped-io-sys/CHECKLIST.md) | not started | M14: finish the contract audit -- categories 1, 2, 6, 8, 9 were not examined -- and sweep `outstanding()` for the advisory-predicate hazard. | [crates/windows-overlapped-io-sys/DESIGN-NOTES.md](crates/windows-overlapped-io-sys/DESIGN-NOTES.md) | -| [crates/windows-ioring-sys/CHECKLIST.md](crates/windows-ioring-sys/CHECKLIST.md) | in progress | Memory-safe Rust over the Windows `IoRing` submission/completion ring, as a new crate. M1-M7 (ring lifecycle through the `ring-copy` topology-aligned sample) are complete and archived. The parked, pinned-thread `M6+` work and the new M10 contract audit remain. | [crates/windows-ioring-sys/DESIGN-NOTES.md](crates/windows-ioring-sys/DESIGN-NOTES.md) | +| [crates/windows-ioring-sys/CHECKLIST.md](crates/windows-ioring-sys/CHECKLIST.md) | in progress | Memory-safe Rust over the Windows `IoRing` submission/completion ring, as a new crate. M1-M19 are complete (0.2.0 shipped 2026-08-30, restoring availability after all three 0.1.x versions were yanked); M1-M18 are archived. **M20** queues documentation and policy-test repairs from the 2026-08-30 NUMA-sharding measurement, and the pinned-thread `M6+` work stays parked. | [crates/windows-ioring-sys/DESIGN-NOTES.md](crates/windows-ioring-sys/DESIGN-NOTES.md) | | [crates/windows-platform-probes/CHECKLIST.md](crates/windows-platform-probes/CHECKLIST.md) | not started | M1: stream a probe's report as it is measured. The one-sink refactor has each renderer compose its report into a `String`, which buys the seam that lets a probe's findings be asserted rather than eyeballed, and gives up output appearing as it is measured. `emit_report` recovers that for an unwinding panic only; Ctrl-C and an abort during unwinding still discard the buffer. Costs most on `probe-cancel-io`, which runs about twenty seconds precisely when the wedge it hunts for occurs -- precisely when a reader interrupts. | [crates/windows-platform-probes/DESIGN-NOTES.md](crates/windows-platform-probes/DESIGN-NOTES.md#d-buffered-report) | -| [CHECKLIST-mutation-survivors.md](CHECKLIST-mutation-survivors.md) | not started | Work queued from the workspace-wide cargo-mutants sweep of 2026-09-02, whose findings are kept in [mutation-sweeps/2026-09-02/](mutation-sweeps/2026-09-02/README.md) rather than re-derived -- the run took roughly fourteen hours. 2,792 caught, 1,112 survived, 198 timed out. **The headline numbers mislead in three ways and the README says how**: a timeout in a blocking-API crate is usually a detection that lost its name rather than a gap (measured: one of `windows-waitable-queues`' 120 timeouts fails four tests in 0.00s when re-injected alone), a low score on an executable probe crate is measuring the wrong thing, and three kinds of survivor -- equivalent mutants, unreachable code, and constants that want a `const` assertion -- are not missing tests at all. M1 covers the shipping crates; M2 holds the two crates that are not libraries and whose scope is an engineer's decision; M3 re-runs and prunes rather than hand-editing the tool's output into a second source of truth. | [mutation-sweeps/2026-09-02/README.md](mutation-sweeps/2026-09-02/README.md) | Add a row here when new work is planned, against [CHECKLIST.md](CHECKLIST.md) or any crate's. diff --git a/crates/topology-planner/CHECKLIST.md b/crates/topology-planner/CHECKLIST.md new file mode 100644 index 00000000..3273e095 --- /dev/null +++ b/crates/topology-planner/CHECKLIST.md @@ -0,0 +1,221 @@ +# Checklist: the topology planner + +Plans an arrangement of execution domains from a stated **goal** plus an **abstracted idealized** +description of a machine. See [COMPONENT.md](COMPONENT.md) for what this crate is and why it is +separate from both the topology crate and the runtime, and +[EP-D-4](DESIGN-NOTES.md#ep-d-4) for the architecture it now sits in. + +**The component has been re-scoped**, per [EP-D-4](DESIGN-NOTES.md#ep-d-4) and +[EP-D-5](DESIGN-NOTES.md#ep-d-5). It is named `topology-planner` and the directory now matches; it +queries an abstract model covering processors, memory, storage, interconnects, distances and +bottlenecks rather than `MachineMemoryTopology` directly; and **adapters** bracket it -- one exposing +the model's traits over the Windows topology objects, one realizing a plan as buffers, rings and +threads. The model, its traits, and the plan type live in a separate `topology-model` crate that +everything depends on and that depends on nothing. +**M2+ onward are written against the superseded shape and are not yet re-cut.** + +## Where this stands + +**Nothing is implemented.** M1 is the only active milestone, and it is deliberately a +*requirements* milestone rather than an implementation one: its output is the concrete statement +of what the model must answer, which the open design session needs in order to settle it. + +**Deferred past PR #56, by direction.** This component contributes only planning documents to that +PR and no code. `windows-topology-sys`'s reshape lands there without it: per +[D-21](../windows-topology-sys/DESIGN-NOTES.md#d-21) that crate publishes a refined view of what the +platform publishes, and an **adapter** absorbs whatever this component needs beyond it, so the two +are no longer coupled. + +The design session that gated M2 onward **has concluded** -- its questions were answered as `D-13` +through `D-21`, and the `MMT-*` plan is what it produced. M2+ is now gated on this component's own +prerequisites rather than on someone else's decision. + +| Milestone | State | What it is waiting on | +|---|---|---| +| M1 the input contract | 4 done, 1 open | `EP-1.5`'s coverage half, which wants a settled model | +| M1+ scenario and naming | **partly answered** | the name is settled (EP-D-4); the goal input is deferred for litigation, by direction | +| M2+ the plan as a value | parked, **and needs re-cutting** | re-cut against EP-D-4/EP-D-5, then the topology reshape landing | +| M3+ the policies | parked | M2+ | +| M-inf parked | ungated | not scheduled, deliberately | + +## M1: state what the planner needs from the topology + +The point of doing this first: the design session asks what representation is most useful to +consumers, and **this crate is the consumer**. Answering in the abstract has already produced one +wrong answer this session. Each item below states a query the planner makes, why it makes it, and +whether the topology can answer it today -- so the model is designed against a real caller. + +- [x] **EP-1.1** -- **The shard-set query.** Which processors may host a domain: online, with + identity carried as `(group, number)` rather than a bare number, with efficiency class and SMT + structure available so a policy can choose one domain per core or per thread and can decide + whether efficiency cores are peers. **Gap already identified:** parked and allocated state is not + available at all, and pinning a domain to a parked processor is a defect a client cannot detect. + Tracked as `SH-16.10`. + **Done:** stated as [EP-D-1](DESIGN-NOTES.md#ep-d-1), with each of its five inputs checked against + the model rather than assumed. Three are answered cleanly; availability is not answered at all; + and the fourth turned up a defect the item had not anticipated. + **`Processor::capacity` is unsafe for reading efficiency class.** It is + `online.then(find owning Core).flatten().unwrap_or(0)`, so `0` means offline, *or* in no core + domain, *or* genuinely class zero -- and the third is every processor on every non-hybrid machine, + so the sentinel collides with the common legitimate value. Worse here than elsewhere, because + Windows orders class `0` as *least* performant: on a hybrid part an unknown processor is + indistinguishable from an efficiency core, so a policy excluding them silently drops a possible + performance core and a policy tiering them mis-tiers it. Neither fails a functional test. Filed + against the owning crate as `SH-16.12`; use `DomainKind::Core { efficiency_class }` meanwhile. + +- [x] **EP-1.2** -- **The proximity query, which is the crux.** For an ~~*ordered pair*~~ + **unordered pair** of processors, how close are they -- because that is what chooses SPSC versus + MPSC versus a routed hop, and it is asked once per pair rather than once per machine. **The + current model cannot answer it**: `outermost_partitioning_cache` reports one global level and + `same_cache_domain` reduces it to a boolean at that level, so a client reconstructs the rest and, + per `SH-16.9`, reconstructs it differently each time. State the query precisely enough that the + session can design against it. + **Done:** stated as [EP-D-2](DESIGN-NOTES.md#ep-d-2). + **This item said "ordered pair" and was wrong**, corrected in place rather than quietly. The + repository had already settled it: `windows-placement-probe` documents that its placement labels + are "deliberately symmetric", that "the *relationship* between two processors genuinely is + symmetric", and that "direction therefore lives where it is real, not in the label" -- with the + measured side putting it as "a hop is not symmetric even though the link is". Proximity is the + link and is unordered; direction is the hop, and belongs to EP-1.3's residency question. + Three requirements came out of stating it. The answer needs the **membership** of the shared + granularity, not just its identity, or the planner re-derives the grouping to size an MPSC + fan-in. It needs to distinguish "tightest shared is X" from "**at most** X, and finer was not + observed", since under the model's bar the planner cannot go and check. And the order being by + inclusion rather than by firmware numbering means two granularities can be **incomparable**, so + the answer is a set of minimal shared granularities -- almost always one, but not by construction. + +- [x] **EP-1.3** -- **The residency query.** Which memory domain each processor belongs to, and -- + for a pair spanning two of them -- what it costs to place a shared buffer on one side rather than + the other. **Gap already identified:** `MachineMemoryTopology::distances` exists, is never populated, and Win32 + cannot populate it; the measurement exists in `windows-placement-probe` and reaches nothing. + Tracked as `SH-16.11`. The probe measures this per node pair with a dedicated ring-placement + column precisely because it was found to matter. + **Done:** stated as [EP-D-3](DESIGN-NOTES.md#ep-d-3). This is where the direction EP-1.2 refused + lands -- proximity is the link and symmetric, residency is the hop and is not. + The processor-to-node half is answered, with one asymmetry worth preserving: an unknown *cache* + domain costs an optimisation, but an unknown *memory* domain has no honest fallback, since the + pool must be allocated somewhere and guessing means quietly allocating remote memory for the life + of the process. `windows-placement-probe` already refuses on the second while tolerating the + first, and that judgement was correct. + **The cost half needs SH-16.11 restated, and it was.** That item read as though someone had + forgotten to populate a field. Two sharper problems replace it: `distances` can never carry + `Measured` provenance **by construction** -- its only inputs are a literal (`Synthetic`) and a file + (capped at `Restored`) -- so populating it would not help; and even populated it is SLIT-shaped, + one symmetric workload-independent scalar, while the question is directional. `D-9` in the + topology crate already deferred the attributed edge list that would answer it, naming *asymmetry* + among what it would absorb, with the trigger being that a scalar "demonstrably mismodels a machine + somebody is tuning for" -- and this planner is that machine-tuner. + **The trigger is approached, not met**, and the gap is a measurement nobody here can take: both + development hosts are single-node, so every directional run prints "VACUOUS ON THIS MACHINE". + Recorded so D-9 is reopened on evidence rather than on argument. + +- [ ] **EP-1.4** -- **What the planner does with an unanswered query**, given the model's bar is + that it answers without further measurement. A fact that was not observed cannot be acquired at + planning time, so decide per query whether the planner degrades to a documented weaker policy, + refuses to plan, or emits a plan carrying an explicit "this was chosen without knowing X" marker. + The third is the only one that survives review of a plan by a human, which is one of the reasons + a plan is a value. + **BLOCKED, and not merely because it is downstream.** EP-1.1 through EP-1.3 push requirements + *into* the model's design, which is why they were worth doing against today's model and found real + defects in it. This item reads behaviour *out* of the model -- it asks what the planner does when + the answer is "not observed", a state the model cannot currently express reliably -- so doing it + now would be analysing a shape that does not exist yet. + **It is also a duplicate.** The design session's fourth open question, "what a consumer does when a + needed fact is `not measured`", is this same decision seen from the model's side; the two were + filed independently before anyone noticed. Taken separately they can disagree: a planner that + degrades in a way the model does not support, or a model offering a fallback no consumer wants. + Answer them together, in the session. + **Narrowed by [D-19](../windows-topology-sys/DESIGN-NOTES.md#d-19).** The item says "decide per + query", and that is now more work than the model requires. A subject the two sources genuinely + contested is one the unified view does not cover, which is indistinguishable from not-observed to a + consumer -- so this is one decision about one degradation path, not one per reason a fact is + missing. The three candidate behaviours are unchanged. + **And it is no longer a duplicate**, per [D-21](../windows-topology-sys/DESIGN-NOTES.md#d-21). + `windows-topology-sys` publishes a refined view of what the platform publishes; what a consumer + *does* with an unobserved fact is not a question about that view. That crate owes only that the + absence be representable and distinguishable, which its `M2+.5` implements. `MMT-1.3` is closed on + those grounds, so **this item is now this component's decision alone** -- there is nothing left to + take jointly, and it no longer blocks anything in the model crate. + +- [ ] **EP-1.5** -- **Hand the resulting requirements to the design session** as the consumer-side + input it asked for, and record in the session which of them the settled model answers and which + it deliberately does not. + **Half done, and split because the halves have different prerequisites.** The *handover* is + complete: the session now carries the three queries in a table, plus the four model properties that + follow from them -- a pairwise query must exist, the order must be total, an answer must be able to + be an upper bound, and a measured number must carry what it measured. That was the part the model + designer needs in front of them, and it did not depend on the model existing. + **One of the four has since been corrected**, and it is recorded here rather than rewritten in the + session, which is an append-only record of what was handed over. "A pairwise query must exist" is + right about the requirement and wrong about the shape: per + [windows-topology-sys](../windows-topology-sys/COMPLETED-CHECKLIST.md) `M4+.1` the ordered collection is the + surface and the pairwise query is derived from it, because an answer obliged to carry the block + containing both processors is a question about the partition rather than about the pair. + The *coverage* half -- recording which requirements the settled model answers and which it + deliberately does not -- can only be written once there is a settled model. It stays open here. + > **-> CROSS-COMPONENT HANDOFF:** next work is in the repository root -> + > [DESIGN-SESSION-2026-09-02-cache-locality-model.md](../../design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md) + > -> `SH-16.8` in + > [CHECKLIST-ship-topology-and-queues.md](../../CHECKLIST-ship-topology-and-queues.md). + +## M1+: the scenario input, and the naming + +Raised when the engineer described this component's function, which turned out to be richer than +"takes a topology, applies policy". Both are gated on the locality-model session, but neither is a +model question -- they are this component's own. + +- [ ] **EP-1+.1** -- **Describe the scenario input.** The synthesizer takes *two* inputs and only one + is described anywhere. The scenario says what the caller intends to run, and it is what makes a + measurement meaningful: [EP-D-3](DESIGN-NOTES.md#ep-d-3) established that a measured number means + nothing without knowing what it measured, so at minimum the scenario must distinguish small-message + handoff from large-buffer streaming. Its absence is why "what is most useful for consumers" was + hard to answer in the abstract for so long. + +- [ ] **EP-1+.2** -- **Decide what the caller-callback traits ask.** Planning is a negotiation: the + component may call back for clarification the scenario did not settle. Enumerating those questions + is what decides whether this is one trait or several, and it cannot be done before EP-1+.1 says + what the scenario already answers. + +- [ ] **EP-1+.3** -- **Settle the naming, before any type is written.** Both inputs and the output + are graphs of processors and their relations, so "topology" fits all of them and distinguishes + none -- and a reader seeing the word twice will eventually take one for the other. Decide whether + the observed machine keeps the bare name (qualified only by its crate), gains a qualifier, or is + renamed outright, and what the synthesized arrangement is called. Cheap now; expensive once either + name is public. This one blocks nothing but should not be settled by whoever writes the first type. + +## M2+: the plan as a value + +Parked, not pending. Gated on the topology model landing. Shape recorded so it is not lost, per the +`M{n}+` convention. + +- [ ] **M2+.1** -- The plan type: domains, each with its processor, its memory domain and its + channels; inspectable and comparable, constructible against a synthetic topology so a machine + nobody has can be planned for and reviewed. + +- [ ] **M2+.2** -- Rendering a plan for a human to read before anything is pinned or allocated, + including which queries were unanswered and what was assumed in their place. + +- [ ] **M2+.3** -- Validation against synthetic topologies drawn from the shapes this repository has + actually met: the ARM64 host with no L3, the x64 host whose outermost partitioning cache is L2 + shared by SMT siblings, a hybrid part with efficiency classes, and a machine with more than 64 + processors so the group boundary is exercised rather than assumed. + +## M3+: the policies + +Parked. These are the choices the crate exists to make, and each is a decision item rather than an +implementation one. + +- [ ] **M3+.1** -- Domain-per-core versus domain-per-thread, and whether efficiency cores are peers, + excluded, or a second tier. + +- [ ] **M3+.2** -- The channel policy: what proximity justifies SPSC, what falls back to MPSC, and + whether any pair is deliberately not connected directly at all. + +- [ ] **M3+.3** -- Buffer residency for a channel spanning two memory domains, which the placement + probe already measures and which has no default that is right on both sides. + +## M-inf: parked, ungated + +- [ ] **M-inf.1** -- Re-planning at runtime, when processors are parked, hot-added, or the process + is given a different CPU-set allocation than it started with. Deliberately not scheduled: it needs + the static case to exist first, and it is a different problem. diff --git a/crates/topology-planner/COMPONENT.md b/crates/topology-planner/COMPONENT.md new file mode 100644 index 00000000..439bac09 --- /dev/null +++ b/crates/topology-planner/COMPONENT.md @@ -0,0 +1,123 @@ +# topology-planner + +**Planned, not built.** This directory currently holds a plan and no code. It becomes a crate +when [CHECKLIST.md](CHECKLIST.md) M2 begins; until then it exists so the work has an owner and a +place, rather than living as an assumption inside somebody else's milestone. + +Named without a `windows-` prefix on purpose: it plans against an abstracted idealized machine and +emits a platform-neutral plan, so nothing in it is Windows-specific. See +[DESIGN-NOTES.md](DESIGN-NOTES.md) -> `EP-D-4` for the architecture, and `EP-D-5` for the layout. + +## What it is + +A **planner**. It takes two inputs and produces a third thing: + +- **a stated goal** -- what the caller intends the arrangement to achieve. Its shape is deliberately + **deferred for litigation**; that is a named deferral, not an omission. +- **an abstracted idealized description of a machine** -- processors, memory, storage, interconnects, + distances and bottlenecks. Not Windows-shaped, and richer than any single platform reports. It is + **mockable by construction**: a description of a machine nobody has is an ordinary input, which is + what makes this component testable without the hardware it plans for. + +From those it produces **a plan**: which processors host domains, where each thread pins, which +memory node each allocates from, what channel connects each pair, and where each channel's buffer +lives. The plan **serializes to JSON** and stays abstracted from Windows. + +**It may ask.** Planning is a negotiation, not a pure function: the component may call back to its +caller through traits for clarifying information the goal did not settle. Which questions those are +is not yet known, and knowing them is what decides whether that is one trait or several. + +## The four components, and which way the arrows point + +| Component | Platform | Depends on | +|---|---|---| +| `topology-model` | neutral | nothing | +| `topology-planner` (this one) | neutral | `topology-model` | +| the inward adapter | Windows | `topology-model`, `windows-topology-sys` | +| the outward adapter (the realizer) | Windows | `topology-model`, the runtime crates | + +`topology-model` holds the abstract machine description, **the traits the planner queries**, and +**the plan type**. Everything depends on it; **nothing depends on this crate**. + +That is the whole point of the arrangement. If the traits lived here, an adapter whose only job is +to describe a machine would have to depend on a planner, and anyone wanting to read a topology would +pull in planning policy they did not ask for. The plan type is here for the same reason one level +down: the realizer *executes* a plan and has no business depending on the policy that chose it. + +## Two kinds of adapter + +**Inward** -- exposes the model's traits over the topology objects already designed, so +`windows_topology_sys::MachineMemoryTopology` becomes one source feeding the abstract model. It is +one source among several: storage and interconnect facts do not come from there, and neither do +measured numbers. + +**Outward (the realizer)** -- takes a plan and **realizes** it in the current process: buffers, +rings and threads, with the user's processing code inserted at the appropriate steps. + +They are separate crates despite both being Windows adapters, because their dependency sets barely +overlap -- the inward one needs only `windows-topology-sys`, while the realizer needs the runtime. +Fusing them would mean anyone reading a topology pulls in the whole runtime. + +## Why the planner is separate from the facts + +Because two different kinds of statement were being made by one crate. + +**`windows-topology-sys` states facts.** Which processors exist, what they share, at what +granularity, how that was established, and what was measured. It never says "use an SPSC ring +here", because that is not a fact about the machine. + +**This crate applies policy.** One domain per core or per thread? Are efficiency cores peers or +excluded? SPSC everywhere, or SPSC within a cache domain and something else across one? Those are +choices, they depend on the workload, and reasonable clients will differ. + +Keeping them in one crate has a specific failure mode, already observed: a policy answer gets +mistaken for a fact and consumers bind to it. `outermost_partitioning_cache` is that -- a single +policy choice ("give me one boundary to shard on") sitting in the facts crate, which three +consumers then re-derived differently. See +[CHECKLIST-ship-topology-and-queues.md](../../CHECKLIST-ship-topology-and-queues.md) SH-16.9. + +## The seam, and how to tell if it is in the right place + +**The planner must not re-derive anything.** If it has to work out for itself which cache level +partitions the machine, or reconstruct a mapping the model already knows, the seam is wrong and the +missing query belongs in `topology-model` -- or, if it is a Windows fact, in the inward adapter. + +That test is the reason this component is being planned *before* the topology model is finished +rather than after: its input requirements are the concrete statement of what the model has to +answer, and they feed the open design session directly. + +## Why it is not the runtime either + +The runtime (M33+, spanning `windows-ioring-sys`, `windows-thread-ambient-sys` and +`windows-namespace-request-sys`) *executes* a plan: it creates the threads, binds them, allocates +the pools, constructs the rings. This crate decides what that plan should be, and the realizer +bridges the two. + +Separating them means a plan is a **value** -- inspectable, comparable, testable against a +synthetic topology for a machine nobody has, and reviewable by a human before anything is pinned +or allocated. A planner fused into the runtime can only be tested by running it on the machine it +plans for, which is exactly the class of test this repository has repeatedly found inadequate. + +The arrangement it plans for is the one +[CHECKLIST-io-domains.md](../../CHECKLIST-io-domains.md) M33+ describes -- "one pinned thread, its +`IoRing`, its node-local registered pool, its shard" -- which is a Seastar-style shard-per-core +runtime. + +## Status and gating + +**Deferred past PR #56, by the engineer's direction.** This component contributes only planning +documents to that PR and no code. The topology reshape it fed requirements into is landing there +without it, because [D-21](../windows-topology-sys/DESIGN-NOTES.md#d-21) establishes that +`windows-topology-sys` publishes a refined view of what the platform publishes and an **adapter** +absorbs whatever this component needs beyond that -- so the reshape is self-justified and the two are +no longer coupled. + +The design session that previously blocked this component has concluded: its questions were answered +as `D-13` through `D-21` in +[windows-topology-sys/DESIGN-NOTES.md](../windows-topology-sys/DESIGN-NOTES.md), and the central +query -- "how close are these two processors?" -- is answered by the ordered relation set that `MMT` +M2 and M4 build. What remains here is this component's own work, not a wait on someone else's. + +The name is settled; the crate does not exist yet. It is deliberately absent from +`release-please-config.json`, the publish workflow's tag patterns, and the workspace manifest until +there is code to publish. diff --git a/crates/topology-planner/DESIGN-NOTES.md b/crates/topology-planner/DESIGN-NOTES.md new file mode 100644 index 00000000..430306c9 --- /dev/null +++ b/crates/topology-planner/DESIGN-NOTES.md @@ -0,0 +1,477 @@ +# Design notes: the topology planner + +Current canonical decisions for this component. See [COMPONENT.md](COMPONENT.md) for what the +component is; see [CHECKLIST.md](CHECKLIST.md) for what is planned. + +`EP-D-1` through `EP-D-3` are **queries** rather than choices: the planner's requirements, stated +precisely enough that the topology model could be designed against a real caller instead of against +a guess. They were written when the planner was to read `windows_topology_sys::MachineMemoryTopology` +directly; [EP-D-4](#ep-d-4) rebinds them to traits over an abstract model, which changes what +satisfies them and not what they require. + +[EP-D-4](#ep-d-4) is the first genuine **choice** here, and it re-scopes the component: the planner +is `topology-planner`, it plans against an abstracted idealized machine, and adapters bracket it on +both sides. [EP-D-5](#ep-d-5) then settles the layout EP-D-4 left open, and the directory has been +renamed to match. + +## Decision index + +| ID | Decision | +|---|---| +| EP-D-1 | **The shard-set query**: what the planner must know to choose which processors host a domain, and what today's model cannot tell it. | +| EP-D-2 | **The proximity query**: how close two processors are, which selects the channel between their domains. Takes an **unordered** pair; the model has no answer today. | +| EP-D-3 | **The residency query**: where a domain's pool lives, and which side of a cross-domain pair should host a shared ring. **Ordered**, and the half the model cannot answer is structurally unanswerable rather than merely unpopulated. | +| EP-D-4 | **The four-part architecture, and the planner's name.** The engineer's position: the planner is **`topology-planner`** (no `windows-` prefix); it takes a **goal** description (shape deferred for litigation), queries an **abstracted idealized** model covering processors, memory, storage, interconnects, distances and bottlenecks, and emits a **JSON-serializable, platform-neutral** plan. Two kinds of **adapter** bracket it: one exposing the planner's traits over the Windows topology objects, one **realizing** a plan as buffers, rings and threads with the user's code inserted at the right steps. Settles `MMT-1.5` (the facts crate keeps its `-sys` name), the "two graphs, one word" ambiguity, and where distance lives -- the attributed interconnect shape D-9 sketched goes in the abstract model, so D-9's deferral in the facts crate stands unreopened. | +| EP-D-5 | **The component layout: `topology-model` is its own crate, and dependencies point one way.** The abstract model and the traits the planner queries live in `topology-model`, which the planner and both adapters depend on; nothing depends on `topology-planner`. Putting the traits in the planner would make a crate whose job is to *describe a machine* depend on one that applies *policy* -- the same defect as `outermost_partitioning_cache`, arriving as a dependency edge instead of an API. Two consequences derived from the same rule rather than decided separately: **the plan type also lives in `topology-model`** (otherwise the realizer depends on the planner), and the inward adapter and the realizer are **separate crates** (their dependency sets barely overlap, and fusing them would make reading a topology pull in the whole runtime). | + +## EP-D-1: the shard-set query + +*Recorded by [CHECKLIST.md](CHECKLIST.md) EP-1.1.* + +### What the planner is choosing + +Which processors may host an execution domain, and how to group them so a policy can pick between +one domain per core and one per logical processor, and can decide whether efficiency cores are +peers, a second tier, or excluded. + +This is the first step of the construction and it fixes the domain count, which everything +downstream is shaped by: the number of rings is quadratic in it, and each domain's memory pool is +sized against it. + +### What it must know, and why + +1. **Identity, as `(group, number)`.** Not a bare index. A processor number without its group names + a different processor in every group and the wrong one in all but the first, and pinning is a + `GROUP_AFFINITY` -- `SetThreadGroupAffinity`, not `SetThreadAffinityMask`, which cannot name + another group at all. A planner that flattens this produces a plan that is silently wrong above + 64 processors. + +2. **Whether the processor is online.** An offline slot exists and counts toward a group's maximum; + planning a domain onto one is planning a thread that cannot run. + +3. **Core membership and whether the core is SMT.** The choice between one domain per core and one + per logical processor is the single largest policy lever, and it needs the sibling grouping, not + just a count. + +4. **Efficiency class.** On a hybrid part, putting latency-sensitive domains on efficiency cores is + a defect the client will not see in a functional test, only in a percentile. + +5. **Whether the processor is available to this process at all** -- parked by the scheduler, or + outside the CPU-set allocation the process was given. + +### What today's model answers + +Points 1 through 3 cleanly. `ProcessorId` is `(group, number)` by construction and documents why +(D-7). `Processor::online` is exactly the distinction in point 2. `DomainKind::Core` carries +`simultaneous_multithreading` and the sibling set, so point 3 is a walk of `MachineMemoryTopology::cores()`. + +Point 4 is answered, but **twice, in two shapes, and one of them is unsafe to use** -- see below. + +Point 5 is **not answered at all**. `GetSystemCpuSetInformation` is consumed nowhere in the +workspace, so `Parked`, `Allocated` and `AllocatedToTargetProcess` are unavailable. A planner +cannot currently avoid pinning a domain to a parked processor, and the client cannot detect that it +happened. Tracked as `SH-16.10` in +[CHECKLIST-ship-topology-and-queues.md](../../CHECKLIST-ship-topology-and-queues.md). + +### `Processor::capacity` must not be used for point 4 + +**Use `DomainKind::Core { efficiency_class, .. }`. Do not use `Processor::capacity`.** + +`capacity` is computed as `online.then(|| find the owning Core domain).flatten().unwrap_or(0)`, so +the value `0` means any of three different things: + +- the processor is offline; +- the processor is online but no `Core` domain names it, which the topology tolerates by design + since firmware coverage is not guaranteed; +- the processor is online, has a core, and its efficiency class genuinely **is** `0`. + +The third is not an edge case. It is **every processor on every non-hybrid machine**, so the +sentinel collides with the overwhelmingly common legitimate value. + +For this planner the collision is worse than for most consumers, because Windows orders efficiency +class with `0` as the *least* performant. On a hybrid part an unknown processor is therefore +indistinguishable from an efficiency core, and a policy that excludes efficiency cores would +silently drop a processor that might be a performance core -- while a policy that tiers them would +place it in the wrong tier. Both failures are invisible in a functional test. + +`Core { efficiency_class }` carries the firmware value with no sentinel, and absence is represented +by the processor being in no `Core` domain, which is a distinguishable state rather than a value. + +**This is the same defect the locality-model session exists to fix, in a third place.** The others: +`ProcessorPlace::cache_domain: Option`, where `None` conflates "no level partitions this +machine" with "this processor was not named at the level that does" (`SH-16.5`); and +`MachineDescription::cpu_model`, where the same conflation was noticed and solved with a side +boolean. Recorded here so the sweep that fixes the model does not stop at the two already known. + +### Partial core coverage is a real state, not a corruption + +A processor in no `Core` domain is a firmware gap, not a contradiction, and the topology crate +tolerates it deliberately. The planner must therefore decide what to do with a processor it cannot +group -- it is a candidate host whose SMT relationships and class are unknown, which is exactly the +"unanswered query" case that [CHECKLIST.md](CHECKLIST.md) EP-1.4 owns. It is named here so that +item is not written as though the case were hypothetical. + +### What this asks of the topology model + +Nothing new in shape; three things in substance. + +- Availability (parked, allocated) has to become expressible, since no policy can be correct + without it. +- Efficiency class has to have exactly one representation, and it must distinguish "class zero" + from "not known". +- Core membership has to admit that a processor may be in no core, without that being an error. + +## EP-D-2: the proximity query + +*Recorded by [CHECKLIST.md](CHECKLIST.md) EP-1.2.* + +### What the planner is choosing + +For two domains, what connects them: a dedicated SPSC ring, a shared MPSC ring fanning several +producers into one consumer, or a routed hop through an intermediate domain. That choice is made +once per pair, and it is made from how close the two processors are. + +This is the query the whole model question turns on. Everything else the planner asks is either +per-processor (EP-D-1) or per-memory-domain (EP-1.3); this is the only one that is *relational*, +and it is the one today's model cannot answer. + +### It takes an unordered pair. The checklist item said ordered, and was wrong. + +`windows-placement-probe` already settled this and stated the reasoning, which is worth quoting +because it is easy to get backwards: + +> These names are deliberately symmetric, and that is not an oversight left over from before hops +> became directed. The *relationship* between two processors genuinely is symmetric -- two +> processors either are SMT siblings or are not, share a cache domain or do not -- so there is no +> honest `CrossNumaNodeForward` to name. Splitting the labels by direction would invent a +> distinction the topology does not have. +> +> The *workload* is what is asymmetric: the producer writes and the consumer reads, so swapping +> them swaps which side pays. Direction therefore lives where it is real, not in the label. + +And on the measured side: "a hop is not symmetric even though the link is." + +So the split is clean, and the planner needs both halves in different places: + +- **Proximity is the link.** Symmetric, unordered pair, answered here. +- **Residency is the hop.** Asymmetric -- which side hosts the ring buffer, which the probe + measures with a dedicated ring-placement column because it was found to matter. That belongs to + EP-1.3, not here. + +Putting direction in the proximity query would invent an asymmetry the topology does not have, and +would double the size of an answer that has no second half to fill. + +### What the answer must contain + +Not a boolean, and not a bare identifier. Three things: + +1. **The tightest granularity the two share.** Comparable against other pairs' answers, because the + policy's threshold ("SPSC within this, MPSC beyond it") is a comparison. The *identity* of the + granularity matters less than its position. + +2. **The membership of that granularity.** Selecting MPSC is not enough -- the planner must size the + fan-in, which is "how many other domains sit at this same proximity". Without membership the + planner would ask the proximity query O(n^2) times and reconstruct the grouping itself, which is + the re-derivation the seam exists to prevent. + +3. **Whether a finer granularity went unobserved.** This is the part that a naive design drops. If + L3 was observed and L2 was not, "tightest shared is L3" is *not* the answer -- the answer is "at + most L3, and finer was not looked at". A planner told the first would choose a slower channel + than the machine can support and never learn why. Under the model's bar -- usable without further + measurement -- it cannot go and check, so the distinction has to be in the answer. + +### The query should be total, which needs a top element + +Two processors in the same machine always share *something*: one address space, one scheduler, one +memory system, however far apart. If the granularity order has no top, the query returns "nothing +in common" for a cross-node pair and every caller writes the same empty-case branch. + +Making "the machine" an explicit top granularity is honest -- it is a real, if loose, locality tier +-- and makes the query total. A bottom ("this processor alone") is the same argument at the other +end and makes `proximity(a, a)` answerable rather than a special case, though a planner has no +reason to ask it. + +### A partial order means the answer may not be a single granularity + +If the order is by observed set inclusion rather than by firmware numbering, two granularities can +be **incomparable** -- neither refines the other. The tightest shared granularity is then not +unique, and the honest answer is the set of *minimal* shared granularities, which is almost always +exactly one. + +This is a cost, and it is worth naming rather than discovering later: every caller either handles a +multi-element answer or documents that it takes the first. But the alternative -- forcing a linear +order -- means silently discarding a real boundary on a machine whose levels do not nest, and this +repository has been bitten specifically by structure that was assumed rather than checked. + +### What today's model answers: nothing + +`MachineMemoryTopology::outermost_partitioning_cache` reports **one level for the whole machine**, and +`Slice::same_cache_domain` reduces that to a boolean at that one level. Neither is pairwise. There +is no query anywhere in `windows-topology-sys` that takes two processors. + +So a planner today reconstructs proximity from the partition list -- which is exactly what +`SH-16.9` records three consumers already doing, in two mutually inconsistent ways. The absence of +this query is the cause of that defect, not a separate problem. + +### What this asks of the model + +- A granularity order derived from **observed set inclusion**, not firmware level numbers, so a + measured-only tier and a machine with no L3 both have positions. +- Access to that order **as a collection**, with a pairwise helper derived from it, returning minimal + shared granularities plus their membership. Stated here first as a pairwise query, which + [windows-topology-sys](../windows-topology-sys/COMPLETED-CHECKLIST.md) `M4+.1` corrected: requiring the answer + to carry the block containing both processors makes it a question about the partition, not the pair, + and a pairwise-primary surface would force the planner into the O(n^2) reconstruction that `SH-16.9` + records going wrong three times. The three *requirements* below are unchanged; only the shape is. +- Unobserved granularities represented, so an answer can be an upper bound and say so. +- A top element, so the query is total. + +## EP-D-3: the residency query + +*Recorded by [CHECKLIST.md](CHECKLIST.md) EP-1.3.* + +### What the planner is choosing + +Two things, and they are different questions that happen to share a subject: + +- **Where each domain's own pool lives.** A domain allocates node-locally to the processor it is + pinned to. Per-processor, unordered, cheap. +- **Which side of a cross-domain pair hosts their shared ring.** Ordered, because the producer + writes and the consumer reads, so the placement decides which of them pays for the crossing. + +This is where the direction that [EP-D-2](#ep-d-2) deliberately refused lands. Proximity is the +link and is symmetric; residency is the hop and is not. + +### The first half is answered, with one asymmetry worth keeping + +`MachineMemoryTopology::memory_domains()` yields the memory domains with their processor sets, so +processor-to-domain is a lookup. + +Partial coverage exists here as it does for caches -- a processor may be named by no memory domain +-- but **the right response is different, and `windows-placement-probe` already got this right**. +Its `places_from_topology` refuses on a missing NUMA node while tolerating a missing cache domain, +and the asymmetry is principled: an unknown cache domain costs an optimisation, whereas an unknown +memory domain has no honest fallback at all, since the pool has to be allocated *somewhere* and +guessing means quietly allocating remote memory for the life of the process. + +So the planner inherits that: an unplaced processor may still host a domain, but not with a +node-local pool, and the difference has to be visible in the plan rather than assumed away. + +### The second half is not merely unpopulated -- it cannot be measured, by construction + +`MachineMemoryTopology::distances` exists, and it is easy to read its permanent `None` as an oversight. It is +not. The field is documented as being for a fed-in description, because "Windows exposes no +user-mode SLIT reader", and that is accurate. + +The sharper problem is what follows from it. `distances` has exactly two input paths: hand +construction, which defaults to `Provenance::Synthetic`, and deserialization, which +`downgraded_to(Provenance::Restored)` caps. `MachineMemoryTopology::discover` hardcodes `None`. So **no path +exists by which `distances` can ever carry `Measured` provenance** -- not because nobody wrote the +code, but because the only sources are a literal and a file, and a file cannot establish that it +describes the machine you are on. + +Under the model's bar -- usable without further measurement -- a planner on a real machine +therefore cannot obtain trustworthy distance for that machine today, and no amount of populating +the existing field would change that. + +### A scalar distance cannot express what this query asks + +Even a populated `Distances` would not answer it. The matrix is SLIT-shaped: one scalar per pair, +with `matrix[i][i]` conventionally `10`. That is a *symmetric, workload-independent* abstraction, +and the residency question is neither. It asks which of two directions is cheaper for a specific +access pattern -- a ring one side writes and the other reads. + +`windows-topology-sys` D-9 already anticipated this precisely, and excluded it deliberately: + +> **HMAT-style attributed relations.** ACPI's Heterogeneous Memory Attribute Table supersedes SLIT, +> giving per-initiator/per-target read and write latency and bandwidth -- four numbers where SLIT +> gives one scalar [...] A general edge list (`{ from, to, read_latency_ns, read_bandwidth_mbps, +> ... }`) would absorb HMAT, **asymmetry**, and multi-hop CXL fabrics; the scalar distance matrix +> this schema keeps will [be revisited when] scalar distance demonstrably mismodels a machine +> somebody is tuning for. + +**This planner is the machine-tuner that deferral names, and asymmetry is exactly the property it +needs.** D-8 makes the revision cheap by keeping the JSON schema outside the semver contract, which +that decision says is "precisely what makes D-9's deferrals safe rather than merely convenient". + +### The trigger is approached, not met, and saying which matters + +D-9's condition is *demonstrable* mismodelling, and honesty requires separating what is shown from +what is expected. + +What is shown: `windows-placement-probe` measures per-hop cost as four numbers per undirected edge +-- two directions times two ring placements -- and its code states that "a hop is not symmetric even +though the link is". The apparatus treats direction as real. + +What is **not** shown: any measurement demonstrating that the four numbers differ. Both development +hosts report a single NUMA node, so every such run is vacuous -- the spike says so itself, printing +"VACUOUS ON THIS MACHINE" and "Apparatus works; question unanswered". So the claim "scalar distance +mismodels this machine" is currently unproven on hardware anyone here has. + +The requirement is real either way, because the planner must choose a side and today has nothing to +choose with. But the *specific* claim that a scalar is insufficient needs a multi-node measurement, +and that measurement should be taken before D-9 is reopened on those grounds rather than after. + +### A measured locality fact must carry what it measured + +The probe's numbers are nanoseconds for one ring-handoff pattern at one message size. Promoting +them into the topology as "the distance" would bake one workload into a model other consumers share +-- and a different consumer, streaming large buffers rather than handing off small messages, would +read them as authoritative and be wrong. + +So a measured relation has to name its measurement, not just its value. This is the concrete reason +per-relation provenance has to be more than a trust label: "measured" is not a sufficient +description of a number whose meaning depends on how it was obtained. + +### What this asks of the model + +- Processor-to-memory-domain, with the unplaced case distinguishable rather than defaulted, because + here it has no honest default. +- A **directed** cost between memory domains, which SLIT's scalar cannot express and which D-9 + already sketched as an attributed edge list. +- Provenance rich enough to say *what* a measured number measured, so one consumer's workload does + not become every consumer's constant. +- And, before reopening D-9 on the asymmetry argument: a multi-node measurement showing the + directions actually differ. + +## EP-D-4: the four-part architecture, and the planner's name + +*The engineer's position, 2026-09-03. This is a **choice**, not one of M1's queries, and it +re-scopes the component that records it.* + +### What was decided + +**The planner is `topology-planner`** -- deliberately with no `windows-` prefix. + +- **Input**: a description of the **goal** of the topology -- what the caller intends the + arrangement to achieve. Its shape is **explicitly deferred for litigation**, which is a named + deferral rather than an omission. +- **What it queries**: an **abstracted, idealized** description of the machine, covering + **processors, memory, storage (NVMe), interconnects, distances, and bottlenecks**. Not + Windows-shaped, and materially richer than what any one platform reports. +- **Output**: a data structure that **serializes to JSON** and is **still abstracted from Windows**. +- **Adapters, in two directions**: + - *inward* -- exposing the traits the planner needs **over the topology objects already designed**, + so `windows_topology_sys::MachineMemoryTopology` becomes one source feeding the abstract model; + - *outward* -- **realizing** a planned topology in the current process as buffers, rings and + threads, with the user's processing code inserted at the appropriate steps. + +### What it settles + +**The crate-naming question** (`MMT-1.5` in +[windows-topology-sys](../windows-topology-sys/COMPLETED-CHECKLIST.md)). The planner does not live in +`windows-topology-sys`, which therefore stays a pure Win32 wrapper and keeps its `-sys` name. The +decisive point is not preference but the adapter boundary: a crate on one side of an adapter is +exactly what `-sys` names, and [D-20](../windows-topology-sys/DESIGN-NOTES.md#d-20) already scoped +that crate to "what the Win32 topology APIs report". + +**"Two graphs, one word"** -- [COMPONENT.md](COMPONENT.md) flagged that both the input and the output +are graphs of processors and relations, so "topology" named all of them and distinguished none. Three +things are now distinct: the **machine memory topology** (Windows facts), the **abstract topology** +(idealized, multi-source, platform-neutral), and the **planned topology** (the output). The word is +shared deliberately; the qualifier carries the distinction. + +**Where distance lives**, which three decisions had left in tension: + +- [D-20](../windows-topology-sys/DESIGN-NOTES.md#d-20) removed `distances` from the facts crate, + because Win32 does not report it and that crate does not go below Win32. +- [EP-D-3](#ep-d-3) established that the planner needs a **directed** cost, which a SLIT-shaped + scalar cannot express. +- `windows-topology-sys` D-9 deferred HMAT-style attributed relations until scalar distance + "demonstrably mismodels a machine somebody is tuning for" -- a trigger this component *approaches* + and, lacking multi-node hardware, has not met. + +The abstract model resolves all three without disturbing any: **interconnects and bottlenecks** are +the attributed-edge shape D-9 sketched, and they live in the abstract model, so D-9's deferral in the +facts crate **stands unreopened** while the need it named is met elsewhere. The measurement condition +still applies before claiming asymmetry is real; it just no longer gates the schema. + +**Storage becomes representable**, which `windows-topology-sys` D-9 also excluded -- on the grounds +that it "changes the crate's identity from processor topology to system topology". That exclusion was +about *that crate* and still holds. NVMe belongs to the abstract model, which was never scoped to a +processor topology. + +### What it opens + +- **Component layout.** How many crates, and where the traits live. If the traits are defined in the + planner, the inward adapter depends on the planner, which points the wrong way for a crate whose + job is to describe a machine. An abstract-model crate that both depend on avoids that, at the cost + of a fourth component. **Not yet decided.** +- **Who measures.** The previous framing had this component measuring with permission. If distance is + a property of the abstract model, measurement plausibly belongs to whatever *populates* that model + -- an adapter -- rather than to the planner. The three-stage split (observe / synthesize / execute) + survives; which component owns the middle stage does not obviously. +- **`MMT-1.3` / `EP-1.4`'s consumer changed.** Both ask what a consumer does with a fact that was not + observed. That consumer is no longer the planner reading `MachineMemoryTopology` directly -- it is + the **inward adapter**, deciding how an absent Windows fact appears in the abstract model. The + decision is still one decision, and it is still to be taken jointly, but it is taken at a boundary + that did not exist when both items were written. + +### What survives unchanged + +[EP-D-1](#ep-d-1), [EP-D-2](#ep-d-2) and [EP-D-3](#ep-d-3) are **requirements**, and requirements +survive a change of binding. Each stated what the planner must know and why; what changes is that +they are now satisfied by traits over an abstract model rather than by methods on a Windows type. +They were written against a real caller, which is what makes them portable in this way. + +## EP-D-5: the component layout, and which way dependencies point + +*The engineer's choice, following [EP-D-4](#ep-d-4). Recorded separately because EP-D-4 explicitly +left it open.* + +### The decision + +**The abstract model and the traits the planner queries live in their own crate, `topology-model`, +which both the planner and the adapters depend on.** + +| Component | Platform | Depends on | +|---|---|---| +| `topology-model` | neutral | nothing | +| `topology-planner` | neutral | `topology-model` | +| inward adapter | Windows | `topology-model`, `windows-topology-sys` | +| outward adapter (realizer) | Windows | `topology-model`, the runtime crates | + +Everything depends on `topology-model`; **nothing depends on `topology-planner`** except a caller +that actually wants to plan. + +### Why not put the traits in the planner + +Because the arrow points the wrong way. An adapter whose job is to describe a machine would have to +depend on a planner in order to describe it, and anyone wanting to read a topology would pull in +planning policy they did not ask for. That is the same defect +[COMPONENT.md](COMPONENT.md) already records in a different place -- `outermost_partitioning_cache`, +a policy answer sitting where facts are stated -- arriving as a dependency edge rather than as an +API. + +### The same rule decides where the plan type goes, one level down + +This is a **derived** consequence rather than a separately-taken decision, and it is called out +because it is easy to miss: the realizer consumes a plan. If the plan type lived in +`topology-planner`, the realizer would depend on the planner -- policy dragged in by a component +whose only job is to execute. + +So **the plan type lives in `topology-model` too**, alongside the machine vocabulary. The crate is +"the shared vocabulary", not merely "the machine description". This is consistent with +[COMPONENT.md](COMPONENT.md)'s existing argument that a plan is a **value** -- inspectable, +comparable, reviewable before anything is pinned or allocated. A value type belongs with the +vocabulary, not with the policy that produced it. + +### Two Windows adapters, not one + +Also derived. They are both Windows adapters and it is tempting to fuse them, but their dependency +sets barely overlap: the inward one needs `windows-topology-sys`, the realizer needs the runtime +(`windows-ioring-sys`, `windows-waitable-queues`, `windows-thread-ambient-sys`). Fusing them would +mean anyone reading a topology pulls in the whole runtime, which is the same "do not drag in what +the caller did not ask for" rule that decided the layout in the first place. + +### What is still open + +- **The adapters' names.** Deliberately not settled here; naming has been getting decided by + whoever writes the first type, and this component has already been renamed once. +- **Who measures.** Carried forward from [EP-D-4](#ep-d-4) and not resolved by the layout: if + distance is a property of the abstract model, measurement plausibly belongs to whatever populates + that model. `topology-model` depends on nothing, so it cannot measure; that puts the measurement in + an adapter or in a fifth thing. +- **Whether `topology-model` is one crate or eventually two.** The machine description and the plan + vocabulary are different enough that they might separate later. They are together now because + splitting on speculation costs more than merging on evidence. diff --git a/crates/windows-ioring-sys/CHECKLIST.md b/crates/windows-ioring-sys/CHECKLIST.md index 2e0a2666..3424739b 100644 --- a/crates/windows-ioring-sys/CHECKLIST.md +++ b/crates/windows-ioring-sys/CHECKLIST.md @@ -128,6 +128,10 @@ conclusions belong to it until it converges. PDO advertises a proximity domain. Write the correction now (the documentation defect is independent of the measurement) and leave the empirical question open. +- [x] **M20.5** -- Dissolved by [D-47](DESIGN-NOTES.md#d-47-detail) rather than decided: the + `flush_barrier` assertion was measuring a claim the platform does not honour, so it was never a + flaky test. -> [completed 2026-09-07](COMPLETED-CHECKLIST.md#m205) + - [ ] **M20.6** -- Re-evaluate `CommitStrategy::AlternatingRings` and the epoch-log benchmark's conclusion against [D-47](DESIGN-NOTES.md#d-47-detail). The strategy comparison in [strategy.rs](examples/epoch_log/strategy.rs) was designed around D-24's claim that a covering flush holds @@ -141,8 +145,8 @@ conclusions belong to it until it converges. answer is that the strategy no longer earns its place, that is an API change to a published example. The corrected prose in [strategy.rs](examples/epoch_log/strategy.rs) and [DESIGN-NOTES.md](DESIGN-NOTES.md) both point here. - *(Numbered M20.6 rather than M20.5 deliberately: M20.5 is in flight on `mikegrier/deferred-namespace-ops` - and the gap reserves it, so the two do not collide when that branch merges.)* + *(Numbered M20.6 rather than M20.5 because M20.5 was in flight on a separate branch when this was + written. That branch has since merged, and M20.5 arrived dissolved -- see above.)* ## M6+ -- Model B: explicit-thread delivery and affinity diff --git a/crates/windows-ioring-sys/COMPLETED-CHECKLIST.md b/crates/windows-ioring-sys/COMPLETED-CHECKLIST.md index eb0eceda..47066e81 100644 --- a/crates/windows-ioring-sys/COMPLETED-CHECKLIST.md +++ b/crates/windows-ioring-sys/COMPLETED-CHECKLIST.md @@ -1494,3 +1494,13 @@ Independent of M17; may run in parallel. `unwrap_or_else(PoisonError::into_inner)`. `scope()` absorbs poisoning internally, matching what the wait callback's own drain already did, so the question no longer reaches callers at all. Verified: 148 tests pass, both affected examples (`model_a_delivery`, `epoch_log`) still run to exit 0. + +## Moved 2026-09-07 -- M20.5: dissolved by D-47 rather than decided + +### M20.5 -- Dissolved by [D-47](DESIGN-NOTES.md#d-47-detail) rather than decided: the `flush_barrier` assertion was measuring a claim the platform does not honour, so it was never a flaky test. *(completed 2026-09-07 21:40:23 -04:00)* + +**Dissolved by [D-47](DESIGN-NOTES.md#d-47-detail), not decided.** This asked whether to make the +load-sensitive `flush_barrier` assertion load-independent or mark the test serial. Neither: the +assertion was measuring a claim the platform does not honour, so it was not a flaky test at all. Its +own contention hypothesis was disproved in the same measurement -- an idle ring failed too. Recorded +in [RESOLVED-TEST-FAILURES.md](RESOLVED-TEST-FAILURES.md). diff --git a/crates/windows-platform-probes/COMPLETED-CHECKLIST.md b/crates/windows-platform-probes/COMPLETED-CHECKLIST.md new file mode 100644 index 00000000..9a975505 --- /dev/null +++ b/crates/windows-platform-probes/COMPLETED-CHECKLIST.md @@ -0,0 +1,217 @@ +# Completed checklists: windows-platform-probes + +Append-only. Newest groups at the bottom. + +## Moved 2026-09-05 -- claim-word layout: measured the apportionment, then shipped it as a caller's choice + +# Checklist: claim-word layout + +Measures how the `reserving_mpsc` claim word's bit apportionment and width +affect push throughput, then offers the layouts as documented, caller-selectable +options in `windows-waitable-queues`. + +Design decisions land in [DESIGN-NOTES.md](DESIGN-NOTES.md) for the measurement +and in the queue crate's +[DESIGN-NOTES.md](../windows-waitable-queues/DESIGN-NOTES.md) for the API +(`D-36`, `D-37`). + +## Background + +`reserving_mpsc` packs `reserved` and `position` into one `AtomicU64` because +the claim protocol needs a single compare-and-swap to update both (`D-17`, +`D-34`). The split is 32/32, which caps positions at 2^32 and is the whole +source of the `SH-14.1` recurrence hazard disclosed by `D-36`. + +**The 32/32 split is not forced by the platform.** It follows from a capacity +ceiling of 2^31, because the `reserved` half must be able to hold the entire +capacity. Two independent constraints bound the capacity: + +- ring arithmetic: `capacity <= 2^(POSITION_BITS - 1)` +- packing: `capacity <= 2^(64 - POSITION_BITS) - 1` + +`BOUNDS_MAX` is currently derived from the first alone and the second is only +*asserted*, so widening the position raises the ceiling while shrinking the +field obliged to hold it -- which is why widening trips the assertion instead of +working. + +## M1: measure the layouts -- done + +Built as a duplicated path in this crate so the measurement added no third-party +dependency to a publishable crate and could not disturb the +`windows-waitable-queues` branch being peeled off PR #56. + +- [x] **CW-1.1** -- Add `portable-atomic` with `default-features = false` to + this crate only, and record whether `AtomicU128` exists and is lock-free. + Measured: `is_always_lock_free()` is true and `cmpxchg16b` is a default target + feature here, so no CPUID branch was timed as though it were the algorithm. + +- [x] **CW-1.2** -- Implement the three claim-word layouts as self-contained + `u64`-item queues in [claim_layout.rs](src/claim_layout.rs). + +- [x] **CW-1.3** -- Wire the three layouts into `probe-queue-contention` as + named shapes in both regimes. + +- [x] **CW-1.4** -- Run the probe and capture the report. Result: + re-apportioning is free (16/48 tracks 32/32 within noise in both regimes); + widening to `u128` costs 2-3x isolated and 5-12% drained, and the drained + figure understates it because a slower producer earns fewer refusals. + +- [x] **CW-1.5** -- Record the measurement and the rollover table in + [DESIGN-NOTES.md](DESIGN-NOTES.md). + +## M2: offer the layouts as options + +**Decided: offer a set of named layouts rather than one, on the condition that +each carries its own ramifications.** The engineer's direction was that options +are right "as long as quality is maintained" and "the ramifications of the +choices are available". Both halves are binding, and the second is the one an +options API usually fails: a caller who cannot see what a layout costs will pick +by name, and the names are the least informative thing about them. + +Three obligations apply to every item in this milestone: + +- **Each layout states its own consequences where it is named** -- reservation + ceiling, capacity ceiling, and time-to-recurrence at a stated push rate. The + rollover figures in [DESIGN-NOTES.md](DESIGN-NOTES.md) are the source; the + crate documentation restates them once and nothing else does. +- **Quality is per-layout, not per-crate.** Every layout gets the same const + assertions, tests, and mutation coverage as the shipping one. A layout + exercised only by a doctest is worse than no option, because its presence + claims a support level nothing verifies. +- **Adding a layout must not weaken the default.** The layout parameter must + not leak into the signatures of callers who do not use it. If it cannot be + kept out, say so rather than accepting the churn. + +**This milestone is no longer parked.** It was gated on the peel merging, on the +reasoning that touching `windows-waitable-queues` would re-grow a branch under +review. That reasoning expired: `mikegrier/waitable-queues` has no pull request +open, so there is no review to disturb, and the `u64` layouts need no new +dependency at all -- only 64/64 does, which is `CW-2.3`. + +- [x] **CW-2.1** -- Introduce the layout as a compile-time parameter, widen the + position to 64 bits, and decouple the reservation ceiling from the capacity. + + **Merged from two items during execution, because they cannot be verified + apart.** Decoupling the ceiling is numerically invisible at 32/32: + `MAX_RESERVED` is 2^32-1 while `BOUNDS_MAX` is 2^31, so a cap on outstanding + reservations can never bind and no test can reach it. It becomes observable + only once a layout makes the reservation half narrow. Landing them separately + would have meant committing a branch nothing could exercise and calling it + done. + + The three parts: + + - Cap *outstanding reservations* at `MAX_RESERVED` in `reserve`, and drop the + `BOUNDS_MAX <= MAX_RESERVED` const assertion that ties the capacity to the + reservation field. `BOUNDS_MAX` then follows from ring arithmetic and the + crate-wide bound alone. + - Widen `position`, `head`, and the per-slot `sequence` to 64 bits for every + layout, since a position of more than 32 bits cannot be read out through + `position_of`'s `u32`. Uniform 64-bit metadata is measured-safe rather than + assumed: `CW-1.4` compared 32/32 with 32-bit metadata against 16/48 with + 64-bit metadata and found no difference, and for a `u64` payload the slot is + 16 bytes either way once alignment is applied. + - Add the layout parameter with a default preserving today's behaviour. + Generic defaults are permitted on types but not on functions, so `bounded` + keeps its signature and returns the defaulted types, and a second entry + point names a layout explicitly. + + **This is a contract change**: the shipping shape promises every slot may be + reserved at once, and this replaces that with a fixed reservation ceiling. A + different promise rather than a broken one, but it must be stated, not slipped + in. + +- [x] **CW-2.3** -- Decide whether a 128-bit claim word ships at all. + + **Decided: yes, behind an opt-in `dwcas` feature.** The `Wide` layout packs a + `u128` divided 64 / 64. Without the feature the crate depends on + `windows-sys` alone and every layout uses `AtomicU64`; with it, + `portable-atomic` appears. So a caller who does not want the dependency does + not carry it, and one who wants a guarantee rather than a twenty-year + argument can have it. + + This resolves `CW-1.6`'s scope the other way from what the item anticipated: + the shipping crate *can* now express a 128-bit layout, so the probe does not + need to keep its own `wide` implementation to measure one. + + **Not a dependency question.** An earlier form of this item framed it as + whether `portable-atomic` becomes a dependency of a published crate, which was + wrong: `core::arch::x86_64::cmpxchg16b` is stable on the pinned toolchain, so + a 64/64 layout needs no third-party crate. `D-7`'s and `D-37`'s dependency + cost does not apply, and the decision must not be made on it. + + What it actually costs: hand-written `unsafe` with manual orderings in the + file where that is worst to get wrong, x86-64 only (no ARM64 `casp`, no + i686), and a `target-feature` or runtime-detection decision. Against that, + `CW-1.4` measured the 128-bit exchange 2-3x slower on the claim in the + isolated regime, and `CW-2.1` has since made `Perpetual` reach about 20 years + before recurrence on a plain `AtomicU64` at no measured cost. + + So the question is narrow: is going from unreachable-in-any-deployment to + unreachable-in-principle worth that? The engineer has said 32-bit Windows + deployment is not a present concern, which changes `D-18`'s premise and must + be recorded rather than assumed. + + **`CW-1.6`'s scope is decided by this item**: if `portable-atomic` is + declined, this crate must keep its `wide` implementation, because a layout the + queue crate cannot express is one the probe cannot instantiate. + +- [x] **CW-2.4** -- Document the layouts as a choice, in the crate documentation + and the README, with the rollover table and the two axes a caller trades + between: outstanding reservations against time-to-recurrence. Lead with what + `CW-1.4` measured -- re-apportioning is free, widening is not -- so a caller + is not left assuming the safest option must be the slowest. State the push + rate the figures assume, and that a draining queue cannot sustain the fastest + of them. + + **Compiled, not merely written.** Any README example naming a layout is a + doctest per this repository's CONTRACT INTEGRITY rule, so a renamed or removed + layout breaks the build instead of leaving the documentation teaching a name + that no longer exists. + +- [x] **CW-2.5** -- Reopen `D-36` with the measurement in hand, then sweep every + statement of the hazard. + + **`D-36`'s premise is falsified, and that is the finding, not the sweep.** It + decided 0.1.0 ships `SH-14.1` disclosed rather than fixed *because the fix is + a claim-protocol replacement (`D-35`) gated on an open question*. Re- + apportionment is a second fix that neither `D-36` nor `D-37` considered, and + `CW-1.4` measured it free. It does not eliminate the recurrence -- only moves + it -- but 8/56 moves it from about 37 seconds to about 20 years at the + disclosed rate, which takes `D-36`'s "computed exposure" from reachable in + under a minute to unreachable in any real deployment. + + So the question is whether the crate ships this hazard at all. Answer that + first; the sweep follows from the answer. + + The sweep is blast-radius, not an edit of one reported site: `D-36` states the + hazard in the crate documentation, the README, and `reserving_mpsc`'s module + documentation, each leading with "on every target, not only 32-bit ones", and + `lib.rs` separately claims the shape is "sound below the wrap". Every one is + scoped to a 32-bit position. Grep the distinguishing terms across `src/`, + `tests/`, `examples/` and `*.md` for the crate and its dependents, fix every + hit or say why it is out of scope, and record the sweep in the commit message. + +## M3: retire the duplicate + +- [x] **CW-1.6** -- Delete the duplicated *implementation* in + [claim_layout.rs](src/claim_layout.rs), keeping only what `CW-2.3` leaves no + other way to measure. + + **This is not a decision about which layouts to offer.** That is settled -- + multiple layouts ship as caller-selectable options, per `M2`. This item is + only about the private copy of the reserving protocol in this crate, which + existed so the layouts could be measured without touching + `windows-waitable-queues`. + + **`M2` makes the copy obsolete.** Once the shipping crate takes the layout as + a compile-time parameter, the probe instantiates the *real* type at any layout + it wants to compare, including candidates that are not defaults -- so + exploring a new apportionment no longer needs a duplicate. + + Deleting it is not tidiness. A second implementation of the same protocol + drifts, and this one already did: `CW-1.4`'s first run measured 3.7x against + the shipping shape on an entirely different scaling curve, because the + duplicate had not cache-padded `head` and the claim word. Corrected, it still + sits about 1.26x off. A duplicate that diverges silently produces a + measurement that looks healthy and describes something nobody ships. diff --git a/crates/windows-platform-probes/COMPLETED-PLANS.md b/crates/windows-platform-probes/COMPLETED-PLANS.md new file mode 100644 index 00000000..7497bbda --- /dev/null +++ b/crates/windows-platform-probes/COMPLETED-PLANS.md @@ -0,0 +1,5 @@ +# Completed plans: windows-platform-probes + +| Path to CHECKLIST.md | Completion Date | Brief description | Design Notes | +|---|---|---|---| +| [COMPLETED-CHECKLIST.md](COMPLETED-CHECKLIST.md) | 2026-09-05 | Measured how `reserving_mpsc`'s claim word apportionment and width affect throughput, then shipped the layouts as caller-selectable options: `Balanced`, `Enduring`, `Perpetual`, and `Wide` behind the `dwcas` feature. Superseded D-36, whose premise was that fixing SH-14.1 required the claim-protocol replacement. | [DESIGN-NOTES.md](DESIGN-NOTES.md) | diff --git a/crates/windows-platform-probes/Cargo.toml b/crates/windows-platform-probes/Cargo.toml index d7f7f6e0..911f789f 100644 --- a/crates/windows-platform-probes/Cargo.toml +++ b/crates/windows-platform-probes/Cargo.toml @@ -15,6 +15,10 @@ description = "Executable probes for the undocumented Windows behaviour this wor [lib] path = "src/lib.rs" +[[bin]] +name = "probe-core-affinity" +path = "src/bin/core_affinity.rs" + [[bin]] name = "probe-error-mode" path = "src/bin/error_mode.rs" @@ -47,6 +51,26 @@ path = "src/bin/ioring.rs" name = "probe-pool-growth" path = "src/bin/pool_growth.rs" +[[bin]] +name = "probe-doorbell-cost" +path = "src/bin/doorbell_cost.rs" + +[[bin]] +name = "probe-request-cost" +path = "src/bin/request_cost.rs" + +[[bin]] +name = "probe-topology" +path = "src/bin/topology.rs" + +[[bin]] +name = "probe-queue-contention" +path = "src/bin/queue_contention.rs" + +[[bin]] +name = "probe-peer-index-cache" +path = "src/bin/peer_index_cache.rs" + # These two are the same code, and that is the measurement: they differ only in # whether `build.rs` embeds the `longPathAware` manifest, which is not a runtime # switch and so cannot be a flag on one binary. @@ -68,17 +92,42 @@ path = "src/bin/long_path_unaware.rs" # A `version` beside a `path` is consulted only when the depending crate is # packaged, but cargo still requires the path crate's own version to satisfy it # at every build -- so a pin left behind by a bump breaks the whole workspace's -# resolution rather than only this crate's. +# resolution rather than only this crate's. With six such pins, that was six +# standing chances to break `main` in exchange for nothing. # # The pool-growth probe measures the shipping API rather than a # reimplementation of the SDK's inline environment helpers, so it depends on the # real crate. windows-threadpool-sys = { path = "../windows-threadpool-sys" } -# Every probe's report opens with a line naming the machine that produced it and -# whether the measurement is tainted, so a captured finding cannot be pasted -# somewhere and compared against something it does not describe. That banner is -# `windows-placement-probe`'s to render, not a second copy here. +# Same reason: the topology probe measures what the shipping parse produces, not +# a second parse written here, which would only measure itself. The raw Win32 +# counters it cross-checks against are read independently through windows-sys. +windows-topology-sys = { path = "../windows-topology-sys" } +# The placement measurement moved out to its own crate so it could be shared +# with people running it on hardware this workspace does not own. The probes +# here call into it rather than keeping a second copy: two renderings of one +# measurement disagreeing is a defect this investigation has already hit. +# +# It renders the banner too: every probe's report opens with a line naming the +# machine that produced it and whether the measurement is tainted, so a captured +# finding cannot be pasted somewhere and compared against something it does not +# describe. That banner is this crate's to render, not a second copy here. windows-placement-probe = { path = "../windows-placement-probe" } +# The request-cost probe measures the real request types the design would put on +# a queue, not a stand-in, for the same reason. +windows-namespace-request-sys = { path = "../windows-namespace-request-sys" } +# The contention probe measures the shipping queue shapes rather than a +# reimplementation, for the same reason: a stand-in would only measure itself, +# and the whole question is what the real tail claim costs. +# The experimental permit claim is enabled here because this probe is what +# decides its fate (SH-15.5): it must be measured against the shipping shapes +# on the same host, in the same run, by the same harness. +windows-waitable-queues = { path = "../windows-waitable-queues", features = [ + "experimental-permit-claim", + # So the probe can instantiate the 128-bit layout. The 64-bit ones need no + # feature; this is the only one that costs the queue crate a dependency. + "dwcas", +] } # The long-path probe measures a length against `MAX_PATH`, and `MAX_PATH` counts # UTF-16 code units. `OsStr::len` counts Rust's platform encoding -- WTF-8 here -- # so the two disagree the moment a non-ASCII character appears in `%TEMP%`, which @@ -104,6 +153,9 @@ features = [ # half of what a relative path resolves against, so the probe has to place # it deliberately rather than inherit whatever launched it. "Win32_System_Environment", + # GetSystemDirectoryW, so the request probe measures the real system + # directory instead of assuming Windows is installed on C:. + "Win32_System_SystemInformation", "Win32_System_Diagnostics_Debug", "Win32_System_IO", "Win32_System_Pipes", diff --git a/crates/windows-platform-probes/DESIGN-NOTES.md b/crates/windows-platform-probes/DESIGN-NOTES.md index b0b88b74..c2ed7d16 100644 --- a/crates/windows-platform-probes/DESIGN-NOTES.md +++ b/crates/windows-platform-probes/DESIGN-NOTES.md @@ -140,6 +140,27 @@ test suite for this workspace's crates: a probe answers "what does Windows do?", never "does our code work?". A probe that starts asserting our own behaviour belongs in the crate that owns that behaviour. +## This crate is never distributed, and its dependencies carry no versions + +Not to a registry, and not as a released binary either -- unlike +`windows-placement-probe`, which ships a CI-built binary to people running it on +hardware this workspace does not own. These probes are a development +instrument, run from a checkout by someone who has the checkout. `publish = +false` is the whole story, and it is permanent rather than "not yet". + +**The consequence is that every workspace dependency here is path-only.** A +`version` beside a `path` exists to tell a registry what to resolve when the +depending crate is packaged. Nothing packages this crate, so those pins named a +version no one would ever consult -- while still having to be correct, because +cargo requires the path crate's own version to satisfy the pin **at every +build**, not merely at publication. + +That is not a theoretical tidy-up. Measured on 2026-09-02: bumping +`windows-topology-sys` to 0.2.0 while this crate pinned `"0.1.0"` failed +`cargo metadata` for the whole workspace. Six such pins were six standing +chances to break `main` on someone else's release, in exchange for nothing, +and they are gone. + ## The earlier probes are migrated, and two of them corrected in the move @@ -460,6 +481,331 @@ architecture. It would fail on any host with a longer user name, on either architecture. Recorded here because it is exactly the kind of result this comparison exists to classify correctly: a red build that is **not** a finding. +## The queue-contention probe, and why it must not run in the CI probe job + +`probe-queue-contention` measures what M31.5 of +[CHECKLIST-io-domains.md](../../CHECKLIST-io-domains.md) exists to decide: whether the bounded array +queue's tail claim contends badly enough to justify the linked and sharded MPSC shapes, and what +`reserving_mpsc`'s extra read of the consumer's position actually costs. + +**It is deliberately absent from the `platform-probes` CI job, unlike every other probe, and the reason is +a measurement rather than a preference.** That job runs `cargo run` without `--release`. Measured in a +debug build, `mpsc` and `reserving_mpsc` come out at 249.7 and 254.0 ns/push at sixteen producers -- +indistinguishable. In release, on the same machine in the same minute, they are 193.5 and 52.2. The +un-inlined overhead of a debug build swamps the cache-coherence effects that *are* the finding, so a debug +run of this probe does not merely lose precision: it reports the two shapes as equivalent, which is a +confident wrong answer of exactly the kind this crate's `doorbell_cost` notes warn about. + +Two further reasons it stays out. A contention curve needs more cores than a hosted runner has, and the +32-producer rows on a four-core runner would measure the scheduler. And the run costs about two minutes in +release, against a job whose other probes are seconds. + +So this one is run by hand, on a known machine, and its numbers are recorded with the machine attached. + +### Reading it + +Two regimes, and the pair is the point. + +**Isolated** gives producers a capacity large enough that nothing is ever refused and runs no consumer, so +whatever curve appears against N is the claim and nothing else. **Drained** runs a consumer popping +continuously, which is the only regime that can price `reserving_mpsc`'s read of `head` -- that read is +cheap until a consumer is *writing* the line, and measuring it in isolation would report it as free. + +The drained regime has a **single** consumer, because that is what MPSC means, so at high producer counts +it becomes consumer-bound and a plateau there says nothing about the claim. Each row carries the refusal +count from the queue's own `Observable` counters precisely so that is visible as a fact rather than +mistaken for contention: the sixteen- and thirty-two-producer drained rows show millions of refusals and +should be read as measurements of the consumer. + +## `probe-core-affinity`: placement costs 5.6x, and it refuted the hypothesis it was written to test + +This probe pins an SPSC producer and consumer to chosen logical processors and measures the handoff +under each placement the machine can express. It exists because +[`probe-peer-index-cache`](#probe-peer-index-cache-a-result-that-inverts-by-host-which-is-why-it-is-kept) +gave opposite answers on two hosts, and the obvious suspect was *placement*: a machine with two +efficiency classes might be decoupling the two threads in a way a homogeneous one does not. + +**The plain answer, which is the useful one.** On the ARM64 development host the unoptimised handoff +costs **38.5 ns/item within a domain and 215.3 ns/item across domains -- 5.6x, for no change but where +the two threads run.** Within a class, the performance cores (class 1) run the same handoff at 30.4 ns +against the efficiency cores' 38.7, about 27% apart, which is a real but far smaller effect than +crossing the boundary. Medians of three, stable across three invocations. + +**The hypothesis was refuted, and backwards.** The prediction was that mismatched core speeds would +decouple the two sides, letting a backlog form and giving peer-index caching the deep batch it needs. +Measured, threads placed *together* batch **~135x deeper** than threads placed apart (49.6 against 0.4 +items per shared read). A coherent reading is that a cheap handoff lets the producer race ahead and +build a backlog while an expensive one throttles it into lockstep -- so cost drives depth rather than +core speed driving it -- but **this run does not test that**, and the probe says so rather than +recording a replacement conclusion it did not earn. What is established is only that the original +prediction is wrong. + +**It also failed to explain the host disagreement, which was its main purpose.** Caching wins at *both* +placements here (14.4x together, 3.0x apart), so placement alone does not account for x64 rejecting the +technique while ARM64 accepts it. That question stays open under `D-28` and M-inf.4. + +**A confound this machine cannot escape, stated because it bounds every reading above.** Its efficiency +classes and its cache domains coincide exactly -- processors 0-5 are class 0 behind one L2, 6-11 are +class 1 behind the other -- so every cross-class pair is also a cross-cache pair. The 5.6x is +"across domains", and attributing it to core speed *or* to cache would need a machine whose classes and +caches cut differently. The probe detects this and prints a CAUTION rather than letting a reader draw +the finer conclusion; two of its four placement rows come back `n/a`, and reporting a placement as +inexpressible is deliberately not the same as reporting that it made no difference. + +Two construction notes. **Pinning failures panic** rather than warn: a silently unpinned thread turns a +placement experiment into a measurement of the scheduler's preferences while still printing a confident +number. And **batch depth is read from the cached runs only** -- the baseline strategy reads the shared +line on every operation by definition, so its depth is ~1 at every placement and carries no +information. An earlier revision compared the baseline depths and duly reported 0.8 against 0.4, which +is noise around a constant being read as a finding. + +## `probe-peer-index-cache`: a result that inverts by host, which is why it is kept + +This probe measures peer-index caching -- each side of an SPSC ring keeping a plain copy of the other +side's position, so the shared line is read once per batch instead of once per item -- against the +`windows-waitable-queues` `spsc` shape. **It gives opposite answers on our two architectures**: roughly +1.8x slower on x64, roughly 17x faster on ARM64, because the batch depth it amortises over is set by how +the two threads interleave on that host rather than by our code. The full reasoning lives with the queue +as [DESIGN-NOTES.md](../windows-waitable-queues/DESIGN-NOTES.md) -> `D-28`. + +This section previously described the probe as recording a settled rejection, on x64 evidence alone. + +Three things about its construction are deliberate and worth keeping if it is ever edited. + +**It counts shared reads, not just time.** A timing-only result would have been unreadable: "caching is +slower" is indistinguishable from "the caching was implemented wrongly and never engaged". The read +counters settle that directly, and they are also what made the two hosts comparable -- the reads reveal +a batch depth near 1 on x64 against roughly 150 on ARM64, which is the mechanism rather than the +symptom. Any future variant added here must keep the counters for the same reason. + +**Its interpretation is derived from the run, and must never go back to prose.** It used to print the +x64 conclusion as a fixed paragraph -- "the technique WORKED and still lost", "roughly 3.6x", "the +producer count goes UP" -- with only the speedup ratio computed. Run on ARM64 it printed all three while +its own table three lines above showed the opposite, and the contradiction was noticed by a reader +rather than by the tool. A probe that states its finding regardless of what it measured is worse than no +probe, because it is believed. The interpretation now computes the batch depths and says outright that +this verdict is host-dependent. + +**It carries a calibration row and a warming control.** The calibration times the real shipping `spsc` +beside the model, and the probe prints a CAUTION when they diverge by more than 25% -- which they +currently do, so the probe says out loud that its rows describe the model rather than the shipped +queue. That guard earned its place immediately: the first run's 3x gap would otherwise have been read +straight past. The warming variant is a control for the hypothesis that a discarded prefetch could +substitute for the real thing; it removes no read and moves no time, which is exactly what a control +that confirms the null should do. + +Like `probe-queue-contention`, this probe is **absent from the CI probe job**, and for the same +measured reason: the effects it studies are coherence effects that a debug build's overhead buries. + +## The fingerprint carries provenance inside the string, not beside it + +The fingerprint is a **canonical summary of a machine's marginal shape**: two hosts rendering the +same string have the same processor, core, cache-domain, class and node sizes, so string equality +is a supported way to group results by shape. (It does *not* mean the two can express the same +placements -- the sizes are recorded without how the partitions intersect. See +[`Fingerprint::provenance`](../windows-placement-probe/src/fingerprint.rs).) That the string is +compared at all is what forces the provenance marker to live *inside* the rendered form. A marker kept alongside -- a separate field, a +second printed line, a note in the surrounding prose -- would leave a fabricated machine claiming the +exact shape of a real one **comparing equal to it**. That is a concrete bug rather than a display +preference, and it has a test named for it. + +Three details are deliberate: + +- **A measured host renders exactly as before, with no prefix.** Every fingerprint already recorded in + a checklist or design note came from a real machine, so those strings stay valid and comparable + rather than being silently reinterpreted by this change. +- **The prefix leads**, so a reader scanning a column of pasted results cannot skip it, and it is + removable -- stripping `!!SYNTHETIC!! ` yields exactly the measured rendering, so a synthetic host + can still be compared against a real one on purpose. +- **`RESTORED` and `SYNTHETIC` are distinguished** rather than collapsed into one "untrusted". They + are different claims: one describes some real machine, the other describes none, and a reader + deciding how far to believe a number needs to know which. + +`Fingerprint::from_topology` exists so provenance *flows* from the topology rather than being stamped +on afterwards. `discover` is now a thin wrapper over it, which means there is no path that invents an +answer -- whatever the topology says is what the fingerprint reports. + +`print_banner` was split so the line is available as a string. The taint marker reaching that line is +the entire point of carrying provenance, and a property that load-bearing should not rest on someone +having read a format string correctly. + +## Which seams are safe: data may be injected, labels may not reach hardware + +Two topology-injection seams were considered during this work and they were decided opposite ways. +The rule that separates them is worth stating on its own, because "add a seam for testability" reads +as unambiguously good and here it is only half true: + +**A seam that only moves data is safe. A seam that lets fabricated labels reach real hardware is +not.** + +- [`places_from_topology`](../windows-placement-probe/src/fingerprint.rs) **has** a seam. It is a pure conversion -- topology in, + processor positions out, nothing pinned and nothing timed. A synthetic topology yields synthetic + positions, which is what the caller asked for and cannot be mistaken for a measurement. +- [`measure`](../windows-placement-probe/src/core_affinity.rs) **must not**, and its documentation says so at the definition. + A synthetic topology's processor *numbers* are still valid on the real host, so every pin would + succeed and the run would produce genuine timings filed under fabricated node ids -- output + indistinguishable from a real NUMA measurement that measured no such thing. The pin assertion does + not catch it: it rejects a processor that does not exist, not a label that is wrong. + +The absence of the second seam is also what lets `Slice` carry no provenance marker of its own, so +the two decisions hold each other up. + +### The hole this closed, and how it was proven + +`discover_places` took no argument and appeared in no test. It was untestable, not merely untested, +and it carries the rules for the partitioning cache level, core and class membership, and the NUMA +node. The NUMA lookup in particular was **unverifiable on every host available to this workspace**: +with a single node, a correct map and a completely broken one both yield node 0. + +Replacing the entire lookup with a hardcoded `0` was run against the suite as it stood before this +change. **It passed everything.** Against the suite now, three tests fail. That is the difference the +seam bought, and it is why the existing `ProcessorPlace` fixtures were kept rather than treated as +sufficient: they encode what a test author assumed the conversion produces, which is precisely the +thing that cannot catch the conversion being wrong. + +## The claim word's width costs 2-3x in isolation and much less in use + +Measured by `probe-queue-contention` for +[CHECKLIST-claim-word-layout.md](CHECKLIST-claim-word-layout.md) `CW-1.4`, on +one host, `x86_64-pc-windows-msvc`. Three apportionments of `reserving_mpsc`'s +claim word, built as duplicates in [claim_layout.rs](src/claim_layout.rs) so the +shipping crate was not disturbed: 32/32 and 16/48 over `AtomicU64`, and 64/64 +over `AtomicU128`. + +`AtomicU128::is_always_lock_free()` is **true** on this target and +`cfg(target_feature = "cmpxchg16b")` is enabled by default, so the 128-bit +exchange is a compile-time-guaranteed native instruction here and no CPUID +branch was measured as though it were the algorithm. + +| producers | 16/48 vs 32/32 (isolated) | 64/64 vs 32/32 (isolated) | 64/64 vs 32/32 (drained) | +|---|---|---|---| +| 1 | 1.14x | 2.05x | 1.05x | +| 4 | 1.21x | 1.37x | 1.12x | +| 8 | 1.00x | 2.33x | 1.07x | +| 16 | 0.88x | 2.37x | 1.00x | +| 32 | 0.98x | 2.99x | 1.11x | + +**Re-apportioning the bits is free.** 16/48 tracks 32/32 within noise in both +regimes, which is the expected result and worth stating as a confirmed +prediction rather than a discovery: both issue the same `lock cmpxchg` on the +same `u64`, so only the shift and mask constants differ. The 48-bit position +does force `head` and the per-slot `sequence` to 64 bits, and that cost does not +show up either. What this buys is the recurrence moving from 2^32 to 2^48 -- +from about 37 seconds of sustained maximum-rate pushing to about 28 days. + +**Widening the word is not free, and how much it costs depends entirely on the +regime.** Isolated, where the claim is the only thing happening, `cmpxchg16b` +costs 2-3x and the penalty *grows* with contention. Drained, with a consumer +running, it is 5-12%. + +### The drained regime flatters the slower layout, and the refusal counts say so + +The two regimes must not be averaged, and the drained one must not be read as +the answer on its own. **A slower producer is less backpressured**, so it earns +fewer refusals, and refusal retries are inside the timed region. At eight +producers the 64/64 layout took 12,149 refusals against 32/32's 74,181 -- so +part of what makes its per-push number look close is that it spent less time +being turned away. The drained figures are therefore an *understatement* of the +128-bit word's cost, not a measurement of it under load. + +The isolated regime is the clean measurement of the claim itself; the drained +one shows that in a queue doing real work the claim is not the dominant cost. A +real application sits between them, nearer the drained end the more +consumer-bound it is. + +### What the control caught + +The first run reported 3.7x against the shipping shape and a completely +different scaling curve. The cause was that the duplicate had not padded `head` +and the claim word onto separate cache lines, which `reserving_mpsc` does +deliberately -- every producer reads `head` on every push, so sharing a line +puts the consumer's writes in their path. Aligned, the duplicate tracks the +shipping shape's curve. + +A residual gap remains: the duplicate runs about 1.26x slower than +`reserving_mpsc` at high producer counts. That offset applies equally to all +three layouts, so the ratios above stand, but it means these figures are **not** +absolute numbers for the shipping shape and must not be quoted as such. + +**Comparing a duplicate against the original it stands in for is what made both +of these visible.** A run of three layouts that agreed with each other and +disagreed with reality would have looked entirely healthy. + +### What each apportionment actually buys + +The rollover figures for candidate splits, computed from the rates above. The +rate model reproduces the crate's own published figure -- 32/32 at 116M/s gives +37 seconds, which is what `reserving_mpsc`'s module documentation discloses -- so +these are an extension of that disclosure rather than a competing estimate. + +| split (reserved/position) | max outstanding reservations | @257M/s | @116M/s | @33M/s | +|---|---|---|---|---| +| 32/32 (ships) | 2^32 | 17 s | 37 s | 2.2 min | +| 24/40 | 2^24 | 71 min | 2.6 hr | 9.2 hr | +| 21/43 | 2^21 | 9.5 hr | 21.1 hr | 3.1 days | +| 20/44 | 2^20 | 19.0 hr | 42.1 hr | 6.1 days | +| 16/48 | 2^16 | 12.7 days | 28.1 days | 98 days | +| 12/52 | 2^12 | 202 days | 449 days | 4 yr | +| 8/56 | 2^8 | 9 yr | 20 yr | 69 yr | +| 64/64 (`u128`) | 2^64 | 2,270 yr | 5,039 yr | 17,607 yr | + +Rates: 257M/s is the measured isolated peak at one producer, which has no +consumer and so is not a rate any draining queue can sustain -- it is a +conservative floor on time-to-wrap. 33M/s is the measured drained rate at one +producer. 116M/s is the crate's own disclosed figure and is the honest planning +number. + +**The reservation half is where the bits are being spent, and it is the half +worth least.** Outstanding reservations are bounded by how many producers are +mid-flight -- hundreds, perhaps thousands -- and the field currently holds four +billion. Giving up reservations nobody will allocate is what buys the position +bits: 2^21 reservations leaves about a day, 2^12 leaves over a year, and 2^8 +leaves twenty years. The last is the same practical answer a 128-bit word gives, +on a plain `AtomicU64`, at no measured cost, without a third-party dependency and +without reopening `D-18`'s i686 question. + +So the candidates worth considering are **12/52 and 8/56**, not the 16/48 first +sketched here: 16/48's 12.7 days at the conservative floor is still reachable by +a busy long-lived process, and 12/52 is the first row that is not. + +### Re-measured on the shipping type, and the duplicate had understated the wide word + +`CW-1.6` deleted the duplicated protocol in this crate once +`windows-waitable-queues` took the layout as a parameter, so the probe now +instantiates the real type at each layout. The numbers below supersede the ones +above, which were taken from the stand-in. + +| producers | 16/48 vs 32/32 | 8/56 vs 32/32 | 64/64 vs 32/32 | +|---|---|---|---| +| 1 | 1.02x | 0.98x | 1.45x | +| 4 | 1.04x | 1.01x | 1.33x | +| 8 | 1.05x | 1.05x | 1.59x | +| 16 | 1.21x | 1.21x | **3.83x** | +| 32 | 1.05x | 1.13x | **3.99x** | + +**The finding about apportionment survives contact with the real type.** Both +`u64` re-apportionments track the default within noise, including `Perpetual`'s +8/56 -- so buying twenty years of headroom really is free, and it is now +measured on the code that ships rather than on something resembling it. + +**The finding about width did not survive unchanged.** The duplicate reported +the 128-bit exchange at 2.37x and 2.99x at sixteen and thirty-two producers; the +real type reports 3.83x and 3.99x. The stand-in was *understating* the cost of +the layout it was built to evaluate, and by the widest margin exactly where the +decision is most sensitive. The conclusion is unaltered in direction and firmer +in degree. + +**The residual offset is gone, which is the point of the deletion.** The +duplicate ran about 1.26x slower than `reserving_mpsc` at high producer counts, +an error that had to be carried as a caveat on every figure. Running the same +configuration twice through the shipping type now agrees within noise -- 50.3 ns +against 52.1 ns at thirty-two producers -- because both rows are the same code. + +The general lesson is worth keeping even though the duplicate is gone: +**a stand-in is only evidence about the thing it stands in for while something +checks that it still does.** This one was checked, which is how the missing +cache padding was caught; but the checking only ever bounded the error, and the +bound was loose enough to hide a third of the wide word's cost. ## The report is buffered, and what that costs diff --git a/crates/windows-platform-probes/README.md b/crates/windows-platform-probes/README.md index 3bc2d949..d3d0612d 100644 --- a/crates/windows-platform-probes/README.md +++ b/crates/windows-platform-probes/README.md @@ -60,3 +60,20 @@ once set at process scope -- irreversible, so no test performs it. its source, with the control that makes that attributable; that closing a duplicate leaves the source usable; and that single-shot metadata queries do not disturb an enumeration in progress, on the handle or on a duplicate. + +**Queue claim contention, and the claim word's layout.** How `slotwise_mpsc`, +`reserving_mpsc`, and the experimental permit claim scale as producers are +added, against an uncontended `fetch_add` floor, in two regimes: producers alone +so the compare-and-swap is the only thing happening, and producers against a +continuously draining consumer. Reports each run's refusal count, so a run that +was bounded by the consumer rather than by the claim is visible as a fact rather +than mistaken for contention. + +Also measures the four apportionments of `reserving_mpsc`'s claim word -- +32/32, 16/48, 8/56, and the 128-bit 64/64 -- which is what established that +re-apportioning the bits is free while widening the word is not. That decided +how the layouts ship. The probe instantiates the shipping type at each layout +rather than a stand-in, and the reason is recorded in +[DESIGN-NOTES.md](DESIGN-NOTES.md): an earlier version carried its own copy of +the protocol and was found to be *understating* the cost of the layout it +existed to evaluate. diff --git a/crates/windows-platform-probes/src/bin/core_affinity.rs b/crates/windows-platform-probes/src/bin/core_affinity.rs new file mode 100644 index 00000000..ebf3aacf --- /dev/null +++ b/crates/windows-platform-probes/src/bin/core_affinity.rs @@ -0,0 +1,604 @@ +// Copyright (c) Mike Grier. + +//! Prints whether it matters where the two ends of a queue run. + +use std::fmt::Write as _; + +use windows_placement_probe::core_affinity::{Observation, Placement, measure}; +use windows_placement_probe::peer_index_cache::Strategy; +use windows_platform_probes::report::{Stdout, emit}; +use windows_topology_sys::Observed; + +fn main() -> std::io::Result<()> { + // The only place that names the real stream. Everything below composes + // text; nothing below knows where it goes. + emit(&mut Stdout, &render(&measure()?)); + Ok(()) +} + +/// The probe's whole report, as text. +fn render(observation: &Observation) -> String { + let mut out = String::new(); + // First line of the report, and part of the returned text rather than + // written out here: a captured report must carry the line naming the + // machine that produced it, and the taint marker with it. + let _ = writeln!( + out, + "{}", + windows_placement_probe::fingerprint::banner_line() + ); + let _ = writeln!( + out, + "== does it matter where the two ends of a queue run? ==\n" + ); + + let _ = writeln!(out, "processors, as discovered:"); + let _ = writeln!( + out, + " {:>8} {:>16} {:>13}", + "cpu", "efficiency class", "cache domain" + ); + for place in &observation.processors { + // Group and number together: a number is unique only within its group, + // so two distinct processors on a machine with more than 64 of them + // would otherwise both render as `cpu5`. + let _ = writeln!( + out, + " {:>8} {:>16} {:>13}", + format!("g{}/cpu{}", place.group, place.number), + place.efficiency_class, + match place.cache_domain { + Observed::Known(id) => id.to_string(), + Observed::Absent => "none".to_owned(), + Observed::NotObserved => "unknown".to_owned(), + } + ); + } + + let classes: Vec = { + let mut seen: Vec = observation + .processors + .iter() + .map(|p| p.efficiency_class) + .collect(); + seen.sort_unstable(); + seen.dedup(); + seen + }; + let _ = writeln!( + out, + "\n {} efficiency class(es), {} cache domain(s)", + classes.len(), + { + let mut seen: Vec<_> = observation + .processors + .iter() + .map(|p| p.cache_domain) + .collect(); + seen.sort_unstable(); + seen.dedup(); + seen.len() + } + ); + + if !observation.by_class.is_empty() { + let _ = writeln!( + out, + "\n-- the same handoff, within each efficiency class --" + ); + let _ = writeln!( + out, + "{:<12} {:>8} {:>8} {:>12} {:>12} {:>10}", + "class", "prod", "cons", "base ns/it", "cached ns/it", "cach depth" + ); + let mut classes: Vec = observation + .by_class + .iter() + .map(|m| m.producer.efficiency_class) + .collect(); + classes.sort_unstable(); + classes.dedup(); + for class in classes { + let base = observation + .by_class + .iter() + .find(|m| m.producer.efficiency_class == class && m.strategy == Strategy::Baseline); + let cached = observation + .by_class + .iter() + .find(|m| m.producer.efficiency_class == class && m.strategy == Strategy::Cached); + if let (Some(base), Some(cached)) = (base, cached) { + let _ = writeln!( + out, + "{:<12} {:>8} {:>8} {:>12.1} {:>12.1} {:>10.1}", + format!("class {class}"), + format!("g{}/cpu{}", base.producer.group, base.producer.number), + format!("g{}/cpu{}", base.consumer.group, base.consumer.number), + base.nanos_per_item, + cached.nanos_per_item, + cached.consumer_batch + ); + } + } + let _ = writeln!( + out, + " (Windows numbers efficiency classes with the FASTER cores higher, so\n \ + the highest class here is the performance one.)" + ); + } + + let _ = writeln!(out, "\n-- the handoff, by placement --"); + let _ = writeln!( + out, + "{:<26} {:>8} {:>8} {:>12} {:>12} {:>10} {:>10}", + "placement", "prod", "cons", "base ns/it", "cached ns/it", "base depth", "cach depth" + ); + + // Every variant, tightest coupling first. `SameCoreSiblings` MUST be here: + // it is the placement the caching hypothesis is about, and on an SMT host it + // is where the interesting result lives. Omitting it once already produced a + // table that disagreed with the interpretation printed directly beneath it. + let all = [ + Placement::SameCoreSiblings, + Placement::SameCacheSameClass, + Placement::SameCacheCrossClass, + Placement::CrossCacheSameClass, + Placement::CrossCacheCrossClass, + Placement::CrossNumaNode, + ]; + + for placement in all { + let (Some(base), Some(cached)) = ( + observation.get(placement, Strategy::Baseline), + observation.get(placement, Strategy::Cached), + ) else { + // Absent is a finding, not a gap: it means this machine cannot + // express the placement at all. + let _ = writeln!( + out, + "{:<26} {:>8} {:>8} {:>12} {:>12} {:>10} {:>10}", + placement.label(), + "-", + "-", + "n/a", + "n/a", + "-", + "-" + ); + continue; + }; + let _ = writeln!( + out, + "{:<26} {:>8} {:>8} {:>12.1} {:>12.1} {:>10.1} {:>10.1}", + placement.label(), + format!("g{}/cpu{}", base.producer.group, base.producer.number), + format!("g{}/cpu{}", base.consumer.group, base.consumer.number), + base.nanos_per_item, + cached.nanos_per_item, + base.consumer_batch, + cached.consumer_batch + ); + } + + let _ = writeln!(out, "\nthe slice each row was measured on:"); + for placement in all { + if let Some(base) = observation.get(placement, Strategy::Baseline) { + let _ = writeln!(out, " {:<26} {}", placement.label(), base.slice); + } + } + + render_node_distances(&mut out, observation); + + let _ = writeln!( + out, + " +interpretation: +" + ); + + let expressible = observation.placements(); + if expressible.len() < 2 { + let _ = writeln!( + out, + " This machine expresses only one placement, so it cannot answer" + ); + let _ = writeln!( + out, + " the question. That is a fact about the host, not a null result:" + ); + let _ = writeln!( + out, + " a homogeneous single-cache machine has nowhere else to put the" + ); + let _ = writeln!(out, " two threads."); + return out; + } + + // Whether the two factors can be told apart at all on this host. If every + // cross-class pair is also cross-cache, they are perfectly confounded and + // no amount of measurement here separates them -- which is a fact to state, + // not to reason past. + let confounded = !expressible.contains(&Placement::SameCacheCrossClass) + && !expressible.contains(&Placement::CrossCacheSameClass); + if confounded { + let _ = writeln!( + out, + " CAUTION: on this machine the efficiency classes and the cache" + ); + let _ = writeln!( + out, + " domains coincide exactly, so every cross-class pair is also a" + ); + let _ = writeln!( + out, + " cross-cache pair. The two effects are perfectly CONFOUNDED here" + ); + let _ = writeln!( + out, + " and nothing below separates them. Read the rows as 'within a" + ); + let _ = writeln!( + out, + " domain' versus 'across domains', and do not attribute the" + ); + let _ = writeln!( + out, + " difference to core speed or to cache without a machine whose" + ); + let _ = writeln!(out, " classes and caches cut differently.\n"); + } + + // Batch depth is read from the CACHED runs, never the baseline ones. + // Baseline reads the shared line on every operation by definition, so its + // depth is ~1 whatever the placement and carries no information at all. An + // earlier version of this probe compared the baseline depths and duly + // reported ~0.8 against ~0.4, which is noise around a constant being read + // as a finding. + let same_class: Vec<_> = expressible + .iter() + .filter(|p| { + matches!( + p, + Placement::SameCacheSameClass | Placement::CrossCacheSameClass + ) + }) + .filter_map(|p| observation.get(*p, Strategy::Cached)) + .collect(); + let cross_class: Vec<_> = expressible + .iter() + .filter(|p| { + matches!( + p, + Placement::SameCacheCrossClass | Placement::CrossCacheCrossClass + ) + }) + .filter_map(|p| observation.get(*p, Strategy::Cached)) + .collect(); + + let mean = |runs: &[_], f: fn(&_) -> f64| -> Option { + if runs.is_empty() { + None + } else { + Some(runs.iter().map(f).sum::() / runs.len() as f64) + } + }; + + if let (Some(same), Some(cross)) = ( + mean( + &same_class, + |m: &windows_placement_probe::core_affinity::Measurement| m.consumer_batch, + ), + mean( + &cross_class, + |m: &windows_placement_probe::core_affinity::Measurement| m.consumer_batch, + ), + ) { + let within = if confounded { + "within a domain " + } else { + "same-class " + }; + let across = if confounded { + "across domains " + } else { + "cross-class " + }; + let _ = writeln!( + out, + " batch depth with caching on, {within}: {same:.1} items per shared read" + ); + let _ = writeln!( + out, + " batch depth with caching on, {across}: {cross:.1} items per shared read" + ); + if cross > same * 2.0 { + let _ = writeln!( + out, + "\n SEPARATION DEEPENS THE BATCH. The two sides decouple: one runs" + ); + let _ = writeln!( + out, + " ahead, a real backlog forms, and each shared read is amortised" + ); + let _ = writeln!( + out, + " over it. That is the condition peer-index caching needs, and it" + ); + let _ = writeln!( + out, + " is a property of PLACEMENT -- not of the architecture." + ); + } else if same > cross * 2.0 { + let _ = writeln!( + out, + "\n THE HYPOTHESIS IS REFUTED, AND BACKWARDS. Threads placed" + ); + let _ = writeln!( + out, + " TOGETHER batch {:.0}x deeper than threads placed apart, where the", + same / cross.max(0.001) + ); + let _ = writeln!( + out, + " prediction was the reverse -- that mismatched cores would" + ); + let _ = writeln!(out, " decouple and batch deeply."); + let _ = writeln!( + out, + " A coherent reading: a cheap handoff lets the producer race ahead" + ); + let _ = writeln!( + out, + " and build a backlog, while an expensive one throttles it into" + ); + let _ = writeln!( + out, + " lockstep, so each side arrives to find exactly one item. Cost" + ); + let _ = writeln!( + out, + " drives depth, rather than depth being set by core speed." + ); + let _ = writeln!( + out, + " That is a hypothesis this run does not test, and it must not be" + ); + let _ = writeln!( + out, + " recorded as a finding -- what IS established is that the" + ); + let _ = writeln!(out, " original prediction is wrong."); + } else { + let _ = writeln!( + out, + "\n Placement does NOT move batch depth here ({:.2}x).", + cross / same + ); + let _ = writeln!( + out, + " The hypothesis that unequal core speeds drive the batching is" + ); + let _ = writeln!( + out, + " not supported, and the difference between hosts needs another" + ); + let _ = writeln!( + out, + " explanation. Recording a refutation is the point of running it." + ); + } + } + + // The plainest answer to "does placement matter", independent of caching. + // "Near" falls back to SMT siblings, because a host whose outermost + // partitioning cache is per-core has no same-cache-different-core pair at + // all -- its nearest expressible placement IS the sibling pair. + if let (Some(near), Some(far)) = ( + observation + .get(Placement::SameCacheSameClass, Strategy::Baseline) + .or_else(|| observation.get(Placement::SameCoreSiblings, Strategy::Baseline)), + observation + .get(Placement::CrossCacheCrossClass, Strategy::Baseline) + .or_else(|| observation.get(Placement::CrossCacheSameClass, Strategy::Baseline)), + ) { + let _ = writeln!( + out, + "\n the unoptimised handoff costs {:.1} ns/item together and {:.1} ns/item", + near.nanos_per_item, far.nanos_per_item + ); + let _ = writeln!( + out, + " apart -- {:.1}x for crossing the boundary, with no code change.", + far.nanos_per_item / near.nanos_per_item + ); + } + + let _ = writeln!( + out, + "\n does the verdict on caching depend on placement?\n" + ); + let mut verdicts = Vec::new(); + for placement in expressible { + let (Some(base), Some(cached)) = ( + observation.get(placement, Strategy::Baseline), + observation.get(placement, Strategy::Cached), + ) else { + continue; + }; + let speedup = base.nanos_per_item / cached.nanos_per_item; + let verdict = if speedup >= 1.1 { + "caching WINS" + } else if speedup <= 0.9 { + "caching LOSES" + } else { + "no effect" + }; + let _ = writeln!( + out, + " {:<26} {:>7.2}x {verdict}", + placement.label(), + speedup + ); + verdicts.push(verdict); + } + verdicts.sort_unstable(); + verdicts.dedup(); + + if verdicts.len() > 1 { + let _ = writeln!( + out, + "\n THE VERDICT FLIPS WITHIN ONE MACHINE. A technique whose sign" + ); + let _ = writeln!( + out, + " depends on where two threads are scheduled cannot be adopted or" + ); + let _ = writeln!( + out, + " rejected by a fixed decision. Any answer has to name the" + ); + let _ = writeln!(out, " placement it holds for."); + } else { + let _ = writeln!( + out, + "\n The verdict is the same at every placement on this host, so" + ); + let _ = writeln!( + out, + " placement alone does not explain the disagreement between hosts." + ); + } + + out +} + +/// Print the per-node-pair handoff cost, when the host has nodes to cross. +/// +/// Silent on a single-node machine: there is nothing to say, and a header over +/// an empty table invites the reader to wonder what went wrong. +fn render_node_distances(out: &mut String, observation: &Observation) { + let pairs = observation.node_pairs_measured(); + if pairs.is_empty() { + return; + } + + let _ = writeln!(out, "\n-- the handoff, by NUMA node pair --"); + // A ring-placement column, because a pair and a strategy no longer identify + // one row: every hop is measured once with the ring on the producer's node + // and once on the consumer's. Rendering one of them would drop half the + // measurements and, worse, could pair a baseline taken at one placement + // against a cached run taken at the other. + let _ = writeln!( + out, + "{:<14} {:>8} {:>8} {:>8} {:>12} {:>12} {:>10}", + "prod -> cons", "ring on", "prod", "cons", "base ns/it", "cached ns/it", "cach depth" + ); + // Stated rather than left as a mystery glyph. `ring on` names the node the + // run asked for, since that is what identifies the row; a `!` means the + // memory did not land there, so that row does not measure the placement it + // names. + let _ = writeln!( + out, + " (`ring on` is the node requested; `!` means it landed elsewhere)" + ); + + let mut slowest: Option<(f64, (u32, u32))> = None; + let mut fastest: Option<(f64, (u32, u32))> = None; + + for pair in &pairs { + for base in observation.node_pair_rows(*pair, Strategy::Baseline) { + // Matched on the ring placement as well, so the two columns + // describe the same configuration -- and on the placement that was + // *requested*, not the one that was achieved. Windows may redirect + // an allocation, so two rows can share an achieved node while + // describing different placements; keyed on that, this pairs a + // baseline taken at one placement against a cached run taken at the + // other, which is the exact error the comment above says the key + // exists to prevent. + let Some(cached) = + observation.node_pair(*pair, Strategy::Cached, base.requested_memory_node) + else { + continue; + }; + let _ = writeln!( + out, + "{:<14} {:>8} {:>8} {:>8} {:>12.1} {:>12.1} {:>10.1}", + // `->`, not `<->`: hops are directed, because the producer + // writes and the consumer reads. The probe crate's own report + // was corrected for this and this second renderer of the same + // data was not, which is how two views of one measurement drift + // apart. + format!("{} -> {}", pair.0, pair.1), + // The requested node, matching the key above and the probe + // crate's own report. A trailing `!` marks a row whose memory + // did not land where it was asked to go, so a redirected run + // is not read as a measurement of the placement it names. + match (base.requested_memory_node, base.memory_node) { + (Some(asked), Some(got)) if asked == got => format!("node {asked}"), + (Some(asked), _) => format!("node {asked}!"), + (None, _) => "unspecified".to_owned(), + }, + format!("g{}/cpu{}", base.producer.group, base.producer.number), + format!("g{}/cpu{}", base.consumer.group, base.consumer.number), + base.nanos_per_item, + cached.nanos_per_item, + cached.consumer_batch + ); + let seen = (base.nanos_per_item, *pair); + if slowest.is_none_or(|(worst, _)| seen.0 > worst) { + slowest = Some(seen); + } + if fastest.is_none_or(|(best, _)| seen.0 < best) { + fastest = Some(seen); + } + } + } + + if pairs.len() == 1 { + let _ = writeln!( + out, + "\n One node pair, so this restates the `cross NUMA node` row above\n \ + rather than adding to it. The table earns its place from three\n \ + nodes upward, where the hops stop being interchangeable." + ); + return; + } + + let (Some((worst, worst_pair)), Some((best, best_pair))) = (slowest, fastest) else { + return; + }; + let _ = writeln!( + out, + "\n {} node pairs. Cheapest hop {} <-> {} at {:.1} ns/item; dearest\n \ + {} <-> {} at {:.1} ns/item -- a spread of {:.1}x.", + pairs.len(), + best_pair.0, + best_pair.1, + best, + worst_pair.0, + worst_pair.1, + worst, + worst / best + ); + if worst / best < 1.2 { + let _ = writeln!( + out, + " That spread is small enough that this host's nodes are close to\n \ + equidistant, so the single `cross NUMA node` row above is a fair\n \ + summary of it." + ); + } else { + let _ = writeln!( + out, + " The hops are NOT interchangeable, so the single `cross NUMA node`\n \ + row above reports whichever one was enumerated first and should not\n \ + be read as 'the' cost of leaving a node." + ); + } + let _ = writeln!( + out, + " This measures the handoff between two nodes; it is not a distance\n \ + matrix read from firmware. Windows exposes no NUMA distance table, so\n \ + these numbers are the observable rather than a restatement of ACPI." + ); +} diff --git a/crates/windows-platform-probes/src/bin/doorbell_cost.rs b/crates/windows-platform-probes/src/bin/doorbell_cost.rs new file mode 100644 index 00000000..80d0c429 --- /dev/null +++ b/crates/windows-platform-probes/src/bin/doorbell_cost.rs @@ -0,0 +1,193 @@ +// Copyright (c) Mike Grier. + +//! Prints how expensive a doorbell is relative to the syscall it would guard. +//! +//! **An experiment, not a component.** These probes measure platform behaviour +//! and are not for production use. Do not call them from production code, and +//! do not lift a technique out of here. See this crate's DESIGN-NOTES.md. +//! +//! This decides whether the two-layer ring design needs an eventcount at all. +//! If a doorbell is a meaningful fraction of `SubmitIoRing`, the skip-when-busy +//! rules are load-bearing. If it is noise, a simple always-signal queue is +//! adequate and the more delicate protocol -- publish intent, re-check, park -- +//! can wait for evidence that it is worth its lost-wakeup risk. + +use std::fmt::Write as _; +use windows_platform_probes::doorbell_cost::{Observation, measure, measure_park_and_wake}; + +use windows_platform_probes::report::{Stdout, emit}; + +fn main() { + // The only place that names the real stream. Everything above composes + // text; nothing above knows where it goes. + emit( + &mut Stdout, + &render(&measure(), measure_park_and_wake(20_000)), + ); +} + +/// The probe's whole report, as text. +fn render(observation: &Observation, park: Option) -> String { + let mut out = String::new(); + // First line of the report, and part of the returned text rather than + // written out here: a captured report must carry the line naming the + // machine that produced it, and the taint marker with it. Without it a + // timing number can be pasted anywhere and compared against anything. + let _ = writeln!( + out, + "{}", + windows_placement_probe::fingerprint::banner_line() + ); + let _ = writeln!( + out, + "== what does a doorbell cost, against the syscall it guards? ==\n" + ); + + let _ = writeln!(out, "{:<30} {:>12}", "operation", "ns/op"); + for timing in &observation.timings { + let _ = writeln!(out, "{:<30} {:>12.1}", timing.label, timing.nanos_per_op); + } + + match park { + Some(ns) => { + let _ = writeln!(out, "{:<30} {:>12.1}", "park_and_wake round trip", ns); + } + None => { + let _ = writeln!( + out, + "{:<30} {:>12}", + "park_and_wake round trip", "TIMED OUT" + ); + } + } + + let _ = writeln!(out, "\ninterpretation:"); + + if let Some(atomic) = observation.get("atomic_fetch_add") + && let Some(doorbell) = observation.get("set_reset_event") + && atomic > 0.0 + { + let _ = writeln!( + out, + " a doorbell cycle costs {:.0}x an uncontended atomic ({:.0} ns vs {:.1} ns).", + doorbell / atomic, + doorbell, + atomic + ); + if let Some(park) = park { + let _ = writeln!( + out, + " an actual park-and-wake round trip costs {:.0}x that again ({:.0} ns),", + park / doorbell, + park + ); + let _ = writeln!( + out, + " which is what is paid when the consumer genuinely sleeps." + ); + } + } + + // Deliberately NOT expressed as a share of the empty submit. See below. + if let Some(submit) = observation.submit_nanos { + let _ = writeln!( + out, + "\n CAUTION: an empty SubmitIoRing measured {submit:.0} ns, which is far too" + ); + let _ = writeln!( + out, + " cheap for a kernel transition -- it is almost certainly short-" + ); + let _ = writeln!( + out, + " circuiting in user mode when there is nothing queued. It is" + ); + let _ = writeln!( + out, + " therefore NOT a fair denominator, and any 'doorbell is N% of a" + ); + let _ = writeln!( + out, + " syscall' figure derived from it would be a confident wrong answer." + ); + let _ = writeln!( + out, + " The honest denominator is the cost of the real work a submission" + ); + let _ = writeln!(out, " carries, which this probe does not measure."); + } + + // What can be said without a denominator: how much batching it takes for + // the doorbell to disappear, which is the lever the design actually has. + if let Some(doorbell) = observation.get("set_reset_event") + && let Some(atomic) = observation.get("atomic_fetch_add") + && atomic > 0.0 + { + let _ = writeln!( + out, + "\n batching is the lever, and it is a strong one. One doorbell per" + ); + let _ = writeln!(out, " drained batch costs, per operation:"); + for batch in [1_u32, 8, 32, 128] { + let _ = writeln!( + out, + " batch of {batch:>4}: {:>7.1} ns/op ({:.1}x an atomic)", + doorbell / f64::from(batch), + doorbell / f64::from(batch) / atomic + ); + } + let break_even = (doorbell / atomic).ceil() as u32; + let _ = writeln!( + out, + " so at a batch of about {break_even}, the doorbell costs less per" + ); + let _ = writeln!(out, " operation than the atomic push it accompanies."); + } + + let _ = writeln!( + out, + "\n => The skip-when-busy rule is a refinement, not a prerequisite." + ); + let _ = writeln!( + out, + " Batching alone drives the doorbell below the cost of the push," + ); + let _ = writeln!( + out, + " so a first implementation can always-signal and stay honest." + ); + let _ = writeln!( + out, + " Adopt the eventcount when a measurement against real work" + ); + let _ = writeln!(out, " justifies its lost-wakeup risk -- not before."); + + let atomic = observation.get("atomic_fetch_add").unwrap_or(f64::NAN); + let already = observation + .get("set_event_already_signalled") + .unwrap_or(f64::NAN); + let cycle = observation.get("set_reset_event").unwrap_or(f64::NAN); + let wait0 = observation.get("wait_zero_signalled").unwrap_or(f64::NAN); + let _ = writeln!( + out, + concat!( + r#"{{"reason":"x-probe-doorbell-cost","arch":"{}","atomic_ns":{:.1},"#, + r#""set_event_already_signalled_ns":{:.1},"set_reset_event_ns":{:.1},"#, + r#""wait_zero_signalled_ns":{:.1},"park_and_wake_round_trip_ns":{},"#, + r#""submit_io_ring_empty_ns":{},"doorbell_share_of_submit":{}}}"# + ), + std::env::consts::ARCH, + atomic, + already, + cycle, + wait0, + park.map_or("null".to_string(), |n| format!("{n:.1}")), + observation + .submit_nanos + .map_or("null".to_string(), |n| format!("{n:.1}")), + observation + .doorbell_share_of_submit() + .map_or("null".to_string(), |s| format!("{s:.4}")), + ); + out +} diff --git a/crates/windows-platform-probes/src/bin/peer_index_cache.rs b/crates/windows-platform-probes/src/bin/peer_index_cache.rs new file mode 100644 index 00000000..f4a5185b --- /dev/null +++ b/crates/windows-platform-probes/src/bin/peer_index_cache.rs @@ -0,0 +1,315 @@ +// Copyright (c) Mike Grier. + +//! Prints what caching the peer's index buys an SPSC ring. +//! +//! **An experiment, not a component.** These probes measure platform behaviour +//! and are not for production use. Do not call them from production code, and +//! do not lift a technique out of here. See this crate's DESIGN-NOTES.md. + +use std::fmt::Write as _; +use windows_placement_probe::peer_index_cache::{CAPACITY, ITEMS, Strategy, measure}; +use windows_platform_probes::report::{Stdout, emit}; + +fn main() { + // The only place that names the real stream. Everything below composes + // text; nothing below knows where it goes. + emit(&mut Stdout, &render()); +} + +/// The probe's whole report, as text. +fn render() -> String { + let mut out = String::new(); + // First line of the report, and part of the returned text rather than + // written out here: a captured report must carry the line naming the + // machine that produced it, and the taint marker with it. + let _ = writeln!( + out, + "{}", + windows_placement_probe::fingerprint::banner_line() + ); + let _ = writeln!( + out, + "== what does caching the peer's index buy an SPSC ring? ==\n" + ); + + let observation = measure(); + + let _ = writeln!( + out, + "{:<24} {:>10} {:>14} {:>14} {:>14}", + "configuration", "ns/item", "items/sec", "cons. reads", "prod. reads" + ); + for run in std::iter::once(&observation.calibration).chain(&observation.strategies) { + let _ = writeln!( + out, + "{:<24} {:>10.1} {:>14.0} {:>14} {:>14}", + run.label, + run.nanos_per_item, + run.items_per_second, + run.consumer_refreshes, + run.producer_refreshes + ); + } + let _ = writeln!( + out, + " + ({ITEMS} items, capacity {CAPACITY}. The two read columns count how often each + \ + side actually loaded the *other* side's position -- the shared line the + \ + technique exists to avoid touching.)" + ); + + let _ = writeln!(out, "\ninterpretation:\n"); + + let Some(baseline) = observation.get(Strategy::Baseline) else { + return out; + }; + + // The model has to reproduce the shipping queue before anything it says + // about variants is worth reading. + let drift = (baseline.nanos_per_item - observation.calibration.nanos_per_item).abs() + / observation.calibration.nanos_per_item; + let _ = writeln!( + out, + " calibration: the model's baseline differs from the shipping spsc by + \ + {:.0}% ({:.1} vs {:.1} ns/item).", + drift * 100.0, + baseline.nanos_per_item, + observation.calibration.nanos_per_item + ); + if drift > 0.25 { + let _ = writeln!( + out, + " CAUTION: that is a wide gap, so the rows below describe the MODEL" + ); + let _ = writeln!( + out, + " and not the shipping queue. The model has only the ring mechanics;" + ); + let _ = writeln!( + out, + " the shipping push also consults the reservation count, updates the" + ); + let _ = writeln!( + out, + " depth metric and rings the doorbell. This probe does NOT attribute" + ); + let _ = writeln!( + out, + " the gap between those, and no such attribution should be read into" + ); + let _ = writeln!( + out, + " it. What the gap does establish is a floor: whatever the shared" + ); + let _ = writeln!( + out, + " read costs, it is a minority of what the shipping queue spends per" + ); + let _ = writeln!(out, " item, so removing it cannot be the large win."); + } else { + let _ = writeln!( + out, + " Close enough to treat the model as a stand-in for the real ring." + ); + } + + for strategy in [Strategy::Cached, Strategy::Warmed] { + let Some(run) = observation.get(strategy) else { + continue; + }; + let speedup = baseline.nanos_per_item / run.nanos_per_item; + let _ = writeln!( + out, + "\n {:<22} {:.2}x the baseline ({:.1} -> {:.1} ns/item)", + match strategy { + Strategy::Cached => "peer-index caching:", + Strategy::Warmed => "warming load only:", + Strategy::Baseline => unreachable!(), + }, + speedup, + baseline.nanos_per_item, + run.nanos_per_item + ); + } + + // Everything below is DERIVED from this run's numbers, and none of it may + // go back to being prose. + // + // It used to be a fixed paragraph concluding that the technique "WORKED and + // still lost", that consumer reads fell "roughly 3.6x", and that producer + // reads "go UP". Those were true of the x64 host it was written on. Run on + // an ARM64 host they were all three false -- caching was 17x FASTER, and + // producer reads fell by ~580x -- and the probe printed the old conclusion + // anyway, contradicting the table directly above it. An instrument that + // states its finding regardless of what it measured is worse than no + // instrument, because it is believed. + let Some(cached) = observation.get(Strategy::Cached) else { + return out; + }; + + // The batch depth is the mechanism, so compute it rather than assert it: it + // is how many items each shared read is amortised over, and it is what + // decides whether trading freshness for fewer reads pays. + let consumer_batch = ITEMS as f64 / cached.consumer_refreshes.max(1) as f64; + let producer_batch = ITEMS as f64 / cached.producer_refreshes.max(1) as f64; + let consumer_reduction = + baseline.consumer_refreshes as f64 / cached.consumer_refreshes.max(1) as f64; + let producer_reduction = + baseline.producer_refreshes as f64 / cached.producer_refreshes.max(1) as f64; + let speedup = baseline.nanos_per_item / cached.nanos_per_item; + + let _ = writeln!(out); + let _ = writeln!( + out, + " how far each shared read was amortised, with caching on:" + ); + let _ = writeln!( + out, + " consumer: {consumer_batch:.1} items per read ({consumer_reduction:.1}x fewer reads than baseline)" + ); + let _ = writeln!( + out, + " producer: {producer_batch:.1} items per read ({producer_reduction:.1}x fewer reads than baseline)" + ); + let _ = writeln!(out); + + let engaged = consumer_reduction > 1.5; + if !engaged { + let _ = writeln!( + out, + " The technique did NOT engage: the consumer's shared reads barely" + ); + let _ = writeln!( + out, + " moved. Any throughput difference below is noise about something" + ); + let _ = writeln!(out, " else, and says nothing about peer-index caching."); + } else if speedup >= 1.1 { + let _ = writeln!(out, " The technique engaged AND won, by {speedup:.2}x."); + let _ = writeln!( + out, + " Peer-index caching trades freshness for fewer reads, and that" + ); + let _ = writeln!( + out, + " trade pays when the batch it amortises over is deep. At the" + ); + let _ = writeln!(out, " depths above it is paying."); + } else if speedup <= 0.9 { + let _ = writeln!( + out, + " The technique engaged and still LOST, at {speedup:.2}x the baseline." + ); + let _ = writeln!( + out, + " This is a real result about the shape rather than a failed" + ); + let _ = writeln!( + out, + " implementation. Caching trades freshness for fewer reads; at the" + ); + let _ = writeln!( + out, + " batch depths above, each side idles on a stale bound it could" + ); + let _ = writeln!( + out, + " have refreshed, and that idling costs more than the reads saved." + ); + if producer_reduction < 1.0 { + let _ = writeln!( + out, + " Note the producer count went UP: a cached index is consulted" + ); + let _ = writeln!( + out, + " only when it says 'no room', so a blocked producer refreshes on" + ); + let _ = writeln!(out, " every spin and gains nothing."); + } + } else { + let _ = writeln!( + out, + " The technique engaged and changed throughput by {speedup:.2}x, which" + ); + let _ = writeln!( + out, + " is inside the noise of this probe. Treat it as no effect." + ); + } + + let _ = writeln!(out); + let _ = writeln!( + out, + " BATCH DEPTH IS THE VARIABLE, AND IT IS NOT A CONSTANT OF THE CODE." + ); + let _ = writeln!( + out, + " It depends on how the producer and consumer interleave, which" + ); + let _ = writeln!( + out, + " depends on the host: core count, whether siblings share a core," + ); + let _ = writeln!( + out, + " and how the scheduler places the two threads. The same binary has" + ); + let _ = writeln!( + out, + " measured a depth near 1 on one machine and in the hundreds on" + ); + let _ = writeln!( + out, + " another, and the verdict inverted with it. Do not carry a" + ); + let _ = writeln!( + out, + " conclusion from one host to another -- run it on the host you" + ); + let _ = writeln!(out, " intend to make the decision for."); + + let Some(warmed) = observation.get(Strategy::Warmed) else { + return out; + }; + let warm_reduction = + baseline.consumer_refreshes as f64 / warmed.consumer_refreshes.max(1) as f64; + let _ = writeln!(out); + let _ = writeln!( + out, + " control (warming load): {:.2}x throughput, {:.2}x fewer consumer reads.", + baseline.nanos_per_item / warmed.nanos_per_item, + warm_reduction + ); + if warm_reduction < 1.5 { + let _ = writeln!( + out, + " It removed no shared read, which is what a control should do. A" + ); + let _ = writeln!( + out, + " discarded load cannot help: the authoritative load still happens," + ); + let _ = writeln!( + out, + " and in a tight handoff loop the prefetch has no time to land." + ); + let _ = writeln!( + out, + " So the technique works by REMOVING the load, not by warming it." + ); + } else { + let _ = writeln!( + out, + " UNEXPECTED: the control removed shared reads, so it is not acting" + ); + let _ = writeln!( + out, + " as a control. Distrust the comparison above until that is explained." + ); + } + out +} diff --git a/crates/windows-platform-probes/src/bin/queue_contention.rs b/crates/windows-platform-probes/src/bin/queue_contention.rs new file mode 100644 index 00000000..bc812b07 --- /dev/null +++ b/crates/windows-platform-probes/src/bin/queue_contention.rs @@ -0,0 +1,328 @@ +// Copyright (c) Mike Grier. + +//! Prints how the array queue's tail claim behaves as producers are added. +//! +//! **An experiment, not a component.** These probes measure platform behaviour +//! and are not for production use. Do not call them from production code, and +//! do not lift a technique out of here. See this crate's DESIGN-NOTES.md. +//! +//! This decides two things that are otherwise decided by taste: whether the +//! linked and sharded MPSC shapes are ever needed, and whether `mpsc` and +//! `reserving_mpsc` should merge. See `queue_contention`'s module docs. + +use std::fmt::Write as _; +use windows_platform_probes::queue_contention::{PRODUCER_COUNTS, Run, measure, shapes}; +use windows_platform_probes::report::{Stdout, emit}; + +fn main() { + // The only place that names the real stream. Everything below composes + // text; nothing below knows where it goes. + emit(&mut Stdout, &render()); +} + +/// The probe's whole report, as text. +fn render() -> String { + let mut out = String::new(); + // First line of the report, and part of the returned text rather than + // written out here: a captured report must carry the line naming the + // machine that produced it, and the taint marker with it. + let _ = writeln!( + out, + "{}", + windows_placement_probe::fingerprint::banner_line() + ); + let _ = writeln!(out, "== does the array queue's tail claim contend? ==\n"); + + let observation = measure(); + let _ = writeln!( + out, + "host reports {} logical processors\n", + observation.logical_processors + ); + + let _ = writeln!( + out, + "-- isolated: producers only, capacity large enough that nothing is refused --" + ); + render_table(&mut out, &observation.isolated); + + let _ = writeln!( + out, + "\n-- drained: a consumer popping continuously, capacity 1024 --" + ); + render_table(&mut out, &observation.drained); + + let _ = writeln!(out, "\ninterpretation:\n"); + + // Question 1: does the claim collapse as producers are added? + let _ = writeln!(out, " 1. tail-claim contention (isolated regime)\n"); + let _ = writeln!( + out, + " {:<18} {:>12} {:>12} {:>12} {:>14}", + "producers", "slotwise x1thr", "reserving", "permit", "atomic floor" + ); + for &producers in PRODUCER_COUNTS { + let mpsc = observation.scaling(&observation.isolated, shapes::SLOTWISE_MPSC, producers); + let reserving = + observation.scaling(&observation.isolated, shapes::RESERVING_MPSC, producers); + let permit = observation.scaling(&observation.isolated, shapes::PERMIT_MPSC, producers); + let floor = + observation.scaling(&observation.isolated, shapes::BASELINE_FETCH_ADD, producers); + let _ = writeln!( + out, + " {producers:<18} {:>12} {:>12} {:>12} {:>14}", + format_scaling(mpsc), + format_scaling(reserving), + format_scaling(permit), + format_scaling(floor) + ); + } + let _ = writeln!( + out, + "\n Read as: throughput at N producers divided by throughput at one." + ); + let _ = writeln!( + out, + " 1.00 means N threads together push no faster than one did." + ); + let _ = writeln!( + out, + " The atomic floor is the cheapest possible contended operation," + ); + let _ = writeln!( + out, + " so it says how much of any curve is the queue and how much is" + ); + let _ = writeln!( + out, + " simply what this processor does to a fought-over cache line." + ); + + // Question 2: what does reserving_mpsc's read of `head` actually cost? + let _ = writeln!( + out, + "\n 2. the price of reservation (drained regime, where `head` is written)\n" + ); + let _ = writeln!( + out, + " {:<18} {:>14} {:>14} {:>10} {:>14} {:>16}", + "producers", "slotwise ns/pu", "reserving", "ratio", "permit", "permit/reserving" + ); + for &producers in PRODUCER_COUNTS { + let plain = observation.find(&observation.drained, shapes::SLOTWISE_MPSC, producers); + let reserving = observation.find(&observation.drained, shapes::RESERVING_MPSC, producers); + let permit = observation.find(&observation.drained, shapes::PERMIT_MPSC, producers); + let ratio = format_ratio(reserving, plain); + // The column SH-15.5 exists to fill: the experimental claim against the + // shipping shape it would replace. Below 1.00 means the permit claim is + // cheaper; above means removing the room-decision race costs throughput. + let permit_ratio = format_ratio(permit, reserving); + let _ = writeln!( + out, + " {producers:<18} {:>14} {:>14} {:>10} {:>14} {:>16}", + format_nanos(plain), + format_nanos(reserving), + ratio, + format_nanos(permit), + permit_ratio + ); + } + let _ = writeln!( + out, + "\n `reserving_mpsc` reads the consumer's position on every push and" + ); + let _ = writeln!( + out, + " `mpsc` does not, which is the entire reason they ship as two" + ); + let _ = writeln!( + out, + " shapes. This regime is the one that can price that read, because" + ); + let _ = writeln!(out, " a consumer is writing the line being read."); + let _ = writeln!( + out, + "\n `permit_mpsc` is experimental and is the candidate replacement" + ); + let _ = writeln!( + out, + " for `reserving_mpsc`: it removes that read entirely, and with it" + ); + let _ = writeln!( + out, + " the stale room decision behind SH-14.1, by making admission a" + ); + let _ = writeln!( + out, + " read-modify-write on a permit count instead. The last column is" + ); + let _ = writeln!( + out, + " the trade -- below 1.00 and the safer claim is also the cheaper" + ); + let _ = writeln!( + out, + " one; above 1.00 and closing the hole costs throughput." + ); + + // Question 3: what does the claim word's apportionment and width cost? + let _ = writeln!(out, "\n 3. claim-word layout\n"); + let _ = writeln!( + out, + " Four apportionments of reserving_mpsc's claim word, measured on" + ); + let _ = writeln!( + out, + " the shipping type itself rather than on a stand-in. 32/32 is the" + ); + let _ = writeln!( + out, + " default; 16/48 and 8/56 are the same u64 exchange with the bits" + ); + let _ = writeln!( + out, + " apportioned differently; 64/64 is a u128 exchange (cmpxchg16b)." + ); + let _ = writeln!( + out, + " The three u64 rows issue the SAME instruction, so a difference" + ); + let _ = writeln!( + out, + " between them is noise or slot-metadata density, not the claim." + ); + let _ = writeln!( + out, + " 64/64 vs 32/32 prices the double-width exchange -- what removing" + ); + let _ = writeln!( + out, + " the recurrence outright costs, against 8/56 merely deferring it.\n" + ); + for (label, regime) in [ + ("isolated", &observation.isolated), + ("drained", &observation.drained), + ] { + let _ = writeln!(out, " -- {label} --"); + let _ = writeln!( + out, + " {:<10} {:>11} {:>11} {:>11} {:>11} {:>10} {:>10} {:>10}", + "producers", + "32/32 ns", + "16/48 ns", + "8/56 ns", + "64/64 ns", + "16/48 vs", + "8/56 vs", + "64/64 vs" + ); + for &producers in PRODUCER_COUNTS { + let narrow = observation.find(regime, shapes::CLAIM_NARROW, producers); + let deep = observation.find(regime, shapes::CLAIM_DEEP, producers); + let perpetual = observation.find(regime, shapes::CLAIM_PERPETUAL, producers); + let wide = observation.find(regime, shapes::CLAIM_WIDE, producers); + let _ = writeln!( + out, + " {:<10} {:>11} {:>11} {:>11} {:>11} {:>10} {:>10} {:>10}", + producers, + format_nanos(narrow), + format_nanos(deep), + format_nanos(perpetual), + format_nanos(wide), + format_ratio(deep, narrow), + format_ratio(perpetual, narrow), + format_ratio(wide, narrow) + ); + } + let _ = writeln!(out); + } + let _ = writeln!( + out, + " the 32/32 row and the reserving_mpsc row above are the same" + ); + let _ = writeln!( + out, + " configuration run twice, so they should agree within noise. They" + ); + let _ = writeln!( + out, + " are no longer a control against a duplicated implementation: the" + ); + let _ = writeln!( + out, + " shipping type takes the layout as a parameter, so there is nothing" + ); + let _ = writeln!( + out, + " left that could drift away from what callers actually run." + ); + let _ = writeln!( + out, + "\n CAUTION: the drained regime has ONE consumer, because that is what" + ); + let _ = writeln!( + out, + " MPSC means. At high producer counts it is expected to become" + ); + let _ = writeln!( + out, + " consumer-bound, and a plateau there says nothing about the claim." + ); + let _ = writeln!( + out, + " The refusal counts above are what make that visible: a run with" + ); + let _ = writeln!( + out, + " many refusals was waiting for the consumer, not for the tail." + ); + out +} + +/// Append one regime's table to `out`. +/// +/// Takes the buffer rather than printing, for the reason `pool_growth`'s twin +/// records: a helper writing to stdout while its caller composes a string emits +/// its lines first, reordering the report without losing any of it. +fn render_table(out: &mut String, runs: &[Run]) { + let _ = writeln!( + out, + "{:<18} {:>10} {:>14} {:>16} {:>14}", + "shape", "producers", "ns/push", "pushes/sec", "refusals" + ); + for run in runs { + let _ = writeln!( + out, + "{:<18} {:>10} {:>14.1} {:>16.0} {:>14}", + run.shape, run.producers, run.nanos_per_push, run.pushes_per_second, run.refusals + ); + } +} + +fn format_scaling(scaling: Option) -> String { + scaling.map_or_else(|| "--".to_owned(), |value| format!("{value:.2}x")) +} + +/// `numerator / denominator` as a cost ratio, or `--` when either is missing. +/// +/// Guards the denominator rather than trusting it: a shape that failed to run +/// reports zero, and a division by it would print `inf` or `NaN` in a column a +/// reader would otherwise take for a measurement. +fn format_ratio(numerator: Option, denominator: Option) -> String { + match (numerator, denominator) { + (Some(numerator), Some(denominator)) if denominator.nanos_per_push > 0.0 => { + format!( + "{:.2}x", + numerator.nanos_per_push / denominator.nanos_per_push + ) + } + _ => "--".to_owned(), + } +} + +fn format_nanos(run: Option) -> String { + run.map_or_else( + || "--".to_owned(), + |run| format!("{:.1}", run.nanos_per_push), + ) +} diff --git a/crates/windows-platform-probes/src/bin/request_cost.rs b/crates/windows-platform-probes/src/bin/request_cost.rs new file mode 100644 index 00000000..1c65ceb1 --- /dev/null +++ b/crates/windows-platform-probes/src/bin/request_cost.rs @@ -0,0 +1,281 @@ +// Copyright (c) Mike Grier. + +//! Prints what a namespace request costs to build, against the queue that would +//! carry it. +//! +//! **An experiment, not a component.** These probes measure platform behaviour +//! and are not for production use. Do not call them from production code, and +//! do not lift a technique out of here. See this crate's DESIGN-NOTES.md. +//! +//! Read alongside `probe-doorbell-cost`: together they say whether the queue's +//! mechanics or the request's allocation model deserves the attention. + +use std::fmt::Write as _; +use windows_platform_probes::report::{Stdout, emit}; +use windows_platform_probes::request_cost::measure; + +/// Measured by `probe-doorbell-cost` on the development machine, and recorded in +/// [the 2026-08-30 design session]. Restated here only to render a ratio; the +/// authoritative number is whatever that probe prints on the host this runs on. +/// +/// **The build profile behind these is not recorded**, which is why the report +/// below calls the ratios indicative rather than quoting them as results. They +/// are not re-baselined here: the figure is the one that session recorded, and +/// silently replacing it would leave the session describing a number that no +/// longer exists anywhere. CI runs both this probe and `probe-doorbell-cost` +/// under `--release` in the same job, so the like-for-like comparison a reader +/// actually wants is those two outputs, not this constant. +/// +/// [the 2026-08-30 design session]: ../../../../design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md +const DOORBELL_NS_REFERENCE: f64 = 164.9; +const ATOMIC_NS_REFERENCE: f64 = 7.2; + +fn main() { + // The only place that names the real stream. Everything below composes + // text; nothing below knows where it goes. + emit(&mut Stdout, &render()); +} + +/// The probe's whole report, as text. +fn render() -> String { + let mut out = String::new(); + // First line of the report, and part of the returned text rather than + // written out here: a captured report must carry the line naming the + // machine that produced it, and the taint marker with it. Without it a + // timing number can be pasted anywhere and compared against anything. + let _ = writeln!( + out, + "{}", + windows_placement_probe::fingerprint::banner_line() + ); + let _ = writeln!(out, "== what does a namespace request cost to build? ==\n"); + + let observation = measure(); + + let _ = writeln!( + out, + "{:<26} {:>10} {:>14} {:>16}", + "operation", "ns/op", "x an atomic", "x a doorbell" + ); + for timing in &observation.timings { + let _ = writeln!( + out, + "{:<26} {:>10.1} {:>14.1} {:>16.2}", + timing.label, + timing.nanos_per_op, + timing.nanos_per_op / ATOMIC_NS_REFERENCE, + timing.nanos_per_op / DOORBELL_NS_REFERENCE, + ); + } + let _ = writeln!( + out, + "\n(ratios use the reference doorbell {DOORBELL_NS_REFERENCE:.1} ns and atomic \ + {ATOMIC_NS_REFERENCE:.1} ns measured\n by probe-doorbell-cost on the development \ + machine; re-read that probe on this host\n before trusting them)" + ); + + let _ = writeln!(out, "\ninterpretation:"); + + let build = observation.get("build_open_request"); + let capture = observation.get("capture_handle"); + + if let Some(build) = build { + // The ratio names its own reference, because the two numbers do not + // come from the same machine. `build` was measured on this host just + // now; the doorbell figure is a constant captured once on the + // development machine. This probe now runs on hosted CI runners, which + // are a heterogeneous fleet, so a ratio printed as though both halves + // were local can be wrong even when the measurement is sound. + let _ = writeln!( + out, + " building a pathed request costs {build:.0} ns, which is {:.1}x one", + build / DOORBELL_NS_REFERENCE + ); + let _ = writeln!( + out, + " doorbell AS MEASURED ON THE DEVELOPMENT MACHINE ({DOORBELL_NS_REFERENCE:.1} ns)," + ); + let _ = writeln!( + out, + " not on this one. Run probe-doorbell-cost here to make the ratio local." + ); + let _ = writeln!(out); + let _ = writeln!( + out, + " SCOPE, because this is easy to over-read: that is a statement about" + ); + let _ = writeln!( + out, + " ONE OPERATION TYPE, not about the queue. A namespace open is the" + ); + let _ = writeln!( + out, + " heaviest payload the queue carries -- it resolves a path through" + ); + let _ = writeln!( + out, + " Win32 and may duplicate a handle -- and it ends in a CreateFileW" + ); + let _ = writeln!( + out, + " costing microseconds regardless. A registered-buffer read, which is" + ); + let _ = writeln!( + out, + " the hot path, carries no path and no handle: its descriptor is a slot" + ); + let _ = writeln!( + out, + " index and an offset, and there the queue's own mechanics are the" + ); + let _ = writeln!(out, " whole cost."); + let _ = writeln!(out); + let _ = writeln!( + out, + " Nor is per-operation overhead the same thing as queue efficiency." + ); + let _ = writeln!( + out, + " Throughput under contention, cache behaviour, batching amortization" + ); + let _ = writeln!( + out, + " and backpressure decide that, and a single uncontended construction" + ); + let _ = writeln!(out, " time measures none of them."); + let _ = writeln!(out); + let _ = writeln!( + out, + " What it does support: for an open-heavy workload, doorbell tuning" + ); + let _ = writeln!( + out, + " would be optimizing the small half. That is a finding about" + ); + let _ = writeln!( + out, + " OPERATION MIX, and it says nothing about the read path." + ); + } + + if let Some(capture) = capture { + let _ = writeln!( + out, + "\n duplicating a handle costs {capture:.0} ns -- a kernel transition, not" + ); + let _ = writeln!( + out, + " a memory copy, and easy to under-count when thinking about what an" + ); + let _ = writeln!(out, " SQE holds."); + // Reported beside it because the duplication figure above excludes it by + // construction, and a reader sizing a request's real cost needs both: + // every captured handle is eventually closed, so the lifecycle is the + // pair. Keeping them separate is what stops either being quoted as the + // other. + if let Some(close) = observation.get("close_handle") { + let _ = writeln!( + out, + " Closing one costs a further {close:.0} ns, measured separately, so a" + ); + let _ = writeln!( + out, + " captured handle's whole lifecycle is {:.0} ns. The duplication figure", + capture + close + ); + let _ = writeln!(out, " above is the duplication alone."); + } + if let Some(build) = build { + if capture > build { + let _ = writeln!( + out, + " It is {:.1}x the cost of building the pathed request itself, so a", + capture / build + ); + let _ = writeln!( + out, + " request carrying a handle is dominated by the duplication, and" + ); + let _ = writeln!( + out, + " any allocation tuning on the path would be optimizing the wrong" + ); + let _ = writeln!(out, " half."); + } else { + let _ = writeln!( + out, + " It is {:.2}x the pathed request, so the two are comparable and", + capture / build + ); + let _ = writeln!(out, " neither dominates."); + } + } + } + + // The split that decides whether an allocator change can help at all. + if let Some(build) = build + && let Some(clone) = observation.get("clone_prepared_units") + && build > clone + { + let _ = writeln!( + out, + "\n WHERE THE TIME ACTUALLY GOES, and it is not the allocator:" + ); + let _ = writeln!( + out, + " `prepare` calls GetFullPathNameW to resolve the path against the" + ); + let _ = writeln!( + out, + " process working directory -- a Win32 call, because the CWD is mutable" + ); + let _ = writeln!( + out, + " by any thread and resolving later would be racy. So most of the cost" + ); + let _ = writeln!( + out, + " above is a syscall that no allocation scheme can remove." + ); + let _ = writeln!( + out, + " Cloning already-prepared units is {clone:.0} ns, which bounds what an" + ); + let _ = writeln!( + out, + " inline-storage or recycling scheme could recover at {:.0} ns per request", + build - clone + ); + let _ = writeln!( + out, + " AT MOST -- and only for a caller that can reuse a resolved path." + ); + let _ = writeln!( + out, + " A caller with a fresh path each time pays the resolution regardless." + ); + } + + let get = |label: &str| { + observation + .get(label) + .map_or("null".to_string(), |n| format!("{n:.1}")) + }; + let _ = writeln!( + out, + concat!( + r#"{{"reason":"x-probe-request-cost","arch":"{}","prepare_short_ns":{},"#, + r#""prepare_long_ns":{},"build_open_request_ns":{},"#, + r#""clone_prepared_units_ns":{},"capture_handle_ns":{},"#, + r#""close_handle_ns":{}}}"# + ), + std::env::consts::ARCH, + get("prepare_short_path"), + get("prepare_long_path"), + get("build_open_request"), + get("clone_prepared_units"), + get("capture_handle"), + get("close_handle"), + ); + out +} diff --git a/crates/windows-platform-probes/src/bin/topology.rs b/crates/windows-platform-probes/src/bin/topology.rs new file mode 100644 index 00000000..7b8f8cc5 --- /dev/null +++ b/crates/windows-platform-probes/src/bin/topology.rs @@ -0,0 +1,238 @@ +// Copyright (c) Mike Grier. + +//! Prints the machine's processor topology, and how many execution domains each +//! candidate partitioning policy would produce on it. +//! +//! **An experiment, not a component.** These probes measure platform behaviour +//! and are not for production use. Do not call them from production code, and +//! do not lift a technique out of here. See this crate's DESIGN-NOTES.md. +//! +//! Running this on every CI build is deliberate: hosted runners are a +//! heterogeneous fleet, so the accumulated output is a slow survey of what real +//! machines look like. The line tagged `x-probe-topology` is emitted as a single +//! JSON object so those results can be mined out of build logs mechanically +//! rather than read by eye. + +use std::fmt::Write as _; +use windows_platform_probes::report::{Stdout, emit}; +use windows_platform_probes::topology::measure; + +fn main() { + // The only place that names the real stream. Everything below composes + // text; nothing below knows where it goes. + emit(&mut Stdout, &render()); +} + +/// The probe's whole report, as text. +fn render() -> String { + let mut out = String::new(); + // First line of the report, and part of the returned text rather than + // written out here: a captured report must carry the line naming the + // machine that produced it, and the taint marker with it. Without it a + // timing number can be pasted anywhere and compared against anything. + let _ = writeln!( + out, + "{}", + windows_placement_probe::fingerprint::banner_line() + ); + let _ = writeln!( + out, + "== processor topology, and what each partitioning policy would yield ==\n" + ); + + let observation = match measure() { + Ok(observation) => observation, + Err(error) => { + let _ = writeln!(out, "MachineMemoryTopology::discover failed: {error}"); + let _ = writeln!( + out, + "(Reported rather than measured: a probe that cannot read its" + ); + let _ = writeln!( + out, + "subject must say so instead of printing a misleading shape.)" + ); + return out; + } + }; + + let _ = writeln!( + out, + "processors (online) : {}", + observation.online_processors + ); + let _ = writeln!(out, "processor groups : {}", observation.groups); + let _ = writeln!(out, "packages : {}", observation.packages); + let _ = writeln!( + out, + "NUMA domains : {} ({} with no processors)", + observation.numa_domains, observation.memoryless_numa_domains + ); + let _ = writeln!(out, "physical cores : {}", observation.cores.len()); + + let smt = observation + .cores + .iter() + .filter(|c| c.simultaneous_multithreading) + .count(); + let mut classes: Vec = observation + .cores + .iter() + .map(|c| c.efficiency_class) + .collect(); + classes.sort_unstable(); + classes.dedup(); + let _ = writeln!(out, " cores with SMT : {smt}"); + let _ = writeln!(out, " efficiency classes: {classes:?}"); + if classes.len() > 1 { + let _ = writeln!( + out, + " (heterogeneous: an I/O thread left unconstrained can land on an" + ); + let _ = writeln!( + out, + " efficiency core, which is why even a single domain wants a mask)" + ); + } + + let _ = writeln!(out, "\ncaches:"); + if observation.caches.is_empty() { + let _ = writeln!(out, " none reported"); + } + for cache in &observation.caches { + let _ = writeln!( + out, + " L{:<2} {:>3} domain(s), processors per domain: {:?}", + cache.level, cache.domains, cache.processors_per_domain + ); + } + + match observation.outermost_partitioning_cache() { + Some(cache) => { + let _ = writeln!( + out, + "\noutermost cache that partitions this machine: L{} ({} domains)", + cache.level, cache.domains + ); + } + None => { + // Two different topologies land here and the difference matters, so + // this reports the absence rather than explaining it. `None` means + // "no unique outermost partitioning cache", which is EITHER that no + // level partitions the machine OR that two levels partition it + // incomparably -- neither being a subset of the other, so neither is + // outermost. Naming only the first turns a reported ambiguity into a + // false claim about the hardware. + let _ = writeln!( + out, + "\nno unique outermost partitioning cache was established: either no" + ); + let _ = writeln!( + out, + "level partitions this machine, or two partition it incomparably." + ); + } + } + if !observation.caches.iter().any(|c| c.level == 3) { + let _ = writeln!( + out, + "NOTE: this machine reports no L3 at all, so a policy keyed literally" + ); + let _ = writeln!( + out, + "on \"L3\" would find nothing here. That is the measured case behind" + ); + let _ = writeln!( + out, + "phrasing the rule as \"the outermost level that partitions\"." + ); + } + + let _ = writeln!(out, "\ndomains each policy would produce:"); + for (name, count) in observation.domain_counts() { + let _ = writeln!(out, " {name:<34} {count}"); + } + + let _ = writeln!( + out, + "\ncross-check against independently read Win32 counters:" + ); + let _ = writeln!( + out, + " GetActiveProcessorCount : {}", + observation.raw_active_processors + ); + let _ = writeln!( + out, + " GetActiveProcessorGroupCount: {}", + observation.raw_group_count + ); + match observation.raw_highest_numa_node { + Some(highest) => { + // The identifier, and deliberately no count derived from it. + // `GetNumaHighestNodeNumber` reports the largest node NUMBER, and + // node numbers may be sparse -- a machine with nodes 0 and 2 has two + // nodes and a highest of 2. `highest + 1` would print three, which + // is the same mistake `Observation::cross_check` was corrected to + // stop making; re-deriving it here would put it back in the output + // the cross-check is printed beside. + let _ = writeln!(out, " GetNumaHighestNodeNumber : {highest}"); + let _ = writeln!( + out, + " (the largest node NUMBER, not a count: node numbers can be sparse)" + ); + } + None => { + let _ = writeln!(out, " GetNumaHighestNodeNumber : failed"); + } + } + let complaints = observation.cross_check(); + if complaints.is_empty() { + let _ = writeln!( + out, + " => agree. windows-topology-sys parsed this machine consistently." + ); + } else { + let _ = writeln!(out, " => DISAGREE. This is a finding, not a nuisance:"); + for complaint in &complaints { + let _ = writeln!(out, " - {complaint}"); + } + } + + // One machine-readable line, so accumulated CI logs can be mined without + // parsing the prose above. Kept to a single line on purpose. + let cache_json: Vec = observation + .caches + .iter() + .map(|c| format!(r#"{{"level":{},"domains":{}}}"#, c.level, c.domains)) + .collect(); + let policy_json: Vec = observation + .domain_counts() + .into_iter() + .map(|(name, count)| format!(r#""{name}":{count}"#)) + .collect(); + let _ = writeln!( + out, + concat!( + r#"{{"reason":"x-probe-topology","arch":"{}","processors":{},"groups":{},"#, + r#""packages":{},"numa_domains":{},"memoryless_numa_domains":{},"cores":{},"#, + r#""efficiency_classes":{},"caches":[{}],"outermost_partitioning_cache_level":{},"#, + r#""policies":{{{}}},"cross_check_ok":{}}}"# + ), + std::env::consts::ARCH, + observation.online_processors, + observation.groups, + observation.packages, + observation.numa_domains, + observation.memoryless_numa_domains, + observation.cores.len(), + classes.len(), + cache_json.join(","), + observation + .outermost_partitioning_cache() + .map_or("null".to_string(), |c| c.level.to_string()), + policy_json.join(","), + complaints.is_empty(), + ); + out +} diff --git a/crates/windows-platform-probes/src/doorbell_cost.rs b/crates/windows-platform-probes/src/doorbell_cost.rs new file mode 100644 index 00000000..5d0ed95c --- /dev/null +++ b/crates/windows-platform-probes/src/doorbell_cost.rs @@ -0,0 +1,287 @@ +// Copyright (c) Mike Grier. + +//! How expensive is a doorbell, relative to the syscall it would guard? +//! +//! **An experiment, not a component.** These probes measure platform behaviour +//! and are not for production use. Do not call them from production code, and +//! do not lift a technique out of here. See this crate's DESIGN-NOTES.md. +//! +//! # The decision this exists to inform +//! +//! The two-layer ring design has a client thread push a descriptor onto a +//! bounded MPSC queue and then, sometimes, signal an event so the domain thread +//! wakes. The design assumes that signal is expensive enough to be worth +//! avoiding, and proposes an eventcount -- publish intent to park, re-check the +//! queue, then wait -- so a producer rings the doorbell only on the +//! empty-to-non-empty edge and only when a consumer is actually parked. +//! +//! That protocol is the highest-risk part of the whole design, because +//! publish-recheck-park is exactly where lost wakeups live. Building it because +//! the cost was *assumed* would be taking on that risk without evidence. So: +//! +//! - if `SetEvent` is a meaningful fraction of `SubmitIoRing`, the skip rules +//! are load-bearing and belong in the design from the start; +//! - if it is noise, a simple always-signal queue is adequate and the +//! optimization can wait for a measurement that justifies it. +//! +//! # What is timed +//! +//! Each is a tight loop over a warm path, reported as nanoseconds per +//! operation. Absolute values are host-specific and uninteresting; the +//! **ratios** are the finding. +//! +//! - `atomic_fetch_add` -- the uncontended atomic that a queue push costs, as a +//! floor for "the cheapest useful thing". +//! - `set_event_already_signalled` -- `SetEvent` on an event that is already +//! set, which is the redundant-signal case the skip rule removes. +//! - `set_reset_event` -- `SetEvent` then `ResetEvent`, the honest cost of one +//! doorbell cycle with nobody waiting. +//! - `wait_zero_signalled` -- `WaitForSingleObject(handle, 0)` on a signalled +//! event: the consumer's cost of observing it. +//! - `submit_io_ring_empty` -- `SubmitIoRing` with nothing queued, which is the +//! syscall the doorbell would be amortised against. Absent when `IoRing` is +//! unavailable. +//! +//! # The empty submit is not a fair denominator, and the first run proved it +//! +//! This probe was written expecting to divide the doorbell cost by +//! `submit_io_ring_empty` and read off "the doorbell is N% of a syscall". **Do +//! not do that.** Measured on the development machine, an empty `SubmitIoRing` +//! came in at ~79 ns -- far too cheap for a kernel transition, so it is almost +//! certainly short-circuiting in user mode when there is nothing queued. The +//! resulting "doorbell is 210% of a syscall" would have been a confident wrong +//! answer built on a denominator that never entered the kernel. +//! +//! The honest denominator is the cost of the real work a submission carries, +//! which this probe deliberately does not measure -- so it reports the absolute +//! costs and the *batching* arithmetic instead, and leaves the ratio alone. +//! [`Observation::doorbell_share_of_submit`] is retained only because the raw +//! fact is worth recording; its own documentation repeats this warning. + +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Instant; + +use windows_sys::Win32::Foundation::{CloseHandle, HANDLE, WAIT_OBJECT_0}; +use windows_sys::Win32::System::Threading::{ + CreateEventW, ResetEvent, SetEvent, WaitForSingleObject, +}; + +use crate::ioring; + +/// Nanoseconds per operation for one timed loop. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct Timing { + /// What was timed. + pub label: &'static str, + /// Iterations executed. + pub iterations: u32, + /// Nanoseconds per iteration. + pub nanos_per_op: f64, +} + +/// Every timing, plus the ratios that actually decide the design question. +#[derive(Debug, Clone)] +pub struct Observation { + /// Each timed loop, in the order run. + pub timings: Vec, + /// `None` when `IoRing` is unavailable on this host. + pub submit_nanos: Option, +} + +impl Observation { + /// Look a timing up by label. + #[must_use] + pub fn get(&self, label: &str) -> Option { + self.timings + .iter() + .find(|t| t.label == label) + .map(|t| t.nanos_per_op) + } + + /// One doorbell cycle as a fraction of one **empty** `SubmitIoRing`. + /// + /// **This is not the number the design turns on, and it should not be read + /// as one.** An empty submit does not appear to enter the kernel (see the + /// module documentation), so this ratio has a denominator that is not a + /// syscall. It is exposed because the raw fact is worth recording across + /// hosts -- a machine where the empty submit is *expensive* would itself be + /// a finding -- not because dividing by it answers anything. + #[must_use] + pub fn doorbell_share_of_submit(&self) -> Option { + let doorbell = self.get("set_reset_event")?; + let submit = self.submit_nanos?; + (submit > 0.0).then_some(doorbell / submit) + } +} + +fn time_loop(label: &'static str, iterations: u32, mut body: impl FnMut()) -> Timing { + // Warm the path first: the first call through a syscall stub pays for + // resolution and page faults that a steady-state cost should not include. + for _ in 0..1024 { + body(); + } + let start = Instant::now(); + for _ in 0..iterations { + body(); + } + let elapsed = start.elapsed(); + Timing { + label, + iterations, + nanos_per_op: elapsed.as_nanos() as f64 / f64::from(iterations), + } +} + +/// Run every timing. +/// +/// # Panics +/// +/// Panics if `CreateEventW` fails, which would mean the host cannot create a +/// manual-reset event and nothing here is measurable. +#[must_use] +pub fn measure() -> Observation { + const ITERATIONS: u32 = 200_000; + + // SAFETY: a manual-reset, initially-unsignalled, unnamed event. + let event: HANDLE = unsafe { CreateEventW(std::ptr::null(), 1, 0, std::ptr::null()) }; + assert!(!event.is_null(), "CreateEventW failed"); + + let counter = AtomicU64::new(0); + let mut timings = Vec::new(); + + timings.push(time_loop("atomic_fetch_add", ITERATIONS, || { + counter.fetch_add(1, Ordering::Relaxed); + })); + + // Leave it signalled, so every call in the next loop is redundant. + unsafe { SetEvent(event) }; + timings.push(time_loop("set_event_already_signalled", ITERATIONS, || { + unsafe { SetEvent(event) }; + })); + + unsafe { ResetEvent(event) }; + timings.push(time_loop("set_reset_event", ITERATIONS, || unsafe { + SetEvent(event); + ResetEvent(event); + })); + + unsafe { SetEvent(event) }; + timings.push(time_loop("wait_zero_signalled", ITERATIONS, || { + unsafe { WaitForSingleObject(event, 0) }; + })); + unsafe { + ResetEvent(event); + CloseHandle(event); + } + + // The syscall the doorbell would be amortised against. Far fewer + // iterations: this one is a real kernel transition. `submit_and_wait(0)` + // asks for no completions, so it returns without blocking and measures the + // transition rather than any I/O. + let submit_nanos = ioring::Ring::new().map(|ring| { + const SUBMIT_ITERATIONS: u32 = 20_000; + let timing = time_loop("submit_io_ring_empty", SUBMIT_ITERATIONS, || { + // Both halves of the answer are checked, inside the timed region. + // Discarding them let a host where `SubmitIoRing` fails produce a + // perfectly plausible timing -- a failing call still costs a + // measurable transition -- which the report then read as the cost of + // a successful empty submission. That is the failure mode this whole + // crate exists to avoid: a number that looks like evidence and is + // not. The `submitted` count is checked too, because a call that + // succeeded while submitting entries did not measure what the label + // says it measured. + // + // The cost is a predictable branch against a syscall, which does not + // perturb the figure; leaving the check outside the loop would let + // the timing be taken before anything established it was valid. + let (hr, submitted) = ring.submit_and_wait(0); + assert!(hr >= 0, "SubmitIoRing(0) failed: {hr:#010x}"); + assert_eq!(submitted, 0, "SubmitIoRing(0) submitted entries"); + }); + timings.push(timing); + timing.nanos_per_op + }); + + Observation { + timings, + submit_nanos, + } +} + +/// Keeps the doorbell's own wake path honest: a consumer that actually parks +/// and is woken measures something the zero-timeout poll above does not. +/// +/// Reported separately because it is a two-thread measurement and therefore +/// noisier than the single-threaded loops. The number is a full **round trip** +/// -- wake the peer, park, be woken -- not a single transition, so it is an +/// upper bound on what one wakeup costs rather than the cost itself. +/// +/// # Why the handshake alternates strictly +/// +/// The obvious version -- one thread calling `SetEvent` in a loop while the +/// other calls `WaitForSingleObject` -- **deadlocks**, and did when this probe +/// was first written. An auto-reset event does not count signals: two arriving +/// before one wait collapse into one, the waiter's count never catches up, and +/// it blocks on `INFINITE` for ever. Two events used as ping and pong force +/// strict alternation, so no signal can be lost. +/// +/// Every wait is nevertheless bounded. A probe that can hang is a probe that +/// can hang a build, and the deadlock above is exactly how that happens; a +/// timeout turns it into a reported anomaly instead. +/// +/// Returns `None` if the handshake ever timed out, because a partial run's +/// average would be meaningless -- and for the same reason if `rounds` is zero, +/// which has no average at all rather than an average of nothing. +#[must_use] +pub fn measure_park_and_wake(rounds: u32) -> Option { + const WAIT_TIMEOUT_MS: u32 = 5_000; + + // An empty sample has no average, and the arithmetic below would not say + // so: no round runs, so the elapsed time is zero, and `0.0 / 0.0` is `NaN` + // wrapped in the `Some` this function documents as a meaningful number. A + // caller comparing that against a threshold gets `false` from every + // comparison and no indication why. + if rounds == 0 { + return None; + } + + // SAFETY: two auto-reset, initially-unsignalled, unnamed events. + let ping: HANDLE = unsafe { CreateEventW(std::ptr::null(), 0, 0, std::ptr::null()) }; + let pong: HANDLE = unsafe { CreateEventW(std::ptr::null(), 0, 0, std::ptr::null()) }; + assert!(!ping.is_null() && !pong.is_null(), "CreateEventW failed"); + + let (ping_addr, pong_addr) = (ping as usize, pong as usize); + let peer = std::thread::spawn(move || { + let (ping, pong) = (ping_addr as HANDLE, pong_addr as HANDLE); + for _ in 0..rounds { + // SAFETY: both handles outlive this thread, which is joined below. + let waited = unsafe { WaitForSingleObject(ping, WAIT_TIMEOUT_MS) }; + if waited != WAIT_OBJECT_0 { + return false; + } + unsafe { SetEvent(pong) }; + } + true + }); + + let mut ok = true; + let start = Instant::now(); + for _ in 0..rounds { + // SAFETY: both handles are live for the whole loop. + unsafe { SetEvent(ping) }; + if unsafe { WaitForSingleObject(pong, WAIT_TIMEOUT_MS) } != WAIT_OBJECT_0 { + ok = false; + break; + } + } + let elapsed = start.elapsed(); + + let peer_ok = peer.join().unwrap_or(false); + // SAFETY: the peer has been joined, so nothing else holds these. + unsafe { + CloseHandle(ping); + CloseHandle(pong); + } + + (ok && peer_ok).then(|| elapsed.as_nanos() as f64 / f64::from(rounds)) +} diff --git a/crates/windows-platform-probes/src/lib.rs b/crates/windows-platform-probes/src/lib.rs index c16c9ebe..253d1df2 100644 --- a/crates/windows-platform-probes/src/lib.rs +++ b/crates/windows-platform-probes/src/lib.rs @@ -113,13 +113,17 @@ pub mod cancel_io; pub mod completion_port; pub mod device_map; +pub mod doorbell_cost; pub mod error_mode; pub mod handle_state; pub mod ioring; pub mod long_path; pub mod long_path_report; pub mod pool_growth; +pub mod queue_contention; pub mod report; +pub mod request_cost; +pub mod topology; pub mod worker_context; #[cfg(test)] diff --git a/crates/windows-platform-probes/src/queue_contention.rs b/crates/windows-platform-probes/src/queue_contention.rs new file mode 100644 index 00000000..15d0e4de --- /dev/null +++ b/crates/windows-platform-probes/src/queue_contention.rs @@ -0,0 +1,641 @@ +// Copyright (c) Mike Grier. + +//! Does the array queue's tail claim contend at realistic producer counts? +//! +//! **An experiment, not a component.** These probes measure platform behaviour +//! and are not for production use. Do not call them from production code, and +//! do not lift a technique out of here. See this crate's DESIGN-NOTES.md. +//! +//! # The two decisions this exists to force +//! +//! **1. Are the linked and sharded MPSC shapes needed at all?** They are parked +//! in `CHECKLIST-io-domains.md` as `M-inf.1`, gated on this measurement rather +//! than on taste. If N threads compare-and-swapping one tail does not collapse +//! at the producer counts a real system reaches, the bounded array queue is the +//! only MPSC the queue crate ever needs, and two speculative shapes never get +//! written. +//! +//! **2. Should `slotwise_mpsc` and `reserving_mpsc` merge?** They ship as peers because +//! honouring a reservation costs the producer a read of the consumer's +//! position -- one line every thread touches -- and *how much* that costs was a +//! judgement rather than a measurement. If it is cheap, the two shapes merge and +//! the non-reserving one goes; if it is expensive, the split is vindicated. +//! +//! # Two regimes, because one of them cannot answer the second question +//! +//! Producers are timed twice, and the pair is the point. +//! +//! - **Isolated** -- capacity large enough that nothing is ever refused, and no +//! consumer running. This is the *cleanest* measurement of tail-claim +//! contention: nothing else touches the queue, so whatever curve appears +//! against N is the compare-and-swap and nothing else. +//! +//! - **Drained** -- a consumer popping continuously while the producers push. +//! This is the one that can price `reserving_mpsc`, because its producer reads +//! `head`, and `head` is only expensive to read when a consumer is *writing* +//! it. Measured in isolation that read hits a clean, shared line and looks +//! free -- which would be a confident wrong answer. +//! +//! # What is deliberately not claimed +//! +//! The drained regime has a **single** consumer, because that is what MPSC +//! means. At high producer counts it is therefore expected to become +//! consumer-bound, and a throughput plateau there says nothing about the tail +//! claim. The probe reports each run's refusal count -- from the queue's own +//! `Observable` counters -- so a backpressure-bound run is visible as a fact +//! rather than mistaken for contention. Read the isolated regime for the +//! contention question, and the drained one for the cost of `head`. + +use std::sync::Arc; +use std::sync::Barrier; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::thread; +use std::time::Instant; + +use windows_waitable_queues::{permit_mpsc, reserving_mpsc, slotwise_mpsc}; + +use windows_waitable_queues::reserving_mpsc::{Balanced, ClaimLayout, Enduring, Perpetual, Wide}; + +/// How many pushes each producer thread performs in one timed run. +const PUSHES_PER_PRODUCER: usize = 50_000; + +/// How many times each configuration is repeated; the median is reported. +/// +/// Odd, so the median is an observed value rather than an average of two. Five +/// because these probes run on a virtual machine, where a single run can be +/// perturbed by something entirely outside the process. +const REPETITIONS: usize = 5; + +/// The producer counts measured, in order. +/// +/// Fixed rather than derived from the host's processor count, so two runs on +/// different machines produce comparable rows. The host's own count is reported +/// alongside, since the interesting region is around and beyond it. +pub const PRODUCER_COUNTS: &[usize] = &[1, 2, 4, 8, 16, 32]; + +/// The names a run is filed under. +/// +/// **Named once because a lookup by string literal is a rename waiting to +/// fail, and this one already did.** The `mpsc` -> `slotwise_mpsc` rename +/// updated the recording side and not the reporting binary, which went on +/// asking for `"mpsc"`; every lookup returned `None` and two entire columns of +/// the report rendered as `--` without anything erroring. A wrong shape name is +/// not a compile error, so the only defence is that both sides read the same +/// definition. +pub mod shapes { + /// The bounded-array MPSC. + pub const SLOTWISE_MPSC: &str = "slotwise_mpsc"; + /// The reservation-based MPSC. + pub const RESERVING_MPSC: &str = "reserving_mpsc"; + /// The experimental permit-claiming MPSC, measured against + /// [`RESERVING_MPSC`] because it is a candidate replacement for it. + pub const PERMIT_MPSC: &str = "permit_mpsc"; + /// The uncontended-atomic floor the queues are measured against. + pub const BASELINE_FETCH_ADD: &str = "baseline_fetch_add"; + /// `reserving_mpsc` on its default layout: a `u64` split 32 / 32. + /// + /// The same configuration as [`RESERVING_MPSC`], run again under its own + /// name so the layout comparison reads without a reader having to know + /// which layout the default is. + pub const CLAIM_NARROW: &str = "reserving(32/32)"; + /// `reserving_mpsc` on `Enduring`: a `u64` split 16 / 48. + pub const CLAIM_DEEP: &str = "reserving(16/48)"; + /// `reserving_mpsc` on `Perpetual`: a `u64` split 8 / 56. + pub const CLAIM_PERPETUAL: &str = "reserving(8/56)"; + /// `reserving_mpsc` on `Wide`: a `u128` split 64 / 64. + pub const CLAIM_WIDE: &str = "reserving(64/64)"; +} +/// One configuration's result. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct Run { + /// Which queue shape, or the baseline. + pub shape: &'static str, + /// How many producer threads pushed concurrently. + pub producers: usize, + /// Median nanoseconds per successful push, across all producers. + pub nanos_per_push: f64, + /// Successful pushes per second, summed across producers. + pub pushes_per_second: f64, + /// Pushes refused for want of room during the median run. + /// + /// Non-zero means the run was at least partly bounded by the consumer + /// rather than by the claim, which is a fact about the measurement and not + /// about the queue. + pub refusals: u64, +} + +/// Everything one invocation measured. +#[derive(Debug, Clone)] +pub struct Observation { + /// Producers timed with no consumer and no possibility of refusal. + pub isolated: Vec, + /// Producers timed against a continuously draining consumer. + pub drained: Vec, + /// Logical processors the host reports. + pub logical_processors: usize, +} + +impl Observation { + /// Look one run up. + #[must_use] + pub fn find(&self, regime: &[Run], shape: &str, producers: usize) -> Option { + regime + .iter() + .find(|run| run.shape == shape && run.producers == producers) + .copied() + } + + /// How far throughput scaled from one producer to `producers`. + /// + /// 1.0 means N producers together push no faster than one did, which is + /// what a badly contended claim looks like. Perfect scaling would be N, + /// which no shared-tail queue can reach. + #[must_use] + pub fn scaling(&self, regime: &[Run], shape: &str, producers: usize) -> Option { + let one = self.find(regime, shape, 1)?; + let many = self.find(regime, shape, producers)?; + Some(many.pushes_per_second / one.pushes_per_second) + } +} + +/// Time every configuration. +#[must_use] +pub fn measure() -> Observation { + let mut isolated = Vec::new(); + let mut drained = Vec::new(); + + for &producers in PRODUCER_COUNTS { + isolated.push(median_run(shapes::BASELINE_FETCH_ADD, producers, |count| { + time_contended_atomic(count) + })); + isolated.push(median_run(shapes::SLOTWISE_MPSC, producers, |count| { + time_isolated_mpsc(count) + })); + isolated.push(median_run(shapes::RESERVING_MPSC, producers, |count| { + time_isolated_reserving(count) + })); + isolated.push(median_run(shapes::PERMIT_MPSC, producers, |count| { + time_isolated_permit(count) + })); + + drained.push(median_run(shapes::SLOTWISE_MPSC, producers, |count| { + time_drained_mpsc(count) + })); + drained.push(median_run(shapes::RESERVING_MPSC, producers, |count| { + time_drained_reserving(count) + })); + drained.push(median_run(shapes::PERMIT_MPSC, producers, |count| { + time_drained_permit(count) + })); + + isolated.push(median_run(shapes::CLAIM_NARROW, producers, |count| { + time_isolated_layout::(count) + })); + isolated.push(median_run(shapes::CLAIM_DEEP, producers, |count| { + time_isolated_layout::(count) + })); + isolated.push(median_run(shapes::CLAIM_PERPETUAL, producers, |count| { + time_isolated_layout::(count) + })); + isolated.push(median_run(shapes::CLAIM_WIDE, producers, |count| { + time_isolated_layout::(count) + })); + + drained.push(median_run(shapes::CLAIM_NARROW, producers, |count| { + time_drained_layout::(count) + })); + drained.push(median_run(shapes::CLAIM_DEEP, producers, |count| { + time_drained_layout::(count) + })); + drained.push(median_run(shapes::CLAIM_PERPETUAL, producers, |count| { + time_drained_layout::(count) + })); + drained.push(median_run(shapes::CLAIM_WIDE, producers, |count| { + time_drained_layout::(count) + })); + } + + Observation { + isolated, + drained, + logical_processors: thread::available_parallelism().map_or(0, std::num::NonZeroUsize::get), + } +} + +/// Raw result of one timed repetition: elapsed nanoseconds and refusals. +type Repetition = (f64, u64); + +/// Run one configuration [`REPETITIONS`] times and keep the median. +/// +/// The median rather than the mean, because on a virtual machine the failure +/// mode is one run being hugely slower rather than a spread around a centre, +/// and a mean would carry that outlier into the reported number. +fn median_run( + shape: &'static str, + producers: usize, + mut timer: impl FnMut(usize) -> Repetition, +) -> Run { + // One untimed pass first: the first touch of a fresh allocation faults + // pages in, and that cost belongs to the allocator rather than the queue. + let _ = timer(producers); + + let mut results: Vec = (0..REPETITIONS).map(|_| timer(producers)).collect(); + results.sort_by(|left, right| left.0.total_cmp(&right.0)); + let (elapsed_nanos, refusals) = results[REPETITIONS / 2]; + + let pushes = (producers * PUSHES_PER_PRODUCER) as f64; + Run { + shape, + producers, + nanos_per_push: elapsed_nanos / pushes, + pushes_per_second: pushes / (elapsed_nanos / 1e9), + refusals, + } +} + +/// The floor: N threads incrementing one shared counter. +/// +/// Not a queue, and not trying to be. It is the cheapest possible operation on +/// a contended line, so it says how much of a queue's scaling curve is the +/// queue and how much is simply what this processor does when N cores fight +/// over one cache line. +fn time_contended_atomic(producers: usize) -> Repetition { + let counter = Arc::new(AtomicU64::new(0)); + // One party per worker plus this thread. Every worker is created, then waits + // here; the clock starts as the barrier releases, so neither thread creation + // nor a solo head start by an early worker is inside the measurement. See + // `start_barrier`'s note for why that matters at these producer counts. + let gate = Arc::new(Barrier::new(producers + 1)); + let started = thread::scope(|scope| { + for _ in 0..producers { + let counter = Arc::clone(&counter); + let gate = Arc::clone(&gate); + scope.spawn(move || { + gate.wait(); + for _ in 0..PUSHES_PER_PRODUCER { + counter.fetch_add(1, Ordering::Relaxed); + } + }); + } + gate.wait(); + Instant::now() + }); + (started.elapsed().as_nanos() as f64, 0) +} + +/// Capacity big enough that a whole run fits, so nothing is ever refused. +fn capacity_for(producers: usize) -> usize { + (producers * PUSHES_PER_PRODUCER).next_power_of_two() +} + +/// A gate holding every participant until all of them exist. +/// +/// **Without this the row labelled N producers need not have measured N of +/// them.** Spawning is not instant, and each worker used to start pushing the +/// moment it was created, so at 50,000 pushes an early producer could complete +/// a long uncontended prefix -- or finish entirely -- before the last thread was +/// spawned. The reported interval also began before any worker existed, folding +/// thread-creation cost into a per-push number. The curve against N is the whole +/// output of this probe, and both effects bend it downward exactly where it is +/// steepest. +/// +/// The count includes this thread: the workers arrive and block, this thread +/// arrives last, and the clock starts as the barrier releases them together. +fn start_barrier(participants: usize) -> Arc { + Arc::new(Barrier::new(participants + 1)) +} + +fn time_isolated_mpsc(producers: usize) -> Repetition { + let (tx, rx) = + slotwise_mpsc::bounded::(capacity_for(producers)).expect("a valid capacity"); + let gate = start_barrier(producers); + let started = thread::scope(|scope| { + for producer in 0..producers { + let tx = tx.clone(); + let gate = Arc::clone(&gate); + scope.spawn(move || { + gate.wait(); + for index in 0..PUSHES_PER_PRODUCER { + tx.push((producer * PUSHES_PER_PRODUCER + index) as u64) + .expect("the run fits in the capacity"); + } + }); + } + gate.wait(); + Instant::now() + }); + let elapsed = started.elapsed().as_nanos() as f64; + let refusals = tx.refused(); + // Drain before dropping: teardown would otherwise walk every slot, and that + // is not part of what is being timed. + while rx.pop().is_ok() {} + (elapsed, refusals) +} + +fn time_isolated_reserving(producers: usize) -> Repetition { + let (tx, rx) = + reserving_mpsc::bounded::(capacity_for(producers)).expect("a valid capacity"); + let gate = start_barrier(producers); + let started = thread::scope(|scope| { + for producer in 0..producers { + let tx = tx.clone(); + let gate = Arc::clone(&gate); + scope.spawn(move || { + gate.wait(); + for index in 0..PUSHES_PER_PRODUCER { + tx.push((producer * PUSHES_PER_PRODUCER + index) as u64) + .expect("the run fits in the capacity"); + } + }); + } + gate.wait(); + Instant::now() + }); + let elapsed = started.elapsed().as_nanos() as f64; + let refusals = tx.refused(); + while rx.pop().is_ok() {} + (elapsed, refusals) +} + +/// The experimental permit claim, in the regime that isolates the claim itself. +/// +/// A line-for-line twin of [`time_isolated_reserving`] with one shape +/// substituted. Deliberately not factored into a generic over the two, which +/// would need a trait both implement and would put a dynamic or monomorphised +/// indirection inside the timed region -- in a measurement whose whole output is +/// a difference of a few nanoseconds per push. +fn time_isolated_permit(producers: usize) -> Repetition { + let (tx, rx) = permit_mpsc::bounded::(capacity_for(producers)).expect("a valid capacity"); + let gate = start_barrier(producers); + let started = thread::scope(|scope| { + for producer in 0..producers { + let tx = tx.clone(); + let gate = Arc::clone(&gate); + scope.spawn(move || { + gate.wait(); + for index in 0..PUSHES_PER_PRODUCER { + tx.push((producer * PUSHES_PER_PRODUCER + index) as u64) + .expect("the run fits in the capacity"); + } + }); + } + gate.wait(); + Instant::now() + }); + let elapsed = started.elapsed().as_nanos() as f64; + let refusals = tx.refused(); + while rx.pop().is_ok() {} + (elapsed, refusals) +} + +/// A capacity a real system would choose, so the drained regime exercises +/// backpressure the way a real one would. +const DRAINED_CAPACITY: usize = 1024; + +fn time_drained_mpsc(producers: usize) -> Repetition { + let (tx, rx) = slotwise_mpsc::bounded::(DRAINED_CAPACITY).expect("a valid capacity"); + let done = Arc::new(AtomicBool::new(false)); + let consumer_done = Arc::clone(&done); + // The consumer is a participant too: it is spawned first, but spawning is + // not readiness, and a consumer still starting up while producers push turns + // the opening of the run into an undrained regime -- the one thing this + // measurement is defined against. + let gate = start_barrier(producers + 1); + let consumer_gate = Arc::clone(&gate); + + let consumer = thread::spawn(move || { + consumer_gate.wait(); + // Spin rather than park: the doorbell's cost is `doorbell_cost`'s + // question, and parking here would measure that instead of the claim. + while !consumer_done.load(Ordering::Relaxed) { + while rx.pop().is_ok() {} + std::hint::spin_loop(); + } + while rx.pop().is_ok() {} + rx.refused() + }); + + let started = thread::scope(|scope| { + for producer in 0..producers { + let tx = tx.clone(); + let gate = Arc::clone(&gate); + scope.spawn(move || { + gate.wait(); + for index in 0..PUSHES_PER_PRODUCER { + let mut item = (producer * PUSHES_PER_PRODUCER + index) as u64; + // Retry on a full queue, which is what a real producer + // does. The refusal count is what makes that visible. + while let Err(error) = tx.push(item) { + item = error.into_inner(); + std::hint::spin_loop(); + } + } + }); + } + gate.wait(); + Instant::now() + }); + let elapsed = started.elapsed().as_nanos() as f64; + + done.store(true, Ordering::Relaxed); + drop(tx); + let refusals = consumer.join().expect("the consumer must not panic"); + (elapsed, refusals) +} + +fn time_drained_reserving(producers: usize) -> Repetition { + // **Defaults on both sides, and that is a correction.** This row previously + // enabled high-water tracking here and nowhere else, to "also price the + // switch M31.4 made opt-in". But the number it feeds is presented as the + // cost of *reservation*, and tracking adds an unrelated operation to this + // shape's push path alone -- a load of the consumer's position, which is + // exactly the shared line the other shape's push is built to avoid + // touching. The ratio therefore measured reservation plus a handicap, with + // no way for a reader to separate them. + // + // Nothing consumes the high-water figure here either, so the tracking was + // paying a cost to produce a number nobody read. Pricing that switch is a + // worthwhile measurement and needs its own row, with both shapes tracking, + // rather than being folded into this comparison. + let (tx, rx) = reserving_mpsc::bounded::(DRAINED_CAPACITY).expect("a valid capacity"); + let done = Arc::new(AtomicBool::new(false)); + let consumer_done = Arc::clone(&done); + // The consumer joins the gate here for the reason it does in the slotwise + // twin: a run whose opening is undrained is not the regime being measured. + let gate = start_barrier(producers + 1); + let consumer_gate = Arc::clone(&gate); + + let consumer = thread::spawn(move || { + consumer_gate.wait(); + while !consumer_done.load(Ordering::Relaxed) { + while rx.pop().is_ok() {} + std::hint::spin_loop(); + } + while rx.pop().is_ok() {} + rx.refused() + }); + + let started = thread::scope(|scope| { + for producer in 0..producers { + let tx = tx.clone(); + let gate = Arc::clone(&gate); + scope.spawn(move || { + gate.wait(); + for index in 0..PUSHES_PER_PRODUCER { + let mut item = (producer * PUSHES_PER_PRODUCER + index) as u64; + while let Err(error) = tx.push(item) { + item = error.into_inner(); + std::hint::spin_loop(); + } + } + }); + } + gate.wait(); + Instant::now() + }); + let elapsed = started.elapsed().as_nanos() as f64; + + done.store(true, Ordering::Relaxed); + drop(tx); + let refusals = consumer.join().expect("the consumer must not panic"); + (elapsed, refusals) +} + +/// The experimental permit claim, against a continuously draining consumer. +/// +/// The regime that can price the claim honestly, for the same reason the +/// reserving twin needs it: the shared line a producer touches is only +/// expensive when a consumer is writing it. Measured in isolation, an +/// uncontended line looks free -- which would be a confident wrong answer, and +/// this shape has more riding on that answer than the others, because it trades +/// `reserving_mpsc`'s *load* of the consumer's position for a read-modify-write +/// on a count the consumer also writes. +fn time_drained_permit(producers: usize) -> Repetition { + let (tx, rx) = permit_mpsc::bounded::(DRAINED_CAPACITY).expect("a valid capacity"); + let done = Arc::new(AtomicBool::new(false)); + let consumer_done = Arc::clone(&done); + let gate = start_barrier(producers + 1); + let consumer_gate = Arc::clone(&gate); + + let consumer = thread::spawn(move || { + consumer_gate.wait(); + while !consumer_done.load(Ordering::Relaxed) { + while rx.pop().is_ok() {} + std::hint::spin_loop(); + } + while rx.pop().is_ok() {} + rx.refused() + }); + + let started = thread::scope(|scope| { + for producer in 0..producers { + let tx = tx.clone(); + let gate = Arc::clone(&gate); + scope.spawn(move || { + gate.wait(); + for index in 0..PUSHES_PER_PRODUCER { + let mut item = (producer * PUSHES_PER_PRODUCER + index) as u64; + while let Err(error) = tx.push(item) { + item = error.into_inner(); + std::hint::spin_loop(); + } + } + }); + } + gate.wait(); + Instant::now() + }); + let elapsed = started.elapsed().as_nanos() as f64; + + done.store(true, Ordering::Relaxed); + drop(tx); + let refusals = consumer.join().expect("the consumer must not panic"); + (elapsed, refusals) +} + +/// One claim-word layout, in the regime that isolates the claim. +/// +/// **Generic over the layout, where [`time_isolated_permit`] is deliberately +/// duplicated, and the difference is the point.** That twin compares two +/// *different types*, which a generic could only unify behind a trait, putting +/// an indirection that might not inline identically inside the timed region. +/// These are the *same type* at different layout parameters, so this +/// monomorphises to exactly the code a hand-written copy would produce -- there +/// is nothing left to dispatch. +/// +/// Measures `reserving_mpsc` itself rather than a stand-in. An earlier form of +/// this probe carried its own duplicated implementation of the claim protocol, +/// built so the layouts could be compared before the shipping crate had them; +/// it drifted from the original twice while doing so. The shipping type takes +/// the layout as a parameter now, so the duplicate is gone. +fn time_isolated_layout(producers: usize) -> Repetition { + let (tx, rx) = + reserving_mpsc::bounded_as::(capacity_for(producers)).expect("a valid capacity"); + let gate = start_barrier(producers); + let started = thread::scope(|scope| { + for producer in 0..producers { + let tx = tx.clone(); + let gate = Arc::clone(&gate); + scope.spawn(move || { + gate.wait(); + for index in 0..PUSHES_PER_PRODUCER { + tx.push((producer * PUSHES_PER_PRODUCER + index) as u64) + .expect("the run fits in the capacity"); + } + }); + } + gate.wait(); + Instant::now() + }); + let elapsed = started.elapsed().as_nanos() as f64; + let refusals = tx.refused(); + while rx.pop().is_ok() {} + (elapsed, refusals) +} + +/// One claim-word layout, against a continuously draining consumer. +/// +/// Generic for [`time_isolated_layout`]'s reason. +fn time_drained_layout(producers: usize) -> Repetition { + let (tx, rx) = + reserving_mpsc::bounded_as::(DRAINED_CAPACITY).expect("a valid capacity"); + let done = Arc::new(AtomicBool::new(false)); + let consumer_done = Arc::clone(&done); + // The consumer joins the gate for the reason its twins do: a run whose + // opening is undrained is not the regime being measured. + let gate = start_barrier(producers + 1); + let consumer_gate = Arc::clone(&gate); + + let consumer = thread::spawn(move || { + consumer_gate.wait(); + while !consumer_done.load(Ordering::Relaxed) { + while rx.pop().is_ok() {} + std::hint::spin_loop(); + } + while rx.pop().is_ok() {} + rx.refused() + }); + + let started = thread::scope(|scope| { + for producer in 0..producers { + let tx = tx.clone(); + let gate = Arc::clone(&gate); + scope.spawn(move || { + gate.wait(); + for index in 0..PUSHES_PER_PRODUCER { + let mut item = (producer * PUSHES_PER_PRODUCER + index) as u64; + while let Err(error) = tx.push(item) { + item = error.into_inner(); + std::hint::spin_loop(); + } + } + }); + } + gate.wait(); + Instant::now() + }); + let elapsed = started.elapsed().as_nanos() as f64; + done.store(true, Ordering::Relaxed); + let refusals = consumer.join().expect("the consumer must not panic"); + (elapsed, refusals) +} diff --git a/crates/windows-platform-probes/src/request_cost.rs b/crates/windows-platform-probes/src/request_cost.rs new file mode 100644 index 00000000..9a3a4486 --- /dev/null +++ b/crates/windows-platform-probes/src/request_cost.rs @@ -0,0 +1,250 @@ +// Copyright (c) Mike Grier. + +//! What does it cost to build a namespace request, against the queue that would +//! carry it? +//! +//! **An experiment, not a component.** These probes measure platform behaviour +//! and are not for production use. Do not call them from production code, and +//! do not lift a technique out of here. See this crate's DESIGN-NOTES.md. +//! +//! # The decision this exists to inform +//! +//! The two-layer ring's submission queue was specified to carry **POD +//! descriptors with no allocation on push**. A deferred `CreateFileW` carries a +//! path, and a path is neither fixed-size nor POD, so that requirement and the +//! namespace plane's needs cannot both hold as written. +//! +//! `windows-namespace-request-sys` already solves the hard part: an `OpenFile` +//! is an *owned, `Send`* parameter set, built on one thread and performed +//! faithfully on another. So the queue can carry a request by value and the +//! lifetime hazard disappears. What remains is a cost question about **this +//! operation type**: how does building one compare with the doorbell that would +//! carry it (~165 ns, per `probe-doorbell-cost`)? +//! +//! # What this does not measure, stated because the number invites over-reading +//! +//! **This says nothing about whether the queue is efficient.** It measures the +//! construction cost of the queue's *heaviest* payload. Two distinctions the +//! result must not be stretched across: +//! +//! - **Operation type.** A namespace open resolves a path through Win32 and +//! may duplicate a handle. A registered-buffer read -- the hot path -- does +//! neither: its descriptor is a slot index and an offset, and there the +//! queue's own mechanics are the whole per-operation cost. +//! - **Overhead against efficiency.** Throughput under contention, the ring's +//! cache behaviour, batching amortization, and backpressure under load are +//! what make a queue good or bad. A single uncontended construction time +//! measures none of them. +//! +//! The conclusion it *does* support is about **operation mix**: for an +//! open-heavy workload, effort spent shaving the doorbell would be spent on the +//! small half of the cost. +//! +//! # Handle duplication is the part that is easy to under-count +//! +//! A request that carries a handle -- a template handle for an open, or the +//! subject of a query -- must **duplicate** it, because the submitting thread +//! may close its own copy the moment it returns. `CapturedHandle::capture` does +//! that with `DuplicateHandle`, which is a kernel transition, not a memory +//! copy. So "what does a request cost" is not only an allocation question, and +//! measuring only the path would understate it. +//! +//! # Preparing a path is a Win32 call, not an allocation +//! +//! This probe was written expecting `prepare` to be an allocation and a copy. +//! It is not: it calls **`GetFullPathNameW`** to resolve the path against the +//! process working directory, because [the namespace session] settled that the +//! path is resolved at submission -- the process CWD is mutable by any thread, +//! so even perfect remoting would be racy. +//! +//! That means the measured cost is a *syscall* cost and cannot be tuned away by +//! an allocator. An inline-storage or recycling scheme would only recover the +//! allocation part, which `clone_prepared_units` bounds from below. Knowing +//! which half is which is the point of measuring both. +//! +//! [the namespace session]: ../../../design-sessions/DESIGN-SESSION-2026-08-27-pseudo-async-namespace-operations.md +//! +//! Each timing is reported per operation. Absolute values are host-specific; +//! the **ratios against the doorbell and the atomic** are the finding. + +use std::time::Instant; + +use wtf_string::Wtf16String; + +use windows_namespace_request_sys::{CapturedHandle, OpenFile, prepare}; +use windows_sys::Win32::Foundation::GENERIC_READ; +use windows_sys::Win32::Storage::FileSystem::{FILE_SHARE_READ, OPEN_EXISTING}; +use windows_sys::Win32::System::SystemInformation::GetSystemDirectoryW; + +/// Nanoseconds per operation for one timed loop. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct Timing { + /// What was timed. + pub label: &'static str, + /// Iterations executed. + pub iterations: u32, + /// Nanoseconds per iteration. + pub nanos_per_op: f64, +} + +/// Every timing taken by [`measure`]. +#[derive(Debug, Clone)] +pub struct Observation { + /// Each timed loop, in the order run. + pub timings: Vec, +} + +impl Observation { + /// Look a timing up by label. + #[must_use] + pub fn get(&self, label: &str) -> Option { + self.timings + .iter() + .find(|t| t.label == label) + .map(|t| t.nanos_per_op) + } +} + +fn time_loop(label: &'static str, iterations: u32, mut body: impl FnMut() -> T) -> Timing { + // Warm the path: the first pass pays for lazily resolved syscall stubs and + // for the allocator's first touch of a fresh size class. + for _ in 0..256 { + std::hint::black_box(body()); + } + let start = Instant::now(); + for _ in 0..iterations { + std::hint::black_box(body()); + } + let elapsed = start.elapsed(); + Timing { + label, + iterations, + nanos_per_op: elapsed.as_nanos() as f64 / f64::from(iterations), + } +} + +/// Time request construction, path preparation, and handle duplication. +/// +/// # Panics +/// +/// Panics if the fixed test paths fail to prepare, which would mean +/// `prepare` rejects an ordinary absolute path and nothing here is meaningful. +#[must_use] +pub fn measure() -> Observation { + const ITERATIONS: u32 = 100_000; + const HANDLE_ITERATIONS: u32 = 50_000; + + // Resolved, not assumed. Windows is not always on `C:` -- a valid + // installation can sit on any volume -- and hard-coding it made this probe + // panic on such a machine rather than measure it. The same path is used for + // the prepared request and the real open below, so the two stay consistent. + let system_dll = system_directory().join("kernel32.dll"); + let short = Wtf16String::from( + system_dll + .to_str() + .expect("the system directory is representable"), + ); + let long_text = format!(r"C:\{}\file.txt", vec!["directory"; 24].join("\\")); + let long = Wtf16String::from(long_text.as_str()); + + let mut timings = Vec::new(); + + // The allocation and normalization a path costs, at two lengths, because + // the common case and the worst case allocate differently. + timings.push(time_loop("prepare_short_path", ITERATIONS, || { + prepare(&short).expect("an absolute path prepares") + })); + timings.push(time_loop("prepare_long_path", ITERATIONS, || { + prepare(&long).expect("an absolute path prepares") + })); + + // A whole request, which is a prepared path plus the builder chain. This is + // what the queue would actually carry. + timings.push(time_loop("build_open_request", ITERATIONS, || { + let path = prepare(&short).expect("an absolute path prepares"); + OpenFile::new(path) + .with_desired_access(GENERIC_READ) + .with_share_mode(FILE_SHARE_READ) + .with_creation_disposition(OPEN_EXISTING) + })); + + // Cloning the prepared path alone, which is what a request-recycling scheme + // would avoid paying. + let prepared_units = prepare(&short) + .expect("an absolute path prepares") + .into_wtf16(); + timings.push(time_loop("clone_prepared_units", ITERATIONS, || { + prepared_units.clone() + })); + + // The kernel transition a captured handle costs. Measured against a handle + // this process already owns, so nothing here depends on the filesystem. + // + // Capture and close are timed SEPARATELY, and that separation is the whole + // point. `time_loop` black-boxes its closure's return value and drops it at + // the end of the statement, so a loop that captures and returns a + // `CapturedHandle` -- which owns an `OwnedHandle` -- also calls + // `CloseHandle` inside the timed region. That is two kernel transitions + // reported as one number, and the report reads that number as the cost of + // duplication alone, so it overstated it by however much a close costs. + // + // Retaining every duplicate in a pre-sized `Vec` keeps the close out of the + // capture loop, and timing the drop of that same `Vec` recovers the close as + // its own figure rather than discarding it. The `push` is a pointer bump + // into reserved capacity, which is not free but is nowhere near a syscall. + let file = std::fs::File::open(&system_dll).expect("kernel32.dll is readable"); + let borrowed = std::os::windows::io::AsHandle::as_handle(&file); + + // Warmed the same way `time_loop` warms, and for the same reason: the first + // pass pays for lazily resolved syscall stubs and the allocator's first + // touch of a fresh size class. + for _ in 0..256 { + let _ = + std::hint::black_box(CapturedHandle::capture(borrowed).expect("duplicating a handle")); + } + + let mut captured = Vec::with_capacity(HANDLE_ITERATIONS as usize); + let start = Instant::now(); + for _ in 0..HANDLE_ITERATIONS { + captured.push(CapturedHandle::capture(borrowed).expect("duplicating an owned handle")); + } + let capture_elapsed = start.elapsed(); + timings.push(Timing { + label: "capture_handle", + iterations: HANDLE_ITERATIONS, + nanos_per_op: capture_elapsed.as_nanos() as f64 / f64::from(HANDLE_ITERATIONS), + }); + + let start = Instant::now(); + drop(captured); + let close_elapsed = start.elapsed(); + timings.push(Timing { + label: "close_handle", + iterations: HANDLE_ITERATIONS, + nanos_per_op: close_elapsed.as_nanos() as f64 / f64::from(HANDLE_ITERATIONS), + }); + + Observation { timings } +} + +/// Where Windows is actually installed, rather than where it usually is. +/// +/// Falls back to the conventional path only when the system will not say, which +/// keeps the probe running on a machine that answers and keeps the failure +/// visible on one that does not. +fn system_directory() -> std::path::PathBuf { + let mut buffer = [0_u16; 260]; + // SAFETY: writes at most `buffer.len()` units into a buffer of that size. + let written = unsafe { GetSystemDirectoryW(buffer.as_mut_ptr(), buffer.len() as u32) }; + let written = written as usize; + // `>=`, not `>`. On success the count excludes the terminator, so it can + // reach at most `buffer.len() - 1`; on failure it is the required size + // *including* the terminator, so it is at least `buffer.len() + 1`. Exactly + // `buffer.len()` is therefore unreachable from either branch -- and treating + // it as a failure costs nothing while removing the need for the next reader + // to redo that analysis before trusting a possibly-unterminated buffer. + if written == 0 || written >= buffer.len() { + return std::path::PathBuf::from(r"C:\Windows\System32"); + } + std::path::PathBuf::from(String::from_utf16_lossy(&buffer[..written])) +} diff --git a/crates/windows-platform-probes/src/tests.rs b/crates/windows-platform-probes/src/tests.rs index b716b9e8..0e0a937c 100644 --- a/crates/windows-platform-probes/src/tests.rs +++ b/crates/windows-platform-probes/src/tests.rs @@ -693,3 +693,315 @@ fn the_impersonation_guard_reverts_even_while_unwinding() { "the thread must not have been left impersonating after the unwind" ); } + +// --- topology ------------------------------------------------------------- +// +// Every interesting number the topology probe prints is host-specific, so +// nothing here asserts a *value*. What is asserted is internal consistency, +// which must hold on any machine and therefore catches a parsing regression in +// `windows-topology-sys` on whatever hardware CI happens to run on -- which is +// the whole reason the probe reads the shipping crate rather than a second +// parse written here. + +#[test] +fn the_machine_reports_at_least_one_processor_one_group_and_one_core() { + let observation = crate::topology::measure().expect("topology discovery"); + + assert!( + observation.online_processors >= 1, + "a running process implies at least one online processor" + ); + assert!( + observation.groups >= 1, + "every machine has at least one processor group" + ); + assert!( + !observation.cores.is_empty(), + "a machine with processors must report cores" + ); + assert!( + observation.packages >= 1, + "every machine has at least one package" + ); +} + +#[test] +fn the_shipping_parse_agrees_with_the_raw_win32_counters() { + let observation = crate::topology::measure().expect("topology discovery"); + + // This is the cross-check that makes the probe worth running everywhere: a + // disagreement means windows-topology-sys parsed + // GetLogicalProcessorInformationEx differently from what the simple + // counters report on this host. + let complaints = observation.cross_check(); + assert!( + complaints.is_empty(), + "topology crate disagrees with the raw counters: {complaints:?}" + ); +} + +/// An observation with nothing to complain about, for a test to perturb one +/// field of. Every host reachable here has a single NUMA node, so the sparse +/// case below cannot be measured and has to be constructed. +fn agreeing_observation() -> crate::topology::Observation { + crate::topology::Observation { + online_processors: 4, + groups: 1, + numa_domains: 1, + memoryless_numa_domains: 0, + highest_numa_node: Some(0), + packages: 1, + cores: Vec::new(), + caches: Vec::new(), + partitioning_cache_level: None, + raw_active_processors: 4, + raw_group_count: 1, + raw_highest_numa_node: Some(0), + } +} + +#[test] +fn sparse_numa_node_numbers_are_not_reported_as_a_parsing_regression() { + // `GetNumaHighestNodeNumber` reports the highest node *number*, which + // Windows does not promise equals the node count. Nodes 0 and 2 are a valid + // sparse topology: two domains, highest number two. Comparing the count + // against `highest + 1` called that a disagreement, so the probe's asserted + // test would fail on hardware that is reporting itself correctly. + let mut observation = agreeing_observation(); + observation.numa_domains = 2; + observation.highest_numa_node = Some(2); + observation.raw_highest_numa_node = Some(2); + + assert!( + observation.cross_check().is_empty(), + "a sparse node numbering is a valid machine, not a parse error" + ); +} + +#[test] +fn a_numa_node_the_topology_crate_never_saw_is_still_reported() { + // The other direction, so the sparse tolerance cannot pass by never + // complaining: Windows names a node the crate's parse did not produce, and + // that is the disagreement this cross-check exists to surface. + let mut observation = agreeing_observation(); + observation.raw_highest_numa_node = Some(3); + + let complaints = observation.cross_check(); + assert_eq!(complaints.len(), 1, "{complaints:?}"); + assert!(complaints[0].contains("NUMA nodes"), "{complaints:?}"); +} + +#[test] +fn a_topology_reporting_no_numa_node_at_all_disagrees_with_a_raw_one() { + // The `None` arm, which the count form could not express: Windows names a + // node and the crate's parse produced no memory domain whatsoever. + let mut observation = agreeing_observation(); + observation.numa_domains = 0; + observation.highest_numa_node = None; + + let complaints = observation.cross_check(); + assert_eq!(complaints.len(), 1, "{complaints:?}"); + assert!(complaints[0].contains("none"), "{complaints:?}"); +} + +#[test] +fn every_core_reports_processors_and_smt_agrees_with_the_count() { + let observation = crate::topology::measure().expect("topology discovery"); + + for core in &observation.cores { + assert!( + core.processors >= 1, + "a core with no processors is a parse error, not a machine" + ); + assert_eq!( + core.simultaneous_multithreading, + core.processors > 1, + "SMT is exactly the condition of a core carrying more than one processor" + ); + } +} + +#[test] +fn every_cache_level_reports_at_least_one_domain_with_at_least_one_processor() { + let observation = crate::topology::measure().expect("topology discovery"); + + for cache in &observation.caches { + assert!( + cache.level >= 1, + "a cache level of zero is a parse error, not a machine" + ); + assert_eq!( + cache.domains, + cache.processors_per_domain.len(), + "the domain count must be the length of the per-domain spans" + ); + assert!( + cache.processors_per_domain.iter().all(|&span| span >= 1), + "a cache domain covering no processors is a parse error" + ); + } +} + +#[test] +fn the_outermost_partitioning_cache_is_the_deepest_level_that_splits_the_machine() { + let observation = crate::topology::measure().expect("topology discovery"); + + // What this can check, and deliberately no more. `Observation`'s caches are + // SUMMARIES -- a level, a domain count, and the size of each domain -- with + // processor membership discarded. "Outermost" is defined by set inclusion + // between partitions, so nothing here can re-derive the selection, and the + // two attempts to do so anyway were both wrong: + // + // - Ordering candidates by LEVEL NUMBER is the rule the topology crate + // abandoned, because a higher number is not always coarser. This module's + // own `outermost_partitioning_cache` doc records it, and the synthetic + // test below builds the counterexample: a valid L2 outermost partition + // alongside a finer L3 that still partitions. A level-number assertion + // here would fail that legitimate topology. + // + // - Reading `None` as "no level partitions this machine" is false for the + // same reason the renderer was corrected: `None` is also the answer when + // two partitionings are incomparable, which is a deliberate ambiguity + // result rather than a claim about the hardware. + // + // So this asserts the two properties that survive the summarising: a + // selection is one of the surveyed levels and genuinely partitions, and the + // selection agrees with the level the survey captured. + match observation.outermost_partitioning_cache() { + Some(chosen) => { + assert!( + chosen.domains > 1, + "a level that does not partition cannot be the partitioning level" + ); + assert_eq!( + Some(chosen.level), + observation.partitioning_cache_level, + "the looked-up summary must be the level the survey selected" + ); + assert!( + observation.caches.iter().any(|c| c.level == chosen.level), + "the selected level must be one this survey actually recorded" + ); + } + None => assert!( + // Only the internal consistency, which is all that is knowable: a + // survey that captured no level must look one up, and a survey that + // captured one must find it. Why the level is absent is not + // recoverable from here, and guessing was the defect. + observation.partitioning_cache_level.is_none() + || !observation + .caches + .iter() + .any(|c| Some(c.level) == observation.partitioning_cache_level), + "a level was selected but its summary was not found" + ), + } +} + +#[test] +fn every_policy_would_produce_at_least_one_domain() { + let observation = crate::topology::measure().expect("topology discovery"); + + // The point of this one is the degenerate cases: a machine reporting zero + // NUMA nodes, or no cache that partitions, must still yield a usable domain + // count rather than zero. A fleet sized at zero domains does no I/O at all, + // and that is exactly the shape the ARM64 laptop in the 2026-08-30 session + // would have produced under a policy keyed literally on L3. + for (name, count) in observation.domain_counts() { + assert!( + count >= 1, + "policy {name} would produce {count} domains, and a fleet of zero \ + domains can perform no I/O" + ); + } +} + +// --- M5+.3: the partitioning rule has one implementation --- + +#[test] +fn the_survey_reports_the_topology_crates_partitioning_level_not_its_own() { + // The restatement this removes differed from the crate's answer in two + // ways: it omitted the pairwise-disjointness check, and it ordered + // candidates by LEVEL NUMBER, which the topology crate stopped doing + // because a higher number is not always coarser. + // + // Here the higher level is the finer partition, so the two rules disagree: + // the old `max_by_key(level)` would answer L3, and asking the crate answers + // L2. The survey must report what the crate says. + let mut observation = agreeing_observation(); + observation.caches = vec![ + crate::topology::CacheLevel { + level: 2, + domains: 2, + processors_per_domain: vec![2, 2], + }, + crate::topology::CacheLevel { + level: 3, + domains: 4, + processors_per_domain: vec![1, 1, 1, 1], + }, + ]; + observation.partitioning_cache_level = Some(2); + + assert_eq!( + observation.outermost_partitioning_cache().map(|c| c.level), + Some(2), + "the survey must not re-derive; it looks up what the crate decided" + ); +} + +#[test] +fn no_partitioning_level_is_a_real_answer_in_the_survey_too() { + let mut observation = agreeing_observation(); + observation.caches = vec![crate::topology::CacheLevel { + level: 3, + domains: 1, + processors_per_domain: vec![4], + }]; + observation.partitioning_cache_level = None; + + assert!(observation.outermost_partitioning_cache().is_none()); +} + +// --- the doorbell's park-and-wake handshake --------------------------------- +// +// Its own documentation records that the first implementation DEADLOCKED: one +// thread setting an auto-reset event while the other waited, two signals +// collapsing into one, and the waiter blocking on INFINITE for ever. Nothing +// tested it, so the rewrite that fixed it could have been undone silently. +// +// Both tests are bounded by construction. A test that can hang is worse than +// the defect it guards, because it takes the whole suite with it: the handshake +// itself waits with a 5-second timeout and reports `None` rather than blocking, +// so a reintroduced deadlock surfaces here as a failed assertion within seconds +// rather than as a suite that never finishes. + +#[test] +fn a_zero_round_handshake_has_no_average_rather_than_a_meaningless_one() { + // The deterministic half of the contract, and the one a caller is most + // likely to break by "simplifying". With no rounds the elapsed time is zero + // and the average would be `0.0 / 0.0` -- NaN, wrapped in the `Some` this + // function documents as a meaningful number, against which every comparison + // a caller makes returns false with no indication why. + assert_eq!( + crate::doorbell_cost::measure_park_and_wake(0), + None, + "zero rounds has no average, and must not report one" + ); +} + +#[test] +fn a_small_handshake_completes_and_reports_a_positive_round_trip() { + // The liveness half: the two-event alternation actually runs to completion + // and produces a number. Deliberately few rounds -- this is a real + // cross-thread measurement, and the assertion is that it terminates and is + // sane, not that it is fast. A machine under load may make each round + // arbitrarily slow without making it wrong. + let average = crate::doorbell_cost::measure_park_and_wake(64) + .expect("a bounded handshake of 64 rounds must complete rather than time out"); + + assert!( + average.is_finite() && average > 0.0, + "a completed handshake must report a positive finite round trip, got {average}" + ); +} diff --git a/crates/windows-platform-probes/src/topology.rs b/crates/windows-platform-probes/src/topology.rs new file mode 100644 index 00000000..c4d6e021 --- /dev/null +++ b/crates/windows-platform-probes/src/topology.rs @@ -0,0 +1,325 @@ +// Copyright (c) Mike Grier. + +//! What shape is the machine, and which cache level actually partitions it? +//! +//! **An experiment, not a component.** These probes measure platform behaviour +//! and are not for production use. Do not call them from production code, and +//! do not lift a technique out of here. See this crate's DESIGN-NOTES.md. +//! +//! # Why this is a probe rather than a test +//! +//! Almost every number here is host-specific, so there is nothing to assert +//! about its *value* -- only about its internal consistency. That is the +//! binary-plus-asserted split this crate is built around: the binary prints the +//! shape for whoever is reading, and the tests pin the invariants that must hold +//! on any machine, so a parsing regression fails the build even though a core +//! count cannot. +//! +//! Running it in CI is the point. Hosted runners are a heterogeneous fleet, so +//! printing the discovered shape on every build turns ordinary CI into a slow +//! survey of what real machines look like -- including the negative result that +//! cloud runners are consistently single-node, which is itself evidence for how +//! the [uniform tunable architecture](../../../design-sessions/DESIGN-SESSION-2026-08-30-numa-sharded-io-execution-domains.md) +//! should size itself by default. +//! +//! # It measures the shipping crate, deliberately +//! +//! The parse comes from [`windows_topology_sys::MachineMemoryTopology::discover`] rather +//! than from a reimplementation here, for the same reason the pool-growth probe +//! uses the real thread-pool crate: a reimplementation would measure the +//! reimplementation. The raw counters below are then read *independently* +//! through Win32 and compared against it, so this probe doubles as a +//! cross-check on that crate's parsing across every machine CI ever runs on. + +use std::io; + +use windows_sys::Win32::System::Threading::{ + ALL_PROCESSOR_GROUPS, GetActiveProcessorCount, GetActiveProcessorGroupCount, + GetNumaHighestNodeNumber, +}; + +use windows_topology_sys::{DomainKind, MachineMemoryTopology, Source}; + +/// One cache level, summarised across the machine. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CacheLevel { + /// 1, 2, 3, ... as the firmware reports it. + pub level: u8, + /// How many distinct processor *partitions* exist at this level. + /// + /// Not the number of caches: a level Windows reports once per cache -- L1 + /// as separate `data` and `instruction` domains over the same processors -- + /// is several relationships but one partition per processor set, and it is + /// the partition a caller dividing work cares about. + pub domains: usize, + /// Processors per partition, in discovery order. + pub processors_per_domain: Vec, +} + +/// One core, summarised. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct CoreShape { + /// Whether this core carries more than one logical processor. + pub simultaneous_multithreading: bool, + /// The firmware's performance ranking for this core. More than one distinct + /// value across the machine means heterogeneous cores, and therefore that + /// an unconstrained thread can be scheduled onto a slow one. + pub efficiency_class: u8, + /// Logical processors this core covers. + pub processors: usize, +} + +/// The machine's shape, as the shipping topology crate sees it, plus the raw +/// counters read independently for cross-checking. +#[derive(Debug, Clone)] +pub struct Observation { + // --- read through windows-topology-sys --- + /// Logical processors reported as online. + pub online_processors: usize, + /// Processor groups. More than one is a hard affinity boundary: a thread's + /// affinity names exactly one group, so above 64 logical processors the + /// partition is forced whether or not it is wanted. + pub groups: usize, + /// NUMA domains, including any that report no processors. + pub numa_domains: usize, + /// NUMA domains that report no processors at all -- ordinary on machines + /// with CXL expanders or HBM tiers, and the reason a domain count cannot be + /// used as a thread count. + pub memoryless_numa_domains: usize, + /// The largest NUMA node number the topology crate reported, or `None` when + /// it reported no memory domain at all. + /// + /// Kept beside the count because the two answer different questions and + /// Windows only promises the second one: node numbers are not guaranteed + /// dense, so a machine with nodes 0 and 2 has a count of two and a highest + /// of two. Comparing the count against `GetNumaHighestNodeNumber` would + /// call that correct machine a parsing regression. + pub highest_numa_node: Option, + /// Physical packages (sockets). + pub packages: usize, + /// Every physical core. + pub cores: Vec, + /// Cache levels, ascending, each summarised across the machine. + pub caches: Vec, + /// Which of [`Self::caches`] the topology crate says actually partitions + /// the machine, if any. + /// + /// Captured from `MachineMemoryTopology::outermost_partitioning_cache` + /// rather than derived from the summaries above, so the rule has one + /// implementation (`SH-16.9`). `None` is a real answer -- no reported level + /// divides this machine -- and not a failure. + pub partitioning_cache_level: Option, + + // --- read independently through Win32 --- + /// `GetActiveProcessorCount(ALL_PROCESSOR_GROUPS)`. + pub raw_active_processors: u32, + /// `GetActiveProcessorGroupCount()`. + pub raw_group_count: u16, + /// `GetNumaHighestNodeNumber()`, or `None` if the call failed. + pub raw_highest_numa_node: Option, +} + +impl Observation { + /// The outermost cache level that actually splits the machine into more + /// than one domain, if any. + /// + /// **Asked of `windows-topology-sys`, not re-derived here.** This method + /// used to restate the rule as "the highest level with more than one + /// domain", and by the time `M4+.4` landed that restatement differed from + /// the crate's own answer in two ways: it omitted the pairwise-disjointness + /// check, so a hand-built topology with overlapping blocks would have been + /// accepted, and it ordered candidates by **level number**, which the + /// topology crate stopped doing because a higher number is not always + /// coarser -- the ARM64 machine with no L3 is the standing counterexample. + /// + /// So the level is now captured at survey time from + /// `MachineMemoryTopology::outermost_partitioning_cache`, and this method + /// only looks up the summary for it. `SH-16.9` records this rule going + /// wrong three times in two crates; there is now one implementation. + #[must_use] + pub fn outermost_partitioning_cache(&self) -> Option<&CacheLevel> { + let level = self.partitioning_cache_level?; + self.caches.iter().find(|c| c.level == level) + } + + /// How many execution domains each candidate policy would produce. + /// + /// Reported rather than recommended. The point of printing all of them is + /// that they disagree, and the disagreement is the finding. + #[must_use] + pub fn domain_counts(&self) -> Vec<(&'static str, usize)> { + vec![ + ("single", 1), + ("by-package", self.packages), + ( + "by-numa-domain-with-processors", + // Clamped to one, as the cache policy beside it already is. A + // host that reports no NUMA relationships leaves both counters + // at zero, and a policy that yields zero domains contradicts + // the execution-domain contract: there is always at least one + // domain, because the machine exists. Saturating for the same + // reason -- these are two independent counts from the + // operating system, and an unsigned subtraction that trusts + // their relationship would panic rather than report. + self.numa_domains + .saturating_sub(self.memoryless_numa_domains) + .max(1), + ), + ( + "by-outermost-partitioning-cache", + self.outermost_partitioning_cache().map_or(1, |c| c.domains), + ), + ("by-core", self.cores.len()), + ] + } + + /// Whether the independently-read Win32 counters agree with what the + /// topology crate parsed. + /// + /// A disagreement is a real finding: it means the shipping crate's parse of + /// `GetLogicalProcessorInformationEx` diverges from what the simple + /// counters report on this machine. + #[must_use] + pub fn cross_check(&self) -> Vec { + let mut complaints = Vec::new(); + if self.online_processors != self.raw_active_processors as usize { + complaints.push(format!( + "online processors: topology crate says {}, GetActiveProcessorCount says {}", + self.online_processors, self.raw_active_processors + )); + } + if self.groups != self.raw_group_count as usize { + complaints.push(format!( + "groups: topology crate says {}, GetActiveProcessorGroupCount says {}", + self.groups, self.raw_group_count + )); + } + if let Some(highest) = self.raw_highest_numa_node + && self.highest_numa_node != Some(highest) + { + // Highest against highest, deliberately, and not a count against + // `highest + 1`. `GetNumaHighestNodeNumber` reports the largest node + // *number*, which Windows does not promise equals the node count -- + // nodes 0 and 2 are a valid sparse topology, and the count form + // would report a regression on hardware that is reporting itself + // correctly. + complaints.push(format!( + "NUMA nodes: topology crate's highest node is {}, GetNumaHighestNodeNumber says {}", + self.highest_numa_node + .map_or_else(|| "none".to_string(), |n| n.to_string()), + highest + )); + } + complaints + } +} + +/// Discover the machine's shape. +/// +/// # Errors +/// +/// Propagates a failure from [`MachineMemoryTopology::discover`]. +pub fn measure() -> io::Result { + let topology = MachineMemoryTopology::discover()?; + + let online_processors = topology.processors.iter().filter(|p| p.online).count(); + + let mut groups = 0usize; + let mut numa_domains = 0usize; + let mut memoryless_numa_domains = 0usize; + let mut highest_numa_node: Option = None; + let mut packages = 0usize; + let mut cores = Vec::new(); + let mut by_level: Vec<(u8, Vec)> = Vec::new(); + + for domain in &topology.domains { + match &domain.kind { + DomainKind::Group => groups += 1, + DomainKind::Package => packages += 1, + DomainKind::Memory { .. } => { + numa_domains += 1; + // The NUMA *node number*, which is specifically what the + // relationship walk reports. Not "the id": a relation may now + // carry a second label from CPU Sets that numbers nodes its own + // way (D-15), so the source is named rather than assumed. + if let Some(node) = domain.label_from(Source::RelationshipWalk) { + highest_numa_node = + Some(highest_numa_node.map_or(node, |seen: u32| seen.max(node))); + } + if domain.processors.is_empty() { + memoryless_numa_domains += 1; + } + } + DomainKind::Core { + simultaneous_multithreading, + efficiency_class, + } => cores.push(CoreShape { + simultaneous_multithreading: *simultaneous_multithreading, + efficiency_class: *efficiency_class, + processors: domain.processors.len(), + }), + _ => {} + } + } + + // Asked of the topology rather than counted from `domains` above, because + // Windows reports one relationship per *cache* and not per partition. + // Measured here: L1 arrives as eight `data` domains plus eight + // `instruction` domains over the same eight processor pairs, so counting + // relationships printed "L1 16 domain(s)" on a machine with eight L1 + // partitions -- and fed a doubled count to every policy in + // `domain_counts`. + for level in topology.cache_levels() { + let spans = topology + .cache_partitions_at_level(level) + .iter() + .map(|domain| domain.processors.len()) + .collect(); + by_level.push((level, spans)); + } + + by_level.sort_by_key(|(level, _)| *level); + let caches = by_level + .into_iter() + .map(|(level, processors_per_domain)| CacheLevel { + level, + domains: processors_per_domain.len(), + processors_per_domain, + }) + .collect(); + + // SAFETY: both take no pointer arguments and cannot fail in a way that + // matters here; `ALL_PROCESSOR_GROUPS` is the documented way to ask for the + // machine-wide count. + let raw_active_processors = unsafe { GetActiveProcessorCount(ALL_PROCESSOR_GROUPS) }; + let raw_group_count = unsafe { GetActiveProcessorGroupCount() }; + + let mut highest = 0u32; + // SAFETY: `highest` is a live local for the duration of the call. + let raw_highest_numa_node = if unsafe { GetNumaHighestNodeNumber(&raw mut highest) } != 0 { + Some(highest) + } else { + None + }; + + // Asked once, here, rather than restated: the crate that owns the topology + // owns the rule (D-21). + let partitioning_cache_level = topology + .outermost_partitioning_cache() + .map(|(level, _)| level); + + Ok(Observation { + online_processors, + groups, + numa_domains, + memoryless_numa_domains, + highest_numa_node, + packages, + cores, + caches, + partitioning_cache_level, + raw_active_processors, + raw_group_count, + raw_highest_numa_node, + }) +} diff --git a/crates/windows-topology-sys/CHECKLIST.md b/crates/windows-topology-sys/CHECKLIST.md index 27885baa..d72db2ca 100644 --- a/crates/windows-topology-sys/CHECKLIST.md +++ b/crates/windows-topology-sys/CHECKLIST.md @@ -19,5 +19,5 @@ Two things were deliberately left out of the reshape rather than forgotten: - **The planner adapters.** Per [D-21](DESIGN-NOTES.md#d-21) this crate is the refined view of what the platform publishes and is self-justified as such; the adapter onto - topology-planner's traits belongs on the planner's side of the + [topology-planner](../topology-planner/CHECKLIST.md)'s traits belongs on the planner's side of the boundary, and is planned there. diff --git a/crates/windows-topology-sys/COMPLETED-CHECKLIST.md b/crates/windows-topology-sys/COMPLETED-CHECKLIST.md index 96cd5e4e..58bcc1fa 100644 --- a/crates/windows-topology-sys/COMPLETED-CHECKLIST.md +++ b/crates/windows-topology-sys/COMPLETED-CHECKLIST.md @@ -98,7 +98,7 @@ gaps the crate did not have and were closed as already-satisfied or re-planned r implemented; the notes below record which, and why. Design record: [DESIGN-NOTES.md](DESIGN-NOTES.md) `D-13` through `D-23`, and -DESIGN-SESSION-2026-09-02-cache-locality-model.md. +[DESIGN-SESSION-2026-09-02-cache-locality-model.md](../../design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md). The context the milestones were written against, preserved because the items refer to it: ## What this is for @@ -213,7 +213,7 @@ be wrong and breaking again later. `discover()` returns a topology stale the instant it returns, so the two-call window is only a larger instance of an unavoidable problem. True, and **not a reason to do nothing**: the two are not equally addressable. Staleness after the fact is the executor's to validate, and is already - owned as `M-inf.1` in topology-planner. + owned as `M-inf.1` in [topology-planner](../topology-planner/CHECKLIST.md). Incoherence *during* collection is ours, detectable, and cheap to fix. The framing is what caused the miss. Asking "what do we **store** when sources disagree" admits @@ -248,9 +248,9 @@ be wrong and breaking again later. far the parts may be **correlated** -- a different question from whether any one part is accurate. Turning that into something actionable, with the identifying provenance an actionable report needs, is the probe tools' job and is tracked as **M7** in - CHECKLIST-placement-tool.md. + [CHECKLIST-placement-tool.md](../../CHECKLIST-placement-tool.md). > **-> CROSS-COMPONENT HANDOFF:** the reporting half is `PT-7.1` and `PT-7.2` in - > CHECKLIST-placement-tool.md. That tool already carries the + > [CHECKLIST-placement-tool.md](../../CHECKLIST-placement-tool.md). That tool already carries the > review this needs -- the runner sees real values before sending (`PT-4.5`), the README lists what > is collected (`PT-4.3`), and suppression is recorded rather than merely absent. Only possible because [D-15](DESIGN-NOTES.md#d-15) keeps both observations: a disagreement cannot @@ -320,7 +320,7 @@ be wrong and breaking again later. landed in a "decisions that shape everything below" milestone because it *looked* foundational, and foundational-looking is not the same as being about this component. > **-> CROSS-COMPONENT HANDOFF:** the behavioural half is `EP-1.4` in - > topology-planner. It no longer has a counterpart here, so it + > [topology-planner](../topology-planner/CHECKLIST.md). It no longer has a counterpart here, so it > is that component's decision alone rather than a joint one. - [x] **MMT-1.4** -- **Does `distances` survive at all?** The two-component architecture says the @@ -339,12 +339,12 @@ be wrong and breaking again later. - [x] **MMT-1.5** -- **Does the synthesizer live in this crate, and therefore what is this crate called?** Recorded as open rather than settled: see - topology-planner/COMPONENT.md. The naming follows + [topology-planner/COMPONENT.md](../topology-planner/COMPONENT.md). The naming follows the merge rather than leading it -- while this crate is only a Win32 wrapper, `-sys` is correct for it; if it gains a synthesizer that measures, it stops being one and the name should change then. **Answered by the engineer's architectural shift, recorded as - EP-D-4: no.** The planner is a separate + [EP-D-4](../topology-planner/DESIGN-NOTES.md#ep-d-4): no.** The planner is a separate component named **`topology-planner`** -- with no `windows-` prefix, because it plans against an abstracted idealized machine and emits a platform-neutral plan. So this crate does not gain the synthesizer, remains a pure Win32 wrapper, and **keeps its name**. @@ -627,12 +627,12 @@ facts about processors and memory stated without sentinels; `M4+.4` fixes a rule restated three times in two crates. Only `M4+.1`'s pairwise helper is consumer-flavoured, and the ordered collection it derives from is what stops that restatement recurring. -They remain cross-referenced to topology-planner as +They remain cross-referenced to [topology-planner](../topology-planner/DESIGN-NOTES.md) as **evidence** the shape is right rather than as its justification -- stating those requirements found the `Processor::capacity` sentinel collision that reviewing the model alone had not. - [x] **M4+.1** -- **The ordered relations are the query surface; pairwise proximity is a method on - them.** The requirement arrived from EP-D-2 as a + them.** The requirement arrived from [EP-D-2](../topology-planner/DESIGN-NOTES.md#ep-d-2) as a *pairwise* query returning the minimal shared granularities, **their membership**, and whether a finer granularity went **unobserved** so the answer can be an upper bound and say so. All three requirements stand. The **shape** does not, and the requirement says so itself: it asks the answer diff --git a/crates/windows-topology-sys/COMPLETED-PLANS.md b/crates/windows-topology-sys/COMPLETED-PLANS.md index aa2afcc2..090accaf 100644 --- a/crates/windows-topology-sys/COMPLETED-PLANS.md +++ b/crates/windows-topology-sys/COMPLETED-PLANS.md @@ -7,5 +7,5 @@ was finished. Individual milestones are archived in [COMPLETED-CHECKLIST.md](COM | Path to CHECKLIST.md | Completion Date | Brief description | Design Notes | |---|---|---|---| | [CHECKLIST.md](CHECKLIST.md) | 2026-08-22 | M1-M4: safe enumeration of Windows processor, cache, and memory topology (a walk-by-`Size`, trailing-array-respecting wrapper over `GetLogicalProcessorInformationEx`), the open-kinded `Domain`/`Topology` description (including a memory domain with no processors, for CXL-shaped systems), JSON serialization behind a default-off `serde` feature with the schema explicitly not semver-covered, and crate documentation plus a worked example printing the host's topology. Unblocks `windows-ioring-sys`'s `M7` (`ring-copy`). | [DESIGN-NOTES.md](DESIGN-NOTES.md) | -| [CHECKLIST.md](CHECKLIST.md) | 2026-09-03 | **MMT-*: reshaping the machine memory topology.** Replaced the ladder-of-levels model with **observed connectivity**: relations held as a set with per-relation provenance rather than reduced on insert (`Observation`, `Source`), presence and observation represented as facts rather than inferred (`Observed`, adopted for `memory_bytes` and `cache_domain`), a granularity order with a `minimal_shared` meet, and the pairwise `proximity` query *derived* from an inclusion-ordered partitioning rather than restated -- which removed the third statement of the partitioning rule. Also dropped `distances` and `Domain::id`, folded CPU Sets into the relation set, and recorded per-processor attribute conflicts. Five of the planned items turned out to assert gaps the crate did not have and were closed as already-satisfied or re-planned. Three breaking changes; the crate goes to **0.2.0**. | [DESIGN-NOTES.md](DESIGN-NOTES.md) (`D-13`-`D-23`), design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md | +| [CHECKLIST.md](CHECKLIST.md) | 2026-09-03 | **MMT-*: reshaping the machine memory topology.** Replaced the ladder-of-levels model with **observed connectivity**: relations held as a set with per-relation provenance rather than reduced on insert (`Observation`, `Source`), presence and observation represented as facts rather than inferred (`Observed`, adopted for `memory_bytes` and `cache_domain`), a granularity order with a `minimal_shared` meet, and the pairwise `proximity` query *derived* from an inclusion-ordered partitioning rather than restated -- which removed the third statement of the partitioning rule. Also dropped `distances` and `Domain::id`, folded CPU Sets into the relation set, and recorded per-processor attribute conflicts. Five of the planned items turned out to assert gaps the crate did not have and were closed as already-satisfied or re-planned. Three breaking changes; the crate goes to **0.2.0**. | [DESIGN-NOTES.md](DESIGN-NOTES.md) (`D-13`-`D-23`), [design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md](../../design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md) | | [CHECKLIST.md](CHECKLIST.md) | 2026-09-04 | **M6: one record walk, per [D-24](DESIGN-NOTES.md#d-24).** The PR #56 diff review found the crate's two record decoders internally coherent and mutually opposite: `cpu_set` bounded every read and stopped on a bad `Size`; `walk` proved one byte, `assert!`ed on a zero `Size`, and read `GroupCount` x 16 bytes unbounded (up to 1,048,560). Resolved by one shared self-bounding walk in [records.rs](src/records.rs) that never panics, records incoherence in the returned data as `enumeration_anomalies`, and draws no trust boundary -- the operating system is relied on for the structural validity of a buffer it just wrote, and careful walking is correct traversal rather than validation. | [DESIGN-NOTES.md](DESIGN-NOTES.md) (`D-24`) | diff --git a/crates/windows-topology-sys/DESIGN-NOTES.md b/crates/windows-topology-sys/DESIGN-NOTES.md index 3138f022..0194d38f 100644 --- a/crates/windows-topology-sys/DESIGN-NOTES.md +++ b/crates/windows-topology-sys/DESIGN-NOTES.md @@ -137,7 +137,7 @@ distinguishes them. That is not fixable by documentation, because the two really are the same value today. It is fixed by the representation, which is the subject of the open locality-model work -- see `SH-16.8` in -CHECKLIST-ship-topology-and-queues.md, where absence +[CHECKLIST-ship-topology-and-queues.md](../../CHECKLIST-ship-topology-and-queues.md), where absence becomes first-class rather than a shape. Recorded here so the gap is not rediscovered, and queued there so it is not merely recorded. @@ -540,7 +540,7 @@ the same requirement. ## D-20: the Win32 boundary, and the deletion of `distances` *Recorded by [CHECKLIST.md](COMPLETED-CHECKLIST.md) MMT-1.4. Supersedes `SH-16.11` in -CHECKLIST-ship-topology-and-queues.md, which proposed +[CHECKLIST-ship-topology-and-queues.md](../../CHECKLIST-ship-topology-and-queues.md), which proposed filling the field.* ### The ruling is about scope, not about the field diff --git a/crates/windows-topology-sys/src/granularity.rs b/crates/windows-topology-sys/src/granularity.rs index 2b6b9ff0..3f4c525f 100644 --- a/crates/windows-topology-sys/src/granularity.rs +++ b/crates/windows-topology-sys/src/granularity.rs @@ -317,12 +317,13 @@ pub struct Proximity<'a> { /// processors appears in *no* instance of, so the platform has said nothing /// about whether they share it. /// - /// This is the third of `EP-D-2`'s requirements, and the one a naive design + /// This is the third of [EP-D-2]'s requirements, and the one a naive design /// drops. A caller told "the tightest shared thing is L3" when in truth L2 /// was never reported for this processor would choose a slower channel than /// the machine can support and never learn why -- and under this crate's own /// bar it cannot go and measure to find out. /// + /// [EP-D-2]: ../../topology-planner/DESIGN-NOTES.md pub finer_unobserved: bool, /// Processors the query named that this platform cannot express, and which /// therefore took no part in [`Self::shared`]. diff --git a/design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md b/design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md new file mode 100644 index 00000000..6a4e894d --- /dev/null +++ b/design-sessions/DESIGN-SESSION-2026-09-02-cache-locality-model.md @@ -0,0 +1,526 @@ +# Design session: the cache-locality model + +**Status: OPEN, with direction settled and the representation converging.** The engineer has +taken sides on both underlying questions (see "Direction taken" below) and settled three +sub-questions about the proposed shape: provenance is **per-relation**, "determined absent" +is a **distinct record**, and the whole-object `Provenance` is **superseded** rather than +re-derived. Nothing is carried over from the old model for its own sake; two of its +properties are kept only because they re-derive independently. The +options section further down predates that direction and is kept as a record of what was +considered -- Options 1 and 2 are now insufficient on their own, because both preserve the +`Option`-shaped absence the direction rejects. + +**This session is on PR #56's critical path.** The work it gates is in scope for that PR by +decision -- #56 does not merge until the new model lands -- because the model being replaced is +the one `windows-topology-sys` 0.2.0 would publish, and a published model cannot be reshaped +without another break. So this session concludes before implementation starts, and +implementation lands before the PR is described or promoted. + +Prompted during PR #56's tenth review round while fixing +[SH-16.5](../CHECKLIST-ship-topology-and-queues.md). That item is **blocked on this +session** and must not be implemented before it concludes: the primitive SH-16.5 was about +to add is itself the thing under design. + +A working prototype of the SH-16.5 fix was written and then reverted so as not to prejudge +the outcome. It is preserved outside the repository, in this session's agent workspace, as +`sh-16.5-prototype.patch`. It compiled, and its topology-side tests passed and were +sabotage-verified; it is evidence about one option, not a commitment to it. + +## How the question arose + +SH-16.5 reported a contract contradiction: `windows-topology-sys` documents that a +partitioning cache level is *not* required to cover every online processor, while +`windows-placement-probe` treats any uncovered processor as corruption and fails the whole +run. The agreed direction was to state the rule once, in the crate that owns the topology, +and have the consumer ask. + +Mid-implementation the engineer raised a broader objection, which is the actual subject +here: + +> i think the problem is that the model has to acknowledge more than exactly 3 levels of +> caching. why not 1? 5? where are the write buffers modeled? i am not saying we have to +> start over from scratch, but it is somewhat ironic that in a crate named "topology", we +> have a network where we assume 3 members and only having 2 confuses it + +## Direction taken + +The engineer separated the subject into two questions and answered both, then gave a +constraint on the representation. Recorded in their terms: + +**1. What model does the Windows API set represent?** Do we expose all the topological +nuance that a real system, described through that API, would reveal -- "not every mechanical +combination of representable values: a reasoned set of logical derived models that would be +represented through the Win32 API set". So the target is completeness with respect to what +Win32 can express about a real machine, not an arbitrary product of enum values. + +**2. What memory-hierarchy concepts might a system we encounter have, whether or not Win32 +exposes them?** These affect analysis regardless of API exposure, and the `-probe` crates +exist partly to establish them by measurement. + +**3. The representation constraint, which is the operative decision.** Where a level of +hierarchy may or may not be present, do **not** model it as a second-class `Option`, because +that conflates two different facts: *Win32 did not provide data* and *the level was +specifically found not to be present*. Instead choose a representation that is designed for +the topology to **represent the observed connectivity**. + +This supersedes the framing further down that treated "collapse to one boundary" as the whole +problem. The collapse is a symptom; the cause is that presence and observation are not +modeled. + +### What this rules in and out + +- Out: `Option`, and equally `Option` plus a side boolean, as the way to say a level is + missing. Also out: the `CachePlacement` prototype from SH-16.5, whose `Unknown` arm merges + "not reported" with "reported, does not name this processor". +- In: a representation where a sharing relation that was *observed not to exist* and one that + was *never observed* are different values, and where a measured relation can sit beside a + firmware-reported one. +- Still open: the concrete shape. See "Proposed representation" below. + +## New evidence gathered after the direction was set + +**The principle is already in-house, written down, and applied inconsistently.** +`MachineDescription::cpu_model` in `windows-placement-probe` records: + +> Suppression is recorded in `model_suppressed` rather than left to be inferred from absence: +> a field withheld by the runner and a field the host would not answer are different facts, +> and a collector that cannot tell them apart will eventually read one as the other. + +That is the engineer's point exactly, reached independently for a different field. It is +solved there with `Option` plus a side boolean, which is the weaker form the direction above +rules out -- but the *reasoning* is settled precedent in this repository, not a new claim. + +**Win32 is not fully consumed: CPU Sets is entirely absent.** The crate consumes seven +`GetLogicalProcessorInformationEx` relations (`ProcessorCore`, `ProcessorPackage`, +`ProcessorDie`, `ProcessorModule`, `Cache`, `NumaNode`, `Group`), which is essentially all of +that API. But `GetSystemCpuSetInformation` / `SYSTEM_CPU_SET_INFORMATION` is not referenced +anywhere in the workspace, and it is a **second, parallel topology model** Windows offers, +carrying at least: `LastLevelCacheIndex` (Windows's own LLC grouping, which is a *different* +answer from "outermost partitioning cache"), `SchedulingClass`, `AllocationTag`, +`EfficiencyClass`, and per-CPU `Parked` / `Allocated` / `RealTime` state. Verify the exact +field list against the SDK before relying on it. This is directly responsive to question 1: +the answer today is **no**, there is a whole Win32 model unexposed. + +**Nothing currently infers a hierarchy level from measurement.** The engineer suspected the +`-probe` crates might already do this and flagged uncertainty. Checked: they do not. The +probes measure *cost per firmware-reported placement* (`core_affinity` times handoffs between +pairs already classified from the topology), and the NUMA spikes infer *policy* -- first-touch +versus creator affinity, per-volume versus per-file -- not structure. `Provenance` already has +a `Measured` variant, but it qualifies a whole `MachineMemoryTopology`, not an individual relation. So the +capability question 2 describes is **new work, not a retrofit**, and the per-relation +provenance it needs does not exist yet. + +**The "outermost partitioning cache" rule is stated three times, and two of the three +disagree.** `MachineMemoryTopology::outermost_partitioning_cache` requires more than one partition **and** +pairwise disjointness. `Observation::outermost_partitioning_cache` in +`windows-platform-probes` is `caches.iter().filter(|c| c.domains > 1).max_by_key(|c| c.level)` +-- no disjointness check -- over a `CacheLevel` summary it builds itself, even though that +crate does depend on `windows-topology-sys`. On a hand-built or deserialized topology with +overlapping domains the two answer differently. `windows-placement-probe` restates it a third +time by rebuilding the map from the partition list, which is SH-16.5. Tracked separately as +SH-16.9. + +## Proposed representation, for reaction + +Offered as a starting shape, not a conclusion. + +Stop modeling a *ladder of levels with optional rungs* and model the *observed sharing +relations* directly. A topology becomes a set of relations, each carrying: + +- **what is shared** -- cache at level N, module, die, package, memory domain; +- **which processors share it** -- the `ProcessorSet` already used; +- **how it was established** -- reported by a named Win32 source, measured by a named probe, + or determined absent. + +Presence then stops being an `Option`. A machine with no L3 has *no L3 relation in the set*, +and that is an observation rather than a missing value; a machine whose firmware was not +queried for L3 carries a *not-observed* record for it. The two are different members, not the +same `None`. + +Connectivity queries follow from it: "at which relations do A and B share?" returns the +observed set, and the difference between *they share nothing* (an empty answer over complete +observations) and *we do not know* (incomplete observations) is representable rather than +collapsed. `outermost_partitioning_cache` survives as one named projection over that set, for +the scheduler question of "give me exactly one boundary to shard on", and is documented as a +projection rather than as the model. + +### Sub-questions, answered + +**Provenance is per-relation.** Asked for a case where it could differ per *source*; there +is none worth having. Any per-source fact is expressible per-relation by repetition, and the +reverse is not true -- so per-relation strictly subsumes it. The case that decides it runs the +other way: **two sources describing the same relation**. Win32 reporting that A and B share +L3 while a probe measures otherwise is expressible only if one relation can hold both +observations; per-source would force two whole topologies and a diff. + +What the per-source instinct was actually reaching for is not provenance but **completeness +of an observation attempt** -- "source S was queried about dies and said nothing" cannot +attach to a relation, because there is no relation. That is the absence record, settled +below. + +**"Determined absent" is a distinct record**, not a relation with an empty processor set. An +empty set already means something else here (`memory_domains` deliberately keeps a +processor-less memory domain, D-5), so overloading it would be a trap. + +**The whole-object `Provenance` is superseded, and should not be replaced by another +whole-object scalar.** Derivation: with trust per-relation, an object-level scalar can only be +the minimum (a topology with ninety-nine measured relations and one synthetic reads +`SYNTHETIC`, which is useless) or the maximum (which is dishonest). Trust belongs to an +**answer** -- "A and B share L3, established by these observations" carries its own -- and that +falls directly out of modeling observed connectivity, since a connectivity model exists to +answer queries and the query result is the thing needing a label. + +### Two questions those answers open + +**Is provenance a scalar or a chain?** Today it is a scalar, and deserialization is a *lossy* +downgrade: `downgraded_to` is `min` against a `Restored` ceiling. A measured relation that +round-trips through a file loses "originally measured, on this machine, at this time", which +matters more per-relation because a measured relation is expensive to establish. From base +principles these are two things one scalar was forced to conflate: **trust assertable now** +(never upgradeable) and **origin history** (recorded, conferring no trust). + +**Can one relation hold more than one observation?** It probably must, and the project already +reasons this way. From `file-handle-numa-spike.rs`: + +> **Agreement is consistent with volume locality; it does not establish it.** A genuinely +> per-file answer may equal its volume's node ... so one file agreeing rules nothing out. Only +> disagreement is decisive, because a per-volume answer cannot differ from itself. + +That is an asymmetric adjudication rule over two independent observations of one underlying +fact, and it only works if the observations coexist. A model storing one winning value per +relation cannot express it -- and detecting a hypervisor that misreports topology is exactly +this shape. + +### Two principles that re-derive rather than being inherited + +The direction is explicitly to find the right model rather than carry anything over. Two +properties of the old `Provenance` are principles rather than model, and both survive that +test on their own merits: + +- **The default is the untrusted value.** Under per-relation provenance this argument is + *stronger*, not weaker: there are far more places to forget. +- **Trust never upgrades.** Identical derivation -- a file still cannot establish that it + describes the machine you are on. + +### Provenance is a scalar, and the topology is a point in time + +Settled. The topology describes **a particular instance at a point in time**, not a historical +record. There may be room for both eventually, but history is deliberately out of scope now: +once historical record is allowed, a much wider set of sources has to be reconciled and the +whole enterprise becomes a mess. The model works from the best data available right now. + +**This does not license paring the model down to what today's consumers need.** The engineer +was explicit, and the repository's history supports it: this project has repeatedly found the +information it had was inadequate -- the ARM64 host with no L3 that forced "outermost level +that partitions" rather than "level 3"; the guard test against a consumer sweeping `1..=4`; +group-awareness, where "a bare `cpu5` cannot tell a reader whether the group was considered and +was zero, or never consulted at all"; and `machine.rs` distinguishing a withheld field from an +unanswerable one. Foreclosing on any single moment's understanding risks losing exactly what is +needed next. + +The two are not in tension, because they are different axes: **breadth in structure, narrowness +in time.** A point-in-time snapshot can be structurally complete. History is the axis that +drags in multiple sources and reconciliation. + +### How concrete: "usable without further measurement" + +The bar, in the engineer's words, is that the abstract model be massaged into something usable +for shaping memory allocations, thread counts and assignments, and ring topology shapes, +**without further measurement -- the model answers from what was already observed, with no +probing at decision time.** + +Three consequences follow, and the third is architectural. + +**1. Measured facts must live in the model, not only in probe output.** If a decision needs a +fact only measurement can supply, and probing at decision time is forbidden, the measurement +has to already be there. This is why per-relation provenance is load-bearing rather than +decorative: a consumer must be able to see whether "these share L3" came from firmware or from +a probe, and cannot go and check. + +**2. The not-observed record gains a second job.** A consumer needing an unmeasured fact +*cannot acquire it*. So the model must say "not measured" plainly and let the caller degrade +deliberately, rather than presenting an absence the caller silently reads as a value. + +**3. There must be an explicit measurement phase on the real machine.** Combined with "trust +never upgrades", this rules out shipping a pre-measured topology: a file caps at `Restored`, so +its measurements cannot be trusted as *this* machine's. And lazy measurement on first need is +just probing at decision time. So the model acquires a lifecycle -- observe (cheap, firmware), +measure (expensive, on-machine), decide (no I/O) -- and something has to own the middle phase. + +### The canonical case, already half-built: NUMA distances + +`MachineMemoryTopology::distances: Option` exists, and every path sets it to `None`. +`MachineMemoryTopology::discover` hardcodes `distances: None`; no consumer reads the field. Meanwhile +**Win32 cannot supply it** -- ACPI carries SLIT, but no Win32 API surfaces node distances -- and +`windows-placement-probe` **already measures the equivalent**, via `node_pairs_measured()`, +producing per-node-pair handoff cost with ring placement and rendering it as a table. + +So the fact is needed, the field exists, the measurement exists, and nothing connects them. A +consumer shaping memory allocation today must either run the probe at decision time, which the +bar above forbids, or guess. This is the whole design in one field, and it is tracked as +SH-16.11. + +### The consumer this is being designed for + +"Most useful for consumers" was answered by naming one: a Seastar-style shard-per-core runtime +building SPSC/MPSC rings between pinned threads over NUMA-local buffers. Walking that construction +produced a requirements list, and it is now owned by +[crates/topology-planner](../crates/topology-planner/CHECKLIST.md) M1 rather than being +carried in this session as prose. + +The walk found the load-bearing query is **pairwise proximity** -- "how close are these two +processors" -- asked once per pair of shards, because that is what selects SPSC versus MPSC versus a +routed hop. The current model answers only the *global* question (`outermost_partitioning_cache`, +one level for the whole machine) reduced to a boolean at that level (`same_cache_domain`), so the +pairwise question has no answer at all today. + +That settles the vocabulary argument on use rather than on aesthetics. Under ordering-by-inclusion, +pairwise proximity is one query over the order. Under a firmware-anchored ladder it is a fixed +sequence of "same L1? same L2? same L3? same die? same node?", which breaks on the ARM64 host with +no L3 and cannot express a measured-only tier at all. + +The walk also found the mapping itself was **unowned**: `CHECKLIST-io-domains.md` M32 lists four +contracts "the runtime cannot be written without" and all four concern the queue, while M33+.1 +presupposes a plan naming which thread, which node and which shard. That is now a component. + +### The consumer's requirements, stated + +Three queries, recorded in full as EP-D-1, EP-D-2 and EP-D-3 in +[the planner's design notes](../crates/topology-planner/DESIGN-NOTES.md). Summarised here +because a model designed without them in view is what produced the current one. + +| Query | Shape | What the model must answer | +|---|---|---| +| **Shard set** (EP-D-1) | per processor | identity as `(group, number)`; online; core membership and SMT; efficiency class **without a sentinel**; and availability -- parked, and allocated to *this* process | +| **Proximity** (EP-D-2) | **unordered** pair | the minimal granularities the two share, **their membership** (to size an MPSC fan-in without re-deriving the grouping), and whether a **finer granularity went unobserved**, so an answer can be an upper bound and say so | +| **Residency** (EP-D-3) | **ordered** pair | processor-to-memory-domain with the unplaced case distinguishable, and a **directed** cost between memory domains, which SLIT's symmetric scalar cannot express | + +Four properties of the model follow from those and are worth stating as requirements rather than +leaving implicit in three separate documents: + +1. **A pairwise query must exist.** No query in `windows-topology-sys` takes two processors today, + and that absence is the direct cause of `SH-16.9`'s three inconsistent reconstructions. +2. **The order must be total**, which needs an explicit "the machine" top granularity -- otherwise + every caller writes the same empty-case branch for a cross-node pair. +3. **An answer must be able to be an upper bound.** "Tightest shared is L3" and "at most L3, finer + not observed" are different answers, and under the no-probing bar the planner cannot go and check. +4. **A measured number must carry what it measured.** The probe's figures are nanoseconds for one + ring-handoff pattern at one message size; promoting them as "the distance" would bake one + workload into a model other consumers share. + +**Where the requirements stop.** They say what the planner must be able to *ask*. They do not say +what it should *do* when the answer is "not observed" -- that is this session's fourth open question +and the planner's EP-1.4, which are the same decision seen from two ends and must be taken together. + +### The two-component architecture, and who measures + +Settled by the engineer, and it answers the measurement question below rather than adding to it. + +There are **two** things, and both are graphs of processors and their relations, which is why +calling both "topology" has been confusing: + +1. **What the machine *is*.** Read from the Windows data model, plus whatever else is trivially + available. Observed, never chosen. This is today's `MachineMemoryTopology`, and it is **mockable** -- a + description of a machine nobody has is a first-class input, which is what makes the second + component testable. + +2. **What we are going to *build* on it.** A concrete description of the arrangement to construct: + which processors host domains, which threads pin where, which rings connect them, where each + buffer lives. + +The second component synthesizes the second from the first, and it takes **two** inputs, not one: + +- the observed machine, and +- **a description of the desired function** -- the scenario. This is the input the design has been + missing, and its absence is why "what is most useful for consumers" kept being hard to answer in + the abstract. + +It may also **call back to its caller** through traits, to ask for clarifying information the +scenario did not settle. So planning is a negotiation rather than a pure function. + +**And it is the component that measures, with the caller's permission**, to determine the optimal +arrangement *for that scenario*. + +### What this resolves + +**Who owns the measurement phase: the synthesizer, permissioned.** Not `discover()`, and not an +enrich step on the topology. This is better than either, for a reason the session had already found +without drawing the conclusion: EP-D-3 established that a measured number is only meaningful +alongside *what it measured* -- the probe's figures are nanoseconds for one ring-handoff pattern at +one message size. A component that knows the scenario can measure the right thing; a `discover()` +that measures cannot, because it does not know what the caller intends to do. + +**The no-probing bar survives, sharpened.** The *observed* topology never measures, so it remains +usable without further measurement. The synthesizer may measure, but that is a distinct, +permissioned, scenario-specific activity producing a **plan**. The plan, once produced, is consumed +without further measurement. Three stages, each honest about its cost: observe (cheap), synthesize +(may measure, with permission), execute (no I/O). + +### A consequence that needs confirming + +If the synthesizer measures for its own scenario, then **measured facts may not belong in the +observed topology at all.** The session earlier concluded they must, on the grounds that a consumer +forbidden from probing needs them present -- but that reasoning assumed one component. With two, the +measurement is the synthesizer's working state and its justification for a choice, not a property of +the machine. + +That would make the observed topology purely what Windows reports, and it would mean +`MachineMemoryTopology::distances` is **deleted rather than filled** -- which is a cleaner answer than SH-16.11's, +and the opposite of what that item currently proposes. Flagged rather than acted on, because it +reverses a conclusion this session reached earlier and should be confirmed before anything is +removed. + +### Still open + +- ~~Who owns the measurement phase?~~ **Answered above: the synthesizer, with permission.** +- **What shape is the scenario input?** It is the newly-named second input and nothing describes it + yet. EP-D-3's finding constrains it: it must carry enough for a measurement to be meaningful, + which at minimum distinguishes small-message handoff from large-buffer streaming. +- **What do the caller-callback traits ask?** Knowing which questions cannot be answered from the + scenario alone is what decides whether this is one trait or several. +- Whether multiple observations per relation are held as a set, or reduced on insert with the + reduction recorded. +- What a query returns when observations disagree: a value plus a conflict marker, or the + conflict itself, forcing the caller to adjudicate. +- What a consumer does when a needed fact is `not measured` -- is degrading its choice, or does + the model offer a documented fallback? + **This is the same decision as the planner's EP-1.4**, seen from the model's side rather than the + consumer's, and the two were filed independently before anyone noticed. They must be taken + together: answering either alone risks a planner that degrades in a way the model does not + support, or a model offering a fallback no consumer wants. EP-1.4 is blocked on this question + rather than on the model as a whole. + +## What the code actually does, verified rather than assumed + +The objection is half right, and the halves point at different files. + +**The base model hardcodes no level count, and that was deliberate.** `DomainKind::Cache` +carries `level: u8`; `MachineMemoryTopology::cache_levels()` returns whatever the firmware reported, +sorted and deduplicated; `caches_at_level` takes any `u8`. There is already a regression +test, `a_partitioning_cache_above_level_four_is_found`, whose comment reads: "`level` is a +`u8`. A consumer sweeping a hard-coded `1..=4` reports this machine as having no +partitioning cache at all." One level works too, and is a distinct answer +(`outermost_partitioning_cache` returns `None`, meaning "nothing divides this machine", +which the docs are explicit is a real answer and not a failure). + +**The base model is also richer than caches.** `DomainKind` has seven variants -- `Group`, +`Package`, `Die`, `Module`, `Core`, `Cache`, `Memory` -- and `Die` and `Module` are +genuinely populated, from `Record::ProcessorDie` and `Record::ProcessorModule`. So the +crate models a multi-tier locality graph, not a three-level cache. + +So the crate named "topology" does model a network. The collapse is downstream of it. + +## Where the collapse actually lives + +Three sites, in increasing severity: + +1. **`MachineMemoryTopology::outermost_partitioning_cache()`** -- selects exactly one level (outermost + first, requiring more than one pairwise-disjoint domain) and discards every other level. + This one sits *inside* the topology crate, which is where the engineer's irony lands + squarely: the crate offers a rich model and then a lossy convenience view that consumers + bind to instead. + +2. **`ProcessorPlace::cache_domain: Option`** -- one scalar, therefore one level. + +3. **`core_affinity::Placement`** -- three tiers of locality: `SameCoreSiblings`, one cache + boundary (`{Same,Cross}Cache` x `{Same,Cross}Class`), and `CrossNumaNode`. `Package`, + `Die`, and `Module` are absent entirely, and every cache level except the selected one is + absent. + +## Consequences + +**A. The label is not portable across machines.** "Same cache" means *same L2* on the x64 +development host (eight L2 domains, a single non-partitioning L3) and would mean *same L3* +on a two-CCD part where L3 has two disjoint domains. The same word denotes a different +boundary depending on the machine. `HostFingerprint` does record `partitioning_cache_level` +alongside `cache_domain_sizes`, so a reader *can* disambiguate -- but only by consulting a +different field, and nothing in the label says to. + +**B. It has already cost this project a row in its own measurement matrix.** +[DESIGN-NOTES.md](../crates/windows-waitable-queues/DESIGN-NOTES.md) records, of the x64 +host: + +> Conversely this host cannot express `same cache, same class` at all: its outermost +> partitioning cache is L2, shared by exactly the two siblings of one core, so any two +> processors sharing a cache domain are siblings. + +That is attributed to hardware. It is at least half the model: those sixteen processors +**do** all share one L3. A per-level model would express "different L2, same L3, same class" +on that very host, which is precisely the row the note reports as inexpressible. The +neighbouring claim that "neither host alone can produce the full table" is therefore partly +self-inflicted, and worth re-checking against whatever this session concludes. + +**C. Two different localities are conflated on any machine with two live boundaries.** On a +part with several L3 domains and several L2 domains within each, `CrossCache` covers both +"different L2, same L3" and "different L3", which are very different costs. Separating costs +by locality is the probe's entire purpose, and D-28's conclusions about peer-index caching +are keyed to these labels. + +## What is outside the model entirely, and why + +Write buffers, store buffers, and line-fill buffers are **not modelable from this source**. +`GetLogicalProcessorInformationEx` reports caches, cores, modules, dies, packages, groups +and NUMA nodes; it does not report store-buffer topology at all. This is a limit of the OS +surface rather than an omission in the crate, and it should be stated somewhere rather than +left as an implied gap -- the question is reasonable and its absence currently reads as an +oversight. + +Whether a *measured* locality tier (something the placement probe establishes empirically +rather than reading from firmware) belongs in this model is a separate and open question. +Note that `Provenance` already exists to distinguish measured from reported claims, so the +crate has a place to put such a thing if the answer is yes. + +## Design space + +Not mutually exclusive; roughly increasing in cost. + +**Option 1 -- name the projection, change nothing else.** Document that +`outermost_partitioning_cache` is one view and that `Placement`'s three tiers are a +deliberate projection, with the portability caveat (consequence A) stated at both. Cheapest, +and it converts an apparent assumption into a recorded choice. Does not address B or C. + +**Option 2 -- add a level-agnostic primitive beside the projection.** Something in the shape +of `shared_cache_levels(a, b) -> Vec` or `deepest_shared_cache(a, b) -> Option`, so +that "do these share a cache?" becomes "at which levels do these share?". The projection +stays for callers that want one boundary to shard on, which is a legitimate scheduler +question. Unblocks SH-16.5 without deepening the collapse. Does not by itself change what the +probe reports. + +**Option 3 -- make a measurement row name its own boundary.** Reshape `Placement` (or the +row that carries it) so "same cache" is qualified by level. Fixes A and C. Changes +measurement output, so it touches D-28's recorded conclusions and the fingerprint's +comparability across existing records -- which is exactly why it is a decision and not a +refactor. + +**Option 4 -- generalise past caches.** `Package`, `Die`, and `Module` are modeled and +discarded. If locality tiers are the real subject, the projection is arguably +"which is the tightest domain these two share, over all kinds" rather than anything +cache-specific. Largest change; also the one that most directly answers "we have a network, +stop assuming three members". + +## Open questions for the session + +**Settled by the direction above:** question 1 below (the projection is kept, but as a named +projection over a connectivity model, not as the model); question 6 (the write-buffer note +belongs with question 2's measured tier, since that is the only mechanism that could ever +establish one). + +1. ~~Is the single-boundary projection right, or wrong?~~ **Settled: it survives as one + projection among others, and is documented as such.** A scheduler sharding work does want + exactly one boundary; the error was letting that answer be the model. +2. If a row names its level, what happens to existing records and to D-28's conclusions? + Are they re-derivable from what was recorded, or would they need re-measuring? +3. Does the matrix hole in consequence B actually close under a per-level model, on the + hardware available? That is checkable and should be checked before it is claimed. +4. Should `Die` / `Module` / `Package` participate, or is cache-level generality enough? +5. Does a measured (as opposed to firmware-reported) locality tier belong in + `windows-topology-sys` at all, given `Provenance` exists? +6. Where should the note about write buffers being outside the OS surface live? + +## Status of dependent work + +- **SH-16.5 is blocked on this session.** The contradiction it reports is real and still + unfixed; `windows-placement-probe` still refuses a partially-covering level that + `windows-topology-sys` deliberately permits. +- No other M16 item is affected. The other six findings from that round are fixed and + committed.