From a5dbf933855d0a7690336a614c17e816f5dd052f Mon Sep 17 00:00:00 2001 From: Charlie Date: Tue, 1 Sep 2026 02:22:57 +0000 Subject: [PATCH] Refactor documentation: conventional README.md + LLM-optimized AGENTS.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the sprawling root README.md (project overview mixed with an ~1400-line performance investigation log) plus deploy/README.md and deploy/DEBUGGING.md with two files: - README.md: a conventional, concise, human-facing overview — what the project is, layout, build/test/deploy commands, and a pointer to AGENTS.md. - AGENTS.md: the single home for all agent documentation and context (for any tool, not a per-tool file), organized for LLM consumption rather than straight-through reading: - A table of contents and a Quick reference section up front — commands, always-true rules, top footguns, current read-ahead settings — each linking to its full section. - Sections ordered rules-first, reference-second: operating rules (what's expected, the full-pass checklist, documentation discipline, conventions) come early since they apply to every task; build/CI/sanitizers/deploy/debugging/perf reference material follows; the two large evidence/design sections are last since they're read occasionally, not every session. - deploy/DEBUGGING.md folded in as its own section rather than a separate file, with content it duplicated against deploy/README.md (the NMI bugcheck notes, the KdBreakPoint() writeup) consolidated into one place. - The playback-stutter investigation compressed into its conclusions, ruled-out hypotheses, the three built-and-reverted experiments, and the reusable measurement-methodology lessons — full round-by-round measurement history stays recoverable from earlier commits, not reproduced in the file every session loads. - The on-disk cache design retitled "Future work: on-disk hot cache (not implemented)" with a banner marking it an unstarted project. - Conventions extended with kernel-stack conservatism (flat call tree, bounded recursion, prefer pool over large stack frames), false-sharing avoidance (pad/align contended per-processor state to a cache line), and avoiding atomics except where correctness requires them — each grounded in existing precedent in this codebase (Statistics.h's padded per-processor counters and its removal of a shared interlocked gauge; the Fcb/Dcb/Vcb refcounts and OnReapList/ThreadsActive claim flags that keep Interlocked ops because correctness genuinely needs them). No source, build, or CI files touched. --- AGENTS.md | 1698 +++++++++++++++++++++++++ CLAUDE.md | 9 - README.md | 2143 +------------------------------- deploy/DEBUGGING.md | 397 ------ deploy/README.md | 279 ----- tools/Measure-BlorgScaling.ps1 | 2 +- 6 files changed, 1734 insertions(+), 2794 deletions(-) create mode 100644 AGENTS.md delete mode 100644 CLAUDE.md delete mode 100644 deploy/DEBUGGING.md delete mode 100644 deploy/README.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..9011d62 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,1698 @@ +# AGENTS.md + +Operating manual for coding agents working on BlorgFS, and the detailed +reference for human contributors too. Read [README.md](README.md) first for +what the project is and its repository layout; this document covers +everything needed to build, test, deploy, debug and modify the driver +correctly, plus the hard-won findings behind the current design. + +Keep this to two markdown files — `README.md` and `AGENTS.md`. All agent +documentation and context, for any tool, lives here — don't create new +markdown files (a `CLAUDE.md` included) for findings; add a section here +instead (see "Documentation discipline" below). + +**Start with the [Quick reference](#quick-reference) below** for the +condensed, current-state version of everything in this file — commands, +settings, and the top footguns, each linking to the full section for +detail and evidence. The rest of the document is organized rules-first, +reference-second: operating rules and conventions come early because they +apply to every task; the two large evidence sections (the read-ahead +investigation and the disk-cache design) are last because they are read +occasionally, not every session. + +| Section | What it's for | +|---|---| +| [Quick reference](#quick-reference) | Condensed cheat sheet — read this first | +| [What is expected of you here](#what-is-expected-of-you-here) | The bar for changes in this codebase | +| [Changing behaviour: the full pass](#changing-behaviour-the-full-pass) | The checklist a behaviour change must clear | +| [Documentation discipline](#documentation-discipline) | Where findings go, and how to keep docs from going stale | +| [Conventions](#conventions) | Naming, style, and hard rules | +| [Build and test tiers](#build-and-test-tiers) | How to build and run the regression tiers | +| [Continuous integration](#continuous-integration) | What each CI workflow gates | +| [Sanitizers](#sanitizers) | ASan/KASAN requirements | +| [Deploying to a VM](#deploying-to-a-vm) | The deploy pipeline and its quirks | +| [Debugging the VM: what's real and what's noise](#debugging-the-vm-whats-real-and-whats-noise) | Decision tree for VM/debugger flakiness | +| [Measuring performance](#measuring-performance) | How to benchmark correctly | +| [Read-ahead policy: current state](#read-ahead-policy-current-state) | What the driver does today, and why, in one place | +| [Evidence trail: the playback-stutter investigation](#evidence-trail-the-playback-stutter-investigation) | Conclusions and reusable measurement lessons from the investigation behind that policy; full round-by-round history is in git log | +| [Future work: on-disk hot cache (not implemented)](#future-work-on-disk-hot-cache-not-implemented) | Design for a not-yet-started project — nothing in it exists in the codebase | + +## Quick reference + +Condensed, current-state facts. Each row links to the section with the +full detail and supporting evidence — read this first, go deeper only when +a task needs it. + +### Commands + +| Do this | Command | Detail | +|---|---|---| +| Build + test (default gate) | `powershell -File tools/Invoke-BlorgChecks.ps1 -Tier Fast` | [Build and test tiers](#build-and-test-tiers) | +| Deploy to the dev VM | `.\deploy\Deploy-ToVM.ps1 -Configuration Release` | [Deploying to a VM](#deploying-to-a-vm) | +| Benchmark (Release, Verifier off) | `powershell -File deploy/Deploy-ToVM.ps1 -ForBenchmark` | [Measuring performance](#measuring-performance) | +| Accept new perf baseline | `powershell -File tools/Invoke-BlorgChecks.ps1 -Tier Perf -PerfFile -UpdateBaseline` | [Measuring performance](#measuring-performance) | +| KASAN driver build | `msbuild src\BlorgFS.vcxproj -p:Configuration=Debug -p:Platform=x64 -p:EnableKASAN=true` | [Sanitizers](#sanitizers) | + +### Always true + +- **Run `-Tier Fast` before calling any change done.** Cheaper tiers don't + run the crypto vectors that catch a `Tls.c` regression. +- **A behaviour change isn't done when the code works** — update the stale + comment, add a regression test in the layer that can actually catch the + bug, and update docs, in the same pass. See + [Changing behaviour: the full pass](#changing-behaviour-the-full-pass). +- **Never create a new markdown file.** Findings go in a section of this + file. See [Documentation discipline](#documentation-discipline). +- **Benchmark only on Release with Driver Verifier off**, and use + `-ForBenchmark`. A Debug build and/or Verifier bias every number and have + invalidated whole studies before. See + [Measuring performance](#measuring-performance). +- **Never deploy by hand-copying `.sys` into `System32\drivers`.** It skips + catalog validation and can leave a stale binary in place. Use + `Deploy-ToVM.ps1`. +- **A guest reboot is mandatory between driver deploys.** `sc stop` wedges + in `STOP_PENDING` — there is no dismount handler yet. See + [Debugging the VM](#debugging-the-vm-whats-real-and-whats-noise). +- **Keep the kernel call tree flat and the stack usage small.** ~3 pages + total, shared with everything the thread calls; running out is a + bugcheck, not a recoverable error. See [Conventions](#conventions). +- **Data-oriented design; avoid false sharing; avoid atomics unless + correctness requires them.** Lay data out for its access pattern + (per-processor where contended), pad/align contended state to a cache + line, and reach for a non-atomic or per-processor design before an + `Interlocked*` op — this codebase has already removed a shared atomic + gauge in favour of per-processor counters for exactly this reason. See + [Conventions](#conventions). + +### Top footguns (full detail in "Debugging the VM") + +| Symptom | It usually means | Not | +|---|---|---| +| Guest looks completely frozen | `vmrun`/VIX host-side flakiness (often low host RAM), or a `KdBreakPoint()` trap in a **Debug** build | A real bugcheck — confirm via KD: is `System Uptime` advancing? | +| `sc stop BlorgFS` hangs in `STOP_PENDING` | No `IRP_MN_DISMOUNT_VOLUME` handler (known, unfixed) — reboot the guest | A refcount leak (already ruled out) | +| `B:\` gives "semaphore timeout period has expired" | Backend unreachable — the INF's default `RemoteHost` resolves to a dead host; use `10.0.50.17:8080` | A filesystem bug | +| Bugcheck `0x80 NMI_HARDWARE_FAILURE` | Known environmental false alarm (correlates with host memory pressure) — confirm bucket via `Get-CrashVerdict.ps1` | A BlorgFS defect | +| `cmd.exe` via `runProgramInGuest` hangs or returns exit 1 on a healthy guest | A known VIX quirk — drive the guest with `powershell.exe` only | A guest health problem | + +### Read-ahead: current settings + +Full derivation in [Read-ahead policy: current state](#read-ahead-policy-current-state). + +| Setting | Value | Registry override | +|---|---|---| +| Starting granule | 128 KB (every file) | `ReadAheadGranularityKb` | +| Growth ceiling | 2 MB | `ReadAheadMaxGranularityKb` | +| Adaptive feedback loop | on | `ReadAheadAdapt=0` pins the granule | +| Slack-based growth | on | `ReadAheadSlackGrowth=0` disables it | + +### Not implemented — don't assume it exists + +The **on-disk hot cache is an unstarted future project** ([design +doc](#future-work-on-disk-hot-cache-not-implemented)). There is no +`DiskCache.c`/`.h` in the tree yet; nothing described there is live driver +behaviour. + +## What is expected of you here + +This is a kernel filesystem driver on a hot path, so the bar is higher than +"it compiles and the tests pass": + +- **Scrutinise for correctness and performance, actively.** Do not wait to be + pointed at a bug. Read the surrounding code when you touch a file and + question it — the double-counted `NonCachedReads`, the vendor IOCTLs that + could never route, and the unload that never ran were all found by looking, + not by being asked. +- **Prefer optimisations that also simplify.** The best change removes code + and instructions at the same time. Collapsing two branches that test the + same predicate, deleting a redundant pass, replacing pointer-chasing with a + flat array — these are wins twice over. A "fast" change that adds + complexity needs to justify itself with a measurement. +- **Fix style inconsistencies as you find them.** The conventions below are + opinionated and deliberate. Code that drifts from them is a defect; correct + it in passing rather than matching the drift. +- **Measure before and after** anything performance-related. `Statistics.h` + counters are always on, `PerfHarness` reports them, and + `tools\Compare-BlorgMetrics.ps1` diffs against a baseline. An unmeasured + performance claim does not land. + +## Changing behaviour: the full pass + +A behaviour change is not done when the code works. Every one of these, in +the same pass: + +1. **Update the comments that described the old behaviour.** The header + comment above the function, the struct-field comment, the file-header + contract — whichever described what you just changed. A stale comment + actively misleads, which is worse than silence. +2. **Add or update a regression test, in the layer that can actually catch + it.** This is the part most often got wrong. The statistics IOCTL bug + passed every existing dispatch test because they call + `BlorgDeviceControl` directly and so never exercise the I/O manager's + routing decision — the only test that could catch it asserts on the + `CTL_CODE` device type itself. Ask what layer the bug lives in, and put + the test there: crypto vectors for `Tls.c`, the usermode sandbox for + `Client.c` logic, the kernel rule-model for IRQL/lock-order, the + systematic scheduler for interleavings, `Test-BlorgCorrectness.ps1` for + anything only the real driver plus real backend can show. +3. **Comment the test too**, to the same standard — say what it is defending + against and why the obvious cheaper test would not catch it. +4. **Adjust the documentation.** `README.md` for changes to what a human + contributor needs — the project description, build/deploy invocations, + layout. This file (`AGENTS.md`) for everything else: conventions, the + deploy pipeline's mechanics, VM debugging, and performance methodology. +5. **Run `-Tier Fast`** before calling it done. + +## Documentation discipline + +Most of the expensive problems in this project were diagnosed once, in a +session that then ended. The write-ups are the only reason the next person +does not pay for them again — so treat them as part of the work, not an +epilogue. + +**Where findings go.** There are exactly two markdown files in this repo: +`README.md` (a conventional, human-facing project overview) and `AGENTS.md` +(this file — build/test/deploy mechanics, VM and debugger findings, +performance methodology, and conventions). All agent documentation and +context, for any tool, lives in `AGENTS.md` — there is no per-tool file +(`CLAUDE.md` or otherwise). **Do not create new markdown files for +findings.** Add a section to `AGENTS.md` instead, so they stay useful to +whoever comes next regardless of tool. + +**Write down what was *not* true, not just what was.** A ruled-out +hypothesis is worth as much as the root cause, because it is the one the +next person will otherwise re-derive from scratch. "Debugging the VM" +records, for example, that a wedged unload is *not* the drain code and +*not* a refcount leak — both plausible, both wrong, both already checked. + +**Fix staleness in the same pass as the change.** A document describing code +you just altered is now wrong, and a confidently wrong document is worse +than none — it sends the next person down a path that no longer exists. When +you change behaviour, grep the docs for what described it and correct them +before you call the change done: + +```bash +grep -rn "" README.md AGENTS.md +``` + +Two real instances: the `KdBreakPoint` calls documented in "Debugging the +VM" were removed but the paragraph describing them survived; and a memory entry carried a +confidently-argued but incorrect root cause for the statistics IOCTL (a +minifilter theory, proposing an architectural device split) that would have +sent the next agent on a substantial and unnecessary refactor. Delete or +correct superseded conclusions outright rather than leaving them to be +weighed against the truth. + +**Prefer evidence to assertion.** Record the command and its output — the +`ln ` that identifies a breakpoint, the counter that read exactly 2x, +the `x BlorgFS!...` that proved a function never ran. A claim someone can +re-verify in one command survives; a bare conclusion does not. + +## Conventions + +These are opinionated and consistently applied. Match them exactly, and +correct drift when you find it. + +### Naming and layout + +- **Parameters are `PascalCase`; locals are `camelCase`.** This is the one + people get wrong most often, and it is load-bearing for readability: at any + line you can tell what came from the caller and what is yours. + `static NTSTATUS HttpParseHeaders(HTTP_CONTEXT* Ctx)` with + `SIZE_T bodyOffset = Ctx->BodyOffset;` inside. +- **Functions are `PascalCase` with a module prefix, and the prefix says + whether the name leaves the file.** A file-static helper takes the bare + module name (`ReadClaimStream`, `HttpFail`, `TlsHandshakeFail`). Anything + with external linkage takes `Blorg`, with no exceptions for layer or + ancestry -- `BlorgSendWskAsync`, `BlorgTlsSha256`, `BlorgFspDispatch`, + `BlorgVolumeRead`. The fastfat-inherited names (`FsdPostRequest`, + `PrePostIrp`, `OplockComplete`) were renamed along with everything else; + they are this driver's functions now, not the reference implementation's. + + So `grep -n '^[A-Za-z].*Blorg'` is the export list, and a bare module + name at file scope is a promise that it is `static`. +- **File-scope statics are `PascalCase` with the module prefix** + (`SocketMaxPoolSize`, `HttpActiveRequests`). Driver-wide mutable state + lives in the `global` struct rather than as loose globals — prefer adding + a field there to introducing a new one. +- **Struct fields are `PascalCase`; macros and constants are + `SCREAMING_SNAKE`** (`READ_AHEAD_GRANULARITY`, `SOCKET_CONNECT_TIMEOUT_MS`). +- **Empty parameter lists are `(VOID)`, never `()`.** +- **Allman braces**, including on `switch` cases, which get their own braced + block. Every `if` gets braces, even single-statement ones. +- **Casts go through `C_CAST`,** not bare C-style casts. +- **Guard clauses and early returns** over nesting — the happy path stays at + the lowest indentation level. + +### Rules + +- **Comments only at top-of-file, above a function, or beside a struct field.** + Never inside a function body — naming carries the meaning there. +- **Padding:** never widen a field's type to satisfy `CHECK_PADDING_END`; add an + explicit named `Reserved[N]` instead. +- **`ProbeForRead` for all user-buffer validation,** including output buffers. + `ProbeForWrite` is legacy and writes every page. + + Two places deliberately do not probe, both following fastfat: the cached + read path in `Read.c` hands `Irp->UserBuffer` to `CcCopyReadEx` under SEH, + and `Security.c` lets `SeQuerySecurityDescriptorInfo` write into + `Irp->UserBuffer` directly. In both the fault is caught rather than + prevented. Do not "fix" either without reading why fastfat does the same. +- **Never `%wZ`/`%Z` in a `DbgPrint` that can run above `PASSIVE_LEVEL`** — + the formatting touches paged code and bugchecks. In practice that means + anywhere on a completion chain (`HttpFail`, `HttpComplete`, + `Blorg*Complete`) or in anything one can call. It is fine, and used + freely, on the PASSIVE-only dispatch paths. +- IRQL is load-bearing throughout the async paths; the file-header comments in + `Client.c` and `Socket.h` state each path's contract. Read + them before changing anything on a completion chain. +- **Keep the call tree flat, and be conservative with the kernel stack.** It + is only about three pages, shared with everything the thread calls, and + running out of it is a bugcheck, not a recoverable error. Avoid deeply + nested calls between internal routines that each pass data on the stack, + and bound the depth of any recursive routine explicitly rather than + trusting the caller. Prefer system-space (pool) allocation over a large + stack frame. `IoGetStackLimits`/`IoGetRemainingStackSize` can check + remaining headroom and `KeExpandKernelStackAndCallout` can grow it, but + neither is a substitute for keeping the tree shallow. The kernel stack can + also be paged out while the thread is in a user-mode wait, so never pass a + stack-based buffer (a local variable) to DMA or to a routine that can run + at `>= DISPATCH_LEVEL`. +- **Data-oriented design.** Lay data out for how it is actually traversed — + hot fields together, cold fields elsewhere, per-processor where + contended, arrays over pointer-chasing, struct-of-arrays over + array-of-structs when a hot path only touches a few fields of many. + Shape the structure around the access pattern rather than around a + conceptual object model, and default to measuring the access pattern + (a counter, a trace) before restructuring rather than guessing it. +- **Avoid false sharing.** Two independently-written fields on the same + cache line serialize their writers against each other even though the + code never intended them to share anything. Pad or align per-processor + or per-CPU state to a cache line boundary — `Statistics.h`'s + `BLORGFS_STATISTICS` entries are one per processor, each padded to a + multiple of 64 bytes, specifically so a CPU updates only its own entry + with a plain `+=` and never dirties a line another core is touching. + When fields are written by different threads at different rates, that's + a sign they belong on different cache lines, not packed together for + compactness. +- **Avoid atomics except where correctness genuinely requires them.** An + `Interlocked*` op is a full memory barrier and a cache line the whole + system can serialize on; on a hot or completion path it can cost more + than the work it's guarding. Reach for a non-atomic, thread- or + processor-confined design first — a per-processor counter summed by the + reader, or reformulating a shared mutable gauge into two independent + monotone counters whose difference the reader computes. `Statistics.h` + did exactly that: a shared interlocked "fetches in flight" gauge was + removed in favour of `FetchesIssued - (FetchesCompleted + + FetchesFailed)`, each term a per-processor counter, because the gauge + was redundant with counters that already existed and an interlocked op + on that path was paid on every fetch for nothing. Where atomicity is + genuinely required — a cross-thread reference count (`Fcb`/`Dcb`/ + `Vcb->RefCount`), a claim/idempotency flag two threads can race to set + (`OnReapList`, `FspQueue.ThreadsActive`) — use `Interlocked*`, not a + lock, and say in a comment why the operation has to be atomic so the + next person doesn't mistake it for a leftover habit. +- **Minimise syscalls and blocking calls.** Batch and amortise instead of + repeating a call per item; prefer an async completion over anything that + waits. A blocking call on a hot or completion path is a design bug, not a + detail — it occupies a worker for the whole duration. +- **Collapse branches with overlapping or shared predicates.** If two + conditions test the same thing, evaluate it once and branch once. Repeated + or nested tests of the same predicate should be merged rather than left + parallel — it is both faster and the only way the invariant stays obvious. + +## Build and test tiers + +**Run `tools\Invoke-BlorgChecks.ps1` — do not hand-roll a build command.** + +```bash +powershell -File tools/Invoke-BlorgChecks.ps1 -Tier Fast +``` + +Tiers, cheapest first: + +| Tier | What it does | Needs | +|---|---|---| +| `Build` | Compile + link everything with PREfast | nothing | +| `Fast` (default) | Build, plus RFC 8448 crypto vectors and the fuzz corpus | nothing | +| `Perf` | Fast, plus PerfHarness workloads compared to a stored baseline | driver loaded, backend reachable | +| `All` | every tier | as above | + +Exit code is 0 only if everything in the tier passed, so it gates cleanly. + +Run `-Tier Fast` before calling any change done — the crypto tests are what +catch a `Tls.c` regression, and cheaper tiers do not run them. + +(In Claude Code specifically, a `Stop` hook in a local `.claude/settings.json` +— a personal, gitignored setup, not something checked into this repo — may +run the `Build` tier automatically after each turn as a non-blocking +backstop. It is not a substitute for running `Fast` yourself.) + +### Where builds land, and why it matters + +**Build output lands in two different places, and it matters.** A *solution* +build writes to the repo root (`x64\\`), because `$(SolutionDir)` is +defined; building a project on its own -- which is what +`tools\Invoke-BlorgChecks.ps1` does -- writes to `tests\\x64\\` +instead. Neither is wrong, but both exist at once and can hold binaries of +different ages. + +Assume the one you want is the stale one until you have checked. This split +has caused three separate failures: a test running against a binary with no +ASan runtime beside it, `Invoke-TestExe` picking a stale exe by glob, and a +32-minute-old `PerfHarness` deployed to the VM measuring nothing. CI works +around it by searching both roots (`build.yml`). + +The sandbox projects put their own directory on the include path so +`src/Driver.h` can pull in `SandboxPrelude.h` without the driver naming a +test directory. + +### Build gotchas + +- **Use the 64-bit MSBuild.** The WDK NuGet picks its PREfast and ApiValidator + directory from the MSBuild process's own architecture; the 32-bit one selects + an x86 directory where those tools are missing, silently losing the static + analysis. `Invoke-BlorgChecks.ps1` already resolves the right one. +- **`inf2cat` "postdated DriverVer" -- fixed, but know the shape.** `BlorgFS.inf` + leaves `DriverVer` empty, so stampinf fills it from local time. inf2cat used + to validate against UTC, so between local midnight and the UTC offset the + two disagreed and catalog generation failed on a tree that compiled fine + (`inf2cat.exe exited with code -2`). `Inf2CatUseLocalTime` in + `BlorgFS.vcxproj` now points the validator at the same clock the stamp came + from. If it ever returns, that mismatch is where to look; the check script + reports it as `CLOCK`, not `FAIL`. + +## Continuous integration + +Three workflows, split by what a failure should cost you. + +| Workflow | Runs on | What it does | +|---|---|---| +| `build.yml` | push and PR to master | Both configurations, Fast tier. The merge gate. | +| `verify.yml` | 03:00 UTC daily, or on demand | CBMC proofs and extended fuzz/interleaving runs. | +| `codeql.yml` | Saturdays 23:41 UTC, on demand, and on any PR touching its own config | CodeQL with the pinned Microsoft driver query packs. | + +The daily and weekly ones are deliberately not gates: a CBMC regression or +a new CodeQL finding is worth waking up to, not worth blocking a merge that +PREfast and the Fast tier already cleared. + +`codeql.yml`'s third trigger is the one worth understanding. Scheduled +workflows only ever run on the default branch, so a change to what CodeQL +analyses -- above all a query-pack pin in `.github/codeql/codeql-config.yml` +-- could otherwise only be merged unrun, and would first show up as a +changed finding set the following Saturday. A PR touching that config or +the workflow runs the analysis it is changing. `workflow_dispatch` covers +the rest: re-scanning after a pack bump or a batch of fixes, without a +seven-day wait. + +Both packs are pinned on purpose. A pack release must not silently change +what a scheduled run reports -- but a pin that is never reviewed is lost +coverage, and the windows-drivers pack is the one carrying the +driver-specific IRQL and annotation queries. Bump it deliberately, let the +PR trigger run it, and re-triage: previous false-positive verdicts do not +carry across a pack version. + +## Sanitizers + +The usermode sandbox targets build with **ASan** (`EnableASAN`) — it owns +memory-safety there, and the gate runs under it. Three things every ASan +target needs, and each fails in its own unhelpful way if missed: + +| Requirement | Symptom when missing | +|---|---| +| `/Zi`, not `/ZI` | ASan and Edit-and-Continue are incompatible | +| `LinkIncremental=false` in a *configuration* PropertyGroup | `LNK4300: ignoring '/INCREMENTAL' because input module contains ASAN metadata` on every link. Setting it inside `ClCompile` is silently ignored. | +| A `CopyAsanRuntime` post-build target | The exe exits `0xC0000135` before `main`, so the suite reports a bare non-zero exit and no output | + +That last one is per project and easy to get wrong, because a *solution* +build puts every binary in one directory — so a target with no copy step +still runs, using the DLL some other project deposited next to it. +`tools\Invoke-BlorgChecks.ps1` builds project-by-project instead, where +output is project-local, and there the missing copy is fatal. A target can +therefore pass in CI and fail in the gate, or the reverse, purely on which +build shape ran last. + +The driver itself builds with **KASAN**: + +```bash +msbuild src\BlorgFS.vcxproj -p:Configuration=Debug -p:Platform=x64 -p:EnableKASAN=true +``` + +`kasan.lib` ships in the WDK and the instrumented `.sys` is roughly double +the size — that size jump is the quickest check that it actually applied. +It is opt-in, so the normal build and the gate are unaffected. Running it +needs `bcdedit /set kasan on` in the guest and a reboot. + +Note `-fsanitize=thread` is **unsupported** for `x86_64-pc-windows-msvc`; +there is no TSan on this platform. Interleaving coverage comes from the +systematic scheduler instead (`tests\sandbox\Scheduler.h`). + +The scheduler's lock contract is **claim-under-the-baton**: a primitive waits +via `KmSchedWaitUntilClaim`, and its claim callback runs while the caller still +holds the baton, immediately after the predicate that justified it. Claiming +anywhere else reopens a TOCTOU window between check and claim -- the spin-lock +double-grant and the ERESOURCE double-hold were both exactly that window. +A deadlocked schedule drains its parked threads serially through the baton +rather than releasing them all at once, so an abandoned run exits cleanly +instead of corrupting every replay after it. `SchedulerAudit` in +`NodeTableSchedTest.cpp` pins both properties. + +## Deploying to a VM + +BlorgFS is a kernel-mode filesystem driver, so it is developed against a +throwaway Windows VM rather than the build machine. This section is +tool-agnostic — nothing here assumes any particular editor, agent, or IDE. + +**If something looks broken while testing — a frozen VM, a mysterious +timeout, a bugcheck — read "Debugging the VM" below first.** Most +alarming-looking failures in this environment are known, diagnosable, and +not BlorgFS bugs; that section is a decision tree for telling the difference +before spending time chasing the wrong thing. + +**Do not deploy by copying `BlorgFS.sys` into `System32\drivers` by hand.** +It skips catalog validation entirely and can silently leave a stale binary in +place from a previous iteration, so the thing you are debugging is not the +thing you just built. Symptoms of that mistake look like driver bugs +(unexplained hangs, behaviour that does not match the source) and cost far +more time than the deploy script does. + +### The short version + +Copy `deploy/blorgfs.env.example` to `deploy/blorgfs.env`, fill it in once, +and then deploying takes no arguments at all: + +```powershell +.\deploy\Deploy-ToVM.ps1 -Configuration Release +``` + +`blorgfs.env` holds the VM path, the guest account, the `.vmx` +config-encryption password, the snapshot to revert to, the backend address, +and the KDNET key. It is **gitignored** -- every one of those is a +credential for the debug VM, and the KDNET key in particular lets anyone on +the network attach a kernel debugger to that guest. The committed +`blorgfs.env.example` is the template and carries no real values. + +The point of it is that a session starting from nothing can deploy and debug +without first being handed five settings by hand. Explicit arguments still +win over the file, so nothing is locked in: + +```powershell +.\deploy\Deploy-ToVM.ps1 ` + -VmxPath "C:\path\to\Win11.vmx" ` + -GuestUser -GuestPassword ` + -SnapshotName ` + -Configuration Debug +``` + +That builds, copies the driver package into the guest, and runs +`Install-BlorgFS.ps1` there. Reverting to a known-good snapshot first is +strongly recommended — a buggy driver load can bugcheck the guest, and a +half-installed package is worse to debug than a clean one. + +For benchmarking: + +```bash +powershell -File deploy/Deploy-ToVM.ps1 -ForBenchmark +``` + +`-ForBenchmark` deploys Release, clears Driver Verifier, reboots so the +change actually applies, and waits for the guest to go idle before +reporting success — see "Measuring performance" below for why each of those +matters. `-Configuration` still wins if given explicitly. + +### How the install actually works + +`BlorgFS.inf` is a Windows 10 1903+ **primitive driver** INF: the driver +creates its own device objects in `DriverEntry` rather than being enumerated +by a bus, so there is no hardware ID, no `[Manufacturer]`/`[Models]` section, +and it never appears in Device Manager. Installing it correctly is a two-step +sequence, and **neither step alone is sufficient**: + +```powershell +# 1. Stage the package into the Driver Store +pnputil.exe /add-driver /install + +# 2. Run the section that does CopyFiles + AddService +rundll32.exe setupapi.dll,InstallHinfSection DefaultInstall.NTamd64 132 +``` + +Why both: + +- `BlorgFS.inf` sets `DestinationDirs = 13`, meaning **run from Driver + Store** (driver package isolation). `InstallHinfSection`'s plain + `CopyFiles` engine cannot write into the protected Driver Store on its + own. Run without staging first, it fails the copy — interactively as a + *"Setup cannot copy the file BlorgFS.sys"* dialog that misleadingly blames + the **source** path (the source is fine; the destination model is the + problem), and non-interactively as a **silent no-op**: exit code 0, and + nothing written to `setupapi.dev.log`. +- `pnputil /install` alone stages the package but never runs + `[DefaultInstall.NTamd64.Services]`. Its `/install` flag installs against + *matching devices*, and a primitive driver has no hardware ID to match, so + it reports success, stages the files, and leaves the service unregistered. + +Other things that bite here: + +- **The section is `DefaultInstall.NTamd64`, not `DefaultInstall`.** The INF + only defines the architecture-decorated variant. Naming the undecorated one + matches nothing and no-ops silently. +- **Deploy the *staged* INF, not the repo-root one.** MSBuild's `stampinf` + step fills in `DriverVer` on the copy at + `x64\\BlorgFS\BlorgFS.inf` *before* `Inf2Cat` hashes it. The + repo-root `BlorgFS.inf` deliberately leaves `DriverVer` blank, so its bytes + — and therefore its hash — differ from what the catalog contains. Deploying + it fails with *"The hash for the file is not present in the specified + catalog file. The file is likely corrupt or the victim of tampering."* + That message sounds like binary corruption or a signing failure; it + actually just means the wrong copy of the INF was deployed. +- **The registered service name is `BlorgFS`.** Not `BLORG`, not anything + from an ad-hoc `sc create`. Verify with `sc query BlorgFS`; a `1060` from + `sc query` means the `AddService` step never ran. +- Test-signing must be on (`bcdedit /set testsigning on`) and the driver's + test certificate trusted, both of which `Install-BlorgFS.ps1` handles. + Enabling test-signing requires a **reboot** before any test-signed driver + will load; the script exits with status 2 to signal exactly that, and + `Deploy-ToVM.ps1` reboots and retries once automatically. + +Confirm a good install: + +```powershell +sc query BlorgFS # STATE : 4 RUNNING +Test-Path B:\ # True -- the driver self-mounts, there is no mount step +``` + +### VMware / `vmrun` quirks + +- The `.vmx` used for this project is **encrypted at the VM-config level**, + which is separate from any guest OS login. Every `vmrun` invocation against + it needs `-vp ` before the command verb — including read-only + commands like `list`. Without it: *"A password is required for this + operation."* +- `vmrun`'s argument order is rigid: + `vmrun [flags] COMMAND [command-args...]`. The vmx path goes + **immediately after the command name**, not at the end. Getting this wrong + produces confusing errors like *"Cannot open VM: C:\some\other\path, + unknown file suffix"* — it is trying to interpret your argument as the vmx. +- Guest credentials go in `-gu` / `-gp`, and VMware Tools must be running in + the guest for any guest command to work. `vmrun checkToolsState ` + is the quick health check; it can transiently report tools as down and + recover on its own. +- `runProgramInGuest` defaults to a **non-interactive, Session-0** context. + Anything that needs the interactive desktop (notably classic setup UI) will + silently do nothing there. Pass `-interactive` to run on the console + session. The two-step install above avoids needing this, but it is + essential for *seeing* a setup dialog when diagnosing why an install + failed. +- `runProgramInGuest` does not surface the guest process's exit code. + `Install-BlorgFS.ps1` works around this by writing its status to a file + that `Deploy-ToVM.ps1` copies back and reads — and deletes any stale status + file first, so a crashed run cannot be misread as the previous run's + success. + +### Avoiding manual guest login + +Windows leaves the guest at the lock/login screen after any reboot +(including the auto-reboot after a bugcheck), and pre-login the network +profile tends to sit as "Public" — which blocks ICMP by default, so `ping` +and some guest-automation calls look like the guest is unreachable when it is +actually just sitting at the lock screen waiting for a human. Set Windows +auto-logon once per VM image so this stops being a recurring interruption: + +```powershell +$k = "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" +Set-ItemProperty $k AutoAdminLogon "1" +Set-ItemProperty $k DefaultUserName "" +Set-ItemProperty $k DefaultPassword "" +Set-ItemProperty $k DefaultDomainName $env:COMPUTERNAME +Set-ItemProperty $k ForceAutoLogon "1" # re-applies after a manual lock, not just cold boot +``` + +Also worth disabling for any long-running test session, so an idle timeout +does not re-lock the session mid-run: + +```powershell +powercfg /change monitor-timeout-ac 0 +powercfg /change standby-timeout-ac 0 +powercfg /change hibernate-timeout-ac 0 +Set-ItemProperty "HKCU:\Control Panel\Desktop" ScreenSaveActive "0" +``` + +This needs to be set once per golden snapshot — a `revertToSnapshot` to an +older snapshot taken before these were set will need it redone. + +### Kernel debugging (KDNET) + +The guest boots with `debug Yes` and a KDNET transport. Check its settings +from inside the guest with `bcdedit /dbgsettings` (key, port, host IP). + +Connecting to an **already-running** target requires an explicit `target=` +with the guest's IP: + +``` +net:port=50000,key=,target= +``` + +`vmrun getGuestIPAddress ` gets the IP. Omitting `target=` only works +against a target actively announcing itself (e.g. right at boot); against a +running guest it simply times out. + +See "Debugging the VM" below for what a normal break-in looks like, the +`g` (resume) timeout, and the `KdBreakPoint()` gotcha in Debug builds — all +of it applies here too. + +### Testing inside the guest + +The `Perf` tier of `tools\Invoke-BlorgChecks.ps1` must run **inside the +guest**, where `B:` is mounted and the HTTP backend is reachable. See +"Measuring performance" below for what the counters mean and how baselines +are updated. + +## Debugging the VM: what's real and what's noise + +This VM setup is flaky in ways that look alarming but usually aren't. This +section is a decision tree for telling the two apart, plus the specific +footguns that have cost real time. Read it before treating anything here as +a BlorgFS bug — most "the VM is frozen/crashed" moments were not. + +**But read the next part first.** One very common "frozen VM" *is* BlorgFS, +and mistaking it for VIX flakiness costs the most time of anything here. + +### The guest freeze that IS BlorgFS: `KdBreakPoint()` in Debug builds + +**Largely fixed on 2026-08-22 — read this before blaming a Debug-build +freeze on the environment, and before re-adding a breakpoint to a rejection +path.** + +`KdBreakPoint()` is `#if DBG`-gated by the WDK headers: an `int 3` in Debug, +nothing in Release. On a debug-enabled guest that `int 3` traps, and **with +no debugger attached the guest simply stops dead** — indistinguishable from +the VIX/host-memory flakiness described below. `vmrun` calls hang, the +console is unresponsive, the VM looks wedged. The tell is that resuming from +a debugger un-freezes it, and it re-freezes the moment something touches +`B:` again. + +`Create.c` used to carry six unconditional `KdBreakPoint()` calls on its +access-mask and disposition rejection paths, so **any** process opening a +file on `B:` with an unsupported access mask froze the whole guest. That +fires constantly without you doing anything: Explorer, Defender (`WdFilter` +attaches to `\Device\BlorgDrive`) and `SearchHost` all probe files with +write masks, and anything registering a library or saving settings (Steam, +game launchers) opens for write as a matter of course. + +**Five of the six are now gone** — refusing a write on a read-only volume is +an expected outcome, not an anomaly worth trapping (see the comment above +`CheckFileAccess`). **Do not re-add them.** One deliberately remains, on the +terminal `STATUS_INVALID_DEVICE_REQUEST` fallthrough at the end of +`BlorgVolumeCreate`, which is a genuine "should not get here" -- the create +matched no case at all. Its own reason is recorded in that function's header +comment too, so grepping for `KdBreakPoint` and landing on the survivor +turns up the explanation without having to find this file first. + +**Confirm it in one command.** Attach KD, break in, and resolve the +instruction pointer: + +``` +ln +``` + +Landing anywhere inside the `BlorgFS` module — `BlorgFS!CheckFileAccess+0x46` +was the original signature — is a driver breakpoint, not a debugger +break-in artifact. Contrast with `nt!DbgBreakPointWithStatus`, which is the +normal forced break-in (see below). Check the module range with +`lm m Blorg*` if the symbol does not resolve. Since the fix this should only +resolve to the `BlorgCreate` fallthrough; anything else means a breakpoint +was reintroduced. + +**Use a Release build for any correctness or performance testing.** +`KdBreakPoint()` compiles to nothing there, and the statistics counters are +always-on in Release anyway (see "Measuring performance" below), so nothing +is lost. This is still the right default even with the rejection-path +breakpoints gone — a Debug build also skews every performance number. +Attach a debugger *before* loading a Debug build so any remaining trap is +diagnosable instead of fatal: + +```powershell +.\deploy\Deploy-ToVM.ps1 -VmxPath -GuestUser -GuestPassword

` + -VmPassword -Configuration Release -RemoteHost 10.0.50.17 +``` + +### First move: is the guest actually unwell, or is it just VIX? + +`vmrun`'s guest-automation channel (VIX) and the guest kernel are two +different things, and they fail independently. A `vmrun` call timing out or +returning *"The VMware Tools are not running in the virtual machine"* +tells you **nothing** about whether the guest is actually broken — check the +guest directly before believing it. + +**Fastest ground truth: attach a kernel debugger and look at two things.** + +``` +net:port=,key=,target= +``` + +(`vmrun getGuestIPAddress ` gets the IP; omitting `target=` only works +against a target actively announcing itself, e.g. right at boot — see +"Kernel debugging (KDNET)" above.) + +1. **Does the register/stack state show a real bugcheck**, or the debugger's + own break-in artifact? Landing in `nt!DbgBreakPointWithStatus` (or, on + some boots, a different but equally unnamed low address — see below) + with a short 2-3 frame garbage stack is the **normal, expected** result + of forcing a break-in on an otherwise-idle or -running system. It is not + evidence of anything in BlorgFS. A **real** bugcheck shows an actual call + chain: `nt!KeBugCheckEx` → `nt!HalpNMIHalt` → `nt!HalBugCheckSystem` (this + session's specific signature — see the NMI section below). +2. **Is `System Uptime` advancing** across repeated break-ins a few seconds + apart? If yes, the kernel is alive and running normally — whatever + `vmrun` is complaining about is a host-side VIX problem, not a guest + problem. + +If both check out, the guest is fine. Stop chasing a "frozen VM" and go +fix whatever is actually making VIX unreliable (almost always host memory — +see below). + +### Driving the guest: what works and what silently does not + +Hard-won specifics for this VM. Each of these looked like a guest or driver +problem and was not. + +- **`cmd.exe` via `runProgramInGuest` hangs indefinitely; `powershell.exe` + works.** Reproducible even for `cmd.exe /c exit` with no redirection, + while `notepad.exe` and `powershell.exe -Command exit` return instantly + on the same guest. Each attempt strands a `cmd.exe` in the guest, so a + pile of them is a symptom of this and not of anything else. **Drive the + guest with `powershell.exe` only.** + + It does not always hang: it also returns **exit 1** on a completely + healthy guest, for both `cmd.exe /c exit 0` and `cmd.exe /c ver`, while + `powershell.exe -Command 'exit 0'` returns 0 in the same second. That + makes `cmd.exe` uniquely bad as a *health probe*, which is the one job it + looks perfect for: the probe reports every guest as dead, and any + recovery escalation behind it then fires against a guest that was fine. + On 2026-08-29 a sweep did exactly that, hard-resetting a working VM and + producing a black console that was then read as evidence of a driver + hang. The rule above already said not to do this. +- **`shutdown /r /t 0` never returns through `runProgramInGuest`.** + `runProgramInGuest` blocks until the guest program exits, and with `/t 0` + the OS tears the process down before it can exit, so vmrun waits forever + on a status that will never arrive. The guest reboots normally and sits + there idle while the host-side script hangs -- twelve minutes of apparent + "slow measurement" that was one stuck call. Use **`shutdown /r /t 5`**: + `shutdown.exe` schedules and returns, PowerShell exits, vmrun returns, + and the reboot fires afterwards. +- **Pass scripts as files, not as `-Command` strings.** Bash mangles `$_` + and `$($...)` before PowerShell ever sees them, which silently corrupts + the script — the usual symptom is an empty output file rather than an + error. Write the script locally, `copyFileFromHostToGuest`, then run it + with `-ExecutionPolicy Bypass -NoProfile -File`. +- **`runProgramInGuest` does not return the guest's stdout.** Redirect + inside the guest (`*> C:\...\out.txt`) and copy the file back. Do the + copy-back even when the run reports a non-zero exit — that is usually + where the actual error message is. +- **MSBuild from Bash mangles `/p:` switches into paths.** Run it from + PowerShell with `-p:` form instead. +- **`C:\BlorgFS-Deploy` in the guest accumulates files across sessions.** + There are results in there from days ago. Always write to a fresh + filename or delete first, or you will read a stale file back and believe + it is this run's output. + +### Footgun: orphaned `kd.exe` processes break the build and freeze the guest + +A `kd` session whose resume timed out (see below) leaves the **process +alive** even though the tooling has dropped the session id. Two things +follow, neither of which points at its cause: + +- It holds `x64\Debug\BlorgFS.pdb` open, so the next driver build dies with + `LNK1201: error writing to program database`. Every other project in the + solution still builds, which makes it look like a driver-specific code + problem. +- It is still attached to the target, so it can hold the guest halted — + looking exactly like the VIX hang above. + +Check for strays and clear the ones from your own session: + +```powershell +Get-Process kd -ErrorAction SilentlyContinue | Select-Object Id,StartTime +``` + +Match `StartTime` against when you opened sessions before killing anything; +a `kd` predating your session may be someone else's live debugger. + +### Footgun: a kernel debugger attached during a bugcheck LOOKS like a frozen VM + +`AutoReboot` is configured on this guest, but a bugcheck that happens while +a KD session is connected **halts and waits for the debugger** instead of +auto-rebooting — completely normal WinDbg behavior, but from outside (or on +the VM's console) it is indistinguishable from a genuinely hung VM: the +screen just sits there, unresponsive, forever. + +**If a hang coincides with a debugger session being open, close that session +with `resume: true` (or send `g`) before concluding the VM itself is stuck.** +This alone explained more than one "it's frozen" moment. + +### Footgun: `g` (resume) reports "Request timed out" and kills the session + +This is expected, not a failure — the resume itself works, but the tool call +wrapping it doesn't get a response until the *next* break, so it times out +and the session id stops being valid. **Open a fresh `open_kd_session` to +check state after resuming** rather than reusing the old session id or +retrying `g` on it. + +### Footgun: an unresolved break address isn't automatically suspicious + +Not every break-in lands in `nt!DbgBreakPointWithStatus` by name — on one +boot, forced break-ins repeatedly landed at an address with no resolvable +symbol at all (`lm` showed nothing there), at both ~1 minute and ~31 minutes +of uptime, with an identical kernel base both times. That pattern (same +address, same kernel base, across what should be a fresh KASLR slide) points +at the debugger's own break-in landing in a stable idle-loop location for +that particular boot, not a driver problem — confirm via the same +uptime-advancing check above rather than assuming an unnamed address means +something is wrong. + +### Root-caused: service stop wedges in `STOP_PENDING` (no dismount handler) + +`sc stop BlorgFS` leaves the service in `STOP_PENDING` indefinitely, after +which `sc start` fails `1056 (already running)` and a reinstall fails +*"The specified service has been marked for deletion"*. Only a guest reboot +clears it. Reproduces against a healthy, reachable backend after normal +successful use, so it is not the dead-backend socket-timeout case below. + +**It is not a reference-count leak — every relevant count is already zero.** +Live KD state while wedged: + +``` +!drvobj \Driver\BlorgFS 2 -> DriverUnload: +!devobj -> RefCount 0 + ExtensionFlags (0x1) DOE_UNLOAD_PENDING + AttachedDevice (Upper) ... \FileSystem\FltMgr +!devobj -> RefCount 0, DOE_UNLOAD_PENDING +dt nt!_VPB -> Flags 1 (VPB_MOUNTED), ReferenceCount 0 +x BlorgFS!HttpActiveRequests -> 0n1 (standing reference, never released) +``` + +Both drain gates still reading their initial standing reference of 1 proves +`DriverUnload` **was never entered** — each drain releases that reference as +its first action. That rules out the two unbounded `KeWaitForSingleObject` +drains in `Client.c`, which is the intuitive suspect and the +wrong ones. + +The actual chain: + +1. `FsCtrlMountVolume` (`FsCtrl.c`) self-mounts: it creates the volume device + object and sets `VPB_MOUNTED`. +2. Filter Manager attaches minifilters to the now-mounted volume — `fltmc + instances` shows `WdFilter`, `UCPD`, `applockerfltr`, `bfs` and + `FileInfo` on `\Device\BlorgDrive`. +3. On stop, `IopUnloadDriver` walks the driver's device objects. The volume + device has `AttachedDevice != NULL` (FltMgr sitting above it), so it + cannot be deleted: the I/O manager sets `DOE_UNLOAD_PENDING` and + **defers `DriverUnload`**. +4. **`IRP_MN_DISMOUNT_VOLUME` has no handler.** `FsCtrl.c` cases + `IRP_MN_USER_FS_REQUEST` and `IRP_MN_MOUNT_VOLUME`; dismount falls into + `default:` and returns `STATUS_INVALID_DEVICE_REQUEST`. Nothing else will + initiate a dismount either, because the driver mounted itself rather than + being mounted by a storage stack. +5. The volume therefore never dismounts, FltMgr never detaches, the deferred + delete never completes, and `DriverUnload` never runs. + +Fix direction (not yet implemented): give the volume a real dismount path — +handle `IRP_MN_DISMOUNT_VOLUME`, tear down the FCB/DCB tree, clear +`VPB_MOUNTED`, and let FltMgr detach. Until then, **a guest +reboot between driver deploys is mandatory**, which `Deploy-ToVM.ps1` does +not currently do on its own. + +### Known environmental issue: NMI_HARDWARE_FAILURE (bugcheck 0x80) + +This VMware/AMD-virtualization setup occasionally bugchecks with +`0x00000080 NMI_HARDWARE_FAILURE`, confirmed unrelated to BlorgFS -- it +happens even with the driver not loaded -- and confirmed to correlate with +**host memory pressure** (see below) and heavy guest load. Two independent +occurrences triaged with `tools\Get-CrashVerdict.ps1 -SymbolServer` both +landed on the identical bucket: + +``` +0x80_4F4454_AuthenticAMD_NOERRREC_IMAGE_AuthenticAMD.sys +DriverOnStack: 0 +``` + +`Arg1` decodes to the ASCII tag `'TDO'` and the captured stack is just +`nt!KeBugCheckEx` with nothing underneath -- a minidump limitation, not a +real call chain. `AuthenticAMD.sys` is WinDbg's generic fallback name when it +has no real driver to blame, not an actual faulting module. This pattern +matches a hypervisor/watchdog-injected NMI forcing a crash dump on a guest +that looked unresponsive, not a genuine hardware or driver fault. + +**Before treating a `0x80` bugcheck as a BlorgFS finding**, triage it: + +``` +tools\Get-CrashVerdict.ps1 -DumpPath -SymbolServer +``` + +and check `DriverOnStack` — if it's `0` and the bucket matches the one +above, this is the known false alarm, not a new bug. + +Heavier allocation load (e.g. a broad recursive directory enumeration) +seems to correlate with triggering this — plausibly because **Driver +Verifier's special pool** turns every allocation into extra TLB-flush IPI +traffic, which is exactly the kind of load that would expose a marginal +hypervisor-level IPI-ack timing issue. Consider disabling Verifier +(`verifier /reset`, then reboot) for routine correctness/perf runs and +reserving it for targeted memory-safety sessions, if this keeps recurring +under load. + +### Host memory pressure breaks VIX reliability, not just VM performance + +**Below roughly 3-4GB of host free RAM, expect `vmrun` guest-automation +calls to fail or time out frequently — copies, `runProgramInGuest`, even +`checkToolsState` — while the guest kernel itself remains completely +healthy.** This was directly observed and measured: + +| Host free RAM | VIX behavior | +|---|---| +| 1.24 GB | Guest looked completely unresponsive; every call failed | +| 2.6 GB | Frequent failures, needed 2-3 retries per call | +| 3.2 GB | Frequent timeouts on trivial commands (`sc query`) | +| 8.1 GB | Fully reliable, no retries needed | + +Check host memory before assuming a guest problem: + +```powershell +Get-CimInstance Win32_OperatingSystem | + Select-Object @{n='FreeGB';e={[math]::Round($_.FreePhysicalMemory/1MB,2)}} +``` + +`vmware-vmx` itself is worth checking too — it has been observed using +noticeably more working set than the VM's own configured `memsize` (e.g. +6.13GB against a 4GB `memsize`), plausibly `mem.hotadd` overhead; disabling +that (requires a VM power-off to edit the `.vmx`) is an untried but +plausible lever if this keeps recurring. + +**Don't burn cycles on rapid VIX retries under memory pressure** — it just +adds more contention. Either wait for more host RAM to free up, or fall back +to the KD-based ground-truth check above, which doesn't depend on VIX at +all. + +### Footgun: a PowerShell-in-guest crash isn't necessarily your script's fault + +`runProgramInGuest` occasionally reports exit code `-196608` on completely +trivial scripts (a bare `sc.exe query` + `Test-Path`) under host memory +pressure. This looks like a script bug but reproduces on scripts with no +possible bug — treat it as another host-pressure symptom, not something to +debug in the script itself. + +### "The semaphore timeout period has expired" on `B:\` is the backend, not the FS + +Enumerating `B:\` failing with *"The semaphore timeout period has expired"* +(an `IOException` from `Get-ChildItem`) means the driver loaded and mounted +fine and is simply not reaching its HTTP backend. + +The INF-seeded default `RemoteHost` is `blorgfs.blorg.lan`, which **resolves +in this environment to `10.0.60.10` and is not reachable on port 8080**. The +working backend is **`10.0.50.17:8080`**. Deploy with `-RemoteHost +10.0.50.17` (or fix `HKLM:\SYSTEM\CurrentControlSet\Services\BlorgFS\Parameters\RemoteHost` +and restart the service). + +Confirm which side is at fault from inside the guest before touching driver +code: + +```powershell +Resolve-DnsName blorgfs.blorg.lan +Test-NetConnection -ComputerName 10.0.50.17 -Port 8080 +``` + +Note that a driver pointed at a dead backend also makes service **stop** +pathological: every in-flight request has to burn its full socket timeout +(connect 15s / send 15s / receive 30s, `Socket.c`) before unload can +proceed, so `STOP_PENDING` can persist for minutes and look like a hang. + +### Solved: the vendor IOCTLs returned `ERROR_INVALID_FUNCTION` (device type) + +`IOCTL_BLORGFS_QUERY_STATISTICS` / `RESET_STATISTICS` / `SET_TLS_PIN` all +failed with `ERROR_INVALID_FUNCTION` (error 1) from usermode — the +long-standing "statistics IOCTL broken" issue. **Root cause: they were +declared `CTL_CODE(FILE_DEVICE_FILE_SYSTEM, ...)`.** + +The I/O manager routes an IOCTL by the device type baked into its +`CTL_CODE`: `FILE_DEVICE_FILE_SYSTEM` becomes `IRP_MJ_FILE_SYSTEM_CONTROL` +(an FSCTL), everything else becomes `IRP_MJ_DEVICE_CONTROL`. All three are +implemented in `DevIoCtrl.c` under `IRP_MJ_DEVICE_CONTROL`, so they never +arrived; they fell into `FsCtrl.c`'s unhandled-FSCTL `default`, which +returns `STATUS_INVALID_DEVICE_REQUEST` → `ERROR_INVALID_FUNCTION`. Fixed +by declaring them `FILE_DEVICE_UNKNOWN`. + +Two traps worth knowing if this ever regresses: + +- **The sandbox tests cannot catch it.** They call `BlorgDeviceControl` + directly, so the routing decision never happens and every test passes + with the broken device type. `DevIoCtrlTest.VendorIoctlsAreNotRoutedAsFsctls` + asserts on the device type specifically for this reason. +- **`PerfHarness.exe` embeds the IOCTL codes at compile time.** After + changing them you must rebuild it, or a stale binary keeps sending the old + codes and still reports error 1 against a fixed driver. Note also that a + **Debug** `PerfHarness.exe` will not start in the guest (missing debug CRT + DLLs, exit code `-1073741515`) — deploy the Release one. + +### Technique: diagnosing "my IOCTL returns the wrong status" via live KD + +When a custom vendor IOCTL fails with a status that doesn't match anything +in the driver's own dispatch code, the fastest way to find out whether the +IRP is even reaching your dispatch routine is to inspect the live device +object stack, not just read source: + +``` +!devobj \BlorgFS +``` + +`AttachedDevice (Upper) ... \FileSystem\FltMgr` in the output means a +minifilter stack sits above your device and gets first look at every IRP — +relevant for any device created as `FILE_DEVICE_DISK_FILE_SYSTEM` and passed +to `IoRegisterFileSystem`, since Filter Manager treats it as a real +filesystem eligible for minifilter attachment regardless of whether the +driver intended it purely as a control device. Cross-reference with, from +inside the guest: + +```powershell +fltmc filters # which minifilters are loaded at all +fltmc instances # which volumes/devices each one is actually attached to +``` + +Note: `Set-MpPreference -DisableRealtimeMonitoring $true` can silently no-op +under Tamper Protection even from an admin token — don't trust it without +checking `Get-MpComputerStatus` afterward. + +## Measuring performance + +**Benchmark on an optimised Release build with Driver Verifier disabled.** +Anything else measures the instrumentation, not the driver, and the two +mistakes compound. + +`deploy\Deploy-ToVM.ps1` defaults to `-Configuration Debug`. A Debug driver +is unoptimised and compiles `BLORGFS_PRINT` in (it is a runtime check on +`global.LogLevel`, not a no-op), so it is the wrong binary to time: + +```bash +powershell -File deploy/Deploy-ToVM.ps1 -ForBenchmark +``` + +`-ForBenchmark` is the whole answer: it deploys Release, clears Driver +Verifier, reboots so the change actually applies, and then **waits for the +guest to go idle** before reporting success. Use it for every performance +run. `-Configuration` still wins if given explicitly. + +That last step is not politeness. A freshly booted Windows guest runs +Defender, SearchIndexer, Windows Update and TiWorker for minutes, which on a +2-vCPU guest is both cores saturated -- it steals the CPU the driver needs +*and* makes `vmrun` calls slow enough to look wedged. A run taken inside that +window measured its usermode control at 25.63 MB/s before the driver and +14.04 MB/s after: a 45% collapse in one cycle. Bracket every driver +measurement with the control run immediately before and after it, and throw +the point away when the two disagree. + +Driver Verifier is worse, because it is invisible in every output the +harness produces -- and because **it lives in the snapshot**. `Deploy-ToVM` +reverts before deploying, so verifier comes back on every deploy regardless +of what was done to the running guest. That is why clearing it is part of +the deploy rather than something to remember. + +The driver reports its own build flavour (`BLORGFS_STATS_FLAG_CHECKED_BUILD`), +so `PerfHarness` refuses to print workload results from a checked driver +without saying so first. Nothing reports the verifier state, so check it: + +```bash +verifier.exe /querysettings +``` + +If `BlorgFS.sys` is listed, the driver is running under whatever flags are +shown -- typically special pool, force IRQL checking, pool tracking, I/O +verification, deadlock detection, DMA checking, security checks and DDI +compliance checking. Every allocation is on its own guarded page and every +DDI call is wrapped. Disable it, reboot, measure, and put it back: + +```bash +verifier.exe /reset +verifier.exe /flags 0x001209bb /driver BlorgFS.sys +``` + +Record the flag word before resetting; the value above is this VM's, not a +universal one. `-ForBenchmark` saves it to `verifier-before.txt` in the guest +deploy directory before resetting. + +Faster still, set `BenchSnapshotName` in `blorgfs.env` to a snapshot whose +guest already has verifier cleared, and `-ForBenchmark` reverts to that +instead -- saving a reboot per run. Take it by hand: boot, `verifier.exe +/reset`, reboot, power off, snapshot. vmrun cannot snapshot an encrypted VM +while it is running. + +This is not hypothetical. An entire performance investigation was run +against a Debug driver under that full verifier flag set and compared +against an *unverified usermode HTTP client*. The comparison was +systematically biased against the driver by an unknown but certainly large +factor, which invalidated a "2.8x the CPU per MB/s" conclusion outright and +left every ratio in the study unsafe to quote. Verifier belongs on for +correctness work and off for measurement, and which one is in force must be +stated alongside the numbers. + +Counters are **always on**, including in release builds (`Statistics.h`/`.c`), +stored per-processor and read two ways: + +- `fsutil fsinfo statistics B:` — the standard `FILESYSTEM_STATISTICS` / + `FAT_STATISTICS` surface, via `FSCTL_FILESYSTEM_GET_STATISTICS(_EX)`. +- `PerfHarness.exe` — the driver-specific counters (read dispatch mix, + chunk-fetch latency histogram, connection pool, TLS) over `IOCTL_BLORGFS_QUERY_STATISTICS` on `\\.\BlorgFS`. + +Two things to know before reading read numbers: + +- **Buffered and unbuffered measure different systems.** Buffered goes + through Cc, which supplies all of this driver's read-ahead, and is what + playback looks like. `streams

unbuffered` bypasses Cc so + every read reaches the driver, which is what to use when comparing the + transport against a usermode HTTP client. +- **Warm runs are not measurements.** A second run against the same files is + served from the Windows cache at thousands of MB/s with zero paging reads. + Reboot the guest between points, and check that `paging reads` is non-zero + before believing a number. + +```bash +PerfHarness.exe seq B:\media\big.mkv --report run.txt +``` + +**One stream is not the workload.** Several readers at once is the normal +case -- a video and its subtitle track, a game streaming assets while its own +data file is open, a library browse overlapping playback -- and it behaves +differently enough that single-stream numbers can look healthy while the +system is starving streams outright. + +```bash +PerfHarness.exe streams B:\ 8 30 +powershell -File tools/Measure-BlorgScaling.ps1 -OutputPath baseline.txt +``` + +`streams` gives each reader its own file and thread and reports the **latency +tail** (p50/p95/p99/max) and a **fairness** ratio alongside aggregate +throughput. Aggregate alone is the wrong metric: a stream that stalls for a +second has failed even when the total looks fine, and equal throughput split +unequally is a different system from one that shares. + +`Measure-BlorgScaling.ps1` sweeps stream counts and adds scaling efficiency +against the single-stream result. **It reboots the guest between runs**, which +is not hygiene: the Windows cache holds the previous run's files, and a +counter reset does not touch it. A second run against the same files is +served from RAM at thousands of MB/s with no paging reads at all, so a sweep +without a real reset measures that from the second row onward. `sc stop` cannot provide that +reset -- it wedges in `STOP_PENDING` (see "Debugging the VM" above). + +Workload commands reset the counters first, so the numbers are attributable to +the workload. `--report` writes flat `key=value` metrics for +`tools\Compare-BlorgMetrics.ps1`, which checks both correctness invariants +(every inline paging read must have a fetch; fetch issues must balance +terminations) and perf deltas against a baseline in `tools\baselines\`. + +Accept new numbers deliberately, never silently: + +```bash +powershell -File tools/Invoke-BlorgChecks.ps1 -Tier Perf -PerfFile B:\media\big.mkv -UpdateBaseline +``` + +The driver runs in a VM for testing — `deploy\Deploy-ToVM.ps1` builds, copies, +and installs it via vmrun. Run the `Perf` tier inside the guest, where the +volume is mounted. + +## Read-ahead policy: current state + +This is the driver's read-ahead behaviour as it stands today, stated +without the history. The full derivation — every dead end, reversal and +measurement that produced these rules — is the evidence trail immediately +below; read it before changing any of this, since several of these rules +exist specifically because a simpler version of them measurably failed. + +**Lookahead is entirely Cc's built-in read-ahead**, sized per file object +via `CcSetReadAheadGranularity`. The driver's own prefetcher and chunk +budget are gone (git history only) and are not coming back as-is — see +"Future work: on-disk hot cache" for the replacement direction. + +| Rule | Detail | +|---|---| +| Starting granule | **128 KB**, every file. Registry: `ReadAheadGranularityKb` (0 = never call `CcSetReadAheadGranularity`, leaves Cc's own default of `PAGE_SIZE`). | +| Shrink | A window that fetched more than **2x** what it consumed (one granule of lead allowed). Applies to every pattern; does not consult the consumer. | +| Grow | Only when **all** of: the consumer never idles (`ReadIsGreedy` in `Read.c`, idle ticks under 25% of a window); the transport is quiet (fewer than `READ_AHEAD_ADAPT_QUIET_DEPTH` fetches in flight, fixed at **6**); 16 consecutive exactly-adjacent reads on the *current* stream (`ReadCurrentStreak`/`ReadLastStreamIndex` — not the longest streak across trackers, which oscillated); and Cc is still honouring the granule last given (largest paging read in the window against the granule asked for — not a rate or a clock). | +| Growth ceiling | **2 MB** (`ReadAheadMaxGranularityKb`), chosen because Cc itself caps around ~1.1 MB on this rig and going further bought nothing. | +| Feedback loop | On by default; `ReadAheadAdapt=0` pins the granule (useful for A/B measurement). | +| Slack-based growth | On by default; `ReadAheadSlackGrowth=0` disables it. | +| **Removed**: "loaded transport grows the granule" | Was in the tree, measured to be up to **12x worse** on paced/deadline workloads, deleted outright. Growth today is slack-only. | + +**What this gets right, measured:** a greedy sequential reader (file copy) +reaches ~1.0x a usermode HTTP client on the same link at the same moment; +amplification stays under 1.2x on every pattern including sparse/random +access; paced playback at realistic bitrates misses effectively no +deadlines. + +**What is not solved, and isn't a granularity problem:** eight paced +consumers demanding the *entire* link at once miss 0.5%-33% of deadlines in +every configuration tried, pinned or adaptive. That's bandwidth +starvation, not something a granule choice can fix. When those streams +starve at the ceiling they also stop idling and get misclassified as +"greedy" by the slack signal — recorded as a known, harmless (net) edge +case, not fixed. + +**The identified next lever, untried:** raise the number of fetches in +flight for a sequential reader without raising the granule. The +512 KB-vs-128 KB gap traces to too few fetches in flight to amortise +time-to-first-byte at the smaller size (depth ~2.7 at the driver's typical +operating point), not to the granule being inherently slower. + +## Evidence trail: the playback-stutter investigation + +The policy in "Read-ahead policy: current state" above didn't come from +first principles — it came from a specific investigation, several dead +ends, and two reverted implementation attempts. This section keeps the +conclusions and the reusable lessons, compressed; the full round-by-round +measurement tables and every superseded hypothesis along the way are not +reproduced here but are recoverable from this file's git history, per the +[documentation discipline](#documentation-discipline) above of writing +down what was *not* true, not just what was. + +### The investigation, compressed + +Reported symptom (2026-08-28, master, Release, Verifier off): video and +subtitle lag watching media off `B:`. The file that reproduces it does +0.33 MB/s sustained against a driver that does 17-28 MB/s, so throughput was +never the question — **the trigger is enabling subtitles**, and until they +are on, nothing about this driver is visible to the player at all. A local +control (the same file copied to the guest's own disk, same player, same +seeking) played smoothly with zero reads recorded by this driver's +counters, which exonerates decode and CPU capacity as independent +explanations. + +Four candidate causes were tested and ruled out — see "What it is not" +below. What was left: subtitles make the demuxer jump around the file +(sequential paging reads fall from 100% to ~70%), which defeats Cc's +read-ahead prediction for a third of reads and forces demand fetches the +player blocks on. A demand fetch is dominated by time-on-the-wire (~65% of +it is body bytes crossing the link), not by anything this driver does with +the CPU. + +Comparing against NTFS (which reads in ~98 KB clusters against this +driver's ~675 KB) pointed at `READ_AHEAD_GRANULARITY` as the lever. A sweep +across four fixed granules (4 KB to 512 KB) showed **no single constant +works**: a large granule wins on throughput and loses badly on tail latency +(a one-second stall on a filesystem whose purpose is playback), a small one +is the reverse, and which one wins depends on load — a lone reader wants a +small granule, a saturated transport wants a large one. That ruled out a +constant and pointed at an adaptive, per-file-object policy, made possible +because `CcSetReadAheadGranularity` turned out to be safely re-callable +mid-stream (undocumented, confirmed by measurement). + +The adaptive policy went through two real bugs before landing on today's +rules. First cut: growth was gated only by an amplification check, which +an operator-precedence mistake (`else if`) silently disabled — a bursty +read pattern inheriting a grown granule from a preceding sequential phase +amplified 2.82x before the waste test was moved back in front of the grow +test. Second: growth keyed off the *longest* streak seen across all stream +trackers on the FCB, so a stale streak from an earlier sequential phase +kept growth armed indefinitely and the granule oscillated; keying off the +*current* stream's own streak (`ReadCurrentStreak`) fixed it. + +The growth ceiling (2 MB) also took two failed attempts at self-tuning +before settling on a fixed, measured constant — see "Reverted experiments" +below. A "grow when the transport is loaded" rule was in the tree for a +while and measured to make paced/deadline workloads up to ~12x worse before +being removed outright; growth today is gated on consumer slack (idle vs. +busy) only, not transport load. + +One gap remains open, and is a limit rather than a defect: eight paced +consumers demanding the entire link at once miss 0.5%-33% of deadlines in +every configuration tried, pinned or adaptive. That's bandwidth starvation, +which no granule choice controls. The identified next lever — raising the +number of fetches in flight for a sequential reader rather than the granule +size, since the real constraint at the driver's typical depth (~2.7) is too +few fetches to amortise time-to-first-byte — has not been attempted. + +### What it is not + +Tested and dead, recorded so they are not re-proposed: + +- **Not the MKV Cues index.** The theory was that each seek drags a read to + the index at the end of the file. `end-of-file` reads: **0**. +- **Not metadata.** The theory was that the player re-opens or re-stats on + seek. `creates` and `file info`: **0 and 0**. +- **Not decode or CPU capacity.** The local-disk control above played + smoothly with zero reads recorded by this driver, on the same two vCPUs. +- **Not seeking.** Twenty-five scripted seeks at two-second intervals, the + rate the reporter used, produced zero reads over a frame interval. An + earlier version of this investigation wrongly concluded seeking was the + cause, from a synchronous-read simulation that exhibited the problem it + was built to look for rather than the real, asynchronous access pattern. + +### Reverted experiments + +Three things were built, measured, and pulled back out. Recorded so they +are not re-proposed without new evidence: + +- **A self-tuning growth ceiling** (two versions: keep-only-if-10%-better, + revert-only-if-10%-worse). Both regressed sequential throughput and + neither shipped. The reason generalizes: per-window throughput can't + resolve an ~11% marginal effect through a link whose own noise is ~30%, + and no threshold fixes that — loose enough to survive the noise is loose + enough to never fire. The ttfb/body ratio was the obvious alternative + signal and doesn't work either, since ttfb grows with the granule too. + What shipped instead: a fixed 2 MB constant, and the byte-count-based + "did Cc honour the granule" signal used elsewhere in the policy, which + doesn't depend on a stable link to measure. +- **Pipelining the receive** (posting the WSK receive at send-issue instead + of send-completion, to shrink time-to-first-byte). Measured consistently + *worse* across interleaved pairs — the send-completion DPC was never what + the request was waiting on. Also surfaced a real use-after-free in the + sandbox (a receive can complete synchronously and free the context before + the send is issued) with a clear fix if this is ever revisited: from the + moment the receive is posted it owns the context, and the send completion + may only stamp a timestamp and record status — never fail, retry, + complete or kick. +- **Splitting one fetch into concurrent range requests** over partial MDLs, + to get a short stall and a large granule at once. Measured worse at every + slice count tried — this guest has two vCPUs, so concurrent WSK receives + and HTTP parses serialize rather than overlap, and splitting just adds + per-slice overhead. `READ_AHEAD_PARAMETERS.PipelinedRequestSize` in + `ntifs.h` describes exactly this and would invite trying it again; the + answer may differ on a guest with more processors, but does not on this + one. + +### Lessons learned: measuring this driver + +Every one of these cost a wrong conclusion before it was understood, and +they apply to any future performance work here, not just read-ahead. + +**Driver-side latency is not what a user feels.** The cache manager sits in +front of all of it and exists to hide exactly those numbers -- driven at +playback rate the driver was issuing 67 ms fetches while the reader saw +0.11 ms. `UserReadLatency*` in `BLORGFS_STATISTICS` is the number to read; +the chunk-fetch block explains it but is not it. + +**Most application reads never become an IRP.** Buffered synchronous reads +of a cached file are served by fast I/O, so timing `IRP_MJ_READ` produced +one sample against a reader's 780. `FastIoRead` is wrapped +(`BlorgFastIoRead`) for this reason: `FsRtlCopyRead` is where a caller +blocks, because it calls `CcCopyRead` inline. + +**Validate instrumentation against something independent before trusting +it.** The user-read timer was checked against a usermode reader measuring +itself on the other side of the syscall boundary: 780 samples against 780, +max 31.174 ms against 31.4 ms. Without that step its first version -- which +saw one read in 780 -- would have looked like a finding. + +**A simulation of a player is not a player.** Read pace, read size, and +whether reads are synchronous all change the answer, and a reader written +to exhibit a hypothesis will exhibit it. Drive the real application: +`PerfHarness reset`, run it, `PerfHarness stats`. + +Two further traps already paid for: reads at full speed measure read-ahead +working rather than playback feeling slow, so pace a simulated reader at +the real bitrate; and every `*MaxUs` field was summed across processors +rather than max-reduced until 2026-08-28, so any maximum taken from this +driver before then is inflated by roughly the processor count. + +**A workload that cannot exhibit the effect will report that there is no +effect.** Four candidate workloads were tried against read-ahead +granularity before one could see it, and each was rejected on its own +measurement rather than on argument: unbuffered random (amplification +1.000, Cc is not in the path), buffered random (1.000, read-ahead is +pattern-triggered and never arms), a demuxer model whose cursors advanced +by their own block size (1.002 at 99.8% sequential -- `seq` in disguise, +read-ahead working perfectly), and the same model with a large stride but +no burst (2.07x at 1% sequential, too scattered to arm anything). What +reproduces the captured trace is `demux`: several cursors, each reading a +short adjacent burst then skipping a stride, which is what a container's +interleaved tracks look like from one file object. + +**Calibrating on one statistic is not validation.** The `demux` model above +was tuned until its amplification matched the trace and its sequential +share was close, and the match was treated as licence to sweep. It reads in +4 KB blocks; the player issues ~262 KB. Matching one number while missing +another by two orders of magnitude is the same class of error as matching +fetch latency and calling it what the user feels. + +**Measure interleaved or not at all.** An ordered A-then-B comparison hands +any host/link drift to whichever setting ran second — the same 128 KB +setting measured 20.2 MB/s at 0.16% in one session and 12.6 MB/s at 1.8% an +hour later. Alternate configurations with a reboot between runs, and rotate +which arm goes first between rounds, or a warm-up bias reads as a real +effect. + +**Compare shares, not counts, across runs of different length.** Runs with +different file sizes or durations produce different read totals, so an +absolute count of reads over a frame ranks the longest run worst. +`UserReadsOverFrameShare` exists for this — it is already a percentage, so +multiplying it again yields impossible values above 100% and, worse, +preserves the ordering while destroying the magnitude. + +## Future work: on-disk hot cache (not implemented) + +> **Not implemented. This is a future project, not current driver +> behaviour.** There is no `DiskCache.c`/`DiskCache.h` in the tree, no +> registry keys for it, and nothing described below exists yet. Don't +> reference this section as if the code is present, and don't be surprised +> the symbols below don't grep-hit anywhere else in the repo — that's +> expected until the project starts. + +The prefetch ring was removed rather than replaced. What follows is the +design for its replacement, which is **not an in-memory prefetcher** -- that +experiment is finished and its evidence is in git history. + +**It lives in the driver, as a module.** A usermode helper owning the store +was the other candidate, on the argument that it removed a re-entrancy +problem by construction. That argument was weaker than it looked: keeping +the store off this volume removes the recursion just as completely, since +nothing in NTFS's completion path calls back into this driver. What is left +is deadlock through the memory manager, which is a discipline problem the +helper would not have solved either -- it is the same problem every +filesystem has when it touches another one. + +### Why this, and not more lookahead + +Measured on the reference rig, 16 concurrent streams, 512 KB range GETs, +interleaved against a usermode HTTP client: + +| | throughput | +| --- | --- | +| network path ceiling, cold or warm | ~30 MB/s | +| driver, cold | 27-30 MB/s (0.93-1.01x the usermode client) | +| driver, warm in the Windows cache | **4800-6800 MB/s** | +| local guest disk, write / read | **601 MB/s / 4.3 GB/s** | + +Cold reads already sit at the link ceiling, so nothing on the fetch path can +add throughput: every delivered byte has to cross a ~30 MB/s wire. The only +way past it is to **not cross the wire**. RAM caching already demonstrates +the payoff and is bounded by RAM -- at four concurrent streams the working +set stopped fitting and a re-read fell from 6677 MB/s to 131 MB/s. Local +disk is ~20x the network on write, ~140x on read, with ~100x the capacity of +RAM. + +### Architecture: a driver module + +`src/DiskCache.c` / `DiskCache.h`, owning a block store in **one ordinary +file on an ordinary live volume**. Location and maximum size configurable +through the registry alongside `RemoteHost`/`RemotePort` in `Parameters`. + +The module boundary is deliberately narrow, and every entry point is either +pure memory or explicitly asynchronous: + +``` +BlorgDiskCacheInitialize / BlorgDiskCacheDrain startup, unload +BlorgDiskCacheLookup(FileId, BlockIndex) IN MEMORY ONLY, no I/O +BlorgDiskCacheReadAsync(Slot, Mdl, Completion) serves a hit +BlorgDiskCacheAdmitAsync(FileId, Block, Mdl) fire-and-forget write-behind +BlorgDiskCacheInvalidate(FileId) validator changed +``` + +`BlorgDiskCacheLookup` touching no I/O is what makes the rest safe: the read +dispatch path can ask "is this cached?" while holding whatever it holds, and +only then decide which asynchronous path to take. + +**The read path keeps the shape it already has.** A paging read today +returns `STATUS_PENDING` and is completed later from a network completion +(`Read.c`). A cache hit is the same shape with a different source: + +1. `BlorgDiskCacheLookup` — in-memory index, no I/O, no blocking. +2. **Hit**: queue a cache read; a worker fills `Irp->MdlAddress` and + completes the IRP. +3. **Miss**: issue the HTTP fetch exactly as now. On completion, complete the + IRP *first*, then queue the write-behind from the buffer already in hand. + +The reader never waits on the cache in either direction. A miss costs +nothing it did not already cost, and a write-behind failure is invisible. + +### What the store being off-volume does, and does not, buy + +**The store must not live on this volume**, checked at open by comparing the +target's volume device object with ours. That single rule is what removes +*recursion*: with the store on NTFS, nothing in the completion path of a +`ZwReadFile` calls back into BlorgFS. There is no cycle in the call graph, +and the usermode-helper alternative bought nothing here that this check does +not. + +What is left is not recursion, and calling it that obscures the actual +risks. Two remain, both mediated by memory manager: + +- **Deadlock through MM, not a nested call.** If a thread holds an FCB + resource and, inside a cache read, memory pressure makes MM trim that + file's pages, MM calls this driver's `AcquireForLazyWrite` and blocks on + the resource the thread is still holding. So: **never hold an FCB resource + across a cache call.** Enqueue, release, return pending. +- **The paging path is the dangerous one.** A paging read can originate from + MM while it is already short of memory. Dependent I/O issued from that + thread can wait on the reclaim that is waiting on us. So: **no cache I/O + on the calling thread** -- all `ZwReadFile`/`ZwWriteFile` happen on the + module's own PASSIVE workers, and the dispatch path only ever enqueues. + +That second rule costs nothing the design was not already paying. The IRP is +completed asynchronously either way, so moving the I/O to a worker changes +which thread finishes it and nothing else. + +**IRQL** is a hard constraint rather than a judgement call: `ZwReadFile` and +`ZwWriteFile` are PASSIVE-only, while network completions run at +`<= DISPATCH`, so an admit queued from a completion reaches PASSIVE through +a work item. That is the same rule the removed prefetcher lived by, and the +one thing from it worth keeping. + +**Open the store `FILE_NO_INTERMEDIATE_BUFFERING`**, for two reasons that +are worth stating accurately. It avoids double-caching bytes Cc already +holds for this volume, and it keeps the store from adding cache-manager +memory pressure at exactly the moment the driver is serving a paging read. +The cost is sector alignment, which a fixed-block store gives for free. + +An earlier draft of this section justified the flag by claiming that +cache-manager pressure from our own store could re-enter this driver through +its own cache callbacks. That is not true for a store on another volume, and +the flag is worth setting anyway for the two reasons above. + +### Store layout, index and recovery + +Fixed-size blocks, each preceded by its own header: magic, file-identity +hash, block index, backend validator, byte length, and the MAC below. Slots +are addressed by index, never by cluster or LCN, so fragmentation, extension +and defragmentation are all transparent. + +**The index is rebuilt from the block headers at startup, not persisted.** +A separate index file is faster to load and introduces a whole failure class +this does not need — an index that disagrees with the store, torn across a +crash, and confidently wrong. Header scan cannot desync because the headers +*are* the store. At 512 KB blocks a 30 GB store is ~61k headers; reading +only the header of each is a few hundred MB against a local disk measured at +4.3 GB/s, so a second or so of startup, off the mount path. + +Persisting an index is a later optimisation, and only worth it if that +startup cost ever shows up as a complaint. + +### Concurrency + +- Slot allocation from a free list under a leaf lock; nothing else is + acquired under it. +- Per-slot reference count so eviction cannot reclaim a slot with a read in + flight -- the same protocol the node table already uses, and the one the + systematic scheduler is set up to explore. +- A per-block "fetch in flight" marker so two readers missing the same block + do not both fetch it and both write it. + +### Cluster pinning: not worth it + +`FSCTL_MARK_HANDLE` with `MARK_HANDLE_PROTECT_CLUSTERS` marks a file so the +defragmenter will not move it. **The conclusion is not to use it.** The +intuition that this is over-engineering is right, and for reasons stronger +than "SSDs do not care about seeks": + +- It solves a problem this design does not have. Protection matters when + something maps a file by LCN and needs that mapping to stay valid -- + hibernation files, page files, block-level VM disks. This store is + addressed through the filesystem by offset, so a moved extent is + transparent. +- It is NTFS-only and volume-specific, and the cache is meant to live + wherever the user points it. +- Marking a large file unmovable is antisocial: it permanently constrains the + volume's own defragmenter on behalf of a cache that is by definition + disposable. +- The cost it avoids is seek cost, which on SSD is near zero and on spinning + media is still small against the ~30 ms network fetch it is competing with. + +Revisit only if profiling ever shows extent-map lookup -- not seek time -- +dominating cache reads, which would be a surprise. + +### Security model + +**Cache integrity is a security boundary.** A process that can write the +store can inject bytes this filesystem then serves as authoritative file +content: a straightforward data-poisoning primitive against every reader of +the share. + +- **ACL the store to SYSTEM and Administrators only**, deny everyone else, + and create it with an explicit security descriptor rather than inheriting + the parent directory's. A cache under a user-writable path with inherited + ACLs is the default-insecure outcome to avoid. +- **Refuse a store whose ownership or ACL is not what was expected**, at + open, rather than repairing it -- repairing races the attacker. +- **ACLs alone are not sufficient.** They do not cover an offline attack + (booting another OS, mounting the volume elsewhere), an administrator-level + compromise, or ordinary corruption. Blocks must carry their own integrity + check. +- **Per-block keyed integrity, verified before use.** Each block records a + MAC over (file identity, block index, backend validator, contents). A + mismatch discards the block and falls back to the network. A plain + checksum detects corruption but not tampering, and the threat here is + tampering. + + The key has to persist for the cache to survive a reboot, and it has to + live somewhere the store does not -- otherwise an attacker who can rewrite + blocks can recompute the tags and the MAC proves nothing. The service's + own registry key is the natural home: same trust boundary as the driver's + configuration, already SYSTEM-only, and already what `TlsPin` uses + (`Driver.c`). Generate it on first use, never log it, and treat a missing + key as an empty cache rather than an error. + + Running in kernel does not change the threat model here. The attacker of + interest is a process that can write the file, not one that can call the + driver -- and MAC verification happens on the module's worker before any + cached byte reaches an IRP, so a forged block is discarded on the same + path that would have discarded a corrupt one. +- **Bind blocks to a backend validator.** `server-rs` returns `etag` and + `last_modified`; a block whose validator does not match the current + response is stale and must not be served. Without this the cache serves + yesterday's bytes for a file that changed. +- **Tampering, truncation or wholesale replacement** must be + indistinguishable in effect from a cold cache: verification fails, blocks + are discarded, reads go to the network. + +The bar is explicit: **the cached path must not be weaker than the uncached +path.** A design step that cannot meet that does not ship. + +### Admission and eviction: start small + +The literature is consistent that the largest wins come from **admission** +control rather than clever eviction, and that the specific thing to avoid is +caching one-hit-wonders -- for a disk-backed store that is wasted write +bandwidth and, on SSD, wasted endurance. + +- **Admit on second miss, not first.** This is the CDN answer to one-hit + wonders, costs a few bytes of state per candidate, and is the single + highest-value policy decision available. +- **Prefer sequential streams.** The `READ_STREAM_TRACKER` array on the FCB + (`Structs.h`) already carries the streak -- it survived the prefetch + removal partly for this. A streaked reader is exactly the case where the + following blocks are worth having. +- **Evict with segmented LRU.** Cheap, well understood, and resistant to one + large scan flushing the whole store. +- **Do not start with TinyLFU/W-TinyLFU or ARC.** W-TinyLFU is the strongest + general result in the literature and is the right thing to *grow into* if + measurement justifies it; its frequency sketch costs about a page. But it + earns its keep on skewed, high-cardinality, small-object workloads -- CDN + edges, key-value caches -- and this workload is a handful of very large, + sequentially-read files. Second-hit admission plus SLRU captures most of + the benefit at a fraction of the complexity, and the counters will show + whether anything more is warranted. + +Measure hit rate, bytes served from cache, and write amplification before +tuning any of it. + +Sources: [TinyLFU (ACM ToS)](https://dl.acm.org/doi/10.1145/3149371), +[size-aware admission for CDN memory caches (CMU)](http://reports-archive.adm.cs.cmu.edu/anon/2016/CMU-CS-16-120.pdf), +[CacheSack: admission optimization for Google datacenter caches (USENIX ATC 22)](https://www.usenix.org/system/files/atc22-yang-tzu-wei.pdf), +[FSCTL_MARK_HANDLE](https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ni-ntifs-fsctl_mark_handle), +[Defragmenting Files](https://learn.microsoft.com/en-us/windows/win32/fileio/defragmenting-files). + diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 27aa5dc..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1,9 +0,0 @@ -# BlorgFS - -Project documentation lives in **[README.md](README.md)** — build and -regression tiers, sanitizers, performance measurement, and the coding -conventions. Deployment and VM/debugger specifics are in -**[deploy/README.md](deploy/README.md)**. - -Read both before making changes. Keep new findings in those documents rather -than here, so they stay useful independently of any particular tool. diff --git a/README.md b/README.md index 8843ac3..2911c64 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ -# BlorgFS +# BlorgFS -Kernel-mode Windows filesystem driver presenting an HTTP backend as a mounted -volume (B:). Read-only. Async WSK networking, an optional hand-rolled TLS 1.3 -client and a keep-alive connection pool. +Kernel-mode Windows filesystem driver that presents an HTTP backend as a +mounted, read-only volume (`B:`). Built on async WSK networking, an optional +hand-rolled TLS 1.3 client, and a keep-alive connection pool. -## Layout +## Repository layout ``` src/ the driver, and only the driver -- one .vcxproj, its INF, @@ -19,2129 +19,56 @@ tests/ everything that verifies it VolumeTester/ volume-level behaviour against a mounted drive tools/ tiered check runner, metric comparison, crash triage, differential correctness harness -deploy/ VM deploy pipeline and the debugging notes +deploy/ VM deploy pipeline (see AGENTS.md) third_party/ submodules: flatcc, picohttpparser, schemas, googletest ``` -**Build output lands in two different places, and it matters.** A *solution* -build writes to the repo root (`x64\\`), because `$(SolutionDir)` is -defined; building a project on its own -- which is what -`tools\Invoke-BlorgChecks.ps1` does -- writes to `tests\\x64\\` -instead. Neither is wrong, but both exist at once and can hold binaries of -different ages. +## Building and testing -Assume the one you want is the stale one until you have checked. This split -has caused three separate failures: a test running against a binary with no -ASan runtime beside it, `Invoke-TestExe` picking a stale exe by glob, and a -32-minute-old `PerfHarness` deployed to the VM measuring nothing. CI works -around it by searching both roots (`build.yml`). - -The sandbox projects put their own directory on the include path so -`src/Driver.h` can pull in `SandboxPrelude.h` without the driver naming a -test directory. - -## Checking for regressions - -**Run `tools\Invoke-BlorgChecks.ps1` — do not hand-roll a build command.** +Run the tiered check script rather than a hand-rolled build: ```bash powershell -File tools/Invoke-BlorgChecks.ps1 -Tier Fast ``` -Tiers, cheapest first: - | Tier | What it does | Needs | |---|---|---| | `Build` | Compile + link everything with PREfast | nothing | | `Fast` (default) | Build, plus RFC 8448 crypto vectors and the fuzz corpus | nothing | -| `Perf` | Fast, plus PerfHarness workloads compared to a stored baseline | driver loaded, backend reachable | +| `Perf` | Fast, plus PerfHarness workloads compared against a stored baseline | driver loaded, backend reachable | | `All` | every tier | as above | -Exit code is 0 only if everything in the tier passed, so it gates cleanly. - -Run `-Tier Fast` before calling any change done — the crypto tests are what -catch a `Tls.c` regression, and cheaper tiers do not run them. - -(For agent-driven sessions, a `Stop` hook in `.claude/settings.json` runs the -`Build` tier automatically after each turn as a non-blocking backstop. It is -not a substitute for running `Fast` yourself.) - -### Build gotchas - -- **Use the 64-bit MSBuild.** The WDK NuGet picks its PREfast and ApiValidator - directory from the MSBuild process's own architecture; the 32-bit one selects - an x86 directory where those tools are missing, silently losing the static - analysis. `Invoke-BlorgChecks.ps1` already resolves the right one. -- **`inf2cat` "postdated DriverVer" -- fixed, but know the shape.** `BlorgFS.inf` - leaves `DriverVer` empty, so stampinf fills it from local time. inf2cat used - to validate against UTC, so between local midnight and the UTC offset the - two disagreed and catalog generation failed on a tree that compiled fine - (`inf2cat.exe exited with code -2`). `Inf2CatUseLocalTime` in - `BlorgFS.vcxproj` now points the validator at the same clock the stamp came - from. If it ever returns, that mismatch is where to look; the check script - reports it as `CLOCK`, not `FAIL`. - -## Continuous integration - -Three workflows, split by what a failure should cost you. - -| Workflow | Runs on | What it does | -|---|---|---| -| `build.yml` | push and PR to master | Both configurations, Fast tier. The merge gate. | -| `verify.yml` | 03:00 UTC daily, or on demand | CBMC proofs and extended fuzz/interleaving runs. | -| `codeql.yml` | Saturdays 23:41 UTC, on demand, and on any PR touching its own config | CodeQL with the pinned Microsoft driver query packs. | - -The daily and weekly ones are deliberately not gates: a CBMC regression or -a new CodeQL finding is worth waking up to, not worth blocking a merge that -PREfast and the Fast tier already cleared. - -`codeql.yml`'s third trigger is the one worth understanding. Scheduled -workflows only ever run on the default branch, so a change to what CodeQL -analyses -- above all a query-pack pin in `.github/codeql/codeql-config.yml` --- could otherwise only be merged unrun, and would first show up as a -changed finding set the following Saturday. A PR touching that config or -the workflow runs the analysis it is changing. `workflow_dispatch` covers -the rest: re-scanning after a pack bump or a batch of fixes, without a -seven-day wait. - -Both packs are pinned on purpose. A pack release must not silently change -what a scheduled run reports -- but a pin that is never reviewed is lost -coverage, and the windows-drivers pack is the one carrying the -driver-specific IRQL and annotation queries. Bump it deliberately, let the -PR trigger run it, and re-triage: previous false-positive verdicts do not -carry across a pack version. - -## Sanitizers - -The usermode sandbox targets build with **ASan** (`EnableASAN`) — it owns -memory-safety there, and the gate runs under it. Three things every ASan -target needs, and each fails in its own unhelpful way if missed: - -| Requirement | Symptom when missing | -|---|---| -| `/Zi`, not `/ZI` | ASan and Edit-and-Continue are incompatible | -| `LinkIncremental=false` in a *configuration* PropertyGroup | `LNK4300: ignoring '/INCREMENTAL' because input module contains ASAN metadata` on every link. Setting it inside `ClCompile` is silently ignored. | -| A `CopyAsanRuntime` post-build target | The exe exits `0xC0000135` before `main`, so the suite reports a bare non-zero exit and no output | - -That last one is per project and easy to get wrong, because a *solution* -build puts every binary in one directory — so a target with no copy step -still runs, using the DLL some other project deposited next to it. -`tools\Invoke-BlorgChecks.ps1` builds project-by-project instead, where -output is project-local, and there the missing copy is fatal. A target can -therefore pass in CI and fail in the gate, or the reverse, purely on which -build shape ran last. - -The driver itself builds with **KASAN**: - -```bash -msbuild src\BlorgFS.vcxproj -p:Configuration=Debug -p:Platform=x64 -p:EnableKASAN=true -``` - -`kasan.lib` ships in the WDK and the instrumented `.sys` is roughly double -the size — that size jump is the quickest check that it actually applied. -It is opt-in, so the normal build and the gate are unaffected. Running it -needs `bcdedit /set kasan on` in the guest and a reboot. - -Note `-fsanitize=thread` is **unsupported** for `x86_64-pc-windows-msvc`; -there is no TSan on this platform. Interleaving coverage comes from the -systematic scheduler instead (`tests\sandbox\Scheduler.h`). - -The scheduler's lock contract is **claim-under-the-baton**: a primitive waits -via `KmSchedWaitUntilClaim`, and its claim callback runs while the caller still -holds the baton, immediately after the predicate that justified it. Claiming -anywhere else reopens a TOCTOU window between check and claim -- the spin-lock -double-grant and the ERESOURCE double-hold were both exactly that window. -A deadlocked schedule drains its parked threads serially through the baton -rather than releasing them all at once, so an abandoned run exits cleanly -instead of corrupting every replay after it. `SchedulerAudit` in -`NodeTableSchedTest.cpp` pins both properties. - -## Measuring performance - -**Benchmark on an optimised Release build with Driver Verifier disabled.** -Anything else measures the instrumentation, not the driver, and the two -mistakes compound. - -`deploy\Deploy-ToVM.ps1` defaults to `-Configuration Debug`. A Debug driver -is unoptimised and compiles `BLORGFS_PRINT` in (it is a runtime check on -`global.LogLevel`, not a no-op), so it is the wrong binary to time: - -```bash -powershell -File deploy/Deploy-ToVM.ps1 -ForBenchmark -``` - -`-ForBenchmark` is the whole answer: it deploys Release, clears Driver -Verifier, reboots so the change actually applies, and then **waits for the -guest to go idle** before reporting success. Use it for every performance -run. `-Configuration` still wins if given explicitly. - -That last step is not politeness. A freshly booted Windows guest runs -Defender, SearchIndexer, Windows Update and TiWorker for minutes, which on a -2-vCPU guest is both cores saturated -- it steals the CPU the driver needs -*and* makes `vmrun` calls slow enough to look wedged. A run taken inside that -window measured its usermode control at 25.63 MB/s before the driver and -14.04 MB/s after: a 45% collapse in one cycle. Bracket every driver -measurement with the control run immediately before and after it, and throw -the point away when the two disagree. - -Driver Verifier is worse, because it is invisible in every output the -harness produces -- and because **it lives in the snapshot**. `Deploy-ToVM` -reverts before deploying, so verifier comes back on every deploy regardless -of what was done to the running guest. That is why clearing it is part of -the deploy rather than something to remember. - -The driver reports its own build flavour (`BLORGFS_STATS_FLAG_CHECKED_BUILD`), -so `PerfHarness` refuses to print workload results from a checked driver -without saying so first. Nothing reports the verifier state, so check it: - -```bash -verifier.exe /querysettings -``` - -If `BlorgFS.sys` is listed, the driver is running under whatever flags are -shown -- typically special pool, force IRQL checking, pool tracking, I/O -verification, deadlock detection, DMA checking, security checks and DDI -compliance checking. Every allocation is on its own guarded page and every -DDI call is wrapped. Disable it, reboot, measure, and put it back: - -```bash -verifier.exe /reset -verifier.exe /flags 0x001209bb /driver BlorgFS.sys -``` - -Record the flag word before resetting; the value above is this VM's, not a -universal one. `-ForBenchmark` saves it to `verifier-before.txt` in the guest -deploy directory before resetting. - -Faster still, set `BenchSnapshotName` in `blorgfs.env` to a snapshot whose -guest already has verifier cleared, and `-ForBenchmark` reverts to that -instead -- saving a reboot per run. Take it by hand: boot, `verifier.exe -/reset`, reboot, power off, snapshot. vmrun cannot snapshot an encrypted VM -while it is running. - -This is not hypothetical. An entire performance investigation was run -against a Debug driver under that full verifier flag set and compared -against an *unverified usermode HTTP client*. The comparison was -systematically biased against the driver by an unknown but certainly large -factor, which invalidated a "2.8x the CPU per MB/s" conclusion outright and -left every ratio in the study unsafe to quote. Verifier belongs on for -correctness work and off for measurement, and which one is in force must be -stated alongside the numbers. - -Counters are **always on**, including in release builds (`Statistics.h`/`.c`), -stored per-processor and read two ways: - -- `fsutil fsinfo statistics B:` — the standard `FILESYSTEM_STATISTICS` / - `FAT_STATISTICS` surface, via `FSCTL_FILESYSTEM_GET_STATISTICS(_EX)`. -- `PerfHarness.exe` — the driver-specific counters (read dispatch mix, - chunk-fetch latency histogram, connection pool, TLS) over `IOCTL_BLORGFS_QUERY_STATISTICS` on `\\.\BlorgFS`. - -Two things to know before reading read numbers: - -- **Buffered and unbuffered measure different systems.** Buffered goes - through Cc, which supplies all of this driver's read-ahead, and is what - playback looks like. `streams unbuffered` bypasses Cc so - every read reaches the driver, which is what to use when comparing the - transport against a usermode HTTP client. -- **Warm runs are not measurements.** A second run against the same files is - served from the Windows cache at thousands of MB/s with zero paging reads. - Reboot the guest between points, and check that `paging reads` is non-zero - before believing a number. - -```bash -PerfHarness.exe seq B:\media\big.mkv --report run.txt -``` - -**One stream is not the workload.** Several readers at once is the normal -case -- a video and its subtitle track, a game streaming assets while its own -data file is open, a library browse overlapping playback -- and it behaves -differently enough that single-stream numbers can look healthy while the -system is starving streams outright. +Exit code is 0 only if everything in the tier passed. Run `-Tier Fast` before +calling any change done — cheaper tiers don't run the crypto tests that catch +a `Tls.c` regression. -```bash -PerfHarness.exe streams B:\ 8 30 -powershell -File tools/Measure-BlorgScaling.ps1 -OutputPath baseline.txt -``` +Three GitHub Actions workflows cover the rest: `build.yml` gates every push +and PR to `master` at the Fast tier, `verify.yml` runs CBMC proofs and +extended fuzz/interleaving coverage nightly, and `codeql.yml` runs weekly +(and on PRs touching its own config) with the pinned Microsoft driver query +packs. -`streams` gives each reader its own file and thread and reports the **latency -tail** (p50/p95/p99/max) and a **fairness** ratio alongside aggregate -throughput. Aggregate alone is the wrong metric: a stream that stalls for a -second has failed even when the total looks fine, and equal throughput split -unequally is a different system from one that shares. +## Deploying to a VM -`Measure-BlorgScaling.ps1` sweeps stream counts and adds scaling efficiency -against the single-stream result. **It reboots the guest between runs**, which -is not hygiene: the Windows cache holds the previous run's files, and a -counter reset does not touch it. A second run against the same files is -served from RAM at thousands of MB/s with no paging reads at all, so a sweep -without a real reset measures that from the second row onward. `sc stop` cannot provide that -reset -- it wedges in `STOP_PENDING` (see `deploy/DEBUGGING.md`). +BlorgFS is a kernel driver, so it's developed and tested against a throwaway +Windows VM rather than the build machine. Copy `deploy/blorgfs.env.example` +to `deploy/blorgfs.env` and fill it in once; after that, deploying takes no +arguments: -Workload commands reset the counters first, so the numbers are attributable to -the workload. `--report` writes flat `key=value` metrics for -`tools\Compare-BlorgMetrics.ps1`, which checks both correctness invariants -(every inline paging read must have a fetch; fetch issues must balance -terminations) and perf deltas against a baseline in `tools\baselines\`. - -Accept new numbers deliberately, never silently: - -```bash -powershell -File tools/Invoke-BlorgChecks.ps1 -Tier Perf -PerfFile B:\media\big.mkv -UpdateBaseline +```powershell +.\deploy\Deploy-ToVM.ps1 -Configuration Release ``` -The driver runs in a VM for testing — `deploy\Deploy-ToVM.ps1` builds, copies, -and installs it via vmrun. Run the `Perf` tier inside the guest, where the -volume is mounted. - -## Playback stutter: subtitles, and why the obvious measurements missed it - -Measured 2026-08-28 against master, Release, Driver Verifier off, on the -reported symptom: video and subtitle lag watching media off `B:`. - -The file that reproduces it is H.264 + AAC + `S_TEXT/UTF8` at 2.6 Mbit/s -- -0.33 MB/s sustained against a driver that does 17-28 MB/s. Throughput was -never the question. **The trigger is enabling subtitles**, and until they -are on, nothing about this driver is visible to the player at all. - -### What subtitles change - -Same file, same player, same session; the only difference is the subtitle -track: - -| | subs off, 25 seeks | subs on | -|---|---|---| -| user reads | 75 | 1211 | -| mean | 713 us | 5794 us | -| max | 16.2 ms | 258 ms | -| **over one frame (41 ms)** | **0 (0.00%)** | **86 (7.10%)** | -| paging reads sequential | 100% | **70.4%** | -| user bytes | 19.6 MB | **428 MB** | - -Three things happen at once. Sequential paging reads fall to 70.4%, so the -cache manager -- which only predicts forward-sequential access -- stops -prefetching for roughly a third of them. The player reads about 22 times -more data than the bitrate needs, because a text subtitle track is sparse -and interleaved and following it drags the demuxer across ranges the video -stream never touches. And the resulting latency distribution is bimodal -with nothing between the modes: 863 reads at 16-64 us because the bytes -were resident, 111 reads at 16-131 ms because they were not. - -### It is this driver, on a clean control - -The same file was copied to the guest's own disk and played by the same -player, same subtitle track, same seeking, same two vCPUs. **It played -smoothly, and this driver's counters recorded zero reads for the duration**, -so the control is clean rather than merely plausible. - -That exonerates decode -- identical decode and compositing work, no -stutter -- and retires CPU capacity as an independent explanation, since -two cores are demonstrably enough for this content. - -It also makes the outlier records legible. The six worst fetches show 88 to -159 ms in `send`, for a request of a couple of hundred bytes, with -`FetchesActive` of 0 or 1: nothing else in flight, and no physical way a -send of that size takes that long. Those are the driver's own completions -being delayed by the driver's own work. Playing from this volume adds -428 MB through WSK receive, HTTP parsing and copies on top of decode, where -the local path adds close to nothing. - -So both driver-side costs share one root, and both scale with over-fetch: - -- a cold read costs a round trip plus a 512 KB to 1 MB body, against - roughly 0.1 ms for the same read locally, and -- moving those bytes costs CPU that delays the completions of the very - fetches being waited on. - -### Concurrency cannot fix this - -`FetchesActive` is 0 on five of the six worst fetches and 1 on the other. -At the moment of every stall the driver had nothing else in flight, so the -pipeline was idle, not saturated. - -That is the shape of the problem. Concurrency scales throughput; this is -latency on a serial dependency chain. The demuxer cannot issue the next -read until the current one returns, because its contents determine the next -offset, and the cache manager only parallelises access it can predict. -Adding width buys nothing here, and would cost something -- more in-flight -completions competing for the two cores already delaying the ones we have. - -Which is why the mitigations that matter attack latency: fetch less per -cold read, or do not go to the network at all. - -### NTFS as the reference - -Measured through the same standard FSCTL this driver implements, same -workload (256 KB requests over a cold region of the same 1.2 GB file): - -| | average read the cache manager issues | -|---|---| -| NTFS (`C:`) | **98 KB** | -| BlorgFS (`B:`) | **675 KB** | - -NTFS issues roughly one disk read per request, sized just above it -- 2,677 -user reads produced 2,681 disk reads -- and does not batch aggressively even -while streaming sequentially at 771 MB/s. This driver clusters about seven -times larger. - -That reframes `READ_AHEAD_GRANULARITY`. The evidence table in `Driver.h` -compared 256 KB, 512 KB and 1 MB and picked the middle, but every option -tested was already far outside where the reference filesystem sits, and all -three were measured on eight concurrent streams -- a throughput workload, -not the latency-shaped one that stalls. It is a well-evidenced choice among -outliers. - -The counter-argument is real and has to be measured rather than assumed: -NTFS reads from a local disk where a request costs about 0.1 ms, so small -reads cost it nothing, while this driver pays a ~12.7 ms round trip per -request. Shrinking granularity trades stall latency for more round trips on -sequential streaming. `ReadAheadGranularityKb` in the service's -`Parameters` key exists to sweep that trade without a rebuild per point; -zero means never call `CcSetReadAheadGranularity` and leave Cc's default. - -### The FSP pool, which had never been measured either - -`FSP_THREAD_COUNT` is `min(max(4 x cores, 8), 16)`, and the argument behind -it is sound -- the pool absorbs blocking rather than CPU, a blocked worker -costs nothing, so small machines keep a floor. But the number came from -reasoning, and nothing could have contradicted it. The only counter was -`ReadsPosted`, which is **zero in every read workload**, because the PASSIVE -bypass keeps reads off the queue entirely. What uses the queue is the -metadata path, and no read benchmark touches it. - -There is no standard constant to copy. In-box filesystems mostly do not run -a private pool at all -- they post to the system worker queues and let the -kernel size them -- but that is closed off here, because posted work waits -on an HTTP round trip and blocking a system worker thread is not allowed. -Once the pool is yours, the convention that applies is the one this driver -already used for `SocketMaxPoolSize`: size it from measured peak -concurrency. That comment records what guessing cost the last time -- a flat -32 justified by an internal notion of pipeline depth, 35-47 fresh connects a -run at 473-660 ms each, p99 635-704 ms, fairness 0.23. - -`FspPosts` and `FspDispatches` make the same measurement possible here. Two -monotone counters, no gauge: depth is their difference, computed by the -reader, which is the resolution this block already reached for fetches in -flight. - -| workload (cold path cache) | posts / dispatches | outstanding at end | -|---|---|---| -| metadata storm, 8 passes (601 creates) | 29 / 29 | 0 | -| metadata storm, 30 passes (2251 creates) | 117 / 117 | 0 | -| sequential read | 0 / 0 | 0 | - -117 posts across a run against a pool of eight, with the queue empty again -by the end. That bounds the work rather than recording the worst instant, -and the bound is what the convention allows: a high-water mark needs an -interlocked global, and this block does not write shared state on any path. -One was written here first and removed for exactly that reason. - -**The first version of this measurement was warm, and wrong.** It reported -11 and 30 posts at an 87-100% path-cache hit rate, because something had run -before it. Cold, the hit rate is 3-12% and the posts quadruple. A metadata -figure measured after anything else has touched the share says very little. - -Still not stressed: a tree far wider than this share, or many processes -opening at once, would post more than 117. - -### Real playback, and where the residual stutter actually is - -The synthetic workloads all read one pattern flat out or to a schedule. A -real player with subtitles and seeking does neither, and two captures of one --- `reset`, play, `stats` -- say more about what is left than any of them. - -| | original diagnosis | capture 1 | capture 2 | -|---|---|---|---| -| reads over one frame | 7.10% | 5.21% | **4.77%** | -| amplification | 22x | 0.62x | **0.46x** | -| sequential share of paging | 70.4% | 68.6% | 63.9% | - -**The amplification the branch set out to fix is fixed.** 22x to under 0.5x -- -read-ahead now fetches half of what the application consumes. - -**The read-ahead policy is correctly inert during playback**, which is the -point of it. Across 1543 evaluation windows it cast 5 shrink votes and zero -grow votes, and both are right: there is nothing to shrink at 0.46x, and the -consumer idles 92% of wall so `ReadIsGreedy` refuses to grow for it. The -granule stays at 128 KB. An earlier worry that a slow consumer would never -complete a window was wrong -- windows complete in abundance. - -**What is left is not read-ahead.** 394 of 646 fetches are DEMAND fetches -- -someone blocked -- because seeks defeat Cc's prediction, and a demand fetch -averages 25.6 ms against a 41.67 ms frame. Decomposed: - -| phase | mean | share of a fetch | -|---|---|---| -| acquire | 7 us | ~0% | -| send | 2.7 ms | 9% | -| wait, the server thinking | 8.3 ms | 27% | -| **body, bytes on the wire** | **20.3 ms** | **65%** | - -Two thirds of a demand fetch is ~290 KB crossing a 24-29 MB/s WiFi link with -a viewer waiting. No policy fixes that. - -### The send, split - -The send looked like the largest unexplained slice: 5.1 ms mean and 164.7 ms -max for a request of about two hundred bytes. `FetchSendSubmit*` and -`FetchSendSettle*` split it at the point WskSend accepts the buffer, because -this driver building the request and the stack delivering the completion -would need opposite fixes. - -It is almost entirely settle -- 2537 us of 2701 mean, and 185846 of 185900 -max. Submit, which is everything this driver does, is 163 us. - -The maximum is not the network. Its outlier record reads `send 185900, -wait 15, activ 0`: the response headers arrived 15 microseconds after the -send completed, so the request had reached the server long before, and -nothing else was in flight. That is a completion delivered late in a -two-processor guest, most plausibly the VM being descheduled. - -The mean is real and reproducible. Paced against greedy, same build, cold: - -| workload | idle share | settle mean | -|---|---|---| -| greedy sequential | 0.08% | 868 us | -| paced 3 MB/s | 99.89% | 2042 us | -| paced 1.5 MB/s | 99.82% | 2058 us | - -**An idle connection costs about 1.2 ms on the next send's completion**, and -the two paced rates agree to within 1%, so it tracks idleness rather than -rate. Real playback's 2537 us sits in that band. - -**It was written up here first as "roughly half the latency the user waits -on", which was wrong.** That compared the send against time-to-first-byte -rather than against the fetch. Against the fetch it is 9%, and the idle -penalty alone is under 5% -- removing it entirely would not move a 41.67 ms -frame budget. Recorded because the effect is real and reproducible, not -because it is the stutter. - -### The full matrix - -Every workload, one reboot per case so none inherits the previous one's -guest cache, and every case on a file no other case touches. Latency and -throughput percentiles are exact (`Sampler`), not from the driver's -power-of-two histogram. Amplification is fetched bytes against what the -workload consumed. - -| case | MB/s | amp | latency p1/p50/p99 ms | tput p1/p50/p99 MB/s | missed | grow/shrink | -|---|---|---|---|---|---|---| -| A1 sequential | 26.94 | 1.00x | 0.004 / 0.007 / 58.1 | 0.00 / 31.75 / 34.75 | -- | 4 / 0 | -| A2 random 64 KB | 10.18 | 0.90x | 0.006 / 6.21 / 12.3 | 8.50 / 9.75 / 12.00 | -- | 0 / 0 | -| A3 bursty demux | 22.11 | 0.72x | 0.004 / 0.006 / 14.8 | 6.25 / 8.00 / 8.75 | -- | 0 / 0 | -| A4 mixed seq to random | 24.57 | 0.88x | 0.003 / 0.007 / 55.5 | 13.25 / 23.50 / 40.00 | -- | 4 / 0 | -| A5 mixed seq to bursty | 22.64 | 1.19x | 0.004 / 0.006 / 55.8 | 5.25 / 9.00 / 48.25 | -- | 4 / 4 | -| A6 paced 3 MB/s | 3.00 | 1.01x | 0.021 / 0.038 / 0.217 | 3.00 / 3.00 / 3.00 | 0.83% | 0 / 0 | -| A7 paced 8 MB/s | 8.00 | 1.01x | 0.011 / 0.031 / 23.3 | 4.00 / 8.00 / 9.00 | 3.88% | 2 / 0 | -| B1 one FCB, seq + seq | 32.25 | 1.01x | 0.004 / 0.006 / 81.1 | 0.00 / 31.75 / 48.50 | -- | 4 / 0 | -| B2 one FCB, seq + bursty | 25.39 | 1.08x | 0.021 / **20.4** / **207** | 0.25 / 1.50 / 5.25 | -- | 4 / 0 | -| B3 one FCB, seq + paced | 24.21 | 1.01x | 0.004 / 0.020 / 68.5 | 1.00 / 3.00 / 4.00 | 9.38% | 3 / 0 | -| C1 eight FCBs, greedy | 24.71 | 1.00x | -- | -- | -- | 0 / 0 | -| C2 eight FCBs, paced | 23.96 | 1.01x | -- | -- | 6.45% | 3 / 0 | -| E0 driver reference | **28.38** | 1.00x | 0.003 / 0.006 / 61.3 | 9.75 / 31.75 / 47.75 | -- | 4 / 0 | -| E1 usermode reference | **28.58** | -- | -- | -- | -- | -- | - -**The ceiling: 28.38 against 28.58, a ratio of 0.993**, on two files nothing -else in the matrix reads, each on its own cold boot. There is no headroom -left against what a plain HTTP client gets from this link. - -**Amplification is solved across the board.** Nothing exceeds 1.19x, and -that worst case is the seq-to-bursty transition unwinding a grown granule. -Random and demux fetch LESS than they consume -- part of their demand is -already resident. The policy grows only where growth is wanted: four grows -on every sequential case, zero on random and bursty, and four grows against -four shrinks on the transition as it re-adapts. - -**Paced playback on a genuinely cold file misses 0.83% at 3 MB/s and 3.88% -at 8 MB/s.** Every earlier figure in this document said zero, and every one -of those was measured on a file some previous run had already pulled through -the backend. Throughput holds at p1 = p50 = p99 = 3.00 MB/s, so the misses -are the cold-start transient rather than a failure to keep rate -- but zero -was wrong, and it was wrong because of test hygiene rather than anything in -the driver. - -**Where the tail actually lives.** Two cases dominate it, and both are -contention rather than policy: - -- B2, a demuxer sharing one FCB with a copy: p50 20.4 ms and p99 207 ms - against 0.006 ms p50 when it runs alone. The copy takes the link. -- B3 and C2, paced consumers against a saturated link: 9.38% and 6.45% - missed. The growth-on/growth-off arms measured earlier put the copy at - 21-27 MB/s of a 24-29 MB/s link either way, so bandwidth is what starves - them, not the granule. - -Throughput p1 reaching 0.00 MB/s on A1 and B1 is the same story seen from -the other end: quarter-second windows where a greedy reader gets nothing -while the link serves something else. - -**Two gaps in the matrix itself.** The eight-FCB cases report the harness's -own p50/p95/p99 rather than `Sampler`'s p1 and throughput distribution, so -their rows are blank above. And the backend's page cache is not controlled -by any of this -- it cannot be from the guest -- so absolute figures are -against unknown backend warmth. The E0/E1 pair is the number to trust, -because both halves of it were measured minutes apart on files of equal -coldness. - -### Where each workload stands - -The read-ahead policy is one rule per direction, and this is what it -produces. Everything below was measured on cold files, one workload per -file, arms alternating with a guest reboot between runs. - -| workload | result | granule | amplification | -|---|---|---|---| -| sequential, greedy (file copy) | **ratio 1.01 to a usermode client** | grows to 2 MB | 1.00x | -| paced playback, 3 MB/s | **0.00% of deadlines missed** | stays 128 KB, never grows | -- | -| paced playback, 8 MB/s | 1 miss in 1600, 1.35 ms late | stays 128 KB | -- | -| paced demux, 4 tracks | 0.17% missed | stays 128 KB, never grows | 1.19x | -| random, 64 KB blocks | -- | stays 128 KB, never grows | 0.91x | -| eight paced players at half the link | 0.19% missed, 13.4 ms worst | stays 128 KB | -- | -| eight paced players AT the link | 0.5-33% missed, every configuration | -- | -- | -| sequential then seeking, one handle | no cost, no unwind needed | keeps 512 KB | 0.56x | -| sequential then **bursty**, one handle | settles, no oscillation | unwinds from 2 MB | **1.56x** | -| two handles, a copy against a paced player | player missed 0 of 1200 | -- | -- | - -Nine of the ten are at their best available answer. The tenth is not a -defect but a limit -- eight players demanding 100% of a variable link miss -deadlines in every arm measured -- pinned or adaptive, 128 KB or 512 KB, any ceiling -- -so granularity does not control it and nothing here is tuned for it. - -**The sequential figure is a ratio, not a rate, and that is deliberate.** -The path to the backend is WiFi, so absolute MB/s moves by tens of percent -between runs and a number like "28 MB/s" means nothing without saying when. -Alternating the driver against a usermode client within one session, each -pair seeing the same medium, puts the driver at 0.96-1.10 across eight runs, -mean 1.02. There is no headroom left against what a plain HTTP client gets -from the same link. - -**Amplification stopped being a trade rather than being balanced.** It was -the price of a constant granule large enough for sequential reads -- 27.5x -on the demux pattern at 512 KB. With 128 KB as the floor every file starts -at, and growth reserved for a reader that is both sequential and never idle, -nothing that seeks ever leaves the floor: the sparse demux and random -patterns fetched byte-identical totals at every ceiling from 512 KB to -16 MB, with zero grows in all of them. Random fetches *less* than it -consumes, because part of what it asks for is already resident. - -The rules that produce this, in full: - -- **Shrink** when a window fetched more than twice what it consumed. This - guards every pattern and consults nothing about the consumer, because - wasted bytes are wasted whether or not anyone has a deadline. -- **Grow** only when the consumer never idles (a copy, not a player), the - transport is quiet, sixteen reads have been exactly adjacent, and Cc is - still honouring the granule it was last given. - -Each condition earns its place by a measured failure without it, recorded in -the sections below: growing for a lone player costs it deadlines, growing on -a busy transport was justified on readers with no deadline, growing for a -seeking reader is where 27.5x amplification came from, and growing past what -Cc will honour is how a ceiling gets fitted to one link. - -### What the means were hiding - -Latency was reported from the driver's power-of-two histogram, so a p99 of -`<=32768 us` meant somewhere between 16 and 32 ms -- a band twice as wide as -most of the differences being argued about here. Throughput had no -distribution at all: every figure in this document is bytes over a whole -run, which cannot tell a workload that ran evenly from one that stalled for -two seconds and then raced. - -`Sampler` records exact per-read latencies and stamps bytes into 250 ms -intervals, so each interval's rate is a sample. Both ends are reported -- -p1 as well as p99 -- because the interesting tail for latency is the high -one and for throughput the low one, and carrying both stops the wrong end -being quoted. - -Two competing handles, each owning half the file so neither reads what the -other cached: - -| run | handle | latency p1/p50/p99 | throughput p1/p50/p99 MB/s | mean MB/s | -|---|---|---|---|---| -| copy vs copy | 0 | 0.004 / 0.006 / 0.039 ms | **2.00 / 13.00 / 9771** | 1361 | -| copy vs copy | 1 | 0.003 / 0.004 / 0.025 ms | **0.00 / 10146 / 15412** | 7837 | -| copy vs paced player | player | 0.011 / 0.021 / 0.254 ms | **3.00 / 3.00 / 3.00** | 2.99 | - -**Throughput p1 is 0.00-3.00 MB/s in every run**, against p99s in the -thousands. There are quarter-second windows where a handle moves nothing at -all, and latency maxima of 182-466 ms in the same runs agree. A mean of -7837 MB/s is an average of cache hits and dead stalls, and it was the only -number this harness produced before. - -**The paced player is flat at its target** -- p1, p50 and p99 all exactly -3.00 MB/s, two misses in 1200, worst 5.63 ms -- while a copy hammers the same -FCB. That is the shared-verdict hazard failing to materialise, and unlike -the earlier run it is not a cache artifact, because the halves are disjoint. - -**Two identical copies took 5.75x different shares** (196 GB against 34 GB). -Same workload, same file, disjoint halves, and no explanation yet. With two -processors, two reader threads and the driver's own work, scheduling is as -plausible as anything in the policy. - -**Above RAM, the picture changes completely.** Half of a 450 MB file fits in -4 GB of guest memory, so both handles went cache-resident after the first -pass and never touched the fetch path. Repeated on a 22 GB file -- halves of -11 GB, which cannot be cached -- the throughputs drop from thousands of MB/s -to tens, and the contention becomes real: - -| pairing | the copy | the other half | other's latency p50 / p99 | -|---|---|---|---| -| copy vs paced player | 23.32 MB/s | 2.99 MB/s, **180 of 1200 missed**, worst 217 ms | 0.021 / 98.6 ms | -| copy vs bursty | 50.50 MB/s | 1.59 MB/s | 7.51 / 213.7 ms | -| copy vs copy | 58.95 MB/s | 22.60 MB/s | 0.006 / 30.3 ms | - -**The 15% figure did not replicate and is withdrawn.** Four further runs of -the same pairing missed 0, 0, 0 and 10 of 1200. One run in twelve produced -180; it was an outlier, and quoting it as a property was wrong. - -**What growth actually costs the player, measured directly.** Forbidding -growth while leaving the copy in place separates the copy's bandwidth from -its granularity decision: - -| slack growth | the copy | player misses of 1200 | -|---|---|---| -| off | 21.63, 20.99 MB/s | 0, 0 | -| on | **26.11, 27.01 MB/s** | 0, 10 (worst 88 ms) | - -The copy gains about 25% and the player pays between nothing and ten misses. -That is the policy working as intended rather than a fairness fault, and it -is the answer to whether `READ_AHEAD_ADAPT_QUIET_DEPTH` needs to become -adaptive: the gate is fragile -- a lone reader runs 2.7 to 4 fetches in -flight and eight paced streams run 3 to 8, so the two distributions overlap --- but what it is guarding against costs ten missed deadlines in twelve -hundred. Fixed at six, documented, and not worth machinery. - -**The asymmetry between identical copies is the backend, not this driver.** -`compete` takes a `swap` argument that inverts which handle owns which half. -The advantage moves with the half, not the handle: - -| run | handle 0 | handle 1 | -|---|---|---| -| normal | 67.87 MB/s (half 0) | 44.23 MB/s (half 1) | -| swapped | 56.70 MB/s (half 1) | **79.88 MB/s (half 0)** | -| normal | 12.39 (half 0) | 12.40 (half 1) | -| swapped | 25.84 (half 1) | 25.72 (half 0) | - -Whichever handle owns the FIRST half wins when there is a difference at all, -and two of the four runs show none. Nothing in the read-ahead policy is -involved -- but the attribution offered here first, that a per-request -`ServeFile` seek makes the start of a 22 GB file cheaper than a point eleven -gigabytes in, is the weaker of the two available explanations. - -The backend holds no block cache of its own: files above 8 MB -(`max_resident_file_bytes`) are cached only as the decision to stream them, -so every range GET reads through the backend HOST's page cache. The first -half of that file is what every run in this document has touched, and the -second half is what almost nothing has. Residency explains an advantage that -follows the offset at least as well as seek geometry does, and better than -seek geometry explains it on any modern storage. Distinguishing them needs -the backend's cache dropped between arms, which nothing here has done. - -### A reader that changes its mind - -Every workload above holds one pattern for its whole run, which left the -interesting moment unmeasured: a granule grown for a file copy is exactly -wrong for the seek that follows it, and the policy only unwinds on evidence --- two agreeing windows per halving, a window being two granules wide, so -the bytes it takes to come down scale with how far it went up. From 2 MB -that is about 15 MB consumed before the granule is back at the floor. - -`mixed` reads sequentially then seeks, on one handle, counting the phases -separately against the driver's own counters: - -| | mixed, 200 MB seq then 1500 x 64 KB random | control, pure random, cold | -|---|---|---| -| random-phase throughput | 13.56 MB/s | 8.90 MB/s | -| random phase fetched | 52.4 MB for 93.8 MB consumed (**0.56x**) | 87.6 MB for 93.8 MB (0.93x) | -| grows / shrinks in random phase | **0 / 0** | 0 / 0 | -| granule entering the random phase | 512 KB | 128 KB | - -**The inherited granule costs nothing, and the slow unwind never has to -happen.** The random phase entered at 512 KB and did not over-fetch: 52.4 MB -against roughly 53 MB of genuinely uncached demand, so about 1.0x once the -sequential phase's cached 200 MB is accounted for. No window ever voted -shrink. - -The reason is structural rather than lucky. `CcSetReadAheadGranularity` only -sizes read-ahead, and Cc arms read-ahead only when its own sequential -detector fires. A seeking reader never arms it, so the granule is inert for -it however large it was left. That also explains something observed earlier -without being understood: demux and random fetched byte-identical totals at -every ceiling from 512 KB to 16 MB. - -**Sequential into a BURSTY pattern was broken, and the cause was an ordering -mistake made in this branch.** Pure random is safe by construction because it -never arms read-ahead. A demuxer does -- short adjacent runs, then a jump -- -and it inherits whatever the sequential half grew to: - -| second phase | fetched vs consumed | shrinks | phase throughput | -|---|---|---|---| -| pure random | 0.56x | 0 | 13.56 MB/s | -| bursty, before the fix | **2.82x** | **0** | 7.43 MB/s | -| bursty, after | **1.87x** | 30 | 7.80 MB/s | - -Two guesses failed before the counters were added, and both are worth -recording because the arithmetic looked convincing each time. Widening the -window to four granules should have moved the shrink trigger from 3x to -2.5x and caught the 2.82x; it changed nothing. Halving the lead allowance -should have done the same; it also changed nothing. - -`ReadAdapt*` in Statistics.h answered it in one run: **56 grow votes, zero -shrink votes, across 89 completed windows.** The windows were completing and -the vote was never being cast, because the shrink test was not being reached -at all: - -``` -if (greedy && honoured >= current && streak >= GROW_STREAK) vote = -1; -else if (fetched > consumed * WASTE + lead) vote = 1; -``` - -A bursty reader going flat out is greedy, and `ReadLongestStreak` takes the -maximum across trackers while `ReadClaimStream` evicts the coldest -- so the -streak the sequential half left behind is the last thing ever evicted and -never decays. The grow arm stayed true, and `else if` meant waste was never -evaluated. - -The reordering came from the commit that freed growth from the amplification -gate. Removing the EARNED comparison was the part that mattered; putting -growth first was gratuitous, and it disabled the shrink rule for exactly the -pattern that rule exists to catch. Waste is tested first again, and the lead -stays at one granule so the trigger sits at 2.5x rather than 3x. - -Testing waste first does not restore the gate that was removed. What blocked -growth was the EARNED comparison, which a reader consuming everything sits -exactly on. The WASTE comparison is a bar at two and a half times, and a -sequential reader measures 1.00-1.04x -- it cast zero shrink votes across -125 windows after the change. - -**The oscillation it left behind is fixed too, by asking a different -question.** The first fix left the bursty phase bouncing -- 26 grows against -30 shrinks -- because the growth arm consulted the LONGEST streak across -trackers, and the stale one from the sequential half is the last thing -ReadClaimStream ever evicts. What growth wants to know is whether the stream -it is on right now is long, not whether any stream on the file ever was. -`ReadCurrentStreak` answers that from `ReadLastStreamIndex`, which the read -path records anyway. - -Decaying the streaks was the obvious alternative and would have broken -growth: at the starting granule a window holds about four reads, so a -decayed streak could never reach the sixteen growth requires. - -| | amplification | grows / shrinks | phase throughput | -|---|---|---|---| -| original defect | 2.82x | 0 / 0 | 7.43 MB/s | -| waste tested first | 1.87x | 26 / 30, oscillating | 13.07 MB/s | -| **current stream's streak** | **1.56x** | **0 / 4**, settles | **15.51 MB/s** | - -Reproduce with `mixed 200 64 1500 1024 4`. - -### The granularity sweep, and why there is no right constant - -That trade has now been measured rather than assumed, and the answer is -that no value of `READ_AHEAD_GRANULARITY` is free. - -First, the fact that decides how to read the table. **Cc's default is not -adaptive.** Between `CcInitializeCacheMap` and `CcSetReadAheadGranularity` -the granularity of a cached file is `PAGE_SIZE` -- a constant, 4 KB. So -"leave the default" is not letting Cc choose, it is pinning the granule to -one page, the smallest value in the sweep. Every point below is a constant; -the sweep has no adaptive arm in it at all. - -Granularity is the rounding unit for read-ahead, so a small read drags its -whole aligned granule, and a large granule means a demand read waits on a -large fetch. That single mechanism produces both columns: throughput rises -with granule size because fetches get larger and fewer, and the tail rises -with it because each stall is longer. - -Measured on one cold guest per matrix, a dataset no other cell touches, and -the granularity assignment alternated within each workload so file identity -cannot track the setting. Sequential rows are the mean of two runs with the -assignment reversed; the others are single runs. - -| granule | seq MB/s | seq tail | streams x8 MB/s | streams x8 tail | demux MB/s | demux amplification | demux tail | -|---|---|---|---|---|---|---|---| -| 512 KB (committed) | **27.1** | 3.26% | **32.2** | 13.3% | 0.75 | 28.1x | 4.2% | -| 128 KB | 20.8 | **0.11%** | **32.1** | 29.2% | 0.64 | 20.6x | 0.44% | -| 64 KB (FastFat) | 16.8 | 0.13% | 28.2 | 15.6% | **0.86** | 10.8x | **0.22%** | -| 4 KB (`PAGE_SIZE`) | 16.1 | **0.09%** | 28.7 | 14.0% | 0.82 | **1.4x** | 1.0% | - -"Tail" is the share of application-visible reads longer than a 24 fps frame -interval -- the number a viewer can see, not the mean. - -The shape is monotone and the endpoints both regress: - -- **512 KB costs the tail.** Sequential buffered reads spend 3.26% of their - reads over a frame against 0.11% at 128 KB, a factor of thirty. At four - concurrent streams its worst read was **1087 ms** against 118 ms at - `PAGE_SIZE`. On a filesystem whose stated purpose is playback, a - one-second read is the whole complaint. -- **`PAGE_SIZE` costs throughput.** Sequential buffered drops from 27.1 to - 16.1 MB/s, **-41%**, because the same bytes now take 4096 fetches instead - of 832 -- five times the round trips at ~12.7 ms each. - -Two controls back the rig up. Unbuffered sequential reads, where Cc is out -of the path, move -5% with amplification 1.00 at both ends -- no effect -where none is possible. Buffered 4 KB random reads are flat at 0.39 MB/s -and amplification 1.00, because Cc's read-ahead is pattern-triggered and -never arms on uniformly random access, so granularity is inert there. - -**Unresolved, and load-bearing before anything is committed:** - -- `streams x8` at 128 KB reported a 29.2% tail, roughly double every other - setting, from a single run against a different directory. That is either - the one real objection to 128 KB or it is noise, and one sample cannot - tell. -- The `meta` storm produced no usable result at any setting, so - metadata-heavy work is unmeasured. -- The demux workload reads in 4 KB blocks while Media Player issues ~262 KB - reads. The granule that minimises amplification is the one matching the - read size, so the playback column is the row most flattering to small - values and the one least entitled to decide this. - -### Why a constant is the wrong shape - -The two regressions above are not in tension by accident. Large granules -win when a file object is being read forwards, because the fetch amortises; -they lose when it is being picked at, because a small read waits for a large -fetch. Those are different file objects, often at the same moment -- a video -track streaming while a subtitle track is picked at -- and one constant has -to serve both. - -The driver already knows which is which. `READ_STREAM_TRACKER` in the FCB -exists to answer exactly that question and `BlorgReadIsSequential` already -reports 54-73% sequential on these workloads. So the shape that fits the -evidence is not a better constant but a granularity that starts small and -is raised for a file object whose reads have proven sequential -- Cc's -per-file-object setting is the right granularity of control for it, and the -adaptivity Cc does not provide is adaptivity this driver has the state to -provide itself. - -`CcSetReadAheadGranularity` is re-callable, which the design depends on and -nothing documents. It is described as setting the value for a cached file, -FastFat calls it once at cache-map time, and a silently-ignored second call -would have killed the idea. Measured rather than assumed: starting a demux -run at 512 KB and switching to `PAGE_SIZE` partway through moved -amplification from 27.5x to **18.7x** on the same file and the same -setting, with the switch point accounting for the size of the move (it -fires after 300 paging reads, and a 512 KB run only produces about 420, so -roughly 71% of the run stayed at 512 KB: 0.71 x 27.5 + 0.29 x 1.4 is about -20). Run-to-run variation on that cell is ~2%, so a 32% drop is the -re-call, not noise. Speculative reads rose 167 to 246 and demand reads 245 -to 361 across the same switch, which is what smaller granules produce. - -What is still unestablished is the microscopic mechanism. Read-ahead is -demonstrably what costs the slow reads -- suppressing it with -FILE_FLAG_RANDOM_ACCESS drops reads over a frame from 46/43 to 2/4 across -replicates, with ReadsSpeculative confirming the suppression actually -happened -- but whether the damage is collision (a reader faulting into a -range an in-flight read-ahead already claimed, and inheriting its latency), -transport congestion from fetching 27x the bytes, or priority inversion is -not settled by aggregates. An earlier reading of this section claimed -collision on the grounds that the worst application read tracked the worst -speculative fetch. That was wrong: the maximum sits at 58-66 ms in every -configuration measured, including with read-ahead entirely suppressed, so -it is a floor common to all of them and not evidence of anything. The -metric that does discriminate is the count of reads over a frame, not the -maximum. - -### The granule depends on load, which is why no constant worked - -Reads over a frame interval are roughly the share of application reads that -have to wait for a fetch, times the chance that wait exceeds the frame. A -granule serves many reads, so halving it roughly doubles how many wait and -halves how long each waits. Which way that trades depends entirely on where -fetch latency sits relative to 41.67 ms -- and load moves it: - -| | fetch mean | reads over a frame | -|---|---|---| -| one reader, 128 KB | 19.8 ms | 0.345% | -| one reader, 512 KB | 58.7 ms | 3.40% | -| eight streams, 128 KB | 41.8 ms | 28.9% | -| eight streams, 512 KB | 102.5 ms | 13.7% | - -Unloaded, a 128 KB fetch lands comfortably inside the frame, so the extra -waits cost nothing and the small granule wins tenfold. Saturated, a 128 KB -fetch takes 41.8 ms -- sitting exactly on the threshold -- so nearly every -extra wait becomes visible and the large granule wins by making far fewer -reads wait at all. Throughput is identical at that point (31.3 against 31.8 -MB/s), so at eight streams this is a latency decision and not a bandwidth -one, and the benchmark that originally chose 512 KB could not have seen it. - -That is the whole reason no constant worked. The variable is load, and -neither a fixed value nor an amplification-driven policy can observe it. -The driver can: fetches in flight run about 1.6 to 2.8 for a single reader -and 21 to 28 at eight streams. - -So growth requires a sequential reader **and** a loaded transport. Sequential -alone is not sufficient and that is the trap -- a lone reader walking a file -forwards has the same long streak and the same 1.0 amplification as eight of -them, and growing for it is exactly the 3.40% case. Files start at 128 KB, -shrink on measured waste, and climb to 512 KB only when depth crosses the -threshold. - -Measured against starting at 512 KB, each cell rebooted with its own dataset -and the configurations alternated: - -| workload | start at 512 KB | load-aware, start at 128 KB | -|---|---|---| -| one sequential reader | 3.39% over a frame, 27.4 MB/s | **0.03%**, 19.5 MB/s | -| eight streams | 15.06%, 31.8 MB/s | **14.47%**, 32.4 MB/s | -| demux (playback) | 0.83%, 0.99 MB/s | **0.08%**, 1.17 MB/s | - -It takes the better of the two constants in each case rather than splitting -the difference, and the eight-stream cell recorded sixteen grows -- eight -file objects each doubling twice -- which is the load gate doing exactly -what it was built for, per file object. - -The cost is 29% of throughput for a lone sequential reader, 19.5 MB/s -against 27.4. That is a file-copy shape rather than a playback one, 19.5 -MB/s remains an order of magnitude above any bitrate this volume serves, and -it is the price of keeping a lone reader's fetches inside the frame budget. -Single runs per cell; the underlying constants were replicated, this -comparison was not. - -**Measure interleaved or not at all.** Every earlier comparison here was -A-then-B in time, and the host drifts: the same 128 KB setting measured 20.2 -MB/s at 0.16% in one session and 12.6 MB/s at 1.8% an hour later. An ordered -comparison hands that drift to whichever setting ran second. The numbers -above alternate configurations with a reboot per run; the fixed-granule -replicates came back at 3.37/3.42% and 0.34/0.35%, tight enough to decide -on, which none of the earlier ordered runs were. The reversal controls used -before this protected against dataset bias and did nothing about drift. - -### Why the trade exists at all: too few fetches in flight to cover ttfb - -Everything above tunes *around* per-fetch cost. A large granule is only -better because it amortises a fixed cost over more bytes, so the size of -that fixed cost decides how sharp the trade is. - -This section previously claimed that 35% of a fetch was this driver's own -overhead, from a phase split against a usermode client: - -| per fetch | driver, 196 KB avg | usermode, same bytes | -|---|---|---| -| ttfb | 10.10 ms | 5.33 ms | -| body transfer | 9.52 ms | ~7.5 ms | - -**That comparison was invalid and the conclusion drawn from it was wrong.** -The driver column was measured at a pipeline depth of ~2.7 fetches in -flight; the usermode column was a strictly serial client, one request at a -time. Sweeping the usermode client across concurrency, with completion -stamped by polling the async handles so that no response is charged the -drain time of its predecessors: - -| concurrency | 1 | 2 | 3 | 4 | 6 | -|---|---|---|---|---|---| -| usermode ttfb, in guest | 6.05 ms | 9.78 ms | 12.46 ms | 17.73 ms | 22.41 ms | - -At the driver's own depth of ~2.7 a usermode client sits near 11.6 ms -against the driver's 9.5. There is no 35% gap. The driver is at or slightly -better than a usermode client at matched depth, and the projected "41 MB/s -at parity" was never available -- it was above the physical ceiling, which -should have been the tell. - -The rise is not queueing inside the server either. Sweeping response body -size from the host, so that neither this driver nor the guest network stack -is in the path: - -| body | conc 1 | conc 2 | conc 3 | conc 4 | conc 6 | -|---|---|---|---|---|---| -| 256 B | 3.80 ms | 2.35 | 2.44 | 1.99 | 2.31 | -| 8 KB | 2.05 ms | 2.11 | 2.70 | 3.17 | 3.48 | -| 64 KB | 2.33 ms | 4.37 | 6.24 | 6.24 | 7.78 | - -At 256 bytes ttfb is flat in concurrency; the climb appears only as bodies -grow, and minimum ttfb stays 1.3-2.3 ms at every point. A file small enough -to be served from the backend's resident cache queues identically to one -large enough to be streamed per request, so it is not the server's file -handling. Concurrent response bodies contend for link bandwidth and delay -later responses' headers. Time to first byte under load is a property of -the link, not of this driver and not of the backend. - -### The ceiling: ~24-29 MB/s, and it is the network - -| measurement | MB/s | -|---|---| -| single stream, from the guest | 23.96 | -| single stream, from the host | 23.70 | -| 2 streams from the guest, aggregate | 21.30 | -| 4 streams from the guest, aggregate | 21.21 | -| 8 MB ranges, advancing (cold) | 28.45 | -| 8 MB ranges, same range repeated (warm) | 29.30 | - -Host and guest are indistinguishable, so it is not the guest's virtual NIC. -Aggregate does not rise with stream count -- four streams get a quarter each --- so it is not per-connection. Warm and cold ranges are identical, so it is -not the backend's disk. It is the network path, and it is the same from -anywhere. - -This agrees with the ~30 MB/s ceiling recorded under "Why this, and not more -lookahead" below, measured independently at 16 streams, and extends it: the -wall is already reached by a *single* connection, so concurrency divides it -rather than growing it. - -It also settles a contradiction this document was carrying. That section -measured the driver cold at 0.93-1.01x a usermode client while the section -above claimed a 35% deficit against one. The 0.93-1.01x figure was right; -the 35% came from comparing unequal pipeline depths. - -This bounds every throughput number here. The 512 KB constant's 27.4 MB/s on -a lone sequential reader was already taking ~96% of the wall; there was -never 40 MB/s to find. - -### What that leaves: depth, not granule - -Per-stream throughput is `granule / (ttfb + granule/bandwidth)`, multiplied -by the number of fetches in flight and capped by the link: - -| granule | body at 28 MB/s | + ttfb | per slot | x2.7 depth | observed | -|---|---|---|---|---|---| -| 512 KB | 18.3 ms | 28.3 ms | 18.1 MB/s | capped at wall | 27.4 | -| 128 KB | 4.6 ms | 14.6 ms | 8.8 MB/s | 23.7 MB/s | 19.5 | - -The small granule does not lose throughput because it is small. It loses -because at a depth of ~2.7 there are not enough fetches in flight to cover -each one's ttfb, so a fixed ~10 ms is amortised over 128 KB instead of over -512 KB. At depth 4 a 128 KB granule reaches the wall on the arithmetic -above, with every stall still one small granule long. - -That is the shape of "best of both worlds", and it is not the same lever as -the fetch splitting removed below. Splitting divided a single *demanded* -fetch into concurrent slices that were all needed at once, so the pieces -only shared bandwidth with each other. Depth adds *speculative* fetches -further ahead in the file, whose bytes will be consumed anyway. Concurrency -on bytes that are already wanted is close to free; concurrency on bytes -wanted right now just divides the same bandwidth finer. - -This is not in tension with "Why this, and not more lookahead" below, which -rejects lookahead as a way to *add* throughput. It is rejected there because -cold reads at a 512 KB granule already sit at the wall, and nothing on the -fetch path can beat the wire. Depth here does not aim past the wall; it aims -to reach it at a granule small enough to keep stalls short. The 128 KB -configuration is at 19.5 MB/s against a ~28 MB/s ceiling, so unlike the -512 KB case there is a gap, and it is the gap the small granule opened. - -Lookahead today is entirely Cc's read-ahead -- the driver's own prefetcher -and its chunk budget are gone -- and Cc puts about 4 fetches in flight for a -single stream, ~2.7 measured on average. Raising effective depth for a -sequential reader without raising the granule is the untried lever, and it -is the one the arithmetic points at. Nothing in this section has been -attempted. - -### The workload with a deadline, which the harness did not have - -Every workload here reads flat out. `seq`, `rand`, `demux` and `streams` all -consume as fast as the driver delivers, and a reader going flat out is -permanently at the read-ahead frontier: it takes each granule the instant Cc -produces it, so it is maximally exposed to fetch latency. The "over one -frame (41 ms)" figure reported for such a reader is hypothetical -- it asks -whether a viewer would have seen a hitch, of a reader that has no deadline. - -That matters because the whole adaptive granularity trade was arbitrated on -that figure. `play` is the missing workload: blocks read on a fixed -schedule, block n due at `start + (n+1) * block / rate`, counting blocks -that missed their due time and by how much. - -At 3072 KB/s in 128 KB blocks -- a 41.67 ms frame at roughly a Blu-ray -bitrate: - -| granule | missed | worst late | app mean | fetch mean / max | speculative : demand | -|---|---|---|---|---|---| -| 128 KB | 0 / 600 | 0.00 ms | 92 us | 16.1 / 66.8 ms | 340 : 1 | -| 512 KB | 0 / 600 | 0.00 ms | 63 us | 41.0 / 55.1 ms | 115 : 1 | - -At 8192 KB/s, a 15.62 ms interval: - -| granule | missed | worst late | app mean | fetch mean / max | speculative : demand | -|---|---|---|---|---|---| -| 128 KB | 1 / 1600 | 1.35 ms | 59 us | 16.4 / 73.7 ms | 499 : 2 | -| 512 KB | **11 / 1600** | **64.15 ms** | 128 us | 42.9 / 106.1 ms | 187 : 2 | - -Two things fall out, and they point opposite ways. - -**Cc absorbs almost everything.** 340 speculative fetches against one demand -fetch: the path a user waits on is essentially never on the wire. The driver -issued fetches of 66-106 ms throughout while the application saw means of -59-128 us. This is the same effect recorded when the user-read timer was -added -- 67 ms fetches behind a reader seeing 0.11 ms -- and it is why fetch -latency is the wrong layer to draw conclusions from. - -**The tail protection is nonetheless real, above a rate.** At 3 MB/s the -granule does not matter; both are perfect and 512 KB is marginally better. -At 8 MB/s the 512 KB granule produces 11 missed deadlines with 64 ms worst -lateness, against one miss 1.35 ms late at 128 KB. That is a hitch a viewer -would see, on a consumer that genuinely has a deadline. The hypothesis that -the greedy harness invented the tail is **wrong** above about 3 MB/s. - -So the adaptive policy is defensible, and the remaining compromise is -narrower than it looked. It is one case: the greedy sequential reader -- -a file copy -- which wants 512 KB for 27.4 MB/s against 19.5, and which has -no deadline to protect. The policy refuses to grow for it because it cannot -tell a copy from a lone player, and it is right to be cautious given the -8 MB/s row above. - -The signal that separates them is not sequentiality, amplification or -transport load, all of which they share. It is **slack**: a paced consumer -idles between reads -- 99% of wall clock at both rates above, since a read -served from cache costs microseconds against a 15-41 ms interval -- and a -copy never idles at all. Measured per stream as the gap between one read -completing and the next arriving, that distinguishes "has a deadline" from -"wants throughput" directly, where every signal the policy uses today is a -proxy for it. Growing only for a stream with no slack would take the file -copy to 512 KB while leaving both paced rates where they are. - -Measured, as instrumentation only -- `ReadIdle*` in the statistics block, -the gap from one application-visible read completing on a file to the next -arriving on it, reported by the harness as an idle share of wall clock: - -| workload | idle share | idle p50 | -|---|---|---| -| greedy sequential, buffered | 0.09% | <=2 us | -| streams x8 | 0.03% | <=2 us | -| demux, 4 tracks, burst 4 | 0.53% | -- | -| rand 64 KB buffered | 2.07% | <=2 us | -| paced 8 MB/s | 94.99% | <=16 ms | -| paced 3 MB/s | 99.85% | <=65 ms | - -Three orders of magnitude apart with nothing in between. Every greedy -workload lands under 2.1% and every paced one above 94%, so any threshold -from 10% to 90% classifies all six correctly. As a discriminator this is -about as clean as a runtime signal gets, and it is the only one measured -here that separates a file copy from a player -- sequentiality, -amplification and transport load are all shared between them. - -Nothing reads these to make a decision. Three things have to be answered -first, and two of them are new: - -- **A starving player stops idling.** Its rate rises until it is asking - continuously, at which point it looks exactly like a copy -- and that is - when a large granule hurts it most. A slack rule needs a floor a - struggling player cannot fall through, which probably means pairing slack - with whether demand rate responds to delivery rate: a copy's does, a - player's does not. - - How close that gets is now measurable rather than hypothetical: eight - players missing 6.57% of their deadlines at the link ceiling still measure - 92.06% idle, because a consumer that is occasionally late is still mostly - waiting on its own schedule. The signal degrades toward greedy only under - sustained starvation, which leaves more headroom above the threshold than - the worry assumed -- but the direction of the failure is unchanged. - -- **The harness had to start representing production before the policy - could key on this**, and now does. `demux` and `streams` take an optional - `pace=`, sharing one `Pacer` with `play` so that paced means one - thing across all three. Paced, they land where the things they stand for - do: - - | workload | idle share | missed | worst late | - |---|---|---|---| - | demux, unpaced | 0.10% | -- | -- | - | demux, paced 3072 KB/s | 99.91% | 1 / 3000 (0.03%) | 0.05 ms | - | streams x8, unpaced | 0.02% | -- | -- | - | streams x8, paced 1536 KB/s each | 98.94% | 9 / 4800 (0.19%) | 13.41 ms | - | streams x8, paced 3072 KB/s each | 92.06% | 631 / 9600 (6.57%) | 208.87 ms | - - The last row is eight players demanding 24 MB/s of a link that delivers - 24-29. It delivered 23.98 at fairness 1.00 and every one of the eight - still stuttered, which is what running a deadline workload at the wall - looks like -- and it is the case where the granule choice matters most, - so it is the one a policy change has to be judged on. - - Two harness bugs had to be fixed to get those numbers, and both had - produced confident nonsense first: - - - An FCB outlives its handles on the delayed close list, so the first read - of a run was charged the gap since the *previous* run's last read on - that file. Unpaced demux reported 99.70% idle from one 402-second - sample, in a run lasting seconds. The stamp is now cleared on open. - - - Eight paced streams started together came due at the same instant, which - is one burst of eight demands per interval rather than eight players. - In phase they missed 7.33% of deadlines with a 500 ms worst case while - asking for 12 MB/s of a link that had just delivered 21.7. Staggered by - `i/N` of the interval, the same configuration misses 0.19% at 13.41 ms. - - The second is worth keeping in mind beyond this harness: synchronised - arrivals make a link look saturated at half its capacity. - -- **Run-to-run variance is larger than the effect on some cells.** The - 8 MB/s paced case missed 1 deadline in 1600 in one run and 65 in another - at the same granule, and greedy sequential measured 12.56 MB/s here - against 19.5 elsewhere. Any decision taken on these numbers needs the - interleaved-with-reboot shape, not two runs compared across a session. - -The amplification shrink rule composes with all of this and is unaffected: -wasted bytes are wasted whether or not anyone has a deadline, which is what -keeps the demux pattern protected however it is classified. - -### Growing on slack: on by default - -The adaptive policy grows the granule for a consumer that never idles as -well as for a loaded transport. `ReadIsGreedy` in Read.c compares a window's -idle ticks against its service ticks, greedy below 25%, and it is consulted -only while fewer than `READ_AHEAD_ADAPT_QUIET_DEPTH` fetches are in flight. -`ReadAheadSlackGrowth` (registry) still switches it off. - -Two rounds with the switch alternating, a guest reboot between every run, -**and the arm order alternating between rounds**: - -| workload | slack | MB/s | missed | worst | grows | -|---|---|---|---|---|---| -| seq, greedy | 0 | 19.04 | -- | -- | 0.0 | -| seq, greedy | 1 | **22.52** | -- | -- | 2.0 | -| play 3072 | 0 | 3.00 | 0.00% | 0.00 ms | 0 | -| play 3072 | 1 | 3.00 | 0.00% | 0.00 ms | 0 | -| demux paced 3072 | 0 | 3.00 | 0.78% | 89.22 ms | 0 | -| demux paced 3072 | 1 | 3.00 | 0.17% | 50.95 ms | 0 | -| streams x8 paced 3072 | 0 | 23.73 | 27.81% | 1347.86 ms | 8.0 | -| streams x8 paced 3072 | 1 | 23.95 | 6.11% | 397.48 ms | 11.0 | - -The sequential case is what this exists for: +18%, `min(slack=1) = 22.44` -above `max(slack=0) = 19.73` so the arms do not overlap, and the mechanism -observed rather than inferred -- exactly two grows in every switch-on run -and none in any switch-off run. Paced playback misses no deadline in either -arm and never grows. - -An earlier sweep put the same gain at +38%, from four rounds that alternated -the arms but always ran slack=0 first. Fixing the order took it to +18%, so -**half of that first number was warm-up**. It is recorded because the -correction is the useful part: alternating the configuration is not enough -if the order within a round is fixed. - -Two rows carry no claim. Demux differs in slack's favour with **zero grows -in both arms**, so the policy cannot have caused it. And the eight-stream -grow counts range from 0 to 16 *within a single arm*, so that workload's -variance swamps any difference between arms; the gate means slack cannot -fire while the transport is busy, which is where that workload lives, but -indistinguishable in noisy data is not the same as identical. - -### The loaded rule is settled, and it is gone - -The policy also grew the granule whenever the transport was busy. That rule -predated slack and its case was this, from the granularity sweep: - -| | 128 KB | 512 KB | -|---|---|---| -| one stream | 0.345% over a frame | 3.40% | -| eight streams | 28.9% | 13.7% | - -Loaded, the large granule looked like it won by a factor of two. Every -number in that table came from a reader going flat out. - -Pinning the granule -- `ReadAheadAdapt=0` in the registry, so the feedback -loop cannot confound the arms -- and running consumers that read to a -schedule reverses it. Two sweeps, two rounds each, arm order rotating: - -| paced workload | pin 128 KB | pin 512 KB | -|---|---|---| -| play 3072 | 0.00% missed | 0.04% | -| demux paced 3072 | 0.39% | 4.76% | -| streams x8 paced 1536 | 0.33% | 1.73% | - -Twelve times worse on the demux pattern, five times worse on eight paced -streams at half the link. So the rule is removed, and growth now happens for -exactly one reason: a sequential consumer that never idles, on a quiet -transport. - -With it gone, the adaptive policy records **zero grows on every paced -workload that is not at the link ceiling** -- it is pin128 for those -- -while still growing twice for a greedy sequential reader: - -| | adaptive | pin 128 KB | pin 512 KB | -|---|---|---|---| -| seq, greedy, cold | **22.52 MB/s** | 16.48 | 20.26 | -| play 3072 | 0.00% | 0.00% | 0.00% | -| demux paced 3072 | 0.17% | 0.41% | 1.02% | -| streams x8 paced 1536 | 0.26% | 0.11% | 1.00% | - -**One workload is not tuned for and cannot be.** Eight paced streams -demanding the whole link miss between 0.5% and 33% of their deadlines in -every arm -- pinned or adaptive, small granule or large: - -| arm | per-run missed, pooled over both sweeps | -|---|---| -| pin 128 KB | 2.30%, 3.48%, 0.52%, 33.12% | -| pin 512 KB | 19.31%, 26.92%, 3.43%, 1.68% | - -A consumer asking for 100% of a link is unstable however it is fetched, and -granularity does not control it. An earlier reading of two runs per arm made -the large granule look eight times worse there and it was written up that -way; the second sweep put the same arm at 2.56% and the claim was withdrawn. -Two runs of a workload whose spread is 0.5% to 33% is not a measurement. - -That row is also the one place the slack rule misfires, and it is worth -being exact about. Eight paced streams run only 3 to 8 fetches in flight -- -they idle 92% of the time, so they are nowhere near the 21-28 a greedy -eight-stream load puts up -- which straddles `READ_AHEAD_ADAPT_QUIET_DEPTH`. -When they starve at the ceiling they stop idling, and the policy grows: 8.5 -times a run, against zero on every other paced workload. This is exactly the -failure predicted when slack was first measured -- a player that cannot keep -up looks like a copy -- and depth cannot separate the two here, because a -lone copy runs 2.7 to 4 in flight and these run 3 to 8. - -It does not show up as harm: adaptive missed 10.07% on that row against -pin128's 16.82% and pin512's 2.56%, all three inside the same 0.5-33% spread. -So it is recorded rather than fixed. Fixing it needs a signal that separates -one starving consumer from one greedy one, and neither slack nor depth is -that signal. - -### Reaching the ceiling, without paying for it anywhere else - -Growth stopped at 512 KB, which is where a sequential reader stopped too. -`ReadAheadMaxGranularityKb` makes that bound configurable, and sweeping it -one boot and one cold file per arm -- the Windows cache survives a counter -reset, so any workload after the first on a file measures Cc rather than -this driver: - -| cap | seq MB/s | grows | -|---|---|---| -| 512 KB | 23.05 | 2 | -| 1 MB | 27.34 | 3 | -| 2 MB | **28.38** | 4 | -| 4 MB | 27.91 | 5 | - -4 MB buys nothing, so 2 MB is the default now. - -**This was written up as reaching 99.8% of the link and that was wrong.** -The 28.38 came from the guest in one sweep; the 28.45 "ceiling" came from -the host in a different session. The path to the backend is WiFi -- an -Intel AX200 at a 649 Mbps link rate, 2 ms away -- so the ceiling is a -variable medium, and comparing two numbers taken hours apart says nothing. -That is the same cross-session error this document warns about twice -elsewhere. - -Measured properly, by alternating the driver and a usermode client within -one session so each pair sees the same medium, on a different cold file -each time: - -| pair | driver | usermode | ratio | -|---|---|---|---| -| ep 01 | 20.74 MB/s | 25.31 | 0.82 | -| ep 02 | 19.83 | 23.94 | 0.83 | -| ep 03 | 18.87 | 23.82 | 0.79 | - -**The driver reaches about 0.8 of what a usermode client gets on the same -link at the same moment**, consistently across three pairs. The granule -sweep above is still valid -- it was internally consistent, one boot per arm --- and 2 MB is still the right cap. But the remaining gap is roughly 20%, -not nothing. - -The shape of it is worth stating because it is not mysterious. The usermode -client issues ONE request for the whole file and pays time-to-first-byte -once. The driver pays it per granule: at 2 MB and ~25 MB/s that is an 80 ms -body behind a ~10 ms ttfb, about 11% overhead, which is the right order for -the gap. Closing it means overlapping a fetch's ttfb with the previous -fetch's body rather than making the granule larger -- 4 MB already showed no -gain. - -The check that matters is not the sequential column but the other two: the -sparse demux pattern fetched **223 MB at every one of those caps** and the -random pattern **85 MB**, both with zero grows. A higher ceiling is -unreachable for them because they never satisfy the streak condition, so -raising it cannot cost them anything. - -Confirmed against the previous default across every workload, each on its -own cold file: - -| workload | 512 KB cap | 2 MB cap | fetched | -|---|---|---|---| -| seq, greedy | 19.91 MB/s | **26.59 MB/s** | 466 MB both | - -| play 3072 | 0.00% missed | 0.00% missed | 76 MB both | -| demux sparse, stride 1024 | 10.77 MB/s | 10.82 | 223 MB both | -| rand 64 KB buffered | 10.30 MB/s | 10.39 | 85 MB both | -| streams x8 paced 3072 | 2.62% missed, 144 ms | 1.23%, 95 ms | -- | - -### Amplification, and why it stopped being a problem - -The granule was once a constant, and a constant large enough for sequential -reads cost the demux pattern 27.5x amplification -- 512 KB fetched to serve -a fraction of it. That is what made the trade look unavoidable. - -It is not a trade any more, because the large granule is no longer a -constant. 128 KB is the floor every file starts at, and the only reader that -ever leaves it is a sequential one that never idles. Measured against -consumption, on cold files: - -| pattern | fetched | consumed | amplification | -|---|---|---|---| -| sparse demux, 64 KB blocks, 1024 KB stride | 223 MB | 187.5 MB | 1.19x | -| random, 64 KB blocks | 85 MB | 93.75 MB | 0.91x | -| sequential, greedy | 466 MB | 465.6 MB | 1.00x | - -Random reads fetch *less* than they consume, because some of what they ask -for is already resident. Nothing here is above 1.2x, and none of it moved -when the growth ceiling was raised four-fold. - -### What made the ceiling portable: asking Cc, not the clock - -Both rate-based tuners failed on the same thing -- an 11% marginal effect -cannot be seen through 30% noise. The signal that works is not a rate at -all. - -Cc does not read ahead in whatever size it is told. It caps somewhere of its -own choosing, around 1.1 MB here, and the cap sweep shows it plainly: the -same 465 MB file was fetched in 418, 413, 421 and 412 requests at ceilings -of 2, 4, 8 and 16 MB. Four ceilings spanning 8x, and the request count does -not move. Everything above ~1 MB was inert. - -So growth now stops when Cc declines to honour the granule -- the largest -paging read seen in the window against the granule asked for. Two byte -counts, no clock, no noise: - -| | cap 2 MB | cap 16 MB | -|---|---|---| -| seq, ratio to usermode | **1.01** | **1.01** | -| seq grows / shrinks | 4 / 0 | 5 / 0 | -| play 3072 missed | 0.09% | 0.00% | -| demux sparse fetched | 223 MB, 0 grows | 223 MB, 0 grows | -| rand 64 KB fetched | 85 MB, 0 grows | 85 MB, 0 grows | -| streams x8 paced missed | 1.58% | 1.94% | - -An eight-fold change in the configured bound moves the ratio not at all, and -the grow count by one step instead of the three the fixed ceiling took -(4, 5, 6, 7 at 2, 4, 8, 16 MB). The ceiling is now mostly a safety bound -rather than the operative limit, which is what portability means here: on a -system where Cc reads further, the policy follows it, and on one where it -reads less, the policy stops earlier -- without either being configured. - -It is not perfectly bound-independent. The test uses the largest paging read -in the window, which is permissive: one oversized read admits another -doubling, which is why 16 MB reached five grows and 2 MB four. Both landed -at ratio 1.01, so the extra step cost nothing here, and a stricter statistic -would be tuning against a single environment again. - -**Amplification cannot regress through this, and not by luck.** Growth -requires a greedy consumer AND sixteen consecutive exactly-adjacent reads. -A seeking reader has neither, which is why demux and random fetched -byte-identical totals with zero grows at every ceiling tried across the -whole sweep series. The new gate only narrows growth further, so it cannot -reach them. - -### A self-tuning ceiling: built twice, measured, reverted - -The growth ceiling is a constant fitted to one link, and where a sequential -reader stops gaining is a property of the link rather than of this driver. -2 MB suits a 2 ms, ~28 MB/s WiFi path; a 50 ms path needs a granule several -times larger before time-to-first-byte is amortised at all. So the ceiling -should be measured, not configured. - -It was built that way: after each doubling, the next window's rate is -compared against the window before it, and a doubling that fails is stepped -back, ending the climb for that file. Rates compare as `bytes1*ticks2` -against `bytes2*ticks1`, exact in integers with no counter frequency needed. - -**Both versions regressed sequential throughput and neither is in the tree.** - -| | seq MB/s | ratio to usermode | grows | shrinks | -|---|---|---|---|---| -| fixed 2 MB ceiling | ~32 | 1.02 | 4-7 | 0 | -| keep only if 10% better | 18.20 / 18.89 | 0.68 / 0.76 | 1.0 | 1.0 | -| revert only if 10% worse | 19.21 / 23.87 | 0.77 / 0.80 | 1.5-2.0 | 1.0 | - -The first version had the test backwards, and the cap sweep says why: 512 KB -to 2 MB is two doublings worth 23% together, so the marginal gain is about -11% per step. Requiring more than 10% improvement per doubling, against a -link whose own noise is tens of percent, fails every trial on its first -judgement -- growth stopped after one step, 40% down. - -Inverting it to stop only on measured harm should have been safe, because -its failure mode is climbing to the bound, which is what the fixed ceiling -already does. It still settled after one or two steps, because a 10% dip -between adjacent windows happens by chance: the same configuration measured -14.36 and 24.07 MB/s in consecutive rounds. - -**The conclusion is about the signal, not the threshold.** Per-window -throughput cannot resolve an 11% marginal effect on a medium that moves by -30%, and no margin fixes that -- a threshold loose enough to survive the -noise is loose enough to never fire. Locating the knee needs either a stable -link or a signal that is not end-to-end throughput. - -The ttfb/body ratio was the obvious alternative and does not work either. -It stays near 2.5 whether the granule is 512 KB or 2 MB, because -time-to-first-byte grows with the granule too -- concurrent fetches queue -behind each other's bodies, so the thing that would signal "stop growing" -scales with the growth. - -What this leaves is an honest constant with its provenance recorded: -2 MB, measured on this link, with the sweep that chose it and a registry -override (`ReadAheadMaxGranularityKb`) for a link where it is wrong. The -work is in `git stash` if a stable link ever makes the measurement possible. - -### Pipelining the receive: built, measured, reverted - -The phase split says a fetch spends about 9.5 ms before its first byte -against a usermode client's 5.3 ms, and the obvious suspect was ordering: -the receive was posted from the send *completion*, so a send-completion DPC -stood between a response already sitting in the socket buffer and anyone -asking for it. Posting the receive as soon as the send was issued should -have removed it. - -It did not. Measured with a runtime switch so the two orderings could -alternate with a reboot each -- the only comparison shape that has held up -on this host: - -| | ttfb | fetch | MB/s | reads over a frame | -|---|---|---|---|---| -| receive posted at send-issue | 9.54 ms | 20.38 ms | 16.28 | 0.498% | -| receive posted at send-completion | **8.96 ms** | **19.48 ms** | **17.64** | **0.448%** | - -Consistently worse across both interleaved pairs, and the `wait` phase -- -the one it exists to shrink -- went up rather than down (7.93/9.61 against -7.23/9.19). A pre-posted receive that does not reduce time-to-headers means -the send-completion DPC was never what the request was waiting on. Reverted. - -Two things worth keeping from it. The sandbox caught a use-after-free in the -first version within one run: posting the receive *before* the send reads as -tidier and is wrong, because a receive can complete synchronously -- the -scripted peer does, and WSK may -- running the request to completion and -freeing the context before the send is issued. ASan named the line, and 23 -allocation-failure scenarios failed beside it. - -And if this is ever attempted again, the invariant that made two outstanding -operations safe without reference counting the whole state machine was -ownership: from the moment the receive is posted it owns the context, and -the send completion may only stamp a timestamp and record a status -- never -fail, retry, complete or kick. That is cheaper than a refcount and it is -checkable by reading one routine. - -The gap it was aimed at turned out not to exist. The 9.5 ms against 5.3 ms -compared this driver at a depth of ~2.7 against a serial usermode client; -at matched depth the usermode client is slower. Ordering was excluded by -the measurement above, and the premise was withdrawn by the one before it. -Pipelining was aimed at a target that was not there, which is why it moved -nothing. - -### Splitting the fetch: built, measured, removed - -Splitting one large fetch into concurrent range requests over partial MDLs -was built to get a short stall and a large granule at once. It measured -worse than not splitting, at every factor tried: mean fetch latency rose -from 46.1 ms to 51.2 ms at four slices, and to 50-57 ms at two. - -The connection counters exclude the obvious explanation -- 100% pool reuse, -no fresh connects, no retries, no timeouts -- and the arithmetic gives the -real one. 51.2 ms across four slices is about 12.8 ms each, which is serial -rather than concurrent. This guest has two processors, and four -simultaneous WSK receives each running a completion and an HTTP parse -cannot overlap on them. Concurrency is not a free lever here, and the code -was removed rather than kept as something that costs requests and returns -nothing. - -Recorded because the idea is a natural one to have again, and because -READ_AHEAD_PARAMETERS.PipelinedRequestSize in ntifs.h describes exactly -this and would invite it. On a guest with more processors the answer may -differ; on this one it does not. - -The same section previously claimed the tail was a reader colliding with an -in-flight read-ahead and inheriting its latency, on the grounds that the -worst application read tracked the worst speculative fetch. That was two -constants sharing a floor -- the maximum sits at 58-66 ms in every -configuration, including with read-ahead suppressed outright -- and it is -withdrawn. The count of reads over a frame is what discriminates; the -maximum never did. - -### What it is not - -Tested and dead, recorded so they are not re-proposed: - -- **Not the MKV Cues index.** The theory was that each seek drags a read to - the index at the end of the file. `end-of-file` reads: **0**. -- **Not metadata.** The theory was that the player re-opens or re-stats on - seek. `creates` and `file info`: **0 and 0**. -- **Not decode.** See the local control above. -- **Not seeking.** Twenty-five scripted seeks at two second intervals, the - rate the reporter used, produced zero reads over a frame interval. - -An earlier version of this section concluded that seeking was the cause, on -a 1:1 seek-to-stall figure taken from a paced *simulation* that issued -synchronous 64 KB reads and blocked on each. Media Player reads ~262 KB -asynchronously behind its own buffer, and paging reads stay 100% sequential -across 25 seeks. The simulation exhibited the problem it was built to look -for. - -### How to measure this, and four ways to get it wrong - -Every one of these cost a wrong conclusion before it was understood. - -**Driver-side latency is not what a user feels.** The cache manager sits in -front of all of it and exists to hide exactly those numbers -- driven at -playback rate the driver was issuing 67 ms fetches while the reader saw -0.11 ms. `UserReadLatency*` in `BLORGFS_STATISTICS` is the number to read; -the chunk-fetch block explains it but is not it. - -**Most application reads never become an IRP.** Buffered synchronous reads -of a cached file are served by fast I/O, so timing `IRP_MJ_READ` produced -one sample against a reader's 780. `FastIoRead` is wrapped -(`BlorgFastIoRead`) for this reason: `FsRtlCopyRead` is where a caller -blocks, because it calls `CcCopyRead` inline. - -**Validate instrumentation against something independent before trusting -it.** The user-read timer was checked against a usermode reader measuring -itself on the other side of the syscall boundary: 780 samples against 780, -max 31.174 ms against 31.4 ms. Without that step its first version -- which -saw one read in 780 -- would have looked like a finding. - -**A simulation of a player is not a player.** Read pace, read size, and -whether reads are synchronous all change the answer, and a reader written -to exhibit a hypothesis will exhibit it. Drive the real application: -`PerfHarness reset`, run it, `PerfHarness stats`. - -Two further traps already paid for: reads at full speed measure read-ahead -working rather than playback feeling slow, so pace a simulated reader at -the real bitrate; and every `*MaxUs` field was summed across processors -rather than max-reduced until 2026-08-28, so any maximum taken from this -driver before then is inflated by roughly the processor count. - -**A workload that cannot exhibit the effect will report that there is no -effect.** Four candidate workloads were tried against read-ahead -granularity before one could see it, and each was rejected on its own -measurement rather than on argument: unbuffered random (amplification -1.000, Cc is not in the path), buffered random (1.000, read-ahead is -pattern-triggered and never arms), a demuxer model whose cursors advanced -by their own block size (1.002 at 99.8% sequential -- `seq` in disguise, -read-ahead working perfectly), and the same model with a large stride but -no burst (2.07x at 1% sequential, too scattered to arm anything). Sweeping -any of the four would have produced a flat line and the confident wrong -conclusion that granularity does not matter. What reproduces the captured -trace is `demux`: several cursors, each reading a short adjacent burst then -skipping a stride, which is what a container's interleaved tracks look like -from one file object. The driver's own sequential test is exact adjacency, -so the trace's 70.4% sequential is itself the evidence that the real -pattern contains runs, not scattered reads. - -**Calibrating on one statistic is not validation.** That `demux` model was -tuned until its amplification matched the trace (24.8x against 22x) and its -sequential share was close, and the match was treated as licence to sweep. -It reads in 4 KB blocks; the player issues ~262 KB. Matching one number -while missing another by two orders of magnitude is the same class of error -as matching fetch latency and calling it what the user feels. - -**Compare shares, not counts, across runs of different length.** Runs with -different file sizes or durations produce different read totals, so an -absolute count of reads over a frame ranks the longest run worst. -`UserReadsOverFrameShare` exists for this. It is already a percentage -- -`SafeRatio` multiplies by 100 -- so multiplying again yields impossible -values above 100% and, worse, preserves the ordering while destroying the -magnitude. - - -## TODO: on-disk hot cache - -The prefetch ring was removed rather than replaced. What follows is the -design for its replacement, which is **not an in-memory prefetcher** -- that -experiment is finished and its evidence is in git history. Nothing in this -section is implemented yet. - -**It lives in the driver, as a module.** A usermode helper owning the store -was the other candidate, on the argument that it removed a re-entrancy -problem by construction. That argument was weaker than it looked: keeping -the store off this volume removes the recursion just as completely, since -nothing in NTFS's completion path calls back into this driver. What is left -is deadlock through the memory manager, which is a discipline problem the -helper would not have solved either -- it is the same problem every -filesystem has when it touches another one. - -### Why this, and not more lookahead - -Measured on the reference rig, 16 concurrent streams, 512 KB range GETs, -interleaved against a usermode HTTP client: - -| | throughput | -| --- | --- | -| network path ceiling, cold or warm | ~30 MB/s | -| driver, cold | 27-30 MB/s (0.93-1.01x the usermode client) | -| driver, warm in the Windows cache | **4800-6800 MB/s** | -| local guest disk, write / read | **601 MB/s / 4.3 GB/s** | - -Cold reads already sit at the link ceiling, so nothing on the fetch path can -add throughput: every delivered byte has to cross a ~30 MB/s wire. The only -way past it is to **not cross the wire**. RAM caching already demonstrates -the payoff and is bounded by RAM -- at four concurrent streams the working -set stopped fitting and a re-read fell from 6677 MB/s to 131 MB/s. Local -disk is ~20x the network on write, ~140x on read, with ~100x the capacity of -RAM. - -### Architecture: a driver module - -`src/DiskCache.c` / `DiskCache.h`, owning a block store in **one ordinary -file on an ordinary live volume**. Location and maximum size configurable -through the registry alongside `RemoteHost`/`RemotePort` in `Parameters`. - -The module boundary is deliberately narrow, and every entry point is either -pure memory or explicitly asynchronous: - -``` -BlorgDiskCacheInitialize / BlorgDiskCacheDrain startup, unload -BlorgDiskCacheLookup(FileId, BlockIndex) IN MEMORY ONLY, no I/O -BlorgDiskCacheReadAsync(Slot, Mdl, Completion) serves a hit -BlorgDiskCacheAdmitAsync(FileId, Block, Mdl) fire-and-forget write-behind -BlorgDiskCacheInvalidate(FileId) validator changed -``` - -`BlorgDiskCacheLookup` touching no I/O is what makes the rest safe: the read -dispatch path can ask "is this cached?" while holding whatever it holds, and -only then decide which asynchronous path to take. - -**The read path keeps the shape it already has.** A paging read today -returns `STATUS_PENDING` and is completed later from a network completion -(`Read.c`). A cache hit is the same shape with a different source: - -1. `BlorgDiskCacheLookup` — in-memory index, no I/O, no blocking. -2. **Hit**: queue a cache read; a worker fills `Irp->MdlAddress` and - completes the IRP. -3. **Miss**: issue the HTTP fetch exactly as now. On completion, complete the - IRP *first*, then queue the write-behind from the buffer already in hand. - -The reader never waits on the cache in either direction. A miss costs -nothing it did not already cost, and a write-behind failure is invisible. - -### What the store being off-volume does, and does not, buy - -**The store must not live on this volume**, checked at open by comparing the -target's volume device object with ours. That single rule is what removes -*recursion*: with the store on NTFS, nothing in the completion path of a -`ZwReadFile` calls back into BlorgFS. There is no cycle in the call graph, -and the usermode-helper alternative bought nothing here that this check does -not. - -What is left is not recursion, and calling it that obscures the actual -risks. Two remain, both mediated by memory manager: - -- **Deadlock through MM, not a nested call.** If a thread holds an FCB - resource and, inside a cache read, memory pressure makes MM trim that - file's pages, MM calls this driver's `AcquireForLazyWrite` and blocks on - the resource the thread is still holding. So: **never hold an FCB resource - across a cache call.** Enqueue, release, return pending. -- **The paging path is the dangerous one.** A paging read can originate from - MM while it is already short of memory. Dependent I/O issued from that - thread can wait on the reclaim that is waiting on us. So: **no cache I/O - on the calling thread** -- all `ZwReadFile`/`ZwWriteFile` happen on the - module's own PASSIVE workers, and the dispatch path only ever enqueues. - -That second rule costs nothing the design was not already paying. The IRP is -completed asynchronously either way, so moving the I/O to a worker changes -which thread finishes it and nothing else. - -**IRQL** is a hard constraint rather than a judgement call: `ZwReadFile` and -`ZwWriteFile` are PASSIVE-only, while network completions run at -`<= DISPATCH`, so an admit queued from a completion reaches PASSIVE through -a work item. That is the same rule the removed prefetcher lived by, and the -one thing from it worth keeping. - -**Open the store `FILE_NO_INTERMEDIATE_BUFFERING`**, for two reasons that -are worth stating accurately. It avoids double-caching bytes Cc already -holds for this volume, and it keeps the store from adding cache-manager -memory pressure at exactly the moment the driver is serving a paging read. -The cost is sector alignment, which a fixed-block store gives for free. - -An earlier draft of this section justified the flag by claiming that -cache-manager pressure from our own store could re-enter this driver through -its own cache callbacks. That is not true for a store on another volume, and -the flag is worth setting anyway for the two reasons above. - -### Store layout, index and recovery - -Fixed-size blocks, each preceded by its own header: magic, file-identity -hash, block index, backend validator, byte length, and the MAC below. Slots -are addressed by index, never by cluster or LCN, so fragmentation, extension -and defragmentation are all transparent. - -**The index is rebuilt from the block headers at startup, not persisted.** -A separate index file is faster to load and introduces a whole failure class -this does not need — an index that disagrees with the store, torn across a -crash, and confidently wrong. Header scan cannot desync because the headers -*are* the store. At 512 KB blocks a 30 GB store is ~61k headers; reading -only the header of each is a few hundred MB against a local disk measured at -4.3 GB/s, so a second or so of startup, off the mount path. - -Persisting an index is a later optimisation, and only worth it if that -startup cost ever shows up as a complaint. - -### Concurrency - -- Slot allocation from a free list under a leaf lock; nothing else is - acquired under it. -- Per-slot reference count so eviction cannot reclaim a slot with a read in - flight -- the same protocol the node table already uses, and the one the - systematic scheduler is set up to explore. -- A per-block "fetch in flight" marker so two readers missing the same block - do not both fetch it and both write it. - -### Cluster pinning: not worth it - -`FSCTL_MARK_HANDLE` with `MARK_HANDLE_PROTECT_CLUSTERS` marks a file so the -defragmenter will not move it. **The conclusion is not to use it.** The -intuition that this is over-engineering is right, and for reasons stronger -than "SSDs do not care about seeks": - -- It solves a problem this design does not have. Protection matters when - something maps a file by LCN and needs that mapping to stay valid -- - hibernation files, page files, block-level VM disks. This store is - addressed through the filesystem by offset, so a moved extent is - transparent. -- It is NTFS-only and volume-specific, and the cache is meant to live - wherever the user points it. -- Marking a large file unmovable is antisocial: it permanently constrains the - volume's own defragmenter on behalf of a cache that is by definition - disposable. -- The cost it avoids is seek cost, which on SSD is near zero and on spinning - media is still small against the ~30 ms network fetch it is competing with. - -Revisit only if profiling ever shows extent-map lookup -- not seek time -- -dominating cache reads, which would be a surprise. - -### Security model - -**Cache integrity is a security boundary.** A process that can write the -store can inject bytes this filesystem then serves as authoritative file -content: a straightforward data-poisoning primitive against every reader of -the share. - -- **ACL the store to SYSTEM and Administrators only**, deny everyone else, - and create it with an explicit security descriptor rather than inheriting - the parent directory's. A cache under a user-writable path with inherited - ACLs is the default-insecure outcome to avoid. -- **Refuse a store whose ownership or ACL is not what was expected**, at - open, rather than repairing it -- repairing races the attacker. -- **ACLs alone are not sufficient.** They do not cover an offline attack - (booting another OS, mounting the volume elsewhere), an administrator-level - compromise, or ordinary corruption. Blocks must carry their own integrity - check. -- **Per-block keyed integrity, verified before use.** Each block records a - MAC over (file identity, block index, backend validator, contents). A - mismatch discards the block and falls back to the network. A plain - checksum detects corruption but not tampering, and the threat here is - tampering. - - The key has to persist for the cache to survive a reboot, and it has to - live somewhere the store does not -- otherwise an attacker who can rewrite - blocks can recompute the tags and the MAC proves nothing. The service's - own registry key is the natural home: same trust boundary as the driver's - configuration, already SYSTEM-only, and already what `TlsPin` uses - (`Driver.c`). Generate it on first use, never log it, and treat a missing - key as an empty cache rather than an error. - - Running in kernel does not change the threat model here. The attacker of - interest is a process that can write the file, not one that can call the - driver -- and MAC verification happens on the module's worker before any - cached byte reaches an IRP, so a forged block is discarded on the same - path that would have discarded a corrupt one. -- **Bind blocks to a backend validator.** `server-rs` returns `etag` and - `last_modified`; a block whose validator does not match the current - response is stale and must not be served. Without this the cache serves - yesterday's bytes for a file that changed. -- **Tampering, truncation or wholesale replacement** must be - indistinguishable in effect from a cold cache: verification fails, blocks - are discarded, reads go to the network. - -The bar is explicit: **the cached path must not be weaker than the uncached -path.** A design step that cannot meet that does not ship. - -### Admission and eviction: start small - -The literature is consistent that the largest wins come from **admission** -control rather than clever eviction, and that the specific thing to avoid is -caching one-hit-wonders -- for a disk-backed store that is wasted write -bandwidth and, on SSD, wasted endurance. - -- **Admit on second miss, not first.** This is the CDN answer to one-hit - wonders, costs a few bytes of state per candidate, and is the single - highest-value policy decision available. -- **Prefer sequential streams.** The `READ_STREAM_TRACKER` array on the FCB - (`Structs.h`) already carries the streak -- it survived the prefetch - removal partly for this. A streaked reader is exactly the case where the - following blocks are worth having. -- **Evict with segmented LRU.** Cheap, well understood, and resistant to one - large scan flushing the whole store. -- **Do not start with TinyLFU/W-TinyLFU or ARC.** W-TinyLFU is the strongest - general result in the literature and is the right thing to *grow into* if - measurement justifies it; its frequency sketch costs about a page. But it - earns its keep on skewed, high-cardinality, small-object workloads -- CDN - edges, key-value caches -- and this workload is a handful of very large, - sequentially-read files. Second-hit admission plus SLRU captures most of - the benefit at a fraction of the complexity, and the counters will show - whether anything more is warranted. - -Measure hit rate, bytes served from cache, and write amplification before -tuning any of it. - -Sources: [TinyLFU (ACM ToS)](https://dl.acm.org/doi/10.1145/3149371), -[size-aware admission for CDN memory caches (CMU)](http://reports-archive.adm.cs.cmu.edu/anon/2016/CMU-CS-16-120.pdf), -[CacheSack: admission optimization for Google datacenter caches (USENIX ATC 22)](https://www.usenix.org/system/files/atc22-yang-tzu-wei.pdf), -[FSCTL_MARK_HANDLE](https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ni-ntifs-fsctl_mark_handle), -[Defragmenting Files](https://learn.microsoft.com/en-us/windows/win32/fileio/defragmenting-files). - - -## Deploying to the VM - -**See `deploy\README.md`** — the full pipeline, the two-step INF install this -primitive driver requires, the staged-vs-repo-root INF hash trap, `vmrun` -argument/password quirks, and KDNET connection notes all live there. That -document is deliberately tool-agnostic; keep new deployment findings in it -rather than here. - -The one rule worth repeating: **never deploy by copying `.sys` into -`System32\drivers` by hand.** It skips catalog validation and can leave a -stale binary in place, so what you debug is not what you built. Use -`deploy\Deploy-ToVM.ps1`. - -## What is expected of you here - -This is a kernel filesystem driver on a hot path, so the bar is higher than -"it compiles and the tests pass": - -- **Scrutinise for correctness and performance, actively.** Do not wait to be - pointed at a bug. Read the surrounding code when you touch a file and - question it — the double-counted `NonCachedReads`, the vendor IOCTLs that - could never route, and the unload that never ran were all found by looking, - not by being asked. -- **Prefer optimisations that also simplify.** The best change removes code - and instructions at the same time. Collapsing two branches that test the - same predicate, deleting a redundant pass, replacing pointer-chasing with a - flat array — these are wins twice over. A "fast" change that adds - complexity needs to justify itself with a measurement. -- **Fix style inconsistencies as you find them.** The conventions below are - opinionated and deliberate. Code that drifts from them is a defect; correct - it in passing rather than matching the drift. -- **Measure before and after** anything performance-related. `Statistics.h` - counters are always on, `PerfHarness` reports them, and - `tools\Compare-BlorgMetrics.ps1` diffs against a baseline. An unmeasured - performance claim does not land. - -## Changing behaviour: the full pass - -A behaviour change is not done when the code works. Every one of these, in -the same pass: - -1. **Update the comments that described the old behaviour.** The header - comment above the function, the struct-field comment, the file-header - contract — whichever described what you just changed. A stale comment - actively misleads, which is worse than silence. -2. **Add or update a regression test, in the layer that can actually catch - it.** This is the part most often got wrong. The statistics IOCTL bug - passed every existing dispatch test because they call - `BlorgDeviceControl` directly and so never exercise the I/O manager's - routing decision — the only test that could catch it asserts on the - `CTL_CODE` device type itself. Ask what layer the bug lives in, and put - the test there: crypto vectors for `Tls.c`, the usermode sandbox for - `Client.c` logic, the kernel rule-model for IRQL/lock-order, the - systematic scheduler for interleavings, `Test-BlorgCorrectness.ps1` for - anything only the real driver plus real backend can show. -3. **Comment the test too**, to the same standard — say what it is defending - against and why the obvious cheaper test would not catch it. -4. **Adjust the documentation files.** `README.md` for build/test/convention - changes, `deploy\DEBUGGING.md` for anything about diagnosing the running - system, `deploy\README.md` for deployment mechanics. -5. **Run `-Tier Fast`** before calling it done. - -## Documentation discipline - -Most of the expensive problems in this project were diagnosed once, in a -session that then ended. The write-ups are the only reason the next person -does not pay for them again — so treat them as part of the work, not an -epilogue. - -**Where findings go.** Environment, VM and debugger findings belong in -`deploy\DEBUGGING.md`; deployment mechanics in `deploy\README.md`; build, -testing and convention material here. Keep them out of any single tool's -memory so they stay useful to whoever comes next. - -**Write down what was *not* true, not just what was.** A ruled-out -hypothesis is worth as much as the root cause, because it is the one the -next person will otherwise re-derive from scratch. `DEBUGGING.md` records, -for example, that a wedged unload is *not* the drain code and *not* a -refcount leak — both plausible, both wrong, both already checked. - -**Fix staleness in the same pass as the change.** A document describing code -you just altered is now wrong, and a confidently wrong document is worse -than none — it sends the next person down a path that no longer exists. When -you change behaviour, grep the docs for what described it and correct them -before you call the change done: - -```bash -grep -rn "" README.md CLAUDE.md deploy/ -``` - -Two real instances: the `KdBreakPoint` calls documented in -`deploy/README.md` were removed but the paragraph describing them survived; -and a memory entry carried a confidently-argued but incorrect root cause for -the statistics IOCTL (a minifilter theory, proposing an architectural device -split) that would have sent the next agent on a substantial and unnecessary -refactor. Delete or correct superseded conclusions outright rather than -leaving them to be weighed against the truth. - -**Prefer evidence to assertion.** Record the command and its output — the -`ln ` that identifies a breakpoint, the counter that read exactly 2x, -the `x BlorgFS!...` that proved a function never ran. A claim someone can -re-verify in one command survives; a bare conclusion does not. - -## Conventions - -These are opinionated and consistently applied. Match them exactly, and -correct drift when you find it. - -### Naming and layout - -- **Parameters are `PascalCase`; locals are `camelCase`.** This is the one - people get wrong most often, and it is load-bearing for readability: at any - line you can tell what came from the caller and what is yours. - `static NTSTATUS HttpParseHeaders(HTTP_CONTEXT* Ctx)` with - `SIZE_T bodyOffset = Ctx->BodyOffset;` inside. -- **Functions are `PascalCase` with a module prefix, and the prefix says - whether the name leaves the file.** A file-static helper takes the bare - module name (`ReadClaimStream`, `HttpFail`, `TlsHandshakeFail`). Anything - with external linkage takes `Blorg`, with no exceptions for layer or - ancestry -- `BlorgSendWskAsync`, `BlorgTlsSha256`, `BlorgFspDispatch`, - `BlorgVolumeRead`. The fastfat-inherited names (`FsdPostRequest`, - `PrePostIrp`, `OplockComplete`) were renamed along with everything else; - they are this driver's functions now, not the reference implementation's. - - So `grep -n '^[A-Za-z].*Blorg'` is the export list, and a bare module - name at file scope is a promise that it is `static`. -- **File-scope statics are `PascalCase` with the module prefix** - (`SocketMaxPoolSize`, `HttpActiveRequests`). Driver-wide mutable state - lives in the `global` struct rather than as loose globals — prefer adding - a field there to introducing a new one. -- **Struct fields are `PascalCase`; macros and constants are - `SCREAMING_SNAKE`** (`READ_AHEAD_GRANULARITY`, `SOCKET_CONNECT_TIMEOUT_MS`). -- **Empty parameter lists are `(VOID)`, never `()`.** -- **Allman braces**, including on `switch` cases, which get their own braced - block. Every `if` gets braces, even single-statement ones. -- **Casts go through `C_CAST`,** not bare C-style casts. -- **Guard clauses and early returns** over nesting — the happy path stays at - the lowest indentation level. - -### Rules +For benchmarking, use `-ForBenchmark` instead, which deploys Release, clears +Driver Verifier, and waits for the guest to go idle before reporting success. -- **Comments only at top-of-file, above a function, or beside a struct field.** - Never inside a function body — naming carries the meaning there. -- **Padding:** never widen a field's type to satisfy `CHECK_PADDING_END`; add an - explicit named `Reserved[N]` instead. -- **`ProbeForRead` for all user-buffer validation,** including output buffers. - `ProbeForWrite` is legacy and writes every page. +## Documentation for contributors and agents - Two places deliberately do not probe, both following fastfat: the cached - read path in `Read.c` hands `Irp->UserBuffer` to `CcCopyReadEx` under SEH, - and `Security.c` lets `SeQuerySecurityDescriptorInfo` write into - `Irp->UserBuffer` directly. In both the fault is caught rather than - prevented. Do not "fix" either without reading why fastfat does the same. -- **Never `%wZ`/`%Z` in a `DbgPrint` that can run above `PASSIVE_LEVEL`** — - the formatting touches paged code and bugchecks. In practice that means - anywhere on a completion chain (`HttpFail`, `HttpComplete`, - `Blorg*Complete`) or in anything one can call. It is fine, and used - freely, on the PASSIVE-only dispatch paths. -- IRQL is load-bearing throughout the async paths; the file-header comments in - `Client.c` and `Socket.h` state each path's contract. Read - them before changing anything on a completion chain. -- **Data-oriented design.** Lay data out for how it is actually traversed — - hot fields together, per-processor where contended, arrays over - pointer-chasing. Shape the structure around the access pattern rather than - around a conceptual object model. -- **Minimise syscalls and blocking calls.** Batch and amortise instead of - repeating a call per item; prefer an async completion over anything that - waits. A blocking call on a hot or completion path is a design bug, not a - detail — it occupies a worker for the whole duration. -- **Collapse branches with overlapping or shared predicates.** If two - conditions test the same thing, evaluate it once and branch once. Repeated - or nested tests of the same predicate should be merged rather than left - parallel — it is both faster and the only way the invariant stays obvious. +**[AGENTS.md](AGENTS.md)** is the detailed reference: coding conventions, +the required steps when changing behaviour, sanitizers, CI, the VM deploy +pipeline and its quirks, a debugging decision tree for the test VM, and the +performance-measurement methodology (with the findings behind the current +read-ahead tuning). It's written for coding agents but is equally the +reference for human contributors — read it before making non-trivial +changes. This includes tool-specific notes (e.g. Claude Code); there is no +separate per-tool file. diff --git a/deploy/DEBUGGING.md b/deploy/DEBUGGING.md deleted file mode 100644 index a999876..0000000 --- a/deploy/DEBUGGING.md +++ /dev/null @@ -1,397 +0,0 @@ -# Debugging this VM: what's real and what's noise - -This VM setup is flaky in ways that look alarming but usually aren't. This -document is a decision tree for telling the two apart, plus the specific -footguns that cost real time this session. Read this before treating -anything here as a BlorgFS bug — most "the VM is frozen/crashed" moments -this session were not. - -**But read the next section first.** One very common "frozen VM" *is* -BlorgFS, and mistaking it for VIX flakiness costs the most time of -anything here. - -## The guest freeze that IS BlorgFS: `KdBreakPoint()` in Debug builds - -**Largely fixed on 2026-08-22 — read this before blaming a Debug-build -freeze on the environment, and before re-adding a breakpoint to a rejection -path.** - -`KdBreakPoint()` is `#if DBG`-gated by the WDK headers: an `int 3` in Debug, -nothing in Release. On a debug-enabled guest that `int 3` traps, and **with -no debugger attached the guest simply stops dead** — indistinguishable from -the VIX/host-memory flakiness described below. `vmrun` calls hang, the -console is unresponsive, the VM looks wedged. The tell is that resuming from -a debugger un-freezes it, and it re-freezes the moment something touches -`B:` again. - -`Create.c` used to carry six unconditional `KdBreakPoint()` calls on its -access-mask and disposition rejection paths, so **any** process opening a -file on `B:` with an unsupported access mask froze the whole guest. That -fires constantly without you doing anything: Explorer, Defender (`WdFilter` -attaches to `\Device\BlorgDrive`) and `SearchHost` all probe files with -write masks, and anything registering a library or saving settings (Steam, -game launchers) opens for write as a matter of course. - -**Five of the six are now gone** — refusing a write on a read-only volume is -an expected outcome, not an anomaly worth trapping (see the comment above -`CheckFileAccess`). **Do not re-add them.** One deliberately remains, on the -terminal `STATUS_INVALID_DEVICE_REQUEST` fallthrough at the end of -`BlorgVolumeCreate`, which is a genuine "should not get here" -- the create -matched no case at all. Its own reason is recorded in that function's header -comment too, so grepping for `KdBreakPoint` and landing on the survivor -turns up the explanation without having to find this file first. - -**Confirm it in one command.** Attach KD, break in, and resolve the -instruction pointer: - -``` -ln -``` - -Landing anywhere inside the `BlorgFS` module — `BlorgFS!CheckFileAccess+0x46` -was the original signature — is a driver breakpoint, not a debugger -break-in artifact. Contrast with `nt!DbgBreakPointWithStatus`, which is the -normal forced break-in (see below). Check the module range with -`lm m Blorg*` if the symbol does not resolve. Since the fix this should only -resolve to the `BlorgCreate` fallthrough; anything else means a breakpoint -was reintroduced. - -**Use a Release build for any correctness or performance testing.** -`KdBreakPoint()` compiles to nothing there, and the statistics counters are -always-on in Release anyway (`README.md`), so nothing is lost. This is still -the right default even with the rejection-path breakpoints gone — a Debug -build also skews every performance number. Attach a debugger *before* -loading a Debug build so any remaining trap is diagnosable instead of fatal: - -```powershell -.\deploy\Deploy-ToVM.ps1 -VmxPath -GuestUser -GuestPassword

` - -VmPassword -Configuration Release -RemoteHost 10.0.50.17 -``` - -## First move: is the guest actually unwell, or is it just VIX? - -`vmrun`'s guest-automation channel (VIX) and the guest kernel are two -different things, and they fail independently. A `vmrun` call timing out or -returning *"The VMware Tools are not running in the virtual machine"* -tells you **nothing** about whether the guest is actually broken — check the -guest directly before believing it. - -**Fastest ground truth: attach a kernel debugger and look at two things.** - -``` -net:port=,key=,target= -``` - -(`vmrun getGuestIPAddress ` gets the IP; omitting `target=` only works -against a target actively announcing itself, e.g. right at boot — see -`deploy/README.md`.) - -1. **Does the register/stack state show a real bugcheck**, or the debugger's - own break-in artifact? Landing in `nt!DbgBreakPointWithStatus` (or, on - some boots, a different but equally unnamed low address — see below) - with a short 2-3 frame garbage stack is the **normal, expected** result - of forcing a break-in on an otherwise-idle or -running system. It is not - evidence of anything in BlorgFS. A **real** bugcheck shows an actual call - chain: `nt!KeBugCheckEx` → `nt!HalpNMIHalt` → `nt!HalBugCheckSystem` (this - session's specific signature — see the NMI section below). -2. **Is `System Uptime` advancing** across repeated break-ins a few seconds - apart? If yes, the kernel is alive and running normally — whatever - `vmrun` is complaining about is a host-side VIX problem, not a guest - problem. - -If both check out, the guest is fine. Stop chasing a "frozen VM" and go -fix whatever is actually making VIX unreliable (almost always host memory — -next section). - -## Driving the guest: what works and what silently does not - -Hard-won specifics for this VM. Each of these looked like a guest or driver -problem and was not. - -- **This `.vmx` is config-encrypted.** Every guest operation needs - `-vp ` before the command verb, or it fails with *"A password - is required for this operation"*. Bare `vmrun list` is the one exception, - which makes the VM look reachable when nothing else is. - `Deploy-ToVM.ps1` takes this as `-VmPassword`. -- **`cmd.exe` via `runProgramInGuest` hangs indefinitely; `powershell.exe` - works.** Reproducible even for `cmd.exe /c exit` with no redirection, - while `notepad.exe` and `powershell.exe -Command exit` return instantly - on the same guest. Each attempt strands a `cmd.exe` in the guest, so a - pile of them is a symptom of this and not of anything else. **Drive the - guest with `powershell.exe` only.** - - It does not always hang: it also returns **exit 1** on a completely - healthy guest, for both `cmd.exe /c exit 0` and `cmd.exe /c ver`, while - `powershell.exe -Command 'exit 0'` returns 0 in the same second. That - makes `cmd.exe` uniquely bad as a *health probe*, which is the one job it - looks perfect for: the probe reports every guest as dead, and any - recovery escalation behind it then fires against a guest that was fine. - On 2026-08-29 a sweep did exactly that, hard-resetting a working VM and - producing a black console that was then read as evidence of a driver - hang. The rule above already said not to do this. -- **`shutdown /r /t 0` never returns through `runProgramInGuest`.** - `runProgramInGuest` blocks until the guest program exits, and with `/t 0` - the OS tears the process down before it can exit, so vmrun waits forever - on a status that will never arrive. The guest reboots normally and sits - there idle while the host-side script hangs -- twelve minutes of apparent - "slow measurement" that was one stuck call. Use **`shutdown /r /t 5`**: - `shutdown.exe` schedules and returns, PowerShell exits, vmrun returns, - and the reboot fires afterwards. -- **Pass scripts as files, not as `-Command` strings.** Bash mangles `$_` - and `$($...)` before PowerShell ever sees them, which silently corrupts - the script — the usual symptom is an empty output file rather than an - error. Write the script locally, `copyFileFromHostToGuest`, then run it - with `-ExecutionPolicy Bypass -NoProfile -File`. -- **`runProgramInGuest` does not return the guest's stdout.** Redirect - inside the guest (`*> C:\...\out.txt`) and copy the file back. Do the - copy-back even when the run reports a non-zero exit — that is usually - where the actual error message is. -- **MSBuild from Bash mangles `/p:` switches into paths.** Run it from - PowerShell with `-p:` form instead. -- **`C:\BlorgFS-Deploy` in the guest accumulates files across sessions.** - There are results in there from days ago. Always write to a fresh - filename or delete first, or you will read a stale file back and believe - it is this run's output. - -## Footgun: orphaned `kd.exe` processes break the build and freeze the guest - -A `kd` session whose resume timed out (see below) leaves the **process -alive** even though the tooling has dropped the session id. Two things -follow, neither of which points at its cause: - -- It holds `x64\Debug\BlorgFS.pdb` open, so the next driver build dies with - `LNK1201: error writing to program database`. Every other project in the - solution still builds, which makes it look like a driver-specific code - problem. -- It is still attached to the target, so it can hold the guest halted — - looking exactly like the VIX hang above. - -Check for strays and clear the ones from your own session: - -```powershell -Get-Process kd -ErrorAction SilentlyContinue | Select-Object Id,StartTime -``` - -Match `StartTime` against when you opened sessions before killing anything; -a `kd` predating your session may be someone else's live debugger. - -## Footgun: a kernel debugger attached during a bugcheck LOOKS like a frozen VM - -`AutoReboot` is configured on this guest, but a bugcheck that happens while -a KD session is connected **halts and waits for the debugger** instead of -auto-rebooting — completely normal WinDbg behavior, but from outside (or on -the VM's console) it is indistinguishable from a genuinely hung VM: the -screen just sits there, unresponsive, forever. - -**If a hang coincides with a debugger session being open, close that session -with `resume: true` (or send `g`) before concluding the VM itself is stuck.** -This alone explained more than one "it's frozen" moment this session. - -## Footgun: `g` (resume) reports "Request timed out" and kills the session - -This is expected, not a failure — the resume itself works, but the tool call -wrapping it doesn't get a response until the *next* break, so it times out -and the session id stops being valid. **Open a fresh `open_kd_session` to -check state after resuming** rather than reusing the old session id or -retrying `g` on it. - -## Footgun: an unresolved break address isn't automatically suspicious - -Not every break-in lands in `nt!DbgBreakPointWithStatus` by name — on one -boot this session, forced break-ins repeatedly landed at an address with no -resolvable symbol at all (`lm` showed nothing there), at both ~1 minute and -~31 minutes of uptime, with an identical kernel base both times. That -pattern (same address, same kernel base, across what should be a fresh -KASLR slide) points at the debugger's own break-in landing in a stable -idle-loop location for that particular boot, not a driver problem — confirm -via the same uptime-advancing check above rather than assuming an unnamed -address means something is wrong. - -## Root-caused: service stop wedges in `STOP_PENDING` (no dismount handler) - -`sc stop BlorgFS` leaves the service in `STOP_PENDING` indefinitely, after -which `sc start` fails `1056 (already running)` and a reinstall fails -*"The specified service has been marked for deletion"*. Only a guest reboot -clears it. Reproduces against a healthy, reachable backend after normal -successful use, so it is not the dead-backend socket-timeout case above. - -**It is not a reference-count leak — every relevant count is already zero.** -Live KD state while wedged: - -``` -!drvobj \Driver\BlorgFS 2 -> DriverUnload: -!devobj -> RefCount 0 - ExtensionFlags (0x1) DOE_UNLOAD_PENDING - AttachedDevice (Upper) ... \FileSystem\FltMgr -!devobj -> RefCount 0, DOE_UNLOAD_PENDING -dt nt!_VPB -> Flags 1 (VPB_MOUNTED), ReferenceCount 0 -x BlorgFS!HttpActiveRequests -> 0n1 (standing reference, never released) -``` - -Both drain gates still reading their initial standing reference of 1 proves -`DriverUnload` **was never entered** — each drain releases that reference as -its first action. That rules out the two unbounded `KeWaitForSingleObject` -drains in `Client.c`, which is the intuitive suspect and the -wrong ones. - -The actual chain: - -1. `FsCtrlMountVolume` (`FsCtrl.c`) self-mounts: it creates the volume device - object and sets `VPB_MOUNTED`. -2. Filter Manager attaches minifilters to the now-mounted volume — `fltmc - instances` shows `WdFilter`, `UCPD`, `applockerfltr`, `bfs` and - `FileInfo` on `\Device\BlorgDrive`. -3. On stop, `IopUnloadDriver` walks the driver's device objects. The volume - device has `AttachedDevice != NULL` (FltMgr sitting above it), so it - cannot be deleted: the I/O manager sets `DOE_UNLOAD_PENDING` and - **defers `DriverUnload`**. -4. **`IRP_MN_DISMOUNT_VOLUME` has no handler.** `FsCtrl.c` cases - `IRP_MN_USER_FS_REQUEST` and `IRP_MN_MOUNT_VOLUME`; dismount falls into - `default:` and returns `STATUS_INVALID_DEVICE_REQUEST`. Nothing else will - initiate a dismount either, because the driver mounted itself rather than - being mounted by a storage stack. -5. The volume therefore never dismounts, FltMgr never detaches, the deferred - delete never completes, and `DriverUnload` never runs. - -Fix direction (not yet implemented): give the volume a real dismount path — -handle `IRP_MN_DISMOUNT_VOLUME`, tear down the FCB/DCB tree, clear -`VPB_MOUNTED`, and let FltMgr detach. Until then, **a guest -reboot between driver deploys is mandatory**, which `Deploy-ToVM.ps1` does -not currently do on its own. - -## Known environmental issue: NMI_HARDWARE_FAILURE (bugcheck 0x80) - -Confirmed pre-existing and unrelated to BlorgFS (occurs even with the driver -not loaded), and now confirmed to correlate with **host memory pressure** -(see below) and heavy guest load. Triage: - -``` -tools\Get-CrashVerdict.ps1 -DumpPath -SymbolServer -``` - -If the bucket matches `0x80_4F4454_AuthenticAMD_NOERRREC_IMAGE_AuthenticAMD.sys` -with `DriverOnStack: 0`, this is the known false alarm — `AuthenticAMD.sys` -is WinDbg's generic fallback name when it has no real driver to blame, and -`Arg1` decodes to the ASCII tag `'TDO'`, consistent with a -hypervisor/watchdog-injected NMI on an apparently-unresponsive guest rather -than a genuine fault. Full detail in `deploy/README.md`. - -## Host memory pressure breaks VIX reliability, not just VM performance - -**Below roughly 3-4GB of host free RAM, expect `vmrun` guest-automation -calls to fail or time out frequently — copies, `runProgramInGuest`, even -`checkToolsState` — while the guest kernel itself remains completely -healthy.** This was directly observed and measured this session: - -| Host free RAM | VIX behavior | -|---|---| -| 1.24 GB | Guest looked completely unresponsive; every call failed | -| 2.6 GB | Frequent failures, needed 2-3 retries per call | -| 3.2 GB | Frequent timeouts on trivial commands (`sc query`) | -| 8.1 GB | Fully reliable, no retries needed | - -Check host memory before assuming a guest problem: - -```powershell -Get-CimInstance Win32_OperatingSystem | - Select-Object @{n='FreeGB';e={[math]::Round($_.FreePhysicalMemory/1MB,2)}} -``` - -`vmware-vmx` itself is worth checking too — it has been observed using -noticeably more working set than the VM's own configured `memsize` (e.g. -6.13GB against a 4GB `memsize`), plausibly `mem.hotadd` overhead; disabling -that (requires a VM power-off to edit the `.vmx`) is an untried but -plausible lever if this keeps recurring. - -**Don't burn cycles on rapid VIX retries under memory pressure** — it just -adds more contention. Either wait for more host RAM to free up, or fall back -to the KD-based ground-truth check above, which doesn't depend on VIX at -all. - -## Footgun: a PowerShell-in-guest crash isn't necessarily your script's fault - -`runProgramInGuest` occasionally reports exit code `-196608` on completely -trivial scripts (a bare `sc.exe query` + `Test-Path`) under host memory -pressure. This looks like a script bug but reproduces on scripts with no -possible bug — treat it as another host-pressure symptom, not something to -debug in the script itself. - -## "The semaphore timeout period has expired" on `B:\` is the backend, not the FS - -Enumerating `B:\` failing with *"The semaphore timeout period has expired"* -(an `IOException` from `Get-ChildItem`) means the driver loaded and mounted -fine and is simply not reaching its HTTP backend. - -The INF-seeded default `RemoteHost` is `blorgfs.blorg.lan`, which **resolves -in this environment to `10.0.60.10` and is not reachable on port 8080**. The -working backend is **`10.0.50.17:8080`**. Deploy with `-RemoteHost -10.0.50.17` (or fix `HKLM:\SYSTEM\CurrentControlSet\Services\BlorgFS\Parameters\RemoteHost` -and restart the service). - -Confirm which side is at fault from inside the guest before touching driver -code: - -```powershell -Resolve-DnsName blorgfs.blorg.lan -Test-NetConnection -ComputerName 10.0.50.17 -Port 8080 -``` - -Note that a driver pointed at a dead backend also makes service **stop** -pathological: every in-flight request has to burn its full socket timeout -(connect 15s / send 15s / receive 30s, `Socket.c`) before unload can -proceed, so `STOP_PENDING` can persist for minutes and look like a hang. - -## Solved: the vendor IOCTLs returned `ERROR_INVALID_FUNCTION` (device type) - -`IOCTL_BLORGFS_QUERY_STATISTICS` / `RESET_STATISTICS` / `SET_TLS_PIN` all -failed with `ERROR_INVALID_FUNCTION` (error 1) from usermode — the -long-standing "statistics IOCTL broken" issue. **Root cause: they were -declared `CTL_CODE(FILE_DEVICE_FILE_SYSTEM, ...)`.** - -The I/O manager routes an IOCTL by the device type baked into its -`CTL_CODE`: `FILE_DEVICE_FILE_SYSTEM` becomes `IRP_MJ_FILE_SYSTEM_CONTROL` -(an FSCTL), everything else becomes `IRP_MJ_DEVICE_CONTROL`. All three are -implemented in `DevIoCtrl.c` under `IRP_MJ_DEVICE_CONTROL`, so they never -arrived; they fell into `FsCtrl.c`'s unhandled-FSCTL `default`, which -returns `STATUS_INVALID_DEVICE_REQUEST` → `ERROR_INVALID_FUNCTION`. Fixed -by declaring them `FILE_DEVICE_UNKNOWN`. - -Two traps worth knowing if this ever regresses: - -- **The sandbox tests cannot catch it.** They call `BlorgDeviceControl` - directly, so the routing decision never happens and every test passes - with the broken device type. `DevIoCtrlTest.VendorIoctlsAreNotRoutedAsFsctls` - asserts on the device type specifically for this reason. -- **`PerfHarness.exe` embeds the IOCTL codes at compile time.** After - changing them you must rebuild it, or a stale binary keeps sending the old - codes and still reports error 1 against a fixed driver. Note also that a - **Debug** `PerfHarness.exe` will not start in the guest (missing debug CRT - DLLs, exit code `-1073741515`) — deploy the Release one. - -## Technique: diagnosing "my IOCTL returns the wrong status" via live KD - -When a custom vendor IOCTL fails with a status that doesn't match anything -in the driver's own dispatch code, the fastest way to find out whether the -IRP is even reaching your dispatch routine is to inspect the live device -object stack, not just read source: - -``` -!devobj \BlorgFS -``` - -`AttachedDevice (Upper) ... \FileSystem\FltMgr` in the output means a -minifilter stack sits above your device and gets first look at every IRP — -relevant for any device created as `FILE_DEVICE_DISK_FILE_SYSTEM` and passed -to `IoRegisterFileSystem`, since Filter Manager treats it as a real -filesystem eligible for minifilter attachment regardless of whether the -driver intended it purely as a control device. Cross-reference with, from -inside the guest: - -```powershell -fltmc filters # which minifilters are loaded at all -fltmc instances # which volumes/devices each one is actually attached to -``` - -Note: `Set-MpPreference -DisableRealtimeMonitoring $true` can silently no-op -under Tamper Protection even from an admin token — don't trust it without -checking `Get-MpComputerStatus` afterward. diff --git a/deploy/README.md b/deploy/README.md deleted file mode 100644 index 71861cc..0000000 --- a/deploy/README.md +++ /dev/null @@ -1,279 +0,0 @@ -# Deploying and testing BlorgFS in a VM - -BlorgFS is a kernel-mode filesystem driver, so it is developed against a -throwaway Windows VM rather than the build machine. This document covers the -deploy pipeline, the VM/debugger plumbing around it, and the quirks that cost -real time to rediscover. It is tool-agnostic — nothing here assumes any -particular editor, agent, or IDE. - -**If something looks broken while testing — a frozen VM, a mysterious -timeout, a bugcheck — read [`DEBUGGING.md`](DEBUGGING.md) first.** Most -alarming-looking failures in this environment are known, diagnosable, and -not BlorgFS bugs; that document is a decision tree for telling the -difference before spending time chasing the wrong thing. - -## The short version - -Copy `deploy/blorgfs.env.example` to `deploy/blorgfs.env`, fill it in once, -and then deploying takes no arguments at all: - -```powershell -.\deploy\Deploy-ToVM.ps1 -Configuration Release -``` - -`blorgfs.env` holds the VM path, the guest account, the `.vmx` -config-encryption password, the snapshot to revert to, the backend address, -and the KDNET key. It is **gitignored** -- every one of those is a -credential for the debug VM, and the KDNET key in particular lets anyone on -the network attach a kernel debugger to that guest. The committed -`blorgfs.env.example` is the template and carries no real values. - -The point of it is that a session starting from nothing can deploy and debug -without first being handed five settings by hand. Explicit arguments still -win over the file, so nothing is locked in: - -```powershell -.\deploy\Deploy-ToVM.ps1 ` - -VmxPath "C:\path\to\Win11.vmx" ` - -GuestUser -GuestPassword ` - -SnapshotName ` - -Configuration Debug -``` - -That builds, copies the driver package into the guest, and runs -`Install-BlorgFS.ps1` there. Reverting to a known-good snapshot first is -strongly recommended — a buggy driver load can bugcheck the guest, and a -half-installed package is worse to debug than a clean one. - -**Do not deploy by copying `BlorgFS.sys` into `System32\drivers` by hand.** -It skips catalog validation entirely and can silently leave a stale binary in -place from a previous iteration, so the thing you are debugging is not the -thing you just built. Symptoms of that mistake look like driver bugs -(unexplained hangs, behaviour that does not match the source) and cost far -more time than the deploy script does. - -## How the install actually works - -`BlorgFS.inf` is a Windows 10 1903+ **primitive driver** INF: the driver -creates its own device objects in `DriverEntry` rather than being enumerated -by a bus, so there is no hardware ID, no `[Manufacturer]`/`[Models]` section, -and it never appears in Device Manager. Installing it correctly is a two-step -sequence, and **neither step alone is sufficient**: - -```powershell -# 1. Stage the package into the Driver Store -pnputil.exe /add-driver /install - -# 2. Run the section that does CopyFiles + AddService -rundll32.exe setupapi.dll,InstallHinfSection DefaultInstall.NTamd64 132 -``` - -Why both: - -- `BlorgFS.inf` sets `DestinationDirs = 13`, meaning **run from Driver - Store** (driver package isolation). `InstallHinfSection`'s plain - `CopyFiles` engine cannot write into the protected Driver Store on its - own. Run without staging first, it fails the copy — interactively as a - *"Setup cannot copy the file BlorgFS.sys"* dialog that misleadingly blames - the **source** path (the source is fine; the destination model is the - problem), and non-interactively as a **silent no-op**: exit code 0, and - nothing written to `setupapi.dev.log`. -- `pnputil /install` alone stages the package but never runs - `[DefaultInstall.NTamd64.Services]`. Its `/install` flag installs against - *matching devices*, and a primitive driver has no hardware ID to match, so - it reports success, stages the files, and leaves the service unregistered. - -Other things that bite here: - -- **The section is `DefaultInstall.NTamd64`, not `DefaultInstall`.** The INF - only defines the architecture-decorated variant. Naming the undecorated one - matches nothing and no-ops silently. -- **Deploy the *staged* INF, not the repo-root one.** MSBuild's `stampinf` - step fills in `DriverVer` on the copy at - `x64\\BlorgFS\BlorgFS.inf` *before* `Inf2Cat` hashes it. The - repo-root `BlorgFS.inf` deliberately leaves `DriverVer` blank, so its bytes - — and therefore its hash — differ from what the catalog contains. Deploying - it fails with *"The hash for the file is not present in the specified - catalog file. The file is likely corrupt or the victim of tampering."* - That message sounds like binary corruption or a signing failure; it - actually just means the wrong copy of the INF was deployed. -- **The registered service name is `BlorgFS`.** Not `BLORG`, not anything - from an ad-hoc `sc create`. Verify with `sc query BlorgFS`; a `1060` from - `sc query` means the `AddService` step never ran. -- Test-signing must be on (`bcdedit /set testsigning on`) and the driver's - test certificate trusted, both of which `Install-BlorgFS.ps1` handles. - Enabling test-signing requires a **reboot** before any test-signed driver - will load; the script exits with status 2 to signal exactly that, and - `Deploy-ToVM.ps1` reboots and retries once automatically. - -Confirm a good install: - -```powershell -sc query BlorgFS # STATE : 4 RUNNING -Test-Path B:\ # True -- the driver self-mounts, there is no mount step -``` - -## VMware / `vmrun` quirks - -- The `.vmx` used for this project is **encrypted at the VM-config level**, - which is separate from any guest OS login. Every `vmrun` invocation against - it needs `-vp ` before the command verb — including read-only - commands like `list`. Without it: *"A password is required for this - operation."* -- `vmrun`'s argument order is rigid: - `vmrun [flags] COMMAND [command-args...]`. The vmx path goes - **immediately after the command name**, not at the end. Getting this wrong - produces confusing errors like *"Cannot open VM: C:\some\other\path, - unknown file suffix"* — it is trying to interpret your argument as the vmx. -- Guest credentials go in `-gu` / `-gp`, and VMware Tools must be running in - the guest for any guest command to work. `vmrun checkToolsState ` - is the quick health check; it can transiently report tools as down and - recover on its own. -- `runProgramInGuest` defaults to a **non-interactive, Session-0** context. - Anything that needs the interactive desktop (notably classic setup UI) will - silently do nothing there. Pass `-interactive` to run on the console - session. The two-step install above avoids needing this, but it is - essential for *seeing* a setup dialog when diagnosing why an install - failed. -- `runProgramInGuest` does not surface the guest process's exit code. - `Install-BlorgFS.ps1` works around this by writing its status to a file - that `Deploy-ToVM.ps1` copies back and reads — and deletes any stale status - file first, so a crashed run cannot be misread as the previous run's - success. - -## Avoiding manual guest login - -Windows leaves the guest at the lock/login screen after any reboot -(including the auto-reboot after a bugcheck), and pre-login the network -profile tends to sit as "Public" — which blocks ICMP by default, so `ping` -and some guest-automation calls look like the guest is unreachable when it is -actually just sitting at the lock screen waiting for a human. Set Windows -auto-logon once per VM image so this stops being a recurring interruption: - -```powershell -$k = "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" -Set-ItemProperty $k AutoAdminLogon "1" -Set-ItemProperty $k DefaultUserName "" -Set-ItemProperty $k DefaultPassword "" -Set-ItemProperty $k DefaultDomainName $env:COMPUTERNAME -Set-ItemProperty $k ForceAutoLogon "1" # re-applies after a manual lock, not just cold boot -``` - -Also worth disabling for any long-running test session, so an idle timeout -does not re-lock the session mid-run: - -```powershell -powercfg /change monitor-timeout-ac 0 -powercfg /change standby-timeout-ac 0 -powercfg /change hibernate-timeout-ac 0 -Set-ItemProperty "HKCU:\Control Panel\Desktop" ScreenSaveActive "0" -``` - -This needs to be set once per golden snapshot — a `revertToSnapshot` to an -older snapshot taken before these were set will need it redone. - -## A known false alarm: NMI_HARDWARE_FAILURE (bugcheck 0x80) - -This VMware/AMD-virtualization setup occasionally bugchecks with -`0x00000080 NMI_HARDWARE_FAILURE`, unrelated to BlorgFS -- confirmed to -happen even with the driver not loaded. Two independent occurrences -triaged with `tools\Get-CrashVerdict.ps1 -SymbolServer` both landed on the -identical bucket: - -``` -0x80_4F4454_AuthenticAMD_NOERRREC_IMAGE_AuthenticAMD.sys -DriverOnStack: 0 -``` - -`Arg1` decodes to the ASCII tag `'TDO'` and the captured stack is just -`nt!KeBugCheckEx` with nothing underneath -- a minidump limitation, not a -real call chain. `AuthenticAMD.sys` is WinDbg's generic fallback name when it -has no real driver to blame, not an actual faulting module. This pattern -matches a hypervisor/watchdog-injected NMI forcing a crash dump on a guest -that looked unresponsive, not a genuine hardware or driver fault. - -**Before treating a `0x80` bugcheck as a BlorgFS finding**, triage it and -check `DriverOnStack` — if it's `0` and the bucket matches the one above, -this is the known false alarm, not a new bug. - -**A kernel debugger left attached across this bugcheck makes the guest look -frozen instead of rebooting.** `AutoReboot` is configured, but a bugcheck -that happens while KDNET is connected halts and waits for the debugger -instead of auto-rebooting — normal WinDbg behaviour, but from outside it is -indistinguishable from a genuinely hung VM: the screen just sits there. -If a hang coincides with a debugger session being open, close that session -with `resume: true` (or send `g`) before assuming the VM itself is stuck. - -Heavier allocation load (e.g. a broad recursive directory enumeration) -seems to correlate with triggering this — plausibly because **Driver -Verifier's special pool** turns every allocation into extra TLB-flush IPI -traffic, which is exactly the kind of load that would expose a marginal -hypervisor-level IPI-ack timing issue. Consider disabling Verifier -(`verifier /reset`, then reboot) for routine correctness/perf runs and -reserving it for targeted memory-safety sessions, if this keeps recurring -under load. - -## Kernel debugging (KDNET) - -The guest boots with `debug Yes` and a KDNET transport. Check its settings -from inside the guest with `bcdedit /dbgsettings` (key, port, host IP). - -Connecting to an **already-running** target requires an explicit `target=` -with the guest's IP: - -``` -net:port=50000,key=,target= -``` - -`vmrun getGuestIPAddress ` gets the IP. Omitting `target=` only works -against a target actively announcing itself (e.g. right at boot); against a -running guest it simply times out. - -Two behaviours that look alarming and are not: - -- **Landing at `nt!DbgBreakPointWithStatus` with a short, garbage-looking - 2–3 frame stack immediately on connect is normal.** That is the debugger's - own forced break-in interrupting whatever the CPU was doing — usually the - idle thread — not a breakpoint in BlorgFS. To tell a live kernel from a - frozen one, break in repeatedly and check that `System Uptime` is - advancing. -- The resume command (`g`) frequently reports a timeout and invalidates the - session, even though the resume itself worked. Expect to open a fresh - debugger session to inspect state afterwards rather than reusing the old - session id. - -`KdBreakPoint()` is `#if DBG`-gated by the WDK headers: it compiles to -`__debugbreak()` (an `int 3`) in **Debug** builds and to **nothing** in -Release. In a Debug build on a debug-enabled guest, hitting one traps into -the debugger; with kernel debugging enabled but no debugger attached, it -will appear to freeze the guest. This is a real difference in -Debug-vs-Release behaviour to keep in mind when a Debug build appears to -hang where a Release build does not — attach a debugger before loading, and -the trap becomes diagnosable instead of fatal. - -`Create.c` used to carry several such calls on its access-mask and -disposition rejection paths, which froze the guest whenever anything opened -a file on `B:` for write. Those were removed on 2026-08-22; only the -terminal `BlorgCreate` fallthrough still traps. See the `KdBreakPoint` -section in [`DEBUGGING.md`](DEBUGGING.md) before reintroducing one. - -## Testing tiers - -Run from the repo root on the build machine: - -```powershell -powershell -File tools/Invoke-BlorgChecks.ps1 -Tier Fast -``` - -| Tier | What it does | Needs | -|---|---|---| -| `Build` | Compile + link everything with PREfast | nothing | -| `Fast` (default) | Build, plus RFC 8448 crypto vectors and the fuzz corpus | nothing | -| `Perf` | Fast, plus PerfHarness workloads compared to a stored baseline | driver loaded, backend reachable | -| `All` | every tier | as above | - -Exit code is 0 only if everything in the tier passed. - -The `Perf` tier must run **inside the guest**, where B: is mounted and the -HTTP backend is reachable. See the performance section in `CLAUDE.md` for -what the counters mean and how baselines are updated. diff --git a/tools/Measure-BlorgScaling.ps1 b/tools/Measure-BlorgScaling.ps1 index 9cdb856..bc05aa1 100644 --- a/tools/Measure-BlorgScaling.ps1 +++ b/tools/Measure-BlorgScaling.ps1 @@ -106,7 +106,7 @@ function Invoke-GuestPowerShell([string]$Command) { # # `sc stop` cannot provide that reset: it wedges in STOP_PENDING because # nothing dismounts a self-mounted volume, and the service then cannot be -# started again without a reboot (see deploy/DEBUGGING.md). Rebooting is +# started again without a reboot (see AGENTS.md, "Debugging the VM"). Rebooting is # slower but it is the only reset that actually works, and a sweep that # silently measured a wedged driver would be worse than a slow one. #