Repository navigation
ci: mounted conformance batteries fail intermittently on the git checks #88
Description
Activity
One more occurrence, with the new diagnostics, from #94's first CI run
(job):when branch job failing check 2026-09-04 18:24 perf/fuse-mount-cachingWinFsp, write-back variant rename directory keeps contentsNotes that may be useful:
- Not a git check this time. The check is
Rename-Item "$M\d\sub" 'sub2'; Test-Path "$M\d\sub2\deeper"— the same
shape as therename fileflakes that fix(agent,client): discard invalidations that predate the client's own mutation (phantom entry after mv) #82 addressed (a rename immediately
followed by a read of the result), but on a directory. - Everything else in the battery passed: 30 passed, 1 failed. The other five CI
jobs on that run were green, including the Linux mounted battery and the
non-write-back Windows battery in the same job. - The diagnostics dump was not informative for this check — it prints the
.git
listings, which were identical between the mount and the backing store, and
the mount root listing. Neither says anything aboutd\sub2. Worth widening
the dump to list the parent of whatever path the failing check touched, not
just.git. - Re-running the failed job passed (2m46s, all checks), so it is a flake and
not a regression from that PR. - Not reproducible locally: the write-back battery passes 31/0 on the same
commit, and the pair of batteries passed repeatedly during that PR's
development.
Since it is a rename-then-read, it has the same smell as the class #82 fixed:
the mutation updates the mirror optimistically and the watcher's event arrives
afterwards, so a read landing in between can see either. #94 touches two things
in that window and it is worth watching whether the rate changes — it fixes
ghost entries left behind when a mutation resolved a path differently from the
mount (#70) and a stale-bytes race after a write (#71) — but neither is an
obvious explanation for this particular check, so I would not call it addressed.- Not a git check this time. The check is
This now reproduces on demand, and it is not new
Following the occurrence I logged above, I suspected #94 had caused it: since
that merged,rename directory keeps contentsfailed 5 of 7 WinFsp battery
runs, against the ~20% this issue recorded. So I measured the baseline directly
— #97 was a throwaway PR pinned at fd1b547, the commit main was at before
#94 — and it fails the same check:FAIL rename directory keeps contents -- condition was false passed: 27 failed: 1So #94 is exonerated, and the useful finding is that the rate has gone up a
lot on the current runner images (that job now reports VS 18 / MSVC 14.51),
independently of our code. That makes it reproducible enough to actually chase,
which it was not when this issue was filed.The family
Every occurrence has the same shape — a mutation immediately followed by a check
that reads the result, "condition was false", and the whole rest of the battery
passing:when check 2026-08-29 20:52 rename file(addressed by #82)2026-08-29 21:05 delete file2026-09-04, repeatedly rename directory keeps contentsThe current one is
Rename-Item "$M\d\sub" 'sub2'; Test-Path "$M\d\sub2\deeper"
— a directory rename followed at once by a lookup of a grandchild. #82 fixed the
rename fileinstance by discarding invalidations that predate the client's own
mutation; this looks like the same race surviving for the directory case, where
the mirror update has to move a whole subtree rather than one node.What I ruled out
- Not the page-cache punch from perf: let the kernel cache file pages; fix read-cache resolution, rescan storms and a stuck-peer stall #94. I gated it off on WinFsp and CI failed
anyway, with the same check. - Not reproducible locally at all: 0 failures in 150 iterations of exactly
that rename against a live mount, 0 in 6 full write-back batteries, and 0 in 5
more with every one of 20 cores pinned at 100%. It needs the runners. - Not write-back specific — I assumed that early and it was wrong. It has
now failed in both the write-through and write-back steps.
Two concrete next steps
- Widen the diagnostic dump. It prints the
.gitlistings, which say
nothing about a non-git check. Havecheck()dump the parent directory of
whatever path the failing check touched, from both the mount and the backing
store, plus the client's current mirror for that path. Without that, each
occurrence still only tells us the assertion was false. - Then chase the directory-rename mirror update against an in-flight
invalidation, which is where the evidence points.
Since it reproduces at a high rate on current runners, a fix attempt can be
validated by re-running the WinFsp job a handful of times, which was not
practical before.- Not the page-cache punch from perf: let the kernel cache file pages; fix read-cache resolution, rescan storms and a stuck-peer stall #94. I gated it off on WinFsp and CI failed
Reopening: the negative-cache fix reduced this a lot, but did not eliminate it
It recurred on #104 — a branch whose only change is
scripts/install.ps1, based
on main with #98 already in it. Same check, same signature as before:FAIL rename directory keeps contents -- condition was false mount [d]: b.txt sub2 backing [d]: b.txt sub2 mount [d\sub2]: deeper backing [d\sub2]: deeper mount [d\sub]: <absent> backing [d\sub]: <absent>Both views correct at dump time; only the per-name lookup disagreed. So the
mechanism is still a stale cached name, andnegative_timeout = 0addressed
some of it but not all.Rates, as honestly as I can put them
WinFsp battery runs failures before #98 7 5 immediately after (deliberate resampling) 10 0 since, across #100 / #101 / #102 / #103 / #104 ~6 1 The before-figure is biased: I was re-running failed jobs preferentially while
hunting it. Treat it as "often" rather than as a measured 70%. The after-figure
is a genuine drop, not a fix.What that leaves
The remaining suspect is the positive entry cache.
entry_timeoutis still
1.0, so a name resolved before the rename can stay cached across it. I measured
zeroing that as far too expensive — stat went 14 ms to ~200 ms — but that
measurement was on Linux/libfuse, and this only ever fails on WinFsp, whose
caching is its own implementation and whose cost I never measured. So
entry_timeout = 0on Windows only is worth measuring before it is dismissed:
Direction A metadata is served from the in-RAM mirror there too.Also worth checking, and cheaper: whether WinFsp-FUSE honours
cfg->negative_timeoutat all. If it does not, #98's improvement on Windows was
luck and the whole effect belongs to something else — 10 consecutive passes
argue against that, but they are not proof.Practical impact
Low for users, high for CI. The mount converges within about a second, so a
real tool retries and moves on; the battery reads back immediately and does not.
It cost several job re-runs across today's merges.Correction: WinFsp does not appear to honour
negative_timeoutI claimed in #98 that stale negative caching was the root cause. That is well
supported on the mechanism side (both views correct, per-name lookup
disagreeing with an enumeration) but I can no longer support it as the cause
on Windows, and the honest thing is to say so.I built a probe that does what the mechanism needs: look a name up while it is
absent, create it, then time how long the mount keeps saying it is missing.build visible after (40 iterations) over 500 ms negative_timeout = 0(current main)min 6, median 14, max 24 ms 0/40 negative_timeout = 1.0(pre-#98)min 13, median 15, max 31 ms 0/40 Identical. If the setting were doing what I assumed, the second row should
have shown roughly a one second stall. So either WinFsp-FUSE ignores
cfg->negative_timeout, or something else invalidates the entry first.Two consequences:
- The ten consecutive green runs after fix(mount): stop caching negative lookups — the root cause of the #88 conformance flake #98 may simply have been luck. fix(mount): stop caching negative lookups — the root cause of the #88 conformance flake #98 also
contained the rename-fallback hardening, and fix(mount): punch the page cache only for the ops that were applied #96 before it changed which
invalidations reach the page-cache punch; either could own whatever
improvement is real. I over-attributed. negative_timeout = 0still stays. It is free on Linux (measured: walk,
stat and read all unchanged) and inert on Windows, so there is nothing to
undo — it is just not the fix I said it was.
Where this leaves it
The probe does not model the failing case closely enough either: it creates the
file on the backing store, so the watcher drives the update, whereas the
battery renames through the mount, which updates the mirror synchronously.
A stress loop doing exactly the latter still gives 0 failures in 150 iterations
locally, and 0 in 6 full batteries, and 0 in 5 more with all 20 cores pinned.So this remains only reproducible on the runners, and the next honest step is
CI-side experimentation rather than another local theory: tryentry_timeout = 0
on Windows only and sample the WinFsp job several times. That cost was measured
at ~13x on stat, but on Linux/libfuse — the Windows cost is unmeasured, and
Direction A serves metadata from the in-RAM mirror there too.- The ten consecutive green runs after fix(mount): stop caching negative lookups — the root cause of the #88 conformance flake #98 may simply have been luck. fix(mount): stop caching negative lookups — the root cause of the #88 conformance flake #98 also
The diagnostics finally separated the two cases
A failure on #108 — a documentation-only PR — produced a dump unlike every
previous one:FAIL rename directory keeps contents -- condition was false mount [d]: b.txt backing [d]: b.txt sub2 ^^ mount and backing store disagree here mount [d\sub2]: deeper backing [d\sub2]: deeper mount [d\sub]: <absent> backing [d\sub]: <absent>Every earlier dump had both views agreeing. This one does not: the mount's
directory listing ofdis missingsub2, while the backing store has it —
and yetd\sub2resolves directly and listsdeepercorrectly.That is a stale cached directory enumeration of
d, not a lost subtree and
not a stale negative lookup.Test-Path "$M\d\sub2\deeper"resolvessub2
throughd's cached listing, which predates the rename, so it fails — while a
direct listing ofd\sub2a moment later succeeds because that entry is fine.Which means the cause is the positive entry cache, and the fix is targeted
entry_timeoutis 1.0 s, so WinFsp may served's contents from cache for up
to a second after the rename. The rename went through the mount, so nothing
told the kernel thatd's listing had changed:- the page-cache punch thread only consumes remote invalidations, the ones the
agent's watcher pushes; - a mutation the mount performs itself (
op_rename,op_create,op_unlink,
op_rmdir) updates the mirror synchronously and punches nothing.
So the fix is not to zero
entry_timeout— measured at ~13x on stat, and that
was on Linux where it would also cost most of the read win. It is to punch the
parent directory after a mutation the mount performed: a rename changes the
contents of both parents, a create/unlink/rmdir changes one. That is a handful
offuse_invalidate_pathcalls on paths we already have in hand, on the
mutation path only, and it leaves the caches doing their job everywhere else.Worth noting this also explains why it was never reproducible locally: it needs
the read to land inside the window between the mutation and the cache expiring,
which the runners hit and this machine does not.Status
negative_timeout = 0from #98 stays — free on Linux, and the earlier probe
showed WinFsp behaves the same either way, so it is simply not the lever I
thought it was. The parent-directory punch is the next thing to try, and unlike
the previous two attempts it is now supported by a dump that shows exactly which
cache is stale.- the page-cache punch thread only consumes remote invalidations, the ones the
The parent-directory punch does not fix it either
Measured, not assumed. #111 punches the parent directory after every mutation
the mount performs. Sampling the WinFsp conformance job on that branch:result with the parent punch (#111) 3 pass / 2 fail baseline rate roughly 1 in 2 Unchanged. That is my third wrong theory here, after the page-cache punch and
the negative-lookup cache, so I am recording what the evidence supports and
stopping rather than trying a fourth.What the failures now look like
Both failures on that branch show the converged dump — mount and backing store
already agreeing by the time the dump runs:FAIL rename directory keeps contents -- condition was false mount [d]: b.txt sub2 backing [d]: b.txt sub2 mount [d\sub2]: deeper backing [d\sub2]: deeperWhich is the same shape as most earlier failures. The one dump that showed
d
genuinely stale (missingsub2) remains a single observation, so I over-read it:
it may have been the same transient caught a moment earlier, not a distinct mode.The hypothesis it does suggest, untested
If the punch is the right idea, the flaw may be that it is asynchronous.
op_renamequeues the paths and returns; the invalidation is applied by a
separate thread. Windows can therefore do its next lookup before the punch has
landed, which would leave the behaviour unchanged - exactly what was measured.Making it synchronous means calling
fuse_invalidate_pathinline from inside the
operation handler, which is the classic libfuse deadlock hazard (the kernel can
be waiting on the very handler that is trying to invalidate). It may be safe
under WinFsp, but that needs establishing before it is worth trying, and it
would need several CI runs to judge given the ~50% rate.Where this leaves the issue
- The diagnostics from fix(mount): stop caching negative lookups — the root cause of the #88 conformance flake #98 are doing their job and should stay.
negative_timeout = 0stays: free on Linux, no measured effect on Windows.- mount: punch the parent directory after a mutation the mount performed itself (does NOT fix #88) #111 is a plausible correctness gap - a mutation through the mount really does
not invalidate the parent's cached listing - but it has no demonstrated
effect on this flake, and I would not merge it as a fix for it. - Nobody should treat the next plausible-sounding cause as settled without
sampling the WinFsp job at least four or five times.
- added a commit that references this issue
on Sep 6, 2026 fuse_invalidate_pathdoes nothing on WinFsp, so two of the three attempts could not have workedNot a theory this time — it is in the header WinFsp ships, and it is what wsldrive compiles against.
inc/fuse3/fuse.h, unchanged from v2.0 through master (CI pins 2.1.25156):FSP_FUSE_SYM( int fuse3_invalidate_path(struct fuse3 *f, const char *path), { (void)f; (void)path; return -ENOENT; })
Two things make that the call wsldrive actually gets:
inc/fuse3/winfsp_fuse.hmaps WinFsp's internal spelling onto the standard one —#define fuse3_invalidate_path fuse_invalidate_path— so this isfuse_invalidate_path.FSP_FUSE_SYM(proto, ...)expands tostatic inline proto { __VA_ARGS__ }for everything but Cygwin, andcmake/WinFsp.cmakesets noCYGFUSEand no override.
So on Windows the call is a
static inlinefunction whose whole body isreturn -ENOENT. It never reaches the DLL, never reaches the FSD, and invalidates nothing.What that retires
- The page-cache punch thread is inert on Direction A.
FuseMount::inval_loopdequeues paths and calls a no-op. The comment on it records "WinFsp showed no difference from the page-cache change either way" — the observation was right, the explanation was not. It costs nothing there because it does nothing there. - mount: punch the parent directory after a mutation the mount performed itself (does NOT fix #88) #111 measured nothing. The parent-directory punch after a mutation is the same call. Its 3 pass / 2 fail sample was the baseline rate with a no-op added, which is exactly what an unchanged result looks like.
- The synchronous-punch hypothesis is moot on Windows. Whether the call is made inline from the handler or from another thread does not matter when the call returns
-ENOENTwithout doing anything. That also removes the deadlock question, at least for Direction A — there is nothing there worth risking it for.
This does not identify the cause of the flake. It does mean Direction A has no cache-invalidation mechanism at all, so any fix shaped like "drop the stale entry" is dead on arrival, and the single dump showing a stale enumeration of
dhas no lever behind it as things stand.What is left
Only one thing wsldrive currently controls about WinFsp's caching: the timeout it advertises.
entry_timeout/attr_timeoutare 1.0 s and become the FSD'sFileInfoTimeout, which is the cache that dump was showing.So the next experiment is the one already named in this thread —
entry_timeout = 0on Windows only — but with a reason to expect something from it rather than a guess, since it is now the only lever there is. The ~13x stat cost measured for that was on Linux/libfuse; the Windows cost is unmeasured, and Direction A answers metadata from the in-RAM mirror. Worth measuring both the flake rate (four or five WinFsp samples, per the note above) and the Direction A benchmark before drawing any conclusion.Beyond that, the real fix is
FspFileSystemNotify, WinFsp's native equivalent — which would also give Direction A the Windows change notifications it does not currently produce at all. The obstacle is that the FUSE compatibility layer does not expose theFSP_FILE_SYSTEM*it wraps:struct fuse_contextcarriesfuse,uid,gid,pid,private_data,umaskand nothing else, and there is no getter in the public headers. Reaching it means going around the FUSE layer for Direction A, which is a much larger change than this issue should decide on its own.🤖 Generated with Claude Code
Everything
op_initsets infuse_configis discarded on WinFsp — and there is a real knob it never reachedFollowing the note above, I read how WinFsp's fuse3 shim calls
init.src/dll/fuse3/fuse2to3.c:struct fuse3_config conf3; memset(&conf3, 0, sizeof conf3); conf3.set_gid = f->set_gid; ... #if 0 /* * Cannot set timeouts because of lack of floating point support. * ... */ conf3.entry_timeout = ... conf3.negative_timeout = 0; conf3.attr_timeout = f->VolumeParams.FileInfoTimeout / 1000; conf3.ac_attr_timeout = conf3.attr_timeout; #endif void *res = f3->ops.init(&conn3, &conf3); conn->max_write = conn3.max_write; conn->max_readahead = conn3.max_readahead; conn->want = ...;
The timeouts are not passed in (the
#if 0, with upstream's own reason: the DLL avoids the standard library and FUSE spells timeouts asdouble), and afterinitreturns only three fields are copied out — all fromconn.conf3is a zeroed stack struct that is thrown away.So on Direction A,
cfg->entry_timeout,cfg->attr_timeout,cfg->negative_timeoutandcfg->auto_cacheare all written into memory nobody reads. That is not a quirk ofnegative_timeout; it is every field offuse_config. The probe in the earlier comment measured identical timings fornegative_timeout = 0and1.0and concluded "either WinFsp-FUSE ignores it, or something else invalidates the entry first" — it is the first, and it applies to the whole struct.What does survive
op_initon Windows isconn->max_write,conn->max_readaheadandconn->want, so the connection tuning and the capability filtering are real. Only the cache policy is inert.The knob that works is a mount option, and there is one aimed exactly at the failing cache
WinFsp's FUSE core parses these at mount time, before
initruns (src/dll/fuse/fuse.c):-o FileInfoTimeout=N metadata timeout (millis, -1 for data caching) -o DirInfoTimeout=N directory info timeout (millis)FileInfoTimeoutdefaults to 1000 ms,attr_timeout=is accepted as an alias for it, andentry_timeout/negative_timeoutareFUSE_OPT_KEY_DISCARD— discarded outright.-o DirInfoTimeout=NsetsVolumeParams.DirInfoTimeoutand itsValidflag; without it, directory info inheritsFileInfoTimeout.wsldrive passes
-ouid=-1,gid=-1,rellinks,FileSystemName=wsldrive,volname=wsldriveand no timeout, so Direction A runs with the 1000 ms default — the same one second the stale-enumeration dump showed, on the same cache.DirInfoTimeoutmatters more than the general timeout here because the one dump that showed a genuine disagreement showed it in exactly that cache:d's listing missingsub2whiled\sub2resolved directly and listeddeepercorrectly. That is directory info being stale while file info is fine.-o DirInfoTimeout=0turns off that one cache and leaves the metadata cache — the thing that makes Direction A fast — untouched, which is a far narrower change than theentry_timeout = 0the thread was contemplating (and which would have done nothing anyway).Next
I will measure the current flake rate before changing anything, since the last figures in this thread are from early September, then sample
DirInfoTimeout=0the four or five times this issue rightly insists on. Posting the mechanism now because it stands on its own regardless of how that comes out.🤖 Generated with Claude Code
Measured the current rate before touching anything, and there is nothing to measure against
Last 40 CI runs, counting the
Filesystem conformance (WinFsp mount)job:passed 35 failed 3 did not complete 2 All three failures are on 2026-09-05 and 2026-09-06, the last at
2026-09-06T01:32Z. The 11 runs
since are clean, including the six on #119 today.Nothing merged in that window can account for it. #115 landed at 2026-09-05 21:00, before the last
failure, and says so itself: "This adds no fix and asserts nothing new." So the flake is dormant,
not fixed — and its-Settlediagnostics have never once fired.Which means I am not going to change the timeout
-o DirInfoTimeout=0is the first candidate in this thread with a mechanism behind it rather than a
guess. It is also, right now, unfalsifiable: with 0 failures in 11 runs there is no rate for it to
improve, and "the flake did not come back" is exactly the evidence that produced the ten-green-runs
over-attribution recorded earlier in this thread. Shipping it now would be the fourth plausible
fix in a row and the first one nobody could check.It is not free either.
DirInfoTimeoutgoverns directory-enumeration caching, which is what makes a
Direction Areaddirfast, and the cost of zeroing it on Windows is unmeasured.What to do when it next fires
The next occurrence arrives with the one thing every previous one lacked, and it discriminates
cleanly:- "settled after ~1000 ms" — that is
FileInfoTimeout's default expiring, so the stale cache is
the FSD's and-o DirInfoTimeout=0(or a lowerFileInfoTimeout) is the fix. Measure the
Direction Areaddirbenchmark alongside it, since that is what pays. - "settled after" something unrelated to 1000 ms — a different timeout, and the value names it.
- "still wrong after 5000 ms" — not a cache at all; state was lost, and everything in this thread
including my own two comments is looking in the wrong place.
Until then the useful output of this investigation is the three things now ruled out by construction
rather than by sampling: the page-cache punch, the parent-directory punch, and the whole
fuse_configcache policy. None of them was ever capable of affecting Direction A.Worth keeping open — the flake has gone quiet before.
🤖 Generated with Claude Code
- "settled after ~1000 ms" — that is
#120 is merged. It removes the dead thread and, more importantly, corrects the comment that implied
Direction A has a cache-coherence mechanism — the assumption behind two of the three fixes this issue
has already tried. No fix for the flake, and none claimed.That is everything I can establish without a reproduction. The state of the investigation:
candidate verdict page-cache punch inert on Direction A — fuse_invalidate_pathis astatic inlinestubnegative_timeout = 0inert — the whole fuse_configis discarded by WinFsp's init shimparent-directory punch (#111) inert, same stub; its 3 pass / 2 fail sample measured nothing -o FileInfoTimeout/-o DirInfoTimeoutuntried, and the only lever that reaches the FSD Three ruled out by construction rather than by sampling, which is the part that was missing: each was
previously judged on a flake rate that could not have responded to it.The remaining candidate cannot be judged the same way, because the flake is dormant — 0 failures in
the last 11 WinFsp jobs, last one 2026-09-06. ShippingDirInfoTimeout=0now would be the fourth
plausible fix and the first that nobody could check.What would actually finish this
Make it reproduce on demand. It has only ever fired on the runners, and the failing check is one
line:Check 'rename directory keeps contents' { Rename-Item "$M\d\sub" 'sub2'; Test-Path "$M\d\sub2\deeper" }
A stress mode that runs that mutation-and-read in a loop, on a runner, under load, would either hit
the window reliably or demonstrate it is gone. Either answer ends the guessing, and the first one
turnsDirInfoTimeout=0from a fourth theory into something measurable — with the Direction A
readdirbenchmark alongside it, since that is what the timeout pays for.Needs a loop flag in the Windows battery and a manual CI trigger, which
ci.ymldoes not have today.
Happy to build it if that is wanted; leaving the issue open and unstarted otherwise.🤖 Generated with Claude Code
It recurred on main after this was closed, and the settle diagnostic finally fired
Run 34790287854, main at
feb7970,
about an hour after this issue was closed:FAIL rename directory keeps contents -- condition was false settled after 21 ms - transient, the mount caught up mount [d]: b.txt sub2 backing [d]: b.txt sub2 mount [d\sub2]: deeper backing [d\sub2]: deeper mount [d\sub]: <absent> backing [d\sub]: <absent>First time #115's
-Settlehas ever produced a number, and it is the discriminator this
investigation has been waiting three months for.21 ms rules out every cache theory, including mine
FileInfoTimeoutdefaults to 1000 ms, andDirInfoTimeoutinherits it. If the stale entry were
the FSD's, the check would have stayed wrong for something close to a second. It was right again
after 21 ms.So
-o DirInfoTimeout=0, the candidate I argued was "the only lever there is" two comments up, would
almost certainly not have fixed this either. That is the fifth theory this issue has produced and the
second of mine. Recording it as refuted before anyone spends a CI campaign on it.What survives from the earlier comments is only the negative half, which does still hold:
fuse_invalidate_pathis a no-op stub on WinFsp and the wholefuse_configis discarded, so the
punch-based fixes could never have worked. They were just wrong about where to look next.Where 21 ms does point
Squarely inside the agent's own invalidation window.
Coalescer::Optionsdefaults are
quiet_period = 2 msandmax_latency = 25 ms, so a watcher event raised by the rename is flushed
to the client somewhere in that range — and the mount became correct at 21 ms.That reframes the whole thing. The rename goes through the mount, so
op_renameforwards it and
RemoteRoot::renameupdates the mirror optimistically; the check then reads the result immediately.
If the optimistic update left the subtree briefly unresolvable, the agent's echoed invalidation is
exactly what would repair it, on exactly that timescale. That is our code, not WinFsp's, which also
explains why three Windows-cache theories in a row went nowhere and why the Linux battery shows the
same shape ongit checkout -b.Stated as the strongest available lead, not a conclusion. This thread has a bad record with
plausible-sounding causes and I have contributed one of them.One more discriminator in the same run
The write-through battery passed
rename directory keeps contentsin that very job; only the
write-back variant failed it. Both ran against the same binaries minutes apart, so whatever this
is,--writebackmakes it more likely — another free constraint on the search.Reopening is your call; commenting rather than reopening since you closed it deliberately. But the
flake is not gone, and the next step is now much better specified than it was: instrument the
optimistic rename path rather than any cache.🤖 Generated with Claude Code
The mounted conformance batteries fail intermittently in CI, always on one of the git checks, and always with the rest of the battery passing. 6 of the last 30 CI runs (2026-08-29):
git checkout -brename file→ fixed by #82rename file→ fixed by #82Two of those (the
rename fileones) had a real cause and were fixed in #82 (a stale invalidation resurrecting a renamed-away path). The remainder are still unexplained; theci/dependabotfailure is on a PR that changes nothing but a YAML file, so it is not a regression from the change under test.Not reproducible locally: 10/10 clean runs of the Linux battery in WSL, and 3× both Windows batteries, on the same commit. The GitHub runners are slower and single-core-ish, which widens the window between a mutation, the watcher event and the flush.
Either there is a residual race in the mount (a genuine, user-visible inconsistency under load) or the battery races itself (checking immediately after a change while an invalidation is in flight). Both matter: the first is a bug, the second makes CI untrustworthy.
Done so far:
fs-conformance.sh'scheck()now prints the failing command's output plus the mount's and the backing store's.gitlistings, and the run dumps the agent and mount logs on failure. The next occurrence should say which git operation failed and how the two views differed.Next: wait for one CI failure with the new diagnostics, then fix the root cause. If it turns out to be a test race, the check should wait for convergence explicitly rather than being loosened.