Skip to content

fix(fileindex): serialize withFolderRO's reload against writers via the cross-process lock (#647)#655

Merged
0kaba0hub merged 1 commit into
mainfrom
fix/ro-reload-cross-process-lock
Jul 19, 2026
Merged

fix(fileindex): serialize withFolderRO's reload against writers via the cross-process lock (#647)#655
0kaba0hub merged 1 commit into
mainfrom
fix/ro-reload-cross-process-lock

Conversation

@0kaba0hub

Copy link
Copy Markdown
Owner

Closes #647.

Problem

withFolderRO reloaded folder state under only the in-process fs.mu, never the cross-process distributed lock (u.b.locker) that the write path (withFolderLock) takes before its own reload. An unlocked reload could interleave with another process's lock-holding compaction (flush + truncateLog) and load a torn view into the shared in-memory folderState — which every subsequent, correctly locked write then trusts as a baseline, regressing the folder's NextUID under concurrent load.

This is the second trigger of the #644 class, distinct from the stale-logFD/inode case fixed in #645: no .log replaced WARN fires here — the fd may not have been replaced at all; the poison is in reading base+log content while a writer compacts. It reproduced live on 2.1.11 (after #645) with the same signature (UIDValidity unchanged, NextUID → 1/2/3), no #645 detector firing.

Fix

  • The locks API is exclusive-per-key only (no shared/read mode), so the read path takes the same exclusive key as writers — correctness over throughput, as the issue accepts. Re-entrancy is handled by the existing HoldsResource shortcut, so a nested / POP3-QUIT caller that already holds the key does not deadlock.
  • Extract withDistLock: the HoldsResource-guarded cross-process acquire (plus the existing debug timings), now shared by withFolderLock and withFolderRO.
  • withFolderRO runs reload() under withDistLock (reload(locked=true)); fn then reads the settled snapshot under fs.mu.RLock without holding the exclusive resource — the lock covers only the reload, not the whole read, minimizing exclusive hold.

Trade-off (deliberate)

Every read on the hot path (LMTP delivery's OpenFolder dedup, FETCH's GetMessages) now takes the exclusive per-(user,folder) lock around its reload — an extra Redis round-trip and serialization of reads against writes. The issue accepts this explicitly. A future optimization (shared/read locks) is only possible if the lock backend gains them; not done here.

Test

TestReadPathSerializesAgainstConcurrentLockHolder: two lock clients (two pods) on one embedded server; client B holds the folder key, a read-only op on the index wired to client A must block until B releases — proving the read reload now serializes against a concurrent compaction. (Without the fix, the read returns immediately → test fails.)

go test ./... + -tags flatcurve build + helm lint green. appVersion → 2.1.16.

…he cross-process lock (#647)

The read path (withFolderRO) reloaded folder state under only the in-process
fs.mu, never the cross-process distributed lock the write path (withFolderLock)
takes before its own reload. An unlocked reload could interleave with another
process's lock-holding compaction (flush + truncateLog) and load a torn view
into the shared in-memory folderState — which every subsequent, correctly
locked write then trusts as a baseline, regressing the folder's NextUID under
concurrent load. This is the second trigger of the #644 class, distinct from the
stale-logFD/inode case fixed in #645 (no ".log replaced" WARN fires here — the
fd may not have been replaced at all; the poison is in reading base+log content
while a writer compacts).

The locks API is exclusive-per-key only (no shared/read mode), so the read path
takes the same exclusive key as writers — correctness over throughput, as the
issue accepts. Re-entrancy is handled by the existing HoldsResource shortcut, so
a nested/POP3-QUIT caller that already holds the key does not deadlock.

- Extract withDistLock: the HoldsResource-guarded cross-process acquire (plus
  the existing debug timings), shared by withFolderLock and withFolderRO.
- withFolderRO now runs reload() under withDistLock (reload(locked=true)); fn
  then reads the settled snapshot under fs.mu.RLock WITHOUT holding the
  exclusive resource, so the lock covers only the reload, not the whole read.

Adds TestReadPathSerializesAgainstConcurrentLockHolder: two lock clients (two
pods) on one embedded server; client B holds the folder key, a read-only op on
the index wired to client A must block until B releases — proving the read
reload now serializes against a concurrent compaction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants