fix(brain): compact sync log on installs with no brain-sync peers - #5462
fix(brain): compact sync log on installs with no brain-sync peers#5462prateeekbuilds wants to merge 1 commit into
Conversation
…omantic#5439) - Compact to getCurrentSeq() in syncAllPeers() when no brain-sync peers are enabled, keeping the last entry so initSyncLog() recovers the sequence counter across restarts - Add early return in brainSyncLog.compactLog() when dropped is 0 to avoid rewriting the log file on idle cycles - Add unit and regression tests in syncOrchestrator.test.js and brainSyncLog.test.js
atomantic
left a comment
atomantic
left a comment
There was a problem hiding this comment.
Reviewed by /do:review — 2 critical, 1 improvement, 0 nits.
Highlights
- server/services/syncOrchestrator.js:881 — the compaction floor can exceed the durable sequence after a failed append, emptying the log and resetting sequence recovery.
- server/services/syncOrchestrator.js:878 — local pull categories do not account for inbound/asymmetric or pre-reconcile delta consumers.
Coherence check
The patch matches issue #5439, but its later-peer reconciliation claim is false for supported pre-#1077 delta-only peers.
Generated by /do:review
| // No brain-sync peer will ever pull these entries; a peer added later | ||
| // converges through the reconcile snapshot (#1077). Keep the last entry so | ||
| // initSyncLog still recovers the sequence counter across restarts. | ||
| await brainSyncLog.compactLog(brainSyncLog.getCurrentSeq()); |
There was a problem hiding this comment.
[CRITICAL] Choose the compaction floor from durable state under the log mutex
appendChange and appendChanges reserve currentSeq before appendFile; a failed append leaves indexLoaded = false but the exported sequence ahead of disk. This call can pass N+1, then compactLog() reloads durable N and still filters with N+1, dropping every line; a restart resets the sequence to 0 and can reuse numbers that retained peer cursors already consumed. Resolve and check the durable current sequence inside brainSyncLog while holding its mutex, such as an atomic compact-to-current operation, rather than reading it here.
| const minSeq = Math.min(...consumedSeqs); | ||
| await brainSyncLog.compactLog(minSeq); | ||
| } else if (brainSyncLog.getCurrentSeq() > 0) { | ||
| // No brain-sync peer will ever pull these entries; a peer added later |
There was a problem hiding this comment.
[CRITICAL] Local pull settings do not prove the log has no remote consumers
getEffectiveCategories(peer).brain controls what this install pulls, but inbound or asymmetric peers can still call GET /api/brain/sync; announced peers begin sync-disabled and older peers may remain one-directional. Pre-#1077 peers also have no reconcile endpoint and explicitly fall back to delta-only, so keeping only the newest delta permanently strands all earlier records. Use a compatibility-preserving bootstrap or compaction representation, or a proven remote-consumer capability gate; the local category map cannot justify destructive truncation.
| expect(dropped).toBe(0); | ||
|
|
||
| const afterContent = readFileSync(syncLogPath(), 'utf8'); | ||
| expect(afterContent).toBe(beforeContent); |
There was a problem hiding this comment.
[IMPROVEMENT] Assert the absence of a write, not only identical bytes
Removing the new if (dropped === 0) return 0 would still reconstruct and atomically write the same bytes, so this before/after content comparison passes while the idle-rewrite regression returns. Wrap atomicWrite with a delegating spy in the fileUtils proxy and assert it was not called after compactLog(1).
3 participants
Description
Fixes #5439.
On installs without enabled brain-sync peers (the default single-machine posture),
compactLogwas previously never called because compaction only ran withinif (brainPeers.length > 0). As a result, mutations across all ten brain types continuously appended entries todata/brain/sync_log.jsonlwithout bound, increasing startup latency and memory spikes ininitSyncLog().Key Changes
server/services/syncOrchestrator.js:else if (brainSyncLog.getCurrentSeq() > 0)branch tosyncAllPeers().getCurrentSeq() > 0, compact the log tobrainSyncLog.getCurrentSeq(). This keeps only the newest entry soinitSyncLog()recovers the sequence counter across restarts, while discarding unpulled delta history that will never be requested (peers added later converge via reconcile snapshot Brain sync is delta-log-only — diverged peers never re-converge (no anti-entropy) #1077).server/services/brainSyncLog.js:if (dropped === 0) return 0;insidecompactLog()before callingatomicWrite(), avoiding disk rewrites and log noise on idle cycles.syncOrchestrator.test.jsverifying compaction behavior when no peers are configured, when peers exist but none is brain-enabled, and whengetCurrentSeq() === 0.brainSyncLog.test.jsverifying thatcompactLog(currentSeq)keeps only the newest entry, recovers sequence counters acrossinitSyncLog()restarts, and that idle compactions do not rewrite the file.Testing & Verification
cd server && npm test services/syncOrchestrator.test.js services/brainSyncLog.test.jspassed (71 tests).cursorForYou.brainSeq, 0 for unreported peer) stay green.