Repository navigation
fix(sync): bound himalaya read concurrency and keep partial sync progress - #34
Merged
Merged
Conversation
A page of envelopes was read with Promise.all, so a page-size of 1000 fired up to 1000 simultaneous himalaya processes (one IMAP login each) and Gmail throttled the fan-out. A single throttled read rejected the whole page and the archive was never synced. Read every envelope through a bounded pool that defaults to 4 concurrent message reads and preserves envelope order, and expose the pool size through HimalayaReadOptions so the CLI can opt into a larger or smaller window. The POSIX stub-himalaya test reproduces the throttle through the real execFile and PATH spawn path; the in-process tests cover the pool on every platform. Ultraworked with [omo](https://github.com/code-yeongyu/oh-my-openagent)
Reads now run in rounds: the first round reads the whole page through the bounded pool, and later rounds retry only the messages that failed, waiting a doubling backoff (250ms, then 500ms) before each round, up to three attempts per message. A read that still fails reports its key, the attempt count, and the redacted himalaya error. A page whose read fails also stops dispatching new reads, so an abandoned page cannot keep spawning provider processes for the rest of the window. Ultraworked with [omo](https://github.com/code-yeongyu/oh-my-openagent)
A page with one unreadable message rejected the whole sync because the CLI called source.list() before archive.sync(). Sources now expose collect(), which returns the messages that were read plus one record per message that could not be read, and sync reports those records as failures[] while still committing the readable messages. Add --concurrency <n> (a validated positive integer, default 4) so the read pool is independent of the envelope window, and exit nonzero only when nothing could be read. Himalaya's own stderr is preserved in the surfaced error so throttling is distinguishable from a malformed message. Ultraworked with [omo](https://github.com/code-yeongyu/oh-my-openagent)
Document --concurrency, the --page-size envelope window, the failures[] sync field, and the non-zero exit rule in the README, the architecture contract, and the agent skill. Ultraworked with [omo](https://github.com/code-yeongyu/oh-my-openagent)
Member
Author
Real-account verification (post-fix)Read-only run on this machine against the reporter's Gmail INBOX with the node dist/cli/index.js --data-dir <tmp> sync --source himalaya \
--account gmail --mailbox INBOX --page-size 25 --concurrency 4 --jsonSync JSON (exit 0): {"added":25,"updated":0,"deleted":0,"unchanged":0,"touchedThreads":25,"rebuiltThreads":25,"chunksAdded":833,"chunksDeleted":0,"embeddingBacklog":833,"excluded":0,"excludedByReason":{},"failures":[]}
Before the fix the same command aborted after ~12s with Two notes on the run:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #33.
Root cause
HimalayaSource.list()read every envelope of a page withPromise.all, so thedefault
--page-size 1000fired up to 1000 simultaneoushimalaya message readprocesses (one IMAP login each). Gmail throttles that fan-out, one read failed,
the whole page rejected, and because the CLI only called
archive.sync()afterlist()resolved, the archive was left untouched: no retry, no partialprogress, no usable chunk store.
What changed
src/source.ts) — a page is read through a poolof at most
--concurrency <n>processes, default 4, decoupled from--page-size, which stays the envelope window. A page whose read fails alsostops dispatching new reads instead of spawning the rest of the window.
src/source.ts) — reads run in rounds:the first round reads the page, later rounds retry only the messages that
failed, after a doubling backoff (250ms, then 500ms), three attempts per
message.
src/source.ts,src/cli/index.ts) —sources expose
collect()returning{ messages, failures }.synccommitsthe readable messages and reports
failures[](providerKey,attempts, andthe redacted error); it exits nonzero only when nothing could be read.
distinguishable from a malformed message.
docs/architecture.md, and the agent skill document thepool, the envelope window,
failures[], and the exit rule.Verification
npm run typecheck,npm run build,npm test(24 files / 80 tests),npm run validate:skill— all green, including the existing fixture sync andstatus/embed/repair/doctor CLI contracts.
tests/issue-33-read-concurrency.test.ts(11 tests): pool bound and orderpreservation, dispatch stop after a failure, retry with growing backoff,
partial progress, stderr preservation, the CLI JSON contract, the
nothing-read exit rule, the empty-page boundary, and
--concurrencyvalidation.
himalayabinary driven through thereal
execFile+PATHspawn path (POSIX-only tests; Windows CI runs thein-process pool, retry, and collect coverage, and the CLI validation case).
(page abort on throttle, first-attempt rejection, sync abort with one
unreadable message, dropped stderr), and mutation proofs confirm the new
assertions can fail (unbounded pool, no backoff, retry disabled,
failuresdropped from the sync JSON, nothing-read rule disabled, empty page treated as
a failure).
Real-account check
Read-only run on this machine against a real Gmail INBOX with the issue's own
repro (
sync --page-size 25 --concurrency 4, temp data dir):archive.sync(), then stopped on a missing Korean-analyzer shell export(
MAILCRAWL_KIWI_MODEL) — an environment prerequisite fromdocs/multilingual-installation.md, unrelated to this change.exit 0,
added: 25,failures: [], andstatus --jsonshowing 25 messages /833 chunks / healthy FTS. Details in the PR comment below.