Skip to content

fix(sync): bound himalaya read concurrency and keep partial sync progress - #34

Merged
vkehfdl1 merged 4 commits into
mainfrom
fix/issue-33-read-concurrency
Sep 19, 2026
Merged

vkehfdl1 merged 4 commits into
mainfrom
fix/issue-33-read-concurrency

Conversation

@vkehfdl1

@vkehfdl1 vkehfdl1 commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

Closes #33.

Root cause

HimalayaSource.list() read every envelope of a page with Promise.all, so the
default --page-size 1000 fired up to 1000 simultaneous himalaya message read
processes (one IMAP login each). Gmail throttles that fan-out, one read failed,
the whole page rejected, and because the CLI only called archive.sync() after
list() resolved, the archive was left untouched: no retry, no partial
progress, no usable chunk store.

What changed

  • Bounded read concurrency (src/source.ts) — a page is read through a pool
    of at most --concurrency <n> processes, default 4, decoupled from
    --page-size, which stays the envelope window. A page whose read fails also
    stops dispatching new reads instead of spawning the rest of the window.
  • Per-message retry with backoff (src/source.ts) — reads run in rounds:
    the first round reads the page, later rounds retry only the messages that
    failed, after a doubling backoff (250ms, then 500ms), three attempts per
    message.
  • Partial progress + diagnostics (src/source.ts, src/cli/index.ts) —
    sources expose collect() returning { messages, failures }. sync commits
    the readable messages and reports failures[] (providerKey, attempts, and
    the redacted error); it exits nonzero only when nothing could be read.
  • himalaya stderr preserved in the surfaced error, so throttling is
    distinguishable from a malformed message.
  • Docs — README, docs/architecture.md, and the agent skill document the
    pool, the envelope window, failures[], and the exit rule.

Verification

  • npm run typecheck, npm run build, npm test (24 files / 80 tests),
    npm run validate:skill — all green, including the existing fixture sync and
    status/embed/repair/doctor CLI contracts.
  • New tests/issue-33-read-concurrency.test.ts (11 tests): pool bound and order
    preservation, dispatch stop after a failure, retry with growing backoff,
    partial progress, stderr preservation, the CLI JSON contract, the
    nothing-read exit rule, the empty-page boundary, and --concurrency
    validation.
  • The throttle is reproduced with a stub himalaya binary driven through the
    real execFile + PATH spawn path (POSIX-only tests; Windows CI runs the
    in-process pool, retry, and collect coverage, and the CLI validation case).
  • Each new assertion was shown to fail for the right reason before the fix
    (page abort on throttle, first-attempt rejection, sync abort with one
    unreadable message, dropped stderr), and mutation proofs confirm the new
    assertions can fail (unbounded pool, no backoff, retry disabled, failures
    dropped from the sync JSON, nothing-read rule disabled, empty page treated as
    a failure).

Real-account check

Read-only run on this machine against a real Gmail INBOX with the issue's own
repro (sync --page-size 25 --concurrency 4, temp data dir):

  • The first attempt read the whole 25-message page at concurrency 4 and reached
    archive.sync(), then stopped on a missing Korean-analyzer shell export
    (MAILCRAWL_KIWI_MODEL) — an environment prerequisite from
    docs/multilingual-installation.md, unrelated to this change.
  • Re-run with the documented analyzer exports (plus the JA/ZH helper paths) —
    exit 0, added: 25, failures: [], and status --json showing 25 messages /
    833 chunks / healthy FTS. Details in the PR comment below.

A page of envelopes was read with Promise.all, so a page-size of 1000 fired up
to 1000 simultaneous himalaya processes (one IMAP login each) and Gmail
throttled the fan-out. A single throttled read rejected the whole page and the
archive was never synced.

Read every envelope through a bounded pool that defaults to 4 concurrent
message reads and preserves envelope order, and expose the pool size through
HimalayaReadOptions so the CLI can opt into a larger or smaller window. The
POSIX stub-himalaya test reproduces the throttle through the real execFile and
PATH spawn path; the in-process tests cover the pool on every platform.

Ultraworked with [omo](https://github.com/code-yeongyu/oh-my-openagent)
Reads now run in rounds: the first round reads the whole page through the
bounded pool, and later rounds retry only the messages that failed, waiting a
doubling backoff (250ms, then 500ms) before each round, up to three attempts
per message. A read that still fails reports its key, the attempt count, and
the redacted himalaya error.

A page whose read fails also stops dispatching new reads, so an abandoned page
cannot keep spawning provider processes for the rest of the window.

Ultraworked with [omo](https://github.com/code-yeongyu/oh-my-openagent)
A page with one unreadable message rejected the whole sync because the CLI
called source.list() before archive.sync(). Sources now expose collect(),
which returns the messages that were read plus one record per message that
could not be read, and sync reports those records as failures[] while still
committing the readable messages.

Add --concurrency <n> (a validated positive integer, default 4) so the read
pool is independent of the envelope window, and exit nonzero only when nothing
could be read. Himalaya's own stderr is preserved in the surfaced error so
throttling is distinguishable from a malformed message.

Ultraworked with [omo](https://github.com/code-yeongyu/oh-my-openagent)
Document --concurrency, the --page-size envelope window, the failures[] sync
field, and the non-zero exit rule in the README, the architecture contract,
and the agent skill.

Ultraworked with [omo](https://github.com/code-yeongyu/oh-my-openagent)
@vkehfdl1

Copy link
Copy Markdown
Member Author

Real-account verification (post-fix)

Read-only run on this machine against the reporter's Gmail INBOX with the
issue's own repro, using the built CLI from this branch:

node dist/cli/index.js --data-dir <tmp> sync --source himalaya \
  --account gmail --mailbox INBOX --page-size 25 --concurrency 4 --json

Sync JSON (exit 0):

{"added":25,"updated":0,"deleted":0,"unchanged":0,"touchedThreads":25,"rebuiltThreads":25,"chunksAdded":833,"chunksDeleted":0,"embeddingBacklog":833,"excluded":0,"excludedByReason":{},"failures":[]}

status --json afterwards: messageCount: 25, chunkCount: 833,
fts: {status: "healthy", rows: 833}.

Before the fix the same command aborted after ~12s with
himalaya message read failed: … FETCH returned no body for the requested message and left messageCount 0 (reported in #33). No read failed this time.

Two notes on the run:

  • The first post-fix attempt read the whole 25-message page at concurrency 4 and
    reached archive.sync(), then stopped on a missing MAILCRAWL_KIWI_MODEL
    shell export — an environment prerequisite from
    docs/multilingual-installation.md, unrelated to this change. The result
    above is from the re-run with the documented analyzer exports (and the JA/ZH
    helper paths) set.
  • The temp archive created by this run was deleted afterwards; no message
    content is included here.

@vkehfdl1
vkehfdl1 merged commit 0ff59e4 into main Sep 19, 2026
4 checks passed
@vkehfdl1
vkehfdl1 deleted the fix/issue-33-read-concurrency branch September 19, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] HimalayaSource reads a whole page with unbounded Promise.all — one throttled message read aborts the entire sync (no retry, no partial progress)

1 participant