Import messages and users concurrently to speed up large migrations - #67
Open
tgvashworth wants to merge 2 commits into
Open
Import messages and users concurrently to speed up large migrations#67tgvashworth wants to merge 2 commits into
tgvashworth wants to merge 2 commits into
Conversation
The script imported one message at a time, fully sequentially, which left most of Gmail's per-user API quota unused (observed ~0.65 messages/sec vs. a ~4/sec quota ceiling). For migrations covering many users and gigabytes of mail this made large imports impractically slow. - Add --message_concurrency (default 4) to import multiple messages from one mbox file in parallel, using a per-thread Gmail service since httplib2 isn't safe to share across threads. - Add --user_concurrency (default 4) to import multiple users in parallel, since each user has an independent Gmail API quota bucket. - Bound in-flight requests via a sliding window so memory use doesn't grow with the size of a (potentially huge) mbox file. - Both default to the old sequential behavior when set to 1. - Document the quota math and new flags in the README. Benchmarked against a real test mailbox: ~6x speedup (0.65 -> ~4 messages/sec), which lines up with Gmail's documented per-user quota ceiling (6,000 units/minute / 25 units per messages.import call).
Self-review caught a serious bug: get_service_for_thread cached "first call wins" per thread, but threads in the --user_concurrency pool are reused across different users. With --message_concurrency 1 (the documented fallback to one-at-a-time behavior) and more users than --user_concurrency, a reused thread would silently keep importing messages using a stale, previously-cached user's credentials instead of the current user's - i.e. mail could land in the wrong mailbox. - Key the per-thread service cache on credentials identity, so a reused thread rebuilds its service when it sees different credentials. - Make --message_concurrency 1 actually skip the per-thread cache and use the passed-in service directly, matching what the docstring already claimed. - Wrap per-thread service construction in a try/except so a transient failure counts as one failed message instead of raising out of process_mbox_file and aborting the rest of that user's import. - Validate --message_concurrency/--user_concurrency >= 1 with a clear argparse error instead of a raw ValueError from ThreadPoolExecutor. - Simplify imap_bounded to track a set of futures instead of a dict with unused values. Reproduced the original bug mechanism in isolation (7/8 mismatches under thread reuse) and confirmed the fix resolves it (0/8), in addition to the new regression tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--message_concurrency(default 4) to import multiple messages from one mbox file in parallel, and--user_concurrency(default 4) to import multiple users in parallel. Each user has an independent Gmail API quota bucket, so user-level concurrency is the main lever for migrations covering many mailboxes.httplib2.Httpisn't safe to share across threads (per the google-api-python-client thread safety guide), so each worker thread gets its own Gmail service built from the same (thread-safe) credentials. In-flight requests are bounded via a sliding window so memory use doesn't grow with the size of a large mbox file.1, so existing invocations are unaffected unless you opt in.Why
Gmail's documented quota for
messages.importis 25 units per call, with a default of 6,000 units/minute per user (~4 imports/sec sustained) and 1,200,000 units/minute per project. The script's one-message-at-a-time loop meant a migration was bottlenecked by its own blocking I/O long before it ever got close to that quota — large migrations (many users, multi-gigabyte mailboxes) could take far longer than necessary.Test plan
python -m unittest tests.test_import— all 11 tests pass (8 existing + 3 new covering concurrent message import,--from_messagecorrectness under concurrency, and multi-user aggregation)--message_concurrency 8, and in 49s (4.1/sec) at--message_concurrency 16— confirming throughput caps out at Gmail's documented per-user quota ceiling rather than being limited by the script, with zero errors or retries in either run--from_messageRefs #19