Bound parallel checker peak memory to O(schedulers), not O(batch)#15544
Closed
aspett wants to merge 1 commit into
Closed
Bound parallel checker peak memory to O(schedulers), not O(batch)#15544aspett wants to merge 1 commit into
aspett wants to merge 1 commit into
Conversation
The group verification phase held every module's expanded definitions AST resident at once: handle_call(:start) caches all modules before checking any, and each checker worker retained its module_tuple from :cache until :check. On a large batch this is the dominant memory term (~0.85 MB/module; verifying 5000 modules held ~3.5 GB of process memory). Three changes keep peak proportional to the scheduler count: * Defer reconstructing the definitions AST from :cache to :check, holding only the beam location and re-reading lazily, so only the ~schedulers modules being checked concurrently materialize their tuple. Guarded on debug_info being kept (the lazy re-read needs it); otherwise keep the in-memory tuple so checking still runs. * Garbage-collect each worker after caching, before it blocks on :check, to reclaim the transient beam read/parse heap it would otherwise hold for the whole batch. * Cache in bounded waves in handle_call(:start) instead of starting every worker at once, while still caching all modules before any checking begins. Warnings and emitted .beam files are unchanged. On a full compile of a 5000-module project: peak RSS ~1934 MB -> ~844 MB (process memory 1156 -> 96 MB).
Member
|
PR #15545 with more conservative changes and a few approaches for us to try! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In a large application (30k+ modules), we see peak RAM consumption in some compilation scenarios reach near 10Gi or more. Upon investigation, the cause appears to be the ParallelChecker is expanding and caching the AST of every module at once before checking begins.
By changing this behaviour to defer expansion until checking, we can reduce peak memory use by 40-80%. (Measured 8.46Gi -> 1.12Gi on last attempt)
The tradeoff is compile time though; deferring on a small recompile on this codebase added ~9%