Skip .worktrees and .phpstan when building the CPG - #1
Open
Braindea7 wants to merge 1 commit into
Open
Conversation
Two directory kinds slipped past DEFAULT_EXCLUDES/IGNORE_DIRS and broke the build on a Laravel monorepo: - `.worktrees/` holds full second checkouts (the layout `git worktree add` gets pointed at most often). Every symbol was indexed once per worktree — in the repo this was measured on, 13,344 of 18,630 indexed files came from three worktrees, so `lmc find` returned every hit four times and `impact` overstated the blast radius by the same factor, mixing in stale branch states that no longer exist anywhere. - `.phpstan/` is PHPStan's cache. Its `resultCache.php` reaches double-digit MB (23 MB here); php2cpg reads the parser output into memory in one go (IOUtils.contentFromBufferedSource) and dies with java.lang.OutOfMemoryError. Measured on the same repo, `lmc build` on 4,493 files: previously it aborted with `joern-parse exit 1` and fell back to the tree-sitter index, now the Joern CPG completes in 18 s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two directory kinds slip past
DEFAULT_EXCLUDES/IGNORE_DIRSand hurt the build on a Laravel monorepo..worktrees/holds full second checkouts — the layoutgit worktree addgets pointed at most often. Every symbol gets indexed once per worktree: in the repo I measured this on, 13,344 of 18,630 indexed files came from three worktrees, solmc findreturned each hit four times andimpactoverstated the blast radius by the same factor, mixing in stale branch states that no longer exist anywhere. Excluding it cut the index from 18,630 to 5,286 files..phpstan/is PHPStan's cache. ItsresultCache.phpreaches double-digit MB (23 MB in this repo); php2cpg reads the parser output into memory in one go (IOUtils.contentFromBufferedSource) and dies withjava.lang.OutOfMemoryError, which surfaces asjoern-parse exit 1and a silent fallback to the tree-sitter index.To be precise about what this PR does and does not fix: excluding
.phpstanwas necessary but not sufficient in my repo. It also carries a vendored legacy tree with 1.5 MB font arrays that contributes to the same OOM, and that directory name is project-specific, so it has no business in a shared list. Measured:.phpstanonlySo this PR removes one of the two causes and the duplicate-indexing problem outright. The project-specific half is what #2 addresses, via an
exclude:key inlumos.yml— with both in place,lmc buildover 4,493 files completes the Joern CPG in 18 s instead of aborting.Existing tests stay green:
tests/test_joern_parse_excludes.pycounts--excludeagainstlen(DEFAULT_EXCLUDES), so the new entries do not break it.