Per-project exclude: key in lumos.yml - #2
Open
Braindea7 wants to merge 2 commits into
Open
Conversation
Two directory kinds slipped past DEFAULT_EXCLUDES/IGNORE_DIRS and broke the build on a Laravel monorepo: - `.worktrees/` holds full second checkouts (the layout `git worktree add` gets pointed at most often). Every symbol was indexed once per worktree — in the repo this was measured on, 13,344 of 18,630 indexed files came from three worktrees, so `lmc find` returned every hit four times and `impact` overstated the blast radius by the same factor, mixing in stale branch states that no longer exist anywhere. - `.phpstan/` is PHPStan's cache. Its `resultCache.php` reaches double-digit MB (23 MB here); php2cpg reads the parser output into memory in one go (IOUtils.contentFromBufferedSource) and dies with java.lang.OutOfMemoryError. Measured on the same repo, `lmc build` on 4,493 files: previously it aborted with `joern-parse exit 1` and fell back to the tree-sitter index, now the Joern CPG completes in 18 s.
DEFAULT_EXCLUDES/IGNORE_DIRS cover dependency and build directories, which are
the same everywhere. What isn't the same everywhere is a directory that only
hurts in one repo — a vendored legacy tree, a generated-code drop. Putting such
a name into the shared lists means every other project carries it too.
# lumos.yml
language: php
codebase_hash: d870a6000cbcd0ae
exclude:
- legacy_app
The key is read once per build and applies to both engines: `build_index()`
adds it to _IGNORE, `joern_parse()` passes it to the frontend as `--exclude`,
de-duplicated against the defaults. No config, no key, or an empty list keeps
the previous behaviour exactly.
Known limitation, documented in `joern_parse`: with `--scope` the mounted path
is the subtree, so a lumos.yml at the worktree root is not seen and only
DEFAULT_EXCLUDES apply.
tests/test_config_excludes.py covers reading the key, the frontend command, the
de-duplication, and — with a real tree-sitter parse — that a configured
directory actually disappears from the index.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #1 — GitHub does not allow a fork branch as the base, so the diff here currently shows both commits. Once #1 is merged this shrinks to the single commit
feat(config): per-project exclude: key in lumos.yml. Review that one.DEFAULT_EXCLUDES/IGNORE_DIRScover dependency and build directories, which are the same everywhere. What is not the same everywhere is a directory that only hurts in one repo — a vendored legacy tree, a generated-code drop. Putting such a name into the shared lists means every other project carries it too. In my case it was a legacy PHP tree whose 1.5 MB font arrays kill php2cpg withjava.lang.OutOfMemoryError; the name means nothing to anyone else.The key is read once per build and applies to both engines:
build_index()adds it to_IGNORE,joern_parse()passes it to the frontend as--exclude, de-duplicated against the defaults. No config, no key, or an empty list keeps the previous behaviour exactly.Known limitation, documented in
joern_parse: with--scopethe mounted path is the subtree, so alumos.ymlat the worktree root is not seen and onlyDEFAULT_EXCLUDESapply.tests/test_config_excludes.pycovers reading the key, slash normalisation, the frontend command, de-duplication against the defaults, and — with a real tree-sitter parse — that a configured directory actually disappears from the index. That last one asserts the baseline first, so it fails if the key silently does nothing.Verified end to end on a Laravel monorepo (4,493 files):
lmc buildproduces an identical index either way, and the Joern CPG now completes in 18 s where it previously aborted.