indexer: emit typed Python call/import edges with enclosing attribution (#84) - #91
Merged
IceRhymers merged 1 commit intoJul 23, 2026
Conversation
…on (#84) extract_file() extends the existing tree-sitter symbol walk in indexer/symbols.py to also emit call/import reference edges in the same single pass (indexer/languages.py's new EDGE_NODE_KINDS map, Python-only for now). Call targets resolve to the rightmost identifier of the callee; import targets are the full dotted path as written, alias-insensitive, with source-faithful relative-import and wildcard handling. Each edge attributes to the innermost named enclosing definition on the walk stack, computed in O(1) with no second walk. indexer/store.py's index_repo writes reference_edges exactly like symbols: an unconditional per-file delete followed by a bulk reinsert inside the same per-(repo, branch) transaction, so a file whose edges all vanish still sheds its stale rows. IndexCounts gains an edges count; indexer/job.py switches to extract_file and logs it. INDEX_SEMANTICS_VERSION bumps 2 -> 3 so every already-indexed branch re-indexes once to backfill reference_edges.
IceRhymers
merged commit Jul 23, 2026
3443fee
into
integration/knowledge-graph-reference-edges
4 checks passed
This was referenced Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #82 (umbrella #89, still draft). Implements #84: the indexer now extracts typed Python call/import reference edges alongside symbols, in a single tree-sitter walk, and writes them into the
reference_edgestable added by #83 (#90).Caveat on
Closes #84: this covers acceptance criteria 1–4 for Python only, matching the epic's delivery sequence — other languages are #85, query-time resolution is #86, MCP/UI are #87/#88. GitHub will auto-close #84 on merge; if the issue is meant to stay open until the epic fully lands, strip the closing keyword before merge.What changed
indexer/languages.py— newEDGE_NODE_KINDSmap (Python:call,import_statement,import_from_statement), new frozen dataclassesExtractedEdge/FileExtraction;IndexCountsgains anedges: intfield.indexer/symbols.py— newextract_file(pf) -> FileExtraction: one parse, one walk, emitting both symbols and edges.extract_symbolsis now a thin wrapper (extract_file(pf).symbols) so the existing unit-test surface and callers are untouched.f()/a.b.f()/self.f()→f); callees with no rightmost identifier (xs[0](), the outer call off()()) are skipped, never crashed.import a.b.c as d→a.b.c); relative imports preserve source fidelity (from . import x→.x,from ..p import q→..p.q); wildcard imports (from a.b import *) emit one edge for the module.None= module scope.node.type -> (tag, value)cache collapses the symbol-map and edge-map lookups into one dict.get()per node, and the walk uses two parallel stacks (node_stack/enclosing_stack) instead of one stack of(node, enclosing)tuples —[enclosing] * len(children)is a single C-level list replication instead of N per-child tuple allocations. See "Performance" below.indexer/store.py—index_repowritesreference_edgesexactly likesymbols: an unconditional per-fileDELETEfollowed by a bulk reinsert, inside the same per-(repo, branch)transaction — a file whose edges all vanish still sheds its stale rows.itemsretypes toIterable[tuple[ParsedFile, FileExtraction]].indexer/job.py— both generator sites switch toextract_file; the per-branch success log line gains an edge count.app/db/models.py—INDEX_SEMANTICS_VERSIONbumps2 -> 3(one deliberate change) with a docstring entry: every already-indexed branch re-indexes once on its next run to backfillreference_edges.docs/runbooks/reference-edges.md§1 rewritten (writer semantics, no longer "dormant");indexer/AGENTS.mdandtests/unit/AGENTS.mdkey-file rows updated;app/db/AGENTS.md's stale "currently 2" note corrected.Out of scope (per the epic's binding plan)
Other languages (#85), query-time resolution/service payloads (#86), MCP tools (#87), Web UI (#88), any resolved
symbol_idpersistence, cross-file joins at write time, uniqueness/dedup of edge rows, query-grammar changes, new migrations, or grant changes (schema-wide grants from #83 already cover DML onreference_edges).Test plan
make lint— ruff check + format + mypy (app, indexer, webui) — clean.make test(pytest -m "unit or observability") — 922 passed, including newtests/unit/test_edges.py(19 tests: bare/nested/method/dotted calls, decorator-with/without-args, non-identifier callees, all import forms incl. relative/wildcard/multi-line, enclosing attribution across function/method/class/module scope, non-Python languages yield no edges, wrapper equivalence, determinism) andtests/unit/test_languages.pyadditions (EDGE_NODE_KINDS⊆EXT_TO_LANG, edge kinds within the DB CHECK set).make test-integrationagainst local Postgres (pgvector/pgvector:pg16, same image as CI's non-Lakebase path) — the reference-edges-relevant suites pass in full:tests/integration/test_store.py(21/21, including 4 new tests: writer proves correct row contents incl. enclosing columns, stale-edge replacement on re-index, no duplication on idempotent re-index, zero-edge shedding) andtests/integration/test_job_reconcile.py(7/7, exercises the realextract_file/index_repopipeline end to end).tests/integration/test_reconcile.pyandtests/integration/test_store_chunk_writer.pywere updated for the newFileExtractionseam but could not be run locally — their fixtures require the Databricks-managedlakebase_vector/lakebase_tokenizerextensions, which don't exist in a stock Postgres image (confirmed this is a pre-existing local-environment gap, not a regression: the same fixture setup fails identically onf33ac6e, before this branch's changes). CI's real-Lakebase job (ci-lakebase.yml) is this project's only environment that can validate them._combined_kindsdegrades safely for a symbol-only language, no SQL injection (all writes parameterized viapg_insert), enclosing denormalization can't diverge fromsymbolswithin a file (sameExtractedSymbolobject, same transaction), and the new integration tests genuinely prove what their names claim (e.g. the stale-edge test confirms the OLD target is gone, not just that the new one exists).Performance (acceptance criterion: ≤ ~15% of parse+walk)
Throwaway local benchmark (not committed, not a CI gate — a perf assertion would be flaky by design): median-of-5
time.perf_countertiming of the pre-changeextract_symbolsvs the newextract_file, over all 97 tracked.pyfiles in this repo plus one 2000-function synthetic file. Measured ~12–13.5% overhead across several runs (parse time, which is identical in both paths, included in the denominator) — under the ≤15% target. The first implementation attempt (two separateSYMBOL_KINDS/EDGE_NODE_KINDSdict lookups, one stack of(node, enclosing)tuples) measured 54% overhead; the merged-lookup + two-stack optimization above brought it back under target without changing extraction semantics (all fixture tests still pass byte-identical).Acceptance-criteria mapping (issue #84)
tests/unit/test_edges.pyfixture matrix;tests/integration/test_store.py::test_indexing_writes_correct_reference_edge_rows(full row contents incl. enclosing columns)test_reindex_replaces_stale_edges_for_the_same_file,test_reindex_with_identical_items_does_not_duplicate_edges,test_reindex_to_zero_edges_sheds_all_rows, updatedtest_mark_and_sweep_removes_deleted_file(now uses the real writer, not a hand-seeded row)INDEX_SEMANTICS_VERSIONbump2 -> 3, one line, docstring entry;tests/unit/test_semantics_version_tripwire.pypasses by constructionNot merging — left ready for orchestrator CI verification and merge.