Skip to content

indexer: emit typed Python call/import edges with enclosing attribution (#84) - #91

Merged
IceRhymers merged 1 commit into
integration/knowledge-graph-reference-edgesfrom
feat/84-python-reference-edges
Jul 23, 2026
Merged

indexer: emit typed Python call/import edges with enclosing attribution (#84)#91
IceRhymers merged 1 commit into
integration/knowledge-graph-reference-edgesfrom
feat/84-python-reference-edges

Conversation

@IceRhymers

Copy link
Copy Markdown
Owner

Summary

Part of #82 (umbrella #89, still draft). Implements #84: the indexer now extracts typed Python call/import reference edges alongside symbols, in a single tree-sitter walk, and writes them into the reference_edges table added by #83 (#90).

Caveat on Closes #84: this covers acceptance criteria 1–4 for Python only, matching the epic's delivery sequence — other languages are #85, query-time resolution is #86, MCP/UI are #87/#88. GitHub will auto-close #84 on merge; if the issue is meant to stay open until the epic fully lands, strip the closing keyword before merge.

What changed

  • indexer/languages.py — new EDGE_NODE_KINDS map (Python: call, import_statement, import_from_statement), new frozen dataclasses ExtractedEdge / FileExtraction; IndexCounts gains an edges: int field.
  • indexer/symbols.py — new extract_file(pf) -> FileExtraction: one parse, one walk, emitting both symbols and edges. extract_symbols is now a thin wrapper (extract_file(pf).symbols) so the existing unit-test surface and callers are untouched.
    • Call target = rightmost identifier of the callee (f()/a.b.f()/self.f()f); callees with no rightmost identifier (xs[0](), the outer call of f()()) are skipped, never crashed.
    • Import target = full dotted path as written, alias-insensitive (import a.b.c as da.b.c); relative imports preserve source fidelity (from . import x.x, from ..p import q..p.q); wildcard imports (from a.b import *) emit one edge for the module.
    • Enclosing attribution = innermost named enclosing definition on the walk stack, computed in O(1) with no second walk and no write-time join; None = module scope.
    • Perf: a merged per-language node.type -> (tag, value) cache collapses the symbol-map and edge-map lookups into one dict .get() per node, and the walk uses two parallel stacks (node_stack / enclosing_stack) instead of one stack of (node, enclosing) tuples — [enclosing] * len(children) is a single C-level list replication instead of N per-child tuple allocations. See "Performance" below.
  • indexer/store.pyindex_repo writes reference_edges exactly like symbols: an unconditional per-file DELETE followed by a bulk reinsert, inside the same per-(repo, branch) transaction — a file whose edges all vanish still sheds its stale rows. items retypes to Iterable[tuple[ParsedFile, FileExtraction]].
  • indexer/job.py — both generator sites switch to extract_file; the per-branch success log line gains an edge count.
  • app/db/models.pyINDEX_SEMANTICS_VERSION bumps 2 -> 3 (one deliberate change) with a docstring entry: every already-indexed branch re-indexes once on its next run to backfill reference_edges.
  • Docsdocs/runbooks/reference-edges.md §1 rewritten (writer semantics, no longer "dormant"); indexer/AGENTS.md and tests/unit/AGENTS.md key-file rows updated; app/db/AGENTS.md's stale "currently 2" note corrected.

Out of scope (per the epic's binding plan)

Other languages (#85), query-time resolution/service payloads (#86), MCP tools (#87), Web UI (#88), any resolved symbol_id persistence, cross-file joins at write time, uniqueness/dedup of edge rows, query-grammar changes, new migrations, or grant changes (schema-wide grants from #83 already cover DML on reference_edges).

Test plan

  • make lint — ruff check + format + mypy (app, indexer, webui) — clean.
  • make test (pytest -m "unit or observability") — 922 passed, including new tests/unit/test_edges.py (19 tests: bare/nested/method/dotted calls, decorator-with/without-args, non-identifier callees, all import forms incl. relative/wildcard/multi-line, enclosing attribution across function/method/class/module scope, non-Python languages yield no edges, wrapper equivalence, determinism) and tests/unit/test_languages.py additions (EDGE_NODE_KINDSEXT_TO_LANG, edge kinds within the DB CHECK set).
  • make test-integration against local Postgres (pgvector/pgvector:pg16, same image as CI's non-Lakebase path) — the reference-edges-relevant suites pass in full: tests/integration/test_store.py (21/21, including 4 new tests: writer proves correct row contents incl. enclosing columns, stale-edge replacement on re-index, no duplication on idempotent re-index, zero-edge shedding) and tests/integration/test_job_reconcile.py (7/7, exercises the real extract_file/index_repo pipeline end to end). tests/integration/test_reconcile.py and tests/integration/test_store_chunk_writer.py were updated for the new FileExtraction seam but could not be run locally — their fixtures require the Databricks-managed lakebase_vector/lakebase_tokenizer extensions, which don't exist in a stock Postgres image (confirmed this is a pre-existing local-environment gap, not a regression: the same fixture setup fails identically on f33ac6e, before this branch's changes). CI's real-Lakebase job (ci-lakebase.yml) is this project's only environment that can validate them.
  • Independent pre-PR code review (separate agent pass, not self-approved): no CRITICAL/HIGH findings. Verified: two-stack lockstep can't desync (enclosing attribution can't misassign), _combined_kinds degrades safely for a symbol-only language, no SQL injection (all writes parameterized via pg_insert), enclosing denormalization can't diverge from symbols within a file (same ExtractedSymbol object, same transaction), and the new integration tests genuinely prove what their names claim (e.g. the stale-edge test confirms the OLD target is gone, not just that the new one exists).

Performance (acceptance criterion: ≤ ~15% of parse+walk)

Throwaway local benchmark (not committed, not a CI gate — a perf assertion would be flaky by design): median-of-5 time.perf_counter timing of the pre-change extract_symbols vs the new extract_file, over all 97 tracked .py files in this repo plus one 2000-function synthetic file. Measured ~12–13.5% overhead across several runs (parse time, which is identical in both paths, included in the denominator) — under the ≤15% target. The first implementation attempt (two separate SYMBOL_KINDS/EDGE_NODE_KINDS dict lookups, one stack of (node, enclosing) tuples) measured 54% overhead; the merged-lookup + two-stack optimization above brought it back under target without changing extraction semantics (all fixture tests still pass byte-identical).

Acceptance-criteria mapping (issue #84)

# Criterion Proven by
1 Typed call/import edges with correct enclosing-symbol attribution and line numbers tests/unit/test_edges.py fixture matrix; tests/integration/test_store.py::test_indexing_writes_correct_reference_edge_rows (full row contents incl. enclosing columns)
2 Edge extraction overhead ≤ ~15% of parse+walk Single-walk design + merged-lookup/two-stack optimization; local benchmark above (~12–13.5%)
3 Delete-and-reinsert + sweep proven by tests test_reindex_replaces_stale_edges_for_the_same_file, test_reindex_with_identical_items_does_not_duplicate_edges, test_reindex_to_zero_edges_sheds_all_rows, updated test_mark_and_sweep_removes_deleted_file (now uses the real writer, not a hand-seeded row)
4 Single deliberate INDEX_SEMANTICS_VERSION bump 2 -> 3, one line, docstring entry; tests/unit/test_semantics_version_tripwire.py passes by construction

Not merging — left ready for orchestrator CI verification and merge.

…on (#84)

extract_file() extends the existing tree-sitter symbol walk in
indexer/symbols.py to also emit call/import reference edges in the same
single pass (indexer/languages.py's new EDGE_NODE_KINDS map, Python-only
for now). Call targets resolve to the rightmost identifier of the callee;
import targets are the full dotted path as written, alias-insensitive,
with source-faithful relative-import and wildcard handling. Each edge
attributes to the innermost named enclosing definition on the walk stack,
computed in O(1) with no second walk.

indexer/store.py's index_repo writes reference_edges exactly like
symbols: an unconditional per-file delete followed by a bulk reinsert
inside the same per-(repo, branch) transaction, so a file whose edges
all vanish still sheds its stale rows. IndexCounts gains an edges count;
indexer/job.py switches to extract_file and logs it.

INDEX_SEMANTICS_VERSION bumps 2 -> 3 so every already-indexed branch
re-indexes once to backfill reference_edges.
@IceRhymers
IceRhymers merged commit 3443fee into integration/knowledge-graph-reference-edges Jul 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant