Skip to content

indexer: extend typed reference edges to JS/TS/TSX/Go/Java/Rust (#85) - #92

Merged
IceRhymers merged 1 commit into
integration/knowledge-graph-reference-edgesfrom
feat/85-multi-language-edges
Jul 23, 2026
Merged

indexer: extend typed reference edges to JS/TS/TSX/Go/Java/Rust (#85)#92
IceRhymers merged 1 commit into
integration/knowledge-graph-reference-edgesfrom
feat/85-multi-language-edges

Conversation

@IceRhymers

Copy link
Copy Markdown
Owner

Summary

Part of #82 (umbrella #89, still draft). Implements #85: extends the typed call/import reference-edge extraction landed for Python in #84 (PR #91) to the remaining six supported languages — JavaScript, TypeScript, TSX, Go, Java, Rust — so all 7 SYMBOL_KINDS languages now emit reference_edges from the same single-parse/single-walk pipeline.

Caveat on Closes #85: this covers the issue's three acceptance checkboxes for all 7 languages. GitHub will auto-close #85 on merge; if the issue should stay open until the full epic (#82) lands, strip the closing keyword before merge.

What changed

  • indexer/languages.py — six new EDGE_NODE_KINDS entries: JS/TS/TSX (call_expression, new_expressioncall; import_statementimport), Go (call_expressioncall; import_specimport, mapped per-spec rather than per-declaration so grouped imports get per-spec line anchors), Java (method_invocation, object_creation_expressioncall; import_declarationimport), Rust (call_expressioncall; use_declarationimport).
  • indexer/symbols.py — one _<lang>_call_edge / _<lang>_import_edges pair per language (JS/TS/TSX share one pair — identical grammar shapes), plus _EDGE_EXTRACTORS, a dict[str, tuple[call_fn, import_fn]] dispatch table. extract_file resolves the pair once per file (not per node) right after the combined symbol/edge map lookup, replacing the hardcoded Python call sites.
    • Calls: rightmost-name resolution per grammar — JS/TS/TSX member_expression.property (a.b.f()/obj?.m()f/m; optional chaining is transparent); Go selector_expression.field; Java method_invocation.name (object/type-argument-independent) plus a dedicated _java_type_name helper for object_creation_expression, since that node has no simple name field — it descends through generic_type and takes the rightmost type_identifier of a scoped_type_identifier (new java.util.ArrayList<String>()ArrayList); Rust scoped_identifier.name / field_expression.field.
    • Imports: JS/TS/TSX cover every specifier shape (default, named incl. aliased, namespace, side-effect, mixed, TS import type, TS import x = require(...)), each anchored per-specifier where the grammar allows it. Go per-import_spec anchoring handles single and grouped forms, including dot/blank aliases (target stays the package path) and empty import () (zero edges, no special-casing). Java covers plain/static/wildcard forms. Rust use is handled by _rust_use_tree_edges, a recursive descent over the use-tree (scoped, grouped, nested groups, prefix-less bare groups, self, wildcard, as-renamed) that accumulates a ::-joined prefix and anchors each emitted edge at its own leaf node.
    • macro_invocation (Rust), dynamic import(...) (JS/TS/TSX), and export ... from re-exports are deliberately unmapped/skipped — not calls or imports in this schema.
  • app/db/models.pyINDEX_SEMANTICS_VERSION bumps 3 -> 4 with a docstring entry: every already-indexed branch re-indexes once on its next run to backfill edges for the six new languages.
  • tests/unit/test_edges.py — per-language shape fixtures (calls: bare/dotted/optional-chain/constructor/generic/static/macro-negative; imports: every specifier form per grammar, incl. negatives — re-export, dynamic import, empty Go import group, Rust macro) and enclosing-attribution fixtures (function/method/class/module scope per language). Split the old combined Python/JS smoke test into a lang=None case and a real JS positive fixture.
  • tests/unit/test_languages.py — three mandatory parity/invariant guards: every SYMBOL_KINDS language has an EDGE_NODE_KINDS entry, every SYMBOL_KINDS language has an _EDGE_EXTRACTORS entry (a missing one is a runtime KeyError for every file of that language), and the symbol/edge node-type sets are disjoint per language (guards the lossless-merge assumption _combined_kinds depends on).

Out of scope (per the epic's binding plan)

indexer/store.py (writes FileExtraction.edges generically already), DB schema/migrations, resolver (#86), MCP tools (#87), Web UI, lexical query grammar/SYMBOL_KINDS, grants.

Grammar-shape verification

Every node-type/field-name assumption below (constructor vs function, Go's interpreted_string_literal_content, Rust's path/list/argument fields, Java's fieldless scoped_type_identifier, TS's import_require_clause.source, etc.) was verified against live tree-sitter-language-pack==1.13.3 parse trees before being coded, not assumed from grammar familiarity — every fixture in the table below passed on the first attempt.

Test plan

  • make lint — ruff check + format --check + mypy (app, indexer, webui) — clean.

  • make test (pytest -m "unit or observability") — 987 passed (previously 984; +3 from a fix-round described below), including 96 tests across test_edges.py/test_languages.py.

  • make test-integrationdeferred to CI, not run locally. This project is Lakebase-only (no local/CI Postgres image; see README.md "Local development" and docs/runbooks/ci-lakebase.md), and the integration suite needs a live ephemeral Lakebase branch via scripts/ci_branch.py. Checked ci-lakebase.yml's run history on the last several related PRs ([Epic] Knowledge graph: typed reference edges in Lakebase with agent-facing graph tools #82 umbrella, db: add raw reference-edge schema with lifecycle-safe grants #83, indexer: emit typed call/import edges with enclosing-symbol attribution (Python) #84) — every run is skipped, because vars.CI_LAKEBASE_ENABLED isn't provisioned yet (a pre-existing gap, unrelated to this change; the prerequisites in docs/runbooks/ci-lakebase.md are still unmet). This diff touches no schema/store/DB-write code — only extraction — so the risk this defers is low, but it is a real gap the CI Lakebase job should close once provisioned.

  • Independent pre-PR code review (separate agent pass, not self-approved): one MEDIUM finding, fixed. _rust_use_tree_edges didn't handle a bare, prefix-less use_list argument (use {std::io, std::fmt}; — valid per the Rust reference grammar's UseTree production, distinct from scoped_use_list), silently dropping to zero edges. Added the missing branch plus a regression fixture (test_rust_use_bare_prefix_less_group). Two non-blocking observations were also addressed: added test_rust_import_attributes_to_enclosing_function (Rust use can appear inside a function body, unlike JS/Go/Java's file-top-level-only imports) and test_javascript_default_and_namespace_import_both_target_the_module, which documents (rather than "fixes") that import d, * as ns from 'm' emits two identical module-target edges — both clauses are module-class bindings with no name-class target to disambiguate, so this is intentional, not a dedup bug.

  • JS/TS-heavy benchmark (A9, manual timing, not committed — per indexer: emit typed call/import edges with enclosing-symbol attribution (Python) #84's precedent, no repo benchmark harness): extract_file (with edges) vs a walk-only baseline (edge branch stubbed, same tree walk and symbol extraction), best-of-5 time.perf_counter, over real corpora per language (this repo's webui/frontend/src for TS/TSX — 21 files, 2654 lines; Go/Java/Rust standard-library sources — /usr/lib/go/src, the JDK's src.zip, and rustup component add rust-src's library/ — since this repo has no Go/Java/Rust source of its own; JS shares the TS/TSX code path exactly, so no separate corpus was needed):

    Language Files Edges Delta
    Python 34 2010 +12.9%
    TypeScript 8 152 +10.1%
    TSX 13 276 +14.2%
    Go 39 4571 +12.5%
    Java 39 767 +9.3%
    Rust 33 1748 +9.3%
    Overall +11.5%
    JS/TS combined +12.5%

    All languages, and the JS/TS delta specifically (the one the epic's ≤~15% budget is keyed to), land comfortably under budget — consistent with the analytic argument that added cost is a bounded number of child_by_field_name/text.decode calls per edge node, not a change to the walk itself.

Acceptance-criteria mapping (issue #85)

# Criterion Proven by
1 All 7 languages emit typed call/import edges with enclosing-symbol attribution EDGE_NODE_KINDS entries for the 6 new languages + existing Python; test_edges.py shape fixtures per language; enclosing-attribution fixtures per language (function/method/class/module scope)
2 Parity test guards SYMBOL_KINDS/edge-map coverage test_languages.py::test_every_symbol_language_has_an_edge_map, ::test_every_symbol_language_has_an_edge_extractor, ::test_symbol_and_edge_node_types_are_disjoint_per_language
3 Fixtures cover each grammar's call and import shapes test_edges.py per-language shape matrix (bare/dotted/optional-chain/constructor/generic calls; every import specifier form per grammar) plus negative fixtures (dynamic import, re-export, empty Go import group, Rust macro, bare prefix-less use group)

Not merging — left ready for orchestrator CI verification and merge.

Adds per-language call/import extractors (JS/TS/TSX share one pair; Go,
Java, and Rust each get their own) and a dispatch table so extract_file
resolves the pair once per file instead of hardcoding Python. Rust use
declarations are handled by a recursive use-tree descent to cover
scoped, grouped, nested, self, and wildcard forms; Java constructor
targets resolve generic and scoped type names since object_creation_expression
has no simple name field. Bumps INDEX_SEMANTICS_VERSION 3 -> 4 so every
already-indexed branch re-indexes once and backfills the new languages.
@IceRhymers
IceRhymers merged commit 0ff53f4 into integration/knowledge-graph-reference-edges Jul 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant