Skip to content

graph graph_store

Jump to bottom
aakash-anko edited this page Jun 20, 2026 · 2 revisions

graph_store.py

Persistent graph storage backed by DuckDB. Stores file dependencies, function/class metadata, module groupings, chunk mappings, and function-level call edges. Lives at .codewalk/graph.duckdb inside each analyzed repo.

Key Concepts

Term Definition Example
edge A connection between two vertices in a graph, representing a relationship (e.g., an import). If pipeline.py imports scanner.py, there's a directed edge pipeline.py → scanner.py.
embedding A numerical vector (list of numbers) that represents the meaning of text. Similar text → similar vectors. The code def add(a, b): return a+b might become [0.12, -0.45, 0.78, ...] (768 numbers for jinaai/jina-code-embeddings-1.5b via Ollama/MPS).
vector store A database optimized for storing embeddings and finding the most similar ones quickly. ChromaDB stores code chunk embeddings and returns the 5 most similar chunks to your query.
ChromaDB An open-source vector database for storing and searching embeddings. Used here to store code chunks. collection.query(query_texts=["scan files"], n_results=5) returns the 5 closest code chunks.
DuckDB An embedded SQL database (like SQLite but optimized for analytics). Used here to store file metadata and import edges. conn.execute("SELECT path FROM files WHERE language='python'").fetchall() returns all Python file paths.
chunk A piece of source code (usually one function or class) stored as a unit for search. The function def scan_directory(root): ... (20 lines) is one chunk.
parent chunk A larger code chunk (e.g., a whole class) that contains smaller child chunks (its methods). Used for context expansion. Class GraphRuntime is a parent chunk; its method get_blast_radius() is a child chunk.
AST Abstract Syntax Tree — a tree representation of source code structure, where each node is a language construct (function, class, if-statement, etc.). def add(a, b): return a+b becomes a tree: FunctionDef → [args: a, b] → [body: Return → BinOp(a + b)].
tree-sitter A fast, multi-language parser that builds ASTs. Supports 100+ languages without needing each language's compiler. tree-sitter parses config.py into an AST, then we extract function/class nodes from it.
RAG Retrieval-Augmented Generation — instead of asking an LLM to answer from memory, first retrieve relevant documents, then include them in the prompt. Question: "What does scan_directory do?" → retrieve the source code of scan_directory → include it in the LLM prompt → get an accurate answer.
hunk A contiguous block of changes within a diff. One diff can contain multiple hunks (changes in different parts of a file). A diff might have hunk 1 (lines 10-15 changed) and hunk 2 (lines 80-85 changed).

Module-level function: _stable_id(*parts: str) -> str (Line 9)

Creates a deterministic 16-character hex ID by SHA-256 hashing the input parts joined with |.

Example

Input: _stable_id("src/config.py")

Line 11 — "|".join(parts) joins all parts with |:

"src/config.py"

Line 11 — .encode() converts to bytes:

b"src/config.py"

Line 11 — hashlib.sha256(...) hashes those bytes and .hexdigest()[:16] takes the first 16 hex chars:

"a1b2c3d4e5f6a7b8"   # (example — actual value depends on SHA-256)

Returns: "a1b2c3d4e5f6a7b8" — a deterministic 16-char hex string. Same input always produces the same ID.

Class: GraphStore (Line 15)

Wraps a DuckDB database with 7 tables: files, imports, symbols, symbol_calls, chunks, modules, module_deps. All graph queries flow through this class.

__init__(self, db_path: str = ".codewalk/graph.duckdb") (Line 29)

Opens (or creates) a DuckDB database at the given path and creates all tables if they don't exist.

Example

Input: GraphStore("/tmp/repo/.codewalk/graph.duckdb")

Line 30 — self.db_path = db_path:

self.db_path = "/tmp/repo/.codewalk/graph.duckdb"

Line 31 — Path(db_path).parent.mkdir(parents=True, exist_ok=True): Creates the directory /tmp/repo/.codewalk/ if it doesn't exist.

Line 32 — self.conn = duckdb.connect(db_path): Opens a DuckDB connection to /tmp/repo/.codewalk/graph.duckdb. Creates the file if it doesn't exist.

Line 33 — self._create_tables(): Calls _create_tables() to ensure all 7 tables exist.

Returns: A GraphStore instance with self.db_path and self.conn set.

_create_tables(self) (Line 35)

Creates all 7 graph tables using CREATE TABLE IF NOT EXISTS, so it's safe to call on every startup without data loss.

Tables created:

Table Primary Key Purpose
files file_id (hash) One row per indexed file. Columns: path, module, language
imports (source_file_id, target_file_id) File-level import edges
symbols symbol_id (hash) Functions/classes with name, file, line range
symbol_calls (caller_symbol_id, callee_symbol_id, line) Function-calls-function edges
chunks chunk_id (hash) Embeddings chunk metadata linking to files/symbols
modules name Detected module names with file counts
module_deps (source, target) Module-level dependency edges

No return value — side effect is the 7 tables existing in DuckDB.

populate_from_analysis(self, files, deps, module_results, embedded_chunks=None) (Line 88)

Master method that populates all 7 tables from scan/analysis results. Clears existing data first, then delegates to private _populate_* methods.

Example

Input:
  files = [
    {"file_path": "config.py", "language": "python", "absolute_path": "/repo/config.py"},
    {"file_path": "main.py", "language": "python", "absolute_path": "/repo/main.py"}
  ]
  deps = {"graph": {"main.py": ["config.py"]}}
  module_results = {
    "modules": {"core": {"files": ["config.py", "main.py"], "file_count": 2}},
    "module_graph": {}
  }
  embedded_chunks = None

Lines 104–112 — Because embedded_chunks is None, delete ALL chunks (full rebuild). Then delete symbol_calls, symbols, imports, module_deps, modules, files — in reverse FK order so no foreign key violations:

All 7 tables now have 0 rows

Line 114 — self._populate_files(files, module_results): Inserts 2 rows into files table with hash IDs.

Line 115 — self._populate_imports(deps): Inserts 1 row into imports: main.py → config.py.

Line 116 — self._populate_symbols(files): Parses each file with tree-sitter, inserts function/class rows into symbols.

Line 117 — self._populate_symbol_calls(files): Extracts call sites, resolves them against symbols, inserts into symbol_calls.

Line 118 — self._populate_modules(module_results): Inserts 1 row into modules ("core", file_count=2) and 0 rows into module_deps.

Line 119–120 — embedded_chunks is None, so _populate_chunks is skipped.

Line 121 — self._get_stats() returns:

{"files": 2, "imports": 1, "symbols": <n>, "symbol_calls": <n>, "chunks": 0, "modules": 1}

Line 123–126 — Logs: [GraphStore] Populated: 2 files, 1 imports, <n> symbols, 1 modules

Returns: None — side effect is all tables populated.

_populate_files(self, files, module_results) (Line 128)

Inserts one row per file into the files table. Each file gets a deterministic hash ID via _stable_id(file_path). The module column is looked up from module_results.

Example

Input:
  files = [{"file_path": "config.py", "language": "python"}]
  module_results = {"modules": {"core": {"files": ["config.py"]}}}

Lines 130–133 — Build file_to_module dict by iterating modules:

file_to_module = {"config.py": "core"}

Lines 135–145 — For each file, create a tuple:

(
  _stable_id("config.py"),     # "a1b2c3d4e5f6a7b8"
  "config.py",                 # path
  "core",                      # module (from file_to_module lookup)
  "python",                    # language
)

Line 135 — self.conn.executemany(...) inserts this tuple into files.

Returns: None — 1 row inserted into files.

_populate_imports(self, deps) (Line 148)

Inserts file-level import edges into the imports table. Only inserts edges where BOTH source and target files exist in the files table (prevents dangling references).

Example

Input: deps = {"graph": {"main.py": ["config.py", "missing.py"]}}
Assume files table contains: main.py (id="aaa"), config.py (id="bbb")

Line 150 — graph = deps.get("graph", {}):

graph = {"main.py": ["config.py", "missing.py"]}

Line 153 — Query all file_id values from the files table:

known_ids = {"aaa", "bbb"}

Lines 155–165 — Loop over graph edges:

  • source = "main.py", source_id = _stable_id("main.py") = "aaa" → in known_ids ✓
    • target = "config.py", target_id = _stable_id("config.py") = "bbb" → in known_ids ✓ → add ("aaa", "bbb") to rows
    • target = "missing.py", target_id = _stable_id("missing.py") = "ccc" → NOT in known_ids ✗ → skipped
rows = [("aaa", "bbb")]

Lines 167–170 — executemany inserts 1 edge into imports.

Returns: None — 1 row inserted.

_populate_symbols(self, files) (Line 172)

Parses each file with tree-sitter via code_parser.parse_file() and inserts function/class records into the symbols table.

Example

Input: files = [{"file_path": "config.py", "language": "python", "absolute_path": "/repo/config.py"}]
Assume config.py contains: class Settings: ... (lines 5-20) and def load(): ... (lines 22-30)

Lines 183–185 — Check file["language"] against GRAMMAR_MAP. "python" is in the map, so continue.

Line 186 — file_id = _stable_id("config.py") → "bbb"

Line 187 — read_path = "/repo/config.py" (from absolute_path)

Line 189 — parse_file("/repo/config.py", "python") returns:

[
  {"name": "Settings", "type": "class", "start_line": 5, "end_line": 20},
  {"name": "load", "type": "function", "start_line": 22, "end_line": 30}
]

Lines 190–200 — For each parsed item, build a tuple. For Settings:

qualified_name = "config.py:Settings"
symbol_id = _stable_id("config.py:Settings", "config.py", "5") → "x1y2z3..."
row = ("x1y2z3...", "Settings", "config.py:Settings", "bbb", "class", 5, 20)

Lines 203–208 — executemany inserts 2 rows into symbols.

Returns: None — 2 rows inserted into symbols.

_populate_symbol_calls(self, files) (Line 230)

Resolves call sites extracted by call_extractor.extract_calls_batch() against the symbols table and inserts resolved edges into symbol_calls.

Example

Input: files = [{"file_path": "main.py", "language": "python", ...}]
Assume call_extractor returns:
  [{"caller": "main.py:run", "callee_name": "load", "line": 10}]
Assume symbols table has:
  ("s1", "main.py:run", "run", "main.py")
  ("s2", "config.py:load", "load", "config.py")

Line 233 — extract_calls_batch(files) returns:

all_calls = [{"caller": "main.py:run", "callee_name": "load", "line": 10}]

Lines 237–243 — Build two lookup dicts from the symbols table:

symbol_by_qname = {"main.py:run": "s1", "config.py:load": "s2"}
symbols_by_name = {"run": [("s1", "main.py")], "load": [("s2", "config.py")]}

Lines 249–251 — For the first call: caller_qname = "main.py:run", callee_name = "load", line = 10.

Line 253 — caller_id = symbol_by_qname.get("main.py:run") → "s1" ✓

Line 257 — caller_file = "main.py:run".rsplit(":", 1)[0] → "main.py"

Line 258 — candidates = symbols_by_name.get("load", []) → [("s2", "config.py")]

Lines 261–264 — Try same-file match first: fpath == "main.py"? No ("config.py" ≠ "main.py"). No same-file match found.

Line 266 — Fallback: callee_id = candidates[0][0] → "s2"

Line 271 — Append ("s1", "s2", 10) to rows. resolved = 1.

Lines 273–277 — executemany inserts 1 row: run calls load at line 10.

Returns: None — logs [GraphStore] Symbol calls: 1 resolved, 0 unresolved.

_populate_modules(self, module_results) (Line 286)

Inserts module records into modules table and module-to-module dependency edges into module_deps.

Example

Input: module_results = {
  "modules": {"core": {"files": ["config.py"], "file_count": 1}},
  "module_graph": {"api": ["core"]}
}

Line 288 — modules = {"core": {"files": ["config.py"], "file_count": 1}}

Line 289 — module_graph = {"api": ["core"]}

Lines 291–297 — executemany inserts into modules:

("core", 1)

Lines 299–302 — Build deps_row from module_graph:

deps_row = [("api", "core")]

Lines 304–307 — executemany inserts 1 row into module_deps.

Returns: None — 1 module + 1 dependency edge inserted.

get_import_edges(self) -> list[tuple[str, str]] (Line 309)

Returns all file-level import edges as (source_path, target_path) tuples by joining imports with files to resolve hash IDs back to paths.

Example

Assume imports table has: (source_file_id="aaa", target_file_id="bbb")
Assume files table has: ("aaa", "main.py"), ("bbb", "config.py")

Lines 316–322 — SQL joins imports → files (twice: once for source, once for target):

SELECT sf.path, tf.path
FROM imports i
JOIN files sf ON i.source_file_id = sf.file_id
JOIN files tf ON i.target_file_id = tf.file_id

Returns: [("main.py", "config.py")]

get_module_dep_edges(self) -> list[tuple[str, str]] (Line 325)

Returns all module-level dependency edges as (source, target) tuples.

Example

Assume module_deps table has: ("api", "core")

Line 328 — SELECT source, target FROM module_deps

Returns: [("api", "core")]

get_module_file(self, file_path: str) -> str | None (Line 331)

Looks up which module a file belongs to by hashing the file path and querying the files table.

Example

Input: file_path = "config.py"

Line 333 — file_id = _stable_id("config.py") → "bbb"

Lines 334–336 — SELECT module FROM files WHERE file_id = "bbb" → ("core",)

Returns: "core"

If the file doesn't exist in the table, returns None.

get_files_in_module(self, module_name: str) -> list[str] (Line 339)

Returns all file paths belonging to a given module.

Example

Input: module_name = "core"
Assume files table has: ("bbb", "config.py", "core", "python"), ("ccc", "utils.py", "core", "python")

Lines 341–344 — SELECT path FROM files WHERE module = 'core'

Returns: ["config.py", "utils.py"]

get_symbols_in_file(self, file_path: str) -> list[dict] (Line 347)

Returns all symbols (functions/classes) in a file, ordered by start line.

Example

Input: file_path = "config.py"
Assume symbols table has Settings (line 5-20) and load (line 22-30)

Line 349 — file_id = _stable_id("config.py") → "bbb"

Lines 350–353 — SQL query returns rows ordered by start_line:

[
  ("x1y2z3", "Settings", "config.py:Settings", "class", 5, 20),
  ("a4b5c6", "load", "config.py:load", "function", 22, 30)
]

Lines 354–365 — Convert each row to a dict:

[
  {"symbol_id": "x1y2z3", "name": "Settings", "qualified_name": "config.py:Settings",
   "symbol_type": "class", "start_line": 5, "end_line": 20},
  {"symbol_id": "a4b5c6", "name": "load", "qualified_name": "config.py:load",
   "symbol_type": "function", "start_line": 22, "end_line": 30}
]

Returns: The list of 2 symbol dicts.

get_all_files(self) -> list[str] (Line 368)

Returns all file paths stored in the graph.

Example

Assume files table has: config.py, main.py, utils.py

Line 370–371 — SELECT path FROM files

Returns: ["config.py", "main.py", "utils.py"]

get_importers(self, file_path: str) -> list[str] (Line 374)

Reverse import lookup: which files import this file?

Example

Input: file_path = "config.py"
Assume imports table has: main.py → config.py, app.py → config.py

Line 376 — file_id = _stable_id("config.py") → "bbb"

Lines 377–381 — SQL joins imports → files where target_file_id = "bbb":

SELECT f.path FROM imports i
JOIN files f ON i.source_file_id = f.file_id
WHERE i.target_file_id = "bbb"

Returns: ["main.py", "app.py"]

get_imports(self, file_path: str) -> list[str] (Line 384)

Forward import lookup: which files does this file import?

Example

Input: file_path = "main.py"
Assume imports table has: main.py → config.py, main.py → utils.py

Line 386 — file_id = _stable_id("main.py") → "aaa"

Lines 387–391 — SQL joins imports → files where source_file_id = "aaa":

Returns: ["config.py", "utils.py"]

_get_stats(self) -> dict (Line 394)

Returns row counts for all 6 tables.

Example

Assume: files=2, imports=1, symbols=4, symbol_calls=2, chunks=0, modules=1

Lines 396–402 — Runs SELECT COUNT(*) on each table.

Returns:

{"files": 2, "imports": 1, "symbols": 4, "symbol_calls": 2, "chunks": 0, "modules": 1}

get_callers_of_symbol(self, qualified_name: str) -> list[dict] (Line 405)

Finds all symbols that call a given symbol. Returns caller name, qualified name, file, and the line of the call site.

Example

Input: qualified_name = "config.py:load"
Assume symbol_calls has: run (s1) calls load (s2) at line 10

Lines 412–415 — Look up symbol_id for "config.py:load":

callee_id = "s2"

Lines 417–424 — SQL joins symbol_calls → symbols → files where callee_symbol_id = "s2":

rows = [("run", "main.py:run", "main.py", 10)]

Lines 426–433 — Convert to dicts:

[{"caller": "run", "caller_qualified": "main.py:run", "file": "main.py", "line": 10}]

Returns: The list of 1 caller dict. Returns [] if the symbol doesn't exist.

get_callees_of_symbol(self, qualified_name: str) -> list[dict] (Line 436)

Finds all symbols that a given symbol calls. Returns callee name, qualified name, file, and line.

Example

Input: qualified_name = "main.py:run"
Assume symbol_calls has: run (s1) calls load (s2) at line 10, run (s1) calls save (s3) at line 15

Lines 438–441 — Look up symbol_id for "main.py:run":

caller_id = "s1"

Lines 442–448 — SQL joins symbol_calls → symbols → files where caller_symbol_id = "s1", ordered by line:

rows = [("load", "config.py:load", "config.py", 10), ("save", "config.py:save", "config.py", 15)]

Returns:

[
  {"callee": "load", "callee_qualified": "config.py:load", "file": "config.py", "line": 10},
  {"callee": "save", "callee_qualified": "config.py:save", "file": "config.py", "line": 15}
]

_populate_chunks(self, embedded_chunks) (Line 475)

Populates the chunks table from embedded chunk data produced by the embeddings pipeline. Maps each chunk to its file, symbol, and ChromaDB embedding ID.

Example

Input: embedded_chunks = [
  {"file_path": "config.py", "chunk_index": 0, "symbol_name": "Settings",
   "start_line": 5, "end_line": 20, "file_hash": "abc123"}
]
Assume symbols table has: ("x1y2z3", "Settings", "config.py") in the join result

Lines 493–497 — Build symbol_lookup from the symbols+files join:

symbol_lookup = {("config.py", "Settings"): "x1y2z3"}

Lines 502–504 — For the chunk: file_path = "config.py", chunk_index = 0, symbol_name = "Settings"

Line 506 — file_id = _stable_id("config.py") → "bbb"

Line 509 — symbol_id = symbol_lookup.get(("config.py", "Settings")) → "x1y2z3"

Line 511 — embedding_id = "config.py::chunk0"

Line 513 — chunk_id = _stable_id("config.py", "0") → "d4e5f6..."

Lines 515–522 — Append tuple:

("d4e5f6...", "bbb", "x1y2z3", 5, 20, "abc123", "config.py::chunk0")

Lines 524–529 — executemany inserts 1 row into chunks.

Returns: None — 1 chunk row inserted.

populate_chunks_from_chromadb(self, vector_store) -> int (Line 531)

Backfills the chunks table from ChromaDB metadata when DuckDB's chunks table is empty but ChromaDB has data (e.g. after server restart).

Example

Input: vector_store with parents_collection containing 3 chunks
Assume chunks table currently has 0 rows

Lines 541–542 — Check vector_store and vector_store.parents_collection are not None.

Lines 545–546 — SELECT COUNT(*) FROM chunks → 0. Table is empty, so proceed.

Lines 549–554 — Build symbol_lookup from symbols+files join (same as _populate_chunks).

Line 557 — vector_store.parents_collection.get(include=["metadatas"]) returns:

{"metadatas": [
  {"file_path": "config.py", "chunk_index": 0, "symbol_name": "Settings", "file_hash": "abc123", "start_line": 5, "end_line": 20},
  {"file_path": "config.py", "chunk_index": 1, "symbol_name": "", "file_hash": "abc123", "start_line": 21, "end_line": 40},
  {"file_path": "main.py", "chunk_index": 0, "symbol_name": "run", "file_hash": "def456", "start_line": 1, "end_line": 15}
]}

Lines 560–580 — For each metadata dict, build the same tuple format as _populate_chunks. Symbol lookup resolves "Settings" → "x1y2z3", empty symbol_name → None.

Lines 582–588 — Delete existing chunks (safety), then executemany inserts 3 rows.

Line 589 — Logs: [GraphStore] Backfilled 3 chunks from ChromaDB

Returns: 3 — the number of chunks inserted. Returns 0 if vector_store is None or chunks table already had data.

close(self) (Line 593)

Closes the DuckDB connection.

Example

store.close()

Line 595 — self.conn.close() — releases the DuckDB file lock.

Returns: None

Summary

Function/Method Line Purpose
_stable_id 9 Deterministic 16-char hex ID from input parts
GraphStore.__init__ 29 Open DuckDB, create tables
GraphStore._create_tables 35 CREATE TABLE IF NOT EXISTS × 7
GraphStore.populate_from_analysis 88 Master populate: clear all → fill all
GraphStore._populate_files 128 Insert file records with module mapping
GraphStore._populate_imports 148 Insert file import edges (validated)
GraphStore._populate_symbols 172 Tree-sitter parse → insert functions/classes
GraphStore._populate_symbol_calls 230 Resolve call sites → insert call edges
GraphStore._populate_modules 286 Insert module records + module dep edges
GraphStore.get_import_edges 309 All import edges as (path, path) tuples
GraphStore.get_module_dep_edges 325 All module dep edges as (name, name) tuples
GraphStore.get_module_file 331 Which module owns this file?
GraphStore.get_files_in_module 339 All file paths in a module
GraphStore.get_symbols_in_file 347 All symbols in a file, ordered by line
GraphStore.get_all_files 368 All file paths in the graph
GraphStore.get_importers 374 Reverse: who imports this file?
GraphStore.get_imports 384 Forward: what does this file import?
GraphStore._get_stats 394 Row counts for all tables
GraphStore.get_callers_of_symbol 405 Who calls this symbol?
GraphStore.get_callees_of_symbol 436 What does this symbol call?
GraphStore._populate_chunks 475 Insert chunk rows from embeddings pipeline
GraphStore.populate_chunks_from_chromadb 531 Backfill chunks from ChromaDB metadata
GraphStore.close 593 Close DuckDB connection

Clone this wiki locally