db: raw reference-edge schema with lifecycle-safe grants (#83)

README.md

@@ -596,6 +596,9 @@ Run the server locally with `make run` (binds `DATABRICKS_APP_PORT`, else 8000).
   semantic search (default-on): the preload assumption, opt-out, embeddings, memory notes
 - [`docs/runbooks/indexing-parallelism.md`](docs/runbooks/indexing-parallelism.md) —
   parallel indexing: worker sizing, skip-if-unchanged, compare-and-set stamping
+- [`docs/runbooks/reference-edges.md`](docs/runbooks/reference-edges.md) — the raw
+  call/import edge schema (`reference_edges`, migration `0005`): what it stores, the
+  no-symbol-FK design, and the grant-coupling this migration introduces
 - [`docs/runbooks/ci-lakebase.md`](docs/runbooks/ci-lakebase.md) — the integration CI
   gate: ephemeral Lakebase branches, prerequisites
 - [`docs/runbooks/webui.md`](docs/runbooks/webui.md) — the web UI app: auth, grants,

app/alembic/AGENTS.md

@@ -4,7 +4,7 @@
 # app/alembic
 
 ## Purpose
-The Alembic migration environment and the single linear core revision chain (0001 → 0004) for the code-search schema on Lakebase Postgres. `env.py` resolves its connection in strict priority order: an injected connection from `scripts/migrate.py` (which owns the Lakebase OAuth engine) wins; else `LAKEBASE_ENDPOINT`/`PGHOST` builds one via `create_db_engine()` (the `make migration` autogenerate path against a disposable Lakebase branch); else it raises — there is no implicit default. Autogenerate diffs against `app.db.models.Base.metadata`, with the semantic `chunks` surface filtered out entirely. `alembic.ini` at the repo root points `script_location` here.
+The Alembic migration environment and the single linear core revision chain (0001 → 0005) for the code-search schema on Lakebase Postgres. `env.py` resolves its connection in strict priority order: an injected connection from `scripts/migrate.py` (which owns the Lakebase OAuth engine) wins; else `LAKEBASE_ENDPOINT`/`PGHOST` builds one via `create_db_engine()` (the `make migration` autogenerate path against a disposable Lakebase branch); else it raises — there is no implicit default. Autogenerate diffs against `app.db.models.Base.metadata`, with the semantic `chunks` surface filtered out entirely. `alembic.ini` at the repo root points `script_location` here.
 
 ## Key Files
 | File | Description |
@@ -20,11 +20,12 @@ The Alembic migration environment and the single linear core revision chain (000
 | `0002_index_semantics_version.py` | Adds + backfills `repos.index_semantics_version`. Backfill value is FROZEN as `_BACKFILL_VERSION = 1` — deliberately never imports the live `INDEX_SEMANTICS_VERSION`. Backfills by cadence (rows touched in the last 48h); untouched rows stay NULL → re-index once |
 | `0003_multi_branch.py` | Multi-branch content dedup: adds `files.content_sha` (backfilled in-DB via `pgcrypto` `digest(...,'sha256')`, proven byte-identical to `indexer.hashing.content_sha`) and GIN-indexed `files.branches`; swaps `uq_files_repo_id_path` → `uq_files_repo_path_sha`; creates `repo_branches` seeded from the legacy `repos` stamp. Downgrade guards FIRST: refuses if any path has multiple content versions |
 | `0004_semantic_chunks.py` | The semantic surface in the core chain (supersedes the retired gated `0002sem`/`versions_semantic`): extensions `lakebase_tokenizer` → `lakebase_vector` → `lakebase_text` with `CASCADE` (load-bearing: `lakebase_vector` declares a dependency on base `vector`), the `chunks` table (embedding dim from `app.config.SEMANTIC_EMBEDDING_DIM`, generated `ts` tsvector, `uq_chunks_file_id_chunk_index` — also the write-path index for per-file DELETE and CASCADE), `ix_chunks_embedding_ann` (`lakebase_ann` with explicit non-default `vector_cosine_ops`; rejects hnsw-style WITH params) and `ix_chunks_ts_bm25`. Idempotency guard: if `to_regclass('chunks')` exists (old gated path), only add `start_line`/`end_line` and drop the orphaned `alembic_version_semantic` |
+| `0005_reference_edges.py` | Adds `reference_edges` (epic #82): raw unresolved call/import edges, `edge_kind IN ('call','import')` CHECK, FKs to `repos`/`files` only (both `ON DELETE CASCADE`) — deliberately NO FK to `symbols` (query-time name-join resolution instead). Autogenerate-shaped (`op.create_table`/`op.create_index`, no raw SQL, no new extension — `pg_trgm` already exists since 0001). Four indexes: btree `target_name`, GIN trgm `target_name`, btree `file_id` (write-path/cascade), btree `(repo_id, edge_kind)`. Grant-coupled like 0003/0004 before it — see `docs/runbooks/reference-edges.md` |
 
 ## For AI Agents
 
 ### Working In This Directory
-- **Chain ordering is strictly linear**: `0001 <- 0002 <- 0003 <- 0004`. A new revision sets `down_revision` to the current head; never branch the chain.
+- **Chain ordering is strictly linear**: `0001 <- 0002 <- 0003 <- 0004 <- 0005`. A new revision sets `down_revision` to the current head; never branch the chain.
 - **Migrations are historical facts.** They must never import mutable app constants — `0002` freezes its backfill value locally, and `models.py` explicitly forbids migrations importing `INDEX_SEMANTICS_VERSION`. The one sanctioned exception is `0004`'s use of `SEMANTIC_EMBEDDING_DIM`, which exists precisely so DDL and the `app/db/semantic.py` table can never drift.
 - **Keep the `chunks` blindness intact.** `include_object` in `env.py` plus `chunks` living outside `Base.metadata` are two halves of one protection; removing either makes `make migration` emit destructive drops.
 - **Extension-before-index ordering** is load-bearing in 0001 and 0004; downgrades never drop extensions (database-wide, potentially shared, and the 0004 preload prerequisite is irreversible).
@@ -33,8 +34,8 @@ The Alembic migration environment and the single linear core revision chain (000
 - Run migrations via `make migrate` (`scripts/migrate.py`, injected connection, optional `ARGS=--apply-grants`); autogenerate via `make migration MSG="..."` against a disposable Lakebase branch (`scripts/ci_branch.py up`) — never against production. `env.py` deliberately raises rather than guessing a target.
 
 ### Testing Requirements
-- `make test`: `tests/unit/test_migration_source.py` / `test_migration_source_semantic.py` (source-level revision-chain and no-app-import checks), `test_semantics_version_tripwire.py`.
-- `make test-integration`: `tests/integration/test_migrations.py` (upgrade/downgrade against real Postgres; models ↔ chain parity), `test_content_sha_parity.py` (pgcrypto digest ≡ Python `content_sha`).
+- `make test`: `tests/unit/test_migration_source.py` / `test_migration_source_semantic.py` (source-level revision-chain and no-app-import checks, including 0005's no-symbol-FK check), `test_semantics_version_tripwire.py`, `test_reference_edge_model.py` (ORM-metadata tripwires for `reference_edges`).
+- `make test-integration`: `tests/integration/test_migrations.py` (upgrade/downgrade against real Postgres; models ↔ chain parity; the `reference_edges` shape/cascade/ADP/EXPLAIN tests run on stock Postgres too via `migrated_edges_capable`), `test_content_sha_parity.py` (pgcrypto digest ≡ Python `content_sha`).
 
 ### Common Patterns
 - Raw `op.execute()` for anything SQLAlchemy can't declare portably (extensions, generated columns, lakebase index access methods, backfill UPDATEs); `op.create_table`/`op.create_index` for the declarable rest.

app/alembic/versions/0005_reference_edges.py

@@ -0,0 +1,101 @@
+"""reference edges (raw, unresolved call/import edges)
+
+Revision ID: 0005
+Revises: 0004
+Create Date: 2026-07-23 00:00:00.000000
+
+Adds ``reference_edges``: one row per raw (unresolved) call/import site found by
+the extractor (epic #82). Deliberately NO FK to ``symbols`` -- symbol ids churn
+on every per-file delete-and-reinsert, and an FK would couple the two rewrite
+orders inside the indexing transaction for no query benefit. Resolution from
+``target_name`` to a concrete symbol happens at query time by name-join (a
+later child of #82), not here. The enclosing symbol is denormalized onto the
+row instead (``enclosing_*``, all nullable -- NULL means module/top-level
+scope). No ``branches`` column: branch membership rides ``files.branches`` at
+query time, exactly as ``symbols`` does.
+
+``pg_trgm`` already exists (created by 0001, database-wide) so no
+``CREATE EXTENSION`` is needed here; extension-before-index ordering is
+satisfied by the chain itself.
+
+**Grant coupling (not schema-only for an already-deployed target):** the
+schema-wide grant builders in ``app/db/grants.py`` (``GRANT ... ON ALL
+TABLES IN SCHEMA`` + ``ALTER DEFAULT PRIVILEGES``) cover this new table
+automatically ONLY when the same identity that ran the original grants also
+runs this migration (Postgres ADP binds to the executing role). A different
+identity running a schema-only ``make migrate`` needs an explicit re-grant.
+See ``docs/runbooks/reference-edges.md`` for the verification query and the
+re-grant command.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision: str = "0005"
+down_revision: str | None = "0004"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "reference_edges",
+        sa.Column("id", sa.BigInteger(), nullable=False),
+        sa.Column("repo_id", sa.Integer(), nullable=False),
+        sa.Column("file_id", sa.Integer(), nullable=False),
+        sa.Column("edge_kind", sa.Text(), nullable=False),
+        sa.Column("target_name", sa.Text(), nullable=False),
+        sa.Column("line", sa.Integer(), nullable=False),
+        sa.Column("enclosing_name", sa.Text(), nullable=True),
+        sa.Column("enclosing_kind", sa.Text(), nullable=True),
+        sa.Column("enclosing_start_line", sa.Integer(), nullable=True),
+        sa.Column("enclosing_end_line", sa.Integer(), nullable=True),
+        sa.CheckConstraint("edge_kind IN ('call', 'import')", name="ck_reference_edges_edge_kind"),
+        sa.ForeignKeyConstraint(["repo_id"], ["repos.id"], ondelete="CASCADE"),
+        sa.ForeignKeyConstraint(["file_id"], ["files.id"], ondelete="CASCADE"),
+        sa.PrimaryKeyConstraint("id"),
+    )
+    op.create_index(
+        "ix_reference_edges_target_name",
+        "reference_edges",
+        ["target_name"],
+        unique=False,
+    )
+    op.create_index(
+        "ix_reference_edges_target_trgm",
+        "reference_edges",
+        ["target_name"],
+        unique=False,
+        postgresql_using="gin",
+        postgresql_ops={"target_name": "gin_trgm_ops"},
+    )
+    # Load-bearing for write performance, not just integrity: Postgres does NOT
+    # auto-index a foreign key, and file_id is the hot lookup for both the
+    # per-file delete-and-reinsert writer and the ON DELETE CASCADE fired by
+    # store.py's mark-and-sweep -- same rationale as ix_symbols' analog on
+    # symbols.file_id (there via the implicit FK) and uq_chunks_file_id_chunk_index.
+    op.create_index(
+        "ix_reference_edges_file_id",
+        "reference_edges",
+        ["file_id"],
+        unique=False,
+    )
+    op.create_index(
+        "ix_reference_edges_repo_kind",
+        "reference_edges",
+        ["repo_id", "edge_kind"],
+        unique=False,
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_reference_edges_repo_kind", table_name="reference_edges")
+    op.drop_index("ix_reference_edges_file_id", table_name="reference_edges")
+    op.drop_index("ix_reference_edges_target_trgm", table_name="reference_edges")
+    op.drop_index("ix_reference_edges_target_name", table_name="reference_edges")
+    op.drop_table("reference_edges")

app/db/AGENTS.md

@@ -4,13 +4,13 @@
 # app/db
 
 ## Purpose
-Database connectivity and schema truth for the code-search corpus. `client.py` is the one and only place that knows how to connect to Lakebase (Autoscaling API, per-connection OAuth token injection) or to plain local Postgres (CI/tests via `PGHOST`). `models.py` holds the SQLAlchemy 2.0 declarative models for the durable core (`repos` / `files` / `symbols` / `repo_branches`) whose `Base.metadata` drives Alembic autogenerate; `semantic.py` declares the `chunks` table in a deliberately separate `MetaData` so autogenerate never sees it; `grants.py` builds least-privilege GRANT SQL strings executed by `scripts/migrate.py`.
+Database connectivity and schema truth for the code-search corpus. `client.py` is the one and only place that knows how to connect to Lakebase (Autoscaling API, per-connection OAuth token injection) or to plain local Postgres (CI/tests via `PGHOST`). `models.py` holds the SQLAlchemy 2.0 declarative models for the durable core (`repos` / `files` / `symbols` / `repo_branches` / `reference_edges`) whose `Base.metadata` drives Alembic autogenerate; `semantic.py` declares the `chunks` table in a deliberately separate `MetaData` so autogenerate never sees it; `grants.py` builds least-privilege GRANT SQL strings executed by `scripts/migrate.py`.
 
 ## Key Files
 | File | Description |
 |------|-------------|
 | `client.py` | `create_db_engine()`: Lakebase-endpoint-wins-over-`PGHOST` dual-mode selection; Lakebase mode closes one `WorkspaceClient` over a `do_connect` handler that mints a fresh OAuth token as the password on every physical connect (never logged); server defaults `pool_size=5`, `pool_recycle=2700` (45 min, under the ~1h token TTL), `pool_pre_ping=True`; SDK import is lazy so the local path never touches it |
-| `models.py` | ORM models + `INDEX_SEMANTICS_VERSION` (currently 2; bump on any indexing-meaning change — CI tripwires it). `File` is content-deduped per (repo_id, path, content_sha) with a `branches ARRAY(Text)` membership column; `RepoBranch` is the authoritative per-(repo, branch) CAS stamp; `repos`' own stamp columns and `files.commit` are deprecated/ambiguous — never add readers. Declares the trgm + branches GIN indexes so autogenerate can't drift |
+| `models.py` | ORM models + `INDEX_SEMANTICS_VERSION` (currently 2; bump on any indexing-meaning change — CI tripwires it). `File` is content-deduped per (repo_id, path, content_sha) with a `branches ARRAY(Text)` membership column; `RepoBranch` is the authoritative per-(repo, branch) CAS stamp; `repos`' own stamp columns and `files.commit` are deprecated/ambiguous — never add readers. `ReferenceEdge` (0005, epic #82) is a raw unresolved call/import edge, deliberately with NO FK to `symbols` (resolution happens at query time by name-join in a later child); FKs to `repos`/`files` only, both `ON DELETE CASCADE`. Declares the trgm + branches GIN indexes so autogenerate can't drift |
 | `semantic.py` | Standalone Core `Table` for `chunks` (BigInteger PK, `Vector(SEMANTIC_EMBEDDING_DIM)` embedding, generated `ts` tsvector, nullable `start_line`/`end_line`) in its own `semantic_metadata` — a typed description only; the real DDL is owned by migration `0004` |
 | `grants.py` | Pure SQL-string builders `build_app_grants` (read-only) / `build_job_grants` (CRUD + sequences, no DDL); identifiers validated against `^[A-Za-z0-9_-]+$` (1..63 chars) then psycopg-quoted. Execution lives in `scripts/migrate.py` |
 | `__init__.py` | Re-exports `Base`, `File`, `Repo`, `Symbol`, `create_db_engine` |
@@ -23,6 +23,7 @@ Database connectivity and schema truth for the code-search corpus. `client.py` i
 - **Do not add `chunks` to `Base.metadata`.** It lives in `semantic.py`'s own `MetaData`; pairing with the `include_object` filter in `app/alembic/env.py`, this is what stops autogenerate emitting `drop_table('chunks')` or spurious diffs of the hand-written vector/tsvector DDL.
 - **`INDEX_SEMANTICS_VERSION` must never be imported by a migration** (migrations freeze their own constants — see `0002`). Bump it whenever `indexer/symbols.py`, `indexer/parse.py` chunking, or `indexer/languages.py` extraction changes meaning.
 - `RepoBranch.last_indexed_commit` is the ONLY commit truth-source; `files.commit` is write-only and ambiguous under multi-branch dedup.
+- **`reference_edges` must never gain a FK to `symbols`** (epic #82 rule) — resolution from `target_name` to a concrete symbol happens at query time by name-join, not via a stored FK. `tests/unit/test_reference_edge_model.py` and `tests/unit/test_migration_source.py::test_0005_no_symbol_fk` are standing tripwires; do not weaken them.
 - The server's pool default (5) is paired with `app/main.py`'s `CapacityLimiter(5)`; the indexer passes `pool_size=` explicitly from its worker count. Changing one side without the other breaks the oversubscription guarantee.
 - Grant changes are deploy-coupled: adding a privilege needs a `deploy.sh --apply-grants` re-run, not just a schema migrate.
 

app/db/__init__.py

@@ -1,6 +1,6 @@
 """Database connectivity and ORM models for the code search service."""
 
 from app.db.client import create_db_engine
-from app.db.models import Base, File, Repo, Symbol
+from app.db.models import Base, File, ReferenceEdge, Repo, Symbol
 
-__all__ = ["Base", "File", "Repo", "Symbol", "create_db_engine"]
+__all__ = ["Base", "File", "ReferenceEdge", "Repo", "Symbol", "create_db_engine"]