feat(planner): use indexes at plan time, index facts become a planner input

[azimafroozeh]

What & why

This PR carries the maintainer-track RFC "Index facts as planner input" (docs/rfcs/2026-10-08-index-facts-as-planner-input.md) and all four of its rollout steps: the v2 planner reads which scalar indexes exist on every table a query reads, at the dataset version the plan is pinned to, and makes every access-path decision at plan time; execution consumes the decision and never re-checks it. The motivating defect is #854: a single String @key equality scanned the key column while the __id BTREE went unused, and no .gqt case could pin index use because the plan never stated it.

• The fact: PlanSource exposes one index fact per index name and keyed column at the pinned version; coverage unions same-name segments and intersects current fragments, an unreadable bitmap stays unknown, and a metadata or table-open failure propagates from planning with its own error class, never as an empty catalog.
• Expand: coverage is derived from the facts instead of hardcoded Indexed, so csr against indexed_scan is priced on real coverage; saved Expand inputs carry coverage provenance, and older saved plans without it keep the runtime coverage probe.
• Scan access: every fresh root scan whose filter is fully known at plan time records access: sequential, or index_probe with the index-query tree and residual that Lance's own scanner put in the read, built at plan time at the pinned version; execution sets use_scalar_index from the plan. A scan that gains inputs at execution (a ranked read, a join-generated filter, a dynamic expression, a dependent scan) records runtime or id_lookup and leaves index use to Lance as today.
• Key narrowing (key_to_id pass): a cast-free equality on a single String @key gains an __id equality (for that key domain the key string is the id) and keeps its written predicate; numeric, temporal, float and composite keys are never narrowed.
• Explain and GQT: one index_facts(<table>) statistics row per read table, access on every fresh physical Scan, and expect plan claims of the form scan Person as $p: access index_probe __id.
• Measured in tests that ship with the diff: cold planning costs the same requests as one ordinary cold scanner build (2 manifest requests on a modern dataset, ceilings pinned per fixture) and reads no row data; a 200,000-row String-key lookup reads the same execution requests and bytes as the __id lookup, where base scanned the key column.

Backing issue / RFC

• Closes A lookup by String @key scans the whole key column instead of using the id index #854: cases/v2/planner/issue_854_key_index_probe.gqt pins a String @key lookup probing the __id BTREE with its written equality kept, which the base tree cannot claim (no access grammar) and did not do (the key column was scanned).
• Is an RFC PR: docs/rfcs/2026-10-08-index-facts-as-planner-input.md (maintainer track, accepted, implementation complete) lands here with its implementation; it composes with RFC 0047's FTS presence fact and leaves RFC 0046's status surface unread.

Checklist

• Focused: one logical change, index facts as a planner input (the fact, Expand coverage, scan access, key narrowing are the RFC's four rollout steps and ship as one feature)
• Tests: 14 new .gqt cases under cases/v2/planner/ (the corpus serial run is 289 cases green) pin the access claims, both Expand coverages, key narrowing on literal, parameter, escaped and empty keys and its refusal on casts and composite keys; Rust tests only where the format cannot reach: index-fact gathering on real snapshots, the cold-planning request and byte ceilings, the 200,000-row I/O comparison, saved-plan provenance round-trips, and a serial failpoint test that a source failure during planning keeps its error class
• Public docs: docs/user/queries/explain.md (the access fields, the index_facts statistic, the key-narrowing note), docs/dev/architecture.md, docs/dev/execution.md
• Release note: changelog.d/planner-index-facts.performance.md
• Invariants: reviewed against docs/dev/invariants.md, no Hard Invariant weakened, no deny-list item hit; nearest one touched is invariant 7 (missing coverage may change cost but not correctness): the access path changes bytes read, never rows, the residual re-evaluates the full predicate on every probed row, and nothing refuses on a scalar fact

Local verification

• cargo test --workspace --exclude omnigraph-gqt --exclude omnigraph-dst --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints --lib engine::plan_source::: 14 passed: cold-planning ceilings (modern 2 manifest requests / 839 bytes; overlay adds one deletion and one row-id request; legacy adds 3 zero-byte index-discovery probes), the 200,000-row key lookup at 12 execution requests / 70,664 bytes for both the key query and the __id query, and a saved plan forced to sequential reading more than twice the bytes
• cargo test -p omnigraph-engine --locked --features failpoints --test failpoints planning_source_failures_keep_their_error_class: 1 passed: both planning seams return the injected error as OmniError::Manifest with kind BadRequest, and the next read runs
• cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked -- --test-threads=1: 289 corpus cases green in 431 s, no deadline hit
• cargo test -p omnigraph-gqt --locked (default parallelism): red locally, 17 cases hit their 10 s deadline under load on this machine, none touching a path this branch changes; the same cases pass serially above
• cargo test --workspace --exclude omnigraph-gqt --exclude omnigraph-dst --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints --no-fail-fast: in the sandbox: 120 suites ok, red only on 3 CLI session-lock tests and 1 blob Unix-socket test (sandbox) and 6 bench watchdog tests that pass rerun alone; the same command passed 4,068 tests with 0 failures outside the sandbox before the review-round fixes

Notes for reviewers

• Default change: a fresh v2 plan sets use_scalar_index from its recorded access; a saved plan without access keeps Lance's default, and saved Expand inputs without coverage provenance keep the runtime coverage probe, so an existing store and its stored plans read as before.
• Planning I/O: gathering index facts opens every table a query reads at planning, root and edge tables included, so a metadata-only query can open tables it previously did not; the per-fixture ceilings pin that at one ordinary cold scanner build.
• Legacy file-format datasets: scan access is decided at planning by reading the Lance plan the real scanner builds for the pinned table. On a legacy file-format dataset Lance may pick MaterializeIndexExec, whose query the public API does not expose; the split then reconstructs it through Planner::create_filter_plan over the dataset's scalar_index_info. When that reconstruction yields no query, or the scan holds two probes, planning fails with a manifest_internal error for a read that would have executed. This follows the RFC's failure posture: failures while opening a pinned table or loading its index metadata propagate from planning and are never treated as an empty catalog. The reconstruction is qualified by pinned source inspection and a public reconstruction guard on both file formats; no legacy-vintage graph fixture with pre-existing indexes exists yet, so a legacy dataset that defeats the reconstruction is a documented limit of this PR, not a tested path.
• Historical single-String ids: key narrowing rewrites a single String @key equality into an __id equality, which returns the key string unchanged for a single non-list String key. Its soundness rests on the identity guarantee of single String keys: every row of such a type, historical or current, carries the key string as its __id. The persisted-historical-keys test shows Date, F32 and composite keys are left alone; it has no String row whose stored id differs from its key, because the loader never wrote one. A graph whose single-String-key rows were written with a non-identity id by an older writer would be narrowed to the wrong id; no such writer exists in the repository's history, and that is the claim this PR relies on.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-09T16:44:20.206645Z	f83d227	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.