Repository navigation
feat(planner): accept every read plan against its checked query (RFC 0047) - #839
Conversation
The query shapes engine v2's planner refuses by design now carry the same diagnostic contract as parse and type refusals. The ranking of a binding a traversal reaches gets code P001, stage `plan`, the refused call as the expression, and a fix that answers the query: declare that binding first, so the ranking starts the traversal. The edge-selection refusals added with edge alternation and wildcard traversal get P002 (no finite traversal work limit), P003 (the limit out of range or recorded twice) and P004 (CSR traversal mode), with fixes where one exists. `PlanError::Unsupported` holds the diagnostic and `PlanError::refused` builds it; the gate's `Unrouted::UnsupportedQuery` carries it, and `plan_source` returns it as `OmniError::Compiler` on the ordinary and the inspected door, so the server answers 400 with the `diagnostic` object and the CLI prints the code, stage and fix. The diagnostic kind `plan` renders its one-line text as `plan error: P001: …`. Fixes ModernRelay#786.
Before the planning that runs, `load_column_statistics` resolves and rewrites the query once to find the tables whose column statistics the cost model needs. It mapped every planning error through `no_plan`, so a refusal by design raised while resolving, such as an edge wildcard under CSR traversal mode (P004), came back as an internal error carrying the diagnostic only as text. It now maps a refusal through the same conversion the gate's result uses, and leaves `no_plan` for defects.
One validator in omnigraph-planner (validate/) constructs the only executable plan type, AcceptedPlan, whose fields are private. Ordinary execution, the explain document, the explain statement and the inspected run call the same accept_query path; explain renders the accepted plan and reports validation.scope. The checked declaration and its type context travel beside the IR (CompiledQuery), and the requirements are derived from them: binding identity, search identity and approximation, retained eligibility predicates, correlated blocks, the projection and score origin, order and cut, and the declared candidate and probe caps. A fresh plan failing a check is a planner defect; an exhausted validation limit is a resource outcome. Replay decodes an envelope (query source and name, scope, schema digest, bound plan) within a byte limit, checks its versions first, refuses a changed schema as incompatible facts, and accepts the plan again against requirements derived afresh. The implicit tie-break now orders declared bindings before the bindings the lowering makes up (anonymous endpoints, cycle temps), as RFC 0047's comparator requires; the validator rejected the old order.
A node's declared ordering is typed order keys (column, expression with direction and null placement, identity, fused rank), derived by one rule from the operator and its inputs: a sort establishes exactly its comparator, identity keys included, and a filter keeps its input's order. The bound plan format carries them (version 2). Acceptance recomputes the root's order from the operators rather than reading declared properties, and each check has a test that breaks the plan it guards.
…coverage Lance scores fragments no full-text segment covers flat, from statistics over the rows its prefilter admits, so a filter applied before scoring changes those rows' BM25 scores. The plan source reads the ranked property's full-text coverage at the pinned dataset version (full, partial or absent), the plan records it in its assumptions, and each ranked scan declares where it applies its eligibility: a bm25 scan before scoring only under full coverage, else after, as Lance's postfilter. Acceptance refuses a bm25 scan that filters before scoring without recorded full coverage. GQT gains the claim `ranked bm25 eligibility <placement>`.
A query of one binding with Boolean, integer and String eligibility, an optional leading bm25() over a String property, direct property sort keys, a limit and a projection of properties and the selected score is a member of the exact fragment. The optimizer records the rules it applies to such a chain (absorb a conjunct into the scan, prune the scan's columns, lower each node, rank the scan by bm25 with its eligibility placement); the validator checks that the IR is the declaration's lowering, rebuilds the canonical chain, re-applies every rule against its precondition and requires the result to equal the plan. A member is accepted as exact_subset, everything else as invariants_only. Replay envelopes carry the derivation, and replay checks it again.
A nearest scan declares its probe policy (the escalation factor, the flat rescan of rows the partition search did not reach, the uncapped fail-closed rescan when Lance's counters are missing, and the flat start when the eligible set fits within the fetch); every pre-pass declares what an empty eligible set decides and whether the recorded full-text coverage of the bm25 scans it feeds admits it. The scan's ladder and both gates read these from the plan instead of their own constants, and the rrf gate's coverage guard is the recorded fact, no longer a dataset read at run time. Acceptance checks that every ladder terminates and keeps its fallbacks, and that every pre-pass draws its eligible set from required first hops and feeds only what its kind may prefilter. The execution report records each gate's verdict with the counts it read and each probe attempt per rung, and a replay must record the same decisions.
Bytes over the evidence limit exhaust before anything is decoded, every envelope version is read before the body, and the replay door answers an envelope of another version as a conflict that asks for the query again.
…ention A table of admitted and excluded forms pins exact-fragment membership; the plain and explained acceptance entries agree on plan, scope and an exhausted limit; an instrument prints derivation bytes, steps, nodes, visits and time as a member query grows. Predicate retention resumes after its last match, so in-order placement checks in linear work, and a derivation charges the conjuncts each absorption copies.
A prefilter changes the BM25 score of a row no full-text segment covers, and a postfilter keeps the whole corpus's score; the eligibility placement of a bm25 scan rests on both.
… built The execution guide describes plan acceptance, the exact fragment and the replay envelope; the explain guide documents validation, a bm25 scan's eligibility placement, a nearest scan's policy, the recorded full-text coverage and the profile's search decision rows, which the profile now carries; the testing map names the validator's owners; the Lance guide adds the eligibility placement fence; the GQT README adds the scope and eligibility claims. RFC 0047 records where the built design differs from its text, and three release notes describe the user-visible changes.
…verage An order key the planner binds to a return item by equal lowering is accepted when the written key lowers to that item, though the two are written differently (`"a" in $d.tags` and `$d.tags contains "a"`), and a tie-break omission whose returned items all lower to order keys is accepted as an equivalence. Before this a legal query failed as a planner defect. The replay door re-establishes every full-text coverage fact a plan records from the pinned snapshot before acceptance: an envelope claiming full coverage of a partially indexed property, with its bm25 scan and derivation switched to filter before scoring, was accepted and would have changed BM25 scores. Both found in a Codex review.
…r probes
Two defects a Codex review found in plan acceptance and the run report.
An order key that is constant for every row (a parameter, `now()`, a
single edge type's `@type`) orders nothing, and the order check discounts
such keys on both sides. When `return` projects the constant under an
alias, the planner binds the key to that alias, and the check compared an
unresolved alias on one side with a discounted constant on the other, so
`return { $x as x } order { $x }` failed acceptance as a planner defect.
Both sides now resolve an alias to the `return` item it names before
discounting constants.
A fusion recorded its gate decision and returned without its nearest arms'
probe attempts, so inspection and replay could not see them, and every
nearest scan wrote one shared report, so two nearest arms would overwrite
each other. Each nearest scan now reports under its own plan node; the
standalone search reads its scan's entry and a fusion records each arm's
attempts after its gate.
Regressions: cases/v2/constant_order_key_bound_to_a_return_alias.gqt,
validate/tests.rs::a_constant_key_bound_to_a_return_alias_is_accepted, and
engine_v2_plan_replay.rs::a_fusion_replays, extended with a two-arm
rrf(nearest, nearest).
# Conflicts: # crates/omnigraph/src/engine/mod.rs
ragnorc
left a comment
There was a problem hiding this comment.
Recommendation: request changes before merge. I found two reproducible gaps in the new acceptance checks. This is a COMMENT review because the authenticated account is also the PR author.
This PR adds a check between a user's query and the plan that runs it. It aims to catch translation errors before they produce wrong answers. Ordinary queries, explain, and saved-plan replay use the same acceptance path. A small set of single-table queries gets checked rewrite evidence. Other queries get narrower checks, reported as invariants_only. The PR also keeps BM25 scores independent of filters on unindexed rows and puts named bindings first in implicit tie breaks.
Two findings need changes:
- P2 — An unwritten row limit passes acceptance. A saved
count_peopleplan with an addedLimit(0)replays successfully. It returns zero rows instead of the required count row. The inline comment identifies the missing check. - P2 — The checker retains quadratic copies of predicate data. A 526,559-byte query retained 68,717,622 bytes of string literals in its derivation arena. Acceptance still succeeded within every default counter. The inline comment includes the fixture and a control.
The design has a sound starting point: an accepted wrapper closes the execution boundary, and requirements come from the checked declaration. That can catch an error shared by later planning stages. It adds protection by construction. The BM25 change also addresses a cause. In pinned Lance 11, fts_search_source uses an empty filter plan for postfilter search. plan_flat_match_query otherwise filters input before flat scoring. The new Lance surface guard passed locally. See Lance scanner and OmniGraph's scan adapter.
The concrete tradeoffs are:
- Assurance costs planning work on every read. Compiled queries retain the checked declaration, but acceptance and derivation reconstruction still run per request. This matters most for frequent small reads. I did not measure production latency or throughput.
- Stable BM25 scores can require more work. Appends leave rows outside index coverage. Postfiltering scores those rows before removing ineligible results, and loses scalar-index filtering at that stage. This favors correct scores over selective-read speed. Full coverage permits prefiltering again.
- Replay favors a fixed contract over portability. Schema and dataset checks reject stale evidence. Format, rule, and semantics versions require future compatibility decisions. This is a hidden Rust replay API, not a new HTTP endpoint.
- The stronger guarantee has a narrow scope. Traversal, aggregation, ANN, and RRF remain outside the exact subset. Fusion and search-aggregate ordering checks are explicitly deferred in RFC 0047. These limits must remain visible when more rules land.
My liability assessment is mixed. The shared acceptance path, typed ordering, and plan-owned search policy remove places where execution and explain can disagree. But this change adds 7,719 lines and removes 627 across 62 files. It adds another account of lowering semantics, a rule catalogue, retained evidence, and versioned replay obligations. The two findings show real costs of that new responsibility. Five similar extensions should compose through the same small rule set and bounded evidence store. Separate copies of each query shape's semantics would increase long-term liability. Fixing the retained-node issue with existing arena slots would reduce that liability without another abstraction.
Validation on ec8b43a25b7a14ea78312693182413d49cbca19b:
- Local clean baseline: planner suite, 121 passed and one ignored instrument.
- Local engine baseline:
engine_v2,engine_v2_plan_replay,engine_v2_scrubbed_replay, andlance_surface_guards: 103 passed and two ignored tests. - Temporary assertions in existing owners reproduced the accepted extra cut at both planner and replay boundaries. A focused rejection check made the replay assertion pass.
- The existing derivation instrument measured retained string data. Releasing obsolete arena nodes reduced it to 524,562 bytes for the same query. This is retained payload, not a process-RSS measurement.
- Exact-head CI reports successful Test Workspace and GQ Logic Tests. I did not rerun the full workspace or cloud suites locally.
I restored all temporary source edits. The restored planner suite passed all 121 tests, and the restored replay owner passed all 24 tests. The checkout is clean. The review leaves the PR unchanged.
P001's fix, declare the ranked binding first, cannot answer an rrf() whose arms rank two bindings one traversal connects: the binding declared first starts the traversal and the other is reached, so following the fix only produced the opposite hint. Such a fusion is now refused before either arm is ranked, as P005, which names both bindings and the fused call and offers no fix. The reorder hint stays for a single ranked binding. The issue-786 owner test now runs the fused shape in both declaration orders and requires the same fixless refusal.
Two review findings on plan acceptance. The row-cut check compared a written `limit` with the root's cut but let any cut through when the query wrote none: a saved count plan with an added Limit(0) replayed as no rows. A read plan now carries exactly the query's cut, one at the root when it writes `limit` and none anywhere when it does not, a correlated block's tree included. The derivation arena stored each rule's successors and kept the nodes they replaced, so absorbing N conjuncts one at a time retained O(N^2) predicate data while every counter passed; a 526 KB query held 68 MB of literals. A replaced node is unreachable already (a rule naming it is refused), so it is now released before its successor is stored, and the arena keeps only each role's current node. RFC 0047's claim that the byte and node limits bound evidence memory is replaced by that linear bound. Regressions: validate/tests.rs::an_unwritten_row_cut_fails_the_row_cut, validate/tests.rs::the_derivation_arena_keeps_only_current_nodes, and engine_v2_plan_replay.rs::an_unwritten_row_cut_refuses_the_replay_of_a_count.
ragnorc
left a comment
There was a problem hiding this comment.
Recommendation: request changes for the remaining P2 row-cut gap in the inline comment. The memory finding is fixed. I reviewed exact head 9f37eca717a122882b842630583f524594a65f9d, including the changes since my review of ec8b43a2.
This PR checks that a physical read plan matches the user's checked query before execution. A small set of single-table queries gets checked rewrite evidence. Other queries get narrower checks, reported as invariants_only. The follow-up addresses two failures in that protection: saved plans could add a row limit, and the evidence checker kept obsolete copies of predicates. It now counts explicit limits and releases replaced arena nodes. The memory change works. The row-cut change still misses limits stored in sort nodes.
The remaining defect has an observable result. The existing liked traversal query returns three rows. A saved plan with an extra Sort { fetch: Some(0) } below its final sort passes acceptance and returns zero rows. The new check rejects equivalent Limit and Page nodes, including those inside correlated blocks. It must also check capped sorts. I did not observe the optimizer generate this bad shape. This is a gap in the acceptance and replay contract.
The arena change fixes the memory cause. Old references were already invalid. Clearing their slots now removes the data that those references could no longer use. With 128 large predicates, the same 526,559-byte query retains 524,562 bytes of literal strings, compared with 68,717,622 without the release. The arena holds four live nodes. Measurements at 32 and 64 predicates also follow linear retained growth.
For frequent small reads, acceptance adds work before useful execution. For larger generated predicates, retained copies and concurrent requests become more important. The check runs in the shared engine, so embedded, file, S3, and Azure use need the same contract. These fixes change planner bookkeeping. They need no new storage mechanism.
The tradeoffs and liability are explicit:
- The memory fix reduces retained data by design. It keeps existing slots, role ownership, and stale-reference checks. It adds no cache, memory manager, or production API.
- Each absorption still copies expressions. Linear retained memory does not imply linear total copying work. The measurements cover literal payload, not peak RSS or production latency. The corrected RFC makes the bound relative to compiled query size.
- The row-cut check adds a charged walk over plan nodes. Its cost follows plan size, not stored row count. Strict placement limits future optimizer freedom until a new rewrite has appropriate evidence.
- The corrective commit adds 147 net lines, mostly tests and documentation. It removes substantial memory liability despite the positive line count. The remaining liability is incomplete coverage of row-reducing fields. Five similar extensions should share the same acceptance rules and arena ownership rules, with one regression owner per contract.
Validation:
- Clean local baseline: 123 planner tests passed, with one ignored instrument. The four engine targets passed 104 tests, with two ignored tests. These include replay, scrubbed replay, and the Lance surface guards.
- Temporary probes confirmed that extra
LimitandPagenodes fail both at the root and inside a correlated block. The capped-sort replay returned the wrong rows. A focused rejection control refused that replay. - Disabling the arena release reproduced the 68,717,622-byte retained payload and failed the new retained-node regression. I restored all temporary edits. The restored planner and replay suites passed, and the checkout is clean.
- Formatting, AGENTS links, and documentation checks passed locally. Workspace CI passed on merge commit
13154c7f7dea2087c508cafaccf137a552e21afd, which includes this head. Its log records all three new regressions as passed. - The complete upstream search documentation and pinned Lance 11 implementation remain consistent with the earlier review. The dependency pin and scan adapter did not change. The reproduced cut occurs in OmniGraph's own sort operator.
The merge CI documentation check still fails because its release-note snapshot is stale. The exact-head documentation check passes locally. Refresh that merge snapshot before merge. I did not rerun the full workspace, HTTP, or cloud suites locally, or measure production performance.
v0.12.0 is tagged and its snapshot generated, but release.json still named it, so the documentation check compared every note added since with that published snapshot and failed. The release procedure's post-publication step moves the configuration to the next version on the published base; this is the same change ModernRelay#871 and ModernRelay#878 carry.
…to feat/plan-validation # Conflicts: # crates/omnigraph-compiler/src/ir/lower.rs # crates/omnigraph-planner/tests/integration/bound_plan.rs # crates/omnigraph/src/engine/plan_source.rs # crates/omnigraph/src/engine/plan_source/key_tests.rs # crates/omnigraph/src/exec/query_doors.rs # crates/omnigraph/src/runtime_cache.rs # docs/dev/lance.md # docs/dev/testing.md
# Conflicts: # crates/omnigraph/src/table_store.rs # docs/dev/lance.md
ragnorc
left a comment
There was a problem hiding this comment.
Recommendation: request changes at d7393077747281e6418200fa4cbe67274548bdcb. One blocking P2 defect is inline: a saved fusion plan can add a predicate after fusion, pass acceptance, and discard required rows. This is a COMMENT review because the authenticated account is the PR author.
This PR checks a read plan before the engine executes it. The purpose is to catch planner errors and reject invalid saved plans. This revision brings that gate onto the typed compiler, key narrowing, late column reads, and the current search ordering. Normal reads, explain, inspected reads, and replay share acceptance. Replay also checks the captured dataset versions and full-text coverage. The reproduced defect is at the embedded Session::replay_bound_plan boundary. I found no HTTP or CLI route that accepts these saved envelopes. See the replay caller.
The direction reduces maintenance liability. Requirements now compare typed IR expressions, which removes a second translator for query expressions. The tradeoff is explicit: compiler lowering remains trusted. These checks do not independently prove that lowering preserves every source expression. Behavioral GQT coverage therefore remains necessary. The exact subset proves equivalence for a narrow set of queries. invariants_only checks selected requirements for the other queries.
The revision also reuses captured index facts and the loader's key-identity derivation. It adds two checked rewrite rules, a key-derivation interface, and an ordered flag that preserves information lost when constant keys are removed. Those additions have concrete owners and purposes. Evidence versions, rule maintenance, and validation work on every read remain costs. Five similar optimizer changes should extend this gate and its existing test owners. They should not create five separate validators or add replay exceptions. For frequent small reads, that planning work can matter. I did not measure latency, throughput, or retained memory in this follow-up.
I surveyed all 29 other open PRs. The direct alignment points are #815, already included here, and the expression/search proposals #805, #792, and #793. Their new expressions, analyzers, and candidate windows should enter through typed lowering and explicit acceptance rules. The draft polymorphism proposal #884 will also need type-aware identity checks before it expands this guarantee. The full-text facts in #905 remain relevant.
The new actor-view draft #951 also intersects this boundary. It proposes a ViewSet predicate and an execution-time checker beside validators that this PR moves into acceptance. That proposal should use the shared acceptance gate and derive authorized view predicates from trusted request context. It should not exempt arbitrary predicates from the origin check. This is future integration work, not a reason to add those APIs here. I found no further direct conflict in the other open work.
I checked the relevant upstream documentation and pinned Lance 11.0.0, source ab6b5bbe46009ed78746b444df8db59a8bc5d842. Its FTS source selection and flat match plan support retaining the coverage-dependent filter placement. I also traced OmniGraph's untrained analyzer declaration, index rebuilds, and replay coverage checks. This PR changes no storage format.
Looking ahead, Lance #9798 adds search_effort and changes probe-option precedence. My inference is that an upgrade must revisit the declared nearest policy and its versioned replay evidence. Those newer semantics must not be assumed for the current pin.
The head changed during review from dbd491d4 to d7393077. I inspected the merge: it adds main’s Blob CLI work, test consolidation, and documentation changes. The compiler, planner, engine implementation, replay test owner, invariants, systems guide, and Lance pin are unchanged. I read the updated repository and testing instructions.
Local validation on the final head, in an isolated checkout:
- Planner: 152 passed, one ignored instrument. The empty doctest target is not counted.
cargo test --locked -p omnigraph-engine --features failpoints --test integration -j 3 engine_v2 -- --nocapture: 80 passed.cargo test --locked -p omnigraph-engine --features failpoints --test lance_surface_guards -j 3 fts_prefilter -- --nocapture: 2 passed. These cover unchanged scores under full coverage and filter-dependent scores in an unindexed tail.- Six existing GQT cases passed: constant order aliases, equally lowered return aliases, edge-selection semantics, BM25 scores over an unindexed fragment, hydration above fusion, and parameter rebinding for key narrowing. Two temporary, smaller order cases also passed.
All Cargo runs used Rust 1.97.1 on macOS ARM64, CARGO_TARGET_DIR=/tmp/review-928/target, and RUSTFLAGS='--cfg tokio_unstable --cfg tokio_unstable'. An initial engine build filled the disk before tests ran. After removing only unused incremental artifacts from that private cache, the retry passed. I did not count the failed build as test evidence.
Regression evidence and its limits
Fusion defect. I temporarily extended engine_v2_plan_replay.rs, using the existing docs fixture and fused query. The query returns three rows. I inserted Filter(false) immediately above RankFuse, below its parent, copied the input properties, and serialized the plan with the original query and pins. The refusal assertion failed because replay succeeded with [].
Command:
cargo test --locked -p omnigraph-engine --features failpoints --test integration -j 3 review_post_fusion_filter_refuses_replay -- --nocaptureAt the reviewed head plus the temporary test: 1 executed, 1 failed, for the predicted behavioral reason. As a diagnostic on predecessor dbd491d4, I called check_eligibility on the complete top-level pipeline as well. The same test passed and required a predicate origin refusal. That second result is from a temporary diagnostic edit, not from the PR. It confirms the missed traversal scope. The production fix should keep per-arm retention and give global predicate-origin obligations complete coverage. GQT cannot inject an edited saved plan, so the existing Rust replay owner is appropriate. Please include this regression with the fix.
Order fixes. The before commit was 256dceb5f85c05465607d2a051d7ec03b5efad0c, immediately before 669ca00593f3310a0938f3fcea56129cf12f0aa6. It failed to compile: engine/plan_source.rs still imports the removed omnigraph_planner::ConstantEvaluator. No behavioral test ran at that commit. Thus I cannot claim an unmodified historical-commit red/green proof.
The command at that before commit was:
cargo run --locked -p omnigraph-gqt -j 3 -- /tmp/review-839-20261012/constant_return.gqt /tmp/review-839-20261012/constant_order.gqt /tmp/review-839-20261012/existing_constant.gqt /tmp/review-839-20261012/existing_alias.gqtFor a controlled behavioral check, I restored only the old constant-order and constant-return checks on dbd491d4. Two small GQT cases reused the existing Person/Knows fixture:
query constant_return() {
match { $p: Person $p $e: knows $d }
return { $p.name as source, $d.name as target, $e.@type as edge_type }
order { $p.name, $d.name, $e.@type }
}
query constant_order() {
match { $p: Person $p $e: knows $d }
return { $d.name as target }
order { $e.@type }
}
cargo run --locked -p omnigraph-gqt -j 3 -- /tmp/review-839-20261012/constant_return.gqt /tmp/review-839-20261012/constant_order.gqt /tmp/review-839-20261012/existing_constant.gqtAll three cases failed under the old checks. The first incorrectly required binding IDs after a complete return comparator. The other two incorrectly treated a constant-key sort as unwritten. With the old tie-break coverage check restored separately, the existing named_constant_only_order step and equally-lowered-alias case also failed on missing identity keys. This establishes that the permanent corpus already detects those fixes, so I do not request duplicate GQT cases.
After restoring dbd491d4, the five order cases passed. On final head d7393077, the following command passed 8/8 cases:
cargo run --locked -p omnigraph-gqt -j 3 -- /tmp/review-839-20261012/constant_return.gqt /tmp/review-839-20261012/constant_order.gqt crates/omnigraph-gqt/cases/v2/constant_order_key_bound_to_a_return_alias.gqt crates/omnigraph-gqt/cases/v2/order_key_bound_to_an_equally_lowered_return_item.gqt crates/omnigraph-gqt/cases/issue_659_edge_selection_semantics.gqt crates/omnigraph-gqt/cases/v2/bm25_score_ignores_a_filter_over_an_unindexed_fragment.gqt crates/omnigraph-gqt/cases/v2/planner/hydrate_return_columns_above_a_fusion.gqt crates/omnigraph-gqt/cases/v2/planner/key_narrowing_rebinds_each_parameter_value.gqtAll temporary source and test edits were restored. The checkout is clean at the reviewed commit. No test changes were pushed.
Workspace CI and GQT CI report success for final PR head d7393077. The workspace checkout log records merge commit 80f6ae8a50e4960aba9ec3594724f1c3adda56b0, which includes that head. This is merge-CI evidence, separate from the exact-head local checks above. I did not rerun the full workspace or cloud suites locally.
Optional: refresh the PR description. Its claim that requirements come from the checked query “not the IR” describes the old implementation. The current RFC decision entry correctly records the typed-lowering dependency.
…iagnostic # Conflicts: # crates/omnigraph-compiler/src/query/codes.rs
…to feat/plan-validation
…iagnostic # Conflicts: # crates/omnigraph-compiler/src/query/codes.rs
…to feat/plan-validation # Conflicts: # crates/omnigraph-planner/src/optimizer.rs
A fused plan's arms each copy the match, so retention stays per arm; but the nodes between the fusion and the root were checked by nothing, and a saved plan could filter fused rows there and pass acceptance. Every arm's pipeline now carries the nodes above the fusion for predicate origin, traversal identity and bindings, and a correlated block above the fusion is refused. `a_fusion_replays` forges a `Filter(false)` above the fusion and requires the replay door's refusal.
…iagnostic # Conflicts: # crates/omnigraph-compiler/src/query/codes.rs
…to feat/plan-validation # Conflicts: # crates/omnigraph-gqt-core/src/plan.rs # crates/omnigraph-planner/src/bound.rs # crates/omnigraph-planner/tests/integration/bound_plan.rs
…parent The fused plan feeds its RankFuse into a Projection, not a Sort; the forged plan now rewires whichever single-input node reads the fusion.
ragnorc
left a comment
There was a problem hiding this comment.
Recommendation: approve at b8d245048a8aca2df348dfdd24a79d74d3ca7fe3. The production fix closes the previously reported fusion predicate gap. The regression now reaches replay and detects the original defect. No blocking defect or optional improvement found in this follow-up. This is a COMMENT review because the authenticated account owns the PR.
This PR checks that a read plan follows its checked query before execution. It also checks saved plans before replay. The latest fix prevents an added predicate above rank fusion from silently removing answers. The previous validator checked only the fusion arms. It missed nodes through which the combined rows then pass.
The revised traversal separates two obligations: each arm must retain the query's required predicates, and every predicate reached by its rows must have a valid origin. It reuses the existing conjunct collection for scans, filters, and joins. It also checks extra bindings and traversals above fusion, and refuses a correlated block placed there. This addresses the omitted part of the plan by design; it does not special-case Filter(false). See the shared checks and predicate retention and origin.
This removes a validation blind spot without a new public API, stored state, or storage format. The cost is temporary node lists and repeated checks above fusion for each arm. Visits and conjuncts use the existing validation budget. This is a reasonable tradeoff for the current two-arm contract. Future operators still require an audit of predicate collection and placement. Five similar changes should extend these shared obligations, not add five plan-shape exceptions. I did not measure latency or throughput.
Replay still captures its read view, checks dataset pins and full-text coverage, recompiles the query, and accepts the plan before execution. The reproduced boundary is the embedded Session::replay_bound_plan API. I found no HTTP or CLI route that accepts these envelopes. The fix does not turn invariant checking into a proof of complete query equivalence. For example, distinct remains outside the exact-equivalence subset.
Validation and regression proof:
- Before: commit
10db7b27a751ce9ed590e5790587b80a06c9951b, the parent of fixf7bbfc573ed1e92cedbe46612d87d2cc482d38a0. I temporarily transplanted the final head's completea_fusion_replayscase unchanged.cargo test --locked -p omnigraph-engine --features failpoints --test integration -j 3 a_fusion_replays -- --nocaptureexecutes one test and fails for the predicted behavior:a filter above the fusion replayed: []. The original query first returns three rows. - After: clean, unmodified head
b8d245048a8aca2df348dfdd24a79d74d3ca7fe3.cargo test --locked -p omnigraph-engine --features failpoints --test integration -j 3 engine_v2 -- --nocapturepasses all 80 selected tests. The same fusion case reaches replay and requiresBadRequestcontainingpredicate origin. - Additional local checks: on production-identical parent
fbb52be1218f4733d32c46fe4585f2b23d42436f,cargo test --locked -p omnigraph-planner -j 3passes 153 tests, with one ignored.cargo test --locked -p omnigraph-gqt --test gq_logic_tests -j 3 -- distinct --test-threads=1passes five cases. Documentation and instruction-link checks pass there. Formatting passes again on the final head. The only change from that parent is the replay test's parent-node rewiring.
All Cargo commands used this prefix to reuse the compatible build cache:
env RUSTUP_TOOLCHAIN=1.97.1 CARGO_TARGET_DIR=/tmp/review-928/target RUSTFLAGS='--cfg tokio_unstable --cfg tokio_unstable' RUST_MIN_STACK=16777216 RUSTC_WRAPPER= rustup run 1.97.1GQT cannot alter a saved physical-plan envelope or call the replay API. The existing Rust replay case owns this contract. Its small extension tests refusal through the real engine boundary. The current setup accepts the fusion's actual parent without changing production plan shape. This gives a behavioral failure before the fix and a pass after it. All temporary test edits were restored; nothing was pushed.
CI and limits: the new CI run is still in progress. The approval rests on the local results and code inspection above, not a claim that all CI has passed. I did not run the full workspace or deployment tests locally. This follow-up covers the fusion fix and intervening acceptance integration; it does not replace every earlier review of the full PR.
I also checked the open PR direction. A shared acceptance boundary fits #815's diagnostics, #805's operator vocabulary, and #792/#793's search contracts. Future polymorphic arms (#884) and actor views (#951) must extend these requirements. Schema changes (#963) and cached read handles (#818/#962) must preserve the captured catalog and dataset authority.
The pinned Lance family remains 11.0.0, source commit ab6b5bbe46009ed78746b444df8db59a8bc5d842. I checked its FTS scan planning, unindexed search path, and probe controls against OmniGraph's callers and the relevant upstream documentation. This fix leaves those storage and search choices unchanged. Upstream Lance #9798 has since merged search_effort; it is absent from the pinned implementation. A future upgrade needs a separate audit of probe-policy precedence and recorded limits. It does not justify a new workaround in this fix.
# Conflicts: # crates/omnigraph-planner/src/gate.rs # crates/omnigraph-planner/src/optimizer.rs # crates/omnigraph-planner/src/physical.rs # crates/omnigraph/src/engine/plan_source.rs
Resolve against the refusal diagnostics (ModernRelay#815) and checked-query read plans (ModernRelay#839): main took bound plan version 8, so the AntiJoin strategy moves to version 9.
RFC 0047 step 2a: every read plan is accepted against its checked query before it runs.
Builds on #815 (step 2, planner refusal diagnostics); its two commits appear here until it merges.
What changes
One acceptance door.
omnigraph-planner/src/validate/constructs the only executable plan type,AcceptedPlan, whose fields are private. The ordinary run,Session::explain_query, theexplainstatement and the inspected run all callaccept_query/accept_query_explained, which share one traced planning run (gate::plan_traced), so explain validates exactly what a run validates.bindtakes anAcceptedPlan;executetakes anAcceptedBoundPlan. A fresh plan failing a check is a planner defect (500); an exhausted validation limit isResourceLimitExceeded(413); a refusal by design keeps its diagnostic (400).Requirements from the checked query, not the IR. The compiled-query cache keeps the
CheckedQuery(declaration and type context) beside the IR. The validator derives binding identity, search identity and approximation, eligibility conjuncts (checked perrrfarm), correlated blocks, the projection and score origin, order and cut, and the declared search policies, and relates written expressions to the IR through the compiler's lowering patterns (constant folding is checked against the engine's own evaluator). Order is recomputed from the operators over typedOrderKeys; a Sort's declared ordering is now its whole comparator.Exact fragment. A single-binding query with Boolean/integer/String eligibility, an optional leading
bm25()over a String property, property sort keys, a limit and a projection of properties and the score is a member. The optimizer records the rules it applies (absorb_scan_filter,prune_scan_columns,lower,rank_bm25_scan); the validator checks the IR is the declaration's lowering, rebuilds the canonical chain, re-applies every rule against its precondition and requires the plan to match. Members are accepted asexact_subset, everything else asinvariants_only;explainreports it asvalidation.scope, and GQT claims it withvalidation scope <scope>.Declared search policy. A
nearestscan declares itsNearestPolicy(probe factor, flat rescan of unreached rows, uncapped fail-closed rescan, flat start); every pre-pass declareson_emptyandcoverage_admits. The ladder and both gates read them from the plan (the rrf gate's coverage guard is now the recorded fact, not a run-time dataset read). The report and the profile record each gate verdict with its counts and each probe attempt per rung.Replay through acceptance.
Session::replay_bound_plantakes a serializedReplayEnvelope(query source and name, scope, schema digest, bound plan, a member's derivation), checks its byte limit and versions before decoding, re-establishes every recorded full-text coverage fact from the pinned snapshot, recompiles the query, and accepts the plan again. Version mismatch, another schema and an unpinned dataset are conflicts (409); an edited plan, derivation or coverage claim is invalid evidence (400).Rework onto main (typed IR, index facts, key narrowing, late materialization, step 5)
Main moved under this PR: the typed lowering (#901), index facts and key narrowing (#924), late materialization (#887), metadata-only schema evolution (#871) and RFC 0047 step 5 (#938). The core design held up under a commit-by-commit review; these are the adaptations, each recorded in RFC 0047 (text and a 2026-10-10 decision-log entry).
Requirements from the typed lowering. Every
QueryIRexpression now carries its compiler type and every conversion is an explicitCast, so the validator derives its requirements from the compiler's lowering of the checked declaration (AcceptInput.ir) and compares plan expressions with it by equality, types included. The lowering is first checked against the declaration (traversal, block, return item, order key and limit counts); the declaration supplies only which return items and order key are the written score calls. The constant evaluator hook and the AST-to-IR matcher are gone.Two-sided predicates and traversal identity. Retention required every written conjunct to be tested; origin now also requires every tested conjunct to be a written one, a constant
true, or the key-narrowing identity equality. A plan that adds a predicate the query never wrote is refused (predicate origin). Each traversal keeps its endpoints, edge selection, direction, endpoint types and hop bounds (traversal identity).One gate. The well-formedness checks #901 added to
execute(aggregate specs, output schemas, fusion row keys) run as acceptance's first phase. Execution keeps only checks that need the snapshot or bound values (traversal admission,validate_plan_values). A saved plan that fails any check is a 400 at the replay door, before execution; the #901 drift tests now assert that refusal, and acceptance names a retyped conjunct or sort key as such ("with stored types the query does not give it").Key narrowing and hydration as checked rules.
narrow_keyadmits the@id = <id>conjunct #924 adds beside a written String key equality only when acceptance re-derives<id>through the catalog (AcceptInput.keys, the loader's derivation);hydrate_return_columnscovers #887'sHydrateColumnsroot. Without the first, every exact-fragment member with a key equality would have failed acceptance as a planner defect.RULES_VERSION2.Step 5 order checks. The fusion and search-ordered aggregate exemptions are gone: a fusion's final sort must lead with
<binding>._rrfdescending, then the written keys and the tie-break; an aggregate's with its remaining keys and then every unwritten group key; anrrfprojection's score must come from an arm. A fusion no longer cuts rows.SEMANTICS_VERSION2,BOUND_PLAN_VERSION8 (main took 7 forcount(distinct)).Full-text coverage from index facts.
FullTextCoverage::of_factsderives the fact from #924'sIndexFacts (the separateTableStore::fts_coveragereader is removed):fullonly when every inverted segment on the column reports known coverage of every fragment; unknown coverage is never full. Replay derives it again from the pinned dataset.Found while porting, each red before its fix:
$e.@typelowers to a constant, and the tie-break check did not count a constant return item as ordered, so a wildcard query ordering by every return failed acceptance (validate/tests.rs::a_constant_return_needs_no_tiebreak).orderclause whose keys all fold to constants (order { $e.@type }over one edge type) still sorts by the identity tie-break; the requirements had dropped the constant keys and then refused the sort as unwritten (validate/tests.rs::a_constant_order_still_sorts, and the GQT runner self-testreference_comparison_refuses_named_type_access_in_every_read_position).cases/v2/order_key_bound_to_an_equally_lowered_return_item.gqt).For @azimafroozeh: the single gate moves #901's plan-shape checks from
executeinto acceptance; a forged saved plan is now a bad request at the replay door instead of an internal error from execution.Merging main after the rework. Main's typed checked query (#947) replaced the type context this PR read and added a
CheckedQueryenum of its own, so this PR's wrapper is nowCheckedRead(the declaration beside main'sCheckedReadQuery), andCheckedReadQuerygained two read accessors,bindings()andtraversals(), which the validator uses. With #909, a declared full-text column carries an untrained segment (empty bitmap) until postings are built; coverage is the union of the column's segments, so such a column is partial and a bm25 scan filters after scoring there (docs/dev/lance.md). The merge of main'sreturn distinct(#957) in10db7b27also teaches acceptance its planning: distinct rows are group rows, so order keys bind to return items, the final sort is total through every unwritten group key with no identity tie-break, and the query is outside the exact fragment (validate/tests.rs::a_distinct_return_sorts_its_groups, red without the change; the three distinct GQT cases pass).Verification of the rework (local, at
dbd491d4): compiler, planner and the engine owners (lib, integration, search, schema_apply, forbidden_apis, Lance surface guards, scrubbed replay) green; GQT corpus 336/336 andrunner_dispatch26/26;omnigraph-benchlib green (with CI'sRUST_MIN_STACK); workspace and GQT clippy, fmt,check-docs.py,check-agents-md.shandtyposclean. Three merge-heavy GQT cases came within about 1.4 s of the 10 s case budget while the machine was loaded.Behavior changes
cases/v2/bm25_score_ignores_a_filter_over_an_unindexed_fragment.gqt(the old prefilter scored the tail row 0.41299206 under a filter, 0.3903352 without), Lance guardfts_prefilter_changes_unindexed_scores_and_postfilter_keeps_them.__anon_*) before named bindings, contrary to RFC 0047's comparator; the validator refused that plan. Regressioncases/v2/anonymous_endpoint_ties_order_by_declared_bindings.gqt.explaingainsvalidation, a bm25 scan'seligibility, a nearest scan'spolicyand the assumptions'full_textcoverage; saved plans move tobound_plan_version8.wildcard_replay_keeps_captured_members_and_pins_every_member_issue_659now asserts that refusal.RFC 0047 amendments
Recorded in the RFC's text and decision log: the canonical form is the planner's resolve order; the derivation stores rule applications, not successors; the rule catalogue as built; BM25 eligibility before scoring only under full coverage; three tie-break omissions as comparator equivalences; correlated blocks required as blocks; evidence memory bounded by byte and node limits (no planning pool exists); the fusion and search-ordered aggregate order checks land with step 5, which implements that order.
Verification
cargo test --workspace --exclude omnigraph-gqt --exclude omnigraph-dst --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints: 122 test targets green (run before the last review fix; the planner suite, the replay owners and clippy pass after it)cargo test -p omnigraph-gqt --locked: 238 of 239 cases pass locally;issue_782_multi_hop_limit_stops_earlyhits its 10 s budget under local machine load (it takes 8.33 s inmain's own CI, so it is near its budget there too)cargo clippy --workspace --all-targets --locked -- -D warnings -W clippy::dbg_macro,cargo fmt --all --check,scripts/check-docs.py,scripts/check-agents-md.sh,typos: cleanvalidate/tests.rs): one broken plan per check, membership over admitted and excluded forms, derivation rejections, entry parity under an exhausted limit;derivation_cost_grows_with_the_query(instrument:) prints evidence growth: 256 conjuncts take about 58 KB, 261 rule applications and 11 ms.cases/v2/order_key_bound_to_an_equally_lowered_return_item.gqt), and a replay trusted a forged full-text coverage claim (a_replay_rechecks_recorded_full_text_coverage). A second review of the fix found nothing.main445f50f) found two more, both fixed with regressions. A constant order key thatreturnprojects under an alias (return { $x as x } order { $x }, alsonow()and a single edge type's grouped@type) failed acceptance as a planner defect, because the planner binds the key to the alias and the order check did not resolve the alias before discounting constants; it now resolves an alias to thereturnitem it names on the written side and on the planned side alike, soorder { x }andorder { $x }are judged the same way (cases/v2/constant_order_key_bound_to_a_return_alias.gqt,validate/tests.rs::a_constant_key_bound_to_a_return_alias_is_accepted). And a fusion returned without recording itsnearestarms' probe attempts, so inspection and replay could not see them; every nearest scan now reports under its own plan node and a fusion records each arm's attempts after its gate (engine_v2_plan_replay.rs::a_fusion_replays, extended with a two-armrrf(nearest, nearest)).