feat(mem_wal): evaluate cross-column predicates on the fresh tier

[hamersaw]

Follow-up to #9172. That PR gave the fresh tier cross-column multi-match by decomposing it into per-column searches; this one gives it cross-column predicates, which cannot be decomposed.

The problem

plan_search bound the whole query to one column, so a boolean or boost whose leaves named different fields had nowhere to go and was refused:

// must: [alpha in title, beta in body]
FtsQuery::Boolean(BooleanQuery::new(vec![
    (Occur::Must, MatchQuery::new("alpha").with_column(Some("title"))),
    (Occur::Must, MatchQuery::new("beta").with_column(Some("body"))),
]))

That is one predicate over both fields — an intersection — so the per-column decomposition #9172 uses for multi-match would answer a different question. It refused rather than guess, and cross_column_boolean_is_refused pinned that.

The committed path has no such gap: CrossColumnCompoundQueryExec serves the bounded fully-indexed shape and the DataFusion fallback serves the rest. Only the fresh tier refused.

The clause algebra was never missing

The committed compound scorer sums MUST (RequiredConjunctionScorer), sums SHOULD into the surviving set (DisjunctionScore::Sum), excludes MUST_NOT, and subtracts negative_boost * negative_score for a boost. search_boolean and search_boost on the memtable already did exactly that, over a HashMap<DocumentKey, f32>.

What was missing was the ability to reach two indexes. FtsQueryExpr carried no column on its leaves, the scanner's FtsQuery was { column, expr }, and FtsIndexExec resolved a single FtsMemIndex.

The change

The tree walk moves out of the index rather than the column moving into it. FtsMemIndex keeps its leaf scorers and grows search_leaf_bounded; combine_compound holds the clause algebra once, parameterized by a leaf evaluator, and both the single-index walk and the new search_cross_column go through it. One implementation, two routings — rather than a second copy that has to agree with the first.

FtsQueryExpr leaves gain an optional column binding, and FtsQuery::new binds unbound leaves to its column, so every existing caller behaves exactly as before.

Leaves from different indexes need one visibility cut. Each index snapshots its own {partitions, tail} view and those can disagree about how far the memtable has advanced, while filter_by_visibility runs after the combine — so a MUST across columns could drop a row both columns actually contain. search_leaf_bounded takes the exec's max_readable_row and clamps each leaf before the clauses meet. This is the one part of the change that is not mechanical.

The planner generalizes to a column slice. Per-column granularity and index settings move into resolve_column_index_contract and are required to agree. Dropping the single-column rebinding in build_source_plan is what lets the base and SSTable arms use the dataset scanner's own cross-column path unchanged — no new code there at all. Row documents only, matching validate_row_leaf_granularities.

One uncovered column sends every queried column through the transient store. The clauses meet on row_position, and positions are only comparable within one store, so a mixed maintained/transient routing would compare across two. Re-indexing a maintained column costs one tokenize pass over a memtable already paying for the missing one; the alternative buys that pass and costs the invariant.

CompoundQueryExec and CrossColumnCompoundQueryExec gain with_limit. Their top-k lives in FtsSearchParams rather than an enclosing fetch node and their segment selections are private, so a caller over-fetching to survive a later dedup had no way to raise it without rebuilding the node over a wider domain.

Ranking

A cross-column MUST sums two per-column BM25 scores, each local to its own index and its own tier, so the local-scoring divergence this module documents compounds rather than merely applies. Matching stays exact; only order moves. Same trade #9172 recorded for multi-match, one step wider.

Tests

cargo test -p lance --lib mem_wal: 720 passed.

New in fts_search: a cross-column MUST returns only the row matching in both columns; a cross-column MUST_NOT drops the row carrying the excluded term in the other field; a cross-column boost demotes rather than drops, and the demoted row's score is strictly lower; a column with no maintained in-memory index still contributes through the transient store; an unbound leaf is refused.

cross_column_boolean_is_refused is superseded — that restriction is what this lifts.

🤖 Generated with Claude Code

[lance-gatekeeper]

✅ Gate recommendation: approve.

The new head removes the cached column list and derives routing directly from the query tree, preventing the public expression and planner state from diverging. The existing cross-column clause algebra, shared readable-row bound, transient-index fallback, and single-column behavior remain intact; focused verification found no blocking issue.

Please mark this PR with the breaking-change label.