feat(index): support flattened JSON sub-doc indexing

[wirybeaver]

Motivation

Lance JSON inverted indexes should preserve array/object structure so path queries over arrays of objects can distinguish exact positions from wildcards. Tantivy documents a known JSON-index flaw: because an array document is treated as a bag of terms, cart.product_type:sneakers AND cart.attributes.color:red can match a document where sneakers and red live in different array objects. Lance's current single-token-stream JSON index has the same class of problem.

This PR introduces FlattenedSubDocs mode for new JSON indexes, flattening each JSON row into one or more sub-documents so those terms must match inside the same flattened array element before the result is mapped back to the original Lance row id.

It also keeps existing JSON indexes readable and provides two controls for documents that would produce many sub-docs: disable_cross_array_unnest=true trades cross-array query accuracy for lower growth, while max_sub_docs_per_row places an explicit per-row ingestion limit.

Summary

• Add explicit JSON tokenizer modes: SingleDocument for existing JSON indexes and FlattenedSubDocs for new JSON indexes.
• Flatten JSON arrays into multiple internal inverted-index sub-docs, each mapped back to the original Lance row id through DocSet.
• Normalize bracketed JSON query paths into value tokens plus $idx constraint tokens, and deduplicate flattened JSON search results by row id using max score.
• Add persisted/index param disable_cross_array_unnest, default false. The default preserves exact Cartesian-product semantics. When set to true, sibling arrays are indexed independently to avoid sub-doc explosion, matching Pinot's DisableCrossArrayUnnest memory tradeoff.
• Add optional persisted/index param max_sub_docs_per_row. It is unlimited when unset. When set, Lance checks the flattened count before allocating the full Cartesian product.
• Add persisted enum param max_sub_docs_per_row_exceed_action, default fail. fail aborts ingestion and suggests increasing the cap or setting disable_cross_array_unnest=true; skip_row omits the offending source row and continues.
• Expose all JSON ingestion controls through Rust, Python, and Java inverted-index params.

Error Handling Implementation

• The recursive flattener checks array accumulation and sibling-array expansion against max_sub_docs_per_row before allocating the complete set of flattened sub-docs. No limit check is performed when the option is unset.
• Limit failures are returned to the shared JsonTokenizer boundary, so immutable index builds and MemWAL indexing use the same behavior.
• With max_sub_docs_per_row_exceed_action=fail, tokenization returns an invalid-input error before any sub-doc from that source row is indexed. The error recommends increasing max_sub_docs_per_row or setting disable_cross_array_unnest=true.
• With max_sub_docs_per_row_exceed_action=skip_row, the tokenizer logs a warning and returns no token streams. Existing writer loops therefore omit the entire source row and continue indexing subsequent rows.

Example 1 - Used to explain the idea

Raw Documents

doc-0

{"foo":[{"bar":["x","y"]}]}

doc-1

{"foo":[{"bar":["y"]},{"bar":"z"}]}

Flattened Documents

Flatten ID	Flattened Path	Value
0	foo[0].bar[0]	"x"
1	foo[0].bar[1]	"y"
2	foo[0].bar[0]	"y"
3	foo[1].bar	"z"

Token Dictionary

Each token is represented as: (path, type, value)

Path	Type	Value	Flatten Posting
foo..bar.	str	x	0
foo..bar.	str	y	1, 2
foo..bar	str	z	3
foo$idx	num	0	0, 1, 2
foo$idx	num	1	3
foo..bar$idx	num	0	0, 2
foo..bar$idx	num	1	1

Notes

• Prior array index [<idx>] is compressed to . when walking through the path from left to right.
• foo$idx represents the index of the element in the foo array.
• foo..bar$idx represents the index inside the bar array.

Search Query A

Query:

foo[0].bar[0] = "y"

Lookup tokens:

Token	Flatten Posting
foo..bar., str, "y"	1, 2
foo..bar$idx, num, 0	0, 2
foo$idx, num, 0	0, 1, 2

Flatten Posting Intersection:

(1, 2) intersection (0, 2) intersection (0, 1, 2) = {2}

Lookup original document:

Flatten ID 2
    ->
Doc ID 1

Search Query B

Query:

foo[0].bar[*] = "y"

Lookup tokens:

Token	Flatten Posting
foo..bar., str, "y"	1, 2
foo$idx, num, 0	0, 1, 2

Notes

• When there's a [*] in the search query, we don't need to generate index constraint token prefix$idx to match the specific array index on that position.

Flatten Posting Intersection:

(1, 2) intersection (0, 1, 2) = {1, 2}

Lookup original document:

Flatten ID [1, 2]
    ->
Doc ID [0, 1]

Example 2: sibling arrays

{"foo":[{"bar":["x","y"]},{"bar":["a","b"]}],"foo2":["u"]}
{"foo":[{"bar":["y","z"]}],"foo2":["u"]}

Expected flattened sub-docs:

row0/sub0: foo$idx=0, foo..bar$idx=0, foo..bar.=x, foo2$idx=0, foo2.=u
row0/sub1: foo$idx=0, foo..bar$idx=1, foo..bar.=y, foo2$idx=0, foo2.=u
row0/sub2: foo$idx=1, foo..bar$idx=0, foo..bar.=a, foo2$idx=0, foo2.=u
row0/sub3: foo$idx=1, foo..bar$idx=1, foo..bar.=b, foo2$idx=0, foo2.=u
row1/sub0: foo$idx=0, foo..bar$idx=0, foo..bar.=y, foo2$idx=0, foo2.=u
row1/sub1: foo$idx=0, foo..bar$idx=1, foo..bar.=z, foo2$idx=0, foo2.=u

Query behavior:

foo[0].bar[0],str,y -> row1 only
foo[0].bar[*],str,y -> row0 and row1

Example 3: disable_cross_array_unnest

Input:

{"a":["x","y"],"b":["u","v"],"c":1}

Default disable_cross_array_unnest=false produces exact Cartesian-product sub-docs:

sub0: a$idx=0, a.=x, b$idx=0, b.=u, c=1
sub1: a$idx=0, a.=x, b$idx=1, b.=v, c=1
sub2: a$idx=1, a.=y, b$idx=0, b.=u, c=1
sub3: a$idx=1, a.=y, b$idx=1, b.=v, c=1

With disable_cross_array_unnest=true, sibling arrays are indexed independently:

sub0: a$idx=0, a.=x, c=1
sub1: a$idx=1, a.=y, c=1
sub2: b$idx=0, b.=u, c=1
sub3: b$idx=1, b.=v, c=1

This avoids combinatorial sub-doc growth. Queries that constrain values across multiple sibling arrays can sacrifice accuracy because no single flattened sub-doc contains terms from both sibling arrays.

Example 4: Tantivy nested-object false positive

Tantivy documents this pitfall for JSON arrays: a document is a bag of terms, so cart.product_type:sneakers AND cart.attributes.color:red can match even when sneakers and red come from different objects in the same cart array. FlattenedSubDocs mode avoids that by preserving each array element as a separate internal inverted-index document.

Raw Documents

row0 should not match because sneakers and red are in different cart objects:

{"cart_id":3234234,"cart":[{"product_type":"sneakers","attributes":{"color":"white"}},{"product_type":"t-shirt","attributes":{"color":"red"}}]}

row1 should match because both terms are in the same cart object:

{"cart_id":3234235,"cart":[{"product_type":"sneakers","attributes":{"color":"red"}}]}

Flattened Sub-Docs

row0/sub0: cart$idx=0, cart..product_type=sneakers, cart..attributes.color=white, cart_id=3234234
row0/sub1: cart$idx=1, cart..product_type=t-shirt, cart..attributes.color=red, cart_id=3234234
row1/sub0: cart$idx=0, cart..product_type=sneakers, cart..attributes.color=red, cart_id=3234235

Correct Query Shape For Nested-Object Semantics

Use one JSON MatchQuery that contains all related JSON triplets separated by ;, and set Operator::And on that MatchQuery:

FtsQuery::Match(
    MatchQuery::new(
        "cart[*].product_type,str,sneakers;cart[*].attributes.color,str,red".to_string(),
    )
    .with_column(Some("json_field".to_string()))
    .with_operator(Operator::And),
)

Expected result:

row1 only

Here ; only separates JSON triplets. Operator::And is what requires all generated tokens to match, and because they are inside one MatchQuery, the intersection happens on flattened sub-doc ids before Lance maps matches back to row ids.

Anti-Example: Row-Level Boolean Composition

Do not express nested-object constraints as separate BooleanQuery.must children:

FtsQuery::Boolean(BooleanQuery {
    must: vec![
        MatchQuery::new("cart[*].product_type,str,sneakers".to_string())
            .with_column(Some("json_field".to_string()))
            .into(),
        MatchQuery::new("cart[*].attributes.color,str,red".to_string())
            .with_column(Some("json_field".to_string()))
            .into(),
    ],
    should: vec![],
    must_not: vec![],
})

Each child MatchQuery runs independently and returns Lance row ids. The outer boolean query then composes those row ids, so row0 can still match at row level even though no single cart object contains both terms.

Apache Pinot JSON Index References

• JSON flattening behavior and array index keys: JsonUtils.flatten
• Pinot disableCrossArrayUnnest config field: JsonIndexConfig
• Mutable JSON index SPI, including write entrypoint and reader inheritance: MutableJsonIndex
• Immutable JSON index writer SPI: JsonIndexCreator
• Immutable JSON index reader SPI: JsonIndexReader
• Immutable writer implementation: BaseJsonIndexCreator
• Mutable implementation read/write behavior: MutableJsonIndexImpl
• Immutable reader implementation: ImmutableJsonIndexReader

Test Plan

Rust:

• cargo fmt --all --check
• cargo check --workspace --tests --benches (passed)
• cargo clippy --all --tests --benches -- -D warnings (passed)
• cargo test -p lance-index scalar::inverted::tokenizer:: --lib (52 passed)

Python:

• rebuilt the extension
• uv run pytest python/tests/test_scalar_index.py::test_json_inverted_match_query from python/ (1 passed)
• uv run make lint from python/ (passed; existing missing-stub warnings only)

Java/JNI:

• ./mvnw spotless:check from java/ (passed)
• focused InvertedIndexParamsTest (14 passed)
• JNI Rust tests (22 passed)

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[wirybeaver]

@beinan @jja725 Would you mind taking a look at this Json Index improvement proposal shown in the PR summary? If no major concerns, I would go ahead to refine the PR.

[westonpace]

This seems like a useful addition to the JSON full text search. I think disable_cross_array_unnest is going to be pretty technical for the user to judge. It might be nice to have some guardrails. For example, if a single document generates more than X subdocs (or maybe more than X subdocs per byte of the original doc) we could log a warning like "more than X subdocuments generated for a single document, if you have sibling arrays you may need to disable cross array unnesting to avoid a combinatorial explosion". Or we could have a "max_subdocs_per_doc" variable that, if exceeded, would fail the index build.

In the other direction we could potentially add some kind of conservative one-time warning. If the user disabled cross array unnesting and then queried the index using multiple constraints. We wouldn't easily know they were multiple sibling array constraints but the warning could be something like "you are querying a JSON text index with multiple constraints and the index was built with cross array unnesting disabled. If those constraints are on two different sibling arrays you may get false negatives. To disable this warning set the environment variable ..."

Ideally we will want some examples added to the docs as well. We could probably expand https://lance.org/guide/json/#full-text-search-on-json-documents

[westonpace]

Should we make this an enum? There are only two choices at the moment and I wouldn't expect there to be many.

[westonpace]

How does the user specify the json tokenizer mode? Is it through the base_tokenizer?

[jja725]

Thanks for this great contribution! this would help a lot with FTS

[jja725]

should we add some error trace here before return

[jja725]

maybe we can Add a max_sub_docs_per_row cap with a clear error or warning when exceeded

[wirybeaver]

@jja725 @westonpace Sorry for the late reply. Somehow I miss the email notification about your folks' feedback during vacation. I will work on the improvement this week. Appreciate you folks compliment on this idea.

In terms of adding the parameter "max_sub_docs_per_row", do you folks prefer failing the whole indexing process OR logging an error but keep the already flattened docs?

[westonpace]

In terms of adding the parameter "max_sub_docs_per_row", do you folks prefer failing the whole indexing process OR logging an error but keep the already flattened docs?

Either approach is fine. Failing the entire operation is safest but could be quite annoying if there is only one bad row. In vector indexing, if we encounter NaN / NULL embeddings, we just exclude that row from the index (so it will never appear in search) so there is some precedence for just skipping too.

[wirybeaver]

Following up on max_sub_docs_per_row, I propose adding two JSON index params before implementation:

• max_sub_docs_per_row: optional cap on how many flattened sub-docs one source JSON row may produce. If unset, behavior remains unlimited, so Lance will keep flattening until normal memory/resource limits apply.
• max_sub_docs_per_row_exceed_action: enum scoped to this cap, with initial values fail and skip_row. fail aborts indexing with an error that suggests increasing max_sub_docs_per_row or setting disable_cross_array_unnest=true; skip_row omits the whole offending source row from the index and continues.

Default proposal: unset max_sub_docs_per_row and max_sub_docs_per_row_exceed_action=fail, so existing behavior is unchanged unless a user opts into the cap.

I'll wait for feedback from @westonpace and @jja725 before implementing this.