Skip to content

feat(index): support flattened JSON sub-doc indexing #7445

Description

@wirybeaver

Parent: #4516
Draft PR: #7377

Motivation

Lance JSON inverted indexes should preserve array/object structure so path queries over arrays of objects can distinguish exact positions, wildcards, and nested-object constraints.

Tantivy documents a known flaw of bag-of-terms JSON array indexing: a query like cart.product_type:sneakers AND cart.attributes.color:red can match even when sneakers and red live in different objects in the same array. Lance's current single-token-stream JSON index has the same class of problem.

Proposal

Introduce FlattenedSubDocs mode for new JSON inverted indexes. A JSON row can flatten into multiple internal sub-documents, and matches are mapped back to the original Lance row id. This lets related JSON path constraints match inside the same flattened array element before row-id remapping.

Also add disable_cross_array_unnest=false by default, with an opt-in memory-safety tradeoff for documents with many sibling arrays, following Pinot's DisableCrossArrayUnnest behavior.

Expected behavior

  • Existing JSON indexes remain readable using SingleDocument mode.
  • New JSON indexes use FlattenedSubDocs mode.
  • Positional and wildcard queries can return different row sets, for example foo[0].bar[0],str,y vs foo[0].bar[*],str,y.
  • Nested-object constraints should be expressed as one JSON MatchQuery with semicolon-separated triplets plus Operator::And, so matching happens at flattened sub-doc granularity.
  • Separate BooleanQuery.must child queries remain row-level composition and are not the query shape for nested-object semantics.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions