From 4ec6e62e88a1f844e83c2692009dee2fba976164 Mon Sep 17 00:00:00 2001 From: jackylee-ch Date: Wed, 26 Aug 2026 15:14:34 +0800 Subject: [PATCH 1/6] docs: correct the SEARCH table function reference search_columns is required (LanceSearchTableFunctions.search throws when it is empty) but has no positional slot, so the documented positional example always fails and SEARCH in fact requires named arguments. offset is documented but never read by search(); unknown named arguments are silently ignored, so it looks accepted. Execution described the removed bespoke single-partition path; the scan now runs server-side through queryTable only when the namespace supports it, and per-fragment otherwise. Validation claimed Docker coverage while the pytest case is xfail and the JVM case is @Disabled. --- docs/src/operations/dql/search.md | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/docs/src/operations/dql/search.md b/docs/src/operations/dql/search.md index c112155ee..0e0db44af 100644 --- a/docs/src/operations/dql/search.md +++ b/docs/src/operations/dql/search.md @@ -6,10 +6,10 @@ Run Lance full-text search from Spark SQL using Lance namespace execution. `SEARCH` requires the Lance Spark SQL extension to be enabled. See [Spark SQL Extensions](../../config.md#spark-sql-extensions) for configuration details. !!! note "Namespace Tables Required" - `SEARCH` resolves the `table` argument through a Spark catalog and executes through the Lance namespace `queryTable` API. Use a Lance namespace catalog table such as `lance.default.documents`, not a raw Lance dataset path. + `SEARCH` resolves the `table` argument through a Spark catalog. Use a Lance namespace catalog table such as `lance.default.documents`, not a raw Lance dataset path. -!!! note "Named Arguments" - Named arguments require Spark 3.5 or later. On Spark 3.4, use the positional form. +!!! note "Named Arguments Required" + `search_columns` is required and has no positional slot, so `SEARCH` must be called with named arguments. Named arguments require Spark 3.5 or later, so `SEARCH` is not available on Spark 3.4. ## Basic Usage @@ -42,27 +42,16 @@ Run Lance full-text search from Spark SQL using Lance namespace execution. See [CREATE INDEX](../ddl/create-index.md#full-text-search-index) for FTS index options. -## Positional Form - -Use positional arguments for simple calls and Spark 3.4 compatibility. - -=== "SQL" - ```sql - SELECT * - FROM SEARCH('lance.default.documents', 'lance', 5); - ``` - ## Arguments | Argument | Type | Required | Description | |----------|------|----------|-------------| | `table` | String | Yes | Catalog table name to search. | | `query` or `search_query` | String | Yes | Full-text query string. | -| `search_columns` | Array string literal | No | Text columns to search. When omitted, Lance uses the indexed columns configured for the FTS index. | +| `search_columns` | Array string literal | Yes | Text columns to search. | | `num_results`, `limit`, or `k` | Integer | No | Number of results. Defaults to `10`. | | `columns` | Array string literal | No | Output table columns. `_score` is always included. Use `array('*')` or omit this argument for all table columns. | | `filter` | String | No | SQL filter expression evaluated by Lance. | -| `offset` | Integer | No | Number of results to skip. | | `version` | Long | No | Lance table version to search. | | `with_row_id` | Boolean | No | Include Lance row ids in the result as `_rowid`. | @@ -72,8 +61,8 @@ The result includes the requested table columns and a nullable `_score` float co ## Execution -Spark plans `SEARCH` as a DataSource V2 batch read with one input partition. The partition reader calls the Lance namespace `queryTable` API. With a directory namespace the search runs in the Spark process executing that reader; with a REST namespace the REST server handles the namespace request. +Spark plans `SEARCH` as a batch read carrying the full-text query as a scan option, wrapped in an optional filter, a projection, `ORDER BY _score DESC`, and `LIMIT k`. The scan then runs one of two ways: a single-partition server-side read through the Lance namespace `queryTable` API when the namespace supports it, or a distributed per-fragment scan for catalog-only namespaces and for reads that target a branch or tag. ## Validation -The Docker integration suite covers `SEARCH` against the directory namespace and a REST namespace backed by a directory namespace. The `Spark Search Docker` GitHub Actions workflow runs both backends for pull requests. +`SEARCH` has no passing end-to-end coverage today: the Docker integration test is marked `xfail` and the JVM test is `@Disabled`, both pending structured full-text query support in lance-core. [`VECTOR_SEARCH`](vector-search.md) and [`HYBRID_SEARCH`](hybrid-search.md) remain covered by the `Spark Search Docker` workflow. From 52baafa7a9185e29c563d16175052403021f6af2 Mon Sep 17 00:00:00 2001 From: jackylee-ch Date: Thu, 3 Sep 2026 11:03:31 +0800 Subject: [PATCH 2/6] docs: drop the branch and tag clause from the SEARCH scan path Review feedback: a tag cannot reach SEARCH at all. The version argument goes through optionalLong, so a tag name never parses, and there is no tag_ identifier suffix to carry one. A branch is reachable, since resolveLanceTable calls loadTable and that matches BRANCH_SUFFIX on the identifier, but nothing tests it. Leave the condition users can act on and drop the rest. --- docs/src/operations/dql/search.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/operations/dql/search.md b/docs/src/operations/dql/search.md index 0e0db44af..dd3847bf0 100644 --- a/docs/src/operations/dql/search.md +++ b/docs/src/operations/dql/search.md @@ -61,7 +61,7 @@ The result includes the requested table columns and a nullable `_score` float co ## Execution -Spark plans `SEARCH` as a batch read carrying the full-text query as a scan option, wrapped in an optional filter, a projection, `ORDER BY _score DESC`, and `LIMIT k`. The scan then runs one of two ways: a single-partition server-side read through the Lance namespace `queryTable` API when the namespace supports it, or a distributed per-fragment scan for catalog-only namespaces and for reads that target a branch or tag. +Spark plans `SEARCH` as a batch read carrying the full-text query as a scan option, wrapped in an optional filter, a projection, `ORDER BY _score DESC`, and `LIMIT k`. The scan then runs one of two ways: a single-partition server-side read through the Lance namespace `queryTable` API when the namespace supports it, or a distributed per-fragment scan for catalog-only namespaces. ## Validation From 4ea4cb31a7af6afe7721dfc87b401f1e50af1f4b Mon Sep 17 00:00:00 2001 From: jackylee-ch Date: Thu, 3 Sep 2026 11:24:43 +0800 Subject: [PATCH 3/6] docs: make the per-fragment scan the general SEARCH fallback I over-corrected: only the tag half of the previous clause was wrong, and dropping branch too made the routing read as if catalog-only namespaces were the sole fallback trigger. State the fallback as the otherwise case and list the three conditions the server-side route needs, matching shouldNamespaceFtsScan: a namespace that implements queryTable, a ref that is not a branch or tag, and no pushed aggregation. Enumerating only the fallback triggers went stale twice. --- docs/src/operations/dql/search.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/operations/dql/search.md b/docs/src/operations/dql/search.md index dd3847bf0..020de7ae2 100644 --- a/docs/src/operations/dql/search.md +++ b/docs/src/operations/dql/search.md @@ -61,7 +61,7 @@ The result includes the requested table columns and a nullable `_score` float co ## Execution -Spark plans `SEARCH` as a batch read carrying the full-text query as a scan option, wrapped in an optional filter, a projection, `ORDER BY _score DESC`, and `LIMIT k`. The scan then runs one of two ways: a single-partition server-side read through the Lance namespace `queryTable` API when the namespace supports it, or a distributed per-fragment scan for catalog-only namespaces. +Spark plans `SEARCH` as a batch read carrying the full-text query as a scan option, wrapped in an optional filter, a projection, `ORDER BY _score DESC`, and `LIMIT k`. The scan then runs either as a single-partition server-side read through the Lance namespace `queryTable` API, or otherwise as a distributed per-fragment scan. The server-side route requires a namespace that implements `queryTable`, a read that does not target a branch or tag, and no pushed-down aggregation. ## Validation From fca528fe7f0a272ee1d5d367b37b520ab6128e71 Mon Sep 17 00:00:00 2001 From: jackylee-ch Date: Tue, 15 Sep 2026 15:24:31 +0800 Subject: [PATCH 4/6] docs: correct SEARCH validation coverage --- docs/src/operations/dql/search.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/operations/dql/search.md b/docs/src/operations/dql/search.md index 020de7ae2..860663bf0 100644 --- a/docs/src/operations/dql/search.md +++ b/docs/src/operations/dql/search.md @@ -65,4 +65,4 @@ Spark plans `SEARCH` as a batch read carrying the full-text query as a scan opti ## Validation -`SEARCH` has no passing end-to-end coverage today: the Docker integration test is marked `xfail` and the JVM test is `@Disabled`, both pending structured full-text query support in lance-core. [`VECTOR_SEARCH`](vector-search.md) and [`HYBRID_SEARCH`](hybrid-search.md) remain covered by the `Spark Search Docker` workflow. +The `Spark Search Docker` workflow exercises `SEARCH` against directory and REST-directory namespaces. The Docker test still carries an `xfail` marker, but it can pass as `XPASS`; the JVM `SEARCH` cases remain `@Disabled`. [`VECTOR_SEARCH`](vector-search.md) and [`HYBRID_SEARCH`](hybrid-search.md) are also exercised by the workflow. From 60ad4c74e24633d228b99146596dab99509f13a1 Mon Sep 17 00:00:00 2001 From: jackylee-ch Date: Wed, 16 Sep 2026 22:31:58 +0800 Subject: [PATCH 5/6] docs: warn that search fails on the server-side route The Validation section leaned on xfail/XPASS/@Disabled and read as "tested and passing", while the Basic Usage example selects _score and therefore fails on a queryTable-capable namespace. Lead with that instead, in the words the repo's own xfail reason uses, and collapse the CI mechanics into one coverage line. Also correct "not available on Spark 3.4": the 3.4 module injects the search table function identically to 3.5 and 4.x, so it is registered but uncallable there. --- docs/src/operations/dql/search.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/src/operations/dql/search.md b/docs/src/operations/dql/search.md index 860663bf0..9982583a8 100644 --- a/docs/src/operations/dql/search.md +++ b/docs/src/operations/dql/search.md @@ -9,7 +9,13 @@ Run Lance full-text search from Spark SQL using Lance namespace execution. `SEARCH` resolves the `table` argument through a Spark catalog. Use a Lance namespace catalog table such as `lance.default.documents`, not a raw Lance dataset path. !!! note "Named Arguments Required" - `search_columns` is required and has no positional slot, so `SEARCH` must be called with named arguments. Named arguments require Spark 3.5 or later, so `SEARCH` is not available on Spark 3.4. + `search_columns` is required and has no positional slot, so `SEARCH` must be called with named arguments. Named arguments require Spark 3.5 or later. On Spark 3.4 the function is registered but cannot be called. + +!!! warning "Full-text search is not supported on the server-side route yet" + A namespace that implements `queryTable` — a directory namespace, for example — takes the + server-side route described in [Execution](#execution). That route does not run the structured + full-text query, so the `_score` column the plan projects and sorts by cannot be produced and + the query fails. ## Basic Usage @@ -65,4 +71,4 @@ Spark plans `SEARCH` as a batch read carrying the full-text query as a scan opti ## Validation -The `Spark Search Docker` workflow exercises `SEARCH` against directory and REST-directory namespaces. The Docker test still carries an `xfail` marker, but it can pass as `XPASS`; the JVM `SEARCH` cases remain `@Disabled`. [`VECTOR_SEARCH`](vector-search.md) and [`HYBRID_SEARCH`](hybrid-search.md) are also exercised by the workflow. +The `Spark Search Docker` workflow covers `SEARCH`, [`VECTOR_SEARCH`](vector-search.md) and [`HYBRID_SEARCH`](hybrid-search.md) against directory and REST-directory namespaces. The `SEARCH` cases are currently expected to fail, for the reason given at the top of this page. From 201b57c1e9308574c032ed297dc6598949d36d62 Mon Sep 17 00:00:00 2001 From: jackylee-ch Date: Wed, 16 Sep 2026 23:47:37 +0800 Subject: [PATCH 6/6] docs: drop the unproven search failure warning I added a warning saying the server-side route cannot produce _score, taking the xfail reason and the review note as fact without reproducing it. Running the disabled JVM case on this head shows a different failure: it throws "SEARCH requires search_columns for full-text search" at analysis time, because the case uses the positional form. It never reaches execution, so nothing here demonstrates the _score claim. Remove the warning and the matching Validation claim, and state only the coverage the workflow provides. The Spark 3.4 rewording stays. --- docs/src/operations/dql/search.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/docs/src/operations/dql/search.md b/docs/src/operations/dql/search.md index 9982583a8..eddf20f56 100644 --- a/docs/src/operations/dql/search.md +++ b/docs/src/operations/dql/search.md @@ -11,12 +11,6 @@ Run Lance full-text search from Spark SQL using Lance namespace execution. !!! note "Named Arguments Required" `search_columns` is required and has no positional slot, so `SEARCH` must be called with named arguments. Named arguments require Spark 3.5 or later. On Spark 3.4 the function is registered but cannot be called. -!!! warning "Full-text search is not supported on the server-side route yet" - A namespace that implements `queryTable` — a directory namespace, for example — takes the - server-side route described in [Execution](#execution). That route does not run the structured - full-text query, so the `_score` column the plan projects and sorts by cannot be produced and - the query fails. - ## Basic Usage `SEARCH` returns the selected table columns plus `_score`. Create an FTS index before querying text columns. @@ -71,4 +65,4 @@ Spark plans `SEARCH` as a batch read carrying the full-text query as a scan opti ## Validation -The `Spark Search Docker` workflow covers `SEARCH`, [`VECTOR_SEARCH`](vector-search.md) and [`HYBRID_SEARCH`](hybrid-search.md) against directory and REST-directory namespaces. The `SEARCH` cases are currently expected to fail, for the reason given at the top of this page. +The `Spark Search Docker` workflow covers `SEARCH`, [`VECTOR_SEARCH`](vector-search.md) and [`HYBRID_SEARCH`](hybrid-search.md) against directory and REST-directory namespaces.