From fc41b65898a01366ee9f1aa60c85ef6dbf3ce520 Mon Sep 17 00:00:00 2001 From: emma Date: Fri, 2 Oct 2026 01:12:19 -0700 Subject: [PATCH 1/5] docs(cosmosdb): filters, projection and limits push down into Cosmos DB --- website/docs/components/data-connectors/cosmosdb/deployment.md | 2 +- website/docs/components/data-connectors/cosmosdb/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/components/data-connectors/cosmosdb/deployment.md b/website/docs/components/data-connectors/cosmosdb/deployment.md index fbf2c4816..ca32c3457 100644 --- a/website/docs/components/data-connectors/cosmosdb/deployment.md +++ b/website/docs/components/data-connectors/cosmosdb/deployment.md @@ -127,7 +127,7 @@ Cosmos DB requests participate in [task history](../../../reference/task_history ## Known Limitations - **Read-only**: Writes (`INSERT` / `UPDATE` / `DELETE`) are not supported. -- **No filter / projection / limit pushdown**: SQL predicates are evaluated locally by DataFusion. Use a custom `query:` to narrow at the Cosmos side. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and `starts_with` on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so results match a query that pushes nothing down. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: Mapping changes after startup require a runtime restart. - **No change feed**: `RefreshMode::Changes` is not wired. - **Mid-stream retries are not safe**: Retries apply to the schema-inference pass only. Errors during a streaming scan propagate immediately; rely on dataset refresh-level retry instead. diff --git a/website/docs/components/data-connectors/cosmosdb/index.md b/website/docs/components/data-connectors/cosmosdb/index.md index 0406200ff..f0ff96764 100644 --- a/website/docs/components/data-connectors/cosmosdb/index.md +++ b/website/docs/components/data-connectors/cosmosdb/index.md @@ -220,7 +220,7 @@ datasets: ## Limitations - **Read-only**: Only `SELECT` scans are supported. Writes (`INSERT` / `UPDATE` / `DELETE`) are not implemented. -- **No filter / projection / limit pushdown into Cosmos**: SQL predicates are evaluated locally by DataFusion after the documents have been streamed in. Use the `query` parameter to narrow at the Cosmos side. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and `starts_with` on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so results match a query that pushes nothing down. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: New fields added in Cosmos after startup are not picked up until the runtime restarts. - **No change feed**: `acceleration.refresh_mode: changes` is not supported. - **No native temporal / decimal / binary types**: Cosmos stores everything as JSON. Round-trip these as strings and `CAST` in SQL. From 9fdbd6fbdf7f95b12df75390d39d5ebde9520fed Mon Sep 17 00:00:00 2001 From: emma Date: Fri, 2 Oct 2026 01:21:24 -0700 Subject: [PATCH 2/5] docs(cosmosdb): name the column types and literal bounds a pushed condition needs --- website/docs/components/data-connectors/cosmosdb/deployment.md | 2 +- website/docs/components/data-connectors/cosmosdb/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/components/data-connectors/cosmosdb/deployment.md b/website/docs/components/data-connectors/cosmosdb/deployment.md index ca32c3457..f711a9f8c 100644 --- a/website/docs/components/data-connectors/cosmosdb/deployment.md +++ b/website/docs/components/data-connectors/cosmosdb/deployment.md @@ -127,7 +127,7 @@ Cosmos DB requests participate in [task history](../../../reference/task_history ## Known Limitations - **Read-only**: Writes (`INSERT` / `UPDATE` / `DELETE`) are not supported. -- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and `starts_with` on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so results match a query that pushes nothing down. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and `starts_with` on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, and an integer literal must be smaller than 2^53 in absolute value. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so results match a query that pushes nothing down. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: Mapping changes after startup require a runtime restart. - **No change feed**: `RefreshMode::Changes` is not wired. - **Mid-stream retries are not safe**: Retries apply to the schema-inference pass only. Errors during a streaming scan propagate immediately; rely on dataset refresh-level retry instead. diff --git a/website/docs/components/data-connectors/cosmosdb/index.md b/website/docs/components/data-connectors/cosmosdb/index.md index f0ff96764..3912df0a4 100644 --- a/website/docs/components/data-connectors/cosmosdb/index.md +++ b/website/docs/components/data-connectors/cosmosdb/index.md @@ -220,7 +220,7 @@ datasets: ## Limitations - **Read-only**: Only `SELECT` scans are supported. Writes (`INSERT` / `UPDATE` / `DELETE`) are not implemented. -- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and `starts_with` on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so results match a query that pushes nothing down. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and `starts_with` on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, and an integer literal must be smaller than 2^53 in absolute value. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so results match a query that pushes nothing down. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: New fields added in Cosmos after startup are not picked up until the runtime restarts. - **No change feed**: `acceleration.refresh_mode: changes` is not supported. - **No native temporal / decimal / binary types**: Cosmos stores everything as JSON. Round-trip these as strings and `CAST` in SQL. From 5f40196fc508e1d1575d40099377b0236ed58220 Mon Sep 17 00:00:00 2001 From: emma Date: Fri, 2 Oct 2026 01:28:25 -0700 Subject: [PATCH 3/5] docs(cosmosdb): scope starts_with pushdown and the decode-error caveat --- website/docs/components/data-connectors/cosmosdb/deployment.md | 2 +- website/docs/components/data-connectors/cosmosdb/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/components/data-connectors/cosmosdb/deployment.md b/website/docs/components/data-connectors/cosmosdb/deployment.md index f711a9f8c..308d1b5d9 100644 --- a/website/docs/components/data-connectors/cosmosdb/deployment.md +++ b/website/docs/components/data-connectors/cosmosdb/deployment.md @@ -127,7 +127,7 @@ Cosmos DB requests participate in [task history](../../../reference/task_history ## Known Limitations - **Read-only**: Writes (`INSERT` / `UPDATE` / `DELETE`) are not supported. -- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and `starts_with` on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, and an integer literal must be smaller than 2^53 in absolute value. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so results match a query that pushes nothing down. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, and an integer literal must be smaller than 2^53 in absolute value. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: Mapping changes after startup require a runtime restart. - **No change feed**: `RefreshMode::Changes` is not wired. - **Mid-stream retries are not safe**: Retries apply to the schema-inference pass only. Errors during a streaming scan propagate immediately; rely on dataset refresh-level retry instead. diff --git a/website/docs/components/data-connectors/cosmosdb/index.md b/website/docs/components/data-connectors/cosmosdb/index.md index 3912df0a4..297cd56aa 100644 --- a/website/docs/components/data-connectors/cosmosdb/index.md +++ b/website/docs/components/data-connectors/cosmosdb/index.md @@ -220,7 +220,7 @@ datasets: ## Limitations - **Read-only**: Only `SELECT` scans are supported. Writes (`INSERT` / `UPDATE` / `DELETE`) are not implemented. -- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and `starts_with` on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, and an integer literal must be smaller than 2^53 in absolute value. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so results match a query that pushes nothing down. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, and an integer literal must be smaller than 2^53 in absolute value. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: New fields added in Cosmos after startup are not picked up until the runtime restarts. - **No change feed**: `acceleration.refresh_mode: changes` is not supported. - **No native temporal / decimal / binary types**: Cosmos stores everything as JSON. Round-trip these as strings and `CAST` in SQL. From ba6d0234f82ded3f3fefa763c9cfad584a9e4936 Mon Sep 17 00:00:00 2001 From: claudespice <270518434+claudespice@users.noreply.github.com> Date: Fri, 2 Oct 2026 03:42:23 -0700 Subject: [PATCH 4/5] docs(cosmosdb): a non-finite floating-point literal is evaluated in Spice Co-Authored-By: Claude --- website/docs/components/data-connectors/cosmosdb/deployment.md | 2 +- website/docs/components/data-connectors/cosmosdb/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/components/data-connectors/cosmosdb/deployment.md b/website/docs/components/data-connectors/cosmosdb/deployment.md index 308d1b5d9..574b641e2 100644 --- a/website/docs/components/data-connectors/cosmosdb/deployment.md +++ b/website/docs/components/data-connectors/cosmosdb/deployment.md @@ -127,7 +127,7 @@ Cosmos DB requests participate in [task history](../../../reference/task_history ## Known Limitations - **Read-only**: Writes (`INSERT` / `UPDATE` / `DELETE`) are not supported. -- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, and an integer literal must be smaller than 2^53 in absolute value. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, an integer literal must be smaller than 2^53 in absolute value, and a floating-point literal must be finite: a condition on `NaN`, `Infinity`, or `-Infinity` is evaluated in Spice. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: Mapping changes after startup require a runtime restart. - **No change feed**: `RefreshMode::Changes` is not wired. - **Mid-stream retries are not safe**: Retries apply to the schema-inference pass only. Errors during a streaming scan propagate immediately; rely on dataset refresh-level retry instead. diff --git a/website/docs/components/data-connectors/cosmosdb/index.md b/website/docs/components/data-connectors/cosmosdb/index.md index 297cd56aa..7c4bd4734 100644 --- a/website/docs/components/data-connectors/cosmosdb/index.md +++ b/website/docs/components/data-connectors/cosmosdb/index.md @@ -220,7 +220,7 @@ datasets: ## Limitations - **Read-only**: Only `SELECT` scans are supported. Writes (`INSERT` / `UPDATE` / `DELETE`) are not implemented. -- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, and an integer literal must be smaller than 2^53 in absolute value. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, an integer literal must be smaller than 2^53 in absolute value, and a floating-point literal must be finite: a condition on `NaN`, `Infinity`, or `-Infinity` is evaluated in Spice. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: New fields added in Cosmos after startup are not picked up until the runtime restarts. - **No change feed**: `acceleration.refresh_mode: changes` is not supported. - **No native temporal / decimal / binary types**: Cosmos stores everything as JSON. Round-trip these as strings and `CAST` in SQL. From 3cbac75ce91b302eb4d81ef57d27f71948d626e5 Mon Sep 17 00:00:00 2001 From: claudespice <270518434+claudespice@users.noreply.github.com> Date: Fri, 2 Oct 2026 04:12:51 -0700 Subject: [PATCH 5/5] docs(cosmosdb): scope the boolean restriction to comparisons, and BETWEEN to literal bounds Co-Authored-By: Claude --- website/docs/components/data-connectors/cosmosdb/deployment.md | 2 +- website/docs/components/data-connectors/cosmosdb/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/components/data-connectors/cosmosdb/deployment.md b/website/docs/components/data-connectors/cosmosdb/deployment.md index 574b641e2..0c972e937 100644 --- a/website/docs/components/data-connectors/cosmosdb/deployment.md +++ b/website/docs/components/data-connectors/cosmosdb/deployment.md @@ -127,7 +127,7 @@ Cosmos DB requests participate in [task history](../../../reference/task_history ## Known Limitations - **Read-only**: Writes (`INSERT` / `UPDATE` / `DELETE`) are not supported. -- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, an integer literal must be smaller than 2^53 in absolute value, and a floating-point literal must be finite: a condition on `NaN`, `Infinity`, or `-Infinity` is evaluated in Spice. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN` with literal bounds, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A comparison on a boolean column supports only `=` and `<>` (and `BETWEEN` does not push down on one), a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, an integer literal must be smaller than 2^53 in absolute value, and a floating-point literal must be finite: a condition on `NaN`, `Infinity`, or `-Infinity` is evaluated in Spice. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: Mapping changes after startup require a runtime restart. - **No change feed**: `RefreshMode::Changes` is not wired. - **Mid-stream retries are not safe**: Retries apply to the schema-inference pass only. Errors during a streaming scan propagate immediately; rely on dataset refresh-level retry instead. diff --git a/website/docs/components/data-connectors/cosmosdb/index.md b/website/docs/components/data-connectors/cosmosdb/index.md index 7c4bd4734..397655700 100644 --- a/website/docs/components/data-connectors/cosmosdb/index.md +++ b/website/docs/components/data-connectors/cosmosdb/index.md @@ -220,7 +220,7 @@ datasets: ## Limitations - **Read-only**: Only `SELECT` scans are supported. Writes (`INSERT` / `UPDATE` / `DELETE`) are not implemented. -- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN`, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A boolean column supports only `=` and `<>`, a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, an integer literal must be smaller than 2^53 in absolute value, and a floating-point literal must be finite: a condition on `NaN`, `Infinity`, or `-Infinity` is evaluated in Spice. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. +- **Partial filter pushdown**: When `query` is left at its default, `SELECT * FROM c`, Spice pushes the projection and supported `WHERE` conditions into the Cosmos DB query. Supported conditions are a column compared with a literal (`=`, `<>`, `<`, `<=`, `>`, `>=`), `IS NULL`, `IS NOT NULL`, `BETWEEN` with literal bounds, an `IN` list of up to 256 non-null literals, `LIKE 'prefix%'` and DataFusion's built-in `starts_with` with a non-empty literal prefix on a string column, a boolean column or its `NOT`, and `AND` and `OR` combinations of these. `NOT IN`, `NOT BETWEEN`, and other functions are evaluated in Spice. A condition pushes down only on a column of type `Utf8`, `Int64`, `Float64`, or `Boolean`. A comparison on a boolean column supports only `=` and `<>` (and `BETWEEN` does not push down on one), a string range comparison (`<`, `<=`, `>`, `>=`) needs an ASCII bound, an integer literal must be smaller than 2^53 in absolute value, and a floating-point literal must be finite: a condition on `NaN`, `Infinity`, or `-Infinity` is evaluated in Spice. Spice evaluates any other condition itself. Each pushed condition reads a superset of the matching documents, and DataFusion filters the rows again, so the returned rows match a query that pushes nothing down. A document that a pushed condition excludes is not read, so a value in it that another column cannot hold no longer fails the query. With a custom `query`, Spice pushes no conditions or projection and reads the documents that query returns. With either kind of query, a `LIMIT` stops the read once enough documents have been read. - **Schema is frozen at registration**: New fields added in Cosmos after startup are not picked up until the runtime restarts. - **No change feed**: `acceleration.refresh_mode: changes` is not supported. - **No native temporal / decimal / binary types**: Cosmos stores everything as JSON. Round-trip these as strings and `CAST` in SQL.