fix(mongodb): push down only filters MongoDB evaluates as SQL does, and stop claiming a sort order by lukekim · Pull Request #77 · spiceai/datafusion-table-providers

lukekim · 2026-09-26T16:05:41Z

Summary

MongoDB filter pushdown selected different rows from the ones SQL keeps, because it translated SQL operators into MongoDB operators one for one. MongoDB and SQL disagree on how those operators treat the documents the connector converts into rows:

$ne, $nin, $not and $nor match a document whose field is null or missing, where SQL's three-valued logic makes the predicate NULL, so age <> 30 returned the rows with no age.
Comparisons are bracketed by BSON type: {f: {$gt: "6"}} never matches the number 7. The conversion, though, renders a non-string value into a Utf8 column: an ObjectId as its hex string, a document as JSON, a symbol as Symbol("x"), anything else as its display string.
A comparison matches an array when any element does, while the conversion turns an array in a scalar column into NULL. A dotted path also traverses arrays, which unnesting does not.
Numeric comparisons cross int, long and double, while the conversion keeps only the BSON types the column's Arrow type accepts, and rounds a long beyond 2^53 into an f64 column.
NaN compares false against every number, while Arrow orders it above them (or below, with the sign bit set). MongoDB also holds -0.0 equal to 0.0, which Arrow orders apart.
A collection's default collation can hold unequal strings equal and order them differently from SQL, which narrows $nin and the range operators.
A cast of a timestamp column to a finer unit overflows for a distant date (TRY_CAST then yields NULL), and a cast of a timestamp without a time zone into another zone moves the instant. The translator looked through both.
A LIKE pattern became a regex that did not anchor at newlines or treat \ as the escape, the way DataFusion evaluates LIKE.
The scan claimed MongoDB's sort order (nulls first, types before values, NaN lowest, arrays by their smallest element) as the SQL one, so DataFusion removed its own sort.

Each comparison is now guarded by the BSON types its column accepts. Arrays are excluded wherever the conversion nulls them. A negation is pushed down to the comparisons (De Morgan, with every operator negated as three-valued logic requires) instead of being wrapped in $nor. Each filter reports Exact only where its document selects exactly the rows the predicate keeps; otherwise it reports Inexact, selects a superset, and DataFusion filters again.

The table reads its collection's collation once. A query that orders or excludes strings names the simple collation on a collated collection. On a view, whose collation cannot be replaced, or where the collation cannot be read, only equality, IN, LIKE and null tests are pushed down, since a collation can only widen $in and does not apply to $regex. A query that needs no ordering keeps the collection's collation, so its indexes stay usable.

The sort-pushdown override is removed. Exact sort pushdown through per-type segmented queries can follow separately.

Two conversion fixes come with it, because the translator has to agree with them:

A date whose milliseconds overflow a microsecond or nanosecond column is now NULL, as Arrow's safe cast makes it. Before, it wrapped to an instant centuries away: a year-2262 date read as 1677-09-21. A column declared timestamp in Spice is nanoseconds, so this was reachable.
With a JSON catch-all, whole documents are read, and unnesting a whole document puts a field whose name contains a dot into the dotted column of the same name. No query path addresses such a field, so dotted columns are not pushed down in that configuration. With a projection, MongoDB returns only the embedded field, and dotted columns are still pushed down.

A NULL predicate, which is what DataFusion leaves of x IN (1, NULL), now translates to a filter that matches nothing under either polarity, so such filters are pushed down. That filter is {_id: {$in: []}}, which matches no document, including on a view that projects _id away.

With unnesting, a field that holds a document in some documents and a scalar in others is both a column and the prefix of others (address, address.city). The projection named both, which MongoDB 4.4 and later refuse (Path collision at address.city remaining portion city), so SELECT * over such a collection failed. It now names only the prefix, whose embedded document unnesting flattens.

Breaking changes

Four public items of the mongodb module change:

utils::expression::expr_to_mongo_filter(expr) is removed. It translated a predicate without the column types or the collection's collation. The query that selects the rows SQL keeps depends on both, so no translation under that signature can be correct. Use translate_filter(expr, schema, scope), which also reports whether the document is exact.
utils::expression::combine_exprs_with_and(exprs) is removed. It is datafusion::logical_expr::utils::conjunction(exprs.iter().cloned()).
connection::MongoDBConnection::query_arrow no longer takes sort_doc, and takes collation: Option<&Collation> after filters_doc. MongoDB's sort order is not SQL's, so no sort is sent. Pass connection.string_comparison(collection).await.collation_for(filters_doc), which is the simple collation on a collated collection when the filter orders or excludes strings, and None otherwise.
table::MongoDBExec::new takes filter_scope: &FilterScope and strings: StringComparison after filters, which the translation needs.

spiceai/spiceai calls none of them. It imports MongoDBTableFactory, connection_pool::MongoDBConnectionPool, Error, projection::project_bson_document, utils::arrow::mongo_docs_to_arrow and utils::unnest, whose signatures do not change.

Evidence

The full differential runs are in spiceai/spiceai#14419, which pins this branch. Each query runs against the federated dataset and against an Arrow acceleration of the same collection, which DataFusion evaluates locally, on mongo:8.0:

Suite	Trunk (`d62271b7a2`)	This branch
Original audit, 25 queries	21 wrong	0 wrong, 20 with a pushed filter
Symbols (`s <> 'x'`, `NOT IN`, `=`)	3 wrong	0 wrong, all pushed
Case-insensitive collation, on a collection	6 of 7 wrong	0 wrong, all pushed; the order-sensitive ones with `collation=[simple]`
The same, on a view	6 of 7 wrong	0 wrong; the order-sensitive ones not pushed
`TRY_CAST(t AS TIMESTAMP) IS NULL` with a year-2262 date	wrong (`[]`, not `[1, 3]`)	right, not pushed
Declared nanosecond column	year 2262 read as 1677	NULL
Dotted field name with a catch-all	2 wrong	0 wrong, not pushed
`SELECT *` with `unnest_depth: 1` where `address` is a document and a string	`Path collision at address.city`	rows

For example, on trunk:

SELECT _id, s FROM coll WHERE s > 'Z'       -- collection collation {locale: en, strength: 2}
federated:  []
truth:      [{"_id": 2, "s": "a"}, {"_id": 3, "s": "x"}, {"_id": 4, "s": "b"}]
plan:       MongoDBExec projection=[_id, s] filters=[{"s":{"$gt":"Z"}}]

On this branch the same query returns the truth rows, with filters=[{"$or":[{"s":{"$gt":"Z"}},…]}] collation=[simple].

A cast that adds a time zone to a naive timestamp column is no longer looked through. Compared with a literal, such a cast never reaches the provider, because DataFusion's unwrap-cast rewrite moves it to the literal. Tested as IS NULL, though, arrow_cast(t, 'Timestamp(Millisecond, Some("UTC"))') IS NULL over a year-275760 date returned [] federated. Local evaluation of the same rows fails with Cast error: Cannot cast timezone to different timezone, because Arrow converts through chrono, whose range is narrower than a BSON date's.

Tests

cargo test -p datafusion-table-providers --lib --no-default-features --features mongodb mongodb::: 164 passed. cargo clippy -p datafusion-table-providers --no-default-features --features mongodb -- -D warnings reports nothing in the MongoDB module. Codex reviewed the translator adversarially twice. Each finding was reproduced against MongoDB before it was fixed, and one it predicted (a zone-adding cast compared with a literal) did not reproduce, as described above. spiceai/spiceai#14419 adds crates/runtime/tests/mongo/pushdown_roundtrip.rs, which runs every case above as a differential test against a MongoDB container, along with filters that match nothing over a view that projects _id away.

…nd stop claiming a sort order The MongoDB table provider translated SQL predicates operator-for-operator and reported every translation as exact, so DataFusion dropped its own filter and returned whatever MongoDB matched. MongoDB's matching differs from SQL's evaluation of the converted rows: - `$ne`, `$nin`, `$not` and `$nor` match a null or missing field, where SQL's three-valued logic evaluates the predicate to NULL. - An `ObjectId` converts to its hex string, so `_id = '<hex>'` compared a string against an `ObjectId` and matched nothing. - Comparisons are bracketed by BSON type, while a `Utf8` column renders numbers, documents and other types as strings. - Arrays match when any element does; the conversion nulls them in a scalar column. - A cast was unwrapped from the column side, so `CAST(ts AS DATE) = d` compared the raw timestamp with midnight. - LIKE was translated without dotall, with `$` as the end anchor, and without DataFusion's default backslash escape. - NaN compares false against every number, which Arrow orders above them. Each comparison is now guarded by the BSON types its column's Arrow type accepts, excludes arrays where the conversion nulls them, and pushes negation down to the comparisons instead of wrapping `$nor`/`$not`. A translation that can only select a superset (a `Utf8` column, a float range, LIKE) is reported `Inexact`, so DataFusion filters it again. The sort pushdown is removed: MongoDB sorts null and missing fields first, orders by BSON type before value, puts NaN below every number and orders an array by its smallest element, so `find().sort()` cannot reproduce the order DataFusion requires, and returning `Exact` removed the `SortExec` that would.

…tant dates consistent with the converted rows A `$nin` over strings also excludes a symbol spelled like the excluded string, which the conversion renders as `Symbol("...")`; the exclusion now keeps every rendered type. A collection's default collation can hold unequal strings equal and order them apart from SQL, which narrows `$nin` and the range operators. A table now reads its collection's collation: a query that orders strings names the simple collation on a collated collection, and on a view (whose collation cannot be replaced) or where the collation cannot be read, only equality, IN, LIKE and null tests on strings are pushed down. A cast of a timestamp column to another unit, or of a timestamp without a time zone into a zone other than UTC, changes values (overflowing, moving or truncating the instant), so such a cast is no longer looked through. A date whose milliseconds overflow a microsecond or nanosecond column is now NULL rather than a wrapped-around instant, and filters on such a column account for it. Unnesting whole documents, as a JSON catch-all requires, reads a field whose name has a dot into the dotted column of that name, which no query path addresses, so dotted columns are not pushed down there.

DataFusion simplifies a short `x IN (1, NULL)` into `x = 1 OR NULL`, whose NULL is neither true nor false, so the filter was not pushed down at all. A NULL predicate now translates to a filter that matches nothing under either polarity, and a disjunction or conjunction with such a side simplifies to what the other side keeps.

…aive timestamp Arrow reads a timestamp without a time zone as wall-clock time in the zone it gains, through chrono, whose range is narrower than a BSON date's; even UTC fails for a distant date, which CAST reports as an error and TRY_CAST as NULL. Such a cast is no longer looked through.

With unnesting, a field that holds a document in some documents and a scalar in others is both a column and the prefix of others (address, address.city). Projecting both is a path collision, which MongoDB 4.4 and later refuses, so any query reading both columns failed. The prefix alone returns the whole embedded document, which unnesting flattens.

Copilot

Copilot review overview

🟡 Changes recommended

The always-false filter can match MongoDB view rows that omit _id, causing exact predicates to return incorrect results.

Review effort: Balanced
Findings: 1

Open (1)

Use field-independent contradiction for empty MongoDB predicates · New

What changed in this PR

Improves MongoDB correctness by aligning pushed filters and converted values with DataFusion semantics.

Changes:

Reworks filter translation with type, null, collation, and exactness safeguards.
Removes unsafe sort pushdown and fixes timestamp overflow conversion.
Avoids MongoDB projection path collisions and adds extensive tests.

File	Description
`core/src/mongodb/utils/expression/tests.rs`	Adds comprehensive filter translation tests.
`core/src/mongodb/utils/expression.rs`	Implements semantics-aware MongoDB filter translation.
`core/src/mongodb/utils/arrow.rs`	Prevents timestamp scaling overflow.
`core/src/mongodb/table.rs`	Integrates exact/inexact filters and removes sort pushdown.
`core/src/mongodb/connection.rs`	Adds collation handling and collision-safe projections.
`core/src/mongodb/connection_pool.rs`	Exposes configured unnest depth.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

A predicate that is never true translated to `{_id: {$exists: false}}`, which assumes every document has an `_id`. A view can project `_id` away, and there it matched every document, so an exact filter such as `i32 = 2.5` on an integer column returned every row. `{_id: {$in: []}}` matches no document, a missing `_id` included.

Copilot

Copilot review overview

🟡 Changes recommended

The replacement removes exported filter helpers without a compatibility or deprecation path.

Review effort: Balanced
Findings: 1

Open (1)

Preserve public expression helpers or stage API removal as breaking · New

Resolved since last review (1)

Use field-independent contradiction for empty MongoDB predicates

Copilot

Copilot review overview

🟡 Changes recommended

A third public API breaking change remains undocumented or requires a compatibility path.

Review effort: Balanced
Findings: 1

Open (1)

Undocumented breaking change to MongoDBConnection::query_arrow API · New

Resolved since last review (1)

Preserve public expression helpers or stage API removal as breaking

Copilot

Copilot review overview

🔵 Needs a closer look

The extensive cross-engine semantic changes and database-specific edge cases warrant final human validation despite strong test coverage.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Undocumented breaking change to MongoDBConnection::query_arrow API

lukekim · 2026-09-26T22:42:24Z

Addressed #77 (review). That review had no line-comment thread. Its headline was that the semantic changes across engines warrant a human's validation. That is a call for a human reviewer, so I'm not requesting another Copilot pass.

What a reviewer can check against MongoDB rather than against the code:

The Evidence table, run on mongo:8.0 against the federated table and an Arrow copy that DataFusion evaluates: 21 of 25 audited queries wrong on the base, 0 on this branch.
fix(dynamodb, cosmosdb, mongodb): push filters down only where the source evaluates them as SQL does spiceai#14419's crates/runtime/tests/mongo/pushdown_roundtrip.rs pins this head and runs 165 such differential queries over five collections and two views, 137 of them pushed down. It passes at c6cbc3a: test result: ok. 3 passed; 0 failed, alongside the DynamoDB and Cosmos DB suites.
The places where judgment about MongoDB matters most: the $type brackets each comparison is guarded by, arrays excluded where the conversion nulls them, De Morgan negation instead of $nor, NaN and -0.0, the simple collation named only when a filter orders or excludes strings (and never on a view), and the removal of sort pushdown.
Breaking changes lists the four public items that change, from a diff of every pub signature under core/src/mongodb/.

The two failing checks on c6cbc3a come from code this PR doesn't touch, and neither is a required check on spiceai-54. The diff is confined to core/src/mongodb/.

DuckDB Tests (release) failed on both attempts, in duckdb::upsert_retention_memory::upsert_with_retention_under_query_load_does_not_grow_memory::case_1_single_column:

terminate called after throwing an instance of 'duckdb::FatalException'
  what():  {"exception_type":"FATAL","exception_message":"IO Error: Corrupt database file: computed checksum 5381 does not match stored checksum 0 in block at location 57683968", …}
process didn't exit successfully: `…/integration-7178f4cf3b4f98c3 --nocapture --test-threads 1` (signal: 6, SIGABRT: process abort signal)

make test-integration-duckdb builds with --no-default-features --features duckdb,duckdb-federation (Makefile:25), and the MongoDB module is behind #[cfg(feature = "mongodb")] (core/src/lib.rs:22). None of this PR's code is in that binary.
The same test fails with other DuckDB engine errors on branches without this change: run 35479343973 (job 105994182197) with Failed to commit: empty child i = 1 for byte 1 in BaseNode::GetChild, and run 35472521238 (job 105975968983) with Out of Memory Error: could not allocate block of size 256.0 KiB.

Python Tests failed on both attempts (jobs 108487428128 and 108496636772) in the ClickHouse setup: test_clickhouse.py::TestClickHouseParameterized::{test_get_tables,test_parameterized_view} raise Exception: ClickHouse connection failed. decompression error: incorrect magic number (17 passed, 2 errors). The test pulls clickhouse/clickhouse-server:latest, which is now 26.9.3.38 (sha256:12b16e744f91d8228705e136bd2b0ab14c939c87ff82443090279b65255c50c1), and the same job passed 19 of 19 on 2026-09-19/20. The failure reproduces on spiceai-54 itself, without this PR. The fork's ClickHouse example (cargo run -p datafusion-table-providers --example clickhouse --no-default-features --features clickhouse), run on origin/spiceai-54:

# clickhouse/clickhouse-server@sha256:12b16e74…, version 26.9.3.38
thread 'main' panicked at core/examples/clickhouse.rs:21:59:
called `Result::unwrap()` on an `Err` value: ConnectionError { source: Decompression("incorrect magic number") }

# the same binary against clickhouse/clickhouse-server:25.8, version 25.8.33.6
thread 'main' panicked at core/examples/clickhouse.rs:33:10:
called `Result::unwrap()` on an `Err` value: UnableToGetSchema { source: RowNotFound }

Against 25.8 the pool connects, and the example then stops at its own missing Reports table. The clickhouse 0.13.3 client can't read ClickHouse 26.9's responses, which is separate from this PR.

…datafusion-table-providers#77 10b7fe71d7 on spiceai-54 squash-merges spiceai/datafusion-table-providers#77 onto 14fdd18233, trunk's previous pin. Its tree is that of c6cbc3a3ca, the branch head this pull request was tested at, so the code is unchanged. The ledger row names spiceai-54 instead of the pull request's branch and drops the TEMPORARY marker, and the MongoDB patch row records the view without `_id` that its guard now queries.

… and guard it with a round-trip suite Pins spiceai/datafusion-table-providers#77, which pushes a MongoDB filter down only where MongoDB selects the rows SQL keeps from the converted documents, stops reporting MongoDB's sort order as the SQL one, projects no path beside one of its prefixes, and reads a date too distant for a microsecond or nanosecond column as NULL. The pin is marked TEMPORARY in docs/dev/fork_patches.md until that pull request merges. Adds a differential round-trip suite over a collection of null, missing, mistyped, array, ObjectId, symbol, NaN, signed-zero and distant values, a case-insensitive collated collection and view, unnested and dotted fields, and a JSON catch-all, each compared with an Arrow acceleration of the same collection.

…datafusion-table-providers#77 10b7fe71d7 on spiceai-54 squash-merges spiceai/datafusion-table-providers#77 onto 14fdd18233, trunk's previous pin. Its tree is that of c6cbc3a3ca, the branch head this pull request was tested at, so the code is unchanged. The ledger row names spiceai-54 instead of the pull request's branch and drops the TEMPORARY marker, and the MongoDB patch row records the view without `_id` that its guard now queries.

lukekim added 5 commits September 25, 2026 23:20

Copilot AI balanced review requested due to automatic review settings September 26, 2026 16:05

Copilot started reviewing on behalf of lukekim September 26, 2026 16:06

View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

Comment thread core/src/mongodb/utils/expression.rs Outdated

lukekim mentioned this pull request Sep 26, 2026

fix(dynamodb, cosmosdb, mongodb): push filters down only where the source evaluates them as SQL does spiceai/spiceai#14419

Merged

lukekim self-assigned this Sep 26, 2026

Copilot AI review requested due to automatic review settings September 26, 2026 21:09

Copilot started reviewing on behalf of lukekim September 26, 2026 21:09

View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

Comment thread core/src/mongodb/utils/expression.rs

lukekim requested a balanced review from Copilot September 26, 2026 22:08

Copilot started reviewing on behalf of lukekim September 26, 2026 22:08

View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

Comment thread core/src/mongodb/connection.rs

lukekim requested a balanced review from Copilot September 26, 2026 22:24

Copilot started reviewing on behalf of lukekim September 26, 2026 22:24

View session

Copilot AI reviewed Sep 26, 2026

View reviewed changes

lukekim added the bug Something isn't working label Sep 26, 2026

krinart approved these changes Sep 27, 2026

View reviewed changes

lukekim merged commit 10b7fe7 into spiceai-54 Sep 27, 2026
26 of 30 checks passed

lukekim deleted the fix/mongodb-filter-sort-pushdown-sql-semantics branch September 27, 2026 00:55

This was referenced Sep 29, 2026

DataFusion 55 / arrow 59.3 in spiceai-55 #78

Closed

Fix CI on the DataFusion 55 upgrade #79

Closed

This was referenced Oct 1, 2026

fix(python): align the datafusion dependency with the DataFusion 55 upgrade #81

Merged

Move to DataFusion 55 and Arrow 59 #80

Merged

Navigation Menu

Repository navigation

fix(mongodb): push down only filters MongoDB evaluates as SQL does, and stop claiming a sort order - #77

fix(mongodb): push down only filters MongoDB evaluates as SQL does, and stop claiming a sort order#77
lukekim merged 6 commits into
spiceai-54from
fix/mongodb-filter-sort-pushdown-sql-semantics

lukekim commented Sep 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

lukekim commented Sep 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lukekim commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking changes

Evidence

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

lukekim commented Sep 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lukekim commented Sep 26, 2026 •

edited

Loading