Skip to content

fix(dynamodb, cosmosdb, mongodb): push filters down only where the source evaluates them as SQL does - #14419

Merged
lukekim merged 11 commits into
trunkfrom
fix/nosql-filter-pushdown
Oct 1, 2026
Merged

lukekim merged 11 commits into
trunkfrom
fix/nosql-filter-pushdown

Conversation

@lukekim

@lukekim lukekim commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

📝 Summary

On trunk, filter pushdown returned wrong rows for MongoDB and DynamoDB and failed some valid queries, and Cosmos DB pushed nothing at all. Each connector now pushes a filter down as Exact only where the source selects exactly the rows SQL keeps from the rows the connector converts. Otherwise it pushes an Inexact superset that DataFusion filters again, and it pushes much more than before.

MongoDB (the translator lives in the fork, spiceai/datafusion-table-providers#77, which this pins):

  • $ne/$nin/$not/$nor match null and missing fields, so age <> 30 returned the rows with no age.
  • Comparisons are bracketed by BSON type, while the conversion renders ObjectIds, documents and symbols into strings.
  • Arrays match element-wise. NaN and -0.0 compare apart from Arrow.
  • A case-insensitive collection collation made s > 'Z' return nothing.
  • Casts were looked through where they change values.
  • The scan claimed MongoDB's sort order as the SQL one.
  • With unnesting, SELECT * failed with Path collision when a field is a document in some documents and a scalar in others.

DynamoDB:

  • NOT, <> and NOT IN are two-valued in DynamoDB, so they matched missing attributes.
  • Timestamp bounds were wrong for any offset and for layouts finer than the column.
  • A map key containing a dot was missed.
  • Attribute names with a space or -, and x IN (1, NULL), failed the query outright.
  • A partition-key equality or IN list now becomes one Query per partition. Sort-key predicates become the key condition, with the rest checked item by item.
  • ORDER BY the sort key sets ScanIndexForward, and a limit stops reading.
  • Filters beyond DynamoDB's 4 KB expression limit are applied to the rows read instead of failing the request.

Cosmos DB: projection, a parameterized WHERE (comparisons, IN, BETWEEN, null tests, LIKE 'prefix%', starts_with) and limits are pushed down. Each condition is guarded by the JSON type it compares, because the emulator and the service disagree on null and undefined. It also keeps a value its column cannot hold, so the query fails as it would unfiltered. An equality on the partition key therefore still reads every partition: a partition key of another JSON type lies in a logical partition of its own.

The starts_with guards translate only DataFusion's own function, never a user function registered under that name, and no other function is pushed.

Each connector gets a differential round-trip suite. Every query runs against the federated dataset and against an Arrow acceleration of the same source, which DataFusion evaluates locally, and the suite requires the same rows, compared cell by cell, plus that each case expected to push down did. The shared harness is crates/runtime/tests/pushdown_roundtrip/mod.rs.

🔗 Related

📚 Docs

  • docs/dev/cosmosdb.md: a Push-down section, and the criteria table no longer says push-down is missing.
  • docs/dev/fork_patches.md: the new pin, and a row for the MongoDB patches with this PR's round-trip suite as their guard.

👀 Notes for Reviewers

Evidence. Every query ran against the federated dataset and against an Arrow acceleration of the same source, on mongo:8.0, amazon/dynamodb-local and the Cosmos DB Linux emulator (vnext-preview). Trunk is d62271b7a2.

Case Trunk This branch
MongoDB audit queries (25) 21 wrong 0 wrong, 20 pushed
MongoDB: symbol, collated collection and view, TRY_CAST, distant dates, dotted names with a catch-all 19 wrong 0 wrong
MongoDB: unnest_depth: 1, a field that is a document and a string Path collision at address.city rows
MongoDB: declared timestamp column, year-2262 date read as 1677-09-21 NULL
DynamoDB audit queries (12) 5 wrong 0 wrong, all pushed
DynamoDB: attribute names with a space or - ExpressionAttributeNames contains invalid key rows
DynamoDB: pk IN ('a', NULL) Invalid FilterExpression: Syntax error right, a Query
DynamoDB: -0.0 bounds, microsecond layout, +15:00 offset, dotted map key, unpadded month wrong right
Cosmos DB right rows, nothing pushed right rows; WHERE and projection in the plan

For example, on trunk, over a collection collated {locale: "en", strength: 2}:

SELECT _id, s FROM coll WHERE s > 'Z'
federated:  []
truth:      [{"_id": 2, "s": "a"}, {"_id": 3, "s": "x"}, {"_id": 4, "s": "b"}]
plan:       MongoDBExec projection=[_id, s] filters=[{"s":{"$gt":"Z"}}]

On this branch the same query returns the truth rows, with collation=[simple] in the plan.

Adversarial reviews of this branch found more cases, each reproduced against the database before it was fixed. Among them: an Int64 sort key holding 9007199254740992.1 passing an IN check through f64 rounding; Cosmos DB's x = i64::MIN overflowing a guard into c["x"] > 9223372036854775807; seven 100-value IN lists reaching DynamoDB at 4,869 bytes after a second optimizer pass; NOT ((a OR b)), which DynamoDB rejects as redundant parentheses. Copilot's reviews found four more. The fork's filter that matches nothing, {_id: {$exists: false}}, matched every document of a view that projects _id away. It is {_id: {$in: []}} now, and the MongoDB suite queries such a view. Cosmos DB's single-partition read for pk = 'a' skipped a partition key of another JSON type, so the query returned rows where the unfiltered scan fails to decode that key. It is removed, and the Cosmos DB suite checks such a container. A DynamoDB condition on a flattened column like m.x.y checked only the nested path. It therefore dropped an item holding the column at two paths (m["x.y"] and m.x.y), which unnesting refuses. The condition now keeps an item with something at any other split of the name, and the DynamoDB suite checks such an item. Its projection also named only the first segment of a flattened column's name, so a column held under a top-level attribute such as m.x read as NULL; that predates this PR, and the projection now names every prefix of the name.

Behavior changes. These are bug fixes, but they change results for malformed or out-of-range data:

  • DynamoDB: a date or timestamp string that is not zero-padded as its layout is (2024- 1-01, 2024-1-02T…) now reads as NULL, as any other unparsable string does.
  • MongoDB: a date too distant for a declared microsecond or nanosecond column reads as NULL instead of wrapping.
  • MongoDB no longer claims a sort order, so ORDER BY … LIMIT sorts in DataFusion. Exact sort pushdown, through per-type queries merged in order, can follow.

Tests.

  • Unit tests: connector-dynamodb 220, connector-cosmosdb 45, and the fork's MongoDB module 164.
  • Integration: cargo test -p runtime --test integration --features mongodb,dynamodb,cosmosdb,duckdb -- --include-ignored pushdown_round_trips passes. That is 165 MongoDB cases (137 pushed), 117 DynamoDB (105 pushed) and 56 Cosmos DB (46 pushed). Over a container whose partition keys are of two JSON types, the Cosmos DB suite also requires a pushed-down equality on the partition key to fail on decoding, as the unfiltered scan does. The DynamoDB suite likewise requires a pushed-down condition to fail on an item that unnesting reads m.x.y from twice, and every item to read m.x.y from whichever attribute holds it.
  • The MongoDB and DynamoDB suites run in integration.yml. The Cosmos DB suite is #[ignore]d because the emulator is a large image, and CI doesn't enable cosmosdb anyway.
  • The DynamoDB integration module's EXPLAIN snapshot of a timestamp filter, dynamodb_timestamp_filter_pushdown, now shows the widened bounds: #n0 >= :v0 AND #n1 <= :v1 over 2024-11-30T12:34:56.123 and 2024-12-02T12:34:56.123~, where trunk compared 2024-12-01T17:34:56.123+05:00 as a string. The merge queue runs that module against a table in AWS and failed on the old snapshot. The new one is the plan the test prints against amazon/dynamodb-local holding the same items, which matched the queue's output byte for byte.
  • Clippy, with the lint gate's flags over these crates and the integration tests, is clean. The remote sign-off of 2a49ffa passed: Signed off by lukekim via remote — lint+test passed in 6922s. The remote sign-off of 9403797, that snapshot and a merge of trunk, is running.

Found along the way, not fixed here: SELECT truncate(1.5, 0) fails on trunk with Truncation width must be a positive Int64, got Float64. The round-trip suites use bucket to check that a Spice function stays out of the pushed filter.

Copilot AI balanced review requested due to automatic review settings September 26, 2026 16:26
@lukekim
lukekim requested a review from a team as a code owner September 26, 2026 16:26
@github-actions github-actions Bot added area/config area/docs area/tests kind/bug Something isn't working kind/dependencies Pull requests that update a dependency file labels Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

✅ Pull with Spice Passed

Passing checks:

  • ✅ Title meets minimum length requirement (10 characters)
  • ✅ No banned labels detected
  • ✅ Has a label from required category kind/
  • ✅ Has a label from required category area/
  • ✅ Has at least one assignee: lukekim

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

DynamoDB and Cosmos DB can apply limits before retained inexact filters, causing valid matching rows to be omitted.

Review effort: Balanced
Findings: 2 High severity

Open (2)
What changed in this PR

Improves SQL-correct filter pushdown for DynamoDB, Cosmos DB, and MongoDB.

Changes:

  • Adds semantics-aware DynamoDB and Cosmos DB filter/request planning.
  • Pins MongoDB pushdown fixes from the provider fork.
  • Adds differential integration tests and documentation.
File Description
Cargo.lock Updates locked dependencies.
Cargo.toml Updates the table-providers fork pin.
connector-cosmosdb/​.../​filter.rs Implements Cosmos DB filter translation.
connector-cosmosdb/​.../​filter/​tests.rs Tests Cosmos filter semantics.
connector-cosmosdb/​.../​mod.rs Registers filter support.
connector-cosmosdb/​.../​provider.rs Adds query, projection, partition, and limit pushdown.
connector-dynamodb/​src/​arrow.rs Tightens date and timestamp parsing.
connector-dynamodb/​src/​filter.rs Implements DynamoDB filter translation.
connector-dynamodb/​src/​filter/​tests.rs Tests translation edge cases.
connector-dynamodb/​src/​lib.rs Registers the filter module.
connector-dynamodb/​src/​provider.rs Integrates request, sort, and limit pushdown.
connector-dynamodb/​src/​request_builder.rs Builds bounded Query and Scan requests.
connector-dynamodb/​src/​request_builder/​tests.rs Tests request planning.
connector-dynamodb/​src/​request_plan.rs Expands request-plan metadata and variants.
connector-dynamodb/​src/​table_schema.rs Adds key types and translation metadata.
connector-dynamodb/​src/​utils.rs Removes the superseded filter visitor.
crates/​runtime/​Cargo.toml Adds Cosmos test dependencies.
runtime/​tests/​cosmosdb/​mod.rs Registers the Cosmos round-trip suite.
runtime/​tests/​cosmosdb/​pushdown_roundtrip.rs Adds Cosmos differential tests.
runtime/​tests/​dynamodb/​mod.rs Registers the DynamoDB suite.
runtime/​tests/​dynamodb/​pushdown_roundtrip.rs Adds DynamoDB differential tests.
runtime/​tests/​integration.rs Registers the shared harness.
runtime/​tests/​mongo/​mod.rs Registers the MongoDB suite.
runtime/​tests/​mongo/​pushdown_roundtrip.rs Adds MongoDB differential tests.
runtime/​tests/​pushdown_roundtrip.rs Provides shared comparison utilities.
docs/​dev/​cosmosdb.md Documents supported pushdown behavior.
docs/​dev/​fork_patches.md Records the MongoDB fork patch.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread crates/data-connectors/connector-cosmosdb/src/cosmosdb/provider.rs
Comment thread crates/data-connectors/connector-dynamodb/src/provider.rs
Copilot AI review requested due to automatic review settings September 26, 2026 20:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Inexact filters can interact incorrectly with pushed limits, and the differential harness can treat distinct results as equal.

Review effort: Balanced
Findings: 2 High severity · 1 Medium severity

Open (3)

Comment thread crates/runtime/tests/pushdown_roundtrip.rs Outdated
Copilot AI review requested due to automatic review settings September 26, 2026 22:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Cosmos single-partition routing can omit wrong-typed partition-key documents that its inexact filter intentionally retains.

Review effort: Balanced
Findings: 1 High severity

Open (1)
Resolved since last review (3)

Comment thread crates/data-connectors/connector-cosmosdb/src/cosmosdb/provider.rs Outdated
Copilot AI review requested due to automatic review settings September 26, 2026 23:13
@cursor

cursor Bot commented Sep 26, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

DynamoDB pushdown can suppress a dotted-key collision error by filtering the ambiguous item before conversion.

Review effort: Balanced
Findings: 1 High severity

Open (1)
Resolved since last review (1)

Comment thread crates/data-connectors/connector-dynamodb/src/filter.rs
Copilot AI review requested due to automatic review settings September 27, 2026 00:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Core query-correctness logic spans a large connector change and an external temporary fork pin, preventing confident automated approval.

Review effort: Balanced
Findings: None

Resolved since last review (1)

@lukekim

lukekim commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

Addressed #14419 (review). That review had no line-comment thread. Its headline was that the query-correctness logic spans a large connector change and a temporary fork pin, which it leaves to a human reviewer, so I'm not requesting another Copilot pass.

The fork pin. spiceai/datafusion-table-providers#77 has to merge first. This PR then repoints Cargo.toml, Cargo.lock and docs/dev/fork_patches.md at the merge commit, drops the TEMPORARY marker, and gets signed off. Until then the required Attestation check cannot pass. make lint-rust runs scripts/check_fork_patches.py (Makefile:361), which refuses the marker:

$ python3 scripts/check_fork_patches.py; echo "exit=$?"
1 pin(s) not ready to land:
  - datafusion-table-providers is pinned to a branch rather than a landed revision: spiceai/datafusion-table-providers#77 has to merge first, then repoint the pin at the merge commit and drop the TEMPORARY marker
exit=1

Correctness. At this head, pushdown_round_trips --include-ignored --test-threads 1 runs 165 MongoDB, 117 DynamoDB and 56 Cosmos DB differential cases, with the two new failure-mode checks: test result: ok. 3 passed; 0 failed. Each finding in the Copilot rounds was reproduced against the database before it was fixed. For each fix, its regression check was then shown to fail with the fix reverted:

#14419 (comment) is Cursor Bugbot's notice that its spend limit was reached. It asks nothing of this PR.

Copilot AI review requested due to automatic review settings September 27, 2026 00:58
@cursor

cursor Bot commented Sep 27, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

… from

A flattened column such as m.x.y is read from the map m, a map attribute
m.x, or an attribute named m.x.y, but the projection named only the first
segment of the column's name. An item whose value sat under a top-level
attribute with a dot in its name came back without it, so m.x.y read as
NULL and a filter on it dropped the row. The projection now names every
prefix of the name that ends before a dot, and the whole name.

The DynamoDB round-trip suite writes an item of each kind and requires all
of them to read m.x.y. The local copy shares the projection, so a round
trip cannot tell.
…lan snapshot

The EXPLAIN snapshot of dynamodb_timestamp_filter_pushdown still held
trunk's request, which compared each timestamp bound as a string in a
single offset. This branch compares the stored local date and time within
a day of the bound instead, and leaves the exact comparison to the
FilterExec above the scan, so the merge queue failed the test on every try.

The new snapshot is the plan the test prints against amazon/dynamodb-local
holding the items init_test_table writes, which is byte for byte the plan
the merge queue printed against the real table.
Copilot AI balanced review requested due to automatic review settings September 30, 2026 06:00
@lukekim
lukekim force-pushed the fix/nosql-filter-pushdown branch from 9403797 to 7c3ef2d Compare September 30, 2026 06:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings September 30, 2026 17:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@lukekim

lukekim commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Addressed #14419 (review) as Copilot's quota-limit error on 207b2f6, which reviews nothing: "Copilot was unable to review this pull request because the user who requested the review has reached their quota limit." All four requests since 2026-09-29 20:12Z got this reply (on 1f26067, 9403797, 7c3ef2d and 207b2f6). Another request would come from the same account, so I haven't sent one. Copilot's last review with content was of 2a49ffa, with no findings: #14419 (review).

The diff hasn't changed since the approvals of 9403797. They were dismissed at 06:00Z by a rewrite of the branch history. 207b2f6 then merged trunk at 197e35d with no conflict. Against its base, the PR adds and removes the same 8803 lines at both heads. The only two lines that differ are unchanged context, next to the imports that trunk's #14412 edited:

$ diff <(git diff 940379797e^2 940379797e | grep -vE '^(index |@@)') \
       <(git diff 207b2f604a^2 207b2f604a | grep -vE '^(index |@@)')
3367d3366
<  use datafusion::prelude::SessionContext;
3369a3369
>      common::project_schema,

The merge-queue test this PR failed on should still pass after the DataFusion bump (#14430) that this merge brings in. dynamodb::dynamodb_timestamp_filter_pushdown runs only in the queue. Its snapshot has seven plan lines, and DataFusion renders six of them. Those six are byte-identical to trunk's copy of the snapshot (diff exit 0). Trunk's copy hasn't changed since #11891. It passed in merge-queue run 36736342521 on bca6fc3, which contains the bump:

PASS [   1.004s] runtime::integration dynamodb::dynamodb_timestamp_filter_pushdown

The seventh line, DynamoDBTableProviderExec { … }, is the request this branch builds from the same partial_filters. Since then, the only trunk change to the connector is #14412. It swaps SessionContext::new() for util::session_state::session_context() in the table bootstrap, and it doesn't touch that request.

Not run: the test itself on 207b2f6. It needs the AWS table whose key only CI holds, so the next queue run is that check.

Attestation stays red until remote sign-off run 36743249414 posts on 207b2f6. The run is queued, and it checks out the branch tip when it starts.

@lukekim
lukekim added this pull request to the merge queue Sep 30, 2026
Merged via the queue into trunk with commit 4a4ecc9 Oct 1, 2026
56 of 58 checks passed
@lukekim
lukekim deleted the fix/nosql-filter-pushdown branch October 1, 2026 09:03
claudespice pushed a commit that referenced this pull request Oct 1, 2026
…14429

This reverts commit ce87784. The
regeneration ran a binary that includes #14419, which moves several
MongoDB filters out of the pushdown, so 19 of the 20 changed snapshots
reflect that change rather than the null-aware NOT IN plan. They are
left for their own review.
spiceemma added a commit that referenced this pull request Oct 3, 2026
Copilot pointed out that s3[parquet]-federated and
abfs_standard_versioned[parquet]-federated still carried the swapped
RightAnti plan. A scan of the trunk b97f85e TPC-H sweep found nine
more arms failing tpch_q16 on the same RightAnti -> null-aware LeftAnti
change. Each snapshot is the new result that arm's trunk run printed.

Left out: mongodb-federated (#14419 scope, as before), oracle and
oracle[catalog] (older drift beyond the anti join), and mysql-arrow /
postgres-arrow (their q16 plan keeps RightAnti).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config area/data-connectors area/docs area/tests kind/bug Something isn't working kind/dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants