chore(deps): upgrade to DataFusion 52 by ethan-tyler · Pull Request #1997 · apache/iceberg-rust

ethan-tyler · 2026-01-06T04:18:22Z

Which issue does this PR close?

Validates and adopts DataFusion 52

What changes are included in this PR?

Upgrade DataFusion integration from 51.x to 52.x
Keep the DataFusion Python dependency dynamic within major version 52
Update the Python FFI table provider bridge for DataFusion 52 API/ABI expectations:
- session-aware __datafusion_table_provider__(session) integration
- DF52 compatible FFI table provider construction with task context and logical codec handling
Update sqllogictest physical plan expectations for DataFusion 52 planner output changes
Refresh lockfiles impacted by the upgrade

DataFusion FFI API change

DataFusion 52 expanded table provider FFI construction to include task context and optional logical codec parameters.

Updated the Rust/Python bridge accordingly allowing filter/logical expression serialization remains compatible across the FFI boundary.

Are these changes tested?

Yes

ethan-tyler · 2026-01-06T04:23:10Z

The audit failure (RUSTSEC-2026-0001 for rkyv) is unrelated to this PR - it's being addressed in #1994. Will rebase once that lands.

ethan-tyler · 2026-01-06T04:42:11Z

Fix for Python Bindings CI Failure

The initial PR failed the Bindings Python CI workflow due to a breaking API change in DataFusion 51+'s FFI module.

Root cause: FFI_TableProvider::new signature changed from 3 to 5 arguments.

Fix (commit 33d5608):

Added datafusion and datafusion-execution dependencies to bindings/python/Cargo.toml
Updated datafusion_table_provider.rs to create a TaskContextProvider from SessionContext and pass it to FFI_TableProvider::new

The core iceberg-rust crates were already compatible with DataFusion 52 - only the Python bindings needed this update.

timsaucer · 2026-01-19T21:12:38Z

Please let me know if you run into difficulties with this PR also regarding the FFI change. I think that my approach in apache/datafusion-python#1337 will help resolve the missing elements here.

Smith-Cruise · 2026-02-21T07:49:27Z

Is there any progress now?

ethan-tyler · 2026-02-22T01:59:26Z

Is there any progress now?

I rebased and got CI cleaned up. DF 52 Python wheels should be landing in the next day or so - https://lists.apache.org/thread/76v9pmqh7cflgjwx4wnqsmdzw00v62bl. To limit an additional follow up, I am waiting to include that and will open the PR.

timsaucer · 2026-02-23T18:53:24Z

Let me know if you need any help with this PR.

ethan-tyler · 2026-02-23T19:02:20Z

Let me know if you need any help with this PR.

Thanks Tim - it's been a hot minute since I worked on Iceberg and got myself into a CI pickle. I think we should be good after this most recent push. I'll let you know if I run into any trouble.

kevinjqliu · 2026-02-23T21:53:04Z

.github/workflows/bindings_python_ci.yml

      - uses: PyO3/maturin-action@v1
        with:
          working-directory: "bindings/python"
+          maturin-version: "v1.12.2"


#2166 should also resolve the CI issue. I would lean towards that solution. Otherwise, we should pin all the PyO3/maturin-action@v1 usages

Thanks! I agree and like this better!

#2166 is merged now, could you rebase and undo the current change?

Aye aye captain

kevinjqliu · 2026-02-23T21:53:35Z

Cargo.toml

-datafusion = "51.0"
-datafusion-cli = "51.0"
-datafusion-sqllogictest = "51.0"
+datafusion = "52.0"


52.1.0 is already out 😄 should we use that instead?

Yes absolutely thanks for the catch

blackmwk · 2026-02-24T01:00:52Z

bindings/python/tests/test_datafusion_table_provider.py

-assert (
-    datafusion.__version__ >= "45"
-)  # iceberg table provider only works for datafusion >= 45
+if Version(datafusion.__version__) < Version("52.0.0"):


Not related to this pr, but is it possible to extract the version from workspace Cargo.toml?

yes - I can do this on a follow up PR

blackmwk · 2026-02-25T01:59:57Z

bindings/python/DEPENDENCIES.rust.tsv

-datafusion-pruning@51.0.0		X															
-datafusion-session@51.0.0		X															
-datafusion-sql@51.0.0		X															
+datafusion@52.0.0		X															


We forgot to update this?

blackmwk · 2026-02-25T02:00:42Z

bindings/python/pyproject.toml

 name = "pyiceberg-core"
 readme = "project-description.md"
-requires-python = ">=3.10,<4"
+requires-python = ">=3.10,<3.13"


Why this change? cc @kevinjqliu Is this reasonable?

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 33d5608 to 6e77511 Compare January 6, 2026 04:45

ethan-tyler mentioned this pull request Jan 6, 2026

Release DataFusion 52.0.0 (Dec 2025 / Jan 2026) apache/datafusion#18566

Closed

41 tasks

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 6a154d0 to f4cf8da Compare January 7, 2026 16:42

ethan-tyler force-pushed the chore/datafusion-52-validation branch 2 times, most recently from ae7b70e to 723e3a6 Compare January 23, 2026 23:59

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 723e3a6 to a19062d Compare February 21, 2026 18:03

ethan-tyler changed the title ~~[WIP] chore(deps): validate DataFusion 52 compatibility~~ chore(deps): upgrade to DataFusion 52 Feb 23, 2026

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 19690bf to 3309d6c Compare February 23, 2026 19:59

ethan-tyler marked this pull request as ready for review February 23, 2026 20:15

mbutrovich self-requested a review February 23, 2026 20:20

kevinjqliu reviewed Feb 23, 2026

View reviewed changes

blackmwk reviewed Feb 24, 2026

View reviewed changes

ethan-tyler added 4 commits February 23, 2026 23:17

chore(deps): upgrade DataFusion/Arrow dependencies to 52.x

2ebf9d4

test(sqllogictest): update DataFusion 52 physical plan expectations

3d0870a

fix(python): align DataFusion 52 FFI bridge and dependency locks

19050fb

chore(deps): bump DataFusion dependencies to 52.1

196c528

ethan-tyler force-pushed the chore/datafusion-52-validation branch from 3309d6c to 196c528 Compare February 24, 2026 04:22

blackmwk reviewed Feb 25, 2026

View reviewed changes

Comments

Conversation

ethan-tyler commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

DataFusion FFI API change

Are these changes tested?

Uh oh!

ethan-tyler commented Jan 6, 2026

Uh oh!

ethan-tyler commented Jan 6, 2026

Fix for Python Bindings CI Failure

Uh oh!

timsaucer commented Jan 19, 2026

Uh oh!

Smith-Cruise commented Feb 21, 2026

Uh oh!

ethan-tyler commented Feb 22, 2026

Uh oh!

timsaucer commented Feb 23, 2026

Uh oh!

ethan-tyler commented Feb 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ethan-tyler commented Jan 6, 2026 •

edited

Loading