Repository navigation
Release DataFusion 54.0.0 (Apr 2026 / May 2026) #21080
Description
Is your feature request related to a problem or challenge?
Previous release is #19692 (Mar, 2026) so next major release would be approximately Apr / May 2026
Steps:
- Create a
branch-54branch: - Update version and changelog:
- Test with DataFusion Python
- Test with DataFusion Comet chore: Datafusion 54 upgrade datafusion-comet#4394
- Test with DataFusion Ballista
- Test with delta.rs:
- Test vortex:
- Test with iceberg-rust:
- Test with LakeSail:
- Test with parquet viewer
- Test with datafusion-materialized-views
- Voting Thread
- Publish to crates.io
- Create ticket for next release: Release DataFusion
55.0.0(Jul / Aug 2026) #22393 - Create blog post: Blog post for the DataFusion
54.0.0release #22392
Features to mention in the blog (if they make it)
- [EPIC] Simplify datetime predicates using "preimages" #19946 from @sdf-jkl (along with the related optimizations for
date_truncanddate_part) - Do not evaluate parquet predicates if they can be proven to be false #19028
- Migrate Avro reader to arrow-avro and remove internal conversion code #17861
- feat: add support for parquet content defined chunking options #21110 from @getChan
- Feature request: Support for lateral joins #10048 lateral joins from @neilconway
- feat: sort file groups by statistics during sort pushdown (Sort pushdown phase 2) #21182 from @zhuqi-lucas
- feat: add support for parquet content defined chunking options #21110 from @kszucs
- API to register behavior for Extension Types #18223 / Basic Extension Type Registry Implementation #20312 from @tobixdev
- perf: Implement physical execution of uncorrelated scalar subqueries #21240 from @neilconway
- feat: Optimize ORDER BY by Pruning Functionally Redundant Sort Keys #21362 from @xiedeyantu
- Add lambda support and array_transform udf #21679 from @gstvg and @rluvaton
- feat: globally reorder files and row groups by statistics for TopK queries #21956 from @zhuqi-lucas
- perf: coalesce batches before sending to distributor channels in RepartitionExec #22010 from @gabotechs
Bugs that should be fixed
- Regression in json performance for local files #21450
- PR #21743 disables Parquet pruning for
= ANY([literals])#22073 - Windows errors out when RANGE frame is decimal #22113
- Windows RANGE frame crashes on overflows #22137
- Windows AVG wrong result for NULLs #22138
- fix: preserve Inexact precision in Statistics #22146
- fix: simplifier on leaf nodes returns null #22368
- DataFusion drops grouped row after inner ORDER BY/LIMIT and outer ORDER BY/OFFSET #22489
- DISTINCT ORDER BY NULLS FIRST LIMIT returns the wrong top row #22554
Community Wishlist
- Fix: compact view buffers in ScalarValue::compact for all container t… #21934
- Optimize metric label cloning #22406
- fix(parquet): converting parquet schema with backward compatible repeated struct/primitive with provided arrow schema arrow-rs#8496
- NTILE returns wrong results #22049
- feat: optional timezone for coerce_int96 #22318
- perf: push_down_filter is pathologically slow for some plans #20002
- Streaming Aggregate operator not being used in deduplication of pre-sorted Parquet files #16919
- Support
ColumnExprin LAG/LEAD for default parameter #22082
Content waiting on / PRs to backport to release-54
- [branch-54] Fix: compact view buffers in ScalarValue::compact for all container types (#21934) #22446
- [branch-54] Revert "Add
ExecutionPlan::apply_expressions()(#20337)" (#22437) #22445 (backport of Revert "AddExecutionPlan::apply_expressions()(#20337)" #22437) - [branch-54]: Backport 22404. Fix Spark slice function on negative OOB #22443
- [branch-54] Support transparent ExecutionPlan downcasts #22565
- [branch-54] Cherry-pick #22493: restore SortExec elimination after stats-based file reorder #22501
- [branch-54] Support transparent ExecutionPlan downcasts #22565 (backport of Support transparent ExecutionPlan downcasts #22559)
- [branch-54] fix: clear handled OFFSET before child recursion in LimitPushdown (#22525) #22631 (backport of fix: clear handled OFFSET before child recursion in LimitPushdown #22525)
- [branch-54] Fix TopK DISTINCT aggregation preserving NULLs (#22571) #22634 (backport of Fix TopK DISTINCT aggregation preserving NULLs #22571)
- [branch-54] Gate new ScalarSubqueryExec node behind session property (#22530) #22690 (backport of Gate new ScalarSubqueryExec node behind session property #22530)
- [branch-54] refactor: wrap HigherOrderUDFImpl in a concrete HigherOrderUDF struct (#22593) #22635 (backport of refactor: wrap HigherOrderUDFImpl in a concrete HigherOrderUDF struct #22593)
Activity
[-]Release DataFusion 54.0.0 (Apr 2026 / May 2026)[/-][+]Release DataFusion `54.0.0` (Apr 2026 / May 2026)[/+]alamb commentedon Mar 24, 2026
Here is another one that would be sweet to mention:
Another would be morsel driven scans
tschwarzinger commentedon Apr 13, 2026
Could be interessting to mention #18223 in the blog.
alamb commentedon Apr 13, 2026
Absolutely -- added to the list
xiedeyantu commentedon Apr 27, 2026
I'm not sure if #21362 needs to be mentioned in the blog.
alamb commentedon Apr 28, 2026
Good call -- will add that
2010YOUY01 commentedon Apr 30, 2026
I think these are worth mentioning in the blog, and I can help write about them:
Additionally, we could maintain an in-progress blog draft for the next release, similar to what we’re doing for the upgrade guide:
That way, we can add items as they’re merged and avoid forgetting them later 🤔
alamb commentedon Apr 30, 2026
We could try for sure! I have been using the release ticket to keep track of things and then have Codex / AI make a summary draft based on the list in the ticket). Then I typically have to heavily edit what is written, but it helps as a good start.
Also I think any particular optimization that you want to explain more fully could be done as its own blog post (not the release post)
comphead commentedon May 1, 2026
@alamb any idea on the release timelines?
alamb commentedon May 3, 2026
@comphead I don't have any partcular plans for this one.
I think I need to personally focus on some arrow-rs releases next week (of 2026-05-04) but could potentially start to work on this one the week after.
57.3.1(May 2026) arrow-rs#985856.2.1(May 2026) arrow-rs#985758.3.0or58.2.1(May 2026) arrow-rs#9859Also, if people are waiting on this one it would be great from my perspective if anyone else wanted to volunteer to run it.
comphead commentedon May 4, 2026
I can drive 54 if mid may sounds reasonable
alamb commentedon May 4, 2026
Sounds great to me
mbutrovich commentedon May 6, 2026
I did a lot of the backporting for 53, so I'm also happy to start learning the process and do as much as possible as a non-PMC.
comphead commentedon May 6, 2026
Thanks @mbutrovich the release process described in refurbished
dev/release/README.mdwould be great to get your feedback. Most of steps don't require PMC status, excepting finalizing the release which includes official publishingtheirix commentedon May 6, 2026
It would be great to mention a functionality change in #21074 (already in the 54.0.0 upgrade notes, not sure if it is worth duplicating), and include #21383 in the various perf improvements section.
82 remaining items
alamb commentedon Jun 3, 2026
I made an RC and it is out for a vote: https://lists.apache.org/thread/bvv9tmw3tj1x6kwzxvgfv2xjnt0z4w8b
comphead commentedon Jun 4, 2026
Started #22765 with initial criteria proposal
mbutrovich commentedon Jun 5, 2026
Comet is seeing performance regressions for TPC-H and TPC-DS SF1000.
I will try to reproduce in DataFusion.
mbutrovich commentedon Jun 5, 2026
This is jumping out at me as alarming:
main:
df54:
The only filters pushed into this scan are three not-NULL filters.
alamb commentedon Jun 5, 2026
FWIW I think @AdamGS found a slowdown in parsing metadata due to a new cap on the amount of memory used for metadata caching. Perhaps something similar is affecting comet. If so, we should perhaps highlight this more aggressively in the upgrade guide / change the default value
54.0.0(Apr 2026 / May 2026) #21080 (comment)mbutrovich commentedon Jun 5, 2026
I'm digging into this but I don't think Comet uses the
ListingTableAPI that @AdamGS mentioned, so not sure it's the issue Comet is hitting.mbutrovich commentedon Jun 5, 2026
I can't reproduce it in DataFusion alone. I will continue to try to make sense of it, but don't think it's a blocker for release if others aren't seeing performance problems. We must have a config or something wrong to trigger a degenerate case. It seems like Comet is downloading the page indexes repeatedly, but don’t see any code changes related to that in 54 (besides refactoring). Page indexes are also probably terrible to download for IS NOT NULL filters, but that’s a separate discussion.
alamb commentedon Jun 8, 2026
The release was approved and I published it.
I think the next thing to work on is the blog:
54.0.0release #22392I'll take a crack at that later in the week if no one else beats me to it 🎣
Next patch release tracked in
54.1.0(minor/patch) Release #22547Next major release is tracked in
55.0.0(Jul / Aug 2026) #22393Upgrade DataFusion to 54 (#8044)
comphead commentedon Jun 9, 2026
I think we need to backport version bump and changelog back to main?
alamb commentedon Jun 9, 2026
Yes, I made a PR to do so
[main] Update version and changelog to 54.0.0 (apache#22855)
[main] Update version and changelog to 54.0.0 (apache#22855)
design: large-state backend (beat Flink) + DataFusion 54.0.0 upgrade …