build: prepare for arrow 59 and DataFusion 55 - #8997
Open
slachiewicz wants to merge 2 commits into
Open
slachiewicz wants to merge 2 commits into
slachiewicz wants to merge 2 commits into
Conversation
github-actions
Bot
added
A-python
Contributor
Hi @slachiewicz, thanks for making a PR. In general, we wait for geodatafusion and datafusion-python to upgrade first be upgrading. If you want you can keep this PR up as a draft until that happens, and then we can rebase at that time. |
slachiewicz
force-pushed
the
agent/arrow-59-datafusion-55
branch
from
September 9, 2026 16:04
456d98f to
8111986
Compare
Xuanwo
added a commit
that referenced
this pull request
Sep 14, 2026
The [cargo-deny job](https://github.com/lance-format/lance/actions/runs/34867471289/job/104054814011) fails on RUSTSEC-2026-0285 because the lockfiles select rustls 0.23.40. Update rustls to the patched 0.23.45 release in the workspace, Python, and Java lockfiles, together with its required aws-lc and webpki dependencies. Validation uses cargo-deny 0.19.0, matching CI: the workspace passes all checks and Java passes the advisory check. Python no longer reports the rustls advisory, but its advisory check remains blocked by existing PyO3 0.28.3 advisories RUSTSEC-2026-0176 and RUSTSEC-2026-0177; the PyO3 upgrade is covered separately by #8997.
slachiewicz
force-pushed
the
agent/arrow-59-datafusion-55
branch
from
September 18, 2026 13:51
8111986 to
dde8cc2
Compare
DataFusion 55 declares rust-version 1.94.0, so the upcoming arrow 59 / DataFusion 55 upgrade cannot pass the MSRV job at 1.91. Rust 1.94 has been stable since 2026-03 and the pinned toolchain is already 1.97.
… keeps arrow 59 removes the infallible `From<Vec<&[u8]>>` impls. `try_from_iter` exists on both 58 and 59, and unlike `try_from(..).unwrap()` it does not trip `clippy::unnecessary_fallible_conversions` while `From` still exists. Also import `record_batch!` from arrow_array rather than the DataFusion re-export.
slachiewicz
force-pushed
the
agent/arrow-59-datafusion-55
branch
from
September 18, 2026 14:14
dde8cc2 to
afc5e2c
Compare
slachiewicz
changed the title
slachiewicz
marked this pull request as ready for review
September 18, 2026 16:41
Author
|
only minimal set of changes to be a bit better prepared for upcoming upgrade. I've expected that will be more but looks it's mainy rust upgrade |
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The reduced, current-dependency-compatible scope is sound: try_from_iter and direct arrow_array macro imports prepare the affected tests for Arrow 59 without changing runtime behavior, while the manifest and CI MSRV updates consistently match DataFusion 55’s Rust 1.94 requirement.
Please mark this PR with the breaking-change label.
lance-gatekeeper
Bot
added
the
K-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for the arrow 59 / DataFusion 55 / pyo3 0.29 upgrade, limited to what compiles against the arrow 58 / DataFusion 54 already on
main, so the upgrade itself can wait forgeodatafusionanddatafusion-pythonand shrink to version bumps plus the API migrations that only exist in the new releases.Cargo.tomlfiles and themsrvCI matrix. DataFusion 55.x declaresrust-version = 1.94.0; the pinned toolchain is already 1.97.FixedSizeBinaryArrayconstruction in tests and a doc example goes throughtry_from_iter, which both arrow 58 and 59 provide. arrow 59 removes the infallibleFrom<Vec<&[u8]>>/From<Vec<Option<&[u8]>>>impls;try_from(..).unwrap()would carry over, but on arrow 58 it tripsclippy::unnecessary_fallible_conversions.record_batch!is imported fromarrow_arrayinstead of thedatafusion::commonre-export inlance-namespace-datafusion.Not in this PR: the dependency bumps and lockfiles,
MutableArrayData::try_extend(new in arrow 59), the DataFusion 55ExecutionPlanmigration (replace_children,EnsureRequirements,ConfigNonZeroUsize),PyCapsule::new_with_value, and droppinggeofromlancedefault features. The complete upgrade is kept rebased onslachiewicz/lance@agent/arrow-59-datafusion-55-fulluntil geoarrow-rs ships an arrow-59 release (geoarrow/geoarrow-rs#1474) and datafusion-python 55 is on PyPI (apache/datafusion-python#1696).This change was created with AI assistance.