Skip to content

dev: Allow new release version to be automatically updated #26007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nuno-faria wants to merge 2 commits into
base: main
Choose a base branch
from
Open

dev: Allow new release version to be automatically updated #26007

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5ff688b
dev: Allow new release version to be automatically updated
nuno-faria Oct 3, 2026
1204ab1
Fix fmt
nuno-faria Oct 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 7 additions & 11 deletions dev/release/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,22 +90,18 @@ git push -u apache branch-50 # push branch to apache remote

### 2. Prepare PR to Update the Release Version

Manually update the DataFusion version in the root `Cargo.toml` to
reflect the new release version. Ensure `Cargo.lock` is updated accordingly by
running:
Update the DataFusion version using the following Python script:

```shell
cargo check -p datafusion
# replace NEW_VERSION before executing (e.g., 50.0.0)
python3 dev/update_datafusion_versions.py NEW_VERSION
```

Within the user documentation there are references to the current version number.
Update these to the current version. At the time of this writing we need to manually
update the following files
This updates the DataFusion version across all files, including documentation. The only extra step required is to update the `Cargo.lock` file, using the following command:

@VVKot VVKot Oct 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does sound like the manual cargo check is actually needed? For me, the script is updating Cargo.lock automatically, and cargo check -p datafusion produces no new diff

@nuno-faria nuno-faria Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Cargo.lock is not updated by the script. Are you maybe running rust-analyzer in the background (e.g., in vscode)? I tested again and these are the files that changed for me:

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   Cargo.toml
        modified:   ci/scripts/release_version_labeler.js
        modified:   docs/source/download.md
        modified:   docs/source/user-guide/configs.md
        modified:   docs/source/user-guide/crate-configuration.md
        modified:   docs/source/user-guide/example-usage.md

@VVKot VVKot Oct 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that's exactly what it was -- background rust-analyzer process updating the Cargo.lock!


- `docs/source/download.md`
- `docs/source/user-guide/configs.md`
- `docs/source/user-guide/crate-configuration.md`
- `docs/source/user-guide/example-usage.md`
```shell
cargo check -p datafusion
```

Then commit the changes and create a PR targeting the release branch `branch-N`.

Expand Down
73 changes: 6 additions & 67 deletions dev/update_datafusion_versions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,29 +26,6 @@
from pathlib import Path
import tomlkit

crates = {
'datafusion-common': 'datafusion/common/Cargo.toml',
'datafusion-common-runtime': 'datafusion/common-runtime/Cargo.toml',
'datafusion': 'datafusion/core/Cargo.toml',
'datafusion-execution': 'datafusion/execution/Cargo.toml',
'datafusion-expr': 'datafusion/expr/Cargo.toml',
'datafusion-ffi': 'datafusion/ffi/Cargo.toml',
'datafusion-functions': 'datafusion/functions/Cargo.toml',
'datafusion-functions-aggregate': 'datafusion/functions-aggregate/Cargo.toml',
'datafusion-functions-nested': 'datafusion/functions-nested/Cargo.toml',
'datafusion-optimizer': 'datafusion/optimizer/Cargo.toml',
'datafusion-physical-expr': 'datafusion/physical-expr/Cargo.toml',
'datafusion-physical-expr-common': 'datafusion/physical-expr-common/Cargo.toml',
'datafusion-physical-plan': 'datafusion/physical-plan/Cargo.toml',
'datafusion-proto': 'datafusion/proto/Cargo.toml',
'datafusion-sql': 'datafusion/sql/Cargo.toml',
'datafusion-sqllogictest': 'datafusion/sqllogictest/Cargo.toml',
'datafusion-substrait': 'datafusion/substrait/Cargo.toml',
'datafusion-wasmtest': 'datafusion/wasmtest/Cargo.toml',
'datafusion-benchmarks': 'benchmarks/Cargo.toml',
'datafusion-cli': 'datafusion-cli/Cargo.toml',
'datafusion-examples': 'datafusion-examples/Cargo.toml',
}
Comment on lines -29 to -51

@nuno-faria nuno-faria Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was not catching everything, and is also not needed since all crates start with "datafusion".


def update_workspace_version(new_version: str):
cargo_toml = 'Cargo.toml'
Expand All @@ -64,47 +41,15 @@ def update_workspace_version(new_version: str):

doc = tomlkit.parse(data)

for crate in crates.keys():
df_dep = doc.get('workspace').get('dependencies', {}).get(crate)
for crate, df_dep in doc['workspace']['dependencies'].items():
if not crate.startswith('datafusion'):

@VVKot VVKot Oct 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per your above comment, this seems like a reasonable way to enumerate all crates now - but I don't know if that will always be the case.

Would love one of the committers to opine whether we can expect all crate names to start with datafusion

@nuno-faria nuno-faria Oct 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe so, all DataFusion subcrates get updated when a new version is released, and they should always have the same prefix.

continue
# skip crates that pin datafusion using git hash
if df_dep is not None and df_dep.get('version') is not None:
print(f'updating {crate} dependency in {cargo_toml}')
df_dep['version'] = new_version

with open(cargo_toml, 'w') as f:
f.write(tomlkit.dumps(doc))


def update_datafusion_version(cargo_toml: str, new_version: str):
print(f'updating {cargo_toml}')
with open(cargo_toml) as f:
data = f.read()

doc = tomlkit.parse(data)
pkg = doc.get('package')
if 'workspace' not in pkg['version']:
pkg['version'] = new_version

with open(cargo_toml, 'w') as f:
f.write(tomlkit.dumps(doc))

def update_downstream_versions(cargo_toml: str, new_version: str):
with open(cargo_toml) as f:
data = f.read()

doc = tomlkit.parse(data)

for crate in crates.keys():
df_dep = doc.get('dependencies', {}).get(crate)
# skip crates that pin datafusion using git hash
if df_dep is not None and df_dep.get('version') is not None:
print(f'updating {crate} dependency in {cargo_toml}')
df_dep['version'] = new_version

df_dep = doc.get('dev-dependencies', {}).get(crate)
if df_dep is not None and df_dep.get('version') is not None:
print(f'updating {crate} dev-dependency in {cargo_toml}')
df_dep['version'] = new_version
Comment on lines -78 to -107

@nuno-faria nuno-faria Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The subcrates use the workspace version directly, so this is not needed anymore.

doc['workspace']['package']['version'] = new_version

with open(cargo_toml, 'w') as f:
f.write(tomlkit.dumps(doc))
Expand All @@ -117,6 +62,7 @@ def update_docs(path: str, new_version: str):
fd.seek(0)
content = re.sub(r'datafusion\s*=\s*"(.+?)"', f'datafusion = "{new_version}"', content)
content = re.sub(r'datafusion\s*=\s*\{\s*version\s*=\s*"(.+?)"', f'datafusion = {{ version = "{new_version}"', content)
content = re.sub(r'datafusion version \d+\.\d+\.\d+', f'datafusion version {new_version}', content)
fd.truncate()
fd.write(content)

Expand Down Expand Up @@ -151,18 +97,11 @@ def main():
print(f'Updating workspace in {repo_root} to {new_version}')
update_workspace_version(new_version)

print(f'Updating datafusion crate versions in {repo_root} to {new_version}')
for cargo_toml in crates.values():
update_datafusion_version(cargo_toml, new_version)

print(f'Updating datafusion dependency versions in {repo_root} to {new_version}')
for cargo_toml in crates.values():
update_downstream_versions(cargo_toml, new_version)

update_docs("README.md", new_version)
update_docs("docs/source/download.md", new_version)
update_docs("docs/source/user-guide/example-usage.md", new_version)
update_docs("docs/source/user-guide/crate-configuration.md", new_version)
update_docs("docs/source/user-guide/configs.md", new_version)

@VVKot VVKot Oct 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see my top-level comment - update_docs is a bit too naive to properly update configs.md

@nuno-faria nuno-faria Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the issue is, the diff looks ok to me.

@VVKot VVKot Oct 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created_by seems to be the default from 55.1.0 but the script will keep updating it #26007 (review) , which is misleading

@nuno-faria nuno-faria Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not sure what the issue is, the created_by version in configs.md needs to be updated to the new one as well. The value should match CARGO_PKG_VERSION.

@VVKot VVKot Oct 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, my initial interpretation of this row in the table was

created_by property is set by default since 55.1.0

whereas it actually means

created_by property will be set to string datafusion version 55.1.0

All good here.


update_ci(new_version)

Expand Down
Loading