[Python] Adopt new features and bugfixes from Cython 3.1

Describe the enhancement requested

#47396 will bump minimum required Cython version to 3.1 and as a follow-up we should evaluate and adopt newly available features and bugfixes in PyArrow's Cython codebase. See https://cython.readthedocs.io/en/latest/src/changes.html#beta-1-2025-04-03

Some are already mentioned in the issue #47370:

The cython command has a new option --cache to cache generated files. (#6091)
std::span declarations were added to libcpp. (#6539)
std::string_view declarations were added to libcpp. (#6651)

Component(s)

Python

@AlenkaF

@AlenkaF @pitrou I was going to create a different issue for the Cython cache issue in order to split here from other improvements, but I''ve been trying locally and I've hit an issue that looks like a bug on Cython when extracting the artifacts from the cache. I've been able to solve it locally by modifying how Cython extracts the artifacts from the cache:

[BUG] Cython cache seems to extracts files to wrong folder when finding zip cache files cython/cython#7559

This might require us to wait to enable Cython cache until Cython fixes it. See the issue above for more details.

As a note for future work, building with:

time python -m pip install -C cmake.build-type=Debug -C build.verbose=true --no-deps --no-build-isolation -vv -e .

The first PyArrow build took:

real	1m7.339s
user	3m30.576s
sys	0m13.188s

Once cached, if the bug mentioned above is locally patched:

real	0m58.111s
user	2m56.534s
sys	0m10.796s

Cache seems to improve just by ~15% as it only caches the generation of the CPP files but the C++ compilation of those is still required. We might have to investigate pairing it with ccache or sccache but scikit-build-core rebuilds in a fresh /tmp/ directory each time that might require some extra configuration.

@raulcd

@raulcd you can try with --config-settings=build-dir=build to reuse the same build directory. With that I got rebuilds of around 10s.

(with conda, I am actually using --config-settings=build-dir=build/$CONDA_DEFAULT_ENV such that if I would have multiple conda envs where I am installing a dev version of pyarrow, it uses separate build dirs)

@jorisvandenbossche

Welp, that's nice! Thanks @jorisvandenbossche . We could add a default build-dir configuration to our pyproject.toml.
Using:

time python -m pip install -C cmake.build-type=Debug -C build.verbose=true --no-deps --no-build-isolation -vv --config-settings=build-dir=build -e .

I got on a rebuild:

real	0m11.357s
user	0m15.001s
sys	0m1.548s

That's indeed an amazing improvement.

I've validated that changing a pxd file or a PyArrow C++ file only compiles the required, example:

real	0m40.480s
user	1m54.172s
sys	0m6.618s

You should be able to use hash_dir = false in your ccache configuration.

My fix to solve the issue with Cython cache has been merged upstream at the Cython repository. It'll be released as part of 3.2.5, we will be able to add a check on our CMake to use the cache if version is > than that.

My fix to solve the issue with Cython cache has been merged upstream at the Cython repository. It'll be released as part of 3.2.5, we will be able to add a check on our CMake to use the cache if version is > than that.

FTR, latest Cython release is 3.2.8.

Navigation Menu

Repository navigation

[Python] Adopt new features and bugfixes from Cython 3.1 #47467

Describe the enhancement requested

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python] Adopt new features and bugfixes from Cython 3.1 #47467

Description

Describe the enhancement requested

Component(s)

Activity

raulcd commented on Mar 10, 2026

raulcd commented on Mar 10, 2026

jorisvandenbossche commented on Mar 10, 2026

raulcd commented on Mar 10, 2026

pitrou commented on Mar 10, 2026

raulcd commented on Mar 16, 2026

pitrou commented on Jul 8, 2026

pitrou commented on Sep 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions