GH-50398/GH-50829 1/6: Use more CPython Limited APIs in python/pyarrow

[fboudra]

Rationale for this change

#50398
#50829

What changes are included in this PR?

First in the series: it replaces macro and struct-field access in
python/pyarrow/src/arrow/python/ with the equivalent public C API calls,
which are also part of the stable API:

• PyBytes_AS_STRING / PyBytes_GET_SIZE become PyBytes_AsString / PyBytes_Size
• PyByteArray_AS_STRING / PyByteArray_GET_SIZE become PyByteArray_AsString / PyByteArray_Size
• Py_TYPE(obj)->tp_name is replaced with helper functions (PyObject_StdStringTypeName) and PyObject_GetAttrString

The calls are plain CPython API, so nothing changes for a normal full API build. No build flags are touched in this PR, which makes it reviewable as a straight behavior-preserving change: configure and test exactly as before.

Also included, found while going through the files: a shadowed variable in AppendUTF32, a missing extension path in the MonthDayNano conversion, and the MonthDayNano docstring.

Verification: local build plus the python/pyarrow test suite, both green.

Are these changes tested?

Yes

Are there any user-facing changes?

No

Was AI used for this PR?

PR code and description written by:

• [x ] Human
• AI

Reviewed before submission by:

• [ x] Human
• AI
• Not reviewed

• GitHub Issue: [Python][Packaging] Support abi3 wheels for pyarrow #50398

[jorisvandenbossche]

@fboudra can you indicate more precisely what this PR contains compared to #50409. Are this all the source code changes from that PR (which already got some review rounds), just with the other changes removed (the benchmark removal)?

[fboudra]

@fboudra can you indicate more precisely what this PR contains compared to #50409. Are this all the source code changes from that PR (which already got some review rounds), just with the other changes removed (the benchmark removal)?

@jorisvandenbossche The comment #50409 (comment) explains why I splitted into multiple PRs in an attempt to make it easier to review. The last PR #51377 has all the patches in the serie (33 patches in total).

The full PR (#51377) contains all of the source changes from #50409, the benchmark removal included, plus 20 new commits on top.

The first 13 commits are the 13 non-merge commits from #50409, cherry-picked onto current main. The five Merge upstream/main commits from #50409 are gone because this branch is rebased instead.

The benchmark removal is kept (Remove python/pyarrow/src/arrow/python/benchmark.cc, Remove pyarrow.benchmark). It's a prerequisite for the limited-API build, so I didn't drop it.

The 20 extra commits are work that isn't in #50409 (the limited-API port is incomplete):

• finishing the limited-API port: the remaining macro fast paths, PyGILState_Check → Py_IsInitialized, the datetime C-API via the capsule, vendored pythoncapi_compat fixes, refcount and stable-ABI semantics fixes, and cp311 compat fixes for Py_REFCNT and PyObject_CallFunction('s#')
• enabling the abi3 build: Py_LIMITED_API=0x030B0000 on arrow_python, Cython --limited-api, .abi3.so module suffix
• packaging and CI: the limited-API symbol audit script, the cp311-abi3 wheel matrix, and the perf gate

So the reviewed changes from #50409 are all here unchanged. If it helps, I can keep #50409 open for the conversions and rebase this PR onto it, so the new work gets reviewed separately.

[jorisvandenbossche]

Yes, I understand why you created multiple PRs (and that is a good idea), but I was specifically asking about this PR.

This PR is just the code changes from #50409 but rebased?

[fboudra]

This one (1/6), yes.

[jorisvandenbossche]

BTW, it would help with reviewing if you left your own further code changes as separate commits, instead of combining it within the rebased commits from the other PR (now it is hard to see which review comments from #50409 you have addressed)

Also included, found while going through the files: a shadowed variable in AppendUTF32, a missing extension path in the MonthDayNano conversion, and the MonthDayNano docstring.

What is this exactly about?

[jorisvandenbossche]

Since this PR is targetting the simple changes like PyList_SET_ITEM replacements, it might be good to complete this aspect here (e.g. there are some remaining PyTuple_GET_SIZE, PyList_GET_SIZE, some remaining places where we access ->tp_name, etc)