Repository navigation
[Python] Define whether user-overridden to_pylist on Array subclasses is honored for nested children #50604
Description
In the review of #50327 it was noted (#50327 (comment)) that the bulk conversion does not consult a user-defined to_pylist override on ExtensionArray subclasses for nested children.
This was already inconsistent before #50327:
ListScalar.as_pycalledchild_values.to_pylist(), so an override on a list child's class was honored;StructScalar.as_pyconverted fields via per-element Scalars, so an override on a struct child's class was ignored.
Top-level overrides dispatch normally through Python method resolution in both the old and the new code, and ExtensionScalar.as_py / __arrow_ext_scalar_class__ — the documented customization point — is honored on all paths.
This issue is to decide the contract:
- document that nested
to_pylistoverrides are not consulted (onlyExtensionScalar.as_pyis), or - detect custom
to_pylistoverrides and route nested conversion through them (sketch in the linked discussion).
Activity
Abhinav0905 commentedon Aug 17, 2026
I can work on a prototype once the intended contract is confirmed. My initial preference is option 1: keep
ExtensionScalar.as_pyas the nested customization point, treat anArray.to_pylistoverride as top-level behavior, and document that distinction explicitly. I would add parity tests covering extension children under both list and struct parents so the behavior is consistent rather than accidental.That approach also keeps the bulk conversion path fast. If option 2 is preferred, I can instead prototype explicit override detection and benchmark nested conversion to quantify the dispatch cost. Would maintainers prefer the documented scalar-customization contract or preservation of nested array overrides?