[Python] Support converting list of multi-dimensional arrays to FixedShapeTensor

@rok

Describe the enhancement requested

When you want to construct a fixed shape tensor array, not from a single contiguous ndarray, but from a list of the individual array elements, this "accidentally" works for 1d arrays (because we support that for ListArray):

>>> pa.array([np.array([1,2,3]), np.array([2,3,4])], type=pa.fixed_shape_tensor(pa.int64(), shape=(3,)))
<pyarrow.lib.FixedShapeTensorArray object at 0x7fadc4d1f3a0>
[
  [
    1,
    2,
    3
  ],
  [
    2,
    3,
    4
  ]
]

But trying the same for >1 d arrays, you get the following error:

>>> pa.array([np.array([[1,2,3]]), np.array([[2,3,4]])], type=pa.fixed_shape_tensor(pa.int64(), shape=(1, 3)))
...
ArrowInvalid: Can only convert 1-dimensional array values

AFAIK, the only way to currently do this conversion, is to convert the list of arrays to a single ndarray yourself, and then use pa.FixedShapeTensorArray.from_numpy_ndarray.

(cc @rok)

Component(s)

Python

And probably worth a separate issue, but also for the conversion the other way around to python, it would be nice to preserve the shape of the ndarrays:

>>> arr = pa.FixedShapeTensorArray.from_numpy_ndarray(np.stack([np.array([[1,2,3], [3,4,5]]), np.array([[2,3,4], [5,6,7]])], axis=1))
# results in an array of 1D arrays
>>> arr.to_numpy(zero_copy_only=False)
array([array([1, 2, 3, 2, 3, 4]), array([3, 4, 5, 5, 6, 7])], dtype=object)
# the scalars do have a method to convert to the correct shape
>>> [v.to_numpy() for v in arr]
[array([[1, 2, 3],
        [2, 3, 4]]),
 array([[3, 4, 5],
        [5, 6, 7]])]

Agreed on both. Let's try to get this into the next release.

apacheGH-49644: [Python] Make fixed_size_list test deterministic and …

apacheGH-49644: [Python] Support converting list of multi-dimensional…

apacheGH-49644: [Python] Make fixed_size_list test deterministic and …

apacheGH-49644: [Python] Only flatten ndim>=2 ndarrays for fixed-size…

apacheGH-49644: [Python] Fix clang-format in AppendNdarray

apacheGH-49644: [Python] Validate tensor shape, reject permuted tenso…

apacheGH-49644: [Python] Trim comments per review

apacheGH-49644: [Python] Keep generic error message for 0-dimensional…

apacheGH-49644: [Python] Validate all sequence inputs, not just list/…

apacheGH-49644: [Python] Address review: allow 0-D arrays, name type …

apacheGH-49644: [Python] Trim test comments per review

apacheGH-49644: [Python] Add a docstring example for building from a …

@aboderinsamuel

@aboderinsamuel please write "take" in a comment below.

take

GH-49644: [Python] Support converting list of multi-dimensional array…

Issue resolved by pull request 50203
#50203

Navigation Menu

Repository navigation

[Python] Support converting list of multi-dimensional arrays to FixedShapeTensor #49644

Describe the enhancement requested

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python] Support converting list of multi-dimensional arrays to FixedShapeTensor #49644

Description

Describe the enhancement requested

Component(s)

Activity

jorisvandenbossche commented on Apr 2, 2026

rok commented on Apr 7, 2026

rok commented on Jun 29, 2026

aboderinsamuel commented on Jun 29, 2026

rok commented on Jun 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions