Repository navigation
[Python] run_end_encode doesn't support structs and lists #44183
Description
Describe the bug, including details regarding any error messages, version, and platform.
When attempting to use a RunEndEncoded array with either a struct or a list, an exception is raised indicating that no matching kernel is available.
Steps to Reproduce:
Please run the following example code to reproduce the issue:
import pyarrow as pa
for data_type, values in [
[pa.struct([pa.field("age", pa.int32())]), {"age": 20}],
[pa.list_(pa.int32()), [20]]
]:
try:
schema = pa.schema([pa.field("data", pa.run_end_encoded(pa.int16(), data_type))])
data = [
{"data": values},
]
table = pa.Table.from_pylist(data, schema=schema)
except Exception as e:
print(f"Failed with {data_type} {e}")Observed Output:
Failed with struct<age: int32> Function 'run_end_encode' has no kernel matching input types (struct<age: int32>)
Failed with list<item: int32> Function 'run_end_encode' has no kernel matching input types (list<item: int32>)
Expected Behavior:
The code should correctly create a RunEndEncoded array using both struct and list types without raising exceptions.
Environment:
- PyArrow version: 18.0.0.dev
- Python version: 3.12.5
- Platform: Mac OS X
Additional Context:
The failure seems to suggest that the run_end_encode function does not currently support struct or list types, but it's not explicitly documented whether this is intentional or an oversight.
Component(s)
Python
Activity
mapleFU commentedon Sep 23, 2024
cc @felipecrv
felipecrv commentedon Sep 23, 2024
Comparing nested types for equality to run-end encode them can be expensive and unlikely to bring good compression rates. Run-end encoding works better on flat columns.
That said, the most likely reason for a kernel not supporting REE arrays yet is usually: it hasn't been implemented. Because, you know, it takes time ($) to implement custom kernels for REE arrays.
rustyconover commentedon Sep 23, 2024
Hi @felipecrv
Why wouldn't it bring great compression rates if the developer knows the column is mostly constant values?
felipecrv commentedon Sep 23, 2024
All it takes is a new random or misaligned column (struct field) to mess up the repetitiveness of the data.
If you know the data is mostly constant values, you don't need
run_end_encode, because you can produce the run-end encoded array directly without comparing the struct values.You can also go for a struct of run-end-encoded fields (not all of them have to be run-end-encoded) and if the whole struct repeats you can share the same
run_endsarray among the fields (no copying needed).jagill commentedon Jan 16, 2025
As a data point, I am running into this in production, from sources I don't control. So maybe they don't know it's not the best compression :).
That doesn't change the fact that it takes effort to implement, of course.
GH-44183: [C++] Support run-end encoded struct, list (view), large li…
pitrou commentedon Jul 30, 2026
Issue resolved by pull request 50534
#50534
pitrou commentedon Jul 30, 2026
@BenMagyar can you post here so that we can assign this issue to you?
BenMagyar commentedon Jul 30, 2026
I can!