Describe the bug, including details regarding any error messages, version, and platform.
pa.show_info() prints its Compression Codecs section from a hardcoded list in python/pyarrow/__init__.py:
codecs = ["brotli", "bz2", "gzip", "lz4_frame", "lz4", "snappy", "zstd"]
That list has drifted from _ensure_compression() in io.pxi, which is what actually decides which codec names are accepted. Two things follow from it:
lz4 and lz4_frame both map to CCompressionType_LZ4_FRAME, so the same codec is reported twice under two names.
lz4_raw maps to CCompressionType_LZ4 and is a separate codec, but it is not listed at all, even though pa.Codec.is_available('lz4_raw') returns True.
So seven lines are printed for six codecs, and the codec that is missing is a real one.
Observed, on pyarrow 25.0.0:
>>> import pyarrow as pa
>>> pa.show_info()
...
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
Expected: each supported codec listed once, including lz4_raw.
Walking every name _ensure_compression() accepts and the enum value each maps to:
bz2 -> BZ2
gzip -> GZIP
brotli -> BROTLI
lz4 -> LZ4_FRAME
lz4_frame -> LZ4_FRAME # same codec as lz4
lz4_raw -> LZ4 # distinct, not reported by show_info()
snappy -> SNAPPY
zstd -> ZSTD
Platform: Windows 11, CPython 3.12, pyarrow 25.0.0. The list is unchanged on current main, so it affects main too.
A fix is proposed in #51753.
Component(s)
Python
Describe the bug, including details regarding any error messages, version, and platform.
pa.show_info()prints its Compression Codecs section from a hardcoded list inpython/pyarrow/__init__.py:That list has drifted from
_ensure_compression()inio.pxi, which is what actually decides which codec names are accepted. Two things follow from it:lz4andlz4_frameboth map toCCompressionType_LZ4_FRAME, so the same codec is reported twice under two names.lz4_rawmaps toCCompressionType_LZ4and is a separate codec, but it is not listed at all, even thoughpa.Codec.is_available('lz4_raw')returns True.So seven lines are printed for six codecs, and the codec that is missing is a real one.
Observed, on pyarrow 25.0.0:
Expected: each supported codec listed once, including
lz4_raw.Walking every name
_ensure_compression()accepts and the enum value each maps to:Platform: Windows 11, CPython 3.12, pyarrow 25.0.0. The list is unchanged on current main, so it affects main too.
A fix is proposed in #51753.
Component(s)
Python