Skip to content

[Python] show_info() reports one codec twice and omits lz4_raw #52024

Description

@Nishuuzz

Describe the bug, including details regarding any error messages, version, and platform.

pa.show_info() prints its Compression Codecs section from a hardcoded list in python/pyarrow/__init__.py:

codecs = ["brotli", "bz2", "gzip", "lz4_frame", "lz4", "snappy", "zstd"]

That list has drifted from _ensure_compression() in io.pxi, which is what actually decides which codec names are accepted. Two things follow from it:

  • lz4 and lz4_frame both map to CCompressionType_LZ4_FRAME, so the same codec is reported twice under two names.
  • lz4_raw maps to CCompressionType_LZ4 and is a separate codec, but it is not listed at all, even though pa.Codec.is_available('lz4_raw') returns True.

So seven lines are printed for six codecs, and the codec that is missing is a real one.

Observed, on pyarrow 25.0.0:

>>> import pyarrow as pa
>>> pa.show_info()
...
Compression Codecs:
  brotli              : Enabled
  bz2                 : Enabled
  gzip                : Enabled
  lz4_frame           : Enabled
  lz4                 : Enabled
  snappy              : Enabled
  zstd                : Enabled

Expected: each supported codec listed once, including lz4_raw.

Walking every name _ensure_compression() accepts and the enum value each maps to:

bz2       -> BZ2
gzip      -> GZIP
brotli    -> BROTLI
lz4       -> LZ4_FRAME
lz4_frame -> LZ4_FRAME   # same codec as lz4
lz4_raw   -> LZ4         # distinct, not reported by show_info()
snappy    -> SNAPPY
zstd      -> ZSTD

Platform: Windows 11, CPython 3.12, pyarrow 25.0.0. The list is unchanged on current main, so it affects main too.

A fix is proposed in #51753.

Component(s)

Python

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions