Describe the bug
Passing a filter expression that contains IS_NULL or NOT(IS_NULL(...)) on a struct or list
column to the libcudf parquet reader raises an error:
ValueError: CUDF failure at: cpp/src/io/parquet/reader_impl_helpers.cpp:66:
Column chunk with schema index 1 not found in row group
Steps/Code to reproduce bug
import tempfile, pathlib
import polars as pl
import pylibcudf as plc
from pylibcudf import expressions as plc_expr
with tempfile.TemporaryDirectory() as tmp:
path = pathlib.Path(tmp) / "struct.parquet"
pl.DataFrame({
"s": pl.Series(
[{"a": None, "b": 1}, {"a": 2, "b": 3}],
dtype=pl.Struct({"a": pl.Int64, "b": pl.Int64}),
)
}).write_parquet(path)
is_null = plc_expr.Operation(
plc_expr.ASTOperator.IS_NULL,
plc_expr.ColumnNameReference("s"),
)
opts = plc.io.parquet.ParquetReaderOptions.builder(
plc.io.SourceInfo([path])
).build()
opts.set_filter(is_null)
plc.io.parquet.read_parquet(opts)
ValueError: CUDF failure at: /home/coder/cudf/cpp/src/io/parquet/reader_impl_helpers.cpp:66: Column chunk with schema index 1 not found in row group
Expected behavior
Skip statistics-based row-group pruning and fallback to apply predicate as a post-read filter.
Additional context
This came up in #23143 which allows IS_NULL predicates to be pushed down.
Describe the bug
Passing a filter expression that contains
IS_NULLorNOT(IS_NULL(...))on a struct or listcolumn to the libcudf parquet reader raises an error:
Steps/Code to reproduce bug
Expected behavior
Skip statistics-based row-group pruning and fallback to apply predicate as a post-read filter.
Additional context
This came up in #23143 which allows
IS_NULLpredicates to be pushed down.