Skip to content

[BUG] Parquet row-group stats filter fails for IS_NULL on nested-type columns #23397

Description

@Matt711

Describe the bug
Passing a filter expression that contains IS_NULL or NOT(IS_NULL(...)) on a struct or list
column to the libcudf parquet reader raises an error:

ValueError: CUDF failure at: cpp/src/io/parquet/reader_impl_helpers.cpp:66:
Column chunk with schema index 1 not found in row group

Steps/Code to reproduce bug

import tempfile, pathlib
import polars as pl
import pylibcudf as plc
from pylibcudf import expressions as plc_expr

with tempfile.TemporaryDirectory() as tmp:
    path = pathlib.Path(tmp) / "struct.parquet"
    pl.DataFrame({
        "s": pl.Series(
            [{"a": None, "b": 1}, {"a": 2, "b": 3}],
            dtype=pl.Struct({"a": pl.Int64, "b": pl.Int64}),
        )
    }).write_parquet(path)

    is_null = plc_expr.Operation(
        plc_expr.ASTOperator.IS_NULL,
        plc_expr.ColumnNameReference("s"),
    )

    opts = plc.io.parquet.ParquetReaderOptions.builder(
        plc.io.SourceInfo([path])
    ).build()
    opts.set_filter(is_null)

    plc.io.parquet.read_parquet(opts)
ValueError: CUDF failure at: /home/coder/cudf/cpp/src/io/parquet/reader_impl_helpers.cpp:66: Column chunk with schema index 1 not found in row group

Expected behavior
Skip statistics-based row-group pruning and fallback to apply predicate as a post-read filter.

Additional context
This came up in #23143 which allows IS_NULL predicates to be pushed down.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcuIOcuIO issuelibcudfAffects libcudf (C++/CUDA) code.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions