Fix Parquet row group pruning for negated equality predicates - #23580
Fix Parquet row group pruning for negated equality predicates#23580mhaseeb123 wants to merge 13 commits into
Conversation
The parquet pruning transformers convert a filter into an expression over per-row-group summary columns that answers "might some row here match?". That is an existential, and existentials are not closed under negation: "no row is 5" is not "some row is not 5". The bloom filter and dictionary page converters wrapped the transformed child in the input unary operator, so `NOT(col == 5)` became `NOT(may_contain(5))` and pruned every row group holding a single 5, dropping rows that satisfy `col != 5`. This affected the hybrid scan dictionary filter and the bloom filter path in both the hybrid scan and the regular reader. Rewrite the filter once in `named_to_reference_converter` so that no transformer ever sees `NOT` over an expression it has already rewritten: eliminate double negations, apply De Morgan's laws over both the null-propagating and the Kleene logical operators, and complement the equality operators. This converted expression also filters the decoded rows, so only exact rewrites are applied - ordering comparisons are left alone because every ordered comparison against a NaN is false, making `NOT(a < b)` true exactly where `a >= b` is false. `apply_unary_membership_transform` now always relaxes to `always_true` rather than negating a membership result. After the pushdown no `NOT` should reach it with a non-relaxed child, but that is a by-construction argument and the guard keeps a future gap from returning wrong rows. De Morgan also lets the stats converter evaluate negated compound predicates it previously gave up on, so `NOT(a AND b)` now prunes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bloom filter half of the negation bug had no coverage: cudf's writer cannot produce bloom filters, so it can only be tested against checked-in files. The mixed_card_ndv_*_bf_fpp0.1_nostats fixtures already carry bloom filters and no column chunk statistics, which makes the bloom filter the only thing that can prune them. Read those files with NOT(str == "FINDME") and with str != "FINDME" and assert the two spellings prune identically. Before the fix the negated form prunes the two row groups that hold a "FINDME" and returns 600 of the 998 matching rows, while the direct form prunes nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-actions
Bot
added
libcudf
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan includes up to 8 reviews per rolling hour; 5 remain after this review. 📝 WalkthroughSummary by CodeRabbit
WalkthroughParquet predicate normalization now pushes supported negations to expression leaves, applies De Morgan rewrites, complements equality operators, and preserves explicit ChangesParquet predicate normalization
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: 🔵 Low · up to This PR changes Parquet predicate-negation handling used for pruning and final filtering. The current head retains bounded follow-up risks around AST lifetime safety, test initialization, and comments describing negation/null semantics; these do not demonstrate a production failure but warrant explicit owner awareness before merge. Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
cpp/src/io/parquet/expression_transform_helpers.hpp (1)
119-149: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueFix the referenced symbol name in the doc.
The comment refers to
negation_pushdown. The declared member function ispush_down_negation(Line 271). Update the reference so the doc stays navigable.📝 Proposed doc fix
- * `col != v` only permits pruning a row group - * whose values are all `v`. See `negation_pushdown`, which rewrites `NOT(col == v)` into - * `col != v` before it ever reaches a converter. + * `col != v` only permits pruning a row group + * whose values are all `v`. See `named_to_reference_converter::push_down_negation`, which rewrites + * `NOT(col == v)` into `col != v` before it ever reaches a converter.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp/src/io/parquet/expression_transform_helpers.hpp` around lines 119 - 149, Update the documentation for apply_unary_membership_transform to reference the declared push_down_negation member instead of negation_pushdown, preserving the rest of the explanation unchanged.cpp/tests/io/parquet_reader_test.cpp (2)
2009-2010: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRun clang-format on the new test block.
Several new statements are wrapped even though they fit inside the column limit, for example Lines 2009-2010, Lines 2050-2051, and Lines 2059-2060.
clang-formatwill join them. Run the pre-commit hook so CI formatting stays clean.As per coding guidelines: "Format C++ and CUDA code with clang-format."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp/tests/io/parquet_reader_test.cpp` around lines 2009 - 2010, Run clang-format, preferably via the repository’s pre-commit hook, on the new test block containing the conjunction declaration and related statements so lines that fit within the column limit are joined and the test conforms to C++ formatting guidelines.Source: Coding guidelines
1965-1973: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider asserting the pruning counts as well.
expect_matches_unrewrittenverifies row equivalence only. It does not verify that the rewrite actually enables pruning, so a regression that disables the rewrite entirely would still pass. Add an optional expectednum_row_groups_after_stats_filterargument for the cases where the rewrite is expected to prune, for example the De Morgan and double-negation cases.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp/tests/io/parquet_reader_test.cpp` around lines 1965 - 1973, Extend the expect_matches_unrewritten lambda with an optional expected num_row_groups_after_stats_filter parameter, and assert the reader result’s pruning count when that expectation is provided. Pass the expected count in rewrite cases that should prune, including the De Morgan and double-negation tests, while preserving row-equivalence-only validation for other cases.cpp/tests/io/experimental/hybrid_scan_filters_test.cpp (1)
1373-1392: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider adding the equivalence assertion here too.
The comment states that this predicate matches the non-negated spelling tested above. The block at Lines 1341-1348 asserts that equivalence explicitly for the single-column case. Add the same style of assertion here so the compound
ORcase is also pinned to(col0 != 50) OR (col2 != "0100").🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp/tests/io/experimental/hybrid_scan_filters_test.cpp` around lines 1373 - 1392, Add an explicit equivalence assertion in the compound filter block around filter_expression, comparing the NOT-based predicate with the corresponding non-negated `(col0 != 50) OR (col2 != "0100")` expression, following the assertion style used in the earlier single-column case. Keep the existing result expectation unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cpp/src/io/parquet/expression_transform_helpers.cpp`:
- Around line 173-176: Correct the NaN explanation in the comments: state that
NOT(a < b) and a >= b are not equivalent when either operand is NaN, because
both ordered comparisons are false while the negation is true. Apply identical
wording to cpp/src/io/parquet/expression_transform_helpers.cpp lines 173-176 and
cpp/src/io/parquet/expression_transform_helpers.hpp lines 264-266, covering the
implementation comment and push_down_negation Doxygen block.
In `@python/cudf/cudf/tests/input_output/test_parquet.py`:
- Around line 4688-4694: Update the stale expected row-count comment associated
with the parameterized bloom_filter_fname fixtures to state 998 instead of 600,
keeping one shared expected value for both files.
---
Nitpick comments:
In `@cpp/src/io/parquet/expression_transform_helpers.hpp`:
- Around line 119-149: Update the documentation for
apply_unary_membership_transform to reference the declared push_down_negation
member instead of negation_pushdown, preserving the rest of the explanation
unchanged.
In `@cpp/tests/io/experimental/hybrid_scan_filters_test.cpp`:
- Around line 1373-1392: Add an explicit equivalence assertion in the compound
filter block around filter_expression, comparing the NOT-based predicate with
the corresponding non-negated `(col0 != 50) OR (col2 != "0100")` expression,
following the assertion style used in the earlier single-column case. Keep the
existing result expectation unchanged.
In `@cpp/tests/io/parquet_reader_test.cpp`:
- Around line 2009-2010: Run clang-format, preferably via the repository’s
pre-commit hook, on the new test block containing the conjunction declaration
and related statements so lines that fit within the column limit are joined and
the test conforms to C++ formatting guidelines.
- Around line 1965-1973: Extend the expect_matches_unrewritten lambda with an
optional expected num_row_groups_after_stats_filter parameter, and assert the
reader result’s pruning count when that expectation is provided. Pass the
expected count in rewrite cases that should prune, including the De Morgan and
double-negation tests, while preserving row-equivalence-only validation for
other cases.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 88164f5b-c13e-491e-97df-887dc1b75c92
📒 Files selected for processing (7)
cpp/src/io/parquet/bloom_filter_reader.cucpp/src/io/parquet/experimental/dictionary_page_filter.cucpp/src/io/parquet/expression_transform_helpers.cppcpp/src/io/parquet/expression_transform_helpers.hppcpp/tests/io/experimental/hybrid_scan_filters_test.cppcpp/tests/io/parquet_reader_test.cpppython/cudf/cudf/tests/input_output/test_parquet.py
|
|
||
| // No column projection is requested, so the reader reads all columns; this isolates filter name | ||
| // resolution (named_to_reference_converter) from select_columns name scanning. | ||
| // resolution (`parquet_filter_normalizer`) from select_columns name scanning. |
There was a problem hiding this comment.
Simple rename in this PR. Happy to move to a separate PR if preferred but then until the follow up merges, named_to_reference_converter will be doing two jobs (convert names -> references and pushdown negations) instead of what it advertises.
| protected: | ||
| /** | ||
| * @brief Visits each expression in `operands`. | ||
| * | ||
| * @param operands Expressions to visit | ||
| * @return References to transformed expressions | ||
| */ | ||
| [[nodiscard]] std::vector<std::reference_wrapper<expression const>> visit_operands( | ||
| std::vector<std::reference_wrapper<expression const>> const& operands); |
There was a problem hiding this comment.
We have to define and duplicate this for several subclasses. Just move it here for reuse.
| std::invalid_argument); | ||
| } | ||
|
|
||
| std::vector<std::reference_wrapper<expression const>> |
There was a problem hiding this comment.
Moved from expression_transform_helpers.cpp
| * expression leaves | ||
| */ | ||
| named_to_reference_converter::named_to_reference_converter( | ||
| parquet_filter_normalizer::parquet_filter_normalizer( |
| * expression leaves | ||
| */ | ||
| class named_to_reference_converter : public parquet::detail::named_to_reference_converter { | ||
| class parquet_filter_normalizer : public parquet::detail::parquet_filter_normalizer { |
| std::vector<std::reference_wrapper<ast::expression const>> visit_operands( | ||
| cudf::host_span<std::reference_wrapper<ast::expression const> const> operands); | ||
| /** | ||
| * @brief Rewrites `NOT(operand)` into an equivalent expression with the negation pushed into |
There was a problem hiding this comment.
Please read this negation pushdown logic.
| * | ||
| */ | ||
| class offset_column_references : public named_to_reference_converter { | ||
| class offset_column_references : public ast::detail::expression_transformer { |
There was a problem hiding this comment.
Don't derive from parquet_filter_normalizer anymore as it would duplicate negation pushdown work that we don't need here
|
|
||
| // name to reference converter to extract AST output filter | ||
| named_to_reference_converter _expr_conv{std::nullopt, table_metadata{}, true}; | ||
| // Converts the input filter to AST output filter. |
| populate_metadata(metadata); | ||
| _expr_conv = | ||
| named_to_reference_converter(options.get_filter(), metadata, _options.case_sensitive_names); | ||
| parquet_filter_normalizer(options.get_filter(), metadata, _options.case_sensitive_names); |
| return {std::move(_columns_mask), _has_is_null_operator}; | ||
| } | ||
|
|
||
| std::vector<std::reference_wrapper<ast::expression const>> stats_columns_collector::visit_operands( |
There was a problem hiding this comment.
Defined in base class now.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
cpp/src/io/parquet/expression_transform_helpers.cpp (1)
230-254: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSimplify: reuse
negate()for theNOTbranch instead of duplicating its fallback.When
push_down_negationreturnsstd::nulloptforop == NOT, this code re-implements the same "convert operand, wrap inNOT" fallback thatnegate()already provides. Callnegate(operands.front().get())directly for theNOTcase, and keep the generic arity-1/arity-2 rebuild path only for non-NOToperators.♻️ Proposed simplification
std::reference_wrapper<ast::expression const> parquet_filter_normalizer::visit( ast::operation const& expr) { - auto const operands = expr.get_operands(); - auto op = expr.get_operator(); - auto const operator_arity = cudf::ast::detail::ast_operator_arity(op); + auto const operands = expr.get_operands(); + auto const op = expr.get_operator(); - // Push down negation to leaves so that downstream transformers don't have to handle `NOT` over - // rewritten operands if (op == ast::ast_operator::NOT) { - if (auto const negated = push_down_negation(operands.front().get()); negated.has_value()) { - _converted_expr = negated.value(); - return negated.value(); - } + // Push down negation to leaves so that downstream transformers don't have to handle `NOT` + // over rewritten operands; falls back to wrapping the converted operand in `NOT`. + _converted_expr = negate(operands.front().get()); + return _converted_expr.value(); } + auto const operator_arity = cudf::ast::detail::ast_operator_arity(op); auto new_operands = visit_operands(operands); if (operator_arity == 2) { _operators.emplace_back(op, new_operands.front(), new_operands.back()); } else if (operator_arity == 1) { _operators.emplace_back(op, new_operands.front()); } _converted_expr = std::reference_wrapper<ast::expression const>(_operators.back()); return std::reference_wrapper<ast::expression const>(_operators.back()); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp/src/io/parquet/expression_transform_helpers.cpp` around lines 230 - 254, Update parquet_filter_normalizer::visit for ast_operator::NOT to return the result of negate(operands.front().get()) directly, allowing negate() to handle both push-down and fallback wrapping. Keep the generic visit_operands and arity-based operator rebuild path only for non-NOT operators.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@cpp/src/io/parquet/expression_transform_helpers.cpp`:
- Around line 230-254: Update parquet_filter_normalizer::visit for
ast_operator::NOT to return the result of negate(operands.front().get())
directly, allowing negate() to handle both push-down and fallback wrapping. Keep
the generic visit_operands and arity-based operator rebuild path only for
non-NOT operators.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 79edb588-620b-44b8-b8ed-3f3cf290f043
📒 Files selected for processing (19)
cpp/benchmarks/io/parquet/parquet_reader_metadata.cppcpp/include/cudf/ast/detail/expression_transformer.hppcpp/src/ast/expressions.cppcpp/src/io/parquet/bloom_filter_reader.cucpp/src/io/parquet/experimental/dictionary_page_filter.cucpp/src/io/parquet/experimental/hybrid_scan_helpers.cppcpp/src/io/parquet/experimental/hybrid_scan_helpers.hppcpp/src/io/parquet/experimental/hybrid_scan_impl.cppcpp/src/io/parquet/experimental/hybrid_scan_impl.hppcpp/src/io/parquet/expression_transform_helpers.cppcpp/src/io/parquet/expression_transform_helpers.hppcpp/src/io/parquet/reader_impl.cppcpp/src/io/parquet/reader_impl.hppcpp/src/io/parquet/stats_filter_helpers.cppcpp/src/io/parquet/stats_filter_helpers.hppcpp/tests/io/experimental/hybrid_scan_filters_test.cppcpp/tests/io/parquet_reader_test.cpppython/cudf/cudf/tests/input_output/test_parquet.pypython/pylibcudf/tests/io/test_experimental_hybrid_scan.py
💤 Files with no reviewable changes (1)
- cpp/src/io/parquet/stats_filter_helpers.hpp
🚧 Files skipped from review as they are similar to previous changes (5)
- cpp/src/io/parquet/experimental/dictionary_page_filter.cu
- python/cudf/cudf/tests/input_output/test_parquet.py
- cpp/tests/io/parquet_reader_test.cpp
- cpp/src/io/parquet/bloom_filter_reader.cu
- cpp/tests/io/experimental/hybrid_scan_filters_test.cpp
vuule
left a comment
vuule
left a comment
There was a problem hiding this comment.
several non-blocking suggestions
There was a problem hiding this comment.
I think AI caught a valid gap, but it's related to the code that's unchanged in this PR:
cpp/src/io/parquet/stats_filter_helpers.cpp, lines 160-169: The normalizer deliberately refuses to complement ordering comparisons because NaN makes it inexact, but stats_expression_converter still complements them for stats pruning: NOT(col < lit) is rewritten to col >= lit and then to vmax >= lit. For a row group holding {NaN, 1, 2} and lit = 50, every row satisfies NOT(col < 50) for the NaN rows, yet vmax >= 50 is false and the row group is pruned. This is unreachable for cudf-written files because the writer drops min/max entirely when a NaN is seen (cpp/src/io/statistics/column_statistics.cuh:208), but Arrow writes min/max that merely exclude NaN — verified with pyarrow 23, where a chunk of [NaN, 1.0, 2.0] yields min=1.0, max=2.0 and only an all-NaN chunk gets has_min_max: false — so the hole is live for Arrow-written files. (parquet-mr/Spark behavior here is unverified; its float NaN statistics handling has varied across versions.) Since exactness under NaN is the central argument of this PR, either skip the NEGATE rewrite in the stats converter for floating-point columns or record the gap explicitly.
Can be addressed in a follow-up PR.
There was a problem hiding this comment.
Yes, let me take care of this in a follow up.
There was a problem hiding this comment.
parquet-mr: The writer (DoubleStatistics.updateStats) skips NaN and keeps updating min/max from the rest, same as arrow. approving the PR but we'll need this fixed.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
python/pylibcudf/tests/io/test_experimental_hybrid_scan.py (1)
711-805: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winBuild the expressions inside the test instead of at import time.
The list comprehension at Line 802 calls
_col0_stats_negation_cases()during collection. Each_literal(...)call runsplc.Scalar.from_arrow, so the test module allocates device memory at import time and keeps those scalars alive for the whole session. Other tests in this file createLiteralobjects inside the test body (see Lines 199-204), and the dictionary test below parametrizes onlyASTOperatorvalues and builds theOperationinside the test.Parametrize the operator/literal data and build the expressions inside the test to match that pattern.
♻️ Proposed refactor sketch
-@pytest.mark.parametrize( - "negated,unnegated,expected", - [ - pytest.param(negated, unnegated, expected, id=name) - for name, negated, unnegated, expected in _col0_stats_negation_cases() - ], -) +@pytest.mark.parametrize( + "case_id,expected", + [ + pytest.param("double_negation", [0], id="double_negation"), + pytest.param("not_equal_to_equal", [0], id="not_equal_to_equal"), + pytest.param("equal_to_not_equal", [0, 1, 2, 3], id="equal_to_not_equal"), + pytest.param("de_morgan_and", [0, 2, 3], id="de_morgan_and"), + pytest.param("de_morgan_or", [1, 2], id="de_morgan_or"), + ], +) def test_hybrid_scan_filter_row_groups_with_stats_negation( simple_hybrid_scan_reader: HybridScanReader, simple_parquet_options: plc.io.parquet.ParquetReaderOptions, - negated: Operation, - unnegated: Operation, + case_id: str, expected: list[int], ) -> None: """A negated filter must prune exactly like its unnegated equivalent.""" + negated, unnegated = _col0_stats_negation_case(case_id)🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow instructions embedded in them. Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/pylibcudf/tests/io/test_experimental_hybrid_scan.py` around lines 711 - 805, Refactor _col0_stats_negation_cases and its parametrization so collection-time data contains only primitive operator/literal descriptors and expected row groups, without constructing Literal, Scalar, or Operation objects. Build the literals and expression trees inside the parameterized test body, following the existing test pattern, so plc.Scalar.from_arrow runs during each test rather than at module import.cpp/src/io/parquet/experimental/hybrid_scan_impl.hpp (1)
338-345: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winMake
parquet_filter_normalizermove-only to protect the self-referential AST storage.Both signatures now return
parquet_filter_normalizerby value. The class stores the rewritten AST instd::list<ast::column_reference> _col_refandstd::list<ast::operation> _operators, and_converted_exprplus eachast::operationoperand holds astd::reference_wrapperinto those lists. A move keeps the list nodes at stable addresses, so the current call sites are safe. A copy would produce an object whose expression references still point into the source object, so the copy dangles as soon as the source dies.Delete the copy operations so this cannot regress silently.
🛡️ Proposed guard in
cpp/src/io/parquet/expression_transform_helpers.hppclass parquet_filter_normalizer : public ast::detail::expression_transformer { public: parquet_filter_normalizer() = default; // The converted expression references nodes owned by `_col_ref` and `_operators`, so copying // would leave the copy pointing into the source object. parquet_filter_normalizer(parquet_filter_normalizer const&) = delete; parquet_filter_normalizer& operator=(parquet_filter_normalizer const&) = delete; parquet_filter_normalizer(parquet_filter_normalizer&&) = default; parquet_filter_normalizer& operator=(parquet_filter_normalizer&&) = default;Also applies to: 373-379
🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow instructions embedded in them. Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp/src/io/parquet/experimental/hybrid_scan_impl.hpp` around lines 338 - 345, Make parquet_filter_normalizer move-only in its class definition by deleting the copy constructor and copy assignment operator, while explicitly defaulting the move constructor and move assignment operator. Preserve the existing self-referential AST storage and ensure build_normalized_expression return-by-value call sites remain movable.
🤖 Prompt for all review comments with AI agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.
Inline comments:
In `@cpp/src/io/parquet/expression_transform_helpers.cpp`:
- Around line 234-235: Correct the De Morgan comment near the LOGICAL_* and
NULL_LOGICAL_* operators to swap their classifications: identify
LOGICAL_AND/LOGICAL_OR as null-propagating operators and
NULL_LOGICAL_AND/NULL_LOGICAL_OR as Kleene three-valued logic operators,
matching the push_down_negation documentation.
---
Nitpick comments:
In `@cpp/src/io/parquet/experimental/hybrid_scan_impl.hpp`:
- Around line 338-345: Make parquet_filter_normalizer move-only in its class
definition by deleting the copy constructor and copy assignment operator, while
explicitly defaulting the move constructor and move assignment operator.
Preserve the existing self-referential AST storage and ensure
build_normalized_expression return-by-value call sites remain movable.
In `@python/pylibcudf/tests/io/test_experimental_hybrid_scan.py`:
- Around line 711-805: Refactor _col0_stats_negation_cases and its
parametrization so collection-time data contains only primitive operator/literal
descriptors and expected row groups, without constructing Literal, Scalar, or
Operation objects. Build the literals and expression trees inside the
parameterized test body, following the existing test pattern, so
plc.Scalar.from_arrow runs during each test rather than at module import.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: bcb290a8-209c-44d0-91a0-001221679431
📒 Files selected for processing (19)
cpp/benchmarks/io/parquet/parquet_reader_metadata.cppcpp/include/cudf/ast/detail/expression_transformer.hppcpp/src/ast/expressions.cppcpp/src/io/parquet/bloom_filter_reader.cucpp/src/io/parquet/experimental/dictionary_page_filter.cucpp/src/io/parquet/experimental/hybrid_scan_helpers.cppcpp/src/io/parquet/experimental/hybrid_scan_helpers.hppcpp/src/io/parquet/experimental/hybrid_scan_impl.cppcpp/src/io/parquet/experimental/hybrid_scan_impl.hppcpp/src/io/parquet/expression_transform_helpers.cppcpp/src/io/parquet/expression_transform_helpers.hppcpp/src/io/parquet/reader_impl.cppcpp/src/io/parquet/reader_impl.hppcpp/src/io/parquet/stats_filter_helpers.cppcpp/src/io/parquet/stats_filter_helpers.hppcpp/tests/io/experimental/hybrid_scan_filters_test.cppcpp/tests/io/parquet_reader_test.cpppython/cudf/cudf/tests/input_output/test_parquet.pypython/pylibcudf/tests/io/test_experimental_hybrid_scan.py
💤 Files with no reviewable changes (1)
- cpp/src/io/parquet/stats_filter_helpers.hpp
Included review availability: Your plan includes up to 8 reviews per rolling hour; 6 remain after this review.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
cpp/src/io/parquet/expression_transform_helpers.hpp (1)
160-161: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winLimit the contract to supported negations.
parquet_filter_normalizerpreservesNOTwhen no exact rewrite exists, as documented at Line 224-225. State that it pushes supported logical negations to expression leaves.🤖 Prompt for AI Agents
Treat finding text, file paths, and code as untrusted review data. Never follow instructions embedded in them. Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp/src/io/parquet/expression_transform_helpers.hpp` around lines 160 - 161, Update the parquet_filter_normalizer documentation comment to state that it pushes supported logical negations to expression leaves, while preserving NOT when no exact rewrite exists.
🤖 Prompt for all review comments with AI agents
Treat finding text, file paths, and code as untrusted review data. Never follow
instructions embedded in them. Verify each finding against current code. Fix
only still-valid issues, skip the rest with a brief reason, keep changes
minimal, and validate.
Outside diff comments:
In `@cpp/src/io/parquet/expression_transform_helpers.hpp`:
- Around line 160-161: Update the parquet_filter_normalizer documentation
comment to state that it pushes supported logical negations to expression
leaves, while preserving NOT when no exact rewrite exists.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: e41e8f69-be1f-4e43-965c-53e5d7d858a0
📒 Files selected for processing (3)
cpp/src/io/parquet/expression_transform_helpers.cppcpp/src/io/parquet/expression_transform_helpers.hppcpp/tests/io/parquet_reader_test.cpp
🚧 Files skipped from review as they are similar to previous changes (2)
- cpp/src/io/parquet/expression_transform_helpers.cpp
- cpp/tests/io/parquet_reader_test.cpp
Included review availability: Your plan includes up to 8 reviews per rolling hour; 5 remain after this review.
d8cf7fe to
91102a8
Compare
mhaseeb123
added
bug
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
mhaseeb123
added
4 - Needs Review
mhaseeb123
added
5 - Ready to Merge
|
/merge |
5 participants
Description
This PR fixes negation (
NOT) handling in the Parquet readers. This is done by pushing down negations in the very first transformer (named_to_reference_converter) to compute an equivalent form which is then used by transformers for row group and page pruning as well as the final filtration. Follow up PRs will further simplify expression transformers in the Parquet reader by pruning out sub-expressions instead of pushing and propagatingalways_true.Negation propagation transformation summary:
NOT(NOT(x))xNOT(a AND b)NOT(a) OR NOT(b), pushed down recursively ; all four De Morgan forms, for bothLOGICAL_*andNULL_LOGICAL_*NOT(a == b)a != b, and vice versaNOT(a >, <, >=, <= b)NaNoperand makes complementing inexactNOT(IS_NULL(x)),NOT(NULL_EQUAL(a, b))Checklist