Forward-merge release/26.08 into main

[Matt711]

Description

Handle forward merge conflict.

Supercedes #23390

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• Bug Fixes
  • Improved Parquet deletion-vector handling to support both deletion and retention modes.
  • Enhanced pandas-compat indexing/reshape behavior, including more reliable handling of NaN keys and missing-driven dtype promotion in pivot/unstack.
• New Features
  • Expanded experimental VARIANT field path parsing to better support bracketed array indexing.
  • Improved streaming join filter pushdown decisions with cost-aware domain scoring.
• Tests
  • Increased test time limits and adjusted test verbosity to reduce flaky runs.
• Build and Reliability
  • Updated GPU test/runtime wiring for broader CUDA-wheel coverage.

Walkthrough

The PR updates CI execution, CUDA dependencies, bloom-filter sizing, Parquet and ORC behavior, Java artifact packaging, pandas-compatible cuDF reshaping, and cudf-polars join-filter pushdown.

Changes

Runtime and CI compatibility

Layer / File(s)	Summary
CI execution updates ci/run_cudf_polars_*.sh, ci/test_*cudf_polars.sh	Python-enabled test timeouts increase to 5400 seconds and pytest verbosity/fail-fast arguments change.
CUDA dependencies and Parquet support dependencies.yaml, cpp/CMakeLists.txt, cpp/src/io/parquet/stats_filter_helpers.hpp	CUDA cuFile requirements, RMM test linkage, and Parquet null-mask construction are updated.

Bloom-filter sizing

Layer / File(s)	Summary
Byte-based filter sizing cpp/libcudf_streaming/..., python/cudf_streaming/..., cpp/libcudf_streaming/benchmarks/...	Bloom filters now use aligned byte storage sizes with validation and updated bindings, callers, and tests.

Parquet, ORC, and Java

Layer / File(s)	Summary
Retention vectors and VARIANT arrays cpp/include/cudf/io/experimental/..., cpp/src/io/parquet/experimental/..., java/src/..., cpp/tests/io/...	Retention-vector metadata propagates through native and Java APIs; VARIANT extraction supports numeric array paths.
ORC statistics and Maven artifacts cpp/src/io/orc/..., java/ci/...	ORC row statistics accumulate using uint64_t, while Java builds and repository assembly validate sources and Javadoc artifacts.

cuDF pandas compatibility

Layer / File(s)	Summary
Index metadata and reshaping python/cudf/cudf/core/{column_accessor,dataframe,indexed_frame,multiindex,reshape}.py	NaN labels, MultiIndex metadata, stack/unstack ordering, and missing-value integer promotion are updated and tested.

cudf-polars planning

Layer / File(s)	Summary
Cost-aware join-filter pushdown python/cudf_polars/cudf_polars/{dsl,streaming,utils}/..., python/cudf_polars/tests/...	Pushdown candidate selection uses source costs, node reference counts, profitability gates, and updated defaults.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• rapidsai/cudf#23402: Related retention-vector support.
• rapidsai/cudf#23067: Related streaming bloom-filter sizing changes.
• rapidsai/cudf#23255: Related pandas-compatible stack and unstack changes.

Suggested labels: 5 - Ready to Merge, ci

Suggested reviewers: vyasr, bdice, mattgara, tomaugspurger, abellina

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.55% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states this is a forward merge from release/26.08 into main, matching the change set.
Description check	✅ Passed	The description is related to the merge and conflict-resolution work in this pull request.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[Matt711]

/ok to test 03a2919

[Matt711]

/merge nosquash

[rapids-bot]

Commit history integrity check failed: not all commits from original PR #23390 appear to be present individually in this PR's history. This usually happens if commits were squashed during the manual resolution process. Please ensure all original commits are preserved individually. You can fix this and try the /merge nosquash command again.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (7)

cpp/tests/io/parquet_deletion_vectors_test.cpp (1)

605-638: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extend retention-mode coverage to CustomRowIndex and MultipleDeletionVectors count tests.

NoRowIndex (lines 616-638) now loops over both are_retention_vectors values, but CustomRowIndex (lines 686-705) and MultipleDeletionVectors (line 739 onward) still only exercise deletion mode. Consider mirroring the NoRowIndex loop pattern in these two tests for consistent coverage of compute_num_deleted_rows under retention mode with custom row indices and multiple bitmaps.

Also applies to: 641-705, 708-768

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/tests/io/parquet_deletion_vectors_test.cpp` around lines 605 - 638,
Extend the retention-mode loop used in DeletionVectorsCountTests.NoRowIndex to
the CustomRowIndex and MultipleDeletionVectors tests. For each
are_retention_vectors value, build the corresponding expected mask and
deletion-vector metadata, then assert compute_num_deleted_rows for the existing
chunk sizes against the matching expected count while preserving their
custom-index and multiple-bitmap scenarios.

cpp/include/cudf/io/experimental/deletion_vectors.hpp (1)

33-54: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Doc comments not updated for the new retention-vector mode.

The struct-level comment (lines 34-36), the new field's one-line doc, the chunked_parquet_reader class doc (lines 56-68, 131-143), and compute_num_deleted_rows's doc (lines 187-195) all still describe only "deletion" semantics. Since are_retention_vectors inverts the meaning of the roaring bitmap (retain vs. delete) and inverts the returned count in compute_num_deleted_rows, it would help API consumers to spell out the retention-mode behavior in these Doxygen comments.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/include/cudf/io/experimental/deletion_vectors.hpp` around lines 33 - 54,
Update the Doxygen comments for deletion_vector_info, are_retention_vectors,
chunked_parquet_reader, and compute_num_deleted_rows to document both
deletion-vector and retention-vector modes. Explain that are_retention_vectors
makes bitmap entries identify rows to retain rather than delete, and that
compute_num_deleted_rows returns the corresponding inverted deleted-row count in
retention mode.

python/cudf_polars/cudf_polars/streaming/actor_graph/join.py (1)

809-811: 🚀 Performance & Scalability | 🔵 Trivial

TODO: hardcoded 32 MiB filter size.

filter_size is now a fixed constant rather than being derived from expected row counts (fitting_num_blocks), which could over/under-size the filter for skewed build-side cardinalities. The TODO acknowledges this.

Want me to help wire this up to BloomFilter.aligned_size(...) and an actual build-side row estimate once one is available in this actor graph?

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/cudf_polars/cudf_polars/streaming/actor_graph/join.py` around lines
809 - 811, Update the BloomFilter construction in the join actor graph to derive
filter_size from the expected build-side row count, such as fitting_num_blocks,
using BloomFilter.aligned_size(...), instead of the hardcoded 32 MiB constant.
Preserve the existing context, communicator, and hash-seed arguments while
sizing the filter for the estimated cardinality.

cpp/libcudf_streaming/tests/streaming/test_bloom_filter.cu (1)

30-35: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use CUDF_KERNEL instead of raw __global__.

As per coding guidelines, cpp/**/*.{cpp,cu} code should "Use CUDF_KERNEL rather than raw __global__, preferably with __launch_bounds__."

🔧 Proposed fix

-__global__ void block_index_kernel(std::uint32_t upper_hash,
-                                   std::size_t num_blocks,
-                                   std::uint32_t* result)
+CUDF_KERNEL void block_index_kernel(std::uint32_t upper_hash,
+                                    std::size_t num_blocks,
+                                    std::uint32_t* result)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/libcudf_streaming/tests/streaming/test_bloom_filter.cu` around lines 30 -
35, Replace the raw __global__ qualifier on block_index_kernel with the
project-standard CUDF_KERNEL macro, preserving the kernel signature and body.

Source: Coding guidelines

cpp/src/io/parquet/bloom_filter_reader.cu (1)

43-56: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Doc comment still describes the old cuco::parametric_filter_policy.

The comment says "Uses cuco's parametric_filter_policy..." but line 56 now aliases cudf::arrow_filter_policy. Worth updating the comment to reflect the actual policy in use (and why, given the new test validating it supports more blocks than the former Arrow limit).

📝 Proposed doc update

- * Uses cuco's `parametric_filter_policy` with the Apache Arrow layout: 256-bit blocks (8 x
- * `uint32_t`), 8 fingerprint bits per key, fully horizontal add (Theta=8) and fully vertical
- * contains (Phi=8). This layout is bit-compatible with Apache Arrow, as verified by cuCollections
- * `tests/bloom_filter/arrow_compat_test.cu`.
+ * Uses `cudf::arrow_filter_policy` with the Apache Arrow layout: 256-bit blocks (8 x
+ * `uint32_t`), 8 fingerprint bits per key, fully horizontal add (Theta=8) and fully vertical
+ * contains (Phi=8). This layout is bit-compatible with Apache Arrow, as verified by cuCollections
+ * `tests/bloom_filter/arrow_compat_test.cu`.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/src/io/parquet/bloom_filter_reader.cu` around lines 43 - 56, Update the
documentation for the arrow_filter_policy alias to describe
cudf::arrow_filter_policy rather than cuco::parametric_filter_policy, and
mention that the policy preserves the Apache Arrow layout while supporting more
blocks than the former Arrow limit, as covered by the compatibility test.

java/src/test/java/ai/rapids/cudf/VariantUtilsTest.java (1)

410-418: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider adding a Java-level success-path test for array indexing.

The malformed-path fix is correct, but only the error path for "[N]" syntax is exercised here. The underlying C++ layer gained several new success-path tests for array indexing (ApacheArrayPrimitiveIndexing, MixedObjectArrayTraversal, etc. in variant_extract_test.cpp), while the Java binding — which is now able to accept the same "[N]" paths — has no corresponding test confirming the JNI round-trip works (e.g. VariantUtils.extractVariantField(variant, "$[0]", DType.INT32)). This would need extending VariantEncoder with an array(...) builder analogous to object(...).

As per path instructions, "Check that Java API matches the underlying C++ API behavior" for java/**/*.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@java/src/test/java/ai/rapids/cudf/VariantUtilsTest.java` around lines 410 -
418, Extend VariantEncoder with an array(...) builder analogous to object(...),
then add a success-path test near malformedPathThrows in VariantUtilsTest that
constructs an array variant and verifies
VariantUtils.extractVariantField(variant, "$[0]", DType.INT32) returns the
expected value, confirming Java/JNI array-indexing behavior matches C++.

Source: Path instructions

cpp/src/io/parquet/experimental/variant_extract.cu (1)

373-430: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add Doxygen documentation to the new device helpers for consistency.

locate_array_element and parse_index_step use plain // comments, while sibling helpers in the same anonymous namespace with comparable bit-layout complexity (locate_object_field, variant_value_length, find_key_in_metadata, decode_object_array_header) all use full /** @brief ... */ Doxygen blocks documenting layout, parameters, and return semantics. Matching that style would keep the file's documentation generation/linting consistent.

As per coding guidelines, "Use doxygen as a documentation generator and linter for C++ and CUDA code" for **/*.{cpp,cu,h,hpp,cuh}.

Also applies to: 463-487

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/src/io/parquet/experimental/variant_extract.cu` around lines 373 - 430,
Replace the plain comments documenting locate_array_element and parse_index_step
with full Doxygen blocks using the surrounding helper style. Include `@brief` text
plus parameter and return documentation, preserving the existing descriptions of
layout, bounds, truncation, and empty-span semantics without changing
implementation behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@java/src/main/java/ai/rapids/cudf/DeletionVector.java`:
- Around line 64-66: Restore the removed public DeletionVectorInfo constructor
overload, mark it deprecated, and delegate to the current constructor with
isRetention set to false. Keep the existing serializedBitmap, rowGroupOffsets,
and rowGroupNumRows parameters and preserve the current constructor’s behavior.

---

Nitpick comments:
In `@cpp/include/cudf/io/experimental/deletion_vectors.hpp`:
- Around line 33-54: Update the Doxygen comments for deletion_vector_info,
are_retention_vectors, chunked_parquet_reader, and compute_num_deleted_rows to
document both deletion-vector and retention-vector modes. Explain that
are_retention_vectors makes bitmap entries identify rows to retain rather than
delete, and that compute_num_deleted_rows returns the corresponding inverted
deleted-row count in retention mode.

In `@cpp/libcudf_streaming/tests/streaming/test_bloom_filter.cu`:
- Around line 30-35: Replace the raw __global__ qualifier on block_index_kernel
with the project-standard CUDF_KERNEL macro, preserving the kernel signature and
body.

In `@cpp/src/io/parquet/bloom_filter_reader.cu`:
- Around line 43-56: Update the documentation for the arrow_filter_policy alias
to describe cudf::arrow_filter_policy rather than
cuco::parametric_filter_policy, and mention that the policy preserves the Apache
Arrow layout while supporting more blocks than the former Arrow limit, as
covered by the compatibility test.

In `@cpp/src/io/parquet/experimental/variant_extract.cu`:
- Around line 373-430: Replace the plain comments documenting
locate_array_element and parse_index_step with full Doxygen blocks using the
surrounding helper style. Include `@brief` text plus parameter and return
documentation, preserving the existing descriptions of layout, bounds,
truncation, and empty-span semantics without changing implementation behavior.

In `@cpp/tests/io/parquet_deletion_vectors_test.cpp`:
- Around line 605-638: Extend the retention-mode loop used in
DeletionVectorsCountTests.NoRowIndex to the CustomRowIndex and
MultipleDeletionVectors tests. For each are_retention_vectors value, build the
corresponding expected mask and deletion-vector metadata, then assert
compute_num_deleted_rows for the existing chunk sizes against the matching
expected count while preserving their custom-index and multiple-bitmap
scenarios.

In `@java/src/test/java/ai/rapids/cudf/VariantUtilsTest.java`:
- Around line 410-418: Extend VariantEncoder with an array(...) builder
analogous to object(...), then add a success-path test near malformedPathThrows
in VariantUtilsTest that constructs an array variant and verifies
VariantUtils.extractVariantField(variant, "$[0]", DType.INT32) returns the
expected value, confirming Java/JNI array-indexing behavior matches C++.

In `@python/cudf_polars/cudf_polars/streaming/actor_graph/join.py`:
- Around line 809-811: Update the BloomFilter construction in the join actor
graph to derive filter_size from the expected build-side row count, such as
fitting_num_blocks, using BloomFilter.aligned_size(...), instead of the
hardcoded 32 MiB constant. Preserve the existing context, communicator, and
hash-seed arguments while sizing the filter for the estimated cardinality.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d1b02066-6bbf-4da3-aa92-fcc31bc5d516

📥 Commits

Reviewing files that changed from the base of the PR and between dadbc30 and a596c3e.

📒 Files selected for processing (58)

• ci/run_cudf_polars_polars_tests.sh
• ci/run_cudf_polars_pytests.sh
• ci/test_python_other.sh
• ci/test_wheel_cudf_polars.sh
• cpp/CMakeLists.txt
• cpp/include/cudf/io/experimental/deletion_vectors.hpp
• cpp/include/cudf/io/experimental/variant.hpp
• cpp/include/cudf/reduction/bloom_filter.cuh
• cpp/libcudf_streaming/benchmarks/streaming/ndsh/q03.cpp
• cpp/libcudf_streaming/benchmarks/streaming/ndsh/q04.cpp
• cpp/libcudf_streaming/benchmarks/streaming/ndsh/q21.cpp
• cpp/libcudf_streaming/include/cudf_streaming/bloom_filter.hpp
• cpp/libcudf_streaming/include/cudf_streaming/detail/device_bloom_filter.hpp
• cpp/libcudf_streaming/src/bloom_filter.cpp
• cpp/libcudf_streaming/src/detail/device_bloom_filter.cu
• cpp/libcudf_streaming/tests/CMakeLists.txt
• cpp/libcudf_streaming/tests/streaming/test_bloom_filter.cu
• cpp/libcudf_streaming/tests/streaming/test_bloom_filter_config.cpp
• cpp/src/io/orc/writer_impl.cu
• cpp/src/io/orc/writer_impl.hpp
• cpp/src/io/parquet/bloom_filter_reader.cu
• cpp/src/io/parquet/experimental/deletion_vectors.cu
• cpp/src/io/parquet/experimental/variant_extract.cu
• cpp/src/io/parquet/experimental/variant_path.cpp
• cpp/src/io/parquet/experimental/variant_path.hpp
• cpp/src/io/parquet/stats_filter_helpers.hpp
• cpp/tests/io/experimental/variant_extract_test.cpp
• cpp/tests/io/orc_test.cpp
• cpp/tests/io/parquet_bloom_filter_test.cu
• cpp/tests/io/parquet_deletion_vectors_test.cpp
• dependencies.yaml
• java/ci/assemble_maven_repo.sh
• java/ci/build_cudf_java_jar_in_container.sh
• java/src/main/java/ai/rapids/cudf/DeletionVector.java
• java/src/main/native/src/DeletionVectorJni.cpp
• java/src/test/java/ai/rapids/cudf/DeletionVectorTableTest.java
• java/src/test/java/ai/rapids/cudf/VariantUtilsTest.java
• python/cudf/cudf/core/column_accessor.py
• python/cudf/cudf/core/dataframe.py
• python/cudf/cudf/core/indexed_frame.py
• python/cudf/cudf/core/multiindex.py
• python/cudf/cudf/core/reshape.py
• python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
• python/cudf/cudf/tests/dataframe/methods/test_swaplevel.py
• python/cudf/cudf/tests/private_objects/test_column_accessor.py
• python/cudf/cudf/tests/reshape/test_pivot_table.py
• python/cudf/cudf/tests/reshape/test_stack.py
• python/cudf/cudf/tests/reshape/test_unstack.py
• python/cudf_polars/cudf_polars/dsl/traversal.py
• python/cudf_polars/cudf_polars/streaming/actor_graph/join.py
• python/cudf_polars/cudf_polars/streaming/join_filter_pushdown.py
• python/cudf_polars/cudf_polars/utils/config.py
• python/cudf_polars/tests/streaming/test_join_filter_pushdown.py
• python/cudf_polars/tests/test_config.py
• python/cudf_streaming/cudf_streaming/bloom_filter.pxd
• python/cudf_streaming/cudf_streaming/bloom_filter.pyi
• python/cudf_streaming/cudf_streaming/bloom_filter.pyx
• python/cudf_streaming/cudf_streaming/tests/test_bloom_filter.py

💤 Files with no reviewable changes (1)

• ci/test_wheel_cudf_polars.sh

🚧 Files skipped from review as they are similar to previous changes (4)

• ci/run_cudf_polars_pytests.sh
• cpp/CMakeLists.txt
• cpp/src/io/parquet/stats_filter_helpers.hpp
• dependencies.yaml

[Matt711]

/ok to test a596c3e