Add config option for JIT parquet filtering by Matt711 · Pull Request #21460 · NVIDIA/cudf

Matt711 · 2026-02-17T15:29:49Z

Description

Adds an option to cudf-polars to to control whether parquet predicates are JIT compiled.

Contribute to #21456

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

…rs/jit-parquet-filter

TomAugspurger · 2026-02-17T15:54:58Z

    duckdb_err = None
 except ImportError as e:
-    duckdb = None
+    duckdb = None  # type: ignore[assignment]


If you needed these locally, you possibly had duckdb installed in your mypy env, while CI doesn't.

You could maybe change this to type: ignore[assignment,unused-ignore]

…tt711/cudf into fea/polars/jit-parquet-filter

copy-pr-bot · 2026-02-18T03:27:47Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Matt711 · 2026-02-18T03:28:12Z

/ok to test 9c12fec

Matt711 · 2026-02-18T03:28:25Z

/merge

Matt711 · 2026-02-18T16:34:56Z

/ok to test 1790ac6

Matt711 · 2026-02-24T14:07:34Z

/ok to test 1a7cc96

Matt711 · 2026-02-26T03:51:38Z

/ok to test 6bc7595

vyasr · 2026-03-04T02:41:21Z

@Matt711 if you can resolve conflicts this PR is probably ready to merge.

vyasr · 2026-03-10T15:46:46Z

/ok to test d6aed0a

vyasr · 2026-03-10T15:50:18Z

/merge

vyasr · 2026-03-16T22:24:25Z

/ok to test 8e5279e

…uet-filter # Conflicts: # python/cudf_polars/cudf_polars/utils/config.py # python/cudf_polars/tests/test_parquet_filters.py

vyasr · 2026-06-16T22:30:52Z

/ok to test 08e36db

coderabbitai · 2026-06-16T22:31:19Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a use_jit_filter boolean field to ParquetOptions, wired to the CUDF_POLARS__PARQUET_OPTIONS__USE_JIT_FILTER env var with a False default. The non-streaming and streaming Parquet read paths now conditionally enable JIT filtering when predicates are present and the option is set. CSV file opening uses explicit UTF-8 encoding.

Changes

JIT Filter Configuration and Wiring

Layer / File(s)	Summary
ParquetOptions.use_jit_filter field and validation `python/cudf_polars/cudf_polars/utils/config.py`	Adds the `use_jit_filter` boolean field to `ParquetOptions` with env-var binding (`CUDF_POLARS__PARQUET_OPTIONS__USE_JIT_FILTER`), a docstring entry, and a `__post_init__` type check that raises `TypeError` for non-boolean values.
JIT filter wiring in non-streaming and streaming read paths `python/cudf_polars/cudf_polars/dsl/ir.py`, `python/cudf_polars/cudf_polars/streaming/actor_graph/io.py`	Refactors `ParquetReaderOptions` construction in both Parquet read paths to use a builder that conditionally calls `use_jit_filter(True)` when a predicate is present and the option is enabled. CSV scan file opening is updated to specify explicit UTF-8 encoding. The streaming function gains a new `parquet_options: ParquetOptions` parameter passed from `config_options` at the call site.
Config and filter tests for `use_jit_filter` `python/cudf_polars/tests/test_config.py`, `python/cudf_polars/tests/test_parquet_filters.py`	Extends `test_config.py` to assert default `False`, env-var activation, and `from_polars_engine` propagation. Adds `test_jit_filter` in `test_parquet_filters.py` that runs a filtered GPU Parquet scan with `use_jit_filter=True`.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

rapidsai/cudf#22758: Both PRs modify python/cudf_polars/cudf_polars/streaming/actor_graph/io.py around the RapidsMPF/parquet read fast path and StreamingScan plumbing.

Suggested labels

improvement

Suggested reviewers

TomAugspurger

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding a JIT parquet filtering config option.
Description check	✅ Passed	The description is directly related to the changeset and accurately states that parquet predicates can be JIT compiled.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

Use executor='in-memory' to avoid triggering DefaultSingletonEngine, which wasn't being shut down before subsequent spmd-small tests could run on the same pytest-xdist worker.

vyasr · 2026-06-17T00:24:38Z

/ok to test 6a3a2ab

Explicitly pass encoding='utf-8' to path.open() in the CSV blank-line skip loop. Without this, the default encoding on systems with LANG=C (e.g. CI containers) is ASCII, which fails when the CSV file contains non-ASCII data (e.g. UTF-8 encoded strings).

vyasr · 2026-06-17T20:43:59Z

/ok to test 0827b5e

vyasr · 2026-06-23T17:38:06Z

/ok to test c32dd63

vyasr · 2026-06-23T21:50:48Z

/ok to test 31e672d

Matt711 · 2026-06-25T18:51:42Z

/ok to test 518c903

Matt711 · 2026-06-25T20:10:41Z

/ok to test 49cc035

Matt711 · 2026-07-06T11:03:34Z

/ok to test 6446f9a

Add config option for JIT parquet filtering

69b0a9a

Matt711 requested a review from a team as a code owner February 17, 2026 15:29

Matt711 requested review from bdice and rjzamora February 17, 2026 15:29

Matt711 added feature request New feature or request non-breaking Non-breaking change labels Feb 17, 2026

github-actions Bot added Python Affects Python cuDF API. cudf-polars Issues specific to cudf-polars labels Feb 17, 2026

github-actions Bot assigned Matt711 Feb 17, 2026

github-project-automation Bot added this to cuDF Python Feb 17, 2026

Merge branch 'main' into fea/polars/jit-parquet-filter

a17c5f7

GPUtester moved this to In Progress in cuDF Python Feb 17, 2026

Merge branch 'main' of https://github.com/rapidsai/cudf into fea/pola…

f4a6e68

…rs/jit-parquet-filter

TomAugspurger approved these changes Feb 17, 2026

View reviewed changes

bdice approved these changes Feb 17, 2026

View reviewed changes

Matt711 and others added 3 commits February 17, 2026 22:25

Merge branch 'main' into fea/polars/jit-parquet-filter

e8281cf

Merge branch 'fea/polars/jit-parquet-filter' of https://github.com/Ma…

4503f1a

…tt711/cudf into fea/polars/jit-parquet-filter

check style

9c12fec

Matt711 and others added 2 commits February 18, 2026 08:17

merge conflict

3c6800a

Merge branch 'main' into fea/polars/jit-parquet-filter

1790ac6

Merge branch 'main' into fea/polars/jit-parquet-filter

1a7cc96

Merge branch 'main' into fea/polars/jit-parquet-filter

6bc7595

merge conflict

cd4c275

Merge branch 'main' into fea/polars/jit-parquet-filter

d6aed0a

github-actions Bot assigned vyasr Mar 10, 2026

Merge branch 'main' into fea/polars/jit-parquet-filter

8e5279e

Merge remote-tracking branch 'upstream/main' into fea/polars/jit-parq…

08e36db

…uet-filter # Conflicts: # python/cudf_polars/cudf_polars/utils/config.py # python/cudf_polars/tests/test_parquet_filters.py

vyasr requested a review from a team as a code owner June 16, 2026 22:27

Fix test_jit_filter leaving DefaultSingletonEngine alive

6a3a2ab

Use executor='in-memory' to avoid triggering DefaultSingletonEngine, which wasn't being shut down before subsequent spmd-small tests could run on the same pytest-xdist worker.

Matt711 and others added 2 commits June 23, 2026 15:39

merge conflict

f8c2c00

Merge branch 'main' into fea/polars/jit-parquet-filter

c32dd63

Merge branch 'main' into fea/polars/jit-parquet-filter

31e672d

Merge branch 'main' into fea/polars/jit-parquet-filter

518c903

Matt711 commented Jun 25, 2026

View reviewed changes

Comment thread python/cudf_polars/cudf_polars/streaming/actor_graph/io.py Outdated

wrong API

49cc035

Matt711 and others added 2 commits July 6, 2026 11:02

utf8-encode

ed849b1

Merge branch 'main' into fea/polars/jit-parquet-filter

6446f9a

rapids-bot Bot merged commit 2c3f616 into NVIDIA:main Jul 6, 2026
108 checks passed

github-project-automation Bot moved this from In Progress to Done in cuDF Python Jul 6, 2026

coderabbitai Bot mentioned this pull request Jul 7, 2026

Improve predicate pushdown coverage in cudf-polars #23143

Open

3 tasks

Uh oh!

Conversation

Matt711 commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

TomAugspurger Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

copy-pr-bot Bot commented Feb 18, 2026

Uh oh!

Matt711 commented Feb 18, 2026

Uh oh!

Matt711 commented Feb 18, 2026

Uh oh!

Matt711 commented Feb 18, 2026

Uh oh!

Matt711 commented Feb 24, 2026

Uh oh!

Matt711 commented Feb 26, 2026

Uh oh!

vyasr commented Mar 4, 2026

Uh oh!

vyasr commented Mar 10, 2026

Uh oh!

vyasr commented Mar 10, 2026

Uh oh!

vyasr commented Mar 16, 2026

Uh oh!

vyasr commented Jun 16, 2026

Uh oh!

coderabbitai Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

vyasr commented Jun 17, 2026

Uh oh!

vyasr commented Jun 17, 2026

Uh oh!

vyasr commented Jun 23, 2026

Uh oh!

vyasr commented Jun 23, 2026

Uh oh!

Matt711 commented Jun 25, 2026

Uh oh!

Uh oh!

Matt711 commented Jun 25, 2026

Uh oh!

Matt711 commented Jul 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Matt711 commented Feb 17, 2026 •

edited

Loading

coderabbitai Bot commented Jun 16, 2026 •

edited

Loading