Repository navigation
Conversation
…oups. Separate GETs for each filter dominate cold reads from object storage. `input_format_parquet_bloom_filter_coalesce_max_bytes` merges contiguous filters, and reads a small file once when that is cheaper. Co-authored-by: Cursor <cursoragent@cursor.com>
… buffer. The bloom-filter header that starts the read is reset as soon as it is parsed, so charging that handle let the scheduler treat a live buffer as free. Related: #2526 Co-authored-by: Cursor <cursoragent@cursor.com>
That field is optional. Filters that only store an offset were left out of coalescing, even though their length is already derived later. Related: #2526 Co-authored-by: Cursor <cursoragent@cursor.com>
…or a whole-file read. Co-authored-by: Cursor <cursoragent@cursor.com>
…compatibility` values restore separate reads. Co-authored-by: Cursor <cursoragent@cursor.com>
…is below `bytes_per_read_task`. Co-authored-by: Cursor <cursoragent@cursor.com>
…and the filters after it stay inside `input_format_parquet_bloom_filter_coalesce_max_bytes`. Co-authored-by: Cursor <cursoragent@cursor.com>
….6 so older `compatibility` values restore separate reads, and count only the bloom ranges `Reader` registers. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This branch has not been deployed
1 participant
Separate GETs for each filter dominate cold reads from object storage.
input_format_parquet_bloom_filter_coalesce_max_bytesmerges contiguous filters, and reads a small file once when that is cheaper.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Read Parquet bloom filters in one request when a file has many row groups
Documentation entry for user-facing changes
ClickHouse can build Bloom-filter for Iceberg tables for queries with 'WHERE' condition not on key columns. For this ClickHouse reads metainfo for each row group from source files. In initial implementation it use single read operation for each row group, which is converted to separate GET request when data files are on object storage like S3. For large table it can produce thousands of requests. In customer test first request takes near 500 seconds, while second (when data are in local cache) less than 4 seconds.
This PR introduced reading metainfo for multiple row groups in single request, when possible. It can increase total traffic, but decrease number of requests, which can reduce time for query execution.
CI/CD Options
Exclude tests:
Regression jobs to run: