Repository navigation
Conversation
# Which issue does this PR close? - Closes apache#10967. # Rationale for this change The mask-backed `RowSelection::and_then` implementation has high per-selected-row overhead for large, dense masks: it walks the outer set bits one at a time and appends each surviving row individually. Expanding the inner mask one 64-bit word at a time is substantially faster for these inputs. # What changes are included in this PR? - Add a word-at-a-time dense mask expansion path. Each output word deposits the next `popcount` bits of the inner mask into the set positions of the outer word (a software `pdep`), iterating over whichever side of the word has fewer bits. - Keep the existing set-index path for short or sparse masks. - Select the dense path for masks with at least 8,192 rows, at least ~75% outer selectivity, and at least ~5% inner selectivity. - Add unit tests for the new path. The dense path costs a fixed amount per output word regardless of how many inner rows survive. The set-index path only walks the outer set bits up to the last inner survivor, so it stays faster when the outer mask is sparse, or when only a few inner rows survive and they are clustered near the start of the selection. The thresholds keep the set-index path for those inputs. The benchmark suite is submitted separately in apache#10969. # Are these changes tested? Yes. Tests cover bitmap offsets, threshold boundaries, and randomized 64-bit deposit inputs across the full mask density range, including every mask with at most one set or unset bit. ```console cargo test -p parquet --lib arrow::arrow_reader::selection::algebra::tests test result: ok. 23 passed; 0 failed ``` Using the benchmark suite in apache#10969, all 38 cases that select the dense path improved, with mean execution time reduced by 41.7% to 97.6% on an Apple M4 with Rust 1.97.1. The remaining 26 cases take the unchanged set-index path and show no measurable difference. | Distribution | Rows | Outer | Inner | `main` | PR | Change | |---|---:|---:|---:|---:|---:|---:| | Pseudo-random | 8,192 | 75% | 5% | 3,691 ns | 1,456 ns | -60.7% | | Clustered | 8,192 | 75% | 5% | 882 ns | 513 ns | -41.7% | | Pseudo-random | 8,192 | 99% | 99% | 18,255 ns | 553 ns | -96.9% | | Clustered | 65,536 | 99% | 99% | 142,840 ns | 3,389 ns | -97.6% | # Are there any user-facing changes? No. This is an internal performance optimization with no API or behavior changes. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
# Which issue does this PR close? - Follow on to apache#10136 # Rationale for this change Applies the review suggestion in apache#10136 (comment) from @mbutrovich to add a doc example for the new public `BitChunks::chunk` API. # What changes are included in this PR? Adds a doctest to `BitChunks::chunk`. # Are these changes tested? Yes, by the new doctest. # Are there any user-facing changes? Docs only. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…the correct number of values (apache#11222) # Which issue does this PR close? - Closes apache#11221. # Rationale for this change In certain instances `ByteArrayDecoderPlain::read` can return prematurely with no indication to the consumer that an incomplete read has been done. # What changes are included in this PR? Modifies `ByteArrayDecoderPlain::read` to return the actual number of values read. It's up to the caller to determine if fewer values than requested is expected. # Are these changes tested? Yes, this adds a test that fails on main # Are there any user-facing changes? No public APIs change, but this changes the behavior of `ByteArrayDecoderPlain::read`, which may not be desirable. We might instead choose to return an error in this case, as happens elsewhere in the same function. # AI Disclaimer This issue was identified and the unit test generated by Codex. I'm responsible for the fix.
…apache#10901) ## Which issue does this PR close? - Part of apache#10774. ## Rationale for this change `cleanup_consumed_batches` rescans batch ids from 0 on every `consume_batch` call, re-removing ids that earlier calls already removed, and takes the shared cache's **write lock** each time even when there is nothing left to remove: ```rust for batch_id_to_remove in 0..(current_batch_id.val - 1) { cache.remove(...); } ``` Over a row group with N cached batches this is O(N²) `remove` calls; the consumer-role reader runs it once per output batch per cached column. ## What changes are included in this PR? Track the already-cleaned frontier (`cleaned_up_to`) and remove only the new range. When the frontier has not advanced, return without touching the lock. ## Are these changes tested? Covered by the existing `cached_array_reader` unit tests and the parquet `--lib` suite (1331 passed; the one failure, `test_int96_interop`, is a missing `parquet-testing` file unrelated to this change). ## Performance Two `arrow_reader_clickbench` bot runs below: **neutral**. With `batch_size = 8192`, N is small enough (~15 per row group on this dataset) that the quadratic costs microseconds — so this is a hygiene fix, not a measurable win at this scale. The pattern grows quadratically with row-group size, and dropping the per-call write-lock acquisition also matters more under concurrent readers than in this single-stream bench. (Per-run flags that did not reproduce: run 1 showed async Q20/Q21 swings that vanished in run 2; async/Q22 shows +11% in both runs, but the same query on the `sync` and `async_object_store` variants — same reader code — is neutral-to-faster in both runs, and the async baseline's ±16ms variance points at that variant's flakiness rather than this change.) ## Are there any user-facing changes? No. `cleanup_consumed_batches` now takes `&mut self`, but it is a private method of a `pub(crate)` type.
…tion in concat (apache#10377) # Which issue does this PR close? - Part of apache#9298. # Rationale for this change `concat_run_arrays` built its result via `RunArray::try_new`, which re-validates the run ends on every call even though they're already known-valid here: inputs are valid `RunArray`s, so the adjusted run ends are strictly increasing by construction. # What changes are included in this PR? - `concat.rs`: `concat_run_arrays` now builds the result directly via `RunArray::new_unchecked(data_type, RunEndBuffer::new_unchecked(...), values)`, skipping `try_new`'s revalidation. - `concat.rs`: adds an explicit overflow guard and rejects up front if the total doesn't fit in the run-end type `R`, before any `R::Native` arithmetic runs. - `nullif.rs`/`take.rs` changes are test-only cleanup. # Are these changes tested? Yes — existing test suite, plus a new regression test for the overflow guard. # Benchmark results ``` cargo bench -p arrow --bench concatenate_kernel --features test_utils -- "concat run" ``` this branch vs `main` (`RunArray<Int32Type>`, `n` arrays concatenated): | Case | main | this PR | Change | |---|---|---|---| | logical=32, physical=4, n=4 | 773.6 ns | 563.4 ns | **−27%** | | logical=1024, physical=128, n=8 | 3.33 µs | 2.68 µs | **−19%** | | logical=8192, physical=1024, n=10 | 25.2 µs | 20.4 µs | **−19%** | # Are there any user-facing changes? No.
… bounds checks (apache#10945) apache#10945 <- here, this PR is stacked on top of apache#10944 apache#10944 # Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#8879. - the only unique commit compared to apache#10944 is 47e2de0 # Rationale for this change `take_record_batch` previously called take per column, which always runs a bounds check on every index even when the caller already knows the indices are valid. For workloads doing repeated record batch takes with pre-validated indices — such as sort, merge, or filter pipelines — this check is redundant and measurable overhead. Benchmarks show ~13% speedup on primitive-column batches at 1024 rows when the check is skipped. <img width="643" height="161" alt="Image 9-1-26 at 2 28 PM" src="https://github.com/user-attachments/assets/5170db51-ff1c-4062-b4fe-c69de894ddbc" /> <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? - Introduces pub unsafe fn take_record_batch_unchecked which bypasses index bounds checking by dispatching through `take_impl::<_, false>` directly <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? yes, existing test + miri <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? If this PR claims a performance improvement, please include evidence such as benchmark results. --> # Are there any user-facing changes? new `take_record_batch_unchecked()` method. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
# Which issue does this PR close? Follow-up to apache#10357; corrects the benchmark used by apache#10358. # Rationale for this change The unshredded object fixture has a literal field named `attr.140`, but `VariantPath::try_from("attr.140")` parses it as two nested fields (`attr` then `140`). Since the fixture has no `attr` field, the benchmark returns NULL for every row instead of extracting the intended value. # What changes are included in this PR? Construct the path with `VariantPathElement::field("attr.140")` and assert that every result is `Some(140)` before entering the timed loop. # Are these changes tested? - `cargo bench --offline -p parquet-variant-compute --bench variant_kernels -- variant_get_unshredded_object_path_262k_rows --test` - `cargo fmt --all -- --check` - `git diff --check` # Are there any user-facing changes? No. This only corrects the benchmark workload; previous timings for this benchmark measured missing-path handling.
…ble to Variant (apache#11053) ## Rationale Closes apache#8628. Previously `VariantArray` could be collected only from `Option<Variant>` items: ```rust let arr: VariantArray = values.iter().map(|v| Some(Variant::from(*v))).collect(); ``` With this change, any type that converts cleanly into `Variant` can be collected directly: ```rust let arr: VariantArray = vec![Some(1_i64), None, Some(42_i64)].into_iter().collect(); let arr: VariantArray = vec![Some("hello"), Some("a very long string exceeding the short string limit")].into_iter().collect(); ``` This follows the suggestion made in apache#8625 (comment). ## Changes - `parquet-variant-compute/src/variant_array.rs`: generalize `FromIterator<Option<Variant<'m, 'v>>>` to `impl<'m, 'v, V: Into<Variant<'m, 'v>>> FromIterator<Option<V>>`, converting each item through `Into::into`. Existing code that collects `Option<Variant>` items keeps working (identity `Into`). - Added regression tests: `i64` items with nulls, `&str` items covering both short-string (ShortString) and long-string (String) storage. ## Verification - `cargo test -p parquet-variant-compute --lib` — 366 tests passed (364 existing + 2 new), including the new `test_from_option_into_variants_into_variant_array` and `test_from_option_str_into_variant_array`. - `cargo clippy -p parquet-variant-compute --all-targets -- -D warnings` — clean. - `cargo fmt --all -- --check` — clean. Note: local verification used the `x86_64-pc-windows-gnu` target; upstream CI covers the full matrix. ## AI disclosure Per the Arrow project's AI-generated submissions guidance: this change was implemented with the assistance of an AI coding agent. The author reviewed the final diff, verified it compiles and passes tests/clippy/fmt locally, and takes full ownership of the change. The ASF ICLA may need to be on file before merge (asf-clabot will report the status).
…ing a batch (apache#11218) # Which issue does this PR close? - Related to apache#6946 # Rationale for this change While discussing apache#6946 with @adriangb, a question came up about whether `ParquetPushDecoder` would decode a batch if only the pages needed for that batch had been pushed or if it insists on having all pages for the row group first. This PR adds a test that answers that question For the current code, the ParquetPushDecoder requires that all ranges for the row group are present # What changes are included in this PR? `test_decoder_first_page_only` test # Are these changes tested? This PR is only a test. # Are there any user-facing changes? No. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
apache#11264) # Which issue does this PR close? - Closes apache#11263. # Rationale for this change Caller-supplied distinct counts describe an individual write, while column metadata describes the entire column chunk. Once earlier data makes the column-wide count unknown, a later write cannot safely restore that count using only its own batch statistics. # What changes are included in this PR? - Use column and page metrics to detect prior data instead of the page-local encoder value count. - Accept a caller-supplied distinct count only when there is no prior data in the column chunk. - Update the regression test to force a page boundary and verify that the column-wide distinct count remains absent while page statistics remain correct. # Are these changes tested? Yes. The updated regression covers distinct-count invalidation across a page flush and checks that page-level statistics are preserved. # Are there any user-facing changes? There are no public API changes. Signed-off-by: Hippolyte Barraud <hippolyte.barraud@datadoghq.com>
…pache#11269) # Which issue does this PR close? NA # Rationale for this change The row-group byte-limit calculation already runs under a guard that ensures the buffered row count is nonzero. Using checked division for that first calculation repeats the guard, while a separate zero-average filter and fallback value make the remaining-budget calculation more complicated than necessary. Simplify the arithmetic while preserving the existing split and flush decisions. # What changes are included in this PR? - Calculate average row bytes directly under the existing nonzero buffered-row guard. - Divide the remaining byte budget by that average using `checked_div`. - Remove the separate zero-average filter and redundant fallback value. # Are these changes tested? All tests passing. # Are there any user-facing changes? No. Signed-off-by: Hippolyte Barraud <hippolyte.barraud@datadoghq.com>
# Which issue does this PR close? - Contributes to apache#9731 . # Rationale for this change We do not track memory usage for the Parquet writer. Native REE support should improve memory usage materially so we need some bench harnesses to keep track of that over time. Add a dedicated memory benchmark with two complementary measurements: incremental peak live memory and cumulative allocated bytes over a complete writer lifecycle. # What changes are included in this PR? - Add the `arrow_writer_peak_memory` Criterion benchmark with peak-live-memory and cumulative-allocation measurements. - Measure writer construction, `write()`, and `close()`, including writer destruction. - Exclude input construction and configuration cloning from the measured window. - Exercise primitive, sparse, dictionary, run-end-encoded, and nested inputs under the default, Parquet 2.0, and content-defined-chunking configurations. - Write to a discarding sink so output retention outside the writer does not dominate the measurements. - Keep allocation tracking local to the benchmark through a thread-local allocator wrapper. The measurements target the synchronous, single-threaded writer; they are not whole-process resident-memory measurements. How memory usage is tracked is the same as the existing mechanism added in apache#9846. # Are these changes tested? NA # Are there any user-facing changes? NA Signed-off-by: Hippolyte Barraud <hippolyte.barraud@datadoghq.com>
) # Which issue does this PR close? - Closes apache#11265. # Rationale for this change The writer accepts some encoding and physical-type combinations that no encoder can represent correctly. Invalid configurations can therefore fail only when values are written, when a dictionary falls back, or when a reader encounters the output. # What changes are included in this PR? - Define the supported value-encoding matrix explicitly. - Validate encoding and physical-type combinations at encoder construction, including the Arrow byte-array fallback path. - Reject invalid dictionary fallback configurations even when the fallback would never be used. - Add a construction-time validation matrix covering physical types, writer versions, and dictionary settings, and update existing error expectations. Supported encodings and their wire representations are unchanged. # Are these changes tested? Yes. The tests cover the supported encoding matrix at column construction, with dictionary encoding both enabled and disabled and with both Parquet writer versions. The Arrow writer regression now expects the invalid DOUBLE/DELTA_BINARY_PACKED configuration to fail earlier. # Are there any user-facing changes? Yes. Unsupported encoding configurations are rejected when the physical column encoder is constructed, before values reach that encoder. Invalid dictionary fallbacks are rejected even if the column would remain dictionary-encoded. This deliberately changes failure timing and error messages for invalid configurations. It does not change supported encodings, valid output, or public API signatures. Signed-off-by: Hippolyte Barraud <hippolyte.barraud@datadoghq.com>
# Which issue does this PR close? - N/A # Rationale for this change We previously went through `write!` when displaying Boolean values, which goes through the general-purpose formatting machinery. It is a one-liner to special-case Boolean values instead. This speeds up CSV output and pretty printing. Benchmark: (M4 Max) * `format_boolean/random`: 96.0 µs -> 77.6 µs, −19.8% # What changes are included in this PR? * Optimize display for Boolean values * Add benchmark # Are these changes tested? Yes, covered by existing tests. # Are there any user-facing changes? No. # AI usage Developed with Claude Code (Opus 5.5). I have reviewed and understand the resulting code.
…argo-deps group across 1 directory (apache#11250) Bumps the all-other-cargo-deps group with 1 update in the / directory: [hyper-util](https://github.com/hyperium/hyper-util). Updates `hyper-util` from 0.1.20 to 0.1.21 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/hyperium/hyper-util/releases">hyper-util's releases</a>.</em></p> <blockquote> <h2>v0.1.21</h2> <h2>Additions</h2> <ul> <li>Add crate-level documentation. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/327">#327</a>)</li> <li>Add <code>client::legacy::Builder::http2_header_table_size()</code> method. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/274">#274</a>)</li> <li>Add <code>client::legacy::Builder::http2_max_concurrent_streams()</code> method. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/274">#274</a>)</li> <li>Add <code>client::legacy::Builder::http2_max_local_error_reset_streams()</code> method. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/277">#277</a>)</li> <li>Add <code>client::legacy::connect::HttpConnector::set_mark()</code> method. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/303">#303</a>)</li> <li>Add <code>rt::tracing::WithSpanExecutor<E></code>, <code>hyper_util::rt::tracing::CurrentSpanExecutor<E></code>, and <code>hyper_util::rt::tracing::MkSpanExecutor<E, F></code> executors. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/323">#323</a>)</li> </ul> <h2>Fixes</h2> <ul> <li>Fix <code>client::legacy::Client</code> so that it properly validates CONNECT responses. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/315">#315</a>)</li> <li>Fix <code>client::legacy::Client</code> to cancel the idle interval once its pool empties. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/292">#292</a>)</li> <li>Fix <code>client::legacy::Client</code> to properly handle IPv6 addresses when using a SOCKS proxy. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/302">#302</a>)</li> <li>Fix <code>client::pool::cache</code> to preserve readiness with clones. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/297">#297</a>)</li> <li>Fix <code>client::pool::cache</code> to wake its waiters in FIFO order. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/298">#298</a>)</li> <li>Fix <code>client::pool::singleton::Singleton</code> to properly handle cancellation. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/299">#299</a>)</li> <li>Fix <code>client::pool::singleton::Singleton</code> to share errors with all waiters. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/296">#296</a>)</li> <li>Fix <code>client::proxy::matcher</code> handling for IP wildcards. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/309">#309</a>)</li> <li>The <code>tokio/net</code> feature is narrowed to the <code>client-legacy</code> feature flag, from the <code>client</code> feature flag. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/276">#276</a>)</li> <li>Various fixes to the <code>client::legacy::Client</code>'s SOCKS proxying. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/302">#302</a>) (<a href="https://redirect.github.com/hyperium/hyper-util/pull/307">#307</a>) (<a href="https://redirect.github.com/hyperium/hyper-util/pull/308">#308</a>) (<a href="https://redirect.github.com/hyperium/hyper-util/pull/310">#310</a>)</li> </ul> <h2>Changes</h2> <p>This release contains a minor behavioral change for users of the <code>tracing</code> feature flag to be aware of.</p> <p>This feature flag was introduced in v0.1.11. When enabled, <code>rt::TokioExecutor<E></code> began propagating the currently active <code>tracing::Span</code> to spawned tasks when <code>hyper::rt::Executor::execute()</code> is called. This caused issues for some users, due to background tasks keeping a span open for the duration of a long-lived connection.</p> <p>This behavior has now been removed from <code>rt::TokioExecutor<E></code> (<a href="https://redirect.github.com/hyperium/hyper-util/pull/322">#322</a>) by default. A collection of executor wrappers have been added to a new <code>rt::tracing</code> submodule, to provide facilities for instrumenting a client or server's spawned tasks. See the module-level documentation of <code>rt::tracing</code> for more information.</p> <p>To temporarily preserve the previous <code>rt::TokioExecutor<E></code> span propagation behavior, enable the <code>rt-tracing-exec-force</code> feature. Note that this feature flag will be removed in a future release.</p> <p>This release bumps the minimal supported Rust version (MSRV) from 1.64 to 1.85.</p> <p>This release bumps the rust edition from 2021 to 2024.</p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/hyperium/hyper-util/blob/master/CHANGELOG.md">hyper-util's changelog</a>.</em></p> <blockquote> <h1>0.1.21 (2026-09-24)</h1> <p>This release bumps the minimal supported Rust version (MSRV) from 1.64 to 1.85.</p> <p>This release bumps the rust edition from 2021 to 2024.</p> <h2>Additions</h2> <ul> <li>Add crate-level documentation. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/327">#327</a>)</li> <li>Add <code>client::legacy::Builder::http2_header_table_size()</code> method. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/274">#274</a>)</li> <li>Add <code>client::legacy::Builder::http2_max_concurrent_streams()</code> method. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/274">#274</a>)</li> <li>Add <code>client::legacy::Builder::http2_max_local_error_reset_streams()</code> method. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/277">#277</a>)</li> <li>Add <code>client::legacy::connect::HttpConnector::set_mark()</code> method. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/303">#303</a>)</li> <li>Add <code>rt::tracing::WithSpanExecutor<E></code>, <code>hyper_util::rt::tracing::CurrentSpanExecutor<E></code>, and <code>hyper_util::rt::tracing::MkSpanExecutor<E, F></code> executors. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/323">#323</a>)</li> </ul> <h2>Fixes</h2> <ul> <li>Fix <code>client::legacy::Client</code> so that it properly validates CONNECT responses. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/315">#315</a>)</li> <li>Fix <code>client::legacy::Client</code> to cancel the idle interval once its pool empties. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/292">#292</a>)</li> <li>Fix <code>client::legacy::Client</code> to properly handle IPv6 addresses when using a SOCKS proxy. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/302">#302</a>)</li> <li>Fix <code>client::pool::cache</code> to preserve readiness with clones. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/297">#297</a>)</li> <li>Fix <code>client::pool::cache</code> to wake its waiters in FIFO order. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/298">#298</a>)</li> <li>Fix <code>client::pool::singleton::Singleton</code> to properly handle cancellation. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/299">#299</a>)</li> <li>Fix <code>client::pool::singleton::Singleton</code> to share errors with all waiters. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/296">#296</a>)</li> <li>Fix <code>client::proxy::matcher</code> handling for IP wildcards. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/309">#309</a>)</li> <li>The <code>tokio/net</code> feature is narrowed to the <code>client-legacy</code> feature flag, from the <code>client</code> feature flag. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/276">#276</a>)</li> <li>Various fixes to the <code>client::legacy::Client</code>'s SOCKS proxying. (<a href="https://redirect.github.com/hyperium/hyper-util/pull/302">#302</a>) (<a href="https://redirect.github.com/hyperium/hyper-util/pull/307">#307</a>) (<a href="https://redirect.github.com/hyperium/hyper-util/pull/308">#308</a>) (<a href="https://redirect.github.com/hyperium/hyper-util/pull/310">#310</a>)</li> </ul> <h2>Changes</h2> <p>This release contains a minor behavioral change for users of the <code>tracing</code> feature flag to be aware of.</p> <p>This feature flag was introduced in v0.1.11. When enabled, <code>rt::TokioExecutor<E></code> began propagating the currently active <code>tracing::Span</code> to spawned tasks when <code>hyper::rt::Executor::execute()</code> is called. This caused issues for some users, due to background tasks keeping a span open for the duration of a long-lived connection.</p> <p>This behavior has now been removed from <code>rt::TokioExecutor<E></code> (<a href="https://redirect.github.com/hyperium/hyper-util/pull/322">#322</a>) by default. A collection of executor wrappers have been added to a new <code>rt::tracing</code> submodule, to provide facilities for instrumenting a client or server's spawned tasks. See the module-level documentation of <code>rt::tracing</code> for more information.</p> <p>To temporarily preserve the previous <code>rt::TokioExecutor<E></code> span propagation behavior, enable the <code>rt-tracing-exec-force</code> feature. Note that this feature flag will be removed in a future release.</p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/hyperium/hyper-util/commit/23a868965964c1d6bb1b94f30ba4c420a4bebe7c"><code>23a8689</code></a> v0.1.21</li> <li><a href="https://github.com/hyperium/hyper-util/commit/cdb2346fed0af8283d5358d1d3fd5f5064db41d7"><code>cdb2346</code></a> doc(client/pool): add <code>broken_intra_doc_links</code> allowance (<a href="https://redirect.github.com/hyperium/hyper-util/issues/326">#326</a>)</li> <li><a href="https://github.com/hyperium/hyper-util/commit/13b61105d417524c2c5b4738d6907a7534dda858"><code>13b6110</code></a> chore(error): remove disabled <code>hyper_util::error</code> submodule (<a href="https://redirect.github.com/hyperium/hyper-util/issues/325">#325</a>)</li> <li><a href="https://github.com/hyperium/hyper-util/commit/b9ee45111645900e0557aecd9f19cfb5fd01d77f"><code>b9ee451</code></a> docs(lib): add crate-level documentation (<a href="https://redirect.github.com/hyperium/hyper-util/issues/327">#327</a>)</li> <li><a href="https://github.com/hyperium/hyper-util/commit/d6b7d7e6a0fc568a1485a8199254305be50dde26"><code>d6b7d7e</code></a> feat(rt/tracing): introduce tracing executors (<a href="https://redirect.github.com/hyperium/hyper-util/issues/323">#323</a>)</li> <li><a href="https://github.com/hyperium/hyper-util/commit/f2da9150f081a6a74e40bd82b0b51f91216fa23b"><code>f2da915</code></a> feat(client): add HttpConnector::set_mark for SO_MARK (<a href="https://redirect.github.com/hyperium/hyper-util/issues/303">#303</a>)</li> <li><a href="https://github.com/hyperium/hyper-util/commit/4dbd4945f566e2b2a40d247c4281164258bbe1a5"><code>4dbd494</code></a> feat(rt): change <code>TokioExecutor</code> to not trace, add rt-tracing-exec-force feat...</li> <li><a href="https://github.com/hyperium/hyper-util/commit/d480d9f802c7062cb0aeece9ee0020ecce840521"><code>d480d9f</code></a> fix(client): parse proxy CONNECT response with httparse (<a href="https://redirect.github.com/hyperium/hyper-util/issues/315">#315</a>)</li> <li><a href="https://github.com/hyperium/hyper-util/commit/7e0958bb537fb8b7901bae60d43a322c8f9a8a63"><code>7e0958b</code></a> chore(ci): simplify msrv check (<a href="https://redirect.github.com/hyperium/hyper-util/issues/317">#317</a>)</li> <li><a href="https://github.com/hyperium/hyper-util/commit/07345689c815e115e616ce78a6a02c5df6c7081f"><code>0734568</code></a> chore(ci): update to actions/checkout@v7 (<a href="https://redirect.github.com/hyperium/hyper-util/issues/316">#316</a>)</li> <li>Additional commits viewable in <a href="https://github.com/hyperium/hyper-util/compare/v0.1.20...v0.1.21">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
### Which issue does this PR close? - Closes apache#11197. ### Rationale for this change new_null took the first union field without checking there was one, so an empty union panicked on the unwrap. len 0 is the only length an empty union can represent. every slot carries a type id naming one of the children, and UnionArray::try_new rejects a type id that names none. ### What changes are included in this PR? an empty union at len 0 builds the empty buffers, type ids plus offsets when dense, and no children. any other length panics saying what is missing, the same shape as FixedSizeBinary with a negative width. new_null returns Self, so the alternative is data that fails later in validate_full. ### Are these changes tested? yes. test_new_null_empty_union covers both modes at len 0 and runs validate_full, test_new_null_empty_union_with_slots pins the panic. arrow-data is 63 + 13 green, arrow-array 725, clippy clean. ### Are there any user-facing changes? new_null on an empty union of length 0 returns an array instead of panicking. no api changes.
# Which issue does this PR close? - N/A # Rationale for this change Casting an array of numeric to boolean iterated over each array element, checking for validity and then appending the new value and validity bit. Using `BooleanArray::from_unary` is much faster, for two reasons: 1. It reuses the input's validity array instead of reconstructing it bit-by-bit 2. Avoiding per-element null checks makes it easier for the compiler to vectorize the loop. Casting an array of boolean to numeric also used a similar value-by-value loop. As before, we can do better by reusing the input's validity bitmap. We also play a further trick: we chunk the input bits into 64-bit words and process one chunk at a time; the fixed-size loop helps LLVM vectorize more effectively. Benchmarks: (M4 Max) - Int32 → Boolean, ~10% nulls: 17.310 → 0.899 µs — 19.26× faster - Boolean → Int32, ~10% nulls: 12.578 → 0.584 µs — 21.54× faster # What changes are included in this PR? * Optimize `cast_numeric_to_bool` and `cast_bool_to_numeric` as described above * Add unit tests * Add benchmarks # Are these changes tested? Yes; new tests added. # Are there any user-facing changes? No. # AI usage Developed with Codex (Astra 6), reviewed and revised with Claude Code (Opus 5.5). I reviewed and understand the resulting code.
# Which issue does this PR close? Closes apache#11282. # Rationale for this change REE slices with a nonzero offset can produce incorrect sums or incorrect overflow results. # What changes are included in this PR? Compute run lengths from slice-relative endpoints, with regression coverage for slices, nulls and overflow. # Are these changes tested? `cargo test --locked -p arrow-arith --all-features`, crate clippy, workspace formatting and the affected file's spelling check pass. # Are there any user-facing changes? Correct sums and overflow detection for sliced REE arrays; no API changes. AI disclosure: Codex generated the fix, regression tests and PR text; Claude Code assisted investigation.
…amps to a named timezone (apache#11038) # Which issue does this PR close? - Closes apache#11037. # Rationale for this change ## What a user hits today Cast a naive timestamp to a named timezone. Pick a wall clock reading that the zone skips or repeats. The cast fails. ```rust use arrow_array::TimestampSecondArray; use arrow_cast::{cast_with_options, CastOptions}; use arrow_schema::{DataType, TimeUnit}; // 1_710_037_800 is 2024-03-10T02:30:00, held as a naive wall clock reading. // America/New_York skips that hour: the clocks go forward at 02:00. let array = TimestampSecondArray::from(vec![1_710_037_800]); let to = DataType::Timestamp(TimeUnit::Second, Some("America/New_York".into())); let strict = CastOptions { safe: false, ..Default::default() }; let lenient = CastOptions { safe: true, ..Default::default() }; cast_with_options(&array, &to, &strict); cast_with_options(&array, &to, &lenient); ``` Today: ```text strict -> Err("Cast error: Cannot cast timezone to different timezone") lenient -> Ok, null_count = 1 ``` After this PR: ```text strict -> Ok, value = 1710055800 (2024-03-10T07:30:00Z, i.e. 03:30 EDT) lenient -> Ok, null_count = 0 ``` The `safe: true` result is the dangerous one. A whole column of DST-boundary readings turns to NULL, and nothing reports it. ## How this reaches a SQL user A DataFusion user reaches the same kernel through ordinary SQL. This is how the bug was found: ```sql SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/New_York'; -- Cast error: Cannot cast timezone to different timezone ``` The engine plans that as `Timestamp(_, None)` to `Timestamp(_, Some("America/New_York"))`, so the error text above comes straight from this kernel. See apache/datafusion#25084. An upcoming type-coercion change inserts the same cast automatically for `timestamptz - timestamp`, so the failure gets more common, not less. ## Why the kernel fails `adjust_timestamp_to_timezone` reads each value as a wall clock time in the target zone. In an IANA timezone that reading is not always one instant: * A "fall back" transition repeats an hour, so the reading is **ambiguous**. * A "spring forward" transition skips an hour, so the reading is **nonexistent**. The kernel resolved the offset with `offset_from_local_datetime(..).single()`. That is `None` for both cases, so both cases fail. # What changes are included in this PR? A new private helper, `resolve_local_offset`, decides the two cases instead of rejecting them. It follows PostgreSQL and DuckDB: * An **ambiguous** reading takes the **later** instant. This is the offset in effect after the transition. `2024-11-03T01:30:00` in `America/New_York` becomes `01:30-05:00` (EST). * A **nonexistent** reading moves **forward** by the length of the gap. `2024-03-10T02:30:00` in `America/New_York` becomes `03:30-04:00` (EDT). `adjust_timestamp_to_timezone` calls the helper in place of `.single()`. That is the whole behavioural change. The forward shift needs the offset in effect *before* the transition. The code recovers it with one probe: it asks for the offset 24 hours earlier and takes the earliest result. `offset_from_utc_datetime` is not a valid shortcut, because it returns the post-transition offset. That is wrong for a southern hemisphere gap such as `Australia/Sydney` `2024-10-06T02:30:00`. Two properties of the timezone database make the probe safe: * No two transitions in any of the 597 zones are closer than **167 hours**. So the probe lands on the correct side of the transition. The smallest interval is `America/Boa_Vista` in 2000. * No local gap is longer than **24 hours**. So the probe lands outside the gap at all. Seven zones sit exactly at 24 hours — the dateline changes, such as `Pacific/Apia` in 2011. The next largest gap is 10 hours. If the probe still fails, the helper returns `None` and the caller keeps the old error or NULL. With current tzdata that branch is unreachable. Nothing else changes. Unit conversion, `safe` handling and the error message stay as they were. A fixed-offset timezone has no transitions, so it is unaffected. # Are these changes tested? Yes. Six new tests sit next to the existing `test_cast_timestamp_with_timezone_*` tests in `arrow-cast/src/cast/mod.rs`: * `test_cast_timestamp_to_named_timezone_dst` — `America/New_York`, with an unambiguous, an ambiguous and a nonexistent reading plus a null, under `safe: false`. * `test_cast_timestamp_to_named_timezone_dst_safe` — the same input under `safe: true`, which no longer produces nulls. * `test_cast_timestamp_to_named_timezone_dst_southern_hemisphere` — `Australia/Sydney`, where the transitions run the other way round. * `test_cast_timestamp_to_named_timezone_dst_nanosecond` and `..._dst_changing_unit` — the resolution composes with the unit paths. * `test_cast_timestamp_to_fixed_offset_timezone_unaffected` — `+08:00` on the same readings is unchanged. Expected values come from `chrono`, built from the wall clock reading and the expected offset. They are not hardcoded. Revert the one-line kernel change and the five DST tests fail, while the fixed-offset test still passes. ## Validation The change was checked against 97,162 cases. These cover every transition of every IANA zone in 1890–1995 and 2023–2025: 15,871 in a gap and 14,370 ambiguous. There were zero disagreements with an independent implementation on the same tzdata. Offsets resolve **per row**. One array that straddles both 2024 `America/New_York` transitions yields three distinct offsets, and every element matches PostgreSQL 17.11. One unrelated discrepancy came up. chrono-tz 0.10.4 bundles tzdata **2025b**, which puts the `Europe/Chisinau` and `Europe/Tiraspol` transitions an hour away from PostgreSQL, ICU and CPython. That reproduces on `main` without this PR. It needs a chrono-tz bump, not a change here. ## A note on the test manifest arrow-cast had no way to name an IANA zone in its own tests. `Tz` parses IANA names only when `arrow-array` is built with `chrono-tz`, and arrow-cast never enabled it. This PR enables it on the **dev-dependency only**: `arrow-array = { workspace = true, features = ["chrono-tz"] }` under `[dev-dependencies]`. The new tests then run in every arrow-cast CI job. No public feature appears, and nothing changes for downstream crates. One test needed an update. `test_cast_string_to_timestamp_invalid_tz` asserted the exact error tail `only offset based timezones supported without chrono-tz feature`, which is now `failed to parse timezone`. It asserts the stable prefix `Parser error: Invalid timezone \"ZZTOP\":` instead. Local runs that pass: `cargo test -p arrow-cast` (default and `--all-features`, debug and release), `cargo test -p arrow --features chrono-tz,prettyprint --test array_cast --test timezone`, clippy with `-D warnings` on the CI feature combinations, and `cargo doc --all-features` with `-D warnings`. # Field research: PostgreSQL 17 and DuckDB The resolution policy is the one arbitrary decision in this change, so it needs more than one reference. Both engines were measured directly: PostgreSQL 17.11 in `postgres:17`, and the DuckDB 1.5.2 CLI with the ICU extension. All times below are UTC. | Case | PostgreSQL 17.11 | DuckDB 1.5.2 | This PR | | --- | --- | --- | --- | | `America/New_York` `2024-11-03 01:30` — **ambiguous** | `2024-11-03 06:30:00` | `2024-11-03 06:30:00` | `2024-11-03 06:30:00` | | `America/Havana` `2024-11-03 00:00` — **ambiguous** local midnight | `2024-11-03 05:00:00` | `2024-11-03 05:00:00` | `2024-11-03 05:00:00` | | `America/New_York` `2024-03-10 02:30` — gap | `2024-03-10 07:30:00` | `2024-03-10 07:30:00` | `2024-03-10 07:30:00` | | `America/Sao_Paulo` `2018-11-04 00:00` — local midnight does not exist | `2018-11-04 03:00:00` | `2018-11-04 03:00:00` | `2018-11-04 03:00:00` | | `Australia/Sydney` `2024-10-06 02:30` — southern hemisphere gap | `2024-10-05 16:30:00` | `2024-10-05 16:30:00` | `2024-10-05 16:30:00` | | `Australia/Lord_Howe` `2024-10-06 02:15` — 30 minute DST step | `2024-10-05 15:45:00` | `2024-10-05 15:45:00` | `2024-10-05 15:45:00` | | `Pacific/Chatham` `2024-09-29 03:00` — +12:45 / +13:45 | `2024-09-28 14:15:00` | `2024-09-28 14:15:00` | `2024-09-28 14:15:00` | **The two ambiguous rows carry the argument.** A gap has only one sensible answer, so it discriminates nothing. An ambiguous reading has two real answers, and the choice between them is the policy. `2024-11-03 01:30` in New York is either `05:30Z` (EDT) or `06:30Z` (EST). Both engines give `06:30Z`. Havana's ambiguous midnight is either `04:00Z` (CDT) or `05:00Z` (CST). Both engines give `05:00Z`. In both cases the answer is **the later instant**, which is what this PR does. ## Spark takes the earlier instant Spark 4.2.0 was measured on the same cases. `CAST(TIMESTAMP_NTZ ... AS TIMESTAMP)`, `to_utc_timestamp`, `convert_timezone` and a string cast all agree, with ANSI mode on and off. Spark matches every gap row above. It takes the **earlier** instant for every ambiguous reading: New York `01:30` gives `05:30Z`, Havana midnight gives `04:00Z`, and `Australia/Sydney` `2024-04-07 02:30` gives `2024-04-06 15:30Z`. Spark follows Java here: the cast calls [`convertTz`](https://github.com/apache/spark/blob/b1b2685058d6359e00842cbd4c723f4ce3810709/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L901), which uses `LocalDateTime.atZone` (`ZonedDateTime.of`). This PR keeps the PostgreSQL and DuckDB rule. ## The case neither engine can arbitrate A fixed-offset string such as `'+05:30'` has no agreed reading, so it is recorded here rather than hidden. Measured: | Spelling | PostgreSQL 17.11 | Convention | | --- | --- | --- | | `SET TimeZone='+05:30'` then `'2024-01-01 12:00:00'::timestamptz` | `17:30Z` | POSIX, west-positive | | `'2024-01-01 12:00:00'::timestamp AT TIME ZONE '+05:30'` | `17:30Z` | POSIX, west-positive | | `timestamptz '2024-01-01 12:00:00 +05:30'` | `06:30Z` | ISO, east-positive | PostgreSQL agrees with arrow-rs when the offset sits inside the literal. It disagrees when the offset acts as a zone name. DuckDB 1.5.2 rejects the zone-name spelling outright: `Not implemented Error: Unknown TimeZone '+05:30'`. So neither engine settles it. This PR does not change that behaviour. It is tracked downstream in apache/datafusion#25170. # Are there any user-facing changes? Yes. A cast that previously raised an error, or returned NULL under `safe: true`, now returns the instants described above. There is no API change. ## This PR does not close DataFusion #25084 on its own There are two independent paths to a DST-boundary failure. This PR fixes one of them: ```sql -- cast kernel, fixed here SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/New_York'; -- string parser, NOT fixed here (see apache#11039) SELECT '2024-03-10 02:30:00' AT TIME ZONE 'America/New_York'; SET datafusion.execution.time_zone = 'America/New_York'; SELECT '2024-03-10 02:30:00'::timestamptz; -- Parser error: Error parsing timestamp from '2024-03-10 02:30:00': error computing timezone offset ``` To a user those are the same query, and the last spelling is the most natural one. Merge this alone and an explicit `::timestamp` in the middle of an expression makes a query start to work. That is harder to explain than the current uniform failure. apache#11039 must land with this PR or close behind it, and must use the same policy so the two paths cannot drift. This PR **is** sufficient for apache/datafusion#10308, which is entirely on the cast path. ## Follow-up: `arrow-array` resolves ambiguity the opposite way After this PR, arrow resolves an ambiguous local time two different ways in two crates: | | Ambiguous | In a gap | | --- | --- | --- | | `arrow-cast::resolve_local_offset` (this PR) | the **later** instant | offset probed 24 hours earlier | | `arrow-array::types::from_naive_datetime` ([`types.rs:347`](https://github.com/apache/arrow-rs/blob/main/arrow-array/src/types.rs#L347)) | the **earlier** instant (`Ambiguous(dt1, _)`) | `None` | That function is deliberately untouched here. This matters downstream. DataFusion calls the `arrow-array` form directly with a `Some(tz)` in `datafusion/functions/src/datetime/date_part.rs` (`date_to_scalar`, four call sites). It builds local-midnight bounds for the `date_part(YEAR, col) = <year>` rewrite. Where that local midnight is ambiguous, the earlier instant becomes an **upper** bound, so valid rows can drop out. The behaviour predates this PR, but this PR widens the gap between the two policies. The two want reconciling, or the difference wants documenting on both functions. Changing `from_naive_datetime` affects every caller of `arrow-array`, so it deserves its own review rather than a ride along with a cast fix. ## Follow-up: a loud error becomes a quiet non-identity `to_local_time(ts) AT TIME ZONE tz` is a documented DataFusion idiom. Across an ambiguous hour it is no longer the identity, and it no longer says so. Before this PR the second leg raised `Cannot cast timezone to different timezone`: ```text orig local roundtrip identity 2021-10-31T02:00:00+02:00 2021-10-31T02:00:00 2021-10-31T02:00:00+01:00 false 2021-10-31T02:30:00+02:00 2021-10-31T02:30:00 2021-10-31T02:30:00+01:00 false 2021-10-31T02:00:00+01:00 2021-10-31T02:00:00 2021-10-31T02:00:00+01:00 true ``` PostgreSQL behaves the same way. `t = ((t AT TIME ZONE 'Europe/Brussels') AT TIME ZONE 'Europe/Brussels')` is `f` for those same two instants. So this is the intended trade, not an argument against the change. But it is a real behaviour change, and downstream docs want to say so. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
…e#10859) # Which issue does this PR close? - Closes apache#10926. - Related to apache#10774 and apache#10776. - apache#11007 adds a benchmark for same-projection predicate chains. # Rationale for this change Consecutive same-projection predicates can repeatedly decode a column or replay it from the predicate cache. The push decoder now wraps eligible groups in a `FusedPredicate` and evaluates them from one decoded stream. Predicates keep their order, and later predicates receive only surviving rows. Fusion is limited to a single top-level, non-repeated leaf. Contiguous survivors use zero-copy slices; fragmented survivors use `filter_record_batch`. Intermediate selections use the default `Auto` heuristic to switch between `RowSelector` runs and bitmaps by run density, independently of the reader's `RowSelectionPolicy`. The synchronous reader is unchanged and is left as a follow-up: it has no predicate cache, so it would benefit at least as much, but this PR keeps fusion inside the push decoder. There is no option to disable fusion. The measured regressions are bounded (see below) and only appear where compaction of ~50% survivors costs more than the saved decode. # What changes are included? - Group eligible consecutive predicates before building the push decoder. - Reuse the existing predicate execution path, including output-limit handling. - Adapt the accumulated selection once per composition, using the default `Auto` heuristic. - Document the fusion in the `RowFilter` docs. # Performance ClickBench Q25 (`SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10`) has two predicates on `SearchPhrase` once the TopK dynamic filter is pushed down, so the pair is fused. Measured with DataFusion `eea8c0961` `dfbench clickbench` on the partitioned 100-file dataset, 12 partitions, batch size 8192, `[patch.crates-io]` pointing the arrow crates at local checkouts. DataFusion builds against the released 59.x API, so `main` is the `59.3.0` tag and `fusion` is `59.3.0` plus this PR's diff. Three interleaved rounds of 40 iterations per variant, 120 samples each: | Variant | Median | Per-round medians | |---|---:|---| | main, pushdown off | 120.73 ms | 121.0 / 120.2 / 121.5 | | main, pushdown on | 146.67 ms | 146.9 / 146.6 / 146.7 | | fusion, pushdown on | 119.46 ms | 120.1 / 117.6 / 119.8 | Fusion removes the 21% cost that enabling pushdown adds on `main` for this query: fusion/on is 18.6% faster than main/on and 1.1% faster than main/off. # Testing - Fused/sequential selection equivalence, including prior selections, nulls, and early termination. - Projection eligibility, adaptive storage, and zero-copy slicing tests. - Push-decoder end-to-end coverage, including output limits. - Parquet library tests and async-reader tests. # User-facing changes No public API changes. `RowFilter` docs now describe when consecutive predicates share one decode. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
# Which issue does this PR close? - Closes apache#11205. # Rationale for this change `filter`, `filter_record_batch`, `BatchCoalescer::push_batch_with_filter`, and `FilterBuilder` start by calling `BooleanArray::true_count` on the caller-provided mask. However, some callers may already know the number of rows selected by the mask -- for example, because they built the mask one row at a time, derived it from another data structure (e.g., a validity bitmap), or needed to compute it for some other reason. In those situations, calling `true_count` requires a redundant full pass over the bitmap. Some examples from DataFusion where the count is known before the kernel runs: - `FilterExec` with a `LIMIT` counts the mask to track how many rows remain, then `push_batch_with_filter` counts it again. - `CASE` evaluation filters the remaining rows for each branch; the count of each new mask is the previous predicate's `count()` subtracted from the remaining row count. - `AND`/`OR` short-circuit evaluation counts the left side to decide whether to pre-select rows, then drops the count and filters the batch. - Aggregate `FILTER` clauses count the selection to decide whether filtering is worthwhile, then filter with the same mask. # What changes are included in this PR? - Add `FilterBuilder::with_count`, a `pub unsafe fn` that supplies the number of rows the mask selects so that `build` does not count them. To make this possible, `FilterBuilder` now counts the mask, picks the iteration strategy, and applies `optimize` in `build`, instead of in `new` and `optimize`. - Add `BatchCoalescer::push_batch_with_filter_builder`, which takes a `FilterBuilder` instead of a mask. It is a safe method: the count is supplied when the `FilterBuilder` is created, so `with_count` is the only new `unsafe` API. - The docs of `filter`, `filter_record_batch`, and `push_batch_with_filter` point to `FilterBuilder::with_count` for callers that already know the count. - Add tests. `with_count` is marked `unsafe` because the filter kernels rely on the count being correct: it sizes the filtered output, and a wrong count leads to out-of-bounds writes. The count is only checked with `debug_assert_eq!`. # Are these changes tested? Yes. Existing tests pass. `test_push_batch_with_filter_builder` covers the new `BatchCoalescer` method, using one builder with a caller-supplied count and one the caller optimized. Doc examples cover `FilterBuilder::with_count` and `BatchCoalescer::push_batch_with_filter_builder`. # Are there any user-facing changes? Two new public APIs: `FilterBuilder::with_count` and `BatchCoalescer::push_batch_with_filter_builder`. The work done by `FilterBuilder::new` and `FilterBuilder::optimize` now happens in `FilterBuilder::build`; this is not visible through the public API. # AI usage Developed with Claude Code (Fable 5.1 and Opus 5.5). I have reviewed and understand the resulting code.
…#11287) # Which issue does this PR close? - Part of apache#9296. - Pre-requisite for apache#11285 # Rationale for this change I'm planning a [PR]( apache#11285) that decodes the stored `ColumnIndex` bytes straight into Arrow arrays, skipping the intermediate `ColumnIndexMetaData`. This is the baseline benchmark for this [PR]( apache#11285). For page pruning, DataFusion's `page_filter.rs` asks `StatisticsConverter` for the data page mins, maxes and null counts of every row group it scans. Each request goes through the fully parsed `ColumnIndexMetaData`, so the column index is decoded into `Vec`s first and then copied into Arrow arrays. # What changes are included in this PR? A new `page_index_benchmark` group in `parquet/benches/arrow_statistics.rs`: - Writes an in-memory file with 20 row groups, 10 rows per data page and page-level statistics, with every 7th value null. - Covers `Int64`, `Utf8`, `Utf8View` and `Decimal128(20, 2)`, each with 2,000 and 10,000 data pages in total. - Reads each row group's raw column index bytes (via `column_index_range()`) and, inside the timed loop, does what DataFusion's page pruning effectively does today: `decode_column_index` → `PageIndexBuilder` → `StatisticsConverter::data_page_{mins,maxes,null_counts,nan_counts}`. # Are these changes tested? No. # Are there any user-facing changes? No. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
# Which issue does this PR close? - Closes apache#11304. # Rationale for this change Nightly Cargo reports redundant `homepage` entries, and nightly rustdoc reports redundant explicit link targets in `arrow-avro`. The rustdoc warnings fail the CI documentation job because it uses `-Dwarnings`. # What changes are included in this PR? - Remove `homepage` values that duplicate the repository URL from workspace members and the two standalone PyArrow test crates. - Let rustdoc infer the targets of five `arrow-avro` intra-doc links. # Are these changes tested? - `RUSTDOCFLAGS='-Dwarnings --enable-index-page -Zunstable-options' cargo +nightly doc --document-private-items --no-deps --workspace --all-features` passes with `rustc 1.101.0-nightly (5c543b0b8 2026-09-29)`. - `cargo +nightly metadata --no-deps --format-version 1` passes for each standalone PyArrow test crate. No runtime tests are needed because the changes affect only manifest metadata and documentation links. # Are there any user-facing changes? Crate listings no longer show a duplicate homepage link. There are no runtime or public API changes. Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
…e#11298) # Which issue does this PR close? N/A — small benchmark correction. # Rationale for this change `parquet_round_trip` clears its output buffer outside `b.iter`, so successive timed calls append complete files to the same Vec. Memory usage grows with Criterion's iteration count, and reallocations can affect the measurement. The subsequent read benchmark also inherits the accumulated bytes. # What changes are included in this PR? Move `buffer.clear()` inside the timed closure. Each iteration writes one file while retaining buffer capacity; the read benchmark receives the final single file. No library behavior changes. # Are these changes tested? - All 92 cases in the patched `parquet_round_trip` executable passed Criterion's `--test` smoke mode. - `rustfmt --check` and `git diff --check` passed. - Compared the actual benchmark executable at base `8208506f8f9ec193c08023ac2477d211d1d86d20` and this patch, using the same lockfile/dependencies. Linux x86-64, Ryzen AI 9 HX PRO 470, Rust 1.91.1, Criterion 0.8.2, optimized bench profile, pinned to CPU 2. Ran before / after / after / before, with 0.5 s warmup, 2 s measurement, 20 samples and 10,000 bootstrap resamples per case. | Benchmark | Before (ms, runs 1 / 2) | After (ms, runs 1 / 2) | Change in mean | | --- | ---: | ---: | ---: | | `write String(20) plain` | 4.548 / 4.496 | 4.382 / 4.340 | -3.6% | | `write int32 dict` | 6.902 / 7.092 | 6.859 / 7.009 | -0.9% | | `write int32 plain` | 2.991 / 3.033 | 2.773 / 2.761 | -8.1% | | `read String(20) plain` | 2.276 / 2.267 | 2.308 / 2.333 | +2.2% | | `read int32 dict` | 1.011 / 1.068 | 1.005 / 0.991 | -4.0% | | `read int32 plain` | 0.529 / 0.527 | 0.516 / 0.506 | -3.1% | Peak RSS for the whole filtered benchmark process was **602.3 / 601.7 MiB before**, versus **84.8 / 84.6 MiB after** (`/usr/bin/time -v`). This includes fixture generation and the harness, not just the timed operation. These short local runs quantify the changed benchmark behavior, not a Parquet implementation speedup. Plain integer writes improved by about 8%; dictionary-write timings were close. Read results varied in both directions, including a 2.2% increase for the string case. Measurement arguments (same on both revisions): ```sh cargo bench -p parquet --bench parquet_round_trip -- \ --warm-up-time 0.5 --measurement-time 2 --sample-size 20 \ --nresamples 10000 --noplot \ '^(write|read) (int32 (dict|plain)|String\(20\) plain)$' ``` # Are there any user-facing changes? None; benchmark harness only.
Bumps the codeql-actions group with 2 updates: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.38.1 to 4.38.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/releases">github/codeql-action/init's releases</a>.</em></p> <blockquote> <h2>v4.38.2</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.27.1">2.27.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4160">#4160</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/init's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <h2>[UNRELEASED]</h2> <p>No user facing changes.</p> <h2>4.38.2 - 24 Sept 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.27.1">2.27.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4160">#4160</a></li> </ul> <h2>4.38.1 - 18 Sept 2026</h2> <ul> <li>The CodeQL Action now has experimental support for CodeQL releases for which per-language bundles are available. Per-language bundles support analysis for a single language and are therefore smaller than the combined bundles that allow analysis for all supported languages. As a result, per-language bundles take up less space on disk and are faster to download. We expect to roll this change out to everyone in the coming weeks. <a href="https://redirect.github.com/github/codeql-action/pull/4146">#4146</a></li> </ul> <h2>4.38.0 - 09 Sept 2026</h2> <ul> <li>On GitHub-hosted runners, the CodeQL Action now deletes unused CodeQL bundles from the toolcache before downloading a different bundle, which frees up disk space for the analysis. We expect to roll this change out to everyone in September. <a href="https://redirect.github.com/github/codeql-action/pull/4124">#4124</a></li> <li>The CodeQL Action now supports CodeQL releases that are compatible with Linux Arm64 and downloads the native <code>linux-arm64</code> CodeQL bundle when available. <a href="https://redirect.github.com/github/codeql-action/pull/4072">#4072</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.27.0">2.27.0</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4129">#4129</a></li> </ul> <h2>4.37.9 - 26 Aug 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.4">2.26.4</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4106">#4106</a></li> </ul> <h2>4.37.8 - 21 Aug 2026</h2> <p>No user facing changes.</p> <h2>4.37.7 - 13 Aug 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.3">2.26.3</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4085">#4085</a></li> </ul> <h2>4.37.6 - 04 Aug 2026</h2> <ul> <li>Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to <code>.github/codeql-config.yml</code> to align it with the suggested path that is used elsewhere. <a href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li> </ul> <h2>4.37.5 - 03 Aug 2026</h2> <ul> <li>Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the <code>init</code> Action instead of falling back to downloading the bundle before extracting it. <a href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li> </ul> <h2>4.37.4 - 29 Jul 2026</h2> <ul> <li>This version of the CodeQL Action adds support for the <code>tools</code> input for the <code>codeql-action/init</code> step to be specified using a <code>github-codeql-tools</code> <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository property</a>. This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to <code>toolcache</code> to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for <code>tools</code> in the workflow definition always takes precedence unless the value of the repository property starts with <code>!</code>. <a href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.2">2.26.2</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4051">#4051</a></li> </ul> <h2>4.37.3 - 22 Jul 2026</h2> <p>No user facing changes.</p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2"><code>2892aa5</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4168">#4168</a> from github/update-v4.38.2-a6ef2c96f</li> <li><a href="https://github.com/github/codeql-action/commit/8ad03a333eb88de8ad6833eda208d0fc51a9c571"><code>8ad03a3</code></a> Trigger workflows</li> <li><a href="https://github.com/github/codeql-action/commit/98af865db5041cee73c7185896319367f8c0adf2"><code>98af865</code></a> Update changelog for v4.38.2</li> <li><a href="https://github.com/github/codeql-action/commit/a6ef2c96fc0e37d0b44fb2bd0b32db4bcb89ae24"><code>a6ef2c9</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4156">#4156</a> from github/mario-campos/fix-validate-cmd</li> <li><a href="https://github.com/github/codeql-action/commit/1ef28a1b7603ca158fd774d1ab328cbd6a40b84b"><code>1ef28a1</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4166">#4166</a> from github/dependabot/github_actions/dot-github/wor...</li> <li><a href="https://github.com/github/codeql-action/commit/26cb08bab0037de74cc66ad9ec0dca31d6d9e8a7"><code>26cb08b</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4163">#4163</a> from github/mbg/fix-getCommitOid-stubs</li> <li><a href="https://github.com/github/codeql-action/commit/f035ce3a985a1223a9f59fb719542598640160b2"><code>f035ce3</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4165">#4165</a> from github/dependabot/npm_and_yarn/npm-minor-8eaed9...</li> <li><a href="https://github.com/github/codeql-action/commit/5e4e2550b48d7f3de205c9d752eb5176bf07f6d9"><code>5e4e255</code></a> Rebuild</li> <li><a href="https://github.com/github/codeql-action/commit/b13f5f47d5398d0fb982942ced6fdfc4e3951804"><code>b13f5f4</code></a> Bump ruby/setup-ruby</li> <li><a href="https://github.com/github/codeql-action/commit/c87fe5756c0c0bcd5e0005d2169945cfee9a232f"><code>c87fe57</code></a> Rebuild</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/1c5b675653bb5c22dbe9b12b556ec555138e09fd...2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2">compare view</a></li> </ul> </details> <br /> Updates `github/codeql-action/analyze` from 4.38.1 to 4.38.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/releases">github/codeql-action/analyze's releases</a>.</em></p> <blockquote> <h2>v4.38.2</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.27.1">2.27.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4160">#4160</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/analyze's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <h2>[UNRELEASED]</h2> <p>No user facing changes.</p> <h2>4.38.2 - 24 Sept 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.27.1">2.27.1</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4160">#4160</a></li> </ul> <h2>4.38.1 - 18 Sept 2026</h2> <ul> <li>The CodeQL Action now has experimental support for CodeQL releases for which per-language bundles are available. Per-language bundles support analysis for a single language and are therefore smaller than the combined bundles that allow analysis for all supported languages. As a result, per-language bundles take up less space on disk and are faster to download. We expect to roll this change out to everyone in the coming weeks. <a href="https://redirect.github.com/github/codeql-action/pull/4146">#4146</a></li> </ul> <h2>4.38.0 - 09 Sept 2026</h2> <ul> <li>On GitHub-hosted runners, the CodeQL Action now deletes unused CodeQL bundles from the toolcache before downloading a different bundle, which frees up disk space for the analysis. We expect to roll this change out to everyone in September. <a href="https://redirect.github.com/github/codeql-action/pull/4124">#4124</a></li> <li>The CodeQL Action now supports CodeQL releases that are compatible with Linux Arm64 and downloads the native <code>linux-arm64</code> CodeQL bundle when available. <a href="https://redirect.github.com/github/codeql-action/pull/4072">#4072</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.27.0">2.27.0</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4129">#4129</a></li> </ul> <h2>4.37.9 - 26 Aug 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.4">2.26.4</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4106">#4106</a></li> </ul> <h2>4.37.8 - 21 Aug 2026</h2> <p>No user facing changes.</p> <h2>4.37.7 - 13 Aug 2026</h2> <ul> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.3">2.26.3</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4085">#4085</a></li> </ul> <h2>4.37.6 - 04 Aug 2026</h2> <ul> <li>Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to <code>.github/codeql-config.yml</code> to align it with the suggested path that is used elsewhere. <a href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li> </ul> <h2>4.37.5 - 03 Aug 2026</h2> <ul> <li>Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the <code>init</code> Action instead of falling back to downloading the bundle before extracting it. <a href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li> </ul> <h2>4.37.4 - 29 Jul 2026</h2> <ul> <li>This version of the CodeQL Action adds support for the <code>tools</code> input for the <code>codeql-action/init</code> step to be specified using a <code>github-codeql-tools</code> <a href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository property</a>. This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to <code>toolcache</code> to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for <code>tools</code> in the workflow definition always takes precedence unless the value of the repository property starts with <code>!</code>. <a href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li> <li>Update default CodeQL bundle version to <a href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.2">2.26.2</a>. <a href="https://redirect.github.com/github/codeql-action/pull/4051">#4051</a></li> </ul> <h2>4.37.3 - 22 Jul 2026</h2> <p>No user facing changes.</p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2"><code>2892aa5</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4168">#4168</a> from github/update-v4.38.2-a6ef2c96f</li> <li><a href="https://github.com/github/codeql-action/commit/8ad03a333eb88de8ad6833eda208d0fc51a9c571"><code>8ad03a3</code></a> Trigger workflows</li> <li><a href="https://github.com/github/codeql-action/commit/98af865db5041cee73c7185896319367f8c0adf2"><code>98af865</code></a> Update changelog for v4.38.2</li> <li><a href="https://github.com/github/codeql-action/commit/a6ef2c96fc0e37d0b44fb2bd0b32db4bcb89ae24"><code>a6ef2c9</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4156">#4156</a> from github/mario-campos/fix-validate-cmd</li> <li><a href="https://github.com/github/codeql-action/commit/1ef28a1b7603ca158fd774d1ab328cbd6a40b84b"><code>1ef28a1</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4166">#4166</a> from github/dependabot/github_actions/dot-github/wor...</li> <li><a href="https://github.com/github/codeql-action/commit/26cb08bab0037de74cc66ad9ec0dca31d6d9e8a7"><code>26cb08b</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4163">#4163</a> from github/mbg/fix-getCommitOid-stubs</li> <li><a href="https://github.com/github/codeql-action/commit/f035ce3a985a1223a9f59fb719542598640160b2"><code>f035ce3</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/4165">#4165</a> from github/dependabot/npm_and_yarn/npm-minor-8eaed9...</li> <li><a href="https://github.com/github/codeql-action/commit/5e4e2550b48d7f3de205c9d752eb5176bf07f6d9"><code>5e4e255</code></a> Rebuild</li> <li><a href="https://github.com/github/codeql-action/commit/b13f5f47d5398d0fb982942ced6fdfc4e3951804"><code>b13f5f4</code></a> Bump ruby/setup-ruby</li> <li><a href="https://github.com/github/codeql-action/commit/c87fe5756c0c0bcd5e0005d2169945cfee9a232f"><code>c87fe57</code></a> Rebuild</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/1c5b675653bb5c22dbe9b12b556ec555138e09fd...2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…che#11290) Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.87.16 to 2.87.21. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/releases">taiki-e/install-action's releases</a>.</em></p> <blockquote> <h2>2.87.21</h2> <ul> <li> <p>Update <code>wasm-bindgen@latest</code> to 0.2.129.</p> </li> <li> <p>Update <code>protoc-gen-connect-openapi@latest</code> to 0.27.3.</p> </li> <li> <p>Update <code>wasmtime@latest</code> to 49.0.1.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.12.19.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.9.13.</p> </li> <li> <p>Update <code>kingfisher@latest</code> to 2.7.0.</p> </li> </ul> <h2>2.87.20</h2> <ul> <li> <p>Update <code>uv@latest</code> to 0.12.18.</p> </li> <li> <p>Update <code>cargo-shear@latest</code> to 1.14.0.</p> </li> </ul> <h2>2.87.19</h2> <ul> <li> <p>Update <code>wasmtime@latest</code> to 49.0.0.</p> </li> <li> <p>Update <code>cargo-shear@latest</code> to 1.13.5.</p> </li> <li> <p>Update <code>cargo-nextest@latest</code> to 0.9.146.</p> </li> </ul> <h2>2.87.18</h2> <ul> <li> <p>Update <code>oxfmt@latest</code> to 1.84.0.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.9.12.</p> </li> <li> <p>Update <code>kache@latest</code> to 0.26.3.</p> </li> <li> <p>Update <code>cargo-tarpaulin@latest</code> to 0.37.4.</p> </li> <li> <p>Update <code>cargo-rdme@latest</code> to 2.2.3.</p> </li> </ul> <h2>2.87.17</h2> <ul> <li> <p>Update <code>uv@latest</code> to 0.12.17.</p> </li> <li> <p>Update <code>release-plz@latest</code> to 0.3.169.</p> </li> <li> <p>Update <code>kingfisher@latest</code> to 2.5.0.</p> </li> <li> <p>Update <code>kache@latest</code> to 0.25.0.</p> </li> <li> <p>Update <code>git-cliff@latest</code> to 2.14.2.</p> </li> <li> <p>Update <code>cargo-tarpaulin@latest</code> to 0.37.3.</p> </li> <li> <p>Update <code>cargo-leptos@latest</code> to 0.3.9.</p> </li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md">taiki-e/install-action's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <p>All notable changes to this project will be documented in this file.</p> <p>This project adheres to <a href="https://semver.org">Semantic Versioning</a>.</p> <!-- raw HTML omitted --> <h2>[Unreleased]</h2> <h2>[2.87.21] - 2026-09-26</h2> <ul> <li> <p>Update <code>wasm-bindgen@latest</code> to 0.2.129.</p> </li> <li> <p>Update <code>protoc-gen-connect-openapi@latest</code> to 0.27.3.</p> </li> <li> <p>Update <code>wasmtime@latest</code> to 49.0.1.</p> </li> <li> <p>Update <code>uv@latest</code> to 0.12.19.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.9.13.</p> </li> <li> <p>Update <code>kingfisher@latest</code> to 2.7.0.</p> </li> </ul> <h2>[2.87.20] - 2026-09-24</h2> <ul> <li> <p>Update <code>uv@latest</code> to 0.12.18.</p> </li> <li> <p>Update <code>cargo-shear@latest</code> to 1.14.0.</p> </li> </ul> <h2>[2.87.19] - 2026-09-23</h2> <ul> <li> <p>Update <code>wasmtime@latest</code> to 49.0.0.</p> </li> <li> <p>Update <code>cargo-shear@latest</code> to 1.13.5.</p> </li> <li> <p>Update <code>cargo-nextest@latest</code> to 0.9.146.</p> </li> </ul> <h2>[2.87.18] - 2026-09-22</h2> <ul> <li> <p>Update <code>oxfmt@latest</code> to 1.84.0.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.9.12.</p> </li> <li> <p>Update <code>kache@latest</code> to 0.26.3.</p> </li> <li> <p>Update <code>cargo-tarpaulin@latest</code> to 0.37.4.</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/taiki-e/install-action/commit/4cef1412cce204788f482e778a0b9187f9626a29"><code>4cef141</code></a> Release 2.87.21</li> <li><a href="https://github.com/taiki-e/install-action/commit/133135e7fd1570baff8964a044bcf0d2147fc57a"><code>133135e</code></a> Update <code>wasm-bindgen@latest</code> to 0.2.129</li> <li><a href="https://github.com/taiki-e/install-action/commit/8589241798155a8c0c86c7c0ae4f642a90a698e1"><code>8589241</code></a> Update <code>protoc-gen-connect-openapi@latest</code> to 0.27.3</li> <li><a href="https://github.com/taiki-e/install-action/commit/95e5bbc4a58633e12ebefd904f798c1eb76c18f7"><code>95e5bbc</code></a> Update wasmtime manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/48ce11d4f97676e64a76a3dc4b83a4f59245b702"><code>48ce11d</code></a> Update <code>wasmtime@latest</code> to 49.0.1</li> <li><a href="https://github.com/taiki-e/install-action/commit/368e6617a899743920fa751bb5d414b6f0352cc0"><code>368e661</code></a> Update wasm-bindgen manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/57f22a457e27d10a13bfcaee43e03dff1c20dedd"><code>57f22a4</code></a> Update <code>uv@latest</code> to 0.12.19</li> <li><a href="https://github.com/taiki-e/install-action/commit/888634e2727c1021a94ebfac9a2d266109a366b8"><code>888634e</code></a> Update typos manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/3bcab01fb9790f533b19647e60396514a9d08f96"><code>3bcab01</code></a> Update protoc-gen-connect-openapi manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/947e90eb285641d8325cc2d55ae24bd51d5c6b8a"><code>947e90e</code></a> Update <code>mise@latest</code> to 2026.9.13</li> <li>Additional commits viewable in <a href="https://github.com/taiki-e/install-action/compare/9114bf4d891761788c546334fd37538eae1bf8b3...4cef1412cce204788f482e778a0b9187f9626a29">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…rgo-deps group across 1 directory (apache#11291) Bumps the all-other-cargo-deps group with 1 update in the / directory: [comfy-table](https://github.com/nukesor/comfy-table). Updates `comfy-table` from 8.0.0 to 8.0.1 <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/Nukesor/comfy-table/blob/main/CHANGELOG.md">comfy-table's changelog</a>.</em></p> <blockquote> <h2>[8.0.1] - 2026-09-25</h2> <ul> <li>Set MSRV to 1.88</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Nukesor/comfy-table/commit/b947cb6bf4a9aef846e4ec00a67f68bd85d41c73"><code>b947cb6</code></a> chore: Release comfy-table version 8.0.1</li> <li><a href="https://github.com/Nukesor/comfy-table/commit/283fed25ada438f0cc1a6a007af5363ccb236693"><code>283fed2</code></a> chore: Set msrv to 1.88</li> <li><a href="https://github.com/Nukesor/comfy-table/commit/bbd884f91c4760b965f56ba136df0e353841bb06"><code>bbd884f</code></a> Merge pull request <a href="https://redirect.github.com/nukesor/comfy-table/issues/212">#212</a> from Nukesor/dependabot/cargo/rstest-0.27</li> <li><a href="https://github.com/Nukesor/comfy-table/commit/f3c6d7ea94f1d610262d4f153603f5e8fec40c70"><code>f3c6d7e</code></a> build(deps): update rstest requirement from 0.26 to 0.27</li> <li>See full diff in <a href="https://github.com/nukesor/comfy-table/compare/v8.0.0...v8.0.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
# Which issue does this PR close? Closes apache#10978. # Rationale for this change Valid negative decimal scales can overflow `i8` precision/scale calculations, causing debug panics or incorrect metadata and cast fast-path selection when overflow checks are disabled. # What changes are included in this PR? - Compute add/subtract/remainder precision and scale differences in `i16`, clamping precision before narrowing. - Reject unrepresentable multiplication scales and overflowing remainder alignment factors. - Widen decimal cast scale differences and precision checks while preserving existing rounding, nulls, and safe/unsafe cast behavior. # Are these changes tested? Regression tests cover negative-scale metadata, adjacent minimum scales, alignment overflow, multiplication scale limits, and safe/unsafe decimal casts. Arithmetic and cast cases are split by expected metadata and failure path, with exact error assertions. The original failure reproduction is recorded against base `b1aa94b0`. - `cargo +stable test -p arrow-arith -p arrow-cast --lib`: 628 passed. - `cargo +stable --config 'profile.test.package.arrow-arith.overflow-checks=false' --config 'profile.test.package.arrow-cast.overflow-checks=false' test -p arrow-arith -p arrow-cast --lib decimal256 --verbose`: 36 passed. This is an unchecked test profile, not an optimized release build. - `cargo +stable fmt --all -- --check` and `git diff --check`: passed. - `cargo +stable clippy -p arrow-arith -p arrow-cast --lib --tests --no-deps -- -D warnings -A clippy::chunks_exact_to_as_chunks`: passed. Local checks used Rust 1.98.0 on Windows because the installed repository-pinned 1.97.1 toolchain was incomplete. Strict Clippy first reported the new `chunks_exact_to_as_chunks` lint in unchanged `arrow-buffer` and `arrow-arith/src/aggregate.rs` code. The scoped run allows only that lint; the repository's existing removed-lint configuration also emits a compatibility warning. No source or lint configuration was changed for these diagnostics. # Are there any user-facing changes? Affected addition, subtraction, multiplication, remainder, and decimal casts now return correctly calculated results or checked errors for extreme negative scales instead of panicking or wrapping metadata. Public API signatures are unchanged. AI assistance: OpenAI Codex generated the implementation, regression tests, review follow-up, and PR text, and ran the checks above. The author personally reviewed the original change at `66680fd2`; the follow-up refactoring and test revisions were reviewed by Codex agents.
…e#11359) # Which issue does this PR close? - N/A # Rationale for this change Casting a List, LargeList, ListView or Map cast its whole child array. When the input to the cast was sliced, this was both inefficient and could also result in spurious errors, if a value outside the visible span made the cast fail (with `safe: false`). For lists and maps, we now cast only the range of child values that the rows span, and rebase the offsets. For ListView, a row can point anywhere in the values buffer, so the visible elements might not form a contiguous span. Casting only the visible values would be expensive and require copies, so we instead settle for an overestimate: we use the lowest starting offset and highest ending offset of any row. This is an improvement over the previous behavior, but it does result in unnecessarily casting values that are inside that range but not part of any row. # What changes are included in this PR? * Only look at the elements in the visible span, for each of the casts in question * The value buffer of the cast result is potentially smaller than the input value buffer, in which case we need to revise the offset buffer * Add unit tests # Are these changes tested? Yes; existing tests pass, new tests added. # Are there any significant user-facing changes? No. # AI usage Developed and revised with Claude Code (Opus 5.5). I have reviewed and understand the resulting code.
# Which issue does this PR close? - Closes apache#10524. # Rationale for this change Parquet 1.0 writers silently ignored explicit dictionary settings for `FIXED_LEN_BYTE_ARRAY` columns, even though dictionary encoding is valid for the physical type. # What changes are included in this PR? Honor explicit global and per-column dictionary opt-ins for Parquet 1.0 FLBA columns while preserving the existing default-off behavior. Update CDC expectations for FixedSizeBinary arrays when dictionary encoding is explicitly enabled. # Are these changes tested? Yes. Added coverage for the default behavior, global opt-in, per-column override, and explicit disable cases. The focused tests, full Parquet suite, Clippy, and formatting checks pass. # Are there any user-facing changes? Callers can now explicitly enable dictionary encoding for Parquet 1.0 FLBA columns. The default behavior is unchanged. --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
apache#11323) # Which issue does this PR close? - N/A # Rationale for this change Arrow does not require bytes outside the visible span of the values buffer to be valid UTF-8. This rule was not consistently applied: * `ArrayData` checked for UTF-8 validity over the visible span, but * `StringArray::try_new` and `try_from_binary` checked for UTF-8 validity of the entire values buffer, which is both inefficient for sliced arrays and also rejects otherwise valid inputs * `substring` assumed that the entire values buffer contained valid UTF-8 bytes, which was sound for inputs from the `StringArray` path but not for the `ArrayData` path * Binary to Utf8 casts (in `safe: true`) reserved enough space for the whole values buffer, not the visible span # What changes are included in this PR? * Both validators now check only the bytes between the first and last offset * `substring` reads only those bytes as a `&str`. * The `safe: true` fallback of the Binary to Utf8 cast only reserves space for the visible span, not the whole values buffer. * Improve some docs * Add unit tests # Are these changes tested? Yes; existing tests pass, new tests added. # Are there any user-facing changes? No. # AI usage Developed with Claude Code (Opus 5.5), reviewed with Codex (Astra 6). I revised and understand the resulting code.
# Which issue does this PR close? Closes apache#11364. # Rationale for this change Writing an empty slice of a nonempty `RunArray` succeeds, but reading the resulting IPC batch fails because the writer emits a run end of zero. For example, slicing run ends `[2, 5]` and values `[10, 20]` with `slice(2, 0)` produces this error. Run ends must be strictly positive. # What changes are included in this PR? Normalize empty run arrays to zero-offset arrays with empty run-end and value children before calculating physical run indices. Add regression tests for Int16, Int32, and Int64 run ends, every legal empty-slice offset in the example array, already-empty arrays, and both Stream and File IPC roundtrips. Implementation and regression tests were generated with OpenAI Codex and reviewed and validated with the checks below. # Are these changes tested? - Default arrow-ipc library tests: 134 passed. - All-features arrow-ipc library tests: 140 passed. - New regression tests with `arrow-data/force_validate`: 3 passed. - Clippy for arrow-ipc with all targets and all features, with warnings denied: passed. - Workspace formatting and diff whitespace checks: passed. The full all-features suite with `arrow-data/force_validate` has six failures in existing reader tests that intentionally construct invalid arrays. The same six failures were reproduced on the unchanged base commit; all new tests pass with forced validation. # Are there any user-facing changes? Empty RunArray slices now roundtrip through Stream and File IPC successfully. No public API changes.
…dation (apache#11362) # Which issue does this PR close? Closes apache#11363. # Rationale for this change FixedSizeList nullability validation compares expanded parent validity with the unsliced child validity, ignoring the parent offset. This can both reject valid arrays and accept nulls in non-nullable items. For example, with list size 2, parent offset 1 and length 2, and child values `[0, 1, null, null, 4, 5]`, parent validity `[false, true]` should be accepted but is rejected; `[true, false]` should be rejected but is accepted. When the parent has no null buffer, child nulls outside the visible range can also cause an incorrect rejection. # What changes are included in this PR? - Slice child validity to `offset * list_size .. (offset + len) * list_size` before checking non-nullable items. - Include the parent offset in the required child length check so validation rejects undersized children before slicing. - Add regression coverage for masked and visible child nulls, child offsets, absent parent null buffers, nulls outside the visible range, empty slices, zero-sized lists, and undersized children. # Are these changes tested? All three new regression tests failed before the fix and pass afterward. - `cargo test -p arrow-buffer -p arrow-data -p arrow-array -p arrow-select -p arrow-ipc --lib`: 1,690 passed, 1 ignored. - `cargo clippy -p arrow-data --all-targets -- -D warnings`: passed. - `cargo fmt --all -- --check`: passed. - `git diff --check`: passed. # Are there any user-facing changes? Validation now correctly accepts child nulls masked by a sliced FixedSizeList parent or outside its visible range, and rejects visible nulls in non-nullable items. Undersized child arrays are rejected when the parent offset requires additional elements. No public API changes. # AI assistance AI assistance was used to investigate the validation semantics, implement the fix and regression tests, review the diff, run the checks above, and prepare this PR description.
# Which issue does this PR close? - N/A # Rationale for this change Casting integers to decimals currently uses `unary_opt` (in safe mode) or `try_unary` (otherwise). This is relatively slow, because neither kernel vectorizes well. In practice, `int` -> `decimal` casts are frequently infallible, either because the target type can accommodate any integer, or because the target type is chosen to be large enough to accommodate all inputs in the source data. We can exploit this by scanning the source data to check if all of the values will fit into the target type. If so, we can use `unary` to do the cast. Empirically, the cost of doing the scan is much smaller than the win from using the faster kernel (see benchmarks below). To make checking for out-of-range values efficient, we use a few techniques: 1. Avoid scanning entirely if the target type is large enough to hold any value of the input type 2. Check for out-of-range values with a branch-less scan, so it vectorizes well 3. Use chunking and early-stopping: for many input distributions, if there is an out of range value it will likely appear in the first few chunks, so we don't need to scan the entire input anyway. Benchmarks: (512 rows, M4 Max) - cast int64 to decimal128(38, 10): 994.360 → 300.580 ns, −69.8% - cast int64 within 1e6 to decimal128(18, 2): 989.400 → 315.160 ns, −68.1% - cast int64 within 1e6 late overflow to decimal128(18, 2): 992.870 → 1060.300 ns, +6.8% - cast int64 to decimal32(9, 0): 738.400 → 781.610 ns, +5.9% - cast int64 to decimal32(9, -1): 957.480 → 991.820 ns, +3.6% In the first case, no input scan is needed because the target type is large enough to hold any `int64`. In the second case, we need to scan the inputs to establish that no out-of-range value is found. The third benchmark is the worst-case, where an out-of-range value is found at the end of the input. The last two cases are existing benchmarks where an out-of-range value is found early in the scan. # What changes are included in this PR? * Add `integers_fit_digits`, which does a chunked, branch-free scan to check for out-of-range values * Use `unary` in `cast_integer_to_decimal`, when the cast is infallible * Add unit test * Extend benchmarks # Are these changes tested? Yes; existing tests pass and new test added. # Are there any user-facing changes? No. Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
… 4 GiB when decoding BinaryView/Utf8View (apache#11224) ## Which issue does this PR close? - Closes apache#11011. ## Rationale for this change `decode_binary_view_inner` casts the data buffer offset to `u32` with no bound, so once the non-inlined values in a batch go over 4 GiB the views silently point at the wrong bytes. The issue left open whether to split the buffer or fail. This PR fails, since it is the smaller change and keeps the hot loop as is. Splitting into multiple buffers can come later if people hit this in practice, and the signatures would not need to change again. ## What changes are included in this PR? - `decode_binary_view` and `decode_string_view` now return `Result`; `decode_column` already returns `Result`, so this is internal only. - New `check_view_buffer_len` helper, called once after the capacity pass on the summed length of non-inlined values, returns `InvalidArgumentError` above `u32::MAX`. - The per-row cast keeps `as u32` with a `debug_assert!`, same approach as `concat_elements_view_array`. ## Are these changes tested? Yes, a unit test on the helper with synthetic lengths (`u32::MAX` passes, `u32::MAX + 1` and `usize::MAX` fail), since actually decoding more than 4 GiB is too much for a test. Existing arrow-row tests pass. I ran the `row_format` string view `convert_rows` benches before and after, but my machine was shared and an A/A run swung by up to 2x, so I can't claim anything from the numbers beyond no obvious regression. The release-mode per-row loop is unchanged. The only added work is one compare per column. ## Are there any user-facing changes? `RowConverter::convert_rows` now returns an error for this case instead of returning wrong data. No public API changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Panchawat <whitepattpon@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
Add the arrow.fixed_closedness_range canonical extension type: a bounded
interval over an orderable type T with a single type-level closedness
("left", "right", "both" or "neither") shared by all values, stored as a
two-field lower/upper struct where a null bound means an unbounded side.
… type Add the arrow.variable_closedness_range canonical extension type, the companion of arrow.fixed_closedness_range for ranges that cannot be canonicalized to a single closedness: each value carries its own lower_inc/upper_inc inclusivity flags alongside the lower/upper bounds, mirroring PostgreSQL's internal range representation.
Hoeze
force-pushed
the
feat/arrow-range-extension
branch
from
October 4, 2026 12:52
27a52a5 to
5b17b76
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
apache/arrow#50028 proposes two canonical extension types for ranges, i.e.
intervals with a lower and an upper bound over an orderable type T. This PR
adds them to
arrow-schemaas an independent Rust implementation. The typesonly become canonical after the vote on the Arrow dev mailing list, so this PR
should not be merged before the vote passes.
What changes are included in this PR?
FixedClosednessRange(arrow.fixed_closedness_range): storageStruct<lower: T, upper: T>, metadata{"closed": "left" | "right" | "both" | "neither"}.VariableClosednessRange(arrow.variable_closedness_range): storageStruct<lower: T, upper: T, lower_inc: Boolean, upper_inc: Boolean>,metadata
{}.CanonicalExtensionType.Metadata handling and storage validation follow the C++ implementation:
unknown metadata keys are ignored, and each bound may be nullable or not.
Are these changes tested?
Yes, with unit tests for both types: metadata round trips, and rejection of
invalid metadata and storage types.
Note that I made heavy use of AI to create this PR and copied many structures
from the existing
Opaqueand tensor extension types. I reviewed each changeand hope the changes I made are meaningful.
Are there any user-facing changes?
Yes, but only additions behind the
canonical_extension_typesfeature.CanonicalExtensionTypeis#[non_exhaustive], so the new variants are notbreaking.