Repository navigation
PERF: use Arrow native path in duration component calculation for Arrow-backed array - #63470
Merged
jbrockmendel merged 36 commits intoJun 19, 2026
Merged
jbrockmendel merged 36 commits into
jbrockmendel merged 36 commits into
Conversation
fangchenli
marked this pull request as ready for review
December 23, 2025 23:13
fangchenli
added
Performance
…nents-arrow-native
…nents-arrow-native
Member
Sure. What script do you want to benchmark? |
jbrockmendel
reviewed
Mar 3, 2026
jbrockmendel
reviewed
Mar 3, 2026
…nents-arrow-native
…nents-arrow-native
…nents-arrow-native
Member
|
pls add a whatsnew for the perf improvement also maybe the fix to setting index for .components |
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jbrockmendel
reviewed
Jun 4, 2026
jbrockmendel
reviewed
Jun 4, 2026
…nents-arrow-native
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nents-arrow-native
Series.dt.microseconds on a coarser duration unit (e.g. duration[ms]) short-circuited to 0 instead of scaling the sub-second portion up to microseconds, so duration[ms].dt.microseconds returned 0 rather than the total microseconds. Compute it by scaling the sub-second remainder to microseconds in either direction, and extend test_dt_timedelta_components_different_units to check the direct .dt.[days, seconds, microseconds, nanoseconds] accessors against the NumPy-backed result at every unit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nents-arrow-native
3 tasks done
…nents-arrow-native
…mainder sort() replaces _pa_array in place, bypassing __setitem__, so the cached _dt_day_remainder (and any other cache_readonly) must be cleared there too, or duration components would reflect the pre-sort order. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jbrockmendel
approved these changes
Jun 17, 2026
Member
|
LGTM ping on green |
…nents-arrow-native
Member
|
Thanks @fangchenli |
hamdanal
pushed a commit
to hamdanal/pandas
that referenced
this pull request
Jun 27, 2026
…ow-backed array (pandas-dev#63470) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rok
reviewed
Jul 28, 2026
Comment on lines
+208
to
+227
| def floor_div_int64( | ||
| arr: pa.Array | pa.ChunkedArray, divisor: int | ||
| ) -> pa.Array | pa.ChunkedArray: | ||
| """ | ||
| Floor division for an int64 (chunked) array, handling negatives correctly. | ||
|
|
||
| PyArrow's ``pc.divide`` truncates toward zero, but we need floor division | ||
| (toward negative infinity) to match pandas behavior for negative durations. | ||
| Uses a fast path for all-non-negative arrays (the common case). | ||
| """ | ||
| min_scalar = pc.min(arr) | ||
| if not min_scalar.is_valid or min_scalar.as_py() >= 0: | ||
| # Fast path: truncation equals floor when all values are non-negative | ||
| return pc.divide(arr, divisor) | ||
| # Truncated division (toward zero) and remainder with the dividend's sign | ||
| trunc_div = pc.divide(arr, divisor) | ||
| remainder = pc.subtract(arr, pc.multiply(trunc_div, divisor)) | ||
| # floor(a/b) = trunc(a/b) - 1 when a < 0 and remainder != 0 | ||
| needs_adjustment = pc.and_(pc.less(arr, 0), pc.not_equal(remainder, 0)) | ||
| return pc.subtract(trunc_div, pc.if_else(needs_adjustment, 1, 0)) |
Contributor
There was a problem hiding this comment.
Why not use pc.floor_temporal instead @fangchenli ?
Member
Author
There was a problem hiding this comment.
pc.floor_temporal has no duration kernel and is much slower in non-negative cases. But it would be nice if Arrow could handle all those.
Contributor
There was a problem hiding this comment.
Duration work is in progress and I expect it to be in the next release apache/arrow#50675.
What do you mean by non-negative cases? That sounds like something we should fix in arrow.
Also please ping me if there are other missing functionalities in pyarrow kernels, especially temporal!
Contributor
There was a problem hiding this comment.
I merged apache/arrow#50675 and you can expect it in the next release.
Member
Author
There was a problem hiding this comment.
Thanks! I'll do more testing to narrow down the slow cases I observed earlier.
5 of 7 tasks
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
.dt.secondsDiscrepancy Between NumPy and PyArrow Timedelta Dtypes #63283doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.AGENTS.md.Computes
Series.dtduration components (days,seconds,microseconds,nanoseconds,components, ...) forArrowDtypedurations directly with PyArrow compute kernels, instead of converting to a NumPyTimedeltaArrayand deriving each field fromTimedeltaArray.components..dt.days.dt.seconds.dt.microseconds.dt.nanoseconds.dt.componentsMethodology:
n=10,000,000random durations spanning ~±115 days, including negatives (so the after path runs the general floor-division branch, not the all-non-negative fast path). Best of 5 cold calls — the_dt_day_remaindercache is cleared before each call. Before is the pre-PR path (_to_timedeltaarray().components); after is the publicSeries.dt.<accessor>. On the before side,.dt.componentscosts ~7× a single field because the old implementation built the DataFrame one component at a time, each re-converting the array; on the after side it computes_dt_day_remainderonce and reuses it across the sub-day fields (~2× vs. recomputing per field). Run on an M1 MacBook Pro — absolute numbers are machine-dependent, but the speedups should hold.