Skip to content

PERF: cascade Series.dt.components for ArrowDtype durations - #221

Open
xirafa wants to merge 1 commit into
mainfrom
aboderinsamuel/perf-63283-dt-components-cascade
Open

xirafa wants to merge 1 commit into
mainfrom
aboderinsamuel/perf-63283-dt-components-cascade

Conversation

@xirafa

@xirafa xirafa commented Aug 7, 2026 •

Copy link
Copy Markdown

Mirror of pandas-dev/pandas#66635. Runs the pandas test suite through dagz.

Summary by CodeRabbit

  • Performance Improvements
    • Improved performance when extracting datetime components from Arrow-backed duration data.
    • Duration components are now calculated more efficiently while preserving accurate cascading values across days, hours, minutes, seconds, milliseconds, microseconds, and nanoseconds.

@coderabbitai

coderabbitai Bot commented Aug 7, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@xirafa, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 62 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6eb3a9c6-09c8-42fa-8da4-2fcf11880de7

📥 Commits

Reviewing files that changed from the base of the PR and between 44bbe2e and 9da7eba.

📒 Files selected for processing (3)
  • doc/source/whatsnew/v3.1.0.rst
  • pandas/core/arrays/arrow/array.py
  • pandas/tests/extension/test_arrow.py
📝 Walkthrough

Walkthrough

ArrowExtensionArray._dt_components now extracts duration sub-day fields from cascading remainders instead of independent modulo operations. The release notes document this performance improvement.

Changes

Arrow duration component extraction

Layer / File(s) Summary
Cascading remainder extraction
pandas/core/arrays/arrow/array.py, doc/source/whatsnew/v3.1.0.rst
_dt_components progressively reduces a non-negative duration remainder from days through nanoseconds. It preserves nulls when finer resolutions are unavailable and avoids separate accessor calls and modulo operations. The release note documents the change.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: jbrockmendel, fangchenli, rhshadrach

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main performance change to cascade Series.dt.components for ArrowDtype durations.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch aboderinsamuel/perf-63283-dt-components-cascade

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
pandas/core/arrays/arrow/array.py (2)

3908-3909: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Compute the day quotient and remainder once.

self._dt_days performs the cast and floor division. _dt_day_remainder then repeats the same work. This makes every Series.dt.components call compute the day decomposition twice. Cache the day quotient and remainder together, then reuse both results here.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pandas/core/arrays/arrow/array.py` around lines 3908 - 3909, Update the
datetime components construction around _dt_days and _dt_day_remainder to
compute the day quotient and remainder together once, cache both values, and
reuse them for the "days" component and remainder processing instead of invoking
both properties independently.

3923-3925: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Skip the final remainder update.

After the last available component for the selected duration unit, Line 3925 assigns a remainder that the loop never reads. This still allocates pc.multiply and pc.subtract results. Use the current remainder directly when the divisor is 1, or guard the update when no finer component exists.

Proposed localized fix
             if divisor is None:
                 # array resolution is coarser than this component -> always 0
                 components[name] = self._dt_zero_or_null_int32()
                 continue
+            if divisor == 1:
+                components[name] = self._from_pyarrow_array(
+                    remainder.cast(pa.int32())
+                )
+                continue
             value = pc.divide(remainder, divisor)
             components[name] = self._from_pyarrow_array(value.cast(pa.int32()))
             remainder = pc.subtract(remainder, pc.multiply(value, divisor))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pandas/core/arrays/arrow/array.py` around lines 3923 - 3925, Update the
component-processing loop around value, components, and remainder so the final
available duration component does not perform an unused remainder update. When
divisor is 1, retain the current remainder directly; otherwise only execute
pc.multiply and pc.subtract when a finer component remains, preserving existing
component values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@doc/source/whatsnew/v3.1.0.rst`:
- Line 361: Update the issue reference in the Series.dt.components ArrowDtype
duration performance bullet to use tracking number 66635 instead of 63283,
leaving the description unchanged.

---

Nitpick comments:
In `@pandas/core/arrays/arrow/array.py`:
- Around line 3908-3909: Update the datetime components construction around
_dt_days and _dt_day_remainder to compute the day quotient and remainder
together once, cache both values, and reuse them for the "days" component and
remainder processing instead of invoking both properties independently.
- Around line 3923-3925: Update the component-processing loop around value,
components, and remainder so the final available duration component does not
perform an unused remainder update. When divisor is 1, retain the current
remainder directly; otherwise only execute pc.multiply and pc.subtract when a
finer component remains, preserving existing component values.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 84d9cd2d-d70e-4bf1-9af9-0ec1034a308d

📥 Commits

Reviewing files that changed from the base of the PR and between 4aac140 and 44bbe2e.

📒 Files selected for processing (2)
  • doc/source/whatsnew/v3.1.0.rst
  • pandas/core/arrays/arrow/array.py

Comment thread doc/source/whatsnew/v3.1.0.rst Outdated
@xirafa
xirafa force-pushed the main branch 5 times, most recently from 5e4f543 to 56632f7 Compare August 7, 2026 22:37
@xirafa
xirafa force-pushed the aboderinsamuel/perf-63283-dt-components-cascade branch from 3d07137 to e975485 Compare August 8, 2026 13:07
@xirafa
xirafa force-pushed the main branch 22 times, most recently from de56a49 to 1096988 Compare August 10, 2026 22:41
@xirafa
xirafa force-pushed the main branch 30 times, most recently from 43ae4ae to aa8f7e4 Compare August 18, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants