Commit 027ea1edde4ee1de97c89cca4f782a00d1955375

[github-actions]

Commit Range

Subsequent benchmarks may have skipped some commits. The link above lists the commits that are between the two benchmark runs where the regression was identified.

• stat_ops.FrameMultiIndexOps.time_op
  • op='kurt' - 46.508% (855.551us)
• stat_ops.FrameOps.time_op
  • op='kurt', dtype='Int64', axis=0 - 51.823% (1.036ms)
• stat_ops.FrameOps.time_op
  • op='kurt', dtype='float', axis=0 - 52.392% (948.162us)
• stat_ops.FrameOps.time_op
  • op='kurt', dtype='int', axis=None - 142.497% (1.341ms)
• stat_ops.SeriesMultiIndexOps.time_op
  • op='kurt' - 41.303% (181.119us)
• stat_ops.SeriesOps.time_op
  • op='kurt', dtype='float' - 39.741% (173.084us)

[rhshadrach]

OP got the commit incorrect, this is due to pandas-dev/pandas#64366. Timings look a bit different than what was reported there. Still, not sure there is anything to do here.

cc @Alvaro-Kothe

[jbrockmendel]

Haven't looked closely, but I suspect these were the regressions we expected. Options to address them are 1) integer dtype cases can go through numpy (which get SIMD), 2) openmp across columns.

[rhshadrach]

Many of those above (e.g. the first one) were not reported in the OP of #64366. I'm assuming this is because the benchmarks produced there showed up as performance neutral (within the tolerance) and here it is showing 40%.

In addition, stat_ops.FrameOps.time_op('kurt', 'int', None) was reported as a 44% regression in the PR, and here we're seeing 1ms -> 3ms which is 300%.

[Alvaro-Kothe]

Most of them were expected.

The benchmark stat_ops.FrameOps.time_op('kurt', 'int', None), which had a regression from 1ms -> 3ms, is running in a similar time to stat_ops.FrameOps.time_op('kurt', 'float', None), that got a performance increase from 5.5ms -> 2.5ms, they are using the same algorithm and are running in similar times.

I am currently working on adding vectorization to pandas-dev/pandas#64582. Unfortunetelly, it's specific for x86_64 with AVX2 support, but for me, increased performane for most of these cases, e.g.:

| -        | 90.1±10μs                  | 71.9±2μs                                |    0.8  | groupby.GroupByMethods.time_dtype_as_field('int16', 'skew', 'direct', 1, 'cython')       |
| -        | 18.5±0.3μs                 | 14.4±0.8μs                              |    0.78 | series_methods.NanOps.time_func('kurt', 1000, 'int32')                                   |
| -        | 19.1±1μs                   | 14.9±2μs                                |    0.78 | series_methods.NanOps.time_func('skew', 1000, 'int8')                                    |
| -        | 22.3±3μs                   | 17.2±0.5μs                              |    0.77 | series_methods.NanOps.time_func('skew', 1000, 'boolean')                                 |
| -        | 18.9±0.9μs                 | 14.6±3μs                                |    0.77 | series_methods.NanOps.time_func('skew', 1000, 'int32')                                   |
| -        | 18.9±0.8μs                 | 14.3±1μs                                |    0.76 | series_methods.NanOps.time_func('kurt', 1000, 'int8')                                    |
| -        | 23.5±2μs                   | 17.9±0.4μs                              |    0.76 | series_methods.NanOps.time_func('skew', 1000, 'Int64')                                   |
| -        | 19.0±2μs                   | 14.3±0.9μs                              |    0.75 | series_methods.NanOps.time_func('kurt', 1000, 'float64')                                 |
| -        | 19.6±2μs                   | 14.8±3μs                                |    0.75 | series_methods.NanOps.time_func('kurt', 1000, 'int64')                                   |
| -        | 19.0±0.8μs                 | 14.2±0.2μs                              |    0.75 | series_methods.NanOps.time_func('skew', 1000, 'float64')                                 |
| -        | 19.5±2μs                   | 14.2±0.7μs                              |    0.73 | series_methods.NanOps.time_func('skew', 1000, 'int64')                                   |
| -        | 24.7±3μs                   | 17.8±0.8μs                              |    0.72 | series_methods.NanOps.time_func('kurt', 1000, 'boolean')                                 |
| -        | 28.5±5μs                   | 18.2±1μs                                |    0.64 | series_methods.NanOps.time_func('kurt', 1000, 'Int64')                                   |
| -        | 5.08±0.09ms                | 1.62±0.2ms                              |    0.32 | stat_ops.FrameOps.time_op('kurt', 'Int64', None)                                         |
| -        | 5.05±0.2ms                 | 1.50±0.08ms                             |    0.3  | stat_ops.FrameOps.time_op('kurt', 'int', None)                                           |
| -        | 4.98±0.3ms                 | 1.48±0.4ms                              |    0.3  | stat_ops.FrameOps.time_op('skew', 'Int64', None)                                         |
| -        | 4.74±0.3ms                 | 1.43±0.1ms                              |    0.3  | stat_ops.FrameOps.time_op('skew', 'int', None)                                           |
| -        | 4.74±0.2ms                 | 1.36±0.4ms                              |    0.29 | stat_ops.FrameOps.time_op('skew', 'Int64', 0)                                            |
| -        | 4.81±0.2ms                 | 1.35±0.3ms                              |    0.28 | stat_ops.FrameOps.time_op('kurt', 'Int64', 0)                                            |
| -        | 10.3±0.5ms                 | 2.22±0.1ms                              |    0.22 | series_methods.NanOps.time_func('kurt', 1000000, 'Int64')                                |
| -        | 9.76±0.03ms                | 2.15±0.07ms                             |    0.22 | series_methods.NanOps.time_func('kurt', 1000000, 'int64')                                |
| -        | 9.59±0.1ms                 | 2.03±0.3ms                              |    0.21 | series_methods.NanOps.time_func('kurt', 1000000, 'boolean')                              |
| -        | 9.51±0.3ms                 | 1.95±0.2ms                              |    0.21 | series_methods.NanOps.time_func('skew', 1000000, 'int64')                                |
| -        | 4.72±0.3ms                 | 934±200μs                               |    0.2  | stat_ops.FrameOps.time_op('kurt', 'int', 0)                                              |
| -        | 9.56±0.08ms                | 1.80±0.1ms                              |    0.19 | series_methods.NanOps.time_func('kurt', 1000000, 'int32')                                |
| -        | 9.41±0.05ms                | 1.77±0.3ms                              |    0.19 | series_methods.NanOps.time_func('kurt', 1000000, 'int8')                                 |
| -        | 10.0±0.5ms                 | 1.94±0.1ms                              |    0.19 | series_methods.NanOps.time_func('skew', 1000000, 'Int64')                                |
| -        | 4.23±0.4ms                 | 825±40μs                                |    0.19 | stat_ops.FrameOps.time_op('skew', 'int', 0)                                              |
| -        | 1.03±0.05ms                | 186±9μs                                 |    0.18 | stat_ops.SeriesOps.time_op('kurt', 'int')                                                |
| -        | 3.77±0.2ms                 | 633±200μs                               |    0.17 | stat_ops.FrameMultiIndexOps.time_op('skew')                                              |
| -        | 1.03±0.04ms                | 172±10μs                                |    0.17 | stat_ops.SeriesOps.time_op('skew', 'int')                                                |
| -        | 9.72±0.5ms                 | 1.59±0.6ms                              |    0.16 | series_methods.NanOps.time_func('skew', 1000000, 'boolean')                              |
| -        | 9.93±0.6ms                 | 1.57±0.1ms                              |    0.16 | series_methods.NanOps.time_func('skew', 1000000, 'int32')                                |
| -        | 9.23±0.6ms                 | 1.44±0.2ms                              |    0.16 | series_methods.NanOps.time_func('skew', 1000000, 'int8')                                 |
| -        | 3.89±0.06ms                | 558±30μs                                |    0.14 | stat_ops.FrameMultiIndexOps.time_op('kurt')                                              |
| -        | 964±20μs                   | 132±30μs                                |    0.14 | stat_ops.SeriesOps.time_op('kurt', 'float')                                              |
| -        | 4.15±0.2ms                 | 535±30μs                                |    0.13 | stat_ops.FrameOps.time_op('kurt', 'float', 0)                                            |
| -        | 3.72±0.02ms                | 461±40μs                                |    0.12 | stat_ops.FrameOps.time_op('skew', 'float', 0)                                            |
| -        | 9.17±0.5ms                 | 948±40μs                                |    0.1  | series_methods.NanOps.time_func('kurt', 1000000, 'float64')                              |
| -        | 4.11±0.1ms                 | 409±90μs                                |    0.1  | stat_ops.FrameOps.time_op('kurt', 'float', None)                                         |
| -        | 1.03±0.03ms                | 106±5μs                                 |    0.1  | stat_ops.SeriesMultiIndexOps.time_op('kurt')                                             |
| -        | 953±30μs                   | 92.2±10μs                               |    0.1  | stat_ops.SeriesMultiIndexOps.time_op('skew')                                             |
| -        | 942±70μs                   | 86.9±30μs                               |    0.09 | stat_ops.SeriesOps.time_op('skew', 'float')                                              |
| -        | 8.53±0.05ms                | 723±40μs                                |    0.08 | series_methods.NanOps.time_func('skew', 1000000, 'float64')                              |
| -        | 3.56±0.01ms                | 297±20μs                                |    0.08 | stat_ops.FrameOps.time_op('skew', 'float', None)                                         |

[rhshadrach]

The benchmark stat_ops.FrameOps.time_op('kurt', 'int', None), which had a regression from 1ms -> 3ms, is running in a similar time to stat_ops.FrameOps.time_op('kurt', 'float', None), that got a performance increase from 5.5ms -> 2.5ms, they are using the same algorithm and are running in similar times.

Agreed with this, my only concern is that in the first instance stat_ops.FrameOps.time_op('kurt', 'int', None) here has a ration of 3.0 whereas the PR reported as 1.42. When I run the asv benchmarks locally on pandas-dev/pandas@b935209 compared to the previous, I see:

| Change   | Before [027ea1ed] <api_ea_methods~2^2~1>   | After [b935209d] <api_ea_methods~2^2>   |   Ratio | Benchmark (Parameter)                            |
|----------|--------------------------------------------|-----------------------------------------|---------|--------------------------------------------------|
| +        | 609±40μs                                   | 1.99±0.01ms                             |    3.27 | stat_ops.FrameOps.time_op('skew', 'int', None)   |
| +        | 652±10μs                                   | 1.87±0.08ms                             |    2.87 | stat_ops.FrameOps.time_op('kurt', 'int', 0)      |
| +        | 678±20μs                                   | 1.81±0.01ms                             |    2.67 | stat_ops.FrameOps.time_op('skew', 'int', 0)      |
| +        | 959±10μs                                   | 1.74±0ms                                |    1.82 | stat_ops.FrameOps.time_op('kurt', 'float', 0)    |
| +        | 977±7μs                                    | 1.70±0ms                                |    1.74 | stat_ops.FrameOps.time_op('skew', 'float', 0)    |
| +        | 1.15±0.02ms                                | 1.89±0.01ms                             |    1.65 | stat_ops.FrameOps.time_op('skew', 'Int64', 0)    |
| -        | 7.71±0.2ms                                 | 6.16±0.2ms                              |    0.8  | stat_ops.FrameOps.time_op('skew', 'int', 1)      |
| -        | 3.27±0.8ms                                 | 1.89±0.02ms                             |    0.58 | stat_ops.FrameOps.time_op('skew', 'Int64', None) |
| -        | 4.69±4ms                                   | 2.08±0.06ms                             |    0.44 | stat_ops.FrameOps.time_op('kurt', 'Int64', None) |
| -        | 4.11±2ms                                   | 1.73±0.02ms                             |    0.42 | stat_ops.FrameOps.time_op('kurt', 'float', None) |
| -        | 4.39±2ms                                   | 1.66±0ms                                |    0.38 | stat_ops.FrameOps.time_op('skew', 'float', None) |
| -        | 20.3±0.1ms                                 | 6.00±0.04ms                             |    0.3  | stat_ops.FrameOps.time_op('kurt', 'float', 1)    |
| -        | 20.8±0.7ms                                 | 5.85±0.06ms                             |    0.28 | stat_ops.FrameOps.time_op('skew', 'float', 1)    |

A second run:

| Change   | Before [027ea1ed] <api_ea_methods~2^2~1>   | After [b935209d] <api_ea_methods~2^2>   |   Ratio | Benchmark (Parameter)                              |
|----------|--------------------------------------------|-----------------------------------------|---------|----------------------------------------------------|
| +        | 629±30μs                                   | 2.02±0.01ms                             |    3.22 | stat_ops.FrameOps.time_op('skew', 'int', None)     |
| +        | 646±100μs                                  | 2.05±0ms                                |    3.17 | stat_ops.FrameOps.time_op('kurt', 'int', None)     |
| +        | 683±5μs                                    | 1.91±0.06ms                             |    2.79 | stat_ops.FrameOps.time_op('skew', 'int', 0)        |
| +        | 669±30μs                                   | 1.85±0.01ms                             |    2.76 | stat_ops.FrameOps.time_op('kurt', 'int', 0)        |
| +        | 956±10μs                                   | 1.76±0ms                                |    1.84 | stat_ops.FrameOps.time_op('kurt', 'float', 0)      |
| +        | 993±10μs                                   | 1.74±0.01ms                             |    1.75 | stat_ops.FrameOps.time_op('skew', 'float', 0)      |
| +        | 1.11±0.01ms                                | 1.94±0.01ms                             |    1.74 | stat_ops.FrameOps.time_op('kurt', 'Int64', 0)      |
| +        | 1.14±0.06ms                                | 1.98±0.04ms                             |    1.73 | stat_ops.FrameOps.time_op('skew', 'Int64', 0)      |
| +        | 1.95±0.06ms                                | 2.27±0.2ms                              |    1.16 | stat_ops.FrameOps.time_op('median', 'Int64', None) |
| -        | 8.05±0.4ms                                 | 6.23±0.07ms                             |    0.77 | stat_ops.FrameOps.time_op('skew', 'int', 1)        |
| -        | 10.8±0.9ms                                 | 6.44±0.07ms                             |    0.6  | stat_ops.FrameOps.time_op('kurt', 'int', 1)        |
| -        | 3.68±0.7ms                                 | 1.95±0.01ms                             |    0.53 | stat_ops.FrameOps.time_op('skew', 'Int64', None)   |
| -        | 3.96±0.6ms                                 | 1.93±0.01ms                             |    0.49 | stat_ops.FrameOps.time_op('kurt', 'Int64', None)   |
| -        | 4.35±2ms                                   | 1.69±0.02ms                             |    0.39 | stat_ops.FrameOps.time_op('skew', 'float', None)   |
| -        | 5.12±1ms                                   | 1.71±0.01ms                             |    0.33 | stat_ops.FrameOps.time_op('kurt', 'float', None)   |
| -        | 22.1±0.3ms                                 | 6.09±0.08ms                             |    0.28 | stat_ops.FrameOps.time_op('kurt', 'float', 1)      |
| -        | 21.6±0.2ms                                 | 5.79±0.07ms                             |    0.27 | stat_ops.FrameOps.time_op('skew', 'float', 1)      |

Looks like it was just a noisy benchmark (as the links above show as well). Interesting, they are no longer noisy. If anyone has a good understanding for the cause of this noise, I have never been able to figure it out.

In any case, I think there is nothing to do here; closing.

[Alvaro-Kothe]

Agreed with this, my only concern is that in the first instance stat_ops.FrameOps.time_op('kurt', 'int', None) here has a ration of 3.0 whereas the PR reported as 1.42.

I believe this is primarily due to the machine that is running the benchmark. Your machine runs in ~0.6ms, while mine takes ~2.7. Because mine is slower, the relative difference (ratio) will be reduced.