BUG: GroupBy.std floating point error

[jbrockmendel]

values = pd.timedelta_range("1 Day", periods=10_000).asi8
ser = pd.Series(values)
gb = ser.groupby(list(range(5)) * 2000)

result = gb.std()
expected = pd.Series([gb.get_group(i).std() for i in gb.groups])

>>> result - expected
0    192.0
1    192.0
2    192.0
3    192.0
4    192.0
dtype: float64

In the groupby std method we cast from ints to floats, which I don't think we do in the relevant Series code (in nanops). This is my best guess for the culprit in this mismatch.

[matt-hoskins]

In writing some test cases for some of my code I had an oddity where I'd compute the std of some values to figure out what the output of a routine doing multiple aggregations of groups should be and the std result via the group by gave a very slightly different result for one set of values.

When I searched the list of open issues I saw this bug report and wondered if what I'm observing is related (or a misunderstanding on my part). I pared it down to a short bit of code to illustrate:

import pandas

score_list = [1,1,2]

stdseries = pandas.Series(score_list).std()
stdgb = pandas.DataFrame({'A':['a' for x in score_list],'B':score_list}).groupby('A').std()['B']['a']

print(stdseries==stdgb)
print(stdseries)
print(stdgb)

And the output:

False
0.5773502691896257
0.5773502691896258

[jbrockmendel]

When I searched the list of open issues I saw this bug report and wondered if what I'm observing is related (or a misunderstanding on my part)

I expect the difference is due to having different implementations of std for Series vs GroupBy, xref #53261

[matt-hoskins]

When I searched the list of open issues I saw this bug report and wondered if what I'm observing is related (or a misunderstanding on my part)

I expect the difference is due to having different implementations of std for Series vs GroupBy, xref #53261

I guess the implementation difference in std for GroupBy is also why the GroupBy version of it seems to give different results depending on the order of the values fed into it...

import pandas

score_list2 = [2.142857142857143, 2, 1.75]
stdgb1 = pandas.DataFrame({'A':['a' for x in score_list2],'B':score_list2}).groupby('A').std(ddof=1)['B']['a']
stdgb2 = pandas.DataFrame({'A':['a' for x in score_list2],'B':sorted(score_list2)}).groupby('A').std(ddof=1)['B']['a']
print(stdgb1 == stdgb2)
print(stdgb1)
print(stdgb2)

...

False
0.19884872724392919
0.19884872724392935

It took me a while to figure out why the test I wrote to test some code I have which aggregates via groupby was calculating a different answer for std in some cases even when using groupby itself until I double checked the problematic group of values and noticed the order was the only difference.

[jbrockmendel]

@Alvaro-Kothe i suspect the solution here is to do the same thing for std that you did for skew/kurt?

[Alvaro-Kothe]

In the groupby std method we cast from ints to floats, which I don't think we do in the relevant Series code (in nanops). This is my best guess for the culprit in this mismatch.

group_var uses the Welford algorithm (one-pass), while nanvar uses the two-pass algorithm. This difference in algorithms explains the discrepancy in results. The two-pass method can compute exact results for integral data, whereas Welford is subject to round-off errors.

Although the absolute error seems large (192), the values themselves are big. The relative error is about 2.5e−16.

Also, trying to migrate nanvar to Welford have shown a negative performance impact in #64848.