Dataframe shuffle: Behaviour for NULL partitions differs from Pandas 3

[crusaderky]

FAILED distributed/shuffle/tests/test_shuffle.py::test_handle_null_partitions_2 - AssertionError: Attributes of DataFrame.iloc[:, 1] (column name="b") are different

Attribute "dtype" are different
[left]: <StringDtype(na_value=)>
[right]: float64

This test has been broken since the day Pandas 3 was released:

Note that the 3.10 environment still uses pandas 2.

[crusaderky]

The issue is that Pandas 3 changed how a concat of a series of None with a series of numbers behaves.
In Pandas 2, the None's were automatically converted to NaN, with a FutureWarning.
In Pandas 3, the numbers are converted to object dtype and the None's are retained.

>>> pd.concat([pd.DataFrame({"a": [None, None]}), pd.DataFrame({"a": [1.0, 2.0]})], axis=0)

# Pandas 2.3
FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated.
In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes.
To retain the old behavior, exclude the relevant entries before the concat operation.
     a
0  NaN
1  NaN
0  1.0
1  2.0

# Pandas 3.0
      a
0  None
1  None
0   1.0
1   2.0

The test is failing because Dask shuffle retains the behaviour of Pandas 2 after you upgrade to Pandas 3 (whereas a plain dd.concat is coherent to Pandas).
This is a genuine issue that needs fixing (Dask behaviour should align as much as possible to Pandas's) but it feels to me quite low in priority.

I'm XFAIL'ing the test for the time being.