Dask DataFrame按常量列rand3排序触发除零错误，求解决方法

当使用Dask DataFrame对全为1的常量列rand3排序时触发除零错误，但对随机值列rand1排序正常。代码及报错如下：

测试代码 #

import numpy as np
import pandas as pd

import dask.dataframe as dd
import dask.array as da
import dask.bag as db

NumberOfSample = 50
NumberOfPartition = 2
SortByColumn = ['rand3', 'rand1', 'rand2']

pandas_df = pd.DataFrame("Happy", index=list(range(0,NumberOfSample)), columns=['x', 'y']) 
pandas_df['rand1'] = np.random.randint(0, 10000, size=NumberOfSample) / 10
pandas_df['rand2'] = np.random.randint(0, 10000, size=NumberOfSample) / 10
pandas_df['rand3'] = 1

ddf = dd.from_pandas(pandas_df, npartitions=NumberOfPartition)

print('ddf before sorting is with len ' + str(len(ddf)))
print(ddf.head(NumberOfSample, NumberOfPartition))

ddf_sorted2 = ddf.sort_values(by=['rand3'], ascending=True, inplace=False)
# ddf_sorted2 = ddf.sort_values(by=['rand1'], ascending=True, inplace=False)

报错信息 #

Traceback (most recent call last):

  File C:\PythonProjects\TradeAnalysis\Test\untitled0.py:29 in <module>
    ddf_sorted2 = ddf.sort_values(by=['rand3'], ascending=True, inplace=False)

  File ~\anaconda3\lib\site-packages\dask\dataframe\core.py:4470 in sort_values
    return sort_values(

  File ~\anaconda3\lib\site-packages\dask\dataframe\shuffle.py:159 in sort_values
    df = rearrange_by_divisions(

  File ~\anaconda3\lib\site-packages\dask\dataframe\shuffle.py:459 in rearrange_by_divisions
    df3 = rearrange_by_column(

  File ~\anaconda3\lib\site-packages\dask\dataframe\shuffle.py:486 in rearrange_by_column
    df = df.repartition(npartitions=npartitions)

  File ~\anaconda3\lib\site-packages\dask\dataframe\core.py:1319 in repartition
    return repartition_npartitions(self, npartitions)

  File ~\anaconda3\lib\site-packages\dask\dataframe\core.py:6970 in repartition_npartitions
    npartitions_ratio = df.npartitions / npartitions

ZeroDivisionError: division by zero

Dask的sort_values默认采用基于分位数的分区排序策略：先计算排序列的分位数，以此为边界将数据划分到不同分区，再对每个分区内部排序，最后合并结果。当排序列的所有值完全相同时，计算分位数无法生成有效的分区边界，导致内部逻辑中npartitions被设置为0，后续执行repartition时触发df.npartitions / npartitions的除零错误。

而随机值列rand1有足够的区分度，能生成有效的分位数分区边界，因此排序正常。

1. 强制使用全量洗牌排序 #

显式指定shuffle='shuffle'参数，绕过分位数分区逻辑，直接对全量数据进行洗牌排序：

ddf_sorted2 = ddf.sort_values(by=['rand3'], ascending=True, shuffle='shuffle')

2. 合并为单分区后排序 #

先将DataFrame合并为单个分区，再执行排序，避免分区划分的问题：

ddf_sorted2 = ddf.repartition(npartitions=1).sort_values(by=['rand3'], ascending=True)

3. 优先使用有区分度的列排序 #

如果是多列排序场景，将有区分度的列放在排序列表的前面，Dask会基于该列生成有效分区边界，常量列仅作为次要排序条件：

# 先按rand1排序，再按rand3排序
ddf_sorted2 = ddf.sort_values(by=['rand1', 'rand3'], ascending=True)

内容的提问来源于stack exchange，提问作者Henry