向Dask DataFrame添加pandas Series时出现NaN值的问题排查

出现NaN的核心是Dask默认按索引对齐列，而从numpy数组创建的Dask DataFrame与从pandas Series创建的Dask Series，即使行数、分区数相同，也可能因为分区的边界划分逻辑不同，导致索引范围无法完全匹配，最终对齐失败产生NaN。

以下是三种可行的解决方法，按需选择：

方法1：按位置直接匹配（推荐） #

利用numpy数组的顺序性，将pandas Series的值转为numpy数组后，用dd.from_array创建与原DataFrame分区完全一致的Dask Series，跳过索引对齐，直接按位置匹配：

from dask.dataframe.core import DataFrame as DaskDataFrame
import dask.dataframe as dd
import pandas as pd
import numpy as np

xy = np.random.rand(int(3e6), 2)
c = pd.Series(np.random.choice(['a', 'b', 'c'], int(3e6)), dtype='category')

table: DaskDataFrame = dd.from_array(xy, columns=['x', 'y'])
# 从Series的值创建同分区的Dask Series，指定category元数据
table['c'] = dd.from_array(c.values, meta=('c', 'category'), npartitions=table.npartitions)
print(table.compute())

方法2：统一索引与分区逻辑 #

先将numpy数组转为pandas DataFrame，再转成Dask DataFrame，确保其索引与目标Series完全一致，再添加列：

from dask.dataframe.core import DataFrame as DaskDataFrame
import dask.dataframe as dd
import pandas as pd
import numpy as np

xy = np.random.rand(int(3e6), 2)
c = pd.Series(np.random.choice(['a', 'b', 'c'], int(3e6)), dtype='category')

# 先转pandas DataFrame，确保索引与c对齐
pd_table = pd.DataFrame(xy, columns=['x', 'y'])
table: DaskDataFrame = dd.from_pandas(pd_table, npartitions=4)
table['c'] = dd.from_pandas(c, npartitions=table.npartitions)
print(table.compute())

方法3：用map_partitions逐分区添加 #

通过map_partitions对每个分区的pandas DataFrame直接添加对应分区的Series数据，完全按位置匹配：

from dask.dataframe.core import DataFrame as DaskDataFrame
import dask.dataframe as dd
import pandas as pd
import numpy as np

xy = np.random.rand(int(3e6), 2)
c = pd.Series(np.random.choice(['a', 'b', 'c'], int(3e6)), dtype='category')

table: DaskDataFrame = dd.from_array(xy, columns=['x', 'y'])
# 转为Dask Series，分区与table一致
dask_c = dd.from_pandas(c, npartitions=table.npartitions)
# 逐分区添加列，指定元数据
table = table.map_partitions(
    lambda df, s: df.assign(c=s),
    dask_c,
    meta=table._meta.append(pd.Series([], dtype='category', name='c'))
)
print(table.compute())

内容的提问来源于stack exchange，提问作者Nisba