拟合NearestNeighbors时出现数组元素赋值序列ValueError

现有结构如下的PySpark DataFrame，包含id与features两个字段，features字段为等长浮点数值组成的特征向量列表：

+------+---------------------------------------------------------------------+
|id    |features                                                             |
+------+---------------------------------------------------------------------+
|2484  |[0.016910851, 0.025989642, 0.0025321299, -0.022232508, -0.00701562]  |
|2504  |[0.015019539, 0.024844216, 0.0029279909, -0.020771071, -0.0061111804]|
|2904  |[0.014104126, 0.02474243, 0.0011707658, -0.021675153, -0.0050868453] |
|3084  |[0.110674664, 0.17139696, 0.059836507, -0.1926481, -0.060425207]     |
|3164  |[0.17688861, 0.2159168, 0.10567094, -0.17365277, -0.016458606]       |
|377784|[0.18425785, 0.34397766, 0.022859085, -0.35151178, -0.07897296]      |
|425114|[0.14556459, 0.25762737, 0.09011796, -0.27128243, 0.011280057]       |
|455074|[0.13579306, 0.3266111, 0.016416805, -0.31139722, -0.054227617]      |
|532624|[0.22281846, 0.1575731, 0.14126688, -0.29887098, -0.09433056]        |
|781654|[0.1381407, 0.14674455, 0.06877328, -0.13415968, -0.06589967]        |
+------+---------------------------------------------------------------------+

基于上述特征执行最近邻检索的初始代码如下：

df_collect = df.toPandas()
#converting list column to array
df_collect['features'] = df_collect['features'].apply(lambda x: np.array(x))
features = df_collect['features'].to_numpy()

knnobj = NearestNeighbors(n_neighbors=5, algorithm='auto').fit(features)

执行fit拟合操作时触发如下报错：

TypeError                                 Traceback (most recent call last)
TypeError: only size-1 arrays can be converted to Python scalars

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
/tmp/ipykernel_6511/1498389666.py in <module>
----> 1 knnobj = NearestNeighbors(n_neighbors=5, algorithm='auto').fit(features)

~/miniconda3/envs/dev_env_37/lib/python3.7/site-packages/sklearn/neighbors/_unsupervised.py in fit(self, X, y)
    164             The fitted nearest neighbors estimator.
    165         """
--> 166         return self._fit(X)

~/miniconda3/envs/dev_env_37/lib/python3.7/site-packages/sklearn/neighbors/_base.py in _fit(self, X, y)
    433         else:
    434             if not isinstance(X, (KDTree, BallTree, NeighborsBase)):
--> 435                 X = self._validate_data(X, accept_sparse="csr")
    436 
    437         self._check_algorithm_metric()

~/miniconda3/envs/dev_env_37/lib/python3.7/site-packages/sklearn/base.py in _validate_data(self, X, y, reset, validate_separately, **check_params)
    564             raise ValueError("Validation should be done on X, y or both.")
    565         elif not no_val_X and no_val_y:
--> 566             X = check_array(X, **check_params)
    567             out = X
    568         elif no_val_X and not no_val_y:

~/miniconda3/envs/dev_env_37/lib/python3.7/site-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
    744                     array = array.astype(dtype, casting="unsafe", copy=False)
    745                 else:
--> 746                     array = np.asarray(array, order=order, dtype=dtype)
    747             except ComplexWarning as complex_warning:
    748                 raise ValueError(

ValueError: setting an array element with a sequence.

已完成排查：所有子数组的长度、数据类型完全一致，打印features变量输出如下：

array([array([ 0.01691085,  0.02598964,  0.00253213, -0.02223251, -0.00701562]),
       array([ 0.01501954,  0.02484422,  0.00292799, -0.02077107, -0.00611118]),
       array([ 0.01410413,  0.02474243,  0.00117077, -0.02167515, -0.00508685]),
       ...,
       array([ 0.01896316,  0.03188267,  0.00258667, -0.02800867, -0.00646481]),
       array([ 0.03538242,  0.07453772,  0.00816828, -0.02914227, -0.0942148 ]),
       array([ 0.02470775,  0.02561068,  0.00401011, -0.02863882, -0.00419102])],
      dtype=object)

scikit-learn的NearestNeighbors.fit()要求输入必须是形状为(n_samples, n_features)的二维数值类型numpy数组，但当前生成的features是dtype=object的一维数组，每个元素是单独的一维numpy数组，属于数组嵌套结构。
即使所有子数组长度、类型完全一致，numpy也无法直接将这种object类型的嵌套数组自动转换为二维浮点数组，因此在sklearn内部调用np.asarray做格式校验时抛出错误。

只需要将嵌套的一维数组堆叠为标准二维数值数组即可，核心修改是用np.stack()处理数组列，修正后的代码如下：

import numpy as np
from sklearn.neighbors import NearestNeighbors

df_collect = df.toPandas()
# 将列表列转换为指定类型的numpy数组
df_collect['features'] = df_collect['features'].apply(lambda x: np.array(x, dtype=np.float64))
# 堆叠为形状(样本数, 特征维度)的二维数值数组
features = np.stack(df_collect['features'].values)

# 可选：验证数组格式
# print(features.shape, features.dtype)
knnobj = NearestNeighbors(n_neighbors=5, algorithm='auto').fit(features)

如果数据量较大，toPandas()会把全量数据拉取到Driver节点容易触发内存溢出，建议直接使用PySpark MLlib自带的最近邻实现，避免单节点内存瓶颈。

内容的提问来源于stack exchange，提问作者Chris_007