Gower聚类后基于预计算矩阵的K-medoids聚类报错求助

我需要对包含二进制和非二进制的混合数据执行Gower距离计算，再基于预计算的距离矩阵dm进行K-medoids聚类，但运行代码时触发了ValueError。以下是我的代码、报错信息及数据样例，请求排查问题。

原代码 #

import gower
from sklearn_extra.cluster import KMedoids

dft = df.T
X = dft.iloc[:-5,:]
y = dft.iloc[-5:,:]

mms = MinMaxScaler()
mms.fit(X)
data_transformed = mms.transform(X)

dm = gower_matrix(X, y)

K = range(1, 10)
for k in K:
    kmedoids = KMedoids(n_clusters=k, metric="precomputed", method="pam").fit(dm, y)
    distortions.append(sum(np.min(cdist(dm, kmedoids.cluster_centers_,'euclidean'), axis=1)) / dm.shape[0])
    inertias.append(kmedoids.inertia_)

报错追踪信息 #

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [33], in <cell line: 1>()
      1 for k in K:
----> 3     kmedoids = KMedoids(n_clusters=k, metric="precomputed", method="pam").fit(dm, y)
      5     distortions.append(sum(np.min(cdist(dm, kmedoids.cluster_centers_,'euclidean'), axis=1)) / dm.shape[0])
      6     inertias.append(kmedoids.inertia_)

File ~/.local/lib/python3.9/site-packages/sklearn_extra/cluster/_k_medoids.py:196, in KMedoids.fit(self, X, y)
    189 if self.n_clusters > X.shape[0]:
    190     raise ValueError(
    191         "The number of medoids (%d) must be less "
    192         "than the number of samples %d."
    193         % (self.n_clusters, X.shape[0])
    194     )
--> 196 D = pairwise_distances(X, metric=self.metric)
    197 medoid_idxs = self._initialize_medoids(
    198     D, self.n_clusters, random_state_
    199 )
    200 labels = None

File /scg/apps/software/jupyter/python_3.9/lib/python3.9/site-packages/sklearn/metrics/pairwise.py:1851, in pairwise_distances(X, Y, metric, n_jobs, force_all_finite, **kwds)
   1845     raise ValueError(
   1846         "Unknown metric %s. Valid metrics are %s, or 'precomputed', or a callable"
   1847         % (metric, _VALID_METRICS)
   1848     )
   1850 if metric == "precomputed":
--> 1851     X, _ = check_pairwise_arrays(
   1852         X, Y, precomputed=True, force_all_finite=force_all_finite
   1853     )
   1855     whom = (
   1856         "`pairwise_distances`. Precomputed distance "
   1857         " need to have non-negative values."
   1858     )
   1859     check_non_negative(X, whom=whom)

File /scg/apps/software/jupyter/python_3.9/lib/python3.9/site-packages/sklearn/metrics/pairwise.py:175, in check_pairwise_arrays(X, Y, precomputed, dtype, accept_sparse, force_all_finite, copy)
    173 if precomputed:
    174     if X.shape[1] != Y.shape[0]:
--> 175         raise ValueError(
    176             "Precomputed metric requires shape "
    177             "(n_queries, n_indexed). Got (%d, %d) "
    178             "for %d indexed." % (X.shape[0], X.shape[1], Y.shape[0])
    179         )
    180 elif X.shape[1] != Y.shape[1]:
    181     raise ValueError(
    182         "Incompatible dimension for X and Y matrices: "
    183         "X.shape[1] == %d while Y.shape[1] == %d" % (X.shape[1], Y.shape[1])
    184     )

ValueError: Precomputed metric requires shape (n_queries, n_indexed). Got (202, 5) for 202 indexed.

数据样例 #

pd.DataFrame({'TCGA-2K-A9WE-01A': {'IGF2R': 0,
  'NBEA': 0,
  ...
  'hsa-miR-30c-5p': 3.3507475510336118}}
)

问题根源 #

1. 距离矩阵形状错误：当KMedoids设置metric="precomputed"时，要求输入的是样本间的两两距离矩阵，形状必须为(n_samples, n_samples)。但你用gower_matrix(X, y)生成的是X（202个样本）到y（5个样本）的交叉距离矩阵，形状(202,5)完全不符合要求。
2. fit参数错误：无监督聚类的fit方法不需要传入y参数，你传入的y会被当作监督学习的标签，进而触发形状校验错误。
3. 冗余数据缩放：Gower距离本身支持混合类型数据，会自动处理数值型特征的归一化，提前用MinMaxScaler属于冗余操作。

修正步骤 #

1. 生成正确的Gower两两距离矩阵 #

如果要对X中的样本聚类，需计算X内部所有样本间的Gower距离：

dm = gower.gower_matrix(X)  # 生成形状为(202,202)的两两距离矩阵

2. 修正KMedoids调用逻辑 #

去掉多余的y参数，同时修正distortions的计算逻辑（预计算距离矩阵下，cluster_centers_是样本索引，无需再用cdist计算）：

K = range(1, 10)
distortions = []
inertias = []
for k in K:
    kmedoids = KMedoids(n_clusters=k, metric="precomputed", method="pam").fit(dm)
    distortions.append(kmedoids.inertia_ / dm.shape[0])
    inertias.append(kmedoids.inertia_)

3. 删除冗余的缩放代码 #

直接去掉MinMaxScaler相关的代码，不影响Gower距离计算。

完整修正代码 #

import gower
from sklearn_extra.cluster import KMedoids

dft = df.T
X = dft.iloc[:-5,:]

# 生成X内部的两两Gower距离矩阵
dm = gower.gower_matrix(X)

K = range(1, 10)
distortions = []
inertias = []
for k in K:
    kmedoids = KMedoids(n_clusters=k, metric="precomputed", method="pam").fit(dm)
    distortions.append(kmedoids.inertia_ / dm.shape[0])
    inertias.append(kmedoids.inertia_)

内容的提问来源于stack exchange，提问作者Anon