KMeans聚类[0,1]特征向量质心越界问题排查求助

问题描述 #

我有一个基于购买月份的聚类数据集，所有特征向量的元素和为1。我并未要求质心严格满足元素和为1，但预期质心元素应处于[0,1]区间内。使用Sklearn的KMeans代码运行后得到了超出该范围的质心，想问我的操作是否存在异常，已在Sklearn提交相关issue。

算法代码 #

import pandas as pd
from os import getcwd
from sklearn.model_selection import train_test_split
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score


this_path = getcwd() + ''
filename = 'product_data.csv'

file_path = this_path + '/' + filename

df = pd.read_csv(file_path)

X = [list(group_df['months']) for _, group_df in df.groupby('product_id')]

# Assuming 'X' is your feature matrix

# Now, we have the following datasets:
# - X_train: Training features
# - X_val: Validation features
# - X_test: Test features
# - y_train: Training target (if applicable)
# - y_val: Validation target (if applicable)
# - y_test: Test target (if applicable)

test_size = 0.2
val_size = 0.2
train_size = 1-test_size

# Step 1: Train-test split
X_train, X_test = train_test_split(X, test_size=test_size, random_state=42)

# Step 2: Train-test split
X_train, X_val = train_test_split(X, test_size=val_size*train_size, random_state=42)

# Step 4: Model training
# Example with 3 clusters
kmeans = KMeans(n_clusters=3, n_init=10, random_state=42)  
kmeans.fit(X_train)

# Step 5: Model evaluation
train_silhouette_score = silhouette_score(X_train, kmeans.labels_)
test_silhouette_score = silhouette_score(X_test, kmeans.predict(X_test))
print(f"Train Silhouette Score: {train_silhouette_score}")
print(f"Test Silhouette Score: {test_silhouette_score}")

输出结果 #

Train Silhouette Score: 0.9338078873850575
Test Silhouette Score: 0.9395121235047589
[[4.82540373e-02 5.09614267e-02 5.32134245e-02 1.03824456e-01
  2.59398992e-01 2.38517675e-01 1.55693519e-01 1.44091717e-01
  6.28019827e-01 9.73606697e-01 1.16770379e+00 7.88320938e-01]
 [1.70372198e+00 2.09958955e+00 2.44583794e+00 2.37549850e+00
  5.03812743e+00 2.22023340e+01 4.85880994e+01 8.07960210e+01
  1.06434009e+02 1.07835291e+02 5.99978649e+01 1.34812973e+01]
 [2.40384615e-01 7.21153846e-01 4.81250000e-01 2.42427885e-01
  9.65685096e-01 2.40931490e+00 2.50085938e+01 7.47704928e+01
  1.04577344e+02 1.04365685e+02 5.25717548e+01 3.85576923e+02]]

核心问题与修正方案 #

1. 质心超出[0,1]的根本原因 #

你的特征向量实际并非处于[0,1]区间，核心是数据处理环节出错：

• 代码中X = [list(group_df['months']) for _, group_df in df.groupby('product_id')]直接将每个产品对应的购买月份原始值（比如购买次数）拼接成样本，而非你预期的归一化比例值。
• 若原始数据是购买次数，样本元素会远大于1，KMeans的质心是样本均值，自然会超出[0,1]区间，和你所说的“特征向量元素和为1”完全不符。

2. 数据处理修正 #

需要将每个产品的月度购买数据转换为比例值，确保每个样本的元素和为1且每个元素在[0,1]之间：

# 替换原X的生成代码
# 假设df包含product_id、month（月份标识）、count（购买次数）列
# 先透视得到每个产品各月的购买次数
pivot_df = df.pivot(index='product_id', columns='month', values='count').fillna(0)
# 归一化得到比例
X = pivot_df.div(pivot_df.sum(axis=1), axis=0).values

3. 数据集拆分错误 #

代码中验证集拆分逻辑存在问题：

• 第二步拆分时，你直接对原始X进行拆分，而非从第一步得到的X_train中拆分，这会导致数据泄露，且训练集、验证集的划分比例不符合预期。
• 修正后的拆分代码：

# Step 1: 拆分训练集+验证集 和 测试集
X_train_val, X_test = train_test_split(X, test_size=test_size, random_state=42)
# Step 2: 从训练集+验证集中拆分训练集和验证集
X_train, X_val = train_test_split(X_train_val, test_size=val_size/(1-test_size), random_state=42)

4. 关于KMeans的特性 #

KMeans本身不约束质心的取值范围，它仅通过最小化样本到质心的欧氏距离更新质心。只要输入样本的元素在[0,1]，质心必然也在[0,1]区间内，这不是Sklearn的bug，而是数据处理逻辑的问题。

内容的提问来源于stack exchange，提问作者Bruno Peixoto