基于KNN构建推荐系统时遇类别样本不足的ValueError问题求助

操作步骤（中文整理） #

• 加载数据：

movies = pd.read_csv('Movielens/ml-small/movies.csv')
ratings = pd.read_csv('Movielens/ml-small/ratings.csv')

• 移除目标值"genres"的唯一值：

mv = movies[movies['genres'].duplicated(keep=False)]

• 合并数据：

moviesdf = ratings.merge(mv, on='movieId')
moviesdf = moviesdf[['userId', 'movieId',
                     'title', 'genres', 'rating', 'ratedate']]

• 分配特征与目标：

y = moviesdf.genres
X = moviesdf.drop(['genres', 'title'], axis=1)

• 编码特征与目标：

from sklearn.preprocessing import OrdinalEncoder
encoder = OrdinalEncoder()
X[["ratedate"]] = encoder.fit_transform(X[["ratedate"]])
print(type(X))

from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
y_encoded = encoder.fit_transform(y) 
print(type(y_encoded))
y_encoded

• 数据集划分（触发错误的步骤）：

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y_encoded, random_state = 100, test_size = 0.25, stratify = y)
print('X_train_shape: ' + str(X_train.shape) + '\nX_test_shape: ' + str(X_test.shape)\
       + '\ny_train_shape: ' + str(y_train.shape) + '\ny_test_shape: ' + str(y_test.shape))

错误信息 #

ValueError: The least populated class in y has only 1 member, which is too few. The minimum number of groups for any class cannot be less than 2.

错误原因 #

使用stratify=y进行分层抽样时，要求每个类别至少有2个样本，才能保证训练集和测试集中都包含该类别的数据。当前y（即genres字段）中存在部分类别仅对应1条样本，无法满足分层抽样的要求。

解决方法 #

1. 过滤小样本类别 #

先统计每个genres的样本量，移除样本数不足2的类别后再划分数据集：

# 统计各类型的样本数量
genre_counts = moviesdf['genres'].value_counts()
# 筛选出样本数≥2的类型
valid_genres = genre_counts[genre_counts >= 2].index
moviesdf_filtered = moviesdf[moviesdf['genres'].isin(valid_genres)]

# 重新定义特征与目标
y_filtered = moviesdf_filtered.genres
X_filtered = moviesdf_filtered.drop(['genres', 'title'], axis=1)

# 重新编码特征与目标
encoder_X = OrdinalEncoder()
X_filtered[["ratedate"]] = encoder_X.fit_transform(X_filtered[["ratedate"]])

encoder_y = LabelEncoder()
y_encoded_filtered = encoder_y.fit_transform(y_filtered)

# 重新划分数据集
X_train, X_test, y_train, y_test = train_test_split(X_filtered, y_encoded_filtered, random_state=100, test_size=0.25, stratify=y_filtered)
print('X_train_shape: ' + str(X_train.shape) + '\nX_test_shape: ' + str(X_test.shape)\
       + '\ny_train_shape: ' + str(y_train.shape) + '\ny_test_shape: ' + str(y_test.shape))

2. 取消分层抽样 #

如果不需要严格保持训练集与测试集的类别分布一致，可以直接移除stratify参数，采用随机划分：

X_train, X_test, y_train, y_test = train_test_split(X, y_encoded, random_state = 100, test_size = 0.25)

注意：这种方式可能导致部分类别仅出现在训练集或测试集中，会影响模型评估的准确性。

3. 小样本类别扩充（进阶） #

若不想丢弃小样本类别，可以通过过采样（如SMOTE算法）对样本数少的类别进行数据扩充，再进行分层抽样。不过推荐系统场景中，冷门类型的样本对整体模型影响有限，方法1是更高效的选择。

内容的提问来源于stack exchange，提问作者Inoue