基于混合协同过滤与句子相似度的医生推荐系统报错排查

问题背景 #

开发一款结合协同过滤与句子相似度的医生推荐系统，根据用户输入的症状和位置推荐医生，执行流程包含缺失值处理、特征编码归一化、BERT生成文本嵌入、协同过滤矩阵构建、相似度计算、模型得分融合及效果评估。运行代码时触发KeyError，错误显示大量索引值不存在于矩阵中。

错误原因分析 #

1. 协同过滤矩阵索引不匹配
   协同过滤矩阵user_item_matrix由训练集X_train生成，其行/列索引仅包含训练集中的specialization_encoded和location_encoded值。但推荐逻辑中similar_doctors来自全量数据集df，可能包含测试集或训练集中未出现的编码值，调用matrix.loc[doctor_specialization, doctor_location]时会因索引不存在触发KeyError。

2. 随机森林特征获取失败
   embeddings_df是单独存储BERT嵌入的DataFrame，未合并到主数据集df中。在推荐函数中尝试从row（来自df）获取embeddings_df的列时，这些列根本不存在，直接触发KeyError。

3. LabelEncoder复用冲突
   同一个LabelEncoder实例同时用于编码specialization和location，会导致两类特征的编码值重叠，后续索引匹配时逻辑混乱。

修复方案 #

1. 拆分LabelEncoder实例
   为specialization和location分别创建独立的LabelEncoder，避免编码冲突。

2. 合并BERT嵌入到主数据集
   将embeddings_df合并到df中，确保推荐时能直接从医生行数据中获取嵌入特征。

3. 处理协同过滤矩阵索引缺失
   改用全量数据集生成协同过滤矩阵，或者在获取CF得分时添加异常处理，对不存在的索引返回默认得分（如0）。这里选择全量生成矩阵，保证所有医生的编码都能匹配。

4. 调整训练集特征选择逻辑
   简化训练集特征选择，直接使用合并后的df中的特征列，避免列名匹配错误。

完整修正代码 #

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score
from transformers import BertTokenizer, BertModel
import torch
from sklearn.metrics.pairwise import cosine_similarity

# Load the dataset
file_path = "Egyptian Doctors.csv"
df = pd.read_csv(file_path)

# Data Preprocessing
# Handle missing values
imputer = SimpleImputer(strategy='median')
df['avg_rate'] = imputer.fit_transform(df[['avg_rate']])
df['Wait_time_Minutes'] = imputer.fit_transform(df[['Wait_time_Minutes']])
df['doctor_visitors'] = imputer.fit_transform(df[['doctor_visitors']])

# Encode categorical features with separate LabelEncoders
spec_encoder = LabelEncoder()
df['specialization_encoded'] = spec_encoder.fit_transform(df['specialization'])

loc_encoder = LabelEncoder()
df['location_encoded'] = loc_encoder.fit_transform(df['location'])

# Normalize numerical features
scaler = StandardScaler()
numerical_features = ['avg_rate', 'fees.1', 'doctors_views', 'doctor_visitors', 'Wait_time_Minutes']
df[numerical_features] = scaler.fit_transform(df[numerical_features])

# Generate BERT embeddings for text features
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

def get_bert_embeddings(text_list):
    text_list = [str(text) for text in text_list]  # Convert all elements to strings
    inputs = tokenizer(text_list, return_tensors='pt', padding=True, truncation=True, max_length=128)
    with torch.no_grad():
        outputs = model(**inputs)
    return outputs.last_hidden_state[:, 0, :].numpy()  # Use [CLS] token embeddings

# Combine specialization and location for BERT embeddings
df['text_features'] = df['specialization'] + " " + df['location']
df['symptoms'] = df['specialization']  # Assuming symptoms are similar to specialization for this example

# Generate BERT embeddings and merge into main df
embeddings = get_bert_embeddings(df['text_features'].tolist())
embeddings_df = pd.DataFrame(embeddings, index=df.index, columns=[f'bert_emb_{i}' for i in range(embeddings.shape[1])])
df = pd.concat([df, embeddings_df], axis=1)

# Split the data into train and test sets
X = df.drop(columns=['text_features', 'specialization', 'location', 'symptoms'])
X.columns = X.columns.astype(str)
y = df['avg_rate']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Select features for RandomForest (all except target)
X_train_ml = X_train.drop(columns=['avg_rate'])
X_test_ml = X_test.drop(columns=['avg_rate'])

# Impute missing values
imputer = SimpleImputer(strategy='median')
X_train_ml = imputer.fit_transform(X_train_ml)
X_test_ml = imputer.transform(X_test_ml)

# Train the RandomForestRegressor
model_rf = RandomForestRegressor(n_estimators=100, random_state=42)
model_rf.fit(X_train_ml, y_train)

# Collaborative Filtering: 使用全量数据生成矩阵，避免索引缺失
user_item_matrix = df.pivot_table(index='specialization_encoded', columns='location_encoded', values='avg_rate', fill_value=0)

def hybrid_recommendation(user_symptoms, user_location, matrix, df, model_rf):
    # Encode user location to match dataset encoding
    try:
        user_location_encoded = loc_encoder.transform([user_location])[0]
    except ValueError:
        user_location_encoded = 0
    
    # Generate user embedding
    user_embedding = get_bert_embeddings([user_symptoms + " " + user_location])[0]
    
    # Calculate cosine similarity with all doctors
    bert_emb_cols = [col for col in df.columns if col.startswith('bert_emb_')]
    doctor_embeddings = df[bert_emb_cols].values
    cosine_similarities = cosine_similarity([user_embedding], doctor_embeddings)[0]
    df['similarity'] = cosine_similarities

    # Get top 10 similar doctors
    similar_doctors = df.nlargest(10, 'similarity')

    recommendations = []

    for _, row in similar_doctors.iterrows():
        doctor_id = row.name
        doctor_specialization = row['specialization_encoded']
        doctor_location = row['location_encoded']
        
        # Get Collaborative Filtering Score (处理索引不存在的情况)
        try:
            cf_score = matrix.loc[doctor_specialization, doctor_location]
        except KeyError:
            cf_score = 0.0
        
        # Get RandomForest Score
        rf_features = row[X_train.drop(columns=['avg_rate']).columns].values.reshape(1, -1)
        rf_score = model_rf.predict(rf_features)[0]

        # Combine scores
        final_score = (cf_score + rf_score) / 2

        recommendations.append((doctor_id, final_score))

    # Sort by final score
    recommendations = sorted(recommendations, key=lambda x: x[1], reverse=True)

    return recommendations

# User input for recommendations
user_symptoms = "Skin Rash"
user_location = "El-Mansoura"
recommendations = hybrid_recommendation(user_symptoms, user_location, user_item_matrix, df, model_rf)

# Print results
print("Top 5 Recommendations:")
for idx, (doc_id, score) in enumerate(recommendations[:5], 1):
    doctor_info = df.loc[doc_id]
    print(f"{idx}. Doctor ID: {doc_id}, Specialization: {doctor_info['specialization']}, Location: {doctor_info['location']}, Final Score: {score:.4f}")

验证说明 #

• 修复了LabelEncoder复用问题，确保两类特征编码独立。
• 合并BERT嵌入到主数据集，解决特征获取时的KeyError。
• 改用全量数据生成协同过滤矩阵，并添加异常处理，避免索引缺失导致的错误。
• 简化了特征选择逻辑，直接使用训练集中的非目标列，避免列名匹配错误。

内容的提问来源于stack exchange，提问作者Sadura Akinrinwa