使用BERT-base-uncased计算余弦相似度时准确率为0的问题排查

核心错误点及修复 #

1. 问题与答案样本无对应关系 #

你分别对问题和答案做了随机采样：

sample_1000_quest = train_ds['questions'].sample(1000)
sample_1000_answer = train_ds['answers'].sample(1000)

这会导致采样的1000个问题和1000个答案完全随机配对，目标问题的真实答案大概率不在采样的答案集合里，准确率必然为0。

修复方式：
按行采样完整数据，保证每个问题对应其真实答案：

# 随机采样1000条完整数据（问题+答案一一对应）
sample_ds = train_ds.sample(1000)
sample_1000_quest = sample_ds['questions']
sample_1000_answer = sample_ds['answers']

2. 真实答案索引逻辑完全错误 #

原代码中：

ground_truth_idx = train_ds['answers'].iloc[selected_question_idx]

你把原数据集的答案内容当成了采样后答案列表的索引，逻辑完全混乱。正确逻辑是找到目标问题对应的真实答案，再定位它在采样答案集合中的位置索引。

修复方式：

# 获取目标问题对应的真实答案内容
ground_truth_answer = sample_ds['answers'].iloc[1]  # 对应selected_question = sample_1000_quest.iloc[1]
# 找到该答案在采样答案列表中的位置索引
ground_truth_pos = sample_1000_answer.index.get_loc(sample_ds.index[1])

后续判断准确率时用ground_truth_pos与most_similar_indices对比。

3. 设备不一致导致计算异常 #

原代码中，问题的embedding在CPU计算，答案的embedding在GPU计算，两个张量设备不匹配会导致余弦相似度计算结果异常。

修复方式：
统一将模型和所有张量放到同一设备：

# 模型移到指定设备
model_bert = model_bert.to(device)
# 问题编码也移到设备
encoded_question = tokenizer_bert(selected_question, return_tensors='pt', padding=True, truncation=True).to(device)
with torch.no_grad():
    outputs = model_bert(**encoded_question)
    question_embedding = outputs.last_hidden_state.mean(dim=1)

4. 模型与文本语言不匹配 #

你使用bert-base-uncased（英文预训练模型）处理土耳其语文本，该模型未学习过土耳其语语义，生成的embedding无法正确表示文本含义，导致匹配结果完全不相关。

修复方式：
使用土耳其语预训练BERT模型，比如dbmdz/bert-base-turkish-uncased：

tokenizer_bert = BertTokenizer.from_pretrained('dbmdz/bert-base-turkish-uncased')
model_bert = BertModel.from_pretrained('dbmdz/bert-base-turkish-uncased', output_hidden_states=True).eval().to(device)

修正后的完整代码示例 #

# 按行采样保证问题答案对应
sample_ds = train_ds.sample(1000)
sample_1000_quest = sample_ds['questions']
sample_1000_answer = sample_ds['answers']

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# 使用土耳其语BERT模型
tokenizer_bert = BertTokenizer.from_pretrained('dbmdz/bert-base-turkish-uncased')
model_bert = BertModel.from_pretrained('dbmdz/bert-base-turkish-uncased', output_hidden_states=True).eval().to(device)

selected_question = sample_1000_quest.iloc[1]
# 获取对应真实答案及位置索引
ground_truth_answer = sample_ds['answers'].iloc[1]
ground_truth_pos = sample_1000_answer.index.get_loc(sample_ds.index[1])

# 编码问题并计算embedding
encoded_question = tokenizer_bert(selected_question, return_tensors='pt', padding=True, truncation=True).to(device)
with torch.no_grad():
    outputs = model_bert(**encoded_question)
    question_embedding = outputs.last_hidden_state.mean(dim=1)

# 编码所有答案并计算embedding
answer_embeddings = []
for answer in sample_1000_answer:
    encoded_answer = tokenizer_bert(answer, return_tensors='pt', padding=True, truncation=True).to(device)
    with torch.no_grad():
        outputs = model_bert(**encoded_answer)
        answer_embedding = outputs.last_hidden_state.mean(dim=1)
        answer_embeddings.append(answer_embedding)

# 计算余弦相似度
similarities = []
for answer_embedding in answer_embeddings:
    similarity = cosine_similarity(question_embedding, answer_embedding)
    similarities.append(similarity.item())

# 获取Top5相似答案的位置索引
most_similar_indices = np.argsort(similarities)[-5:][::-1]

# 计算准确率
top1_accuracy = 1 if most_similar_indices[0] == ground_truth_pos else 0
top5_accuracy = 1 if ground_truth_pos in most_similar_indices else 0

# 输出结果
print("Selected Question:", selected_question)
print("Ground Truth Answer:", ground_truth_answer)
print("Most similar 5 answers:")
for i, idx in enumerate(most_similar_indices):
    print(f"{i+1}. {sample_1000_answer.iloc[idx]}")

print("Top-1 Accuracy:", top1_accuracy)
print("Top-5 Accuracy:", top5_accuracy)

内容的提问来源于stack exchange，提问作者Zephyrus