使用pandas与BERT实现两DataFrame间余弦相似度循环计算方法

解决代码 #

你不需要手动逐行调用cosine_similarity，直接一次性计算全量相似度矩阵后遍历每一行即可，完整可运行代码如下：

import pandas as pd
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

# 示例数据 你可以替换成自己的实际数据源
df = pd.DataFrame({'Element Detail':['Too many competitors in market', 'Highly skilled employees']})
df1 = pd.DataFrame({'Element Details':['Our workers have a lot of talent', 
                                      'this too is a sentence',
                                      'this is very different',
                                      'another sentence is this',
                                      'not much of anything']
                    })

# 模型加载与句向量编码
model_name = 'bert-base-nli-mean-tokens'
model = SentenceTransformer(model_name)
sentence_vecs = model.encode(df['Element Detail'])
sentence_vecs1 = model.encode(df1['Element Details'])

# 计算全量相似度矩阵：每一行对应df的一个句子，每一列对应df1的一个句子
similarity_matrix = cosine_similarity(sentence_vecs, sentence_vecs1)

# 遍历每个df的句子，提取最高相似度匹配结果
result = []
for i in range(len(similarity_matrix)):
    current_scores = similarity_matrix[i]
    max_score = current_scores.max()
    # 匹配的df1序号，按照你原来的规则ID从1开始计数
    match_id = current_scores.argmax() + 1
    result.append([match_id, max_score])

# 构造和你原有格式完全一致的Final DataFrame
Final = pd.DataFrame(result, columns=['XNew_ID', 'X0'])

输出说明 #

运行后Final会包含df所有句子的匹配结果，每一行对应df的一个句子的最高相似度匹配，示例输出如下：

XNew_ID        X0
0        2  0.527691
1        1  0.791345

内容的提问来源于stack exchange，提问作者Conor C