如何使用Python计算数据框每一行文本对的成对余弦相似度分值

完整实现代码 #

首先先下载nltk所需的资源包，仅需首次运行时执行：

import nltk
nltk.download('punkt')
nltk.download('stopwords')

主逻辑代码如下：

import string
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from sklearn.metrics.pairwise import cosine_similarity

# 1. 初始化数据框
df = pd.DataFrame.from_dict({
    "q1": [
        'What is the step by step guide to invest in share market in india?',
        'What is the story of Kohinoor (Koh-i-Noor) Diamond?',
        'How can I increase the speed of my internet connection while using a VPN?',
        'Why am I mentally very lonely? How can I solve it?',
        'Which one dissolve in water quikly sugar, salt, methane and carbon di oxide?'
    ],
    "q2": [
        'What is the step by step guide to invest in share market?',
        'What would happen if the Indian government stole the Kohinoor (Koh-i-Noor) diamond back?',
        'How can Internet speed be increased by hacking through DNS?',
        'Find the remainder when [math]23^{24}[/math] is divided by 24,23?',
        'Which fish would survive in salt water?'
    ]
})

# 2. 修正后的预处理函数（原代码返回值存在变量名笔误，已修复）
def preprocessing(text):
    words = word_tokenize(text)
    tokens = [w for w in words if w.lower() not in string.punctuation]
    stopw = stopwords.words('english')
    tokens = [token for token in tokens if token not in stopw]
    preprocessed_text = ' '.join(tokens)
    return preprocessed_text

# 3. 对q1、q2列分别做文本预处理
df['pre_q1'] = df['q1'].apply(preprocessing)
df['pre_q2'] = df['q2'].apply(preprocessing)

# 4. 拟合TF-IDF向量器（用所有q1+q2的文本统一拟合，保证词表一致）
vectorizer = TfidfVectorizer()
all_text = pd.concat([df['pre_q1'], df['pre_q2']])
vectorizer.fit(all_text)

# 5. 逐行计算余弦相似度存入新列
def cal_cos_sim(row):
    vec_q1 = vectorizer.transform([row['pre_q1']])
    vec_q2 = vectorizer.transform([row['pre_q2']])
    return cosine_similarity(vec_q1, vec_q2)[0][0]

df['cos_sim_q1q2'] = df.apply(cal_cos_sim, axis=1)

# 输出结果验证
print(df[['q1', 'q2', 'cos_sim_q1q2']])

补充说明 #

• TF-IDF向量器需要在所有q1、q2文本上统一拟合词表，避免两列文本词表不一致导致相似度计算结果失真
• 最终cos_sim_q1q2列即为每对q1、q2句子的余弦相似度分值，取值范围为0到1，值越高表示语义相似度越高

内容的提问来源于stack exchange，提问作者Nishant