如何计算含同义词的句子间余弦相似度？附Python代码修改需求

实现思路 #

• 构建同义词映射表：将所有同义词统一映射到同一个基准词（比如把coder和programmer都映射为dev，或指定其中一个作为基准），同时处理双向映射（避免只替换单向词导致匹配遗漏）
• 文本预处理：在将文本转换为词向量前，先把文本中的所有词替换为对应的基准同义词，让同义词在词向量中表现为同一个特征
• 计算余弦相似度：使用替换后的文本生成词向量，再用原有的余弦相似度公式计算即可

修改后的代码实现 #

import math
import re
from collections import Counter

WORD = re.compile(r"\w+")    

def get_cosine(vec1, vec2):
    intersection = set(vec1.keys()) & set(vec2.keys())
    numerator = sum([vec1[x] * vec2[x] for x in intersection])

    sum1 = sum([vec1[x] ** 2 for x in list(vec1.keys())])
    sum2 = sum([vec2[x] ** 2 for x in list(vec2.keys())])
    denominator = math.sqrt(sum1) * math.sqrt(sum2)

    if not denominator:
        return 0.0
    else:
        return float(numerator) / denominator    

# 同义词替换：将文本中的词替换为基准词，统一小写避免大小写差异
def replace_synonyms(text, synonym_map):
    words = WORD.findall(text.lower())
    replaced_words = [synonym_map.get(word, word) for word in words]
    return " ".join(replaced_words)

def text_to_vector(text, synonym_map=None):
    # 若传入同义词映射表，先完成替换再生成向量
    if synonym_map:
        text = replace_synonyms(text, synonym_map)
    words = WORD.findall(text)
    return Counter(words)

# 自定义同义词映射表：多组同义词统一映射到基准词
base_synonyms = {
    "india": "india",
    "hindustan": "india",
    "usa": "usa",
    "america": "usa",
    "coder": "dev",
    "programmer": "dev"
}

# 测试同义词场景
sent1 = "You are a good coder."
sent2 = "I am new programmer"
vector1 = text_to_vector(sent1, base_synonyms)
vector2 = text_to_vector(sent2, base_synonyms)
cosine = get_cosine(vector1, vector2)
print(f"同义词场景余弦相似度: {cosine:.4f}")

# 原示例测试
text2 = "I live in India"    
sentences = [
    "India",
    "He belongs to USA", 
    "Hindustan is synonym of my country name",
    "USA and America is same",
    "You live in a great country.",
    "All countries are great to live",
]    
cosinetolist = []
for i in sentences:
    vector1 = text_to_vector(i, base_synonyms)
    vector2 = text_to_vector(text2, base_synonyms) 
    cosine = get_cosine(vector1, vector2)
    cosinetolist.append((round(cosine,4), i,))

print("\n原示例处理结果:")
for item in cosinetolist:
    print(item)

代码说明 #

1. 同义词映射：通过base_synonyms统一管理同义词，支持多对一的映射关系，同时统一转小写避免大小写干扰
2. 预处理逻辑：新增replace_synonyms函数完成文本同义词替换，确保同义词在词向量中被识别为同一特征
3. 兼容性：text_to_vector保留原功能，通过可选参数控制是否启用同义词替换，不影响原有逻辑使用

内容的提问来源于stack exchange，提问作者Murari Kumar