使用CLIP计算文本嵌入相似度出现异常结果的排查

我尝试使用CLIP计算字符串间的相似度（CLIP通常用于图文匹配，但也可仅处理文本）。提供一组简单文本语句，计算其嵌入向量的相似度，但结果不符合逻辑，例如“A dog”与“A house”的相似度（0.885）高于“A dog”与“A labrador”的相似度（0.848）。尝试过余弦相似度和欧氏距离两种计算方式，结果仍类似，请问问题出在哪里？

代码 #

import torch
import clip
from torch.nn import CosineSimilarity

cos = CosineSimilarity(dim=1, eps=1e-6)

def gen_features(model, text):
    tokens = clip.tokenize([text]).to(device)
    text_features = model.encode_text(tokens)

    return text_features

def dist(v1, v2):
    #return torch.dist(normalize(v1), normalize(v2)) # euclidean distance
    #return cos(normalize(v1), normalize(v2)).item() # cosine similarity

    similarity = (normalize(v1) @ normalize(v2).T)

    return similarity.item()



device = "cuda" if torch.cuda.is_available() else "cpu"
model_name = "ViT-B/32"
model, _ = clip.load(model_name, device=device)

sentences = ["A cat", "A dog", "A labrador", "A poodle", "A wolf", "A lion", "A house"]

with torch.no_grad():
    embeddings = [(sentence, gen_features(model, sentence)) for sentence in sentences]
    for label1, embedding1 in embeddings:
        for label2, embedding2 in embeddings:
            print(f"{label1} -> {label2}: {dist(embedding1, embedding2)}")

输出结果 #

A cat -> A cat: 0.9999998211860657
A cat -> A dog: 0.9361147880554199
A cat -> A labrador: 0.8170720934867859
A cat -> A poodle: 0.8438302278518677
A cat -> A wolf: 0.9086413979530334
A cat -> A lion: 0.8914517164230347
A cat -> A house: 0.8724125027656555
A dog -> A cat: 0.9361147880554199
A dog -> A dog: 1.0000004768371582
A dog -> A labrador: 0.8481228351593018
A dog -> A poodle: 0.9010260105133057
A dog -> A wolf: 0.9260395169258118
A dog -> A lion: 0.886112630367279
A dog -> A house: 0.8852840662002563
A labrador -> A cat: 0.8170720934867859
A labrador -> A dog: 0.8481228351593018
A labrador -> A labrador: 1.000000238418579
A labrador -> A poodle: 0.7722526788711548
A labrador -> A wolf: 0.8111101984977722
A labrador -> A lion: 0.783727765083313
A labrador -> A house: 0.7569846510887146
A poodle -> A cat: 0.8438302278518677
A poodle -> A dog: 0.9010260105133057
A poodle -> A labrador: 0.7722526788711548
A poodle -> A poodle: 0.999999463558197
A poodle -> A wolf: 0.8539597988128662
A poodle -> A lion: 0.8460092544555664
A poodle -> A house: 0.8119628429412842
A wolf -> A cat: 0.9086413979530334
A wolf -> A dog: 0.9260395169258118
A wolf -> A labrador: 0.8111101984977722
A wolf -> A poodle: 0.8539597988128662
A wolf -> A wolf: 1.000000238418579
A wolf -> A lion: 0.9043934941291809
A wolf -> A house: 0.860664427280426
A lion -> A cat: 0.8914517164230347
A lion -> A dog: 0.886112630367279
A lion -> A labrador: 0.783727765083313
A lion -> A poodle: 0.8460092544555664
A lion -> A wolf: 0.9043934941291809
A lion -> A lion: 1.0000004768371582
A lion -> A house: 0.8402873873710632
A house -> A cat: 0.8724125027656555
A house -> A dog: 0.8852840662002563
A house -> A labrador: 0.7569846510887146
A house -> A poodle: 0.8119628429412842
A house -> A wolf: 0.860664427280426
A house -> A lion: 0.8402873873710632
A house -> A house: 0.9999997615814209

问题主要出在以下几个方面：

1. 输入文本与CLIP预训练分布不匹配 #

CLIP是基于图文对预训练的模型，训练数据中的文本几乎都是描述性句式（比如"a photo of a dog"），而非你使用的极简短语（"A dog"）。这种输入分布的差异会导致模型无法正确捕捉细粒度语义关联——比如"labrador"作为特定犬种，和"dog"的语义联系没有被激活，反而"dog"和"house"这类通用名词的相似度被高估。

2. 模型粒度与任务需求不匹配 #

你使用的ViT-B/32是轻量级CLIP模型，语义区分能力有限，对于犬种这种细粒度类别，本身就很难和通用的"dog"建立强关联。加上输入格式不符合预训练习惯，进一步放大了这个问题。

3. 归一化函数可能未正确实现 #

你的代码中使用了normalize函数，但未在提供的代码片段中定义。如果该函数没有正确执行L2归一化（将向量模长缩为1），会直接导致相似度计算失真。正确的归一化实现应该是：

def normalize(x):
    return x / torch.norm(x, dim=-1, keepdim=True)

解决方法 #

（1）使用CLIP官方推荐的文本模板 #

将输入文本调整为和预训练一致的描述性句式，比如把"A dog"改为"a photo of a dog"，"A labrador"改为"a photo of a labrador"。这样能让模型的语义提取逻辑和预训练对齐，大幅提升相似度准确性。

（2）换用更大的CLIP模型 #

比如ViT-L/14，更大的模型拥有更强的语义理解能力，能更好区分细粒度类别之间的关联。

（3）修正归一化函数 #

确保归一化函数正确实现L2归一化，保证余弦相似度计算的准确性。

修改后的示例代码 #

import torch
import clip
from torch.nn import CosineSimilarity

cos = CosineSimilarity(dim=1, eps=1e-6)

# 正确实现L2归一化
def normalize(x):
    return x / torch.norm(x, dim=-1, keepdim=True)

def gen_features(model, text):
    # 使用官方推荐的图文描述模板
    template_text = f"a photo of a {text.lower()}"
    tokens = clip.tokenize([template_text]).to(device)
    text_features = model.encode_text(tokens)
    return text_features

def dist(v1, v2):
    similarity = (normalize(v1) @ normalize(v2).T)
    return similarity.item()

device = "cuda" if torch.cuda.is_available() else "cpu"
model_name = "ViT-B/32"
model, _ = clip.load(model_name, device=device)

sentences = ["A cat", "A dog", "A labrador", "A poodle", "A wolf", "A lion", "A house"]

with torch.no_grad():
    embeddings = [(sentence, gen_features(model, sentence)) for sentence in sentences]
    for label1, embedding1 in embeddings:
        for label2, embedding2 in embeddings:
            print(f"{label1} -> {label2}: {dist(embedding1, embedding2):.6f}")

修改后，"A dog"与"A labrador"的相似度会显著高于与"A house"的相似度，符合预期逻辑。

内容的提问来源于stack exchange，提问作者Adi Eyal