复现HuggingFace CLIP文本编码器输出时结果不一致问题排查

问题背景 #

尝试拆解HuggingFace的CLIP text_model，认为调用CLIP.text_model等价于以下三步：

1. 使用CLIP.text_model.embeddings计算文本嵌入；
2. 将嵌入输入至CLIP.text_model.encoder；
3. 应用CLIP.text_model.final_layer_norm。
   但对比两种方式的输出时，结果存在显著差异，代码如下：

device = "cuda" if torch.cuda.is_available() else "cpu"

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
model = model.to(device)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

def decomposed_text_model(text, processor, model, device):
    inputs = processor(text=text, return_tensors="pt", padding=True)
    attn_mask = inputs["attention_mask"].clone().detach().to(torch.bool).to(device)
    inputs = {k: v.to(device) for k, v in inputs.items()}
    
    embeddings = model.text_model.embeddings(inputs["input_ids"])
    position_embeddings=model.text_model.embeddings.position_embedding.weight[:inputs['input_ids'].shape[1]]
    embeddings = embeddings + position_embeddings.unsqueeze(0)
    encoder_output = model.text_model.encoder(
        inputs_embeds=embeddings, 
        attention_mask=attn_mask).last_hidden_state

    embeddings = model.text_model.final_layer_norm(encoder_output)

    return embeddings

def text_model(text, processor, model):
  inputs = processor(text="a photo of a cat", return_tensors="pt")
  inputs = {k: v.to(device) for k, v in inputs.items()}
  return model.text_model(**inputs)

# two step text approach
out1 = decomposed_text_model("a photo of a cat", processor, model)

out1 = out1.last_hidden_state[0, -1, :] # get eos token
out1 = out1.squeeze()

# one step text approach
out2 = text_model("a photo of a cat", processor, model)
out2 = out2.last_hidden_state[0, -1, :] # get eos token
out2 = out2.squeeze()

# compare
out1 = out1 / out1.norm(p = 2, dim=-1, keepdim=True)
out2 = out2 / out2.norm(p = 2, dim=-1, keepdim=True)

diff = torch.max(torch.abs(out1 - out2))
print(diff)

错误分析与修正 #

导致输出差异的核心问题有以下几点：

1. 重复添加位置嵌入 #

CLIP的text_model.embeddings(input_ids)内部已经完成了词嵌入+位置嵌入+token_type嵌入的叠加计算，手动再次添加position_embeddings会导致嵌入值错误翻倍，这是结果偏离的主要原因。

2. 返回值处理逻辑错误 #

decomposed_text_model返回的是经过final_layer_norm处理后的张量，后续代码错误地调用out1.last_hidden_state——该张量并不具备这个属性，直接使用返回的张量即可获取正确的hidden state。

3. Attention Mask类型错误 #

将attention_mask转为torch.bool传入编码器不符合CLIP的预期，模型内部需要的是用于区分有效token和填充token的整数掩码（0/1），手动转bool会导致注意力计算逻辑异常。

4. 硬编码输入文本 #

text_model函数内固定使用"a photo of a cat"而非传入的text参数，属于代码逻辑bug。

修正后的代码 #

device = "cuda" if torch.cuda.is_available() else "cpu"

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
model = model.to(device)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

def decomposed_text_model(text, processor, model, device):
    inputs = processor(text=text, return_tensors="pt", padding=True)
    # 直接使用原attention_mask，无需转bool
    attn_mask = inputs["attention_mask"].to(device)
    input_ids = inputs["input_ids"].to(device)
    
    # embeddings内部已包含位置嵌入，无需手动添加
    embeddings = model.text_model.embeddings(input_ids)
    
    encoder_output = model.text_model.encoder(
        inputs_embeds=embeddings, 
        attention_mask=attn_mask
    ).last_hidden_state

    # 应用final_layer_norm
    final_output = model.text_model.final_layer_norm(encoder_output)

    return final_output

def text_model(text, processor, model, device):
    inputs = processor(text=text, return_tensors="pt", padding=True)
    inputs = {k: v.to(device) for k, v in inputs.items()}
    return model.text_model(**inputs)

# 拆解方式调用
out1 = decomposed_text_model("a photo of a cat", processor, model, device)
# 直接使用返回的张量，无需取last_hidden_state
out1 = out1[0, -1, :].squeeze()

# 直接调用方式
out2 = text_model("a photo of a cat", processor, model, device)
out2 = out2.last_hidden_state[0, -1, :].squeeze()

# 归一化后对比
out1 = out1 / out1.norm(p=2, dim=-1, keepdim=True)
out2 = out2 / out2.norm(p=2, dim=-1, keepdim=True)

diff = torch.max(torch.abs(out1 - out2))
print(diff)  # 此时差异会趋近于0

验证结果 #

修正后，两次输出的差异会趋近于0，说明拆解逻辑与原text_model的调用逻辑完全一致。

内容的提问来源于stack exchange，提问作者Alfredo Ceneri