如何用HuggingFace CLIP获取同维度的文本与图像嵌入

你完全可以不用额外训练层，直接提取CLIP原生的统一维度（512）嵌入——核心是使用带投影层的模型变体，而非基础的Vision/Text模型。

CLIP的文本和视觉分支本身都内置了线性投影层，负责将各自的隐藏层输出（文本512维、图像768维）映射到共享的512维特征空间。HuggingFace提供了对应的带投影层的模型类：

• 视觉侧：TFCLIPVisionModelWithProjection
• 文本侧：TFCLIPTextModelWithProjection

这两个模型会直接输出投影后的512维嵌入，完全匹配原生CLIP的特征空间。

修改后的图像嵌入生成函数 #

def Image_Embedding_Generator(images, batch_size=32):
    model_name = "openai/clip-vit-base-patch32"
    # 替换为带投影层的视觉模型
    model = TFCLIPVisionModelWithProjection.from_pretrained(model_name)
    processor = CLIPProcessor.from_pretrained(model_name)

    if isinstance(images, (np.ndarray, tf.Tensor)):
        images = tf.unstack(images) if len(images.shape) == 4 else [images]
    elif isinstance(images, dict):
        images = [image for _, image in images.items()]

    image_projection_embeds = []
    pbar = trange(0, len(images), batch_size, desc="Generating Image Embeddings")

    for i in range(0, len(images), batch_size):
        image_batch = images[i:i+batch_size]
        inputs = processor(images=image_batch, return_tensors="tf", do_rescale=False)
        outputs = model(**inputs)
        # 提取投影后的512维嵌入
        batch_projection_embeds = outputs.image_embeds.numpy()
        image_projection_embeds.append(batch_projection_embeds)
        pbar.update(batch_size)
    
    image_projection_embeds = np.concatenate(image_projection_embeds, axis=0)
    return image_projection_embeds

修改后的文本嵌入生成函数 #

def Text_Embedding_Generator(texts, batch_size=32):
    model_name = "openai/clip-vit-base-patch32"
    # 替换为带投影层的文本模型
    model = TFCLIPTextModelWithProjection.from_pretrained(model_name)
    tokenizer = CLIPTokenizer.from_pretrained(model_name)

    if isinstance(texts, str):
        texts = [texts]
    elif isinstance(texts, dict):
        texts = [text for _, text in texts.items()]
    elif isinstance(texts, Iterable):
        texts = list(texts)

    inputs = tokenizer(text=texts, return_tensors="tf", padding="max_length", truncation=True, max_length=256)
    dataset = tf.data.Dataset.from_tensor_slices((inputs['input_ids'], inputs['attention_mask']))
    dataset = dataset.batch(batch_size).prefetch(tf.data.experimental.AUTOTUNE)

    text_projection_embeds = []
    for batch in tqdm(dataset, desc="Generating Text Embeddings"):
        batch_inputs = {'input_ids': batch[0], 'attention_mask': batch[1]}
        outputs = model(**batch_inputs)
        # 提取投影后的512维嵌入
        batch_projection_embeds = outputs.text_embeds.numpy()
        text_projection_embeds.append(batch_projection_embeds)
    
    text_projection_embeds = np.concatenate(text_projection_embeds, axis=0)
    return text_projection_embeds

补充说明 #

1. 两个修改后的函数返回的嵌入均为(batch_size, 512)形状，完全处于CLIP原生的共享特征空间，可直接用于下游图文匹配、检索等任务。
2. 若想更贴近原生CLIP的完整流程，也可直接加载TFCLIPModel，调用内置方法一步获取统一维度嵌入：

from transformers import TFCLIPModel, CLIPProcessor

model = TFCLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# 获取图像嵌入
image_inputs = processor(images=your_images, return_tensors="tf")
image_embeds = model.get_image_features(**image_inputs)

# 获取文本嵌入
text_inputs = processor(text=your_texts, return_tensors="tf", padding=True, truncation=True)
text_embeds = model.get_text_features(**text_inputs)

内容的提问来源于stack exchange，提问作者Arijit Dey