如何为两款微调后的BERT模型绘制隐藏状态的T-SNE图？

1. 安装依赖库 #

打开终端执行以下命令，安装所需Python库：

pip install transformers torch scikit-learn matplotlib seaborn pandas

2. 加载模型与准备数据 #

加载两个模型的Tokenizer和基础模型（用AutoModel而非分类模型，目的是提取隐藏状态），同时准备一批带分类标签的文本数据（建议用模型微调时的验证集，保证标签匹配）：

from transformers import AutoTokenizer, AutoModel
import torch

# 模型标识与对应名称
model_configs = [
    ("samanjoy2/banglaclickbert_finetuned_sequence_classification_clickbait", "BanglaClickBert"),
    ("samanjoy2/banglabert_finetuned_sequence_classification_clickbait", "BanglaBert")
]

tokenizers = []
models = []
model_names = []

for model_path, name in model_configs:
    tokenizer = AutoTokenizer.from_pretrained(model_path)
    model = AutoModel.from_pretrained(model_path)
    model.eval()  # 切换至评估模式，禁用梯度计算
    tokenizers.append(tokenizer)
    models.append(model)
    model_names.append(name)

# 替换为你的实际文本数据与分类标签（1=点击诱饵，0=非点击诱饵）
sample_texts = [
    "আপনার জন্য সেরা টিপস!",
    "বাংলাদেশের নতুন প্রজেক্টের বিস্তারিত",
    "কিভাবে একদিনে ১০ হাজার টাকা ইনকাম করবেন?",
    "শিক্ষার্থীদের জন্য নতুন স্কলারশিপ ঘোষণা"
]
sample_class_labels = [1, 0, 1, 0]

3. 提取隐藏状态 #

编写函数提取每个文本的<[BOS_never_used_51bce0c785ca2f68081bfa7d91973934]>token最后一层隐藏状态（作为句子表征）：

def get_hidden_states(texts, tokenizer, model):
    hidden_states = []
    with torch.no_grad():
        for text in texts:
            inputs = tokenizer(
                text,
                padding="max_length",
                truncation=True,
                max_length=128,
                return_tensors="pt"
            )
            outputs = model(**inputs)
            # 取<[BOS_never_used_51bce0c785ca2f68081bfa7d91973934]>token的最后一层隐藏状态，转换为numpy数组
            cls_state = outputs.last_hidden_state[:, 0, :].squeeze().numpy()
            hidden_states.append(cls_state)
    return hidden_states

# 收集所有模型的隐藏状态与对应标签
all_hidden = []
all_model_tags = []
all_class_tags = []

for idx, (tokenizer, model) in enumerate(zip(tokenizers, models)):
    states = get_hidden_states(sample_texts, tokenizer, model)
    all_hidden.extend(states)
    all_model_tags.extend([model_names[idx]] * len(states))
    all_class_tags.extend(sample_class_labels)

4. T-SNE降维 #

用scikit-learn将高维隐藏状态降至2维：

from sklearn.manifold import TSNE
import numpy as np

hidden_array = np.array(all_hidden)
# 初始化T-SNE，perplexity可根据样本数量调整（建议5-50）
tsne = TSNE(n_components=2, random_state=42, perplexity=5)
tsne_results = tsne.fit_transform(hidden_array)

5. 绘制可视化图 #

用seaborn和matplotlib绘制区分模型与分类标签的T-SNE图：

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# 整理数据为DataFrame
tsne_df = pd.DataFrame({
    "TSNE_1": tsne_results[:, 0],
    "TSNE_2": tsne_results[:, 1],
    "模型": all_model_tags,
    "分类": ["点击诱饵" if tag == 1 else "非点击诱饵" for tag in all_class_tags]
})

# 绘图
plt.figure(figsize=(10, 8))
sns.scatterplot(
    data=tsne_df,
    x="TSNE_1",
    y="TSNE_2",
    hue="模型",
    style="分类",
    palette="bright",
    s=80,
    alpha=0.8
)
plt.title("BanglaClickBert与BanglaBert隐藏状态T-SNE可视化")
plt.xlabel("TSNE维度1")
plt.ylabel("TSNE维度2")
plt.legend(bbox_to_anchor=(1.05, 1), loc=2)
plt.show()

关键提示 #

• 若需提取中间层隐藏状态，加载模型时需设置output_hidden_states=True，再通过outputs.hidden_states获取对应层数据
• 样本量越大，T-SNE可视化效果越具参考性
• 调整perplexity参数可优化聚类效果，需根据样本数量灵活调整

内容的提问来源于stack exchange，提问作者saman.joy