使用HuggingFace微调Vision Encoder Decoder模型遇ValueError问题修复

你遇到的ValueError: expected sequence of length 11 at dim 2 (got 12)是三个核心问题共同导致的：

• 模型初始化时tokenizer未加载，导致配置参数引用未定义变量
• 文本tokenization后保留了多余的batch维度，样本labels为二维张量无法合并
• 使用的default_data_collator不支持Seq2Seq任务的变长序列padding处理

1. 修正tokenizer与模型的初始化顺序 #

原代码先初始化模型再加载tokenizer，属于逻辑错误，必须先加载特征提取器和tokenizer，再初始化模型。同时注意BERT没有默认bos_token_id，需用cls_token_id作为decoder的起始token。

修改后的初始化代码：

from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer, ViTFeatureExtractor, AutoTokenizer
from transformers import VisionEncoderDecoderModel, DataCollatorForSeq2Seq
from datasets import load_dataset, DatasetDict

encoder_checkpoint = "google/vit-base-patch16-224-in21k"
decoder_checkpoint = "bert-base-uncased"

# 先加载特征提取器和tokenizer
feature_extractor = ViTFeatureExtractor.from_pretrained(encoder_checkpoint)
tokenizer = AutoTokenizer.from_pretrained(decoder_checkpoint)

# 再初始化模型
model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
    encoder_checkpoint, decoder_checkpoint
)
# 设置decoder特殊token
model.config.decoder_start_token_id = tokenizer.cls_token_id
model.config.pad_token_id = tokenizer.pad_token_id
model.config.eos_token_id = tokenizer.sep_token_id
# 对齐vocab尺寸
model.config.vocab_size = model.config.decoder.vocab_size

# 配置beam search参数
model.config.max_length = 512
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4
model.decoder.resize_token_embeddings(len(tokenizer))

2. 修正数据集处理的tokenization逻辑 #

原代码中return_tensors='pt'会返回(1, seq_len)的二维张量，导致每个样本的labels维度不一致。需要移除多余维度，同时建议采用批量处理提升效率，可选择固定序列长度或交给后续collator处理。

修改后的数据集处理代码：

dataset = load_dataset("svjack/pokemon-blip-captions-en-zh").remove_columns("zh_text")

# 批量处理图像特征
def process_image(examples):
    pixel_values = feature_extractor([img for img in examples['image']], return_tensors='pt').pixel_values
    return {"pixel_values": pixel_values}

dataset = dataset.map(process_image, batched=True)
dataset = dataset.remove_columns("image")

# 批量处理文本标签，移除多余维度
def process_text(examples):
    tokenized = tokenizer(
        examples['en_text'],
        padding="max_length",
        truncation=True,
        max_length=512,
        return_tensors='pt'
    )
    # 去掉batch维度，确保labels为一维数组
    return {"labels": tokenized.input_ids.squeeze(1)}

dataset = dataset.map(process_text, batched=True)
dataset = dataset.remove_columns("en_text")

# 划分训练/验证/测试集
train_testvalid = dataset["train"].train_test_split(0.1)
test_valid = train_testvalid['test'].train_test_split(0.5)
train_test_valid_dataset = DatasetDict({
    'train': train_testvalid['train'],
    'test': test_valid['test'],
    'valid': test_valid['train']
})

3. 替换数据整理器为Seq2Seq专属实现 #

default_data_collator无法处理变长序列，必须使用DataCollatorForSeq2Seq，它会自动完成序列padding、截断，并生成decoder_input_ids。

修改后的训练器初始化代码：

# 冻结encoder参数
for param in model.encoder.parameters():
    param.requires_grad = False

output_dir = "./checkpoints"
training_args = Seq2SeqTrainingArguments(
    predict_with_generate=True,
    evaluation_strategy="steps",
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    overwrite_output_dir=True,
    fp16=True,
    run_name="first_run",
    load_best_model_at_end=True,
    output_dir=output_dir,
    logging_steps=2000,
    save_steps=2000,
    eval_steps=2000,
)

# 使用Seq2Seq专属数据整理器
data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)

trainer = Seq2SeqTrainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=train_test_valid_dataset['train'],
    eval_dataset=train_test_valid_dataset['valid'],
    data_collator=data_collator,
)

trainer.train()

内容的提问来源于stack exchange，提问作者Rocketq