微调VisionEncoderDecoderModel时DataCollatorForSeq2Seq的ValueError解决

问题根源 #

1. DataCollatorForSeq2Seq是为文本到文本任务设计的，默认会调用文本tokenizer处理输入特征，但你的输入是图像格式的pixel_values，不属于文本tokenizer的处理范围，导致报错。
2. 数据集__getitem__方法错误地将标签存入input_ids字段，而VisionEncoderDecoderModel要求标签字段为labels。
3. Trainer初始化时传入了错误的tokenizer参数（使用了图像特征提取器而非文本tokenizer）。

修正步骤 #

1. 修正数据集__getitem__方法

将返回的input_ids字段改为labels，匹配模型的输入要求：

def __getitem__(self, idx):
    file_name = self.df['file_name'][idx]
    text = self.df['text'][idx]

    assert text.strip() != "", f"ERROR Empty text in {idx}"

    # 读取图像并应用增强
    image = Image.open(self.root_dir + file_name).convert('RGB')
    image = train_transforms(image)
    pixel_values = self.processor(image, return_tensors='pt').pixel_values.squeeze(0)
    
    # 处理文本标签
    labels = self.processor.tokenizer(
        text,
        padding='max_length',
        max_length=self.max_target_length,
        return_tensors='pt'
    ).input_ids.squeeze(0)

    # 将padding token替换为-100（模型会忽略该值的损失计算）
    labels = torch.where(labels == self.processor.tokenizer.pad_token_id, torch.tensor(-100), labels)
    
    # 返回正确字段：图像输入pixel_values、文本标签labels
    return {
        "pixel_values": pixel_values,
        "labels": labels
    }

2. 使用自定义DataCollator

放弃DataCollatorForSeq2Seq，改用适配图像输入的自定义collator，修正之前的实现：

def collate_fn(batch):
    # 过滤无效样本
    batch = list(filter(lambda x: x is not None, batch))
    
    # 堆叠图像张量（已统一尺寸）
    pixel_values = torch.stack([item['pixel_values'] for item in batch])
    # 对标签进行padding
    labels = torch.nn.utils.rnn.pad_sequence(
        [item['labels'] for item in batch],
        batch_first=True,
        padding_value=-100
    )
    
    return {
        "pixel_values": pixel_values,
        "labels": labels
    }

注：若之前出现num_items_in_batch错误，升级transformers到≥4.20.0版本即可解决。

3. 修正Trainer初始化参数

将tokenizer参数改为processor.tokenizer，确保Seq2SeqTrainer能正确处理文本生成逻辑：

# 初始化模型并配置生成参数
model = VisionEncoderDecoderModel.from_pretrained(ModelConfig.MODEL_NAME)
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id

# 初始化Trainer
trainer = Seq2SeqTrainer(
    model=model,
    tokenizer=processor.tokenizer,  # 替换为文本tokenizer
    args=training_args,
    compute_metrics=compute_cer,
    train_dataset=train_dataset,
    eval_dataset=valid_dataset,
    data_collator=collate_fn  # 使用自定义collator
)
trainer.train()

原问题背景 #

我想要微调VisionEncoderDecoderModel.from_pretrained(model_name)，使用来自Learn Open CV的CustomOCRDataset。但默认data_collator因样本形状不同无法堆叠输入，因此尝试使用DataCollatorForSeq2Seq并加入Resize数据增强，却遇到错误：

ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['pixel_values']

修改__getitem__返回input_ids后错误依旧，相关代码及完整报错信息如下：

原数据集代码

def __getitem__(self, idx):
        file_name = self.df['file_name'][idx]
        text = self.df['text'][idx]

        assert text.strip() != "", f"ERROR Empty text in {idx}"

        # Read the image, apply augmentations, and get the transformed pixels.
        image = Image.open(self.root_dir + file_name).convert('RGB')
        image = train_transforms(image)
        pixel_values = self.processor(image, return_tensors='pt').pixel_values
        # Pass the text through the tokenizer and get the labels,
        # i.e. tokenized labels.
        labels = self.processor.tokenizer(
            text,
            padding='max_length',
            max_length=self.max_target_length,
            return_tensors='pt'
        ).input_ids.squeeze(0)

        # We are using -100 as the padding token.
        labels = torch.where(labels == self.processor.tokenizer.pad_token_id, torch.tensor(-100), labels)
        encoding = {"pixel_values": pixel_values.squeeze(0),
                    "input_ids": labels}
        return encoding

配置类代码

@dataclass(frozen=True)
class TrainingConfig:
    BATCH_SIZE:    int = 16
    EPOCHS:        int = 5
    LEARNING_RATE: float = 0.00005

@dataclass(frozen=True)
class DatasetConfig:
    DATA_ROOT:     str = image_dir

@dataclass(frozen=True)
class ModelConfig:
    MODEL_NAME: str = 'microsoft/trocr-base-handwritten'

数据增强代码

# Augmentations.
train_transforms = transforms.Compose([
    transforms.Resize((1024, 880))
])

数据集与训练参数初始化

processor = TrOCRProcessor.from_pretrained(ModelConfig.MODEL_NAME)
train_dataset = CustomOCRDataset(
    root_dir=os.path.join(DatasetConfig.DATA_ROOT, train_destination),
    df=train_df,
    processor=processor
)
valid_dataset = CustomOCRDataset(
    root_dir=os.path.join(DatasetConfig.DATA_ROOT, test_destination),
    df=test_df,
    processor=processor
)

training_args = Seq2SeqTrainingArguments(
    predict_with_generate=True,
    evaluation_strategy='epoch',
    per_device_train_batch_size=TrainingConfig.BATCH_SIZE,
    per_device_eval_batch_size=TrainingConfig.BATCH_SIZE,
    fp16=True,
    output_dir='seq2seq_model_printed/',
    logging_strategy='epoch',
    save_strategy='epoch',
    save_total_limit=5,
    report_to='tensorboard',
    num_train_epochs=TrainingConfig.EPOCHS
)

原Trainer初始化代码

data_collator = DataCollatorForSeq2Seq(tokenizer=processor.tokenizer, model=model, padding=True)
# Initialize trainer.
trainer = Seq2SeqTrainer(
    model=model,
    tokenizer=processor.feature_extractor,
    args=training_args,
    compute_metrics=compute_cer,
    train_dataset=train_dataset,
    eval_dataset=valid_dataset,
    data_collator=data_collator
)
trainer.train()

完整报错信息

File \transformers\data\data_collator.py:599, in DataCollatorForSeq2Seq.__call__(self, features, return_tensors)
    596 non_labels_features = [{k: v for k, v in feature.items() if k != label_name} for feature in features]
    598 # run through tokenizer without labels to ensure no side effects
--> 599 batch = pad_without_fast_tokenizer_warning(
    600     self.tokenizer,
    601     non_labels_features,
    602     padding=self.padding,
    603     max_length=self.max_length,
    604     pad_to_multiple_of=self.pad_to_multiple_of,
    605     return_tensors=return_tensors,
    606 )
    608 # we have to pad the labels manually as we cannot rely on `tokenizer.pad` and we need them to be of the same length to return tensors
    609 no_padding = self.padding is False or self.padding == PaddingStrategy.DO_NOT_PAD

File \transformers\data\data_collator.py:66, in pad_without_fast_tokenizer_warning(tokenizer, *pad_args, **pad_kwargs)
     63 tokenizer.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = True
     65 try:
--> 66     padded = tokenizer.pad(*pad_args, **pad_kwargs)
     67 finally:
     68     # Restore the state of the warning.
     69     tokenizer.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = warning_state

File \transformers\tokenization_utils_base.py:3305, in PreTrainedTokenizerBase.pad(self, encoded_inputs, padding, max_length, pad_to_multiple_of, padding_side, return_attention_mask, return_tensors, verbose)
   3303 # The model's main input name, usually `input_ids`, has been passed for padding
   3304 if self.model_input_names[0] not in encoded_inputs:
--> 3305     raise ValueError(
   3306         "You should supply an encoding or a list of encodings to this method "
   3307         f"that includes {self.model_input_names[0]}, but you provided {list(encoded_inputs.keys())}"
   3308     )
   3310 required_input = encoded_inputs[self.model_input_names[0]]
   3312 if required_input is None or (isinstance(required_input, Sized) and len(required_input) == 0):

ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['pixel_values']

已尝试的方案

• 使用transforms.Resize((1024, 880))统一图像尺寸
• 使用自定义data collator，但出现TypeError: ViTModel.forward() got an unexpected keyword argument 'num_items_in_batch'错误

内容的提问来源于stack exchange，提问作者Milana Alimova