微调BLIP模型实现ROCO胸部X光图像captioning遇整数索引错误求助

我尝试在ROCO数据集上微调BLIP模型，实现胸部X光图像的图像captioning任务，但运行代码时触发了整数索引相关错误，具体报错信息如下：

KeyError: 'Indexing with integers is not available when using Python based feature extractors'

原代码如下：

def read_data(filepath,csv_path,n_samples):
    df = pd.read_csv(csv_path)
    images = []
    capts = []
    for idx in range(len(df)):
        if 'hest x-ray' in df['caption'][idx] or 'hest X-ray' in df['caption'][idx]:
            if len(images)>n_samples:
                break            
            else:
                images.append(Image.open(os.path.join(filepath,df['name'][idx])).convert('L'))
                capts.append(df['caption'][idx])
    return images, capts
    

def get_data():
    imgtrpath = 'all_data/train/radiology/images'
    trcsvpath = 'all_data/train/radiology/traindata.csv'
    imgtspath = 'all_data/test/radiology/images'
    tscsvpath = 'all_data/test/radiology/testdata.csv'
    imgvalpath = 'all_data/validation/radiology/images'
    valcsvpath = 'all_data/validation/radiology/valdata.csv'

    print('Extracting Training Data')
    trainimgs, traincapts = read_data(imgtrpath, trcsvpath, 1800)
    
    print('Extracting Testing Data')
    testimgs, testcapts = read_data(imgtrpath, trcsvpath, 100)
    
    print('Extracting Validation Data')
    valimgs, valcapts = read_data(imgtrpath, trcsvpath, 100)


                
    return trainimgs, traincapts, testimgs, testcapts, valimgs, valcapts

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainimgs, traincapts, testimgs, testcapts, valimgs, valcapts = get_data() 
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")

metric = evaluate.load("accuracy")
traindata = processor(text=traincapts, images=trainimgs, return_tensors="pt", padding=True, truncation=True)
evaldata =  processor(text=testcapts, images=testimgs, return_tensors="pt", padding=True, truncation=True)
training_args = TrainingArguments(output_dir="test_trainer", evaluation_strategy="epoch")
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=traindata,
    eval_dataset=evaldata,
    compute_metrics=compute_metrics
)
trainer.train()

1. 数据集格式不兼容：直接将processor输出的批量张量字典传给Trainer，但Trainer要求数据集支持整数索引（如PyTorch Dataset或Hugging Face Dataset对象），而批量张量字典无法按索引提取单个样本，这是报错核心原因。
2. 数据读取逻辑错误：测试集和验证集错误复用了训练集的文件路径，导致测试/验证数据与训练数据完全重复，失去了评估模型泛化能力的意义。
3. 图像格式不匹配：将图像转为灰度图（convert('L')），但BLIP模型默认处理3通道RGB图像，灰度图会导致输入维度不兼容。
4. 评估指标选择错误：图像captioning是序列生成任务，accuracy逐词匹配的指标完全不适用，无法有效评估生成结果的质量。

1. 转换数据集为Hugging Face Dataset格式 #

导入Dataset类，将原始图像和文本数据转为支持索引的数据集格式，并编写单样本预处理函数：

from datasets import Dataset

# 将读取到的数据转为Dataset对象
train_dataset = Dataset.from_dict({"image": trainimgs, "text": traincapts})
test_dataset = Dataset.from_dict({"image": testimgs, "text": testcapts})
val_dataset = Dataset.from_dict({"image": valimgs, "text": valcapts})

# 定义单样本预处理函数
def preprocess_function(examples):
    inputs = processor(images=examples["image"], text=examples["text"], padding="max_length", truncation=True, return_tensors="pt")
    # 自回归任务需要将input_ids作为labels
    inputs["labels"] = inputs["input_ids"].clone()
    return inputs

# 批量预处理数据集
tokenized_train = train_dataset.map(preprocess_function, batched=True)
tokenized_test = test_dataset.map(preprocess_function, batched=True)
tokenized_val = val_dataset.map(preprocess_function, batched=True)

2. 修正测试/验证集读取路径 #

修改get_data函数，确保测试集和验证集读取对应路径的文件：

def get_data():
    imgtrpath = 'all_data/train/radiology/images'
    trcsvpath = 'all_data/train/radiology/traindata.csv'
    imgtspath = 'all_data/test/radiology/images'
    tscsvpath = 'all_data/test/radiology/testdata.csv'
    imgvalpath = 'all_data/validation/radiology/images'
    valcsvpath = 'all_data/validation/radiology/valdata.csv'

    print('Extracting Training Data')
    trainimgs, traincapts = read_data(imgtrpath, trcsvpath, 1800)
    
    print('Extracting Testing Data')
    testimgs, testcapts = read_data(imgtspath, tscsvpath, 100)
    
    print('Extracting Validation Data')
    valimgs, valcapts = read_data(imgvalpath, valcsvpath, 100)
                
    return trainimgs, traincapts, testimgs, testcapts, valimgs, valcapts

3. 修正图像格式 #

去掉灰度转换，保持图像为RGB格式：

# 替换read_data中的图像读取行
images.append(Image.open(os.path.join(filepath,df['name'][idx])).convert('RGB'))

4. 更换合适的评估指标 #

使用captioning任务专用的BLEU指标替换accuracy：

import evaluate

metric = evaluate.load("bleu")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    # 去除pad token（通常为0），避免影响解码
    labels = np.where(labels != 0, labels, -100)
    # 将token id解码为文本
    pred_texts = processor.batch_decode(predictions, skip_special_tokens=True)
    label_texts = processor.batch_decode(labels, skip_special_tokens=True)
    # 计算BLEU分数
    return metric.compute(predictions=pred_texts, references=[[text] for text in label_texts])

5. 更新Trainer参数 #

将预处理后的数据集传入Trainer，并调整训练参数适配任务：

training_args = TrainingArguments(
    output_dir="test_trainer",
    evaluation_strategy="epoch",
    per_device_train_batch_size=8,  # 根据GPU显存调整
    per_device_eval_batch_size=8,
    num_train_epochs=3,
    logging_dir='./logs',
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_val,
    compute_metrics=compute_metrics
)
trainer.train()

内容的提问来源于stack exchange，提问作者FaiqFF