微调Qwen2-VL-7B-Instruct时遇文本输入类型ValueError的排查求助

大家好，我目前正在尝试微调Qwen/Qwen2-VL-7B-Instruct这类视觉大语言模型，使用的是.jsonl格式的自定义数据集，同时自己写了一套微调代码，但在运行过程中碰到了和文本输入类型相关的ValueError，想请社区的朋友们帮忙排查下问题。

我的数据集格式（.jsonl文件内容） #

{"messages": [{"role": "system", "content": [{"type": "text", "text": "You are an expert assistant for answering questions based on textual and visual data."}]}, {"role": "user", "content": [{"type": "text", "text": "\nBased on the provided context and image, answer the following questions.\n\n## CONTEXT ##\nGerman Text: Schörghuber Unternehmensgruppe\nESG – Status quo, Chancen und Risiken aus Sicht eines \nImmobilien-Bestandshalters\nGastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer\nTranslated Text: Schörghuber group\nESG - status quo, opportunities and risks from the perspective of one\nReal estate stock holder\nGuest lecture Advanced Topics in Sustainable Real Estate on May 12, 2023, Janine Schluer\nTopic: Real Estate Investment\nSummary: Schörghuber Unternehmensgruppe\nESG – Status quo, Chancen und Risiken aus Sicht eines \nImmobilien-Bestandshalters\nGastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer\nKeywords: []\nExtracted Text: nan\nGenerated Questions: ['Schörghuber Unternehmensgruppe ESG – Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text]'\n 'Schörghuber Unternehmensgruppe ESG – Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text: nan]'\n 'Schörghuber Unternehmensgruppe ESG – Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text: nan ]'\n 'Schörghuber Unternehmensgruppe ESG - Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text: nan ]'\n 'Schörghuber Unternehmensgruppe ESG - Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text: nan]']\n\n## IMAGE ##\nImage Path: C:\\Users\\ehsan\\Downloads\\Initial Dataset\\ehsan\\230512_ESG-Vortrag_TUM\\230512_ESG-Vortrag_TUM.pdf_page_1_img_1.png\n\nOnly return the answers as your output.\n"}, {"type": "image", "image": "C:\\Users\\ehsan\\Downloads\\Initial Dataset\\ehsan\\230512_ESG-Vortrag_TUM\\230512_ESG-Vortrag_TUM.pdf_page_1_img_1.png"}]}, {"role": "assistant", "content": [{"type": "text", "text": ""}]}]}
{"messages": [{"role": "system", "content": [{"type": "text", "text": "You are an expert assistant for answering questions based on textual and visual data."}]}, {"role": "user", "content": [{"type": "text", "text": "\nBased on the provided context and image, answer the following questions.\n\n## CONTEXT ##\nGerman Text: Schörghuber Unternehmensgruppe\nESG – Status quo, Chancen und Risiken aus Sicht eines \nImmobilien-Bestandshalters\nGastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer\nTranslated Text: Schörghuber group\nESG - status quo, opportunities and risks from the perspective of one\nReal estate stock holder\nGuest lecture Advanced Topics in Sustainable Real Estate on May 12, 2023, Janine Schluer\nTopic: Real Estate Investment\nSummary: Schörghuber Unternehmensgruppe\nESG – Status quo, Chancen und Risiken aus Sicht eines \nImmobilien-Bestandshalters\nGastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer\nKeywords: []\nExtracted Text: nan\nGenerated Questions: ['Schörghuber Unternehmensgruppe ESG – Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text]'\n 'Schörghuber Unternehmensgruppe ESG – Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text: nan]'\n 'Schörghuber Unternehmensgruppe ESG – Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text: nan ]'\n 'Schörghuber Unternehmensgruppe ESG - Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text: nan ]'\n 'Schörghuber Unternehmensgruppe ESG - Status quo, Chancen und Risiken aus Sicht eines Immobilien-Bestandshalters Gastvortrag Advanced Topics in Sustainable Real Estate am 12. Mai 2023, Janine Schluer Keywords: [Extracted Text: nan]']\n\n## IMAGE ##\nImage Path: C:\\Users\\ehsan\\Downloads\\Initial Dataset\\ehsan\\230512_ESG-Vortrag_TUM\\230512_ESG-Vortrag_TUM.pdf_page_1_img_2.jpeg\n\nOnly return the answers as your output.\n"}, {"type": "image", "image": "C:\\Users\\ehsan\\Downloads\\Initial Dataset\\ehsan\\230512_ESG-Vortrag_TUM\\230512_ESG-Vortrag_TUM.pdf_page_1_img_2.jpeg"}]}, {"role": "assistant", "content": [{"type": "text", "text": ""}]}]}

我使用的微调代码 #

import torch
from transformers import AutoModelForVision2Seq, AutoProcessor, BitsAndBytesConfig
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig
from datasets import load_dataset
from PIL import Image
from typing import List, Dict, Any
import os

# Helper function to process visual references (e.g., images)
def process_vision_info(visual_reference: str) -> List[Any]:
    """
    Load and preprocess images given their file paths.

    :param visual_reference: Filepath to the image.
    :return: Preprocessed image tensor.
    """
    try:
        image = Image.open(visual_reference).convert("RGB")
        return [image]
    except Exception as e:
        print(f"Error processing image {visual_reference}: {e}")
        return [None]  # Placeholder for failed image loads

# Extract relevant information from messages
def extract_information(messages: List[Dict[str, Any]]) -> Dict[str, Any]:
    """
    Extract information from the 'messages' field in each example.

    :param messages: List of message dictionaries.
    :return: A dictionary with extracted text and other fields.
    """
    german_text = ""
    translated_text = ""
    topic = ""
    generated_questions = []
    summary = ""
    image_path = None

    for message in messages:
        if isinstance(message, dict):  # Ensure the message is a dictionary
            for content in message.get("content", []):
                if isinstance(content, dict):  # Ensure content is a dictionary
                    if content["type"] == "text":
                        if "German Text" in content["text"]:
                            german_text = content["text"]
                        elif "Translated Text" in content["text"]:
                            translated_text = content["text"]
                        elif "Topic:" in content["text"]:
                            topic = content["text"].split("Topic:")[-1].strip()
                        elif "Summary:" in content["text"]:
                            summary = content["text"].split("Summary:")[-1].strip()
                        if "Generated Questions" in content["text"]:
                            generated_questions.append(content["text"])
                    elif content["type"] == "image" and content["image"]:
                        image_path = content["image"]
    
    return {
        "German Text": german_text,
        "Translated Text": translated_text,
        "Topic": topic,
        "Generated Questions": generated_questions,
        "Summary": summary,
        "image_path": image_path
    }

def collate_fn(examples: List[Dict[str, Any]], processor: AutoProcessor) -> Dict[str, Any]:
    german_texts, translated_texts, topics, generated_questions, summaries, images = [], [], [], [], [], []

    for example in examples:
        # Extracting text from the "content" field inside the messages
        extracted_data = extract_information(example["messages"])

        german_text = extracted_data.get("German Text", "")
        translated_text = extracted_data.get("Translated Text", "")
        topics.append(extracted_data.get("Topic", ""))
        generated_questions.append(extracted_data.get("Generated Questions", []))
        summaries.append(extracted_data.get("Summary", ""))
        images.append(process_vision_info(extracted_data["image_path"])[0] if extracted_data["image_path"] else None)

麻烦各位帮忙分析下，我的数据集处理或者代码逻辑里哪里可能导致了文本输入类型的ValueError？该如何调整才能顺利完成微调？

备注：内容来源于stack exchange，提问作者Filbadeha