使用CSV数据集微调LLaMA3.2后模型未返回预期结果的问题

问题背景 #

尝试使用本地CSV文件微调LLaMA3.2 1B模型，完成完整训练流程后，测试发现模型输出与数据集不符；修改数据集为29条相同问题的类似答案后，所有问题均返回类似"Unknown"的结果，需解决该异常。

执行的完整训练与部署流程 #

1. 导入所需库 #

import pandas as pd
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
from torch.utils.data import Dataset, DataLoader

2. 加载本地模型与Tokenizer #

model_name = "D://LLama3.2_1B//"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

3. 处理Padding Token #

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token  # Use EOS token as padding token

4. 模型移至GPU #

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

5. 自定义数据集加载 #

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)
class QADataset(Dataset):
    def __init__(self, dataframe, tokenizer, max_length=512):
        self.data = dataframe
        self.tokenizer = tokenizer
        self.max_length = max_length

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        question = self.data.iloc[idx]['question']
        answer = self.data.iloc[idx]['answer']

        # Tokenize question and answer together for training
        encoding = self.tokenizer(
            question,
            answer,
            truncation=True,
            padding='max_length',
            max_length=self.max_length,
            return_tensors='pt'
        )

        return {
            'input_ids': encoding['input_ids'].squeeze(0),
            'attention_mask': encoding['attention_mask'].squeeze(0),
            'labels': encoding['input_ids'].squeeze(0)  # Labels for training
        }

df_qa = pd.read_csv('D://LLama3.2_1B_Dataset//graph.csv')
qa_dataset = QADataset(df_qa, tokenizer)

初始数据集示例：

question,answer
"What is the capital of France?", "Unknow"
"Who developed LLaMA models?", "Microsoft"

6. 模型训练与保存 #

qa_loader = DataLoader(qa_dataset, batch_size=2, shuffle=True)
for batch in qa_loader:
    print(batch)
model.train()
for epoch in range(3):
    for batch in qa_loader:
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        labels = batch['labels'].to(device)
        outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

        print(f"Epoch {epoch}, Loss: {loss.item()}")

output_dir = "D://LLama3.2_1B_output_dir//"
model.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)

7. Flask部署与测试 #

from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments,Trainer
from flask import Flask, jsonify, request
from datasets import load_dataset
import torch,json
app = Flask(__name__)
model_path = "D://LLama3.2_1B_output_dir//"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
@app.route("/ask", methods=["GET"])
def ask_question():
    question = request.args.get("question", "")
    inputs = tokenizer(question, return_tensors="pt").to(device)
    with torch.no_grad():
        outputs = model.generate(
        **inputs,
        max_length=450,
        num_return_sequences=1,
        no_repeat_ngram_size=2,
        top_k=50,
        top_p=0.95,
        temperature=0.7,
        do_sample=True
        )
    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    json_response = jsonify(generated_text)
    json_response.headers['Content-Type'] = 'application/json; charset=utf-8'
    return json_response
if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

异常原因分析与解决方案 #

1. 修复数据集标签逻辑 #

当前训练直接将input_ids作为labels，导致模型被迫学习预测问题部分的token，而非仅答案部分。需屏蔽问题段标签，让模型专注学习生成答案：
修改QADataset的__getitem__方法：

def __getitem__(self, idx):
    question = self.data.iloc[idx]['question']
    answer = self.data.iloc[idx]['answer']

    # 单独编码问题与答案
    question_encoding = self.tokenizer(question, truncation=True, max_length=self.max_length)
    answer_encoding = self.tokenizer(answer, truncation=True, max_length=self.max_length - len(question_encoding['input_ids']))
    
    # 拼接输入与注意力掩码
    input_ids = question_encoding['input_ids'] + answer_encoding['input_ids']
    attention_mask = question_encoding['attention_mask'] + answer_encoding['attention_mask']
    
    # 构建标签：问题段设为-100（模型忽略该部分损失）
    labels = [-100] * len(question_encoding['input_ids']) + answer_encoding['input_ids']
    
    # 补全至最大长度
    padding_length = self.max_length - len(input_ids)
    if padding_length > 0:
        input_ids += [self.tokenizer.pad_token_id] * padding_length
        attention_mask += [0] * padding_length
        labels += [-100] * padding_length
    
    return {
        'input_ids': torch.tensor(input_ids),
        'attention_mask': torch.tensor(attention_mask),
        'labels': torch.tensor(labels)
    }

2. 调整训练参数 #

• 增加训练轮次：将epoch从3提升至10-20，观察损失变化，当损失趋于稳定时停止训练。
• 降低学习率：将lr=5e-5调整为1e-5或2e-5，避免模型震荡或遗忘预训练知识。
• 梯度累积（显存不足时）：若batch_size=2过小，添加梯度累积等效增大batch_size：

accumulation_steps = 4  # 累积4步更新一次参数
model.train()
for epoch in range(10):
    total_loss = 0.0
    for step, batch in enumerate(qa_loader):
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        labels = batch['labels'].to(device)
        outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss / accumulation_steps
        loss.backward()
        total_loss += loss.item() * accumulation_steps
        
        if (step + 1) % accumulation_steps == 0:
            optimizer.step()
            optimizer.zero_grad()
            print(f"Epoch {epoch}, Step {step+1}, Loss: {total_loss/accumulation_steps}")
            total_loss = 0.0

3. 规范数据集 #

• 修正数据错误：将初始数据集中的"Unknow"改为"Unknown"，统一答案表述。
• 优化数据集结构：当使用单一问题的多样本时，确保答案一致性；同时加入少量其他问题样本，避免模型过拟合单一任务。

4. 调整生成参数 #

当前生成参数随机性过高，测试阶段可关闭采样，强制生成确定性结果：
修改生成代码：

outputs = model.generate(
    **inputs,
    max_length=450,
    num_return_sequences=1,
    no_repeat_ngram_size=2,
    do_sample=False,  # 关闭采样
    temperature=1.0,
    max_new_tokens=100  # 限制生成长度，避免冗余
)

5. 验证模型加载 #

加载模型时添加torch_dtype=torch.float16（GPU支持时），减少内存占用并保证精度：

model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16)

内容的提问来源于stack exchange，提问作者graph User