Hey there! Let's dive into your plan for doing abstractive text summarization with Llama 3.1 8B Instruct—your core steps are already solid, but I’ll break down each part and add some key details you might want to consider to make the process smoother and more effective.

First, let’s recap your proposed workflow to make sure we’re on the same page:

• Load the CNN Daily dataset (which includes articles and their corresponding highlights/summaries)
• Load the Llama 3.1 8B Instruct model and tokenizer locally in Kaggle
• Use LoRA for parameter-efficient fine-tuning of the LLM
• Preprocess the dataset with the tokenizer, incorporating task-specific instruction prompts
• Train the model using TrainingArguments and SFTTrainer

Let’s break down each step + add critical tweaks: #

1. Dataset Loading & Prep
   CNN Daily is a great choice for general text summarization tasks—it’s well-curated and widely used. That said, you might want to:

   • Do a quick data cleanup: Remove overly short/long articles, duplicate entries, or samples where the summary is irrelevant to the article.
   • If your use case is domain-specific (e.g., medical, tech), supplement CNN Daily with domain-specific summarization datasets to boost performance on your target content.

2. Model & Tokenizer Loading in Kaggle
   Kaggle’s GPU environment works well for this, but keep these in mind:

   • Make sure you have proper access to Llama 3.1 (you’ll need to accept Meta’s license on Hugging Face Hub first).
   • Use memory-efficient loading to avoid OOM errors: Use load_in_4bit or load_in_8bit via bitsandbytes when loading the model, like this:

     from transformers import AutoModelForCausalLM, AutoTokenizer
     model = AutoModelForCausalLM.from_pretrained(
         "meta-llama/Llama-3.1-8B-Instruct",
         load_in_4bit=True,
         device_map="auto"
     )
     tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
     tokenizer.pad_token = tokenizer.eos_token
     

   • Double-check that the tokenizer’s pad token is set (Llama models don’t have a default pad token, so using eos_token as the pad token is a common fix).

3. LoRA Configuration
   Choosing LoRA is smart—it’s way more memory-efficient than full fine-tuning. To get the most out of it:

   • Use the peft library to set up LoRA: Define parameters like r (rank, typically 8-64), lora_alpha (usually 2x the rank), and target modules (focus on attention layers like q_proj and v_proj for best results).
   • Example LoRA config snippet:

     from peft import LoraConfig
     lora_config = LoraConfig(
         r=16,
         lora_alpha=32,
         target_modules=["q_proj", "v_proj"],
         lora_dropout=0.05,
         bias="none",
         task_type="CAUSAL_LM"
     )
     

4. Dataset Preprocessing with Instruction Prompts
   Since you’re using an Instruct-tuned model, formatting your prompts correctly is make-or-break. Follow Llama 3.1’s official chat template to align with how the model was trained:

   • Use role tags like <|user|> and <|assistant|> to structure your input-output pairs:

     <|user|>
     Please provide an abstractive summary of the following article:
     {article_text}
     <|assistant|>
     {summary_text}<|end_of_solution|>
     

   • Set a reasonable max_length (match the model’s context window—Llama 3.1 8B has an 8k default, some variants go up to 128k) and truncate/pad sequences appropriately to avoid overflow.

5. Training with TrainingArguments & SFTTrainer
   Your choice of SFTTrainer is perfect for supervised fine-tuning here. Tweak these key arguments:

   • Batch size: Start with per_device_train_batch_size=2 or 4 (adjust based on your GPU memory; use gradient_accumulation_steps if you need to simulate larger batches).
   • Learning rate: For LoRA, a range of 1e-4 to 5e-4 works well—avoid setting it too high, which can cause catastrophic forgetting.
   • Evaluation: Add a validation split of your dataset and enable evaluation during training. Use ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L) to measure summarization quality—this helps you spot overfitting and track progress.
   • Example TrainingArguments snippet:

     from transformers import TrainingArguments
     training_args = TrainingArguments(
         output_dir="./llama3_summarizer",
         per_device_train_batch_size=2,
         gradient_accumulation_steps=4,
         learning_rate=2e-4,
         num_train_epochs=3,
         logging_steps=10,
         save_steps=50,
         evaluation_strategy="epoch",
         fp16=True,
         report_to="none"
     )
     

Additional Steps You Might Miss #

• Post-Training Evaluation: After training, run inference on a held-out test set and compute ROUGE scores. You can also do a small manual evaluation to check if summaries are coherent, capture key details, and avoid hallucinations.
• Model Merging (Optional): If you need to deploy the model without loading LoRA weights separately, merge the LoRA adapter with the base model using peft’s merge_and_unload() method.
• Inference Optimization: For faster inference, consider using quantization (e.g., GPTQ) or frameworks like vLLM if you’re scaling up.

Overall, your initial workflow covers all the core bases—adding these details will help you avoid common pitfalls and build a more robust summarization model.

备注：内容来源于stack exchange，提问作者user26732823