运行Llama-2-13b模型提示Killed错误，如何解决？

环境与配置 #

硬件配置 #

• Intel Core i7-13700HX
• NVIDIA RTX 4060
• 32GB DDR5
• 1TB SSD

运行环境 #

Windows 11 WSL2 Bash，执行命令：python3 run.py

代码文件 #

run.py

import torch
import transformers
import requests
print(torch.cuda.is_available())
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load model and adapter weights from local directory
model = transformers.AutoModelForCausalLM.from_pretrained("/home/maxloo/src/pastoring/llama/llama-2-13b")
model.to(device)
adapter = transformers.AutoModelForCausalLM.from_pretrained("/home/maxloo/src/pastoring/adapter", config=transformers.configuration.AdapterConfig.from_json_file("adapter_config.json"))
model.load_state_dict(adapter.state_dict())
adapter.load_state_dict(model.state_dict())
# Define prompt
prompt = "Hello, I am a chatbot."
# Perform inference
response = model.generate(prompt, max_length=50)
# Print response
print(response)

adapter_config.json

{
  "base_model_name_or_path": "../llama/llama-2-13b/",
  "bias": "none",
  "enable_lora": null,
  "fan_in_fan_out": false,
  "inference_mode": true,
  "init_lora_weights": true,
  "lora_alpha": 16,
  "lora_dropout": 0.05,
  "merge_weights": false,
  "modules_to_save": null,
  "peft_type": "LORA",
  "r": 16,
  "target_modules": [
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj"
  ],
  "task_type": "CAUSAL_LM",
  "task": "question_answering",
  "domain": "general"
}

问题现象 #

执行python3 run.py后仅输出Killed，未得到预期的聊天回复。

核心原因 #

Llama-2-13b全精度（FP32）需约52GB显存，半精度（FP16）也需26GB，而RTX4060通常只有8GB显存，直接加载全量模型会触发内存/显存不足，系统为稳定会强制终止进程，即输出"Killed"。同时代码中LoRA适配器加载方式错误，进一步加剧资源占用问题。

具体修复步骤 #

1. 启用模型量化加载 #

用bitsandbytes库做4-bit/8-bit量化，大幅降低显存占用。修改run.py中模型加载代码：

# 替换原模型加载代码
model = transformers.AutoModelForCausalLM.from_pretrained(
    "/home/maxloo/src/pastoring/llama/llama-2-13b",
    load_in_4bit=True,  # 4-bit量化适配8GB显存，也可尝试load_in_8bit=True
    device_map="auto",
    torch_dtype=torch.float16
)

• 先安装依赖：pip install bitsandbytes accelerate

2. 修正LoRA适配器加载逻辑 #

原代码用AutoModelForCausalLM加载适配器是错误的，需用PEFT库的PeftModel正确加载：

# 替换原适配器加载代码
from peft import PeftModel
model = PeftModel.from_pretrained(model, "/home/maxloo/src/pastoring/adapter")

• 原代码中model.load_state_dict(adapter.state_dict())等操作会导致权重覆盖，完全错误，需删除。
• 安装PEFT库：pip install peft

3. 规范推理流程 #

原generate方法调用不规范，需用tokenizer处理输入，同时调整参数减少资源占用：

# 替换原推理代码
tokenizer = transformers.AutoTokenizer.from_pretrained("/home/maxloo/src/pastoring/llama/llama-2-13b")
tokenizer.pad_token = tokenizer.eos_token  # 避免警告

prompt = "Hello, I am a chatbot."
inputs = tokenizer(prompt, return_tensors="pt").to(device)
response = model.generate(
    **inputs,
    max_length=50,
    do_sample=True,
    temperature=0.7,
    pad_token_id=tokenizer.eos_token_id
)

# 输出可读文本，而非token ID
print(tokenizer.decode(response[0], skip_special_tokens=True))

4. 调整WSL2内存分配 #

默认WSL2内存分配可能不足，新建或编辑C:\Users\<你的用户名>\.wslconfig文件：

[wsl2]
memory=24GB  # 分配24GB给WSL2，留8GB给Windows
swap=8GB
localhostForwarding=true

• 修改后重启WSL：wsl --shutdown，再重新打开WSL2。

完整修正后的run.py #

import torch
import transformers
from peft import PeftModel

print(torch.cuda.is_available())
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# 加载量化后的基础模型
model = transformers.AutoModelForCausalLM.from_pretrained(
    "/home/maxloo/src/pastoring/llama/llama-2-13b",
    load_in_4bit=True,
    device_map="auto",
    torch_dtype=torch.float16
)

# 加载LoRA适配器
model = PeftModel.from_pretrained(model, "/home/maxloo/src/pastoring/adapter")

# 加载并配置tokenizer
tokenizer = transformers.AutoTokenizer.from_pretrained("/home/maxloo/src/pastoring/llama/llama-2-13b")
tokenizer.pad_token = tokenizer.eos_token

# 执行推理并输出结果
prompt = "Hello, I am a chatbot."
inputs = tokenizer(prompt, return_tensors="pt").to(device)
response = model.generate(
    **inputs,
    max_length=50,
    do_sample=True,
    temperature=0.7,
    pad_token_id=tokenizer.eos_token_id
)
print(tokenizer.decode(response[0], skip_special_tokens=True))

内容的提问来源于stack exchange，提问作者user22894658