为何微调HuggingFace预训练模型时所有参数均可训练？

严格遵循HuggingFace预训练模型微调文档操作，但得到的模型显示100%参数可训练。原本预期仅部分层会被解冻优化，但实际所有参数均处于可训练状态。

使用的代码如下：

def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param:.2f}"
    )

...
# id2label 和 label2id 对应当前任务的3个类别

model_name = "nvidia/segformer-b5-finetuned-cityscapes-1024-1024"
model = AutoModelForSemanticSegmentation.from_pretrained(model_name, id2label=id2label, label2id=label2id, ignore_mismatched_sizes=True)
print_trainable_parameters(model)

运行代码后输出：

Some weights of SegformerForSemanticSegmentation were not initialized from the model checkpoint at nvidia/segformer-b5-finetuned-cityscapes-1024-1024 and are newly initialized because the shapes did not match:
- decode_head.classifier.weight: found shape torch.Size([19, 768, 1, 1]) in the checkpoint and torch.Size([3, 768, 1, 1]) in the model instantiated
- decode_head.classifier.bias: found shape torch.Size([19]) in the checkpoint and torch.Size([3]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
trainable params: 84595651 || all params: 84595651 || trainable%: 100.00

疑问：为何所有参数都可训练？知道可使用PEFT减少可训练参数，但根据decode_head.classifier层的警告，本以为仅小部分参数可优化。

原因分析 #

• HuggingFace的from_pretrained方法默认会将所有参数的requires_grad设为True，不管参数是从 checkpoint 加载的还是新初始化的。你看到的警告只是提示分类头参数因类别数不匹配被重新初始化，但这不会改变其他参数的可训练状态。
• 文档中提到的“部分层解冻”不会自动生效，需要手动设置参数的requires_grad为False来冻结不需要训练的层，或者使用PEFT框架实现参数高效微调。

解决方法 #

方法1：手动冻结Backbone层，仅训练分类头

# 先冻结所有参数
for param in model.parameters():
    param.requires_grad = False

# 解冻分类头的参数
for param in model.decode_head.classifier.parameters():
    param.requires_grad = True

# 再次检查可训练参数占比
print_trainable_parameters(model)

方法2：使用PEFT实现参数高效微调（以LoRA为例）

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=["query", "value"],  # 根据Segformer结构选择目标模块
    lora_dropout=0.05,
    bias="none",
    task_type="SEMANTIC_SEGMENTATION"
)

model = get_peft_model(model, config)
print_trainable_parameters(model)

内容的提问来源于stack exchange，提问作者Bruno