如何按LLaMA官方标准重新初始化Hugging Face LLaMA v2模型权重？

我需要重新初始化正在使用/下载的LLaMA v2模型权重。查阅Hugging Face相关源码、官方文档及LLaMA v1、v2论文后，未找到模型权重的确切初始化方式（疑为商业机密）。我尝试遍历模型模块与参数，按源码提示重新初始化并打印权重L1范数，但范数未发生变化，不确定是否因PyTorch或Hugging Face模型的保护机制导致操作有误，现寻求LLaMA官方认可的正确权重初始化方法及具体参数值。

尝试代码 #

import torch
from transformers import AutoModelForCausalLM, AutoConfig
import torch.nn as nn

def main_reinit_model():
    """
    ref: https://stackoverflow.com/questions/76971761/how-to-adapt-llama-v2-model-to-less-than-7b-parameters
    ref: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L721
    ref: https://chat.openai.com/c/977d0cb0-b819-48ac-be5c-6e482ad5e518
    """
    print('Starting to reinitialize the model...')
    # Load the pretrained LLaMA v2 config
    config = AutoConfig.from_pretrained("meta-llama/Llama-2-7b-hf")
    # print(f'config: {config} {type(config)}')
    # Print the original number of parameters
    model = AutoModelForCausalLM.from_config(config)
    # put model on device cuda
    model = model.to('cuda')
    # print the model's device
    print(f'{model.device=}')
    # print(f'{model=}')
    # print("Original number of parameters:", sum(p.numel() for p in model.parameters()))
    # go through all parameters and compute the l1 norm and sum it then print it
    norm_model = sum(p.norm(1) for p in model.parameters())
    # loop through modules of model and reinitialize weights with normal_mean, 0.02
    print(f'{norm_model=}')
    """
    go through model and print all laters
    """
    # model.init_weights()  # didn't work
    # model._init_weights(module)  # didn't work needs module
    # for name, param in model.named_parameters():
    #     model._init_weights(param)
    # model.post_init()
    reinitialize_weights(model)
    # model._initialize_weights(module)  # didn't work needs module
    # for name, param in model.named_parameters():
    #     print(f'{name=} {param.shape=}')
    norm_model = sum(p.norm(1) for p in model.parameters())
    print(f'{norm_model=}')

def reinitialize_weights(model) -> None:
    for module in model.modules():
        if isinstance(module, nn.Linear):
            nn.init.normal_(module.weight, mean=0, std=0.02)
            if module.bias is not None:
                nn.init.constant_(module.bias, 0)

def _init_weights(self, module):
    std = self.config.initializer_range
    if isinstance(module, nn.Linear):
        module.weight.data.normal_(mean=100.0, std=std)
        if module.bias is not None:
            module.bias.data.zero_()
    elif isinstance(module, nn.Embedding):
        module.weight.data.normal_(mean=0.0, std=std)
        if module.padding_idx is not None:
            module.weight.data[module.padding_idx].zero_()

def main_generate_smaller_model():
    """
    ref: https://stackoverflow.com/questions/76971761/how-to-adapt-llama-v2-model-to-less-than-7b-parameters
    """
    print('Starting to reinitialize the model...')
    # Load the pretrained LLaMA v2 config
    config = AutoConfig.from_pretrained("meta-llama/Llama-2-7b-hf")
    print(f'config: {config} {type(config)}')
    # Print the original number of parameters
    model = AutoModelForCausalLM.from_config(config)
    print("Original number of parameters:", sum(p.numel() for p in model.parameters()))

    # Modify the config to reduce size
    config.hidden_size = 2048
    config.num_hidden_layers = 12

    # Create new smaller model from modified config
    smaller_model = AutoModelForCausalLM.from_config(config)
    print("New number of parameters:", sum(p.numel() for p in smaller_model.parameters()))

if __name__ == '__main__':
    import time
    start = time.time()
    # main_generate_smaller_model()
    main_reinit_model()
    print('Done!\a\a\a')

输出结果 #

Starting to reinitialize the model...
model.device=device(type='cuda', index=0)
norm_model=tensor(1.0779e+08, device='cuda:0', grad_fn=<AddBackward0>)
norm_model=tensor(1.0779e+08, device='cuda:0', grad_fn=<AddBackward0>)
Done!

解决方案 #

1. 权重未更新的核心原因 #

你的代码中L1范数未变化，主要问题有两个：

• 仅初始化了Linear层，遗漏了Embedding、LayerNorm等LLaMA核心权重层
• 直接修改module.weight时，未明确操作张量的.data属性，可能被计算图机制干扰；同时模型在CUDA上时，需确保修改的是设备上的张量

2. 符合Hugging Face LLaMA实现的初始化方法 #

基于Hugging Face公开的LLaMA源码逻辑，覆盖所有关键层的初始化代码如下：

def correct_reinitialize_weights(model):
    std = model.config.initializer_range  # LLaMA v2默认值为0.02
    for module in model.modules():
        if isinstance(module, nn.Linear):
            nn.init.normal_(module.weight.data, mean=0.0, std=std)
            if module.bias is not None:
                nn.init.constant_(module.bias.data, 0.0)
        elif isinstance(module, nn.Embedding):
            nn.init.normal_(module.weight.data, mean=0.0, std=std)
            if module.padding_idx is not None:
                module.weight.data[module.padding_idx].zero_()
        elif isinstance(module, nn.LayerNorm):
            nn.init.constant_(module.weight.data, 1.0)
            nn.init.constant_(module.bias.data, 0.0)

3. 修复后的完整测试流程 #

修改main_reinit_model，确保范数计算避开计算图干扰：

def main_reinit_model():
    print('Starting to reinitialize the model...')
    config = AutoConfig.from_pretrained("meta-llama/Llama-2-7b-hf")
    model = AutoModelForCausalLM.from_config(config)
    model = model.to('cuda')
    print(f'{model.device=}')
    
    # 计算初始L1范数（取数值避开计算图）
    initial_norm = sum(p.data.norm(1).item() for p in model.parameters())
    print(f'initial_norm={initial_norm:.4e}')
    
    # 执行正确的初始化
    correct_reinitialize_weights(model)
    
    # 计算初始化后的L1范数
    final_norm = sum(p.data.norm(1).item() for p in model.parameters())
    print(f'final_norm={final_norm:.4e}')

4. 关于LLaMA官方初始化参数 #

Meta官方未公开LLaMA的完整初始化细节（属于商业保密内容），但Hugging Face的实现基于公开论文和行业通用验证逻辑，核心参数为：

• Linear/Embedding层：正态分布N(0, 0.02)
• LayerNorm层：权重初始化为1，偏置初始化为0
• 所有偏置项：初始化为0

内容的提问来源于stack exchange，提问作者Charlie Parker