使用transformers.Trainer微调stablelm-3b-4e1t时遇导入错误求助

基于stablelm-3b-4e1t模型，使用TweetSumm数据集，借助PEFT库进行微调训练，但调用transformers.Trainer时出现导入错误。

微调代码 #

# Freezing the original weights
for param in model.parameters():
  param.requires_grad = False  # freeze the model - train adapters later
  if param.ndim == 1:
    # cast the small parameters (e.g. layernorm) to fp32 for stability
    param.data = param.data.to(torch.float32)

model.enable_input_require_grads()

class CastOutputToFloat(nn.Sequential):
  def forward(self, x): return super().forward(x).to(torch.float32)
model.lm_head = CastOutputToFloat(model.lm_head)

# Setting up the LoRa Adapters
def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )

from peft import LoraConfig, get_peft_model 

config = LoraConfig(
    r=16, # attention heads
    lora_alpha=32, # alpha scaling
    target_modules=["q_proj", "v_proj"], 
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM" 
)

model = get_peft_model(model, config)
print_trainable_parameters(model)

# Training
trainer = transformers.Trainer(
    model=model, 
    train_dataset=dataset["train"],
    args=transformers.TrainingArguments(
        per_device_train_batch_size=4, 
        gradient_accumulation_steps=4,
        warmup_steps=100, 
        max_steps=200, 
        learning_rate=2e-4, 
        fp16=True,
        logging_steps=1, 
        output_dir='outputs'
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
model.config.use_cache = False  # silence the warnings. Please re-enable for inference!
trainer.train()

报错信息 #

ImportError                               Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py in _get_module(self, module_name)
   1352         self.__all__ = list(import_structure.keys()) + list(chain(*import_structure.values()))
-> 1353         self.__file__ = module_file
   1354         self.__spec__ = module_spec

24 frames
ImportError: cannot import name 'ACCELERATE_MIN_VERSION' from 'transformers.utils' (/usr/local/lib/python3.10/dist-packages/transformers/utils/__init__.py)

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
RuntimeError: Failed to import transformers.integrations.integration_utils because of the following error (look up to see its traceback):
cannot import name 'ACCELERATE_MIN_VERSION' from 'transformers.utils' (/usr/local/lib/python3.10/dist-packages/transformers/utils/__init__.py)

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py in _get_module(self, module_name)
   1353         self.__file__ = module_file
   1354         self.__spec__ = module_spec
-> 1355         self.__path__ = [os.path.dirname(module_file)]
   1356         self._objects = {} if extra_objects is None else extra_objects
   1357         self._name = name

RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):
Failed to import transformers.integrations.integration_utils because of the following error (look up to see its traceback):
cannot import name 'ACCELERATE_MIN_VERSION' from 'transformers.utils' (/usr/local/lib/python3.10/dist-packages/transformers/utils/__init__.py)

问题原因及解决方法 #

原因

该错误是由于transformers库与accelerate库版本不兼容，或者transformers版本过旧，导致transformers.utils模块中不存在ACCELERATE_MIN_VERSION常量，进而引发后续的模块导入失败。

解决步骤

1. 卸载现有版本的transformers和accelerate：

pip uninstall -y transformers accelerate

2. 安装兼容的稳定版本（推荐经过验证的版本组合）：

pip install transformers==4.35.2 accelerate==0.24.1

或者直接安装最新稳定版：

pip install --upgrade transformers accelerate

内容的提问来源于stack exchange，提问作者Zahra Reyhanian