微调deepset/bert-base-cased-squad2时遇CUDA错误求助

问题描述 #

使用Hugging Face Transformers库微调deepset/bert-base-cased-squad2模型时，触发以下错误：

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

尝试过的无效方案 #

• 将模型词汇量调整为与Tokenizer的词汇量一致
• 尝试更小的批量大小（8、4甚至1）
• 更新PyTorch和Transformers库

相关代码片段 #

主代码（错误版本）

from transformers import AutoModelForQuestionAnswering, AutoTokenizer, Trainer, TrainingArguments

# 加载模型与不匹配的tokenizer
model = AutoModelForQuestionAnswering.from_pretrained("deepset/bert-base-cased-squad2")
tokenizer = AutoTokenizer.from_pretrained("错误的tokenizer路径或名称")

training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=8,
    num_train_epochs=3,
)

# Trainer初始化未传入正确tokenizer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()

train_model函数（错误版本）

def train_model(model, train_dataset, training_args):
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        # 缺失tokenizer参数
    )
    trainer.train()

完整错误堆栈 #

Traceback (most recent call last):
  File "train.py", line 45, in <module>
    train_model(model, train_dataset, training_args)
  File "train.py", line 30, in train_model
    trainer.train()
  File "/usr/local/lib/python3.8/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.8/site-packages/transformers/trainer.py", line 1880, in inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/usr/local/lib/python3.8/site-packages/transformers/trainer.py", line 2776, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.8/site-packages/transformers/trainer.py", line 2808, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 1695, in forward
    outputs = self.bert(
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 994, in forward
    embedding_output = self.embeddings(
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 233, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return self._backend.Embedding.apply(
  File "/usr/local/lib/python3.8/site-packages/torch/nn/_functions/linear.py", line 184, in forward
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

问题根源与解决方法 #

根源

未向Trainer传递正确的Tokenizer，导致数据预处理生成的输入张量形状与模型嵌入层预期不匹配，进而触发CUDA层面的CUBLAS初始化错误。

解决方法

1. 确保加载与模型匹配的Tokenizer：使用deepset/bert-base-cased-squad2对应的Tokenizer
2. 在初始化Trainer时，正确传入tokenizer参数，保证数据预处理逻辑与模型输入要求一致

修正后的Trainer初始化代码：

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,  # 添加正确的tokenizer参数
)

内容的提问来源于stack exchange，提问作者tt40kiwi