Python加载LLaMA模型Tokenizer报错求助：路径正确仍无法加载

目录结构说明 #

• LLaMA模板路径：/home/wisehipoppotamus/LLAMA
• 目录内容：
  • 4个对应不同参数的模型文件夹：7B、13B、30B、65B
  • 两个tokenizer相关文件：tokenizer.model、tokenizer_checklist.chk

运行代码 #

import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('/home/wisehipoppotamus/LLAMA')
model = GPT2LMHeadModel.from_pretrained('/home/wisehipoppotamus/LLAMA/65B')


input_text = "What are penguins?"
encoded_input = tokenizer.encode(input_text, return_tensors='en')
output = model.generate(encoded_input, max_length=200, num_return_sequences=1)
decoded_output = tokenizer.decode(output[0], skip_special_tokens=True)
print(decoded_output)

错误信息 #

python3 ai_model_test.py
Traceback (most recent call last):
  File "/home/wisehipoppotamus/ai_model_test.py", line 4, in <module>
    tokenizer = GPT2Tokenizer.from_pretrained('/home/wisehipoppotamus/LLAMA')
  File "/home/wisehipoppotamus/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1788, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for '/home/wisehipoppotamus/LLAMA'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/home/wisehipoppotamus/LLAMA' is the correct path to a directory containing all relevant files for a GPT2Tokenizer tokenizer.

已确认tokenizer和模型的路径均正确，但无法理解该错误原因，恳请提供帮助。谢谢。

内容的提问来源于stack exchange，提问作者WiseHipoppotamus