调用Hugging Face load_dataset加载GLUE MRPC数据集时遇类型错误

运行以下代码加载GLUE的MRPC数据集时触发类型错误：

from datasets import load_dataset
from transformers import AutoTokenizer, DataCollatorWithPadding
from torch.utils.data import DataLoader

raw_datasets = load_dataset("glue", "mrpc")

报错信息如下：

Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151k/151k [00:00<00:00, 3.35MB/s]
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.1k/11.1k [00:00<00:00, 6.63MB/s]
Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:32<00:00, 10.89s/it]
Extracting data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 127.92it/s]
Traceback (most recent call last):
  File "/Users/ameenizhac/Downloads/transformers_playground.py", line 5, in <module>
    raw_datasets = load_dataset("glue", "mrpc")
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datasets/load.py", line 1782, in load_dataset
    builder_instance.download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datasets/builder.py", line 872, in download_and_prepare
    self._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datasets/builder.py", line 967, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datasets/builder.py", line 1709, in _prepare_split
    split_info = self.info.splits[split_generator.name]
                 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datasets/splits.py", line 530, in __getitem__
    instructions = make_file_instructions(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datasets/arrow_reader.py", line 112, in make_file_instructions
    name2filenames = {
                     ^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datasets/arrow_reader.py", line 113, in <dictcomp>
    info.name: filenames_for_dataset_split(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datasets/naming.py", line 70, in filenames_for_dataset_split
    prefix = filename_prefix_for_split(dataset_name, split)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datasets/naming.py", line 54, in filename_prefix_for_split
    if os.path.basename(name) != name:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 142, in basename
TypeError: expected str, bytes or os.PathLike object, not NoneType

错误原因分析 #

报错栈显示，问题出在datasets库的文件命名处理逻辑中——os.path.basename()接收到None值，触发类型错误。大概率是数据集缓存文件损坏、库版本兼容性问题，或下载过程异常中断导致的。

解决方案 #

• 清理损坏的缓存文件
  找到datasets默认缓存目录（~/.cache/huggingface/datasets），删除其中glue/mrpc相关文件夹后重新运行代码。
• 升级datasets库到最新版本
  执行命令修复已知bug：

  pip install --upgrade datasets
  

• 指定新缓存目录重新下载
  临时指定新缓存路径，避开旧的损坏文件：

  raw_datasets = load_dataset("glue", "mrpc", cache_dir="./new_cache_dir")
  

• 检查Python版本兼容性
  当前使用的Python 3.11可能与旧版datasets库存在兼容问题，升级库后仍报错的话，可尝试切换到Python 3.10版本。

内容的提问来源于stack exchange，提问作者Ameen Izhac