使用datasets库加载SQuAD数据集时遇FileNotFoundError求助

问题描述 #

在Python中使用datasets库加载SQuAD数据集时触发FileNotFoundError，代码如下：

from datasets import load_dataset
dataset = load_dataset("squad")

错误提示显示代码尝试在本地目录/root/retraining-free-pruning/squad中寻找数据集，而非从远程下载，完整错误栈：

Traceback (most recent call last): File "", line 1, in
File "/root/miniconda3/envs/env/lib/python3.7/site-packages/datasets/load.py", line 1797, in load_dataset config_kwargs, File "/root/miniconda3/envs/env/lib/python3.7/site-packages/datasets/load.py", line 1520, in load_dataset_builder
data_files=data_files, File "/root/miniconda3/envs/env/lib/python3.7/site-packages/datasets/load.py", line 1164, in dataset_module_factory
path, data_dir=data_dir, data_files=data_files, download_mode=download_mode
File "/root/miniconda3/envs/env/lib/python3.7/site-packages/datasets/load.py", line 645, in get_module
allowed_extensions=ALL_ALLOWED_EXTENSIONS,
File "/root/miniconda3/envs/env/lib/python3.7/site-packages/datasets/data_files.py", line 798, in from_local_or_remote
if not isinstance(patterns_for_key, DataFilesList)
File "/root/miniconda3/envs/env/lib/python3.7/site-packages/datasets/data_files.py", line 748, in from_local_or_remote
data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions) File "/root/miniconda3/envs/env/lib/python3.7/site-packages/datasets/data_files.py", line 355, in resolve_patterns_locally_or_by_urls
raise FileNotFoundError(error_msg) FileNotFoundError: Unable to resolve any data file that matches '['']' at /root/retraining-free-pruning/squad with any supported extension ['.csv', '.tsv', '.json', '.jsonl', '.parquet', '.arrow', '.txt', '.blp', '.bmp', '.dib', '.bufr', '.cur', '.pcx', '.dcx', '.dds', '.ps', '.eps', '.fit', '.fits', '.fli', '.flc', '.ftc', '.ftu', '.gbr', '.gif', '.grib', '.h5', '.hdf', '.png', '.apng', '.jp2', '.j2k', '.jpc', '.jpf', '.jpx', '.j2c', '.icns', '.ico', '.im', '.iim', '.tif', '.tiff', '.jfif', '.jpe', '.jpg', '.jpeg', '.mpg', '.mpeg', '.msp', '.pcd', '.pxr', '.pbm', '.pgm', '.ppm', '.pnm', '.psd', '.bw', '.rgb', '.rgba', '.sgi', '.ras', '.tga', '.icb', '.vda', '.vst', '.webp', '.wmf', '.emf', '.xbm', '.xpm', '.BLP', '.BMP', '.DIB', '.BUFR', '.CUR', '.PCX', '.DCX', '.DDS', '.PS', '.EPS', '.FIT', '.FITS', '.FLI', '.FLC', '.FTC', '.FTU', '.GBR', '.GIF', '.GRIB', '.H5', '.HDF', '.PNG', '.APNG', '.JP2', '.J2K', '.JPC', '.JPF', '.JPX', '.J2C', '.ICNS', '.ICO', '.IM', '.IIM', '.TIF', '.TIFF', '.JFIF', '.JPE', '.JPG', '.JPEG', '.MPG', '.MPEG', '.MSP', '.PCD', '.PXR', '.PBM', '.PGM', '.PPM', '.PNM', '.PSD', '.BW', '.RGB', '.RGBA', '.SGI', '.RAS', '.TGA', '.ICB', '.VDA', '.VST', '.WEBP', '.WMF', '.EMF', '.XBM', '.XPM', '.aiff', '.au', '.avr', '.caf', '.flac', '.htk', '.svx', '.mat4', '.mat5', '.mpc2k', '.ogg', '.paf', '.pvf', '.raw', '.rf64', '.sd2', '.sds', '.ircam', '.voc', '.w64', '.wav', '.nist', '.wavex', '.wve', '.xi', '.mp3', '.opus', '.AIFF', '.AU', '.AVR', '.CAF', '.FLAC', '.HTK', '.SVX', '.MAT4', '.MAT5', '.MPC2K', '.OGG', '.PAF', '.PVF', '.RAW', '.RF64', '.SD2', '.SDS', '.IRCAM', '.VOC', '.W64', '.WAV', '.NIST', '.WAVEX', '.WVE', '.XI', '.MP3', '.OPUS', '.zip']

环境信息：

• Python版本：3.7
• datasets库版本：2.13.2
• 代码在Colab正常运行，但在AutoDL云GPU平台失败；加载Glue数据集时可找到远程链接，但加载SQuAD时仅提示本地文件缺失。

解决步骤 #

1. 明确指定数据集来源 #

load_dataset会优先在本地目录查找同名文件夹，当前工作目录下的squad文件夹导致逻辑混淆，强制指定从远程Hub加载：

from datasets import load_dataset
dataset = load_dataset("huggingface.co/datasets/squad")

2. 删除本地同名文件夹 #

如果当前工作目录下的squad文件夹是误创建的，直接删除该文件夹，再重新运行原始代码即可恢复默认远程下载逻辑。

3. 强制重新下载覆盖缓存 #

若缓存损坏导致异常，指定download_mode参数强制重新下载：

from datasets import load_dataset, DownloadMode
dataset = load_dataset("squad", download_mode=DownloadMode.REDOWNLOAD)

4. 升级datasets库版本 #

当前使用的datasets 2.13.2在Python 3.7环境下可能存在场景兼容问题，尝试升级到适配的稳定版本：

pip install --upgrade datasets==2.14.0

5. 手动指定缓存路径 #

若平台默认缓存路径存在权限问题，手动指定可读写的缓存目录：

from datasets import load_dataset
dataset = load_dataset("squad", cache_dir="/path/to/your/writable/cache")

内容的提问来源于stack exchange，提问作者Rtttt