加载infoslack/mistral-7b-arxiv-paper-chunked数据集遇路径模式错误

问题重现 #

运行以下代码加载Hugging Face数据集时触发错误：

from datasets import load_dataset

dataset = load_dataset("infoslack/mistral-7b-arxiv-paper-chunked", split="train")

错误提示：
ValueError: 无效模式：'**'只能作为完整的路径组件

完整报错栈（中文翻译）：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[21], line 3
      1 from datasets import load_dataset
----> 3 dataset = load_dataset("infoslack/mistral-7b-arxiv-paper-chunked", split="train")

文件 ~\AppData\Local\anaconda3\Lib\site-packages\datasets\load.py:1773，在 load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   1768 verification_mode = VerificationMode(
   1769     (verification_mode or VerificationMode.BASIC_CHECKS) if not save_infos else VerificationMode.ALL_CHECKS
   1770 )
   1772 # 创建数据集构建器
-> 1773 builder_instance = load_dataset_builder(
   1774     path=path,
   1775     name=name,
   1776     data_dir=data_dir,
   1777     data_files=data_files,
   1778     cache_dir=cache_dir,
   1779     features=features,
   1780     download_config=download_config,
   1781     download_mode=download_mode,
   1782     revision=revision,
   1783     use_auth_token=use_auth_token,
   1784     storage_options=storage_options,
   1785     **config_kwargs,
   1786 )
   1788 # 若为流式返回则返回可迭代数据集
   1789 if streaming:

文件 ~\AppData\Local\anaconda3\Lib\site-packages\datasets\load.py:1502，在 load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, storage_options, **config_kwargs)
   1500     download_config = download_config.copy() if download_config else DownloadConfig()
   1501     download_config.use_auth_token = use_auth_token
-> 1502 dataset_module = dataset_module_factory(
   1503     path=path,
   1504     revision=revision,
   1505     download_config=download_config,
   1506     download_mode=download_mode,
   1507     data_dir=data_dir,
   1508     data_files=data_files,
   1509 )
   1511 # 从处理脚本中获取数据集构建器类
   1512 builder_cls = import_main_class(dataset_module.module_path)

文件 ~\AppData\Local\anaconda3\Lib\site-packages\datasets\load.py:1219，在 dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1214             if isinstance(e1, FileNotFoundError):
   1215                 raise FileNotFoundError(
   1216                     f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1217                     f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1218                 ) from None
-> 1219             raise e1 from None
   1220 else:
   1221     raise FileNotFoundError(
   1222         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory."
   1223     )

文件 ~\AppData\Local\anaconda3\Lib\site-packages\datasets\load.py:1203，在 dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1188         return HubDatasetModuleFactoryWithScript(
   1189             path,
   1190             revision=revision,
   (...)
   1193             dynamic_modules_path=dynamic_modules_path,
   1194         ).get_module()
   1195     else:
   1196         return HubDatasetModuleFactoryWithoutScript(
   1197             path,
   1198             revision=revision,
   1199             data_dir=data_dir,
   1200             data_files=data_files,
   1201             download_config=download_config,
   1202             download_mode=download_mode,
-> 1203         ).get_module()
   1204 except (
   1205     Exception
   1206 ) as e1:  # noqa: all the attempts failed, before raising the error we should check if the module is already cached.
   1207     try:

文件 ~\AppData\Local\anaconda3\Lib\site-packages\datasets\load.py:769，在 HubDatasetModuleFactoryWithoutScript.get_module(self) -> DatasetModule:
    759 def get_module(self) -> DatasetModule:
    760     hfh_dataset_info = HfApi(config.HF_ENDPOINT).dataset_info(
    761         self.name,
    762         revision=self.revision,
    763         token=self.download_config.use_auth_token,
    764         timeout=100.0,
    765     )
    766     patterns = (
    767         sanitize_patterns(self.data_files)
    768         if self.data_files is not None
-> 769         else get_data_patterns_in_dataset_repository(hfh_dataset_info, self.data_dir)
    770     )
    771     data_files = DataFilesDict.from_hf_repo(
    772         patterns,
    773         dataset_info=hfh_dataset_info,
    774         base_path=self.data_dir,
    775         allowed_extensions=ALL_ALLOWED_EXTENSIONS,
    776     )
    777     split_modules = {
    778         split: infer_module_for_data_files(data_files_list, use_auth_token=self.download_config.use_auth_token)
    779         for split, data_files_list in data_files.items()
    780     }

文件 ~\AppData\Local\anaconda3\Lib\site-packages\datasets\data_files.py:662，在 get_data_patterns_in_dataset_repository(dataset_info, base_path)
    660 resolver = partial(_resolve_single_pattern_in_dataset_repository, dataset_info, base_path=base_path)
    661 try:
-> 662     return _get_data_files_patterns(resolver)
    663 except FileNotFoundError:
    664     raise EmptyDatasetError(
    665         f"The dataset repository at '{dataset_info.id}' doesn't contain any data files"
    666     ) from None

文件 ~\AppData\Local\anaconda3\Lib\site-packages\datasets\data_files.py:223，在 _get_data_files_patterns(pattern_resolver)
    221 try:
    222     for pattern in patterns:
-> 223         data_files = pattern_resolver(pattern)
    224         if len(data_files) > 0:
    225             non_empty_splits.append(split)

文件 ~\AppData\Local\anaconda3\Lib\site-packages\datasets\data_files.py:473，在 _resolve_single_pattern_in_dataset_repository(dataset_info, pattern, base_path, allowed_extensions)
    471 else:
    472     base_path = "/"
-> 473 glob_iter = [PurePath(filepath) for filepath in fs.glob(PurePath(pattern).as_posix()) if fs.isfile(filepath)]
    474 matched_paths = [
    475     filepath
    476     for filepath in glob_iter
   (...)
    483     )
    484 ]  # 忽略.ipynb和__pycache__，保留/../
    485 if allowed_extensions is not None:

文件 ~\AppData\Roaming\Python\Python311\site-packages\fsspec\spec.py:606，在 AbstractFileSystem.glob(self, path, maxdepth, **kwargs)
    602         depth = None
    604 allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
-> 606 pattern = glob_translate(path + ("/" if ends_with_sep else ""))
    607 pattern = re.compile(pattern)
    609 out = {
    610     p: info
    611     for p, info in sorted(allpaths.items())
   (...)
    618     )
    619 }

文件 ~\AppData\Roaming\Python\Python311\site-packages\fsspec\utils.py:734，在 glob_translate(pat)
    732     continue
    733 elif "**" in part:
-> 734     raise ValueError(
    735         "Invalid pattern: '**' can only be an entire path component"
    736     )
    737 if part:
    738     results.extend(_translate(part, f"{not_sep}*", not_sep))

ValueError: 无效模式：'**'只能作为完整的路径组件

问题原因 #

该错误源于数据集仓库的自动文件检测逻辑生成了不符合fsspec glob规则的路径模式。fsspec要求**必须作为独立的路径组件（比如**/*.json是合法的，但data**/*.json这类包含部分路径的模式不合法），而目标数据集的文件结构导致自动检测生成了无效的模式。

解决方案 #

方案1：指定具体数据文件路径 #

直接在load_dataset中通过data_files参数指定合法的glob模式，绕过自动检测：

from datasets import load_dataset

# 指定数据集内的具体文件路径模式
dataset = load_dataset("infoslack/mistral-7b-arxiv-paper-chunked", split="train", data_files="**/*.json")

方案2：本地克隆后加载 #

如果远程加载仍有问题，可以先手动克隆数据集仓库到本地，再从本地路径加载：

git clone https://huggingface.co/datasets/infoslack/mistral-7b-arxiv-paper-chunked

然后本地加载：

dataset = load_dataset("./mistral-7b-arxiv-paper-chunked", split="train")

内容的提问来源于stack exchange，提问作者tmrgn