使用Hugging Face parquet-converter遇异常，求正确文件结构

我有包含PNG图片的train.zip压缩包和用于标注的metadata.csv文件，想让Hugging Face Hub上的parquet-converter机器人自动识别并处理数据集。尝试了两种文件结构：

• 根目录放置train.zip与metadata.csv
• train/train.zip加根目录metadata.csv
  但均触发以下异常：

Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code: StreamingRowsError
Exception: ValueError
Message: One or several metadata.csv were found, but not in the same directory or in a parent directory of zip://1.png::hf://datasets/[user]/[repo-name]@[hash]/train/train.zip.
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/job_runners/split/first_rows.py", line 322, in compute
compute_first_rows_from_parquet_response(
File "/src/services/worker/src/worker/job_runners/split/first_rows.py", line 88, in compute_first_rows_from_parquet_response
rows_index = indexer.get_rows_index(
File "/src/libs/libcommon/src/libcommon/parquet_utils.py", line 640, in get_rows_index
return RowsIndex(
File "/src/libs/libcommon/src/libcommon/parquet_utils.py", line 521, in init
self.parquet_index = self._init_parquet_index(
File "/src/libs/libcommon/src/libcommon/parquet_utils.py", line 538, in _init_parquet_index
response = get_previous_step_or_raise(
File "/src/libs/libcommon/src/libcommon/simple_cache.py", line 590, in get_previous_step_or_raise
raise CachedArtifactError(
libcommon.simple_cache.CachedArtifactError: The previous step failed.

During handling of the above exception, another exception occurred:
          
          Traceback (most recent call last):
            File "/src/services/worker/src/worker/utils.py", line 96, in get_rows_or_raise
              return get_rows(
            File "/src/libs/libcommon/src/libcommon/utils.py", line 197, in decorator
              return func(*args, **kwargs)
            File "/src/services/worker/src/worker/utils.py", line 73, in get_rows
              rows_plus_one = list(itertools.islice(ds, rows_max_number + 1))
            File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 1389, in __iter__
              for key, example in ex_iterable:
            File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 234, in __iter__
              yield from self.generate_examples_fn(**self.kwargs)
            File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/folder_based_builder/folder_based_builder.py", line 376, in _generate_examples
              raise ValueError(
          ValueError: One or several metadata.csv were found, but not in the same directory or in a parent directory of zip://1.png::hf://datasets/[user]/[repo-name]@[hash]/train/train.zip.

错误原因 #

异常提示明确说明：metadata.csv不在zip包内图片文件的同级或父目录下。parquet-converter解析zip内的图片时，会将图片的路径视为zip://xxx/train/train.zip/1.png这类“虚拟路径”，此时放在zip外部根目录的metadata.csv，并不属于这个虚拟路径的同级或父目录层级，因此无法关联图片和标注信息。

正确文件结构 #

你可以选择以下两种方式调整：

1. 将metadata.csv打包进train.zip内
   让zip包的内部结构为：

   train.zip
   ├── 1.png
   ├── 2.png
   └── metadata.csv
   

   之后将这个train.zip放在数据集根目录，或者train子目录均可。

2. 不使用zip压缩，直接放置文件
   把PNG图片和metadata.csv放在同一目录下，比如根目录的train文件夹：

   train/
   ├── 1.png
   ├── 2.png
   └── metadata.csv
   

   这种结构下机器人能直接识别到同级的标注文件，完成关联处理。

内容的提问来源于stack exchange，提问作者Artyom Ionash