使用Xgboost 1.7-1训练数据时遭遇tar空文件错误求助

环境与问题 #

使用XGBoost 1.7-1版本在SageMaker上训练，两个结构一致的CSV格式数据集，调用训练时触发以下错误：

2024-01-19:02:57:27:INFO] Imported framework sagemaker_xgboost_container.training
[2024-01-19:02:57:27:INFO] No GPUs detected (normal if no gpus installed)
[2024-01-19:02:57:27:INFO] Invoking user training script.
[2024-01-19:02:57:27:ERROR] Reporting training FAILURE
[2024-01-19:02:57:27:ERROR] framework error: 
Traceback (most recent call last):
  File "/miniconda3/lib/python3.8/tarfile.py", line 2318, in next
    tarinfo = self.tarinfo.fromtarfile(self)
  File "/miniconda3/lib/python3.8/tarfile.py", line 1105, in fromtarfile
    obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
  File "/miniconda3/lib/python3.8/tarfile.py", line 1041, in frombuf
    raise EmptyHeaderError("empty header")
tarfile.EmptyHeaderError: empty header
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_trainer.py", line 84, in train
    entrypoint()
  File "/miniconda3/lib/python3.8/site-packages/sagemaker_xgboost_container/training.py", line 102, in main
    train(framework.training_env())
  File "/miniconda3/lib/python3.8/site-packages/sagemaker_xgboost_container/training.py", line 87, in train
    framework.modules.run_module(
  File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_modules.py", line 290, in run_module
    _files.download_and_extract(uri, _env.code_dir)
  File "/miniconda3/lib/python3.8/site-packages/sagemaker_containers/_files.py", line 131, in download_and_extract
    with tarfile.open(name=dst, mode="r:gz") as t:
  File "/miniconda3/lib/python3.8/tarfile.py", line 1621, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/miniconda3/lib/python3.8/tarfile.py", line 1674, in gzopen
    t = cls.taropen(name, mode, fileobj, **kwargs)
  File "/miniconda3/lib/python3.8/tarfile.py", line 1651, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/miniconda3/lib/python3.8/tarfile.py", line 1514, in __init__
    self.firstmember = self.next()
  File "/miniconda3/lib/python3.8/tarfile.py", line 2333, in next
    raise ReadError("empty file")
tarfile.ReadError: empty file
empty file

原因分析 #

这个报错和CSV数据集无关，问题出在SageMaker容器尝试下载并解压训练代码包时，发现对应的tar.gz文件为空。可能的触发场景：

• 上传到S3的代码tar包本身就是空文件（大小为0字节）
• 打包训练代码时操作失误，未将脚本文件包含进去
• 训练任务配置中指定的代码包S3 URI错误，指向了不存在或空的对象

解决步骤 #

• 检查代码包完整性：本地解压准备的tar包，确认包含训练脚本（如train.py）及必要依赖文件。打包时确保在脚本所在目录执行命令，例如：

  tar -czf training-code.tar.gz .
  

  注意末尾的.代表打包当前目录下所有内容，避免只打包空文件夹。
• 验证S3文件状态：登录S3控制台，找到代码包对应的存储路径，查看文件大小。如果显示为0字节，重新上传正确的tar包。
• 核对训练任务配置：确认SageMaker训练任务的entry_point、source_dir参数配置正确，没有指向错误的代码资源路径。
• 重新生成代码包：如果之前打包过程有误，重新打包训练代码，确保脚本和依赖都被包含在内后再上传到S3。

内容的提问来源于stack exchange，提问作者dewdrops