通过SageMaker上传本地文件至S3时遇FileNotFoundError问题排查

问题描述 #

我是AWS新手，希望通过SageMaker Studio中的JupyterLab将本地文件夹上传至Amazon S3存储。目前可通过JupyterLab的上传图标手动完成操作，但使用以下代码执行时出现FileNotFoundError：

import sagemaker
from sagemaker.tuner import (
    IntegerParameter,
    CategoricalParameter,
    ContinuousParameter,
    HyperparameterTuner,
)

sagemaker_session = sagemaker.Session()
region = sagemaker_session.boto_region_name
bucket = sagemaker_session.default_bucket()
prefix = "sagemaker/my-first-proj"
role = sagemaker.get_execution_role()

local_dir = "/Users/tomi/DevProjects/WeThePeople/datasets"
inputs = sagemaker_session.upload_data(path=local_dir, bucket=bucket, key_prefix=prefix)

执行代码后报错信息如下：

FileNotFoundError                         Traceback (most recent call last)
Cell In[2], line 2
      1 local_dir = "/Users/tomi/DevProjects/WeThePeople/datasets"
----> 2 inputs = sagemaker_session.upload_data(path=local_dir, bucket=bucket, key_prefix=prefix)
      3 print("input spec (in this case, just an S3 path): {}".format(inputs))

File /opt/conda/lib/python3.10/site-packages/sagemaker/session.py:400, in Session.upload_data(self, path, bucket, key_prefix, extra_args)
    397     s3 = self.s3_resource
    399 for local_path, s3_key in files:
--> 400     s3.Object(bucket, s3_key).upload_file(local_path, ExtraArgs=extra_args)
    402 s3_uri = "s3://{}/{}".format(bucket, key_prefix)
    403 # If a specific file was used as input (instead of a directory), we return the full S3 key
    404 # of the uploaded object. This prevents unintentionally using other files under the same
    405 # prefix during training.

File /opt/conda/lib/python3.10/site-packages/boto3/s3/inject.py:318, in object_upload_file(self, Filename, ExtraArgs, Callback, Config)
    287 def object_upload_file(
    288     self, Filename, ExtraArgs=None, Callback=None, Config=None
    289 ):
    290     """Upload a file to an S3 object.
    291 
    292     Usage::
   (...)
    316         transfer.
    317     """
--> 318     return self.meta.client.upload_file(
    319         Filename=Filename,
    320         Bucket=self.bucket_name,
    321         Key=self.key,
    322         ExtraArgs=ExtraArgs,
    323         Callback=Callback,
    324         Config=Config,
    325     )

File /opt/conda/lib/python3.10/site-packages/boto3/s3/inject.py:143, in upload_file(self, Filename, Bucket, Key, ExtraArgs, Callback, Config)
    108 """Upload a file to an S3 object.
    109 
    110 Usage::
   (...)
    140     transfer.
    141 """
    142 with S3Transfer(self, Config) as transfer:
--> 143     return transfer.upload_file(
    144         filename=Filename,
    145         bucket=Bucket,
    146         key=Key,
    147         extra_args=ExtraArgs,
    148         callback=Callback,
    149     )

File /opt/conda/lib/python3.10/site-packages/boto3/s3/transfer.py:292, in S3Transfer.upload_file(self, filename, bucket, key, callback, extra_args)
    288 future = self._manager.upload(
    289     filename, bucket, key, extra_args, subscribers
    290 )
    291 try:
--> 292     future.result()
    293 # If a client error was raised, add the backwards compatibility layer
    294 # that raises a S3UploadFailedError. These specific errors were only
    295 # ever thrown for upload_parts but now can be thrown for any related
    296 # client error.
    297 except ClientError as e:

File /opt/conda/lib/python3.10/site-packages/s3transfer/futures.py:103, in TransferFuture.result(self)
     98 def result(self):
     99     try:
    100         # Usually the result() method blocks until the transfer is done,
    101         # however if a KeyboardInterrupt is raised we want want to exit
    102         # out of this and propagate the exception.
--> 103         return self._coordinator.result()
    104     except KeyboardInterrupt as e:
    105         self.cancel()

File /opt/conda/lib/python3.10/site-packages/s3transfer/futures.py:266, in TransferCoordinator.result(self)
    263 # Once done waiting, raise an exception if present or return the
    264 # final result.
    265 if self._exception:
--> 266     raise self._exception
    267 return self._result

File /opt/conda/lib/python3.10/site-packages/s3transfer/tasks.py:269, in SubmissionTask._main(self, transfer_future, **kwargs)
    265     self._transfer_coordinator.set_status_to_running()
    267     # Call the submit method to start submitting tasks to execute the
    268     # transfer.
--> 269     self._submit(transfer_future=transfer_future, **kwargs)
    270 except BaseException as e:
    271     # If there was an exception raised during the submission of task
    272     # there is a chance that the final task that signals if a transfer
   (...)
    281 
    282     # Set the exception, that caused the process to fail.
    283     self._log_and_set_exception(e)

File /opt/conda/lib/python3.10/site-packages/s3transfer/upload.py:591, in UploadSubmissionTask._submit(self, client, config, osutil, request_executor, transfer_future, bandwidth_limiter)
    589 # Determine the size if it was not provided
    590 if transfer_future.meta.size is None:
--> 591     upload_input_manager.provide_transfer_size(transfer_future)
    593 # Do a multipart upload if needed, otherwise do a regular put object.
    594 if not upload_input_manager.requires_multipart_upload(
    595     transfer_future, config
    596 ):

File /opt/conda/lib/python3.10/site-packages/s3transfer/upload.py:244, in UploadFilenameInputManager.provide_transfer_size(self, transfer_future)
    242 def provide_transfer_size(self, transfer_future):
    243     transfer_future.meta.provide_transfer_size(
--> 244         self._osutil.get_file_size(transfer_future.meta.call_args.fileobj)
    245     )

File /opt/conda/lib/python3.10/site-packages/s3transfer/utils.py:247, in OSUtils.get_file_size(self, filename)
    246 def get_file_size(self, filename):
--> 247     return os.path.getsize(filename)

File /opt/conda/lib/python3.10/genericpath.py:50, in getsize(filename)
     48 def getsize(filename):
     49     """Return the size of a file, reported by os.stat()."""
--> 50     return os.stat(filename).st_size

FileNotFoundError: [Errno 2] No such file or directory: '/Users/tomi/DevProjects/WeThePeople/datasets'

已确认该路径在本地机器上存在，终端可正常访问：

>>> (WeThePeople) tomi@MacBook-Pro-4 datasets % pwd
/Users/tomi/DevProjects/WeThePeople/datasets

当前使用的SageMaker用户配置了AmazonS3FullAccess和AmazonSageMakerFullAccess权限，请问该问题的原因是什么？如何解决？是否还有其他未检查的权限设置？

原因与解决方案 #

核心原因 #

SageMaker Studio的JupyterLab运行在AWS云端的容器实例中，并非你的本地机器。代码中指定的/Users/tomi/DevProjects/WeThePeople/datasets是本地Mac的路径，云端实例无法访问该目录，这是报错的根本原因。

手动上传成功是因为JupyterLab的上传功能会先将本地文件传输到云端实例的临时存储，再同步到S3，和代码直接指定本地路径的逻辑完全不同。

解决方法 #

有两种常用方案可选：

1. 先上传到SageMaker Studio实例，再同步到S3

   • 使用JupyterLab的上传功能，将本地文件夹传到云端实例的某个路径（例如/home/sagemaker-user/datasets）
   • 修改代码中的local_dir为云端实例的路径：

     local_dir = "/home/sagemaker-user/datasets"
     inputs = sagemaker_session.upload_data(path=local_dir, bucket=bucket, key_prefix=prefix)
     

2. 在本地机器运行代码直接上传

   • 在本地安装AWS CLI并配置好凭证（关联你的SageMaker用户权限）
   • 安装sagemaker Python库，直接在本地运行代码，此时代码访问的是本地路径，可正常上传到S3

权限说明 #

你当前配置的AmazonS3FullAccess和AmazonSageMakerFullAccess权限已足够完成上传操作，报错与权限无关，无需额外配置权限。

内容的提问来源于stack exchange，提问作者tomigee