使用S3FS向AWS S3上传文件时遇MalformedXML错误求助

问题场景 #

在已配置IAM角色权限的EC2实例上，通过Python的s3fs库连接AWS S3并上传文本文件时，触发MalformedXML错误。执行aws s3 ls命令可正常列出存储桶并访问目标桶，但上传操作失败。

错误信息 #

warnings.warn(
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/s3fs/core.py", line 112, in _error_wrapper
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aiobotocore/client.py", line 358, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (MalformedXML) when calling the PutObject operation: The XML you provided was not well-formed or did not validate against our published schema

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/airflow/twitter_dag/twitter_etl.py", line 46, in <module>
    run_twitter_etl()
    self.commit()
  File "/usr/local/lib/python3.10/dist-packages/s3fs/core.py", line 2188, in commit
    write_result = self._call_s3(
  File "/usr/local/lib/python3.10/dist-packages/s3fs/core.py", line 2040, in _call_s3
    return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/fsspec/asyn.py", line 113, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/fsspec/asyn.py", line 98, in sync
    raise return_result
  File "/usr/local/lib/python3.10/dist-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.10/dist-packages/s3fs/core.py", line 339, in _call_s3
    return await _error_wrapper(
  File "/usr/local/lib/python3.10/dist-packages/s3fs/core.py", line 139, in _error_wrapper
    raise err
OSError: [Errno 22] The XML you provided was not well-formed or did not validate against our published schema

涉事代码 #

s3 = s3fs.S3FileSystem(anon=False)
with s3.open(f"<bucket-name>/<filename>.txt", 'w') as f:
    f.write('Hello')

解决方法 #

• 校验存储桶与对象键格式：确保<bucket-name>符合S3命名规则（仅小写字母、数字、连字符，不能以连字符首尾），<filename>.txt无特殊字符（如空格、中文、非常规符号），避免因命名不规范导致请求XML解析失败。
• 显式指定AWS区域：初始化S3FileSystem时添加region_name参数，匹配你的存储桶所在区域，示例：

  s3 = s3fs.S3FileSystem(anon=False, region_name='us-east-1')
  

  部分场景下s3fs无法自动识别区域，会导致请求发往错误端点，引发XML格式错误。
• 升级依赖库版本：旧版本的s3fs、botocore或aiobotocore可能存在兼容性问题，执行命令升级：

  pip install --upgrade s3fs botocore aiobotocore
  

• 改用二进制写入模式：将文件打开模式改为wb，写入字节数据，避免文本编码处理引发的异常：

  s3 = s3fs.S3FileSystem(anon=False)
  with s3.open(f"<bucket-name>/<filename>.txt", 'wb') as f:
      f.write(b'Hello')
  

内容的提问来源于stack exchange，提问作者coder2911