如何将Gmail的.xlsx附件上传至AWS S3桶？代码报错排查

问题概述 #

尝试用Python从Gmail提取主题包含“Zillion”和“payment information”的.xlsx附件，上传至AWS S3桶时多次报错：

• 使用upload_fileobj方法时提示ValueError: Fileobj must implement read
• 改用put_object方法则出现ParamValidationError参数Body类型错误
  注释上传代码后，其余逻辑可正常运行并获取附件文件名。

原代码 #

import googleapiclient.discovery
import os.path
import boto3
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow

SCOPES = ['https://mail.google.com/']

def get_credentials():
    if os.path.exists('/home/matillion/gmail_token.json'):
        creds = Credentials.from_authorized_user_file('/home/matillion/gmail_token.json', SCOPES)
    return creds

service = googleapiclient.discovery.build('gmail', 'v1', credentials=get_credentials())

inbox = service.users().messages().list(userId='me', maxResults=2).execute()

for message in inbox['messages']:
    msg = service.users().messages().get(userId='me', id=message['id']).execute()
    payload = msg['payload']
    headers = payload['headers']
    parts = payload['parts']

    for d in headers:
        if d['name'] == 'Subject':
            subject = d['value']

    if 'Zillion' and 'payment information' in subject:
        print(subject)

        for attachment in msg['payload']['parts']:
            if attachment['filename'].endswith('.xlsx'):
                # Get the attachment file name
                filename = attachment['filename']
                # Get the attachment body
                attachment_body = attachment['body']

                # Download the attachment to the S3 bucket
                s3_client = boto3.client('s3')
                s3_client.upload_fileobj(attachment_body, 'super-etl-matillion-east', '/client/From_Client/Payment/' + filename)

        # Delete the email
        #service.users().messages().delete(userId='me', id=message['id']).execute()

        # Print the filename
        print(filename)

报错信息 #

使用upload_fileobj时的报错 #

Traceback (most recent call last):
  File "/tmp/interpreter-input-c83147df-dde3-4343-b927-48d97c404da8.tmp", line 41, in <module>
    s3_client.upload_fileobj(attachment_body, 'super-etl-matillion-east', '/client/From_Client/Payment/' + filename)
  File "/usr/local/lib/python3.6/site-packages/boto3/s3/inject.py", line 618, in upload_fileobj
    raise ValueError('Fileobj must implement read')
ValueError: Fileobj must implement read

使用put_object时的报错 #

Fwd: FW: Zillion - payment information
Traceback (most recent call last):
  File "/tmp/interpreter-input-cf44abe6-4df1-495c-bd9e-7ff21ae8a1b4.tmp", line 41, in <module>
    s3_client.put_object(Body=attachment_body,Bucket='super-etl-matillion-east',Key='client/From_Client/Payment/' + filename)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 692, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 740, in _convert_to_request_dict
    api_params, operation_model)
  File "/usr/local/lib/python3.6/site-packages/botocore/validate.py", line 360, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Body, value: {'attachmentId': 'ANGjdJ_80Rt2EoNqoLnokG_ygvdEnBzBGtCQIr-5idUxO5iVHQzgxL3fuLejJo3lJKsZYRhAfrjrPDsCX9_fBKeg7q1nT8-Ao1oCyxKWqAyN-6zcdKOo93QvoY-8-L5dAMj_2HBH3ZBxoqB8ei4OaiJKYtIPs3cbbrQ2XyTX2xaWRrP26PmNHDV6abDl5I395yjtKIoJVDbaxag28VMFD5P0UYcQFb-nypmGkQh0OpLbYQkd2Q65qz0sTJSeuBfKDD-ooEEJSWrMF4pWdpsq-N2dKqD5SGX9oAr6Udk6SFUa2BbZRknHb3jfXQdyRA0gRy1FO5vmihqAYULQ23WntO4xvw536oVdyRmj5ouQqvCkWdZC2c9yI_xNpQgHxBfkt3mvinyHeyUQVSrJYnFf', 'size': 33477}, type: <class 'dict'>, valid types: <class 'bytes'>, <class 'bytearray'>, file-like object
Script failed with status: 1

问题原因与解决方案 #

核心问题 #

Gmail API返回的attachment['body']不是实际的文件内容，而是一个包含attachmentId的字典。必须通过这个ID调用专门的API接口，才能获取到附件的base64编码内容，解码后才能上传到S3。

另外原代码中主题判断逻辑有误：if 'Zillion' and 'payment information' in subject 会被解析为if ('Zillion') and ('payment information' in subject)，即使主题不含Zillion，只要有payment information就会触发，需修正为同时判断两个关键词都存在。

修改后的完整代码 #

import googleapiclient.discovery
import os.path
import boto3
import base64
from io import BytesIO
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow

SCOPES = ['https://mail.google.com/']

def get_credentials():
    if os.path.exists('/home/matillion/gmail_token.json'):
        creds = Credentials.from_authorized_user_file('/home/matillion/gmail_token.json', SCOPES)
    return creds

service = googleapiclient.discovery.build('gmail', 'v1', credentials=get_credentials())
s3_client = boto3.client('s3')  # 提前初始化，避免循环内重复创建

inbox = service.users().messages().list(userId='me', maxResults=2).execute()

for message in inbox['messages']:
    msg = service.users().messages().get(userId='me', id=message['id']).execute()
    payload = msg['payload']
    headers = payload['headers']
    subject = ""

    for d in headers:
        if d['name'] == 'Subject':
            subject = d['value']
            break

    # 修正主题判断逻辑
    if 'Zillion' in subject and 'payment information' in subject:
        print(subject)

        for attachment in msg['payload']['parts']:
            if attachment['filename'].endswith('.xlsx'):
                filename = attachment['filename']
                attachment_id = attachment['body']['attachmentId']
                
                # 通过attachmentId获取实际附件内容
                attachment_data = service.users().messages().attachments().get(
                    userId='me',
                    messageId=message['id'],
                    id=attachment_id
                ).execute()
                
                # 解码base64数据为bytes
                file_content = base64.urlsafe_b64decode(attachment_data['data'])
                
                # 方法1：使用put_object上传bytes
                s3_client.put_object(
                    Body=file_content,
                    Bucket='super-etl-matillion-east',
                    Key=f'client/From_Client/Payment/{filename}'  # 去掉Key开头的斜杠，S3键不需要开头/
                )
                
                # 方法2：使用upload_fileobj，需要把bytes转成BytesIO文件对象
                # file_obj = BytesIO(file_content)
                # s3_client.upload_fileobj(file_obj, 'super-etl-matillion-east', f'client/From_Client/Payment/{filename}')
                
                print(f"已上传附件：{filename}")

        # Delete the email
        #service.users().messages().delete(userId='me', id=message['id']).execute()

关键修改点说明 #

1. 获取真实附件内容：调用users().messages().attachments().get()接口，传入attachmentId获取base64编码的附件数据。
2. 解码数据：用base64.urlsafe_b64decode()把API返回的data字段解码为二进制bytes，这是S3接受的格式。
3. 修正主题判断：确保两个关键词都存在于主题中。
4. 优化S3客户端初始化：把s3_client = boto3.client('s3')移到循环外避免重复创建连接。
5. 修正S3 Key格式：去掉Key开头的斜杠，S3对象键不需要以/开头。

内容的提问来源于stack exchange，提问作者James Eichelberger