通过面向S3的API Gateway代理读取S3存储桶Feather文件报错求助

原始报错信息 #

ine 239, in read_table
    reader = _feather.FeatherReader(source, use_memory_map=memory_map)
  File "pyarrow\_feather.pyx", line 75, in pyarrow._feather.FeatherReader.__cinit__
  File "pyarrow\error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow\error.pxi", line 114, in pyarrow.lib.check_status
OSError: Verification of flatbuffer-encoded Footer failed.

报错根因 #

API Gateway默认将非配置内的二进制媒体类型按文本编码转码，篡改了Feather文件的二进制内容，导致FlatBuffer编码的Footer校验失败，本地读取正常说明源文件无损坏，问题完全出在API Gateway的二进制响应配置上。

需调整的API Gateway配置 #

• 在API Gateway全局「设置」页，将application/octet-stream添加到二进制媒体类型列表
• 检查对应S3代理集成的「内容处理」参数，设置为CONVERT_TO_BINARY
• 确认S3存储桶内对应Feather文件的元数据Content-Type为application/octet-stream，若为文本类类型需修改
• 配置调整完成后，重新部署API到对应生效阶段，未部署的配置不会生效

优化后的请求代码 #

import pandas as pd
import requests
import io

resp = requests.get(
    'https://<api_gateway>/final/s3?key=naxi143/data.feather',
    headers={'Accept': 'application/octet-stream'}
)
# 提前校验请求状态，避免把错误响应内容当做Feather文件解析
resp.raise_for_status()
mem_fh = io.BytesIO(resp.content)
s3_data = pd.read_feather(mem_fh)

内容的提问来源于stack exchange，提问作者Naxi