通过ApiGateway下载NoSQL二进制文件遇Base64解码错误求助

原始CloudFormation配置 #

DownloadMethodGet:
  Type: AWS::ApiGateway::Method
  Properties:
    HttpMethod: GET
    ApiKeyRequired: true
    ResourceId: !Ref OfflineExtractorJobIdDownloadApiResource
    RestApiId: !Ref RestApi
    AuthorizationType: CUSTOM
    AuthorizerId: !Ref RestApiAuthorizer
    RequestParameters:
      method.request.path.jobid: true
      method.request.header.Accept: false
      method.request.header.Content-Type: false
      method.request.header.Range: false
    Integration:
      Type: AWS
      CacheKeyParameters:
        - 'method.request.path.jobid'
      RequestParameters:
        integration.request.path.jobid: 'method.request.path.jobid'
        integration.request.header.Accept: 'method.request.header.Accept'
        integration.request.header.Content-Type: 'method.request.header.Content-Type'
        integration.request.header.Range: 'method.request.header.Range'
      Uri: !Sub "${DownloadS3BucketArn}/export-{jobid}.db"
      Credentials: !Ref DownloadS3RoleName
      IntegrationHttpMethod: GET
      PassthroughBehavior: WHEN_NO_MATCH
      IntegrationResponses:
        - StatusCode: 200
          ContentHandling: CONVERT_TO_BINARY
          ResponseParameters:
            method.response.header.Content-Type: "integration.response.header.Content-Type"
            method.response.header.Content-Range: "integration.response.header.Content-Range"
          ResponseTemplates:
            application/json: ''
      MethodResponses:
        - StatusCode: 200
          ResponseModels:
            application/json: "Empty"
          ResponseParameters:
            method.response.header.Content-Type: true
            method.response.header.Content-Range: false

测试错误日志 #

Sun Dec 11 06:24:35 UTC 2022 : Endpoint request body after transformations: 
Sun Dec 11 06:24:35 UTC 2022 : Sending request to https://s3-external-1.amazonaws.com/the-bucket/export-the-nosql-file.db
Sun Dec 11 06:24:35 UTC 2022 : Received response. Status: 200, Integration latency: 279 ms
Sun Dec 11 06:24:35 UTC 2022 : Endpoint response headers: {x-amz-id-2=BDwDmSffhPUoXGUB/VxQxZ5XVGufOcxa5an7gmwK9O2zXJROFOGHh/H0BRYPAabt6pVPR9EMubg=, x-amz-request-id=JNQ0Q84HENXZJTPH, Date=Sun, 11 Dec 2022 06:24:36 GMT, Last-Modified=Wed, 13 Jul 2022 08:57:15 GMT, ETag="6f7f45c10069472887d4306164fc84cd", Accept-Ranges=bytes, Content-Type=application/octet-stream, Server=AmazonS3, Content-Length=7102464}
Sun Dec 11 06:24:36 UTC 2022 : Endpoint response body before transformations: SQLite format 3@  (�Y [TRUNCATED]
Sun Dec 11 06:24:36 UTC 2022 : Execution failed due to configuration error: Unable to base64 decode the body.
Sun Dec 11 06:24:36 UTC 2022 : Method completed with status: 500

调整ContentHandling后的测试情况 #

将ContentHandling改为CONVERT_TO_TEXT后，API Gateway测试UI可正常运行，但Curl/Postman请求仍返回200但无内容，且响应Content-Type始终为application/json：

Execution log for request xxx
Sun Dec 11 09:47:02 UTC 2022 : Starting execution for request: xxx
Sun Dec 11 09:47:02 UTC 2022 : HTTP Method: GET, Resource Path: /offline-extractor/jobs2/the-file/download
Sun Dec 11 09:47:02 UTC 2022 : Method request path: {jobid=0014aa18-d3f7-4508-a5df-63fda8573dfc}
Sun Dec 11 09:47:02 UTC 2022 : Method request query string: {}
Sun Dec 11 09:47:02 UTC 2022 : Method request headers: {}
Sun Dec 11 09:47:02 UTC 2022 : Method request body before transformations: 
Sun Dec 11 09:47:02 UTC 2022 : Endpoint request URI: https://s3-external-1.amazonaws.com/acuttera-offline-extractor-prod/export-the-file.db
Sun Dec 11 09:47:02 UTC 2022 : Endpoint request headers: {Authorization=*****************************************************************************************************************************************************************************************************************************************************************************13e589, X-Amz-Date=20221211T094702Z, x-amzn-apigateway-api-id=xxx, Accept=application/json, User-Agent=AmazonAPIGateway_hoypjdvhx7, X-Amz-Security-Token=IQ [TRUNCATED]
Sun Dec 11 09:47:02 UTC 2022 : Endpoint request body after transformations: 
Sun Dec 11 09:47:02 UTC 2022 : Sending request to https://s3-external-1.amazonaws.com/the-bucket/export-the-file.db
Sun Dec 11 09:47:02 UTC 2022 : Received response. Status: 200, Integration latency: 67 ms
Sun Dec 11 09:47:02 UTC 2022 : Endpoint response headers: {x-amz-id-2=xxx x-amz-request-id=xxx, Date=Sun, 11 Dec 2022 09:47:03 GMT, Last-Modified=Sun, 10 Jul 2022 11:43:24 GMT, ETag="xxx", Accept-Ranges=bytes, Content-Type=application/octet-stream, Server=AmazonS3, Content-Length=638976}
Sun Dec 11 09:47:02 UTC 2022 : Endpoint response body before transformations: SQLite format 3@  ��U [TRUNCATED]
Sun Dec 11 09:47:02 UTC 2022 : Method response body after transformations: SQLite format 3@  ��U    [TRUNCATED]
Sun Dec 11 09:47:02 UTC 2022 : Method response headers: {X-Amzn-Trace-Id=Root=1-xxx, Content-Type=application/octet-stream}
Sun Dec 11 09:47:02 UTC 2022 : Successfully completed execution
Sun Dec 11 09:47:02 UTC 2022 : Method completed with status: 200

问题核心原因 #

1. CONVERT_TO_BINARY配置下，API Gateway尝试对S3返回的二进制内容进行base64解码，但部分二进制文件的编码格式触发了解码错误
2. 保留的ResponseTemplates: application/json: ''强制响应转为JSON格式，导致二进制内容被清空
3. MethodResponses绑定的application/json: "Empty"模型，让API Gateway默认返回JSON Content-Type，忽略了S3返回的application/octet-stream

修改后的CloudFormation配置 #

DownloadMethodGet:
  Type: AWS::ApiGateway::Method
  Properties:
    HttpMethod: GET
    ApiKeyRequired: true
    ResourceId: !Ref OfflineExtractorJobIdDownloadApiResource
    RestApiId: !Ref RestApi
    AuthorizationType: CUSTOM
    AuthorizerId: !Ref RestApiAuthorizer
    RequestParameters:
      method.request.path.jobid: true
      method.request.header.Accept: true  # 强制接收客户端Accept头，保证内容类型匹配
      method.request.header.Content-Type: false
      method.request.header.Range: false
    Integration:
      Type: AWS
      CacheKeyParameters:
        - 'method.request.path.jobid'
      RequestParameters:
        integration.request.path.jobid: 'method.request.path.jobid'
        integration.request.header.Accept: 'method.request.header.Accept'
        integration.request.header.Content-Type: 'method.request.header.Content-Type'
        integration.request.header.Range: 'method.request.header.Range'
      Uri: !Sub "${DownloadS3BucketArn}/export-{jobid}.db"
      Credentials: !Ref DownloadS3RoleName
      IntegrationHttpMethod: GET
      PassthroughBehavior: WHEN_NO_MATCH
      IntegrationResponses:
        - StatusCode: 200
          ContentHandling: CONVERT_TO_BINARY
          ResponseParameters:
            method.response.header.Content-Type: "integration.response.header.Content-Type"
            method.response.header.Content-Range: "integration.response.header.Content-Range"
          # 移除JSON响应模板，避免强制转换内容格式
    MethodResponses:
      - StatusCode: 200
        # 移除JSON模型绑定，允许传递原始二进制内容类型
        ResponseParameters:
          method.response.header.Content-Type: true
          method.response.header.Content-Range: false

关键修改说明 #

1. 将method.request.header.Accept设为true，确保客户端发送的Accept头被传递到S3，让S3返回匹配的Content-Type
2. 移除IntegrationResponses中的ResponseTemplates，避免强制将二进制内容转为JSON格式
3. 移除MethodResponses中的ResponseModels，取消JSON模型绑定，让API Gateway直接返回S3的原始响应类型
4. 恢复ContentHandling: CONVERT_TO_BINARY，确保API Gateway正确处理二进制文件，避免文本转换导致的内容损坏

内容的提问来源于stack exchange，提问作者Tomáš Mika