使用S3预签名URL上传原始字符串（CSV/JSON）失败求助

Let's get to the bottom of why your raw string uploads are failing while file uploads work. The core issue here comes down to how S3 validates pre-signed URL requests, and how curl handles request bodies and headers when using --data vs --upload-file.

Why Your Current Raw String Calls Fail #

When you use --upload-file, curl sends the file content as-is, with a correctly calculated Content-Length and (often) an auto-detected Content-Type that aligns with S3's expectations. But when you use --data:

1. curl defaults to setting Content-Type: application/x-www-form-urlencoded unless you explicitly override it. If your pre-signed URL was generated with a different Content-Type (or no restriction), this mismatch triggers S3's signature validation failure.
2. --data automatically escapes certain characters (like newlines), which alters the request body's actual length—another critical check S3 uses to validate pre-signed requests.

Fixing the PUT Request (No Need to Switch to POST) #

You don't have to switch to POST. Just adjust your curl command to match the parameters used to generate the pre-signed URL. Here are the correct formats:

1. If your pre-signed URL was generated with Content-Type: text/plain #

Use --data-binary to send the raw string without escaping, and explicitly set the matching Content-Type header:

curl --request PUT -H "Content-Type: text/plain" --data-binary "this is raw data" "pre-signed-upload-url"

2. If your pre-signed URL has no Content-Type restriction #

You can omit the header, but still use --data-binary to ensure the request body is sent exactly as written:

curl --request PUT --data-binary "this is raw data" "pre-signed-upload-url"

Quick Check: Verify Your Pre-Signed URL Generation #

Double-check your Lambda code that generates the pre-signed URL. If you're using AWS SDKs like boto3, ensure:

• If you want to restrict Content-Type, include it in the Params argument of generate_presigned_url:

  # Example boto3 code with Content-Type restriction
  presigned_url = s3_client.generate_presigned_url(
      'put_object',
      Params={
          'Bucket': 'your-bucket',
          'Key': 'your-object-key',
          'ContentType': 'text/plain'  # Enforces this header in the request
      },
      ExpiresIn=3600
  )
  

• If you want to allow any content type, remove the ContentType parameter from Params.

If You Do Want to Switch to POST #

While PUT is simpler for raw content, if you need to use POST, you'll need to generate a pre-signed POST form (instead of a single URL) in your Lambda. The curl command would look like this:

curl --request POST \
  -F "key=your-object-key" \
  -F "AWSAccessKeyId=your-access-key" \
  -F "policy=your-generated-policy" \
  -F "signature=your-post-signature" \
  -F "file=this is raw data" \
  "https://your-bucket.s3.amazonaws.com"

Note: Your Lambda will need to return all required form fields (key, policy, signature, etc.) along with the bucket URL, not just a single pre-signed URL.

内容的提问来源于stack exchange，提问作者credizian