Python操作GCP存储桶覆盖YAML文件时CRC32C校验不匹配问题

I'm using the Python 3 Google Cloud Storage client to download a YAML file, modify its content, and upload it back to overwrite the original. Here's my code:

from google.cloud import storage
import yaml
client = storage.Client()
bucket = client.get_bucket('bucket_name')
blob = bucket.blob('my_rules_file.yaml')
yaml_file = blob.download_as_string()
doc = yaml.load(yaml_file, Loader=yaml.FullLoader)
doc['email'].clear()
doc['email'].extend(["test@gmail.com"])
yaml_file = yaml.dump(doc)
blob.upload_from_string(yaml_file, content_type="application/octet-stream")

When running the final upload line, I get this error:

BadRequest: 400 POST https://storage.googleapis.com/upload/storage/v1/b/fc-sandbox-datastore/o?uploadType=multipart: { "error": { "code": 400, "message": "Provided CRC32C \"YXQoSg==\" doesn't match calculated CRC32C \"EyDHsA==\".", "errors": [ { "message": "Provided CRC32C \"YXQoSg==\" doesn't match calculated CRC32C \"EyDHsA==\".", "domain": "global", "reason": "invalid" }, { "message": "Provided MD5 hash \"G/rQwQii9moEvc3ZDqW2qQ==\" doesn't match calculated MD5 hash \"GqyZzuvv6yE57q1bLg8HAg==\".", "domain": "global", "reason": "invalid" } ] } } : ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>)

This issue only happens when working with YAML files. What's causing this?

This error boils down to a mismatch between how the GCS client calculates checksums for string inputs and the actual byte content sent to Google Cloud Storage—something that pops up specifically with YAML files because of how the serialization process works.

What's Going On #

When you run yaml.dump(doc), you get a Python Unicode string (str type). The GCS client's upload_from_string method automatically converts this string to bytes using UTF-8 encoding, then calculates CRC32C and MD5 checksums based on those bytes to send along with the request.

For YAML files, though, the serialization process often produces byte content that's logically equivalent to the original but not exactly identical (think differences in line endings, indentation spacing, quote styles, or even key order). While this shouldn't matter for functionality, the strict checksum validation in GCS flags a mismatch if the client's calculated checksums don't perfectly match what GCS computes from the received bytes. In some cases, there can also be edge cases with how Python handles string-to-byte conversion that throw off the checksum calculation.

Fixes to Try #

Here are two reliable ways to resolve this:

1. Explicitly Encode the YAML Output to Bytes

By converting the dumped YAML string to bytes yourself, you remove any ambiguity in how the GCS client processes the content. This ensures the checksums are calculated on the exact bytes that get uploaded:

from google.cloud import storage
import yaml

client = storage.Client()
bucket = client.get_bucket('bucket_name')
blob = bucket.blob('my_rules_file.yaml')

# Download and parse the YAML file
yaml_file = blob.download_as_string()
doc = yaml.load(yaml_file, Loader=yaml.FullLoader)

# Modify the content
doc['email'].clear()
doc['email'].extend(["test@gmail.com"])

# Explicitly encode the dumped string to UTF-8 bytes
yaml_bytes = yaml.dump(doc).encode('utf-8')

# Upload the bytes directly (use the correct YAML MIME type)
blob.upload_from_string(yaml_bytes, content_type="application/yaml")

2. Use a BytesIO Stream with upload_from_file

If you want to avoid any string encoding quirks entirely, wrap the YAML bytes in a BytesIO object and use upload_from_file instead. This treats the content as a binary file, which eliminates checksum mismatches:

from google.cloud import storage
import yaml
from io import BytesIO

client = storage.Client()
bucket = client.get_bucket('bucket_name')
blob = bucket.blob('my_rules_file.yaml')

yaml_file = blob.download_as_string()
doc = yaml.load(yaml_file, Loader=yaml.FullLoader)

doc['email'].clear()
doc['email'].extend(["test@gmail.com"])

yaml_bytes = yaml.dump(doc).encode('utf-8')

# Use BytesIO to simulate a file stream for upload
with BytesIO(yaml_bytes) as stream:
    blob.upload_from_file(stream, content_type="application/yaml")

Quick Note #

I swapped application/octet-stream for application/yaml as the content type—this is optional, but it sets proper metadata for your YAML file in GCS. If your original file uses a non-UTF-8 encoding (like Latin-1), replace utf-8 with that encoding when calling .encode().

内容的提问来源于stack exchange，提问作者Kaushik J