GCP挂载Cloud Storage写入多文件触发OSError: [Errno 5] I/O错误咨询

Let's break down what's happening here—this is a common gotcha when using gcsfuse to mount Cloud Storage buckets as a file system, and it all comes down to how object storage differs from traditional local disks.

First, let's look at your code pattern:

for file_number in range(10):
    for i in range(1000):
        with open('./bucket_storage/output/file_' + str(file_number) + '.txt', 'a') as output_file:
            output_file.write("Some text")

The critical issue here is that you're opening, writing a tiny snippet of text, and closing the same file 1000 times per file number. This plays extremely poorly with how gcsfuse and Cloud Storage operate under the hood.

Why the error hits after a few files #

Cloud Storage is an object storage system, not a block-based file system like your Compute Engine's local disk. gcsfuse acts as a translation layer (via FUSE) to make it look like a local file system, but every file operation maps directly to Cloud Storage API calls:

• When you open a file in append mode and close it, gcsfuse can't just write to a local block—it has to either:
  1. Download the existing object from Cloud Storage, append your new text, then re-upload the entire updated object; or
  2. Use Cloud Storage's appendable object feature, which has strict limits on request frequency and chunk size.

Your code spams 1000 tiny API requests per file. For the first 2-3 files, the request volume is low enough that gcsfuse can keep up with syncing to the cloud. But as you keep going:

• The FUSE layer's IO queue gets backed up with too many small, frequent operations, leading to timeouts.
• You hit implicit rate limits on Cloud Storage API calls (even without an explicit rate-limit error, the sheer volume causes conflicts or unhandled race conditions).
• gcsfuse's local cache can't keep up with rapid state changes, leading to inconsistencies that trigger the OSError: Errno 5 (input/output error).

Why writing locally first works #

When you write to a local Compute Engine disk, you're using a block storage system designed for frequent small writes. The OS handles caching and writing efficiently, and when you move the file to the bucket, it's a single large upload operation—way easier for gcsfuse and Cloud Storage to handle without hitting IO bottlenecks.

Fixes and deeper explanations #

The fix you already considered (opening the file once before the inner loop) works because it reduces API interactions to one per file (or a small number, if gcsfuse batches writes):

for file_number in range(10):
    with open('./bucket_storage/output/file_' + str(file_number) + '.txt', 'a') as output_file:
        for i in range(1000):
            output_file.write("Some text")

Now, all 1000 writes are buffered locally by gcsfuse, and only synced to Cloud Storage once when you close the file. This avoids the flood of tiny API requests that were causing the IO error.

If you need to do frequent small writes for some reason, you can also tweak gcsfuse's mount parameters to optimize for this:

• Use --write-buffer-size to set a larger buffer (e.g., --write-buffer-size=64MiB) so gcsfuse waits until more data is accumulated before syncing to the cloud.
• Increase --max-cache-size to give gcsfuse more local storage for caching pending writes.

Alternatively, consider using the official google-cloud-storage Python client directly instead of gcsfuse—it lets you control exactly when and how data is uploaded to Cloud Storage, which can be more efficient for patterns like this.

内容的提问来源于stack exchange，提问作者Rocke Fella