谷歌云存储大量小图片以实现快速批量访问的方案选型

Great question—dealing with duplicate storage for ML datasets across VMs can get pricey fast, especially when you’re running multiple instances daily. Here are the top cost-effective solutions tailored to your use case on Google Cloud:

1. Google Cloud Storage (GCS) + gcsfuse Mounting #

This is probably the most budget-friendly option for your scenario. Instead of copying datasets to individual persistent disks, store all your image datasets in a single GCS bucket and mount it directly on each VM using gcsfuse (a FUSE adapter for GCS).

How to implement:

• Create a GCS bucket (use the Standard storage class since you access data daily—Nearline/Coldline are cheaper but have higher access fees).
• Upload your dataset to the bucket once using gsutil cp -r ./local-dataset gs://my-ml-datasets/.
• On each VM, install gcsfuse and mount the bucket:

  sudo apt-get install gcsfuse
  mkdir -p /mnt/ml-dataset
  gcsfuse my-ml-datasets /mnt/ml-dataset
  

Pros:

• No data duplication—all VMs access the same source, cutting storage costs drastically.
• Scalable: GCS handles millions of small files seamlessly.
• Cost-efficient: Standard storage costs ~$0.026/GB/month, which is cheaper than persistent disks.
• Add caching flags like --cache-max-size 10G to keep frequently accessed images in VM memory, reducing GCS API calls and speeding up loads.

2. Shared Read-Only Regional Persistent Disk #

If you need lower latency than GCS can provide (since block storage is faster than object storage for small file access), a shared read-only regional PD is a solid alternative.

How to implement:

• Create a single Regional Persistent Disk in your VM’s region.
• Attach it to a temporary VM, format it, and copy your dataset onto it.
• Detach it from the temporary VM, then attach it as read-only to all your ML VMs (Google allows multiple VMs to attach a PD in read-only mode).

Pros:

• Faster access to small images compared to GCS.
• Only one PD to manage and pay for, instead of one per VM.
• Works well if your workload needs consistent low-latency access.

Cons:

• Limited to VMs in the same region.
• Slightly more expensive than GCS (~$0.04/GB/month for regional PD).

3. Cloud Filestore (Managed NFS) #

If you ever need shared read-write access (e.g., for updating datasets across VMs), Cloud Filestore is a managed NFS service that works seamlessly with Google VMs.

How to implement:

• Create a Filestore instance in your region.
• Mount the NFS share on each VM using standard NFS commands.

Pros:

• Fully managed, so you don’t have to maintain NFS servers.
• POSIX-compliant, making it easy to integrate with existing ML workflows.

Cons:

• More expensive than both GCS and shared PDs, so it’s overkill if you only need read access.

Recommendation #

For your specific use case (incremental loading of small images, daily access across multiple VMs), GCS + gcsfuse is the best balance of cost and functionality. If low latency is critical for your ML workloads, go with the shared read-only regional PD.

内容的提问来源于stack exchange，提问作者Tom