如何在Python中向Google Cloud Storage上传视频？含App Engine断点续传疑问

Hey there! Let's work through your video upload issue with Google Cloud Storage (GCS) from your App Engine Python app—especially focusing on resumable uploads, since that's usually the fix for large video upload failures.

Before diving into resumable uploads, let's make sure your foundation is solid:

• IAM Permissions: Ensure your App Engine service account has the Storage Object Creator (or more granular storage.objects.create and storage.objects.update) permissions on your GCS bucket. You can set this in the Cloud Console's IAM section.
• Dependencies: Install the google-cloud-storage library (add it to your requirements.txt for App Engine):

  pip install google-cloud-storage
  

Start with a simple upload to rule out basic issues like invalid paths or bucket access. This works for small videos (under ~100MB), but will fail for larger files due to timeouts or size limits:

from google.cloud import storage

def upload_small_video(bucket_name, source_file_path, dest_blob_name):
    """Basic upload for small video files."""
    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(dest_blob_name)

    # Set correct content type so GCS recognizes it as a video
    blob.content_type = "video/mp4"  # adjust to your format (e.g., video/mov)

    try:
        blob.upload_from_filename(source_file_path)
        print(f"Success! Video uploaded to gs://{bucket_name}/{dest_blob_name}")
        return True
    except Exception as e:
        print(f"Basic upload failed: {str(e)}")
        return False

If this fails, debug permissions, bucket existence, or file path validity first.

How Resumable Uploads Work #

Resumable uploads split your large video into smaller chunks (default 8MB) and upload them one at a time. If the upload gets interrupted (network drop, App Engine timeout), GCS keeps track of the chunks you've already uploaded. Next time you resume, it picks up where it left off—no need to reupload the entire file.

Python Implementation for App Engine #

Here's a robust resumable upload script tailored for App Engine:

from google.cloud import storage
from google.resumable_media.requests import ResumableUpload
from google.resumable_media.common import DataReader
import requests
import time

def resumable_upload_large_video(bucket_name, source_file_path, dest_blob_name):
    """Resumable upload for large video files (works around App Engine timeouts)."""
    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(dest_blob_name)
    blob.content_type = "video/mp4"

    # Initialize a resumable upload session with GCS
    upload_url = blob.create_resumable_upload_session()

    # Open the video file and create a reader for chunked uploads
    with open(source_file_path, "rb") as video_file:
        file_reader = DataReader(video_file)
        upload = ResumableUpload(upload_url, file_reader)

        # Start the upload session
        init_response = upload.initiate(
            headers={"Content-Type": blob.content_type},
            method="POST"
        )

        # Check if upload is already complete (rare for videos)
        if init_response.status_code == 200:
            print("Upload was already completed!")
            return True

        # Upload chunks until finished
        while not upload.finished:
            try:
                transmit_response = upload.transmit_next_chunk()
                # Optional: Track progress
                progress = (upload.bytes_uploaded / upload.total_bytes) * 100
                print(f"Upload progress: {progress:.1f}%")
            except requests.exceptions.RequestException as e:
                print(f"Upload interrupted—retrying in 5s: {str(e)}")
                time.sleep(5)
                continue

        print(f"Resumable upload complete! Video saved to gs://{bucket_name}/{dest_blob_name}")
        return True

    except Exception as e:
        print(f"Resumable upload failed: {str(e)}")
        return False

Key Tips for App Engine #

• Avoid Frontend Timeouts: App Engine standard environment has a 60-second frontend request limit. For large videos, offload the upload to a Cloud Task instead of running it in a user-facing request. This lets the upload run in the background without timing out.
• Adjust Chunk Size: You can customize the chunk size (e.g., chunk_size=32*1024*1024 for 32MB chunks) in the ResumableUpload constructor—tweak this based on your network stability.
• Save Session URLs: If your app might restart mid-upload, save the upload_url to a database. You can use this URL later to resume the upload without starting over.
• Retry Logic: The example uses a simple sleep-and-retry, but for production, use exponential backoff (try the tenacity library) to handle repeated failures gracefully.

• Permission Denied: Double-check your App Engine service account's IAM roles—resumable uploads need both create and update permissions on bucket objects.
• File Not Found: In App Engine, make sure your source file is stored in the /tmp directory (the only writable path in standard environment) if it's a user-uploaded file.
• Timeout Errors: As mentioned, use Cloud Tasks for large uploads to bypass frontend timeouts.

内容的提问来源于stack exchange，提问作者Dannick Montanede