如何通过Google Cloud AI Jupyter Notebook(Python)连接GCP云存储读写指定路径文件

Hey there! If you're working in a Google Cloud AI Jupyter Notebook and need to interact with files in the bucket_name bucket under the user/test folder, I've got you covered. Below's a step-by-step breakdown of how to authenticate, connect, and perform read/write operations.

Prerequisites #

• You're working in a Google Cloud AI Platform Jupyter Notebook (hosted on GCP) — this means you already have access to your GCP project's resources (assuming the notebook's service account has permissions for GCS).
• The google-cloud-storage library is installed (it's usually pre-installed, but we'll cover installing it just in case).

Step 1: Verify Authentication #

In most cases, GCP-hosted notebooks are automatically authenticated with a service account tied to your project. To confirm this works, you can run a quick test to list your buckets:

from google.cloud import storage

# Initialize the GCS client
client = storage.Client()

# List all buckets in your project to confirm auth works
for bucket in client.list_buckets():
    print(bucket.name)

If you see your bucket_name listed here, you're good to go. If not, you can manually authenticate by running this in a notebook cell:

!gcloud auth application-default login

Follow the prompts to log in with your GCP account, and you'll be authenticated.

Step 2: Install the GCS Client Library (If Needed) #

If the google-cloud-storage library isn't already installed, run this in a notebook cell:

!pip install google-cloud-storage

Step 3: Read/Write Files in bucket_name/user/test #

Let's dive into the actual file operations. We'll cover writing a file to the path, reading it back, and listing files in the folder.

Writing a File to bucket_name/user/test #

Suppose you want to write a text file, or upload a local file from your notebook to GCS. Here's how:

Option 1: Write a string directly to a GCS file

from google.cloud import storage

client = storage.Client()
bucket = client.get_bucket('bucket_name')

# Define the full path in the bucket (folder + filename)
blob_path = 'user/test/my_test_file.txt'
blob = bucket.blob(blob_path)

# Write a string to the file
content = "Hello from GCP AI Notebook!"
blob.upload_from_string(content)

print(f"Successfully wrote to {blob_path} in bucket {bucket.name}")

Option 2: Upload a local file from your notebook to GCS

If you have a file saved in your notebook's local filesystem (e.g., local_file.csv), you can upload it like this:

from google.cloud import storage

client = storage.Client()
bucket = client.get_bucket('bucket_name')

blob_path = 'user/test/uploaded_file.csv'
blob = bucket.blob(blob_path)

# Upload from local file path
blob.upload_from_filename('local_file.csv')

print(f"Uploaded local file to {blob_path}")

Reading a File from bucket_name/user/test #

To read the content of a file from the GCS path, use these methods:

Option 1: Read file content as a string

from google.cloud import storage

client = storage.Client()
bucket = client.get_bucket('bucket_name')

blob_path = 'user/test/my_test_file.txt'
blob = bucket.blob(blob_path)

# Read content as string
file_content = blob.download_as_string().decode('utf-8')
print("File content:")
print(file_content)

Option 2: Download the file to your notebook's local filesystem

If you want to save the GCS file locally:

from google.cloud import storage

client = storage.Client()
bucket = client.get_bucket('bucket_name')

blob_path = 'user/test/uploaded_file.csv'
blob = bucket.blob(blob_path)

# Download to local path
blob.download_to_filename('downloaded_file.csv')

print(f"Downloaded {blob_path} to local file downloaded_file.csv")

List All Files in bucket_name/user/test #

If you want to see what's in the user/test folder, you can list all blobs under that prefix:

from google.cloud import storage

client = storage.Client()
bucket = client.get_bucket('bucket_name')

# Prefix is the folder path (make sure it ends with a slash to match only the folder)
prefix = 'user/test/'

# List all blobs in the folder
blobs = bucket.list_blobs(prefix=prefix)

print(f"Files in {prefix}:")
for blob in blobs:
    print(blob.name)

Important Notes #

• Make sure the service account associated with your notebook has the Storage Object Admin (or appropriate) permissions on the bucket_name bucket. If you run into permission errors, check the IAM settings for your bucket in the GCP Console.
• Folder paths in GCS are just prefixes — there's no actual "folder" structure, but using prefixes works exactly like folders for most operations.

内容的提问来源于stack exchange，提问作者Adrish Ray