如何在Google Cloud ML中存储Keras训练模型并跨运行从存储桶加载？

Got it, let's walk through exactly how to load that Keras model you saved to your specified Cloud Storage path in subsequent Google Cloud ML runs. The approach mirrors how you saved the model—we'll use GCP's file I/O tools to pull the model from GCS to a local temp path first, then load it with Keras.

Step-by-Step Implementation #

First, here's a reusable function you can use to load your model. We'll stick with TensorFlow's file_io module (matching your save code) to handle the GCS file transfer:

from tensorflow.keras.models import load_model
from tensorflow.python.lib.io import file_io
import os

def load_keras_model_from_gcs(gcs_model_path):
    # Define a local temporary path (GCP ML jobs have access to /tmp for temp storage)
    local_temp_path = "/tmp/model.h5"
    
    # Copy the model file from GCS to local temp storage
    with file_io.FileIO(gcs_model_path, mode='rb') as gcs_file:
        with file_io.FileIO(local_temp_path, mode='wb+') as local_file:
            local_file.write(gcs_file.read())
    
    # Load the model using Keras
    trained_model = load_model(local_temp_path)
    
    # Optional: Clean up the local temp file if you don't need it anymore
    os.remove(local_temp_path)
    
    return trained_model

# Call the function with your exact GCS model path (match what you used when saving)
# Example: If data_folder was "gs://your-bucket/custom-models/", then:
model = load_keras_model_from_gcs(data_folder + "model.h5")

Alternative Using TensorFlow's tf.io.gfile (TF 2.x+) #

If you're using TensorFlow 2.x or later, you can also use the more modern tf.io.gfile API for the file transfer:

import tensorflow as tf
from tensorflow.keras.models import load_model
import os

def load_keras_model_from_gcs(gcs_model_path):
    local_temp_path = "/tmp/model.h5"
    
    with tf.io.gfile.GFile(gcs_model_path, 'rb') as gcs_file:
        with open(local_temp_path, 'wb') as local_file:
            local_file.write(gcs_file.read())
    
    trained_model = load_model(local_temp_path)
    os.remove(local_temp_path)
    return trained_model

Key Notes to Remember #

• Path Consistency: Double-check that the gcs_model_path you pass matches exactly the path you used when saving the model (including the gs:// prefix and full bucket/path structure).
• Permissions: Ensure the service account running your GCP ML job has the storage.objects.get permission on the target Cloud Storage bucket. You can configure this via the IAM section in the GCP Console.
• Temp Storage: The /tmp directory in GCP ML jobs is ephemeral but has enough space for most Keras models. For extremely large models, you might need to adjust the temp path, but /tmp works for most use cases.

内容的提问来源于stack exchange，提问作者user2427317