如何用Python自动化AutoML Vision图像分类的CSV上传与模型训练？

Got it, let's walk through how to automate uploading your training label CSV and kick off an AutoML Vision image classification model training job using Python. I’ve tackled this exact workflow before, so here’s a practical, step-by-step solution using Google’s official client libraries.

First, make sure you have these sorted:

• A Google Cloud project with AutoML Vision enabled
• A service account key with permissions: AutoML Editor and Storage Object Admin
• The required Python libraries installed

Run this command to get the necessary packages:

pip install google-cloud-automl google-cloud-storage

Export your service account key path as an environment variable (or load it directly in code if you prefer):

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json"

Here’s a complete script that handles uploading the CSV to Cloud Storage (required for AutoML), creating a dataset, importing the labels, and starting training. I’ve added comments to explain each part:

from google.cloud import automl_v1beta1 as automl
from google.cloud import storage
from google.api_core.exceptions import GoogleAPICallError

def upload_csv_to_gcs(bucket_name, local_csv_path, gcs_destination_path):
    """Uploads local CSV file to Google Cloud Storage"""
    try:
        storage_client = storage.Client()
        bucket = storage_client.bucket(bucket_name)
        blob = bucket.blob(gcs_destination_path)
        
        blob.upload_from_filename(local_csv_path)
        print(f"✅ Successfully uploaded {local_csv_path} to gs://{bucket_name}/{gcs_destination_path}")
        return f"gs://{bucket_name}/{gcs_destination_path}"
    except GoogleAPICallError as e:
        print(f"❌ Failed to upload CSV: {e}")
        raise

def create_and_populate_dataset(project_id, region, dataset_name, gcs_csv_uri):
    """Creates an AutoML Vision dataset and imports label data from CSV"""
    client = automl.AutoMlClient()
    project_location = client.location_path(project_id, region)

    # Define dataset type (use MULTILABEL if you need multi-classification)
    dataset_metadata = {"classification_type": "MULTICLASS"}
    dataset_config = {
        "display_name": dataset_name,
        "image_classification_dataset_metadata": dataset_metadata
    }

    # Create the dataset
    try:
        dataset_response = client.create_dataset(project_location, dataset_config)
        dataset_id = dataset_response.name.split("/")[-1]
        print(f"✅ Created dataset with ID: {dataset_id}")
    except GoogleAPICallError as e:
        print(f"❌ Failed to create dataset: {e}")
        raise

    # Import CSV data into the dataset
    import_config = {"gcs_source": {"input_uris": [gcs_csv_uri]}}
    import_operation = client.import_data(dataset_response.name, import_config)
    
    # Wait for import to finish (this can take time depending on dataset size)
    print("⏳ Waiting for data import to complete...")
    import_operation.result()
    print("✅ Data imported successfully!")
    return dataset_response.name

def start_model_training(project_id, region, dataset_full_name, model_name, training_budget=1000):
    """Starts training the AutoML Vision model"""
    client = automl.AutoMlClient()

    # Configure model training (adjust budget as needed; 1000 = 1 node hour)
    model_metadata = {
        "image_classification_model_metadata": {
            "train_budget_milli_node_hours": training_budget
        }
    }
    model_config = {
        "display_name": model_name,
        "dataset_id": dataset_full_name.split("/")[-1],
        "image_classification_model_metadata": model_metadata["image_classification_model_metadata"]
    }

    # Kick off training
    try:
        training_operation = client.create_model(dataset_full_name, model_config)
        print(f"✅ Training started! Model resource name: {training_operation.name}")
        print("You can monitor progress in the AutoML Vision Console or via the API.")
        return training_operation.name
    except GoogleAPICallError as e:
        print(f"❌ Failed to start training: {e}")
        raise

if __name__ == "__main__":
    # Fill in your configuration details here
    PROJECT_ID = "your-google-cloud-project-id"
    REGION = "us-central1"  # Use your preferred region (e.g., europe-west1)
    GCS_BUCKET_NAME = "your-gcs-bucket-name"
    LOCAL_CSV_FILE = "/path/to/your/training-labels.csv"
    GCS_CSV_DESTINATION = "autoML-data/training-labels.csv"
    DATASET_DISPLAY_NAME = "product-image-classifier"
    MODEL_DISPLAY_NAME = "product-classifier-v1"
    TRAINING_BUDGET_MILLI_HOURS = 2000  # 2 node hours

    # Run the workflow
    gcs_csv_uri = upload_csv_to_gcs(GCS_BUCKET_NAME, LOCAL_CSV_FILE, GCS_CSV_DESTINATION)
    dataset_full_name = create_and_populate_dataset(PROJECT_ID, REGION, DATASET_DISPLAY_NAME, gcs_csv_uri)
    start_model_training(PROJECT_ID, REGION, dataset_full_name, MODEL_DISPLAY_NAME, TRAINING_BUDGET_MILLI_HOURS)

• CSV Format: Ensure your CSV follows AutoML’s requirements: each line should be image_uri,label for single-label classification, or image_uri,label1,label2 for multi-label. Image URIs must be gs:// paths (Cloud Storage) or public URLs.
• Training Budget: Adjust train_budget_milli_node_hours based on your needs. 1000 milli node hours equals 1 hour of training on a single node.
• Error Handling: The script includes basic error handling with GoogleAPICallError to catch common issues like permission errors or invalid CSV formats.
• Monitoring: After starting training, you can check progress via the AutoML Console, or extend the script to poll the model status using client.get_model(model_name).

内容的提问来源于stack exchange，提问作者Anurag Gupta