如何启动Compute实例并部署容器？Cloud Composer相关技术问询

Got it, let's break down how to launch a Compute Engine instance and deploy a container on it from Cloud Composer—you’re totally right that the Python Operator runs directly on Airflow workers instead of external VMs, so we’ll need to combine a few GCP tools and Airflow operators to make this work.

Step 1: Start (or Create) Your Compute Instance #

First, use Airflow’s dedicated GCP operators to handle instance lifecycle management. If you’re starting an existing instance, use GCEInstanceStartOperator; if you need to spin up a new one from scratch, use GCEInstanceCreateOperator.

Example: Start an Existing Instance #

from airflow.providers.google.cloud.operators.compute import GCEInstanceStartOperator

start_instance = GCEInstanceStartOperator(
    task_id="start_target_gce_instance",
    project_id="your-gcp-project-id",
    zone="us-central1-a",
    instance_name="your-preconfigured-instance",
    gcp_conn_id="google_cloud_default",  # Ensure your Composer env has this connection set up
)

Example: Create a New Instance (Optional) #

If you don’t have an instance yet, you can define it directly in the operator:

from airflow.providers.google.cloud.operators.compute import GCEInstanceCreateOperator

create_instance = GCEInstanceCreateOperator(
    task_id="create_new_gce_instance",
    project_id="your-gcp-project-id",
    zone="us-central1-a",
    body={
        "name": "fresh-instance-for-containers",
        "machineType": "n1-standard-1",
        "disks": [{"boot": True, "initializeParams": {"sourceImage": "debian-cloud/debian-11"}}],
        "networkInterfaces": [{"network": "default", "accessConfigs": [{"type": "ONE_TO_ONE_NAT"}]}]
    },
)

Step 2: Wait for the Instance to Be Ready (Recommended) #

Don’t rush to deploy—wait until the instance is fully running to avoid connection issues. Use GCEInstanceSensor to monitor its status:

from airflow.providers.google.cloud.sensors.compute import GCEInstanceSensor

wait_for_instance = GCEInstanceSensor(
    task_id="wait_for_instance_running",
    project_id="your-gcp-project-id",
    zone="us-central1-a",
    instance_name="your-target-instance",
    target_status="RUNNING",
    poke_interval=10,  # Check every 10 seconds
    timeout=300,  # Time out after 5 minutes
)

Step 3: Deploy the Container on the Instance #

Now for the core part—getting your container running on the Compute instance. Here are two practical approaches:

Option 1: Deploy via SSH (Simple, Direct) #

Use SSHOperator to connect to the instance and run Docker commands directly. Make sure your instance has Docker pre-installed (or include installation in the command), and that Composer’s workers can reach the instance’s SSH port (update firewall rules if needed).

from airflow.providers.ssh.operators.ssh import SSHOperator

deploy_container = SSHOperator(
    task_id="deploy_container_via_ssh",
    ssh_conn_id="gce_instance_ssh_connection",  # Configure this in Airflow's Connections tab
    command="""
        # Uncomment below if Docker isn't pre-installed on the instance
        # sudo apt-get update && sudo apt-get install -y docker.io && sudo systemctl start docker
        
        # Run your container (example: Nginx)
        sudo docker run -d -p 80:80 nginx:latest
    """,
)

Option 2: Use a Startup Script (Automate on Instance Creation) #

If you’re creating a new instance, embed a startup script to automatically set up Docker and launch your container as soon as the instance boots:

# Add this to the `body` parameter of GCEInstanceCreateOperator
"metadata": {
    "items": [
        {
            "key": "startup-script",
            "value": """
                #!/bin/bash
                apt-get update -y
                apt-get install docker.io -y
                systemctl enable --now docker
                docker run -d -p 80:80 nginx:latest
            """
        }
    ]
}

Step 4: Wire Up the Task Dependencies #

Link your tasks together to create a linear workflow:

# For starting an existing instance
start_instance >> wait_for_instance >> deploy_container

# For creating a new instance with startup script
# create_instance  # No need for extra steps since the startup script handles deployment

Key Notes for Production #

• Permissions: Ensure your Cloud Composer service account has roles like Compute Instance Admin and SSH Viewer to interact with the instance. Also, the instance’s service account needs permission to pull your container image (e.g., from GCR).
• Security: Avoid hardcoding project IDs or credentials—use Airflow Variables or Connections instead.
• Firewall Rules: Allow inbound SSH (port 22) from your Composer environment’s IP range if using the SSH method.

内容的提问来源于stack exchange，提问作者JY2k