GCP Cloud Composer（基于GKE/GCE）是否支持自动扩缩容？DAG扩容咨询

First: Does Cloud Composer support autoscaling on GKE/GCE? #

Absolutely—though it’s important to break down autoscaling across two key layers of a Composer environment:

• Control plane autoscaling (Scheduler, Webserver, etc.):

  • For Cloud Composer 2 (which runs on GKE by default), control plane components like the Scheduler support autoscaling out of the box. If your scheduling load spikes (e.g., more DAGs needing parsing and scheduling), Composer can automatically spin up additional Scheduler instances. Plus, if you’ve enabled the GKE Cluster Autoscaler for your underlying cluster, GKE nodes will scale up on their own if control plane pods need more resources.
  • For older Composer 1 environments running on GCE, control plane autoscaling is more limited—you’d typically adjust instance counts manually, though you can build custom scaling workflows using Cloud Monitoring alerts and Cloud Functions.

• Celery Worker autoscaling:
  This is the layer that matters most for handling increased task load from more DAGs. When using the CeleryExecutor (as you mentioned), Cloud Composer supports autoscaling worker pods via the Horizontal Pod Autoscaler (HPA). You can set scaling triggers based on:

  • CPU/memory usage of worker pods
  • The number of pending tasks in your Airflow queue (often the most reliable trigger for task-heavy workloads)

Second: Will my cluster auto-scale when DAGs grow from 100 to 300 with Celery Workers? #

The short answer is yes—if you have the right autoscaling configurations in place:

• DAG count alone doesn’t trigger scaling; what matters is the number of running/pending tasks those 300 DAGs generate, and whether that load outpaces your current cluster’s capacity.
• If you’ve set up Celery Worker HPA (e.g., scaling when pending tasks exceed a threshold) and enabled the GKE Cluster Autoscaler:
  1. As task volume grows, the HPA will detect pending tasks and spin up more Celery Worker pods.
  2. If your existing 3 nodes don’t have enough resources for these new pods, the GKE Cluster Autoscaler will automatically add more nodes to the cluster to accommodate them.
• Critical note: Make sure you’ve set a reasonable maximum node count for your GKE node pool (or GCE instance group for Composer 1). If you cap node counts too low, the HPA won’t be able to scale workers beyond what your existing nodes can handle, leading to stuck pending pods.

Quick Pro Tips #

• Monitor metrics like airflow_task_queue_length, worker CPU/memory usage, and GKE node utilization to fine-tune your autoscaling thresholds.
• For Composer 2, you can configure Celery Worker autoscaling directly via the GCP Console or gcloud commands—look for the "Worker autoscaling" settings when updating your environment.

内容的提问来源于stack exchange，提问作者Ching-Chien Chen