Cloud Composer 2工作节点存储扩容及BigQuery任务性能咨询

我们近期从Composer 1迁移至Composer 2，迁移后某一基于BigqueryOperator、处理50TB数据的任务性能大幅下降。原Composer 1中该查询耗时约40分钟，现Composer 2中耗时约1小时40分钟。

Composer 1配置 #

Web server machine type
composer-n1-webserver-2 (2 vCPU, 1.6 GB memory)
  
Cloud SQL machine type
db-n1-standard-2 (2 vCPU, 7.5 GB memory)
  
Worker nodes
Node count
3   
Disk size (GB)
50
Machine type
e2-standard-4
Number of schedulers
1   

Composer 2配置 #

Resources
Workloads configuration
Scheduler
4 vCPUs, 7.5 GB memory, 5 GB storage
Number of schedulers
2
Web server
2 vCPUs, 7.5 GB memory, 10 GB storage
Worker
4 vCPUs, 16 GB memory, 10 GB storage
Number of workers
Autoscaling between 4 and 8 workers

咨询问题 #

• 工作节点存储从50GB降至10GB是否影响查询运行？
• 工作节点是否参与查询计算，还是仅从队列获取任务并提交至BigQuery？
• 如何扩容Composer 2中默认最大为10GB的工作节点存储？

内容的提问来源于stack exchange，提问作者codninja0908