Cloud Composer对接Cloud SQL运行TFX Pipeline时元数据访问异常排查
故障场景概述
非常感谢Google团队推出的各类优质云产品,当前遇到的问题为:通过Cloud Composer(托管Airflow)编排TFX pipeline,pipeline在本地环境可正常运行,迁移到Cloud Composer环境后,在执行CsvExampleGenComponent环节,通过配置的元数据连接读写Cloud SQL存储的元数据时报错,需梳理当前实现方案中遗漏的配置环节。
核心的精简版pipeline代码如下:
# other code setting up unrelated variables above ... metadata_config = metadata.mysql_metadata_connection_config(host=DATABASE_IP, port=3306, database=DATABASE_NAME, username=USERNAME, password=PASSWORD) def create_pipeline(pipeline_name, pipeline_root, data_root, transform_module, train_module, serving_root, beam_pipeline_args, metadata_config=None): """create beam pipeline""" example_gen = CsvExampleGen(input_base=data_root) # unrelated tfx code .... return pipeline.Pipeline( pipeline_name=pipeline_name, pipeline_root=pipeline_root, components=[ example_gen, # ...other components ... ], enable_cache=True, beam_pipeline_args=beam_pipeline_args, metadata_connection_config=metadata_config ) # var for airflow to detect DAG DAG = AirflowDagRunner(AirflowPipelineConfig(airflow_config)).run( create_pipeline(pipeline_name=beam_pipeline_name, pipeline_root=beam_pipeline_root, data_root=data_root, transform_module=transform_module, train_module=train_module, serving_root=serving_root, metadata_config=metadata_config, beam_pipeline_args=local_pipeline_args) )
已完成的配置操作
配置过程参照两份官方文档执行:
- 《从Google Kubernetes Engine连接Cloud SQL》
- 《Cloud Composer中管理Airflow连接》
具体操作项如下:
- 调整官方Cloud SQL Proxy部署模板编写Deployment配置:因业务pipeline由Cloud Composer托管,移除了原模板中配套的业务应用容器,仅保留Cloud SQL Proxy容器部署。Deployment配置如下:
apiVersion: apps/v1 kind: Deployment metadata: name: cloudsql-proxy spec: selector: matchLabels: app: cloudsql-proxy template: metadata: labels: app: cloudsql-proxy spec: containers: - name: cloudsql-proxy #image: gcr.io/cloudsql-docker/gce-proxy:1.17 # ... other container configuration env: - name: DB_USER valueFrom: secretKeyRef: name: cloudsql-token key: username - name: DB_PASS valueFrom: secretKeyRef: name: cloudsql-token key: password - name: DB_NAME valueFrom: secretKeyRef: name: cloudsql-token key: database - name: "PORT" value: "50001" #- name: cloud-sql-proxy # It is recommended to use the latest version of the Cloud SQL proxy # Make sure to update on a regular schedule! image: gcr.io/cloudsql-docker/gce-proxy:1.17 command: - "/cloud_sql_proxy" # If connecting from a VPC-native GKE cluster, you can use the # following flag to have the proxy connect over private IP # - "-ip_address_types=PRIVATE" # Replace DB_PORT with the port the proxy should listen on # Defaults: MySQL: 3306, Postgres: 5432, SQLServer: 1433 - "-instances=vm-intro-285512:us-central1:recommender-metadata=tcp:3306" # [START cloud_sql_proxy_k8s_volume_mount] # This flag specifies where the service account key can be found - "-credential_file=/Users/michaelma/.gcp/credentials/credentials.json" securityContext: # The default Cloud SQL proxy image runs as the # "nonroot" user and group (uid: 65532) by default. runAsNonRoot: true volumeMounts: - name: service-account-token mountPath: /Users/michaelma/.gcp/credentials readOnly: true # [END cloud_sql_proxy_k8s_volume_mount] # [START cloud_sql_proxy_k8s_volume_secret] volumes: - name: service-account-token secret: secretName: service-account-token # [START cloud_sql_proxy_k8s_volume_secret]
调整说明:原官方模板包含业务应用、Cloud SQL Proxy两个容器,因业务应用由Cloud Composer通过Airflow托管,因此移除了业务应用容器,仅保留代理容器部署,暂不确定该调整是否为故障根源。
- 按照Cloud Composer连接管理要求,编写Service配置暴露Cloud SQL Proxy服务:
apiVersion: v1 kind: Service metadata: name: cloudsql-proxy-service spec: type: LoadBalancer selector: app: cloudsql-proxy ports: - protocol: TCP port: 60000 targetPort: 50001
- 完成Cloud Composer环境初始化:
- 创建默认规格的Cloud Composer环境
- 在云控制台安装以下PyPi依赖:
numpy==1.16.0 tfx==0.25.0 tensorflow-model-analysis==0.25.0
- 将DAG文件、业务数据文件上传到Cloud Composer环境对应存储桶的指定路径 - 通过`kubectl`命令启动Cloud SQL Proxy Deployment及对应暴露Service
已排查无效的操作项
- 观察到Cloud SQL Proxy Pod与对应Service已正常运行,但在Airflow UI触发DAG后,仍然复现相同报错,DAG无法访问Cloud SQL实例
- 尝试将Cloud SQL Proxy的外部IP加入Cloud SQL实例的可连接IP列表,故障无改善
- 确认项目关联的服务账号已配置
Editor权限
当前故障排查方向
目前怀疑问题来自YAML配置细节疏漏、Cloud SQL网络链路配置不完整,需定位故障根因。
内容的提问来源于stack exchange,提问作者Michael
相关产品推荐
相关产品推荐

