You need to enable JavaScript to run this app. Cloud Composer对接Cloud SQL运行TFX Pipeline时元数据访问异常排查
优惠活动
大模型
产品
解决方案
定价
更多

Cloud Composer对接Cloud SQL运行TFX Pipeline时元数据访问异常排查

故障场景概述

非常感谢Google团队推出的各类优质云产品,当前遇到的问题为:通过Cloud Composer(托管Airflow)编排TFX pipeline,pipeline在本地环境可正常运行,迁移到Cloud Composer环境后,在执行CsvExampleGenComponent环节,通过配置的元数据连接读写Cloud SQL存储的元数据时报错,需梳理当前实现方案中遗漏的配置环节。

核心的精简版pipeline代码如下:

# other code setting up unrelated variables above ...
metadata_config = metadata.mysql_metadata_connection_config(host=DATABASE_IP, port=3306,
                                                            database=DATABASE_NAME, username=USERNAME,
                                                            password=PASSWORD)

def create_pipeline(pipeline_name, pipeline_root, data_root, transform_module, train_module, serving_root,
                    beam_pipeline_args, metadata_config=None):
    """create beam pipeline"""
    example_gen = CsvExampleGen(input_base=data_root)

    # unrelated tfx code ....
    
    return pipeline.Pipeline(
        pipeline_name=pipeline_name,
        pipeline_root=pipeline_root,
        components=[
            example_gen,
            # ...other components ...
        ],
        enable_cache=True,
        beam_pipeline_args=beam_pipeline_args,
        metadata_connection_config=metadata_config
    )

# var for airflow to detect DAG
DAG = AirflowDagRunner(AirflowPipelineConfig(airflow_config)).run(
    create_pipeline(pipeline_name=beam_pipeline_name, pipeline_root=beam_pipeline_root, data_root=data_root,
                    transform_module=transform_module, train_module=train_module, serving_root=serving_root,
                    metadata_config=metadata_config,
                    beam_pipeline_args=local_pipeline_args)
)
已完成的配置操作

配置过程参照两份官方文档执行:

  • 《从Google Kubernetes Engine连接Cloud SQL》
  • 《Cloud Composer中管理Airflow连接》
    具体操作项如下:
  1. 调整官方Cloud SQL Proxy部署模板编写Deployment配置:因业务pipeline由Cloud Composer托管,移除了原模板中配套的业务应用容器,仅保留Cloud SQL Proxy容器部署。Deployment配置如下:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudsql-proxy
spec:
  selector:
    matchLabels:
      app: cloudsql-proxy
  template:
    metadata:
      labels:
        app: cloudsql-proxy
    spec:
      containers:
      - name: cloudsql-proxy
        #image: gcr.io/cloudsql-docker/gce-proxy:1.17
        # ... other container configuration
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: cloudsql-token
              key: username
        - name: DB_PASS
          valueFrom:
            secretKeyRef:
              name: cloudsql-token
              key: password
        - name: DB_NAME
          valueFrom:
            secretKeyRef:
              name: cloudsql-token
              key: database
        - name: "PORT"
          value: "50001"
      #- name: cloud-sql-proxy
        # It is recommended to use the latest version of the Cloud SQL proxy
        # Make sure to update on a regular schedule!
        image: gcr.io/cloudsql-docker/gce-proxy:1.17
        command:
          - "/cloud_sql_proxy"

          # If connecting from a VPC-native GKE cluster, you can use the
          # following flag to have the proxy connect over private IP
          # - "-ip_address_types=PRIVATE"

          # Replace DB_PORT with the port the proxy should listen on
          # Defaults: MySQL: 3306, Postgres: 5432, SQLServer: 1433
          - "-instances=vm-intro-285512:us-central1:recommender-metadata=tcp:3306"

        # [START cloud_sql_proxy_k8s_volume_mount]
          # This flag specifies where the service account key can be found
          - "-credential_file=/Users/michaelma/.gcp/credentials/credentials.json"
        securityContext:
          # The default Cloud SQL proxy image runs as the
          # "nonroot" user and group (uid: 65532) by default.
          runAsNonRoot: true
        volumeMounts:
        - name: service-account-token
          mountPath: /Users/michaelma/.gcp/credentials
          readOnly: true
          # [END cloud_sql_proxy_k8s_volume_mount]
      # [START cloud_sql_proxy_k8s_volume_secret]
      volumes:
      - name: service-account-token
        secret:
          secretName: service-account-token
      # [START cloud_sql_proxy_k8s_volume_secret]

调整说明:原官方模板包含业务应用、Cloud SQL Proxy两个容器,因业务应用由Cloud Composer通过Airflow托管,因此移除了业务应用容器,仅保留代理容器部署,暂不确定该调整是否为故障根源。

  1. 按照Cloud Composer连接管理要求,编写Service配置暴露Cloud SQL Proxy服务:
apiVersion: v1
kind: Service
metadata:
  name: cloudsql-proxy-service
spec:
  type: LoadBalancer
  selector:
    app: cloudsql-proxy
  ports:
  - protocol: TCP
    port: 60000
    targetPort: 50001
  1. 完成Cloud Composer环境初始化:
    • 创建默认规格的Cloud Composer环境
    • 在云控制台安装以下PyPi依赖:
numpy==1.16.0
tfx==0.25.0
tensorflow-model-analysis==0.25.0
- 将DAG文件、业务数据文件上传到Cloud Composer环境对应存储桶的指定路径
- 通过`kubectl`命令启动Cloud SQL Proxy Deployment及对应暴露Service
已排查无效的操作项
  • 观察到Cloud SQL Proxy Pod与对应Service已正常运行,但在Airflow UI触发DAG后,仍然复现相同报错,DAG无法访问Cloud SQL实例
  • 尝试将Cloud SQL Proxy的外部IP加入Cloud SQL实例的可连接IP列表,故障无改善
  • 确认项目关联的服务账号已配置Editor权限
当前故障排查方向

目前怀疑问题来自YAML配置细节疏漏、Cloud SQL网络链路配置不完整,需定位故障根因。


内容的提问来源于stack exchange,提问作者Michael

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.11.14 08:45:04