Airflow Docker部署无法找到GCP凭证JSON文件求助

报错信息：

google.auth.exceptions.DefaultCredentialsError: File /.google/credentials/google_credentials.json was not found.

环境背景 #

• 在GCP Compute Engine的Ubuntu虚拟机上，通过Docker运行Airflow（使用DataTalksClub数据工程训练营Week2的docker-compose配置）
• GCP服务账号认证JSON文件存放在VM的~/.google/credentials/目录下

DAG代码 #

import os
import logging
from datetime import datetime

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator

from google.cloud import storage
from airflow.providers.google.cloud.operators.bigquery import BigQueryCreateExternalTableOperator

PROJECT_ID = os.environ.get("GCP_PROJECT_ID")
BUCKET = os.environ.get("GCP_GCS_BUCKET")

FILE_DATE = '2022-10'
FILE_NAME = f'yellow_tripdata_{FILE_DATE}.parquet'
XCOM_FILE_NAME = f'yellow_tripdata_{FILE_DATE}'
SOURCE_URL = 'https://d37ci6vzurychx.cloudfront.net/trip-data/'
URL_TEMPLATE = SOURCE_URL + FILE_NAME

PATH_TO_LOCAL_HOME = os.environ.get("AIRFLOW_HOME", "/opt/airflow/")


def upload_to_gcs(bucket, object_name, local_file):
    storage.blob._MAX_MULTIPART_SIZE = 5 * 1024 * 1024 
    storage.blob._DEFAULT_CHUNKSIZE = 5 * 1024 * 1024 

    client = storage.Client()
    bucket = client.bucket(bucket)

    blob = bucket.blob(object_name)
    blob.upload_from_filename(local_file)
    

default_args = {
    "owner": "airflow",
    "start_date": datetime(2019, 1, 1),
    "depends_on_past": False,
    "retries": 1,
}

with DAG(
    dag_id="ingest_yellowtaxi_to_CGP",
    schedule_interval="0 6 2 * *",
    default_args=default_args,
    catchup=False,
    max_active_runs=1,
    tags=['dtc-de-hw-task'],
) as dag:

    download_datafile_task = BashOperator(
        task_id="download_datafile_task",
        bash_command=f"curl -sSLf {URL_TEMPLATE} > {PATH_TO_LOCAL_HOME}/{FILE_NAME}"
    )

    local_to_gcs_task = PythonOperator(
        task_id="local_to_gcs_task",
        python_callable=upload_to_gcs,
        op_kwargs={
            "bucket": BUCKET,
            "object_name": f"raw/{FILE_NAME}",
            "local_file": f"{PATH_TO_LOCAL_HOME}/{FILE_NAME}",
        },
    )

    delete_data_file = BashOperator(
        task_id="delete_data_file",
        bash_command=f"cd {PATH_TO_LOCAL_HOME} && rm {FILE_NAME}"
    )

download_datafile_task >> local_to_gcs_task >> delete_data_file

已尝试但无效的操作 #

• 修改Docker卷挂载配置，明确指定认证文件路径和文件名
• 在VM中执行export GOOGLE_APPLICATION_CREDENTIALS=<path/filename.json>和gcloud auth activate-service-account --key-file $GOOGLE_APPLICATION_CREDENTIALS，但环境变量未在Airflow容器中生效
• 进入Airflow Worker容器，未找到/.google/credentials目录
• 尝试通过Airflow UI配置GCP连接，持续收到Bad Request错误

问题排查与解决步骤 #

1. 修正Docker卷挂载配置 #

Docker的~符号不会自动解析为用户家目录，需替换为VM中认证目录的绝对路径。在docker-compose.yaml的volumes段添加：

volumes:
  - /home/ubuntu/.google/credentials/:/.google/credentials:ro

（将/home/ubuntu替换为你的VM用户实际家目录路径）

2. 在容器中配置环境变量 #

直接在VM中export的变量不会传递给Docker容器，必须在docker-compose的environment段声明：

environment:
  - GOOGLE_APPLICATION_CREDENTIALS=/.google/credentials/google_credentials.json
  - GCP_PROJECT_ID=你的GCP项目ID
  - GCP_GCS_BUCKET=你的GCS存储桶名称

3. 验证容器内文件是否存在 #

启动容器后，进入Worker容器检查文件：

# 查找Worker容器ID
docker ps | grep airflow-worker
# 进入容器
docker exec -it <worker容器ID> bash
# 检查文件路径
ls -la /.google/credentials/

如果文件不存在，说明卷挂载路径错误，需重新核对VM本地路径与容器内路径。

4. 改用Airflow GCP Hook（推荐） #

避开本地文件挂载问题，使用Airflow官方GCS Hook处理认证：

• 在Airflow UI的Admin > Connections中创建GCP连接：
  • Conn Id: google_cloud_default
  • Conn Type: Google Cloud
  • 在Keyfile JSON字段直接粘贴服务账号JSON的完整内容
• 修改DAG中的上传函数：

  from airflow.providers.google.cloud.hooks.gcs import GCSHook
  
  def upload_to_gcs(bucket, object_name, local_file):
      hook = GCSHook(gcp_conn_id="google_cloud_default")
      hook.upload(bucket_name=bucket, object_name=object_name, filename=local_file)
  

5. 调整认证文件权限 #

确保VM中的JSON文件权限允许容器读取：

chmod 644 ~/.google/credentials/google_credentials.json

内容的提问来源于stack exchange，提问作者Michael.s