如何让Google Cloud Composer中基于Google Service Account创建的Airflow连接仅为特定DAG专属使用？

Absolutely, you can lock down a Google Service Account-based Airflow connection so only your target DAG can access it—critical for sensitive data workflows. Below are two secure, practical approaches tailored to your use case:

Approach 1: Use Airflow RBAC to Limit Connection Access #

Cloud Composer enables Airflow's Role-Based Access Control (RBAC) by default, which you can leverage to restrict connection visibility and usage:

1. Create your target connection in the Airflow UI (under Admin > Connections) as usual—name something like conn_sensitive_gcp_sa.
2. Create a custom Airflow role:
   • Go to Security > Roles in the Airflow UI, then click "Create".
   • Name it something like dag_sensitive_connection_access, then scroll to the "Connections" section.
   • Grant only the permissions your DAG needs (typically can_read and can_edit if you need to modify the connection later) specifically for your conn_sensitive_gcp_sa connection.
3. Bind the role to your DAG's execution identity:
   • If you're using Cloud IAM integration with Airflow (enabled by default in newer Composer versions), locate the GCP service account that runs your target DAG (either the Composer environment's default worker SA, or a dedicated SA you created for this DAG).
   • In the Airflow UI's Security > Users section, find the user corresponding to this SA, and assign your custom role to them.

This way, only the service account running your target DAG will have access to the connection—other DAGs (and their execution identities) won't see it in the Airflow UI or be able to use it. Note: Ensure only trusted users have Airflow's Admin role, as admins can view all connections regardless of RBAC rules.

Approach 2: Store Credentials in Secret Manager (No Global Airflow Connection) #

For even stronger security (especially with highly sensitive data), skip the global Airflow connection entirely and store your Service Account credentials in GCP Secret Manager. This keeps credentials out of the Airflow UI and restricts access via GCP IAM:

1. Create a Secret Manager secret:
   • In the GCP Console, go to Secret Manager and create a new secret. Upload your Service Account's JSON key file or paste its content as the secret value.
2. Restrict secret access:
   • Grant the roles/secretmanager.secretAccessor IAM role to the service account that runs your target DAG—only this SA will be able to fetch the secret.
3. Fetch credentials directly in your DAG code:
   Use Airflow's GcpSecretManagerHook to pull the secret at runtime, then initialize your GCP hooks with the retrieved credentials. Example code:

from airflow import DAG
from airflow.providers.google.cloud.hooks.secret_manager import SecretManagerHook
from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook
from datetime import datetime

def handle_sensitive_data():
    # Fetch the Service Account key from Secret Manager
    secret_hook = SecretManagerHook(gcp_conn_id='google_cloud_default')
    sa_json_key = secret_hook.get_secret(
        secret_id='sensitive-sa-key',
        project_id='your-gcp-project-id'
    )
    
    # Initialize a BigQuery hook using the fetched credentials
    bq_hook = BigQueryHook(
        gcp_conn_id=None,  # Skip using a global Airflow connection
        service_account_json=sa_json_key,
        use_legacy_sql=False
    )
    
    # Execute your sensitive data operation
    query_result = bq_hook.run_query(
        sql='SELECT * FROM your-sensitive-dataset.sensitive-table',
        destination_dataset_table=None
    )

with DAG(
    dag_id='sensitive_data_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily',
    catchup=False,
    # Optional: Assign the dedicated service account to this DAG
    default_args={'gcp_conn_id': 'google_cloud_default', 'service_account': 'your-dedicated-sa@your-project.iam.gserviceaccount.com'}
) as dag:
    handle_sensitive_data()

With this approach, no other DAG can access the credentials because their execution service accounts won't have permission to read the Secret Manager secret.

Key Notes #

• Always follow the principle of least privilege: Only grant the minimum permissions needed for your DAG to function, both in Airflow RBAC and GCP IAM.
• Approach 2 is preferred for highly sensitive data, as it eliminates the risk of credentials being exposed in the Airflow UI or accessed by unauthorized Airflow users/roles.

内容的提问来源于stack exchange，提问作者luisvenezian