Apache Airflow双向SSL配置问题及替代方案咨询

Great question! Let's break down your options clearly, since you're already familiar with one-way SSL and need to lock down communication to only App Engine and Airflow.

Can You Use SSH for This? #

Short answer: It's not practical or recommended. Here's why:

• App Engine is a managed serverless environment—you can't maintain a persistent SSH tunnel between it and your Airflow instance. Tunnels would need to be recreated on every instance spin-up, leading to instability.
• SSH port forwarding adds unnecessary complexity: you'd need to manage tunnel endpoints, ensure reliability, and it doesn't natively enforce "only these two services can talk" without extra firewall rules that make the setup redundant.
• App Engine doesn't support inbound SSH connections, so you'd have to rely on outbound tunnels from Airflow to App Engine, which isn't scalable for production.

The Optimal Solution: Mutual TLS (mTLS) + Network Access Controls #

This is the industry standard for exactly your use case—mutual authentication ensures both sides verify each other's identity, and network rules add an extra layer of protection. Here's how to implement it with your Airflow setup:

1. Prepare Your Certificate Chain #

First, you'll need a trusted Certificate Authority (CA) to issue certificates for both Airflow (server) and App Engine (client):

• Generate or use a private CA (you can use tools like openssl or cloud-managed CAs like Google Cloud Certificate Authority Service)
• Issue a server certificate for your Airflow webserver (include your Airflow instance's domain/IP in the SAN field)
• Issue a client certificate exclusively for your App Engine service (store this securely—never share it outside your App Engine deployment)

2. Update Airflow Config to Enable mTLS #

Modify your airflow.cfg (focus on the [webserver] section) to enforce mutual authentication:

[webserver]
# Enable SSL for the webserver
web_server_ssl_cert = /airflow/certs/airflow-server-cert.pem
web_server_ssl_key = /airflow/certs/airflow-server-key.pem
# Specify your CA cert to verify client certificates
web_server_ssl_ca = /airflow/certs/root-ca.pem
# Force all clients to present a valid certificate
ssl_verify_client = required
# Use standard HTTPS port
web_server_port = 443
# Ensure API uses HTTPS
base_url = https://your-airflow-domain-or-ip

[cli]
# Update CLI endpoint to use HTTPS
endpoint_url = https://your-airflow-domain-or-ip:443

This configuration will reject any request that doesn't present a client certificate signed by your CA—only your App Engine service (with its valid client cert) will get through.

3. Configure App Engine to Use the Client Certificate #

In your App Engine code, configure your HTTP client to send the client certificate when making requests to Airflow. For example, using Python's requests library:

import requests
from google.cloud import secretmanager

# Load client cert/key from Secret Manager (better than hardcoding)
def get_secret(secret_name):
    client = secretmanager.SecretManagerServiceClient()
    response = client.access_secret_version(name=secret_name)
    return response.payload.data.decode('utf-8')

client_cert_data = get_secret("projects/your-project/secrets/appengine-client-cert/versions/latest")
client_key_data = get_secret("projects/your-project/secrets/appengine-client-key/versions/latest")

# Save temp files (or use in-memory certs if your client supports it)
with open('/tmp/client-cert.pem', 'w') as f:
    f.write(client_cert_data)
with open('/tmp/client-key.pem', 'w') as f:
    f.write(client_key_data)

# Make request to Airflow with client cert
response = requests.get(
    "https://your-airflow-domain/api/v1/dags",
    cert=('/tmp/client-cert.pem', '/tmp/client-key.pem'),
    verify='/tmp/root-ca.pem'  # Verify Airflow's server cert
)

Using Google Cloud Secret Manager ensures your certificates aren't exposed in your code or deployment artifacts.

4. Add Network Firewall Rules (Optional but Critical) #

To add a second layer of defense, restrict incoming traffic to your Airflow instance's 443 port only to App Engine's outbound IP ranges. You can get App Engine's IP ranges from Google's published IP list, then configure your server's firewall (or VPC firewall if Airflow is in GCP) to allow traffic only from those IPs.

Quick Notes on Your Existing Airflow Config #

• You have duplicate remote_log_conn_id entries in the [core] section—clean that up to use datalake_gcp_connection consistently.
• Ensure enable_xcom_pickling is set to False if you're using Airflow 2.x (it's deprecated and insecure—use XCom with JSON instead).

Final Result #

With mTLS + firewall rules, you get:

• Mutual authentication: Airflow verifies App Engine's identity via its client cert, and App Engine verifies Airflow's identity via its server cert.
• Access restriction: Only App Engine's IPs and valid client certificates can reach Airflow—all other requests are blocked at the SSL or network level.

内容的提问来源于stack exchange，提问作者Tameem