Airflow升级后依赖冲突及多轮修复失败技术求助

Alright, let's work through this dependency mess you're facing. First, let's nail down the core issue that's causing all the headaches:

Core Conflict Breakdown #

Your original setup (composer-1.11.2-airflow-1.10.6, Python 3.6, dbt==0.17.0, protobuf==3.12.2) worked because the dependency check didn't catch the hidden conflict. Once you upgraded to Airflow 1.10.13, pip's dependency resolver started enforcing stricter checks, and it flagged that:

• dbt-bigquery 0.17.0 requires protobuf >=3.6.0, <3.12
• You explicitly set protobuf==3.12.2, which falls outside that range.

All the issues you ran into after trying fixes stem from either mismatched sub-dependencies or incompatible package versions with Python 3.6. Let's walk through reliable solutions tailored to your setup.

Solution 1: Adjust Protobuf & Docutils to Match dbt 0.17.0's Requirements #

This is the safest fix if you want to keep using dbt==0.17.0 and Python 3.6. We'll pick versions that play nice with all your dependencies:

Step 1: Update Your Requirements File #

Use this exact set of dependencies to avoid conflicts:

dbt==0.17.0
google-cloud-storage
google-cloud-secret-manager==1.0.0
protobuf==3.11.4  # Fits dbt-bigquery's <3.12 requirement, stable for Python 3.6
docutils==0.15.1  # Avoids the post1 metadata mismatch issue, meets botocore's <0.16 rule

Step 2: Clean & Reinstall #

First, remove any conflicting existing packages:

pip uninstall -y protobuf docutils dbt

Then install the updated requirements:

pip install -r requirements.txt

Why this works:

• protobuf==3.11.4 is the last stable release in the 3.11 series, fully compatible with dbt-bigquery 0.17.0 and google-cloud-secret-manager 1.0.0.
• docutils==0.15.1 directly matches the version botocore <1.15 expects, avoiding the metadata mismatch error you saw earlier.

Solution 2: Upgrade dbt to Support Protobuf 3.12.2 #

If you're open to updating dbt, versions starting from dbt==0.18.0 relaxed the protobuf restriction to >=3.6.0, <3.19.0, which includes your protobuf==3.12.2.

Step 1: Update Requirements #

dbt==0.18.1  # Stable release compatible with protobuf 3.12.2
google-cloud-storage
google-cloud-secret-manager==1.0.0
protobuf==3.12.2

Step 2: Adapt Your dbt Project #

Note: dbt 0.18.x has breaking changes (e.g., model configuration syntax, macro updates). Make sure to back up your dbt project and adjust it to match the 0.18.x standards before deploying.

Solution 3: Use a Composer-Airflow Version with Pre-Tested Dependencies #

Google Cloud Composer versions come with pre-vetted dependency sets to avoid conflicts. For Airflow 1.10.13, use composer-1.14.0-airflow-1.10.13 (the official compatible Composer version).

• Add dependencies via the Cloud Console or gcloud composer environments update command instead of manual pip installs—this ensures you're using versions that Google has tested with that specific Composer/Airflow stack.

Key Takeaways to Avoid Future Issues #

• Don't mix Python versions: Composer 1.x uses Python 3.6, and you can't manually upgrade to 3.8 without switching to Composer 2.x.
• Watch for Python 2 leftovers: The SyntaxError about missing parentheses in print means you're installing a Python 2-only package version—always verify that packages support Python 3.6.
• Check dependencies early: Use pip check regularly to catch hidden conflicts before they break your setup.

内容的提问来源于stack exchange，提问作者user2640679