通过CI/CD更新Cloud Composer依赖时Poetry导出requirements报错排查

需要确保Python依赖能安装到Cloud Composer实例中：开发者提交含依赖的代码到Bitbucket后，Bitbucket流水线需在部署阶段完成Cloud Composer实例的依赖安装。我们必须用poetry管理依赖，部署流程包括同步DAG目录到存储桶、安装poetry、生成requirements.txt文件，再通过gcloud CLI按官方文档更新Composer的PyPI包，但最后一步安装依赖失败。

以下是bitbucket-pipeline.yaml中的相关步骤：

- /etc/poetry/bin/poetry export --without-hashes --format requirements.txt --output /tmp/requirements_tmp.txt
- sed "s/\"/'/g"  /tmp/requirements_tmp.txt >> /tmp/requirements.txt
- gcloud composer environments update ${COMPOSER_PROJECT_ID}-composer --location ${COMPOSER_LOCATION} --update-pypi-packages-from-file /tmp/requirements.txt

• poetry安装正常
• 第二步用sed替换双引号为单引号的尝试无效，调整库与版本间空格后仍报错
• 第三步命令导致流水线失败

已知信息 #

• 生成的requirements.txt本地执行pip install -r requirements.txt可正常安装，报错的8项正是新增的依赖
• 已排除权限问题

1. poetry生成的requirements.txt是否不符合PEP规范？是否需要额外转换？（参考PEP-508、PEP-496）
2. 是否需要更换实现方案？

Poetry生成的requirements.txt（sed替换后） #

colorama==0.4.6 ; python_full_version == "3.8.12" and sys_platform == "win32"
exceptiongroup==1.0.1 ; python_full_version == "3.8.12"
iniconfig==2.0.0 ; python_full_version == "3.8.12"
packaging==21.3 ; python_full_version == "3.8.12"
pluggy==1.0.0 ; python_full_version == "3.8.12"
pyparsing==3.0.9 ; python_full_version == "3.8.12"
pytest==7.3.1 ; python_full_version == "3.8.12"
tomli==2.0.1 ; python_full_version == "3.8.12"

错误信息 #

gcloud composer environments update ${COMPOSER_PROJECT_ID}-composer --location ${COMPOSER_LOCATION} --update-pypi-packages-from-file /tmp/requirements.txt

ERROR: (gcloud.composer.environments.update) INVALID_ARGUMENT: Found 8 problems:
    1) Error validating PyPI package extras and version ==0.4.6 ; python_full_version == '3.8.12' and sys_platform == 'win32'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508.
    2) Error validating PyPI package extras and version ==1.0.1 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508.
    3) Error validating PyPI package extras and version ==2.0.0 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508.
    4) Error validating PyPI package extras and version ==21.3 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508.
    5) Error validating PyPI package extras and version ==1.0.0 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508.
    6) Error validating PyPI package extras and version ==3.0.9 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508.
    7) Error validating PyPI package extras and version ==7.3.1 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508.
    8) Error validating PyPI package extras and version ==2.0.1 ; python_full_version == '3.8.12'. It must be provided as optional extras followed by an optional versionspec. For example, '>=1.10.3'. For details, see the grammars for 'extras' and 'versionspec' in PEP 508.

1. 关于PEP规范与转换问题：
   Poetry生成的requirements.txt符合PEP 508规范，但Cloud Composer的gcloud composer environments update命令不支持其中的环境标记（environment markers）（即分号后的条件语句）。本地pip能正常安装是因为pip完全兼容PEP 508的标记语法，而Composer的CLI工具仅接受简单的包版本格式（如package==x.y.z）。

解决方法：修改Poetry导出命令，添加--without-markers参数去掉环境标记，同时无需再用sed替换引号：

/etc/poetry/bin/poetry export --without-hashes --without-markers --format requirements.txt --output /tmp/requirements.txt

2. 是否需要更换方案：
   不需要更换现有方案，仅需调整Poetry的导出参数即可解决问题。如果后续遇到其他兼容问题，再考虑替代方案（如手动维护核心依赖的requirements.txt），但优先使用上述参数调整的方式。

内容的提问来源于stack exchange，提问作者cestlarumba