升级至composer-2.3.5-airflow-2.5.3后GKE长时Pod故障求助

升级Cloud Composer到composer-2.3.5-airflow-2.5.3版本后，所有长时间运行的GKE Pod均出现故障，核心错误为Kubernetes API返回401 Unauthorized，日志示例如下：

[2023-08-01, 14:20:23 CEST] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 2nd time calling it.
[2023-08-01, 14:20:27 CEST] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 3rd time calling it.
[2023-08-01, 14:20:31 CEST] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 4th time calling it.
[2023-08-01, 14:20:35 CEST] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 5th time calling it.
[2023-08-01, 14:20:39 CEST] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 6th time calling it.
[2023-08-01, 14:20:43 CEST] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 7th time calling it.
[2023-08-01, 14:20:47 CEST] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 8th time calling it.
[2023-08-01, 14:20:51 CEST] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 9th time calling it.
[2023-08-01, 14:20:55 CEST] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 10th time calling it.
[2023-08-01, 14:20:58 CEST] {pod.py:934} ERROR - (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '<REDACTED>', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 01 Aug 2023 12:20:58 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

经排查确认这是Google Provider的凭证过期问题，相关修复已在开发中。现需解决：

1. 如何在Composer环境中应用该修复？
2. 除了禁用日志外，还有哪些临时修复方案？

在Composer中应用官方修复的方法 #

• 等待Composer版本更新：Google会将修复后的apache-airflow-providers-google包集成到后续的Composer版本中，关注Composer版本发布日志，待包含该修复的版本推出后，直接升级环境即可。
• 手动安装修复版本的Provider包：如果需要立即修复，可通过Composer的自定义PyPI包功能，安装包含该修复的apache-airflow-providers-google版本（该问题的修复已包含在>=8.9.0版本中）。操作步骤：
  1. 进入Composer环境配置页面，添加自定义PyPI包，指定apache-airflow-providers-google==<修复版本号>
  2. 等待环境更新完成，新包会替换默认版本

其他临时修复建议 #

• 调整日志拉取参数：不建议完全禁用日志，可修改KubernetesPodOperator的日志拉取配置：
  • 设置log_fetch_timeout延长日志拉取超时时间
  • 增大get_logs_interval降低日志请求频率，减少凭证刷新压力
• 使用静态服务账号凭证：为Composer工作节点配置具有GKE访问权限的服务账号静态密钥（注意做好密钥安全管控），替代自动过期的临时凭证，避免长时间运行时凭证失效。
• 拆分长任务：将单Pod长时间运行的任务拆分为多个短任务，每个任务运行时长控制在Google临时凭证有效期内（默认1小时），从根源上避免凭证过期问题。
• 临时禁用自动日志拉取：如果必须快速规避故障，可在KubernetesPodOperator中设置get_logs=False，但需通过GKE控制台、Cloud Logging等其他方式监控Pod日志和运行状态，避免遗漏故障信息。

内容的提问来源于stack exchange，提问作者Jonny5