部署至Heroku时Airflow命令因utils.py中TypeError失败

问题场景 #

本地Docker环境运行Airflow 2.4.1（Python 3.10）完全正常，但通过Heroku容器注册表部署后，执行airflow webserver命令失败，核心报错回溯如下：

____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:55811
Timeout: 120
Logfiles: - -
Access Logformat: 
=================================================================
[2022-11-10 00:19:39 +0000] [43] [INFO] Starting gunicorn 20.1.0
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/site-packages/gunicorn/__main__.py", line 7, in <module>
    run()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 198, in run
    self.start()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 138, in start
    self.cfg.on_starting(self)
  File "/usr/local/lib/python3.10/site-packages/airflow/www/gunicorn_config.py", line 40, in on_starting
    ProvidersManager().connection_form_widgets
  File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 908, in connection_form_widgets
    self.initialize_providers_hooks()
  File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 300, in wrapped_function
    func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 386, in initialize_providers_hooks
    self.initialize_providers_list()
  File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 300, in wrapped_function
    func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 363, in initialize_providers_list
    self._discover_all_providers_from_packages()
  File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 429, in _discover_all_providers_from_packages
    for entry_point, dist in entry_points_with_dist('apache_airflow_provider'):
  File "/usr/local/lib/python3.10/site-packages/airflow/utils/entry_points.py", line 40, in entry_points_with_dist
    key = canonicalize_name(dist.metadata["Name"])
  File "/usr/local/lib/python3.10/site-packages/packaging/utils.py", line 34, in canonicalize_name
    value = _canonicalize_regex.sub("-", name).lower()
TypeError: expected string or bytes-like object
[2022-11-10 00:21:38,935] {webserver_command.py:217} ERROR - No response from gunicorn master within 120 seconds
[2022-11-10 00:21:38,935] {webserver_command.py:218} ERROR - Shutting down webserver

错误原因 #

核心问题是某个已安装依赖包的metadata中"Name"字段不是字符串类型，Airflow在扫描apache_airflow_provider入口点时，调用packaging.utils.canonicalize_name处理该字段时触发类型错误。本地环境正常但部署环境异常，通常是依赖版本不一致、缓存损坏或安装包不完整导致。

解决步骤 #

• 清理Heroku构建缓存
  Heroku的缓存可能留存了损坏的依赖包，执行以下命令清理后重新部署：

  heroku builds:cache:purge -a your-app-name
  

• 锁定依赖版本与本地一致
  将本地验证通过的requirements.txt或Pipfile.lock同步到部署环境，确保所有依赖包版本（尤其是apache-airflow、packaging及各类provider包）与本地完全一致，避免版本差异引发的元数据异常。

• 检查依赖完整性
  在Dockerfile中添加依赖检查步骤，确保安装的包无损坏：

  RUN pip check
  

  若检查出问题，根据提示修复依赖冲突或重新安装异常包。

• 排查异常包
  如果上述方法无效，可在启动webserver前添加临时脚本排查哪个包的元数据异常：

  # 添加到Dockerfile中，在airflow webserver命令前执行
  RUN python -c "import pkg_resources; for dist in pkg_resources.working_set: try: name = dist.metadata['Name']; if not isinstance(name, str): print(f'Invalid package: {dist.project_name}, Name type: {type(name)}') except Exception as e: print(f'Error checking {dist.project_name}: {e}')"
  

  运行后找到异常包，移除或替换为正常版本。

• 降级packaging包版本
  部分packaging版本存在元数据处理兼容问题，尝试指定稳定版本，在requirements.txt中添加：

  packaging==20.9
  

内容的提问来源于stack exchange，提问作者Jerome