Docker部署Celery Beat Worker遇OSError: Bad file descriptor错误

我在Docker容器中分别运行Celery Beat Worker和普通Celery Worker，任务部署在Flask应用里。普通Worker运行正常，但Beat Worker启动时抛出以下错误：

[2024-08-07 09:57:24,692: INFO/MainProcess] mingle: searching for neighbors
[2024-08-07 09:57:25,413: INFO/Beat] beat: Starting...
[2024-08-07 09:57:25,416: ERROR/Beat] Process Beat
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/billiard-4.2.0-py3.11.egg/billiard/process.py", line 323, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/site-packages/celery-5.4.0-py3.11.egg/celery/beat.py", line 718, in run
    self.service.start(embedded_process=True)
  File "/usr/local/lib/python3.11/site-packages/celery-5.4.0-py3.11.egg/celery/beat.py", line 634, in start
    humanize_seconds(self.scheduler.max_interval))
                     ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kombu/utils/objects.py", line 40, in __get__
    return super().__get__(instance, owner)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery-5.4.0-py3.11.egg/celery/beat.py", line 677, in scheduler
    return self.get_scheduler()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery-5.4.0-py3.11.egg/celery/beat.py", line 667, in get_scheduler
    aliases = dict(load_extension_class_names(extension_namespace))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery-5.4.0-py3.11.egg/celery/utils/imports.py", line 150, in load_extension_class_names
    _entry_points = entry_points(group=namespace)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 1041, in entry_points
    return SelectableGroups.load(eps).select(**params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 476, in load
    ordered = sorted(eps, key=by_group)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 1038, in <genexpr>
    eps = itertools.chain.from_iterable(
                                       ^
  File "/usr/local/lib/python3.11/importlib/metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
        ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 955, in _normalized_name
    or super()._normalized_name
       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 627, in _normalized_name
    return Prepared.normalize(self.name)
                              ^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 622, in name
    return self.metadata['Name']
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 611, in metadata
    or self.read_text('PKG-INFO')
       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 939, in read_text
    return self._path.joinpath(filename).read_text(encoding='utf-8')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/zipfile.py", line 2468, in read_text
    with self.open('r', encoding, *args, **kwargs) as strm:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/zipfile.py", line 2434, in open
    stream = self.root.open(self.at, zip_mode, pwd=pwd)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/zipfile.py", line 1579, in open
    fheader = zef_file.read(sizeFileHeader)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/zipfile.py", line 786, in read
    self._pos = self._file.tell()
                ^^^^^^^^^^^^^^^^^
OSError: [Errno 9] Bad file descriptor

环境信息 #

Beat Worker使用python:3.11.9-slim作为基础镜像，容器内安装的依赖包版本如下：

amqp==5.2.0
attrs==24.1.0
billiard==4.2.0
blinker==1.8.2
boto3==1.34.145
botocore==1.34.145
celery==5.4.0
certifi==2024.7.4
cffi==1.17.0
charset-normalizer==3.3.2
CIRAUtils==9.0.0
click==8.1.7
click-didyoumean==0.3.1
click-plugins==1.1.1
click-repl==0.3.0
cobble==0.1.4
cryptography==43.0.0
cssselect==1.2.0
dnspython==2.6.1
et-xmlfile==1.1.0
flasgger==0.9.7.1
Flask==3.0.2
Flask-Cors==4.0.1
Flask-JWT-Extended==4.6.0
Flask-PyMongo==2.3.0
gunicorn==22.0.0
idna==3.7
importlib_metadata==8.2.0
itsdangerous==2.2.0
Jinja2==3.1.4
jmespath==1.0.1
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
kombu==5.3.7
lxml==5.2.2
mammoth==1.8.0
MarkupSafe==2.1.5
mistune==3.0.2
msal==1.30.0
numpy==2.0.1
openpyxl==3.1.5
packaging==24.1
pandas==2.2.2
prompt_toolkit==3.0.47
pycparser==2.22
pycryptodome==3.20.0
PyJWT==2.8.0
pymongo==4.8.0
pyquery==2.0.0
python-dateutil==2.9.0
pytz==2024.1
PyYAML==6.0.1
redis==5.0.7
referencing==0.35.1
requests==2.32.3
rpds-py==0.19.1
s3transfer==0.10.2
sentry-sdk==2.10.0
six==1.16.0
tzdata==2024.1
urllib3==2.2.2
vine==5.1.0
wcwidth==0.2.13
Werkzeug==3.0.3
xmltodict==0.12.0
zipp==3.19.2

其中CIRAUtils==9.0.0是自定义包。

解决步骤 #

1. 检查自定义包完整性
   错误发生在读取包元数据阶段，优先排查自定义包CIRAUtils的安装包是否损坏。重新构建该包的wheel/sdist文件，确保PKG-INFO文件完整，然后在容器中重新安装。

2. 禁用元数据缓存
   启动Beat Worker前设置环境变量，跳过importlib_metadata的缓存：

   IMPORTLIB_METADATA_CACHE_TIMEOUT=-1 celery -A your_flask_app beat -l info
   

3. 指定调度器跳过扩展加载
   直接指定默认调度器，避免加载扩展时触发元数据读取错误：

   celery -A your_flask_app beat -l info --scheduler celery.beat:PersistentScheduler
   

4. 更换基础镜像或补充系统依赖
   python:slim镜像精简了系统组件，可能缺少zip文件操作相关依赖。尝试使用python:3.11.9完整镜像，或在Dockerfile中添加：

   RUN apt-get update && apt-get install -y zip unzip && rm -rf /var/lib/apt/lists/*
   

5. 清理pip缓存重装依赖
   在Dockerfile中使用--no-cache-dir参数安装依赖，避免缓存的损坏包被复用：

   RUN pip install --no-cache-dir -r requirements.txt
   

内容的提问来源于stack exchange，提问作者Albin Antony