Elasticbeanstalk上Django-Celery Daemon无法连接Redis求助

我在Django中使用Celery处理定时任务，本地运行正常，但部署到Elasticbeanstalk后，Celery Daemon始终尝试连接未安装的RabbitMQ并失败。Redis已在EB实例本地安装且运行正常，但无论尝试硬编码配置、加载环境变量等操作，Celery完全无视Redis的broker配置。

已尝试的解决方法 #

• 在配置文件中导出变量
• 将localhost替换为127.0.0.1
• 尝试将环境变量改为小写

配置文件详情 #

celery.py #

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings

# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'demo_django.settings')

app = Celery('app_django', broker=settings.CELERY_BROKER_URL)

# Use Django settings for Celery
app.config_from_object('django.conf:settings', namespace='CELERY')

# Auto-discover tasks from installed apps
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

settings.py #

# Celery settings
CELERY_BROKER_URL = os.getenv('REDIS_URL')
CELERY_RESULT_BACKEND = os.getenv('REDIS_URL')
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

.ebextensions/01_celery_config #

files:
  "/etc/tmpfiles.d/celery.conf":
    mode: "000777"
    content: |
      d /run/celery 0755 celery celery -
      d /var/log/celery 0755 celery celery -
  "/etc/conf.d/celery":
    mode: "000777"
    content: |
      # Name of nodes to start
      # here we have a single node
      CELERYD_NODES="w1"
      # or we could have three nodes:
      #CELERYD_NODES="w1 w2 w3"

      # Absolute or relative path to the 'celery' command:
      CELERY_BIN="/var/app/venv/staging-LQM1lest/bin/celery"
      #CELERY_BIN="/virtualenvs/def/bin/celery"

      # App instance to use
      # comment out this line if you don't use an app
      CELERY_APP="demo_django"
      # or fully qualified:
      #CELERY_APP="proj.tasks:app"

      # How to call manage.py
      CELERYD_MULTI="multi"

      # Extra command-line arguments to the worker
      CELERYD_OPTS="--time-limit=300 --autoscale=8,3"

      # - %n will be replaced with the first part of the nodename.
      # - %I will be replaced with the current child process index
      #   and is important when using the prefork pool to avoid race conditions.
      CELERYD_PID_FILE="/var/run/celery/%n.pid"
      CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
      CELERYD_LOG_LEVEL="INFO"

      # you may wish to add these options for Celery Beat
      CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"
      CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"
      
      # Desperate Attempt to make it work even tried hardcoding
      export DJANGO_SETTINGS_MODULE="demo_django.settings" 
      export CELERY_BROKER_URL=redis://127.0.0.1:6379
      export BROKER_URL=redis://127.0.0.1:6379
      export $(/opt/elasticbeanstalk/bin/get-config --output YAML environment | sed -r 's/: /=/' | xargs)

  "/etc/systemd/system/celery.service":
    mode: "000777"
    content: |
      [Unit]
      Description=Celery Service
      After=network.target

      [Service]
      Type=forking
      User=celery
      Group=celery
      EnvironmentFile=/etc/conf.d/celery
      WorkingDirectory=/var/app/current
      ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \
          --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
          --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
      ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \
          --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
          --loglevel="${CELERYD_LOG_LEVEL}"'
      ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \
          --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
          --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
      Restart=always

      [Install]
      WantedBy=multi-user.target

排查与修复建议 #

1. 调整环境变量加载顺序
   在/etc/conf.d/celery中，你将EB环境变量加载命令放在了硬编码配置之后，这会导致EB环境变量覆盖硬编码值。如果EB环境中未设置REDIS_URL，CELERY_BROKER_URL会为空，Celery会自动 fallback到默认的RabbitMQ地址。建议将硬编码配置移到环境变量加载之后，或确保EB环境中正确配置REDIS_URL。

2. 修正Celery初始化逻辑
   celery.py中先通过app = Celery(..., broker=settings.CELERY_BROKER_URL)设置broker，再调用app.config_from_object。如果settings.CELERY_BROKER_URL为空，初始化时就会使用默认值。可以在settings.py中添加日志输出确认REDIS_URL是否被正确获取，或在EB实例中执行echo $REDIS_URL验证变量存在性。

3. 修复EnvironmentFile的变量格式
   systemd的EnvironmentFile不支持export语句，只需要键值对格式。修改/etc/conf.d/celery，去掉所有export前缀：

   DJANGO_SETTINGS_MODULE="demo_django.settings"
   CELERY_BROKER_URL=redis://127.0.0.1:6379
   BROKER_URL=redis://127.0.0.1:6379
   

   若需要加载EB环境变量，可在celery.service的Environment字段中添加EnvironmentFile=/opt/elasticbeanstalk/deployment/env。

4. 确认虚拟环境路径有效性
   EB每次部署的虚拟环境路径可能变化，CELERY_BIN中的固定路径可能失效。建议改用通配符匹配：

   CELERY_BIN="/var/app/venv/*/bin/celery"
   

   或在部署脚本中动态生成该路径。

5. 通过日志与手动运行定位问题
   查看/var/log/celery/w1.log，搜索broker关键词确认实际使用的地址。若显示默认RabbitMQ地址，可手动在EB实例中运行测试命令：

   cd /var/app/current
   source /var/app/venv/staging-LQM1lest/bin/activate
   celery -A demo_django worker --loglevel=info
   

   验证手动运行时是否能连接Redis，以此区分是代码配置问题还是服务启动配置问题。

内容的提问来源于stack exchange，提问作者Rohail Najam