Ubuntu20.04下Celery Worker/Beat开机自启配置及Redis问题求助

环境信息 #

• 系统：Ubuntu 20.04
• 组件：Redis、Celery、Python 3.8.10、Django 4.0.7
• 虚拟环境路径：/home/worker/Fusion/envFCorp

已完成操作 #

1. 安装并加固Redis，设置requirepass为foobared、禁用危险命令，redis-cli ping可正常返回pong；
2. 配置/etc/default/celeryd文件，内容如下：

#Where your Celery is present
CELERY_BIN="/home/worker/Fusion/envFCorp/bin/celery"

# App instance to use 
CELERY_APP="app.celery"
#CELERY_APP="FusionCorp"

# Where to chdir at start
CELERYD_CHDIR="/home/worker/Fusion/FusionCorp/FusionCorp/"

# Extra command-line arguments to the worker 
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# %n will be replaced with the first part of the nodename. 
CELERYD_LOG_FILE="/var/log/celery/%n%I.log" 
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
# You need to create this user manually (or you can choose
# A user/group combination that already exists (e.g., nobody). 
CELERYD_USER="celery" 
CELERYD_GROUP="celery"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured. 
CELERY_CREATE_DIRS=1
export SECRET_KEY="MYSECRETPASSWORD"

遇到的问题 #

1. 执行sudo systemctl enable redis报错：Failed to enable unit: Refusing to operate on alias name or linked unit file: redis.service；
2. Redis开启requirepass后，手动启动Celery Worker/Beat命令报错，关闭密码则正常运行；
3. 激活虚拟环境执行sudo /etc/init.d/celeryd start返回启动提示，但sudo /etc/init.d/celeryd status显示celeryd down: no pidfiles found；
4. 尝试supervisor配置未成功，需实现Celery Worker与Beat开机自启。

问题1：Redis服务启用失败 #

Ubuntu 20.04中Redis的系统服务标准名称是redis-server.service，直接操作redis.service别名会触发报错。执行以下命令启用并启动Redis：

sudo systemctl enable redis-server.service
sudo systemctl start redis-server.service

问题2：Celery连接带密码的Redis失败 #

在Django项目的settings.py中添加带密码的Redis连接配置，确保Celery能正确完成身份认证：

# settings.py
CELERY_BROKER_URL = 'redis://:foobared@localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://:foobared@localhost:6379/0'

格式说明：redis://:密码@主机地址:端口/数据库编号，若Redis绑定了特定IP，替换localhost为对应IP即可。

问题3：celeryd启动后无PID文件 #

步骤1：修复目录权限 #

确保celery用户对日志、PID目录拥有读写权限：

sudo chown -R celery:celery /var/log/celery /var/run/celery
sudo chmod -R 755 /var/log/celery /var/run/celery

步骤2：修正celeryd配置参数 #

• 调整CELERY_APP：若Celery实例文件是FusionCorp/celery.py，则将CELERY_APP设为FusionCorp.celery，注释掉app.celery；
• 修正CELERYD_CHDIR：该路径需指向Django项目根目录（即包含manage.py的目录），将当前配置的/home/worker/Fusion/FusionCorp/FusionCorp/调整为/home/worker/Fusion/FusionCorp/；
• 添加Redis连接环境变量：在/etc/default/celeryd末尾补充：

export CELERY_BROKER_URL="redis://:foobared@localhost:6379/0"
export CELERY_RESULT_BACKEND="redis://:foobared@localhost:6379/0"

步骤3：重新启动并检查状态 #

sudo /etc/init.d/celeryd stop
sudo /etc/init.d/celeryd start
sudo /etc/init.d/celeryd status

若仍有问题，查看/var/log/celery下的日志文件排查具体错误。

问题4：Celery Worker与Beat开机自启（推荐systemd方案） #

传统celeryd init脚本兼容性较差，推荐使用systemd配置自启：

配置Celery Worker服务 #

创建/etc/systemd/system/celery-worker.service文件：

[Unit]
Description=Celery Worker Service
After=network.target redis-server.service

[Service]
Type=forking
User=celery
Group=celery
WorkingDirectory=/home/worker/Fusion/FusionCorp/
Environment="PATH=/home/worker/Fusion/envFCorp/bin"
ExecStart=/home/worker/Fusion/envFCorp/bin/celery multi start worker -A FusionCorp.celery --time-limit=300 --concurrency=8 --loglevel=info --logfile=/var/log/celery/worker.log --pidfile=/var/run/celery/worker.pid
ExecStop=/home/worker/Fusion/envFCorp/bin/celery multi stopwait worker --pidfile=/var/run/celery/worker.pid
ExecReload=/home/worker/Fusion/envFCorp/bin/celery multi restart worker -A FusionCorp.celery --time-limit=300 --concurrency=8 --loglevel=info --logfile=/var/log/celery/worker.log --pidfile=/var/run/celery/worker.pid

[Install]
WantedBy=multi-user.target

配置Celery Beat服务 #

创建/etc/systemd/system/celery-beat.service文件：

[Unit]
Description=Celery Beat Service
After=network.target redis-server.service celery-worker.service

[Service]
Type=simple
User=celery
Group=celery
WorkingDirectory=/home/worker/Fusion/FusionCorp/
Environment="PATH=/home/worker/Fusion/envFCorp/bin"
ExecStart=/home/worker/Fusion/envFCorp/bin/celery -A FusionCorp.celery beat --loglevel=info --logfile=/var/log/celery/beat.log --pidfile=/var/run/celery/beat.pid

[Install]
WantedBy=multi-user.target

启用并启动服务 #

# 重新加载systemd配置
sudo systemctl daemon-reload

# 启用开机自启
sudo systemctl enable celery-worker.service
sudo systemctl enable celery-beat.service

# 启动服务
sudo systemctl start celery-worker.service
sudo systemctl start celery-beat.service

# 检查状态
sudo systemctl status celery-worker.service
sudo systemctl status celery-beat.service

内容的提问来源于stack exchange，提问作者AhmedRaza