Celery执行简单任务时触发KeyError: 'tasks.add'问题排查求助

错误详情 #

执行Celery任务时触发如下错误：

The full contents of the message body was:
b'[[4, 4], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (81b)

The full contents of the message headers:
{'lang': 'py', 'task': 'tasks.add', 'id': '8f2b3e82-8a3b-4ee3-9038-574931e0022b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '8f2b3e82-8a3b-4ee3-9038-574931e0022b', 'parent_id': None, 'argsrepr': '(4, 4)', 'kwargsrepr': '{}', 'origin': 'gen130049@marlowe.cc.ucsf.edu', 'ignore_result': False}

The delivery info for this task is:
{'exchange': '', 'routing_key': 'celery'}
Traceback (most recent call last):
  File ".../celery/worker/consumer/consumer.py", line 591, in on_task_received
    strategy = strategies[type_]
KeyError: 'tasks.add'

相关代码文件 #

tasks.py（任务定义） #

# tasks.py
from celery_config import app
from save_dataframe import save_dataframe
import pandas as pd
    
@app.task
def add(x, y):
    return x + y

celery_config.py（Celery配置） #

from celery import Celery

broker_url = 'redis://:password@ec2-...compute.amazonaws.com:6379/0'
result_backend = 'redis://:password@ec2-...compute.amazonaws.com:6379/0'

app = Celery('tasks', broker=broker_url, backend=result_backend)

Worker启动情况 #

使用命令启动Worker：

celery -A celery_config worker --loglevel=info

启动输出如下（关键可见[tasks]部分为空，说明Worker未加载到任务）：

celery -A celery_config worker --loglevel=info
 
 -------------- celery@marlowe.cc.ucsf.edu v5.2.7 (dawn-chorus)
--- ***** ----- 
-- ******* ---- Linux-3.10.0-1127.18.2.el7.x86_64-x86_64-with-glibc2.17 2023-03-26 10:42:17
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x7f709e0066b0
- ** ---------- .> transport:   redis://:**@...amazonaws.com:6379/0
- ** ---------- .> results:     redis://:**@...amazonaws.com:6379/0
- *** --- * --- .> concurrency: 28 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]


[2023-03-26 10:42:18,717: INFO/MainProcess] Connected to redis://...compute.amazonaws.com:6379/0
[2023-03-26 10:42:18,788: INFO/MainProcess] mingle: searching for neighbors
[2023-03-26 10:42:20,117: INFO/MainProcess] mingle: all alone
[2023-03-26 10:42:20,561: INFO/MainProcess] celery@... ready.

解决方案 #

问题根源是Celery Worker未自动发现并注册tasks.py中的任务。在celery_config.py中添加任务自动发现代码即可解决：

app.autodiscover_tasks(['tasks'], related_name='tasks', force=True)

内容的提问来源于stack exchange，提问作者Will