Celery+AWS SQS环境下设置整数retry_backoff导致重试混乱、任务激增问题求助
Celery+AWS SQS环境下设置整数retry_backoff导致重试混乱、任务激增问题求助
大家好,我目前在使用Celery搭配AWS SQS处理异步任务,最近碰到了一个棘手的问题,想向各位大佬求助。
我的任务定义代码如下:
@app.task( autoretry_for=(Exception,), max_retries=5, retry_backoff=True, retry_jitter=False, acks_late=True, ) @onfailure_reject(non_traced_exceptions=NON_TRACED_EXCEPTIONS) def send_order_update_event_task(order_id, data): .........
但当我把retry_backoff参数设置为整数值时,重试模式彻底混乱了,任务数量直接失控。
相关错误日志如下:
2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [53285c923f-79232a3856] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [1052f09663-c19b42589a] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [dd021828dd-4f6b8ae6f8] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [116bef9273-e4dbfb526b] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [913697ae7e-d4f65d45a5] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [d99e889882-a76718b549] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [d99e889882-30bac3e515] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [d7f01e5b4f-edfa22355f] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [8ba15966ae-2266247e56] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [738688f34d-34067ca58b] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [c790586783-b363d38520] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [6231986f4c-7696b7cf47] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10 2024-12-10 05:16:10 ERROR [1b810665-c0b1-4527-8cd9-c142f67d6605] [e020ded4ca-f11c933d87] tasks.order_request_task - [ send_order_update_event_task] Exception for order: 700711926: Order absent 700711926, retry_count: 10
我在代码里打印了每次重试的计数,结果发现同一个重试计数对应了大量重复任务——比如重试计数1有20个任务,计数2有40个,完全超出了预期。
补充下环境信息:所有任务都使用特定队列celery-requests-primary,并且运行在名为celery-requests-primary的K8s部署中,该部署包含多个Pod。
想请教各位,这种情况可能是什么原因导致的?如果需要进一步排查,我还需要提供哪些信息?
备注:内容来源于stack exchange,提问作者Dev
相关产品推荐
相关产品推荐

