如何避免Celery在django_celery_results_taskresult中截断task_args文本

问题现象 #

在Django项目中使用Celery执行异步任务时，出现django_celery_results_taskresult表的task_args列内容被截断、末尾带省略号的问题。预期存储完整的process_4_items参数值，但实际运行时无法实现，Celery Worker的命令行输出中也能看到同样的截断内容，仅逐行调试项目时task_args可以显示完整内容。

相关配置 #

celeryconfig.py #

broker_url = 'amqp://localhost:5672//'
imports = ('operation_hawkeye.tasks',)
result_backend = 'django-db'
accept_content = ['json']
task_always_eager = False

views.py 任务调用代码 #

from celery import group
process_4_items = [
  {
    "service_id": "7f7962b082cb8da55e01f4f013d8fefbc97948ec03400c93423b050d",
    "service_id_unique": "c30ed891987f4d8d3debcaef7edb0f582e6b634f1ebed00fb88f0ec7",
    "service_type": "buzz_collect_compare",
    "service_file_pf_folder": "",
    "service_file_nanonsportsproductivity_folder": "",
    "service_file_tba_report": "",
    "service_file_buzz_collect_compare": "services/buzzcollect_xml_compare/2021/06/01/file_2_name_Update_COOKINGHD_May_2021_Listin_yWoWXo1.xml",
    "service_file_type": "XML 1.0 document, ASCII text",
    "service_file_name": "file_2_name_Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml",
    "service_file_size": 396557,
    "service_status": "In Queue for Upload",
    "service_status_remarks": {
      "source_information": {
        "source_id": "5191",
        "source_name": "10NEWS2",
        "file_1_name": "Original_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml",
        "file_2_name": "Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml"
      }
    },
    "service_creation_date": "2021-06-01T18:18:40.968Z",
    "service_owner": "p.premlal@spi-global.com"
  },
  {
    "service_id": "7f7962b082cb8da55e01f4f013d8fefbc97948ec03400c93423b050d",
    "service_id_unique": "2edaf2535945992d13b9ee85e908b21cbd520a02642bd7faf7f229bc",
    "service_type": "buzz_collect_compare",
    "service_file_pf_folder": "",
    "service_file_nanonsportsproductivity_folder": "",
    "service_file_tba_report": "",
    "service_file_buzz_collect_compare": "services/buzzcollect_xml_compare/2021/06/01/file_1_name_Original_COOKINGHD_May_2021_List_P2PlYY7.xml",
    "service_file_type": "XML 1.0 document, ASCII text",
    "service_file_name": "file_1_name_Original_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml",
    "service_file_size": 396409,
    "service_status": "In Queue for Upload",
    "service_status_remarks": {
      "source_information": {
        "source_id": "5191",
        "source_name": "10NEWS2",
        "file_1_name": "Original_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml",
        "file_2_name": "Update_COOKINGHD_May 2021 Listings(4-26-21 thru 5-30-21).xml"
      }
    },
    "service_creation_date": "2021-06-01T18:18:40.935Z",
    "service_owner": "p.premlal@spi-global.com"
  }
]
jobs_4 = group(tasks.buzz_collect_compare_main.s(process_4_items))
result_4 = jobs_4.apply_async()

tasks.py 任务定义代码 #

@app.task(bind=True)
def buzz_collect_compare_main(self, items):
    process_items = items
    ...

解决方案 #

Celery默认对任务参数的字符串表示长度做了限制，导致过长的参数在日志、结果存储时被截断。只需要在celery.py中调整app.amqp.argsrepr_maxsize参数，调大最大长度限制即可解决问题，示例配置如下：

import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hawkeye.settings')
os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')
app = Celery('hawkeye')
default_config = 'hawkeye.celeryconfig'
app.config_from_object(default_config)
# 调整参数表示最大长度为10MB，可根据实际需求修改
app.amqp.argsrepr_maxsize = 10485760

内容的提问来源于stack exchange，提问作者Premlal Premkumar