FastAPI集成Celery时无法通过YAML配置日志的问题

环境与现状 #

• 采用FastAPI（Uvicorn启动）作为Web服务器，Celery作为后台任务处理器
• FastAPI通过以下命令加载logging.yaml日志配置，运行正常：

  uvicorn syncapp.main:app --workers 4 --host 0.0.0.0 --port 8084 --log-config logging.yaml
  

• Celery无法复用logging.yaml中定义的日志器，尤其是基于QueueListenerHandler的异步日志配置完全失效

Celery核心代码 #

Worker定义（路径：syncapp/worker.py） #

from celery import Celery

from syncapp.settings import get_settings

settings = get_settings()

app = Celery(
    main=settings.app_name,
    broker=settings.celery_broker_url,
    backend=settings.celery_result_backend,
    include="syncapp.tasks.copy_data",
)

app.conf.update(
    broker_connection_retry_on_startup=True,
    worker_hijack_root_logger=False,
)

任务示例（路径：syncapp/tasks/copy_data.py） #

from celery.utils.log import get_task_logger
import celery

from syncapp.commons.enums import WorkerQueue
from syncapp.worker import app

logger = get_task_logger(__name__)


@app.task(queue=WorkerQueue.COPY_DATA.value)
def copy_index_data(index_name: str) -> bool:
    """Copies index data up until the current date."""
    task_id = celery.current_task.request.id
    logger.info("Job %s: Copying data from source index %s", task_id, index_name)
    return True

logging.yaml关键配置片段 #

version: 1

...
handlers:
  hl1_console:
    class: logging.StreamHandler
    formatter: simple
    stream: ext://sys.stdout

  hl1_syncapp_file_handler:
    class: logging.FileHandler
    filename: logs/syncapp.log
    formatter: extended

  hl1_synctask_file_handler:
    class: logging.FileHandler
    filename: logs/synctask.log
    formatter: extended

  hl1_celery_file_handler:
    class: logging.FileHandler
    filename: logs/celery.log
    formatter: extended

  hl2_synctask_queue_handler:
    class: logging_.handlers.QueueListenerHandler
    handlers:
      - cfg://handlers.hl1_console
      - cfg://handlers.hl1_synctask_file_handler
    queue: cfg://objects.synctask_queue

  hl2_celery_queue_handler:
    class: logging_.handlers.QueueListenerHandler
    handlers:
      - cfg://handlers.hl1_console
      - cfg://handlers.hl1_celery_file_handler
    queue: cfg://objects.celery_queue

  hl2_syncapp_queue_handler:
    class: logging_.handlers.QueueListenerHandler
    handlers:
      - cfg://handlers.hl1_console
      - cfg://handlers.hl1_syncapp_file_handler
    queue: cfg://objects.syncapp_queue

loggers:
  celery.task:
    level: DEBUG
    handlers:
      - hl2_synctask_queue_handler
    propagate: no
  celery:
    level: INFO
    handlers:
      - hl2_celery_queue_handler
    propagate: no
  syncapp:
    level: DEBUG
    handlers:
      - hl2_syncapp_queue_handler
    propagate: no

root:
  level: INFO
  handlers:
    - hl1_console

Celery Worker启动命令 #

celery -A syncapp.worker:app worker --concurrency=4 --hostname=copy@%h --queues=syncapp.tasks.copy_data --loglevel=INFO

期望日志规则 #

• Celery后台运行事件由celery日志器记录
• 任务执行日志由syncapp.tasks日志器记录
• 主应用日志（已正常工作）由syncapp日志器记录

已尝试方案与未解决问题 #

1. 通过Celery的signals.after_setup_logger和signals.after_setup_task_logger信号手动初始化日志器：
   • QueueListenerHandler因多线程访问队列对象无法输出日志
   • 必须指定精确的日志器名称，适配性差
2. 开启日志传播到root后，能看到基础日志输出，但QueueListenerHandler仍完全失效
3. 临时改用直接的文件/控制台处理器替代QueueListenerHandler，现寻求线程安全使用QueueListenerHandler的可行方案

内容的提问来源于stack exchange，提问作者Zobayer Hasan