能否利用Celery与Redis实现Dead Letter Queue？Redis是否支持死信队列？

Great questions! Let's break them down clearly:

Celery doesn't have native Dead Letter Queue (DLQ) support when using Redis as the broker, but you absolutely can build a DLQ-like system with custom logic. Here are a few practical approaches:

• Leverage Celery's failure callbacks: Define an on_failure method in your task class that sends failed task details (arguments, exception info, timestamp) to a dedicated Redis list (your makeshift DLQ). You can then spin up a separate worker process to consume and handle tasks from this list.
  Example snippet:

  from celery import Task
  import redis
  
  r = redis.Redis()
  
  class DLQEnabledTask(Task):
      def on_failure(self, exc, task_id, args, kwargs, einfo):
          # Push failed task to Redis DLQ
          r.rpush('celery_dlq', {
              'task_id': task_id,
              'args': args,
              'kwargs': kwargs,
              'exception': str(exc),
              'einfo': str(einfo)
          })
  
  # Use this base class for your tasks
  @app.task(base=DLQEnabledTask)
  def my_task():
      # Task logic that might fail
      raise ValueError("Something went wrong!")
  

• Retry limits with a fallback queue: Set a maximum retry count for tasks; once retries are exhausted, route the task to a separate, dedicated queue (configured in Celery) that acts as your DLQ. You can then process this queue independently.

• Community extensions: Some third-party tools like celery-deadletter add DLQ capabilities, though you'll want to verify compatibility with Redis as your broker.

No, Redis does not have native DLQ functionality built-in (unlike RabbitMQ which offers direct DLX/DLQ support). That said, you can simulate a DLQ using Redis's existing features:

• Keyspace notifications + expiry: Enable Redis keyspace notifications for expired keys. When a task (stored as a key) expires, trigger a script that moves the task data to a dedicated "dead letter" list or stream.

• Redis Streams + Consumer Groups: Use Redis Streams for task queues, and leverage consumer groups' ACK mechanism. If a message isn't acknowledged within a set timeframe, write custom logic to move it to a separate stream (your DLQ) for later review or reprocessing.

• Manual failure tracking: For simpler use cases, track task failure counts manually. If a task fails a specified number of times, move it to a dedicated Redis list that functions as your DLQ.

All these approaches require you to implement the DLQ logic yourself—Redis doesn't handle this workflow out of the box.

内容的提问来源于stack exchange，提问作者paulo.pinda