Great question—this is a super common pain point when trying to prioritize tasks without wasting dyno resources on Heroku. The traditional multi-queue approach does have those idle worker issues, but there are smarter ways to handle this that keep your workers flexible and scalable. Let’s break down the most effective solutions:

1. Weighted Queue Consumption (Simplest & Most Maintainable) #

Instead of locking workers to specific queues, configure all your workers to listen to both high-priority and default queues, but give the high-priority queue a higher weight. This way, workers will prioritize grabbing tasks from the high-priority queue first, but fall back to the default queue when there’s nothing urgent to process. No idle workers, no split dyno groups—all workers are identical and scalable.

How to set this up: #

• Celery Worker Start Command: Use the --queue-weight flag to assign weights (higher numbers mean more priority). For example:

  celery -A your_django_app worker -Q high_priority,default --queue-weight high_priority=3,default=1
  

  This tells each worker to check the high_priority queue 3 times for every 1 time it checks default.

• Task Routing: Assign high-priority tasks to their queue using the Celery decorator:

  from celery import shared_task
  
  @shared_task(queue='high_priority')
  def urgent_processing_task(data):
      # Your high-priority logic here
      pass
  
  @shared_task  # Uses default queue automatically
  def regular_processing_task(data):
      # Your standard logic here
      pass
  

• Heroku Scaling: Since all workers are identical, you can scale your worker dynos uniformly with heroku ps:scale worker=5—no need to manage separate dyno types.

2. Redis Sorted Sets for Single-Queue Priority (Resource-Efficient) #

While single-queue priority is often warned against, it’s actually feasible with Redis by leveraging sorted sets instead of the default list storage. Celery supports this natively, letting you assign priority values to tasks so workers pull higher-priority tasks first—all within a single queue. This eliminates queue fragmentation entirely.

Configuration Steps: #

• Update your Django settings to enable priority handling in Redis:

  CELERY_BROKER_URL = os.environ.get('REDIS_URL')
  CELERY_BROKER_TRANSPORT_OPTIONS = {
      'priority_steps': list(range(10)),  # 0 = lowest, 9 = highest
      'queue_order_strategy': 'priority'
  }
  

• Assign priority to tasks when calling them:

  # Send a high-priority task (priority 9)
  regular_processing_task.apply_async(args=[data], priority=9)
  
  # Send a low-priority task (priority 1)
  regular_processing_task.apply_async(args=[data], priority=1)
  

This keeps all tasks in one queue, so every worker is always busy if there’s work to do. Heroku scaling stays simple, and you avoid the overhead of managing multiple queues. Just note that Redis sorted sets have a tiny performance hit compared to lists, but it’s negligible for most Django/Heroku workloads.

3. Dynamic Worker Redirection (For Complex Workloads) #

If you have extreme traffic spikes where you need temporary dedicated workers for high-priority tasks, you can automate worker queue assignments using Heroku’s API and a simple monitoring script. For example:

• Write a small Python script that checks the length of the high_priority queue via Redis.
• When the queue exceeds a threshold (e.g., 100 tasks), use the Heroku API to scale up a temporary high_priority_worker dyno type that only listens to the high-priority queue.
• When the queue clears, scale those dynos back down to 0.

This is more complex but useful if you have rare, large bursts of urgent tasks. However, it adds operational overhead, so I’d only recommend it if the first two solutions don’t meet your needs.

Final Recommendation #

For most Heroku + Celery + Redis setups, weighted queue consumption is the sweet spot—it’s simple to implement, keeps workers fully utilized, and maintains easy scalability. If you want to avoid multiple queues entirely, the sorted set single-queue approach works great too. Both eliminate idle dynos and let you scale workers uniformly.

内容的提问来源于stack exchange，提问作者Martin Faucheux