Absolutely! You can absolutely set concurrency limits per queue in Celery—this is a super common need when dealing with resource-heavy tasks like your memory-intensive ones in q2. Let me break down the most straightforward, reliable ways to make this happen:

Method 1: Launch Separate Worker Processes (Quick & Direct) #

Celery workers are independent processes, so the easiest way to control concurrency per queue is to spin up dedicated workers for each queue, each with their own concurrency setting.

• For your general tasks in q1, start a worker with a higher concurrency (adjust the number based on your server's resources):

  celery -A your_celery_app worker -Q q1 --concurrency=4
  

  (The --concurrency=4 means this worker can run up to 4 tasks at once.)

• For your memory-heavy q2 tasks, start a separate worker with concurrency set to 1 (so only one task runs at a time):

  celery -A your_celery_app worker -Q q2 --concurrency=1
  

This approach gives you clean resource isolation—your q2 tasks won't hog resources from q1, and you're guaranteed no more than one memory-intensive task runs simultaneously.

Method 2: Use a Config File (For Production/Repeatability) #

If you want to formalize your queue setup and avoid typing long commands every time, create a Celery config file (e.g., celeryconfig.py) to define your queues and routing rules:

# celeryconfig.py
from celery import Celery

app = Celery('your_app_name')

# Define your queues
app.conf.task_queues = {
    'q1': {
        'exchange': 'q1',
        'exchange_type': 'direct',
        'routing_key': 'q1',
    },
    'q2': {
        'exchange': 'q2',
        'exchange_type': 'direct',
        'routing_key': 'q2',
    },
}

# Route tasks to the correct queues
app.conf.task_routes = {
    # Replace with your actual task paths
    'your_app.tasks.memory_intensive_task': {'queue': 'q2'},
    'your_app.tasks.*': {'queue': 'q1'},  # Catch-all for other tasks
}

Then start your workers using this config:

# For q1
celery -A your_celery_app worker -Q q1 --concurrency=4 --config=celeryconfig

# For q2
celery -A your_celery_app worker -Q q2 --concurrency=1 --config=celeryconfig

Important Note #

Celery doesn't support setting per-queue concurrency limits within a single worker process. So splitting into dedicated workers is the only reliable way to enforce this restriction. This is actually a good thing—it keeps your resource-heavy tasks isolated from other workloads, preventing performance bottlenecks.

If you're running Celery in production (e.g., with systemd), you'll want to create separate service files for each worker to ensure they start automatically and stay running.

内容的提问来源于stack exchange，提问作者Yuanzheng Xiao