Celery+RabbitMQ+Docker环境下‘Lookup timed out’错误排查与解决

我的FastAPI应用运行在Docker Compose中，搭配Celery和RabbitMQ，此前正常运行数月，现在FastAPI容器无法连接RabbitMQ，报错[Errno -3] Lookup timed out。RabbitMQ日志正常，Flower容器能正常连接，但FastAPI容器尝试更换AMQP地址、更新依赖、改用Redis都无效。

Docker Compose 配置 #

services:
scraper:
    build:
      context: fastapi-scrapers
      dockerfile: Dockerfile
    cpus: "0.7"
    # ports:
    #   - '8000:8000'
    environment:
      - TIMEOUT="120"
      - WEB_CONCURRENCY=2
    networks:
      - scrape-net
    volumes:
      - ../images/:/app/images/:rw
    extra_hosts:
      - "host.docker.internal:host-gateway"

flower:
    image: mher/flower
    ports:
      - '5555:5555'
    environment:
      - CELERY_BROKER_URL=amqp://admin:pass@rabbitmq:5672/
      # - CELERY_BROKER_URL=redis://redis:6379/0
      - FLOWER_BASIC_AUTH=admin:pass
    depends_on:
      - scraper
    networks:
      - scrape-net

rabbitmq:
    image: "rabbitmq:latest"
    # ports:
      # - '5672:5672'
      # - "15672:15672"
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=pass
    networks:
      - scrape-net
    extra_hosts:
      - "host.docker.internal:host-gateway"

networks:
  scrape-net:
    driver: bridge

FastAPI Dockerfile #

FROM python:3.9

WORKDIR /code

COPY ./requirements.txt /code/requirements.txt

RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

COPY ./app /code/app

# CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
CMD ["bash", "-c", "celery -A app.celery.tasks worker --loglevel=info --concurrency=8 -E -P eventlet & uvicorn app.main:app --host 0.0.0.0 --port 8000"]

Celery 配置代码 #

celery_app = Celery('tasks', broker='amqp://admin:pass@rabbitmq:5672/')

celery_app.conf.update(
    CELERY_RESULT_EXPIRES=3600,
    CELERY_AMQP_TASK_RESULT_EXPIRES=3600
)

FastAPI 容器日志片段 #

site-scraper-1  | INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
site-scraper-1  | INFO:     Started parent process [8]
site-scraper-1  | INFO:     Started server process [11]
site-scraper-1  | INFO:     Waiting for application startup.
site-scraper-1  | INFO:     Application startup complete.
site-scraper-1  | INFO:     Started server process [10]
site-scraper-1  | INFO:     Waiting for application startup.
site-scraper-1  | INFO:     Application startup complete.
site-scraper-1  | /usr/local/lib/python3.9/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
site-scraper-1  | absolutely not recommended!
site-scraper-1  | 
site-scraper-1  | Please specify a different user using the --uid option.
site-scraper-1  | 
site-scraper-1  | User information: uid=0 euid=0 gid=0 egid=0
site-scraper-1  | 
site-scraper-1  |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
site-scraper-1  |  
site-scraper-1  |  -------------- celery@d4dec9482220 v5.3.6 (emerald-rush)
site-scraper-1  | --- ***** ----- 
site-scraper-1  | -- ******* ---- Linux-5.10.0-26-amd64-x86_64-with-glibc2.36 2024-03-10 22:26:52
site-scraper-1  | - *** --- * --- 
site-scraper-1  | - ** ---------- [config]
site-scraper-1  | - ** ---------- .> app:         tasks:0x7fd3a198f2e0
site-scraper-1  | - ** ---------- .> transport:   amqp://admin:**@rabbitmq:5672//
site-scraper-1  | - ** ---------- .> results:     disabled://
site-scraper-1  | - *** --- * --- .> concurrency: 8 (eventlet)
site-scraper-1  | -- ******* ---- .> task events: ON
site-scraper-1  | --- ***** ----- 
site-scraper-1  |  -------------- [queues]
site-scraper-1  |                 .> celery           exchange=celery(direct) key=celery
site-scraper-1  |                 
site-scraper-1  | 
site-scraper-1  | [tasks]
site-scraper-1  |   . app.celery.tasks.start_scrape
site-scraper-1  | 
site-scraper-1  | [2024-03-10 22:26:52,909: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
site-scraper-1  | whether broker connection retries are made during startup in Celery 6.0 and above.
site-scraper-1  | If you wish to retain the existing behavior for retrying connections on startup,
site-scraper-1  | you should set broker_connection_retry_on_startup to True.
site-scraper-1  |   warnings.warn(
site-scraper-1  | 
site-scraper-1  | [2024-03-10 22:27:13,338: ERROR/MainProcess] consumer: Cannot connect to amqp://admin:**@rabbitmq:5672//: [Errno -3] Lookup timed out.
site-scraper-1  | Trying again in 2.00 seconds... (1/100)
site-scraper-1  | 
site-scraper-1  | [2024-03-10 22:27:35,769: ERROR/MainProcess] consumer: Cannot connect to amqp://admin:**@rabbitmq:5672//: [Errno -3] Lookup timed out.
site-scraper-1  | Trying again in 4.00 seconds... (2/100)

1. 修复 Docker Compose 依赖顺序 #

当前flower依赖scraper，但scraper需要等待rabbitmq完全启动才能连接。修改depends_on并添加健康检查，确保RabbitMQ就绪后再启动scraper：

更新docker-compose.yml：

services:
  scraper:
    # ... 其他配置
    depends_on:
      rabbitmq:
        condition: service_healthy
    # ... 其他配置

  rabbitmq:
    image: "rabbitmq:latest"
    # ... 其他配置
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
      interval: 5s
      timeout: 10s
      retries: 5

2. 检查 Eventlet 兼容性问题 #

Celery 使用eventlet作为并发池时，可能和最新的AMQP库存在兼容性问题。尝试：

• 切换到gevent或者默认的prefork并发池：
  修改Dockerfile中的CMD：

  celery -A app.celery.tasks worker --loglevel=info --concurrency=8 -E -P gevent & uvicorn app.main:app --host 0.0.0.0 --port 8000
  

  或者去掉-P eventlet使用默认prefork：

  celery -A app.celery.tasks worker --loglevel=info --concurrency=8 -E & uvicorn app.main:app --host 0.0.0.0 --port 8000
  

• 固定kombu和amqp库的版本到之前能正常运行的版本，比如在requirements.txt中添加：

  kombu==5.3.2
  amqp==5.1.1
  

3. 验证容器网络连通性 #

进入scraper容器，手动测试DNS解析和端口连通：

# 进入容器
docker exec -it site-scraper-1 bash

# 解析 rabbitmq 主机名
nslookup rabbitmq

# 测试端口连通性
telnet rabbitmq 5672

如果解析失败，检查Docker网络是否正常，尝试重建网络：

docker-compose down -v
docker-compose up -d

4. 修正 Celery 配置中的过时参数 #

当前Celery配置使用了过时的CELERY_*前缀，改为新的无前缀配置，同时添加连接重试参数：

celery_app = Celery('tasks', broker='amqp://admin:pass@rabbitmq:5672/')

celery_app.conf.update(
    result_expires=3600,
    amqp_task_result_expires=3600,
    broker_connection_retry_on_startup=True  # 解决启动时连接重试的警告
)

5. 检查容器资源限制 #

scraper容器设置了cpus: "0.7"，如果宿主机资源紧张，可能导致DNS解析超时。尝试临时移除CPU限制，观察是否恢复正常：

scraper:
    # cpus: "0.7"  # 注释掉这行
    # ... 其他配置

内容的提问来源于stack exchange，提问作者sleep3000