Traefik连接Docker Swarm跨节点服务时出现504错误求助

我用Docker Swarm部署服务，Traefik（v2.9.6）和web1（Nginx）在同一管理节点（主机名arasaac2）时运行正常；将web1迁移到工作节点（主机名swarmtest）后，访问web1出现504网关超时错误。Traefik日志显示请求超时，但从Traefik容器能成功ping通web1的容器IP。

Stack配置文件 #

services:

  traefik:
    image: traefik:v2.9.6
    ports:
      - 80:80
      - 443:443
    deploy:
      placement:
        constraints:
          - 'node.hostname == arasaac2'
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik-public
        - traefik.http.middlewares.admin-auth.basicauth.users=${USERNAME?Variable not set}:${HASHED_PASSWORD?Variable not set}
        - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
        - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true
        - traefik.http.routers.traefik-public-http.rule=Host(`${DOMAIN?Variable not set}`)
        - traefik.http.routers.traefik-public-http.entrypoints=http
        - traefik.http.routers.traefik-public-http.middlewares=https-redirect
        - traefik.http.routers.traefik-public-https.rule=Host(`${DOMAIN?Variable not set}`)
        - traefik.http.routers.traefik-public-https.entrypoints=https
        - traefik.http.routers.traefik-public-https.tls=true
        - traefik.http.routers.traefik-public-https.service=api@internal
        - traefik.http.routers.traefik-public-https.tls.certresolver=le
        - traefik.http.routers.traefik-public-https.middlewares=admin-auth
        - traefik.http.services.traefik-public.loadbalancer.server.port=8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik-public-certificates:/certificates
    command:
      - --providers.docker
      - --providers.docker.exposedbydefault=false
      - --providers.docker.network=traefik-public
      - --providers.docker.swarmmode
      - --entrypoints.http.address=:80
      - --entrypoints.https.address=:443
      - --certificatesresolvers.le.acme.email=${EMAIL?Variable not set}
      - --certificatesresolvers.le.acme.storage=/certificates/acme.json
      - --certificatesresolvers.le.acme.tlschallenge=true
      - --accesslog
      - --log=DEBUG
      - --api
    networks:
      - traefik-public

  web1:
    image: 'nginx'
    networks:
      - traefik-public
    deploy:
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - 'node.hostname == swarmtest'
      replicas: 1
      labels:
        - traefik.enable=true
        - traefik.http.routers.web1.entrypoints=http
        - traefik.docker.network=traefik-public
        - traefik.http.services.web1.loadbalancer.server.port=80
        - traefik.http.services.web1.loadbalancer.server.scheme=http
        - traefik.http.routers.web1.tls=false
        # - traefik.http.routers.web1.tls.certresolver=production
        - traefik.http.routers.web1.rule=Host(`web1.arasaac.org`)

volumes:
  traefik-public-certificates:

networks:
  traefik-public:
    external: true

Traefik相关日志 #

proxy_traefik.1.6sld8nuc0wo6@arasaac2    | 10.0.0.2 - - [26/Apr/2023:11:05:12 +0000] "GET / HTTP/1.1" 499 21 "-" "-" 27 "web1@docker" "http://10.0.5.30:80" 3925ms
proxy_traefik.1.6sld8nuc0wo6@arasaac2    | 10.0.0.2 - - proxy_traefik.1.6sld8nuc0wo6@arasaac2    | 10.0.0.2 - - [26/Apr/2023:11:05:16 +0000] "GET / HTTP/1.1" 504 15 "-" "-" 28 "web1@docker" "http://10.0.5.30:80" 30000ms

排查与解决思路 #

• 检查节点间防火墙规则
  工作节点swarmtest可能存在防火墙限制，阻止了Traefik所在节点（arasaac2）访问web1容器的80端口。确保所有Swarm节点间开放以下端口：7946（TCP/UDP，节点间通信）、4789（UDP，覆盖网络数据传输）以及容器服务使用的80端口。

• 确认覆盖网络配置
  外部网络traefik-public需是Swarm覆盖网络，执行以下命令验证：

  docker network inspect traefik-public
  

  检查输出中Scope是否为swarm，且所有节点都在Peers列表中。如果不是覆盖网络，重新创建：

  docker network create --driver=overlay --attachable traefik-public
  

• 调整Traefik负载均衡超时设置
  在Traefik的command中添加超时配置：

  - --serversTransport.forwardingTimeouts.dialTimeout=60s
  - --serversTransport.forwardingTimeouts.responseHeaderTimeout=60s
  

  同时在web1的服务标签中添加：

  - traefik.http.services.web1.loadbalancer.timeout.serverResponseTimeout=60s
  

• 验证web1容器端口监听
  进入swarmtest节点的web1容器，确认Nginx监听80端口：

  docker exec -it <web1-container-id> netstat -tulpn
  

  确保输出中有0.0.0.0:80的监听记录。

内容的提问来源于stack exchange，提问作者user2670996