请求页面30秒后Traefik返回500错误排查求助

此前在3台服务器上搭建的Docker Swarm环境中，以Keycloak为认证提供商、搭配forwardauth和errors中间件的OAuth2-proxy认证服务运行正常。但新增服务器部署后，请求页面会在30秒后返回HTTP 500错误。即使将Traefik日志级别设为DEBUG，仅能得到如下日志：

14.168.51.149 - - [26/Sep/2022:04:29:09 +0000] "GET / HTTP/2.0" 500 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" 2412 "auth_testing@docker" "-" 30000ms

OAuth2-proxy配置 #

oauth:
    image: quay.io/oauth2-proxy/oauth2-proxy:v7.1.1
    deploy:
      placement:
        constraints: [node.labels.web==web]
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.oauth.rule=Host(`oauth.${HOST_ABBR}.domain.org`) || PathPrefix(`/oauth2`)"
        - "traefik.http.routers.oauth.entrypoints=websecure"
        - "traefik.http.routers.oauth.tls=true"
        - "traefik.http.routers.oauth.tls.certresolver=leresolver"
        # Set up service
        - "traefik.http.routers.oauth.service=oauth@docker"
        - "traefik.http.services.oauth.loadbalancer.server.port=4185"
        # Set up middlewares
        - 'traefik.http.middlewares.oauth.forwardauth.address=https://oauth.${HOST_ABBR}.doamin.org/oauth2/auth'
        - 'traefik.http.middlewares.oauth.forwardauth.trustForwardHeader=true'
        - 'traefik.http.middlewares.oauth.forwardauth.authResponseHeaders=X-Auth-Request-User, X-Auth-Request-Groups, X-Auth-Request-Email, X-Auth-Request-Preferred-Username, X-Auth-Request-Access-Token'
        - "traefik.http.middlewares.oauth-signin.errors.service=oauth@docker"
        - "traefik.http.middlewares.oauth-signin.errors.status=401,403"
        - "traefik.http.middlewares.oauth-signin.errors.query=/oauth2/sign_in"
    environment:
      OAUTH2_PROXY_CLIENT_ID: 'coffee_authentication'
      OAUTH2_PROXY_CLIENT_SECRET: 'xxxxxxx99af13b51' 
      
      OAUTH2_PROXY_PROVIDER: 'keycloak'
      OAUTH2_PROXY_SCOPE: 'openid profile email'
      OAUTH2_PROXY_OIDC_ISSUER_URL: '${MY_DOMAIN}/auth/realms/coffee'
      OAUTH2_PROXY_LOGIN_URL: '${MY_DOMAIN}/auth/realms/coffee/protocol/openid-connect/auth'
      OAUTH2_PROXY_REDEEM_URL: '${MY_DOMAIN}/auth/realms/coffee/protocol/openid-connect/token'
      OAUTH2_PROXY_PROFILE_URL: '${MY_DOMAIN}/auth/realms/coffee/protocol/openid-connect/userinfo'
      OAUTH2_PROXY_VALIDATE_URL: '${MY_DOMAIN}/auth/realms/coffee/protocol/openid-connect/userinfo'
      
      OAUTH2_PROXY_PASS_ACCESS_TOKEN: 'true'
      OAUTH2_PROXY_PASS_USER_HEADERS: 'true'
      OAUTH2_PROXY_PASS_BASIC_AUTH: 'true'
      OAUTH2_PROXY_SET_XAUTHREQUEST: 'true'
      OAUTH2_PROXY_SET_AUTHORIZATION_HEADER: 'true'
      OAUTH2_PROXY_COOKIE_DOMAINS: '${MY_DOMAIN}'
      OAUTH2_PROXY_HTTP_ADDRESS: '0.0.0.0:4185'
      OAUTH2_PROXY_COOKIE_REFRESH: '12h'
      OAUTH2_PROXY_COOKIE_SECURE: 'false'
      # python -c 'import os,base64; print(base64.urlsafe_b64encode(os.urandom(32)).decode())'
      OAUTH2_PROXY_COOKIE_SECRET: '${SECRET_COOKIE}'
      OAUTH2_PROXY_AUTHENTICATED_EMAILS_FILE: '/oauth-conf/authorized_emails.txt'
      #OAUTH2_PROXY_EMAIL_DOMAINS: '*'
      OAUTH2_PROXY_WHITELIST_DOMAINS: '*'
      OAUTH2_PROXY_REVERSE_PROXY: 'true'

测试服务Traefik标签配置 #

labels:
        - "traefik.enable=true"
        - "traefik.http.routers.auth_testing.rule=Host(`test.${HOST_ABBR}.domain.org`)"
        - "traefik.http.routers.auth_testing.entrypoints=websecure"
        - "traefik.http.routers.auth_testing.tls=true"
        - "traefik.http.routers.auth_testing.tls.certresolver=leresolver"
        #- "traefik.http.middlewares.auth_testing_auth.basicauth.users=${HOST_USER}:${HOST_HASHED_PASS}"
        #- "traefik.http.routers.auth_testing.middlewares=auth_testing_auth"
        - "traefik.http.routers.auth_testing.middlewares=oauth-signin,oauth"
        # Set up service
        - "traefik.http.services.auth_testing_svc.loadbalancer.server.port=5000"
        - "traefik.http.routers.auth_testing.service=auth_testing_svc"

排查方向 #

• 域名拼写错误：检查OAuth中间件的forwardauth地址，配置中写的是doamin.org，正确应为domain.org，这个错误会导致Traefik无法转发请求到OAuth2-proxy，超时后返回500，这是最优先排查的点。
• 容器网络连通性：在新服务器的Traefik容器内执行curl http://oauth:4185/oauth2/auth，验证是否能正常响应，排查Swarm overlay网络在新节点的连通性。
• OAuth2-proxy日志调试：添加环境变量OAUTH2_PROXY_LOG_LEVEL=debug，开启DEBUG级日志，查看认证流程中是否存在与Keycloak通信失败、授权文件读取异常等问题。
• 节点标签验证：确认新服务器已正确设置web==web的节点标签，否则OAuth2-proxy容器不会部署到该节点，导致Traefik找不到后端服务超时。
• Cookie配置一致性：检查新服务器上的SECRET_COOKIE变量是否与其他节点完全一致，不一致会导致Cookie验证失败。
• Keycloak可达性：在OAuth2-proxy容器内curl Keycloak的issuer URL，确认新节点能正常访问Keycloak服务，排查防火墙规则或DNS解析问题。

内容的提问来源于stack exchange，提问作者tandathuynh148