跨服务器Concourse Worker无法连接Concourse Web排查求助

环境背景 #

• 服务器A（IP: 212.77.7.255）运行Concourse Web + Worker容器，版本7.8.1
• 服务器B新增Concourse Worker容器，正常运行5天后突然无法连接服务器A的Web节点

Worker日志报错（服务器B） #

{
    "timestamp": "2022-07-12T11:15:59.542985762Z",
    "level": "error",
    "source": "worker",
    "message": "worker.container-sweeper.tick.failed-to-connect-to-tsa",
    "data": {
        "error": "dial tcp 212.77.7.255:2222: i/o timeout",
        "session": "6.4"
    }
}
{
    "timestamp": "2022-07-12T11:15:59.543044656Z",
    "level": "error",
    "source": "worker",
    "message": "worker.container-sweeper.tick.dial.failed-to-connect-to-any-tsa",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "6.4.2"
    }
}
{
    "timestamp": "2022-07-12T11:15:59.543060804Z",
    "level": "error",
    "source": "worker",
    "message": "worker.container-sweeper.tick.failed-to-dial",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "6.4"
    }
}
{
    "timestamp": "2022-07-12T11:15:59.543068953Z",
    "level": "error",
    "source": "worker",
    "message": "worker.container-sweeper.tick.failed-to-get-containers-to-destroy",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "6.4"
    }
}
{
    "timestamp": "2022-07-12T11:15:59.554118751Z",
    "level": "error",
    "source": "worker",
    "message": "worker.volume-sweeper.tick.failed-to-connect-to-tsa",
    "data": {
        "error": "dial tcp 212.77.7.255:2222: i/o timeout",
        "session": "7.4"
    }
}
{
    "timestamp": "2022-07-12T11:15:59.554164844Z",
    "level": "error",
    "source": "worker",
    "message": "worker.volume-sweeper.tick.dial.failed-to-connect-to-any-tsa",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "7.4.3"
    }
}
{
    "timestamp": "2022-07-12T11:15:59.554172593Z",
    "level": "error",
    "source": "worker",
    "message": "worker.volume-sweeper.tick.failed-to-dial",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "7.4"
    }
}
{
    "timestamp": "2022-07-12T11:15:59.554179789Z",
    "level": "error",
    "source": "worker",
    "message": "worker.volume-sweeper.tick.failed-to-get-volumes-to-destroy",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "7.4"
    }
}
{
    "timestamp": "2022-07-12T11:16:04.580220012Z",
    "level": "error",
    "source": "worker",
    "message": "worker.beacon-runner.beacon.failed-to-connect-to-tsa",
    "data": {
        "error": "dial tcp 212.77.7.255:2222: i/o timeout",
        "session": "4.1"
    }
}
{
    "timestamp": "2022-07-12T11:16:04.580284659Z",
    "level": "error",
    "source": "worker",
    "message": "worker.beacon-runner.beacon.dial.failed-to-connect-to-any-tsa",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "4.1.10"
    }
}
{
    "timestamp": "2022-07-12T11:16:04.580335377Z",
    "level": "error",
    "source": "worker",
    "message": "worker.beacon-runner.beacon.failed-to-dial",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "4.1"
    }
}
{
    "timestamp": "2022-07-12T11:16:04.580359868Z",
    "level": "error",
    "source": "worker",
    "message": "worker.beacon-runner.beacon.exited-with-error",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "4.1"
    }
}
{
    "timestamp": "2022-07-12T11:16:04.580372552Z",
    "level": "debug",
    "source": "worker",
    "message": "worker.beacon-runner.beacon.done",
    "data": {
        "session": "4.1"
    }
}
{
    "timestamp": "2022-07-12T11:16:04.580394879Z",
    "level": "error",
    "source": "worker",
    "message": "worker.beacon-runner.failed",
    "data": {
        "error": "all worker SSH gateways unreachable",
        "session": "4"
    }
}

已验证现象 #

• 服务器A的Web日志无服务器B Worker的连接记录
• 服务器B执行nc 212.77.7.255 2222可收到响应：SSH-2.0-Go

已尝试无效操作 #

• 重启Worker容器
• 重启Web容器
• 清理服务器B上的停滞Worker
• 执行docker system prune

进一步调试与修复方案 #

1. 网络层深度排查 #

• 在服务器B的Worker容器内部执行nc 212.77.7.255 2222，确认容器内网络是否正常（排除宿主机网络正常但容器网络异常的情况）
• 在服务器B上用tcpdump抓包分析2222端口流量：

  tcpdump -i any host 212.77.7.255 and port 2222 -w concourse-tsa.pcap
  

  查看是否存在SYN包发送后无ACK响应、数据包丢失等异常
• 检查服务器A和B之间的防火墙/安全组规则，确认2222端口双向流量未被临时拦截（部分安全策略会触发流量阈值封禁）

2. TSA服务状态验证 #

• 在服务器A上检查Web容器内TSA服务监听状态：

  docker exec <web-container-id> netstat -tulpn | grep 2222
  

• 启用Web节点debug级别日志（添加启动参数--log-level debug），查看TSA服务的连接处理细节
• 验证Worker启动时指定的--tsa-public-key与Web节点的tsa_host_key.pub是否完全一致，避免密钥不匹配

3. Worker状态与配置检查 #

• 检查服务器B的Worker容器资源使用情况：

  docker stats <worker-container-id>
  

  确认CPU、内存、磁盘IO是否异常，资源耗尽可能导致网络请求超时
• 在服务器A上重新生成Worker注册令牌，用新令牌重启服务器B的Worker：

  # 清理旧Worker注册信息
  docker exec <web-container-id> fly -t local prune-worker -w <worker-name>
  # 生成新注册命令
  docker exec <web-container-id> fly -t local worker register --name <new-worker-name> --host <worker-ip> --public-key <worker-public-key>
  

• 检查Worker容器与Web节点的系统时间是否同步，时间偏差过大可能导致SSH握手失败

4. 容器网络配置排查 #

• 检查服务器B的Docker自定义网络状态：

  docker network inspect <network-name>
  

• 尝试将Worker容器切换为host网络模式启动，排除容器网络NAT或端口映射问题
• 检查服务器B的DNS配置，确认Worker能正确解析Web节点IP（部分组件依赖DNS验证逻辑）

内容的提问来源于stack exchange，提问作者mles