Docker Swarm中Go API无法从etcd获取gauth.rpc键，但etcdctl可查询到

场景描述 #

使用go-zero框架部署Docker Swarm应用栈，架构以RESTful API服务作为请求入口，API通过ETCD查询对应RPC服务的端点，再调用RPC服务实现业务逻辑。Swarm集群启动后，所有RPC服务和ETCD均处于健康状态，但API服务持续报错退出。

Docker Compose配置 #

version: '3.8'

services:
  g_api:
    image: <Host IP>:5000/g-api:1.0.5
    ports:
      - "8888:8888"
    environment:
      - TZ=Asia/Shanghai
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    depends_on:
      - g_etcd
      - g_authrpc
      - g_commentrpc
      - g_userrpc
      - g_msgrpc
      - g_postrpc
    networks:
      - gnet
  g_authrpc:
    image: <Host IP>:5000/g-authrpc:1.0.5
    ports:
      - "8080:8080"
    environment:
      - TZ=Asia/Shanghai
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    depends_on:
      - g_etcd
    networks:
      - gnet
  g_userrpc:
    image: <Host IP>:5000/g-userrpc:1.0.5
    ports:
      - "8081:8081"
    environment:
      - TZ=Asia/Shanghai
    command: "./main"
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    depends_on:
      - g_etcd
    networks:
      - gnet
  g_msgrpc:
    image: <Host IP>:5000/g-msgrpc:1.0.5
    ports:
      - "8086:8086"
    environment:
      - TZ=Asia/Shanghai
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    depends_on:
      - g_etcd
    networks:
      - gnet
  g_postrpc:
    image: <Host IP>:5000/g-postrpc:1.0.5
    ports:
      - "8082:8082"
    environment:
      - TZ=Asia/Shanghai
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    depends_on:
      - g_etcd
    networks:
      - gnet
  g_commentrpc:
    image: <Host IP>:5000/g-commentrpc:1.0.5
    ports:
      - "8152:8152"
    environment:
      - TZ=Asia/Shanghai
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    depends_on:
      - g_etcd
    networks:
      - gnet
  g_etcd:
    image: <Host IP>:5000/etcd:3.5
    ports:
      - "2379:2379"
    environment:
      - TZ=Asia/Shanghai
      - ALLOW_NONE_AUTHENTICATION=yes
      - ETCD_ADVERTISE_CLIENT_URLS=http://g_etcd:2379
    deploy:
      replicas: 1
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    networks:
      - gnet
networks:
  gnet:
    driver: overlay

注：<Host IP>为云服务器IP

API服务错误日志 #

global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    | 2024/07/01 03:05:57 rpc dial: etcd://141.2.18.177:2379/gauth.rpc, error: context deadline exceeded, make sure rpc service "gauth.rpc" is already started
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    | 
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    | goroutine 1 [running]:
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    | runtime/debug.Stack()
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    |   C:/Program Files/Go/src/runtime/debug/stack.go:24 +0x5e
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    | github.com/zeromicro/go-zero/core/logx.Must({0x20288e0?, 0xc00038a480?})
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    |   C:/Users/vipda/go/pkg/mod/github.com/zeromicro/go-zero@v1.6.0/core/logx/logs.go:225 +0x4b
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    | github.com/zeromicro/go-zero/zrpc.MustNewClient({{{0xc000403220, 0x1, 0x1}, {0xc000406810, 0x9}, 0x0, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}, ...)
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    |   C:/Users/vipda/go/pkg/mod/github.com/zeromicro/go-zero@v1.6.0/zrpc/client.go:45 +0x65
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    | globalfun/app/api/internal/svc.NewServiceContext({{{{0xc000406846, 0x3}, {{...}, {...}, {...}, {...}, {...}, {...}, 0x0, 0x0, ...}, ...}, ...}, ...})
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    |   C:/Users/person/projects/realworld/globalfun-go/globalfun/app/api/internal/svc/servicecontext.go:36 +0xc9
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    | main.main()
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    |   C:/Users/person/projects/realworld/globalfun-go/globalfun/app/api/app.go:25 +0x137
global_g_api.1.yv6gvbm93sq7@b3saglvue9qgqseg    | {"@timestamp":"2024-07-01T03:05:57.451Z","content":"rpc dial: etcd://141.2.18.177:2379/gauth.rpc, error: context deadline exceeded, make sure rpc service \"gauth.rpc\" is already started\n\ngoroutine 1 [running]:\nruntime/debug.Stack()\n\tC:/Program Files/Go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/zeromicro/go-zero/core/logx.Must({0x20288e0?, 0xc00038a480?})\n\tC:/Users/vipda/go/pkg/mod/github.com/zeromicro/go-zero@v1.6.0/core/logx/logs.go:225 +0x4b\ngithub.com/zeromicro/go-zero/zrpc.MustNewClient({{{0xc000403220, 0x1, 0x1}, {0xc000406810, 0x9}, 0x0, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}, ...)\n\tC:/Users/vipda/go/pkg/mod/github.com/zeromicro/go-zero@v1.6.0/zrpc/client.go:45 +0x65\nglobalfun/app/api/internal/svc.NewServiceContext({{{{0xc000406846, 0x3}, {{...}, {...}, {...}, {...}, {...}, {...}, 0x0, 0x0, ...}, ...}, ...}, ...})\n\tC:/Users/person/projects/realworld/globalfun-go/globalfun/app/api/internal/svc/servicecontext.go:36 +0xc9\nmain.main()\n\tC:/Users/person/projects/realworld/globalfun-go/globalfun/app/api/app.go:25 +0x137\n","level":"fatal"}

ETCD键查询结果 #

commentrpc.rpc/7587879719612179989
10.0.7.12:8152
commentrpc.rpc/7587879719612180006
10.0.7.13:8152
gauth.rpc/7587879719612179977
10.0.7.20:8080
gauth.rpc/7587879719612179981
10.0.7.21:8080
msg.rpc/7587879719612179993
10.0.7.7:8086
msg.rpc/7587879719612179997
10.0.7.6:8086
posts.rpc/7587879719612179973
10.0.7.9:8082
posts.rpc/7587879719612179985
10.0.7.10:8082
users.rpc/7587879719612180001
10.0.7.4:8081
users.rpc/7587879719612180011
10.0.0.3:8081

排查解决方案 #

1. 修正ETCD地址配置 #

API当前配置的ETCD地址是云服务器IP（141.2.18.177:2379），但在Docker Swarm的overlay网络中，容器应通过服务名g_etcd:2379访问ETCD服务。宿主机IP可能受防火墙规则、Swarm网络策略限制，导致容器内部无法连通。修改API的配置文件，将ETCD地址改为http://g_etcd:2379。

2. 验证RPC服务网络连通性 #

RPC服务注册到ETCD的是Swarm overlay网络的内部IP（如10.0.7.20:8080），进入API容器内部，执行以下命令验证网络连通性：

# ping RPC服务内部IP
ping 10.0.7.20
# telnet RPC端口
telnet 10.0.7.20 8080

如果无法连通，检查Swarm overlay网络gnet的配置，确保所有服务都加入了该网络，且无自定义网络策略限制流量。

3. 优化服务启动顺序 #

Docker Swarm的depends_on仅保证服务启动顺序，不保证服务完全就绪。API启动时，ETCD中可能还未完成RPC服务的注册，导致启动超时。可以：

• 在API的启动脚本中添加轮询逻辑，用etcdctl查询gauth.rpc前缀的键，直到检测到存在再启动API程序：

  while ! etcdctl --endpoints=http://g_etcd:2379 get --prefix gauth.rpc; do
    sleep 2
  done
  ./main
  

• 修改API的zrpc客户端配置，增加启动重试机制，避免因临时未就绪导致启动失败。

4. 延长ETCD客户端超时时间 #

go-zero的zrpc客户端默认超时时间可能过短，在Swarm网络环境中容易出现超时。在API的zrpc客户端配置中增加timeout参数，延长超时时间（例如设置为10s）：

# 示例zrpc客户端配置
GauthRpc:
  Etcd:
    Hosts:
    - g_etcd:2379
    Key: gauth.rpc
  Timeout: 10s

内容的提问来源于stack exchange，提问作者AchillesHeel