Ubuntu 20单节点Docker Swarm跨Stack容器服务解析失败排查
Swarm单节点集群跨Stack通信DNS失效问题排查指南
问题场景
在Ubuntu 20环境搭建完成单节点Swarm集群,共部署3个Stack,所有服务默认使用overlay网络。按照Swarm overlay网络的官方设计规则,同集群内容器无需额外特殊配置即可互相通信,但实际运行中出现跨Stack容器无法完成服务名DNS解析的问题,同一Stack内的不同容器通信完全正常。从IP段分配规则来看,跨栈容器本应具备基础通信条件,需要明确这类问题的标准排查流程与解决方法。
基础环境配置
# 系统版本 uname -a Linux ubuntu20 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux # Stack列表 docker stack ls NAME SERVICES ORCHESTRATOR management 1 Swarm tech 3 Swarm web 1 Swarm # 服务列表 docker service ls ID NAME MODE REPLICAS IMAGE PORTS cla322jwbgm3 management_portainer-ce replicated 1/1 portainer/portainer-ce:latest *:8000->8000/tcp, *:9000->9000/tcp qlts0otlc4c8 tech_grafana replicated 1/1 grafana/grafana:latest *:3000->3000/tcp yujqvwim2okb tech_influxdb replicated 1/1 influxdb:latest *:8086->8086/tcp v14haj0oa8dc tech_mosquitto replicated 1/1 eclipse-mosquitto:latest *:1883->1883/tcp 3dra8oyo7swo web_nginx replicated 1/1 nginx:latest *:80->80/tcp # 运行中容器列表 docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9025b777f04c portainer/portainer-ce:latest "/portainer" 29 minutes ago Up 29 minutes 8000/tcp, 9000/tcp management_portainer-ce.1.w910sug8vaormu1pgs62wrj2j 918705a22e73 influxdb:latest "/entrypoint.sh infl…" 29 minutes ago Up 29 minutes 8086/tcp tech_influxdb.1.4d47iflbvoybnzsl1lquf89ej 4911abc53cf0 grafana/grafana:latest "/run.sh" 29 minutes ago Up 29 minutes 3000/tcp tech_grafana.1.oyoot8zkxl92yi05s1ac7nano a00639d4aa76 eclipse-mosquitto:latest "/docker-entrypoint.…" 29 minutes ago Up 29 minutes 1883/tcp tech_mosquitto.1.toob1isapbmy0h0i4wa97uqq6 ca60cb673379 nginx:latest "/docker-entrypoint.…" 30 minutes ago Up 30 minutes 80/tcp web_nginx.1.mgyk64ggouqi9pyzgkyknmuk4
网络配置现状
# 自定义创建ingress网络 docker network create --driver overlay --ingress --subnet=10.11.0.0/16 --gateway=10.11.0.1 ingress # 查看ingress网络详情 docker inspect network ingress [ { "Name": "ingress", "Id": "9ykfwv0ahhk7a9s8g8a2nh53q", "Created": "2021-02-07T13:52:09.317600628-05:00", "Scope": "swarm", "Driver": "overlay", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "10.11.0.0/16", "Gateway": "10.11.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": true, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "4911abc53cf0021e0fbad4392a1666e17d500167589460a77bdbd73bc52a75a1": { "Name": "tech_grafana.1.oyoot8zkxl92yi05s1ac7nano", "EndpointID": "c9d609ffe49a7aad7be596f6f36bf9b494af109a4c530fc8b5393d0ff3c16753", "MacAddress": "02:42:0a:0b:00:08", "IPv4Address": "10.11.0.8/16", "IPv6Address": "" }, "9025b777f04c86742b87308374cda51b11530556040faad623dd5db51f246de4": { "Name": "management_portainer-ce.1.w910sug8vaormu1pgs62wrj2j", "EndpointID": "bfd0613488921f39d69f4dc4e45e1f2b1b80b7c41a411040f1c2f7518c06f3c6", "MacAddress": "02:42:0a:0b:00:0c", "IPv4Address": "10.11.0.12/16", "IPv6Address": "" }, "918705a22e73c0f6ea81b063bd95db964d74cbb0f99b5fe19bbfb0cb352b0bfc": { "Name": "tech_influxdb.1.4d47iflbvoybnzsl1lquf89ej", "EndpointID": "07a5cac92d0c5fc53e93bf6c049f7b8977dd31c1a2d6fc27a033f1769370c53b", "MacAddress": "02:42:0a:0b:00:0a", "IPv4Address": "10.11.0.10/16", "IPv6Address": "" }, "a00639d4aa76485b305b41e4052fe955eef4ee2c0f0e90722097d6a7995ca9bc": { "Name": "tech_mosquitto.1.toob1isapbmy0h0i4wa97uqq6", "EndpointID": "080d882f2cc09a011261a11c6f46d8c04f26a2329d00f5fa530133bf67a4d8c0", "MacAddress": "02:42:0a:0b:00:06", "IPv4Address": "10.11.0.6/16", "IPv6Address": "" }, "ca60cb6733797cd6867fb7146217f92ad18d1f759c86dbb65f0804199129651a": { "Name": "web_nginx.1.mgyk64ggouqi9pyzgkyknmuk4", "EndpointID": "019506a9da152f923600580121c373e7a74ef99cd05bf3c5094ab92dfe39a472", "MacAddress": "02:42:0a:0b:00:03", "IPv4Address": "10.11.0.3/16", "IPv6Address": "" }, "ingress-sbox": { "Name": "ingress-endpoint", "EndpointID": "b89f39cca8cfc6553385247315adaf543900a4e56ca769c2989e6ad63baa0ebc", "MacAddress": "02:42:0a:0b:00:04", "IPv4Address": "10.11.0.4/16", "IPv6Address": "" } }, "Options": { "com.docker.network.driver.overlay.vxlanid_list": "4104" }, "Labels": {}, "Peers": [ { "Name": "933a5c2c1e76", "IP": "192.168.1.103" } ] } ]
连通性测试结果
同一Stack内容器互通正常,但跨Stack的服务名解析全部失败,具体测试过程如下:
# 从tech stack的influxdb容器发起测试 docker exec -it 918705a22e73 bash root@918705a22e73:/# ping nginx ping: nginx: Name or service not known root@918705a22e73:/# ping grafana PING grafana (10.0.9.5) 56(84) bytes of data. 64 bytes from 10.0.9.5 (10.0.9.5): icmp_seq=1 ttl=64 time=0.099 ms ^C --- grafana ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.099/0.099/0.099/0.000 ms # 从web stack的nginx容器发起测试 root@ca60cb673379:/# curl grafana curl: (6) Could not resolve host: grafana root@ca60cb673379:/# curl nginx <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> root@ca60cb673379:/# curl influxdb curl: (6) Could not resolve host: influxdb root@ca60cb673379:/#
问题根因与排查逻辑
- Swarm的overlay网络DNS解析范围严格以网络为边界:只有接入完全相同网络的服务,才能通过服务名互相解析,跨网络的服务名不会被内置DNS暴露
- 每个Stack部署时,Swarm会自动创建一个仅当前Stack可访问的专属overlay网络(命名格式为
<stack名>_default),同栈服务自动接入这个网络,因此同栈内DNS解析完全正常 - 当前所有服务因为配置了端口发布,会被自动接入ingress网络,但ingress的设计目标是承载外部入口流量的负载均衡,默认
Attachable属性为false,不会在网络上注册跨Stack的服务名DNS记录,这是Swarm的默认安全隔离机制,避免不同Stack的服务被随意跨域访问 - 从测试结果可验证:同栈服务解析到的IP为10.0.9.5,不属于ingress网络的10.11.0.0/16网段,说明不同Stack的服务分别接入了互相隔离的专属网络,因此跨栈无法解析
标准解决步骤
- 第一步:创建全局共享的自定义overlay网络,该网络不绑定任何Stack,所有需要跨栈通信的服务都可接入
docker network create --driver overlay swarm_shared - 第二步:修改各Stack的compose部署文件,将需要跨栈通信的服务配置为同时接入上述共享网络,配置示例:
services: # 对应web Stack的nginx服务 nginx: image: nginx:latest networks: - swarm_shared # 对应tech Stack的grafana服务 grafana: image: grafana/grafana:latest networks: - swarm_shared networks: # 声明外部已存在的共享网络,避免重复创建 swarm_shared: external: true - 第三步:重新部署所有涉及跨栈通信的Stack,确保服务接入共享网络
- 第四步:验证连通性:跨栈容器内直接ping/调用对端服务名,确认DNS解析正常
内容的提问来源于stack exchange,提问作者Adarsha
相关产品推荐
相关产品推荐

