如何连接不同主机上的Docker容器与Swarm集群中的malamute服务

问题描述 #

我有大量待处理任务需要分配到多台主机执行，因为熟悉且偏好zmq，当前选择zmq的malamute实现，也欢迎推荐其他适合向工作节点分发任务的库。

我正在开发需要以全局模式部署的Docker服务，开发阶段需要将作为工作节点的容器连接到Swarm管理节点上的malamute broker，同时需要交互式输入命令编写代码，因此使用了挂载共享overlay网络的容器。

# 管理节点（主机1）操作
docker service create --replicas 0 --name malamute zeromqorg/malamute
#docker service update --publish-add 9999 malamute
docker service update --publish-add published=9999,target=9999 malamute
docker network create --attachable --driver overlay primarynet
docker service update --network-add primarynet malamute

docker inspect primarynet输出如下：

[
    {
        "Name": "primarynet",
        "Id": "b7vq0p0purgiykebtykdn7syh",
        "Created": "2021-11-08T13:34:08.600557322-05:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.2.0/24",
                    "Gateway": "10.0.2.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "f50adb23eebcac36e4c1c5816d140da469df1e1ee72ae2c7bada832ba69e1535": {
                "Name": "malamute.1.e29sipw9i0rijvldeqbmh4r9r",
                "EndpointID": "2fe7789b4c50eeca7d19007e447aa32a3bb8d6a33435051d84ba04e34f206446",
                "MacAddress": "02:42:0a:00:02:0f",
                "IPv4Address": "10.0.2.15/24",
                "IPv6Address": ""
            },
            "lb-primarynet": {
                "Name": "primarynet-endpoint",
                "EndpointID": "9b3926bcfea39d77a48ac4fcc1c5df37c1dd5e7add6b790bc9cbc549a5bea589",
                "MacAddress": "02:42:0a:00:02:04",
                "IPv4Address": "10.0.2.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "582d161489c7",
                "IP": "192.168.1.106"
            },
            {
                "Name": "c71e09e3cd2c",
                "IP": "192.168.1.107"
            }
        ]
    }
]

docker inspect ingress输出如下：

[
    {
        "Name": "ingress",
        "Id": "od7bn815iuxyq4v9jzwe17n4p",
        "Created": "2021-11-08T10:59:00.304709057-05:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "f50adb23eebcac36e4c1c5816d140da469df1e1ee72ae2c7bada832ba69e1535": {
                "Name": "malamute.1.e29sipw9i0rijvldeqbmh4r9r",
                "EndpointID": "bb87dbf390d23c8992c4f2b27597af80bb8d3b96b3bd62aa58618cca82cc0426",
                "MacAddress": "02:42:0a:00:00:0a",
                "IPv4Address": "10.0.0.10/24",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "2c1ef914e0aa506756b779ece181c7d8efb8697b71cb5dce0db1d9426137660f",
                "MacAddress": "02:42:0a:00:00:02",
                "IPv4Address": "10.0.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "582d161489c7",
                "IP": "192.168.1.106"
            },
            {
                "Name": "c71e09e3cd2c",
                "IP": "192.168.1.107"
            },
            {
                "Name": "c67bb64801c0",
                "IP": "192.168.1.143"
            }
        ]
    }
]

错误发生环节 #

# 工作节点（主机2）操作
docker swarm join --token <token> 192.168.1.106:2377
docker run --net primarynet -ti zeromqorg/malamute  /bin/bash
# 容器内操作
export PYTHONPATH=/home/zmq/malamute/bindings/python/:/home/zmq/czmq/bindings/python
python3
>>> from malamute import MalamuteClient
>>> client=MalamuteClient()
>>> client.connect(b'tcp://manager.addr.ip.addr:9999', 100, b'service')
# 此处连接失败，预期可正常连通
# 错误输出如下：
I: 21-11-10 07:12:59 My address is 'service'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/zmq/malamute/bindings/python/malamute/__init__.py", line 51, in connect
    "Could not connect to malamute server at {!r}", endpoint,
  File "/home/zmq/malamute/bindings/python/malamute/__init__.py", line 41, in _check_error
    fmt.format(*args, **kw) + ': ' + str(reason)
malamute.MalamuteError: Could not connect to malamute server at <addr>: Server is not reachable

工作节点容器内可正常ping通malamute服务的容器IP：

zmq@8e2256f6a806:~/malamute$ ping 10.0.2.15
PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
64 bytes from 10.0.2.15: icmp_seq=1 ttl=64 time=0.646 ms
64 bytes from 10.0.2.15: icmp_seq=2 ttl=64 time=0.399 ms
64 bytes from 10.0.2.15: icmp_seq=3 ttl=64 time=0.398 ms
64 bytes from 10.0.2.15: icmp_seq=4 ttl=64 time=0.401 ms
^C
--- 10.0.2.15 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3079ms
rtt min/avg/max/mdev = 0.398/0.461/0.646/0.106 ms

工作节点和管理节点的主机名也可正常ping通，输出与上述结果类似；但如果在工作节点容器内ping Swarm管理节点上运行的服务的Docker ID，则无法连通。

我查阅相关指南后仍未定位到连接报错的原因，希望解决该连接问题，使connect调用正常执行以完成后续脚本开发，最终将逻辑封装为名为worker的Docker服务。

解决方案 #

1. 优先使用Swarm内置服务名通信：你的工作容器和malamute服务都在同一个attachable overlay网络primarynet下，不需要走管理节点对外暴露的端口，直接将连接地址改为b'tcp://malamute:9999'即可，Swarm内置的DNS会自动将服务名解析为对应服务的虚拟IP，避免手动填写IP带来的路由问题。
2. 验证端口连通性：如果替换服务名后仍报错，先在工作节点容器内执行nc -zv 10.0.2.15 9999验证端口是否可达：
   • 如果端口连通，说明是malamute服务端配置问题，可进入malamute容器确认服务是否正常监听0.0.0.0:9999，而非本地回环地址。
   • 如果端口不通，检查两个节点的防火墙规则，放开Swarm集群所需的端口：2377（集群管理）、7946（节点通信）、4789（vxlan overlay网络通信）。
3. Overlay网络加密配置：跨节点overlay网络通信异常时，可重建primarynet网络添加加密参数：docker network create --attachable --driver overlay --opt encrypted primarynet，避免底层网络拦截加密前的业务流量。
4. 后续worker服务封装：正式打包为worker服务时，直接在docker service create时指定--network primarynet，不需要额外配置端口映射，服务之间可直接通过服务名互相调用。

内容的提问来源于stack exchange，提问作者Adam Miller