Docker Swarm部署Milvus集群时milvus-proxy端口不可用问题

问题 #

在本地集群通过Docker Compose结合Docker Swarm部署Milvus时，所有服务均已启动，但milvus-proxy的端口映射未在docker ps输出中显示（minio和etcd的端口映射正常）。milvus-proxy日志显示已成功监听19530端口，但外部无法访问。

操作步骤 #

1. 初始化Swarm集群：

docker swarm init --advertise-addr <master-ip>
docker swarm join --token <token> <master-ip>

2. 使用的Docker Compose配置文件：

services:
  etcd:
    image: quay.io/coreos/etcd:v3.5.5
    container_name: etcd
    environment:
      - ETCD_AUTO_COMPACTION_RETENTION=1
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ALLOW_NONE_AUTHENTICATION=yes
      - ETCDCTL_API=3
    ports:
      - "2379:2379"
      - "2380:2380"
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  minio:
    image: minio/minio
    container_name: minio
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    command: minio server /minio_data
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    ports:
      - "9000:9000"
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  milvus-proxy:
    image: milvusdb/milvus:v2.2.10
    container_name: milvus-proxy
    command: ["milvus", "run", "proxy"]
    environment:
      ETCD_ENDPOINTS: <master_node_ip>:2379
      MINIO_ADDRESS: <master_node_ip>:9000
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
      CUDA_VISIBLE_DEVICES: 0
    ports:
      - "<master_node_ip>:19530:19530"
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  milvus-rootcoord:
    image: milvusdb/milvus:v2.2.10
    container_name: milvus-rootcoord
    command: ["milvus", "run", "rootcoord"]
    environment:
      ETCD_ENDPOINTS: <master_node_ip>:2379
      MINIO_ADDRESS: <master_node_ip>:9000
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
      CUDA_VISIBLE_DEVICES: 0
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  milvus-datacoord:
    image: milvusdb/milvus:v2.2.10
    container_name: milvus-datacoord
    command: ["milvus", "run", "datacoord"]
    environment:
      ETCD_ENDPOINTS: <master_node_ip>:2379
      MINIO_ADDRESS: <master_node_ip>:9000
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
      CUDA_VISIBLE_DEVICES: 0
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  milvus-indexcoord:
    image: milvusdb/milvus:v2.2.10
    container_name: milvus-indexcoord
    command: ["milvus", "run", "indexcoord"]
    environment:
      ETCD_ENDPOINTS: <master_node_ip>:2379
      MINIO_ADDRESS: <master_node_ip>:9000
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
      CUDA_VISIBLE_DEVICES: 0
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  milvus-querycoord:
    image: milvusdb/milvus:v2.2.10
    container_name: milvus-querycoord
    command: ["milvus", "run", "querycoord"]
    environment:
      ETCD_ENDPOINTS: <master_node_ip>:2379
      MINIO_ADDRESS: <master_node_ip>:9000
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
      CUDA_VISIBLE_DEVICES: 0
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  milvus-datanode-1:
    image: milvusdb/milvus:v2.2.10
    container_name: milvus-datanode-1
    command: ["milvus", "run", "datanode"]
    environment:
      ETCD_ENDPOINTS: <master_node_ip>:2379
      MINIO_ADDRESS: <master_node_ip>:9000
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
      CUDA_VISIBLE_DEVICES: 0
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  milvus-datanode-2:
     --same as milvus-datanode-1 config --
        constraints: [node.hostname == worker_node]

  milvus-indexnode-1:
    image: milvusdb/milvus:v2.2.10
    container_name: milvus-indexnode-1
    command: ["milvus", "run", "indexnode"]
    environment:
      ETCD_ENDPOINTS: <master_node_ip>:2379
      MINIO_ADDRESS: <master_node_ip>:9000
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
      CUDA_VISIBLE_DEVICES: 0
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  milvus-indexnode-2:
    --same as milvus-indexnode-1 config --
    deploy:
      placement:
        constraints: [node.hostname == worker_node]

  milvus-querynode-1:
    image: milvusdb/milvus:v2.2.10
    container_name: milvus-querynode-1
    command: ["milvus", "run", "querynode"]
    environment:
      ETCD_ENDPOINTS: <master_node_ip>:2379
      MINIO_ADDRESS: <master_node_ip>:9000
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
      CUDA_VISIBLE_DEVICES: 0
    deploy:
      placement:
        constraints: [node.hostname == master_node]

  milvus-querynode-2:
  --same as milvus-querynode-1 config --
    deploy:
      placement:
        constraints: [node.hostname == worker_node]

volumes:
  minio-data:

3. milvus-proxy日志片段：

["Proxy internal server listen on tcp"] [port=19529]
[2024/07/11 15:35:05.996 +00:00] [INFO] [proxy/service.go:263] ["Proxy internal server already listen on tcp"] [port=19529]
[2024/07/11 15:35:05.996 +00:00] [INFO] [proxy/service.go:281] ["create Proxy internal grpc server"] ["enforcement policy"="{\"MinTime\":5000000000,\"PermitWithoutStream\":true}"] ["server parameters"="{\"MaxConnectionIdle\":0,\"MaxConnectionAge\":0,\"MaxConnectionAgeGrace\":0,\"Time\":60000000000,\"Timeout\":10000000000}"]
[2024/07/11 15:35:05.996 +00:00] [INFO] [proxy/service.go:155] ["Proxy server listen on tcp"] [port=19530]
[2024/07/11 15:35:05.996 +00:00] [INFO] [proxy/service.go:162] ["Proxy server already listen on tcp"] [port=19530]
[2024/07/11 15:35:05.996 +00:00] [INFO] [proxy/service.go:170] ["Get proxy rate limiter done"] [port=19530]
[2024/07/11 15:35:05.996 +00:00] [INFO] [proxy/service.go:233] ["create Proxy grpc server"] ["enforcement policy"="{\"MinTime\":5000000000,\"PermitWithoutStream\":true}"] ["server parameters"="{\"MaxConnectionIdle\":0,\"MaxConnectionAge\":0,\"MaxConnectionAgeGrace\":0,\"Time\":60000000000,\"Timeout\":10000000000}"]
[2024/07/11 15:35:05.996 +00:00] [INFO] [proxy/service.go:363] ["register http server of proxy"]

原因分析与解决方案 #

原因1：Swarm模式下端口映射格式不兼容 #

Docker Swarm对端口映射中指定固定IP的格式（<master_node_ip>:19530:19530）支持存在问题，路由网格无法正确解析该配置，导致端口映射不生效。而etcd和minio使用的是端口:端口无IP指定格式，所以正常工作。

解决办法

修改milvus-proxy的ports配置为无IP绑定的通用格式：

ports:
  - "19530:19530"

如果需要限制仅在master节点暴露端口，可在deploy中添加endpoint_mode: host（该模式下端口只能在单个节点暴露，适合单节点部署proxy的场景）：

milvus-proxy:
  # 其他配置不变
  ports:
    - "19530:19530"
  deploy:
    placement:
      constraints: [node.hostname == master_node]
    endpoint_mode: host

原因2：错误使用docker ps查看Swarm服务端口 #

Swarm模式下，docker ps仅显示容器级别的信息，服务层面的端口映射需要通过Swarm专属命令查看，直接用docker ps无法看到路由网格管理的端口配置。

解决办法

使用以下命令查看milvus-proxy服务的端口状态：

# 查看所有服务端口概览
docker service ls
# 查看服务详细端口配置
docker service inspect milvus-proxy

只要服务状态正常且端口配置正确，即使docker ps看不到映射，外部也可通过Swarm节点IP访问19530端口（路由网格会自动转发到运行proxy的节点）。

原因3：容器内部监听地址限制 #

若Milvus Proxy监听的是localhost而非0.0.0.0，即使端口映射正确，外部也无法访问。虽然日志显示监听了tcp端口，但仍需确认监听地址配置。

解决办法

在milvus-proxy的环境变量中添加监听地址配置，确保服务绑定所有网卡：

milvus-proxy:
  environment:
    # 其他环境变量不变
    PROXY_LISTEN_ADDRESS: 0.0.0.0

验证步骤 #

1. 更新Compose文件后重新部署：

docker stack deploy -c docker-compose.yml milvus-stack

2. 等待服务更新完成，用docker service ls确认milvus-proxy的端口状态。
3. 测试端口连通性：

telnet <master-node-ip> 19530
# 或
nc -zv <master-node-ip> 19530

内容的提问来源于stack exchange，提问作者Brave