Docker Swarm部署Apache Flink时TaskManager无法注册至JobManager

问题描述 #

在两台Google Cloud实例上通过Docker Swarm部署Apache Flink 1.16.1，JobManager部署在管理节点，TaskManager部署在工作节点，TaskManager无法通过akka.tcp与JobManager上的ResourceManager建立通信，最终触发注册超时错误。两台实例处于同一VPC和子网，可互相ping通、curl访问，防火墙无阻碍规则。此前在私有Microstack云环境中使用相同部署配置可正常运行。

错误日志 #

flink_taskmanager.1.e7sxy43lsb49@workernode.gcp    | 2023-03-29 13:50:34,061 INFO  org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Start job leader service.
flink_taskmanager.1.e7sxy43lsb49@workernode.gcp    | 2023-03-29 13:50:34,066 INFO  org.apache.flink.runtime.filecache.FileCache                 [] - User file cache uses directory /tmp/flink-dist-cache-705fdca5-c285-4875-9b68-556ccd1b56c3
flink_taskmanager.1.e7sxy43lsb49@workernode.gcp    | 2023-03-29 13:50:34,073 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Connecting to ResourceManager akka.tcp://flink@jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000).
flink_taskmanager.1.e7sxy43lsb49@workernode.gcp    | 2023-03-29 13:50:34,420 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Resolved ResourceManager address, beginning registration
flink_taskmanager.1.e7sxy43lsb49@workernode.gcp    | 2023-03-29 13:55:34,086 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Fatal error occurred in TaskExecutor akka.tcp://flink@10.0.1.14:6127/user/rpc/taskmanager_0.
flink_taskmanager.1.e7sxy43lsb49@workernode.gcp    | org.apache.flink.runtime.taskexecutor.exceptions.RegistrationTimeoutException: Could not register at the ResourceManager within the specified maximum registration duration PT5M. This indicates a problem with this instance. Terminating now.

部署配置 #

Docker Compose文件（通过docker stack deploy部署） #

version: '3.8'

services:
  jobmanager:
    image: halo93/fixed-ports-flink-docker:1.16.1-scala_2.12-java11-custom
    deploy:
      replicas: 1
      placement:
        constraints: [node.hostname == managernode.gcp]
    ports:
      - "8081:8081"
      - "6123:6123"
      - "6124:6124"
      - "6125:6125"
    command: jobmanager
    environment:
      - FLINK_PROPERTIES=${FLINK_PROPERTIES}
    networks:
      - flink-network
  taskmanager:
    image: halo93/fixed-ports-flink-docker:1.16.1-scala_2.12-java11-custom
    deploy:
      replicas: 1
      placement:
        constraints: [node.hostname == workernode.gcp]
    depends_on:
      - jobmanager
    ports:
      - "6121:6121"
      - "6122:6122"
      - "6126:6126"
      - "6127:6127"
      - "6128:6128"
      - "5005:5005/udp"
    command:
      - taskmanager
    environment:
      - FLINK_PROPERTIES=${FLINK_PROPERTIES}
    networks:
      - flink-network

networks:
  flink-network:
    driver: overlay
    attachable: true

FLINK_PROPERTIES配置 #

FLINK_PROPERTIES=$'\njobmanager.rpc.address: jobmanager\nparallelism.default: 2\n'

额外说明与诉求 #

• 使用自定义Flink镜像固定了taskmanager.data.port=6126、taskmanager.rpc.port=6127
• 尝试将jobmanager.rpc.address改为私有IP或区域DNS时，TaskManager可成功注册，但flink-metrics功能失效
• 期望在Google Cloud环境下实现正常部署，解决TaskManager注册超时及metrics失效的问题

内容的提问来源于stack exchange，提问作者halo93