You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

Flink1.5作业运行一段时间后释放TaskManager异常原因咨询

avatar
架构老林
2026-8-30
问题描述

提交Flink程序后，作业可正常运行一段时间，之后抛出异常，客户端侧异常栈如下：

org.apache.flink.client.program.ProgramInvocationException: org.apache.flink.util.FlinkException: Releasing TaskManager 98fba59750780a0a7d400f5198636e3b.
    at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:464)
    at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
    at flink.KafkaFlink.main(KafkaFlink.java:140)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
    at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:781)
    at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:275)
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
    at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
    at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
    at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
Caused by: org.apache.flink.util.FlinkException: Releasing TaskManager 98fba59750780a0a7d400f5198636e3b.
    at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManagerInternal(SlotPool.java:1067)
    at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManager(SlotPool.java:1050)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
    at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
    at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

TaskManager侧报错信息如下：

2022-06-15 17:23:47,494 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerDetailsHandler  - Implementation error: Unhandled exception.
org.apache.flink.runtime.resourcemanager.exceptions.UnknownTaskExecutorException: No TaskExecutor registered under 7aa5950ae70a2e4aeef2c1240b3c3dca.
    at org.apache.flink.runtime.resourcemanager.ResourceManager.requestTaskManagerInfo(ResourceManager.java:550)
    at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
    at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)
    at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
    at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
    at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
    at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

当前使用的Flink版本为1.5，需定位触发该异常的根本原因。

根本原因

该异常是Flink 1.5版本RPC心跳容错机制缺陷叠加集群运行异常导致，异常传导逻辑如下：

  • 核心触发条件是TaskManager（TaskExecutor）与ResourceManager之间的RPC心跳超时。Flink 1.5默认Akka RPC心跳间隔为10s、akka.ask.timeout超时阈值默认10s，当集群出现Full GC长停顿、网络闪断丢包、节点CPU负载过高时，心跳包无法在超时窗口内送达，ResourceManager会直接将该TaskExecutor从注册列表中剔除，没有重试缓冲机制。
  • 剔除动作触发后，JobMaster的SlotPool组件会收到TaskManager释放通知，执行releaseTaskManagerInternal逻辑释放该节点上持有的所有计算slot，抛出Releasing TaskManager异常，运行中的作业因为依赖的slot被强制回收，直接失败抛出ProgramInvocationException。
  • TaskManager侧的UnknownTaskExecutorException属于连锁次生异常：TaskManager进程本身并未退出，当REST接口收到查询该TaskManager详情的请求时，ResourceManager已经删除了它的注册信息，无法查到对应记录才抛出该错误，不是故障根因。
  • Flink 1.5版本本身的心跳容错设计存在缺陷，只要单次心跳超时就直接剔除TaskExecutor，没有设计失败重试、超时自适应调整逻辑，该问题在1.6及之后版本才被修复。

常见的实际触发场景包括：

  • TaskManager节点出现长时间Full GC，停顿时间超过心跳超时阈值
  • 集群网络出现分区、高丢包率，导致RPC心跳包无法正常传输
  • TaskManager进程CPU、内存资源被同节点其他业务进程抢占，无法及时处理RPC请求
  • 配置的心跳超时阈值过小，无法适配业务数据高峰时的节点负载压力

内容的提问来源于stack exchange，提问作者user1112259

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.08.30 00:03:27