You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

批量下线BE导致Master FE指标采集阻塞问题排查

avatar
API 探路者
2026-6-12
Doris 2.1.7 集群批量下线BE后8030端口部分接口阻塞问题

问题背景

Doris版本为2.1.7，集群原包含25台BE，每台BE约有20000个tablets。批量下线5台BE后，出现以下异常：

  • 访问http://fe_ip:8030/metrics接口阻塞，监控页面相关指标消失
  • 9030、8010端口未受影响，8030 UI页面可正常登录
  • system -> transactions页面可正常访问，但system -> cluster_balance页面访问阻塞

JVM栈信息 

"thrift-server-pool-2330" #27579 daemon prio=5 os_prio=0 tid=0x00007f8ef921d800 nid=0xf727 waiting on condition [0x00007f88ba749000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000003e2481b00> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
    at org.apache.doris.common.lock.MonitoredReentrantReadWriteLock$WriteLock.lock(MonitoredReentrantReadWriteLock.java:98)
    at org.apache.doris.catalog.Table.writeLockIfExist(Table.java:211)
    at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:940)
    at org.apache.doris.clone.TabletSchedCtx.releaseResource(TabletSchedCtx.java:898)
    at org.apache.doris.clone.TabletScheduler.releaseTabletCtx(TabletScheduler.java:1743)
    at org.apache.doris.clone.TabletScheduler.finalizeTabletCtx(TabletScheduler.java:1625)
    at org.apache.doris.clone.TabletScheduler.addTablet(TabletScheduler.java:287)
    - locked <0x00000003c0b7da20> (a org.apache.doris.clone.TabletScheduler)
    at org.apache.doris.clone.TabletScheduler.tryAddRepairTablet(TabletScheduler.java:1738)
    at org.apache.doris.clone.TabletScheduler.tryAddAfterFinished(TabletScheduler.java:1666)
    at org.apache.doris.clone.TabletScheduler.finalizeTabletCtx(TabletScheduler.java:1629)
    at org.apache.doris.clone.TabletScheduler.finishCloneTask(TabletScheduler.java:1881)
    at org.apache.doris.master.MasterImpl.finishClone(MasterImpl.java:551)
    at org.apache.doris.master.MasterImpl.finishTask(MasterImpl.java:179)
    at org.apache.doris.service.FrontendServiceImpl.finishTask(FrontendServiceImpl.java:991)
    at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60)
    at org.apache.doris.service.FeServer$$Lambda$213/1237024911.invoke(Unknown Source)
    at com.sun.proxy.$Proxy41.finishTask(Unknown Source)
    at org.apache.doris.thrift.FrontendService$Processor$finishTask.getResult(FrontendService.java:3717)
    at org.apache.doris.thrift.FrontendService$Processor$finishTask.getResult(FrontendService.java:3697)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)

   Locked ownable synchronizers:
    - <0x00000003cb1c7d60> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"qtp2026237784-22068" #22068 prio=5 os_prio=0 tid=0x00007f8e2803a000 nid=0xd569 waiting for monitor entry [0x00007f8b8afa8000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.doris.metric.MetricRepo.getMetric(MetricRepo.java:662)
    - waiting to lock <0x00000003c04d0198> (a java.lang.Class for org.apache.doris.metric.MetricRepo)
    at org.apache.doris.httpv2.rest.MetricsAction.execute(MetricsAction.java:61)
    at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:903)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
    at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:292)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
    at org.apache.doris.httpv2.interceptor.ServletTraceIterceptor.doFilter(ServletTraceIterceptor.java:55)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    at org.eclipse.jetty.server.HttpChannel$$Lambda$1932/133417100.dispatch(Unknown Source)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.lang.Thread.run(Thread.java:750)

   Locked ownable synchronizers:
    - None

问题分析

1. Thrift线程池锁等待

thrift-server-pool-2330线程处于WAITING状态，正在等待Table对象的写锁。BE下线后，集群触发大量分片修复任务，在完成分片克隆后释放调度上下文时，需要获取Table的写锁来更新分片状态，大量此类请求导致锁竞争，占用Thrift线程池资源。

2. Metrics接口类锁阻塞

qtp2026237784-22068线程处于BLOCKED状态，等待MetricRepo的类锁。/metrics接口和cluster_balance页面都依赖MetricRepo获取集群状态指标，而该类锁被其他长时间运行的线程持有，导致接口阻塞。

解决方案

临时缓解

  • 暂停部分修复任务：执行ADMIN CANCEL REPAIR TABLE <table_name>;，优先暂停大表的修复，降低锁竞争
  • 调整FE线程池参数：修改fe.conf中的thrift_server_threads（默认100）和jetty_threads（默认200），增大线程池容量，重启FE生效
  • 手动推进平衡：执行ADMIN SET FRONTEND CONFIG ("disable_balance" = "false");，确保平衡任务正常执行，逐步释放锁资源

长期优化

  • 分批下线BE：避免一次性下线过多节点，建议每次下线1-2台，待集群平衡完成后再继续操作
  • 调优平衡参数：降低balance_max_batches和balance_batch_size，减少单次调度的分片数量，降低锁竞争压力
  • 升级Doris版本：后续版本针对分片修复和锁竞争场景有优化，建议升级到2.1.x稳定版或更高版本

内容的提问来源于stack exchange，提问作者Chris

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.06.12 22:54:50