Apache Druid index_hadoop导入失败，Hadoop MR应用报OutOfMemoryError

AI 小北

2026-8-5

问题：Apache Druid index_hadoop导入任务因Yarn MRAppMaster内存溢出失败

使用Apache Druid通过index_hadoop类型导入数据时，索引任务执行失败，但Druid索引日志中无有效报错信息。进一步排查发现对应的MR应用（应用名称：OELandingType-determine_partitions_hashed-Optional.of([2022-12-25T00:00:00.000Z/2022-12-26T00:00:00.000Z]，其中OELandingType为Druid数据源名称）执行失败，Yarn日志报错如下：

2023-01-05 17:04:55,291 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1672909107121_0001 to jobTokenSecretManager
2023-01-05 17:04:55,349 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1672909107121_0001 because: too many reduces;
2023-01-05 17:04:55,397 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1672909107121_0001 = 115735. Number of splits = 1
2023-01-05 17:08:51,299 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.concurrent.locks.ReentrantReadWriteLock.<init>(ReentrantReadWriteLock.java:240)
    at java.util.concurrent.locks.ReentrantReadWriteLock.<init>(ReentrantReadWriteLock.java:230)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl.<init>(TaskImpl.java:304)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.ReduceTaskImpl.<init>(ReduceTaskImpl.java:47)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createReduceTasks(JobImpl.java:1569)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1494)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1414)
    at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
    at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
    at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
    at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1008)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:139)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1436)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1207)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1654)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1650)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1583)
2023-01-05 17:09:05,108 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
2023-01-05 17:09:45,419 ERROR [Thread-0] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Thread-0,5,main] threw an Throwable, but we are shutting down, so ignoring this
java.lang.OutOfMemoryError: GC overhead limit exceeded

从日志可见本次任务数据量极小，具体为：Input size for job job_1672909107121_0001 = 115735. Number of splits = 1