DruidDB index_kafka_histogram任务Java堆内存溢出问题求助

问题描述 #

我是DruidDB新手，在通过Kafka向DruidDB摄入数据时，初始阶段一切正常，但运行一段时间后，index_kafka_histogram任务出现**Java堆内存溢出（OutOfMemoryError: Java heap space）**错误。已尝试任务hard reset操作及Stack Overflow相关方案，仍未解决问题，特此求助。

相关配置片段 #

...
"metricsSpec": [
      {
        "type": "longMin",
        "name": "min",
        "fieldName": "min",
        "expression": null
      },
      {
        "type": "longMax",
        "name": "max",
        "fieldName": "max",
        "expression": null
      },
      {
        "type": "longSum",
        "name": "count",
        "fieldName": "count",
        "expression": null
      },
      {
        "type": "longSum",
        "name": "sum",
        "fieldName": "sum",
        "expression": null
      },
      {
        "type": "quantilesDoublesSketch",
        "name": "quantilesDoubleSketch",
        "fieldName": "sketch",
        "k": 128
      }
    ],
    "granularitySpec": {
      "type": "uniform",
      "segmentGranularity": "HOUR",
      "queryGranularity": "MINUTE",
      "rollup": true,
      "intervals": null
    },
...

"tuningConfig": {
    "type": "kafka",
    "maxRowsInMemory": 10000,
    "maxBytesInMemory": 200000,
    "maxRowsPerSegment": 5000000,
    "intermediatePersistPeriod": "PT10M",
    "basePersistDirectory": "/tmp/druid-realtime-persist15059426147899962275",
    "maxPendingPersists": 0,
    "indexSpec": {
      "bitmap": {
        "type": "roaring",
        "compressRunOnSerialization": true
      },
      "dimensionCompression": "lz4",
      "metricCompression": "lz4",
      "longEncoding": "longs"
    },
    "indexSpecForIntermediatePersists": {
      "bitmap": {
        "type": "roaring",
        "compressRunOnSerialization": true
      },
      "dimensionCompression": "lz4",
      "metricCompression": "lz4",
      "longEncoding": "longs"
    },
    "buildV9Directly": true,
    "reportParseExceptions": false,
    "handoffConditionTimeout": 0,
    "resetOffsetAutomatically": false,
    "chatRetries": 8,
    "httpTimeout": "PT10S",
    "shutdownTimeout": "PT80S",
    "offsetFetchPeriod": "PT30S",
    "intermediateHandoffPeriod": "P2147483647D",
    "logParseExceptions": true,
    "maxParseExceptions": 2147483647,
    "maxSavedParseExceptions": 0,
    "skipSequenceNumberAvailabilityCheck": false,
    "repartitionTransitionDuration": "PT120S"
  }
...

错误日志 #

09:55:49.134 [task-runner-0-priority-0] ERROR org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Uncaught Throwable while running task[AbstractTask{id='index_kafka_histogram_25c6328c09f15d7_nofamgdk', groupId='index_kafka_histogram', taskResource=TaskResource{availabilityGroup='index_kafka_histogram_25c6328c09f15d7', requiredCapacity=1}, dataSource='histogram', context={checkpoints={"0":{"0":0,"1":0,"2":0,"3":0,"4":0,"5":0,"6":0,"7":0,"8":0,"9":0,"10":0,"11":0,"12":0,"13":0,"14":0,"15":0,"16":0,"17":0,"18":0,"19":0,"20":0,"21":0,"22":0,"23":0,"24":0,"25":0,"26":0,"27":0,"28":0,"29":0,"30":0,"31":0,"32":0,"33":0,"34":0,"35":0,"36":0,"37":0,"38":0,"39":0,"40":0,"41":0,"42":0,"43":0,"44":0,"45":0,"46":0,"47":0,"48":0,"49":0}}, IS_INCREMENTAL_HANDOFF_SUPPORTED=true, forceTimeChunkLock=true}}]
java.lang.OutOfMemoryError: Java heap space
Error!
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
    at org.apache.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:215)
    at org.apache.druid.cli.CliPeon.run(CliPeon.java:295)
    at org.apache.druid.cli.Main.main(Main.java:113)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
    at org.apache.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:212)
    ... 2 more
Caused by: java.lang.OutOfMemoryError: Java heap space

排查与优化建议 #

• 调整Peon进程JVM内存：检查Druid配置中druid.indexer.runner.javaOpts参数，默认堆内存可能不足以支撑聚合任务，根据服务器物理内存调整，例如设置为-Xmx8g -Xms8g（建议不超过物理内存的70%）。
• 优化内存阈值配置：
  • 当前maxBytesInMemory设置为200KB过小，易导致频繁持久化但也可能在数据突增时引发内存压力，可调整为100MB左右（"maxBytesInMemory": 100000000），配合maxRowsInMemory平衡内存使用。
  • 缩短intermediatePersistPeriod，从PT10M改为PT5M，让任务更频繁地将内存数据持久化到磁盘，减少内存占用。
• 优化Sketch指标：quantilesDoublesSketch的k=128会占用较多内存，若数据基数极大，可尝试降低k值（如64），同时确认Rollup是否正常生效，避免重复聚合未合并的Sketch数据。
• 调整粒度配置：queryGranularity=MINUTE结合高基数维度会导致内存中堆积大量聚合桶，可尝试将queryGranularity调整为更大粒度（如FIVE_MINUTE），或检查维度数量与基数是否过高。
• 监控内存变化：启用Druid内置监控，观察任务运行时堆内存的增长趋势，定位内存占用过高的阶段（聚合/持久化），排查是否存在内存泄漏。
• 检查数据波动：确认Kafka主题是否存在数据量突增或单条消息过大的情况，导致短时间内内存堆积过多未处理数据。

内容的提问来源于stack exchange，提问作者Mukesh Kumar