Apache Pinot Server组件内存占用异常过高问题排查求助

大家好，我这边在生产环境用Docker部署Apache Pinot时遇到了棘手的内存问题，想请社区的大神们帮忙排查下根因。

我们完全按照Pinot官方文档的步骤在VM服务器上部署集群，但现在Pinot Server的内存占用远超我们根据数据量和复制因子算出的预期值，最终导致容器频繁触发OOMKilled错误。

已尝试的排查/优化操作 #

• 在JAVA_OPTS环境变量中配置了JVM的Xms和Xmx参数，调整了GC相关设置
• 给所有服务器部署了监控，试图获取系统运行的可观测性数据
• 从表定义中移除了倒排索引这类额外的索引配置

系统规格 #

我们的集群由3台Server、2台Controller、2台Broker组成，每台机器的硬件配置如下：

• CPU：24核
• 内存：64GB
• 磁盘：738GB SSD

相关配置文件示例 #

单台Server上的Docker Compose文件 #

version: '3.7'

services:
  pinot-server:
    image: apachepinot/pinot:0.11.0
    command: "StartServer -clusterName bigdata-pinot-ansible -zkAddress 172.16.24.14:2181,172.16.24.15:2181 -configFileName /server.conf"
    restart: unless-stopped
    hostname: server1
    container_name: server1
    ports:
      - "8096-8099:8096-8099"
      - "9000:9000"
      - "8008:8008"
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx20G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log -javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml"
    volumes:
      - ./server.conf:/server.conf
      - ./data/server_data/segment:/var/pinot/server/data/segment
      - ./data/server_data/index:/var/pinot/server/data/index

实时表配置 #

{
  "tableName": "<table-name>",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "schemaName": "<schema-name>",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "60",
    "replication": "3",
    "timeColumnName": "date",
    "allowNullTimeValue": false,
    "replicasPerPartition": "3",
    "segmentPushType": "APPEND",
    "completionConfig": {
      "completionMode": "DOWNLOAD"
    }
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant",
    "tagOverrideConfig": {
      "realtimeCompleted": "DefaultTenant_OFFLINE"
    }
  },
  "tableIndexConfig": {
    "noDictionaryColumns": [
      <some-fileds>
    ],
    "rangeIndexColumns": [
      <some-fileds>
    ],
    "rangeIndexVersion": 1,
    "autoGeneratedInvertedIndex": false,
    "createInvertedIndexDuringSegmentGeneration": false,
    "sortedColumn": [
      "date",
      "id"
    ],
    "bloomFilterColumns": [],
    "loadMode": "MMAP",
    "onHeapDictionaryColumns": [],
    "varLengthDictionaryColumns": [],
    "enableDefaultStarTree": false,
    "enableDynamicStarTreeCreation": false,
    "aggregateMetrics": false,
    "nullHandlingEnabled": false
  },
  "metadata": {},
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  },
  "query": {},
  "fieldConfigList": [],
  "upsertConfig": {
    "mode": "FULL",
    "hashFunction": "NONE"
  },
  "ingestionConfig": {
    "streamIngestionConfig": {
      "streamConfigMaps": [
        {
          "streamType": "kafka",
          "stream.kafka.topic.name": "<topic-name>",
          "stream.kafka.broker.list": "<kafka-brokers-list>",
          "stream.kafka.consumer.type": "lowlevel",
          "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
          "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
          "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
          "stream.kafka.decoder.prop.format": "JSON",
          "realtime.segment.flush.threshold.rows": "0",
          "realtime.segment.flush.threshold.time": "1h",
          "realtime.segment.flush.segment.size": "300M"
        }
      ]
    }
  },
  "isDimTable": false
}

server.conf配置文件 #

pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/var/pinot/server/data/index
pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment
pinot.set.instance.id.to.hostname=true

核心疑问 #

从Kafka实时摄入数据后，Pinot Server的内存就开始持续增长，完全不受控，最终导致容器被OOMKilled。我们尝试了几种常规优化都没效果，现在完全摸不清问题出在哪里，恳请大家帮忙分析下根因！

补充说明：

1. 我们的Pinot部署完整流程可以参考对应的GitHub仓库
2. 我们清楚Pinot数据量的计算公式：数据大小 = 单周期数据量 * 保留周期 * 复制因子，比如保留2天、每天2GB数据、复制因子3的话，预期数据量是223=12GB，但实际内存占用远高于这个值。

备注：内容来源于stack exchange，提问作者Mostafa Ghadimi