HDP 2.6.4集群Standby NameNode JVM停顿及GC参数调优咨询

集群环境信息 #

• Hadoop版本：Hortonworks HDP 2.6.4
• NameNode配置：2个节点（1 Active + 1 Standby），操作系统RHEL 7.2，OS层面无异常
• Worker节点：12台，每台32核，部署DataNode和NodeManager
• HDFS文件数量：700万
• NameNode堆内存：已从60G调至100G

问题现象 #

Standby NameNode触发冒烟测试脚本告警：Detected pause in JVM or host machine，堆内存调整后告警仍存在。

NameNode日志显示每1-2小时出现一次JVM停顿，时长约1698-2524ms，日志片段：

2022-10-27 14:04:49,728 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2044ms
2022-10-27 16:21:33,973 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2524ms
2022-10-27 17:31:35,333 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2444ms
2022-10-27 18:55:55,387 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2134ms
2022-10-27 19:42:00,816 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2153ms
2022-10-27 20:50:23,624 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2050ms
2022-10-27 21:07:01,240 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2343ms
2022-10-27 23:53:00,507 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2120ms
2022-10-28 00:43:30,633 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1811ms
2022-10-28 00:53:35,120 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2192ms
2022-10-28 02:07:39,660 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2353ms
2022-10-28 02:49:25,018 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1698ms
2022-10-28 03:00:20,592 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2432ms
2022-10-28 05:02:15,093 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2016ms
2022-10-28 06:52:46,672 INFO  util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approxim

排查过程 #

通过jstat排查PID为1837的NameNode进程，FGCT数值正常，未指向堆内存问题，结果如下：

# /usr/jdk64/jdk1.8.0_112/bin/jstat -gcutil    1837 10 10
  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720
  0.00   1.95  32.30  34.74  97.89      -    197  173.922     2    1.798  175.720

当前NameNode JVM参数配置：

export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=8 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Xms1G -Xmx1G -XX:NewSize=128M -XX:MaxNewSize=128M -XX:PermSize=128M -XX:MaxPermSize=256M -verbose:gc -Xloggc:/Users/chris/hadoop-deploy-trunk/hadoop-3.0.0-SNAPSHOT/logs/gc.log-`date +'%Y%m%d%H%M'` -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:ErrorFile=/Users/chris/hadoop-deploy-trunk/hadoop-3.0.0-SNAPSHOT/logs/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError $HADOOP_NAMENODE_OPTS"

问题解答 #

1. 是否可以将CMSInitiatingOccupancyFraction调整为90？ #

可以调整，但需逐步测试验证：

• 该参数设置为90时，CMS会在老年代占用率达90%时触发并发收集。当前老年代占用率仅34.74%，调高阈值能减少CMS收集频率，但如果老年代内存增长速度超出预期，可能导致并发收集赶不上内存消耗，最终触发Full GC（Stop-the-World），反而加重停顿。
• 建议先逐步调高，比如先设为80，观察GC日志和停顿情况，确认老年代内存增长平稳后，再考虑调至90。同时需保持-XX:+UseCMSInitiatingOccupancyOnly开启（当前已配置），确保JVM严格按设定阈值触发CMS收集。

2. 当前ParallelGCThreads配置为8，而节点拥有32核，是否可以将其调整为32？这是否有助于解决JVM停顿问题？ #

不建议直接调至32，需合理设置：

• ParallelGCThreads控制CMS初始标记、重新标记阶段的并行线程数（这两个阶段为Stop-the-World）。线程数过多会导致CPU上下文切换开销剧增，反而延长停顿时间。
• 通常建议设置为CPU核心数的50%-75%，32核节点可尝试设为16-24。
• 结合当前情况，停顿每1-2小时出现一次且FGC次数极少，这些停顿更可能来自CMS重新标记阶段或系统层面因素（如磁盘IO、进程抢占CPU）。调整ParallelGCThreads到合理范围可能缩短停顿时长，但需结合GC日志分析停顿具体发生阶段。

内容的提问来源于stack exchange，提问作者Judy