RHEL 7.9 Kafka客户端生产集群低CPU空闲率分析及阈值设置咨询
RHEL 7.9 Kafka客户端生产集群低CPU空闲率分析及阈值设置咨询
问题背景
我们有一个由RHEL 7.9物理机构成的集群,这些机器作为Kafka客户端生产者使用。单节点硬件规格如下(DELL物理机):
- 48核心CPU
- 128GB内存
观察到的现象
- 多数节点通过
sar命令观测到CPU空闲率极低,维持在2%-6%区间 - 偶尔出现节点短暂卡顿(HANG几秒)
- CPU负载平均值在40-60区间,暂时未表现出明显异常
核心疑问
- 2%-6%的CPU空闲率是否属于正常范围?是否需要介入处理?
- 若需要设置低空闲率告警阈值,该如何合理定义?比如设置10%或20%是否合适?
以下是我们采集的系统监控数据:
系统监控数据
1. vmstat 1 3输出
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 60 0 0 65249020 3364 1082656 0 0 2 1 115 43 86 3 11 0 0 46 0 0 65240956 3364 1082656 0 0 0 0 167113 10096 92 3 5 0 0 53 0 0 65248888 3364 1082656 0 0 0 0 208360 9795 92 4 4 0 0
2. sar 5 5输出
09:46:10 AM CPU %user %nice %system %iowait %steal %idle 09:46:15 AM all 91.93 0.00 4.03 0.00 0.00 4.04 09:46:20 AM all 91.90 0.00 3.48 0.00 0.00 4.62 09:46:25 AM all 91.76 0.00 3.21 0.00 0.00 5.04 09:46:30 AM all 91.69 0.00 2.84 0.00 0.00 5.47 09:46:35 AM all 92.17 0.00 4.50 0.00 0.00 3.34 Average: all 91.89 0.00 3.61 0.00 0.00 4.50
3. CPU使用率计算结果
执行命令:
top -bn2 | grep '%Cpu' | tail -1 | grep -P '(....|...) id,'|awk '{print "CPU Usage: " 100-$8 "%"}'
输出:
CPU Usage: 96.2%
4. sar -P ALL 1 1输出(单CPU核心统计)
Average: CPU %user %nice %system %iowait %steal %idle Average: all 91.94 0.00 4.75 0.00 0.00 3.31 Average: 0 12.24 0.00 51.02 0.00 0.00 36.73 Average: 1 17.35 0.00 41.84 0.00 0.00 40.82 Average: 2 100.00 0.00 0.00 0.00 0.00 0.00 Average: 3 98.02 0.00 1.98 0.00 0.00 0.00 Average: 4 99.00 0.00 1.00 0.00 0.00 0.00 Average: 5 98.00 0.00 2.00 0.00 0.00 0.00 Average: 6 98.02 0.00 1.98 0.00 0.00 0.00 Average: 7 98.00 0.00 2.00 0.00 0.00 0.00 Average: 8 98.00 0.00 2.00 0.00 0.00 0.00 Average: 9 99.00 0.00 1.00 0.00 0.00 0.00 Average: 10 99.00 0.00 1.00 0.00 0.00 0.00 Average: 11 98.02 0.00 1.98 0.00 0.00 0.00 Average: 12 98.00 0.00 2.00 0.00 0.00 0.00 Average: 13 98.00 0.00 2.00 0.00 0.00 0.00 Average: 14 98.99 0.00 1.01 0.00 0.00 0.00 Average: 15 99.00 0.00 1.00 0.00 0.00 0.00 Average: 16 98.99 0.00 1.01 0.00 0.00 0.00 Average: 17 99.00 0.00 1.00 0.00 0.00 0.00 Average: 18 98.00 0.00 2.00 0.00 0.00 0.00 Average: 19 99.00 0.00 1.00 0.00 0.00 0.00 Average: 20 99.00 0.00 1.00 0.00 0.00 0.00 Average: 21 97.06 0.00 2.94 0.00 0.00 0.00 Average: 22 98.00 0.00 2.00 0.00 0.00 0.00 Average: 23 98.02 0.00 1.98 0.00 0.00 0.00 Average: 24 20.20 0.00 41.41 0.00 0.00 38.38 Average: 25 31.31 0.00 23.23 0.00 0.00 45.45 Average: 26 99.01 0.00 0.99 0.00 0.00 0.00 Average: 27 98.02 0.00 1.98 0.00 0.00 0.00 Average: 28 98.02 0.00 1.98 0.00 0.00 0.00 Average: 29 98.02 0.00 1.98 0.00 0.00 0.00 Average: 30 98.99 0.00 1.01 0.00 0.00 0.00 Average: 31 98.02 0.00 1.98 0.00 0.00 0.00 Average: 32 98.99 0.00 1.01 0.00 0.00 0.00 Average: 33 98.02 0.00 1.98 0.00 0.00 0.00 Average: 34 98.02 0.00 1.98 0.00 0.00 0.00 Average: 35 99.00 0.00 1.00 0.00 0.00 0.00 Average: 36 97.06 0.00 2.94 0.00 0.00 0.00 Average: 37 98.00 0.00 2.00 0.00 0.00 0.00 Average: 38 97.06 0.00 2.94 0.00 0.00 0.00 Average: 39 98.99 0.00 1.01 0.00 0.00 0.00 Average: 40 98.00 0.00 2.00 0.00 0.00 0.00 Average: 41 98.00 0.00 2.00 0.00 0.00 0.00 Average: 42 98.99 0.00 1.01 0.00 0.00 0.00 Average: 43 98.00 0.00 2.00 0.00 0.00 0.00 Average: 44 98.00 0.00 2.00 0.00 0.00 0.00 Average: 45 98.99 0.00 1.01 0.00 0.00 0.00 Average: 46 98.00 0.00 2.00 0.00 0.00 0.00 Average: 47 98.00 0.00 2.00 0.00 0.00 0.00
5. uptime输出
09:53:23 up 2:07, 4 users, load average: 49.94, 49.17, 49.17
6. free -g输出
total used free shared buff/cache available Mem: 125 61 62 0 1 62 Swap: 15 0 15
7. iostat输出
Linux 4.18.0-305.el8.x86_64 (dragon12) 06/29/2023 _x86_64_ (48 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 86.30 0.00 3.27 0.00 0.00 10.42 Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 2.59 91.45 60.94 754657 502839 dm-0 1.95 83.37 54.04 687962 445952 dm-1 0.01 0.27 0.00 2220 0 dm-2 0.57 2.76 6.65 22810 54839
8. sar -B 2 5输出(内存分页统计)
Linux 4.18.0-305.el8.x86_64 (dragon12) 06/29/2023 _x86_64_ (48 CPU) 10:05:30 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 10:05:32 AM 0.00 1.50 23815.00 0.00 43641.00 0.00 0.00 0.00 0.00 10:05:34 AM 0.00 0.00 27231.50 0.00 45495.00 0.00 0.00 0.00 0.00 10:05:36 AM 0.00 0.00 28570.50 0.00 47603.50 0.00 0.00 0.00 0.00 10:05:38 AM 0.00 0.00 27766.50 0.00 48434.50 0.00 0.00 0.00 0.00 10:05:40 AM 0.00 14.00 28007.00 0.00 48733.50 0.00 0.00 0.00 0.00 Average: 0.00 3.10 27078.10 0.00 46781.50 0.00 0.00 0.00 0
相关产品推荐
相关产品推荐

