You need to enable JavaScript to run this app. CentOS 6.5使用perf做持续性能剖析时触发内核panic咨询
优惠活动
大模型
产品
解决方案
定价
更多

CentOS 6.5使用perf做持续性能剖析时触发内核panic咨询

问题场景
  • 为实现运行中进程的持续性能剖析,通过crontab定时调用Python脚本拉起perf子进程,采集supervise托管的守护进程性能数据,实际使用的perf采集命令为:
    perf record -p {target process} -e cycles:u -a -q -g -- sleep {some time}
  • 日常采集流程运行正常,仅在通过svc -t重启目标进程(更新可执行文件)的场景下,有概率触发内核panic,必须重启整机才能恢复。
  • 运行环境基线:
    • 系统发行版:CentOS release 6.5 (Final)
    • 初始部署内核版本:2.6.32-431.23.3.el6.x86_64
    • 触发故障时实际运行内核版本:2.6.32-573.22.1.el6.x86_64
  • 已知环境异常特征:
    • 崩溃发生在被perf监控的进程退出阶段,属于偶发问题,无稳定复现路径,初步判断为内核层面的竞态类缺陷
    • 系统自带的旧版本perf存在行为异常:被监控的目标进程终止后,perf不会自动退出,会持续运行直到被手动中断,该行为对目标进程退出流程的影响暂未明确。
采集到的内核崩溃日志

通用保护故障基础信息

general protection fault: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 1
Modules linked in: AliSecGuard(U) AliSecProcFilter64(U) tcp_diag inet_diag joydev microcode virtio_net virtio_balloon shpchp i2c_piix4 i2c_core ext4 jbd2 mbcache virtio_blk virtio_console virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 22748, comm: server Not tainted 2.6.32-573.22.1.el6.x86_64 #1 Alibaba Cloud Alibaba Cloud ECS
RIP: 0010:[<ffffffff8111db57>]  [<ffffffff8111db57>] ring_buffer_put+0x77/0xf0
RSP: 0018:ffff8801afadbda8  EFLAGS: 00010006
RAX: ffff880416d81e60 RBX: ffff8803d335f000 RCX: 63496d6165727473
RDX: 676e697274736f5f RSI: 0000000000000003 RDI: ffff880416d81c00
RBP: ffff8801afadbdd8 R08: 0000000000000001 R09: 00000000ffffffff
R10: 00000000ffffffff R11: dead000000200200 R12: ffff8803d335f058
R13: 676e697274736cff R14: ffff8803d335f060 R15: 0000000000000202
FS:  0000000000000000(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000004264d70 CR3: 0000000001a8d000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Process gameserver (pid: 22748, threadinfo ffff8801afad8000, task ffff8803a19cd520)
Stack:
 ffff8801afadbdf8 ffff8803a1b03800 ffff8804138bf78c ffff8804138bf790
<d> ffff88001a9fb800 ffff8804182a1c80 ffff8801afadbdf8 ffffffff8111e377
<d> ffff8804138bf790 ffff8803a1b03800 ffff8801afadbe28 ffffffff8111fe72
Call Trace:
 [<ffffffff8111e377>] free_event+0x37/0x170
 [<ffffffff8111fe72>] perf_event_release_kernel+0x72/0xb0
 [<ffffffff8111ff49>] put_event+0x99/0xd0
 [<ffffffff81123a65>] __perf_event_exit_task+0xf5/0x150
 [<ffffffff81123c91>] perf_event_exit_task+0x1d1/0x210
 [<ffffffff8107ca24>] do_exit+0x1e4/0x870
 [<ffffffff8107d1b7>] sys_exit+0x17/0x20
 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
Code: ff ff 4c 39 f0 48 8b 97 60 02 00 00 74 5f 4c 8d aa a0 fd ff ff eb 08 0f 1f 44 00 00 49 89 cd 48 8b 8f 68 02 00 00 be 03 00 00 00 <48> 89 4a 08 48 89 11 31 c9 48 89 87 60 02 00 00 48 89 87 68 02
RIP  [<ffffffff8111db57>] ring_buffer_put+0x77/0xf0
 RSP <ffff8801afadbda8>

kexec捕获的崩溃调用栈

PID: 22748  TASK: ffff8803a19cd520  CPU: 1   COMMAND: "server"
 #0 [ffff8801afadbb30] machine_kexec at ffffffff8103d1fb
 #1 [ffff8801afadbb90] crash_kexec at ffffffff810cc882
 #2 [ffff8801afadbc60] oops_end at ffffffff8153da20
 #3 [ffff8801afadbc90] die at ffffffff81010fab
 #4 [ffff8801afadbcc0] do_general_protection at ffffffff8153d512
 #5 [ffff8801afadbcf0] general_protection at ffffffff8153cce5
    [exception RIP: ring_buffer_put+119]
    RIP: ffffffff8111db57  RSP: ffff8801afadbda8  RFLAGS: 00010006
    RAX: ffff880416d81e60  RBX: ffff8803d335f000  RCX: 63496d6165727473
    RDX: 676e697274736f5f  RSI: 0000000000000003  RDI: ffff880416d81c00
    RBP: ffff8801afadbdd8   R8: 0000000000000001   R9: 00000000ffffffff
    R10: 00000000ffffffff  R11: dead000000200200  R12: ffff8803d335f058
    R13: 676e697274736cff  R14: ffff8803d335f060  R15: 0000000000000202
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #6 [ffff8801afadbda0] ring_buffer_put at ffffffff8111db20
 #7 [ffff8801afadbde0] free_event at ffffffff8111e377
 #8 [ffff8801afadbe00] perf_event_release_kernel at ffffffff8111fe72
 #9 [ffff8801afadbe30] put_event at ffffffff8111ff49
#10 [ffff8801afadbe60] __perf_event_exit_task at ffffffff81123a65
#11 [ffff8801afadbe90] perf_event_exit_task at ffffffff81123c91
#12 [ffff8801afadbef0] do_exit at ffffffff8107ca24
#13 [ffff8801afadbf70] sys_exit at ffffffff8107d1b7
#14 [ffff8801afadbf80] system_call_fastpath at ffffffff8100b0d2
    RIP: 0000003026207c41  RSP: 00007fe4d1e56e50  RFLAGS: 00000246
    RAX: 000000000000003c  RBX: ffffffff8100b0d2  RCX: 0000000000000001
    RDX: 0000000000000004  RSI: 00000000009fb000  RDI: 0000000000000000
    RBP: 0000000000000000   R8: 000000000598f280   R9: 00000000000058dc
    R10: 00007fe4d259f3ac  R11: 0000000000000246  R12: ffffffff8107d1b7
    R13: ffff8801afadbf78  R14: 0000000000000003  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
崩溃特征分析
  • 崩溃触发点位于perf子系统的ring_buffer_put函数,属于引用计数操作阶段的通用保护故障:执行操作时传入的ring_buffer内存指针已经被异常释放,触发非法内存访问。
  • 完整崩溃路径完全匹配场景特征:目标进程退出时,内核会遍历所有绑定到该进程的perf事件,依次触发perf_event_exit_task流程,释放对应事件占用的ring_buffer资源,整个流程直接关联perf工具端的状态。
  • 旧版perf的异常行为是竞态触发的重要诱因:当目标进程退出后,perf端没有及时释放自身持有的perf事件文件描述符,会导致进程退出阶段的内核资源释放流程,与perf端残留的事件引用同时操作同一个ring_buffer内存块,触发双重释放、野指针访问的竞态问题。

内容的提问来源于stack exchange,提问作者Adachi Sakura

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.11.14 20:06:08