CentOS 6.5使用perf做持续性能剖析时触发内核panic咨询
问题场景
- 为实现运行中进程的持续性能剖析,通过crontab定时调用Python脚本拉起perf子进程,采集supervise托管的守护进程性能数据,实际使用的perf采集命令为:
perf record -p {target process} -e cycles:u -a -q -g -- sleep {some time} - 日常采集流程运行正常,仅在通过
svc -t重启目标进程(更新可执行文件)的场景下,有概率触发内核panic,必须重启整机才能恢复。 - 运行环境基线:
- 系统发行版:CentOS release 6.5 (Final)
- 初始部署内核版本:
2.6.32-431.23.3.el6.x86_64 - 触发故障时实际运行内核版本:
2.6.32-573.22.1.el6.x86_64
- 已知环境异常特征:
- 崩溃发生在被perf监控的进程退出阶段,属于偶发问题,无稳定复现路径,初步判断为内核层面的竞态类缺陷
- 系统自带的旧版本perf存在行为异常:被监控的目标进程终止后,perf不会自动退出,会持续运行直到被手动中断,该行为对目标进程退出流程的影响暂未明确。
采集到的内核崩溃日志
通用保护故障基础信息
general protection fault: 0000 [#1] SMP last sysfs file: /sys/devices/system/cpu/online CPU 1 Modules linked in: AliSecGuard(U) AliSecProcFilter64(U) tcp_diag inet_diag joydev microcode virtio_net virtio_balloon shpchp i2c_piix4 i2c_core ext4 jbd2 mbcache virtio_blk virtio_console virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Pid: 22748, comm: server Not tainted 2.6.32-573.22.1.el6.x86_64 #1 Alibaba Cloud Alibaba Cloud ECS RIP: 0010:[<ffffffff8111db57>] [<ffffffff8111db57>] ring_buffer_put+0x77/0xf0 RSP: 0018:ffff8801afadbda8 EFLAGS: 00010006 RAX: ffff880416d81e60 RBX: ffff8803d335f000 RCX: 63496d6165727473 RDX: 676e697274736f5f RSI: 0000000000000003 RDI: ffff880416d81c00 RBP: ffff8801afadbdd8 R08: 0000000000000001 R09: 00000000ffffffff R10: 00000000ffffffff R11: dead000000200200 R12: ffff8803d335f058 R13: 676e697274736cff R14: ffff8803d335f060 R15: 0000000000000202 FS: 0000000000000000(0000) GS:ffff880028240000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000004264d70 CR3: 0000000001a8d000 CR4: 00000000001406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Process gameserver (pid: 22748, threadinfo ffff8801afad8000, task ffff8803a19cd520) Stack: ffff8801afadbdf8 ffff8803a1b03800 ffff8804138bf78c ffff8804138bf790 <d> ffff88001a9fb800 ffff8804182a1c80 ffff8801afadbdf8 ffffffff8111e377 <d> ffff8804138bf790 ffff8803a1b03800 ffff8801afadbe28 ffffffff8111fe72 Call Trace: [<ffffffff8111e377>] free_event+0x37/0x170 [<ffffffff8111fe72>] perf_event_release_kernel+0x72/0xb0 [<ffffffff8111ff49>] put_event+0x99/0xd0 [<ffffffff81123a65>] __perf_event_exit_task+0xf5/0x150 [<ffffffff81123c91>] perf_event_exit_task+0x1d1/0x210 [<ffffffff8107ca24>] do_exit+0x1e4/0x870 [<ffffffff8107d1b7>] sys_exit+0x17/0x20 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b Code: ff ff 4c 39 f0 48 8b 97 60 02 00 00 74 5f 4c 8d aa a0 fd ff ff eb 08 0f 1f 44 00 00 49 89 cd 48 8b 8f 68 02 00 00 be 03 00 00 00 <48> 89 4a 08 48 89 11 31 c9 48 89 87 60 02 00 00 48 89 87 68 02 RIP [<ffffffff8111db57>] ring_buffer_put+0x77/0xf0 RSP <ffff8801afadbda8>
kexec捕获的崩溃调用栈
PID: 22748 TASK: ffff8803a19cd520 CPU: 1 COMMAND: "server" #0 [ffff8801afadbb30] machine_kexec at ffffffff8103d1fb #1 [ffff8801afadbb90] crash_kexec at ffffffff810cc882 #2 [ffff8801afadbc60] oops_end at ffffffff8153da20 #3 [ffff8801afadbc90] die at ffffffff81010fab #4 [ffff8801afadbcc0] do_general_protection at ffffffff8153d512 #5 [ffff8801afadbcf0] general_protection at ffffffff8153cce5 [exception RIP: ring_buffer_put+119] RIP: ffffffff8111db57 RSP: ffff8801afadbda8 RFLAGS: 00010006 RAX: ffff880416d81e60 RBX: ffff8803d335f000 RCX: 63496d6165727473 RDX: 676e697274736f5f RSI: 0000000000000003 RDI: ffff880416d81c00 RBP: ffff8801afadbdd8 R8: 0000000000000001 R9: 00000000ffffffff R10: 00000000ffffffff R11: dead000000200200 R12: ffff8803d335f058 R13: 676e697274736cff R14: ffff8803d335f060 R15: 0000000000000202 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #6 [ffff8801afadbda0] ring_buffer_put at ffffffff8111db20 #7 [ffff8801afadbde0] free_event at ffffffff8111e377 #8 [ffff8801afadbe00] perf_event_release_kernel at ffffffff8111fe72 #9 [ffff8801afadbe30] put_event at ffffffff8111ff49 #10 [ffff8801afadbe60] __perf_event_exit_task at ffffffff81123a65 #11 [ffff8801afadbe90] perf_event_exit_task at ffffffff81123c91 #12 [ffff8801afadbef0] do_exit at ffffffff8107ca24 #13 [ffff8801afadbf70] sys_exit at ffffffff8107d1b7 #14 [ffff8801afadbf80] system_call_fastpath at ffffffff8100b0d2 RIP: 0000003026207c41 RSP: 00007fe4d1e56e50 RFLAGS: 00000246 RAX: 000000000000003c RBX: ffffffff8100b0d2 RCX: 0000000000000001 RDX: 0000000000000004 RSI: 00000000009fb000 RDI: 0000000000000000 RBP: 0000000000000000 R8: 000000000598f280 R9: 00000000000058dc R10: 00007fe4d259f3ac R11: 0000000000000246 R12: ffffffff8107d1b7 R13: ffff8801afadbf78 R14: 0000000000000003 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
崩溃特征分析
- 崩溃触发点位于perf子系统的
ring_buffer_put函数,属于引用计数操作阶段的通用保护故障:执行操作时传入的ring_buffer内存指针已经被异常释放,触发非法内存访问。 - 完整崩溃路径完全匹配场景特征:目标进程退出时,内核会遍历所有绑定到该进程的perf事件,依次触发
perf_event_exit_task流程,释放对应事件占用的ring_buffer资源,整个流程直接关联perf工具端的状态。 - 旧版perf的异常行为是竞态触发的重要诱因:当目标进程退出后,perf端没有及时释放自身持有的perf事件文件描述符,会导致进程退出阶段的内核资源释放流程,与perf端残留的事件引用同时操作同一个ring_buffer内存块,触发双重释放、野指针访问的竞态问题。
内容的提问来源于stack exchange,提问作者Adachi Sakura
相关产品推荐
相关产品推荐

