Linux内核调试：crash工具报kmem_cache_s_num偏移错误的解决办法

环境信息 #

• 内核版本：Linux pegasusv 5.19.0-41-generic #42-Ubuntu SMP PREEMPT_DYNAMIC Mon Apr 17 19:17:03 UTC 2023 x86_64
• 运行环境：QEMU/KVM
• panic触发命令：echo c > /proc/sysrq-trigger
• 前置操作：已配置内核崩溃转储、下载对应内核调试符号，参考旧版Ubuntu教程完成配置

错误现象 #

运行crash工具加载内核镜像和崩溃转储文件时，出现如下错误：

pegasus@pegasusv:/usr/lib/debug/boot$ crash vmlinux-5.19.0-41-generic /var/crash/202305100142/dump.202305100142 

crash 8.0.0
Copyright (C) 2002-2021  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter \"help copying\" to see the conditions.
This program has absolutely no warranty.  Enter \"help warranty\" for details.
 
GNU gdb (GDB) 10.2                             
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type \"show copying\" and \"show warranty\" for details.
This GDB was configured as \"x86_64-pc-linux-gnu\".
Type \"show configuration\" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type \"help\".
Type \"apropos word\" to search for commands related to \"word\"...

please wait... (gathering kmem slab cache data)
crash: invalid structure member offset: kmem_cache_s_num
       FILE: memory.c  LINE: 9619  FUNCTION: kmem_cache_init()

[/usr/bin/crash] error trace: 56207d0cb69e => 56207d09f2f4 => 56207d16d11b => 56207d16d09c

解决方案 #

1. 升级crash工具版本 #

crash 8.0.0版本对Linux 5.19内核的slab结构解析存在兼容性问题，需升级到8.1.x及以上版本：

• 编译安装最新版crash工具：

  git clone https://github.com/crash-utility/crash.git
  cd crash
  make
  sudo make install
  

• 安装完成后，重新运行crash命令加载转储文件即可。

2. 验证调试符号与内核的匹配性 #

确保调试镜像与当前内核版本完全一致：

• 检查调试镜像的调试段：

  objdump -h /usr/lib/debug/boot/vmlinux-5.19.0-41-generic | grep -i debug
  

• 若缺少调试段，重新安装对应内核的调试符号包：

  sudo apt reinstall linux-image-5.19.0-41-generic-dbgsym
  

3. 临时绕过slab结构解析（Workaround） #

若无法立即升级crash工具，可通过内核启动参数禁用slab缓存的相关特性：

• 编辑/etc/default/grub文件，修改GRUB_CMDLINE_LINUX参数：

  GRUB_CMDLINE_LINUX="slab_nomerge kmem_cache_debug=0"
  

• 更新grub配置并重启系统：

  sudo update-grub
  sudo reboot
  

• 重启后重新触发panic，再尝试用crash工具调试。

内容的提问来源于stack exchange，提问作者NobinPegasus