ARMv8-A架构Serror Panic故障排查求助及问题分析

我熟悉x86架构内核panic排查，近期碰到ARMv8-A架构下的Serror panic问题需要解决，求Serror问题的通用排查步骤指导。

触发panic的错误码为0xbe000411，根据ARMv8手册，ISS:0x411对应的AET为001，属于不可恢复错误（UEU）。目前无法定位故障指令，dmesg指向tcp_set_state+0x4/0x1e8，该位置是一条简单的mov指令：0xffffce3b1c776b2c <tcp_set_state+4>: mov x29, sp。由于panic日志未反映实际故障位置，且ARMv8手册未提及如何确定引发Serror的内存地址，不知如何继续排查。

该问题在多台机器上出现，排除硬件问题可能，所有机器均运行64k页表的ARM版5.4.210内核。故障共性为：在efi_runtime_service的get_time执行过程中被IRQ中断，IRQ处理时在tcp_set_state+0x4处触发Serror。出于保密，BIOS及内核版本细节已隐去。

注意到进入EFI runtime service时MM会切换为efi_mm，怀疑efi_mm的页表遍历存在问题，但此类问题应易于复现，我使用stress-ng对所有核心进行TCP流压力测试，并持续调用hwclock触发efi_runtime_service，运行约半天仍未复现问题。

附panic日志：

[14020380.440647] SError Interrupt on CPU46, code 0xbe000411 -- SError
[14020380.440648] CPU: 46 PID: 3999171 Comm: kworker/u256:5 Kdump: loaded Tainted: G           OE     5.4.210.xxx-64k-arm64 #64k
[14020380.440648] Hardware name: xxxxxxxxxxxxxxxxxxx, BIOS xxxxxxxxxx
[14020380.440649] Workqueue: efi_rts_wq efi_call_rts
[14020380.440651] pstate: 60400009 (nZCv daif +PAN -UAO)
[14020380.440651] pc : tcp_set_state+0x4/0x1e8
[14020380.440652] lr : tcp_fin+0xf8/0x1c0
[14020380.440652] sp : ffff8000105cf6e0
[14020380.440653] x29: ffff8000105cf760 x28: ffffa4e9cdf192c0 
[14020380.440654] x27: ffff40861a80b800 x26: ffff44831e5757a8 
[14020380.440655] x25: ffff445be3adc380 x24: 0000000000000020 
[14020380.440656] x23: 0000000000000000 x22: ffff44831e5757a8 
[14020380.440656] x21: ffffa4e9cd3f62f8 x20: ffff44831e575780 
[14020380.440657] x19: ffff445be3adc380 x18: 0000000000000000 
[14020380.440658] x17: 0000000000000000 x16: ffffa4e9ccfa1fa8 
[14020380.440659] x15: 00000000000000f4 x14: ccccccccccccce3c 
[14020380.440660] x13: ffffffff00000000 x12: 0000000000000002 
[14020380.440661] x11: 0000000000000000 x10: 00000000000000a8 
[14020380.440662] x9 : 0000000000000004 x8 : ffff78860b5e7a00 
[14020380.440662] x7 : 0000000000000001 x6 : 00013eca54043561 
[14020380.440663] x5 : 000000000001bc00 x4 : 000000000000001b 
[14020380.440664] x3 : 0000000000000302 x2 : 0000000000000001 
[14020380.440665] x1 : 0000000000000008 x0 : ffff445be3adc380 
[14020380.440666] Kernel panic - not syncing: Asynchronous SError Interrupt
[14020380.440667] CPU: 46 PID: 3999171 Comm: kworker/u256:5 Kdump: loaded Tainted: G           OE     5.4.210.xxx-64k-arm64 #64k
[14020380.440668] Hardware name: xxxxxxxxxxxxxxxxxxx, BIOS xxxxxxxxxx
[14020380.440668] Workqueue: efi_rts_wq efi_call_rts
[14020380.440669] Call trace:
[14020380.440669]  dump_backtrace+0x0/0x1f0
[14020380.440670]  show_stack+0x24/0x30
[14020380.440671]  dump_stack+0xbc/0x114
[14020380.440671]  panic+0x158/0x36c
[14020380.440672]  nmi_panic+0x68/0xa0
[14020380.440672]  arm64_serror_panic+0x84/0x90
[14020380.440672]  do_serror+0x84/0x128
[14020380.440673]  el1_error+0x8c/0x108
[14020380.440674]  tcp_set_state+0x4/0x1e8
[14020380.440674]  tcp_data_queue+0x610/0xb90
[14020380.440674]  tcp_rcv_established+0x1a8/0x558
[14020380.440675]  tcp_v6_do_rcv+0xc0/0x490
[14020380.440676]  tcp_v6_rcv+0xcd0/0xda8
[14020380.440676]  ip6_protocol_deliver_rcu+0xf0/0x538
[14020380.440676]  ip6_input_finish+0x30/0x40
[14020380.440677]  ip6_input+0xb4/0xc0
[14020380.440677]  ip6_sublist_rcv_finish+0x64/0x88
[14020380.440678]  ip6_sublist_rcv+0x1f4/0x278
[14020380.440678]  ipv6_list_rcv+0x130/0x160
[14020380.440679]  __netif_receive_skb_list_core+0x158/0x270
[14020380.440680]  netif_receive_skb_list_internal+0x1a8/0x2d8
[14020380.440680]  gro_normal_list.part.151+0x24/0x40
[14020380.440681]  napi_complete_done+0x118/0x1d0
[14020380.440681]  mlx5e_napi_poll+0x25c/0x688 [mlx5_core]
[14020380.440682]  net_rx_action+0x184/0x478
[14020380.440682]  __do_softirq+0x128/0x34c
[14020380.440683]  irq_exit+0xa4/0xc0
[14020380.440684]  __handle_domain_irq+0x70/0xc0
[14020380.440684]  gic_handle_irq+0xec/0x288
[14020380.440684]  el1_irq+0xcc/0x180
[14020380.440685]  0x2b263354
[14020380.440686]  0x2b26c150
[14020380.440686]  0x2b26c35c
[14020380.440686]  0x2b26c79c
[14020380.440687]  0x2b265184
[14020380.440687]  0x2b265390
[14020380.440687]  0x2b264334
[14020380.440687]  0x2b260c1c
[14020380.440688]  __efi_rt_asm_wrapper+0x28/0x44
[14020380.440688]  efi_call_rts+0x88/0x458
[14020380.440688]  process_one_work+0x1b4/0x410
[14020380.440688]  worker_thread+0x150/0x528
[14020380.440688]  kthread+0x124/0x128
[14020380.440689]  ret_from_fork+0x10/0x1c

内容的提问来源于stack exchange，提问作者Yuchen Zhang