OrangePI R1（Allwinner sun8i）自定义系统rcu_sched报错原因及调试咨询

我正在为搭载Allwinner sun8i SoC的OrangePI R1制作自定义发行版，为将自定义Linux系统适配到16M SPI NOR中，精简了kernel_defconfig。设备运行数天后，串口控制台出现如下报错：

admin@orange-pi-r1:~# [65779.614485] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[65779.620458] rcu:     3-...!: (4 GPs behind) idle=200/0/0x0 softirq=17870/17870 fqs=0  (false positive?)
[65779.629630]  (detected by 2, t=2103 jiffies, g=68925, q=83)
[65779.635224] Sending NMI from CPU 2 to CPUs 3:
[65779.639605] NMI backtrace for cpu 3
[65779.639619] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           O      5.15.35 #1
[65779.639636] Hardware name: Allwinner sun8i Family
[65779.639644] PC is at 0xc0106330
[65779.639651] LR is at 0xc0106340
[65779.639657] pc : [<c0106330>]    lr : [<c0106340>]    psr: 60000013
[65779.639669] sp : c0cadfa8  ip : 00000000  fp : c0805f90
[65779.639679] r10: c0cadfb8  r9 : 410fc075  r8 : c0805f4c
[65779.639689] r7 : c0cac000  r6 : c0cac000  r5 : 00000000  r4 : 00000000
[65779.639701] r3 : c0113ca0  r2 : 12e10204  r1 : 00000000  r0 : 12e10204
[65779.639714] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[65779.639729] Control: 10c5387d  Table: 42a7c06a  DAC: 00000051
[65779.639738] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           O      5.15.35 #1
[65779.639754] Hardware name: Allwinner sun8i Family
[65779.639767] Function entered at [<c010c0b8>] from [<c0108bf4>]
[65779.639779] Function entered at [<c0108bf4>] from [<c0540794>]
[65779.639791] Function entered at [<c0540794>] from [<c037e2e0>]
[65779.639803] Function entered at [<c037e2e0>] from [<c010ad00>]
[65779.639814] Function entered at [<c010ad00>] from [<c010ad50>]
[65779.639825] Function entered at [<c010ad50>] from [<c016d288>]
[65779.639837] Function entered at [<c016d288>] from [<c0167b94>]
[65779.639848] Function entered at [<c0167b94>] from [<c0168218>]
[65779.639860] Function entered at [<c0168218>] from [<c038dff0>]
[65779.639871] Function entered at [<c038dff0>] from [<c0100b7c>]
[65779.639881] Exception stack(0xc0cadf58 to 0xc0cadfa0)
[65779.639896] df40:                                                       12e10204 00000000
[65779.639915] df60: 12e10204 c0113ca0 00000000 00000000 c0cac000 c0cac000 c0805f4c 410fc075
[65779.639934] df80: c0cadfb8 c0805f90 00000000 c0cadfa8 c0106340 c0106330 60000013 ffffffff
[65779.639946] Function entered at [<c0100b7c>] from [<c0106330>]
[65779.639957] Function entered at [<c0106330>] from [<c0546b94>]
[65779.639969] Function entered at [<c0546b94>] from [<c0148244>]
[65779.639980] Function entered at [<c0148244>] from [<c0148680>]
[65779.639991] Function entered at [<c0148680>] from [<401014d0>]
[65779.640602] rcu: rcu_sched kthread timer wakeup didn't happen for 2103 jiffies! g68925 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[65779.845052] rcu:     Possible timer handling issue on cpu=1 timer-softirq=33908
[65779.852117] rcu: rcu_sched kthread starved for 2125 jiffies! g68925 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
[65779.862406] rcu:     Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
[65779.871383] rcu: RCU grace-period kthread stack dump:
[65779.876447] task:rcu_sched       state:I stack:    0 pid:   12 ppid:     2 flags:0x00000000
[65779.884836] Function entered at [<c0543560>] from [<c0543780>]
[65779.890688] Function entered at [<c0543780>] from [<c05462f4>]
[65779.896539] Function entered at [<c05462f4>] from [<c0176720>]
[65779.902391] Function entered at [<c0176720>] from [<c01791e4>]
[65779.908241] Function entered at [<c01791e4>] from [<c013cc18>]
[65779.914093] Function entered at [<c013cc18>] from [<c0100130>]
[65779.919942] Exception stack(0xc0c71fb0 to 0xc0c71ff8)
[65779.925008] 1fa0:                                     00000000 00000000 00000000 00000000
[65779.933212] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[65779.941415] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000

可能的原因 #

• RCU调度器阻塞：报错核心是rcu_sched detected stalls，说明CPU3在RCU宽限期内未响应，无法完成RCU回调处理。
• 内核配置精简过度：误删了RCU、CPU idle或定时器相关的关键配置，比如CONFIG_RCU_SCHED、CONFIG_CPU_IDLE、CONFIG_HIGH_RES_TIMERS等，导致系统无法在idle状态下正确处理RCU事件。
• CPU idle路径异常：从backtrace看CPU3处于idle线程（swapper/3），可能idle状态下的中断或时钟处理被破坏，导致RCU无法唤醒该CPU。
• 定时器软中断故障：日志提到Possible timer handling issue on cpu=1，CPU1的定时器软中断异常会影响RCU调度线程的唤醒，导致rcu_sched线程被饿死。
• SPI NOR IO阻塞：系统运行在SPI NOR上，频繁的低速IO操作可能长时间屏蔽中断，或引发内存访问异常，间接导致RCU stall。

调试步骤 #

1. 恢复关键内核配置

   • 对比官方OrangePI R1的默认kernel_defconfig，检查并恢复以下配置：
     • 确保CONFIG_RCU_SUPPORT=y、CONFIG_RCU_SCHED=y、CONFIG_RCU_STALL_COMMON=y开启；
     • 保留CPU idle相关选项（CONFIG_CPU_IDLE=y、CONFIG_CPU_IDLE_GOV_MENU=y）；
     • 开启定时器相关配置（CONFIG_HIGH_RES_TIMERS=y、CONFIG_PREEMPT=y）；
     • 若开启了CONFIG_RCU_NOCB_CPU，暂时关闭该选项。

2. 解析内核符号表定位问题函数

   • 使用addr2line工具将backtrace中的内存地址转换为具体函数：

     addr2line -e vmlinux c0106330 c0540794 c037e2e0
     

   • 通过转换结果确定CPU3在idle状态下卡住的具体函数，判断是idle路径还是RCU回调的问题。

3. 监控RCU运行状态

   • 查看RCU stall统计：cat /sys/kernel/debug/rcu/rcu_sched/stall_detected；
   • 实时监控RCU调度器状态：watch -n 1 cat /proc/rcu_sched，观察是否有回调积压。

4. 排查SPI NOR性能与驱动问题

   • 测试SPI NOR读写性能：dd if=/dev/mtdblock0 of=/dev/null bs=1M count=10；
   • 查看SPI驱动日志：dmesg | grep spi，检查是否有超时或错误信息。

5. 开启RCU调试日志

   • 在kernel_defconfig中添加CONFIG_RCU_TRACE=y、CONFIG_RCU_DEBUG=y，重新编译内核；
   • 下次出现stall时，系统会输出更详细的RCU回调信息，帮助定位具体触发点。

内容的提问来源于stack exchange，提问作者BlameCapitalism