服务器平台DeviceTree创建PCI节点及RCU调度停滞问题解决咨询

我们的目标是通过DeviceTree启动Linux内核，由Bootloader传递dtb文件完成PCI设备枚举，当前在Ampere Altra平台尝试枚举单个PCI主机桥时，出现RCU调度停滞问题。

硬件信息（lspci输出） #

0000:00:00.0 Host bridge: Ampere Computing, LLC Device 0300
Subsystem: Ampere Computing, LLC Device e100
Flags: fast devsel, NUMA node 0

0000:00:01.0 PCI bridge: Ampere Computing, LLC Altra PCI Express Root Port a1 (prog-if 00 [Normal decode])

Flags: bus master, fast devsel, latency 0, IRQ 70, NUMA node 0, IOMMU group 27
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: 04000000-041fffff [size=2M]
Prefetchable memory behind bridge: 00006c0100000000-00006c01001fffff [size=2M]
Capabilities: [40] Power Management version 3
Capabilities: [70] Express Root Port (Slot+), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Virtual Channel
Capabilities: [170] Secondary PCI Express
Capabilities: [1a0] Physical Layer 16.0 GT/s <?>
Capabilities: [1d0] Lane Margining at the Receiver <?>
Capabilities: [218] Extended Capability ID 0x2a
Capabilities: [258] Access Control Services
Capabilities: [270] Downstream Port Containment
Capabilities: [2b4] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Capabilities: [3b4] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [3ec] Data Link Feature <?>
Capabilities: [3f8] Designated Vendor-Specific: Vendor=0001 ID=0000 Rev=1 Len=64 <?>
Capabilities: [43c] Vendor Specific Information: ID=0005 Rev=1 Len=018 <?>
Kernel driver in use: pcieport

编写的PCI DeviceTree节点 #

pcie0: pcie@6ffff0000000 {
#address-cells = <3>;
#size-cells = <2>;
compatible = "pci-host-cam-generic", "pci-host-ecam-generic";
device_type = "pci";
dma-coherent;
linux,pci-domain = <0x00>;
bus-range = <0x0 0xff>;
reg = <0x6fff 0xf0000000  0x0 0x10000000>;
ranges = <0x2000000 0x00006c01 0x00000000 0x00006c01 0x00000000 0x000003fe 0xe0000000
          0xc3000800 0x00006c00 0x04000000 0x00006c00 0x04000000 0x00000000 0x10000000>;  
};

启动后内核输出日志 #

pci-host-generic 6ffff0000000.pcie: host bridge /pcie@6ffff0000000 ranges:
pci-host-generic 6ffff0000000.pcie: Parsing ranges property...
pci-host-generic 6ffff0000000.pcie:      MEM 0x6c0100000000..0x6fffdfffffff -> 0x6c0100000000
pci-host-generic 6ffff0000000.pcie:      MEM 0x6c0004000000..0x6c0013ffffff -> 0x6c0004000000
pci-host-generic 6ffff0000000.pcie: ECAM at [mem 0x6ffff0000000-0x6fffffffffff] for [bus 00-ff]
pci-host-generic 6ffff0000000.pcie: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [bus 00-ff]
pci_bus 0000:00: root bus resource [mem 0x6c0100000000-0x6fffdfffffff]
pci_bus 0000:00: root bus resource [mem 0x6c0004000000-0x6c0013ffffff pref]
pci_bus 0000:00: scanning bus
pci 0000:00:00.0: [1def:0300] type 00 class 0x060000
rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
rcu:    1-...0: (10 ticks this GP) idle=702/1/0x4000000000000000 softirq=67/67 fqs=2625 
    (detected by 10, t=5255 jiffies, g=-1055, q=1)
Task dump for CPU 1:
task:swapper/0       state:R  running task     stack:    0 pid:    1 ppid:     0 flags:0x0000002a
Call trace:
 __switch_to+0xf0/0x128
 0x0

排查解决思路 #

• 修正PCI地址范围配置
  日志显示当前ranges配置的地址范围远大于硬件实际提供的2M内存空间（预取/非预取各2M），过大的范围可能导致内核访问未映射的物理内存，触发总线挂起进而引发RCU stall。按照硬件实际地址调整ranges：

  • 预取内存（64bit）：0x2000000 0x6c01 0x0 0x6c01 0x0 0x0 0x200000
  • 非预取内存（32bit）：0x02000000 0x6c00 0x4000000 0x6c00 0x4000000 0x0 0x200000
    注意原节点中0xc3000800的类型字段不符合PCI ECAM编码规则，非预取内存应使用0x02000000（高3bit为1，表示32bit非预取内存）。

• 验证ECAM区域配置
  核对reg属性中的ECAM基地址和大小是否与Ampere Altra平台手册一致，避免覆盖其他设备地址空间引发冲突。同时缩小bus-range为实际支持的范围（比如<0x0 0x1>），减少不必要的总线扫描范围。

• 检查DMA一致性设置
  若平台依赖IOMMU实现DMA地址转换，错误开启dma-coherent会导致DMA操作异常引发死锁。尝试移除dma-coherent属性，并确认IOMMU相关DeviceTree配置是否正确。

• 排查PCI枚举时的总线挂起
  RCU stall发生在总线扫描阶段，说明枚举过程中可能存在异常PCI访问。可添加内核启动参数：

  • pci=earlydump：提前打印PCI配置空间信息，定位异常设备
  • pci=no_scan_bus：跳过总线扫描，验证是否为扫描过程导致的问题
    同时检查PCI桥的中断配置（硬件IRQ70）是否在DeviceTree中正确映射，错误的中断配置可能引发中断风暴或挂起。

• 核对内核版本兼容性
  确认使用的Linux内核版本是否支持Ampere Altra的PCIe控制器，旧版本可能存在枚举bug。建议升级至5.15及以上的LTS内核，并检查是否有相关补丁修复了类似RCU stall问题。

内容的提问来源于stack exchange，提问作者Jayaprakash