CPU向PCIe设备BAR区域发送MMIO请求的载荷单元大小问询

问题描述 #

我想了解CPU向PCIe设备BAR区域发送MMIO请求时的载荷单元大小。我仅使用32位值（指针）对PCIe设备MMIO进行读写请求，但不确定该请求是否会被拆分或合并（若后续有连续读写请求），若发生拆分或合并，其大小会是多少？载荷单元大小具体为多少？

测试代码 #

#define MMIO_PHY_ADDR 0x20bf80000000
#define MMIO_SIZE 4096 // 4KB, that I want to map on the memory (Total BAR is 1GB but I used only 4KB for test)
#define MMIO_ACCESS_SIZE (MMIO_SIZE / sizeof(uint32_t)) // Number of 32-bit units

int main() {
    int fd = open("/dev/mem", O_RDWR | O_SYNC);
    if (fd < 0) {
        perror("Failed to open /dev/mem");
        return EXIT_FAILURE;
    }

    void* mmio_base = mmap(nullptr, MMIO_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, MMIO_PHY_ADDR);
    if (mmio_base == MAP_FAILED) {
        perror("Failed to mmap");
        close(fd);
        return EXIT_FAILURE;
    }
    
    ////////////////////////////////////////////////////////////
    printf("Memory mapped at virtual address %p.\n", mmio_base);

    uint32_t value = *((uint32_t*)mmio_base);
    std::cout << "Read value: 0x" << std::hex << value << std::endl;

    *((uint32_t*)mmio_base) = 0x12345678;
    std::cout << "Wrote 0x12345678 to the mapped memory" << std::endl;

    value = *((uint32_t*)mmio_base);
    std::cout << "Read value: 0x" << std::hex << value << std::endl;
    ////////////////////////////////////////////////////////////
}

环境与设备信息 #

运行环境：Ubuntu系统，Linux内核版本v5.4.0-172-generic
PCIe设备：自行配置Xilinx PCIe IP的Xilinx FPGA，lspci输出如下：

27:00.0 Multimedia video controller: Device ****:**** (rev ****)
    Subsystem: **** Device ****
    Physical Slot: 2
    Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 16
    NUMA node: 0
    Region 0: Memory at 20bfc0000000 (64-bit, prefetchable) [size=4K]
    Region 2: Memory at 20bf80000000 (64-bit, prefetchable) [size=1G]
    Capabilities: [80] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <256ns, L1 <2us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
        DevCtl: CorrErr- NonFatalErr- FatalErr+ UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
        LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s (ok), Width x8 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR-
             10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS-, TPHComp-, ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
             AtomicOpsCtl: ReqEn+
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
    Capabilities: [e0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [f8] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [100 v1] Vendor Specific Information: ID=**** Rev=**** Len=**** <?>
    Capabilities: [120 v1] Address Translation Service (ATS)
        ATSCap: Invalidate Queue Depth: 00
        ATSCtl: Enable-, Smallest Translation Unit: 00
    Capabilities: [200 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC+ UnsupReq+ ACSViol-
        UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout- AdvNonFatalErr+
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        AERCap: First Error Pointer: 0f, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [300 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
        LaneErrStat: LaneErr at lane: 0 5 6 7

核心解答 #

1. 单次32位MMIO请求的载荷大小 #

• 代码中对uint32_t的直接读写，对应**32位（4字节）**的MMIO请求，不会被拆分。x86架构下对齐的32位访问是原生支持的最小有效单元，且符合PCIe设备的处理要求。

2. 连续请求的拆分与合并规则 #

• 合并场景：对连续32位地址批量读写时，Linux内核和PCIe根复合体可能合并请求，合并后的最大载荷受两个参数限制：
  • 设备的MaxPayload Size：从lspci输出可知，设备配置为MaxPayload 256 bytes，这是设备能接收的最大单请求载荷。
  • CPU缓存行大小：x86架构通常为64字节，连续访问会优先按缓存行粒度合并，再适配PCIe的MaxPayload限制。
• 拆分场景：如果请求跨4KB页边界（你的MMIO是页对齐的，同页内连续访问不会触发），或者合并后的请求大小超过设备MaxPayload，就会被拆分成多个不超过256字节的请求。
• 注意：使用O_SYNC打开/dev/mem会强制读写直接提交到硬件，减少内核层面的合并概率，但PCIe根复合体仍可能做小范围优化合并。

3. 实际载荷单元的具体值 #

• 单次对齐32位访问：固定为4字节。
• 连续批量访问：最大合并载荷为256字节（受限于设备MaxPayload）；若CPU缓存行更小（如64字节），则会先按64字节合并，再发送到PCIe总线。

内容的提问来源于stack exchange，提问作者horse-master