主机上virtio-net vDPA多VF配置时DMA地址冲突问题求助

我正在为自研NIC开发vDPA驱动，测试多VF的virtio-net vDPA场景时发现，内核vDPA框架为多个VF分配了相同的DMA地址，不同VF操作同一DMA地址（比如更新已用索引）会导致内核virtqueue工作异常。

操作步骤 #

1. 启用NIC的SR-IOV并创建4个VF，确认4个vdpa管理设备创建成功：

[root@localhost ~]# echo 4 > /sys/class/net/enp1s0np0/device/sriov_numvfs
[root@localhost ~]# vdpa mgmtdev show
pci/0000:01:08.0:
supported_classes net
pci/0000:01:08.1:
supported_classes net
pci/0000:01:08.2:
supported_classes net
pci/0000:01:08.3:
supported_classes net

2. 添加vdpa设备并绑定virtio_vdpa驱动，确认绑定成功：

[root@localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.0 name vdpa0
[root@localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.1 name vdpa1
[root@localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.2 name vdpa2
[root@localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.3 name vdpa3
[root@localhost ~]# modprobe virtio_vdpa
[root@localhost ~]# ls -l /sys/bus/vdpa/drivers/virtio_vdpa/
total 0
--w-------. 1 root root 4096 Nov 21 16:55 bind
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 module -> ../../../../module/virtio_vdpa
--w-------. 1 root root 4096 Nov 21 16:55 uevent
--w-------. 1 root root 4096 Nov 21 16:55 unbind
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 vdpa0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.0/vdpa0
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 vdpa1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.1/vdpa1
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 vdpa2 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.2/vdpa2
lrwxrwxrwx. 1 root root    0 Nov 21 16:55 vdpa3 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.3/vdpa3

3. 查看内核日志，可见4个VF的virtqueue分配了相同的DMA地址：

[Mon Nov 21 16:55:42 2022] virtio_net virtio0: devname virtio0 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio0: devname virtio0 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio1: devname virtio1 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio1: devname virtio1 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio2: devname virtio2 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio2: devname virtio2 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio0 enp1s0v0: renamed from eth5
[Mon Nov 21 16:55:42 2022] virtio_net virtio3: devname virtio3 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio3: devname virtio3 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] virtio_net virtio1 enp1s0v1: renamed from eth6
[Mon Nov 21 16:55:42 2022] virtio_net virtio2 enp1s0v2: renamed from eth7
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio3 enp1s0v3: renamed from eth5

内核版本与配置 #

使用内核版本5.15.15，编译时已启用所有vDPA相关选项：

[root@localhost linux-5.15.15]# cat .config | grep VDPA
CONFIG_VIRTIO_VDPA=m
CONFIG_VDPA=y
CONFIG_VDPA_SIM=m
CONFIG_VDPA_SIM_NET=m
CONFIG_VDPA_SIM_BLOCK=m
#CONFIG_VDPA_USER is not set
CONFIG_MLX5_VDPA=y
CONFIG_MLX5_VDPA_NET=m
CONFIG_VP_VDPA=m
CONFIG_VHOST_VDPA=m

补充测试 #

测试vhost-vdpa结合QEMU启动虚拟机的场景时，多VF工作正常。

我是否遗漏或误解了内核vDPA框架的某些内容？恳请提供建议。

内容的提问来源于stack exchange，提问作者kyle