如何将主机PCIe虚拟函数共享至Docker容器？

Since you're already familiar with SR-IOV in bare-metal and VM scenarios, getting Docker to access QAT VFs is totally achievable—you just need to use the right PCI passthrough method with Docker, leveraging the VFIO framework for secure device isolation. Here's a step-by-step breakdown:

Prerequisites #

First, confirm these are already set up (you likely have most of them done):

• SR-IOV is enabled on your host, with VFs properly exposed (your lspci output confirms 32 VFs for the QAT PF)
• IOMMU is enabled on the host (add intel_iommu=on to your GRUB kernel params and reboot—critical for PCI passthrough)
• Docker 17.06 or newer (supports modern device passthrough features)

Step 1: Bind QAT VFs to the VFIO-PCI Driver #

Before passing VFs to Docker, you need to unbind them from their native QAT VF driver and bind them to vfio-pci (the standard PCI passthrough driver):

1. Get the vendor and device ID of your QAT VF (replace 85:01.0 with your target VF's BDF):

   lspci -n -s 85:01.0
   

   You'll get output like 85:01.0 0b40: 8086:0443—note the pair 8086 0443 (vendor:device).

2. Load the vfio-pci driver if it's not already loaded:

   modprobe vfio-pci
   

3. Register the QAT VF ID with vfio-pci:

   echo 8086 0443 > /sys/bus/pci/drivers/vfio-pci/new_id
   

4. Unbind the VF from its current driver (e.g., the QAT VF driver you wrote):

   echo 0000:85:01.0 > /sys/bus/pci/devices/0000:85:01.0/driver/unbind
   

5. Bind the VF to vfio-pci:

   echo 0000:85:01.0 > /sys/bus/pci/drivers/vfio-pci/bind
   

Step 2: Identify the VFIO Group for the VF #

Each PCI device belongs to an IOMMU group, which you need to pass to Docker:

1. Find the group ID for your VF:

   readlink /sys/bus/pci/devices/0000:85:01.0/iommu_group | cut -d'/' -f5
   

   This will return a number (e.g., 25)—this is your VFIO group ID.

Step 3: Launch Docker Container with the VF #

Use Docker's --device flag to pass the VFIO group and control device to the container, along with necessary capabilities:

docker run -it --rm \
  --device /dev/vfio/<GROUP_ID> \
  --device /dev/vfio/vfio \
  --cap-add IPC_LOCK \
  --cap-add SYS_RAWIO \
  your-qat-container-image:tag

• Replace <GROUP_ID> with the number you found in Step 2
• /dev/vfio/vfio is the global VFIO control device, required for passthrough
• The --cap-add flags grant the container necessary permissions to interact with the PCI device

Step 4: Verify VF Access in the Container #

Once inside the container, run lspci—you should now see the QAT VF listed. Additionally, test your QAT driver to confirm it can initialize and use the VF for encryption/compression tasks.

Batch Passing Multiple VFs #

If you need to pass multiple VFs to a single container, repeat Steps 1-2 for each VF, then add multiple --device /dev/vfio/<GROUP_ID> flags to your docker run command. Most QAT VFs from the same PF will belong to the same IOMMU group, so you might only need to pass one group ID for all VFs (confirm with the readlink command for each VF).

Key Notes #

• Avoid passing the PF to containers: Keep the Physical Function (85:00.0) on the host—it manages the VFs and shouldn't be shared with containers.
• Container Driver Setup: Ensure your QAT VF driver is installed in the container image (or mounted in) to recognize and use the passed VF.
• Security: Using VFIO is far more secure than --privileged mode, as it isolates the VF to the container without granting full host access.

内容的提问来源于stack exchange，提问作者user3109016