| 1 | =============================== |
| 2 | IOMMUFD BACKEND usage with VFIO |
| 3 | =============================== |
| 4 | |
| 5 | (Same meaning for backend/container/BE) |
| 6 | |
| 7 | With the introduction of iommufd, the Linux kernel provides a generic |
| 8 | interface for user space drivers to propagate their DMA mappings to kernel |
| 9 | for assigned devices. While the legacy kernel interface is group-centric, |
| 10 | the new iommufd interface is device-centric, relying on device fd and iommufd. |
| 11 | |
| 12 | To support both interfaces in the QEMU VFIO device, introduce a base container |
| 13 | to abstract the common part of VFIO legacy and iommufd container. So that the |
| 14 | generic VFIO code can use either container. |
| 15 | |
| 16 | The base container implements generic functions such as memory_listener and |
| 17 | address space management whereas the derived container implements callbacks |
| 18 | specific to either legacy or iommufd. Each container has its own way to setup |
| 19 | secure context and dma management interface. The below diagram shows how it |
| 20 | looks like with both containers. |
| 21 | |
| 22 | :: |
| 23 | |
| 24 | VFIO AddressSpace/Memory |
| 25 | +-------+ +----------+ +-----+ +-----+ |
| 26 | | pci | | platform | | ap | | ccw | |
| 27 | +---+---+ +----+-----+ +--+--+ +--+--+ +----------------------+ |
| 28 | | | | | | AddressSpace | |
| 29 | | | | | +------------+---------+ |
| 30 | +---V-----------V-----------V--------V----+ / |
| 31 | | VFIOAddressSpace | <------------+ |
| 32 | | | | MemoryListener |
| 33 | | VFIOContainerBase list | |
| 34 | +-------+----------------------------+----+ |
| 35 | | | |
| 36 | | | |
| 37 | +-------V------+ +--------V----------+ |
| 38 | | iommufd | | vfio legacy | |
| 39 | | container | | container | |
| 40 | +-------+------+ +--------+----------+ |
| 41 | | | |
| 42 | | /dev/iommu | /dev/vfio/vfio |
| 43 | | /dev/vfio/devices/vfioX | /dev/vfio/$group_id |
| 44 | Userspace | | |
| 45 | ============+============================+=========================== |
| 46 | Kernel | device fd | |
| 47 | +---------------+ | group/container fd |
| 48 | | (BIND_IOMMUFD | | (SET_CONTAINER/SET_IOMMU) |
| 49 | | ATTACH_IOAS) | | device fd |
| 50 | | | | |
| 51 | | +-------V------------V-----------------+ |
| 52 | iommufd | | vfio | |
| 53 | (map/unmap | +---------+--------------------+-------+ |
| 54 | ioas_copy) | | | map/unmap |
| 55 | | | | |
| 56 | +------V------+ +-----V------+ +------V--------+ |
| 57 | | iommfd core | | device | | vfio iommu | |
| 58 | +-------------+ +------------+ +---------------+ |
| 59 | |
| 60 | * Secure Context setup |
| 61 | |
| 62 | - iommufd BE: uses device fd and iommufd to setup secure context |
| 63 | (bind_iommufd, attach_ioas) |
| 64 | - vfio legacy BE: uses group fd and container fd to setup secure context |
| 65 | (set_container, set_iommu) |
| 66 | |
| 67 | * Device access |
| 68 | |
| 69 | - iommufd BE: device fd is opened through ``/dev/vfio/devices/vfioX`` |
| 70 | - vfio legacy BE: device fd is retrieved from group fd ioctl |
| 71 | |
| 72 | * DMA Mapping flow |
| 73 | |
| 74 | 1. VFIOAddressSpace receives MemoryRegion add/del via MemoryListener |
| 75 | 2. VFIO populates DMA map/unmap via the container BEs |
| 76 | * iommufd BE: uses iommufd |
| 77 | * vfio legacy BE: uses container fd |
| 78 | |
| 79 | Example configuration |
| 80 | ===================== |
| 81 | |
| 82 | Step 1: configure the host device |
| 83 | --------------------------------- |
| 84 | |
| 85 | It's exactly same as the VFIO device with legacy VFIO container. |
| 86 | |
| 87 | Step 2: configure QEMU |
| 88 | ---------------------- |
| 89 | |
| 90 | Interactions with the ``/dev/iommu`` are abstracted by a new iommufd |
| 91 | object (compiled in with the ``CONFIG_IOMMUFD`` option). |
| 92 | |
| 93 | Any QEMU device (e.g. VFIO device) wishing to use ``/dev/iommu`` must |
| 94 | be linked with an iommufd object. It gets a new optional property |
| 95 | named iommufd which allows to pass an iommufd object. Take ``vfio-pci`` |
| 96 | device for example: |
| 97 | |
| 98 | .. code-block:: bash |
| 99 | |
| 100 | -object iommufd,id=iommufd0 |
| 101 | -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0 |
| 102 | |
| 103 | Note the ``/dev/iommu`` and VFIO cdev can be externally opened by a |
| 104 | management layer. In such a case the fd is passed, the fd supports a |
| 105 | string naming the fd or a number, for example: |
| 106 | |
| 107 | .. code-block:: bash |
| 108 | |
| 109 | -object iommufd,id=iommufd0,fd=22 |
| 110 | -device vfio-pci,iommufd=iommufd0,fd=23 |
| 111 | |
| 112 | If the ``fd`` property is not passed, the fd is opened by QEMU. |
| 113 | |
| 114 | If no ``iommufd`` object is passed to the ``vfio-pci`` device, iommufd |
| 115 | is not used and the user gets the behavior based on the legacy VFIO |
| 116 | container: |
| 117 | |
| 118 | .. code-block:: bash |
| 119 | |
| 120 | -device vfio-pci,host=0000:02:00.0 |
| 121 | |
| 122 | Supported platform |
| 123 | ================== |
| 124 | |
| 125 | Supports x86, Arm and s390x currently. |
| 126 | |
| 127 | Caveats |
| 128 | ======= |
| 129 | |
| 130 | P2P DMA |
| 131 | ------- |
| 132 | |
| 133 | PCI p2p DMA is unsupported as IOMMUFD doesn't support mapping hardware PCI |
| 134 | BAR region yet. Below warning shows for assigned PCI device, it's not a bug. |
| 135 | |
| 136 | .. code-block:: none |
| 137 | |
| 138 | qemu-system-x86_64: warning: IOMMU_IOAS_MAP failed: Bad address, PCI BAR? |
| 139 | qemu-system-x86_64: vfio_container_dma_map(0x560cb6cb1620, 0xe000000021000, 0x3000, 0x7f32ed55c000) = -14 (Bad address) |
| 140 | |
| 141 | FD passing with mdev |
| 142 | -------------------- |
| 143 | |
| 144 | ``vfio-pci`` device checks sysfsdev property to decide if backend is a mdev. |
| 145 | If FD passing is used, there is no way to know that and the mdev is treated |
| 146 | like a real PCI device. There is an error as below if user wants to enable |
| 147 | RAM discarding for mdev. |
| 148 | |
| 149 | .. code-block:: none |
| 150 | |
| 151 | qemu-system-x86_64: -device vfio-pci,iommufd=iommufd0,x-balloon-allowed=on,fd=9: vfio VFIO_FD9: x-balloon-allowed only potentially compatible with mdev devices |
| 152 | |
| 153 | ``vfio-ap`` and ``vfio-ccw`` devices don't have same issue as their backend |
| 154 | devices are always mdev and RAM discarding is force enabled. |
| 155 | |
| 156 | Usage with intel_iommu featuring fsts=on |
| 157 | ------------------------------------------ |
| 158 | |
| 159 | Only IOMMUFD backed VFIO device is supported when intel_iommu is configured |
| 160 | with fsts=on, for legacy container backed VFIO device, below error shows: |
| 161 | |
| 162 | .. code-block:: none |
| 163 | |
| 164 | qemu-system-x86_64: -device vfio-pci,host=0000:02:00.0: vfio 0000:02:00.0: Failed to set vIOMMU: Need IOMMUFD backend when fsts=on |
| 165 | |
| 166 | VFIO device under PCI bridge is unsupported, use PCIE bridge if necessary, |
| 167 | otherwise below error shows: |
| 168 | |
| 169 | .. code-block:: none |
| 170 | |
| 171 | qemu-system-x86_64: -device vfio-pci,host=0000:02:00.0,bus=bridge1,iommufd=iommufd0: vfio 0000:02:00.0: Failed to set vIOMMU: Host device downstream to a PCI bridge is unsupported when fsts=on |
| 172 | |
| 173 | If host IOMMU has ERRATA_772415_SPR17, running guest with "intel_iommu=on,sm_off" |
| 174 | is unsupported, kexec or reboot guest from "intel_iommu=on,sm_on" to |
| 175 | "intel_iommu=on,sm_off" is also unsupported. Configure scalable mode off as |
| 176 | below if it's not needed by guest: |
| 177 | |
| 178 | .. code-block:: bash |
| 179 | |
| 180 | -device intel-iommu,scalable-mode=off |