| 1 | Intel Graphics Device (IGD) assignment with vfio-pci |
| 2 | ==================================================== |
| 3 | |
| 4 | Using vfio-pci, we can passthrough Intel Graphics Device (IGD) to guest, either |
| 5 | serve as primary and exclusive graphics adapter, or used in combination with an |
| 6 | emulated primary graphics device, depending on the config and guest driver |
| 7 | support. However, IGD devices are not "clean" PCI devices, they use extra |
| 8 | memory regions other than BARs. Special handling is required to make them work |
| 9 | properly, including: |
| 10 | |
| 11 | * OpRegion for accessing Virtual BIOS Table (VBT) that contains display output |
| 12 | information. |
| 13 | * Data Stolen Memory (DSM) region used as VRAM at early stage (BIOS/UEFI) |
| 14 | |
| 15 | Certain guest software also depends on following conditions to work: |
| 16 | (*-Required by) |
| 17 | |
| 18 | | Condition | Linux | Windows | VBIOS | EFI GOP | |
| 19 | |---------------------------------------------|-------|---------|-------|---------| |
| 20 | | #1 IGD has a valid OpRegion containing VBT | * ^1 | * | * | * | |
| 21 | | #2 VID/DID of LPC bridge at 00:1f.0 matches | | | * | * | |
| 22 | | #3 IGD is assigned to BDF 00:02.0 | | | * | * | |
| 23 | | #4 IGD has VGA controller device class | | | * | * | |
| 24 | | #5 Host's VGA ranges are mapped to IGD | | | * | | |
| 25 | | #6 Guest has valid VBIOS or UEFI Option ROM | | | * | * | |
| 26 | |
| 27 | ^1 Though i915 driver is able to mock a OpRegion, it is still recommended to |
| 28 | use the VBT copied from host OpRegion to prevent incorrect configuration. |
| 29 | |
| 30 | For #1, the "x-igd-opregion=on" option exposes a copy of host IGD OpRegion to |
| 31 | guest via fw_cfg, where guest firmware can set up guest OpRegion with it. |
| 32 | |
| 33 | For #2, "x-igd-lpc=on" option copies the IDs of host LPC bridge and host bridge |
| 34 | to guest. Currently this is only supported on i440fx machines as there is |
| 35 | already an ICH9 LPC bridge present on q35 machines, overwriting its IDs may |
| 36 | lead to unexpected behavior. |
| 37 | |
| 38 | For #3, "addr=2.0" assigns IGD to 00:02.0. |
| 39 | |
| 40 | For #4, the primary display must be set to IGD in host BIOS. |
| 41 | |
| 42 | For #5, "x-vga=on" enables guest access to standard VGA IO/MMIO ranges. |
| 43 | |
| 44 | For #6, ROM either provided via the ROM BAR or romfile= option is needed, this |
| 45 | Intel document [1] shows how to dump VBIOS to file. For UEFI Option ROM, see |
| 46 | "Guest firmware" section. |
| 47 | |
| 48 | QEMU also provides a "Legacy" mode that implicitly enables full functionality |
| 49 | on IGD, it is automatically enabled when |
| 50 | * IGD generation is 6 to 9 (Sandy Bridge to Comet Lake) |
| 51 | * IGD claims VGA cycles on host (IGD is VGA controller on host) |
| 52 | * Machine type is i440fx |
| 53 | * IGD is assigned to guest BDF 00:02.0 |
| 54 | * ROM BAR or romfile is present |
| 55 | |
| 56 | In "Legacy" mode, QEMU will automatically setup OpRegion, LPC bridge IDs and |
| 57 | VGA range access, which is equivalent to: |
| 58 | x-igd-opregion=on,x-igd-lpc=on,x-vga=on |
| 59 | |
| 60 | By default, "Legacy" mode won't fail, it continues on error. User can set |
| 61 | "x-igd-legacy-mode=on" to force enabling legacy mode, this also checks if the |
| 62 | conditions above for legacy mode is met, and if any error occurs, QEMU will |
| 63 | fail immediately. Users can also set "x-igd-legacy-mode=off" to disable legacy |
| 64 | mode. |
| 65 | |
| 66 | In legacy mode, as the guest VGA ranges are assigned to IGD device, all other |
| 67 | graphics devices should be removed, this can be done using "-nographic" or |
| 68 | "-vga none" or "-nodefaults", along with adding the device using vfio-pci. |
| 69 | |
| 70 | For either mode, depending on the host kernel, the i915 driver in the host |
| 71 | may generate faults and errors upon re-binding to an IGD device after it |
| 72 | has been assigned to a VM. It's therefore generally recommended to prevent |
| 73 | such driver binding unless the host driver is known to work well for this. |
| 74 | There are numerous ways to do this, i915 can be blacklisted on the host, |
| 75 | the driver_override option can be used to ensure that only vfio-pci can bind |
| 76 | to the device on the host[2], virsh nodedev-detach can be used to bind the |
| 77 | device to vfio drivers and then managed='no' set in the VM xml to prevent |
| 78 | re-binding to i915, etc. Also note that IGD is also typically the primary |
| 79 | graphics in the host and special options may be required beyond simply |
| 80 | blacklisting i915 or using pci-stub/vfio-pci to take ownership of IGD as a |
| 81 | PCI class device. Lower level drivers exist that may still claim the device. |
| 82 | It may therefore be necessary to use kernel boot options video=vesafb:off or |
| 83 | video=efifb:off (depending on host BIOS/UEFI) or these can be combined to |
| 84 | a catch-all, video=vesafb:off,efifb:off. Error messages such as: |
| 85 | |
| 86 | Failed to mmap 0000:00:02.0 BAR <>. Performance may be slow |
| 87 | |
| 88 | are a good indicator that such a problem exists. The host files /proc/iomem |
| 89 | and /proc/ioports are often useful for identifying drivers consuming ranges |
| 90 | of the device to cause such conflicts. |
| 91 | |
| 92 | Additionally, IGD device are known to generate small numbers of DMAR faults |
| 93 | when initially assigned. It is believed that this is simply the IGD attempting |
| 94 | to access the reserved GTT space after reset, which it no longer has access to |
| 95 | when accessed from userspace. So long as the DMAR faults are small in number |
| 96 | and most importantly, not ongoing, these are not an indication of an error. |
| 97 | |
| 98 | Additionally++, analog VGA output (as opposed to digital outputs like HDMI, |
| 99 | DVI, or DisplayPort) may be unsupported in some use cases. In the author's |
| 100 | experience, even DP to VGA adapters can be troublesome while adapters between |
| 101 | digital formats work well. |
| 102 | |
| 103 | |
| 104 | Options |
| 105 | ======= |
| 106 | * x-igd-opregion=[*on*|off] |
| 107 | Copy host IGD OpRegion and expose it to guest with fw_cfg |
| 108 | |
| 109 | * x-igd-lpc=[on|*off*] |
| 110 | Creates a dummy LPC bridge at 00:1f:0 with host VID/DID (i440fx only) |
| 111 | |
| 112 | * x-igd-legacy-mode=[on|off|*auto*] |
| 113 | Enable/Disable legacy mode |
| 114 | |
| 115 | * x-igd-gms=[hex, default 0] |
| 116 | Overriding DSM region size in GGC register, 0 means uses host value. |
| 117 | Use this only when the DSM size cannot be changed through the |
| 118 | 'DVMT Pre-Allocated' option in host BIOS. |
| 119 | |
| 120 | |
| 121 | Examples |
| 122 | ======== |
| 123 | * Adding IGD with automatically legacy mode support |
| 124 | -device vfio-pci,host=00:02.0,id=hostdev0,addr=2.0 |
| 125 | |
| 126 | * Adding IGD with OpRegion and LPC ID hack, but without VGA ranges |
| 127 | (For UEFI guests) |
| 128 | -device vfio-pci,host=00:02.0,id=hostdev0,addr=2.0,x-igd-legacy-mode=off,x-igd-lpc=on,romfile=efi_oprom.rom |
| 129 | |
| 130 | |
| 131 | Guest firmware |
| 132 | ============== |
| 133 | Guest firmware is responsible for setting up OpRegion and Base of Data Stolen |
| 134 | Memory (BDSM) in guest address space. IGD passthrough support imposes two |
| 135 | fw_cfg requirements on the VM firmware: |
| 136 | |
| 137 | 1) "etc/igd-opregion" |
| 138 | |
| 139 | This fw_cfg file exposes the OpRegion for the IGD device. A reserved |
| 140 | region should be created below 4GB (recommended 4KB alignment), sized |
| 141 | sufficient for the fw_cfg file size, and the content of this file copied |
| 142 | to it. The dword based address of this reserved memory region must also |
| 143 | be written to the ASLS register at offset 0xFC on the IGD device. It is |
| 144 | recommended that firmware should make use of this fw_cfg entry for any |
| 145 | PCI class VGA device with Intel vendor ID. Multiple of such devices |
| 146 | within a VM is undefined. |
| 147 | |
| 148 | 2) "etc/igd-bdsm-size" |
| 149 | |
| 150 | This fw_cfg file contains an 8-byte, little endian integer indicating |
| 151 | the size of the reserved memory region required for IGD stolen memory. |
| 152 | Firmware must allocate a reserved memory below 4GB with required 1MB |
| 153 | alignment equal to this size. Additionally the base address of this |
| 154 | reserved region must be written to the dword BDSM register in PCI config |
| 155 | space of the IGD device at offset 0x5C (or 0xC0 for Gen 11+ devices using |
| 156 | 64-bit BDSM). As this support is related to running the IGD ROM, which |
| 157 | has other dependencies on the device appearing at guest address 00:02.0, |
| 158 | it's expected that this fw_cfg file is only relevant to a single PCI |
| 159 | class VGA device with Intel vendor ID, appearing at PCI bus address 00:02.0. |
| 160 | |
| 161 | Starting from Meteor Lake, IGD devices access stolen memory via its MMIO |
| 162 | BAR2 (LMEMBAR) and removed the BDSM register in config space. There is |
| 163 | no need for guest firmware to allocate data stolen memory in guest address |
| 164 | space and write it to BDSM register. Value of this fw_cfg file is 0 in |
| 165 | such case. |
| 166 | |
| 167 | Upstream Seabios has OpRegion and BDSM (pre-Gen11 device only) support. |
| 168 | However, the support is not accepted by upstream EDK2/OVMF. A recommended |
| 169 | solution is to create a virtual OpRom with following DXE drivers: |
| 170 | |
| 171 | * IgdAssignmentDxe: Set up OpRegion and BDSM according to fw_cfg (must) |
| 172 | * IntelGopDriver: Closed-source Intel GOP driver |
| 173 | * PlatformGopPolicy: Protocol required by IntelGopDriver |
| 174 | |
| 175 | IntelGopDriver and PlatformGopPolicy is only required when enabling GOP on IGD. |
| 176 | |
| 177 | The original IgdAssignmentDxe can be found at [3]. A Intel maintained version |
| 178 | with PlatformGopPolicy for industrial computing is at [4]. There is also an |
| 179 | unofficially maintained version with newer Gen11+ device support at [5]. |
| 180 | You need to build them with EDK2. |
| 181 | |
| 182 | For the IntelGopDriver, Intel never released it to public. You may contact |
| 183 | Intel support to get one as [4] said, if you are an Intel Premier Support |
| 184 | customer, or you can try extracting it from your host firmware using |
| 185 | "UEFI BIOS Updater"[6]. |
| 186 | |
| 187 | Once you got all the required DXE drivers, a Option ROM can be generated with |
| 188 | EfiRom utility in EDK2, using |
| 189 | EfiRom -f 0x8086 -i <Device ID of your IGD> -o output.rom \ |
| 190 | -e IgdAssignmentDxe.efi PlatformGOPPolicy.efi IntelGopDriver.efi |
| 191 | |
| 192 | |
| 193 | Known issues |
| 194 | ============ |
| 195 | When using OVMF as guest firmware, you may encounter the following warning: |
| 196 | warning: vfio_container_dma_map(0x55fab36ce610, 0x380010000000, 0x108000, 0x7fd336000000) = -22 (Invalid argument) |
| 197 | |
| 198 | Solution: |
| 199 | Set the host physical address bits to IOMMU address width using |
| 200 | -cpu host,host-phys-bits-limit=<IOMMU address width> |
| 201 | Or in libvirt XML with |
| 202 | <cpu> |
| 203 | <maxphysaddr mode='passthrough' limit='<IOMMU address width>'/> |
| 204 | </cpu> |
| 205 | The IOMMU address width can be determined with |
| 206 | echo $(( ((0x$(cat /sys/devices/virtual/iommu/dmar0/intel-iommu/cap) & 0x3F0000) >> 16) + 1 )) |
| 207 | Refer https://edk2.groups.io/g/devel/topic/patch_v1/102359124 for more details |
| 208 | |
| 209 | |
| 210 | Memory View |
| 211 | =========== |
| 212 | IGD has it own address space. To use system RAM as VRAM, a single-level page |
| 213 | table named Global Graphics Translation Table (GTT) is used for the address |
| 214 | translation. Each page table entry points a 4KB page. Illustration below shows |
| 215 | the translation flow on IGD with 64-bit GTT PTEs. |
| 216 | |
| 217 | (PTE_SIZE == 8) +-------------+---+ |
| 218 | | Address | V | V: Valid Bit |
| 219 | +-------------+---+ |
| 220 | | ... | | |
| 221 | IGD:0x01ae9010 0xd740| 0x70ffc000 | 1 | Mem:0x42ba3e010^ |
| 222 | -----------------------> 0xd748| 0x42ba3e000 | 1 +------------------> |
| 223 | (addr >> 12) * PTE_SIZE 0xd750| 0x42ba3f000 | 1 | |
| 224 | | ... | | |
| 225 | +-------------+---+ |
| 226 | ^ The address may be remapped by IOMMU |
| 227 | |
| 228 | The memory region store GTT is called GTT Stolen Memory (GSM) it is located |
| 229 | right below the Data Stolen Memory (DSM). Accessing this region directly is |
| 230 | not allowed, any access will immediately freeze the whole system. The only way |
| 231 | to access it is through the second half of MMIO BAR0. |
| 232 | |
| 233 | The Data Stolen Memory is reserved by firmware, and acts as the VRAM in pre-OS |
| 234 | environments. In QEMU, guest firmware (Seabios/OVMF) is responsible for |
| 235 | reserving a continuous region and program its base address to BDSM register, |
| 236 | then let VBIOS/GOP driver initializing this region. Illustration below shows |
| 237 | how DSM is mapped. |
| 238 | |
| 239 | IGD Addr Space Host Addr Space Guest Addr Space |
| 240 | +-------------+ +-------------+ +-------------+ |
| 241 | | | | | | | |
| 242 | | | | | | | |
| 243 | | | +-------------+ +-------------+ |
| 244 | | | | Data Stolen | | Data Stolen | |
| 245 | | | | (Guest) | | (Guest) | |
| 246 | | | +------------>+-------------+<------->+-------------+<--Guest BDSM |
| 247 | | | | Passthrough | | EPT | | Emulated by QEMU |
| 248 | DSMSIZE+-------------+ | with IOMMU | | Mapping | | Programmed by guest FW |
| 249 | | | | | | | | |
| 250 | | | | | | | | |
| 251 | 0+-------------+--+ | | | | |
| 252 | | +-------------+ | | |
| 253 | | | Data Stolen | +-------------+ |
| 254 | | | (Host) | |
| 255 | +------------>+-------------+<--Host BDSM |
| 256 | Non- | | "real" one in HW |
| 257 | Passthrough | | Programmed by host FW |
| 258 | +-------------+ |
| 259 | |
| 260 | Footnotes |
| 261 | ========= |
| 262 | [1] https://www.intel.com/content/www/us/en/docs/graphics-for-linux/developer-reference/1-0/dump-video-bios.html |
| 263 | [2] # echo "vfio-pci" > /sys/bus/pci/devices/0000:00:02.0/driver_override |
| 264 | [3] https://web.archive.org/web/20240827012422/https://bugzilla.tianocore.org/show_bug.cgi?id=935 |
| 265 | Tianocore bugzilla was down since Jan 2025 :( |
| 266 | [4] https://eci.intel.com/docs/3.3/components/kvm-hypervisor.html, Patch 0001-0004 |
| 267 | [5] https://github.com/tomitamoeko/VfioIgdPkg |
| 268 | [6] https://winraid.level1techs.com/t/tool-guide-news-uefi-bios-updater-ubu/30357 |