| 1 | XIVE for sPAPR (pseries machines) |
| 2 | ================================= |
| 3 | |
| 4 | The POWER9 processor comes with a new interrupt controller |
| 5 | architecture, called XIVE as "eXternal Interrupt Virtualization |
| 6 | Engine". It supports a larger number of interrupt sources and offers |
| 7 | virtualization features which enables the HW to deliver interrupts |
| 8 | directly to virtual processors without hypervisor assistance. |
| 9 | |
| 10 | A QEMU ``pseries`` machine (which is PAPR compliant) using POWER9 |
| 11 | processors can run under two interrupt modes: |
| 12 | |
| 13 | - *Legacy Compatibility Mode* |
| 14 | |
| 15 | the hypervisor provides identical interfaces and similar |
| 16 | functionality to PAPR+ Version 2.7. This is the default mode |
| 17 | |
| 18 | It is also referred as *XICS* in QEMU. |
| 19 | |
| 20 | - *XIVE native exploitation mode* |
| 21 | |
| 22 | the hypervisor provides new interfaces to manage the XIVE control |
| 23 | structures, and provides direct control for interrupt management |
| 24 | through MMIO pages. |
| 25 | |
| 26 | Which interrupt modes can be used by the machine is negotiated with |
| 27 | the guest O/S during the Client Architecture Support negotiation |
| 28 | sequence. The two modes are mutually exclusive. |
| 29 | |
| 30 | Both interrupt mode share the same IRQ number space. See below for the |
| 31 | layout. |
| 32 | |
| 33 | CAS Negotiation |
| 34 | --------------- |
| 35 | |
| 36 | QEMU advertises the supported interrupt modes in the device tree |
| 37 | property ``ibm,arch-vec-5-platform-support`` in byte 23 and the OS |
| 38 | Selection for XIVE is indicated in the ``ibm,architecture-vec-5`` |
| 39 | property byte 23. |
| 40 | |
| 41 | The interrupt modes supported by the machine depend on the CPU type |
| 42 | (POWER9 is required for XIVE) but also on the machine property |
| 43 | ``ic-mode`` which can be set on the command line. It can take the |
| 44 | following values: ``xics``, ``xive``, and ``dual`` which is the |
| 45 | default mode. ``dual`` means that both modes XICS **and** XIVE are |
| 46 | supported and if the guest OS supports XIVE, this mode will be |
| 47 | selected. |
| 48 | |
| 49 | The chosen interrupt mode is activated after a reconfiguration done |
| 50 | in a machine reset. |
| 51 | |
| 52 | KVM negotiation |
| 53 | --------------- |
| 54 | |
| 55 | When the guest starts under KVM, the capabilities of the host kernel |
| 56 | and QEMU are also negotiated. Depending on the version of the host |
| 57 | kernel, KVM will advertise the XIVE capability to QEMU or not. |
| 58 | |
| 59 | Nevertheless, the available interrupt modes in the machine should not |
| 60 | depend on the XIVE KVM capability of the host. On older kernels |
| 61 | without XIVE KVM support, QEMU will use the emulated XIVE device as a |
| 62 | fallback and on newer kernels (>=5.2), the KVM XIVE device. |
| 63 | |
| 64 | XIVE native exploitation mode is not supported for KVM nested guests, |
| 65 | VMs running under a L1 hypervisor (KVM on pSeries). In that case, the |
| 66 | hypervisor will not advertise the KVM capability and QEMU will use the |
| 67 | emulated XIVE device, same as for older versions of KVM. |
| 68 | |
| 69 | As a final refinement, the user can also switch the use of the KVM |
| 70 | device with the machine option ``kernel_irqchip``. |
| 71 | |
| 72 | |
| 73 | XIVE support in KVM |
| 74 | ~~~~~~~~~~~~~~~~~~~ |
| 75 | |
| 76 | For guest OSes supporting XIVE, the resulting interrupt modes on host |
| 77 | kernels with XIVE KVM support are the following: |
| 78 | |
| 79 | ============== ============= ============= ================ |
| 80 | ic-mode kernel_irqchip |
| 81 | -------------- ---------------------------------------------- |
| 82 | / allowed off on |
| 83 | (default) |
| 84 | ============== ============= ============= ================ |
| 85 | dual (default) XIVE KVM XIVE emul. XIVE KVM |
| 86 | xive XIVE KVM XIVE emul. XIVE KVM |
| 87 | xics XICS KVM XICS emul. XICS KVM |
| 88 | ============== ============= ============= ================ |
| 89 | |
| 90 | For legacy guest OSes without XIVE support, the resulting interrupt |
| 91 | modes are the following: |
| 92 | |
| 93 | ============== ============= ============= ================ |
| 94 | ic-mode kernel_irqchip |
| 95 | -------------- ---------------------------------------------- |
| 96 | / allowed off on |
| 97 | (default) |
| 98 | ============== ============= ============= ================ |
| 99 | dual (default) XICS KVM XICS emul. XICS KVM |
| 100 | xive QEMU error(3) QEMU error(3) QEMU error(3) |
| 101 | xics XICS KVM XICS emul. XICS KVM |
| 102 | ============== ============= ============= ================ |
| 103 | |
| 104 | (3) QEMU fails at CAS with ``Guest requested unavailable interrupt |
| 105 | mode (XICS), either don't set the ic-mode machine property or try |
| 106 | ic-mode=xics or ic-mode=dual`` |
| 107 | |
| 108 | |
| 109 | No XIVE support in KVM |
| 110 | ~~~~~~~~~~~~~~~~~~~~~~ |
| 111 | |
| 112 | For guest OSes supporting XIVE, the resulting interrupt modes on host |
| 113 | kernels without XIVE KVM support are the following: |
| 114 | |
| 115 | ============== ============= ============= ================ |
| 116 | ic-mode kernel_irqchip |
| 117 | -------------- ---------------------------------------------- |
| 118 | / allowed off on |
| 119 | (default) |
| 120 | ============== ============= ============= ================ |
| 121 | dual (default) XIVE emul.(1) XIVE emul. QEMU error (2) |
| 122 | xive XIVE emul.(1) XIVE emul. QEMU error (2) |
| 123 | xics XICS KVM XICS emul. XICS KVM |
| 124 | ============== ============= ============= ================ |
| 125 | |
| 126 | |
| 127 | (1) QEMU warns with ``warning: kernel_irqchip requested but unavailable: |
| 128 | IRQ_XIVE capability must be present for KVM`` |
| 129 | In some cases (old host kernels or KVM nested guests), one may hit a |
| 130 | QEMU/KVM incompatibility due to device destruction in reset. QEMU fails |
| 131 | with ``KVM is incompatible with ic-mode=dual,kernel-irqchip=on`` |
| 132 | (2) QEMU fails with ``kernel_irqchip requested but unavailable: |
| 133 | IRQ_XIVE capability must be present for KVM`` |
| 134 | |
| 135 | |
| 136 | For legacy guest OSes without XIVE support, the resulting interrupt |
| 137 | modes are the following: |
| 138 | |
| 139 | ============== ============= ============= ================ |
| 140 | ic-mode kernel_irqchip |
| 141 | -------------- ---------------------------------------------- |
| 142 | / allowed off on |
| 143 | (default) |
| 144 | ============== ============= ============= ================ |
| 145 | dual (default) QEMU error(4) XICS emul. QEMU error(4) |
| 146 | xive QEMU error(3) QEMU error(3) QEMU error(3) |
| 147 | xics XICS KVM XICS emul. XICS KVM |
| 148 | ============== ============= ============= ================ |
| 149 | |
| 150 | (3) QEMU fails at CAS with ``Guest requested unavailable interrupt |
| 151 | mode (XICS), either don't set the ic-mode machine property or try |
| 152 | ic-mode=xics or ic-mode=dual`` |
| 153 | (4) QEMU/KVM incompatibility due to device destruction in reset. QEMU fails |
| 154 | with ``KVM is incompatible with ic-mode=dual,kernel-irqchip=on`` |
| 155 | |
| 156 | |
| 157 | XIVE Device tree properties |
| 158 | --------------------------- |
| 159 | |
| 160 | The properties for the PAPR interrupt controller node when the *XIVE |
| 161 | native exploitation mode* is selected should contain: |
| 162 | |
| 163 | - ``device_type`` |
| 164 | |
| 165 | value should be "power-ivpe". |
| 166 | |
| 167 | - ``compatible`` |
| 168 | |
| 169 | value should be "ibm,power-ivpe". |
| 170 | |
| 171 | - ``reg`` |
| 172 | |
| 173 | contains the base address and size of the thread interrupt |
| 174 | managnement areas (TIMA), for the User level and for the Guest OS |
| 175 | level. Only the Guest OS level is taken into account today. |
| 176 | |
| 177 | - ``ibm,xive-eq-sizes`` |
| 178 | |
| 179 | the size of the event queues. One cell per size supported, contains |
| 180 | log2 of size, in ascending order. |
| 181 | |
| 182 | - ``ibm,xive-lisn-ranges`` |
| 183 | |
| 184 | the IRQ interrupt number ranges assigned to the guest for the IPIs. |
| 185 | |
| 186 | The root node also exports : |
| 187 | |
| 188 | - ``ibm,plat-res-int-priorities`` |
| 189 | |
| 190 | contains a list of priorities that the hypervisor has reserved for |
| 191 | its own use. |
| 192 | |
| 193 | IRQ number space |
| 194 | ---------------- |
| 195 | |
| 196 | IRQ Number space of the ``pseries`` machine is 8K wide and is the same |
| 197 | for both interrupt mode. The different ranges are defined as follow : |
| 198 | |
| 199 | - ``0x0000 .. 0x0FFF`` 4K CPU IPIs (only used under XIVE) |
| 200 | - ``0x1000 .. 0x1000`` 1 EPOW |
| 201 | - ``0x1001 .. 0x1001`` 1 HOTPLUG |
| 202 | - ``0x1002 .. 0x10FF`` unused |
| 203 | - ``0x1100 .. 0x11FF`` 256 VIO devices |
| 204 | - ``0x1200 .. 0x127F`` 32x4 LSIs for PHB devices |
| 205 | - ``0x1280 .. 0x12FF`` unused |
| 206 | - ``0x1300 .. 0x1FFF`` PHB MSIs (dynamically allocated) |
| 207 | |
| 208 | Monitoring XIVE |
| 209 | --------------- |
| 210 | |
| 211 | The state of the XIVE interrupt controller can be queried through the |
| 212 | monitor commands ``info pic``. The output comes in two parts. |
| 213 | |
| 214 | First, the state of the thread interrupt context registers is dumped |
| 215 | for each CPU : |
| 216 | |
| 217 | :: |
| 218 | |
| 219 | (qemu) info pic |
| 220 | CPU[0000]: QW NSR CPPR IPB LSMFB ACK# INC AGE PIPR W2 |
| 221 | CPU[0000]: USER 00 00 00 00 00 00 00 00 00000000 |
| 222 | CPU[0000]: OS 00 ff 00 00 ff 00 ff ff 80000400 |
| 223 | CPU[0000]: POOL 00 00 00 00 00 00 00 00 00000000 |
| 224 | CPU[0000]: PHYS 00 00 00 00 00 00 00 ff 00000000 |
| 225 | ... |
| 226 | |
| 227 | In the case of a ``pseries`` machine, QEMU acts as the hypervisor and only |
| 228 | the O/S and USER register rings make sense. ``W2`` contains the vCPU CAM |
| 229 | line which is set to the VP identifier. |
| 230 | |
| 231 | Then comes the routing information which aggregates the EAS and the |
| 232 | END configuration: |
| 233 | |
| 234 | :: |
| 235 | |
| 236 | ... |
| 237 | LISN PQ EISN CPU/PRIO EQ |
| 238 | 00000000 MSI -- 00000010 0/6 380/16384 @1fe3e0000 ^1 [ 80000010 ... ] |
| 239 | 00000001 MSI -- 00000010 1/6 305/16384 @1fc230000 ^1 [ 80000010 ... ] |
| 240 | 00000002 MSI -- 00000010 2/6 220/16384 @1fc2f0000 ^1 [ 80000010 ... ] |
| 241 | 00000003 MSI -- 00000010 3/6 201/16384 @1fc390000 ^1 [ 80000010 ... ] |
| 242 | 00000004 MSI -Q M 00000000 |
| 243 | 00000005 MSI -Q M 00000000 |
| 244 | 00000006 MSI -Q M 00000000 |
| 245 | 00000007 MSI -Q M 00000000 |
| 246 | 00001000 MSI -- 00000012 0/6 380/16384 @1fe3e0000 ^1 [ 80000010 ... ] |
| 247 | 00001001 MSI -- 00000013 0/6 380/16384 @1fe3e0000 ^1 [ 80000010 ... ] |
| 248 | 00001100 MSI -- 00000100 1/6 305/16384 @1fc230000 ^1 [ 80000010 ... ] |
| 249 | 00001101 MSI -Q M 00000000 |
| 250 | 00001200 LSI -Q M 00000000 |
| 251 | 00001201 LSI -Q M 00000000 |
| 252 | 00001202 LSI -Q M 00000000 |
| 253 | 00001203 LSI -Q M 00000000 |
| 254 | 00001300 MSI -- 00000102 1/6 305/16384 @1fc230000 ^1 [ 80000010 ... ] |
| 255 | 00001301 MSI -- 00000103 2/6 220/16384 @1fc2f0000 ^1 [ 80000010 ... ] |
| 256 | 00001302 MSI -- 00000104 3/6 201/16384 @1fc390000 ^1 [ 80000010 ... ] |
| 257 | |
| 258 | The source information and configuration: |
| 259 | |
| 260 | - The ``LISN`` column outputs the interrupt number of the source in |
| 261 | range ``[ 0x0 ... 0x1FFF ]`` and its type : ``MSI`` or ``LSI`` |
| 262 | - The ``PQ`` column reflects the state of the PQ bits of the source : |
| 263 | |
| 264 | - ``--`` source is ready to take events |
| 265 | - ``P-`` an event was sent and an EOI is PENDING |
| 266 | - ``PQ`` an event was QUEUED |
| 267 | - ``-Q`` source is OFF |
| 268 | |
| 269 | a ``M`` indicates that source is *MASKED* at the EAS level, |
| 270 | |
| 271 | The targeting configuration : |
| 272 | |
| 273 | - The ``EISN`` column is the event data that will be queued in the event |
| 274 | queue of the O/S. |
| 275 | - The ``CPU/PRIO`` column is the tuple defining the CPU number and |
| 276 | priority queue serving the source. |
| 277 | - The ``EQ`` column outputs : |
| 278 | |
| 279 | - the current index of the event queue/ the max number of entries |
| 280 | - the O/S event queue address |
| 281 | - the toggle bit |
| 282 | - the last entries that were pushed in the event queue. |