| 1 | .. _cpu-topology-s390x: |
| 2 | |
| 3 | CPU topology on s390x |
| 4 | ===================== |
| 5 | |
| 6 | Since QEMU 8.2, CPU topology on s390x provides up to 3 levels of |
| 7 | topology containers: drawers, books and sockets. They define a |
| 8 | tree-shaped hierarchy. |
| 9 | |
| 10 | The socket container has one or more CPU entries. |
| 11 | Each of these CPU entries consists of a bitmap and three CPU attributes: |
| 12 | |
| 13 | - CPU type |
| 14 | - entitlement |
| 15 | - dedication |
| 16 | |
| 17 | Each bit set in the bitmap correspond to a core-id of a vCPU with matching |
| 18 | attributes. |
| 19 | |
| 20 | This documentation provides general information on S390 CPU topology, |
| 21 | how to enable it and explains the new CPU attributes. |
| 22 | For information on how to modify the S390 CPU topology and how to |
| 23 | monitor polarization changes, see ``docs/devel/s390-cpu-topology.rst``. |
| 24 | |
| 25 | Prerequisites |
| 26 | ------------- |
| 27 | |
| 28 | To use the CPU topology, you currently need to choose the KVM accelerator. |
| 29 | See :ref:`Accelerators` for more details about accelerators and how to select them. |
| 30 | |
| 31 | The s390x host needs to use a Linux kernel v6.0 or newer (which provides the so-called |
| 32 | ``KVM_CAP_S390_CPU_TOPOLOGY`` capability that allows QEMU to signal the |
| 33 | CPU topology facility via the so-called STFLE bit 11 to the VM). |
| 34 | |
| 35 | Enabling CPU topology |
| 36 | --------------------- |
| 37 | |
| 38 | Currently, CPU topology is enabled by default only in the "host" CPU model. |
| 39 | |
| 40 | Enabling CPU topology in another CPU model is done by setting the CPU flag |
| 41 | ``ctop`` to ``on`` as in: |
| 42 | |
| 43 | .. code-block:: bash |
| 44 | |
| 45 | -cpu gen16b,ctop=on |
| 46 | |
| 47 | Having the topology disabled by default allows migration between |
| 48 | old and new QEMU without adding new flags. |
| 49 | |
| 50 | Default topology usage |
| 51 | ---------------------- |
| 52 | |
| 53 | The CPU topology can be specified on the QEMU command line |
| 54 | with the ``-smp`` or the ``-device`` QEMU command arguments. |
| 55 | |
| 56 | Note also that since 7.2 threads are no longer supported in the topology |
| 57 | and the ``-smp`` command line argument accepts only ``threads=1``. |
| 58 | |
| 59 | If none of the containers attributes (drawers, books, sockets) are |
| 60 | specified for the ``-smp`` flag, the number of these containers |
| 61 | is 1. |
| 62 | |
| 63 | Thus the following two options will result in the same topology: |
| 64 | |
| 65 | .. code-block:: bash |
| 66 | |
| 67 | -smp cpus=5,drawer=1,books=1,sockets=8,cores=4,maxcpus=32 |
| 68 | |
| 69 | and |
| 70 | |
| 71 | .. code-block:: bash |
| 72 | |
| 73 | -smp cpus=5,sockets=8,cores=4,maxcpus=32 |
| 74 | |
| 75 | When a CPU is defined by the ``-smp`` command argument, its position |
| 76 | inside the topology is calculated by adding the CPUs to the topology |
| 77 | based on the core-id starting with core-0 at position 0 of socket-0, |
| 78 | book-0, drawer-0 and filling all CPUs of socket-0 before filling socket-1 |
| 79 | of book-0 and so on up to the last socket of the last book of the last |
| 80 | drawer. |
| 81 | |
| 82 | When a CPU is defined by the ``-device`` command argument, the |
| 83 | tree topology attributes must all be defined or all not defined. |
| 84 | |
| 85 | .. code-block:: bash |
| 86 | |
| 87 | -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 |
| 88 | |
| 89 | or |
| 90 | |
| 91 | .. code-block:: bash |
| 92 | |
| 93 | -device gen16b-s390x-cpu,core-id=1,dedicated=true |
| 94 | |
| 95 | If none of the tree attributes (drawer, book, sockets), are specified |
| 96 | for the ``-device`` argument, like for all CPUs defined with the ``-smp`` |
| 97 | command argument the topology tree attributes will be set by simply |
| 98 | adding the CPUs to the topology based on the core-id. |
| 99 | |
| 100 | QEMU will not try to resolve collisions and will report an error if the |
| 101 | CPU topology defined explicitly or implicitly on a ``-device`` |
| 102 | argument collides with the definition of a CPU implicitly defined |
| 103 | on the ``-smp`` argument. |
| 104 | |
| 105 | When the topology modifier attributes are not defined for the |
| 106 | ``-device`` command argument they takes following default values: |
| 107 | |
| 108 | - dedicated: ``false`` |
| 109 | - entitlement: ``medium`` |
| 110 | |
| 111 | |
| 112 | Hot plug |
| 113 | ++++++++ |
| 114 | |
| 115 | New CPUs can be plugged using the device_add hmp command as in: |
| 116 | |
| 117 | .. code-block:: bash |
| 118 | |
| 119 | (qemu) device_add gen16b-s390x-cpu,core-id=9 |
| 120 | |
| 121 | The placement of the CPU is derived from the core-id as described above. |
| 122 | |
| 123 | The topology can of course also be fully defined: |
| 124 | |
| 125 | .. code-block:: bash |
| 126 | |
| 127 | (qemu) device_add gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 |
| 128 | |
| 129 | |
| 130 | Examples |
| 131 | ++++++++ |
| 132 | |
| 133 | In the following machine we define 8 sockets with 4 cores each. |
| 134 | |
| 135 | .. code-block:: bash |
| 136 | |
| 137 | $ qemu-system-s390x -accel kvm -m 2G \ |
| 138 | -cpu gen16b,ctop=on \ |
| 139 | -smp cpus=5,sockets=8,cores=4,maxcpus=32 \ |
| 140 | -device host-s390x-cpu,core-id=14 \ |
| 141 | |
| 142 | A new CPUs can be plugged using the device_add hmp command as before: |
| 143 | |
| 144 | .. code-block:: bash |
| 145 | |
| 146 | (qemu) device_add gen16b-s390x-cpu,core-id=9 |
| 147 | |
| 148 | The core-id defines the placement of the core in the topology by |
| 149 | starting with core 0 in socket 0 up to maxcpus. |
| 150 | |
| 151 | In the example above: |
| 152 | |
| 153 | * There are 5 CPUs provided to the guest with the ``-smp`` command line |
| 154 | They will take the core-ids 0,1,2,3,4 |
| 155 | As we have 4 cores in a socket, we have 4 CPUs provided |
| 156 | to the guest in socket 0, with core-ids 0,1,2,3. |
| 157 | The last CPU, with core-id 4, will be on socket 1. |
| 158 | |
| 159 | * the core with ID 14 provided by the ``-device`` command line will |
| 160 | be placed in socket 3, with core-id 14 |
| 161 | |
| 162 | * the core with ID 9 provided by the ``device_add`` qmp command will |
| 163 | be placed in socket 2, with core-id 9 |
| 164 | |
| 165 | |
| 166 | Polarization, entitlement and dedication |
| 167 | ---------------------------------------- |
| 168 | |
| 169 | Polarization |
| 170 | ++++++++++++ |
| 171 | |
| 172 | The polarization affects how the CPUs of a shared host are utilized/distributed |
| 173 | among guests. |
| 174 | The guest determines the polarization by using the PTF instruction. |
| 175 | |
| 176 | Polarization defines two models of CPU provisioning: horizontal |
| 177 | and vertical. |
| 178 | |
| 179 | The horizontal polarization is the default model on boot and after |
| 180 | subsystem reset. When horizontal polarization is in effect all vCPUs should |
| 181 | have about equal resource provisioning. |
| 182 | |
| 183 | In the vertical polarization model vCPUs are unequal, but overall more resources |
| 184 | might be available. |
| 185 | The guest can make use of the vCPU entitlement information provided by the host |
| 186 | to optimize kernel thread scheduling. |
| 187 | |
| 188 | A subsystem reset puts all vCPU of the configuration into the |
| 189 | horizontal polarization. |
| 190 | |
| 191 | Entitlement |
| 192 | +++++++++++ |
| 193 | |
| 194 | The vertical polarization specifies that the guest's vCPU can get |
| 195 | different real CPU provisioning: |
| 196 | |
| 197 | - a vCPU with vertical high entitlement specifies that this |
| 198 | vCPU gets 100% of the real CPU provisioning. |
| 199 | |
| 200 | - a vCPU with vertical medium entitlement specifies that this |
| 201 | vCPU shares the real CPU with other vCPUs. |
| 202 | |
| 203 | - a vCPU with vertical low entitlement specifies that this |
| 204 | vCPU only gets real CPU provisioning when no other vCPUs needs it. |
| 205 | |
| 206 | In the case a vCPU with vertical high entitlement does not use |
| 207 | the real CPU, the unused "slack" can be dispatched to other vCPU |
| 208 | with medium or low entitlement. |
| 209 | |
| 210 | A vCPU can be "dedicated" in which case the vCPU is fully dedicated to a single |
| 211 | real CPU. |
| 212 | |
| 213 | The dedicated bit is an indication of affinity of a vCPU for a real CPU |
| 214 | while the entitlement indicates the sharing or exclusivity of use. |
| 215 | |
| 216 | Defining the topology on the command line |
| 217 | ----------------------------------------- |
| 218 | |
| 219 | The topology can entirely be defined using -device cpu statements, |
| 220 | with the exception of CPU 0 which must be defined with the -smp |
| 221 | argument. |
| 222 | |
| 223 | For example, here we set the position of the cores 1,2,3 to |
| 224 | drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0, |
| 225 | book 0, socket 0 without defining entitlement or dedication. |
| 226 | Core 4 will be set on its default position on socket 1 |
| 227 | (since we have 4 core per socket) and we define it as dedicated and |
| 228 | with vertical high entitlement. |
| 229 | |
| 230 | .. code-block:: bash |
| 231 | |
| 232 | $ qemu-system-s390x -accel kvm -m 2G \ |
| 233 | -cpu gen16b,ctop=on \ |
| 234 | -smp cpus=1,sockets=8,cores=4,maxcpus=32 \ |
| 235 | \ |
| 236 | -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \ |
| 237 | -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \ |
| 238 | -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \ |
| 239 | \ |
| 240 | -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \ |
| 241 | -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \ |
| 242 | \ |
| 243 | -device gen16b-s390x-cpu,core-id=4,dedicated=on,entitlement=high |
| 244 | |
| 245 | The entitlement defined for the CPU 4 will only be used after the guest |
| 246 | successfully enables vertical polarization by using the PTF instruction. |