| 1 | ============== |
| 2 | NVMe Emulation |
| 3 | ============== |
| 4 | |
| 5 | QEMU provides NVMe emulation through the ``nvme``, ``nvme-ns`` and |
| 6 | ``nvme-subsys`` devices. |
| 7 | |
| 8 | See the following sections for specific information on |
| 9 | |
| 10 | * `Adding NVMe Devices`_, `additional namespaces`_ and `NVM subsystems`_. |
| 11 | * Configuration of `Optional Features`_ such as `Controller Memory Buffer`_, |
| 12 | `Simple Copy`_, `Zoned Namespaces`_, `metadata`_ and `End-to-End Data |
| 13 | Protection`_, |
| 14 | |
| 15 | Adding NVMe Devices |
| 16 | =================== |
| 17 | |
| 18 | Controller Emulation |
| 19 | -------------------- |
| 20 | |
| 21 | The QEMU emulated NVMe controller implements version 1.4 of the NVM Express |
| 22 | specification. All mandatory features are implement with a couple of exceptions |
| 23 | and limitations: |
| 24 | |
| 25 | * Accounting numbers in the SMART/Health log page are reset when the device |
| 26 | is power cycled. |
| 27 | * Interrupt Coalescing is not supported and is disabled by default. |
| 28 | |
| 29 | The simplest way to attach an NVMe controller on the QEMU PCI bus is to add the |
| 30 | following parameters: |
| 31 | |
| 32 | .. code-block:: console |
| 33 | |
| 34 | -drive file=nvm.img,if=none,id=nvm |
| 35 | -device nvme,serial=deadbeef,drive=nvm |
| 36 | |
| 37 | There are a number of optional general parameters for the ``nvme`` device. Some |
| 38 | are mentioned here, but see ``-device nvme,help`` to list all possible |
| 39 | parameters. |
| 40 | |
| 41 | ``max_ioqpairs=UINT32`` (default: ``64``) |
| 42 | Set the maximum number of allowed I/O queue pairs. This replaces the |
| 43 | deprecated ``num_queues`` parameter. |
| 44 | |
| 45 | ``msix_qsize=UINT16`` (default: ``65``) |
| 46 | The number of MSI-X vectors that the device should support. |
| 47 | |
| 48 | ``mdts=UINT8`` (default: ``7``) |
| 49 | Set the Maximum Data Transfer Size of the device. |
| 50 | |
| 51 | ``use-intel-id`` (default: ``off``) |
| 52 | Since QEMU 5.2, the device uses a QEMU allocated "Red Hat" PCI Device and |
| 53 | Vendor ID. Set this to ``on`` to revert to the unallocated Intel ID |
| 54 | previously used. |
| 55 | |
| 56 | ``ocp`` (default: ``off``) |
| 57 | The Open Compute Project defines the Datacenter NVMe SSD Specification that |
| 58 | sits on top of NVMe. It describes additional commands and NVMe behaviors |
| 59 | specific for the Datacenter. When this option is ``on`` OCP features such as |
| 60 | the SMART / Health information extended log become available in the |
| 61 | controller. We emulate version 5 of this log page. |
| 62 | |
| 63 | ``model`` (default: ``QEMU NVMe Ctrl``) |
| 64 | Override the default reported model, which can be used when needing |
| 65 | to more closely impersonate a particular device type. The model name |
| 66 | can be a maximum of 40 characters in length. |
| 67 | |
| 68 | ``firmware-version`` (default: current QEMU version number) |
| 69 | Override the default reported firmware version, which can be used when |
| 70 | needing to more closely impersonate a particular device type. The version |
| 71 | can be a maximum of 8 characters in length. |
| 72 | |
| 73 | Additional Namespaces |
| 74 | --------------------- |
| 75 | |
| 76 | In the simplest possible invocation sketched above, the device only support a |
| 77 | single namespace with the namespace identifier ``1``. To support multiple |
| 78 | namespaces and additional features, the ``nvme-ns`` device must be used. |
| 79 | |
| 80 | .. code-block:: console |
| 81 | |
| 82 | -device nvme,id=nvme-ctrl-0,serial=deadbeef |
| 83 | -drive file=nvm-1.img,if=none,id=nvm-1 |
| 84 | -device nvme-ns,drive=nvm-1 |
| 85 | -drive file=nvm-2.img,if=none,id=nvm-2 |
| 86 | -device nvme-ns,drive=nvm-2 |
| 87 | |
| 88 | The namespaces defined by the ``nvme-ns`` device will attach to the most |
| 89 | recently defined ``nvme-bus`` that is created by the ``nvme`` device. Namespace |
| 90 | identifiers are allocated automatically, starting from ``1``. |
| 91 | |
| 92 | There are a number of parameters available: |
| 93 | |
| 94 | ``nsid`` (default: ``0``) |
| 95 | Explicitly set the namespace identifier. |
| 96 | |
| 97 | ``uuid`` (default: *autogenerated*) |
| 98 | Set the UUID of the namespace. This will be reported as a "Namespace UUID" |
| 99 | descriptor in the Namespace Identification Descriptor List. |
| 100 | |
| 101 | ``nguid`` |
| 102 | Set the NGUID of the namespace. This will be reported as a "Namespace Globally |
| 103 | Unique Identifier" descriptor in the Namespace Identification Descriptor List. |
| 104 | It is specified as a string of hexadecimal digits containing exactly 16 bytes |
| 105 | or "auto" for a random value. An optional '-' separator could be used to group |
| 106 | bytes. If not specified the NGUID will remain all zeros. |
| 107 | |
| 108 | ``eui64`` |
| 109 | Set the EUI-64 of the namespace. This will be reported as a "IEEE Extended |
| 110 | Unique Identifier" descriptor in the Namespace Identification Descriptor List. |
| 111 | Since machine type 6.1 a non-zero default value is used if the parameter |
| 112 | is not provided. For earlier machine types the field defaults to 0. |
| 113 | |
| 114 | ``bus`` |
| 115 | If there are more ``nvme`` devices defined, this parameter may be used to |
| 116 | attach the namespace to a specific ``nvme`` device (identified by an ``id`` |
| 117 | parameter on the controller device). |
| 118 | |
| 119 | NVM Subsystems |
| 120 | -------------- |
| 121 | |
| 122 | Additional features becomes available if the controller device (``nvme``) is |
| 123 | linked to an NVM Subsystem device (``nvme-subsys``). |
| 124 | |
| 125 | The NVM Subsystem emulation allows features such as shared namespaces and |
| 126 | multipath I/O. |
| 127 | |
| 128 | .. code-block:: console |
| 129 | |
| 130 | -device nvme-subsys,id=nvme-subsys-0,nqn=subsys0 |
| 131 | -device nvme,serial=deadbeef,subsys=nvme-subsys-0 |
| 132 | -device nvme,serial=deadbeef,subsys=nvme-subsys-0 |
| 133 | |
| 134 | This will create an NVM subsystem with two controllers. Having controllers |
| 135 | linked to an ``nvme-subsys`` device allows additional ``nvme-ns`` parameters: |
| 136 | |
| 137 | ``shared`` (default: ``on`` since 6.2) |
| 138 | Specifies that the namespace will be attached to all controllers in the |
| 139 | subsystem. If set to ``off``, the namespace will remain a private namespace |
| 140 | and may only be attached to a single controller at a time. Shared namespaces |
| 141 | are always automatically attached to all controllers (also when controllers |
| 142 | are hotplugged). |
| 143 | |
| 144 | ``detached`` (default: ``off``) |
| 145 | If set to ``on``, the namespace will be be available in the subsystem, but |
| 146 | not attached to any controllers initially. A shared namespace with this set |
| 147 | to ``on`` will never be automatically attached to controllers. |
| 148 | |
| 149 | Thus, adding |
| 150 | |
| 151 | .. code-block:: console |
| 152 | |
| 153 | -drive file=nvm-1.img,if=none,id=nvm-1 |
| 154 | -device nvme-ns,drive=nvm-1,nsid=1 |
| 155 | -drive file=nvm-2.img,if=none,id=nvm-2 |
| 156 | -device nvme-ns,drive=nvm-2,nsid=3,shared=off,detached=on |
| 157 | |
| 158 | will cause NSID 1 will be a shared namespace that is initially attached to both |
| 159 | controllers. NSID 3 will be a private namespace due to ``shared=off`` and only |
| 160 | attachable to a single controller at a time. Additionally it will not be |
| 161 | attached to any controller initially (due to ``detached=on``) or to hotplugged |
| 162 | controllers. |
| 163 | |
| 164 | Optional Features |
| 165 | ================= |
| 166 | |
| 167 | Controller Memory Buffer |
| 168 | ------------------------ |
| 169 | |
| 170 | ``nvme`` device parameters related to the Controller Memory Buffer support: |
| 171 | |
| 172 | ``cmb_size_mb=UINT32`` (default: ``0``) |
| 173 | This adds a Controller Memory Buffer of the given size at offset zero in BAR |
| 174 | 2. |
| 175 | |
| 176 | ``legacy-cmb`` (default: ``off``) |
| 177 | By default, the device uses the "v1.4 scheme" for the Controller Memory |
| 178 | Buffer support (i.e, the CMB is initially disabled and must be explicitly |
| 179 | enabled by the host). Set this to ``on`` to behave as a v1.3 device wrt. the |
| 180 | CMB. |
| 181 | |
| 182 | Simple Copy |
| 183 | ----------- |
| 184 | |
| 185 | The device includes support for TP 4065 ("Simple Copy Command"). A number of |
| 186 | additional ``nvme-ns`` device parameters may be used to control the Copy |
| 187 | command limits: |
| 188 | |
| 189 | ``mssrl=UINT16`` (default: ``128``) |
| 190 | Set the Maximum Single Source Range Length (``MSSRL``). This is the maximum |
| 191 | number of logical blocks that may be specified in each source range. |
| 192 | |
| 193 | ``mcl=UINT32`` (default: ``128``) |
| 194 | Set the Maximum Copy Length (``MCL``). This is the maximum number of logical |
| 195 | blocks that may be specified in a Copy command (the total for all source |
| 196 | ranges). |
| 197 | |
| 198 | ``msrc=UINT8`` (default: ``127``) |
| 199 | Set the Maximum Source Range Count (``MSRC``). This is the maximum number of |
| 200 | source ranges that may be used in a Copy command. This is a 0's based value. |
| 201 | |
| 202 | Zoned Namespaces |
| 203 | ---------------- |
| 204 | |
| 205 | A namespaces may be "Zoned" as defined by TP 4053 ("Zoned Namespaces"). Set |
| 206 | ``zoned=on`` on an ``nvme-ns`` device to configure it as a zoned namespace. |
| 207 | |
| 208 | The namespace may be configured with additional parameters |
| 209 | |
| 210 | ``zoned.zone_size=SIZE`` (default: ``128MiB``) |
| 211 | Define the zone size (``ZSZE``). |
| 212 | |
| 213 | ``zoned.zone_capacity=SIZE`` (default: ``0``) |
| 214 | Define the zone capacity (``ZCAP``). If left at the default (``0``), the zone |
| 215 | capacity will equal the zone size. |
| 216 | |
| 217 | ``zoned.descr_ext_size=UINT32`` (default: ``0``) |
| 218 | Set the Zone Descriptor Extension Size (``ZDES``). Must be a multiple of 64 |
| 219 | bytes. |
| 220 | |
| 221 | ``zoned.cross_read=BOOL`` (default: ``off``) |
| 222 | Set to ``on`` to allow reads to cross zone boundaries. |
| 223 | |
| 224 | ``zoned.max_active=UINT32`` (default: ``0``) |
| 225 | Set the maximum number of active resources (``MAR``). The default (``0``) |
| 226 | allows all zones to be active. |
| 227 | |
| 228 | ``zoned.max_open=UINT32`` (default: ``0``) |
| 229 | Set the maximum number of open resources (``MOR``). The default (``0``) |
| 230 | allows all zones to be open. If ``zoned.max_active`` is specified, this value |
| 231 | must be less than or equal to that. |
| 232 | |
| 233 | ``zoned.zasl=UINT8`` (default: ``0``) |
| 234 | Set the maximum data transfer size for the Zone Append command. Like |
| 235 | ``mdts``, the value is specified as a power of two (2^n) and is in units of |
| 236 | the minimum memory page size (CAP.MPSMIN). The default value (``0``) |
| 237 | has this property inherit the ``mdts`` value. |
| 238 | |
| 239 | Flexible Data Placement |
| 240 | ----------------------- |
| 241 | |
| 242 | The device may be configured to support TP4146 ("Flexible Data Placement") by |
| 243 | configuring it (``fdp=on``) on the subsystem:: |
| 244 | |
| 245 | -device nvme-subsys,id=nvme-subsys-0,nqn=subsys0,fdp=on,fdp.nruh=16 |
| 246 | |
| 247 | The subsystem emulates a single Endurance Group, on which Flexible Data |
| 248 | Placement will be supported. Also note that the device emulation deviates |
| 249 | slightly from the specification, by always enabling the "FDP Mode" feature on |
| 250 | the controller if the subsystems is configured for Flexible Data Placement. |
| 251 | |
| 252 | Enabling Flexible Data Placement on the subsyste enables the following |
| 253 | parameters: |
| 254 | |
| 255 | ``fdp.nrg`` (default: ``1``) |
| 256 | Set the number of Reclaim Groups. |
| 257 | |
| 258 | ``fdp.nruh`` (default: ``0``) |
| 259 | Set the number of Reclaim Unit Handles. This is a mandatory parameter and |
| 260 | must be non-zero. |
| 261 | |
| 262 | ``fdp.runs`` (default: ``96M``) |
| 263 | Set the Reclaim Unit Nominal Size. Defaults to 96 MiB. |
| 264 | |
| 265 | Namespaces within this subsystem may requests Reclaim Unit Handles:: |
| 266 | |
| 267 | -device nvme-ns,drive=nvm-1,fdp.ruhs=RUHLIST |
| 268 | |
| 269 | The ``RUHLIST`` is a semicolon separated list (i.e. ``0;1;2;3``) and may |
| 270 | include ranges (i.e. ``0;8-15``). If no reclaim unit handle list is specified, |
| 271 | the controller will assign the controller-specified reclaim unit handle to |
| 272 | placement handle identifier 0. |
| 273 | |
| 274 | Metadata |
| 275 | -------- |
| 276 | |
| 277 | The virtual namespace device supports LBA metadata in the form separate |
| 278 | metadata (``MPTR``-based) and extended LBAs. |
| 279 | |
| 280 | ``ms=UINT16`` (default: ``0``) |
| 281 | Defines the number of metadata bytes per LBA. |
| 282 | |
| 283 | ``mset=UINT8`` (default: ``0``) |
| 284 | Set to ``1`` to enable extended LBAs. |
| 285 | |
| 286 | End-to-End Data Protection |
| 287 | -------------------------- |
| 288 | |
| 289 | The virtual namespace device supports DIF- and DIX-based protection information |
| 290 | (depending on ``mset``). |
| 291 | |
| 292 | ``pi=UINT8`` (default: ``0``) |
| 293 | Enable protection information of the specified type (type ``1``, ``2`` or |
| 294 | ``3``). |
| 295 | |
| 296 | ``pil=UINT8`` (default: ``0``) |
| 297 | Controls the location of the protection information within the metadata. Set |
| 298 | to ``1`` to transfer protection information as the first bytes of metadata. |
| 299 | Otherwise, the protection information is transferred as the last bytes of |
| 300 | metadata. |
| 301 | |
| 302 | ``pif=UINT8`` (default: ``0``) |
| 303 | By default, the namespace device uses 16 bit guard protection information |
| 304 | format (``pif=0``). Set to ``2`` to enable 64 bit guard protection |
| 305 | information format. This requires at least 16 bytes of metadata. Note that |
| 306 | ``pif=1`` (32 bit guards) is currently not supported. |
| 307 | |
| 308 | Virtualization Enhancements and SR-IOV (Experimental Support) |
| 309 | ------------------------------------------------------------- |
| 310 | |
| 311 | The ``nvme`` device supports Single Root I/O Virtualization and Sharing |
| 312 | along with Virtualization Enhancements. The controller has to be linked to |
| 313 | an NVM Subsystem device (``nvme-subsys``) for use with SR-IOV. |
| 314 | |
| 315 | A number of parameters are present (**please note, that they may be |
| 316 | subject to change**): |
| 317 | |
| 318 | ``sriov_max_vfs`` (default: ``0``) |
| 319 | Indicates the maximum number of PCIe virtual functions supported |
| 320 | by the controller. Specifying a non-zero value enables reporting of both |
| 321 | SR-IOV and ARI (Alternative Routing-ID Interpretation) capabilities |
| 322 | by the NVMe device. Virtual function controllers will not report SR-IOV. |
| 323 | |
| 324 | ``sriov_vq_flexible`` |
| 325 | Indicates the total number of flexible queue resources assignable to all |
| 326 | the secondary controllers. Implicitly sets the number of primary |
| 327 | controller's private resources to ``(max_ioqpairs - sriov_vq_flexible)``. |
| 328 | |
| 329 | ``sriov_vi_flexible`` |
| 330 | Indicates the total number of flexible interrupt resources assignable to |
| 331 | all the secondary controllers. Implicitly sets the number of primary |
| 332 | controller's private resources to ``(msix_qsize - sriov_vi_flexible)``. |
| 333 | |
| 334 | ``sriov_max_vi_per_vf`` (default: ``0``) |
| 335 | Indicates the maximum number of virtual interrupt resources assignable |
| 336 | to a secondary controller. The default ``0`` resolves to |
| 337 | ``(sriov_vi_flexible / sriov_max_vfs)`` |
| 338 | |
| 339 | ``sriov_max_vq_per_vf`` (default: ``0``) |
| 340 | Indicates the maximum number of virtual queue resources assignable to |
| 341 | a secondary controller. The default ``0`` resolves to |
| 342 | ``(sriov_vq_flexible / sriov_max_vfs)`` |
| 343 | |
| 344 | The simplest possible invocation enables the capability to set up one VF |
| 345 | controller and assign an admin queue, an IO queue, and a MSI-X interrupt. |
| 346 | |
| 347 | .. code-block:: console |
| 348 | |
| 349 | -device nvme-subsys,id=subsys0 |
| 350 | -device nvme,serial=deadbeef,subsys=subsys0,sriov_max_vfs=1, |
| 351 | sriov_vq_flexible=2,sriov_vi_flexible=1 |
| 352 | |
| 353 | The minimum steps required to configure a functional NVMe secondary |
| 354 | controller are: |
| 355 | |
| 356 | * unbind flexible resources from the primary controller |
| 357 | |
| 358 | .. code-block:: console |
| 359 | |
| 360 | nvme virt-mgmt /dev/nvme0 -c 0 -r 1 -a 1 -n 0 |
| 361 | nvme virt-mgmt /dev/nvme0 -c 0 -r 0 -a 1 -n 0 |
| 362 | |
| 363 | * perform a Function Level Reset on the primary controller to actually |
| 364 | release the resources |
| 365 | |
| 366 | .. code-block:: console |
| 367 | |
| 368 | echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset |
| 369 | |
| 370 | * enable VF |
| 371 | |
| 372 | .. code-block:: console |
| 373 | |
| 374 | echo 1 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs |
| 375 | |
| 376 | * assign the flexible resources to the VF and set it ONLINE |
| 377 | |
| 378 | .. code-block:: console |
| 379 | |
| 380 | nvme virt-mgmt /dev/nvme0 -c 1 -r 1 -a 8 -n 1 |
| 381 | nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 8 -n 2 |
| 382 | nvme virt-mgmt /dev/nvme0 -c 1 -r 0 -a 9 -n 0 |
| 383 | |
| 384 | * bind the NVMe driver to the VF |
| 385 | |
| 386 | .. code-block:: console |
| 387 | |
| 388 | echo 0000:01:00.1 > /sys/bus/pci/drivers/nvme/bind |