@samitouri / QOSamiQemu
Code Commits Issues Pulls Sessions CI/CD
master
master
rst 2,191 lines 80.7 KB
Raw
1 .. _vhost_user_proto:
2
3 ===================
4 Vhost-user Protocol
5 ===================
6
7 ..
8 Copyright 2014 Virtual Open Systems Sarl.
9 Copyright 2019 Intel Corporation
10 Licence: This work is licensed under the terms of the GNU GPL,
11 version 2 or later. See the COPYING file in the top-level
12 directory.
13
14 .. contents:: Table of Contents
15
16 Introduction
17 ============
18
19 This protocol is aiming to complement the ``ioctl`` interface used to
20 control the vhost implementation in the Linux kernel. It implements
21 the control plane needed to establish virtqueue sharing with a user
22 space process on the same host. It uses communication over a Unix
23 domain socket to share file descriptors in the ancillary data of the
24 message.
25
26 The protocol defines 2 sides of the communication, *front-end* and
27 *back-end*. The *front-end* is the application that shares its virtqueues, in
28 our case QEMU. The *back-end* is the consumer of the virtqueues.
29
30 In the current implementation QEMU is the *front-end*, and the *back-end*
31 is the external process consuming the virtio queues, for example a
32 software Ethernet switch running in user space, such as Snabbswitch,
33 or a block device back-end processing read & write to a virtual
34 disk. In order to facilitate interoperability between various back-end
35 implementations, it is recommended to follow the :ref:`Backend program
36 conventions <backend_conventions>`.
37
38 The *front-end* and *back-end* can be either a client (i.e. connecting) or
39 server (listening) in the socket communication.
40
41 Support for platforms other than Linux
42 --------------------------------------
43
44 While vhost-user was initially developed targeting Linux, nowadays it
45 is supported on any platform that provides the following features:
46
47 - A way for requesting shared memory represented by a file descriptor
48 so it can be passed over a UNIX domain socket and then mapped by the
49 other process.
50
51 - AF_UNIX sockets with SCM_RIGHTS, so QEMU and the other process can
52 exchange messages through it, including ancillary data when needed.
53
54 - Either eventfd or pipe/pipe2. On platforms where eventfd is not
55 available, QEMU will automatically fall back to pipe2 or, as a last
56 resort, pipe. Each file descriptor will be used for receiving or
57 sending events by reading or writing (respectively) an 8-byte value
58 to the corresponding it. The 8-value itself has no meaning and
59 should not be interpreted.
60
61 Message Specification
62 =====================
63
64 .. Note:: All numbers are in the machine native byte order.
65
66 A vhost-user message consists of 3 header fields and a payload.
67
68 +---------+-------+------+---------+
69 | request | flags | size | payload |
70 +---------+-------+------+---------+
71
72 Header
73 ------
74
75 :request: 32-bit type of the request
76
77 :flags: 32-bit bit field
78
79 - Lower 2 bits are the version (currently 0x01)
80 - Bit 2 is the reply flag - needs to be sent on each reply from the back-end
81 - Bit 3 is the need_reply flag - see :ref:`REPLY_ACK <reply_ack>` for
82 details.
83
84 :size: 32-bit size of the payload
85
86 Payload
87 -------
88
89 Depending on the request type, **payload** can be:
90
91 A single 64-bit integer
92 ^^^^^^^^^^^^^^^^^^^^^^^
93
94 +-----+
95 | u64 |
96 +-----+
97
98 :u64: a 64-bit unsigned integer
99
100 A vring state description
101 ^^^^^^^^^^^^^^^^^^^^^^^^^
102
103 +-------+-----+
104 | index | num |
105 +-------+-----+
106
107 :index: a 32-bit index
108
109 :num: a 32-bit number
110
111 A vring descriptor index for split virtqueues
112 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
113
114 +-------------+---------------------+
115 | vring index | index in avail ring |
116 +-------------+---------------------+
117
118 :vring index: 32-bit index of the respective virtqueue
119
120 :index in avail ring: 32-bit value, of which currently only the lower 16
121 bits are used:
122
123 - Bits 0–15: Index of the next *Available Ring* descriptor that the
124 back-end will process. This is a free-running index that is not
125 wrapped by the ring size.
126 - Bits 16–31: Reserved (set to zero)
127
128 Vring descriptor indices for packed virtqueues
129 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
130
131 +-------------+--------------------+
132 | vring index | descriptor indices |
133 +-------------+--------------------+
134
135 :vring index: 32-bit index of the respective virtqueue
136
137 :descriptor indices: 32-bit value:
138
139 - Bits 0–14: Index of the next *Available Ring* descriptor that the
140 back-end will process. This is a free-running index that is not
141 wrapped by the ring size.
142 - Bit 15: Driver (Available) Ring Wrap Counter
143 - Bits 16–30: Index of the entry in the *Used Ring* where the back-end
144 will place the next descriptor. This is a free-running index that
145 is not wrapped by the ring size.
146 - Bit 31: Device (Used) Ring Wrap Counter
147
148 A vring address description
149 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
150
151 +-------+-------+------------+------+-----------+-----+
152 | index | flags | descriptor | used | available | log |
153 +-------+-------+------------+------+-----------+-----+
154
155 :index: a 32-bit vring index
156
157 :flags: a 32-bit vring flags
158
159 :descriptor: a 64-bit ring address of the vring descriptor table
160
161 :used: a 64-bit ring address of the vring used ring
162
163 :available: a 64-bit ring address of the vring available ring
164
165 :log: a 64-bit guest address for logging
166
167 .. Note::
168 When ``VIRTIO_F_IOMMU_PLATFORM`` is negotiated, ring addresses are IOVAs.
169
170 Otherwise, when ``VHOST_USER_PROTOCOL_F_GPA_ADDRESSES`` is negotiated, the
171 ring addresses are guest physical addresses for frontend messages. That
172 does not apply to backend replies.
173
174 Finally, when neither ``VIRTIO_F_IOMMU_PLATFORM`` nor
175 ``VHOST_USER_PROTOCOL_F_GPA_ADDRESSES`` features are negotiated, ring
176 addresses are user virtual addresses.
177
178 .. _memory_region_description:
179
180 Memory region description
181 ^^^^^^^^^^^^^^^^^^^^^^^^^
182
183 +---------------+------+--------------+-------------+
184 | guest address | size | user address | mmap offset |
185 +---------------+------+--------------+-------------+
186
187 :guest address: a 64-bit guest address of the region
188
189 :size: a 64-bit size
190
191 :user address: a 64-bit user address. When ``VHOST_USER_PROTOCOL_F_GPA_ADDRESSES``
192 is negotiated, this field contain guest physical address instead and must
193 duplicate ``guest address`` field.
194
195 :mmap offset: a 64-bit offset where region starts in the mapped memory
196
197 When the ``VHOST_USER_PROTOCOL_F_XEN_MMAP`` protocol feature has been
198 successfully negotiated, the memory region description contains two extra
199 fields at the end.
200
201 +---------------+------+--------------+-------------+----------------+-------+
202 | guest address | size | user address | mmap offset | xen mmap flags | domid |
203 +---------------+------+--------------+-------------+----------------+-------+
204
205 :xen mmap flags: a 32-bit bit field
206
207 - Bit 0 is set for Xen foreign memory mapping.
208 - Bit 1 is set for Xen grant memory mapping.
209 - Bit 8 is set if the memory region can not be mapped in advance, and memory
210 areas within this region must be mapped / unmapped only when required by the
211 back-end. The back-end shouldn't try to map the entire region at once, as the
212 front-end may not allow it. The back-end should rather map only the required
213 amount of memory at once and unmap it after it is used.
214
215 :domid: a 32-bit Xen hypervisor specific domain id.
216
217 For all memory regions active at a given time:
218
219 - ``[guest address, guest address + size)`` of one memory region never overlaps
220 the ``[guest address, guest address + size)`` of another memory region.
221
222 - ``[user address, user address + size)`` of one memory region never overlaps
223 the ``[user address, user address + size)`` of another memory region.
224
225 Violating any of these is a bug in the front-end. This ensures that a guest
226 address or user address always refers to at most one location in memory.
227 The front-end must remove a region before it can add an overlapping one.
228
229 Single memory region description
230 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
231
232 +---------+--------+
233 | padding | region |
234 +---------+--------+
235
236 :padding: 64-bit
237
238 :region: region is represented by :ref:`Memory region description <memory_region_description>`.
239
240 Multiple Memory regions description
241 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
242
243 +-------------+---------+---------+-----+---------+
244 | num regions | padding | region0 | ... | region7 |
245 +-------------+---------+---------+-----+---------+
246
247 :num regions: a 32-bit number of regions
248
249 :padding: 32-bit
250
251 :regions: regions field contains 8 regions of type :ref:`Memory region description <memory_region_description>`.
252
253 Log description
254 ^^^^^^^^^^^^^^^
255
256 +----------+------------+
257 | log size | log offset |
258 +----------+------------+
259
260 :log size: a 64-bit size of area used for logging
261
262 :log offset: a 64-bit offset from start of supplied file descriptor where
263 logging starts (i.e. where guest address 0 would be
264 logged)
265
266 An IOTLB message
267 ^^^^^^^^^^^^^^^^
268
269 +------+------+--------------+-------------------+------+
270 | iova | size | user address | permissions flags | type |
271 +------+------+--------------+-------------------+------+
272
273 :iova: a 64-bit I/O virtual address programmed by the guest
274
275 :size: a 64-bit size
276
277 :user address: a 64-bit user address. When ``VHOST_USER_PROTOCOL_F_GPA_ADDRESSES``
278 is negotiated, this field contain guest physical address instead, except for
279 ``VHOST_USER_BACKEND_IOTLB_MSG``, where it's user address anyway.
280
281 :permissions flags: an 8-bit value:
282 - 0: No access
283 - 1: Read access
284 - 2: Write access
285 - 3: Read/Write access
286
287 :type: an 8-bit IOTLB message type:
288 - 1: IOTLB miss
289 - 2: IOTLB update
290 - 3: IOTLB invalidate
291 - 4: IOTLB access fail
292
293 Virtio device config space
294 ^^^^^^^^^^^^^^^^^^^^^^^^^^
295
296 +--------+------+-------+---------+
297 | offset | size | flags | payload |
298 +--------+------+-------+---------+
299
300 :offset: a 32-bit offset of virtio device's configuration space
301
302 :size: a 32-bit configuration space access size in bytes
303
304 :flags: a 32-bit value:
305 - 0: Vhost front-end messages used for writable fields
306 - 1: Vhost front-end messages used for live migration
307
308 :payload: Size bytes array holding the contents of the virtio
309 device's configuration space
310
311 Vring area description
312 ^^^^^^^^^^^^^^^^^^^^^^
313
314 +-----+------+--------+
315 | u64 | size | offset |
316 +-----+------+--------+
317
318 :u64: a 64-bit integer contains vring index and flags
319
320 :size: a 64-bit size of this area
321
322 :offset: a 64-bit offset of this area from the start of the
323 supplied file descriptor
324
325 Inflight description
326 ^^^^^^^^^^^^^^^^^^^^
327
328 +-----------+-------------+------------+------------+
329 | mmap size | mmap offset | num queues | queue size |
330 +-----------+-------------+------------+------------+
331
332 :mmap size: a 64-bit size of area to track inflight I/O
333
334 :mmap offset: a 64-bit offset of this area from the start
335 of the supplied file descriptor
336
337 :num queues: a 16-bit number of virtqueues
338
339 :queue size: a 16-bit size of virtqueues
340
341 VhostUserShared
342 ^^^^^^^^^^^^^^^
343
344 +------+
345 | UUID |
346 +------+
347
348 :UUID: 16 bytes UUID, whose first three components (a 32-bit value, then
349 two 16-bit values) are stored in big endian.
350
351 Device state transfer parameters
352 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
353
354 +--------------------+-----------------+
355 | transfer direction | migration phase |
356 +--------------------+-----------------+
357
358 :transfer direction: a 32-bit enum, describing the direction in which
359 the state is transferred:
360
361 - 0: Save: Transfer the state from the back-end to the front-end,
362 which happens on the source side of migration
363 - 1: Load: Transfer the state from the front-end to the back-end,
364 which happens on the destination side of migration
365
366 :migration phase: a 32-bit enum, describing the state in which the VM
367 guest and devices are:
368
369 - 0: Stopped (in the period after the transfer of memory-mapped
370 regions before switch-over to the destination): The VM guest is
371 stopped, and the vhost-user device is suspended (see
372 :ref:`Suspended device state <suspended_device_state>`).
373
374 In the future, additional phases might be added e.g. to allow
375 iterative migration while the device is running.
376
377 MMAP request
378 ^^^^^^^^^^^^
379
380 +-------+---------+-----------+------------+-----+-------+
381 | shmid | padding | fd_offset | shm_offset | len | flags |
382 +-------+---------+-----------+------------+-----+-------+
383
384 :shmid: a 8-bit shared memory region identifier
385
386 :fd_offset: a 64-bit offset of this area from the start
387 of the supplied file descriptor
388
389 :shm_offset: a 64-bit offset from the start of the
390 pointed shared memory region
391
392 :len: a 64-bit size of the memory to map
393
394 :flags: a 64-bit value:
395
396 - 0: Pages are mapped read-only
397 - 1: Pages are mapped read-write
398
399 VIRTIO Shared Memory Region configuration
400 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
401
402 +-------------+---------+------------+----+--------------+
403 | num regions | padding | mem size 0 | .. | mem size 255 |
404 +-------------+---------+------------+----+--------------+
405
406 :num regions: a 32-bit number of regions
407
408 :padding: 32-bit
409
410 :mem size: an array of 256 64-bit fields representing the size of each
411 VIRTIO Shared Memory Region. ``num regions`` specifies the
412 number of valid regions (non-zero size). The array index
413 corresponds to the shared memory ID (shmid).
414
415 C structure
416 -----------
417
418 In QEMU the vhost-user message is implemented with the following struct:
419
420 .. code:: c
421
422 typedef struct VhostUserMsg {
423 VhostUserRequest request;
424 uint32_t flags;
425 uint32_t size;
426 union {
427 uint64_t u64;
428 struct vhost_vring_state state;
429 struct vhost_vring_addr addr;
430 VhostUserMemory memory;
431 VhostUserMemRegMsg mem_reg;
432 VhostUserLog log;
433 struct vhost_iotlb_msg iotlb;
434 VhostUserConfig config;
435 VhostUserCryptoSession session;
436 VhostUserVringArea area;
437 VhostUserInflight inflight;
438 VhostUserShared object;
439 VhostUserTransferDeviceState transfer_state;
440 VhostUserMMap mmap;
441 VhostUserShMemConfig shmem;
442 };
443 } QEMU_PACKED VhostUserMsg;
444
445 Communication
446 =============
447
448 The protocol for vhost-user is based on the existing implementation of
449 vhost for the Linux Kernel. Most messages that can be sent via the
450 Unix domain socket implementing vhost-user have an equivalent ioctl to
451 the kernel implementation.
452
453 The communication consists of the *front-end* sending message requests and
454 the *back-end* sending message replies. Most of the requests don't require
455 replies, except for the following requests:
456
457 * ``VHOST_USER_GET_FEATURES``
458 * ``VHOST_USER_GET_PROTOCOL_FEATURES``
459 * ``VHOST_USER_GET_VRING_BASE``
460 * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
461 * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
462
463 .. seealso::
464
465 :ref:`REPLY_ACK <reply_ack>`
466 The section on ``REPLY_ACK`` protocol extension.
467
468 There are several messages that the front-end sends with file descriptors passed
469 in the ancillary data:
470
471 * ``VHOST_USER_ADD_MEM_REG``
472 * ``VHOST_USER_SET_MEM_TABLE``
473 * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
474 * ``VHOST_USER_SET_LOG_FD``
475 * ``VHOST_USER_SET_VRING_KICK``
476 * ``VHOST_USER_SET_VRING_CALL``
477 * ``VHOST_USER_SET_VRING_ERR``
478 * ``VHOST_USER_SET_BACKEND_REQ_FD`` (previous name ``VHOST_USER_SET_SLAVE_REQ_FD``)
479 * ``VHOST_USER_SET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
480 * ``VHOST_USER_SET_DEVICE_STATE_FD``
481
482 When sending file descriptors in ancillary data, *front-end* should
483 associate the ancillary data with a ``sendmsg`` operation (or
484 equivalent) that sends bytes starting with the first byte of the
485 message header. *back-end* can therefore expect that file descriptors
486 will only be received in the first ``recvmsg`` operation for a message
487 header.
488
489 If *front-end* is unable to send the full message or receives a wrong
490 reply it will close the connection. An optional reconnection mechanism
491 can be implemented.
492
493 If *back-end* detects some error such as incompatible features, it may also
494 close the connection. This should only happen in exceptional circumstances.
495
496 Any protocol extensions are gated by protocol feature bits, which
497 allows full backwards compatibility on both front-end and back-end. As
498 older back-ends don't support negotiating protocol features, a feature
499 bit was dedicated for this purpose::
500
501 #define VHOST_USER_F_PROTOCOL_FEATURES 30
502
503 Note that VHOST_USER_F_PROTOCOL_FEATURES is the UNUSED (30) feature
504 bit defined in `VIRTIO 1.1 6.3 Legacy Interface: Reserved Feature Bits
505 <https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-4130003>`_.
506 VIRTIO devices do not advertise this feature bit and therefore VIRTIO
507 drivers cannot negotiate it.
508
509 This reserved feature bit was reused by the vhost-user protocol to add
510 vhost-user protocol feature negotiation in a backwards compatible
511 fashion. Old vhost-user front-end and back-end implementations continue to
512 work even though they are not aware of vhost-user protocol feature
513 negotiation.
514
515 Ring states
516 -----------
517
518 Rings have two independent states: started/stopped, and enabled/disabled.
519
520 * While a ring is stopped, the back-end must not process the ring at
521 all, regardless of whether it is enabled or disabled. The
522 enabled/disabled state should still be tracked, though, so it can come
523 into effect once the ring is started.
524
525 * started and disabled: The back-end must process the ring without
526 causing any side effects. For example, for a networking device,
527 in the disabled state the back-end must not supply any new RX packets,
528 but must process and discard any TX packets.
529
530 * started and enabled: The back-end must process the ring normally, i.e.
531 process all requests and execute them.
532
533 Each ring is initialized in a stopped and disabled state. Rings are started
534 with ``VHOST_USER_SET_VRING_KICK`` (or ``VHOST_USER_VRING_KICK`` if
535 ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` is negotiated) and stopped with
536 ``VHOST_USER_GET_VRING_BASE``. A stopped ring enters the started state again
537 with ``VHOST_USER_SET_VRING_KICK`` (or ``VHOST_USER_VRING_KICK`` if
538 ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` is negotiated) and the back-end
539 resumes processing requests.
540
541 Note that previous versions of this specification stated that rings start when
542 the back-end receives a kick (that is, detecting that file descriptor is
543 readable) on the descriptor specified by ``VHOST_USER_SET_VRING_KICK`` or
544 receiving the in-band message ``VHOST_USER_VRING_KICK`` if negotiated.
545 Widely-used front-ends and back-ends did not implement this behavior and it
546 complicates poll mode back-ends that do not rely on the kick file descriptor.
547
548 For compatibility with back-ends that implemented the start on kick behavior,
549 front-ends SHOULD inject a kick after ``VHOST_USER_SET_VRING_KICK``. This
550 ensures that the back-end processes any available requests in the ring.
551 Back-ends SHOULD NOT rely on receiving a kick after
552 ``VHOST_USER_SET_VRING_KICK``.
553
554 Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``.
555
556 In addition, upon receiving a ``VHOST_USER_SET_FEATURES`` message from
557 the front-end without ``VHOST_USER_F_PROTOCOL_FEATURES`` set, the
558 back-end must enable all rings immediately.
559
560 While processing the rings (whether they are enabled or not), the back-end
561 must support changing some configuration aspects on the fly.
562
563 .. _suspended_device_state:
564
565 Suspended device state
566 ^^^^^^^^^^^^^^^^^^^^^^
567
568 While all vrings are stopped, the device is *suspended*. In addition to
569 not processing any vring (because they are stopped), the device must:
570
571 * not write to any guest memory regions,
572 * not send any notifications to the guest,
573 * not send any messages to the front-end,
574 * still process and reply to messages from the front-end.
575
576 Multiple queue support
577 ----------------------
578
579 Many devices have a fixed number of virtqueues. In this case the front-end
580 already knows the number of available virtqueues without communicating with the
581 back-end.
582
583 Some devices do not have a fixed number of virtqueues. Instead the maximum
584 number of virtqueues is chosen by the back-end. The number can depend on host
585 resource availability or back-end implementation details. Such devices are called
586 multiple queue devices.
587
588 Multiple queue support allows the back-end to advertise the maximum number of
589 queues. This is treated as a protocol extension, hence the back-end has to
590 implement protocol features first. The multiple queues feature is supported
591 only when the protocol feature ``VHOST_USER_PROTOCOL_F_MQ`` (bit 0) is set.
592
593 The max number of queues the back-end supports can be queried with message
594 ``VHOST_USER_GET_QUEUE_NUM``. Front-end should stop when the number of requested
595 queues is bigger than that.
596
597 As all queues share one connection, the front-end uses a unique index for each
598 queue in the sent message to identify a specified queue.
599
600 The front-end enables queues by sending message ``VHOST_USER_SET_VRING_ENABLE``.
601 vhost-user-net has historically automatically enabled the first queue pair.
602
603 Back-ends should always implement the ``VHOST_USER_PROTOCOL_F_MQ`` protocol
604 feature, even for devices with a fixed number of virtqueues, since it is simple
605 to implement and offers a degree of introspection.
606
607 Front-ends must not rely on the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature for
608 devices with a fixed number of virtqueues. Only true multiqueue devices
609 require this protocol feature.
610
611 Migration
612 ---------
613
614 During live migration, the front-end may need to track the modifications
615 the back-end makes to the memory mapped regions. The front-end should mark
616 the dirty pages in a log. Once it complies to this logging, it may
617 declare the ``VHOST_F_LOG_ALL`` vhost feature.
618
619 To start/stop logging of data/used ring writes, the front-end may send
620 messages ``VHOST_USER_SET_FEATURES`` with ``VHOST_F_LOG_ALL`` and
621 ``VHOST_USER_SET_VRING_ADDR`` with ``VHOST_VRING_F_LOG`` in ring's
622 flags set to 1/0, respectively.
623
624 All the modifications to memory pointed by vring "descriptor" should
625 be marked. Modifications to "used" vring should be marked if
626 ``VHOST_VRING_F_LOG`` is part of ring's flags.
627
628 Dirty pages are of size::
629
630 #define VHOST_LOG_PAGE 0x1000
631
632 The log memory fd is provided in the ancillary data of
633 ``VHOST_USER_SET_LOG_BASE`` message when the back-end has
634 ``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature.
635
636 The size of the log is supplied as part of ``VhostUserMsg`` which
637 should be large enough to cover all known guest addresses. Log starts
638 at the supplied offset in the supplied file descriptor. The log
639 covers from address 0 to the maximum of guest regions. In pseudo-code,
640 to mark page at ``addr`` as dirty::
641
642 page = addr / VHOST_LOG_PAGE
643 log[page / 8] |= 1 << page % 8
644
645 Where ``addr`` is the guest physical address.
646
647 Use atomic operations, as the log may be concurrently manipulated.
648
649 Note that when logging modifications to the used ring (when
650 ``VHOST_VRING_F_LOG`` is set for this ring), ``log_guest_addr`` should
651 be used to calculate the log offset: the write to first byte of the
652 used ring is logged at this offset from log start. Also note that this
653 value might be outside the legal guest physical address range
654 (i.e. does not have to be covered by the ``VhostUserMemory`` table), but
655 the bit offset of the last byte of the ring must fall within the size
656 supplied by ``VhostUserLog``.
657
658 ``VHOST_USER_SET_LOG_FD`` is an optional message with an eventfd in
659 ancillary data, it may be used to inform the front-end that the log has
660 been modified.
661
662 Once the source has finished migration, rings will be stopped by the
663 source (:ref:`Suspended device state <suspended_device_state>`). No
664 further update must be done before rings are restarted.
665
666 In postcopy migration the back-end is started before all the memory has
667 been received from the source host, and care must be taken to avoid
668 accessing pages that have yet to be received. The back-end opens a
669 'userfault'-fd and registers the memory with it; this fd is then
670 passed back over to the front-end. The front-end services requests on the
671 userfaultfd for pages that are accessed and when the page is available
672 it performs WAKE ioctl's on the userfaultfd to wake the stalled
673 back-end. The front-end indicates support for this via the
674 ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` feature.
675
676 .. _migrating_backend_state:
677
678 Migrating back-end state
679 ^^^^^^^^^^^^^^^^^^^^^^^^
680
681 Migrating device state involves transferring the state from one
682 back-end, called the source, to another back-end, called the
683 destination. After migration, the destination transparently resumes
684 operation without requiring the driver to re-initialize the device at
685 the VIRTIO level. If the migration fails, then the source can
686 transparently resume operation until another migration attempt is made.
687
688 Generally, the front-end is connected to a virtual machine guest (which
689 contains the driver), which has its own state to transfer between source
690 and destination, and therefore will have an implementation-specific
691 mechanism to do so. The ``VHOST_USER_PROTOCOL_F_DEVICE_STATE`` feature
692 provides functionality to have the front-end include the back-end's
693 state in this transfer operation so the back-end does not need to
694 implement its own mechanism, and so the virtual machine may have its
695 complete state, including vhost-user devices' states, contained within a
696 single stream of data.
697
698 To do this, the back-end state is transferred from back-end to front-end
699 on the source side, and vice versa on the destination side. This
700 transfer happens over a channel that is negotiated using the
701 ``VHOST_USER_SET_DEVICE_STATE_FD`` message. This message has two
702 parameters:
703
704 * Direction of transfer: On the source, the data is saved, transferring
705 it from the back-end to the front-end. On the destination, the data
706 is loaded, transferring it from the front-end to the back-end.
707
708 * Migration phase: Currently, the only supported phase is the period
709 after the transfer of memory-mapped regions before switch-over to the
710 destination, when both the source and destination devices are
711 suspended (:ref:`Suspended device state <suspended_device_state>`).
712 In the future, additional phases might be supported to allow iterative
713 migration while the device is running.
714
715 The nature of the channel is implementation-defined, but it must
716 generally behave like a pipe: The writing end will write all the data it
717 has into it, signalling the end of data by closing its end. The reading
718 end must read all of this data (until encountering the end of file) and
719 process it.
720
721 * When saving, the writing end is the source back-end, and the reading
722 end is the source front-end. After reading the state data from the
723 channel, the source front-end must transfer it to the destination
724 front-end through an implementation-defined mechanism.
725
726 * When loading, the writing end is the destination front-end, and the
727 reading end is the destination back-end. After reading the state data
728 from the channel, the destination back-end must deserialize its
729 internal state from that data and set itself up to allow the driver to
730 seamlessly resume operation on the VIRTIO level.
731
732 Seamlessly resuming operation means that the migration must be
733 transparent to the guest driver, which operates on the VIRTIO level.
734 This driver will not perform any re-initialization steps, but continue
735 to use the device as if no migration had occurred. The vhost-user
736 front-end, however, will re-initialize the vhost state on the
737 destination, following the usual protocol for establishing a connection
738 to a vhost-user back-end: This includes, for example, setting up memory
739 mappings and kick and call FDs as necessary, negotiating protocol
740 features, or setting the initial vring base indices (to the same value
741 as on the source side, so that operation can resume). The vhost-user front-end
742 may also write to the kick FDs of vrings containing unused buffers or send
743 ``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
744 since the driver likely already kicked them in the source and won't do it again.
745
746 Both on the source and on the destination side, after the respective
747 front-end has seen all data transferred (when the transfer FD has been
748 closed), it sends the ``VHOST_USER_CHECK_DEVICE_STATE`` message to
749 verify that data transfer was successful in the back-end, too. The
750 back-end responds once it knows whether the transfer and processing was
751 successful or not.
752
753 Memory access
754 -------------
755
756 The front-end sends a list of vhost memory regions to the back-end using the
757 ``VHOST_USER_SET_MEM_TABLE`` message. Each region has two base
758 addresses: a guest address and a user address.
759
760 Memory regions can be added via the ``VHOST_USER_ADD_MEM_REG`` message. They
761 can be removed via the ``VHOST_USER_REM_MEM_REG`` message. These messages can
762 only be used if the ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol
763 feature has been successfully negotiated.
764
765 Guest addresses are physical addresses in the guest. User addresses are
766 arbitrary opaque values, though they typically refer to userspace addresses in
767 the client process.
768
769 Messages contain guest addresses and/or user addresses to reference locations
770 within the shared memory. The mapping of these addresses works as follows.
771
772 User addresses map to the vhost memory region containing that user address.
773
774 When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has not been negotiated:
775
776 * Guest addresses map to the vhost memory region containing that guest
777 address.
778
779 When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated:
780
781 * Guest addresses are also called I/O virtual addresses (IOVAs). They are
782 translated to user addresses via the IOTLB.
783
784 * The vhost memory region guest address is not used.
785
786 IOMMU support
787 -------------
788
789 When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated, the
790 front-end sends IOTLB entries update & invalidation by sending
791 ``VHOST_USER_IOTLB_MSG`` requests to the back-end with a ``struct
792 vhost_iotlb_msg`` as payload. For update events, the ``iotlb`` payload
793 has to be filled with the update message type (2), the I/O virtual
794 address, the size, the user virtual address, and the permissions
795 flags. Addresses and size must be within vhost memory regions set via
796 the ``VHOST_USER_SET_MEM_TABLE`` request. For invalidation events, the
797 ``iotlb`` payload has to be filled with the invalidation message type
798 (3), the I/O virtual address and the size. On success, the back-end is
799 expected to reply with a zero payload, non-zero otherwise.
800
801 The back-end relies on the back-end communication channel (see :ref:`Back-end
802 communication <backend_communication>` section below) to send IOTLB miss
803 and access failure events, by sending ``VHOST_USER_BACKEND_IOTLB_MSG``
804 requests to the front-end with a ``struct vhost_iotlb_msg`` as
805 payload. For miss events, the iotlb payload has to be filled with the
806 miss message type (1), the I/O virtual address and the permissions
807 flags. For access failure event, the iotlb payload has to be filled
808 with the access failure message type (4), the I/O virtual address and
809 the permissions flags. For synchronization purpose, the back-end may
810 rely on the reply-ack feature, so the front-end may send a reply when
811 operation is completed if the reply-ack feature is negotiated and
812 back-ends requests a reply. For miss events, completed operation means
813 either front-end sent an update message containing the IOTLB entry
814 containing requested address and permission, or front-end sent nothing if
815 the IOTLB miss message is invalid (invalid IOVA or permission).
816
817 The front-end isn't expected to take the initiative to send IOTLB update
818 messages, as the back-end sends IOTLB miss messages for the guest virtual
819 memory areas it needs to access.
820
821 .. _backend_communication:
822
823 Back-end communication
824 ----------------------
825
826 An optional communication channel is provided if the back-end declares
827 ``VHOST_USER_PROTOCOL_F_BACKEND_REQ`` protocol feature, to allow the
828 back-end to make requests to the front-end.
829
830 The fd is provided via ``VHOST_USER_SET_BACKEND_REQ_FD`` ancillary data.
831
832 A back-end may then send ``VHOST_USER_BACKEND_*`` messages to the front-end
833 using this fd communication channel.
834
835 If ``VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD`` protocol feature is
836 negotiated, back-end can send file descriptors (at most 8 descriptors in
837 each message) to front-end via ancillary data using this fd communication
838 channel.
839
840 .. _inflight_io_tracking:
841
842 Inflight I/O tracking
843 ---------------------
844
845 To support reconnecting after restart or crash, back-end may need to
846 resubmit inflight I/Os. If virtqueue is processed in order, we can
847 easily achieve that by getting the inflight descriptors from
848 descriptor table (split virtqueue) or descriptor ring (packed
849 virtqueue). However, it can't work when we process descriptors
850 out-of-order because some entries which store the information of
851 inflight descriptors in available ring (split virtqueue) or descriptor
852 ring (packed virtqueue) might be overridden by new entries. To solve
853 this problem, the back-end need to allocate an extra buffer to store this
854 information of inflight descriptors and share it with front-end for
855 persistent. ``VHOST_USER_GET_INFLIGHT_FD`` and
856 ``VHOST_USER_SET_INFLIGHT_FD`` are used to transfer this buffer
857 between front-end and back-end. And the format of this buffer is described
858 below:
859
860 +---------------+---------------+-----+---------------+
861 | queue0 region | queue1 region | ... | queueN region |
862 +---------------+---------------+-----+---------------+
863
864 N is the number of available virtqueues. The back-end could get it from num
865 queues field of ``VhostUserInflight``.
866
867 For split virtqueue, queue region can be implemented as:
868
869 .. code:: c
870
871 typedef struct DescStateSplit {
872 /* Indicate whether this descriptor is inflight or not.
873 * Only available for head-descriptor. */
874 uint8_t inflight;
875
876 /* Padding */
877 uint8_t padding[5];
878
879 /* Maintain a list for the last batch of used descriptors.
880 * Only available when batching is used for submitting */
881 uint16_t next;
882
883 /* Used to preserve the order of fetching available descriptors.
884 * Only available for head-descriptor. */
885 uint64_t counter;
886 } DescStateSplit;
887
888 typedef struct QueueRegionSplit {
889 /* The feature flags of this region. Now it's initialized to 0. */
890 uint64_t features;
891
892 /* The version of this region. It's 1 currently.
893 * Zero value indicates an uninitialized buffer */
894 uint16_t version;
895
896 /* The size of DescStateSplit array. It's equal to the virtqueue size.
897 * The back-end could get it from queue size field of VhostUserInflight. */
898 uint16_t desc_num;
899
900 /* The head of list that track the last batch of used descriptors. */
901 uint16_t last_batch_head;
902
903 /* Store the idx value of used ring */
904 uint16_t used_idx;
905
906 /* Used to track the state of each descriptor in descriptor table */
907 DescStateSplit desc[];
908 } QueueRegionSplit;
909
910 To track inflight I/O, the queue region should be processed as follows:
911
912 When receiving available buffers from the driver:
913
914 #. Get the next available head-descriptor index from available ring, ``i``
915
916 #. Set ``desc[i].counter`` to the value of global counter
917
918 #. Increase global counter by 1
919
920 #. Set ``desc[i].inflight`` to 1
921
922 When supplying used buffers to the driver:
923
924 1. Get corresponding used head-descriptor index, i
925
926 2. Set ``desc[i].next`` to ``last_batch_head``
927
928 3. Set ``last_batch_head`` to ``i``
929
930 #. Steps 1,2,3 may be performed repeatedly if batching is possible
931
932 #. Increase the ``idx`` value of used ring by the size of the batch
933
934 #. Set the ``inflight`` field of each ``DescStateSplit`` entry in the batch to 0
935
936 #. Set ``used_idx`` to the ``idx`` value of used ring
937
938 When reconnecting:
939
940 #. If the value of ``used_idx`` does not match the ``idx`` value of
941 used ring (means the inflight field of ``DescStateSplit`` entries in
942 last batch may be incorrect),
943
944 a. Subtract the value of ``used_idx`` from the ``idx`` value of
945 used ring to get last batch size of ``DescStateSplit`` entries
946
947 #. Set the ``inflight`` field of each ``DescStateSplit`` entry to 0 in last batch
948 list which starts from ``last_batch_head``
949
950 #. Set ``used_idx`` to the ``idx`` value of used ring
951
952 #. Resubmit inflight ``DescStateSplit`` entries in order of their
953 counter value
954
955 For packed virtqueue, queue region can be implemented as:
956
957 .. code:: c
958
959 typedef struct DescStatePacked {
960 /* Indicate whether this descriptor is inflight or not.
961 * Only available for head-descriptor. */
962 uint8_t inflight;
963
964 /* Padding */
965 uint8_t padding;
966
967 /* Link to the next free entry */
968 uint16_t next;
969
970 /* Link to the last entry of descriptor list.
971 * Only available for head-descriptor. */
972 uint16_t last;
973
974 /* The length of descriptor list.
975 * Only available for head-descriptor. */
976 uint16_t num;
977
978 /* Used to preserve the order of fetching available descriptors.
979 * Only available for head-descriptor. */
980 uint64_t counter;
981
982 /* The buffer id */
983 uint16_t id;
984
985 /* The descriptor flags */
986 uint16_t flags;
987
988 /* The buffer length */
989 uint32_t len;
990
991 /* The buffer address */
992 uint64_t addr;
993 } DescStatePacked;
994
995 typedef struct QueueRegionPacked {
996 /* The feature flags of this region. Now it's initialized to 0. */
997 uint64_t features;
998
999 /* The version of this region. It's 1 currently.
1000 * Zero value indicates an uninitialized buffer */
1001 uint16_t version;
1002
1003 /* The size of DescStatePacked array. It's equal to the virtqueue size.
1004 * The back-end could get it from queue size field of VhostUserInflight. */
1005 uint16_t desc_num;
1006
1007 /* The head of free DescStatePacked entry list */
1008 uint16_t free_head;
1009
1010 /* The old head of free DescStatePacked entry list */
1011 uint16_t old_free_head;
1012
1013 /* The used index of descriptor ring */
1014 uint16_t used_idx;
1015
1016 /* The old used index of descriptor ring */
1017 uint16_t old_used_idx;
1018
1019 /* Device ring wrap counter */
1020 uint8_t used_wrap_counter;
1021
1022 /* The old device ring wrap counter */
1023 uint8_t old_used_wrap_counter;
1024
1025 /* Padding */
1026 uint8_t padding[7];
1027
1028 /* Used to track the state of each descriptor fetched from descriptor ring */
1029 DescStatePacked desc[];
1030 } QueueRegionPacked;
1031
1032 To track inflight I/O, the queue region should be processed as follows:
1033
1034 When receiving available buffers from the driver:
1035
1036 #. Get the next available descriptor entry from descriptor ring, ``d``
1037
1038 #. If ``d`` is head descriptor,
1039
1040 a. Set ``desc[old_free_head].num`` to 0
1041
1042 #. Set ``desc[old_free_head].counter`` to the value of global counter
1043
1044 #. Increase global counter by 1
1045
1046 #. Set ``desc[old_free_head].inflight`` to 1
1047
1048 #. If ``d`` is last descriptor, set ``desc[old_free_head].last`` to
1049 ``free_head``
1050
1051 #. Increase ``desc[old_free_head].num`` by 1
1052
1053 #. Set ``desc[free_head].addr``, ``desc[free_head].len``,
1054 ``desc[free_head].flags``, ``desc[free_head].id`` to ``d.addr``,
1055 ``d.len``, ``d.flags``, ``d.id``
1056
1057 #. Set ``free_head`` to ``desc[free_head].next``
1058
1059 #. If ``d`` is last descriptor, set ``old_free_head`` to ``free_head``
1060
1061 When supplying used buffers to the driver:
1062
1063 1. Get corresponding used head-descriptor entry from descriptor ring,
1064 ``d``
1065
1066 2. Get corresponding ``DescStatePacked`` entry, ``e``
1067
1068 3. Set ``desc[e.last].next`` to ``free_head``
1069
1070 4. Set ``free_head`` to the index of ``e``
1071
1072 #. Steps 1,2,3,4 may be performed repeatedly if batching is possible
1073
1074 #. Increase ``used_idx`` by the size of the batch and update
1075 ``used_wrap_counter`` if needed
1076
1077 #. Update ``d.flags``
1078
1079 #. Set the ``inflight`` field of each head ``DescStatePacked`` entry
1080 in the batch to 0
1081
1082 #. Set ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
1083 to ``free_head``, ``used_idx``, ``used_wrap_counter``
1084
1085 When reconnecting:
1086
1087 #. If ``used_idx`` does not match ``old_used_idx`` (means the
1088 ``inflight`` field of ``DescStatePacked`` entries in last batch may
1089 be incorrect),
1090
1091 a. Get the next descriptor ring entry through ``old_used_idx``, ``d``
1092
1093 #. Use ``old_used_wrap_counter`` to calculate the available flags
1094
1095 #. If ``d.flags`` is not equal to the calculated flags value (means
1096 back-end has submitted the buffer to guest driver before crash, so
1097 it has to commit the in-progress update), set ``old_free_head``,
1098 ``old_used_idx``, ``old_used_wrap_counter`` to ``free_head``,
1099 ``used_idx``, ``used_wrap_counter``
1100
1101 #. Set ``free_head``, ``used_idx``, ``used_wrap_counter`` to
1102 ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
1103 (roll back any in-progress update)
1104
1105 #. Set the ``inflight`` field of each ``DescStatePacked`` entry in
1106 free list to 0
1107
1108 #. Resubmit inflight ``DescStatePacked`` entries in order of their
1109 counter value
1110
1111 In-band notifications
1112 ---------------------
1113
1114 In some limited situations (e.g. for simulation) it is desirable to
1115 have the kick, call and error (if used) signals done via in-band
1116 messages instead of asynchronous eventfd notifications. This can be
1117 done by negotiating the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS``
1118 protocol feature.
1119
1120 Note that due to the fact that too many messages on the sockets can
1121 cause the sending application(s) to block, it is not advised to use
1122 this feature unless absolutely necessary. It is also considered an
1123 error to negotiate this feature without also negotiating
1124 ``VHOST_USER_PROTOCOL_F_BACKEND_REQ`` and ``VHOST_USER_PROTOCOL_F_REPLY_ACK``,
1125 the former is necessary for getting a message channel from the back-end
1126 to the front-end, while the latter needs to be used with the in-band
1127 notification messages to block until they are processed, both to avoid
1128 blocking later and for proper processing (at least in the simulation
1129 use case.) As it has no other way of signalling this error, the back-end
1130 should close the connection as a response to a
1131 ``VHOST_USER_SET_PROTOCOL_FEATURES`` message that sets the in-band
1132 notifications feature flag without the other two.
1133
1134 Protocol features
1135 -----------------
1136
1137 .. code:: c
1138
1139 #define VHOST_USER_PROTOCOL_F_MQ 0
1140 #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
1141 #define VHOST_USER_PROTOCOL_F_RARP 2
1142 #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3
1143 #define VHOST_USER_PROTOCOL_F_MTU 4
1144 #define VHOST_USER_PROTOCOL_F_BACKEND_REQ 5
1145 #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
1146 #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7
1147 #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8
1148 #define VHOST_USER_PROTOCOL_F_CONFIG 9
1149 #define VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD 10
1150 #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11
1151 #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12
1152 #define VHOST_USER_PROTOCOL_F_RESET_DEVICE 13
1153 #define VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS 14
1154 #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
1155 #define VHOST_USER_PROTOCOL_F_STATUS 16
1156 #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
1157 #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT 18
1158 #define VHOST_USER_PROTOCOL_F_DEVICE_STATE 19
1159 #define VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT 20
1160 #define VHOST_USER_PROTOCOL_F_GPA_ADDRESSES 21
1161 #define VHOST_USER_PROTOCOL_F_SHMEM_MAP 22
1162
1163 Front-end message types
1164 -----------------------
1165
1166 ``VHOST_USER_GET_FEATURES``
1167 :id: 1
1168 :equivalent ioctl: ``VHOST_GET_FEATURES``
1169 :request payload: N/A
1170 :reply payload: ``u64``
1171
1172 Get from the underlying vhost implementation the features bitmask.
1173 Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals back-end support
1174 for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
1175 ``VHOST_USER_SET_PROTOCOL_FEATURES``.
1176
1177 ``VHOST_USER_SET_FEATURES``
1178 :id: 2
1179 :equivalent ioctl: ``VHOST_SET_FEATURES``
1180 :request payload: ``u64``
1181 :reply payload: N/A
1182
1183 Enable features in the underlying vhost implementation using a
1184 bitmask. Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals
1185 back-end support for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
1186 ``VHOST_USER_SET_PROTOCOL_FEATURES``.
1187
1188 ``VHOST_USER_GET_PROTOCOL_FEATURES``
1189 :id: 15
1190 :equivalent ioctl: ``VHOST_GET_FEATURES``
1191 :request payload: N/A
1192 :reply payload: ``u64``
1193
1194 Get the protocol feature bitmask from the underlying vhost
1195 implementation. Only legal if feature bit
1196 ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
1197 ``VHOST_USER_GET_FEATURES``. It does not need to be acknowledged by
1198 ``VHOST_USER_SET_FEATURES``.
1199
1200 .. Note::
1201 Back-ends that report ``VHOST_USER_F_PROTOCOL_FEATURES`` must
1202 support this message even before ``VHOST_USER_SET_FEATURES`` was
1203 called.
1204
1205 ``VHOST_USER_SET_PROTOCOL_FEATURES``
1206 :id: 16
1207 :equivalent ioctl: ``VHOST_SET_FEATURES``
1208 :request payload: ``u64``
1209 :reply payload: N/A
1210
1211 Enable protocol features in the underlying vhost implementation.
1212
1213 Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
1214 ``VHOST_USER_GET_FEATURES``. It does not need to be acknowledged by
1215 ``VHOST_USER_SET_FEATURES``.
1216
1217 .. Note::
1218 Back-ends that report ``VHOST_USER_F_PROTOCOL_FEATURES`` must support
1219 this message even before ``VHOST_USER_SET_FEATURES`` was called.
1220
1221 ``VHOST_USER_SET_OWNER``
1222 :id: 3
1223 :equivalent ioctl: ``VHOST_SET_OWNER``
1224 :request payload: N/A
1225 :reply payload: N/A
1226
1227 Issued when a new connection is established. It marks the sender
1228 as the front-end that owns of the session. This can be used on the *back-end*
1229 as a "session start" flag.
1230
1231 ``VHOST_USER_RESET_OWNER``
1232 :id: 4
1233 :request payload: N/A
1234 :reply payload: N/A
1235
1236 .. admonition:: Deprecated
1237
1238 This is no longer used. Used to be sent to request disabling all
1239 rings, but some back-ends interpreted it to also discard connection
1240 state (this interpretation would lead to bugs). It is recommended
1241 that back-ends either ignore this message, or use it to disable all
1242 rings.
1243
1244 ``VHOST_USER_SET_MEM_TABLE``
1245 :id: 5
1246 :equivalent ioctl: ``VHOST_SET_MEM_TABLE``
1247 :request payload: multiple memory regions description
1248 :reply payload: (postcopy only) multiple memory regions description
1249
1250 Sets the memory map regions on the back-end so it can translate the
1251 vring addresses. In the ancillary data there is an array of file
1252 descriptors for each memory mapped region. The size and ordering of
1253 the fds matches the number and ordering of memory regions.
1254
1255 When ``VHOST_USER_POSTCOPY_LISTEN`` has been received,
1256 ``SET_MEM_TABLE`` replies with the bases of the memory mapped
1257 regions to the front-end. The back-end must have mmap'd the regions but
1258 not yet accessed them and should not yet generate a userfault
1259 event.
1260
1261 .. Note::
1262 ``NEED_REPLY_MASK`` is not set in this case. QEMU will then
1263 reply back to the list of mappings with an empty
1264 ``VHOST_USER_SET_MEM_TABLE`` as an acknowledgement; only upon
1265 reception of this message may the guest start accessing the memory
1266 and generating faults.
1267
1268 ``VHOST_USER_SET_LOG_BASE``
1269 :id: 6
1270 :equivalent ioctl: ``VHOST_SET_LOG_BASE``
1271 :request payload: u64
1272 :reply payload: N/A
1273
1274 Sets logging shared memory space.
1275
1276 When the back-end has ``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature,
1277 the log memory fd is provided in the ancillary data of
1278 ``VHOST_USER_SET_LOG_BASE`` message, the size and offset of shared
1279 memory area provided in the message.
1280
1281 ``VHOST_USER_SET_LOG_FD``
1282 :id: 7
1283 :equivalent ioctl: ``VHOST_SET_LOG_FD``
1284 :request payload: N/A
1285 :reply payload: N/A
1286
1287 Sets the logging file descriptor, which is passed as ancillary data.
1288
1289 ``VHOST_USER_SET_VRING_NUM``
1290 :id: 8
1291 :equivalent ioctl: ``VHOST_SET_VRING_NUM``
1292 :request payload: vring state description
1293 :reply payload: N/A
1294
1295 Set the size of the queue.
1296
1297 ``VHOST_USER_SET_VRING_ADDR``
1298 :id: 9
1299 :equivalent ioctl: ``VHOST_SET_VRING_ADDR``
1300 :request payload: vring address description
1301 :reply payload: N/A
1302
1303 Sets the addresses of the different aspects of the vring.
1304
1305 ``VHOST_USER_SET_VRING_BASE``
1306 :id: 10
1307 :equivalent ioctl: ``VHOST_SET_VRING_BASE``
1308 :request payload: vring descriptor index/indices
1309 :reply payload: N/A
1310
1311 Sets the next index to use for descriptors in this vring:
1312
1313 * For a split virtqueue, sets only the next descriptor index to
1314 process in the *Available Ring*. The device is supposed to read the
1315 next index in the *Used Ring* from the respective vring structure in
1316 guest memory.
1317
1318 * For a packed virtqueue, both indices are supplied, as they are not
1319 explicitly available in memory.
1320
1321 Consequently, the payload type is specific to the type of virt queue
1322 (*a vring descriptor index for split virtqueues* vs. *vring descriptor
1323 indices for packed virtqueues*).
1324
1325 ``VHOST_USER_GET_VRING_BASE``
1326 :id: 11
1327 :equivalent ioctl: ``VHOST_USER_GET_VRING_BASE``
1328 :request payload: vring state description
1329 :reply payload: vring descriptor index/indices
1330
1331 Stops the vring and returns the current descriptor index or indices:
1332
1333 * For a split virtqueue, returns only the 16-bit next descriptor
1334 index to process in the *Available Ring*. Note that this may
1335 differ from the available ring index in the vring structure in
1336 memory, which points to where the driver will put new available
1337 descriptors. For the *Used Ring*, the device only needs the next
1338 descriptor index at which to put new descriptors, which is the
1339 value in the vring structure in memory, so this value is not
1340 covered by this message.
1341
1342 * For a packed virtqueue, neither index is explicitly available to
1343 read from memory, so both indices (as maintained by the device) are
1344 returned.
1345
1346 Consequently, the payload type is specific to the type of virt queue
1347 (*a vring descriptor index for split virtqueues* vs. *vring descriptor
1348 indices for packed virtqueues*).
1349
1350 When and as long as all of a device's vrings are stopped, it is
1351 *suspended*, see :ref:`Suspended device state
1352 <suspended_device_state>`.
1353
1354 The request payload's *num* field is currently reserved and must be
1355 set to 0.
1356
1357 By default, the back-end must complete all inflight I/O requests for the
1358 specified vring before stopping it.
1359
1360 If the ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT`` protocol
1361 feature has been negotiated, the back-end may suspend in-flight I/O
1362 requests and record them as described in :ref:`Inflight I/O tracking
1363 <inflight_io_tracking>` instead of completing them before stopping the vring.
1364 How to suspend an in-flight request depends on the implementation of the back-end
1365 but it typically can be done by aborting or cancelling the underlying I/O
1366 request. The ``VHOST_USER_PROTOCOL_F_GET_VRING_BASE_INFLIGHT``
1367 protocol feature must only be negotiated if
1368 ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` is also negotiated.
1369
1370 ``VHOST_USER_SET_VRING_KICK``
1371 :id: 12
1372 :equivalent ioctl: ``VHOST_SET_VRING_KICK``
1373 :request payload: ``u64``
1374 :reply payload: N/A
1375
1376 Set the event file descriptor for adding buffers to the vring. It is
1377 passed in the ancillary data.
1378
1379 Bits (0-7) of the payload contain the vring index. Bit 8 is the
1380 invalid FD flag. This flag is set when there is no file descriptor
1381 in the ancillary data. This signals that polling should be used
1382 instead of waiting for the kick. Note that if the protocol feature
1383 ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` has been negotiated
1384 this message isn't necessary as the ring is also started on the
1385 ``VHOST_USER_VRING_KICK`` message, it may however still be used to
1386 set an event file descriptor (which will be preferred over the
1387 message) or to enable polling.
1388
1389 ``VHOST_USER_SET_VRING_CALL``
1390 :id: 13
1391 :equivalent ioctl: ``VHOST_SET_VRING_CALL``
1392 :request payload: ``u64``
1393 :reply payload: N/A
1394
1395 Set the event file descriptor to signal when buffers are used. It is
1396 passed in the ancillary data.
1397
1398 Bits (0-7) of the payload contain the vring index. Bit 8 is the
1399 invalid FD flag. This flag is set when there is no file descriptor
1400 in the ancillary data. This signals that polling will be used
1401 instead of waiting for the call. Note that if the protocol features
1402 ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` and
1403 ``VHOST_USER_PROTOCOL_F_BACKEND_REQ`` have been negotiated this message
1404 isn't necessary as the ``VHOST_USER_BACKEND_VRING_CALL`` message can be
1405 used, it may however still be used to set an event file descriptor
1406 or to enable polling.
1407
1408 ``VHOST_USER_SET_VRING_ERR``
1409 :id: 14
1410 :equivalent ioctl: ``VHOST_SET_VRING_ERR``
1411 :request payload: ``u64``
1412 :reply payload: N/A
1413
1414 Set the event file descriptor to signal when error occurs. It is
1415 passed in the ancillary data.
1416
1417 Bits (0-7) of the payload contain the vring index. Bit 8 is the
1418 invalid FD flag. This flag is set when there is no file descriptor
1419 in the ancillary data. Note that if the protocol features
1420 ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` and
1421 ``VHOST_USER_PROTOCOL_F_BACKEND_REQ`` have been negotiated this message
1422 isn't necessary as the ``VHOST_USER_BACKEND_VRING_ERR`` message can be
1423 used, it may however still be used to set an event file descriptor
1424 (which will be preferred over the message).
1425
1426 ``VHOST_USER_GET_QUEUE_NUM``
1427 :id: 17
1428 :equivalent ioctl: N/A
1429 :request payload: N/A
1430 :reply payload: u64
1431
1432 Query how many queues the back-end supports.
1433
1434 This request should be sent only when ``VHOST_USER_PROTOCOL_F_MQ``
1435 is set in queried protocol features by
1436 ``VHOST_USER_GET_PROTOCOL_FEATURES``.
1437
1438 ``VHOST_USER_SET_VRING_ENABLE``
1439 :id: 18
1440 :equivalent ioctl: N/A
1441 :request payload: vring state description
1442 :reply payload: N/A
1443
1444 Signal the back-end to enable or disable corresponding vring.
1445
1446 This request should be sent only when
1447 ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated.
1448
1449 ``VHOST_USER_SEND_RARP``
1450 :id: 19
1451 :equivalent ioctl: N/A
1452 :request payload: ``u64``
1453 :reply payload: N/A
1454
1455 Ask vhost user back-end to broadcast a fake RARP to notify the migration
1456 is terminated for guest that does not support GUEST_ANNOUNCE.
1457
1458 Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is
1459 present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
1460 ``VHOST_USER_PROTOCOL_F_RARP`` is present in
1461 ``VHOST_USER_GET_PROTOCOL_FEATURES``. The first 6 bytes of the
1462 payload contain the mac address of the guest to allow the vhost user
1463 back-end to construct and broadcast the fake RARP.
1464
1465 ``VHOST_USER_NET_SET_MTU``
1466 :id: 20
1467 :equivalent ioctl: N/A
1468 :request payload: ``u64``
1469 :reply payload: N/A
1470
1471 Set host MTU value exposed to the guest.
1472
1473 This request should be sent only when ``VIRTIO_NET_F_MTU`` feature
1474 has been successfully negotiated, ``VHOST_USER_F_PROTOCOL_FEATURES``
1475 is present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
1476 ``VHOST_USER_PROTOCOL_F_NET_MTU`` is present in
1477 ``VHOST_USER_GET_PROTOCOL_FEATURES``.
1478
1479 If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, the back-end must
1480 respond with zero in case the specified MTU is valid, or non-zero
1481 otherwise.
1482
1483 ``VHOST_USER_SET_BACKEND_REQ_FD`` (previous name ``VHOST_USER_SET_SLAVE_REQ_FD``)
1484 :id: 21
1485 :equivalent ioctl: N/A
1486 :request payload: N/A
1487 :reply payload: N/A
1488
1489 Set the socket file descriptor for back-end initiated requests. It is passed
1490 in the ancillary data.
1491
1492 This request should be sent only when
1493 ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, and protocol
1494 feature bit ``VHOST_USER_PROTOCOL_F_BACKEND_REQ`` bit is present in
1495 ``VHOST_USER_GET_PROTOCOL_FEATURES``. If
1496 ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, the back-end must
1497 respond with zero for success, non-zero otherwise.
1498
1499 ``VHOST_USER_IOTLB_MSG``
1500 :id: 22
1501 :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
1502 :request payload: ``struct vhost_iotlb_msg``
1503 :reply payload: ``u64``
1504
1505 Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
1506
1507 The front-end sends such requests to update and invalidate entries in the
1508 device IOTLB. The back-end has to acknowledge the request with sending
1509 zero as ``u64`` payload for success, non-zero otherwise.
1510
1511 This request should be send only when ``VIRTIO_F_IOMMU_PLATFORM``
1512 feature has been successfully negotiated.
1513
1514 ``VHOST_USER_SET_VRING_ENDIAN``
1515 :id: 23
1516 :equivalent ioctl: ``VHOST_SET_VRING_ENDIAN``
1517 :request payload: vring state description
1518 :reply payload: N/A
1519
1520 Set the endianness of a VQ for legacy devices. Little-endian is
1521 indicated with state.num set to 0 and big-endian is indicated with
1522 state.num set to 1. Other values are invalid.
1523
1524 This request should be sent only when
1525 ``VHOST_USER_PROTOCOL_F_CROSS_ENDIAN`` has been negotiated.
1526 Backends that negotiated this feature should handle both
1527 endiannesses and expect this message once (per VQ) during device
1528 configuration (ie. before the front-end starts the VQ).
1529
1530 ``VHOST_USER_GET_CONFIG``
1531 :id: 24
1532 :equivalent ioctl: N/A
1533 :request payload: virtio device config space
1534 :reply payload: virtio device config space
1535
1536 When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
1537 submitted by the vhost-user front-end to fetch the contents of the
1538 virtio device configuration space, vhost-user back-end's payload size
1539 MUST match the front-end's request, vhost-user back-end uses zero length of
1540 payload to indicate an error to the vhost-user front-end. The vhost-user
1541 front-end may cache the contents to avoid repeated
1542 ``VHOST_USER_GET_CONFIG`` calls.
1543
1544 ``VHOST_USER_SET_CONFIG``
1545 :id: 25
1546 :equivalent ioctl: N/A
1547 :request payload: virtio device config space
1548 :reply payload: N/A
1549
1550 When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
1551 submitted by the vhost-user front-end when the Guest changes the virtio
1552 device configuration space and also can be used for live migration
1553 on the destination host. The vhost-user back-end must check the flags
1554 field, and back-ends MUST NOT accept SET_CONFIG for read-only
1555 configuration space fields unless the live migration bit is set.
1556
1557 ``VHOST_USER_CREATE_CRYPTO_SESSION``
1558 :id: 26
1559 :equivalent ioctl: N/A
1560 :request payload: crypto session description
1561 :reply payload: crypto session description
1562
1563 Create a session for crypto operation. The back-end must return
1564 the session id, 0 or positive for success, negative for failure.
1565 This request should be sent only when
1566 ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
1567 successfully negotiated. It's a required feature for crypto
1568 devices.
1569
1570 ``VHOST_USER_CLOSE_CRYPTO_SESSION``
1571 :id: 27
1572 :equivalent ioctl: N/A
1573 :request payload: ``u64``
1574 :reply payload: N/A
1575
1576 Close a session for crypto operation which was previously
1577 created by ``VHOST_USER_CREATE_CRYPTO_SESSION``.
1578
1579 This request should be sent only when
1580 ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
1581 successfully negotiated. It's a required feature for crypto
1582 devices.
1583
1584 ``VHOST_USER_POSTCOPY_ADVISE``
1585 :id: 28
1586 :request payload: N/A
1587 :reply payload: userfault fd
1588
1589 When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, the front-end
1590 advises back-end that a migration with postcopy enabled is underway,
1591 the back-end must open a userfaultfd for later use. Note that at this
1592 stage the migration is still in precopy mode.
1593
1594 ``VHOST_USER_POSTCOPY_LISTEN``
1595 :id: 29
1596 :request payload: N/A
1597 :reply payload: N/A
1598
1599 The front-end advises back-end that a transition to postcopy mode has
1600 happened. The back-end must ensure that shared memory is registered
1601 with userfaultfd to cause faulting of non-present pages.
1602
1603 This is always sent sometime after a ``VHOST_USER_POSTCOPY_ADVISE``,
1604 and thus only when ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported.
1605
1606 ``VHOST_USER_POSTCOPY_END``
1607 :id: 30
1608 :request payload: N/A
1609 :reply payload: ``u64``
1610
1611 The front-end advises that postcopy migration has now completed. The back-end
1612 must disable the userfaultfd. The reply is an acknowledgement
1613 only.
1614
1615 When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, this message
1616 is sent at the end of the migration, after
1617 ``VHOST_USER_POSTCOPY_LISTEN`` was previously sent.
1618
1619 The value returned is an error indication; 0 is success.
1620
1621 ``VHOST_USER_GET_INFLIGHT_FD``
1622 :id: 31
1623 :equivalent ioctl: N/A
1624 :request payload: inflight description
1625 :reply payload: N/A
1626
1627 When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
1628 been successfully negotiated, this message is submitted by the front-end to
1629 get a shared buffer from back-end. The shared buffer will be used to
1630 track inflight I/O by back-end. QEMU should retrieve a new one when vm
1631 reset.
1632
1633 ``VHOST_USER_SET_INFLIGHT_FD``
1634 :id: 32
1635 :equivalent ioctl: N/A
1636 :request payload: inflight description
1637 :reply payload: N/A
1638
1639 When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
1640 been successfully negotiated, this message is submitted by the front-end to
1641 send the shared inflight buffer back to the back-end so that the back-end
1642 could get inflight I/O after a crash or restart.
1643
1644 ``VHOST_USER_GPU_SET_SOCKET``
1645 :id: 33
1646 :equivalent ioctl: N/A
1647 :request payload: N/A
1648 :reply payload: N/A
1649
1650 Sets the GPU protocol socket file descriptor, which is passed as
1651 ancillary data. The GPU protocol is used to inform the front-end of
1652 rendering state and updates. See vhost-user-gpu.rst for details.
1653
1654 ``VHOST_USER_RESET_DEVICE``
1655 :id: 34
1656 :equivalent ioctl: N/A
1657 :request payload: N/A
1658 :reply payload: N/A
1659
1660 Ask the vhost user back-end to disable all rings and reset all
1661 internal device state to the initial state, ready to be
1662 reinitialized. The back-end retains ownership of the device
1663 throughout the reset operation.
1664
1665 Only valid if the ``VHOST_USER_PROTOCOL_F_RESET_DEVICE`` protocol
1666 feature is set by the back-end.
1667
1668 ``VHOST_USER_VRING_KICK``
1669 :id: 35
1670 :equivalent ioctl: N/A
1671 :request payload: vring state description
1672 :reply payload: N/A
1673
1674 When the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` protocol
1675 feature has been successfully negotiated, this message may be
1676 submitted by the front-end to indicate that a buffer was added to
1677 the vring instead of signalling it using the vring's kick file
1678 descriptor or having the back-end rely on polling.
1679
1680 The state.num field is currently reserved and must be set to 0.
1681
1682 ``VHOST_USER_GET_MAX_MEM_SLOTS``
1683 :id: 36
1684 :equivalent ioctl: N/A
1685 :request payload: N/A
1686 :reply payload: u64
1687
1688 When the ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol
1689 feature has been successfully negotiated, this message is submitted
1690 by the front-end to the back-end. The back-end should return the message with a
1691 u64 payload containing the maximum number of memory slots for
1692 QEMU to expose to the guest. The value returned by the back-end
1693 will be capped at the maximum number of ram slots which can be
1694 supported by the target platform.
1695
1696 ``VHOST_USER_ADD_MEM_REG``
1697 :id: 37
1698 :equivalent ioctl: N/A
1699 :request payload: N/A
1700 :reply payload: single memory region description
1701
1702 When the ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol
1703 feature has been successfully negotiated, this message is submitted
1704 by the front-end to the back-end. The message payload contains a memory
1705 region descriptor struct, describing a region of guest memory which
1706 the back-end device must map in. When the
1707 ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol feature has
1708 been successfully negotiated, along with the
1709 ``VHOST_USER_REM_MEM_REG`` message, this message is used to set and
1710 update the memory tables of the back-end device.
1711
1712 Exactly one file descriptor from which the memory is mapped is
1713 passed in the ancillary data.
1714
1715 In postcopy mode (see ``VHOST_USER_POSTCOPY_LISTEN``), the back-end
1716 replies with the bases of the memory mapped region to the front-end.
1717 For further details on postcopy, see ``VHOST_USER_SET_MEM_TABLE``.
1718 They apply to ``VHOST_USER_ADD_MEM_REG`` accordingly.
1719
1720 ``VHOST_USER_REM_MEM_REG``
1721 :id: 38
1722 :equivalent ioctl: N/A
1723 :request payload: N/A
1724 :reply payload: single memory region description
1725
1726 When the ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol
1727 feature has been successfully negotiated, this message is submitted
1728 by the front-end to the back-end. The message payload contains a memory
1729 region descriptor struct, describing a region of guest memory which
1730 the back-end device must unmap. When the
1731 ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol feature has
1732 been successfully negotiated, along with the
1733 ``VHOST_USER_ADD_MEM_REG`` message, this message is used to set and
1734 update the memory tables of the back-end device.
1735
1736 The memory region to be removed is identified by its guest address,
1737 user address and size. The mmap offset is ignored.
1738
1739 No file descriptors SHOULD be passed in the ancillary data. For
1740 compatibility with existing incorrect implementations, the back-end MAY
1741 accept messages with one file descriptor. If a file descriptor is
1742 passed, the back-end MUST close it without using it otherwise.
1743
1744 ``VHOST_USER_SET_STATUS``
1745 :id: 39
1746 :equivalent ioctl: VHOST_VDPA_SET_STATUS
1747 :request payload: ``u64``
1748 :reply payload: N/A
1749
1750 When the ``VHOST_USER_PROTOCOL_F_STATUS`` protocol feature has been
1751 successfully negotiated, this message is submitted by the front-end to
1752 notify the back-end with updated device status as defined in the Virtio
1753 specification.
1754
1755 ``VHOST_USER_GET_STATUS``
1756 :id: 40
1757 :equivalent ioctl: VHOST_VDPA_GET_STATUS
1758 :request payload: N/A
1759 :reply payload: ``u64``
1760
1761 When the ``VHOST_USER_PROTOCOL_F_STATUS`` protocol feature has been
1762 successfully negotiated, this message is submitted by the front-end to
1763 query the back-end for its device status as defined in the Virtio
1764 specification.
1765
1766 ``VHOST_USER_GET_SHARED_OBJECT``
1767 :id: 41
1768 :equivalent ioctl: N/A
1769 :request payload: ``struct VhostUserShared``
1770 :reply payload: dmabuf fd
1771
1772 When the ``VHOST_USER_PROTOCOL_F_SHARED_OBJECT`` protocol
1773 feature has been successfully negotiated, and the UUID is found
1774 in the exporters cache, this message is submitted by the front-end
1775 to retrieve a given dma-buf fd from a given back-end, determined by
1776 the requested UUID. Back-end will reply passing the fd when the operation
1777 is successful, or no fd otherwise.
1778
1779 ``VHOST_USER_SET_DEVICE_STATE_FD``
1780 :id: 42
1781 :equivalent ioctl: N/A
1782 :request payload: device state transfer parameters
1783 :reply payload: ``u64``
1784
1785 Front-end and back-end negotiate a channel over which to transfer the
1786 back-end's internal state during migration. Either side (front-end or
1787 back-end) may create the channel. The nature of this channel is not
1788 restricted or defined in this document, but whichever side creates it
1789 must create a file descriptor that is provided to the respectively
1790 other side, allowing access to the channel. This FD must behave as
1791 follows:
1792
1793 * For the writing end, it must allow writing the whole back-end state
1794 sequentially. Closing the file descriptor signals the end of
1795 transfer.
1796
1797 * For the reading end, it must allow reading the whole back-end state
1798 sequentially. The end of file signals the end of the transfer.
1799
1800 For example, the channel may be a pipe, in which case the two ends of
1801 the pipe fulfill these requirements respectively.
1802
1803 Initially, the front-end creates a channel along with such an FD. It
1804 passes the FD to the back-end as ancillary data of a
1805 ``VHOST_USER_SET_DEVICE_STATE_FD`` message. The back-end may create a
1806 different transfer channel, passing the respective FD back to the
1807 front-end as ancillary data of the reply. If so, the front-end must
1808 then discard its channel and use the one provided by the back-end.
1809
1810 Whether the back-end should decide to use its own channel is decided
1811 based on efficiency: If the channel is a pipe, both ends will most
1812 likely need to copy data into and out of it. Any channel that allows
1813 for more efficient processing on at least one end, e.g. through
1814 zero-copy, is considered more efficient and thus preferred. If the
1815 back-end can provide such a channel, it should decide to use it.
1816
1817 The request payload contains parameters for the subsequent data
1818 transfer, as described in the :ref:`Migrating back-end state
1819 <migrating_backend_state>` section.
1820
1821 The value returned is both an indication for success, and whether a
1822 file descriptor for a back-end-provided channel is returned: Bits 0–7
1823 are 0 on success, and non-zero on error. Bit 8 is the invalid FD
1824 flag; this flag is set when there is no file descriptor returned.
1825 When this flag is not set, the front-end must use the returned file
1826 descriptor as its end of the transfer channel. The back-end must not
1827 both indicate an error and return a file descriptor.
1828
1829 Using this function requires prior negotiation of the
1830 ``VHOST_USER_PROTOCOL_F_DEVICE_STATE`` feature.
1831
1832 ``VHOST_USER_CHECK_DEVICE_STATE``
1833 :id: 43
1834 :equivalent ioctl: N/A
1835 :request payload: N/A
1836 :reply payload: ``u64``
1837
1838 After transferring the back-end's internal state during migration (see
1839 the :ref:`Migrating back-end state <migrating_backend_state>`
1840 section), check whether the back-end was able to successfully fully
1841 process the state.
1842
1843 The value returned indicates success or error; 0 is success, any
1844 non-zero value is an error.
1845
1846 Using this function requires prior negotiation of the
1847 ``VHOST_USER_PROTOCOL_F_DEVICE_STATE`` feature.
1848
1849 ``VHOST_USER_GET_SHMEM_CONFIG``
1850 :id: 44
1851 :equivalent ioctl: N/A
1852 :request payload: N/A
1853 :reply payload: ``struct VhostUserShMemConfig``
1854
1855 When the ``VHOST_USER_PROTOCOL_F_SHMEM`` protocol feature has been
1856 successfully negotiated, this message can be submitted by the front-end
1857 to gather the VIRTIO Shared Memory Region configuration. The back-end will
1858 respond with the number of VIRTIO Shared Memory Regions it requires, and
1859 each shared memory region size in an array. The shared memory IDs are
1860 represented by the array index. The information returned shall comply
1861 with the following rules:
1862
1863 * The shared information will remain valid and unchanged for the entire
1864 lifetime of the connection.
1865
1866 * The Shared Memory Region size must be a multiple of the page size
1867 supported by mmap(2).
1868
1869 * The size may be 0 if the region is unused.
1870
1871 Back-end message types
1872 ----------------------
1873
1874 For this type of message, the request is sent by the back-end and the reply
1875 is sent by the front-end.
1876
1877 ``VHOST_USER_BACKEND_IOTLB_MSG`` (previous name ``VHOST_USER_SLAVE_IOTLB_MSG``)
1878 :id: 1
1879 :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
1880 :request payload: ``struct vhost_iotlb_msg``
1881 :reply payload: N/A
1882
1883 Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
1884 The back-end sends such requests to notify of an IOTLB miss, or an IOTLB
1885 access failure. If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is
1886 negotiated, and back-end set the ``VHOST_USER_NEED_REPLY`` flag, the front-end
1887 must respond with zero when operation is successfully completed, or
1888 non-zero otherwise. This request should be send only when
1889 ``VIRTIO_F_IOMMU_PLATFORM`` feature has been successfully
1890 negotiated.
1891
1892 ``VHOST_USER_BACKEND_CONFIG_CHANGE_MSG`` (previous name ``VHOST_USER_SLAVE_CONFIG_CHANGE_MSG``)
1893 :id: 2
1894 :equivalent ioctl: N/A
1895 :request payload: N/A
1896 :reply payload: N/A
1897
1898 When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, vhost-user
1899 back-end sends such messages to notify that the virtio device's
1900 configuration space has changed, for those host devices which can
1901 support such feature, host driver can send ``VHOST_USER_GET_CONFIG``
1902 message to the back-end to get the latest content. If
1903 ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and the back-end sets the
1904 ``VHOST_USER_NEED_REPLY`` flag, the front-end must respond with zero when
1905 operation is successfully completed, or non-zero otherwise.
1906
1907 ``VHOST_USER_BACKEND_VRING_HOST_NOTIFIER_MSG`` (previous name ``VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG``)
1908 :id: 3
1909 :equivalent ioctl: N/A
1910 :request payload: vring area description
1911 :reply payload: N/A
1912
1913 Sets host notifier for a specified queue. The queue index is
1914 contained in the ``u64`` field of the vring area description. The
1915 host notifier is described by the file descriptor (typically it's a
1916 VFIO device fd) which is passed as ancillary data and the size
1917 (which is mmap size and should be the same as host page size) and
1918 offset (which is mmap offset) carried in the vring area
1919 description. QEMU can mmap the file descriptor based on the size and
1920 offset to get a memory range. Registering a host notifier means
1921 mapping this memory range to the VM as the specified queue's notify
1922 MMIO region. The back-end sends this request to tell QEMU to de-register
1923 the existing notifier if any and register the new notifier if the
1924 request is sent with a file descriptor.
1925
1926 This request should be sent only when
1927 ``VHOST_USER_PROTOCOL_F_HOST_NOTIFIER`` protocol feature has been
1928 successfully negotiated.
1929
1930 ``VHOST_USER_BACKEND_VRING_CALL`` (previous name ``VHOST_USER_SLAVE_VRING_CALL``)
1931 :id: 4
1932 :equivalent ioctl: N/A
1933 :request payload: vring state description
1934 :reply payload: N/A
1935
1936 When the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` protocol
1937 feature has been successfully negotiated, this message may be
1938 submitted by the back-end to indicate that a buffer was used from
1939 the vring instead of signalling this using the vring's call file
1940 descriptor or having the front-end relying on polling.
1941
1942 The state.num field is currently reserved and must be set to 0.
1943
1944 ``VHOST_USER_BACKEND_VRING_ERR`` (previous name ``VHOST_USER_SLAVE_VRING_ERR``)
1945 :id: 5
1946 :equivalent ioctl: N/A
1947 :request payload: vring state description
1948 :reply payload: N/A
1949
1950 When the ``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` protocol
1951 feature has been successfully negotiated, this message may be
1952 submitted by the back-end to indicate that an error occurred on the
1953 specific vring, instead of signalling the error file descriptor
1954 set by the front-end via ``VHOST_USER_SET_VRING_ERR``.
1955
1956 The state.num field is currently reserved and must be set to 0.
1957
1958 ``VHOST_USER_BACKEND_SHARED_OBJECT_ADD``
1959 :id: 6
1960 :equivalent ioctl: N/A
1961 :request payload: ``struct VhostUserShared``
1962 :reply payload: N/A
1963
1964 When the ``VHOST_USER_PROTOCOL_F_SHARED_OBJECT`` protocol
1965 feature has been successfully negotiated, this message can be submitted
1966 by the backends to add themselves as exporters to the virtio shared lookup
1967 table. The back-end device gets associated with a UUID in the shared table.
1968 The back-end is responsible of keeping its own table with exported dma-buf fds.
1969 When another back-end tries to import the resource associated with the UUID,
1970 it will send a message to the front-end, which will act as a proxy to the
1971 exporter back-end. If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and
1972 the back-end sets the ``VHOST_USER_NEED_REPLY`` flag, the front-end must
1973 respond with zero when operation is successfully completed, or non-zero
1974 otherwise.
1975
1976 ``VHOST_USER_BACKEND_SHARED_OBJECT_REMOVE``
1977 :id: 7
1978 :equivalent ioctl: N/A
1979 :request payload: ``struct VhostUserShared``
1980 :reply payload: N/A
1981
1982 When the ``VHOST_USER_PROTOCOL_F_SHARED_OBJECT`` protocol
1983 feature has been successfully negotiated, this message can be submitted
1984 by the backend to remove themselves from to the virtio-dmabuf shared
1985 table API. Only the back-end owning the entry (i.e., the one that first added
1986 it) will have permission to remove it. Otherwise, the message is ignored.
1987 The shared table will remove the back-end device associated with
1988 the UUID. If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and the
1989 back-end sets the ``VHOST_USER_NEED_REPLY`` flag, the front-end must respond
1990 with zero when operation is successfully completed, or non-zero otherwise.
1991
1992 ``VHOST_USER_BACKEND_SHARED_OBJECT_LOOKUP``
1993 :id: 8
1994 :equivalent ioctl: N/A
1995 :request payload: ``struct VhostUserShared``
1996 :reply payload: dmabuf fd and ``u64``
1997
1998 When the ``VHOST_USER_PROTOCOL_F_SHARED_OBJECT`` protocol
1999 feature has been successfully negotiated, this message can be submitted
2000 by the backends to retrieve a given dma-buf fd from the virtio-dmabuf
2001 shared table given a UUID. Frontend will reply passing the fd and a zero
2002 when the operation is successful, or non-zero otherwise. Note that if the
2003 operation fails, no fd is sent to the backend.
2004
2005 ``VHOST_USER_BACKEND_SHMEM_MAP``
2006 :id: 9
2007 :equivalent ioctl: N/A
2008 :request payload: fd and ``struct VhostUserMMap``
2009 :reply payload: N/A
2010
2011 When the ``VHOST_USER_PROTOCOL_F_SHMEM`` protocol feature has been
2012 successfully negotiated, this message can be submitted by the backends to
2013 advertise a new mapping to be made in a given VIRTIO Shared Memory Region.
2014 Upon receiving the message, the front-end will mmap the given fd into the
2015 VIRTIO Shared Memory Region with the requested ``shmid``.
2016
2017 If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and
2018 back-end set the ``VHOST_USER_NEED_REPLY`` flag, the front-end
2019 must respond with zero when operation is successfully completed,
2020 or non-zero otherwise.
2021
2022 Mapping over an already existing map is not allowed and requests shall fail.
2023 Therefore, the memory range in the request must correspond with a valid,
2024 free region of the VIRTIO Shared Memory Region. Also, note that mappings
2025 consume resources and that the request can fail when there are no resources
2026 available. Lastly, mappings are automatically unmapped by the front-end
2027 across device reset operation.
2028
2029 ``VHOST_USER_BACKEND_SHMEM_UNMAP``
2030 :id: 10
2031 :equivalent ioctl: N/A
2032 :request payload: ``struct VhostUserMMap``
2033 :reply payload: N/A
2034
2035 When the ``VHOST_USER_PROTOCOL_F_SHMEM`` protocol feature has been
2036 successfully negotiated, this message can be submitted by the backends so
2037 that the front-end un-mmaps a given range (``shm_offset``, ``len``) in the
2038 VIRTIO Shared Memory Region with the requested ``shmid``. Note that the
2039 given range shall correspond to the entirety of a valid mapped region.
2040
2041 If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and the back-end
2042 sets the ``VHOST_USER_NEED_REPLY`` flag, the front-end must respond with
2043 zero when operation is successfully completed, or non-zero otherwise.
2044
2045 .. _reply_ack:
2046
2047 VHOST_USER_PROTOCOL_F_REPLY_ACK
2048 -------------------------------
2049
2050 The original vhost-user specification only demands replies for certain
2051 commands. This differs from the vhost protocol implementation where
2052 commands are sent over an ``ioctl()`` call and block until the back-end
2053 has completed.
2054
2055 With this protocol extension negotiated, the sender (QEMU) can set the
2056 ``need_reply`` [Bit 3] flag to any command. This indicates that the
2057 back-end MUST respond with a Payload ``VhostUserMsg`` indicating success
2058 or failure. The payload should be set to zero on success or non-zero
2059 on failure, unless the message already has an explicit reply body.
2060
2061 The reply payload gives QEMU a deterministic indication of the result
2062 of the command. Today, QEMU is expected to terminate the main vhost-user
2063 loop upon receiving such errors. In future, qemu could be taught to be more
2064 resilient for selective requests.
2065
2066 For the message types that already solicit a reply from the back-end,
2067 the presence of ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` or need_reply bit
2068 being set brings no behavioural change. (See the Communication_
2069 section for details.)
2070
2071 .. _backend_conventions:
2072
2073 Backend program conventions
2074 ===========================
2075
2076 vhost-user back-ends can provide various devices & services and may
2077 need to be configured manually depending on the use case. However, it
2078 is a good idea to follow the conventions listed here when
2079 possible. Users, QEMU or libvirt, can then rely on some common
2080 behaviour to avoid heterogeneous configuration and management of the
2081 back-end programs and facilitate interoperability.
2082
2083 Each back-end installed on a host system should come with at least one
2084 JSON file that conforms to the vhost-user.json schema. Each file
2085 informs the management applications about the back-end type, and binary
2086 location. In addition, it defines rules for management apps for
2087 picking the highest priority back-end when multiple match the search
2088 criteria (see ``@VhostUserBackend`` documentation in the schema file).
2089
2090 If the back-end is not capable of enabling a requested feature on the
2091 host (such as 3D acceleration with virgl), or the initialization
2092 failed, the back-end should fail to start early and exit with a status
2093 != 0. It may also print a message to stderr for further details.
2094
2095 The back-end program must not daemonize itself, but it may be
2096 daemonized by the management layer. It may also have a restricted
2097 access to the system.
2098
2099 File descriptors 0, 1 and 2 will exist, and have regular
2100 stdin/stdout/stderr usage (they may have been redirected to /dev/null
2101 by the management layer, or to a log handler).
2102
2103 The back-end program must end (as quickly and cleanly as possible) when
2104 the SIGTERM signal is received. Eventually, it may receive SIGKILL by
2105 the management layer after a few seconds.
2106
2107 The following command line options have an expected behaviour. They
2108 are mandatory, unless explicitly said differently:
2109
2110 --socket-path=PATH
2111
2112 This option specify the location of the vhost-user Unix domain socket.
2113 It is incompatible with --fd.
2114
2115 --fd=FDNUM
2116
2117 When this argument is given, the back-end program is started with the
2118 vhost-user socket as file descriptor FDNUM. It is incompatible with
2119 --socket-path.
2120
2121 --print-capabilities
2122
2123 Output to stdout the back-end capabilities in JSON format, and then
2124 exit successfully. Other options and arguments should be ignored, and
2125 the back-end program should not perform its normal function. The
2126 capabilities can be reported dynamically depending on the host
2127 capabilities.
2128
2129 The JSON output is described in the ``vhost-user.json`` schema, by
2130 ```@VHostUserBackendCapabilities``. Example:
2131
2132 .. code:: json
2133
2134 {
2135 "type": "foo",
2136 "features": [
2137 "feature-a",
2138 "feature-b"
2139 ]
2140 }
2141
2142 vhost-user-input
2143 ----------------
2144
2145 Command line options:
2146
2147 --evdev-path=PATH
2148
2149 Specify the linux input device.
2150
2151 (optional)
2152
2153 --no-grab
2154
2155 Do no request exclusive access to the input device.
2156
2157 (optional)
2158
2159 vhost-user-gpu
2160 --------------
2161
2162 Command line options:
2163
2164 --render-node=PATH
2165
2166 Specify the GPU DRM render node.
2167
2168 (optional)
2169
2170 --virgl
2171
2172 Enable virgl rendering support.
2173
2174 (optional)
2175
2176 vhost-user-blk
2177 --------------
2178
2179 Command line options:
2180
2181 --blk-file=PATH
2182
2183 Specify block device or file path.
2184
2185 (optional)
2186
2187 --read-only
2188
2189 Enable read-only.
2190
2191 (optional)