@samitouri / QOSamiQemu
Code Commits Issues Pulls Sessions CI/CD
master
master
rst 2,004 lines 81.8 KB
Raw
1 .. include:: <isonum.txt>
2 .. SPDX-License-Identifier: GPL-2.0-or-later
3
4 ================================
5 vfio-user Protocol Specification
6 ================================
7
8 .. contents:: Table of Contents
9
10 Introduction
11 ============
12 vfio-user is a protocol that allows a device to be emulated in a separate
13 process outside of a Virtual Machine Monitor (VMM). vfio-user devices consist
14 of a generic VFIO device type, living inside the VMM, which we call the client,
15 and the core device implementation, living outside the VMM, which we call the
16 server.
17
18 The vfio-user specification is partly based on the
19 `Linux VFIO ioctl interface <https://www.kernel.org/doc/html/latest/driver-api/vfio.html>`_.
20
21 VFIO is a mature and stable API, backed by an extensively used framework. The
22 existing VFIO client implementation in QEMU (``qemu/hw/vfio/``) can be largely
23 re-used, though there is nothing in this specification that requires that
24 particular implementation. None of the VFIO kernel modules are required for
25 supporting the protocol, on either the client or server side. Some source
26 definitions in VFIO are re-used for vfio-user.
27
28 The main idea is to allow a virtual device to function in a separate process in
29 the same host over a UNIX domain socket. A UNIX domain socket (``AF_UNIX``) is
30 chosen because file descriptors can be trivially sent over it, which in turn
31 allows:
32
33 * Sharing of client memory for DMA with the server.
34 * Sharing of server memory with the client for fast MMIO.
35 * Efficient sharing of eventfd's for triggering interrupts.
36
37 Other socket types could be used which allow the server to run in a separate
38 guest in the same host (``AF_VSOCK``) or remotely (``AF_INET``). Theoretically
39 the underlying transport does not necessarily have to be a socket, however we do
40 not examine such alternatives. In this protocol version we focus on using a UNIX
41 domain socket and introduce basic support for the other two types of sockets
42 without considering performance implications.
43
44 While passing of file descriptors is desirable for performance reasons, support
45 is not necessary for either the client or the server in order to implement the
46 protocol. There is always an in-band, message-passing fall back mechanism.
47
48 Overview
49 ========
50
51 VFIO is a framework that allows a physical device to be securely passed through
52 to a user space process; the device-specific kernel driver does not drive the
53 device at all. Typically, the user space process is a VMM and the device is
54 passed through to it in order to achieve high performance. VFIO provides an API
55 and the required functionality in the kernel. QEMU has adopted VFIO to allow a
56 guest to directly access physical devices, instead of emulating them in
57 software.
58
59 vfio-user reuses the core VFIO concepts defined in its API, but implements them
60 as messages to be sent over a socket. It does not change the kernel-based VFIO
61 in any way, in fact none of the VFIO kernel modules need to be loaded to use
62 vfio-user. It is also possible for the client to concurrently use the current
63 kernel-based VFIO for one device, and vfio-user for another device.
64
65 VFIO Device Model
66 -----------------
67
68 A device under VFIO presents a standard interface to the user process. Many of
69 the VFIO operations in the existing interface use the ``ioctl()`` system call, and
70 references to the existing interface are called the ``ioctl()`` implementation in
71 this document.
72
73 The following sections describe the set of messages that implement the vfio-user
74 interface over a socket. In many cases, the messages are analogous to data
75 structures used in the ``ioctl()`` implementation. Messages derived from the
76 ``ioctl()`` will have a name derived from the ``ioctl()`` command name. E.g., the
77 ``VFIO_DEVICE_GET_INFO`` ``ioctl()`` command becomes a
78 ``VFIO_USER_DEVICE_GET_INFO`` message. The purpose of this reuse is to share as
79 much code as feasible with the ``ioctl()`` implementation.
80
81 Connection Initiation
82 ^^^^^^^^^^^^^^^^^^^^^
83
84 After the client connects to the server, the initial client message is
85 ``VFIO_USER_VERSION`` to propose a protocol version and set of capabilities to
86 apply to the session. The server replies with a compatible version and set of
87 capabilities it supports, or closes the connection if it cannot support the
88 advertised version.
89
90 Device Information
91 ^^^^^^^^^^^^^^^^^^
92
93 The client uses a ``VFIO_USER_DEVICE_GET_INFO`` message to query the server for
94 information about the device. This information includes:
95
96 * The device type and whether it supports reset (``VFIO_DEVICE_FLAGS_``),
97 * the number of device regions, and
98 * the device presents to the client the number of interrupt types the device
99 supports.
100
101 Region Information
102 ^^^^^^^^^^^^^^^^^^
103
104 The client uses ``VFIO_USER_DEVICE_GET_REGION_INFO`` messages to query the
105 server for information about the device's regions. This information describes:
106
107 * Read and write permissions, whether it can be memory mapped, and whether it
108 supports additional capabilities (``VFIO_REGION_INFO_CAP_``).
109 * Region index, size, and offset.
110
111 When a device region can be mapped by the client, the server provides a file
112 descriptor which the client can ``mmap()``. The server is responsible for
113 polling for client updates to memory mapped regions.
114
115 Region Capabilities
116 """""""""""""""""""
117
118 Some regions have additional capabilities that cannot be described adequately
119 by the region info data structure. These capabilities are returned in the
120 region info reply in a list similar to PCI capabilities in a PCI device's
121 configuration space.
122
123 Sparse Regions
124 """"""""""""""
125 A region can be memory-mappable in whole or in part. When only a subset of a
126 region can be mapped by the client, a ``VFIO_REGION_INFO_CAP_SPARSE_MMAP``
127 capability is included in the region info reply. This capability describes
128 which portions can be mapped by the client.
129
130 .. Note::
131 For example, in a virtual NVMe controller, sparse regions can be used so
132 that accesses to the NVMe registers (found in the beginning of BAR0) are
133 trapped (an infrequent event), while allowing direct access to the doorbells
134 (an extremely frequent event as every I/O submission requires a write to
135 BAR0), found in the next page after the NVMe registers in BAR0.
136
137 Device-Specific Regions
138 """""""""""""""""""""""
139
140 A device can define regions additional to the standard ones (e.g. PCI indexes
141 0-8). This is achieved by including a ``VFIO_REGION_INFO_CAP_TYPE`` capability
142 in the region info reply of a device-specific region. Such regions are reflected
143 in ``struct vfio_user_device_info.num_regions``. Thus, for PCI devices this
144 value can be equal to, or higher than, ``VFIO_PCI_NUM_REGIONS``.
145
146 Region I/O via file descriptors
147 -------------------------------
148
149 For unmapped regions, region I/O from the client is done via
150 ``VFIO_USER_REGION_READ/WRITE``. As an optimization, ioeventfds or ioregionfds
151 may be configured for sub-regions of some regions. A client may request
152 information on these sub-regions via ``VFIO_USER_DEVICE_GET_REGION_IO_FDS``; by
153 configuring the returned file descriptors as ioeventfds or ioregionfds, the
154 server can be directly notified of I/O (for example, by KVM) without taking a
155 trip through the client.
156
157 Interrupts
158 ^^^^^^^^^^
159
160 The client uses ``VFIO_USER_DEVICE_GET_IRQ_INFO`` messages to query the server
161 for the device's interrupt types. The interrupt types are specific to the bus
162 the device is attached to, and the client is expected to know the capabilities
163 of each interrupt type. The server can signal an interrupt by directly injecting
164 interrupts into the guest via an event file descriptor. The client configures
165 how the server signals an interrupt with ``VFIO_USER_SET_IRQS`` messages.
166
167 Device Read and Write
168 ^^^^^^^^^^^^^^^^^^^^^
169
170 When the guest executes load or store operations to an unmapped device region,
171 the client forwards these operations to the server with
172 ``VFIO_USER_REGION_READ`` or ``VFIO_USER_REGION_WRITE`` messages. The server
173 will reply with data from the device on read operations or an acknowledgement on
174 write operations. See `Read and Write Operations`_.
175
176 Client memory access
177 --------------------
178
179 The client uses ``VFIO_USER_DMA_MAP`` and ``VFIO_USER_DMA_UNMAP`` messages to
180 inform the server of the valid DMA ranges that the server can access on behalf
181 of a device (typically, VM guest memory). DMA memory may be accessed by the
182 server via ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages over the
183 socket. In this case, the "DMA" part of the naming is a misnomer.
184
185 Actual direct memory access of client memory from the server is possible if the
186 client provides file descriptors the server can ``mmap()``. Note that ``mmap()``
187 privileges cannot be revoked by the client, therefore file descriptors should
188 only be exported in environments where the client trusts the server not to
189 corrupt guest memory.
190
191 See `Read and Write Operations`_.
192
193 Client/server interactions
194 ==========================
195
196 Socket
197 ------
198
199 A server can serve:
200
201 1) one or more clients, and/or
202 2) one or more virtual devices, belonging to one or more clients.
203
204 The current protocol specification requires dedicated sockets per
205 client/server connection. Commands in the client-to-server direction are
206 handled on the main communication socket which the client connects to, and
207 replies to these commands are passed on the same socket. Commands sent in the
208 other direction from the server to the client as well as their corresponding
209 replies can optionally be passed across a separate socket, which is set up
210 during negotiation (AF_UNIX servers just pass the file descriptor).
211
212 Using separate sockets for each command channel avoids introducing an
213 artificial point of synchronization between the channels. This simplifies
214 implementations since it obviates the need to demultiplex incoming messages
215 into commands and replies and interleave command handling and reply processing.
216 Note that it is still illegal for implementations to stall command or reply
217 processing indefinitely while waiting for replies on the other channel, as this
218 may lead to deadlocks. However, since incoming commands and requests arrive on
219 different sockets, it's possible to meet this requirement e.g. by running two
220 independent request processing threads that can internally operate
221 synchronously. It is expected that this is simpler to implement than fully
222 asynchronous message handling code. Implementations may still choose a fully
223 asynchronous, event-based design for other reasons, and the protocol fully
224 supports it.
225
226 It is a server-side implementation detail whether a single server handles
227 multiple virtual devices from the same or multiple clients. The location of the
228 socket is implementation-specific. Multiplexing clients, devices, and servers
229 over the same socket is not supported in this version of the protocol.
230
231 Authentication
232 --------------
233
234 For ``AF_UNIX``, we rely on OS mandatory access controls on the socket files,
235 therefore it is up to the management layer to set up the socket as required.
236 Socket types that span guests or hosts will require a proper authentication
237 mechanism. Defining that mechanism is deferred to a future version of the
238 protocol.
239
240 Command Concurrency
241 -------------------
242
243 A client may pipeline multiple commands without waiting for previous command
244 replies. The server will process commands in the order they are received. A
245 consequence of this is if a client issues a command with the *No_reply* bit,
246 then subsequently issues a command without *No_reply*, the older command will
247 have been processed before the reply to the younger command is sent by the
248 server. The client must be aware of the device's capability to process
249 concurrent commands if pipelining is used. For example, pipelining allows
250 multiple client threads to concurrently access device regions; the client must
251 ensure these accesses obey device semantics.
252
253 An example is a frame buffer device, where the device may allow concurrent
254 access to different areas of video memory, but may have indeterminate behavior
255 if concurrent accesses are performed to command or status registers.
256
257 Note that unrelated messages sent from the server to the client can appear in
258 between a client to server request/reply and vice versa.
259
260 Implementers should be prepared for certain commands to exhibit potentially
261 unbounded latencies. For example, ``VFIO_USER_DEVICE_RESET`` may take an
262 arbitrarily long time to complete; clients should take care not to block
263 unnecessarily.
264
265 Socket Disconnection Behavior
266 -----------------------------
267 The server and the client can disconnect from each other, either intentionally
268 or unexpectedly. Both the client and the server need to know how to handle such
269 events.
270
271 Server Disconnection
272 ^^^^^^^^^^^^^^^^^^^^
273 A server disconnecting from the client may indicate that:
274
275 1) A virtual device has been restarted, either intentionally (e.g. because of a
276 device update) or unintentionally (e.g. because of a crash).
277 2) A virtual device has been shut down with no intention to be restarted.
278
279 It is impossible for the client to know whether or not a failure is
280 intermittent or innocuous and should be retried, therefore the client should
281 reset the VFIO device when it detects the socket has been disconnected.
282 Error recovery will be driven by the guest's device error handling
283 behavior.
284
285 Client Disconnection
286 ^^^^^^^^^^^^^^^^^^^^
287 The client disconnecting from the server primarily means that the client
288 has exited. Currently, this means that the guest is shut down so the device is
289 no longer needed therefore the server can automatically exit. However, there
290 can be cases where a client disconnection should not result in a server exit:
291
292 1) A single server serving multiple clients.
293 2) A multi-process QEMU upgrading itself step by step, which is not yet
294 implemented.
295
296 Therefore in order for the protocol to be forward compatible, the server should
297 respond to a client disconnection as follows:
298
299 - all client memory regions are unmapped and cleaned up (including closing any
300 passed file descriptors)
301 - all IRQ file descriptors passed from the old client are closed
302 - the device state should otherwise be retained
303
304 The expectation is that when a client reconnects, it will re-establish IRQ and
305 client memory mappings.
306
307 If anything happens to the client (such as qemu really did exit), the control
308 stack will know about it and can clean up resources accordingly.
309
310 Security Considerations
311 -----------------------
312
313 Speaking generally, vfio-user clients should not trust servers, and vice versa.
314 Standard tools and mechanisms should be used on both sides to validate input and
315 prevent against denial of service scenarios, buffer overflow, etc.
316
317 Request Retry and Response Timeout
318 ----------------------------------
319 A failed command is a command that has been successfully sent and has been
320 responded to with an error code. Failure to send the command in the first place
321 (e.g. because the socket is disconnected) is a different type of error examined
322 earlier in the disconnect section.
323
324 .. Note::
325 QEMU's VFIO retries certain operations if they fail. While this makes sense
326 for real HW, we don't know for sure whether it makes sense for virtual
327 devices.
328
329 Defining a retry and timeout scheme is deferred to a future version of the
330 protocol.
331
332 Message sizes
333 -------------
334
335 Some requests have an ``argsz`` field. In a request, it defines the maximum
336 expected reply payload size, which should be at least the size of the fixed
337 reply payload headers defined here. The *request* payload size is defined by the
338 usual ``msg_size`` field in the header, not the ``argsz`` field.
339
340 In a reply, the server sets ``argsz`` field to the size needed for a full
341 payload size. This may be less than the requested maximum size. This may be
342 larger than the requested maximum size: in that case, the full payload is not
343 included in the reply, but the ``argsz`` field in the reply indicates the needed
344 size, allowing a client to allocate a larger buffer for holding the reply before
345 trying again.
346
347 In addition, during negotiation (see `Version`_), the client and server may
348 each specify a ``max_data_xfer_size`` value; this defines the maximum data that
349 may be read or written via one of the ``VFIO_USER_DMA/REGION_READ/WRITE``
350 messages; see `Read and Write Operations`_.
351
352 Protocol Specification
353 ======================
354
355 To distinguish from the base VFIO symbols, all vfio-user symbols are prefixed
356 with ``vfio_user`` or ``VFIO_USER``. In this revision, all data is in the
357 endianness of the host system, although this may be relaxed in future
358 revisions in cases where the client and server run on different hosts
359 with different endianness.
360
361 Unless otherwise specified, all sizes should be presumed to be in bytes.
362
363 .. _Commands:
364
365 Commands
366 --------
367 The following table lists the VFIO message command IDs, and whether the
368 message command is sent from the client or the server.
369
370 ====================================== ========= =================
371 Name Command Request Direction
372 ====================================== ========= =================
373 ``VFIO_USER_VERSION`` 1 client -> server
374 ``VFIO_USER_DMA_MAP`` 2 client -> server
375 ``VFIO_USER_DMA_UNMAP`` 3 client -> server
376 ``VFIO_USER_DEVICE_GET_INFO`` 4 client -> server
377 ``VFIO_USER_DEVICE_GET_REGION_INFO`` 5 client -> server
378 ``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` 6 client -> server
379 ``VFIO_USER_DEVICE_GET_IRQ_INFO`` 7 client -> server
380 ``VFIO_USER_DEVICE_SET_IRQS`` 8 client -> server
381 ``VFIO_USER_REGION_READ`` 9 client -> server
382 ``VFIO_USER_REGION_WRITE`` 10 client -> server
383 ``VFIO_USER_DMA_READ`` 11 server -> client
384 ``VFIO_USER_DMA_WRITE`` 12 server -> client
385 ``VFIO_USER_DEVICE_RESET`` 13 client -> server
386 ``VFIO_USER_REGION_WRITE_MULTI`` 15 client -> server
387 ``VFIO_USER_DEVICE_FEATURE`` 16 client -> server
388 ``VFIO_USER_MIG_DATA_READ`` 17 client -> server
389 ``VFIO_USER_MIG_DATA_WRITE`` 18 client -> server
390 ====================================== ========= =================
391
392 Header
393 ------
394
395 All messages, both command messages and reply messages, are preceded by a
396 16-byte header that contains basic information about the message. The header is
397 followed by message-specific data described in the sections below.
398
399 +----------------+--------+-------------+
400 | Name | Offset | Size |
401 +================+========+=============+
402 | Message ID | 0 | 2 |
403 +----------------+--------+-------------+
404 | Command | 2 | 2 |
405 +----------------+--------+-------------+
406 | Message size | 4 | 4 |
407 +----------------+--------+-------------+
408 | Flags | 8 | 4 |
409 +----------------+--------+-------------+
410 | | +-----+------------+ |
411 | | | Bit | Definition | |
412 | | +=====+============+ |
413 | | | 0-3 | Type | |
414 | | +-----+------------+ |
415 | | | 4 | No_reply | |
416 | | +-----+------------+ |
417 | | | 5 | Error | |
418 | | +-----+------------+ |
419 +----------------+--------+-------------+
420 | Error | 12 | 4 |
421 +----------------+--------+-------------+
422 | <message data> | 16 | variable |
423 +----------------+--------+-------------+
424
425 * *Message ID* identifies the message, and is echoed in the command's reply
426 message. Message IDs belong entirely to the sender, can be re-used (even
427 concurrently) and the receiver must not make any assumptions about their
428 uniqueness.
429 * *Command* specifies the command to be executed, listed in Commands_. It is
430 also set in the reply header.
431 * *Message size* contains the size of the entire message, including the header.
432 * *Flags* contains attributes of the message:
433
434 * The *Type* bits indicate the message type.
435
436 * *Command* (value 0x0) indicates a command message.
437 * *Reply* (value 0x1) indicates a reply message acknowledging a previous
438 command with the same message ID.
439 * *No_reply* in a command message indicates that no reply is needed for this
440 command. This is commonly used when multiple commands are sent, and only
441 the last needs acknowledgement.
442 * *Error* in a reply message indicates the command being acknowledged had
443 an error. In this case, the *Error* field will be valid.
444
445 * *Error* in a reply message is an optional UNIX errno value. It may be zero
446 even if the Error bit is set in Flags. It is reserved in a command message.
447
448 Each command message in Commands_ must be replied to with a reply message,
449 unless the message sets the *No_Reply* bit. The reply consists of the header
450 with the *Reply* bit set, plus any additional data.
451
452 If an error occurs, the reply message must only include the reply header.
453
454 As the header is standard in both requests and replies, it is not included in
455 the command-specific specifications below; each message definition should be
456 appended to the standard header, and the offsets are given from the end of the
457 standard header.
458
459 ``VFIO_USER_VERSION``
460 ---------------------
461
462 .. _Version:
463
464 This is the initial message sent by the client after the socket connection is
465 established; the same format is used for the server's reply.
466
467 Upon establishing a connection, the client must send a ``VFIO_USER_VERSION``
468 message proposing a protocol version and a set of capabilities. The server
469 compares these with the versions and capabilities it supports and sends a
470 ``VFIO_USER_VERSION`` reply according to the following rules.
471
472 * The major version in the reply must be the same as proposed. If the client
473 does not support the proposed major, it closes the connection.
474 * The minor version in the reply must be equal to or less than the minor
475 version proposed.
476 * The capability list must be a subset of those proposed. If the server
477 requires a capability the client did not include, it closes the connection.
478
479 The protocol major version will only change when incompatible protocol changes
480 are made, such as changing the message format. The minor version may change
481 when compatible changes are made, such as adding new messages or capabilities,
482 Both the client and server must support all minor versions less than the
483 maximum minor version it supports. E.g., an implementation that supports
484 version 1.3 must also support 1.0 through 1.2.
485
486 When making a change to this specification, the protocol version number must
487 be included in the form "added in version X.Y"
488
489 Request
490 ^^^^^^^
491
492 ============== ====== ====
493 Name Offset Size
494 ============== ====== ====
495 version major 0 2
496 version minor 2 2
497 version data 4 variable (including terminating NUL). Optional.
498 ============== ====== ====
499
500 The version data is an optional UTF-8 encoded JSON byte array with the following
501 format:
502
503 +--------------+--------+-----------------------------------+
504 | Name | Type | Description |
505 +==============+========+===================================+
506 | capabilities | object | Contains common capabilities that |
507 | | | the sender supports. Optional. |
508 +--------------+--------+-----------------------------------+
509
510 Capabilities:
511
512 +--------------------+---------+-----------------------------------------------+
513 | Name | Type | Description |
514 +====================+=========+===============================================+
515 | max_msg_fds | number | Maximum number of file descriptors that can |
516 | | | be received by the sender in one message. |
517 | | | Optional. If not specified then the receiver |
518 | | | must assume a value of ``1``. |
519 +--------------------+---------+-----------------------------------------------+
520 | max_data_xfer_size | number | Maximum ``count`` for data transfer messages; |
521 | | | see `Read and Write Operations`_. Optional, |
522 | | | with a default value of 1048576 bytes. |
523 +--------------------+---------+-----------------------------------------------+
524 | max_dma_maps | number | Maximum number DMA map windows that can be |
525 | | | valid simultaneously. Optional, with a |
526 | | | value of 65535 (64k-1). |
527 +--------------------+---------+-----------------------------------------------+
528 | pgsizes | number | Page sizes supported in DMA map operations |
529 | | | or'ed together. Optional, with a default |
530 | | | value of supporting only 4k pages. |
531 +--------------------+---------+-----------------------------------------------+
532 | twin_socket | object | Parameters for twin-socket mode, which |
533 | | | handles server-to-client commands and their |
534 | | | replies on a separate socket. Optional. |
535 +--------------------+---------+-----------------------------------------------+
536 | write_multiple | boolean | ``VFIO_USER_REGION_WRITE_MULTI`` messages |
537 | | | are supported if the value is ``true``. |
538 +--------------------+---------+-----------------------------------------------+
539
540 The ``twin_socket`` capability object holds these name/value pairs:
541
542 +-----------+---------+--------------------------------------------------------+
543 | Name | Type | Description |
544 +===========+=========+========================================================+
545 | supported | boolean | Indicates whether the sender supports twin-socket |
546 | | | mode. Optional, defaults to false. |
547 +-----------+---------+--------------------------------------------------------+
548 | fd_index | number | Specifies an index in the file descriptor array |
549 | | | included with the message. The designated file |
550 | | | descriptor is a socket which is to be used for the |
551 | | | server-to-client command channel. Optional, only valid |
552 | | | in the reply message. |
553 +-----------+---------+--------------------------------------------------------+
554
555 Reply
556 ^^^^^
557
558 The same message format is used in the server's reply with the semantics
559 described above.
560
561 If and only if the client has indicated support for twin-socket mode by setting
562 ``twin_socket.supported`` to true in its capabilities, the server may optionally
563 set up a separate command channel for server-to-client commands and their
564 replies. The server enables twin-socket mode as follows:
565
566 * Create a fresh socket pair.
567 * Keep the server end of the socket pair and pass the client end in the file
568 descriptor array included with the reply message.
569 * Set ``twin_socket.supported`` to true in the reply.
570 * Indicate the index in the file descriptor array by the
571 ``twin_socket.fd_index`` capability field in the reply, so the client can
572 identify the correct file descriptor to use.
573
574 A client requesting twin-socket mode must examine the ``twin_socket`` capability
575 in the reply:
576
577 * If ``twin_socket.supported`` is false, the field is missing, or the entire
578 ``twin_socket`` object is absent, the server does not support twin-socket mode
579 or decided not to enable it. The client can choose whether it wants to proceed
580 without twin-socket mode, or close the connection if not.
581 * If ``twin_socket.supported`` is true and ``twin_socket.fd_index`` is present
582 and refers to a valid file descriptor, twin-socket mode negotiation has
583 succeeded. The client monitors the provided file descriptor for commands from
584 the server.
585 * Otherwise, the reply from the server is inconsistent. The client must abort
586 and close the connection since it is potentially unable to receive commands
587 from the server.
588
589 The twin-socket feature is optional, so some servers may not support it.
590 However, for server implementations that do send server-to-client commands it is
591 strongly recommended to implement twin-socket support.
592
593 ``VFIO_USER_DMA_MAP``
594 ---------------------
595
596 This command message is sent by the client to the server to inform it of the
597 memory regions the server can access. It must be sent before the server can
598 perform any DMA to the client. It is normally sent directly after the version
599 handshake is completed, but may also occur when memory is added to the client,
600 or if the client uses a vIOMMU.
601
602 Request
603 ^^^^^^^
604
605 The request payload for this message is a structure of the following format:
606
607 +-------------+--------+------------------------+
608 | Name | Offset | Size |
609 +=============+========+========================+
610 | argsz | 0 | 4 |
611 +-------------+--------+------------------------+
612 | flags | 4 | 4 |
613 +-------------+--------+------------------------+
614 | | +-----+-----------------------+ |
615 | | | Bit | Definition | |
616 | | +=====+=======================+ |
617 | | | 0 | readable | |
618 | | +-----+-----------------------+ |
619 | | | 1 | writeable | |
620 | | +-----+-----------------------+ |
621 | | | 2 | access mode: mmap | |
622 | | +-----+-----------------------+ |
623 | | | 3 | access mode: file I/O | |
624 | | +-----+-----------------------+ |
625 +-------------+--------+------------------------+
626 | offset | 8 | 8 |
627 +-------------+--------+------------------------+
628 | address | 16 | 8 |
629 +-------------+--------+------------------------+
630 | size | 24 | 8 |
631 +-------------+--------+------------------------+
632
633 * *argsz* is the size of the above structure. Note there is no reply payload,
634 so this field differs from other message types.
635 * *flags* contains the following region attributes:
636
637 * *readable* indicates that the region can be read from.
638
639 * *writeable* indicates that the region can be written to.
640
641 * *access mode* bits indicate how the region is to be accessed by the server.
642
643 * *offset* is the file offset of the region with respect to the associated file
644 descriptor, or zero if the region is not mappable
645 * *address* is the base DMA address of the region.
646 * *size* is the size of the region.
647
648 This structure is 32 bytes in size, so the message size is 16 + 32 bytes.
649
650 There are several alternative access modes for the server to use when accessing
651 the region:
652
653 * ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages, explained in
654 `Read and Write Operations`_.
655
656 * ``mmap()`` a client-provided file descriptor, then perform direct accesses to
657 the underlying memory.
658
659 * File I/O system calls (such as ``pread()`` / ``pwrite()``) against a
660 client-provided file descriptor.
661
662 The access mode bits in the flags field indicate which access mode to use. If
663 an access mode requiring a file descriptor is specified, but the client does
664 not provide a file descriptor, the server must fail the request with
665 ``EINVAL``. If no access mode flag bit is set, the server should use ``mmap()``
666 based access if the client provided a file descriptor and message-based access
667 otherwise.
668
669 On ``AF_UNIX`` sockets, the file descriptor must be passed as ``SCM_RIGHTS``
670 type ancillary data.
671
672 A command to map over an existing region must be failed by the server with
673 ``EEXIST`` set in the error field in the reply.
674
675 Reply
676 ^^^^^
677
678 There is no payload in the reply message.
679
680 ``VFIO_USER_DMA_UNMAP``
681 -----------------------
682
683 This command message is sent by the client to the server to inform it that a
684 DMA region, previously made available via a ``VFIO_USER_DMA_MAP`` command
685 message, is no longer available for DMA. It typically occurs when memory is
686 subtracted from the client or if the client uses a vIOMMU. The DMA region is
687 described by the following structure:
688
689 Request
690 ^^^^^^^
691
692 The request payload for this message is a structure of the following format:
693
694 +--------------+--------+------------------------+
695 | Name | Offset | Size |
696 +==============+========+========================+
697 | argsz | 0 | 4 |
698 +--------------+--------+------------------------+
699 | flags | 4 | 4 |
700 +--------------+--------+------------------------+
701 | address | 8 | 8 |
702 +--------------+--------+------------------------+
703 | size | 16 | 8 |
704 +--------------+--------+------------------------+
705
706 * *argsz* is the maximum size of the reply payload.
707 * *flags* is unused in this version.
708 * *address* is the base DMA address of the DMA region.
709 * *size* is the size of the DMA region.
710
711 The address and size of the DMA region being unmapped must match exactly a
712 previous mapping.
713
714 Reply
715 ^^^^^
716
717 Upon receiving a ``VFIO_USER_DMA_UNMAP`` command, if the file descriptor is
718 mapped then the server must release all references to that DMA region before
719 replying, which potentially includes in-flight DMA transactions.
720
721 The server responds with the original DMA entry in the request.
722
723
724 ``VFIO_USER_DEVICE_GET_INFO``
725 -----------------------------
726
727 This command message is sent by the client to the server to query for basic
728 information about the device.
729
730 Request
731 ^^^^^^^
732
733 +-------------+--------+--------------------------+
734 | Name | Offset | Size |
735 +=============+========+==========================+
736 | argsz | 0 | 4 |
737 +-------------+--------+--------------------------+
738 | flags | 4 | 4 |
739 +-------------+--------+--------------------------+
740 | | +-----+-------------------------+ |
741 | | | Bit | Definition | |
742 | | +=====+=========================+ |
743 | | | 0 | VFIO_DEVICE_FLAGS_RESET | |
744 | | +-----+-------------------------+ |
745 | | | 1 | VFIO_DEVICE_FLAGS_PCI | |
746 | | +-----+-------------------------+ |
747 +-------------+--------+--------------------------+
748 | num_regions | 8 | 4 |
749 +-------------+--------+--------------------------+
750 | num_irqs | 12 | 4 |
751 +-------------+--------+--------------------------+
752
753 * *argsz* is the maximum size of the reply payload
754 * all other fields must be zero.
755
756 Reply
757 ^^^^^
758
759 +-------------+--------+--------------------------+
760 | Name | Offset | Size |
761 +=============+========+==========================+
762 | argsz | 0 | 4 |
763 +-------------+--------+--------------------------+
764 | flags | 4 | 4 |
765 +-------------+--------+--------------------------+
766 | | +-----+-------------------------+ |
767 | | | Bit | Definition | |
768 | | +=====+=========================+ |
769 | | | 0 | VFIO_DEVICE_FLAGS_RESET | |
770 | | +-----+-------------------------+ |
771 | | | 1 | VFIO_DEVICE_FLAGS_PCI | |
772 | | +-----+-------------------------+ |
773 +-------------+--------+--------------------------+
774 | num_regions | 8 | 4 |
775 +-------------+--------+--------------------------+
776 | num_irqs | 12 | 4 |
777 +-------------+--------+--------------------------+
778
779 * *argsz* is the size required for the full reply payload (16 bytes today)
780 * *flags* contains the following device attributes.
781
782 * ``VFIO_DEVICE_FLAGS_RESET`` indicates that the device supports the
783 ``VFIO_USER_DEVICE_RESET`` message.
784 * ``VFIO_DEVICE_FLAGS_PCI`` indicates that the device is a PCI device.
785
786 * *num_regions* is the number of memory regions that the device exposes.
787 * *num_irqs* is the number of distinct interrupt types that the device supports.
788
789 This version of the protocol only supports PCI devices. Additional devices may
790 be supported in future versions.
791
792 ``VFIO_USER_DEVICE_GET_REGION_INFO``
793 ------------------------------------
794
795 This command message is sent by the client to the server to query for
796 information about device regions. The VFIO region info structure is defined in
797 ``<linux/vfio.h>`` (``struct vfio_region_info``).
798
799 Request
800 ^^^^^^^
801
802 +------------+--------+------------------------------+
803 | Name | Offset | Size |
804 +============+========+==============================+
805 | argsz | 0 | 4 |
806 +------------+--------+------------------------------+
807 | flags | 4 | 4 |
808 +------------+--------+------------------------------+
809 | index | 8 | 4 |
810 +------------+--------+------------------------------+
811 | cap_offset | 12 | 4 |
812 +------------+--------+------------------------------+
813 | size | 16 | 8 |
814 +------------+--------+------------------------------+
815 | offset | 24 | 8 |
816 +------------+--------+------------------------------+
817
818 * *argsz* the maximum size of the reply payload
819 * *index* is the index of memory region being queried, it is the only field
820 that is required to be set in the command message.
821 * all other fields must be zero.
822
823 Reply
824 ^^^^^
825
826 +------------+--------+------------------------------+
827 | Name | Offset | Size |
828 +============+========+==============================+
829 | argsz | 0 | 4 |
830 +------------+--------+------------------------------+
831 | flags | 4 | 4 |
832 +------------+--------+------------------------------+
833 | | +-----+-----------------------------+ |
834 | | | Bit | Definition | |
835 | | +=====+=============================+ |
836 | | | 0 | VFIO_REGION_INFO_FLAG_READ | |
837 | | +-----+-----------------------------+ |
838 | | | 1 | VFIO_REGION_INFO_FLAG_WRITE | |
839 | | +-----+-----------------------------+ |
840 | | | 2 | VFIO_REGION_INFO_FLAG_MMAP | |
841 | | +-----+-----------------------------+ |
842 | | | 3 | VFIO_REGION_INFO_FLAG_CAPS | |
843 | | +-----+-----------------------------+ |
844 +------------+--------+------------------------------+
845 +------------+--------+------------------------------+
846 | index | 8 | 4 |
847 +------------+--------+------------------------------+
848 | cap_offset | 12 | 4 |
849 +------------+--------+------------------------------+
850 | size | 16 | 8 |
851 +------------+--------+------------------------------+
852 | offset | 24 | 8 |
853 +------------+--------+------------------------------+
854
855 * *argsz* is the size required for the full reply payload (region info structure
856 plus the size of any region capabilities)
857 * *flags* are attributes of the region:
858
859 * ``VFIO_REGION_INFO_FLAG_READ`` allows client read access to the region.
860 * ``VFIO_REGION_INFO_FLAG_WRITE`` allows client write access to the region.
861 * ``VFIO_REGION_INFO_FLAG_MMAP`` specifies the client can mmap() the region.
862 When this flag is set, the reply will include a file descriptor in its
863 meta-data. On ``AF_UNIX`` sockets, the file descriptors will be passed as
864 ``SCM_RIGHTS`` type ancillary data.
865 * ``VFIO_REGION_INFO_FLAG_CAPS`` indicates additional capabilities found in the
866 reply.
867
868 * *index* is the index of memory region being queried, it is the only field
869 that is required to be set in the command message.
870 * *cap_offset* describes where additional region capabilities can be found.
871 cap_offset is relative to the beginning of the VFIO region info structure.
872 The data structure it points is a VFIO cap header defined in
873 ``<linux/vfio.h>``.
874 * *size* is the size of the region.
875 * *offset* is the offset that should be given to the mmap() system call for
876 regions with the MMAP attribute. It is also used as the base offset when
877 mapping a VFIO sparse mmap area, described below.
878
879 VFIO region capabilities
880 """"""""""""""""""""""""
881
882 The VFIO region information can also include a capabilities list. This list is
883 similar to a PCI capability list - each entry has a common header that
884 identifies a capability and where the next capability in the list can be found.
885 The VFIO capability header format is defined in ``<linux/vfio.h>`` (``struct
886 vfio_info_cap_header``).
887
888 VFIO cap header format
889 """"""""""""""""""""""
890
891 +---------+--------+------+
892 | Name | Offset | Size |
893 +=========+========+======+
894 | id | 0 | 2 |
895 +---------+--------+------+
896 | version | 2 | 2 |
897 +---------+--------+------+
898 | next | 4 | 4 |
899 +---------+--------+------+
900
901 * *id* is the capability identity.
902 * *version* is a capability-specific version number.
903 * *next* specifies the offset of the next capability in the capability list. It
904 is relative to the beginning of the VFIO region info structure.
905
906 VFIO sparse mmap cap header
907 """""""""""""""""""""""""""
908
909 +------------------+----------------------------------+
910 | Name | Value |
911 +==================+==================================+
912 | id | VFIO_REGION_INFO_CAP_SPARSE_MMAP |
913 +------------------+----------------------------------+
914 | version | 0x1 |
915 +------------------+----------------------------------+
916 | next | <next> |
917 +------------------+----------------------------------+
918 | sparse mmap info | VFIO region info sparse mmap |
919 +------------------+----------------------------------+
920
921 This capability is defined when only a subrange of the region supports
922 direct access by the client via mmap(). The VFIO sparse mmap area is defined in
923 ``<linux/vfio.h>`` (``struct vfio_region_sparse_mmap_area`` and ``struct
924 vfio_region_info_cap_sparse_mmap``).
925
926 VFIO region info cap sparse mmap
927 """"""""""""""""""""""""""""""""
928
929 +----------+--------+------+
930 | Name | Offset | Size |
931 +==========+========+======+
932 | nr_areas | 0 | 4 |
933 +----------+--------+------+
934 | reserved | 4 | 4 |
935 +----------+--------+------+
936 | offset | 8 | 8 |
937 +----------+--------+------+
938 | size | 16 | 8 |
939 +----------+--------+------+
940 | ... | | |
941 +----------+--------+------+
942
943 * *nr_areas* is the number of sparse mmap areas in the region.
944 * *offset* and size describe a single area that can be mapped by the client.
945 There will be *nr_areas* pairs of offset and size. The offset will be added to
946 the base offset given in the ``VFIO_USER_DEVICE_GET_REGION_INFO`` to form the
947 offset argument of the subsequent mmap() call.
948
949 The VFIO sparse mmap area is defined in ``<linux/vfio.h>`` (``struct
950 vfio_region_info_cap_sparse_mmap``).
951
952
953 ``VFIO_USER_DEVICE_GET_REGION_IO_FDS``
954 --------------------------------------
955
956 Clients can access regions via ``VFIO_USER_REGION_READ/WRITE`` or, if provided, by
957 ``mmap()`` of a file descriptor provided by the server.
958
959 ``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` provides an alternative access mechanism via
960 file descriptors. This is an optional feature intended for performance
961 improvements where an underlying sub-system (such as KVM) supports communication
962 across such file descriptors to the vfio-user server, without needing to
963 round-trip through the client.
964
965 The server returns an array of sub-regions for the requested region. Each
966 sub-region describes a span (offset and size) of a region, along with the
967 requested file descriptor notification mechanism to use. Each sub-region in the
968 response message may choose to use a different method, as defined below. The
969 two mechanisms supported in this specification are ioeventfds and ioregionfds.
970
971 The server in addition returns a file descriptor in the ancillary data; clients
972 are expected to configure each sub-region's file descriptor with the requested
973 notification method. For example, a client could configure KVM with the
974 requested ioeventfd via a ``KVM_IOEVENTFD`` ``ioctl()``.
975
976 Request
977 ^^^^^^^
978
979 +-------------+--------+------+
980 | Name | Offset | Size |
981 +=============+========+======+
982 | argsz | 0 | 4 |
983 +-------------+--------+------+
984 | flags | 4 | 4 |
985 +-------------+--------+------+
986 | index | 8 | 4 |
987 +-------------+--------+------+
988 | count | 12 | 4 |
989 +-------------+--------+------+
990
991 * *argsz* the maximum size of the reply payload
992 * *index* is the index of memory region being queried
993 * all other fields must be zero
994
995 The client must set ``flags`` to zero and specify the region being queried in
996 the ``index``.
997
998 Reply
999 ^^^^^
1000
1001 +-------------+--------+------+
1002 | Name | Offset | Size |
1003 +=============+========+======+
1004 | argsz | 0 | 4 |
1005 +-------------+--------+------+
1006 | flags | 4 | 4 |
1007 +-------------+--------+------+
1008 | index | 8 | 4 |
1009 +-------------+--------+------+
1010 | count | 12 | 4 |
1011 +-------------+--------+------+
1012 | sub-regions | 16 | ... |
1013 +-------------+--------+------+
1014
1015 * *argsz* is the size of the region IO FD info structure plus the
1016 total size of the sub-region array. Thus, each array entry "i" is at offset
1017 i * ((argsz - 32) / count). Note that currently this is 40 bytes for both IO
1018 FD types, but this is not to be relied on. As elsewhere, this indicates the
1019 full reply payload size needed.
1020 * *flags* must be zero
1021 * *index* is the index of memory region being queried
1022 * *count* is the number of sub-regions in the array
1023 * *sub-regions* is the array of Sub-Region IO FD info structures
1024
1025 The reply message will additionally include at least one file descriptor in the
1026 ancillary data. Note that more than one sub-region may share the same file
1027 descriptor.
1028
1029 Note that it is the client's responsibility to verify the requested values (for
1030 example, that the requested offset does not exceed the region's bounds).
1031
1032 Each sub-region given in the response has one of two possible structures,
1033 depending whether *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD`` or
1034 ``VFIO_USER_IO_FD_TYPE_IOREGIONFD``:
1035
1036 Sub-Region IO FD info format (ioeventfd)
1037 """"""""""""""""""""""""""""""""""""""""
1038
1039 +-----------+--------+------+
1040 | Name | Offset | Size |
1041 +===========+========+======+
1042 | offset | 0 | 8 |
1043 +-----------+--------+------+
1044 | size | 8 | 8 |
1045 +-----------+--------+------+
1046 | fd_index | 16 | 4 |
1047 +-----------+--------+------+
1048 | type | 20 | 4 |
1049 +-----------+--------+------+
1050 | flags | 24 | 4 |
1051 +-----------+--------+------+
1052 | padding | 28 | 4 |
1053 +-----------+--------+------+
1054 | datamatch | 32 | 8 |
1055 +-----------+--------+------+
1056
1057 * *offset* is the offset of the start of the sub-region within the region
1058 requested ("physical address offset" for the region)
1059 * *size* is the length of the sub-region. This may be zero if the access size is
1060 not relevant, which may allow for optimizations
1061 * *fd_index* is the index in the ancillary data of the FD to use for ioeventfd
1062 notification; it may be shared.
1063 * *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD``
1064 * *flags* is any of:
1065
1066 * ``KVM_IOEVENTFD_FLAG_DATAMATCH``
1067 * ``KVM_IOEVENTFD_FLAG_PIO``
1068 * ``KVM_IOEVENTFD_FLAG_VIRTIO_CCW_NOTIFY`` (FIXME: makes sense?)
1069
1070 * *datamatch* is the datamatch value if needed
1071
1072 See https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt, *4.59
1073 KVM_IOEVENTFD* for further context on the ioeventfd-specific fields.
1074
1075 Sub-Region IO FD info format (ioregionfd)
1076 """""""""""""""""""""""""""""""""""""""""
1077
1078 +-----------+--------+------+
1079 | Name | Offset | Size |
1080 +===========+========+======+
1081 | offset | 0 | 8 |
1082 +-----------+--------+------+
1083 | size | 8 | 8 |
1084 +-----------+--------+------+
1085 | fd_index | 16 | 4 |
1086 +-----------+--------+------+
1087 | type | 20 | 4 |
1088 +-----------+--------+------+
1089 | flags | 24 | 4 |
1090 +-----------+--------+------+
1091 | padding | 28 | 4 |
1092 +-----------+--------+------+
1093 | user_data | 32 | 8 |
1094 +-----------+--------+------+
1095
1096 * *offset* is the offset of the start of the sub-region within the region
1097 requested ("physical address offset" for the region)
1098 * *size* is the length of the sub-region. This may be zero if the access size is
1099 not relevant, which may allow for optimizations; ``KVM_IOREGION_POSTED_WRITES``
1100 must be set in *flags* in this case
1101 * *fd_index* is the index in the ancillary data of the FD to use for ioregionfd
1102 messages; it may be shared
1103 * *type* is ``VFIO_USER_IO_FD_TYPE_IOREGIONFD``
1104 * *flags* is any of:
1105
1106 * ``KVM_IOREGION_PIO``
1107 * ``KVM_IOREGION_POSTED_WRITES``
1108
1109 * *user_data* is an opaque value passed back to the server via a message on the
1110 file descriptor
1111
1112 For further information on the ioregionfd-specific fields, see:
1113 https://lore.kernel.org/kvm/cover.1613828726.git.eafanasova@gmail.com/
1114
1115 (FIXME: update with final API docs.)
1116
1117 ``VFIO_USER_DEVICE_GET_IRQ_INFO``
1118 ---------------------------------
1119
1120 This command message is sent by the client to the server to query for
1121 information about device interrupt types. The VFIO IRQ info structure is
1122 defined in ``<linux/vfio.h>`` (``struct vfio_irq_info``).
1123
1124 Request
1125 ^^^^^^^
1126
1127 +-------+--------+---------------------------+
1128 | Name | Offset | Size |
1129 +=======+========+===========================+
1130 | argsz | 0 | 4 |
1131 +-------+--------+---------------------------+
1132 | flags | 4 | 4 |
1133 +-------+--------+---------------------------+
1134 | | +-----+--------------------------+ |
1135 | | | Bit | Definition | |
1136 | | +=====+==========================+ |
1137 | | | 0 | VFIO_IRQ_INFO_EVENTFD | |
1138 | | +-----+--------------------------+ |
1139 | | | 1 | VFIO_IRQ_INFO_MASKABLE | |
1140 | | +-----+--------------------------+ |
1141 | | | 2 | VFIO_IRQ_INFO_AUTOMASKED | |
1142 | | +-----+--------------------------+ |
1143 | | | 3 | VFIO_IRQ_INFO_NORESIZE | |
1144 | | +-----+--------------------------+ |
1145 +-------+--------+---------------------------+
1146 | index | 8 | 4 |
1147 +-------+--------+---------------------------+
1148 | count | 12 | 4 |
1149 +-------+--------+---------------------------+
1150
1151 * *argsz* is the maximum size of the reply payload (16 bytes today)
1152 * index is the index of IRQ type being queried (e.g. ``VFIO_PCI_MSIX_IRQ_INDEX``)
1153 * all other fields must be zero
1154
1155 Reply
1156 ^^^^^
1157
1158 +-------+--------+---------------------------+
1159 | Name | Offset | Size |
1160 +=======+========+===========================+
1161 | argsz | 0 | 4 |
1162 +-------+--------+---------------------------+
1163 | flags | 4 | 4 |
1164 +-------+--------+---------------------------+
1165 | | +-----+--------------------------+ |
1166 | | | Bit | Definition | |
1167 | | +=====+==========================+ |
1168 | | | 0 | VFIO_IRQ_INFO_EVENTFD | |
1169 | | +-----+--------------------------+ |
1170 | | | 1 | VFIO_IRQ_INFO_MASKABLE | |
1171 | | +-----+--------------------------+ |
1172 | | | 2 | VFIO_IRQ_INFO_AUTOMASKED | |
1173 | | +-----+--------------------------+ |
1174 | | | 3 | VFIO_IRQ_INFO_NORESIZE | |
1175 | | +-----+--------------------------+ |
1176 +-------+--------+---------------------------+
1177 | index | 8 | 4 |
1178 +-------+--------+---------------------------+
1179 | count | 12 | 4 |
1180 +-------+--------+---------------------------+
1181
1182 * *argsz* is the size required for the full reply payload (16 bytes today)
1183 * *flags* defines IRQ attributes:
1184
1185 * ``VFIO_IRQ_INFO_EVENTFD`` indicates the IRQ type can support server eventfd
1186 signalling.
1187 * ``VFIO_IRQ_INFO_MASKABLE`` indicates that the IRQ type supports the ``MASK``
1188 and ``UNMASK`` actions in a ``VFIO_USER_DEVICE_SET_IRQS`` message.
1189 * ``VFIO_IRQ_INFO_AUTOMASKED`` indicates the IRQ type masks itself after being
1190 triggered, and the client must send an ``UNMASK`` action to receive new
1191 interrupts.
1192 * ``VFIO_IRQ_INFO_NORESIZE`` indicates ``VFIO_USER_SET_IRQS`` operations setup
1193 interrupts as a set, and new sub-indexes cannot be enabled without disabling
1194 the entire type.
1195 * index is the index of IRQ type being queried
1196 * count describes the number of interrupts of the queried type.
1197
1198 ``VFIO_USER_DEVICE_SET_IRQS``
1199 -----------------------------
1200
1201 This command message is sent by the client to the server to set actions for
1202 device interrupt types. The VFIO IRQ set structure is defined in
1203 ``<linux/vfio.h>`` (``struct vfio_irq_set``).
1204
1205 Request
1206 ^^^^^^^
1207
1208 +-------+--------+------------------------------+
1209 | Name | Offset | Size |
1210 +=======+========+==============================+
1211 | argsz | 0 | 4 |
1212 +-------+--------+------------------------------+
1213 | flags | 4 | 4 |
1214 +-------+--------+------------------------------+
1215 | | +-----+-----------------------------+ |
1216 | | | Bit | Definition | |
1217 | | +=====+=============================+ |
1218 | | | 0 | VFIO_IRQ_SET_DATA_NONE | |
1219 | | +-----+-----------------------------+ |
1220 | | | 1 | VFIO_IRQ_SET_DATA_BOOL | |
1221 | | +-----+-----------------------------+ |
1222 | | | 2 | VFIO_IRQ_SET_DATA_EVENTFD | |
1223 | | +-----+-----------------------------+ |
1224 | | | 3 | VFIO_IRQ_SET_ACTION_MASK | |
1225 | | +-----+-----------------------------+ |
1226 | | | 4 | VFIO_IRQ_SET_ACTION_UNMASK | |
1227 | | +-----+-----------------------------+ |
1228 | | | 5 | VFIO_IRQ_SET_ACTION_TRIGGER | |
1229 | | +-----+-----------------------------+ |
1230 +-------+--------+------------------------------+
1231 | index | 8 | 4 |
1232 +-------+--------+------------------------------+
1233 | start | 12 | 4 |
1234 +-------+--------+------------------------------+
1235 | count | 16 | 4 |
1236 +-------+--------+------------------------------+
1237 | data | 20 | variable |
1238 +-------+--------+------------------------------+
1239
1240 * *argsz* is the size of the VFIO IRQ set request payload, including any *data*
1241 field. Note there is no reply payload, so this field differs from other
1242 message types.
1243 * *flags* defines the action performed on the interrupt range. The ``DATA``
1244 flags describe the data field sent in the message; the ``ACTION`` flags
1245 describe the action to be performed. The flags are mutually exclusive for
1246 both sets.
1247
1248 * ``VFIO_IRQ_SET_DATA_NONE`` indicates there is no data field in the command.
1249 The action is performed unconditionally.
1250 * ``VFIO_IRQ_SET_DATA_BOOL`` indicates the data field is an array of boolean
1251 bytes. The action is performed if the corresponding boolean is true.
1252 * ``VFIO_IRQ_SET_DATA_EVENTFD`` indicates an array of event file descriptors
1253 was sent in the message meta-data. These descriptors will be signalled when
1254 the action defined by the action flags occurs. In ``AF_UNIX`` sockets, the
1255 descriptors are sent as ``SCM_RIGHTS`` type ancillary data.
1256 If no file descriptors are provided, this de-assigns the specified
1257 previously configured interrupts.
1258 * ``VFIO_IRQ_SET_ACTION_MASK`` indicates a masking event. It can be used with
1259 ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to mask an interrupt,
1260 or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the guest masks
1261 the interrupt.
1262 * ``VFIO_IRQ_SET_ACTION_UNMASK`` indicates an unmasking event. It can be used
1263 with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to unmask an
1264 interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the
1265 guest unmasks the interrupt.
1266 * ``VFIO_IRQ_SET_ACTION_TRIGGER`` indicates a triggering event. It can be used
1267 with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to trigger an
1268 interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the
1269 server triggers the interrupt.
1270
1271 * *index* is the index of IRQ type being setup.
1272 * *start* is the start of the sub-index being set.
1273 * *count* describes the number of sub-indexes being set. As a special case, a
1274 count (and start) of 0, with data flags of ``VFIO_IRQ_SET_DATA_NONE`` disables
1275 all interrupts of the index.
1276 * *data* is an optional field included when the
1277 ``VFIO_IRQ_SET_DATA_BOOL`` flag is present. It contains an array of booleans
1278 that specify whether the action is to be performed on the corresponding
1279 index. It's used when the action is only performed on a subset of the range
1280 specified.
1281
1282 Not all interrupt types support every combination of data and action flags.
1283 The client must know the capabilities of the device and IRQ index before it
1284 sends a ``VFIO_USER_DEVICE_SET_IRQ`` message.
1285
1286 In typical operation, a specific IRQ may operate as follows:
1287
1288 1. The client sends a ``VFIO_USER_DEVICE_SET_IRQ`` message with
1289 ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_TRIGGER)`` along
1290 with an eventfd. This associates the IRQ with a particular eventfd on the
1291 server side.
1292
1293 #. The client may send a ``VFIO_USER_DEVICE_SET_IRQ`` message with
1294 ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_MASK/UNMASK)`` along
1295 with another eventfd. This associates the given eventfd with the
1296 mask/unmask state on the server side.
1297
1298 #. The server may trigger the IRQ by writing 1 to the eventfd.
1299
1300 #. The server may mask/unmask an IRQ which will write 1 to the corresponding
1301 mask/unmask eventfd, if there is one.
1302
1303 5. A client may trigger a device IRQ itself, by sending a
1304 ``VFIO_USER_DEVICE_SET_IRQ`` message with
1305 ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_TRIGGER)``.
1306
1307 6. A client may mask or unmask the IRQ, by sending a
1308 ``VFIO_USER_DEVICE_SET_IRQ`` message with
1309 ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_MASK/UNMASK)``.
1310
1311 Reply
1312 ^^^^^
1313
1314 There is no payload in the reply.
1315
1316 .. _Read and Write Operations:
1317
1318 Note that all of these operations must be supported by the client and/or server,
1319 even if the corresponding memory or device region has been shared as mappable.
1320
1321 The ``count`` field must not exceed the value of ``max_data_xfer_size`` of the
1322 peer, for both reads and writes.
1323
1324 ``VFIO_USER_REGION_READ``
1325 -------------------------
1326
1327 If a device region is not mappable, it's not directly accessible by the client
1328 via ``mmap()`` of the underlying file descriptor. In this case, a client can
1329 read from a device region with this message.
1330
1331 Request
1332 ^^^^^^^
1333
1334 +--------+--------+----------+
1335 | Name | Offset | Size |
1336 +========+========+==========+
1337 | offset | 0 | 8 |
1338 +--------+--------+----------+
1339 | region | 8 | 4 |
1340 +--------+--------+----------+
1341 | count | 12 | 4 |
1342 +--------+--------+----------+
1343
1344 * *offset* into the region being accessed.
1345 * *region* is the index of the region being accessed.
1346 * *count* is the size of the data to be transferred.
1347
1348 Reply
1349 ^^^^^
1350
1351 +--------+--------+----------+
1352 | Name | Offset | Size |
1353 +========+========+==========+
1354 | offset | 0 | 8 |
1355 +--------+--------+----------+
1356 | region | 8 | 4 |
1357 +--------+--------+----------+
1358 | count | 12 | 4 |
1359 +--------+--------+----------+
1360 | data | 16 | variable |
1361 +--------+--------+----------+
1362
1363 * *offset* into the region accessed.
1364 * *region* is the index of the region accessed.
1365 * *count* is the size of the data transferred.
1366 * *data* is the data that was read from the device region.
1367
1368 ``VFIO_USER_REGION_WRITE``
1369 --------------------------
1370
1371 If a device region is not mappable, it's not directly accessible by the client
1372 via mmap() of the underlying fd. In this case, a client can write to a device
1373 region with this message.
1374
1375 Request
1376 ^^^^^^^
1377
1378 +--------+--------+----------+
1379 | Name | Offset | Size |
1380 +========+========+==========+
1381 | offset | 0 | 8 |
1382 +--------+--------+----------+
1383 | region | 8 | 4 |
1384 +--------+--------+----------+
1385 | count | 12 | 4 |
1386 +--------+--------+----------+
1387 | data | 16 | variable |
1388 +--------+--------+----------+
1389
1390 * *offset* into the region being accessed.
1391 * *region* is the index of the region being accessed.
1392 * *count* is the size of the data to be transferred.
1393 * *data* is the data to write
1394
1395 Reply
1396 ^^^^^
1397
1398 +--------+--------+----------+
1399 | Name | Offset | Size |
1400 +========+========+==========+
1401 | offset | 0 | 8 |
1402 +--------+--------+----------+
1403 | region | 8 | 4 |
1404 +--------+--------+----------+
1405 | count | 12 | 4 |
1406 +--------+--------+----------+
1407
1408 * *offset* into the region accessed.
1409 * *region* is the index of the region accessed.
1410 * *count* is the size of the data transferred.
1411
1412 ``VFIO_USER_DMA_READ``
1413 -----------------------
1414
1415 If the client has not shared mappable memory, the server can use this message to
1416 read from guest memory. This message and its reply are passed over the separate
1417 server-to-client socket if twin-socket mode has been negotiated during
1418 connection setup.
1419
1420 Request
1421 ^^^^^^^
1422
1423 +---------+--------+----------+
1424 | Name | Offset | Size |
1425 +=========+========+==========+
1426 | address | 0 | 8 |
1427 +---------+--------+----------+
1428 | count | 8 | 8 |
1429 +---------+--------+----------+
1430
1431 * *address* is the client DMA memory address being accessed. This address must have
1432 been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message.
1433 * *count* is the size of the data to be transferred.
1434
1435 Reply
1436 ^^^^^
1437
1438 +---------+--------+----------+
1439 | Name | Offset | Size |
1440 +=========+========+==========+
1441 | address | 0 | 8 |
1442 +---------+--------+----------+
1443 | count | 8 | 8 |
1444 +---------+--------+----------+
1445 | data | 16 | variable |
1446 +---------+--------+----------+
1447
1448 * *address* is the client DMA memory address being accessed.
1449 * *count* is the size of the data transferred.
1450 * *data* is the data read.
1451
1452 Note that whether short reads return an error or just set count appropriately is
1453 a client-side choice; servers should be prepared to handle both cases.
1454
1455 ``VFIO_USER_DMA_WRITE``
1456 -----------------------
1457
1458 If the client has not shared mappable memory, the server can use this message to
1459 write to guest memory. This message and its reply are passed over the separate
1460 server-to-client socket if twin-socket mode has been negotiated during
1461 connection setup.
1462
1463 Request
1464 ^^^^^^^
1465
1466 +---------+--------+----------+
1467 | Name | Offset | Size |
1468 +=========+========+==========+
1469 | address | 0 | 8 |
1470 +---------+--------+----------+
1471 | count | 8 | 8 |
1472 +---------+--------+----------+
1473 | data | 16 | variable |
1474 +---------+--------+----------+
1475
1476 * *address* is the client DMA memory address being accessed. This address must have
1477 been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message.
1478 * *count* is the size of the data to be transferred.
1479 * *data* is the data to write
1480
1481 Reply
1482 ^^^^^
1483
1484 +---------+--------+----------+
1485 | Name | Offset | Size |
1486 +=========+========+==========+
1487 | address | 0 | 8 |
1488 +---------+--------+----------+
1489 | count | 8 | 8 |
1490 +---------+--------+----------+
1491
1492 * *address* is the client DMA memory address being accessed.
1493 * *count* is the size of the data transferred.
1494
1495 Note that whether short writes return an error or just set count appropriately
1496 is a client-side choice; servers should be prepared to handle both cases.
1497
1498 ``VFIO_USER_DEVICE_RESET``
1499 --------------------------
1500
1501 This command message is sent from the client to the server to reset the device.
1502 Neither the request or reply have a payload.
1503
1504 ``VFIO_USER_REGION_WRITE_MULTI``
1505 --------------------------------
1506
1507 This message can be used to coalesce multiple device write operations
1508 into a single messgage. It is only used as an optimization when the
1509 outgoing message queue is relatively full.
1510
1511 Request
1512 ^^^^^^^
1513
1514 +---------+--------+----------+
1515 | Name | Offset | Size |
1516 +=========+========+==========+
1517 | wr_cnt | 0 | 8 |
1518 +---------+--------+----------+
1519 | wrs | 8 | variable |
1520 +---------+--------+----------+
1521
1522 * *wr_cnt* is the number of device writes coalesced in the message
1523 * *wrs* is an array of device writes defined below
1524
1525 Single Device Write Format
1526 """"""""""""""""""""""""""
1527
1528 +--------+--------+----------+
1529 | Name | Offset | Size |
1530 +========+========+==========+
1531 | offset | 0 | 8 |
1532 +--------+--------+----------+
1533 | region | 8 | 4 |
1534 +--------+--------+----------+
1535 | count | 12 | 4 |
1536 +--------+--------+----------+
1537 | data | 16 | 8 |
1538 +--------+--------+----------+
1539
1540 * *offset* into the region being accessed.
1541 * *region* is the index of the region being accessed.
1542 * *count* is the size of the data to be transferred. This format can
1543 only describe writes of 8 bytes or less.
1544 * *data* is the data to write.
1545
1546 Reply
1547 ^^^^^
1548
1549 +---------+--------+----------+
1550 | Name | Offset | Size |
1551 +=========+========+==========+
1552 | wr_cnt | 0 | 8 |
1553 +---------+--------+----------+
1554
1555 * *wr_cnt* is the number of device writes completed.
1556
1557 ``VFIO_USER_DEVICE_FEATURE``
1558 ----------------------------
1559
1560 This command is analogous to ``VFIO_DEVICE_FEATURE``. It is used to get, set, or
1561 probe feature data of the device.
1562
1563 Request
1564 ^^^^^^^
1565
1566 The request payload for this message is a structure of the following format.
1567
1568 +-------+--------+--------------------------------+
1569 | Name | Offset | Size |
1570 +=======+========+================================+
1571 | argsz | 0 | 4 |
1572 +-------+--------+--------------------------------+
1573 | flags | 4 | 4 |
1574 +-------+--------+--------------------------------+
1575 | | +---------+---------------------------+ |
1576 | | | Bit | Definition | |
1577 | | +=========+===========================+ |
1578 | | | 0 to 15 | Feature index | |
1579 | | +---------+---------------------------+ |
1580 | | | 16 | VFIO_DEVICE_FEATURE_GET | |
1581 | | +---------+---------------------------+ |
1582 | | | 17 | VFIO_DEVICE_FEATURE_SET | |
1583 | | +---------+---------------------------+ |
1584 | | | 18 | VFIO_DEVICE_FEATURE_PROBE | |
1585 | | +---------+---------------------------+ |
1586 +-------+--------+--------------------------------+
1587 | data | 8 | variable |
1588 +-------+--------+--------------------------------+
1589
1590 * *argsz* is the maximum size of the reply payload.
1591
1592 * *flags* defines the action to be performed by the server and upon which
1593 feature:
1594
1595 * The feature index consists of the least significant 16 bits of the flags
1596 field, and can be accessed using the ``VFIO_DEVICE_FEATURE_MASK`` bit mask.
1597
1598 * ``VFIO_DEVICE_FEATURE_GET`` instructs the server to get the data for the
1599 given feature.
1600
1601 * ``VFIO_DEVICE_FEATURE_SET`` instructs the server to set the feature data to
1602 that given in the ``data`` field of the payload.
1603
1604 * ``VFIO_DEVICE_FEATURE_PROBE`` instructs the server to probe for feature
1605 support. If ``VFIO_DEVICE_FEATURE_GET`` and/or ``VFIO_DEVICE_FEATURE_SET``
1606 are also set, the probe will only return success if all of the indicated
1607 methods are supported.
1608
1609 ``VFIO_DEVICE_FEATURE_GET`` and ``VFIO_DEVICE_FEATURE_SET`` are mutually
1610 exclusive, except for use with ``VFIO_DEVICE_FEATURE_PROBE``.
1611
1612 * *data* is specific to the particular feature. It is not used for probing.
1613
1614 This part of the request is analogous to VFIO's ``struct vfio_device_feature``.
1615
1616 Reply
1617 ^^^^^
1618
1619 The reply payload must be the same as the request payload for setting or
1620 probing a feature. For getting a feature's data, the data is added in the data
1621 section and its length is added to ``argsz``.
1622
1623 Device Features
1624 ^^^^^^^^^^^^^^^
1625
1626 The only device features supported by vfio-user are those related to migration,
1627 although this may change in the future. They are a subset of those supported in
1628 the VFIO implementation of the Linux kernel.
1629
1630 +----------------------------------------+---------------+
1631 | Name | Feature Index |
1632 +========================================+===============+
1633 | VFIO_DEVICE_FEATURE_MIGRATION | 1 |
1634 +----------------------------------------+---------------+
1635 | VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE | 2 |
1636 +----------------------------------------+---------------+
1637 | VFIO_DEVICE_FEATURE_DMA_LOGGING_START | 6 |
1638 +----------------------------------------+---------------+
1639 | VFIO_DEVICE_FEATURE_DMA_LOGGING_STOP | 7 |
1640 +----------------------------------------+---------------+
1641 | VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT | 8 |
1642 +----------------------------------------+---------------+
1643
1644 ``VFIO_DEVICE_FEATURE_MIGRATION``
1645 """""""""""""""""""""""""""""""""
1646
1647 This feature indicates that the device can support the migration API through
1648 ``VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE``. If ``GET`` succeeds, the ``RUNNING``
1649 and ``ERROR`` states are always supported. Support for additional states is
1650 indicated via the flags field; at least ``VFIO_MIGRATION_STOP_COPY`` must be
1651 set.
1652
1653 There is no data field of the request message.
1654
1655 The data field of the reply message is structured as follows:
1656
1657 +-------+--------+---------------------------+
1658 | Name | Offset | Size |
1659 +=======+========+===========================+
1660 | flags | 0 | 8 |
1661 +-------+--------+---------------------------+
1662 | | +-----+--------------------------+ |
1663 | | | Bit | Definition | |
1664 | | +=====+==========================+ |
1665 | | | 0 | VFIO_MIGRATION_STOP_COPY | |
1666 | | +-----+--------------------------+ |
1667 | | | 1 | VFIO_MIGRATION_P2P | |
1668 | | +-----+--------------------------+ |
1669 | | | 2 | VFIO_MIGRATION_PRE_COPY | |
1670 | | +-----+--------------------------+ |
1671 +-------+--------+---------------------------+
1672
1673 These flags are interpreted in the same way as VFIO.
1674
1675 ``VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE``
1676 """"""""""""""""""""""""""""""""""""""""
1677
1678 Upon ``VFIO_DEVICE_FEATURE_SET``, execute a migration state change on the VFIO
1679 device. The new state is supplied in ``device_state``. The state transition must
1680 fully complete before the reply is sent.
1681
1682 The data field of the reply message, as well as the ``SET`` request message, is
1683 structured as follows:
1684
1685 +--------------+--------+------+
1686 | Name | Offset | Size |
1687 +==============+========+======+
1688 | device_state | 0 | 4 |
1689 +--------------+--------+------+
1690 | data_fd | 4 | 4 |
1691 +--------------+--------+------+
1692
1693 * *device_state* is the current state of the device (for ``GET``) or the
1694 state to transition to (for ``SET``). It is defined by the
1695 ``vfio_device_mig_state`` enum as detailed below. These states are the states
1696 of the device migration Finite State Machine.
1697
1698 +--------------------------------+-------+---------------------------------------------------------------------+
1699 | Name | State | Description |
1700 +================================+=======+=====================================================================+
1701 | VFIO_DEVICE_STATE_ERROR | 0 | The device has failed and must be reset. |
1702 +--------------------------------+-------+---------------------------------------------------------------------+
1703 | VFIO_DEVICE_STATE_STOP | 1 | The device does not change the internal or external state. |
1704 +--------------------------------+-------+---------------------------------------------------------------------+
1705 | VFIO_DEVICE_STATE_RUNNING | 2 | The device is running normally. |
1706 +--------------------------------+-------+---------------------------------------------------------------------+
1707 | VFIO_DEVICE_STATE_STOP_COPY | 3 | The device internal state can be read out. |
1708 +--------------------------------+-------+---------------------------------------------------------------------+
1709 | VFIO_DEVICE_STATE_RESUMING | 4 | The device is stopped and is loading a new internal state. |
1710 +--------------------------------+-------+---------------------------------------------------------------------+
1711 | VFIO_DEVICE_STATE_RUNNING_P2P | 5 | (not used in vfio-user) |
1712 +--------------------------------+-------+---------------------------------------------------------------------+
1713 | VFIO_DEVICE_STATE_PRE_COPY | 6 | The device is running normally but tracking internal state changes. |
1714 +--------------------------------+-------+---------------------------------------------------------------------+
1715 | VFIO_DEVICE_STATE_PRE_COPY_P2P | 7 | (not used in vfio-user) |
1716 +--------------------------------+-------+---------------------------------------------------------------------+
1717
1718 * *data_fd* is unused in vfio-user, as the ``VFIO_USER_MIG_DATA_READ`` and
1719 ``VFIO_USER_MIG_DATA_WRITE`` messages are used instead for migration data
1720 transport.
1721
1722 Direct State Transitions
1723 """"""""""""""""""""""""
1724
1725 The device migration FSM is a Mealy machine, so actions are taken upon the arcs
1726 between FSM states. The following transitions need to be supported by the
1727 server, a subset of those defined in ``<linux/vfio.h>``
1728 (``enum vfio_device_mig_state``).
1729
1730 * ``RUNNING -> STOP``, ``STOP_COPY -> STOP``: Stop the operation of the device.
1731 The ``STOP_COPY`` arc terminates the data transfer session.
1732
1733 * ``RESUMING -> STOP``: Terminate the data transfer session. Complete processing
1734 of the migration data. Stop the operation of the device. If the delivered data
1735 is found to be incomplete, inconsistent, or otherwise invalid, fail the
1736 ``SET`` command and optionally transition to the ``ERROR`` state.
1737
1738 * ``PRE_COPY -> RUNNING``: Terminate the data transfer session. The device is
1739 now fully operational.
1740
1741 * ``STOP -> RUNNING``: Start the operation of the device.
1742
1743 * ``RUNNING -> PRE_COPY``, ``STOP -> STOP_COPY``: Begin the process of saving
1744 the device state. The device operation is unchanged, but data transfer begins.
1745 ``PRE_COPY`` and ``STOP_COPY`` are referred to as the "saving group" of
1746 states.
1747
1748 * ``PRE_COPY -> STOP_COPY``: Continue to transfer migration data, but stop
1749 device operation.
1750
1751 * ``STOP -> RESUMING``: Start the process of restoring the device state. The
1752 internal device state may be changed to prepare the device to receive the
1753 migration data.
1754
1755 The ``STOP_COPY -> PRE_COPY`` transition is explicitly not allowed and should
1756 return an error if requested.
1757
1758 ``ERROR`` cannot be specified as a device state, but any transition request can
1759 be failed and then move the state into ``ERROR`` if the server was unable to
1760 execute the requested arc AND was unable to restore the device into any valid
1761 state. To recover from ``ERROR``, ``VFIO_USER_DEVICE_RESET`` must be used to
1762 return back to ``RUNNING``.
1763
1764 If ``PRE_COPY`` is not supported, arcs touching it are removed.
1765
1766 Complex State Transitions
1767 """""""""""""""""""""""""
1768
1769 The remaining possible transitions are to be implemented as combinations of the
1770 above FSM arcs. As there are multiple paths, the path should be selected based
1771 on the following rules:
1772
1773 * Select the shortest path.
1774
1775 * The path cannot have saving group states as interior arcs, only start/end
1776 states.
1777
1778 ``VFIO_DEVICE_FEATURE_DMA_LOGGING_START`` / ``VFIO_DEVICE_FEATURE_DMA_LOGGING_STOP``
1779 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
1780
1781 Upon ``VFIO_DEVICE_FEATURE_SET``, start/stop DMA logging. These features can
1782 also be probed to determine whether the device supports DMA logging.
1783
1784 When DMA logging is started, a range of IOVAs to monitor is provided and the
1785 device can optimize its logging to cover only the IOVA range given. Only DMA
1786 writes are logged.
1787
1788 The data field of the ``SET`` request is structured as follows:
1789
1790 +------------+--------+----------+
1791 | Name | Offset | Size |
1792 +============+========+==========+
1793 | page_size | 0 | 8 |
1794 +------------+--------+----------+
1795 | num_ranges | 8 | 4 |
1796 +------------+--------+----------+
1797 | reserved | 12 | 4 |
1798 +------------+--------+----------+
1799 | ranges | 16 | variable |
1800 +------------+--------+----------+
1801
1802 * *page_size* hints what tracking granularity the device should try to achieve.
1803 If the device cannot do the hinted page size then it's the driver's choice
1804 which page size to pick based on its support. On output the device will return
1805 the page size it selected.
1806
1807 * *num_ranges* is the number of IOVA ranges to monitor. A value of zero
1808 indicates that all writes should be logged.
1809
1810 * *ranges* is an array of ``vfio_user_device_feature_dma_logging_range``
1811 entries:
1812
1813 +--------+--------+------+
1814 | Name | Offset | Size |
1815 +========+========+======+
1816 | iova | 0 | 8 |
1817 +--------+--------+------+
1818 | length | 8 | 8 |
1819 +--------+--------+------+
1820
1821 * *iova* is the base IO virtual address
1822 * *length* is the length of the range to log
1823
1824 Upon success, the response data field will be the same as the request, unless
1825 the page size was changed, in which case this will be reflected in the response.
1826
1827 ``VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT``
1828 """"""""""""""""""""""""""""""""""""""""""
1829
1830 Upon ``VFIO_DEVICE_FEATURE_GET``, returns the dirty bitmap for a specific IOVA
1831 range. This operation is only valid if logging of dirty pages has been
1832 previously started by setting ``VFIO_DEVICE_FEATURE_DMA_LOGGING_START``.
1833
1834 The data field of the request is structured as follows:
1835
1836 +-----------+--------+------+
1837 | Name | Offset | Size |
1838 +===========+========+======+
1839 | iova | 0 | 8 |
1840 +-----------+--------+------+
1841 | length | 8 | 8 |
1842 +-----------+--------+------+
1843 | page_size | 16 | 8 |
1844 +-----------+--------+------+
1845
1846 * *iova* is the base IO virtual address
1847
1848 * *length* is the length of the range
1849
1850 * *page_size* is the unit of granularity of the bitmap, and must be a power of
1851 two. It doesn't have to match the value given to
1852 ``VFIO_DEVICE_FEATURE_DMA_LOGGING_START`` because the driver will format its
1853 internal logging to match the reporting page size possibly by replicating bits
1854 if the internal page size is lower than requested
1855
1856 The data field of the response is identical, except with the bitmap added on
1857 the end at offset 24.
1858
1859 The bitmap is an array of u64s that holds the output bitmap, with 1 bit
1860 reporting a *page_size* unit of IOVA. The bits outside of the requested range
1861 must be zero.
1862
1863 The mapping of IOVA to bits is given by:
1864
1865 ``bitmap[(addr - iova)/page_size] & (1ULL << (addr % 64))``
1866
1867 ``VFIO_USER_MIG_DATA_READ``
1868 ---------------------------
1869
1870 This command is used to read data from the source migration server while it is
1871 in a saving group state (``PRE_COPY`` or ``STOP_COPY``).
1872
1873 This command, and ``VFIO_USER_MIG_DATA_WRITE``, are used in place of the
1874 ``data_fd`` file descriptor in ``<linux/vfio.h>``
1875 (``struct vfio_device_feature_mig_state``) to enable all data transport to use
1876 the single already-established UNIX socket. Hence, the migration data is
1877 treated like a stream, so the client must continue reading until no more
1878 migration data remains.
1879
1880 Request
1881 ^^^^^^^
1882
1883 The request payload for this message is a structure of the following format.
1884
1885 +-------+--------+------+
1886 | Name | Offset | Size |
1887 +=======+========+======+
1888 | argsz | 0 | 4 |
1889 +-------+--------+------+
1890 | size | 4 | 4 |
1891 +-------+--------+------+
1892
1893 * *argsz* is the maximum size of the reply payload.
1894
1895 * *size* is the size of the migration data to read.
1896
1897 Reply
1898 ^^^^^
1899
1900 The reply payload for this message is a structure of the following format.
1901
1902 +-------+--------+----------+
1903 | Name | Offset | Size |
1904 +=======+========+==========+
1905 | argsz | 0 | 4 |
1906 +-------+--------+----------+
1907 | size | 4 | 4 |
1908 +-------+--------+----------+
1909 | data | 8 | variable |
1910 +-------+--------+----------+
1911
1912 * *argsz* is the size of the above structure, including the size of the data.
1913
1914 * *size* indicates the size of returned migration data. If this is less than the
1915 requested size, there is no more migration data to read.
1916
1917 * *data* contains the migration data.
1918
1919 ``VFIO_USER_MIG_DATA_WRITE``
1920 ----------------------------
1921
1922 This command is used to write data to the destination migration server while it
1923 is in the ``RESUMING`` state.
1924
1925 As above, this replaces the ``data_fd`` file descriptor for transport of
1926 migration data, and as such, the migration data is treated like a stream.
1927
1928 Request
1929 ^^^^^^^
1930
1931 The request payload for this message is a structure of the following format.
1932
1933 +-------+--------+----------+
1934 | Name | Offset | Size |
1935 +=======+========+==========+
1936 | argsz | 0 | 4 |
1937 +-------+--------+----------+
1938 | size | 4 | 4 |
1939 +-------+--------+----------+
1940 | data | 8 | variable |
1941 +-------+--------+----------+
1942
1943 * *argsz* is the maximum size of the reply payload.
1944
1945 * *size* is the size of the migration data to be written.
1946
1947 * *data* contains the migration data.
1948
1949 Reply
1950 ^^^^^
1951
1952 There is no reply payload for this message.
1953
1954 Appendices
1955 ==========
1956
1957 Unused VFIO ``ioctl()`` commands
1958 --------------------------------
1959
1960 The following VFIO commands do not have an equivalent vfio-user command:
1961
1962 * ``VFIO_GET_API_VERSION``
1963 * ``VFIO_CHECK_EXTENSION``
1964 * ``VFIO_SET_IOMMU``
1965 * ``VFIO_GROUP_GET_STATUS``
1966 * ``VFIO_GROUP_SET_CONTAINER``
1967 * ``VFIO_GROUP_UNSET_CONTAINER``
1968 * ``VFIO_GROUP_GET_DEVICE_FD``
1969 * ``VFIO_IOMMU_GET_INFO``
1970
1971 However, once support for live migration for VFIO devices is finalized some
1972 of the above commands may have to be handled by the client in their
1973 corresponding vfio-user form. This will be addressed in a future protocol
1974 version.
1975
1976 VFIO groups and containers
1977 ^^^^^^^^^^^^^^^^^^^^^^^^^^
1978
1979 The current VFIO implementation includes group and container idioms that
1980 describe how a device relates to the host IOMMU. In the vfio-user
1981 implementation, the IOMMU is implemented in SW by the client, and is not
1982 visible to the server. The simplest idea would be that the client put each
1983 device into its own group and container.
1984
1985 Backend Program Conventions
1986 ---------------------------
1987
1988 vfio-user backend program conventions are based on the vhost-user ones.
1989
1990 * The backend program must not daemonize itself.
1991 * No assumptions must be made as to what access the backend program has on the
1992 system.
1993 * File descriptors 0, 1 and 2 must exist, must have regular
1994 stdin/stdout/stderr semantics, and can be redirected.
1995 * The backend program must honor the SIGTERM signal.
1996 * The backend program must accept the following commands line options:
1997
1998 * ``--socket-path=PATH``: path to UNIX domain socket,
1999 * ``--fd=FDNUM``: file descriptor for UNIX domain socket, incompatible with
2000 ``--socket-path``
2001 * The backend program must be accompanied with a JSON file stored under
2002 ``/usr/share/vfio-user``.
2003
2004 TODO add schema similar to docs/interop/vhost-user.json.