| 1 | ===================== |
| 2 | VFIO device migration |
| 3 | ===================== |
| 4 | |
| 5 | Migration of virtual machine involves saving the state for each device that |
| 6 | the guest is running on source host and restoring this saved state on the |
| 7 | destination host. This document details how saving and restoring of VFIO |
| 8 | devices is done in QEMU. |
| 9 | |
| 10 | Migration of VFIO devices consists of two phases: the optional pre-copy phase, |
| 11 | and the stop-and-copy phase. The pre-copy phase is iterative and allows to |
| 12 | accommodate VFIO devices that have a large amount of data that needs to be |
| 13 | transferred. The iterative pre-copy phase of migration allows for the guest to |
| 14 | continue whilst the VFIO device state is transferred to the destination, this |
| 15 | helps to reduce the total downtime of the VM. VFIO devices opt-in to pre-copy |
| 16 | support by reporting the VFIO_MIGRATION_PRE_COPY flag in the |
| 17 | VFIO_DEVICE_FEATURE_MIGRATION ioctl. |
| 18 | |
| 19 | When pre-copy is supported, it's possible to further reduce downtime by |
| 20 | enabling "switchover-ack" migration capability. |
| 21 | VFIO migration uAPI defines "initial bytes" as part of its pre-copy data stream |
| 22 | and recommends that the initial bytes are sent and loaded in the destination |
| 23 | before stopping the source VM. Enabling this migration capability will |
| 24 | guarantee that and thus, can potentially reduce downtime even further. |
| 25 | |
| 26 | For example, in mlx5 devices, the initial bytes hold metadata used for time |
| 27 | consuming pre-allocations of resources on the destination. Although init bytes |
| 28 | may be small in size and sending them may take little time, loading them in the |
| 29 | destination can take a significant amount of time. Switchover-ack guarantees |
| 30 | that this pre-allocation doesn't happen during downtime. |
| 31 | |
| 32 | Initial bytes was originally defined to be monotonically decreasing, however |
| 33 | there are cases where a new chunk of initial bytes should be transferred during |
| 34 | precopy, e.g., due to a device reconfiguration, etc. The |
| 35 | VFIO_PRECOPY_INFO_REINIT feature addresses this and when supported, allows to |
| 36 | report a new initial bytes value regardless of any previously reported values. |
| 37 | In this case, a new switchover ACK will be requested to make sure the new |
| 38 | initial bytes are loaded in the destination before switching over. |
| 39 | |
| 40 | To support migration of multiple devices that might do P2P transactions between |
| 41 | themselves, VFIO migration uAPI defines an intermediate P2P quiescent state. |
| 42 | While in the P2P quiescent state, P2P DMA transactions cannot be initiated by |
| 43 | the device, but the device can respond to incoming ones. Additionally, all |
| 44 | outstanding P2P transactions are guaranteed to have been completed by the time |
| 45 | the device enters this state. |
| 46 | |
| 47 | All the devices that support P2P migration are first transitioned to the P2P |
| 48 | quiescent state and only then are they stopped or started. This makes migration |
| 49 | safe P2P-wise, since starting and stopping the devices is not done atomically |
| 50 | for all the devices together. |
| 51 | |
| 52 | Thus, multiple VFIO devices migration is allowed only if all the devices |
| 53 | support P2P migration. Single VFIO device migration is allowed regardless of |
| 54 | P2P migration support. |
| 55 | |
| 56 | A detailed description of the UAPI for VFIO device migration can be found in |
| 57 | the comment for the ``vfio_device_mig_state`` structure in the header file |
| 58 | linux-headers/linux/vfio.h. |
| 59 | |
| 60 | VFIO implements the device hooks for the iterative approach as follows: |
| 61 | |
| 62 | * A ``save_setup`` function that sets up migration on the source. |
| 63 | |
| 64 | * A ``load_setup`` function that sets the VFIO device on the destination in |
| 65 | _RESUMING state. |
| 66 | |
| 67 | * A ``save_query_pending`` function that reports the remaining data that |
| 68 | the vendor driver has yet to save for the VFIO device. |
| 69 | |
| 70 | * An ``is_active_iterate`` function that indicates ``save_live_iterate`` is |
| 71 | active only when the VFIO device is in pre-copy states. |
| 72 | |
| 73 | * A ``save_live_iterate`` function that reads the VFIO device's data from the |
| 74 | vendor driver during iterative pre-copy phase. |
| 75 | |
| 76 | * A ``switchover_start`` function that in the multifd mode starts a thread that |
| 77 | reassembles the multifd received data and loads it in-order into the device. |
| 78 | In the non-multifd mode this function is a NOP. |
| 79 | |
| 80 | * A ``save_state`` function to save the device config space if it is present |
| 81 | in the non-multifd mode. |
| 82 | In the multifd mode it just emits either a dummy EOS marker. |
| 83 | |
| 84 | * A ``save_complete`` function that sets the VFIO device in _STOP_COPY |
| 85 | state and iteratively copies the data for the VFIO device until the |
| 86 | vendor driver indicates that no data remains. In the multifd mode it |
| 87 | just emits a dummy EOS marker. |
| 88 | |
| 89 | * A ``save_complete_precopy_thread`` function that in the multifd mode |
| 90 | provides thread handler performing multifd device state transfer. |
| 91 | It sets the VFIO device to _STOP_COPY state, iteratively reads the data |
| 92 | from the VFIO device and queues it for multifd transmission until the vendor |
| 93 | driver indicates that no data remains. |
| 94 | After that, it saves the device config space and queues it for multifd |
| 95 | transfer too. |
| 96 | In the non-multifd mode this thread is a NOP. |
| 97 | |
| 98 | * A ``load_state`` function that loads the config section and the data |
| 99 | sections that are generated by the save functions above. |
| 100 | |
| 101 | * A ``load_state_buffer`` function that loads the device state and the device |
| 102 | config that arrived via multifd channels. |
| 103 | It's used only in the multifd mode. |
| 104 | |
| 105 | * ``cleanup`` functions for both save and load that perform any migration |
| 106 | related cleanup. |
| 107 | |
| 108 | |
| 109 | The VFIO migration code uses a VM state change handler to change the VFIO |
| 110 | device state when the VM state changes from running to not-running, and |
| 111 | vice versa. |
| 112 | |
| 113 | Similarly, a migration state change handler is used to trigger a transition of |
| 114 | the VFIO device state when certain changes of the migration state occur. For |
| 115 | example, the VFIO device state is transitioned back to _RUNNING in case a |
| 116 | migration failed or was canceled. |
| 117 | |
| 118 | System memory dirty pages tracking |
| 119 | ---------------------------------- |
| 120 | |
| 121 | A ``log_global_start`` and ``log_global_stop`` memory listener callback informs |
| 122 | the VFIO dirty tracking module to start and stop dirty page tracking. A |
| 123 | ``log_sync`` memory listener callback queries the dirty page bitmap from the |
| 124 | dirty tracking module and marks system memory pages which were DMA-ed by the |
| 125 | VFIO device as dirty. The dirty page bitmap is queried per container. |
| 126 | |
| 127 | Currently there are two ways dirty page tracking can be done: |
| 128 | (1) Device dirty tracking: |
| 129 | In this method the device is responsible to log and report its DMAs. This |
| 130 | method can be used only if the device is capable of tracking its DMAs. |
| 131 | Discovering device capability, starting and stopping dirty tracking, and |
| 132 | syncing the dirty bitmaps from the device are done using the DMA logging uAPI. |
| 133 | More info about the uAPI can be found in the comments of the |
| 134 | ``vfio_device_feature_dma_logging_control`` and |
| 135 | ``vfio_device_feature_dma_logging_report`` structures in the header file |
| 136 | linux-headers/linux/vfio.h. |
| 137 | |
| 138 | (2) VFIO IOMMU module: |
| 139 | In this method dirty tracking is done by IOMMU. However, there is currently no |
| 140 | IOMMU support for dirty page tracking. For this reason, all pages are |
| 141 | perpetually marked dirty, unless the device driver pins pages through external |
| 142 | APIs in which case only those pinned pages are perpetually marked dirty. |
| 143 | |
| 144 | If the above two methods are not supported, all pages are perpetually marked |
| 145 | dirty by QEMU. |
| 146 | |
| 147 | By default, dirty pages are tracked during pre-copy as well as stop-and-copy |
| 148 | phase. So, a page marked as dirty will be copied to the destination in both |
| 149 | phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it can |
| 150 | achieve its downtime tolerances. If QEMU during pre-copy phase keeps finding |
| 151 | dirty pages continuously, then it understands that even in stop-and-copy phase, |
| 152 | it is likely to find dirty pages and can predict the downtime accordingly. |
| 153 | |
| 154 | QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking`` |
| 155 | which disables querying the dirty bitmap during pre-copy phase. If it is set to |
| 156 | off, all dirty pages will be copied to the destination in stop-and-copy phase |
| 157 | only. |
| 158 | |
| 159 | System memory dirty pages tracking when vIOMMU is enabled |
| 160 | --------------------------------------------------------- |
| 161 | |
| 162 | With vIOMMU, an IO virtual address range can get unmapped while in pre-copy |
| 163 | phase of migration. In that case, the unmap ioctl returns any dirty pages in |
| 164 | that range and QEMU reports corresponding guest physical pages dirty. During |
| 165 | stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped |
| 166 | pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those |
| 167 | mapped ranges. If device dirty tracking is enabled with vIOMMU, live migration |
| 168 | will be blocked. |
| 169 | |
| 170 | Flow of state changes during Live migration |
| 171 | =========================================== |
| 172 | |
| 173 | Below is the state change flow during live migration for a VFIO device that |
| 174 | supports both precopy and P2P migration. The flow for devices that don't |
| 175 | support it is similar, except that the relevant states for precopy and P2P are |
| 176 | skipped. |
| 177 | The values in the parentheses represent the VM state, the migration state, and |
| 178 | the VFIO device state, respectively. |
| 179 | |
| 180 | Live migration save path |
| 181 | ------------------------ |
| 182 | |
| 183 | :: |
| 184 | |
| 185 | QEMU normal running state |
| 186 | (RUNNING, _NONE, _RUNNING) |
| 187 | | |
| 188 | migrate_init spawns migration_thread |
| 189 | Migration thread then calls each device's .save_setup() |
| 190 | (RUNNING, _SETUP, _PRE_COPY) |
| 191 | | |
| 192 | (RUNNING, _ACTIVE, _PRE_COPY) |
| 193 | If device is active, get pending_bytes by .state_pending_{estimate,exact}() |
| 194 | If total pending_bytes >= threshold_size, call .save_live_iterate() |
| 195 | Data of VFIO device for pre-copy phase is copied |
| 196 | Iterate till total pending bytes converge and are less than threshold |
| 197 | | |
| 198 | On migration completion, the vCPUs and the VFIO device are stopped |
| 199 | The VFIO device is first put in P2P quiescent state |
| 200 | (FINISH_MIGRATE, _ACTIVE, _PRE_COPY_P2P) |
| 201 | | |
| 202 | Then the VFIO device is put in _STOP_COPY state |
| 203 | (FINISH_MIGRATE, _ACTIVE, _STOP_COPY) |
| 204 | .save_complete() is called for each active device |
| 205 | For the VFIO device: in the non-multifd mode iterate in |
| 206 | .save_complete() until |
| 207 | pending data is 0 |
| 208 | In the multifd mode this iteration is done in |
| 209 | .save_complete_precopy_thread() instead. |
| 210 | | |
| 211 | (POSTMIGRATE, _COMPLETED, _STOP_COPY) |
| 212 | Migraton thread schedules cleanup bottom half and exits |
| 213 | | |
| 214 | .save_cleanup() is called |
| 215 | (POSTMIGRATE, _COMPLETED, _STOP) |
| 216 | |
| 217 | Live migration resume path |
| 218 | -------------------------- |
| 219 | |
| 220 | :: |
| 221 | |
| 222 | Incoming migration calls .load_setup() for each device |
| 223 | (RESTORE_VM, _ACTIVE, _STOP) |
| 224 | | |
| 225 | For each device, .load_state() is called for that device section data |
| 226 | transmitted via the main migration channel. |
| 227 | For data transmitted via multifd channels .load_state_buffer() is called |
| 228 | instead. |
| 229 | (RESTORE_VM, _ACTIVE, _RESUMING) |
| 230 | | |
| 231 | At the end, .load_cleanup() is called for each device and vCPUs are started |
| 232 | The VFIO device is first put in P2P quiescent state |
| 233 | (RUNNING, _ACTIVE, _RUNNING_P2P) |
| 234 | | |
| 235 | (RUNNING, _NONE, _RUNNING) |
| 236 | |
| 237 | Postcopy |
| 238 | ======== |
| 239 | |
| 240 | Postcopy migration is currently not supported for VFIO devices. |
| 241 | |
| 242 | Multifd |
| 243 | ======= |
| 244 | |
| 245 | Starting from QEMU version 10.0 there's a possibility to transfer VFIO device |
| 246 | _STOP_COPY state via multifd channels. This helps reduce downtime - especially |
| 247 | with multiple VFIO devices or with devices having a large migration state. |
| 248 | As an additional benefit, setting the VFIO device to _STOP_COPY state and |
| 249 | saving its config space is also parallelized (run in a separate thread) in |
| 250 | such migration mode. |
| 251 | |
| 252 | The multifd VFIO device state transfer is controlled by |
| 253 | "x-migration-multifd-transfer" VFIO device property. This property defaults to |
| 254 | AUTO, which means that VFIO device state transfer via multifd channels is |
| 255 | attempted in configurations that otherwise support it. |
| 256 | |
| 257 | Since the target QEMU needs to load device state buffers in-order it needs to |
| 258 | queue incoming buffers until they can be loaded into the device. |
| 259 | This means that a malicious QEMU source could theoretically cause the target |
| 260 | QEMU to allocate unlimited amounts of memory for such buffers-in-flight. |
| 261 | |
| 262 | The "x-migration-max-queued-buffers-size" property allows capping the total size |
| 263 | of these VFIO device state buffers queued at the destination. |
| 264 | |
| 265 | Because a malicious QEMU source causing OOM on the target is not expected to be |
| 266 | a realistic threat in most of VFIO live migration use cases and the right value |
| 267 | depends on the particular setup by default this queued buffers size limit is |
| 268 | disabled by setting it to UINT64_MAX. |
| 269 | |
| 270 | Some host platforms (like ARM64) require that VFIO device config is loaded only |
| 271 | after all iterables were loaded, during non-iterables loading phase. |
| 272 | Such interlocking is controlled by "x-migration-load-config-after-iter" VFIO |
| 273 | device property, which in its default setting (AUTO) does so only on platforms |
| 274 | that actually require it. |