| 1 | .. _migration: |
| 2 | |
| 3 | =================== |
| 4 | Migration framework |
| 5 | =================== |
| 6 | |
| 7 | QEMU has code to load/save the state of the guest that it is running. |
| 8 | These are two complementary operations. Saving the state just does |
| 9 | that, saves the state for each device that the guest is running. |
| 10 | Restoring a guest is just the opposite operation: we need to load the |
| 11 | state of each device. |
| 12 | |
| 13 | For this to work, QEMU has to be launched with the same arguments the |
| 14 | two times. I.e. it can only restore the state in one guest that has |
| 15 | the same devices that the one it was saved (this last requirement can |
| 16 | be relaxed a bit, but for now we can consider that configuration has |
| 17 | to be exactly the same). |
| 18 | |
| 19 | Once that we are able to save/restore a guest, a new functionality is |
| 20 | requested: migration. This means that QEMU is able to start in one |
| 21 | machine and being "migrated" to another machine. I.e. being moved to |
| 22 | another machine. |
| 23 | |
| 24 | Next was the "live migration" functionality. This is important |
| 25 | because some guests run with a lot of state (specially RAM), and it |
| 26 | can take a while to move all state from one machine to another. Live |
| 27 | migration allows the guest to continue running while the state is |
| 28 | transferred. Only while the last part of the state is transferred has |
| 29 | the guest to be stopped. Typically the time that the guest is |
| 30 | unresponsive during live migration is the low hundred of milliseconds |
| 31 | (notice that this depends on a lot of things). |
| 32 | |
| 33 | .. contents:: |
| 34 | |
| 35 | Transports |
| 36 | ========== |
| 37 | |
| 38 | The migration stream is normally just a byte stream that can be passed |
| 39 | over any transport. |
| 40 | |
| 41 | - tcp migration: do the migration using tcp sockets |
| 42 | - unix migration: do the migration using unix sockets |
| 43 | - exec migration: do the migration using the stdin/stdout through a process. |
| 44 | - fd migration: do the migration using a file descriptor that is |
| 45 | passed to QEMU. QEMU doesn't care how this file descriptor is opened. |
| 46 | - file migration: do the migration using a file that is passed to QEMU |
| 47 | by path. A file offset option is supported to allow a management |
| 48 | application to add its own metadata to the start of the file without |
| 49 | QEMU interference. Note that QEMU does not flush cached file |
| 50 | data/metadata at the end of migration. |
| 51 | |
| 52 | The file migration also supports using a file that has already been |
| 53 | opened. A set of file descriptors is passed to QEMU via an "fdset" |
| 54 | (see add-fd QMP command documentation). This method allows a |
| 55 | management application to have control over the migration file |
| 56 | opening operation. There are, however, strict requirements to this |
| 57 | interface if the multifd capability is enabled: |
| 58 | |
| 59 | - the fdset must contain two file descriptors that are not |
| 60 | duplicates between themselves; |
| 61 | - if the direct-io capability is to be used, exactly one of the |
| 62 | file descriptors must have the O_DIRECT flag set; |
| 63 | - the file must be opened with WRONLY on the migration source side |
| 64 | and RDONLY on the migration destination side. |
| 65 | |
| 66 | - rdma migration: support is included for migration using RDMA, which |
| 67 | transports the page data using ``RDMA``, where the hardware takes |
| 68 | care of transporting the pages, and the load on the CPU is much |
| 69 | lower. While the internals of RDMA migration are a bit different, |
| 70 | this isn't really visible outside the RAM migration code. |
| 71 | |
| 72 | All these migration protocols use the same infrastructure to |
| 73 | save/restore state devices. This infrastructure is shared with the |
| 74 | savevm/loadvm functionality. |
| 75 | |
| 76 | Common infrastructure |
| 77 | ===================== |
| 78 | |
| 79 | The files, sockets or fd's that carry the migration stream are abstracted by |
| 80 | the ``QEMUFile`` type (see ``migration/qemu-file.h``). In most cases this |
| 81 | is connected to a subtype of ``QIOChannel`` (see ``io/``). |
| 82 | |
| 83 | |
| 84 | Saving the state of one device |
| 85 | ============================== |
| 86 | |
| 87 | For most devices, the state is saved in a single call to the migration |
| 88 | infrastructure; these are *non-iterative* devices. The data for these |
| 89 | devices is sent at the end of precopy migration, when the CPUs are paused. |
| 90 | There are also *iterative* devices, which contain a very large amount of |
| 91 | data (e.g. RAM or large tables). See the iterative device section below. |
| 92 | |
| 93 | General advice for device developers |
| 94 | ------------------------------------ |
| 95 | |
| 96 | - The migration state saved should reflect the device being modelled rather |
| 97 | than the way your implementation works. That way if you change the implementation |
| 98 | later the migration stream will stay compatible. That model may include |
| 99 | internal state that's not directly visible in a register. |
| 100 | |
| 101 | - When saving a migration stream the device code may walk and check |
| 102 | the state of the device. These checks might fail in various ways (e.g. |
| 103 | discovering internal state is corrupt or that the guest has done something bad). |
| 104 | Consider carefully before asserting/aborting at this point, since the |
| 105 | normal response from users is that *migration broke their VM* since it had |
| 106 | apparently been running fine until then. In these error cases, the device |
| 107 | should log a message indicating the cause of error, and should consider |
| 108 | putting the device into an error state, allowing the rest of the VM to |
| 109 | continue execution. |
| 110 | |
| 111 | - The migration might happen at an inconvenient point, |
| 112 | e.g. right in the middle of the guest reprogramming the device, during |
| 113 | guest reboot or shutdown or while the device is waiting for external IO. |
| 114 | It's strongly preferred that migrations do not fail in this situation, |
| 115 | since in the cloud environment migrations might happen automatically to |
| 116 | VMs that the administrator doesn't directly control. |
| 117 | |
| 118 | - If you do need to fail a migration, ensure that sufficient information |
| 119 | is logged to identify what went wrong. |
| 120 | |
| 121 | - The destination should treat an incoming migration stream as hostile |
| 122 | (which we do to varying degrees in the existing code). Check that offsets |
| 123 | into buffers and the like can't cause overruns. Fail the incoming migration |
| 124 | in the case of a corrupted stream like this. |
| 125 | |
| 126 | - Take care with internal device state or behaviour that might become |
| 127 | migration version dependent. For example, the order of PCI capabilities |
| 128 | is required to stay constant across migration. Another example would |
| 129 | be that a special case handled by subsections (see below) might become |
| 130 | much more common if a default behaviour is changed. |
| 131 | |
| 132 | - The state of the source should not be changed or destroyed by the |
| 133 | outgoing migration. Migrations timing out or being failed by |
| 134 | higher levels of management, or failures of the destination host are |
| 135 | not unusual, and in that case the VM is restarted on the source. |
| 136 | Note that the management layer can validly revert the migration |
| 137 | even though the QEMU level of migration has succeeded as long as it |
| 138 | does it before starting execution on the destination. |
| 139 | |
| 140 | - Buses and devices should be able to explicitly specify addresses when |
| 141 | instantiated, and management tools should use those. For example, |
| 142 | when hot adding USB devices it's important to specify the ports |
| 143 | and addresses, since implicit ordering based on the command line order |
| 144 | may be different on the destination. This can result in the |
| 145 | device state being loaded into the wrong device. |
| 146 | |
| 147 | VMState |
| 148 | ------- |
| 149 | |
| 150 | Most device data can be described using the ``VMSTATE`` macros (mostly defined |
| 151 | in ``include/migration/vmstate.h``). |
| 152 | |
| 153 | An example (from hw/input/pckbd.c) |
| 154 | |
| 155 | .. code:: c |
| 156 | |
| 157 | static const VMStateDescription vmstate_kbd = { |
| 158 | .name = "pckbd", |
| 159 | .version_id = 3, |
| 160 | .minimum_version_id = 3, |
| 161 | .fields = (const VMStateField[]) { |
| 162 | VMSTATE_UINT8(write_cmd, KBDState), |
| 163 | VMSTATE_UINT8(status, KBDState), |
| 164 | VMSTATE_UINT8(mode, KBDState), |
| 165 | VMSTATE_UINT8(pending, KBDState), |
| 166 | VMSTATE_END_OF_LIST() |
| 167 | } |
| 168 | }; |
| 169 | |
| 170 | We are declaring the state with name "pckbd". The ``version_id`` is |
| 171 | 3, and there are 4 uint8_t fields in the KBDState structure. We |
| 172 | registered this ``VMSTATEDescription`` with one of the following |
| 173 | functions. The first one will generate a device ``instance_id`` |
| 174 | different for each registration. Use the second one if you already |
| 175 | have an id that is different for each instance of the device: |
| 176 | |
| 177 | .. code:: c |
| 178 | |
| 179 | vmstate_register_any(NULL, &vmstate_kbd, s); |
| 180 | vmstate_register(NULL, instance_id, &vmstate_kbd, s); |
| 181 | |
| 182 | For devices that are ``qdev`` based, we can register the device in the class |
| 183 | init function: |
| 184 | |
| 185 | .. code:: c |
| 186 | |
| 187 | dc->vmsd = &vmstate_kbd_isa; |
| 188 | |
| 189 | The VMState macros take care of ensuring that the device data section |
| 190 | is formatted portably (normally big endian) and make some compile time checks |
| 191 | against the types of the fields in the structures. |
| 192 | |
| 193 | VMState macros can include other VMStateDescriptions to store substructures |
| 194 | (see ``VMSTATE_STRUCT_``), arrays (``VMSTATE_ARRAY_``) and variable length |
| 195 | arrays (``VMSTATE_VARRAY_``). Various other macros exist for special |
| 196 | cases. |
| 197 | |
| 198 | Note that the format on the wire is still very raw; i.e. a VMSTATE_UINT32 |
| 199 | ends up with a 4 byte bigendian representation on the wire; in the future |
| 200 | it might be possible to use a more structured format. |
| 201 | |
| 202 | Legacy way |
| 203 | ---------- |
| 204 | |
| 205 | This way is going to disappear as soon as all current users are ported to VMSTATE; |
| 206 | although converting existing code can be tricky, and thus 'soon' is relative. |
| 207 | |
| 208 | Each device has to register two functions, one to save the state and |
| 209 | another to load the state back. |
| 210 | |
| 211 | .. code:: c |
| 212 | |
| 213 | int register_savevm_live(const char *idstr, |
| 214 | int instance_id, |
| 215 | int version_id, |
| 216 | SaveVMHandlers *ops, |
| 217 | void *opaque); |
| 218 | |
| 219 | Two functions in the ``ops`` structure are the ``save_state`` |
| 220 | and ``load_state`` functions. Notice that ``load_state`` receives a version_id |
| 221 | parameter to know what state format is receiving. ``save_state`` doesn't |
| 222 | have a version_id parameter because it always uses the latest version. |
| 223 | |
| 224 | Note that because the VMState macros still save the data in a raw |
| 225 | format, in many cases it's possible to replace legacy code |
| 226 | with a carefully constructed VMState description that matches the |
| 227 | byte layout of the existing code. |
| 228 | |
| 229 | Changing migration data structures |
| 230 | ---------------------------------- |
| 231 | |
| 232 | When we migrate a device, we save/load the state as a series |
| 233 | of fields. Sometimes, due to bugs or new functionality, we need to |
| 234 | change the state to store more/different information. Changing the migration |
| 235 | state saved for a device can break migration compatibility unless |
| 236 | care is taken to use the appropriate techniques. In general QEMU tries |
| 237 | to maintain forward migration compatibility (i.e. migrating from |
| 238 | QEMU n->n+1) and there are users who benefit from backward compatibility |
| 239 | as well. |
| 240 | |
| 241 | Subsections |
| 242 | ----------- |
| 243 | |
| 244 | The most common structure change is adding new data, e.g. when adding |
| 245 | a newer form of device, or adding that state that you previously |
| 246 | forgot to migrate. This is best solved using a subsection. |
| 247 | |
| 248 | A subsection is "like" a device vmstate, but with a particularity, it |
| 249 | has a Boolean function that tells if that values are needed to be sent |
| 250 | or not. If this functions returns false, the subsection is not sent. |
| 251 | Subsections have a unique name, that is looked for on the receiving |
| 252 | side. |
| 253 | |
| 254 | On the receiving side, if we found a subsection for a device that we |
| 255 | don't understand, we just fail the migration. If we understand all |
| 256 | the subsections, then we load the state with success. There's no check |
| 257 | that a subsection is loaded, so a newer QEMU that knows about a subsection |
| 258 | can (with care) load a stream from an older QEMU that didn't send |
| 259 | the subsection. |
| 260 | |
| 261 | If the new data is only needed in a rare case, then the subsection |
| 262 | can be made conditional on that case and the migration will still |
| 263 | succeed to older QEMUs in most cases. This is OK for data that's |
| 264 | critical, but in some use cases it's preferred that the migration |
| 265 | should succeed even with the data missing. To support this the |
| 266 | subsection can be connected to a device property and from there |
| 267 | to a versioned machine type. |
| 268 | |
| 269 | The 'pre_load' and 'post_load' functions on subsections are only |
| 270 | called if the subsection is loaded. |
| 271 | |
| 272 | One important note is that the outer post_load() function is called "after" |
| 273 | loading all subsections, because a newer subsection could change the same |
| 274 | value that it uses. A flag, and the combination of outer pre_load and |
| 275 | post_load can be used to detect whether a subsection was loaded, and to |
| 276 | fall back on default behaviour when the subsection isn't present. |
| 277 | |
| 278 | Example: |
| 279 | |
| 280 | .. code:: c |
| 281 | |
| 282 | static bool ide_drive_pio_state_needed(void *opaque) |
| 283 | { |
| 284 | IDEState *s = opaque; |
| 285 | |
| 286 | return ((s->status & DRQ_STAT) != 0) |
| 287 | || (s->bus->error_status & BM_STATUS_PIO_RETRY); |
| 288 | } |
| 289 | |
| 290 | const VMStateDescription vmstate_ide_drive_pio_state = { |
| 291 | .name = "ide_drive/pio_state", |
| 292 | .version_id = 1, |
| 293 | .minimum_version_id = 1, |
| 294 | .pre_save = ide_drive_pio_pre_save, |
| 295 | .post_load = ide_drive_pio_post_load, |
| 296 | .needed = ide_drive_pio_state_needed, |
| 297 | .fields = (const VMStateField[]) { |
| 298 | VMSTATE_INT32(req_nb_sectors, IDEState), |
| 299 | VMSTATE_VARRAY_INT32(io_buffer, IDEState, io_buffer_total_len, 1, |
| 300 | vmstate_info_uint8, uint8_t), |
| 301 | VMSTATE_INT32(cur_io_buffer_offset, IDEState), |
| 302 | VMSTATE_INT32(cur_io_buffer_len, IDEState), |
| 303 | VMSTATE_UINT8(end_transfer_fn_idx, IDEState), |
| 304 | VMSTATE_INT32(elementary_transfer_size, IDEState), |
| 305 | VMSTATE_INT32(packet_transfer_size, IDEState), |
| 306 | VMSTATE_END_OF_LIST() |
| 307 | } |
| 308 | }; |
| 309 | |
| 310 | const VMStateDescription vmstate_ide_drive = { |
| 311 | .name = "ide_drive", |
| 312 | .version_id = 3, |
| 313 | .minimum_version_id = 0, |
| 314 | .post_load = ide_drive_post_load, |
| 315 | .fields = (const VMStateField[]) { |
| 316 | .... several fields .... |
| 317 | VMSTATE_END_OF_LIST() |
| 318 | }, |
| 319 | .subsections = (const VMStateDescription * const []) { |
| 320 | &vmstate_ide_drive_pio_state, |
| 321 | NULL |
| 322 | } |
| 323 | }; |
| 324 | |
| 325 | Here we have a subsection for the pio state. We only need to |
| 326 | save/send this state when we are in the middle of a pio operation |
| 327 | (that is what ``ide_drive_pio_state_needed()`` checks). If DRQ_STAT is |
| 328 | not enabled, the values on that fields are garbage and don't need to |
| 329 | be sent. |
| 330 | |
| 331 | Connecting subsections to properties |
| 332 | ------------------------------------ |
| 333 | |
| 334 | Using a condition function that checks a 'property' to determine whether |
| 335 | to send a subsection allows backward migration compatibility when |
| 336 | new subsections are added, especially when combined with versioned |
| 337 | machine types. |
| 338 | |
| 339 | For example: |
| 340 | |
| 341 | a) Add a new property using ``DEFINE_PROP_BOOL`` - e.g. support-foo and |
| 342 | default it to true. |
| 343 | b) Add an entry to the ``hw_compat_`` for the previous version that sets |
| 344 | the property to false. |
| 345 | c) Add a static bool support_foo function that tests the property. |
| 346 | d) Add a subsection with a .needed set to the support_foo function |
| 347 | e) (potentially) Add an outer pre_load that sets up a default value |
| 348 | for 'foo' to be used if the subsection isn't loaded. |
| 349 | |
| 350 | Now that subsection will not be generated when using an older |
| 351 | machine type and the migration stream will be accepted by older |
| 352 | QEMU versions. |
| 353 | |
| 354 | Not sending existing elements |
| 355 | ----------------------------- |
| 356 | |
| 357 | Sometimes members of the VMState are no longer needed: |
| 358 | |
| 359 | - removing them will break migration compatibility |
| 360 | |
| 361 | - making them version dependent and bumping the version will break backward migration |
| 362 | compatibility. |
| 363 | |
| 364 | Adding a dummy field into the migration stream is normally the best way to preserve |
| 365 | compatibility. |
| 366 | |
| 367 | If the field really does need to be removed then: |
| 368 | |
| 369 | a) Add a new property/compatibility/function in the same way for subsections above. |
| 370 | b) replace the VMSTATE macro with the _TEST version of the macro, e.g.: |
| 371 | |
| 372 | ``VMSTATE_UINT32(foo, barstruct)`` |
| 373 | |
| 374 | becomes |
| 375 | |
| 376 | ``VMSTATE_UINT32_TEST(foo, barstruct, pre_version_baz)`` |
| 377 | |
| 378 | Sometime in the future when we no longer care about the ancient versions these can be killed off. |
| 379 | Note that for backward compatibility it's important to fill in the structure with |
| 380 | data that the destination will understand. |
| 381 | |
| 382 | Any difference in the predicates on the source and destination will end up |
| 383 | with different fields being enabled and data being loaded into the wrong |
| 384 | fields; for this reason conditional fields like this are very fragile. |
| 385 | |
| 386 | Versions |
| 387 | -------- |
| 388 | |
| 389 | Version numbers are intended for major incompatible changes to the |
| 390 | migration of a device, and using them breaks backward-migration |
| 391 | compatibility; in general most changes can be made by adding Subsections |
| 392 | (see above) or _TEST macros (see above) which won't break compatibility. |
| 393 | |
| 394 | Each version is associated with a series of fields saved. The ``save_state`` always saves |
| 395 | the state as the newer version. But ``load_state`` sometimes is able to |
| 396 | load state from an older version. |
| 397 | |
| 398 | You can see that there are two version fields: |
| 399 | |
| 400 | - ``version_id``: the maximum version_id supported by VMState for that device. |
| 401 | - ``minimum_version_id``: the minimum version_id that VMState is able to understand |
| 402 | for that device. |
| 403 | |
| 404 | VMState is able to read versions from minimum_version_id to version_id. |
| 405 | |
| 406 | There are *_V* forms of many ``VMSTATE_`` macros to load fields for version dependent fields, |
| 407 | e.g. |
| 408 | |
| 409 | .. code:: c |
| 410 | |
| 411 | VMSTATE_UINT16_V(ip_id, Slirp, 2), |
| 412 | |
| 413 | only loads that field for versions 2 and newer. |
| 414 | |
| 415 | Saving state will always create a section with the 'version_id' value |
| 416 | and thus can't be loaded by any older QEMU. |
| 417 | |
| 418 | Massaging functions |
| 419 | ------------------- |
| 420 | |
| 421 | Sometimes, it is not enough to be able to save the state directly |
| 422 | from one structure, we need to fill the correct values there. One |
| 423 | example is when we are using kvm. Before saving the cpu state, we |
| 424 | need to ask kvm to copy to QEMU the state that it is using. And the |
| 425 | opposite when we are loading the state, we need a way to tell kvm to |
| 426 | load the state for the cpu that we have just loaded from the QEMUFile. |
| 427 | |
| 428 | The functions to do that are inside a vmstate definition, and are called: |
| 429 | |
| 430 | - ``int (*pre_load)(void *opaque);`` |
| 431 | |
| 432 | This function is called before we load the state of one device. |
| 433 | |
| 434 | - ``int (*post_load)(void *opaque, int version_id);`` |
| 435 | |
| 436 | This function is called after we load the state of one device. |
| 437 | |
| 438 | - ``int (*pre_save)(void *opaque);`` |
| 439 | |
| 440 | This function is called before we save the state of one device. |
| 441 | |
| 442 | - ``void (*post_save)(void *opaque);`` |
| 443 | |
| 444 | This function is called after we save the state of one device |
| 445 | (even upon failure, unless the call to pre_save returned an error). |
| 446 | |
| 447 | Following are the errp variants of these functions. |
| 448 | |
| 449 | - ``bool (*pre_load_errp)(void *opaque, Error **errp);`` |
| 450 | |
| 451 | This function is called before we load the state of one device. |
| 452 | |
| 453 | - ``bool (*post_load_errp)(void *opaque, int version_id, Error **errp);`` |
| 454 | |
| 455 | This function is called after we load the state of one device. |
| 456 | |
| 457 | - ``bool (*pre_save_errp)(void *opaque, Error **errp);`` |
| 458 | |
| 459 | This function is called before we save the state of one device. |
| 460 | |
| 461 | New impls should preferentally use 'errp' variants of these |
| 462 | methods and existing impls incrementally converted. |
| 463 | The variants without 'errp' are intended to be removed |
| 464 | once all usage is converted. |
| 465 | |
| 466 | Example: You can look at hpet.c, that uses the first three functions |
| 467 | to massage the state that is transferred. |
| 468 | |
| 469 | The ``VMSTATE_WITH_TMP`` macro may be useful when the migration |
| 470 | data doesn't match the stored device data well; it allows an |
| 471 | intermediate temporary structure to be populated with migration |
| 472 | data and then transferred to the main structure. |
| 473 | |
| 474 | If you use memory or portio_list API functions that update memory layout outside |
| 475 | initialization (i.e., in response to a guest action), this is a strong |
| 476 | indication that you need to call these functions in a ``post_load`` callback. |
| 477 | Examples of such API functions are: |
| 478 | |
| 479 | - memory_region_add_subregion() |
| 480 | - memory_region_del_subregion() |
| 481 | - memory_region_set_readonly() |
| 482 | - memory_region_set_nonvolatile() |
| 483 | - memory_region_set_enabled() |
| 484 | - memory_region_set_address() |
| 485 | - memory_region_set_alias_offset() |
| 486 | - portio_list_set_address() |
| 487 | - portio_list_set_enabled() |
| 488 | |
| 489 | Since the order of device save/restore is not defined, you must |
| 490 | avoid accessing or changing any other device's state in one of these |
| 491 | callbacks. (For instance, don't do anything that calls ``update_irq()`` |
| 492 | in a ``post_load`` hook.) Otherwise, restore will not be deterministic, |
| 493 | and this will break execution record/replay. |
| 494 | |
| 495 | Iterative device migration |
| 496 | -------------------------- |
| 497 | |
| 498 | Some devices, such as RAM or certain platform devices, |
| 499 | have large amounts of data that would mean that the CPUs would be |
| 500 | paused for too long if they were sent in one section. For these |
| 501 | devices an *iterative* approach is taken. |
| 502 | |
| 503 | The iterative devices generally don't use VMState macros |
| 504 | (although it may be possible in some cases) and instead use |
| 505 | qemu_put_*/qemu_get_* macros to read/write data to the stream. Specialist |
| 506 | versions exist for high bandwidth IO. |
| 507 | |
| 508 | |
| 509 | An iterative device must provide: |
| 510 | |
| 511 | - A ``save_setup`` function that initialises the data structures and |
| 512 | transmits a first section containing information on the device. In the |
| 513 | case of RAM this transmits a list of RAMBlocks and sizes. |
| 514 | |
| 515 | - A ``load_setup`` function that initialises the data structures on the |
| 516 | destination. |
| 517 | |
| 518 | - A ``save_query_pending`` function that indicates how much more |
| 519 | data we must save. |
| 520 | |
| 521 | - A ``save_live_iterate`` function should send a chunk of data until |
| 522 | the point that stream bandwidth limits tell it to stop. Each call |
| 523 | generates one section. |
| 524 | |
| 525 | - A ``save_complete`` function that must transmit the last section for |
| 526 | the device containing any remaining data. |
| 527 | |
| 528 | - A ``load_state`` function used to load sections generated by |
| 529 | any of the save functions that generate sections. |
| 530 | |
| 531 | - ``cleanup`` functions for both save and load that are called |
| 532 | at the end of migration. |
| 533 | |
| 534 | Note that the contents of the sections for iterative migration tend |
| 535 | to be open-coded by the devices; care should be taken in parsing |
| 536 | the results and structuring the stream to make them easy to validate. |
| 537 | |
| 538 | Device ordering |
| 539 | --------------- |
| 540 | |
| 541 | There are cases in which the ordering of device loading matters; for |
| 542 | example in some systems where a device may assert an interrupt during loading, |
| 543 | if the interrupt controller is loaded later then it might lose the state. |
| 544 | |
| 545 | Some ordering is implicitly provided by the order in which the machine |
| 546 | definition creates devices, however this is somewhat fragile. |
| 547 | |
| 548 | The ``MigrationPriority`` enum provides a means of explicitly enforcing |
| 549 | ordering. Numerically higher priorities are loaded earlier. |
| 550 | The priority is set by setting the ``priority`` field of the top level |
| 551 | ``VMStateDescription`` for the device. |
| 552 | |
| 553 | Stream structure |
| 554 | ================ |
| 555 | |
| 556 | The stream tries to be word and endian agnostic, allowing migration between hosts |
| 557 | of different characteristics running the same VM. |
| 558 | |
| 559 | - Header |
| 560 | |
| 561 | - Magic |
| 562 | - Version |
| 563 | - VM configuration section |
| 564 | |
| 565 | - Machine type |
| 566 | - Target page bits |
| 567 | - List of sections |
| 568 | Each section contains a device, or one iteration of a device save. |
| 569 | |
| 570 | - section type |
| 571 | - section id |
| 572 | - ID string (First section of each device) |
| 573 | - instance id (First section of each device) |
| 574 | - version id (First section of each device) |
| 575 | - <device data> |
| 576 | - Footer mark |
| 577 | - EOF mark |
| 578 | - VM Description structure |
| 579 | Consisting of a JSON description of the contents for analysis only |
| 580 | |
| 581 | The ``device data`` in each section consists of the data produced |
| 582 | by the code described above. For non-iterative devices they have a single |
| 583 | section; iterative devices have an initial and last section and a set |
| 584 | of parts in between. |
| 585 | Note that there is very little checking by the common code of the integrity |
| 586 | of the ``device data`` contents, that's up to the devices themselves. |
| 587 | The ``footer mark`` provides a little bit of protection for the case where |
| 588 | the receiving side reads more or less data than expected. |
| 589 | |
| 590 | The ``ID string`` is normally unique, having been formed from a bus name |
| 591 | and device address, PCI devices and storage devices hung off PCI controllers |
| 592 | fit this pattern well. Some devices are fixed single instances (e.g. "pc-ram"). |
| 593 | Others (especially either older devices or system devices which for |
| 594 | some reason don't have a bus concept) make use of the ``instance id`` |
| 595 | for otherwise identically named devices. |
| 596 | |
| 597 | Return path |
| 598 | ----------- |
| 599 | |
| 600 | Only a unidirectional stream is required for normal migration, however a |
| 601 | ``return path`` can be created when bidirectional communication is desired. |
| 602 | This is primarily used by postcopy, but is also used to return a success |
| 603 | flag to the source at the end of migration. |
| 604 | |
| 605 | ``qemu_file_get_return_path(QEMUFile* fwdpath)`` gives the QEMUFile* for the return |
| 606 | path. |
| 607 | |
| 608 | Source side |
| 609 | |
| 610 | Forward path - written by migration thread |
| 611 | Return path - opened by main thread, read by return-path thread |
| 612 | |
| 613 | Destination side |
| 614 | |
| 615 | Forward path - read by main thread |
| 616 | Return path - opened by main thread, written by main thread AND postcopy |
| 617 | thread (protected by rp_mutex) |
| 618 |