| 1 | ================== |
| 2 | Fast Snapshot Load |
| 3 | ================== |
| 4 | |
| 5 | Overview |
| 6 | ======== |
| 7 | Fast snapshot load is an extension of the postcopy migration feature |
| 8 | to disk loads. |
| 9 | |
| 10 | Unlike a usual snapshot load, which requires all VM data (RAM as well |
| 11 | as device states) to be loaded into host RAM from the snapshot file |
| 12 | for the guest to run, fast snapshot load uses postcopy infrastructure |
| 13 | to load in only the required device states and allows RAM pages to |
| 14 | be loaded after the guest starts execution. The idea is to start the |
| 15 | guest and serve its page faults on the go, reducing the perceived |
| 16 | resume time for large snapshots. |
| 17 | |
| 18 | Architecture |
| 19 | ============ |
| 20 | This feature combines postcopy migration and mapped-ram capabilities |
| 21 | to load RAM pages on demand. It is done by catching guest faults using |
| 22 | Linux ``userfaultfd`` and loading the page by calculating the offset |
| 23 | of its location in the snapshot file using mapped-ram capabilities. |
| 24 | |
| 25 | Fault Thread |
| 26 | ------------ |
| 27 | The fault thread uses Linux ``userfaultfd`` to catch page faults caused |
| 28 | by guest and directly load the page from the snapshot file. It is |
| 29 | very similar to network postcopy fault thread, with primary difference |
| 30 | being it loads pages directly by reading from the snapshot file. |
| 31 | |
| 32 | Eager Thread |
| 33 | ------------ |
| 34 | Eager thread iterates over all pages in RAM and loads each page not |
| 35 | yet loaded by fault thread. It is required as unlike network postcopy |
| 36 | where majority of RAM has already been loaded via precopy, here entire |
| 37 | RAM is waiting to be loaded. If there is no eager loading thread each |
| 38 | page will only be loaded when it is required by guest. In case there |
| 39 | are some background pages that are never/rarely accessed by guest, |
| 40 | the system will be locked in migration state indefinitely. |
| 41 | |
| 42 | Synchronization |
| 43 | --------------- |
| 44 | In order to make sure both of these threads do not load the same page |
| 45 | twice potentially overwriting and corrupting user RAM, a bitmap is |
| 46 | used (``RAMBlock->pending_bmap``) which tracks the pages claimed to |
| 47 | be loaded by threads. This prevents race condition when one thread |
| 48 | is loading the page and other one tries to do the same. |
| 49 | |
| 50 | Usage |
| 51 | ===== |
| 52 | |
| 53 | Simply enable ``mapped-ram`` and ``postcopy-ram`` capabilities on |
| 54 | the destination: |
| 55 | |
| 56 | .. code-block:: text |
| 57 | |
| 58 | migrate_set_capability mapped-ram on |
| 59 | migrate_set_capability postcopy-ram on |
| 60 | |
| 61 | Use a ``file:`` URI for migration: |
| 62 | |
| 63 | .. code-block:: text |
| 64 | |
| 65 | migrate_incoming file:/path/to/snapshot/file |
| 66 | |
| 67 | Limitations |
| 68 | =========== |
| 69 | |
| 70 | - Multifd |
| 71 | Fast snapshot load is currently incompatible with ``multifd`` |
| 72 | capability. While ``mapped-ram`` allows for parallel disk I/O, |
| 73 | coupling it with ``postcopy`` capability requires additional |
| 74 | infrastructure. |
| 75 | |
| 76 | - Host OS support |
| 77 | Because this feautre essentially depends on ``userfaultfd`` |
| 78 | to trap page faults, it is supported only on Linux hosts. |
| 79 | |
| 80 | - vhost-user |
| 81 | Fast snapshot load does not currently support ``vhost-user`` |
| 82 | backends. |