| 1 | Mapped-ram |
| 2 | ========== |
| 3 | |
| 4 | Mapped-ram is a new stream format for the RAM section designed to |
| 5 | supplement the existing ``file:`` migration and make it compatible |
| 6 | with ``multifd``. This enables parallel migration of a guest's RAM to |
| 7 | a file. |
| 8 | |
| 9 | The core of the feature is to ensure that RAM pages are mapped |
| 10 | directly to offsets in the resulting migration file. This enables the |
| 11 | ``multifd`` threads to write exclusively to those offsets even if the |
| 12 | guest is constantly dirtying pages (i.e. live migration). Another |
| 13 | benefit is that the resulting file will have a bounded size, since |
| 14 | pages which are dirtied multiple times will always go to a fixed |
| 15 | location in the file, rather than constantly being added to a |
| 16 | sequential stream. Having the pages at fixed offsets also allows the |
| 17 | usage of O_DIRECT for save/restore of the migration stream as the |
| 18 | pages are ensured to be written respecting O_DIRECT alignment |
| 19 | restrictions. |
| 20 | |
| 21 | Usage |
| 22 | ----- |
| 23 | |
| 24 | On both source and destination, enable the ``multifd`` and |
| 25 | ``mapped-ram`` capabilities: |
| 26 | |
| 27 | ``migrate_set_capability multifd on`` |
| 28 | |
| 29 | ``migrate_set_capability mapped-ram on`` |
| 30 | |
| 31 | Use a ``file:`` URL for migration: |
| 32 | |
| 33 | ``migrate file:/path/to/migration/file`` |
| 34 | |
| 35 | Mapped-ram migration is best done non-live, i.e. by stopping the VM on |
| 36 | the source side before migrating. |
| 37 | |
| 38 | For best performance enable the ``direct-io`` parameter as well: |
| 39 | |
| 40 | ``migrate_set_parameter direct-io on`` |
| 41 | |
| 42 | Use-cases |
| 43 | --------- |
| 44 | |
| 45 | The mapped-ram feature was designed for use cases where the migration |
| 46 | stream will be directed to a file in the filesystem and not |
| 47 | immediately restored on the destination VM\ [#alternatives]_. These could be |
| 48 | thought of as snapshots. We can further categorize them into live and |
| 49 | non-live. |
| 50 | |
| 51 | - Non-live snapshot |
| 52 | |
| 53 | If the use case requires a VM to be stopped before taking a snapshot, |
| 54 | that's the ideal scenario for mapped-ram migration. Not having to |
| 55 | track dirty pages, the migration will write the RAM pages to the disk |
| 56 | as fast as it can. |
| 57 | |
| 58 | Note: if a snapshot is taken of a running VM, but the VM will be |
| 59 | stopped after the snapshot by the admin, then consider stopping it |
| 60 | right before the snapshot to take benefit of the performance gains |
| 61 | mentioned above. |
| 62 | |
| 63 | - Live snapshot |
| 64 | |
| 65 | If the use case requires that the VM keeps running during and after |
| 66 | the snapshot operation, then mapped-ram migration can still be used, |
| 67 | but will be less performant. Other strategies such as |
| 68 | background-snapshot should be evaluated as well. One benefit of |
| 69 | mapped-ram in this scenario is portability since background-snapshot |
| 70 | depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not |
| 71 | supported outside of Linux. |
| 72 | |
| 73 | .. [#alternatives] While this same effect could be obtained with the usage of |
| 74 | snapshots or the ``file:`` migration alone, mapped-ram provides |
| 75 | a performance increase for VMs with larger RAM sizes (10s to |
| 76 | 100s of GiBs), specially if the VM has been stopped beforehand. |
| 77 | |
| 78 | RAM section format |
| 79 | ------------------ |
| 80 | |
| 81 | Instead of having a sequential stream of pages that follow the |
| 82 | RAMBlock headers, the dirty pages for a RAMBlock follow its header |
| 83 | instead. This ensures that each RAM page has a fixed offset in the |
| 84 | resulting migration file. |
| 85 | |
| 86 | A bitmap is introduced to track which pages have been written in the |
| 87 | migration file. Pages are written at a fixed location for every |
| 88 | ramblock. Zero pages are ignored as they'd be zero in the destination |
| 89 | migration as well. |
| 90 | |
| 91 | :: |
| 92 | |
| 93 | Without mapped-ram: With mapped-ram: |
| 94 | |
| 95 | --------------------- -------------------------------- |
| 96 | | ramblock 1 header | | ramblock 1 header | |
| 97 | --------------------- -------------------------------- |
| 98 | | ramblock 2 header | | ramblock 1 mapped-ram header | |
| 99 | --------------------- -------------------------------- |
| 100 | | ... | | padding to next 1MB boundary | |
| 101 | --------------------- | ... | |
| 102 | | ramblock n header | -------------------------------- |
| 103 | --------------------- | ramblock 1 pages | |
| 104 | | RAM_SAVE_FLAG_EOS | | ... | |
| 105 | --------------------- -------------------------------- |
| 106 | | stream of pages | | ramblock 2 header | |
| 107 | | (iter 1) | -------------------------------- |
| 108 | | ... | | ramblock 2 mapped-ram header | |
| 109 | --------------------- -------------------------------- |
| 110 | | RAM_SAVE_FLAG_EOS | | padding to next 1MB boundary | |
| 111 | --------------------- | ... | |
| 112 | | stream of pages | -------------------------------- |
| 113 | | (iter 2) | | ramblock 2 pages | |
| 114 | | ... | | ... | |
| 115 | --------------------- -------------------------------- |
| 116 | | ... | | ... | |
| 117 | --------------------- -------------------------------- |
| 118 | | RAM_SAVE_FLAG_EOS | |
| 119 | -------------------------------- |
| 120 | | ... | |
| 121 | -------------------------------- |
| 122 | |
| 123 | where: |
| 124 | - ramblock header: the generic information for a ramblock, such as |
| 125 | idstr, used_len, etc. |
| 126 | |
| 127 | - ramblock mapped-ram header: the information added by this feature: |
| 128 | bitmap of pages written, bitmap size and offset of pages in the |
| 129 | migration file. |
| 130 | |
| 131 | Restrictions |
| 132 | ------------ |
| 133 | |
| 134 | Since pages are written to their relative offsets and out of order |
| 135 | (due to the memory dirtying patterns), streaming channels such as |
| 136 | sockets are not supported. A seekable channel such as a file is |
| 137 | required. This can be verified in the QIOChannel by the presence of |
| 138 | the QIO_CHANNEL_FEATURE_SEEKABLE. |
| 139 | |
| 140 | The improvements brought by this feature apply only to guest physical |
| 141 | RAM. Other types of memory such as VRAM are migrated as part of device |
| 142 | states. |