| 1 | QEMU VM templating |
| 2 | ================== |
| 3 | |
| 4 | This document explains how to use VM templating in QEMU. |
| 5 | |
| 6 | For now, the focus is on VM memory aspects, and not about how to save and |
| 7 | restore other VM state (i.e., migrate-to-file with ``x-ignore-shared``). |
| 8 | |
| 9 | Overview |
| 10 | -------- |
| 11 | |
| 12 | With VM templating, a single template VM serves as the starting point for |
| 13 | new VMs. This allows for fast and efficient replication of VMs, resulting |
| 14 | in fast startup times and reduced memory consumption. |
| 15 | |
| 16 | Conceptually, the VM state is frozen, to then be used as a basis for new |
| 17 | VMs. The Copy-On-Write mechanism in the operating systems makes sure that |
| 18 | new VMs are able to read template VM memory; however, any modifications |
| 19 | stay private and don't modify the original template VM or any other |
| 20 | created VM. |
| 21 | |
| 22 | !!! Security Alert !!! |
| 23 | ---------------------- |
| 24 | |
| 25 | When effectively cloning VMs by VM templating, hardware identifiers |
| 26 | (such as UUIDs and NIC MAC addresses), and similar data in the guest OS |
| 27 | (such as machine IDs, SSH keys, certificates) that are supposed to be |
| 28 | *unique* are no longer unique, which can be a security concern. |
| 29 | |
| 30 | Please be aware of these implications and how to mitigate them for your |
| 31 | use case, which might involve vmgenid, hot(un)plug of NIC, etc.. |
| 32 | |
| 33 | Memory configuration |
| 34 | -------------------- |
| 35 | |
| 36 | In order to create the template VM, we have to make sure that VM memory |
| 37 | ends up in a file, from where it can be reused for the new VMs: |
| 38 | |
| 39 | Supply VM RAM via memory-backend-file, with ``share=on`` (modifications go |
| 40 | to the file) and ``readonly=off`` (open the file writable). Note that |
| 41 | ``readonly=off`` is implicit. |
| 42 | |
| 43 | In the following command-line example, a 2GB VM is created, whereby VM RAM |
| 44 | is to be stored in the ``template`` file. |
| 45 | |
| 46 | .. parsed-literal:: |
| 47 | |
| 48 | |qemu_system| [...] -m 2g \\ |
| 49 | -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,share=on,... \\ |
| 50 | -machine q35,memory-backend=pc.ram |
| 51 | |
| 52 | If multiple memory backends are used (vNUMA, DIMMs), configure all |
| 53 | memory backends accordingly. |
| 54 | |
| 55 | Once the VM is in the desired state, stop the VM and save other VM state, |
| 56 | leaving the current state of VM RAM reside in the file. |
| 57 | |
| 58 | In order to have a new VM be based on a template VM, we have to |
| 59 | configure VM RAM to be based on a template VM RAM file; however, the VM |
| 60 | should not be able to modify file content. |
| 61 | |
| 62 | Supply VM RAM via memory-backend-file, with ``share=off`` (modifications |
| 63 | stay private), ``readonly=on`` (open the file readonly) and ``rom=off`` |
| 64 | (don't make the memory readonly for the VM). Note that ``share=off`` is |
| 65 | implicit and that other VM state has to be restored separately. |
| 66 | |
| 67 | In the following command-line example, a 2GB VM is created based on the |
| 68 | existing 2GB file ``template``. |
| 69 | |
| 70 | .. parsed-literal:: |
| 71 | |
| 72 | |qemu_system| [...] -m 2g \\ |
| 73 | -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,readonly=on,rom=off,... \\ |
| 74 | -machine q35,memory-backend=pc.ram |
| 75 | |
| 76 | If multiple memory backends are used (vNUMA, DIMMs), configure all |
| 77 | memory backends accordingly. |
| 78 | |
| 79 | Note that ``-mem-path`` cannot be used for VM templating when creating the |
| 80 | template VM or when starting new VMs based on a template VM. |
| 81 | |
| 82 | Incompatible features |
| 83 | --------------------- |
| 84 | |
| 85 | Some features are incompatible with VM templating, as the underlying file |
| 86 | cannot be modified to discard VM RAM, or to actually share memory with |
| 87 | another process. |
| 88 | |
| 89 | vhost-user and multi-process QEMU |
| 90 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 91 | |
| 92 | vhost-user and multi-process QEMU are incompatible with VM templating. |
| 93 | These technologies rely on shared memory, however, the template VMs |
| 94 | don't actually share memory (``share=off``), even though they are |
| 95 | file-based. |
| 96 | |
| 97 | virtio-balloon |
| 98 | ~~~~~~~~~~~~~~ |
| 99 | |
| 100 | virtio-balloon inflation and "free page reporting" cannot discard VM RAM |
| 101 | and will repeatedly report errors. While virtio-balloon can be used |
| 102 | for template VMs (e.g., report VM RAM stats), "free page reporting" |
| 103 | should be disabled and the balloon should not be inflated. |
| 104 | |
| 105 | virtio-mem |
| 106 | ~~~~~~~~~~ |
| 107 | |
| 108 | virtio-mem cannot discard VM RAM that is managed by the virtio-mem |
| 109 | device. virtio-mem will fail early when realizing the device. To use |
| 110 | VM templating with virtio-mem, either hotplug virtio-mem devices to the |
| 111 | new VM, or don't supply any memory to the template VM using virtio-mem |
| 112 | (requested-size=0), not using a template VM file as memory backend for the |
| 113 | virtio-mem device. |
| 114 | |
| 115 | VM migration |
| 116 | ~~~~~~~~~~~~ |
| 117 | |
| 118 | For VM migration, "x-release-ram" similarly relies on discarding of VM |
| 119 | RAM on the migration source to free up migrated RAM, and will |
| 120 | repeatedly report errors. |
| 121 | |
| 122 | Postcopy live migration fails discarding VM RAM on the migration |
| 123 | destination early and refuses to activate postcopy live migration. Note |
| 124 | that postcopy live migration usually only works on selected filesystems |
| 125 | (shmem/tmpfs, hugetlbfs) either way. |