| 1 | CheckPoint and Restart (CPR) |
| 2 | ============================ |
| 3 | |
| 4 | CPR is the umbrella name for a set of migration modes in which the |
| 5 | VM is migrated to a new QEMU instance on the same host. It is |
| 6 | intended for use when the goal is to update host software components |
| 7 | that run the VM, such as QEMU or even the host kernel. At this time, |
| 8 | the cpr-reboot, cpr-transfer, and cpr-exec modes are available. |
| 9 | |
| 10 | Because QEMU is restarted on the same host, with access to the same |
| 11 | local devices, CPR is allowed in certain cases where normal migration |
| 12 | would be blocked. However, the user must not modify the contents of |
| 13 | guest block devices between quitting old QEMU and starting new QEMU. |
| 14 | |
| 15 | CPR unconditionally stops VM execution before memory is saved, and |
| 16 | thus does not depend on any form of dirty page tracking. |
| 17 | |
| 18 | cpr-reboot mode |
| 19 | --------------- |
| 20 | |
| 21 | In this mode, QEMU stops the VM, and writes VM state to the migration |
| 22 | URI, which will typically be a file. After quitting QEMU, the user |
| 23 | resumes by running QEMU with the ``-incoming`` option. Because the |
| 24 | old and new QEMU instances are not active concurrently, the URI cannot |
| 25 | be a type that streams data from one instance to the other. |
| 26 | |
| 27 | Guest RAM can be saved in place if backed by shared memory, or can be |
| 28 | copied to a file. The former is more efficient and is therefore |
| 29 | preferred. |
| 30 | |
| 31 | After state and memory are saved, the user may update userland host |
| 32 | software before restarting QEMU and resuming the VM. Further, if |
| 33 | the RAM is backed by persistent shared memory, such as a DAX device, |
| 34 | then the user may reboot to a new host kernel before restarting QEMU. |
| 35 | |
| 36 | This mode supports VFIO devices provided the user first puts the |
| 37 | guest in the suspended runstate, such as by issuing the |
| 38 | ``guest-suspend-ram`` command to the QEMU guest agent. The agent |
| 39 | must be pre-installed in the guest, and the guest must support |
| 40 | suspend to RAM. Beware that suspension can take a few seconds, so |
| 41 | the user should poll to see the suspended state before proceeding |
| 42 | with the CPR operation. |
| 43 | |
| 44 | Usage |
| 45 | ^^^^^ |
| 46 | |
| 47 | It is recommended that guest RAM be backed with some type of shared |
| 48 | memory, such as ``memory-backend-file,share=on``, and that the |
| 49 | ``x-ignore-shared`` capability be set. This combination allows memory |
| 50 | to be saved in place. Otherwise, after QEMU stops the VM, all guest |
| 51 | RAM is copied to the migration URI. |
| 52 | |
| 53 | Outgoing: |
| 54 | * Set the migration mode parameter to ``cpr-reboot``. |
| 55 | * Set the ``x-ignore-shared`` capability if desired. |
| 56 | * Issue the ``migrate`` command. It is recommended the URI be a |
| 57 | ``file`` type, but one can use other types such as ``exec``, |
| 58 | provided the command captures all the data from the outgoing side, |
| 59 | and provides all the data to the incoming side. |
| 60 | * Quit when QEMU reaches the postmigrate state. |
| 61 | |
| 62 | Incoming: |
| 63 | * Start QEMU with the ``-incoming defer`` option. |
| 64 | * Set the migration mode parameter to ``cpr-reboot``. |
| 65 | * Set the ``x-ignore-shared`` capability if desired. |
| 66 | * Issue the ``migrate-incoming`` command. |
| 67 | * If the VM was running when the outgoing ``migrate`` command was |
| 68 | issued, then QEMU automatically resumes VM execution. |
| 69 | |
| 70 | Example 1 |
| 71 | ^^^^^^^^^ |
| 72 | :: |
| 73 | |
| 74 | # qemu-kvm -monitor stdio |
| 75 | -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G |
| 76 | ... |
| 77 | |
| 78 | (qemu) info status |
| 79 | VM status: running |
| 80 | (qemu) migrate_set_parameter mode cpr-reboot |
| 81 | (qemu) migrate_set_capability x-ignore-shared on |
| 82 | (qemu) migrate -d file:vm.state |
| 83 | (qemu) info status |
| 84 | VM status: paused (postmigrate) |
| 85 | (qemu) quit |
| 86 | |
| 87 | ### optionally update kernel and reboot |
| 88 | # systemctl kexec |
| 89 | kexec_core: Starting new kernel |
| 90 | ... |
| 91 | |
| 92 | # qemu-kvm ... -incoming defer |
| 93 | (qemu) info status |
| 94 | VM status: paused (inmigrate) |
| 95 | (qemu) migrate_set_parameter mode cpr-reboot |
| 96 | (qemu) migrate_set_capability x-ignore-shared on |
| 97 | (qemu) migrate_incoming file:vm.state |
| 98 | (qemu) info status |
| 99 | VM status: running |
| 100 | |
| 101 | Example 2: VFIO |
| 102 | ^^^^^^^^^^^^^^^ |
| 103 | :: |
| 104 | |
| 105 | # qemu-kvm -monitor stdio |
| 106 | -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/dax0.0,align=2M,share=on -m 4G |
| 107 | -device vfio-pci, ... |
| 108 | -chardev socket,id=qga0,path=qga.sock,server=on,wait=off |
| 109 | -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 |
| 110 | ... |
| 111 | |
| 112 | (qemu) info status |
| 113 | VM status: running |
| 114 | |
| 115 | # echo '{"execute":"guest-suspend-ram"}' | ncat --send-only -U qga.sock |
| 116 | |
| 117 | (qemu) info status |
| 118 | VM status: paused (suspended) |
| 119 | (qemu) migrate_set_parameter mode cpr-reboot |
| 120 | (qemu) migrate_set_capability x-ignore-shared on |
| 121 | (qemu) migrate -d file:vm.state |
| 122 | (qemu) info status |
| 123 | VM status: paused (postmigrate) |
| 124 | (qemu) quit |
| 125 | |
| 126 | ### optionally update kernel and reboot |
| 127 | # systemctl kexec |
| 128 | kexec_core: Starting new kernel |
| 129 | ... |
| 130 | |
| 131 | # qemu-kvm ... -incoming defer |
| 132 | (qemu) info status |
| 133 | VM status: paused (inmigrate) |
| 134 | (qemu) migrate_set_parameter mode cpr-reboot |
| 135 | (qemu) migrate_set_capability x-ignore-shared on |
| 136 | (qemu) migrate_incoming file:vm.state |
| 137 | (qemu) info status |
| 138 | VM status: paused (suspended) |
| 139 | (qemu) system_wakeup |
| 140 | (qemu) info status |
| 141 | VM status: running |
| 142 | |
| 143 | Caveats |
| 144 | ^^^^^^^ |
| 145 | |
| 146 | cpr-reboot mode may not be used with postcopy, background-snapshot, |
| 147 | or COLO. |
| 148 | |
| 149 | cpr-transfer mode |
| 150 | ----------------- |
| 151 | |
| 152 | This mode allows the user to transfer a guest to a new QEMU instance |
| 153 | on the same host with minimal guest pause time, by preserving guest |
| 154 | RAM in place, albeit with new virtual addresses in new QEMU. Devices |
| 155 | and their pinned memory pages are also preserved for VFIO and IOMMUFD. |
| 156 | |
| 157 | The user starts new QEMU on the same host as old QEMU, with command- |
| 158 | line arguments to create the same machine, plus the ``-incoming`` |
| 159 | option for the main migration channel, like normal live migration. |
| 160 | In addition, the user adds a second -incoming option with channel |
| 161 | type ``cpr``. This CPR channel must support file descriptor transfer |
| 162 | with SCM_RIGHTS, i.e. it must be a UNIX domain socket. |
| 163 | |
| 164 | To initiate CPR, the user issues a migrate command to old QEMU, |
| 165 | adding a second migration channel of type ``cpr`` in the channels |
| 166 | argument. Old QEMU stops the VM, saves state to the migration |
| 167 | channels, and enters the postmigrate state. Execution resumes in |
| 168 | new QEMU. |
| 169 | |
| 170 | New QEMU reads the CPR channel before opening a monitor, hence |
| 171 | the CPR channel cannot be specified in the list of channels for a |
| 172 | migrate-incoming command. It may only be specified on the command |
| 173 | line. |
| 174 | |
| 175 | Usage |
| 176 | ^^^^^ |
| 177 | |
| 178 | Memory backend objects must have the ``share=on`` attribute. |
| 179 | |
| 180 | The VM must be started with the ``-machine aux-ram-share=on`` |
| 181 | option. This causes implicit RAM blocks (those not described by |
| 182 | a memory-backend object) to be allocated by mmap'ing a memfd. |
| 183 | Examples include VGA and ROM. |
| 184 | |
| 185 | Outgoing: |
| 186 | * Set the migration mode parameter to ``cpr-transfer``. |
| 187 | * Issue the ``migrate`` command, containing a main channel and |
| 188 | a cpr channel. |
| 189 | |
| 190 | Incoming: |
| 191 | * Start new QEMU with two ``-incoming`` options. |
| 192 | * If the VM was running when the outgoing ``migrate`` command was |
| 193 | issued, then QEMU automatically resumes VM execution. |
| 194 | |
| 195 | Caveats |
| 196 | ^^^^^^^ |
| 197 | |
| 198 | cpr-transfer mode may not be used with postcopy, background-snapshot, |
| 199 | or COLO. |
| 200 | |
| 201 | memory-backend-epc is not supported. |
| 202 | |
| 203 | The main incoming migration channel address cannot be a file type. |
| 204 | |
| 205 | If the main incoming channel address is an inet socket, then the port |
| 206 | cannot be 0 (meaning dynamically choose a port). |
| 207 | |
| 208 | When using ``-incoming defer``, you must issue the migrate command to |
| 209 | old QEMU before issuing any monitor commands to new QEMU, because new |
| 210 | QEMU blocks waiting to read from the cpr channel before starting its |
| 211 | monitor, and old QEMU does not write to the channel until the migrate |
| 212 | command is issued. However, new QEMU does not open and read the |
| 213 | main migration channel until you issue the migrate incoming command. |
| 214 | |
| 215 | Example 1: incoming channel |
| 216 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 217 | |
| 218 | In these examples, we simply restart the same version of QEMU, but |
| 219 | in a real scenario one would start new QEMU on the incoming side. |
| 220 | Note that new QEMU does not print the monitor prompt until old QEMU |
| 221 | has issued the migrate command. The outgoing side uses QMP because |
| 222 | HMP cannot specify a CPR channel. Some QMP responses are omitted for |
| 223 | brevity. |
| 224 | |
| 225 | :: |
| 226 | |
| 227 | Outgoing: Incoming: |
| 228 | |
| 229 | # qemu-kvm -qmp stdio |
| 230 | -object memory-backend-file,id=ram0,size=4G, |
| 231 | mem-path=/dev/shm/ram0,share=on -m 4G |
| 232 | -machine memory-backend=ram0 |
| 233 | -machine aux-ram-share=on |
| 234 | ... |
| 235 | # qemu-kvm -monitor stdio |
| 236 | -incoming tcp:0:44444 |
| 237 | -incoming '{"channel-type": "cpr", |
| 238 | "addr": { "transport": "socket", |
| 239 | "type": "unix", "path": "cpr.sock"}}' |
| 240 | ... |
| 241 | {"execute":"qmp_capabilities"} |
| 242 | |
| 243 | {"execute": "query-status"} |
| 244 | {"return": {"status": "running", |
| 245 | "running": true}} |
| 246 | |
| 247 | {"execute":"migrate-set-parameters", |
| 248 | "arguments":{"mode":"cpr-transfer"}} |
| 249 | |
| 250 | {"execute": "migrate", "arguments": { "channels": [ |
| 251 | {"channel-type": "main", |
| 252 | "addr": { "transport": "socket", "type": "inet", |
| 253 | "host": "0", "port": "44444" }}, |
| 254 | {"channel-type": "cpr", |
| 255 | "addr": { "transport": "socket", "type": "unix", |
| 256 | "path": "cpr.sock" }}]}} |
| 257 | |
| 258 | QEMU 10.0.50 monitor |
| 259 | (qemu) info status |
| 260 | VM status: running |
| 261 | |
| 262 | {"execute": "query-status"} |
| 263 | {"return": {"status": "postmigrate", |
| 264 | "running": false}} |
| 265 | |
| 266 | Example 2: incoming defer |
| 267 | ^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 268 | |
| 269 | This example uses ``-incoming defer`` to hot plug a device before |
| 270 | accepting the main migration channel. Again note you must issue the |
| 271 | migrate command to old QEMU before you can issue any monitor |
| 272 | commands to new QEMU. |
| 273 | |
| 274 | |
| 275 | :: |
| 276 | |
| 277 | Outgoing: Incoming: |
| 278 | |
| 279 | # qemu-kvm -monitor stdio |
| 280 | -object memory-backend-file,id=ram0,size=4G, |
| 281 | mem-path=/dev/shm/ram0,share=on -m 4G |
| 282 | -machine memory-backend=ram0 |
| 283 | -machine aux-ram-share=on |
| 284 | ... |
| 285 | # qemu-kvm -monitor stdio |
| 286 | -incoming defer |
| 287 | -incoming '{"channel-type": "cpr", |
| 288 | "addr": { "transport": "socket", |
| 289 | "type": "unix", "path": "cpr.sock"}}' |
| 290 | ... |
| 291 | {"execute":"qmp_capabilities"} |
| 292 | |
| 293 | {"execute": "device_add", |
| 294 | "arguments": {"driver": "pcie-root-port"}} |
| 295 | |
| 296 | {"execute":"migrate-set-parameters", |
| 297 | "arguments":{"mode":"cpr-transfer"}} |
| 298 | |
| 299 | {"execute": "migrate", "arguments": { "channels": [ |
| 300 | {"channel-type": "main", |
| 301 | "addr": { "transport": "socket", "type": "inet", |
| 302 | "host": "0", "port": "44444" }}, |
| 303 | {"channel-type": "cpr", |
| 304 | "addr": { "transport": "socket", "type": "unix", |
| 305 | "path": "cpr.sock" }}]}} |
| 306 | |
| 307 | QEMU 10.0.50 monitor |
| 308 | (qemu) info status |
| 309 | VM status: paused (inmigrate) |
| 310 | (qemu) device_add pcie-root-port |
| 311 | (qemu) migrate_incoming tcp:0:44444 |
| 312 | (qemu) info status |
| 313 | VM status: running |
| 314 | |
| 315 | {"execute": "query-status"} |
| 316 | {"return": {"status": "postmigrate", |
| 317 | "running": false}} |
| 318 | |
| 319 | Futures |
| 320 | ^^^^^^^ |
| 321 | |
| 322 | cpr-transfer mode is based on a capability to transfer open file |
| 323 | descriptors from old to new QEMU. In the future, descriptors for |
| 324 | vhost, and char devices could be transferred, |
| 325 | preserving those devices and their kernel state without interruption, |
| 326 | even if they do not explicitly support live migration. |
| 327 | |
| 328 | cpr-exec mode |
| 329 | ------------- |
| 330 | |
| 331 | In this mode, QEMU stops the VM, writes VM state to the migration |
| 332 | URI, and directly exec's a new version of QEMU on the same host, |
| 333 | replacing the original process while retaining its PID. Guest RAM is |
| 334 | preserved in place, albeit with new virtual addresses. The user |
| 335 | completes the migration by specifying the ``-incoming`` option, and |
| 336 | by issuing the ``migrate-incoming`` command if necessary; see details |
| 337 | below. |
| 338 | |
| 339 | This mode supports VFIO/IOMMUFD devices by preserving device |
| 340 | descriptors and hence kernel state across the exec, even for devices |
| 341 | that do not support live migration. |
| 342 | |
| 343 | Because the old and new QEMU instances are not active concurrently, |
| 344 | the URI cannot be a type that streams data from one instance to the |
| 345 | other. |
| 346 | |
| 347 | This mode does not require a channel of type ``cpr``. The information |
| 348 | that is passed over that channel for cpr-transfer mode is instead |
| 349 | serialized to a memfd, the number of the fd is saved in the |
| 350 | QEMU_CPR_EXEC_STATE environment variable during the exec of new QEMU. |
| 351 | and new QEMU mmaps the memfd. |
| 352 | |
| 353 | Usage |
| 354 | ^^^^^ |
| 355 | |
| 356 | Arguments for the new QEMU process are taken from the |
| 357 | @cpr-exec-command parameter. The first argument should be the |
| 358 | path of a new QEMU binary, or a prefix command that exec's the |
| 359 | new QEMU binary, and the arguments should include the ''-incoming'' |
| 360 | option. |
| 361 | |
| 362 | Memory backend objects must have the ``share=on`` attribute. |
| 363 | The VM must be started with the ``-machine aux-ram-share=on`` option. |
| 364 | |
| 365 | Outgoing: |
| 366 | * Set the migration mode parameter to ``cpr-exec``. |
| 367 | * Set the ``cpr-exec-command`` parameter. |
| 368 | * Issue the ``migrate`` command. It is recommended that the URI be |
| 369 | a ``file`` type, but one can use other types such as ``exec``, |
| 370 | provided the command captures all the data from the outgoing side, |
| 371 | and provides all the data to the incoming side. |
| 372 | |
| 373 | Incoming: |
| 374 | * You do not need to explicitly start new QEMU. It is started as |
| 375 | a side effect of the migrate command above. |
| 376 | * If the VM was running when the outgoing ``migrate`` command was |
| 377 | issued, then QEMU automatically resumes VM execution. |
| 378 | |
| 379 | Example 1: incoming URI |
| 380 | ^^^^^^^^^^^^^^^^^^^^^^^ |
| 381 | |
| 382 | In these examples, we simply restart the same version of QEMU, but in |
| 383 | a real scenario one would set a new QEMU binary path in |
| 384 | cpr-exec-command. |
| 385 | |
| 386 | :: |
| 387 | |
| 388 | # qemu-kvm -monitor stdio |
| 389 | -object memory-backend-memfd,id=ram0,size=4G |
| 390 | -machine memory-backend=ram0 |
| 391 | -machine aux-ram-share=on |
| 392 | ... |
| 393 | |
| 394 | QEMU 10.2.50 monitor - type 'help' for more information |
| 395 | (qemu) info status |
| 396 | VM status: running |
| 397 | (qemu) migrate_set_parameter mode cpr-exec |
| 398 | (qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming file:vm.state |
| 399 | (qemu) migrate -d file:vm.state |
| 400 | (qemu) QEMU 10.2.50 monitor - type 'help' for more information |
| 401 | (qemu) info status |
| 402 | VM status: running |
| 403 | |
| 404 | Example 2: incoming defer |
| 405 | ^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 406 | :: |
| 407 | |
| 408 | # qemu-kvm -monitor stdio |
| 409 | -object memory-backend-memfd,id=ram0,size=4G |
| 410 | -machine memory-backend=ram0 |
| 411 | -machine aux-ram-share=on |
| 412 | ... |
| 413 | |
| 414 | QEMU 10.2.50 monitor - type 'help' for more information |
| 415 | (qemu) info status |
| 416 | VM status: running |
| 417 | (qemu) migrate_set_parameter mode cpr-exec |
| 418 | (qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming defer |
| 419 | (qemu) migrate -d file:vm.state |
| 420 | (qemu) QEMU 10.2.50 monitor - type 'help' for more information |
| 421 | (qemu) info status |
| 422 | status: paused (inmigrate) |
| 423 | (qemu) migrate_incoming file:vm.state |
| 424 | (qemu) info status |
| 425 | VM status: running |
| 426 | |
| 427 | Caveats |
| 428 | ^^^^^^^ |
| 429 | |
| 430 | cpr-exec mode may not be used with postcopy, background-snapshot, |
| 431 | or COLO. |
| 432 | |
| 433 | cpr-exec mode requires permission to use the exec system call, which |
| 434 | is denied by certain sandbox options, such as spawn. |
| 435 | |
| 436 | The guest pause time increases for large guest RAM backed by small pages. |