| 1 | .. |
| 2 | Copyright 2019 John Snow <jsnow@redhat.com> and Red Hat, Inc. |
| 3 | All rights reserved. |
| 4 | |
| 5 | This file is licensed via The FreeBSD Documentation License, the full |
| 6 | text of which is included at the end of this document. |
| 7 | |
| 8 | ==================================== |
| 9 | Dirty Bitmaps and Incremental Backup |
| 10 | ==================================== |
| 11 | |
| 12 | Dirty Bitmaps are in-memory objects that track writes to block devices. They |
| 13 | can be used in conjunction with various block job operations to perform |
| 14 | incremental or differential backup regimens. |
| 15 | |
| 16 | This document explains the conceptual mechanisms, as well as up-to-date, |
| 17 | complete and comprehensive documentation on the API to manipulate them. |
| 18 | (Hopefully, the "why", "what", and "how".) |
| 19 | |
| 20 | The intended audience for this document is developers who are adding QEMU |
| 21 | backup features to management applications, or power users who run and |
| 22 | administer QEMU directly via QMP. |
| 23 | |
| 24 | .. contents:: |
| 25 | |
| 26 | Overview |
| 27 | -------- |
| 28 | |
| 29 | Bitmaps are bit vectors where each '1' bit in the vector indicates a modified |
| 30 | ("dirty") segment of the corresponding block device. The size of the segment |
| 31 | that is tracked is the granularity of the bitmap. If the granularity of a |
| 32 | bitmap is 64K, each '1' bit means that a 64K region as a whole may have |
| 33 | changed in some way, possibly by as little as one byte. |
| 34 | |
| 35 | Smaller granularities mean more accurate tracking of modified disk data, but |
| 36 | requires more computational overhead and larger bitmap sizes. Larger |
| 37 | granularities mean smaller bitmap sizes, but less targeted backups. |
| 38 | |
| 39 | The size of a bitmap (in bytes) can be computed as such: |
| 40 | ``size`` = ceil(ceil(``image_size`` / ``granularity``) / 8) |
| 41 | |
| 42 | e.g. the size of a 64KiB granularity bitmap on a 2TiB image is: |
| 43 | ``size`` = ((2147483648K / 64K) / 8) |
| 44 | = 4194304B = 4MiB. |
| 45 | |
| 46 | QEMU uses these bitmaps when making incremental backups to know which sections |
| 47 | of the file to copy out. They are not enabled by default and must be |
| 48 | explicitly added in order to begin tracking writes. |
| 49 | |
| 50 | Bitmaps can be created at any time and can be attached to any arbitrary block |
| 51 | node in the storage graph, but are most useful conceptually when attached to |
| 52 | the root node attached to the guest's storage device model. |
| 53 | |
| 54 | That is to say: It's likely most useful to track the guest's writes to disk, |
| 55 | but you could theoretically track things like qcow2 metadata changes by |
| 56 | attaching the bitmap elsewhere in the storage graph. This is beyond the scope |
| 57 | of this document. |
| 58 | |
| 59 | QEMU supports persisting these bitmaps to disk via the qcow2 image format. |
| 60 | Bitmaps which are stored or loaded in this way are called "persistent", |
| 61 | whereas bitmaps that are not are called "transient". |
| 62 | |
| 63 | QEMU also supports the migration of both transient bitmaps (tracking any |
| 64 | arbitrary image format) or persistent bitmaps (qcow2) via live migration. |
| 65 | |
| 66 | Supported Image Formats |
| 67 | ----------------------- |
| 68 | |
| 69 | QEMU supports all documented features below on the qcow2 image format. |
| 70 | |
| 71 | However, qcow2 is only strictly necessary for the persistence feature, which |
| 72 | writes bitmap data to disk upon close. If persistence is not required for a |
| 73 | specific use case, all bitmap features excepting persistence are available for |
| 74 | any arbitrary image format. |
| 75 | |
| 76 | For example, Dirty Bitmaps can be combined with the 'raw' image format, but |
| 77 | any changes to the bitmap will be discarded upon exit. |
| 78 | |
| 79 | .. warning:: Transient bitmaps will not be saved on QEMU exit! Persistent |
| 80 | bitmaps are available only on qcow2 images. |
| 81 | |
| 82 | Dirty Bitmap Names |
| 83 | ------------------ |
| 84 | |
| 85 | Bitmap objects need a method to reference them in the API. All API-created and |
| 86 | managed bitmaps have a human-readable name chosen by the user at creation |
| 87 | time. |
| 88 | |
| 89 | - A bitmap's name is unique to the node, but bitmaps attached to different |
| 90 | nodes can share the same name. Therefore, all bitmaps are addressed via |
| 91 | their (node, name) pair. |
| 92 | |
| 93 | - The name of a user-created bitmap cannot be empty (""). |
| 94 | |
| 95 | - Transient bitmaps can have JSON unicode names that are effectively not |
| 96 | length limited. (QMP protocol may restrict messages to less than 64MiB.) |
| 97 | |
| 98 | - Persistent storage formats may impose their own requirements on bitmap names |
| 99 | and namespaces. Presently, only qcow2 supports persistent bitmaps. See |
| 100 | :doc:`qcow2` for more details on restrictions. Notably: |
| 101 | |
| 102 | - qcow2 bitmap names are limited to between 1 and 1023 bytes long. |
| 103 | |
| 104 | - No two bitmaps saved to the same qcow2 file may share the same name. |
| 105 | |
| 106 | - QEMU occasionally uses bitmaps for internal use which have no name. They are |
| 107 | hidden from API query calls, cannot be manipulated by the external API, are |
| 108 | never persistent, nor ever migrated. |
| 109 | |
| 110 | Bitmap Status |
| 111 | ------------- |
| 112 | |
| 113 | Dirty Bitmap objects can be queried with the QMP command `query-block |
| 114 | <qemu-qmp-ref.html#index-query_002dblock>`_, and are visible via the |
| 115 | `BlockDirtyInfo <qemu-qmp-ref.html#index-BlockDirtyInfo>`_ QAPI structure. |
| 116 | |
| 117 | This struct shows the name, granularity, and dirty byte count for each bitmap. |
| 118 | Additionally, it shows several boolean status indicators: |
| 119 | |
| 120 | - ``recording``: This bitmap is recording writes. |
| 121 | - ``busy``: This bitmap is in-use by an operation. |
| 122 | - ``persistent``: This bitmap is a persistent type. |
| 123 | - ``inconsistent``: This bitmap is corrupted and cannot be used. |
| 124 | |
| 125 | The ``+busy`` status prohibits you from deleting, clearing, or otherwise |
| 126 | modifying a bitmap, and happens when the bitmap is being used for a backup |
| 127 | operation or is in the process of being loaded from a migration. Many of the |
| 128 | commands documented below will refuse to work on such bitmaps. |
| 129 | |
| 130 | The ``+inconsistent`` status similarly prohibits almost all operations, |
| 131 | notably allowing only the ``block-dirty-bitmap-remove`` operation. |
| 132 | |
| 133 | There is also a deprecated ``status`` field of type `DirtyBitmapStatus |
| 134 | <qemu-qmp-ref.html#index-DirtyBitmapStatus>`_. A bitmap historically had |
| 135 | five visible states: |
| 136 | |
| 137 | #. ``Frozen``: This bitmap is currently in-use by an operation and is |
| 138 | immutable. It can't be deleted, renamed, reset, etc. |
| 139 | |
| 140 | (This is now ``+busy``.) |
| 141 | |
| 142 | #. ``Disabled``: This bitmap is not recording new writes. |
| 143 | |
| 144 | (This is now ``-recording -busy``.) |
| 145 | |
| 146 | #. ``Active``: This bitmap is recording new writes. |
| 147 | |
| 148 | (This is now ``+recording -busy``.) |
| 149 | |
| 150 | #. ``Locked``: This bitmap is in-use by an operation, and is immutable. |
| 151 | The difference from "Frozen" was primarily implementation details. |
| 152 | |
| 153 | (This is now ``+busy``.) |
| 154 | |
| 155 | #. ``Inconsistent``: This persistent bitmap was not saved to disk |
| 156 | correctly, and can no longer be used. It remains in memory to serve as |
| 157 | an indicator of failure. |
| 158 | |
| 159 | (This is now ``+inconsistent``.) |
| 160 | |
| 161 | These states are directly replaced by the status indicators and should not be |
| 162 | used. The difference between ``Frozen`` and ``Locked`` is an implementation |
| 163 | detail and should not be relevant to external users. |
| 164 | |
| 165 | Basic QMP Usage |
| 166 | --------------- |
| 167 | |
| 168 | The primary interface to manipulating bitmap objects is via the QMP |
| 169 | interface. If you are not familiar, see the :doc:`qmp-spec` for the |
| 170 | protocol, and :doc:`qemu-qmp-ref` for a full reference of all QMP |
| 171 | commands. |
| 172 | |
| 173 | Supported Commands |
| 174 | ~~~~~~~~~~~~~~~~~~ |
| 175 | |
| 176 | There are six primary bitmap-management API commands: |
| 177 | |
| 178 | - ``block-dirty-bitmap-add`` |
| 179 | - ``block-dirty-bitmap-remove`` |
| 180 | - ``block-dirty-bitmap-clear`` |
| 181 | - ``block-dirty-bitmap-disable`` |
| 182 | - ``block-dirty-bitmap-enable`` |
| 183 | - ``block-dirty-bitmap-merge`` |
| 184 | |
| 185 | And one related query command: |
| 186 | |
| 187 | - ``query-block`` |
| 188 | |
| 189 | Creation: block-dirty-bitmap-add |
| 190 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 191 | |
| 192 | `block-dirty-bitmap-add |
| 193 | <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dadd>`_: |
| 194 | |
| 195 | Creates a new bitmap that tracks writes to the specified node. granularity, |
| 196 | persistence, and recording state can be adjusted at creation time. |
| 197 | |
| 198 | .. admonition:: Example |
| 199 | |
| 200 | to create a new, actively recording persistent bitmap: |
| 201 | |
| 202 | .. code-block:: QMP |
| 203 | |
| 204 | -> { "execute": "block-dirty-bitmap-add", |
| 205 | "arguments": { |
| 206 | "node": "drive0", |
| 207 | "name": "bitmap0", |
| 208 | "persistent": true, |
| 209 | } |
| 210 | } |
| 211 | |
| 212 | <- { "return": {} } |
| 213 | |
| 214 | - This bitmap will have a default granularity that matches the cluster size of |
| 215 | its associated drive, if available, clamped to between [4KiB, 64KiB]. The |
| 216 | current default for qcow2 is 64KiB. |
| 217 | |
| 218 | .. admonition:: Example |
| 219 | |
| 220 | To create a new, disabled (``-recording``), transient bitmap that tracks |
| 221 | changes in 32KiB segments: |
| 222 | |
| 223 | .. code-block:: QMP |
| 224 | |
| 225 | -> { "execute": "block-dirty-bitmap-add", |
| 226 | "arguments": { |
| 227 | "node": "drive0", |
| 228 | "name": "bitmap1", |
| 229 | "granularity": 32768, |
| 230 | "disabled": true |
| 231 | } |
| 232 | } |
| 233 | |
| 234 | <- { "return": {} } |
| 235 | |
| 236 | Deletion: block-dirty-bitmap-remove |
| 237 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 238 | |
| 239 | `block-dirty-bitmap-remove |
| 240 | <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dremove>`_: |
| 241 | |
| 242 | Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed. |
| 243 | |
| 244 | - Deleting a bitmap does not impact any other bitmaps attached to the same |
| 245 | node, nor does it affect any backups already created from this bitmap or |
| 246 | node. |
| 247 | |
| 248 | - Because bitmaps are only unique to the node to which they are attached, you |
| 249 | must specify the node/drive name here, too. |
| 250 | |
| 251 | - Deleting a persistent bitmap will remove it from the qcow2 file. |
| 252 | |
| 253 | .. admonition:: Example |
| 254 | |
| 255 | Remove a bitmap named ``bitmap0`` from node ``drive0``: |
| 256 | |
| 257 | .. code-block:: QMP |
| 258 | |
| 259 | -> { "execute": "block-dirty-bitmap-remove", |
| 260 | "arguments": { |
| 261 | "node": "drive0", |
| 262 | "name": "bitmap0" |
| 263 | } |
| 264 | } |
| 265 | |
| 266 | <- { "return": {} } |
| 267 | |
| 268 | Resetting: block-dirty-bitmap-clear |
| 269 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 270 | |
| 271 | `block-dirty-bitmap-clear |
| 272 | <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dclear>`_: |
| 273 | |
| 274 | Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared. |
| 275 | |
| 276 | - An incremental backup created from an empty bitmap will copy no data, as if |
| 277 | nothing has changed. |
| 278 | |
| 279 | .. admonition:: Example |
| 280 | |
| 281 | Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``: |
| 282 | |
| 283 | .. code-block:: QMP |
| 284 | |
| 285 | -> { "execute": "block-dirty-bitmap-clear", |
| 286 | "arguments": { |
| 287 | "node": "drive0", |
| 288 | "name": "bitmap0" |
| 289 | } |
| 290 | } |
| 291 | |
| 292 | <- { "return": {} } |
| 293 | |
| 294 | Enabling: block-dirty-bitmap-enable |
| 295 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 296 | |
| 297 | `block-dirty-bitmap-enable |
| 298 | <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002denable>`_: |
| 299 | |
| 300 | "Enables" a bitmap, setting the ``recording`` bit to true, causing writes to |
| 301 | begin being recorded. ``+busy`` bitmaps cannot be enabled. |
| 302 | |
| 303 | - Bitmaps default to being enabled when created, unless configured otherwise. |
| 304 | |
| 305 | - Persistent enabled bitmaps will remember their ``+recording`` status on |
| 306 | load. |
| 307 | |
| 308 | .. admonition:: Example |
| 309 | |
| 310 | To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``: |
| 311 | |
| 312 | .. code-block:: QMP |
| 313 | |
| 314 | -> { "execute": "block-dirty-bitmap-enable", |
| 315 | "arguments": { |
| 316 | "node": "drive0", |
| 317 | "name": "bitmap0" |
| 318 | } |
| 319 | } |
| 320 | |
| 321 | <- { "return": {} } |
| 322 | |
| 323 | Enabling: block-dirty-bitmap-disable |
| 324 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 325 | |
| 326 | `block-dirty-bitmap-disable |
| 327 | <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002ddisable>`_: |
| 328 | |
| 329 | "Disables" a bitmap, setting the ``recording`` bit to false, causing further |
| 330 | writes to begin being ignored. ``+busy`` bitmaps cannot be disabled. |
| 331 | |
| 332 | .. warning:: |
| 333 | |
| 334 | This is potentially dangerous: QEMU makes no effort to stop any writes if |
| 335 | there are disabled bitmaps on a node, and will not mark any disabled bitmaps |
| 336 | as ``+inconsistent`` if any such writes do happen. Backups made from such |
| 337 | bitmaps will not be able to be used to reconstruct a coherent image. |
| 338 | |
| 339 | - Disabling a bitmap may be useful for examining which sectors of a disk |
| 340 | changed during a specific time period, or for explicit management of |
| 341 | differential backup windows. |
| 342 | |
| 343 | - Persistent disabled bitmaps will remember their ``-recording`` status on |
| 344 | load. |
| 345 | |
| 346 | .. admonition:: Example |
| 347 | |
| 348 | To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``: |
| 349 | |
| 350 | .. code-block:: QMP |
| 351 | |
| 352 | -> { "execute": "block-dirty-bitmap-disable", |
| 353 | "arguments": { |
| 354 | "node": "drive0", |
| 355 | "name": "bitmap0" |
| 356 | } |
| 357 | } |
| 358 | |
| 359 | <- { "return": {} } |
| 360 | |
| 361 | Merging, Copying: block-dirty-bitmap-merge |
| 362 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 363 | |
| 364 | `block-dirty-bitmap-merge |
| 365 | <qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dmerge>`_: |
| 366 | |
| 367 | Merges one or more bitmaps into a target bitmap. For any segment that is dirty |
| 368 | in any one source bitmap, the target bitmap will mark that segment dirty. |
| 369 | |
| 370 | - Merge takes one or more bitmaps as a source and merges them together into a |
| 371 | single destination, such that any segment marked as dirty in any source |
| 372 | bitmap(s) will be marked dirty in the destination bitmap. |
| 373 | |
| 374 | - Merge does not create the destination bitmap if it does not exist. A blank |
| 375 | bitmap can be created beforehand to achieve the same effect. |
| 376 | |
| 377 | - The destination is not cleared prior to merge, so subsequent merge |
| 378 | operations will continue to cumulatively mark more segments as dirty. |
| 379 | |
| 380 | - If the merge operation should fail, the destination bitmap is guaranteed to |
| 381 | be unmodified. The operation may fail if the source or destination bitmaps |
| 382 | are busy, or have different granularities. |
| 383 | |
| 384 | - Bitmaps can only be merged on the same node. There is only one "node" |
| 385 | argument, so all bitmaps must be attached to that same node. |
| 386 | |
| 387 | - Copy can be achieved by merging from a single source to an empty |
| 388 | destination. |
| 389 | |
| 390 | .. admonition:: Example |
| 391 | |
| 392 | Merge the data from ``bitmap0`` into the bitmap ``new_bitmap`` on node |
| 393 | ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves |
| 394 | a copy. |
| 395 | |
| 396 | .. code-block:: QMP |
| 397 | |
| 398 | -> { "execute": "block-dirty-bitmap-merge", |
| 399 | "arguments": { |
| 400 | "node": "drive0", |
| 401 | "target": "new_bitmap", |
| 402 | "bitmaps": [ "bitmap0" ] |
| 403 | } |
| 404 | } |
| 405 | |
| 406 | <- { "return": {} } |
| 407 | |
| 408 | Querying: query-block |
| 409 | ~~~~~~~~~~~~~~~~~~~~~ |
| 410 | |
| 411 | `query-block |
| 412 | <qemu-qmp-ref.html#index-query_002dblock>`_: |
| 413 | |
| 414 | Not strictly a bitmaps command, but will return information about any bitmaps |
| 415 | attached to nodes serving as the root for guest devices. |
| 416 | |
| 417 | - The "inconsistent" bit will not appear when it is false, appearing only when |
| 418 | the value is true to indicate there is a problem. |
| 419 | |
| 420 | .. admonition:: Example |
| 421 | |
| 422 | Query the block sub-system of QEMU. The following json has trimmed irrelevant |
| 423 | keys from the response to highlight only the bitmap-relevant portions of the |
| 424 | API. This result highlights a bitmap ``bitmap0`` attached to the root node of |
| 425 | device ``drive0``. |
| 426 | |
| 427 | .. code-block:: QMP |
| 428 | |
| 429 | -> { |
| 430 | "execute": "query-block", |
| 431 | "arguments": {} |
| 432 | } |
| 433 | |
| 434 | <- { |
| 435 | "return": [ { |
| 436 | "dirty-bitmaps": [ { |
| 437 | "status": "active", |
| 438 | "count": 0, |
| 439 | "busy": false, |
| 440 | "name": "bitmap0", |
| 441 | "persistent": false, |
| 442 | "recording": true, |
| 443 | "granularity": 65536 |
| 444 | } ], |
| 445 | "device": "drive0", |
| 446 | } ] |
| 447 | } |
| 448 | |
| 449 | Bitmap Persistence |
| 450 | ------------------ |
| 451 | |
| 452 | As outlined in `Supported Image Formats`_, QEMU can persist bitmaps to qcow2 |
| 453 | files. Demonstrated in `Creation: block-dirty-bitmap-add`_, passing |
| 454 | ``persistent: true`` to ``block-dirty-bitmap-add`` will persist that bitmap to |
| 455 | disk. |
| 456 | |
| 457 | Persistent bitmaps will be automatically loaded into memory upon load, and |
| 458 | will be written back to disk upon close. Their usage should be mostly |
| 459 | transparent. |
| 460 | |
| 461 | However, if QEMU does not get a chance to close the file cleanly, the bitmap |
| 462 | will be marked as ``+inconsistent`` at next load and considered unsafe to use |
| 463 | for any operation. At this point, the only valid operation on such bitmaps is |
| 464 | ``block-dirty-bitmap-remove``. |
| 465 | |
| 466 | Losing a bitmap in this way does not invalidate any existing backups that have |
| 467 | been made from this bitmap, but no further backups will be able to be issued |
| 468 | for this chain. |
| 469 | |
| 470 | Transactions |
| 471 | ------------ |
| 472 | |
| 473 | Transactions are a QMP feature that allows you to submit multiple QMP commands |
| 474 | at once, being guaranteed that they will all succeed or fail atomically, |
| 475 | together. The interaction of bitmaps and transactions are demonstrated below. |
| 476 | |
| 477 | See `transaction <qemu-qmp.ref.html#index-transaction>`_ in the QMP reference |
| 478 | for more details. |
| 479 | |
| 480 | Justification |
| 481 | ~~~~~~~~~~~~~ |
| 482 | |
| 483 | Bitmaps can generally be modified at any time, but certain operations often |
| 484 | only make sense when paired directly with other commands. When a VM is paused, |
| 485 | it's easy to ensure that no guest writes occur between individual QMP |
| 486 | commands. When a VM is running, this is difficult to accomplish with |
| 487 | individual QMP commands that may allow guest writes to occur between each |
| 488 | command. |
| 489 | |
| 490 | For example, using only individual QMP commands, we could: |
| 491 | |
| 492 | #. Boot the VM in a paused state. |
| 493 | #. Create a full drive backup of drive0. |
| 494 | #. Create a new bitmap attached to drive0, confident that nothing has been |
| 495 | written to drive0 in the meantime. |
| 496 | #. Resume execution of the VM. |
| 497 | #. At a later point, issue incremental backups from ``bitmap0``. |
| 498 | |
| 499 | At this point, the bitmap and drive backup would be correctly in sync, and |
| 500 | incremental backups made from this point forward would be correctly aligned to |
| 501 | the full drive backup. |
| 502 | |
| 503 | This is not particularly useful if we decide we want to start incremental |
| 504 | backups after the VM has been running for a while, for which we would want to |
| 505 | perform actions such as the following: |
| 506 | |
| 507 | #. Boot the VM and begin execution. |
| 508 | #. Using a single transaction, perform the following operations: |
| 509 | |
| 510 | - Create ``bitmap0``. |
| 511 | - Create a full drive backup of ``drive0``. |
| 512 | |
| 513 | #. At a later point, issue incremental backups from ``bitmap0``. |
| 514 | |
| 515 | .. note:: As a consideration, if ``bitmap0`` is created prior to the full |
| 516 | drive backup, incremental backups can still be authored from this |
| 517 | bitmap, but they will copy extra segments reflecting writes that |
| 518 | occurred prior to the backup operation. Transactions allow us to |
| 519 | narrow critical points in time to reduce waste, or, in the other |
| 520 | direction, to ensure that no segments are omitted. |
| 521 | |
| 522 | Supported Bitmap Transactions |
| 523 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 524 | |
| 525 | - ``block-dirty-bitmap-add`` |
| 526 | - ``block-dirty-bitmap-clear`` |
| 527 | - ``block-dirty-bitmap-enable`` |
| 528 | - ``block-dirty-bitmap-disable`` |
| 529 | - ``block-dirty-bitmap-merge`` |
| 530 | |
| 531 | The usages for these commands are identical to their respective QMP commands, |
| 532 | but see the sections below for concrete examples. |
| 533 | |
| 534 | Incremental Backups - Push Model |
| 535 | -------------------------------- |
| 536 | |
| 537 | Incremental backups are simply partial disk images that can be combined with |
| 538 | other partial disk images on top of a base image to reconstruct a full backup |
| 539 | from the point in time at which the incremental backup was issued. |
| 540 | |
| 541 | The "Push Model" here references the fact that QEMU is "pushing" the modified |
| 542 | blocks out to a destination. We will be using the `blockdev-backup |
| 543 | <qemu-qmp-ref.html#index-blockdev_002dbackup>`_ QMP command to create both |
| 544 | full and incremental backups. |
| 545 | |
| 546 | The command is a background job, which has its own QMP API for querying and |
| 547 | management documented in `Background jobs |
| 548 | <qemu-qmp-ref.html#Background-jobs>`_. |
| 549 | |
| 550 | Example: New Incremental Backup Anchor Point |
| 551 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 552 | |
| 553 | As outlined in the Transactions - `Justification`_ section, perhaps we want to |
| 554 | create a new incremental backup chain attached to a drive. |
| 555 | |
| 556 | This example creates a new, full backup of "drive0" and accompanies it with a |
| 557 | new, empty bitmap that records writes from this point in time forward. |
| 558 | |
| 559 | The target can be created with the help of `blockdev-add |
| 560 | <qemu-qmp-ref.html#index-blockdev_002dadd>`_ or `blockdev-create |
| 561 | <qemu-qmp-ref.html#index-blockdev_002dcreate>`_ command. |
| 562 | |
| 563 | .. note:: Any new writes that happen after this command is issued, even while |
| 564 | the backup job runs, will be written locally and not to the backup |
| 565 | destination. These writes will be recorded in the bitmap |
| 566 | accordingly. |
| 567 | |
| 568 | .. code-block:: QMP |
| 569 | |
| 570 | -> { |
| 571 | "execute": "transaction", |
| 572 | "arguments": { |
| 573 | "actions": [ |
| 574 | { |
| 575 | "type": "block-dirty-bitmap-add", |
| 576 | "data": { |
| 577 | "node": "drive0", |
| 578 | "name": "bitmap0" |
| 579 | } |
| 580 | }, |
| 581 | { |
| 582 | "type": "blockdev-backup", |
| 583 | "data": { |
| 584 | "device": "drive0", |
| 585 | "target": "target0", |
| 586 | "sync": "full" |
| 587 | } |
| 588 | } |
| 589 | ] |
| 590 | } |
| 591 | } |
| 592 | |
| 593 | <- { "return": {} } |
| 594 | |
| 595 | <- { |
| 596 | "timestamp": { |
| 597 | "seconds": 1555436945, |
| 598 | "microseconds": 179620 |
| 599 | }, |
| 600 | "data": { |
| 601 | "status": "created", |
| 602 | "id": "drive0" |
| 603 | }, |
| 604 | "event": "JOB_STATUS_CHANGE" |
| 605 | } |
| 606 | |
| 607 | ... |
| 608 | |
| 609 | <- { |
| 610 | "timestamp": {...}, |
| 611 | "data": { |
| 612 | "device": "drive0", |
| 613 | "type": "backup", |
| 614 | "speed": 0, |
| 615 | "len": 68719476736, |
| 616 | "offset": 68719476736 |
| 617 | }, |
| 618 | "event": "BLOCK_JOB_COMPLETED" |
| 619 | } |
| 620 | |
| 621 | <- { |
| 622 | "timestamp": {...}, |
| 623 | "data": { |
| 624 | "status": "concluded", |
| 625 | "id": "drive0" |
| 626 | }, |
| 627 | "event": "JOB_STATUS_CHANGE" |
| 628 | } |
| 629 | |
| 630 | <- { |
| 631 | "timestamp": {...}, |
| 632 | "data": { |
| 633 | "status": "null", |
| 634 | "id": "drive0" |
| 635 | }, |
| 636 | "event": "JOB_STATUS_CHANGE" |
| 637 | } |
| 638 | |
| 639 | A full explanation of the job transition semantics and the JOB_STATUS_CHANGE |
| 640 | event are beyond the scope of this document and will be omitted in all |
| 641 | subsequent examples; above, several more events have been omitted for brevity. |
| 642 | |
| 643 | .. note:: Subsequent examples will omit all events except BLOCK_JOB_COMPLETED |
| 644 | except where necessary to illustrate workflow differences. |
| 645 | |
| 646 | Omitted events and json objects will be represented by ellipses: |
| 647 | ``...`` |
| 648 | |
| 649 | Example: Resetting an Incremental Backup Anchor Point |
| 650 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 651 | |
| 652 | If we want to start a new backup chain with an existing bitmap, we can also |
| 653 | use a transaction to reset the bitmap while making a new full backup: |
| 654 | |
| 655 | .. code-block:: QMP |
| 656 | |
| 657 | -> { |
| 658 | "execute": "transaction", |
| 659 | "arguments": { |
| 660 | "actions": [ |
| 661 | { |
| 662 | "type": "block-dirty-bitmap-clear", |
| 663 | "data": { |
| 664 | "node": "drive0", |
| 665 | "name": "bitmap0" |
| 666 | } |
| 667 | }, |
| 668 | { |
| 669 | "type": "blockdev-backup", |
| 670 | "data": { |
| 671 | "device": "drive0", |
| 672 | "target": "target0", |
| 673 | "sync": "full" |
| 674 | } |
| 675 | } |
| 676 | ] |
| 677 | } |
| 678 | } |
| 679 | |
| 680 | <- { "return": {} } |
| 681 | |
| 682 | ... |
| 683 | |
| 684 | <- { |
| 685 | "timestamp": {...}, |
| 686 | "data": { |
| 687 | "device": "drive0", |
| 688 | "type": "backup", |
| 689 | "speed": 0, |
| 690 | "len": 68719476736, |
| 691 | "offset": 68719476736 |
| 692 | }, |
| 693 | "event": "BLOCK_JOB_COMPLETED" |
| 694 | } |
| 695 | |
| 696 | ... |
| 697 | |
| 698 | The result of this example is identical to the first, but we clear an existing |
| 699 | bitmap instead of adding a new one. |
| 700 | |
| 701 | .. tip:: In both of these examples, "bitmap0" is tied conceptually to the |
| 702 | creation of new, full backups. This relationship is not saved or |
| 703 | remembered by QEMU; it is up to the operator or management layer to |
| 704 | remember which bitmaps are associated with which backups. |
| 705 | |
| 706 | Example: First Incremental Backup |
| 707 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 708 | |
| 709 | #. Create a full backup and sync it to a dirty bitmap using any method: |
| 710 | |
| 711 | - Either of the two live backup method demonstrated above, |
| 712 | - Using QMP commands with the VM paused as in the `Justification`_ section, |
| 713 | or |
| 714 | - With the VM offline, manually copy the image and start the VM in a paused |
| 715 | state, careful to add a new bitmap before the VM begins execution. |
| 716 | |
| 717 | Whichever method is chosen, let's assume that at the end of this step: |
| 718 | |
| 719 | - The full backup is named ``drive0.full.qcow2``. |
| 720 | - The bitmap we created is named ``bitmap0``, attached to ``drive0``. |
| 721 | |
| 722 | #. Create a destination image for the incremental backup that utilizes the |
| 723 | full backup as a backing image. |
| 724 | |
| 725 | - Let's assume the new incremental image is named ``drive0.inc0.qcow2``: |
| 726 | |
| 727 | .. code:: bash |
| 728 | |
| 729 | $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ |
| 730 | -b drive0.full.qcow2 -F qcow2 |
| 731 | |
| 732 | #. Add target block node: |
| 733 | |
| 734 | .. code-block:: QMP |
| 735 | |
| 736 | -> { |
| 737 | "execute": "blockdev-add", |
| 738 | "arguments": { |
| 739 | "node-name": "target0", |
| 740 | "driver": "qcow2", |
| 741 | "file": { |
| 742 | "driver": "file", |
| 743 | "filename": "drive0.inc0.qcow2" |
| 744 | } |
| 745 | } |
| 746 | } |
| 747 | |
| 748 | <- { "return": {} } |
| 749 | |
| 750 | #. Issue an incremental backup command: |
| 751 | |
| 752 | .. code-block:: QMP |
| 753 | |
| 754 | -> { |
| 755 | "execute": "blockdev-backup", |
| 756 | "arguments": { |
| 757 | "device": "drive0", |
| 758 | "bitmap": "bitmap0", |
| 759 | "target": "target0", |
| 760 | "sync": "incremental" |
| 761 | } |
| 762 | } |
| 763 | |
| 764 | <- { "return": {} } |
| 765 | |
| 766 | ... |
| 767 | |
| 768 | <- { |
| 769 | "timestamp": {...}, |
| 770 | "data": { |
| 771 | "device": "drive0", |
| 772 | "type": "backup", |
| 773 | "speed": 0, |
| 774 | "len": 68719476736, |
| 775 | "offset": 68719476736 |
| 776 | }, |
| 777 | "event": "BLOCK_JOB_COMPLETED" |
| 778 | } |
| 779 | |
| 780 | ... |
| 781 | |
| 782 | This copies any blocks modified since the full backup was created into the |
| 783 | ``drive0.inc0.qcow2`` file. During the operation, ``bitmap0`` is marked |
| 784 | ``+busy``. If the operation is successful, ``bitmap0`` will be cleared to |
| 785 | reflect the "incremental" backup regimen, which only copies out new changes |
| 786 | from each incremental backup. |
| 787 | |
| 788 | .. note:: Any new writes that occur after the backup operation starts do not |
| 789 | get copied to the destination. The backup's "point in time" is when |
| 790 | the backup starts, not when it ends. These writes are recorded in a |
| 791 | special bitmap that gets re-added to bitmap0 when the backup ends so |
| 792 | that the next incremental backup can copy them out. |
| 793 | |
| 794 | Example: Second Incremental Backup |
| 795 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 796 | |
| 797 | #. Create a new destination image for the incremental backup that points to |
| 798 | the previous one, e.g.: ``drive0.inc1.qcow2`` |
| 799 | |
| 800 | .. code:: bash |
| 801 | |
| 802 | $ qemu-img create -f qcow2 drive0.inc1.qcow2 \ |
| 803 | -b drive0.inc0.qcow2 -F qcow2 |
| 804 | |
| 805 | #. Add target block node: |
| 806 | |
| 807 | .. code-block:: QMP |
| 808 | |
| 809 | -> { |
| 810 | "execute": "blockdev-add", |
| 811 | "arguments": { |
| 812 | "node-name": "target0", |
| 813 | "driver": "qcow2", |
| 814 | "file": { |
| 815 | "driver": "file", |
| 816 | "filename": "drive0.inc1.qcow2" |
| 817 | } |
| 818 | } |
| 819 | } |
| 820 | |
| 821 | <- { "return": {} } |
| 822 | |
| 823 | #. Issue a new incremental backup command. The only difference here is that we |
| 824 | have changed the target image below. |
| 825 | |
| 826 | .. code-block:: QMP |
| 827 | |
| 828 | -> { |
| 829 | "execute": "blockdev-backup", |
| 830 | "arguments": { |
| 831 | "device": "drive0", |
| 832 | "bitmap": "bitmap0", |
| 833 | "target": "target0", |
| 834 | "sync": "incremental" |
| 835 | } |
| 836 | } |
| 837 | |
| 838 | <- { "return": {} } |
| 839 | |
| 840 | ... |
| 841 | |
| 842 | <- { |
| 843 | "timestamp": {...}, |
| 844 | "data": { |
| 845 | "device": "drive0", |
| 846 | "type": "backup", |
| 847 | "speed": 0, |
| 848 | "len": 68719476736, |
| 849 | "offset": 68719476736 |
| 850 | }, |
| 851 | "event": "BLOCK_JOB_COMPLETED" |
| 852 | } |
| 853 | |
| 854 | ... |
| 855 | |
| 856 | Because the first incremental backup from the previous example completed |
| 857 | successfully, ``bitmap0`` was synchronized with ``drive0.inc0.qcow2``. Here, |
| 858 | we use ``bitmap0`` again to create a new incremental backup that targets the |
| 859 | previous one, creating a chain of three images: |
| 860 | |
| 861 | .. admonition:: Diagram |
| 862 | |
| 863 | .. code:: text |
| 864 | |
| 865 | +-------------------+ +-------------------+ +-------------------+ |
| 866 | | drive0.full.qcow2 |<--| drive0.inc0.qcow2 |<--| drive0.inc1.qcow2 | |
| 867 | +-------------------+ +-------------------+ +-------------------+ |
| 868 | |
| 869 | Each new incremental backup re-synchronizes the bitmap to the latest backup |
| 870 | authored, allowing a user to continue to "consume" it to create new backups on |
| 871 | top of an existing chain. |
| 872 | |
| 873 | In the above diagram, neither drive0.inc1.qcow2 nor drive0.inc0.qcow2 are |
| 874 | complete images by themselves, but rely on their backing chain to reconstruct |
| 875 | a full image. The dependency terminates with each full backup. |
| 876 | |
| 877 | Each backup in this chain remains independent, and is unchanged by new entries |
| 878 | made later in the chain. For instance, drive0.inc0.qcow2 remains a perfectly |
| 879 | valid backup of the disk as it was when that backup was issued. |
| 880 | |
| 881 | Example: Incremental Push Backups without Backing Files |
| 882 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 883 | |
| 884 | Backup images are best kept off-site, so we often will not have the preceding |
| 885 | backups in a chain available to link against. This is not a problem at backup |
| 886 | time; we simply do not set the backing image when creating the destination |
| 887 | image: |
| 888 | |
| 889 | #. Create a new destination image with no backing file set. We will need to |
| 890 | specify the size of the base image, because the backing file isn't |
| 891 | available for QEMU to use to determine it. |
| 892 | |
| 893 | .. code:: bash |
| 894 | |
| 895 | $ qemu-img create -f qcow2 drive0.inc2.qcow2 64G |
| 896 | |
| 897 | .. note:: Alternatively, you can omit ``mode: "existing"`` from the push |
| 898 | backup commands to have QEMU create an image without a backing |
| 899 | file for you, but you lose control over format options like |
| 900 | compatibility and preallocation presets. |
| 901 | |
| 902 | #. Add target block node: |
| 903 | |
| 904 | .. code-block:: QMP |
| 905 | |
| 906 | -> { |
| 907 | "execute": "blockdev-add", |
| 908 | "arguments": { |
| 909 | "node-name": "target0", |
| 910 | "driver": "qcow2", |
| 911 | "file": { |
| 912 | "driver": "file", |
| 913 | "filename": "drive0.inc2.qcow2" |
| 914 | } |
| 915 | } |
| 916 | } |
| 917 | |
| 918 | <- { "return": {} } |
| 919 | |
| 920 | #. Issue a new incremental backup command. Apart from the new destination |
| 921 | image, there is no difference from the last two examples. |
| 922 | |
| 923 | .. code-block:: QMP |
| 924 | |
| 925 | -> { |
| 926 | "execute": "blockdev-backup", |
| 927 | "arguments": { |
| 928 | "device": "drive0", |
| 929 | "bitmap": "bitmap0", |
| 930 | "target": "target0", |
| 931 | "sync": "incremental" |
| 932 | } |
| 933 | } |
| 934 | |
| 935 | <- { "return": {} } |
| 936 | |
| 937 | ... |
| 938 | |
| 939 | <- { |
| 940 | "timestamp": {...}, |
| 941 | "data": { |
| 942 | "device": "drive0", |
| 943 | "type": "backup", |
| 944 | "speed": 0, |
| 945 | "len": 68719476736, |
| 946 | "offset": 68719476736 |
| 947 | }, |
| 948 | "event": "BLOCK_JOB_COMPLETED" |
| 949 | } |
| 950 | |
| 951 | ... |
| 952 | |
| 953 | The only difference from the perspective of the user is that you will need to |
| 954 | set the backing image when attempting to restore the backup: |
| 955 | |
| 956 | .. code:: bash |
| 957 | |
| 958 | $ qemu-img rebase drive0.inc2.qcow2 \ |
| 959 | -u -b drive0.inc1.qcow2 |
| 960 | |
| 961 | This uses the "unsafe" rebase mode to simply set the backing file to a file |
| 962 | that isn't present. |
| 963 | |
| 964 | It is also possible to use ``--image-opts`` to specify the entire backing |
| 965 | chain by hand as an ephemeral property at runtime, but that is beyond the |
| 966 | scope of this document. |
| 967 | |
| 968 | Example: Multi-drive Incremental Backup |
| 969 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 970 | |
| 971 | Assume we have a VM with two drives, "drive0" and "drive1" and we wish to back |
| 972 | both of them up such that the two backups represent the same crash-consistent |
| 973 | point in time. |
| 974 | |
| 975 | #. For each drive, create an empty image: |
| 976 | |
| 977 | .. code:: bash |
| 978 | |
| 979 | $ qemu-img create -f qcow2 drive0.full.qcow2 64G |
| 980 | $ qemu-img create -f qcow2 drive1.full.qcow2 64G |
| 981 | |
| 982 | #. Add target block nodes: |
| 983 | |
| 984 | .. code-block:: QMP |
| 985 | |
| 986 | -> { |
| 987 | "execute": "blockdev-add", |
| 988 | "arguments": { |
| 989 | "node-name": "target0", |
| 990 | "driver": "qcow2", |
| 991 | "file": { |
| 992 | "driver": "file", |
| 993 | "filename": "drive0.full.qcow2" |
| 994 | } |
| 995 | } |
| 996 | } |
| 997 | |
| 998 | <- { "return": {} } |
| 999 | |
| 1000 | -> { |
| 1001 | "execute": "blockdev-add", |
| 1002 | "arguments": { |
| 1003 | "node-name": "target1", |
| 1004 | "driver": "qcow2", |
| 1005 | "file": { |
| 1006 | "driver": "file", |
| 1007 | "filename": "drive1.full.qcow2" |
| 1008 | } |
| 1009 | } |
| 1010 | } |
| 1011 | |
| 1012 | <- { "return": {} } |
| 1013 | |
| 1014 | #. Create a full (anchor) backup for each drive, with accompanying bitmaps: |
| 1015 | |
| 1016 | .. code-block:: QMP |
| 1017 | |
| 1018 | -> { |
| 1019 | "execute": "transaction", |
| 1020 | "arguments": { |
| 1021 | "actions": [ |
| 1022 | { |
| 1023 | "type": "block-dirty-bitmap-add", |
| 1024 | "data": { |
| 1025 | "node": "drive0", |
| 1026 | "name": "bitmap0" |
| 1027 | } |
| 1028 | }, |
| 1029 | { |
| 1030 | "type": "block-dirty-bitmap-add", |
| 1031 | "data": { |
| 1032 | "node": "drive1", |
| 1033 | "name": "bitmap0" |
| 1034 | } |
| 1035 | }, |
| 1036 | { |
| 1037 | "type": "blockdev-backup", |
| 1038 | "data": { |
| 1039 | "device": "drive0", |
| 1040 | "target": "target0", |
| 1041 | "sync": "full" |
| 1042 | } |
| 1043 | }, |
| 1044 | { |
| 1045 | "type": "blockdev-backup", |
| 1046 | "data": { |
| 1047 | "device": "drive1", |
| 1048 | "target": "target1", |
| 1049 | "sync": "full" |
| 1050 | } |
| 1051 | } |
| 1052 | ] |
| 1053 | } |
| 1054 | } |
| 1055 | |
| 1056 | <- { "return": {} } |
| 1057 | |
| 1058 | ... |
| 1059 | |
| 1060 | <- { |
| 1061 | "timestamp": {...}, |
| 1062 | "data": { |
| 1063 | "device": "drive0", |
| 1064 | "type": "backup", |
| 1065 | "speed": 0, |
| 1066 | "len": 68719476736, |
| 1067 | "offset": 68719476736 |
| 1068 | }, |
| 1069 | "event": "BLOCK_JOB_COMPLETED" |
| 1070 | } |
| 1071 | |
| 1072 | ... |
| 1073 | |
| 1074 | <- { |
| 1075 | "timestamp": {...}, |
| 1076 | "data": { |
| 1077 | "device": "drive1", |
| 1078 | "type": "backup", |
| 1079 | "speed": 0, |
| 1080 | "len": 68719476736, |
| 1081 | "offset": 68719476736 |
| 1082 | }, |
| 1083 | "event": "BLOCK_JOB_COMPLETED" |
| 1084 | } |
| 1085 | |
| 1086 | ... |
| 1087 | |
| 1088 | #. Later, create new destination images for each of the incremental backups |
| 1089 | that point to their respective full backups: |
| 1090 | |
| 1091 | .. code:: bash |
| 1092 | |
| 1093 | $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ |
| 1094 | -b drive0.full.qcow2 -F qcow2 |
| 1095 | $ qemu-img create -f qcow2 drive1.inc0.qcow2 \ |
| 1096 | -b drive1.full.qcow2 -F qcow2 |
| 1097 | |
| 1098 | #. Add target block nodes: |
| 1099 | |
| 1100 | .. code-block:: QMP |
| 1101 | |
| 1102 | -> { |
| 1103 | "execute": "blockdev-add", |
| 1104 | "arguments": { |
| 1105 | "node-name": "target0", |
| 1106 | "driver": "qcow2", |
| 1107 | "file": { |
| 1108 | "driver": "file", |
| 1109 | "filename": "drive0.inc0.qcow2" |
| 1110 | } |
| 1111 | } |
| 1112 | } |
| 1113 | |
| 1114 | <- { "return": {} } |
| 1115 | |
| 1116 | -> { |
| 1117 | "execute": "blockdev-add", |
| 1118 | "arguments": { |
| 1119 | "node-name": "target1", |
| 1120 | "driver": "qcow2", |
| 1121 | "file": { |
| 1122 | "driver": "file", |
| 1123 | "filename": "drive1.inc0.qcow2" |
| 1124 | } |
| 1125 | } |
| 1126 | } |
| 1127 | |
| 1128 | <- { "return": {} } |
| 1129 | |
| 1130 | #. Issue a multi-drive incremental push backup transaction: |
| 1131 | |
| 1132 | .. code-block:: QMP |
| 1133 | |
| 1134 | -> { |
| 1135 | "execute": "transaction", |
| 1136 | "arguments": { |
| 1137 | "actions": [ |
| 1138 | { |
| 1139 | "type": "blockev-backup", |
| 1140 | "data": { |
| 1141 | "device": "drive0", |
| 1142 | "bitmap": "bitmap0", |
| 1143 | "sync": "incremental", |
| 1144 | "target": "target0" |
| 1145 | } |
| 1146 | }, |
| 1147 | { |
| 1148 | "type": "blockdev-backup", |
| 1149 | "data": { |
| 1150 | "device": "drive1", |
| 1151 | "bitmap": "bitmap0", |
| 1152 | "sync": "incremental", |
| 1153 | "target": "target1" |
| 1154 | } |
| 1155 | }, |
| 1156 | ] |
| 1157 | } |
| 1158 | } |
| 1159 | |
| 1160 | <- { "return": {} } |
| 1161 | |
| 1162 | ... |
| 1163 | |
| 1164 | <- { |
| 1165 | "timestamp": {...}, |
| 1166 | "data": { |
| 1167 | "device": "drive0", |
| 1168 | "type": "backup", |
| 1169 | "speed": 0, |
| 1170 | "len": 68719476736, |
| 1171 | "offset": 68719476736 |
| 1172 | }, |
| 1173 | "event": "BLOCK_JOB_COMPLETED" |
| 1174 | } |
| 1175 | |
| 1176 | ... |
| 1177 | |
| 1178 | <- { |
| 1179 | "timestamp": {...}, |
| 1180 | "data": { |
| 1181 | "device": "drive1", |
| 1182 | "type": "backup", |
| 1183 | "speed": 0, |
| 1184 | "len": 68719476736, |
| 1185 | "offset": 68719476736 |
| 1186 | }, |
| 1187 | "event": "BLOCK_JOB_COMPLETED" |
| 1188 | } |
| 1189 | |
| 1190 | ... |
| 1191 | |
| 1192 | Push Backup Errors & Recovery |
| 1193 | ----------------------------- |
| 1194 | |
| 1195 | In the event of an error that occurs after a push backup job is successfully |
| 1196 | launched, either by an individual QMP command or a QMP transaction, the user |
| 1197 | will receive a ``BLOCK_JOB_COMPLETE`` event with a failure message, |
| 1198 | accompanied by a ``BLOCK_JOB_ERROR`` event. |
| 1199 | |
| 1200 | In the case of a job being cancelled, the user will receive a |
| 1201 | ``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR |
| 1202 | events. |
| 1203 | |
| 1204 | In either failure case, the bitmap used for the failed operation is not |
| 1205 | cleared. It will contain all of the dirty bits it did at the start of the |
| 1206 | operation, plus any new bits that got marked during the operation. |
| 1207 | |
| 1208 | Effectively, the "point in time" that a bitmap is recording differences |
| 1209 | against is kept at the issuance of the last successful incremental backup, |
| 1210 | instead of being moved forward to the start of this now-failed backup. |
| 1211 | |
| 1212 | Once the underlying problem is addressed (e.g. more storage space is allocated |
| 1213 | on the destination), the incremental backup command can be retried with the |
| 1214 | same bitmap. |
| 1215 | |
| 1216 | Example: Individual Failures |
| 1217 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 1218 | |
| 1219 | Incremental Push Backup jobs that fail individually behave simply as |
| 1220 | described above. This example demonstrates the single-job failure case: |
| 1221 | |
| 1222 | #. Create a target image: |
| 1223 | |
| 1224 | .. code:: bash |
| 1225 | |
| 1226 | $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ |
| 1227 | -b drive0.full.qcow2 -F qcow2 |
| 1228 | |
| 1229 | #. Add target block node: |
| 1230 | |
| 1231 | .. code-block:: QMP |
| 1232 | |
| 1233 | -> { |
| 1234 | "execute": "blockdev-add", |
| 1235 | "arguments": { |
| 1236 | "node-name": "target0", |
| 1237 | "driver": "qcow2", |
| 1238 | "file": { |
| 1239 | "driver": "file", |
| 1240 | "filename": "drive0.inc0.qcow2" |
| 1241 | } |
| 1242 | } |
| 1243 | } |
| 1244 | |
| 1245 | <- { "return": {} } |
| 1246 | |
| 1247 | #. Attempt to create an incremental backup via QMP: |
| 1248 | |
| 1249 | .. code-block:: QMP |
| 1250 | |
| 1251 | -> { |
| 1252 | "execute": "blockdev-backup", |
| 1253 | "arguments": { |
| 1254 | "device": "drive0", |
| 1255 | "bitmap": "bitmap0", |
| 1256 | "target": "target0", |
| 1257 | "sync": "incremental" |
| 1258 | } |
| 1259 | } |
| 1260 | |
| 1261 | <- { "return": {} } |
| 1262 | |
| 1263 | #. Receive a pair of events indicating failure: |
| 1264 | |
| 1265 | .. code-block:: QMP |
| 1266 | |
| 1267 | <- { |
| 1268 | "timestamp": {...}, |
| 1269 | "data": { |
| 1270 | "device": "drive0", |
| 1271 | "action": "report", |
| 1272 | "operation": "write" |
| 1273 | }, |
| 1274 | "event": "BLOCK_JOB_ERROR" |
| 1275 | } |
| 1276 | |
| 1277 | <- { |
| 1278 | "timestamp": {...}, |
| 1279 | "data": { |
| 1280 | "speed": 0, |
| 1281 | "offset": 0, |
| 1282 | "len": 67108864, |
| 1283 | "error": "No space left on device", |
| 1284 | "device": "drive0", |
| 1285 | "type": "backup" |
| 1286 | }, |
| 1287 | "event": "BLOCK_JOB_COMPLETED" |
| 1288 | } |
| 1289 | |
| 1290 | #. Remove target node: |
| 1291 | |
| 1292 | .. code-block:: QMP |
| 1293 | |
| 1294 | -> { |
| 1295 | "execute": "blockdev-del", |
| 1296 | "arguments": { |
| 1297 | "node-name": "target0", |
| 1298 | } |
| 1299 | } |
| 1300 | |
| 1301 | <- { "return": {} } |
| 1302 | |
| 1303 | #. Delete the failed image, and re-create it. |
| 1304 | |
| 1305 | .. code:: bash |
| 1306 | |
| 1307 | $ rm drive0.inc0.qcow2 |
| 1308 | $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ |
| 1309 | -b drive0.full.qcow2 -F qcow2 |
| 1310 | |
| 1311 | #. Add target block node: |
| 1312 | |
| 1313 | .. code-block:: QMP |
| 1314 | |
| 1315 | -> { |
| 1316 | "execute": "blockdev-add", |
| 1317 | "arguments": { |
| 1318 | "node-name": "target0", |
| 1319 | "driver": "qcow2", |
| 1320 | "file": { |
| 1321 | "driver": "file", |
| 1322 | "filename": "drive0.inc0.qcow2" |
| 1323 | } |
| 1324 | } |
| 1325 | } |
| 1326 | |
| 1327 | <- { "return": {} } |
| 1328 | |
| 1329 | #. Retry the command after fixing the underlying problem, such as |
| 1330 | freeing up space on the backup volume: |
| 1331 | |
| 1332 | .. code-block:: QMP |
| 1333 | |
| 1334 | -> { |
| 1335 | "execute": "blockdev-backup", |
| 1336 | "arguments": { |
| 1337 | "device": "drive0", |
| 1338 | "bitmap": "bitmap0", |
| 1339 | "target": "target0", |
| 1340 | "sync": "incremental" |
| 1341 | } |
| 1342 | } |
| 1343 | |
| 1344 | <- { "return": {} } |
| 1345 | |
| 1346 | #. Receive confirmation that the job completed successfully: |
| 1347 | |
| 1348 | .. code-block:: QMP |
| 1349 | |
| 1350 | <- { |
| 1351 | "timestamp": {...}, |
| 1352 | "data": { |
| 1353 | "device": "drive0", |
| 1354 | "type": "backup", |
| 1355 | "speed": 0, |
| 1356 | "len": 67108864, |
| 1357 | "offset": 67108864 |
| 1358 | }, |
| 1359 | "event": "BLOCK_JOB_COMPLETED" |
| 1360 | } |
| 1361 | |
| 1362 | Example: Partial Transactional Failures |
| 1363 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 1364 | |
| 1365 | QMP commands like `blockdev-backup |
| 1366 | <qemu-qmp-ref.html#index-blockdev_002dbackup>`_ |
| 1367 | conceptually only start a job, and so transactions containing these commands |
| 1368 | may succeed even if the job it created later fails. This might have surprising |
| 1369 | interactions with notions of how a "transaction" ought to behave. |
| 1370 | |
| 1371 | This distinction means that on occasion, a transaction containing such job |
| 1372 | launching commands may appear to succeed and return success, but later |
| 1373 | individual jobs associated with the transaction may fail. It is possible that |
| 1374 | a management application may have to deal with a partial backup failure after |
| 1375 | a "successful" transaction. |
| 1376 | |
| 1377 | If multiple backup jobs are specified in a single transaction, if one of those |
| 1378 | jobs fails, it will not interact with the other backup jobs in any way by |
| 1379 | default. The job(s) that succeeded will clear the dirty bitmap associated with |
| 1380 | the operation, but the job(s) that failed will not. It is therefore not safe |
| 1381 | to delete any incremental backups that were created successfully in this |
| 1382 | scenario, even though others failed. |
| 1383 | |
| 1384 | This example illustrates a transaction with two backup jobs, where one fails |
| 1385 | and one succeeds: |
| 1386 | |
| 1387 | #. Issue the transaction to start a backup of both drives. |
| 1388 | |
| 1389 | .. code-block:: QMP |
| 1390 | |
| 1391 | -> { |
| 1392 | "execute": "transaction", |
| 1393 | "arguments": { |
| 1394 | "actions": [ |
| 1395 | { |
| 1396 | "type": "blockdev-backup", |
| 1397 | "data": { |
| 1398 | "device": "drive0", |
| 1399 | "bitmap": "bitmap0", |
| 1400 | "sync": "incremental", |
| 1401 | "target": "target0" |
| 1402 | } |
| 1403 | }, |
| 1404 | { |
| 1405 | "type": "blockdev-backup", |
| 1406 | "data": { |
| 1407 | "device": "drive1", |
| 1408 | "bitmap": "bitmap0", |
| 1409 | "sync": "incremental", |
| 1410 | "target": "target1" |
| 1411 | } |
| 1412 | }] |
| 1413 | } |
| 1414 | } |
| 1415 | |
| 1416 | #. Receive notice that the Transaction was accepted, and jobs were |
| 1417 | launched: |
| 1418 | |
| 1419 | .. code-block:: QMP |
| 1420 | |
| 1421 | <- { "return": {} } |
| 1422 | |
| 1423 | #. Receive notice that the first job has completed: |
| 1424 | |
| 1425 | .. code-block:: QMP |
| 1426 | |
| 1427 | <- { |
| 1428 | "timestamp": {...}, |
| 1429 | "data": { |
| 1430 | "device": "drive0", |
| 1431 | "type": "backup", |
| 1432 | "speed": 0, |
| 1433 | "len": 67108864, |
| 1434 | "offset": 67108864 |
| 1435 | }, |
| 1436 | "event": "BLOCK_JOB_COMPLETED" |
| 1437 | } |
| 1438 | |
| 1439 | #. Receive notice that the second job has failed: |
| 1440 | |
| 1441 | .. code-block:: QMP |
| 1442 | |
| 1443 | <- { |
| 1444 | "timestamp": {...}, |
| 1445 | "data": { |
| 1446 | "device": "drive1", |
| 1447 | "action": "report", |
| 1448 | "operation": "read" |
| 1449 | }, |
| 1450 | "event": "BLOCK_JOB_ERROR" |
| 1451 | } |
| 1452 | |
| 1453 | ... |
| 1454 | |
| 1455 | <- { |
| 1456 | "timestamp": {...}, |
| 1457 | "data": { |
| 1458 | "speed": 0, |
| 1459 | "offset": 0, |
| 1460 | "len": 67108864, |
| 1461 | "error": "Input/output error", |
| 1462 | "device": "drive1", |
| 1463 | "type": "backup" |
| 1464 | }, |
| 1465 | "event": "BLOCK_JOB_COMPLETED" |
| 1466 | } |
| 1467 | |
| 1468 | At the conclusion of the above example, ``drive0.inc0.qcow2`` is valid and |
| 1469 | must be kept, but ``drive1.inc0.qcow2`` is incomplete and should be |
| 1470 | deleted. If a VM-wide incremental backup of all drives at a point-in-time is |
| 1471 | to be made, new backups for both drives will need to be made, taking into |
| 1472 | account that a new incremental backup for drive0 needs to be based on top of |
| 1473 | ``drive0.inc0.qcow2``. |
| 1474 | |
| 1475 | For this example, an incremental backup for ``drive0`` was created, but not |
| 1476 | for ``drive1``. The last VM-wide crash-consistent backup that is available in |
| 1477 | this case is the full backup: |
| 1478 | |
| 1479 | .. code:: text |
| 1480 | |
| 1481 | [drive0.full.qcow2] <-- [drive0.inc0.qcow2] |
| 1482 | [drive1.full.qcow2] |
| 1483 | |
| 1484 | To repair this, issue a new incremental backup across both drives. The result |
| 1485 | will be backup chains that resemble the following: |
| 1486 | |
| 1487 | .. code:: text |
| 1488 | |
| 1489 | [drive0.full.qcow2] <-- [drive0.inc0.qcow2] <-- [drive0.inc1.qcow2] |
| 1490 | [drive1.full.qcow2] <-------------------------- [drive1.inc1.qcow2] |
| 1491 | |
| 1492 | Example: Grouped Completion Mode |
| 1493 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 1494 | |
| 1495 | While jobs launched by transactions normally complete or fail individually, |
| 1496 | it's possible to instruct them to complete or fail together as a group. QMP |
| 1497 | transactions take an optional properties structure that can affect the |
| 1498 | behavior of the transaction. |
| 1499 | |
| 1500 | The ``completion-mode`` transaction property can be either ``individual`` |
| 1501 | which is the default legacy behavior described above, or ``grouped``, detailed |
| 1502 | below. |
| 1503 | |
| 1504 | In ``grouped`` completion mode, no jobs will report success until all jobs are |
| 1505 | ready to report success. If any job fails, all other jobs will be cancelled. |
| 1506 | |
| 1507 | Regardless of if a participating incremental backup job failed or was |
| 1508 | cancelled, their associated bitmaps will all be held at their existing |
| 1509 | points-in-time, as in individual failure cases. |
| 1510 | |
| 1511 | Here's the same multi-drive backup scenario from `Example: Partial |
| 1512 | Transactional Failures`_, but with the ``grouped`` completion-mode property |
| 1513 | applied: |
| 1514 | |
| 1515 | #. Issue the multi-drive incremental backup transaction: |
| 1516 | |
| 1517 | .. code-block:: QMP |
| 1518 | |
| 1519 | -> { |
| 1520 | "execute": "transaction", |
| 1521 | "arguments": { |
| 1522 | "properties": { |
| 1523 | "completion-mode": "grouped" |
| 1524 | }, |
| 1525 | "actions": [ |
| 1526 | { |
| 1527 | "type": "blockdev-backup", |
| 1528 | "data": { |
| 1529 | "device": "drive0", |
| 1530 | "bitmap": "bitmap0", |
| 1531 | "sync": "incremental", |
| 1532 | "target": "target0" |
| 1533 | } |
| 1534 | }, |
| 1535 | { |
| 1536 | "type": "blockdev-backup", |
| 1537 | "data": { |
| 1538 | "device": "drive1", |
| 1539 | "bitmap": "bitmap0", |
| 1540 | "sync": "incremental", |
| 1541 | "target": "target1" |
| 1542 | } |
| 1543 | }] |
| 1544 | } |
| 1545 | } |
| 1546 | |
| 1547 | #. Receive notice that the Transaction was accepted, and jobs were launched: |
| 1548 | |
| 1549 | .. code-block:: QMP |
| 1550 | |
| 1551 | <- { "return": {} } |
| 1552 | |
| 1553 | #. Receive notification that the backup job for ``drive1`` has failed: |
| 1554 | |
| 1555 | .. code-block:: QMP |
| 1556 | |
| 1557 | <- { |
| 1558 | "timestamp": {...}, |
| 1559 | "data": { |
| 1560 | "device": "drive1", |
| 1561 | "action": "report", |
| 1562 | "operation": "read" |
| 1563 | }, |
| 1564 | "event": "BLOCK_JOB_ERROR" |
| 1565 | } |
| 1566 | |
| 1567 | <- { |
| 1568 | "timestamp": {...}, |
| 1569 | "data": { |
| 1570 | "speed": 0, |
| 1571 | "offset": 0, |
| 1572 | "len": 67108864, |
| 1573 | "error": "Input/output error", |
| 1574 | "device": "drive1", |
| 1575 | "type": "backup" |
| 1576 | }, |
| 1577 | "event": "BLOCK_JOB_COMPLETED" |
| 1578 | } |
| 1579 | |
| 1580 | #. Receive notification that the job for ``drive0`` has been cancelled: |
| 1581 | |
| 1582 | .. code-block:: QMP |
| 1583 | |
| 1584 | <- { |
| 1585 | "timestamp": {...}, |
| 1586 | "data": { |
| 1587 | "device": "drive0", |
| 1588 | "type": "backup", |
| 1589 | "speed": 0, |
| 1590 | "len": 67108864, |
| 1591 | "offset": 16777216 |
| 1592 | }, |
| 1593 | "event": "BLOCK_JOB_CANCELLED" |
| 1594 | } |
| 1595 | |
| 1596 | At the conclusion of *this* example, both jobs have been aborted due to a |
| 1597 | failure. Both destination images should be deleted and are no longer of use. |
| 1598 | |
| 1599 | The transaction as a whole can simply be re-issued at a later time. |
| 1600 | |
| 1601 | .. raw:: html |
| 1602 | |
| 1603 | <!-- |
| 1604 | The FreeBSD Documentation License |
| 1605 | |
| 1606 | Redistribution and use in source (ReST) and 'compiled' forms (SGML, HTML, |
| 1607 | PDF, PostScript, RTF and so forth) with or without modification, are |
| 1608 | permitted provided that the following conditions are met: |
| 1609 | |
| 1610 | Redistributions of source code (ReST) must retain the above copyright notice, |
| 1611 | this list of conditions and the following disclaimer of this file unmodified. |
| 1612 | |
| 1613 | Redistributions in compiled form (transformed to other DTDs, converted to |
| 1614 | PDF, PostScript, RTF and other formats) must reproduce the above copyright |
| 1615 | notice, this list of conditions and the following disclaimer in the |
| 1616 | documentation and/or other materials provided with the distribution. |
| 1617 | |
| 1618 | THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS |
| 1619 | IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE |
| 1620 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE |
| 1621 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE |
| 1622 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR |
| 1623 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF |
| 1624 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS |
| 1625 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN |
| 1626 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) |
| 1627 | ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF |
| 1628 | THE POSSIBILITY OF SUCH DAMAGE. |
| 1629 | --> |