| 1 | Block I/O error injection using ``blkdebug`` |
| 2 | ============================================ |
| 3 | |
| 4 | .. |
| 5 | Copyright (C) 2014-2015 Red Hat Inc |
| 6 | |
| 7 | This work is licensed under the terms of the GNU GPL, version 2 or later. See |
| 8 | the COPYING file in the top-level directory. |
| 9 | |
| 10 | The ``blkdebug`` block driver is a rule-based error injection engine. It can be |
| 11 | used to exercise error code paths in block drivers including ``ENOSPC`` (out of |
| 12 | space) and ``EIO``. |
| 13 | |
| 14 | This document gives an overview of the features available in ``blkdebug``. |
| 15 | |
| 16 | Background |
| 17 | ---------- |
| 18 | Block drivers have many error code paths that handle I/O errors. Image formats |
| 19 | are especially complex since metadata I/O errors during cluster allocation or |
| 20 | while updating tables happen halfway through request processing and require |
| 21 | discipline to keep image files consistent. |
| 22 | |
| 23 | Error injection allows test cases to trigger I/O errors at specific points. |
| 24 | This way, all error paths can be tested to make sure they are correct. |
| 25 | |
| 26 | Rules |
| 27 | ----- |
| 28 | The ``blkdebug`` block driver takes a list of "rules" that tell the error injection |
| 29 | engine when to fail an I/O request. |
| 30 | |
| 31 | Each I/O request is evaluated against the rules. If a rule matches the request |
| 32 | then its "action" is executed. |
| 33 | |
| 34 | Rules can be placed in a configuration file; the configuration file |
| 35 | follows the same .ini-like format used by QEMU's ``-readconfig`` option, and |
| 36 | each section of the file represents a rule. |
| 37 | |
| 38 | The following configuration file defines a single rule:: |
| 39 | |
| 40 | $ cat blkdebug.conf |
| 41 | [inject-error] |
| 42 | event = "read_aio" |
| 43 | errno = "28" |
| 44 | |
| 45 | This rule fails all aio read requests with ``ENOSPC`` (28). Note that the errno |
| 46 | value depends on the host. On Linux, see |
| 47 | ``/usr/include/asm-generic/errno-base.h`` for errno values. |
| 48 | |
| 49 | Invoke QEMU as follows:: |
| 50 | |
| 51 | $ qemu-system-x86_64 |
| 52 | -drive if=none,cache=none,file=blkdebug:blkdebug.conf:test.img,id=drive0 \ |
| 53 | -device virtio-blk-pci,drive=drive0,id=virtio-blk-pci0 |
| 54 | |
| 55 | Rules support the following attributes: |
| 56 | |
| 57 | ``event`` |
| 58 | which type of operation to match (e.g. ``read_aio``, ``write_aio``, |
| 59 | ``flush_to_os``, ``flush_to_disk``). See `Events`_ for |
| 60 | information on events. |
| 61 | |
| 62 | ``state`` |
| 63 | (optional) the engine must be in this state number in order for this |
| 64 | rule to match. See `State transitions`_ for information |
| 65 | on states. |
| 66 | |
| 67 | ``errno`` |
| 68 | the numeric errno value to return when a request matches this rule. |
| 69 | The errno values depend on the host since the numeric values are not |
| 70 | standardized in the POSIX specification. |
| 71 | |
| 72 | ``sector`` |
| 73 | (optional) a sector number that the request must overlap in order to |
| 74 | match this rule |
| 75 | |
| 76 | ``once`` |
| 77 | (optional, default ``off``) only execute this action on the first |
| 78 | matching request |
| 79 | |
| 80 | ``immediately`` |
| 81 | (optional, default ``off``) return a NULL ``BlockAIOCB`` |
| 82 | pointer and fail without an errno instead. This |
| 83 | exercises the code path where ``BlockAIOCB`` fails and the |
| 84 | caller's ``BlockCompletionFunc`` is not invoked. |
| 85 | |
| 86 | Events |
| 87 | ------ |
| 88 | Block drivers provide information about the type of I/O request they are about |
| 89 | to make so rules can match specific types of requests. For example, the ``qcow2`` |
| 90 | block driver tells ``blkdebug`` when it accesses the L1 table so rules can match |
| 91 | only L1 table accesses and not other metadata or guest data requests. |
| 92 | |
| 93 | The core events are: |
| 94 | |
| 95 | ``read_aio`` |
| 96 | guest data read |
| 97 | |
| 98 | ``write_aio`` |
| 99 | guest data write |
| 100 | |
| 101 | ``flush_to_os`` |
| 102 | write out unwritten block driver state (e.g. cached metadata) |
| 103 | |
| 104 | ``flush_to_disk`` |
| 105 | flush the host block device's disk cache |
| 106 | |
| 107 | See ``qapi/block-core.json:BlkdebugEvent`` for the full list of events. |
| 108 | You may need to grep block driver source code to understand the |
| 109 | meaning of specific events. |
| 110 | |
| 111 | State transitions |
| 112 | ----------------- |
| 113 | There are cases where more power is needed to match a particular I/O request in |
| 114 | a longer sequence of requests. For example:: |
| 115 | |
| 116 | write_aio |
| 117 | flush_to_disk |
| 118 | write_aio |
| 119 | |
| 120 | How do we match the 2nd ``write_aio`` but not the first? This is where state |
| 121 | transitions come in. |
| 122 | |
| 123 | The error injection engine has an integer called the "state" that always starts |
| 124 | initialized to 1. The state integer is internal to ``blkdebug`` and cannot be |
| 125 | observed from outside but rules can interact with it for powerful matching |
| 126 | behavior. |
| 127 | |
| 128 | Rules can be conditional on the current state and they can transition to a new |
| 129 | state. |
| 130 | |
| 131 | When a rule's "state" attribute is non-zero then the current state must equal |
| 132 | the attribute in order for the rule to match. |
| 133 | |
| 134 | For example, to match the 2nd write_aio:: |
| 135 | |
| 136 | [set-state] |
| 137 | event = "write_aio" |
| 138 | state = "1" |
| 139 | new_state = "2" |
| 140 | |
| 141 | [inject-error] |
| 142 | event = "write_aio" |
| 143 | state = "2" |
| 144 | errno = "5" |
| 145 | |
| 146 | The first ``write_aio`` request matches the ``set-state`` rule and transitions from |
| 147 | state 1 to state 2. Once state 2 has been entered, the ``set-state`` rule no |
| 148 | longer matches since it requires state 1. But the ``inject-error`` rule now |
| 149 | matches the next ``write_aio`` request and injects ``EIO`` (5). |
| 150 | |
| 151 | State transition rules support the following attributes: |
| 152 | |
| 153 | ``event`` |
| 154 | which type of operation to match (e.g. ``read_aio``, ``write_aio``, |
| 155 | ``flush_to_os`, ``flush_to_disk``). See `Events`_ for |
| 156 | information on events. |
| 157 | |
| 158 | ``state`` |
| 159 | (optional) the engine must be in this state number in order for this |
| 160 | rule to match |
| 161 | |
| 162 | ``new_state`` |
| 163 | transition to this state number |
| 164 | |
| 165 | Suspend and resume |
| 166 | ------------------ |
| 167 | Exercising code paths in block drivers may require specific ordering amongst |
| 168 | concurrent requests. The "breakpoint" feature allows requests to be halted on |
| 169 | a ``blkdebug`` event and resumed later. This makes it possible to achieve |
| 170 | deterministic ordering when multiple requests are in flight. |
| 171 | |
| 172 | Breakpoints on ``blkdebug`` events are associated with a user-defined ``tag`` string. |
| 173 | This tag serves as an identifier by which the request can be resumed at a later |
| 174 | point. |
| 175 | |
| 176 | See the ``qemu-io(1)`` ``break``, ``resume``, ``remove_break``, and ``wait_break`` |
| 177 | commands for details. |