| 1 | ======== |
| 2 | Fuzzing |
| 3 | ======== |
| 4 | |
| 5 | This document describes the virtual-device fuzzing infrastructure in QEMU and |
| 6 | how to use it to implement additional fuzzers. |
| 7 | |
| 8 | Basics |
| 9 | ------ |
| 10 | |
| 11 | Fuzzing operates by passing inputs to an entry point/target function. The |
| 12 | fuzzer tracks the code coverage triggered by the input. Based on these |
| 13 | findings, the fuzzer mutates the input and repeats the fuzzing. |
| 14 | |
| 15 | To fuzz QEMU, we rely on libfuzzer. Unlike other fuzzers such as AFL, libfuzzer |
| 16 | is an *in-process* fuzzer. For the developer, this means that it is their |
| 17 | responsibility to ensure that state is reset between fuzzing-runs. |
| 18 | |
| 19 | Building the fuzzers |
| 20 | -------------------- |
| 21 | |
| 22 | To build the fuzzers, install a recent version of clang: |
| 23 | Configure with (substitute the clang binaries with the version you installed). |
| 24 | Here, enable-asan and enable-ubsan are optional but they allow us to reliably |
| 25 | detect bugs such as out-of-bounds accesses, uses-after-free, double-frees |
| 26 | etc.:: |
| 27 | |
| 28 | CC=clang-8 CXX=clang++-8 /path/to/configure \ |
| 29 | --enable-fuzzing --enable-asan --enable-ubsan |
| 30 | |
| 31 | Fuzz targets are built similarly to system targets:: |
| 32 | |
| 33 | make qemu-fuzz-i386 |
| 34 | |
| 35 | This builds ``./qemu-fuzz-i386`` |
| 36 | |
| 37 | The first option to this command is: ``--fuzz-target=FUZZ_NAME`` |
| 38 | To list all of the available fuzzers run ``qemu-fuzz-i386`` with no arguments. |
| 39 | |
| 40 | For example:: |
| 41 | |
| 42 | ./qemu-fuzz-i386 --fuzz-target=virtio-scsi-fuzz |
| 43 | |
| 44 | Internally, libfuzzer parses all arguments that do not begin with ``"--"``. |
| 45 | Information about these is available by passing ``-help=1`` |
| 46 | |
| 47 | Now the only thing left to do is wait for the fuzzer to trigger potential |
| 48 | crashes. |
| 49 | |
| 50 | Useful libFuzzer flags |
| 51 | ---------------------- |
| 52 | |
| 53 | As mentioned above, libFuzzer accepts some arguments. Passing ``-help=1`` will |
| 54 | list the available arguments. In particular, these arguments might be helpful: |
| 55 | |
| 56 | * ``CORPUS_DIR/`` : Specify a directory as the last argument to libFuzzer. |
| 57 | libFuzzer stores each "interesting" input in this corpus directory. The next |
| 58 | time you run libFuzzer, it will read all of the inputs from the corpus, and |
| 59 | continue fuzzing from there. You can also specify multiple directories. |
| 60 | libFuzzer loads existing inputs from all specified directories, but will only |
| 61 | write new ones to the first one specified. |
| 62 | |
| 63 | * ``-max_len=4096`` : specify the maximum byte-length of the inputs libFuzzer |
| 64 | will generate. |
| 65 | |
| 66 | * ``-close_fd_mask={1,2,3}`` : close, stderr, or both. Useful for targets that |
| 67 | trigger many debug/error messages, or create output on the serial console. |
| 68 | |
| 69 | * ``-jobs=4 -workers=4`` : These arguments configure libFuzzer to run 4 fuzzers in |
| 70 | parallel (4 fuzzing jobs in 4 worker processes). Alternatively, with only |
| 71 | ``-jobs=N``, libFuzzer automatically spawns a number of workers less than or equal |
| 72 | to half the available CPU cores. Replace 4 with a number appropriate for your |
| 73 | machine. Make sure to specify a ``CORPUS_DIR``, which will allow the parallel |
| 74 | fuzzers to share information about the interesting inputs they find. |
| 75 | |
| 76 | * ``-use_value_profile=1`` : For each comparison operation, libFuzzer computes |
| 77 | ``(caller_pc&4095) | (popcnt(Arg1 ^ Arg2) << 12)`` and places this in the |
| 78 | coverage table. Useful for targets with "magic" constants. If Arg1 came from |
| 79 | the fuzzer's input and Arg2 is a magic constant, then each time the Hamming |
| 80 | distance between Arg1 and Arg2 decreases, libFuzzer adds the input to the |
| 81 | corpus. |
| 82 | |
| 83 | * ``-shrink=1`` : Tries to make elements of the corpus "smaller". Might lead to |
| 84 | better coverage performance, depending on the target. |
| 85 | |
| 86 | Note that libFuzzer's exact behavior will depend on the version of |
| 87 | clang and libFuzzer used to build the device fuzzers. |
| 88 | |
| 89 | Generating Coverage Reports |
| 90 | --------------------------- |
| 91 | |
| 92 | Code coverage is a crucial metric for evaluating a fuzzer's performance. |
| 93 | libFuzzer's output provides a "cov: " column that provides a total number of |
| 94 | unique blocks/edges covered. To examine coverage on a line-by-line basis we |
| 95 | can use Clang coverage: |
| 96 | |
| 97 | 1. Configure libFuzzer to store a corpus of all interesting inputs (see |
| 98 | CORPUS_DIR above) |
| 99 | 2. ``./configure`` the QEMU build with :: |
| 100 | |
| 101 | --enable-fuzzing \ |
| 102 | --extra-cflags="-fprofile-instr-generate -fcoverage-mapping" |
| 103 | |
| 104 | 3. Re-run the fuzzer. Specify $CORPUS_DIR/* as an argument, telling libfuzzer |
| 105 | to execute all of the inputs in $CORPUS_DIR and exit. Once the process |
| 106 | exits, you should find a file, "default.profraw" in the working directory. |
| 107 | 4. Execute these commands to generate a detailed HTML coverage-report:: |
| 108 | |
| 109 | llvm-profdata merge -output=default.profdata default.profraw |
| 110 | llvm-cov show ./path/to/qemu-fuzz-i386 -instr-profile=default.profdata \ |
| 111 | --format html -output-dir=/path/to/output/report |
| 112 | |
| 113 | Adding a new fuzzer |
| 114 | ------------------- |
| 115 | |
| 116 | Coverage over virtual devices can be improved by adding additional fuzzers. |
| 117 | Fuzzers are kept in ``tests/qtest/fuzz/`` and should be added to |
| 118 | ``tests/qtest/fuzz/meson.build`` |
| 119 | |
| 120 | Fuzzers can rely on both qtest and libqos to communicate with virtual devices. |
| 121 | |
| 122 | 1. Create a new source file. For example ``tests/qtest/fuzz/foo-device-fuzz.c``. |
| 123 | |
| 124 | 2. Write the fuzzing code using the libqtest/libqos API. See existing fuzzers |
| 125 | for reference. |
| 126 | |
| 127 | 3. Add the fuzzer to ``tests/qtest/fuzz/meson.build``. |
| 128 | |
| 129 | Fuzzers can be more-or-less thought of as special qtest programs which can |
| 130 | modify the qtest commands and/or qtest command arguments based on inputs |
| 131 | provided by libfuzzer. Libfuzzer passes a byte array and length. Commonly the |
| 132 | fuzzer loops over the byte-array interpreting it as a list of qtest commands, |
| 133 | addresses, or values. |
| 134 | |
| 135 | The Generic Fuzzer |
| 136 | ------------------ |
| 137 | |
| 138 | Writing a fuzz target can be a lot of effort (especially if a device driver has |
| 139 | not be built-out within libqos). Many devices can be fuzzed to some degree, |
| 140 | without any device-specific code, using the generic-fuzz target. |
| 141 | |
| 142 | The generic-fuzz target is capable of fuzzing devices over their PIO, MMIO, |
| 143 | and DMA input-spaces. To apply the generic-fuzz to a device, we need to define |
| 144 | two env-variables, at minimum: |
| 145 | |
| 146 | * ``QEMU_FUZZ_ARGS=`` is the set of QEMU arguments used to configure a machine, with |
| 147 | the device attached. For example, if we want to fuzz the virtio-net device |
| 148 | attached to a pc-i440fx machine, we can specify:: |
| 149 | |
| 150 | QEMU_FUZZ_ARGS="-M pc -nodefaults -netdev user,id=user0 \ |
| 151 | -device virtio-net,netdev=user0" |
| 152 | |
| 153 | * ``QEMU_FUZZ_OBJECTS=`` is a set of space-delimited strings used to identify |
| 154 | the MemoryRegions that will be fuzzed. These strings are compared against |
| 155 | MemoryRegion names and MemoryRegion owner names, to decide whether each |
| 156 | MemoryRegion should be fuzzed. These strings support globbing. For the |
| 157 | virtio-net example, we could use one of :: |
| 158 | |
| 159 | QEMU_FUZZ_OBJECTS='virtio-net' |
| 160 | QEMU_FUZZ_OBJECTS='virtio*' |
| 161 | QEMU_FUZZ_OBJECTS='virtio* pcspk' # Fuzz the virtio devices and the speaker |
| 162 | QEMU_FUZZ_OBJECTS='*' # Fuzz the whole machine`` |
| 163 | |
| 164 | The ``"info mtree"`` and ``"info qom-tree"`` monitor commands can be especially |
| 165 | useful for identifying the ``MemoryRegion`` and ``Object`` names used for |
| 166 | matching. |
| 167 | |
| 168 | As a generic rule-of-thumb, the more ``MemoryRegions``/Devices we match, the |
| 169 | greater the input-space, and the smaller the probability of finding crashing |
| 170 | inputs for individual devices. As such, it is usually a good idea to limit the |
| 171 | fuzzer to only a few ``MemoryRegions``. |
| 172 | |
| 173 | To ensure that these env variables have been configured correctly, we can use:: |
| 174 | |
| 175 | ./qemu-fuzz-i386 --fuzz-target=generic-fuzz -runs=0 |
| 176 | |
| 177 | The output should contain a complete list of matched MemoryRegions. |
| 178 | |
| 179 | OSS-Fuzz |
| 180 | -------- |
| 181 | QEMU is continuously fuzzed on `OSS-Fuzz |
| 182 | <https://github.com/google/oss-fuzz>`_. By default, the OSS-Fuzz build |
| 183 | will try to fuzz every fuzz-target. Since the generic-fuzz target |
| 184 | requires additional information provided in environment variables, we |
| 185 | pre-define some generic-fuzz configs in |
| 186 | ``tests/qtest/fuzz/generic_fuzz_configs.h``. Each config must specify: |
| 187 | |
| 188 | - ``.name``: To identify the fuzzer config |
| 189 | |
| 190 | - ``.args`` OR ``.argfunc``: A string or pointer to a function returning a |
| 191 | string. These strings are used to specify the ``QEMU_FUZZ_ARGS`` |
| 192 | environment variable. ``argfunc`` is useful when the config relies on e.g. |
| 193 | a dynamically created temp directory, or a free tcp/udp port. |
| 194 | |
| 195 | - ``.objects``: A string that specifies the ``QEMU_FUZZ_OBJECTS`` environment |
| 196 | variable. |
| 197 | |
| 198 | To fuzz additional devices/device configuration on OSS-Fuzz, send patches for |
| 199 | either a new device-specific fuzzer or a new generic-fuzz config. |
| 200 | |
| 201 | Build details: |
| 202 | |
| 203 | - The Dockerfile that sets up the environment for building QEMU's |
| 204 | fuzzers on OSS-Fuzz can be fund in the OSS-Fuzz repository |
| 205 | __(https://github.com/google/oss-fuzz/blob/master/projects/qemu/Dockerfile) |
| 206 | |
| 207 | - The script responsible for building the fuzzers can be found in the |
| 208 | QEMU source tree at ``scripts/oss-fuzz/build.sh`` |
| 209 | |
| 210 | Building Crash Reproducers |
| 211 | ----------------------------------------- |
| 212 | When we find a crash, we should try to create an independent reproducer, that |
| 213 | can be used on a non-fuzzer build of QEMU. This filters out any potential |
| 214 | false-positives, and improves the debugging experience for developers. |
| 215 | Here are the steps for building a reproducer for a crash found by the |
| 216 | generic-fuzz target. |
| 217 | |
| 218 | - Ensure the crash reproduces:: |
| 219 | |
| 220 | qemu-fuzz-i386 --fuzz-target... ./crash-... |
| 221 | |
| 222 | - Gather the QTest output for the crash:: |
| 223 | |
| 224 | QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \ |
| 225 | qemu-fuzz-i386 --fuzz-target... ./crash-... &> /tmp/trace |
| 226 | |
| 227 | - Reorder and clean-up the resulting trace:: |
| 228 | |
| 229 | scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/trace > /tmp/reproducer |
| 230 | |
| 231 | - Get the arguments needed to start qemu, and provide a path to qemu:: |
| 232 | |
| 233 | less /tmp/trace # The args should be logged at the top of this file |
| 234 | export QEMU_ARGS="-machine ..." |
| 235 | export QEMU_PATH="path/to/qemu-system" |
| 236 | |
| 237 | - Ensure the crash reproduces in qemu-system:: |
| 238 | |
| 239 | $QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer |
| 240 | |
| 241 | - From the crash output, obtain some string that identifies the crash. This |
| 242 | can be a line in the stack-trace, for example:: |
| 243 | |
| 244 | export CRASH_TOKEN="hw/usb/hcd-xhci.c:1865" |
| 245 | |
| 246 | - Minimize the reproducer:: |
| 247 | |
| 248 | scripts/oss-fuzz/minimize_qtest_trace.py -M1 -M2 \ |
| 249 | /tmp/reproducer /tmp/reproducer-minimized |
| 250 | |
| 251 | - Confirm that the minimized reproducer still crashes:: |
| 252 | |
| 253 | $QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer-minimized |
| 254 | |
| 255 | - Create a one-liner reproducer that can be sent over email:: |
| 256 | |
| 257 | ./scripts/oss-fuzz/output_reproducer.py -bash /tmp/reproducer-minimized |
| 258 | |
| 259 | - Output the C source code for a test case that will reproduce the bug:: |
| 260 | |
| 261 | ./scripts/oss-fuzz/output_reproducer.py -owner "John Smith <john@smith.com>"\ |
| 262 | -name "test_function_name" /tmp/reproducer-minimized |
| 263 | |
| 264 | - Report the bug and send a patch with the C reproducer upstream |
| 265 | |
| 266 | QEMU can also read the reproducer directly from a file rather than |
| 267 | from standard input:: |
| 268 | |
| 269 | $QEMU_PATH $QEMU_ARGS -qtest chardev:repro \ |
| 270 | -chardev file,id=repro,path=/dev/null,input-path=/tmp/reproducer |
| 271 | |
| 272 | This is useful if you want to run QEMU under a debugger to investigate |
| 273 | the failure. |
| 274 | |
| 275 | Implementation Details / Fuzzer Lifecycle |
| 276 | ----------------------------------------- |
| 277 | |
| 278 | The fuzzer has two entrypoints that libfuzzer calls. libfuzzer provides it's |
| 279 | own ``main()``, which performs some setup, and calls the entrypoints: |
| 280 | |
| 281 | ``LLVMFuzzerInitialize``: called prior to fuzzing. Used to initialize all of the |
| 282 | necessary state |
| 283 | |
| 284 | ``LLVMFuzzerTestOneInput``: called for each fuzzing run. Processes the input and |
| 285 | resets the state at the end of each run. |
| 286 | |
| 287 | In more detail: |
| 288 | |
| 289 | ``LLVMFuzzerInitialize`` parses the arguments to the fuzzer (must start with two |
| 290 | dashes, so they are ignored by libfuzzer ``main()``). Currently, the arguments |
| 291 | select the fuzz target. Then, the qtest client is initialized. If the target |
| 292 | requires qos, qgraph is set up and the QOM/LIBQOS modules are initialized. |
| 293 | Then the QGraph is walked and the QEMU cmd_line is determined and saved. |
| 294 | |
| 295 | After this, the ``vl.c:main`` is called to set up the guest. There are |
| 296 | target-specific hooks that can be called before and after main, for |
| 297 | additional setup(e.g. PCI setup, or VM snapshotting). |
| 298 | |
| 299 | ``LLVMFuzzerTestOneInput``: Uses qtest/qos functions to act based on the fuzz |
| 300 | input. It is also responsible for manually calling ``main_loop_wait`` to ensure |
| 301 | that bottom halves are executed and any cleanup required before the next input. |
| 302 | |
| 303 | Since the same process is reused for many fuzzing runs, QEMU state needs to |
| 304 | be reset at the end of each run. For example, this can be done by rebooting the |
| 305 | VM, after each run. |
| 306 | |
| 307 | - *Pros*: Straightforward and fast for simple fuzz targets. |
| 308 | |
| 309 | - *Cons*: Depending on the device, does not reset all device state. If the |
| 310 | device requires some initialization prior to being ready for fuzzing (common |
| 311 | for QOS-based targets), this initialization needs to be done after each |
| 312 | reboot. |
| 313 | |
| 314 | - *Example target*: ``i440fx-qtest-reboot-fuzz`` |