| 1 | .. _testing: |
| 2 | |
| 3 | Testing in QEMU |
| 4 | =============== |
| 5 | |
| 6 | QEMU's testing infrastructure is fairly complex as it covers |
| 7 | everything from unit testing and exercising specific sub-systems all |
| 8 | the way to full blown functional tests. To get an overview of the |
| 9 | tests you can run ``make check-help`` from either the source or build |
| 10 | tree. |
| 11 | |
| 12 | Most (but not all) tests are also integrated as an automated test into |
| 13 | the meson build system so can be run directly from the build tree, |
| 14 | for example:: |
| 15 | |
| 16 | [./pyvenv/bin/]meson test --suite qemu:softfloat |
| 17 | |
| 18 | will run just the softfloat tests. |
| 19 | |
| 20 | An automated test is written with one of the test frameworks using its |
| 21 | generic test functions/classes. The test framework can run the tests and |
| 22 | report their success or failure [1]_. |
| 23 | |
| 24 | An automated test has essentially three parts: |
| 25 | |
| 26 | 1. The test initialization of the parameters, where the expected parameters, |
| 27 | like inputs and expected results, are set up; |
| 28 | 2. The call to the code that should be tested; |
| 29 | 3. An assertion, comparing the result from the previous call with the expected |
| 30 | result set during the initialization of the parameters. If the result |
| 31 | matches the expected result, the test has been successful; otherwise, it has |
| 32 | failed. |
| 33 | |
| 34 | The rest of this document will cover the details for specific test |
| 35 | groups. |
| 36 | |
| 37 | Testing with "make check" |
| 38 | ------------------------- |
| 39 | |
| 40 | The "make check" testing family includes most of the C based tests in QEMU. |
| 41 | |
| 42 | The usual way to run these tests is: |
| 43 | |
| 44 | .. code:: |
| 45 | |
| 46 | make check |
| 47 | |
| 48 | which includes QAPI schema tests, unit tests, QTests and some iotests. |
| 49 | Different sub-types of "make check" tests will be explained below. |
| 50 | |
| 51 | Before running tests, it is best to build QEMU programs first. Some tests |
| 52 | expect the executables to exist and will fail with obscure messages if they |
| 53 | cannot find them. |
| 54 | |
| 55 | The timeouts for QEMU tests are set conservatively so you should not |
| 56 | in general find that tests time out. However, if you are running on a |
| 57 | particularly slow host or with a slow configuration (such as a build |
| 58 | with the clang address-sanitizer enabled) you can globally raise all |
| 59 | the timeouts, by setting the ``TIMEOUT_MULTIPLIER`` environment |
| 60 | variable. For instance: |
| 61 | |
| 62 | .. code:: |
| 63 | |
| 64 | TIMEOUT_MULTIPLIER=3 make check |
| 65 | |
| 66 | will run with all the default timeouts multiplied by three. You can |
| 67 | also disable timeouts entirely by setting the environment variable to |
| 68 | ``0``. |
| 69 | |
| 70 | .. _unit-tests: |
| 71 | |
| 72 | Unit tests |
| 73 | ~~~~~~~~~~ |
| 74 | |
| 75 | A unit test is responsible for exercising individual software components as a |
| 76 | unit, like interfaces, data structures, and functionality, uncovering errors |
| 77 | within the boundaries of a component. The verification effort is in the |
| 78 | smallest software unit and focuses on the internal processing logic and data |
| 79 | structures. A test case of unit tests should be designed to uncover errors |
| 80 | due to erroneous computations, incorrect comparisons, or improper control |
| 81 | flow [2]_. |
| 82 | |
| 83 | In QEMU, unit tests can be invoked with ``make check-unit``. They are |
| 84 | simple C tests that typically link to individual QEMU object files and |
| 85 | exercise them by calling exported functions. |
| 86 | |
| 87 | If you are writing new code in QEMU, consider adding a unit test, especially |
| 88 | for utility modules that are relatively stateless or have few dependencies. To |
| 89 | add a new unit test: |
| 90 | |
| 91 | 1. Create a new source file. For example, ``tests/unit/foo-test.c``. |
| 92 | |
| 93 | 2. Write the test. Normally you would include the header file which exports |
| 94 | the module API, then verify the interface behaves as expected from your |
| 95 | test. The test code should be organized with the glib testing framework. |
| 96 | Copying and modifying an existing test is usually a good idea. |
| 97 | |
| 98 | 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a |
| 99 | dictionary called ``tests``. The values are any additional sources and |
| 100 | dependencies to be linked with the test. For a simple test whose source |
| 101 | is in ``tests/unit/foo-test.c``, it is enough to add an entry like:: |
| 102 | |
| 103 | { |
| 104 | ... |
| 105 | 'foo-test': [], |
| 106 | ... |
| 107 | } |
| 108 | |
| 109 | Since unit tests don't require environment variables, the simplest way to debug |
| 110 | a unit test failure is often directly invoking it or even running it under |
| 111 | ``gdb``. However there can still be differences in behavior between ``make`` |
| 112 | invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment |
| 113 | variable (which affects memory reclamation and catches invalid pointers better) |
| 114 | and gtester options. If necessary, you can run |
| 115 | |
| 116 | .. code:: |
| 117 | |
| 118 | make check-unit V=1 |
| 119 | |
| 120 | and copy the actual command line which executes the unit test, then run |
| 121 | it from the command line. |
| 122 | |
| 123 | QTest |
| 124 | ~~~~~ |
| 125 | |
| 126 | QTest is a device emulation testing framework. It can be very useful to test |
| 127 | device models; it could also control certain aspects of QEMU (such as virtual |
| 128 | clock stepping), with a special purpose "qtest" protocol. Refer to |
| 129 | :doc:`qtest` for more details. |
| 130 | |
| 131 | QTest cases can be executed with |
| 132 | |
| 133 | .. code:: |
| 134 | |
| 135 | make check-qtest |
| 136 | |
| 137 | Writing portable test cases |
| 138 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 139 | Both unit tests and qtests can run on POSIX hosts as well as Windows hosts. |
| 140 | Care must be taken when writing portable test cases that can be built and run |
| 141 | successfully on various hosts. The following list shows some best practices: |
| 142 | |
| 143 | * Use portable APIs from glib whenever necessary, e.g.: g_setenv(), |
| 144 | g_mkdtemp(), g_mkdir(). |
| 145 | * Avoid using hardcoded /tmp for temporary file directory. |
| 146 | Use g_get_tmp_dir() instead. |
| 147 | * Bear in mind that Windows has different special string representation for |
| 148 | stdin/stdout/stderr and null devices. For example if your test case uses |
| 149 | "/dev/fd/2" and "/dev/null" on Linux, remember to use "2" and "nul" on |
| 150 | Windows instead. Also IO redirection does not work on Windows, so avoid |
| 151 | using "2>nul" whenever necessary. |
| 152 | * If your test cases uses the blkdebug feature, use relative path to pass |
| 153 | the config and image file paths in the command line as Windows absolute |
| 154 | path contains the delimiter ":" which will confuse the blkdebug parser. |
| 155 | * Use double quotes in your extra QEMU command line in your test cases |
| 156 | instead of single quotes, as Windows does not drop single quotes when |
| 157 | passing the command line to QEMU. |
| 158 | * Windows opens a file in text mode by default, while a POSIX compliant |
| 159 | implementation treats text files and binary files the same. So if your |
| 160 | test cases opens a file to write some data and later wants to compare the |
| 161 | written data with the original one, be sure to pass the letter 'b' as |
| 162 | part of the mode string to fopen(), or O_BINARY flag for the open() call. |
| 163 | * If a certain test case can only run on POSIX or Linux hosts, use a proper |
| 164 | #ifdef in the codes. If the whole test suite cannot run on Windows, disable |
| 165 | the build in the meson.build file. |
| 166 | |
| 167 | .. _qapi-tests: |
| 168 | |
| 169 | QAPI schema tests |
| 170 | ~~~~~~~~~~~~~~~~~ |
| 171 | |
| 172 | The QAPI schema tests validate the QAPI parser used by QMP, by feeding |
| 173 | predefined input to the parser and comparing the result with the reference |
| 174 | output. |
| 175 | |
| 176 | The input/output data is managed under the ``tests/qapi-schema`` directory. |
| 177 | Each test case includes four files that have a common base name: |
| 178 | |
| 179 | * ``${casename}.json`` - the file contains the JSON input for feeding the |
| 180 | parser |
| 181 | * ``${casename}.out`` - the file contains the expected stdout from the parser |
| 182 | * ``${casename}.err`` - the file contains the expected stderr from the parser |
| 183 | * ``${casename}.exit`` - the expected error code |
| 184 | |
| 185 | Consider adding a new QAPI schema test when you are making a change on the QAPI |
| 186 | parser (either fixing a bug or extending/modifying the syntax). To do this: |
| 187 | |
| 188 | 1. Add four files for the new case as explained above. For example: |
| 189 | |
| 190 | ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``. |
| 191 | |
| 192 | 2. Add the new test in ``tests/Makefile.include``. For example: |
| 193 | |
| 194 | ``qapi-schema += foo.json`` |
| 195 | |
| 196 | The reference output can be automatically updated to match the latest QAPI |
| 197 | code generator by running the tests with the QEMU_TEST_REGENERATE environment |
| 198 | variable set. |
| 199 | |
| 200 | .. code:: |
| 201 | |
| 202 | QEMU_TEST_REGENERATE=1 make check-qapi-schema |
| 203 | |
| 204 | The resulting changes must be reviewed by the author to ensure they match |
| 205 | the intended results before adding the updated reference output to the |
| 206 | same commit that alters the generator code. |
| 207 | |
| 208 | .. _tracetool-tests: |
| 209 | |
| 210 | Tracetool tests |
| 211 | ~~~~~~~~~~~~~~~ |
| 212 | |
| 213 | The tracetool tests validate the generated source files used for defining |
| 214 | probes for various tracing backends and source formats. The test operates |
| 215 | by running the tracetool program against a sample trace-events file, and |
| 216 | comparing the generated output against known good reference output. The |
| 217 | tests can be run with: |
| 218 | |
| 219 | .. code:: |
| 220 | |
| 221 | make check-tracetool |
| 222 | |
| 223 | The reference output is stored in files under tests/tracetool, and when |
| 224 | the tracetool backend/format output is intentionally changed, the reference |
| 225 | files need to be updated. This can be automated by setting the |
| 226 | QEMU_TEST_REGENERATE=1 environment variable: |
| 227 | |
| 228 | .. code:: |
| 229 | |
| 230 | QEMU_TEST_REGENERATE=1 make check-tracetool |
| 231 | |
| 232 | The resulting changes must be reviewed by the author to ensure they match |
| 233 | the intended results, before adding the updated reference output to the |
| 234 | same commit that alters the generator code. |
| 235 | |
| 236 | check-block |
| 237 | ~~~~~~~~~~~ |
| 238 | |
| 239 | There are a variety of ways to exercise the block layer I/O tests |
| 240 | via make targets for a selection of formats / protocols (collectively |
| 241 | referred to as ``drivers`` below). |
| 242 | |
| 243 | A default ``make check`` or ``make check-block`` command will exercise |
| 244 | the ``qcow2`` format, using the tests tagged into the ``auto`` group |
| 245 | only. |
| 246 | |
| 247 | These targets accept the ``SPEED`` variable to augment the set of tests |
| 248 | to run. A slightly more comprehensive test plan can be run by defining |
| 249 | ``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw`` |
| 250 | drivers. The most comprehensive test plan can be run by defining |
| 251 | ``SPEED=thorough``, which enables all available tests for the drivers |
| 252 | ``luks``, ``nbd``, ``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``, |
| 253 | ``vhdx``, ``vmdk``, and ``vpc``. |
| 254 | |
| 255 | Each of drivers also has its own dedicated make target, named |
| 256 | ``make check-block-$DRIVER`` which will run all available tests for |
| 257 | the designated driver and does not require the ``SPEED`` variable |
| 258 | to be set. |
| 259 | |
| 260 | See the "QEMU iotests" section below for more information on the |
| 261 | block I/O test framework that is leveraged by these ``make`` targets. |
| 262 | |
| 263 | .. _qemu-iotests: |
| 264 | |
| 265 | QEMU iotests |
| 266 | ------------ |
| 267 | |
| 268 | QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing |
| 269 | framework widely used to test block layer related features. It is higher level |
| 270 | than "make check" tests and 99% of the code is written in bash or Python |
| 271 | scripts. The testing success criteria is golden output comparison, and the |
| 272 | test files are named with numbers. |
| 273 | |
| 274 | To run iotests, make sure QEMU is built successfully, then switch to the |
| 275 | ``tests/qemu-iotests`` directory under the build directory, and run ``./check`` |
| 276 | with desired arguments from there. |
| 277 | |
| 278 | By default, "raw" format and "file" protocol is used; all tests will be |
| 279 | executed, except the unsupported ones. You can override the format and protocol |
| 280 | with arguments: |
| 281 | |
| 282 | .. code:: |
| 283 | |
| 284 | # test with qcow2 format |
| 285 | ./check -qcow2 |
| 286 | # or test a different protocol |
| 287 | ./check -nbd |
| 288 | |
| 289 | It's also possible to list test numbers explicitly: |
| 290 | |
| 291 | .. code:: |
| 292 | |
| 293 | # run selected cases with qcow2 format |
| 294 | ./check -qcow2 001 030 153 |
| 295 | |
| 296 | Cache mode can be selected with the "-c" option, which may help reveal bugs |
| 297 | that are specific to certain cache mode. |
| 298 | |
| 299 | More options are supported by the ``./check`` script, run ``./check -h`` for |
| 300 | help. |
| 301 | |
| 302 | Writing a new test case |
| 303 | ~~~~~~~~~~~~~~~~~~~~~~~ |
| 304 | |
| 305 | Consider writing a tests case when you are making any changes to the block |
| 306 | layer. An iotest case is usually the choice for that. There are already many |
| 307 | test cases, so it is possible that extending one of them may achieve the goal |
| 308 | and save the boilerplate to create one. (Unfortunately, there isn't a 100% |
| 309 | reliable way to find a related one out of hundreds of tests. One approach is |
| 310 | using ``git grep``.) |
| 311 | |
| 312 | Usually an iotest case consists of two files. One is an executable that |
| 313 | produces output to stdout and stderr, the other is the expected reference |
| 314 | output. They are given the same number in file names. E.g. Test script ``055`` |
| 315 | and reference output ``055.out``. |
| 316 | |
| 317 | In rare cases, when outputs differ between cache mode ``none`` and others, a |
| 318 | ``.out.nocache`` file is added. In other cases, when outputs differ between |
| 319 | image formats, more than one ``.out`` files are created ending with the |
| 320 | respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``. |
| 321 | |
| 322 | There isn't a hard rule about how to write a test script, but a new test is |
| 323 | usually a (copy and) modification of an existing case. There are a few |
| 324 | commonly used ways to create a test: |
| 325 | |
| 326 | * A Bash script. It will make use of several environmental variables related |
| 327 | to the testing procedure, and could source a group of ``common.*`` libraries |
| 328 | for some common helper routines. |
| 329 | |
| 330 | * A Python unittest script. Import ``iotests`` and create a subclass of |
| 331 | ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of |
| 332 | this approach is that the output is too scarce, and the script is considered |
| 333 | harder to debug. |
| 334 | |
| 335 | * A simple Python script without using unittest module. This could also import |
| 336 | ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit |
| 337 | from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest |
| 338 | execution. This is a combination of 1 and 2. |
| 339 | |
| 340 | Pick the language per your preference since both Bash and Python have |
| 341 | comparable library support for invoking and interacting with QEMU programs. If |
| 342 | you opt for Python, it is strongly recommended to write Python 3 compatible |
| 343 | code. |
| 344 | |
| 345 | Both Python and Bash frameworks in iotests provide helpers to manage test |
| 346 | images. They can be used to create and clean up images under the test |
| 347 | directory. If no I/O or any protocol specific feature is needed, it is often |
| 348 | more convenient to use the pseudo block driver, ``null-co://``, as the test |
| 349 | image, which doesn't require image creation or cleaning up. Avoid system-wide |
| 350 | devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``. |
| 351 | Otherwise, image locking implications have to be considered. For example, |
| 352 | another application on the host may have locked the file, possibly leading to a |
| 353 | test failure. If using such devices are explicitly desired, consider adding |
| 354 | ``locking=off`` option to disable image locking. |
| 355 | |
| 356 | Debugging a test case |
| 357 | ~~~~~~~~~~~~~~~~~~~~~ |
| 358 | |
| 359 | The following options to the ``check`` script can be useful when debugging |
| 360 | a failing test: |
| 361 | |
| 362 | * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a |
| 363 | connection from a gdb client. The options given to ``gdbserver`` (e.g. the |
| 364 | address on which to listen for connections) are taken from the ``$GDB_OPTIONS`` |
| 365 | environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on |
| 366 | ``localhost:12345``. |
| 367 | It is possible to connect to it for example with |
| 368 | ``gdb -iex "target remote $addr"``, where ``$addr`` is the address |
| 369 | ``gdbserver`` listens on. |
| 370 | If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored, |
| 371 | regardless of whether it is set or not. |
| 372 | |
| 373 | * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects |
| 374 | warnings, it will print and save the log in |
| 375 | ``$TEST_DIR/<valgrind_pid>.valgrind``. |
| 376 | The final command line will be ``valgrind --log-file=$TEST_DIR/ |
| 377 | <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...`` |
| 378 | |
| 379 | * ``-d`` (debug) just increases the logging verbosity, showing |
| 380 | for example the QMP commands and answers. |
| 381 | |
| 382 | * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output, |
| 383 | instead of saving it into a log file in |
| 384 | ``$TEST_DIR/qemu-machine-<random_string>``. |
| 385 | |
| 386 | Test case groups |
| 387 | ~~~~~~~~~~~~~~~~ |
| 388 | |
| 389 | "Tests may belong to one or more test groups, which are defined in the form |
| 390 | of a comment in the test source file. By convention, test groups are listed |
| 391 | in the second line of the test file, after the "#!/..." line, like this: |
| 392 | |
| 393 | .. code:: |
| 394 | |
| 395 | #!/usr/bin/env python3 |
| 396 | # group: auto quick |
| 397 | # |
| 398 | ... |
| 399 | |
| 400 | Another way of defining groups is creating the tests/qemu-iotests/group.local |
| 401 | file. This should be used only for downstream (this file should never appear |
| 402 | in upstream). This file may be used for defining some downstream test groups |
| 403 | or for temporarily disabling tests, like this: |
| 404 | |
| 405 | .. code:: |
| 406 | |
| 407 | # groups for some company downstream process |
| 408 | # |
| 409 | # ci - tests to run on build |
| 410 | # down - our downstream tests, not for upstream |
| 411 | # |
| 412 | # Format of each line is: |
| 413 | # TEST_NAME TEST_GROUP [TEST_GROUP ]... |
| 414 | |
| 415 | 013 ci |
| 416 | 210 disabled |
| 417 | 215 disabled |
| 418 | our-ugly-workaround-test down ci |
| 419 | |
| 420 | Note that the following group names have a special meaning: |
| 421 | |
| 422 | - quick: Tests in this group should finish within a few seconds. |
| 423 | |
| 424 | - auto: Tests in this group are used during "make check" and should be |
| 425 | runnable in any case. That means they should run with every QEMU binary |
| 426 | (also non-x86), with every QEMU configuration (i.e. must not fail if |
| 427 | an optional feature is not compiled in - but reporting a "skip" is ok), |
| 428 | work at least with the qcow2 file format, work with all kind of host |
| 429 | filesystems and users (e.g. "nobody" or "root") and must not take too |
| 430 | much memory and disk space (since CI pipelines tend to fail otherwise). |
| 431 | |
| 432 | - disabled: Tests in this group are disabled and ignored by check. |
| 433 | |
| 434 | .. _container-ref: |
| 435 | |
| 436 | Container based tests |
| 437 | --------------------- |
| 438 | |
| 439 | Introduction |
| 440 | ~~~~~~~~~~~~ |
| 441 | |
| 442 | The container testing framework in QEMU utilizes public images to |
| 443 | build and test QEMU in predefined and widely accessible Linux |
| 444 | environments. This makes it possible to expand the test coverage |
| 445 | across distros, toolchain flavors and library versions. The support |
| 446 | was originally written for Docker although we also support Podman as |
| 447 | an alternative container runtime. Although many of the target |
| 448 | names and scripts are prefixed with "docker" the system will |
| 449 | automatically run on whichever is configured. |
| 450 | |
| 451 | The container images are also used to augment the generation of tests |
| 452 | for testing TCG. See :ref:`checktcg-ref` for more details. |
| 453 | |
| 454 | Docker Prerequisites |
| 455 | ~~~~~~~~~~~~~~~~~~~~ |
| 456 | |
| 457 | Install "docker" with the system package manager and start the Docker service |
| 458 | on your development machine, then make sure you have the privilege to run |
| 459 | Docker commands. Typically it means setting up passwordless ``sudo docker`` |
| 460 | command or login as root. For example: |
| 461 | |
| 462 | .. code:: |
| 463 | |
| 464 | $ sudo yum install docker |
| 465 | $ # or `apt-get install docker` for Ubuntu, etc. |
| 466 | $ sudo systemctl start docker |
| 467 | $ sudo docker ps |
| 468 | |
| 469 | The last command should print an empty table, to verify the system is ready. |
| 470 | |
| 471 | An alternative method to set up permissions is by adding the current user to |
| 472 | "docker" group and making the docker daemon socket file (by default |
| 473 | ``/var/run/docker.sock``) accessible to the group: |
| 474 | |
| 475 | .. code:: |
| 476 | |
| 477 | $ sudo groupadd docker |
| 478 | $ sudo usermod $USER -a -G docker |
| 479 | $ sudo chown :docker /var/run/docker.sock |
| 480 | |
| 481 | Note that any one of above configurations makes it possible for the user to |
| 482 | exploit the whole host with Docker bind mounting or other privileged |
| 483 | operations. So only do it on development machines. |
| 484 | |
| 485 | Podman Prerequisites |
| 486 | ~~~~~~~~~~~~~~~~~~~~ |
| 487 | |
| 488 | Install "podman" with the system package manager. |
| 489 | |
| 490 | .. code:: |
| 491 | |
| 492 | $ sudo dnf install podman |
| 493 | $ podman ps |
| 494 | |
| 495 | The last command should print an empty table, to verify the system is ready. |
| 496 | |
| 497 | Quickstart |
| 498 | ~~~~~~~~~~ |
| 499 | |
| 500 | From source tree, type ``make docker-help`` to see the help. Testing |
| 501 | can be started without configuring or building QEMU (``configure`` and |
| 502 | ``make`` are done in the container, with parameters defined by the |
| 503 | make target): |
| 504 | |
| 505 | .. code:: |
| 506 | |
| 507 | make docker-test-build@debian |
| 508 | |
| 509 | This will create a container instance using the ``debian`` image (the image |
| 510 | is downloaded and initialized automatically), in which the ``test-build`` job |
| 511 | is executed. |
| 512 | |
| 513 | Registry |
| 514 | ~~~~~~~~ |
| 515 | |
| 516 | The QEMU project has a container registry hosted by GitLab at |
| 517 | ``registry.gitlab.com/qemu-project/qemu`` which will automatically be |
| 518 | used to pull in pre-built layers. This avoids unnecessary strain on |
| 519 | the distro archives created by multiple developers running the same |
| 520 | container build steps over and over again. This can be overridden |
| 521 | locally by using the ``NOCACHE`` build option: |
| 522 | |
| 523 | .. code:: |
| 524 | |
| 525 | make docker-image-debian-arm64-cross NOCACHE=1 |
| 526 | |
| 527 | Images |
| 528 | ~~~~~~ |
| 529 | |
| 530 | Along with many other images, the ``debian`` image is defined in a Dockerfile |
| 531 | in ``tests/docker/dockerfiles/``, called ``debian.docker``. ``make docker-help`` |
| 532 | command will list all the available images. |
| 533 | |
| 534 | A ``.pre`` script can be added beside the ``.docker`` file, which will be |
| 535 | executed before building the image under the build context directory. This is |
| 536 | mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, |
| 537 | for example, to make qemu-user powered cross build containers work. |
| 538 | |
| 539 | Most of the existing Dockerfiles were written by hand, simply by creating a |
| 540 | a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory. |
| 541 | This has led to an inconsistent set of packages being present across the |
| 542 | different containers. |
| 543 | |
| 544 | Thus going forward, QEMU is aiming to automatically generate the Dockerfiles |
| 545 | using the ``lcitool`` program provided by the ``libvirt-ci`` project: |
| 546 | |
| 547 | https://gitlab.com/libvirt/libvirt-ci |
| 548 | |
| 549 | ``libvirt-ci`` contains an ``lcitool`` program as well as a list of |
| 550 | mappings to distribution package names for a wide variety of third |
| 551 | party projects. ``lcitool`` applies the mappings to a list of build |
| 552 | pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the |
| 553 | list of native packages to install on each distribution, and uses them |
| 554 | to generate build environments (dockerfiles) that are consistent across OS |
| 555 | distribution. |
| 556 | |
| 557 | |
| 558 | Adding new build pre-requisites |
| 559 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 560 | |
| 561 | When preparing a patch series that adds a new build |
| 562 | pre-requisite to QEMU, the prerequisites should to be added to |
| 563 | ``tests/lcitool/projects/qemu.yml`` in order to make the dependency |
| 564 | available in the CI build environments. |
| 565 | |
| 566 | In the simple case where the pre-requisite is already known to ``libvirt-ci`` |
| 567 | the following steps are needed: |
| 568 | |
| 569 | * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite |
| 570 | |
| 571 | * Run ``make lcitool-refresh`` to re-generate all relevant build environment |
| 572 | manifests |
| 573 | |
| 574 | It may be that ``libvirt-ci`` does not know about the new pre-requisite. |
| 575 | If that is the case, some extra preparation steps will be required |
| 576 | first to contribute the mapping to the ``libvirt-ci`` project: |
| 577 | |
| 578 | * Fork the ``libvirt-ci`` project on gitlab |
| 579 | |
| 580 | * Add an entry for the new build prerequisite to |
| 581 | ``lcitool/facts/mappings.yml``, listing its native package name on as |
| 582 | many OS distros as practical. Run ``python -m pytest --regenerate-output`` |
| 583 | and check that the changes are correct. |
| 584 | |
| 585 | * Commit the ``mappings.yml`` change together with the regenerated test |
| 586 | files, and submit a merge request to the ``libvirt-ci`` project. |
| 587 | Please note in the description that this is a new build pre-requisite |
| 588 | desired for use with QEMU. |
| 589 | |
| 590 | * CI pipeline will run to validate that the changes to ``mappings.yml`` |
| 591 | are correct, by attempting to install the newly listed package on |
| 592 | all OS distributions supported by ``libvirt-ci``. |
| 593 | |
| 594 | * Once the merge request is accepted, go back to QEMU and update |
| 595 | the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that |
| 596 | contains the ``mappings.yml`` update. Then add the prerequisite and |
| 597 | run ``make lcitool-refresh``. |
| 598 | |
| 599 | * Please also trigger gitlab container generation pipelines on your change |
| 600 | for as many OS distros as practical to make sure that there are no |
| 601 | obvious breakages when adding the new pre-requisite. Please see |
| 602 | `CI <https://www.qemu.org/docs/master/devel/ci.html>`__ documentation |
| 603 | page on how to trigger gitlab CI pipelines on your change. |
| 604 | |
| 605 | For enterprise distros that default to old, end-of-life versions of the |
| 606 | Python runtime, QEMU uses a separate set of mappings that work with more |
| 607 | recent versions. These can be found in ``tests/lcitool/mappings.yml``. |
| 608 | Modifying this file should not be necessary unless the new pre-requisite |
| 609 | is a Python library or tool. |
| 610 | |
| 611 | |
| 612 | Adding new OS distros |
| 613 | ^^^^^^^^^^^^^^^^^^^^^ |
| 614 | |
| 615 | In some cases ``libvirt-ci`` will not know about the OS distro that is |
| 616 | desired to be tested. Before adding a new OS distro, discuss the proposed |
| 617 | addition: |
| 618 | |
| 619 | * Send a mail to qemu-devel, copying people listed in the |
| 620 | MAINTAINERS file for ``Build and test automation``. |
| 621 | |
| 622 | There are limited CI compute resources available to QEMU, so the |
| 623 | cost/benefit tradeoff of adding new OS distros needs to be considered. |
| 624 | |
| 625 | * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues |
| 626 | pointing to the qemu-devel mail thread in the archives. |
| 627 | |
| 628 | This alerts other people who might be interested in the work |
| 629 | to avoid duplication, as well as to get feedback from libvirt-ci |
| 630 | maintainers on any tips to ease the addition |
| 631 | |
| 632 | Assuming there is agreement to add a new OS distro then |
| 633 | |
| 634 | * Fork the ``libvirt-ci`` project on gitlab |
| 635 | |
| 636 | * Add metadata under ``lcitool/facts/targets/`` for the new OS |
| 637 | distro. There might be code changes required if the OS distro |
| 638 | uses a package format not currently known. The ``libvirt-ci`` |
| 639 | maintainers can advise on this when the issue is filed. |
| 640 | |
| 641 | * Edit the ``lcitool/facts/mappings.yml`` change to add entries for |
| 642 | the new OS, listing the native package names for as many packages |
| 643 | as practical. Run ``python -m pytest --regenerate-output`` and |
| 644 | check that the changes are correct. |
| 645 | |
| 646 | * Commit the changes to ``lcitool/facts`` and the regenerated test |
| 647 | files, and submit a merge request to the ``libvirt-ci`` project. |
| 648 | Please note in the description that this is a new build pre-requisite |
| 649 | desired for use with QEMU |
| 650 | |
| 651 | * CI pipeline will run to validate that the changes to ``mappings.yml`` |
| 652 | are correct, by attempting to install the newly listed package on |
| 653 | all OS distributions supported by ``libvirt-ci``. |
| 654 | |
| 655 | * Once the merge request is accepted, go back to QEMU and update |
| 656 | the ``libvirt-ci`` submodule to point to a commit that contains |
| 657 | the ``mappings.yml`` update. |
| 658 | |
| 659 | |
| 660 | Tests |
| 661 | ~~~~~ |
| 662 | |
| 663 | Different tests are added to cover various configurations to build and test |
| 664 | QEMU. Docker tests are the executables under ``tests/docker`` named |
| 665 | ``test-*``. They are typically shell scripts and are built on top of a shell |
| 666 | library, ``tests/docker/common.rc``, which provides helpers to find the QEMU |
| 667 | source and build it. |
| 668 | |
| 669 | The full list of tests is printed in the ``make docker-help`` help. |
| 670 | |
| 671 | Debugging a Docker test failure |
| 672 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 673 | |
| 674 | When CI tasks, maintainers or yourself report a Docker test failure, follow the |
| 675 | below steps to debug it: |
| 676 | |
| 677 | 1. Locally reproduce the failure with the reported command line. E.g. run |
| 678 | ``make docker-test-mingw@fedora-win64-cross J=8``. |
| 679 | 2. Add "V=1" to the command line, try again, to see the verbose output. |
| 680 | 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt |
| 681 | in the container right before testing starts. You could either manually |
| 682 | build QEMU and run tests from there, or press :kbd:`Ctrl+d` to let the Docker |
| 683 | testing continue. |
| 684 | 4. If you press :kbd:`Ctrl+d`, the same building and testing procedure will begin, and |
| 685 | will hopefully run into the error again. After that, you will be dropped to |
| 686 | the prompt for debug. |
| 687 | |
| 688 | Options |
| 689 | ~~~~~~~ |
| 690 | |
| 691 | Various options can be used to affect how Docker tests are done. The full |
| 692 | list is in the ``make docker`` help text. The frequently used ones are: |
| 693 | |
| 694 | * ``V=1``: the same as in top level ``make``. It will be propagated to the |
| 695 | container and enable verbose output. |
| 696 | * ``J=$N``: the number of parallel tasks in make commands in the container, |
| 697 | similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in |
| 698 | top level ``make`` will not be propagated into the container.) |
| 699 | * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test |
| 700 | failure" section. |
| 701 | |
| 702 | Thread Sanitizer |
| 703 | ---------------- |
| 704 | |
| 705 | Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports |
| 706 | building and testing with this tool. |
| 707 | |
| 708 | For more information on TSan: |
| 709 | |
| 710 | https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual |
| 711 | |
| 712 | Thread Sanitizer in Docker |
| 713 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 714 | TSan is currently supported in all our host docker images (for instance, ubuntu2404). |
| 715 | |
| 716 | The test-tsan test will build using TSan and then run make check. |
| 717 | |
| 718 | .. code:: |
| 719 | |
| 720 | make docker-test-tsan@ubuntu2404 |
| 721 | |
| 722 | TSan warnings under docker are placed in files located at build/tsan/. |
| 723 | |
| 724 | We recommend using DEBUG=1 to allow launching the test from inside the docker, |
| 725 | and to allow review of the warnings generated by TSan. |
| 726 | |
| 727 | Building and Testing with TSan |
| 728 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 729 | |
| 730 | It is possible to build and test with TSan, with a few additional steps. |
| 731 | These steps are normally done automatically in the docker. |
| 732 | |
| 733 | TSan is supported for clang and gcc. |
| 734 | One particularity of sanitizers is that all the code, including shared objects |
| 735 | dependencies, should be built with it. |
| 736 | In the case of TSan, any synchronization primitive from glib (GMutex for |
| 737 | instance) will not be recognized, and will lead to false positives. |
| 738 | |
| 739 | To build a tsan version of glib: |
| 740 | |
| 741 | .. code:: |
| 742 | |
| 743 | $ git clone --depth=1 --branch=2.81.0 https://github.com/GNOME/glib.git |
| 744 | $ cd glib |
| 745 | $ CFLAGS="-O2 -g -fsanitize=thread" meson build |
| 746 | $ ninja -C build |
| 747 | |
| 748 | To configure the build for TSan: |
| 749 | |
| 750 | .. code:: |
| 751 | |
| 752 | ../configure --enable-tsan \ |
| 753 | --disable-werror --extra-cflags="-O0" |
| 754 | |
| 755 | When executing qemu, don't forget to point to tsan glib: |
| 756 | |
| 757 | .. code:: |
| 758 | |
| 759 | $ glib_dir=/path/to/glib |
| 760 | $ export LD_LIBRARY_PATH=$glib_dir/build/gio:$glib_dir/build/glib:$glib_dir/build/gmodule:$glib_dir/build/gobject:$glib_dir/build/gthread |
| 761 | # check correct version is used |
| 762 | $ ldd build/qemu-x86_64 | grep glib |
| 763 | $ qemu-system-x86_64 ... |
| 764 | |
| 765 | The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment |
| 766 | variable. |
| 767 | |
| 768 | More information on the TSAN_OPTIONS can be found here: |
| 769 | |
| 770 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags |
| 771 | |
| 772 | For example: |
| 773 | |
| 774 | .. code:: |
| 775 | |
| 776 | export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \ |
| 777 | detect_deadlocks=false history_size=7 exitcode=0 \ |
| 778 | log_path=<build path>/tsan/tsan_warning |
| 779 | |
| 780 | The above exitcode=0 has TSan continue without error if any warnings are found. |
| 781 | This allows for running the test and then checking the warnings afterwards. |
| 782 | If you want TSan to stop and exit with error on warnings, use exitcode=66. |
| 783 | |
| 784 | .. _tsan-suppressions: |
| 785 | |
| 786 | TSan Suppressions |
| 787 | ~~~~~~~~~~~~~~~~~ |
| 788 | Keep in mind that for any data race warning, although there might be a data race |
| 789 | detected by TSan, there might be no actual bug here. TSan provides several |
| 790 | different mechanisms for suppressing warnings. In general it is recommended |
| 791 | to fix the code if possible to eliminate the data race rather than suppress |
| 792 | the warning. |
| 793 | |
| 794 | A few important files for suppressing warnings are: |
| 795 | |
| 796 | tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime. |
| 797 | The comment on each suppression will typically indicate why we are |
| 798 | suppressing it. More information on the file format can be found here: |
| 799 | |
| 800 | https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions |
| 801 | |
| 802 | tests/tsan/ignore.tsan - Has TSan warnings we wish to disable |
| 803 | at compile time for test or debug. |
| 804 | Add flags to configure to enable: |
| 805 | |
| 806 | "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/ignore.tsan" |
| 807 | |
| 808 | More information on the file format can be found here under "Blacklist Format": |
| 809 | |
| 810 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags |
| 811 | |
| 812 | TSan Annotations |
| 813 | ~~~~~~~~~~~~~~~~ |
| 814 | include/qemu/tsan.h defines annotations. See this file for more descriptions |
| 815 | of the annotations themselves. Annotations can be used to suppress |
| 816 | TSan warnings or give TSan more information so that it can detect proper |
| 817 | relationships between accesses of data. |
| 818 | |
| 819 | Annotation examples can be found here: |
| 820 | |
| 821 | https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/ |
| 822 | |
| 823 | Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp |
| 824 | |
| 825 | The full set of annotations can be found here: |
| 826 | |
| 827 | https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp |
| 828 | |
| 829 | docker-binfmt-image-debian-% targets |
| 830 | ------------------------------------ |
| 831 | |
| 832 | It is possible to combine Debian's bootstrap scripts with a configured |
| 833 | ``binfmt_misc`` to bootstrap a number of Debian's distros including |
| 834 | experimental ports not yet supported by a released OS. This can |
| 835 | simplify setting up a rootfs by using docker to contain the foreign |
| 836 | rootfs rather than manually invoking chroot. |
| 837 | |
| 838 | Setting up ``binfmt_misc`` |
| 839 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 840 | |
| 841 | You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU |
| 842 | user binary to automatically run binaries for the foreign |
| 843 | architecture. While the scripts will try their best to work with |
| 844 | dynamically linked QEMU's a statically linked one will present less |
| 845 | potential complications when copying into the docker image. Modern |
| 846 | kernels support the ``F`` (fix binary) flag which will open the QEMU |
| 847 | executable on setup and avoids the need to find and re-open in the |
| 848 | chroot environment. This is triggered with the ``--persistent`` flag. |
| 849 | |
| 850 | Example invocation |
| 851 | ~~~~~~~~~~~~~~~~~~ |
| 852 | |
| 853 | For example to setup the HPPA ports builds of Debian:: |
| 854 | |
| 855 | make docker-binfmt-image-debian-sid-hppa \ |
| 856 | DEB_TYPE=sid DEB_ARCH=hppa \ |
| 857 | DEB_URL=http://ftp.ports.debian.org/debian-ports/ \ |
| 858 | DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \ |
| 859 | EXECUTABLE=(pwd)/qemu-hppa V=1 |
| 860 | |
| 861 | The ``DEB_`` variables are substitutions used by |
| 862 | ``debian-bootstrap.pre`` which is called to do the initial debootstrap |
| 863 | of the rootfs before it is copied into the container. The second stage |
| 864 | is run as part of the build. The final image will be tagged as |
| 865 | ``qemu/debian-sid-hppa``. |
| 866 | |
| 867 | VM testing |
| 868 | ---------- |
| 869 | |
| 870 | This test suite contains scripts that bootstrap various guest images that have |
| 871 | necessary packages to build QEMU. The basic usage is documented in ``Makefile`` |
| 872 | help which is displayed with ``make vm-help``. |
| 873 | |
| 874 | Quickstart |
| 875 | ~~~~~~~~~~ |
| 876 | |
| 877 | Run ``make vm-help`` to list available make targets. Invoke a specific make |
| 878 | command to run build test in an image. For example, ``make vm-build-freebsd`` |
| 879 | will build the source tree in the FreeBSD image. The command can be executed |
| 880 | from either the source tree or the build dir; if the former, ``./configure`` is |
| 881 | not needed. The command will then generate the test image in ``./tests/vm/`` |
| 882 | under the working directory. |
| 883 | |
| 884 | Note: images created by the scripts accept a well-known RSA key pair for SSH |
| 885 | access, so they SHOULD NOT be exposed to external interfaces if you are |
| 886 | concerned about attackers taking control of the guest and potentially |
| 887 | exploiting a QEMU security bug to compromise the host. |
| 888 | |
| 889 | QEMU binaries |
| 890 | ~~~~~~~~~~~~~ |
| 891 | |
| 892 | By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If |
| 893 | there isn't one, or if it is older than 2.10, the test won't work. In this case, |
| 894 | provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. |
| 895 | |
| 896 | Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable. |
| 897 | |
| 898 | Make jobs |
| 899 | ~~~~~~~~~ |
| 900 | |
| 901 | The ``-j$X`` option in the make command line is not propagated into the VM, |
| 902 | specify ``J=$X`` to control the make jobs in the guest. |
| 903 | |
| 904 | Debugging |
| 905 | ~~~~~~~~~ |
| 906 | |
| 907 | Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive |
| 908 | debugging and verbose output. If this is not enough, see the next section. |
| 909 | ``V=1`` will be propagated down into the make jobs in the guest. |
| 910 | |
| 911 | Manual invocation |
| 912 | ~~~~~~~~~~~~~~~~~ |
| 913 | |
| 914 | Each guest script is an executable script with the same command line options. |
| 915 | For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: |
| 916 | |
| 917 | .. code:: |
| 918 | |
| 919 | $ cd $QEMU_SRC/tests/vm |
| 920 | |
| 921 | # To bootstrap the image |
| 922 | $ ./netbsd --build-image --image /var/tmp/netbsd.img |
| 923 | <...> |
| 924 | |
| 925 | # To run an arbitrary command in guest (the output will not be echoed unless |
| 926 | # --debug is added) |
| 927 | $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a |
| 928 | |
| 929 | # To build QEMU in guest |
| 930 | $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC |
| 931 | |
| 932 | # To get to an interactive shell |
| 933 | $ ./netbsd --interactive --image /var/tmp/netbsd.img sh |
| 934 | |
| 935 | Adding new guests |
| 936 | ~~~~~~~~~~~~~~~~~ |
| 937 | |
| 938 | Please look at existing guest scripts for how to add new guests. |
| 939 | |
| 940 | Most importantly, create a subclass of BaseVM and implement ``build_image()`` |
| 941 | method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from |
| 942 | the script's ``main()``. |
| 943 | |
| 944 | * Usually in ``build_image()``, a template image is downloaded from a |
| 945 | predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and |
| 946 | the checksum, so consider using it. |
| 947 | |
| 948 | * Once the image is downloaded, users, SSH server and QEMU build deps should |
| 949 | be set up: |
| 950 | |
| 951 | - Root password set to ``BaseVM.ROOT_PASS`` |
| 952 | - User ``BaseVM.GUEST_USER`` is created, and password set to |
| 953 | ``BaseVM.GUEST_PASS`` |
| 954 | - SSH service is enabled and started on boot, |
| 955 | ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys`` |
| 956 | file of both root and the normal user |
| 957 | - DHCP client service is enabled and started on boot, so that it can |
| 958 | automatically configure the virtio-net-pci NIC and communicate with QEMU |
| 959 | user net (10.0.2.2) |
| 960 | - Necessary packages are installed to untar the source tarball and build |
| 961 | QEMU |
| 962 | |
| 963 | * Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that |
| 964 | untars a raw virtio-blk block device, which is the tarball data blob of the |
| 965 | QEMU source tree, then configure/build it. Running "make check" is also |
| 966 | recommended. |
| 967 | |
| 968 | Image fuzzer testing |
| 969 | -------------------- |
| 970 | |
| 971 | An image fuzzer was added to exercise format drivers. Currently only qcow2 is |
| 972 | supported. To start the fuzzer, run |
| 973 | |
| 974 | .. code:: |
| 975 | |
| 976 | tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2 |
| 977 | |
| 978 | Alternatively, some command different from ``qemu-img info`` can be tested, by |
| 979 | changing the ``-c`` option. |
| 980 | |
| 981 | Functional tests using Python |
| 982 | ----------------------------- |
| 983 | |
| 984 | A functional test focuses on the functional requirement of the software, |
| 985 | attempting to find errors like incorrect functions, interface errors, |
| 986 | behavior errors, and initialization and termination errors [3]_. |
| 987 | |
| 988 | The ``tests/functional`` directory hosts functional tests written in |
| 989 | Python. You can run the functional tests simply by executing: |
| 990 | |
| 991 | .. code:: |
| 992 | |
| 993 | make check-functional |
| 994 | |
| 995 | See :ref:`checkfunctional-ref` for more details. |
| 996 | |
| 997 | The harness for the functional tests also honours the |
| 998 | ``TIMEOUT_MULTIPLIER`` environment variable. |
| 999 | |
| 1000 | .. _checktcg-ref: |
| 1001 | |
| 1002 | Testing with "make check-tcg" |
| 1003 | ----------------------------- |
| 1004 | |
| 1005 | The check-tcg tests are intended for simple smoke tests of both |
| 1006 | linux-user and softmmu TCG functionality. However to build test |
| 1007 | programs for guest targets you need to have cross compilers available. |
| 1008 | If your distribution supports cross compilers you can do something as |
| 1009 | simple as:: |
| 1010 | |
| 1011 | apt install gcc-aarch64-linux-gnu |
| 1012 | |
| 1013 | The configure script will automatically pick up their presence. |
| 1014 | Sometimes compilers have slightly odd names so the availability of |
| 1015 | them can be prompted by passing in the appropriate configure option |
| 1016 | for the architecture in question, for example:: |
| 1017 | |
| 1018 | $(configure) --cross-cc-aarch64=aarch64-cc |
| 1019 | |
| 1020 | There is also a ``--cross-cc-cflags-ARCH`` flag in case additional |
| 1021 | compiler flags are needed to build for a given target. |
| 1022 | |
| 1023 | If you have the ability to run containers as the user the build system |
| 1024 | will automatically use them where no system compiler is available. For |
| 1025 | architectures where we also support building QEMU we will generally |
| 1026 | use the same container to build tests. However there are a number of |
| 1027 | additional containers defined that have a minimal cross-build |
| 1028 | environment that is only suitable for building test cases. Sometimes |
| 1029 | we may use a bleeding edge distribution for compiler features needed |
| 1030 | for test cases that aren't yet in the LTS distros we support for QEMU |
| 1031 | itself. |
| 1032 | |
| 1033 | See :ref:`container-ref` for more details. |
| 1034 | |
| 1035 | Running subset of tests |
| 1036 | ~~~~~~~~~~~~~~~~~~~~~~~ |
| 1037 | |
| 1038 | You can build the tests for one architecture:: |
| 1039 | |
| 1040 | make build-tcg-tests-$TARGET |
| 1041 | |
| 1042 | And run with:: |
| 1043 | |
| 1044 | make run-tcg-tests-$TARGET |
| 1045 | |
| 1046 | Adding ``V=1`` to the invocation will show the details of how to |
| 1047 | invoke QEMU for the test which is useful for debugging tests. |
| 1048 | |
| 1049 | Running individual tests |
| 1050 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
| 1051 | |
| 1052 | Tests can also be run directly from the test build directory. If you |
| 1053 | run ``make help`` from the test build directory you will get a list of |
| 1054 | all the tests that can be run. Please note that same binaries are used |
| 1055 | in multiple tests, for example:: |
| 1056 | |
| 1057 | make run-plugin-test-mmap-with-libinline.so |
| 1058 | |
| 1059 | will run the mmap test with the ``libinline.so`` TCG plugin. The |
| 1060 | gdbstub tests also re-use the test binaries but while exercising gdb. |
| 1061 | |
| 1062 | TCG test dependencies |
| 1063 | ~~~~~~~~~~~~~~~~~~~~~ |
| 1064 | |
| 1065 | The TCG tests are deliberately very light on dependencies and are |
| 1066 | either totally bare with minimal gcc lib support (for system-mode tests) |
| 1067 | or just glibc (for linux-user tests). This is because getting a cross |
| 1068 | compiler to work with additional libraries can be challenging. |
| 1069 | |
| 1070 | Other TCG Tests |
| 1071 | --------------- |
| 1072 | |
| 1073 | There are a number of out-of-tree test suites that are used for more |
| 1074 | extensive testing of processor features. |
| 1075 | |
| 1076 | KVM Unit Tests |
| 1077 | ~~~~~~~~~~~~~~ |
| 1078 | |
| 1079 | The KVM unit tests are designed to run as a Guest OS under KVM but |
| 1080 | there is no reason why they can't exercise the TCG as well. It |
| 1081 | provides a minimal OS kernel with hooks for enabling the MMU as well |
| 1082 | as reporting test results via a special device:: |
| 1083 | |
| 1084 | https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git |
| 1085 | |
| 1086 | Linux Test Project |
| 1087 | ~~~~~~~~~~~~~~~~~~ |
| 1088 | |
| 1089 | The LTP is focused on exercising the syscall interface of a Linux |
| 1090 | kernel. It checks that syscalls behave as documented and strives to |
| 1091 | exercise as many corner cases as possible. It is a useful test suite |
| 1092 | to run to exercise QEMU's linux-user code:: |
| 1093 | |
| 1094 | https://linux-test-project.github.io/ |
| 1095 | |
| 1096 | GCC gcov support |
| 1097 | ---------------- |
| 1098 | |
| 1099 | ``gcov`` is a GCC tool to analyze the testing coverage by |
| 1100 | instrumenting the tested code. To use it, configure QEMU with |
| 1101 | ``--enable-gcov`` option and build. Then run the tests as usual. |
| 1102 | |
| 1103 | If you want to gather coverage information on a single test the ``make |
| 1104 | clean-gcda`` target can be used to delete any existing coverage |
| 1105 | information before running a single test. |
| 1106 | |
| 1107 | You can generate a HTML coverage report by executing ``make |
| 1108 | coverage-html`` which will create |
| 1109 | ``meson-logs/coveragereport/index.html``. |
| 1110 | |
| 1111 | Further analysis can be conducted by running the ``gcov`` command |
| 1112 | directly on the various .gcda output files. Please read the ``gcov`` |
| 1113 | documentation for more information. |
| 1114 | |
| 1115 | Flaky tests |
| 1116 | ----------- |
| 1117 | |
| 1118 | A flaky test is defined as a test that exhibits both a passing and a failing |
| 1119 | result with the same code on different runs. Some usual reasons for an |
| 1120 | intermittent/flaky test are async wait, concurrency, and test order dependency |
| 1121 | [4]_. |
| 1122 | |
| 1123 | In QEMU, tests that are identified to be flaky are normally disabled by |
| 1124 | default. Set the QEMU_TEST_FLAKY_TESTS environment variable before running |
| 1125 | the tests to enable them. |
| 1126 | |
| 1127 | References |
| 1128 | ---------- |
| 1129 | |
| 1130 | .. [1] Sommerville, Ian (2016). Software Engineering. p. 233. |
| 1131 | .. [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, |
| 1132 | A Practitioner’s Approach. p. 48, 376, 378, 381. |
| 1133 | .. [3] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, |
| 1134 | A Practitioner’s Approach. p. 388. |
| 1135 | .. [4] Luo, Qingzhou, et al. An empirical analysis of flaky tests. |
| 1136 | Proceedings of the 22nd ACM SIGSOFT International Symposium on |
| 1137 | Foundations of Software Engineering. 2014. |