| 1 | .. _checkfunctional-ref: |
| 2 | |
| 3 | Functional testing with Python |
| 4 | ============================== |
| 5 | |
| 6 | The ``tests/functional`` directory hosts functional tests written in |
| 7 | Python. They are usually higher level tests, and may interact with |
| 8 | external resources and with various guest operating systems. |
| 9 | |
| 10 | The tests should be written in the style of the Python `unittest`_ framework, |
| 11 | using stdio for the TAP protocol. The folder ``tests/functional/qemu_test`` |
| 12 | provides classes (e.g. the ``QemuBaseTest``, ``QemuUserTest`` and the |
| 13 | ``QemuSystemTest`` classes) and utility functions that help to get your test |
| 14 | into the right shape, e.g. by replacing the 'stdout' python object to redirect |
| 15 | the normal output of your test to stderr instead. |
| 16 | |
| 17 | Note that if you don't use one of the QemuBaseTest based classes for your |
| 18 | test, or if you spawn subprocesses from your test, you have to make sure |
| 19 | that there is no TAP-incompatible output written to stdio, e.g. either by |
| 20 | prefixing every line with a "# " to mark the output as a TAP comment, or |
| 21 | e.g. by capturing the stdout output of subprocesses (redirecting it to |
| 22 | stderr is OK). |
| 23 | |
| 24 | Tests based on ``qemu_test.QemuSystemTest`` can easily: |
| 25 | |
| 26 | * Customize the command line arguments given to the convenience |
| 27 | ``self.vm`` attribute (a QEMUMachine instance) |
| 28 | |
| 29 | * Interact with the QEMU monitor, send QMP commands and check |
| 30 | their results |
| 31 | |
| 32 | * Interact with the guest OS, using the convenience console device |
| 33 | (which may be useful to assert the effectiveness and correctness of |
| 34 | command line arguments or QMP commands) |
| 35 | |
| 36 | * Download (and cache) remote data files, such as firmware and kernel |
| 37 | images |
| 38 | |
| 39 | Running tests |
| 40 | ------------- |
| 41 | |
| 42 | You can run the functional tests simply by executing: |
| 43 | |
| 44 | .. code:: |
| 45 | |
| 46 | make check-functional |
| 47 | |
| 48 | It is also possible to run tests for a certain target only, for example |
| 49 | the following line will only run the tests for the x86_64 target: |
| 50 | |
| 51 | .. code:: |
| 52 | |
| 53 | make check-functional-x86_64 |
| 54 | |
| 55 | To run a single test file without the meson test runner, you can also |
| 56 | execute the file directly by specifying the name of the emulator target |
| 57 | binary as an env variable. |
| 58 | |
| 59 | Assuming the current working directory is the top level source checkout |
| 60 | and the build directory is './build':: |
| 61 | |
| 62 | $ export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64 |
| 63 | |
| 64 | Run all tests from a test file:: |
| 65 | |
| 66 | $ ./build/run tests/functional/x86_64/test_virtio_version.py |
| 67 | |
| 68 | Run all tests from a test class:: |
| 69 | |
| 70 | $ ./build/run tests/functional/x86_64/test_virtio_version.py VirtioVersionCheck |
| 71 | |
| 72 | Or a single test:: |
| 73 | |
| 74 | $ ./build/run tests/functional/x86_64/test_virtio_version.py VirtioVersionCheck.test_modern_only_devs |
| 75 | |
| 76 | Filtering test names also works:: |
| 77 | |
| 78 | $ ./build/run tests/functional/x86_64/test_virtio_version.py -k modern |
| 79 | |
| 80 | The test framework will automatically purge any scratch files created during |
| 81 | the tests. If needing to debug a failed test, it is possible to keep these |
| 82 | files around on disk by setting ``QEMU_TEST_KEEP_SCRATCH=1`` as an env |
| 83 | variable. Any preserved files will be deleted the next time the test is run |
| 84 | without this variable set. |
| 85 | |
| 86 | Logging |
| 87 | ------- |
| 88 | |
| 89 | The framework collects log files for each test in the build directory |
| 90 | in the following subfolder:: |
| 91 | |
| 92 | <builddir>/tests/functional/<arch>/<fileid>.<classid>.<testname>/ |
| 93 | |
| 94 | There are usually three log files: |
| 95 | |
| 96 | * ``base.log`` contains the generic logging information that is written |
| 97 | by the calls to the logging functions in the test code (e.g. by calling |
| 98 | the ``self.log.info()`` or ``self.log.debug()`` functions). |
| 99 | * ``console.log`` contains the output of the serial console of the guest. |
| 100 | * ``default.log`` contains the output of QEMU. This file could be named |
| 101 | differently if the test chooses to use a different identifier for |
| 102 | the guest VM (e.g. when the test spins up multiple VMs). |
| 103 | |
| 104 | Introduction to writing tests |
| 105 | ----------------------------- |
| 106 | |
| 107 | The ``tests/functional/qemu_test`` directory provides the ``qemu_test`` |
| 108 | Python module, containing the ``qemu_test.QemuSystemTest`` class. |
| 109 | Here is a simple usage example: |
| 110 | |
| 111 | .. code:: |
| 112 | |
| 113 | #!/usr/bin/env python3 |
| 114 | |
| 115 | from qemu_test import QemuSystemTest |
| 116 | |
| 117 | class Version(QemuSystemTest): |
| 118 | |
| 119 | def test_qmp_human_info_version(self): |
| 120 | self.vm.launch() |
| 121 | res = self.vm.cmd('human-monitor-command', |
| 122 | command_line='info version') |
| 123 | self.assertRegex(res, r'^(\d+\.\d+\.\d)') |
| 124 | |
| 125 | if __name__ == '__main__': |
| 126 | QemuSystemTest.main() |
| 127 | |
| 128 | By providing the "hash bang" line at the beginning of the script, marking |
| 129 | the file as executable and by calling into QemuSystemTest.main(), the test |
| 130 | can also be run stand-alone, without a test runner. OTOH when run via a test |
| 131 | runner, the QemuSystemTest.main() function takes care of running the test |
| 132 | functions in the right fassion (e.g. with TAP output that is required by the |
| 133 | meson test runner). |
| 134 | |
| 135 | The ``qemu_test.QemuSystemTest`` base test class |
| 136 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 137 | |
| 138 | The ``qemu_test.QemuSystemTest`` class has a number of characteristics |
| 139 | that are worth being mentioned. |
| 140 | |
| 141 | First of all, it attempts to give each test a ready to use QEMUMachine |
| 142 | instance, available at ``self.vm``. Because many tests will tweak the |
| 143 | QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``) |
| 144 | is left to the test writer. |
| 145 | |
| 146 | The base test class has also support for tests with more than one |
| 147 | QEMUMachine. The way to get machines is through the ``self.get_vm()`` |
| 148 | method which will return a QEMUMachine instance. The ``self.get_vm()`` |
| 149 | method accepts arguments that will be passed to the QEMUMachine creation |
| 150 | and also an optional ``name`` attribute so you can identify a specific |
| 151 | machine and get it more than once through the tests methods. A simple |
| 152 | and hypothetical example follows: |
| 153 | |
| 154 | .. code:: |
| 155 | |
| 156 | from qemu_test import QemuSystemTest |
| 157 | |
| 158 | class MultipleMachines(QemuSystemTest): |
| 159 | def test_multiple_machines(self): |
| 160 | first_machine = self.get_vm() |
| 161 | second_machine = self.get_vm() |
| 162 | self.get_vm(name='third_machine').launch() |
| 163 | |
| 164 | first_machine.launch() |
| 165 | second_machine.launch() |
| 166 | |
| 167 | first_res = first_machine.cmd( |
| 168 | 'human-monitor-command', |
| 169 | command_line='info version') |
| 170 | |
| 171 | second_res = second_machine.cmd( |
| 172 | 'human-monitor-command', |
| 173 | command_line='info version') |
| 174 | |
| 175 | third_res = self.get_vm(name='third_machine').cmd( |
| 176 | 'human-monitor-command', |
| 177 | command_line='info version') |
| 178 | |
| 179 | self.assertEqual(first_res, second_res, third_res) |
| 180 | |
| 181 | At test "tear down", ``qemu_test.QemuSystemTest`` handles all the QEMUMachines |
| 182 | shutdown. |
| 183 | |
| 184 | QEMUMachine |
| 185 | ----------- |
| 186 | |
| 187 | The QEMUMachine API is already widely used in the Python iotests, |
| 188 | device-crash-test and other Python scripts. It's a wrapper around the |
| 189 | execution of a QEMU binary, giving its users: |
| 190 | |
| 191 | * the ability to set command line arguments to be given to the QEMU |
| 192 | binary |
| 193 | |
| 194 | * a ready to use QMP connection and interface, which can be used to |
| 195 | send commands and inspect its results, as well as asynchronous |
| 196 | events |
| 197 | |
| 198 | * convenience methods to set commonly used command line arguments in |
| 199 | a more succinct and intuitive way |
| 200 | |
| 201 | QEMU binary selection |
| 202 | ^^^^^^^^^^^^^^^^^^^^^ |
| 203 | |
| 204 | The QEMU binary used for the ``self.vm`` QEMUMachine instance will |
| 205 | primarily depend on the value of the ``qemu_bin`` instance attribute. |
| 206 | If it is not explicitly set by the test code, its default value will |
| 207 | be the result the QEMU_TEST_QEMU_BINARY environment variable. |
| 208 | |
| 209 | Debugging hung QEMU |
| 210 | ^^^^^^^^^^^^^^^^^^^ |
| 211 | |
| 212 | When test cases go wrong it may be helpful to debug a stalled QEMU |
| 213 | process. While the QEMUMachine class owns the primary QMP monitor |
| 214 | socket, it is possible to request a second QMP monitor be created |
| 215 | by setting the ``QEMU_TEST_QMP_BACKDOOR`` env variable to refer |
| 216 | to a UNIX socket name. The ``qmp-shell`` command can then be |
| 217 | attached to the stalled QEMU to examine its live state. |
| 218 | |
| 219 | Attribute reference |
| 220 | ------------------- |
| 221 | |
| 222 | QemuBaseTest |
| 223 | ^^^^^^^^^^^^ |
| 224 | |
| 225 | The following attributes are available on any ``qemu_test.QemuBaseTest`` |
| 226 | instance. |
| 227 | |
| 228 | arch |
| 229 | """" |
| 230 | |
| 231 | The target architecture of the QEMU binary. |
| 232 | |
| 233 | Tests are also free to use this attribute value, for their own needs. |
| 234 | A test may, for instance, use this value when selecting the architecture |
| 235 | of a kernel or disk image to boot a VM with. |
| 236 | |
| 237 | qemu_bin |
| 238 | """""""" |
| 239 | |
| 240 | The preserved value of the ``QEMU_TEST_QEMU_BINARY`` environment |
| 241 | variable. |
| 242 | |
| 243 | QemuUserTest |
| 244 | ^^^^^^^^^^^^ |
| 245 | |
| 246 | The QemuUserTest class can be used for running an executable via the |
| 247 | usermode emulation binaries. |
| 248 | |
| 249 | QemuSystemTest |
| 250 | ^^^^^^^^^^^^^^ |
| 251 | |
| 252 | The QemuSystemTest class can be used for running tests via one of the |
| 253 | qemu-system-* binaries. |
| 254 | |
| 255 | vm |
| 256 | "" |
| 257 | |
| 258 | A QEMUMachine instance, initially configured according to the given |
| 259 | ``qemu_bin`` parameter. |
| 260 | |
| 261 | cpu |
| 262 | """ |
| 263 | |
| 264 | The cpu model that will be set to all QEMUMachine instances created |
| 265 | by the test. |
| 266 | |
| 267 | machine |
| 268 | """"""" |
| 269 | |
| 270 | The machine type that will be set to all QEMUMachine instances created |
| 271 | by the test. By using the set_machine() function of the QemuSystemTest |
| 272 | class to set this attribute, you can automatically check whether the |
| 273 | machine is available to skip the test in case it is not built into the |
| 274 | QEMU binary. |
| 275 | |
| 276 | Asset handling |
| 277 | -------------- |
| 278 | |
| 279 | Many functional tests download assets (e.g. Linux kernels, initrds, |
| 280 | firmware images, etc.) from the internet to be able to run tests with |
| 281 | them. This imposes additional challenges to the test framework. |
| 282 | |
| 283 | First there is the problem that some people might not have an |
| 284 | unconstrained internet connection, so such tests should not be run by |
| 285 | default when running ``make check``. To accomplish this situation, |
| 286 | the tests that download files should only be added to the "thorough" |
| 287 | speed mode in the meson.build file, while the "quick" speed mode is |
| 288 | fine for functional tests that can be run without downloading files. |
| 289 | ``make check`` then only runs the quick functional tests along with |
| 290 | the other quick tests from the other test suites. If you choose to |
| 291 | run only ``make check-functional``, the "thorough" tests will be |
| 292 | executed, too. And to run all functional tests along with the others, |
| 293 | you can use something like:: |
| 294 | |
| 295 | make -j$(nproc) check SPEED=thorough |
| 296 | |
| 297 | The second problem with downloading files from the internet are time |
| 298 | constraints. The time for downloading files should not be taken into |
| 299 | account when the test is running and the timeout of the test is ticking |
| 300 | (since downloading can be very slow, depending on the network bandwidth). |
| 301 | This problem is solved by downloading the assets ahead of time, before |
| 302 | the tests are run. This pre-caching is done with the qemu_test.Asset |
| 303 | class. To use it in your test, declare an asset in your test class with |
| 304 | its URL and SHA256 checksum like this:: |
| 305 | |
| 306 | from qemu_test import Asset |
| 307 | |
| 308 | ASSET_somename = Asset( |
| 309 | ('https://www.qemu.org/assets/images/qemu_head_200.png'), |
| 310 | '34b74cad46ea28a2966c1d04e102510daf1fd73e6582b6b74523940d5da029dd') |
| 311 | |
| 312 | In your test function, you can then get the file name of the cached |
| 313 | asset like this:: |
| 314 | |
| 315 | def test_function(self): |
| 316 | file_path = self.ASSET_somename.fetch() |
| 317 | |
| 318 | The pre-caching will be done automatically when running |
| 319 | ``make check-functional`` (but not when running e.g. |
| 320 | ``make check-functional-<target>``). In case you just want to download |
| 321 | the assets without running the tests, you can do so by running:: |
| 322 | |
| 323 | make precache-functional |
| 324 | |
| 325 | The cache is populated in the ``~/.cache/qemu/download`` directory by |
| 326 | default, but the location can be changed by setting the |
| 327 | ``QEMU_TEST_CACHE_DIR`` environment variable. |
| 328 | |
| 329 | To force the test suite to re-download the cache, even if still valid, |
| 330 | set the ``QEMU_TEST_REFRESH_CACHE`` environment variable. |
| 331 | |
| 332 | Skipping tests |
| 333 | -------------- |
| 334 | |
| 335 | Since the test framework is based on the common Python unittest framework, |
| 336 | you can use the usual Python decorators which allow for easily skipping |
| 337 | tests running under certain conditions, for example, on the lack of a binary |
| 338 | on the test system or when the running environment is a CI system. For further |
| 339 | information about those decorators, please refer to: |
| 340 | |
| 341 | https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures |
| 342 | |
| 343 | While the conditions for skipping tests are often specifics of each one, there |
| 344 | are recurring scenarios identified by the QEMU developers and the use of |
| 345 | environment variables became a kind of standard way to enable/disable tests. |
| 346 | |
| 347 | Here is a list of the most used variables: |
| 348 | |
| 349 | QEMU_TEST_ALLOW_LARGE_STORAGE |
| 350 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 351 | Tests which are going to fetch or produce assets considered *large* are not |
| 352 | going to run unless that ``QEMU_TEST_ALLOW_LARGE_STORAGE=1`` is exported on |
| 353 | the environment. |
| 354 | |
| 355 | The definition of *large* is a bit arbitrary here, but it usually means an |
| 356 | asset which occupies at least 1GB of size on disk when uncompressed. |
| 357 | |
| 358 | QEMU_TEST_ALLOW_UNTRUSTED_CODE |
| 359 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 360 | There are tests which will boot a kernel image or firmware that can be |
| 361 | considered not safe to run on the developer's workstation, thus they are |
| 362 | skipped by default. The definition of *not safe* is also arbitrary but |
| 363 | usually it means a blob which either its source or build process aren't |
| 364 | public available. |
| 365 | |
| 366 | You should export ``QEMU_TEST_ALLOW_UNTRUSTED_CODE=1`` on the environment in |
| 367 | order to allow tests which make use of those kind of assets. |
| 368 | |
| 369 | QEMU_TEST_FLAKY_TESTS |
| 370 | ^^^^^^^^^^^^^^^^^^^^^ |
| 371 | Some tests are not working reliably and thus are disabled by default. |
| 372 | This includes tests that don't run reliably on GitLab's CI which |
| 373 | usually expose real issues that are rarely seen on developer machines |
| 374 | due to the constraints of the CI environment. If you encounter a |
| 375 | similar situation then raise a bug and then mark the test as shown on |
| 376 | the code snippet below: |
| 377 | |
| 378 | .. code:: |
| 379 | |
| 380 | # See https://gitlab.com/qemu-project/qemu/-/issues/nnnn |
| 381 | @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') |
| 382 | def test(self): |
| 383 | do_something() |
| 384 | |
| 385 | Tests should not live in this state forever and should either be fixed |
| 386 | or eventually removed. |
| 387 | |
| 388 | QEMU_TEST_ALLOW_SLOW |
| 389 | ^^^^^^^^^^^^^^^^^^^^ |
| 390 | Tests that have a very long runtime and might run into timeout issues |
| 391 | e.g. if the QEMU binary has been compiled with debugging options enabled. |
| 392 | To avoid these timeout issues by default and to save some precious CPU |
| 393 | cycles during normal testing, such tests are disabled by default unless |
| 394 | the QEMU_TEST_ALLOW_SLOW environment variable has been set. |
| 395 | |
| 396 | |
| 397 | .. _unittest: https://docs.python.org/3/library/unittest.html |