| 1 | ================================== |
| 2 | ACPI/SMBIOS testing using biosbits |
| 3 | ================================== |
| 4 | ************ |
| 5 | Introduction |
| 6 | ************ |
| 7 | Biosbits is a software written by Josh Triplett that can be downloaded |
| 8 | from https://biosbits.org/. The github codebase can be found |
| 9 | `here <https://github.com/biosbits/bits/tree/master>`__. It is a software that |
| 10 | executes the bios components such as acpi and smbios tables directly through |
| 11 | acpica bios interpreter (a freely available C based library written by Intel, |
| 12 | downloadable from https://acpica.org/ and is included with biosbits) without an |
| 13 | operating system getting involved in between. Bios-bits has python integration |
| 14 | with grub so actual routines that executes bios components can be written in |
| 15 | python instead of bash-ish (grub's native scripting language). |
| 16 | There are several advantages to directly testing the bios in a real physical |
| 17 | machine or in a VM as opposed to indirectly discovering bios issues through the |
| 18 | operating system (the OS). Operating systems tend to bypass bios problems and |
| 19 | hide them from the end user. We have more control of what we wanted to test and |
| 20 | how by being as close to the bios on a running system as possible without a |
| 21 | complicated software component such as an operating system coming in between. |
| 22 | Another issue is that we cannot exercise bios components such as ACPI and |
| 23 | SMBIOS without being in the highest hardware privilege level, ring 0 for |
| 24 | example in case of x86. Since the OS executes from ring 0 whereas normal user |
| 25 | land software resides in unprivileged ring 3, operating system must be modified |
| 26 | in order to write our test routines that exercise and test the bios. This is |
| 27 | not possible in all cases. Lastly, test frameworks and routines are preferably |
| 28 | written using a high level scripting language such as python. OSes and |
| 29 | OS modules are generally written using low level languages such as C and |
| 30 | low level assembly machine language. Writing test routines in a low level |
| 31 | language makes things more cumbersome. These and other reasons makes using |
| 32 | bios-bits very attractive for testing bioses. More details on the inspiration |
| 33 | for developing biosbits and its real life uses were presented `at Plumbers |
| 34 | in 2011 <Plumbers_>`__ and `at Linux.conf.au in 2012 <Linux.conf.au_>`__. |
| 35 | |
| 36 | For QEMU, we maintain a fork of bios bits in `gitlab`_, along with all |
| 37 | the dependent submodules. This fork contains numerous fixes, a newer |
| 38 | acpica and changes specific to running these functional QEMU tests using |
| 39 | bits. The author of this document is the current maintainer of the QEMU |
| 40 | fork of bios bits repository. For more information, please see `the |
| 41 | author's FOSDEM presentation <FOSDEM_>`__ on this bios-bits based test framework. |
| 42 | |
| 43 | .. _Plumbers: https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf |
| 44 | .. _Linux.conf.au: https://www.youtube.com/watch?v=36QIepyUuhg |
| 45 | .. _gitlab: https://gitlab.com/qemu-project/biosbits-bits |
| 46 | .. _FOSDEM: https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/ |
| 47 | |
| 48 | ********************************* |
| 49 | Description of the test framework |
| 50 | ********************************* |
| 51 | |
| 52 | Under the directory ``tests/functional/``, ``test_acpi_bits.py`` is a QEMU |
| 53 | functional test that drives all this. |
| 54 | |
| 55 | A brief description of the various test files follows. |
| 56 | |
| 57 | Under ``tests/functional/`` as the root we have: |
| 58 | |
| 59 | :: |
| 60 | |
| 61 | ├── acpi-bits |
| 62 | │ ├── bits-config |
| 63 | │ │ └── bits-cfg.txt |
| 64 | │ ├── bits-tests |
| 65 | │ ├── smbios.py2 |
| 66 | │ ├── testacpi.py2 |
| 67 | │ └── testcpuid.py2 |
| 68 | ├── test_acpi_bits.py |
| 69 | |
| 70 | * ``tests/functional``: |
| 71 | |
| 72 | ``test_acpi_bits.py``: |
| 73 | This is the main python functional test script that generates a |
| 74 | biosbits iso. It then spawns a QEMU VM with it, collects the log and reports |
| 75 | test failures. This is the script one would be interested in if they wanted |
| 76 | to add or change some component of the log parsing, add a new command line |
| 77 | to alter how QEMU is spawned etc. Test writers typically would not need to |
| 78 | modify this script unless they wanted to enhance or change the log parsing |
| 79 | for their tests. In order to enable debugging, you can set **V=1** |
| 80 | environment variable. This enables verbose mode for the test and also dumps |
| 81 | the entire log from bios bits and more information in case failure happens. |
| 82 | You can also set **BITS_DEBUG=1** to turn on debug mode. It will enable |
| 83 | verbose logs and also retain the temporary work directory the test used for |
| 84 | you to inspect and run the specific commands manually. |
| 85 | |
| 86 | In order to run this test, please perform the following steps from the QEMU |
| 87 | build directory (assuming that the sources are in ".."): |
| 88 | :: |
| 89 | |
| 90 | $ export PYTHONPATH=../python:../tests/functional |
| 91 | $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64 |
| 92 | $ python3 ../tests/functional/test_acpi_bits.py |
| 93 | |
| 94 | The above will run all acpi-bits functional tests (producing output in |
| 95 | tap format). |
| 96 | |
| 97 | You can inspect the log files in tests/functional/x86_64/test_acpi_bits.*/ |
| 98 | for more information about the run or in order to diagnoze issues. |
| 99 | If you pass V=1 in the environment, more diagnostic logs will be put into |
| 100 | the test log. |
| 101 | |
| 102 | * ``tests/functional/acpi-bits/bits-config``: |
| 103 | |
| 104 | This location contains biosbits configuration files that determine how the |
| 105 | software runs the tests. |
| 106 | |
| 107 | ``bits-config.txt``: |
| 108 | This is the biosbits config file that determines what tests |
| 109 | or actions are performed by bits. The description of the config options are |
| 110 | provided in the file itself. |
| 111 | |
| 112 | * ``tests/functional/acpi-bits/bits-tests``: |
| 113 | |
| 114 | This directory contains biosbits python based tests that are run from within |
| 115 | the biosbits environment in the spawned VM. New additions of test cases can |
| 116 | be made in the appropriate test file. For example, new acpi tests can go |
| 117 | into testacpi.py2 and one would call testsuite.add_test() to register the new |
| 118 | test so that it gets executed as a part of the ACPI tests. |
| 119 | It might be occasionally necessary to disable some subtests or add a new |
| 120 | test that belongs to a test suite not already present in this directory. To |
| 121 | do this, please clone the bits source from |
| 122 | https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits. |
| 123 | Note that this is the "qemu-bits" branch and not the "bits" branch of the |
| 124 | repository. "qemu-bits" is the branch where we have made all the QEMU |
| 125 | specific enhancements and we must use the source from this branch only. |
| 126 | Copy the test suite/script that needs modification (addition of new tests |
| 127 | or disabling them) from python directory into this directory. For |
| 128 | example, in order to change cpuid related tests, copy the following |
| 129 | file into this directory and rename it with .py2 extension: |
| 130 | https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py |
| 131 | Then make your additions and changes here. Therefore, the steps are: |
| 132 | |
| 133 | (a) Copy unmodified test script to this directory from bits source. |
| 134 | (b) Add a SPDX license header. |
| 135 | (c) Perform modifications to the test. |
| 136 | |
| 137 | Commits (a), (b) and (c) preferably should go under separate commits so that |
| 138 | the original test script and the changes we have made are separated and |
| 139 | clear. (a) and (b) can sometimes be combined into a single step. |
| 140 | |
| 141 | The test framework will then use your modified test script to run the test. |
| 142 | No further changes would be needed. Please check the logs to make sure that |
| 143 | appropriate changes have taken effect. |
| 144 | |
| 145 | The tests have an extension .py2 in order to indicate that: |
| 146 | |
| 147 | (a) They are python2.7 based scripts and not python 3 scripts. |
| 148 | (b) They are run from within the bios bits VM and is not subjected to QEMU |
| 149 | build/test python script maintenance and dependency resolutions. |
| 150 | (c) They need not be loaded by the test framework by accident when running |
| 151 | tests. |
| 152 | |
| 153 | |
| 154 | Author: Ani Sinha <anisinha@redhat.com> |
| 155 |