| 1 | Emulation |
| 2 | ========= |
| 3 | |
| 4 | QEMU's Tiny Code Generator (TCG) provides the ability to emulate a |
| 5 | number of CPU architectures on any supported host platform. Both |
| 6 | :ref:`System Emulation` and :ref:`User Mode Emulation` are supported |
| 7 | depending on the guest architecture. |
| 8 | |
| 9 | .. list-table:: Supported Guest Architectures for Emulation |
| 10 | :widths: 30 10 10 50 |
| 11 | :header-rows: 1 |
| 12 | |
| 13 | * - Architecture (qemu name) |
| 14 | - System |
| 15 | - User |
| 16 | - Notes |
| 17 | * - Alpha |
| 18 | - Yes |
| 19 | - Yes |
| 20 | - Legacy 64 bit RISC ISA developed by DEC |
| 21 | * - Arm (arm, aarch64) |
| 22 | - :ref:`Yes<ARM-System-emulator>` |
| 23 | - Yes |
| 24 | - Wide range of features, see :ref:`Arm Emulation` for details |
| 25 | * - AVR |
| 26 | - :ref:`Yes<AVR-System-emulator>` |
| 27 | - No |
| 28 | - 8 bit micro controller, often used in maker projects |
| 29 | * - Hexagon |
| 30 | - No |
| 31 | - Yes |
| 32 | - Family of DSPs by Qualcomm |
| 33 | * - PA-RISC (hppa) |
| 34 | - Yes |
| 35 | - Yes |
| 36 | - A legacy RISC system used in HP's old minicomputers |
| 37 | * - x86 (i386, x86_64) |
| 38 | - :ref:`Yes<QEMU-PC-System-emulator>` |
| 39 | - Yes |
| 40 | - The ubiquitous desktop PC CPU architecture, 32 and 64 bit. |
| 41 | * - LoongArch |
| 42 | - Yes |
| 43 | - Yes |
| 44 | - A MIPS-like 64bit RISC architecture developed in China |
| 45 | * - m68k |
| 46 | - :ref:`Yes<ColdFire-System-emulator>` |
| 47 | - Yes |
| 48 | - Motorola 68000 variants and ColdFire |
| 49 | * - Microblaze |
| 50 | - Yes |
| 51 | - Yes |
| 52 | - RISC based soft-core by Xilinx |
| 53 | * - MIPS (mips*) |
| 54 | - :ref:`Yes<MIPS-System-emulator>` |
| 55 | - Yes |
| 56 | - Venerable RISC architecture originally out of Stanford University |
| 57 | * - OpenRISC |
| 58 | - :ref:`Yes<OpenRISC-System-emulator>` |
| 59 | - Yes |
| 60 | - Open source RISC architecture developed by the OpenRISC community |
| 61 | * - Power (ppc, ppc64) |
| 62 | - :ref:`Yes<PowerPC-System-emulator>` |
| 63 | - Yes |
| 64 | - A general purpose RISC architecture now managed by IBM |
| 65 | * - RISC-V |
| 66 | - :ref:`Yes<RISC-V-System-emulator>` |
| 67 | - Yes |
| 68 | - An open standard RISC ISA maintained by RISC-V International |
| 69 | * - RX |
| 70 | - :ref:`Yes<RX-System-emulator>` |
| 71 | - No |
| 72 | - A 32 bit micro controller developed by Renesas |
| 73 | * - s390x |
| 74 | - :ref:`Yes<s390x-System-emulator>` |
| 75 | - Yes |
| 76 | - A 64 bit CPU found in IBM's System Z mainframes |
| 77 | * - sh4 |
| 78 | - Yes |
| 79 | - Yes |
| 80 | - A 32 bit RISC embedded CPU developed by Hitachi |
| 81 | * - SPARC (sparc, sparc64) |
| 82 | - :ref:`Yes<Sparc32-System-emulator>` |
| 83 | - Yes |
| 84 | - A RISC ISA originally developed by Sun Microsystems |
| 85 | * - Tricore |
| 86 | - Yes |
| 87 | - No |
| 88 | - A 32 bit RISC/uController/DSP developed by Infineon |
| 89 | * - Xtensa |
| 90 | - :ref:`Yes<Xtensa-System-emulator>` |
| 91 | - Yes |
| 92 | - A configurable 32 bit soft core now owned by Cadence |
| 93 | |
| 94 | .. _Semihosting: |
| 95 | |
| 96 | Semihosting |
| 97 | ----------- |
| 98 | |
| 99 | Semihosting is a feature defined by the owner of the architecture to |
| 100 | allow programs to interact with a debugging host system. On real |
| 101 | hardware this is usually provided by an In-circuit emulator (ICE) |
| 102 | hooked directly to the board. QEMU's implementation allows for |
| 103 | semihosting calls to be passed to the host system or via the |
| 104 | ``gdbstub``. |
| 105 | |
| 106 | Generally semihosting makes it easier to bring up low level code before a |
| 107 | more fully functional operating system has been enabled. On QEMU it |
| 108 | also allows for embedded micro-controller code which typically doesn't |
| 109 | have a full libc to be run as "bare-metal" code under QEMU's user-mode |
| 110 | emulation. It is also useful for writing test cases and indeed a |
| 111 | number of compiler suites as well as QEMU itself use semihosting calls |
| 112 | to exit test code while reporting the success state. |
| 113 | |
| 114 | Semihosting is only available using TCG emulation. This is because the |
| 115 | instructions to trigger a semihosting call are typically reserved |
| 116 | causing most hypervisors to trap and fault on them. |
| 117 | |
| 118 | .. warning:: |
| 119 | Semihosting inherently bypasses any isolation there may be between |
| 120 | the guest and the host. As a result a program using semihosting can |
| 121 | happily trash your host system. Some semihosting calls (e.g. |
| 122 | ``SYS_READC``) can block execution indefinitely. You should only |
| 123 | ever run trusted code with semihosting enabled. |
| 124 | |
| 125 | Redirection |
| 126 | ~~~~~~~~~~~ |
| 127 | |
| 128 | Semihosting calls can be re-directed to a (potentially remote) gdb |
| 129 | during debugging via the :ref:`gdbstub<GDB usage>`. Output to the |
| 130 | semihosting console is configured as a ``chardev`` so can be |
| 131 | redirected to a file, pipe or socket like any other ``chardev`` |
| 132 | device. |
| 133 | |
| 134 | Supported Targets |
| 135 | ~~~~~~~~~~~~~~~~~ |
| 136 | |
| 137 | Most targets offer similar semihosting implementations with some |
| 138 | minor changes to define the appropriate instruction to encode the |
| 139 | semihosting call and which registers hold the parameters. They tend to |
| 140 | presents a simple POSIX-like API which allows your program to read and |
| 141 | write files, access the console and some other basic interactions. |
| 142 | |
| 143 | For full details of the ABI for a particular target, and the set of |
| 144 | calls it provides, you should consult the semihosting specification |
| 145 | for that architecture. |
| 146 | |
| 147 | .. note:: |
| 148 | QEMU makes an implementation decision to implement all file |
| 149 | access in ``O_BINARY`` mode. The user-visible effect of this is |
| 150 | regardless of the text/binary mode the program sets QEMU will |
| 151 | always select a binary mode ensuring no line-terminator conversion |
| 152 | is performed on input or output. This is because gdb semihosting |
| 153 | support doesn't make the distinction between the modes and |
| 154 | magically processing line endings can be confusing. |
| 155 | |
| 156 | .. list-table:: Guest Architectures supporting Semihosting |
| 157 | :widths: 10 10 80 |
| 158 | :header-rows: 1 |
| 159 | |
| 160 | * - Architecture |
| 161 | - Modes |
| 162 | - Specification |
| 163 | * - Arm |
| 164 | - System and User-mode |
| 165 | - https://github.com/ARM-software/abi-aa/blob/main/semihosting/semihosting.rst |
| 166 | * - m68k |
| 167 | - System |
| 168 | - https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=libgloss/m68k/m68k-semi.txt;hb=HEAD |
| 169 | * - MIPS |
| 170 | - System |
| 171 | - Unified Hosting Interface (MD01069) |
| 172 | * - RISC-V |
| 173 | - System and User-mode |
| 174 | - https://github.com/riscv-non-isa/riscv-semihosting/blob/main/riscv-semihosting.adoc |
| 175 | * - Xtensa |
| 176 | - System |
| 177 | - Tensilica ISS SIMCALL |
| 178 | |
| 179 | .. _tcg-plugins: |
| 180 | |
| 181 | TCG Plugins |
| 182 | ----------- |
| 183 | |
| 184 | QEMU TCG plugins provide a way for users to run experiments taking |
| 185 | advantage of the total system control emulation can have over a guest. |
| 186 | It provides a mechanism for plugins to subscribe to events during |
| 187 | translation and execution and optionally callback into the plugin |
| 188 | during these events. TCG plugins are unable to change the system state |
| 189 | only monitor it passively. However they can do this down to an |
| 190 | individual instruction granularity including potentially subscribing |
| 191 | to all load and store operations. |
| 192 | |
| 193 | See the developer section of the manual for details about |
| 194 | :ref:`writing plugins<TCG Plugins>`. |
| 195 | |
| 196 | Usage |
| 197 | ~~~~~ |
| 198 | |
| 199 | Any QEMU binary with TCG support has plugins enabled by default. |
| 200 | Earlier releases needed to be explicitly enabled with:: |
| 201 | |
| 202 | configure --enable-plugins |
| 203 | |
| 204 | Once built a program can be run with multiple plugins loaded each with |
| 205 | their own arguments:: |
| 206 | |
| 207 | $QEMU $OTHER_QEMU_ARGS \ |
| 208 | -plugin contrib/plugins/libhowvec.so,inline=on,count=hint \ |
| 209 | -plugin contrib/plugins/libhotblocks.so |
| 210 | |
| 211 | Arguments are plugin specific and can be used to modify their |
| 212 | behaviour. In this case the howvec plugin is being asked to use inline |
| 213 | ops to count and break down the hint instructions by type. |
| 214 | |
| 215 | Linux user-mode emulation also evaluates the environment variable |
| 216 | ``QEMU_PLUGIN``:: |
| 217 | |
| 218 | QEMU_PLUGIN="file=contrib/plugins/libhowvec.so,inline=on,count=hint" $QEMU |
| 219 | |
| 220 | QEMU plugins avoid to write directly to stdin/stderr, and use the log provided |
| 221 | by the API (see function ``qemu_plugin_outs``). |
| 222 | To show output, you may use this additional parameter:: |
| 223 | |
| 224 | $QEMU $OTHER_QEMU_ARGS \ |
| 225 | -d plugin \ |
| 226 | -plugin contrib/plugins/libhowvec.so,inline=on,count=hint |
| 227 | |
| 228 | Example Plugins |
| 229 | ~~~~~~~~~~~~~~~ |
| 230 | |
| 231 | There are a number of plugins included with QEMU and you are |
| 232 | encouraged to contribute your own plugins plugins upstream. There is a |
| 233 | ``contrib/plugins`` directory where they can go. There are also some |
| 234 | basic plugins that are used to test and exercise the API during the |
| 235 | ``make check-tcg`` target in ``tests/tcg/plugins`` that are never the |
| 236 | less useful for basic analysis. |
| 237 | |
| 238 | Empty |
| 239 | ..... |
| 240 | |
| 241 | ``tests/tcg/plugins/empty.c`` |
| 242 | |
| 243 | Purely a test plugin for measuring the overhead of the plugins system |
| 244 | itself. Does no instrumentation. |
| 245 | |
| 246 | Basic Blocks |
| 247 | ............ |
| 248 | |
| 249 | ``tests/tcg/plugins/bb.c`` |
| 250 | |
| 251 | A very basic plugin which will measure execution in coarse terms as |
| 252 | each basic block is executed. By default the results are shown once |
| 253 | execution finishes:: |
| 254 | |
| 255 | $ qemu-aarch64 -plugin tests/plugin/libbb.so \ |
| 256 | -d plugin ./tests/tcg/aarch64-linux-user/sha1 |
| 257 | SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6 |
| 258 | bb's: 2277338, insns: 158483046 |
| 259 | |
| 260 | Behaviour can be tweaked with the following arguments: |
| 261 | |
| 262 | .. list-table:: Basic Block plugin arguments |
| 263 | :widths: 20 80 |
| 264 | :header-rows: 1 |
| 265 | |
| 266 | * - Option |
| 267 | - Description |
| 268 | * - inline=true|false |
| 269 | - Use faster inline addition of a single counter. |
| 270 | * - idle=true|false |
| 271 | - Dump the current execution stats whenever the guest vCPU idles |
| 272 | |
| 273 | Basic Block Vectors |
| 274 | ................... |
| 275 | |
| 276 | ``contrib/plugins/bbv.c`` |
| 277 | |
| 278 | The bbv plugin allows you to generate basic block vectors for use with the |
| 279 | `SimPoint <https://cseweb.ucsd.edu/~calder/simpoint/>`__ analysis tool. |
| 280 | |
| 281 | .. list-table:: Basic block vectors arguments |
| 282 | :widths: 20 80 |
| 283 | :header-rows: 1 |
| 284 | |
| 285 | * - Option |
| 286 | - Description |
| 287 | * - interval=N |
| 288 | - The interval to generate a basic block vector specified by the number of |
| 289 | instructions (Default: N = 100000000) |
| 290 | * - outfile=PATH |
| 291 | - The path to output files. |
| 292 | It will be suffixed with ``.N.bb`` where ``N`` is a vCPU index. |
| 293 | |
| 294 | Example:: |
| 295 | |
| 296 | $ qemu-aarch64 \ |
| 297 | -plugin contrib/plugins/libbbv.so,interval=100,outfile=sha1 \ |
| 298 | tests/tcg/aarch64-linux-user/sha1 |
| 299 | SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6 |
| 300 | $ du sha1.0.bb |
| 301 | 23128 sha1.0.bb |
| 302 | |
| 303 | Instruction |
| 304 | ........... |
| 305 | |
| 306 | ``tests/tcg/plugins/insn.c`` |
| 307 | |
| 308 | This is a basic instruction level instrumentation which can count the |
| 309 | number of instructions executed on each core/thread:: |
| 310 | |
| 311 | $ qemu-aarch64 -plugin tests/plugin/libinsn.so \ |
| 312 | -d plugin ./tests/tcg/aarch64-linux-user/threadcount |
| 313 | Created 10 threads |
| 314 | Done |
| 315 | cpu 0 insns: 46765 |
| 316 | cpu 1 insns: 3694 |
| 317 | cpu 2 insns: 3694 |
| 318 | cpu 3 insns: 2994 |
| 319 | cpu 4 insns: 1497 |
| 320 | cpu 5 insns: 1497 |
| 321 | cpu 6 insns: 1497 |
| 322 | cpu 7 insns: 1497 |
| 323 | total insns: 63135 |
| 324 | |
| 325 | Behaviour can be tweaked with the following arguments: |
| 326 | |
| 327 | .. list-table:: Instruction plugin arguments |
| 328 | :widths: 20 80 |
| 329 | :header-rows: 1 |
| 330 | |
| 331 | * - Option |
| 332 | - Description |
| 333 | * - inline=true|false |
| 334 | - Use faster inline addition of a single counter. |
| 335 | * - sizes=true|false |
| 336 | - Give a summary of the instruction sizes for the execution |
| 337 | * - match=<string> |
| 338 | - Only instrument instructions matching the string prefix |
| 339 | |
| 340 | The ``match`` option will show some basic stats including how many |
| 341 | instructions have executed since the last execution. For |
| 342 | example:: |
| 343 | |
| 344 | $ qemu-aarch64 -plugin tests/plugin/libinsn.so,match=bl \ |
| 345 | -d plugin ./tests/tcg/aarch64-linux-user/sha512-vector |
| 346 | ... |
| 347 | 0x40069c, 'bl #0x4002b0', 10 hits, 1093 match hits, Δ+1257 since last match, 98 avg insns/match |
| 348 | 0x4006ac, 'bl #0x403690', 10 hits, 1094 match hits, Δ+47 since last match, 98 avg insns/match |
| 349 | 0x4037fc, 'bl #0x4002b0', 18 hits, 1095 match hits, Δ+22 since last match, 98 avg insns/match |
| 350 | 0x400720, 'bl #0x403690', 10 hits, 1096 match hits, Δ+58 since last match, 98 avg insns/match |
| 351 | 0x4037fc, 'bl #0x4002b0', 19 hits, 1097 match hits, Δ+22 since last match, 98 avg insns/match |
| 352 | 0x400730, 'bl #0x403690', 10 hits, 1098 match hits, Δ+33 since last match, 98 avg insns/match |
| 353 | 0x4037ac, 'bl #0x4002b0', 12 hits, 1099 match hits, Δ+20 since last match, 98 avg insns/match |
| 354 | ... |
| 355 | |
| 356 | For more detailed execution tracing see the ``execlog`` plugin for |
| 357 | other options. |
| 358 | |
| 359 | Memory |
| 360 | ...... |
| 361 | |
| 362 | ``tests/tcg/plugins/mem.c`` |
| 363 | |
| 364 | Basic instruction level memory instrumentation:: |
| 365 | |
| 366 | $ qemu-aarch64 -plugin tests/plugin/libmem.so,inline=true \ |
| 367 | -d plugin ./tests/tcg/aarch64-linux-user/sha1 |
| 368 | SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6 |
| 369 | inline mem accesses: 79525013 |
| 370 | |
| 371 | Behaviour can be tweaked with the following arguments: |
| 372 | |
| 373 | .. list-table:: Memory plugin arguments |
| 374 | :widths: 20 80 |
| 375 | :header-rows: 1 |
| 376 | |
| 377 | * - Option |
| 378 | - Description |
| 379 | * - inline=true|false |
| 380 | - Use faster inline addition of a single counter |
| 381 | * - callback=true|false |
| 382 | - Use callbacks on each memory instrumentation. |
| 383 | * - hwaddr=true|false |
| 384 | - Count IO accesses (only for system emulation) |
| 385 | |
| 386 | System Calls |
| 387 | ............ |
| 388 | |
| 389 | ``tests/tcg/plugins/syscall.c`` |
| 390 | |
| 391 | A basic syscall tracing plugin. This only works for user-mode. By |
| 392 | default it will give a summary of syscall stats at the end of the |
| 393 | run:: |
| 394 | |
| 395 | $ qemu-aarch64 -plugin tests/plugin/libsyscall \ |
| 396 | -d plugin ./tests/tcg/aarch64-linux-user/threadcount |
| 397 | Created 10 threads |
| 398 | Done |
| 399 | syscall no. calls errors |
| 400 | 226 12 0 |
| 401 | 99 11 11 |
| 402 | 115 11 0 |
| 403 | 222 11 0 |
| 404 | 93 10 0 |
| 405 | 220 10 0 |
| 406 | 233 10 0 |
| 407 | 215 8 0 |
| 408 | 214 4 0 |
| 409 | 134 2 0 |
| 410 | 64 2 0 |
| 411 | 96 1 0 |
| 412 | 94 1 0 |
| 413 | 80 1 0 |
| 414 | 261 1 0 |
| 415 | 78 1 0 |
| 416 | 160 1 0 |
| 417 | 135 1 0 |
| 418 | |
| 419 | Behaviour can be tweaked with the following arguments: |
| 420 | |
| 421 | .. list-table:: Syscall plugin arguments |
| 422 | :widths: 20 80 |
| 423 | :header-rows: 1 |
| 424 | |
| 425 | * - Option |
| 426 | - Description |
| 427 | * - print=true|false |
| 428 | - Print the number of times each syscall is called |
| 429 | * - log_writes=true|false |
| 430 | - Log the buffer of each write syscall in hexdump format |
| 431 | |
| 432 | Test inline operations |
| 433 | ...................... |
| 434 | |
| 435 | ``tests/plugins/inline.c`` |
| 436 | |
| 437 | This plugin is used for testing all inline operations, conditional callbacks and |
| 438 | scoreboard. It prints a per-cpu summary of all events. |
| 439 | |
| 440 | |
| 441 | Hot Blocks |
| 442 | .......... |
| 443 | |
| 444 | ``contrib/plugins/hotblocks.c`` |
| 445 | |
| 446 | The hotblocks plugin allows you to examine the where hot paths of |
| 447 | execution are in your program. Once the program has finished you will |
| 448 | get a sorted list of blocks reporting the starting PC, translation |
| 449 | count, number of instructions and execution count. This will work best |
| 450 | with linux-user execution as system emulation tends to generate |
| 451 | re-translations as blocks from different programs get swapped in and |
| 452 | out of system memory. |
| 453 | |
| 454 | Example:: |
| 455 | |
| 456 | $ qemu-aarch64 \ |
| 457 | -plugin contrib/plugins/libhotblocks.so -d plugin \ |
| 458 | ./tests/tcg/aarch64-linux-user/sha1 |
| 459 | SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6 |
| 460 | collected 903 entries in the hash table |
| 461 | pc, tcount, icount, ecount |
| 462 | 0x0000000041ed10, 1, 5, 66087 |
| 463 | 0x000000004002b0, 1, 4, 66087 |
| 464 | ... |
| 465 | |
| 466 | Behaviour can be tweaked with the following arguments: |
| 467 | |
| 468 | .. list-table:: Hot Blocks plugin arguments |
| 469 | :widths: 20 80 |
| 470 | :header-rows: 1 |
| 471 | |
| 472 | * - Option |
| 473 | - Description |
| 474 | * - inline=true|false |
| 475 | - Use faster inline addition of a single counter. |
| 476 | * - limit=N |
| 477 | - The number of blocks to be printed. (Default: N = 20, use 0 for no limit). |
| 478 | |
| 479 | Hot Pages |
| 480 | ......... |
| 481 | |
| 482 | ``contrib/plugins/hotpages.c`` |
| 483 | |
| 484 | Similar to hotblocks but this time tracks memory accesses:: |
| 485 | |
| 486 | $ qemu-aarch64 \ |
| 487 | -plugin contrib/plugins/libhotpages.so -d plugin \ |
| 488 | ./tests/tcg/aarch64-linux-user/sha1 |
| 489 | SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6 |
| 490 | Addr, RCPUs, Reads, WCPUs, Writes |
| 491 | 0x000055007fe000, 0x0001, 31747952, 0x0001, 8835161 |
| 492 | 0x000055007ff000, 0x0001, 29001054, 0x0001, 8780625 |
| 493 | 0x00005500800000, 0x0001, 687465, 0x0001, 335857 |
| 494 | 0x0000000048b000, 0x0001, 130594, 0x0001, 355 |
| 495 | 0x0000000048a000, 0x0001, 1826, 0x0001, 11 |
| 496 | |
| 497 | The hotpages plugin can be configured using the following arguments: |
| 498 | |
| 499 | .. list-table:: Hot pages arguments |
| 500 | :widths: 20 80 |
| 501 | :header-rows: 1 |
| 502 | |
| 503 | * - Option |
| 504 | - Description |
| 505 | * - sortby=reads|writes|address |
| 506 | - Log the data sorted by either the number of reads, the number of writes, or |
| 507 | memory address. (Default: entries are sorted by the sum of reads and writes) |
| 508 | * - io=on |
| 509 | - Track IO addresses. Only relevant to full system emulation. (Default: off) |
| 510 | * - pagesize=N |
| 511 | - The page size used. (Default: N = 4096) |
| 512 | |
| 513 | Instruction Distribution |
| 514 | ........................ |
| 515 | |
| 516 | ``contrib/plugins/howvec.c`` |
| 517 | |
| 518 | This is an instruction classifier so can be used to count different |
| 519 | types of instructions. It has a number of options to refine which get |
| 520 | counted. You can give a value to the ``count`` argument for a class of |
| 521 | instructions to break it down fully, so for example to see all the system |
| 522 | registers accesses:: |
| 523 | |
| 524 | $ qemu-system-aarch64 $(QEMU_ARGS) \ |
| 525 | -append "root=/dev/sda2 systemd.unit=benchmark.service" \ |
| 526 | -smp 4 -plugin ./contrib/plugins/libhowvec.so,count=sreg -d plugin |
| 527 | |
| 528 | which will lead to a sorted list after the class breakdown:: |
| 529 | |
| 530 | Instruction Classes: |
| 531 | Class: UDEF not counted |
| 532 | Class: SVE (68 hits) |
| 533 | Class: PCrel addr (47789483 hits) |
| 534 | Class: Add/Sub (imm) (192817388 hits) |
| 535 | Class: Logical (imm) (93852565 hits) |
| 536 | Class: Move Wide (imm) (76398116 hits) |
| 537 | Class: Bitfield (44706084 hits) |
| 538 | Class: Extract (5499257 hits) |
| 539 | Class: Cond Branch (imm) (147202932 hits) |
| 540 | Class: Exception Gen (193581 hits) |
| 541 | Class: NOP not counted |
| 542 | Class: Hints (6652291 hits) |
| 543 | Class: Barriers (8001661 hits) |
| 544 | Class: PSTATE (1801695 hits) |
| 545 | Class: System Insn (6385349 hits) |
| 546 | Class: System Reg counted individually |
| 547 | Class: Branch (reg) (69497127 hits) |
| 548 | Class: Branch (imm) (84393665 hits) |
| 549 | Class: Cmp & Branch (110929659 hits) |
| 550 | Class: Tst & Branch (44681442 hits) |
| 551 | Class: AdvSimd ldstmult (736 hits) |
| 552 | Class: ldst excl (9098783 hits) |
| 553 | Class: Load Reg (lit) (87189424 hits) |
| 554 | Class: ldst noalloc pair (3264433 hits) |
| 555 | Class: ldst pair (412526434 hits) |
| 556 | Class: ldst reg (imm) (314734576 hits) |
| 557 | Class: Loads & Stores (2117774 hits) |
| 558 | Class: Data Proc Reg (223519077 hits) |
| 559 | Class: Scalar FP (31657954 hits) |
| 560 | Individual Instructions: |
| 561 | Instr: mrs x0, sp_el0 (2682661 hits) (op=0xd5384100/ System Reg) |
| 562 | Instr: mrs x1, tpidr_el2 (1789339 hits) (op=0xd53cd041/ System Reg) |
| 563 | Instr: mrs x2, tpidr_el2 (1513494 hits) (op=0xd53cd042/ System Reg) |
| 564 | Instr: mrs x0, tpidr_el2 (1490823 hits) (op=0xd53cd040/ System Reg) |
| 565 | Instr: mrs x1, sp_el0 (933793 hits) (op=0xd5384101/ System Reg) |
| 566 | Instr: mrs x2, sp_el0 (699516 hits) (op=0xd5384102/ System Reg) |
| 567 | Instr: mrs x4, tpidr_el2 (528437 hits) (op=0xd53cd044/ System Reg) |
| 568 | Instr: mrs x30, ttbr1_el1 (480776 hits) (op=0xd538203e/ System Reg) |
| 569 | Instr: msr ttbr1_el1, x30 (480713 hits) (op=0xd518203e/ System Reg) |
| 570 | Instr: msr vbar_el1, x30 (480671 hits) (op=0xd518c01e/ System Reg) |
| 571 | ... |
| 572 | |
| 573 | To find the argument shorthand for the class you need to examine the |
| 574 | source code of the plugin at the moment, specifically the ``*opt`` |
| 575 | argument in the InsnClassExecCount tables. |
| 576 | |
| 577 | Lockstep Execution |
| 578 | .................. |
| 579 | |
| 580 | ``contrib/plugins/lockstep.c`` |
| 581 | |
| 582 | This is a debugging tool for developers who want to find out when and |
| 583 | where execution diverges after a subtle change to TCG code generation. |
| 584 | It is not an exact science and results are likely to be mixed once |
| 585 | asynchronous events are introduced. While the use of -icount can |
| 586 | introduce determinism to the execution flow it doesn't always follow |
| 587 | the translation sequence will be exactly the same. Typically this is |
| 588 | caused by a timer firing to service the GUI causing a block to end |
| 589 | early. However in some cases it has proved to be useful in pointing |
| 590 | people at roughly where execution diverges. The only argument you need |
| 591 | for the plugin is a path for the socket the two instances will |
| 592 | communicate over:: |
| 593 | |
| 594 | |
| 595 | $ qemu-system-sparc -monitor none -parallel none \ |
| 596 | -net none -M SS-20 -m 256 -kernel day11/zImage.elf \ |
| 597 | -plugin ./contrib/plugins/liblockstep.so,sockpath=lockstep-sparc.sock \ |
| 598 | -d plugin,nochain |
| 599 | |
| 600 | which will eventually report:: |
| 601 | |
| 602 | qemu-system-sparc: warning: nic lance.0 has no peer |
| 603 | @ 0x000000ffd06678 vs 0x000000ffd001e0 (2/1 since last) |
| 604 | @ 0x000000ffd07d9c vs 0x000000ffd06678 (3/1 since last) |
| 605 | Δ insn_count @ 0x000000ffd07d9c (809900609) vs 0x000000ffd06678 (809900612) |
| 606 | previously @ 0x000000ffd06678/10 (809900609 insns) |
| 607 | previously @ 0x000000ffd001e0/4 (809900599 insns) |
| 608 | previously @ 0x000000ffd080ac/2 (809900595 insns) |
| 609 | previously @ 0x000000ffd08098/5 (809900593 insns) |
| 610 | previously @ 0x000000ffd080c0/1 (809900588 insns) |
| 611 | |
| 612 | |
| 613 | Hardware Profile |
| 614 | ................ |
| 615 | |
| 616 | ``contrib/plugins/hwprofile.c`` |
| 617 | |
| 618 | The hwprofile tool can only be used with system emulation and allows |
| 619 | the user to see what hardware is accessed how often. It has a number of options: |
| 620 | |
| 621 | .. list-table:: Hardware Profile arguments |
| 622 | :widths: 20 80 |
| 623 | :header-rows: 1 |
| 624 | |
| 625 | * - Option |
| 626 | - Description |
| 627 | * - track=[read|write] |
| 628 | - By default the plugin tracks both reads and writes. You can use |
| 629 | this option to limit the tracking to just one class of accesses. |
| 630 | * - source |
| 631 | - Will include a detailed break down of what the guest PC that made the |
| 632 | access was. Not compatible with the pattern option. Example output:: |
| 633 | |
| 634 | cirrus-low-memory @ 0xfffffd00000a0000 |
| 635 | pc:fffffc0000005cdc, 1, 256 |
| 636 | pc:fffffc0000005ce8, 1, 256 |
| 637 | pc:fffffc0000005cec, 1, 256 |
| 638 | |
| 639 | * - pattern |
| 640 | - Instead break down the accesses based on the offset into the HW |
| 641 | region. This can be useful for seeing the most used registers of |
| 642 | a device. Example output:: |
| 643 | |
| 644 | pci0-conf @ 0xfffffd01fe000000 |
| 645 | off:00000004, 1, 1 |
| 646 | off:00000010, 1, 3 |
| 647 | off:00000014, 1, 3 |
| 648 | off:00000018, 1, 2 |
| 649 | off:0000001c, 1, 2 |
| 650 | off:00000020, 1, 2 |
| 651 | ... |
| 652 | |
| 653 | |
| 654 | Execution Log |
| 655 | ............. |
| 656 | |
| 657 | ``contrib/plugins/execlog.c`` |
| 658 | |
| 659 | The execlog tool traces executed instructions with memory access. It can be used |
| 660 | for debugging and security analysis purposes. |
| 661 | Please be aware that this will generate a lot of output. |
| 662 | |
| 663 | The plugin needs default argument:: |
| 664 | |
| 665 | $ qemu-system-arm $(QEMU_ARGS) \ |
| 666 | -plugin ./contrib/plugins/libexeclog.so -d plugin |
| 667 | |
| 668 | which will output an execution trace following this structure:: |
| 669 | |
| 670 | # vCPU, vAddr, opcode, disassembly[, load/store, memory addr, device]... |
| 671 | 0, 0xa12, 0xf8012400, "movs r4, #0" |
| 672 | 0, 0xa14, 0xf87f42b4, "cmp r4, r6" |
| 673 | 0, 0xa16, 0xd206, "bhs #0xa26" |
| 674 | 0, 0xa18, 0xfff94803, "ldr r0, [pc, #0xc]", load, 0x00010a28, RAM |
| 675 | 0, 0xa1a, 0xf989f000, "bl #0xd30" |
| 676 | 0, 0xd30, 0xfff9b510, "push {r4, lr}", store, 0x20003ee0, RAM, store, 0x20003ee4, RAM |
| 677 | 0, 0xd32, 0xf9893014, "adds r0, #0x14" |
| 678 | 0, 0xd34, 0xf9c8f000, "bl #0x10c8" |
| 679 | 0, 0x10c8, 0xfff96c43, "ldr r3, [r0, #0x44]", load, 0x200000e4, RAM |
| 680 | |
| 681 | Please note that you need to configure QEMU with Capstone support to get disassembly. |
| 682 | |
| 683 | The output can be filtered to only track certain instructions or |
| 684 | addresses using the ``ifilter`` or ``afilter`` options. You can stack the |
| 685 | arguments if required:: |
| 686 | |
| 687 | $ qemu-system-arm $(QEMU_ARGS) \ |
| 688 | -plugin ./contrib/plugins/libexeclog.so,ifilter=st1w,afilter=0x40001808 -d plugin |
| 689 | |
| 690 | This plugin can also dump registers when they change value. Specify the name of the |
| 691 | registers with multiple ``reg`` options. You can also use glob style matching if you wish:: |
| 692 | |
| 693 | $ qemu-system-arm $(QEMU_ARGS) \ |
| 694 | -plugin ./contrib/plugins/libexeclog.so,reg=\*_el2,reg=sp -d plugin |
| 695 | |
| 696 | Be aware that each additional register to check will slow down |
| 697 | execution quite considerably. You can optimise the number of register |
| 698 | checks done by using the rdisas option. This will only instrument |
| 699 | instructions that mention the registers in question in disassembly. |
| 700 | This is not foolproof as some instructions implicitly change |
| 701 | instructions. You can use the ifilter to catch these cases:: |
| 702 | |
| 703 | $ qemu-system-arm $(QEMU_ARGS) \ |
| 704 | -plugin ./contrib/plugins/libexeclog.so,ifilter=msr,ifilter=blr,reg=x30,reg=\*_el1,rdisas=on |
| 705 | |
| 706 | Cache Modelling |
| 707 | ............... |
| 708 | |
| 709 | ``contrib/plugins/cache.c`` |
| 710 | |
| 711 | Cache modelling plugin that measures the performance of a given L1 cache |
| 712 | configuration, and optionally a unified L2 per-core cache when a given working |
| 713 | set is run:: |
| 714 | |
| 715 | $ qemu-x86_64 -plugin ./contrib/plugins/libcache.so \ |
| 716 | -d plugin -D cache.log ./tests/tcg/x86_64-linux-user/float_convs |
| 717 | |
| 718 | will report the following:: |
| 719 | |
| 720 | core #, data accesses, data misses, dmiss rate, insn accesses, insn misses, imiss rate |
| 721 | 0 996695 508 0.0510% 2642799 18617 0.7044% |
| 722 | |
| 723 | address, data misses, instruction |
| 724 | 0x424f1e (_int_malloc), 109, movq %rax, 8(%rcx) |
| 725 | 0x41f395 (_IO_default_xsputn), 49, movb %dl, (%rdi, %rax) |
| 726 | 0x42584d (ptmalloc_init.part.0), 33, movaps %xmm0, (%rax) |
| 727 | 0x454d48 (__tunables_init), 20, cmpb $0, (%r8) |
| 728 | ... |
| 729 | |
| 730 | address, fetch misses, instruction |
| 731 | 0x4160a0 (__vfprintf_internal), 744, movl $1, %ebx |
| 732 | 0x41f0a0 (_IO_setb), 744, endbr64 |
| 733 | 0x415882 (__vfprintf_internal), 744, movq %r12, %rdi |
| 734 | 0x4268a0 (__malloc), 696, andq $0xfffffffffffffff0, %rax |
| 735 | ... |
| 736 | |
| 737 | The plugin has a number of arguments, all of them are optional: |
| 738 | |
| 739 | .. list-table:: Cache modelling arguments |
| 740 | :widths: 20 80 |
| 741 | :header-rows: 1 |
| 742 | |
| 743 | * - Option |
| 744 | - Description |
| 745 | * - limit=N |
| 746 | - Print top N icache and dcache thrashing instructions along with |
| 747 | their address, number of misses, and its disassembly. (default: 32) |
| 748 | * - icachesize=N |
| 749 | iblksize=B |
| 750 | iassoc=A |
| 751 | - Instruction cache configuration arguments. They specify the |
| 752 | cache size, block size, and associativity of the instruction |
| 753 | cache, respectively. (default: N = 16384, B = 64, A = 8) |
| 754 | * - dcachesize=N |
| 755 | - Data cache size (default: 16834) |
| 756 | * - dblksize=B |
| 757 | - Data cache block size (default: 64) |
| 758 | * - dassoc=A |
| 759 | - Data cache associativity (default: 8) |
| 760 | * - evict=POLICY |
| 761 | - Sets the eviction policy to POLICY. Available policies are: |
| 762 | ``lru``, ``fifo``, and ``rand``. The plugin will use |
| 763 | the specified policy for both instruction and data caches. |
| 764 | (default: POLICY = ``lru``) |
| 765 | * - cores=N |
| 766 | - Sets the number of cores for which we maintain separate icache |
| 767 | and dcache. (default: for linux-user, N = 1, for full system |
| 768 | emulation: N = cores available to guest) |
| 769 | * - l2=on |
| 770 | - Simulates a unified L2 cache (stores blocks for both |
| 771 | instructions and data) using the default L2 configuration (cache |
| 772 | size = 2MB, associativity = 16-way, block size = 64B). |
| 773 | * - l2cachesize=N |
| 774 | - L2 cache size (default: 2097152 (2MB)), implies ``l2=on`` |
| 775 | * - l2blksize=B |
| 776 | - L2 cache block size (default: 64), implies ``l2=on`` |
| 777 | * - l2assoc=A |
| 778 | - L2 cache associativity (default: 16), implies ``l2=on`` |
| 779 | |
| 780 | Stop on Trigger |
| 781 | ............... |
| 782 | |
| 783 | ``contrib/plugins/stoptrigger.c`` |
| 784 | |
| 785 | The stoptrigger plugin allows to setup triggers to stop emulation. |
| 786 | It can be used for research purposes to launch some code and precisely stop it |
| 787 | and understand where its execution flow went. |
| 788 | |
| 789 | Two types of triggers can be configured: a count of instructions to stop at, |
| 790 | or an address to stop at. Multiple triggers can be set at once. |
| 791 | |
| 792 | By default, QEMU will exit with return code 0. A custom return code can be |
| 793 | configured for each trigger using ``:CODE`` syntax. |
| 794 | |
| 795 | For example, to stop at the 20-th instruction with return code 41, at address |
| 796 | 0xd4 with return code 0 or at address 0xd8 with return code 42:: |
| 797 | |
| 798 | $ qemu-system-aarch64 $(QEMU_ARGS) \ |
| 799 | -plugin ./contrib/plugins/libstoptrigger.so,icount=20:41,addr=0xd4,addr=0xd8:42 -d plugin |
| 800 | |
| 801 | The plugin will log the reason of exit, for example:: |
| 802 | |
| 803 | 0xd4 reached, exiting |
| 804 | |
| 805 | Limit instructions per second |
| 806 | ............................. |
| 807 | |
| 808 | This plugin can limit the number of Instructions Per Second that are executed:: |
| 809 | |
| 810 | # get number of instructions |
| 811 | $ num_insn=$(./build/qemu-x86_64 -plugin ./build/tests/plugin/libinsn.so -d plugin /bin/true |& grep total | sed -e 's/.*: //') |
| 812 | # limit speed to execute in 10 seconds |
| 813 | $ time ./build/qemu-x86_64 -plugin ./build/contrib/plugins/libips.so,ips=$(($num_insn/10)) /bin/true |
| 814 | real 10.000s |
| 815 | |
| 816 | |
| 817 | .. list-table:: IPS arguments |
| 818 | :widths: 20 80 |
| 819 | :header-rows: 1 |
| 820 | |
| 821 | * - Option |
| 822 | - Description |
| 823 | * - ips=N |
| 824 | - Maximum number of instructions per cpu that can be executed in one second. |
| 825 | The plugin will sleep when the given number of instructions is reached. |
| 826 | * - ipq=N |
| 827 | - Instructions per quantum. How many instructions before we re-calculate time. |
| 828 | The lower the number the more accurate time will be, but the less efficient the plugin. |
| 829 | Defaults to ips/10 |
| 830 | |
| 831 | Uftrace |
| 832 | ....... |
| 833 | |
| 834 | ``contrib/plugins/uftrace.c`` |
| 835 | |
| 836 | This plugin generates a binary trace compatible with |
| 837 | `uftrace <https://github.com/namhyung/uftrace>`_. |
| 838 | |
| 839 | Plugin supports aarch64, x64 and riscv64, and works in user and system mode, |
| 840 | allowing to trace a system boot, which is not something possible usually. |
| 841 | |
| 842 | In user mode, the memory mapping is directly copied from ``/proc/self/maps`` at |
| 843 | the end of execution. Uftrace should be able to retrieve symbols by itself, |
| 844 | without any additional step. |
| 845 | In system mode, the default memory mapping is empty, and you can generate |
| 846 | one (and associated symbols) using ``contrib/plugins/uftrace_symbols.py``. |
| 847 | Symbols must be present in ELF binaries. |
| 848 | |
| 849 | It tracks the call stack (based on frame pointer analysis). Thus, your program |
| 850 | and its dependencies must be compiled using ``-fno-omit-frame-pointer |
| 851 | -mno-omit-leaf-frame-pointer``. In 2024, `Ubuntu and Fedora enabled it by |
| 852 | default again on 64-bit platforms |
| 853 | <https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html>`_. |
| 854 | On aarch64, this is less of a problem, as they are usually part of the ABI, |
| 855 | except for leaf functions. That's true for user space applications, but not |
| 856 | necessarily for bare metal code. You can read this `section |
| 857 | <uftrace_build_system_example>` to easily build a system with frame pointers. |
| 858 | |
| 859 | When tracing long scenarios (> 1 min), the generated trace can become very long, |
| 860 | making it hard to extract data from it. In this case, a simple solution is to |
| 861 | trace execution while generating a timestamped output log using |
| 862 | ``qemu-system-aarch64 ... | ts "%s"``. Then, ``uftrace --time-range=start~end`` |
| 863 | can be used to reduce trace for only this part of execution. |
| 864 | |
| 865 | Performance wise, overhead compared to normal tcg execution is around x5-x15. |
| 866 | |
| 867 | .. list-table:: Uftrace plugin arguments |
| 868 | :widths: 20 80 |
| 869 | :header-rows: 1 |
| 870 | |
| 871 | * - Option |
| 872 | - Description |
| 873 | * - trace-privilege-level=[on|off] |
| 874 | - Generate separate traces for each privilege level (Exception Level + |
| 875 | Security State on aarch64, Privilege levels on riscv64 and Rings on x64). |
| 876 | |
| 877 | .. list-table:: uftrace_symbols.py arguments |
| 878 | :widths: 20 80 |
| 879 | :header-rows: 1 |
| 880 | |
| 881 | * - Option |
| 882 | - Description |
| 883 | * - elf_file [elf_file ...] |
| 884 | - path to an ELF file. Use /path/to/file:0xdeadbeef to add a mapping offset. |
| 885 | * - --prefix-symbols |
| 886 | - prepend binary name to symbols |
| 887 | |
| 888 | Example user trace |
| 889 | ++++++++++++++++++ |
| 890 | |
| 891 | As an example, we can trace qemu itself running git:: |
| 892 | |
| 893 | $ ./build/qemu-aarch64 -plugin \ |
| 894 | build/contrib/plugins/libuftrace.so \ |
| 895 | ./build/qemu-aarch64 /usr/bin/git --help |
| 896 | |
| 897 | # and generate a chrome trace directly |
| 898 | $ uftrace dump --chrome | gzip > ~/qemu_aarch64_git_help.json.gz |
| 899 | |
| 900 | For convenience, you can download this trace `qemu_aarch64_git_help.json.gz |
| 901 | <https://github.com/p-b-o/qemu-assets/raw/refs/heads/master/qemu-uftrace/qemu_aarch64_git_help.json.gz>`_. |
| 902 | Download it and open this trace on https://ui.perfetto.dev/. You can zoom in/out |
| 903 | using :kbd:`W`, :kbd:`A`, :kbd:`S`, :kbd:`D` keys. |
| 904 | Some sequences taken from this trace: |
| 905 | |
| 906 | - Loading program and its interpreter |
| 907 | |
| 908 | .. image:: https://github.com/p-b-o/qemu-assets/blob/master/qemu-uftrace/loader_exec.png?raw=true |
| 909 | :height: 200px |
| 910 | |
| 911 | - open syscall |
| 912 | |
| 913 | .. image:: https://github.com/p-b-o/qemu-assets/blob/master/qemu-uftrace/open_syscall.png?raw=true |
| 914 | :height: 200px |
| 915 | |
| 916 | - TB creation |
| 917 | |
| 918 | .. image:: https://github.com/p-b-o/qemu-assets/blob/master/qemu-uftrace/tb_translation.png?raw=true |
| 919 | :height: 200px |
| 920 | |
| 921 | It's usually better to use ``uftrace record`` directly. However, tracing |
| 922 | binaries through qemu-user can be convenient when you don't want to recompile |
| 923 | them (``uftrace record`` requires instrumentation), as long as symbols are |
| 924 | present. |
| 925 | |
| 926 | Example system trace |
| 927 | ++++++++++++++++++++ |
| 928 | |
| 929 | A full trace example (chrome trace, from instructions below) generated from a |
| 930 | system boot can be found `here |
| 931 | <https://github.com/p-b-o/qemu-assets/raw/refs/heads/master/qemu-uftrace/aarch64_boot.json.gz>`_. |
| 932 | Download it and open this trace on https://ui.perfetto.dev/. You can see code |
| 933 | executed for all privilege levels, and zoom in/out using |
| 934 | :kbd:`W`, :kbd:`A`, :kbd:`S`, :kbd:`D` keys. You can find below some sequences |
| 935 | taken from this trace: |
| 936 | |
| 937 | - Two first stages of boot sequence in Arm Trusted Firmware (EL3 and S-EL1) |
| 938 | |
| 939 | .. image:: https://github.com/p-b-o/qemu-assets/blob/master/qemu-uftrace/bl3_to_bl1.png?raw=true |
| 940 | :height: 200px |
| 941 | |
| 942 | - U-boot initialization (until code relocation, after which we can't track it) |
| 943 | |
| 944 | .. image:: https://github.com/p-b-o/qemu-assets/blob/master/qemu-uftrace/uboot.png?raw=true |
| 945 | :height: 200px |
| 946 | |
| 947 | - Stat and open syscalls in kernel |
| 948 | |
| 949 | .. image:: https://github.com/p-b-o/qemu-assets/blob/master/qemu-uftrace/stat.png?raw=true |
| 950 | :height: 200px |
| 951 | |
| 952 | - Timer interrupt |
| 953 | |
| 954 | .. image:: https://github.com/p-b-o/qemu-assets/blob/master/qemu-uftrace/timer_interrupt.png?raw=true |
| 955 | :height: 200px |
| 956 | |
| 957 | - Poweroff sequence (from kernel back to firmware, NS-EL2 to EL3) |
| 958 | |
| 959 | .. image:: https://github.com/p-b-o/qemu-assets/blob/master/qemu-uftrace/poweroff.png?raw=true |
| 960 | :height: 200px |
| 961 | |
| 962 | Build and run system example |
| 963 | ++++++++++++++++++++++++++++ |
| 964 | |
| 965 | .. _uftrace_build_system_example: |
| 966 | |
| 967 | Building a full system image with frame pointers is not trivial. |
| 968 | |
| 969 | We provide a `simple way <https://github.com/p-b-o/qemu-linux-stack>`_ to |
| 970 | build an aarch64 system, combining Arm Trusted firmware, U-boot, Linux kernel |
| 971 | and debian userland. It's based on containers (``podman`` only) and |
| 972 | ``qemu-user-static (binfmt)`` to make sure it's easily reproducible and does not depend |
| 973 | on machine where you build it. |
| 974 | |
| 975 | You can follow the exact same instructions for a x64 system, combining edk2, |
| 976 | Linux, and Ubuntu, simply by switching to |
| 977 | `x86_64 <https://github.com/p-b-o/qemu-linux-stack/tree/x86_64>`_ branch. |
| 978 | |
| 979 | You can follow the exact same instructions for a riscv64 system, combining |
| 980 | opensbi, Linux, and Ubuntu, simply by switching to |
| 981 | `riscv64 <https://github.com/p-b-o/qemu-linux-stack/tree/riscv64>`_ branch. |
| 982 | |
| 983 | To build and run the system:: |
| 984 | |
| 985 | # Install dependencies |
| 986 | $ sudo apt install -y podman qemu-user-static |
| 987 | |
| 988 | $ git clone https://github.com/p-b-o/qemu-linux-stack |
| 989 | $ cd qemu-linux-stack |
| 990 | $ ./build.sh |
| 991 | |
| 992 | # system can be started using: |
| 993 | $ ./run.sh /path/to/qemu-system-aarch64 |
| 994 | |
| 995 | # generate a uftrace for execution (collect symbols automatically) |
| 996 | $ ./trace.sh /path/to/qemu-system-aarch64 |
| 997 | # show output log to read timestamps |
| 998 | $ cat uftrace.data/exec.log |
| 999 | # generate final trace (compressed) for perfetto |
| 1000 | $ ./perfetto.sh <start_timestamp> <end_timestamp> ~/trace.gz |
| 1001 | |
| 1002 | To generate manually the same trace:: |
| 1003 | |
| 1004 | # run true and poweroff the system |
| 1005 | $ env INIT=true ./run.sh path/to/qemu-system-aarch64 \ |
| 1006 | -plugin path/to/contrib/plugins/libuftrace.so,trace-privilege-level=on |
| 1007 | |
| 1008 | # generate symbols and memory mapping |
| 1009 | $ path/to/contrib/plugins/uftrace_symbols.py \ |
| 1010 | --prefix-symbols \ |
| 1011 | arm-trusted-firmware/build/qemu/debug/bl1/bl1.elf \ |
| 1012 | arm-trusted-firmware/build/qemu/debug/bl2/bl2.elf \ |
| 1013 | arm-trusted-firmware/build/qemu/debug/bl31/bl31.elf \ |
| 1014 | u-boot/u-boot:0x60000000 \ |
| 1015 | linux/vmlinux |
| 1016 | |
| 1017 | # inspect trace with |
| 1018 | $ uftrace replay |
| 1019 | |
| 1020 | Uftrace allows to filter the trace, and dump flamegraphs, or a chrome trace. |
| 1021 | This last one is very interesting to see visually the boot process:: |
| 1022 | |
| 1023 | $ uftrace dump --chrome > boot.json |
| 1024 | # Open your browser, and load boot.json on https://ui.perfetto.dev/. |
| 1025 | |
| 1026 | Long visual chrome traces can't be easily opened, thus, it might be |
| 1027 | interesting to generate them around a particular point of execution:: |
| 1028 | |
| 1029 | # execute qemu and timestamp output log |
| 1030 | $ env INIT=true ./run.sh path/to/qemu-system-aarch64 \ |
| 1031 | -plugin path/to/contrib/plugins/libuftrace.so,trace-privilege-level=on |& |
| 1032 | ts "%s" | tee exec.log |
| 1033 | |
| 1034 | $ cat exec.log | grep 'Run /init' |
| 1035 | 1753122320 [ 11.834391] Run /init as init process |
| 1036 | # init was launched at 1753122320 |
| 1037 | |
| 1038 | # generate trace around init execution (2 seconds): |
| 1039 | $ uftrace dump --chrome --time-range=1753122320~1753122322 > init.json |
| 1040 | |
| 1041 | Count traps |
| 1042 | ........... |
| 1043 | |
| 1044 | ``contrib/plugins/traps.c`` |
| 1045 | |
| 1046 | This plugin counts the number of interrupts (asynchronous events), exceptions |
| 1047 | (synchronous events) and host calls (e.g. semihosting) per cpu. |
| 1048 | |
| 1049 | Dynamic Linking Call |
| 1050 | .................... |
| 1051 | |
| 1052 | ``contrib/plugins/dlcall.c`` |
| 1053 | |
| 1054 | This plugin provides a dynamic linking function call interception mechanism |
| 1055 | for linux-user guests: the guest hands a call off to the host, where the plugin |
| 1056 | runs native code in its place instead of the guest emulating it. Interception |
| 1057 | alone enables several uses, for instance tracing or auditing guest calls. |
| 1058 | One use is acceleration by leveraging the host's native shared libraries. For |
| 1059 | example, a thunk layer can run the stock zlib ``minizip`` utility under |
| 1060 | emulation while forwarding its ``deflate`` calls to the host's native zlib |
| 1061 | library (libz). This avoids emulating those selected library calls instruction |
| 1062 | by instruction. |
| 1063 | |
| 1064 | The guest issues a reserved "magic" system call (4096 by default, configurable |
| 1065 | with ``syscall_num=N``) whose first argument selects a pass-through operation: |
| 1066 | dlopen/dlclose a host library, dlsym a symbol, and invoke a resolved host |
| 1067 | function. The plugin performs the operation on the host and consumes the |
| 1068 | syscall, so the real kernel never sees it. |
| 1069 | |
| 1070 | .. warning:: |
| 1071 | |
| 1072 | Trusted guests only. The guest can load arbitrary host libraries and run |
| 1073 | arbitrary code in the QEMU host process. The plugin is not a sandbox and |
| 1074 | provides no isolation. It also requires ``guest_base == 0`` (qemu-user's |
| 1075 | default) and a guest whose pointer width and endianness match the host's, as |
| 1076 | guest pointers are dereferenced as host addresses with no translation. |
| 1077 | |
| 1078 | The plugin intentionally keeps the QEMU side lightweight and knows nothing |
| 1079 | about any particular library or its calling convention. Producing the thunks |
| 1080 | for a real library is done entirely in userspace, and any toolchain can |
| 1081 | implement the interface. |
| 1082 | |
| 1083 | Loading the plugin is all that is required from QEMU's side: |
| 1084 | |
| 1085 | .. code-block:: shell |
| 1086 | |
| 1087 | qemu-x86_64 -plugin contrib/plugins/libdlcall.so <guest-program> ... |
| 1088 | |
| 1089 | If the default number does not suit the guest ABI, pick another one, and build |
| 1090 | the userspace side to issue the same one: |
| 1091 | |
| 1092 | .. code-block:: shell |
| 1093 | |
| 1094 | qemu-x86_64 -plugin contrib/plugins/libdlcall.so,syscall_num=8192 \ |
| 1095 | <guest-program> ... |
| 1096 | |
| 1097 | `Lorelei <https://github.com/rover2024/lorelei>`_ is one end-to-end userspace |
| 1098 | implementation of this: it provides the guest and host runtimes and an |
| 1099 | automated toolchain that generates the thunks from a library's headers, so guest |
| 1100 | library calls run on the host's native libraries. How it handles the parts the |
| 1101 | plugin leaves out, including argument marshalling, callbacks and variadic |
| 1102 | functions, can serve as a reference. It supports an x86_64 guest running on an |
| 1103 | x86_64, aarch64 or riscv64 host. |
| 1104 | |
| 1105 | A minimal end-to-end example uses a one-function library, ``libhello.so``, built |
| 1106 | two ways: the guest build tags its output ``(from the guest)`` and the host |
| 1107 | build ``(from the host)``. An unmodified guest program ``main`` calls |
| 1108 | ``hello("World", 7)``, and the thunk makes that same binary reach the host build |
| 1109 | in place of its own. The sources live under ``src/``: |
| 1110 | |
| 1111 | .. code-block:: c |
| 1112 | |
| 1113 | /* src/hello.h */ |
| 1114 | void hello(const char *name, int lucky); |
| 1115 | |
| 1116 | .. code-block:: c |
| 1117 | |
| 1118 | /* src/hello_guest.c */ |
| 1119 | #include "hello.h" |
| 1120 | #include <stdio.h> |
| 1121 | |
| 1122 | void hello(const char *name, int lucky) |
| 1123 | { |
| 1124 | printf("Hello, %s! Your lucky number is %d. (from the guest)\n", name, lucky); |
| 1125 | } |
| 1126 | |
| 1127 | .. code-block:: c |
| 1128 | |
| 1129 | /* src/hello_host.c */ |
| 1130 | #include "hello.h" |
| 1131 | #include <stdio.h> |
| 1132 | |
| 1133 | void hello(const char *name, int lucky) |
| 1134 | { |
| 1135 | printf("Hello, %s! Your lucky number is %d. (from the host)\n", name, lucky); |
| 1136 | } |
| 1137 | |
| 1138 | .. code-block:: c |
| 1139 | |
| 1140 | /* src/main.c */ |
| 1141 | #include "hello.h" |
| 1142 | |
| 1143 | int main(void) |
| 1144 | { |
| 1145 | hello("World", 7); |
| 1146 | return 0; |
| 1147 | } |
| 1148 | |
| 1149 | Lorelei ships a prebuilt toolchain (a "devkit") in its releases. Download the |
| 1150 | one for your host and unpack it: |
| 1151 | |
| 1152 | .. code-block:: shell |
| 1153 | |
| 1154 | # ARCH is your host's architecture: x86_64, aarch64 or riscv64. This example uses aarch64. |
| 1155 | # See https://github.com/rover2024/lorelei/releases |
| 1156 | ARCH=aarch64 |
| 1157 | VERSION=$(curl -fsSL -o /dev/null -w '%{url_effective}' \ |
| 1158 | https://github.com/rover2024/lorelei/releases/latest | sed 's|.*/tag/v||') |
| 1159 | wget "https://github.com/rover2024/lorelei/releases/download/v$VERSION/lorelei-devkit-$ARCH-$VERSION.tar.xz" |
| 1160 | tar -xf lorelei-devkit-$ARCH-$VERSION.tar.xz |
| 1161 | DEVKIT=lorelei-devkit-$ARCH |
| 1162 | |
| 1163 | Build the guest ``libhello.so`` (x86_64) and the host ``libhello.so`` (this |
| 1164 | host's architecture), then the guest program: |
| 1165 | |
| 1166 | .. code-block:: shell |
| 1167 | |
| 1168 | mkdir -p build/guest build/host |
| 1169 | $DEVKIT/bin/x86_64-linux-gnu-clang -shared -fPIC src/hello_guest.c -o build/guest/libhello.so |
| 1170 | cc -shared -fPIC src/hello_host.c -o build/host/libhello.so |
| 1171 | $DEVKIT/bin/x86_64-linux-gnu-clang src/main.c -Isrc -Lbuild/guest -lhello -o build/guest/main |
| 1172 | |
| 1173 | Run it under qemu: |
| 1174 | |
| 1175 | .. code-block:: shell |
| 1176 | |
| 1177 | qemu-x86_64 -L /usr/x86_64-linux-gnu/ -E LD_LIBRARY_PATH=build/guest build/guest/main |
| 1178 | |
| 1179 | which prints:: |
| 1180 | |
| 1181 | Hello, World! Your lucky number is 7. (from the guest) |
| 1182 | |
| 1183 | Now generate the thunk from the host ``libhello.so``. This produces a guest-side |
| 1184 | ``libhello.so`` that stands in for the guest build, and a host-side thunk library |
| 1185 | that dispatches to the host build: |
| 1186 | |
| 1187 | .. code-block:: shell |
| 1188 | |
| 1189 | $DEVKIT/bin/LoreMakeThunk.py --name hello --lib build/host/libhello.so \ |
| 1190 | --header hello.h -o thunks -- -Isrc |
| 1191 | |
| 1192 | Run the same ``main`` under the plugin. The call reaches the host build now: |
| 1193 | |
| 1194 | .. code-block:: shell |
| 1195 | |
| 1196 | LD_LIBRARY_PATH=$DEVKIT/lib:build/host \ |
| 1197 | qemu-x86_64 -plugin contrib/plugins/libdlcall.so \ |
| 1198 | -E LD_LIBRARY_PATH=$DEVKIT/x86_64/lib:thunks/x86_64 \ |
| 1199 | -L /usr/x86_64-linux-gnu/ \ |
| 1200 | build/guest/main |
| 1201 | |
| 1202 | which prints:: |
| 1203 | |
| 1204 | Hello, World! Your lucky number is 7. (from the host) |
| 1205 | |
| 1206 | .. list-table:: Dynamic Linking Call arguments |
| 1207 | :widths: 20 80 |
| 1208 | :header-rows: 1 |
| 1209 | |
| 1210 | * - Option |
| 1211 | - Description |
| 1212 | * - syscall_num=N |
| 1213 | - The magic syscall number the guest issues (default 4096). It must be a |
| 1214 | number the guest ABI does not use for a real syscall, and does not |
| 1215 | reject before the plugin sees it, which bounds the choice from both |
| 1216 | sides. |
| 1217 | |
| 1218 | Other emulation features |
| 1219 | ------------------------ |
| 1220 | |
| 1221 | When running system emulation you can also enable deterministic |
| 1222 | execution which allows for repeatable record/replay debugging. See |
| 1223 | :ref:`Record/Replay<replay>` for more details. |