master
rst 499 lines 20.6 KB
Raw
1 .. |msrv| replace:: 1.83.0
2
3 Rust in QEMU
4 ============
5
6 Rust in QEMU is a project to enable using the Rust programming language
7 to add new functionality to QEMU.
8
9 Right now, the focus is on making it possible to write devices that inherit
10 from ``SysBusDevice`` in `*safe*`__ Rust. Later, it may become possible
11 to write other kinds of devices (e.g. PCI devices that can do DMA),
12 complete boards, or backends (e.g. block device formats).
13
14 __ https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html
15
16 Building the Rust in QEMU code
17 ------------------------------
18
19 The Rust in QEMU code is included in the emulators via Meson. Meson
20 invokes rustc directly, building static libraries that are then linked
21 together with the C code. This is completely automatic when you run
22 ``make`` or ``ninja``.
23
24 However, Meson is able to consume ``Cargo.toml`` files and tries
25 to be easy to use for people who are accustomed to the more "normal"
26 Cargo-based development workflow. In the case of QEMU, in addition,
27 it is possible to use ``cargo`` for common Rust-specific coding
28 tasks, in particular to invoke ``clippy``, ``rustfmt`` and ``rustdoc``.
29
30 To this end, QEMU includes a ``build.rs`` build script that picks up
31 generated sources from QEMU's build directory and puts it in Cargo's
32 output directory (typically ``target/``). A vanilla invocation
33 of Cargo will complain that it cannot find the generated sources,
34 which can be fixed in different ways:
35
36 * by using Makefile targets, provided by Meson, that run ``clippy`` or
37 ``rustdoc``:
38
39 make clippy
40 make rustdoc
41
42 A target for ``rustfmt`` is also declared in ``rust/meson.build``:
43
44 make rustfmt
45
46 * by invoking ``cargo`` through the Meson `development environment`__
47 feature::
48
49 pyvenv/bin/meson devenv -w ../rust cargo clippy --tests
50 pyvenv/bin/meson devenv -w ../rust cargo fmt
51
52 If you are going to use ``cargo`` repeatedly, ``pyvenv/bin/meson devenv``
53 will enter a shell where commands like ``cargo fmt`` just work.
54
55 __ https://mesonbuild.com/Commands.html#devenv
56
57 * by pointing the ``MESON_BUILD_ROOT`` to the top of your QEMU build
58 tree. This third method is useful if you are using ``rust-analyzer``;
59 you can set the environment variable through the
60 ``rust-analyzer.cargo.extraEnv`` setting.
61
62 As shown above, you can use the ``--tests`` option as usual to operate on test
63 code. Note however that you cannot *build* or run tests via ``cargo``, because
64 they need support C code from QEMU that Cargo does not know about. Tests can
65 be run via Meson (``pyvenv/bin/meson test``) or ``make``::
66
67 make check-rust
68
69 Note that doctests require all ``.o`` files from the build to be available.
70
71 Supported tools
72 '''''''''''''''
73
74 QEMU supports rustc version 1.83.0 and newer. The following features
75 from relatively new versions of Rust are not used for historical reasons;
76 patches are welcome:
77
78 * associated constants are still explicitly marked ``'static`` (`changed in
79 1.81.0`__)
80
81 * ``&raw`` (stable in 1.82.0).
82
83 * NUL-terminated file names with ``#[track_caller]`` are scheduled for
84 inclusion as ``#![feature(location_file_nul)]``, but it will be a while
85 before QEMU can use them. For now, there is special code in
86 ``util/error.c`` to support non-NUL-terminated file names.
87
88 Associated const equality would be nice to have for some users of
89 ``callbacks::FnCall``, but is still experimental. Const assertions
90 are used instead.
91
92 __ https://github.com/rust-lang/rust/pull/125258
93
94 QEMU also supports version 0.60.x of bindgen, which is missing option
95 ``--generate-cstr``. This option requires version 0.66.x and will
96 be adopted as soon as supporting these older versions is not necessary
97 anymore.
98
99 Writing Rust code in QEMU
100 -------------------------
101
102 QEMU includes several crates:
103
104 * ``common`` provides Rust-only utilities
105
106 * ``bql``, ``chardev``, ``hw/core``, ``migration``, ``qom``, ``system``,
107 ``util`` for bindings to respective QEMU C library APIs
108
109 * ``qemu_macros`` defines several procedural macros that are useful when
110 writing C code
111
112 * ``pl011`` (under ``rust/hw/char/pl011``) and ``hpet`` (under ``rust/hw/timer/hpet``)
113 are sample devices that demonstrate Rust binding usage and ``qemu_macros``, and are
114 used to further develop them. These two crates are functional\ [#issues]_ replacements
115 for the ``hw/char/pl011.c`` and ``hw/timer/hpet.c`` files.
116
117 .. [#issues] The ``pl011`` crate is synchronized with ``hw/char/pl011.c``
118 as of commit 3e0f118f82. The ``hpet`` crate is synchronized as of
119 commit 1433e38cc8. Both are lacking tracing functionality.
120
121 This section explains how to work with them.
122
123 Status
124 ''''''
125
126 The stability of the modules can be defined as:
127
128 - *complete*: ready for use in new devices; if applicable, the API supports the
129 full functionality available in C
130
131 - *stable*: ready for production use, the API is safe and should not undergo
132 major changes
133
134 - *proof of concept*: the API is subject to change but allows working with safe
135 Rust
136
137 - *initial*: the API is in its initial stages; it requires large amount of
138 unsafe code; it might have soundness or type-safety issues
139
140 The status of the modules is as follows:
141
142 ========================== ======================
143 module status
144 ========================== ======================
145 ``bql::cell`` stable
146 ``common::assertions`` stable
147 ``common::bitops`` complete
148 ``common::callbacks`` complete
149 ``common::errno`` complete
150 ``common::zeroable`` stable
151 ``hwcore::irq`` complete
152 ``hwcore::qdev`` stable
153 ``hwcore::sysbus`` stable
154 ``migration::migratable`` proof of concept
155 ``migration::vmstate`` stable
156 ``qom`` stable
157 ``system::memory`` stable
158 ``util::error`` stable
159 ``util::log`` proof of concept
160 ``util::module`` complete
161 ``util::timer`` stable
162 ========================== ======================
163
164 .. note::
165 API stability is not a promise, if anything because the C APIs are not a stable
166 interface either. Also, ``unsafe`` interfaces may be replaced by safe interfaces
167 later.
168
169 Naming convention
170 '''''''''''''''''
171
172 C function names usually are prefixed according to the data type that they
173 apply to, for example ``timer_mod`` or ``sysbus_connect_irq``. Furthermore,
174 both function and structs sometimes have a ``qemu_`` or ``QEMU`` prefix.
175 Generally speaking, these are all removed in the corresponding Rust functions:
176 ``QEMUTimer`` becomes ``timer::Timer``, ``timer_mod`` becomes ``Timer::modify``,
177 ``sysbus_connect_irq`` becomes ``SysBusDeviceMethods::connect_irq``.
178
179 Sometimes however a name appears multiple times in the QOM class hierarchy,
180 and the only difference is in the prefix. An example is ``qdev_realize`` and
181 ``sysbus_realize``. In such cases, whenever a name is not unique in
182 the hierarchy, always add the prefix to the classes that are lower in
183 the hierarchy; for the top class, decide on a case by case basis.
184
185 For example:
186
187 ========================== =========================================
188 ``device_cold_reset()`` ``DeviceMethods::cold_reset()``
189 ``pci_device_reset()`` ``PciDeviceMethods::pci_device_reset()``
190 ``pci_bridge_reset()`` ``PciBridgeMethods::pci_bridge_reset()``
191 ========================== =========================================
192
193 Here, the name is not exactly the same, but nevertheless ``PciDeviceMethods``
194 adds the prefix to avoid confusion, because the functionality of
195 ``device_cold_reset()`` and ``pci_device_reset()`` is subtly different.
196
197 In this case, however, no prefix is needed:
198
199 ========================== =========================================
200 ``device_realize()`` ``DeviceMethods::realize()``
201 ``sysbus_realize()`` ``SysbusDeviceMethods::sysbus_realize()``
202 ``pci_realize()`` ``PciDeviceMethods::pci_realize()``
203 ========================== =========================================
204
205 Here, the lower classes do not add any functionality, and mostly
206 provide extra compile-time checking; the basic *realize* functionality
207 is the same for all devices. Therefore, ``DeviceMethods`` does not
208 add the prefix.
209
210 Whenever a name is unique in the hierarchy, instead, you should
211 always remove the class name prefix.
212
213 Common pitfalls
214 '''''''''''''''
215
216 Rust has very strict rules with respect to how you get an exclusive (``&mut``)
217 reference; failure to respect those rules is a source of undefined behavior.
218 In particular, even if a value is loaded from a raw mutable pointer (``*mut``),
219 it *cannot* be casted to ``&mut`` unless the value was stored to the ``*mut``
220 from a mutable reference. Furthermore, it is undefined behavior if any
221 shared reference was created between the store to the ``*mut`` and the load::
222
223 let mut p: u32 = 42;
224 let p_mut = &mut p; // 1
225 let p_raw = p_mut as *mut u32; // 2
226
227 // p_raw keeps the mutable reference "alive"
228
229 let p_shared = &p; // 3
230 println!("access from &u32: {}", *p_shared);
231
232 // Bring back the mutable reference, its lifetime overlaps
233 // with that of a shared reference.
234 let p_mut = unsafe { &mut *p_raw }; // 4
235 println!("access from &mut 32: {}", *p_mut);
236
237 println!("access from &u32: {}", *p_shared); // 5
238
239 These rules can be tested with `MIRI`__, for example.
240
241 __ https://github.com/rust-lang/miri
242
243 Almost all Rust code in QEMU will involve QOM objects, and pointers to these
244 objects are *shared*, for example because they are part of the QOM composition
245 tree. This creates exactly the above scenario:
246
247 1. a QOM object is created
248
249 2. a ``*mut`` is created, for example as the opaque value for a ``MemoryRegion``
250
251 3. the QOM object is placed in the composition tree
252
253 4. a memory access dereferences the opaque value to a ``&mut``
254
255 5. but the shared reference is still present in the composition tree
256
257 Because of this, QOM objects should almost always use ``&self`` instead
258 of ``&mut self``; access to internal fields must use *interior mutability*
259 to go from a shared reference to a ``&mut``.
260
261 Whenever C code provides you with an opaque ``void *``, avoid converting it
262 to a Rust mutable reference, and use a shared reference instead. The
263 ``bql::cell`` module provides wrappers that can be used to tell the
264 Rust compiler about interior mutability, and optionally to enforce locking
265 rules for the "Big QEMU Lock". In the future, similar cell types might
266 also be provided for ``AioContext``-based locking as well.
267
268 In particular, device code will usually rely on the ``BqlRefCell`` and
269 ``BqlCell`` type to ensure that data is accessed correctly under the
270 "Big QEMU Lock". These cell types are also known to the ``vmstate``
271 crate, which is able to "look inside" them when building an in-memory
272 representation of a ``struct``'s layout. Note that the same is not true
273 of a ``RefCell`` or ``Mutex``.
274
275 Bindings code instead will usually use the ``Opaque`` type, which hides
276 the contents of the underlying struct and can be easily converted to
277 a raw pointer, for use in calls to C functions. It can be used for
278 example as follows::
279
280 #[repr(transparent)]
281 #[derive(Debug, common::Wrapper)]
282 pub struct Object(Opaque<bindings::Object>);
283
284 where the special ``derive`` macro provides useful methods such as
285 ``from_raw``, ``as_ptr`, ``as_mut_ptr`` and ``raw_get``. The bindings will
286 then manually check for the big QEMU lock with assertions, which allows
287 the wrapper to be declared thread-safe::
288
289 unsafe impl Send for Object {}
290 unsafe impl Sync for Object {}
291
292 Writing bindings to C code
293 ''''''''''''''''''''''''''
294
295 Here are some things to keep in mind when working on the QEMU Rust crate.
296
297 **Look at existing code**
298 Very often, similar idioms in C code correspond to similar tricks in
299 Rust bindings. If the C code uses ``offsetof``, look at qdev properties
300 or ``vmstate``. If the C code has a complex const struct, look at
301 ``MemoryRegion``. Reuse existing patterns for handling lifetimes;
302 for example use ``&T`` for QOM objects that do not need a reference
303 count (including those that can be embedded in other objects) and
304 ``Owned<T>`` for those that need it.
305
306 **Use the type system**
307 Bindings often will need access information that is specific to a type
308 (either a builtin one or a user-defined one) in order to pass it to C
309 functions. Put them in a trait and access it through generic parameters.
310 The ``vmstate`` module has examples of how to retrieve type information
311 for the fields of a Rust ``struct``.
312
313 **Prefer unsafe traits to unsafe functions**
314 Unsafe traits are much easier to prove correct than unsafe functions.
315 They are an excellent place to store metadata that can later be accessed
316 by generic functions. C code usually places metadata in global variables;
317 in Rust, they can be stored in traits and then turned into ``static``
318 variables. Often, unsafe traits can be generated by procedural macros.
319
320 **Document limitations due to old Rust versions**
321 If you need to settle for an inferior solution because of the currently
322 supported set of Rust versions, document it in the source and in this
323 file. This ensures that it can be fixed when the minimum supported
324 version is bumped.
325
326 **Keep locking in mind**.
327 When marking a type ``Sync``, be careful of whether it needs the big
328 QEMU lock. Use ``BqlCell`` and ``BqlRefCell`` for interior data,
329 or assert ``bql_locked()``.
330
331 **Don't be afraid of complexity, but document and isolate it**
332 It's okay to be tricky; device code is written more often than bindings
333 code and it's important that it is idiomatic. However, you should strive
334 to isolate any tricks in a place (for example a ``struct``, a trait
335 or a macro) where it can be documented and tested. If needed, include
336 toy versions of the code in the documentation.
337
338 FFI Binding Generation
339 ''''''''''''''''''''''
340
341 QEMU's Rust integration uses multiple ``*-sys`` crates that contain raw FFI
342 bindings to different QEMU subsystems. These crates mirror the dependency
343 structure that meson.build uses for C code, and which is reflected in
344 ``static_library()`` declarations. For example:
345
346 * util-sys: Basic utilities (no dependencies)
347 * qom-sys: QEMU Object Model (depends on util-sys)
348 * chardev-sys: Character devices (depends on qom-sys, util-sys)
349 * hwcore-sys: Hardware core (depends on qom-sys, util-sys)
350 * migration-sys: Migration (depends on util-sys)
351 * system-sys: System-level APIs (depends on all others)
352
353 Having multiple crates avoids massive rebuilds of all Rust code when C headers
354 are changed. On the other hand, bindgen is not aware of how headers are split
355 across crates, and therefore it would generate declarations for dependencies
356 again. These duplicate declarations are not only large, they create distinct
357 types and therefore they are incompatible with each other.
358
359 Bindgen Configuration
360 ~~~~~~~~~~~~~~~~~~~~~
361
362 Bindgen options such as symbol blocklists or how to configure enums can be
363 defined in each crate's ``Cargo.toml`` via a ``[package.metadata.bindgen]`` section.
364 For example::
365
366 [package.metadata.bindgen]
367 header = "wrapper.h" # Main header file for this crate
368 rustified-enum = ["QEMUClockType"] # Enums to generate as Rust enums
369 bitfield-enum = ["VMStateFlags"] # Enums to treat as bitfields
370 blocklist-function = [ # Functions to exclude
371 "vmstate_register_ram",
372 "vmstate_unregister_ram"
373 ]
374 additional-files = [ # Extra files to allowlist
375 "include/system/memory_ldst.*"
376 ]
377
378 All bindgen options are supported in the metadata section. The complete list
379 can be found in ``rust/bindings/generate_bindgen_args.py``.
380
381 Dependency Management
382 ~~~~~~~~~~~~~~~~~~~~~
383
384 By examining the dependency chain before bindgen creates the code for
385 the ``*-sys`` crates, the build system ensures that header files included in
386 one crate are blocked from appearing in dependent crates, thus avoiding
387 duplicate definitions. Dependent crates can import the definition via
388 "use" statements.
389
390 This dependency-aware binding generation is handled automatically by
391 ``rust/bindings/generate_bindgen_args.py``, which processes the Cargo.toml
392 files in dependency order and generates appropriate ``--allowlist-file`` and
393 ``--blocklist-file`` arguments for bindgen.
394
395 Writing procedural macros
396 '''''''''''''''''''''''''
397
398 By conventions, procedural macros are split in two functions, one
399 returning ``Result<proc_macro2::TokenStream, syn::Error>`` with the body of
400 the procedural macro, and the second returning ``proc_macro::TokenStream``
401 which is the actual procedural macro. The former's name is the same as
402 the latter with the ``_or_error`` suffix. The code for the latter is more
403 or less fixed; it follows the following template, which is fixed apart
404 from the type after ``as`` in the invocation of ``parse_macro_input!``::
405
406 #[proc_macro_derive(Object)]
407 pub fn derive_object(input: TokenStream) -> TokenStream {
408 let input = parse_macro_input!(input as DeriveInput);
409
410 derive_object_or_error(input)
411 .unwrap_or_else(syn::Error::into_compile_error)
412 .into()
413 }
414
415 The ``qemu_macros`` crate has utility functions to examine a
416 ``DeriveInput`` and perform common checks (e.g. looking for a struct
417 with named fields). These functions return ``Result<..., syn::Error>``
418 and can be used easily in the procedural macro function::
419
420 fn derive_object_or_error(input: DeriveInput) ->
421 Result<proc_macro2::TokenStream, Error>
422 {
423 is_c_repr(&input, "#[derive(Object)]")?;
424
425 let name = &input.ident;
426 let parent = &get_fields(&input, "#[derive(Object)]")?[0].ident;
427 ...
428 }
429
430 Use procedural macros with care. They are mostly useful for two purposes:
431
432 * Performing consistency checks; for example ``#[derive(Object)]`` checks
433 that the structure has ``#[repr[C])`` and that the type of the first field
434 is consistent with the ``ObjectType`` declaration.
435
436 * Extracting information from Rust source code into traits, typically based
437 on types and attributes. For example, ``#[derive(TryInto)]`` builds an
438 implementation of ``TryFrom``, and it uses the ``#[repr(...)]`` attribute
439 as the ``TryFrom`` source and error types.
440
441 Procedural macros can be hard to debug and test; if the code generation
442 exceeds a few lines of code, it may be worthwhile to delegate work to
443 "regular" declarative (``macro_rules!``) macros and write unit tests for
444 those instead.
445
446
447 Coding style
448 ''''''''''''
449
450 Code should pass clippy and be formatted with rustfmt.
451
452 Right now, only the nightly version of ``rustfmt`` is supported. This
453 might change in the future. While CI checks for correct formatting via
454 ``cargo fmt --check``, maintainers can fix this for you when applying patches.
455
456 It is expected that QEMU Rust crates provides full ``rustdoc`` documentation for
457 bindings that are in their final shape or close.
458
459 Adding dependencies
460 -------------------
461
462 Generally, the set of dependent crates is kept small. Think twice before
463 adding a new external crate, especially if it comes with a large set of
464 dependencies itself. Sometimes QEMU only needs a small subset of the
465 functionality; see for example QEMU's ``assertions`` module. Also,
466 choose a version of the crate that works with QEMU's minimum supported
467 Rust version (|msrv|).
468
469 On top of this recommendation, adding external crates to QEMU is a
470 slightly complicated process, mostly due to the need to teach Meson how
471 to download them. While QEMU uses Meson's support for parsing ``Cargo.toml``
472 files, it ships ``.wrap`` files instead of using ``Cargo.lock``; this way,
473 distros can adjust the set of dependencies to the exact versions they use.
474 The versions specified in QEMU's ``Cargo.lock`` must be the same as the
475 one in the wrap file.
476
477 The wrap file must be named ``NAME-SEMVER-rs.wrap``, where ``NAME``
478 is the name of the crate and ``SEMVER`` is the version up to and including the
479 first non-zero number. For example, a crate with version ``0.2.3`` will use
480 ``0.2`` for its ``SEMVER``, while a crate with version ``1.0.84`` will use ``1``.
481
482 Usually, Meson is able to figure out how to build the crate, and also handles
483 cross compilation correctly. For crates that have a ``build.rs`` file,
484 equivalent rules must be added to
485 ``subprojects/packagefiles/NAME-SEMVER-rs/meson/meson.build``.
486 The file can modify the ``extra_args`` and ``extra_deps`` variables,
487 which contain respectively the compiler arguments and external dependencies
488 for the crate.
489
490 After every change to the ``meson/meson.build`` file you have to update the
491 patched version with ``meson subprojects update --reset ``NAME-SEMVER-rs``.
492 This might be automated in the future.
493
494 Also, after every change to the file it is strongly suggested to do a dummy
495 change to the ``.wrap`` file (for example adding a comment like ``# version 2``),
496 which will help Meson notice that the subproject is out of date.
497
498 As a last step, add the new subproject to ``scripts/archive-source.sh``,
499 ``scripts/make-release`` and ``subprojects/.gitignore``.