hw/arm/tegra241-cmdqv: Document the CMDQV design and lifecycle
Add an overview describing the Tegra241 CMDQV passthrough model, MMIO layout, guest-driven lifecycle, and per-VM isolation. Tested-by: Nicolin Chen <nicolinc@nvidia.com> Signed-off-by: Shameer Kolothum <skolothumtho@nvidia.com> Tested-by: Eric Auger <eric.auger@redhat.com> Reviewed-by: Eric Auger <eric.auger@redhat.com> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com> Message-id: 20260609112552.378999-31-skolothumtho@nvidia.com Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Shameer Kolothum committed
Jun 9, 2026 at 12:25 UTC
295ca0e14c6169e831c4a277e48df3a4aa0d8606
1 file changed
+100
hw/arm/tegra241-cmdqv.c
+100
@@ -7,6 +7,106 @@
7
* SPDX-License-Identifier: GPL-2.0-or-later
8
*/
9
10
+/*
11
+ * Tegra241 CMDQV - overview
12
+ * =========================
13
+ *
14
+ * NVIDIA Tegra241 extends SMMUv3 with a Command Queue Virtualization (CMDQ-V)
15
+ * block. It lets a guest issue SMMU invalidation commands directly to
16
+ * dedicated hardware queues (vCMDQs) without trapping into the hypervisor on
17
+ * the fast path. vCMDQs are exclusively allocated to Virtual Interfaces
18
+ * (VINTFs); the host kernel allocates one VINTF per emulated SMMUv3 instance
19
+ * via iommufd. QEMU emulates the CMDQV MMIO region and drives the host kernel
20
+ * calls (VIOMMU_ALLOC, HW_QUEUE_ALLOC, mmap); the actual command processing
21
+ * happens on real hardware.
22
+ *
23
+ * A vCMDQ becomes functional only once allocated to the host VINTF; until then
24
+ * no command processing happens, and trapped register accesses fall back to a
25
+ * QEMU-side cache. After allocation, the cached register state is migrated to
26
+ * the hardware and command processing runs on the host; guest accesses to the
27
+ * live control/status registers then bypass QEMU and reach the host directly.
28
+ *
29
+ * MMIO layout (64KB pages, total TEGRA241_CMDQV_IO_LEN)
30
+ * -----------------------------------------------------
31
+ * 0x00000 CMDQV Config page: QEMU-trapped.
32
+ * 0x10000 Direct vCMDQ Page 0 (control/status): QEMU-trapped and routed
33
+ * to either the mmap'd host VINTF Page 0 (if the vCMDQ has been
34
+ * allocated to a VINTF) or a per-vCMDQ register cache (otherwise).
35
+ * 0x20000 Direct vCMDQ Page 1 (BASE / DRAM addresses): QEMU-trapped.
36
+ * 0x30000 VINTF Page 0 (per-VINTF control/status): the guest's virtual
37
+ * VINTF Page 0 aperture, backed by the host VINTF Page 0 (mmap'd
38
+ * via iommufd) and installed into guest MMIO as a RAM-device
39
+ * subregion when VINTF is enabled; subsequent accesses bypass QEMU.
40
+ * 0x40000 VINTF Page 1 (per-VINTF BASE): QEMU-trapped. Although this is
41
+ * a HW alias of the direct Page 1, the kernel only exposes mmap
42
+ * for the host VINTF Page 0; the host VINTF Page 1 is not mmap'd
43
+ * and stays trapped.
44
+ *
45
+ * The direct vCMDQ apertures (0x10000/0x20000) are HW aliases of the VINTF
46
+ * apertures (0x30000/0x40000); they expose the same per-vCMDQ register slots
47
+ * under different addressing.
48
+ *
49
+ * The direct vCMDQ Page 0 stays trapped rather than aliased to the host VINTF
50
+ * Page 0 mmap. The CMDQV architecture allows software to program a vCMDQ
51
+ * through the direct aperture before allocating it to a VINTF; aliasing to
52
+ * the host VINTF Page 0 mmap would route those accesses into unallocated
53
+ * logical slots where the hardware silently drops them, so trapping keeps
54
+ * accesses well-defined for an unallocated vCMDQ.
55
+ *
56
+ * Lifecycle (driven by guest events)
57
+ * ----------------------------------
58
+ * 1. First vfio-pci device attach (.set_iommu_device) triggers:
59
+ * - tegra241_cmdqv_probe(): IOMMU_GET_HW_INFO confirms host CMDQV support.
60
+ * - IOMMU_VIOMMU_ALLOC: the kernel allocates and enables a VINTF for this
61
+ * VM, configures the VM's VMID (from its stage-2 HWPT) in VINTF_CONFIG,
62
+ * forces HYP_OWN=0, and returns the mmap offset/length for the host
63
+ * VINTF Page 0, which QEMU then mmap()s.
64
+ *
65
+ * 2. Guest writes VINTF_CONFIG.ENABLE = 1:
66
+ * QEMU installs the mmap'd host VINTF Page 0 into guest MMIO as the guest's
67
+ * virtual VINTF Page 0 aperture (a RAM-device subregion) and reports
68
+ * STATUS.ENABLE_OK = 1. The aperture is now a direct window onto the host
69
+ * page, so accesses no longer trap into QEMU; a vCMDQ within it operates as
70
+ * a real command queue only once it has been allocated (step 3).
71
+ *
72
+ * 3. Guest completes vCMDQ setup (BASE, CMDQ_ALLOC_MAP.ALLOC, CMDQV_EN,
73
+ * VINTF.ENABLE, in any order; each precondition write retries the HW queue
74
+ * allocation):
75
+ * IOMMU_HW_QUEUE_ALLOC grants the guest a new host vCMDQ in this VM's
76
+ * VINTF, binding the guest BASE GPA (translated through stage-2 and pinned
77
+ * by the kernel) to it.
78
+ *
79
+ * 4. Guest SMMU driver programs a Stream Table Entry for a passthrough
80
+ * device: IOMMU_VDEVICE_ALLOC programs SID_MATCH/SID_REPLACE in this VM's
81
+ * VINTF so that the HW translates the device's guest vSID into its host
82
+ * pSID. Commands referencing unmapped SIDs are rejected by HW.
83
+ *
84
+ * This reflects the current accel SMMUv3 design, which allocates the
85
+ * vDEVICE when the guest programs the STE.
86
+ *
87
+ * Per-VM isolation
88
+ * ----------------
89
+ * - Each VM has its own iommufd FD; all iommufd objects (VINTF, vdevices,
90
+ * hw_queues, mmap regions) belong to that FD. Cross-FD lookups fail, so
91
+ * one VM cannot reach another VM's IDs.
92
+ * - IOMMU_VIOMMU_ALLOC configures the VM's VMID in VINTF_CONFIG; the CMDQV
93
+ * hardware substitutes / checks VMID on every command the guest issues.
94
+ * - The kernel allocates the VINTF with HYP_OWN = 0, which restricts the
95
+ * guest to a safe subset of commands.
96
+ * - IOMMU_VDEVICE_ALLOC populates SID_MATCH/SID_REPLACE so invalidations
97
+ * only reach the host StreamIDs assigned to this VM (see step 4).
98
+ * - IOMMU_HW_QUEUE_ALLOC binds each vCMDQ to a single VINTF, so a guest
99
+ * cannot reach a vCMDQ that belongs to another VM.
100
+ *
101
+ * Limits exposed to the guest
102
+ * ---------------------------
103
+ * One VINTF per emulated SMMUv3 and two vCMDQs per VINTF. The HW maximum
104
+ * vCMDQ size is 8MiB, but the size QEMU exposes to the guest may be smaller.
105
+ * The queue must be physically contiguous in host memory, so QEMU caps the
106
+ * exposed size to the host memory-backend page size. Use hugepage backing to
107
+ * reach the 8MiB maximum.
108
+ */
109
+
110
#include "qemu/osdep.h"
111
#include "qemu/log.h"
112