| 1 | Compute Express Link (CXL) |
| 2 | ========================== |
| 3 | From the view of a single host, CXL is an interconnect standard that |
| 4 | targets accelerators and memory devices attached to a CXL host. |
| 5 | This description will focus on those aspects visible either to |
| 6 | software running on a QEMU emulated host or to the internals of |
| 7 | functional emulation. As such, it will skip over many of the |
| 8 | electrical and protocol elements that would be more of interest |
| 9 | for real hardware and will dominate more general introductions to CXL. |
| 10 | It will also completely ignore the fabric management aspects of CXL |
| 11 | by considering only a single host and a static configuration. |
| 12 | |
| 13 | CXL shares many concepts and much of the infrastructure of PCI Express, |
| 14 | with CXL Host Bridges, which have CXL Root Ports which may be directly |
| 15 | attached to CXL or PCI End Points. Alternatively there may be CXL Switches |
| 16 | with CXL and PCI Endpoints attached below them. In many cases additional |
| 17 | control and capabilities are exposed via PCI Express interfaces. |
| 18 | This sharing of interfaces and hence emulation code is reflected |
| 19 | in how the devices are emulated in QEMU. In most cases the various |
| 20 | CXL elements are built upon an equivalent PCIe devices. |
| 21 | |
| 22 | CXL devices support the following interfaces: |
| 23 | |
| 24 | * Most conventional PCIe interfaces |
| 25 | |
| 26 | - Configuration space access |
| 27 | - BAR mapped memory accesses used for registers and mailboxes. |
| 28 | - MSI/MSI-X |
| 29 | - AER |
| 30 | - DOE mailboxes |
| 31 | - IDE |
| 32 | - Many other PCI express defined interfaces.. |
| 33 | |
| 34 | * Memory operations |
| 35 | |
| 36 | - Equivalent of accessing DRAM / NVDIMMs. Any access / feature |
| 37 | supported by the host for normal memory should also work for |
| 38 | CXL attached memory devices. |
| 39 | |
| 40 | * Cache operations. The are mostly irrelevant to QEMU emulation as |
| 41 | QEMU is not emulating a coherency protocol. Any emulation related |
| 42 | to these will be device specific and is out of the scope of this |
| 43 | document. |
| 44 | |
| 45 | CXL 2.0 Device Types |
| 46 | -------------------- |
| 47 | CXL 2.0 End Points are often categorized into three types. |
| 48 | |
| 49 | **Type 1:** These support coherent caching of host memory. Example might |
| 50 | be a crypto accelerators. May also have device private memory accessible |
| 51 | via means such as PCI memory reads and writes to BARs. |
| 52 | |
| 53 | **Type 2:** These support coherent caching of host memory and host |
| 54 | managed device memory (HDM) for which the coherency protocol is managed |
| 55 | by the host. This is a complex topic, so for more information on CXL |
| 56 | coherency see the CXL 2.0 specification. |
| 57 | |
| 58 | **Type 3 Memory devices:** These devices act as a means of attaching |
| 59 | additional memory (HDM) to a CXL host including both volatile and |
| 60 | persistent memory. The CXL topology may support interleaving across a |
| 61 | number of Type 3 memory devices using HDM Decoders in the host, host |
| 62 | bridge, switch upstream port and endpoints. |
| 63 | |
| 64 | Scope of CXL emulation in QEMU |
| 65 | ------------------------------ |
| 66 | The focus of CXL emulation is CXL revision 2.0 and later. Earlier CXL |
| 67 | revisions defined a smaller set of features, leaving much of the control |
| 68 | interface as implementation defined or device specific, making generic |
| 69 | emulation challenging with host specific firmware being responsible |
| 70 | for setup and the Endpoints being presented to operating systems |
| 71 | as Root Complex Integrated End Points. CXL rev 2.0 looks a lot |
| 72 | more like PCI Express, with fully specified discoverability |
| 73 | of the CXL topology. |
| 74 | |
| 75 | CXL System components |
| 76 | ---------------------- |
| 77 | A CXL system is made up a Host with a number of 'standard components' |
| 78 | the control and capabilities of which are discoverable by system software |
| 79 | using means described in the CXL 2.0 specification. |
| 80 | |
| 81 | CXL Fixed Memory Windows (CFMW) |
| 82 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 83 | A CFMW consists of a particular range of Host Physical Address space |
| 84 | which is routed to particular CXL Host Bridges. At time of generic |
| 85 | software initialization it will have a particularly interleaving |
| 86 | configuration and associated Quality of Service Throttling Group (QTG). |
| 87 | This information is available to system software, when making |
| 88 | decisions about how to configure interleave across available CXL |
| 89 | memory devices. It is provide as CFMW Structures (CFMWS) in |
| 90 | the CXL Early Discovery Table, an ACPI table. |
| 91 | |
| 92 | Note: QTG 0 is the only one currently supported in QEMU. |
| 93 | |
| 94 | CXL Host Bridge (CXL HB) |
| 95 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
| 96 | A CXL host bridge is similar to the PCIe equivalent, but with a |
| 97 | specification defined register interface called CXL Host Bridge |
| 98 | Component Registers (CHBCR). The location of this CHBCR MMIO |
| 99 | space is described to system software via a CXL Host Bridge |
| 100 | Structure (CHBS) in the CEDT ACPI table. The actual interfaces |
| 101 | are identical to those used for other parts of the CXL hierarchy |
| 102 | as CXL Component Registers in PCI BARs. |
| 103 | |
| 104 | Interfaces provided include: |
| 105 | |
| 106 | * Configuration of HDM Decoders to route CXL Memory accesses with |
| 107 | a particularly Host Physical Address range to the target port |
| 108 | below which the CXL device servicing that address lies. This |
| 109 | may be a mapping to a single Root Port (RP) or across a set of |
| 110 | target RPs. |
| 111 | |
| 112 | CXL Root Ports (CXL RP) |
| 113 | ~~~~~~~~~~~~~~~~~~~~~~~ |
| 114 | A CXL Root Port serves the same purpose as a PCIe Root Port. |
| 115 | There are a number of CXL specific Designated Vendor Specific |
| 116 | Extended Capabilities (DVSEC) in PCIe Configuration Space |
| 117 | and associated component register access via PCI bars. |
| 118 | |
| 119 | CXL Switch |
| 120 | ~~~~~~~~~~ |
| 121 | Here we consider a simple CXL switch with only a single |
| 122 | virtual hierarchy. Whilst more complex devices exist, their |
| 123 | visibility to a particular host is generally the same as for |
| 124 | a simple switch design. Hosts often have no awareness |
| 125 | of complex rerouting and device pooling, they simply see |
| 126 | devices being hot added or hot removed. |
| 127 | |
| 128 | A CXL switch has a similar architecture to those in PCIe, |
| 129 | with a single upstream port, internal PCI bus and multiple |
| 130 | downstream ports. |
| 131 | |
| 132 | Both the CXL upstream and downstream ports have CXL specific |
| 133 | DVSECs in configuration space, and component registers in PCI |
| 134 | BARs. The Upstream Port has the configuration interfaces for |
| 135 | the HDM decoders which route incoming memory accesses to the |
| 136 | appropriate downstream port. |
| 137 | |
| 138 | A CXL switch is created in a similar fashion to PCI switches |
| 139 | by creating an upstream port (cxl-upstream) and a number of |
| 140 | downstream ports on the internal switch bus (cxl-downstream). |
| 141 | |
| 142 | CXL Memory Devices - Type 3 |
| 143 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 144 | CXL type 3 devices use a PCI class code and are intended to be supported |
| 145 | by a generic operating system driver. They have HDM decoders |
| 146 | though in these EP devices, the decoder is responsible not for |
| 147 | routing but for translation of the incoming host physical address (HPA) |
| 148 | into a Device Physical Address (DPA). |
| 149 | |
| 150 | CXL Memory Interleave |
| 151 | --------------------- |
| 152 | To understand the interaction of different CXL hardware components which |
| 153 | are emulated in QEMU, let us consider a memory read in a fully configured |
| 154 | CXL topology. Note that system software is responsible for configuration |
| 155 | of all components with the exception of the CFMWs. System software is |
| 156 | responsible for allocating appropriate ranges from within the CFMWs |
| 157 | and exposing those via normal memory configurations as would be done |
| 158 | for system RAM. |
| 159 | |
| 160 | Example system topology. x marks the match in each decoder level:: |
| 161 | |
| 162 | |<------------------SYSTEM PHYSICAL ADDRESS MAP (1)----------------->| |
| 163 | | __________ __________________________________ __________ | |
| 164 | | | | | | | | | |
| 165 | | | CFMW 0 | | CXL Fixed Memory Window 1 | | CFMW 2 | | |
| 166 | | | HB0 only | | Configured to interleave memory | | HB1 only | | |
| 167 | | | | | memory accesses across HB0/HB1 | | | | |
| 168 | | |__________| |_____x____________________________| |__________| | |
| 169 | | | | | |
| 170 | | | | | |
| 171 | | | | | |
| 172 | | Interleave Decoder | | |
| 173 | | Matches this HB | | |
| 174 | \_____________| |_____________/ |
| 175 | __________|__________ _____|_______________ |
| 176 | | | | | |
| 177 | (2) | CXL HB 0 | | CXL HB 1 | |
| 178 | | HB IntLv Decoders | | HB IntLv Decoders | |
| 179 | | PCI/CXL Root Bus 0c | | PCI/CXL Root Bus 0d | |
| 180 | | | | | |
| 181 | |___x_________________| |_____________________| |
| 182 | | | | | |
| 183 | | | | | |
| 184 | A HB 0 HDM Decoder | | | |
| 185 | matches this Port | | | |
| 186 | | | | | |
| 187 | ___________|___ __________|__ __|_________ ___|_________ |
| 188 | (3)| Root Port 0 | | Root Port 1 | | Root Port 2| | Root Port 3 | |
| 189 | | Appears in | | Appears in | | Appears in | | Appear in | |
| 190 | | PCI topology | | PCI topology| | PCI topo | | PCI topo | |
| 191 | | as 0c:00.0 | | as 0c:01.0 | | as de:00.0 | | as de:01.0 | |
| 192 | |_______________| |_____________| |____________| |_____________| |
| 193 | | | | | |
| 194 | | | | | |
| 195 | _____|_________ ______|______ ______|_____ ______|_______ |
| 196 | (4)| x | | | | | | | |
| 197 | | CXL Type3 0 | | CXL Type3 1 | | CXL type3 2| | CLX Type 3 3 | |
| 198 | | | | | | | | | |
| 199 | | PMEM0(Vol LSA)| | PMEM1 (...) | | PMEM2 (...)| | PMEM3 (...) | |
| 200 | | Decoder to go | | | | | | | |
| 201 | | from host PA | | PCI 0e:00.0 | | PCI df:00.0| | PCI e0:00.0 | |
| 202 | | to device PA | | | | | | | |
| 203 | | PCI as 0d:00.0| | | | | | | |
| 204 | |_______________| |_____________| |____________| |______________| |
| 205 | |
| 206 | Notes: |
| 207 | |
| 208 | (1) **3 CXL Fixed Memory Windows (CFMW)** corresponding to different |
| 209 | ranges of the system physical address map. Each CFMW has |
| 210 | particular interleave setup across the CXL Host Bridges (HB) |
| 211 | CFMW0 provides uninterleaved access to HB0, CFMW2 provides |
| 212 | uninterleaved access to HB1. CFMW1 provides interleaved memory access |
| 213 | across HB0 and HB1. |
| 214 | |
| 215 | (2) **Two CXL Host Bridges**. Each of these has 2 CXL Root Ports and |
| 216 | programmable HDM decoders to route memory accesses either to |
| 217 | a single port or interleave them across multiple ports. |
| 218 | A complex configuration here, might be to use the following HDM |
| 219 | decoders in HB0. HDM0 routes CFMW0 requests to RP0 and hence |
| 220 | part of CXL Type3 0. HDM1 routes CFMW0 requests from a |
| 221 | different region of the CFMW0 PA range to RP1 and hence part |
| 222 | of CXL Type 3 1. HDM2 routes yet another PA range from within |
| 223 | CFMW0 to be interleaved across RP0 and RP1, providing 2 way |
| 224 | interleave of part of the memory provided by CXL Type3 0 and |
| 225 | CXL Type 3 1. HDM3 routes those interleaved accesses from |
| 226 | CFMW1 that target HB0 to RP 0 and another part of the memory of |
| 227 | CXL Type 3 0 (as part of a 2 way interleave at the system level |
| 228 | across for example CXL Type3 0 and CXL Type3 2). |
| 229 | HDM4 is used to enable system wide 4 way interleave across all |
| 230 | the present CXL type3 devices, by interleaving those (interleaved) |
| 231 | requests that HB0 receives from CFMW1 across RP 0 and |
| 232 | RP 1 and hence to yet more regions of the memory of the |
| 233 | attached Type3 devices. Note this is a representative subset |
| 234 | of the full range of possible HDM decoder configurations in this |
| 235 | topology. |
| 236 | |
| 237 | (3) **Four CXL Root Ports.** In this case the CXL Type 3 devices are |
| 238 | directly attached to these ports. |
| 239 | |
| 240 | (4) **Four CXL Type3 memory expansion devices.** These will each have |
| 241 | HDM decoders, but in this case rather than performing interleave |
| 242 | they will take the Host Physical Addresses of accesses and map |
| 243 | them to their own local Device Physical Address Space (DPA). |
| 244 | |
| 245 | Example topology involving a switch:: |
| 246 | |
| 247 | |<------------------SYSTEM PHYSICAL ADDRESS MAP (1)----------------->| |
| 248 | | __________ __________________________________ __________ | |
| 249 | | | | | | | | | |
| 250 | | | CFMW 0 | | CXL Fixed Memory Window 1 | | CFMW 2 | | |
| 251 | | | HB0 only | | Configured to interleave memory | | HB1 only | | |
| 252 | | | | | memory accesses across HB0/HB1 | | | | |
| 253 | | |____x_____| |__________________________________| |__________| | |
| 254 | | | | | |
| 255 | | | | | |
| 256 | | | | |
| 257 | Interleave Decoder | | | |
| 258 | Matches this HB | | | |
| 259 | \_____________| |_____________/ |
| 260 | __________|__________ _____|_______________ |
| 261 | | | | | |
| 262 | | CXL HB 0 | | CXL HB 1 | |
| 263 | | HB IntLv Decoders | | HB IntLv Decoders | |
| 264 | | PCI/CXL Root Bus 0c | | PCI/CXL Root Bus 0d | |
| 265 | | | | | |
| 266 | |___x_________________| |_____________________| |
| 267 | | | | | |
| 268 | | |
| 269 | A HB 0 HDM Decoder |
| 270 | matches this Port |
| 271 | ___________|___ |
| 272 | | Root Port 0 | |
| 273 | | Appears in | |
| 274 | | PCI topology | |
| 275 | | as 0c:00.0 | |
| 276 | |___________x___| |
| 277 | | |
| 278 | | |
| 279 | \_____________________ |
| 280 | | |
| 281 | | |
| 282 | --------------------------------------------------- |
| 283 | | Switch 0 USP as PCI 0d:00.0 | |
| 284 | | USP has HDM decoder which direct traffic to | |
| 285 | | appropriate downstream port | |
| 286 | | Switch BUS appears as 0e | |
| 287 | |x__________________________________________________| |
| 288 | | | | | |
| 289 | | | | | |
| 290 | _____|_________ ______|______ ______|_____ ______|_______ |
| 291 | (4)| x | | | | | | | |
| 292 | | CXL Type3 0 | | CXL Type3 1 | | CXL type3 2| | CLX Type 3 3 | |
| 293 | | | | | | | | | |
| 294 | | PMEM0(Vol LSA)| | PMEM1 (...) | | PMEM2 (...)| | PMEM3 (...) | |
| 295 | | Decoder to go | | | | | | | |
| 296 | | from host PA | | PCI 10:00.0 | | PCI 11:00.0| | PCI 12:00.0 | |
| 297 | | to device PA | | | | | | | |
| 298 | | PCI as 0f:00.0| | | | | | | |
| 299 | |_______________| |_____________| |____________| |______________| |
| 300 | |
| 301 | Example command lines |
| 302 | --------------------- |
| 303 | A very simple setup with just one directly attached CXL Type 3 Persistent Memory device:: |
| 304 | |
| 305 | qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \ |
| 306 | ... |
| 307 | -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \ |
| 308 | -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \ |
| 309 | -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ |
| 310 | -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ |
| 311 | -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,sn=0x1 \ |
| 312 | -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G |
| 313 | |
| 314 | A very simple setup with just one directly attached CXL Type 3 Volatile Memory device:: |
| 315 | |
| 316 | qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \ |
| 317 | ... |
| 318 | -object memory-backend-ram,id=vmem0,share=on,size=256M \ |
| 319 | -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ |
| 320 | -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ |
| 321 | -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \ |
| 322 | -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G |
| 323 | |
| 324 | The same volatile setup may optionally include an LSA region:: |
| 325 | |
| 326 | qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \ |
| 327 | ... |
| 328 | -object memory-backend-ram,id=vmem0,share=on,size=256M \ |
| 329 | -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \ |
| 330 | -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ |
| 331 | -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ |
| 332 | -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,lsa=cxl-lsa0,id=cxl-vmem0 \ |
| 333 | -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G |
| 334 | |
| 335 | A setup suitable for 4 way interleave. Only one fixed window provided, to enable 2 way |
| 336 | interleave across 2 CXL host bridges. Each host bridge has 2 CXL Root Ports, with |
| 337 | the CXL Type3 device directly attached (no switches).:: |
| 338 | |
| 339 | qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \ |
| 340 | ... |
| 341 | -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \ |
| 342 | -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M \ |
| 343 | -object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M \ |
| 344 | -object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M \ |
| 345 | -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \ |
| 346 | -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M \ |
| 347 | -object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M \ |
| 348 | -object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M \ |
| 349 | -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ |
| 350 | -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \ |
| 351 | -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ |
| 352 | -device cxl-type3,bus=root_port13,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,sn=0x1 \ |
| 353 | -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \ |
| 354 | -device cxl-type3,bus=root_port14,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,sn=0x2 \ |
| 355 | -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \ |
| 356 | -device cxl-type3,bus=root_port15,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,sn=0x3 \ |
| 357 | -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \ |
| 358 | -device cxl-type3,bus=root_port16,persistent-memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,sn=0x4 \ |
| 359 | -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k |
| 360 | |
| 361 | An example of 4 devices below a switch suitable for 1, 2 or 4 way interleave:: |
| 362 | |
| 363 | qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \ |
| 364 | ... |
| 365 | -object memory-backend-file,id=cxl-mem0,share=on,mem-path=/tmp/cxltest.raw,size=256M \ |
| 366 | -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest1.raw,size=256M \ |
| 367 | -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M \ |
| 368 | -object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M \ |
| 369 | -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa0.raw,size=256M \ |
| 370 | -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa1.raw,size=256M \ |
| 371 | -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M \ |
| 372 | -object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M \ |
| 373 | -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ |
| 374 | -device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \ |
| 375 | -device cxl-rp,port=1,bus=cxl.1,id=root_port1,chassis=0,slot=1 \ |
| 376 | -device cxl-upstream,bus=root_port0,id=us0 \ |
| 377 | -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ |
| 378 | -device cxl-type3,bus=swport0,persistent-memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0,sn=0x1 \ |
| 379 | -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ |
| 380 | -device cxl-type3,bus=swport1,persistent-memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1,sn=0x2 \ |
| 381 | -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ |
| 382 | -device cxl-type3,bus=swport2,persistent-memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2,sn=0x3 \ |
| 383 | -device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \ |
| 384 | -device cxl-type3,bus=swport3,persistent-memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3,sn=0x4 \ |
| 385 | -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k |
| 386 | |
| 387 | An example of 4 type3 devices with volatile memory below a switch. Two of the devices |
| 388 | use HDM-DB for coherence, which requires operating in Flit mode:: |
| 389 | |
| 390 | qemu-system-x86_64 -M q35,cxl=on -m 4G,maxmem=8G,slots=8 -smp 4 \ |
| 391 | ... |
| 392 | -object memory-backend-ram,id=cxl-mem0,share=on,size=256M \ |
| 393 | -object memory-backend-ram,id=cxl-mem1,share=on,size=256M \ |
| 394 | -object memory-backend-ram,id=cxl-mem2,share=on,size=256M \ |
| 395 | -object memory-backend-ram,id=cxl-mem3,share=on,size=256M \ |
| 396 | -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ |
| 397 | -device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \ |
| 398 | -device cxl-rp,port=1,bus=cxl.1,id=root_port1,chassis=0,slot=1 \ |
| 399 | -device cxl-upstream,bus=root_port0,id=us0,x-256b-flit=on \ |
| 400 | -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ |
| 401 | -device cxl-type3,bus=swport0,volatile-memdev=cxl-mem0,id=cxl-mem0,sn=0x1,x-256b-flit=on,hdm-db=on \ |
| 402 | -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ |
| 403 | -device cxl-type3,bus=swport1,volatile-memdev=cxl-mem1,id=cxl-mem1,sn=0x2,x-256b-flit=on,hdm-db=on \ |
| 404 | -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ |
| 405 | -device cxl-type3,bus=swport2,volatile-memdev=cxl-mem2,id=cxl-mem2,sn=0x3 \ |
| 406 | -device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \ |
| 407 | -device cxl-type3,bus=swport3,volatile-memdev=cxl-mem3,id=cxl-mem3,sn=0x4 \ |
| 408 | -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k |
| 409 | |
| 410 | A simple arm/virt example featuring a single direct connected CXL Type 3 |
| 411 | Volatile Memory device:: |
| 412 | |
| 413 | qemu-system-aarch64 -M virt,gic-version=3,cxl=on -m 4g,maxmem=8g,slots=4 -cpu max -smp 4 \ |
| 414 | ... |
| 415 | -object memory-backend-ram,id=vmem0,share=on,size=256M \ |
| 416 | -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ |
| 417 | -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ |
| 418 | -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \ |
| 419 | -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G |
| 420 | |
| 421 | Deprecations |
| 422 | ------------ |
| 423 | |
| 424 | The Type 3 device [memdev] attribute has been deprecated in favor of the |
| 425 | [persistent-memdev] attributes. [memdev] will default to a persistent memory |
| 426 | device for backward compatibility and is incapable of being used in combination |
| 427 | with [persistent-memdev]. |
| 428 | |
| 429 | Kernel Configuration Options |
| 430 | ---------------------------- |
| 431 | |
| 432 | In Linux 5.18 the following options are necessary to make use of |
| 433 | OS management of CXL memory devices as described here. |
| 434 | |
| 435 | * CONFIG_CXL_BUS |
| 436 | * CONFIG_CXL_PCI |
| 437 | * CONFIG_CXL_ACPI |
| 438 | * CONFIG_CXL_PMEM |
| 439 | * CONFIG_CXL_MEM |
| 440 | * CONFIG_CXL_PORT |
| 441 | * CONFIG_CXL_REGION |
| 442 | |
| 443 | References |
| 444 | ---------- |
| 445 | |
| 446 | - Consortium website for specifications etc: |
| 447 | http://www.computeexpresslink.org |
| 448 | - Compute Express Link (CXL) Specification, Revision 3.1, August 2023 |