| 1 | Software Guard eXtensions (SGX) |
| 2 | =============================== |
| 3 | |
| 4 | Overview |
| 5 | -------- |
| 6 | |
| 7 | Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms |
| 8 | for memory accesses in order to provide security accesses for sensitive |
| 9 | applications and data. SGX allows an application to use its particular |
| 10 | address space as an *enclave*, which is a protected area provides confidentiality |
| 11 | and integrity even in the presence of privileged malware. Accesses to the |
| 12 | enclave memory area from any software not resident in the enclave are prevented, |
| 13 | including those from privileged software. |
| 14 | |
| 15 | Virtual SGX |
| 16 | ----------- |
| 17 | |
| 18 | SGX feature is exposed to guest via SGX CPUID. Looking at SGX CPUID, we can |
| 19 | report the same CPUID info to guest as on host for most of SGX CPUID. With |
| 20 | reporting the same CPUID guest is able to use full capacity of SGX, and KVM |
| 21 | doesn't need to emulate those info. |
| 22 | |
| 23 | The guest's EPC base and size are determined by QEMU, and KVM needs QEMU to |
| 24 | notify such info to it before it can initialize SGX for guest. |
| 25 | |
| 26 | Virtual EPC |
| 27 | ~~~~~~~~~~~ |
| 28 | |
| 29 | By default, QEMU does not assign EPC to a VM, i.e. fully enabling SGX in a VM |
| 30 | requires explicit allocation of EPC to the VM. Similar to other specialized |
| 31 | memory types, e.g. hugetlbfs, EPC is exposed as a memory backend. |
| 32 | |
| 33 | SGX EPC is enumerated through CPUID, i.e. EPC "devices" need to be realized |
| 34 | prior to realizing the vCPUs themselves, which occurs long before generic |
| 35 | devices are parsed and realized. This limitation means that EPC does not |
| 36 | require -maxmem as EPC is not treated as {cold,hot}plugged memory. |
| 37 | |
| 38 | QEMU does not artificially restrict the number of EPC sections exposed to a |
| 39 | guest, e.g. QEMU will happily allow you to create 64 1M EPC sections. Be aware |
| 40 | that some kernels may not recognize all EPC sections, e.g. the Linux SGX driver |
| 41 | is hardwired to support only 8 EPC sections. |
| 42 | |
| 43 | The following QEMU snippet creates two EPC sections, with 64M pre-allocated |
| 44 | to the VM and an additional 28M mapped but not allocated:: |
| 45 | |
| 46 | -object memory-backend-epc,id=mem1,size=64M,prealloc=on \ |
| 47 | -object memory-backend-epc,id=mem2,size=28M \ |
| 48 | -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2 |
| 49 | |
| 50 | Note: |
| 51 | |
| 52 | The size and location of the virtual EPC are far less restricted compared |
| 53 | to physical EPC. Because physical EPC is protected via range registers, |
| 54 | the size of the physical EPC must be a power of two (though software sees |
| 55 | a subset of the full EPC, e.g. 92M or 128M) and the EPC must be naturally |
| 56 | aligned. KVM SGX's virtual EPC is purely a software construct and only |
| 57 | requires the size and location to be page aligned. QEMU enforces the EPC |
| 58 | size is a multiple of 4k and will ensure the base of the EPC is 4k aligned. |
| 59 | To simplify the implementation, EPC is always located above 4g in the guest |
| 60 | physical address space. |
| 61 | |
| 62 | Migration |
| 63 | ~~~~~~~~~ |
| 64 | |
| 65 | QEMU/KVM doesn't prevent live migrating SGX VMs, although from hardware's |
| 66 | perspective, SGX doesn't support live migration, since both EPC and the SGX |
| 67 | key hierarchy are bound to the physical platform. However live migration |
| 68 | can be supported in the sense if guest software stack can support recreating |
| 69 | enclaves when it suffers sudden lose of EPC; and if guest enclaves can detect |
| 70 | SGX keys being changed, and handle gracefully. For instance, when ERESUME fails |
| 71 | with #PF.SGX, guest software can gracefully detect it and recreate enclaves; |
| 72 | and when enclave fails to unseal sensitive information from outside, it can |
| 73 | detect such error and sensitive information can be provisioned to it again. |
| 74 | |
| 75 | CPUID |
| 76 | ~~~~~ |
| 77 | |
| 78 | Due to its myriad dependencies, SGX is currently not listed as supported |
| 79 | in any of QEMU's built-in CPU configuration. To expose SGX (and SGX Launch |
| 80 | Control) to a guest, you must either use ``-cpu host`` to pass-through the |
| 81 | host CPU model, or explicitly enable SGX when using a built-in CPU model, |
| 82 | e.g. via ``-cpu <model>,+sgx`` or ``-cpu <model>,+sgx,+sgxlc``. |
| 83 | |
| 84 | All SGX sub-features enumerated through CPUID, e.g. SGX2, MISCSELECT, |
| 85 | ATTRIBUTES, etc... can be restricted via CPUID flags. Be aware that enforcing |
| 86 | restriction of MISCSELECT, ATTRIBUTES and XFRM requires intercepting ECREATE, |
| 87 | i.e. may marginally reduce SGX performance in the guest. All SGX sub-features |
| 88 | controlled via -cpu are prefixed with "sgx", e.g.:: |
| 89 | |
| 90 | $ qemu-system-x86_64 -cpu help | xargs printf "%s\n" | grep sgx |
| 91 | sgx |
| 92 | sgx-debug |
| 93 | sgx-encls-c |
| 94 | sgx-enclv |
| 95 | sgx-exinfo |
| 96 | sgx-kss |
| 97 | sgx-mode64 |
| 98 | sgx-provisionkey |
| 99 | sgx-tokenkey |
| 100 | sgx1 |
| 101 | sgx2 |
| 102 | sgxlc |
| 103 | |
| 104 | The following QEMU snippet passes through the host CPU but restricts access to |
| 105 | the provision and EINIT token keys:: |
| 106 | |
| 107 | -cpu host,-sgx-provisionkey,-sgx-tokenkey |
| 108 | |
| 109 | SGX sub-features cannot be emulated, i.e. sub-features that are not present |
| 110 | in hardware cannot be forced on via '-cpu'. |
| 111 | |
| 112 | Virtualize SGX Launch Control |
| 113 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 114 | |
| 115 | QEMU SGX support for Launch Control (LC) is passive, in the sense that it |
| 116 | does not actively change the LC configuration. QEMU SGX provides the user |
| 117 | the ability to set/clear the CPUID flag (and by extension the associated |
| 118 | IA32_FEATURE_CONTROL MSR bit in fw_cfg) and saves/restores the LE Hash MSRs |
| 119 | when getting/putting guest state, but QEMU does not add new controls to |
| 120 | directly modify the LC configuration. Similar to hardware behavior, locking |
| 121 | the LC configuration to a non-Intel value is left to guest firmware. Unlike |
| 122 | host bios setting for SGX launch control(LC), there is no special bios setting |
| 123 | for SGX guest by our design. If host is in locked mode, we can still allow |
| 124 | creating VM with SGX. |
| 125 | |
| 126 | Feature Control |
| 127 | ~~~~~~~~~~~~~~~ |
| 128 | |
| 129 | QEMU SGX updates the ``etc/msr_feature_control`` fw_cfg entry to set the SGX |
| 130 | (bit 18) and SGX LC (bit 17) flags based on their respective CPUID support, |
| 131 | i.e. existing guest firmware will automatically set SGX and SGX LC accordingly, |
| 132 | assuming said firmware supports fw_cfg.msr_feature_control. |
| 133 | |
| 134 | Launching a guest |
| 135 | ----------------- |
| 136 | |
| 137 | To launch a SGX guest: |
| 138 | |
| 139 | .. parsed-literal:: |
| 140 | |
| 141 | |qemu_system_x86| \\ |
| 142 | -cpu host,+sgx-provisionkey \\ |
| 143 | -object memory-backend-epc,id=mem1,size=64M,prealloc=on \\ |
| 144 | -M sgx-epc.0.memdev=mem1,sgx-epc.0.node=0 |
| 145 | |
| 146 | Utilizing SGX in the guest requires a kernel/OS with SGX support. |
| 147 | The support can be determined in guest by:: |
| 148 | |
| 149 | $ grep sgx /proc/cpuinfo |
| 150 | |
| 151 | and SGX epc info by:: |
| 152 | |
| 153 | $ dmesg | grep sgx |
| 154 | [ 0.182807] sgx: EPC section 0x140000000-0x143ffffff |
| 155 | [ 0.183695] sgx: [Firmware Bug]: Unable to map EPC section to online node. Fallback to the NUMA node 0. |
| 156 | |
| 157 | To launch a SGX numa guest: |
| 158 | |
| 159 | .. parsed-literal:: |
| 160 | |
| 161 | |qemu_system_x86| \\ |
| 162 | -cpu host,+sgx-provisionkey \\ |
| 163 | -object memory-backend-ram,size=2G,host-nodes=0,policy=bind,id=node0 \\ |
| 164 | -object memory-backend-epc,id=mem0,size=64M,prealloc=on,host-nodes=0,policy=bind \\ |
| 165 | -numa node,nodeid=0,cpus=0-1,memdev=node0 \\ |
| 166 | -object memory-backend-ram,size=2G,host-nodes=1,policy=bind,id=node1 \\ |
| 167 | -object memory-backend-epc,id=mem1,size=28M,prealloc=on,host-nodes=1,policy=bind \\ |
| 168 | -numa node,nodeid=1,cpus=2-3,memdev=node1 \\ |
| 169 | -M sgx-epc.0.memdev=mem0,sgx-epc.0.node=0,sgx-epc.1.memdev=mem1,sgx-epc.1.node=1 |
| 170 | |
| 171 | and SGX epc numa info by:: |
| 172 | |
| 173 | $ dmesg | grep sgx |
| 174 | [ 0.369937] sgx: EPC section 0x180000000-0x183ffffff |
| 175 | [ 0.370259] sgx: EPC section 0x184000000-0x185bfffff |
| 176 | |
| 177 | $ dmesg | grep SRAT |
| 178 | [ 0.009981] ACPI: SRAT: Node 0 PXM 0 [mem 0x180000000-0x183ffffff] |
| 179 | [ 0.009982] ACPI: SRAT: Node 1 PXM 1 [mem 0x184000000-0x185bfffff] |
| 180 | |
| 181 | References |
| 182 | ---------- |
| 183 | |
| 184 | - `SGX Homepage <https://software.intel.com/sgx>`__ |
| 185 | |
| 186 | - `SGX SDK <https://github.com/intel/linux-sgx.git>`__ |
| 187 | |
| 188 | - SGX specification: Intel SDM Volume 3 |