@samitouri / QOSamiQemu
Code Commits Issues Pulls Sessions CI/CD
master
master
rst 160 lines 4.94 KB
Raw
1 XBZRLE (Xor Based Zero Run Length Encoding)
2 ===========================================
3
4 Using XBZRLE (Xor Based Zero Run Length Encoding) allows for the reduction
5 of VM downtime and the total live-migration time of Virtual machines.
6 It is particularly useful for virtual machines running memory write intensive
7 workloads that are typical of large enterprise applications such as SAP ERP
8 Systems, and generally speaking for any application that uses a sparse memory
9 update pattern.
10
11 Instead of sending the changed guest memory page this solution will send a
12 compressed version of the updates, thus reducing the amount of data sent during
13 live migration.
14 In order to be able to calculate the update, the previous memory pages need to
15 be stored on the source. Those pages are stored in a dedicated cache
16 (hash table) and are accessed by their address.
17 The larger the cache size the better the chances are that the page has already
18 been stored in the cache.
19 A small cache size will result in high cache miss rate.
20 Cache size can be changed before and during migration.
21
22 Format
23 ------
24
25 The compression format performs a XOR between the previous and current content
26 of the page, where zero represents an unchanged value.
27 The page data delta is represented by zero and non zero runs.
28 A zero run is represented by its length (in bytes).
29 A non zero run is represented by its length (in bytes) and the new data.
30 The run length is encoded using ULEB128 (http://en.wikipedia.org/wiki/LEB128)
31
32 There can be more than one valid encoding, the sender may send a longer
33 encoding for the benefit of reducing computation cost.
34
35 ::
36
37 page = zrun nzrun
38 | zrun nzrun page
39
40 zrun = length
41
42 nzrun = length byte...
43
44 length = uleb128 encoded integer
45
46 On the sender side XBZRLE is used as a compact delta encoding of page updates,
47 retrieving the old page content from the cache (default size of 64MB). The
48 receiving side uses the existing page's content and XBZRLE to decode the new
49 page's content.
50
51 This work was originally based on research results published
52 VEE 2011: Evaluation of Delta Compression Techniques for Efficient Live
53 Migration of Large Virtual Machines by Benoit, Svard, Tordsson and Elmroth.
54 Additionally the delta encoder XBRLE was improved further using the XBZRLE
55 instead.
56
57 XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making it
58 ideal for in-line, real-time encoding such as is needed for live-migration.
59
60 Example:
61
62 old buffer:
63
64 .. code:: batch
65
66 1001 zeros
67 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 68 00 00 6b 00 6d
68 3074 zeros
69
70 new buffer:
71
72 .. code:: batch
73
74 1001 zeros
75 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 68 00 00 67 00 69
76 3074 zeros
77
78 encoded buffer:
79
80 .. code:: batch
81
82 encoded length 24
83 e9 07 0f 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 03 01 67 01 01 69
84
85 Cache update strategy
86 ---------------------
87
88 Keeping the hot pages in the cache is effective for decreasing cache
89 misses. XBZRLE uses a counter as the age of each page. The counter will
90 increase after each ram dirty bitmap sync. When a cache conflict is
91 detected, XBZRLE will only evict pages in the cache that are older than
92 a threshold.
93
94 Usage
95 -----
96
97 1. Verify the destination QEMU version is able to decode the new format::
98
99 (qemu) info migrate_capabilities
100 xbzrle: off
101 ...
102
103 2. Activate xbzrle on both source and destination::
104
105 (qemu) migrate_set_capability xbzrle on
106
107 3. Set the XBZRLE cache size - the cache size is in MBytes and should be a
108 power of 2. The cache default value is 64 MBytes (on source only)::
109
110 (qemu) migrate_set_parameter xbzrle-cache-size 256m
111
112 4. Start outgoing migration::
113
114 (qemu) migrate -d tcp:destination.host:4444
115 (qemu) info migrate
116 capabilities: xbzrle: on
117 Migration status: active
118 transferred ram: A kbytes
119 remaining ram: B kbytes
120 total ram: C kbytes
121 total time: D milliseconds
122 duplicate: E pages
123 normal: F pages
124 normal bytes: G kbytes
125 cache size: H bytes
126 xbzrle transferred: I kbytes
127 xbzrle pages: J pages
128 xbzrle cache miss: K pages
129 xbzrle cache miss rate: L
130 xbzrle encoding rate: M
131 xbzrle overflow: N
132
133 xbzrle cache miss: the number of cache misses to date - high cache-miss rate
134 indicates that the cache size is set too low.
135
136 xbzrle overflow: the number of overflows in the decoding which where the delta
137 could not be compressed. This can happen if the changes in the pages are too
138 large or there are many short changes; for example, changing every second byte
139 (half a page).
140
141 Testing: Testing indicated that live migration with XBZRLE was completed in 110
142 seconds, whereas without it would not be able to complete.
143
144 A simple synthetic memory r/w load generator:
145
146 .. code-block:: c
147
148 #include <stdlib.h>
149 #include <stdio.h>
150 int main()
151 {
152 char *buf = (char *) calloc(4096, 4096);
153 while (1) {
154 int i;
155 for (i = 0; i < 4096 * 4; i++) {
156 buf[i * 4096 / 4]++;
157 }
158 printf(".");
159 }
160 }