| 1 | Dirty limit |
| 2 | =========== |
| 3 | |
| 4 | The dirty limit, short for dirty page rate upper limit, is a new capability |
| 5 | introduced in the 8.1 QEMU release that uses a new algorithm based on the KVM |
| 6 | dirty ring to throttle down the guest during live migration. |
| 7 | |
| 8 | The algorithm framework is as follows: |
| 9 | |
| 10 | :: |
| 11 | |
| 12 | ------------------------------------------------------------------------------ |
| 13 | main --------------> throttle thread ------------> PREPARE(1) <-------- |
| 14 | thread \ | | |
| 15 | \ | | |
| 16 | \ V | |
| 17 | -\ CALCULATE(2) | |
| 18 | \ | | |
| 19 | \ | | |
| 20 | \ V | |
| 21 | \ SET PENALTY(3) ----- |
| 22 | -\ | |
| 23 | \ | |
| 24 | \ V |
| 25 | -> virtual CPU thread -------> ACCEPT PENALTY(4) |
| 26 | ------------------------------------------------------------------------------ |
| 27 | |
| 28 | When the qmp command qmp_set_vcpu_dirty_limit is called for the first time, |
| 29 | the QEMU main thread starts the throttle thread. The throttle thread, once |
| 30 | launched, executes the loop, which consists of three steps: |
| 31 | |
| 32 | - PREPARE (1) |
| 33 | |
| 34 | The entire work of PREPARE (1) is preparation for the second stage, |
| 35 | CALCULATE(2), as the name implies. It involves preparing the dirty |
| 36 | page rate value and the corresponding upper limit of the VM: |
| 37 | The dirty page rate is calculated via the KVM dirty ring mechanism, |
| 38 | which tells QEMU how many dirty pages a virtual CPU has had since the |
| 39 | last KVM_EXIT_DIRTY_RING_FULL exception; The dirty page rate upper |
| 40 | limit is specified by caller, therefore fetch it directly. |
| 41 | |
| 42 | - CALCULATE (2) |
| 43 | |
| 44 | Calculate a suitable sleep period for each virtual CPU, which will be |
| 45 | used to determine the penalty for the target virtual CPU. The |
| 46 | computation must be done carefully in order to reduce the dirty page |
| 47 | rate progressively down to the upper limit without oscillation. To |
| 48 | achieve this, two strategies are provided: the first is to add or |
| 49 | subtract sleep time based on the ratio of the current dirty page rate |
| 50 | to the limit, which is used when the current dirty page rate is far |
| 51 | from the limit; the second is to add or subtract a fixed time when |
| 52 | the current dirty page rate is close to the limit. |
| 53 | |
| 54 | - SET PENALTY (3) |
| 55 | |
| 56 | Set the sleep time for each virtual CPU that should be penalized based |
| 57 | on the results of the calculation supplied by step CALCULATE (2). |
| 58 | |
| 59 | After completing the three above stages, the throttle thread loops back |
| 60 | to step PREPARE (1) until the dirty limit is reached. |
| 61 | |
| 62 | On the other hand, each virtual CPU thread reads the sleep duration and |
| 63 | sleeps in the path of the KVM_EXIT_DIRTY_RING_FULL exception handler, that |
| 64 | is ACCEPT PENALTY (4). Virtual CPUs tied with writing processes will |
| 65 | obviously exit to the path and get penalized, whereas virtual CPUs involved |
| 66 | with read processes will not. |
| 67 | |
| 68 | In summary, thanks to the KVM dirty ring technology, the dirty limit |
| 69 | algorithm will restrict virtual CPUs as needed to keep their dirty page |
| 70 | rate inside the limit. This leads to more steady reading performance during |
| 71 | live migration and can aid in improving large guest responsiveness. |