| 1 | # Benchmark KV Cache Offloading with Multi-Turn Conversations |
| 2 | |
| 3 | The requirements (pip) for `benchmark_serving_multi_turn.py` can be found in `requirements.txt` |
| 4 | |
| 5 | First start serving your model |
| 6 | |
| 7 | ```bash |
| 8 | export MODEL_PATH=/models/meta-llama/Meta-Llama-3.1-8B-Instruct/ |
| 9 | |
| 10 | vllm serve $MODEL_PATH --served-model-name Llama |
| 11 | ``` |
| 12 | |
| 13 | The variable `MODEL_PATH` should be a path to the model files (e.g. downloaded from huggingface). |
| 14 | |
| 15 | ## Synthetic Multi-Turn Conversations |
| 16 | |
| 17 | Download the following text file (used for generation of synthetic conversations) |
| 18 | |
| 19 | ```bash |
| 20 | wget https://www.gutenberg.org/ebooks/1184.txt.utf-8 |
| 21 | mv 1184.txt.utf-8 pg1184.txt |
| 22 | ``` |
| 23 | |
| 24 | The filename `pg1184.txt` is used in `generate_multi_turn.json` (see `"text_files"`). |
| 25 | |
| 26 | But you may use other text files if you prefer (using this specific file is not required). |
| 27 | |
| 28 | Then run the benchmarking script |
| 29 | |
| 30 | ```bash |
| 31 | export MODEL_PATH=/models/meta-llama/Meta-Llama-3.1-8B-Instruct/ |
| 32 | |
| 33 | python benchmark_serving_multi_turn.py --model $MODEL_PATH --served-model-name Llama \ |
| 34 | --input-file generate_multi_turn.json --num-clients 2 --max-active-conversations 6 |
| 35 | ``` |
| 36 | |
| 37 | You can edit the file `generate_multi_turn.json` to change the conversation parameters (number of turns, etc.). |
| 38 | |
| 39 | If successful, you will see the following output |
| 40 | |
| 41 | ```bash |
| 42 | ---------------------------------------------------------------------------------------------------- |
| 43 | Statistics summary: |
| 44 | runtime_sec = 215.810 |
| 45 | requests_per_sec = 0.769 |
| 46 | ---------------------------------------------------------------------------------------------------- |
| 47 | count mean std min 25% 50% 75% 90% 99% max |
| 48 | ttft_ms 166.0 78.22 67.63 45.91 59.94 62.26 64.43 69.66 353.18 567.54 |
| 49 | tpot_ms 166.0 25.37 0.57 24.40 25.07 25.31 25.50 25.84 27.50 28.05 |
| 50 | latency_ms 166.0 2591.07 326.90 1998.53 2341.62 2573.01 2860.10 3003.50 3268.46 3862.94 |
| 51 | input_num_turns 166.0 7.43 4.57 1.00 3.00 7.00 11.00 13.00 17.00 17.00 |
| 52 | input_num_tokens 166.0 2006.20 893.56 522.00 1247.75 2019.00 2718.00 3233.00 3736.45 3899.00 |
| 53 | output_num_tokens 166.0 100.01 11.80 80.00 91.00 99.00 109.75 116.00 120.00 120.00 |
| 54 | output_num_chunks 166.0 99.01 11.80 79.00 90.00 98.00 108.75 115.00 119.00 119.00 |
| 55 | ---------------------------------------------------------------------------------------------------- |
| 56 | ``` |
| 57 | |
| 58 | If you run with `--warmup-step`, the summary will also include `warmup_runtime_sec` |
| 59 | and `total_runtime_incl_warmup_sec` (while `runtime_sec` continues to reflect the |
| 60 | benchmark-only runtime so the reported throughput stays comparable). |
| 61 | |
| 62 | ### JSON configuration file for synthetic conversations generation |
| 63 | |
| 64 | The input flag `--input-file` is used to determine the input conversations for the benchmark.<br/> |
| 65 | When the input is a JSON file with the field `"filetype": "generate_conversations"` the tool will generate synthetic multi-turn (questions and answers) conversations. |
| 66 | |
| 67 | The file `generate_multi_turn.json` is an example file. |
| 68 | |
| 69 | The file must contain the sections `prompt_input` and `prompt_output`. |
| 70 | |
| 71 | The `prompt_input` section must contain `num_turns`, `prefix_num_tokens` and `num_tokens`: |
| 72 | |
| 73 | * `num_turns` - Number of total turns in the conversation (both user & assistant).<br/> |
| 74 | The final value will always be rounded to an even number so each user turn has a reply. |
| 75 | * `prefix_num_tokens` - Tokens added at the start of only the **first user turn** in a conversation (unique per conversation). |
| 76 | * `num_tokens` - Total token length of each **user** message (one turn). |
| 77 | |
| 78 | The `prompt_output` section must contain `num_tokens`: |
| 79 | |
| 80 | * `num_tokens` - Total token length of each **assistant** message (one turn). |
| 81 | |
| 82 | ### Random distributions for synthetic conversations generation |
| 83 | |
| 84 | When creating an input JSON file (such as `generate_multi_turn.json`),<br/> |
| 85 | every numeric field (such as `num_turns` or `num_tokens`) requires a distribution.<br/> |
| 86 | The distribution determines how to randomly sample values for the field. |
| 87 | |
| 88 | The available distributions are listed below. |
| 89 | |
| 90 | **Note:** The optional `max` field (for lognormal, zipf, and poisson) can be used to cap sampled values at an upper bound.</br> |
| 91 | Can be used to make sure that the total number of tokens in every request does not exceed `--max-model-len`. |
| 92 | |
| 93 | #### constant |
| 94 | |
| 95 | ```json |
| 96 | { |
| 97 | "distribution": "constant", |
| 98 | "value": 500 |
| 99 | } |
| 100 | ``` |
| 101 | |
| 102 | * `value` - the fixed integer value (always returns the same number). |
| 103 | |
| 104 | #### uniform |
| 105 | |
| 106 | ```json |
| 107 | { |
| 108 | "distribution": "uniform", |
| 109 | "min": 12, |
| 110 | "max": 18 |
| 111 | } |
| 112 | ``` |
| 113 | |
| 114 | * `min` - minimum value (inclusive). |
| 115 | * `max` - maximum value (inclusive), should be equal or larger than min. |
| 116 | |
| 117 | #### lognormal |
| 118 | |
| 119 | ```json |
| 120 | { |
| 121 | "distribution": "lognormal", |
| 122 | "average": 1000, |
| 123 | "max": 5000 |
| 124 | } |
| 125 | ``` |
| 126 | |
| 127 | You can parameterize the lognormal distribution in one of two ways: |
| 128 | |
| 129 | Using the average and optional median ratio: |
| 130 | |
| 131 | * `average` - target average value of the distribution. |
| 132 | * `median_ratio` - the ratio of the median to the average; controls the skewness. Must be in the range (0, 1). |
| 133 | |
| 134 | Using the parameters of the underlying normal distribution: |
| 135 | |
| 136 | * `mean` - mean of the underlying normal distribution. |
| 137 | * `sigma` - standard deviation of the underlying normal distribution. |
| 138 | |
| 139 | #### zipf |
| 140 | |
| 141 | ```json |
| 142 | { |
| 143 | "distribution": "zipf", |
| 144 | "alpha": 1.2, |
| 145 | "max": 100 |
| 146 | } |
| 147 | ``` |
| 148 | |
| 149 | * `alpha` - skew parameter (> 1). Larger values produce stronger skew toward smaller integers. |
| 150 | |
| 151 | #### poisson |
| 152 | |
| 153 | ```json |
| 154 | { |
| 155 | "distribution": "poisson", |
| 156 | "alpha": 10, |
| 157 | "max": 50 |
| 158 | } |
| 159 | ``` |
| 160 | |
| 161 | * `alpha` - expected value (λ). Also the variance of the distribution. |
| 162 | |
| 163 | ## ShareGPT Conversations |
| 164 | |
| 165 | To run with the ShareGPT data, download the following ShareGPT dataset: |
| 166 | `https://huggingface.co/datasets/philschmid/sharegpt-raw/blob/main/sharegpt_20230401_clean_lang_split.json` |
| 167 | |
| 168 | Use the `convert_sharegpt_to_openai.py` script to convert the dataset to a format supported by `benchmark_serving_multi_turn.py` |
| 169 | |
| 170 | ```bash |
| 171 | python convert_sharegpt_to_openai.py sharegpt_20230401_clean_lang_split.json sharegpt_conv_128.json --seed=99 --max-items=128 |
| 172 | ``` |
| 173 | |
| 174 | The script will convert the ShareGPT dataset to a dataset with the standard user/assistant roles. |
| 175 | |
| 176 | The flag `--max-items=128` is used to sample 128 conversations from the original dataset (change as needed). |
| 177 | |
| 178 | Use the output JSON file `sharegpt_conv_128.json` as the `--input-file` for `benchmark_serving_multi_turn.py`. |