@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 8bbcf8b6e7
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 12, 06:00 UTC

[vLLM Benchmark Suite] Add default parameters section and update CPU benchmark cases (#29381)

Signed-off-by: Tsai, Louie <louie.tsai@intel.com> Signed-off-by: Louie Tsai <louie.tsai@intel.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Li, Jiang <bigpyj64@gmail.com>

Louie Tsai committed Dec 2, 2025 at 01:00 UTC 8bbcf8b6e7ad0cdeaef010bd834bd723f4e00445
6 files changed +361 -1898
.buildkite/performance-benchmarks/README.md
+59
@@ -108,6 +108,65 @@ The number of this test is less stable compared to the delay and latency benchma
108
109 WARNING: The benchmarking script will save json results by itself, so please do not configure `--save-results` or other results-saving-related parameters in `serving-tests.json`.
110
111 +#### Default Parameters Field
112 +
113 +We can specify default parameters in a JSON field with key `defaults`. Parameters defined in the field are applied globally to all serving tests, and can be overridden in test case fields. Here is an example:
114 +
115 +<details>
116 +<summary> An Example of default parameters field </summary>
117 +
118 +```json
119 +{
120 + "defaults": {
121 + "qps_list": [
122 + "inf"
123 + ],
124 + "server_environment_variables": {
125 + "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1
126 + },
127 + "server_parameters": {
128 + "tensor_parallel_size": 1,
129 + "dtype": "bfloat16",
130 + "block_size": 128,
131 + "disable_log_stats": "",
132 + "load_format": "dummy"
133 + },
134 + "client_parameters": {
135 + "backend": "vllm",
136 + "dataset_name": "random",
137 + "random-input-len": 128,
138 + "random-output-len": 128,
139 + "num_prompts": 200,
140 + "ignore-eos": ""
141 + }
142 + },
143 + "tests": [
144 + {
145 + "test_name": "serving_llama3B_tp2_random_128_128",
146 + "server_parameters": {
147 + "model": "meta-llama/Llama-3.2-3B-Instruct",
148 + "tensor_parallel_size": 2,
149 + },
150 + "client_parameters": {
151 + "model": "meta-llama/Llama-3.2-3B-Instruct",
152 + }
153 + },
154 + {
155 + "test_name": "serving_qwen3_tp4_random_128_128",
156 + "server_parameters": {
157 + "model": "Qwen/Qwen3-14B",
158 + "tensor_parallel_size": 4,
159 + },
160 + "client_parameters": {
161 + "model": "Qwen/Qwen3-14B",
162 + }
163 + },
164 + ]
165 +}
166 +```
167 +
168 +</details>
169 +
170 ### Visualizing the results
171
172 The `convert-results-json-to-markdown.py` helps you put the benchmarking results inside a markdown table, by formatting [descriptions.md](performance-benchmarks-descriptions.md) with real benchmarking results.
.buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh
+43 -5
@@ -110,7 +110,8 @@ json2envs() {
110 wait_for_server() {
111 # wait for vllm server to start
112 # return 1 if vllm server crashes
113 - timeout 1200 bash -c '
113 + local timeout_val="1200"
114 + timeout "$timeout_val" bash -c '
115 until curl -X POST localhost:8000/v1/completions; do
116 sleep 1
117 done' && return 0 || return 1
@@ -316,12 +317,44 @@ run_throughput_tests() {
317 run_serving_tests() {
318 # run serving tests using `vllm bench serve` command
319 # $1: a json file specifying serving test cases
320 + #
321 + # Supported JSON formats:
322 + # 1) Plain format: top-level array
323 + # [ { "test_name": "...", "server_parameters": {...}, ... }, ... ]
324 + #
325 + # 2) Default parameters field + plain format tests
326 + # {
327 + # "defaults": { ... },
328 + # "tests": [ { "test_name": "...", "server_parameters": {...}, ... }, ... ]
329 + # }
330
331 local serving_test_file
332 serving_test_file=$1
333
334 # Iterate over serving tests
324 - jq -c '.[]' "$serving_test_file" | while read -r params; do
335 + jq -c '
336 + if type == "array" then
337 + # Plain format: test cases array
338 + .[]
339 + elif (type == "object" and has("tests")) then
340 + # merge the default parameters into each test cases
341 + . as $root
342 + | ($root.defaults // {}) as $d
343 + | ($root.tests // [])[]
344 + # default qps / max_concurrency from defaults if missing
345 + | .qps_list = (.qps_list // $d.qps_list)
346 + | .max_concurrency_list = (.max_concurrency_list // $d.max_concurrency_list)
347 + # merge envs / params: test overrides defaults
348 + | .server_environment_variables =
349 + (($d.server_environment_variables // {}) + (.server_environment_variables // {}))
350 + | .server_parameters =
351 + (($d.server_parameters // {}) + (.server_parameters // {}))
352 + | .client_parameters =
353 + (($d.client_parameters // {}) + (.client_parameters // {}))
354 + else
355 + error("Unsupported serving test file format: must be array or object with .tests")
356 + end
357 + ' "$serving_test_file" | while read -r params; do
358 # get the test name, and append the GPU type back to it.
359 test_name=$(echo "$params" | jq -r '.test_name')
360 if [[ ! "$test_name" =~ ^serving_ ]]; then
@@ -335,20 +368,25 @@ run_serving_tests() {
368 continue
369 fi
370
338 - # get client and server arguments
371 + # get client and server arguments (after merged the default parameters)
372 server_params=$(echo "$params" | jq -r '.server_parameters')
373 server_envs=$(echo "$params" | jq -r '.server_environment_variables')
374 client_params=$(echo "$params" | jq -r '.client_parameters')
375 +
376 server_args=$(json2args "$server_params")
377 server_envs=$(json2envs "$server_envs")
378 client_args=$(json2args "$client_params")
379 +
380 + # qps_list
381 qps_list=$(echo "$params" | jq -r '.qps_list')
382 qps_list=$(echo "$qps_list" | jq -r '.[] | @sh')
383 echo "Running over qps list $qps_list"
384 +
385 + # max_concurrency_list (fallback to num_prompts if missing)
386 max_concurrency_list=$(echo "$params" | jq -r '.max_concurrency_list')
387 if [[ -z "$max_concurrency_list" || "$max_concurrency_list" == "null" ]]; then
350 - num_prompts=$(echo "$client_params" | jq -r '.num_prompts')
351 - max_concurrency_list="[$num_prompts]"
388 + num_prompts=$(echo "$client_params" | jq -r '.num_prompts')
389 + max_concurrency_list="[$num_prompts]"
390 fi
391 max_concurrency_list=$(echo "$max_concurrency_list" | jq -r '.[] | @sh')
392 echo "Running over max concurrency list $max_concurrency_list"
.buildkite/performance-benchmarks/tests/serving-tests-cpu-snc2.json deleted
-610
@@ -1,610 +0,0 @@
1 -[
2 - {
3 - "test_name": "serving_llama8B_bf16_tp1_sharegpt",
4 - "qps_list": ["inf"],
5 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
6 - "server_environment_variables": {
7 - "VLLM_RPC_TIMEOUT": 100000,
8 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
9 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
10 - "VLLM_CPU_SGL_KERNEL": 1,
11 - "VLLM_CPU_KVCACHE_SPACE": 40
12 - },
13 - "server_parameters": {
14 - "model": "meta-llama/Llama-3.1-8B-Instruct",
15 - "tensor_parallel_size": 1,
16 - "dtype": "bfloat16",
17 - "distributed_executor_backend": "mp",
18 - "block_size": 128,
19 - "trust_remote_code": "",
20 - "disable_log_stats": "",
21 - "enforce_eager": "",
22 - "max_num_batched_tokens": 2048,
23 - "max_num_seqs": 256,
24 - "load_format": "dummy"
25 - },
26 - "client_parameters": {
27 - "model": "meta-llama/Llama-3.1-8B-Instruct",
28 - "backend": "vllm",
29 - "dataset_name": "sharegpt",
30 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
31 - "num_prompts": 200
32 - }
33 - },
34 - {
35 - "test_name": "serving_llama8B_bf16_tp2_sharegpt",
36 - "qps_list": ["inf"],
37 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
38 - "server_environment_variables": {
39 - "VLLM_RPC_TIMEOUT": 100000,
40 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
41 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
42 - "VLLM_CPU_SGL_KERNEL": 1,
43 - "VLLM_CPU_KVCACHE_SPACE": 40
44 - },
45 - "server_parameters": {
46 - "model": "meta-llama/Llama-3.1-8B-Instruct",
47 - "tensor_parallel_size": 2,
48 - "dtype": "bfloat16",
49 - "distributed_executor_backend": "mp",
50 - "block_size": 128,
51 - "trust_remote_code": "",
52 - "disable_log_stats": "",
53 - "enforce_eager": "",
54 - "max_num_batched_tokens": 2048,
55 - "max_num_seqs": 256,
56 - "load_format": "dummy"
57 - },
58 - "client_parameters": {
59 - "model": "meta-llama/Llama-3.1-8B-Instruct",
60 - "backend": "vllm",
61 - "dataset_name": "sharegpt",
62 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
63 - "num_prompts": 200
64 - }
65 - },
66 - {
67 - "test_name": "serving_llama8B_bf16_tp4_sharegpt",
68 - "qps_list": ["inf"],
69 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
70 - "server_environment_variables": {
71 - "VLLM_RPC_TIMEOUT": 100000,
72 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
73 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
74 - "VLLM_CPU_SGL_KERNEL": 1,
75 - "VLLM_CPU_KVCACHE_SPACE": 40
76 - },
77 - "server_parameters": {
78 - "model": "meta-llama/Llama-3.1-8B-Instruct",
79 - "tensor_parallel_size": 4,
80 - "dtype": "bfloat16",
81 - "distributed_executor_backend": "mp",
82 - "block_size": 128,
83 - "trust_remote_code": "",
84 - "disable_log_stats": "",
85 - "enforce_eager": "",
86 - "max_num_batched_tokens": 2048,
87 - "max_num_seqs": 256,
88 - "load_format": "dummy"
89 - },
90 - "client_parameters": {
91 - "model": "meta-llama/Llama-3.1-8B-Instruct",
92 - "backend": "vllm",
93 - "dataset_name": "sharegpt",
94 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
95 - "num_prompts": 200
96 - }
97 - },
98 - {
99 - "test_name": "serving_llama8B_bf16_tp1_random_128_128",
100 - "qps_list": ["inf"],
101 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
102 - "server_environment_variables": {
103 - "VLLM_RPC_TIMEOUT": 100000,
104 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
105 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
106 - "VLLM_CPU_SGL_KERNEL": 1,
107 - "VLLM_CPU_KVCACHE_SPACE": 40
108 - },
109 - "server_parameters": {
110 - "model": "meta-llama/Llama-3.1-8B-Instruct",
111 - "tensor_parallel_size": 1,
112 - "dtype": "bfloat16",
113 - "distributed_executor_backend": "mp",
114 - "block_size": 128,
115 - "trust_remote_code": "",
116 - "enable_chunked_prefill": "",
117 - "disable_log_stats": "",
118 - "enforce_eager": "",
119 - "max_num_batched_tokens": 2048,
120 - "max_num_seqs": 256,
121 - "load_format": "dummy"
122 - },
123 - "client_parameters": {
124 - "model": "meta-llama/Llama-3.1-8B-Instruct",
125 - "backend": "vllm",
126 - "dataset_name": "random",
127 - "random-input-len": 128,
128 - "random-output-len": 128,
129 - "ignore-eos": "",
130 - "num_prompts": 1000
131 - }
132 - },
133 - {
134 - "test_name": "serving_llama8B_bf16_tp2_random_128_128",
135 - "qps_list": ["inf"],
136 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
137 - "server_environment_variables": {
138 - "VLLM_RPC_TIMEOUT": 100000,
139 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
140 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
141 - "VLLM_CPU_SGL_KERNEL": 1,
142 - "VLLM_CPU_KVCACHE_SPACE": 40
143 - },
144 - "server_parameters": {
145 - "model": "meta-llama/Llama-3.1-8B-Instruct",
146 - "tensor_parallel_size": 2,
147 - "dtype": "bfloat16",
148 - "distributed_executor_backend": "mp",
149 - "block_size": 128,
150 - "trust_remote_code": "",
151 - "enable_chunked_prefill": "",
152 - "disable_log_stats": "",
153 - "enforce_eager": "",
154 - "max_num_batched_tokens": 2048,
155 - "max_num_seqs": 256,
156 - "load_format": "dummy"
157 - },
158 - "client_parameters": {
159 - "model": "meta-llama/Llama-3.1-8B-Instruct",
160 - "backend": "vllm",
161 - "dataset_name": "random",
162 - "random-input-len": 128,
163 - "random-output-len": 128,
164 - "ignore-eos": "",
165 - "num_prompts": 1000
166 - }
167 - },
168 - {
169 - "test_name": "serving_llama8B_bf16_tp4_random_128_128",
170 - "qps_list": ["inf"],
171 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
172 - "server_environment_variables": {
173 - "VLLM_RPC_TIMEOUT": 100000,
174 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
175 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
176 - "VLLM_CPU_SGL_KERNEL": 1,
177 - "VLLM_CPU_KVCACHE_SPACE": 40
178 - },
179 - "server_parameters": {
180 - "model": "meta-llama/Llama-3.1-8B-Instruct",
181 - "tensor_parallel_size": 4,
182 - "dtype": "bfloat16",
183 - "distributed_executor_backend": "mp",
184 - "block_size": 128,
185 - "trust_remote_code": "",
186 - "enable_chunked_prefill": "",
187 - "disable_log_stats": "",
188 - "enforce_eager": "",
189 - "max_num_batched_tokens": 2048,
190 - "max_num_seqs": 256,
191 - "load_format": "dummy"
192 - },
193 - "client_parameters": {
194 - "model": "meta-llama/Llama-3.1-8B-Instruct",
195 - "backend": "vllm",
196 - "dataset_name": "random",
197 - "random-input-len": 128,
198 - "random-output-len": 128,
199 - "num_prompts": 1000
200 - }
201 - },
202 - {
203 - "test_name": "serving_llama8B_int8_tp1_sharegpt",
204 - "qps_list": ["inf"],
205 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
206 - "server_environment_variables": {
207 - "VLLM_RPC_TIMEOUT": 100000,
208 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
209 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
210 - "VLLM_CPU_SGL_KERNEL": 1,
211 - "VLLM_CPU_KVCACHE_SPACE": 40
212 - },
213 - "server_parameters": {
214 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
215 - "tensor_parallel_size": 1,
216 - "dtype": "bfloat16",
217 - "distributed_executor_backend": "mp",
218 - "block_size": 128,
219 - "trust_remote_code": "",
220 - "disable_log_stats": "",
221 - "enforce_eager": "",
222 - "max_num_batched_tokens": 2048,
223 - "max_num_seqs": 256,
224 - "load_format": "dummy"
225 - },
226 - "client_parameters": {
227 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
228 - "backend": "vllm",
229 - "dataset_name": "sharegpt",
230 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
231 - "num_prompts": 200
232 - }
233 - },
234 - {
235 - "test_name": "serving_llama8B_int8_tp2_sharegpt",
236 - "qps_list": ["inf"],
237 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
238 - "server_environment_variables": {
239 - "VLLM_RPC_TIMEOUT": 100000,
240 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
241 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
242 - "VLLM_CPU_SGL_KERNEL": 1,
243 - "VLLM_CPU_KVCACHE_SPACE": 40
244 - },
245 - "server_parameters": {
246 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
247 - "tensor_parallel_size": 2,
248 - "dtype": "bfloat16",
249 - "distributed_executor_backend": "mp",
250 - "block_size": 128,
251 - "trust_remote_code": "",
252 - "disable_log_stats": "",
253 - "enforce_eager": "",
254 - "max_num_batched_tokens": 2048,
255 - "max_num_seqs": 256,
256 - "load_format": "dummy"
257 - },
258 - "client_parameters": {
259 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
260 - "backend": "vllm",
261 - "dataset_name": "sharegpt",
262 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
263 - "num_prompts": 200
264 - }
265 - },
266 - {
267 - "test_name": "serving_llama8B_int8_tp4_sharegpt",
268 - "qps_list": ["inf"],
269 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
270 - "server_environment_variables": {
271 - "VLLM_RPC_TIMEOUT": 100000,
272 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
273 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
274 - "VLLM_CPU_SGL_KERNEL": 1,
275 - "VLLM_CPU_KVCACHE_SPACE": 40
276 - },
277 - "server_parameters": {
278 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
279 - "tensor_parallel_size": 4,
280 - "dtype": "bfloat16",
281 - "distributed_executor_backend": "mp",
282 - "block_size": 128,
283 - "trust_remote_code": "",
284 - "disable_log_stats": "",
285 - "enforce_eager": "",
286 - "max_num_batched_tokens": 2048,
287 - "max_num_seqs": 256,
288 - "load_format": "dummy"
289 - },
290 - "client_parameters": {
291 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
292 - "backend": "vllm",
293 - "dataset_name": "sharegpt",
294 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
295 - "num_prompts": 200
296 - }
297 - },
298 - {
299 - "test_name": "serving_llama8B_int8_tp1_random_128_128",
300 - "qps_list": ["inf"],
301 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
302 - "server_environment_variables": {
303 - "VLLM_RPC_TIMEOUT": 100000,
304 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
305 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
306 - "VLLM_CPU_SGL_KERNEL": 1,
307 - "VLLM_CPU_KVCACHE_SPACE": 40
308 - },
309 - "server_parameters": {
310 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
311 - "tensor_parallel_size": 1,
312 - "dtype": "bfloat16",
313 - "distributed_executor_backend": "mp",
314 - "block_size": 128,
315 - "trust_remote_code": "",
316 - "enable_chunked_prefill": "",
317 - "disable_log_stats": "",
318 - "enforce_eager": "",
319 - "max_num_batched_tokens": 2048,
320 - "max_num_seqs": 256,
321 - "load_format": "dummy"
322 - },
323 - "client_parameters": {
324 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
325 - "backend": "vllm",
326 - "dataset_name": "random",
327 - "random-input-len": 128,
328 - "random-output-len": 128,
329 - "ignore-eos": "",
330 - "num_prompts": 1000
331 - }
332 - },
333 - {
334 - "test_name": "serving_llama8B_int8_tp2_random_128_128",
335 - "qps_list": ["inf"],
336 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
337 - "server_environment_variables": {
338 - "VLLM_RPC_TIMEOUT": 100000,
339 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
340 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
341 - "VLLM_CPU_SGL_KERNEL": 1,
342 - "VLLM_CPU_KVCACHE_SPACE": 40
343 - },
344 - "server_parameters": {
345 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
346 - "tensor_parallel_size": 2,
347 - "dtype": "bfloat16",
348 - "distributed_executor_backend": "mp",
349 - "block_size": 128,
350 - "trust_remote_code": "",
351 - "enable_chunked_prefill": "",
352 - "disable_log_stats": "",
353 - "enforce_eager": "",
354 - "max_num_batched_tokens": 2048,
355 - "max_num_seqs": 256,
356 - "load_format": "dummy"
357 - },
358 - "client_parameters": {
359 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
360 - "backend": "vllm",
361 - "dataset_name": "random",
362 - "random-input-len": 128,
363 - "random-output-len": 128,
364 - "ignore-eos": "",
365 - "num_prompts": 1000
366 - }
367 - },
368 - {
369 - "test_name": "serving_llama8B_int8_tp4_random_128_128",
370 - "qps_list": ["inf"],
371 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
372 - "server_environment_variables": {
373 - "VLLM_RPC_TIMEOUT": 100000,
374 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
375 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
376 - "VLLM_CPU_SGL_KERNEL": 1,
377 - "VLLM_CPU_KVCACHE_SPACE": 40
378 - },
379 - "server_parameters": {
380 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
381 - "tensor_parallel_size": 4,
382 - "dtype": "bfloat16",
383 - "distributed_executor_backend": "mp",
384 - "block_size": 128,
385 - "trust_remote_code": "",
386 - "enable_chunked_prefill": "",
387 - "disable_log_stats": "",
388 - "enforce_eager": "",
389 - "max_num_batched_tokens": 2048,
390 - "max_num_seqs": 256,
391 - "load_format": "dummy"
392 - },
393 - "client_parameters": {
394 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
395 - "backend": "vllm",
396 - "dataset_name": "random",
397 - "random-input-len": 128,
398 - "random-output-len": 128,
399 - "ignore-eos": "",
400 - "num_prompts": 1000
401 - }
402 - },
403 - {
404 - "test_name": "serving_llama8B_int4_tp1_sharegpt",
405 - "qps_list": ["inf"],
406 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
407 - "server_environment_variables": {
408 - "VLLM_RPC_TIMEOUT": 100000,
409 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
410 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
411 - "VLLM_CPU_SGL_KERNEL": 1,
412 - "VLLM_CPU_KVCACHE_SPACE": 40
413 - },
414 - "server_parameters": {
415 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
416 - "quantization": "awq",
417 - "tensor_parallel_size": 1,
418 - "dtype": "bfloat16",
419 - "distributed_executor_backend": "mp",
420 - "block_size": 128,
421 - "trust_remote_code": "",
422 - "disable_log_stats": "",
423 - "enforce_eager": "",
424 - "max_num_batched_tokens": 2048,
425 - "max_num_seqs": 256,
426 - "load_format": "dummy"
427 - },
428 - "client_parameters": {
429 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
430 - "backend": "vllm",
431 - "dataset_name": "sharegpt",
432 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
433 - "num_prompts": 200
434 - }
435 - },
436 - {
437 - "test_name": "serving_llama8B_int4_tp2_sharegpt",
438 - "qps_list": ["inf"],
439 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
440 - "server_environment_variables": {
441 - "VLLM_RPC_TIMEOUT": 100000,
442 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
443 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
444 - "VLLM_CPU_SGL_KERNEL": 1,
445 - "VLLM_CPU_KVCACHE_SPACE": 40
446 - },
447 - "server_parameters": {
448 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
449 - "quantization": "awq",
450 - "tensor_parallel_size": 2,
451 - "dtype": "bfloat16",
452 - "distributed_executor_backend": "mp",
453 - "block_size": 128,
454 - "trust_remote_code": "",
455 - "disable_log_stats": "",
456 - "enforce_eager": "",
457 - "max_num_batched_tokens": 2048,
458 - "max_num_seqs": 256,
459 - "load_format": "dummy"
460 - },
461 - "client_parameters": {
462 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
463 - "backend": "vllm",
464 - "dataset_name": "sharegpt",
465 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
466 - "num_prompts": 200
467 - }
468 - },
469 - {
470 - "test_name": "serving_llama8B_int4_tp4_sharegpt",
471 - "qps_list": ["inf"],
472 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
473 - "server_environment_variables": {
474 - "VLLM_RPC_TIMEOUT": 100000,
475 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
476 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
477 - "VLLM_CPU_SGL_KERNEL": 1,
478 - "VLLM_CPU_KVCACHE_SPACE": 40
479 - },
480 - "server_parameters": {
481 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
482 - "quantization": "awq",
483 - "tensor_parallel_size": 4,
484 - "dtype": "bfloat16",
485 - "distributed_executor_backend": "mp",
486 - "block_size": 128,
487 - "trust_remote_code": "",
488 - "disable_log_stats": "",
489 - "enforce_eager": "",
490 - "max_num_batched_tokens": 2048,
491 - "max_num_seqs": 256,
492 - "load_format": "dummy"
493 - },
494 - "client_parameters": {
495 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
496 - "backend": "vllm",
497 - "dataset_name": "sharegpt",
498 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
499 - "num_prompts": 200
500 - }
501 - },
502 - {
503 - "test_name": "serving_llama8B_int4_tp1_random_128_128",
504 - "qps_list": ["inf"],
505 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
506 - "server_environment_variables": {
507 - "VLLM_RPC_TIMEOUT": 100000,
508 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
509 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
510 - "VLLM_CPU_SGL_KERNEL": 1,
511 - "VLLM_CPU_KVCACHE_SPACE": 40
512 - },
513 - "server_parameters": {
514 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
515 - "quantization": "awq",
516 - "tensor_parallel_size": 1,
517 - "dtype": "bfloat16",
518 - "distributed_executor_backend": "mp",
519 - "block_size": 128,
520 - "trust_remote_code": "",
521 - "enable_chunked_prefill": "",
522 - "disable_log_stats": "",
523 - "enforce_eager": "",
524 - "max_num_batched_tokens": 2048,
525 - "max_num_seqs": 256,
526 - "load_format": "dummy"
527 - },
528 - "client_parameters": {
529 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
530 - "backend": "vllm",
531 - "dataset_name": "random",
532 - "random-input-len": 128,
533 - "random-output-len": 128,
534 - "ignore-eos": "",
535 - "num_prompts": 1000
536 - }
537 - },
538 - {
539 - "test_name": "serving_llama8B_int4_tp2_random_128_128",
540 - "qps_list": ["inf"],
541 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
542 - "server_environment_variables": {
543 - "VLLM_RPC_TIMEOUT": 100000,
544 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
545 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
546 - "VLLM_CPU_SGL_KERNEL": 1,
547 - "VLLM_CPU_KVCACHE_SPACE": 40
548 - },
549 - "server_parameters": {
550 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
551 - "quantization": "awq",
552 - "tensor_parallel_size": 2,
553 - "dtype": "bfloat16",
554 - "distributed_executor_backend": "mp",
555 - "block_size": 128,
556 - "trust_remote_code": "",
557 - "enable_chunked_prefill": "",
558 - "disable_log_stats": "",
559 - "enforce_eager": "",
560 - "max_num_batched_tokens": 2048,
561 - "max_num_seqs": 256,
562 - "load_format": "dummy"
563 - },
564 - "client_parameters": {
565 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
566 - "backend": "vllm",
567 - "dataset_name": "random",
568 - "random-input-len": 128,
569 - "random-output-len": 128,
570 - "ignore-eos": "",
571 - "num_prompts": 1000
572 - }
573 - },
574 - {
575 - "test_name": "serving_llama8B_int4_tp4_random_128_128",
576 - "qps_list": ["inf"],
577 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
578 - "server_environment_variables": {
579 - "VLLM_RPC_TIMEOUT": 100000,
580 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
581 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
582 - "VLLM_CPU_SGL_KERNEL": 1,
583 - "VLLM_CPU_KVCACHE_SPACE": 40
584 - },
585 - "server_parameters": {
586 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
587 - "quantization": "awq",
588 - "tensor_parallel_size": 4,
589 - "dtype": "bfloat16",
590 - "distributed_executor_backend": "mp",
591 - "block_size": 128,
592 - "trust_remote_code": "",
593 - "enable_chunked_prefill": "",
594 - "disable_log_stats": "",
595 - "enforce_eager": "",
596 - "max_num_batched_tokens": 2048,
597 - "max_num_seqs": 256,
598 - "load_format": "dummy"
599 - },
600 - "client_parameters": {
601 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
602 - "backend": "vllm",
603 - "dataset_name": "random",
604 - "random-input-len": 128,
605 - "random-output-len": 128,
606 - "ignore-eos": "",
607 - "num_prompts": 1000
608 - }
609 - }
610 -]
.buildkite/performance-benchmarks/tests/serving-tests-cpu-snc3.json deleted
-1023
@@ -1,1023 +0,0 @@
1 -[
2 - {
3 - "test_name": "serving_llama8B_bf16_pp1_sharegpt",
4 - "qps_list": ["inf"],
5 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
6 - "server_environment_variables": {
7 - "VLLM_RPC_TIMEOUT": 100000,
8 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
9 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
10 - "VLLM_CPU_SGL_KERNEL": 1,
11 - "VLLM_CPU_KVCACHE_SPACE": 40
12 - },
13 - "server_parameters": {
14 - "model": "meta-llama/Llama-3.1-8B-Instruct",
15 - "pipeline_parallel_size": 1,
16 - "dtype": "bfloat16",
17 - "distributed_executor_backend": "mp",
18 - "block_size": 128,
19 - "trust_remote_code": "",
20 - "disable_log_stats": "",
21 - "enforce_eager": "",
22 - "max_num_batched_tokens": 2048,
23 - "max_num_seqs": 256,
24 - "load_format": "dummy"
25 - },
26 - "client_parameters": {
27 - "model": "meta-llama/Llama-3.1-8B-Instruct",
28 - "backend": "vllm",
29 - "dataset_name": "sharegpt",
30 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
31 - "num_prompts": 200
32 - }
33 - },
34 - {
35 - "test_name": "serving_llama8B_bf16_tp2_sharegpt",
36 - "qps_list": ["inf"],
37 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
38 - "server_environment_variables": {
39 - "VLLM_RPC_TIMEOUT": 100000,
40 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
41 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
42 - "VLLM_CPU_SGL_KERNEL": 1,
43 - "VLLM_CPU_KVCACHE_SPACE": 40
44 - },
45 - "server_parameters": {
46 - "model": "meta-llama/Llama-3.1-8B-Instruct",
47 - "tensor_parallel_size": 2,
48 - "dtype": "bfloat16",
49 - "distributed_executor_backend": "mp",
50 - "block_size": 128,
51 - "trust_remote_code": "",
52 - "disable_log_stats": "",
53 - "enforce_eager": "",
54 - "max_num_batched_tokens": 2048,
55 - "max_num_seqs": 256,
56 - "load_format": "dummy"
57 - },
58 - "client_parameters": {
59 - "model": "meta-llama/Llama-3.1-8B-Instruct",
60 - "backend": "vllm",
61 - "dataset_name": "sharegpt",
62 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
63 - "num_prompts": 200
64 - }
65 - },
66 - {
67 - "test_name": "serving_llama8B_bf16_pp3_sharegpt",
68 - "qps_list": ["inf"],
69 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
70 - "server_environment_variables": {
71 - "VLLM_RPC_TIMEOUT": 100000,
72 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
73 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
74 - "VLLM_CPU_SGL_KERNEL": 1,
75 - "VLLM_CPU_KVCACHE_SPACE": 40
76 - },
77 - "server_parameters": {
78 - "model": "meta-llama/Llama-3.1-8B-Instruct",
79 - "pipeline_parallel_size": 3,
80 - "dtype": "bfloat16",
81 - "distributed_executor_backend": "mp",
82 - "block_size": 128,
83 - "trust_remote_code": "",
84 - "disable_log_stats": "",
85 - "enforce_eager": "",
86 - "max_num_batched_tokens": 2048,
87 - "max_num_seqs": 256,
88 - "load_format": "dummy"
89 - },
90 - "client_parameters": {
91 - "model": "meta-llama/Llama-3.1-8B-Instruct",
92 - "backend": "vllm",
93 - "dataset_name": "sharegpt",
94 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
95 - "num_prompts": 200
96 - }
97 - },
98 - {
99 - "test_name": "serving_llama8B_bf16_tp4_sharegpt",
100 - "qps_list": ["inf"],
101 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
102 - "server_environment_variables": {
103 - "VLLM_RPC_TIMEOUT": 100000,
104 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
105 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
106 - "VLLM_CPU_SGL_KERNEL": 1,
107 - "VLLM_CPU_KVCACHE_SPACE": 40
108 - },
109 - "server_parameters": {
110 - "model": "meta-llama/Llama-3.1-8B-Instruct",
111 - "tensor_parallel_size": 4,
112 - "dtype": "bfloat16",
113 - "distributed_executor_backend": "mp",
114 - "block_size": 128,
115 - "trust_remote_code": "",
116 - "disable_log_stats": "",
117 - "enforce_eager": "",
118 - "max_num_batched_tokens": 2048,
119 - "max_num_seqs": 256,
120 - "load_format": "dummy"
121 - },
122 - "client_parameters": {
123 - "model": "meta-llama/Llama-3.1-8B-Instruct",
124 - "backend": "vllm",
125 - "dataset_name": "sharegpt",
126 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
127 - "num_prompts": 200
128 - }
129 - },
130 - {
131 - "test_name": "serving_llama8B_bf16_tp2pp3_sharegpt",
132 - "qps_list": ["inf"],
133 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
134 - "server_environment_variables": {
135 - "VLLM_RPC_TIMEOUT": 100000,
136 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
137 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
138 - "VLLM_CPU_SGL_KERNEL": 1,
139 - "VLLM_CPU_KVCACHE_SPACE": 40
140 - },
141 - "server_parameters": {
142 - "model": "meta-llama/Llama-3.1-8B-Instruct",
143 - "tensor_parallel_size": 2,
144 - "pipeline_parallel_size": 3,
145 - "dtype": "bfloat16",
146 - "distributed_executor_backend": "mp",
147 - "block_size": 128,
148 - "trust_remote_code": "",
149 - "disable_log_stats": "",
150 - "enforce_eager": "",
151 - "max_num_batched_tokens": 2048,
152 - "max_num_seqs": 256,
153 - "load_format": "dummy"
154 - },
155 - "client_parameters": {
156 - "model": "meta-llama/Llama-3.1-8B-Instruct",
157 - "backend": "vllm",
158 - "dataset_name": "sharegpt",
159 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
160 - "num_prompts": 200
161 - }
162 - },
163 - {
164 - "test_name": "serving_llama8B_bf16_pp1_random_128_128",
165 - "qps_list": ["inf"],
166 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
167 - "server_environment_variables": {
168 - "VLLM_RPC_TIMEOUT": 100000,
169 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
170 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
171 - "VLLM_CPU_SGL_KERNEL": 1,
172 - "VLLM_CPU_KVCACHE_SPACE": 40
173 - },
174 - "server_parameters": {
175 - "model": "meta-llama/Llama-3.1-8B-Instruct",
176 - "pipeline_parallel_size": 1,
177 - "dtype": "bfloat16",
178 - "distributed_executor_backend": "mp",
179 - "block_size": 128,
180 - "trust_remote_code": "",
181 - "enable_chunked_prefill": "",
182 - "disable_log_stats": "",
183 - "enforce_eager": "",
184 - "max_num_batched_tokens": 2048,
185 - "max_num_seqs": 256,
186 - "load_format": "dummy"
187 - },
188 - "client_parameters": {
189 - "model": "meta-llama/Llama-3.1-8B-Instruct",
190 - "backend": "vllm",
191 - "dataset_name": "random",
192 - "random-input-len": 128,
193 - "random-output-len": 128,
194 - "ignore-eos": "",
195 - "num_prompts": 1000
196 - }
197 - },
198 - {
199 - "test_name": "serving_llama8B_bf16_tp2_random_128_128",
200 - "qps_list": ["inf"],
201 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
202 - "server_environment_variables": {
203 - "VLLM_RPC_TIMEOUT": 100000,
204 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
205 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
206 - "VLLM_CPU_SGL_KERNEL": 1,
207 - "VLLM_CPU_KVCACHE_SPACE": 40
208 - },
209 - "server_parameters": {
210 - "model": "meta-llama/Llama-3.1-8B-Instruct",
211 - "tensor_parallel_size": 2,
212 - "dtype": "bfloat16",
213 - "distributed_executor_backend": "mp",
214 - "block_size": 128,
215 - "trust_remote_code": "",
216 - "enable_chunked_prefill": "",
217 - "disable_log_stats": "",
218 - "enforce_eager": "",
219 - "max_num_batched_tokens": 2048,
220 - "max_num_seqs": 256,
221 - "load_format": "dummy"
222 - },
223 - "client_parameters": {
224 - "model": "meta-llama/Llama-3.1-8B-Instruct",
225 - "backend": "vllm",
226 - "dataset_name": "random",
227 - "random-input-len": 128,
228 - "random-output-len": 128,
229 - "ignore-eos": "",
230 - "num_prompts": 1000
231 - }
232 - },
233 - {
234 - "test_name": "serving_llama8B_bf16_pp3_random_128_128",
235 - "qps_list": ["inf"],
236 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
237 - "server_environment_variables": {
238 - "VLLM_RPC_TIMEOUT": 100000,
239 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
240 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
241 - "VLLM_CPU_SGL_KERNEL": 1,
242 - "VLLM_CPU_KVCACHE_SPACE": 40
243 - },
244 - "server_parameters": {
245 - "model": "meta-llama/Llama-3.1-8B-Instruct",
246 - "pipeline_parallel_size": 3,
247 - "dtype": "bfloat16",
248 - "distributed_executor_backend": "mp",
249 - "block_size": 128,
250 - "trust_remote_code": "",
251 - "enable_chunked_prefill": "",
252 - "disable_log_stats": "",
253 - "enforce_eager": "",
254 - "max_num_batched_tokens": 2048,
255 - "max_num_seqs": 256,
256 - "load_format": "dummy"
257 - },
258 - "client_parameters": {
259 - "model": "meta-llama/Llama-3.1-8B-Instruct",
260 - "backend": "vllm",
261 - "dataset_name": "random",
262 - "random-input-len": 128,
263 - "random-output-len": 128,
264 - "ignore-eos": "",
265 - "num_prompts": 1000
266 - }
267 - },
268 - {
269 - "test_name": "serving_llama8B_bf16_tp4_random_128_128",
270 - "qps_list": ["inf"],
271 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
272 - "server_environment_variables": {
273 - "VLLM_RPC_TIMEOUT": 100000,
274 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
275 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
276 - "VLLM_CPU_SGL_KERNEL": 1,
277 - "VLLM_CPU_KVCACHE_SPACE": 40
278 - },
279 - "server_parameters": {
280 - "model": "meta-llama/Llama-3.1-8B-Instruct",
281 - "tensor_parallel_size": 4,
282 - "dtype": "bfloat16",
283 - "distributed_executor_backend": "mp",
284 - "block_size": 128,
285 - "trust_remote_code": "",
286 - "enable_chunked_prefill": "",
287 - "disable_log_stats": "",
288 - "enforce_eager": "",
289 - "max_num_batched_tokens": 2048,
290 - "max_num_seqs": 256,
291 - "load_format": "dummy"
292 - },
293 - "client_parameters": {
294 - "model": "meta-llama/Llama-3.1-8B-Instruct",
295 - "backend": "vllm",
296 - "dataset_name": "random",
297 - "random-input-len": 128,
298 - "random-output-len": 128,
299 - "ignore-eos": "",
300 - "num_prompts": 1000
301 - }
302 - },
303 - {
304 - "test_name": "serving_llama8B_bf16_tp2pp3_random_128_128",
305 - "qps_list": ["inf"],
306 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
307 - "server_environment_variables": {
308 - "VLLM_RPC_TIMEOUT": 100000,
309 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
310 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
311 - "VLLM_CPU_SGL_KERNEL": 1,
312 - "VLLM_CPU_KVCACHE_SPACE": 40
313 - },
314 - "server_parameters": {
315 - "model": "meta-llama/Llama-3.1-8B-Instruct",
316 - "tensor_parallel_size": 2,
317 - "pipeline_parallel_size": 3,
318 - "dtype": "bfloat16",
319 - "distributed_executor_backend": "mp",
320 - "block_size": 128,
321 - "trust_remote_code": "",
322 - "enable_chunked_prefill": "",
323 - "disable_log_stats": "",
324 - "enforce_eager": "",
325 - "max_num_batched_tokens": 2048,
326 - "max_num_seqs": 256,
327 - "load_format": "dummy"
328 - },
329 - "client_parameters": {
330 - "model": "meta-llama/Llama-3.1-8B-Instruct",
331 - "backend": "vllm",
332 - "dataset_name": "random",
333 - "random-input-len": 128,
334 - "random-output-len": 128,
335 - "ignore-eos": "",
336 - "num_prompts": 1000
337 - }
338 - },
339 - {
340 - "test_name": "serving_llama8B_int8_pp1_sharegpt",
341 - "qps_list": ["inf"],
342 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
343 - "server_environment_variables": {
344 - "VLLM_RPC_TIMEOUT": 100000,
345 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
346 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
347 - "VLLM_CPU_SGL_KERNEL": 1,
348 - "VLLM_CPU_KVCACHE_SPACE": 40
349 - },
350 - "server_parameters": {
351 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
352 - "pipeline_parallel_size": 1,
353 - "dtype": "bfloat16",
354 - "distributed_executor_backend": "mp",
355 - "block_size": 128,
356 - "trust_remote_code": "",
357 - "disable_log_stats": "",
358 - "enforce_eager": "",
359 - "max_num_batched_tokens": 2048,
360 - "max_num_seqs": 256,
361 - "load_format": "dummy"
362 - },
363 - "client_parameters": {
364 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
365 - "backend": "vllm",
366 - "dataset_name": "sharegpt",
367 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
368 - "num_prompts": 200
369 - }
370 - },
371 - {
372 - "test_name": "serving_llama8B_int8_tp2_sharegpt",
373 - "qps_list": ["inf"],
374 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
375 - "server_environment_variables": {
376 - "VLLM_RPC_TIMEOUT": 100000,
377 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
378 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
379 - "VLLM_CPU_SGL_KERNEL": 1,
380 - "VLLM_CPU_KVCACHE_SPACE": 40
381 - },
382 - "server_parameters": {
383 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
384 - "tensor_parallel_size": 2,
385 - "dtype": "bfloat16",
386 - "distributed_executor_backend": "mp",
387 - "block_size": 128,
388 - "trust_remote_code": "",
389 - "disable_log_stats": "",
390 - "enforce_eager": "",
391 - "max_num_batched_tokens": 2048,
392 - "max_num_seqs": 256,
393 - "load_format": "dummy"
394 - },
395 - "client_parameters": {
396 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
397 - "backend": "vllm",
398 - "dataset_name": "sharegpt",
399 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
400 - "num_prompts": 200
401 - }
402 - },
403 - {
404 - "test_name": "serving_llama8B_int8_pp3_sharegpt",
405 - "qps_list": ["inf"],
406 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
407 - "server_environment_variables": {
408 - "VLLM_RPC_TIMEOUT": 100000,
409 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
410 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
411 - "VLLM_CPU_SGL_KERNEL": 1,
412 - "VLLM_CPU_KVCACHE_SPACE": 40
413 - },
414 - "server_parameters": {
415 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
416 - "pipeline_parallel_size": 3,
417 - "dtype": "bfloat16",
418 - "distributed_executor_backend": "mp",
419 - "block_size": 128,
420 - "trust_remote_code": "",
421 - "disable_log_stats": "",
422 - "enforce_eager": "",
423 - "max_num_batched_tokens": 2048,
424 - "max_num_seqs": 256,
425 - "load_format": "dummy"
426 - },
427 - "client_parameters": {
428 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
429 - "backend": "vllm",
430 - "dataset_name": "sharegpt",
431 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
432 - "num_prompts": 200
433 - }
434 - },
435 - {
436 - "test_name": "serving_llama8B_int8_tp4_sharegpt",
437 - "qps_list": ["inf"],
438 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
439 - "server_environment_variables": {
440 - "VLLM_RPC_TIMEOUT": 100000,
441 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
442 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
443 - "VLLM_CPU_SGL_KERNEL": 1,
444 - "VLLM_CPU_KVCACHE_SPACE": 40
445 - },
446 - "server_parameters": {
447 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
448 - "tensor_parallel_size": 4,
449 - "dtype": "bfloat16",
450 - "distributed_executor_backend": "mp",
451 - "block_size": 128,
452 - "trust_remote_code": "",
453 - "disable_log_stats": "",
454 - "enforce_eager": "",
455 - "max_num_batched_tokens": 2048,
456 - "max_num_seqs": 256,
457 - "load_format": "dummy"
458 - },
459 - "client_parameters": {
460 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
461 - "backend": "vllm",
462 - "dataset_name": "sharegpt",
463 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
464 - "num_prompts": 200
465 - }
466 - },
467 - {
468 - "test_name": "serving_llama8B_int8_tp2pp3_sharegpt",
469 - "qps_list": ["inf"],
470 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
471 - "server_environment_variables": {
472 - "VLLM_RPC_TIMEOUT": 100000,
473 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
474 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
475 - "VLLM_CPU_SGL_KERNEL": 1,
476 - "VLLM_CPU_KVCACHE_SPACE": 40
477 - },
478 - "server_parameters": {
479 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
480 - "tensor_parallel_size": 2,
481 - "pipeline_parallel_size": 3,
482 - "dtype": "bfloat16",
483 - "distributed_executor_backend": "mp",
484 - "block_size": 128,
485 - "trust_remote_code": "",
486 - "disable_log_stats": "",
487 - "enforce_eager": "",
488 - "max_num_batched_tokens": 2048,
489 - "max_num_seqs": 256,
490 - "load_format": "dummy"
491 - },
492 - "client_parameters": {
493 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
494 - "backend": "vllm",
495 - "dataset_name": "sharegpt",
496 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
497 - "num_prompts": 200
498 - }
499 - },
500 - {
501 - "test_name": "serving_llama8B_int8_pp1_random_128_128",
502 - "qps_list": ["inf"],
503 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
504 - "server_environment_variables": {
505 - "VLLM_RPC_TIMEOUT": 100000,
506 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
507 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
508 - "VLLM_CPU_SGL_KERNEL": 1,
509 - "VLLM_CPU_KVCACHE_SPACE": 40
510 - },
511 - "server_parameters": {
512 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
513 - "pipeline_parallel_size": 1,
514 - "dtype": "bfloat16",
515 - "distributed_executor_backend": "mp",
516 - "block_size": 128,
517 - "trust_remote_code": "",
518 - "enable_chunked_prefill": "",
519 - "disable_log_stats": "",
520 - "enforce_eager": "",
521 - "max_num_batched_tokens": 2048,
522 - "max_num_seqs": 256,
523 - "load_format": "dummy"
524 - },
525 - "client_parameters": {
526 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
527 - "backend": "vllm",
528 - "dataset_name": "random",
529 - "random-input-len": 128,
530 - "random-output-len": 128,
531 - "ignore-eos": "",
532 - "num_prompts": 1000
533 - }
534 - },
535 - {
536 - "test_name": "serving_llama8B_int8_tp2_random_128_128",
537 - "qps_list": ["inf"],
538 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
539 - "server_environment_variables": {
540 - "VLLM_RPC_TIMEOUT": 100000,
541 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
542 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
543 - "VLLM_CPU_SGL_KERNEL": 1,
544 - "VLLM_CPU_KVCACHE_SPACE": 40
545 - },
546 - "server_parameters": {
547 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
548 - "tensor_parallel_size": 2,
549 - "dtype": "bfloat16",
550 - "distributed_executor_backend": "mp",
551 - "block_size": 128,
552 - "trust_remote_code": "",
553 - "enable_chunked_prefill": "",
554 - "disable_log_stats": "",
555 - "enforce_eager": "",
556 - "max_num_batched_tokens": 2048,
557 - "max_num_seqs": 256,
558 - "load_format": "dummy"
559 - },
560 - "client_parameters": {
561 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
562 - "backend": "vllm",
563 - "dataset_name": "random",
564 - "random-input-len": 128,
565 - "random-output-len": 128,
566 - "ignore-eos": "",
567 - "num_prompts": 1000
568 - }
569 - },
570 - {
571 - "test_name": "serving_llama8B_int8_pp3_random_128_128",
572 - "qps_list": ["inf"],
573 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
574 - "server_environment_variables": {
575 - "VLLM_RPC_TIMEOUT": 100000,
576 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
577 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
578 - "VLLM_CPU_SGL_KERNEL": 1,
579 - "VLLM_CPU_KVCACHE_SPACE": 40
580 - },
581 - "server_parameters": {
582 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
583 - "pipeline_parallel_size": 3,
584 - "dtype": "bfloat16",
585 - "distributed_executor_backend": "mp",
586 - "block_size": 128,
587 - "trust_remote_code": "",
588 - "enable_chunked_prefill": "",
589 - "disable_log_stats": "",
590 - "enforce_eager": "",
591 - "max_num_batched_tokens": 2048,
592 - "max_num_seqs": 256,
593 - "load_format": "dummy"
594 - },
595 - "client_parameters": {
596 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
597 - "backend": "vllm",
598 - "dataset_name": "random",
599 - "random-input-len": 128,
600 - "random-output-len": 128,
601 - "ignore-eos": "",
602 - "num_prompts": 1000
603 - }
604 - },
605 - {
606 - "test_name": "serving_llama8B_int8_tp4_random_128_128",
607 - "qps_list": ["inf"],
608 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
609 - "server_environment_variables": {
610 - "VLLM_RPC_TIMEOUT": 100000,
611 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
612 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
613 - "VLLM_CPU_SGL_KERNEL": 1,
614 - "VLLM_CPU_KVCACHE_SPACE": 40
615 - },
616 - "server_parameters": {
617 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
618 - "tensor_parallel_size": 4,
619 - "dtype": "bfloat16",
620 - "distributed_executor_backend": "mp",
621 - "block_size": 128,
622 - "trust_remote_code": "",
623 - "enable_chunked_prefill": "",
624 - "disable_log_stats": "",
625 - "enforce_eager": "",
626 - "max_num_batched_tokens": 2048,
627 - "max_num_seqs": 256,
628 - "load_format": "dummy"
629 - },
630 - "client_parameters": {
631 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
632 - "backend": "vllm",
633 - "dataset_name": "random",
634 - "random-input-len": 128,
635 - "random-output-len": 128,
636 - "ignore-eos": "",
637 - "num_prompts": 1000
638 - }
639 - },
640 - {
641 - "test_name": "serving_llama8B_int8_tp2pp3_random_128_128",
642 - "qps_list": ["inf"],
643 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
644 - "server_environment_variables": {
645 - "VLLM_RPC_TIMEOUT": 100000,
646 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
647 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
648 - "VLLM_CPU_SGL_KERNEL": 1,
649 - "VLLM_CPU_KVCACHE_SPACE": 40
650 - },
651 - "server_parameters": {
652 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
653 - "tensor_parallel_size": 2,
654 - "pipeline_parallel_size": 3,
655 - "dtype": "bfloat16",
656 - "distributed_executor_backend": "mp",
657 - "block_size": 128,
658 - "trust_remote_code": "",
659 - "enable_chunked_prefill": "",
660 - "disable_log_stats": "",
661 - "enforce_eager": "",
662 - "max_num_batched_tokens": 2048,
663 - "max_num_seqs": 256,
664 - "load_format": "dummy"
665 - },
666 - "client_parameters": {
667 - "model": "RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8",
668 - "backend": "vllm",
669 - "dataset_name": "random",
670 - "random-input-len": 128,
671 - "random-output-len": 128,
672 - "ignore-eos": "",
673 - "num_prompts": 1000
674 - }
675 - },
676 - {
677 - "test_name": "serving_llama8B_int4_pp1_sharegpt",
678 - "qps_list": ["inf"],
679 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
680 - "server_environment_variables": {
681 - "VLLM_RPC_TIMEOUT": 100000,
682 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
683 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
684 - "VLLM_CPU_SGL_KERNEL": 1,
685 - "VLLM_CPU_KVCACHE_SPACE": 40
686 - },
687 - "server_parameters": {
688 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
689 - "quantization": "awq",
690 - "pipeline_parallel_size": 1,
691 - "dtype": "bfloat16",
692 - "distributed_executor_backend": "mp",
693 - "block_size": 128,
694 - "trust_remote_code": "",
695 - "disable_log_stats": "",
696 - "enforce_eager": "",
697 - "max_num_batched_tokens": 2048,
698 - "max_num_seqs": 256,
699 - "load_format": "dummy"
700 - },
701 - "client_parameters": {
702 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
703 - "backend": "vllm",
704 - "dataset_name": "sharegpt",
705 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
706 - "num_prompts": 200
707 - }
708 - },
709 - {
710 - "test_name": "serving_llama8B_int4_tp2_sharegpt",
711 - "qps_list": ["inf"],
712 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
713 - "server_environment_variables": {
714 - "VLLM_RPC_TIMEOUT": 100000,
715 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
716 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
717 - "VLLM_CPU_SGL_KERNEL": 1,
718 - "VLLM_CPU_KVCACHE_SPACE": 40
719 - },
720 - "server_parameters": {
721 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
722 - "quantization": "awq",
723 - "tensor_parallel_size": 2,
724 - "dtype": "bfloat16",
725 - "distributed_executor_backend": "mp",
726 - "block_size": 128,
727 - "trust_remote_code": "",
728 - "disable_log_stats": "",
729 - "enforce_eager": "",
730 - "max_num_batched_tokens": 2048,
731 - "max_num_seqs": 256,
732 - "load_format": "dummy"
733 - },
734 - "client_parameters": {
735 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
736 - "backend": "vllm",
737 - "dataset_name": "sharegpt",
738 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
739 - "num_prompts": 200
740 - }
741 - },
742 - {
743 - "test_name": "serving_llama8B_int4_pp3_sharegpt",
744 - "qps_list": ["inf"],
745 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
746 - "server_environment_variables": {
747 - "VLLM_RPC_TIMEOUT": 100000,
748 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
749 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
750 - "VLLM_CPU_SGL_KERNEL": 1,
751 - "VLLM_CPU_KVCACHE_SPACE": 40
752 - },
753 - "server_parameters": {
754 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
755 - "quantization": "awq",
756 - "pipeline_parallel_size": 3,
757 - "dtype": "bfloat16",
758 - "distributed_executor_backend": "mp",
759 - "block_size": 128,
760 - "trust_remote_code": "",
761 - "disable_log_stats": "",
762 - "enforce_eager": "",
763 - "max_num_batched_tokens": 2048,
764 - "max_num_seqs": 256,
765 - "load_format": "dummy"
766 - },
767 - "client_parameters": {
768 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
769 - "backend": "vllm",
770 - "dataset_name": "sharegpt",
771 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
772 - "num_prompts": 200
773 - }
774 - },
775 - {
776 - "test_name": "serving_llama8B_int4_tp4_sharegpt",
777 - "qps_list": ["inf"],
778 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
779 - "server_environment_variables": {
780 - "VLLM_RPC_TIMEOUT": 100000,
781 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
782 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
783 - "VLLM_CPU_SGL_KERNEL": 1,
784 - "VLLM_CPU_KVCACHE_SPACE": 40
785 - },
786 - "server_parameters": {
787 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
788 - "quantization": "awq",
789 - "tensor_parallel_size": 4,
790 - "dtype": "bfloat16",
791 - "distributed_executor_backend": "mp",
792 - "block_size": 128,
793 - "trust_remote_code": "",
794 - "disable_log_stats": "",
795 - "enforce_eager": "",
796 - "max_num_batched_tokens": 2048,
797 - "max_num_seqs": 256,
798 - "load_format": "dummy"
799 - },
800 - "client_parameters": {
801 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
802 - "backend": "vllm",
803 - "dataset_name": "sharegpt",
804 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
805 - "num_prompts": 200
806 - }
807 - },
808 - {
809 - "test_name": "serving_llama8B_int4_tp2pp3_sharegpt",
810 - "qps_list": ["inf"],
811 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
812 - "server_environment_variables": {
813 - "VLLM_RPC_TIMEOUT": 100000,
814 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
815 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
816 - "VLLM_CPU_SGL_KERNEL": 1,
817 - "VLLM_CPU_KVCACHE_SPACE": 40
818 - },
819 - "server_parameters": {
820 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
821 - "quantization": "awq",
822 - "tensor_parallel_size": 2,
823 - "pipeline_parallel_size": 3,
824 - "dtype": "bfloat16",
825 - "distributed_executor_backend": "mp",
826 - "block_size": 128,
827 - "trust_remote_code": "",
828 - "disable_log_stats": "",
829 - "enforce_eager": "",
830 - "max_num_batched_tokens": 2048,
831 - "max_num_seqs": 256,
832 - "load_format": "dummy"
833 - },
834 - "client_parameters": {
835 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
836 - "backend": "vllm",
837 - "dataset_name": "sharegpt",
838 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
839 - "num_prompts": 200
840 - }
841 - },
842 - {
843 - "test_name": "serving_llama8B_int4_pp1_random_128_128",
844 - "qps_list": ["inf"],
845 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
846 - "server_environment_variables": {
847 - "VLLM_RPC_TIMEOUT": 100000,
848 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
849 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
850 - "VLLM_CPU_SGL_KERNEL": 1,
851 - "VLLM_CPU_KVCACHE_SPACE": 40
852 - },
853 - "server_parameters": {
854 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
855 - "quantization": "awq",
856 - "pipeline_parallel_size": 1,
857 - "dtype": "bfloat16",
858 - "distributed_executor_backend": "mp",
859 - "block_size": 128,
860 - "trust_remote_code": "",
861 - "enable_chunked_prefill": "",
862 - "disable_log_stats": "",
863 - "enforce_eager": "",
864 - "max_num_batched_tokens": 2048,
865 - "max_num_seqs": 256,
866 - "load_format": "dummy"
867 - },
868 - "client_parameters": {
869 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
870 - "backend": "vllm",
871 - "dataset_name": "random",
872 - "random-input-len": 128,
873 - "random-output-len": 128,
874 - "ignore-eos": "",
875 - "num_prompts": 1000
876 - }
877 - },
878 - {
879 - "test_name": "serving_llama8B_int4_tp2_random_128_128",
880 - "qps_list": ["inf"],
881 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
882 - "server_environment_variables": {
883 - "VLLM_RPC_TIMEOUT": 100000,
884 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
885 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
886 - "VLLM_CPU_SGL_KERNEL": 1,
887 - "VLLM_CPU_KVCACHE_SPACE": 40
888 - },
889 - "server_parameters": {
890 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
891 - "quantization": "awq",
892 - "tensor_parallel_size": 2,
893 - "dtype": "bfloat16",
894 - "distributed_executor_backend": "mp",
895 - "block_size": 128,
896 - "trust_remote_code": "",
897 - "enable_chunked_prefill": "",
898 - "disable_log_stats": "",
899 - "enforce_eager": "",
900 - "max_num_batched_tokens": 2048,
901 - "max_num_seqs": 256,
902 - "load_format": "dummy"
903 - },
904 - "client_parameters": {
905 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
906 - "backend": "vllm",
907 - "dataset_name": "random",
908 - "random-input-len": 128,
909 - "random-output-len": 128,
910 - "ignore-eos": "",
911 - "num_prompts": 1000
912 - }
913 - },
914 - {
915 - "test_name": "serving_llama8B_int4_pp3_random_128_128",
916 - "qps_list": ["inf"],
917 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
918 - "server_environment_variables": {
919 - "VLLM_RPC_TIMEOUT": 100000,
920 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
921 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
922 - "VLLM_CPU_SGL_KERNEL": 1,
923 - "VLLM_CPU_KVCACHE_SPACE": 40
924 - },
925 - "server_parameters": {
926 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
927 - "quantization": "awq",
928 - "pipeline_parallel_size": 3,
929 - "dtype": "bfloat16",
930 - "distributed_executor_backend": "mp",
931 - "block_size": 128,
932 - "trust_remote_code": "",
933 - "enable_chunked_prefill": "",
934 - "disable_log_stats": "",
935 - "enforce_eager": "",
936 - "max_num_batched_tokens": 2048,
937 - "max_num_seqs": 256,
938 - "load_format": "dummy"
939 - },
940 - "client_parameters": {
941 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
942 - "backend": "vllm",
943 - "dataset_name": "random",
944 - "random-input-len": 128,
945 - "random-output-len": 128,
946 - "ignore-eos": "",
947 - "num_prompts": 1000
948 - }
949 - },
950 - {
951 - "test_name": "serving_llama8B_int4_tp4_random_128_128",
952 - "qps_list": ["inf"],
953 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
954 - "server_environment_variables": {
955 - "VLLM_RPC_TIMEOUT": 100000,
956 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
957 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
958 - "VLLM_CPU_SGL_KERNEL": 1,
959 - "VLLM_CPU_KVCACHE_SPACE": 40
960 - },
961 - "server_parameters": {
962 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
963 - "quantization": "awq",
964 - "tensor_parallel_size": 4,
965 - "dtype": "bfloat16",
966 - "distributed_executor_backend": "mp",
967 - "block_size": 128,
968 - "trust_remote_code": "",
969 - "enable_chunked_prefill": "",
970 - "disable_log_stats": "",
971 - "enforce_eager": "",
972 - "max_num_batched_tokens": 2048,
973 - "max_num_seqs": 256,
974 - "load_format": "dummy"
975 - },
976 - "client_parameters": {
977 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
978 - "backend": "vllm",
979 - "dataset_name": "random",
980 - "random-input-len": 128,
981 - "random-output-len": 128,
982 - "ignore-eos": "",
983 - "num_prompts": 1000
984 - }
985 - },
986 - {
987 - "test_name": "serving_llama8B_int4_tp2pp3_random_128_128",
988 - "qps_list": ["inf"],
989 - "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200, 1000],
990 - "server_environment_variables": {
991 - "VLLM_RPC_TIMEOUT": 100000,
992 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
993 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
994 - "VLLM_CPU_SGL_KERNEL": 1,
995 - "VLLM_CPU_KVCACHE_SPACE": 40
996 - },
997 - "server_parameters": {
998 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
999 - "quantization": "awq",
1000 - "tensor_parallel_size": 2,
1001 - "pipeline_parallel_size": 3,
1002 - "dtype": "bfloat16",
1003 - "distributed_executor_backend": "mp",
1004 - "block_size": 128,
1005 - "trust_remote_code": "",
1006 - "enable_chunked_prefill": "",
1007 - "disable_log_stats": "",
1008 - "enforce_eager": "",
1009 - "max_num_batched_tokens": 2048,
1010 - "max_num_seqs": 256,
1011 - "load_format": "dummy"
1012 - },
1013 - "client_parameters": {
1014 - "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
1015 - "backend": "vllm",
1016 - "dataset_name": "random",
1017 - "random-input-len": 128,
1018 - "random-output-len": 128,
1019 - "ignore-eos": "",
1020 - "num_prompts": 1000
1021 - }
1022 - }
1023 -]
.buildkite/performance-benchmarks/tests/serving-tests-cpu.json
+230 -260
@@ -1,276 +1,246 @@
1 -[
1 +{
2 + "defaults": {
3 + "qps_list": [
4 + "inf"
5 + ],
6 + "max_concurrency_list": [12, 16, 24, 32, 64, 128, 200],
7 + "server_environment_variables": {
8 + "VLLM_RPC_TIMEOUT": 100000,
9 + "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
10 + "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
11 + "VLLM_CPU_SGL_KERNEL": 1,
12 + "VLLM_CPU_KVCACHE_SPACE": 40
13 + },
14 + "server_parameters": {
15 + "model": "meta-llama/Llama-3.1-8B-Instruct",
16 + "tensor_parallel_size": 1,
17 + "dtype": "bfloat16",
18 + "distributed_executor_backend": "mp",
19 + "block_size": 128,
20 + "trust_remote_code": "",
21 + "disable_log_stats": "",
22 + "enforce_eager": "",
23 + "max_num_batched_tokens": 2048,
24 + "max_num_seqs": 256,
25 + "load_format": "dummy"
26 + },
27 + "client_parameters": {
28 + "model": "meta-llama/Llama-3.1-8B-Instruct",
29 + "backend": "vllm",
30 + "ignore-eos": "",
31 + "num_prompts": 200
32 + }
33 + },
34 + "tests": [
35 + {
36 + "test_name": "serving_llama8B_tp1_sharegpt",
37 + "server_parameters": {
38 + "tensor_parallel_size": 1
39 + },
40 + "client_parameters": {
41 + "dataset_name": "sharegpt",
42 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json"
43 + }
44 + },
45 + {
46 + "test_name": "serving_llama8B_tp2_sharegpt",
47 + "server_parameters": {
48 + "tensor_parallel_size": 2
49 + },
50 + "client_parameters": {
51 + "dataset_name": "sharegpt",
52 + "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json"
53 + }
54 + },
55 + {
56 + "test_name": "serving_llama8B_tp1_random_128_128",
57 + "server_parameters": {
58 + "tensor_parallel_size": 1
59 + },
60 + "client_parameters": {
61 + "dataset_name": "random",
62 + "random-input-len": 128,
63 + "random-output-len": 128
64 + }
65 + },
66 + {
67 + "test_name": "serving_llama8B_tp2_random_128_128",
68 + "server_parameters": {
69 + "tensor_parallel_size": 2
70 + },
71 + "client_parameters": {
72 + "dataset_name": "random",
73 + "random-input-len": 128,
74 + "random-output-len": 128
75 + }
76 + },
77 + {
78 + "test_name": "serving_llama8B_tp4_random_128_128",
79 + "server_parameters": {
80 + "tensor_parallel_size": 4
81 + },
82 + "client_parameters": {
83 + "dataset_name": "random",
84 + "random-input-len": 128,
85 + "random-output-len": 128
86 + }
87 + },
88 + {
89 + "test_name": "serving_llama8B_tp1_random_128_2048",
90 + "server_parameters": {
91 + "tensor_parallel_size": 1
92 + },
93 + "client_parameters": {
94 + "dataset_name": "random",
95 + "random-input-len": 128,
96 + "random-output-len": 2048
97 + }
98 + },
99 + {
100 + "test_name": "serving_llama8B_tp2_random_128_2048",
101 + "server_parameters": {
102 + "tensor_parallel_size": 2
103 + },
104 + "client_parameters": {
105 + "dataset_name": "random",
106 + "random-input-len": 128,
107 + "random-output-len": 2048
108 + }
109 + },
110 + {
111 + "test_name": "serving_llama8B_tp4_random_128_2048",
112 + "server_parameters": {
113 + "tensor_parallel_size": 4
114 + },
115 + "client_parameters": {
116 + "dataset_name": "random",
117 + "random-input-len": 128,
118 + "random-output-len": 2048
119 + }
120 + },
121 + {
122 + "test_name": "serving_llama8B_tp1_random_2048_128",
123 + "server_parameters": {
124 + "tensor_parallel_size": 1
125 + },
126 + "client_parameters": {
127 + "dataset_name": "random",
128 + "random-input-len": 2048,
129 + "random-output-len": 128
130 + }
131 + },
132 + {
133 + "test_name": "serving_llama8B_tp2_random_2048_128",
134 + "server_parameters": {
135 + "tensor_parallel_size": 2
136 + },
137 + "client_parameters": {
138 + "dataset_name": "random",
139 + "random-input-len": 2048,
140 + "random-output-len": 128
141 + }
142 + },
143 {
3 - "test_name": "serving_llama8B_tp1_sharegpt",
4 - "qps_list": [1, 4, 16, "inf"],
5 - "max_concurrency_list": [32],
6 - "server_environment_variables": {
7 - "VLLM_RPC_TIMEOUT": 100000,
8 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
9 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
10 - "VLLM_CPU_SGL_KERNEL": 1,
11 - "VLLM_CPU_KVCACHE_SPACE": 40
12 - },
13 - "server_parameters": {
14 - "model": "meta-llama/Llama-3.1-8B-Instruct",
15 - "tensor_parallel_size": 1,
16 - "dtype": "bfloat16",
17 - "distributed_executor_backend": "mp",
18 - "block_size": 128,
19 - "trust_remote_code": "",
20 - "disable_log_stats": "",
21 - "enforce_eager": "",
22 - "max_num_batched_tokens": 2048,
23 - "max_num_seqs": 256,
24 - "load_format": "dummy"
25 - },
26 - "client_parameters": {
27 - "model": "meta-llama/Llama-3.1-8B-Instruct",
28 - "backend": "vllm",
29 - "dataset_name": "sharegpt",
30 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
31 - "num_prompts": 32
32 - }
144 + "test_name": "serving_llama8B_tp4_random_2048_128",
145 + "server_parameters": {
146 + "tensor_parallel_size": 4
147 + },
148 + "client_parameters": {
149 + "dataset_name": "random",
150 + "random-input-len": 2048,
151 + "random-output-len": 128
152 + }
153 },
154 {
35 - "test_name": "serving_llama8B_tp2_sharegpt",
36 - "qps_list": [1, 4, 16, "inf"],
37 - "max_concurrency_list": [32],
38 - "server_environment_variables": {
39 - "VLLM_RPC_TIMEOUT": 100000,
40 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
41 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
42 - "VLLM_CPU_SGL_KERNEL": 1,
43 - "VLLM_CPU_KVCACHE_SPACE": 40
44 - },
45 - "server_parameters": {
46 - "model": "meta-llama/Llama-3.1-8B-Instruct",
47 - "tensor_parallel_size": 2,
48 - "dtype": "bfloat16",
49 - "distributed_executor_backend": "mp",
50 - "block_size": 128,
51 - "trust_remote_code": "",
52 - "disable_log_stats": "",
53 - "enforce_eager": "",
54 - "max_num_batched_tokens": 2048,
55 - "max_num_seqs": 256,
56 - "load_format": "dummy"
57 - },
58 - "client_parameters": {
59 - "model": "meta-llama/Llama-3.1-8B-Instruct",
60 - "backend": "vllm",
61 - "dataset_name": "sharegpt",
62 - "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json",
63 - "num_prompts": 32
64 - }
155 + "test_name": "serving_llama3B_tp1_random_128_128",
156 + "server_parameters": {
157 + "model": "meta-llama/Llama-3.2-3B-Instruct",
158 + "tensor_parallel_size": 1
159 + },
160 + "client_parameters": {
161 + "model": "meta-llama/Llama-3.2-3B-Instruct",
162 + "dataset_name": "random",
163 + "random-input-len": 128,
164 + "random-output-len": 128
165 + }
166 },
167 {
67 - "test_name": "serving_llama8B_tp1_random_128_128",
68 - "qps_list": [1, 4, 16, "inf"],
69 - "max_concurrency_list": [32],
70 - "server_environment_variables": {
71 - "VLLM_RPC_TIMEOUT": 100000,
72 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
73 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
74 - "VLLM_CPU_SGL_KERNEL": 1,
75 - "VLLM_CPU_KVCACHE_SPACE": 40
76 - },
77 - "server_parameters": {
78 - "model": "meta-llama/Llama-3.1-8B-Instruct",
79 - "tensor_parallel_size": 1,
80 - "dtype": "bfloat16",
81 - "distributed_executor_backend": "mp",
82 - "block_size": 128,
83 - "trust_remote_code": "",
84 - "enable_chunked_prefill": "",
85 - "disable_log_stats": "",
86 - "enforce_eager": "",
87 - "max_num_batched_tokens": 2048,
88 - "max_num_seqs": 256,
89 - "load_format": "dummy"
90 - },
91 - "client_parameters": {
92 - "model": "meta-llama/Llama-3.1-8B-Instruct",
93 - "backend": "vllm",
94 - "dataset_name": "random",
95 - "random-input-len": 128,
96 - "random-output-len": 128,
97 - "ignore-eos": "",
98 - "num_prompts": 32
99 - }
168 + "test_name": "serving_granite2B_tp1_random_128_128",
169 + "server_parameters": {
170 + "model": "ibm-granite/granite-3.2-2b-instruct",
171 + "tensor_parallel_size": 1
172 + },
173 + "client_parameters": {
174 + "model": "ibm-granite/granite-3.2-2b-instruct",
175 + "dataset_name": "random",
176 + "random-input-len": 128,
177 + "random-output-len": 128
178 + }
179 },
180 {
102 - "test_name": "serving_llama8B_tp2_random_128_128",
103 - "qps_list": [1, 4, 16, "inf"],
104 - "max_concurrency_list": [32],
105 - "server_environment_variables": {
106 - "VLLM_RPC_TIMEOUT": 100000,
107 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
108 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
109 - "VLLM_CPU_SGL_KERNEL": 1,
110 - "VLLM_CPU_KVCACHE_SPACE": 40
111 - },
112 - "server_parameters": {
113 - "model": "meta-llama/Llama-3.1-8B-Instruct",
114 - "tensor_parallel_size": 2,
115 - "dtype": "bfloat16",
116 - "distributed_executor_backend": "mp",
117 - "block_size": 128,
118 - "trust_remote_code": "",
119 - "enable_chunked_prefill": "",
120 - "disable_log_stats": "",
121 - "enforce_eager": "",
122 - "max_num_batched_tokens": 2048,
123 - "max_num_seqs": 256,
124 - "load_format": "dummy"
125 - },
126 - "client_parameters": {
127 - "model": "meta-llama/Llama-3.1-8B-Instruct",
128 - "backend": "vllm",
129 - "dataset_name": "random",
130 - "random-input-len": 128,
131 - "random-output-len": 128,
132 - "ignore-eos": "",
133 - "num_prompts": 32
134 - }
181 + "test_name": "serving_qwen1.7B_tp1_random_128_128",
182 + "server_parameters": {
183 + "model": "Qwen/Qwen3-1.7B",
184 + "tensor_parallel_size": 1
185 + },
186 + "client_parameters": {
187 + "model": "Qwen/Qwen3-1.7B",
188 + "dataset_name": "random",
189 + "random-input-len": 128,
190 + "random-output-len": 128
191 + }
192 },
193 {
137 - "test_name": "serving_llama8B_tp1_random_128_2048",
138 - "qps_list": [1, 4, 16, "inf"],
139 - "max_concurrency_list": [32],
140 - "server_environment_variables": {
141 - "VLLM_RPC_TIMEOUT": 100000,
142 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
143 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
144 - "VLLM_CPU_SGL_KERNEL": 1,
145 - "VLLM_CPU_KVCACHE_SPACE": 40
146 - },
147 - "server_parameters": {
148 - "model": "meta-llama/Llama-3.1-8B-Instruct",
149 - "tensor_parallel_size": 1,
150 - "dtype": "bfloat16",
151 - "distributed_executor_backend": "mp",
152 - "block_size": 128,
153 - "trust_remote_code": "",
154 - "enable_chunked_prefill": "",
155 - "disable_log_stats": "",
156 - "enforce_eager": "",
157 - "max_num_batched_tokens": 2048,
158 - "max_num_seqs": 256,
159 - "load_format": "dummy"
160 - },
161 - "client_parameters": {
162 - "model": "meta-llama/Llama-3.1-8B-Instruct",
163 - "backend": "vllm",
164 - "dataset_name": "random",
165 - "random-input-len": 128,
166 - "random-output-len": 2048,
167 - "ignore-eos": "",
168 - "num_prompts": 32
169 - }
194 + "test_name": "serving_qwen4B_tp1_random_128_128",
195 + "server_parameters": {
196 + "model": "Qwen/Qwen3-4B",
197 + "tensor_parallel_size": 1
198 + },
199 + "client_parameters": {
200 + "model": "Qwen/Qwen3-4B",
201 + "dataset_name": "random",
202 + "random-input-len": 128,
203 + "random-output-len": 128
204 + }
205 },
206 {
172 - "test_name": "serving_llama8B_tp2_random_128_2048",
173 - "qps_list": [1, 4, 16, "inf"],
174 - "max_concurrency_list": [32],
175 - "server_environment_variables": {
176 - "VLLM_RPC_TIMEOUT": 100000,
177 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
178 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
179 - "VLLM_CPU_SGL_KERNEL": 1,
180 - "VLLM_CPU_KVCACHE_SPACE": 40
181 - },
182 - "server_parameters": {
183 - "model": "meta-llama/Llama-3.1-8B-Instruct",
184 - "tensor_parallel_size": 2,
185 - "dtype": "bfloat16",
186 - "distributed_executor_backend": "mp",
187 - "block_size": 128,
188 - "trust_remote_code": "",
189 - "enable_chunked_prefill": "",
190 - "disable_log_stats": "",
191 - "enforce_eager": "",
192 - "max_num_batched_tokens": 2048,
193 - "max_num_seqs": 256,
194 - "load_format": "dummy"
195 - },
196 - "client_parameters": {
197 - "model": "meta-llama/Llama-3.1-8B-Instruct",
198 - "backend": "vllm",
199 - "dataset_name": "random",
200 - "random-input-len": 128,
201 - "random-output-len": 2048,
202 - "ignore-eos": "",
203 - "num_prompts": 32
204 - }
207 + "test_name": "serving_qwen8B_tp1_random_128_128",
208 + "server_parameters": {
209 + "model": "Qwen/Qwen3-8B",
210 + "tensor_parallel_size": 1
211 + },
212 + "client_parameters": {
213 + "model": "Qwen/Qwen3-8B",
214 + "dataset_name": "random",
215 + "random-input-len": 128,
216 + "random-output-len": 128
217 + }
218 },
219 {
207 - "test_name": "serving_llama8B_tp1_random_2048_128",
208 - "qps_list": [1, 4, 16, "inf"],
209 - "max_concurrency_list": [32],
210 - "server_environment_variables": {
211 - "VLLM_RPC_TIMEOUT": 100000,
212 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
213 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
214 - "VLLM_CPU_SGL_KERNEL": 1,
215 - "VLLM_CPU_KVCACHE_SPACE": 40
216 - },
217 - "server_parameters": {
218 - "model": "meta-llama/Llama-3.1-8B-Instruct",
219 - "tensor_parallel_size": 1,
220 - "dtype": "bfloat16",
221 - "distributed_executor_backend": "mp",
222 - "block_size": 128,
223 - "trust_remote_code": "",
224 - "enable_chunked_prefill": "",
225 - "disable_log_stats": "",
226 - "enforce_eager": "",
227 - "max_num_batched_tokens": 2048,
228 - "max_num_seqs": 256,
229 - "load_format": "dummy"
230 - },
231 - "client_parameters": {
232 - "model": "meta-llama/Llama-3.1-8B-Instruct",
233 - "backend": "vllm",
234 - "dataset_name": "random",
235 - "random-input-len": 2048,
236 - "random-output-len": 128,
237 - "ignore-eos": "",
238 - "num_prompts": 32
239 - }
220 + "test_name": "serving_glm9B_tp1_random_128_128",
221 + "server_parameters": {
222 + "model": "zai-org/glm-4-9b-hf",
223 + "tensor_parallel_size": 1
224 + },
225 + "client_parameters": {
226 + "model": "zai-org/glm-4-9b-hf",
227 + "dataset_name": "random",
228 + "random-input-len": 128,
229 + "random-output-len": 128
230 + }
231 },
232 {
242 - "test_name": "serving_llama8B_tp2_random_2048_128",
243 - "qps_list": [1, 4, 16, "inf"],
244 - "max_concurrency_list": [32],
245 - "server_environment_variables": {
246 - "VLLM_RPC_TIMEOUT": 100000,
247 - "VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
248 - "VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
249 - "VLLM_CPU_SGL_KERNEL": 1,
250 - "VLLM_CPU_KVCACHE_SPACE": 40
251 - },
252 - "server_parameters": {
253 - "model": "meta-llama/Llama-3.1-8B-Instruct",
254 - "tensor_parallel_size": 2,
255 - "dtype": "bfloat16",
256 - "distributed_executor_backend": "mp",
257 - "block_size": 128,
258 - "trust_remote_code": "",
259 - "enable_chunked_prefill": "",
260 - "disable_log_stats": "",
261 - "enforce_eager": "",
262 - "max_num_batched_tokens": 2048,
263 - "max_num_seqs": 256,
264 - "load_format": "dummy"
265 - },
266 - "client_parameters": {
267 - "model": "meta-llama/Llama-3.1-8B-Instruct",
268 - "backend": "vllm",
269 - "dataset_name": "random",
270 - "random-input-len": 2048,
271 - "random-output-len": 128,
272 - "ignore-eos": "",
273 - "num_prompts": 32
274 - }
233 + "test_name": "serving_gemma7B_tp1_random_128_128",
234 + "server_parameters": {
235 + "model": "google/gemma-7b",
236 + "tensor_parallel_size": 1
237 + },
238 + "client_parameters": {
239 + "model": "google/gemma-7b",
240 + "dataset_name": "random",
241 + "random-input-len": 128,
242 + "random-output-len": 128
243 + }
244 }
276 -]
245 + ]
246 +}
docs/getting_started/installation/cpu.md
+29
@@ -138,6 +138,35 @@ vllm serve facebook/opt-125m --dtype=bfloat16
138
139 Note, it is recommended to manually reserve 1 CPU for vLLM front-end process when `world_size == 1`.
140
141 +### What are supported models on CPU?
142 +
143 +For the full and up-to-date list of models validated on CPU platforms, please see the official documentation: [Supported Models on CPU](https://docs.vllm.ai/en/latest/models/hardware_supported_models/cpu)
144 +
145 +### How to find benchmark configuration examples for supported CPU models?
146 +
147 +For any model listed under [Supported Models on CPU](https://docs.vllm.ai/en/latest/models/hardware_supported_models/cpu), optimized runtime configurations are provided in the vLLM Benchmark Suite’s CPU test cases, defined in [cpu test cases](https://github.com/vllm-project/vllm/blob/main/.buildkite/performance-benchmarks/tests/serving-tests-cpu.json)
148 +For details on how these optimized configurations are determined, see: [performance-benchmark-details](https://github.com/vllm-project/vllm/tree/main/.buildkite/performance-benchmarks#performance-benchmark-details).
149 +To benchmark the supported models using these optimized settings, follow the steps in [running vLLM Benchmark Suite manually](https://docs.vllm.ai/en/latest/contributing/benchmarks/#manually-trigger-the-benchmark) and run the Benchmark Suite on a CPU environment.
150 +
151 +Below is an example command to benchmark all CPU-supported models using optimized configurations.
152 +
153 +```bash
154 +ON_CPU=1 bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh
155 +```
156 +
157 +The benchmark results will be saved in `./benchmark/results/`.
158 +In the directory, the generated `.commands` files contain all example commands for the benchmark.
159 +
160 +We recommend configuring tensor-parallel-size to match the number of NUMA nodes on your system. Note that the current release does not support tensor-parallel-size=6.
161 +To determine the number of NUMA nodes available, use the following command:
162 +
163 +```bash
164 +lscpu | grep "NUMA node(s):" | awk '{print $3}'
165 +```
166 +
167 +For performance reference, users may also consult the [vLLM Performance Dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm&deviceName=cpu)
168 +, which publishes default-model CPU results produced using the same Benchmark Suite.
169 +
170 ### How to decide `VLLM_CPU_OMP_THREADS_BIND`?
171
172 - Default `auto` thread-binding is recommended for most cases. Ideally, each OpenMP thread will be bound to a dedicated physical core respectively, threads of each rank will be bound to the same NUMA node respectively, and 1 CPU per rank will be reserved for other vLLM components when `world_size > 1`. If you have any performance problems or unexpected binding behaviours, please try to bind threads as following.