@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 65 lines 2.43 KB
Rendered Raw
1 # Performance benchmarks descriptions
2
3 ## Latency tests
4
5 - Input length: 32 tokens.
6 - Output length: 128 tokens.
7 - Batch size: fixed (8).
8 - GPU/HPU Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
9 - CPU Models: llama-3.1 8B.
10 - Evaluation metrics: end-to-end latency (mean, median, p99).
11
12 {latency_tests_markdown_table}
13
14 ## Throughput tests
15
16 - Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
17 - Output length: the corresponding output length of these 200 prompts.
18 - Batch size: dynamically determined by vllm to achieve maximum throughput.
19 - GPU/HPU Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
20 - CPU Models: llama-3.1 8B.
21 - Evaluation metrics: throughput.
22
23 {throughput_tests_markdown_table}
24
25 ## Serving tests
26
27 - Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
28 - Output length: the corresponding output length of these 200 prompts.
29 - Batch size: dynamically determined by vllm and the arrival pattern of the requests.
30 - **Average QPS (query per second)**: 1, 4, 16 and inf. QPS = inf means all requests come at once. For other QPS values, the arrival time of each query is determined using a random Poisson process (with fixed random seed).
31 - GPU/HPU Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
32 - We also added a speculative decoding test for llama-3 70B on GPU, under QPS 2
33 - CPU Models: llama-3.1 8B.
34 - Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).
35 - For CPU, we added random dataset tests to benchmark fixed input/output length with 100 prompts.
36
37 {serving_tests_markdown_table}
38
39 ## Platform Information
40
41 {platform_markdown_table}
42
43 ## json version of the benchmarking tables
44
45 This section contains the data of the markdown tables above in JSON format.
46 You can load the benchmarking tables into pandas dataframes as follows:
47
48 ```python
49 import json
50 import pandas as pd
51
52 benchmarking_results_json = """The json string"""
53 benchmarking_results = json.loads(benchmarking_results_json)
54 latency_results = pd.DataFrame.from_dict(benchmarking_results["latency"])
55 throughput_results = pd.DataFrame.from_dict(benchmarking_results["throughput"])
56 serving_results = pd.DataFrame.from_dict(benchmarking_results["serving"])
57 ```
58
59 The json string for all benchmarking tables:
60
61 ```json
62 {benchmarking_results_in_json_string}
63 ```
64
65 You can also check the raw experiment data in the Artifact tab of the Buildkite page.