@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 8a69e0e20e
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 23:30 UTC

[CI/Build] Auto-fix Markdown files (#12941)

Cyrus Leung committed Feb 8, 2025 at 20:25 UTC 8a69e0e20e72d429aaf379ae7647f0434a0e9c9e
20 files changed +158 -141
.buildkite/nightly-benchmarks/README.md
+18 -28
@@ -1,15 +1,13 @@
1 # vLLM benchmark suite
2
3 -
3 ## Introduction
4
5 This directory contains two sets of benchmark for vllm.
6 +
7 - Performance benchmark: benchmark vllm's performance under various workload, for **developers** to gain clarity on whether their PR improves/degrades vllm's performance
8 - Nightly benchmark: compare vllm's performance against alternatives (tgi, trt-llm and lmdeploy), for **the public** to know when to choose vllm.
9
10 -
11 -See [vLLM performance dashboard](https://perf.vllm.ai) for the latest performance benchmark results and [vLLM GitHub README](https://github.com/vllm-project/vllm/blob/main/README.md) for latest nightly benchmark results.
12 -
10 +See [vLLM performance dashboard](https://perf.vllm.ai) for the latest performance benchmark results and [vLLM GitHub README](https://github.com/vllm-project/vllm/blob/main/README.md) for latest nightly benchmark results.
11
12 ## Performance benchmark quick overview
13
@@ -19,17 +17,14 @@ See [vLLM performance dashboard](https://perf.vllm.ai) for the latest performan
17
18 **For benchmarking developers**: please try your best to constraint the duration of benchmarking to about 1 hr so that it won't take forever to run.
19
22 -
20 ## Nightly benchmark quick overview
21
25 -**Benchmarking Coverage**: Fix-qps serving on A100 (the support for FP8 benchmark on H100 is coming!) on Llama-3 8B, 70B and Mixtral 8x7B.
22 +**Benchmarking Coverage**: Fix-qps serving on A100 (the support for FP8 benchmark on H100 is coming!) on Llama-3 8B, 70B and Mixtral 8x7B.
23
24 **Benchmarking engines**: vllm, TGI, trt-llm and lmdeploy.
25
26 **Benchmarking Duration**: about 3.5hrs.
27
31 -
32 -
28 ## Trigger the benchmark
29
30 Performance benchmark will be triggered when:
@@ -39,16 +34,11 @@ Performance benchmark will be triggered when:
34 Nightly benchmark will be triggered when:
35 - Every commit for those PRs with `perf-benchmarks` label and `nightly-benchmarks` label.
36
42 -
43 -
44 -
37 ## Performance benchmark details
38
47 -
39 See [performance-benchmarks-descriptions.md](performance-benchmarks-descriptions.md) for detailed descriptions, and use `tests/latency-tests.json`, `tests/throughput-tests.json`, `tests/serving-tests.json` to configure the test cases.
40
50 -
51 -#### Latency test
41 +### Latency test
42
43 Here is an example of one test inside `latency-tests.json`:
44
@@ -68,23 +58,25 @@ Here is an example of one test inside `latency-tests.json`:
58 ```
59
60 In this example:
71 -- The `test_name` attributes is a unique identifier for the test. In `latency-tests.json`, it must start with `latency_`.
72 -- The `parameters` attribute control the command line arguments to be used for `benchmark_latency.py`. Note that please use underline `_` instead of the dash `-` when specifying the command line arguments, and `run-performance-benchmarks.sh` will convert the underline to dash when feeding the arguments to `benchmark_latency.py`. For example, the corresponding command line arguments for `benchmark_latency.py` will be `--model meta-llama/Meta-Llama-3-8B --tensor-parallel-size 1 --load-format dummy --num-iters-warmup 5 --num-iters 15`
61 +
62 +- The `test_name` attributes is a unique identifier for the test. In `latency-tests.json`, it must start with `latency_`.
63 +- The `parameters` attribute control the command line arguments to be used for `benchmark_latency.py`. Note that please use underline `_` instead of the dash `-` when specifying the command line arguments, and `run-performance-benchmarks.sh` will convert the underline to dash when feeding the arguments to `benchmark_latency.py`. For example, the corresponding command line arguments for `benchmark_latency.py` will be `--model meta-llama/Meta-Llama-3-8B --tensor-parallel-size 1 --load-format dummy --num-iters-warmup 5 --num-iters 15`
64
65 Note that the performance numbers are highly sensitive to the value of the parameters. Please make sure the parameters are set correctly.
66
67 WARNING: The benchmarking script will save json results by itself, so please do not configure `--output-json` parameter in the json file.
68
69 +### Throughput test
70
79 -#### Throughput test
71 The tests are specified in `throughput-tests.json`. The syntax is similar to `latency-tests.json`, except for that the parameters will be fed forward to `benchmark_throughput.py`.
72
73 The number of this test is also stable -- a slight change on the value of this number might vary the performance numbers by a lot.
74
84 -#### Serving test
75 +### Serving test
76 +
77 We test the throughput by using `benchmark_serving.py` with request rate = inf to cover the online serving overhead. The corresponding parameters are in `serving-tests.json`, and here is an example:
78
87 -```
79 +```json
80 [
81 {
82 "test_name": "serving_llama8B_tp1_sharegpt",
@@ -109,6 +101,7 @@ We test the throughput by using `benchmark_serving.py` with request rate = inf t
101 ```
102
103 Inside this example:
104 +
105 - The `test_name` attribute is also a unique identifier for the test. It must start with `serving_`.
106 - The `server-parameters` includes the command line arguments for vLLM server.
107 - The `client-parameters` includes the command line arguments for `benchmark_serving.py`.
@@ -118,36 +111,33 @@ The number of this test is less stable compared to the delay and latency benchma
111
112 WARNING: The benchmarking script will save json results by itself, so please do not configure `--save-results` or other results-saving-related parameters in `serving-tests.json`.
113
121 -#### Visualizing the results
114 +### Visualizing the results
115 +
116 The `convert-results-json-to-markdown.py` helps you put the benchmarking results inside a markdown table, by formatting [descriptions.md](tests/descriptions.md) with real benchmarking results.
117 You can find the result presented as a table inside the `buildkite/performance-benchmark` job page.
118 If you do not see the table, please wait till the benchmark finish running.
119 The json version of the table (together with the json version of the benchmark) will be also attached to the markdown file.
120 The raw benchmarking results (in the format of json files) are in the `Artifacts` tab of the benchmarking.
121
128 -
129 -
122 ## Nightly test details
123
124 See [nightly-descriptions.md](nightly-descriptions.md) for the detailed description on test workload, models and docker containers of benchmarking other llm engines.
125
126 +### Workflow
127
135 -#### Workflow
136 -
137 -- The [nightly-pipeline.yaml](nightly-pipeline.yaml) specifies the docker containers for different LLM serving engines.
128 +- The [nightly-pipeline.yaml](nightly-pipeline.yaml) specifies the docker containers for different LLM serving engines.
129 - Inside each container, we run [run-nightly-suite.sh](run-nightly-suite.sh), which will probe the serving engine of the current container.
130 - The `run-nightly-suite.sh` will redirect the request to `tests/run-[llm serving engine name]-nightly.sh`, which parses the workload described in [nightly-tests.json](tests/nightly-tests.json) and performs the benchmark.
131 - At last, we run [scripts/plot-nightly-results.py](scripts/plot-nightly-results.py) to collect and plot the final benchmarking results, and update the results to buildkite.
132
142 -#### Nightly tests
133 +### Nightly tests
134
135 In [nightly-tests.json](tests/nightly-tests.json), we include the command line arguments for benchmarking commands, together with the benchmarking test cases. The format is highly similar to performance benchmark.
136
146 -#### Docker containers
137 +### Docker containers
138
139 The docker containers for benchmarking are specified in `nightly-pipeline.yaml`.
140
141 WARNING: the docker versions are HARD-CODED and SHOULD BE ALIGNED WITH `nightly-descriptions.md`. The docker versions need to be hard-coded as there are several version-specific bug fixes inside `tests/run-[llm serving engine name]-nightly.sh`.
142
143 WARNING: populating `trt-llm` to latest version is not easy, as it requires updating several protobuf files in [tensorrt-demo](https://github.com/neuralmagic/tensorrt-demo.git).
153 -
.buildkite/nightly-benchmarks/nightly-annotation.md
+10 -11
@@ -9,20 +9,19 @@ This file contains the downloading link for benchmarking results.
9
10 Please download the visualization scripts in the post
11
12 -
12 ## Results reproduction
13
14 - Find the docker we use in `benchmarking pipeline`
15 - Deploy the docker, and inside the docker:
17 - - Download `nightly-benchmarks.zip`.
18 - - In the same folder, run the following code
19 -```
20 -export HF_TOKEN=<your HF token>
21 -apt update
22 -apt install -y git
23 -unzip nightly-benchmarks.zip
24 -VLLM_SOURCE_CODE_LOC=./ bash .buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
25 -```
16 + - Download `nightly-benchmarks.zip`.
17 + - In the same folder, run the following code:
18
27 -And the results will be inside `./benchmarks/results`.
19 + ```console
20 + export HF_TOKEN=<your HF token>
21 + apt update
22 + apt install -y git
23 + unzip nightly-benchmarks.zip
24 + VLLM_SOURCE_CODE_LOC=./ bash .buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh
25 + ```
26
27 +And the results will be inside `./benchmarks/results`.
.buildkite/nightly-benchmarks/nightly-descriptions.md
+3 -3
@@ -2,6 +2,7 @@
2 # Nightly benchmark
3
4 This benchmark aims to:
5 +
6 - Provide performance clarity: Provide clarity on which one (vllm, tensorrt-llm, lmdeploy and SGLang) leads in performance in what workload.
7 - Be reproducible: one can run the exact same set of benchmarking commands inside the exact same docker by following reproducing instructions.
8
@@ -9,7 +10,6 @@ Latest results: [results link](https://blog.vllm.ai/2024/09/05/perf-update.html)
10
11 Latest reproduction guilde: [github issue link](https://github.com/vllm-project/vllm/issues/8176)
12
12 -
13 ## Setup
14
15 - Docker images:
@@ -33,7 +33,7 @@ Latest reproduction guilde: [github issue link](https://github.com/vllm-project/
33 - Queries are randomly sampled, and arrival patterns are determined via Poisson process, but all with fixed random seed.
34 - Evaluation metrics: Throughput (higher the better), TTFT (time to the first token, lower the better), ITL (inter-token latency, lower the better).
35
36 -# Known issues
36 +## Known issues
37
38 - TRT-LLM crashes with Llama 3.1 8B [issue](https://github.com/NVIDIA/TensorRT-LLM/issues/2105).
39 -- TGI does not support `ignore-eos` flag.
\ No newline at end of file
39 +- TGI does not support `ignore-eos` flag.
.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md
+2 -8
@@ -7,10 +7,8 @@
7 - Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
8 - Evaluation metrics: end-to-end latency (mean, median, p99).
9
10 -
10 {latency_tests_markdown_table}
11
13 -
12 ## Throughput tests
13
14 - Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
@@ -19,10 +17,8 @@
17 - Models: llama-3.1 8B, llama-3 70B, mixtral 8x7B.
18 - Evaluation metrics: throughput.
19
22 -
20 {throughput_tests_markdown_table}
21
25 -
22 ## Serving tests
23
24 - Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed).
@@ -33,13 +29,11 @@
29 - We also added a speculative decoding test for llama-3 70B, under QPS 2
30 - Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99).
31
36 -
32 {serving_tests_markdown_table}
33
39 -
34 ## json version of the benchmarking tables
35
42 -This section contains the data of the markdown tables above in JSON format.
36 +This section contains the data of the markdown tables above in JSON format.
37 You can load the benchmarking tables into pandas dataframes as follows:
38
39 ```python
@@ -54,9 +48,9 @@ serving_results = pd.DataFrame.from_dict(benchmarking_results["serving"])
48 ```
49
50 The json string for all benchmarking tables:
51 +
52 ```json
53 {benchmarking_results_in_json_string}
54 ```
55
56 You can also check the raw experiment data in the Artifact tab of the Buildkite page.
62 -
.github/PULL_REQUEST_TEMPLATE.md
+2 -1
@@ -2,4 +2,5 @@ FILL IN THE PR DESCRIPTION HERE
2
3 FIX #xxxx (*link existing issues this PR will resolve*)
4
5 -**BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing/overview.html **
5 +<!--- pyml disable-next-line no-emphasis-as-heading -->
6 +**BEFORE SUBMITTING, PLEASE READ <https://docs.vllm.ai/en/latest/contributing/overview.html>**
.pre-commit-config.yaml
+1 -1
@@ -33,7 +33,7 @@ repos:
33 rev: v0.9.27
34 hooks:
35 - id: pymarkdown
36 - files: docs/.*
36 + args: [fix]
37 - repo: https://github.com/rhysd/actionlint
38 rev: v1.7.7
39 hooks:
CODE_OF_CONDUCT.md
-1
@@ -125,4 +125,3 @@ Community Impact Guidelines were inspired by
125 For answers to common questions about this code of conduct, see the
126 [Contributor Covenant FAQ](https://www.contributor-covenant.org/faq). Translations are available at
127 [Contributor Covenant translations](https://www.contributor-covenant.org/translations).
128 -
README.md
+9 -5
@@ -16,6 +16,7 @@ Easy, fast, and cheap LLM serving for everyone
16 ---
17
18 *Latest News* 🔥
19 +
20 - [2025/01] We are excited to announce the alpha release of vLLM V1: A major architectural upgrade with 1.7x speedup! Clean code, optimized execution loop, zero-overhead prefix caching, enhanced multimodal support, and more. Please check out our blog post [here](https://blog.vllm.ai/2025/01/27/v1-alpha-release.html).
21 - [2025/01] We hosted [the eighth vLLM meetup](https://lu.ma/zep56hui) with Google Cloud! Please find the meetup slides from vLLM team [here](https://docs.google.com/presentation/d/1epVkt4Zu8Jz_S5OhEHPc798emsYh2BwYfRuDDVEF7u4/edit?usp=sharing), and Google Cloud team [here](https://drive.google.com/file/d/1h24pHewANyRL11xy5dXUbvRC9F9Kkjix/view?usp=sharing).
22 - [2024/12] vLLM joins [pytorch ecosystem](https://pytorch.org/blog/vllm-joins-pytorch)! Easy, Fast, and Cheap LLM Serving for Everyone!
@@ -33,7 +34,9 @@ Easy, fast, and cheap LLM serving for everyone
34 - [2023/06] We officially released vLLM! FastChat-vLLM integration has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid-April. Check out our [blog post](https://vllm.ai).
35
36 ---
37 +
38 ## About
39 +
40 vLLM is a fast and easy-to-use library for LLM inference and serving.
41
42 Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.
@@ -127,6 +130,7 @@ We also have an official fundraising venue through [OpenCollective](https://open
130 ## Citation
131
132 If you use vLLM for your research, please cite our [paper](https://arxiv.org/abs/2309.06180):
133 +
134 ```bibtex
135 @inproceedings{kwon2023efficient,
136 title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
@@ -138,11 +142,11 @@ If you use vLLM for your research, please cite our [paper](https://arxiv.org/abs
142
143 ## Contact Us
144
141 -* For technical questions and feature requests, please use Github issues or discussions.
142 -* For discussing with fellow users and coordinating contributions and development, please use Slack.
143 -* For security disclosures, please use Github's security advisory feature.
144 -* For collaborations and partnerships, please contact us at vllm-questions AT lists.berkeley.edu.
145 +- For technical questions and feature requests, please use Github issues or discussions.
146 +- For discussing with fellow users and coordinating contributions and development, please use Slack.
147 +- For security disclosures, please use Github's security advisory feature.
148 +- For collaborations and partnerships, please contact us at vllm-questions AT lists.berkeley.edu.
149
150 ## Media Kit
151
148 -* If you wish to use vLLM's logo, please refer to [our media kit repo](https://github.com/vllm-project/media-kit).
152 +- If you wish to use vLLM's logo, please refer to [our media kit repo](https://github.com/vllm-project/media-kit).
benchmarks/README.md
+2
@@ -3,6 +3,7 @@
3 ## Downloading the ShareGPT dataset
4
5 You can download the dataset by running:
6 +
7 ```bash
8 wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
9 ```
@@ -11,6 +12,7 @@ wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/r
12
13 The json file refers to several image datasets (coco, llava, etc.). The benchmark scripts
14 will ignore a datapoint if the referred image is missing.
15 +
16 ```bash
17 wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json
18 mkdir coco -p
csrc/quantization/cutlass_w8a8/Epilogues.md
+32 -12
@@ -1,17 +1,19 @@
1 # CUTLASS Epilogues
2
3 ## Introduction
4 -This document describes the various CUTLASS epilogues implemented for fusing de-quantization operations onto GEMMs.
4 +
5 +This document describes the various CUTLASS epilogues implemented for fusing de-quantization operations onto GEMMs.
6
7 Currently, we only support symmetric quantization for weights,
8 and symmetric and asymmetric quantization for activations.
9 Both can be quantized per-tensor or per-channel (weights) / per-token (activations).
10
11 There are 4 epilogues:
11 -1. ScaledEpilogue: symmetric quantization for activations, no bias.
12 -1. ScaledEpilogueBias: symmetric quantization for activations, supports bias.
13 -1. ScaledEpilogueAzp: asymmetric per-tensor quantization for activations, supports bias.
14 -1. ScaledEpilogueAzpPerToken: asymmetric per-token quantization for activations, supports bias.
12 +
13 +1. `ScaledEpilogue`: symmetric quantization for activations, no bias.
14 +1. `ScaledEpilogueBias`: symmetric quantization for activations, supports bias.
15 +1. `ScaledEpilogueAzp`: asymmetric per-tensor quantization for activations, supports bias.
16 +1. `ScaledEpilogueAzpPerToken`: asymmetric per-token quantization for activations, supports bias.
17
18 We do not have epilogues for asymmetric quantization of activations without bias in order to reduce final binary size.
19 Instead, if no bias is passed, the epilogue will use 0 as the bias.
@@ -26,12 +28,15 @@ If $` \widehat X `$ is the quantized $` X `$, our matrices become the following
28 ```math
29 A = s_a (\widehat A - J_a z_a)
30 ```
31 +
32 ```math
33 B = s_b \widehat B
34 ```
35 +
36 ```math
37 D = A B + C
38 ```
39 +
40 ```math
41 D = s_a s_b \widehat D + C
42 ```
@@ -48,9 +53,11 @@ Expanding further, we can calculate $` \widehat D `$ as follows:
53 ```math
54 A B = s_a ( \widehat A - J_a z_a ) s_b \widehat B
55 ```
56 +
57 ```math
58 A B = s_a s_b \left( \widehat A \widehat B - J_a z_a \widehat B \right)
59 ```
60 +
61 ```math
62 \widehat D = \widehat A \widehat B - z_a J_a \widehat B
63 ```
@@ -61,16 +68,19 @@ Each row of it is equal to $` \mathbf 1 \widehat B `$, which is a row-vector of
68
69 ## Epilogues
70
64 -### ScaledEpilogue
71 +### `ScaledEpilogue`
72 +
73 This epilogue computes the symmetric quantization for activations without bias, meaning $` C = 0 `$ and $` z_a = 0 `$.
74 The output of the GEMM is:
75
76 ```math
77 \widehat D = \widehat A \widehat B
78 ```
79 +
80 ```math
81 D = s_a s_b \widehat D
82 ```
83 +
84 ```math
85 D = s_a s_b \widehat A \widehat B
86 ```
@@ -79,44 +89,51 @@ Epilogue parameters:
89 - `scale_a` is the scale for activations, can be per-tensor (scalar) or per-token (column-vector).
90 - `scale_b` is the scale for weights, can be per-tensor (scalar) or per-channel (row-vector).
91
82 -### ScaledEpilogueBias
92 +### `ScaledEpilogueBias`
93 +
94 This epilogue computes the symmetric quantization for activations with bias, meaning $` z_a = 0 `$.
95 The output of the GEMM is:
96
97 ```math
98 \widehat D = \widehat A \widehat B
99 ```
100 +
101 ```math
102 D = s_a s_b \widehat D + C
103 ```
104 +
105 ```math
106 D = s_a s_b \widehat A \widehat B + C
107 ```
108
96 -
109 Epilogue parameters:
110 +
111 - `scale_a` is the scale for activations, can be per-tensor (scalar) or per-token (column-vector).
112 - `scale_b` is the scale for weights, can be per-tensor (scalar) or per-channel (row-vector).
113 - `bias` is the bias, is always per-channel (row-vector).
114
102 -### ScaledEpilogueAzp
115 +### `ScaledEpilogueAzp`
116 +
117 This epilogue computes the asymmetric per-tensor quantization for activations with bias.
118 The output of the GEMM is:
119
120 ```math
121 \widehat D = \widehat A \widehat B - z_a J_a \widehat B
122 ```
123 +
124 ```math
125 D = s_a s_b \widehat D + C
126 ```
127 +
128 ```math
129 D = s_a s_b \left( \widehat A \widehat B - z_a J_a \widehat B \right) + C
130 ```
131
116 -Because $` z_a `$ is a scalar, the zero-point term $` z_a J_a \widehat B `$ has every row equal to $` z_a \mathbf 1 B `$.
132 +Because $` z_a `$ is a scalar, the zero-point term $` z_a J_a \widehat B `$ has every row equal to $` z_a \mathbf 1 B `$.
133 That is precomputed and stored in `azp_with_adj` as a row-vector.
134
135 Epilogue parameters:
136 +
137 - `scale_a` is the scale for activations, can be per-tensor (scalar) or per-token (column-vector).
138 - Generally this will be per-tensor as the zero-points are per-tensor.
139 - `scale_b` is the scale for weights, can be per-tensor (scalar) or per-channel (row-vector).
@@ -125,13 +142,15 @@ Epilogue parameters:
142
143 To use these kernels efficiently, users must precompute the `azp_with_adj` term offline and pass it to the kernel.
144
128 -### ScaledEpilogueAzpPerToken
145 +### `ScaledEpilogueAzpPerToken`
146 +
147 This epilogue computes the asymmetric per-token quantization for activations with bias.
148
149 The output of the GEMM is the same as above, but the $` z_a `$ is a column-vector.
150 That means the zero-point term $` z_a J_a \widehat B `$ becomes an outer product of $` z_a `$ and $` \mathbf 1 \widehat B `$.
151
152 Epilogue parameters:
153 +
154 - `scale_a` is the scale for activations, can be per-tensor (scalar) or per-token (column-vector).
155 - Generally this will be per-token as the zero-points are per-token.
156 - `scale_b` is the scale for weights, can be per-tensor (scalar) or per-channel (row-vector).
@@ -142,6 +161,7 @@ Epilogue parameters:
161 To use these kernels efficiently, users must precompute the `azp_adj` term offline and pass it to the kernel.
162
163 The epilogue performs the following computation (where `Dq` is the raw quantized output of the GEMM):
145 -```
164 +
165 +```math
166 out = scale_a * scale_b * (Dq - azp_adj * azp) + bias
167 ```
csrc/quantization/machete/Readme.md
+7 -7
@@ -6,25 +6,25 @@ Machete is a spiritual successor to the Marlin kernel but optimized for Hopper a
6
7 Machete effectively performs
8
9 -```
9 +```python
10 scale_type = w_s.dtype
11 compute_type = a.dtype
12 out = (w_q.to(scale_type) * w_s - w_z.to(scale_type)) @ a
13 ```
14
15 -Where `w_q` is a quantized weight matrix, `w_s` is the quantization scales, and
15 +Where `w_q` is a quantized weight matrix, `w_s` is the quantization scales, and
16 `w_z` is the quantization zeropoints.
17
18 -> **_NOTE:_** `w_z` is added after the scales so we can
18 +> **_NOTE:_** `w_z` is added after the scales so we can
19 use FMA operations, but this means they must have the scales pre-applied if the
20 -supplied zeropoints assume that they will be subtracted before the scales are
20 +supplied zeropoints assume that they will be subtracted before the scales are
21 applied.
22
23 ## API
24
25 The main optimization within Machete is prepacking the weight matrix to more closely match the tensor core layouts, allowing for wider shared memory loads when loading the weight matrix. This means that the weight matrix must be prepacked before calling `machete_gemm`. The flow looks something like:
26
27 -```
27 +```python
28 from vllm import _custom_ops as ops
29
30 ...
@@ -40,6 +40,6 @@ output = ops.machete_gemm(
40
41 ## Code Generation
42
43 -Since Machete is based on Cutlass, we can generate multiple type pairs and different tile shapes using the same kernel template. We generate multiple instantiations of this template using `generate.py`.
43 +Since Machete is based on Cutlass, we can generate multiple type pairs and different tile shapes using the same kernel template. We generate multiple instantiations of this template using `generate.py`.
44
45 -New type pairs (`TypeConfig`s) can be appended to `impl_configs` (in `generate()`), and these will get automatically generated (assuming they can be supported without issues). For each `TypeConfig`, you must also provide an `ImplConfig`, which bundles a `TypeConfig` with a list of `ScheduleConfig`s, `Specialization`s, and a default heuristic. The `ScheduleConfig`s (which contain info on tile shapes, tile scheduler, etc.) can perform differently for different problem shapes, and there is almost never one `ScheduleConfig` that works well for all problem shapes, so it is generally beneficial to generate different `ScheduleConfig`s for different potential problem shapes. This is where the heuristic comes in. For each `TypeConfig`, a default heuristic should be provided. This maps different problem shapes to different `ScheduleConfig`s and is used when the user does not provide the `schedule` parameter to `machete_gemm`. The `Specialization`s define what feature combinations to generate, i.e., `with_zeropoints`, `with_scales`, etc. We can reduce compile times and the final binary size by limiting the set of feature combinations we generate.
\ No newline at end of file
45 +New type pairs (`TypeConfig`s) can be appended to `impl_configs` (in `generate()`), and these will get automatically generated (assuming they can be supported without issues). For each `TypeConfig`, you must also provide an `ImplConfig`, which bundles a `TypeConfig` with a list of `ScheduleConfig`s, `Specialization`s, and a default heuristic. The `ScheduleConfig`s (which contain info on tile shapes, tile scheduler, etc.) can perform differently for different problem shapes, and there is almost never one `ScheduleConfig` that works well for all problem shapes, so it is generally beneficial to generate different `ScheduleConfig`s for different potential problem shapes. This is where the heuristic comes in. For each `TypeConfig`, a default heuristic should be provided. This maps different problem shapes to different `ScheduleConfig`s and is used when the user does not provide the `schedule` parameter to `machete_gemm`. The `Specialization`s define what feature combinations to generate, i.e., `with_zeropoints`, `with_scales`, etc. We can reduce compile times and the final binary size by limiting the set of feature combinations we generate.
docs/source/getting_started/installation/gpu/rocm.inc.md
+4 -5
@@ -93,12 +93,11 @@ Currently, there are no pre-built ROCm wheels.
93
94 This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm installation.
95
96 -<!--- pyml disable-num-lines 5 ul-indent-->
96 :::{tip}
98 - - Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.
99 - - Triton flash attention does not currently support sliding window attention. If using half precision, please use CK flash-attention for sliding window support.
100 - - To use CK flash-attention or PyTorch naive attention, please use this flag `export VLLM_USE_TRITON_FLASH_ATTN=0` to turn off triton flash attention.
101 - - The ROCm version of PyTorch, ideally, should match the ROCm driver version.
97 + - Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.
98 + - Triton flash attention does not currently support sliding window attention. If using half precision, please use CK flash-attention for sliding window support.
99 + - To use CK flash-attention or PyTorch naive attention, please use this flag `export VLLM_USE_TRITON_FLASH_ATTN=0` to turn off triton flash attention.
100 + - The ROCm version of PyTorch, ideally, should match the ROCm driver version.
101 :::
102
103 :::{tip}
docs/source/serving/engine_args.md
+2 -2
@@ -4,7 +4,7 @@
4
5 Below, you can find an explanation of every engine argument for vLLM:
6
7 -<!--- pyml disable-num-lines 7 no-space-in-emphasis-->
7 +<!--- pyml disable-num-lines 7 no-space-in-emphasis -->
8 ```{eval-rst}
9 .. argparse::
10 :module: vllm.engine.arg_utils
@@ -17,7 +17,7 @@ Below, you can find an explanation of every engine argument for vLLM:
17
18 Below are the additional arguments related to the asynchronous engine:
19
20 -<!--- pyml disable-num-lines 7 no-space-in-emphasis-->
20 +<!--- pyml disable-num-lines 7 no-space-in-emphasis -->
21 ```{eval-rst}
22 .. argparse::
23 :module: vllm.engine.arg_utils
examples/offline_inference/openai/openai_batch.md
+29 -30
@@ -5,50 +5,49 @@ This is a guide to performing batch inference using the OpenAI batch file format
5 ```
6
7 ## File Format
8 -
8 +
9 The OpenAI batch file format consists of a series of json objects on new lines.
10 -
10 +
11 [See here for an example file.](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/openai/openai_example_batch.jsonl)
12 -
12 +
13 Each line represents a separate request. See the [OpenAI package reference](https://platform.openai.com/docs/api-reference/batch/requestInput) for more details.
14 -
14 +
15 ```{note}
16 We currently support `/v1/chat/completions`, `/v1/embeddings`, and `/v1/score` endpoints (completions coming soon).
17 ```
18 -
18 +
19 ## Pre-requisites
20
21 * The examples in this document use `meta-llama/Meta-Llama-3-8B-Instruct`.
22 - Create a [user access token](https://huggingface.co/docs/hub/en/security-tokens)
23 - Install the token on your machine (Run `huggingface-cli login`).
24 - Get access to the gated model by [visiting the model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and agreeing to the terms and conditions.
25 -
26 -
25 +
26 ## Example 1: Running with a local file
27
28 ### Step 1: Create your batch file
29
30 To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
31
33 -```
32 +```console
33 wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai/openai_example_batch.jsonl
34 ```
35
36 Once you've created your batch file it should look like this
37
39 -```
38 +```console
39 $ cat offline_inference/openai/openai_example_batch.jsonl
40 {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
41 {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
42 ```
43
44 ### Step 2: Run the batch
46 -
45 +
46 The batch running tool is designed to be used from the command line.
47
48 You can run the batch with the following command, which will write its results to a file called `results.jsonl`
49
51 -```
50 +```console
51 python -m vllm.entrypoints.openai.run_batch -i offline_inference/openai/openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
52 ```
53
@@ -56,7 +55,7 @@ python -m vllm.entrypoints.openai.run_batch -i offline_inference/openai/openai_e
55
56 You should now have your results at `results.jsonl`. You can check your results by running `cat results.jsonl`
57
59 -```
58 +```console
59 $ cat results.jsonl
60 {"id":"vllm-383d1c59835645aeb2e07d004d62a826","custom_id":"request-1","response":{"id":"cmpl-61c020e54b964d5a98fa7527bfcdd378","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! It's great to meet you! I'm here to help with any questions or tasks you may have. What's on your mind today?"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":25,"total_tokens":56,"completion_tokens":31}},"error":null}
61 {"id":"vllm-42e3d09b14b04568afa3f1797751a267","custom_id":"request-2","response":{"id":"cmpl-f44d049f6b3a42d4b2d7850bb1e31bcc","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"*silence*"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":27,"total_tokens":32,"completion_tokens":5}},"error":null}
@@ -68,7 +67,7 @@ The batch runner supports remote input and output urls that are accessible via h
67
68 For example, to run against our example input file located at `https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai/openai_example_batch.jsonl`, you can run
69
71 -```
70 +```console
71 python -m vllm.entrypoints.openai.run_batch -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai/openai_example_batch.jsonl -o results.jsonl --model meta-llama/Meta-Llama-3-8B-Instruct
72 ```
73
@@ -80,7 +79,7 @@ To integrate with cloud blob storage, we recommend using presigned urls.
79
80 ### Additional prerequisites
81
83 -* [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
82 +* [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html).
83 * The `awscli` package (Run `pip install awscli`) to configure your credentials and interactively use s3.
84 - [Configure your credentials](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).
85 * The `boto3` python package (Run `pip install boto3`) to generate presigned urls.
@@ -89,13 +88,13 @@ To integrate with cloud blob storage, we recommend using presigned urls.
88
89 To follow along with this example, you can download the example batch, or create your own batch file in your working directory.
90
92 -```
91 +```console
92 wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai/openai_example_batch.jsonl
93 ```
94
95 Once you've created your batch file it should look like this
96
98 -```
97 +```console
98 $ cat offline_inference/openai/openai_example_batch.jsonl
99 {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
100 {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}
@@ -103,7 +102,7 @@ $ cat offline_inference/openai/openai_example_batch.jsonl
102
103 Now upload your batch file to your S3 bucket.
104
106 -```
105 +```console
106 aws s3 cp offline_inference/openai/openai_example_batch.jsonl s3://MY_BUCKET/MY_INPUT_FILE.jsonl
107 ```
108
@@ -111,9 +110,9 @@ aws s3 cp offline_inference/openai/openai_example_batch.jsonl s3://MY_BUCKET/MY_
110
111 Presigned urls can only be generated via the SDK. You can run the following python script to generate your presigned urls. Be sure to replace the `MY_BUCKET`, `MY_INPUT_FILE.jsonl`, and `MY_OUTPUT_FILE.jsonl` placeholders with your bucket and file names.
112
114 -(The script is adapted from https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/s3/s3_basics/presigned_url.py)
113 +(The script is adapted from <https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/s3/s3_basics/presigned_url.py>)
114
116 -```
115 +```python
116 import boto3
117 from botocore.exceptions import ClientError
118
@@ -149,7 +148,7 @@ print(f"{output_url=}")
148
149 This script should output
150
152 -```
151 +```text
152 input_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091'
153 output_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091'
154 ```
@@ -158,7 +157,7 @@ output_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AW
157
158 You can now run the batch runner, using the urls generated in the previous section.
159
161 -```
160 +```console
161 python -m vllm.entrypoints.openai.run_batch \
162 -i "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
163 -o "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \
@@ -169,7 +168,7 @@ python -m vllm.entrypoints.openai.run_batch \
168
169 Your results are now on S3. You can view them in your terminal by running
170
172 -```
171 +```console
172 aws s3 cp s3://MY_BUCKET/MY_OUTPUT_FILE.jsonl -
173 ```
174
@@ -180,10 +179,10 @@ aws s3 cp s3://MY_BUCKET/MY_OUTPUT_FILE.jsonl -
179 * Ensure you are using `vllm >= 0.5.5`.
180
181 ### Step 1: Create your batch file
183 -
182 +
183 Add embedding requests to your batch file. The following is an example:
185 -
186 -```
184 +
185 +```text
186 {"custom_id": "request-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are a helpful assistant."}}
187 {"custom_id": "request-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are an unhelpful assistant."}}
188 ```
@@ -198,7 +197,7 @@ You can run the batch using the same command as in earlier examples.
197
198 You can check your results by running `cat results.jsonl`
199
201 -```
200 +```console
201 $ cat results.jsonl
202 {"id":"vllm-db0f71f7dec244e6bce530e0b4ef908b","custom_id":"request-1","response":{"status_code":200,"request_id":"vllm-batch-3580bf4d4ae54d52b67eee266a6eab20","body":{"id":"embd-33ac2efa7996430184461f2e38529746","object":"list","created":444647,"model":"intfloat/e5-mistral-7b-instruct","data":[{"index":0,"object":"embedding","embedding":[0.016204833984375,0.0092010498046875,0.0018358230590820312,-0.0028228759765625,0.001422882080078125,-0.0031147003173828125,...]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0}}},"error":null}
203 ...
@@ -211,10 +210,10 @@ $ cat results.jsonl
210 * Ensure you are using `vllm >= 0.7.0`.
211
212 ### Step 1: Create your batch file
214 -
213 +
214 Add score requests to your batch file. The following is an example:
216 -
217 -```
215 +
216 +```text
217 {"custom_id": "request-1", "method": "POST", "url": "/v1/score", "body": {"model": "BAAI/bge-reranker-v2-m3", "text_1": "What is the capital of France?", "text_2": ["The capital of Brazil is Brasilia.", "The capital of France is Paris."]}}
218 {"custom_id": "request-2", "method": "POST", "url": "/v1/score", "body": {"model": "BAAI/bge-reranker-v2-m3", "text_1": "What is the capital of France?", "text_2": ["The capital of Brazil is Brasilia.", "The capital of France is Paris."]}}
219 ```
@@ -229,7 +228,7 @@ You can run the batch using the same command as in earlier examples.
228
229 You can check your results by running `cat results.jsonl`
230
232 -```
231 +```console
232 $ cat results.jsonl
233 {"id":"vllm-f87c5c4539184f618e555744a2965987","custom_id":"request-1","response":{"status_code":200,"request_id":"vllm-batch-806ab64512e44071b37d3f7ccd291413","body":{"id":"score-4ee45236897b4d29907d49b01298cdb1","object":"list","created":1737847944,"model":"BAAI/bge-reranker-v2-m3","data":[{"index":0,"object":"score","score":0.0010900497436523438},{"index":1,"object":"score","score":1.0}],"usage":{"prompt_tokens":37,"total_tokens":37,"completion_tokens":0,"prompt_tokens_details":null}}},"error":null}
234 {"id":"vllm-41990c51a26d4fac8419077f12871099","custom_id":"request-2","response":{"status_code":200,"request_id":"vllm-batch-73ce66379026482699f81974e14e1e99","body":{"id":"score-13f2ffe6ba40460fbf9f7f00ad667d75","object":"list","created":1737847944,"model":"BAAI/bge-reranker-v2-m3","data":[{"index":0,"object":"score","score":0.001094818115234375},{"index":1,"object":"score","score":1.0}],"usage":{"prompt_tokens":37,"total_tokens":37,"completion_tokens":0,"prompt_tokens_details":null}}},"error":null}
examples/offline_inference/profiling_tpu/README.md
+3 -3
@@ -29,7 +29,6 @@ python3 profiling.py \
29 --profile-result-dir profiles
30 ```
31
32 -
32 ### Generate Decode Trace
33
34 This example runs Llama 3.1 70B with a batch of 32 requests where each has 1 input token and 128 output tokens. This is set up in attempt to profile just the 32 decodes running in parallel by having an extremely small prefill of 1 token and setting `VLLM_TPU_PROFILE_DELAY_MS=1000` to skip the first second of inference (hopefully prefill).
@@ -51,17 +50,18 @@ python3 profiling.py \
50 --max-model-len 2048 --tensor-parallel-size 8
51 ```
52
54 -
53 ## Visualizing the profiles
54
55 Once you have collected your profiles with this script, you can visualize them using [TensorBoard](https://cloud.google.com/tpu/docs/pytorch-xla-performance-profiling-tpu-vm).
56
57 Here are most likely the dependencies you need to install:
58 +
59 ```bash
60 pip install tensorflow-cpu tensorboard-plugin-profile etils importlib_resources
61 ```
62
63 Then you just need to point TensorBoard to the directory where you saved the profiles and visit `http://localhost:6006/` in your browser:
64 +
65 ```bash
66 tensorboard --logdir profiles/ --port 6006
67 -```
\ No newline at end of file
67 +```
examples/online_serving/chart-helm/README.md
+1 -1
@@ -18,4 +18,4 @@ This directory contains a Helm chart for deploying the vllm application. The cha
18 - templates/poddisruptionbudget.yaml: Template for Pod Disruption Budget.
19 - templates/pvc.yaml: Template for Persistent Volume Claims.
20 - templates/secrets.yaml: Template for Kubernetes Secrets.
21 -- templates/service.yaml: Template for creating Services.
\ No newline at end of file
21 +- templates/service.yaml: Template for creating Services.
examples/online_serving/opentelemetry/Otel.md
+22 -10
@@ -1,7 +1,8 @@
1 # Setup OpenTelemetry POC
2
3 1. Install OpenTelemetry packages:
4 - ```
4 +
5 + ```console
6 pip install \
7 'opentelemetry-sdk>=1.26.0,<1.27.0' \
8 'opentelemetry-api>=1.26.0,<1.27.0' \
@@ -10,7 +11,8 @@
11 ```
12
13 1. Start Jaeger in a docker container:
13 - ```
14 +
15 + ```console
16 # From: https://www.jaegertracing.io/docs/1.57/getting-started/
17 docker run --rm --name jaeger \
18 -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
@@ -28,19 +30,23 @@
30 ```
31
32 1. In a new shell, export Jaeger IP:
31 - ```
33 +
34 + ```console
35 export JAEGER_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' jaeger)
36 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
37 ```
38 +
39 Then set vLLM's service name for OpenTelemetry, enable insecure connections to Jaeger and run vLLM:
36 - ```
40 +
41 + ```console
42 export OTEL_SERVICE_NAME="vllm-server"
43 export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
44 vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
45 ```
46
47 1. In a new shell, send requests with trace context from a dummy client
43 - ```
48 +
49 + ```console
50 export JAEGER_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' jaeger)
51 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
52 export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
@@ -48,7 +54,7 @@
54 python dummy_client.py
55 ```
56
51 -1. Open Jaeger webui: http://localhost:16686/
57 +1. Open Jaeger webui: <http://localhost:16686/>
58
59 In the search pane, select `vllm-server` service and hit `Find Traces`. You should get a list of traces, one for each request.
60 ![Traces](https://i.imgur.com/GYHhFjo.png)
@@ -57,26 +63,32 @@
63 ![Spans details](https://i.imgur.com/OPf6CBL.png)
64
65 ## Exporter Protocol
66 +
67 OpenTelemetry supports either `grpc` or `http/protobuf` as the transport protocol for trace data in the exporter.
68 By default, `grpc` is used. To set `http/protobuf` as the protocol, configure the `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` environment variable as follows:
62 -```
69 +
70 +```console
71 export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
72 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
73 vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
74 ```
75
76 ## Instrumentation of FastAPI
77 +
78 OpenTelemetry allows automatic instrumentation of FastAPI.
79 +
80 1. Install the instrumentation library
71 - ```
81 +
82 + ```console
83 pip install opentelemetry-instrumentation-fastapi
84 ```
85
86 1. Run vLLM with `opentelemetry-instrument`
76 - ```
87 +
88 + ```console
89 opentelemetry-instrument vllm serve facebook/opt-125m
90 ```
91
92 1. Send a request to vLLM and find its trace in Jaeger. It should contain spans from FastAPI.
93
82 -![FastAPI Spans](https://i.imgur.com/hywvoOJ.png)
\ No newline at end of file
94 +![FastAPI Spans](https://i.imgur.com/hywvoOJ.png)
examples/online_serving/prometheus_grafana/README.md
+9 -5
@@ -1,14 +1,16 @@
1 -# Prometheus and Grafana
1 +# Prometheus and Grafana
2
3 -This is a simple example that shows you how to connect vLLM metric logging to the Prometheus/Grafana stack. For this example, we launch Prometheus and Grafana via Docker. You can checkout other methods through [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) websites.
3 +This is a simple example that shows you how to connect vLLM metric logging to the Prometheus/Grafana stack. For this example, we launch Prometheus and Grafana via Docker. You can checkout other methods through [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) websites.
4 +
5 +Install:
6
5 -Install:
7 - [`docker`](https://docs.docker.com/engine/install/)
8 - [`docker compose`](https://docs.docker.com/compose/install/linux/#install-using-the-repository)
9
10 ## Launch
11
12 Prometheus metric logging is enabled by default in the OpenAI-compatible server. Launch via the entrypoint:
13 +
14 ```bash
15 vllm serve mistralai/Mistral-7B-v0.1 \
16 --max-model-len 2048 \
@@ -16,11 +18,13 @@ vllm serve mistralai/Mistral-7B-v0.1 \
18 ```
19
20 Launch Prometheus and Grafana servers with `docker compose`:
21 +
22 ```bash
23 docker compose up
24 ```
25
26 Submit some sample requests to the server:
27 +
28 ```bash
29 wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
30
@@ -41,13 +45,13 @@ Navigate to [`http://localhost:3000`](http://localhost:3000). Log in with the de
45
46 ### Add Prometheus Data Source
47
44 -Navigate to [`http://localhost:3000/connections/datasources/new`](http://localhost:3000/connections/datasources/new) and select Prometheus.
48 +Navigate to [`http://localhost:3000/connections/datasources/new`](http://localhost:3000/connections/datasources/new) and select Prometheus.
49
50 On Prometheus configuration page, we need to add the `Prometheus Server URL` in `Connection`. For this setup, Grafana and Prometheus are running in separate containers, but Docker creates DNS name for each containers. You can just use `http://prometheus:9090`.
51
52 Click `Save & Test`. You should get a green check saying "Successfully queried the Prometheus API.".
53
50 -### Import Dashboard
54 +### Import Dashboard
55
56 Navigate to [`http://localhost:3000/dashboard/import`](http://localhost:3000/dashboard/import), upload `grafana.json`, and select the `prometheus` datasource. You should see a screen that looks like the following:
57
examples/other/logging_configuration.md
-5
@@ -15,7 +15,6 @@ more-complex-and-more-flexible.
15 - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1` and
16 set `VLLM_LOGGING_CONFIG_PATH=<path-to-logging-config.json>`
17
18 -
18 ## Logging Configuration Environment Variables
19
20 ### `VLLM_CONFIGURE_LOGGING`
@@ -45,7 +44,6 @@ schema](https://docs.python.org/3/library/logging.config.html#dictionary-schema-
44 If `VLLM_LOGGING_CONFIG_PATH` is specified, but `VLLM_CONFIGURE_LOGGING` is
45 disabled, an error will occur while starting vLLM.
46
48 -
47 ## Examples
48
49 ### Example 1: Customize vLLM root logger
@@ -98,7 +96,6 @@ VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
96 vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048
97 ```
98
101 -
99 ### Example 2: Silence a particular vLLM logger
100
101 To silence a particular vLLM logger, it is necessary to provide custom logging
@@ -153,7 +150,6 @@ VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
150 vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048
151 ```
152
156 -
153 ### Example 3: Disable vLLM default logging configuration
154
155 To disable vLLM's default logging configuration and silence all vLLM loggers,
@@ -166,7 +162,6 @@ VLLM_CONFIGURE_LOGGING=0 \
162 vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048
163 ```
164
169 -
165 ## Additional resources
166
167 - [`logging.config` Dictionary Schema Details](https://docs.python.org/3/library/logging.config.html#dictionary-schema-details)
vllm/distributed/kv_transfer/README.md
+2 -3
@@ -14,8 +14,8 @@ The KV cache transfer contains three layer of abstractions:
14
15 Why we need KV lookup buffer: FIFO pipe itself is not enough as prefill vLLM worker may process requests in a different order compared to decode vLLM worker. Say the QPS is really high, prefill worker may handle requests in order A -> B -> C, but the decode worker may process request C first. This is not the case that can be naturally handled by FIFO pipe, so we provide KV lookup buffer to help translate a FIFO pipe to a lookup buffer.
16
17 -NOTE: KV pipe layer is bypassible: you can skip this layer if your distributed
18 -communication service already supports key-value-based lookup (like redis or
17 +NOTE: KV pipe layer is bypassible: you can skip this layer if your distributed
18 +communication service already supports key-value-based lookup (like redis or
19 RDMA database).
20
21 NOTE: If you want to not only transfer KV caches, but adjust the model execution flow of vLLM as well (for example, allow vLLM to receive KV caches on some tokens and do prefill on the remaining tokens), you can bypass both KV pipe layer and KV lookup buffer layer, and directly implement on KV connector layer. Bear in mind that as vLLM's model input is constantly changing, this implementation will likely be broken when vLLM has new updates.
@@ -27,4 +27,3 @@ The example usage is in [this file](../../../examples/online_serving/disaggregat
27 Here is the diagram of how we run disaggretgated prefilling.
28
29 ![Disaggregated prefill workflow](./disagg_prefill_workflow.jpg)
30 -