A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 04:45 UTC
[doc] Fold long code blocks to improve readability (#19926)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
Jun 23, 2025 at 13:24 UTC
f17aec0d6350303b46ee58d27a6fc83ddf9583b2
50 files changed
+3687
-3412
docs/ci/update_pytorch_version.md
+3
-3
@@ -91,7 +91,7 @@ source to unblock the update process.
91
### FlashInfer
92
Here is how to build and install it from source with torch2.7.0+cu128 in vLLM [Dockerfile](https://github.com/vllm-project/vllm/blob/27bebcd89792d5c4b08af7a65095759526f2f9e1/docker/Dockerfile#L259-L271):
93
94
-```
94
+```bash
95
export TORCH_CUDA_ARCH_LIST='7.5 8.0 8.9 9.0 10.0+PTX'
96
export FLASHINFER_ENABLE_SM90=1
97
uv pip install --system --no-build-isolation "git+https://github.com/flashinfer-ai/flashinfer@v0.2.6.post1"
@@ -105,14 +105,14 @@ team if you want to get the package published there.
105
### xFormers
106
Similar to FlashInfer, here is how to build and install xFormers from source:
107
108
-```
108
+```bash
109
export TORCH_CUDA_ARCH_LIST='7.0 7.5 8.0 8.9 9.0 10.0+PTX'
110
MAX_JOBS=16 uv pip install --system --no-build-isolation "git+https://github.com/facebookresearch/xformers@v0.0.30"
111
```
112
113
### Mamba
114
115
-```
115
+```bash
116
uv pip install --system --no-build-isolation "git+https://github.com/state-spaces/mamba@v2.2.4"
117
```
118
docs/cli/README.md
+22
-27
@@ -16,35 +16,33 @@ vllm {chat,complete,serve,bench,collect-env,run-batch}
16
17
Start the vLLM OpenAI Compatible API server.
18
19
-Examples:
19
+??? Examples
20
21
-```bash
22
-# Start with a model
23
-vllm serve meta-llama/Llama-2-7b-hf
21
+ ```bash
22
+ # Start with a model
23
+ vllm serve meta-llama/Llama-2-7b-hf
24
25
-# Specify the port
26
-vllm serve meta-llama/Llama-2-7b-hf --port 8100
25
+ # Specify the port
26
+ vllm serve meta-llama/Llama-2-7b-hf --port 8100
27
28
-# Check with --help for more options
29
-# To list all groups
30
-vllm serve --help=listgroup
28
+ # Check with --help for more options
29
+ # To list all groups
30
+ vllm serve --help=listgroup
31
32
-# To view a argument group
33
-vllm serve --help=ModelConfig
32
+ # To view a argument group
33
+ vllm serve --help=ModelConfig
34
35
-# To view a single argument
36
-vllm serve --help=max-num-seqs
35
+ # To view a single argument
36
+ vllm serve --help=max-num-seqs
37
38
-# To search by keyword
39
-vllm serve --help=max
40
-```
38
+ # To search by keyword
39
+ vllm serve --help=max
40
+ ```
41
42
## chat
43
44
Generate chat completions via the running API server.
45
46
-Examples:
47
-
46
```bash
47
# Directly connect to localhost API without arguments
48
vllm chat
@@ -60,8 +58,6 @@ vllm chat --quick "hi"
58
59
Generate text completions based on the given prompt via the running API server.
60
63
-Examples:
64
-
61
```bash
62
# Directly connect to localhost API without arguments
63
vllm complete
@@ -73,6 +69,8 @@ vllm complete --url http://{vllm-serve-host}:{vllm-serve-port}/v1
69
vllm complete --quick "The future of AI is"
70
```
71
72
+</details>
73
+
74
## bench
75
76
Run benchmark tests for latency online serving throughput and offline inference throughput.
@@ -89,8 +87,6 @@ vllm bench {latency, serve, throughput}
87
88
Benchmark the latency of a single batch of requests.
89
92
-Example:
93
-
90
```bash
91
vllm bench latency \
92
--model meta-llama/Llama-3.2-1B-Instruct \
@@ -104,8 +100,6 @@ vllm bench latency \
100
101
Benchmark the online serving throughput.
102
107
-Example:
108
-
103
```bash
104
vllm bench serve \
105
--model meta-llama/Llama-3.2-1B-Instruct \
@@ -120,8 +114,6 @@ vllm bench serve \
114
115
Benchmark offline inference throughput.
116
123
-Example:
124
-
117
```bash
118
vllm bench throughput \
119
--model meta-llama/Llama-3.2-1B-Instruct \
@@ -143,7 +135,8 @@ vllm collect-env
135
136
Run batch prompts and write results to file.
137
146
-Examples:
138
+<details>
139
+<summary>Examples</summary>
140
141
```bash
142
# Running with a local file
@@ -159,6 +152,8 @@ vllm run-batch \
152
--model meta-llama/Meta-Llama-3-8B-Instruct
153
```
154
155
+</details>
156
+
157
## More Help
158
159
For detailed options of any subcommand, use:
docs/configuration/conserving_memory.md
+31
-27
@@ -57,19 +57,21 @@ By default, we optimize model inference using CUDA graphs which take up extra me
57
58
You can adjust `compilation_config` to achieve a better balance between inference speed and memory usage:
59
60
-```python
61
-from vllm import LLM
62
-from vllm.config import CompilationConfig, CompilationLevel
63
-
64
-llm = LLM(
65
- model="meta-llama/Llama-3.1-8B-Instruct",
66
- compilation_config=CompilationConfig(
67
- level=CompilationLevel.PIECEWISE,
68
- # By default, it goes up to max_num_seqs
69
- cudagraph_capture_sizes=[1, 2, 4, 8, 16],
70
- ),
71
-)
72
-```
60
+??? Code
61
+
62
+ ```python
63
+ from vllm import LLM
64
+ from vllm.config import CompilationConfig, CompilationLevel
65
+
66
+ llm = LLM(
67
+ model="meta-llama/Llama-3.1-8B-Instruct",
68
+ compilation_config=CompilationConfig(
69
+ level=CompilationLevel.PIECEWISE,
70
+ # By default, it goes up to max_num_seqs
71
+ cudagraph_capture_sizes=[1, 2, 4, 8, 16],
72
+ ),
73
+ )
74
+ ```
75
76
You can disable graph capturing completely via the `enforce_eager` flag:
77
@@ -127,18 +129,20 @@ reduce the size of the processed multi-modal inputs, which in turn saves memory.
129
130
Here are some examples:
131
130
-```python
131
-from vllm import LLM
132
+??? Code
133
133
-# Available for Qwen2-VL series models
134
-llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
135
- mm_processor_kwargs={
136
- "max_pixels": 768 * 768, # Default is 1280 * 28 * 28
137
- })
138
-
139
-# Available for InternVL series models
140
-llm = LLM(model="OpenGVLab/InternVL2-2B",
141
- mm_processor_kwargs={
142
- "max_dynamic_patch": 4, # Default is 12
143
- })
144
-```
134
+ ```python
135
+ from vllm import LLM
136
+
137
+ # Available for Qwen2-VL series models
138
+ llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
139
+ mm_processor_kwargs={
140
+ "max_pixels": 768 * 768, # Default is 1280 * 28 * 28
141
+ })
142
+
143
+ # Available for InternVL series models
144
+ llm = LLM(model="OpenGVLab/InternVL2-2B",
145
+ mm_processor_kwargs={
146
+ "max_dynamic_patch": 4, # Default is 12
147
+ })
148
+ ```
docs/configuration/env_vars.md
+5
-3
@@ -7,6 +7,8 @@ vLLM uses the following environment variables to configure the system:
7
8
All environment variables used by vLLM are prefixed with `VLLM_`. **Special care should be taken for Kubernetes users**: please do not name the service as `vllm`, otherwise environment variables set by Kubernetes might conflict with vLLM's environment variables, because [Kubernetes sets environment variables for each service with the capitalized service name as the prefix](https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables).
9
10
-```python
11
---8<-- "vllm/envs.py:env-vars-definition"
12
-```
10
+??? Code
11
+
12
+ ```python
13
+ --8<-- "vllm/envs.py:env-vars-definition"
14
+ ```
docs/contributing/README.md
+16
-14
@@ -93,25 +93,27 @@ For additional features and advanced configurations, refer to the official [MkDo
93
94
## Testing
95
96
-```bash
97
-pip install -r requirements/dev.txt
96
+??? note "Commands"
97
99
-# Linting, formatting and static type checking
100
-pre-commit install --hook-type pre-commit --hook-type commit-msg
98
+ ```bash
99
+ pip install -r requirements/dev.txt
100
102
-# You can manually run pre-commit with
103
-pre-commit run --all-files
101
+ # Linting, formatting and static type checking
102
+ pre-commit install --hook-type pre-commit --hook-type commit-msg
103
105
-# To manually run something from CI that does not run
106
-# locally by default, you can run:
107
-pre-commit run mypy-3.9 --hook-stage manual --all-files
104
+ # You can manually run pre-commit with
105
+ pre-commit run --all-files
106
109
-# Unit tests
110
-pytest tests/
107
+ # To manually run something from CI that does not run
108
+ # locally by default, you can run:
109
+ pre-commit run mypy-3.9 --hook-stage manual --all-files
110
112
-# Run tests for a single test file with detailed output
113
-pytest -s -v tests/test_logger.py
114
-```
111
+ # Unit tests
112
+ pytest tests/
113
+
114
+ # Run tests for a single test file with detailed output
115
+ pytest -s -v tests/test_logger.py
116
+ ```
117
118
!!! tip
119
Since the <gh-file:docker/Dockerfile> ships with Python 3.12, all tests in CI (except `mypy`) are run with Python 3.12.
docs/contributing/model/basic.md
+29
-27
@@ -27,33 +27,35 @@ All vLLM modules within the model must include a `prefix` argument in their cons
27
28
The initialization code should look like this:
29
30
-```python
31
-from torch import nn
32
-from vllm.config import VllmConfig
33
-from vllm.attention import Attention
34
-
35
-class MyAttention(nn.Module):
36
- def __init__(self, vllm_config: VllmConfig, prefix: str):
37
- super().__init__()
38
- self.attn = Attention(prefix=f"{prefix}.attn")
39
-
40
-class MyDecoderLayer(nn.Module):
41
- def __init__(self, vllm_config: VllmConfig, prefix: str):
42
- super().__init__()
43
- self.self_attn = MyAttention(prefix=f"{prefix}.self_attn")
44
-
45
-class MyModel(nn.Module):
46
- def __init__(self, vllm_config: VllmConfig, prefix: str):
47
- super().__init__()
48
- self.layers = nn.ModuleList(
49
- [MyDecoderLayer(vllm_config, prefix=f"{prefix}.layers.{i}") for i in range(vllm_config.model_config.hf_config.num_hidden_layers)]
50
- )
51
-
52
-class MyModelForCausalLM(nn.Module):
53
- def __init__(self, vllm_config: VllmConfig, prefix: str = ""):
54
- super().__init__()
55
- self.model = MyModel(vllm_config, prefix=f"{prefix}.model")
56
-```
30
+??? Code
31
+
32
+ ```python
33
+ from torch import nn
34
+ from vllm.config import VllmConfig
35
+ from vllm.attention import Attention
36
+
37
+ class MyAttention(nn.Module):
38
+ def __init__(self, vllm_config: VllmConfig, prefix: str):
39
+ super().__init__()
40
+ self.attn = Attention(prefix=f"{prefix}.attn")
41
+
42
+ class MyDecoderLayer(nn.Module):
43
+ def __init__(self, vllm_config: VllmConfig, prefix: str):
44
+ super().__init__()
45
+ self.self_attn = MyAttention(prefix=f"{prefix}.self_attn")
46
+
47
+ class MyModel(nn.Module):
48
+ def __init__(self, vllm_config: VllmConfig, prefix: str):
49
+ super().__init__()
50
+ self.layers = nn.ModuleList(
51
+ [MyDecoderLayer(vllm_config, prefix=f"{prefix}.layers.{i}") for i in range(vllm_config.model_config.hf_config.num_hidden_layers)]
52
+ )
53
+
54
+ class MyModelForCausalLM(nn.Module):
55
+ def __init__(self, vllm_config: VllmConfig, prefix: str = ""):
56
+ super().__init__()
57
+ self.model = MyModel(vllm_config, prefix=f"{prefix}.model")
58
+ ```
59
60
### Computation Code
61
docs/contributing/model/multimodal.md
+424
-385
@@ -25,59 +25,63 @@ Further update the model as follows:
25
26
- Implement [get_multimodal_embeddings][vllm.model_executor.models.interfaces.SupportsMultiModal.get_multimodal_embeddings] that returns the embeddings from running the multimodal inputs through the multimodal tokenizer of the model. Below we provide a boilerplate of a typical implementation pattern, but feel free to adjust it to your own needs.
27
28
- ```python
29
- class YourModelForImage2Seq(nn.Module):
30
- ...
28
+ ??? Code
29
32
- def _process_image_input(self, image_input: YourModelImageInputs) -> torch.Tensor:
30
+ ```python
31
+ class YourModelForImage2Seq(nn.Module):
32
+ ...
33
34
- assert self.vision_encoder is not None
35
- image_features = self.vision_encoder(image_input)
36
- return self.multi_modal_projector(image_features)
34
+ def _process_image_input(self, image_input: YourModelImageInputs) -> torch.Tensor:
35
38
- def get_multimodal_embeddings(
39
- self, **kwargs: object) -> Optional[MultiModalEmbeddings]:
36
+ assert self.vision_encoder is not None
37
+ image_features = self.vision_encoder(image_input)
38
+ return self.multi_modal_projector(image_features)
39
41
- # Validate the multimodal input keyword arguments
42
- image_input = self._parse_and_validate_image_input(**kwargs)
43
- if image_input is None:
44
- return None
40
+ def get_multimodal_embeddings(
41
+ self, **kwargs: object) -> Optional[MultiModalEmbeddings]:
42
46
- # Run multimodal inputs through encoder and projector
47
- vision_embeddings = self._process_image_input(image_input)
48
- return vision_embeddings
49
- ```
43
+ # Validate the multimodal input keyword arguments
44
+ image_input = self._parse_and_validate_image_input(**kwargs)
45
+ if image_input is None:
46
+ return None
47
+
48
+ # Run multimodal inputs through encoder and projector
49
+ vision_embeddings = self._process_image_input(image_input)
50
+ return vision_embeddings
51
+ ```
52
53
!!! important
54
The returned `multimodal_embeddings` must be either a **3D [torch.Tensor][]** of shape `(num_items, feature_size, hidden_size)`, or a **list / tuple of 2D [torch.Tensor][]'s** of shape `(feature_size, hidden_size)`, so that `multimodal_embeddings[i]` retrieves the embeddings generated from the `i`-th multimodal data item (e.g, image) of the request.
55
56
- Implement [get_input_embeddings][vllm.model_executor.models.interfaces.SupportsMultiModal.get_input_embeddings] to merge `multimodal_embeddings` with text embeddings from the `input_ids`. If input processing for the model is implemented correctly (see sections below), then you can leverage the utility function we provide to easily merge the embeddings.
57
56
- ```python
57
- from .utils import merge_multimodal_embeddings
58
+ ??? Code
59
59
- class YourModelForImage2Seq(nn.Module):
60
- ...
60
+ ```python
61
+ from .utils import merge_multimodal_embeddings
62
62
- def get_input_embeddings(
63
- self,
64
- input_ids: torch.Tensor,
65
- multimodal_embeddings: Optional[MultiModalEmbeddings] = None,
66
- ) -> torch.Tensor:
67
-
68
- # `get_input_embeddings` should already be implemented for the language
69
- # model as one of the requirements of basic vLLM model implementation.
70
- inputs_embeds = self.language_model.get_input_embeddings(input_ids)
71
-
72
- if multimodal_embeddings is not None:
73
- inputs_embeds = merge_multimodal_embeddings(
74
- input_ids=input_ids,
75
- inputs_embeds=inputs_embeds,
76
- multimodal_embeddings=multimodal_embeddings,
77
- placeholder_token_id=self.config.image_token_index)
78
-
79
- return inputs_embeds
80
- ```
63
+ class YourModelForImage2Seq(nn.Module):
64
+ ...
65
+
66
+ def get_input_embeddings(
67
+ self,
68
+ input_ids: torch.Tensor,
69
+ multimodal_embeddings: Optional[MultiModalEmbeddings] = None,
70
+ ) -> torch.Tensor:
71
+
72
+ # `get_input_embeddings` should already be implemented for the language
73
+ # model as one of the requirements of basic vLLM model implementation.
74
+ inputs_embeds = self.language_model.get_input_embeddings(input_ids)
75
+
76
+ if multimodal_embeddings is not None:
77
+ inputs_embeds = merge_multimodal_embeddings(
78
+ input_ids=input_ids,
79
+ inputs_embeds=inputs_embeds,
80
+ multimodal_embeddings=multimodal_embeddings,
81
+ placeholder_token_id=self.config.image_token_index)
82
+
83
+ return inputs_embeds
84
+ ```
85
86
- Implement [get_language_model][vllm.model_executor.models.interfaces.SupportsMultiModal.get_language_model] getter to provide stable access to the underlying language model.
87
@@ -135,42 +139,46 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
139
140
Looking at the code of HF's `LlavaForConditionalGeneration`:
141
138
- ```python
139
- # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L530-L544
140
- n_image_tokens = (input_ids == self.config.image_token_index).sum().item()
141
- n_image_features = image_features.shape[0] * image_features.shape[1]
142
+ ??? Code
143
143
- if n_image_tokens != n_image_features:
144
- raise ValueError(
145
- f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {n_image_features}"
144
+ ```python
145
+ # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L530-L544
146
+ n_image_tokens = (input_ids == self.config.image_token_index).sum().item()
147
+ n_image_features = image_features.shape[0] * image_features.shape[1]
148
+
149
+ if n_image_tokens != n_image_features:
150
+ raise ValueError(
151
+ f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {n_image_features}"
152
+ )
153
+ special_image_mask = (
154
+ (input_ids == self.config.image_token_index)
155
+ .unsqueeze(-1)
156
+ .expand_as(inputs_embeds)
157
+ .to(inputs_embeds.device)
158
)
147
- special_image_mask = (
148
- (input_ids == self.config.image_token_index)
149
- .unsqueeze(-1)
150
- .expand_as(inputs_embeds)
151
- .to(inputs_embeds.device)
152
- )
153
- image_features = image_features.to(inputs_embeds.device, inputs_embeds.dtype)
154
- inputs_embeds = inputs_embeds.masked_scatter(special_image_mask, image_features)
155
- ```
159
+ image_features = image_features.to(inputs_embeds.device, inputs_embeds.dtype)
160
+ inputs_embeds = inputs_embeds.masked_scatter(special_image_mask, image_features)
161
+ ```
162
163
The number of placeholder feature tokens per image is `image_features.shape[1]`.
164
`image_features` is calculated inside the `get_image_features` method:
165
160
- ```python
161
- # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L290-L300
162
- image_outputs = self.vision_tower(pixel_values, output_hidden_states=True)
163
-
164
- selected_image_feature = image_outputs.hidden_states[vision_feature_layer]
165
- if vision_feature_select_strategy == "default":
166
- selected_image_feature = selected_image_feature[:, 1:]
167
- elif vision_feature_select_strategy == "full":
168
- selected_image_feature = selected_image_feature
169
- else:
170
- raise ValueError(f"Unexpected select feature strategy: {self.config.vision_feature_select_strategy}")
171
- image_features = self.multi_modal_projector(selected_image_feature)
172
- return image_features
173
- ```
166
+ ??? Code
167
+
168
+ ```python
169
+ # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L290-L300
170
+ image_outputs = self.vision_tower(pixel_values, output_hidden_states=True)
171
+
172
+ selected_image_feature = image_outputs.hidden_states[vision_feature_layer]
173
+ if vision_feature_select_strategy == "default":
174
+ selected_image_feature = selected_image_feature[:, 1:]
175
+ elif vision_feature_select_strategy == "full":
176
+ selected_image_feature = selected_image_feature
177
+ else:
178
+ raise ValueError(f"Unexpected select feature strategy: {self.config.vision_feature_select_strategy}")
179
+ image_features = self.multi_modal_projector(selected_image_feature)
180
+ return image_features
181
+ ```
182
183
We can infer that `image_features.shape[1]` is based on `image_outputs.hidden_states.shape[1]` from the vision tower
184
(`CLIPVisionModel` for the [`llava-hf/llava-1.5-7b-hf`](https://huggingface.co/llava-hf/llava-1.5-7b-hf) model).
@@ -193,20 +201,22 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
201
202
To find the sequence length, we turn to the code of `CLIPVisionEmbeddings`:
203
196
- ```python
197
- # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L247-L257
198
- target_dtype = self.patch_embedding.weight.dtype
199
- patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype)) # shape = [*, width, grid, grid]
200
- patch_embeds = patch_embeds.flatten(2).transpose(1, 2)
201
-
202
- class_embeds = self.class_embedding.expand(batch_size, 1, -1)
203
- embeddings = torch.cat([class_embeds, patch_embeds], dim=1)
204
- if interpolate_pos_encoding:
205
- embeddings = embeddings + self.interpolate_pos_encoding(embeddings, height, width)
206
- else:
207
- embeddings = embeddings + self.position_embedding(self.position_ids)
208
- return embeddings
209
- ```
204
+ ??? Code
205
+
206
+ ```python
207
+ # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L247-L257
208
+ target_dtype = self.patch_embedding.weight.dtype
209
+ patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype)) # shape = [*, width, grid, grid]
210
+ patch_embeds = patch_embeds.flatten(2).transpose(1, 2)
211
+
212
+ class_embeds = self.class_embedding.expand(batch_size, 1, -1)
213
+ embeddings = torch.cat([class_embeds, patch_embeds], dim=1)
214
+ if interpolate_pos_encoding:
215
+ embeddings = embeddings + self.interpolate_pos_encoding(embeddings, height, width)
216
+ else:
217
+ embeddings = embeddings + self.position_embedding(self.position_ids)
218
+ return embeddings
219
+ ```
220
221
We can infer that `embeddings.shape[1] == self.num_positions`, where
222
@@ -218,55 +228,59 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
228
229
Overall, the number of placeholder feature tokens for an image can be calculated as:
230
221
- ```python
222
- def get_num_image_tokens(
223
- self,
224
- *,
225
- image_width: int,
226
- image_height: int,
227
- ) -> int:
228
- hf_config = self.get_hf_config()
229
- hf_processor = self.get_hf_processor()
231
+ ??? Code
232
231
- image_size = hf_config.vision_config.image_size
232
- patch_size = hf_config.vision_config.patch_size
233
+ ```python
234
+ def get_num_image_tokens(
235
+ self,
236
+ *,
237
+ image_width: int,
238
+ image_height: int,
239
+ ) -> int:
240
+ hf_config = self.get_hf_config()
241
+ hf_processor = self.get_hf_processor()
242
234
- num_image_tokens = (image_size // patch_size) ** 2 + 1
235
- if hf_processor.vision_feature_select_strategy == "default":
236
- num_image_tokens -= 1
243
+ image_size = hf_config.vision_config.image_size
244
+ patch_size = hf_config.vision_config.patch_size
245
238
- return num_image_tokens
239
- ```
246
+ num_image_tokens = (image_size // patch_size) ** 2 + 1
247
+ if hf_processor.vision_feature_select_strategy == "default":
248
+ num_image_tokens -= 1
249
+
250
+ return num_image_tokens
251
+ ```
252
253
Notice that the number of image tokens doesn't depend on the image width and height.
254
We can simply use a dummy `image_size` to calculate the multimodal profiling data:
255
244
- ```python
245
- # NOTE: In actuality, this is usually implemented as part of the
246
- # model's subclass of `BaseProcessingInfo`, but we show it as is
247
- # here for simplicity.
248
- def get_image_size_with_most_features(self) -> ImageSize:
249
- hf_config = self.get_hf_config()
250
- width = height = hf_config.image_size
251
- return ImageSize(width=width, height=height)
256
+ ??? Code
257
253
- def get_dummy_mm_data(
254
- self,
255
- seq_len: int,
256
- mm_counts: Mapping[str, int],
257
- ) -> MultiModalDataDict:
258
- num_images = mm_counts.get("image", 0)
259
-
260
- target_width, target_height = \
261
- self.info.get_image_size_with_most_features()
258
+ ```python
259
+ # NOTE: In actuality, this is usually implemented as part of the
260
+ # model's subclass of `BaseProcessingInfo`, but we show it as is
261
+ # here for simplicity.
262
+ def get_image_size_with_most_features(self) -> ImageSize:
263
+ hf_config = self.get_hf_config()
264
+ width = height = hf_config.image_size
265
+ return ImageSize(width=width, height=height)
266
263
- return {
264
- "image":
265
- self._get_dummy_images(width=target_width,
266
- height=target_height,
267
- num_images=num_images)
268
- }
269
- ```
267
+ def get_dummy_mm_data(
268
+ self,
269
+ seq_len: int,
270
+ mm_counts: Mapping[str, int],
271
+ ) -> MultiModalDataDict:
272
+ num_images = mm_counts.get("image", 0)
273
+
274
+ target_width, target_height = \
275
+ self.info.get_image_size_with_most_features()
276
+
277
+ return {
278
+ "image":
279
+ self._get_dummy_images(width=target_width,
280
+ height=target_height,
281
+ num_images=num_images)
282
+ }
283
+ ```
284
285
For the text, we simply expand the multimodal image token from the model config to match the desired number of images.
286
@@ -284,21 +298,23 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
298
299
Looking at the code of HF's `FuyuForCausalLM`:
300
287
- ```python
288
- # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/modeling_fuyu.py#L311-L322
289
- if image_patches is not None and past_key_values is None:
290
- patch_embeddings = [
291
- self.vision_embed_tokens(patch.to(self.vision_embed_tokens.weight.dtype))
292
- .squeeze(0)
293
- .to(inputs_embeds.device)
294
- for patch in image_patches
295
- ]
296
- inputs_embeds = self.gather_continuous_embeddings(
297
- word_embeddings=inputs_embeds,
298
- continuous_embeddings=patch_embeddings,
299
- image_patch_input_indices=image_patches_indices,
300
- )
301
- ```
301
+ ??? Code
302
+
303
+ ```python
304
+ # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/modeling_fuyu.py#L311-L322
305
+ if image_patches is not None and past_key_values is None:
306
+ patch_embeddings = [
307
+ self.vision_embed_tokens(patch.to(self.vision_embed_tokens.weight.dtype))
308
+ .squeeze(0)
309
+ .to(inputs_embeds.device)
310
+ for patch in image_patches
311
+ ]
312
+ inputs_embeds = self.gather_continuous_embeddings(
313
+ word_embeddings=inputs_embeds,
314
+ continuous_embeddings=patch_embeddings,
315
+ image_patch_input_indices=image_patches_indices,
316
+ )
317
+ ```
318
319
The number of placeholder feature tokens for the `i`th item in the batch is `patch_embeddings[i].shape[0]`,
320
which is the same as `image_patches[i].shape[0]`, i.e. `num_total_patches`.
@@ -312,92 +328,98 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
328
In `FuyuImageProcessor.preprocess`, the images are resized and padded to the target `FuyuImageProcessor.size`,
329
returning the dimensions after resizing (but before padding) as metadata.
330
315
- ```python
316
- # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L541-L544
317
- image_encoding = self.image_processor.preprocess(images, **output_kwargs["images_kwargs"])
318
- batch_images = image_encoding["images"]
319
- image_unpadded_heights = image_encoding["image_unpadded_heights"]
320
- image_unpadded_widths = image_encoding["image_unpadded_widths"]
321
-
322
- # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L480-L
323
- if do_resize:
324
- batch_images = [
325
- [self.resize(image, size=size, input_data_format=input_data_format) for image in images]
326
- for images in batch_images
327
- ]
328
-
329
- image_sizes = [get_image_size(images[0], channel_dim=input_data_format) for images in batch_images]
330
- image_unpadded_heights = [[image_size[0]] for image_size in image_sizes]
331
- image_unpadded_widths = [[image_size[1]] for image_size in image_sizes]
332
-
333
- if do_pad:
334
- batch_images = [
335
- [
336
- self.pad_image(
337
- image,
338
- size=size,
339
- mode=padding_mode,
340
- constant_values=padding_value,
341
- input_data_format=input_data_format,
342
- )
343
- for image in images
331
+ ??? Code
332
+
333
+ ```python
334
+ # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L541-L544
335
+ image_encoding = self.image_processor.preprocess(images, **output_kwargs["images_kwargs"])
336
+ batch_images = image_encoding["images"]
337
+ image_unpadded_heights = image_encoding["image_unpadded_heights"]
338
+ image_unpadded_widths = image_encoding["image_unpadded_widths"]
339
+
340
+ # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L480-L
341
+ if do_resize:
342
+ batch_images = [
343
+ [self.resize(image, size=size, input_data_format=input_data_format) for image in images]
344
+ for images in batch_images
345
]
345
- for images in batch_images
346
- ]
347
- ```
346
349
- In `FuyuImageProcessor.preprocess_with_tokenizer_info`, the images are split into patches based on this metadata:
347
+ image_sizes = [get_image_size(images[0], channel_dim=input_data_format) for images in batch_images]
348
+ image_unpadded_heights = [[image_size[0]] for image_size in image_sizes]
349
+ image_unpadded_widths = [[image_size[1]] for image_size in image_sizes]
350
+
351
+ if do_pad:
352
+ batch_images = [
353
+ [
354
+ self.pad_image(
355
+ image,
356
+ size=size,
357
+ mode=padding_mode,
358
+ constant_values=padding_value,
359
+ input_data_format=input_data_format,
360
+ )
361
+ for image in images
362
+ ]
363
+ for images in batch_images
364
+ ]
365
+ ```
366
351
- ```python
352
- # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L425
353
- model_image_input = self.image_processor.preprocess_with_tokenizer_info(
354
- image_input=tensor_batch_images,
355
- image_present=image_present,
356
- image_unpadded_h=image_unpadded_heights,
357
- image_unpadded_w=image_unpadded_widths,
358
- image_placeholder_id=image_placeholder_id,
359
- image_newline_id=image_newline_id,
360
- variable_sized=True,
361
- )
367
+ In `FuyuImageProcessor.preprocess_with_tokenizer_info`, the images are split into patches based on this metadata:
368
363
- # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L638-L658
364
- image_height, image_width = image.shape[1], image.shape[2]
365
- if variable_sized: # variable_sized=True
366
- new_h = min(
367
- image_height,
368
- math.ceil(image_unpadded_h[batch_index, subseq_index] / patch_height) * patch_height,
369
- )
370
- new_w = min(
371
- image_width,
372
- math.ceil(image_unpadded_w[batch_index, subseq_index] / patch_width) * patch_width,
369
+ ??? Code
370
+
371
+ ```python
372
+ # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L425
373
+ model_image_input = self.image_processor.preprocess_with_tokenizer_info(
374
+ image_input=tensor_batch_images,
375
+ image_present=image_present,
376
+ image_unpadded_h=image_unpadded_heights,
377
+ image_unpadded_w=image_unpadded_widths,
378
+ image_placeholder_id=image_placeholder_id,
379
+ image_newline_id=image_newline_id,
380
+ variable_sized=True,
381
)
374
- image = image[:, :new_h, :new_w]
375
- image_height, image_width = new_h, new_w
382
377
- num_patches = self.get_num_patches(image_height=image_height, image_width=image_width)
378
- tensor_of_image_ids = torch.full(
379
- [num_patches], image_placeholder_id, dtype=torch.int32, device=image_input.device
380
- )
381
- patches = self.patchify_image(image=image.unsqueeze(0)).squeeze(0)
382
- assert num_patches == patches.shape[0]
383
- ```
383
+ # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L638-L658
384
+ image_height, image_width = image.shape[1], image.shape[2]
385
+ if variable_sized: # variable_sized=True
386
+ new_h = min(
387
+ image_height,
388
+ math.ceil(image_unpadded_h[batch_index, subseq_index] / patch_height) * patch_height,
389
+ )
390
+ new_w = min(
391
+ image_width,
392
+ math.ceil(image_unpadded_w[batch_index, subseq_index] / patch_width) * patch_width,
393
+ )
394
+ image = image[:, :new_h, :new_w]
395
+ image_height, image_width = new_h, new_w
396
+
397
+ num_patches = self.get_num_patches(image_height=image_height, image_width=image_width)
398
+ tensor_of_image_ids = torch.full(
399
+ [num_patches], image_placeholder_id, dtype=torch.int32, device=image_input.device
400
+ )
401
+ patches = self.patchify_image(image=image.unsqueeze(0)).squeeze(0)
402
+ assert num_patches == patches.shape[0]
403
+ ```
404
405
The number of patches is in turn defined by `FuyuImageProcessor.get_num_patches`:
406
387
- ```python
388
- # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L552-L562
389
- patch_size = patch_size if patch_size is not None else self.patch_size
390
- patch_height, patch_width = self.patch_size["height"], self.patch_size["width"]
391
-
392
- if image_height % patch_height != 0:
393
- raise ValueError(f"{image_height=} must be divisible by {patch_height}")
394
- if image_width % patch_width != 0:
395
- raise ValueError(f"{image_width=} must be divisible by {patch_width}")
396
-
397
- num_patches_per_dim_h = image_height // patch_height
398
- num_patches_per_dim_w = image_width // patch_width
399
- num_patches = num_patches_per_dim_h * num_patches_per_dim_w
400
- ```
407
+ ??? Code
408
+
409
+ ```python
410
+ # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L552-L562
411
+ patch_size = patch_size if patch_size is not None else self.patch_size
412
+ patch_height, patch_width = self.patch_size["height"], self.patch_size["width"]
413
+
414
+ if image_height % patch_height != 0:
415
+ raise ValueError(f"{image_height=} must be divisible by {patch_height}")
416
+ if image_width % patch_width != 0:
417
+ raise ValueError(f"{image_width=} must be divisible by {patch_width}")
418
+
419
+ num_patches_per_dim_h = image_height // patch_height
420
+ num_patches_per_dim_w = image_width // patch_width
421
+ num_patches = num_patches_per_dim_h * num_patches_per_dim_w
422
+ ```
423
424
These image patches correspond to placeholder tokens (`|SPEAKER|`). So, we just need to maximize the number of image patches. Since input images are first resized
425
to fit within `image_processor.size`, we can maximize the number of image patches by inputting an image with size equal to `image_processor.size`.
@@ -419,23 +441,25 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
441
442
For the multimodal image profiling data, the logic is very similar to LLaVA:
443
422
- ```python
423
- def get_dummy_mm_data(
424
- self,
425
- seq_len: int,
426
- mm_counts: Mapping[str, int],
427
- ) -> MultiModalDataDict:
428
- target_width, target_height = \
429
- self.info.get_image_size_with_most_features()
430
- num_images = mm_counts.get("image", 0)
444
+ ??? Code
445
432
- return {
433
- "image":
434
- self._get_dummy_images(width=target_width,
435
- height=target_height,
436
- num_images=num_images)
437
- }
438
- ```
446
+ ```python
447
+ def get_dummy_mm_data(
448
+ self,
449
+ seq_len: int,
450
+ mm_counts: Mapping[str, int],
451
+ ) -> MultiModalDataDict:
452
+ target_width, target_height = \
453
+ self.info.get_image_size_with_most_features()
454
+ num_images = mm_counts.get("image", 0)
455
+
456
+ return {
457
+ "image":
458
+ self._get_dummy_images(width=target_width,
459
+ height=target_height,
460
+ num_images=num_images)
461
+ }
462
+ ```
463
464
## 4. Specify processing details
465
@@ -455,6 +479,7 @@ return a schema of the tensors outputted by the HF processor that are related to
479
The output of `CLIPImageProcessor` is a simple tensor with shape
480
`(num_images, num_channels, image_height, image_width)`:
481
482
+
483
```python
484
# https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/image_processing_clip.py#L339-L345
485
images = [
@@ -505,35 +530,37 @@ return a schema of the tensors outputted by the HF processor that are related to
530
In order to support the use of [MultiModalFieldConfig.batched][] like in LLaVA,
531
we remove the extra batch dimension by overriding [BaseMultiModalProcessor._call_hf_processor][]:
532
508
- ```python
509
- def _call_hf_processor(
510
- self,
511
- prompt: str,
512
- mm_data: Mapping[str, object],
513
- mm_kwargs: Mapping[str, object],
514
- ) -> BatchFeature:
515
- processed_outputs = super()._call_hf_processor(
516
- prompt=prompt,
517
- mm_data=mm_data,
518
- mm_kwargs=mm_kwargs,
519
- )
533
+ ??? Code
534
521
- image_patches = processed_outputs.get("image_patches")
522
- if image_patches is not None:
523
- images = mm_data["images"]
524
- assert isinstance(images, list)
535
+ ```python
536
+ def _call_hf_processor(
537
+ self,
538
+ prompt: str,
539
+ mm_data: Mapping[str, object],
540
+ mm_kwargs: Mapping[str, object],
541
+ ) -> BatchFeature:
542
+ processed_outputs = super()._call_hf_processor(
543
+ prompt=prompt,
544
+ mm_data=mm_data,
545
+ mm_kwargs=mm_kwargs,
546
+ )
547
526
- # Original output: (1, num_images, Pn, Px * Py * C)
527
- # New output: (num_images, Pn, Px * Py * C)
528
- assert (isinstance(image_patches, list)
529
- and len(image_patches) == 1)
530
- assert (isinstance(image_patches[0], torch.Tensor)
531
- and len(image_patches[0]) == len(images))
548
+ image_patches = processed_outputs.get("image_patches")
549
+ if image_patches is not None:
550
+ images = mm_data["images"]
551
+ assert isinstance(images, list)
552
533
- processed_outputs["image_patches"] = image_patches[0]
553
+ # Original output: (1, num_images, Pn, Px * Py * C)
554
+ # New output: (num_images, Pn, Px * Py * C)
555
+ assert (isinstance(image_patches, list)
556
+ and len(image_patches) == 1)
557
+ assert (isinstance(image_patches[0], torch.Tensor)
558
+ and len(image_patches[0]) == len(images))
559
535
- return processed_outputs
536
- ```
560
+ processed_outputs["image_patches"] = image_patches[0]
561
+
562
+ return processed_outputs
563
+ ```
564
565
!!! note
566
Our [actual code](gh-file:vllm/model_executor/models/fuyu.py) has special handling
@@ -573,35 +600,37 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
600
It simply repeats each input `image_token` a number of times equal to the number of placeholder feature tokens (`num_image_tokens`).
601
Based on this, we override [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] as follows:
602
576
- ```python
577
- def _get_prompt_updates(
578
- self,
579
- mm_items: MultiModalDataItems,
580
- hf_processor_mm_kwargs: Mapping[str, object],
581
- out_mm_kwargs: MultiModalKwargs,
582
- ) -> Sequence[PromptUpdate]:
583
- hf_config = self.info.get_hf_config()
584
- image_token_id = hf_config.image_token_index
603
+ ??? Code
604
586
- def get_replacement(item_idx: int):
587
- images = mm_items.get_items("image", ImageProcessorItems)
588
-
589
- image_size = images.get_image_size(item_idx)
590
- num_image_tokens = self.info.get_num_image_tokens(
591
- image_width=image_size.width,
592
- image_height=image_size.height,
593
- )
605
+ ```python
606
+ def _get_prompt_updates(
607
+ self,
608
+ mm_items: MultiModalDataItems,
609
+ hf_processor_mm_kwargs: Mapping[str, object],
610
+ out_mm_kwargs: MultiModalKwargs,
611
+ ) -> Sequence[PromptUpdate]:
612
+ hf_config = self.info.get_hf_config()
613
+ image_token_id = hf_config.image_token_index
614
+
615
+ def get_replacement(item_idx: int):
616
+ images = mm_items.get_items("image", ImageProcessorItems)
617
+
618
+ image_size = images.get_image_size(item_idx)
619
+ num_image_tokens = self.info.get_num_image_tokens(
620
+ image_width=image_size.width,
621
+ image_height=image_size.height,
622
+ )
623
595
- return [image_token_id] * num_image_tokens
624
+ return [image_token_id] * num_image_tokens
625
597
- return [
598
- PromptReplacement(
599
- modality="image",
600
- target=[image_token_id],
601
- replacement=get_replacement,
602
- ),
603
- ]
604
- ```
626
+ return [
627
+ PromptReplacement(
628
+ modality="image",
629
+ target=[image_token_id],
630
+ replacement=get_replacement,
631
+ ),
632
+ ]
633
+ ```
634
635
=== "Handling additional tokens: Fuyu"
636
@@ -616,117 +645,90 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
645
646
We define a helper function to return `ncols` and `nrows` directly:
647
619
- ```python
620
- def get_image_feature_grid_size(
621
- self,
622
- *,
623
- image_width: int,
624
- image_height: int,
625
- ) -> tuple[int, int]:
626
- image_processor = self.get_image_processor()
627
- target_width = image_processor.size["width"]
628
- target_height = image_processor.size["height"]
629
- patch_width = image_processor.patch_size["width"]
630
- patch_height = image_processor.patch_size["height"]
631
-
632
- if not (image_width <= target_width and image_height <= target_height):
633
- height_scale_factor = target_height / image_height
634
- width_scale_factor = target_width / image_width
635
- optimal_scale_factor = min(height_scale_factor, width_scale_factor)
636
-
637
- image_height = int(image_height * optimal_scale_factor)
638
- image_width = int(image_width * optimal_scale_factor)
639
-
640
- ncols = math.ceil(image_width / patch_width)
641
- nrows = math.ceil(image_height / patch_height)
642
- return ncols, nrows
643
- ```
648
+ ??? Code
649
+
650
+ ```python
651
+ def get_image_feature_grid_size(
652
+ self,
653
+ *,
654
+ image_width: int,
655
+ image_height: int,
656
+ ) -> tuple[int, int]:
657
+ image_processor = self.get_image_processor()
658
+ target_width = image_processor.size["width"]
659
+ target_height = image_processor.size["height"]
660
+ patch_width = image_processor.patch_size["width"]
661
+ patch_height = image_processor.patch_size["height"]
662
+
663
+ if not (image_width <= target_width and image_height <= target_height):
664
+ height_scale_factor = target_height / image_height
665
+ width_scale_factor = target_width / image_width
666
+ optimal_scale_factor = min(height_scale_factor, width_scale_factor)
667
+
668
+ image_height = int(image_height * optimal_scale_factor)
669
+ image_width = int(image_width * optimal_scale_factor)
670
+
671
+ ncols = math.ceil(image_width / patch_width)
672
+ nrows = math.ceil(image_height / patch_height)
673
+ return ncols, nrows
674
+ ```
675
676
Based on this, we can initially define our replacement tokens as:
677
647
- ```python
648
- def get_replacement(item_idx: int):
649
- images = mm_items.get_items("image", ImageProcessorItems)
650
- image_size = images.get_image_size(item_idx)
678
+ ??? Code
679
652
- ncols, nrows = self.info.get_image_feature_grid_size(
653
- image_width=image_size.width,
654
- image_height=image_size.height,
655
- )
680
+ ```python
681
+ def get_replacement(item_idx: int):
682
+ images = mm_items.get_items("image", ImageProcessorItems)
683
+ image_size = images.get_image_size(item_idx)
684
657
- # `_IMAGE_TOKEN_ID` corresponds to `|SPEAKER|`
658
- # `_NEWLINE_TOKEN_ID` corresponds to `|NEWLINE|`
659
- return ([_IMAGE_TOKEN_ID] * ncols + [_NEWLINE_TOKEN_ID]) * nrows
660
- ```
685
+ ncols, nrows = self.info.get_image_feature_grid_size(
686
+ image_width=image_size.width,
687
+ image_height=image_size.height,
688
+ )
689
+
690
+ # `_IMAGE_TOKEN_ID` corresponds to `|SPEAKER|`
691
+ # `_NEWLINE_TOKEN_ID` corresponds to `|NEWLINE|`
692
+ return ([_IMAGE_TOKEN_ID] * ncols + [_NEWLINE_TOKEN_ID]) * nrows
693
+ ```
694
695
However, this is not entirely correct. After `FuyuImageProcessor.preprocess_with_tokenizer_info` is called,
696
a BOS token (`<s>`) is also added to the promopt:
697
665
- ```python
666
- # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L435
667
- model_image_input = self.image_processor.preprocess_with_tokenizer_info(
668
- image_input=tensor_batch_images,
669
- image_present=image_present,
670
- image_unpadded_h=image_unpadded_heights,
671
- image_unpadded_w=image_unpadded_widths,
672
- image_placeholder_id=image_placeholder_id,
673
- image_newline_id=image_newline_id,
674
- variable_sized=True,
675
- )
676
- prompt_tokens, prompts_length = _tokenize_prompts_with_image_and_batch(
677
- tokenizer=self.tokenizer,
678
- prompts=prompts,
679
- scale_factors=scale_factors,
680
- max_tokens_to_generate=self.max_tokens_to_generate,
681
- max_position_embeddings=self.max_position_embeddings,
682
- add_BOS=True,
683
- add_beginning_of_answer_token=True,
684
- )
685
- ```
698
+ ??? Code
699
+
700
+ ```python
701
+ # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L435
702
+ model_image_input = self.image_processor.preprocess_with_tokenizer_info(
703
+ image_input=tensor_batch_images,
704
+ image_present=image_present,
705
+ image_unpadded_h=image_unpadded_heights,
706
+ image_unpadded_w=image_unpadded_widths,
707
+ image_placeholder_id=image_placeholder_id,
708
+ image_newline_id=image_newline_id,
709
+ variable_sized=True,
710
+ )
711
+ prompt_tokens, prompts_length = _tokenize_prompts_with_image_and_batch(
712
+ tokenizer=self.tokenizer,
713
+ prompts=prompts,
714
+ scale_factors=scale_factors,
715
+ max_tokens_to_generate=self.max_tokens_to_generate,
716
+ max_position_embeddings=self.max_position_embeddings,
717
+ add_BOS=True,
718
+ add_beginning_of_answer_token=True,
719
+ )
720
+ ```
721
722
To assign the vision embeddings to only the image tokens, instead of a string
723
you can return an instance of [PromptUpdateDetails][vllm.multimodal.processing.PromptUpdateDetails]:
724
690
- ```python
691
- hf_config = self.info.get_hf_config()
692
- bos_token_id = hf_config.bos_token_id # `<s>`
693
- assert isinstance(bos_token_id, int)
694
-
695
- def get_replacement_fuyu(item_idx: int):
696
- images = mm_items.get_items("image", ImageProcessorItems)
697
- image_size = images.get_image_size(item_idx)
725
+ ??? Code
726
699
- ncols, nrows = self.info.get_image_feature_grid_size(
700
- image_width=image_size.width,
701
- image_height=image_size.height,
702
- )
703
- image_tokens = ([_IMAGE_TOKEN_ID] * ncols +
704
- [_NEWLINE_TOKEN_ID]) * nrows
705
-
706
- return PromptUpdateDetails.select_token_id(
707
- image_tokens + [bos_token_id],
708
- embed_token_id=_IMAGE_TOKEN_ID,
709
- )
710
- ```
711
-
712
- Finally, noticing that the HF processor removes the `|ENDOFTEXT|` token from the tokenized prompt,
713
- we can search for it to conduct the replacement at the start of the string:
714
-
715
- ```python
716
- def _get_prompt_updates(
717
- self,
718
- mm_items: MultiModalDataItems,
719
- hf_processor_mm_kwargs: Mapping[str, object],
720
- out_mm_kwargs: MultiModalKwargs,
721
- ) -> Sequence[PromptUpdate]:
727
+ ```python
728
hf_config = self.info.get_hf_config()
723
- bos_token_id = hf_config.bos_token_id
729
+ bos_token_id = hf_config.bos_token_id # `<s>`
730
assert isinstance(bos_token_id, int)
731
726
- tokenizer = self.info.get_tokenizer()
727
- eot_token_id = tokenizer.bos_token_id
728
- assert isinstance(eot_token_id, int)
729
-
732
def get_replacement_fuyu(item_idx: int):
733
images = mm_items.get_items("image", ImageProcessorItems)
734
image_size = images.get_image_size(item_idx)
@@ -742,15 +744,52 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
744
image_tokens + [bos_token_id],
745
embed_token_id=_IMAGE_TOKEN_ID,
746
)
747
+ ```
748
746
- return [
747
- PromptReplacement(
748
- modality="image",
749
- target=[eot_token_id],
750
- replacement=get_replacement_fuyu,
751
- )
752
- ]
753
- ```
749
+ Finally, noticing that the HF processor removes the `|ENDOFTEXT|` token from the tokenized prompt,
750
+ we can search for it to conduct the replacement at the start of the string:
751
+
752
+ ??? Code
753
+
754
+ ```python
755
+ def _get_prompt_updates(
756
+ self,
757
+ mm_items: MultiModalDataItems,
758
+ hf_processor_mm_kwargs: Mapping[str, object],
759
+ out_mm_kwargs: MultiModalKwargs,
760
+ ) -> Sequence[PromptUpdate]:
761
+ hf_config = self.info.get_hf_config()
762
+ bos_token_id = hf_config.bos_token_id
763
+ assert isinstance(bos_token_id, int)
764
+
765
+ tokenizer = self.info.get_tokenizer()
766
+ eot_token_id = tokenizer.bos_token_id
767
+ assert isinstance(eot_token_id, int)
768
+
769
+ def get_replacement_fuyu(item_idx: int):
770
+ images = mm_items.get_items("image", ImageProcessorItems)
771
+ image_size = images.get_image_size(item_idx)
772
+
773
+ ncols, nrows = self.info.get_image_feature_grid_size(
774
+ image_width=image_size.width,
775
+ image_height=image_size.height,
776
+ )
777
+ image_tokens = ([_IMAGE_TOKEN_ID] * ncols +
778
+ [_NEWLINE_TOKEN_ID]) * nrows
779
+
780
+ return PromptUpdateDetails.select_token_id(
781
+ image_tokens + [bos_token_id],
782
+ embed_token_id=_IMAGE_TOKEN_ID,
783
+ )
784
+
785
+ return [
786
+ PromptReplacement(
787
+ modality="image",
788
+ target=[eot_token_id],
789
+ replacement=get_replacement_fuyu,
790
+ )
791
+ ]
792
+ ```
793
794
## 5. Register processor-related classes
795
docs/contributing/profiling.md
+20
-20
@@ -97,26 +97,26 @@ to manually kill the profiler and generate your `nsys-rep` report.
97
98
You can view these profiles either as summaries in the CLI, using `nsys stats [profile-file]`, or in the GUI by installing Nsight [locally following the directions here](https://developer.nvidia.com/nsight-systems/get-started).
99
100
-CLI example:
101
-
102
-```bash
103
-nsys stats report1.nsys-rep
104
-...
105
- ** CUDA GPU Kernel Summary (cuda_gpu_kern_sum):
106
-
107
- Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name
108
- -------- --------------- --------- ----------- ----------- -------- --------- ----------- ----------------------------------------------------------------------------------------------------
109
- 46.3 10,327,352,338 17,505 589,965.9 144,383.0 27,040 3,126,460 944,263.8 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize128x128x64_warpgroupsize1x1x1_execute_segment_k_of…
110
- 14.8 3,305,114,764 5,152 641,520.7 293,408.0 287,296 2,822,716 867,124.9 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize256x128x64_warpgroupsize2x1x1_execute_segment_k_of…
111
- 12.1 2,692,284,876 14,280 188,535.4 83,904.0 19,328 2,862,237 497,999.9 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize64x128x64_warpgroupsize1x1x1_execute_segment_k_off…
112
- 9.5 2,116,600,578 33,920 62,399.8 21,504.0 15,326 2,532,285 290,954.1 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize64x64x64_warpgroupsize1x1x1_execute_segment_k_off_…
113
- 5.0 1,119,749,165 18,912 59,208.4 9,056.0 6,784 2,578,366 271,581.7 void vllm::act_and_mul_kernel<c10::BFloat16, &vllm::silu_kernel<c10::BFloat16>, (bool)1>(T1 *, cons…
114
- 4.1 916,662,515 21,312 43,011.6 19,776.0 8,928 2,586,205 199,790.1 void cutlass::device_kernel<flash::enable_sm90_or_later<flash::FlashAttnFwdSm90<flash::CollectiveMa…
115
- 2.6 587,283,113 37,824 15,526.7 3,008.0 2,719 2,517,756 139,091.1 std::enable_if<T2>(int)0&&vllm::_typeConvert<T1>::exists, void>::type vllm::fused_add_rms_norm_kern…
116
- 1.9 418,362,605 18,912 22,121.5 3,871.0 3,328 2,523,870 175,248.2 void vllm::rotary_embedding_kernel<c10::BFloat16, (bool)1>(const long *, T1 *, T1 *, const T1 *, in…
117
- 0.7 167,083,069 18,880 8,849.7 2,240.0 1,471 2,499,996 101,436.1 void vllm::reshape_and_cache_flash_kernel<__nv_bfloat16, __nv_bfloat16, (vllm::Fp8KVCacheDataType)0…
118
-...
119
-```
100
+??? CLI example
101
+
102
+ ```bash
103
+ nsys stats report1.nsys-rep
104
+ ...
105
+ ** CUDA GPU Kernel Summary (cuda_gpu_kern_sum):
106
+
107
+ Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name
108
+ -------- --------------- --------- ----------- ----------- -------- --------- ----------- ----------------------------------------------------------------------------------------------------
109
+ 46.3 10,327,352,338 17,505 589,965.9 144,383.0 27,040 3,126,460 944,263.8 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize128x128x64_warpgroupsize1x1x1_execute_segment_k_of…
110
+ 14.8 3,305,114,764 5,152 641,520.7 293,408.0 287,296 2,822,716 867,124.9 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize256x128x64_warpgroupsize2x1x1_execute_segment_k_of…
111
+ 12.1 2,692,284,876 14,280 188,535.4 83,904.0 19,328 2,862,237 497,999.9 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize64x128x64_warpgroupsize1x1x1_execute_segment_k_off…
112
+ 9.5 2,116,600,578 33,920 62,399.8 21,504.0 15,326 2,532,285 290,954.1 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize64x64x64_warpgroupsize1x1x1_execute_segment_k_off_…
113
+ 5.0 1,119,749,165 18,912 59,208.4 9,056.0 6,784 2,578,366 271,581.7 void vllm::act_and_mul_kernel<c10::BFloat16, &vllm::silu_kernel<c10::BFloat16>, (bool)1>(T1 *, cons…
114
+ 4.1 916,662,515 21,312 43,011.6 19,776.0 8,928 2,586,205 199,790.1 void cutlass::device_kernel<flash::enable_sm90_or_later<flash::FlashAttnFwdSm90<flash::CollectiveMa…
115
+ 2.6 587,283,113 37,824 15,526.7 3,008.0 2,719 2,517,756 139,091.1 std::enable_if<T2>(int)0&&vllm::_typeConvert<T1>::exists, void>::type vllm::fused_add_rms_norm_kern…
116
+ 1.9 418,362,605 18,912 22,121.5 3,871.0 3,328 2,523,870 175,248.2 void vllm::rotary_embedding_kernel<c10::BFloat16, (bool)1>(const long *, T1 *, T1 *, const T1 *, in…
117
+ 0.7 167,083,069 18,880 8,849.7 2,240.0 1,471 2,499,996 101,436.1 void vllm::reshape_and_cache_flash_kernel<__nv_bfloat16, __nv_bfloat16, (vllm::Fp8KVCacheDataType)0…
118
+ ...
119
+ ```
120
121
GUI example:
122
docs/deployment/docker.md
+15
-13
@@ -97,19 +97,21 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--
97
flags to speed up build process. However, ensure your `max_jobs` is substantially larger than `nvcc_threads` to get the most benefits.
98
Keep an eye on memory usage with parallel jobs as it can be substantial (see example below).
99
100
-```console
101
-# Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
102
-python3 use_existing_torch.py
103
-DOCKER_BUILDKIT=1 docker build . \
104
- --file docker/Dockerfile \
105
- --target vllm-openai \
106
- --platform "linux/arm64" \
107
- -t vllm/vllm-gh200-openai:latest \
108
- --build-arg max_jobs=66 \
109
- --build-arg nvcc_threads=2 \
110
- --build-arg torch_cuda_arch_list="9.0 10.0+PTX" \
111
- --build-arg vllm_fa_cmake_gpu_arches="90-real"
112
-```
100
+??? Command
101
+
102
+ ```console
103
+ # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
104
+ python3 use_existing_torch.py
105
+ DOCKER_BUILDKIT=1 docker build . \
106
+ --file docker/Dockerfile \
107
+ --target vllm-openai \
108
+ --platform "linux/arm64" \
109
+ -t vllm/vllm-gh200-openai:latest \
110
+ --build-arg max_jobs=66 \
111
+ --build-arg nvcc_threads=2 \
112
+ --build-arg torch_cuda_arch_list="9.0 10.0+PTX" \
113
+ --build-arg vllm_fa_cmake_gpu_arches="90-real"
114
+ ```
115
116
!!! note
117
If you are building the `linux/arm64` image on a non-ARM host (e.g., an x86_64 machine), you need to ensure your system is set up for cross-compilation using QEMU. This allows your host machine to emulate ARM64 execution.
docs/deployment/frameworks/autogen.md
+47
-45
@@ -30,51 +30,53 @@ python -m vllm.entrypoints.openai.api_server \
30
31
- Call it with AutoGen:
32
33
-```python
34
-import asyncio
35
-from autogen_core.models import UserMessage
36
-from autogen_ext.models.openai import OpenAIChatCompletionClient
37
-from autogen_core.models import ModelFamily
38
-
39
-
40
-async def main() -> None:
41
- # Create a model client
42
- model_client = OpenAIChatCompletionClient(
43
- model="mistralai/Mistral-7B-Instruct-v0.2",
44
- base_url="http://{your-vllm-host-ip}:{your-vllm-host-port}/v1",
45
- api_key="EMPTY",
46
- model_info={
47
- "vision": False,
48
- "function_calling": False,
49
- "json_output": False,
50
- "family": ModelFamily.MISTRAL,
51
- "structured_output": True,
52
- },
53
- )
54
-
55
- messages = [UserMessage(content="Write a very short story about a dragon.", source="user")]
56
-
57
- # Create a stream.
58
- stream = model_client.create_stream(messages=messages)
59
-
60
- # Iterate over the stream and print the responses.
61
- print("Streamed responses:")
62
- async for response in stream:
63
- if isinstance(response, str):
64
- # A partial response is a string.
65
- print(response, flush=True, end="")
66
- else:
67
- # The last response is a CreateResult object with the complete message.
68
- print("\n\n------------\n")
69
- print("The complete response:", flush=True)
70
- print(response.content, flush=True)
71
-
72
- # Close the client when done.
73
- await model_client.close()
74
-
75
-
76
-asyncio.run(main())
77
-```
33
+??? Code
34
+
35
+ ```python
36
+ import asyncio
37
+ from autogen_core.models import UserMessage
38
+ from autogen_ext.models.openai import OpenAIChatCompletionClient
39
+ from autogen_core.models import ModelFamily
40
+
41
+
42
+ async def main() -> None:
43
+ # Create a model client
44
+ model_client = OpenAIChatCompletionClient(
45
+ model="mistralai/Mistral-7B-Instruct-v0.2",
46
+ base_url="http://{your-vllm-host-ip}:{your-vllm-host-port}/v1",
47
+ api_key="EMPTY",
48
+ model_info={
49
+ "vision": False,
50
+ "function_calling": False,
51
+ "json_output": False,
52
+ "family": ModelFamily.MISTRAL,
53
+ "structured_output": True,
54
+ },
55
+ )
56
+
57
+ messages = [UserMessage(content="Write a very short story about a dragon.", source="user")]
58
+
59
+ # Create a stream.
60
+ stream = model_client.create_stream(messages=messages)
61
+
62
+ # Iterate over the stream and print the responses.
63
+ print("Streamed responses:")
64
+ async for response in stream:
65
+ if isinstance(response, str):
66
+ # A partial response is a string.
67
+ print(response, flush=True, end="")
68
+ else:
69
+ # The last response is a CreateResult object with the complete message.
70
+ print("\n\n------------\n")
71
+ print("The complete response:", flush=True)
72
+ print(response.content, flush=True)
73
+
74
+ # Close the client when done.
75
+ await model_client.close()
76
+
77
+
78
+ asyncio.run(main())
79
+ ```
80
81
For details, see the tutorial:
82
docs/deployment/frameworks/cerebrium.md
+59
-53
@@ -34,25 +34,27 @@ vllm = "latest"
34
35
Next, let us add our code to handle inference for the LLM of your choice (`mistralai/Mistral-7B-Instruct-v0.1` for this example), add the following code to your `main.py`:
36
37
-```python
38
-from vllm import LLM, SamplingParams
37
+??? Code
38
40
-llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.1")
39
+ ```python
40
+ from vllm import LLM, SamplingParams
41
42
-def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95):
42
+ llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.1")
43
44
- sampling_params = SamplingParams(temperature=temperature, top_p=top_p)
45
- outputs = llm.generate(prompts, sampling_params)
44
+ def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95):
45
47
- # Print the outputs.
48
- results = []
49
- for output in outputs:
50
- prompt = output.prompt
51
- generated_text = output.outputs[0].text
52
- results.append({"prompt": prompt, "generated_text": generated_text})
46
+ sampling_params = SamplingParams(temperature=temperature, top_p=top_p)
47
+ outputs = llm.generate(prompts, sampling_params)
48
54
- return {"results": results}
55
-```
49
+ # Print the outputs.
50
+ results = []
51
+ for output in outputs:
52
+ prompt = output.prompt
53
+ generated_text = output.outputs[0].text
54
+ results.append({"prompt": prompt, "generated_text": generated_text})
55
+
56
+ return {"results": results}
57
+ ```
58
59
Then, run the following code to deploy it to the cloud:
60
@@ -62,47 +64,51 @@ cerebrium deploy
64
65
If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case`/run`)
66
65
-```python
66
-curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
67
- -H 'Content-Type: application/json' \
68
- -H 'Authorization: <JWT TOKEN>' \
69
- --data '{
70
- "prompts": [
71
- "Hello, my name is",
72
- "The president of the United States is",
73
- "The capital of France is",
74
- "The future of AI is"
75
- ]
76
- }'
77
-```
67
+??? Command
68
+
69
+ ```python
70
+ curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
71
+ -H 'Content-Type: application/json' \
72
+ -H 'Authorization: <JWT TOKEN>' \
73
+ --data '{
74
+ "prompts": [
75
+ "Hello, my name is",
76
+ "The president of the United States is",
77
+ "The capital of France is",
78
+ "The future of AI is"
79
+ ]
80
+ }'
81
+ ```
82
83
You should get a response like:
84
81
-```python
82
-{
83
- "run_id": "52911756-3066-9ae8-bcc9-d9129d1bd262",
84
- "result": {
85
- "result": [
86
- {
87
- "prompt": "Hello, my name is",
88
- "generated_text": " Sarah, and I'm a teacher. I teach elementary school students. One of"
89
- },
90
- {
91
- "prompt": "The president of the United States is",
92
- "generated_text": " elected every four years. This is a democratic system.\n\n5. What"
93
- },
94
- {
95
- "prompt": "The capital of France is",
96
- "generated_text": " Paris.\n"
97
- },
98
- {
99
- "prompt": "The future of AI is",
100
- "generated_text": " bright, but it's important to approach it with a balanced and nuanced perspective."
101
- }
102
- ]
103
- },
104
- "run_time_ms": 152.53663063049316
105
-}
106
-```
85
+??? Response
86
+
87
+ ```python
88
+ {
89
+ "run_id": "52911756-3066-9ae8-bcc9-d9129d1bd262",
90
+ "result": {
91
+ "result": [
92
+ {
93
+ "prompt": "Hello, my name is",
94
+ "generated_text": " Sarah, and I'm a teacher. I teach elementary school students. One of"
95
+ },
96
+ {
97
+ "prompt": "The president of the United States is",
98
+ "generated_text": " elected every four years. This is a democratic system.\n\n5. What"
99
+ },
100
+ {
101
+ "prompt": "The capital of France is",
102
+ "generated_text": " Paris.\n"
103
+ },
104
+ {
105
+ "prompt": "The future of AI is",
106
+ "generated_text": " bright, but it's important to approach it with a balanced and nuanced perspective."
107
+ }
108
+ ]
109
+ },
110
+ "run_time_ms": 152.53663063049316
111
+ }
112
+ ```
113
114
You now have an autoscaling endpoint where you only pay for the compute you use!
docs/deployment/frameworks/dstack.md
+69
-63
@@ -26,75 +26,81 @@ dstack init
26
27
Next, to provision a VM instance with LLM of your choice (`NousResearch/Llama-2-7b-chat-hf` for this example), create the following `serve.dstack.yml` file for the dstack `Service`:
28
29
-```yaml
30
-type: service
31
-
32
-python: "3.11"
33
-env:
34
- - MODEL=NousResearch/Llama-2-7b-chat-hf
35
-port: 8000
36
-resources:
37
- gpu: 24GB
38
-commands:
39
- - pip install vllm
40
- - vllm serve $MODEL --port 8000
41
-model:
42
- format: openai
43
- type: chat
44
- name: NousResearch/Llama-2-7b-chat-hf
45
-```
29
+??? Config
30
+
31
+ ```yaml
32
+ type: service
33
+
34
+ python: "3.11"
35
+ env:
36
+ - MODEL=NousResearch/Llama-2-7b-chat-hf
37
+ port: 8000
38
+ resources:
39
+ gpu: 24GB
40
+ commands:
41
+ - pip install vllm
42
+ - vllm serve $MODEL --port 8000
43
+ model:
44
+ format: openai
45
+ type: chat
46
+ name: NousResearch/Llama-2-7b-chat-hf
47
+ ```
48
49
Then, run the following CLI for provisioning:
50
49
-```console
50
-$ dstack run . -f serve.dstack.yml
51
-
52
-⠸ Getting run plan...
53
- Configuration serve.dstack.yml
54
- Project deep-diver-main
55
- User deep-diver
56
- Min resources 2..xCPU, 8GB.., 1xGPU (24GB)
57
- Max price -
58
- Max duration -
59
- Spot policy auto
60
- Retry policy no
61
-
62
- # BACKEND REGION INSTANCE RESOURCES SPOT PRICE
63
- 1 gcp us-central1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804
64
- 2 gcp us-east1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804
65
- 3 gcp us-west1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804
66
- ...
67
- Shown 3 of 193 offers, $5.876 max
68
-
69
-Continue? [y/n]: y
70
-⠙ Submitting run...
71
-⠏ Launching spicy-treefrog-1 (pulling)
72
-spicy-treefrog-1 provisioning completed (running)
73
-Service is published at ...
74
-```
51
+??? Command
52
+
53
+ ```console
54
+ $ dstack run . -f serve.dstack.yml
55
+
56
+ ⠸ Getting run plan...
57
+ Configuration serve.dstack.yml
58
+ Project deep-diver-main
59
+ User deep-diver
60
+ Min resources 2..xCPU, 8GB.., 1xGPU (24GB)
61
+ Max price -
62
+ Max duration -
63
+ Spot policy auto
64
+ Retry policy no
65
+
66
+ # BACKEND REGION INSTANCE RESOURCES SPOT PRICE
67
+ 1 gcp us-central1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804
68
+ 2 gcp us-east1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804
69
+ 3 gcp us-west1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804
70
+ ...
71
+ Shown 3 of 193 offers, $5.876 max
72
+
73
+ Continue? [y/n]: y
74
+ ⠙ Submitting run...
75
+ ⠏ Launching spicy-treefrog-1 (pulling)
76
+ spicy-treefrog-1 provisioning completed (running)
77
+ Service is published at ...
78
+ ```
79
80
After the provisioning, you can interact with the model by using the OpenAI SDK:
81
78
-```python
79
-from openai import OpenAI
80
-
81
-client = OpenAI(
82
- base_url="https://gateway.<gateway domain>",
83
- api_key="<YOUR-DSTACK-SERVER-ACCESS-TOKEN>"
84
-)
85
-
86
-completion = client.chat.completions.create(
87
- model="NousResearch/Llama-2-7b-chat-hf",
88
- messages=[
89
- {
90
- "role": "user",
91
- "content": "Compose a poem that explains the concept of recursion in programming.",
92
- }
93
- ]
94
-)
95
-
96
-print(completion.choices[0].message.content)
97
-```
82
+??? Code
83
+
84
+ ```python
85
+ from openai import OpenAI
86
+
87
+ client = OpenAI(
88
+ base_url="https://gateway.<gateway domain>",
89
+ api_key="<YOUR-DSTACK-SERVER-ACCESS-TOKEN>"
90
+ )
91
+
92
+ completion = client.chat.completions.create(
93
+ model="NousResearch/Llama-2-7b-chat-hf",
94
+ messages=[
95
+ {
96
+ "role": "user",
97
+ "content": "Compose a poem that explains the concept of recursion in programming.",
98
+ }
99
+ ]
100
+ )
101
+
102
+ print(completion.choices[0].message.content)
103
+ ```
104
105
!!! note
106
dstack automatically handles authentication on the gateway using dstack's tokens. Meanwhile, if you don't want to configure a gateway, you can provision dstack `Task` instead of `Service`. The `Task` is for development purpose only. If you want to know more about hands-on materials how to serve vLLM using dstack, check out [this repository](https://github.com/dstackai/dstack-examples/tree/main/deployment/vllm)
docs/deployment/frameworks/haystack.md
+23
-23
@@ -27,29 +27,29 @@ vllm serve mistralai/Mistral-7B-Instruct-v0.1
27
28
- Use the `OpenAIGenerator` and `OpenAIChatGenerator` components in Haystack to query the vLLM server.
29
30
-```python
31
-from haystack.components.generators.chat import OpenAIChatGenerator
32
-from haystack.dataclasses import ChatMessage
33
-from haystack.utils import Secret
34
-
35
-generator = OpenAIChatGenerator(
36
- # for compatibility with the OpenAI API, a placeholder api_key is needed
37
- api_key=Secret.from_token("VLLM-PLACEHOLDER-API-KEY"),
38
- model="mistralai/Mistral-7B-Instruct-v0.1",
39
- api_base_url="http://{your-vLLM-host-ip}:{your-vLLM-host-port}/v1",
40
- generation_kwargs = {"max_tokens": 512}
41
-)
42
-
43
-response = generator.run(
44
- messages=[ChatMessage.from_user("Hi. Can you help me plan my next trip to Italy?")]
45
-)
46
-
47
-print("-"*30)
48
-print(response)
49
-print("-"*30)
50
-```
51
-
52
-Output e.g.:
30
+??? Code
31
+
32
+ ```python
33
+ from haystack.components.generators.chat import OpenAIChatGenerator
34
+ from haystack.dataclasses import ChatMessage
35
+ from haystack.utils import Secret
36
+
37
+ generator = OpenAIChatGenerator(
38
+ # for compatibility with the OpenAI API, a placeholder api_key is needed
39
+ api_key=Secret.from_token("VLLM-PLACEHOLDER-API-KEY"),
40
+ model="mistralai/Mistral-7B-Instruct-v0.1",
41
+ api_base_url="http://{your-vLLM-host-ip}:{your-vLLM-host-port}/v1",
42
+ generation_kwargs = {"max_tokens": 512}
43
+ )
44
+
45
+ response = generator.run(
46
+ messages=[ChatMessage.from_user("Hi. Can you help me plan my next trip to Italy?")]
47
+ )
48
+
49
+ print("-"*30)
50
+ print(response)
51
+ print("-"*30)
52
+ ```
53
54
```console
55
------------------------------
docs/deployment/frameworks/litellm.md
+15
-13
@@ -34,21 +34,23 @@ vllm serve qwen/Qwen1.5-0.5B-Chat
34
35
- Call it with litellm:
36
37
-```python
38
-import litellm
37
+??? Code
38
40
-messages = [{ "content": "Hello, how are you?","role": "user"}]
39
+ ```python
40
+ import litellm
41
42
-# hosted_vllm is prefix key word and necessary
43
-response = litellm.completion(
44
- model="hosted_vllm/qwen/Qwen1.5-0.5B-Chat", # pass the vllm model name
45
- messages=messages,
46
- api_base="http://{your-vllm-server-host}:{your-vllm-server-port}/v1",
47
- temperature=0.2,
48
- max_tokens=80)
49
-
50
-print(response)
51
-```
42
+ messages = [{ "content": "Hello, how are you?","role": "user"}]
43
+
44
+ # hosted_vllm is prefix key word and necessary
45
+ response = litellm.completion(
46
+ model="hosted_vllm/qwen/Qwen1.5-0.5B-Chat", # pass the vllm model name
47
+ messages=messages,
48
+ api_base="http://{your-vllm-server-host}:{your-vllm-server-port}/v1",
49
+ temperature=0.2,
50
+ max_tokens=80)
51
+
52
+ print(response)
53
+ ```
54
55
### Embeddings
56
docs/deployment/frameworks/lws.md
+117
-113
@@ -17,99 +17,101 @@ vLLM can be deployed with [LWS](https://github.com/kubernetes-sigs/lws) on Kuber
17
18
Deploy the following yaml file `lws.yaml`
19
20
-```yaml
21
-apiVersion: leaderworkerset.x-k8s.io/v1
22
-kind: LeaderWorkerSet
23
-metadata:
24
- name: vllm
25
-spec:
26
- replicas: 2
27
- leaderWorkerTemplate:
28
- size: 2
29
- restartPolicy: RecreateGroupOnPodRestart
30
- leaderTemplate:
31
- metadata:
32
- labels:
33
- role: leader
34
- spec:
35
- containers:
36
- - name: vllm-leader
37
- image: docker.io/vllm/vllm-openai:latest
38
- env:
39
- - name: HUGGING_FACE_HUB_TOKEN
40
- value: <your-hf-token>
41
- command:
42
- - sh
43
- - -c
44
- - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
45
- python3 -m vllm.entrypoints.openai.api_server --port 8080 --model meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline_parallel_size 2"
46
- resources:
47
- limits:
48
- nvidia.com/gpu: "8"
49
- memory: 1124Gi
50
- ephemeral-storage: 800Gi
51
- requests:
52
- ephemeral-storage: 800Gi
53
- cpu: 125
54
- ports:
55
- - containerPort: 8080
56
- readinessProbe:
57
- tcpSocket:
58
- port: 8080
59
- initialDelaySeconds: 15
60
- periodSeconds: 10
61
- volumeMounts:
62
- - mountPath: /dev/shm
63
- name: dshm
64
- volumes:
65
- - name: dshm
66
- emptyDir:
67
- medium: Memory
68
- sizeLimit: 15Gi
69
- workerTemplate:
70
- spec:
71
- containers:
72
- - name: vllm-worker
73
- image: docker.io/vllm/vllm-openai:latest
74
- command:
75
- - sh
76
- - -c
77
- - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"
78
- resources:
79
- limits:
80
- nvidia.com/gpu: "8"
81
- memory: 1124Gi
82
- ephemeral-storage: 800Gi
83
- requests:
84
- ephemeral-storage: 800Gi
85
- cpu: 125
86
- env:
87
- - name: HUGGING_FACE_HUB_TOKEN
88
- value: <your-hf-token>
89
- volumeMounts:
90
- - mountPath: /dev/shm
91
- name: dshm
92
- volumes:
93
- - name: dshm
94
- emptyDir:
95
- medium: Memory
96
- sizeLimit: 15Gi
97
----
98
-apiVersion: v1
99
-kind: Service
100
-metadata:
101
- name: vllm-leader
102
-spec:
103
- ports:
104
- - name: http
105
- port: 8080
106
- protocol: TCP
107
- targetPort: 8080
108
- selector:
109
- leaderworkerset.sigs.k8s.io/name: vllm
110
- role: leader
111
- type: ClusterIP
112
-```
20
+??? Yaml
21
+
22
+ ```yaml
23
+ apiVersion: leaderworkerset.x-k8s.io/v1
24
+ kind: LeaderWorkerSet
25
+ metadata:
26
+ name: vllm
27
+ spec:
28
+ replicas: 2
29
+ leaderWorkerTemplate:
30
+ size: 2
31
+ restartPolicy: RecreateGroupOnPodRestart
32
+ leaderTemplate:
33
+ metadata:
34
+ labels:
35
+ role: leader
36
+ spec:
37
+ containers:
38
+ - name: vllm-leader
39
+ image: docker.io/vllm/vllm-openai:latest
40
+ env:
41
+ - name: HUGGING_FACE_HUB_TOKEN
42
+ value: <your-hf-token>
43
+ command:
44
+ - sh
45
+ - -c
46
+ - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
47
+ python3 -m vllm.entrypoints.openai.api_server --port 8080 --model meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline_parallel_size 2"
48
+ resources:
49
+ limits:
50
+ nvidia.com/gpu: "8"
51
+ memory: 1124Gi
52
+ ephemeral-storage: 800Gi
53
+ requests:
54
+ ephemeral-storage: 800Gi
55
+ cpu: 125
56
+ ports:
57
+ - containerPort: 8080
58
+ readinessProbe:
59
+ tcpSocket:
60
+ port: 8080
61
+ initialDelaySeconds: 15
62
+ periodSeconds: 10
63
+ volumeMounts:
64
+ - mountPath: /dev/shm
65
+ name: dshm
66
+ volumes:
67
+ - name: dshm
68
+ emptyDir:
69
+ medium: Memory
70
+ sizeLimit: 15Gi
71
+ workerTemplate:
72
+ spec:
73
+ containers:
74
+ - name: vllm-worker
75
+ image: docker.io/vllm/vllm-openai:latest
76
+ command:
77
+ - sh
78
+ - -c
79
+ - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"
80
+ resources:
81
+ limits:
82
+ nvidia.com/gpu: "8"
83
+ memory: 1124Gi
84
+ ephemeral-storage: 800Gi
85
+ requests:
86
+ ephemeral-storage: 800Gi
87
+ cpu: 125
88
+ env:
89
+ - name: HUGGING_FACE_HUB_TOKEN
90
+ value: <your-hf-token>
91
+ volumeMounts:
92
+ - mountPath: /dev/shm
93
+ name: dshm
94
+ volumes:
95
+ - name: dshm
96
+ emptyDir:
97
+ medium: Memory
98
+ sizeLimit: 15Gi
99
+ ---
100
+ apiVersion: v1
101
+ kind: Service
102
+ metadata:
103
+ name: vllm-leader
104
+ spec:
105
+ ports:
106
+ - name: http
107
+ port: 8080
108
+ protocol: TCP
109
+ targetPort: 8080
110
+ selector:
111
+ leaderworkerset.sigs.k8s.io/name: vllm
112
+ role: leader
113
+ type: ClusterIP
114
+ ```
115
116
```bash
117
kubectl apply -f lws.yaml
@@ -175,25 +177,27 @@ curl http://localhost:8080/v1/completions \
177
178
The output should be similar to the following
179
178
-```text
179
-{
180
- "id": "cmpl-1bb34faba88b43f9862cfbfb2200949d",
181
- "object": "text_completion",
182
- "created": 1715138766,
183
- "model": "meta-llama/Meta-Llama-3.1-405B-Instruct",
184
- "choices": [
180
+??? Output
181
+
182
+ ```text
183
{
186
- "index": 0,
187
- "text": " top destination for foodies, with",
188
- "logprobs": null,
189
- "finish_reason": "length",
190
- "stop_reason": null
184
+ "id": "cmpl-1bb34faba88b43f9862cfbfb2200949d",
185
+ "object": "text_completion",
186
+ "created": 1715138766,
187
+ "model": "meta-llama/Meta-Llama-3.1-405B-Instruct",
188
+ "choices": [
189
+ {
190
+ "index": 0,
191
+ "text": " top destination for foodies, with",
192
+ "logprobs": null,
193
+ "finish_reason": "length",
194
+ "stop_reason": null
195
+ }
196
+ ],
197
+ "usage": {
198
+ "prompt_tokens": 5,
199
+ "total_tokens": 12,
200
+ "completion_tokens": 7
201
+ }
202
}
192
- ],
193
- "usage": {
194
- "prompt_tokens": 5,
195
- "total_tokens": 12,
196
- "completion_tokens": 7
197
- }
198
-}
199
-```
203
+ ```
docs/deployment/frameworks/skypilot.md
+200
-206
@@ -24,48 +24,50 @@ sky check
24
25
See the vLLM SkyPilot YAML for serving, [serving.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve.yaml).
26
27
-```yaml
28
-resources:
29
- accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
30
- use_spot: True
31
- disk_size: 512 # Ensure model checkpoints can fit.
32
- disk_tier: best
33
- ports: 8081 # Expose to internet traffic.
34
-
35
-envs:
36
- MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
37
- HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
38
-
39
-setup: |
40
- conda create -n vllm python=3.10 -y
41
- conda activate vllm
42
-
43
- pip install vllm==0.4.0.post1
44
- # Install Gradio for web UI.
45
- pip install gradio openai
46
- pip install flash-attn==2.5.7
47
-
48
-run: |
49
- conda activate vllm
50
- echo 'Starting vllm api server...'
51
- python -u -m vllm.entrypoints.openai.api_server \
52
- --port 8081 \
53
- --model $MODEL_NAME \
54
- --trust-remote-code \
55
- --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
56
- 2>&1 | tee api_server.log &
57
-
58
- echo 'Waiting for vllm api server to start...'
59
- while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
60
-
61
- echo 'Starting gradio server...'
62
- git clone https://github.com/vllm-project/vllm.git || true
63
- python vllm/examples/online_serving/gradio_openai_chatbot_webserver.py \
64
- -m $MODEL_NAME \
65
- --port 8811 \
66
- --model-url http://localhost:8081/v1 \
67
- --stop-token-ids 128009,128001
68
-```
27
+??? Yaml
28
+
29
+ ```yaml
30
+ resources:
31
+ accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
32
+ use_spot: True
33
+ disk_size: 512 # Ensure model checkpoints can fit.
34
+ disk_tier: best
35
+ ports: 8081 # Expose to internet traffic.
36
+
37
+ envs:
38
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
39
+ HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
40
+
41
+ setup: |
42
+ conda create -n vllm python=3.10 -y
43
+ conda activate vllm
44
+
45
+ pip install vllm==0.4.0.post1
46
+ # Install Gradio for web UI.
47
+ pip install gradio openai
48
+ pip install flash-attn==2.5.7
49
+
50
+ run: |
51
+ conda activate vllm
52
+ echo 'Starting vllm api server...'
53
+ python -u -m vllm.entrypoints.openai.api_server \
54
+ --port 8081 \
55
+ --model $MODEL_NAME \
56
+ --trust-remote-code \
57
+ --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
58
+ 2>&1 | tee api_server.log &
59
+
60
+ echo 'Waiting for vllm api server to start...'
61
+ while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done
62
+
63
+ echo 'Starting gradio server...'
64
+ git clone https://github.com/vllm-project/vllm.git || true
65
+ python vllm/examples/online_serving/gradio_openai_chatbot_webserver.py \
66
+ -m $MODEL_NAME \
67
+ --port 8811 \
68
+ --model-url http://localhost:8081/v1 \
69
+ --stop-token-ids 128009,128001
70
+ ```
71
72
Start the serving the Llama-3 8B model on any of the candidate GPUs listed (L4, A10g, ...):
73
@@ -93,68 +95,67 @@ HF_TOKEN="your-huggingface-token" \
95
96
SkyPilot can scale up the service to multiple service replicas with built-in autoscaling, load-balancing and fault-tolerance. You can do it by adding a services section to the YAML file.
97
96
-```yaml
97
-service:
98
- replicas: 2
99
- # An actual request for readiness probe.
100
- readiness_probe:
101
- path: /v1/chat/completions
102
- post_data:
103
- model: $MODEL_NAME
104
- messages:
105
- - role: user
106
- content: Hello! What is your name?
107
- max_completion_tokens: 1
108
-```
109
-
110
-<details>
111
-<summary>Click to see the full recipe YAML</summary>
112
-
113
-```yaml
114
-service:
115
- replicas: 2
116
- # An actual request for readiness probe.
117
- readiness_probe:
118
- path: /v1/chat/completions
119
- post_data:
120
- model: $MODEL_NAME
121
- messages:
122
- - role: user
123
- content: Hello! What is your name?
98
+??? Yaml
99
+
100
+ ```yaml
101
+ service:
102
+ replicas: 2
103
+ # An actual request for readiness probe.
104
+ readiness_probe:
105
+ path: /v1/chat/completions
106
+ post_data:
107
+ model: $MODEL_NAME
108
+ messages:
109
+ - role: user
110
+ content: Hello! What is your name?
111
max_completion_tokens: 1
112
+ ```
113
126
-resources:
127
- accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
128
- use_spot: True
129
- disk_size: 512 # Ensure model checkpoints can fit.
130
- disk_tier: best
131
- ports: 8081 # Expose to internet traffic.
132
-
133
-envs:
134
- MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
135
- HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
136
-
137
-setup: |
138
- conda create -n vllm python=3.10 -y
139
- conda activate vllm
140
-
141
- pip install vllm==0.4.0.post1
142
- # Install Gradio for web UI.
143
- pip install gradio openai
144
- pip install flash-attn==2.5.7
145
-
146
-run: |
147
- conda activate vllm
148
- echo 'Starting vllm api server...'
149
- python -u -m vllm.entrypoints.openai.api_server \
150
- --port 8081 \
151
- --model $MODEL_NAME \
152
- --trust-remote-code \
153
- --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
154
- 2>&1 | tee api_server.log
155
-```
156
-
157
-</details>
114
+??? Yaml
115
+
116
+ ```yaml
117
+ service:
118
+ replicas: 2
119
+ # An actual request for readiness probe.
120
+ readiness_probe:
121
+ path: /v1/chat/completions
122
+ post_data:
123
+ model: $MODEL_NAME
124
+ messages:
125
+ - role: user
126
+ content: Hello! What is your name?
127
+ max_completion_tokens: 1
128
+
129
+ resources:
130
+ accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
131
+ use_spot: True
132
+ disk_size: 512 # Ensure model checkpoints can fit.
133
+ disk_tier: best
134
+ ports: 8081 # Expose to internet traffic.
135
+
136
+ envs:
137
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
138
+ HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
139
+
140
+ setup: |
141
+ conda create -n vllm python=3.10 -y
142
+ conda activate vllm
143
+
144
+ pip install vllm==0.4.0.post1
145
+ # Install Gradio for web UI.
146
+ pip install gradio openai
147
+ pip install flash-attn==2.5.7
148
+
149
+ run: |
150
+ conda activate vllm
151
+ echo 'Starting vllm api server...'
152
+ python -u -m vllm.entrypoints.openai.api_server \
153
+ --port 8081 \
154
+ --model $MODEL_NAME \
155
+ --trust-remote-code \
156
+ --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
157
+ 2>&1 | tee api_server.log
158
+ ```
159
160
Start the serving the Llama-3 8B model on multiple replicas:
161
@@ -170,8 +171,7 @@ Wait until the service is ready:
171
watch -n10 sky serve status vllm
172
```
173
173
-<details>
174
-<summary>Example outputs:</summary>
174
+Example outputs:
175
176
```console
177
Services
@@ -184,29 +184,29 @@ vllm 1 1 xx.yy.zz.121 18 mins ago 1x GCP([Spot]{'L4': 1}) R
184
vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4
185
```
186
187
-</details>
188
-
187
After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:
188
191
-```console
192
-ENDPOINT=$(sky serve status --endpoint 8081 vllm)
193
-curl -L http://$ENDPOINT/v1/chat/completions \
194
- -H "Content-Type: application/json" \
195
- -d '{
196
- "model": "meta-llama/Meta-Llama-3-8B-Instruct",
197
- "messages": [
198
- {
199
- "role": "system",
200
- "content": "You are a helpful assistant."
201
- },
202
- {
203
- "role": "user",
204
- "content": "Who are you?"
205
- }
206
- ],
207
- "stop_token_ids": [128009, 128001]
208
- }'
209
-```
189
+??? Commands
190
+
191
+ ```bash
192
+ ENDPOINT=$(sky serve status --endpoint 8081 vllm)
193
+ curl -L http://$ENDPOINT/v1/chat/completions \
194
+ -H "Content-Type: application/json" \
195
+ -d '{
196
+ "model": "meta-llama/Meta-Llama-3-8B-Instruct",
197
+ "messages": [
198
+ {
199
+ "role": "system",
200
+ "content": "You are a helpful assistant."
201
+ },
202
+ {
203
+ "role": "user",
204
+ "content": "Who are you?"
205
+ }
206
+ ],
207
+ "stop_token_ids": [128009, 128001]
208
+ }'
209
+ ```
210
211
To enable autoscaling, you could replace the `replicas` with the following configs in `service`:
212
@@ -220,57 +220,54 @@ service:
220
221
This will scale the service up to when the QPS exceeds 2 for each replica.
222
223
-<details>
224
-<summary>Click to see the full recipe YAML</summary>
225
-
226
-```yaml
227
-service:
228
- replica_policy:
229
- min_replicas: 2
230
- max_replicas: 4
231
- target_qps_per_replica: 2
232
- # An actual request for readiness probe.
233
- readiness_probe:
234
- path: /v1/chat/completions
235
- post_data:
236
- model: $MODEL_NAME
237
- messages:
238
- - role: user
239
- content: Hello! What is your name?
240
- max_completion_tokens: 1
241
-
242
-resources:
243
- accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
244
- use_spot: True
245
- disk_size: 512 # Ensure model checkpoints can fit.
246
- disk_tier: best
247
- ports: 8081 # Expose to internet traffic.
248
-
249
-envs:
250
- MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
251
- HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
252
-
253
-setup: |
254
- conda create -n vllm python=3.10 -y
255
- conda activate vllm
256
-
257
- pip install vllm==0.4.0.post1
258
- # Install Gradio for web UI.
259
- pip install gradio openai
260
- pip install flash-attn==2.5.7
261
-
262
-run: |
263
- conda activate vllm
264
- echo 'Starting vllm api server...'
265
- python -u -m vllm.entrypoints.openai.api_server \
266
- --port 8081 \
267
- --model $MODEL_NAME \
268
- --trust-remote-code \
269
- --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
270
- 2>&1 | tee api_server.log
271
-```
272
-
273
-</details>
223
+??? Yaml
224
+
225
+ ```yaml
226
+ service:
227
+ replica_policy:
228
+ min_replicas: 2
229
+ max_replicas: 4
230
+ target_qps_per_replica: 2
231
+ # An actual request for readiness probe.
232
+ readiness_probe:
233
+ path: /v1/chat/completions
234
+ post_data:
235
+ model: $MODEL_NAME
236
+ messages:
237
+ - role: user
238
+ content: Hello! What is your name?
239
+ max_completion_tokens: 1
240
+
241
+ resources:
242
+ accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
243
+ use_spot: True
244
+ disk_size: 512 # Ensure model checkpoints can fit.
245
+ disk_tier: best
246
+ ports: 8081 # Expose to internet traffic.
247
+
248
+ envs:
249
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
250
+ HF_TOKEN: <your-huggingface-token> # Change to your own huggingface token, or use --env to pass.
251
+
252
+ setup: |
253
+ conda create -n vllm python=3.10 -y
254
+ conda activate vllm
255
+
256
+ pip install vllm==0.4.0.post1
257
+ # Install Gradio for web UI.
258
+ pip install gradio openai
259
+ pip install flash-attn==2.5.7
260
+
261
+ run: |
262
+ conda activate vllm
263
+ echo 'Starting vllm api server...'
264
+ python -u -m vllm.entrypoints.openai.api_server \
265
+ --port 8081 \
266
+ --model $MODEL_NAME \
267
+ --trust-remote-code \
268
+ --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
269
+ 2>&1 | tee api_server.log
270
+ ```
271
272
To update the service with the new config:
273
@@ -288,38 +285,35 @@ sky serve down vllm
285
286
It is also possible to access the Llama-3 service with a separate GUI frontend, so the user requests send to the GUI will be load-balanced across replicas.
287
291
-<details>
292
-<summary>Click to see the full GUI YAML</summary>
288
+??? Yaml
289
294
-```yaml
295
-envs:
296
- MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
297
- ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
298
-
299
-resources:
300
- cpus: 2
301
-
302
-setup: |
303
- conda create -n vllm python=3.10 -y
304
- conda activate vllm
305
-
306
- # Install Gradio for web UI.
307
- pip install gradio openai
308
-
309
-run: |
310
- conda activate vllm
311
- export PATH=$PATH:/sbin
312
-
313
- echo 'Starting gradio server...'
314
- git clone https://github.com/vllm-project/vllm.git || true
315
- python vllm/examples/online_serving/gradio_openai_chatbot_webserver.py \
316
- -m $MODEL_NAME \
317
- --port 8811 \
318
- --model-url http://$ENDPOINT/v1 \
319
- --stop-token-ids 128009,128001 | tee ~/gradio.log
320
-```
290
+ ```yaml
291
+ envs:
292
+ MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
293
+ ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
294
+
295
+ resources:
296
+ cpus: 2
297
322
-</details>
298
+ setup: |
299
+ conda create -n vllm python=3.10 -y
300
+ conda activate vllm
301
+
302
+ # Install Gradio for web UI.
303
+ pip install gradio openai
304
+
305
+ run: |
306
+ conda activate vllm
307
+ export PATH=$PATH:/sbin
308
+
309
+ echo 'Starting gradio server...'
310
+ git clone https://github.com/vllm-project/vllm.git || true
311
+ python vllm/examples/online_serving/gradio_openai_chatbot_webserver.py \
312
+ -m $MODEL_NAME \
313
+ --port 8811 \
314
+ --model-url http://$ENDPOINT/v1 \
315
+ --stop-token-ids 128009,128001 | tee ~/gradio.log
316
+ ```
317
318
1. Start the chat web UI:
319
docs/deployment/integrations/production-stack.md
+43
-41
@@ -60,22 +60,22 @@ And then you can send out a query to the OpenAI-compatible API to check the avai
60
curl -o- http://localhost:30080/models
61
```
62
63
-Expected output:
63
+??? Output
64
65
-```json
66
-{
67
- "object": "list",
68
- "data": [
65
+ ```json
66
{
70
- "id": "facebook/opt-125m",
71
- "object": "model",
72
- "created": 1737428424,
73
- "owned_by": "vllm",
74
- "root": null
67
+ "object": "list",
68
+ "data": [
69
+ {
70
+ "id": "facebook/opt-125m",
71
+ "object": "model",
72
+ "created": 1737428424,
73
+ "owned_by": "vllm",
74
+ "root": null
75
+ }
76
+ ]
77
}
76
- ]
77
-}
78
-```
78
+ ```
79
80
To send an actual chatting request, you can issue a curl request to the OpenAI `/completion` endpoint:
81
@@ -89,23 +89,23 @@ curl -X POST http://localhost:30080/completions \
89
}'
90
```
91
92
-Expected output:
92
+??? Output
93
94
-```json
95
-{
96
- "id": "completion-id",
97
- "object": "text_completion",
98
- "created": 1737428424,
99
- "model": "facebook/opt-125m",
100
- "choices": [
94
+ ```json
95
{
102
- "text": " there was a brave knight who...",
103
- "index": 0,
104
- "finish_reason": "length"
96
+ "id": "completion-id",
97
+ "object": "text_completion",
98
+ "created": 1737428424,
99
+ "model": "facebook/opt-125m",
100
+ "choices": [
101
+ {
102
+ "text": " there was a brave knight who...",
103
+ "index": 0,
104
+ "finish_reason": "length"
105
+ }
106
+ ]
107
}
106
- ]
107
-}
108
-```
108
+ ```
109
110
### Uninstall
111
@@ -121,23 +121,25 @@ sudo helm uninstall vllm
121
122
The core vLLM production stack configuration is managed with YAML. Here is the example configuration used in the installation above:
123
124
-```yaml
125
-servingEngineSpec:
126
- runtimeClassName: ""
127
- modelSpec:
128
- - name: "opt125m"
129
- repository: "vllm/vllm-openai"
130
- tag: "latest"
131
- modelURL: "facebook/opt-125m"
124
+??? Yaml
125
133
- replicaCount: 1
126
+ ```yaml
127
+ servingEngineSpec:
128
+ runtimeClassName: ""
129
+ modelSpec:
130
+ - name: "opt125m"
131
+ repository: "vllm/vllm-openai"
132
+ tag: "latest"
133
+ modelURL: "facebook/opt-125m"
134
135
- requestCPU: 6
136
- requestMemory: "16Gi"
137
- requestGPU: 1
135
+ replicaCount: 1
136
139
- pvcStorage: "10Gi"
140
-```
137
+ requestCPU: 6
138
+ requestMemory: "16Gi"
139
+ requestGPU: 1
140
+
141
+ pvcStorage: "10Gi"
142
+ ```
143
144
In this YAML configuration:
145
* **`modelSpec`** includes:
docs/deployment/k8s.md
+98
-74
@@ -29,85 +29,89 @@ Alternatively, you can deploy vLLM to Kubernetes using any of the following:
29
30
First, create a Kubernetes PVC and Secret for downloading and storing Hugging Face model:
31
32
-```bash
33
-cat <<EOF |kubectl apply -f -
34
-apiVersion: v1
35
-kind: PersistentVolumeClaim
36
-metadata:
37
- name: vllm-models
38
-spec:
39
- accessModes:
40
- - ReadWriteOnce
41
- volumeMode: Filesystem
42
- resources:
43
- requests:
44
- storage: 50Gi
45
----
46
-apiVersion: v1
47
-kind: Secret
48
-metadata:
49
- name: hf-token-secret
50
-type: Opaque
51
-data:
52
- token: $(HF_TOKEN)
53
-EOF
54
-```
32
+??? Config
33
+
34
+ ```bash
35
+ cat <<EOF |kubectl apply -f -
36
+ apiVersion: v1
37
+ kind: PersistentVolumeClaim
38
+ metadata:
39
+ name: vllm-models
40
+ spec:
41
+ accessModes:
42
+ - ReadWriteOnce
43
+ volumeMode: Filesystem
44
+ resources:
45
+ requests:
46
+ storage: 50Gi
47
+ ---
48
+ apiVersion: v1
49
+ kind: Secret
50
+ metadata:
51
+ name: hf-token-secret
52
+ type: Opaque
53
+ data:
54
+ token: $(HF_TOKEN)
55
+ EOF
56
+ ```
57
58
Next, start the vLLM server as a Kubernetes Deployment and Service:
59
58
-```bash
59
-cat <<EOF |kubectl apply -f -
60
-apiVersion: apps/v1
61
-kind: Deployment
62
-metadata:
63
- name: vllm-server
64
-spec:
65
- replicas: 1
66
- selector:
67
- matchLabels:
68
- app.kubernetes.io/name: vllm
69
- template:
60
+??? Config
61
+
62
+ ```bash
63
+ cat <<EOF |kubectl apply -f -
64
+ apiVersion: apps/v1
65
+ kind: Deployment
66
metadata:
71
- labels:
72
- app.kubernetes.io/name: vllm
67
+ name: vllm-server
68
spec:
74
- containers:
75
- - name: vllm
76
- image: vllm/vllm-openai:latest
77
- command: ["/bin/sh", "-c"]
78
- args: [
79
- "vllm serve meta-llama/Llama-3.2-1B-Instruct"
80
- ]
81
- env:
82
- - name: HUGGING_FACE_HUB_TOKEN
83
- valueFrom:
84
- secretKeyRef:
85
- name: hf-token-secret
86
- key: token
87
- ports:
88
- - containerPort: 8000
89
- volumeMounts:
69
+ replicas: 1
70
+ selector:
71
+ matchLabels:
72
+ app.kubernetes.io/name: vllm
73
+ template:
74
+ metadata:
75
+ labels:
76
+ app.kubernetes.io/name: vllm
77
+ spec:
78
+ containers:
79
+ - name: vllm
80
+ image: vllm/vllm-openai:latest
81
+ command: ["/bin/sh", "-c"]
82
+ args: [
83
+ "vllm serve meta-llama/Llama-3.2-1B-Instruct"
84
+ ]
85
+ env:
86
+ - name: HUGGING_FACE_HUB_TOKEN
87
+ valueFrom:
88
+ secretKeyRef:
89
+ name: hf-token-secret
90
+ key: token
91
+ ports:
92
+ - containerPort: 8000
93
+ volumeMounts:
94
+ - name: llama-storage
95
+ mountPath: /root/.cache/huggingface
96
+ volumes:
97
- name: llama-storage
91
- mountPath: /root/.cache/huggingface
92
- volumes:
93
- - name: llama-storage
94
- persistentVolumeClaim:
95
- claimName: vllm-models
96
----
97
-apiVersion: v1
98
-kind: Service
99
-metadata:
100
- name: vllm-server
101
-spec:
102
- selector:
103
- app.kubernetes.io/name: vllm
104
- ports:
105
- - protocol: TCP
106
- port: 8000
107
- targetPort: 8000
108
- type: ClusterIP
109
-EOF
110
-```
98
+ persistentVolumeClaim:
99
+ claimName: vllm-models
100
+ ---
101
+ apiVersion: v1
102
+ kind: Service
103
+ metadata:
104
+ name: vllm-server
105
+ spec:
106
+ selector:
107
+ app.kubernetes.io/name: vllm
108
+ ports:
109
+ - protocol: TCP
110
+ port: 8000
111
+ targetPort: 8000
112
+ type: ClusterIP
113
+ EOF
114
+ ```
115
116
We can verify that the vLLM server has started successfully via the logs (this might take a couple of minutes to download the model):
117
@@ -128,6 +132,9 @@ INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
132
133
PVC is used to store the model cache and it is optional, you can use hostPath or other storage options
134
135
+ <details>
136
+ <summary>Yaml</summary>
137
+
138
```yaml
139
apiVersion: v1
140
kind: PersistentVolumeClaim
@@ -144,6 +151,8 @@ INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
151
volumeMode: Filesystem
152
```
153
154
+ </details>
155
+
156
Secret is optional and only required for accessing gated models, you can skip this step if you are not using gated models
157
158
```yaml
@@ -156,13 +165,16 @@ INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
165
stringData:
166
token: "REPLACE_WITH_TOKEN"
167
```
159
-
168
+
169
Next to create the deployment file for vLLM to run the model server. The following example deploys the `Mistral-7B-Instruct-v0.3` model.
170
171
Here are two examples for using NVIDIA GPU and AMD GPU.
172
173
NVIDIA GPU:
174
175
+ <details>
176
+ <summary>Yaml</summary>
177
+
178
```yaml
179
apiVersion: apps/v1
180
kind: Deployment
@@ -233,10 +245,15 @@ INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
245
periodSeconds: 5
246
```
247
248
+ </details>
249
+
250
AMD GPU:
251
252
You can refer to the `deployment.yaml` below if using AMD ROCm GPU like MI300X.
253
254
+ <details>
255
+ <summary>Yaml</summary>
256
+
257
```yaml
258
apiVersion: apps/v1
259
kind: Deployment
@@ -305,12 +322,17 @@ INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
322
mountPath: /dev/shm
323
```
324
325
+ </details>
326
+
327
You can get the full example with steps and sample yaml files from <https://github.com/ROCm/k8s-device-plugin/tree/master/example/vllm-serve>.
328
329
2. Create a Kubernetes Service for vLLM
330
331
Next, create a Kubernetes Service file to expose the `mistral-7b` deployment:
332
333
+ <details>
334
+ <summary>Yaml</summary>
335
+
336
```yaml
337
apiVersion: v1
338
kind: Service
@@ -330,6 +352,8 @@ INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
352
type: ClusterIP
353
```
354
355
+ </details>
356
+
357
3. Deploy and Test
358
359
Apply the deployment and service configurations using `kubectl apply -f <filename>`:
docs/deployment/nginx.md
+44
-40
@@ -36,23 +36,25 @@ docker build . -f Dockerfile.nginx --tag nginx-lb
36
37
Create a file named `nginx_conf/nginx.conf`. Note that you can add as many servers as you'd like. In the below example we'll start with two. To add more, add another `server vllmN:8000 max_fails=3 fail_timeout=10000s;` entry to `upstream backend`.
38
39
-```console
40
-upstream backend {
41
- least_conn;
42
- server vllm0:8000 max_fails=3 fail_timeout=10000s;
43
- server vllm1:8000 max_fails=3 fail_timeout=10000s;
44
-}
45
-server {
46
- listen 80;
47
- location / {
48
- proxy_pass http://backend;
49
- proxy_set_header Host $host;
50
- proxy_set_header X-Real-IP $remote_addr;
51
- proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
52
- proxy_set_header X-Forwarded-Proto $scheme;
39
+??? Config
40
+
41
+ ```console
42
+ upstream backend {
43
+ least_conn;
44
+ server vllm0:8000 max_fails=3 fail_timeout=10000s;
45
+ server vllm1:8000 max_fails=3 fail_timeout=10000s;
46
}
54
-}
55
-```
47
+ server {
48
+ listen 80;
49
+ location / {
50
+ proxy_pass http://backend;
51
+ proxy_set_header Host $host;
52
+ proxy_set_header X-Real-IP $remote_addr;
53
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
54
+ proxy_set_header X-Forwarded-Proto $scheme;
55
+ }
56
+ }
57
+ ```
58
59
[](){ #nginxloadbalancer-nginx-vllm-container }
60
@@ -93,30 +95,32 @@ Notes:
95
- The below example assumes GPU backend used. If you are using CPU backend, remove `--gpus device=ID`, add `VLLM_CPU_KVCACHE_SPACE` and `VLLM_CPU_OMP_THREADS_BIND` environment variables to the docker run command.
96
- Adjust the model name that you want to use in your vLLM servers if you don't want to use `Llama-2-7b-chat-hf`.
97
96
-```console
97
-mkdir -p ~/.cache/huggingface/hub/
98
-hf_cache_dir=~/.cache/huggingface/
99
-docker run \
100
- -itd \
101
- --ipc host \
102
- --network vllm_nginx \
103
- --gpus device=0 \
104
- --shm-size=10.24gb \
105
- -v $hf_cache_dir:/root/.cache/huggingface/ \
106
- -p 8081:8000 \
107
- --name vllm0 vllm \
108
- --model meta-llama/Llama-2-7b-chat-hf
109
-docker run \
110
- -itd \
111
- --ipc host \
112
- --network vllm_nginx \
113
- --gpus device=1 \
114
- --shm-size=10.24gb \
115
- -v $hf_cache_dir:/root/.cache/huggingface/ \
116
- -p 8082:8000 \
117
- --name vllm1 vllm \
118
- --model meta-llama/Llama-2-7b-chat-hf
119
-```
98
+??? Commands
99
+
100
+ ```console
101
+ mkdir -p ~/.cache/huggingface/hub/
102
+ hf_cache_dir=~/.cache/huggingface/
103
+ docker run \
104
+ -itd \
105
+ --ipc host \
106
+ --network vllm_nginx \
107
+ --gpus device=0 \
108
+ --shm-size=10.24gb \
109
+ -v $hf_cache_dir:/root/.cache/huggingface/ \
110
+ -p 8081:8000 \
111
+ --name vllm0 vllm \
112
+ --model meta-llama/Llama-2-7b-chat-hf
113
+ docker run \
114
+ -itd \
115
+ --ipc host \
116
+ --network vllm_nginx \
117
+ --gpus device=1 \
118
+ --shm-size=10.24gb \
119
+ -v $hf_cache_dir:/root/.cache/huggingface/ \
120
+ -p 8082:8000 \
121
+ --name vllm1 vllm \
122
+ --model meta-llama/Llama-2-7b-chat-hf
123
+ ```
124
125
!!! note
126
If you are behind proxy, you can pass the proxy settings to the docker run command via `-e http_proxy=$http_proxy -e https_proxy=$https_proxy`.
docs/design/arch_overview.md
+55
-51
@@ -22,31 +22,33 @@ server.
22
23
Here is a sample of `LLM` class usage:
24
25
-```python
26
-from vllm import LLM, SamplingParams
27
-
28
-# Define a list of input prompts
29
-prompts = [
30
- "Hello, my name is",
31
- "The capital of France is",
32
- "The largest ocean is",
33
-]
34
-
35
-# Define sampling parameters
36
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
37
-
38
-# Initialize the LLM engine with the OPT-125M model
39
-llm = LLM(model="facebook/opt-125m")
40
-
41
-# Generate outputs for the input prompts
42
-outputs = llm.generate(prompts, sampling_params)
43
-
44
-# Print the generated outputs
45
-for output in outputs:
46
- prompt = output.prompt
47
- generated_text = output.outputs[0].text
48
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
49
-```
25
+??? Code
26
+
27
+ ```python
28
+ from vllm import LLM, SamplingParams
29
+
30
+ # Define a list of input prompts
31
+ prompts = [
32
+ "Hello, my name is",
33
+ "The capital of France is",
34
+ "The largest ocean is",
35
+ ]
36
+
37
+ # Define sampling parameters
38
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
39
+
40
+ # Initialize the LLM engine with the OPT-125M model
41
+ llm = LLM(model="facebook/opt-125m")
42
+
43
+ # Generate outputs for the input prompts
44
+ outputs = llm.generate(prompts, sampling_params)
45
+
46
+ # Print the generated outputs
47
+ for output in outputs:
48
+ prompt = output.prompt
49
+ generated_text = output.outputs[0].text
50
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
51
+ ```
52
53
More API details can be found in the [Offline Inference](#offline-inference-api) section of the API docs.
54
@@ -178,32 +180,34 @@ vision-language model.
180
181
To avoid accidentally passing incorrect arguments, the constructor is now keyword-only. This ensures that the constructor will raise an error if old configurations are passed. vLLM developers have already made this change for all models within vLLM. For out-of-tree registered models, developers need to update their models, for example by adding shim code to adapt the old constructor signature to the new one:
182
181
- ```python
182
- class MyOldModel(nn.Module):
183
- def __init__(
184
- self,
185
- config,
186
- cache_config: Optional[CacheConfig] = None,
187
- quant_config: Optional[QuantizationConfig] = None,
188
- lora_config: Optional[LoRAConfig] = None,
189
- prefix: str = "",
190
- ) -> None:
191
- ...
192
-
193
- from vllm.config import VllmConfig
194
- class MyNewModel(MyOldModel):
195
- def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
196
- config = vllm_config.model_config.hf_config
197
- cache_config = vllm_config.cache_config
198
- quant_config = vllm_config.quant_config
199
- lora_config = vllm_config.lora_config
200
- super().__init__(config, cache_config, quant_config, lora_config, prefix)
201
-
202
- if __version__ >= "0.6.4":
203
- MyModel = MyNewModel
204
- else:
205
- MyModel = MyOldModel
206
- ```
183
+ ??? Code
184
+
185
+ ```python
186
+ class MyOldModel(nn.Module):
187
+ def __init__(
188
+ self,
189
+ config,
190
+ cache_config: Optional[CacheConfig] = None,
191
+ quant_config: Optional[QuantizationConfig] = None,
192
+ lora_config: Optional[LoRAConfig] = None,
193
+ prefix: str = "",
194
+ ) -> None:
195
+ ...
196
+
197
+ from vllm.config import VllmConfig
198
+ class MyNewModel(MyOldModel):
199
+ def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
200
+ config = vllm_config.model_config.hf_config
201
+ cache_config = vllm_config.cache_config
202
+ quant_config = vllm_config.quant_config
203
+ lora_config = vllm_config.lora_config
204
+ super().__init__(config, cache_config, quant_config, lora_config, prefix)
205
+
206
+ if __version__ >= "0.6.4":
207
+ MyModel = MyNewModel
208
+ else:
209
+ MyModel = MyOldModel
210
+ ```
211
212
This way, the model can work with both old and new versions of vLLM.
213
docs/design/kernel/paged_attention.md
+20
-18
@@ -448,27 +448,29 @@ elements of the entire head for all context tokens. However, overall,
448
all results for output have been calculated but are just stored in
449
different thread register memory.
450
451
-```cpp
452
-float* out_smem = reinterpret_cast<float*>(shared_mem);
453
-for (int i = NUM_WARPS; i > 1; i /= 2) {
454
- // Upper warps write to shared memory.
455
- ...
456
- float* dst = &out_smem[(warp_idx - mid) * HEAD_SIZE];
457
- for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
458
- ...
459
- dst[row_idx] = accs[i];
460
- }
451
+??? Code
452
462
- // Lower warps update the output.
463
- const float* src = &out_smem[warp_idx * HEAD_SIZE];
464
- for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
453
+ ```cpp
454
+ float* out_smem = reinterpret_cast<float*>(shared_mem);
455
+ for (int i = NUM_WARPS; i > 1; i /= 2) {
456
+ // Upper warps write to shared memory.
457
...
466
- accs[i] += src[row_idx];
458
+ float* dst = &out_smem[(warp_idx - mid) * HEAD_SIZE];
459
+ for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
460
+ ...
461
+ dst[row_idx] = accs[i];
462
+ }
463
+
464
+ // Lower warps update the output.
465
+ const float* src = &out_smem[warp_idx * HEAD_SIZE];
466
+ for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
467
+ ...
468
+ accs[i] += src[row_idx];
469
+ }
470
+
471
+ // Write out the accs.
472
}
468
-
469
- // Write out the accs.
470
-}
471
-```
473
+ ```
474
475
## Output
476
docs/design/plugin_system.md
+24
-22
@@ -13,28 +13,30 @@ Plugins are user-registered code that vLLM executes. Given vLLM's architecture (
13
14
vLLM's plugin system uses the standard Python `entry_points` mechanism. This mechanism allows developers to register functions in their Python packages for use by other packages. An example of a plugin:
15
16
-```python
17
-# inside `setup.py` file
18
-from setuptools import setup
19
-
20
-setup(name='vllm_add_dummy_model',
21
- version='0.1',
22
- packages=['vllm_add_dummy_model'],
23
- entry_points={
24
- 'vllm.general_plugins':
25
- ["register_dummy_model = vllm_add_dummy_model:register"]
26
- })
27
-
28
-# inside `vllm_add_dummy_model.py` file
29
-def register():
30
- from vllm import ModelRegistry
31
-
32
- if "MyLlava" not in ModelRegistry.get_supported_archs():
33
- ModelRegistry.register_model(
34
- "MyLlava",
35
- "vllm_add_dummy_model.my_llava:MyLlava",
36
- )
37
-```
16
+??? Code
17
+
18
+ ```python
19
+ # inside `setup.py` file
20
+ from setuptools import setup
21
+
22
+ setup(name='vllm_add_dummy_model',
23
+ version='0.1',
24
+ packages=['vllm_add_dummy_model'],
25
+ entry_points={
26
+ 'vllm.general_plugins':
27
+ ["register_dummy_model = vllm_add_dummy_model:register"]
28
+ })
29
+
30
+ # inside `vllm_add_dummy_model.py` file
31
+ def register():
32
+ from vllm import ModelRegistry
33
+
34
+ if "MyLlava" not in ModelRegistry.get_supported_archs():
35
+ ModelRegistry.register_model(
36
+ "MyLlava",
37
+ "vllm_add_dummy_model.my_llava:MyLlava",
38
+ )
39
+ ```
40
41
For more information on adding entry points to your package, please check the [official documentation](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
42
docs/features/lora.md
+103
-97
@@ -29,24 +29,26 @@ We can now submit the prompts and call `llm.generate` with the `lora_request` pa
29
of `LoRARequest` is a human identifiable name, the second parameter is a globally unique ID for the adapter and
30
the third parameter is the path to the LoRA adapter.
31
32
-```python
33
-sampling_params = SamplingParams(
34
- temperature=0,
35
- max_tokens=256,
36
- stop=["[/assistant]"]
37
-)
38
-
39
-prompts = [
40
- "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport [/user] [assistant]",
41
- "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_11 (nationality VARCHAR, elector VARCHAR)\n\n question: When Anchero Pantaleone was the elector what is under nationality? [/user] [assistant]",
42
-]
43
-
44
-outputs = llm.generate(
45
- prompts,
46
- sampling_params,
47
- lora_request=LoRARequest("sql_adapter", 1, sql_lora_path)
48
-)
49
-```
32
+??? Code
33
+
34
+ ```python
35
+ sampling_params = SamplingParams(
36
+ temperature=0,
37
+ max_tokens=256,
38
+ stop=["[/assistant]"]
39
+ )
40
+
41
+ prompts = [
42
+ "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport [/user] [assistant]",
43
+ "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_11 (nationality VARCHAR, elector VARCHAR)\n\n question: When Anchero Pantaleone was the elector what is under nationality? [/user] [assistant]",
44
+ ]
45
+
46
+ outputs = llm.generate(
47
+ prompts,
48
+ sampling_params,
49
+ lora_request=LoRARequest("sql_adapter", 1, sql_lora_path)
50
+ )
51
+ ```
52
53
Check out <gh-file:examples/offline_inference/multilora_inference.py> for an example of how to use LoRA adapters with the async engine and how to use more advanced configuration options.
54
@@ -68,24 +70,26 @@ The server entrypoint accepts all other LoRA configuration parameters (`max_lora
70
etc.), which will apply to all forthcoming requests. Upon querying the `/models` endpoint, we should see our LoRA along
71
with its base model (if `jq` is not installed, you can follow [this guide](https://jqlang.org/download/) to install it.):
72
71
-```bash
72
-curl localhost:8000/v1/models | jq .
73
-{
74
- "object": "list",
75
- "data": [
76
- {
77
- "id": "meta-llama/Llama-2-7b-hf",
78
- "object": "model",
79
- ...
80
- },
81
- {
82
- "id": "sql-lora",
83
- "object": "model",
84
- ...
85
- }
86
- ]
87
-}
88
-```
73
+??? Command
74
+
75
+ ```bash
76
+ curl localhost:8000/v1/models | jq .
77
+ {
78
+ "object": "list",
79
+ "data": [
80
+ {
81
+ "id": "meta-llama/Llama-2-7b-hf",
82
+ "object": "model",
83
+ ...
84
+ },
85
+ {
86
+ "id": "sql-lora",
87
+ "object": "model",
88
+ ...
89
+ }
90
+ ]
91
+ }
92
+ ```
93
94
Requests can specify the LoRA adapter as if it were any other model via the `model` request parameter. The requests will be
95
processed according to the server-wide LoRA configuration (i.e. in parallel with base model requests, and potentially other
@@ -168,36 +172,36 @@ Alternatively, follow these example steps to implement your own plugin:
172
173
1. Implement the LoRAResolver interface.
174
171
- Example of a simple S3 LoRAResolver implementation:
172
-
173
- ```python
174
- import os
175
- import s3fs
176
- from vllm.lora.request import LoRARequest
177
- from vllm.lora.resolver import LoRAResolver
178
-
179
- class S3LoRAResolver(LoRAResolver):
180
- def __init__(self):
181
- self.s3 = s3fs.S3FileSystem()
182
- self.s3_path_format = os.getenv("S3_PATH_TEMPLATE")
183
- self.local_path_format = os.getenv("LOCAL_PATH_TEMPLATE")
184
-
185
- async def resolve_lora(self, base_model_name, lora_name):
186
- s3_path = self.s3_path_format.format(base_model_name=base_model_name, lora_name=lora_name)
187
- local_path = self.local_path_format.format(base_model_name=base_model_name, lora_name=lora_name)
188
-
189
- # Download the LoRA from S3 to the local path
190
- await self.s3._get(
191
- s3_path, local_path, recursive=True, maxdepth=1
192
- )
193
-
194
- lora_request = LoRARequest(
195
- lora_name=lora_name,
196
- lora_path=local_path,
197
- lora_int_id=abs(hash(lora_name))
198
- )
199
- return lora_request
200
- ```
175
+ ??? Example of a simple S3 LoRAResolver implementation
176
+
177
+ ```python
178
+ import os
179
+ import s3fs
180
+ from vllm.lora.request import LoRARequest
181
+ from vllm.lora.resolver import LoRAResolver
182
+
183
+ class S3LoRAResolver(LoRAResolver):
184
+ def __init__(self):
185
+ self.s3 = s3fs.S3FileSystem()
186
+ self.s3_path_format = os.getenv("S3_PATH_TEMPLATE")
187
+ self.local_path_format = os.getenv("LOCAL_PATH_TEMPLATE")
188
+
189
+ async def resolve_lora(self, base_model_name, lora_name):
190
+ s3_path = self.s3_path_format.format(base_model_name=base_model_name, lora_name=lora_name)
191
+ local_path = self.local_path_format.format(base_model_name=base_model_name, lora_name=lora_name)
192
+
193
+ # Download the LoRA from S3 to the local path
194
+ await self.s3._get(
195
+ s3_path, local_path, recursive=True, maxdepth=1
196
+ )
197
+
198
+ lora_request = LoRARequest(
199
+ lora_name=lora_name,
200
+ lora_path=local_path,
201
+ lora_int_id=abs(hash(lora_name))
202
+ )
203
+ return lora_request
204
+ ```
205
206
2. Register `LoRAResolver` plugin.
207
@@ -234,38 +238,40 @@ The new format of `--lora-modules` is mainly to support the display of parent mo
238
- The `parent` field of LoRA model `sql-lora` now links to its base model `meta-llama/Llama-2-7b-hf`. This correctly reflects the hierarchical relationship between the base model and the LoRA adapter.
239
- The `root` field points to the artifact location of the lora adapter.
240
237
-```bash
238
-$ curl http://localhost:8000/v1/models
239
-
240
-{
241
- "object": "list",
242
- "data": [
243
- {
244
- "id": "meta-llama/Llama-2-7b-hf",
245
- "object": "model",
246
- "created": 1715644056,
247
- "owned_by": "vllm",
248
- "root": "~/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf/snapshots/01c7f73d771dfac7d292323805ebc428287df4f9/",
249
- "parent": null,
250
- "permission": [
241
+??? Command output
242
+
243
+ ```bash
244
+ $ curl http://localhost:8000/v1/models
245
+
246
+ {
247
+ "object": "list",
248
+ "data": [
249
{
252
- .....
253
- }
254
- ]
255
- },
256
- {
257
- "id": "sql-lora",
258
- "object": "model",
259
- "created": 1715644056,
260
- "owned_by": "vllm",
261
- "root": "~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/",
262
- "parent": meta-llama/Llama-2-7b-hf,
263
- "permission": [
250
+ "id": "meta-llama/Llama-2-7b-hf",
251
+ "object": "model",
252
+ "created": 1715644056,
253
+ "owned_by": "vllm",
254
+ "root": "~/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf/snapshots/01c7f73d771dfac7d292323805ebc428287df4f9/",
255
+ "parent": null,
256
+ "permission": [
257
+ {
258
+ .....
259
+ }
260
+ ]
261
+ },
262
{
265
- ....
263
+ "id": "sql-lora",
264
+ "object": "model",
265
+ "created": 1715644056,
266
+ "owned_by": "vllm",
267
+ "root": "~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/",
268
+ "parent": meta-llama/Llama-2-7b-hf,
269
+ "permission": [
270
+ {
271
+ ....
272
+ }
273
+ ]
274
}
275
]
268
- }
269
- ]
270
-}
271
-```
276
+ }
277
+ ```
docs/features/multimodal_inputs.md
+360
-340
@@ -20,111 +20,117 @@ To input multi-modal data, follow this schema in [vllm.inputs.PromptType][]:
20
21
You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:
22
23
-```python
24
-from vllm import LLM
25
-
26
-llm = LLM(model="llava-hf/llava-1.5-7b-hf")
27
-
28
-# Refer to the HuggingFace repo for the correct format to use
29
-prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
30
-
31
-# Load the image using PIL.Image
32
-image = PIL.Image.open(...)
33
-
34
-# Single prompt inference
35
-outputs = llm.generate({
36
- "prompt": prompt,
37
- "multi_modal_data": {"image": image},
38
-})
39
-
40
-for o in outputs:
41
- generated_text = o.outputs[0].text
42
- print(generated_text)
43
-
44
-# Batch inference
45
-image_1 = PIL.Image.open(...)
46
-image_2 = PIL.Image.open(...)
47
-outputs = llm.generate(
48
- [
49
- {
50
- "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
51
- "multi_modal_data": {"image": image_1},
52
- },
53
- {
54
- "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
55
- "multi_modal_data": {"image": image_2},
56
- }
57
- ]
58
-)
23
+??? Code
24
60
-for o in outputs:
61
- generated_text = o.outputs[0].text
62
- print(generated_text)
63
-```
25
+ ```python
26
+ from vllm import LLM
27
+
28
+ llm = LLM(model="llava-hf/llava-1.5-7b-hf")
29
+
30
+ # Refer to the HuggingFace repo for the correct format to use
31
+ prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
32
+
33
+ # Load the image using PIL.Image
34
+ image = PIL.Image.open(...)
35
+
36
+ # Single prompt inference
37
+ outputs = llm.generate({
38
+ "prompt": prompt,
39
+ "multi_modal_data": {"image": image},
40
+ })
41
+
42
+ for o in outputs:
43
+ generated_text = o.outputs[0].text
44
+ print(generated_text)
45
+
46
+ # Batch inference
47
+ image_1 = PIL.Image.open(...)
48
+ image_2 = PIL.Image.open(...)
49
+ outputs = llm.generate(
50
+ [
51
+ {
52
+ "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
53
+ "multi_modal_data": {"image": image_1},
54
+ },
55
+ {
56
+ "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
57
+ "multi_modal_data": {"image": image_2},
58
+ }
59
+ ]
60
+ )
61
+
62
+ for o in outputs:
63
+ generated_text = o.outputs[0].text
64
+ print(generated_text)
65
+ ```
66
67
Full example: <gh-file:examples/offline_inference/vision_language.py>
68
69
To substitute multiple images inside the same text prompt, you can pass in a list of images instead:
70
69
-```python
70
-from vllm import LLM
71
-
72
-llm = LLM(
73
- model="microsoft/Phi-3.5-vision-instruct",
74
- trust_remote_code=True, # Required to load Phi-3.5-vision
75
- max_model_len=4096, # Otherwise, it may not fit in smaller GPUs
76
- limit_mm_per_prompt={"image": 2}, # The maximum number to accept
77
-)
78
-
79
-# Refer to the HuggingFace repo for the correct format to use
80
-prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
81
-
82
-# Load the images using PIL.Image
83
-image1 = PIL.Image.open(...)
84
-image2 = PIL.Image.open(...)
85
-
86
-outputs = llm.generate({
87
- "prompt": prompt,
88
- "multi_modal_data": {
89
- "image": [image1, image2]
90
- },
91
-})
92
-
93
-for o in outputs:
94
- generated_text = o.outputs[0].text
95
- print(generated_text)
96
-```
71
+??? Code
72
+
73
+ ```python
74
+ from vllm import LLM
75
+
76
+ llm = LLM(
77
+ model="microsoft/Phi-3.5-vision-instruct",
78
+ trust_remote_code=True, # Required to load Phi-3.5-vision
79
+ max_model_len=4096, # Otherwise, it may not fit in smaller GPUs
80
+ limit_mm_per_prompt={"image": 2}, # The maximum number to accept
81
+ )
82
+
83
+ # Refer to the HuggingFace repo for the correct format to use
84
+ prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
85
+
86
+ # Load the images using PIL.Image
87
+ image1 = PIL.Image.open(...)
88
+ image2 = PIL.Image.open(...)
89
+
90
+ outputs = llm.generate({
91
+ "prompt": prompt,
92
+ "multi_modal_data": {
93
+ "image": [image1, image2]
94
+ },
95
+ })
96
+
97
+ for o in outputs:
98
+ generated_text = o.outputs[0].text
99
+ print(generated_text)
100
+ ```
101
102
Full example: <gh-file:examples/offline_inference/vision_language_multi_image.py>
103
104
Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:
105
102
-```python
103
-from vllm import LLM
106
+??? Code
107
105
-# Specify the maximum number of frames per video to be 4. This can be changed.
106
-llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
108
+ ```python
109
+ from vllm import LLM
110
108
-# Create the request payload.
109
-video_frames = ... # load your video making sure it only has the number of frames specified earlier.
110
-message = {
111
- "role": "user",
112
- "content": [
113
- {"type": "text", "text": "Describe this set of frames. Consider the frames to be a part of the same video."},
114
- ],
115
-}
116
-for i in range(len(video_frames)):
117
- base64_image = encode_image(video_frames[i]) # base64 encoding.
118
- new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
119
- message["content"].append(new_image)
120
-
121
-# Perform inference and log output.
122
-outputs = llm.chat([message])
123
-
124
-for o in outputs:
125
- generated_text = o.outputs[0].text
126
- print(generated_text)
127
-```
111
+ # Specify the maximum number of frames per video to be 4. This can be changed.
112
+ llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
113
+
114
+ # Create the request payload.
115
+ video_frames = ... # load your video making sure it only has the number of frames specified earlier.
116
+ message = {
117
+ "role": "user",
118
+ "content": [
119
+ {"type": "text", "text": "Describe this set of frames. Consider the frames to be a part of the same video."},
120
+ ],
121
+ }
122
+ for i in range(len(video_frames)):
123
+ base64_image = encode_image(video_frames[i]) # base64 encoding.
124
+ new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
125
+ message["content"].append(new_image)
126
+
127
+ # Perform inference and log output.
128
+ outputs = llm.chat([message])
129
+
130
+ for o in outputs:
131
+ generated_text = o.outputs[0].text
132
+ print(generated_text)
133
+ ```
134
135
### Video Inputs
136
@@ -144,68 +150,72 @@ Full example: <gh-file:examples/offline_inference/audio_language.py>
150
To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
151
pass a tensor of shape `(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
152
147
-```python
148
-from vllm import LLM
153
+??? Code
154
150
-# Inference with image embeddings as input
151
-llm = LLM(model="llava-hf/llava-1.5-7b-hf")
155
+ ```python
156
+ from vllm import LLM
157
153
-# Refer to the HuggingFace repo for the correct format to use
154
-prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
158
+ # Inference with image embeddings as input
159
+ llm = LLM(model="llava-hf/llava-1.5-7b-hf")
160
156
-# Embeddings for single image
157
-# torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
158
-image_embeds = torch.load(...)
161
+ # Refer to the HuggingFace repo for the correct format to use
162
+ prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
163
160
-outputs = llm.generate({
161
- "prompt": prompt,
162
- "multi_modal_data": {"image": image_embeds},
163
-})
164
+ # Embeddings for single image
165
+ # torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
166
+ image_embeds = torch.load(...)
167
165
-for o in outputs:
166
- generated_text = o.outputs[0].text
167
- print(generated_text)
168
-```
168
+ outputs = llm.generate({
169
+ "prompt": prompt,
170
+ "multi_modal_data": {"image": image_embeds},
171
+ })
172
+
173
+ for o in outputs:
174
+ generated_text = o.outputs[0].text
175
+ print(generated_text)
176
+ ```
177
178
For Qwen2-VL and MiniCPM-V, we accept additional parameters alongside the embeddings:
179
172
-```python
173
-# Construct the prompt based on your model
174
-prompt = ...
175
-
176
-# Embeddings for multiple images
177
-# torch.Tensor of shape (num_images, image_feature_size, hidden_size of LM)
178
-image_embeds = torch.load(...)
179
-
180
-# Qwen2-VL
181
-llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
182
-mm_data = {
183
- "image": {
184
- "image_embeds": image_embeds,
185
- # image_grid_thw is needed to calculate positional encoding.
186
- "image_grid_thw": torch.load(...), # torch.Tensor of shape (1, 3),
180
+??? Code
181
+
182
+ ```python
183
+ # Construct the prompt based on your model
184
+ prompt = ...
185
+
186
+ # Embeddings for multiple images
187
+ # torch.Tensor of shape (num_images, image_feature_size, hidden_size of LM)
188
+ image_embeds = torch.load(...)
189
+
190
+ # Qwen2-VL
191
+ llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
192
+ mm_data = {
193
+ "image": {
194
+ "image_embeds": image_embeds,
195
+ # image_grid_thw is needed to calculate positional encoding.
196
+ "image_grid_thw": torch.load(...), # torch.Tensor of shape (1, 3),
197
+ }
198
}
188
-}
189
-
190
-# MiniCPM-V
191
-llm = LLM("openbmb/MiniCPM-V-2_6", trust_remote_code=True, limit_mm_per_prompt={"image": 4})
192
-mm_data = {
193
- "image": {
194
- "image_embeds": image_embeds,
195
- # image_sizes is needed to calculate details of the sliced image.
196
- "image_sizes": [image.size for image in images], # list of image sizes
199
+
200
+ # MiniCPM-V
201
+ llm = LLM("openbmb/MiniCPM-V-2_6", trust_remote_code=True, limit_mm_per_prompt={"image": 4})
202
+ mm_data = {
203
+ "image": {
204
+ "image_embeds": image_embeds,
205
+ # image_sizes is needed to calculate details of the sliced image.
206
+ "image_sizes": [image.size for image in images], # list of image sizes
207
+ }
208
}
198
-}
209
200
-outputs = llm.generate({
201
- "prompt": prompt,
202
- "multi_modal_data": mm_data,
203
-})
210
+ outputs = llm.generate({
211
+ "prompt": prompt,
212
+ "multi_modal_data": mm_data,
213
+ })
214
205
-for o in outputs:
206
- generated_text = o.outputs[0].text
207
- print(generated_text)
208
-```
215
+ for o in outputs:
216
+ generated_text = o.outputs[0].text
217
+ print(generated_text)
218
+ ```
219
220
## Online Serving
221
@@ -235,51 +245,53 @@ vllm serve microsoft/Phi-3.5-vision-instruct --task generate \
245
246
Then, you can use the OpenAI client as follows:
247
238
-```python
239
-from openai import OpenAI
240
-
241
-openai_api_key = "EMPTY"
242
-openai_api_base = "http://localhost:8000/v1"
243
-
244
-client = OpenAI(
245
- api_key=openai_api_key,
246
- base_url=openai_api_base,
247
-)
248
-
249
-# Single-image input inference
250
-image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
251
-
252
-chat_response = client.chat.completions.create(
253
- model="microsoft/Phi-3.5-vision-instruct",
254
- messages=[{
255
- "role": "user",
256
- "content": [
257
- # NOTE: The prompt formatting with the image token `<image>` is not needed
258
- # since the prompt will be processed automatically by the API server.
259
- {"type": "text", "text": "What’s in this image?"},
260
- {"type": "image_url", "image_url": {"url": image_url}},
261
- ],
262
- }],
263
-)
264
-print("Chat completion output:", chat_response.choices[0].message.content)
265
-
266
-# Multi-image input inference
267
-image_url_duck = "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"
268
-image_url_lion = "https://upload.wikimedia.org/wikipedia/commons/7/77/002_The_lion_king_Snyggve_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent.jpg"
269
-
270
-chat_response = client.chat.completions.create(
271
- model="microsoft/Phi-3.5-vision-instruct",
272
- messages=[{
273
- "role": "user",
274
- "content": [
275
- {"type": "text", "text": "What are the animals in these images?"},
276
- {"type": "image_url", "image_url": {"url": image_url_duck}},
277
- {"type": "image_url", "image_url": {"url": image_url_lion}},
278
- ],
279
- }],
280
-)
281
-print("Chat completion output:", chat_response.choices[0].message.content)
282
-```
248
+??? Code
249
+
250
+ ```python
251
+ from openai import OpenAI
252
+
253
+ openai_api_key = "EMPTY"
254
+ openai_api_base = "http://localhost:8000/v1"
255
+
256
+ client = OpenAI(
257
+ api_key=openai_api_key,
258
+ base_url=openai_api_base,
259
+ )
260
+
261
+ # Single-image input inference
262
+ image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
263
+
264
+ chat_response = client.chat.completions.create(
265
+ model="microsoft/Phi-3.5-vision-instruct",
266
+ messages=[{
267
+ "role": "user",
268
+ "content": [
269
+ # NOTE: The prompt formatting with the image token `<image>` is not needed
270
+ # since the prompt will be processed automatically by the API server.
271
+ {"type": "text", "text": "What’s in this image?"},
272
+ {"type": "image_url", "image_url": {"url": image_url}},
273
+ ],
274
+ }],
275
+ )
276
+ print("Chat completion output:", chat_response.choices[0].message.content)
277
+
278
+ # Multi-image input inference
279
+ image_url_duck = "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"
280
+ image_url_lion = "https://upload.wikimedia.org/wikipedia/commons/7/77/002_The_lion_king_Snyggve_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent.jpg"
281
+
282
+ chat_response = client.chat.completions.create(
283
+ model="microsoft/Phi-3.5-vision-instruct",
284
+ messages=[{
285
+ "role": "user",
286
+ "content": [
287
+ {"type": "text", "text": "What are the animals in these images?"},
288
+ {"type": "image_url", "image_url": {"url": image_url_duck}},
289
+ {"type": "image_url", "image_url": {"url": image_url_lion}},
290
+ ],
291
+ }],
292
+ )
293
+ print("Chat completion output:", chat_response.choices[0].message.content)
294
+ ```
295
296
Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
297
@@ -311,44 +323,46 @@ vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --task generate --max-model
323
324
Then, you can use the OpenAI client as follows:
325
314
-```python
315
-from openai import OpenAI
326
+??? Code
327
317
-openai_api_key = "EMPTY"
318
-openai_api_base = "http://localhost:8000/v1"
328
+ ```python
329
+ from openai import OpenAI
330
320
-client = OpenAI(
321
- api_key=openai_api_key,
322
- base_url=openai_api_base,
323
-)
331
+ openai_api_key = "EMPTY"
332
+ openai_api_base = "http://localhost:8000/v1"
333
325
-video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
334
+ client = OpenAI(
335
+ api_key=openai_api_key,
336
+ base_url=openai_api_base,
337
+ )
338
327
-## Use video url in the payload
328
-chat_completion_from_url = client.chat.completions.create(
329
- messages=[{
330
- "role":
331
- "user",
332
- "content": [
333
- {
334
- "type": "text",
335
- "text": "What's in this video?"
336
- },
337
- {
338
- "type": "video_url",
339
- "video_url": {
340
- "url": video_url
341
- },
342
- },
343
- ],
344
- }],
345
- model=model,
346
- max_completion_tokens=64,
347
-)
339
+ video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
340
349
-result = chat_completion_from_url.choices[0].message.content
350
-print("Chat completion output from image url:", result)
351
-```
341
+ ## Use video url in the payload
342
+ chat_completion_from_url = client.chat.completions.create(
343
+ messages=[{
344
+ "role":
345
+ "user",
346
+ "content": [
347
+ {
348
+ "type": "text",
349
+ "text": "What's in this video?"
350
+ },
351
+ {
352
+ "type": "video_url",
353
+ "video_url": {
354
+ "url": video_url
355
+ },
356
+ },
357
+ ],
358
+ }],
359
+ model=model,
360
+ max_completion_tokens=64,
361
+ )
362
+
363
+ result = chat_completion_from_url.choices[0].message.content
364
+ print("Chat completion output from image url:", result)
365
+ ```
366
367
Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
368
@@ -373,84 +387,88 @@ vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b
387
388
Then, you can use the OpenAI client as follows:
389
376
-```python
377
-import base64
378
-import requests
379
-from openai import OpenAI
380
-from vllm.assets.audio import AudioAsset
390
+??? Code
391
382
-def encode_base64_content_from_url(content_url: str) -> str:
383
- """Encode a content retrieved from a remote url to base64 format."""
392
+ ```python
393
+ import base64
394
+ import requests
395
+ from openai import OpenAI
396
+ from vllm.assets.audio import AudioAsset
397
385
- with requests.get(content_url) as response:
386
- response.raise_for_status()
387
- result = base64.b64encode(response.content).decode('utf-8')
398
+ def encode_base64_content_from_url(content_url: str) -> str:
399
+ """Encode a content retrieved from a remote url to base64 format."""
400
389
- return result
401
+ with requests.get(content_url) as response:
402
+ response.raise_for_status()
403
+ result = base64.b64encode(response.content).decode('utf-8')
404
391
-openai_api_key = "EMPTY"
392
-openai_api_base = "http://localhost:8000/v1"
405
+ return result
406
394
-client = OpenAI(
395
- api_key=openai_api_key,
396
- base_url=openai_api_base,
397
-)
407
+ openai_api_key = "EMPTY"
408
+ openai_api_base = "http://localhost:8000/v1"
409
399
-# Any format supported by librosa is supported
400
-audio_url = AudioAsset("winning_call").url
401
-audio_base64 = encode_base64_content_from_url(audio_url)
410
+ client = OpenAI(
411
+ api_key=openai_api_key,
412
+ base_url=openai_api_base,
413
+ )
414
403
-chat_completion_from_base64 = client.chat.completions.create(
404
- messages=[{
405
- "role": "user",
406
- "content": [
407
- {
408
- "type": "text",
409
- "text": "What's in this audio?"
410
- },
411
- {
412
- "type": "input_audio",
413
- "input_audio": {
414
- "data": audio_base64,
415
- "format": "wav"
416
- },
417
- },
418
- ],
419
- }],
420
- model=model,
421
- max_completion_tokens=64,
422
-)
415
+ # Any format supported by librosa is supported
416
+ audio_url = AudioAsset("winning_call").url
417
+ audio_base64 = encode_base64_content_from_url(audio_url)
418
424
-result = chat_completion_from_base64.choices[0].message.content
425
-print("Chat completion output from input audio:", result)
426
-```
419
+ chat_completion_from_base64 = client.chat.completions.create(
420
+ messages=[{
421
+ "role": "user",
422
+ "content": [
423
+ {
424
+ "type": "text",
425
+ "text": "What's in this audio?"
426
+ },
427
+ {
428
+ "type": "input_audio",
429
+ "input_audio": {
430
+ "data": audio_base64,
431
+ "format": "wav"
432
+ },
433
+ },
434
+ ],
435
+ }],
436
+ model=model,
437
+ max_completion_tokens=64,
438
+ )
439
+
440
+ result = chat_completion_from_base64.choices[0].message.content
441
+ print("Chat completion output from input audio:", result)
442
+ ```
443
444
Alternatively, you can pass `audio_url`, which is the audio counterpart of `image_url` for image input:
445
430
-```python
431
-chat_completion_from_url = client.chat.completions.create(
432
- messages=[{
433
- "role": "user",
434
- "content": [
435
- {
436
- "type": "text",
437
- "text": "What's in this audio?"
438
- },
439
- {
440
- "type": "audio_url",
441
- "audio_url": {
442
- "url": audio_url
443
- },
444
- },
445
- ],
446
- }],
447
- model=model,
448
- max_completion_tokens=64,
449
-)
446
+??? Code
447
451
-result = chat_completion_from_url.choices[0].message.content
452
-print("Chat completion output from audio url:", result)
453
-```
448
+ ```python
449
+ chat_completion_from_url = client.chat.completions.create(
450
+ messages=[{
451
+ "role": "user",
452
+ "content": [
453
+ {
454
+ "type": "text",
455
+ "text": "What's in this audio?"
456
+ },
457
+ {
458
+ "type": "audio_url",
459
+ "audio_url": {
460
+ "url": audio_url
461
+ },
462
+ },
463
+ ],
464
+ }],
465
+ model=model,
466
+ max_completion_tokens=64,
467
+ )
468
+
469
+ result = chat_completion_from_url.choices[0].message.content
470
+ print("Chat completion output from audio url:", result)
471
+ ```
472
473
Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
474
@@ -470,61 +488,63 @@ pass a tensor of shape to the corresponding field of the multi-modal dictionary.
488
For image embeddings, you can pass the base64-encoded tensor to the `image_embeds` field.
489
The following example demonstrates how to pass image embeddings to the OpenAI server:
490
473
-```python
474
-image_embedding = torch.load(...)
475
-grid_thw = torch.load(...) # Required by Qwen/Qwen2-VL-2B-Instruct
476
-
477
-buffer = io.BytesIO()
478
-torch.save(image_embedding, buffer)
479
-buffer.seek(0)
480
-binary_data = buffer.read()
481
-base64_image_embedding = base64.b64encode(binary_data).decode('utf-8')
482
-
483
-client = OpenAI(
484
- # defaults to os.environ.get("OPENAI_API_KEY")
485
- api_key=openai_api_key,
486
- base_url=openai_api_base,
487
-)
488
-
489
-# Basic usage - this is equivalent to the LLaVA example for offline inference
490
-model = "llava-hf/llava-1.5-7b-hf"
491
-embeds = {
492
- "type": "image_embeds",
493
- "image_embeds": f"{base64_image_embedding}"
494
-}
495
-
496
-# Pass additional parameters (available to Qwen2-VL and MiniCPM-V)
497
-model = "Qwen/Qwen2-VL-2B-Instruct"
498
-embeds = {
499
- "type": "image_embeds",
500
- "image_embeds": {
501
- "image_embeds": f"{base64_image_embedding}" , # Required
502
- "image_grid_thw": f"{base64_image_grid_thw}" # Required by Qwen/Qwen2-VL-2B-Instruct
503
- },
504
-}
505
-model = "openbmb/MiniCPM-V-2_6"
506
-embeds = {
507
- "type": "image_embeds",
508
- "image_embeds": {
509
- "image_embeds": f"{base64_image_embedding}" , # Required
510
- "image_sizes": f"{base64_image_sizes}" # Required by openbmb/MiniCPM-V-2_6
511
- },
512
-}
513
-chat_completion = client.chat.completions.create(
514
- messages=[
515
- {"role": "system", "content": "You are a helpful assistant."},
516
- {"role": "user", "content": [
517
- {
518
- "type": "text",
519
- "text": "What's in this image?",
491
+??? Code
492
+
493
+ ```python
494
+ image_embedding = torch.load(...)
495
+ grid_thw = torch.load(...) # Required by Qwen/Qwen2-VL-2B-Instruct
496
+
497
+ buffer = io.BytesIO()
498
+ torch.save(image_embedding, buffer)
499
+ buffer.seek(0)
500
+ binary_data = buffer.read()
501
+ base64_image_embedding = base64.b64encode(binary_data).decode('utf-8')
502
+
503
+ client = OpenAI(
504
+ # defaults to os.environ.get("OPENAI_API_KEY")
505
+ api_key=openai_api_key,
506
+ base_url=openai_api_base,
507
+ )
508
+
509
+ # Basic usage - this is equivalent to the LLaVA example for offline inference
510
+ model = "llava-hf/llava-1.5-7b-hf"
511
+ embeds = {
512
+ "type": "image_embeds",
513
+ "image_embeds": f"{base64_image_embedding}"
514
+ }
515
+
516
+ # Pass additional parameters (available to Qwen2-VL and MiniCPM-V)
517
+ model = "Qwen/Qwen2-VL-2B-Instruct"
518
+ embeds = {
519
+ "type": "image_embeds",
520
+ "image_embeds": {
521
+ "image_embeds": f"{base64_image_embedding}" , # Required
522
+ "image_grid_thw": f"{base64_image_grid_thw}" # Required by Qwen/Qwen2-VL-2B-Instruct
523
},
521
- embeds,
522
- ],
523
- },
524
-],
525
- model=model,
526
-)
527
-```
524
+ }
525
+ model = "openbmb/MiniCPM-V-2_6"
526
+ embeds = {
527
+ "type": "image_embeds",
528
+ "image_embeds": {
529
+ "image_embeds": f"{base64_image_embedding}" , # Required
530
+ "image_sizes": f"{base64_image_sizes}" # Required by openbmb/MiniCPM-V-2_6
531
+ },
532
+ }
533
+ chat_completion = client.chat.completions.create(
534
+ messages=[
535
+ {"role": "system", "content": "You are a helpful assistant."},
536
+ {"role": "user", "content": [
537
+ {
538
+ "type": "text",
539
+ "text": "What's in this image?",
540
+ },
541
+ embeds,
542
+ ],
543
+ },
544
+ ],
545
+ model=model,
546
+ )
547
+ ```
548
549
!!! note
550
Only one message can contain `{"type": "image_embeds"}`.
docs/features/quantization/auto_awq.md
+46
-42
@@ -15,29 +15,31 @@ pip install autoawq
15
16
After installing AutoAWQ, you are ready to quantize a model. Please refer to the [AutoAWQ documentation](https://casper-hansen.github.io/AutoAWQ/examples/#basic-quantization) for further details. Here is an example of how to quantize `mistralai/Mistral-7B-Instruct-v0.2`:
17
18
-```python
19
-from awq import AutoAWQForCausalLM
20
-from transformers import AutoTokenizer
18
+??? Code
19
22
-model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
23
-quant_path = 'mistral-instruct-v0.2-awq'
24
-quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
20
+ ```python
21
+ from awq import AutoAWQForCausalLM
22
+ from transformers import AutoTokenizer
23
26
-# Load model
27
-model = AutoAWQForCausalLM.from_pretrained(
28
- model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
29
-)
30
-tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
24
+ model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
25
+ quant_path = 'mistral-instruct-v0.2-awq'
26
+ quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
27
32
-# Quantize
33
-model.quantize(tokenizer, quant_config=quant_config)
28
+ # Load model
29
+ model = AutoAWQForCausalLM.from_pretrained(
30
+ model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
31
+ )
32
+ tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
33
35
-# Save quantized model
36
-model.save_quantized(quant_path)
37
-tokenizer.save_pretrained(quant_path)
34
+ # Quantize
35
+ model.quantize(tokenizer, quant_config=quant_config)
36
39
-print(f'Model is quantized and saved at "{quant_path}"')
40
-```
37
+ # Save quantized model
38
+ model.save_quantized(quant_path)
39
+ tokenizer.save_pretrained(quant_path)
40
+
41
+ print(f'Model is quantized and saved at "{quant_path}"')
42
+ ```
43
44
To run an AWQ model with vLLM, you can use [TheBloke/Llama-2-7b-Chat-AWQ](https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ) with the following command:
45
@@ -49,27 +51,29 @@ python examples/offline_inference/llm_engine_example.py \
51
52
AWQ models are also supported directly through the LLM entrypoint:
53
52
-```python
53
-from vllm import LLM, SamplingParams
54
-
55
-# Sample prompts.
56
-prompts = [
57
- "Hello, my name is",
58
- "The president of the United States is",
59
- "The capital of France is",
60
- "The future of AI is",
61
-]
62
-# Create a sampling params object.
63
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
64
-
65
-# Create an LLM.
66
-llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="AWQ")
67
-# Generate texts from the prompts. The output is a list of RequestOutput objects
68
-# that contain the prompt, generated text, and other information.
69
-outputs = llm.generate(prompts, sampling_params)
70
-# Print the outputs.
71
-for output in outputs:
72
- prompt = output.prompt
73
- generated_text = output.outputs[0].text
74
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
75
-```
54
+??? Code
55
+
56
+ ```python
57
+ from vllm import LLM, SamplingParams
58
+
59
+ # Sample prompts.
60
+ prompts = [
61
+ "Hello, my name is",
62
+ "The president of the United States is",
63
+ "The capital of France is",
64
+ "The future of AI is",
65
+ ]
66
+ # Create a sampling params object.
67
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
68
+
69
+ # Create an LLM.
70
+ llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="AWQ")
71
+ # Generate texts from the prompts. The output is a list of RequestOutput objects
72
+ # that contain the prompt, generated text, and other information.
73
+ outputs = llm.generate(prompts, sampling_params)
74
+ # Print the outputs.
75
+ for output in outputs:
76
+ prompt = output.prompt
77
+ generated_text = output.outputs[0].text
78
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
79
+ ```
docs/features/quantization/bitblas.md
+16
-14
@@ -43,17 +43,19 @@ llm = LLM(
43
44
## Read gptq format checkpoint
45
46
-```python
47
-from vllm import LLM
48
-import torch
49
-
50
-# "hxbgsyxh/llama-13b-4bit-g-1" is a pre-quantized checkpoint.
51
-model_id = "hxbgsyxh/llama-13b-4bit-g-1"
52
-llm = LLM(
53
- model=model_id,
54
- dtype=torch.float16,
55
- trust_remote_code=True,
56
- quantization="bitblas",
57
- max_model_len=1024
58
-)
59
-```
46
+??? Code
47
+
48
+ ```python
49
+ from vllm import LLM
50
+ import torch
51
+
52
+ # "hxbgsyxh/llama-13b-4bit-g-1" is a pre-quantized checkpoint.
53
+ model_id = "hxbgsyxh/llama-13b-4bit-g-1"
54
+ llm = LLM(
55
+ model=model_id,
56
+ dtype=torch.float16,
57
+ trust_remote_code=True,
58
+ quantization="bitblas",
59
+ max_model_len=1024
60
+ )
61
+ ```
docs/features/quantization/fp8.md
+15
-13
@@ -58,22 +58,24 @@ For FP8 quantization, we can recover accuracy with simple RTN quantization. We r
58
59
Since simple RTN does not require data for weight quantization and the activations are quantized dynamically, we do not need any calibration data for this quantization flow.
60
61
-```python
62
-from llmcompressor.transformers import oneshot
63
-from llmcompressor.modifiers.quantization import QuantizationModifier
61
+??? Code
62
65
-# Configure the simple PTQ quantization
66
-recipe = QuantizationModifier(
67
- targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
63
+ ```python
64
+ from llmcompressor.transformers import oneshot
65
+ from llmcompressor.modifiers.quantization import QuantizationModifier
66
69
-# Apply the quantization algorithm.
70
-oneshot(model=model, recipe=recipe)
67
+ # Configure the simple PTQ quantization
68
+ recipe = QuantizationModifier(
69
+ targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
70
72
-# Save the model: Meta-Llama-3-8B-Instruct-FP8-Dynamic
73
-SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
74
-model.save_pretrained(SAVE_DIR)
75
-tokenizer.save_pretrained(SAVE_DIR)
76
-```
71
+ # Apply the quantization algorithm.
72
+ oneshot(model=model, recipe=recipe)
73
+
74
+ # Save the model: Meta-Llama-3-8B-Instruct-FP8-Dynamic
75
+ SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
76
+ model.save_pretrained(SAVE_DIR)
77
+ tokenizer.save_pretrained(SAVE_DIR)
78
+ ```
79
80
### 3. Evaluating Accuracy
81
docs/features/quantization/gguf.md
+41
-39
@@ -41,42 +41,44 @@ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
41
42
You can also use the GGUF model directly through the LLM entrypoint:
43
44
-```python
45
-from vllm import LLM, SamplingParams
46
-
47
-# In this script, we demonstrate how to pass input to the chat method:
48
-conversation = [
49
- {
50
- "role": "system",
51
- "content": "You are a helpful assistant"
52
- },
53
- {
54
- "role": "user",
55
- "content": "Hello"
56
- },
57
- {
58
- "role": "assistant",
59
- "content": "Hello! How can I assist you today?"
60
- },
61
- {
62
- "role": "user",
63
- "content": "Write an essay about the importance of higher education.",
64
- },
65
-]
66
-
67
-# Create a sampling params object.
68
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
69
-
70
-# Create an LLM.
71
-llm = LLM(model="./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf",
72
- tokenizer="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
73
-# Generate texts from the prompts. The output is a list of RequestOutput objects
74
-# that contain the prompt, generated text, and other information.
75
-outputs = llm.chat(conversation, sampling_params)
76
-
77
-# Print the outputs.
78
-for output in outputs:
79
- prompt = output.prompt
80
- generated_text = output.outputs[0].text
81
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
82
-```
44
+??? Code
45
+
46
+ ```python
47
+ from vllm import LLM, SamplingParams
48
+
49
+ # In this script, we demonstrate how to pass input to the chat method:
50
+ conversation = [
51
+ {
52
+ "role": "system",
53
+ "content": "You are a helpful assistant"
54
+ },
55
+ {
56
+ "role": "user",
57
+ "content": "Hello"
58
+ },
59
+ {
60
+ "role": "assistant",
61
+ "content": "Hello! How can I assist you today?"
62
+ },
63
+ {
64
+ "role": "user",
65
+ "content": "Write an essay about the importance of higher education.",
66
+ },
67
+ ]
68
+
69
+ # Create a sampling params object.
70
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
71
+
72
+ # Create an LLM.
73
+ llm = LLM(model="./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf",
74
+ tokenizer="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
75
+ # Generate texts from the prompts. The output is a list of RequestOutput objects
76
+ # that contain the prompt, generated text, and other information.
77
+ outputs = llm.chat(conversation, sampling_params)
78
+
79
+ # Print the outputs.
80
+ for output in outputs:
81
+ prompt = output.prompt
82
+ generated_text = output.outputs[0].text
83
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
84
+ ```
docs/features/quantization/gptqmodel.md
+48
-44
@@ -31,28 +31,30 @@ After installing GPTQModel, you are ready to quantize a model. Please refer to t
31
32
Here is an example of how to quantize `meta-llama/Llama-3.2-1B-Instruct`:
33
34
-```python
35
-from datasets import load_dataset
36
-from gptqmodel import GPTQModel, QuantizeConfig
34
+??? Code
35
38
-model_id = "meta-llama/Llama-3.2-1B-Instruct"
39
-quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"
36
+ ```python
37
+ from datasets import load_dataset
38
+ from gptqmodel import GPTQModel, QuantizeConfig
39
41
-calibration_dataset = load_dataset(
42
- "allenai/c4",
43
- data_files="en/c4-train.00001-of-01024.json.gz",
44
- split="train"
45
- ).select(range(1024))["text"]
40
+ model_id = "meta-llama/Llama-3.2-1B-Instruct"
41
+ quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"
42
47
-quant_config = QuantizeConfig(bits=4, group_size=128)
43
+ calibration_dataset = load_dataset(
44
+ "allenai/c4",
45
+ data_files="en/c4-train.00001-of-01024.json.gz",
46
+ split="train"
47
+ ).select(range(1024))["text"]
48
49
-model = GPTQModel.load(model_id, quant_config)
49
+ quant_config = QuantizeConfig(bits=4, group_size=128)
50
51
-# increase `batch_size` to match gpu/vram specs to speed up quantization
52
-model.quantize(calibration_dataset, batch_size=2)
51
+ model = GPTQModel.load(model_id, quant_config)
52
54
-model.save(quant_path)
55
-```
53
+ # increase `batch_size` to match gpu/vram specs to speed up quantization
54
+ model.quantize(calibration_dataset, batch_size=2)
55
+
56
+ model.save(quant_path)
57
+ ```
58
59
## Running a quantized model with vLLM
60
@@ -67,32 +69,34 @@ python examples/offline_inference/llm_engine_example.py \
69
70
GPTQModel quantized models are also supported directly through the LLM entrypoint:
71
70
-```python
71
-from vllm import LLM, SamplingParams
72
-
73
-# Sample prompts.
74
-prompts = [
75
- "Hello, my name is",
76
- "The president of the United States is",
77
- "The capital of France is",
78
- "The future of AI is",
79
-]
80
-
81
-# Create a sampling params object.
82
-sampling_params = SamplingParams(temperature=0.6, top_p=0.9)
83
-
84
-# Create an LLM.
85
-llm = LLM(model="ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2")
86
-
87
-# Generate texts from the prompts. The output is a list of RequestOutput objects
88
-# that contain the prompt, generated text, and other information.
89
-outputs = llm.generate(prompts, sampling_params)
90
-
91
-# Print the outputs.
92
-print("-"*50)
93
-for output in outputs:
94
- prompt = output.prompt
95
- generated_text = output.outputs[0].text
96
- print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}")
72
+??? Code
73
+
74
+ ```python
75
+ from vllm import LLM, SamplingParams
76
+
77
+ # Sample prompts.
78
+ prompts = [
79
+ "Hello, my name is",
80
+ "The president of the United States is",
81
+ "The capital of France is",
82
+ "The future of AI is",
83
+ ]
84
+
85
+ # Create a sampling params object.
86
+ sampling_params = SamplingParams(temperature=0.6, top_p=0.9)
87
+
88
+ # Create an LLM.
89
+ llm = LLM(model="ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2")
90
+
91
+ # Generate texts from the prompts. The output is a list of RequestOutput objects
92
+ # that contain the prompt, generated text, and other information.
93
+ outputs = llm.generate(prompts, sampling_params)
94
+
95
+ # Print the outputs.
96
print("-"*50)
98
-```
97
+ for output in outputs:
98
+ prompt = output.prompt
99
+ generated_text = output.outputs[0].text
100
+ print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}")
101
+ print("-"*50)
102
+ ```
docs/features/quantization/int4.md
+68
-62
@@ -53,51 +53,55 @@ When quantizing weights to INT4, you need sample data to estimate the weight upd
53
It's best to use calibration data that closely matches your deployment data.
54
For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
55
56
-```python
57
-from datasets import load_dataset
56
+??? Code
57
59
-NUM_CALIBRATION_SAMPLES = 512
60
-MAX_SEQUENCE_LENGTH = 2048
58
+ ```python
59
+ from datasets import load_dataset
60
62
-# Load and preprocess the dataset
63
-ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
64
-ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
61
+ NUM_CALIBRATION_SAMPLES = 512
62
+ MAX_SEQUENCE_LENGTH = 2048
63
66
-def preprocess(example):
67
- return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
68
-ds = ds.map(preprocess)
64
+ # Load and preprocess the dataset
65
+ ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
66
+ ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
67
70
-def tokenize(sample):
71
- return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
72
-ds = ds.map(tokenize, remove_columns=ds.column_names)
73
-```
68
+ def preprocess(example):
69
+ return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
70
+ ds = ds.map(preprocess)
71
+
72
+ def tokenize(sample):
73
+ return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
74
+ ds = ds.map(tokenize, remove_columns=ds.column_names)
75
+ ```
76
77
### 3. Applying Quantization
78
79
Now, apply the quantization algorithms:
80
79
-```python
80
-from llmcompressor.transformers import oneshot
81
-from llmcompressor.modifiers.quantization import GPTQModifier
82
-from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
83
-
84
-# Configure the quantization algorithms
85
-recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
86
-
87
-# Apply quantization
88
-oneshot(
89
- model=model,
90
- dataset=ds,
91
- recipe=recipe,
92
- max_seq_length=MAX_SEQUENCE_LENGTH,
93
- num_calibration_samples=NUM_CALIBRATION_SAMPLES,
94
-)
81
+??? Code
82
96
-# Save the compressed model: Meta-Llama-3-8B-Instruct-W4A16-G128
97
-SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
98
-model.save_pretrained(SAVE_DIR, save_compressed=True)
99
-tokenizer.save_pretrained(SAVE_DIR)
100
-```
83
+ ```python
84
+ from llmcompressor.transformers import oneshot
85
+ from llmcompressor.modifiers.quantization import GPTQModifier
86
+ from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
87
+
88
+ # Configure the quantization algorithms
89
+ recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
90
+
91
+ # Apply quantization
92
+ oneshot(
93
+ model=model,
94
+ dataset=ds,
95
+ recipe=recipe,
96
+ max_seq_length=MAX_SEQUENCE_LENGTH,
97
+ num_calibration_samples=NUM_CALIBRATION_SAMPLES,
98
+ )
99
+
100
+ # Save the compressed model: Meta-Llama-3-8B-Instruct-W4A16-G128
101
+ SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
102
+ model.save_pretrained(SAVE_DIR, save_compressed=True)
103
+ tokenizer.save_pretrained(SAVE_DIR)
104
+ ```
105
106
This process creates a W4A16 model with weights quantized to 4-bit integers.
107
@@ -137,34 +141,36 @@ $ lm_eval --model vllm \
141
142
The following is an example of an expanded quantization recipe you can tune to your own use case:
143
140
-```python
141
-from compressed_tensors.quantization import (
142
- QuantizationArgs,
143
- QuantizationScheme,
144
- QuantizationStrategy,
145
- QuantizationType,
146
-)
147
-recipe = GPTQModifier(
148
- targets="Linear",
149
- config_groups={
150
- "config_group": QuantizationScheme(
151
- targets=["Linear"],
152
- weights=QuantizationArgs(
153
- num_bits=4,
154
- type=QuantizationType.INT,
155
- strategy=QuantizationStrategy.GROUP,
156
- group_size=128,
157
- symmetric=True,
158
- dynamic=False,
159
- actorder="weight",
144
+??? Code
145
+
146
+ ```python
147
+ from compressed_tensors.quantization import (
148
+ QuantizationArgs,
149
+ QuantizationScheme,
150
+ QuantizationStrategy,
151
+ QuantizationType,
152
+ )
153
+ recipe = GPTQModifier(
154
+ targets="Linear",
155
+ config_groups={
156
+ "config_group": QuantizationScheme(
157
+ targets=["Linear"],
158
+ weights=QuantizationArgs(
159
+ num_bits=4,
160
+ type=QuantizationType.INT,
161
+ strategy=QuantizationStrategy.GROUP,
162
+ group_size=128,
163
+ symmetric=True,
164
+ dynamic=False,
165
+ actorder="weight",
166
+ ),
167
),
161
- ),
162
- },
163
- ignore=["lm_head"],
164
- update_size=NUM_CALIBRATION_SAMPLES,
165
- dampening_frac=0.01
166
-)
167
-```
168
+ },
169
+ ignore=["lm_head"],
170
+ update_size=NUM_CALIBRATION_SAMPLES,
171
+ dampening_frac=0.01
172
+ )
173
+ ```
174
175
## Troubleshooting and Support
176
docs/features/quantization/int8.md
+45
-39
@@ -54,54 +54,60 @@ When quantizing activations to INT8, you need sample data to estimate the activa
54
It's best to use calibration data that closely matches your deployment data.
55
For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
56
57
-```python
58
-from datasets import load_dataset
57
+??? Code
58
60
-NUM_CALIBRATION_SAMPLES = 512
61
-MAX_SEQUENCE_LENGTH = 2048
59
+ ```python
60
+ from datasets import load_dataset
61
63
-# Load and preprocess the dataset
64
-ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
65
-ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
62
+ NUM_CALIBRATION_SAMPLES = 512
63
+ MAX_SEQUENCE_LENGTH = 2048
64
67
-def preprocess(example):
68
- return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
69
-ds = ds.map(preprocess)
65
+ # Load and preprocess the dataset
66
+ ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
67
+ ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
68
71
-def tokenize(sample):
72
- return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
73
-ds = ds.map(tokenize, remove_columns=ds.column_names)
74
-```
69
+ def preprocess(example):
70
+ return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
71
+ ds = ds.map(preprocess)
72
+
73
+ def tokenize(sample):
74
+ return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
75
+ ds = ds.map(tokenize, remove_columns=ds.column_names)
76
+ ```
77
+
78
+</details>
79
80
### 3. Applying Quantization
81
82
Now, apply the quantization algorithms:
83
80
-```python
81
-from llmcompressor.transformers import oneshot
82
-from llmcompressor.modifiers.quantization import GPTQModifier
83
-from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
84
-
85
-# Configure the quantization algorithms
86
-recipe = [
87
- SmoothQuantModifier(smoothing_strength=0.8),
88
- GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
89
-]
90
-
91
-# Apply quantization
92
-oneshot(
93
- model=model,
94
- dataset=ds,
95
- recipe=recipe,
96
- max_seq_length=MAX_SEQUENCE_LENGTH,
97
- num_calibration_samples=NUM_CALIBRATION_SAMPLES,
98
-)
99
-
100
-# Save the compressed model: Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token
101
-SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
102
-model.save_pretrained(SAVE_DIR, save_compressed=True)
103
-tokenizer.save_pretrained(SAVE_DIR)
104
-```
84
+??? Code
85
+
86
+ ```python
87
+ from llmcompressor.transformers import oneshot
88
+ from llmcompressor.modifiers.quantization import GPTQModifier
89
+ from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
90
+
91
+ # Configure the quantization algorithms
92
+ recipe = [
93
+ SmoothQuantModifier(smoothing_strength=0.8),
94
+ GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
95
+ ]
96
+
97
+ # Apply quantization
98
+ oneshot(
99
+ model=model,
100
+ dataset=ds,
101
+ recipe=recipe,
102
+ max_seq_length=MAX_SEQUENCE_LENGTH,
103
+ num_calibration_samples=NUM_CALIBRATION_SAMPLES,
104
+ )
105
+
106
+ # Save the compressed model: Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token
107
+ SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
108
+ model.save_pretrained(SAVE_DIR, save_compressed=True)
109
+ tokenizer.save_pretrained(SAVE_DIR)
110
+ ```
111
112
This process creates a W8A8 model with weights and activations quantized to 8-bit integers.
113
docs/features/quantization/modelopt.md
+39
-35
@@ -14,24 +14,26 @@ You can quantize HuggingFace models using the example scripts provided in the Te
14
15
Below is an example showing how to quantize a model using modelopt's PTQ API:
16
17
-```python
18
-import modelopt.torch.quantization as mtq
19
-from transformers import AutoModelForCausalLM
17
+??? Code
18
21
-# Load the model from HuggingFace
22
-model = AutoModelForCausalLM.from_pretrained("<path_or_model_id>")
19
+ ```python
20
+ import modelopt.torch.quantization as mtq
21
+ from transformers import AutoModelForCausalLM
22
24
-# Select the quantization config, for example, FP8
25
-config = mtq.FP8_DEFAULT_CFG
23
+ # Load the model from HuggingFace
24
+ model = AutoModelForCausalLM.from_pretrained("<path_or_model_id>")
25
27
-# Define a forward loop function for calibration
28
-def forward_loop(model):
29
- for data in calib_set:
30
- model(data)
26
+ # Select the quantization config, for example, FP8
27
+ config = mtq.FP8_DEFAULT_CFG
28
32
-# PTQ with in-place replacement of quantized modules
33
-model = mtq.quantize(model, config, forward_loop)
34
-```
29
+ # Define a forward loop function for calibration
30
+ def forward_loop(model):
31
+ for data in calib_set:
32
+ model(data)
33
+
34
+ # PTQ with in-place replacement of quantized modules
35
+ model = mtq.quantize(model, config, forward_loop)
36
+ ```
37
38
After the model is quantized, you can export it to a quantized checkpoint using the export API:
39
@@ -48,31 +50,33 @@ with torch.inference_mode():
50
51
The quantized checkpoint can then be deployed with vLLM. As an example, the following code shows how to deploy `nvidia/Llama-3.1-8B-Instruct-FP8`, which is the FP8 quantized checkpoint derived from `meta-llama/Llama-3.1-8B-Instruct`, using vLLM:
52
51
-```python
52
-from vllm import LLM, SamplingParams
53
+??? Code
54
54
-def main():
55
+ ```python
56
+ from vllm import LLM, SamplingParams
57
56
- model_id = "nvidia/Llama-3.1-8B-Instruct-FP8"
57
- # Ensure you specify quantization='modelopt' when loading the modelopt checkpoint
58
- llm = LLM(model=model_id, quantization="modelopt", trust_remote_code=True)
58
+ def main():
59
60
- sampling_params = SamplingParams(temperature=0.8, top_p=0.9)
60
+ model_id = "nvidia/Llama-3.1-8B-Instruct-FP8"
61
+ # Ensure you specify quantization='modelopt' when loading the modelopt checkpoint
62
+ llm = LLM(model=model_id, quantization="modelopt", trust_remote_code=True)
63
62
- prompts = [
63
- "Hello, my name is",
64
- "The president of the United States is",
65
- "The capital of France is",
66
- "The future of AI is",
67
- ]
64
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.9)
65
69
- outputs = llm.generate(prompts, sampling_params)
66
+ prompts = [
67
+ "Hello, my name is",
68
+ "The president of the United States is",
69
+ "The capital of France is",
70
+ "The future of AI is",
71
+ ]
72
71
- for output in outputs:
72
- prompt = output.prompt
73
- generated_text = output.outputs[0].text
74
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
73
+ outputs = llm.generate(prompts, sampling_params)
74
76
-if __name__ == "__main__":
77
- main()
78
-```
75
+ for output in outputs:
76
+ prompt = output.prompt
77
+ generated_text = output.outputs[0].text
78
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
79
+
80
+ if __name__ == "__main__":
81
+ main()
82
+ ```
docs/features/quantization/quantized_kvcache.md
+75
-71
@@ -35,20 +35,22 @@ Studies have shown that FP8 E4M3 quantization typically only minimally degrades
35
36
Here is an example of how to enable FP8 quantization:
37
38
-```python
39
-# To calculate kv cache scales on the fly enable the calculate_kv_scales
40
-# parameter
38
+??? Code
39
42
-from vllm import LLM, SamplingParams
40
+ ```python
41
+ # To calculate kv cache scales on the fly enable the calculate_kv_scales
42
+ # parameter
43
44
-sampling_params = SamplingParams(temperature=0.7, top_p=0.8)
45
-llm = LLM(model="meta-llama/Llama-2-7b-chat-hf",
46
- kv_cache_dtype="fp8",
47
- calculate_kv_scales=True)
48
-prompt = "London is the capital of"
49
-out = llm.generate(prompt, sampling_params)[0].outputs[0].text
50
-print(out)
51
-```
44
+ from vllm import LLM, SamplingParams
45
+
46
+ sampling_params = SamplingParams(temperature=0.7, top_p=0.8)
47
+ llm = LLM(model="meta-llama/Llama-2-7b-chat-hf",
48
+ kv_cache_dtype="fp8",
49
+ calculate_kv_scales=True)
50
+ prompt = "London is the capital of"
51
+ out = llm.generate(prompt, sampling_params)[0].outputs[0].text
52
+ print(out)
53
+ ```
54
55
The `kv_cache_dtype` argument specifies the data type for KV cache storage:
56
- `"auto"`: Uses the model's default "unquantized" data type
@@ -71,67 +73,69 @@ pip install llmcompressor
73
74
Here's a complete example using `meta-llama/Llama-3.1-8B-Instruct` (most models can use this same pattern):
75
74
-```python
75
-from datasets import load_dataset
76
-from transformers import AutoModelForCausalLM, AutoTokenizer
77
-from llmcompressor.transformers import oneshot
78
-
79
-# Select model and load it
80
-MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
81
-model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto")
82
-tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
83
-
84
-# Select calibration dataset
85
-DATASET_ID = "HuggingFaceH4/ultrachat_200k"
86
-DATASET_SPLIT = "train_sft"
87
-
88
-# Configure calibration parameters
89
-NUM_CALIBRATION_SAMPLES = 512 # 512 samples is a good starting point
90
-MAX_SEQUENCE_LENGTH = 2048
91
-
92
-# Load and preprocess dataset
93
-ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
94
-ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
95
-
96
-def process_and_tokenize(example):
97
- text = tokenizer.apply_chat_template(example["messages"], tokenize=False)
98
- return tokenizer(
99
- text,
100
- padding=False,
101
- max_length=MAX_SEQUENCE_LENGTH,
102
- truncation=True,
103
- add_special_tokens=False,
76
+??? Code
77
+
78
+ ```python
79
+ from datasets import load_dataset
80
+ from transformers import AutoModelForCausalLM, AutoTokenizer
81
+ from llmcompressor.transformers import oneshot
82
+
83
+ # Select model and load it
84
+ MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
85
+ model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto")
86
+ tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
87
+
88
+ # Select calibration dataset
89
+ DATASET_ID = "HuggingFaceH4/ultrachat_200k"
90
+ DATASET_SPLIT = "train_sft"
91
+
92
+ # Configure calibration parameters
93
+ NUM_CALIBRATION_SAMPLES = 512 # 512 samples is a good starting point
94
+ MAX_SEQUENCE_LENGTH = 2048
95
+
96
+ # Load and preprocess dataset
97
+ ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
98
+ ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
99
+
100
+ def process_and_tokenize(example):
101
+ text = tokenizer.apply_chat_template(example["messages"], tokenize=False)
102
+ return tokenizer(
103
+ text,
104
+ padding=False,
105
+ max_length=MAX_SEQUENCE_LENGTH,
106
+ truncation=True,
107
+ add_special_tokens=False,
108
+ )
109
+
110
+ ds = ds.map(process_and_tokenize, remove_columns=ds.column_names)
111
+
112
+ # Configure quantization settings
113
+ recipe = """
114
+ quant_stage:
115
+ quant_modifiers:
116
+ QuantizationModifier:
117
+ kv_cache_scheme:
118
+ num_bits: 8
119
+ type: float
120
+ strategy: tensor
121
+ dynamic: false
122
+ symmetric: true
123
+ """
124
+
125
+ # Apply quantization
126
+ oneshot(
127
+ model=model,
128
+ dataset=ds,
129
+ recipe=recipe,
130
+ max_seq_length=MAX_SEQUENCE_LENGTH,
131
+ num_calibration_samples=NUM_CALIBRATION_SAMPLES,
132
)
133
106
-ds = ds.map(process_and_tokenize, remove_columns=ds.column_names)
107
-
108
-# Configure quantization settings
109
-recipe = """
110
-quant_stage:
111
- quant_modifiers:
112
- QuantizationModifier:
113
- kv_cache_scheme:
114
- num_bits: 8
115
- type: float
116
- strategy: tensor
117
- dynamic: false
118
- symmetric: true
119
-"""
120
-
121
-# Apply quantization
122
-oneshot(
123
- model=model,
124
- dataset=ds,
125
- recipe=recipe,
126
- max_seq_length=MAX_SEQUENCE_LENGTH,
127
- num_calibration_samples=NUM_CALIBRATION_SAMPLES,
128
-)
129
-
130
-# Save quantized model: Llama-3.1-8B-Instruct-FP8-KV
131
-SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-KV"
132
-model.save_pretrained(SAVE_DIR, save_compressed=True)
133
-tokenizer.save_pretrained(SAVE_DIR)
134
-```
134
+ # Save quantized model: Llama-3.1-8B-Instruct-FP8-KV
135
+ SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-KV"
136
+ model.save_pretrained(SAVE_DIR, save_compressed=True)
137
+ tokenizer.save_pretrained(SAVE_DIR)
138
+ ```
139
140
The above script will create a folder in your current directory containing your quantized model (e.g., `Llama-3.1-8B-Instruct-FP8-KV`) with calibrated scales.
141
docs/features/quantization/quark.md
+122
-112
@@ -42,20 +42,22 @@ The Quark quantization process can be listed for 5 steps as below:
42
Quark uses [Transformers](https://huggingface.co/docs/transformers/en/index)
43
to fetch model and tokenizer.
44
45
-```python
46
-from transformers import AutoTokenizer, AutoModelForCausalLM
45
+??? Code
46
48
-MODEL_ID = "meta-llama/Llama-2-70b-chat-hf"
49
-MAX_SEQ_LEN = 512
47
+ ```python
48
+ from transformers import AutoTokenizer, AutoModelForCausalLM
49
51
-model = AutoModelForCausalLM.from_pretrained(
52
- MODEL_ID, device_map="auto", torch_dtype="auto",
53
-)
54
-model.eval()
50
+ MODEL_ID = "meta-llama/Llama-2-70b-chat-hf"
51
+ MAX_SEQ_LEN = 512
52
56
-tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, model_max_length=MAX_SEQ_LEN)
57
-tokenizer.pad_token = tokenizer.eos_token
58
-```
53
+ model = AutoModelForCausalLM.from_pretrained(
54
+ MODEL_ID, device_map="auto", torch_dtype="auto",
55
+ )
56
+ model.eval()
57
+
58
+ tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, model_max_length=MAX_SEQ_LEN)
59
+ tokenizer.pad_token = tokenizer.eos_token
60
+ ```
61
62
### 2. Prepare the Calibration Dataloader
63
@@ -63,22 +65,24 @@ Quark uses the [PyTorch Dataloader](https://pytorch.org/tutorials/beginner/basic
65
to load calibration data. For more details about how to use calibration datasets efficiently, please refer
66
to [Adding Calibration Datasets](https://quark.docs.amd.com/latest/pytorch/calibration_datasets.html).
67
66
-```python
67
-from datasets import load_dataset
68
-from torch.utils.data import DataLoader
68
+??? Code
69
70
-BATCH_SIZE = 1
71
-NUM_CALIBRATION_DATA = 512
70
+ ```python
71
+ from datasets import load_dataset
72
+ from torch.utils.data import DataLoader
73
73
-# Load the dataset and get calibration data.
74
-dataset = load_dataset("mit-han-lab/pile-val-backup", split="validation")
75
-text_data = dataset["text"][:NUM_CALIBRATION_DATA]
74
+ BATCH_SIZE = 1
75
+ NUM_CALIBRATION_DATA = 512
76
77
-tokenized_outputs = tokenizer(text_data, return_tensors="pt",
78
- padding=True, truncation=True, max_length=MAX_SEQ_LEN)
79
-calib_dataloader = DataLoader(tokenized_outputs['input_ids'],
80
- batch_size=BATCH_SIZE, drop_last=True)
81
-```
77
+ # Load the dataset and get calibration data.
78
+ dataset = load_dataset("mit-han-lab/pile-val-backup", split="validation")
79
+ text_data = dataset["text"][:NUM_CALIBRATION_DATA]
80
+
81
+ tokenized_outputs = tokenizer(text_data, return_tensors="pt",
82
+ padding=True, truncation=True, max_length=MAX_SEQ_LEN)
83
+ calib_dataloader = DataLoader(tokenized_outputs['input_ids'],
84
+ batch_size=BATCH_SIZE, drop_last=True)
85
+ ```
86
87
### 3. Set the Quantization Configuration
88
@@ -94,42 +98,44 @@ kv-cache and the quantization algorithm is AutoSmoothQuant.
98
AutoSmoothQuant config file for Llama is
99
`examples/torch/language_modeling/llm_ptq/models/llama/autosmoothquant_config.json`.
100
97
-```python
98
-from quark.torch.quantization import (Config, QuantizationConfig,
99
- FP8E4M3PerTensorSpec,
100
- load_quant_algo_config_from_file)
101
-
102
-# Define fp8/per-tensor/static spec.
103
-FP8_PER_TENSOR_SPEC = FP8E4M3PerTensorSpec(observer_method="min_max",
104
- is_dynamic=False).to_quantization_spec()
105
-
106
-# Define global quantization config, input tensors and weight apply FP8_PER_TENSOR_SPEC.
107
-global_quant_config = QuantizationConfig(input_tensors=FP8_PER_TENSOR_SPEC,
108
- weight=FP8_PER_TENSOR_SPEC)
109
-
110
-# Define quantization config for kv-cache layers, output tensors apply FP8_PER_TENSOR_SPEC.
111
-KV_CACHE_SPEC = FP8_PER_TENSOR_SPEC
112
-kv_cache_layer_names_for_llama = ["*k_proj", "*v_proj"]
113
-kv_cache_quant_config = {name :
114
- QuantizationConfig(input_tensors=global_quant_config.input_tensors,
115
- weight=global_quant_config.weight,
116
- output_tensors=KV_CACHE_SPEC)
117
- for name in kv_cache_layer_names_for_llama}
118
-layer_quant_config = kv_cache_quant_config.copy()
119
-
120
-# Define algorithm config by config file.
121
-LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE =
122
- 'examples/torch/language_modeling/llm_ptq/models/llama/autosmoothquant_config.json'
123
-algo_config = load_quant_algo_config_from_file(LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE)
124
-
125
-EXCLUDE_LAYERS = ["lm_head"]
126
-quant_config = Config(
127
- global_quant_config=global_quant_config,
128
- layer_quant_config=layer_quant_config,
129
- kv_cache_quant_config=kv_cache_quant_config,
130
- exclude=EXCLUDE_LAYERS,
131
- algo_config=algo_config)
132
-```
101
+??? Code
102
+
103
+ ```python
104
+ from quark.torch.quantization import (Config, QuantizationConfig,
105
+ FP8E4M3PerTensorSpec,
106
+ load_quant_algo_config_from_file)
107
+
108
+ # Define fp8/per-tensor/static spec.
109
+ FP8_PER_TENSOR_SPEC = FP8E4M3PerTensorSpec(observer_method="min_max",
110
+ is_dynamic=False).to_quantization_spec()
111
+
112
+ # Define global quantization config, input tensors and weight apply FP8_PER_TENSOR_SPEC.
113
+ global_quant_config = QuantizationConfig(input_tensors=FP8_PER_TENSOR_SPEC,
114
+ weight=FP8_PER_TENSOR_SPEC)
115
+
116
+ # Define quantization config for kv-cache layers, output tensors apply FP8_PER_TENSOR_SPEC.
117
+ KV_CACHE_SPEC = FP8_PER_TENSOR_SPEC
118
+ kv_cache_layer_names_for_llama = ["*k_proj", "*v_proj"]
119
+ kv_cache_quant_config = {name :
120
+ QuantizationConfig(input_tensors=global_quant_config.input_tensors,
121
+ weight=global_quant_config.weight,
122
+ output_tensors=KV_CACHE_SPEC)
123
+ for name in kv_cache_layer_names_for_llama}
124
+ layer_quant_config = kv_cache_quant_config.copy()
125
+
126
+ # Define algorithm config by config file.
127
+ LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE =
128
+ 'examples/torch/language_modeling/llm_ptq/models/llama/autosmoothquant_config.json'
129
+ algo_config = load_quant_algo_config_from_file(LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE)
130
+
131
+ EXCLUDE_LAYERS = ["lm_head"]
132
+ quant_config = Config(
133
+ global_quant_config=global_quant_config,
134
+ layer_quant_config=layer_quant_config,
135
+ kv_cache_quant_config=kv_cache_quant_config,
136
+ exclude=EXCLUDE_LAYERS,
137
+ algo_config=algo_config)
138
+ ```
139
140
### 4. Quantize the Model and Export
141
@@ -139,63 +145,67 @@ HuggingFace `safetensors`, you can refer to
145
[HuggingFace format exporting](https://quark.docs.amd.com/latest/pytorch/export/quark_export_hf.html)
146
for more exporting format details.
147
142
-```python
143
-import torch
144
-from quark.torch import ModelQuantizer, ModelExporter
145
-from quark.torch.export import ExporterConfig, JsonExporterConfig
146
-
147
-# Apply quantization.
148
-quantizer = ModelQuantizer(quant_config)
149
-quant_model = quantizer.quantize_model(model, calib_dataloader)
150
-
151
-# Freeze quantized model to export.
152
-freezed_model = quantizer.freeze(model)
153
-
154
-# Define export config.
155
-LLAMA_KV_CACHE_GROUP = ["*k_proj", "*v_proj"]
156
-export_config = ExporterConfig(json_export_config=JsonExporterConfig())
157
-export_config.json_export_config.kv_cache_group = LLAMA_KV_CACHE_GROUP
158
-
159
-# Model: Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant
160
-EXPORT_DIR = MODEL_ID.split("/")[1] + "-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant"
161
-exporter = ModelExporter(config=export_config, export_dir=EXPORT_DIR)
162
-with torch.no_grad():
163
- exporter.export_safetensors_model(freezed_model,
164
- quant_config=quant_config, tokenizer=tokenizer)
165
-```
148
+??? Code
149
+
150
+ ```python
151
+ import torch
152
+ from quark.torch import ModelQuantizer, ModelExporter
153
+ from quark.torch.export import ExporterConfig, JsonExporterConfig
154
+
155
+ # Apply quantization.
156
+ quantizer = ModelQuantizer(quant_config)
157
+ quant_model = quantizer.quantize_model(model, calib_dataloader)
158
+
159
+ # Freeze quantized model to export.
160
+ freezed_model = quantizer.freeze(model)
161
+
162
+ # Define export config.
163
+ LLAMA_KV_CACHE_GROUP = ["*k_proj", "*v_proj"]
164
+ export_config = ExporterConfig(json_export_config=JsonExporterConfig())
165
+ export_config.json_export_config.kv_cache_group = LLAMA_KV_CACHE_GROUP
166
+
167
+ # Model: Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant
168
+ EXPORT_DIR = MODEL_ID.split("/")[1] + "-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant"
169
+ exporter = ModelExporter(config=export_config, export_dir=EXPORT_DIR)
170
+ with torch.no_grad():
171
+ exporter.export_safetensors_model(freezed_model,
172
+ quant_config=quant_config, tokenizer=tokenizer)
173
+ ```
174
175
### 5. Evaluation in vLLM
176
177
Now, you can load and run the Quark quantized model directly through the LLM entrypoint:
178
171
-```python
172
-from vllm import LLM, SamplingParams
173
-
174
-# Sample prompts.
175
-prompts = [
176
- "Hello, my name is",
177
- "The president of the United States is",
178
- "The capital of France is",
179
- "The future of AI is",
180
-]
181
-# Create a sampling params object.
182
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
183
-
184
-# Create an LLM.
185
-llm = LLM(model="Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant",
186
- kv_cache_dtype='fp8',quantization='quark')
187
-# Generate texts from the prompts. The output is a list of RequestOutput objects
188
-# that contain the prompt, generated text, and other information.
189
-outputs = llm.generate(prompts, sampling_params)
190
-# Print the outputs.
191
-print("\nGenerated Outputs:\n" + "-" * 60)
192
-for output in outputs:
193
- prompt = output.prompt
194
- generated_text = output.outputs[0].text
195
- print(f"Prompt: {prompt!r}")
196
- print(f"Output: {generated_text!r}")
197
- print("-" * 60)
198
-```
179
+??? Code
180
+
181
+ ```python
182
+ from vllm import LLM, SamplingParams
183
+
184
+ # Sample prompts.
185
+ prompts = [
186
+ "Hello, my name is",
187
+ "The president of the United States is",
188
+ "The capital of France is",
189
+ "The future of AI is",
190
+ ]
191
+ # Create a sampling params object.
192
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
193
+
194
+ # Create an LLM.
195
+ llm = LLM(model="Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant",
196
+ kv_cache_dtype='fp8',quantization='quark')
197
+ # Generate texts from the prompts. The output is a list of RequestOutput objects
198
+ # that contain the prompt, generated text, and other information.
199
+ outputs = llm.generate(prompts, sampling_params)
200
+ # Print the outputs.
201
+ print("\nGenerated Outputs:\n" + "-" * 60)
202
+ for output in outputs:
203
+ prompt = output.prompt
204
+ generated_text = output.outputs[0].text
205
+ print(f"Prompt: {prompt!r}")
206
+ print(f"Output: {generated_text!r}")
207
+ print("-" * 60)
208
+ ```
209
210
Or, you can use `lm_eval` to evaluate accuracy:
211
docs/features/quantization/torchao.md
+23
-21
@@ -15,26 +15,28 @@ pip install \
15
## Quantizing HuggingFace Models
16
You can quantize your own huggingface model with torchao, e.g. [transformers](https://huggingface.co/docs/transformers/main/en/quantization/torchao) and [diffusers](https://huggingface.co/docs/diffusers/en/quantization/torchao), and save the checkpoint to huggingface hub like [this](https://huggingface.co/jerryzh168/llama3-8b-int8wo) with the following example code:
17
18
-```Python
19
-import torch
20
-from transformers import TorchAoConfig, AutoModelForCausalLM, AutoTokenizer
21
-from torchao.quantization import Int8WeightOnlyConfig
22
-
23
-model_name = "meta-llama/Meta-Llama-3-8B"
24
-quantization_config = TorchAoConfig(Int8WeightOnlyConfig())
25
-quantized_model = AutoModelForCausalLM.from_pretrained(
26
- model_name,
27
- torch_dtype="auto",
28
- device_map="auto",
29
- quantization_config=quantization_config
30
-)
31
-tokenizer = AutoTokenizer.from_pretrained(model_name)
32
-input_text = "What are we having for dinner?"
33
-input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
34
-
35
-hub_repo = # YOUR HUB REPO ID
36
-tokenizer.push_to_hub(hub_repo)
37
-quantized_model.push_to_hub(hub_repo, safe_serialization=False)
38
-```
18
+??? Code
19
+
20
+ ```Python
21
+ import torch
22
+ from transformers import TorchAoConfig, AutoModelForCausalLM, AutoTokenizer
23
+ from torchao.quantization import Int8WeightOnlyConfig
24
+
25
+ model_name = "meta-llama/Meta-Llama-3-8B"
26
+ quantization_config = TorchAoConfig(Int8WeightOnlyConfig())
27
+ quantized_model = AutoModelForCausalLM.from_pretrained(
28
+ model_name,
29
+ torch_dtype="auto",
30
+ device_map="auto",
31
+ quantization_config=quantization_config
32
+ )
33
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
34
+ input_text = "What are we having for dinner?"
35
+ input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
36
+
37
+ hub_repo = # YOUR HUB REPO ID
38
+ tokenizer.push_to_hub(hub_repo)
39
+ quantized_model.push_to_hub(hub_repo, safe_serialization=False)
40
+ ```
41
42
Alternatively, you can use the [TorchAO Quantization space](https://huggingface.co/spaces/medmekk/TorchAO_Quantization) for quantizing models with a simple UI.
docs/features/reasoning_outputs.md
+208
-196
@@ -33,34 +33,36 @@ vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
33
34
Next, make a request to the model that should return the reasoning content in the response.
35
36
-```python
37
-from openai import OpenAI
36
+??? Code
37
39
-# Modify OpenAI's API key and API base to use vLLM's API server.
40
-openai_api_key = "EMPTY"
41
-openai_api_base = "http://localhost:8000/v1"
38
+ ```python
39
+ from openai import OpenAI
40
43
-client = OpenAI(
44
- api_key=openai_api_key,
45
- base_url=openai_api_base,
46
-)
41
+ # Modify OpenAI's API key and API base to use vLLM's API server.
42
+ openai_api_key = "EMPTY"
43
+ openai_api_base = "http://localhost:8000/v1"
44
48
-models = client.models.list()
49
-model = models.data[0].id
45
+ client = OpenAI(
46
+ api_key=openai_api_key,
47
+ base_url=openai_api_base,
48
+ )
49
51
-# Round 1
52
-messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
53
-# For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
54
-# For Qwen3 series, if you want to disable thinking in reasoning mode, add:
55
-# extra_body={"chat_template_kwargs": {"enable_thinking": False}}
56
-response = client.chat.completions.create(model=model, messages=messages)
50
+ models = client.models.list()
51
+ model = models.data[0].id
52
58
-reasoning_content = response.choices[0].message.reasoning_content
59
-content = response.choices[0].message.content
53
+ # Round 1
54
+ messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
55
+ # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
56
+ # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
57
+ # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
58
+ response = client.chat.completions.create(model=model, messages=messages)
59
61
-print("reasoning_content:", reasoning_content)
62
-print("content:", content)
63
-```
60
+ reasoning_content = response.choices[0].message.reasoning_content
61
+ content = response.choices[0].message.content
62
+
63
+ print("reasoning_content:", reasoning_content)
64
+ print("content:", content)
65
+ ```
66
67
The `reasoning_content` field contains the reasoning steps that led to the final conclusion, while the `content` field contains the final conclusion.
68
@@ -68,77 +70,81 @@ The `reasoning_content` field contains the reasoning steps that led to the final
70
71
Streaming chat completions are also supported for reasoning models. The `reasoning_content` field is available in the `delta` field in [chat completion response chunks](https://platform.openai.com/docs/api-reference/chat/streaming).
72
71
-```json
72
-{
73
- "id": "chatcmpl-123",
74
- "object": "chat.completion.chunk",
75
- "created": 1694268190,
76
- "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
77
- "system_fingerprint": "fp_44709d6fcb",
78
- "choices": [
79
- {
80
- "index": 0,
81
- "delta": {
82
- "role": "assistant",
83
- "reasoning_content": "is",
84
- },
85
- "logprobs": null,
86
- "finish_reason": null
87
- }
88
- ]
89
-}
90
-```
73
+??? Json
74
+
75
+ ```json
76
+ {
77
+ "id": "chatcmpl-123",
78
+ "object": "chat.completion.chunk",
79
+ "created": 1694268190,
80
+ "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
81
+ "system_fingerprint": "fp_44709d6fcb",
82
+ "choices": [
83
+ {
84
+ "index": 0,
85
+ "delta": {
86
+ "role": "assistant",
87
+ "reasoning_content": "is",
88
+ },
89
+ "logprobs": null,
90
+ "finish_reason": null
91
+ }
92
+ ]
93
+ }
94
+ ```
95
96
OpenAI Python client library does not officially support `reasoning_content` attribute for streaming output. But the client supports extra attributes in the response. You can use `hasattr` to check if the `reasoning_content` attribute is present in the response. For example:
97
94
-```python
95
-from openai import OpenAI
96
-
97
-# Modify OpenAI's API key and API base to use vLLM's API server.
98
-openai_api_key = "EMPTY"
99
-openai_api_base = "http://localhost:8000/v1"
100
-
101
-client = OpenAI(
102
- api_key=openai_api_key,
103
- base_url=openai_api_base,
104
-)
105
-
106
-models = client.models.list()
107
-model = models.data[0].id
108
-
109
-messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
110
-# For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
111
-# For Qwen3 series, if you want to disable thinking in reasoning mode, add:
112
-# extra_body={"chat_template_kwargs": {"enable_thinking": False}}
113
-stream = client.chat.completions.create(model=model,
114
- messages=messages,
115
- stream=True)
116
-
117
-print("client: Start streaming chat completions...")
118
-printed_reasoning_content = False
119
-printed_content = False
120
-
121
-for chunk in stream:
122
- reasoning_content = None
123
- content = None
124
- # Check the content is reasoning_content or content
125
- if hasattr(chunk.choices[0].delta, "reasoning_content"):
126
- reasoning_content = chunk.choices[0].delta.reasoning_content
127
- elif hasattr(chunk.choices[0].delta, "content"):
128
- content = chunk.choices[0].delta.content
129
-
130
- if reasoning_content is not None:
131
- if not printed_reasoning_content:
132
- printed_reasoning_content = True
133
- print("reasoning_content:", end="", flush=True)
134
- print(reasoning_content, end="", flush=True)
135
- elif content is not None:
136
- if not printed_content:
137
- printed_content = True
138
- print("\ncontent:", end="", flush=True)
139
- # Extract and print the content
140
- print(content, end="", flush=True)
141
-```
98
+??? Code
99
+
100
+ ```python
101
+ from openai import OpenAI
102
+
103
+ # Modify OpenAI's API key and API base to use vLLM's API server.
104
+ openai_api_key = "EMPTY"
105
+ openai_api_base = "http://localhost:8000/v1"
106
+
107
+ client = OpenAI(
108
+ api_key=openai_api_key,
109
+ base_url=openai_api_base,
110
+ )
111
+
112
+ models = client.models.list()
113
+ model = models.data[0].id
114
+
115
+ messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
116
+ # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
117
+ # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
118
+ # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
119
+ stream = client.chat.completions.create(model=model,
120
+ messages=messages,
121
+ stream=True)
122
+
123
+ print("client: Start streaming chat completions...")
124
+ printed_reasoning_content = False
125
+ printed_content = False
126
+
127
+ for chunk in stream:
128
+ reasoning_content = None
129
+ content = None
130
+ # Check the content is reasoning_content or content
131
+ if hasattr(chunk.choices[0].delta, "reasoning_content"):
132
+ reasoning_content = chunk.choices[0].delta.reasoning_content
133
+ elif hasattr(chunk.choices[0].delta, "content"):
134
+ content = chunk.choices[0].delta.content
135
+
136
+ if reasoning_content is not None:
137
+ if not printed_reasoning_content:
138
+ printed_reasoning_content = True
139
+ print("reasoning_content:", end="", flush=True)
140
+ print(reasoning_content, end="", flush=True)
141
+ elif content is not None:
142
+ if not printed_content:
143
+ printed_content = True
144
+ print("\ncontent:", end="", flush=True)
145
+ # Extract and print the content
146
+ print(content, end="", flush=True)
147
+ ```
148
149
Remember to check whether the `reasoning_content` exists in the response before accessing it. You could checkout the [example](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py).
150
@@ -146,41 +152,43 @@ Remember to check whether the `reasoning_content` exists in the response before
152
153
The reasoning content is also available when both tool calling and the reasoning parser are enabled. Additionally, tool calling only parses functions from the `content` field, not from the `reasoning_content`.
154
149
-```python
150
-from openai import OpenAI
151
-
152
-client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
153
-
154
-tools = [{
155
- "type": "function",
156
- "function": {
157
- "name": "get_weather",
158
- "description": "Get the current weather in a given location",
159
- "parameters": {
160
- "type": "object",
161
- "properties": {
162
- "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
163
- "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
164
- },
165
- "required": ["location", "unit"]
155
+??? Code
156
+
157
+ ```python
158
+ from openai import OpenAI
159
+
160
+ client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
161
+
162
+ tools = [{
163
+ "type": "function",
164
+ "function": {
165
+ "name": "get_weather",
166
+ "description": "Get the current weather in a given location",
167
+ "parameters": {
168
+ "type": "object",
169
+ "properties": {
170
+ "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
171
+ "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
172
+ },
173
+ "required": ["location", "unit"]
174
+ }
175
}
167
- }
168
-}]
176
+ }]
177
170
-response = client.chat.completions.create(
171
- model=client.models.list().data[0].id,
172
- messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
173
- tools=tools,
174
- tool_choice="auto"
175
-)
178
+ response = client.chat.completions.create(
179
+ model=client.models.list().data[0].id,
180
+ messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
181
+ tools=tools,
182
+ tool_choice="auto"
183
+ )
184
177
-print(response)
178
-tool_call = response.choices[0].message.tool_calls[0].function
185
+ print(response)
186
+ tool_call = response.choices[0].message.tool_calls[0].function
187
180
-print(f"reasoning_content: {response.choices[0].message.reasoning_content}")
181
-print(f"Function called: {tool_call.name}")
182
-print(f"Arguments: {tool_call.arguments}")
183
-```
188
+ print(f"reasoning_content: {response.choices[0].message.reasoning_content}")
189
+ print(f"Function called: {tool_call.name}")
190
+ print(f"Arguments: {tool_call.arguments}")
191
+ ```
192
193
For more examples, please refer to <gh-file:examples/online_serving/openai_chat_completion_tool_calls_with_reasoning.py>.
194
@@ -192,85 +200,89 @@ For more examples, please refer to <gh-file:examples/online_serving/openai_chat_
200
201
You can add a new `ReasoningParser` similar to <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
202
195
-```python
196
-# import the required packages
197
-
198
-from vllm.reasoning import ReasoningParser, ReasoningParserManager
199
-from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
200
- DeltaMessage)
201
-
202
-# define a reasoning parser and register it to vllm
203
-# the name list in register_module can be used
204
-# in --reasoning-parser.
205
-@ReasoningParserManager.register_module(["example"])
206
-class ExampleParser(ReasoningParser):
207
- def __init__(self, tokenizer: AnyTokenizer):
208
- super().__init__(tokenizer)
209
-
210
- def extract_reasoning_content_streaming(
211
- self,
212
- previous_text: str,
213
- current_text: str,
214
- delta_text: str,
215
- previous_token_ids: Sequence[int],
216
- current_token_ids: Sequence[int],
217
- delta_token_ids: Sequence[int],
218
- ) -> Union[DeltaMessage, None]:
219
- """
220
- Instance method that should be implemented for extracting reasoning
221
- from an incomplete response; for use when handling reasoning calls and
222
- streaming. Has to be an instance method because it requires state -
223
- the current tokens/diffs, but also the information about what has
224
- previously been parsed and extracted (see constructor)
225
- """
226
-
227
- def extract_reasoning_content(
228
- self, model_output: str, request: ChatCompletionRequest
229
- ) -> tuple[Optional[str], Optional[str]]:
230
- """
231
- Extract reasoning content from a complete model-generated string.
232
-
233
- Used for non-streaming responses where we have the entire model response
234
- available before sending to the client.
203
+??? Code
204
+
205
+ ```python
206
+ # import the required packages
207
+
208
+ from vllm.reasoning import ReasoningParser, ReasoningParserManager
209
+ from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
210
+ DeltaMessage)
211
+
212
+ # define a reasoning parser and register it to vllm
213
+ # the name list in register_module can be used
214
+ # in --reasoning-parser.
215
+ @ReasoningParserManager.register_module(["example"])
216
+ class ExampleParser(ReasoningParser):
217
+ def __init__(self, tokenizer: AnyTokenizer):
218
+ super().__init__(tokenizer)
219
+
220
+ def extract_reasoning_content_streaming(
221
+ self,
222
+ previous_text: str,
223
+ current_text: str,
224
+ delta_text: str,
225
+ previous_token_ids: Sequence[int],
226
+ current_token_ids: Sequence[int],
227
+ delta_token_ids: Sequence[int],
228
+ ) -> Union[DeltaMessage, None]:
229
+ """
230
+ Instance method that should be implemented for extracting reasoning
231
+ from an incomplete response; for use when handling reasoning calls and
232
+ streaming. Has to be an instance method because it requires state -
233
+ the current tokens/diffs, but also the information about what has
234
+ previously been parsed and extracted (see constructor)
235
+ """
236
+
237
+ def extract_reasoning_content(
238
+ self, model_output: str, request: ChatCompletionRequest
239
+ ) -> tuple[Optional[str], Optional[str]]:
240
+ """
241
+ Extract reasoning content from a complete model-generated string.
242
+
243
+ Used for non-streaming responses where we have the entire model response
244
+ available before sending to the client.
245
+
246
+ Parameters:
247
+ model_output: str
248
+ The model-generated string to extract reasoning content from.
249
+
250
+ request: ChatCompletionRequest
251
+ The request object that was used to generate the model_output.
252
+
253
+ Returns:
254
+ tuple[Optional[str], Optional[str]]
255
+ A tuple containing the reasoning content and the content.
256
+ """
257
+ ```
258
236
- Parameters:
237
- model_output: str
238
- The model-generated string to extract reasoning content from.
259
+Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
260
240
- request: ChatCompletionRequest
241
- The request object that was used to generate the model_output.
261
+??? Code
262
243
- Returns:
244
- tuple[Optional[str], Optional[str]]
245
- A tuple containing the reasoning content and the content.
263
+ ```python
264
+ @dataclass
265
+ class DeepSeekReasoner(Reasoner):
266
"""
247
-```
248
-
249
-Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
250
-
251
-```python
252
-@dataclass
253
-class DeepSeekReasoner(Reasoner):
254
- """
255
- Reasoner for DeepSeek R series models.
256
- """
257
- start_token_id: int
258
- end_token_id: int
259
-
260
- start_token: str = "<think>"
261
- end_token: str = "</think>"
262
-
263
- @classmethod
264
- def from_tokenizer(cls, tokenizer: PreTrainedTokenizer) -> Reasoner:
265
- return cls(start_token_id=tokenizer.encode(
266
- "<think>", add_special_tokens=False)[0],
267
- end_token_id=tokenizer.encode("</think>",
268
- add_special_tokens=False)[0])
269
-
270
- def is_reasoning_end(self, input_ids: list[int]) -> bool:
271
- return self.end_token_id in input_ids
272
- ...
273
-```
267
+ Reasoner for DeepSeek R series models.
268
+ """
269
+ start_token_id: int
270
+ end_token_id: int
271
+
272
+ start_token: str = "<think>"
273
+ end_token: str = "</think>"
274
+
275
+ @classmethod
276
+ def from_tokenizer(cls, tokenizer: PreTrainedTokenizer) -> Reasoner:
277
+ return cls(start_token_id=tokenizer.encode(
278
+ "<think>", add_special_tokens=False)[0],
279
+ end_token_id=tokenizer.encode("</think>",
280
+ add_special_tokens=False)[0])
281
+
282
+ def is_reasoning_end(self, input_ids: list[int]) -> bool:
283
+ return self.end_token_id in input_ids
284
+ ...
285
+ ```
286
287
The structured output engine like [xgrammar](https://github.com/mlc-ai/xgrammar) will use `end_token_id` to check if the reasoning content is present in the model output and skip the structured output if it is the case.
288
docs/features/spec_decode.md
+133
-123
@@ -18,29 +18,31 @@ Speculative decoding is a technique which improves inter-token latency in memory
18
19
The following code configures vLLM in an offline mode to use speculative decoding with a draft model, speculating 5 tokens at a time.
20
21
-```python
22
-from vllm import LLM, SamplingParams
23
-
24
-prompts = [
25
- "The future of AI is",
26
-]
27
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
28
-
29
-llm = LLM(
30
- model="facebook/opt-6.7b",
31
- tensor_parallel_size=1,
32
- speculative_config={
33
- "model": "facebook/opt-125m",
34
- "num_speculative_tokens": 5,
35
- },
36
-)
37
-outputs = llm.generate(prompts, sampling_params)
38
-
39
-for output in outputs:
40
- prompt = output.prompt
41
- generated_text = output.outputs[0].text
42
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
43
-```
21
+??? Code
22
+
23
+ ```python
24
+ from vllm import LLM, SamplingParams
25
+
26
+ prompts = [
27
+ "The future of AI is",
28
+ ]
29
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
30
+
31
+ llm = LLM(
32
+ model="facebook/opt-6.7b",
33
+ tensor_parallel_size=1,
34
+ speculative_config={
35
+ "model": "facebook/opt-125m",
36
+ "num_speculative_tokens": 5,
37
+ },
38
+ )
39
+ outputs = llm.generate(prompts, sampling_params)
40
+
41
+ for output in outputs:
42
+ prompt = output.prompt
43
+ generated_text = output.outputs[0].text
44
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
45
+ ```
46
47
To perform the same with an online mode launch the server:
48
@@ -60,69 +62,73 @@ python -m vllm.entrypoints.openai.api_server \
62
63
Then use a client:
64
63
-```python
64
-from openai import OpenAI
65
-
66
-# Modify OpenAI's API key and API base to use vLLM's API server.
67
-openai_api_key = "EMPTY"
68
-openai_api_base = "http://localhost:8000/v1"
69
-
70
-client = OpenAI(
71
- # defaults to os.environ.get("OPENAI_API_KEY")
72
- api_key=openai_api_key,
73
- base_url=openai_api_base,
74
-)
75
-
76
-models = client.models.list()
77
-model = models.data[0].id
78
-
79
-# Completion API
80
-stream = False
81
-completion = client.completions.create(
82
- model=model,
83
- prompt="The future of AI is",
84
- echo=False,
85
- n=1,
86
- stream=stream,
87
-)
88
-
89
-print("Completion results:")
90
-if stream:
91
- for c in completion:
92
- print(c)
93
-else:
94
- print(completion)
95
-```
65
+??? Code
66
+
67
+ ```python
68
+ from openai import OpenAI
69
+
70
+ # Modify OpenAI's API key and API base to use vLLM's API server.
71
+ openai_api_key = "EMPTY"
72
+ openai_api_base = "http://localhost:8000/v1"
73
+
74
+ client = OpenAI(
75
+ # defaults to os.environ.get("OPENAI_API_KEY")
76
+ api_key=openai_api_key,
77
+ base_url=openai_api_base,
78
+ )
79
+
80
+ models = client.models.list()
81
+ model = models.data[0].id
82
+
83
+ # Completion API
84
+ stream = False
85
+ completion = client.completions.create(
86
+ model=model,
87
+ prompt="The future of AI is",
88
+ echo=False,
89
+ n=1,
90
+ stream=stream,
91
+ )
92
+
93
+ print("Completion results:")
94
+ if stream:
95
+ for c in completion:
96
+ print(c)
97
+ else:
98
+ print(completion)
99
+ ```
100
101
## Speculating by matching n-grams in the prompt
102
103
The following code configures vLLM to use speculative decoding where proposals are generated by
104
matching n-grams in the prompt. For more information read [this thread.](https://x.com/joao_gante/status/1747322413006643259)
105
102
-```python
103
-from vllm import LLM, SamplingParams
104
-
105
-prompts = [
106
- "The future of AI is",
107
-]
108
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
109
-
110
-llm = LLM(
111
- model="facebook/opt-6.7b",
112
- tensor_parallel_size=1,
113
- speculative_config={
114
- "method": "ngram",
115
- "num_speculative_tokens": 5,
116
- "prompt_lookup_max": 4,
117
- },
118
-)
119
-outputs = llm.generate(prompts, sampling_params)
120
-
121
-for output in outputs:
122
- prompt = output.prompt
123
- generated_text = output.outputs[0].text
124
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
125
-```
106
+??? Code
107
+
108
+ ```python
109
+ from vllm import LLM, SamplingParams
110
+
111
+ prompts = [
112
+ "The future of AI is",
113
+ ]
114
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
115
+
116
+ llm = LLM(
117
+ model="facebook/opt-6.7b",
118
+ tensor_parallel_size=1,
119
+ speculative_config={
120
+ "method": "ngram",
121
+ "num_speculative_tokens": 5,
122
+ "prompt_lookup_max": 4,
123
+ },
124
+ )
125
+ outputs = llm.generate(prompts, sampling_params)
126
+
127
+ for output in outputs:
128
+ prompt = output.prompt
129
+ generated_text = output.outputs[0].text
130
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
131
+ ```
132
133
## Speculating using MLP speculators
134
@@ -131,29 +137,31 @@ draft models that conditioning draft predictions on both context vectors and sam
137
For more information see [this blog](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/) or
138
[this technical report](https://arxiv.org/abs/2404.19124).
139
134
-```python
135
-from vllm import LLM, SamplingParams
136
-
137
-prompts = [
138
- "The future of AI is",
139
-]
140
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
141
-
142
-llm = LLM(
143
- model="meta-llama/Meta-Llama-3.1-70B-Instruct",
144
- tensor_parallel_size=4,
145
- speculative_config={
146
- "model": "ibm-ai-platform/llama3-70b-accelerator",
147
- "draft_tensor_parallel_size": 1,
148
- },
149
-)
150
-outputs = llm.generate(prompts, sampling_params)
151
-
152
-for output in outputs:
153
- prompt = output.prompt
154
- generated_text = output.outputs[0].text
155
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
156
-```
140
+??? Code
141
+
142
+ ```python
143
+ from vllm import LLM, SamplingParams
144
+
145
+ prompts = [
146
+ "The future of AI is",
147
+ ]
148
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
149
+
150
+ llm = LLM(
151
+ model="meta-llama/Meta-Llama-3.1-70B-Instruct",
152
+ tensor_parallel_size=4,
153
+ speculative_config={
154
+ "model": "ibm-ai-platform/llama3-70b-accelerator",
155
+ "draft_tensor_parallel_size": 1,
156
+ },
157
+ )
158
+ outputs = llm.generate(prompts, sampling_params)
159
+
160
+ for output in outputs:
161
+ prompt = output.prompt
162
+ generated_text = output.outputs[0].text
163
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
164
+ ```
165
166
Note that these speculative models currently need to be run without tensor parallelism, although
167
it is possible to run the main model using tensor parallelism (see example above). Since the
@@ -177,31 +185,33 @@ A variety of speculative models of this type are available on HF hub:
185
The following code configures vLLM to use speculative decoding where proposals are generated by
186
an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found [here](gh-file:examples/offline_inference/eagle.py).
187
180
-```python
181
-from vllm import LLM, SamplingParams
188
+??? Code
189
183
-prompts = [
184
- "The future of AI is",
185
-]
186
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
190
+ ```python
191
+ from vllm import LLM, SamplingParams
192
188
-llm = LLM(
189
- model="meta-llama/Meta-Llama-3-8B-Instruct",
190
- tensor_parallel_size=4,
191
- speculative_config={
192
- "model": "yuhuili/EAGLE-LLaMA3-Instruct-8B",
193
- "draft_tensor_parallel_size": 1,
194
- },
195
-)
193
+ prompts = [
194
+ "The future of AI is",
195
+ ]
196
+ sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
197
197
-outputs = llm.generate(prompts, sampling_params)
198
+ llm = LLM(
199
+ model="meta-llama/Meta-Llama-3-8B-Instruct",
200
+ tensor_parallel_size=4,
201
+ speculative_config={
202
+ "model": "yuhuili/EAGLE-LLaMA3-Instruct-8B",
203
+ "draft_tensor_parallel_size": 1,
204
+ },
205
+ )
206
199
-for output in outputs:
200
- prompt = output.prompt
201
- generated_text = output.outputs[0].text
202
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
207
+ outputs = llm.generate(prompts, sampling_params)
208
204
-```
209
+ for output in outputs:
210
+ prompt = output.prompt
211
+ generated_text = output.outputs[0].text
212
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
213
+
214
+ ```
215
216
A few important things to consider when using the EAGLE based draft models:
217
docs/features/structured_outputs.md
+195
-181
@@ -33,39 +33,43 @@ text.
33
34
Now let´s see an example for each of the cases, starting with the `guided_choice`, as it´s the easiest one:
35
36
-```python
37
-from openai import OpenAI
38
-client = OpenAI(
39
- base_url="http://localhost:8000/v1",
40
- api_key="-",
41
-)
42
-model = client.models.list().data[0].id
43
-
44
-completion = client.chat.completions.create(
45
- model=model,
46
- messages=[
47
- {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
48
- ],
49
- extra_body={"guided_choice": ["positive", "negative"]},
50
-)
51
-print(completion.choices[0].message.content)
52
-```
36
+??? Code
37
+
38
+ ```python
39
+ from openai import OpenAI
40
+ client = OpenAI(
41
+ base_url="http://localhost:8000/v1",
42
+ api_key="-",
43
+ )
44
+ model = client.models.list().data[0].id
45
+
46
+ completion = client.chat.completions.create(
47
+ model=model,
48
+ messages=[
49
+ {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
50
+ ],
51
+ extra_body={"guided_choice": ["positive", "negative"]},
52
+ )
53
+ print(completion.choices[0].message.content)
54
+ ```
55
56
The next example shows how to use the `guided_regex`. The idea is to generate an email address, given a simple regex template:
57
56
-```python
57
-completion = client.chat.completions.create(
58
- model=model,
59
- messages=[
60
- {
61
- "role": "user",
62
- "content": "Generate an example email address for Alan Turing, who works in Enigma. End in .com and new line. Example result: alan.turing@enigma.com\n",
63
- }
64
- ],
65
- extra_body={"guided_regex": r"\w+@\w+\.com\n", "stop": ["\n"]},
66
-)
67
-print(completion.choices[0].message.content)
68
-```
58
+??? Code
59
+
60
+ ```python
61
+ completion = client.chat.completions.create(
62
+ model=model,
63
+ messages=[
64
+ {
65
+ "role": "user",
66
+ "content": "Generate an example email address for Alan Turing, who works in Enigma. End in .com and new line. Example result: alan.turing@enigma.com\n",
67
+ }
68
+ ],
69
+ extra_body={"guided_regex": r"\w+@\w+\.com\n", "stop": ["\n"]},
70
+ )
71
+ print(completion.choices[0].message.content)
72
+ ```
73
74
One of the most relevant features in structured text generation is the option to generate a valid JSON with pre-defined fields and formats.
75
For this we can use the `guided_json` parameter in two different ways:
@@ -75,41 +79,43 @@ For this we can use the `guided_json` parameter in two different ways:
79
80
The next example shows how to use the `guided_json` parameter with a Pydantic model:
81
78
-```python
79
-from pydantic import BaseModel
80
-from enum import Enum
81
-
82
-class CarType(str, Enum):
83
- sedan = "sedan"
84
- suv = "SUV"
85
- truck = "Truck"
86
- coupe = "Coupe"
87
-
88
-class CarDescription(BaseModel):
89
- brand: str
90
- model: str
91
- car_type: CarType
92
-
93
-json_schema = CarDescription.model_json_schema()
94
-
95
-completion = client.chat.completions.create(
96
- model=model,
97
- messages=[
98
- {
99
- "role": "user",
100
- "content": "Generate a JSON with the brand, model and car_type of the most iconic car from the 90's",
101
- }
102
- ],
103
- "response_format": {
104
- "type": "json_schema",
105
- "json_schema": {
106
- "name": "car-description",
107
- "schema": CarDescription.model_json_schema()
82
+??? Code
83
+
84
+ ```python
85
+ from pydantic import BaseModel
86
+ from enum import Enum
87
+
88
+ class CarType(str, Enum):
89
+ sedan = "sedan"
90
+ suv = "SUV"
91
+ truck = "Truck"
92
+ coupe = "Coupe"
93
+
94
+ class CarDescription(BaseModel):
95
+ brand: str
96
+ model: str
97
+ car_type: CarType
98
+
99
+ json_schema = CarDescription.model_json_schema()
100
+
101
+ completion = client.chat.completions.create(
102
+ model=model,
103
+ messages=[
104
+ {
105
+ "role": "user",
106
+ "content": "Generate a JSON with the brand, model and car_type of the most iconic car from the 90's",
107
+ }
108
+ ],
109
+ "response_format": {
110
+ "type": "json_schema",
111
+ "json_schema": {
112
+ "name": "car-description",
113
+ "schema": CarDescription.model_json_schema()
114
+ },
115
},
109
- },
110
-)
111
-print(completion.choices[0].message.content)
112
-```
116
+ )
117
+ print(completion.choices[0].message.content)
118
+ ```
119
120
!!! tip
121
While not strictly necessary, normally it´s better to indicate in the prompt the
@@ -121,33 +127,35 @@ difficult to use, but it´s really powerful. It allows us to define complete
127
languages like SQL queries. It works by using a context free EBNF grammar.
128
As an example, we can use to define a specific format of simplified SQL queries:
129
124
-```python
125
-simplified_sql_grammar = """
126
- root ::= select_statement
130
+??? Code
131
128
- select_statement ::= "SELECT " column " from " table " where " condition
132
+ ```python
133
+ simplified_sql_grammar = """
134
+ root ::= select_statement
135
130
- column ::= "col_1 " | "col_2 "
136
+ select_statement ::= "SELECT " column " from " table " where " condition
137
132
- table ::= "table_1 " | "table_2 "
138
+ column ::= "col_1 " | "col_2 "
139
134
- condition ::= column "= " number
140
+ table ::= "table_1 " | "table_2 "
141
136
- number ::= "1 " | "2 "
137
-"""
142
+ condition ::= column "= " number
143
139
-completion = client.chat.completions.create(
140
- model=model,
141
- messages=[
142
- {
143
- "role": "user",
144
- "content": "Generate an SQL query to show the 'username' and 'email' from the 'users' table.",
145
- }
146
- ],
147
- extra_body={"guided_grammar": simplified_sql_grammar},
148
-)
149
-print(completion.choices[0].message.content)
150
-```
144
+ number ::= "1 " | "2 "
145
+ """
146
+
147
+ completion = client.chat.completions.create(
148
+ model=model,
149
+ messages=[
150
+ {
151
+ "role": "user",
152
+ "content": "Generate an SQL query to show the 'username' and 'email' from the 'users' table.",
153
+ }
154
+ ],
155
+ extra_body={"guided_grammar": simplified_sql_grammar},
156
+ )
157
+ print(completion.choices[0].message.content)
158
+ ```
159
160
See also: [full example](https://docs.vllm.ai/en/latest/examples/online_serving/structured_outputs.html)
161
@@ -161,34 +169,36 @@ vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --reasoning-parser deepseek_r
169
170
Note that you can use reasoning with any provided structured outputs feature. The following uses one with JSON schema:
171
164
-```python
165
-from pydantic import BaseModel
166
-
167
-
168
-class People(BaseModel):
169
- name: str
170
- age: int
171
-
172
-
173
-completion = client.chat.completions.create(
174
- model=model,
175
- messages=[
176
- {
177
- "role": "user",
178
- "content": "Generate a JSON with the name and age of one random person.",
179
- }
180
- ],
181
- response_format={
182
- "type": "json_schema",
183
- "json_schema": {
184
- "name": "people",
185
- "schema": People.model_json_schema()
186
- }
187
- },
188
-)
189
-print("reasoning_content: ", completion.choices[0].message.reasoning_content)
190
-print("content: ", completion.choices[0].message.content)
191
-```
172
+??? Code
173
+
174
+ ```python
175
+ from pydantic import BaseModel
176
+
177
+
178
+ class People(BaseModel):
179
+ name: str
180
+ age: int
181
+
182
+
183
+ completion = client.chat.completions.create(
184
+ model=model,
185
+ messages=[
186
+ {
187
+ "role": "user",
188
+ "content": "Generate a JSON with the name and age of one random person.",
189
+ }
190
+ ],
191
+ response_format={
192
+ "type": "json_schema",
193
+ "json_schema": {
194
+ "name": "people",
195
+ "schema": People.model_json_schema()
196
+ }
197
+ },
198
+ )
199
+ print("reasoning_content: ", completion.choices[0].message.reasoning_content)
200
+ print("content: ", completion.choices[0].message.content)
201
+ ```
202
203
See also: [full example](https://docs.vllm.ai/en/latest/examples/online_serving/structured_outputs.html)
204
@@ -202,33 +212,33 @@ For the following examples, vLLM was setup using `vllm serve meta-llama/Llama-3.
212
213
Here is a simple example demonstrating how to get structured output using Pydantic models:
214
205
-```python
206
-from pydantic import BaseModel
207
-from openai import OpenAI
208
-
209
-class Info(BaseModel):
210
- name: str
211
- age: int
212
-
213
-client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
214
-model = client.models.list().data[0].id
215
-completion = client.beta.chat.completions.parse(
216
- model=model,
217
- messages=[
218
- {"role": "system", "content": "You are a helpful assistant."},
219
- {"role": "user", "content": "My name is Cameron, I'm 28. What's my name and age?"},
220
- ],
221
- response_format=Info,
222
-)
223
-
224
-message = completion.choices[0].message
225
-print(message)
226
-assert message.parsed
227
-print("Name:", message.parsed.name)
228
-print("Age:", message.parsed.age)
229
-```
230
-
231
-Output:
215
+??? Code
216
+
217
+ ```python
218
+ from pydantic import BaseModel
219
+ from openai import OpenAI
220
+
221
+ class Info(BaseModel):
222
+ name: str
223
+ age: int
224
+
225
+ client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
226
+ model = client.models.list().data[0].id
227
+ completion = client.beta.chat.completions.parse(
228
+ model=model,
229
+ messages=[
230
+ {"role": "system", "content": "You are a helpful assistant."},
231
+ {"role": "user", "content": "My name is Cameron, I'm 28. What's my name and age?"},
232
+ ],
233
+ response_format=Info,
234
+ )
235
+
236
+ message = completion.choices[0].message
237
+ print(message)
238
+ assert message.parsed
239
+ print("Name:", message.parsed.name)
240
+ print("Age:", message.parsed.age)
241
+ ```
242
243
```console
244
ParsedChatCompletionMessage[Testing](content='{"name": "Cameron", "age": 28}', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], parsed=Testing(name='Cameron', age=28))
@@ -238,35 +248,37 @@ Age: 28
248
249
Here is a more complex example using nested Pydantic models to handle a step-by-step math solution:
250
241
-```python
242
-from typing import List
243
-from pydantic import BaseModel
244
-from openai import OpenAI
245
-
246
-class Step(BaseModel):
247
- explanation: str
248
- output: str
249
-
250
-class MathResponse(BaseModel):
251
- steps: list[Step]
252
- final_answer: str
253
-
254
-completion = client.beta.chat.completions.parse(
255
- model=model,
256
- messages=[
257
- {"role": "system", "content": "You are a helpful expert math tutor."},
258
- {"role": "user", "content": "Solve 8x + 31 = 2."},
259
- ],
260
- response_format=MathResponse,
261
-)
262
-
263
-message = completion.choices[0].message
264
-print(message)
265
-assert message.parsed
266
-for i, step in enumerate(message.parsed.steps):
267
- print(f"Step #{i}:", step)
268
-print("Answer:", message.parsed.final_answer)
269
-```
251
+??? Code
252
+
253
+ ```python
254
+ from typing import List
255
+ from pydantic import BaseModel
256
+ from openai import OpenAI
257
+
258
+ class Step(BaseModel):
259
+ explanation: str
260
+ output: str
261
+
262
+ class MathResponse(BaseModel):
263
+ steps: list[Step]
264
+ final_answer: str
265
+
266
+ completion = client.beta.chat.completions.parse(
267
+ model=model,
268
+ messages=[
269
+ {"role": "system", "content": "You are a helpful expert math tutor."},
270
+ {"role": "user", "content": "Solve 8x + 31 = 2."},
271
+ ],
272
+ response_format=MathResponse,
273
+ )
274
+
275
+ message = completion.choices[0].message
276
+ print(message)
277
+ assert message.parsed
278
+ for i, step in enumerate(message.parsed.steps):
279
+ print(f"Step #{i}:", step)
280
+ print("Answer:", message.parsed.final_answer)
281
+ ```
282
283
Output:
284
@@ -296,19 +308,21 @@ These parameters can be used in the same way as the parameters from the Online
308
Serving examples above. One example for the usage of the `choice` parameter is
309
shown below:
310
299
-```python
300
-from vllm import LLM, SamplingParams
301
-from vllm.sampling_params import GuidedDecodingParams
311
+??? Code
312
303
-llm = LLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
313
+ ```python
314
+ from vllm import LLM, SamplingParams
315
+ from vllm.sampling_params import GuidedDecodingParams
316
305
-guided_decoding_params = GuidedDecodingParams(choice=["Positive", "Negative"])
306
-sampling_params = SamplingParams(guided_decoding=guided_decoding_params)
307
-outputs = llm.generate(
308
- prompts="Classify this sentiment: vLLM is wonderful!",
309
- sampling_params=sampling_params,
310
-)
311
-print(outputs[0].outputs[0].text)
312
-```
317
+ llm = LLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
318
+
319
+ guided_decoding_params = GuidedDecodingParams(choice=["Positive", "Negative"])
320
+ sampling_params = SamplingParams(guided_decoding=guided_decoding_params)
321
+ outputs = llm.generate(
322
+ prompts="Classify this sentiment: vLLM is wonderful!",
323
+ sampling_params=sampling_params,
324
+ )
325
+ print(outputs[0].outputs[0].text)
326
+ ```
327
328
See also: [full example](https://docs.vllm.ai/en/latest/examples/online_serving/structured_outputs.html)
docs/features/tool_calling.md
+84
-80
@@ -15,44 +15,46 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \
15
16
Next, make a request to the model that should result in it using the available tools:
17
18
-```python
19
-from openai import OpenAI
20
-import json
21
-
22
-client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
23
-
24
-def get_weather(location: str, unit: str):
25
- return f"Getting the weather for {location} in {unit}..."
26
-tool_functions = {"get_weather": get_weather}
27
-
28
-tools = [{
29
- "type": "function",
30
- "function": {
31
- "name": "get_weather",
32
- "description": "Get the current weather in a given location",
33
- "parameters": {
34
- "type": "object",
35
- "properties": {
36
- "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
37
- "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
38
- },
39
- "required": ["location", "unit"]
18
+??? Code
19
+
20
+ ```python
21
+ from openai import OpenAI
22
+ import json
23
+
24
+ client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
25
+
26
+ def get_weather(location: str, unit: str):
27
+ return f"Getting the weather for {location} in {unit}..."
28
+ tool_functions = {"get_weather": get_weather}
29
+
30
+ tools = [{
31
+ "type": "function",
32
+ "function": {
33
+ "name": "get_weather",
34
+ "description": "Get the current weather in a given location",
35
+ "parameters": {
36
+ "type": "object",
37
+ "properties": {
38
+ "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
39
+ "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
40
+ },
41
+ "required": ["location", "unit"]
42
+ }
43
}
41
- }
42
-}]
43
-
44
-response = client.chat.completions.create(
45
- model=client.models.list().data[0].id,
46
- messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
47
- tools=tools,
48
- tool_choice="auto"
49
-)
50
-
51
-tool_call = response.choices[0].message.tool_calls[0].function
52
-print(f"Function called: {tool_call.name}")
53
-print(f"Arguments: {tool_call.arguments}")
54
-print(f"Result: {get_weather(**json.loads(tool_call.arguments))}")
55
-```
44
+ }]
45
+
46
+ response = client.chat.completions.create(
47
+ model=client.models.list().data[0].id,
48
+ messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
49
+ tools=tools,
50
+ tool_choice="auto"
51
+ )
52
+
53
+ tool_call = response.choices[0].message.tool_calls[0].function
54
+ print(f"Function called: {tool_call.name}")
55
+ print(f"Arguments: {tool_call.arguments}")
56
+ print(f"Result: {get_weather(**json.loads(tool_call.arguments))}")
57
+ ```
58
59
Example output:
60
@@ -301,49 +303,51 @@ A tool parser plugin is a Python file containing one or more ToolParser implemen
303
304
Here is a summary of a plugin file:
305
304
-```python
305
-
306
-# import the required packages
307
-
308
-# define a tool parser and register it to vllm
309
-# the name list in register_module can be used
310
-# in --tool-call-parser. you can define as many
311
-# tool parsers as you want here.
312
-@ToolParserManager.register_module(["example"])
313
-class ExampleToolParser(ToolParser):
314
- def __init__(self, tokenizer: AnyTokenizer):
315
- super().__init__(tokenizer)
316
-
317
- # adjust request. e.g.: set skip special tokens
318
- # to False for tool call output.
319
- def adjust_request(
320
- self, request: ChatCompletionRequest) -> ChatCompletionRequest:
321
- return request
322
-
323
- # implement the tool call parse for stream call
324
- def extract_tool_calls_streaming(
325
- self,
326
- previous_text: str,
327
- current_text: str,
328
- delta_text: str,
329
- previous_token_ids: Sequence[int],
330
- current_token_ids: Sequence[int],
331
- delta_token_ids: Sequence[int],
332
- request: ChatCompletionRequest,
333
- ) -> Union[DeltaMessage, None]:
334
- return delta
335
-
336
- # implement the tool parse for non-stream call
337
- def extract_tool_calls(
338
- self,
339
- model_output: str,
340
- request: ChatCompletionRequest,
341
- ) -> ExtractedToolCallInformation:
342
- return ExtractedToolCallInformation(tools_called=False,
343
- tool_calls=[],
344
- content=text)
345
-
346
-```
306
+??? Code
307
+
308
+ ```python
309
+
310
+ # import the required packages
311
+
312
+ # define a tool parser and register it to vllm
313
+ # the name list in register_module can be used
314
+ # in --tool-call-parser. you can define as many
315
+ # tool parsers as you want here.
316
+ @ToolParserManager.register_module(["example"])
317
+ class ExampleToolParser(ToolParser):
318
+ def __init__(self, tokenizer: AnyTokenizer):
319
+ super().__init__(tokenizer)
320
+
321
+ # adjust request. e.g.: set skip special tokens
322
+ # to False for tool call output.
323
+ def adjust_request(
324
+ self, request: ChatCompletionRequest) -> ChatCompletionRequest:
325
+ return request
326
+
327
+ # implement the tool call parse for stream call
328
+ def extract_tool_calls_streaming(
329
+ self,
330
+ previous_text: str,
331
+ current_text: str,
332
+ delta_text: str,
333
+ previous_token_ids: Sequence[int],
334
+ current_token_ids: Sequence[int],
335
+ delta_token_ids: Sequence[int],
336
+ request: ChatCompletionRequest,
337
+ ) -> Union[DeltaMessage, None]:
338
+ return delta
339
+
340
+ # implement the tool parse for non-stream call
341
+ def extract_tool_calls(
342
+ self,
343
+ model_output: str,
344
+ request: ChatCompletionRequest,
345
+ ) -> ExtractedToolCallInformation:
346
+ return ExtractedToolCallInformation(tools_called=False,
347
+ tool_calls=[],
348
+ content=text)
349
+
350
+ ```
351
352
Then you can use this plugin in the command line like this.
353
docs/getting_started/installation/cpu.md
+45
-41
@@ -76,21 +76,23 @@ Currently, there are no pre-built CPU wheels.
76
77
### Build image from source
78
79
-```console
80
-$ docker build -f docker/Dockerfile.cpu --tag vllm-cpu-env --target vllm-openai .
81
-
82
-# Launching OpenAI server
83
-$ docker run --rm \
84
- --privileged=true \
85
- --shm-size=4g \
86
- -p 8000:8000 \
87
- -e VLLM_CPU_KVCACHE_SPACE=<KV cache space> \
88
- -e VLLM_CPU_OMP_THREADS_BIND=<CPU cores for inference> \
89
- vllm-cpu-env \
90
- --model=meta-llama/Llama-3.2-1B-Instruct \
91
- --dtype=bfloat16 \
92
- other vLLM OpenAI server arguments
93
-```
79
+??? Commands
80
+
81
+ ```console
82
+ $ docker build -f docker/Dockerfile.cpu --tag vllm-cpu-env --target vllm-openai .
83
+
84
+ # Launching OpenAI server
85
+ $ docker run --rm \
86
+ --privileged=true \
87
+ --shm-size=4g \
88
+ -p 8000:8000 \
89
+ -e VLLM_CPU_KVCACHE_SPACE=<KV cache space> \
90
+ -e VLLM_CPU_OMP_THREADS_BIND=<CPU cores for inference> \
91
+ vllm-cpu-env \
92
+ --model=meta-llama/Llama-3.2-1B-Instruct \
93
+ --dtype=bfloat16 \
94
+ other vLLM OpenAI server arguments
95
+ ```
96
97
!!! tip
98
For ARM or Apple silicon, use `docker/Dockerfile.arm`
@@ -144,32 +146,34 @@ vllm serve facebook/opt-125m
146
147
- If using vLLM CPU backend on a machine with hyper-threading, it is recommended to bind only one OpenMP thread on each physical CPU core using `VLLM_CPU_OMP_THREADS_BIND` or using auto thread binding feature by default. On a hyper-threading enabled platform with 16 logical CPU cores / 8 physical CPU cores:
148
147
-```console
148
-$ lscpu -e # check the mapping between logical CPU cores and physical CPU cores
149
-
150
-# The "CPU" column means the logical CPU core IDs, and the "CORE" column means the physical core IDs. On this platform, two logical cores are sharing one physical core.
151
-CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ MHZ
152
-0 0 0 0 0:0:0:0 yes 2401.0000 800.0000 800.000
153
-1 0 0 1 1:1:1:0 yes 2401.0000 800.0000 800.000
154
-2 0 0 2 2:2:2:0 yes 2401.0000 800.0000 800.000
155
-3 0 0 3 3:3:3:0 yes 2401.0000 800.0000 800.000
156
-4 0 0 4 4:4:4:0 yes 2401.0000 800.0000 800.000
157
-5 0 0 5 5:5:5:0 yes 2401.0000 800.0000 800.000
158
-6 0 0 6 6:6:6:0 yes 2401.0000 800.0000 800.000
159
-7 0 0 7 7:7:7:0 yes 2401.0000 800.0000 800.000
160
-8 0 0 0 0:0:0:0 yes 2401.0000 800.0000 800.000
161
-9 0 0 1 1:1:1:0 yes 2401.0000 800.0000 800.000
162
-10 0 0 2 2:2:2:0 yes 2401.0000 800.0000 800.000
163
-11 0 0 3 3:3:3:0 yes 2401.0000 800.0000 800.000
164
-12 0 0 4 4:4:4:0 yes 2401.0000 800.0000 800.000
165
-13 0 0 5 5:5:5:0 yes 2401.0000 800.0000 800.000
166
-14 0 0 6 6:6:6:0 yes 2401.0000 800.0000 800.000
167
-15 0 0 7 7:7:7:0 yes 2401.0000 800.0000 800.000
168
-
169
-# On this platform, it is recommend to only bind openMP threads on logical CPU cores 0-7 or 8-15
170
-$ export VLLM_CPU_OMP_THREADS_BIND=0-7
171
-$ python examples/offline_inference/basic/basic.py
172
-```
149
+??? Commands
150
+
151
+ ```console
152
+ $ lscpu -e # check the mapping between logical CPU cores and physical CPU cores
153
+
154
+ # The "CPU" column means the logical CPU core IDs, and the "CORE" column means the physical core IDs. On this platform, two logical cores are sharing one physical core.
155
+ CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ MHZ
156
+ 0 0 0 0 0:0:0:0 yes 2401.0000 800.0000 800.000
157
+ 1 0 0 1 1:1:1:0 yes 2401.0000 800.0000 800.000
158
+ 2 0 0 2 2:2:2:0 yes 2401.0000 800.0000 800.000
159
+ 3 0 0 3 3:3:3:0 yes 2401.0000 800.0000 800.000
160
+ 4 0 0 4 4:4:4:0 yes 2401.0000 800.0000 800.000
161
+ 5 0 0 5 5:5:5:0 yes 2401.0000 800.0000 800.000
162
+ 6 0 0 6 6:6:6:0 yes 2401.0000 800.0000 800.000
163
+ 7 0 0 7 7:7:7:0 yes 2401.0000 800.0000 800.000
164
+ 8 0 0 0 0:0:0:0 yes 2401.0000 800.0000 800.000
165
+ 9 0 0 1 1:1:1:0 yes 2401.0000 800.0000 800.000
166
+ 10 0 0 2 2:2:2:0 yes 2401.0000 800.0000 800.000
167
+ 11 0 0 3 3:3:3:0 yes 2401.0000 800.0000 800.000
168
+ 12 0 0 4 4:4:4:0 yes 2401.0000 800.0000 800.000
169
+ 13 0 0 5 5:5:5:0 yes 2401.0000 800.0000 800.000
170
+ 14 0 0 6 6:6:6:0 yes 2401.0000 800.0000 800.000
171
+ 15 0 0 7 7:7:7:0 yes 2401.0000 800.0000 800.000
172
+
173
+ # On this platform, it is recommend to only bind openMP threads on logical CPU cores 0-7 or 8-15
174
+ $ export VLLM_CPU_OMP_THREADS_BIND=0-7
175
+ $ python examples/offline_inference/basic/basic.py
176
+ ```
177
178
- If using vLLM CPU backend on a multi-socket machine with NUMA, be aware to set CPU cores using `VLLM_CPU_OMP_THREADS_BIND` to avoid cross NUMA node memory access.
179
docs/getting_started/installation/gpu/rocm.inc.md
+35
-31
@@ -90,24 +90,26 @@ Currently, there are no pre-built ROCm wheels.
90
91
4. Build vLLM. For example, vLLM on ROCM 6.3 can be built with the following steps:
92
93
- ```bash
94
- pip install --upgrade pip
95
-
96
- # Build & install AMD SMI
97
- pip install /opt/rocm/share/amd_smi
98
-
99
- # Install dependencies
100
- pip install --upgrade numba \
101
- scipy \
102
- huggingface-hub[cli,hf_transfer] \
103
- setuptools_scm
104
- pip install "numpy<2"
105
- pip install -r requirements/rocm.txt
106
-
107
- # Build vLLM for MI210/MI250/MI300.
108
- export PYTORCH_ROCM_ARCH="gfx90a;gfx942"
109
- python3 setup.py develop
110
- ```
93
+ ??? Commands
94
+
95
+ ```bash
96
+ pip install --upgrade pip
97
+
98
+ # Build & install AMD SMI
99
+ pip install /opt/rocm/share/amd_smi
100
+
101
+ # Install dependencies
102
+ pip install --upgrade numba \
103
+ scipy \
104
+ huggingface-hub[cli,hf_transfer] \
105
+ setuptools_scm
106
+ pip install "numpy<2"
107
+ pip install -r requirements/rocm.txt
108
+
109
+ # Build vLLM for MI210/MI250/MI300.
110
+ export PYTORCH_ROCM_ARCH="gfx90a;gfx942"
111
+ python3 setup.py develop
112
+ ```
113
114
This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm installation.
115
@@ -201,19 +203,21 @@ DOCKER_BUILDKIT=1 docker build \
203
204
To run the above docker image `vllm-rocm`, use the below command:
205
204
-```console
205
-docker run -it \
206
- --network=host \
207
- --group-add=video \
208
- --ipc=host \
209
- --cap-add=SYS_PTRACE \
210
- --security-opt seccomp=unconfined \
211
- --device /dev/kfd \
212
- --device /dev/dri \
213
- -v <path/to/model>:/app/model \
214
- vllm-rocm \
215
- bash
216
-```
206
+??? Command
207
+
208
+ ```console
209
+ docker run -it \
210
+ --network=host \
211
+ --group-add=video \
212
+ --ipc=host \
213
+ --cap-add=SYS_PTRACE \
214
+ --security-opt seccomp=unconfined \
215
+ --device /dev/kfd \
216
+ --device /dev/dri \
217
+ -v <path/to/model>:/app/model \
218
+ vllm-rocm \
219
+ bash
220
+ ```
221
222
Where the `<path/to/model>` is the location where the model is stored, for example, the weights for llama2 or llama3 models.
223
docs/getting_started/installation/intel_gaudi.md
+50
-46
@@ -200,7 +200,7 @@ INFO 08-01 21:37:59 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 1
200
201
`min` determines the lowest value of the bucket. `step` determines the interval between buckets, and `max` determines the upper bound of the bucket. Furthermore, interval between `min` and `step` has special handling -- `min` gets multiplied by consecutive powers of two, until `step` gets reached. We call this the ramp-up phase and it is used for handling lower batch sizes with minimum wastage, while allowing larger padding on larger batch sizes.
202
203
-Example (with ramp-up)
203
+Example (with ramp-up):
204
205
```text
206
min = 2, step = 32, max = 64
@@ -209,7 +209,7 @@ min = 2, step = 32, max = 64
209
=> buckets = ramp_up + stable => (2, 4, 8, 16, 32, 64)
210
```
211
212
-Example (without ramp-up)
212
+Example (without ramp-up):
213
214
```text
215
min = 128, step = 128, max = 512
@@ -232,19 +232,21 @@ As an example, if a request of 3 sequences, with max sequence length of 412 come
232
233
Warmup is an optional, but highly recommended step occurring before vLLM server starts listening. It executes a forward pass for each bucket with dummy data. The goal is to pre-compile all graphs and not incur any graph compilation overheads within bucket boundaries during server runtime. Each warmup step is logged during vLLM startup:
234
235
-```text
236
-INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:79.16 GiB
237
-INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][2/24] batch_size:4 seq_len:896 free_mem:55.43 GiB
238
-INFO 08-01 22:26:48 hpu_model_runner.py:1066] [Warmup][Prompt][3/24] batch_size:4 seq_len:768 free_mem:55.43 GiB
239
-...
240
-INFO 08-01 22:26:59 hpu_model_runner.py:1066] [Warmup][Prompt][24/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
241
-INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][1/48] batch_size:4 seq_len:2048 free_mem:55.43 GiB
242
-INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][2/48] batch_size:4 seq_len:1920 free_mem:55.43 GiB
243
-INFO 08-01 22:27:01 hpu_model_runner.py:1066] [Warmup][Decode][3/48] batch_size:4 seq_len:1792 free_mem:55.43 GiB
244
-...
245
-INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][47/48] batch_size:2 seq_len:128 free_mem:55.43 GiB
246
-INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
247
-```
235
+??? Logs
236
+
237
+ ```text
238
+ INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:79.16 GiB
239
+ INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][2/24] batch_size:4 seq_len:896 free_mem:55.43 GiB
240
+ INFO 08-01 22:26:48 hpu_model_runner.py:1066] [Warmup][Prompt][3/24] batch_size:4 seq_len:768 free_mem:55.43 GiB
241
+ ...
242
+ INFO 08-01 22:26:59 hpu_model_runner.py:1066] [Warmup][Prompt][24/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
243
+ INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][1/48] batch_size:4 seq_len:2048 free_mem:55.43 GiB
244
+ INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][2/48] batch_size:4 seq_len:1920 free_mem:55.43 GiB
245
+ INFO 08-01 22:27:01 hpu_model_runner.py:1066] [Warmup][Decode][3/48] batch_size:4 seq_len:1792 free_mem:55.43 GiB
246
+ ...
247
+ INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][47/48] batch_size:2 seq_len:128 free_mem:55.43 GiB
248
+ INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
249
+ ```
250
251
This example uses the same buckets as in the [Bucketing Mechanism][gaudi-bucketing-mechanism] section. Each output line corresponds to execution of a single bucket. When bucket is executed for the first time, its graph is compiled and can be reused later on, skipping further graph compilations.
252
@@ -279,37 +281,39 @@ When there's large amount of requests pending, vLLM scheduler will attempt to fi
281
282
Each described step is logged by vLLM server, as follows (negative values correspond to memory being released):
283
282
-```text
283
-INFO 08-02 17:37:44 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]
284
-INFO 08-02 17:37:44 hpu_model_runner.py:499] Generated 24 prompt buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024)]
285
-INFO 08-02 17:37:44 hpu_model_runner.py:504] Decode bucket config (min, step, max_warmup) bs:[1, 128, 4], seq:[128, 128, 2048]
286
-INFO 08-02 17:37:44 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
287
-INFO 08-02 17:37:52 hpu_model_runner.py:430] Pre-loading model weights on hpu:0 took 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
288
-INFO 08-02 17:37:52 hpu_model_runner.py:438] Wrapping in HPU Graph took 0 B of device memory (14.97 GiB/94.62 GiB used) and -252 KiB of host memory (475.2 GiB/1007 GiB used)
289
-INFO 08-02 17:37:52 hpu_model_runner.py:442] Loading model weights took in total 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
290
-INFO 08-02 17:37:54 hpu_worker.py:134] Model profiling run took 504 MiB of device memory (15.46 GiB/94.62 GiB used) and 180.9 MiB of host memory (475.4 GiB/1007 GiB used)
291
-INFO 08-02 17:37:54 hpu_worker.py:158] Free device memory: 79.16 GiB, 39.58 GiB usable (gpu_memory_utilization=0.5), 15.83 GiB reserved for HPUGraphs (VLLM_GRAPH_RESERVED_MEM=0.4), 23.75 GiB reserved for KV cache
292
-INFO 08-02 17:37:54 hpu_executor.py:85] # HPU blocks: 1519, # CPU blocks: 0
293
-INFO 08-02 17:37:54 hpu_worker.py:190] Initializing cache engine took 23.73 GiB of device memory (39.2 GiB/94.62 GiB used) and -1.238 MiB of host memory (475.4 GiB/1007 GiB used)
294
-INFO 08-02 17:37:54 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:55.43 GiB
295
-...
296
-INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
297
-INFO 08-02 17:38:22 hpu_model_runner.py:1159] Using 15.85 GiB/55.43 GiB of free device memory for HPUGraphs, 7.923 GiB for prompt and 7.923 GiB for decode (VLLM_GRAPH_PROMPT_RATIO=0.3)
298
-INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][1/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
299
-...
300
-INFO 08-02 17:38:26 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][11/24] batch_size:1 seq_len:896 free_mem:48.77 GiB
301
-INFO 08-02 17:38:27 hpu_model_runner.py:1066] [Warmup][Graph/Decode][1/48] batch_size:4 seq_len:128 free_mem:47.51 GiB
302
-...
303
-INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Decode][48/48] batch_size:1 seq_len:2048 free_mem:47.35 GiB
304
-INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][12/24] batch_size:4 seq_len:256 free_mem:47.35 GiB
305
-INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][13/24] batch_size:2 seq_len:512 free_mem:45.91 GiB
306
-INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][14/24] batch_size:1 seq_len:1024 free_mem:44.48 GiB
307
-INFO 08-02 17:38:43 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][15/24] batch_size:2 seq_len:640 free_mem:43.03 GiB
308
-INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Prompt captured:15 (62.5%) used_mem:14.03 GiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (4, 128), (4, 256)]
309
-INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Decode captured:48 (100.0%) used_mem:161.9 MiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
310
-INFO 08-02 17:38:43 hpu_model_runner.py:1206] Warmup finished in 49 secs, allocated 14.19 GiB of device memory
311
-INFO 08-02 17:38:43 hpu_executor.py:91] init_cache_engine took 37.92 GiB of device memory (53.39 GiB/94.62 GiB used) and 57.86 MiB of host memory (475.4 GiB/1007 GiB used)
312
-```
284
+??? Logs
285
+
286
+ ```text
287
+ INFO 08-02 17:37:44 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]
288
+ INFO 08-02 17:37:44 hpu_model_runner.py:499] Generated 24 prompt buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024)]
289
+ INFO 08-02 17:37:44 hpu_model_runner.py:504] Decode bucket config (min, step, max_warmup) bs:[1, 128, 4], seq:[128, 128, 2048]
290
+ INFO 08-02 17:37:44 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
291
+ INFO 08-02 17:37:52 hpu_model_runner.py:430] Pre-loading model weights on hpu:0 took 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
292
+ INFO 08-02 17:37:52 hpu_model_runner.py:438] Wrapping in HPU Graph took 0 B of device memory (14.97 GiB/94.62 GiB used) and -252 KiB of host memory (475.2 GiB/1007 GiB used)
293
+ INFO 08-02 17:37:52 hpu_model_runner.py:442] Loading model weights took in total 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
294
+ INFO 08-02 17:37:54 hpu_worker.py:134] Model profiling run took 504 MiB of device memory (15.46 GiB/94.62 GiB used) and 180.9 MiB of host memory (475.4 GiB/1007 GiB used)
295
+ INFO 08-02 17:37:54 hpu_worker.py:158] Free device memory: 79.16 GiB, 39.58 GiB usable (gpu_memory_utilization=0.5), 15.83 GiB reserved for HPUGraphs (VLLM_GRAPH_RESERVED_MEM=0.4), 23.75 GiB reserved for KV cache
296
+ INFO 08-02 17:37:54 hpu_executor.py:85] # HPU blocks: 1519, # CPU blocks: 0
297
+ INFO 08-02 17:37:54 hpu_worker.py:190] Initializing cache engine took 23.73 GiB of device memory (39.2 GiB/94.62 GiB used) and -1.238 MiB of host memory (475.4 GiB/1007 GiB used)
298
+ INFO 08-02 17:37:54 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:55.43 GiB
299
+ ...
300
+ INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
301
+ INFO 08-02 17:38:22 hpu_model_runner.py:1159] Using 15.85 GiB/55.43 GiB of free device memory for HPUGraphs, 7.923 GiB for prompt and 7.923 GiB for decode (VLLM_GRAPH_PROMPT_RATIO=0.3)
302
+ INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][1/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
303
+ ...
304
+ INFO 08-02 17:38:26 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][11/24] batch_size:1 seq_len:896 free_mem:48.77 GiB
305
+ INFO 08-02 17:38:27 hpu_model_runner.py:1066] [Warmup][Graph/Decode][1/48] batch_size:4 seq_len:128 free_mem:47.51 GiB
306
+ ...
307
+ INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Decode][48/48] batch_size:1 seq_len:2048 free_mem:47.35 GiB
308
+ INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][12/24] batch_size:4 seq_len:256 free_mem:47.35 GiB
309
+ INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][13/24] batch_size:2 seq_len:512 free_mem:45.91 GiB
310
+ INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][14/24] batch_size:1 seq_len:1024 free_mem:44.48 GiB
311
+ INFO 08-02 17:38:43 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][15/24] batch_size:2 seq_len:640 free_mem:43.03 GiB
312
+ INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Prompt captured:15 (62.5%) used_mem:14.03 GiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (4, 128), (4, 256)]
313
+ INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Decode captured:48 (100.0%) used_mem:161.9 MiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
314
+ INFO 08-02 17:38:43 hpu_model_runner.py:1206] Warmup finished in 49 secs, allocated 14.19 GiB of device memory
315
+ INFO 08-02 17:38:43 hpu_executor.py:91] init_cache_engine took 37.92 GiB of device memory (53.39 GiB/94.62 GiB used) and 57.86 MiB of host memory (475.4 GiB/1007 GiB used)
316
+ ```
317
318
### Recommended vLLM Parameters
319
docs/getting_started/quickstart.md
+38
-34
@@ -147,20 +147,22 @@ curl http://localhost:8000/v1/completions \
147
148
Since this server is compatible with OpenAI API, you can use it as a drop-in replacement for any applications using OpenAI API. For example, another way to query the server is via the `openai` Python package:
149
150
-```python
151
-from openai import OpenAI
152
-
153
-# Modify OpenAI's API key and API base to use vLLM's API server.
154
-openai_api_key = "EMPTY"
155
-openai_api_base = "http://localhost:8000/v1"
156
-client = OpenAI(
157
- api_key=openai_api_key,
158
- base_url=openai_api_base,
159
-)
160
-completion = client.completions.create(model="Qwen/Qwen2.5-1.5B-Instruct",
161
- prompt="San Francisco is a")
162
-print("Completion result:", completion)
163
-```
150
+??? Code
151
+
152
+ ```python
153
+ from openai import OpenAI
154
+
155
+ # Modify OpenAI's API key and API base to use vLLM's API server.
156
+ openai_api_key = "EMPTY"
157
+ openai_api_base = "http://localhost:8000/v1"
158
+ client = OpenAI(
159
+ api_key=openai_api_key,
160
+ base_url=openai_api_base,
161
+ )
162
+ completion = client.completions.create(model="Qwen/Qwen2.5-1.5B-Instruct",
163
+ prompt="San Francisco is a")
164
+ print("Completion result:", completion)
165
+ ```
166
167
A more detailed client example can be found here: <gh-file:examples/online_serving/openai_completion_client.py>
168
@@ -184,26 +186,28 @@ curl http://localhost:8000/v1/chat/completions \
186
187
Alternatively, you can use the `openai` Python package:
188
187
-```python
188
-from openai import OpenAI
189
-# Set OpenAI's API key and API base to use vLLM's API server.
190
-openai_api_key = "EMPTY"
191
-openai_api_base = "http://localhost:8000/v1"
192
-
193
-client = OpenAI(
194
- api_key=openai_api_key,
195
- base_url=openai_api_base,
196
-)
197
-
198
-chat_response = client.chat.completions.create(
199
- model="Qwen/Qwen2.5-1.5B-Instruct",
200
- messages=[
201
- {"role": "system", "content": "You are a helpful assistant."},
202
- {"role": "user", "content": "Tell me a joke."},
203
- ]
204
-)
205
-print("Chat response:", chat_response)
206
-```
189
+??? Code
190
+
191
+ ```python
192
+ from openai import OpenAI
193
+ # Set OpenAI's API key and API base to use vLLM's API server.
194
+ openai_api_key = "EMPTY"
195
+ openai_api_base = "http://localhost:8000/v1"
196
+
197
+ client = OpenAI(
198
+ api_key=openai_api_key,
199
+ base_url=openai_api_base,
200
+ )
201
+
202
+ chat_response = client.chat.completions.create(
203
+ model="Qwen/Qwen2.5-1.5B-Instruct",
204
+ messages=[
205
+ {"role": "system", "content": "You are a helpful assistant."},
206
+ {"role": "user", "content": "Tell me a joke."},
207
+ ]
208
+ )
209
+ print("Chat response:", chat_response)
210
+ ```
211
212
## On Attention Backends
213
docs/models/generative_models.md
+31
-29
@@ -85,35 +85,37 @@ and automatically applies the model's [chat template](https://huggingface.co/doc
85
In general, only instruction-tuned models have a chat template.
86
Base models may perform poorly as they are not trained to respond to the chat conversation.
87
88
-```python
89
-from vllm import LLM
90
-
91
-llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
92
-conversation = [
93
- {
94
- "role": "system",
95
- "content": "You are a helpful assistant"
96
- },
97
- {
98
- "role": "user",
99
- "content": "Hello"
100
- },
101
- {
102
- "role": "assistant",
103
- "content": "Hello! How can I assist you today?"
104
- },
105
- {
106
- "role": "user",
107
- "content": "Write an essay about the importance of higher education.",
108
- },
109
-]
110
-outputs = llm.chat(conversation)
111
-
112
-for output in outputs:
113
- prompt = output.prompt
114
- generated_text = output.outputs[0].text
115
- print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
116
-```
88
+??? Code
89
+
90
+ ```python
91
+ from vllm import LLM
92
+
93
+ llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
94
+ conversation = [
95
+ {
96
+ "role": "system",
97
+ "content": "You are a helpful assistant"
98
+ },
99
+ {
100
+ "role": "user",
101
+ "content": "Hello"
102
+ },
103
+ {
104
+ "role": "assistant",
105
+ "content": "Hello! How can I assist you today?"
106
+ },
107
+ {
108
+ "role": "user",
109
+ "content": "Write an essay about the importance of higher education.",
110
+ },
111
+ ]
112
+ outputs = llm.chat(conversation)
113
+
114
+ for output in outputs:
115
+ prompt = output.prompt
116
+ generated_text = output.outputs[0].text
117
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
118
+ ```
119
120
A code example can be found here: <gh-file:examples/offline_inference/basic/chat.py>
121
docs/models/supported_models.md
+40
-23
@@ -70,7 +70,10 @@ To make your model compatible with the Transformers backend, it needs:
70
2. `MyAttention` must use `ALL_ATTENTION_FUNCTIONS` to call attention.
71
3. `MyModel` must contain `_supports_attention_backend = True`.
72
73
-```python title="modeling_my_model.py"
73
+<details>
74
+<summary>modeling_my_model.py</summary>
75
+
76
+```python
77
78
from transformers import PreTrainedModel
79
from torch import nn
@@ -93,6 +96,8 @@ class MyModel(PreTrainedModel):
96
_supports_attention_backend = True
97
```
98
99
+</details>
100
+
101
Here is what happens in the background when this model is loaded:
102
103
1. The config is loaded.
@@ -103,7 +108,10 @@ That's it!
108
109
For your model to be compatible with vLLM's tensor parallel and/or pipeline parallel features, you must add `base_model_tp_plan` and/or `base_model_pp_plan` to your model's config class:
110
106
-```python title="configuration_my_model.py"
111
+<details>
112
+<summary>configuration_my_model.py</summary>
113
+
114
+```python
115
116
from transformers import PretrainedConfig
117
@@ -123,6 +131,8 @@ class MyConfig(PretrainedConfig):
131
}
132
```
133
134
+</details>
135
+
136
- `base_model_tp_plan` is a `dict` that maps fully qualified layer name patterns to tensor parallel styles (currently only `"colwise"` and `"rowwise"` are supported).
137
- `base_model_pp_plan` is a `dict` that maps direct child layer names to `tuple`s of `list`s of `str`s:
138
* You only need to do this for layers which are not present on all pipeline stages
@@ -198,6 +208,9 @@ huggingface-cli scan-cache --dir ~/.cache/huggingface/hub
208
209
Use the Hugging Face CLI to interactively [delete downloaded model](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#clean-your-cache) from the cache:
210
211
+<details>
212
+<summary>Commands</summary>
213
+
214
```console
215
# The `delete-cache` command requires extra dependencies to work with the TUI.
216
# Please run `pip install huggingface_hub[cli]` to install them.
@@ -224,6 +237,8 @@ Start deletion.
237
Done. Deleted 1 repo(s) and 0 revision(s) for a total of 438.9M.
238
```
239
240
+</details>
241
+
242
#### Using a proxy
243
244
Here are some tips for loading/downloading models from Hugging Face using a proxy:
@@ -601,27 +616,29 @@ Specified using `--task generate`.
616
617
For the best results, we recommend using the following dependency versions (tested on A10 and L40):
618
604
- ```text
605
- # Core vLLM-compatible dependencies with Molmo accuracy setup (tested on L40)
606
- torch==2.5.1
607
- torchvision==0.20.1
608
- transformers==4.48.1
609
- tokenizers==0.21.0
610
- tiktoken==0.7.0
611
- vllm==0.7.0
612
-
613
- # Optional but recommended for improved performance and stability
614
- triton==3.1.0
615
- xformers==0.0.28.post3
616
- uvloop==0.21.0
617
- protobuf==5.29.3
618
- openai==1.60.2
619
- opencv-python-headless==4.11.0.86
620
- pillow==10.4.0
621
-
622
- # Installed FlashAttention (for float16 only)
623
- flash-attn>=2.5.6 # Not used in float32, but should be documented
624
- ```
619
+ ??? Dependency versions
620
+
621
+ ```text
622
+ # Core vLLM-compatible dependencies with Molmo accuracy setup (tested on L40)
623
+ torch==2.5.1
624
+ torchvision==0.20.1
625
+ transformers==4.48.1
626
+ tokenizers==0.21.0
627
+ tiktoken==0.7.0
628
+ vllm==0.7.0
629
+
630
+ # Optional but recommended for improved performance and stability
631
+ triton==3.1.0
632
+ xformers==0.0.28.post3
633
+ uvloop==0.21.0
634
+ protobuf==5.29.3
635
+ openai==1.60.2
636
+ opencv-python-headless==4.11.0.86
637
+ pillow==10.4.0
638
+
639
+ # Installed FlashAttention (for float16 only)
640
+ flash-attn>=2.5.6 # Not used in float32, but should be documented
641
+ ```
642
643
**Note:** Make sure you understand the security implications of using outdated packages.
644
docs/serving/integrations/langchain.md
+16
-14
@@ -13,19 +13,21 @@ pip install langchain langchain_community -q
13
14
To run inference on a single or multiple GPUs, use `VLLM` class from `langchain`.
15
16
-```python
17
-from langchain_community.llms import VLLM
18
-
19
-llm = VLLM(model="mosaicml/mpt-7b",
20
- trust_remote_code=True, # mandatory for hf models
21
- max_new_tokens=128,
22
- top_k=10,
23
- top_p=0.95,
24
- temperature=0.8,
25
- # tensor_parallel_size=... # for distributed inference
26
-)
27
-
28
-print(llm("What is the capital of France ?"))
29
-```
16
+??? Code
17
+
18
+ ```python
19
+ from langchain_community.llms import VLLM
20
+
21
+ llm = VLLM(model="mosaicml/mpt-7b",
22
+ trust_remote_code=True, # mandatory for hf models
23
+ max_new_tokens=128,
24
+ top_k=10,
25
+ top_p=0.95,
26
+ temperature=0.8,
27
+ # tensor_parallel_size=... # for distributed inference
28
+ )
29
+
30
+ print(llm("What is the capital of France ?"))
31
+ ```
32
33
Please refer to this [Tutorial](https://python.langchain.com/docs/integrations/llms/vllm) for more details.
docs/serving/openai_compatible_server.md
+286
-270
@@ -15,22 +15,24 @@ vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
15
16
To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).
17
18
-```python
19
-from openai import OpenAI
20
-client = OpenAI(
21
- base_url="http://localhost:8000/v1",
22
- api_key="token-abc123",
23
-)
18
+??? Code
19
25
-completion = client.chat.completions.create(
26
- model="NousResearch/Meta-Llama-3-8B-Instruct",
27
- messages=[
28
- {"role": "user", "content": "Hello!"}
29
- ]
30
-)
20
+ ```python
21
+ from openai import OpenAI
22
+ client = OpenAI(
23
+ base_url="http://localhost:8000/v1",
24
+ api_key="token-abc123",
25
+ )
26
32
-print(completion.choices[0].message)
33
-```
27
+ completion = client.chat.completions.create(
28
+ model="NousResearch/Meta-Llama-3-8B-Instruct",
29
+ messages=[
30
+ {"role": "user", "content": "Hello!"}
31
+ ]
32
+ )
33
+
34
+ print(completion.choices[0].message)
35
+ ```
36
37
!!! tip
38
vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
@@ -147,27 +149,29 @@ with `--enable-request-id-headers`.
149
> rather than within the vLLM layer for this reason.
150
> See [this PR](https://github.com/vllm-project/vllm/pull/11529) for more details.
151
150
-```python
151
-completion = client.chat.completions.create(
152
- model="NousResearch/Meta-Llama-3-8B-Instruct",
153
- messages=[
154
- {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
155
- ],
156
- extra_headers={
157
- "x-request-id": "sentiment-classification-00001",
158
- }
159
-)
160
-print(completion._request_id)
152
+??? Code
153
162
-completion = client.completions.create(
163
- model="NousResearch/Meta-Llama-3-8B-Instruct",
164
- prompt="A robot may not injure a human being",
165
- extra_headers={
166
- "x-request-id": "completion-test",
167
- }
168
-)
169
-print(completion._request_id)
170
-```
154
+ ```python
155
+ completion = client.chat.completions.create(
156
+ model="NousResearch/Meta-Llama-3-8B-Instruct",
157
+ messages=[
158
+ {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
159
+ ],
160
+ extra_headers={
161
+ "x-request-id": "sentiment-classification-00001",
162
+ }
163
+ )
164
+ print(completion._request_id)
165
+
166
+ completion = client.completions.create(
167
+ model="NousResearch/Meta-Llama-3-8B-Instruct",
168
+ prompt="A robot may not injure a human being",
169
+ extra_headers={
170
+ "x-request-id": "completion-test",
171
+ }
172
+ )
173
+ print(completion._request_id)
174
+ ```
175
176
## API Reference
177
@@ -184,15 +188,19 @@ Code example: <gh-file:examples/online_serving/openai_completion_client.py>
188
189
The following [sampling parameters][sampling-params] are supported.
190
187
-```python
188
---8<-- "vllm/entrypoints/openai/protocol.py:completion-sampling-params"
189
-```
191
+??? Code
192
+
193
+ ```python
194
+ --8<-- "vllm/entrypoints/openai/protocol.py:completion-sampling-params"
195
+ ```
196
197
The following extra parameters are supported:
198
193
-```python
194
---8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"
195
-```
199
+??? Code
200
+
201
+ ```python
202
+ --8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"
203
+ ```
204
205
[](){ #chat-api }
206
@@ -212,15 +220,19 @@ Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
220
221
The following [sampling parameters][sampling-params] are supported.
222
215
-```python
216
---8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-sampling-params"
217
-```
223
+??? Code
224
+
225
+ ```python
226
+ --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-sampling-params"
227
+ ```
228
229
The following extra parameters are supported:
230
221
-```python
222
---8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"
223
-```
231
+??? Code
232
+
233
+ ```python
234
+ --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"
235
+ ```
236
237
[](){ #embeddings-api }
238
@@ -259,29 +271,31 @@ and passing a list of `messages` in the request. Refer to the examples below for
271
272
Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
273
262
- ```python
263
- import requests
264
-
265
- image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
266
-
267
- response = requests.post(
268
- "http://localhost:8000/v1/embeddings",
269
- json={
270
- "model": "TIGER-Lab/VLM2Vec-Full",
271
- "messages": [{
272
- "role": "user",
273
- "content": [
274
- {"type": "image_url", "image_url": {"url": image_url}},
275
- {"type": "text", "text": "Represent the given image."},
276
- ],
277
- }],
278
- "encoding_format": "float",
279
- },
280
- )
281
- response.raise_for_status()
282
- response_json = response.json()
283
- print("Embedding output:", response_json["data"][0]["embedding"])
284
- ```
274
+ ??? Code
275
+
276
+ ```python
277
+ import requests
278
+
279
+ image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
280
+
281
+ response = requests.post(
282
+ "http://localhost:8000/v1/embeddings",
283
+ json={
284
+ "model": "TIGER-Lab/VLM2Vec-Full",
285
+ "messages": [{
286
+ "role": "user",
287
+ "content": [
288
+ {"type": "image_url", "image_url": {"url": image_url}},
289
+ {"type": "text", "text": "Represent the given image."},
290
+ ],
291
+ }],
292
+ "encoding_format": "float",
293
+ },
294
+ )
295
+ response.raise_for_status()
296
+ response_json = response.json()
297
+ print("Embedding output:", response_json["data"][0]["embedding"])
298
+ ```
299
300
=== "DSE-Qwen2-MRL"
301
@@ -316,15 +330,19 @@ The following [pooling parameters][pooling-params] are supported.
330
331
The following extra parameters are supported by default:
332
319
-```python
320
---8<-- "vllm/entrypoints/openai/protocol.py:embedding-extra-params"
321
-```
333
+??? Code
334
+
335
+ ```python
336
+ --8<-- "vllm/entrypoints/openai/protocol.py:embedding-extra-params"
337
+ ```
338
339
For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:
340
325
-```python
326
---8<-- "vllm/entrypoints/openai/protocol.py:chat-embedding-extra-params"
327
-```
341
+??? Code
342
+
343
+ ```python
344
+ --8<-- "vllm/entrypoints/openai/protocol.py:chat-embedding-extra-params"
345
+ ```
346
347
[](){ #transcriptions-api }
348
@@ -343,15 +361,19 @@ Code example: <gh-file:examples/online_serving/openai_transcription_client.py>
361
362
The following [sampling parameters][sampling-params] are supported.
363
346
-```python
347
---8<-- "vllm/entrypoints/openai/protocol.py:transcription-sampling-params"
348
-```
364
+??? Code
365
+
366
+ ```python
367
+ --8<-- "vllm/entrypoints/openai/protocol.py:transcription-sampling-params"
368
+ ```
369
370
The following extra parameters are supported:
371
352
-```python
353
---8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"
354
-```
372
+??? Code
373
+
374
+ ```python
375
+ --8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"
376
+ ```
377
378
[](){ #tokenizer-api }
379
@@ -387,8 +409,6 @@ Code example: <gh-file:examples/online_serving/openai_classification_client.py>
409
410
You can classify multiple texts by passing an array of strings:
411
390
-Request:
391
-
412
```bash
413
curl -v "http://127.0.0.1:8000/classify" \
414
-H "Content-Type: application/json" \
@@ -401,47 +421,45 @@ curl -v "http://127.0.0.1:8000/classify" \
421
}'
422
```
423
404
-Response:
424
+??? Response
425
406
-```bash
407
-{
408
- "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",
409
- "object": "list",
410
- "created": 1745383065,
411
- "model": "jason9693/Qwen2.5-1.5B-apeach",
412
- "data": [
413
- {
414
- "index": 0,
415
- "label": "Default",
416
- "probs": [
417
- 0.565970778465271,
418
- 0.4340292513370514
419
- ],
420
- "num_classes": 2
421
- },
426
+ ```bash
427
{
423
- "index": 1,
424
- "label": "Spoiled",
425
- "probs": [
426
- 0.26448777318000793,
427
- 0.7355121970176697
428
+ "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",
429
+ "object": "list",
430
+ "created": 1745383065,
431
+ "model": "jason9693/Qwen2.5-1.5B-apeach",
432
+ "data": [
433
+ {
434
+ "index": 0,
435
+ "label": "Default",
436
+ "probs": [
437
+ 0.565970778465271,
438
+ 0.4340292513370514
439
+ ],
440
+ "num_classes": 2
441
+ },
442
+ {
443
+ "index": 1,
444
+ "label": "Spoiled",
445
+ "probs": [
446
+ 0.26448777318000793,
447
+ 0.7355121970176697
448
+ ],
449
+ "num_classes": 2
450
+ }
451
],
429
- "num_classes": 2
452
+ "usage": {
453
+ "prompt_tokens": 20,
454
+ "total_tokens": 20,
455
+ "completion_tokens": 0,
456
+ "prompt_tokens_details": null
457
+ }
458
}
431
- ],
432
- "usage": {
433
- "prompt_tokens": 20,
434
- "total_tokens": 20,
435
- "completion_tokens": 0,
436
- "prompt_tokens_details": null
437
- }
438
-}
439
-```
459
+ ```
460
461
You can also pass a string directly to the `input` field:
462
443
-Request:
444
-
463
```bash
464
curl -v "http://127.0.0.1:8000/classify" \
465
-H "Content-Type: application/json" \
@@ -451,33 +469,33 @@ curl -v "http://127.0.0.1:8000/classify" \
469
}'
470
```
471
454
-Response:
472
+??? Response
473
456
-```bash
457
-{
458
- "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
459
- "object": "list",
460
- "created": 1745383213,
461
- "model": "jason9693/Qwen2.5-1.5B-apeach",
462
- "data": [
474
+ ```bash
475
{
464
- "index": 0,
465
- "label": "Default",
466
- "probs": [
467
- 0.565970778465271,
468
- 0.4340292513370514
476
+ "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
477
+ "object": "list",
478
+ "created": 1745383213,
479
+ "model": "jason9693/Qwen2.5-1.5B-apeach",
480
+ "data": [
481
+ {
482
+ "index": 0,
483
+ "label": "Default",
484
+ "probs": [
485
+ 0.565970778465271,
486
+ 0.4340292513370514
487
+ ],
488
+ "num_classes": 2
489
+ }
490
],
470
- "num_classes": 2
491
+ "usage": {
492
+ "prompt_tokens": 10,
493
+ "total_tokens": 10,
494
+ "completion_tokens": 0,
495
+ "prompt_tokens_details": null
496
+ }
497
}
472
- ],
473
- "usage": {
474
- "prompt_tokens": 10,
475
- "total_tokens": 10,
476
- "completion_tokens": 0,
477
- "prompt_tokens_details": null
478
- }
479
-}
480
-```
498
+ ```
499
500
#### Extra parameters
501
@@ -508,8 +526,6 @@ Code example: <gh-file:examples/online_serving/openai_cross_encoder_score.py>
526
527
You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.
528
511
-Request:
512
-
529
```bash
530
curl -X 'POST' \
531
'http://127.0.0.1:8000/score' \
@@ -523,24 +539,24 @@ curl -X 'POST' \
539
}'
540
```
541
526
-Response:
542
+??? Response
543
528
-```bash
529
-{
530
- "id": "score-request-id",
531
- "object": "list",
532
- "created": 693447,
533
- "model": "BAAI/bge-reranker-v2-m3",
534
- "data": [
544
+ ```bash
545
{
536
- "index": 0,
537
- "object": "score",
538
- "score": 1
546
+ "id": "score-request-id",
547
+ "object": "list",
548
+ "created": 693447,
549
+ "model": "BAAI/bge-reranker-v2-m3",
550
+ "data": [
551
+ {
552
+ "index": 0,
553
+ "object": "score",
554
+ "score": 1
555
+ }
556
+ ],
557
+ "usage": {}
558
}
540
- ],
541
- "usage": {}
542
-}
543
-```
559
+ ```
560
561
#### Batch inference
562
@@ -548,95 +564,95 @@ You can pass a string to `text_1` and a list to `text_2`, forming multiple sente
564
where each pair is built from `text_1` and a string in `text_2`.
565
The total number of pairs is `len(text_2)`.
566
551
-Request:
567
+??? Request
568
553
-```bash
554
-curl -X 'POST' \
555
- 'http://127.0.0.1:8000/score' \
556
- -H 'accept: application/json' \
557
- -H 'Content-Type: application/json' \
558
- -d '{
559
- "model": "BAAI/bge-reranker-v2-m3",
560
- "text_1": "What is the capital of France?",
561
- "text_2": [
562
- "The capital of Brazil is Brasilia.",
563
- "The capital of France is Paris."
564
- ]
565
-}'
566
-```
569
+ ```bash
570
+ curl -X 'POST' \
571
+ 'http://127.0.0.1:8000/score' \
572
+ -H 'accept: application/json' \
573
+ -H 'Content-Type: application/json' \
574
+ -d '{
575
+ "model": "BAAI/bge-reranker-v2-m3",
576
+ "text_1": "What is the capital of France?",
577
+ "text_2": [
578
+ "The capital of Brazil is Brasilia.",
579
+ "The capital of France is Paris."
580
+ ]
581
+ }'
582
+ ```
583
568
-Response:
584
+??? Response
585
570
-```bash
571
-{
572
- "id": "score-request-id",
573
- "object": "list",
574
- "created": 693570,
575
- "model": "BAAI/bge-reranker-v2-m3",
576
- "data": [
577
- {
578
- "index": 0,
579
- "object": "score",
580
- "score": 0.001094818115234375
581
- },
586
+ ```bash
587
{
583
- "index": 1,
584
- "object": "score",
585
- "score": 1
588
+ "id": "score-request-id",
589
+ "object": "list",
590
+ "created": 693570,
591
+ "model": "BAAI/bge-reranker-v2-m3",
592
+ "data": [
593
+ {
594
+ "index": 0,
595
+ "object": "score",
596
+ "score": 0.001094818115234375
597
+ },
598
+ {
599
+ "index": 1,
600
+ "object": "score",
601
+ "score": 1
602
+ }
603
+ ],
604
+ "usage": {}
605
}
587
- ],
588
- "usage": {}
589
-}
590
-```
606
+ ```
607
608
You can pass a list to both `text_1` and `text_2`, forming multiple sentence pairs
609
where each pair is built from a string in `text_1` and the corresponding string in `text_2` (similar to `zip()`).
610
The total number of pairs is `len(text_2)`.
611
596
-Request:
612
+??? Request
613
598
-```bash
599
-curl -X 'POST' \
600
- 'http://127.0.0.1:8000/score' \
601
- -H 'accept: application/json' \
602
- -H 'Content-Type: application/json' \
603
- -d '{
604
- "model": "BAAI/bge-reranker-v2-m3",
605
- "encoding_format": "float",
606
- "text_1": [
607
- "What is the capital of Brazil?",
608
- "What is the capital of France?"
609
- ],
610
- "text_2": [
611
- "The capital of Brazil is Brasilia.",
612
- "The capital of France is Paris."
613
- ]
614
-}'
615
-```
614
+ ```bash
615
+ curl -X 'POST' \
616
+ 'http://127.0.0.1:8000/score' \
617
+ -H 'accept: application/json' \
618
+ -H 'Content-Type: application/json' \
619
+ -d '{
620
+ "model": "BAAI/bge-reranker-v2-m3",
621
+ "encoding_format": "float",
622
+ "text_1": [
623
+ "What is the capital of Brazil?",
624
+ "What is the capital of France?"
625
+ ],
626
+ "text_2": [
627
+ "The capital of Brazil is Brasilia.",
628
+ "The capital of France is Paris."
629
+ ]
630
+ }'
631
+ ```
632
617
-Response:
633
+??? Response
634
619
-```bash
620
-{
621
- "id": "score-request-id",
622
- "object": "list",
623
- "created": 693447,
624
- "model": "BAAI/bge-reranker-v2-m3",
625
- "data": [
626
- {
627
- "index": 0,
628
- "object": "score",
629
- "score": 1
630
- },
635
+ ```bash
636
{
632
- "index": 1,
633
- "object": "score",
634
- "score": 1
637
+ "id": "score-request-id",
638
+ "object": "list",
639
+ "created": 693447,
640
+ "model": "BAAI/bge-reranker-v2-m3",
641
+ "data": [
642
+ {
643
+ "index": 0,
644
+ "object": "score",
645
+ "score": 1
646
+ },
647
+ {
648
+ "index": 1,
649
+ "object": "score",
650
+ "score": 1
651
+ }
652
+ ],
653
+ "usage": {}
654
}
636
- ],
637
- "usage": {}
638
-}
639
-```
655
+ ```
656
657
#### Extra parameters
658
@@ -675,51 +691,51 @@ Code example: <gh-file:examples/online_serving/jinaai_rerank_client.py>
691
Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.
692
Result documents will be sorted by relevance, and the `index` property can be used to determine original order.
693
678
-Request:
694
+??? Request
695
680
-```bash
681
-curl -X 'POST' \
682
- 'http://127.0.0.1:8000/v1/rerank' \
683
- -H 'accept: application/json' \
684
- -H 'Content-Type: application/json' \
685
- -d '{
686
- "model": "BAAI/bge-reranker-base",
687
- "query": "What is the capital of France?",
688
- "documents": [
689
- "The capital of Brazil is Brasilia.",
690
- "The capital of France is Paris.",
691
- "Horses and cows are both animals"
692
- ]
693
-}'
694
-```
696
+ ```bash
697
+ curl -X 'POST' \
698
+ 'http://127.0.0.1:8000/v1/rerank' \
699
+ -H 'accept: application/json' \
700
+ -H 'Content-Type: application/json' \
701
+ -d '{
702
+ "model": "BAAI/bge-reranker-base",
703
+ "query": "What is the capital of France?",
704
+ "documents": [
705
+ "The capital of Brazil is Brasilia.",
706
+ "The capital of France is Paris.",
707
+ "Horses and cows are both animals"
708
+ ]
709
+ }'
710
+ ```
711
696
-Response:
712
+??? Response
713
698
-```bash
699
-{
700
- "id": "rerank-fae51b2b664d4ed38f5969b612edff77",
701
- "model": "BAAI/bge-reranker-base",
702
- "usage": {
703
- "total_tokens": 56
704
- },
705
- "results": [
706
- {
707
- "index": 1,
708
- "document": {
709
- "text": "The capital of France is Paris."
710
- },
711
- "relevance_score": 0.99853515625
712
- },
714
+ ```bash
715
{
714
- "index": 0,
715
- "document": {
716
- "text": "The capital of Brazil is Brasilia."
716
+ "id": "rerank-fae51b2b664d4ed38f5969b612edff77",
717
+ "model": "BAAI/bge-reranker-base",
718
+ "usage": {
719
+ "total_tokens": 56
720
},
718
- "relevance_score": 0.0005860328674316406
721
+ "results": [
722
+ {
723
+ "index": 1,
724
+ "document": {
725
+ "text": "The capital of France is Paris."
726
+ },
727
+ "relevance_score": 0.99853515625
728
+ },
729
+ {
730
+ "index": 0,
731
+ "document": {
732
+ "text": "The capital of Brazil is Brasilia."
733
+ },
734
+ "relevance_score": 0.0005860328674316406
735
+ }
736
+ ]
737
}
720
- ]
721
-}
722
-```
738
+ ```
739
740
#### Extra parameters
741
docs/usage/metrics.md
+23
-19
@@ -12,28 +12,32 @@ vllm serve unsloth/Llama-3.2-1B-Instruct
12
13
Then query the endpoint to get the latest metrics from the server:
14
15
-```console
16
-$ curl http://0.0.0.0:8000/metrics
17
-
18
-# HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
19
-# TYPE vllm:iteration_tokens_total histogram
20
-vllm:iteration_tokens_total_sum{model_name="unsloth/Llama-3.2-1B-Instruct"} 0.0
21
-vllm:iteration_tokens_total_bucket{le="1.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
22
-vllm:iteration_tokens_total_bucket{le="8.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
23
-vllm:iteration_tokens_total_bucket{le="16.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
24
-vllm:iteration_tokens_total_bucket{le="32.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
25
-vllm:iteration_tokens_total_bucket{le="64.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
26
-vllm:iteration_tokens_total_bucket{le="128.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
27
-vllm:iteration_tokens_total_bucket{le="256.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
28
-vllm:iteration_tokens_total_bucket{le="512.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
29
-...
30
-```
15
+??? Output
16
+
17
+ ```console
18
+ $ curl http://0.0.0.0:8000/metrics
19
+
20
+ # HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
21
+ # TYPE vllm:iteration_tokens_total histogram
22
+ vllm:iteration_tokens_total_sum{model_name="unsloth/Llama-3.2-1B-Instruct"} 0.0
23
+ vllm:iteration_tokens_total_bucket{le="1.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
24
+ vllm:iteration_tokens_total_bucket{le="8.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
25
+ vllm:iteration_tokens_total_bucket{le="16.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
26
+ vllm:iteration_tokens_total_bucket{le="32.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
27
+ vllm:iteration_tokens_total_bucket{le="64.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
28
+ vllm:iteration_tokens_total_bucket{le="128.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
29
+ vllm:iteration_tokens_total_bucket{le="256.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
30
+ vllm:iteration_tokens_total_bucket{le="512.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
31
+ ...
32
+ ```
33
34
The following metrics are exposed:
35
34
-```python
35
---8<-- "vllm/engine/metrics.py:metrics-definitions"
36
-```
36
+??? Code
37
+
38
+ ```python
39
+ --8<-- "vllm/engine/metrics.py:metrics-definitions"
40
+ ```
41
42
Note: when metrics are deprecated in version `X.Y`, they are hidden in version `X.Y+1`
43
but can be re-enabled using the `--show-hidden-metrics-for-version=X.Y` escape hatch,
docs/usage/troubleshooting.md
+91
-85
@@ -60,68 +60,70 @@ To identify the particular CUDA operation that causes the error, you can add `--
60
61
If GPU/CPU communication cannot be established, you can use the following Python script and follow the instructions below to confirm whether the GPU/CPU communication is working correctly.
62
63
-```python
64
-# Test PyTorch NCCL
65
-import torch
66
-import torch.distributed as dist
67
-dist.init_process_group(backend="nccl")
68
-local_rank = dist.get_rank() % torch.cuda.device_count()
69
-torch.cuda.set_device(local_rank)
70
-data = torch.FloatTensor([1,] * 128).to("cuda")
71
-dist.all_reduce(data, op=dist.ReduceOp.SUM)
72
-torch.cuda.synchronize()
73
-value = data.mean().item()
74
-world_size = dist.get_world_size()
75
-assert value == world_size, f"Expected {world_size}, got {value}"
76
-
77
-print("PyTorch NCCL is successful!")
78
-
79
-# Test PyTorch GLOO
80
-gloo_group = dist.new_group(ranks=list(range(world_size)), backend="gloo")
81
-cpu_data = torch.FloatTensor([1,] * 128)
82
-dist.all_reduce(cpu_data, op=dist.ReduceOp.SUM, group=gloo_group)
83
-value = cpu_data.mean().item()
84
-assert value == world_size, f"Expected {world_size}, got {value}"
85
-
86
-print("PyTorch GLOO is successful!")
87
-
88
-if world_size <= 1:
89
- exit()
90
-
91
-# Test vLLM NCCL, with cuda graph
92
-from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator
93
-
94
-pynccl = PyNcclCommunicator(group=gloo_group, device=local_rank)
95
-# pynccl is enabled by default for 0.6.5+,
96
-# but for 0.6.4 and below, we need to enable it manually.
97
-# keep the code for backward compatibility when because people
98
-# prefer to read the latest documentation.
99
-pynccl.disabled = False
100
-
101
-s = torch.cuda.Stream()
102
-with torch.cuda.stream(s):
103
- data.fill_(1)
104
- out = pynccl.all_reduce(data, stream=s)
105
- value = out.mean().item()
63
+??? Code
64
+
65
+ ```python
66
+ # Test PyTorch NCCL
67
+ import torch
68
+ import torch.distributed as dist
69
+ dist.init_process_group(backend="nccl")
70
+ local_rank = dist.get_rank() % torch.cuda.device_count()
71
+ torch.cuda.set_device(local_rank)
72
+ data = torch.FloatTensor([1,] * 128).to("cuda")
73
+ dist.all_reduce(data, op=dist.ReduceOp.SUM)
74
+ torch.cuda.synchronize()
75
+ value = data.mean().item()
76
+ world_size = dist.get_world_size()
77
assert value == world_size, f"Expected {world_size}, got {value}"
78
108
-print("vLLM NCCL is successful!")
79
+ print("PyTorch NCCL is successful!")
80
110
-g = torch.cuda.CUDAGraph()
111
-with torch.cuda.graph(cuda_graph=g, stream=s):
112
- out = pynccl.all_reduce(data, stream=torch.cuda.current_stream())
81
+ # Test PyTorch GLOO
82
+ gloo_group = dist.new_group(ranks=list(range(world_size)), backend="gloo")
83
+ cpu_data = torch.FloatTensor([1,] * 128)
84
+ dist.all_reduce(cpu_data, op=dist.ReduceOp.SUM, group=gloo_group)
85
+ value = cpu_data.mean().item()
86
+ assert value == world_size, f"Expected {world_size}, got {value}"
87
114
-data.fill_(1)
115
-g.replay()
116
-torch.cuda.current_stream().synchronize()
117
-value = out.mean().item()
118
-assert value == world_size, f"Expected {world_size}, got {value}"
88
+ print("PyTorch GLOO is successful!")
89
120
-print("vLLM NCCL with cuda graph is successful!")
90
+ if world_size <= 1:
91
+ exit()
92
122
-dist.destroy_process_group(gloo_group)
123
-dist.destroy_process_group()
124
-```
93
+ # Test vLLM NCCL, with cuda graph
94
+ from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator
95
+
96
+ pynccl = PyNcclCommunicator(group=gloo_group, device=local_rank)
97
+ # pynccl is enabled by default for 0.6.5+,
98
+ # but for 0.6.4 and below, we need to enable it manually.
99
+ # keep the code for backward compatibility when because people
100
+ # prefer to read the latest documentation.
101
+ pynccl.disabled = False
102
+
103
+ s = torch.cuda.Stream()
104
+ with torch.cuda.stream(s):
105
+ data.fill_(1)
106
+ out = pynccl.all_reduce(data, stream=s)
107
+ value = out.mean().item()
108
+ assert value == world_size, f"Expected {world_size}, got {value}"
109
+
110
+ print("vLLM NCCL is successful!")
111
+
112
+ g = torch.cuda.CUDAGraph()
113
+ with torch.cuda.graph(cuda_graph=g, stream=s):
114
+ out = pynccl.all_reduce(data, stream=torch.cuda.current_stream())
115
+
116
+ data.fill_(1)
117
+ g.replay()
118
+ torch.cuda.current_stream().synchronize()
119
+ value = out.mean().item()
120
+ assert value == world_size, f"Expected {world_size}, got {value}"
121
+
122
+ print("vLLM NCCL with cuda graph is successful!")
123
+
124
+ dist.destroy_process_group(gloo_group)
125
+ dist.destroy_process_group()
126
+ ```
127
128
If you are testing with a single node, adjust `--nproc-per-node` to the number of GPUs you want to use:
129
@@ -165,25 +167,27 @@ WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
167
168
or an error from Python that looks like this:
169
168
-```console
169
-RuntimeError:
170
- An attempt has been made to start a new process before the
171
- current process has finished its bootstrapping phase.
170
+??? Logs
171
173
- This probably means that you are not using fork to start your
174
- child processes and you have forgotten to use the proper idiom
175
- in the main module:
172
+ ```console
173
+ RuntimeError:
174
+ An attempt has been made to start a new process before the
175
+ current process has finished its bootstrapping phase.
176
177
- if __name__ == '__main__':
178
- freeze_support()
179
- ...
177
+ This probably means that you are not using fork to start your
178
+ child processes and you have forgotten to use the proper idiom
179
+ in the main module:
180
181
- The "freeze_support()" line can be omitted if the program
182
- is not going to be frozen to produce an executable.
181
+ if __name__ == '__main__':
182
+ freeze_support()
183
+ ...
184
184
- To fix this issue, refer to the "Safe importing of main module"
185
- section in https://docs.python.org/3/library/multiprocessing.html
186
-```
185
+ The "freeze_support()" line can be omitted if the program
186
+ is not going to be frozen to produce an executable.
187
+
188
+ To fix this issue, refer to the "Safe importing of main module"
189
+ section in https://docs.python.org/3/library/multiprocessing.html
190
+ ```
191
192
then you must update your Python code to guard usage of `vllm` behind a `if
193
__name__ == '__main__':` block. For example, instead of this:
@@ -207,20 +211,22 @@ if __name__ == '__main__':
211
212
vLLM heavily depends on `torch.compile` to optimize the model for better performance, which introduces the dependency on the `torch.compile` functionality and the `triton` library. By default, we use `torch.compile` to [optimize some functions](https://github.com/vllm-project/vllm/pull/10406) in the model. Before running vLLM, you can check if `torch.compile` is working as expected by running the following script:
213
210
-```python
211
-import torch
212
-
213
-@torch.compile
214
-def f(x):
215
- # a simple function to test torch.compile
216
- x = x + 1
217
- x = x * 2
218
- x = x.sin()
219
- return x
220
-
221
-x = torch.randn(4, 4).cuda()
222
-print(f(x))
223
-```
214
+??? Code
215
+
216
+ ```python
217
+ import torch
218
+
219
+ @torch.compile
220
+ def f(x):
221
+ # a simple function to test torch.compile
222
+ x = x + 1
223
+ x = x * 2
224
+ x = x.sin()
225
+ return x
226
+
227
+ x = torch.randn(4, 4).cuda()
228
+ print(f(x))
229
+ ```
230
231
If it raises errors from `torch/_inductor` directory, usually it means you have a custom `triton` library that is not compatible with the version of PyTorch you are using. See [this issue](https://github.com/vllm-project/vllm/issues/12219) for example.
232
docs/usage/usage_stats.md
+32
-30
@@ -10,36 +10,38 @@ The list of data collected by the latest version of vLLM can be found here: <gh-
10
11
Here is an example as of v0.4.0:
12
13
-```json
14
-{
15
- "uuid": "fbe880e9-084d-4cab-a395-8984c50f1109",
16
- "provider": "GCP",
17
- "num_cpu": 24,
18
- "cpu_type": "Intel(R) Xeon(R) CPU @ 2.20GHz",
19
- "cpu_family_model_stepping": "6,85,7",
20
- "total_memory": 101261135872,
21
- "architecture": "x86_64",
22
- "platform": "Linux-5.10.0-28-cloud-amd64-x86_64-with-glibc2.31",
23
- "gpu_count": 2,
24
- "gpu_type": "NVIDIA L4",
25
- "gpu_memory_per_device": 23580639232,
26
- "model_architecture": "OPTForCausalLM",
27
- "vllm_version": "0.3.2+cu123",
28
- "context": "LLM_CLASS",
29
- "log_time": 1711663373492490000,
30
- "source": "production",
31
- "dtype": "torch.float16",
32
- "tensor_parallel_size": 1,
33
- "block_size": 16,
34
- "gpu_memory_utilization": 0.9,
35
- "quantization": null,
36
- "kv_cache_dtype": "auto",
37
- "enable_lora": false,
38
- "enable_prefix_caching": false,
39
- "enforce_eager": false,
40
- "disable_custom_all_reduce": true
41
-}
42
-```
13
+??? Output
14
+
15
+ ```json
16
+ {
17
+ "uuid": "fbe880e9-084d-4cab-a395-8984c50f1109",
18
+ "provider": "GCP",
19
+ "num_cpu": 24,
20
+ "cpu_type": "Intel(R) Xeon(R) CPU @ 2.20GHz",
21
+ "cpu_family_model_stepping": "6,85,7",
22
+ "total_memory": 101261135872,
23
+ "architecture": "x86_64",
24
+ "platform": "Linux-5.10.0-28-cloud-amd64-x86_64-with-glibc2.31",
25
+ "gpu_count": 2,
26
+ "gpu_type": "NVIDIA L4",
27
+ "gpu_memory_per_device": 23580639232,
28
+ "model_architecture": "OPTForCausalLM",
29
+ "vllm_version": "0.3.2+cu123",
30
+ "context": "LLM_CLASS",
31
+ "log_time": 1711663373492490000,
32
+ "source": "production",
33
+ "dtype": "torch.float16",
34
+ "tensor_parallel_size": 1,
35
+ "block_size": 16,
36
+ "gpu_memory_utilization": 0.9,
37
+ "quantization": null,
38
+ "kv_cache_dtype": "auto",
39
+ "enable_lora": false,
40
+ "enable_prefix_caching": false,
41
+ "enforce_eager": false,
42
+ "disable_custom_all_reduce": true
43
+ }
44
+ ```
45
46
You can preview the collected data by running the following command:
47