@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / dd6a3a02cb
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 00:45 UTC

[Doc] Convert docs to use colon fences (#12471)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Jan 29, 2025 at 03:38 UTC dd6a3a02cb3bf2a7bc6cb84c85dcd57c6eaf2bf9
68 files changed +2091 -2080
docs/requirements-docs.txt
+2 -2
@@ -1,10 +1,10 @@
1 sphinx==6.2.1
2 +sphinx-argparse==0.4.0
3 sphinx-book-theme==1.0.1
4 sphinx-copybutton==0.5.2
4 -myst-parser==3.0.1
5 -sphinx-argparse==0.4.0
5 sphinx-design==0.6.1
6 sphinx-togglebutton==0.3.2
7 +myst-parser==3.0.1
8 msgspec
9 cloudpickle
10
docs/source/api/engine/index.md
+2 -2
@@ -8,10 +8,10 @@
8 .. currentmodule:: vllm.engine
9 ```
10
11 -```{toctree}
11 +:::{toctree}
12 :caption: Engines
13 :maxdepth: 2
14
15 llm_engine
16 async_llm_engine
17 -```
17 +:::
docs/source/api/model/index.md
+2 -2
@@ -2,10 +2,10 @@
2
3 ## Submodules
4
5 -```{toctree}
5 +:::{toctree}
6 :maxdepth: 1
7
8 interfaces_base
9 interfaces
10 adapters
11 -```
11 +:::
docs/source/api/multimodal/index.md
+2 -2
@@ -17,7 +17,7 @@ Looking to add your own multi-modal model? Please follow the instructions listed
17
18 ## Submodules
19
20 -```{toctree}
20 +:::{toctree}
21 :maxdepth: 1
22
23 inputs
@@ -25,4 +25,4 @@ parse
25 processing
26 profiling
27 registry
28 -```
28 +:::
docs/source/api/offline_inference/index.md
+2 -2
@@ -1,9 +1,9 @@
1 # Offline Inference
2
3 -```{toctree}
3 +:::{toctree}
4 :caption: Contents
5 :maxdepth: 1
6
7 llm
8 llm_inputs
9 -```
9 +:::
docs/source/contributing/dockerfile/dockerfile.md
+2 -2
@@ -17,11 +17,11 @@ The edges of the build graph represent:
17
18 - `RUN --mount=(.\*)from=...` dependencies (with a dotted line and an empty diamond arrow head)
19
20 - > ```{figure} /assets/contributing/dockerfile-stages-dependency.png
20 + > :::{figure} /assets/contributing/dockerfile-stages-dependency.png
21 > :align: center
22 > :alt: query
23 > :width: 100%
24 - > ```
24 + > :::
25 >
26 > Made using: <https://github.com/patrickhoefler/dockerfilegraph>
27 >
docs/source/contributing/model/basic.md
+4 -4
@@ -10,9 +10,9 @@ First, clone the PyTorch model code from the source repository.
10 For instance, vLLM's [OPT model](gh-file:vllm/model_executor/models/opt.py) was adapted from
11 HuggingFace's [modeling_opt.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py) file.
12
13 -```{warning}
13 +:::{warning}
14 Make sure to review and adhere to the original code's copyright and licensing terms!
15 -```
15 +:::
16
17 ## 2. Make your code compatible with vLLM
18
@@ -80,10 +80,10 @@ def forward(
80 ...
81 ```
82
83 -```{note}
83 +:::{note}
84 Currently, vLLM supports the basic multi-head attention mechanism and its variant with rotary positional embeddings.
85 If your model employs a different attention mechanism, you will need to implement a new attention layer in vLLM.
86 -```
86 +:::
87
88 For reference, check out our [Llama implementation](gh-file:vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out <gh-dir:vllm/model_executor/models> for more examples.
89
docs/source/contributing/model/index.md
+6 -6
@@ -4,7 +4,7 @@
4
5 This section provides more information on how to integrate a [PyTorch](https://pytorch.org/) model into vLLM.
6
7 -```{toctree}
7 +:::{toctree}
8 :caption: Contents
9 :maxdepth: 1
10
@@ -12,16 +12,16 @@ basic
12 registration
13 tests
14 multimodal
15 -```
15 +:::
16
17 -```{note}
17 +:::{note}
18 The complexity of adding a new model depends heavily on the model's architecture.
19 The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM.
20 However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
21 -```
21 +:::
22
23 -```{tip}
23 +:::{tip}
24 If you are encountering issues while integrating your model into vLLM, feel free to open a [GitHub issue](https://github.com/vllm-project/vllm/issues)
25 or ask on our [developer slack](https://slack.vllm.ai).
26 We will be happy to help you out!
27 -```
27 +:::
docs/source/contributing/model/multimodal.md
+16 -16
@@ -48,9 +48,9 @@ Further update the model as follows:
48 return vision_embeddings
49 ```
50
51 - ```{important}
51 + :::{important}
52 The returned `multimodal_embeddings` must be either a **3D {class}`torch.Tensor`** of shape `(num_items, feature_size, hidden_size)`, or a **list / tuple of 2D {class}`torch.Tensor`'s** of shape `(feature_size, hidden_size)`, so that `multimodal_embeddings[i]` retrieves the embeddings generated from the `i`-th multimodal data item (e.g, image) of the request.
53 - ```
53 + :::
54
55 - Implement {meth}`~vllm.model_executor.models.interfaces.SupportsMultiModal.get_input_embeddings` to merge `multimodal_embeddings` with text embeddings from the `input_ids`. If input processing for the model is implemented correctly (see sections below), then you can leverage the utility function we provide to easily merge the embeddings.
56
@@ -89,10 +89,10 @@ Further update the model as follows:
89 + class YourModelForImage2Seq(nn.Module, SupportsMultiModal):
90 ```
91
92 - ```{note}
92 + :::{note}
93 The model class does not have to be named {code}`*ForCausalLM`.
94 Check out [the HuggingFace Transformers documentation](https://huggingface.co/docs/transformers/model_doc/auto#multimodal) for some examples.
95 - ```
95 + :::
96
97 ## 2. Specify processing information
98
@@ -120,8 +120,8 @@ When calling the model, the output embeddings from the visual encoder are assign
120 containing placeholder feature tokens. Therefore, the number of placeholder feature tokens should be equal
121 to the size of the output embeddings.
122
123 -::::{tab-set}
124 -:::{tab-item} Basic example: LLaVA
123 +:::::{tab-set}
124 +::::{tab-item} Basic example: LLaVA
125 :sync: llava
126
127 Looking at the code of HF's `LlavaForConditionalGeneration`:
@@ -254,12 +254,12 @@ def get_mm_max_tokens_per_item(self, seq_len: int) -> Mapping[str, int]:
254 return {"image": self.get_max_image_tokens()}
255 ```
256
257 -```{note}
257 +:::{note}
258 Our [actual code](gh-file:vllm/model_executor/models/llava.py) is more abstracted to support vision encoders other than CLIP.
259 -```
260 -
259 :::
260 +
261 ::::
262 +:::::
263
264 ## 3. Specify dummy inputs
265
@@ -315,17 +315,17 @@ def get_dummy_processor_inputs(
315 Afterwards, create a subclass of {class}`~vllm.multimodal.processing.BaseMultiModalProcessor`
316 to fill in the missing details about HF processing.
317
318 -```{seealso}
318 +:::{seealso}
319 [Multi-Modal Data Processing](#mm-processing)
320 -```
320 +:::
321
322 ### Multi-modal fields
323
324 Override {class}`~vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config` to
325 return a schema of the tensors outputted by the HF processor that are related to the input multi-modal items.
326
327 -::::{tab-set}
328 -:::{tab-item} Basic example: LLaVA
327 +:::::{tab-set}
328 +::::{tab-item} Basic example: LLaVA
329 :sync: llava
330
331 Looking at the model's `forward` method:
@@ -367,13 +367,13 @@ def _get_mm_fields_config(
367 )
368 ```
369
370 -```{note}
370 +:::{note}
371 Our [actual code](gh-file:vllm/model_executor/models/llava.py) additionally supports
372 pre-computed image embeddings, which can be passed to be model via the `image_embeds` argument.
373 -```
374 -
373 :::
374 +
375 ::::
376 +:::::
377
378 ### Prompt replacements
379
docs/source/contributing/model/registration.md
+8 -8
@@ -17,17 +17,17 @@ After you have implemented your model (see [tutorial](#new-model-basic)), put it
17 Then, add your model class to `_VLLM_MODELS` in <gh-file:vllm/model_executor/models/registry.py> so that it is automatically registered upon importing vLLM.
18 Finally, update our [list of supported models](#supported-models) to promote your model!
19
20 -```{important}
20 +:::{important}
21 The list of models in each section should be maintained in alphabetical order.
22 -```
22 +:::
23
24 ## Out-of-tree models
25
26 You can load an external model using a plugin without modifying the vLLM codebase.
27
28 -```{seealso}
28 +:::{seealso}
29 [vLLM's Plugin System](#plugin-system)
30 -```
30 +:::
31
32 To register the model, use the following code:
33
@@ -45,11 +45,11 @@ from vllm import ModelRegistry
45 ModelRegistry.register_model("YourModelForCausalLM", "your_code:YourModelForCausalLM")
46 ```
47
48 -```{important}
48 +:::{important}
49 If your model is a multimodal model, ensure the model class implements the {class}`~vllm.model_executor.models.interfaces.SupportsMultiModal` interface.
50 Read more about that [here](#supports-multimodal).
51 -```
51 +:::
52
53 -```{note}
53 +:::{note}
54 Although you can directly put these code snippets in your script using `vllm.LLM`, the recommended way is to place these snippets in a vLLM plugin. This ensures compatibility with various vLLM features like distributed inference and the API server.
55 -```
55 +:::
docs/source/contributing/model/tests.md
+4 -4
@@ -14,14 +14,14 @@ Without them, the CI for your PR will fail.
14 Include an example HuggingFace repository for your model in <gh-file:tests/models/registry.py>.
15 This enables a unit test that loads dummy weights to ensure that the model can be initialized in vLLM.
16
17 -```{important}
17 +:::{important}
18 The list of models in each section should be maintained in alphabetical order.
19 -```
19 +:::
20
21 -```{tip}
21 +:::{tip}
22 If your model requires a development version of HF Transformers, you can set
23 `min_transformers_version` to skip the test in CI until the model is released.
24 -```
24 +:::
25
26 ## Optional Tests
27
docs/source/contributing/overview.md
+6 -6
@@ -35,17 +35,17 @@ pre-commit run --all-files
35 pytest tests/
36 ```
37
38 -```{note}
38 +:::{note}
39 Currently, the repository is not fully checked by `mypy`.
40 -```
40 +:::
41
42 ## Issues
43
44 If you encounter a bug or have a feature request, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible.
45
46 -```{important}
46 +:::{important}
47 If you discover a security vulnerability, please follow the instructions [here](gh-file:SECURITY.md#reporting-a-vulnerability).
48 -```
48 +:::
49
50 ## Pull Requests & Code Reviews
51
@@ -81,9 +81,9 @@ appropriately to indicate the type of change. Please use one of the following:
81 - `[Misc]` for PRs that do not fit the above categories. Please use this
82 sparingly.
83
84 -```{note}
84 +:::{note}
85 If the PR spans more than one category, please include all relevant prefixes.
86 -```
86 +:::
87
88 ### Code Quality
89
docs/source/contributing/profiling/profiling_index.md
+6 -6
@@ -6,21 +6,21 @@ The OpenAI server also needs to be started with the `VLLM_TORCH_PROFILER_DIR` en
6
7 When using `benchmarks/benchmark_serving.py`, you can enable profiling by passing the `--profile` flag.
8
9 -```{warning}
9 +:::{warning}
10 Only enable profiling in a development environment.
11 -```
11 +:::
12
13 Traces can be visualized using <https://ui.perfetto.dev/>.
14
15 -```{tip}
15 +:::{tip}
16 Only send a few requests through vLLM when profiling, as the traces can get quite large. Also, no need to untar the traces, they can be viewed directly.
17 -```
17 +:::
18
19 -```{tip}
19 +:::{tip}
20 To stop the profiler - it flushes out all the profile trace files to the directory. This takes time, for example for about 100 requests worth of data for a llama 70b, it takes about 10 minutes to flush out on a H100.
21 Set the env variable VLLM_RPC_TIMEOUT to a big number before you start the server. Say something like 30 minutes.
22 `export VLLM_RPC_TIMEOUT=1800000`
23 -```
23 +:::
24
25 ## Example commands and usage
26
docs/source/deployment/docker.md
+8 -8
@@ -21,11 +21,11 @@ $ docker run --runtime nvidia --gpus all \
21
22 You can add any other <project:#engine-args> you need after the image tag (`vllm/vllm-openai:latest`).
23
24 -```{note}
24 +:::{note}
25 You can either use the `ipc=host` flag or `--shm-size` flag to allow the
26 container to access the host's shared memory. vLLM uses PyTorch, which uses shared
27 memory to share data between processes under the hood, particularly for tensor parallel inference.
28 -```
28 +:::
29
30 (deployment-docker-build-image-from-source)=
31
@@ -38,25 +38,25 @@ You can build and run vLLM from source via the provided <gh-file:Dockerfile>. To
38 DOCKER_BUILDKIT=1 docker build . --target vllm-openai --tag vllm/vllm-openai
39 ```
40
41 -```{note}
41 +:::{note}
42 By default vLLM will build for all GPU types for widest distribution. If you are just building for the
43 current GPU type the machine is running on, you can add the argument `--build-arg torch_cuda_arch_list=""`
44 for vLLM to find the current GPU type and build for that.
45
46 If you are using Podman instead of Docker, you might need to disable SELinux labeling by
47 adding `--security-opt label=disable` when running `podman build` command to avoid certain [existing issues](https://github.com/containers/buildah/discussions/4184).
48 -```
48 +:::
49
50 ## Building for Arm64/aarch64
51
52 A docker container can be built for aarch64 systems such as the Nvidia Grace-Hopper. At time of this writing, this requires the use
53 of PyTorch Nightly and should be considered **experimental**. Using the flag `--platform "linux/arm64"` will attempt to build for arm64.
54
55 -```{note}
55 +:::{note}
56 Multiple modules must be compiled, so this process can take a while. Recommend using `--build-arg max_jobs=` & `--build-arg nvcc_threads=`
57 flags to speed up build process. However, ensure your `max_jobs` is substantially larger than `nvcc_threads` to get the most benefits.
58 Keep an eye on memory usage with parallel jobs as it can be substantial (see example below).
59 -```
59 +:::
60
61 ```console
62 # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
@@ -85,6 +85,6 @@ $ docker run --runtime nvidia --gpus all \
85
86 The argument `vllm/vllm-openai` specifies the image to run, and should be replaced with the name of the custom-built image (the `-t` tag from the build command).
87
88 -```{note}
88 +:::{note}
89 **For version 0.4.1 and 0.4.2 only** - the vLLM docker images under these versions are supposed to be run under the root user since a library under the root user's home directory, i.e. `/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` is required to be loaded during runtime. If you are running the container under a different user, you may need to first change the permissions of the library (and all the parent directories) to allow the user to access it, then run vLLM with environment variable `VLLM_NCCL_SO_PATH=/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` .
90 -```
90 +:::
docs/source/deployment/frameworks/cerebrium.md
+2 -2
@@ -2,11 +2,11 @@
2
3 # Cerebrium
4
5 -```{raw} html
5 +:::{raw} html
6 <p align="center">
7 <img src="https://i.ibb.co/hHcScTT/Screenshot-2024-06-13-at-10-14-54.png" alt="vLLM_plus_cerebrium"/>
8 </p>
9 -```
9 +:::
10
11 vLLM can be run on a cloud based GPU machine with [Cerebrium](https://www.cerebrium.ai/), a serverless AI infrastructure platform that makes it easier for companies to build and deploy AI based applications.
12
docs/source/deployment/frameworks/dstack.md
+4 -4
@@ -2,11 +2,11 @@
2
3 # dstack
4
5 -```{raw} html
5 +:::{raw} html
6 <p align="center">
7 <img src="https://i.ibb.co/71kx6hW/vllm-dstack.png" alt="vLLM_plus_dstack"/>
8 </p>
9 -```
9 +:::
10
11 vLLM can be run on a cloud based GPU machine with [dstack](https://dstack.ai/), an open-source framework for running LLMs on any cloud. This tutorial assumes that you have already configured credentials, gateway, and GPU quotas on your cloud environment.
12
@@ -97,6 +97,6 @@ completion = client.chat.completions.create(
97 print(completion.choices[0].message.content)
98 ```
99
100 -```{note}
100 +:::{note}
101 dstack automatically handles authentication on the gateway using dstack's tokens. Meanwhile, if you don't want to configure a gateway, you can provision dstack `Task` instead of `Service`. The `Task` is for development purpose only. If you want to know more about hands-on materials how to serve vLLM using dstack, check out [this repository](https://github.com/dstackai/dstack-examples/tree/main/deployment/vllm)
102 -```
102 +:::
docs/source/deployment/frameworks/helm.md
+204 -204
@@ -38,213 +38,213 @@ chart **including persistent volumes** and deletes the release.
38
39 ## Architecture
40
41 -```{image} /assets/deployment/architecture_helm_deployment.png
42 -```
41 +:::{image} /assets/deployment/architecture_helm_deployment.png
42 +:::
43
44 ## Values
45
46 -```{list-table}
46 +:::{list-table}
47 :widths: 25 25 25 25
48 :header-rows: 1
49
50 -* - Key
51 - - Type
52 - - Default
53 - - Description
54 -* - autoscaling
55 - - object
56 - - {"enabled":false,"maxReplicas":100,"minReplicas":1,"targetCPUUtilizationPercentage":80}
57 - - Autoscaling configuration
58 -* - autoscaling.enabled
59 - - bool
60 - - false
61 - - Enable autoscaling
62 -* - autoscaling.maxReplicas
63 - - int
64 - - 100
65 - - Maximum replicas
66 -* - autoscaling.minReplicas
67 - - int
68 - - 1
69 - - Minimum replicas
70 -* - autoscaling.targetCPUUtilizationPercentage
71 - - int
72 - - 80
73 - - Target CPU utilization for autoscaling
74 -* - configs
75 - - object
76 - - {}
77 - - Configmap
78 -* - containerPort
79 - - int
80 - - 8000
81 - - Container port
82 -* - customObjects
83 - - list
84 - - []
85 - - Custom Objects configuration
86 -* - deploymentStrategy
87 - - object
88 - - {}
89 - - Deployment strategy configuration
90 -* - externalConfigs
91 - - list
92 - - []
93 - - External configuration
94 -* - extraContainers
95 - - list
96 - - []
97 - - Additional containers configuration
98 -* - extraInit
99 - - object
100 - - {"pvcStorage":"1Gi","s3modelpath":"relative_s3_model_path/opt-125m", "awsEc2MetadataDisabled": true}
101 - - Additional configuration for the init container
102 -* - extraInit.pvcStorage
103 - - string
104 - - "50Gi"
105 - - Storage size of the s3
106 -* - extraInit.s3modelpath
107 - - string
108 - - "relative_s3_model_path/opt-125m"
109 - - Path of the model on the s3 which hosts model weights and config files
110 -* - extraInit.awsEc2MetadataDisabled
111 - - boolean
112 - - true
113 - - Disables the use of the Amazon EC2 instance metadata service
114 -* - extraPorts
115 - - list
116 - - []
117 - - Additional ports configuration
118 -* - gpuModels
119 - - list
120 - - ["TYPE_GPU_USED"]
121 - - Type of gpu used
122 -* - image
123 - - object
124 - - {"command":["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"],"repository":"vllm/vllm-openai","tag":"latest"}
125 - - Image configuration
126 -* - image.command
127 - - list
128 - - ["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"]
129 - - Container launch command
130 -* - image.repository
131 - - string
132 - - "vllm/vllm-openai"
133 - - Image repository
134 -* - image.tag
135 - - string
136 - - "latest"
137 - - Image tag
138 -* - livenessProbe
139 - - object
140 - - {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":15,"periodSeconds":10}
141 - - Liveness probe configuration
142 -* - livenessProbe.failureThreshold
143 - - int
144 - - 3
145 - - Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not alive
146 -* - livenessProbe.httpGet
147 - - object
148 - - {"path":"/health","port":8000}
149 - - Configuration of the Kubelet http request on the server
150 -* - livenessProbe.httpGet.path
151 - - string
152 - - "/health"
153 - - Path to access on the HTTP server
154 -* - livenessProbe.httpGet.port
155 - - int
156 - - 8000
157 - - Name or number of the port to access on the container, on which the server is listening
158 -* - livenessProbe.initialDelaySeconds
159 - - int
160 - - 15
161 - - Number of seconds after the container has started before liveness probe is initiated
162 -* - livenessProbe.periodSeconds
163 - - int
164 - - 10
165 - - How often (in seconds) to perform the liveness probe
166 -* - maxUnavailablePodDisruptionBudget
167 - - string
168 - - ""
169 - - Disruption Budget Configuration
170 -* - readinessProbe
171 - - object
172 - - {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":5,"periodSeconds":5}
173 - - Readiness probe configuration
174 -* - readinessProbe.failureThreshold
175 - - int
176 - - 3
177 - - Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not ready
178 -* - readinessProbe.httpGet
179 - - object
180 - - {"path":"/health","port":8000}
181 - - Configuration of the Kubelet http request on the server
182 -* - readinessProbe.httpGet.path
183 - - string
184 - - "/health"
185 - - Path to access on the HTTP server
186 -* - readinessProbe.httpGet.port
187 - - int
188 - - 8000
189 - - Name or number of the port to access on the container, on which the server is listening
190 -* - readinessProbe.initialDelaySeconds
191 - - int
192 - - 5
193 - - Number of seconds after the container has started before readiness probe is initiated
194 -* - readinessProbe.periodSeconds
195 - - int
196 - - 5
197 - - How often (in seconds) to perform the readiness probe
198 -* - replicaCount
199 - - int
200 - - 1
201 - - Number of replicas
202 -* - resources
203 - - object
204 - - {"limits":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1},"requests":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1}}
205 - - Resource configuration
206 -* - resources.limits."nvidia.com/gpu"
207 - - int
208 - - 1
209 - - Number of gpus used
210 -* - resources.limits.cpu
211 - - int
212 - - 4
213 - - Number of CPUs
214 -* - resources.limits.memory
215 - - string
216 - - "16Gi"
217 - - CPU memory configuration
218 -* - resources.requests."nvidia.com/gpu"
219 - - int
220 - - 1
221 - - Number of gpus used
222 -* - resources.requests.cpu
223 - - int
224 - - 4
225 - - Number of CPUs
226 -* - resources.requests.memory
227 - - string
228 - - "16Gi"
229 - - CPU memory configuration
230 -* - secrets
231 - - object
232 - - {}
233 - - Secrets configuration
234 -* - serviceName
235 - - string
236 - -
237 - - Service name
238 -* - servicePort
239 - - int
240 - - 80
241 - - Service port
242 -* - labels.environment
243 - - string
244 - - test
245 - - Environment name
246 -* - labels.release
247 - - string
248 - - test
249 - - Release name
250 -```
50 +- * Key
51 + * Type
52 + * Default
53 + * Description
54 +- * autoscaling
55 + * object
56 + * {"enabled":false,"maxReplicas":100,"minReplicas":1,"targetCPUUtilizationPercentage":80}
57 + * Autoscaling configuration
58 +- * autoscaling.enabled
59 + * bool
60 + * false
61 + * Enable autoscaling
62 +- * autoscaling.maxReplicas
63 + * int
64 + * 100
65 + * Maximum replicas
66 +- * autoscaling.minReplicas
67 + * int
68 + * 1
69 + * Minimum replicas
70 +- * autoscaling.targetCPUUtilizationPercentage
71 + * int
72 + * 80
73 + * Target CPU utilization for autoscaling
74 +- * configs
75 + * object
76 + * {}
77 + * Configmap
78 +- * containerPort
79 + * int
80 + * 8000
81 + * Container port
82 +- * customObjects
83 + * list
84 + * []
85 + * Custom Objects configuration
86 +- * deploymentStrategy
87 + * object
88 + * {}
89 + * Deployment strategy configuration
90 +- * externalConfigs
91 + * list
92 + * []
93 + * External configuration
94 +- * extraContainers
95 + * list
96 + * []
97 + * Additional containers configuration
98 +- * extraInit
99 + * object
100 + * {"pvcStorage":"1Gi","s3modelpath":"relative_s3_model_path/opt-125m", "awsEc2MetadataDisabled": true}
101 + * Additional configuration for the init container
102 +- * extraInit.pvcStorage
103 + * string
104 + * "50Gi"
105 + * Storage size of the s3
106 +- * extraInit.s3modelpath
107 + * string
108 + * "relative_s3_model_path/opt-125m"
109 + * Path of the model on the s3 which hosts model weights and config files
110 +- * extraInit.awsEc2MetadataDisabled
111 + * boolean
112 + * true
113 + * Disables the use of the Amazon EC2 instance metadata service
114 +- * extraPorts
115 + * list
116 + * []
117 + * Additional ports configuration
118 +- * gpuModels
119 + * list
120 + * ["TYPE_GPU_USED"]
121 + * Type of gpu used
122 +- * image
123 + * object
124 + * {"command":["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"],"repository":"vllm/vllm-openai","tag":"latest"}
125 + * Image configuration
126 +- * image.command
127 + * list
128 + * ["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"]
129 + * Container launch command
130 +- * image.repository
131 + * string
132 + * "vllm/vllm-openai"
133 + * Image repository
134 +- * image.tag
135 + * string
136 + * "latest"
137 + * Image tag
138 +- * livenessProbe
139 + * object
140 + * {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":15,"periodSeconds":10}
141 + * Liveness probe configuration
142 +- * livenessProbe.failureThreshold
143 + * int
144 + * 3
145 + * Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not alive
146 +- * livenessProbe.httpGet
147 + * object
148 + * {"path":"/health","port":8000}
149 + * Configuration of the Kubelet http request on the server
150 +- * livenessProbe.httpGet.path
151 + * string
152 + * "/health"
153 + * Path to access on the HTTP server
154 +- * livenessProbe.httpGet.port
155 + * int
156 + * 8000
157 + * Name or number of the port to access on the container, on which the server is listening
158 +- * livenessProbe.initialDelaySeconds
159 + * int
160 + * 15
161 + * Number of seconds after the container has started before liveness probe is initiated
162 +- * livenessProbe.periodSeconds
163 + * int
164 + * 10
165 + * How often (in seconds) to perform the liveness probe
166 +- * maxUnavailablePodDisruptionBudget
167 + * string
168 + * ""
169 + * Disruption Budget Configuration
170 +- * readinessProbe
171 + * object
172 + * {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":5,"periodSeconds":5}
173 + * Readiness probe configuration
174 +- * readinessProbe.failureThreshold
175 + * int
176 + * 3
177 + * Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not ready
178 +- * readinessProbe.httpGet
179 + * object
180 + * {"path":"/health","port":8000}
181 + * Configuration of the Kubelet http request on the server
182 +- * readinessProbe.httpGet.path
183 + * string
184 + * "/health"
185 + * Path to access on the HTTP server
186 +- * readinessProbe.httpGet.port
187 + * int
188 + * 8000
189 + * Name or number of the port to access on the container, on which the server is listening
190 +- * readinessProbe.initialDelaySeconds
191 + * int
192 + * 5
193 + * Number of seconds after the container has started before readiness probe is initiated
194 +- * readinessProbe.periodSeconds
195 + * int
196 + * 5
197 + * How often (in seconds) to perform the readiness probe
198 +- * replicaCount
199 + * int
200 + * 1
201 + * Number of replicas
202 +- * resources
203 + * object
204 + * {"limits":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1},"requests":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1}}
205 + * Resource configuration
206 +- * resources.limits."nvidia.com/gpu"
207 + * int
208 + * 1
209 + * Number of gpus used
210 +- * resources.limits.cpu
211 + * int
212 + * 4
213 + * Number of CPUs
214 +- * resources.limits.memory
215 + * string
216 + * "16Gi"
217 + * CPU memory configuration
218 +- * resources.requests."nvidia.com/gpu"
219 + * int
220 + * 1
221 + * Number of gpus used
222 +- * resources.requests.cpu
223 + * int
224 + * 4
225 + * Number of CPUs
226 +- * resources.requests.memory
227 + * string
228 + * "16Gi"
229 + * CPU memory configuration
230 +- * secrets
231 + * object
232 + * {}
233 + * Secrets configuration
234 +- * serviceName
235 + * string
236 + *
237 + * Service name
238 +- * servicePort
239 + * int
240 + * 80
241 + * Service port
242 +- * labels.environment
243 + * string
244 + * test
245 + * Environment name
246 +- * labels.release
247 + * string
248 + * test
249 + * Release name
250 +:::
docs/source/deployment/frameworks/index.md
+2 -2
@@ -1,6 +1,6 @@
1 # Using other frameworks
2
3 -```{toctree}
3 +:::{toctree}
4 :maxdepth: 1
5
6 bentoml
@@ -11,4 +11,4 @@ lws
11 modal
12 skypilot
13 triton
14 -```
14 +:::
docs/source/deployment/frameworks/skypilot.md
+18 -18
@@ -2,11 +2,11 @@
2
3 # SkyPilot
4
5 -```{raw} html
5 +:::{raw} html
6 <p align="center">
7 <img src="https://imgur.com/yxtzPEu.png" alt="vLLM"/>
8 </p>
9 -```
9 +:::
10
11 vLLM can be **run and scaled to multiple service replicas on clouds and Kubernetes** with [SkyPilot](https://github.com/skypilot-org/skypilot), an open-source framework for running LLMs on any cloud. More examples for various open models, such as Llama-3, Mixtral, etc, can be found in [SkyPilot AI gallery](https://skypilot.readthedocs.io/en/latest/gallery/index.html).
12
@@ -104,10 +104,10 @@ service:
104 max_completion_tokens: 1
105 ```
106
107 -```{raw} html
107 +:::{raw} html
108 <details>
109 <summary>Click to see the full recipe YAML</summary>
110 -```
110 +:::
111
112 ```yaml
113 service:
@@ -153,9 +153,9 @@ run: |
153 2>&1 | tee api_server.log
154 ```
155
156 -```{raw} html
156 +:::{raw} html
157 </details>
158 -```
158 +:::
159
160 Start the serving the Llama-3 8B model on multiple replicas:
161
@@ -169,10 +169,10 @@ Wait until the service is ready:
169 watch -n10 sky serve status vllm
170 ```
171
172 -```{raw} html
172 +:::{raw} html
173 <details>
174 <summary>Example outputs:</summary>
175 -```
175 +:::
176
177 ```console
178 Services
@@ -185,9 +185,9 @@ vllm 1 1 xx.yy.zz.121 18 mins ago 1x GCP([Spot]{'L4': 1}) R
185 vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4
186 ```
187
188 -```{raw} html
188 +:::{raw} html
189 </details>
190 -```
190 +:::
191
192 After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:
193
@@ -223,10 +223,10 @@ service:
223
224 This will scale the service up to when the QPS exceeds 2 for each replica.
225
226 -```{raw} html
226 +:::{raw} html
227 <details>
228 <summary>Click to see the full recipe YAML</summary>
229 -```
229 +:::
230
231 ```yaml
232 service:
@@ -275,9 +275,9 @@ run: |
275 2>&1 | tee api_server.log
276 ```
277
278 -```{raw} html
278 +:::{raw} html
279 </details>
280 -```
280 +:::
281
282 To update the service with the new config:
283
@@ -295,10 +295,10 @@ sky serve down vllm
295
296 It is also possible to access the Llama-3 service with a separate GUI frontend, so the user requests send to the GUI will be load-balanced across replicas.
297
298 -```{raw} html
298 +:::{raw} html
299 <details>
300 <summary>Click to see the full GUI YAML</summary>
301 -```
301 +:::
302
303 ```yaml
304 envs:
@@ -328,9 +328,9 @@ run: |
328 --stop-token-ids 128009,128001 | tee ~/gradio.log
329 ```
330
331 -```{raw} html
331 +:::{raw} html
332 </details>
333 -```
333 +:::
334
335 1. Start the chat web UI:
336
docs/source/deployment/integrations/index.md
+2 -2
@@ -1,9 +1,9 @@
1 # External Integrations
2
3 -```{toctree}
3 +:::{toctree}
4 :maxdepth: 1
5
6 kserve
7 kubeai
8 llamastack
9 -```
9 +:::
docs/source/deployment/nginx.md
+2 -2
@@ -105,9 +105,9 @@ docker run -itd --ipc host --privileged --network vllm_nginx --gpus all --shm-si
105 docker run -itd --ipc host --privileged --network vllm_nginx --gpus all --shm-size=10.24gb -v $hf_cache_dir:/root/.cache/huggingface/ -p 8082:8000 --name vllm1 vllm --model meta-llama/Llama-2-7b-chat-hf
106 ```
107
108 -```{note}
108 +:::{note}
109 If you are behind proxy, you can pass the proxy settings to the docker run command via `-e http_proxy=$http_proxy -e https_proxy=$https_proxy`.
110 -```
110 +:::
111
112 (nginxloadbalancer-nginx-launch-nginx)=
113
docs/source/design/arch_overview.md
+10 -10
@@ -4,19 +4,19 @@
4
5 This document provides an overview of the vLLM architecture.
6
7 -```{contents} Table of Contents
7 +:::{contents} Table of Contents
8 :depth: 2
9 :local: true
10 -```
10 +:::
11
12 ## Entrypoints
13
14 vLLM provides a number of entrypoints for interacting with the system. The
15 following diagram shows the relationship between them.
16
17 -```{image} /assets/design/arch_overview/entrypoints.excalidraw.png
17 +:::{image} /assets/design/arch_overview/entrypoints.excalidraw.png
18 :alt: Entrypoints Diagram
19 -```
19 +:::
20
21 ### LLM Class
22
@@ -84,9 +84,9 @@ More details on the API server can be found in the [OpenAI-Compatible Server](#o
84 The `LLMEngine` and `AsyncLLMEngine` classes are central to the functioning of
85 the vLLM system, handling model inference and asynchronous request processing.
86
87 -```{image} /assets/design/arch_overview/llm_engine.excalidraw.png
87 +:::{image} /assets/design/arch_overview/llm_engine.excalidraw.png
88 :alt: LLMEngine Diagram
89 -```
89 +:::
90
91 ### LLMEngine
92
@@ -144,11 +144,11 @@ configurations affect the class we ultimately get.
144
145 The following figure shows the class hierarchy of vLLM:
146
147 -> ```{figure} /assets/design/hierarchy.png
147 +> :::{figure} /assets/design/hierarchy.png
148 > :align: center
149 > :alt: query
150 > :width: 100%
151 -> ```
151 +> :::
152
153 There are several important design choices behind this class hierarchy:
154
@@ -178,7 +178,7 @@ of a vision model and a language model. By making the constructor uniform, we
178 can easily create a vision model and a language model and compose them into a
179 vision-language model.
180
181 -````{note}
181 +:::{note}
182 To support this change, all vLLM models' signatures have been updated to:
183
184 ```python
@@ -215,7 +215,7 @@ else:
215 ```
216
217 This way, the model can work with both old and new versions of vLLM.
218 -````
218 +:::
219
220 3\. **Sharding and Quantization at Initialization**: Certain features require
221 changing the model weights. For example, tensor parallelism needs to shard the
docs/source/design/kernel/paged_attention.md
+16 -16
@@ -139,26 +139,26 @@
139 const scalar_t* q_ptr = q + seq_idx * q_stride + head_idx * HEAD_SIZE;
140 ```
141
142 - ```{figure} ../../assets/kernel/query.png
142 + :::{figure} ../../assets/kernel/query.png
143 :align: center
144 :alt: query
145 :width: 70%
146
147 Query data of one token at one head
148 - ```
148 + :::
149
150 - Each thread defines its own `q_ptr` which points to the assigned
151 query token data on global memory. For example, if `VEC_SIZE` is 4
152 and `HEAD_SIZE` is 128, the `q_ptr` points to data that contains
153 total of 128 elements divided into 128 / 4 = 32 vecs.
154
155 - ```{figure} ../../assets/kernel/q_vecs.png
155 + :::{figure} ../../assets/kernel/q_vecs.png
156 :align: center
157 :alt: q_vecs
158 :width: 70%
159
160 `q_vecs` for one thread group
161 - ```
161 + :::
162
163 ```cpp
164 __shared__ Q_vec q_vecs[THREAD_GROUP_SIZE][NUM_VECS_PER_THREAD];
@@ -195,13 +195,13 @@
195 points to key token data based on `k_cache` at assigned block,
196 assigned head and assigned token.
197
198 - ```{figure} ../../assets/kernel/key.png
198 + :::{figure} ../../assets/kernel/key.png
199 :align: center
200 :alt: key
201 :width: 70%
202
203 Key data of all context tokens at one head
204 - ```
204 + :::
205
206 - The diagram above illustrates the memory layout for key data. It
207 assumes that the `BLOCK_SIZE` is 16, `HEAD_SIZE` is 128, `x` is
@@ -214,13 +214,13 @@
214 elements for one token) that will be processed by 2 threads (one
215 thread group) separately.
216
217 - ```{figure} ../../assets/kernel/k_vecs.png
217 + :::{figure} ../../assets/kernel/k_vecs.png
218 :align: center
219 :alt: k_vecs
220 :width: 70%
221
222 `k_vecs` for one thread
223 - ```
223 + :::
224
225 ```cpp
226 K_vec k_vecs[NUM_VECS_PER_THREAD]
@@ -289,14 +289,14 @@
289 should be performed across the entire thread block, encompassing
290 results between the query token and all context key tokens.
291
292 - ```{math}
292 + :::{math}
293 :nowrap: true
294
295 \begin{gather*}
296 m(x):=\max _i \quad x_i \\ \quad f(x):=\left[\begin{array}{lll}e^{x_1-m(x)} & \ldots & e^{x_B-m(x)}\end{array}\right]\\ \quad \ell(x):=\sum_i f(x)_i \\
297 \quad \operatorname{softmax}(x):=\frac{f(x)}{\ell(x)}
298 \end{gather*}
299 - ```
299 + :::
300
301 ### `qk_max` and `logits`
302
@@ -379,29 +379,29 @@
379
380 ## Value
381
382 -```{figure} ../../assets/kernel/value.png
382 +:::{figure} ../../assets/kernel/value.png
383 :align: center
384 :alt: value
385 :width: 70%
386
387 Value data of all context tokens at one head
388 -```
388 +:::
389
390 -```{figure} ../../assets/kernel/logits_vec.png
390 +:::{figure} ../../assets/kernel/logits_vec.png
391 :align: center
392 :alt: logits_vec
393 :width: 50%
394
395 `logits_vec` for one thread
396 -```
396 +:::
397
398 -```{figure} ../../assets/kernel/v_vec.png
398 +:::{figure} ../../assets/kernel/v_vec.png
399 :align: center
400 :alt: v_vec
401 :width: 70%
402
403 List of `v_vec` for one thread
404 -```
404 +:::
405
406 - Now we need to retrieve the value data and perform dot multiplication
407 with `logits`. Unlike query and key, there is no thread group
docs/source/design/multiprocessing.md
+2 -2
@@ -7,9 +7,9 @@ page for information on known issues and how to solve them.
7
8 ## Introduction
9
10 -```{important}
10 +:::{important}
11 The source code references are to the state of the code at the time of writing in December, 2024.
12 -```
12 +:::
13
14 The use of Python multiprocessing in vLLM is complicated by:
15
docs/source/features/automatic_prefix_caching.md
+2 -2
@@ -6,9 +6,9 @@
6
7 Automatic Prefix Caching (APC in short) caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries, allowing the new query to skip the computation of the shared part.
8
9 -```{note}
9 +:::{note}
10 Technical details on how vLLM implements APC can be found [here](#design-automatic-prefix-caching).
11 -```
11 +:::
12
13 ## Enabling APC in vLLM
14
docs/source/features/compatibility_matrix.md
+439 -440
@@ -4,13 +4,13 @@
4
5 The tables below show mutually exclusive features and the support on some hardware.
6
7 -```{note}
7 +:::{note}
8 Check the '✗' with links to see tracking issue for unsupported feature/hardware combination.
9 -```
9 +:::
10
11 ## Feature x Feature
12
13 -```{raw} html
13 +:::{raw} html
14 <style>
15 /* Make smaller to try to improve readability */
16 td {
@@ -23,448 +23,447 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
23 font-size: 0.8rem;
24 }
25 </style>
26 -```
26 +:::
27
28 -```{list-table}
29 - :header-rows: 1
30 - :stub-columns: 1
31 - :widths: auto
28 +:::{list-table}
29 +:header-rows: 1
30 +:stub-columns: 1
31 +:widths: auto
32
33 - * - Feature
34 - - [CP](#chunked-prefill)
35 - - [APC](#automatic-prefix-caching)
36 - - [LoRA](#lora-adapter)
37 - - <abbr title="Prompt Adapter">prmpt adptr</abbr>
38 - - [SD](#spec_decode)
39 - - CUDA graph
40 - - <abbr title="Pooling Models">pooling</abbr>
41 - - <abbr title="Encoder-Decoder Models">enc-dec</abbr>
42 - - <abbr title="Logprobs">logP</abbr>
43 - - <abbr title="Prompt Logprobs">prmpt logP</abbr>
44 - - <abbr title="Async Output Processing">async output</abbr>
45 - - multi-step
46 - - <abbr title="Multimodal Inputs">mm</abbr>
47 - - best-of
48 - - beam-search
49 - - <abbr title="Guided Decoding">guided dec</abbr>
50 - * - [CP](#chunked-prefill)
51 - -
52 - -
53 - -
54 - -
55 - -
56 - -
57 - -
58 - -
59 - -
60 - -
61 - -
62 - -
63 - -
64 - -
65 - -
66 - -
67 - * - [APC](#automatic-prefix-caching)
68 - - ✅
69 - -
70 - -
71 - -
72 - -
73 - -
74 - -
75 - -
76 - -
77 - -
78 - -
79 - -
80 - -
81 - -
82 - -
83 - -
84 - * - [LoRA](#lora-adapter)
85 - - [✗](gh-pr:9057)
86 - - ✅
87 - -
88 - -
89 - -
90 - -
91 - -
92 - -
93 - -
94 - -
95 - -
96 - -
97 - -
98 - -
99 - -
100 - -
101 - * - <abbr title="Prompt Adapter">prmpt adptr</abbr>
102 - - ✅
103 - - ✅
104 - - ✅
105 - -
106 - -
107 - -
108 - -
109 - -
110 - -
111 - -
112 - -
113 - -
114 - -
115 - -
116 - -
117 - -
118 - * - [SD](#spec_decode)
119 - - ✅
120 - - ✅
121 - - ✗
122 - - ✅
123 - -
124 - -
125 - -
126 - -
127 - -
128 - -
129 - -
130 - -
131 - -
132 - -
133 - -
134 - -
135 - * - CUDA graph
136 - - ✅
137 - - ✅
138 - - ✅
139 - - ✅
140 - - ✅
141 - -
142 - -
143 - -
144 - -
145 - -
146 - -
147 - -
148 - -
149 - -
150 - -
151 - -
152 - * - <abbr title="Pooling Models">pooling</abbr>
153 - - ✗
154 - - ✗
155 - - ✗
156 - - ✗
157 - - ✗
158 - - ✗
159 - -
160 - -
161 - -
162 - -
163 - -
164 - -
165 - -
166 - -
167 - -
168 - -
169 - * - <abbr title="Encoder-Decoder Models">enc-dec</abbr>
170 - - ✗
171 - - [✗](gh-issue:7366)
172 - - ✗
173 - - ✗
174 - - [✗](gh-issue:7366)
175 - - ✅
176 - - ✅
177 - -
178 - -
179 - -
180 - -
181 - -
182 - -
183 - -
184 - -
185 - -
186 - * - <abbr title="Logprobs">logP</abbr>
187 - - ✅
188 - - ✅
189 - - ✅
190 - - ✅
191 - - ✅
192 - - ✅
193 - - ✗
194 - - ✅
195 - -
196 - -
197 - -
198 - -
199 - -
200 - -
201 - -
202 - -
203 - * - <abbr title="Prompt Logprobs">prmpt logP</abbr>
204 - - ✅
205 - - ✅
206 - - ✅
207 - - ✅
208 - - [✗](gh-pr:8199)
209 - - ✅
210 - - ✗
211 - - ✅
212 - - ✅
213 - -
214 - -
215 - -
216 - -
217 - -
218 - -
219 - -
220 - * - <abbr title="Async Output Processing">async output</abbr>
221 - - ✅
222 - - ✅
223 - - ✅
224 - - ✅
225 - - ✗
226 - - ✅
227 - - ✗
228 - - ✗
229 - - ✅
230 - - ✅
231 - -
232 - -
233 - -
234 - -
235 - -
236 - -
237 - * - multi-step
238 - - ✗
239 - - ✅
240 - - ✗
241 - - ✅
242 - - ✗
243 - - ✅
244 - - ✗
245 - - ✗
246 - - ✅
247 - - [✗](gh-issue:8198)
248 - - ✅
249 - -
250 - -
251 - -
252 - -
253 - -
254 - * - <abbr title="Multimodal Inputs">mm</abbr>
255 - - ✅
256 - - [✗](gh-pr:8348)
257 - - [✗](gh-pr:7199)
258 - - ?
259 - - ?
260 - - ✅
261 - - ✅
262 - - ✅
263 - - ✅
264 - - ✅
265 - - ✅
266 - - ?
267 - -
268 - -
269 - -
270 - -
271 - * - best-of
272 - - ✅
273 - - ✅
274 - - ✅
275 - - ✅
276 - - [✗](gh-issue:6137)
277 - - ✅
278 - - ✗
279 - - ✅
280 - - ✅
281 - - ✅
282 - - ?
283 - - [✗](gh-issue:7968)
284 - - ✅
285 - -
286 - -
287 - -
288 - * - beam-search
289 - - ✅
290 - - ✅
291 - - ✅
292 - - ✅
293 - - [✗](gh-issue:6137)
294 - - ✅
295 - - ✗
296 - - ✅
297 - - ✅
298 - - ✅
299 - - ?
300 - - [✗](gh-issue:7968>)
301 - - ?
302 - - ✅
303 - -
304 - -
305 - * - <abbr title="Guided Decoding">guided dec</abbr>
306 - - ✅
307 - - ✅
308 - - ?
309 - - ?
310 - - [✗](gh-issue:11484)
311 - - ✅
312 - - ✗
313 - - ?
314 - - ✅
315 - - ✅
316 - - ✅
317 - - [✗](gh-issue:9893)
318 - - ?
319 - - ✅
320 - - ✅
321 - -
322 -
323 -```
33 +- * Feature
34 + * [CP](#chunked-prefill)
35 + * [APC](#automatic-prefix-caching)
36 + * [LoRA](#lora-adapter)
37 + * <abbr title="Prompt Adapter">prmpt adptr</abbr>
38 + * [SD](#spec_decode)
39 + * CUDA graph
40 + * <abbr title="Pooling Models">pooling</abbr>
41 + * <abbr title="Encoder-Decoder Models">enc-dec</abbr>
42 + * <abbr title="Logprobs">logP</abbr>
43 + * <abbr title="Prompt Logprobs">prmpt logP</abbr>
44 + * <abbr title="Async Output Processing">async output</abbr>
45 + * multi-step
46 + * <abbr title="Multimodal Inputs">mm</abbr>
47 + * best-of
48 + * beam-search
49 + * <abbr title="Guided Decoding">guided dec</abbr>
50 +- * [CP](#chunked-prefill)
51 + *
52 + *
53 + *
54 + *
55 + *
56 + *
57 + *
58 + *
59 + *
60 + *
61 + *
62 + *
63 + *
64 + *
65 + *
66 + *
67 +- * [APC](#automatic-prefix-caching)
68 + * ✅
69 + *
70 + *
71 + *
72 + *
73 + *
74 + *
75 + *
76 + *
77 + *
78 + *
79 + *
80 + *
81 + *
82 + *
83 + *
84 +- * [LoRA](#lora-adapter)
85 + * [✗](gh-pr:9057)
86 + * ✅
87 + *
88 + *
89 + *
90 + *
91 + *
92 + *
93 + *
94 + *
95 + *
96 + *
97 + *
98 + *
99 + *
100 + *
101 +- * <abbr title="Prompt Adapter">prmpt adptr</abbr>
102 + * ✅
103 + * ✅
104 + * ✅
105 + *
106 + *
107 + *
108 + *
109 + *
110 + *
111 + *
112 + *
113 + *
114 + *
115 + *
116 + *
117 + *
118 +- * [SD](#spec_decode)
119 + * ✅
120 + * ✅
121 + * ✗
122 + * ✅
123 + *
124 + *
125 + *
126 + *
127 + *
128 + *
129 + *
130 + *
131 + *
132 + *
133 + *
134 + *
135 +- * CUDA graph
136 + * ✅
137 + * ✅
138 + * ✅
139 + * ✅
140 + * ✅
141 + *
142 + *
143 + *
144 + *
145 + *
146 + *
147 + *
148 + *
149 + *
150 + *
151 + *
152 +- * <abbr title="Pooling Models">pooling</abbr>
153 + * ✗
154 + * ✗
155 + * ✗
156 + * ✗
157 + * ✗
158 + * ✗
159 + *
160 + *
161 + *
162 + *
163 + *
164 + *
165 + *
166 + *
167 + *
168 + *
169 +- * <abbr title="Encoder-Decoder Models">enc-dec</abbr>
170 + * ✗
171 + * [✗](gh-issue:7366)
172 + * ✗
173 + * ✗
174 + * [✗](gh-issue:7366)
175 + * ✅
176 + * ✅
177 + *
178 + *
179 + *
180 + *
181 + *
182 + *
183 + *
184 + *
185 + *
186 +- * <abbr title="Logprobs">logP</abbr>
187 + * ✅
188 + * ✅
189 + * ✅
190 + * ✅
191 + * ✅
192 + * ✅
193 + * ✗
194 + * ✅
195 + *
196 + *
197 + *
198 + *
199 + *
200 + *
201 + *
202 + *
203 +- * <abbr title="Prompt Logprobs">prmpt logP</abbr>
204 + * ✅
205 + * ✅
206 + * ✅
207 + * ✅
208 + * [✗](gh-pr:8199)
209 + * ✅
210 + * ✗
211 + * ✅
212 + * ✅
213 + *
214 + *
215 + *
216 + *
217 + *
218 + *
219 + *
220 +- * <abbr title="Async Output Processing">async output</abbr>
221 + * ✅
222 + * ✅
223 + * ✅
224 + * ✅
225 + * ✗
226 + * ✅
227 + * ✗
228 + * ✗
229 + * ✅
230 + * ✅
231 + *
232 + *
233 + *
234 + *
235 + *
236 + *
237 +- * multi-step
238 + * ✗
239 + * ✅
240 + * ✗
241 + * ✅
242 + * ✗
243 + * ✅
244 + * ✗
245 + * ✗
246 + * ✅
247 + * [✗](gh-issue:8198)
248 + * ✅
249 + *
250 + *
251 + *
252 + *
253 + *
254 +- * <abbr title="Multimodal Inputs">mm</abbr>
255 + * ✅
256 + * [✗](gh-pr:8348)
257 + * [✗](gh-pr:7199)
258 + * ?
259 + * ?
260 + * ✅
261 + * ✅
262 + * ✅
263 + * ✅
264 + * ✅
265 + * ✅
266 + * ?
267 + *
268 + *
269 + *
270 + *
271 +- * best-of
272 + * ✅
273 + * ✅
274 + * ✅
275 + * ✅
276 + * [✗](gh-issue:6137)
277 + * ✅
278 + * ✗
279 + * ✅
280 + * ✅
281 + * ✅
282 + * ?
283 + * [✗](gh-issue:7968)
284 + * ✅
285 + *
286 + *
287 + *
288 +- * beam-search
289 + * ✅
290 + * ✅
291 + * ✅
292 + * ✅
293 + * [✗](gh-issue:6137)
294 + * ✅
295 + * ✗
296 + * ✅
297 + * ✅
298 + * ✅
299 + * ?
300 + * [✗](gh-issue:7968>)
301 + * ?
302 + * ✅
303 + *
304 + *
305 +- * <abbr title="Guided Decoding">guided dec</abbr>
306 + * ✅
307 + * ✅
308 + * ?
309 + * ?
310 + * [✗](gh-issue:11484)
311 + * ✅
312 + * ✗
313 + * ?
314 + * ✅
315 + * ✅
316 + * ✅
317 + * [✗](gh-issue:9893)
318 + * ?
319 + * ✅
320 + * ✅
321 + *
322 +:::
323
324 (feature-x-hardware)=
325
326 ## Feature x Hardware
327
329 -```{list-table}
330 - :header-rows: 1
331 - :stub-columns: 1
332 - :widths: auto
328 +:::{list-table}
329 +:header-rows: 1
330 +:stub-columns: 1
331 +:widths: auto
332
334 - * - Feature
335 - - Volta
336 - - Turing
337 - - Ampere
338 - - Ada
339 - - Hopper
340 - - CPU
341 - - AMD
342 - * - [CP](#chunked-prefill)
343 - - [✗](gh-issue:2729)
344 - - ✅
345 - - ✅
346 - - ✅
347 - - ✅
348 - - ✅
349 - - ✅
350 - * - [APC](#automatic-prefix-caching)
351 - - [✗](gh-issue:3687)
352 - - ✅
353 - - ✅
354 - - ✅
355 - - ✅
356 - - ✅
357 - - ✅
358 - * - [LoRA](#lora-adapter)
359 - - ✅
360 - - ✅
361 - - ✅
362 - - ✅
363 - - ✅
364 - - ✅
365 - - ✅
366 - * - <abbr title="Prompt Adapter">prmpt adptr</abbr>
367 - - ✅
368 - - ✅
369 - - ✅
370 - - ✅
371 - - ✅
372 - - [✗](gh-issue:8475)
373 - - ✅
374 - * - [SD](#spec_decode)
375 - - ✅
376 - - ✅
377 - - ✅
378 - - ✅
379 - - ✅
380 - - ✅
381 - - ✅
382 - * - CUDA graph
383 - - ✅
384 - - ✅
385 - - ✅
386 - - ✅
387 - - ✅
388 - - ✗
389 - - ✅
390 - * - <abbr title="Pooling Models">pooling</abbr>
391 - - ✅
392 - - ✅
393 - - ✅
394 - - ✅
395 - - ✅
396 - - ✅
397 - - ?
398 - * - <abbr title="Encoder-Decoder Models">enc-dec</abbr>
399 - - ✅
400 - - ✅
401 - - ✅
402 - - ✅
403 - - ✅
404 - - ✅
405 - - ✗
406 - * - <abbr title="Multimodal Inputs">mm</abbr>
407 - - ✅
408 - - ✅
409 - - ✅
410 - - ✅
411 - - ✅
412 - - ✅
413 - - ✅
414 - * - <abbr title="Logprobs">logP</abbr>
415 - - ✅
416 - - ✅
417 - - ✅
418 - - ✅
419 - - ✅
420 - - ✅
421 - - ✅
422 - * - <abbr title="Prompt Logprobs">prmpt logP</abbr>
423 - - ✅
424 - - ✅
425 - - ✅
426 - - ✅
427 - - ✅
428 - - ✅
429 - - ✅
430 - * - <abbr title="Async Output Processing">async output</abbr>
431 - - ✅
432 - - ✅
433 - - ✅
434 - - ✅
435 - - ✅
436 - - ✗
437 - - ✗
438 - * - multi-step
439 - - ✅
440 - - ✅
441 - - ✅
442 - - ✅
443 - - ✅
444 - - [✗](gh-issue:8477)
445 - - ✅
446 - * - best-of
447 - - ✅
448 - - ✅
449 - - ✅
450 - - ✅
451 - - ✅
452 - - ✅
453 - - ✅
454 - * - beam-search
455 - - ✅
456 - - ✅
457 - - ✅
458 - - ✅
459 - - ✅
460 - - ✅
461 - - ✅
462 - * - <abbr title="Guided Decoding">guided dec</abbr>
463 - - ✅
464 - - ✅
465 - - ✅
466 - - ✅
467 - - ✅
468 - - ✅
469 - - ✅
470 -```
333 +- * Feature
334 + * Volta
335 + * Turing
336 + * Ampere
337 + * Ada
338 + * Hopper
339 + * CPU
340 + * AMD
341 +- * [CP](#chunked-prefill)
342 + * [✗](gh-issue:2729)
343 + * ✅
344 + * ✅
345 + * ✅
346 + * ✅
347 + * ✅
348 + * ✅
349 +- * [APC](#automatic-prefix-caching)
350 + * [✗](gh-issue:3687)
351 + * ✅
352 + * ✅
353 + * ✅
354 + * ✅
355 + * ✅
356 + * ✅
357 +- * [LoRA](#lora-adapter)
358 + * ✅
359 + * ✅
360 + * ✅
361 + * ✅
362 + * ✅
363 + * ✅
364 + * ✅
365 +- * <abbr title="Prompt Adapter">prmpt adptr</abbr>
366 + * ✅
367 + * ✅
368 + * ✅
369 + * ✅
370 + * ✅
371 + * [✗](gh-issue:8475)
372 + * ✅
373 +- * [SD](#spec_decode)
374 + * ✅
375 + * ✅
376 + * ✅
377 + * ✅
378 + * ✅
379 + * ✅
380 + * ✅
381 +- * CUDA graph
382 + * ✅
383 + * ✅
384 + * ✅
385 + * ✅
386 + * ✅
387 + * ✗
388 + * ✅
389 +- * <abbr title="Pooling Models">pooling</abbr>
390 + * ✅
391 + * ✅
392 + * ✅
393 + * ✅
394 + * ✅
395 + * ✅
396 + * ?
397 +- * <abbr title="Encoder-Decoder Models">enc-dec</abbr>
398 + * ✅
399 + * ✅
400 + * ✅
401 + * ✅
402 + * ✅
403 + * ✅
404 + * ✗
405 +- * <abbr title="Multimodal Inputs">mm</abbr>
406 + * ✅
407 + * ✅
408 + * ✅
409 + * ✅
410 + * ✅
411 + * ✅
412 + * ✅
413 +- * <abbr title="Logprobs">logP</abbr>
414 + * ✅
415 + * ✅
416 + * ✅
417 + * ✅
418 + * ✅
419 + * ✅
420 + * ✅
421 +- * <abbr title="Prompt Logprobs">prmpt logP</abbr>
422 + * ✅
423 + * ✅
424 + * ✅
425 + * ✅
426 + * ✅
427 + * ✅
428 + * ✅
429 +- * <abbr title="Async Output Processing">async output</abbr>
430 + * ✅
431 + * ✅
432 + * ✅
433 + * ✅
434 + * ✅
435 + * ✗
436 + * ✗
437 +- * multi-step
438 + * ✅
439 + * ✅
440 + * ✅
441 + * ✅
442 + * ✅
443 + * [✗](gh-issue:8477)
444 + * ✅
445 +- * best-of
446 + * ✅
447 + * ✅
448 + * ✅
449 + * ✅
450 + * ✅
451 + * ✅
452 + * ✅
453 +- * beam-search
454 + * ✅
455 + * ✅
456 + * ✅
457 + * ✅
458 + * ✅
459 + * ✅
460 + * ✅
461 +- * <abbr title="Guided Decoding">guided dec</abbr>
462 + * ✅
463 + * ✅
464 + * ✅
465 + * ✅
466 + * ✅
467 + * ✅
468 + * ✅
469 +:::
docs/source/features/disagg_prefill.md
+10 -10
@@ -4,9 +4,9 @@
4
5 This page introduces you the disaggregated prefilling feature in vLLM.
6
7 -```{note}
7 +:::{note}
8 This feature is experimental and subject to change.
9 -```
9 +:::
10
11 ## Why disaggregated prefilling?
12
@@ -15,9 +15,9 @@ Two main reasons:
15 - **Tuning time-to-first-token (TTFT) and inter-token-latency (ITL) separately**. Disaggregated prefilling put prefill and decode phase of LLM inference inside different vLLM instances. This gives you the flexibility to assign different parallel strategies (e.g. `tp` and `pp`) to tune TTFT without affecting ITL, or to tune ITL without affecting TTFT.
16 - **Controlling tail ITL**. Without disaggregated prefilling, vLLM may insert some prefill jobs during the decoding of one request. This results in higher tail latency. Disaggregated prefilling helps you solve this issue and control tail ITL. Chunked prefill with a proper chunk size also can achieve the same goal, but in practice it's hard to figure out the correct chunk size value. So disaggregated prefilling is a much more reliable way to control tail ITL.
17
18 -```{note}
18 +:::{note}
19 Disaggregated prefill DOES NOT improve throughput.
20 -```
20 +:::
21
22 ## Usage example
23
@@ -39,21 +39,21 @@ Key abstractions for disaggregated prefilling:
39 - **LookupBuffer**: LookupBuffer provides two API: `insert` KV cache and `drop_select` KV cache. The semantics of `insert` and `drop_select` are similar to SQL, where `insert` inserts a KV cache into the buffer, and `drop_select` returns the KV cache that matches the given condition and drop it from the buffer.
40 - **Pipe**: A single-direction FIFO pipe for tensor transmission. It supports `send_tensor` and `recv_tensor`.
41
42 -```{note}
42 +:::{note}
43 `insert` is non-blocking operation but `drop_select` is blocking operation.
44 -```
44 +:::
45
46 Here is a figure illustrating how the above 3 abstractions are organized:
47
48 -```{image} /assets/features/disagg_prefill/abstraction.jpg
48 +:::{image} /assets/features/disagg_prefill/abstraction.jpg
49 :alt: Disaggregated prefilling abstractions
50 -```
50 +:::
51
52 The workflow of disaggregated prefilling is as follows:
53
54 -```{image} /assets/features/disagg_prefill/overview.jpg
54 +:::{image} /assets/features/disagg_prefill/overview.jpg
55 :alt: Disaggregated prefilling workflow
56 -```
56 +:::
57
58 The `buffer` corresponds to `insert` API in LookupBuffer, and the `drop_select` corresponds to `drop_select` API in LookupBuffer.
59
docs/source/features/lora.md
+2 -2
@@ -60,9 +60,9 @@ vllm serve meta-llama/Llama-2-7b-hf \
60 --lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/
61 ```
62
63 -```{note}
63 +:::{note}
64 The commit ID `0dfa347e8877a4d4ed19ee56c140fa518470028c` may change over time. Please check the latest commit ID in your environment to ensure you are using the correct one.
65 -```
65 +:::
66
67 The server entrypoint accepts all other LoRA configuration parameters (`max_loras`, `max_lora_rank`, `max_cpu_loras`,
68 etc.), which will apply to all forthcoming requests. Upon querying the `/models` endpoint, we should see our LoRA along
docs/source/features/quantization/auto_awq.md
+2 -2
@@ -2,11 +2,11 @@
2
3 # AutoAWQ
4
5 -```{warning}
5 +:::{warning}
6 Please note that AWQ support in vLLM is under-optimized at the moment. We would recommend using the unquantized version of the model for better
7 accuracy and higher throughput. Currently, you can use AWQ as a way to reduce memory footprint. As of now, it is more suitable for low latency
8 inference with small number of concurrent requests. vLLM's AWQ implementation have lower throughput than unquantized version.
9 -```
9 +:::
10
11 To create a new 4-bit quantized model, you can leverage [AutoAWQ](https://github.com/casper-hansen/AutoAWQ).
12 Quantizing reduces the model's precision from FP16 to INT4 which effectively reduces the file size by ~70%.
docs/source/features/quantization/fp8.md
+8 -8
@@ -14,10 +14,10 @@ The FP8 types typically supported in hardware have two distinct representations,
14 - **E4M3**: Consists of 1 sign bit, 4 exponent bits, and 3 bits of mantissa. It can store values up to +/-448 and `nan`.
15 - **E5M2**: Consists of 1 sign bit, 5 exponent bits, and 2 bits of mantissa. It can store values up to +/-57344, +/- `inf`, and `nan`. The tradeoff for the increased dynamic range is lower precision of the stored values.
16
17 -```{note}
17 +:::{note}
18 FP8 computation is supported on NVIDIA GPUs with compute capability > 8.9 (Ada Lovelace, Hopper).
19 FP8 models will run on compute capability > 8.0 (Ampere) as weight-only W8A16, utilizing FP8 Marlin.
20 -```
20 +:::
21
22 ## Quick Start with Online Dynamic Quantization
23
@@ -32,9 +32,9 @@ model = LLM("facebook/opt-125m", quantization="fp8")
32 result = model.generate("Hello, my name is")
33 ```
34
35 -```{warning}
35 +:::{warning}
36 Currently, we load the model at original precision before quantizing down to 8-bits, so you need enough memory to load the whole model.
37 -```
37 +:::
38
39 ## Installation
40
@@ -110,9 +110,9 @@ model.generate("Hello my name is")
110
111 Evaluate accuracy with `lm_eval` (for example on 250 samples of `gsm8k`):
112
113 -```{note}
113 +:::{note}
114 Quantized models can be sensitive to the presence of the `bos` token. `lm_eval` does not add a `bos` token by default, so make sure to include the `add_bos_token=True` argument when running your evaluations.
115 -```
115 +:::
116
117 ```console
118 $ MODEL=$PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic
@@ -137,10 +137,10 @@ If you encounter any issues or have feature requests, please open an issue on th
137
138 ## Deprecated Flow
139
140 -```{note}
140 +:::{note}
141 The following information is preserved for reference and search purposes.
142 The quantization method described below is deprecated in favor of the `llmcompressor` method described above.
143 -```
143 +:::
144
145 For static per-tensor offline quantization to FP8, please install the [AutoFP8 library](https://github.com/neuralmagic/autofp8).
146
docs/source/features/quantization/gguf.md
+6 -6
@@ -2,13 +2,13 @@
2
3 # GGUF
4
5 -```{warning}
5 +:::{warning}
6 Please note that GGUF support in vLLM is highly experimental and under-optimized at the moment, it might be incompatible with other features. Currently, you can use GGUF as a way to reduce memory footprint. If you encounter any issues, please report them to the vLLM team.
7 -```
7 +:::
8
9 -```{warning}
9 +:::{warning}
10 Currently, vllm only supports loading single-file GGUF models. If you have a multi-files GGUF model, you can use [gguf-split](https://github.com/ggerganov/llama.cpp/pull/6135) tool to merge them to a single-file model.
11 -```
11 +:::
12
13 To run a GGUF model with vLLM, you can download and use the local GGUF model from [TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF](https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF) with the following command:
14
@@ -25,9 +25,9 @@ You can also add `--tensor-parallel-size 2` to enable tensor parallelism inferen
25 vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 --tensor-parallel-size 2
26 ```
27
28 -```{warning}
28 +:::{warning}
29 We recommend using the tokenizer from base model instead of GGUF model. Because the tokenizer conversion from GGUF is time-consuming and unstable, especially for some models with large vocab size.
30 -```
30 +:::
31
32 You can also use the GGUF model directly through the LLM entrypoint:
33
docs/source/features/quantization/index.md
+2 -2
@@ -4,7 +4,7 @@
4
5 Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.
6
7 -```{toctree}
7 +:::{toctree}
8 :caption: Contents
9 :maxdepth: 1
10
@@ -15,4 +15,4 @@ gguf
15 int8
16 fp8
17 quantized_kvcache
18 -```
18 +:::
docs/source/features/quantization/int8.md
+4 -4
@@ -7,9 +7,9 @@ This quantization method is particularly useful for reducing model size while ma
7
8 Please visit the HF collection of [quantized INT8 checkpoints of popular LLMs ready to use with vLLM](https://huggingface.co/collections/neuralmagic/int8-llms-for-vllm-668ec32c049dca0369816415).
9
10 -```{note}
10 +:::{note}
11 INT8 computation is supported on NVIDIA GPUs with compute capability > 7.5 (Turing, Ampere, Ada Lovelace, Hopper).
12 -```
12 +:::
13
14 ## Prerequisites
15
@@ -119,9 +119,9 @@ $ lm_eval --model vllm \
119 --batch_size 'auto'
120 ```
121
122 -```{note}
122 +:::{note}
123 Quantized models can be sensitive to the presence of the `bos` token. Make sure to include the `add_bos_token=True` argument when running evaluations.
124 -```
124 +:::
125
126 ## Best Practices
127
docs/source/features/quantization/supported_hardware.md
+115 -114
@@ -4,128 +4,129 @@
4
5 The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
6
7 -```{list-table}
7 +:::{list-table}
8 :header-rows: 1
9 :widths: 20 8 8 8 8 8 8 8 8 8 8
10
11 -* - Implementation
12 - - Volta
13 - - Turing
14 - - Ampere
15 - - Ada
16 - - Hopper
17 - - AMD GPU
18 - - Intel GPU
19 - - x86 CPU
20 - - AWS Inferentia
21 - - Google TPU
22 -* - AWQ
23 - - ✗
24 - - ✅︎
25 - - ✅︎
26 - - ✅︎
27 - - ✅︎
28 - - ✗
29 - - ✅︎
30 - - ✅︎
31 - - ✗
32 - - ✗
33 -* - GPTQ
34 - - ✅︎
35 - - ✅︎
36 - - ✅︎
37 - - ✅︎
38 - - ✅︎
39 - - ✗
40 - - ✅︎
41 - - ✅︎
42 - - ✗
43 - - ✗
44 -* - Marlin (GPTQ/AWQ/FP8)
45 - - ✗
46 - - ✗
47 - - ✅︎
48 - - ✅︎
49 - - ✅︎
50 - - ✗
51 - - ✗
52 - - ✗
53 - - ✗
54 - - ✗
55 -* - INT8 (W8A8)
56 - - ✗
57 - - ✅︎
58 - - ✅︎
59 - - ✅︎
60 - - ✅︎
61 - - ✗
62 - - ✗
63 - - ✅︎
64 - - ✗
65 - - ✗
66 -* - FP8 (W8A8)
67 - - ✗
68 - - ✗
69 - - ✗
70 - - ✅︎
71 - - ✅︎
72 - - ✅︎
73 - - ✗
74 - - ✗
75 - - ✗
76 - - ✗
77 -* - AQLM
78 - - ✅︎
79 - - ✅︎
80 - - ✅︎
81 - - ✅︎
82 - - ✅︎
83 - - ✗
84 - - ✗
85 - - ✗
86 - - ✗
87 - - ✗
88 -* - bitsandbytes
89 - - ✅︎
90 - - ✅︎
91 - - ✅︎
92 - - ✅︎
93 - - ✅︎
94 - - ✗
95 - - ✗
96 - - ✗
97 - - ✗
98 - - ✗
99 -* - DeepSpeedFP
100 - - ✅︎
101 - - ✅︎
102 - - ✅︎
103 - - ✅︎
104 - - ✅︎
105 - - ✗
106 - - ✗
107 - - ✗
108 - - ✗
109 - - ✗
110 -* - GGUF
111 - - ✅︎
112 - - ✅︎
113 - - ✅︎
114 - - ✅︎
115 - - ✅︎
116 - - ✅︎
117 - - ✗
118 - - ✗
119 - - ✗
120 - - ✗
121 -```
11 +- * Implementation
12 + * Volta
13 + * Turing
14 + * Ampere
15 + * Ada
16 + * Hopper
17 + * AMD GPU
18 + * Intel GPU
19 + * x86 CPU
20 + * AWS Inferentia
21 + * Google TPU
22 +- * AWQ
23 + * ✗
24 + * ✅︎
25 + * ✅︎
26 + * ✅︎
27 + * ✅︎
28 + * ✗
29 + * ✅︎
30 + * ✅︎
31 + * ✗
32 + * ✗
33 +- * GPTQ
34 + * ✅︎
35 + * ✅︎
36 + * ✅︎
37 + * ✅︎
38 + * ✅︎
39 + * ✗
40 + * ✅︎
41 + * ✅︎
42 + * ✗
43 + * ✗
44 +- * Marlin (GPTQ/AWQ/FP8)
45 + * ✗
46 + * ✗
47 + * ✅︎
48 + * ✅︎
49 + * ✅︎
50 + * ✗
51 + * ✗
52 + * ✗
53 + * ✗
54 + * ✗
55 +- * INT8 (W8A8)
56 + * ✗
57 + * ✅︎
58 + * ✅︎
59 + * ✅︎
60 + * ✅︎
61 + * ✗
62 + * ✗
63 + * ✅︎
64 + * ✗
65 + * ✗
66 +- * FP8 (W8A8)
67 + * ✗
68 + * ✗
69 + * ✗
70 + * ✅︎
71 + * ✅︎
72 + * ✅︎
73 + * ✗
74 + * ✗
75 + * ✗
76 + * ✗
77 +- * AQLM
78 + * ✅︎
79 + * ✅︎
80 + * ✅︎
81 + * ✅︎
82 + * ✅︎
83 + * ✗
84 + * ✗
85 + * ✗
86 + * ✗
87 + * ✗
88 +- * bitsandbytes
89 + * ✅︎
90 + * ✅︎
91 + * ✅︎
92 + * ✅︎
93 + * ✅︎
94 + * ✗
95 + * ✗
96 + * ✗
97 + * ✗
98 + * ✗
99 +- * DeepSpeedFP
100 + * ✅︎
101 + * ✅︎
102 + * ✅︎
103 + * ✅︎
104 + * ✅︎
105 + * ✗
106 + * ✗
107 + * ✗
108 + * ✗
109 + * ✗
110 +- * GGUF
111 + * ✅︎
112 + * ✅︎
113 + * ✅︎
114 + * ✅︎
115 + * ✅︎
116 + * ✅︎
117 + * ✗
118 + * ✗
119 + * ✗
120 + * ✗
121 +
122 +:::
123
124 - Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
125 - "✅︎" indicates that the quantization method is supported on the specified hardware.
126 - "✗" indicates that the quantization method is not supported on the specified hardware.
127
127 -```{note}
128 +:::{note}
129 This compatibility chart is subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
130
131 For the most up-to-date information on hardware support and quantization methods, please refer to <gh-dir:vllm/model_executor/layers/quantization> or consult with the vLLM development team.
131 -```
132 +:::
docs/source/features/spec_decode.md
+4 -4
@@ -2,15 +2,15 @@
2
3 # Speculative Decoding
4
5 -```{warning}
5 +:::{warning}
6 Please note that speculative decoding in vLLM is not yet optimized and does
7 not usually yield inter-token latency reductions for all prompt datasets or sampling parameters.
8 The work to optimize it is ongoing and can be followed here: <gh-issue:4630>
9 -```
9 +:::
10
11 -```{warning}
11 +:::{warning}
12 Currently, speculative decoding in vLLM is not compatible with pipeline parallelism.
13 -```
13 +:::
14
15 This document shows how to use [Speculative Decoding](https://x.com/karpathy/status/1697318534555336961) with vLLM.
16 Speculative decoding is a technique which improves inter-token latency in memory-bound LLM inference.
docs/source/features/structured_outputs.md
+2 -2
@@ -95,10 +95,10 @@ completion = client.chat.completions.create(
95 print(completion.choices[0].message.content)
96 ```
97
98 -```{tip}
98 +:::{tip}
99 While not strictly necessary, normally it´s better to indicate in the prompt that a JSON needs to be generated and which fields and how should the LLM fill them.
100 This can improve the results notably in most cases.
101 -```
101 +:::
102
103 Finally we have the `guided_grammar`, which probably is the most difficult one to use but it´s really powerful, as it allows us to define complete languages like SQL queries.
104 It works by using a context free EBNF grammar, which for example we can use to define a specific format of simplified SQL queries, like in the example below:
docs/source/generate_examples.py
+2 -2
@@ -57,9 +57,9 @@ class Index:
57
58 def generate(self) -> str:
59 content = f"# {self.title}\n\n{self.description}\n\n"
60 - content += "```{toctree}\n"
60 + content += ":::{toctree}\n"
61 content += f":caption: {self.caption}\n:maxdepth: {self.maxdepth}\n"
62 - content += "\n".join(self.documents) + "\n```\n"
62 + content += "\n".join(self.documents) + "\n:::\n"
63 return content
64
65
docs/source/getting_started/installation/ai_accelerator/hpu-gaudi.inc.md
+38 -38
@@ -86,9 +86,9 @@ docker build -f Dockerfile.hpu -t vllm-hpu-env .
86 docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --rm vllm-hpu-env
87 ```
88
89 -```{tip}
89 +:::{tip}
90 If you're observing the following error: `docker: Error response from daemon: Unknown runtime specified habana.`, please refer to "Install Using Containers" section of [Intel Gaudi Software Stack and Driver Installation](https://docs.habana.ai/en/v1.18.0/Installation_Guide/Bare_Metal_Fresh_OS.html). Make sure you have `habana-container-runtime` package installed and that `habana` container runtime is registered.
91 -```
91 +:::
92
93 ## Extra information
94
@@ -155,30 +155,30 @@ Gaudi2 devices. Configurations that are not listed may or may not work.
155
156 Currently in vLLM for HPU we support four execution modes, depending on selected HPU PyTorch Bridge backend (via `PT_HPU_LAZY_MODE` environment variable), and `--enforce-eager` flag.
157
158 -```{list-table} vLLM execution modes
158 +:::{list-table} vLLM execution modes
159 :widths: 25 25 50
160 :header-rows: 1
161
162 -* - `PT_HPU_LAZY_MODE`
163 - - `enforce_eager`
164 - - execution mode
165 -* - 0
166 - - 0
167 - - torch.compile
168 -* - 0
169 - - 1
170 - - PyTorch eager mode
171 -* - 1
172 - - 0
173 - - HPU Graphs
174 -* - 1
175 - - 1
176 - - PyTorch lazy mode
177 -```
178 -
179 -```{warning}
162 +- * `PT_HPU_LAZY_MODE`
163 + * `enforce_eager`
164 + * execution mode
165 +- * 0
166 + * 0
167 + * torch.compile
168 +- * 0
169 + * 1
170 + * PyTorch eager mode
171 +- * 1
172 + * 0
173 + * HPU Graphs
174 +- * 1
175 + * 1
176 + * PyTorch lazy mode
177 +:::
178 +
179 +:::{warning}
180 In 1.18.0, all modes utilizing `PT_HPU_LAZY_MODE=0` are highly experimental and should be only used for validating functional correctness. Their performance will be improved in the next releases. For obtaining the best performance in 1.18.0, please use HPU Graphs, or PyTorch lazy mode.
181 -```
181 +:::
182
183 (gaudi-bucketing-mechanism)=
184
@@ -187,9 +187,9 @@ In 1.18.0, all modes utilizing `PT_HPU_LAZY_MODE=0` are highly experimental and
187 Intel Gaudi accelerators work best when operating on models with fixed tensor shapes. [Intel Gaudi Graph Compiler](https://docs.habana.ai/en/latest/Gaudi_Overview/Intel_Gaudi_Software_Suite.html#graph-compiler-and-runtime) is responsible for generating optimized binary code that implements the given model topology on Gaudi. In its default configuration, the produced binary code may be heavily dependent on input and output tensor shapes, and can require graph recompilation when encountering differently shaped tensors within the same topology. While the resulting binaries utilize Gaudi efficiently, the compilation itself may introduce a noticeable overhead in end-to-end execution.
188 In a dynamic inference serving scenario, there is a need to minimize the number of graph compilations and reduce the risk of graph compilation occurring during server runtime. Currently it is achieved by "bucketing" model's forward pass across two dimensions - `batch_size` and `sequence_length`.
189
190 -```{note}
190 +:::{note}
191 Bucketing allows us to reduce the number of required graphs significantly, but it does not handle any graph compilation and device code generation - this is done in warmup and HPUGraph capture phase.
192 -```
192 +:::
193
194 Bucketing ranges are determined with 3 parameters - `min`, `step` and `max`. They can be set separately for prompt and decode phase, and for batch size and sequence length dimension. These parameters can be observed in logs during vLLM startup:
195
@@ -222,15 +222,15 @@ min = 128, step = 128, max = 512
222
223 In the logged scenario, 24 buckets were generated for prompt (prefill) runs, and 48 buckets for decode runs. Each bucket corresponds to a separate optimized device binary for a given model with specified tensor shapes. Whenever a batch of requests is processed, it is padded across batch and sequence length dimension to the smallest possible bucket.
224
225 -```{warning}
225 +:::{warning}
226 If a request exceeds maximum bucket size in any dimension, it will be processed without padding, and its processing may require a graph compilation, potentially significantly increasing end-to-end latency. The boundaries of the buckets are user-configurable via environment variables, and upper bucket boundaries can be increased to avoid such scenario.
227 -```
227 +:::
228
229 As an example, if a request of 3 sequences, with max sequence length of 412 comes in to an idle vLLM server, it will be padded executed as `(4, 512)` prefill bucket, as `batch_size` (number of sequences) will be padded to 4 (closest batch_size dimension higher than 3), and max sequence length will be padded to 512 (closest sequence length dimension higher than 412). After prefill stage, it will be executed as `(4, 512)` decode bucket and will continue as that bucket until either batch dimension changes (due to request being finished) - in which case it will become a `(2, 512)` bucket, or context length increases above 512 tokens, in which case it will become `(4, 640)` bucket.
230
231 -```{note}
231 +:::{note}
232 Bucketing is transparent to a client -- padding in sequence length dimension is never returned to the client, and padding in batch dimension does not create new requests.
233 -```
233 +:::
234
235 ### Warmup
236
@@ -252,9 +252,9 @@ INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size
252
253 This example uses the same buckets as in the [Bucketing Mechanism](#gaudi-bucketing-mechanism) section. Each output line corresponds to execution of a single bucket. When bucket is executed for the first time, its graph is compiled and can be reused later on, skipping further graph compilations.
254
255 -```{tip}
255 +:::{tip}
256 Compiling all the buckets might take some time and can be turned off with `VLLM_SKIP_WARMUP=true` environment variable. Keep in mind that if you do that, you may face graph compilations once executing a given bucket for the first time. It is fine to disable warmup for development, but it's highly recommended to enable it in deployment.
257 -```
257 +:::
258
259 ### HPU Graph capture
260
@@ -269,9 +269,9 @@ With its default value (`VLLM_GRAPH_RESERVED_MEM=0.1`), 10% of usable memory wil
269 Environment variable `VLLM_GRAPH_PROMPT_RATIO` determines the ratio of usable graph memory reserved for prefill and decode graphs. By default (`VLLM_GRAPH_PROMPT_RATIO=0.3`), both stages have equal memory constraints.
270 Lower value corresponds to less usable graph memory reserved for prefill stage, e.g. `VLLM_GRAPH_PROMPT_RATIO=0.2` will reserve 20% of usable graph memory for prefill graphs, and 80% of usable graph memory for decode graphs.
271
272 -```{note}
272 +:::{note}
273 `gpu_memory_utilization` does not correspond to the absolute memory usage across HPU. It specifies the memory margin after loading the model and performing a profile run. If device has 100 GiB of total memory, and 50 GiB of free memory after loading model weights and executing profiling run, `gpu_memory_utilization` at its default value will mark 90% of 50 GiB as usable, leaving 5 GiB of margin, regardless of total device memory.
274 -```
274 +:::
275
276 User can also configure the strategy for capturing HPU Graphs for prompt and decode stages separately. Strategy affects the order of capturing graphs. There are two strategies implemented:
277 \- `max_bs` - graph capture queue will sorted in descending order by their batch sizes. Buckets with equal batch sizes are sorted by sequence length in ascending order (e.g. `(64, 128)`, `(64, 256)`, `(32, 128)`, `(32, 256)`, `(1, 128)`, `(1,256)`), default strategy for decode
@@ -279,9 +279,9 @@ User can also configure the strategy for capturing HPU Graphs for prompt and dec
279
280 When there's large amount of requests pending, vLLM scheduler will attempt to fill the maximum batch size for decode as soon as possible. When a request is finished, decode batch size decreases. When that happens, vLLM will attempt to schedule a prefill iteration for requests in the waiting queue, to fill the decode batch size to its previous state. This means that in a full load scenario, decode batch size is often at its maximum, which makes large batch size HPU Graphs crucial to capture, as reflected by `max_bs` strategy. On the other hand, prefills will be executed most frequently with very low batch sizes (1-4), which is reflected in `min_tokens` strategy.
281
282 -```{note}
282 +:::{note}
283 `VLLM_GRAPH_PROMPT_RATIO` does not set a hard limit on memory taken by graphs for each stage (prefill and decode). vLLM will first attempt to use up entirety of usable prefill graph memory (usable graph memory * `VLLM_GRAPH_PROMPT_RATIO`) for capturing prefill HPU Graphs, next it will attempt do the same for decode graphs and usable decode graph memory pool. If one stage is fully captured, and there is unused memory left within usable graph memory pool, vLLM will attempt further graph capture for the other stage, until no more HPU Graphs can be captured without exceeding reserved memory pool. The behavior on that mechanism can be observed in the example below.
284 -```
284 +:::
285
286 Each described step is logged by vLLM server, as follows (negative values correspond to memory being released):
287
@@ -352,13 +352,13 @@ INFO 08-02 17:38:43 hpu_executor.py:91] init_cache_engine took 37.92 GiB of devi
352
353 - `VLLM_{phase}_{dim}_BUCKET_{param}` - collection of 12 environment variables configuring ranges of bucketing mechanism
354
355 - - `{phase}` is either `PROMPT` or `DECODE`
355 + * `{phase}` is either `PROMPT` or `DECODE`
356
357 - - `{dim}` is either `BS`, `SEQ` or `BLOCK`
357 + * `{dim}` is either `BS`, `SEQ` or `BLOCK`
358
359 - - `{param}` is either `MIN`, `STEP` or `MAX`
359 + * `{param}` is either `MIN`, `STEP` or `MAX`
360
361 - - Default values:
361 + * Default values:
362
363 - Prompt:
364 - batch size min (`VLLM_PROMPT_BS_BUCKET_MIN`): `1`
docs/source/getting_started/installation/ai_accelerator/index.md
+136 -136
@@ -2,374 +2,374 @@
2
3 vLLM is a Python library that supports the following AI accelerators. Select your AI accelerator type to see vendor specific instructions:
4
5 -::::{tab-set}
5 +:::::{tab-set}
6 :sync-group: device
7
8 -:::{tab-item} TPU
8 +::::{tab-item} TPU
9 :sync: tpu
10
11 -```{include} tpu.inc.md
11 +:::{include} tpu.inc.md
12 :start-after: "# Installation"
13 :end-before: "## Requirements"
14 -```
15 -
14 :::
15
18 -:::{tab-item} Intel Gaudi
16 +::::
17 +
18 +::::{tab-item} Intel Gaudi
19 :sync: hpu-gaudi
20
21 -```{include} hpu-gaudi.inc.md
21 +:::{include} hpu-gaudi.inc.md
22 :start-after: "# Installation"
23 :end-before: "## Requirements"
24 -```
25 -
24 :::
25
28 -:::{tab-item} Neuron
26 +::::
27 +
28 +::::{tab-item} Neuron
29 :sync: neuron
30
31 -```{include} neuron.inc.md
31 +:::{include} neuron.inc.md
32 :start-after: "# Installation"
33 :end-before: "## Requirements"
34 -```
35 -
34 :::
35
38 -:::{tab-item} OpenVINO
36 +::::
37 +
38 +::::{tab-item} OpenVINO
39 :sync: openvino
40
41 -```{include} openvino.inc.md
41 +:::{include} openvino.inc.md
42 :start-after: "# Installation"
43 :end-before: "## Requirements"
44 -```
45 -
44 :::
45
46 ::::
47
48 +:::::
49 +
50 ## Requirements
51
52 -::::{tab-set}
52 +:::::{tab-set}
53 :sync-group: device
54
55 -:::{tab-item} TPU
55 +::::{tab-item} TPU
56 :sync: tpu
57
58 -```{include} tpu.inc.md
58 +:::{include} tpu.inc.md
59 :start-after: "## Requirements"
60 :end-before: "## Configure a new environment"
61 -```
62 -
61 :::
62
65 -:::{tab-item} Intel Gaudi
63 +::::
64 +
65 +::::{tab-item} Intel Gaudi
66 :sync: hpu-gaudi
67
68 -```{include} hpu-gaudi.inc.md
68 +:::{include} hpu-gaudi.inc.md
69 :start-after: "## Requirements"
70 :end-before: "## Configure a new environment"
71 -```
72 -
71 :::
72
75 -:::{tab-item} Neuron
73 +::::
74 +
75 +::::{tab-item} Neuron
76 :sync: neuron
77
78 -```{include} neuron.inc.md
78 +:::{include} neuron.inc.md
79 :start-after: "## Requirements"
80 :end-before: "## Configure a new environment"
81 -```
82 -
81 :::
82
85 -:::{tab-item} OpenVINO
83 +::::
84 +
85 +::::{tab-item} OpenVINO
86 :sync: openvino
87
88 -```{include} openvino.inc.md
88 +:::{include} openvino.inc.md
89 :start-after: "## Requirements"
90 :end-before: "## Set up using Python"
91 -```
92 -
91 :::
92
93 ::::
94
95 +:::::
96 +
97 ## Configure a new environment
98
99 -::::{tab-set}
99 +:::::{tab-set}
100 :sync-group: device
101
102 -:::{tab-item} TPU
102 +::::{tab-item} TPU
103 :sync: tpu
104
105 -```{include} tpu.inc.md
105 +:::{include} tpu.inc.md
106 :start-after: "## Configure a new environment"
107 :end-before: "## Set up using Python"
108 -```
109 -
108 :::
109
112 -:::{tab-item} Intel Gaudi
110 +::::
111 +
112 +::::{tab-item} Intel Gaudi
113 :sync: hpu-gaudi
114
115 -```{include} hpu-gaudi.inc.md
115 +:::{include} hpu-gaudi.inc.md
116 :start-after: "## Configure a new environment"
117 :end-before: "## Set up using Python"
118 -```
119 -
118 :::
119
122 -:::{tab-item} Neuron
120 +::::
121 +
122 +::::{tab-item} Neuron
123 :sync: neuron
124
125 -```{include} neuron.inc.md
125 +:::{include} neuron.inc.md
126 :start-after: "## Configure a new environment"
127 :end-before: "## Set up using Python"
128 -```
129 -
128 :::
129
132 -:::{tab-item} OpenVINO
133 -:sync: openvino
130 +::::
131
135 -```{include} ../python_env_setup.inc.md
136 -```
132 +::::{tab-item} OpenVINO
133 +:sync: openvino
134
135 +:::{include} ../python_env_setup.inc.md
136 :::
137
138 ::::
139
140 +:::::
141 +
142 ## Set up using Python
143
144 ### Pre-built wheels
145
146 -::::{tab-set}
146 +:::::{tab-set}
147 :sync-group: device
148
149 -:::{tab-item} TPU
149 +::::{tab-item} TPU
150 :sync: tpu
151
152 -```{include} tpu.inc.md
152 +:::{include} tpu.inc.md
153 :start-after: "### Pre-built wheels"
154 :end-before: "### Build wheel from source"
155 -```
156 -
155 :::
156
159 -:::{tab-item} Intel Gaudi
157 +::::
158 +
159 +::::{tab-item} Intel Gaudi
160 :sync: hpu-gaudi
161
162 -```{include} hpu-gaudi.inc.md
162 +:::{include} hpu-gaudi.inc.md
163 :start-after: "### Pre-built wheels"
164 :end-before: "### Build wheel from source"
165 -```
166 -
165 :::
166
169 -:::{tab-item} Neuron
167 +::::
168 +
169 +::::{tab-item} Neuron
170 :sync: neuron
171
172 -```{include} neuron.inc.md
172 +:::{include} neuron.inc.md
173 :start-after: "### Pre-built wheels"
174 :end-before: "### Build wheel from source"
175 -```
176 -
175 :::
176
179 -:::{tab-item} OpenVINO
177 +::::
178 +
179 +::::{tab-item} OpenVINO
180 :sync: openvino
181
182 -```{include} openvino.inc.md
182 +:::{include} openvino.inc.md
183 :start-after: "### Pre-built wheels"
184 :end-before: "### Build wheel from source"
185 -```
186 -
185 :::
186
187 ::::
188
189 +:::::
190 +
191 ### Build wheel from source
192
193 -::::{tab-set}
193 +:::::{tab-set}
194 :sync-group: device
195
196 -:::{tab-item} TPU
196 +::::{tab-item} TPU
197 :sync: tpu
198
199 -```{include} tpu.inc.md
199 +:::{include} tpu.inc.md
200 :start-after: "### Build wheel from source"
201 :end-before: "## Set up using Docker"
202 -```
203 -
202 :::
203
206 -:::{tab-item} Intel Gaudi
204 +::::
205 +
206 +::::{tab-item} Intel Gaudi
207 :sync: hpu-gaudi
208
209 -```{include} hpu-gaudi.inc.md
209 +:::{include} hpu-gaudi.inc.md
210 :start-after: "### Build wheel from source"
211 :end-before: "## Set up using Docker"
212 -```
213 -
212 :::
213
216 -:::{tab-item} Neuron
214 +::::
215 +
216 +::::{tab-item} Neuron
217 :sync: neuron
218
219 -```{include} neuron.inc.md
219 +:::{include} neuron.inc.md
220 :start-after: "### Build wheel from source"
221 :end-before: "## Set up using Docker"
222 -```
223 -
222 :::
223
226 -:::{tab-item} OpenVINO
224 +::::
225 +
226 +::::{tab-item} OpenVINO
227 :sync: openvino
228
229 -```{include} openvino.inc.md
229 +:::{include} openvino.inc.md
230 :start-after: "### Build wheel from source"
231 :end-before: "## Set up using Docker"
232 -```
233 -
232 :::
233
234 ::::
235
236 +:::::
237 +
238 ## Set up using Docker
239
240 ### Pre-built images
241
242 -::::{tab-set}
242 +:::::{tab-set}
243 :sync-group: device
244
245 -:::{tab-item} TPU
245 +::::{tab-item} TPU
246 :sync: tpu
247
248 -```{include} tpu.inc.md
248 +:::{include} tpu.inc.md
249 :start-after: "### Pre-built images"
250 :end-before: "### Build image from source"
251 -```
252 -
251 :::
252
255 -:::{tab-item} Intel Gaudi
253 +::::
254 +
255 +::::{tab-item} Intel Gaudi
256 :sync: hpu-gaudi
257
258 -```{include} hpu-gaudi.inc.md
258 +:::{include} hpu-gaudi.inc.md
259 :start-after: "### Pre-built images"
260 :end-before: "### Build image from source"
261 -```
262 -
261 :::
262
265 -:::{tab-item} Neuron
263 +::::
264 +
265 +::::{tab-item} Neuron
266 :sync: neuron
267
268 -```{include} neuron.inc.md
268 +:::{include} neuron.inc.md
269 :start-after: "### Pre-built images"
270 :end-before: "### Build image from source"
271 -```
272 -
271 :::
272
275 -:::{tab-item} OpenVINO
273 +::::
274 +
275 +::::{tab-item} OpenVINO
276 :sync: openvino
277
278 -```{include} openvino.inc.md
278 +:::{include} openvino.inc.md
279 :start-after: "### Pre-built images"
280 :end-before: "### Build image from source"
281 -```
282 -
281 :::
282
283 ::::
284
285 +:::::
286 +
287 ### Build image from source
288
289 -::::{tab-set}
289 +:::::{tab-set}
290 :sync-group: device
291
292 -:::{tab-item} TPU
292 +::::{tab-item} TPU
293 :sync: tpu
294
295 -```{include} tpu.inc.md
295 +:::{include} tpu.inc.md
296 :start-after: "### Build image from source"
297 :end-before: "## Extra information"
298 -```
299 -
298 :::
299
302 -:::{tab-item} Intel Gaudi
300 +::::
301 +
302 +::::{tab-item} Intel Gaudi
303 :sync: hpu-gaudi
304
305 -```{include} hpu-gaudi.inc.md
305 +:::{include} hpu-gaudi.inc.md
306 :start-after: "### Build image from source"
307 :end-before: "## Extra information"
308 -```
309 -
308 :::
309
312 -:::{tab-item} Neuron
310 +::::
311 +
312 +::::{tab-item} Neuron
313 :sync: neuron
314
315 -```{include} neuron.inc.md
315 +:::{include} neuron.inc.md
316 :start-after: "### Build image from source"
317 :end-before: "## Extra information"
318 -```
319 -
318 :::
319
322 -:::{tab-item} OpenVINO
320 +::::
321 +
322 +::::{tab-item} OpenVINO
323 :sync: openvino
324
325 -```{include} openvino.inc.md
325 +:::{include} openvino.inc.md
326 :start-after: "### Build image from source"
327 :end-before: "## Extra information"
328 -```
329 -
328 :::
329
330 ::::
331
332 +:::::
333 +
334 ## Extra information
335
336 -::::{tab-set}
336 +:::::{tab-set}
337 :sync-group: device
338
339 -:::{tab-item} TPU
339 +::::{tab-item} TPU
340 :sync: tpu
341
342 -```{include} tpu.inc.md
342 +:::{include} tpu.inc.md
343 :start-after: "## Extra information"
344 -```
345 -
344 :::
345
348 -:::{tab-item} Intel Gaudi
346 +::::
347 +
348 +::::{tab-item} Intel Gaudi
349 :sync: hpu-gaudi
350
351 -```{include} hpu-gaudi.inc.md
351 +:::{include} hpu-gaudi.inc.md
352 :start-after: "## Extra information"
353 -```
354 -
353 :::
354
357 -:::{tab-item} Neuron
355 +::::
356 +
357 +::::{tab-item} Neuron
358 :sync: neuron
359
360 -```{include} neuron.inc.md
360 +:::{include} neuron.inc.md
361 :start-after: "## Extra information"
362 -```
363 -
362 :::
363
366 -:::{tab-item} OpenVINO
364 +::::
365 +
366 +::::{tab-item} OpenVINO
367 :sync: openvino
368
369 -```{include} openvino.inc.md
369 +:::{include} openvino.inc.md
370 :start-after: "## Extra information"
371 -```
372 -
371 :::
372
373 ::::
374 +
375 +:::::
docs/source/getting_started/installation/ai_accelerator/neuron.inc.md
+2 -2
@@ -67,9 +67,9 @@ Currently, there are no pre-built Neuron wheels.
67
68 ### Build wheel from source
69
70 -```{note}
70 +:::{note}
71 The currently supported version of Pytorch for Neuron installs `triton` version `2.1.0`. This is incompatible with `vllm >= 0.5.3`. You may see an error `cannot import name 'default_dump_dir...`. To work around this, run a `pip install --upgrade triton==3.0.0` after installing the vLLM wheel.
72 -```
72 +:::
73
74 Following instructions are applicable to Neuron SDK 2.16 and beyond.
75
docs/source/getting_started/installation/ai_accelerator/tpu.inc.md
+26 -25
@@ -47,10 +47,10 @@ When you request queued resources, the request is added to a queue maintained by
47 the Cloud TPU service. When the requested resource becomes available, it's
48 assigned to your Google Cloud project for your immediate exclusive use.
49
50 -```{note}
50 +:::{note}
51 In all of the following commands, replace the ALL CAPS parameter names with
52 appropriate values. See the parameter descriptions table for more information.
53 -```
53 +:::
54
55 ### Provision Cloud TPUs with GKE
56
@@ -75,33 +75,33 @@ gcloud alpha compute tpus queued-resources create QUEUED_RESOURCE_ID \
75 --service-account SERVICE_ACCOUNT
76 ```
77
78 -```{list-table} Parameter descriptions
78 +:::{list-table} Parameter descriptions
79 :header-rows: 1
80
81 -* - Parameter name
82 - - Description
83 -* - QUEUED_RESOURCE_ID
84 - - The user-assigned ID of the queued resource request.
85 -* - TPU_NAME
86 - - The user-assigned name of the TPU which is created when the queued
81 +- * Parameter name
82 + * Description
83 +- * QUEUED_RESOURCE_ID
84 + * The user-assigned ID of the queued resource request.
85 +- * TPU_NAME
86 + * The user-assigned name of the TPU which is created when the queued
87 resource request is allocated.
88 -* - PROJECT_ID
89 - - Your Google Cloud project
90 -* - ZONE
91 - - The GCP zone where you want to create your Cloud TPU. The value you use
88 +- * PROJECT_ID
89 + * Your Google Cloud project
90 +- * ZONE
91 + * The GCP zone where you want to create your Cloud TPU. The value you use
92 depends on the version of TPUs you are using. For more information, see
93 `TPU regions and zones <https://cloud.google.com/tpu/docs/regions-zones>`_
94 -* - ACCELERATOR_TYPE
95 - - The TPU version you want to use. Specify the TPU version, for example
94 +- * ACCELERATOR_TYPE
95 + * The TPU version you want to use. Specify the TPU version, for example
96 `v5litepod-4` specifies a v5e TPU with 4 cores. For more information,
97 see `TPU versions <https://cloud.devsite.corp.google.com/tpu/docs/system-architecture-tpu-vm#versions>`_.
98 -* - RUNTIME_VERSION
99 - - The TPU VM runtime version to use. For more information see `TPU VM images <https://cloud.google.com/tpu/docs/runtimes>`_.
100 -* - SERVICE_ACCOUNT
101 - - The email address for your service account. You can find it in the IAM
98 +- * RUNTIME_VERSION
99 + * The TPU VM runtime version to use. For more information see `TPU VM images <https://cloud.google.com/tpu/docs/runtimes>`_.
100 +- * SERVICE_ACCOUNT
101 + * The email address for your service account. You can find it in the IAM
102 Cloud Console under *Service Accounts*. For example:
103 `tpu-service-account@<your_project_ID>.iam.gserviceaccount.com`
104 -```
104 +:::
105
106 Connect to your TPU using SSH:
107
@@ -178,15 +178,15 @@ Run the Docker image with the following command:
178 docker run --privileged --net host --shm-size=16G -it vllm-tpu
179 ```
180
181 -```{note}
181 +:::{note}
182 Since TPU relies on XLA which requires static shapes, vLLM bucketizes the
183 possible input shapes and compiles an XLA graph for each shape. The
184 compilation time may take 20~30 minutes in the first run. However, the
185 compilation time reduces to ~5 minutes afterwards because the XLA graphs are
186 cached in the disk (in {code}`VLLM_XLA_CACHE_PATH` or {code}`~/.cache/vllm/xla_cache` by default).
187 -```
187 +:::
188
189 -````{tip}
189 +:::{tip}
190 If you encounter the following error:
191
192 ```console
@@ -198,9 +198,10 @@ file or directory
198 Install OpenBLAS with the following command:
199
200 ```console
201 -$ sudo apt-get install libopenblas-base libopenmpi-dev libomp-dev
201 +sudo apt-get install libopenblas-base libopenmpi-dev libomp-dev
202 ```
203 -````
203 +
204 +:::
205
206 ## Extra information
207
docs/source/getting_started/installation/cpu/apple.inc.md
+2 -2
@@ -25,9 +25,9 @@ pip install -r requirements-cpu.txt
25 pip install -e .
26 ```
27
28 -```{note}
28 +:::{note}
29 On macOS the `VLLM_TARGET_DEVICE` is automatically set to `cpu`, which currently is the only supported device.
30 -```
30 +:::
31
32 #### Troubleshooting
33
docs/source/getting_started/installation/cpu/index.md
+43 -43
@@ -2,86 +2,86 @@
2
3 vLLM is a Python library that supports the following CPU variants. Select your CPU type to see vendor specific instructions:
4
5 -::::{tab-set}
5 +:::::{tab-set}
6 :sync-group: device
7
8 -:::{tab-item} x86
8 +::::{tab-item} x86
9 :sync: x86
10
11 -```{include} x86.inc.md
11 +:::{include} x86.inc.md
12 :start-after: "# Installation"
13 :end-before: "## Requirements"
14 -```
15 -
14 :::
15
18 -:::{tab-item} ARM
16 +::::
17 +
18 +::::{tab-item} ARM
19 :sync: arm
20
21 -```{include} arm.inc.md
21 +:::{include} arm.inc.md
22 :start-after: "# Installation"
23 :end-before: "## Requirements"
24 -```
25 -
24 :::
25
28 -:::{tab-item} Apple silicon
26 +::::
27 +
28 +::::{tab-item} Apple silicon
29 :sync: apple
30
31 -```{include} apple.inc.md
31 +:::{include} apple.inc.md
32 :start-after: "# Installation"
33 :end-before: "## Requirements"
34 -```
35 -
34 :::
35
36 ::::
37
38 +:::::
39 +
40 ## Requirements
41
42 - Python: 3.9 -- 3.12
43
44 -::::{tab-set}
44 +:::::{tab-set}
45 :sync-group: device
46
47 -:::{tab-item} x86
47 +::::{tab-item} x86
48 :sync: x86
49
50 -```{include} x86.inc.md
50 +:::{include} x86.inc.md
51 :start-after: "## Requirements"
52 :end-before: "## Set up using Python"
53 -```
54 -
53 :::
54
57 -:::{tab-item} ARM
55 +::::
56 +
57 +::::{tab-item} ARM
58 :sync: arm
59
60 -```{include} arm.inc.md
60 +:::{include} arm.inc.md
61 :start-after: "## Requirements"
62 :end-before: "## Set up using Python"
63 -```
64 -
63 :::
64
67 -:::{tab-item} Apple silicon
65 +::::
66 +
67 +::::{tab-item} Apple silicon
68 :sync: apple
69
70 -```{include} apple.inc.md
70 +:::{include} apple.inc.md
71 :start-after: "## Requirements"
72 :end-before: "## Set up using Python"
73 -```
74 -
73 :::
74
75 ::::
76
77 +:::::
78 +
79 ## Set up using Python
80
81 ### Create a new Python environment
82
83 -```{include} ../python_env_setup.inc.md
84 -```
83 +:::{include} ../python_env_setup.inc.md
84 +:::
85
86 ### Pre-built wheels
87
@@ -89,41 +89,41 @@ Currently, there are no pre-built CPU wheels.
89
90 ### Build wheel from source
91
92 -::::{tab-set}
92 +:::::{tab-set}
93 :sync-group: device
94
95 -:::{tab-item} x86
95 +::::{tab-item} x86
96 :sync: x86
97
98 -```{include} x86.inc.md
98 +:::{include} x86.inc.md
99 :start-after: "### Build wheel from source"
100 :end-before: "## Set up using Docker"
101 -```
102 -
101 :::
102
105 -:::{tab-item} ARM
103 +::::
104 +
105 +::::{tab-item} ARM
106 :sync: arm
107
108 -```{include} arm.inc.md
108 +:::{include} arm.inc.md
109 :start-after: "### Build wheel from source"
110 :end-before: "## Set up using Docker"
111 -```
112 -
111 :::
112
115 -:::{tab-item} Apple silicon
113 +::::
114 +
115 +::::{tab-item} Apple silicon
116 :sync: apple
117
118 -```{include} apple.inc.md
118 +:::{include} apple.inc.md
119 :start-after: "### Build wheel from source"
120 :end-before: "## Set up using Docker"
121 -```
122 -
121 :::
122
123 ::::
124
125 +:::::
126 +
127 ## Set up using Docker
128
129 ### Pre-built images
@@ -142,9 +142,9 @@ $ docker run -it \
142 vllm-cpu-env
143 ```
144
145 -:::{tip}
145 +::::{tip}
146 For ARM or Apple silicon, use `Dockerfile.arm`
147 -:::
147 +::::
148
149 ## Supported features
150
docs/source/getting_started/installation/cpu/x86.inc.md
+2 -2
@@ -17,10 +17,10 @@ vLLM initially supports basic model inferencing and serving on x86 CPU platform,
17 :::{include} build.inc.md
18 :::
19
20 -```{note}
20 +:::{note}
21 - AVX512_BF16 is an extension ISA provides native BF16 data type conversion and vector product instructions, which brings some performance improvement compared with pure AVX512. The CPU backend build script will check the host CPU flags to determine whether to enable AVX512_BF16.
22 - If you want to force enable AVX512_BF16 for the cross-compilation, please set environment variable `VLLM_CPU_AVX512BF16=1` before the building.
23 -```
23 +:::
24
25 ## Set up using Docker
26
docs/source/getting_started/installation/gpu/cuda.inc.md
+6 -6
@@ -10,9 +10,9 @@ vLLM contains pre-compiled C++ and CUDA (12.1) binaries.
10
11 ### Create a new Python environment
12
13 -```{note}
13 +:::{note}
14 PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
15 -```
15 +:::
16
17 In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.
18
@@ -100,10 +100,10 @@ pip install --editable .
100
101 You can find more information about vLLM's wheels in <project:#install-the-latest-code>.
102
103 -```{note}
103 +:::{note}
104 There is a possibility that your source code may have a different commit ID compared to the latest vLLM wheel, which could potentially lead to unknown errors.
105 It is recommended to use the same commit ID for the source code as the vLLM wheel you have installed. Please refer to <project:#install-the-latest-code> for instructions on how to install a specified wheel.
106 -```
106 +:::
107
108 #### Full build (with compilation)
109
@@ -115,7 +115,7 @@ cd vllm
115 pip install -e .
116 ```
117
118 -```{tip}
118 +:::{tip}
119 Building from source requires a lot of compilation. If you are building from source repeatedly, it's more efficient to cache the compilation results.
120
121 For example, you can install [ccache](https://github.com/ccache/ccache) using `conda install ccache` or `apt install ccache` .
@@ -123,7 +123,7 @@ As long as `which ccache` command can find the `ccache` binary, it will be used
123
124 [sccache](https://github.com/mozilla/sccache) works similarly to `ccache`, but has the capability to utilize caching in remote storage environments.
125 The following environment variables can be set to configure the vLLM `sccache` remote: `SCCACHE_BUCKET=vllm-build-sccache SCCACHE_REGION=us-west-2 SCCACHE_S3_NO_CREDENTIALS=1`. We also recommend setting `SCCACHE_IDLE_TIMEOUT=0`.
126 -```
126 +:::
127
128 ##### Use an existing PyTorch installation
129
docs/source/getting_started/installation/gpu/index.md
+103 -103
@@ -2,299 +2,299 @@
2
3 vLLM is a Python library that supports the following GPU variants. Select your GPU type to see vendor specific instructions:
4
5 -::::{tab-set}
5 +:::::{tab-set}
6 :sync-group: device
7
8 -:::{tab-item} CUDA
8 +::::{tab-item} CUDA
9 :sync: cuda
10
11 -```{include} cuda.inc.md
11 +:::{include} cuda.inc.md
12 :start-after: "# Installation"
13 :end-before: "## Requirements"
14 -```
15 -
14 :::
15
18 -:::{tab-item} ROCm
16 +::::
17 +
18 +::::{tab-item} ROCm
19 :sync: rocm
20
21 -```{include} rocm.inc.md
21 +:::{include} rocm.inc.md
22 :start-after: "# Installation"
23 :end-before: "## Requirements"
24 -```
25 -
24 :::
25
28 -:::{tab-item} XPU
26 +::::
27 +
28 +::::{tab-item} XPU
29 :sync: xpu
30
31 -```{include} xpu.inc.md
31 +:::{include} xpu.inc.md
32 :start-after: "# Installation"
33 :end-before: "## Requirements"
34 -```
35 -
34 :::
35
36 ::::
37
38 +:::::
39 +
40 ## Requirements
41
42 - OS: Linux
43 - Python: 3.9 -- 3.12
44
45 -::::{tab-set}
45 +:::::{tab-set}
46 :sync-group: device
47
48 -:::{tab-item} CUDA
48 +::::{tab-item} CUDA
49 :sync: cuda
50
51 -```{include} cuda.inc.md
51 +:::{include} cuda.inc.md
52 :start-after: "## Requirements"
53 :end-before: "## Set up using Python"
54 -```
55 -
54 :::
55
58 -:::{tab-item} ROCm
56 +::::
57 +
58 +::::{tab-item} ROCm
59 :sync: rocm
60
61 -```{include} rocm.inc.md
61 +:::{include} rocm.inc.md
62 :start-after: "## Requirements"
63 :end-before: "## Set up using Python"
64 -```
65 -
64 :::
65
68 -:::{tab-item} XPU
66 +::::
67 +
68 +::::{tab-item} XPU
69 :sync: xpu
70
71 -```{include} xpu.inc.md
71 +:::{include} xpu.inc.md
72 :start-after: "## Requirements"
73 :end-before: "## Set up using Python"
74 -```
75 -
74 :::
75
76 ::::
77
78 +:::::
79 +
80 ## Set up using Python
81
82 ### Create a new Python environment
83
84 -```{include} ../python_env_setup.inc.md
85 -```
84 +:::{include} ../python_env_setup.inc.md
85 +:::
86
87 -::::{tab-set}
87 +:::::{tab-set}
88 :sync-group: device
89
90 -:::{tab-item} CUDA
90 +::::{tab-item} CUDA
91 :sync: cuda
92
93 -```{include} cuda.inc.md
93 +:::{include} cuda.inc.md
94 :start-after: "## Create a new Python environment"
95 :end-before: "### Pre-built wheels"
96 -```
97 -
96 :::
97
100 -:::{tab-item} ROCm
98 +::::
99 +
100 +::::{tab-item} ROCm
101 :sync: rocm
102
103 There is no extra information on creating a new Python environment for this device.
104
105 -:::
105 +::::
106
107 -:::{tab-item} XPU
107 +::::{tab-item} XPU
108 :sync: xpu
109
110 There is no extra information on creating a new Python environment for this device.
111
112 -:::
113 -
112 ::::
113
114 +:::::
115 +
116 ### Pre-built wheels
117
118 -::::{tab-set}
118 +:::::{tab-set}
119 :sync-group: device
120
121 -:::{tab-item} CUDA
121 +::::{tab-item} CUDA
122 :sync: cuda
123
124 -```{include} cuda.inc.md
124 +:::{include} cuda.inc.md
125 :start-after: "### Pre-built wheels"
126 :end-before: "### Build wheel from source"
127 -```
128 -
127 :::
128
131 -:::{tab-item} ROCm
129 +::::
130 +
131 +::::{tab-item} ROCm
132 :sync: rocm
133
134 -```{include} rocm.inc.md
134 +:::{include} rocm.inc.md
135 :start-after: "### Pre-built wheels"
136 :end-before: "### Build wheel from source"
137 -```
138 -
137 :::
138
141 -:::{tab-item} XPU
139 +::::
140 +
141 +::::{tab-item} XPU
142 :sync: xpu
143
144 -```{include} xpu.inc.md
144 +:::{include} xpu.inc.md
145 :start-after: "### Pre-built wheels"
146 :end-before: "### Build wheel from source"
147 -```
148 -
147 :::
148
149 ::::
150
151 +:::::
152 +
153 (build-from-source)=
154
155 ### Build wheel from source
156
157 -::::{tab-set}
157 +:::::{tab-set}
158 :sync-group: device
159
160 -:::{tab-item} CUDA
160 +::::{tab-item} CUDA
161 :sync: cuda
162
163 -```{include} cuda.inc.md
163 +:::{include} cuda.inc.md
164 :start-after: "### Build wheel from source"
165 :end-before: "## Set up using Docker"
166 -```
167 -
166 :::
167
170 -:::{tab-item} ROCm
168 +::::
169 +
170 +::::{tab-item} ROCm
171 :sync: rocm
172
173 -```{include} rocm.inc.md
173 +:::{include} rocm.inc.md
174 :start-after: "### Build wheel from source"
175 :end-before: "## Set up using Docker"
176 -```
177 -
176 :::
177
180 -:::{tab-item} XPU
178 +::::
179 +
180 +::::{tab-item} XPU
181 :sync: xpu
182
183 -```{include} xpu.inc.md
183 +:::{include} xpu.inc.md
184 :start-after: "### Build wheel from source"
185 :end-before: "## Set up using Docker"
186 -```
187 -
186 :::
187
188 ::::
189
190 +:::::
191 +
192 ## Set up using Docker
193
194 ### Pre-built images
195
196 -::::{tab-set}
196 +:::::{tab-set}
197 :sync-group: device
198
199 -:::{tab-item} CUDA
199 +::::{tab-item} CUDA
200 :sync: cuda
201
202 -```{include} cuda.inc.md
202 +:::{include} cuda.inc.md
203 :start-after: "### Pre-built images"
204 :end-before: "### Build image from source"
205 -```
206 -
205 :::
206
209 -:::{tab-item} ROCm
207 +::::
208 +
209 +::::{tab-item} ROCm
210 :sync: rocm
211
212 -```{include} rocm.inc.md
212 +:::{include} rocm.inc.md
213 :start-after: "### Pre-built images"
214 :end-before: "### Build image from source"
215 -```
216 -
215 :::
216
219 -:::{tab-item} XPU
217 +::::
218 +
219 +::::{tab-item} XPU
220 :sync: xpu
221
222 -```{include} xpu.inc.md
222 +:::{include} xpu.inc.md
223 :start-after: "### Pre-built images"
224 :end-before: "### Build image from source"
225 -```
226 -
225 :::
226
227 ::::
228
229 +:::::
230 +
231 ### Build image from source
232
233 -::::{tab-set}
233 +:::::{tab-set}
234 :sync-group: device
235
236 -:::{tab-item} CUDA
236 +::::{tab-item} CUDA
237 :sync: cuda
238
239 -```{include} cuda.inc.md
239 +:::{include} cuda.inc.md
240 :start-after: "### Build image from source"
241 :end-before: "## Supported features"
242 -```
243 -
242 :::
243
246 -:::{tab-item} ROCm
244 +::::
245 +
246 +::::{tab-item} ROCm
247 :sync: rocm
248
249 -```{include} rocm.inc.md
249 +:::{include} rocm.inc.md
250 :start-after: "### Build image from source"
251 :end-before: "## Supported features"
252 -```
253 -
252 :::
253
256 -:::{tab-item} XPU
254 +::::
255 +
256 +::::{tab-item} XPU
257 :sync: xpu
258
259 -```{include} xpu.inc.md
259 +:::{include} xpu.inc.md
260 :start-after: "### Build image from source"
261 :end-before: "## Supported features"
262 -```
263 -
262 :::
263
264 ::::
265
266 +:::::
267 +
268 ## Supported features
269
270 -::::{tab-set}
270 +:::::{tab-set}
271 :sync-group: device
272
273 -:::{tab-item} CUDA
273 +::::{tab-item} CUDA
274 :sync: cuda
275
276 -```{include} cuda.inc.md
276 +:::{include} cuda.inc.md
277 :start-after: "## Supported features"
278 -```
279 -
278 :::
279
282 -:::{tab-item} ROCm
280 +::::
281 +
282 +::::{tab-item} ROCm
283 :sync: rocm
284
285 -```{include} rocm.inc.md
285 +:::{include} rocm.inc.md
286 :start-after: "## Supported features"
287 -```
288 -
287 :::
288
291 -:::{tab-item} XPU
289 +::::
290 +
291 +::::{tab-item} XPU
292 :sync: xpu
293
294 -```{include} xpu.inc.md
294 +:::{include} xpu.inc.md
295 :start-after: "## Supported features"
296 -```
297 -
296 :::
297
298 ::::
299 +
300 +:::::
docs/source/getting_started/installation/gpu/rocm.inc.md
+13 -12
@@ -16,10 +16,10 @@ Currently, there are no pre-built ROCm wheels.
16 However, the [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
17 docker image designed for validating inference performance on the AMD Instinct™ MI300X accelerator.
18
19 -```{tip}
19 +:::{tip}
20 Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)
21 for instructions on how to use this prebuilt docker image.
22 -```
22 +:::
23
24 ### Build wheel from source
25
@@ -47,9 +47,9 @@ for instructions on how to use this prebuilt docker image.
47 cd ../..
48 ```
49
50 - ```{note}
51 - - If you see HTTP issue related to downloading packages during building triton, please try again as the HTTP error is intermittent.
52 - ```
50 + :::{note}
51 + If you see HTTP issue related to downloading packages during building triton, please try again as the HTTP error is intermittent.
52 + :::
53
54 2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/ROCm/flash-attention/tree/ck_tile)
55
@@ -67,9 +67,9 @@ for instructions on how to use this prebuilt docker image.
67 cd ..
68 ```
69
70 - ```{note}
71 - - You might need to downgrade the "ninja" version to 1.10 it is not used when compiling flash-attention-2 (e.g. `pip install ninja==1.10.2.4`)
72 - ```
70 + :::{note}
71 + You might need to downgrade the "ninja" version to 1.10 it is not used when compiling flash-attention-2 (e.g. `pip install ninja==1.10.2.4`)
72 + :::
73
74 3. Build vLLM. For example, vLLM on ROCM 6.2 can be built with the following steps:
75
@@ -95,17 +95,18 @@ for instructions on how to use this prebuilt docker image.
95
96 This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm installation.
97
98 - ```{tip}
98 +<!--- pyml disable-num-lines 5 ul-indent-->
99 + :::{tip}
100 - Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.
101 - Triton flash attention does not currently support sliding window attention. If using half precision, please use CK flash-attention for sliding window support.
102 - To use CK flash-attention or PyTorch naive attention, please use this flag `export VLLM_USE_TRITON_FLASH_ATTN=0` to turn off triton flash attention.
103 - The ROCm version of PyTorch, ideally, should match the ROCm driver version.
103 - ```
104 + :::
105
105 -```{tip}
106 +:::{tip}
107 - For MI300x (gfx942) users, to achieve optimal performance, please refer to [MI300x tuning guide](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) for performance optimization and tuning tips on system and workflow level.
108 For vLLM, please refer to [vLLM performance optimization](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/workload.html#vllm-performance-optimization).
108 -```
109 +:::
110
111 ## Set up using Docker
112
docs/source/getting_started/installation/gpu/xpu.inc.md
+2 -2
@@ -30,10 +30,10 @@ pip install -v -r requirements-xpu.txt
30 VLLM_TARGET_DEVICE=xpu python setup.py install
31 ```
32
33 -```{note}
33 +:::{note}
34 - FP16 is the default data type in the current XPU backend. The BF16 data
35 type will be supported in the future.
36 -```
36 +:::
37
38 ## Set up using Docker
39
docs/source/getting_started/installation/index.md
+2 -2
@@ -4,10 +4,10 @@
4
5 vLLM supports the following hardware platforms:
6
7 -```{toctree}
7 +:::{toctree}
8 :maxdepth: 1
9
10 gpu/index
11 cpu/index
12 ai_accelerator/index
13 -```
13 +:::
docs/source/getting_started/installation/python_env_setup.inc.md
+2 -2
@@ -6,9 +6,9 @@ conda create -n myenv python=3.12 -y
6 conda activate myenv
7 ```
8
9 -```{note}
9 +:::{note}
10 [PyTorch has deprecated the conda release channel](https://github.com/pytorch/pytorch/issues/138506). If you use `conda`, please only use it to create Python environment rather than installing packages.
11 -```
11 +:::
12
13 Or you can create a new Python environment using [uv](https://docs.astral.sh/uv/), a very fast Python environment manager. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following command:
14
docs/source/getting_started/quickstart.md
+6 -6
@@ -32,9 +32,9 @@ conda activate myenv
32 pip install vllm
33 ```
34
35 -```{note}
35 +:::{note}
36 For non-CUDA platforms, please refer [here](#installation-index) for specific instructions on how to install vLLM.
37 -```
37 +:::
38
39 (quickstart-offline)=
40
@@ -69,9 +69,9 @@ The {class}`~vllm.LLM` class initializes vLLM's engine and the [OPT-125M model](
69 llm = LLM(model="facebook/opt-125m")
70 ```
71
72 -```{note}
72 +:::{note}
73 By default, vLLM downloads models from [HuggingFace](https://huggingface.co/). If you would like to use models from [ModelScope](https://www.modelscope.cn), set the environment variable `VLLM_USE_MODELSCOPE` before initializing the engine.
74 -```
74 +:::
75
76 Now, the fun part! The outputs are generated using `llm.generate`. It adds the input prompts to the vLLM engine's waiting queue and executes the vLLM engine to generate the outputs with high throughput. The outputs are returned as a list of `RequestOutput` objects, which include all of the output tokens.
77
@@ -97,10 +97,10 @@ Run the following command to start the vLLM server with the [Qwen2.5-1.5B-Instru
97 vllm serve Qwen/Qwen2.5-1.5B-Instruct
98 ```
99
100 -```{note}
100 +:::{note}
101 By default, the server uses a predefined chat template stored in the tokenizer.
102 You can learn about overriding it [here](#chat-template).
103 -```
103 +:::
104
105 This server can be queried in the same format as OpenAI API. For example, to list the models:
106
docs/source/getting_started/troubleshooting.md
+6 -6
@@ -4,9 +4,9 @@
4
5 This document outlines some troubleshooting strategies you can consider. If you think you've discovered a bug, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible.
6
7 -```{note}
7 +:::{note}
8 Once you've debugged a problem, remember to turn off any debugging environment variables defined, or simply start a new shell to avoid being affected by lingering debugging settings. Otherwise, the system might be slow with debugging functionalities left activated.
9 -```
9 +:::
10
11 ## Hangs downloading a model
12
@@ -18,9 +18,9 @@ It's recommended to download the model first using the [huggingface-cli](https:/
18 If the model is large, it can take a long time to load it from disk. Pay attention to where you store the model. Some clusters have shared filesystems across nodes, e.g. a distributed filesystem or a network filesystem, which can be slow.
19 It'd be better to store the model in a local disk. Additionally, have a look at the CPU memory usage, when the model is too large it might take a lot of CPU memory, slowing down the operating system because it needs to frequently swap between disk and memory.
20
21 -```{note}
21 +:::{note}
22 To isolate the model downloading and loading issue, you can use the `--load-format dummy` argument to skip loading the model weights. This way, you can check if the model downloading and loading is the bottleneck.
23 -```
23 +:::
24
25 ## Out of memory
26
@@ -132,14 +132,14 @@ If the script runs successfully, you should see the message `sanity check is suc
132
133 If the test script hangs or crashes, usually it means the hardware/drivers are broken in some sense. You should try to contact your system administrator or hardware vendor for further assistance. As a common workaround, you can try to tune some NCCL environment variables, such as `export NCCL_P2P_DISABLE=1` to see if it helps. Please check [their documentation](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html) for more information. Please only use these environment variables as a temporary workaround, as they might affect the performance of the system. The best solution is still to fix the hardware/drivers so that the test script can run successfully.
134
135 -```{note}
135 +:::{note}
136 A multi-node environment is more complicated than a single-node one. If you see errors such as `torch.distributed.DistNetworkError`, it is likely that the network/DNS setup is incorrect. In that case, you can manually assign node rank and specify the IP via command line arguments:
137
138 - In the first node, run `NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=2 --node-rank 0 --master_addr $MASTER_ADDR test.py`.
139 - In the second node, run `NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=2 --node-rank 1 --master_addr $MASTER_ADDR test.py`.
140
141 Adjust `--nproc-per-node`, `--nnodes`, and `--node-rank` according to your setup, being sure to execute different commands (with different `--node-rank`) on different nodes.
142 -```
142 +:::
143
144 (troubleshooting-python-multiprocessing)=
145
docs/source/index.md
+24 -24
@@ -1,13 +1,13 @@
1 # Welcome to vLLM
2
3 -```{figure} ./assets/logos/vllm-logo-text-light.png
3 +:::{figure} ./assets/logos/vllm-logo-text-light.png
4 :align: center
5 :alt: vLLM
6 :class: no-scaled-link
7 :width: 60%
8 -```
8 +:::
9
10 -```{raw} html
10 +:::{raw} html
11 <p style="text-align:center">
12 <strong>Easy, fast, and cheap LLM serving for everyone
13 </strong>
@@ -19,7 +19,7 @@
19 <a class="github-button" href="https://github.com/vllm-project/vllm/subscription" data-icon="octicon-eye" data-size="large" aria-label="Watch">Watch</a>
20 <a class="github-button" href="https://github.com/vllm-project/vllm/fork" data-icon="octicon-repo-forked" data-size="large" aria-label="Fork">Fork</a>
21 </p>
22 -```
22 +:::
23
24 vLLM is a fast and easy-to-use library for LLM inference and serving.
25
@@ -58,7 +58,7 @@ For more information, check out the following:
58
59 % How to start using vLLM?
60
61 -```{toctree}
61 +:::{toctree}
62 :caption: Getting Started
63 :maxdepth: 1
64
@@ -67,11 +67,11 @@ getting_started/quickstart
67 getting_started/examples/examples_index
68 getting_started/troubleshooting
69 getting_started/faq
70 -```
70 +:::
71
72 % What does vLLM support?
73
74 -```{toctree}
74 +:::{toctree}
75 :caption: Models
76 :maxdepth: 1
77
@@ -79,11 +79,11 @@ models/generative_models
79 models/pooling_models
80 models/supported_models
81 models/extensions/index
82 -```
82 +:::
83
84 % Additional capabilities
85
86 -```{toctree}
86 +:::{toctree}
87 :caption: Features
88 :maxdepth: 1
89
@@ -96,11 +96,11 @@ features/automatic_prefix_caching
96 features/disagg_prefill
97 features/spec_decode
98 features/compatibility_matrix
99 -```
99 +:::
100
101 % Details about running vLLM
102
103 -```{toctree}
103 +:::{toctree}
104 :caption: Inference and Serving
105 :maxdepth: 1
106
@@ -113,11 +113,11 @@ serving/engine_args
113 serving/env_vars
114 serving/usage_stats
115 serving/integrations/index
116 -```
116 +:::
117
118 % Scaling up vLLM for production
119
120 -```{toctree}
120 +:::{toctree}
121 :caption: Deployment
122 :maxdepth: 1
123
@@ -126,21 +126,21 @@ deployment/k8s
126 deployment/nginx
127 deployment/frameworks/index
128 deployment/integrations/index
129 -```
129 +:::
130
131 % Making the most out of vLLM
132
133 -```{toctree}
133 +:::{toctree}
134 :caption: Performance
135 :maxdepth: 1
136
137 performance/optimization
138 performance/benchmarks
139 -```
139 +:::
140
141 % Explanation of vLLM internals
142
143 -```{toctree}
143 +:::{toctree}
144 :caption: Design Documents
145 :maxdepth: 2
146
@@ -151,11 +151,11 @@ design/kernel/paged_attention
151 design/mm_processing
152 design/automatic_prefix_caching
153 design/multiprocessing
154 -```
154 +:::
155
156 % How to contribute to the vLLM project
157
158 -```{toctree}
158 +:::{toctree}
159 :caption: Developer Guide
160 :maxdepth: 2
161
@@ -164,11 +164,11 @@ contributing/profiling/profiling_index
164 contributing/dockerfile/dockerfile
165 contributing/model/index
166 contributing/vulnerability_management
167 -```
167 +:::
168
169 % Technical API specifications
170
171 -```{toctree}
171 +:::{toctree}
172 :caption: API Reference
173 :maxdepth: 2
174
@@ -177,18 +177,18 @@ api/engine/index
177 api/inference_params
178 api/multimodal/index
179 api/model/index
180 -```
180 +:::
181
182 % Latest news and acknowledgements
183
184 -```{toctree}
184 +:::{toctree}
185 :caption: Community
186 :maxdepth: 1
187
188 community/blog
189 community/meetups
190 community/sponsors
191 -```
191 +:::
192
193 ## Indices and tables
194
docs/source/models/extensions/index.md
+2 -2
@@ -1,8 +1,8 @@
1 # Built-in Extensions
2
3 -```{toctree}
3 +:::{toctree}
4 :maxdepth: 1
5
6 runai_model_streamer
7 tensorizer
8 -```
8 +:::
docs/source/models/extensions/runai_model_streamer.md
+2 -2
@@ -48,6 +48,6 @@ You can read further about CPU buffer memory limiting [here](https://github.com/
48 vllm serve /home/meta-llama/Llama-3.2-3B-Instruct --load-format runai_streamer --model-loader-extra-config '{"memory_limit":5368709120}'
49 ```
50
51 -```{note}
51 +:::{note}
52 For further instructions about tunable parameters and additional parameters configurable through environment variables, read the [Environment Variables Documentation](https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/env-vars.md).
53 -```
53 +:::
docs/source/models/extensions/tensorizer.md
+2 -2
@@ -11,6 +11,6 @@ For more information on CoreWeave's Tensorizer, please refer to
11 [CoreWeave's Tensorizer documentation](https://github.com/coreweave/tensorizer). For more information on serializing a vLLM model, as well a general usage guide to using Tensorizer with vLLM, see
12 the [vLLM example script](https://docs.vllm.ai/en/stable/getting_started/examples/offline_inference/tensorize_vllm_model.html).
13
14 -```{note}
14 +:::{note}
15 Note that to use this feature you will need to install `tensorizer` by running `pip install vllm[tensorizer]`.
16 -```
16 +:::
docs/source/models/generative_models.md
+2 -2
@@ -70,10 +70,10 @@ The {class}`~vllm.LLM.chat` method implements chat functionality on top of {clas
70 In particular, it accepts input similar to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat)
71 and automatically applies the model's [chat template](https://huggingface.co/docs/transformers/en/chat_templating) to format the prompt.
72
73 -```{important}
73 +:::{important}
74 In general, only instruction-tuned models have a chat template.
75 Base models may perform poorly as they are not trained to respond to the chat conversation.
76 -```
76 +:::
77
78 ```python
79 llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
docs/source/models/pooling_models.md
+30 -30
@@ -8,54 +8,54 @@ In vLLM, pooling models implement the {class}`~vllm.model_executor.models.VllmMo
8 These models use a {class}`~vllm.model_executor.layers.Pooler` to extract the final hidden states of the input
9 before returning them.
10
11 -```{note}
11 +:::{note}
12 We currently support pooling models primarily as a matter of convenience.
13 As shown in the [Compatibility Matrix](#compatibility-matrix), most vLLM features are not applicable to
14 pooling models as they only work on the generation or decode stage, so performance may not improve as much.
15 -```
15 +:::
16
17 For pooling models, we support the following `--task` options.
18 The selected option sets the default pooler used to extract the final hidden states:
19
20 -```{list-table}
20 +:::{list-table}
21 :widths: 50 25 25 25
22 :header-rows: 1
23
24 -* - Task
25 - - Pooling Type
26 - - Normalization
27 - - Softmax
28 -* - Embedding (`embed`)
29 - - `LAST`
30 - - ✅︎
31 - - ✗
32 -* - Classification (`classify`)
33 - - `LAST`
34 - - ✗
35 - - ✅︎
36 -* - Sentence Pair Scoring (`score`)
37 - - \*
38 - - \*
39 - - \*
40 -* - Reward Modeling (`reward`)
41 - - `ALL`
42 - - ✗
43 - - ✗
44 -```
24 +- * Task
25 + * Pooling Type
26 + * Normalization
27 + * Softmax
28 +- * Embedding (`embed`)
29 + * `LAST`
30 + * ✅︎
31 + * ✗
32 +- * Classification (`classify`)
33 + * `LAST`
34 + * ✗
35 + * ✅︎
36 +- * Sentence Pair Scoring (`score`)
37 + * \*
38 + * \*
39 + * \*
40 +- * Reward Modeling (`reward`)
41 + * `ALL`
42 + * ✗
43 + * ✗
44 +:::
45
46 \*The default pooler is always defined by the model.
47
48 -```{note}
48 +:::{note}
49 If the model's implementation in vLLM defines its own pooler, the default pooler is set to that instead of the one specified in this table.
50 -```
50 +:::
51
52 When loading [Sentence Transformers](https://huggingface.co/sentence-transformers) models,
53 we attempt to override the default pooler based on its Sentence Transformers configuration file (`modules.json`).
54
55 -```{tip}
55 +:::{tip}
56 You can customize the model's pooling method via the `--override-pooler-config` option,
57 which takes priority over both the model's and Sentence Transformers's defaults.
58 -```
58 +:::
59
60 ## Offline Inference
61
@@ -111,10 +111,10 @@ The {class}`~vllm.LLM.score` method outputs similarity scores between sentence p
111 It is primarily designed for [cross-encoder models](https://www.sbert.net/examples/applications/cross-encoder/README.html).
112 These types of models serve as rerankers between candidate query-document pairs in RAG systems.
113
114 -```{note}
114 +:::{note}
115 vLLM can only perform the model inference component (e.g. embedding, reranking) of RAG.
116 To handle RAG at a higher level, you should use integration frameworks such as [LangChain](https://github.com/langchain-ai/langchain).
117 -```
117 +:::
118
119 ```python
120 llm = LLM(model="BAAI/bge-reranker-v2-m3", task="score")
docs/source/models/supported_models.md
+633 -630
@@ -17,7 +17,7 @@ By default, vLLM loads models from [HuggingFace (HF) Hub](https://huggingface.co
17 To determine whether a given model is supported, you can check the `config.json` file inside the HF repository.
18 If the `"architectures"` field contains a model architecture listed below, then it should be supported in theory.
19
20 -````{tip}
20 +:::{tip}
21 The easiest way to check if your model is really supported at runtime is to run the program below:
22
23 ```python
@@ -35,7 +35,7 @@ print(output)
35 ```
36
37 If vLLM successfully returns text (for generative models) or hidden states (for pooling models), it indicates that your model is supported.
38 -````
38 +:::
39
40 Otherwise, please refer to [Adding a New Model](#new-model) for instructions on how to implement your model in vLLM.
41 Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) to request vLLM support.
@@ -72,364 +72,364 @@ See [this page](#generative-models) for more information on how to use generativ
72
73 #### Text Generation (`--task generate`)
74
75 -```{list-table}
75 +:::{list-table}
76 :widths: 25 25 50 5 5
77 :header-rows: 1
78
79 -* - Architecture
80 - - Models
81 - - Example HF Models
82 - - [LoRA](#lora-adapter)
83 - - [PP](#distributed-serving)
84 -* - `AquilaForCausalLM`
85 - - Aquila, Aquila2
86 - - `BAAI/Aquila-7B`, `BAAI/AquilaChat-7B`, etc.
87 - - ✅︎
88 - - ✅︎
89 -* - `ArcticForCausalLM`
90 - - Arctic
91 - - `Snowflake/snowflake-arctic-base`, `Snowflake/snowflake-arctic-instruct`, etc.
92 - -
93 - - ✅︎
94 -* - `BaiChuanForCausalLM`
95 - - Baichuan2, Baichuan
96 - - `baichuan-inc/Baichuan2-13B-Chat`, `baichuan-inc/Baichuan-7B`, etc.
97 - - ✅︎
98 - - ✅︎
99 -* - `BloomForCausalLM`
100 - - BLOOM, BLOOMZ, BLOOMChat
101 - - `bigscience/bloom`, `bigscience/bloomz`, etc.
102 - -
103 - - ✅︎
104 -* - `BartForConditionalGeneration`
105 - - BART
106 - - `facebook/bart-base`, `facebook/bart-large-cnn`, etc.
107 - -
108 - -
109 -* - `ChatGLMModel`
110 - - ChatGLM
111 - - `THUDM/chatglm2-6b`, `THUDM/chatglm3-6b`, etc.
112 - - ✅︎
113 - - ✅︎
114 -* - `CohereForCausalLM`, `Cohere2ForCausalLM`
115 - - Command-R
116 - - `CohereForAI/c4ai-command-r-v01`, `CohereForAI/c4ai-command-r7b-12-2024`, etc.
117 - - ✅︎
118 - - ✅︎
119 -* - `DbrxForCausalLM`
120 - - DBRX
121 - - `databricks/dbrx-base`, `databricks/dbrx-instruct`, etc.
122 - -
123 - - ✅︎
124 -* - `DeciLMForCausalLM`
125 - - DeciLM
126 - - `Deci/DeciLM-7B`, `Deci/DeciLM-7B-instruct`, etc.
127 - -
128 - - ✅︎
129 -* - `DeepseekForCausalLM`
130 - - DeepSeek
131 - - `deepseek-ai/deepseek-llm-67b-base`, `deepseek-ai/deepseek-llm-7b-chat` etc.
132 - -
133 - - ✅︎
134 -* - `DeepseekV2ForCausalLM`
135 - - DeepSeek-V2
136 - - `deepseek-ai/DeepSeek-V2`, `deepseek-ai/DeepSeek-V2-Chat` etc.
137 - -
138 - - ✅︎
139 -* - `DeepseekV3ForCausalLM`
140 - - DeepSeek-V3
141 - - `deepseek-ai/DeepSeek-V3-Base`, `deepseek-ai/DeepSeek-V3` etc.
142 - -
143 - - ✅︎
144 -* - `ExaoneForCausalLM`
145 - - EXAONE-3
146 - - `LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
147 - - ✅︎
148 - - ✅︎
149 -* - `FalconForCausalLM`
150 - - Falcon
151 - - `tiiuae/falcon-7b`, `tiiuae/falcon-40b`, `tiiuae/falcon-rw-7b`, etc.
152 - -
153 - - ✅︎
154 -* - `FalconMambaForCausalLM`
155 - - FalconMamba
156 - - `tiiuae/falcon-mamba-7b`, `tiiuae/falcon-mamba-7b-instruct`, etc.
157 - - ✅︎
158 - - ✅︎
159 -* - `GemmaForCausalLM`
160 - - Gemma
161 - - `google/gemma-2b`, `google/gemma-7b`, etc.
162 - - ✅︎
163 - - ✅︎
164 -* - `Gemma2ForCausalLM`
165 - - Gemma2
166 - - `google/gemma-2-9b`, `google/gemma-2-27b`, etc.
167 - - ✅︎
168 - - ✅︎
169 -* - `GlmForCausalLM`
170 - - GLM-4
171 - - `THUDM/glm-4-9b-chat-hf`, etc.
172 - - ✅︎
173 - - ✅︎
174 -* - `GPT2LMHeadModel`
175 - - GPT-2
176 - - `gpt2`, `gpt2-xl`, etc.
177 - -
178 - - ✅︎
179 -* - `GPTBigCodeForCausalLM`
180 - - StarCoder, SantaCoder, WizardCoder
181 - - `bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, `WizardLM/WizardCoder-15B-V1.0`, etc.
182 - - ✅︎
183 - - ✅︎
184 -* - `GPTJForCausalLM`
185 - - GPT-J
186 - - `EleutherAI/gpt-j-6b`, `nomic-ai/gpt4all-j`, etc.
187 - -
188 - - ✅︎
189 -* - `GPTNeoXForCausalLM`
190 - - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
191 - - `EleutherAI/gpt-neox-20b`, `EleutherAI/pythia-12b`, `OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, `databricks/dolly-v2-12b`, `stabilityai/stablelm-tuned-alpha-7b`, etc.
192 - -
193 - - ✅︎
194 -* - `GraniteForCausalLM`
195 - - Granite 3.0, Granite 3.1, PowerLM
196 - - `ibm-granite/granite-3.0-2b-base`, `ibm-granite/granite-3.1-8b-instruct`, `ibm/PowerLM-3b`, etc.
197 - - ✅︎
198 - - ✅︎
199 -* - `GraniteMoeForCausalLM`
200 - - Granite 3.0 MoE, PowerMoE
201 - - `ibm-granite/granite-3.0-1b-a400m-base`, `ibm-granite/granite-3.0-3b-a800m-instruct`, `ibm/PowerMoE-3b`, etc.
202 - - ✅︎
203 - - ✅︎
204 -* - `GritLM`
205 - - GritLM
206 - - `parasail-ai/GritLM-7B-vllm`.
207 - - ✅︎
208 - - ✅︎
209 -* - `InternLMForCausalLM`
210 - - InternLM
211 - - `internlm/internlm-7b`, `internlm/internlm-chat-7b`, etc.
212 - - ✅︎
213 - - ✅︎
214 -* - `InternLM2ForCausalLM`
215 - - InternLM2
216 - - `internlm/internlm2-7b`, `internlm/internlm2-chat-7b`, etc.
217 - - ✅︎
218 - - ✅︎
219 -* - `InternLM3ForCausalLM`
220 - - InternLM3
221 - - `internlm/internlm3-8b-instruct`, etc.
222 - - ✅︎
223 - - ✅︎
224 -* - `JAISLMHeadModel`
225 - - Jais
226 - - `inceptionai/jais-13b`, `inceptionai/jais-13b-chat`, `inceptionai/jais-30b-v3`, `inceptionai/jais-30b-chat-v3`, etc.
227 - -
228 - - ✅︎
229 -* - `JambaForCausalLM`
230 - - Jamba
231 - - `ai21labs/AI21-Jamba-1.5-Large`, `ai21labs/AI21-Jamba-1.5-Mini`, `ai21labs/Jamba-v0.1`, etc.
232 - - ✅︎
233 - - ✅︎
234 -* - `LlamaForCausalLM`
235 - - Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
236 - - `meta-llama/Meta-Llama-3.1-405B-Instruct`, `meta-llama/Meta-Llama-3.1-70B`, `meta-llama/Meta-Llama-3-70B-Instruct`, `meta-llama/Llama-2-70b-hf`, `01-ai/Yi-34B`, etc.
237 - - ✅︎
238 - - ✅︎
239 -* - `MambaForCausalLM`
240 - - Mamba
241 - - `state-spaces/mamba-130m-hf`, `state-spaces/mamba-790m-hf`, `state-spaces/mamba-2.8b-hf`, etc.
242 - -
243 - - ✅︎
244 -* - `MiniCPMForCausalLM`
245 - - MiniCPM
246 - - `openbmb/MiniCPM-2B-sft-bf16`, `openbmb/MiniCPM-2B-dpo-bf16`, `openbmb/MiniCPM-S-1B-sft`, etc.
247 - - ✅︎
248 - - ✅︎
249 -* - `MiniCPM3ForCausalLM`
250 - - MiniCPM3
251 - - `openbmb/MiniCPM3-4B`, etc.
252 - - ✅︎
253 - - ✅︎
254 -* - `MistralForCausalLM`
255 - - Mistral, Mistral-Instruct
256 - - `mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc.
257 - - ✅︎
258 - - ✅︎
259 -* - `MixtralForCausalLM`
260 - - Mixtral-8x7B, Mixtral-8x7B-Instruct
261 - - `mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc.
262 - - ✅︎
263 - - ✅︎
264 -* - `MPTForCausalLM`
265 - - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
266 - - `mosaicml/mpt-7b`, `mosaicml/mpt-7b-storywriter`, `mosaicml/mpt-30b`, etc.
267 - -
268 - - ✅︎
269 -* - `NemotronForCausalLM`
270 - - Nemotron-3, Nemotron-4, Minitron
271 - - `nvidia/Minitron-8B-Base`, `mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
272 - - ✅︎
273 - - ✅︎
274 -* - `OLMoForCausalLM`
275 - - OLMo
276 - - `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc.
277 - -
278 - - ✅︎
279 -* - `OLMo2ForCausalLM`
280 - - OLMo2
281 - - `allenai/OLMo2-7B-1124`, etc.
282 - -
283 - - ✅︎
284 -* - `OLMoEForCausalLM`
285 - - OLMoE
286 - - `allenai/OLMoE-1B-7B-0924`, `allenai/OLMoE-1B-7B-0924-Instruct`, etc.
287 - - ✅︎
288 - - ✅︎
289 -* - `OPTForCausalLM`
290 - - OPT, OPT-IML
291 - - `facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.
292 - -
293 - - ✅︎
294 -* - `OrionForCausalLM`
295 - - Orion
296 - - `OrionStarAI/Orion-14B-Base`, `OrionStarAI/Orion-14B-Chat`, etc.
297 - -
298 - - ✅︎
299 -* - `PhiForCausalLM`
300 - - Phi
301 - - `microsoft/phi-1_5`, `microsoft/phi-2`, etc.
302 - - ✅︎
303 - - ✅︎
304 -* - `Phi3ForCausalLM`
305 - - Phi-4, Phi-3
306 - - `microsoft/Phi-4`, `microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-mini-128k-instruct`, `microsoft/Phi-3-medium-128k-instruct`, etc.
307 - - ✅︎
308 - - ✅︎
309 -* - `Phi3SmallForCausalLM`
310 - - Phi-3-Small
311 - - `microsoft/Phi-3-small-8k-instruct`, `microsoft/Phi-3-small-128k-instruct`, etc.
312 - -
313 - - ✅︎
314 -* - `PhiMoEForCausalLM`
315 - - Phi-3.5-MoE
316 - - `microsoft/Phi-3.5-MoE-instruct`, etc.
317 - - ✅︎
318 - - ✅︎
319 -* - `PersimmonForCausalLM`
320 - - Persimmon
321 - - `adept/persimmon-8b-base`, `adept/persimmon-8b-chat`, etc.
322 - -
323 - - ✅︎
324 -* - `QWenLMHeadModel`
325 - - Qwen
326 - - `Qwen/Qwen-7B`, `Qwen/Qwen-7B-Chat`, etc.
327 - - ✅︎
328 - - ✅︎
329 -* - `Qwen2ForCausalLM`
330 - - QwQ, Qwen2
331 - - `Qwen/QwQ-32B-Preview`, `Qwen/Qwen2-7B-Instruct`, `Qwen/Qwen2-7B`, etc.
332 - - ✅︎
333 - - ✅︎
334 -* - `Qwen2MoeForCausalLM`
335 - - Qwen2MoE
336 - - `Qwen/Qwen1.5-MoE-A2.7B`, `Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
337 - -
338 - - ✅︎
339 -* - `StableLmForCausalLM`
340 - - StableLM
341 - - `stabilityai/stablelm-3b-4e1t`, `stabilityai/stablelm-base-alpha-7b-v2`, etc.
342 - -
343 - - ✅︎
344 -* - `Starcoder2ForCausalLM`
345 - - Starcoder2
346 - - `bigcode/starcoder2-3b`, `bigcode/starcoder2-7b`, `bigcode/starcoder2-15b`, etc.
347 - -
348 - - ✅︎
349 -* - `SolarForCausalLM`
350 - - Solar Pro
351 - - `upstage/solar-pro-preview-instruct`, etc.
352 - - ✅︎
353 - - ✅︎
354 -* - `TeleChat2ForCausalLM`
355 - - TeleChat2
356 - - `TeleAI/TeleChat2-3B`, `TeleAI/TeleChat2-7B`, `TeleAI/TeleChat2-35B`, etc.
357 - - ✅︎
358 - - ✅︎
359 -* - `XverseForCausalLM`
360 - - XVERSE
361 - - `xverse/XVERSE-7B-Chat`, `xverse/XVERSE-13B-Chat`, `xverse/XVERSE-65B-Chat`, etc.
362 - - ✅︎
363 - - ✅︎
364 -```
365 -
366 -```{note}
79 +- * Architecture
80 + * Models
81 + * Example HF Models
82 + * [LoRA](#lora-adapter)
83 + * [PP](#distributed-serving)
84 +- * `AquilaForCausalLM`
85 + * Aquila, Aquila2
86 + * `BAAI/Aquila-7B`, `BAAI/AquilaChat-7B`, etc.
87 + * ✅︎
88 + * ✅︎
89 +- * `ArcticForCausalLM`
90 + * Arctic
91 + * `Snowflake/snowflake-arctic-base`, `Snowflake/snowflake-arctic-instruct`, etc.
92 + *
93 + * ✅︎
94 +- * `BaiChuanForCausalLM`
95 + * Baichuan2, Baichuan
96 + * `baichuan-inc/Baichuan2-13B-Chat`, `baichuan-inc/Baichuan-7B`, etc.
97 + * ✅︎
98 + * ✅︎
99 +- * `BloomForCausalLM`
100 + * BLOOM, BLOOMZ, BLOOMChat
101 + * `bigscience/bloom`, `bigscience/bloomz`, etc.
102 + *
103 + * ✅︎
104 +- * `BartForConditionalGeneration`
105 + * BART
106 + * `facebook/bart-base`, `facebook/bart-large-cnn`, etc.
107 + *
108 + *
109 +- * `ChatGLMModel`
110 + * ChatGLM
111 + * `THUDM/chatglm2-6b`, `THUDM/chatglm3-6b`, etc.
112 + * ✅︎
113 + * ✅︎
114 +- * `CohereForCausalLM`, `Cohere2ForCausalLM`
115 + * Command-R
116 + * `CohereForAI/c4ai-command-r-v01`, `CohereForAI/c4ai-command-r7b-12-2024`, etc.
117 + * ✅︎
118 + * ✅︎
119 +- * `DbrxForCausalLM`
120 + * DBRX
121 + * `databricks/dbrx-base`, `databricks/dbrx-instruct`, etc.
122 + *
123 + * ✅︎
124 +- * `DeciLMForCausalLM`
125 + * DeciLM
126 + * `Deci/DeciLM-7B`, `Deci/DeciLM-7B-instruct`, etc.
127 + *
128 + * ✅︎
129 +- * `DeepseekForCausalLM`
130 + * DeepSeek
131 + * `deepseek-ai/deepseek-llm-67b-base`, `deepseek-ai/deepseek-llm-7b-chat` etc.
132 + *
133 + * ✅︎
134 +- * `DeepseekV2ForCausalLM`
135 + * DeepSeek-V2
136 + * `deepseek-ai/DeepSeek-V2`, `deepseek-ai/DeepSeek-V2-Chat` etc.
137 + *
138 + * ✅︎
139 +- * `DeepseekV3ForCausalLM`
140 + * DeepSeek-V3
141 + * `deepseek-ai/DeepSeek-V3-Base`, `deepseek-ai/DeepSeek-V3` etc.
142 + *
143 + * ✅︎
144 +- * `ExaoneForCausalLM`
145 + * EXAONE-3
146 + * `LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
147 + * ✅︎
148 + * ✅︎
149 +- * `FalconForCausalLM`
150 + * Falcon
151 + * `tiiuae/falcon-7b`, `tiiuae/falcon-40b`, `tiiuae/falcon-rw-7b`, etc.
152 + *
153 + * ✅︎
154 +- * `FalconMambaForCausalLM`
155 + * FalconMamba
156 + * `tiiuae/falcon-mamba-7b`, `tiiuae/falcon-mamba-7b-instruct`, etc.
157 + * ✅︎
158 + * ✅︎
159 +- * `GemmaForCausalLM`
160 + * Gemma
161 + * `google/gemma-2b`, `google/gemma-7b`, etc.
162 + * ✅︎
163 + * ✅︎
164 +- * `Gemma2ForCausalLM`
165 + * Gemma2
166 + * `google/gemma-2-9b`, `google/gemma-2-27b`, etc.
167 + * ✅︎
168 + * ✅︎
169 +- * `GlmForCausalLM`
170 + * GLM-4
171 + * `THUDM/glm-4-9b-chat-hf`, etc.
172 + * ✅︎
173 + * ✅︎
174 +- * `GPT2LMHeadModel`
175 + * GPT-2
176 + * `gpt2`, `gpt2-xl`, etc.
177 + *
178 + * ✅︎
179 +- * `GPTBigCodeForCausalLM`
180 + * StarCoder, SantaCoder, WizardCoder
181 + * `bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, `WizardLM/WizardCoder-15B-V1.0`, etc.
182 + * ✅︎
183 + * ✅︎
184 +- * `GPTJForCausalLM`
185 + * GPT-J
186 + * `EleutherAI/gpt-j-6b`, `nomic-ai/gpt4all-j`, etc.
187 + *
188 + * ✅︎
189 +- * `GPTNeoXForCausalLM`
190 + * GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
191 + * `EleutherAI/gpt-neox-20b`, `EleutherAI/pythia-12b`, `OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, `databricks/dolly-v2-12b`, `stabilityai/stablelm-tuned-alpha-7b`, etc.
192 + *
193 + * ✅︎
194 +- * `GraniteForCausalLM`
195 + * Granite 3.0, Granite 3.1, PowerLM
196 + * `ibm-granite/granite-3.0-2b-base`, `ibm-granite/granite-3.1-8b-instruct`, `ibm/PowerLM-3b`, etc.
197 + * ✅︎
198 + * ✅︎
199 +- * `GraniteMoeForCausalLM`
200 + * Granite 3.0 MoE, PowerMoE
201 + * `ibm-granite/granite-3.0-1b-a400m-base`, `ibm-granite/granite-3.0-3b-a800m-instruct`, `ibm/PowerMoE-3b`, etc.
202 + * ✅︎
203 + * ✅︎
204 +- * `GritLM`
205 + * GritLM
206 + * `parasail-ai/GritLM-7B-vllm`.
207 + * ✅︎
208 + * ✅︎
209 +- * `InternLMForCausalLM`
210 + * InternLM
211 + * `internlm/internlm-7b`, `internlm/internlm-chat-7b`, etc.
212 + * ✅︎
213 + * ✅︎
214 +- * `InternLM2ForCausalLM`
215 + * InternLM2
216 + * `internlm/internlm2-7b`, `internlm/internlm2-chat-7b`, etc.
217 + * ✅︎
218 + * ✅︎
219 +- * `InternLM3ForCausalLM`
220 + * InternLM3
221 + * `internlm/internlm3-8b-instruct`, etc.
222 + * ✅︎
223 + * ✅︎
224 +- * `JAISLMHeadModel`
225 + * Jais
226 + * `inceptionai/jais-13b`, `inceptionai/jais-13b-chat`, `inceptionai/jais-30b-v3`, `inceptionai/jais-30b-chat-v3`, etc.
227 + *
228 + * ✅︎
229 +- * `JambaForCausalLM`
230 + * Jamba
231 + * `ai21labs/AI21-Jamba-1.5-Large`, `ai21labs/AI21-Jamba-1.5-Mini`, `ai21labs/Jamba-v0.1`, etc.
232 + * ✅︎
233 + * ✅︎
234 +- * `LlamaForCausalLM`
235 + * Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
236 + * `meta-llama/Meta-Llama-3.1-405B-Instruct`, `meta-llama/Meta-Llama-3.1-70B`, `meta-llama/Meta-Llama-3-70B-Instruct`, `meta-llama/Llama-2-70b-hf`, `01-ai/Yi-34B`, etc.
237 + * ✅︎
238 + * ✅︎
239 +- * `MambaForCausalLM`
240 + * Mamba
241 + * `state-spaces/mamba-130m-hf`, `state-spaces/mamba-790m-hf`, `state-spaces/mamba-2.8b-hf`, etc.
242 + *
243 + * ✅︎
244 +- * `MiniCPMForCausalLM`
245 + * MiniCPM
246 + * `openbmb/MiniCPM-2B-sft-bf16`, `openbmb/MiniCPM-2B-dpo-bf16`, `openbmb/MiniCPM-S-1B-sft`, etc.
247 + * ✅︎
248 + * ✅︎
249 +- * `MiniCPM3ForCausalLM`
250 + * MiniCPM3
251 + * `openbmb/MiniCPM3-4B`, etc.
252 + * ✅︎
253 + * ✅︎
254 +- * `MistralForCausalLM`
255 + * Mistral, Mistral-Instruct
256 + * `mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc.
257 + * ✅︎
258 + * ✅︎
259 +- * `MixtralForCausalLM`
260 + * Mixtral-8x7B, Mixtral-8x7B-Instruct
261 + * `mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc.
262 + * ✅︎
263 + * ✅︎
264 +- * `MPTForCausalLM`
265 + * MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
266 + * `mosaicml/mpt-7b`, `mosaicml/mpt-7b-storywriter`, `mosaicml/mpt-30b`, etc.
267 + *
268 + * ✅︎
269 +- * `NemotronForCausalLM`
270 + * Nemotron-3, Nemotron-4, Minitron
271 + * `nvidia/Minitron-8B-Base`, `mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
272 + * ✅︎
273 + * ✅︎
274 +- * `OLMoForCausalLM`
275 + * OLMo
276 + * `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc.
277 + *
278 + * ✅︎
279 +- * `OLMo2ForCausalLM`
280 + * OLMo2
281 + * `allenai/OLMo2-7B-1124`, etc.
282 + *
283 + * ✅︎
284 +- * `OLMoEForCausalLM`
285 + * OLMoE
286 + * `allenai/OLMoE-1B-7B-0924`, `allenai/OLMoE-1B-7B-0924-Instruct`, etc.
287 + * ✅︎
288 + * ✅︎
289 +- * `OPTForCausalLM`
290 + * OPT, OPT-IML
291 + * `facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.
292 + *
293 + * ✅︎
294 +- * `OrionForCausalLM`
295 + * Orion
296 + * `OrionStarAI/Orion-14B-Base`, `OrionStarAI/Orion-14B-Chat`, etc.
297 + *
298 + * ✅︎
299 +- * `PhiForCausalLM`
300 + * Phi
301 + * `microsoft/phi-1_5`, `microsoft/phi-2`, etc.
302 + * ✅︎
303 + * ✅︎
304 +- * `Phi3ForCausalLM`
305 + * Phi-4, Phi-3
306 + * `microsoft/Phi-4`, `microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-mini-128k-instruct`, `microsoft/Phi-3-medium-128k-instruct`, etc.
307 + * ✅︎
308 + * ✅︎
309 +- * `Phi3SmallForCausalLM`
310 + * Phi-3-Small
311 + * `microsoft/Phi-3-small-8k-instruct`, `microsoft/Phi-3-small-128k-instruct`, etc.
312 + *
313 + * ✅︎
314 +- * `PhiMoEForCausalLM`
315 + * Phi-3.5-MoE
316 + * `microsoft/Phi-3.5-MoE-instruct`, etc.
317 + * ✅︎
318 + * ✅︎
319 +- * `PersimmonForCausalLM`
320 + * Persimmon
321 + * `adept/persimmon-8b-base`, `adept/persimmon-8b-chat`, etc.
322 + *
323 + * ✅︎
324 +- * `QWenLMHeadModel`
325 + * Qwen
326 + * `Qwen/Qwen-7B`, `Qwen/Qwen-7B-Chat`, etc.
327 + * ✅︎
328 + * ✅︎
329 +- * `Qwen2ForCausalLM`
330 + * QwQ, Qwen2
331 + * `Qwen/QwQ-32B-Preview`, `Qwen/Qwen2-7B-Instruct`, `Qwen/Qwen2-7B`, etc.
332 + * ✅︎
333 + * ✅︎
334 +- * `Qwen2MoeForCausalLM`
335 + * Qwen2MoE
336 + * `Qwen/Qwen1.5-MoE-A2.7B`, `Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
337 + *
338 + * ✅︎
339 +- * `StableLmForCausalLM`
340 + * StableLM
341 + * `stabilityai/stablelm-3b-4e1t`, `stabilityai/stablelm-base-alpha-7b-v2`, etc.
342 + *
343 + * ✅︎
344 +- * `Starcoder2ForCausalLM`
345 + * Starcoder2
346 + * `bigcode/starcoder2-3b`, `bigcode/starcoder2-7b`, `bigcode/starcoder2-15b`, etc.
347 + *
348 + * ✅︎
349 +- * `SolarForCausalLM`
350 + * Solar Pro
351 + * `upstage/solar-pro-preview-instruct`, etc.
352 + * ✅︎
353 + * ✅︎
354 +- * `TeleChat2ForCausalLM`
355 + * TeleChat2
356 + * `TeleAI/TeleChat2-3B`, `TeleAI/TeleChat2-7B`, `TeleAI/TeleChat2-35B`, etc.
357 + * ✅︎
358 + * ✅︎
359 +- * `XverseForCausalLM`
360 + * XVERSE
361 + * `xverse/XVERSE-7B-Chat`, `xverse/XVERSE-13B-Chat`, `xverse/XVERSE-65B-Chat`, etc.
362 + * ✅︎
363 + * ✅︎
364 +:::
365 +
366 +:::{note}
367 Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.
368 -```
368 +:::
369
370 ### Pooling Models
371
372 See [this page](pooling-models) for more information on how to use pooling models.
373
374 -```{important}
374 +:::{important}
375 Since some model architectures support both generative and pooling tasks,
376 you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.
377 -```
377 +:::
378
379 #### Text Embedding (`--task embed`)
380
381 -```{list-table}
381 +:::{list-table}
382 :widths: 25 25 50 5 5
383 :header-rows: 1
384
385 -* - Architecture
386 - - Models
387 - - Example HF Models
388 - - [LoRA](#lora-adapter)
389 - - [PP](#distributed-serving)
390 -* - `BertModel`
391 - - BERT-based
392 - - `BAAI/bge-base-en-v1.5`, etc.
393 - -
394 - -
395 -* - `Gemma2Model`
396 - - Gemma2-based
397 - - `BAAI/bge-multilingual-gemma2`, etc.
398 - -
399 - - ✅︎
400 -* - `GritLM`
401 - - GritLM
402 - - `parasail-ai/GritLM-7B-vllm`.
403 - - ✅︎
404 - - ✅︎
405 -* - `LlamaModel`, `LlamaForCausalLM`, `MistralModel`, etc.
406 - - Llama-based
407 - - `intfloat/e5-mistral-7b-instruct`, etc.
408 - - ✅︎
409 - - ✅︎
410 -* - `Qwen2Model`, `Qwen2ForCausalLM`
411 - - Qwen2-based
412 - - `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc.
413 - - ✅︎
414 - - ✅︎
415 -* - `RobertaModel`, `RobertaForMaskedLM`
416 - - RoBERTa-based
417 - - `sentence-transformers/all-roberta-large-v1`, `sentence-transformers/all-roberta-large-v1`, etc.
418 - -
419 - -
420 -* - `XLMRobertaModel`
421 - - XLM-RoBERTa-based
422 - - `intfloat/multilingual-e5-large`, etc.
423 - -
424 - -
425 -```
426 -
427 -```{note}
385 +- * Architecture
386 + * Models
387 + * Example HF Models
388 + * [LoRA](#lora-adapter)
389 + * [PP](#distributed-serving)
390 +- * `BertModel`
391 + * BERT-based
392 + * `BAAI/bge-base-en-v1.5`, etc.
393 + *
394 + *
395 +- * `Gemma2Model`
396 + * Gemma2-based
397 + * `BAAI/bge-multilingual-gemma2`, etc.
398 + *
399 + * ✅︎
400 +- * `GritLM`
401 + * GritLM
402 + * `parasail-ai/GritLM-7B-vllm`.
403 + * ✅︎
404 + * ✅︎
405 +- * `LlamaModel`, `LlamaForCausalLM`, `MistralModel`, etc.
406 + * Llama-based
407 + * `intfloat/e5-mistral-7b-instruct`, etc.
408 + * ✅︎
409 + * ✅︎
410 +- * `Qwen2Model`, `Qwen2ForCausalLM`
411 + * Qwen2-based
412 + * `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc.
413 + * ✅︎
414 + * ✅︎
415 +- * `RobertaModel`, `RobertaForMaskedLM`
416 + * RoBERTa-based
417 + * `sentence-transformers/all-roberta-large-v1`, `sentence-transformers/all-roberta-large-v1`, etc.
418 + *
419 + *
420 +- * `XLMRobertaModel`
421 + * XLM-RoBERTa-based
422 + * `intfloat/multilingual-e5-large`, etc.
423 + *
424 + *
425 +:::
426 +
427 +:::{note}
428 `ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.
429 You should manually set mean pooling by passing `--override-pooler-config '{"pooling_type": "MEAN"}'`.
430 -```
430 +:::
431
432 -```{note}
432 +:::{note}
433 Unlike base Qwen2, `Alibaba-NLP/gte-Qwen2-7B-instruct` uses bi-directional attention.
434 You can set `--hf-overrides '{"is_causal": false}'` to change the attention mask accordingly.
435
@@ -438,7 +438,7 @@ despite being described otherwise on its model card.
438
439 Regardless of the variant, you need to enable `--trust-remote-code` for the correct tokenizer to be
440 loaded. See [relevant issue on HF Transformers](https://github.com/huggingface/transformers/issues/34882).
441 -```
441 +:::
442
443 If your model is not in the above list, we will try to automatically convert the model using
444 {func}`~vllm.model_executor.models.adapters.as_embedding_model`. By default, the embeddings
@@ -446,98 +446,98 @@ of the whole prompt are extracted from the normalized hidden state corresponding
446
447 #### Reward Modeling (`--task reward`)
448
449 -```{list-table}
449 +:::{list-table}
450 :widths: 25 25 50 5 5
451 :header-rows: 1
452
453 -* - Architecture
454 - - Models
455 - - Example HF Models
456 - - [LoRA](#lora-adapter)
457 - - [PP](#distributed-serving)
458 -* - `InternLM2ForRewardModel`
459 - - InternLM2-based
460 - - `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc.
461 - - ✅︎
462 - - ✅︎
463 -* - `LlamaForCausalLM`
464 - - Llama-based
465 - - `peiyi9979/math-shepherd-mistral-7b-prm`, etc.
466 - - ✅︎
467 - - ✅︎
468 -* - `Qwen2ForRewardModel`
469 - - Qwen2-based
470 - - `Qwen/Qwen2.5-Math-RM-72B`, etc.
471 - - ✅︎
472 - - ✅︎
473 -* - `Qwen2ForProcessRewardModel`
474 - - Qwen2-based
475 - - `Qwen/Qwen2.5-Math-PRM-7B`, `Qwen/Qwen2.5-Math-PRM-72B`, etc.
476 - - ✅︎
477 - - ✅︎
478 -```
453 +- * Architecture
454 + * Models
455 + * Example HF Models
456 + * [LoRA](#lora-adapter)
457 + * [PP](#distributed-serving)
458 +- * `InternLM2ForRewardModel`
459 + * InternLM2-based
460 + * `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc.
461 + * ✅︎
462 + * ✅︎
463 +- * `LlamaForCausalLM`
464 + * Llama-based
465 + * `peiyi9979/math-shepherd-mistral-7b-prm`, etc.
466 + * ✅︎
467 + * ✅︎
468 +- * `Qwen2ForRewardModel`
469 + * Qwen2-based
470 + * `Qwen/Qwen2.5-Math-RM-72B`, etc.
471 + * ✅︎
472 + * ✅︎
473 +- * `Qwen2ForProcessRewardModel`
474 + * Qwen2-based
475 + * `Qwen/Qwen2.5-Math-PRM-7B`, `Qwen/Qwen2.5-Math-PRM-72B`, etc.
476 + * ✅︎
477 + * ✅︎
478 +:::
479
480 If your model is not in the above list, we will try to automatically convert the model using
481 {func}`~vllm.model_executor.models.adapters.as_reward_model`. By default, we return the hidden states of each token directly.
482
483 -```{important}
483 +:::{important}
484 For process-supervised reward models such as `peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
485 e.g.: `--override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.
486 -```
486 +:::
487
488 #### Classification (`--task classify`)
489
490 -```{list-table}
490 +:::{list-table}
491 :widths: 25 25 50 5 5
492 :header-rows: 1
493
494 -* - Architecture
495 - - Models
496 - - Example HF Models
497 - - [LoRA](#lora-adapter)
498 - - [PP](#distributed-serving)
499 -* - `JambaForSequenceClassification`
500 - - Jamba
501 - - `ai21labs/Jamba-tiny-reward-dev`, etc.
502 - - ✅︎
503 - - ✅︎
504 -* - `Qwen2ForSequenceClassification`
505 - - Qwen2-based
506 - - `jason9693/Qwen2.5-1.5B-apeach`, etc.
507 - - ✅︎
508 - - ✅︎
509 -```
494 +- * Architecture
495 + * Models
496 + * Example HF Models
497 + * [LoRA](#lora-adapter)
498 + * [PP](#distributed-serving)
499 +- * `JambaForSequenceClassification`
500 + * Jamba
501 + * `ai21labs/Jamba-tiny-reward-dev`, etc.
502 + * ✅︎
503 + * ✅︎
504 +- * `Qwen2ForSequenceClassification`
505 + * Qwen2-based
506 + * `jason9693/Qwen2.5-1.5B-apeach`, etc.
507 + * ✅︎
508 + * ✅︎
509 +:::
510
511 If your model is not in the above list, we will try to automatically convert the model using
512 {func}`~vllm.model_executor.models.adapters.as_classification_model`. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.
513
514 #### Sentence Pair Scoring (`--task score`)
515
516 -```{list-table}
516 +:::{list-table}
517 :widths: 25 25 50 5 5
518 :header-rows: 1
519
520 -* - Architecture
521 - - Models
522 - - Example HF Models
523 - - [LoRA](#lora-adapter)
524 - - [PP](#distributed-serving)
525 -* - `BertForSequenceClassification`
526 - - BERT-based
527 - - `cross-encoder/ms-marco-MiniLM-L-6-v2`, etc.
528 - -
529 - -
530 -* - `RobertaForSequenceClassification`
531 - - RoBERTa-based
532 - - `cross-encoder/quora-roberta-base`, etc.
533 - -
534 - -
535 -* - `XLMRobertaForSequenceClassification`
536 - - XLM-RoBERTa-based
537 - - `BAAI/bge-reranker-v2-m3`, etc.
538 - -
539 - -
540 -```
520 +- * Architecture
521 + * Models
522 + * Example HF Models
523 + * [LoRA](#lora-adapter)
524 + * [PP](#distributed-serving)
525 +- * `BertForSequenceClassification`
526 + * BERT-based
527 + * `cross-encoder/ms-marco-MiniLM-L-6-v2`, etc.
528 + *
529 + *
530 +- * `RobertaForSequenceClassification`
531 + * RoBERTa-based
532 + * `cross-encoder/quora-roberta-base`, etc.
533 + *
534 + *
535 +- * `XLMRobertaForSequenceClassification`
536 + * XLM-RoBERTa-based
537 + * `BAAI/bge-reranker-v2-m3`, etc.
538 + *
539 + *
540 +:::
541
542 (supported-mm-models)=
543
@@ -560,11 +560,12 @@ On the other hand, modalities separated by `/` are mutually exclusive.
560
561 See [this page](#multimodal-inputs) on how to pass multi-modal inputs to the model.
562
563 -````{important}
563 +:::{important}
564 To enable multiple multi-modal items per text prompt, you have to set `limit_mm_per_prompt` (offline inference)
565 or `--limit-mm-per-prompt` (online serving). For example, to enable passing up to 4 images per text prompt:
566
567 Offline inference:
568 +
569 ```python
570 llm = LLM(
571 model="Qwen/Qwen2-VL-7B-Instruct",
@@ -573,14 +574,16 @@ llm = LLM(
574 ```
575
576 Online serving:
577 +
578 ```bash
579 vllm serve Qwen/Qwen2-VL-7B-Instruct --limit-mm-per-prompt image=4
580 ```
579 -````
581
581 -```{note}
582 +:::
583 +
584 +:::{note}
585 vLLM currently only supports adding LoRA to the language backbone of multimodal models.
583 -```
586 +:::
587
588 ### Generative Models
589
@@ -588,256 +591,256 @@ See [this page](#generative-models) for more information on how to use generativ
591
592 #### Text Generation (`--task generate`)
593
591 -```{list-table}
594 +:::{list-table}
595 :widths: 25 25 15 20 5 5 5
596 :header-rows: 1
597
595 -* - Architecture
596 - - Models
597 - - Inputs
598 - - Example HF Models
599 - - [LoRA](#lora-adapter)
600 - - [PP](#distributed-serving)
601 - - [V1](gh-issue:8779)
602 -* - `AriaForConditionalGeneration`
603 - - Aria
604 - - T + I<sup>+</sup>
605 - - `rhymes-ai/Aria`
606 - -
607 - - ✅︎
608 - - ✅︎
609 -* - `Blip2ForConditionalGeneration`
610 - - BLIP-2
611 - - T + I<sup>E</sup>
612 - - `Salesforce/blip2-opt-2.7b`, `Salesforce/blip2-opt-6.7b`, etc.
613 - -
614 - - ✅︎
615 - - ✅︎
616 -* - `ChameleonForConditionalGeneration`
617 - - Chameleon
618 - - T + I
619 - - `facebook/chameleon-7b` etc.
620 - -
621 - - ✅︎
622 - - ✅︎
623 -* - `DeepseekVLV2ForCausalLM`
624 - - DeepSeek-VL2
625 - - T + I<sup>+</sup>
626 - - `deepseek-ai/deepseek-vl2-tiny`, `deepseek-ai/deepseek-vl2-small`, `deepseek-ai/deepseek-vl2` etc. (see note)
627 - -
628 - - ✅︎
629 - - ✅︎
630 -* - `FuyuForCausalLM`
631 - - Fuyu
632 - - T + I
633 - - `adept/fuyu-8b` etc.
634 - -
635 - - ✅︎
636 - - ✅︎
637 -* - `ChatGLMModel`
638 - - GLM-4V
639 - - T + I
640 - - `THUDM/glm-4v-9b` etc.
641 - - ✅︎
642 - - ✅︎
643 - -
644 -* - `H2OVLChatModel`
645 - - H2OVL
646 - - T + I<sup>E+</sup>
647 - - `h2oai/h2ovl-mississippi-800m`, `h2oai/h2ovl-mississippi-2b`, etc.
648 - -
649 - - ✅︎
650 - -
651 -* - `Idefics3ForConditionalGeneration`
652 - - Idefics3
653 - - T + I
654 - - `HuggingFaceM4/Idefics3-8B-Llama3` etc.
655 - - ✅︎
656 - -
657 - -
658 -* - `InternVLChatModel`
659 - - InternVL 2.5, Mono-InternVL, InternVL 2.0
660 - - T + I<sup>E+</sup>
661 - - `OpenGVLab/InternVL2_5-4B`, `OpenGVLab/Mono-InternVL-2B`, `OpenGVLab/InternVL2-4B`, etc.
662 - -
663 - - ✅︎
664 - - ✅︎
665 -* - `LlavaForConditionalGeneration`
666 - - LLaVA-1.5
667 - - T + I<sup>E+</sup>
668 - - `llava-hf/llava-1.5-7b-hf`, `TIGER-Lab/Mantis-8B-siglip-llama3` (see note), etc.
669 - -
670 - - ✅︎
671 - - ✅︎
672 -* - `LlavaNextForConditionalGeneration`
673 - - LLaVA-NeXT
674 - - T + I<sup>E+</sup>
675 - - `llava-hf/llava-v1.6-mistral-7b-hf`, `llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
676 - -
677 - - ✅︎
678 - - ✅︎
679 -* - `LlavaNextVideoForConditionalGeneration`
680 - - LLaVA-NeXT-Video
681 - - T + V
682 - - `llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
683 - -
684 - - ✅︎
685 - - ✅︎
686 -* - `LlavaOnevisionForConditionalGeneration`
687 - - LLaVA-Onevision
688 - - T + I<sup>+</sup> + V<sup>+</sup>
689 - - `llava-hf/llava-onevision-qwen2-7b-ov-hf`, `llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
690 - -
691 - - ✅︎
692 - - ✅︎
693 -* - `MiniCPMV`
694 - - MiniCPM-V
695 - - T + I<sup>E+</sup>
696 - - `openbmb/MiniCPM-V-2` (see note), `openbmb/MiniCPM-Llama3-V-2_5`, `openbmb/MiniCPM-V-2_6`, etc.
697 - - ✅︎
698 - - ✅︎
699 - -
700 -* - `MllamaForConditionalGeneration`
701 - - Llama 3.2
702 - - T + I<sup>+</sup>
703 - - `meta-llama/Llama-3.2-90B-Vision-Instruct`, `meta-llama/Llama-3.2-11B-Vision`, etc.
704 - -
705 - -
706 - -
707 -* - `MolmoForCausalLM`
708 - - Molmo
709 - - T + I
710 - - `allenai/Molmo-7B-D-0924`, `allenai/Molmo-72B-0924`, etc.
711 - - ✅︎
712 - - ✅︎
713 - - ✅︎
714 -* - `NVLM_D_Model`
715 - - NVLM-D 1.0
716 - - T + I<sup>E+</sup>
717 - - `nvidia/NVLM-D-72B`, etc.
718 - -
719 - - ✅︎
720 - - ✅︎
721 -* - `PaliGemmaForConditionalGeneration`
722 - - PaliGemma, PaliGemma 2
723 - - T + I<sup>E</sup>
724 - - `google/paligemma-3b-pt-224`, `google/paligemma-3b-mix-224`, `google/paligemma2-3b-ft-docci-448`, etc.
725 - -
726 - - ✅︎
727 - -
728 -* - `Phi3VForCausalLM`
729 - - Phi-3-Vision, Phi-3.5-Vision
730 - - T + I<sup>E+</sup>
731 - - `microsoft/Phi-3-vision-128k-instruct`, `microsoft/Phi-3.5-vision-instruct`, etc.
732 - -
733 - - ✅︎
734 - - ✅︎
735 -* - `PixtralForConditionalGeneration`
736 - - Pixtral
737 - - T + I<sup>+</sup>
738 - - `mistralai/Pixtral-12B-2409`, `mistral-community/pixtral-12b` (see note), etc.
739 - -
740 - - ✅︎
741 - - ✅︎
742 -* - `QWenLMHeadModel`
743 - - Qwen-VL
744 - - T + I<sup>E+</sup>
745 - - `Qwen/Qwen-VL`, `Qwen/Qwen-VL-Chat`, etc.
746 - - ✅︎
747 - - ✅︎
748 - - ✅︎
749 -* - `Qwen2AudioForConditionalGeneration`
750 - - Qwen2-Audio
751 - - T + A<sup>+</sup>
752 - - `Qwen/Qwen2-Audio-7B-Instruct`
753 - -
754 - - ✅︎
755 - - ✅︎
756 -* - `Qwen2VLForConditionalGeneration`
757 - - QVQ, Qwen2-VL
758 - - T + I<sup>E+</sup> + V<sup>E+</sup>
759 - - `Qwen/QVQ-72B-Preview`, `Qwen/Qwen2-VL-7B-Instruct`, `Qwen/Qwen2-VL-72B-Instruct`, etc.
760 - - ✅︎
761 - - ✅︎
762 - - ✅︎
763 -* - `UltravoxModel`
764 - - Ultravox
765 - - T + A<sup>E+</sup>
766 - - `fixie-ai/ultravox-v0_3`
767 - -
768 - - ✅︎
769 - - ✅︎
770 -```
598 +- * Architecture
599 + * Models
600 + * Inputs
601 + * Example HF Models
602 + * [LoRA](#lora-adapter)
603 + * [PP](#distributed-serving)
604 + * [V1](gh-issue:8779)
605 +- * `AriaForConditionalGeneration`
606 + * Aria
607 + * T + I<sup>+</sup>
608 + * `rhymes-ai/Aria`
609 + *
610 + * ✅︎
611 + * ✅︎
612 +- * `Blip2ForConditionalGeneration`
613 + * BLIP-2
614 + * T + I<sup>E</sup>
615 + * `Salesforce/blip2-opt-2.7b`, `Salesforce/blip2-opt-6.7b`, etc.
616 + *
617 + * ✅︎
618 + * ✅︎
619 +- * `ChameleonForConditionalGeneration`
620 + * Chameleon
621 + * T + I
622 + * `facebook/chameleon-7b` etc.
623 + *
624 + * ✅︎
625 + * ✅︎
626 +- * `DeepseekVLV2ForCausalLM`
627 + * DeepSeek-VL2
628 + * T + I<sup>+</sup>
629 + * `deepseek-ai/deepseek-vl2-tiny`, `deepseek-ai/deepseek-vl2-small`, `deepseek-ai/deepseek-vl2` etc. (see note)
630 + *
631 + * ✅︎
632 + * ✅︎
633 +- * `FuyuForCausalLM`
634 + * Fuyu
635 + * T + I
636 + * `adept/fuyu-8b` etc.
637 + *
638 + * ✅︎
639 + * ✅︎
640 +- * `ChatGLMModel`
641 + * GLM-4V
642 + * T + I
643 + * `THUDM/glm-4v-9b` etc.
644 + * ✅︎
645 + * ✅︎
646 + *
647 +- * `H2OVLChatModel`
648 + * H2OVL
649 + * T + I<sup>E+</sup>
650 + * `h2oai/h2ovl-mississippi-800m`, `h2oai/h2ovl-mississippi-2b`, etc.
651 + *
652 + * ✅︎
653 + *
654 +- * `Idefics3ForConditionalGeneration`
655 + * Idefics3
656 + * T + I
657 + * `HuggingFaceM4/Idefics3-8B-Llama3` etc.
658 + * ✅︎
659 + *
660 + *
661 +- * `InternVLChatModel`
662 + * InternVL 2.5, Mono-InternVL, InternVL 2.0
663 + * T + I<sup>E+</sup>
664 + * `OpenGVLab/InternVL2_5-4B`, `OpenGVLab/Mono-InternVL-2B`, `OpenGVLab/InternVL2-4B`, etc.
665 + *
666 + * ✅︎
667 + * ✅︎
668 +- * `LlavaForConditionalGeneration`
669 + * LLaVA-1.5
670 + * T + I<sup>E+</sup>
671 + * `llava-hf/llava-1.5-7b-hf`, `TIGER-Lab/Mantis-8B-siglip-llama3` (see note), etc.
672 + *
673 + * ✅︎
674 + * ✅︎
675 +- * `LlavaNextForConditionalGeneration`
676 + * LLaVA-NeXT
677 + * T + I<sup>E+</sup>
678 + * `llava-hf/llava-v1.6-mistral-7b-hf`, `llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
679 + *
680 + * ✅︎
681 + * ✅︎
682 +- * `LlavaNextVideoForConditionalGeneration`
683 + * LLaVA-NeXT-Video
684 + * T + V
685 + * `llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
686 + *
687 + * ✅︎
688 + * ✅︎
689 +- * `LlavaOnevisionForConditionalGeneration`
690 + * LLaVA-Onevision
691 + * T + I<sup>+</sup> + V<sup>+</sup>
692 + * `llava-hf/llava-onevision-qwen2-7b-ov-hf`, `llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
693 + *
694 + * ✅︎
695 + * ✅︎
696 +- * `MiniCPMV`
697 + * MiniCPM-V
698 + * T + I<sup>E+</sup>
699 + * `openbmb/MiniCPM-V-2` (see note), `openbmb/MiniCPM-Llama3-V-2_5`, `openbmb/MiniCPM-V-2_6`, etc.
700 + * ✅︎
701 + * ✅︎
702 + *
703 +- * `MllamaForConditionalGeneration`
704 + * Llama 3.2
705 + * T + I<sup>+</sup>
706 + * `meta-llama/Llama-3.2-90B-Vision-Instruct`, `meta-llama/Llama-3.2-11B-Vision`, etc.
707 + *
708 + *
709 + *
710 +- * `MolmoForCausalLM`
711 + * Molmo
712 + * T + I
713 + * `allenai/Molmo-7B-D-0924`, `allenai/Molmo-72B-0924`, etc.
714 + * ✅︎
715 + * ✅︎
716 + * ✅︎
717 +- * `NVLM_D_Model`
718 + * NVLM-D 1.0
719 + * T + I<sup>E+</sup>
720 + * `nvidia/NVLM-D-72B`, etc.
721 + *
722 + * ✅︎
723 + * ✅︎
724 +- * `PaliGemmaForConditionalGeneration`
725 + * PaliGemma, PaliGemma 2
726 + * T + I<sup>E</sup>
727 + * `google/paligemma-3b-pt-224`, `google/paligemma-3b-mix-224`, `google/paligemma2-3b-ft-docci-448`, etc.
728 + *
729 + * ✅︎
730 + *
731 +- * `Phi3VForCausalLM`
732 + * Phi-3-Vision, Phi-3.5-Vision
733 + * T + I<sup>E+</sup>
734 + * `microsoft/Phi-3-vision-128k-instruct`, `microsoft/Phi-3.5-vision-instruct`, etc.
735 + *
736 + * ✅︎
737 + * ✅︎
738 +- * `PixtralForConditionalGeneration`
739 + * Pixtral
740 + * T + I<sup>+</sup>
741 + * `mistralai/Pixtral-12B-2409`, `mistral-community/pixtral-12b` (see note), etc.
742 + *
743 + * ✅︎
744 + * ✅︎
745 +- * `QWenLMHeadModel`
746 + * Qwen-VL
747 + * T + I<sup>E+</sup>
748 + * `Qwen/Qwen-VL`, `Qwen/Qwen-VL-Chat`, etc.
749 + * ✅︎
750 + * ✅︎
751 + * ✅︎
752 +- * `Qwen2AudioForConditionalGeneration`
753 + * Qwen2-Audio
754 + * T + A<sup>+</sup>
755 + * `Qwen/Qwen2-Audio-7B-Instruct`
756 + *
757 + * ✅︎
758 + * ✅︎
759 +- * `Qwen2VLForConditionalGeneration`
760 + * QVQ, Qwen2-VL
761 + * T + I<sup>E+</sup> + V<sup>E+</sup>
762 + * `Qwen/QVQ-72B-Preview`, `Qwen/Qwen2-VL-7B-Instruct`, `Qwen/Qwen2-VL-72B-Instruct`, etc.
763 + * ✅︎
764 + * ✅︎
765 + * ✅︎
766 +- * `UltravoxModel`
767 + * Ultravox
768 + * T + A<sup>E+</sup>
769 + * `fixie-ai/ultravox-v0_3`
770 + *
771 + * ✅︎
772 + * ✅︎
773 +:::
774
775 <sup>E</sup> Pre-computed embeddings can be inputted for this modality.
776 <sup>+</sup> Multiple items can be inputted per text prompt for this modality.
777
775 -```{note}
778 +:::{note}
779 To use `DeepSeek-VL2` series models, you have to pass `--hf_overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}'` when running vLLM.
777 -```
780 +:::
781
779 -```{note}
782 +:::{note}
783 To use `TIGER-Lab/Mantis-8B-siglip-llama3`, you have to pass `--hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}'` when running vLLM.
781 -```
784 +:::
785
783 -```{note}
786 +:::{note}
787 The official `openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork (`HwwwH/MiniCPM-V-2`) for now.
788 For more details, please see: <gh-pr:4087#issuecomment-2250397630>
786 -```
789 +:::
790
788 -```{note}
791 +:::{note}
792 The chat template for Pixtral-HF is incorrect (see [discussion](https://huggingface.co/mistral-community/pixtral-12b/discussions/22)).
793 A corrected version is available at <gh-file:examples/template_pixtral_hf.jinja>.
791 -```
794 +:::
795
796 ### Pooling Models
797
798 See [this page](pooling-models) for more information on how to use pooling models.
799
797 -```{important}
800 +:::{important}
801 Since some model architectures support both generative and pooling tasks,
802 you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.
800 -```
803 +:::
804
805 #### Text Embedding (`--task embed`)
806
807 Any text generation model can be converted into an embedding model by passing `--task embed`.
808
806 -```{note}
809 +:::{note}
810 To get the best results, you should use pooling models that are specifically trained as such.
808 -```
811 +:::
812
813 The following table lists those that are tested in vLLM.
814
812 -```{list-table}
815 +:::{list-table}
816 :widths: 25 25 15 25 5 5
817 :header-rows: 1
818
816 -* - Architecture
817 - - Models
818 - - Inputs
819 - - Example HF Models
820 - - [LoRA](#lora-adapter)
821 - - [PP](#distributed-serving)
822 -* - `LlavaNextForConditionalGeneration`
823 - - LLaVA-NeXT-based
824 - - T / I
825 - - `royokong/e5-v`
826 - -
827 - - ✅︎
828 -* - `Phi3VForCausalLM`
829 - - Phi-3-Vision-based
830 - - T + I
831 - - `TIGER-Lab/VLM2Vec-Full`
832 - - 🚧
833 - - ✅︎
834 -* - `Qwen2VLForConditionalGeneration`
835 - - Qwen2-VL-based
836 - - T + I
837 - - `MrLight/dse-qwen2-2b-mrl-v1`
838 - -
839 - - ✅︎
840 -```
819 +- * Architecture
820 + * Models
821 + * Inputs
822 + * Example HF Models
823 + * [LoRA](#lora-adapter)
824 + * [PP](#distributed-serving)
825 +- * `LlavaNextForConditionalGeneration`
826 + * LLaVA-NeXT-based
827 + * T / I
828 + * `royokong/e5-v`
829 + *
830 + * ✅︎
831 +- * `Phi3VForCausalLM`
832 + * Phi-3-Vision-based
833 + * T + I
834 + * `TIGER-Lab/VLM2Vec-Full`
835 + * 🚧
836 + * ✅︎
837 +- * `Qwen2VLForConditionalGeneration`
838 + * Qwen2-VL-based
839 + * T + I
840 + * `MrLight/dse-qwen2-2b-mrl-v1`
841 + *
842 + * ✅︎
843 +:::
844
845 _________________
846
@@ -849,9 +852,9 @@ At vLLM, we are committed to facilitating the integration and support of third-p
852
853 2. **Best-Effort Consistency**: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.
854
852 - ```{tip}
855 + :::{tip}
856 When comparing the output of `model.generate` from HuggingFace Transformers with the output of `llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
854 - ```
857 + :::
858
859 3. **Issue Resolution and Model Updates**: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.
860
docs/source/serving/distributed_serving.md
+6 -6
@@ -14,9 +14,9 @@ In short, you should increase the number of GPUs and the number of nodes until y
14
15 After adding enough GPUs and nodes to hold the model, you can run vLLM first, which will print some logs like `# GPU blocks: 790`. Multiply the number by `16` (the block size), and you can get roughly the maximum number of tokens that can be served on the current configuration. If this number is not satisfying, e.g. you want higher throughput, you can further increase the number of GPUs or nodes, until the number of blocks is enough.
16
17 -```{note}
17 +:::{note}
18 There is one edge case: if the model fits in a single node with multiple GPUs, but the number of GPUs cannot divide the model size evenly, you can use pipeline parallelism, which splits the model along layers and supports uneven splits. In this case, the tensor parallel size should be 1 and the pipeline parallel size should be the number of GPUs.
19 -```
19 +:::
20
21 ## Running vLLM on a single node
22
@@ -94,12 +94,12 @@ vllm serve /path/to/the/model/in/the/container \
94
95 To make tensor parallel performant, you should make sure the communication between nodes is efficient, e.g. using high-speed network cards like Infiniband. To correctly set up the cluster to use Infiniband, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the `run_cluster.sh` script. Please contact your system administrator for more information on how to set up the flags. One way to confirm if the Infiniband is working is to run vLLM with `NCCL_DEBUG=TRACE` environment variable set, e.g. `NCCL_DEBUG=TRACE vllm serve ...` and check the logs for the NCCL version and the network used. If you find `[send] via NET/Socket` in the logs, it means NCCL uses raw TCP Socket, which is not efficient for cross-node tensor parallel. If you find `[send] via NET/IB/GDRDMA` in the logs, it means NCCL uses Infiniband with GPU-Direct RDMA, which is efficient.
96
97 -```{warning}
97 +:::{warning}
98 After you start the Ray cluster, you'd better also check the GPU-GPU communication between nodes. It can be non-trivial to set up. Please refer to the [sanity check script](#troubleshooting-incorrect-hardware-driver) for more information. If you need to set some environment variables for the communication configuration, you can append them to the `run_cluster.sh` script, e.g. `-e NCCL_SOCKET_IFNAME=eth0`. Note that setting environment variables in the shell (e.g. `NCCL_SOCKET_IFNAME=eth0 vllm serve ...`) only works for the processes in the same node, not for the processes in the other nodes. Setting environment variables when you create the cluster is the recommended way. See <gh-issue:6803> for more information.
99 -```
99 +:::
100
101 -```{warning}
101 +:::{warning}
102 Please make sure you downloaded the model to all the nodes (with the same path), or the model is downloaded to some distributed file system that is accessible by all nodes.
103
104 When you use huggingface repo id to refer to the model, you should append your huggingface token to the `run_cluster.sh` script, e.g. `-e HF_TOKEN=`. The recommended way is to download the model first, and then use the path to refer to the model.
105 -```
105 +:::
docs/source/serving/engine_args.md
+2
@@ -4,6 +4,7 @@
4
5 Below, you can find an explanation of every engine argument for vLLM:
6
7 +<!--- pyml disable-num-lines 7 no-space-in-emphasis-->
8 ```{eval-rst}
9 .. argparse::
10 :module: vllm.engine.arg_utils
@@ -16,6 +17,7 @@ Below, you can find an explanation of every engine argument for vLLM:
17
18 Below are the additional arguments related to the asynchronous engine:
19
20 +<!--- pyml disable-num-lines 7 no-space-in-emphasis-->
21 ```{eval-rst}
22 .. argparse::
23 :module: vllm.engine.arg_utils
docs/source/serving/env_vars.md
+4 -4
@@ -2,14 +2,14 @@
2
3 vLLM uses the following environment variables to configure the system:
4
5 -```{warning}
5 +:::{warning}
6 Please note that `VLLM_PORT` and `VLLM_HOST_IP` set the port and ip for vLLM's **internal usage**. It is not the port and ip for the API server. If you use `--host $VLLM_HOST_IP` and `--port $VLLM_PORT` to start the API server, it will not work.
7
8 All environment variables used by vLLM are prefixed with `VLLM_`. **Special care should be taken for Kubernetes users**: please do not name the service as `vllm`, otherwise environment variables set by Kubernetes might conflict with vLLM's environment variables, because [Kubernetes sets environment variables for each service with the capitalized service name as the prefix](https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables).
9 -```
9 +:::
10
11 -```{literalinclude} ../../../vllm/envs.py
11 +:::{literalinclude} ../../../vllm/envs.py
12 :end-before: end-env-vars-definition
13 :language: python
14 :start-after: begin-env-vars-definition
15 -```
15 +:::
docs/source/serving/integrations/index.md
+2 -2
@@ -1,8 +1,8 @@
1 # External Integrations
2
3 -```{toctree}
3 +:::{toctree}
4 :maxdepth: 1
5
6 langchain
7 llamaindex
8 -```
8 +:::
docs/source/serving/metrics.md
+2 -2
@@ -31,8 +31,8 @@ vllm:iteration_tokens_total_bucket{le="512.0",model_name="unsloth/Llama-3.2-1B-I
31
32 The following metrics are exposed:
33
34 -```{literalinclude} ../../../vllm/engine/metrics.py
34 +:::{literalinclude} ../../../vllm/engine/metrics.py
35 :end-before: end-metrics-definitions
36 :language: python
37 :start-after: begin-metrics-definitions
38 -```
38 +:::
docs/source/serving/multimodal_inputs.md
+28 -25
@@ -4,10 +4,10 @@
4
5 This page teaches you how to pass multi-modal inputs to [multi-modal models](#supported-mm-models) in vLLM.
6
7 -```{note}
7 +:::{note}
8 We are actively iterating on multi-modal support. See [this RFC](gh-issue:4194) for upcoming changes,
9 and [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) if you have any feedback or feature requests.
10 -```
10 +:::
11
12 ## Offline Inference
13
@@ -203,13 +203,13 @@ for o in outputs:
203
204 Our OpenAI-compatible server accepts multi-modal data via the [Chat Completions API](https://platform.openai.com/docs/api-reference/chat).
205
206 -```{important}
206 +:::{important}
207 A chat template is **required** to use Chat Completions API.
208
209 Although most models come with a chat template, for others you have to define one yourself.
210 The chat template can be inferred based on the documentation on the model's HuggingFace repo.
211 For example, LLaVA-1.5 (`llava-hf/llava-1.5-7b-hf`) requires a chat template that can be found here: <gh-file:examples/template_llava.jinja>
212 -```
212 +:::
213
214 ### Image
215
@@ -273,24 +273,25 @@ print("Chat completion output:", chat_response.choices[0].message.content)
273
274 Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
275
276 -```{tip}
276 +:::{tip}
277 Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via `--allowed-local-media-path` when launching the API server/engine,
278 and pass the file path as `url` in the API request.
279 -```
279 +:::
280
281 -```{tip}
281 +:::{tip}
282 There is no need to place image placeholders in the text content of the API request - they are already represented by the image content.
283 In fact, you can place image placeholders in the middle of the text by interleaving text and image content.
284 -```
284 +:::
285
286 -````{note}
286 +:::{note}
287 By default, the timeout for fetching images through HTTP URL is `5` seconds.
288 You can override this by setting the environment variable:
289
290 ```console
291 -$ export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
291 +export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
292 ```
293 -````
293 +
294 +:::
295
296 ### Video
297
@@ -345,14 +346,15 @@ print("Chat completion output from image url:", result)
346
347 Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
348
348 -````{note}
349 +:::{note}
350 By default, the timeout for fetching videos through HTTP URL is `30` seconds.
351 You can override this by setting the environment variable:
352
353 ```console
353 -$ export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>
354 +export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>
355 ```
355 -````
356 +
357 +:::
358
359 ### Audio
360
@@ -448,24 +450,25 @@ print("Chat completion output from audio url:", result)
450
451 Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
452
451 -````{note}
453 +:::{note}
454 By default, the timeout for fetching audios through HTTP URL is `10` seconds.
455 You can override this by setting the environment variable:
456
457 ```console
456 -$ export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>
458 +export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>
459 ```
458 -````
460 +
461 +:::
462
463 ### Embedding
464
465 vLLM's Embeddings API is a superset of OpenAI's [Embeddings API](https://platform.openai.com/docs/api-reference/embeddings),
466 where a list of chat `messages` can be passed instead of batched `inputs`. This enables multi-modal inputs to be passed to embedding models.
467
465 -```{tip}
468 +:::{tip}
469 The schema of `messages` is exactly the same as in Chat Completions API.
470 You can refer to the above tutorials for more details on how to pass each type of multi-modal data.
468 -```
471 +:::
472
473 Usually, embedding models do not expect chat-based input, so we need to use a custom chat template to format the text and images.
474 Refer to the examples below for illustration.
@@ -477,13 +480,13 @@ vllm serve TIGER-Lab/VLM2Vec-Full --task embed \
480 --trust-remote-code --max-model-len 4096 --chat-template examples/template_vlm2vec.jinja
481 ```
482
480 -```{important}
483 +:::{important}
484 Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--task embed`
485 to run this model in embedding mode instead of text generation mode.
486
487 The custom chat template is completely different from the original one for this model,
488 and can be found here: <gh-file:examples/template_vlm2vec.jinja>
486 -```
489 +:::
490
491 Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
492
@@ -518,16 +521,16 @@ vllm serve MrLight/dse-qwen2-2b-mrl-v1 --task embed \
521 --trust-remote-code --max-model-len 8192 --chat-template examples/template_dse_qwen2_vl.jinja
522 ```
523
521 -```{important}
524 +:::{important}
525 Like with VLM2Vec, we have to explicitly pass `--task embed`.
526
527 Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled
528 by a custom chat template: <gh-file:examples/template_dse_qwen2_vl.jinja>
526 -```
529 +:::
530
528 -```{important}
531 +:::{important}
532 Also important, `MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code
533 example below for details.
531 -```
534 +:::
535
536 Full example: <gh-file:examples/online_serving/openai_chat_embedding_client_for_multimodal.py>
docs/source/serving/offline_inference.md
+4 -4
@@ -22,9 +22,9 @@ The available APIs depend on the type of model that is being run:
22
23 Please refer to the above pages for more details about each API.
24
25 -```{seealso}
25 +:::{seealso}
26 [API Reference](/api/offline_inference/index)
27 -```
27 +:::
28
29 ## Configuration Options
30
@@ -70,12 +70,12 @@ llm = LLM(model="ibm-granite/granite-3.1-8b-instruct",
70 tensor_parallel_size=2)
71 ```
72
73 -```{important}
73 +:::{important}
74 To ensure that vLLM initializes CUDA correctly, you should avoid calling related functions (e.g. {func}`torch.cuda.set_device`)
75 before initializing vLLM. Otherwise, you may run into an error like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`.
76
77 To control which devices are used, please instead set the `CUDA_VISIBLE_DEVICES` environment variable.
78 -```
78 +:::
79
80 #### Quantization
81
docs/source/serving/openai_compatible_server.md
+28 -28
@@ -161,11 +161,11 @@ print(completion._request_id)
161
162 The `vllm serve` command is used to launch the OpenAI-compatible server.
163
164 -```{argparse}
164 +:::{argparse}
165 :module: vllm.entrypoints.openai.cli_args
166 :func: create_parser_for_docs
167 :prog: vllm serve
168 -```
168 +:::
169
170 #### Configuration file
171
@@ -188,10 +188,10 @@ To use the above config file:
188 vllm serve SOME_MODEL --config config.yaml
189 ```
190
191 -```{note}
191 +:::{note}
192 In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
193 The order of priorities is `command line > config file values > defaults`.
194 -```
194 +:::
195
196 ## API Reference
197
@@ -208,19 +208,19 @@ Code example: <gh-file:examples/online_serving/openai_completion_client.py>
208
209 The following [sampling parameters](#sampling-params) are supported.
210
211 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
211 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
212 :language: python
213 :start-after: begin-completion-sampling-params
214 :end-before: end-completion-sampling-params
215 -```
215 +:::
216
217 The following extra parameters are supported:
218
219 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
219 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
220 :language: python
221 :start-after: begin-completion-extra-params
222 :end-before: end-completion-extra-params
223 -```
223 +:::
224
225 (chat-api)=
226
@@ -240,19 +240,19 @@ Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
240
241 The following [sampling parameters](#sampling-params) are supported.
242
243 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
243 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
244 :language: python
245 :start-after: begin-chat-completion-sampling-params
246 :end-before: end-chat-completion-sampling-params
247 -```
247 +:::
248
249 The following extra parameters are supported:
250
251 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
251 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
252 :language: python
253 :start-after: begin-chat-completion-extra-params
254 :end-before: end-chat-completion-extra-params
255 -```
255 +:::
256
257 (embeddings-api)=
258
@@ -264,9 +264,9 @@ you can use the [official OpenAI Python client](https://github.com/openai/openai
264 If the model has a [chat template](#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](#chat-api))
265 which will be treated as a single prompt to the model.
266
267 -```{tip}
267 +:::{tip}
268 This enables multi-modal inputs to be passed to embedding models, see [this page](#multimodal-inputs) for details.
269 -```
269 +:::
270
271 Code example: <gh-file:examples/online_serving/openai_embedding_client.py>
272
@@ -274,27 +274,27 @@ Code example: <gh-file:examples/online_serving/openai_embedding_client.py>
274
275 The following [pooling parameters](#pooling-params) are supported.
276
277 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
277 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
278 :language: python
279 :start-after: begin-embedding-pooling-params
280 :end-before: end-embedding-pooling-params
281 -```
281 +:::
282
283 The following extra parameters are supported by default:
284
285 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
285 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
286 :language: python
287 :start-after: begin-embedding-extra-params
288 :end-before: end-embedding-extra-params
289 -```
289 +:::
290
291 For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:
292
293 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
293 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
294 :language: python
295 :start-after: begin-chat-embedding-extra-params
296 :end-before: end-chat-embedding-extra-params
297 -```
297 +:::
298
299 (tokenizer-api)=
300
@@ -465,19 +465,19 @@ Response:
465
466 The following [pooling parameters](#pooling-params) are supported.
467
468 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
468 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
469 :language: python
470 :start-after: begin-score-pooling-params
471 :end-before: end-score-pooling-params
472 -```
472 +:::
473
474 The following extra parameters are supported:
475
476 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
476 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
477 :language: python
478 :start-after: begin-score-extra-params
479 :end-before: end-score-extra-params
480 -```
480 +:::
481
482 (rerank-api)=
483
@@ -552,16 +552,16 @@ Response:
552
553 The following [pooling parameters](#pooling-params) are supported.
554
555 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
555 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
556 :language: python
557 :start-after: begin-rerank-pooling-params
558 :end-before: end-rerank-pooling-params
559 -```
559 +:::
560
561 The following extra parameters are supported:
562
563 -```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
563 +:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
564 :language: python
565 :start-after: begin-rerank-extra-params
566 :end-before: end-rerank-extra-params
567 -```
567 +:::
pyproject.toml
+1
@@ -111,6 +111,7 @@ markers = [
111 ]
112
113 [tool.pymarkdown]
114 +plugins.md004.style = "sublist" # ul-style
115 plugins.md013.enabled = false # line-length
116 plugins.md041.enabled = false # first-line-h1
117 plugins.md033.enabled = false # inline-html