@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 483ea64611

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

[Docs] Replace all explicit anchors with real links (#27087)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Oct 17, 2025 at 10:22 UTC 483ea646111b5ef576c2c6c833c321520de72fb2

28 files changed +54 -139

.markdownlint.yaml

-1

@@ -4,7 +4,6 @@ MD013: false

4 MD024:

5 siblings_only: true

6 MD033: false

7 -MD042: false

7 MD045: false

8 MD046: false

9 MD051: false

docs/api/README.md

+1 -7

@@ -20,8 +20,6 @@ API documentation for vLLM's configuration classes.

20 - [vllm.config.CompilationConfig][]

21 - [vllm.config.VllmConfig][]

23 -[](){ #offline-inference-api }

24 -

23 ## Offline Inference

25 LLM Class.

@@ -45,18 +43,14 @@ Engine classes for offline and online inference.

44 Inference parameters for vLLM APIs.

48 -[](){ #sampling-params }

49 -

46 - [vllm.SamplingParams][]

47 - [vllm.PoolingParams][]

53 -[](){ #multi-modality }

54 -

49 ## Multi-Modality

51 vLLM provides experimental support for multi-modal models through the [vllm.multimodal][] package.

59 -Multi-modal inputs can be passed alongside text and token prompts to [supported models][supported-mm-models]

53 +Multi-modal inputs can be passed alongside text and token prompts to [supported models](../models/supported_models.md#list-of-multimodal-language-models)

54 via the `multi_modal_data` field in [vllm.inputs.PromptType][].

56 Looking to add your own multi-modal model? Please follow the instructions listed [here](../contributing/model/multimodal.md).

docs/configuration/README.md

+1 -1

@@ -4,6 +4,6 @@ This section lists the most common options for running vLLM.

5 There are three main levels of configuration, from highest priority to lowest priority:

7 -- [Request parameters][completions-api] and [input arguments][sampling-params]

7 +- [Request parameters](../serving/openai_compatible_server.md#completions-api) and [input arguments](../api/README.md#inference-parameters)

8 - [Engine arguments](./engine_args.md)

9 - [Environment variables](./env_vars.md)

docs/configuration/optimization.md

-2

@@ -27,8 +27,6 @@ You can monitor the number of preemption requests through Prometheus metrics exp

28 In vLLM V1, the default preemption mode is `RECOMPUTE` rather than `SWAP`, as recomputation has lower overhead in the V1 architecture.

30 -[](){ #chunked-prefill }

31 -

30 ## Chunked Prefill

32 Chunked prefill allows vLLM to process large prefills in smaller chunks and batch them together with decode requests. This feature helps improve both throughput and latency by better balancing compute-bound (prefill) and memory-bound (decode) operations.

docs/contributing/benchmarks.md

+2 -6

@@ -7,8 +7,8 @@ toc_depth: 4

7 vLLM provides comprehensive benchmarking tools for performance testing and evaluation:

9 - **[Benchmark CLI]**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing

10 -- **[Performance benchmarks][performance-benchmarks]**: Automated CI benchmarks for development

11 -- **[Nightly benchmarks][nightly-benchmarks]**: Comparative benchmarks against alternatives

10 +- **[Performance benchmarks](#performance-benchmarks)**: Automated CI benchmarks for development

11 +- **[Nightly benchmarks](#nightly-benchmarks)**: Comparative benchmarks against alternatives

13 [Benchmark CLI]: #benchmark-cli

@@ -924,8 +924,6 @@ throughput numbers correctly is also adjusted.

924

925 </details>

926

927 -[](){ #performance-benchmarks }

928 -

927 ## Performance Benchmarks

928

929 The performance benchmarks are used for development to confirm whether new changes improve performance under various workloads. They are triggered on every commit with both the `perf-benchmarks` and `ready` labels, and when a PR is merged into vLLM.

@@ -988,8 +986,6 @@ The benchmarking currently runs on a predefined set of models configured in the

986

987 All continuous benchmarking results are automatically published to the public [vLLM Performance Dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm).

988

991 -[](){ #nightly-benchmarks }

992 -

989 ## Nightly Benchmarks

990

991 These compare vLLM's performance against alternatives (`tgi`, `trt-llm`, and `lmdeploy`) when there are major updates of vLLM (e.g., bumping up to a new version). They are primarily intended for consumers to evaluate when to choose vLLM over other options and are triggered on every commit with both the `perf-benchmarks` and `nightly-benchmarks` labels.

docs/contributing/model/README.md

+1 -1

@@ -1,7 +1,7 @@

1 # Summary

3 !!! important

4 - Many decoder language models can now be automatically loaded using the [Transformers backend][transformers-backend] without having to implement them in vLLM. See if `vllm serve <model>` works first!

4 + Many decoder language models can now be automatically loaded using the [Transformers backend](../../models/supported_models.md#transformers) without having to implement them in vLLM. See if `vllm serve <model>` works first!

6 vLLM models are specialized [PyTorch](https://pytorch.org/) models that take advantage of various [features](../../features/README.md#compatibility-matrix) to optimize their performance.

docs/contributing/model/registration.md

+1 -1

@@ -8,7 +8,7 @@ This page provides detailed instructions on how to do so.

9 ## Built-in models

11 -To add a model directly to the vLLM library, start by forking our [GitHub repository](https://github.com/vllm-project/vllm) and then [build it from source][build-from-source].

11 +To add a model directly to the vLLM library, start by forking our [GitHub repository](https://github.com/vllm-project/vllm) and then [build it from source](../../getting_started/installation/gpu.md#build-wheel-from-source).

12 This gives you the ability to modify the codebase and test your model.

14 After you have implemented your model (see [tutorial](basic.md)), put it into the [vllm/model_executor/models](../../../vllm/model_executor/models) directory.

docs/contributing/model/tests.md

-2

@@ -39,8 +39,6 @@ For [generative models](../../models/generative_models.md), there are two levels

40 For [pooling models](../../models/pooling_models.md), we simply check the cosine similarity, as defined in [tests/models/utils.py](../../../tests/models/utils.py).

42 -[](){ #mm-processing-tests }

43 -

42 ### Multi-modal processing

44 #### Common tests

docs/deployment/docker.md

-4

@@ -1,7 +1,5 @@

1 # Using Docker

3 -[](){ #deployment-docker-pre-built-image }

4 -

3 ## Use vLLM's Official Docker Image

5 vLLM offers an official Docker image for deployment.

@@ -62,8 +60,6 @@ You can add any other [engine-args](../configuration/engine_args.md) you need af

60 RUN uv pip install --system git+https://github.com/huggingface/transformers.git

61 ```

65 -[](){ #deployment-docker-build-image-from-source }

66 -

63 ## Building vLLM's Docker Image from Source

65 You can build and run vLLM from source via the provided [docker/Dockerfile](../../docker/Dockerfile). To build vLLM:

docs/deployment/frameworks/anyscale.md

-2

@@ -1,7 +1,5 @@

1 # Anyscale

3 -[](){ #deployment-anyscale }

4 -

3 [Anyscale](https://www.anyscale.com) is a managed, multi-cloud platform developed by the creators of Ray.

5 Anyscale automates the entire lifecycle of Ray clusters in your AWS, GCP, or Azure account, delivering the flexibility of open-source Ray

docs/deployment/nginx.md

-14

@@ -2,8 +2,6 @@

3 This document shows how to launch multiple vLLM serving containers and use Nginx to act as a load balancer between the servers.

5 -[](){ #nginxloadbalancer-nginx-build }

6 -

5 ## Build Nginx Container

7 This guide assumes that you have just cloned the vLLM project and you're currently in the vllm root directory.

@@ -27,8 +25,6 @@ Build the container:

25 docker build . -f Dockerfile.nginx --tag nginx-lb

26 ```

30 -[](){ #nginxloadbalancer-nginx-conf }

31 -

28 ## Create Simple Nginx Config file

30 Create a file named `nginx_conf/nginx.conf`. Note that you can add as many servers as you'd like. In the below example we'll start with two. To add more, add another `server vllmN:8000 max_fails=3 fail_timeout=10000s;` entry to `upstream backend`.

@@ -53,8 +49,6 @@ Create a file named `nginx_conf/nginx.conf`. Note that you can add as many serve

49 }

50 ```

56 -[](){ #nginxloadbalancer-nginx-vllm-container }

57 -

52 ## Build vLLM Container

54 ```bash

@@ -73,16 +67,12 @@ docker build \

67 --build-arg https_proxy=$https_proxy

68 ```

76 -[](){ #nginxloadbalancer-nginx-docker-network }

77 -

70 ## Create Docker Network

72 ```bash

73 docker network create vllm_nginx

74 ```

84 -[](){ #nginxloadbalancer-nginx-launch-container }

85 -

76 ## Launch vLLM Containers

78 Notes:

@@ -122,8 +112,6 @@ Notes:

112 !!! note

113 If you are behind proxy, you can pass the proxy settings to the docker run command via `-e http_proxy=$http_proxy -e https_proxy=$https_proxy`.

114

125 -[](){ #nginxloadbalancer-nginx-launch-nginx }

126 -

115 ## Launch Nginx

116

117 ```bash

@@ -135,8 +123,6 @@ docker run \

123 --name nginx-lb nginx-lb:latest

124 ```

125

138 -[](){ #nginxloadbalancer-nginx-verify-nginx }

139 -

126 ## Verify That vLLM Servers Are Ready

127

128 ```bash

docs/design/arch_overview.md

+1 -1

@@ -47,7 +47,7 @@ Here is a sample of `LLM` class usage:

47 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

48 ```

50 -More API details can be found in the [Offline Inference](#offline-inference-api) section of the API docs.

50 +More API details can be found in the [Offline Inference](../api/README.md#offline-inference) section of the API docs.

52 The code for the `LLM` class can be found in [vllm/entrypoints/llm.py](../../vllm/entrypoints/llm.py).

docs/design/mm_processing.md

+2 -6

@@ -1,6 +1,6 @@

1 # Multi-Modal Data Processing

3 -To enable various optimizations in vLLM such as [chunked prefill][chunked-prefill] and [prefix caching](../features/automatic_prefix_caching.md), we use [BaseMultiModalProcessor][vllm.multimodal.processing.BaseMultiModalProcessor] to provide the correspondence between placeholder feature tokens (e.g. `<image>`) and multi-modal inputs (e.g. the raw input image) based on the outputs of HF processor.

3 +To enable various optimizations in vLLM such as [chunked prefill](../configuration/optimization.md#chunked-prefill) and [prefix caching](../features/automatic_prefix_caching.md), we use [BaseMultiModalProcessor][vllm.multimodal.processing.BaseMultiModalProcessor] to provide the correspondence between placeholder feature tokens (e.g. `<image>`) and multi-modal inputs (e.g. the raw input image) based on the outputs of HF processor.

5 Here are the main features of [BaseMultiModalProcessor][vllm.multimodal.processing.BaseMultiModalProcessor]:

@@ -41,14 +41,10 @@ While HF processors support text + multi-modal inputs natively, this is not so f

42 Moreover, since the tokenized text has not passed through the HF processor, we have to apply Step 3 by ourselves to keep the output tokens and multi-modal data consistent with each other.

44 -[](){ #mm-dummy-text }

45 -

44 ### Dummy text

46 We work around the first issue by requiring each model to define how to generate dummy text based on the number of multi-modal inputs, via [get_dummy_text][vllm.multimodal.profiling.BaseDummyInputsBuilder.get_dummy_text]. This lets us generate dummy text corresponding to the multi-modal inputs and input them together to obtain the processed multi-modal data.

50 -[](){ #mm-automatic-prompt-updating }

51 -

48 ### Automatic prompt updating

50 We address the second issue by implementing model-agnostic code in

@@ -64,4 +60,4 @@ Some HF processors, such as the one for Qwen2-VL, are [very slow](https://github

61 When new data is passed in, we first check which items are in the cache, and which ones are missing. The missing items are passed into the HF processor in a single batch and cached, before being merged with the existing items in the cache.

67 -Since we only process the missing multi-modal data items, the number of input placeholder tokens no longer corresponds to the number of the multi-modal inputs, so they can't be passed alongside the text prompt to HF processor. Therefore, we process the text and multi-modal inputs separately, using [dummy text][mm-dummy-text] to avoid HF errors. Since this skips HF's prompt updating code, we apply [automatic prompt updating][mm-automatic-prompt-updating] afterwards to keep the output tokens and multi-modal data consistent with each other.

63 +Since we only process the missing multi-modal data items, the number of input placeholder tokens no longer corresponds to the number of the multi-modal inputs, so they can't be passed alongside the text prompt to HF processor. Therefore, we process the text and multi-modal inputs separately, using [dummy text](#dummy-text) to avoid HF errors. Since this skips HF's prompt updating code, we apply [automatic prompt updating](#automatic-prompt-updating) afterwards to keep the output tokens and multi-modal data consistent with each other.

docs/design/multiprocessing.md

+1 -1

@@ -2,7 +2,7 @@

3 ## Debugging

5 -Please see the [Troubleshooting][troubleshooting-python-multiprocessing]

5 +Please see the [Troubleshooting](../usage/troubleshooting.md#python-multiprocessing)

6 page for information on known issues and how to solve them.

8 ## Introduction

docs/features/README.md

+3 -5

@@ -36,9 +36,9 @@ th:not(:first-child) {

36 }

37 </style>

40 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

41 -| [CP][chunked-prefill] | ✅ | | | | | | | | | | | | | | |

41 +| [CP](../configuration/optimization.md#chunked-prefill) | ✅ | | | | | | | | | | | | | | |

42 | [APC](automatic_prefix_caching.md) | ✅ | ✅ | | | | | | | | | | | | | |

43 | [LoRA](lora.md) | ✅ | ✅ | ✅ | | | | | | | | | | | | |

44 | [SD](spec_decode.md) | ✅ | ✅ | ❌ | ✅ | | | | | | | | | | | |

@@ -57,13 +57,11 @@ th:not(:first-child) {

57 \* Chunked prefill and prefix caching are only applicable to last-token pooling.

58 <sup>^</sup> LoRA is only applicable to the language backbone of multimodal models.

60 -[](){ #feature-x-hardware }

61 -

60 ### Feature x Hardware

62 | Feature | Volta | Turing | Ampere | Ada | Hopper | CPU | AMD | TPU | Intel GPU |

63 |-----------------------------------------------------------|---------------------|-----------|-----------|--------|------------|--------------------|--------|-----| ------------|

66 -| [CP][chunked-prefill] | [❌](https://github.com/vllm-project/vllm/issues/2729) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

64 +| [CP](../configuration/optimization.md#chunked-prefill) | [❌](https://github.com/vllm-project/vllm/issues/2729) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

65 | [APC](automatic_prefix_caching.md) | [❌](https://github.com/vllm-project/vllm/issues/3687) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

66 | [LoRA](lora.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

67 | [SD](spec_decode.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | [🟠](https://github.com/vllm-project/vllm/issues/26963) |

docs/features/multimodal_inputs.md

+1 -1

@@ -1,6 +1,6 @@

1 # Multimodal Inputs

3 -This page teaches you how to pass multi-modal inputs to [multi-modal models][supported-mm-models] in vLLM.

3 +This page teaches you how to pass multi-modal inputs to [multi-modal models](../models/supported_models.md#list-of-multimodal-language-models) in vLLM.

5 !!! note

6 We are actively iterating on multi-modal support. See [this RFC](https://github.com/vllm-project/vllm/issues/4194) for upcoming changes,

docs/getting_started/installation/google_tpu.md

+1 -1

@@ -153,7 +153,7 @@ VLLM_TARGET_DEVICE="tpu" python -m pip install -e .

153

154 ### Pre-built images

155

156 -See [deployment-docker-pre-built-image][deployment-docker-pre-built-image] for instructions on using the official Docker image, making sure to substitute the image name `vllm/vllm-openai` with `vllm/vllm-tpu`.

156 +See [Using Docker](../../deployment/docker.md) for instructions on using the official Docker image, making sure to substitute the image name `vllm/vllm-openai` with `vllm/vllm-tpu`.

157

158 ### Build image from source

159

docs/getting_started/installation/gpu.cuda.inc.md

+6 -8

@@ -15,7 +15,7 @@ vLLM contains pre-compiled C++ and CUDA (12.8) binaries.

16 In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.

18 -Therefore, it is recommended to install vLLM with a **fresh new** environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See [below][build-from-source] for more details.

18 +Therefore, it is recommended to install vLLM with a **fresh new** environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See [below](#build-wheel-from-source) for more details.

20 # --8<-- [end:set-up-using-python]

21 # --8<-- [start:pre-built-wheels]

@@ -44,8 +44,6 @@ export CUDA_VERSION=118 # or 126

44 uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu${CUDA_VERSION}-cp38-abi3-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu${CUDA_VERSION}

45 ```

47 -[](){ #install-the-latest-code }

48 -

47 #### Install the latest code

49 LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides wheels for Linux running on an x86 platform with CUDA 12 for every commit since `v0.5.3`.

@@ -128,11 +126,11 @@ export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/${VLLM_COMMIT}/vll

126 uv pip install --editable .

127 ```

128

131 -You can find more information about vLLM's wheels in [install-the-latest-code][install-the-latest-code].

129 +You can find more information about vLLM's wheels in [Install the latest code](#install-the-latest-code).

130

131 !!! note

132 There is a possibility that your source code may have a different commit ID compared to the latest vLLM wheel, which could potentially lead to unknown errors.

135 - It is recommended to use the same commit ID for the source code as the vLLM wheel you have installed. Please refer to [install-the-latest-code][install-the-latest-code] for instructions on how to install a specified wheel.

133 + It is recommended to use the same commit ID for the source code as the vLLM wheel you have installed. Please refer to [Install the latest code](#install-the-latest-code) for instructions on how to install a specified wheel.

134

135 #### Full build (with compilation)

136

@@ -250,7 +248,7 @@ uv pip install -e .

248 # --8<-- [end:build-wheel-from-source]

249 # --8<-- [start:pre-built-images]

250

253 -See [deployment-docker-pre-built-image][deployment-docker-pre-built-image] for instructions on using the official Docker image.

251 +See [Using Docker](../../deployment/docker.md) for instructions on using the official Docker image.

252

253 Another way to access the latest code is to use the docker images:

254

@@ -266,11 +264,11 @@ The latest code can contain bugs and may not be stable. Please use it with cauti

264 # --8<-- [end:pre-built-images]

265 # --8<-- [start:build-image-from-source]

266

269 -See [deployment-docker-build-image-from-source][deployment-docker-build-image-from-source] for instructions on building the Docker image.

267 +See [Building vLLM's Docker Image from Source](../../deployment/docker.md#building-vllms-docker-image-from-source) for instructions on building the Docker image.

268

269 # --8<-- [end:build-image-from-source]

270 # --8<-- [start:supported-features]

271

274 -See [feature-x-hardware][feature-x-hardware] compatibility matrix for feature support information.

272 +See [Feature x Hardware](../../features/README.md#feature-x-hardware) compatibility matrix for feature support information.

273

274 # --8<-- [end:supported-features]

docs/getting_started/installation/gpu.md

-2

@@ -66,8 +66,6 @@ vLLM is a Python library that supports the following GPU variants. Select your G

67 --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:pre-built-wheels"

69 -[](){ #build-from-source }

70 -

69 ### Build wheel from source

71 === "NVIDIA CUDA"

docs/getting_started/installation/gpu.rocm.inc.md

+1 -1

@@ -217,6 +217,6 @@ Where the `<path/to/model>` is the location where the model is stored, for examp

217 # --8<-- [end:build-image-from-source]

218 # --8<-- [start:supported-features]

219

220 -See [feature-x-hardware][feature-x-hardware] compatibility matrix for feature support information.

220 +See [Feature x Hardware](../../features/README.md#feature-x-hardware) compatibility matrix for feature support information.

221

222 # --8<-- [end:supported-features]

docs/getting_started/quickstart.md

+4 -8

@@ -2,8 +2,8 @@

3 This guide will help you quickly get started with vLLM to perform:

5 -- [Offline batched inference][quickstart-offline]

6 -- [Online serving using OpenAI-compatible server][quickstart-online]

5 +- [Offline batched inference](#offline-batched-inference)

6 +- [Online serving using OpenAI-compatible server](#openai-compatible-server)

8 ## Prerequisites

@@ -42,8 +42,6 @@ uv pip install vllm --torch-backend=auto

42 !!! note

43 For more detail and non-CUDA platforms, please refer [here](installation/README.md) for specific instructions on how to install vLLM.

45 -[](){ #quickstart-offline }

46 -

45 ## Offline Batched Inference

47 With vLLM installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing). See the example script: [examples/offline_inference/basic/basic.py](../../examples/offline_inference/basic/basic.py)

@@ -57,7 +55,7 @@ The first line of this example imports the classes [LLM][vllm.LLM] and [Sampling

55 from vllm import LLM, SamplingParams

56 ```

60 -The next section defines a list of input prompts and sampling parameters for text generation. The [sampling temperature](https://arxiv.org/html/2402.05201v1) is set to `0.8` and the [nucleus sampling probability](https://en.wikipedia.org/wiki/Top-p_sampling) is set to `0.95`. You can find more information about the sampling parameters [here][sampling-params].

58 +The next section defines a list of input prompts and sampling parameters for text generation. The [sampling temperature](https://arxiv.org/html/2402.05201v1) is set to `0.8` and the [nucleus sampling probability](https://en.wikipedia.org/wiki/Top-p_sampling) is set to `0.95`. You can find more information about the sampling parameters [here](../api/README.md#inference-parameters).

60 !!! important

61 By default, vLLM will use sampling parameters recommended by model creator by applying the `generation_config.json` from the Hugging Face model repository if it exists. In most cases, this will provide you with the best results by default if [SamplingParams][vllm.SamplingParams] is not specified.

@@ -135,8 +133,6 @@ for output in outputs:

133 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

134 ```

135

138 -[](){ #quickstart-online }

139 -

136 ## OpenAI-Compatible Server

137

138 vLLM can be deployed as a server that implements the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API.

@@ -150,7 +146,7 @@ vllm serve Qwen/Qwen2.5-1.5B-Instruct

146

147 !!! note

148 By default, the server uses a predefined chat template stored in the tokenizer.

153 - You can learn about overriding it [here][chat-template].

149 + You can learn about overriding it [here](../serving/openai_compatible_server.md#chat-template).

150 !!! important

151 By default, the server applies `generation_config.json` from the huggingface model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator.

152

docs/models/generative_models.md

+2 -2

@@ -140,5 +140,5 @@ outputs = llm.chat(conversation, chat_template=custom_template)

140

141 Our [OpenAI-Compatible Server](../serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs:

142

143 -- [Completions API][completions-api] is similar to `LLM.generate` but only accepts text.

144 -- [Chat API][chat-api] is similar to `LLM.chat`, accepting both text and [multi-modal inputs](../features/multimodal_inputs.md) for models with a chat template.

143 +- [Completions API](../serving/openai_compatible_server.md#completions-api) is similar to `LLM.generate` but only accepts text.

144 +- [Chat API](../serving/openai_compatible_server.md#chat-api) is similar to `LLM.chat`, accepting both text and [multi-modal inputs](../features/multimodal_inputs.md) for models with a chat template.

docs/models/pooling_models.md

+4 -4

@@ -185,10 +185,10 @@ print(f"Data: {data!r}")

185

186 Our [OpenAI-Compatible Server](../serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs:

187

188 -- [Pooling API][pooling-api] is similar to `LLM.encode`, being applicable to all types of pooling models.

189 -- [Embeddings API][embeddings-api] is similar to `LLM.embed`, accepting both text and [multi-modal inputs](../features/multimodal_inputs.md) for embedding models.

190 -- [Classification API][classification-api] is similar to `LLM.classify` and is applicable to sequence classification models.

191 -- [Score API][score-api] is similar to `LLM.score` for cross-encoder models.

188 +- [Pooling API](../serving/openai_compatible_server.md#pooling-api) is similar to `LLM.encode`, being applicable to all types of pooling models.

189 +- [Embeddings API](../serving/openai_compatible_server.md#embeddings-api) is similar to `LLM.embed`, accepting both text and [multi-modal inputs](../features/multimodal_inputs.md) for embedding models.

190 +- [Classification API](../serving/openai_compatible_server.md#classification-api) is similar to `LLM.classify` and is applicable to sequence classification models.

191 +- [Score API](../serving/openai_compatible_server.md#score-api) is similar to `LLM.score` for cross-encoder models.

192

193 ## Matryoshka Embeddings

194

docs/models/supported_models.md

+3 -13

@@ -11,9 +11,7 @@ Alongside each architecture, we include some popular models that use it.

12 If vLLM natively supports a model, its implementation can be found in [vllm/model_executor/models](../../vllm/model_executor/models).

14 -These models are what we list in [supported-text-models][supported-text-models] and [supported-mm-models][supported-mm-models].

15 -

16 -[](){ #transformers-backend }

14 +These models are what we list in [supported text models](#list-of-text-only-language-models) and [supported multimodal models](#list-of-multimodal-language-models).

16 ### Transformers

@@ -60,7 +58,7 @@ For a model to be compatible with the Transformers backend for vLLM it must:

58 - be a Transformers compatible custom model (see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)):

59 - The model directory must have the correct structure (e.g. `config.json` is present).

60 - `config.json` must contain `auto_map.AutoModel`.

63 -- be a Transformers backend for vLLM compatible model (see [writing-custom-models][writing-custom-models]):

61 +- be a Transformers backend for vLLM compatible model (see [Writing custom models](#writing-custom-models)):

62 - Customisation should be done in the base model (e.g. in `MyModel`, not `MyModelForCausalLM`).

64 If the compatible model is:

@@ -70,8 +68,6 @@ If the compatible model is:

69 This means that, with the Transformers backend for vLLM, new models can be used before they are officially supported in Transformers or vLLM!

73 -[](){ #writing-custom-models }

74 -

71 #### Writing custom models

73 This section details the necessary modifications to make to a Transformers compatible custom model that make it compatible with the Transformers backend for vLLM. (We assume that a Transformers compatible custom model has already been created, see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)).

@@ -164,7 +160,7 @@ To determine whether a given model is natively supported, you can check the `con

160 If the `"architectures"` field contains a model architecture listed below, then it should be natively supported.

161

162 Models do not _need_ to be natively supported to be used in vLLM.

167 -The [Transformers backend][transformers-backend] enables you to run models directly using their Transformers implementation (or even remote code on the Hugging Face Model Hub!).

163 +The [Transformers backend](#transformers) enables you to run models directly using their Transformers implementation (or even remote code on the Hugging Face Model Hub!).

164

165 !!! tip

166 The easiest way to check if your model is really supported at runtime is to run the program below:

@@ -306,8 +302,6 @@ output = llm.encode("Hello, my name is")

302 print(output)

303 ```

304

309 -[](){ #feature-status-legend }

310 -

305 ## Feature Status Legend

306

307 - ✅︎ indicates that the feature is supported for the model.

@@ -316,8 +310,6 @@ print(output)

310

311 - ⚠️ indicates that the feature is available but may have known issues or limitations.

312

319 -[](){ #supported-text-models }

320 -

313 ## List of Text-only Language Models

314

315 ### Generative Models

@@ -583,8 +575,6 @@ These models primarily support the [`LLM.encode`](./pooling_models.md#llmencode)

575 !!! note

576 Named Entity Recognition (NER) usage, please refer to [examples/offline_inference/pooling/ner.py](../../examples/offline_inference/pooling/ner.py), [examples/online_serving/pooling/ner_client.py](../../examples/online_serving/pooling/ner_client.py).

577

586 -[](){ #supported-mm-models }

587 -

578 ## List of Multimodal Language Models

579

580 The following modalities are supported depending on the model:

docs/serving/distributed_troubleshooting.md

+1 -1

@@ -4,7 +4,7 @@ For general troubleshooting, see [Troubleshooting](../usage/troubleshooting.md).

5 ## Verify inter-node GPU communication

7 -After you start the Ray cluster, verify GPU-to-GPU communication across nodes. Proper configuration can be non-trivial. For more information, see [troubleshooting script][troubleshooting-incorrect-hardware-driver]. If you need additional environment variables for communication configuration, append them to [examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh), for example `-e NCCL_SOCKET_IFNAME=eth0`. Setting environment variables during cluster creation is recommended because the variables propagate to all nodes. In contrast, setting environment variables in the shell affects only the local node. For more information, see <https://github.com/vllm-project/vllm/issues/6803>.

7 +After you start the Ray cluster, verify GPU-to-GPU communication across nodes. Proper configuration can be non-trivial. For more information, see [troubleshooting script](../usage/troubleshooting.md#incorrect-hardwaredriver). If you need additional environment variables for communication configuration, append them to [examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh), for example `-e NCCL_SOCKET_IFNAME=eth0`. Setting environment variables during cluster creation is recommended because the variables propagate to all nodes. In contrast, setting environment variables in the shell affects only the local node. For more information, see <https://github.com/vllm-project/vllm/issues/6803>.

9 ## No available node types can fulfill resource request

docs/serving/offline_inference.md

+1 -1

@@ -19,7 +19,7 @@ The available APIs depend on the model type:

19 - [Pooling models](../models/pooling_models.md) output their hidden states directly.

21 !!! info

22 - [API Reference][offline-inference-api]

22 + [API Reference](../api/README.md#offline-inference)

24 ## Ray Data LLM API

docs/serving/openai_compatible_server.md

+17 -39

@@ -44,37 +44,35 @@ To call the server, in your preferred text editor, create a script that uses an

45 We currently support the following OpenAI APIs:

47 -- [Completions API][completions-api] (`/v1/completions`)

47 +- [Completions API](#completions-api) (`/v1/completions`)

48 - Only applicable to [text generation models](../models/generative_models.md).

49 - *Note: `suffix` parameter is not supported.*

50 -- [Chat Completions API][chat-api] (`/v1/chat/completions`)

51 - - Only applicable to [text generation models](../models/generative_models.md) with a [chat template][chat-template].

50 +- [Chat Completions API](#chat-api) (`/v1/chat/completions`)

51 + - Only applicable to [text generation models](../models/generative_models.md) with a [chat template](../serving/openai_compatible_server.md#chat-template).

52 - *Note: `parallel_tool_calls` and `user` parameters are ignored.*

53 -- [Embeddings API][embeddings-api] (`/v1/embeddings`)

53 +- [Embeddings API](#embeddings-api) (`/v1/embeddings`)

54 - Only applicable to [embedding models](../models/pooling_models.md).

55 -- [Transcriptions API][transcriptions-api] (`/v1/audio/transcriptions`)

55 +- [Transcriptions API](#transcriptions-api) (`/v1/audio/transcriptions`)

56 - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).

57 -- [Translation API][translations-api] (`/v1/audio/translations`)

57 +- [Translation API](#translations-api) (`/v1/audio/translations`)

58 - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).

60 In addition, we have the following custom APIs:

62 -- [Tokenizer API][tokenizer-api] (`/tokenize`, `/detokenize`)

62 +- [Tokenizer API](#tokenizer-api) (`/tokenize`, `/detokenize`)

63 - Applicable to any model with a tokenizer.

64 -- [Pooling API][pooling-api] (`/pooling`)

64 +- [Pooling API](#pooling-api) (`/pooling`)

65 - Applicable to all [pooling models](../models/pooling_models.md).

66 -- [Classification API][classification-api] (`/classify`)

66 +- [Classification API](#classification-api) (`/classify`)

67 - Only applicable to [classification models](../models/pooling_models.md).

68 -- [Score API][score-api] (`/score`)

68 +- [Score API](#score-api) (`/score`)

69 - Applicable to [embedding models and cross-encoder models](../models/pooling_models.md).

70 -- [Re-rank API][rerank-api] (`/rerank`, `/v1/rerank`, `/v2/rerank`)

70 +- [Re-rank API](#re-rank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)

71 - Implements [Jina AI's v1 re-rank API](https://jina.ai/reranker/)

72 - Also compatible with [Cohere's v1 & v2 re-rank APIs](https://docs.cohere.com/v2/reference/rerank)

73 - Jina and Cohere's APIs are very similar; Jina's includes extra information in the rerank endpoint's response.

74 - Only applicable to [cross-encoder models](../models/pooling_models.md).

76 -[](){ #chat-template }

77 -

76 ## Chat Template

78 In order for the language model to support chat protocol, vLLM requires the model to include

@@ -174,8 +172,6 @@ with `--enable-request-id-headers`.

172

173 ## API Reference

174

177 -[](){ #completions-api }

178 -

175 ### Completions API

176

177 Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);

@@ -185,7 +181,7 @@ Code example: [examples/online_serving/openai_completion_client.py](../../exampl

181

182 #### Extra parameters

183

188 -The following [sampling parameters][sampling-params] are supported.

184 +The following [sampling parameters](../api/README.md#inference-parameters) are supported.

185

186 ??? code

187

@@ -201,8 +197,6 @@ The following extra parameters are supported:

197 --8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"

198 ```

199

204 -[](){ #chat-api }

205 -

200 ### Chat API

201

202 Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat);

@@ -218,7 +212,7 @@ Code example: [examples/online_serving/openai_chat_completion_client.py](../../e

212

213 #### Extra parameters

214

221 -The following [sampling parameters][sampling-params] are supported.

215 +The following [sampling parameters](../api/README.md#inference-parameters) are supported.

216

217 ??? code

218

@@ -234,8 +228,6 @@ The following extra parameters are supported:

228 --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"

229 ```

230

237 -[](){ #embeddings-api }

238 -

231 ### Embeddings API

232

233 Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);

@@ -243,7 +235,7 @@ you can use the [official OpenAI Python client](https://github.com/openai/openai

235

236 Code example: [examples/online_serving/pooling/openai_embedding_client.py](../../examples/online_serving/pooling/openai_embedding_client.py)

237

246 -If the model has a [chat template][chat-template], you can replace `inputs` with a list of `messages` (same schema as [Chat API][chat-api])

238 +If the model has a [chat template](../serving/openai_compatible_server.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](#chat-api))

239 which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations:

240

241 ??? code

@@ -369,8 +361,6 @@ For chat-like input (i.e. if `messages` is passed), these extra parameters are s

361 --8<-- "vllm/entrypoints/openai/protocol.py:chat-embedding-extra-params"

362 ```

363

372 -[](){ #transcriptions-api }

373 -

364 ### Transcriptions API

365

366 Our Transcriptions API is compatible with [OpenAI's Transcriptions API](https://platform.openai.com/docs/api-reference/audio/createTranscription);

@@ -468,7 +458,7 @@ For `verbose_json` response format:

458

459 #### Extra Parameters

460

471 -The following [sampling parameters][sampling-params] are supported.

461 +The following [sampling parameters](../api/README.md#inference-parameters) are supported.

462

463 ??? code

464

@@ -484,8 +474,6 @@ The following extra parameters are supported:

474 --8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"

475 ```

476

487 -[](){ #translations-api }

488 -

477 ### Translations API

478

479 Our Translation API is compatible with [OpenAI's Translations API](https://platform.openai.com/docs/api-reference/audio/createTranslation);

@@ -500,7 +488,7 @@ Code example: [examples/online_serving/openai_translation_client.py](../../examp

488

489 #### Extra Parameters

490

503 -The following [sampling parameters][sampling-params] are supported.

491 +The following [sampling parameters](../api/README.md#inference-parameters) are supported.

492

493 ```python

494 --8<-- "vllm/entrypoints/openai/protocol.py:translation-sampling-params"

@@ -512,8 +500,6 @@ The following extra parameters are supported:

500 --8<-- "vllm/entrypoints/openai/protocol.py:translation-extra-params"

501 ```

502

515 -[](){ #tokenizer-api }

516 -

503 ### Tokenizer API

504

505 Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).

@@ -522,18 +508,14 @@ It consists of two endpoints:

508 - `/tokenize` corresponds to calling `tokenizer.encode()`.

509 - `/detokenize` corresponds to calling `tokenizer.decode()`.

510

525 -[](){ #pooling-api }

526 -

511 ### Pooling API

512

513 Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.

514

531 -The input format is the same as [Embeddings API][embeddings-api], but the output data can contain an arbitrary nested list, not just a 1-D list of floats.

515 +The input format is the same as [Embeddings API](#embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.

516

517 Code example: [examples/online_serving/pooling/openai_pooling_client.py](../../examples/online_serving/pooling/openai_pooling_client.py)

518

535 -[](){ #classification-api }

536 -

519 ### Classification API

520

521 Our Classification API directly supports Hugging Face sequence-classification models such as [ai21labs/Jamba-tiny-reward-dev](https://huggingface.co/ai21labs/Jamba-tiny-reward-dev) and [jason9693/Qwen2.5-1.5B-apeach](https://huggingface.co/jason9693/Qwen2.5-1.5B-apeach).

@@ -649,8 +631,6 @@ The following extra parameters are supported:

631 --8<-- "vllm/entrypoints/openai/protocol.py:classification-extra-params"

632 ```

633

652 -[](){ #score-api }

653 -

634 ### Score API

635

636 Our Score API can apply a cross-encoder model or an embedding model to predict scores for sentence or multimodal pairs. When using an embedding model the score corresponds to the cosine similarity between each embedding pair.

@@ -856,8 +836,6 @@ The following extra parameters are supported:

836 --8<-- "vllm/entrypoints/openai/protocol.py:score-extra-params"

837 ```

838

859 -[](){ #rerank-api }

860 -

839 ### Re-rank API

840

841 Our Re-rank API can apply an embedding model or a cross-encoder model to predict relevant scores between a single query, and

docs/usage/troubleshooting.md

-4

@@ -80,8 +80,6 @@ You might also need to set `export NCCL_SOCKET_IFNAME=<your_network_interface>`

80 If vLLM crashes and the error trace captures it somewhere around `self.graph.replay()` in `vllm/worker/model_runner.py`, it is a CUDA error inside CUDAGraph.

81 To identify the particular CUDA operation that causes the error, you can add `--enforce-eager` to the command line, or `enforce_eager=True` to the [LLM][vllm.LLM] class to disable the CUDAGraph optimization and isolate the exact CUDA operation that causes the error.

83 -[](){ #troubleshooting-incorrect-hardware-driver }

84 -

83 ## Incorrect hardware/driver

85 If GPU/CPU communication cannot be established, you can use the following Python script and follow the instructions below to confirm whether the GPU/CPU communication is working correctly.

@@ -178,8 +176,6 @@ If the test script hangs or crashes, usually it means the hardware/drivers are b

176

177 Adjust `--nproc-per-node`, `--nnodes`, and `--node-rank` according to your setup, being sure to execute different commands (with different `--node-rank`) on different nodes.

178

181 -[](){ #troubleshooting-python-multiprocessing }

182 -

179 ## Python multiprocessing

180

181 ### `RuntimeError` Exception