A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Doc][3/N] Reorganize Serving section (#11766)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Jan 7, 2025 at 11:20 UTC
8ceffbf3152d3b26d293ba1e157d0c187884572b
40 files changed
+248
-133
README.md
+1
-1
@@ -77,7 +77,7 @@ pip install vllm
77
Visit our [documentation](https://vllm.readthedocs.io/en/latest/) to learn more.
78
- [Installation](https://vllm.readthedocs.io/en/latest/getting_started/installation.html)
79
- [Quickstart](https://vllm.readthedocs.io/en/latest/getting_started/quickstart.html)
80
-- [Supported Models](https://vllm.readthedocs.io/en/latest/models/supported_models.html)
80
+- [List of Supported Models](https://vllm.readthedocs.io/en/latest/models/supported_models.html)
81
82
## Contributing
83
docs/source/assets/deployment/architecture_helm_deployment.png
renamed
docs/source/contributing/dockerfile/dockerfile.md
+1
-1
@@ -1,7 +1,7 @@
1
# Dockerfile
2
3
We provide a <gh-file:Dockerfile> to construct the image for running an OpenAI compatible server with vLLM.
4
-More information about deploying with Docker can be found [here](../../serving/deploying_with_docker.md).
4
+More information about deploying with Docker can be found [here](#deployment-docker).
5
6
Below is a visual representation of the multi-stage Dockerfile. The build graph contains the following nodes:
7
docs/source/contributing/model/registration.md
+2
-2
@@ -3,7 +3,7 @@
3
# Model Registration
4
5
vLLM relies on a model registry to determine how to run each model.
6
-A list of pre-registered architectures can be found on the [Supported Models](#supported-models) page.
6
+A list of pre-registered architectures can be found [here](#supported-models).
7
8
If your model is not on this list, you must register it to vLLM.
9
This page provides detailed instructions on how to do so.
@@ -16,7 +16,7 @@ This gives you the ability to modify the codebase and test your model.
16
After you have implemented your model (see [tutorial](#new-model-basic)), put it into the <gh-dir:vllm/model_executor/models> directory.
17
Then, add your model class to `_VLLM_MODELS` in <gh-file:vllm/model_executor/models/registry.py> so that it is automatically registered upon importing vLLM.
18
You should also include an example HuggingFace repository for this model in <gh-file:tests/models/registry.py> to run the unit tests.
19
-Finally, update the [Supported Models](#supported-models) documentation page to promote your model!
19
+Finally, update our [list of supported models](#supported-models) to promote your model!
20
21
```{important}
22
The list of models in each section should be maintained in alphabetical order.
docs/source/deployment/docker.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(deploying-with-docker)=
1
+(deployment-docker)=
2
3
-# Deploying with Docker
3
+# Using Docker
4
5
## Use vLLM's Official Docker Image
6
docs/source/deployment/frameworks/bentoml.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(deploying-with-bentoml)=
1
+(deployment-bentoml)=
2
3
-# Deploying with BentoML
3
+# BentoML
4
5
[BentoML](https://github.com/bentoml/BentoML) allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-complicant image and deploy it on Kubernetes.
6
docs/source/deployment/frameworks/cerebrium.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(deploying-with-cerebrium)=
1
+(deployment-cerebrium)=
2
3
-# Deploying with Cerebrium
3
+# Cerebrium
4
5
```{raw} html
6
<p align="center">
docs/source/deployment/frameworks/dstack.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(deploying-with-dstack)=
1
+(deployment-dstack)=
2
3
-# Deploying with dstack
3
+# dstack
4
5
```{raw} html
6
<p align="center">
docs/source/deployment/frameworks/helm.md
renamed
+3
-3
@@ -1,6 +1,6 @@
1
-(deploying-with-helm)=
1
+(deployment-helm)=
2
3
-# Deploying with Helm
3
+# Helm
4
5
A Helm chart to deploy vLLM for Kubernetes
6
@@ -38,7 +38,7 @@ chart **including persistent volumes** and deletes the release.
38
39
## Architecture
40
41
-```{image} architecture_helm_deployment.png
41
+```{image} /assets/deployment/architecture_helm_deployment.png
42
```
43
44
## Values
docs/source/deployment/frameworks/index.md
new
+13
@@ -0,0 +1,13 @@
1
+# Using other frameworks
2
+
3
+```{toctree}
4
+:maxdepth: 1
5
+
6
+bentoml
7
+cerebrium
8
+dstack
9
+helm
10
+lws
11
+skypilot
12
+triton
13
+```
docs/source/deployment/frameworks/lws.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(deploying-with-lws)=
1
+(deployment-lws)=
2
3
-# Deploying with LWS
3
+# LWS
4
5
LeaderWorkerSet (LWS) is a Kubernetes API that aims to address common deployment patterns of AI/ML inference workloads.
6
A major use case is for multi-host/multi-node distributed inference.
docs/source/deployment/frameworks/skypilot.md
renamed
+4
-4
@@ -1,6 +1,6 @@
1
-(on-cloud)=
1
+(deployment-skypilot)=
2
3
-# Deploying and scaling up with SkyPilot
3
+# SkyPilot
4
5
```{raw} html
6
<p align="center">
@@ -12,9 +12,9 @@ vLLM can be **run and scaled to multiple service replicas on clouds and Kubernet
12
13
## Prerequisites
14
15
-- Go to the [HuggingFace model page](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and request access to the model {code}`meta-llama/Meta-Llama-3-8B-Instruct`.
15
+- Go to the [HuggingFace model page](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and request access to the model `meta-llama/Meta-Llama-3-8B-Instruct`.
16
- Check that you have installed SkyPilot ([docs](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html)).
17
-- Check that {code}`sky check` shows clouds or Kubernetes are enabled.
17
+- Check that `sky check` shows clouds or Kubernetes are enabled.
18
19
```console
20
pip install skypilot-nightly
docs/source/deployment/frameworks/triton.md
renamed
+2
-2
@@ -1,5 +1,5 @@
1
-(deploying-with-triton)=
1
+(deployment-triton)=
2
3
-# Deploying with NVIDIA Triton
3
+# NVIDIA Triton
4
5
The [Triton Inference Server](https://github.com/triton-inference-server) hosts a tutorial demonstrating how to quickly deploy a simple [facebook/opt-125m](https://huggingface.co/facebook/opt-125m) model using vLLM. Please see [Deploying a vLLM model in Triton](https://github.com/triton-inference-server/tutorials/blob/main/Quick_Deploy/vLLM/README.md#deploying-a-vllm-model-in-triton) for more details.
docs/source/deployment/integrations/index.md
new
+9
@@ -0,0 +1,9 @@
1
+# External Integrations
2
+
3
+```{toctree}
4
+:maxdepth: 1
5
+
6
+kserve
7
+kubeai
8
+llamastack
9
+```
docs/source/deployment/integrations/kserve.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(deploying-with-kserve)=
1
+(deployment-kserve)=
2
3
-# Deploying with KServe
3
+# KServe
4
5
vLLM can be deployed with [KServe](https://github.com/kserve/kserve) on Kubernetes for highly scalable distributed model serving.
6
docs/source/deployment/integrations/kubeai.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(deploying-with-kubeai)=
1
+(deployment-kubeai)=
2
3
-# Deploying with KubeAI
3
+# KubeAI
4
5
[KubeAI](https://github.com/substratusai/kubeai) is a Kubernetes operator that enables you to deploy and manage AI models on Kubernetes. It provides a simple and scalable way to deploy vLLM in production. Functionality such as scale-from-zero, load based autoscaling, model caching, and much more is provided out of the box with zero external dependencies.
6
docs/source/deployment/integrations/llamastack.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(run-on-llamastack)=
1
+(deployment-llamastack)=
2
3
-# Serving with Llama Stack
3
+# Llama Stack
4
5
vLLM is also available via [Llama Stack](https://github.com/meta-llama/llama-stack) .
6
docs/source/deployment/k8s.md
renamed
+2
-2
@@ -1,6 +1,6 @@
1
-(deploying-with-k8s)=
1
+(deployment-k8s)=
2
3
-# Deploying with Kubernetes
3
+# Using Kubernetes
4
5
Using Kubernetes to deploy vLLM is a scalable and efficient way to serve machine learning models. This guide will walk you through the process of deploying vLLM with Kubernetes, including the necessary prerequisites, steps for deployment, and testing.
6
docs/source/deployment/nginx.md
renamed
+1
-1
@@ -1,6 +1,6 @@
1
(nginxloadbalancer)=
2
3
-# Deploying with Nginx Loadbalancer
3
+# Using Nginx
4
5
This document shows how to launch multiple vLLM serving containers and use Nginx to act as a load balancer between the servers.
6
docs/source/design/arch_overview.md
+1
-1
@@ -57,7 +57,7 @@ More API details can be found in the {doc}`Offline Inference
57
58
The code for the `LLM` class can be found in <gh-file:vllm/entrypoints/llm.py>.
59
60
-### OpenAI-compatible API server
60
+### OpenAI-Compatible API Server
61
62
The second primary interface to vLLM is via its OpenAI-compatible API server.
63
This server can be started using the `vllm serve` command.
docs/source/features/disagg_prefill.md
+6
-2
@@ -1,8 +1,12 @@
1
(disagg-prefill)=
2
3
-# Disaggregated prefilling (experimental)
3
+# Disaggregated Prefilling (experimental)
4
5
-This page introduces you the disaggregated prefilling feature in vLLM. This feature is experimental and subject to change.
5
+This page introduces you the disaggregated prefilling feature in vLLM.
6
+
7
+```{note}
8
+This feature is experimental and subject to change.
9
+```
10
11
## Why disaggregated prefilling?
12
docs/source/features/spec_decode.md
+1
-1
@@ -1,6 +1,6 @@
1
(spec-decode)=
2
3
-# Speculative decoding
3
+# Speculative Decoding
4
5
```{warning}
6
Please note that speculative decoding in vLLM is not yet optimized and does
docs/source/getting_started/installation/gpu-rocm.md
+1
-1
@@ -148,7 +148,7 @@ $ export PYTORCH_ROCM_ARCH="gfx90a;gfx942"
148
$ python3 setup.py develop
149
```
150
151
-This may take 5-10 minutes. Currently, {code}`pip install .` does not work for ROCm installation.
151
+This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm installation.
152
153
```{tip}
154
- Triton flash attention is used by default. For benchmarking purposes, it is recommended to run a warm up step before collecting perf numbers.
docs/source/getting_started/installation/hpu-gaudi.md
+1
-1
@@ -82,7 +82,7 @@ $ python setup.py develop
82
83
## Supported Features
84
85
-- [Offline batched inference](#offline-batched-inference)
85
+- [Offline inference](#offline-inference)
86
- Online inference via [OpenAI-Compatible Server](#openai-compatible-server)
87
- HPU autodetection - no need to manually select device within vLLM
88
- Paged KV cache with algorithms enabled for Intel Gaudi accelerators
docs/source/getting_started/quickstart.md
+10
-8
@@ -2,20 +2,20 @@
2
3
# Quickstart
4
5
-This guide will help you quickly get started with vLLM to:
5
+This guide will help you quickly get started with vLLM to perform:
6
7
-- [Run offline batched inference](#offline-batched-inference)
8
-- [Run OpenAI-compatible inference](#openai-compatible-server)
7
+- [Offline batched inference](#quickstart-offline)
8
+- [Online inference using OpenAI-compatible server](#quickstart-online)
9
10
## Prerequisites
11
12
- OS: Linux
13
- Python: 3.9 -- 3.12
14
-- GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
14
15
## Installation
16
18
-You can install vLLM using pip. It's recommended to use [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) to create and manage Python environments.
17
+If you are using NVIDIA GPUs, you can install vLLM using [pip](https://pypi.org/project/vllm/) directly.
18
+It's recommended to use [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) to create and manage Python environments.
19
20
```console
21
$ conda create -n myenv python=3.10 -y
@@ -23,9 +23,11 @@ $ conda activate myenv
23
$ pip install vllm
24
```
25
26
-Please refer to the [installation documentation](#installation-index) for more details on installing vLLM.
26
+```{note}
27
+For non-CUDA platforms, please refer [here](#installation-index) for specific instructions on how to install vLLM.
28
+```
29
28
-(offline-batched-inference)=
30
+(quickstart-offline)=
31
32
## Offline Batched Inference
33
@@ -73,7 +75,7 @@ for output in outputs:
75
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
76
```
77
76
-(openai-compatible-server)=
78
+(quickstart-online)=
79
80
## OpenAI-Compatible Server
81
docs/source/index.md
+28
-21
@@ -65,32 +65,14 @@ getting_started/troubleshooting
65
getting_started/faq
66
```
67
68
-```{toctree}
69
-:caption: Serving
70
-:maxdepth: 1
71
-
72
-serving/openai_compatible_server
73
-serving/deploying_with_docker
74
-serving/deploying_with_k8s
75
-serving/deploying_with_helm
76
-serving/deploying_with_nginx
77
-serving/distributed_serving
78
-serving/metrics
79
-serving/integrations
80
-serving/tensorizer
81
-serving/runai_model_streamer
82
-serving/engine_args
83
-serving/env_vars
84
-serving/usage_stats
85
-```
86
-
68
```{toctree}
69
:caption: Models
70
:maxdepth: 1
71
91
-models/supported_models
72
models/generative_models
73
models/pooling_models
74
+models/supported_models
75
+models/extensions/index
76
```
77
78
```{toctree}
@@ -99,7 +81,6 @@ models/pooling_models
81
82
features/quantization/index
83
features/lora
102
-features/multimodal_inputs
84
features/tool_calling
85
features/structured_outputs
86
features/automatic_prefix_caching
@@ -108,6 +89,32 @@ features/spec_decode
89
features/compatibility_matrix
90
```
91
92
+```{toctree}
93
+:caption: Inference and Serving
94
+:maxdepth: 1
95
+
96
+serving/offline_inference
97
+serving/openai_compatible_server
98
+serving/multimodal_inputs
99
+serving/distributed_serving
100
+serving/metrics
101
+serving/engine_args
102
+serving/env_vars
103
+serving/usage_stats
104
+serving/integrations/index
105
+```
106
+
107
+```{toctree}
108
+:caption: Deployment
109
+:maxdepth: 1
110
+
111
+deployment/docker
112
+deployment/k8s
113
+deployment/nginx
114
+deployment/frameworks/index
115
+deployment/integrations/index
116
+```
117
+
118
```{toctree}
119
:caption: Performance
120
:maxdepth: 1
docs/source/models/extensions/index.md
new
+8
@@ -0,0 +1,8 @@
1
+# Built-in Extensions
2
+
3
+```{toctree}
4
+:maxdepth: 1
5
+
6
+runai_model_streamer
7
+tensorizer
8
+```
docs/source/models/extensions/runai_model_streamer.md
renamed
+1
-1
@@ -1,6 +1,6 @@
1
(runai-model-streamer)=
2
3
-# Loading Models with Run:ai Model Streamer
3
+# Loading models with Run:ai Model Streamer
4
5
Run:ai Model Streamer is a library to read tensors in concurrency, while streaming it to GPU memory.
6
Further reading can be found in [Run:ai Model Streamer Documentation](https://github.com/run-ai/runai-model-streamer/blob/master/docs/README.md).
docs/source/models/extensions/tensorizer.md
renamed
+1
-1
@@ -1,6 +1,6 @@
1
(tensorizer)=
2
3
-# Loading Models with CoreWeave's Tensorizer
3
+# Loading models with CoreWeave's Tensorizer
4
5
vLLM supports loading models with [CoreWeave's Tensorizer](https://docs.coreweave.com/coreweave-machine-learning-and-ai/inference/tensorizer).
6
vLLM model tensors that have been serialized to disk, an HTTP/HTTPS endpoint, or S3 endpoint can be deserialized
docs/source/models/supported_models.md
+22
-22
@@ -1,9 +1,9 @@
1
(supported-models)=
2
3
-# Supported Models
3
+# List of Supported Models
4
5
vLLM supports generative and pooling models across various tasks.
6
-If a model supports more than one task, you can set the task via the {code}`--task` argument.
6
+If a model supports more than one task, you can set the task via the `--task` argument.
7
8
For each task, we list the model architectures that have been implemented in vLLM.
9
Alongside each architecture, we include some popular models that use it.
@@ -14,8 +14,8 @@ Alongside each architecture, we include some popular models that use it.
14
15
By default, vLLM loads models from [HuggingFace (HF) Hub](https://huggingface.co/models).
16
17
-To determine whether a given model is supported, you can check the {code}`config.json` file inside the HF repository.
18
-If the {code}`"architectures"` field contains a model architecture listed below, then it should be supported in theory.
17
+To determine whether a given model is supported, you can check the `config.json` file inside the HF repository.
18
+If the `"architectures"` field contains a model architecture listed below, then it should be supported in theory.
19
20
````{tip}
21
The easiest way to check if your model is really supported at runtime is to run the program below:
@@ -48,7 +48,7 @@ To use models from [ModelScope](https://www.modelscope.cn) instead of HuggingFac
48
$ export VLLM_USE_MODELSCOPE=True
49
```
50
51
-And use with {code}`trust_remote_code=True`.
51
+And use with `trust_remote_code=True`.
52
53
```python
54
from vllm import LLM
@@ -420,15 +420,15 @@ you should explicitly specify the task type to ensure that the model is used in
420
```
421
422
```{note}
423
-{code}`ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.
424
-You should manually set mean pooling by passing {code}`--override-pooler-config '{"pooling_type": "MEAN"}'`.
423
+`ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.
424
+You should manually set mean pooling by passing `--override-pooler-config '{"pooling_type": "MEAN"}'`.
425
```
426
427
```{note}
428
-Unlike base Qwen2, {code}`Alibaba-NLP/gte-Qwen2-7B-instruct` uses bi-directional attention.
429
-You can set {code}`--hf-overrides '{"is_causal": false}'` to change the attention mask accordingly.
428
+Unlike base Qwen2, `Alibaba-NLP/gte-Qwen2-7B-instruct` uses bi-directional attention.
429
+You can set `--hf-overrides '{"is_causal": false}'` to change the attention mask accordingly.
430
431
-On the other hand, its 1.5B variant ({code}`Alibaba-NLP/gte-Qwen2-1.5B-instruct`) uses causal attention
431
+On the other hand, its 1.5B variant (`Alibaba-NLP/gte-Qwen2-1.5B-instruct`) uses causal attention
432
despite being described otherwise on its model card.
433
```
434
@@ -468,8 +468,8 @@ If your model is not in the above list, we will try to automatically convert the
468
{func}`vllm.model_executor.models.adapters.as_reward_model`. By default, we return the hidden states of each token directly.
469
470
```{important}
471
-For process-supervised reward models such as {code}`peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
472
-e.g.: {code}`--override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.
471
+For process-supervised reward models such as `peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
472
+e.g.: `--override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.
473
```
474
475
#### Classification (`--task classify`)
@@ -537,13 +537,13 @@ The following modalities are supported depending on the model:
537
- **V**ideo
538
- **A**udio
539
540
-Any combination of modalities joined by {code}`+` are supported.
540
+Any combination of modalities joined by `+` are supported.
541
542
-- e.g.: {code}`T + I` means that the model supports text-only, image-only, and text-with-image inputs.
542
+- e.g.: `T + I` means that the model supports text-only, image-only, and text-with-image inputs.
543
544
-On the other hand, modalities separated by {code}`/` are mutually exclusive.
544
+On the other hand, modalities separated by `/` are mutually exclusive.
545
546
-- e.g.: {code}`T / I` means that the model supports text-only and image-only inputs, but not text-with-image inputs.
546
+- e.g.: `T / I` means that the model supports text-only and image-only inputs, but not text-with-image inputs.
547
548
See [this page](#multimodal-inputs) on how to pass multi-modal inputs to the model.
549
@@ -731,8 +731,8 @@ See [this page](#generative-models) for more information on how to use generativ
731
<sup>+</sup> Multiple items can be inputted per text prompt for this modality.
732
733
````{important}
734
-To enable multiple multi-modal items per text prompt, you have to set {code}`limit_mm_per_prompt` (offline inference)
735
-or {code}`--limit-mm-per-prompt` (online inference). For example, to enable passing up to 4 images per text prompt:
734
+To enable multiple multi-modal items per text prompt, you have to set `limit_mm_per_prompt` (offline inference)
735
+or `--limit-mm-per-prompt` (online inference). For example, to enable passing up to 4 images per text prompt:
736
737
```python
738
llm = LLM(
@@ -751,11 +751,11 @@ vLLM currently only supports adding LoRA to the language backbone of multimodal
751
```
752
753
```{note}
754
-To use {code}`TIGER-Lab/Mantis-8B-siglip-llama3`, you have pass {code}`--hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}'` when running vLLM.
754
+To use `TIGER-Lab/Mantis-8B-siglip-llama3`, you have pass `--hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}'` when running vLLM.
755
```
756
757
```{note}
758
-The official {code}`openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork ({code}`HwwwH/MiniCPM-V-2`) for now.
758
+The official `openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork (`HwwwH/MiniCPM-V-2`) for now.
759
For more details, please see: <gh-pr:4087#issuecomment-2250397630>
760
```
761
@@ -770,7 +770,7 @@ you should explicitly specify the task type to ensure that the model is used in
770
771
#### Text Embedding (`--task embed`)
772
773
-Any text generation model can be converted into an embedding model by passing {code}`--task embed`.
773
+Any text generation model can be converted into an embedding model by passing `--task embed`.
774
775
```{note}
776
To get the best results, you should use pooling models that are specifically trained as such.
@@ -818,7 +818,7 @@ At vLLM, we are committed to facilitating the integration and support of third-p
818
2. **Best-Effort Consistency**: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.
819
820
```{tip}
821
-When comparing the output of {code}`model.generate` from HuggingFace Transformers with the output of {code}`llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
821
+When comparing the output of `model.generate` from HuggingFace Transformers with the output of `llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
822
```
823
824
3. **Issue Resolution and Model Updates**: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.
docs/source/serving/distributed_serving.md
+6
-6
@@ -18,13 +18,13 @@ After adding enough GPUs and nodes to hold the model, you can run vLLM first, wh
18
There is one edge case: if the model fits in a single node with multiple GPUs, but the number of GPUs cannot divide the model size evenly, you can use pipeline parallelism, which splits the model along layers and supports uneven splits. In this case, the tensor parallel size should be 1 and the pipeline parallel size should be the number of GPUs.
19
```
20
21
-## Details for Distributed Inference and Serving
21
+## Running vLLM on a single node
22
23
vLLM supports distributed tensor-parallel and pipeline-parallel inference and serving. Currently, we support [Megatron-LM's tensor parallel algorithm](https://arxiv.org/pdf/1909.08053.pdf). We manage the distributed runtime with either [Ray](https://github.com/ray-project/ray) or python native multiprocessing. Multiprocessing can be used when deploying on a single node, multi-node inferencing currently requires Ray.
24
25
-Multiprocessing will be used by default when not running in a Ray placement group and if there are sufficient GPUs available on the same node for the configured {code}`tensor_parallel_size`, otherwise Ray will be used. This default can be overridden via the {code}`LLM` class {code}`distributed_executor_backend` argument or {code}`--distributed-executor-backend` API server argument. Set it to {code}`mp` for multiprocessing or {code}`ray` for Ray. It's not required for Ray to be installed for the multiprocessing case.
25
+Multiprocessing will be used by default when not running in a Ray placement group and if there are sufficient GPUs available on the same node for the configured `tensor_parallel_size`, otherwise Ray will be used. This default can be overridden via the `LLM` class `distributed_executor_backend` argument or `--distributed-executor-backend` API server argument. Set it to `mp` for multiprocessing or `ray` for Ray. It's not required for Ray to be installed for the multiprocessing case.
26
27
-To run multi-GPU inference with the {code}`LLM` class, set the {code}`tensor_parallel_size` argument to the number of GPUs you want to use. For example, to run inference on 4 GPUs:
27
+To run multi-GPU inference with the `LLM` class, set the `tensor_parallel_size` argument to the number of GPUs you want to use. For example, to run inference on 4 GPUs:
28
29
```python
30
from vllm import LLM
@@ -32,14 +32,14 @@ llm = LLM("facebook/opt-13b", tensor_parallel_size=4)
32
output = llm.generate("San Franciso is a")
33
```
34
35
-To run multi-GPU serving, pass in the {code}`--tensor-parallel-size` argument when starting the server. For example, to run API server on 4 GPUs:
35
+To run multi-GPU serving, pass in the `--tensor-parallel-size` argument when starting the server. For example, to run API server on 4 GPUs:
36
37
```console
38
$ vllm serve facebook/opt-13b \
39
$ --tensor-parallel-size 4
40
```
41
42
-You can also additionally specify {code}`--pipeline-parallel-size` to enable pipeline parallelism. For example, to run API server on 8 GPUs with pipeline parallelism and tensor parallelism:
42
+You can also additionally specify `--pipeline-parallel-size` to enable pipeline parallelism. For example, to run API server on 8 GPUs with pipeline parallelism and tensor parallelism:
43
44
```console
45
$ vllm serve gpt2 \
@@ -47,7 +47,7 @@ $ --tensor-parallel-size 4 \
47
$ --pipeline-parallel-size 2
48
```
49
50
-## Multi-Node Inference and Serving
50
+## Running vLLM on multiple nodes
51
52
If a single node does not have enough GPUs to hold the model, you can run the model using multiple nodes. It is important to make sure the execution environment is the same on all nodes, including the model path, the Python environment. The recommended way is to use docker images to ensure the same environment, and hide the heterogeneity of the host machines via mapping them into the same docker configuration.
53
docs/source/serving/integrations.md
deleted
-17
@@ -1,17 +0,0 @@
1
-# Integrations
2
-
3
-```{toctree}
4
-:maxdepth: 1
5
-
6
-run_on_sky
7
-deploying_with_kserve
8
-deploying_with_kubeai
9
-deploying_with_triton
10
-deploying_with_bentoml
11
-deploying_with_cerebrium
12
-deploying_with_lws
13
-deploying_with_dstack
14
-serving_with_langchain
15
-serving_with_llamaindex
16
-serving_with_llamastack
17
-```
docs/source/serving/integrations/index.md
new
+8
@@ -0,0 +1,8 @@
1
+# External Integrations
2
+
3
+```{toctree}
4
+:maxdepth: 1
5
+
6
+langchain
7
+llamaindex
8
+```
docs/source/serving/integrations/langchain.md
renamed
+4
-4
@@ -1,10 +1,10 @@
1
-(run-on-langchain)=
1
+(serving-langchain)=
2
3
-# Serving with Langchain
3
+# LangChain
4
5
-vLLM is also available via [Langchain](https://github.com/langchain-ai/langchain) .
5
+vLLM is also available via [LangChain](https://github.com/langchain-ai/langchain) .
6
7
-To install langchain, run
7
+To install LangChain, run
8
9
```console
10
$ pip install langchain langchain_community -q
docs/source/serving/integrations/llamaindex.md
renamed
+4
-4
@@ -1,10 +1,10 @@
1
-(run-on-llamaindex)=
1
+(serving-llamaindex)=
2
3
-# Serving with llama_index
3
+# LlamaIndex
4
5
-vLLM is also available via [llama_index](https://github.com/run-llama/llama_index) .
5
+vLLM is also available via [LlamaIndex](https://github.com/run-llama/llama_index) .
6
7
-To install llamaindex, run
7
+To install LlamaIndex, run
8
9
```console
10
$ pip install llama-index-llms-vllm -q
docs/source/serving/metrics.md
+1
-1
@@ -4,7 +4,7 @@ vLLM exposes a number of metrics that can be used to monitor the health of the
4
system. These metrics are exposed via the `/metrics` endpoint on the vLLM
5
OpenAI compatible API server.
6
7
-You can start the server using Python, or using [Docker](deploying_with_docker.md):
7
+You can start the server using Python, or using [Docker](#deployment-docker):
8
9
```console
10
$ vllm serve unsloth/Llama-3.2-1B-Instruct
docs/source/serving/multimodal_inputs.md
renamed
+6
-6
@@ -18,7 +18,7 @@ To input multi-modal data, follow this schema in {class}`vllm.inputs.PromptType`
18
19
### Image
20
21
-You can pass a single image to the {code}`'image'` field of the multi-modal dictionary, as shown in the following examples:
21
+You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:
22
23
```python
24
llm = LLM(model="llava-hf/llava-1.5-7b-hf")
@@ -122,21 +122,21 @@ for o in outputs:
122
123
### Video
124
125
-You can pass a list of NumPy arrays directly to the {code}`'video'` field of the multi-modal dictionary
125
+You can pass a list of NumPy arrays directly to the `'video'` field of the multi-modal dictionary
126
instead of using multi-image input.
127
128
Full example: <gh-file:examples/offline_inference_vision_language.py>
129
130
### Audio
131
132
-You can pass a tuple {code}`(array, sampling_rate)` to the {code}`'audio'` field of the multi-modal dictionary.
132
+You can pass a tuple `(array, sampling_rate)` to the `'audio'` field of the multi-modal dictionary.
133
134
Full example: <gh-file:examples/offline_inference_audio_language.py>
135
136
### Embedding
137
138
To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
139
-pass a tensor of shape {code}`(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
139
+pass a tensor of shape `(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
140
141
```python
142
# Inference with image embeddings as input
@@ -294,7 +294,7 @@ $ export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
294
295
### Video
296
297
-Instead of {code}`image_url`, you can pass a video file via {code}`video_url`. Here is a simple example using [LLaVA-OneVision](https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf).
297
+Instead of `image_url`, you can pass a video file via `video_url`. Here is a simple example using [LLaVA-OneVision](https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf).
298
299
First, launch the OpenAI-compatible server:
300
@@ -418,7 +418,7 @@ result = chat_completion_from_base64.choices[0].message.content
418
print("Chat completion output from input audio:", result)
419
```
420
421
-Alternatively, you can pass {code}`audio_url`, which is the audio counterpart of {code}`image_url` for image input:
421
+Alternatively, you can pass `audio_url`, which is the audio counterpart of `image_url` for image input:
422
423
```python
424
chat_completion_from_url = client.chat.completions.create(
docs/source/serving/offline_inference.md
new
+79
@@ -0,0 +1,79 @@
1
+(offline-inference)=
2
+
3
+# Offline Inference
4
+
5
+You can run vLLM in your own code on a list of prompts.
6
+
7
+The offline API is based on the {class}`~vllm.LLM` class.
8
+To initialize the vLLM engine, create a new instance of `LLM` and specify the model to run.
9
+
10
+For example, the following code downloads the [`facebook/opt-125m`](https://huggingface.co/facebook/opt-125m) model from HuggingFace
11
+and runs it in vLLM using the default configuration.
12
+
13
+```python
14
+llm = LLM(model="facebook/opt-125m")
15
+```
16
+
17
+After initializing the `LLM` instance, you can perform model inference using various APIs.
18
+The available APIs depend on the type of model that is being run:
19
+
20
+- [Generative models](#generative-models) output logprobs which are sampled from to obtain the final output text.
21
+- [Pooling models](#pooling-models) output their hidden states directly.
22
+
23
+Please refer to the above pages for more details about each API.
24
+
25
+```{seealso}
26
+[API Reference](/dev/offline_inference/offline_index)
27
+```
28
+
29
+## Configuration Options
30
+
31
+This section lists the most common options for running the vLLM engine.
32
+For a full list, refer to the [Engine Arguments](#engine-args) page.
33
+
34
+### Reducing memory usage
35
+
36
+Large models might cause your machine to run out of memory (OOM). Here are some options that help alleviate this problem.
37
+
38
+#### Tensor Parallelism (TP)
39
+
40
+Tensor parallelism (`tensor_parallel_size` option) can be used to split the model across multiple GPUs.
41
+
42
+The following code splits the model across 2 GPUs.
43
+
44
+```python
45
+llm = LLM(model="ibm-granite/granite-3.1-8b-instruct",
46
+ tensor_parallel_size=2)
47
+```
48
+
49
+```{important}
50
+To ensure that vLLM initializes CUDA correctly, you should avoid calling related functions (e.g. {func}`torch.cuda.set_device`)
51
+before initializing vLLM. Otherwise, you may run into an error like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`.
52
+
53
+To control which devices are used, please instead set the `CUDA_VISIBLE_DEVICES` environment variable.
54
+```
55
+
56
+#### Quantization
57
+
58
+Quantized models take less memory at the cost of lower precision.
59
+
60
+Statically quantized models can be downloaded from HF Hub (some popular ones are available at [Neural Magic](https://huggingface.co/neuralmagic))
61
+and used directly without extra configuration.
62
+
63
+Dynamic quantization is also supported via the `quantization` option -- see [here](#quantization-index) for more details.
64
+
65
+#### Context length and batch size
66
+
67
+You can further reduce memory usage by limit the context length of the model (`max_model_len` option)
68
+and the maximum batch size (`max_num_seqs` option).
69
+
70
+```python
71
+llm = LLM(model="adept/fuyu-8b",
72
+ max_model_len=2048,
73
+ max_num_seqs=2)
74
+```
75
+
76
+### Performance optimization and tuning
77
+
78
+You can potentially improve the performance of vLLM by finetuning various options.
79
+Please refer to [this guide](#optimization-and-tuning) for more details.
docs/source/serving/openai_compatible_server.md
+5
-3
@@ -1,8 +1,10 @@
1
-# OpenAI Compatible Server
1
+(openai-compatible-server)=
2
3
-vLLM provides an HTTP server that implements OpenAI's [Completions](https://platform.openai.com/docs/api-reference/completions) and [Chat](https://platform.openai.com/docs/api-reference/chat) API, and more!
3
+# OpenAI-Compatible Server
4
5
-You can start the server via the [`vllm serve`](#vllm-serve) command, or through [Docker](deploying_with_docker.md):
5
+vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more!
6
+
7
+You can start the server via the [`vllm serve`](#vllm-serve) command, or through [Docker](#deployment-docker):
8
```bash
9
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
10
```
docs/source/serving/usage_stats.md
+1
-1
@@ -45,7 +45,7 @@ You can preview the collected data by running the following command:
45
tail ~/.config/vllm/usage_stats.json
46
```
47
48
-## Opt-out of Usage Stats Collection
48
+## Opting out
49
50
You can opt-out of usage stats collection by setting the `VLLM_NO_USAGE_STATS` or `DO_NOT_TRACK` environment variable, or by creating a `~/.config/vllm/do_not_track` file:
51