@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 402d378360
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

[Doc] [1/N] Reorganize Getting Started section (#11645)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Jan 6, 2025 at 10:18 UTC 402d37836059463c7ec8b1e25d40c29138f1dd40
22 files changed +54 -41
docs/source/design/arch_overview.md
+1 -2
@@ -77,8 +77,7 @@ python -m vllm.entrypoints.openai.api_server --model <model>
77
78 That code can be found in <gh-file:vllm/entrypoints/openai/api_server.py>.
79
80 -More details on the API server can be found in the {doc}`OpenAI Compatible
81 -Server </serving/openai_compatible_server>` document.
80 +More details on the API server can be found in the [OpenAI-Compatible Server](#openai-compatible-server) document.
81
82 ## LLM Engine
83
docs/source/design/multiprocessing.md
+1 -1
@@ -2,7 +2,7 @@
2
3 ## Debugging
4
5 -Please see the [Debugging Tips](#debugging-python-multiprocessing)
5 +Please see the [Troubleshooting](#troubleshooting-python-multiprocessing)
6 page for information on known issues and how to solve them.
7
8 ## Introduction
docs/source/getting_started/faq.md renamed
docs/source/getting_started/installation/cpu-arm.md renamed
+1 -1
@@ -2,7 +2,7 @@
2
3 # Installation for ARM CPUs
4
5 -vLLM has been adapted to work on ARM64 CPUs with NEON support, leveraging the CPU backend initially developed for the x86 platform. This guide provides installation instructions specific to ARM. For additional details on supported features, refer to the x86 platform documentation covering:
5 +vLLM has been adapted to work on ARM64 CPUs with NEON support, leveraging the CPU backend initially developed for the x86 platform. This guide provides installation instructions specific to ARM. For additional details on supported features, refer to the [x86 CPU documentation](#installation-x86) covering:
6
7 - CPU backend inference capabilities
8 - Relevant runtime environment variables
docs/source/getting_started/installation/cpu-x86.md renamed
+3 -3
@@ -1,6 +1,6 @@
1 -(installation-cpu)=
1 +(installation-x86)=
2
3 -# Installation with CPU
3 +# Installation for x86 CPUs
4
5 vLLM initially supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16. vLLM CPU backend supports the following vLLM features:
6
@@ -151,4 +151,4 @@ $ python examples/offline_inference.py
151 $ VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" vllm serve meta-llama/Llama-2-7b-chat-hf -tp=2 --distributed-executor-backend mp
152 ```
153
154 - - Using Data Parallel for maximum throughput: to launch an LLM serving endpoint on each NUMA node along with one additional load balancer to dispatch the requests to those endpoints. Common solutions like [Nginx](../serving/deploying_with_nginx.md) or HAProxy are recommended. Anyscale Ray project provides the feature on LLM [serving](https://docs.ray.io/en/latest/serve/index.html). Here is the example to setup a scalable LLM serving with [Ray Serve](https://github.com/intel/llm-on-ray/blob/main/docs/setup.md).
154 + - Using Data Parallel for maximum throughput: to launch an LLM serving endpoint on each NUMA node along with one additional load balancer to dispatch the requests to those endpoints. Common solutions like [Nginx](#nginxloadbalancer) or HAProxy are recommended. Anyscale Ray project provides the feature on LLM [serving](https://docs.ray.io/en/latest/serve/index.html). Here is the example to setup a scalable LLM serving with [Ray Serve](https://github.com/intel/llm-on-ray/blob/main/docs/setup.md).
docs/source/getting_started/installation/gpu-cuda.md renamed
+2 -2
@@ -1,6 +1,6 @@
1 -(installation)=
1 +(installation-cuda)=
2
3 -# Installation
3 +# Installation for CUDA
4
5 vLLM is a Python library that also contains pre-compiled C++ and CUDA (12.1) binaries.
6
docs/source/getting_started/installation/gpu-rocm.md renamed
+1 -1
@@ -1,6 +1,6 @@
1 (installation-rocm)=
2
3 -# Installation with ROCm
3 +# Installation for ROCm
4
5 vLLM supports AMD GPUs with ROCm 6.2.
6
docs/source/getting_started/installation/hpu-gaudi.md renamed
+3 -1
@@ -1,4 +1,6 @@
1 -# Installation with Intel® Gaudi® AI Accelerators
1 +(installation-gaudi)=
2 +
3 +# Installation for Intel® Gaudi®
4
5 This README provides instructions on running vLLM with Intel Gaudi devices.
6
docs/source/getting_started/installation/index.md new
+19
@@ -0,0 +1,19 @@
1 +(installation-index)=
2 +
3 +# Installation
4 +
5 +vLLM supports the following hardware platforms:
6 +
7 +```{toctree}
8 +:maxdepth: 1
9 +
10 +gpu-cuda
11 +gpu-rocm
12 +cpu-x86
13 +cpu-arm
14 +hpu-gaudi
15 +tpu
16 +xpu
17 +openvino
18 +neuron
19 +```
docs/source/getting_started/installation/neuron.md renamed
+1 -1
@@ -1,6 +1,6 @@
1 (installation-neuron)=
2
3 -# Installation with Neuron
3 +# Installation for Neuron
4
5 vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK with continuous batching.
6 Paged Attention and Chunked Prefill are currently in development and will be available soon.
docs/source/getting_started/installation/openvino.md renamed
+2 -2
@@ -1,8 +1,8 @@
1 (installation-openvino)=
2
3 -# Installation with OpenVINO
3 +# Installation for OpenVINO
4
5 -vLLM powered by OpenVINO supports all LLM models from {doc}`vLLM supported models list <../models/supported_models>` and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support, as well as on both integrated and discrete Intel® GPUs ([the list of supported GPUs](https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html#gpu)). OpenVINO vLLM backend supports the following advanced vLLM features:
5 +vLLM powered by OpenVINO supports all LLM models from [vLLM supported models list](#supported-models) and can perform optimal model serving on all x86-64 CPUs with, at least, AVX2 support, as well as on both integrated and discrete Intel® GPUs ([the list of supported GPUs](https://docs.openvino.ai/2024/about-openvino/release-notes-openvino/system-requirements.html#gpu)). OpenVINO vLLM backend supports the following advanced vLLM features:
6
7 - Prefix caching (`--enable-prefix-caching`)
8 - Chunked prefill (`--enable-chunked-prefill`)
docs/source/getting_started/installation/tpu.md renamed
+1 -1
@@ -1,6 +1,6 @@
1 (installation-tpu)=
2
3 -# Installation with TPU
3 +# Installation for TPUs
4
5 Tensor Processing Units (TPUs) are Google's custom-developed application-specific
6 integrated circuits (ASICs) used to accelerate machine learning workloads. TPUs
docs/source/getting_started/installation/xpu.md renamed
+1 -1
@@ -1,6 +1,6 @@
1 (installation-xpu)=
2
3 -# Installation with XPU
3 +# Installation for XPUs
4
5 vLLM initially supports basic model inferencing and serving on Intel GPU platform.
6
docs/source/getting_started/quickstart.md
+1 -1
@@ -23,7 +23,7 @@ $ conda activate myenv
23 $ pip install vllm
24 ```
25
26 -Please refer to the {ref}`installation documentation <installation>` for more details on installing vLLM.
26 +Please refer to the [installation documentation](#installation-index) for more details on installing vLLM.
27
28 (offline-batched-inference)=
29
docs/source/getting_started/troubleshooting.md renamed
+6 -5
@@ -1,8 +1,8 @@
1 -(debugging)=
1 +(troubleshooting)=
2
3 -# Debugging Tips
3 +# Troubleshooting
4
5 -This document outlines some debugging strategies you can consider. If you think you've discovered a bug, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible.
5 +This document outlines some troubleshooting strategies you can consider. If you think you've discovered a bug, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible.
6
7 ```{note}
8 Once you've debugged a problem, remember to turn off any debugging environment variables defined, or simply start a new shell to avoid being affected by lingering debugging settings. Otherwise, the system might be slow with debugging functionalities left activated.
@@ -47,6 +47,7 @@ You might also need to set `export NCCL_SOCKET_IFNAME=<your_network_interface>`
47 If vLLM crashes and the error trace captures it somewhere around `self.graph.replay()` in `vllm/worker/model_runner.py`, it is a CUDA error inside CUDAGraph.
48 To identify the particular CUDA operation that causes the error, you can add `--enforce-eager` to the command line, or `enforce_eager=True` to the {class}`~vllm.LLM` class to disable the CUDAGraph optimization and isolate the exact CUDA operation that causes the error.
49
50 +(troubleshooting-incorrect-hardware-driver)=
51 ## Incorrect hardware/driver
52
53 If GPU/CPU communication cannot be established, you can use the following Python script and follow the instructions below to confirm whether the GPU/CPU communication is working correctly.
@@ -139,7 +140,7 @@ A multi-node environment is more complicated than a single-node one. If you see
140 Adjust `--nproc-per-node`, `--nnodes`, and `--node-rank` according to your setup, being sure to execute different commands (with different `--node-rank`) on different nodes.
141 ```
142
142 -(debugging-python-multiprocessing)=
143 +(troubleshooting-python-multiprocessing)=
144 ## Python multiprocessing
145
146 ### `RuntimeError` Exception
@@ -150,7 +151,7 @@ If you have seen a warning in your logs like this:
151 WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
152 initialized. We must use the `spawn` multiprocessing start method. Setting
153 VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
153 - https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing
154 + https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#python-multiprocessing
155 for more information.
156 ```
157
docs/source/index.md
+4 -12
@@ -50,7 +50,7 @@ For more information, check out the following:
50 - [vLLM announcing blog post](https://vllm.ai) (intro to PagedAttention)
51 - [vLLM paper](https://arxiv.org/abs/2309.06180) (SOSP 2023)
52 - [How continuous batching enables 23x throughput in LLM inference while reducing p50 latency](https://www.anyscale.com/blog/continuous-batching-llm-inference) by Cade Daniel et al.
53 -- {ref}`vLLM Meetups <meetups>`.
53 +- [vLLM Meetups](#meetups)
54
55 ## Documentation
56
@@ -58,18 +58,11 @@ For more information, check out the following:
58 :caption: Getting Started
59 :maxdepth: 1
60
61 -getting_started/installation
62 -getting_started/amd-installation
63 -getting_started/openvino-installation
64 -getting_started/cpu-installation
65 -getting_started/gaudi-installation
66 -getting_started/arm-installation
67 -getting_started/neuron-installation
68 -getting_started/tpu-installation
69 -getting_started/xpu-installation
61 +getting_started/installation/index
62 getting_started/quickstart
71 -getting_started/debugging
63 getting_started/examples/examples_index
64 +getting_started/troubleshooting
65 +getting_started/faq
66 ```
67
68 ```{toctree}
@@ -110,7 +103,6 @@ usage/structured_outputs
103 usage/spec_decode
104 usage/compatibility_matrix
105 usage/performance
113 -usage/faq
106 usage/engine_args
107 usage/env_vars
108 usage/usage_stats
docs/source/models/generative_models.md
+1 -1
@@ -120,7 +120,7 @@ outputs = llm.chat(conversation, chat_template=custom_template)
120
121 ## Online Inference
122
123 -Our [OpenAI Compatible Server](../serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs:
123 +Our [OpenAI-Compatible Server](#openai-compatible-server) provides endpoints that correspond to the offline APIs:
124
125 - [Completions API](#completions-api) is similar to `LLM.generate` but only accepts text.
126 - [Chat API](#chat-api) is similar to `LLM.chat`, accepting both text and [multi-modal inputs](#multimodal-inputs) for models with a chat template.
docs/source/models/pooling_models.md
+1 -1
@@ -106,7 +106,7 @@ A code example can be found here: <gh-file:examples/offline_inference_scoring.py
106
107 ## Online Inference
108
109 -Our [OpenAI Compatible Server](../serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs:
109 +Our [OpenAI-Compatible Server](#openai-compatible-server) provides endpoints that correspond to the offline APIs:
110
111 - [Pooling API](#pooling-api) is similar to `LLM.encode`, being applicable to all types of pooling models.
112 - [Embeddings API](#embeddings-api) is similar to `LLM.embed`, accepting both text and [multi-modal inputs](#multimodal-inputs) for embedding models.
docs/source/serving/distributed_serving.md
+1 -1
@@ -95,7 +95,7 @@ $ --tensor-parallel-size 16
95 To make tensor parallel performant, you should make sure the communication between nodes is efficient, e.g. using high-speed network cards like Infiniband. To correctly set up the cluster to use Infiniband, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the `run_cluster.sh` script. Please contact your system administrator for more information on how to set up the flags. One way to confirm if the Infiniband is working is to run vLLM with `NCCL_DEBUG=TRACE` environment variable set, e.g. `NCCL_DEBUG=TRACE vllm serve ...` and check the logs for the NCCL version and the network used. If you find `[send] via NET/Socket` in the logs, it means NCCL uses raw TCP Socket, which is not efficient for cross-node tensor parallel. If you find `[send] via NET/IB/GDRDMA` in the logs, it means NCCL uses Infiniband with GPU-Direct RDMA, which is efficient.
96
97 ```{warning}
98 -After you start the Ray cluster, you'd better also check the GPU-GPU communication between nodes. It can be non-trivial to set up. Please refer to the [sanity check script](../getting_started/debugging.md) for more information. If you need to set some environment variables for the communication configuration, you can append them to the `run_cluster.sh` script, e.g. `-e NCCL_SOCKET_IFNAME=eth0`. Note that setting environment variables in the shell (e.g. `NCCL_SOCKET_IFNAME=eth0 vllm serve ...`) only works for the processes in the same node, not for the processes in the other nodes. Setting environment variables when you create the cluster is the recommended way. See <gh-issue:6803> for more information.
98 +After you start the Ray cluster, you'd better also check the GPU-GPU communication between nodes. It can be non-trivial to set up. Please refer to the [sanity check script](#troubleshooting-incorrect-hardware-driver) for more information. If you need to set some environment variables for the communication configuration, you can append them to the `run_cluster.sh` script, e.g. `-e NCCL_SOCKET_IFNAME=eth0`. Note that setting environment variables in the shell (e.g. `NCCL_SOCKET_IFNAME=eth0 vllm serve ...`) only works for the processes in the same node, not for the processes in the other nodes. Setting environment variables when you create the cluster is the recommended way. See <gh-issue:6803> for more information.
99 ```
100
101 ```{warning}
docs/source/usage/spec_decode.md
+2 -2
@@ -182,7 +182,7 @@ speculative decoding, breaking down the guarantees into three key areas:
182 3. **vLLM Logprob Stability**
183 \- vLLM does not currently guarantee stable token log probabilities (logprobs). This can result in different outputs for the
184 same request across runs. For more details, see the FAQ section
185 - titled *Can the output of a prompt vary across runs in vLLM?* in the {ref}`FAQs <faq>`.
185 + titled *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](#faq).
186
187 **Conclusion**
188
@@ -195,7 +195,7 @@ can occur due to following factors:
195
196 **Mitigation Strategies**
197
198 -For mitigation strategies, please refer to the FAQ entry *Can the output of a prompt vary across runs in vLLM?* in the {ref}`FAQs <faq>`.
198 +For mitigation strategies, please refer to the FAQ entry *Can the output of a prompt vary across runs in vLLM?* in the [FAQs](#faq).
199
200 ## Resources for vLLM contributors
201
docs/source/usage/structured_outputs.md
+1 -1
@@ -18,7 +18,7 @@ The following parameters are supported, which must be added as extra parameters:
18 - `guided_whitespace_pattern`: used to override the default whitespace pattern for guided json decoding.
19 - `guided_decoding_backend`: used to select the guided decoding backend to use.
20
21 -You can see the complete list of supported parameters on the [OpenAI Compatible Server](../serving/openai_compatible_server.md) page.
21 +You can see the complete list of supported parameters on the [OpenAI-Compatible Server](#openai-compatible-server)page.
22
23 Now let´s see an example for each of the cases, starting with the `guided_choice`, as it´s the easiest one:
24
vllm/utils.py
+1 -1
@@ -1938,7 +1938,7 @@ def _check_multiproc_method():
1938 "the `spawn` multiprocessing start method. Setting "
1939 "VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. "
1940 "See https://docs.vllm.ai/en/latest/getting_started/"
1941 - "debugging.html#python-multiprocessing "
1941 + "troubleshooting.html#python-multiprocessing "
1942 "for more information.")
1943 os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
1944