@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 6ad909fdda
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 04:30 UTC

[Doc] Improve GitHub links (#11491)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Dec 26, 2024 at 06:49 UTC 6ad909fdda54f91379bbee7590a37e38600b6204
31 files changed +147 -136
docs/source/conf.py
+29
@@ -74,6 +74,35 @@ html_theme_options = {
74 html_static_path = ["_static"]
75 html_js_files = ["custom.js"]
76
77 +myst_url_schemes = {
78 + 'http': None,
79 + 'https': None,
80 + 'mailto': None,
81 + 'ftp': None,
82 + "gh-issue": {
83 + "url":
84 + "https://github.com/vllm-project/vllm/issues/{{path}}#{{fragment}}",
85 + "title": "Issue #{{path}}",
86 + "classes": ["github"],
87 + },
88 + "gh-pr": {
89 + "url":
90 + "https://github.com/vllm-project/vllm/pull/{{path}}#{{fragment}}",
91 + "title": "Pull Request #{{path}}",
92 + "classes": ["github"],
93 + },
94 + "gh-dir": {
95 + "url": "https://github.com/vllm-project/vllm/tree/main/{{path}}",
96 + "title": "{{path}}",
97 + "classes": ["github"],
98 + },
99 + "gh-file": {
100 + "url": "https://github.com/vllm-project/vllm/blob/main/{{path}}",
101 + "title": "{{path}}",
102 + "classes": ["github"],
103 + },
104 +}
105 +
106 # see https://docs.readthedocs.io/en/stable/reference/environment-variables.html # noqa
107 READTHEDOCS_VERSION_TYPE = os.environ.get('READTHEDOCS_VERSION_TYPE')
108 if READTHEDOCS_VERSION_TYPE == "tag":
docs/source/contributing/dockerfile/dockerfile.md
+2 -2
@@ -1,7 +1,7 @@
1 # Dockerfile
2
3 -See [here](https://github.com/vllm-project/vllm/blob/main/Dockerfile) for the main Dockerfile to construct
4 -the image for running an OpenAI compatible server with vLLM. More information about deploying with Docker can be found [here](https://docs.vllm.ai/en/stable/serving/deploying_with_docker.html).
3 +We provide a <gh-file:Dockerfile> to construct the image for running an OpenAI compatible server with vLLM.
4 +More information about deploying with Docker can be found [here](../../serving/deploying_with_docker.md).
5
6 Below is a visual representation of the multi-stage Dockerfile. The build graph contains the following nodes:
7
docs/source/contributing/overview.md
+7 -7
@@ -13,11 +13,12 @@ Finally, one of the most impactful ways to support us is by raising awareness ab
13
14 ## License
15
16 -See [LICENSE](https://github.com/vllm-project/vllm/tree/main/LICENSE).
16 +See <gh-file:LICENSE>.
17
18 ## Developing
19
20 -Depending on the kind of development you'd like to do (e.g. Python, CUDA), you can choose to build vLLM with or without compilation. Check out the [building from source](https://docs.vllm.ai/en/latest/getting_started/installation.html#build-from-source) documentation for details.
20 +Depending on the kind of development you'd like to do (e.g. Python, CUDA), you can choose to build vLLM with or without compilation.
21 +Check out the [building from source](#build-from-source) documentation for details.
22
23 ## Testing
24
@@ -43,7 +44,7 @@ Currently, the repository does not pass the `mypy` tests.
44 If you encounter a bug or have a feature request, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible.
45
46 ```{important}
46 -If you discover a security vulnerability, please follow the instructions [here](https://github.com/vllm-project/vllm/tree/main/SECURITY.md#reporting-a-vulnerability).
47 +If you discover a security vulnerability, please follow the instructions [here](gh-file:SECURITY.md#reporting-a-vulnerability).
48 ```
49
50 ## Pull Requests & Code Reviews
@@ -54,9 +55,9 @@ code quality and improve the efficiency of the review process.
55
56 ### DCO and Signed-off-by
57
57 -When contributing changes to this project, you must agree to the [DCO](https://github.com/vllm-project/vllm/tree/main/DCO).
58 +When contributing changes to this project, you must agree to the <gh-file:DCO>.
59 Commits must include a `Signed-off-by:` header which certifies agreement with
59 -the terms of the [DCO](https://github.com/vllm-project/vllm/tree/main/DCO).
60 +the terms of the DCO.
61
62 Using `-s` with `git commit` will automatically add this header.
63
@@ -89,8 +90,7 @@ If the PR spans more than one category, please include all relevant prefixes.
90 The PR needs to meet the following code quality standards:
91
92 - We adhere to [Google Python style guide](https://google.github.io/styleguide/pyguide.html) and [Google C++ style guide](https://google.github.io/styleguide/cppguide.html).
92 -- Pass all linter checks. Please use [format.sh](https://github.com/vllm-project/vllm/blob/main/format.sh) to format your
93 - code.
93 +- Pass all linter checks. Please use <gh-file:format.sh> to format your code.
94 - The code needs to be well-documented to ensure future contributors can easily
95 understand the code.
96 - Include sufficient tests to ensure the project stays correct and robust. This
docs/source/contributing/profiling/profiling_index.md
+4 -4
@@ -22,13 +22,13 @@ Set the env variable VLLM_RPC_TIMEOUT to a big number before you start the serve
22 `export VLLM_RPC_TIMEOUT=1800000`
23 ```
24
25 -## Example commands and usage:
25 +## Example commands and usage
26
27 -### Offline Inference:
27 +### Offline Inference
28
29 -Refer to [examples/offline_inference_with_profiler.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_with_profiler.py) for an example.
29 +Refer to <gh-file:examples/offline_inference_with_profiler.py> for an example.
30
31 -### OpenAI Server:
31 +### OpenAI Server
32
33 ```bash
34 VLLM_TORCH_PROFILER_DIR=./vllm_profile python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-70B
docs/source/design/arch_overview.md
+6 -11
@@ -55,7 +55,7 @@ for output in outputs:
55 More API details can be found in the {doc}`Offline Inference
56 </dev/offline_inference/offline_index>` section of the API docs.
57
58 -The code for the `LLM` class can be found in [vllm/entrypoints/llm.py](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py).
58 +The code for the `LLM` class can be found in <gh-file:vllm/entrypoints/llm.py>.
59
60 ### OpenAI-compatible API server
61
@@ -66,7 +66,7 @@ This server can be started using the `vllm serve` command.
66 vllm serve <model>
67 ```
68
69 -The code for the `vllm` CLI can be found in [vllm/scripts.py](https://github.com/vllm-project/vllm/blob/main/vllm/scripts.py).
69 +The code for the `vllm` CLI can be found in <gh-file:vllm/scripts.py>.
70
71 Sometimes you may see the API server entrypoint used directly instead of via the
72 `vllm` CLI command. For example:
@@ -75,7 +75,7 @@ Sometimes you may see the API server entrypoint used directly instead of via the
75 python -m vllm.entrypoints.openai.api_server --model <model>
76 ```
77
78 -That code can be found in [vllm/entrypoints/openai/api_server.py](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/api_server.py).
78 +That code can be found in <gh-file:vllm/entrypoints/openai/api_server.py>.
79
80 More details on the API server can be found in the {doc}`OpenAI Compatible
81 Server </serving/openai_compatible_server>` document.
@@ -105,7 +105,7 @@ processing.
105 - **Output Processing**: Processes the outputs generated by the model, decoding the
106 token IDs from a language model into human-readable text.
107
108 -The code for `LLMEngine` can be found in [vllm/engine/llm_engine.py].
108 +The code for `LLMEngine` can be found in <gh-file:vllm/engine/llm_engine.py>.
109
110 ### AsyncLLMEngine
111
@@ -115,10 +115,9 @@ incoming requests. The `AsyncLLMEngine` is designed for online serving, where it
115 can handle multiple concurrent requests and stream outputs to clients.
116
117 The OpenAI-compatible API server uses the `AsyncLLMEngine`. There is also a demo
118 -API server that serves as a simpler example in
119 -[vllm/entrypoints/api_server.py].
118 +API server that serves as a simpler example in <gh-file:vllm/entrypoints/api_server.py>.
119
121 -The code for `AsyncLLMEngine` can be found in [vllm/engine/async_llm_engine.py].
120 +The code for `AsyncLLMEngine` can be found in <gh-file:vllm/engine/async_llm_engine.py>.
121
122 ## Worker
123
@@ -252,7 +251,3 @@ big problem.
251
252 In summary, the complete config object `VllmConfig` can be treated as an
253 engine-level global state that is shared among all vLLM classes.
255 -
256 -[vllm/engine/async_llm_engine.py]: https://github.com/vllm-project/vllm/tree/main/vllm/engine/async_llm_engine.py
257 -[vllm/engine/llm_engine.py]: https://github.com/vllm-project/vllm/tree/main/vllm/engine/llm_engine.py
258 -[vllm/entrypoints/api_server.py]: https://github.com/vllm-project/vllm/tree/main/vllm/entrypoints/api_server.py
docs/source/design/multiprocessing.md
+14 -13
@@ -2,13 +2,14 @@
2
3 ## Debugging
4
5 -Please see the [Debugging
6 -Tips](https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing)
5 +Please see the [Debugging Tips](#debugging-python-multiprocessing)
6 page for information on known issues and how to solve them.
7
8 ## Introduction
9
11 -*Note that source code references are to the state of the code at the time of writing in December, 2024.*
10 +```{important}
11 +The source code references are to the state of the code at the time of writing in December, 2024.
12 +```
13
14 The use of Python multiprocessing in vLLM is complicated by:
15
@@ -20,7 +21,7 @@ This document describes how vLLM deals with these challenges.
21
22 ## Multiprocessing Methods
23
23 -[Python multiprocessing methods](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) include:
24 +[Python multiprocessing methods](https://docs.python.org/3/library/multiprocessing.html.md#contexts-and-start-methods) include:
25
26 - `spawn` - spawn a new Python process. This will be the default as of Python
27 3.14.
@@ -82,7 +83,7 @@ There are other miscellaneous places hard-coding the use of `spawn`:
83
84 Related PRs:
85
85 -- <https://github.com/vllm-project/vllm/pull/8823>
86 +- <gh-pr:8823>
87
88 ## Prior State in v1
89
@@ -96,7 +97,7 @@ engine core.
97
98 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L93-L95>
99 - <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L70-L77>
99 -- https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/core_client.py#L44-L45
100 +- <https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/core_client.py#L44-L45>
101
102 It was off by default for all the reasons mentioned above - compatibility with
103 dependencies and code using vLLM as a library.
@@ -119,17 +120,17 @@ instruct users to either add a `__main__` guard or to disable multiprocessing.
120 If that known-failure case occurs, the user will see two messages that explain
121 what is happening. First, a log message from vLLM:
122
122 -```
123 - WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
124 - initialized. We must use the `spawn` multiprocessing start method. Setting
125 - VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
126 - https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing
127 - for more information.
123 +```console
124 +WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
125 + initialized. We must use the `spawn` multiprocessing start method. Setting
126 + VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
127 + https://docs.vllm.ai/en/latest/getting_started/debugging.html#python-multiprocessing
128 + for more information.
129 ```
130
131 Second, Python itself will raise an exception with a nice explanation:
132
132 -```
133 +```console
134 RuntimeError:
135 An attempt has been made to start a new process before the
136 current process has finished its bootstrapping phase.
docs/source/generate_examples.py
+1 -2
@@ -36,11 +36,10 @@ def generate_examples():
36
37 # Generate the example docs for each example script
38 for script_path, doc_path in zip(script_paths, doc_paths):
39 - script_url = f"https://github.com/vllm-project/vllm/blob/main/examples/{script_path.name}"
39 # Make script_path relative to doc_path and call it include_path
40 include_path = '../../../..' / script_path.relative_to(root_dir)
41 content = (f"{generate_title(doc_path.stem)}\n\n"
43 - f"Source: <{script_url}>.\n\n"
42 + f"Source: <gh-file:examples/{script_path.name}>.\n\n"
43 f"```{{literalinclude}} {include_path}\n"
44 ":language: python\n"
45 ":linenos:\n```")
docs/source/getting_started/amd-installation.md
+2 -2
@@ -22,7 +22,7 @@ Installation options:
22
23 You can build and install vLLM from source.
24
25 -First, build a docker image from [Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/Dockerfile.rocm) and launch a docker container from the image.
25 +First, build a docker image from <gh-file:Dockerfile.rocm> and launch a docker container from the image.
26 It is important that the user kicks off the docker build using buildkit. Either the user put DOCKER_BUILDKIT=1 as environment variable when calling docker build command, or the user needs to setup buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
27
28 ```console
@@ -33,7 +33,7 @@ It is important that the user kicks off the docker build using buildkit. Either
33 }
34 ```
35
36 -[Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/Dockerfile.rocm) uses ROCm 6.2 by default, but also supports ROCm 5.7, 6.0 and 6.1 in older vLLM branches.
36 +<gh-file:Dockerfile.rocm> uses ROCm 6.2 by default, but also supports ROCm 5.7, 6.0 and 6.1 in older vLLM branches.
37 It provides flexibility to customize the build of docker image using the following arguments:
38
39 - `BASE_IMAGE`: specifies the base image used when running `docker build`, specifically the PyTorch on ROCm base image.
docs/source/getting_started/cpu-installation.md
+2 -2
@@ -145,10 +145,10 @@ $ python examples/offline_inference.py
145
146 - On CPU based setup with NUMA enabled, the memory access performance may be largely impacted by the [topology](https://github.com/intel/intel-extension-for-pytorch/blob/main/docs/tutorials/performance_tuning/tuning_guide.md#non-uniform-memory-access-numa). For NUMA architecture, two optimizations are to recommended: Tensor Parallel or Data Parallel.
147
148 - - Using Tensor Parallel for a latency constraints deployment: following GPU backend design, a Megatron-LM's parallel algorithm will be used to shard the model, based on the number of NUMA nodes (e.g. TP = 2 for a two NUMA node system). With [TP feature on CPU](https://github.com/vllm-project/vllm/pull/6125) merged, Tensor Parallel is supported for serving and offline inferencing. In general each NUMA node is treated as one GPU card. Below is the example script to enable Tensor Parallel = 2 for serving:
148 + - Using Tensor Parallel for a latency constraints deployment: following GPU backend design, a Megatron-LM's parallel algorithm will be used to shard the model, based on the number of NUMA nodes (e.g. TP = 2 for a two NUMA node system). With [TP feature on CPU](gh-pr:6125) merged, Tensor Parallel is supported for serving and offline inferencing. In general each NUMA node is treated as one GPU card. Below is the example script to enable Tensor Parallel = 2 for serving:
149
150 ```console
151 $ VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" vllm serve meta-llama/Llama-2-7b-chat-hf -tp=2 --distributed-executor-backend mp
152 ```
153
154 - - Using Data Parallel for maximum throughput: to launch an LLM serving endpoint on each NUMA node along with one additional load balancer to dispatch the requests to those endpoints. Common solutions like [Nginx](../serving/deploying_with_nginx) or HAProxy are recommended. Anyscale Ray project provides the feature on LLM [serving](https://docs.ray.io/en/latest/serve/index.html). Here is the example to setup a scalable LLM serving with [Ray Serve](https://github.com/intel/llm-on-ray/blob/main/docs/setup.md).
154 + - Using Data Parallel for maximum throughput: to launch an LLM serving endpoint on each NUMA node along with one additional load balancer to dispatch the requests to those endpoints. Common solutions like [Nginx](../serving/deploying_with_nginx.md) or HAProxy are recommended. Anyscale Ray project provides the feature on LLM [serving](https://docs.ray.io/en/latest/serve/index.html). Here is the example to setup a scalable LLM serving with [Ray Serve](https://github.com/intel/llm-on-ray/blob/main/docs/setup.md).
docs/source/getting_started/debugging.md
+4 -3
@@ -24,7 +24,7 @@ To isolate the model downloading and loading issue, you can use the `--load-form
24
25 ## Model is too large
26
27 -If the model is too large to fit in a single GPU, you might want to [consider tensor parallelism](https://docs.vllm.ai/en/latest/serving/distributed_serving.html#distributed-inference-and-serving) to split the model across multiple GPUs. In that case, every process will read the whole model and split it into chunks, which makes the disk reading time even longer (proportional to the size of tensor parallelism). You can convert the model checkpoint to a sharded checkpoint using [this example](https://docs.vllm.ai/en/latest/getting_started/examples/save_sharded_state.html) . The conversion process might take some time, but later you can load the sharded checkpoint much faster. The model loading time should remain constant regardless of the size of tensor parallelism.
27 +If the model is too large to fit in a single GPU, you might want to [consider tensor parallelism](#distributed-serving) to split the model across multiple GPUs. In that case, every process will read the whole model and split it into chunks, which makes the disk reading time even longer (proportional to the size of tensor parallelism). You can convert the model checkpoint to a sharded checkpoint using <gh-file:examples/save_sharded_state.py>. The conversion process might take some time, but later you can load the sharded checkpoint much faster. The model loading time should remain constant regardless of the size of tensor parallelism.
28
29 ## Enable more logging
30
@@ -139,6 +139,7 @@ A multi-node environment is more complicated than a single-node one. If you see
139 Adjust `--nproc-per-node`, `--nnodes`, and `--node-rank` according to your setup, being sure to execute different commands (with different `--node-rank`) on different nodes.
140 ```
141
142 +(debugging-python-multiprocessing)=
143 ## Python multiprocessing
144
145 ### `RuntimeError` Exception
@@ -195,5 +196,5 @@ if __name__ == '__main__':
196
197 ## Known Issues
198
198 -- In `v0.5.2`, `v0.5.3`, and `v0.5.3.post1`, there is a bug caused by [zmq](https://github.com/zeromq/pyzmq/issues/2000) , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of `vllm` to include the [fix](https://github.com/vllm-project/vllm/pull/6759).
199 -- To circumvent a NCCL [bug](https://github.com/NVIDIA/nccl/issues/1234) , all vLLM processes will set an environment variable ``NCCL_CUMEM_ENABLE=0`` to disable NCCL's ``cuMem`` allocator. It does not affect performance but only gives memory benefits. When external processes want to set up a NCCL connection with vLLM's processes, they should also set this environment variable, otherwise, inconsistent environment setup will cause NCCL to hang or crash, as observed in the [RLHF integration](https://github.com/OpenRLHF/OpenRLHF/pull/604) and the [discussion](https://github.com/vllm-project/vllm/issues/5723#issuecomment-2554389656) .
199 +- In `v0.5.2`, `v0.5.3`, and `v0.5.3.post1`, there is a bug caused by [zmq](https://github.com/zeromq/pyzmq/issues/2000) , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of `vllm` to include the [fix](gh-pr:6759).
200 +- To circumvent a NCCL [bug](https://github.com/NVIDIA/nccl/issues/1234) , all vLLM processes will set an environment variable ``NCCL_CUMEM_ENABLE=0`` to disable NCCL's ``cuMem`` allocator. It does not affect performance but only gives memory benefits. When external processes want to set up a NCCL connection with vLLM's processes, they should also set this environment variable, otherwise, inconsistent environment setup will cause NCCL to hang or crash, as observed in the [RLHF integration](https://github.com/OpenRLHF/OpenRLHF/pull/604) and the [discussion](gh-issue:5723#issuecomment-2554389656) .
docs/source/getting_started/gaudi-installation.md
+2 -4
@@ -80,10 +80,8 @@ $ python setup.py develop
80
81 ## Supported Features
82
83 -- [Offline batched
84 - inference](https://docs.vllm.ai/en/latest/getting_started/quickstart.html#offline-batched-inference)
85 -- Online inference via [OpenAI-Compatible
86 - Server](https://docs.vllm.ai/en/latest/getting_started/quickstart.html#openai-compatible-server)
83 +- [Offline batched inference](#offline-batched-inference)
84 +- Online inference via [OpenAI-Compatible Server](#openai-compatible-server)
85 - HPU autodetection - no need to manually select device within vLLM
86 - Paged KV cache with algorithms enabled for Intel Gaudi accelerators
87 - Custom Intel Gaudi implementations of Paged Attention, KV cache ops,
docs/source/getting_started/installation.md
+1 -1
@@ -24,7 +24,7 @@ $ pip install vllm
24 ```
25
26 ```{note}
27 -Although we recommend using `conda` to create and manage Python environments, it is highly recommended to use `pip` to install vLLM. This is because `pip` can install `torch` with separate library packages like `NCCL`, while `conda` installs `torch` with statically linked `NCCL`. This can cause issues when vLLM tries to use `NCCL`. See [this issue](https://github.com/vllm-project/vllm/issues/8420) for more details.
27 +Although we recommend using `conda` to create and manage Python environments, it is highly recommended to use `pip` to install vLLM. This is because `pip` can install `torch` with separate library packages like `NCCL`, while `conda` installs `torch` with statically linked `NCCL`. This can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
28 ```
29
30 ````{note}
docs/source/getting_started/quickstart.md
+4 -3
@@ -29,7 +29,7 @@ Please refer to the {ref}`installation documentation <installation>` for more de
29
30 ## Offline Batched Inference
31
32 -With vLLM installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing). The example script for this section can be found [here](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference.py).
32 +With vLLM installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing). See the example script: <gh-file:examples/offline_inference.py>
33
34 The first line of this example imports the classes {class}`~vllm.LLM` and {class}`~vllm.SamplingParams`:
35
@@ -87,7 +87,8 @@ $ vllm serve Qwen/Qwen2.5-1.5B-Instruct
87 ```
88
89 ```{note}
90 -By default, the server uses a predefined chat template stored in the tokenizer. You can learn about overriding it [here](https://github.com/vllm-project/vllm/blob/main/docs/source/serving/openai_compatible_server.md#chat-template).
90 +By default, the server uses a predefined chat template stored in the tokenizer.
91 +You can learn about overriding it [here](#chat-template).
92 ```
93
94 This server can be queried in the same format as OpenAI API. For example, to list the models:
@@ -130,7 +131,7 @@ completion = client.completions.create(model="Qwen/Qwen2.5-1.5B-Instruct",
131 print("Completion result:", completion)
132 ```
133
133 -A more detailed client example can be found [here](https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py).
134 +A more detailed client example can be found here: <gh-file:examples/openai_completion_client.py>
135
136 ### OpenAI Chat Completions API with vLLM
137
docs/source/getting_started/tpu-installation.md
+1 -2
@@ -154,8 +154,7 @@ For more information about using TPUs with GKE, see
154
155 ## Build a docker image with {code}`Dockerfile.tpu`
156
157 -You can use [Dockerfile.tpu](https://github.com/vllm-project/vllm/blob/main/Dockerfile.tpu)
158 -to build a Docker image with TPU support.
157 +You can use <gh-file:Dockerfile.tpu> to build a Docker image with TPU support.
158
159 ```console
160 $ docker build -f Dockerfile.tpu -t vllm-tpu .
docs/source/getting_started/xpu-installation.md
+1 -1
@@ -71,4 +71,4 @@ $ --pipeline-parallel-size=2 \
71 $ -tp=8
72 ```
73
74 -By default, a ray instance will be launched automatically if no existing one is detected in system, with `num-gpus` equals to `parallel_config.world_size`. We recommend properly starting a ray cluster before execution, referring helper [script](https://github.com/vllm-project/vllm/tree/main/examples/run_cluster.sh).
74 +By default, a ray instance will be launched automatically if no existing one is detected in system, with `num-gpus` equals to `parallel_config.world_size`. We recommend properly starting a ray cluster before execution, referring to the <gh-file:examples/run_cluster.sh> helper script.
docs/source/models/adding_model.md
+4 -4
@@ -31,8 +31,8 @@ If you don't want to fork the repository and modify vLLM's codebase, please refe
31
32 ## 1. Bring your model code
33
34 -Clone the PyTorch model code from the HuggingFace Transformers repository and put it into the [vllm/model_executor/models](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) directory.
35 -For instance, vLLM's [OPT model](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/opt.py) was adapted from the HuggingFace's [modeling_opt.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py) file.
34 +Clone the PyTorch model code from the HuggingFace Transformers repository and put it into the <gh-dir:vllm/model_executor/models> directory.
35 +For instance, vLLM's [OPT model](gh-file:vllm/model_executor/models/opt.py) was adapted from the HuggingFace's [modeling_opt.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py) file.
36
37 ```{warning}
38 When copying the model code, make sure to review and adhere to the code's copyright and licensing terms.
@@ -99,7 +99,7 @@ Currently, vLLM supports the basic multi-head attention mechanism and its varian
99 If your model employs a different attention mechanism, you will need to implement a new attention layer in vLLM.
100 ```
101
102 -For reference, check out the [LLAMA model](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out the [vLLM models](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) directory for more examples.
102 +For reference, check out our [Llama implementation](gh-file:vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out <gh-dir:vllm/model_executor/models> for more examples.
103
104 ## 3. (Optional) Implement tensor parallelism and quantization support
105
@@ -123,7 +123,7 @@ This method should load the weights from the HuggingFace's checkpoint file and a
123
124 ## 5. Register your model
125
126 -Finally, register your {code}`*ForCausalLM` class to the {code}`_VLLM_MODELS` in [vllm/model_executor/models/registry.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/registry.py).
126 +Finally, register your {code}`*ForCausalLM` class to the {code}`_VLLM_MODELS` in <gh-file:vllm/model_executor/models/registry.py>.
127
128 ## 6. Out-of-Tree Model Integration
129
docs/source/models/enabling_multimodal_inputs.md
+6 -6
@@ -78,8 +78,8 @@ and register it via {meth}`INPUT_REGISTRY.register_dummy_data <vllm.inputs.regis
78
79 Here are some examples:
80
81 -- Image inputs (static feature size): [LLaVA-1.5 Model](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava.py)
82 -- Image inputs (dynamic feature size): [LLaVA-NeXT Model](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava_next.py)
81 +- Image inputs (static feature size): [LLaVA-1.5 Model](gh-file:vllm/model_executor/models/llava.py)
82 +- Image inputs (dynamic feature size): [LLaVA-NeXT Model](gh-file:vllm/model_executor/models/llava_next.py)
83
84 ```{seealso}
85 [Input Processing Pipeline](#input-processing-pipeline)
@@ -107,8 +107,8 @@ The dummy data should have the maximum possible number of multi-modal tokens, as
107
108 Here are some examples:
109
110 -- Image inputs (static feature size): [LLaVA-1.5 Model](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava.py)
111 -- Image inputs (dynamic feature size): [LLaVA-NeXT Model](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava_next.py)
110 +- Image inputs (static feature size): [LLaVA-1.5 Model](gh-file:vllm/model_executor/models/llava.py)
111 +- Image inputs (dynamic feature size): [LLaVA-NeXT Model](gh-file:vllm/model_executor/models/llava_next.py)
112
113 ```{seealso}
114 [Input Processing Pipeline](#input-processing-pipeline)
@@ -135,8 +135,8 @@ You can register input processors via {meth}`INPUT_REGISTRY.register_input_proce
135 A common use case of input processors is inserting placeholder tokens to leverage the vLLM framework for attention mask generation.
136 Here are some examples:
137
138 -- Insert static number of image tokens: [LLaVA-1.5 Model](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava.py)
139 -- Insert dynamic number of image tokens: [LLaVA-NeXT Model](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava_next.py)
138 +- Insert static number of image tokens: [LLaVA-1.5 Model](gh-file:vllm/model_executor/models/llava.py)
139 +- Insert dynamic number of image tokens: [LLaVA-NeXT Model](gh-file:vllm/model_executor/models/llava_next.py)
140
141 ```{seealso}
142 [Input Processing Pipeline](#input-processing-pipeline)
docs/source/models/generative_models.md
+3 -3
@@ -46,7 +46,7 @@ for output in outputs:
46 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
47 ```
48
49 -A code example can be found in [examples/offline_inference.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference.py).
49 +A code example can be found here: <gh-file:examples/offline_inference.py>
50
51 ### `LLM.beam_search`
52
@@ -103,7 +103,7 @@ for output in outputs:
103 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
104 ```
105
106 -A code example can be found in [examples/offline_inference_chat.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_chat.py).
106 +A code example can be found here: <gh-file:examples/offline_inference_chat.py>
107
108 If the model doesn't have a chat template or you want to specify another one,
109 you can explicitly pass a chat template:
@@ -120,7 +120,7 @@ outputs = llm.chat(conversation, chat_template=custom_template)
120
121 ## Online Inference
122
123 -Our [OpenAI Compatible Server](../serving/openai_compatible_server) provides endpoints that correspond to the offline APIs:
123 +Our [OpenAI Compatible Server](../serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs:
124
125 - [Completions API](#completions-api) is similar to `LLM.generate` but only accepts text.
126 - [Chat API](#chat-api) is similar to `LLM.chat`, accepting both text and [multi-modal inputs](#multimodal-inputs) for models with a chat template.
docs/source/models/pooling_models.md
+3 -3
@@ -65,7 +65,7 @@ embeds = output.outputs.embedding
65 print(f"Embeddings: {embeds!r} (size={len(embeds)})")
66 ```
67
68 -A code example can be found in [examples/offline_inference_embedding.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_embedding.py).
68 +A code example can be found here: <gh-file:examples/offline_inference_embedding.py>
69
70 ### `LLM.classify`
71
@@ -80,7 +80,7 @@ probs = output.outputs.probs
80 print(f"Class Probabilities: {probs!r} (size={len(probs)})")
81 ```
82
83 -A code example can be found in [examples/offline_inference_classification.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_classification.py).
83 +A code example can be found here: <gh-file:examples/offline_inference_classification.py>
84
85 ### `LLM.score`
86
@@ -102,7 +102,7 @@ score = output.outputs.score
102 print(f"Score: {score}")
103 ```
104
105 -A code example can be found in [examples/offline_inference_scoring.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_scoring.py).
105 +A code example can be found here: <gh-file:examples/offline_inference_scoring.py>
106
107 ## Online Inference
108
docs/source/models/supported_models.md
+2 -2
@@ -756,7 +756,7 @@ and pass {code}`--hf_overrides '{"architectures": ["MantisForConditionalGenerati
756
757 ```{note}
758 The official {code}`openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork ({code}`HwwwH/MiniCPM-V-2`) for now.
759 -For more details, please see: <https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630>
759 +For more details, please see: <gh-pr:4087#issuecomment-2250397630>
760 ```
761
762 ### Pooling Models
@@ -834,5 +834,5 @@ We have the following levels of testing for models:
834
835 1. **Strict Consistency**: We compare the output of the model with the output of the model in the HuggingFace Transformers library under greedy decoding. This is the most stringent test. Please refer to [models tests](https://github.com/vllm-project/vllm/blob/main/tests/models) for the models that have passed this test.
836 2. **Output Sensibility**: We check if the output of the model is sensible and coherent, by measuring the perplexity of the output and checking for any obvious errors. This is a less stringent test.
837 -3. **Runtime Functionality**: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to [functionality tests](https://github.com/vllm-project/vllm/tree/main/tests) and [examples](https://github.com/vllm-project/vllm/tree/main/examples) for the models that have passed this test.
837 +3. **Runtime Functionality**: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to [functionality tests](gh-dir:tests) and [examples](gh-dir:main/examples) for the models that have passed this test.
838 4. **Community Feedback**: We rely on the community to provide feedback on the models. If a model is broken or not working as expected, we encourage users to raise issues to report it or open pull requests to fix it. The rest of the models fall under this category.
docs/source/performance/benchmarks.md
+2 -2
@@ -15,7 +15,7 @@ The performance benchmarks are used for development to confirm whether new chang
15
16 The latest performance results are hosted on the public [vLLM Performance Dashboard](https://perf.vllm.ai).
17
18 -More information on the performance benchmarks and their parameters can be found [here](https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md).
18 +More information on the performance benchmarks and their parameters can be found [here](gh-file:.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md).
19
20 (nightly-benchmarks)=
21
@@ -25,4 +25,4 @@ These compare vLLM's performance against alternatives (`tgi`, `trt-llm`, and `lm
25
26 The latest nightly benchmark results are shared in major release blog posts such as [vLLM v0.6.0](https://blog.vllm.ai/2024/09/05/perf-update.html).
27
28 -More information on the nightly benchmarks and their parameters can be found [here](https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/nightly-descriptions.md).
28 +More information on the nightly benchmarks and their parameters can be found [here](gh-file:.buildkite/nightly-benchmarks/nightly-descriptions.md).
docs/source/quantization/supported_hardware.md
+1 -1
@@ -129,4 +129,4 @@ The table below shows the compatibility of various quantization implementations
129
130 Please note that this compatibility chart may be subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
131
132 -For the most up-to-date information on hardware support and quantization methods, please check the [quantization directory](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization) or consult with the vLLM development team.
132 +For the most up-to-date information on hardware support and quantization methods, please refer to <gh-dir:vllm/model_executor/layers/quantization> or consult with the vLLM development team.
docs/source/serving/deploying_with_docker.md
+1 -1
@@ -25,7 +25,7 @@ memory to share data between processes under the hood, particularly for tensor p
25
26 ## Building vLLM's Docker Image from Source
27
28 -You can build and run vLLM from source via the provided [Dockerfile](https://github.com/vllm-project/vllm/blob/main/Dockerfile). To build vLLM:
28 +You can build and run vLLM from source via the provided <gh-file:Dockerfile>. To build vLLM:
29
30 ```console
31 $ # optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2
docs/source/serving/distributed_serving.md
+2 -2
@@ -51,7 +51,7 @@ $ --pipeline-parallel-size 2
51
52 If a single node does not have enough GPUs to hold the model, you can run the model using multiple nodes. It is important to make sure the execution environment is the same on all nodes, including the model path, the Python environment. The recommended way is to use docker images to ensure the same environment, and hide the heterogeneity of the host machines via mapping them into the same docker configuration.
53
54 -The first step, is to start containers and organize them into a cluster. We have provided a helper [script](https://github.com/vllm-project/vllm/tree/main/examples/run_cluster.sh) to start the cluster. Please note, this script launches docker without administrative privileges that would be required to access GPU performance counters when running profiling and tracing tools. For that purpose, the script can have `CAP_SYS_ADMIN` to the docker container by using the `--cap-add` option in the docker run command.
54 +The first step, is to start containers and organize them into a cluster. We have provided the helper script <gh-file:examples/run_cluster.sh> to start the cluster. Please note, this script launches docker without administrative privileges that would be required to access GPU performance counters when running profiling and tracing tools. For that purpose, the script can have `CAP_SYS_ADMIN` to the docker container by using the `--cap-add` option in the docker run command.
55
56 Pick a node as the head node, and run the following command:
57
@@ -95,7 +95,7 @@ $ --tensor-parallel-size 16
95 To make tensor parallel performant, you should make sure the communication between nodes is efficient, e.g. using high-speed network cards like Infiniband. To correctly set up the cluster to use Infiniband, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the `run_cluster.sh` script. Please contact your system administrator for more information on how to set up the flags. One way to confirm if the Infiniband is working is to run vLLM with `NCCL_DEBUG=TRACE` environment variable set, e.g. `NCCL_DEBUG=TRACE vllm serve ...` and check the logs for the NCCL version and the network used. If you find `[send] via NET/Socket` in the logs, it means NCCL uses raw TCP Socket, which is not efficient for cross-node tensor parallel. If you find `[send] via NET/IB/GDRDMA` in the logs, it means NCCL uses Infiniband with GPU-Direct RDMA, which is efficient.
96
97 ```{warning}
98 -After you start the Ray cluster, you'd better also check the GPU-GPU communication between nodes. It can be non-trivial to set up. Please refer to the [sanity check script](https://docs.vllm.ai/en/latest/getting_started/debugging.html) for more information. If you need to set some environment variables for the communication configuration, you can append them to the `run_cluster.sh` script, e.g. `-e NCCL_SOCKET_IFNAME=eth0`. Note that setting environment variables in the shell (e.g. `NCCL_SOCKET_IFNAME=eth0 vllm serve ...`) only works for the processes in the same node, not for the processes in the other nodes. Setting environment variables when you create the cluster is the recommended way. See the [discussion](https://github.com/vllm-project/vllm/issues/6803) for more information.
98 +After you start the Ray cluster, you'd better also check the GPU-GPU communication between nodes. It can be non-trivial to set up. Please refer to the [sanity check script](../getting_started/debugging.md) for more information. If you need to set some environment variables for the communication configuration, you can append them to the `run_cluster.sh` script, e.g. `-e NCCL_SOCKET_IFNAME=eth0`. Note that setting environment variables in the shell (e.g. `NCCL_SOCKET_IFNAME=eth0 vllm serve ...`) only works for the processes in the same node, not for the processes in the other nodes. Setting environment variables when you create the cluster is the recommended way. See <gh-issue:6803> for more information.
99 ```
100
101 ```{warning}
docs/source/serving/openai_compatible_server.md
+6 -17
@@ -65,8 +65,7 @@ and all chat requests will error.
65 vllm serve <model> --chat-template ./path-to-chat-template.jinja
66 ```
67
68 -vLLM community provides a set of chat templates for popular models. You can find them in the examples
69 -directory [here](https://github.com/vllm-project/vllm/tree/main/examples/)
68 +vLLM community provides a set of chat templates for popular models. You can find them under the <gh-dir:examples> directory.
69
70 With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
71 both a `type` and a `text` field. An example is provided below:
@@ -184,9 +183,7 @@ The order of priorities is `command line > config file values > defaults`.
183 Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
184 you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
185
187 -#### Code example
188 -
189 -See [examples/openai_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py).
186 +Code example: <gh-file:examples/openai_completion_client.py>
187
188 #### Extra parameters
189
@@ -217,9 +214,7 @@ We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
214 see our [Multimodal Inputs](../usage/multimodal_inputs.md) guide for more information.
215 - *Note: `image_url.detail` parameter is not supported.*
216
220 -#### Code example
221 -
222 -See [examples/openai_chat_completion_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client.py).
217 +Code example: <gh-file:examples/openai_chat_completion_client.py>
218
219 #### Extra parameters
220
@@ -252,9 +247,7 @@ which will be treated as a single prompt to the model.
247 This enables multi-modal inputs to be passed to embedding models, see [this page](#multimodal-inputs) for details.
248 ```
249
255 -#### Code example
256 -
257 -See [examples/openai_embedding_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_embedding_client.py).
250 +Code example: <gh-file:examples/openai_embedding_client.py>
251
252 #### Extra parameters
253
@@ -298,9 +291,7 @@ Our Pooling API encodes input prompts using a [pooling model](../models/pooling_
291
292 The input format is the same as [Embeddings API](#embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.
293
301 -#### Code example
302 -
303 -See [examples/openai_pooling_client.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_pooling_client.py).
294 +Code example: <gh-file:examples/openai_pooling_client.py>
295
296 (score-api)=
297 ### Score API
@@ -310,9 +301,7 @@ Usually, the score for a sentence pair refers to the similarity between two sent
301
302 You can find the documentation for these kind of models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
303
313 -#### Code example
314 -
315 -See [examples/openai_cross_encoder_score.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_cross_encoder_score.py).
304 +Code example: <gh-file:examples/openai_cross_encoder_score.py>
305
306 #### Single inference
307
docs/source/usage/compatibility_matrix.md
+17 -17
@@ -82,7 +82,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
82 -
83 -
84 * - [LoRA](#lora-adapter)
85 - - [✗](https://github.com/vllm-project/vllm/pull/9057)
85 + - [✗](gh-pr:9057)
86 - ✅
87 -
88 -
@@ -168,10 +168,10 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
168 -
169 * - <abbr title="Encoder-Decoder Models">enc-dec</abbr>
170 - ✗
171 - - [✗](https://github.com/vllm-project/vllm/issues/7366)
171 + - [✗](gh-issue:7366)
172 - ✗
173 - ✗
174 - - [✗](https://github.com/vllm-project/vllm/issues/7366)
174 + - [✗](gh-issue:7366)
175 - ✅
176 - ✅
177 -
@@ -205,7 +205,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
205 - ✅
206 - ✅
207 - ✅
208 - - [✗](https://github.com/vllm-project/vllm/pull/8199)
208 + - [✗](gh-pr:8199)
209 - ✅
210 - ✗
211 - ✅
@@ -244,7 +244,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
244 - ✗
245 - ✗
246 - ✅
247 - - [✗](https://github.com/vllm-project/vllm/issues/8198)
247 + - [✗](gh-issue:8198)
248 - ✅
249 -
250 -
@@ -253,8 +253,8 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
253 -
254 * - <abbr title="Multimodal Inputs">mm</abbr>
255 - ✅
256 - - [✗](https://github.com/vllm-project/vllm/pull/8348)
257 - - [✗](https://github.com/vllm-project/vllm/pull/7199)
256 + - [✗](gh-pr:8348)
257 + - [✗](gh-pr:7199)
258 - ?
259 - ?
260 - ✅
@@ -273,14 +273,14 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
273 - ✅
274 - ✅
275 - ✅
276 - - [✗](https://github.com/vllm-project/vllm/issues/6137)
276 + - [✗](gh-issue:6137)
277 - ✅
278 - ✗
279 - ✅
280 - ✅
281 - ✅
282 - ?
283 - - [✗](https://github.com/vllm-project/vllm/issues/7968)
283 + - [✗](gh-issue:7968)
284 - ✅
285 -
286 -
@@ -290,14 +290,14 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
290 - ✅
291 - ✅
292 - ✅
293 - - [✗](https://github.com/vllm-project/vllm/issues/6137)
293 + - [✗](gh-issue:6137)
294 - ✅
295 - ✗
296 - ✅
297 - ✅
298 - ✅
299 - ?
300 - - [✗](https://github.com/vllm-project/vllm/issues/7968>)
300 + - [✗](gh-issue:7968>)
301 - ?
302 - ✅
303 -
@@ -314,7 +314,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
314 - ✅
315 - ✅
316 - ✅
317 - - [✗](https://github.com/vllm-project/vllm/issues/9893)
317 + - [✗](gh-issue:9893)
318 - ?
319 - ✅
320 - ✅
@@ -338,7 +338,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
338 - CPU
339 - AMD
340 * - [CP](#chunked-prefill)
341 - - [✗](https://github.com/vllm-project/vllm/issues/2729)
341 + - [✗](gh-issue:2729)
342 - ✅
343 - ✅
344 - ✅
@@ -346,7 +346,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
346 - ✅
347 - ✅
348 * - [APC](#apc)
349 - - [✗](https://github.com/vllm-project/vllm/issues/3687)
349 + - [✗](gh-issue:3687)
350 - ✅
351 - ✅
352 - ✅
@@ -359,7 +359,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
359 - ✅
360 - ✅
361 - ✅
362 - - [✗](https://github.com/vllm-project/vllm/pull/4830)
362 + - [✗](gh-pr:4830)
363 - ✅
364 * - <abbr title="Prompt Adapter">prmpt adptr</abbr>
365 - ✅
@@ -367,7 +367,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
367 - ✅
368 - ✅
369 - ✅
370 - - [✗](https://github.com/vllm-project/vllm/issues/8475)
370 + - [✗](gh-issue:8475)
371 - ✅
372 * - [SD](#spec_decode)
373 - ✅
@@ -439,7 +439,7 @@ Check the '✗' with links to see tracking issue for unsupported feature/hardwar
439 - ✅
440 - ✅
441 - ✅
442 - - [✗](https://github.com/vllm-project/vllm/issues/8477)
442 + - [✗](gh-issue:8477)
443 - ✅
444 * - best-of
445 - ✅
docs/source/usage/lora.md
+1 -2
@@ -47,8 +47,7 @@ outputs = llm.generate(
47 )
48 ```
49
50 -Check out [examples/multilora_inference.py](https://github.com/vllm-project/vllm/blob/main/examples/multilora_inference.py)
51 -for an example of how to use LoRA adapters with the async engine and how to use more advanced configuration options.
50 +Check out <gh-file:examples/multilora_inference.py> for an example of how to use LoRA adapters with the async engine and how to use more advanced configuration options.
51
52 ## Serving LoRA Adapters
53
docs/source/usage/multimodal_inputs.md
+12 -12
@@ -5,7 +5,7 @@
5 This page teaches you how to pass multi-modal inputs to [multi-modal models](#supported-mm-models) in vLLM.
6
7 ```{note}
8 -We are actively iterating on multi-modal support. See [this RFC](https://github.com/vllm-project/vllm/issues/4194) for upcoming changes,
8 +We are actively iterating on multi-modal support. See [this RFC](gh-issue:4194) for upcoming changes,
9 and [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) if you have any feedback or feature requests.
10 ```
11
@@ -60,7 +60,7 @@ for o in outputs:
60 print(generated_text)
61 ```
62
63 -A code example can be found in [examples/offline_inference_vision_language.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py).
63 +Full example: <gh-file:examples/offline_inference_vision_language.py>
64
65 To substitute multiple images inside the same text prompt, you can pass in a list of images instead:
66
@@ -91,7 +91,7 @@ for o in outputs:
91 print(generated_text)
92 ```
93
94 -A code example can be found in [examples/offline_inference_vision_language_multi_image.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language_multi_image.py).
94 +Full example: <gh-file:examples/offline_inference_vision_language_multi_image.py>
95
96 Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:
97
@@ -125,13 +125,13 @@ for o in outputs:
125 You can pass a list of NumPy arrays directly to the {code}`'video'` field of the multi-modal dictionary
126 instead of using multi-image input.
127
128 -Please refer to [examples/offline_inference_vision_language.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py) for more details.
128 +Full example: <gh-file:examples/offline_inference_vision_language.py>
129
130 ### Audio
131
132 You can pass a tuple {code}`(array, sampling_rate)` to the {code}`'audio'` field of the multi-modal dictionary.
133
134 -Please refer to [examples/offline_inference_audio_language.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_audio_language.py) for more details.
134 +Full example: <gh-file:examples/offline_inference_audio_language.py>
135
136 ### Embedding
137
@@ -208,7 +208,7 @@ A chat template is **required** to use Chat Completions API.
208
209 Although most models come with a chat template, for others you have to define one yourself.
210 The chat template can be inferred based on the documentation on the model's HuggingFace repo.
211 -For example, LLaVA-1.5 (`llava-hf/llava-1.5-7b-hf`) requires a chat template that can be found [here](https://github.com/vllm-project/vllm/blob/main/examples/template_llava.jinja).
211 +For example, LLaVA-1.5 (`llava-hf/llava-1.5-7b-hf`) requires a chat template that can be found here: <gh-file:examples/template_llava.jinja>
212 ```
213
214 ### Image
@@ -271,7 +271,7 @@ chat_response = client.chat.completions.create(
271 print("Chat completion output:", chat_response.choices[0].message.content)
272 ```
273
274 -A full code example can be found in [examples/openai_chat_completion_client_for_multimodal.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client_for_multimodal.py).
274 +Full example: <gh-file:examples/openai_chat_completion_client_for_multimodal.py>
275
276 ```{tip}
277 Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via `--allowed-local-media-path` when launching the API server/engine,
@@ -296,7 +296,7 @@ $ export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
296
297 Instead of {code}`image_url`, you can pass a video file via {code}`video_url`.
298
299 -You can use [these tests](https://github.com/vllm-project/vllm/blob/main/tests/entrypoints/openai/test_video.py) as reference.
299 +You can use [these tests](gh-file:entrypoints/openai/test_video.py) as reference.
300
301 ````{note}
302 By default, the timeout for fetching videos through HTTP URL url is `30` seconds.
@@ -399,7 +399,7 @@ result = chat_completion_from_url.choices[0].message.content
399 print("Chat completion output from audio url:", result)
400 ```
401
402 -A full code example can be found in [examples/openai_chat_completion_client_for_multimodal.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_client_for_multimodal.py).
402 +Full example: <gh-file:examples/openai_chat_completion_client_for_multimodal.py>
403
404 ````{note}
405 By default, the timeout for fetching audios through HTTP URL is `10` seconds.
@@ -435,7 +435,7 @@ Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to expl
435 to run this model in embedding mode instead of text generation mode.
436
437 The custom chat template is completely different from the original one for this model,
438 -and can be found [here](https://github.com/vllm-project/vllm/blob/main/examples/template_vlm2vec.jinja).
438 +and can be found here: <gh-file:examples/template_vlm2vec.jinja>
439 ```
440
441 Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
@@ -475,7 +475,7 @@ vllm serve MrLight/dse-qwen2-2b-mrl-v1 --task embed \
475 Like with VLM2Vec, we have to explicitly pass `--task embed`.
476
477 Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled
478 -by [this custom chat template](https://github.com/vllm-project/vllm/blob/main/examples/template_dse_qwen2_vl.jinja).
478 +by a custom chat template: <gh-file:examples/template_dse_qwen2_vl.jinja>
479 ```
480
481 ```{important}
@@ -483,4 +483,4 @@ Also important, `MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of th
483 example below for details.
484 ```
485
486 -A full code example can be found in [examples/openai_chat_embedding_client_for_multimodal.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_embedding_client_for_multimodal.py).
486 +Full example: <gh-file:examples/openai_chat_embedding_client_for_multimodal.py>
docs/source/usage/spec_decode.md
+4 -4
@@ -4,8 +4,8 @@
4
5 ```{warning}
6 Please note that speculative decoding in vLLM is not yet optimized and does
7 -not usually yield inter-token latency reductions for all prompt datasets or sampling parameters. The work
8 -to optimize it is ongoing and can be followed in [this issue.](https://github.com/vllm-project/vllm/issues/4630)
7 +not usually yield inter-token latency reductions for all prompt datasets or sampling parameters.
8 +The work to optimize it is ongoing and can be followed here: <gh-issue:4630>
9 ```
10
11 ```{warning}
@@ -176,7 +176,7 @@ speculative decoding, breaking down the guarantees into three key areas:
176 > distribution. [View Test Code](https://github.com/vllm-project/vllm/blob/47b65a550866c7ffbd076ecb74106714838ce7da/tests/samplers/test_rejection_sampler.py#L252)
177 > - **Greedy Sampling Equality**: Confirms that greedy sampling with speculative decoding matches greedy sampling
178 > without it. This verifies that vLLM's speculative decoding framework, when integrated with the vLLM forward pass and the vLLM rejection sampler,
179 - > provides a lossless guarantee. Almost all of the tests in [this directory](https://github.com/vllm-project/vllm/tree/b67ae00cdbbe1a58ffc8ff170f0c8d79044a684a/tests/spec_decode/e2e)
179 + > provides a lossless guarantee. Almost all of the tests in <gh-dir:tests/spec_decode/e2e>.
180 > verify this property using [this assertion implementation](https://github.com/vllm-project/vllm/blob/b67ae00cdbbe1a58ffc8ff170f0c8d79044a684a/tests/spec_decode/e2e/conftest.py#L291)
181
182 3. **vLLM Logprob Stability**
@@ -202,4 +202,4 @@ For mitigation strategies, please refer to the FAQ entry *Can the output of a pr
202 - [A Hacker's Guide to Speculative Decoding in vLLM](https://www.youtube.com/watch?v=9wNAgpX6z_4)
203 - [What is Lookahead Scheduling in vLLM?](https://docs.google.com/document/d/1Z9TvqzzBPnh5WHcRwjvK2UEeFeq5zMZb5mFE8jR0HCs/edit#heading=h.1fjfb0donq5a)
204 - [Information on batch expansion](https://docs.google.com/document/d/1T-JaS2T1NRfdP51qzqpyakoCXxSXTtORppiwaj5asxA/edit#heading=h.kk7dq05lc6q8)
205 -- [Dynamic speculative decoding](https://github.com/vllm-project/vllm/issues/4565)
205 +- [Dynamic speculative decoding](gh-issue:4565)
docs/source/usage/structured_outputs.md
+2 -2
@@ -131,7 +131,7 @@ completion = client.chat.completions.create(
131 print(completion.choices[0].message.content)
132 ```
133
134 -The complete code of the examples can be found on [examples/openai_chat_completion_structured_outputs.py](https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_structured_outputs.py).
134 +Full example: <gh-file:examples/openai_chat_completion_structured_outputs.py>
135
136 ## Experimental Automatic Parsing (OpenAI API)
137
@@ -257,4 +257,4 @@ outputs = llm.generate(
257 print(outputs[0].outputs[0].text)
258 ```
259
260 -A complete example with all options can be found in [examples/offline_inference_structured_outputs.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_structured_outputs.py).
260 +Full example: <gh-file:examples/offline_inference_structured_outputs.py>
docs/source/usage/usage_stats.md
+1 -1
@@ -4,7 +4,7 @@ vLLM collects anonymous usage data by default to help the engineering team bette
4
5 ## What data is collected?
6
7 -You can see the up to date list of data collected by vLLM in the [usage_lib.py](https://github.com/vllm-project/vllm/blob/main/vllm/usage/usage_lib.py).
7 +The list of data collected by the latest version of vLLM can be found here: <gh-file:vllm/usage/usage_lib.py>
8
9 Here is an example as of v0.4.0:
10