@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 4ffd6e8942
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 05:30 UTC

[Docs] Reduce custom syntax used in docs (#27009)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Harry Mellor committed Oct 17, 2025 at 04:05 UTC 4ffd6e8942710310f8e4f1a84090784cc2f09e70
65 files changed +382 -403
docs/configuration/conserving_memory.md
+1 -1
@@ -23,7 +23,7 @@ llm = LLM(model="ibm-granite/granite-3.1-8b-instruct", tensor_parallel_size=2)
23 !!! note
24 With tensor parallelism enabled, each process will read the whole model and split it into chunks, which makes the disk reading time even longer (proportional to the size of tensor parallelism).
25
26 - You can convert the model checkpoint to a sharded checkpoint using <gh-file:examples/offline_inference/save_sharded_state.py>. The conversion process might take some time, but later you can load the sharded checkpoint much faster. The model loading time should remain constant regardless of the size of tensor parallelism.
26 + You can convert the model checkpoint to a sharded checkpoint using [examples/offline_inference/save_sharded_state.py](../../examples/offline_inference/save_sharded_state.py). The conversion process might take some time, but later you can load the sharded checkpoint much faster. The model loading time should remain constant regardless of the size of tensor parallelism.
27
28 ## Quantization
29
docs/configuration/optimization.md
+8 -8
@@ -174,14 +174,14 @@ Regardless, you need to set `mm_encoder_tp_mode="data"` in engine arguments to u
174
175 Known supported models (with corresponding benchmarks):
176
177 -- dots_ocr (<gh-pr:25466>)
178 -- GLM-4.1V or above (<gh-pr:23168>)
179 -- InternVL (<gh-pr:23909>)
180 -- Kimi-VL (<gh-pr:23817>)
181 -- Llama4 (<gh-pr:18368>)
182 -- MiniCPM-V-2.5 or above (<gh-pr:23327>, <gh-pr:23948>)
183 -- Qwen2-VL or above (<gh-pr:22742>, <gh-pr:24955>, <gh-pr:25445>)
184 -- Step3 (<gh-pr:22697>)
177 +- dots_ocr (<https://github.com/vllm-project/vllm/pull/25466>)
178 +- GLM-4.1V or above (<https://github.com/vllm-project/vllm/pull/23168>)
179 +- InternVL (<https://github.com/vllm-project/vllm/pull/23909>)
180 +- Kimi-VL (<https://github.com/vllm-project/vllm/pull/23817>)
181 +- Llama4 (<https://github.com/vllm-project/vllm/pull/18368>)
182 +- MiniCPM-V-2.5 or above (<https://github.com/vllm-project/vllm/pull/23327>, <https://github.com/vllm-project/vllm/pull/23948>)
183 +- Qwen2-VL or above (<https://github.com/vllm-project/vllm/pull/22742>, <https://github.com/vllm-project/vllm/pull/24955>, <https://github.com/vllm-project/vllm/pull/25445>)
184 +- Step3 (<https://github.com/vllm-project/vllm/pull/22697>)
185
186 ## Input Processing
187
docs/configuration/tpu.md
+1 -1
@@ -96,7 +96,7 @@ Although it’s common to do this with GPUs, don't try to fragment 2 or 8 differ
96
97 ### Tune your workloads
98
99 -Although we try to have great default configs, we strongly recommend you check out the [vLLM auto-tuner](gh-file:benchmarks/auto_tune/README.md) to optimize your workloads for your use case.
99 +Although we try to have great default configs, we strongly recommend you check out the [vLLM auto-tuner](../../benchmarks/auto_tune/README.md) to optimize your workloads for your use case.
100
101 ### Future Topics We'll Cover
102
docs/contributing/README.md
+5 -5
@@ -22,7 +22,7 @@ Unsure on where to start? Check out the following links for tasks to work on:
22
23 ## License
24
25 -See <gh-file:LICENSE>.
25 +See [LICENSE](../../LICENSE).
26
27 ## Developing
28
@@ -54,7 +54,7 @@ For more details about installing from source and installing for other hardware,
54 For an optimized workflow when iterating on C++/CUDA kernels, see the [Incremental Compilation Workflow](./incremental_build.md) for recommendations.
55
56 !!! tip
57 - vLLM is compatible with Python versions 3.10 to 3.13. However, vLLM's default [Dockerfile](gh-file:docker/Dockerfile) ships with Python 3.12 and tests in CI (except `mypy`) are run with Python 3.12.
57 + vLLM is compatible with Python versions 3.10 to 3.13. However, vLLM's default [Dockerfile](../../docker/Dockerfile) ships with Python 3.12 and tests in CI (except `mypy`) are run with Python 3.12.
58
59 Therefore, we recommend developing with Python 3.12 to minimise the chance of your local environment clashing with our CI environment.
60
@@ -88,7 +88,7 @@ vLLM's `pre-commit` hooks will now run automatically every time you commit.
88
89 ### Documentation
90
91 -MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file, <gh-file:mkdocs.yaml>.
91 +MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file, [mkdocs.yaml](../../mkdocs.yaml).
92
93 Get started with:
94
@@ -152,7 +152,7 @@ pytest -s -v tests/test_logger.py
152 If you encounter a bug or have a feature request, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible.
153
154 !!! important
155 - If you discover a security vulnerability, please follow the instructions [here](gh-file:SECURITY.md#reporting-a-vulnerability).
155 + If you discover a security vulnerability, please follow the instructions [here](../../SECURITY.md).
156
157 ## Pull Requests & Code Reviews
158
@@ -162,7 +162,7 @@ code quality and improve the efficiency of the review process.
162
163 ### DCO and Signed-off-by
164
165 -When contributing changes to this project, you must agree to the <gh-file:DCO>.
165 +When contributing changes to this project, you must agree to the [DCO](../../DCO).
166 Commits must include a `Signed-off-by:` header which certifies agreement with
167 the terms of the DCO.
168
docs/contributing/benchmarks.md
+3 -3
@@ -822,7 +822,7 @@ you should set `--endpoint /v1/embeddings` to use the Embeddings API. The backen
822 - CLIP: `--backend openai-embeddings-clip`
823 - VLM2Vec: `--backend openai-embeddings-vlm2vec`
824
825 -For other models, please add your own implementation inside <gh-file:vllm/benchmarks/lib/endpoint_request_func.py> to match the expected instruction format.
825 +For other models, please add your own implementation inside [vllm/benchmarks/lib/endpoint_request_func.py](../../vllm/benchmarks/lib/endpoint_request_func.py) to match the expected instruction format.
826
827 You can use any text or multi-modal dataset to benchmark the model, as long as the model supports it.
828 For example, you can use ShareGPT and VisionArena to benchmark vision-language embeddings.
@@ -962,7 +962,7 @@ For more results visualization, check the [visualizing the results](https://gith
962
963 The latest performance results are hosted on the public [vLLM Performance Dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm).
964
965 -More information on the performance benchmarks and their parameters can be found in [Benchmark README](https://github.com/intel-ai-tce/vllm/blob/more_cpu_models/.buildkite/nightly-benchmarks/README.md) and [performance benchmark description](gh-file:.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md).
965 +More information on the performance benchmarks and their parameters can be found in [Benchmark README](https://github.com/intel-ai-tce/vllm/blob/more_cpu_models/.buildkite/nightly-benchmarks/README.md) and [performance benchmark description](../../.buildkite/nightly-benchmarks/performance-benchmarks-descriptions.md).
966
967 ### Continuous Benchmarking
968
@@ -996,4 +996,4 @@ These compare vLLM's performance against alternatives (`tgi`, `trt-llm`, and `lm
996
997 The latest nightly benchmark results are shared in major release blog posts such as [vLLM v0.6.0](https://blog.vllm.ai/2024/09/05/perf-update.html).
998
999 -More information on the nightly benchmarks and their parameters can be found [here](gh-file:.buildkite/nightly-benchmarks/nightly-descriptions.md).
999 +More information on the nightly benchmarks and their parameters can be found [here](../../.buildkite/nightly-benchmarks/nightly-descriptions.md).
docs/contributing/ci/failures.md
+2 -2
@@ -64,7 +64,7 @@ Download the full log file from Buildkite locally.
64
65 Strip timestamps and colorization:
66
67 -<gh-file:.buildkite/scripts/ci-clean-log.sh>
67 +[.buildkite/scripts/ci-clean-log.sh](../../../.buildkite/scripts/ci-clean-log.sh)
68
69 ```bash
70 ./ci-clean-log.sh ci.log
@@ -87,7 +87,7 @@ tail -525 ci_build.log | wl-copy
87
88 CI test failures may be flaky. Use a bash loop to run repeatedly:
89
90 -<gh-file:.buildkite/scripts/rerun-test.sh>
90 +[.buildkite/scripts/rerun-test.sh](../../../.buildkite/scripts/rerun-test.sh)
91
92 ```bash
93 ./rerun-test.sh tests/v1/engine/test_engine_core_client.py::test_kv_cache_events[True-tcp]
docs/contributing/ci/update_pytorch_version.md
+4 -4
@@ -5,7 +5,7 @@ release in CI/CD. It is standard practice to submit a PR to update the
5 PyTorch version as early as possible when a new [PyTorch stable
6 release](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-cadence) becomes available.
7 This process is non-trivial due to the gap between PyTorch
8 -releases. Using <gh-pr:16859> as an example, this document outlines common steps to achieve this
8 +releases. Using <https://github.com/vllm-project/vllm/pull/16859> as an example, this document outlines common steps to achieve this
9 update along with a list of potential issues and how to address them.
10
11 ## Test PyTorch release candidates (RCs)
@@ -85,7 +85,7 @@ and timeout. Additionally, since vLLM's fastcheck pipeline runs in read-only mod
85 it doesn't populate the cache, so re-running it to warm up the cache
86 is ineffective.
87
88 -While ongoing efforts like [#17419](gh-issue:17419)
88 +While ongoing efforts like <https://github.com/vllm-project/vllm/issues/17419>
89 address the long build time at its source, the current workaround is to set `VLLM_CI_BRANCH`
90 to a custom branch provided by @khluu (`VLLM_CI_BRANCH=khluu/use_postmerge_q`)
91 when manually triggering a build on Buildkite. This branch accomplishes two things:
@@ -138,5 +138,5 @@ to handle some platforms separately. The separation of requirements and Dockerfi
138 for different platforms in vLLM CI/CD allows us to selectively choose
139 which platforms to update. For instance, updating XPU requires the corresponding
140 release from [Intel Extension for PyTorch](https://github.com/intel/intel-extension-for-pytorch) by Intel.
141 -While <gh-pr:16859> updated vLLM to PyTorch 2.7.0 on CPU, CUDA, and ROCm,
142 -<gh-pr:17444> completed the update for XPU.
141 +While <https://github.com/vllm-project/vllm/pull/16859> updated vLLM to PyTorch 2.7.0 on CPU, CUDA, and ROCm,
142 +<https://github.com/vllm-project/vllm/pull/17444> completed the update for XPU.
docs/contributing/dockerfile/dockerfile.md
+1 -1
@@ -1,6 +1,6 @@
1 # Dockerfile
2
3 -We provide a <gh-file:docker/Dockerfile> to construct the image for running an OpenAI compatible server with vLLM.
3 +We provide a [docker/Dockerfile](../../../docker/Dockerfile) to construct the image for running an OpenAI compatible server with vLLM.
4 More information about deploying with Docker can be found [here](../../deployment/docker.md).
5
6 Below is a visual representation of the multi-stage Dockerfile. The build graph contains the following nodes:
docs/contributing/model/basic.md
+10 -10
@@ -5,7 +5,7 @@ This guide walks you through the steps to implement a basic vLLM model.
5 ## 1. Bring your model code
6
7 First, clone the PyTorch model code from the source repository.
8 -For instance, vLLM's [OPT model](gh-file:vllm/model_executor/models/opt.py) was adapted from
8 +For instance, vLLM's [OPT model](../../../vllm/model_executor/models/opt.py) was adapted from
9 HuggingFace's [modeling_opt.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py) file.
10
11 !!! warning
@@ -83,7 +83,7 @@ def forward(
83 Currently, vLLM supports the basic multi-head attention mechanism and its variant with rotary positional embeddings.
84 If your model employs a different attention mechanism, you will need to implement a new attention layer in vLLM.
85
86 -For reference, check out our [Llama implementation](gh-file:vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out <gh-dir:vllm/model_executor/models> for more examples.
86 +For reference, check out our [Llama implementation](../../../vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out [vllm/model_executor/models](../../../vllm/model_executor/models) for more examples.
87
88 ## 3. (Optional) Implement tensor parallelism and quantization support
89
@@ -130,22 +130,22 @@ We consider 3 different scenarios:
130 2. Models that combine Mamba layers (either Mamba-1 or Mamba-2) together with attention layers.
131 3. Models that combine Mamba-like mechanisms (e.g., Linear Attention, ShortConv) together with attention layers.
132
133 -For case (1), we recommend looking at the implementation of [`MambaForCausalLM`](gh-file:vllm/model_executor/models/mamba.py) (for Mamba-1) or [`Mamba2ForCausalLM`](gh-file:vllm/model_executor/models/mamba2.py) (for Mamba-2) as a reference.
133 +For case (1), we recommend looking at the implementation of [`MambaForCausalLM`](../../../vllm/model_executor/models/mamba.py) (for Mamba-1) or [`Mamba2ForCausalLM`](../../../vllm/model_executor/models/mamba2.py) (for Mamba-2) as a reference.
134 The model should inherit protocol `IsAttentionFree` and also implement class methods `get_mamba_state_dtype_from_config` and `get_mamba_state_shape_from_config` to calculate the state shapes and data types from the config.
135 -For the mamba layers themselves, please use the [`MambaMixer`](gh-file:vllm/model_executor/layers/mamba/mamba_mixer.py) (for Mamba-1) or [`MambaMixer2`](gh-file:vllm/model_executor/layers/mamba/mamba_mixer2.py) (for Mamba-2) classes.
135 +For the mamba layers themselves, please use the [`MambaMixer`](../../../vllm/model_executor/layers/mamba/mamba_mixer.py) (for Mamba-1) or [`MambaMixer2`](../../../vllm/model_executor/layers/mamba/mamba_mixer2.py) (for Mamba-2) classes.
136 Please *do not* use the `MambaCacheManager` (deprecated in V1) or replicate any of the V0-specific code paths in the existing model implementations.
137 V0-only classes and code will be removed in the very near future.
138 -The model should also be added to the `MODELS_CONFIG_MAP` dictionary in <gh-file:vllm/model_executor/models/config.py> to ensure that the runtime defaults are optimized.
138 +The model should also be added to the `MODELS_CONFIG_MAP` dictionary in [vllm/model_executor/models/config.py](../../../vllm/model_executor/models/config.py) to ensure that the runtime defaults are optimized.
139
140 -For case (2), we recommend using as a reference the implementation of [`JambaForCausalLM`](gh-file:vllm/model_executor/models/jamba.py) (for an example of a model that uses Mamba-1 and attention together) or [`BambaForCausalLM`](gh-file:vllm/model_executor/models/bamba.py) (for an example of a model that uses Mamba-2 and attention together).
140 +For case (2), we recommend using as a reference the implementation of [`JambaForCausalLM`](../../../vllm/model_executor/models/jamba.py) (for an example of a model that uses Mamba-1 and attention together) or [`BambaForCausalLM`](../../../vllm/model_executor/models/bamba.py) (for an example of a model that uses Mamba-2 and attention together).
141 These models should follow the same instructions as case (1), but they should inherit protocol `IsHybrid` (instead of `IsAttentionFree`) and it is *not* necessary to add them to the `MODELS_CONFIG_MAP` (their runtime defaults will be inferred from the protocol).
142
143 -For case (3), we recommend looking at the implementation of [`MiniMaxText01ForCausalLM`](gh-file:vllm/model_executor/models/minimax_text_01.py) or [`Lfm2ForCausalLM`](gh-file:vllm/model_executor/models/lfm2.py) as a reference, which use custom "mamba-like" layers `MiniMaxText01LinearAttention` and `ShortConv` respectively.
143 +For case (3), we recommend looking at the implementation of [`MiniMaxText01ForCausalLM`](../../../vllm/model_executor/models/minimax_text_01.py) or [`Lfm2ForCausalLM`](../../../vllm/model_executor/models/lfm2.py) as a reference, which use custom "mamba-like" layers `MiniMaxText01LinearAttention` and `ShortConv` respectively.
144 Please follow the same guidelines as case (2) for implementing these models.
145 We use "mamba-like" to refer to layers that posses a state that is updated in-place, rather than being appended-to (like KV cache for attention).
146 For implementing new custom mamba-like layers, one should inherit from `MambaBase` and implement the methods `get_state_dtype`, `get_state_shape` to calculate the data types and state shapes at runtime, as well as `mamba_type` and `get_attn_backend`.
147 It is also necessary to implement the "attention meta-data" class which handles the meta-data that is common across all layers.
148 -Please see [`LinearAttentionMetadata`](gh-file:vllm/v1/attention/backends/linear_attn.py) or [`ShortConvAttentionMetadata`](gh-file:v1/attention/backends/short_conv_attn.py) for examples of this.
148 +Please see [`LinearAttentionMetadata`](../../../vllm/v1/attention/backends/linear_attn.py) or [`ShortConvAttentionMetadata`](../../../vllm/v1/attention/backends/short_conv_attn.py) for examples of this.
149 Finally, if one wants to support torch compile and CUDA graphs, it necessary to wrap the call to the mamba-like layer inside a custom op and register it.
150 -Please see the calls to `direct_register_custom_op` in <gh-file:vllm/model_executor/models/minimax_text_01.py> or <gh-file:vllm/model_executor/layers/mamba/short_conv.py> for examples of this.
151 -The new custom op should then be added to the list `_attention_ops` in <gh-file:vllm/config/compilation.py> to ensure that piecewise CUDA graphs works as intended.
150 +Please see the calls to `direct_register_custom_op` in [vllm/model_executor/models/minimax_text_01.py](../../../vllm/model_executor/models/minimax_text_01.py) or [vllm/model_executor/layers/mamba/short_conv.py](../../../vllm/model_executor/layers/mamba/short_conv.py) for examples of this.
151 +The new custom op should then be added to the list `_attention_ops` in [vllm/config/compilation.py](../../../vllm/config/compilation.py) to ensure that piecewise CUDA graphs works as intended.
docs/contributing/model/multimodal.md
+10 -10
@@ -507,7 +507,7 @@ return a schema of the tensors outputted by the HF processor that are related to
507 ```
508
509 !!! note
510 - Our [actual code](gh-file:vllm/model_executor/models/llava.py) additionally supports
510 + Our [actual code](../../../vllm/model_executor/models/llava.py) additionally supports
511 pre-computed image embeddings, which can be passed to be model via the `image_embeds` argument.
512
513 === "With postprocessing: Fuyu"
@@ -569,7 +569,7 @@ return a schema of the tensors outputted by the HF processor that are related to
569 ```
570
571 !!! note
572 - Our [actual code](gh-file:vllm/model_executor/models/fuyu.py) has special handling
572 + Our [actual code](../../../vllm/model_executor/models/fuyu.py) has special handling
573 for text-only inputs to prevent unnecessary warnings from HF processor.
574
575 !!! note
@@ -828,8 +828,8 @@ Some HF processors directly insert feature tokens without replacing anything in
828
829 Examples:
830
831 -- BLIP-2 (insert at start of prompt): <gh-file:vllm/model_executor/models/blip2.py>
832 -- Molmo (insert after `<|endoftext|>` token): <gh-file:vllm/model_executor/models/molmo.py>
831 +- BLIP-2 (insert at start of prompt): [vllm/model_executor/models/blip2.py](../../../vllm/model_executor/models/blip2.py)
832 +- Molmo (insert after `<|endoftext|>` token): [vllm/model_executor/models/molmo.py](../../../vllm/model_executor/models/molmo.py)
833
834 ### Handling prompt updates unrelated to multi-modal data
835
@@ -837,9 +837,9 @@ Examples:
837
838 Examples:
839
840 -- Chameleon (appends `sep_token`): <gh-file:vllm/model_executor/models/chameleon.py>
841 -- Fuyu (appends `boa_token`): <gh-file:vllm/model_executor/models/fuyu.py>
842 -- Molmo (applies chat template which is not defined elsewhere): <gh-file:vllm/model_executor/models/molmo.py>
840 +- Chameleon (appends `sep_token`): [vllm/model_executor/models/chameleon.py](../../../vllm/model_executor/models/chameleon.py)
841 +- Fuyu (appends `boa_token`): [vllm/model_executor/models/fuyu.py](../../../vllm/model_executor/models/fuyu.py)
842 +- Molmo (applies chat template which is not defined elsewhere): [vllm/model_executor/models/molmo.py](../../../vllm/model_executor/models/molmo.py)
843
844 ### Custom HF processor
845
@@ -847,6 +847,6 @@ Some models don't define an HF processor class on HF Hub. In that case, you can
847
848 Examples:
849
850 -- DeepSeek-VL2: <gh-file:vllm/model_executor/models/deepseek_vl2.py>
851 -- InternVL: <gh-file:vllm/model_executor/models/internvl.py>
852 -- Qwen-VL: <gh-file:vllm/model_executor/models/qwen_vl.py>
850 +- DeepSeek-VL2: [vllm/model_executor/models/deepseek_vl2.py](../../../vllm/model_executor/models/deepseek_vl2.py)
851 +- InternVL: [vllm/model_executor/models/internvl.py](../../../vllm/model_executor/models/internvl.py)
852 +- Qwen-VL: [vllm/model_executor/models/qwen_vl.py](../../../vllm/model_executor/models/qwen_vl.py)
docs/contributing/model/registration.md
+2 -2
@@ -11,8 +11,8 @@ This page provides detailed instructions on how to do so.
11 To add a model directly to the vLLM library, start by forking our [GitHub repository](https://github.com/vllm-project/vllm) and then [build it from source][build-from-source].
12 This gives you the ability to modify the codebase and test your model.
13
14 -After you have implemented your model (see [tutorial](basic.md)), put it into the <gh-dir:vllm/model_executor/models> directory.
15 -Then, add your model class to `_VLLM_MODELS` in <gh-file:vllm/model_executor/models/registry.py> so that it is automatically registered upon importing vLLM.
14 +After you have implemented your model (see [tutorial](basic.md)), put it into the [vllm/model_executor/models](../../../vllm/model_executor/models) directory.
15 +Then, add your model class to `_VLLM_MODELS` in [vllm/model_executor/models/registry.py](../../../vllm/model_executor/models/registry.py) so that it is automatically registered upon importing vLLM.
16 Finally, update our [list of supported models](../../models/supported_models.md) to promote your model!
17
18 !!! important
docs/contributing/model/tests.md
+7 -7
@@ -9,7 +9,7 @@ Without them, the CI for your PR will fail.
9
10 ### Model loading
11
12 -Include an example HuggingFace repository for your model in <gh-file:tests/models/registry.py>.
12 +Include an example HuggingFace repository for your model in [tests/models/registry.py](../../../tests/models/registry.py).
13 This enables a unit test that loads dummy weights to ensure that the model can be initialized in vLLM.
14
15 !!! important
@@ -26,18 +26,18 @@ Passing these tests provides more confidence that your implementation is correct
26
27 ### Model correctness
28
29 -These tests compare the model outputs of vLLM against [HF Transformers](https://github.com/huggingface/transformers). You can add new tests under the subdirectories of <gh-dir:tests/models>.
29 +These tests compare the model outputs of vLLM against [HF Transformers](https://github.com/huggingface/transformers). You can add new tests under the subdirectories of [tests/models](../../../tests/models).
30
31 #### Generative models
32
33 -For [generative models](../../models/generative_models.md), there are two levels of correctness tests, as defined in <gh-file:tests/models/utils.py>:
33 +For [generative models](../../models/generative_models.md), there are two levels of correctness tests, as defined in [tests/models/utils.py](../../../tests/models/utils.py):
34
35 - Exact correctness (`check_outputs_equal`): The text outputted by vLLM should exactly match the text outputted by HF.
36 - Logprobs similarity (`check_logprobs_close`): The logprobs outputted by vLLM should be in the top-k logprobs outputted by HF, and vice versa.
37
38 #### Pooling models
39
40 -For [pooling models](../../models/pooling_models.md), we simply check the cosine similarity, as defined in <gh-file:tests/models/utils.py>.
40 +For [pooling models](../../models/pooling_models.md), we simply check the cosine similarity, as defined in [tests/models/utils.py](../../../tests/models/utils.py).
41
42 [](){ #mm-processing-tests }
43
@@ -45,7 +45,7 @@ For [pooling models](../../models/pooling_models.md), we simply check the cosine
45
46 #### Common tests
47
48 -Adding your model to <gh-file:tests/models/multimodal/processing/test_common.py> verifies that the following input combinations result in the same outputs:
48 +Adding your model to [tests/models/multimodal/processing/test_common.py](../../../tests/models/multimodal/processing/test_common.py) verifies that the following input combinations result in the same outputs:
49
50 - Text + multi-modal data
51 - Tokens + multi-modal data
@@ -54,6 +54,6 @@ Adding your model to <gh-file:tests/models/multimodal/processing/test_common.py>
54
55 #### Model-specific tests
56
57 -You can add a new file under <gh-dir:tests/models/multimodal/processing> to run tests that only apply to your model.
57 +You can add a new file under [tests/models/multimodal/processing](../../../tests/models/multimodal/processing) to run tests that only apply to your model.
58
59 -For example, if the HF processor for your model accepts user-specified keyword arguments, you can verify that the keyword arguments are being applied correctly, such as in <gh-file:tests/models/multimodal/processing/test_phi3v.py>.
59 +For example, if the HF processor for your model accepts user-specified keyword arguments, you can verify that the keyword arguments are being applied correctly, such as in [tests/models/multimodal/processing/test_phi3v.py](../../../tests/models/multimodal/processing/test_phi3v.py).
docs/contributing/model/transcription.md
+4 -4
@@ -248,9 +248,9 @@ No extra registration is required beyond having your model class available via t
248
249 ## Examples in-tree
250
251 -- Whisper encoder–decoder (audio-only): <gh-file:vllm/model_executor/models/whisper.py>
252 -- Voxtral decoder-only (audio embeddings + LLM): <gh-file:vllm/model_executor/models/voxtral.py>
253 -- Gemma3n decoder-only with fixed instruction prompt: <gh-file:vllm/model_executor/models/gemma3n_mm.py>
251 +- Whisper encoder–decoder (audio-only): [vllm/model_executor/models/whisper.py](../../../vllm/model_executor/models/whisper.py)
252 +- Voxtral decoder-only (audio embeddings + LLM): [vllm/model_executor/models/voxtral.py](../../../vllm/model_executor/models/voxtral.py)
253 +- Gemma3n decoder-only with fixed instruction prompt: [vllm/model_executor/models/gemma3n_mm.py](../../../vllm/model_executor/models/gemma3n_mm.py)
254
255 ## Test with the API
256
@@ -278,7 +278,7 @@ Once your model implements `SupportsTranscription`, you can test the endpoints (
278 http://localhost:8000/v1/audio/translations
279 ```
280
281 -Or check out more examples in <gh-file:examples/online_serving>.
281 +Or check out more examples in [examples/online_serving](../../../examples/online_serving).
282
283 !!! note
284 - If your model handles chunking internally (e.g., via its processor or encoder), set `min_energy_split_window_size=None` in the returned `SpeechToTextConfig` to disable server-side chunking.
docs/contributing/profiling.md
+1 -1
@@ -33,7 +33,7 @@ Traces can be visualized using <https://ui.perfetto.dev/>.
33
34 #### Offline Inference
35
36 -Refer to <gh-file:examples/offline_inference/simple_profiling.py> for an example.
36 +Refer to [examples/offline_inference/simple_profiling.py](../../examples/offline_inference/simple_profiling.py) for an example.
37
38 #### OpenAI Server
39
docs/deployment/docker.md
+2 -2
@@ -37,7 +37,7 @@ You can add any other [engine-args](../configuration/engine_args.md) you need af
37 memory to share data between processes under the hood, particularly for tensor parallel inference.
38
39 !!! note
40 - Optional dependencies are not included in order to avoid licensing issues (e.g. <gh-issue:8030>).
40 + Optional dependencies are not included in order to avoid licensing issues (e.g. <https://github.com/vllm-project/vllm/issues/8030>).
41
42 If you need to use those dependencies (having accepted the license terms),
43 create a custom Dockerfile on top of the base image with an extra layer that installs them:
@@ -66,7 +66,7 @@ You can add any other [engine-args](../configuration/engine_args.md) you need af
66
67 ## Building vLLM's Docker Image from Source
68
69 -You can build and run vLLM from source via the provided <gh-file:docker/Dockerfile>. To build vLLM:
69 +You can build and run vLLM from source via the provided [docker/Dockerfile](../../docker/Dockerfile). To build vLLM:
70
71 ```bash
72 # optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2
docs/deployment/frameworks/anyscale.md
+1 -1
@@ -5,7 +5,7 @@
5 [Anyscale](https://www.anyscale.com) is a managed, multi-cloud platform developed by the creators of Ray.
6
7 Anyscale automates the entire lifecycle of Ray clusters in your AWS, GCP, or Azure account, delivering the flexibility of open-source Ray
8 -without the operational overhead of maintaining Kubernetes control planes, configuring autoscalers, managing observability stacks, or manually managing head and worker nodes with helper scripts like <gh-file:examples/online_serving/run_cluster.sh>.
8 +without the operational overhead of maintaining Kubernetes control planes, configuring autoscalers, managing observability stacks, or manually managing head and worker nodes with helper scripts like [examples/online_serving/run_cluster.sh](../../../examples/online_serving/run_cluster.sh).
9
10 When serving large language models with vLLM, Anyscale can rapidly provision [production-ready HTTPS endpoints](https://docs.anyscale.com/examples/deploy-ray-serve-llms) or [fault-tolerant batch inference jobs](https://docs.anyscale.com/examples/ray-data-llm).
11
docs/deployment/frameworks/retrieval_augmented_generation.md
+2 -2
@@ -36,7 +36,7 @@ pip install -U vllm \
36 vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001
37 ```
38
39 -1. Use the script: <gh-file:examples/online_serving/retrieval_augmented_generation_with_langchain.py>
39 +1. Use the script: [examples/online_serving/retrieval_augmented_generation_with_langchain.py](../../../examples/online_serving/retrieval_augmented_generation_with_langchain.py)
40
41 1. Run the script
42
@@ -74,7 +74,7 @@ pip install vllm \
74 vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001
75 ```
76
77 -1. Use the script: <gh-file:examples/online_serving/retrieval_augmented_generation_with_llamaindex.py>
77 +1. Use the script: [examples/online_serving/retrieval_augmented_generation_with_llamaindex.py](../../../examples/online_serving/retrieval_augmented_generation_with_llamaindex.py)
78
79 1. Run the script:
80
docs/deployment/frameworks/streamlit.md
+1 -1
@@ -20,7 +20,7 @@ pip install vllm streamlit openai
20 vllm serve Qwen/Qwen1.5-0.5B-Chat
21 ```
22
23 -1. Use the script: <gh-file:examples/online_serving/streamlit_openai_chatbot_webserver.py>
23 +1. Use the script: [examples/online_serving/streamlit_openai_chatbot_webserver.py](../../../examples/online_serving/streamlit_openai_chatbot_webserver.py)
24
25 1. Start the streamlit web UI and start to chat:
26
docs/design/arch_overview.md
+6 -6
@@ -49,7 +49,7 @@ Here is a sample of `LLM` class usage:
49
50 More API details can be found in the [Offline Inference](#offline-inference-api) section of the API docs.
51
52 -The code for the `LLM` class can be found in <gh-file:vllm/entrypoints/llm.py>.
52 +The code for the `LLM` class can be found in [vllm/entrypoints/llm.py](../../vllm/entrypoints/llm.py).
53
54 ### OpenAI-Compatible API Server
55
@@ -60,7 +60,7 @@ This server can be started using the `vllm serve` command.
60 vllm serve <model>
61 ```
62
63 -The code for the `vllm` CLI can be found in <gh-file:vllm/entrypoints/cli/main.py>.
63 +The code for the `vllm` CLI can be found in [vllm/entrypoints/cli/main.py](../../vllm/entrypoints/cli/main.py).
64
65 Sometimes you may see the API server entrypoint used directly instead of via the
66 `vllm` CLI command. For example:
@@ -74,7 +74,7 @@ python -m vllm.entrypoints.openai.api_server --model <model>
74 `python -m vllm.entrypoints.openai.api_server` is deprecated
75 and may become unsupported in a future release.
76
77 -That code can be found in <gh-file:vllm/entrypoints/openai/api_server.py>.
77 +That code can be found in [vllm/entrypoints/openai/api_server.py](../../vllm/entrypoints/openai/api_server.py).
78
79 More details on the API server can be found in the [OpenAI-Compatible Server](../serving/openai_compatible_server.md) document.
80
@@ -101,7 +101,7 @@ processing.
101 - **Output Processing**: Processes the outputs generated by the model, decoding the
102 token IDs from a language model into human-readable text.
103
104 -The code for `LLMEngine` can be found in <gh-file:vllm/engine/llm_engine.py>.
104 +The code for `LLMEngine` can be found in [vllm/engine/llm_engine.py](../../vllm/engine/llm_engine.py).
105
106 ### AsyncLLMEngine
107
@@ -111,9 +111,9 @@ incoming requests. The `AsyncLLMEngine` is designed for online serving, where it
111 can handle multiple concurrent requests and stream outputs to clients.
112
113 The OpenAI-compatible API server uses the `AsyncLLMEngine`. There is also a demo
114 -API server that serves as a simpler example in <gh-file:vllm/entrypoints/api_server.py>.
114 +API server that serves as a simpler example in [vllm/entrypoints/api_server.py](../../vllm/entrypoints/api_server.py).
115
116 -The code for `AsyncLLMEngine` can be found in <gh-file:vllm/engine/async_llm_engine.py>.
116 +The code for `AsyncLLMEngine` can be found in [vllm/engine/async_llm_engine.py](../../vllm/engine/async_llm_engine.py).
117
118 ## Worker
119
docs/design/cuda_graphs.md
+2 -2
@@ -17,7 +17,7 @@ In this document we will discuss the:
17 In this document, we refer to pure decode (`max_query_len=1`) or speculative decode (`max_query_len =1+num_spec_tokens`) as **uniform decode** batches, and the opposite would be **non-uniform** batches (i.e., prefill or mixed prefill-decode batches).
18
19 !!! note
20 - The following contents are mostly based on the last commit of <gh-pr:20059>.
20 + The following contents are mostly based on the last commit of <https://github.com/vllm-project/vllm/pull/20059>.
21
22 ## Motivation
23
@@ -92,7 +92,7 @@ where `num_tokens` can be the padded token length, and `uniform_decode` is deter
92 The goal of this structure is to uniquely identify a (padded) batch with minimal possible items corresponding to a CUDA Graphs item. We are safe to exclude items like `uniform_query_len` because it is a constant at runtime for a certain setup currently. For example, it should be either `1` for a commonly pure decode or `1+num_spec_tokens` for a validation phase of speculative decode.
93
94 !!! note
95 - The prototype of `BatchDescriptor` may be extended for more general situations in the future, e.g., include more items, like `uniform_query_len` to support multiple different uniform decode lengths settings (<gh-pr:23679>), or other modifications needed to support CUDA Graphs for models whose inputs are not necessarily token length aware (for example, some multi-modal inputs).
95 + The prototype of `BatchDescriptor` may be extended for more general situations in the future, e.g., include more items, like `uniform_query_len` to support multiple different uniform decode lengths settings (<https://github.com/vllm-project/vllm/pull/23679>), or other modifications needed to support CUDA Graphs for models whose inputs are not necessarily token length aware (for example, some multi-modal inputs).
96
97 ### `CudagraphDispatcher`
98
docs/design/fused_moe_modular_kernel.md
+8 -8
@@ -2,7 +2,7 @@
2
3 ## Introduction
4
5 -FusedMoEModularKernel is implemented [here](gh-file:/vllm/model_executor/layers/fused_moe/modular_kernel.py)
5 +FusedMoEModularKernel is implemented [here](../..//vllm/model_executor/layers/fused_moe/modular_kernel.py)
6
7 Based on the format of the input activations, FusedMoE implementations are broadly classified into 2 types.
8
@@ -44,7 +44,7 @@ FusedMoEModularKernel splits the FusedMoE operation into 3 parts,
44
45 The TopK Weight Application and Reduction components happen right after the Unpermute operation and before the All2All Combine. Note that the `FusedMoEPermuteExpertsUnpermute` is responsible for the Unpermute and `FusedMoEPrepareAndFinalize` is responsible for the All2All Combine. There is value in doing the TopK Weight Application and Reduction in the `FusedMoEPermuteExpertsUnpermute`. But some implementations choose to do it `FusedMoEPrepareAndFinalize`. In order to enable this flexibility, we have a TopKWeightAndReduce abstract class.
46
47 -Please find the implementations of TopKWeightAndReduce [here](gh-file:vllm/model_executor/layers/fused_moe/topk_weight_and_reduce.py).
47 +Please find the implementations of TopKWeightAndReduce [here](../../vllm/model_executor/layers/fused_moe/topk_weight_and_reduce.py).
48
49 `FusedMoEPrepareAndFinalize::finalize()` method accepts a `TopKWeightAndReduce` argument that is invoked inside the method.
50 The `FusedMoEModularKernel` acts as a bridge between the `FusedMoEPermuteExpertsUnpermute` and `FusedMoEPerpareAndFinalize` implementations to determine where the TopK Weight Application and Reduction happens.
@@ -138,7 +138,7 @@ Typically a FusedMoEPrepareAndFinalize type is backed by an All2All Dispatch & C
138
139 #### Step 1: Add an All2All manager
140
141 -The purpose of the All2All Manager is to set up the All2All kernel implementations. The `FusedMoEPrepareAndFinalize` implementations typically fetch a kernel-implementation "handle" from the All2All Manager to invoke the Dispatch and Combine functions. Please look at the All2All Manager implementations [here](gh-file:vllm/distributed/device_communicators/all2all.py).
141 +The purpose of the All2All Manager is to set up the All2All kernel implementations. The `FusedMoEPrepareAndFinalize` implementations typically fetch a kernel-implementation "handle" from the All2All Manager to invoke the Dispatch and Combine functions. Please look at the All2All Manager implementations [here](../../vllm/distributed/device_communicators/all2all.py).
142
143 #### Step 2: Add a FusedMoEPrepareAndFinalize Type
144
@@ -213,29 +213,29 @@ Please take a look at [init_prepare_finalize](https://github.com/vllm-project/vl
213
214 ### How To Unit Test
215
216 -We have `FusedMoEModularKernel` unit tests at [test_modular_kernel_combinations.py](gh-file:tests/kernels/moe/test_modular_kernel_combinations.py).
216 +We have `FusedMoEModularKernel` unit tests at [test_modular_kernel_combinations.py](../../tests/kernels/moe/test_modular_kernel_combinations.py).
217
218 The unit test iterates through all combinations of `FusedMoEPrepareAndFinalize` and `FusedMoEPremuteExpertsUnpermute` types and if they are
219 compatible, runs some correctness tests.
220 If you are adding some `FusedMoEPrepareAndFinalize` / `FusedMoEPermuteExpertsUnpermute` implementations,
221
222 -1. Add the implementation type to `MK_ALL_PREPARE_FINALIZE_TYPES` and `MK_FUSED_EXPERT_TYPES` in [mk_objects.py](gh-file:tests/kernels/moe/modular_kernel_tools/mk_objects.py) respectively.
222 +1. Add the implementation type to `MK_ALL_PREPARE_FINALIZE_TYPES` and `MK_FUSED_EXPERT_TYPES` in [mk_objects.py](../../tests/kernels/moe/modular_kernel_tools/mk_objects.py) respectively.
223 2. Update `Config::is_batched_prepare_finalize()`, `Config::is_batched_fused_experts()`, `Config::is_standard_fused_experts()`,
224 `Config::is_fe_16bit_supported()`, `Config::is_fe_fp8_supported()`, `Config::is_fe_block_fp8_supported()`,
225 -`Config::is_fe_supports_chunking()` methods in [/tests/kernels/moe/modular_kernel_tools/common.py](gh-file:tests/kernels/moe/modular_kernel_tools/common.py)
225 +`Config::is_fe_supports_chunking()` methods in [/tests/kernels/moe/modular_kernel_tools/common.py](../../tests/kernels/moe/modular_kernel_tools/common.py)
226
227 Doing this will add the new implementation to the test suite.
228
229 ### How To Check `FusedMoEPrepareAndFinalize` & `FusedMoEPermuteExpertsUnpermute` Compatibility
230
231 -The unit test file [test_modular_kernel_combinations.py](gh-file:tests/kernels/moe/test_modular_kernel_combinations.py) can also be executed as a standalone script.
231 +The unit test file [test_modular_kernel_combinations.py](../../tests/kernels/moe/test_modular_kernel_combinations.py) can also be executed as a standalone script.
232 Example: `python3 -m tests.kernels.moe.test_modular_kernel_combinations --pf-type PplxPrepareAndFinalize --experts-type BatchedTritonExperts`
233 As a side effect, this script can be used to test `FusedMoEPrepareAndFinalize` & `FusedMoEPermuteExpertsUnpermute` compatibility. When invoked
234 with incompatible types, the script will error.
235
236 ### How To Profile
237
238 -Please take a look at [profile_modular_kernel.py](gh-file:tests/kernels/moe/modular_kernel_tools/profile_modular_kernel.py)
238 +Please take a look at [profile_modular_kernel.py](../../tests/kernels/moe/modular_kernel_tools/profile_modular_kernel.py)
239 The script can be used to generate Torch traces for a single `FusedMoEModularKernel::forward()` call for any compatible
240 `FusedMoEPrepareAndFinalize` and `FusedMoEPermuteExpertsUnpermute` types.
241 Example: `python3 -m tests.kernels.moe.modular_kernel_tools.profile_modular_kernel --pf-type PplxPrepareAndFinalize --experts-type BatchedTritonExperts`
docs/design/io_processor_plugins.md
+3 -3
@@ -6,7 +6,7 @@ When performing an inference with IO Processor plugins, the prompt type is defin
6
7 ## Writing an IO Processor Plugin
8
9 -IO Processor plugins implement the `IOProcessor` interface (<gh-file:vllm/plugins/io_processors/interface.py>):
9 +IO Processor plugins implement the [`IOProcessor`][vllm.plugins.io_processors.interface.IOProcessor] interface:
10
11 ```python
12 IOProcessorInput = TypeVar("IOProcessorInput")
@@ -67,9 +67,9 @@ The `parse_request` method is used for validating the user prompt and converting
67 The `pre_process*` methods take the validated plugin input to generate vLLM's model prompts for regular inference.
68 The `post_process*` methods take `PoolingRequestOutput` objects as input and generate a custom plugin output.
69
70 -The `output_to_response` method is used only for online serving and converts the plugin output to the `IOProcessorResponse` type that is then returned by the API Server. The implementation of the `/io_processor_pooling` serving endpoint is available here <gh-file:vllm/entrypoints/openai/serving_pooling_with_io_plugin.py>.
70 +The `output_to_response` method is used only for online serving and converts the plugin output to the `IOProcessorResponse` type that is then returned by the API Server. The implementation of the `/pooling` serving endpoint is available here [vllm/entrypoints/openai/serving_pooling.py](../../vllm/entrypoints/openai/serving_pooling.py).
71
72 -An example implementation of a plugin that enables generating geotiff images with the PrithviGeospatialMAE model is available [here](https://github.com/christian-pinto/prithvi_io_processor_plugin). Please, also refer to our online (<gh-file:examples/online_serving/prithvi_geospatial_mae.py>) and offline (<gh-file:examples/offline_inference/prithvi_geospatial_mae_io_processor.py>) inference examples.
72 +An example implementation of a plugin that enables generating geotiff images with the PrithviGeospatialMAE model is available [here](https://github.com/christian-pinto/prithvi_io_processor_plugin). Please, also refer to our online ([examples/online_serving/prithvi_geospatial_mae.py](../../examples/online_serving/prithvi_geospatial_mae.py)) and offline ([examples/offline_inference/prithvi_geospatial_mae_io_processor.py](../../examples/offline_inference/prithvi_geospatial_mae_io_processor.py)) inference examples.
73
74 ## Using an IO Processor plugin
75
docs/design/metrics.md
+38 -38
@@ -80,13 +80,13 @@ The subset of metrics exposed in the Grafana dashboard gives us an indication of
80 - `vllm:request_decode_time_seconds` - Requests decode time.
81 - `vllm:request_max_num_generation_tokens` - Max generation tokens in a sequence group.
82
83 -See [the PR which added this Dashboard](gh-pr:2316) for interesting and useful background on the choices made here.
83 +See [the PR which added this Dashboard](https://github.com/vllm-project/vllm/pull/2316) for interesting and useful background on the choices made here.
84
85 ### Prometheus Client Library
86
87 -Prometheus support was initially added [using the aioprometheus library](gh-pr:1890), but a switch was made quickly to [prometheus_client](gh-pr:2730). The rationale is discussed in both linked PRs.
87 +Prometheus support was initially added [using the aioprometheus library](https://github.com/vllm-project/vllm/pull/1890), but a switch was made quickly to [prometheus_client](https://github.com/vllm-project/vllm/pull/2730). The rationale is discussed in both linked PRs.
88
89 -With the switch to `aioprometheus`, we lost a `MetricsMiddleware` to track HTTP metrics, but this was reinstated [using prometheus_fastapi_instrumentator](gh-pr:15657):
89 +With the switch to `aioprometheus`, we lost a `MetricsMiddleware` to track HTTP metrics, but this was reinstated [using prometheus_fastapi_instrumentator](https://github.com/vllm-project/vllm/pull/15657):
90
91 ```bash
92 $ curl http://0.0.0.0:8000/metrics 2>/dev/null | grep -P '^http_(?!.*(_bucket|_created|_sum)).*'
@@ -99,7 +99,7 @@ http_request_duration_seconds_count{handler="/v1/completions",method="POST"} 201
99
100 ### Multi-process Mode
101
102 -In v0, metrics are collected in the engine core process and we use multiprocess mode to make them available in the API server process. See <gh-pr:7279>.
102 +In v0, metrics are collected in the engine core process and we use multiprocess mode to make them available in the API server process. See <https://github.com/vllm-project/vllm/pull/7279>.
103
104 ### Built in Python/Process Metrics
105
@@ -125,32 +125,32 @@ vLLM instance.
125
126 For background, these are some of the relevant PRs which added the v0 metrics:
127
128 -- <gh-pr:1890>
129 -- <gh-pr:2316>
130 -- <gh-pr:2730>
131 -- <gh-pr:4464>
132 -- <gh-pr:7279>
128 +- <https://github.com/vllm-project/vllm/pull/1890>
129 +- <https://github.com/vllm-project/vllm/pull/2316>
130 +- <https://github.com/vllm-project/vllm/pull/2730>
131 +- <https://github.com/vllm-project/vllm/pull/4464>
132 +- <https://github.com/vllm-project/vllm/pull/7279>
133
134 -Also note the ["Even Better Observability"](gh-issue:3616) feature where e.g. [a detailed roadmap was laid out](gh-issue:3616#issuecomment-2030858781).
134 +Also note the ["Even Better Observability"](https://github.com/vllm-project/vllm/issues/3616) feature where e.g. [a detailed roadmap was laid out](https://github.com/vllm-project/vllm/issues/3616#issuecomment-2030858781).
135
136 ## v1 Design
137
138 ### v1 PRs
139
140 For background, here are the relevant v1 PRs relating to the v1
141 -metrics issue <gh-issue:10582>:
142 -
143 -- <gh-pr:11962>
144 -- <gh-pr:11973>
145 -- <gh-pr:10907>
146 -- <gh-pr:12416>
147 -- <gh-pr:12478>
148 -- <gh-pr:12516>
149 -- <gh-pr:12530>
150 -- <gh-pr:12561>
151 -- <gh-pr:12579>
152 -- <gh-pr:12592>
153 -- <gh-pr:12644>
141 +metrics issue <https://github.com/vllm-project/vllm/issues/10582>:
142 +
143 +- <https://github.com/vllm-project/vllm/pull/11962>
144 +- <https://github.com/vllm-project/vllm/pull/11973>
145 +- <https://github.com/vllm-project/vllm/pull/10907>
146 +- <https://github.com/vllm-project/vllm/pull/12416>
147 +- <https://github.com/vllm-project/vllm/pull/12478>
148 +- <https://github.com/vllm-project/vllm/pull/12516>
149 +- <https://github.com/vllm-project/vllm/pull/12530>
150 +- <https://github.com/vllm-project/vllm/pull/12561>
151 +- <https://github.com/vllm-project/vllm/pull/12579>
152 +- <https://github.com/vllm-project/vllm/pull/12592>
153 +- <https://github.com/vllm-project/vllm/pull/12644>
154
155 ### Metrics Collection
156
@@ -369,7 +369,7 @@ vllm:cache_config_info{block_size="16",cache_dtype="auto",calculate_kv_scales="F
369
370 However, `prometheus_client` has
371 [never supported Info metrics in multiprocessing mode](https://github.com/prometheus/client_python/pull/300) -
372 -for [unclear reasons](gh-pr:7279#discussion_r1710417152). We
372 +for [unclear reasons](https://github.com/vllm-project/vllm/pull/7279#discussion_r1710417152). We
373 simply use a `Gauge` metric set to 1 and
374 `multiprocess_mode="mostrecent"` instead.
375
@@ -394,7 +394,7 @@ distinguish between per-adapter counts. This should be revisited.
394 Note that `multiprocess_mode="livemostrecent"` is used - the most
395 recent metric is used, but only from currently running processes.
396
397 -This was added in <gh-pr:9477> and there is
397 +This was added in <https://github.com/vllm-project/vllm/pull/9477> and there is
398 [at least one known user](https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/54).
399 If we revisit this design and deprecate the old metric, we should reduce
400 the need for a significant deprecation period by making the change in
@@ -402,7 +402,7 @@ v0 also and asking this project to move to the new metric.
402
403 ### Prefix Cache metrics
404
405 -The discussion in <gh-issue:10582> about adding prefix cache metrics yielded
405 +The discussion in <https://github.com/vllm-project/vllm/issues/10582> about adding prefix cache metrics yielded
406 some interesting points which may be relevant to how we approach
407 future metrics.
408
@@ -439,8 +439,8 @@ suddenly (from their perspective) when it is removed, even if there is
439 an equivalent metric for them to use.
440
441 As an example, see how `vllm:avg_prompt_throughput_toks_per_s` was
442 -[deprecated](gh-pr:2764) (with a comment in the code),
443 -[removed](gh-pr:12383), and then [noticed by a user](gh-issue:13218).
442 +[deprecated](https://github.com/vllm-project/vllm/pull/2764) (with a comment in the code),
443 +[removed](https://github.com/vllm-project/vllm/pull/12383), and then [noticed by a user](https://github.com/vllm-project/vllm/issues/13218).
444
445 In general:
446
@@ -460,20 +460,20 @@ the project-wide deprecation policy.
460
461 ### Unimplemented - `vllm:tokens_total`
462
463 -Added by <gh-pr:4464>, but apparently never implemented. This can just be
463 +Added by <https://github.com/vllm-project/vllm/pull/4464>, but apparently never implemented. This can just be
464 removed.
465
466 ### Duplicated - Queue Time
467
468 The `vllm:time_in_queue_requests` Histogram metric was added by
469 -<gh-pr:9659> and its calculation is:
469 +<https://github.com/vllm-project/vllm/pull/9659> and its calculation is:
470
471 ```python
472 self.metrics.first_scheduled_time = now
473 self.metrics.time_in_queue = now - self.metrics.arrival_time
474 ```
475
476 -Two weeks later, <gh-pr:4464> added `vllm:request_queue_time_seconds` leaving
476 +Two weeks later, <https://github.com/vllm-project/vllm/pull/4464> added `vllm:request_queue_time_seconds` leaving
477 us with:
478
479 ```python
@@ -513,7 +513,7 @@ cache to complete other requests), we swap kv cache blocks out to CPU
513 memory. This is also known as "KV cache offloading" and is configured
514 with `--swap-space` and `--preemption-mode`.
515
516 -In v0, [vLLM has long supported beam search](gh-issue:6226). The
516 +In v0, [vLLM has long supported beam search](https://github.com/vllm-project/vllm/issues/6226). The
517 SequenceGroup encapsulated the idea of N Sequences which
518 all shared the same prompt kv blocks. This enabled KV cache block
519 sharing between requests, and copy-on-write to do branching. CPU
@@ -526,7 +526,7 @@ and the part of the prompt that was evicted can be recomputed.
526
527 SequenceGroup was removed in V1, although a replacement will be
528 required for "parallel sampling" (`n>1`).
529 -[Beam search was moved out of the core (in V0)](gh-issue:8306). There was a
529 +[Beam search was moved out of the core (in V0)](https://github.com/vllm-project/vllm/issues/8306). There was a
530 lot of complex code for a very uncommon feature.
531
532 In V1, with prefix caching being better (zero over head) and therefore
@@ -541,7 +541,7 @@ Some v0 metrics are only relevant in the context of "parallel
541 sampling". This is where the `n` parameter in a request is used to
542 request multiple completions from the same prompt.
543
544 -As part of adding parallel sampling support in <gh-pr:10980>, we should
544 +As part of adding parallel sampling support in <https://github.com/vllm-project/vllm/pull/10980>, we should
545 also add these metrics.
546
547 - `vllm:request_params_n` (Histogram)
@@ -566,7 +566,7 @@ model and then validate those tokens with the larger model.
566 - `vllm:spec_decode_num_draft_tokens_total` (Counter)
567 - `vllm:spec_decode_num_emitted_tokens_total` (Counter)
568
569 -There is a PR under review (<gh-pr:12193>) to add "prompt lookup (ngram)"
569 +There is a PR under review (<https://github.com/vllm-project/vllm/pull/12193>) to add "prompt lookup (ngram)"
570 speculative decoding to v1. Other techniques will follow. We should
571 revisit the v0 metrics in this context.
572
@@ -587,7 +587,7 @@ see:
587 - [Standardizing Large Model Server Metrics in Kubernetes](https://docs.google.com/document/d/1SpSp1E6moa4HSrJnS4x3NpLuj88sMXr2tbofKlzTZpk)
588 - [Benchmarking LLM Workloads for Performance Evaluation and Autoscaling in Kubernetes](https://docs.google.com/document/d/1k4Q4X14hW4vftElIuYGDu5KDe2LtV1XammoG-Xi3bbQ)
589 - [Inference Perf](https://github.com/kubernetes-sigs/wg-serving/tree/main/proposals/013-inference-perf)
590 -- <gh-issue:5041> and <gh-pr:12726>.
590 +- <https://github.com/vllm-project/vllm/issues/5041> and <https://github.com/vllm-project/vllm/pull/12726>.
591
592 This is a non-trivial topic. Consider this comment from Rob:
593
@@ -654,7 +654,7 @@ fall under the more general heading of "Observability".
654
655 v0 has support for OpenTelemetry tracing:
656
657 -- Added by <gh-pr:4687>
657 +- Added by <https://github.com/vllm-project/vllm/pull/4687>
658 - Configured with `--oltp-traces-endpoint` and `--collect-detailed-traces`
659 - [OpenTelemetry blog post](https://opentelemetry.io/blog/2024/llm-observability/)
660 - [User-facing docs](../examples/online_serving/opentelemetry.md)
@@ -685,7 +685,7 @@ documentation for this option states:
685 > use of possibly costly and or blocking operations and hence might
686 > have a performance impact.
687
688 -The metrics were added by <gh-pr:7089> and who up in an OpenTelemetry trace
688 +The metrics were added by <https://github.com/vllm-project/vllm/pull/7089> and who up in an OpenTelemetry trace
689 as:
690
691 ```text
docs/design/mm_processing.md
+1 -1
@@ -60,7 +60,7 @@ With the help of dummy text and automatic prompt updating, our multi-modal proce
60
61 ## Processor Output Caching
62
63 -Some HF processors, such as the one for Qwen2-VL, are [very slow](gh-issue:9238). To alleviate this problem, we cache the multi-modal outputs of HF processor to avoid processing the same multi-modal input (e.g. image) again.
63 +Some HF processors, such as the one for Qwen2-VL, are [very slow](https://github.com/vllm-project/vllm/issues/9238). To alleviate this problem, we cache the multi-modal outputs of HF processor to avoid processing the same multi-modal input (e.g. image) again.
64
65 When new data is passed in, we first check which items are in the cache, and which ones are missing. The missing items are passed into the HF processor in a single batch and cached, before being merged with the existing items in the cache.
66
docs/design/multiprocessing.md
+1 -1
@@ -82,7 +82,7 @@ There are other miscellaneous places hard-coding the use of `spawn`:
82
83 Related PRs:
84
85 -- <gh-pr:8823>
85 +- <https://github.com/vllm-project/vllm/pull/8823>
86
87 ## Prior State in v1
88
docs/design/torch_compile.md
+2 -2
@@ -19,8 +19,8 @@ vLLM will take all the available factors into consideration, and decide a direct
19
20 The factors considered include:
21
22 -- All the related configs (see the `compute_hash` functions in their respective configs in the [config folder](gh-file:vllm/config))
23 -- PyTorch configs (see the `compute_hash` functions in the [compiler_interface.py](gh-file:vllm/compilation/compiler_interface.py))
22 +- All the related configs (see the `compute_hash` functions in their respective configs in the [config folder](../../vllm/config))
23 +- PyTorch configs (see the `compute_hash` functions in the [compiler_interface.py](../../vllm/compilation/compiler_interface.py))
24 - The model's forward function and the relevant functions called by the forward function (see below)
25
26 With all these factors taken into consideration, usually we can guarantee that the cache is safe to use, and will not cause any unexpected behavior. Therefore, the cache is enabled by default. If you want to debug the compilation process, or if you suspect the cache is causing some issues, you can disable it by setting the environment variable `VLLM_DISABLE_COMPILE_CACHE=1`.
docs/features/README.md
+12 -12
@@ -44,15 +44,15 @@ th:not(:first-child) {
44 | [SD](spec_decode.md) | ✅ | ✅ | ❌ | ✅ | | | | | | | | | | | |
45 | CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | | | | |
46 | [pooling](../models/pooling_models.md) | 🟠\* | 🟠\* | ✅ | ❌ | ✅ | ✅ | | | | | | | | | |
47 -| <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ❌ | [❌](gh-issue:7366) | ❌ | [❌](gh-issue:7366) | ✅ | ✅ | ✅ | | | | | | | | |
47 +| <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ❌ | [❌](https://github.com/vllm-project/vllm/issues/7366) | ❌ | [❌](https://github.com/vllm-project/vllm/issues/7366) | ✅ | ✅ | ✅ | | | | | | | | |
48 | <abbr title="Logprobs">logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | | | | | | | |
49 | <abbr title="Prompt Logprobs">prmpt logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | | | | | | |
50 | <abbr title="Async Output Processing">async output</abbr> | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | | | | | |
51 | multi-step | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | | | | |
52 -| [mm](multimodal_inputs.md) | ✅ | ✅ | [🟠](gh-pr:4194)<sup>^</sup> | ❔ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | | | |
53 -| best-of | ✅ | ✅ | ✅ | [❌](gh-issue:6137) | ✅ | ❌ | ✅ | ✅ | ✅ | ❔ | [❌](gh-issue:7968) | ✅ | ✅ | | |
54 -| beam-search | ✅ | ✅ | ✅ | [❌](gh-issue:6137) | ✅ | ❌ | ✅ | ✅ | ✅ | ❔ | [❌](gh-issue:7968) | ❔ | ✅ | ✅ | |
55 -| [prompt-embeds](prompt_embeds.md) | ✅ | [❌](gh-issue:25096) | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❔ | ❔ | ❌ | ❔ | ❔ | ✅ |
52 +| [mm](multimodal_inputs.md) | ✅ | ✅ | [🟠](https://github.com/vllm-project/vllm/pull/4194)<sup>^</sup> | ❔ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❔ | ✅ | | | |
53 +| best-of | ✅ | ✅ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/6137) | ✅ | ❌ | ✅ | ✅ | ✅ | ❔ | [❌](https://github.com/vllm-project/vllm/issues/7968) | ✅ | ✅ | | |
54 +| beam-search | ✅ | ✅ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/6137) | ✅ | ❌ | ✅ | ✅ | ✅ | ❔ | [❌](https://github.com/vllm-project/vllm/issues/7968) | ❔ | ✅ | ✅ | |
55 +| [prompt-embeds](prompt_embeds.md) | ✅ | [❌](https://github.com/vllm-project/vllm/issues/25096) | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❔ | ❔ | ❌ | ❔ | ❔ | ✅ |
56
57 \* Chunked prefill and prefix caching are only applicable to last-token pooling.
58 <sup>^</sup> LoRA is only applicable to the language backbone of multimodal models.
@@ -63,18 +63,18 @@ th:not(:first-child) {
63
64 | Feature | Volta | Turing | Ampere | Ada | Hopper | CPU | AMD | TPU | Intel GPU |
65 |-----------------------------------------------------------|---------------------|-----------|-----------|--------|------------|--------------------|--------|-----| ------------|
66 -| [CP][chunked-prefill] | [❌](gh-issue:2729) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
67 -| [APC](automatic_prefix_caching.md) | [❌](gh-issue:3687) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
66 +| [CP][chunked-prefill] | [❌](https://github.com/vllm-project/vllm/issues/2729) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
67 +| [APC](automatic_prefix_caching.md) | [❌](https://github.com/vllm-project/vllm/issues/3687) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
68 | [LoRA](lora.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
69 -| [SD](spec_decode.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | [🟠](gh-issue:26963) |
70 -| CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | [❌](gh-issue:26970) |
69 +| [SD](spec_decode.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | [🟠](https://github.com/vllm-project/vllm/issues/26963) |
70 +| CUDA graph | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | [❌](https://github.com/vllm-project/vllm/issues/26970) |
71 | [pooling](../models/pooling_models.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
72 | <abbr title="Encoder-Decoder Models">enc-dec</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
73 -| [mm](multimodal_inputs.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | [🟠](gh-issue:26965) |
73 +| [mm](multimodal_inputs.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | [🟠](https://github.com/vllm-project/vllm/issues/26965) |
74 | <abbr title="Logprobs">logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
75 | <abbr title="Prompt Logprobs">prmpt logP</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
76 | <abbr title="Async Output Processing">async output</abbr> | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
77 -| multi-step | ✅ | ✅ | ✅ | ✅ | ✅ | [❌](gh-issue:8477) | ✅ | ❌ | ✅ |
77 +| multi-step | ✅ | ✅ | ✅ | ✅ | ✅ | [❌](https://github.com/vllm-project/vllm/issues/8477) | ✅ | ❌ | ✅ |
78 | best-of | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
79 | beam-search | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
80 -| [prompt-embeds](prompt_embeds.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ? | [❌](gh-issue:25097) | ✅ |
80 +| [prompt-embeds](prompt_embeds.md) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ? | [❌](https://github.com/vllm-project/vllm/issues/25097) | ✅ |
docs/features/automatic_prefix_caching.md
+1 -1
@@ -11,7 +11,7 @@ Automatic Prefix Caching (APC in short) caches the KV cache of existing queries,
11
12 Set `enable_prefix_caching=True` in vLLM engine to enable APC. Here is an example:
13
14 -<gh-file:examples/offline_inference/automatic_prefix_caching.py>
14 +[examples/offline_inference/automatic_prefix_caching.py](../../examples/offline_inference/automatic_prefix_caching.py)
15
16 ## Example workloads
17
docs/features/disagg_prefill.md
+6 -6
@@ -17,14 +17,14 @@ Two main reasons:
17
18 ## Usage example
19
20 -Please refer to <gh-file:examples/online_serving/disaggregated_prefill.sh> for the example usage of disaggregated prefilling.
20 +Please refer to [examples/online_serving/disaggregated_prefill.sh](../../examples/online_serving/disaggregated_prefill.sh) for the example usage of disaggregated prefilling.
21
22 Now supports 5 types of connectors:
23
24 -- **SharedStorageConnector**: refer to <gh-file:examples/offline_inference/disaggregated-prefill-v1/run.sh> for the example usage of SharedStorageConnector disaggregated prefilling.
25 -- **LMCacheConnectorV1**: refer to <gh-file:examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh> for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission.
26 -- **NixlConnector**: refer to <gh-file:tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh> for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md).
27 -- **P2pNcclConnector**: refer to <gh-file:examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh> for the example usage of P2pNcclConnector disaggregated prefilling.
24 +- **SharedStorageConnector**: refer to [examples/offline_inference/disaggregated-prefill-v1/run.sh](../../examples/offline_inference/disaggregated-prefill-v1/run.sh) for the example usage of SharedStorageConnector disaggregated prefilling.
25 +- **LMCacheConnectorV1**: refer to [examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission.
26 +- **NixlConnector**: refer to [tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md).
27 +- **P2pNcclConnector**: refer to [examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh](../../examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh) for the example usage of P2pNcclConnector disaggregated prefilling.
28 - **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
29
30 ```bash
@@ -45,7 +45,7 @@ For NixlConnector, you may also specify one or multiple NIXL_Backend. Such as:
45
46 ## Benchmarks
47
48 -Please refer to <gh-file:benchmarks/disagg_benchmarks> for disaggregated prefilling benchmarks.
48 +Please refer to [benchmarks/disagg_benchmarks](../../benchmarks/disagg_benchmarks) for disaggregated prefilling benchmarks.
49
50 ## Development
51
docs/features/lora.md
+1 -1
@@ -47,7 +47,7 @@ the third parameter is the path to the LoRA adapter.
47 )
48 ```
49
50 -Check out <gh-file:examples/offline_inference/multilora_inference.py> for an example of how to use LoRA adapters with the async engine and how to use more advanced configuration options.
50 +Check out [examples/offline_inference/multilora_inference.py](../../examples/offline_inference/multilora_inference.py) for an example of how to use LoRA adapters with the async engine and how to use more advanced configuration options.
51
52 ## Serving LoRA Adapters
53
docs/features/multimodal_inputs.md
+11 -11
@@ -3,7 +3,7 @@
3 This page teaches you how to pass multi-modal inputs to [multi-modal models][supported-mm-models] in vLLM.
4
5 !!! note
6 - We are actively iterating on multi-modal support. See [this RFC](gh-issue:4194) for upcoming changes,
6 + We are actively iterating on multi-modal support. See [this RFC](https://github.com/vllm-project/vllm/issues/4194) for upcoming changes,
7 and [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) if you have any feedback or feature requests.
8
9 !!! tip
@@ -129,7 +129,7 @@ You can pass a single image to the `'image'` field of the multi-modal dictionary
129 print(generated_text)
130 ```
131
132 -Full example: <gh-file:examples/offline_inference/vision_language.py>
132 +Full example: [examples/offline_inference/vision_language.py](../../examples/offline_inference/vision_language.py)
133
134 To substitute multiple images inside the same text prompt, you can pass in a list of images instead:
135
@@ -162,7 +162,7 @@ To substitute multiple images inside the same text prompt, you can pass in a lis
162 print(generated_text)
163 ```
164
165 -Full example: <gh-file:examples/offline_inference/vision_language_multi_image.py>
165 +Full example: [examples/offline_inference/vision_language_multi_image.py](../../examples/offline_inference/vision_language_multi_image.py)
166
167 If using the [LLM.chat](../models/generative_models.md#llmchat) method, you can pass images directly in the message content using various formats: image URLs, PIL Image objects, or pre-computed embeddings:
168
@@ -346,13 +346,13 @@ Instead of NumPy arrays, you can also pass `'torch.Tensor'` instances, as shown
346 !!! note
347 'process_vision_info' is only applicable to Qwen2.5-VL and similar models.
348
349 -Full example: <gh-file:examples/offline_inference/vision_language.py>
349 +Full example: [examples/offline_inference/vision_language.py](../../examples/offline_inference/vision_language.py)
350
351 ### Audio Inputs
352
353 You can pass a tuple `(array, sampling_rate)` to the `'audio'` field of the multi-modal dictionary.
354
355 -Full example: <gh-file:examples/offline_inference/audio_language.py>
355 +Full example: [examples/offline_inference/audio_language.py](../../examples/offline_inference/audio_language.py)
356
357 ### Embedding Inputs
358
@@ -434,11 +434,11 @@ Our OpenAI-compatible server accepts multi-modal data via the [Chat Completions
434 A chat template is **required** to use Chat Completions API.
435 For HF format models, the default chat template is defined inside `chat_template.json` or `tokenizer_config.json`.
436
437 - If no default chat template is available, we will first look for a built-in fallback in <gh-file:vllm/transformers_utils/chat_templates/registry.py>.
437 + If no default chat template is available, we will first look for a built-in fallback in [vllm/transformers_utils/chat_templates/registry.py](../../vllm/transformers_utils/chat_templates/registry.py).
438 If no fallback is available, an error is raised and you have to provide the chat template manually via the `--chat-template` argument.
439
440 - For certain models, we provide alternative chat templates inside <gh-dir:examples>.
441 - For example, VLM2Vec uses <gh-file:examples/template_vlm2vec_phi3v.jinja> which is different from the default one for Phi-3-Vision.
440 + For certain models, we provide alternative chat templates inside [examples](../../examples).
441 + For example, VLM2Vec uses [examples/template_vlm2vec_phi3v.jinja](../../examples/template_vlm2vec_phi3v.jinja) which is different from the default one for Phi-3-Vision.
442
443 ### Image Inputs
444
@@ -524,7 +524,7 @@ Then, you can use the OpenAI client as follows:
524 print("Chat completion output:", chat_response.choices[0].message.content)
525 ```
526
527 -Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
527 +Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
528
529 !!! tip
530 Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via `--allowed-local-media-path` when launching the API server/engine,
@@ -595,7 +595,7 @@ Then, you can use the OpenAI client as follows:
595 print("Chat completion output from image url:", result)
596 ```
597
598 -Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
598 +Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
599
600 !!! note
601 By default, the timeout for fetching videos through HTTP URL is `30` seconds.
@@ -719,7 +719,7 @@ Alternatively, you can pass `audio_url`, which is the audio counterpart of `imag
719 print("Chat completion output from audio url:", result)
720 ```
721
722 -Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
722 +Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
723
724 !!! note
725 By default, the timeout for fetching audios through HTTP URL is `10` seconds.
docs/features/nixl_connector_usage.md
+4 -4
@@ -9,7 +9,7 @@ NixlConnector is a high-performance KV cache transfer connector for vLLM's disag
9 Install the NIXL library: `uv pip install nixl`, as a quick start.
10
11 - Refer to [NIXL official repository](https://github.com/ai-dynamo/nixl) for more installation instructions
12 -- The specified required NIXL version can be found in [requirements/kv_connectors.txt](gh-file:requirements/kv_connectors.txt) and other relevant config files
12 +- The specified required NIXL version can be found in [requirements/kv_connectors.txt](../../requirements/kv_connectors.txt) and other relevant config files
13
14 For non-cuda platform, please install nixl with ucx build from source, instructed as below.
15
@@ -170,6 +170,6 @@ Support use case: Prefill with 'HND' and decode with 'NHD' with experimental con
170
171 Refer to these example scripts in the vLLM repository:
172
173 -- [run_accuracy_test.sh](gh-file:tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh)
174 -- [toy_proxy_server.py](gh-file:tests/v1/kv_connector/nixl_integration/toy_proxy_server.py)
175 -- [test_accuracy.py](gh-file:tests/v1/kv_connector/nixl_integration/test_accuracy.py)
173 +- [run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh)
174 +- [toy_proxy_server.py](../../tests/v1/kv_connector/nixl_integration/toy_proxy_server.py)
175 +- [test_accuracy.py](../../tests/v1/kv_connector/nixl_integration/test_accuracy.py)
docs/features/prompt_embeds.md
+2 -2
@@ -16,7 +16,7 @@ To input multi-modal data, follow this schema in [vllm.inputs.EmbedsPrompt][]:
16
17 You can pass prompt embeddings from Hugging Face Transformers models to the `'prompt_embeds'` field of the prompt embedding dictionary, as shown in the following examples:
18
19 -<gh-file:examples/offline_inference/prompt_embed_inference.py>
19 +[examples/offline_inference/prompt_embed_inference.py](../../examples/offline_inference/prompt_embed_inference.py)
20
21 ## Online Serving
22
@@ -37,4 +37,4 @@ vllm serve meta-llama/Llama-3.2-1B-Instruct --runner generate \
37
38 Then, you can use the OpenAI client as follows:
39
40 -<gh-file:examples/online_serving/prompt_embed_inference_with_openai_client.py>
40 +[examples/online_serving/prompt_embed_inference_with_openai_client.py](../../examples/online_serving/prompt_embed_inference_with_openai_client.py)
docs/features/quantization/README.md
+1 -1
@@ -64,4 +64,4 @@ th:not(:first-child) {
64 !!! note
65 This compatibility chart is subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.
66
67 - For the most up-to-date information on hardware support and quantization methods, please refer to <gh-dir:vllm/model_executor/layers/quantization> or consult with the vLLM development team.
67 + For the most up-to-date information on hardware support and quantization methods, please refer to [vllm/model_executor/layers/quantization](../../../vllm/model_executor/layers/quantization) or consult with the vLLM development team.
docs/features/reasoning_outputs.md
+3 -3
@@ -196,7 +196,7 @@ The reasoning content is also available when both tool calling and the reasoning
196 print(f"Arguments: {tool_call.arguments}")
197 ```
198
199 -For more examples, please refer to <gh-file:examples/online_serving/openai_chat_completion_tool_calls_with_reasoning.py>.
199 +For more examples, please refer to [examples/online_serving/openai_chat_completion_tool_calls_with_reasoning.py](../../examples/online_serving/openai_chat_completion_tool_calls_with_reasoning.py).
200
201 ## Limitations
202
@@ -204,7 +204,7 @@ For more examples, please refer to <gh-file:examples/online_serving/openai_chat_
204
205 ## How to support a new reasoning model
206
207 -You can add a new `ReasoningParser` similar to <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
207 +You can add a new `ReasoningParser` similar to [vllm/reasoning/deepseek_r1_reasoning_parser.py](../../vllm/reasoning/deepseek_r1_reasoning_parser.py).
208
209 ??? code
210
@@ -264,7 +264,7 @@ You can add a new `ReasoningParser` similar to <gh-file:vllm/reasoning/deepseek_
264 """
265 ```
266
267 -Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
267 +Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in [vllm/reasoning/deepseek_r1_reasoning_parser.py](../../vllm/reasoning/deepseek_r1_reasoning_parser.py).
268
269 ??? code
270
docs/features/spec_decode.md
+7 -7
@@ -3,7 +3,7 @@
3 !!! warning
4 Please note that speculative decoding in vLLM is not yet optimized and does
5 not usually yield inter-token latency reductions for all prompt datasets or sampling parameters.
6 - The work to optimize it is ongoing and can be followed here: <gh-issue:4630>
6 + The work to optimize it is ongoing and can be followed here: <https://github.com/vllm-project/vllm/issues/4630>
7
8 !!! warning
9 Currently, speculative decoding in vLLM is not compatible with pipeline parallelism.
@@ -183,7 +183,7 @@ A variety of speculative models of this type are available on HF hub:
183 ## Speculating using EAGLE based draft models
184
185 The following code configures vLLM to use speculative decoding where proposals are generated by
186 -an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found [here](gh-file:examples/offline_inference/eagle.py).
186 +an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found [here](../../examples/offline_inference/spec_decode.py).
187
188 ??? code
189
@@ -218,8 +218,8 @@ an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https
218 A few important things to consider when using the EAGLE based draft models:
219
220 1. The EAGLE draft models available in the [HF repository for EAGLE models](https://huggingface.co/yuhuili) should
221 - be able to be loaded and used directly by vLLM after <gh-pr:12304>.
222 - If you are using vllm version before <gh-pr:12304>, please use the
221 + be able to be loaded and used directly by vLLM after <https://github.com/vllm-project/vllm/pull/12304>.
222 + If you are using vllm version before <https://github.com/vllm-project/vllm/pull/12304>, please use the
223 [script](https://gist.github.com/abhigoyal1997/1e7a4109ccb7704fbc67f625e86b2d6d) to convert the speculative model,
224 and specify `"model": "path/to/modified/eagle/model"` in `speculative_config`. If weight-loading problems still occur when using the latest version of vLLM, please leave a comment or raise an issue.
225
@@ -229,7 +229,7 @@ A few important things to consider when using the EAGLE based draft models:
229
230 3. When using EAGLE-based speculators with vLLM, the observed speedup is lower than what is
231 reported in the reference implementation [here](https://github.com/SafeAILab/EAGLE). This issue is under
232 - investigation and tracked here: <gh-issue:9565>.
232 + investigation and tracked here: <https://github.com/vllm-project/vllm/issues/9565>.
233
234 4. When using EAGLE-3 based draft model, option "method" must be set to "eagle3".
235 That is, to specify `"method": "eagle3"` in `speculative_config`.
@@ -267,7 +267,7 @@ speculative decoding, breaking down the guarantees into three key areas:
267 > distribution. [View Test Code](https://github.com/vllm-project/vllm/blob/47b65a550866c7ffbd076ecb74106714838ce7da/tests/samplers/test_rejection_sampler.py#L252)
268 > - **Greedy Sampling Equality**: Confirms that greedy sampling with speculative decoding matches greedy sampling
269 > without it. This verifies that vLLM's speculative decoding framework, when integrated with the vLLM forward pass and the vLLM rejection sampler,
270 - > provides a lossless guarantee. Almost all of the tests in <gh-dir:tests/spec_decode/e2e>.
270 + > provides a lossless guarantee. Almost all of the tests in [tests/spec_decode/e2e](../../tests/spec_decode/e2e).
271 > verify this property using [this assertion implementation](https://github.com/vllm-project/vllm/blob/b67ae00cdbbe1a58ffc8ff170f0c8d79044a684a/tests/spec_decode/e2e/conftest.py#L291)
272
273 3. **vLLM Logprob Stability**
@@ -289,4 +289,4 @@ For mitigation strategies, please refer to the FAQ entry *Can the output of a pr
289 - [A Hacker's Guide to Speculative Decoding in vLLM](https://www.youtube.com/watch?v=9wNAgpX6z_4)
290 - [What is Lookahead Scheduling in vLLM?](https://docs.google.com/document/d/1Z9TvqzzBPnh5WHcRwjvK2UEeFeq5zMZb5mFE8jR0HCs/edit#heading=h.1fjfb0donq5a)
291 - [Information on batch expansion](https://docs.google.com/document/d/1T-JaS2T1NRfdP51qzqpyakoCXxSXTtORppiwaj5asxA/edit#heading=h.kk7dq05lc6q8)
292 -- [Dynamic speculative decoding](gh-issue:4565)
292 +- [Dynamic speculative decoding](https://github.com/vllm-project/vllm/issues/4565)
docs/features/structured_outputs.md
+1 -1
@@ -298,7 +298,7 @@ Step #2: explanation="Next, let's isolate 'x' by dividing both sides of the equa
298 Answer: x = -29/8
299 ```
300
301 -An example of using `structural_tag` can be found here: <gh-file:examples/online_serving/structured_outputs>
301 +An example of using `structural_tag` can be found here: [examples/online_serving/structured_outputs](../../examples/online_serving/structured_outputs)
302
303 ## Offline Inference
304
docs/features/tool_calling.md
+19 -19
@@ -151,9 +151,9 @@ Known issues:
151 much shorter than what vLLM generates. Since an exception is thrown when this condition
152 is not met, the following additional chat templates are provided:
153
154 - * <gh-file:examples/tool_chat_template_mistral.jinja> - this is the "official" Mistral chat template, but tweaked so that
154 + * [examples/tool_chat_template_mistral.jinja](../../examples/tool_chat_template_mistral.jinja) - this is the "official" Mistral chat template, but tweaked so that
155 it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)
156 - * <gh-file:examples/tool_chat_template_mistral_parallel.jinja> - this is a "better" version that adds a tool-use system prompt
156 + * [examples/tool_chat_template_mistral_parallel.jinja](../../examples/tool_chat_template_mistral_parallel.jinja) - this is a "better" version that adds a tool-use system prompt
157 when tools are provided, that results in much better reliability when working with parallel tool calling.
158
159 Recommended flags:
@@ -187,16 +187,16 @@ Known issues:
187
188 VLLM provides two JSON-based chat templates for Llama 3.1 and 3.2:
189
190 -* <gh-file:examples/tool_chat_template_llama3.1_json.jinja> - this is the "official" chat template for the Llama 3.1
190 +* [examples/tool_chat_template_llama3.1_json.jinja](../../examples/tool_chat_template_llama3.1_json.jinja) - this is the "official" chat template for the Llama 3.1
191 models, but tweaked so that it works better with vLLM.
192 -* <gh-file:examples/tool_chat_template_llama3.2_json.jinja> - this extends upon the Llama 3.1 chat template by adding support for
192 +* [examples/tool_chat_template_llama3.2_json.jinja](../../examples/tool_chat_template_llama3.2_json.jinja) - this extends upon the Llama 3.1 chat template by adding support for
193 images.
194
195 Recommended flags: `--tool-call-parser llama3_json --chat-template {see_above}`
196
197 VLLM also provides a pythonic and JSON-based chat template for Llama 4, but pythonic tool calling is recommended:
198
199 -* <gh-file:examples/tool_chat_template_llama4_pythonic.jinja> - this is based on the [official chat template](https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/) for the Llama 4 models.
199 +* [examples/tool_chat_template_llama4_pythonic.jinja](../../examples/tool_chat_template_llama4_pythonic.jinja) - this is based on the [official chat template](https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/) for the Llama 4 models.
200
201 For Llama 4 model, use `--tool-call-parser llama4_pythonic --chat-template examples/tool_chat_template_llama4_pythonic.jinja`.
202
@@ -212,7 +212,7 @@ Supported models:
212
213 Recommended flags: `--tool-call-parser granite --chat-template examples/tool_chat_template_granite.jinja`
214
215 - <gh-file:examples/tool_chat_template_granite.jinja>: this is a modified chat template from the original on Hugging Face. Parallel function calls are supported.
215 + [examples/tool_chat_template_granite.jinja](../../examples/tool_chat_template_granite.jinja): this is a modified chat template from the original on Hugging Face. Parallel function calls are supported.
216
217 * `ibm-granite/granite-3.1-8b-instruct`
218
@@ -224,7 +224,7 @@ Supported models:
224
225 Recommended flags: `--tool-call-parser granite-20b-fc --chat-template examples/tool_chat_template_granite_20b_fc.jinja`
226
227 - <gh-file:examples/tool_chat_template_granite_20b_fc.jinja>: this is a modified chat template from the original on Hugging Face, which is not vLLM-compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
227 + [examples/tool_chat_template_granite_20b_fc.jinja](../../examples/tool_chat_template_granite_20b_fc.jinja): this is a modified chat template from the original on Hugging Face, which is not vLLM-compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
228
229 ### InternLM Models (`internlm`)
230
@@ -282,8 +282,8 @@ Flags: `--tool-call-parser hermes`
282
283 Supported models:
284
285 -* `MiniMaxAi/MiniMax-M1-40k` (use with <gh-file:examples/tool_chat_template_minimax_m1.jinja>)
286 -* `MiniMaxAi/MiniMax-M1-80k` (use with <gh-file:examples/tool_chat_template_minimax_m1.jinja>)
285 +* `MiniMaxAi/MiniMax-M1-40k` (use with [examples/tool_chat_template_minimax_m1.jinja](../../examples/tool_chat_template_minimax_m1.jinja))
286 +* `MiniMaxAi/MiniMax-M1-80k` (use with [examples/tool_chat_template_minimax_m1.jinja](../../examples/tool_chat_template_minimax_m1.jinja))
287
288 Flags: `--tool-call-parser minimax --chat-template examples/tool_chat_template_minimax_m1.jinja`
289
@@ -291,8 +291,8 @@ Flags: `--tool-call-parser minimax --chat-template examples/tool_chat_template_m
291
292 Supported models:
293
294 -* `deepseek-ai/DeepSeek-V3-0324` (use with <gh-file:examples/tool_chat_template_deepseekv3.jinja>)
295 -* `deepseek-ai/DeepSeek-R1-0528` (use with <gh-file:examples/tool_chat_template_deepseekr1.jinja>)
294 +* `deepseek-ai/DeepSeek-V3-0324` (use with [examples/tool_chat_template_deepseekv3.jinja](../../examples/tool_chat_template_deepseekv3.jinja))
295 +* `deepseek-ai/DeepSeek-R1-0528` (use with [examples/tool_chat_template_deepseekr1.jinja](../../examples/tool_chat_template_deepseekr1.jinja))
296
297 Flags: `--tool-call-parser deepseek_v3 --chat-template {see_above}`
298
@@ -300,7 +300,7 @@ Flags: `--tool-call-parser deepseek_v3 --chat-template {see_above}`
300
301 Supported models:
302
303 -* `deepseek-ai/DeepSeek-V3.1` (use with <gh-file:examples/tool_chat_template_deepseekv31.jinja>)
303 +* `deepseek-ai/DeepSeek-V3.1` (use with [examples/tool_chat_template_deepseekv31.jinja](../../examples/tool_chat_template_deepseekv31.jinja))
304
305 Flags: `--tool-call-parser deepseek_v31 --chat-template {see_above}`
306
@@ -379,12 +379,12 @@ Limitations:
379
380 Example supported models:
381
382 -* `meta-llama/Llama-3.2-1B-Instruct` ⚠️ (use with <gh-file:examples/tool_chat_template_llama3.2_pythonic.jinja>)
383 -* `meta-llama/Llama-3.2-3B-Instruct` ⚠️ (use with <gh-file:examples/tool_chat_template_llama3.2_pythonic.jinja>)
384 -* `Team-ACE/ToolACE-8B` (use with <gh-file:examples/tool_chat_template_toolace.jinja>)
385 -* `fixie-ai/ultravox-v0_4-ToolACE-8B` (use with <gh-file:examples/tool_chat_template_toolace.jinja>)
386 -* `meta-llama/Llama-4-Scout-17B-16E-Instruct` ⚠️ (use with <gh-file:examples/tool_chat_template_llama4_pythonic.jinja>)
387 -* `meta-llama/Llama-4-Maverick-17B-128E-Instruct` ⚠️ (use with <gh-file:examples/tool_chat_template_llama4_pythonic.jinja>)
382 +* `meta-llama/Llama-3.2-1B-Instruct` ⚠️ (use with [examples/tool_chat_template_llama3.2_pythonic.jinja](../../examples/tool_chat_template_llama3.2_pythonic.jinja))
383 +* `meta-llama/Llama-3.2-3B-Instruct` ⚠️ (use with [examples/tool_chat_template_llama3.2_pythonic.jinja](../../examples/tool_chat_template_llama3.2_pythonic.jinja))
384 +* `Team-ACE/ToolACE-8B` (use with [examples/tool_chat_template_toolace.jinja](../../examples/tool_chat_template_toolace.jinja))
385 +* `fixie-ai/ultravox-v0_4-ToolACE-8B` (use with [examples/tool_chat_template_toolace.jinja](../../examples/tool_chat_template_toolace.jinja))
386 +* `meta-llama/Llama-4-Scout-17B-16E-Instruct` ⚠️ (use with [examples/tool_chat_template_llama4_pythonic.jinja](../../examples/tool_chat_template_llama4_pythonic.jinja))
387 +* `meta-llama/Llama-4-Maverick-17B-128E-Instruct` ⚠️ (use with [examples/tool_chat_template_llama4_pythonic.jinja](../../examples/tool_chat_template_llama4_pythonic.jinja))
388
389 Flags: `--tool-call-parser pythonic --chat-template {see_above}`
390
@@ -393,7 +393,7 @@ Flags: `--tool-call-parser pythonic --chat-template {see_above}`
393
394 ## How to Write a Tool Parser Plugin
395
396 -A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in <gh-file:vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py>.
396 +A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in [vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py](../../vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py).
397
398 Here is a summary of a plugin file:
399
docs/getting_started/installation/cpu.apple.inc.md renamed
docs/getting_started/installation/cpu.arm.inc.md renamed
docs/getting_started/installation/cpu.md
+17 -17
@@ -4,19 +4,19 @@ vLLM is a Python library that supports the following CPU variants. Select your C
4
5 === "Intel/AMD x86"
6
7 - --8<-- "docs/getting_started/installation/cpu/x86.inc.md:installation"
7 + --8<-- "docs/getting_started/installation/cpu.x86.inc.md:installation"
8
9 === "ARM AArch64"
10
11 - --8<-- "docs/getting_started/installation/cpu/arm.inc.md:installation"
11 + --8<-- "docs/getting_started/installation/cpu.arm.inc.md:installation"
12
13 === "Apple silicon"
14
15 - --8<-- "docs/getting_started/installation/cpu/apple.inc.md:installation"
15 + --8<-- "docs/getting_started/installation/cpu.apple.inc.md:installation"
16
17 === "IBM Z (S390X)"
18
19 - --8<-- "docs/getting_started/installation/cpu/s390x.inc.md:installation"
19 + --8<-- "docs/getting_started/installation/cpu.s390x.inc.md:installation"
20
21 ## Requirements
22
@@ -24,19 +24,19 @@ vLLM is a Python library that supports the following CPU variants. Select your C
24
25 === "Intel/AMD x86"
26
27 - --8<-- "docs/getting_started/installation/cpu/x86.inc.md:requirements"
27 + --8<-- "docs/getting_started/installation/cpu.x86.inc.md:requirements"
28
29 === "ARM AArch64"
30
31 - --8<-- "docs/getting_started/installation/cpu/arm.inc.md:requirements"
31 + --8<-- "docs/getting_started/installation/cpu.arm.inc.md:requirements"
32
33 === "Apple silicon"
34
35 - --8<-- "docs/getting_started/installation/cpu/apple.inc.md:requirements"
35 + --8<-- "docs/getting_started/installation/cpu.apple.inc.md:requirements"
36
37 === "IBM Z (S390X)"
38
39 - --8<-- "docs/getting_started/installation/cpu/s390x.inc.md:requirements"
39 + --8<-- "docs/getting_started/installation/cpu.s390x.inc.md:requirements"
40
41 ## Set up using Python
42
@@ -52,19 +52,19 @@ Currently, there are no pre-built CPU wheels.
52
53 === "Intel/AMD x86"
54
55 - --8<-- "docs/getting_started/installation/cpu/x86.inc.md:build-wheel-from-source"
55 + --8<-- "docs/getting_started/installation/cpu.x86.inc.md:build-wheel-from-source"
56
57 === "ARM AArch64"
58
59 - --8<-- "docs/getting_started/installation/cpu/arm.inc.md:build-wheel-from-source"
59 + --8<-- "docs/getting_started/installation/cpu.arm.inc.md:build-wheel-from-source"
60
61 === "Apple silicon"
62
63 - --8<-- "docs/getting_started/installation/cpu/apple.inc.md:build-wheel-from-source"
63 + --8<-- "docs/getting_started/installation/cpu.apple.inc.md:build-wheel-from-source"
64
65 === "IBM Z (s390x)"
66
67 - --8<-- "docs/getting_started/installation/cpu/s390x.inc.md:build-wheel-from-source"
67 + --8<-- "docs/getting_started/installation/cpu.s390x.inc.md:build-wheel-from-source"
68
69 ## Set up using Docker
70
@@ -72,24 +72,24 @@ Currently, there are no pre-built CPU wheels.
72
73 === "Intel/AMD x86"
74
75 - --8<-- "docs/getting_started/installation/cpu/x86.inc.md:pre-built-images"
75 + --8<-- "docs/getting_started/installation/cpu.x86.inc.md:pre-built-images"
76
77 ### Build image from source
78
79 === "Intel/AMD x86"
80
81 - --8<-- "docs/getting_started/installation/cpu/x86.inc.md:build-image-from-source"
81 + --8<-- "docs/getting_started/installation/cpu.x86.inc.md:build-image-from-source"
82
83 === "ARM AArch64"
84
85 - --8<-- "docs/getting_started/installation/cpu/arm.inc.md:build-image-from-source"
85 + --8<-- "docs/getting_started/installation/cpu.arm.inc.md:build-image-from-source"
86
87 === "Apple silicon"
88
89 - --8<-- "docs/getting_started/installation/cpu/arm.inc.md:build-image-from-source"
89 + --8<-- "docs/getting_started/installation/cpu.arm.inc.md:build-image-from-source"
90
91 === "IBM Z (S390X)"
92 - --8<-- "docs/getting_started/installation/cpu/s390x.inc.md:build-image-from-source"
92 + --8<-- "docs/getting_started/installation/cpu.s390x.inc.md:build-image-from-source"
93
94 ## Related runtime environment variables
95
docs/getting_started/installation/cpu.s390x.inc.md renamed
docs/getting_started/installation/cpu.x86.inc.md renamed
docs/getting_started/installation/google_tpu.md
+1 -1
@@ -157,7 +157,7 @@ See [deployment-docker-pre-built-image][deployment-docker-pre-built-image] for i
157
158 ### Build image from source
159
160 -You can use <gh-file:docker/Dockerfile.tpu> to build a Docker image with TPU support.
160 +You can use [docker/Dockerfile.tpu](../../../docker/Dockerfile.tpu) to build a Docker image with TPU support.
161
162 ```bash
163 docker build -f docker/Dockerfile.tpu -t vllm-tpu .
docs/getting_started/installation/gpu.cuda.inc.md renamed
+1 -1
@@ -11,7 +11,7 @@ vLLM contains pre-compiled C++ and CUDA (12.8) binaries.
11 # --8<-- [start:set-up-using-python]
12
13 !!! note
14 - PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
14 + PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <https://github.com/vllm-project/vllm/issues/8420> for more details.
15
16 In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.
17
docs/getting_started/installation/gpu.md
+24 -24
@@ -4,15 +4,15 @@ vLLM is a Python library that supports the following GPU variants. Select your G
4
5 === "NVIDIA CUDA"
6
7 - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:installation"
7 + --8<-- "docs/getting_started/installation/gpu.cuda.inc.md:installation"
8
9 === "AMD ROCm"
10
11 - --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:installation"
11 + --8<-- "docs/getting_started/installation/gpu.rocm.inc.md:installation"
12
13 === "Intel XPU"
14
15 - --8<-- "docs/getting_started/installation/gpu/xpu.inc.md:installation"
15 + --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:installation"
16
17 ## Requirements
18
@@ -24,15 +24,15 @@ vLLM is a Python library that supports the following GPU variants. Select your G
24
25 === "NVIDIA CUDA"
26
27 - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:requirements"
27 + --8<-- "docs/getting_started/installation/gpu.cuda.inc.md:requirements"
28
29 === "AMD ROCm"
30
31 - --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:requirements"
31 + --8<-- "docs/getting_started/installation/gpu.rocm.inc.md:requirements"
32
33 === "Intel XPU"
34
35 - --8<-- "docs/getting_started/installation/gpu/xpu.inc.md:requirements"
35 + --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:requirements"
36
37 ## Set up using Python
38
@@ -42,29 +42,29 @@ vLLM is a Python library that supports the following GPU variants. Select your G
42
43 === "NVIDIA CUDA"
44
45 - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:set-up-using-python"
45 + --8<-- "docs/getting_started/installation/gpu.cuda.inc.md:set-up-using-python"
46
47 === "AMD ROCm"
48
49 - --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:set-up-using-python"
49 + --8<-- "docs/getting_started/installation/gpu.rocm.inc.md:set-up-using-python"
50
51 === "Intel XPU"
52
53 - --8<-- "docs/getting_started/installation/gpu/xpu.inc.md:set-up-using-python"
53 + --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:set-up-using-python"
54
55 ### Pre-built wheels
56
57 === "NVIDIA CUDA"
58
59 - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:pre-built-wheels"
59 + --8<-- "docs/getting_started/installation/gpu.cuda.inc.md:pre-built-wheels"
60
61 === "AMD ROCm"
62
63 - --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:pre-built-wheels"
63 + --8<-- "docs/getting_started/installation/gpu.rocm.inc.md:pre-built-wheels"
64
65 === "Intel XPU"
66
67 - --8<-- "docs/getting_started/installation/gpu/xpu.inc.md:pre-built-wheels"
67 + --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:pre-built-wheels"
68
69 [](){ #build-from-source }
70
@@ -72,15 +72,15 @@ vLLM is a Python library that supports the following GPU variants. Select your G
72
73 === "NVIDIA CUDA"
74
75 - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:build-wheel-from-source"
75 + --8<-- "docs/getting_started/installation/gpu.cuda.inc.md:build-wheel-from-source"
76
77 === "AMD ROCm"
78
79 - --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:build-wheel-from-source"
79 + --8<-- "docs/getting_started/installation/gpu.rocm.inc.md:build-wheel-from-source"
80
81 === "Intel XPU"
82
83 - --8<-- "docs/getting_started/installation/gpu/xpu.inc.md:build-wheel-from-source"
83 + --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:build-wheel-from-source"
84
85 ## Set up using Docker
86
@@ -88,40 +88,40 @@ vLLM is a Python library that supports the following GPU variants. Select your G
88
89 === "NVIDIA CUDA"
90
91 - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:pre-built-images"
91 + --8<-- "docs/getting_started/installation/gpu.cuda.inc.md:pre-built-images"
92
93 === "AMD ROCm"
94
95 - --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:pre-built-images"
95 + --8<-- "docs/getting_started/installation/gpu.rocm.inc.md:pre-built-images"
96
97 === "Intel XPU"
98
99 - --8<-- "docs/getting_started/installation/gpu/xpu.inc.md:pre-built-images"
99 + --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:pre-built-images"
100
101 ### Build image from source
102
103 === "NVIDIA CUDA"
104
105 - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:build-image-from-source"
105 + --8<-- "docs/getting_started/installation/gpu.cuda.inc.md:build-image-from-source"
106
107 === "AMD ROCm"
108
109 - --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:build-image-from-source"
109 + --8<-- "docs/getting_started/installation/gpu.rocm.inc.md:build-image-from-source"
110
111 === "Intel XPU"
112
113 - --8<-- "docs/getting_started/installation/gpu/xpu.inc.md:build-image-from-source"
113 + --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:build-image-from-source"
114
115 ## Supported features
116
117 === "NVIDIA CUDA"
118
119 - --8<-- "docs/getting_started/installation/gpu/cuda.inc.md:supported-features"
119 + --8<-- "docs/getting_started/installation/gpu.cuda.inc.md:supported-features"
120
121 === "AMD ROCm"
122
123 - --8<-- "docs/getting_started/installation/gpu/rocm.inc.md:supported-features"
123 + --8<-- "docs/getting_started/installation/gpu.rocm.inc.md:supported-features"
124
125 === "Intel XPU"
126
127 - --8<-- "docs/getting_started/installation/gpu/xpu.inc.md:supported-features"
127 + --8<-- "docs/getting_started/installation/gpu.xpu.inc.md:supported-features"
docs/getting_started/installation/gpu.rocm.inc.md renamed
+4 -4
@@ -146,7 +146,7 @@ Building the Docker image from source is the recommended way to use vLLM with RO
146
147 #### (Optional) Build an image with ROCm software stack
148
149 -Build a docker image from <gh-file:docker/Dockerfile.rocm_base> which setup ROCm software stack needed by the vLLM.
149 +Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM.
150 **This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**
151 If you choose to build this rocm_base image yourself, the steps are as follows.
152
@@ -170,7 +170,7 @@ DOCKER_BUILDKIT=1 docker build \
170
171 #### Build an image with vLLM
172
173 -First, build a docker image from <gh-file:docker/Dockerfile.rocm> and launch a docker container from the image.
173 +First, build a docker image from [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) and launch a docker container from the image.
174 It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
175
176 ```bash
@@ -181,10 +181,10 @@ It is important that the user kicks off the docker build using buildkit. Either
181 }
182 ```
183
184 -<gh-file:docker/Dockerfile.rocm> uses ROCm 6.3 by default, but also supports ROCm 5.7, 6.0, 6.1, and 6.2, in older vLLM branches.
184 +[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 6.3 by default, but also supports ROCm 5.7, 6.0, 6.1, and 6.2, in older vLLM branches.
185 It provides flexibility to customize the build of docker image using the following arguments:
186
187 -- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using <gh-file:docker/Dockerfile.rocm_base>
187 +- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base)
188 - `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image
189
190 Their values can be passed in when running `docker build` with `--build-arg` options.
docs/getting_started/installation/gpu.xpu.inc.md renamed
+1 -1
@@ -75,7 +75,7 @@ vllm serve facebook/opt-13b \
75 -tp=8
76 ```
77
78 -By default, a ray instance will be launched automatically if no existing one is detected in the system, with `num-gpus` equals to `parallel_config.world_size`. We recommend properly starting a ray cluster before execution, referring to the <gh-file:examples/online_serving/run_cluster.sh> helper script.
78 +By default, a ray instance will be launched automatically if no existing one is detected in the system, with `num-gpus` equals to `parallel_config.world_size`. We recommend properly starting a ray cluster before execution, referring to the [examples/online_serving/run_cluster.sh](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/run_cluster.sh) helper script.
79
80 # --8<-- [end:supported-features]
81 # --8<-- [start:distributed-backend]
docs/getting_started/quickstart.md
+3 -3
@@ -46,7 +46,7 @@ uv pip install vllm --torch-backend=auto
46
47 ## Offline Batched Inference
48
49 -With vLLM installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing). See the example script: <gh-file:examples/offline_inference/basic/basic.py>
49 +With vLLM installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing). See the example script: [examples/offline_inference/basic/basic.py](../../examples/offline_inference/basic/basic.py)
50
51 The first line of this example imports the classes [LLM][vllm.LLM] and [SamplingParams][vllm.SamplingParams]:
52
@@ -201,7 +201,7 @@ Since this server is compatible with OpenAI API, you can use it as a drop-in rep
201 print("Completion result:", completion)
202 ```
203
204 -A more detailed client example can be found here: <gh-file:examples/online_serving/openai_completion_client.py>
204 +A more detailed client example can be found here: [examples/offline_inference/basic/basic.py](../../examples/offline_inference/basic/basic.py)
205
206 ### OpenAI Chat Completions API with vLLM
207
@@ -253,4 +253,4 @@ Currently, vLLM supports multiple backends for efficient Attention computation a
253 If desired, you can also manually set the backend of your choice by configuring the environment variable `VLLM_ATTENTION_BACKEND` to one of the following options: `FLASH_ATTN`, `FLASHINFER` or `XFORMERS`.
254
255 !!! warning
256 - There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see <gh-file:docker/Dockerfile> for instructions on how to install it.
256 + There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see [docker/Dockerfile](../../docker/Dockerfile) for instructions on how to install it.
docs/mkdocs/hooks/generate_examples.py
+9 -2
@@ -137,13 +137,20 @@ class Example:
137 gh_file = (self.main_file.parent / relative_path).resolve()
138 gh_file = gh_file.relative_to(ROOT_DIR)
139
140 - return f"[{link_text}](gh-file:{gh_file})"
140 + # Make GitHub URL
141 + url = "https://github.com/vllm-project/vllm/"
142 + url += "tree/main" if self.path.is_dir() else "blob/main"
143 + gh_url = f"{url}/{gh_file}"
144 +
145 + return f"[{link_text}]({gh_url})"
146
147 return re.sub(link_pattern, replace_link, content)
148
149 def generate(self) -> str:
150 content = f"# {self.title}\n\n"
146 - content += f"Source <gh-file:{self.path.relative_to(ROOT_DIR)}>.\n\n"
151 + url = "https://github.com/vllm-project/vllm/"
152 + url += "tree/main" if self.path.is_dir() else "blob/main"
153 + content += f"Source <{url}/{self.path.relative_to(ROOT_DIR)}>.\n\n"
154
155 # Use long code fence to avoid issues with
156 # included files containing code fences too
docs/mkdocs/hooks/url_schemes.py
+71 -99
@@ -1,123 +1,95 @@
1 # SPDX-License-Identifier: Apache-2.0
2 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 """
4 -This is basically a port of MyST parser’s external URL resolution mechanism
5 -(https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html#customising-external-url-resolution)
6 -to work with MkDocs.
4 +MkDocs hook to enable the following links to render correctly:
5
8 -It allows Markdown authors to use GitHub shorthand links like:
9 -
10 - - [Text](gh-issue:123)
11 - - <gh-pr:456>
12 - - [File](gh-file:path/to/file.py#L10)
13 -
14 -These are automatically rewritten into fully qualified GitHub URLs pointing to
15 -issues, pull requests, files, directories, or projects in the
16 -`vllm-project/vllm` repository.
6 +- Relative file links outside of the `docs/` directory, e.g.:
7 + - [Text](../some_file.py)
8 + - [Directory](../../some_directory/)
9 +- GitHub URLs for issues, pull requests, and projects, e.g.:
10 + - Adds GitHub icon before links
11 + - Replaces raw links with descriptive text,
12 + e.g. <...pull/123> -> [Pull Request #123](.../pull/123)
13 + - Works for external repos too by including the `owner/repo` in the link title
14
15 The goal is to simplify cross-referencing common GitHub resources
16 in project docs.
17 """
18
19 +from pathlib import Path
20 +
21 import regex as re
22 from mkdocs.config.defaults import MkDocsConfig
23 from mkdocs.structure.files import Files
24 from mkdocs.structure.pages import Page
25
26 +ROOT_DIR = Path(__file__).parent.parent.parent.parent.resolve()
27 +DOC_DIR = ROOT_DIR / "docs"
28 +
29 +
30 +gh_icon = ":octicons-mark-github-16:"
31 +
32 +# Regex pieces
33 +TITLE = r"(?P<title>[^\[\]<>]+?)"
34 +REPO = r"(?P<repo>.+?/.+?)"
35 +TYPE = r"(?P<type>issues|pull|projects)"
36 +NUMBER = r"(?P<number>\d+)"
37 +FRAGMENT = r"(?P<fragment>#[^\s]+)?"
38 +URL = f"https://github.com/{REPO}/{TYPE}/{NUMBER}{FRAGMENT}"
39 +RELATIVE = r"(?!(https?|ftp)://|#)(?P<path>[^\s]+?)"
40 +
41 +# Common titles to use for GitHub links when none is provided in the link.
42 +TITLES = {"issues": "Issue ", "pull": "Pull Request ", "projects": "Project "}
43 +
44 +# Regex to match GitHub issue, PR, and project links with optional titles.
45 +github_link = re.compile(rf"(\[{TITLE}\]\(|<){URL}(\)|>)")
46 +# Regex to match relative file links with optional titles.
47 +relative_link = re.compile(rf"\[{TITLE}\]\({RELATIVE}\)")
48 +
49
50 def on_page_markdown(
51 markdown: str, *, page: Page, config: MkDocsConfig, files: Files
52 ) -> str:
31 - """
32 - Custom MkDocs plugin hook to rewrite special GitHub reference links
33 - in Markdown.
34 -
35 - This function scans the given Markdown content for specially formatted
36 - GitHub shorthand links, such as:
37 - - `[Link text](gh-issue:123)`
38 - - `<gh-pr:456>`
39 -
40 - And rewrites them into fully-qualified GitHub URLs with GitHub icons:
41 - - `[:octicons-mark-github-16: Link text](https://github.com/vllm-project/vllm/issues/123)`
42 - - `[:octicons-mark-github-16: Pull Request #456](https://github.com/vllm-project/vllm/pull/456)`
43 -
44 - Supported shorthand types:
45 - - `gh-issue`
46 - - `gh-pr`
47 - - `gh-project`
48 - - `gh-dir`
49 - - `gh-file`
50 -
51 - Args:
52 - markdown (str): The raw Markdown content of the page.
53 - page (Page): The MkDocs page object being processed.
54 - config (MkDocsConfig): The MkDocs site configuration.
55 - files (Files): The collection of files in the MkDocs build.
56 -
57 - Returns:
58 - str: The updated Markdown content with GitHub shorthand links replaced.
59 - """
60 - gh_icon = ":octicons-mark-github-16:"
61 - gh_url = "https://github.com"
62 - repo_url = f"{gh_url}/vllm-project/vllm"
63 - org_url = f"{gh_url}/orgs/vllm-project"
64 -
65 - # Mapping of shorthand types to their corresponding GitHub base URLs
66 - urls = {
67 - "issue": f"{repo_url}/issues",
68 - "pr": f"{repo_url}/pull",
69 - "project": f"{org_url}/projects",
70 - "dir": f"{repo_url}/tree/main",
71 - "file": f"{repo_url}/blob/main",
72 - }
73 -
74 - # Default title prefixes for auto links
75 - titles = {
76 - "issue": "Issue #",
77 - "pr": "Pull Request #",
78 - "project": "Project #",
79 - "dir": "",
80 - "file": "",
81 - }
82 -
83 - # Regular expression to match GitHub shorthand links
84 - scheme = r"gh-(?P<type>.+?):(?P<path>.+?)(#(?P<fragment>.+?))?"
85 - inline_link = re.compile(r"\[(?P<title>[^\[]+?)\]\(" + scheme + r"\)")
86 - auto_link = re.compile(f"<{scheme}>")
87 -
88 - def replace_inline_link(match: re.Match) -> str:
89 - """
90 - Replaces a matched inline-style GitHub shorthand link
91 - with a full Markdown link.
92 -
93 - Example:
94 - [My issue](gh-issue:123) → [:octicons-mark-github-16: My issue](https://github.com/vllm-project/vllm/issues/123)
95 - """
96 - url = f"{urls[match.group('type')]}/{match.group('path')}"
97 - if fragment := match.group("fragment"):
98 - url += f"#{fragment}"
99 -
100 - return f"[{gh_icon} {match.group('title')}]({url})"
101 -
102 - def replace_auto_link(match: re.Match) -> str:
103 - """
104 - Replaces a matched autolink-style GitHub shorthand
105 - with a full Markdown link.
106 -
107 - Example:
108 - <gh-pr:456> → [:octicons-mark-github-16: Pull Request #456](https://github.com/vllm-project/vllm/pull/456)
109 - """
110 - type = match.group("type")
53 + def replace_relative_link(match: re.Match) -> str:
54 + """Replace relative file links with URLs if they point outside the docs dir."""
55 + title = match.group("title")
56 path = match.group("path")
112 - title = f"{titles[type]}{path}"
113 - url = f"{urls[type]}/{path}"
114 - if fragment := match.group("fragment"):
115 - url += f"#{fragment}"
57 + path = (Path(page.file.abs_src_path).parent / path).resolve()
58 +
59 + # Check if the path exists and is outside the docs dir
60 + if not path.exists() or path.is_relative_to(DOC_DIR):
61 + return match.group(0)
62 +
63 + # Files and directories have different URL schemes on GitHub
64 + slug = "tree/main" if path.is_dir() else "blob/main"
65
66 + path = path.relative_to(ROOT_DIR)
67 + url = f"https://github.com/vllm-project/vllm/{slug}/{path}"
68 return f"[{gh_icon} {title}]({url})"
69
119 - # Replace both inline and autolinks
120 - markdown = inline_link.sub(replace_inline_link, markdown)
121 - markdown = auto_link.sub(replace_auto_link, markdown)
70 + def replace_github_link(match: re.Match) -> str:
71 + """Replace GitHub issue, PR, and project links with enhanced Markdown links."""
72 + repo = match.group("repo")
73 + type = match.group("type")
74 + number = match.group("number")
75 + # Title and fragment could be None
76 + title = match.group("title") or ""
77 + fragment = match.group("fragment") or ""
78 +
79 + # Use default titles for raw links
80 + if not title:
81 + title = TITLES[type]
82 + if "vllm-project" not in repo:
83 + title += repo
84 + title += f"#{number}"
85 +
86 + url = f"https://github.com/{repo}/{type}/{number}{fragment}"
87 + return f"[{gh_icon} {title}]({url})"
88 +
89 + markdown = relative_link.sub(replace_relative_link, markdown)
90 + markdown = github_link.sub(replace_github_link, markdown)
91 +
92 + if "interface" in str(page.file.abs_src_path):
93 + print(markdown)
94
95 return markdown
docs/models/extensions/runai_model_streamer.md
+1 -1
@@ -82,7 +82,7 @@ vllm serve /path/to/sharded/model \
82 --model-loader-extra-config '{"pattern":"custom-model-rank-{rank}-part-{part}.safetensors"}'
83 ```
84
85 -To create sharded model files, you can use the script provided in <gh-file:examples/offline_inference/save_sharded_state.py>. This script demonstrates how to save a model in the sharded format that is compatible with the Run:ai Model Streamer sharded loader.
85 +To create sharded model files, you can use the script provided in [examples/offline_inference/save_sharded_state.py](../../../examples/offline_inference/save_sharded_state.py). This script demonstrates how to save a model in the sharded format that is compatible with the Run:ai Model Streamer sharded loader.
86
87 The sharded loader supports all the same tunable parameters as the regular Run:ai Model Streamer, including `concurrency` and `memory_limit`. These can be configured in the same way:
88
docs/models/generative_models.md
+2 -2
@@ -59,7 +59,7 @@ for output in outputs:
59 By default, vLLM will use sampling parameters recommended by model creator by applying the `generation_config.json` from the huggingface model repository if it exists. In most cases, this will provide you with the best results by default if [SamplingParams][vllm.SamplingParams] is not specified.
60
61 However, if vLLM's default sampling parameters are preferred, please pass `generation_config="vllm"` when creating the [LLM][vllm.LLM] instance.
62 -A code example can be found here: <gh-file:examples/offline_inference/basic/basic.py>
62 +A code example can be found here: [examples/offline_inference/basic/basic.py](../../examples/offline_inference/basic/basic.py)
63
64 ### `LLM.beam_search`
65
@@ -121,7 +121,7 @@ and automatically applies the model's [chat template](https://huggingface.co/doc
121 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
122 ```
123
124 -A code example can be found here: <gh-file:examples/offline_inference/basic/chat.py>
124 +A code example can be found here: [examples/offline_inference/basic/chat.py](../../examples/offline_inference/basic/chat.py)
125
126 If the model doesn't have a chat template or you want to specify another one,
127 you can explicitly pass a chat template:
docs/models/pooling_models.md
+7 -7
@@ -9,7 +9,7 @@ before returning them.
9 !!! note
10 We currently support pooling models primarily as a matter of convenience. This is not guaranteed to have any performance improvement over using HF Transformers / Sentence Transformers directly.
11
12 - We are now planning to optimize pooling models in vLLM. Please comment on <gh-issue:21796> if you have any suggestions!
12 + We are now planning to optimize pooling models in vLLM. Please comment on <https://github.com/vllm-project/vllm/issues/21796> if you have any suggestions!
13
14 ## Configuration
15
@@ -98,7 +98,7 @@ embeds = output.outputs.embedding
98 print(f"Embeddings: {embeds!r} (size={len(embeds)})")
99 ```
100
101 -A code example can be found here: <gh-file:examples/offline_inference/basic/embed.py>
101 +A code example can be found here: [examples/offline_inference/basic/embed.py](../../examples/offline_inference/basic/embed.py)
102
103 ### `LLM.classify`
104
@@ -115,7 +115,7 @@ probs = output.outputs.probs
115 print(f"Class Probabilities: {probs!r} (size={len(probs)})")
116 ```
117
118 -A code example can be found here: <gh-file:examples/offline_inference/basic/classify.py>
118 +A code example can be found here: [examples/offline_inference/basic/classify.py](../../examples/offline_inference/basic/classify.py)
119
120 ### `LLM.score`
121
@@ -139,7 +139,7 @@ score = output.outputs.score
139 print(f"Score: {score}")
140 ```
141
142 -A code example can be found here: <gh-file:examples/offline_inference/basic/score.py>
142 +A code example can be found here: [examples/offline_inference/basic/score.py](../../examples/offline_inference/basic/score.py)
143
144 ### `LLM.reward`
145
@@ -156,7 +156,7 @@ data = output.outputs.data
156 print(f"Data: {data!r}")
157 ```
158
159 -A code example can be found here: <gh-file:examples/offline_inference/basic/reward.py>
159 +A code example can be found here: [examples/offline_inference/basic/reward.py](../../examples/offline_inference/basic/reward.py)
160
161 ### `LLM.encode`
162
@@ -234,7 +234,7 @@ outputs = llm.embed(
234 print(outputs[0].outputs)
235 ```
236
237 -A code example can be found here: <gh-file:examples/offline_inference/pooling/embed_matryoshka_fy.py>
237 +A code example can be found here: [examples/offline_inference/pooling/embed_matryoshka_fy.py](../../examples/offline_inference/pooling/embed_matryoshka_fy.py)
238
239 ### Online Inference
240
@@ -264,4 +264,4 @@ Expected output:
264 {"id":"embd-5c21fc9a5c9d4384a1b021daccaf9f64","object":"list","created":1745476417,"model":"jinaai/jina-embeddings-v3","data":[{"index":0,"object":"embedding","embedding":[-0.3828125,-0.1357421875,0.03759765625,0.125,0.21875,0.09521484375,-0.003662109375,0.1591796875,-0.130859375,-0.0869140625,-0.1982421875,0.1689453125,-0.220703125,0.1728515625,-0.2275390625,-0.0712890625,-0.162109375,-0.283203125,-0.055419921875,-0.0693359375,0.031982421875,-0.04052734375,-0.2734375,0.1826171875,-0.091796875,0.220703125,0.37890625,-0.0888671875,-0.12890625,-0.021484375,-0.0091552734375,0.23046875]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0,"prompt_tokens_details":null}}
265 ```
266
267 -An OpenAI client example can be found here: <gh-file:examples/online_serving/pooling/openai_embedding_matryoshka_fy.py>
267 +An OpenAI client example can be found here: [examples/online_serving/pooling/openai_embedding_matryoshka_fy.py](../../examples/online_serving/pooling/openai_embedding_matryoshka_fy.py)
docs/models/supported_models.md
+6 -6
@@ -9,7 +9,7 @@ Alongside each architecture, we include some popular models that use it.
9
10 ### vLLM
11
12 -If vLLM natively supports a model, its implementation can be found in <gh-file:vllm/model_executor/models>.
12 +If vLLM natively supports a model, its implementation can be found in [vllm/model_executor/models](../../vllm/model_executor/models).
13
14 These models are what we list in [supported-text-models][supported-text-models] and [supported-mm-models][supported-mm-models].
15
@@ -116,7 +116,7 @@ Here is what happens in the background when this model is loaded:
116
117 1. The config is loaded.
118 2. `MyModel` Python class is loaded from the `auto_map` in config, and we check that the model `is_backend_compatible()`.
119 -3. `MyModel` is loaded into one of the Transformers backend classes in <gh-file:vllm/model_executor/models/transformers.py> which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.
119 +3. `MyModel` is loaded into one of the Transformers backend classes in [vllm/model_executor/models/transformers.py](../../vllm/model_executor/models/transformers.py) which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.
120
121 That's it!
122
@@ -543,7 +543,7 @@ These models primarily support the [`LLM.score`](./pooling_models.md#llmscore) A
543 ```
544
545 !!! note
546 - Load the official original `Qwen3 Reranker` by using the following command. More information can be found at: <gh-file:examples/offline_inference/pooling/qwen3_reranker.py>.
546 + Load the official original `Qwen3 Reranker` by using the following command. More information can be found at: [examples/offline_inference/pooling/qwen3_reranker.py](../../examples/offline_inference/pooling/qwen3_reranker.py).
547
548 ```bash
549 vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'
@@ -581,7 +581,7 @@ These models primarily support the [`LLM.encode`](./pooling_models.md#llmencode)
581 | `ModernBertForTokenClassification` | ModernBERT-based | `disham993/electrical-ner-ModernBERT-base` | | |
582
583 !!! note
584 - Named Entity Recognition (NER) usage, please refer to <gh-file:examples/offline_inference/pooling/ner.py>, <gh-file:examples/online_serving/pooling/ner_client.py>.
584 + Named Entity Recognition (NER) usage, please refer to [examples/offline_inference/pooling/ner.py](../../examples/offline_inference/pooling/ner.py), [examples/online_serving/pooling/ner_client.py](../../examples/online_serving/pooling/ner_client.py).
585
586 [](){ #supported-mm-models }
587
@@ -776,7 +776,7 @@ Some models are supported only via the [Transformers backend](#transformers). Th
776
777 !!! note
778 The official `openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork (`HwwwH/MiniCPM-V-2`) for now.
779 - For more details, please see: <gh-pr:4087#issuecomment-2250397630>
779 + For more details, please see: <https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630>
780
781 !!! warning
782 Our PaliGemma implementations have the same problem as Gemma 3 (see above) for both V0 and V1.
@@ -856,5 +856,5 @@ We have the following levels of testing for models:
856
857 1. **Strict Consistency**: We compare the output of the model with the output of the model in the HuggingFace Transformers library under greedy decoding. This is the most stringent test. Please refer to [models tests](https://github.com/vllm-project/vllm/blob/main/tests/models) for the models that have passed this test.
858 2. **Output Sensibility**: We check if the output of the model is sensible and coherent, by measuring the perplexity of the output and checking for any obvious errors. This is a less stringent test.
859 -3. **Runtime Functionality**: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to [functionality tests](gh-dir:tests) and [examples](gh-dir:examples) for the models that have passed this test.
859 +3. **Runtime Functionality**: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to [functionality tests](../../tests) and [examples](../../examples) for the models that have passed this test.
860 4. **Community Feedback**: We rely on the community to provide feedback on the models. If a model is broken or not working as expected, we encourage users to raise issues to report it or open pull requests to fix it. The rest of the models fall under this category.
docs/serving/data_parallel_deployment.md
+1 -1
@@ -16,7 +16,7 @@ For MoE models, when any requests are in progress in any rank, we must ensure th
16
17 In all cases, it is beneficial to load-balance requests between DP ranks. For online deployments, this balancing can be optimized by taking into account the state of each DP engine - in particular its currently scheduled and waiting (queued) requests, and KV cache state. Each DP engine has an independent KV cache, and the benefit of prefix caching can be maximized by directing prompts intelligently.
18
19 -This document focuses on online deployments (with the API server). DP + EP is also supported for offline usage (via the LLM class), for an example see <gh-file:examples/offline_inference/data_parallel.py>.
19 +This document focuses on online deployments (with the API server). DP + EP is also supported for offline usage (via the LLM class), for an example see [examples/offline_inference/data_parallel.py](../../examples/offline_inference/data_parallel.py).
20
21 There are two distinct modes supported for online deployments - self-contained with internal load balancing, or externally per-rank process deployment and load balancing.
22
docs/serving/distributed_troubleshooting.md
+2 -2
@@ -4,11 +4,11 @@ For general troubleshooting, see [Troubleshooting](../usage/troubleshooting.md).
4
5 ## Verify inter-node GPU communication
6
7 -After you start the Ray cluster, verify GPU-to-GPU communication across nodes. Proper configuration can be non-trivial. For more information, see [troubleshooting script][troubleshooting-incorrect-hardware-driver]. If you need additional environment variables for communication configuration, append them to <gh-file:examples/online_serving/run_cluster.sh>, for example `-e NCCL_SOCKET_IFNAME=eth0`. Setting environment variables during cluster creation is recommended because the variables propagate to all nodes. In contrast, setting environment variables in the shell affects only the local node. For more information, see <gh-issue:6803>.
7 +After you start the Ray cluster, verify GPU-to-GPU communication across nodes. Proper configuration can be non-trivial. For more information, see [troubleshooting script][troubleshooting-incorrect-hardware-driver]. If you need additional environment variables for communication configuration, append them to [examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh), for example `-e NCCL_SOCKET_IFNAME=eth0`. Setting environment variables during cluster creation is recommended because the variables propagate to all nodes. In contrast, setting environment variables in the shell affects only the local node. For more information, see <https://github.com/vllm-project/vllm/issues/6803>.
8
9 ## No available node types can fulfill resource request
10
11 -The error message `Error: No available node types can fulfill resource request` can appear even when the cluster has enough GPUs. The issue often occurs when nodes have multiple IP addresses and vLLM can't select the correct one. Ensure that vLLM and Ray use the same IP address by setting `VLLM_HOST_IP` in <gh-file:examples/online_serving/run_cluster.sh> (with a different value on each node). Use `ray status` and `ray list nodes` to verify the chosen IP address. For more information, see <gh-issue:7815>.
11 +The error message `Error: No available node types can fulfill resource request` can appear even when the cluster has enough GPUs. The issue often occurs when nodes have multiple IP addresses and vLLM can't select the correct one. Ensure that vLLM and Ray use the same IP address by setting `VLLM_HOST_IP` in [examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh) (with a different value on each node). Use `ray status` and `ray list nodes` to verify the chosen IP address. For more information, see <https://github.com/vllm-project/vllm/issues/7815>.
12
13 ## Ray observability
14
docs/serving/expert_parallel_deployment.md
+3 -3
@@ -8,9 +8,9 @@ EP is typically coupled with Data Parallelism (DP). While DP can be used indepen
8
9 Before using EP, you need to install the necessary dependencies. We are actively working on making this easier in the future:
10
11 -1. **Install DeepEP and pplx-kernels**: Set up host environment following vLLM's guide for EP kernels [here](gh-file:tools/ep_kernels).
11 +1. **Install DeepEP and pplx-kernels**: Set up host environment following vLLM's guide for EP kernels [here](../../tools/ep_kernels).
12 2. **Install DeepGEMM library**: Follow the [official instructions](https://github.com/deepseek-ai/DeepGEMM#installation).
13 -3. **For disaggregated serving**: Install `gdrcopy` by running the [`install_gdrcopy.sh`](gh-file:tools/install_gdrcopy.sh) script (e.g., `install_gdrcopy.sh "${GDRCOPY_OS_VERSION}" "12.8" "x64"`). You can find available OS versions [here](https://developer.download.nvidia.com/compute/redist/gdrcopy/CUDA%2012.8/).
13 +3. **For disaggregated serving**: Install `gdrcopy` by running the [`install_gdrcopy.sh`](../../tools/install_gdrcopy.sh) script (e.g., `install_gdrcopy.sh "${GDRCOPY_OS_VERSION}" "12.8" "x64"`). You can find available OS versions [here](https://developer.download.nvidia.com/compute/redist/gdrcopy/CUDA%2012.8/).
14
15 ### Backend Selection Guide
16
@@ -195,7 +195,7 @@ For production deployments requiring strict SLA guarantees for time-to-first-tok
195
196 ### Setup Steps
197
198 -1. **Install gdrcopy/ucx/nixl**: For maximum performance, run the [install_gdrcopy.sh](gh-file:tools/install_gdrcopy.sh) script to install `gdrcopy` (e.g., `install_gdrcopy.sh "${GDRCOPY_OS_VERSION}" "12.8" "x64"`). You can find available OS versions [here](https://developer.download.nvidia.com/compute/redist/gdrcopy/CUDA%2012.8/). If `gdrcopy` is not installed, things will still work with a plain `pip install nixl`, just with lower performance. `nixl` and `ucx` are installed as dependencies via pip. For non-cuda platform to install nixl with non-cuda UCX build, run the [install_nixl_from_source_ubuntu.py](gh-file:tools/install_nixl_from_source_ubuntu.py) script.
198 +1. **Install gdrcopy/ucx/nixl**: For maximum performance, run the [install_gdrcopy.sh](../../tools/install_gdrcopy.sh) script to install `gdrcopy` (e.g., `install_gdrcopy.sh "${GDRCOPY_OS_VERSION}" "12.8" "x64"`). You can find available OS versions [here](https://developer.download.nvidia.com/compute/redist/gdrcopy/CUDA%2012.8/). If `gdrcopy` is not installed, things will still work with a plain `pip install nixl`, just with lower performance. `nixl` and `ucx` are installed as dependencies via pip. For non-cuda platform to install nixl with non-cuda UCX build, run the [install_nixl_from_source_ubuntu.py](../../tools/install_nixl_from_source_ubuntu.py) script.
199
200 2. **Configure Both Instances**: Add this flag to both prefill and decode instances `--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}`. Noted, you may also specify one or multiple NIXL_Backend. Such as: `--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'`
201
docs/serving/openai_compatible_server.md
+15 -15
@@ -92,7 +92,7 @@ and all chat requests will error.
92 vllm serve <model> --chat-template ./path-to-chat-template.jinja
93 ```
94
95 -vLLM community provides a set of chat templates for popular models. You can find them under the <gh-dir:examples> directory.
95 +vLLM community provides a set of chat templates for popular models. You can find them under the [examples](../../examples) directory.
96
97 With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
98 both a `type` and a `text` field. An example is provided below:
@@ -181,7 +181,7 @@ with `--enable-request-id-headers`.
181 Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
182 you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
183
184 -Code example: <gh-file:examples/online_serving/openai_completion_client.py>
184 +Code example: [examples/online_serving/openai_completion_client.py](../../examples/online_serving/openai_completion_client.py)
185
186 #### Extra parameters
187
@@ -214,7 +214,7 @@ see our [Multimodal Inputs](../features/multimodal_inputs.md) guide for more inf
214
215 - *Note: `image_url.detail` parameter is not supported.*
216
217 -Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
217 +Code example: [examples/online_serving/openai_chat_completion_client.py](../../examples/online_serving/openai_chat_completion_client.py)
218
219 #### Extra parameters
220
@@ -241,7 +241,7 @@ The following extra parameters are supported:
241 Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
242 you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
243
244 -Code example: <gh-file:examples/online_serving/pooling/openai_embedding_client.py>
244 +Code example: [examples/online_serving/pooling/openai_embedding_client.py](../../examples/online_serving/pooling/openai_embedding_client.py)
245
246 If the model has a [chat template][chat-template], you can replace `inputs` with a list of `messages` (same schema as [Chat API][chat-api])
247 which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations:
@@ -289,7 +289,7 @@ and passing a list of `messages` in the request. Refer to the examples below for
289 to run this model in embedding mode instead of text generation mode.
290
291 The custom chat template is completely different from the original one for this model,
292 - and can be found here: <gh-file:examples/template_vlm2vec_phi3v.jinja>
292 + and can be found here: [examples/template_vlm2vec_phi3v.jinja](../../examples/template_vlm2vec_phi3v.jinja)
293
294 Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
295
@@ -336,13 +336,13 @@ and passing a list of `messages` in the request. Refer to the examples below for
336 Like with VLM2Vec, we have to explicitly pass `--runner pooling`.
337
338 Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled
339 - by a custom chat template: <gh-file:examples/template_dse_qwen2_vl.jinja>
339 + by a custom chat template: [examples/template_dse_qwen2_vl.jinja](../../examples/template_dse_qwen2_vl.jinja)
340
341 !!! important
342 `MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code
343 example below for details.
344
345 -Full example: <gh-file:examples/online_serving/pooling/openai_chat_embedding_client_for_multimodal.py>
345 +Full example: [examples/online_serving/pooling/openai_chat_embedding_client_for_multimodal.py](../../examples/online_serving/pooling/openai_chat_embedding_client_for_multimodal.py)
346
347 #### Extra parameters
348
@@ -379,7 +379,7 @@ you can use the [official OpenAI Python client](https://github.com/openai/openai
379 !!! note
380 To use the Transcriptions API, please install with extra audio dependencies using `pip install vllm[audio]`.
381
382 -Code example: <gh-file:examples/online_serving/openai_transcription_client.py>
382 +Code example: [examples/online_serving/openai_transcription_client.py](../../examples/online_serving/openai_transcription_client.py)
383
384 #### API Enforced Limits
385
@@ -496,7 +496,7 @@ Please mind that the popular `openai/whisper-large-v3-turbo` model does not supp
496 !!! note
497 To use the Translation API, please install with extra audio dependencies using `pip install vllm[audio]`.
498
499 -Code example: <gh-file:examples/online_serving/openai_translation_client.py>
499 +Code example: [examples/online_serving/openai_translation_client.py](../../examples/online_serving/openai_translation_client.py)
500
501 #### Extra Parameters
502
@@ -530,7 +530,7 @@ Our Pooling API encodes input prompts using a [pooling model](../models/pooling_
530
531 The input format is the same as [Embeddings API][embeddings-api], but the output data can contain an arbitrary nested list, not just a 1-D list of floats.
532
533 -Code example: <gh-file:examples/online_serving/pooling/openai_pooling_client.py>
533 +Code example: [examples/online_serving/pooling/openai_pooling_client.py](../../examples/online_serving/pooling/openai_pooling_client.py)
534
535 [](){ #classification-api }
536
@@ -540,7 +540,7 @@ Our Classification API directly supports Hugging Face sequence-classification mo
540
541 We automatically wrap any other transformer via `as_seq_cls_model()`, which pools on the last token, attaches a `RowParallelLinear` head, and applies a softmax to produce per-class probabilities.
542
543 -Code example: <gh-file:examples/online_serving/pooling/openai_classification_client.py>
543 +Code example: [examples/online_serving/pooling/openai_classification_client.py](../../examples/online_serving/pooling/openai_classification_client.py)
544
545 #### Example Requests
546
@@ -658,7 +658,7 @@ Usually, the score for a sentence pair refers to the similarity between two sent
658
659 You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
660
661 -Code example: <gh-file:examples/online_serving/openai_cross_encoder_score.py>
661 +Code example: [examples/online_serving/openai_cross_encoder_score.py](../../examples/online_serving/openai_cross_encoder_score.py)
662
663 #### Single inference
664
@@ -839,7 +839,7 @@ You can pass multi-modal inputs to scoring models by passing `content` including
839 print("Scoring output:", response_json["data"][0]["score"])
840 print("Scoring output:", response_json["data"][1]["score"])
841 ```
842 -Full example: <gh-file:examples/online_serving/openai_cross_encoder_score_for_multimodal.py>
842 +Full example: [examples/online_serving/openai_cross_encoder_score_for_multimodal.py](../../examples/online_serving/openai_cross_encoder_score_for_multimodal.py)
843
844 #### Extra parameters
845
@@ -871,7 +871,7 @@ endpoints are compatible with both [Jina AI's re-rank API interface](https://jin
871 [Cohere's re-rank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with
872 popular open-source tools.
873
874 -Code example: <gh-file:examples/online_serving/pooling/jinaai_rerank_client.py>
874 +Code example: [examples/online_serving/pooling/jinaai_rerank_client.py](../../examples/online_serving/pooling/jinaai_rerank_client.py)
875
876 #### Example Request
877
@@ -949,6 +949,6 @@ Key capabilities:
949 - Scales from a single GPU to a multi-node cluster without code changes.
950 - Provides observability and autoscaling policies through Ray dashboards and metrics.
951
952 -The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: <gh-file:examples/online_serving/ray_serve_deepseek.py>.
952 +The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: [examples/online_serving/ray_serve_deepseek.py](../../examples/online_serving/ray_serve_deepseek.py).
953
954 Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/serving-llms.html).
docs/serving/parallelism_scaling.md
+2 -2
@@ -72,7 +72,7 @@ For details, see the [Ray documentation](https://docs.ray.io/en/latest/index.htm
72
73 ### Ray cluster setup with containers
74
75 -The helper script <gh-file:examples/online_serving/run_cluster.sh> starts containers across nodes and initializes Ray. By default, the script runs Docker without administrative privileges, which prevents access to the GPU performance counters when profiling or tracing. To enable admin privileges, add the `--cap-add=CAP_SYS_ADMIN` flag to the Docker command.
75 +The helper script [examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh) starts containers across nodes and initializes Ray. By default, the script runs Docker without administrative privileges, which prevents access to the GPU performance counters when profiling or tracing. To enable admin privileges, add the `--cap-add=CAP_SYS_ADMIN` flag to the Docker command.
76
77 Choose one node as the head node and run:
78
@@ -132,7 +132,7 @@ vllm serve /path/to/the/model/in/the/container \
132
133 Efficient tensor parallelism requires fast inter-node communication, preferably through high-speed network adapters such as InfiniBand.
134 To set up the cluster to use InfiniBand, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the
135 -<gh-file:examples/online_serving/run_cluster.sh> helper script.
135 +[examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh) helper script.
136 Contact your system administrator for more information about the required flags.
137
138 ## Enabling GPUDirect RDMA
docs/usage/reproducibility.md
+2 -2
@@ -6,7 +6,7 @@ reproducible results:
6 - For V1: Turn off multiprocessing to make the scheduling deterministic by setting `VLLM_ENABLE_V1_MULTIPROCESSING=0`.
7 - For V0: Set the global seed (see below).
8
9 -Example: <gh-file:examples/offline_inference/reproducibility.py>
9 +Example: [examples/offline_inference/reproducibility.py](../../examples/offline_inference/reproducibility.py)
10
11 !!! warning
12
@@ -39,7 +39,7 @@ In V1, the `seed` parameter defaults to `0` which sets the random state for each
39 It is impossible to un-specify a seed for V1 because different workers need to sample the same outputs
40 for workflows such as speculative decoding.
41
42 - For more information, see: <gh-pr:17929>
42 + For more information, see: <https://github.com/vllm-project/vllm/pull/17929>
43
44 ### Locality of random state
45
docs/usage/troubleshooting.md
+5 -5
@@ -24,7 +24,7 @@ If the model is too large to fit in a single GPU, you will get an out-of-memory
24
25 ## Generation quality changed
26
27 -In v0.8.0, the source of default sampling parameters was changed in <gh-pr:12622>. Prior to v0.8.0, the default sampling parameters came from vLLM's set of neutral defaults. From v0.8.0 onwards, the default sampling parameters come from the `generation_config.json` provided by the model creator.
27 +In v0.8.0, the source of default sampling parameters was changed in <https://github.com/vllm-project/vllm/pull/12622>. Prior to v0.8.0, the default sampling parameters came from vLLM's set of neutral defaults. From v0.8.0 onwards, the default sampling parameters come from the `generation_config.json` provided by the model creator.
28
29 In most cases, this should lead to higher quality responses, because the model creator is likely to know which sampling parameters are best for their model. However, in some cases the defaults provided by the model creator can lead to degraded performance.
30
@@ -238,7 +238,7 @@ if __name__ == '__main__':
238
239 ## `torch.compile` Error
240
241 -vLLM heavily depends on `torch.compile` to optimize the model for better performance, which introduces the dependency on the `torch.compile` functionality and the `triton` library. By default, we use `torch.compile` to [optimize some functions](gh-pr:10406) in the model. Before running vLLM, you can check if `torch.compile` is working as expected by running the following script:
241 +vLLM heavily depends on `torch.compile` to optimize the model for better performance, which introduces the dependency on the `torch.compile` functionality and the `triton` library. By default, we use `torch.compile` to [optimize some functions](https://github.com/vllm-project/vllm/pull/10406) in the model. Before running vLLM, you can check if `torch.compile` is working as expected by running the following script:
242
243 ??? code
244
@@ -257,7 +257,7 @@ vLLM heavily depends on `torch.compile` to optimize the model for better perform
257 print(f(x))
258 ```
259
260 -If it raises errors from `torch/_inductor` directory, usually it means you have a custom `triton` library that is not compatible with the version of PyTorch you are using. See <gh-issue:12219> for example.
260 +If it raises errors from `torch/_inductor` directory, usually it means you have a custom `triton` library that is not compatible with the version of PyTorch you are using. See <https://github.com/vllm-project/vllm/issues/12219> for example.
261
262 ## Model failed to be inspected
263
@@ -297,7 +297,7 @@ But you are sure that the model is in the [list of supported models](../models/s
297
298 ## Failed to infer device type
299
300 -If you see an error like `RuntimeError: Failed to infer device type`, it means that vLLM failed to infer the device type of the runtime environment. You can check [the code](gh-file:vllm/platforms/__init__.py) to see how vLLM infers the device type and why it is not working as expected. After [this PR](gh-pr:14195), you can also set the environment variable `VLLM_LOGGING_LEVEL=DEBUG` to see more detailed logs to help debug the issue.
300 +If you see an error like `RuntimeError: Failed to infer device type`, it means that vLLM failed to infer the device type of the runtime environment. You can check [the code](../../vllm/platforms/__init__.py) to see how vLLM infers the device type and why it is not working as expected. After [this PR](https://github.com/vllm-project/vllm/pull/14195), you can also set the environment variable `VLLM_LOGGING_LEVEL=DEBUG` to see more detailed logs to help debug the issue.
301
302 ## NCCL error: unhandled system error during `ncclCommInitRank`
303
@@ -322,6 +322,6 @@ This indicates vLLM failed to initialize the NCCL communicator, possibly due to
322
323 ## Known Issues
324
325 -- In `v0.5.2`, `v0.5.3`, and `v0.5.3.post1`, there is a bug caused by [zmq](https://github.com/zeromq/pyzmq/issues/2000) , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of `vllm` to include the [fix](gh-pr:6759).
325 +- In `v0.5.2`, `v0.5.3`, and `v0.5.3.post1`, there is a bug caused by [zmq](https://github.com/zeromq/pyzmq/issues/2000) , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of `vllm` to include the [fix](https://github.com/vllm-project/vllm/pull/6759).
326 - To address a memory overhead issue in older NCCL versions (see [bug](https://github.com/NVIDIA/nccl/issues/1234)), vLLM versions `>= 0.4.3, <= 0.10.1.1` would set the environment variable `NCCL_CUMEM_ENABLE=0`. External processes connecting to vLLM also needed to set this variable to prevent hangs or crashes. Since the underlying NCCL bug was fixed in NCCL 2.22.3, this override was removed in newer vLLM versions to allow for NCCL performance optimizations.
327 - In some PCIe machines (e.g. machines without NVLink), if you see an error like `transport/shm.cc:590 NCCL WARN Cuda failure 217 'peer access is not supported between these two devices'`, it's likely caused by a driver bug. See [this issue](https://github.com/NVIDIA/nccl/issues/1838) for more details. In that case, you can try to set `NCCL_CUMEM_HOST_ENABLE=0` to disable the feature, or upgrade your driver to the latest version.
docs/usage/usage_stats.md
+1 -1
@@ -6,7 +6,7 @@ A subset of the data, after cleaning and aggregation, will be publicly released
6
7 ## What data is collected?
8
9 -The list of data collected by the latest version of vLLM can be found here: <gh-file:vllm/usage/usage_lib.py>
9 +The list of data collected by the latest version of vLLM can be found here: [vllm/usage/usage_lib.py](../../vllm/usage/usage_lib.py)
10
11 Here is an example as of v0.4.0:
12
docs/usage/v1_guide.md
+9 -9
@@ -2,7 +2,7 @@
2
3 !!! announcement
4
5 - We have started the process of deprecating V0. Please read [RFC #18571](gh-issue:18571) for more details.
5 + We have started the process of deprecating V0. Please read [RFC #18571](https://github.com/vllm-project/vllm/issues/18571) for more details.
6
7 V1 is now enabled by default for all supported use cases, and we will gradually enable it for every use case we plan to support. Please share any feedback on [GitHub](https://github.com/vllm-project/vllm) or in the [vLLM Slack](https://inviter.co/vllm-slack).
8
@@ -94,8 +94,8 @@ See below for the status of models that are not yet supported or have more featu
94
95 The initial basic support is now functional.
96
97 -Later, we will consider using [hidden states processor](gh-issue:12249),
98 -which is based on [global logits processor](gh-pr:13360)
97 +Later, we will consider using [hidden states processor](https://github.com/vllm-project/vllm/issues/12249),
98 +which is based on [global logits processor](https://github.com/vllm-project/vllm/pull/13360)
99 to enable simultaneous generation and embedding using the same engine instance in V1.
100
101 #### Mamba Models
@@ -124,13 +124,13 @@ encoder and decoder (e.g., `BartForConditionalGeneration`,
124 | **Chunked Prefill** | <nobr>🚀 Optimized</nobr> |
125 | **LoRA** | <nobr>🚀 Optimized</nobr> |
126 | **Logprobs Calculation** | <nobr>🟢 Functional</nobr> |
127 -| **FP8 KV Cache** | <nobr>🟢 Functional on Hopper devices (<gh-pr:15191>)</nobr>|
127 +| **FP8 KV Cache** | <nobr>🟢 Functional on Hopper devices (<https://github.com/vllm-project/vllm/pull/15191>)</nobr>|
128 | **Spec Decode** | <nobr>🚀 Optimized</nobr> |
129 -| **Prompt Logprobs with Prefix Caching** | <nobr>🟡 Planned ([RFC #13414](gh-issue:13414))</nobr>|
129 +| **Prompt Logprobs with Prefix Caching** | <nobr>🟡 Planned ([RFC #13414](https://github.com/vllm-project/vllm/issues/13414))</nobr>|
130 | **Structured Output Alternative Backends** | <nobr>🟢 Functional</nobr> |
131 | **Request-level Structured Output Backend** | <nobr>🔴 Deprecated</nobr> |
132 -| **best_of** | <nobr>🔴 Deprecated ([RFC #13361](gh-issue:13361))</nobr>|
133 -| **Per-Request Logits Processors** | <nobr>🔴 Deprecated ([RFC #13360](gh-pr:13360))</nobr> |
132 +| **best_of** | <nobr>🔴 Deprecated ([RFC #13361](https://github.com/vllm-project/vllm/issues/13361))</nobr>|
133 +| **Per-Request Logits Processors** | <nobr>🔴 Deprecated ([RFC #13360](https://github.com/vllm-project/vllm/pull/13360))</nobr> |
134 | **GPU <> CPU KV Cache Swapping** | <nobr>🔴 Deprecated</nobr> |
135
136 !!! note
@@ -168,11 +168,11 @@ As part of the major architectural rework in vLLM V1, several legacy features ha
168
169 ##### Sampling features
170
171 -- **best_of**: This feature has been deprecated due to limited usage. See details at [RFC #13361](gh-issue:13361).
171 +- **best_of**: This feature has been deprecated due to limited usage. See details at [RFC #13361](https://github.com/vllm-project/vllm/issues/13361).
172 - **Per-Request Logits Processors**: In V0, users could pass custom
173 processing functions to adjust logits on a per-request basis. In vLLM V1, this
174 feature has been deprecated. Instead, the design is moving toward supporting **global logits
175 - processors**, a feature the team is actively working on for future releases. See details at [RFC #13360](gh-pr:13360).
175 + processors**, a feature the team is actively working on for future releases. See details at [RFC #13360](https://github.com/vllm-project/vllm/pull/13360).
176
177 ##### KV Cache features
178
examples/offline_inference/pooling/README.md
+1 -1
@@ -14,7 +14,7 @@ python examples/offline_inference/pooling/convert_model_to_seq_cls.py --model_na
14
15 ## Embed jina_embeddings_v3 usage
16
17 -Only text matching task is supported for now. See <gh-pr:16120>
17 +Only text matching task is supported for now. See <https://github.com/vllm-project/vllm/pull/16120>
18
19 ```bash
20 python examples/offline_inference/pooling/embed_jina_embeddings_v3.py