A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 08:30 UTC
[Doc] Fix broken links and unlinked docs, add shortcuts to home sidebar (#18627)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
May 24, 2025 at 01:22 UTC
371f7e4ca2a44fbd4a63cd641efb279274a717f4
6 files changed
+16
-11
docs/.nav.yml
+8
-3
@@ -9,8 +9,13 @@ nav:
9
- getting_started/examples/offline_inference
10
- getting_started/examples/online_serving
11
- getting_started/examples/other
12
- - Roadmap: https://roadmap.vllm.ai
13
- - Releases: https://github.com/vllm-project/vllm/releases
12
+ - Quick Links:
13
+ - User Guide: serving/offline_inference.md
14
+ - Developer Guide: contributing/overview.md
15
+ - API Reference: api/README.md
16
+ - Timeline:
17
+ - Roadmap: https://roadmap.vllm.ai
18
+ - Releases: https://github.com/vllm-project/vllm/releases
19
- User Guide:
20
- Inference and Serving:
21
- serving/offline_inference.md
@@ -38,7 +43,7 @@ nav:
43
- contributing/overview.md
44
- glob: contributing/*
45
flatten_single_child_sections: true
41
- - contributing/model
46
+ - Model Implementation: contributing/model
47
- Design Documents:
48
- V0: design
49
- V1: design/v1
docs/contributing/model/tests.md
+2
-2
@@ -33,14 +33,14 @@ These tests compare the model outputs of vLLM against [HF Transformers](https://
33
34
#### Generative models
35
36
-For [generative models][generative-models], there are two levels of correctness tests, as defined in <gh-file:tests/models/utils.py>:
36
+For [generative models](../../models/generative_models.md), there are two levels of correctness tests, as defined in <gh-file:tests/models/utils.py>:
37
38
- Exact correctness (`check_outputs_equal`): The text outputted by vLLM should exactly match the text outputted by HF.
39
- Logprobs similarity (`check_logprobs_close`): The logprobs outputted by vLLM should be in the top-k logprobs outputted by HF, and vice versa.
40
41
#### Pooling models
42
43
-For [pooling models][pooling-models], we simply check the cosine similarity, as defined in <gh-file:tests/models/embedding/utils.py>.
43
+For [pooling models](../../models/pooling_models.md), we simply check the cosine similarity, as defined in <gh-file:tests/models/utils.py>.
44
45
[](){ #mm-processing-tests }
46
docs/features/spec_decode.md
+1
-1
@@ -170,7 +170,7 @@ A variety of speculative models of this type are available on HF hub:
170
## Speculating using EAGLE based draft models
171
172
The following code configures vLLM to use speculative decoding where proposals are generated by
173
-an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found [here](<gh-file:examples/offline_inference/eagle.py>).
173
+an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found [here](gh-file:examples/offline_inference/eagle.py).
174
175
```python
176
from vllm import LLM, SamplingParams
docs/models/supported_models.md
+3
-3
@@ -3,7 +3,7 @@ title: Supported Models
3
---
4
[](){ #supported-models }
5
6
-vLLM supports [generative](generative-models) and [pooling](pooling-models) models across various tasks.
6
+vLLM supports [generative](./generative_models.md) and [pooling](./pooling_models.md) models across various tasks.
7
If a model supports more than one task, you can set the task via the `--task` argument.
8
9
For each task, we list the model architectures that have been implemented in vLLM.
@@ -376,7 +376,7 @@ Specified using `--task generate`.
376
377
### Pooling Models
378
379
-See [this page](pooling-models) for more information on how to use pooling models.
379
+See [this page](./pooling_models.md) for more information on how to use pooling models.
380
381
!!! warning
382
Since some model architectures support both generative and pooling tasks,
@@ -628,7 +628,7 @@ Specified using `--task generate`.
628
629
### Pooling Models
630
631
-See [this page](pooling-models) for more information on how to use pooling models.
631
+See [this page](./pooling_models.md) for more information on how to use pooling models.
632
633
!!! warning
634
Since some model architectures support both generative and pooling tasks,
docs/serving/openai_compatible_server.md
+1
-1
@@ -5,7 +5,7 @@ title: OpenAI-Compatible Server
5
6
vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client.
7
8
-In your terminal, you can [install](../getting_started/installation.md) vLLM, then start the server with the [`vllm serve`][serve-args] command. (You can also use our [Docker][deployment-docker] image.)
8
+In your terminal, you can [install](../getting_started/installation/README.md) vLLM, then start the server with the [`vllm serve`][serve-args] command. (You can also use our [Docker][deployment-docker] image.)
9
10
```bash
11
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
docs/serving/seed_parameter_behavior.md
renamed
+1
-1
@@ -1,4 +1,4 @@
1
-# Seed Parameter Behavior in vLLM
1
+# Seed Parameter Behavior
2
3
## Overview
4