A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Doc][4/N] Reorganize API Reference (#11843)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Jan 8, 2025 at 21:34 UTC
6cd40a5bfed24ef0ceca83b0450be6920d8ca6d4
24 files changed
+38
-67
.buildkite/test-pipeline.yaml
+1
-1
@@ -38,7 +38,7 @@ steps:
38
- pip install -r requirements-docs.txt
39
- SPHINXOPTS=\"-W\" make html
40
# Check API reference (if it fails, you may have missing mock imports)
41
- - grep \"sig sig-object py\" build/html/dev/sampling_params.html
41
+ - grep \"sig sig-object py\" build/html/api/params.html
42
43
- label: Async Engine, Inputs, Utils, Worker Test # 24min
44
fast_check: true
Dockerfile
+2
-2
@@ -2,8 +2,8 @@
2
# to run the OpenAI compatible server.
3
4
# Please update any changes made here to
5
-# docs/source/dev/dockerfile/dockerfile.md and
6
-# docs/source/assets/dev/dockerfile-stages-dependency.png
5
+# docs/source/contributing/dockerfile/dockerfile.md and
6
+# docs/source/assets/contributing/dockerfile-stages-dependency.png
7
8
ARG CUDA_VERSION=12.4.1
9
#################### BASE BUILD IMAGE ####################
docs/source/api/engine/async_llm_engine.md
renamed
docs/source/api/engine/index.md
renamed
docs/source/api/engine/llm_engine.md
renamed
docs/source/api/multimodal/index.md
renamed
-10
@@ -11,18 +11,8 @@ vLLM provides experimental support for multi-modal models through the {mod}`vllm
11
Multi-modal inputs can be passed alongside text and token prompts to [supported models](#supported-mm-models)
12
via the `multi_modal_data` field in {class}`vllm.inputs.PromptType`.
13
14
-Currently, vLLM only has built-in support for image data. You can extend vLLM to process additional modalities
15
-by following [this guide](#adding-multimodal-plugin).
16
-
14
Looking to add your own multi-modal model? Please follow the instructions listed [here](#enabling-multimodal-inputs).
15
19
-## Guides
20
-
21
-```{toctree}
22
-:maxdepth: 1
23
-
24
-adding_multimodal_plugin
25
-```
16
17
## Module Contents
18
docs/source/api/offline_inference/index.md
renamed
docs/source/api/offline_inference/llm.md
renamed
docs/source/api/offline_inference/llm_inputs.md
renamed
docs/source/api/params.md
new
+22
@@ -0,0 +1,22 @@
1
+# Optional Parameters
2
+
3
+Optional parameters for vLLM APIs.
4
+
5
+(sampling-params)=
6
+
7
+## Sampling Parameters
8
+
9
+```{eval-rst}
10
+.. autoclass:: vllm.SamplingParams
11
+ :members:
12
+```
13
+
14
+(pooling-params)=
15
+
16
+## Pooling Parameters
17
+
18
+```{eval-rst}
19
+.. autoclass:: vllm.PoolingParams
20
+ :members:
21
+```
22
+
docs/source/assets/contributing/dockerfile-stages-dependency.png
renamed
docs/source/contributing/dockerfile/dockerfile.md
+1
-1
@@ -17,7 +17,7 @@ The edges of the build graph represent:
17
18
- `RUN --mount=(.\*)from=...` dependencies (with a dotted line and an empty diamond arrow head)
19
20
- > ```{figure} ../../assets/dev/dockerfile-stages-dependency.png
20
+ > ```{figure} /assets/contributing/dockerfile-stages-dependency.png
21
> :align: center
22
> :alt: query
23
> :width: 100%
docs/source/design/arch_overview.md
+1
-1
@@ -53,7 +53,7 @@ for output in outputs:
53
```
54
55
More API details can be found in the {doc}`Offline Inference
56
-</dev/offline_inference/offline_index>` section of the API docs.
56
+</api/offline_inference/index>` section of the API docs.
57
58
The code for the `LLM` class can be found in <gh-file:vllm/entrypoints/llm.py>.
59
docs/source/design/multimodal/adding_multimodal_plugin.md
deleted
-16
@@ -1,16 +0,0 @@
1
-(adding-multimodal-plugin)=
2
-
3
-# Adding a Multimodal Plugin
4
-
5
-This document teaches you how to add a new modality to vLLM.
6
-
7
-Each modality in vLLM is represented by a {class}`~vllm.multimodal.MultiModalPlugin` and registered to {data}`~vllm.multimodal.MULTIMODAL_REGISTRY`.
8
-For vLLM to recognize a new modality type, you have to create a new plugin and then pass it to {meth}`~vllm.multimodal.MultiModalRegistry.register_plugin`.
9
-
10
-The remainder of this document details how to define custom {class}`~vllm.multimodal.MultiModalPlugin` s.
11
-
12
-```{note}
13
-This article is a work in progress.
14
-```
15
-
16
-% TODO: Add more instructions on how to add new plugins once embeddings is in.
docs/source/dev/pooling_params.md
deleted
-6
@@ -1,6 +0,0 @@
1
-# Pooling Parameters
2
-
3
-```{eval-rst}
4
-.. autoclass:: vllm.PoolingParams
5
- :members:
6
-```
docs/source/dev/sampling_params.md
deleted
-6
@@ -1,6 +0,0 @@
1
-# Sampling Parameters
2
-
3
-```{eval-rst}
4
-.. autoclass:: vllm.SamplingParams
5
- :members:
6
-```
docs/source/getting_started/quickstart.md
+1
-1
@@ -42,7 +42,7 @@ The first line of this example imports the classes {class}`~vllm.LLM` and {class
42
from vllm import LLM, SamplingParams
43
```
44
45
-The next section defines a list of input prompts and sampling parameters for text generation. The [sampling temperature](https://arxiv.org/html/2402.05201v1) is set to `0.8` and the [nucleus sampling probability](https://en.wikipedia.org/wiki/Top-p_sampling) is set to `0.95`. You can find more information about the sampling parameters [here](https://docs.vllm.ai/en/stable/dev/sampling_params.html).
45
+The next section defines a list of input prompts and sampling parameters for text generation. The [sampling temperature](https://arxiv.org/html/2402.05201v1) is set to `0.8` and the [nucleus sampling probability](https://en.wikipedia.org/wiki/Top-p_sampling) is set to `0.95`. You can find more information about the sampling parameters [here](#sampling-params).
46
47
```python
48
prompts = [
docs/source/index.md
+4
-5
@@ -137,10 +137,10 @@ community/sponsors
137
:caption: API Reference
138
:maxdepth: 2
139
140
-dev/sampling_params
141
-dev/pooling_params
142
-dev/offline_inference/offline_index
143
-dev/engine/engine_index
140
+api/offline_inference/index
141
+api/engine/index
142
+api/multimodal/index
143
+api/params
144
```
145
146
% Design Documents: Details about vLLM internals
@@ -154,7 +154,6 @@ design/huggingface_integration
154
design/plugin_system
155
design/kernel/paged_attention
156
design/input_processing/model_inputs_index
157
-design/multimodal/multimodal_index
157
design/automatic_prefix_caching
158
design/multiprocessing
159
```
docs/source/serving/offline_inference.md
+1
-1
@@ -23,7 +23,7 @@ The available APIs depend on the type of model that is being run:
23
Please refer to the above pages for more details about each API.
24
25
```{seealso}
26
-[API Reference](/dev/offline_inference/offline_index)
26
+[API Reference](/api/offline_inference/index)
27
```
28
29
## Configuration Options
docs/source/serving/openai_compatible_server.md
+4
-4
@@ -195,7 +195,7 @@ Code example: <gh-file:examples/online_serving/openai_completion_client.py>
195
196
#### Extra parameters
197
198
-The following [sampling parameters (click through to see documentation)](../dev/sampling_params.md) are supported.
198
+The following [sampling parameters](#sampling-params) are supported.
199
200
```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
201
:language: python
@@ -226,7 +226,7 @@ Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
226
227
#### Extra parameters
228
229
-The following [sampling parameters (click through to see documentation)](../dev/sampling_params.md) are supported.
229
+The following [sampling parameters](#sampling-params) are supported.
230
231
```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
232
:language: python
@@ -259,7 +259,7 @@ Code example: <gh-file:examples/online_serving/openai_embedding_client.py>
259
260
#### Extra parameters
261
262
-The following [pooling parameters (click through to see documentation)](../dev/pooling_params.md) are supported.
262
+The following [pooling parameters](#pooling-params) are supported.
263
264
```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
265
:language: python
@@ -447,7 +447,7 @@ Response:
447
448
#### Extra parameters
449
450
-The following [pooling parameters (click through to see documentation)](../dev/pooling_params.md) are supported.
450
+The following [pooling parameters](#pooling-params) are supported.
451
452
```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
453
:language: python
vllm/multimodal/base.py
-3
@@ -49,9 +49,6 @@ class MultiModalPlugin(ABC):
49
process the same data differently). This registry is in turn used by
50
:class:`~MultiModalRegistry` which acts at a higher level
51
(i.e., the modality of the data).
52
-
53
- See also:
54
- :ref:`adding-multimodal-plugin`
52
"""
53
54
def __init__(self) -> None:
vllm/multimodal/inputs.py
-6
@@ -99,12 +99,6 @@ class MultiModalDataBuiltins(TypedDict, total=False):
99
MultiModalDataDict: TypeAlias = Mapping[str, ModalityData[Any]]
100
"""
101
A dictionary containing an entry for each modality type to input.
102
-
103
-Note:
104
- This dictionary also accepts modality keys defined outside
105
- :class:`MultiModalDataBuiltins` as long as a customized plugin
106
- is registered through the :class:`~vllm.multimodal.MULTIMODAL_REGISTRY`.
107
- Read more on that :ref:`here <adding-multimodal-plugin>`.
102
"""
103
104
vllm/multimodal/registry.py
-3
@@ -125,9 +125,6 @@ class MultiModalRegistry:
125
def register_plugin(self, plugin: MultiModalPlugin) -> None:
126
"""
127
Register a multi-modal plugin so it can be recognized by vLLM.
128
-
129
- See also:
130
- :ref:`adding-multimodal-plugin`
128
"""
129
data_type_key = plugin.get_data_key()
130
vllm/pooling_params.py
+1
-1
@@ -7,7 +7,7 @@ class PoolingParams(
7
msgspec.Struct,
8
omit_defaults=True, # type: ignore[call-arg]
9
array_like=True): # type: ignore[call-arg]
10
- """Pooling parameters for embeddings API.
10
+ """API parameters for pooling models. This is currently a placeholder.
11
12
Attributes:
13
additional_data: Any additional data needed for pooling.