@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 6cd40a5bfe
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

[Doc][4/N] Reorganize API Reference (#11843)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Cyrus Leung committed Jan 8, 2025 at 21:34 UTC 6cd40a5bfed24ef0ceca83b0450be6920d8ca6d4
24 files changed +38 -67
.buildkite/test-pipeline.yaml
+1 -1
@@ -38,7 +38,7 @@ steps:
38 - pip install -r requirements-docs.txt
39 - SPHINXOPTS=\"-W\" make html
40 # Check API reference (if it fails, you may have missing mock imports)
41 - - grep \"sig sig-object py\" build/html/dev/sampling_params.html
41 + - grep \"sig sig-object py\" build/html/api/params.html
42
43 - label: Async Engine, Inputs, Utils, Worker Test # 24min
44 fast_check: true
Dockerfile
+2 -2
@@ -2,8 +2,8 @@
2 # to run the OpenAI compatible server.
3
4 # Please update any changes made here to
5 -# docs/source/dev/dockerfile/dockerfile.md and
6 -# docs/source/assets/dev/dockerfile-stages-dependency.png
5 +# docs/source/contributing/dockerfile/dockerfile.md and
6 +# docs/source/assets/contributing/dockerfile-stages-dependency.png
7
8 ARG CUDA_VERSION=12.4.1
9 #################### BASE BUILD IMAGE ####################
docs/source/api/engine/async_llm_engine.md renamed
docs/source/api/engine/index.md renamed
docs/source/api/engine/llm_engine.md renamed
docs/source/api/multimodal/index.md renamed
-10
@@ -11,18 +11,8 @@ vLLM provides experimental support for multi-modal models through the {mod}`vllm
11 Multi-modal inputs can be passed alongside text and token prompts to [supported models](#supported-mm-models)
12 via the `multi_modal_data` field in {class}`vllm.inputs.PromptType`.
13
14 -Currently, vLLM only has built-in support for image data. You can extend vLLM to process additional modalities
15 -by following [this guide](#adding-multimodal-plugin).
16 -
14 Looking to add your own multi-modal model? Please follow the instructions listed [here](#enabling-multimodal-inputs).
15
19 -## Guides
20 -
21 -```{toctree}
22 -:maxdepth: 1
23 -
24 -adding_multimodal_plugin
25 -```
16
17 ## Module Contents
18
docs/source/api/offline_inference/index.md renamed
docs/source/api/offline_inference/llm.md renamed
docs/source/api/offline_inference/llm_inputs.md renamed
docs/source/api/params.md new
+22
@@ -0,0 +1,22 @@
1 +# Optional Parameters
2 +
3 +Optional parameters for vLLM APIs.
4 +
5 +(sampling-params)=
6 +
7 +## Sampling Parameters
8 +
9 +```{eval-rst}
10 +.. autoclass:: vllm.SamplingParams
11 + :members:
12 +```
13 +
14 +(pooling-params)=
15 +
16 +## Pooling Parameters
17 +
18 +```{eval-rst}
19 +.. autoclass:: vllm.PoolingParams
20 + :members:
21 +```
22 +
docs/source/assets/contributing/dockerfile-stages-dependency.png renamed
docs/source/contributing/dockerfile/dockerfile.md
+1 -1
@@ -17,7 +17,7 @@ The edges of the build graph represent:
17
18 - `RUN --mount=(.\*)from=...` dependencies (with a dotted line and an empty diamond arrow head)
19
20 - > ```{figure} ../../assets/dev/dockerfile-stages-dependency.png
20 + > ```{figure} /assets/contributing/dockerfile-stages-dependency.png
21 > :align: center
22 > :alt: query
23 > :width: 100%
docs/source/design/arch_overview.md
+1 -1
@@ -53,7 +53,7 @@ for output in outputs:
53 ```
54
55 More API details can be found in the {doc}`Offline Inference
56 -</dev/offline_inference/offline_index>` section of the API docs.
56 +</api/offline_inference/index>` section of the API docs.
57
58 The code for the `LLM` class can be found in <gh-file:vllm/entrypoints/llm.py>.
59
docs/source/design/multimodal/adding_multimodal_plugin.md deleted
-16
@@ -1,16 +0,0 @@
1 -(adding-multimodal-plugin)=
2 -
3 -# Adding a Multimodal Plugin
4 -
5 -This document teaches you how to add a new modality to vLLM.
6 -
7 -Each modality in vLLM is represented by a {class}`~vllm.multimodal.MultiModalPlugin` and registered to {data}`~vllm.multimodal.MULTIMODAL_REGISTRY`.
8 -For vLLM to recognize a new modality type, you have to create a new plugin and then pass it to {meth}`~vllm.multimodal.MultiModalRegistry.register_plugin`.
9 -
10 -The remainder of this document details how to define custom {class}`~vllm.multimodal.MultiModalPlugin` s.
11 -
12 -```{note}
13 -This article is a work in progress.
14 -```
15 -
16 -% TODO: Add more instructions on how to add new plugins once embeddings is in.
docs/source/dev/pooling_params.md deleted
-6
@@ -1,6 +0,0 @@
1 -# Pooling Parameters
2 -
3 -```{eval-rst}
4 -.. autoclass:: vllm.PoolingParams
5 - :members:
6 -```
docs/source/dev/sampling_params.md deleted
-6
@@ -1,6 +0,0 @@
1 -# Sampling Parameters
2 -
3 -```{eval-rst}
4 -.. autoclass:: vllm.SamplingParams
5 - :members:
6 -```
docs/source/getting_started/quickstart.md
+1 -1
@@ -42,7 +42,7 @@ The first line of this example imports the classes {class}`~vllm.LLM` and {class
42 from vllm import LLM, SamplingParams
43 ```
44
45 -The next section defines a list of input prompts and sampling parameters for text generation. The [sampling temperature](https://arxiv.org/html/2402.05201v1) is set to `0.8` and the [nucleus sampling probability](https://en.wikipedia.org/wiki/Top-p_sampling) is set to `0.95`. You can find more information about the sampling parameters [here](https://docs.vllm.ai/en/stable/dev/sampling_params.html).
45 +The next section defines a list of input prompts and sampling parameters for text generation. The [sampling temperature](https://arxiv.org/html/2402.05201v1) is set to `0.8` and the [nucleus sampling probability](https://en.wikipedia.org/wiki/Top-p_sampling) is set to `0.95`. You can find more information about the sampling parameters [here](#sampling-params).
46
47 ```python
48 prompts = [
docs/source/index.md
+4 -5
@@ -137,10 +137,10 @@ community/sponsors
137 :caption: API Reference
138 :maxdepth: 2
139
140 -dev/sampling_params
141 -dev/pooling_params
142 -dev/offline_inference/offline_index
143 -dev/engine/engine_index
140 +api/offline_inference/index
141 +api/engine/index
142 +api/multimodal/index
143 +api/params
144 ```
145
146 % Design Documents: Details about vLLM internals
@@ -154,7 +154,6 @@ design/huggingface_integration
154 design/plugin_system
155 design/kernel/paged_attention
156 design/input_processing/model_inputs_index
157 -design/multimodal/multimodal_index
157 design/automatic_prefix_caching
158 design/multiprocessing
159 ```
docs/source/serving/offline_inference.md
+1 -1
@@ -23,7 +23,7 @@ The available APIs depend on the type of model that is being run:
23 Please refer to the above pages for more details about each API.
24
25 ```{seealso}
26 -[API Reference](/dev/offline_inference/offline_index)
26 +[API Reference](/api/offline_inference/index)
27 ```
28
29 ## Configuration Options
docs/source/serving/openai_compatible_server.md
+4 -4
@@ -195,7 +195,7 @@ Code example: <gh-file:examples/online_serving/openai_completion_client.py>
195
196 #### Extra parameters
197
198 -The following [sampling parameters (click through to see documentation)](../dev/sampling_params.md) are supported.
198 +The following [sampling parameters](#sampling-params) are supported.
199
200 ```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
201 :language: python
@@ -226,7 +226,7 @@ Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
226
227 #### Extra parameters
228
229 -The following [sampling parameters (click through to see documentation)](../dev/sampling_params.md) are supported.
229 +The following [sampling parameters](#sampling-params) are supported.
230
231 ```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
232 :language: python
@@ -259,7 +259,7 @@ Code example: <gh-file:examples/online_serving/openai_embedding_client.py>
259
260 #### Extra parameters
261
262 -The following [pooling parameters (click through to see documentation)](../dev/pooling_params.md) are supported.
262 +The following [pooling parameters](#pooling-params) are supported.
263
264 ```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
265 :language: python
@@ -447,7 +447,7 @@ Response:
447
448 #### Extra parameters
449
450 -The following [pooling parameters (click through to see documentation)](../dev/pooling_params.md) are supported.
450 +The following [pooling parameters](#pooling-params) are supported.
451
452 ```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
453 :language: python
vllm/multimodal/base.py
-3
@@ -49,9 +49,6 @@ class MultiModalPlugin(ABC):
49 process the same data differently). This registry is in turn used by
50 :class:`~MultiModalRegistry` which acts at a higher level
51 (i.e., the modality of the data).
52 -
53 - See also:
54 - :ref:`adding-multimodal-plugin`
52 """
53
54 def __init__(self) -> None:
vllm/multimodal/inputs.py
-6
@@ -99,12 +99,6 @@ class MultiModalDataBuiltins(TypedDict, total=False):
99 MultiModalDataDict: TypeAlias = Mapping[str, ModalityData[Any]]
100 """
101 A dictionary containing an entry for each modality type to input.
102 -
103 -Note:
104 - This dictionary also accepts modality keys defined outside
105 - :class:`MultiModalDataBuiltins` as long as a customized plugin
106 - is registered through the :class:`~vllm.multimodal.MULTIMODAL_REGISTRY`.
107 - Read more on that :ref:`here <adding-multimodal-plugin>`.
102 """
103
104
vllm/multimodal/registry.py
-3
@@ -125,9 +125,6 @@ class MultiModalRegistry:
125 def register_plugin(self, plugin: MultiModalPlugin) -> None:
126 """
127 Register a multi-modal plugin so it can be recognized by vLLM.
128 -
129 - See also:
130 - :ref:`adding-multimodal-plugin`
128 """
129 data_type_key = plugin.get_data_key()
130
vllm/pooling_params.py
+1 -1
@@ -7,7 +7,7 @@ class PoolingParams(
7 msgspec.Struct,
8 omit_defaults=True, # type: ignore[call-arg]
9 array_like=True): # type: ignore[call-arg]
10 - """Pooling parameters for embeddings API.
10 + """API parameters for pooling models. This is currently a placeholder.
11
12 Attributes:
13 additional_data: Any additional data needed for pooling.