A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
[Doc] Add API reference for offline inference (#4710)
Cyrus Leung committed
May 14, 2024 at 08:47 UTC
4bfa7e7f75eb5b1a397c93aeea1dea1afa867b2a
4 files changed
+17
-5
docs/source/index.rst
+7
-1
@@ -67,6 +67,13 @@ Documentation
67
getting_started/quickstart
68
getting_started/examples/examples_index
69
70
+.. toctree::
71
+ :maxdepth: 1
72
+ :caption: Offline Inference
73
+
74
+ offline_inference/llm
75
+ offline_inference/sampling_params
76
+
77
.. toctree::
78
:maxdepth: 1
79
:caption: Serving
@@ -101,7 +108,6 @@ Documentation
108
:maxdepth: 2
109
:caption: Developer Documentation
110
104
- dev/sampling_params
111
dev/engine/engine_index
112
dev/kernel/paged_attention
113
dev/dockerfile/dockerfile
docs/source/offline_inference/llm.rst
new
+6
@@ -0,0 +1,6 @@
1
+LLM Class
2
+==========
3
+
4
+.. autoclass:: vllm.LLM
5
+ :members:
6
+ :show-inheritance:
docs/source/offline_inference/sampling_params.rst
renamed
+2
-2
@@ -1,5 +1,5 @@
1
-Sampling Params
2
-===============
1
+Sampling Parameters
2
+===================
3
4
.. autoclass:: vllm.SamplingParams
5
:members:
docs/source/serving/openai_compatible_server.md
+2
-2
@@ -48,7 +48,7 @@ completion = client.chat.completions.create(
48
```
49
50
### Extra Parameters for Chat API
51
-The following [sampling parameters (click through to see documentation)](../dev/sampling_params.rst) are supported.
51
+The following [sampling parameters (click through to see documentation)](../offline_inference/sampling_params.rst) are supported.
52
53
```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
54
:language: python
@@ -65,7 +65,7 @@ The following extra parameters are supported:
65
```
66
67
### Extra Parameters for Completions API
68
-The following [sampling parameters (click through to see documentation)](../dev/sampling_params.rst) are supported.
68
+The following [sampling parameters (click through to see documentation)](../offline_inference/sampling_params.rst) are supported.
69
70
```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
71
:language: python