A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 09:45 UTC
Move cli args docs to its own page (#18228) (#18264)
Signed-off-by: Trevor Royer <troyer@redhat.com>
Trevor Royer committed
May 16, 2025 at 19:43 UTC
55f1a468d97fbf9387e577e901b3f290ed8aa15b
4 files changed
+51
-49
docs/source/index.md
+1
@@ -117,6 +117,7 @@ training/rlhf.md
117
118
serving/offline_inference
119
serving/openai_compatible_server
120
+serving/serve_args
121
serving/multimodal_inputs
122
serving/distributed_serving
123
serving/metrics
docs/source/serving/engine_args.md
+2
@@ -7,6 +7,8 @@ Engine arguments control the behavior of the vLLM engine.
7
- For [offline inference](#offline-inference), they are part of the arguments to `LLM` class.
8
- For [online serving](#openai-compatible-server), they are part of the arguments to `vllm serve`.
9
10
+For references to all arguments available from `vllm serve` see the [serve args](#serve-args) documentation.
11
+
12
Below, you can find an explanation of every engine argument:
13
14
<!--- pyml disable-num-lines 7 no-space-in-emphasis -->
docs/source/serving/openai_compatible_server.md
+1
-49
@@ -4,7 +4,7 @@
4
5
vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client.
6
7
-In your terminal, you can [install](../getting_started/installation.md) vLLM, then start the server with the [`vllm serve`](#vllm-serve) command. (You can also use our [Docker](#deployment-docker) image.)
7
+In your terminal, you can [install](../getting_started/installation.md) vLLM, then start the server with the [`vllm serve`](#serve-args) command. (You can also use our [Docker](#deployment-docker) image.)
8
9
```bash
10
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
@@ -168,54 +168,6 @@ completion = client.completions.create(
168
print(completion._request_id)
169
```
170
171
-## CLI Reference
172
-
173
-(vllm-serve)=
174
-
175
-### `vllm serve`
176
-
177
-The `vllm serve` command is used to launch the OpenAI-compatible server.
178
-
179
-:::{tip}
180
-The vast majority of command-line arguments are based on those for offline inference.
181
-
182
-See [here](configuration-options) for some common options.
183
-:::
184
-
185
-:::{argparse}
186
-:module: vllm.entrypoints.openai.cli_args
187
-:func: create_parser_for_docs
188
-:prog: vllm serve
189
-:::
190
-
191
-#### Configuration file
192
-
193
-You can load CLI arguments via a [YAML](https://yaml.org/) config file.
194
-The argument names must be the long form of those outlined [above](#vllm-serve).
195
-
196
-For example:
197
-
198
-```yaml
199
-# config.yaml
200
-
201
-model: meta-llama/Llama-3.1-8B-Instruct
202
-host: "127.0.0.1"
203
-port: 6379
204
-uvicorn-log-level: "info"
205
-```
206
-
207
-To use the above config file:
208
-
209
-```bash
210
-vllm serve --config config.yaml
211
-```
212
-
213
-:::{note}
214
-In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
215
-The order of priorities is `command line > config file values > defaults`.
216
-e.g. `vllm serve SOME_MODEL --config config.yaml`, SOME_MODEL takes precedence over `model` in config file.
217
-:::
218
-
171
## API Reference
172
173
(completions-api)=
docs/source/serving/serve_args.md
new
+47
@@ -0,0 +1,47 @@
1
+(serve-args)=
2
+
3
+# Server Arguments
4
+
5
+The `vllm serve` command is used to launch the OpenAI-compatible server.
6
+
7
+## CLI Arguments
8
+
9
+The following are all arguments available from the `vllm serve` command:
10
+
11
+<!--- pyml disable-num-lines 7 no-space-in-emphasis -->
12
+```{eval-rst}
13
+.. argparse::
14
+ :module: vllm.entrypoints.openai.cli_args
15
+ :func: create_parser_for_docs
16
+ :prog: vllm serve
17
+ :nodefaultconst:
18
+ :markdownhelp:
19
+```
20
+
21
+## Configuration file
22
+
23
+You can load CLI arguments via a [YAML](https://yaml.org/) config file.
24
+The argument names must be the long form of those outlined [above](#serve-args).
25
+
26
+For example:
27
+
28
+```yaml
29
+# config.yaml
30
+
31
+model: meta-llama/Llama-3.1-8B-Instruct
32
+host: "127.0.0.1"
33
+port: 6379
34
+uvicorn-log-level: "info"
35
+```
36
+
37
+To use the above config file:
38
+
39
+```bash
40
+vllm serve --config config.yaml
41
+```
42
+
43
+:::{note}
44
+In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
45
+The order of priorities is `command line > config file values > defaults`.
46
+e.g. `vllm serve SOME_MODEL --config config.yaml`, SOME_MODEL takes precedence over `model` in config file.
47
+:::