A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Docs] Add comprehensive CLI reference for all large `vllm` subcommands (#22601)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Harry Mellor committed
Aug 11, 2025 at 08:13 UTC
bc1d02ac85d834c98ec2794f1122b269f4c3e45b
20 files changed
+199
-104
docs/.nav.yml
+4
-6
@@ -11,7 +11,7 @@ nav:
11
- Quick Links:
12
- User Guide: usage/README.md
13
- Developer Guide: contributing/README.md
14
- - API Reference: api/summary.md
14
+ - API Reference: api/README.md
15
- CLI Reference: cli/README.md
16
- Timeline:
17
- Roadmap: https://roadmap.vllm.ai
@@ -58,11 +58,9 @@ nav:
58
- CI: contributing/ci
59
- Design Documents: design
60
- API Reference:
61
- - Summary: api/summary.md
62
- - Contents:
63
- - api/vllm/*
64
- - CLI Reference:
65
- - Summary: cli/README.md
61
+ - api/README.md
62
+ - api/vllm/*
63
+ - CLI Reference: cli
64
- Community:
65
- community/*
66
- Blog: https://blog.vllm.ai
docs/api/README.md
renamed
docs/cli/.meta.yml
new
+1
@@ -0,0 +1 @@
1
+toc_depth: 3
\ No newline at end of file
docs/cli/.nav.yml
new
+8
@@ -0,0 +1,8 @@
1
+nav:
2
+ - README.md
3
+ - serve.md
4
+ - chat.md
5
+ - complete.md
6
+ - run-batch.md
7
+ - vllm bench:
8
+ - bench/*.md
docs/cli/README.md
+43
-42
@@ -1,7 +1,3 @@
1
----
2
-toc_depth: 4
3
----
4
-
1
# vLLM CLI Guide
2
3
The vllm command-line tool is used to run and manage vLLM models. You can start by viewing the help message with:
@@ -16,52 +12,48 @@ Available Commands:
12
vllm {chat,complete,serve,bench,collect-env,run-batch}
13
```
14
19
-When passing JSON CLI arguments, the following sets of arguments are equivalent:
20
-
21
-- `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'`
22
-- `--json-arg.key1 value1 --json-arg.key2.key3 value2`
23
-
24
-Additionally, list elements can be passed individually using `+`:
15
+## serve
16
26
-- `--json-arg '{"key4": ["value3", "value4", "value5"]}'`
27
-- `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'`
17
+Starts the vLLM OpenAI Compatible API server.
18
29
-## serve
19
+Start with a model:
20
31
-Start the vLLM OpenAI Compatible API server.
21
+```bash
22
+vllm serve meta-llama/Llama-2-7b-hf
23
+```
24
33
-??? console "Examples"
25
+Specify the port:
26
35
- ```bash
36
- # Start with a model
37
- vllm serve meta-llama/Llama-2-7b-hf
27
+```bash
28
+vllm serve meta-llama/Llama-2-7b-hf --port 8100
29
+```
30
39
- # Specify the port
40
- vllm serve meta-llama/Llama-2-7b-hf --port 8100
31
+Serve over a Unix domain socket:
32
42
- # Serve over a Unix domain socket
43
- vllm serve meta-llama/Llama-2-7b-hf --uds /tmp/vllm.sock
33
+```bash
34
+vllm serve meta-llama/Llama-2-7b-hf --uds /tmp/vllm.sock
35
+```
36
45
- # Check with --help for more options
46
- # To list all groups
47
- vllm serve --help=listgroup
37
+Check with --help for more options:
38
49
- # To view a argument group
50
- vllm serve --help=ModelConfig
39
+```bash
40
+# To list all groups
41
+vllm serve --help=listgroup
42
52
- # To view a single argument
53
- vllm serve --help=max-num-seqs
43
+# To view a argument group
44
+vllm serve --help=ModelConfig
45
55
- # To search by keyword
56
- vllm serve --help=max
46
+# To view a single argument
47
+vllm serve --help=max-num-seqs
48
58
- # To view full help with pager (less/more)
59
- vllm serve --help=page
60
- ```
49
+# To search by keyword
50
+vllm serve --help=max
51
62
-### Options
52
+# To view full help with pager (less/more)
53
+vllm serve --help=page
54
+```
55
64
---8<-- "docs/argparse/serve.md"
56
+See [vllm serve](./serve.md) for the full reference of all available arguments.
57
58
## chat
59
@@ -78,6 +70,8 @@ vllm chat --url http://{vllm-serve-host}:{vllm-serve-port}/v1
70
vllm chat --quick "hi"
71
```
72
73
+See [vllm chat](./chat.md) for the full reference of all available arguments.
74
+
75
## complete
76
77
Generate text completions based on the given prompt via the running API server.
@@ -93,7 +87,7 @@ vllm complete --url http://{vllm-serve-host}:{vllm-serve-port}/v1
87
vllm complete --quick "The future of AI is"
88
```
89
96
-</details>
90
+See [vllm complete](./complete.md) for the full reference of all available arguments.
91
92
## bench
93
@@ -120,6 +114,8 @@ vllm bench latency \
114
--load-format dummy
115
```
116
117
+See [vllm bench latency](./bench/latency.md) for the full reference of all available arguments.
118
+
119
### serve
120
121
Benchmark the online serving throughput.
@@ -134,6 +130,8 @@ vllm bench serve \
130
--num-prompts 5
131
```
132
133
+See [vllm bench serve](./bench/serve.md) for the full reference of all available arguments.
134
+
135
### throughput
136
137
Benchmark offline inference throughput.
@@ -147,6 +145,8 @@ vllm bench throughput \
145
--load-format dummy
146
```
147
148
+See [vllm bench throughput](./bench/throughput.md) for the full reference of all available arguments.
149
+
150
## collect-env
151
152
Start collecting environment information.
@@ -159,24 +159,25 @@ vllm collect-env
159
160
Run batch prompts and write results to file.
161
162
-<details>
163
-<summary>Examples</summary>
162
+Running with a local file:
163
164
```bash
166
-# Running with a local file
165
vllm run-batch \
166
-i offline_inference/openai_batch/openai_example_batch.jsonl \
167
-o results.jsonl \
168
--model meta-llama/Meta-Llama-3-8B-Instruct
169
+```
170
172
-# Using remote file
171
+Using remote file:
172
+
173
+```bash
174
vllm run-batch \
175
-i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/openai_batch/openai_example_batch.jsonl \
176
-o results.jsonl \
177
--model meta-llama/Meta-Llama-3-8B-Instruct
178
```
179
179
-</details>
180
+See [vllm run-batch](./run-batch.md) for the full reference of all available arguments.
181
182
## More Help
183
docs/cli/bench/latency.md
new
+9
@@ -0,0 +1,9 @@
1
+# vllm bench latency
2
+
3
+## JSON CLI Arguments
4
+
5
+--8<-- "docs/cli/json_tip.inc.md"
6
+
7
+## Options
8
+
9
+--8<-- "docs/argparse/bench_latency.md"
docs/cli/bench/serve.md
new
+9
@@ -0,0 +1,9 @@
1
+# vllm bench serve
2
+
3
+## JSON CLI Arguments
4
+
5
+--8<-- "docs/cli/json_tip.inc.md"
6
+
7
+## Options
8
+
9
+--8<-- "docs/argparse/bench_serve.md"
docs/cli/bench/throughput.md
new
+9
@@ -0,0 +1,9 @@
1
+# vllm bench throughput
2
+
3
+## JSON CLI Arguments
4
+
5
+--8<-- "docs/cli/json_tip.inc.md"
6
+
7
+## Options
8
+
9
+--8<-- "docs/argparse/bench_throughput.md"
docs/cli/chat.md
new
+5
@@ -0,0 +1,5 @@
1
+# vllm chat
2
+
3
+## Options
4
+
5
+--8<-- "docs/argparse/chat.md"
docs/cli/complete.md
new
+5
@@ -0,0 +1,5 @@
1
+# vllm complete
2
+
3
+## Options
4
+
5
+--8<-- "docs/argparse/complete.md"
docs/cli/json_tip.inc.md
new
+9
@@ -0,0 +1,9 @@
1
+When passing JSON CLI arguments, the following sets of arguments are equivalent:
2
+
3
+- `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'`
4
+- `--json-arg.key1 value1 --json-arg.key2.key3 value2`
5
+
6
+Additionally, list elements can be passed individually using `+`:
7
+
8
+- `--json-arg '{"key4": ["value3", "value4", "value5"]}'`
9
+- `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'`
\ No newline at end of file
docs/cli/run-batch.md
new
+9
@@ -0,0 +1,9 @@
1
+# vllm run-batch
2
+
3
+## JSON CLI Arguments
4
+
5
+--8<-- "docs/cli/json_tip.inc.md"
6
+
7
+## Options
8
+
9
+--8<-- "docs/argparse/run-batch.md"
docs/cli/serve.md
new
+9
@@ -0,0 +1,9 @@
1
+# vllm serve
2
+
3
+## JSON CLI Arguments
4
+
5
+--8<-- "docs/cli/json_tip.inc.md"
6
+
7
+## Options
8
+
9
+--8<-- "docs/argparse/serve.md"
docs/configuration/engine_args.md
+1
-9
@@ -11,15 +11,7 @@ Engine arguments control the behavior of the vLLM engine.
11
12
The engine argument classes, [EngineArgs][vllm.engine.arg_utils.EngineArgs] and [AsyncEngineArgs][vllm.engine.arg_utils.AsyncEngineArgs], are a combination of the configuration classes defined in [vllm.config][]. Therefore, if you are interested in developer documentation, we recommend looking at these configuration classes as they are the source of truth for types, defaults and docstrings.
13
14
-When passing JSON CLI arguments, the following sets of arguments are equivalent:
15
-
16
-- `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'`
17
-- `--json-arg.key1 value1 --json-arg.key2.key3 value2`
18
-
19
-Additionally, list elements can be passed individually using `+`:
20
-
21
-- `--json-arg '{"key4": ["value3", "value4", "value5"]}'`
22
-- `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'`
14
+--8<-- "docs/cli/json_tip.inc.md"
15
16
## `EngineArgs`
17
docs/mkdocs/hooks/generate_argparse.py
+32
-17
@@ -15,8 +15,14 @@ sys.modules["aiohttp"] = MagicMock()
15
sys.modules["blake3"] = MagicMock()
16
sys.modules["vllm._C"] = MagicMock()
17
18
+from vllm.benchmarks import latency # noqa: E402
19
+from vllm.benchmarks import serve # noqa: E402
20
+from vllm.benchmarks import throughput # noqa: E402
21
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs # noqa: E402
19
-from vllm.entrypoints.openai.cli_args import make_arg_parser # noqa: E402
22
+from vllm.entrypoints.cli.openai import ChatCommand # noqa: E402
23
+from vllm.entrypoints.cli.openai import CompleteCommand # noqa: E402
24
+from vllm.entrypoints.openai import cli_args # noqa: E402
25
+from vllm.entrypoints.openai import run_batch # noqa: E402
26
from vllm.utils import FlexibleArgumentParser # noqa: E402
27
28
logger = logging.getLogger("mkdocs")
@@ -68,7 +74,8 @@ class MarkdownFormatter(HelpFormatter):
74
self._markdown_output.append(
75
f"Possible choices: {metavar}\n\n")
76
71
- self._markdown_output.append(f"{action.help}\n\n")
77
+ if action.help:
78
+ self._markdown_output.append(f"{action.help}\n\n")
79
80
if (default := action.default) != SUPPRESS:
81
self._markdown_output.append(f"Default: `{default}`\n\n")
@@ -78,7 +85,7 @@ class MarkdownFormatter(HelpFormatter):
85
return "".join(self._markdown_output)
86
87
81
-def create_parser(cls, **kwargs) -> FlexibleArgumentParser:
88
+def create_parser(add_cli_args, **kwargs) -> FlexibleArgumentParser:
89
"""Create a parser for the given class with markdown formatting.
90
91
Args:
@@ -88,18 +95,12 @@ def create_parser(cls, **kwargs) -> FlexibleArgumentParser:
95
Returns:
96
FlexibleArgumentParser: A parser with markdown formatting for the class.
97
"""
91
- parser = FlexibleArgumentParser()
98
+ parser = FlexibleArgumentParser(add_json_tip=False)
99
parser.formatter_class = MarkdownFormatter
100
with patch("vllm.config.DeviceConfig.__post_init__"):
94
- return cls.add_cli_args(parser, **kwargs)
95
-
96
-
97
-def create_serve_parser() -> FlexibleArgumentParser:
98
- """Create a parser for the serve command with markdown formatting."""
99
- parser = FlexibleArgumentParser()
100
- parser.formatter_class = lambda prog: MarkdownFormatter(
101
- prog, starting_heading_level=4)
102
- return make_arg_parser(parser)
101
+ _parser = add_cli_args(parser, **kwargs)
102
+ # add_cli_args might be in-place so return parser if _parser is None
103
+ return _parser or parser
104
105
106
def on_startup(command: Literal["build", "gh-deploy", "serve"], dirty: bool):
@@ -113,10 +114,24 @@ def on_startup(command: Literal["build", "gh-deploy", "serve"], dirty: bool):
114
115
# Create parsers to document
116
parsers = {
116
- "engine_args": create_parser(EngineArgs),
117
- "async_engine_args": create_parser(AsyncEngineArgs,
118
- async_args_only=True),
119
- "serve": create_serve_parser(),
117
+ "engine_args":
118
+ create_parser(EngineArgs.add_cli_args),
119
+ "async_engine_args":
120
+ create_parser(AsyncEngineArgs.add_cli_args, async_args_only=True),
121
+ "serve":
122
+ create_parser(cli_args.make_arg_parser),
123
+ "chat":
124
+ create_parser(ChatCommand.add_cli_args),
125
+ "complete":
126
+ create_parser(CompleteCommand.add_cli_args),
127
+ "bench_latency":
128
+ create_parser(latency.add_cli_args),
129
+ "bench_throughput":
130
+ create_parser(throughput.add_cli_args),
131
+ "bench_serve":
132
+ create_parser(serve.add_cli_args),
133
+ "run-batch":
134
+ create_parser(run_batch.make_arg_parser),
135
}
136
137
# Generate documentation for each parser
requirements/docs.txt
+2
@@ -29,3 +29,5 @@ setproctitle
29
torch
30
transformers
31
zmq
32
+uvloop
33
+prometheus-client
vllm/benchmarks/throughput.py
+2
-2
@@ -24,8 +24,6 @@ from vllm.benchmarks.datasets import (AIMODataset, BurstGPTDataset,
24
from vllm.benchmarks.lib.utils import (convert_to_pytorch_benchmark_format,
25
write_to_json)
26
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
27
-from vllm.entrypoints.openai.api_server import (
28
- build_async_engine_client_from_engine_args)
27
from vllm.inputs import TextPrompt, TokensPrompt
28
from vllm.lora.request import LoRARequest
29
from vllm.outputs import RequestOutput
@@ -146,6 +144,8 @@ async def run_vllm_async(
144
disable_detokenize: bool = False,
145
) -> float:
146
from vllm import SamplingParams
147
+ from vllm.entrypoints.openai.api_server import (
148
+ build_async_engine_client_from_engine_args)
149
150
async with build_async_engine_client_from_engine_args(
151
engine_args,
vllm/entrypoints/cli/openai.py
+35
-25
@@ -130,28 +130,33 @@ class ChatCommand(CLISubcommand):
130
conversation.append(response_message) # type: ignore
131
print(output)
132
133
+ @staticmethod
134
+ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
135
+ """Add CLI arguments for the chat command."""
136
+ _add_query_options(parser)
137
+ parser.add_argument(
138
+ "--system-prompt",
139
+ type=str,
140
+ default=None,
141
+ help=("The system prompt to be added to the chat template, "
142
+ "used for models that support system prompts."))
143
+ parser.add_argument("-q",
144
+ "--quick",
145
+ type=str,
146
+ metavar="MESSAGE",
147
+ help=("Send a single prompt as MESSAGE "
148
+ "and print the response, then exit."))
149
+ return parser
150
+
151
def subparser_init(
152
self,
153
subparsers: argparse._SubParsersAction) -> FlexibleArgumentParser:
136
- chat_parser = subparsers.add_parser(
154
+ parser = subparsers.add_parser(
155
"chat",
156
help="Generate chat completions via the running API server.",
157
description="Generate chat completions via the running API server.",
158
usage="vllm chat [options]")
141
- _add_query_options(chat_parser)
142
- chat_parser.add_argument(
143
- "--system-prompt",
144
- type=str,
145
- default=None,
146
- help=("The system prompt to be added to the chat template, "
147
- "used for models that support system prompts."))
148
- chat_parser.add_argument("-q",
149
- "--quick",
150
- type=str,
151
- metavar="MESSAGE",
152
- help=("Send a single prompt as MESSAGE "
153
- "and print the response, then exit."))
154
- return chat_parser
159
+ return ChatCommand.add_cli_args(parser)
160
161
162
class CompleteCommand(CLISubcommand):
@@ -179,25 +184,30 @@ class CompleteCommand(CLISubcommand):
184
output = completion.choices[0].text
185
print(output)
186
187
+ @staticmethod
188
+ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
189
+ """Add CLI arguments for the complete command."""
190
+ _add_query_options(parser)
191
+ parser.add_argument(
192
+ "-q",
193
+ "--quick",
194
+ type=str,
195
+ metavar="PROMPT",
196
+ help=
197
+ "Send a single prompt and print the completion output, then exit.")
198
+ return parser
199
+
200
def subparser_init(
201
self,
202
subparsers: argparse._SubParsersAction) -> FlexibleArgumentParser:
185
- complete_parser = subparsers.add_parser(
203
+ parser = subparsers.add_parser(
204
"complete",
205
help=("Generate text completions based on the given prompt "
206
"via the running API server."),
207
description=("Generate text completions based on the given prompt "
208
"via the running API server."),
209
usage="vllm complete [options]")
192
- _add_query_options(complete_parser)
193
- complete_parser.add_argument(
194
- "-q",
195
- "--quick",
196
- type=str,
197
- metavar="PROMPT",
198
- help=
199
- "Send a single prompt and print the completion output, then exit.")
200
- return complete_parser
210
+ return CompleteCommand.add_cli_args(parser)
211
212
213
def cmd_init() -> list[CLISubcommand]:
vllm/entrypoints/openai/run_batch.py
+3
-2
@@ -20,7 +20,6 @@ from vllm.engine.arg_utils import AsyncEngineArgs, optional_type
20
from vllm.engine.protocol import EngineClient
21
from vllm.entrypoints.logger import RequestLogger
22
# yapf: disable
23
-from vllm.entrypoints.openai.api_server import build_async_engine_client
23
from vllm.entrypoints.openai.protocol import (BatchRequestInput,
24
BatchRequestOutput,
25
BatchResponseData,
@@ -34,7 +33,6 @@ from vllm.entrypoints.openai.serving_models import (BaseModelPath,
33
OpenAIServingModels)
34
from vllm.entrypoints.openai.serving_score import ServingScores
35
from vllm.logger import init_logger
37
-from vllm.usage.usage_lib import UsageContext
36
from vllm.utils import FlexibleArgumentParser, random_uuid
37
from vllm.version import __version__ as VLLM_VERSION
38
@@ -469,6 +467,9 @@ async def run_batch(
467
468
469
async def main(args: Namespace):
470
+ from vllm.entrypoints.openai.api_server import build_async_engine_client
471
+ from vllm.usage.usage_lib import UsageContext
472
+
473
async with build_async_engine_client(
474
args,
475
usage_context=UsageContext.OPENAI_BATCH_RUNNER,
vllm/utils/__init__.py
+4
-1
@@ -1682,6 +1682,8 @@ class FlexibleArgumentParser(ArgumentParser):
1682
# Set the default "formatter_class" to SortedHelpFormatter
1683
if "formatter_class" not in kwargs:
1684
kwargs["formatter_class"] = SortedHelpFormatter
1685
+ # Pop kwarg "add_json_tip" to control whether to add the JSON tip
1686
+ self.add_json_tip = kwargs.pop("add_json_tip", True)
1687
super().__init__(*args, **kwargs)
1688
1689
if sys.version_info < (3, 13):
@@ -1726,7 +1728,8 @@ class FlexibleArgumentParser(ArgumentParser):
1728
def format_help(self) -> str:
1729
# Add tip about JSON arguments to the epilog
1730
epilog = self.epilog or ""
1729
- if not epilog.startswith(FlexibleArgumentParser._json_tip):
1731
+ if (self.add_json_tip
1732
+ and not epilog.startswith(FlexibleArgumentParser._json_tip)):
1733
self.epilog = FlexibleArgumentParser._json_tip + epilog
1734
return super().format_help()
1735