@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 248 lines 6.03 KB
Rendered Raw
1 # vLLM CLI Guide
2
3 The vllm command-line tool is used to run and manage vLLM models. You can start by viewing the help message with:
4
5 ```bash
6 vllm --help
7 ```
8
9 Available Commands:
10
11 ```bash
12 vllm {chat,complete,serve,launch,bench,collect-env,run-batch,preload}
13 ```
14
15 ## serve
16
17 Starts the vLLM OpenAI Compatible API server.
18
19 Start with a model:
20
21 ```bash
22 vllm serve meta-llama/Llama-2-7b-hf
23 ```
24
25 Specify the port:
26
27 ```bash
28 vllm serve meta-llama/Llama-2-7b-hf --port 8100
29 ```
30
31 Serve over a Unix domain socket:
32
33 ```bash
34 vllm serve meta-llama/Llama-2-7b-hf --uds /tmp/vllm.sock
35 ```
36
37 Check with --help for more options:
38
39 ```bash
40 # To list all flags
41 vllm serve --help=all
42
43 # To view an argument group
44 vllm serve --help=ModelConfig
45
46 # To view a single argument
47 vllm serve --help=max-num-seqs
48
49 # To search by keyword or flag name
50 vllm serve --help=max
51 ```
52
53 !!! tip "Human-readable integer arguments"
54 Many integer arguments accept human-readable suffixes for convenience. For example:
55
56 - `1k` = 1,000 (decimal kilo)
57 - `1K` = 1,024 (binary kibibyte)
58 - `1m` = 1,000,000 (decimal mega)
59 - `1M` = 1,048,576 (binary mebibyte)
60 - `1g` / `1G` = 1 billion / 1 gibibyte
61 - `1t` / `1T` = 1 trillion / 1 tebibyte
62
63 Decimal suffixes (`k`, `m`, `g`, `t`) also accept floating point: `25.6k` = 25,600.
64 Binary suffixes (`K`, `M`, `G`, `T`) require integers: `32K` = 32,768.
65
66 Supported arguments include: `--max-model-len`, `--max-num-batched-tokens`, `--max-num-scheduled-tokens`, `--kv-cache-memory-bytes`, `--safetensors-prefetch-block-size`.
67
68 See [vllm serve](./serve.md) for the full reference of all available arguments.
69
70 ## launch
71
72 Launch individual vLLM components.
73
74 ```bash
75 # Launch the rendering server component
76 vllm launch render meta-llama/Llama-3.2-1B-Instruct
77
78 # Inspect all available flags for the render component
79 vllm launch render --help=all
80 ```
81
82 See [vllm launch render](./launch/render.md) for the current launch
83 component reference.
84
85 ## chat
86
87 Generate chat completions via the running API server.
88
89 ```bash
90 # Directly connect to localhost API without arguments
91 vllm chat
92
93 # Specify API url
94 vllm chat --url http://{vllm-serve-host}:{vllm-serve-port}/v1
95
96 # Quick chat with a single prompt
97 vllm chat --quick "hi"
98
99 # Print TTFT and throughput statistics after each response
100 vllm chat --stats
101 ```
102
103 See [vllm chat](./chat.md) for the full reference of all available arguments.
104
105 ## complete
106
107 Generate text completions based on the given prompt via the running API server.
108
109 ```bash
110 # Directly connect to localhost API without arguments
111 vllm complete
112
113 # Specify API url
114 vllm complete --url http://{vllm-serve-host}:{vllm-serve-port}/v1
115
116 # Quick complete with a single prompt
117 vllm complete --quick "The future of AI is"
118
119 # Print TTFT and throughput statistics after each response
120 vllm complete --stats
121 ```
122
123 See [vllm complete](./complete.md) for the full reference of all available arguments.
124
125 ## bench
126
127 Run benchmark tests for latency online serving throughput and offline inference throughput.
128
129 To use benchmark commands, please install with extra dependencies using `pip install vllm[bench]`.
130
131 Available Commands:
132
133 ```bash
134 vllm bench {latency, serve, throughput}
135 ```
136
137 ### latency
138
139 Benchmark the latency of a single batch of requests.
140
141 ```bash
142 vllm bench latency \
143 --model meta-llama/Llama-3.2-1B-Instruct \
144 --input-len 32 \
145 --output-len 1 \
146 --enforce-eager \
147 --load-format dummy
148 ```
149
150 See [vllm bench latency](./bench/latency.md) for the full reference of all available arguments.
151
152 ### serve
153
154 Benchmark the online serving throughput.
155
156 ```bash
157 vllm bench serve \
158 --model meta-llama/Llama-3.2-1B-Instruct \
159 --host server-host \
160 --port server-port \
161 --random-input-len 32 \
162 --random-output-len 4 \
163 --num-prompts 5
164 ```
165
166 See [vllm bench serve](./bench/serve.md) for the full reference of all available arguments.
167
168 ### throughput
169
170 Benchmark offline inference throughput.
171
172 ```bash
173 vllm bench throughput \
174 --model meta-llama/Llama-3.2-1B-Instruct \
175 --input-len 32 \
176 --output-len 1 \
177 --enforce-eager \
178 --load-format dummy
179 ```
180
181 See [vllm bench throughput](./bench/throughput.md) for the full reference of all available arguments.
182
183 ## collect-env
184
185 Start collecting environment information.
186
187 ```bash
188 vllm collect-env
189 ```
190
191 ## run-batch
192
193 Run batch prompts and write results to file.
194
195 Running with a local file:
196
197 ```bash
198 vllm run-batch \
199 -i examples/features/openai_batch/openai_example_batch.jsonl \
200 -o results.jsonl \
201 --model meta-llama/Meta-Llama-3-8B-Instruct
202 ```
203
204 Using remote file:
205
206 ```bash
207 vllm run-batch \
208 -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl \
209 -o results.jsonl \
210 --model meta-llama/Meta-Llama-3-8B-Instruct
211 ```
212
213 See [vllm run-batch](./run-batch.md) for the full reference of all available arguments.
214
215 ## preload
216
217 Launch weight cache daemons (one per GPU) that hold the post-quantized,
218 TP-sharded weights in GPU memory and serve CUDA IPC handles to vLLM engines
219 over a Unix domain socket. Restarting engines then map the weights via
220 zero-copy IPC instead of reloading from disk, enabling fast engine restarts.
221
222 ```bash
223 # Launch one daemon per GPU
224 vllm preload --model meta-llama/Llama-3.2-1B-Instruct --tensor-parallel-size 4
225
226 # Engines then load from the daemons
227 vllm serve meta-llama/Llama-3.2-1B-Instruct --tensor-parallel-size 4 \
228 --load-format ipc_cache
229 ```
230
231 The daemon accepts the standard engine arguments (model, dtype, quantization,
232 tensor-parallel-size, ...) plus `--weight-cache-socket-dir` to override the
233 directory holding the per-GPU Unix sockets, and `--weight-cache-master-port` /
234 `--weight-cache-draft-master-port` to pin the daemon rendezvous ports for
235 multi-node and speculative-decoding setups. Tensor, expert and data
236 parallelism are supported; pipeline parallelism is rejected at launch.
237
238 See [Preload](../features/preload.md) for how it works, cache modes, and
239 limitations, and [vllm preload](./preload.md) for the full reference of all
240 available arguments.
241
242 ## More Help
243
244 For detailed options of any subcommand, use:
245
246 ```bash
247 vllm <subcommand> --help
248 ```