A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 04:45 UTC
[doc] improve readability for long commands (#19920)
Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>
Reid committed
Jun 23, 2025 at 22:27 UTC
53243e5c42b006f56e239c4f946b7ca33b7db5cc
3 files changed
+48
-9
docs/contributing/profiling.md
+33
-5
@@ -30,13 +30,21 @@ Refer to <gh-file:examples/offline_inference/simple_profiling.py> for an example
30
#### OpenAI Server
31
32
```bash
33
-VLLM_TORCH_PROFILER_DIR=./vllm_profile python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-70B
33
+VLLM_TORCH_PROFILER_DIR=./vllm_profile \
34
+ python -m vllm.entrypoints.openai.api_server \
35
+ --model meta-llama/Meta-Llama-3-70B
36
```
37
38
benchmark_serving.py:
39
40
```bash
39
-python benchmarks/benchmark_serving.py --backend vllm --model meta-llama/Meta-Llama-3-70B --dataset-name sharegpt --dataset-path sharegpt.json --profile --num-prompts 2
41
+python benchmarks/benchmark_serving.py \
42
+ --backend vllm \
43
+ --model meta-llama/Meta-Llama-3-70B \
44
+ --dataset-name sharegpt \
45
+ --dataset-path sharegpt.json \
46
+ --profile \
47
+ --num-prompts 2
48
```
49
50
## Profile with NVIDIA Nsight Systems
@@ -64,7 +72,16 @@ For basic usage, you can just append `nsys profile -o report.nsys-rep --trace-fo
72
The following is an example using the `benchmarks/benchmark_latency.py` script:
73
74
```bash
67
-nsys profile -o report.nsys-rep --trace-fork-before-exec=true --cuda-graph-trace=node python benchmarks/benchmark_latency.py --model meta-llama/Llama-3.1-8B-Instruct --num-iters-warmup 5 --num-iters 1 --batch-size 16 --input-len 512 --output-len 8
75
+nsys profile -o report.nsys-rep \
76
+ --trace-fork-before-exec=true \
77
+ --cuda-graph-trace=node \
78
+ python benchmarks/benchmark_latency.py \
79
+ --model meta-llama/Llama-3.1-8B-Instruct \
80
+ --num-iters-warmup 5 \
81
+ --num-iters 1 \
82
+ --batch-size 16 \
83
+ --input-len 512 \
84
+ --output-len 8
85
```
86
87
#### OpenAI Server
@@ -73,10 +90,21 @@ To profile the server, you will want to prepend your `vllm serve` command with `
90
91
```bash
92
# server
76
-nsys profile -o report.nsys-rep --trace-fork-before-exec=true --cuda-graph-trace=node --delay 30 --duration 60 vllm serve meta-llama/Llama-3.1-8B-Instruct
93
+nsys profile -o report.nsys-rep \
94
+ --trace-fork-before-exec=true \
95
+ --cuda-graph-trace=node \
96
+ --delay 30 \
97
+ --duration 60 \
98
+ vllm serve meta-llama/Llama-3.1-8B-Instruct
99
100
# client
79
-python benchmarks/benchmark_serving.py --backend vllm --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 1 --dataset-name random --random-input 1024 --random-output 512
101
+python benchmarks/benchmark_serving.py \
102
+ --backend vllm \
103
+ --model meta-llama/Llama-3.1-8B-Instruct \
104
+ --num-prompts 1 \
105
+ --dataset-name random \
106
+ --random-input 1024 \
107
+ --random-output 512
108
```
109
110
In practice, you should set the `--duration` argument to a large value. Whenever you want the server to stop profiling, run:
docs/getting_started/installation/cpu.md
+11
-3
@@ -79,7 +79,9 @@ Currently, there are no pre-built CPU wheels.
79
??? Commands
80
81
```console
82
- $ docker build -f docker/Dockerfile.cpu --tag vllm-cpu-env --target vllm-openai .
82
+ $ docker build -f docker/Dockerfile.cpu \
83
+ --tag vllm-cpu-env \
84
+ --target vllm-openai .
85
86
# Launching OpenAI server
87
$ docker run --rm \
@@ -188,13 +190,19 @@ vllm serve facebook/opt-125m
190
- Tensor Parallel is supported for serving and offline inferencing. In general each NUMA node is treated as one GPU card. Below is the example script to enable Tensor Parallel = 2 for serving:
191
192
```console
191
- VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" vllm serve meta-llama/Llama-2-7b-chat-hf -tp=2 --distributed-executor-backend mp
193
+ VLLM_CPU_KVCACHE_SPACE=40 VLLM_CPU_OMP_THREADS_BIND="0-31|32-63" \
194
+ vllm serve meta-llama/Llama-2-7b-chat-hf \
195
+ -tp=2 \
196
+ --distributed-executor-backend mp
197
```
198
199
or using default auto thread binding:
200
201
```console
197
- VLLM_CPU_KVCACHE_SPACE=40 vllm serve meta-llama/Llama-2-7b-chat-hf -tp=2 --distributed-executor-backend mp
202
+ VLLM_CPU_KVCACHE_SPACE=40 \
203
+ vllm serve meta-llama/Llama-2-7b-chat-hf \
204
+ -tp=2 \
205
+ --distributed-executor-backend mp
206
```
207
208
- For each thread id list in `VLLM_CPU_OMP_THREADS_BIND`, users should guarantee threads in the list belong to a same NUMA node.
docs/usage/troubleshooting.md
+4
-1
@@ -134,7 +134,10 @@ NCCL_DEBUG=TRACE torchrun --nproc-per-node=<number-of-GPUs> test.py
134
If you are testing with multi-nodes, adjust `--nproc-per-node` and `--nnodes` according to your setup and set `MASTER_ADDR` to the correct IP address of the master node, reachable from all nodes. Then, run:
135
136
```console
137
-NCCL_DEBUG=TRACE torchrun --nnodes 2 --nproc-per-node=2 --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR test.py
137
+NCCL_DEBUG=TRACE torchrun --nnodes 2 \
138
+ --nproc-per-node=2 \
139
+ --rdzv_backend=c10d \
140
+ --rdzv_endpoint=$MASTER_ADDR test.py
141
```
142
143
If the script runs successfully, you should see the message `sanity check is successful!`.