@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / b82e0f82cb
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 04:45 UTC

[doc] use MkDocs collapsible blocks - supplement (#19973)

Signed-off-by: reidliu41 <reid201711@gmail.com> Co-authored-by: reidliu41 <reid201711@gmail.com>

Reid committed Jun 23, 2025 at 18:54 UTC b82e0f82cb24bc2cfccbd816a46f535a8ff64eda
3 files changed +290 -264
docs/design/v1/p2p_nccl_connector.md
+188 -168
@@ -61,23 +61,25 @@ To address the above issues, I have designed and developed a local Tensor memory
61
62 # Install vLLM
63
64 -```shell
65 -# Enter the home directory or your working directory.
66 -cd /home
64 +??? Commands
65
68 -# Download the installation package, and I will update the commit-id in time. You can directly copy the command.
69 -wget https://vllm-wheels.s3.us-west-2.amazonaws.com/9112b443a042d8d815880b8780633882ad32b183/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
66 + ```shell
67 + # Enter the home directory or your working directory.
68 + cd /home
69
71 -# Download the code repository.
72 -git clone -b xpyd-v1 https://github.com/Abatom/vllm.git
73 -cd vllm
70 + # Download the installation package, and I will update the commit-id in time. You can directly copy the command.
71 + wget https://vllm-wheels.s3.us-west-2.amazonaws.com/9112b443a042d8d815880b8780633882ad32b183/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
72
75 -# Set the installation package path.
76 -export VLLM_PRECOMPILED_WHEEL_LOCATION=/home/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
73 + # Download the code repository.
74 + git clone -b xpyd-v1 https://github.com/Abatom/vllm.git
75 + cd vllm
76
78 -# installation
79 -pip install -e . -v
80 -```
77 + # Set the installation package path.
78 + export VLLM_PRECOMPILED_WHEEL_LOCATION=/home/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
79 +
80 + # installation
81 + pip install -e . -v
82 + ```
83
84 # Run xPyD
85
@@ -104,83 +106,91 @@ python3 disagg_prefill_proxy_xpyd.py &
106
107 ### Prefill1 (e.g. 10.0.1.2 or 10.0.1.1)
108
107 -```shell
108 -VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=0 vllm serve {your model directory} \
109 - --host 0.0.0.0 \
110 - --port 20005 \
111 - --tensor-parallel-size 1 \
112 - --seed 1024 \
113 - --served-model-name base_model \
114 - --dtype float16 \
115 - --max-model-len 10000 \
116 - --max-num-batched-tokens 10000 \
117 - --max-num-seqs 256 \
118 - --trust-remote-code \
119 - --gpu-memory-utilization 0.9 \
120 - --disable-log-request \
121 - --kv-transfer-config \
122 - '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"21001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20005","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
123 -```
109 +??? Command
110 +
111 + ```shell
112 + VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=0 vllm serve {your model directory} \
113 + --host 0.0.0.0 \
114 + --port 20005 \
115 + --tensor-parallel-size 1 \
116 + --seed 1024 \
117 + --served-model-name base_model \
118 + --dtype float16 \
119 + --max-model-len 10000 \
120 + --max-num-batched-tokens 10000 \
121 + --max-num-seqs 256 \
122 + --trust-remote-code \
123 + --gpu-memory-utilization 0.9 \
124 + --disable-log-request \
125 + --kv-transfer-config \
126 + '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"21001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20005","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
127 + ```
128
129 ### Decode1 (e.g. 10.0.1.3 or 10.0.1.1)
130
127 -```shell
128 -VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=1 vllm serve {your model directory} \
129 - --host 0.0.0.0 \
130 - --port 20009 \
131 - --tensor-parallel-size 1 \
132 - --seed 1024 \
133 - --served-model-name base_model \
134 - --dtype float16 \
135 - --max-model-len 10000 \
136 - --max-num-batched-tokens 10000 \
137 - --max-num-seqs 256 \
138 - --trust-remote-code \
139 - --gpu-memory-utilization 0.7 \
140 - --disable-log-request \
141 - --kv-transfer-config \
142 - '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"22001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20009","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
143 -```
131 +??? Command
132 +
133 + ```shell
134 + VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=1 vllm serve {your model directory} \
135 + --host 0.0.0.0 \
136 + --port 20009 \
137 + --tensor-parallel-size 1 \
138 + --seed 1024 \
139 + --served-model-name base_model \
140 + --dtype float16 \
141 + --max-model-len 10000 \
142 + --max-num-batched-tokens 10000 \
143 + --max-num-seqs 256 \
144 + --trust-remote-code \
145 + --gpu-memory-utilization 0.7 \
146 + --disable-log-request \
147 + --kv-transfer-config \
148 + '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"22001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20009","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
149 + ```
150
151 ### Decode2 (e.g. 10.0.1.4 or 10.0.1.1)
152
147 -```shell
148 -VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=2 vllm serve {your model directory} \
149 - --host 0.0.0.0 \
150 - --port 20003 \
151 - --tensor-parallel-size 1 \
152 - --seed 1024 \
153 - --served-model-name base_model \
154 - --dtype float16 \
155 - --max-model-len 10000 \
156 - --max-num-batched-tokens 10000 \
157 - --max-num-seqs 256 \
158 - --trust-remote-code \
159 - --gpu-memory-utilization 0.7 \
160 - --disable-log-request \
161 - --kv-transfer-config \
162 - '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"23001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20003","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
163 -```
153 +??? Command
154 +
155 + ```shell
156 + VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=2 vllm serve {your model directory} \
157 + --host 0.0.0.0 \
158 + --port 20003 \
159 + --tensor-parallel-size 1 \
160 + --seed 1024 \
161 + --served-model-name base_model \
162 + --dtype float16 \
163 + --max-model-len 10000 \
164 + --max-num-batched-tokens 10000 \
165 + --max-num-seqs 256 \
166 + --trust-remote-code \
167 + --gpu-memory-utilization 0.7 \
168 + --disable-log-request \
169 + --kv-transfer-config \
170 + '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"23001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20003","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
171 + ```
172
173 ### Decode3 (e.g. 10.0.1.5 or 10.0.1.1)
174
167 -```shell
168 -VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=3 vllm serve {your model directory} \
169 - --host 0.0.0.0 \
170 - --port 20008 \
171 - --tensor-parallel-size 1 \
172 - --seed 1024 \
173 - --served-model-name base_model \
174 - --dtype float16 \
175 - --max-model-len 10000 \
176 - --max-num-batched-tokens 10000 \
177 - --max-num-seqs 256 \
178 - --trust-remote-code \
179 - --gpu-memory-utilization 0.7 \
180 - --disable-log-request \
181 - --kv-transfer-config \
182 - '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"24001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20008","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
183 -```
175 +??? Command
176 +
177 + ```shell
178 + VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=3 vllm serve {your model directory} \
179 + --host 0.0.0.0 \
180 + --port 20008 \
181 + --tensor-parallel-size 1 \
182 + --seed 1024 \
183 + --served-model-name base_model \
184 + --dtype float16 \
185 + --max-model-len 10000 \
186 + --max-num-batched-tokens 10000 \
187 + --max-num-seqs 256 \
188 + --trust-remote-code \
189 + --gpu-memory-utilization 0.7 \
190 + --disable-log-request \
191 + --kv-transfer-config \
192 + '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"24001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20008","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
193 + ```
194
195 ## Run 3P1D
196
@@ -193,83 +203,91 @@ python3 disagg_prefill_proxy_xpyd.py &
203
204 ### Prefill1 (e.g. 10.0.1.2 or 10.0.1.1)
205
196 -```shell
197 -VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=0 vllm serve {your model directory} \
198 - --host 0.0.0.0 \
199 - --port 20005 \
200 - --tensor-parallel-size 1 \
201 - --seed 1024 \
202 - --served-model-name base_model \
203 - --dtype float16 \
204 - --max-model-len 10000 \
205 - --max-num-batched-tokens 10000 \
206 - --max-num-seqs 256 \
207 - --trust-remote-code \
208 - --gpu-memory-utilization 0.9 \
209 - --disable-log-request \
210 - --kv-transfer-config \
211 - '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"21001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20005","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
212 -```
206 +??? Command
207 +
208 + ```shell
209 + VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=0 vllm serve {your model directory} \
210 + --host 0.0.0.0 \
211 + --port 20005 \
212 + --tensor-parallel-size 1 \
213 + --seed 1024 \
214 + --served-model-name base_model \
215 + --dtype float16 \
216 + --max-model-len 10000 \
217 + --max-num-batched-tokens 10000 \
218 + --max-num-seqs 256 \
219 + --trust-remote-code \
220 + --gpu-memory-utilization 0.9 \
221 + --disable-log-request \
222 + --kv-transfer-config \
223 + '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"21001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20005","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
224 + ```
225
226 ### Prefill2 (e.g. 10.0.1.3 or 10.0.1.1)
227
216 -```shell
217 -VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=1 vllm serve {your model directory} \
218 - --host 0.0.0.0 \
219 - --port 20009 \
220 - --tensor-parallel-size 1 \
221 - --seed 1024 \
222 - --served-model-name base_model \
223 - --dtype float16 \
224 - --max-model-len 10000 \
225 - --max-num-batched-tokens 10000 \
226 - --max-num-seqs 256 \
227 - --trust-remote-code \
228 - --gpu-memory-utilization 0.9 \
229 - --disable-log-request \
230 - --kv-transfer-config \
231 - '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"22001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20009","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
232 -```
228 +??? Command
229 +
230 + ```shell
231 + VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=1 vllm serve {your model directory} \
232 + --host 0.0.0.0 \
233 + --port 20009 \
234 + --tensor-parallel-size 1 \
235 + --seed 1024 \
236 + --served-model-name base_model \
237 + --dtype float16 \
238 + --max-model-len 10000 \
239 + --max-num-batched-tokens 10000 \
240 + --max-num-seqs 256 \
241 + --trust-remote-code \
242 + --gpu-memory-utilization 0.9 \
243 + --disable-log-request \
244 + --kv-transfer-config \
245 + '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"22001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20009","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
246 + ```
247
248 ### Prefill3 (e.g. 10.0.1.4 or 10.0.1.1)
249
236 -```shell
237 -VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=2 vllm serve {your model directory} \
238 - --host 0.0.0.0 \
239 - --port 20003 \
240 - --tensor-parallel-size 1 \
241 - --seed 1024 \
242 - --served-model-name base_model \
243 - --dtype float16 \
244 - --max-model-len 10000 \
245 - --max-num-batched-tokens 10000 \
246 - --max-num-seqs 256 \
247 - --trust-remote-code \
248 - --gpu-memory-utilization 0.9 \
249 - --disable-log-request \
250 - --kv-transfer-config \
251 - '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"23001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20003","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
252 -```
250 +??? Command
251 +
252 + ```shell
253 + VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=2 vllm serve {your model directory} \
254 + --host 0.0.0.0 \
255 + --port 20003 \
256 + --tensor-parallel-size 1 \
257 + --seed 1024 \
258 + --served-model-name base_model \
259 + --dtype float16 \
260 + --max-model-len 10000 \
261 + --max-num-batched-tokens 10000 \
262 + --max-num-seqs 256 \
263 + --trust-remote-code \
264 + --gpu-memory-utilization 0.9 \
265 + --disable-log-request \
266 + --kv-transfer-config \
267 + '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"23001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20003","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
268 + ```
269
270 ### Decode1 (e.g. 10.0.1.5 or 10.0.1.1)
271
256 -```shell
257 -VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=3 vllm serve {your model directory} \
258 - --host 0.0.0.0 \
259 - --port 20008 \
260 - --tensor-parallel-size 1 \
261 - --seed 1024 \
262 - --served-model-name base_model \
263 - --dtype float16 \
264 - --max-model-len 10000 \
265 - --max-num-batched-tokens 10000 \
266 - --max-num-seqs 256 \
267 - --trust-remote-code \
268 - --gpu-memory-utilization 0.7 \
269 - --disable-log-request \
270 - --kv-transfer-config \
271 - '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"24001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20008","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
272 -```
272 +??? Command
273 +
274 + ```shell
275 + VLLM_USE_V1=1 CUDA_VISIBLE_DEVICES=3 vllm serve {your model directory} \
276 + --host 0.0.0.0 \
277 + --port 20008 \
278 + --tensor-parallel-size 1 \
279 + --seed 1024 \
280 + --served-model-name base_model \
281 + --dtype float16 \
282 + --max-model-len 10000 \
283 + --max-num-batched-tokens 10000 \
284 + --max-num-seqs 256 \
285 + --trust-remote-code \
286 + --gpu-memory-utilization 0.7 \
287 + --disable-log-request \
288 + --kv-transfer-config \
289 + '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"24001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20008","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}' > /var/vllm.log 2>&1 &
290 + ```
291
292 # Single request
293
@@ -286,25 +304,27 @@ curl -X POST -s http://10.0.1.1:10001/v1/completions \
304
305 # Benchmark
306
289 -```shell
290 -python3 benchmark_serving.py \
291 - --backend vllm \
292 - --model base_model \
293 - --tokenizer meta-llama/Llama-3.1-8B-Instruct \
294 - --dataset-name "random" \
295 - --host 10.0.1.1 \
296 - --port 10001 \
297 - --random-input-len 1024 \
298 - --random-output-len 1024 \
299 - --ignore-eos \
300 - --burstiness 100 \
301 - --percentile-metrics "ttft,tpot,itl,e2el" \
302 - --metric-percentiles "90,95,99" \
303 - --seed $(date +%s) \
304 - --trust-remote-code \
305 - --request-rate 3 \
306 - --num-prompts 1000
307 -```
307 +??? Command
308 +
309 + ```shell
310 + python3 benchmark_serving.py \
311 + --backend vllm \
312 + --model base_model \
313 + --tokenizer meta-llama/Llama-3.1-8B-Instruct \
314 + --dataset-name "random" \
315 + --host 10.0.1.1 \
316 + --port 10001 \
317 + --random-input-len 1024 \
318 + --random-output-len 1024 \
319 + --ignore-eos \
320 + --burstiness 100 \
321 + --percentile-metrics "ttft,tpot,itl,e2el" \
322 + --metric-percentiles "90,95,99" \
323 + --seed $(date +%s) \
324 + --trust-remote-code \
325 + --request-rate 3 \
326 + --num-prompts 1000
327 + ```
328
329 # Shut down
330
docs/design/v1/torch_compile.md
+45 -39
@@ -28,27 +28,29 @@ A unique aspect of vLLM's `torch.compile` integration, is that we guarantee all
28
29 In the very verbose logs, we can see:
30
31 -```
32 -DEBUG 03-07 03:06:52 [decorators.py:203] Start compiling function <code object forward at 0x7f08acf40c90, file "xxx/vllm/model_executor/models/llama.py", line 339>
33 -
34 -DEBUG 03-07 03:06:54 [backends.py:370] Traced files (to be considered for compilation cache):
35 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/torch/_dynamo/polyfills/builtins.py
36 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/torch/nn/modules/container.py
37 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/torch/nn/modules/module.py
38 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/attention/layer.py
39 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/distributed/communication_op.py
40 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/distributed/parallel_state.py
41 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/custom_op.py
42 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/activation.py
43 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/layernorm.py
44 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/linear.py
45 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/rotary_embedding.py
46 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/vocab_parallel_embedding.py
47 -DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/models/llama.py
48 -
49 -DEBUG 03-07 03:07:07 [backends.py:462] Computation graph saved to ~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/computation_graph.py
50 -DEBUG 03-07 03:07:07 [wrapper.py:105] Dynamo transformed code saved to ~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/transformed_code.py
51 -```
31 +??? Logs
32 +
33 + ```text
34 + DEBUG 03-07 03:06:52 [decorators.py:203] Start compiling function <code object forward at 0x7f08acf40c90, file "xxx/vllm/model_executor/models/llama.py", line 339>
35 +
36 + DEBUG 03-07 03:06:54 [backends.py:370] Traced files (to be considered for compilation cache):
37 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/torch/_dynamo/polyfills/builtins.py
38 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/torch/nn/modules/container.py
39 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/torch/nn/modules/module.py
40 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/attention/layer.py
41 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/distributed/communication_op.py
42 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/distributed/parallel_state.py
43 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/custom_op.py
44 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/activation.py
45 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/layernorm.py
46 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/linear.py
47 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/rotary_embedding.py
48 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/vocab_parallel_embedding.py
49 + DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/models/llama.py
50 +
51 + DEBUG 03-07 03:07:07 [backends.py:462] Computation graph saved to ~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/computation_graph.py
52 + DEBUG 03-07 03:07:07 [wrapper.py:105] Dynamo transformed code saved to ~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/transformed_code.py
53 + ```
54
55 This is about the Python code compilation, i.e. graph capture by Dynamo. It tries to trace the function with code `xxx/vllm/model_executor/models/llama.py:339`, which is the `forward` function of the model we compile. During the forward pass, there are also other functions called and inlined by Dynamo, as shown by the logs, including some PyTorch functions from `xxx/torch/nn/modules/module.py` (used by PyTorch `nn.Module`, because module attribute access will trigger a function call), some communication / attention / activation functions from vLLM. All the traced files will be considered when we decide the cache directory to use. This way, any code change in the above files will trigger compilation cache miss, and therefore recompilation.
56
@@ -99,28 +101,31 @@ This time, Inductor compilation is completely bypassed, and we will load from di
101
102 The above example just uses Inductor to compile for a general shape (i.e. symbolic shape). We can also use Inductor to compile for some of the specific shapes, for example:
103
102 -```
103 -vllm serve meta-llama/Llama-3.2-1B --compilation_config '{"compile_sizes": [1, 2, 4, 8]}'
104 +```bash
105 +vllm serve meta-llama/Llama-3.2-1B \
106 + --compilation_config '{"compile_sizes": [1, 2, 4, 8]}'
107 ```
108
109 Then it will also compile a specific kernel just for batch size `1, 2, 4, 8`. At this time, all of the shapes in the computation graph are static and known, and we will turn on auto-tuning to tune for max performance. This can be slow when you run it for the first time, but the next time you run it, we can directly bypass the tuning and run the tuned kernel.
110
111 When all the shapes are known, `torch.compile` can compare different configs, and often find some better configs to run the kernel. For example, we can see the following log:
112
110 -```
111 -AUTOTUNE mm(8x2048, 2048x3072)
112 - triton_mm_4 0.0130 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=2
113 - triton_mm_8 0.0134 ms 97.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4
114 - triton_mm_12 0.0148 ms 87.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=4
115 - mm 0.0160 ms 81.6%
116 - triton_mm_16 0.0165 ms 78.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=8
117 - triton_mm_3 0.0199 ms 65.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=16, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=2
118 - triton_mm_1 0.0203 ms 64.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=2
119 - triton_mm_7 0.0203 ms 64.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=16, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
120 - triton_mm_2 0.0208 ms 62.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=16, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4
121 - triton_mm_11 0.0215 ms 60.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
122 -SingleProcess AUTOTUNE benchmarking takes 2.0428 seconds and 7.5727 seconds precompiling
123 -```
113 +??? Logs
114 +
115 + ```
116 + AUTOTUNE mm(8x2048, 2048x3072)
117 + triton_mm_4 0.0130 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=2
118 + triton_mm_8 0.0134 ms 97.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4
119 + triton_mm_12 0.0148 ms 87.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=4
120 + mm 0.0160 ms 81.6%
121 + triton_mm_16 0.0165 ms 78.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=8
122 + triton_mm_3 0.0199 ms 65.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=16, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=2
123 + triton_mm_1 0.0203 ms 64.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=2
124 + triton_mm_7 0.0203 ms 64.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=16, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
125 + triton_mm_2 0.0208 ms 62.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=16, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4
126 + triton_mm_11 0.0215 ms 60.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
127 + SingleProcess AUTOTUNE benchmarking takes 2.0428 seconds and 7.5727 seconds precompiling
128 + ```
129
130 It means, for a matrix multiplication with shape `8x2048x3072`, `torch.compile` tries triton template with various configs, and it is much faster than the default code (which dispatches to cublas library).
131
@@ -136,8 +141,9 @@ The cudagraphs are captured and managed by the compiler backend, and replayed wh
141
142 By default, vLLM will try to determine a set of sizes to capture cudagraph. You can also override it using the config `cudagraph_capture_sizes`:
143
139 -```
140 -vllm serve meta-llama/Llama-3.2-1B --compilation-config '{"cudagraph_capture_sizes": [1, 2, 4, 8]}'
144 +```bash
145 +vllm serve meta-llama/Llama-3.2-1B \
146 + --compilation-config '{"cudagraph_capture_sizes": [1, 2, 4, 8]}'
147 ```
148
149 Then it will only capture cudagraph for the specified sizes. It can be useful to have fine-grained control over the cudagraph capture.
examples/others/logging_configuration.md
+57 -57
@@ -55,33 +55,33 @@ STDOUT of the console in JSON format with a log level of `INFO`.
55
56 To begin, first, create an appropriate JSON logging configuration file:
57
58 -**/path/to/logging_config.json:**
59 -
60 -```json
61 -{
62 - "formatters": {
63 - "json": {
64 - "class": "pythonjsonlogger.jsonlogger.JsonFormatter"
65 - }
66 - },
67 - "handlers": {
68 - "console": {
69 - "class" : "logging.StreamHandler",
70 - "formatter": "json",
71 - "level": "INFO",
72 - "stream": "ext://sys.stdout"
58 +??? note "/path/to/logging_config.json"
59 +
60 + ```json
61 + {
62 + "formatters": {
63 + "json": {
64 + "class": "pythonjsonlogger.jsonlogger.JsonFormatter"
65 + }
66 + },
67 + "handlers": {
68 + "console": {
69 + "class" : "logging.StreamHandler",
70 + "formatter": "json",
71 + "level": "INFO",
72 + "stream": "ext://sys.stdout"
73 + }
74 + },
75 + "loggers": {
76 + "vllm": {
77 + "handlers": ["console"],
78 + "level": "INFO",
79 + "propagate": false
80 + }
81 + },
82 + "version": 1
83 }
74 - },
75 - "loggers": {
76 - "vllm": {
77 - "handlers": ["console"],
78 - "level": "INFO",
79 - "propagate": false
80 - }
81 - },
82 - "version": 1
83 -}
84 -```
84 + ```
85
86 Finally, run vLLM with the `VLLM_LOGGING_CONFIG_PATH` environment variable set
87 to the path of the custom logging configuration JSON file:
@@ -104,38 +104,38 @@ configuration overrides the built-in default logging configuration used by vLLM.
104 First, create an appropriate JSON logging configuration file that includes
105 configuration for the root vLLM logger and for the logger you wish to silence:
106
107 -**/path/to/logging_config.json:**
108 -
109 -```json
110 -{
111 - "formatters": {
112 - "vllm": {
113 - "class": "vllm.logging_utils.NewLineFormatter",
114 - "datefmt": "%m-%d %H:%M:%S",
115 - "format": "%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s"
116 - }
117 - },
118 - "handlers": {
119 - "vllm": {
120 - "class" : "logging.StreamHandler",
121 - "formatter": "vllm",
122 - "level": "INFO",
123 - "stream": "ext://sys.stdout"
107 +??? note "/path/to/logging_config.json"
108 +
109 + ```json
110 + {
111 + "formatters": {
112 + "vllm": {
113 + "class": "vllm.logging_utils.NewLineFormatter",
114 + "datefmt": "%m-%d %H:%M:%S",
115 + "format": "%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s"
116 + }
117 + },
118 + "handlers": {
119 + "vllm": {
120 + "class" : "logging.StreamHandler",
121 + "formatter": "vllm",
122 + "level": "INFO",
123 + "stream": "ext://sys.stdout"
124 + }
125 + },
126 + "loggers": {
127 + "vllm": {
128 + "handlers": ["vllm"],
129 + "level": "DEBUG",
130 + "propagate": false
131 + },
132 + "vllm.example_noisy_logger": {
133 + "propagate": false
134 + }
135 + },
136 + "version": 1
137 }
125 - },
126 - "loggers": {
127 - "vllm": {
128 - "handlers": ["vllm"],
129 - "level": "DEBUG",
130 - "propagate": false
131 - },
132 - "vllm.example_noisy_logger": {
133 - "propagate": false
134 - }
135 - },
136 - "version": 1
137 -}
138 -```
138 + ```
139
140 Finally, run vLLM with the `VLLM_LOGGING_CONFIG_PATH` environment variable set
141 to the path of the custom logging configuration JSON file: