A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 03:30 UTC
[Misc] small update (#20462)
Signed-off-by: reidliu41 <reid201711@gmail.com>
Reid committed
Jul 4, 2025 at 11:33 UTC
a7bab0c9e58c412cb7fa66c4f86de9911de26a3c
3 files changed
+18
-9
examples/offline_inference/profiling_tpu/README.md
+4
-1
@@ -57,7 +57,10 @@ Once you have collected your profiles with this script, you can visualize them u
57
Here are most likely the dependencies you need to install:
58
59
```bash
60
-pip install tensorflow-cpu tensorboard-plugin-profile etils importlib_resources
60
+pip install tensorflow-cpu \
61
+ tensorboard-plugin-profile \
62
+ etils \
63
+ importlib_resources
64
```
65
66
Then you just need to point TensorBoard to the directory where you saved the profiles and visit `http://localhost:6006/` in your browser:
examples/online_serving/structured_outputs/README.md
+7
-3
@@ -13,13 +13,15 @@ vllm serve Qwen/Qwen2.5-3B-Instruct
13
To serve a reasoning model, you can use the following command:
14
15
```bash
16
-vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --reasoning-parser deepseek_r1
16
+vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
17
+ --reasoning-parser deepseek_r1
18
```
19
20
If you want to run this script standalone with `uv`, you can use the following:
21
22
```bash
22
-uvx --from git+https://github.com/vllm-project/vllm#subdirectory=examples/online_serving/structured_outputs structured-output
23
+uvx --from git+https://github.com/vllm-project/vllm#subdirectory=examples/online_serving/structured_outputs \
24
+ structured-output
25
```
26
27
See [feature docs](https://docs.vllm.ai/en/latest/features/structured_outputs.html) for more information.
@@ -44,7 +46,9 @@ uv run structured_outputs.py --stream
46
Run certain constraints, for example `structural_tag` and `regex`, streaming:
47
48
```bash
47
-uv run structured_outputs.py --constraint structural_tag regex --stream
49
+uv run structured_outputs.py \
50
+ --constraint structural_tag regex \
51
+ --stream
52
```
53
54
Run all constraints, with reasoning models and streaming:
examples/others/tensorize_vllm_model.py
+7
-5
@@ -202,7 +202,7 @@ def parse_args():
202
203
204
205
-def deserialize():
205
+def deserialize(args, tensorizer_config):
206
if args.lora_path:
207
tensorizer_config.lora_dir = tensorizer_config.tensorizer_dir
208
llm = LLM(model=args.model,
@@ -242,7 +242,7 @@ def deserialize():
242
return llm
243
244
245
-if __name__ == '__main__':
245
+def main():
246
args = parse_args()
247
248
s3_access_key_id = (getattr(args, 's3_access_key_id', None)
@@ -260,8 +260,6 @@ if __name__ == '__main__':
260
261
model_ref = args.model
262
263
- model_name = model_ref.split("/")[1]
264
-
263
if args.command == "serialize" or args.command == "deserialize":
264
keyfile = args.keyfile
265
else:
@@ -309,6 +307,10 @@ if __name__ == '__main__':
307
encryption_keyfile = keyfile,
308
**credentials
309
)
312
- deserialize()
310
+ deserialize(args, tensorizer_config)
311
else:
312
raise ValueError("Either serialize or deserialize must be specified.")
313
+
314
+
315
+if __name__ == "__main__":
316
+ main()