@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 161 lines 4.73 KB
Rendered Raw
1 # vLLM Recipes Tools
2
3 Convert a vLLM Recipes deployment rendering into files for `vllm serve`.
4
5 ## Optimized Deployment Flow
6
7 ```mermaid
8 flowchart LR
9 R["vLLM Recipe"] --> C["Recipe Converter"]
10 H["Hardware Info (optional)"] --> C
11 W["Workload Info (optional)"] --> C
12 C --> F["config.yml + env.sh"]
13 C -.-> S["Sweep Tuning (optional)"]
14 S -.-> F
15 F --> D["vLLM Docker Image"]
16 D --> E["OpenAI Endpoint"]
17
18 style S stroke-dasharray: 5 5
19 ```
20
21 The recipe is the baseline. Hardware and workload information can optionally
22 refine the initial configuration. Sweep tuning is an optional validation step.
23
24 ## Getting Started
25
26 Use `serve_with_recipe.sh` to fetch the selected deployment from
27 [vLLM Recipes](https://recipes.vllm.ai/), generate the configuration, and start
28 vLLM in one step:
29
30 ```bash
31 tools/recipes/serve_with_recipe.sh \
32 --model meta-llama/Llama-3.1-8B-Instruct \
33 --hardware xeon6
34 ```
35
36 The converter queries `recipes.vllm.ai` at runtime, so it can use the latest
37 published recipe for the requested model and hardware without rebuilding the
38 vLLM image.
39
40 ### Find the hardware name
41
42 Open [vLLM Recipes](https://recipes.vllm.ai/), select the model, and choose the
43 target in the **Hardware** picker. The selected page URL contains the hardware
44 key. For example, `?hardware=xeon6` maps to `--hardware xeon6`.
45
46 You can also run the converter without arguments for interactive model,
47 hardware, and strategy discovery:
48
49 ```bash
50 python3 tools/recipes/recipe_json_to_vllm_config.py
51 ```
52
53 For `xeon6`, the script enables hardware detection automatically. Generated
54 `config.yml` and `env.sh` files remain in the directory where the script was
55 invoked.
56
57 ## 1. vLLM Recipes Only
58
59 Use the converter directly when the recipe already contains the deployment
60 settings you need. This path requires only PyYAML; the vLLM Python package is
61 not required unless optional runtime tuning or sweep generation is requested.
62
63 ```bash
64 pip install pyyaml
65 ```
66
67 Choose whichever recipe-selection method fits the workflow:
68
69 **Interactive discovery** — search models, then choose hardware and strategy:
70
71 ```bash
72 python3 tools/recipes/recipe_json_to_vllm_config.py
73 ```
74
75 **Non-interactive discovery** — provide model and hardware and use the
76 Recipes-recommended strategy:
77
78 ```bash
79 python3 tools/recipes/recipe_json_to_vllm_config.py \
80 --model meta-llama/Llama-3.1-8B-Instruct \
81 --hardware xeon6
82 ```
83
84 **Direct JSON input** — use a Recipes JSON URL or a local JSON file:
85
86 ```bash
87 python3 tools/recipes/recipe_json_to_vllm_config.py \
88 https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
89
90 python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json
91 ```
92
93 All paths generate `config.yml` and `env.sh`. See
94 [REFERENCE.md](REFERENCE.md) for recipe discovery, strategy selection, direct
95 JSON input, custom output files, and deployment scope.
96
97 ## 2. Hardware Information (Optional)
98
99 Add `--detect-hardware` when the target host's effective CPU/NUMA/memory
100 resources should refine deployment-sensitive values such as
101 `tensor-parallel-size` and `gpu-memory-utilization`.
102
103 ```bash
104 python3 tools/recipes/recipe_json_to_vllm_config.py \
105 --model meta-llama/Llama-3.1-8B-Instruct \
106 --hardware xeon6 \
107 --detect-hardware
108 ```
109
110 Hardware detection is optional and uses vLLM CPU resource utilities only when
111 requested. See [RUNTIME_TUNING.md](RUNTIME_TUNING.md#hardware-information).
112
113 ## 3. Workload Information (Optional)
114
115 Workload hints can refine scheduler settings for one initial deployment
116 suggestion. Inputs include token lengths, concurrency, and optional latency or
117 capacity objectives.
118
119 ```bash
120 python3 tools/recipes/recipe_json_to_vllm_config.py \
121 --model meta-llama/Llama-3.1-8B-Instruct \
122 --hardware xeon6 \
123 --input-tokens 128 \
124 --output-tokens 128 \
125 --concurrency 32 \
126 --ttft-sla-ms 3000 \
127 --tpot-sla-ms 100
128 ```
129
130 Hardware detection and workload information are independent optional inputs;
131 they can also be supplied together. See
132 [RUNTIME_TUNING.md](RUNTIME_TUNING.md#workload-information) for the supported
133 inputs and how runtime parameters are calculated.
134
135 ## 4. Sweep Tuning (Optional)
136
137 Use `--generate-sweep` when the initial scheduler suggestion should be validated
138 with `vllm bench sweep serve`. The sweep benchmarks nearby scheduler values and
139 `recommend.py` produces one measured `recommended-config.yml` plus the
140 selection evidence in `recommendation.json`.
141
142 ```bash
143 python3 tools/recipes/recipe_json_to_vllm_config.py \
144 --model meta-llama/Llama-3.1-8B-Instruct \
145 --hardware xeon6 \
146 --detect-hardware \
147 --input-tokens 128 \
148 --output-tokens 128 \
149 --concurrency 32 \
150 --generate-sweep
151 ```
152
153 See [SWEEP_TUNING.md](SWEEP_TUNING.md) for the benchmark, recommendation, and
154 vLLM CPU Docker-shell workflow.
155
156 ## Start vLLM
157
158 ```bash
159 source env.sh
160 vllm serve --config config.yml
161 ```