@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 177 lines 5.45 KB
Rendered Raw
1 # vLLM Recipes Tools
2
3 Utilities for consuming deployment configurations from [vLLM Recipes](https://recipes.vllm.ai/) and converting them into files that can be used directly with vLLM.
4
5 ## Optimized Deployment Flow
6
7 The recipe provides the validated deployment baseline. Hardware and workload
8 information are optional inputs that can refine deployment-sensitive values
9 before the generated configuration is passed to vLLM.
10
11 ```mermaid
12 flowchart LR
13 R["vLLM Recipe"] --> C["Recipe Converter"]
14 H["Hardware Info (optional)"] --> C
15 W["Workload Info (optional)"] --> C
16 C --> F["config.yml + env.sh"]
17 F --> D["vLLM Docker Image"]
18 D --> E["OpenAI Endpoint"]
19 ```
20
21 ## `recipe_json_to_vllm_config.py`
22
23 Converts a hardware-specific vLLM Recipes JSON rendering into:
24
25 - `config.yml` — native configuration for `vllm serve --config`
26 - `env.sh` — environment variables required by the recipe
27
28 The converter uses the Recipes JSON API as the source of truth. It does not reimplement model-variant, hardware, or strategy compatibility rules.
29
30 ## Install Dependency
31
32 ```bash
33 pip install pyyaml
34 ```
35
36 ## Recipe Selection Model
37
38 Discovery follows the JSON links published by vLLM Recipes:
39
40 ```text
41 /models.json
42 -> /{hf_id}.json
43 -> recommended_command.by_hardware[{hardware}]
44 -> /{hf_id}/hw/{hardware}.json
45 -> recommended strategy
46 -> alternatives[{strategy}]
47 ```
48
49 The per-hardware JSON is the recommended deployment rendering for that model and hardware. Alternative strategy renderings are referenced through the JSON's `alternatives` map.
50
51 The converter follows those exact links instead of constructing strategy paths locally. This also handles the Recipes API's legacy default-hardware strategy path automatically.
52
53 ## Interactive Recipe Discovery
54
55 If you do not know the recipe JSON URL, run the script without an input:
56
57 ```bash
58 python3 tools/recipes/recipe_json_to_vllm_config.py
59 ```
60
61 The script will:
62
63 1. Search the Recipes model index, including promoted variant model IDs.
64 2. Ask which model to use.
65 3. Show the hardware configurations available for that model.
66 4. Show the recommended strategy and generated alternative strategies.
67 5. Resolve the exact Recipes JSON endpoint.
68 6. Generate `config.yml` and `env.sh`.
69
70 Example interaction:
71
72 ```text
73 No recipe JSON supplied; starting Recipes API discovery.
74 Model search (for example: llama 3.1): llama 3.1 fp8
75
76 Matching models:
77 [1] nvidia/Llama-3.1-8B-Instruct-FP8 — Llama-3.1-8B-Instruct [Meta]
78 Select model: 1
79
80 Available hardware:
81 [1] arc_pro_b70
82 [2] b200
83 [3] h100
84 Select hardware: 1
85
86 Available strategies:
87 [1] single_node_tp (recommended)
88 [2] multi_node_tp
89 Select strategy: 1
90
91 Resolved recipe:
92 Model: nvidia/Llama-3.1-8B-Instruct-FP8
93 Hardware: arc_pro_b70
94 Strategy: single_node_tp
95 JSON: https://recipes.vllm.ai/nvidia/Llama-3.1-8B-Instruct-FP8/hw/arc_pro_b70.json
96 ```
97
98 ## Non-Interactive Recipe Discovery
99
100 For automation or CI, specify the model and hardware directly:
101
102 ```bash
103 python3 tools/recipes/recipe_json_to_vllm_config.py \
104 --model meta-llama/Llama-3.1-8B-Instruct \
105 --hardware xeon6
106 ```
107
108 When `--strategy` is omitted with both `--model` and `--hardware`, the converter uses the Recipes-recommended strategy from the per-hardware JSON.
109
110 To request a generated alternative strategy explicitly:
111
112 ```bash
113 python3 tools/recipes/recipe_json_to_vllm_config.py \
114 --model nvidia/Llama-3.1-8B-Instruct-FP8 \
115 --hardware arc_pro_b70 \
116 --strategy multi_node_tp
117 ```
118
119 The converter does not synthesize a URL such as `.../strategies/multi_node_tp.json`. It reads the selected hardware JSON and follows the exact `alternatives["multi_node_tp"]` link published by the Recipes API.
120
121 ## Direct Recipe JSON Input
122
123 If you already know the recipe JSON URL:
124
125 ```bash
126 python3 tools/recipes/recipe_json_to_vllm_config.py \
127 https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
128 ```
129
130 A local JSON file is also supported:
131
132 ```bash
133 python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json
134 ```
135
136 When a positional JSON source is supplied, do not combine it with `--model`, `--hardware`, or `--strategy`.
137
138 ## Start vLLM
139
140 After generating the files:
141
142 ```bash
143 source env.sh
144 vllm serve --config config.yml
145 ```
146
147 Example generated `config.yml`:
148
149 ```yaml
150 model: meta-llama/Llama-3.1-8B-Instruct
151 tensor-parallel-size: 1
152 ```
153
154 If the selected recipe does not require environment variables, `env.sh` will contain no additional exports.
155
156 ## Custom Output Files
157
158 ```bash
159 python3 tools/recipes/recipe_json_to_vllm_config.py \
160 --model meta-llama/Llama-3.1-8B-Instruct \
161 --hardware xeon6 \
162 --config-out llama31-xeon6.yml \
163 --env-out llama31-xeon6-env.sh
164 ```
165
166 ## Strategy and Deployment Scope
167
168 Strategy discovery and config conversion are separate concerns.
169
170 The converter can resolve any generated strategy JSON exposed by the Recipes API. It currently emits one `config.yml`, so the selected rendering must contain a single `vllm serve` `argv`.
171
172 Single-process renderings such as `single_node_tp` can be converted directly. Multi-node, disaggregated prefill/decode, or other multi-process renderings expose fields such as `head_argv`, `worker_argv`, `prefill`, or `decode`; the converter intentionally exits instead of generating an incomplete single-process config.
173
174 ## Related Guides
175
176 - [RUNTIME_TUNING.md](RUNTIME_TUNING.md) — optional hardware and workload inputs, runtime parameter policies, and calculations.
177 - [SWEEP_TUNING.md](SWEEP_TUNING.md) — optional benchmark sweep and measured recommendation workflow.