@tduncan / Nvidia-Optimization-Stack-VLLM / commits / 4af586e185

Vllm recipes tool improve (#53325)

Vllm recipes tool improve (#53325) Signed-off-by: louie-tsai <louie.tsai@intel.com>

Louie Tsai committed Aug 24, 2026 at 22:53 UTC 4af586e185b028acf08312a4dee381b5998a137e
9 files changed +1930 -105
tools/recipes/README.md
+76 -104
@@ -1,87 +1,46 @@
1 # vLLM Recipes Tools
2
3 -Utilities for consuming deployment configurations from [vLLM Recipes](https://recipes.vllm.ai/) and converting them into files that can be used directly with vLLM.
4 -
5 -## `recipe_json_to_vllm_config.py`
6 -
7 -Converts a hardware-specific vLLM Recipes JSON rendering into:
3 +Convert a vLLM Recipes deployment rendering into files for `vllm serve`.
4 +
5 +## Optimized Deployment Flow
6 +
7 +```mermaid
8 +flowchart LR
9 + R["vLLM Recipe"] --> C["Recipe Converter"]
10 + H["Hardware Info (optional)"] --> C
11 + W["Workload Info (optional)"] --> C
12 + C --> F["config.yml + env.sh"]
13 + C -.-> S["Sweep Tuning (optional)"]
14 + S -.-> F
15 + F --> D["vLLM Docker Image"]
16 + D --> E["OpenAI Endpoint"]
17 +
18 + style S stroke-dasharray: 5 5
19 +```
20
9 -- `config.yml` — native configuration for `vllm serve --config`
10 -- `env.sh` — environment variables required by the recipe
21 +The recipe is the baseline. Hardware and workload information can optionally
22 +refine the initial configuration. Sweep tuning is an optional validation step.
23
12 -The converter uses the Recipes JSON API as the source of truth. It does not reimplement model-variant, hardware, or strategy compatibility rules.
24 +## 1. vLLM Recipes Only
25
14 -## Install Dependency
26 +Use the converter directly when the recipe already contains the deployment
27 +settings you need. This path requires only PyYAML; the vLLM Python package is
28 +not required unless optional runtime tuning or sweep generation is requested.
29
30 ```bash
31 pip install pyyaml
32 ```
33
20 -## Recipe Selection Model
21 -
22 -Discovery follows the JSON links published by vLLM Recipes:
23 -
24 -```text
25 -/models.json
26 - -> /{hf_id}.json
27 - -> recommended_command.by_hardware[{hardware}]
28 - -> /{hf_id}/hw/{hardware}.json
29 - -> recommended strategy
30 - -> alternatives[{strategy}]
31 -```
32 -
33 -The per-hardware JSON is the recommended deployment rendering for that model and hardware. Alternative strategy renderings are referenced through the JSON's `alternatives` map.
34 -
35 -The converter follows those exact links instead of constructing strategy paths locally. This also handles the Recipes API's legacy default-hardware strategy path automatically.
34 +Choose whichever recipe-selection method fits the workflow:
35
37 -## Interactive Recipe Discovery
38 -
39 -If you do not know the recipe JSON URL, run the script without an input:
36 +**Interactive discovery** — search models, then choose hardware and strategy:
37
38 ```bash
39 python3 tools/recipes/recipe_json_to_vllm_config.py
40 ```
41
45 -The script will:
46 -
47 -1. Search the Recipes model index, including promoted variant model IDs.
48 -2. Ask which model to use.
49 -3. Show the hardware configurations available for that model.
50 -4. Show the recommended strategy and generated alternative strategies.
51 -5. Resolve the exact Recipes JSON endpoint.
52 -6. Generate `config.yml` and `env.sh`.
53 -
54 -Example interaction:
55 -
56 -```text
57 -No recipe JSON supplied; starting Recipes API discovery.
58 -Model search (for example: llama 3.1): llama 3.1 fp8
59 -
60 -Matching models:
61 - [1] nvidia/Llama-3.1-8B-Instruct-FP8 — Llama-3.1-8B-Instruct [Meta]
62 -Select model: 1
63 -
64 -Available hardware:
65 - [1] arc_pro_b70
66 - [2] b200
67 - [3] h100
68 -Select hardware: 1
69 -
70 -Available strategies:
71 - [1] single_node_tp (recommended)
72 - [2] multi_node_tp
73 -Select strategy: 1
74 -
75 -Resolved recipe:
76 - Model: nvidia/Llama-3.1-8B-Instruct-FP8
77 - Hardware: arc_pro_b70
78 - Strategy: single_node_tp
79 - JSON: https://recipes.vllm.ai/nvidia/Llama-3.1-8B-Instruct-FP8/hw/arc_pro_b70.json
80 -```
81 -
82 -## Non-Interactive Recipe Discovery
83 -
84 -For automation or CI, specify the model and hardware directly:
42 +**Non-interactive discovery** — provide model and hardware and use the
43 +Recipes-recommended strategy:
44
45 ```bash
46 python3 tools/recipes/recipe_json_to_vllm_config.py \
@@ -89,68 +48,81 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \
48 --hardware xeon6
49 ```
50
92 -When `--strategy` is omitted with both `--model` and `--hardware`, the converter uses the Recipes-recommended strategy from the per-hardware JSON.
93 -
94 -To request a generated alternative strategy explicitly:
51 +**Direct JSON input** — use a Recipes JSON URL or a local JSON file:
52
53 ```bash
54 python3 tools/recipes/recipe_json_to_vllm_config.py \
98 - --model nvidia/Llama-3.1-8B-Instruct-FP8 \
99 - --hardware arc_pro_b70 \
100 - --strategy multi_node_tp
55 + https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
56 +
57 +python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json
58 ```
59
103 -The converter does not synthesize a URL such as `.../strategies/multi_node_tp.json`. It reads the selected hardware JSON and follows the exact `alternatives["multi_node_tp"]` link published by the Recipes API.
60 +All paths generate `config.yml` and `env.sh`. See
61 +[REFERENCE.md](REFERENCE.md) for recipe discovery, strategy selection, direct
62 +JSON input, custom output files, and deployment scope.
63
105 -## Direct Recipe JSON Input
64 +## 2. Hardware Information (Optional)
65
107 -If you already know the recipe JSON URL:
66 +Add `--detect-hardware` when the target host's effective CPU/NUMA/memory
67 +resources should refine deployment-sensitive values such as
68 +`tensor-parallel-size` and `gpu-memory-utilization`.
69
70 ```bash
71 python3 tools/recipes/recipe_json_to_vllm_config.py \
111 - https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
112 -```
113 -
114 -A local JSON file is also supported:
115 -
116 -```bash
117 -python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json
72 + --model meta-llama/Llama-3.1-8B-Instruct \
73 + --hardware xeon6 \
74 + --detect-hardware
75 ```
76
120 -When a positional JSON source is supplied, do not combine it with `--model`, `--hardware`, or `--strategy`.
77 +Hardware detection is optional and uses vLLM CPU resource utilities only when
78 +requested. See [RUNTIME_TUNING.md](RUNTIME_TUNING.md#hardware-information).
79
122 -## Start vLLM
80 +## 3. Workload Information (Optional)
81
124 -After generating the files:
82 +Workload hints can refine scheduler settings for one initial deployment
83 +suggestion. Inputs include token lengths, concurrency, and optional latency or
84 +capacity objectives.
85
86 ```bash
127 -source env.sh
128 -vllm serve --config config.yml
87 +python3 tools/recipes/recipe_json_to_vllm_config.py \
88 + --model meta-llama/Llama-3.1-8B-Instruct \
89 + --hardware xeon6 \
90 + --input-tokens 2048 \
91 + --output-tokens 128 \
92 + --concurrency 32 \
93 + --ttft-sla-ms 3000 \
94 + --tpot-sla-ms 100
95 ```
96
131 -Example generated `config.yml`:
132 -
133 -```yaml
134 -model: meta-llama/Llama-3.1-8B-Instruct
135 -tensor-parallel-size: 1
136 -```
97 +Hardware detection and workload information are independent optional inputs;
98 +they can also be supplied together. See
99 +[RUNTIME_TUNING.md](RUNTIME_TUNING.md#workload-information) for the supported
100 +inputs and how runtime parameters are calculated.
101
138 -If the selected recipe does not require environment variables, `env.sh` will contain no additional exports.
102 +## 4. Sweep Tuning (Optional)
103
140 -## Custom Output Files
104 +Use `--generate-sweep` when the initial scheduler suggestion should be validated
105 +with `vllm bench sweep serve`. The sweep benchmarks nearby scheduler values and
106 +`recommend.py` produces one measured `recommended-config.yml` plus the
107 +selection evidence in `recommendation.json`.
108
109 ```bash
110 python3 tools/recipes/recipe_json_to_vllm_config.py \
111 --model meta-llama/Llama-3.1-8B-Instruct \
112 --hardware xeon6 \
146 - --config-out llama31-xeon6.yml \
147 - --env-out llama31-xeon6-env.sh
113 + --detect-hardware \
114 + --input-tokens 2048 \
115 + --output-tokens 128 \
116 + --concurrency 32 \
117 + --generate-sweep
118 ```
119
150 -## Strategy and Deployment Scope
120 +See [SWEEP_TUNING.md](SWEEP_TUNING.md) for the benchmark, recommendation, and
121 +vLLM CPU Docker-shell workflow.
122
152 -Strategy discovery and config conversion are separate concerns.
153 -
154 -The converter can resolve any generated strategy JSON exposed by the Recipes API. It currently emits one `config.yml`, so the selected rendering must contain a single `vllm serve` `argv`.
123 +## Start vLLM
124
156 -Single-process renderings such as `single_node_tp` can be converted directly. Multi-node, disaggregated prefill/decode, or other multi-process renderings expose fields such as `head_argv`, `worker_argv`, `prefill`, or `decode`; the converter intentionally exits instead of generating an incomplete single-process config.
125 +```bash
126 +source env.sh
127 +vllm serve --config config.yml
128 +```
tools/recipes/REFERENCE.md new
+177
@@ -0,0 +1,177 @@
1 +# vLLM Recipes Tools
2 +
3 +Utilities for consuming deployment configurations from [vLLM Recipes](https://recipes.vllm.ai/) and converting them into files that can be used directly with vLLM.
4 +
5 +## Optimized Deployment Flow
6 +
7 +The recipe provides the validated deployment baseline. Hardware and workload
8 +information are optional inputs that can refine deployment-sensitive values
9 +before the generated configuration is passed to vLLM.
10 +
11 +```mermaid
12 +flowchart LR
13 + R["vLLM Recipe"] --> C["Recipe Converter"]
14 + H["Hardware Info (optional)"] --> C
15 + W["Workload Info (optional)"] --> C
16 + C --> F["config.yml + env.sh"]
17 + F --> D["vLLM Docker Image"]
18 + D --> E["OpenAI Endpoint"]
19 +```
20 +
21 +## `recipe_json_to_vllm_config.py`
22 +
23 +Converts a hardware-specific vLLM Recipes JSON rendering into:
24 +
25 +- `config.yml` — native configuration for `vllm serve --config`
26 +- `env.sh` — environment variables required by the recipe
27 +
28 +The converter uses the Recipes JSON API as the source of truth. It does not reimplement model-variant, hardware, or strategy compatibility rules.
29 +
30 +## Install Dependency
31 +
32 +```bash
33 +pip install pyyaml
34 +```
35 +
36 +## Recipe Selection Model
37 +
38 +Discovery follows the JSON links published by vLLM Recipes:
39 +
40 +```text
41 +/models.json
42 + -> /{hf_id}.json
43 + -> recommended_command.by_hardware[{hardware}]
44 + -> /{hf_id}/hw/{hardware}.json
45 + -> recommended strategy
46 + -> alternatives[{strategy}]
47 +```
48 +
49 +The per-hardware JSON is the recommended deployment rendering for that model and hardware. Alternative strategy renderings are referenced through the JSON's `alternatives` map.
50 +
51 +The converter follows those exact links instead of constructing strategy paths locally. This also handles the Recipes API's legacy default-hardware strategy path automatically.
52 +
53 +## Interactive Recipe Discovery
54 +
55 +If you do not know the recipe JSON URL, run the script without an input:
56 +
57 +```bash
58 +python3 tools/recipes/recipe_json_to_vllm_config.py
59 +```
60 +
61 +The script will:
62 +
63 +1. Search the Recipes model index, including promoted variant model IDs.
64 +2. Ask which model to use.
65 +3. Show the hardware configurations available for that model.
66 +4. Show the recommended strategy and generated alternative strategies.
67 +5. Resolve the exact Recipes JSON endpoint.
68 +6. Generate `config.yml` and `env.sh`.
69 +
70 +Example interaction:
71 +
72 +```text
73 +No recipe JSON supplied; starting Recipes API discovery.
74 +Model search (for example: llama 3.1): llama 3.1 fp8
75 +
76 +Matching models:
77 + [1] nvidia/Llama-3.1-8B-Instruct-FP8 — Llama-3.1-8B-Instruct [Meta]
78 +Select model: 1
79 +
80 +Available hardware:
81 + [1] arc_pro_b70
82 + [2] b200
83 + [3] h100
84 +Select hardware: 1
85 +
86 +Available strategies:
87 + [1] single_node_tp (recommended)
88 + [2] multi_node_tp
89 +Select strategy: 1
90 +
91 +Resolved recipe:
92 + Model: nvidia/Llama-3.1-8B-Instruct-FP8
93 + Hardware: arc_pro_b70
94 + Strategy: single_node_tp
95 + JSON: https://recipes.vllm.ai/nvidia/Llama-3.1-8B-Instruct-FP8/hw/arc_pro_b70.json
96 +```
97 +
98 +## Non-Interactive Recipe Discovery
99 +
100 +For automation or CI, specify the model and hardware directly:
101 +
102 +```bash
103 +python3 tools/recipes/recipe_json_to_vllm_config.py \
104 + --model meta-llama/Llama-3.1-8B-Instruct \
105 + --hardware xeon6
106 +```
107 +
108 +When `--strategy` is omitted with both `--model` and `--hardware`, the converter uses the Recipes-recommended strategy from the per-hardware JSON.
109 +
110 +To request a generated alternative strategy explicitly:
111 +
112 +```bash
113 +python3 tools/recipes/recipe_json_to_vllm_config.py \
114 + --model nvidia/Llama-3.1-8B-Instruct-FP8 \
115 + --hardware arc_pro_b70 \
116 + --strategy multi_node_tp
117 +```
118 +
119 +The converter does not synthesize a URL such as `.../strategies/multi_node_tp.json`. It reads the selected hardware JSON and follows the exact `alternatives["multi_node_tp"]` link published by the Recipes API.
120 +
121 +## Direct Recipe JSON Input
122 +
123 +If you already know the recipe JSON URL:
124 +
125 +```bash
126 +python3 tools/recipes/recipe_json_to_vllm_config.py \
127 + https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
128 +```
129 +
130 +A local JSON file is also supported:
131 +
132 +```bash
133 +python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json
134 +```
135 +
136 +When a positional JSON source is supplied, do not combine it with `--model`, `--hardware`, or `--strategy`.
137 +
138 +## Start vLLM
139 +
140 +After generating the files:
141 +
142 +```bash
143 +source env.sh
144 +vllm serve --config config.yml
145 +```
146 +
147 +Example generated `config.yml`:
148 +
149 +```yaml
150 +model: meta-llama/Llama-3.1-8B-Instruct
151 +tensor-parallel-size: 1
152 +```
153 +
154 +If the selected recipe does not require environment variables, `env.sh` will contain no additional exports.
155 +
156 +## Custom Output Files
157 +
158 +```bash
159 +python3 tools/recipes/recipe_json_to_vllm_config.py \
160 + --model meta-llama/Llama-3.1-8B-Instruct \
161 + --hardware xeon6 \
162 + --config-out llama31-xeon6.yml \
163 + --env-out llama31-xeon6-env.sh
164 +```
165 +
166 +## Strategy and Deployment Scope
167 +
168 +Strategy discovery and config conversion are separate concerns.
169 +
170 +The converter can resolve any generated strategy JSON exposed by the Recipes API. It currently emits one `config.yml`, so the selected rendering must contain a single `vllm serve` `argv`.
171 +
172 +Single-process renderings such as `single_node_tp` can be converted directly. Multi-node, disaggregated prefill/decode, or other multi-process renderings expose fields such as `head_argv`, `worker_argv`, `prefill`, or `decode`; the converter intentionally exits instead of generating an incomplete single-process config.
173 +
174 +## Related Guides
175 +
176 +- [RUNTIME_TUNING.md](RUNTIME_TUNING.md) — optional hardware and workload inputs, runtime parameter policies, and calculations.
177 +- [SWEEP_TUNING.md](SWEEP_TUNING.md) — optional benchmark sweep and measured recommendation workflow.
tools/recipes/RUNTIME_TUNING.md new
+292
@@ -0,0 +1,292 @@
1 +# Runtime Tuning
2 +
3 +The Recipes JSON remains the baseline. vLLM Recipes already provide validated
4 +model, hardware, strategy, environment variables, and serving arguments.
5 +Runtime tuning is optional and is intended for parameters whose best value can
6 +depend on the actual deployment resources or expected request workload.
7 +
8 +## Information Sources
9 +
10 +Runtime tuning combines up to three inputs:
11 +
12 +- **vLLM Recipe** — the required baseline from the Recipes JSON API or a direct
13 + recipe JSON file.
14 +- **Hardware information (optional)** — detected with `--detect-hardware`,
15 + including effective CPU/NUMA topology and memory availability.
16 +- **Workload information (optional)** — supplied through CLI hints such as input
17 + and output token lengths, concurrency, TTFT/TPOT objectives, and target QPS.
18 +
19 +If no optional hardware or workload information is supplied, the converter keeps
20 +the normal Recipes conversion behavior.
21 +
22 +See [Deployment-Time Parameters](#deployment-time-parameters) for the mapping
23 +between these inputs and individual runtime parameters.
24 +
25 +## Hardware Information
26 +
27 +Hardware detection is enabled only with `--detect-hardware`. It reports the
28 +effective resources available to the process or container, including CPU/NUMA
29 +topology and memory information. The current policy can use those values to
30 +refine `tensor-parallel-size` and `gpu-memory-utilization`.
31 +
32 +```bash
33 +python3 tools/recipes/recipe_json_to_vllm_config.py \
34 + --model meta-llama/Llama-3.1-8B-Instruct \
35 + --hardware xeon6 \
36 + --detect-hardware
37 +```
38 +
39 +Hardware detection is optional. Recipe hardware selects the tuning policy; host
40 +hardware detection provides resource information to that policy.
41 +
42 +## Workload Information
43 +
44 +Workload information is supplied explicitly because the converter runs before a
45 +vLLM server exists. Supported hints include:
46 +
47 +- `--input-tokens`
48 +- `--output-tokens`
49 +- `--concurrency`
50 +- `--ttft-sla-ms`
51 +- `--tpot-sla-ms`
52 +- `--target-qps`
53 +
54 +Example:
55 +
56 +```bash
57 +python3 tools/recipes/recipe_json_to_vllm_config.py \
58 + --model meta-llama/Llama-3.1-8B-Instruct \
59 + --hardware xeon6 \
60 + --input-tokens 2048 \
61 + --output-tokens 128 \
62 + --concurrency 32 \
63 + --ttft-sla-ms 3000 \
64 + --tpot-sla-ms 100
65 +```
66 +
67 +`--output-tokens` is used to estimate steady-state request turnover for
68 +`max-num-batched-tokens`. When both `--target-qps` and `--tpot-sla-ms` are
69 +available, they are also used to estimate prompt arrival pressure per scheduler
70 +step. `--ttft-sla-ms` is collected but is not yet converted directly into a
71 +batch-size formula because the relationship between TTFT and scheduler budget is
72 +model- and hardware-dependent.
73 +
74 +## Deployment-Time Parameters
75 +
76 +The parameters below may already exist in a recipe. They are candidates for
77 +deployment-time refinement when the recipe value is missing, generic, or based
78 +on a validation environment that differs from the user's target deployment.
79 +
80 +| Runtime parameter | Why it may need deployment-time refinement | Main decision input | Current policy |
81 +| --- | --- | --- | --- |
82 +| `tensor-parallel-size` | The effective CPU/NUMA topology available to a container or pod can differ from the system used to validate the recipe. | Hardware topology | Use the largest power-of-two TP value that does not exceed the effective NUMA-node count. |
83 +| `gpu-memory-utilization` | Available memory can differ by machine size, container limits, and other memory use. The vLLM option name is also used by the CPU backend. | Hardware memory + recipe baseline | Calculate a conservative fraction from the most constrained NUMA node. |
84 +| `max-num-seqs` | The useful scheduler concurrency depends on the number of requests expected to be active at the same time. | Workload concurrency | Set `max-num-seqs` to `--concurrency` when supplied. |
85 +| `max-num-batched-tokens` | Each scheduler iteration must share its token budget between active decodes and incoming prefills. | Input/output token shape, concurrency, and optional QPS/TPOT | Calculate decode budget + expected prefill demand, with vLLM scheduler constraints as floors. |
86 +| `data-parallel-size` | The required replica count depends on the requested throughput and the capacity of one replica. | Capacity target | Keep the recipe value today because per-replica SLA capacity is not known. |
87 +
88 +## How Each Runtime Parameter Is Calculated
89 +
90 +### `tensor-parallel-size`
91 +
92 +TP is derived from the effective NUMA topology reported by hardware detection:
93 +
94 +```text
95 +TP = largest power of two <= effective NUMA-node count
96 +```
97 +
98 +For example, 2 effective NUMA nodes produce TP=2, while 6 nodes currently
99 +produce TP=4. This is a topology-based starting point and avoids automatically
100 +selecting unusual non-power-of-two TP sizes before they are validated.
101 +
102 +### `gpu-memory-utilization`
103 +
104 +For every effective NUMA memory node, the policy calculates:
105 +
106 +```text
107 +node_safe_fraction = available_memory / total_memory - 0.10
108 +safe_fraction = min(0.90, minimum node_safe_fraction)
109 +```
110 +
111 +The 10% reserve leaves memory for model/runtime overhead outside the vLLM cache
112 +budget. If the recipe already contains a smaller value, the policy preserves the
113 +smaller value. If it is absent, `0.80` is used as the initial ceiling:
114 +
115 +```text
116 +candidate = min(recipe value or 0.80, safe_fraction)
117 +```
118 +
119 +### `max-num-seqs`
120 +
121 +When `--concurrency` is supplied, it directly represents the requested maximum
122 +number of simultaneously active requests:
123 +
124 +```text
125 +max-num-seqs = concurrency
126 +```
127 +
128 +If concurrency is not supplied, the converter does not override this parameter.
129 +
130 +### `max-num-batched-tokens`
131 +
132 +The token budget is workload-derived and no longer assumes a fixed number of
133 +parallel prefills.
134 +
135 +First, the policy determines the active sequence count:
136 +
137 +```text
138 +active_sequences =
139 + --concurrency
140 + or recipe max-num-seqs
141 + or vLLM default max_num_seqs (128)
142 +```
143 +
144 +Decode requests consume approximately one token per active sequence in a
145 +scheduler iteration:
146 +
147 +```text
148 +decode_budget = active_sequences
149 +```
150 +
151 +The policy then estimates how many new prompts need prefill work per scheduler
152 +step:
153 +
154 +```text
155 +prefills_per_step = max(
156 + 1,
157 + active_sequences / output_tokens,
158 + target_qps * tpot_sla_ms / 1000
159 +)
160 +
161 +prefills_per_step = min(active_sequences, prefills_per_step)
162 +prefill_budget = ceil(input_tokens * prefills_per_step)
163 +```
164 +
165 +The final scheduler budget is:
166 +
167 +```text
168 +max-num-batched-tokens = max(
169 + vLLM default max_num_batched_tokens (2048),
170 + active_sequences,
171 + decode_budget + prefill_budget
172 +)
173 +```
174 +
175 +For example, with 2048 input tokens, 128 output tokens, and concurrency 32:
176 +
177 +```text
178 +decode_budget = 32
179 +prefills_per_step = max(1, 32 / 128) = 1
180 +prefill_budget = 2048
181 +candidate = max(2048, 32, 32 + 2048) = 2080
182 +```
183 +
184 +If chunked prefill is explicitly disabled and `max-model-len` is available, the
185 +policy also ensures:
186 +
187 +```text
188 +max-num-batched-tokens >= max-model-len
189 +```
190 +
191 +### `data-parallel-size`
192 +
193 +`--target-qps` alone is not enough to choose DP safely. A correct capacity rule
194 +also needs measured per-replica throughput that still satisfies TTFT/TPOT:
195 +
196 +```text
197 +DP = ceil(target_qps / qps_per_replica_at_SLO)
198 +```
199 +
200 +Because the converter does not have that measured capacity yet, it intentionally
201 +keeps the recipe DP value rather than guessing.
202 +
203 +## Precedence
204 +
205 +```text
206 +vLLM defaults
207 + -> vLLM Recipes baseline
208 + -> hardware refinement (optional)
209 + -> workload / SLO refinement (optional)
210 + -> config.yml + env.sh
211 +```
212 +
213 +## Runtime-Tuning Hardware Scope
214 +
215 +Runtime tuning is selected from the resolved recipe JSON's `hardware` field,
216 +rather than by inspecting which physical devices happen to be present on the
217 +host. This matters because a GPU server also exposes its host CPU topology.
218 +
219 +The current runtime-tuning policy registry contains `xeon6`. A tuning request
220 +for unregistered recipe hardware, such as `b200`, fails before host hardware
221 +detection:
222 +
223 +```text
224 +ERROR: Runtime tuning is not supported for recipe hardware 'b200'.
225 +Currently supported: xeon6.
226 +```
227 +
228 +Plain recipe conversion remains available for all recipe hardware. The gate only
229 +applies when optional runtime-tuning inputs are requested.
230 +
231 +## vLLM CPU Docker Shell
232 +
233 +Run hardware detection and runtime tuning inside the target vLLM CPU container
234 +so detected CPU, NUMA, memory, and cgroup limits match the deployment
235 +environment.
236 +
237 +From the vLLM source tree:
238 +
239 +```bash
240 +mkdir -p recipe-output
241 +
242 +docker run --rm -it \
243 + --entrypoint bash \
244 + --security-opt seccomp=unconfined \
245 + --cap-add SYS_NICE \
246 + --shm-size=4g \
247 + -p 8000:8000 \
248 + -v "$PWD/tools/recipes:/recipes:ro" \
249 + -v "$PWD/recipe-output:/output" \
250 + -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
251 + -w /output \
252 + vllm/vllm-openai-cpu:latest-x86_64
253 +```
254 +
255 +`/output` is writable; `/recipes` remains read-only.
256 +
257 +Inside the container, generate one runtime-tuned initial configuration:
258 +
259 +```bash
260 +python3 /recipes/recipe_json_to_vllm_config.py \
261 + --model meta-llama/Llama-3.1-8B-Instruct \
262 + --hardware xeon6 \
263 + --detect-hardware \
264 + --input-tokens 2048 \
265 + --output-tokens 128 \
266 + --concurrency 32 \
267 + --ttft-sla-ms 3000 \
268 + --tpot-sla-ms 100 \
269 + --config-out /output/config.yml \
270 + --env-out /output/env.sh
271 +```
272 +
273 +Deploy the generated initial configuration:
274 +
275 +```bash
276 +source /output/env.sh
277 +vllm serve --config /output/config.yml
278 +```
279 +
280 +For optional benchmark-backed validation of the scheduler settings, continue
281 +with [SWEEP_TUNING.md](SWEEP_TUNING.md).
282 +
283 +## Implementation
284 +
285 +- `hardware_detection.py` collects effective CPU/NUMA/memory information.
286 +- `runtime_tuning.py` owns hardware-policy selection and independent parameter
287 + tuning functions.
288 +- `recipe_json_to_vllm_config.py` resolves the recipe, collects optional inputs,
289 + applies the selected policy, and generates `config.yml` and `env.sh`.
290 +
291 +For benchmark-backed validation of the initial scheduler suggestion, see
292 +[SWEEP_TUNING.md](SWEEP_TUNING.md).
tools/recipes/SWEEP_TUNING.md new
+124
@@ -0,0 +1,124 @@
1 +# Sweep Tuning
2 +
3 +Sweep tuning is optional. The converter always creates one initial `config.yml`
4 +first. Users can deploy it immediately or benchmark nearby scheduler settings
5 +and generate one measured recommendation.
6 +
7 +## Generate
8 +
9 +```bash
10 +python3 tools/recipes/recipe_json_to_vllm_config.py \
11 + --model meta-llama/Llama-3.1-8B-Instruct \
12 + --hardware xeon6 \
13 + --detect-hardware \
14 + --input-tokens 2048 \
15 + --output-tokens 128 \
16 + --concurrency 32 \
17 + --ttft-sla-ms 3000 \
18 + --tpot-sla-ms 100 \
19 + --generate-sweep
20 +```
21 +
22 +The optional package is:
23 +
24 +```text
25 +sweep/
26 +├── serve_params.json
27 +├── bench_params.json
28 +├── run_sweep.sh
29 +├── recommend.py
30 +└── SWEEP.md
31 +```
32 +
33 +The sweep currently varies only `max-num-seqs` and
34 +`max-num-batched-tokens`.
35 +
36 +## Run and Recommend
37 +
38 +```bash
39 +sweep/run_sweep.sh --dry-run
40 +sweep/run_sweep.sh
41 +sweep/recommend.py
42 +```
43 +
44 +Resume an interrupted sweep with:
45 +
46 +```bash
47 +sweep/run_sweep.sh --resume
48 +```
49 +
50 +`recommend.py` writes:
51 +
52 +```text
53 +sweep/recommended-config.yml
54 +sweep/recommendation.json
55 +```
56 +
57 +With TTFT/TPOT objectives, the benchmark uses vLLM `--goodput`, and the
58 +recommender selects highest mean request goodput across repeated runs. Without
59 +latency objectives it selects highest mean output-token throughput. Failed
60 +benchmark configurations are excluded.
61 +
62 +`recommended-config.yml` changes only the two swept scheduler parameters.
63 +`recommendation.json` records the measured evidence.
64 +
65 +Deploy:
66 +
67 +```bash
68 +source env.sh
69 +vllm serve --config sweep/recommended-config.yml
70 +```
71 +
72 +## vLLM CPU Docker Shell
73 +
74 +Use the common CPU container setup in
75 +[RUNTIME_TUNING.md](RUNTIME_TUNING.md#vllm-cpu-docker-shell). It ensures
76 +hardware detection runs against the same effective CPU, NUMA, memory, and
77 +cgroup limits used by the deployment.
78 +
79 +Inside that container, generate the initial suggestion plus sweep package:
80 +
81 +```bash
82 +python3 /recipes/recipe_json_to_vllm_config.py \
83 + --model meta-llama/Llama-3.1-8B-Instruct \
84 + --hardware xeon6 \
85 + --detect-hardware \
86 + --input-tokens 2048 \
87 + --output-tokens 128 \
88 + --concurrency 32 \
89 + --ttft-sla-ms 3000 \
90 + --tpot-sla-ms 100 \
91 + --config-out /output/config.yml \
92 + --env-out /output/env.sh \
93 + --generate-sweep \
94 + --sweep-out-dir /output/sweep
95 +```
96 +
97 +The initial configuration can be deployed directly:
98 +
99 +```bash
100 +source /output/env.sh
101 +vllm serve --config /output/config.yml
102 +```
103 +
104 +Stop that manually started server before running the sweep, because
105 +`run_sweep.sh` starts and stops its own vLLM servers.
106 +
107 +```bash
108 +/output/sweep/run_sweep.sh --dry-run
109 +/output/sweep/run_sweep.sh
110 +/output/sweep/recommend.py
111 +```
112 +
113 +Inspect:
114 +
115 +```bash
116 +cat /output/sweep/recommendation.json
117 +```
118 +
119 +Deploy the measured result:
120 +
121 +```bash
122 +source /output/env.sh
123 +vllm serve --config /output/sweep/recommended-config.yml
124 +```
tools/recipes/hardware_detection.py new
+115
@@ -0,0 +1,115 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +
4 +"""CPU hardware discovery for optional vLLM Recipes runtime tuning.
5 +
6 +Keep this module focused on facts exposed to the current process/container.
7 +Tuning decisions belong in runtime_tuning.py.
8 +"""
9 +
10 +from __future__ import annotations
11 +
12 +import json
13 +import platform
14 +from dataclasses import asdict, dataclass
15 +from pathlib import Path
16 +
17 +from vllm.utils.cpu_resource_utils import (
18 + get_allowed_cpu_list,
19 + get_memory_affinity,
20 + get_memory_node_info,
21 +)
22 +
23 +
24 +@dataclass(frozen=True)
25 +class NumaNodeInfo:
26 + node_id: int
27 + logical_cpu_ids: tuple[int, ...]
28 + physical_core_count: int
29 + total_memory_bytes: int
30 + available_memory_bytes: int
31 +
32 +
33 +@dataclass(frozen=True)
34 +class HardwareInfo:
35 + architecture: str
36 + socket_count: int | None
37 + numa_nodes: tuple[NumaNodeInfo, ...]
38 + allowed_logical_cpu_count: int
39 + physical_core_count: int
40 +
41 + @property
42 + def numa_node_count(self) -> int:
43 + return len(self.numa_nodes)
44 +
45 + def to_dict(self) -> dict[str, object]:
46 + return asdict(self)
47 +
48 +
49 +def _read_socket_id(cpu_id: int) -> int | None:
50 + path = Path(f"/sys/devices/system/cpu/cpu{cpu_id}/topology/physical_package_id")
51 + try:
52 + return int(path.read_text().strip())
53 + except (OSError, ValueError):
54 + return None
55 +
56 +
57 +def detect_hardware() -> HardwareInfo:
58 + """Return CPU resources effectively available to this process.
59 +
60 + vLLM's CPU helpers already honor process CPU affinity and cgroup-aware
61 + memory limits. We intentionally use get_memory_affinity() rather than
62 + get_visible_memory_node() so CPU_VISIBLE_MEMORY_NODES does not become an
63 + input to the detector; auto binding remains owned by the CPU backend.
64 + """
65 + cpus = get_allowed_cpu_list()
66 + if not cpus:
67 + raise RuntimeError("vLLM reported no CPUs available to this process.")
68 +
69 + cpu_numa_nodes = {cpu.numa_node for cpu in cpus if cpu.numa_node >= 0}
70 + memory_numa_nodes = set(get_memory_affinity())
71 + effective_numa_nodes = sorted(cpu_numa_nodes & memory_numa_nodes)
72 + if not effective_numa_nodes:
73 + effective_numa_nodes = sorted(cpu_numa_nodes)
74 +
75 + physical_cores = {
76 + (cpu.numa_node, cpu.physical_core) for cpu in cpus if cpu.physical_core >= 0
77 + }
78 +
79 + numa_nodes = []
80 + for node_id in effective_numa_nodes:
81 + node_cpus = [cpu for cpu in cpus if cpu.numa_node == node_id]
82 + node_physical_cores = {
83 + cpu.physical_core for cpu in node_cpus if cpu.physical_core >= 0
84 + }
85 + memory = get_memory_node_info(node_id)
86 + numa_nodes.append(
87 + NumaNodeInfo(
88 + node_id=node_id,
89 + logical_cpu_ids=tuple(sorted(cpu.id for cpu in node_cpus)),
90 + physical_core_count=len(node_physical_cores),
91 + total_memory_bytes=memory.total_memory,
92 + available_memory_bytes=memory.available_memory,
93 + )
94 + )
95 +
96 + socket_ids = {
97 + socket_id for cpu in cpus if (socket_id := _read_socket_id(cpu.id)) is not None
98 + }
99 +
100 + return HardwareInfo(
101 + architecture=platform.machine(),
102 + socket_count=len(socket_ids) if socket_ids else None,
103 + numa_nodes=tuple(numa_nodes),
104 + allowed_logical_cpu_count=len(cpus),
105 + physical_core_count=len(physical_cores),
106 + )
107 +
108 +
109 +def main() -> int:
110 + print(json.dumps(detect_hardware().to_dict(), indent=2))
111 + return 0
112 +
113 +
114 +if __name__ == "__main__":
115 + raise SystemExit(main())
tools/recipes/recipe_json_to_vllm_config.py
+162 -1
@@ -57,7 +57,6 @@ try:
57 except ImportError as exc:
58 raise SystemExit("PyYAML is required. Install it with: pip install pyyaml") from exc
59
60 -
60 DEFAULT_API_BASE = "https://recipes.vllm.ai"
61
62 SHORT_ALIASES = {
@@ -117,6 +116,71 @@ def parse_args() -> argparse.Namespace:
116 default="env.sh",
117 help="Output shell environment file (default: env.sh)",
118 )
119 +
120 + tuning = p.add_argument_group(
121 + "optional runtime tuning",
122 + (
123 + "Refine the recipe baseline only when additional information is supplied. "
124 + "Runtime tuning is enabled only for recipe hardware with a registered "
125 + "policy (currently: xeon6)."
126 + ),
127 + )
128 + tuning.add_argument(
129 + "--detect-hardware",
130 + action="store_true",
131 + help=(
132 + "Detect effective CPU/NUMA/memory resources and allow the selected "
133 + "recipe-hardware policy to override recipe runtime arguments."
134 + ),
135 + )
136 + tuning.add_argument(
137 + "--input-tokens",
138 + type=int,
139 + help="Expected input-token length. Optional workload hint.",
140 + )
141 + tuning.add_argument(
142 + "--output-tokens",
143 + type=int,
144 + help="Expected output-token length. Optional workload hint.",
145 + )
146 + tuning.add_argument(
147 + "--concurrency",
148 + type=int,
149 + help="Expected maximum concurrent requests. Optional workload hint.",
150 + )
151 + tuning.add_argument(
152 + "--ttft-sla-ms",
153 + type=float,
154 + help="Optional time-to-first-token objective in milliseconds.",
155 + )
156 + tuning.add_argument(
157 + "--tpot-sla-ms",
158 + type=float,
159 + help="Optional time-per-output-token objective in milliseconds.",
160 + )
161 + tuning.add_argument(
162 + "--target-qps",
163 + type=float,
164 + help="Optional capacity target for future DP/capacity tuning.",
165 + )
166 +
167 + sweep = p.add_argument_group(
168 + "optional performance sweep",
169 + ("Generate benchmark files after creating one initial runtime suggestion."),
170 + )
171 + sweep.add_argument(
172 + "--generate-sweep",
173 + action="store_true",
174 + help=(
175 + "Generate an optional vllm bench sweep package around the single "
176 + "initial runtime suggestion."
177 + ),
178 + )
179 + sweep.add_argument(
180 + "--sweep-out-dir",
181 + default="sweep",
182 + help="Output directory for optional sweep files (default: sweep).",
183 + )
184 return p.parse_args()
185
186
@@ -642,18 +706,115 @@ def main() -> int:
706
707 argv = recipe_argv(recipe)
708 config = argv_to_config(argv)
709 +
710 + tuning_requested = (
711 + args.detect_hardware
712 + or args.generate_sweep
713 + or any(
714 + value is not None
715 + for value in (
716 + args.input_tokens,
717 + args.output_tokens,
718 + args.concurrency,
719 + args.ttft_sla_ms,
720 + args.tpot_sla_ms,
721 + args.target_qps,
722 + )
723 + )
724 + )
725 +
726 + tuning = None
727 + workload = None
728 + sweep_writer = None
729 + if tuning_requested:
730 + # Keep plain Recipes conversion lightweight. vLLM-specific modules
731 + # are imported only for optional runtime tuning or sweep generation.
732 + from runtime_tuning import (
733 + WorkloadHints,
734 + finetune_runtime_config,
735 + get_runtime_tuning_policies,
736 + )
737 +
738 + workload = WorkloadHints(
739 + input_tokens=args.input_tokens,
740 + output_tokens=args.output_tokens,
741 + concurrency=args.concurrency,
742 + ttft_sla_ms=args.ttft_sla_ms,
743 + tpot_sla_ms=args.tpot_sla_ms,
744 + target_qps=args.target_qps,
745 + )
746 +
747 + if args.generate_sweep:
748 + from sweep_generation import (
749 + validate_sweep_workload,
750 + write_sweep_files,
751 + )
752 +
753 + validate_sweep_workload(workload)
754 + sweep_writer = write_sweep_files
755 +
756 + recipe_hardware = recipe.get("hardware")
757 + policies = get_runtime_tuning_policies(recipe_hardware)
758 +
759 + hardware = None
760 + if args.detect_hardware:
761 + from hardware_detection import detect_hardware
762 +
763 + hardware = detect_hardware()
764 +
765 + tuning = finetune_runtime_config(
766 + config,
767 + hardware=hardware,
768 + workload=workload,
769 + policies=policies,
770 + )
771 + config.update(tuning.overrides)
772 +
773 write_config(args.config_out, source, recipe, config)
774 write_env(args.env_out, source, recipe)
775 +
776 + sweep_files: list[Path] = []
777 + if args.generate_sweep:
778 + assert workload is not None
779 + assert sweep_writer is not None
780 + sweep_files = sweep_writer(
781 + args.sweep_out_dir,
782 + config_path=args.config_out,
783 + env_path=args.env_out,
784 + config=config,
785 + workload=workload,
786 + )
787 +
788 + if tuning is not None:
789 + if tuning.overrides:
790 + print("Initial runtime suggestion:")
791 + for key, value in tuning.overrides.items():
792 + print(f" {key}: {value}")
793 + for note in tuning.notes:
794 + print(f" tuning: {note}")
795 except Exception as exc:
796 print(f"ERROR: {exc}", file=sys.stderr)
797 return 1
798
799 print(f"Wrote {args.config_out}")
800 print(f"Wrote {args.env_out}")
801 + if sweep_files:
802 + print(f"Wrote optional sweep package under {args.sweep_out_dir}/")
803 print()
804 print("Run:")
805 print(f" source {shlex.quote(args.env_out)}")
806 print(f" vllm serve --config {shlex.quote(args.config_out)}")
807 + if sweep_files:
808 + sweep_dir = Path(args.sweep_out_dir)
809 + run_sweep = sweep_dir / "run_sweep.sh"
810 + recommend = sweep_dir / "recommend.py"
811 + print()
812 + print("Optional performance sweep:")
813 + print(f" {shlex.quote(str(run_sweep))} --dry-run")
814 + print(f" {shlex.quote(str(run_sweep))}")
815 + print()
816 + print("After the sweep:")
817 + print(f" {shlex.quote(str(recommend))}")
818 return 0
819
820
tools/recipes/runtime_tuning.py new
+314
@@ -0,0 +1,314 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +
4 +"""Optional runtime refinement for configs produced from vLLM Recipes.
5 +
6 +The recipe remains the baseline. Each policy may return an override only when
7 +enough information is available. Keep policies independent so the tuning
8 +algorithms can evolve without changing the converter or hardware detector.
9 +
10 +The policies in this draft are intentionally simple heuristics, not benchmark-
11 +proven universal optima.
12 +"""
13 +
14 +from __future__ import annotations
15 +
16 +from collections.abc import Callable
17 +from dataclasses import dataclass, field
18 +from math import ceil
19 +from typing import Any
20 +
21 +from hardware_detection import HardwareInfo
22 +
23 +from vllm.config.scheduler import SchedulerConfig
24 +
25 +
26 +@dataclass(frozen=True)
27 +class WorkloadHints:
28 + input_tokens: int | None = None
29 + output_tokens: int | None = None
30 + concurrency: int | None = None
31 + ttft_sla_ms: float | None = None
32 + tpot_sla_ms: float | None = None
33 + target_qps: float | None = None
34 +
35 + def validate(self) -> None:
36 + for name in ("input_tokens", "output_tokens", "concurrency"):
37 + value = getattr(self, name)
38 + if value is not None and value <= 0:
39 + raise ValueError(f"{name} must be greater than zero")
40 +
41 + for name in ("ttft_sla_ms", "tpot_sla_ms", "target_qps"):
42 + value = getattr(self, name)
43 + if value is not None and value <= 0:
44 + raise ValueError(f"{name} must be greater than zero")
45 +
46 +
47 +@dataclass
48 +class TuningResult:
49 + overrides: dict[str, Any] = field(default_factory=dict)
50 + notes: list[str] = field(default_factory=list)
51 +
52 +
53 +Policy = Callable[
54 + [dict[str, Any], HardwareInfo | None, WorkloadHints, TuningResult],
55 + None,
56 +]
57 +
58 +
59 +def _record_override(
60 + result: TuningResult,
61 + key: str,
62 + value: Any,
63 + reason: str,
64 +) -> None:
65 + result.overrides[key] = value
66 + result.notes.append(f"{key}={value!r}: {reason}")
67 +
68 +
69 +def _get_active_sequence_count(
70 + config: dict[str, Any],
71 + workload: WorkloadHints,
72 +) -> int:
73 + """Return the sequence count used to size one scheduler iteration."""
74 + if workload.concurrency is not None:
75 + return workload.concurrency
76 +
77 + configured = config.get("max-num-seqs")
78 + if (
79 + isinstance(configured, int)
80 + and not isinstance(configured, bool)
81 + and configured > 0
82 + ):
83 + return configured
84 +
85 + return SchedulerConfig.DEFAULT_MAX_NUM_SEQS
86 +
87 +
88 +def _estimate_prefills_per_step(
89 + workload: WorkloadHints,
90 + active_sequences: int,
91 +) -> float:
92 + """Estimate steady-state prompt arrivals per decode scheduler step."""
93 + # Reserve enough prefill budget for at least one representative prompt.
94 + prefills_per_step = 1.0
95 +
96 + # In steady state, active_sequences / output_tokens approximates how many
97 + # requests finish and are replaced by new prefills per decode step.
98 + if workload.output_tokens is not None:
99 + prefills_per_step = max(
100 + prefills_per_step,
101 + active_sequences / workload.output_tokens,
102 + )
103 +
104 + # With target QPS + TPOT, approximate how many prompts arrive during one
105 + # decode scheduler-step interval.
106 + if workload.target_qps is not None and workload.tpot_sla_ms is not None:
107 + prefills_per_step = max(
108 + prefills_per_step,
109 + workload.target_qps * workload.tpot_sla_ms / 1000.0,
110 + )
111 +
112 + # A step cannot replace more requests than are active.
113 + return min(float(active_sequences), prefills_per_step)
114 +
115 +
116 +def _resolve_tensor_parallel_size(
117 + config: dict[str, Any],
118 + hardware: HardwareInfo | None,
119 + workload: WorkloadHints,
120 + result: TuningResult,
121 +) -> None:
122 + del config, workload
123 + if hardware is None or hardware.numa_node_count < 1:
124 + return
125 +
126 + # Draft policy: use the largest power-of-two TP that does not exceed the
127 + # effective NUMA-node count. This deliberately avoids hard-coding unusual
128 + # non-power-of-two topologies into the converter.
129 + candidate = 1 << (hardware.numa_node_count.bit_length() - 1)
130 + _record_override(
131 + result,
132 + "tensor-parallel-size",
133 + candidate,
134 + "derived from the effective NUMA-node count",
135 + )
136 +
137 +
138 +def _resolve_gpu_memory_utilization(
139 + config: dict[str, Any],
140 + hardware: HardwareInfo | None,
141 + workload: WorkloadHints,
142 + result: TuningResult,
143 +) -> None:
144 + del workload
145 + if hardware is None or not hardware.numa_nodes:
146 + return
147 +
148 + reserve_fraction = 0.10
149 + safe_fractions = [
150 + max(
151 + 0.0,
152 + node.available_memory_bytes / node.total_memory_bytes - reserve_fraction,
153 + )
154 + for node in hardware.numa_nodes
155 + if node.total_memory_bytes > 0
156 + ]
157 + if not safe_fractions:
158 + return
159 +
160 + safe_fraction = round(min(0.90, min(safe_fractions)), 2)
161 + if safe_fraction <= 0:
162 + return
163 +
164 + recipe_value = config.get("gpu-memory-utilization")
165 + if isinstance(recipe_value, (int, float)):
166 + candidate = min(float(recipe_value), safe_fraction)
167 + else:
168 + candidate = min(0.80, safe_fraction)
169 +
170 + _record_override(
171 + result,
172 + "gpu-memory-utilization",
173 + round(candidate, 2),
174 + "constrained by effective per-NUMA available memory",
175 + )
176 +
177 +
178 +def _resolve_max_num_seqs(
179 + config: dict[str, Any],
180 + hardware: HardwareInfo | None,
181 + workload: WorkloadHints,
182 + result: TuningResult,
183 +) -> None:
184 + del config, hardware
185 + if workload.concurrency is None:
186 + return
187 +
188 + _record_override(
189 + result,
190 + "max-num-seqs",
191 + workload.concurrency,
192 + "derived from the optional target concurrency",
193 + )
194 +
195 +
196 +def _resolve_max_num_batched_tokens(
197 + config: dict[str, Any],
198 + hardware: HardwareInfo | None,
199 + workload: WorkloadHints,
200 + result: TuningResult,
201 +) -> None:
202 + del hardware
203 + if workload.input_tokens is None:
204 + return
205 +
206 + active_sequences = _get_active_sequence_count(config, workload)
207 +
208 + # Decode requests consume roughly one token per active sequence in one
209 + # scheduler iteration. Prefills share the remaining scheduler token budget.
210 + decode_budget = active_sequences
211 + prefills_per_step = _estimate_prefills_per_step(workload, active_sequences)
212 + prefill_budget = ceil(workload.input_tokens * prefills_per_step)
213 +
214 + # Use vLLM's scheduler defaults/constraints as floors rather than local
215 + # recipe-specific magic constants.
216 + candidate = max(
217 + SchedulerConfig.DEFAULT_MAX_NUM_BATCHED_TOKENS,
218 + active_sequences,
219 + decode_budget + prefill_budget,
220 + )
221 +
222 + # Chunked prefill normally lets a prompt span scheduler iterations. If the
223 + # recipe explicitly disables it, vLLM requires the token budget to cover
224 + # max_model_len when max_model_len is known.
225 + if config.get("enable-chunked-prefill") is False:
226 + max_model_len = config.get("max-model-len")
227 + if (
228 + isinstance(max_model_len, int)
229 + and not isinstance(max_model_len, bool)
230 + and max_model_len > 0
231 + ):
232 + candidate = max(candidate, max_model_len)
233 +
234 + _record_override(
235 + result,
236 + "max-num-batched-tokens",
237 + candidate,
238 + (
239 + "workload-derived scheduler budget "
240 + f"(active_sequences={active_sequences}, "
241 + f"prefills_per_step={prefills_per_step:.2f}, "
242 + f"decode_budget={decode_budget}, "
243 + f"prefill_budget={prefill_budget})"
244 + ),
245 + )
246 +
247 +
248 +def _resolve_data_parallel_size(
249 + config: dict[str, Any],
250 + hardware: HardwareInfo | None,
251 + workload: WorkloadHints,
252 + result: TuningResult,
253 +) -> None:
254 + del config, hardware
255 + if workload.target_qps is None:
256 + return
257 +
258 + result.notes.append(
259 + "data-parallel-size: target QPS supplied; kept recipe value because "
260 + "per-replica capacity is not known yet"
261 + )
262 +
263 +
264 +DEFAULT_POLICIES: tuple[Policy, ...] = (
265 + _resolve_tensor_parallel_size,
266 + _resolve_gpu_memory_utilization,
267 + _resolve_max_num_seqs,
268 + _resolve_max_num_batched_tokens,
269 + _resolve_data_parallel_size,
270 +)
271 +
272 +# Select tuning behavior from the hardware requested by the Recipes rendering,
273 +# not from devices physically present in the host. A GPU server still exposes
274 +# its host CPU topology, so host inspection alone cannot determine deployment
275 +# intent.
276 +HARDWARE_TUNING_POLICIES: dict[str, tuple[Policy, ...]] = {
277 + "xeon6": DEFAULT_POLICIES,
278 +}
279 +
280 +
281 +def get_runtime_tuning_policies(
282 + recipe_hardware: object,
283 +) -> tuple[Policy, ...]:
284 + if not isinstance(recipe_hardware, str) or not recipe_hardware:
285 + raise ValueError(
286 + "Runtime tuning requires the resolved Recipes JSON to declare "
287 + "a non-empty `hardware` field."
288 + )
289 +
290 + policies = HARDWARE_TUNING_POLICIES.get(recipe_hardware)
291 + if policies is None:
292 + supported = ", ".join(sorted(HARDWARE_TUNING_POLICIES))
293 + raise ValueError(
294 + "Runtime tuning is not supported for recipe hardware "
295 + f"{recipe_hardware!r}. Currently supported: {supported}."
296 + )
297 + return policies
298 +
299 +
300 +def finetune_runtime_config(
301 + config: dict[str, Any],
302 + *,
303 + hardware: HardwareInfo | None = None,
304 + workload: WorkloadHints | None = None,
305 + policies: tuple[Policy, ...] = DEFAULT_POLICIES,
306 +) -> TuningResult:
307 + """Return overrides; never mutate the recipe-derived config in place."""
308 + hints = workload or WorkloadHints()
309 + hints.validate()
310 +
311 + result = TuningResult()
312 + for policy in policies:
313 + policy(config, hardware, hints, result)
314 + return result
tools/recipes/sweep_generation.py new
+341
@@ -0,0 +1,341 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +
4 +"""Generate optional vLLM benchmark sweep files from one initial suggestion."""
5 +
6 +from __future__ import annotations
7 +
8 +import json
9 +import os
10 +import shlex
11 +from pathlib import Path
12 +from typing import Any
13 +
14 +from runtime_tuning import WorkloadHints
15 +
16 +
17 +def _positive_int(config: dict[str, Any], key: str) -> int:
18 + value = config.get(key)
19 + if not isinstance(value, int) or isinstance(value, bool) or value <= 0:
20 + raise ValueError(f"Sweep generation requires a positive initial {key!r} value.")
21 + return value
22 +
23 +
24 +def _strict_lower_power_of_two(value: int) -> int:
25 + if value <= 1:
26 + return 1
27 + return 1 << ((value - 1).bit_length() - 1)
28 +
29 +
30 +def _strict_upper_power_of_two(value: int) -> int:
31 + return 1 << value.bit_length()
32 +
33 +
34 +def validate_sweep_workload(workload: WorkloadHints) -> None:
35 + required = {
36 + "--input-tokens": workload.input_tokens,
37 + "--output-tokens": workload.output_tokens,
38 + "--concurrency": workload.concurrency,
39 + }
40 + missing = [name for name, value in required.items() if value is None]
41 + if missing:
42 + raise ValueError("--generate-sweep requires " + ", ".join(missing) + ".")
43 +
44 +
45 +def build_serve_params(config: dict[str, Any]) -> list[dict[str, Any]]:
46 + """Build a small directed sweep around the single initial suggestion."""
47 + initial_seqs = _positive_int(config, "max-num-seqs")
48 + initial_batch = _positive_int(config, "max-num-batched-tokens")
49 +
50 + minimum_batch = initial_seqs
51 + if config.get("enable-chunked-prefill") is False:
52 + max_model_len = config.get("max-model-len")
53 + if (
54 + isinstance(max_model_len, int)
55 + and not isinstance(max_model_len, bool)
56 + and max_model_len > 0
57 + ):
58 + minimum_batch = max(minimum_batch, max_model_len)
59 +
60 + lower_seqs = max(1, (initial_seqs + 1) // 2)
61 + lower_batch = max(
62 + minimum_batch,
63 + _strict_lower_power_of_two(initial_batch),
64 + )
65 + higher_batch = max(
66 + minimum_batch,
67 + _strict_upper_power_of_two(initial_batch),
68 + )
69 +
70 + candidates: list[dict[str, Any]] = []
71 + seen: set[tuple[int, int]] = set()
72 +
73 + def add(name: str, max_num_seqs: int, max_num_batched_tokens: int) -> None:
74 + max_num_batched_tokens = max(
75 + max_num_batched_tokens,
76 + max_num_seqs,
77 + )
78 + signature = (max_num_seqs, max_num_batched_tokens)
79 + if signature in seen:
80 + return
81 + seen.add(signature)
82 + candidates.append(
83 + {
84 + "_benchmark_name": name,
85 + "max_num_seqs": max_num_seqs,
86 + "max_num_batched_tokens": max_num_batched_tokens,
87 + }
88 + )
89 +
90 + # Keep the exact initial suggestion as the measured baseline. Additional
91 + # values exist only in the optional sweep package.
92 + add("initial", initial_seqs, initial_batch)
93 + add("lower_scheduler_concurrency", lower_seqs, initial_batch)
94 + add("lower_batch_budget", initial_seqs, lower_batch)
95 + add("latency_focused", lower_seqs, lower_batch)
96 + add("higher_batch_budget", initial_seqs, higher_batch)
97 +
98 + return candidates
99 +
100 +
101 +def build_bench_params(workload: WorkloadHints) -> list[dict[str, Any]]:
102 + validate_sweep_workload(workload)
103 + assert workload.input_tokens is not None
104 + assert workload.output_tokens is not None
105 + assert workload.concurrency is not None
106 +
107 + return [
108 + {
109 + "_benchmark_name": "user_workload",
110 + "random_input_len": workload.input_tokens,
111 + "random_output_len": workload.output_tokens,
112 + "max_concurrency": workload.concurrency,
113 + }
114 + ]
115 +
116 +
117 +def _benchmark_models(config: dict[str, Any]) -> tuple[str, str]:
118 + model = config.get("model")
119 + if not isinstance(model, str) or not model:
120 + raise ValueError("Sweep generation requires a model in config.yml.")
121 +
122 + served_model_name = config.get("served-model-name")
123 + if isinstance(served_model_name, str) and served_model_name:
124 + return served_model_name, model
125 + if (
126 + isinstance(served_model_name, list)
127 + and served_model_name
128 + and isinstance(served_model_name[0], str)
129 + ):
130 + return served_model_name[0], model
131 +
132 + return model, model
133 +
134 +
135 +def _relative_to(directory: Path, target: str) -> str:
136 + return os.path.relpath(Path(target).resolve(), directory.resolve())
137 +
138 +
139 +def _write_json(path: Path, value: object) -> None:
140 + path.write_text(
141 + json.dumps(value, indent=2) + "\n",
142 + encoding="utf-8",
143 + )
144 +
145 +
146 +def _write_run_script(
147 + path: Path,
148 + *,
149 + config_rel: str,
150 + env_rel: str,
151 + request_model: str,
152 + tokenizer: str,
153 + workload: WorkloadHints,
154 +) -> None:
155 + bench_parts = [
156 + "vllm bench serve",
157 + "--backend vllm",
158 + f"--model {shlex.quote(request_model)}",
159 + f"--tokenizer {shlex.quote(tokenizer)}",
160 + "--dataset-name random",
161 + "--request-rate inf",
162 + "--ignore-eos",
163 + "--metric-percentiles 99",
164 + ]
165 +
166 + goodput_pairs: list[str] = []
167 + if workload.ttft_sla_ms is not None:
168 + goodput_pairs.append(f"ttft:{workload.ttft_sla_ms:g}")
169 + if workload.tpot_sla_ms is not None:
170 + goodput_pairs.append(f"tpot:{workload.tpot_sla_ms:g}")
171 + if goodput_pairs:
172 + bench_parts.append(
173 + "--goodput " + " ".join(shlex.quote(pair) for pair in goodput_pairs)
174 + )
175 +
176 + bench_cmd = " ".join(bench_parts)
177 + script = f"""#!/usr/bin/env bash
178 +set -euo pipefail
179 +
180 +SCRIPT_DIR="$(cd -- "$(dirname -- "${{BASH_SOURCE[0]}}")" && pwd)"
181 +CONFIG_PATH="${{SCRIPT_DIR}}/{config_rel}"
182 +ENV_PATH="${{SCRIPT_DIR}}/{env_rel}"
183 +
184 +source "${{ENV_PATH}}"
185 +
186 +vllm bench sweep serve \
187 + --serve-cmd "vllm serve --config '${{CONFIG_PATH}}'" \
188 + --bench-cmd "{bench_cmd}" \
189 + --serve-params "${{SCRIPT_DIR}}/serve_params.json" \
190 + --bench-params "${{SCRIPT_DIR}}/bench_params.json" \
191 + --output-dir "${{SCRIPT_DIR}}/results" \
192 + --experiment-name runtime-tuning \
193 + "$@"
194 +"""
195 + path.write_text(script, encoding="utf-8")
196 + path.chmod(path.stat().st_mode | 0o111)
197 +
198 +
199 +def _write_recommend_script(
200 + path: Path,
201 + *,
202 + config_rel: str,
203 + env_rel: str,
204 + workload: WorkloadHints,
205 +) -> None:
206 + template_path = Path(__file__).with_name("sweep_recommendation.py")
207 + source = template_path.read_text(encoding="utf-8")
208 +
209 + replacements = {
210 + "DEFAULT_CONFIG_PATH: str | None = None": (
211 + f"DEFAULT_CONFIG_PATH: str | None = {config_rel!r}"
212 + ),
213 + "DEFAULT_ENV_PATH: str | None = None": (
214 + f"DEFAULT_ENV_PATH: str | None = {env_rel!r}"
215 + ),
216 + "DEFAULT_TTFT_SLA_MS: float | None = None": (
217 + f"DEFAULT_TTFT_SLA_MS: float | None = {workload.ttft_sla_ms!r}"
218 + ),
219 + "DEFAULT_TPOT_SLA_MS: float | None = None": (
220 + f"DEFAULT_TPOT_SLA_MS: float | None = {workload.tpot_sla_ms!r}"
221 + ),
222 + }
223 + for marker, replacement in replacements.items():
224 + if marker not in source:
225 + raise ValueError(f"Recommender template marker not found: {marker}")
226 + source = source.replace(marker, replacement, 1)
227 +
228 + path.write_text(source, encoding="utf-8")
229 + path.chmod(path.stat().st_mode | 0o111)
230 +
231 +
232 +def _write_guide(path: Path, workload: WorkloadHints) -> None:
233 + sla_lines: list[str] = []
234 + if workload.ttft_sla_ms is not None:
235 + sla_lines.append(f"- TTFT objective: `{workload.ttft_sla_ms:g} ms`")
236 + if workload.tpot_sla_ms is not None:
237 + sla_lines.append(f"- TPOT objective: `{workload.tpot_sla_ms:g} ms`")
238 +
239 + sla_text = "\n".join(sla_lines)
240 + if sla_text:
241 + sla_text = (
242 + "\n## Supplied latency objectives\n\n"
243 + + sla_text
244 + + "\n\nThe generated benchmark uses these values with vLLM "
245 + "`--goodput`.\n"
246 + )
247 +
248 + content = f"""# Optional Runtime Tuning Sweep
249 +
250 +The generated `config.yml` is the **single initial suggestion** and can be
251 +deployed directly. The sweep benchmarks nearby values for:
252 +
253 +- `max-num-seqs`
254 +- `max-num-batched-tokens`
255 +
256 +## Run
257 +
258 +```bash
259 +./run_sweep.sh --dry-run
260 +./run_sweep.sh
261 +./recommend.py
262 +```
263 +
264 +For a quick one-run experiment:
265 +
266 +```bash
267 +rm -rf results/runtime-tuning
268 +./run_sweep.sh --num-runs 1
269 +./recommend.py
270 +```
271 +
272 +Resume an interrupted sweep:
273 +
274 +```bash
275 +./run_sweep.sh --resume
276 +```
277 +
278 +By default, vLLM benchmarks each parameter combination three times.
279 +{sla_text}
280 +## Outputs
281 +
282 +`recommend.py` writes:
283 +
284 +```text
285 +recommended-config.yml
286 +recommendation.json
287 +```
288 +
289 +With TTFT/TPOT objectives it selects highest mean request goodput across
290 +repeated runs. Without latency objectives it selects highest mean output-token
291 +throughput. Configurations with failed requests are excluded.
292 +
293 +`recommended-config.yml` copies the initial configuration and changes only
294 +`max-num-seqs` and `max-num-batched-tokens`.
295 +"""
296 + path.write_text(content, encoding="utf-8")
297 +
298 +
299 +def write_sweep_files(
300 + output_dir: str,
301 + *,
302 + config_path: str,
303 + env_path: str,
304 + config: dict[str, Any],
305 + workload: WorkloadHints,
306 +) -> list[Path]:
307 + """Write an optional sweep package around the initial config suggestion."""
308 + validate_sweep_workload(workload)
309 +
310 + directory = Path(output_dir)
311 + directory.mkdir(parents=True, exist_ok=True)
312 +
313 + serve_params = directory / "serve_params.json"
314 + bench_params = directory / "bench_params.json"
315 + run_script = directory / "run_sweep.sh"
316 + recommend_script = directory / "recommend.py"
317 + guide = directory / "SWEEP.md"
318 +
319 + config_rel = _relative_to(directory, config_path)
320 + env_rel = _relative_to(directory, env_path)
321 +
322 + _write_json(serve_params, build_serve_params(config))
323 + _write_json(bench_params, build_bench_params(workload))
324 + request_model, tokenizer = _benchmark_models(config)
325 + _write_run_script(
326 + run_script,
327 + config_rel=config_rel,
328 + env_rel=env_rel,
329 + request_model=request_model,
330 + tokenizer=tokenizer,
331 + workload=workload,
332 + )
333 + _write_recommend_script(
334 + recommend_script,
335 + config_rel=config_rel,
336 + env_rel=env_rel,
337 + workload=workload,
338 + )
339 + _write_guide(guide, workload)
340 +
341 + return [serve_params, bench_params, run_script, recommend_script, guide]
tools/recipes/sweep_recommendation.py new
+329
@@ -0,0 +1,329 @@
1 +#!/usr/bin/env python3
2 +# SPDX-License-Identifier: Apache-2.0
3 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
4 +
5 +"""Select one measured scheduler recommendation from vLLM sweep results."""
6 +
7 +from __future__ import annotations
8 +
9 +import argparse
10 +import json
11 +from collections import defaultdict
12 +from pathlib import Path
13 +from statistics import mean
14 +from typing import Any
15 +
16 +try:
17 + import yaml
18 +except ImportError as exc:
19 + raise SystemExit("PyYAML is required. Install it with: pip install pyyaml") from exc
20 +
21 +
22 +DEFAULT_CONFIG_PATH: str | None = None
23 +DEFAULT_ENV_PATH: str | None = None
24 +DEFAULT_TTFT_SLA_MS: float | None = None
25 +DEFAULT_TPOT_SLA_MS: float | None = None
26 +
27 +
28 +def _number(value: object) -> float | None:
29 + if isinstance(value, bool) or not isinstance(value, (int, float)):
30 + return None
31 + return float(value)
32 +
33 +
34 +def _mean(rows: list[dict[str, Any]], key: str) -> float | None:
35 + values = [_number(row.get(key)) for row in rows]
36 + numbers = [value for value in values if value is not None]
37 + return mean(numbers) if numbers else None
38 +
39 +
40 +def _positive_int(row: dict[str, Any], key: str) -> int | None:
41 + value = row.get(key)
42 + if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
43 + return None
44 + return value
45 +
46 +
47 +def _resolve(script_dir: Path, value: str | None) -> Path | None:
48 + if value is None:
49 + return None
50 + path = Path(value)
51 + return path if path.is_absolute() else script_dir / path
52 +
53 +
54 +def _load_runs(results_dir: Path) -> list[dict[str, Any]]:
55 + rows: list[dict[str, Any]] = []
56 + for path in sorted(results_dir.rglob("summary.json")):
57 + data = json.loads(path.read_text(encoding="utf-8"))
58 + if not isinstance(data, list):
59 + continue
60 + for row in data:
61 + if isinstance(row, dict):
62 + copied = dict(row)
63 + copied["_summary_path"] = str(path)
64 + rows.append(copied)
65 + return rows
66 +
67 +
68 +def _aggregate_candidates(
69 + rows: list[dict[str, Any]],
70 + *,
71 + use_goodput: bool,
72 +) -> list[dict[str, Any]]:
73 + grouped: dict[tuple[int, int], list[dict[str, Any]]] = defaultdict(list)
74 +
75 + for row in rows:
76 + seqs = _positive_int(row, "max_num_seqs")
77 + batch = _positive_int(row, "max_num_batched_tokens")
78 + if seqs is None or batch is None:
79 + continue
80 + grouped[(seqs, batch)].append(row)
81 +
82 + candidates: list[dict[str, Any]] = []
83 + for (seqs, batch), runs in sorted(grouped.items()):
84 + failed_requests = sum(int(_number(run.get("failed")) or 0) for run in runs)
85 + output_throughput = _mean(runs, "output_throughput")
86 + request_goodput = _mean(runs, "request_goodput")
87 + p99_ttft_ms = _mean(runs, "p99_ttft_ms")
88 + p99_tpot_ms = _mean(runs, "p99_tpot_ms")
89 +
90 + valid = failed_requests == 0 and output_throughput is not None
91 + reason = None
92 + if failed_requests:
93 + valid = False
94 + reason = f"{failed_requests} failed request(s) across repeated runs"
95 + elif use_goodput and request_goodput is None:
96 + valid = False
97 + reason = "request_goodput is missing; rerun sweep with SLO goodput enabled"
98 +
99 + candidates.append(
100 + {
101 + "max_num_seqs": seqs,
102 + "max_num_batched_tokens": batch,
103 + "run_count": len(runs),
104 + "failed_requests": failed_requests,
105 + "mean_request_goodput": request_goodput,
106 + "mean_output_throughput": output_throughput,
107 + "mean_p99_ttft_ms": p99_ttft_ms,
108 + "mean_p99_tpot_ms": p99_tpot_ms,
109 + "valid": valid,
110 + "invalid_reason": reason,
111 + "summary_files": sorted({str(run["_summary_path"]) for run in runs}),
112 + }
113 + )
114 +
115 + return candidates
116 +
117 +
118 +def _select_candidate(
119 + candidates: list[dict[str, Any]],
120 + *,
121 + use_goodput: bool,
122 +) -> tuple[dict[str, Any], str]:
123 + valid = [candidate for candidate in candidates if candidate["valid"]]
124 + if not valid:
125 + raise ValueError(
126 + "No valid sweep configuration was found. Inspect the sweep results "
127 + "for failed requests or missing metrics."
128 + )
129 +
130 + if use_goodput:
131 + objective = "highest_mean_request_goodput"
132 + winner = max(
133 + valid,
134 + key=lambda candidate: (
135 + candidate["mean_request_goodput"],
136 + candidate["mean_output_throughput"],
137 + -candidate["max_num_batched_tokens"],
138 + -candidate["max_num_seqs"],
139 + ),
140 + )
141 + if winner["mean_request_goodput"] <= 0:
142 + raise ValueError(
143 + "No sweep configuration produced non-zero request goodput for "
144 + "the supplied TTFT/TPOT objectives."
145 + )
146 + else:
147 + objective = "highest_mean_output_throughput"
148 + winner = max(
149 + valid,
150 + key=lambda candidate: (
151 + candidate["mean_output_throughput"],
152 + -candidate["max_num_batched_tokens"],
153 + -candidate["max_num_seqs"],
154 + ),
155 + )
156 +
157 + return winner, objective
158 +
159 +
160 +def _load_config(path: Path) -> dict[str, Any]:
161 + data = yaml.safe_load(path.read_text(encoding="utf-8"))
162 + if not isinstance(data, dict):
163 + raise ValueError(f"{path} does not contain a YAML configuration object.")
164 + return data
165 +
166 +
167 +def _write_config(
168 + path: Path,
169 + *,
170 + source_path: Path,
171 + config: dict[str, Any],
172 + objective: str,
173 +) -> None:
174 + body = yaml.safe_dump(
175 + config,
176 + sort_keys=False,
177 + default_flow_style=False,
178 + allow_unicode=True,
179 + )
180 + header = (
181 + "# Generated from the initial recipe config after a vLLM parameter sweep.\n"
182 + f"# Initial config: {source_path}\n"
183 + f"# Selection objective: {objective}\n"
184 + )
185 + path.write_text(header + body, encoding="utf-8")
186 +
187 +
188 +def parse_args() -> argparse.Namespace:
189 + parser = argparse.ArgumentParser(
190 + description=(
191 + "Select one recommended max-num-seqs/max-num-batched-tokens pair "
192 + "from generated vLLM sweep results."
193 + )
194 + )
195 + parser.add_argument(
196 + "--results-dir",
197 + default="results/runtime-tuning",
198 + help="Sweep experiment directory (default: results/runtime-tuning).",
199 + )
200 + parser.add_argument(
201 + "--config",
202 + default=DEFAULT_CONFIG_PATH,
203 + help="Initial config.yml used by the sweep.",
204 + )
205 + parser.add_argument(
206 + "--env",
207 + default=DEFAULT_ENV_PATH,
208 + help="Environment file to show in the deployment instructions.",
209 + )
210 + parser.add_argument(
211 + "--ttft-sla-ms",
212 + type=float,
213 + default=DEFAULT_TTFT_SLA_MS,
214 + help="TTFT objective used to generate request goodput.",
215 + )
216 + parser.add_argument(
217 + "--tpot-sla-ms",
218 + type=float,
219 + default=DEFAULT_TPOT_SLA_MS,
220 + help="TPOT objective used to generate request goodput.",
221 + )
222 + parser.add_argument(
223 + "--output-config",
224 + default="recommended-config.yml",
225 + help="Recommended config output path.",
226 + )
227 + parser.add_argument(
228 + "--output-json",
229 + default="recommendation.json",
230 + help="Recommendation evidence output path.",
231 + )
232 + return parser.parse_args()
233 +
234 +
235 +def main() -> int:
236 + args = parse_args()
237 + script_dir = Path(__file__).resolve().parent
238 +
239 + config_path = _resolve(script_dir, args.config)
240 + env_path = _resolve(script_dir, args.env)
241 + results_dir = _resolve(script_dir, args.results_dir)
242 + output_config = _resolve(script_dir, args.output_config)
243 + output_json = _resolve(script_dir, args.output_json)
244 +
245 + if config_path is None:
246 + raise ValueError("--config is required.")
247 + assert results_dir is not None
248 + assert output_config is not None
249 + assert output_json is not None
250 +
251 + if not results_dir.exists():
252 + raise ValueError(
253 + f"Sweep results were not found at {results_dir}. Run the sweep first."
254 + )
255 +
256 + rows = _load_runs(results_dir)
257 + if not rows:
258 + raise ValueError(f"No summary.json sweep results found under {results_dir}.")
259 +
260 + use_goodput = args.ttft_sla_ms is not None or args.tpot_sla_ms is not None
261 + candidates = _aggregate_candidates(rows, use_goodput=use_goodput)
262 + winner, objective = _select_candidate(candidates, use_goodput=use_goodput)
263 +
264 + initial_config = _load_config(config_path)
265 + recommended_config = dict(initial_config)
266 + recommended_config["max-num-seqs"] = winner["max_num_seqs"]
267 + recommended_config["max-num-batched-tokens"] = winner["max_num_batched_tokens"]
268 +
269 + _write_config(
270 + output_config,
271 + source_path=config_path,
272 + config=recommended_config,
273 + objective=objective,
274 + )
275 +
276 + recommendation = {
277 + "selection_objective": objective,
278 + "slo": {
279 + "ttft_ms": args.ttft_sla_ms,
280 + "tpot_ms": args.tpot_sla_ms,
281 + },
282 + "initial": {
283 + "max_num_seqs": initial_config.get("max-num-seqs"),
284 + "max_num_batched_tokens": initial_config.get("max-num-batched-tokens"),
285 + },
286 + "recommended": {
287 + "max_num_seqs": winner["max_num_seqs"],
288 + "max_num_batched_tokens": winner["max_num_batched_tokens"],
289 + },
290 + "measured": {
291 + "run_count": winner["run_count"],
292 + "mean_request_goodput": winner["mean_request_goodput"],
293 + "mean_output_throughput": winner["mean_output_throughput"],
294 + "mean_p99_ttft_ms": winner["mean_p99_ttft_ms"],
295 + "mean_p99_tpot_ms": winner["mean_p99_tpot_ms"],
296 + },
297 + "candidates": candidates,
298 + }
299 + output_json.write_text(
300 + json.dumps(recommendation, indent=2) + "\n",
301 + encoding="utf-8",
302 + )
303 +
304 + print("Recommended runtime configuration")
305 + print()
306 + print(f" max-num-seqs: {winner['max_num_seqs']}")
307 + print(f" max-num-batched-tokens: {winner['max_num_batched_tokens']}")
308 + print()
309 + print(f"Selection objective: {objective}")
310 + if use_goodput:
311 + print(f" mean request goodput: {winner['mean_request_goodput']:.2f} req/s")
312 + print(f" mean output throughput: {winner['mean_output_throughput']:.2f} tok/s")
313 + if winner["mean_p99_ttft_ms"] is not None:
314 + print(f" mean P99 TTFT: {winner['mean_p99_ttft_ms']:.2f} ms")
315 + if winner["mean_p99_tpot_ms"] is not None:
316 + print(f" mean P99 TPOT: {winner['mean_p99_tpot_ms']:.2f} ms")
317 + print()
318 + print(f"Wrote {output_config}")
319 + print(f"Wrote {output_json}")
320 + print()
321 + print("Deploy:")
322 + if env_path is not None:
323 + print(f" source {env_path}")
324 + print(f" vllm serve --config {output_config}")
325 + return 0
326 +
327 +
328 +if __name__ == "__main__":
329 + raise SystemExit(main())