@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 292 lines 9.57 KB
Rendered Raw
1 # Runtime Tuning
2
3 The Recipes JSON remains the baseline. vLLM Recipes already provide validated
4 model, hardware, strategy, environment variables, and serving arguments.
5 Runtime tuning is optional and is intended for parameters whose best value can
6 depend on the actual deployment resources or expected request workload.
7
8 ## Information Sources
9
10 Runtime tuning combines up to three inputs:
11
12 - **vLLM Recipe** — the required baseline from the Recipes JSON API or a direct
13 recipe JSON file.
14 - **Hardware information (optional)** — detected with `--detect-hardware`,
15 including effective CPU/NUMA topology and memory availability.
16 - **Workload information (optional)** — supplied through CLI hints such as input
17 and output token lengths, concurrency, TTFT/TPOT objectives, and target QPS.
18
19 If no optional hardware or workload information is supplied, the converter keeps
20 the normal Recipes conversion behavior.
21
22 See [Deployment-Time Parameters](#deployment-time-parameters) for the mapping
23 between these inputs and individual runtime parameters.
24
25 ## Hardware Information
26
27 Hardware detection is enabled only with `--detect-hardware`. It reports the
28 effective resources available to the process or container, including CPU/NUMA
29 topology and memory information. The current policy can use those values to
30 refine `tensor-parallel-size` and `gpu-memory-utilization`.
31
32 ```bash
33 python3 tools/recipes/recipe_json_to_vllm_config.py \
34 --model meta-llama/Llama-3.1-8B-Instruct \
35 --hardware xeon6 \
36 --detect-hardware
37 ```
38
39 Hardware detection is optional. Recipe hardware selects the tuning policy; host
40 hardware detection provides resource information to that policy.
41
42 ## Workload Information
43
44 Workload information is supplied explicitly because the converter runs before a
45 vLLM server exists. Supported hints include:
46
47 - `--input-tokens`
48 - `--output-tokens`
49 - `--concurrency`
50 - `--ttft-sla-ms`
51 - `--tpot-sla-ms`
52 - `--target-qps`
53
54 Example:
55
56 ```bash
57 python3 tools/recipes/recipe_json_to_vllm_config.py \
58 --model meta-llama/Llama-3.1-8B-Instruct \
59 --hardware xeon6 \
60 --input-tokens 128 \
61 --output-tokens 128 \
62 --concurrency 32 \
63 --ttft-sla-ms 3000 \
64 --tpot-sla-ms 100
65 ```
66
67 `--output-tokens` is used to estimate steady-state request turnover for
68 `max-num-batched-tokens`. When both `--target-qps` and `--tpot-sla-ms` are
69 available, they are also used to estimate prompt arrival pressure per scheduler
70 step. `--ttft-sla-ms` is collected but is not yet converted directly into a
71 batch-size formula because the relationship between TTFT and scheduler budget is
72 model- and hardware-dependent.
73
74 ## Deployment-Time Parameters
75
76 The parameters below may already exist in a recipe. They are candidates for
77 deployment-time refinement when the recipe value is missing, generic, or based
78 on a validation environment that differs from the user's target deployment.
79
80 | Runtime parameter | Why it may need deployment-time refinement | Main decision input | Current policy |
81 | --- | --- | --- | --- |
82 | `tensor-parallel-size` | The effective CPU/NUMA topology available to a container or pod can differ from the system used to validate the recipe. | Hardware topology | Use the largest power-of-two TP value that does not exceed the effective NUMA-node count. |
83 | `gpu-memory-utilization` | Available memory can differ by machine size, container limits, and other memory use. The vLLM option name is also used by the CPU backend. | Hardware memory + recipe baseline | Calculate a conservative fraction from the most constrained NUMA node. |
84 | `max-num-seqs` | The useful scheduler concurrency depends on the number of requests expected to be active at the same time. | Workload concurrency | Set `max-num-seqs` to `--concurrency` when supplied. |
85 | `max-num-batched-tokens` | Each scheduler iteration must share its token budget between active decodes and incoming prefills. | Input/output token shape, concurrency, and optional QPS/TPOT | Calculate decode budget + expected prefill demand, with vLLM scheduler constraints as floors. |
86 | `data-parallel-size` | The required replica count depends on the requested throughput and the capacity of one replica. | Capacity target | Keep the recipe value today because per-replica SLA capacity is not known. |
87
88 ## How Each Runtime Parameter Is Calculated
89
90 ### `tensor-parallel-size`
91
92 TP is derived from the effective NUMA topology reported by hardware detection:
93
94 ```text
95 TP = largest power of two <= effective NUMA-node count
96 ```
97
98 For example, 2 effective NUMA nodes produce TP=2, while 6 nodes currently
99 produce TP=4. This is a topology-based starting point and avoids automatically
100 selecting unusual non-power-of-two TP sizes before they are validated.
101
102 ### `gpu-memory-utilization`
103
104 For every effective NUMA memory node, the policy calculates:
105
106 ```text
107 node_safe_fraction = available_memory / total_memory - 0.10
108 safe_fraction = min(0.90, minimum node_safe_fraction)
109 ```
110
111 The 10% reserve leaves memory for model/runtime overhead outside the vLLM cache
112 budget. If the recipe already contains a smaller value, the policy preserves the
113 smaller value. If it is absent, `0.80` is used as the initial ceiling:
114
115 ```text
116 candidate = min(recipe value or 0.80, safe_fraction)
117 ```
118
119 ### `max-num-seqs`
120
121 When `--concurrency` is supplied, it directly represents the requested maximum
122 number of simultaneously active requests:
123
124 ```text
125 max-num-seqs = concurrency
126 ```
127
128 If concurrency is not supplied, the converter does not override this parameter.
129
130 ### `max-num-batched-tokens`
131
132 The token budget is workload-derived and no longer assumes a fixed number of
133 parallel prefills.
134
135 First, the policy determines the active sequence count:
136
137 ```text
138 active_sequences =
139 --concurrency
140 or recipe max-num-seqs
141 or vLLM default max_num_seqs (128)
142 ```
143
144 Decode requests consume approximately one token per active sequence in a
145 scheduler iteration:
146
147 ```text
148 decode_budget = active_sequences
149 ```
150
151 The policy then estimates how many new prompts need prefill work per scheduler
152 step:
153
154 ```text
155 prefills_per_step = max(
156 1,
157 active_sequences / output_tokens,
158 target_qps * tpot_sla_ms / 1000
159 )
160
161 prefills_per_step = min(active_sequences, prefills_per_step)
162 prefill_budget = ceil(input_tokens * prefills_per_step)
163 ```
164
165 The final scheduler budget is:
166
167 ```text
168 max-num-batched-tokens = max(
169 vLLM default max_num_batched_tokens (2048),
170 active_sequences,
171 decode_budget + prefill_budget
172 )
173 ```
174
175 For example, with 128 input tokens, 128 output tokens, and concurrency 32:
176
177 ```text
178 decode_budget = 32
179 prefills_per_step = max(1, 32 / 128) = 1
180 prefill_budget = 128
181 candidate = max(2048, 32, 32 + 128) = 2048
182 ```
183
184 If chunked prefill is explicitly disabled and `max-model-len` is available, the
185 policy also ensures:
186
187 ```text
188 max-num-batched-tokens >= max-model-len
189 ```
190
191 ### `data-parallel-size`
192
193 `--target-qps` alone is not enough to choose DP safely. A correct capacity rule
194 also needs measured per-replica throughput that still satisfies TTFT/TPOT:
195
196 ```text
197 DP = ceil(target_qps / qps_per_replica_at_SLO)
198 ```
199
200 Because the converter does not have that measured capacity yet, it intentionally
201 keeps the recipe DP value rather than guessing.
202
203 ## Precedence
204
205 ```text
206 vLLM defaults
207 -> vLLM Recipes baseline
208 -> hardware refinement (optional)
209 -> workload / SLO refinement (optional)
210 -> config.yml + env.sh
211 ```
212
213 ## Runtime-Tuning Hardware Scope
214
215 Runtime tuning is selected from the resolved recipe JSON's `hardware` field,
216 rather than by inspecting which physical devices happen to be present on the
217 host. This matters because a GPU server also exposes its host CPU topology.
218
219 The current runtime-tuning policy registry contains `xeon6`. A tuning request
220 for unregistered recipe hardware, such as `b200`, fails before host hardware
221 detection:
222
223 ```text
224 ERROR: Runtime tuning is not supported for recipe hardware 'b200'.
225 Currently supported: xeon6.
226 ```
227
228 Plain recipe conversion remains available for all recipe hardware. The gate only
229 applies when optional runtime-tuning inputs are requested.
230
231 ## vLLM CPU Docker Shell
232
233 Run hardware detection and runtime tuning inside the target vLLM CPU container
234 so detected CPU, NUMA, memory, and cgroup limits match the deployment
235 environment.
236
237 From the vLLM source tree:
238
239 ```bash
240 mkdir -p recipe-output
241
242 docker run --rm -it \
243 --entrypoint bash \
244 --security-opt seccomp=unconfined \
245 --cap-add SYS_NICE \
246 --shm-size=4g \
247 -p 8000:8000 \
248 -v "$PWD/tools/recipes:/recipes:ro" \
249 -v "$PWD/recipe-output:/output" \
250 -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
251 -w /output \
252 vllm/vllm-openai-cpu:latest-x86_64
253 ```
254
255 `/output` is writable; `/recipes` remains read-only.
256
257 Inside the container, generate one runtime-tuned initial configuration:
258
259 ```bash
260 python3 /recipes/recipe_json_to_vllm_config.py \
261 --model meta-llama/Llama-3.1-8B-Instruct \
262 --hardware xeon6 \
263 --detect-hardware \
264 --input-tokens 128 \
265 --output-tokens 128 \
266 --concurrency 32 \
267 --ttft-sla-ms 3000 \
268 --tpot-sla-ms 100 \
269 --config-out /output/config.yml \
270 --env-out /output/env.sh
271 ```
272
273 Deploy the generated initial configuration:
274
275 ```bash
276 source /output/env.sh
277 vllm serve --config /output/config.yml
278 ```
279
280 For optional benchmark-backed validation of the scheduler settings, continue
281 with [SWEEP_TUNING.md](SWEEP_TUNING.md).
282
283 ## Implementation
284
285 - `hardware_detection.py` collects effective CPU/NUMA/memory information.
286 - `runtime_tuning.py` owns hardware-policy selection and independent parameter
287 tuning functions.
288 - `recipe_json_to_vllm_config.py` resolves the recipe, collects optional inputs,
289 applies the selected policy, and generates `config.yml` and `env.sh`.
290
291 For benchmark-backed validation of the initial scheduler suggestion, see
292 [SWEEP_TUNING.md](SWEEP_TUNING.md).