@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 137 lines 3.56 KB
Rendered Raw
1 # Sweep Tuning
2
3 Sweep tuning is optional. The converter always creates one initial `config.yml`
4 first. Users can deploy it immediately or benchmark nearby scheduler settings
5 and generate one measured recommendation.
6
7 ## Generate
8
9 ```bash
10 python3 tools/recipes/recipe_json_to_vllm_config.py \
11 --model meta-llama/Llama-3.1-8B-Instruct \
12 --hardware xeon6 \
13 --detect-hardware \
14 --input-tokens 128 \
15 --output-tokens 128 \
16 --concurrency 32 \
17 --ttft-sla-ms 3000 \
18 --tpot-sla-ms 100 \
19 --generate-sweep
20 ```
21
22 The optional package is:
23
24 ```text
25 sweep/
26 ├── serve_params.json
27 ├── bench_params.json
28 ├── run_sweep.sh
29 ├── recommend.py
30 └── SWEEP.md
31 ```
32
33 The sweep varies only `max-num-seqs` and `max-num-batched-tokens`. It uses an
34 eight-point directed design: a batch-budget curve at the initial sequence count
35 plus lower/higher batch interactions at three-quarters and one-half of that
36 count. This gives broader coverage than a one-parameter-at-a-time sweep without
37 the cost of a full Cartesian grid.
38
39 ## Run and Recommend
40
41 ```bash
42 sweep/run_sweep.sh --dry-run
43 sweep/run_sweep.sh
44 sweep/recommend.py
45 ```
46
47 Resume an interrupted sweep with:
48
49 ```bash
50 sweep/run_sweep.sh --resume
51 ```
52
53 `recommend.py` writes:
54
55 ```text
56 sweep/recommended-config.yml
57 sweep/recommendation.json
58 ```
59
60 With TTFT/TPOT objectives, the benchmark uses vLLM `--goodput`. The recommender
61 calculates duration-weighted combined compliance across repeated runs and
62 requires both median P99 latency objectives and the minimum compliance ratio
63 (default `0.99`). Among eligible configurations it selects the highest mean
64 output-token throughput. Use `recommend.py --minimum-compliance VALUE` to
65 change the compliance requirement.
66
67 If no configuration is eligible, `recommend.py` records the highest-goodput
68 configuration as `best_effort` in `recommendation.json`, does not write a
69 deployable `recommended-config.yml`, and exits with status 2. Without latency
70 objectives it selects highest mean output-token throughput. Failed benchmark
71 configurations are excluded in either mode.
72
73 When an eligible configuration exists, `recommended-config.yml` changes only
74 the two swept scheduler parameters. `recommendation.json` records mean, median,
75 and worst-run P99 values, combined compliance, and the measured evidence for
76 every candidate.
77
78 Deploy:
79
80 ```bash
81 source env.sh
82 vllm serve --config sweep/recommended-config.yml
83 ```
84
85 ## vLLM CPU Docker Shell
86
87 Use the common CPU container setup in
88 [RUNTIME_TUNING.md](RUNTIME_TUNING.md#vllm-cpu-docker-shell). It ensures
89 hardware detection runs against the same effective CPU, NUMA, memory, and
90 cgroup limits used by the deployment.
91
92 Inside that container, generate the initial suggestion plus sweep package:
93
94 ```bash
95 python3 /recipes/recipe_json_to_vllm_config.py \
96 --model meta-llama/Llama-3.1-8B-Instruct \
97 --hardware xeon6 \
98 --detect-hardware \
99 --input-tokens 128 \
100 --output-tokens 128 \
101 --concurrency 32 \
102 --ttft-sla-ms 3000 \
103 --tpot-sla-ms 100 \
104 --config-out /output/config.yml \
105 --env-out /output/env.sh \
106 --generate-sweep \
107 --sweep-out-dir /output/sweep
108 ```
109
110 The initial configuration can be deployed directly:
111
112 ```bash
113 source /output/env.sh
114 vllm serve --config /output/config.yml
115 ```
116
117 Stop that manually started server before running the sweep, because
118 `run_sweep.sh` starts and stops its own vLLM servers.
119
120 ```bash
121 /output/sweep/run_sweep.sh --dry-run
122 /output/sweep/run_sweep.sh
123 /output/sweep/recommend.py
124 ```
125
126 Inspect:
127
128 ```bash
129 cat /output/sweep/recommendation.json
130 ```
131
132 Deploy the measured result:
133
134 ```bash
135 source /output/env.sh
136 vllm serve --config /output/sweep/recommended-config.yml
137 ```