@tduncan / Nvidia-Optimization-Stack-VLLM / commits / 5b184f775a

connects vLLM Recipes with vLLM's native config-based deployment and benchmark (#51308)

Signed-off-by: louie-tsai <louie.tsai@intel.com>

Louie Tsai committed Aug 11, 2026 at 06:53 UTC 5b184f775ae0f71d20cee222314b8866fb2b9d3c
6 files changed +773 -43
docs/benchmarking/README.md
+2 -1
@@ -1,7 +1,8 @@
1 # Benchmark Suites
2
3 -vLLM provides comprehensive benchmarking tools for performance testing and evaluation:
3 +vLLM provides comprehensive benchmarking tools for performance testing and evaluation.
4
5 - **[Benchmark CLI](./cli.md)**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing.
6 - **[Parameter Sweeps](./sweeps.md)**: Automate `vllm bench` runs for multiple configurations, useful for [optimization and tuning](../configuration/optimization.md).
7 - **[Performance Dashboard](./dashboard.md)**: Automated CI that publishes benchmarks on each commit.
8 +- **[Benchmark using vLLM Recipes](../../tools/recipes/README.md)**: Use the Recipe conversion tool to generate `config.yaml` and `env.sh` from [vLLM Recipes](https://recipes.vllm.ai/) for Benchmarking.
docs/configuration/serve_args.md
+13
@@ -29,6 +29,19 @@ To use the above config file:
29 vllm serve --config config.yaml
30 ```
31
32 +### Generate a configuration from vLLM Recipes
33 +
34 +[vLLM Recipes](https://recipes.vllm.ai/) can be converted into `config.yaml`
35 +and `env.sh`. See the
36 +[Recipes conversion tool README](../../tools/recipes/README.md) for usage.
37 +
38 +Source the generated environment before starting vLLM:
39 +
40 +```bash
41 +source env.sh
42 +vllm serve --config config.yaml
43 +```
44 +
45 !!! note
46 In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
47 The order of priorities is `command line > config file values > defaults`.
docs/deployment/docker.md
+21
@@ -8,6 +8,27 @@ toc_depth: 2
8
9 --8<-- "docs/getting_started/installation/gpu.md:pre-built-images"
10
11 +## Run a vLLM Recipes configuration
12 +
13 +[vLLM Recipes](https://recipes.vllm.ai/) can be converted into `config.yaml`
14 +and `env.sh`. See the
15 +[Recipes conversion tool README](../../tools/recipes/README.md) for usage.
16 +
17 +For Docker, mount both files and source `env.sh` inside the container before
18 +starting vLLM:
19 +
20 +```bash
21 +docker run --rm --gpus all \
22 + -v ~/.cache/huggingface:/root/.cache/huggingface \
23 + -v "$PWD/config.yaml:/recipe/config.yaml:ro" \
24 + -v "$PWD/env.sh:/recipe/env.sh:ro" \
25 + -p 8000:8000 \
26 + --ipc=host \
27 + --entrypoint /bin/bash \
28 + vllm/vllm-openai:latest \
29 + -lc 'source /recipe/env.sh && exec vllm serve --config /recipe/config.yaml'
30 +```
31 +
32 ## Persist the compile cache across containers
33
34 Mounting the Hugging Face cache keeps model weights across containers, but each
docs/models/hardware_supported_models/cpu.md
+62 -42
@@ -10,54 +10,74 @@
10 | [Intel® Xeon® 6 Processors](https://www.intel.com/content/www/us/en/products/details/processors/xeon.html) |
11 | [Intel® Xeon® 5 Processors](https://www.intel.com/content/www/us/en/products/docs/processors/xeon/5th-gen-xeon-scalable-processors.html) |
12
13 +## Deploy from a vLLM Recipe
14 +
15 +The **Recipe** column below links to validated Xeon 6 configurations when
16 +available. Use the Recipe conversion tool to generate `config.yaml` and
17 +`env.sh`; see the
18 +[Recipes conversion tool README](../../../tools/recipes/README.md) for usage.
19 +
20 +Load the generated environment before starting vLLM:
21 +
22 +```bash
23 +source env.sh
24 +vllm serve --config config.yaml
25 +```
26 +
27 ## Recommended Models
28
29 ### Text-only Language Models
30
17 -| Model | Architecture | Supported |
18 -| ------------------------------------ | ---------------------------------------- | --------- |
19 -| unsloth/gpt-oss-20b | GptOssForCausalLM | ✅ |
20 -| meta-llama/Llama-3.1-8B-Instruct | LlamaForCausalLM | ✅ |
21 -| meta-llama/Llama-3.2-1B | LlamaForCausalLM | ✅ |
22 -| meta-llama/Llama-3.2-3B-Instruct | LlamaForCausalLM | ✅ |
23 -| meta-llama/Llama-3.3-70B-Instruct | LlamaForCausalLM | ✅ |
24 -| RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 | LlamaForCausalLM | ✅ |
25 -| RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ |
26 -| RedHatAI/Llama-3.2-1B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ |
27 -| RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ |
28 -| RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8 | LlamaForCausalLM | ✅ |
29 -| hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 | LlamaForCausalLM | ✅ |
30 -| AMead10/Llama-3.2-1B-Instruct-AWQ | LlamaForCausalLM | ✅ |
31 -| AMead10/Llama-3.2-3B-Instruct-AWQ | LlamaForCausalLM | ✅ |
32 -| TheBloke/TinyLlama-1.1B-Chat-v1.0-AWQ | LlamaForCausalLM | ✅ |
33 -| TheBloke/TinyLlama-1.1B-Chat-v1.0-GPTQ | LlamaForCausalLM | ✅ |
34 -| ibm-granite/granite-3.2-2b-instruct | GraniteForCausalLM | ✅ |
35 -| Qwen/Qwen3-1.7B | Qwen3ForCausalLM | ✅ |
36 -| Qwen/Qwen3-4B | Qwen3ForCausalLM | ✅ |
37 -| Qwen/Qwen3-8B | Qwen3ForCausalLM | ✅ |
38 -| Qwen/Qwen3-14B | Qwen3ForCausalLM | ✅ |
39 -| Qwen/Qwen3-14B-AWQ | Qwen3ForCausalLM | ✅ |
40 -| Qwen/Qwen3-30B-A3B | Qwen3MoeForCausalLM | ✅ |
41 -| Qwen/QwQ-32B-AWQ | Qwen2ForCausalLM | ✅ |
42 -| Qwen/Qwen1.5-0.5B-Chat-GPTQ-Int4 | Qwen2ForCausalLM | ✅ |
43 -| RedHatAI/QwQ-32B-quantized.w8a8 | Qwen2ForCausalLM | ✅ |
44 -| zai-org/glm-4-9b-hf | GLMForCausalLM | ✅ |
45 -| google/gemma-7b | GemmaForCausalLM | ✅ |
46 -| microsoft/Phi-4-reasoning | Phi3ForCausalLM | ✅ |
47 -| TheBloke/Mistral-7B-Instruct-v0.2-AWQ | MistralForCausalLM | ✅ |
31 +| Model | Architecture | Supported | Recipe |
32 +| ------------------------------------ | ---------------------------------------- | --------- | ------ |
33 +| unsloth/gpt-oss-20b | GptOssForCausalLM | ✅ | — |
34 +| openai/gpt-oss-20b | GptOssForCausalLM | ✅ | — |
35 +| meta-llama/Llama-3.1-8B-Instruct | LlamaForCausalLM | ✅ | [Xeon 6](https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct?hardware=xeon6) |
36 +| meta-llama/Llama-3.2-1B | LlamaForCausalLM | ✅ | — |
37 +| meta-llama/Llama-3.2-3B-Instruct | LlamaForCausalLM | ✅ | — |
38 +| meta-llama/Llama-3.3-70B-Instruct | LlamaForCausalLM | ✅ | [Xeon 6](https://recipes.vllm.ai/meta-llama/Llama-3.3-70B-Instruct?hardware=xeon6) |
39 +| RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
40 +| RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
41 +| RedHatAI/Llama-3.2-1B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
42 +| RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
43 +| RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
44 +| hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 | LlamaForCausalLM | ✅ | — |
45 +| AMead10/Llama-3.2-1B-Instruct-AWQ | LlamaForCausalLM | ✅ | — |
46 +| AMead10/Llama-3.2-3B-Instruct-AWQ | LlamaForCausalLM | ✅ | — |
47 +| TheBloke/TinyLlama-1.1B-Chat-v1.0-AWQ | LlamaForCausalLM | ✅ | — |
48 +| TheBloke/TinyLlama-1.1B-Chat-v1.0-GPTQ | LlamaForCausalLM | ✅ | — |
49 +| ibm-granite/granite-3.2-2b-instruct | GraniteForCausalLM | ✅ | — |
50 +| Qwen/Qwen3-1.7B | Qwen3ForCausalLM | ✅ | — |
51 +| Qwen/Qwen3-4B | Qwen3ForCausalLM | ✅ | [Xeon 6](https://recipes.vllm.ai/Qwen/Qwen3-4B?hardware=xeon6) |
52 +| Qwen/Qwen3-8B | Qwen3ForCausalLM | ✅ | — |
53 +| Qwen/Qwen3-14B | Qwen3ForCausalLM | ✅ | — |
54 +| Qwen/Qwen3-14B-FP8 | Qwen3ForCausalLM | ✅ | — |
55 +| Qwen/Qwen3-14B-AWQ | Qwen3ForCausalLM | ✅ | — |
56 +| Qwen/Qwen3-30B-A3B | Qwen3MoeForCausalLM | ✅ | — |
57 +| Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 | Qwen3MoeForCausalLM | ✅ | — |
58 +| Qwen/QwQ-32B | Qwen2ForCausalLM | ✅ | — |
59 +| Qwen/QwQ-32B-AWQ | Qwen2ForCausalLM | ✅ | — |
60 +| Qwen/Qwen1.5-0.5B-Chat-GPTQ-Int4 | Qwen2ForCausalLM | ✅ | — |
61 +| RedHatAI/QwQ-32B-quantized.w8a8 | Qwen2ForCausalLM | ✅ | — |
62 +| zai-org/glm-4-9b-hf | GLMForCausalLM | ✅ | — |
63 +| google/gemma-7b | GemmaForCausalLM | ✅ | — |
64 +| microsoft/Phi-4-reasoning | Phi3ForCausalLM | ✅ | — |
65 +| mistralai/Mistral-7B-Instruct-v0.2 | MistralForCausalLM | ✅ | — |
66 +| TheBloke/Mistral-7B-Instruct-v0.2-AWQ | MistralForCausalLM | ✅ | — |
67
68 ### Multimodal Language Models
69
51 -| Model | Architecture | Supported |
52 -| ------------------------------------ | ---------------------------------------- | --------- |
53 -| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama4ForConditionalGeneration | ✅ |
54 -| google/gemma-3-4b-it | Gemma3ForConditionalGeneration | ✅ |
55 -| google/gemma-3-12b-it | Gemma3ForConditionalGeneration | ✅ |
56 -| google/gemma-4-E4B-it | Gemma4ForConditionalGeneration | ✅ |
57 -| google/gemma-4-E2B-it | Gemma4ForConditionalGeneration | ✅ |
58 -| google/gemma-4-26B-A4B-it | Gemma4ForConditionalGeneration | ✅ |
59 -| microsoft/Phi-4-multimodal-instruct | Phi4MMForCausalLM | ✅ |
60 -| Qwen/Qwen2.5-VL-7B-Instruct | Qwen2VLForConditionalGeneration | ✅ |
61 -| openai/whisper-large-v3 | WhisperForConditionalGeneration | ✅ |
70 +| Model | Architecture | Supported | Recipe |
71 +| ------------------------------------ | ---------------------------------------- | --------- | ------ |
72 +| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama4ForConditionalGeneration | ✅ | [Xeon 6](https://recipes.vllm.ai/meta-llama/Llama-4-Scout-17B-16E-Instruct?hardware=xeon6) |
73 +| google/gemma-3-4b-it | Gemma3ForConditionalGeneration | ✅ | — |
74 +| google/gemma-3-12b-it | Gemma3ForConditionalGeneration | ✅ | — |
75 +| google/gemma-4-E4B-it | Gemma4ForConditionalGeneration | ✅ | [Xeon 6](https://recipes.vllm.ai/Google/gemma-4-E4B-it?hardware=xeon6) |
76 +| google/gemma-4-E2B-it | Gemma4ForConditionalGeneration | ✅ | [Xeon 6](https://recipes.vllm.ai/Google/gemma-4-E2B-it?hardware=xeon6) |
77 +| google/gemma-4-26B-A4B-it | Gemma4ForConditionalGeneration | ✅ | [Xeon 6](https://recipes.vllm.ai/Google/gemma-4-26B-A4B-it?hardware=xeon6) |
78 +| microsoft/Phi-4-multimodal-instruct | Phi4MMForCausalLM | ✅ | — |
79 +| Qwen/Qwen2.5-VL-7B-Instruct | Qwen2VLForConditionalGeneration | ✅ | — |
80 +| Qwen/Qwen3-VL-30B-A3B-Instruct | Qwen3VLMoeForConditionalGeneration | ✅ | — |
81 +| openai/whisper-large-v3 | WhisperForConditionalGeneration | ✅ | — |
82
83 ✅ Runs and optimized.
tools/recipes/README.md new
+116
@@ -0,0 +1,116 @@
1 +# vLLM Recipes Tools
2 +
3 +Utilities for consuming deployment configurations from [vLLM Recipes](https://recipes.vllm.ai/) and converting them into files that can be used directly with vLLM.
4 +
5 +## `recipe_json_to_vllm_config.py`
6 +
7 +Converts a hardware-specific vLLM Recipes JSON rendering into:
8 +
9 +- `config.yml` — native configuration for `vllm serve --config`
10 +- `env.sh` — environment variables required by the recipe
11 +
12 +## Install Dependency
13 +
14 +```bash
15 +pip install pyyaml
16 +```
17 +
18 +## Interactive Recipe Discovery
19 +
20 +If you do not know the recipe JSON URL, run the script without an input:
21 +
22 +```bash
23 +python3 tools/recipes/recipe_json_to_vllm_config.py
24 +```
25 +
26 +The script will:
27 +
28 +1. Search the Recipes model index.
29 +2. Ask which model to use.
30 +3. Show the hardware configurations available for that model.
31 +4. Resolve the correct hardware-specific JSON endpoint.
32 +5. Generate `config.yml` and `env.sh`.
33 +
34 +Example interaction:
35 +
36 +```text
37 +No recipe JSON supplied; starting Recipes API discovery.
38 +Model search (for example: llama 3.1): llama 3.1
39 +
40 +Matching models:
41 + [1] meta-llama/Llama-3.1-8B-Instruct — Llama-3.1-8B-Instruct [Meta]
42 +Select model: 1
43 +
44 +Available hardware:
45 + [1] h100
46 + [2] h200
47 + [3] trillium
48 + [4] xeon6
49 +Select hardware: 4
50 +
51 +Resolved recipe:
52 + Model: meta-llama/Llama-3.1-8B-Instruct
53 + Hardware: xeon6
54 + JSON: https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
55 +```
56 +
57 +## Non-Interactive Recipe Discovery
58 +
59 +For automation or CI, specify the model and hardware directly:
60 +
61 +```bash
62 +python3 tools/recipes/recipe_json_to_vllm_config.py \
63 + --model meta-llama/Llama-3.1-8B-Instruct \
64 + --hardware xeon6
65 +```
66 +
67 +The script verifies that the requested hardware rendering exists before using it.
68 +
69 +## Direct Recipe JSON Input
70 +
71 +If you already know the recipe JSON URL:
72 +
73 +```bash
74 +python3 tools/recipes/recipe_json_to_vllm_config.py \
75 + https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
76 +```
77 +
78 +A local JSON file is also supported:
79 +
80 +```bash
81 +python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json
82 +```
83 +
84 +## Start vLLM
85 +
86 +After generating the files:
87 +
88 +```bash
89 +source env.sh
90 +vllm serve --config config.yml
91 +```
92 +
93 +Example generated `config.yml`:
94 +
95 +```yaml
96 +model: meta-llama/Llama-3.1-8B-Instruct
97 +tensor-parallel-size: 1
98 +```
99 +
100 +If the selected recipe does not require environment variables, `env.sh` will contain no additional exports.
101 +
102 +## Custom Output Files
103 +
104 +```bash
105 +python3 tools/recipes/recipe_json_to_vllm_config.py \
106 + --model meta-llama/Llama-3.1-8B-Instruct \
107 + --hardware xeon6 \
108 + --config-out llama31-xeon6.yml \
109 + --env-out llama31-xeon6-env.sh
110 +```
111 +
112 +## Scope
113 +
114 +The converter currently targets recipes that resolve to a single `vllm serve` process.
115 +
116 +Multi-node, disaggregated prefill/decode, or other multi-process deployment recipes require multiple commands and cannot be represented by a single `config.yml`.
tools/recipes/recipe_json_to_vllm_config.py new
+559
@@ -0,0 +1,559 @@
1 +#!/usr/bin/env python3
2 +# SPDX-License-Identifier: Apache-2.0
3 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
4 +
5 +"""
6 +Convert a vLLM Recipes per-hardware JSON rendering into:
7 +
8 + 1) config.yml - native `vllm serve --config` YAML
9 + 2) env.sh - environment variables required by the recipe
10 +
11 +Examples:
12 + # Interactive discovery
13 + python3 recipe_json_to_vllm_config.py
14 +
15 + # Direct JSON URL
16 + python3 recipe_json_to_vllm_config.py \
17 + https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
18 +
19 + # Non-interactive discovery
20 + python3 recipe_json_to_vllm_config.py \
21 + --model meta-llama/Llama-3.1-8B-Instruct --hardware xeon6
22 +
23 +Then:
24 + source env.sh
25 + vllm serve --config config.yml
26 +
27 +The converter intentionally targets a single `vllm serve` process. If the
28 +recipe rendering is multi-node, PD-disaggregated, or another multi-process
29 +deployment, it exits instead of silently generating an incomplete config.
30 +"""
31 +
32 +from __future__ import annotations
33 +
34 +import argparse
35 +import difflib
36 +import json
37 +import shlex
38 +import sys
39 +import urllib.parse
40 +import urllib.request
41 +from pathlib import Path
42 +from typing import Any
43 +
44 +try:
45 + import yaml
46 +except ImportError as exc:
47 + raise SystemExit("PyYAML is required. Install it with: pip install pyyaml") from exc
48 +
49 +
50 +DEFAULT_API_BASE = "https://recipes.vllm.ai"
51 +
52 +SHORT_ALIASES = {
53 + "-tp": "tensor-parallel-size",
54 + "-pp": "pipeline-parallel-size",
55 + "-dp": "data-parallel-size",
56 +}
57 +
58 +
59 +def parse_args() -> argparse.Namespace:
60 + p = argparse.ArgumentParser(
61 + description="Convert vLLM Recipes JSON to native vllm serve config.yml + env.sh"
62 + )
63 + p.add_argument(
64 + "source",
65 + nargs="?",
66 + help=(
67 + "Per-hardware recipe JSON URL or local JSON file. "
68 + "If omitted, discover a recipe from the Recipes API."
69 + ),
70 + )
71 + p.add_argument(
72 + "--model",
73 + help=(
74 + "Model hf_id or search text for recipe discovery. "
75 + "Use with --hardware for non-interactive operation."
76 + ),
77 + )
78 + p.add_argument(
79 + "--hardware",
80 + help="Hardware ID for recipe discovery, for example xeon6 or h100.",
81 + )
82 + p.add_argument(
83 + "--api-base",
84 + default=DEFAULT_API_BASE,
85 + help=f"Recipes API base URL (default: {DEFAULT_API_BASE})",
86 + )
87 + p.add_argument(
88 + "--config-out",
89 + default="config.yml",
90 + help="Output vLLM YAML config (default: config.yml)",
91 + )
92 + p.add_argument(
93 + "--env-out",
94 + default="env.sh",
95 + help="Output shell environment file (default: env.sh)",
96 + )
97 + return p.parse_args()
98 +
99 +
100 +def api_url(base: str, path: str) -> str:
101 + return urllib.parse.urljoin(base.rstrip("/") + "/", path.lstrip("/"))
102 +
103 +
104 +def load_json(source: str) -> dict[str, Any] | list[Any]:
105 + if source.startswith(("http://", "https://")):
106 + req = urllib.request.Request(
107 + source,
108 + headers={"User-Agent": "vllm-recipe-config-converter/1.0"},
109 + )
110 + with urllib.request.urlopen(req, timeout=30) as response:
111 + return json.load(response)
112 +
113 + with open(source, encoding="utf-8") as f:
114 + return json.load(f)
115 +
116 +
117 +def prompt(text: str) -> str:
118 + try:
119 + return input(text).strip()
120 + except EOFError as exc:
121 + raise ValueError(
122 + "Interactive input is unavailable. Pass --model and --hardware, "
123 + "or provide a recipe JSON URL/file."
124 + ) from exc
125 +
126 +
127 +def model_label(model: dict[str, Any]) -> str:
128 + label = str(model.get("hf_id", ""))
129 + title = model.get("title")
130 + provider = model.get("provider")
131 + if title and title != label:
132 + label += f" — {title}"
133 + if provider:
134 + label += f" [{provider}]"
135 + return label
136 +
137 +
138 +def search_models(
139 + models: list[dict[str, Any]], query: str, limit: int = 20
140 +) -> list[dict[str, Any]]:
141 + q = query.strip().lower()
142 + if not q:
143 + return []
144 +
145 + exact = [model for model in models if str(model.get("hf_id", "")).lower() == q]
146 + if exact:
147 + return exact
148 +
149 + tokens = [token for token in q.replace("/", " ").replace("-", " ").split() if token]
150 + scored: list[tuple[float, dict[str, Any]]] = []
151 +
152 + for model in models:
153 + hf_id = str(model.get("hf_id", "")).lower()
154 + title = str(model.get("title", "")).lower()
155 + provider = str(model.get("provider", "")).lower()
156 + text = f"{hf_id} {title} {provider}"
157 +
158 + if q in hf_id:
159 + score = 1000.0
160 + elif q in title:
161 + score = 900.0
162 + elif tokens and all(token in text for token in tokens):
163 + score = 800.0
164 + elif tokens and any(token in text for token in tokens):
165 + score = 500.0
166 + else:
167 + ratio = difflib.SequenceMatcher(None, q, hf_id).ratio()
168 + if ratio < 0.30:
169 + continue
170 + score = ratio * 100.0
171 +
172 + scored.append((score, model))
173 +
174 + scored.sort(key=lambda pair: (-pair[0], str(pair[1].get("hf_id", "")).lower()))
175 + return [model for _, model in scored[:limit]]
176 +
177 +
178 +def choose_from_menu(items: list[Any], label_fn, prompt_text: str) -> Any:
179 + if not items:
180 + raise ValueError("No selectable items found.")
181 +
182 + if len(items) == 1:
183 + print(f"Selected: {label_fn(items[0])}")
184 + return items[0]
185 +
186 + for index, item in enumerate(items, start=1):
187 + print(f" [{index}] {label_fn(item)}")
188 +
189 + while True:
190 + answer = prompt(prompt_text)
191 + try:
192 + index = int(answer)
193 + except ValueError:
194 + print(f"Enter a number from 1 to {len(items)}.")
195 + continue
196 + if 1 <= index <= len(items):
197 + return items[index - 1]
198 + print(f"Enter a number from 1 to {len(items)}.")
199 +
200 +
201 +def select_model(models: list[dict[str, Any]], requested: str | None) -> dict[str, Any]:
202 + query = requested
203 + while True:
204 + if not query:
205 + query = prompt("Model search (for example: llama 3.1): ")
206 +
207 + matches = search_models(models, query)
208 + if matches:
209 + print("\nMatching models:")
210 + return choose_from_menu(matches, model_label, "Select model: ")
211 +
212 + if requested:
213 + raise ValueError(f"No recipe model matched {requested!r}.")
214 +
215 + print(f"No recipe model matched {query!r}. Try again.")
216 + query = None
217 +
218 +
219 +def select_hardware(
220 + by_hardware: dict[str, str], requested: str | None
221 +) -> tuple[str, str]:
222 + hardware_ids = sorted(by_hardware)
223 +
224 + if requested:
225 + selected = next(
226 + (hw for hw in hardware_ids if hw.lower() == requested.lower()),
227 + None,
228 + )
229 + if selected is None:
230 + raise ValueError(
231 + f"Hardware {requested!r} is not available for this model. "
232 + f"Available: {', '.join(hardware_ids)}"
233 + )
234 + return selected, by_hardware[selected]
235 +
236 + print("\nAvailable hardware:")
237 + selected = choose_from_menu(hardware_ids, lambda value: value, "Select hardware: ")
238 + return selected, by_hardware[selected]
239 +
240 +
241 +def discover_recipe_source(
242 + api_base: str,
243 + requested_model: str | None,
244 + requested_hardware: str | None,
245 +) -> str:
246 + print("No recipe JSON supplied; starting Recipes API discovery.")
247 +
248 + models_url = api_url(api_base, "/models.json")
249 + models_data = load_json(models_url)
250 + if not isinstance(models_data, list):
251 + raise ValueError(f"{models_url} did not return a model list.")
252 +
253 + models = [model for model in models_data if isinstance(model, dict)]
254 + model = select_model(models, requested_model)
255 +
256 + model_json_path = model.get("json")
257 + if not isinstance(model_json_path, str) or not model_json_path:
258 + raise ValueError(f"Selected model {model.get('hf_id')!r} has no JSON API path.")
259 +
260 + model_json_url = api_url(api_base, model_json_path)
261 + model_data = load_json(model_json_url)
262 + if not isinstance(model_data, dict):
263 + raise ValueError(f"{model_json_url} did not return a JSON object.")
264 +
265 + recommended = model_data.get("recommended_command")
266 + if not isinstance(recommended, dict):
267 + raise ValueError(
268 + f"Model {model.get('hf_id')!r} has no rendered "
269 + "recommended_command in the Recipes API."
270 + )
271 +
272 + raw_by_hardware = recommended.get("by_hardware")
273 + if not isinstance(raw_by_hardware, dict) or not raw_by_hardware:
274 + raise ValueError(
275 + f"Model {model.get('hf_id')!r} has no per-hardware renderings "
276 + "in recommended_command.by_hardware."
277 + )
278 +
279 + by_hardware = {
280 + str(hw): path
281 + for hw, path in raw_by_hardware.items()
282 + if isinstance(path, str) and path
283 + }
284 + if not by_hardware:
285 + raise ValueError(
286 + f"Model {model.get('hf_id')!r} has no usable hardware JSON paths."
287 + )
288 +
289 + hardware, path = select_hardware(by_hardware, requested_hardware)
290 + resolved = api_url(api_base, path)
291 +
292 + print("\nResolved recipe:")
293 + print(f" Model: {model.get('hf_id')}")
294 + print(f" Hardware: {hardware}")
295 + print(f" JSON: {resolved}")
296 + print()
297 +
298 + return resolved
299 +
300 +
301 +def coerce(value: str) -> Any:
302 + """Convert CLI string values to useful YAML scalar/object types."""
303 + try:
304 + return json.loads(value)
305 + except (json.JSONDecodeError, TypeError):
306 + return value
307 +
308 +
309 +def is_option_token(token: str) -> bool:
310 + if token.startswith("--"):
311 + return True
312 + if token in SHORT_ALIASES or token == "-O":
313 + return True
314 + return token.startswith("-O") and len(token) > 2
315 +
316 +
317 +def merge_value(dst: dict[str, Any], path: list[str], value: Any) -> None:
318 + """Merge dotted CLI args into nested YAML dictionaries."""
319 + cur = dst
320 + for part in path[:-1]:
321 + existing = cur.get(part)
322 + if existing is None:
323 + cur[part] = {}
324 + elif not isinstance(existing, dict):
325 + raise ValueError(
326 + f"Cannot merge nested option {'.'.join(path)!r}: "
327 + f"{part!r} is already a scalar"
328 + )
329 + cur = cur[part]
330 +
331 + leaf = path[-1]
332 + if leaf not in cur:
333 + cur[leaf] = value
334 + return
335 +
336 + # Repeated CLI option. Preserve all values.
337 + old = cur[leaf]
338 + if not isinstance(old, list):
339 + old = [old]
340 + if isinstance(value, list):
341 + old.extend(value)
342 + else:
343 + old.append(value)
344 + cur[leaf] = old
345 +
346 +
347 +def normalize_key(raw_key: str) -> list[str]:
348 + """
349 + Convert the CLI key to config-file spelling.
350 +
351 + Only the top-level CLI option name gets underscore -> dash normalization.
352 + Nested JSON field names after a dot are preserved.
353 + """
354 + parts = raw_key.split(".")
355 + parts[0] = parts[0].replace("_", "-")
356 + return parts
357 +
358 +
359 +def argv_to_config(argv: list[Any]) -> dict[str, Any]:
360 + argv = [str(x) for x in argv]
361 +
362 + if len(argv) < 3 or argv[0:2] != ["vllm", "serve"]:
363 + raise ValueError(
364 + "Expected recipe argv to start with: ['vllm', 'serve', MODEL, ...]. "
365 + f"Got: {argv[:4]!r}"
366 + )
367 +
368 + model = argv[2]
369 + if model.startswith("-"):
370 + raise ValueError(f"Expected model after 'vllm serve', got {model!r}")
371 +
372 + config: dict[str, Any] = {"model": model}
373 +
374 + i = 3
375 + while i < len(argv):
376 + token = argv[i]
377 +
378 + # -O3 / -O=3
379 + if token.startswith("-O") and token != "-O":
380 + value = token[3:] if token.startswith("-O=") else token[2:]
381 + merge_value(config, ["optimization-level"], coerce(value))
382 + i += 1
383 + continue
384 +
385 + # -O 3
386 + if token == "-O":
387 + if i + 1 >= len(argv):
388 + raise ValueError("-O is missing its value")
389 + merge_value(config, ["optimization-level"], coerce(argv[i + 1]))
390 + i += 2
391 + continue
392 +
393 + # Selected common short aliases.
394 + if token in SHORT_ALIASES:
395 + if i + 1 >= len(argv):
396 + raise ValueError(f"{token} is missing its value")
397 + merge_value(
398 + config,
399 + [SHORT_ALIASES[token]],
400 + coerce(argv[i + 1]),
401 + )
402 + i += 2
403 + continue
404 +
405 + if not token.startswith("--"):
406 + raise ValueError(
407 + f"Unexpected positional/short argument {token!r}. "
408 + "The converter expects Recipes to emit long-form vLLM serve options."
409 + )
410 +
411 + # --key=value
412 + if "=" in token:
413 + key, raw_value = token[2:].split("=", 1)
414 + merge_value(config, normalize_key(key), coerce(raw_value))
415 + i += 1
416 + continue
417 +
418 + key = token[2:]
419 + i += 1
420 +
421 + # Gather values until the next option. This supports both scalar and
422 + # nargs-style vLLM options.
423 + raw_values: list[str] = []
424 + while i < len(argv) and not is_option_token(argv[i]):
425 + raw_values.append(argv[i])
426 + i += 1
427 +
428 + if not raw_values:
429 + # Store flags exactly by their long-form spelling.
430 + # Example:
431 + # --trust-remote-code -> trust-remote-code: true
432 + # --no-enable-prefix-caching -> no-enable-prefix-caching: true
433 + value: Any = True
434 + elif len(raw_values) == 1:
435 + value = coerce(raw_values[0])
436 + else:
437 + value = [coerce(v) for v in raw_values]
438 +
439 + merge_value(config, normalize_key(key), value)
440 +
441 + return config
442 +
443 +
444 +def recipe_argv(recipe: dict[str, Any]) -> list[Any]:
445 + deploy_type = recipe.get("deploy_type")
446 +
447 + # Current Recipes API's single-node rendering exposes `argv`.
448 + if isinstance(recipe.get("argv"), list):
449 + if deploy_type not in (None, "single_node"):
450 + raise ValueError(
451 + f"Recipe deploy_type={deploy_type!r} is not a single-node deployment. "
452 + "A single config.yml would not fully represent this deployment."
453 + )
454 + return recipe["argv"]
455 +
456 + # Give a useful failure for other known rendered shapes.
457 + multi_process_fields = [
458 + k
459 + for k in (
460 + "head_argv",
461 + "worker_argv",
462 + "worker_argvs",
463 + "prefill",
464 + "decode",
465 + "vllm_argv",
466 + )
467 + if k in recipe
468 + ]
469 + if multi_process_fields:
470 + raise ValueError(
471 + "This recipe is a multi-process deployment and cannot be represented "
472 + "by one config.yml. Found fields: " + ", ".join(multi_process_fields)
473 + )
474 +
475 + raise ValueError("Recipe JSON does not contain an `argv` field")
476 +
477 +
478 +def write_config(
479 + path: str,
480 + source: str,
481 + recipe: dict[str, Any],
482 + config: dict[str, Any],
483 +) -> None:
484 + metadata = [
485 + "# Generated from vLLM Recipes JSON.",
486 + f"# Source: {source}",
487 + ]
488 + for key in ("hardware", "strategy", "variant", "deploy_type"):
489 + if recipe.get(key) is not None:
490 + metadata.append(f"# {key}: {recipe[key]}")
491 +
492 + body = yaml.safe_dump(
493 + config,
494 + sort_keys=False,
495 + default_flow_style=False,
496 + allow_unicode=True,
497 + )
498 + Path(path).write_text("\n".join(metadata) + "\n" + body, encoding="utf-8")
499 +
500 +
501 +def write_env(path: str, source: str, recipe: dict[str, Any]) -> None:
502 + env = recipe.get("env") or {}
503 + if not isinstance(env, dict):
504 + raise ValueError(f"Recipe `env` must be an object, got {type(env).__name__}")
505 +
506 + lines = [
507 + "#!/usr/bin/env bash",
508 + "# Generated from vLLM Recipes JSON.",
509 + f"# Source: {source}",
510 + "",
511 + ]
512 +
513 + if env:
514 + for key, value in env.items():
515 + lines.append(f"export {key}={shlex.quote(str(value))}")
516 + else:
517 + lines.append("# No recipe-specific environment variables.")
518 +
519 + lines.append("")
520 + Path(path).write_text("\n".join(lines), encoding="utf-8")
521 + Path(path).chmod(Path(path).stat().st_mode | 0o111)
522 +
523 +
524 +def main() -> int:
525 + args = parse_args()
526 +
527 + try:
528 + source = args.source
529 + if source is None:
530 + source = discover_recipe_source(args.api_base, args.model, args.hardware)
531 + elif args.model or args.hardware:
532 + raise ValueError(
533 + "Do not combine a positional recipe JSON source with "
534 + "--model/--hardware discovery options."
535 + )
536 +
537 + recipe = load_json(source)
538 + if not isinstance(recipe, dict):
539 + raise ValueError("Recipe JSON must be a JSON object.")
540 +
541 + argv = recipe_argv(recipe)
542 + config = argv_to_config(argv)
543 + write_config(args.config_out, source, recipe, config)
544 + write_env(args.env_out, source, recipe)
545 + except Exception as exc:
546 + print(f"ERROR: {exc}", file=sys.stderr)
547 + return 1
548 +
549 + print(f"Wrote {args.config_out}")
550 + print(f"Wrote {args.env_out}")
551 + print()
552 + print("Run:")
553 + print(f" source {shlex.quote(args.env_out)}")
554 + print(f" vllm serve --config {shlex.quote(args.config_out)}")
555 + return 0
556 +
557 +
558 +if __name__ == "__main__":
559 + raise SystemExit(main())