| 1 | # Structured Outputs |
| 2 | |
| 3 | vLLM supports the generation of structured outputs using |
| 4 | [xgrammar](https://github.com/mlc-ai/xgrammar) or |
| 5 | [guidance](https://github.com/guidance-ai/llguidance) as backends. |
| 6 | This document shows you some examples of the different options that are |
| 7 | available to generate structured outputs. |
| 8 | |
| 9 | !!! warning |
| 10 | If you are still using the following deprecated API fields which were removed in v0.12.0, please update your code to use `structured_outputs` as demonstrated in the rest of this document: |
| 11 | |
| 12 | - `guided_json` -> `{"structured_outputs": {"json": ...}}` or `StructuredOutputsParams(json=...)` |
| 13 | - `guided_regex` -> `{"structured_outputs": {"regex": ...}}` or `StructuredOutputsParams(regex=...)` |
| 14 | - `guided_choice` -> `{"structured_outputs": {"choice": ...}}` or `StructuredOutputsParams(choice=...)` |
| 15 | - `guided_grammar` -> `{"structured_outputs": {"grammar": ...}}` or `StructuredOutputsParams(grammar=...)` |
| 16 | - `guided_whitespace_pattern` -> `{"structured_outputs": {"whitespace_pattern": ...}}` or `StructuredOutputsParams(whitespace_pattern=...)` |
| 17 | - `structural_tag` -> `{"structured_outputs": {"structural_tag": ...}}` or `StructuredOutputsParams(structural_tag=...)` |
| 18 | - `guided_decoding_backend` -> Remove this field from your request |
| 19 | |
| 20 | ## Online Serving (OpenAI API) |
| 21 | |
| 22 | You can generate structured outputs using the OpenAI's [Completions](https://platform.openai.com/docs/api-reference/completions) and [Chat](https://platform.openai.com/docs/api-reference/chat) API. |
| 23 | |
| 24 | The following parameters are supported, which must be added as extra parameters: |
| 25 | |
| 26 | - `choice`: the output will be exactly one of the choices. |
| 27 | - `regex`: the output will follow the regex pattern. |
| 28 | - `json`: the output will follow the JSON schema. |
| 29 | - `grammar`: the output will follow the context free grammar. |
| 30 | - `structural_tag`: Follow a JSON schema within a set of specified tags within the generated text. |
| 31 | |
| 32 | You can see the complete list of supported parameters on the [OpenAI-Compatible Server](../serving/online_serving/openai_compatible_server.md) page. |
| 33 | |
| 34 | Structured outputs are supported by default in the OpenAI-Compatible Server. You |
| 35 | may choose to specify the backend to use by setting the |
| 36 | `--structured-outputs-config.backend` flag to `vllm serve`. The default backend is `auto`, |
| 37 | which will try to choose an appropriate backend based on the details of the |
| 38 | request. You may also choose a specific backend, along with |
| 39 | some options. A full set of options is available in the `vllm serve --help` |
| 40 | text. |
| 41 | |
| 42 | Now let's see an example for each of the cases, starting with the `choice`, as it's the easiest one: |
| 43 | |
| 44 | ??? code |
| 45 | |
| 46 | ```python |
| 47 | from openai import OpenAI |
| 48 | client = OpenAI( |
| 49 | base_url="http://localhost:8000/v1", |
| 50 | api_key="-", |
| 51 | ) |
| 52 | model = client.models.list().data[0].id |
| 53 | |
| 54 | completion = client.chat.completions.create( |
| 55 | model=model, |
| 56 | messages=[ |
| 57 | {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"} |
| 58 | ], |
| 59 | extra_body={"structured_outputs": {"choice": ["positive", "negative"]}}, |
| 60 | ) |
| 61 | print(completion.choices[0].message.content) |
| 62 | ``` |
| 63 | |
| 64 | The next example shows how to use the `regex`. The supported regex syntax depends on the structured output backend. For example, `xgrammar`, `guidance`, and `outlines` use Rust-style regex, while `lm-format-enforcer` uses Python's `re` module. The idea is to generate an email address, given a simple regex template: |
| 65 | |
| 66 | ??? code |
| 67 | |
| 68 | ```python |
| 69 | completion = client.chat.completions.create( |
| 70 | model=model, |
| 71 | messages=[ |
| 72 | { |
| 73 | "role": "user", |
| 74 | "content": "Generate an example email address for Alan Turing, who works in Enigma. End in .com and new line. Example result: alan.turing@enigma.com\n", |
| 75 | } |
| 76 | ], |
| 77 | extra_body={"structured_outputs": {"regex": r"\w+@\w+\.com\n"}, "stop": ["\n"]}, |
| 78 | ) |
| 79 | print(completion.choices[0].message.content) |
| 80 | ``` |
| 81 | |
| 82 | One of the most relevant features in structured text generation is the option to generate a valid JSON with pre-defined fields and formats. |
| 83 | For this we can use the `json` parameter in two different ways: |
| 84 | |
| 85 | - Using directly a [JSON Schema](https://json-schema.org/) |
| 86 | - Defining a [Pydantic model](https://docs.pydantic.dev/latest/) and then extracting the JSON Schema from it (which is normally an easier option). |
| 87 | |
| 88 | The next example shows how to use the `response_format` parameter with a Pydantic model: |
| 89 | |
| 90 | ??? code |
| 91 | |
| 92 | ```python |
| 93 | from pydantic import BaseModel |
| 94 | from enum import Enum |
| 95 | |
| 96 | class CarType(str, Enum): |
| 97 | sedan = "sedan" |
| 98 | suv = "SUV" |
| 99 | truck = "Truck" |
| 100 | coupe = "Coupe" |
| 101 | |
| 102 | class CarDescription(BaseModel): |
| 103 | brand: str |
| 104 | model: str |
| 105 | car_type: CarType |
| 106 | |
| 107 | json_schema = CarDescription.model_json_schema() |
| 108 | |
| 109 | completion = client.chat.completions.create( |
| 110 | model=model, |
| 111 | messages=[ |
| 112 | { |
| 113 | "role": "user", |
| 114 | "content": "Generate a JSON with the brand, model and car_type of the most iconic car from the 90's", |
| 115 | } |
| 116 | ], |
| 117 | response_format={ |
| 118 | "type": "json_schema", |
| 119 | "json_schema": { |
| 120 | "name": "car-description", |
| 121 | "schema": CarDescription.model_json_schema() |
| 122 | }, |
| 123 | }, |
| 124 | ) |
| 125 | print(completion.choices[0].message.content) |
| 126 | ``` |
| 127 | |
| 128 | !!! tip |
| 129 | While not strictly necessary, normally it's better to indicate in the prompt the |
| 130 | JSON schema and how the fields should be populated. This can improve the |
| 131 | results notably in most cases. |
| 132 | |
| 133 | Finally we have the `grammar` option, which is probably the most |
| 134 | difficult to use, but it's really powerful. It allows us to define complete |
| 135 | languages like SQL queries. It works by using a context free EBNF grammar. |
| 136 | As an example, we can use to define a specific format of simplified SQL queries: |
| 137 | |
| 138 | ??? code |
| 139 | |
| 140 | ```python |
| 141 | simplified_sql_grammar = """ |
| 142 | root ::= select_statement |
| 143 | |
| 144 | select_statement ::= "SELECT " column " from " table " where " condition |
| 145 | |
| 146 | column ::= "col_1 " | "col_2 " |
| 147 | |
| 148 | table ::= "table_1 " | "table_2 " |
| 149 | |
| 150 | condition ::= column "= " number |
| 151 | |
| 152 | number ::= "1 " | "2 " |
| 153 | """ |
| 154 | |
| 155 | completion = client.chat.completions.create( |
| 156 | model=model, |
| 157 | messages=[ |
| 158 | { |
| 159 | "role": "user", |
| 160 | "content": "Generate an SQL query to show the 'username' and 'email' from the 'users' table.", |
| 161 | } |
| 162 | ], |
| 163 | extra_body={"structured_outputs": {"grammar": simplified_sql_grammar}}, |
| 164 | ) |
| 165 | print(completion.choices[0].message.content) |
| 166 | ``` |
| 167 | |
| 168 | See also: [full example](../../examples/features/structured_outputs/README.md) |
| 169 | |
| 170 | ## Reasoning Outputs |
| 171 | |
| 172 | You can also use structured outputs with <project:#reasoning-outputs> for reasoning models. |
| 173 | |
| 174 | ```bash |
| 175 | vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --reasoning-parser deepseek_r1 |
| 176 | ``` |
| 177 | |
| 178 | Note that you can use reasoning with any provided structured outputs feature. The following uses one with JSON schema: |
| 179 | |
| 180 | ??? code |
| 181 | |
| 182 | ```python |
| 183 | from pydantic import BaseModel |
| 184 | |
| 185 | |
| 186 | class People(BaseModel): |
| 187 | name: str |
| 188 | age: int |
| 189 | |
| 190 | |
| 191 | completion = client.chat.completions.create( |
| 192 | model=model, |
| 193 | messages=[ |
| 194 | { |
| 195 | "role": "user", |
| 196 | "content": "Generate a JSON with the name and age of one random person.", |
| 197 | } |
| 198 | ], |
| 199 | response_format={ |
| 200 | "type": "json_schema", |
| 201 | "json_schema": { |
| 202 | "name": "people", |
| 203 | "schema": People.model_json_schema() |
| 204 | } |
| 205 | }, |
| 206 | ) |
| 207 | print("reasoning: ", completion.choices[0].message.reasoning) |
| 208 | print("content: ", completion.choices[0].message.content) |
| 209 | ``` |
| 210 | |
| 211 | See also: [full example](../../examples/features/structured_outputs/README.md) |
| 212 | |
| 213 | !!! note |
| 214 | When using Qwen3 Coder models with reasoning enabled, structured outputs might become disabled if the reasoning content does not get parsed into the `reasoning` field separately (v0.11.2+). |
| 215 | To use both features together, you must explicitly enable structured outputs in reasoning mode. |
| 216 | To do so, add the following flag when starting the vLLM server: `--structured-outputs-config.enable_in_reasoning=True`. |
| 217 | See also: [Reasoning Outputs](reasoning_outputs.md) documentation. |
| 218 | |
| 219 | ## Experimental Automatic Parsing (OpenAI API) |
| 220 | |
| 221 | This section covers the OpenAI beta wrapper over the `client.chat.completions.create()` method that provides richer integrations with Python specific types. |
| 222 | |
| 223 | At the time of writing (`openai==1.54.4`), this is a "beta" feature in the OpenAI client library. Code reference can be found [here](https://github.com/openai/openai-python/blob/52357cff50bee57ef442e94d78a0de38b4173fc2/src/openai/resources/beta/chat/completions.py#L100-L104). |
| 224 | |
| 225 | For the following examples, vLLM was set up using `vllm serve meta-llama/Llama-3.1-8B-Instruct` |
| 226 | |
| 227 | Here is a simple example demonstrating how to get structured output using Pydantic models: |
| 228 | |
| 229 | ??? code |
| 230 | |
| 231 | ```python |
| 232 | from pydantic import BaseModel |
| 233 | from openai import OpenAI |
| 234 | |
| 235 | class Info(BaseModel): |
| 236 | name: str |
| 237 | age: int |
| 238 | |
| 239 | client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy") |
| 240 | model = client.models.list().data[0].id |
| 241 | completion = client.beta.chat.completions.parse( |
| 242 | model=model, |
| 243 | messages=[ |
| 244 | {"role": "system", "content": "You are a helpful assistant."}, |
| 245 | {"role": "user", "content": "My name is Cameron, I'm 28. What's my name and age?"}, |
| 246 | ], |
| 247 | response_format=Info, |
| 248 | ) |
| 249 | |
| 250 | message = completion.choices[0].message |
| 251 | print(message) |
| 252 | assert message.parsed |
| 253 | print("Name:", message.parsed.name) |
| 254 | print("Age:", message.parsed.age) |
| 255 | ``` |
| 256 | |
| 257 | ```console |
| 258 | ParsedChatCompletionMessage[Testing](content='{"name": "Cameron", "age": 28}', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], parsed=Testing(name='Cameron', age=28)) |
| 259 | Name: Cameron |
| 260 | Age: 28 |
| 261 | ``` |
| 262 | |
| 263 | Here is a more complex example using nested Pydantic models to handle a step-by-step math solution: |
| 264 | |
| 265 | ??? code |
| 266 | |
| 267 | ```python |
| 268 | from typing import List |
| 269 | from pydantic import BaseModel |
| 270 | from openai import OpenAI |
| 271 | |
| 272 | class Step(BaseModel): |
| 273 | explanation: str |
| 274 | output: str |
| 275 | |
| 276 | class MathResponse(BaseModel): |
| 277 | steps: list[Step] |
| 278 | final_answer: str |
| 279 | |
| 280 | completion = client.beta.chat.completions.parse( |
| 281 | model=model, |
| 282 | messages=[ |
| 283 | {"role": "system", "content": "You are a helpful expert math tutor."}, |
| 284 | {"role": "user", "content": "Solve 8x + 31 = 2."}, |
| 285 | ], |
| 286 | response_format=MathResponse, |
| 287 | ) |
| 288 | |
| 289 | message = completion.choices[0].message |
| 290 | print(message) |
| 291 | assert message.parsed |
| 292 | for i, step in enumerate(message.parsed.steps): |
| 293 | print(f"Step #{i}:", step) |
| 294 | print("Answer:", message.parsed.final_answer) |
| 295 | ``` |
| 296 | |
| 297 | Output: |
| 298 | |
| 299 | ```console |
| 300 | ParsedChatCompletionMessage[MathResponse](content='{ "steps": [{ "explanation": "First, let\'s isolate the term with the variable \'x\'. To do this, we\'ll subtract 31 from both sides of the equation.", "output": "8x + 31 - 31 = 2 - 31"}, { "explanation": "By subtracting 31 from both sides, we simplify the equation to 8x = -29.", "output": "8x = -29"}, { "explanation": "Next, let\'s isolate \'x\' by dividing both sides of the equation by 8.", "output": "8x / 8 = -29 / 8"}], "final_answer": "x = -29/8" }', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], parsed=MathResponse(steps=[Step(explanation="First, let's isolate the term with the variable 'x'. To do this, we'll subtract 31 from both sides of the equation.", output='8x + 31 - 31 = 2 - 31'), Step(explanation='By subtracting 31 from both sides, we simplify the equation to 8x = -29.', output='8x = -29'), Step(explanation="Next, let's isolate 'x' by dividing both sides of the equation by 8.", output='8x / 8 = -29 / 8')], final_answer='x = -29/8')) |
| 301 | Step #0: explanation="First, let's isolate the term with the variable 'x'. To do this, we'll subtract 31 from both sides of the equation." output='8x + 31 - 31 = 2 - 31' |
| 302 | Step #1: explanation='By subtracting 31 from both sides, we simplify the equation to 8x = -29.' output='8x = -29' |
| 303 | Step #2: explanation="Next, let's isolate 'x' by dividing both sides of the equation by 8." output='8x / 8 = -29 / 8' |
| 304 | Answer: x = -29/8 |
| 305 | ``` |
| 306 | |
| 307 | An example of using `structural_tag` can be found here: [examples/features/structured_outputs](../../examples/features/structured_outputs/README.md) |
| 308 | |
| 309 | ## Offline Inference |
| 310 | |
| 311 | Offline inference allows for the same types of structured outputs. |
| 312 | To use it, we'll need to configure the structured outputs using the class `StructuredOutputsParams` inside `SamplingParams`. |
| 313 | The main available options inside `StructuredOutputsParams` are: |
| 314 | |
| 315 | - `json` |
| 316 | - `regex` |
| 317 | - `choice` |
| 318 | - `grammar` |
| 319 | - `structural_tag` |
| 320 | |
| 321 | These parameters can be used in the same way as the parameters from the Online |
| 322 | Serving examples above. One example for the usage of the `choice` parameter is |
| 323 | shown below: |
| 324 | |
| 325 | ??? code |
| 326 | |
| 327 | ```python |
| 328 | from vllm import LLM, SamplingParams |
| 329 | from vllm.sampling_params import StructuredOutputsParams |
| 330 | |
| 331 | llm = LLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct") |
| 332 | |
| 333 | structured_outputs_params = StructuredOutputsParams(choice=["Positive", "Negative"]) |
| 334 | sampling_params = SamplingParams(structured_outputs=structured_outputs_params) |
| 335 | outputs = llm.generate( |
| 336 | prompts="Classify this sentiment: vLLM is wonderful!", |
| 337 | sampling_params=sampling_params, |
| 338 | ) |
| 339 | print(outputs[0].outputs[0].text) |
| 340 | ``` |
| 341 | |
| 342 | See also: [full example](../../examples/features/structured_outputs/structured_outputs_offline.py) |