@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / dba68f9159
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 06:00 UTC

[Doc] Unify structured outputs examples (#18196)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

Aaron Pham committed Jun 12, 2025 at 18:50 UTC dba68f9159e769c2783d96577dabb3aea15ad823
8 files changed +397 -491
docs/features/reasoning_outputs.md
-45
@@ -142,51 +142,6 @@ for chunk in stream:
142
143 Remember to check whether the `reasoning_content` exists in the response before accessing it. You could checkout the [example](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py).
144
145 -## Structured output
146 -
147 -The reasoning content is also available in the structured output. The structured output engine like `xgrammar` will use the reasoning content to generate structured output. It is only supported in v0 engine now.
148 -
149 -```bash
150 -vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --reasoning-parser deepseek_r1
151 -```
152 -
153 -The following is an example client:
154 -
155 -```python
156 -from openai import OpenAI
157 -from pydantic import BaseModel
158 -
159 -# Modify OpenAI's API key and API base to use vLLM's API server.
160 -openai_api_key = "EMPTY"
161 -openai_api_base = "http://localhost:8000/v1"
162 -
163 -client = OpenAI(
164 - api_key=openai_api_key,
165 - base_url=openai_api_base,
166 -)
167 -
168 -models = client.models.list()
169 -model = models.data[0].id
170 -
171 -class People(BaseModel):
172 - name: str
173 - age: int
174 -
175 -json_schema = People.model_json_schema()
176 -
177 -prompt = ("Generate a JSON with the name and age of one random person.")
178 -completion = client.chat.completions.create(
179 - model=model,
180 - messages=[{
181 - "role": "user",
182 - "content": prompt,
183 - }],
184 - extra_body={"guided_json": json_schema},
185 -)
186 -print("reasoning_content: ", completion.choices[0].message.reasoning_content)
187 -print("content: ", completion.choices[0].message.content)
188 -```
189 -
145 ## Tool Calling
146
147 The reasoning content is also available when both tool calling and the reasoning parser are enabled. Additionally, tool calling only parses functions from the `content` field, not from the `reasoning_content`.
docs/features/structured_outputs.md
+63 -17
@@ -39,9 +39,10 @@ client = OpenAI(
39 base_url="http://localhost:8000/v1",
40 api_key="-",
41 )
42 +model = client.models.list().data[0].id
43
44 completion = client.chat.completions.create(
44 - model="Qwen/Qwen2.5-3B-Instruct",
45 + model=model,
46 messages=[
47 {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
48 ],
@@ -54,7 +55,7 @@ The next example shows how to use the `guided_regex`. The idea is to generate an
55
56 ```python
57 completion = client.chat.completions.create(
57 - model="Qwen/Qwen2.5-3B-Instruct",
58 + model=model,
59 messages=[
60 {
61 "role": "user",
@@ -92,26 +93,32 @@ class CarDescription(BaseModel):
93 json_schema = CarDescription.model_json_schema()
94
95 completion = client.chat.completions.create(
95 - model="Qwen/Qwen2.5-3B-Instruct",
96 + model=model,
97 messages=[
98 {
99 "role": "user",
100 "content": "Generate a JSON with the brand, model and car_type of the most iconic car from the 90's",
101 }
102 ],
102 - extra_body={"guided_json": json_schema},
103 + "response_format": {
104 + "type": "json_schema",
105 + "json_schema": {
106 + "name": "car-description",
107 + "schema": CarDescription.model_json_schema()
108 + },
109 + },
110 )
111 print(completion.choices[0].message.content)
112 ```
113
114 !!! tip
115 While not strictly necessary, normally it´s better to indicate in the prompt the
109 - JSON schema and how the fields should be populated. This can improve the
116 + JSON schema and how the fields should be populated. This can improve the
117 results notably in most cases.
118
119 Finally we have the `guided_grammar` option, which is probably the most
120 difficult to use, but it´s really powerful. It allows us to define complete
114 -languages like SQL queries. It works by using a context free EBNF grammar.
121 +languages like SQL queries. It works by using a context free EBNF grammar.
122 As an example, we can use to define a specific format of simplified SQL queries:
123
124 ```python
@@ -130,7 +137,7 @@ simplified_sql_grammar = """
137 """
138
139 completion = client.chat.completions.create(
133 - model="Qwen/Qwen2.5-3B-Instruct",
140 + model=model,
141 messages=[
142 {
143 "role": "user",
@@ -142,7 +149,48 @@ completion = client.chat.completions.create(
149 print(completion.choices[0].message.content)
150 ```
151
145 -Full example: <gh-file:examples/online_serving/openai_chat_completion_structured_outputs.py>
152 +See also: [full example](../../examples/online_serving/structured_outputs)
153 +
154 +## Reasoning Outputs
155 +
156 +You can also use structured outputs with <project:#reasoning-outputs> for reasoning models.
157 +
158 +```bash
159 +vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --reasoning-parser deepseek_r1
160 +```
161 +
162 +Note that you can use reasoning with any provided structured outputs feature. The following uses one with JSON schema:
163 +
164 +```python
165 +from pydantic import BaseModel
166 +
167 +
168 +class People(BaseModel):
169 + name: str
170 + age: int
171 +
172 +
173 +completion = client.chat.completions.create(
174 + model=model,
175 + messages=[
176 + {
177 + "role": "user",
178 + "content": "Generate a JSON with the name and age of one random person.",
179 + }
180 + ],
181 + response_format={
182 + "type": "json_schema",
183 + "json_schema": {
184 + "name": "people",
185 + "schema": People.model_json_schema()
186 + }
187 + },
188 +)
189 +print("reasoning_content: ", completion.choices[0].message.reasoning_content)
190 +print("content: ", completion.choices[0].message.content)
191 +```
192 +
193 +See also: [full example](../../examples/online_serving/structured_outputs)
194
195 ## Experimental Automatic Parsing (OpenAI API)
196
@@ -163,14 +211,14 @@ class Info(BaseModel):
211 age: int
212
213 client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
214 +model = client.models.list().data[0].id
215 completion = client.beta.chat.completions.parse(
167 - model="meta-llama/Llama-3.1-8B-Instruct",
216 + model=model,
217 messages=[
218 {"role": "system", "content": "You are a helpful assistant."},
219 {"role": "user", "content": "My name is Cameron, I'm 28. What's my name and age?"},
220 ],
221 response_format=Info,
173 - extra_body=dict(guided_decoding_backend="outlines"),
222 )
223
224 message = completion.choices[0].message
@@ -203,15 +251,13 @@ class MathResponse(BaseModel):
251 steps: list[Step]
252 final_answer: str
253
206 -client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
254 completion = client.beta.chat.completions.parse(
208 - model="meta-llama/Llama-3.1-8B-Instruct",
255 + model=model,
256 messages=[
257 {"role": "system", "content": "You are a helpful expert math tutor."},
258 {"role": "user", "content": "Solve 8x + 31 = 2."},
259 ],
260 response_format=MathResponse,
214 - extra_body=dict(guided_decoding_backend="outlines"),
261 )
262
263 message = completion.choices[0].message
@@ -232,11 +278,11 @@ Step #2: explanation="Next, let's isolate 'x' by dividing both sides of the equa
278 Answer: x = -29/8
279 ```
280
235 -An example of using `structural_tag` can be found here: <gh-file:examples/online_serving/openai_chat_completion_structured_outputs_structural_tag.py>
281 +An example of using `structural_tag` can be found here: <gh-file:examples/online_serving/structured_outputs>
282
283 ## Offline Inference
284
239 -Offline inference allows for the same types of guided decoding.
285 +Offline inference allows for the same types of structured outputs.
286 To use it, we´ll need to configure the guided decoding using the class `GuidedDecodingParams` inside `SamplingParams`.
287 The main available options inside `GuidedDecodingParams` are:
288
@@ -247,7 +293,7 @@ The main available options inside `GuidedDecodingParams` are:
293 - `structural_tag`
294
295 These parameters can be used in the same way as the parameters from the Online
250 -Serving examples above. One example for the usage of the `choice` parameter is
296 +Serving examples above. One example for the usage of the `choice` parameter is
297 shown below:
298
299 ```python
@@ -265,4 +311,4 @@ outputs = llm.generate(
311 print(outputs[0].outputs[0].text)
312 ```
313
268 -Full example: <gh-file:examples/offline_inference/structured_outputs.py>
314 +See also: [full example](../../examples/online_serving/structured_outputs)
examples/online_serving/openai_chat_completion_structured_outputs.py deleted
-175
@@ -1,175 +0,0 @@
1 -# SPDX-License-Identifier: Apache-2.0
2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 -"""
4 -To run this example, you need to start the vLLM server:
5 -
6 -```bash
7 -vllm serve Qwen/Qwen2.5-3B-Instruct
8 -```
9 -"""
10 -
11 -from enum import Enum
12 -
13 -from openai import BadRequestError, OpenAI
14 -from pydantic import BaseModel
15 -
16 -openai_api_key = "EMPTY"
17 -openai_api_base = "http://localhost:8000/v1"
18 -
19 -
20 -# Guided decoding by Choice (list of possible options)
21 -def guided_choice_completion(client: OpenAI, model: str):
22 - completion = client.chat.completions.create(
23 - model=model,
24 - messages=[
25 - {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
26 - ],
27 - extra_body={"guided_choice": ["positive", "negative"]},
28 - )
29 - return completion.choices[0].message.content
30 -
31 -
32 -# Guided decoding by Regex
33 -def guided_regex_completion(client: OpenAI, model: str):
34 - prompt = (
35 - "Generate an email address for Alan Turing, who works in Enigma."
36 - "End in .com and new line. Example result:"
37 - "alan.turing@enigma.com\n"
38 - )
39 -
40 - completion = client.chat.completions.create(
41 - model=model,
42 - messages=[
43 - {
44 - "role": "user",
45 - "content": prompt,
46 - }
47 - ],
48 - extra_body={"guided_regex": r"\w+@\w+\.com\n", "stop": ["\n"]},
49 - )
50 - return completion.choices[0].message.content
51 -
52 -
53 -# Guided decoding by JSON using Pydantic schema
54 -class CarType(str, Enum):
55 - sedan = "sedan"
56 - suv = "SUV"
57 - truck = "Truck"
58 - coupe = "Coupe"
59 -
60 -
61 -class CarDescription(BaseModel):
62 - brand: str
63 - model: str
64 - car_type: CarType
65 -
66 -
67 -def guided_json_completion(client: OpenAI, model: str):
68 - json_schema = CarDescription.model_json_schema()
69 -
70 - prompt = (
71 - "Generate a JSON with the brand, model and car_type of"
72 - "the most iconic car from the 90's"
73 - )
74 - completion = client.chat.completions.create(
75 - model=model,
76 - messages=[
77 - {
78 - "role": "user",
79 - "content": prompt,
80 - }
81 - ],
82 - extra_body={"guided_json": json_schema},
83 - )
84 - return completion.choices[0].message.content
85 -
86 -
87 -# Guided decoding by Grammar
88 -def guided_grammar_completion(client: OpenAI, model: str):
89 - simplified_sql_grammar = """
90 - root ::= select_statement
91 -
92 - select_statement ::= "SELECT " column " from " table " where " condition
93 -
94 - column ::= "col_1 " | "col_2 "
95 -
96 - table ::= "table_1 " | "table_2 "
97 -
98 - condition ::= column "= " number
99 -
100 - number ::= "1 " | "2 "
101 - """
102 -
103 - prompt = (
104 - "Generate an SQL query to show the 'username' and 'email'"
105 - "from the 'users' table."
106 - )
107 - completion = client.chat.completions.create(
108 - model=model,
109 - messages=[
110 - {
111 - "role": "user",
112 - "content": prompt,
113 - }
114 - ],
115 - extra_body={"guided_grammar": simplified_sql_grammar},
116 - )
117 - return completion.choices[0].message.content
118 -
119 -
120 -# Extra backend options
121 -def extra_backend_options_completion(client: OpenAI, model: str):
122 - prompt = (
123 - "Generate an email address for Alan Turing, who works in Enigma."
124 - "End in .com and new line. Example result:"
125 - "alan.turing@enigma.com\n"
126 - )
127 -
128 - try:
129 - # The guided_decoding_disable_fallback option forces vLLM to use
130 - # xgrammar, so when it fails you get a 400 with the reason why
131 - completion = client.chat.completions.create(
132 - model=model,
133 - messages=[
134 - {
135 - "role": "user",
136 - "content": prompt,
137 - }
138 - ],
139 - extra_body={
140 - "guided_regex": r"\w+@\w+\.com\n",
141 - "stop": ["\n"],
142 - "guided_decoding_disable_fallback": True,
143 - },
144 - )
145 - return completion.choices[0].message.content
146 - except BadRequestError as e:
147 - print("This error is expected:", e)
148 -
149 -
150 -def main():
151 - client: OpenAI = OpenAI(
152 - base_url=openai_api_base,
153 - api_key=openai_api_key,
154 - )
155 -
156 - model = client.models.list().data[0].id
157 -
158 - print("Guided Choice Completion:")
159 - print(guided_choice_completion(client, model))
160 -
161 - print("\nGuided Regex Completion:")
162 - print(guided_regex_completion(client, model))
163 -
164 - print("\nGuided JSON Completion:")
165 - print(guided_json_completion(client, model))
166 -
167 - print("\nGuided Grammar Completion:")
168 - print(guided_grammar_completion(client, model))
169 -
170 - print("\nExtra Backend Options Completion:")
171 - print(extra_backend_options_completion(client, model))
172 -
173 -
174 -if __name__ == "__main__":
175 - main()
examples/online_serving/openai_chat_completion_structured_outputs_structural_tag.py deleted
-87
@@ -1,87 +0,0 @@
1 -# SPDX-License-Identifier: Apache-2.0
2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 -from openai import OpenAI
4 -
5 -# This example demonstrates the `structural_tag` response format.
6 -# It can be used to specify a structured output format that occurs between
7 -# specific tags in the response. This example shows how it could be used
8 -# to enforce the format of a tool call response, but it could be used for
9 -# any structured output within a subset of the response.
10 -
11 -openai_api_key = "EMPTY"
12 -openai_api_base = "http://localhost:8000/v1"
13 -
14 -
15 -def main():
16 - client = OpenAI(
17 - base_url=openai_api_base,
18 - api_key=openai_api_key,
19 - )
20 -
21 - messages = [
22 - {
23 - "role": "user",
24 - "content": """
25 -You have access to the following function to retrieve the weather in a city:
26 -
27 - {
28 - "name": "get_weather",
29 - "parameters": {
30 - "city": {
31 - "param_type": "string",
32 - "description": "The city to get the weather for",
33 - "required": True
34 - }
35 - }
36 - }
37 -
38 -If a you choose to call a function ONLY reply in the following format:
39 -<{start_tag}={function_name}>{parameters}{end_tag}
40 -where
41 -
42 -start_tag => `<function`
43 -parameters => a JSON dict with the function argument name as key and function
44 - argument value as value.
45 -end_tag => `</function>`
46 -
47 -Here is an example,
48 -<function=example_function_name>{"example_name": "example_value"}</function>
49 -
50 -Reminder:
51 -- Function calls MUST follow the specified format
52 -- Required parameters MUST be specified
53 -- Only call one function at a time
54 -- Put the entire function call reply on one line
55 -- Always add your sources when using search results to answer the user query
56 -
57 -You are a helpful assistant.
58 -
59 -Given the previous instructions, what is the weather in New York City, Boston,
60 -and San Francisco?
61 -""",
62 - }
63 - ]
64 -
65 - response = client.chat.completions.create(
66 - model=client.models.list().data[0].id,
67 - messages=messages,
68 - response_format={
69 - "type": "structural_tag",
70 - "structures": [
71 - {
72 - "begin": "<function=get_weather>",
73 - "schema": {
74 - "type": "object",
75 - "properties": {"city": {"type": "string"}},
76 - },
77 - "end": "</function>",
78 - }
79 - ],
80 - "triggers": ["<function="],
81 - },
82 - )
83 - print(response)
84 -
85 -
86 -if __name__ == "__main__":
87 - main()
examples/online_serving/openai_chat_completion_structured_outputs_with_reasoning.py deleted
-167
@@ -1,167 +0,0 @@
1 -# SPDX-License-Identifier: Apache-2.0
2 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 -"""
4 -An example shows how to generate structured outputs from reasoning models
5 -like DeepSeekR1. The thinking process will not be guided by the JSON
6 -schema provided by the user. Only the final output will be structured.
7 -
8 -To run this example, you need to start the vLLM server with the reasoning
9 -parser:
10 -
11 -```bash
12 -vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
13 - --reasoning-parser deepseek_r1
14 -```
15 -
16 -This example demonstrates how to generate chat completions from reasoning models
17 -using the OpenAI Python client library.
18 -"""
19 -
20 -from enum import Enum
21 -
22 -from openai import OpenAI
23 -from pydantic import BaseModel
24 -
25 -# Modify OpenAI's API key and API base to use vLLM's API server.
26 -openai_api_key = "EMPTY"
27 -openai_api_base = "http://localhost:8000/v1"
28 -
29 -
30 -def print_completion_details(completion):
31 - print("reasoning_content: ", completion.choices[0].message.reasoning_content)
32 - print("content: ", completion.choices[0].message.content)
33 -
34 -
35 -# Guided decoding by Regex
36 -def guided_regex_completion(client: OpenAI, model: str):
37 - prompt = "What is the capital of France?"
38 -
39 - completion = client.chat.completions.create(
40 - model=model,
41 - messages=[
42 - {
43 - "role": "user",
44 - "content": prompt,
45 - }
46 - ],
47 - extra_body={
48 - "guided_regex": "(Paris|London)",
49 - },
50 - )
51 - print_completion_details(completion)
52 -
53 -
54 -class People(BaseModel):
55 - name: str
56 - age: int
57 -
58 -
59 -def guided_json_completion(client: OpenAI, model: str):
60 - json_schema = People.model_json_schema()
61 -
62 - prompt = "Generate a JSON with the name and age of one random person."
63 - completion = client.chat.completions.create(
64 - model=model,
65 - messages=[
66 - {
67 - "role": "user",
68 - "content": prompt,
69 - }
70 - ],
71 - extra_body={"guided_json": json_schema},
72 - )
73 - print_completion_details(completion)
74 -
75 -
76 -# Guided decoding by JSON using Pydantic schema
77 -class CarType(str, Enum):
78 - sedan = "sedan"
79 - suv = "SUV"
80 - truck = "Truck"
81 - coupe = "Coupe"
82 -
83 -
84 -class CarDescription(BaseModel):
85 - brand: str
86 - model: str
87 - car_type: CarType
88 -
89 -
90 -def guided_car_json_completion(client: OpenAI, model: str):
91 - json_schema = CarDescription.model_json_schema()
92 -
93 - prompt = (
94 - "Generate a JSON with the brand, model and car_type of"
95 - "the most iconic car from the 90's"
96 - )
97 - completion = client.chat.completions.create(
98 - model=model,
99 - messages=[
100 - {
101 - "role": "user",
102 - "content": prompt,
103 - }
104 - ],
105 - extra_body={"guided_json": json_schema},
106 - )
107 - print_completion_details(completion)
108 -
109 -
110 -# Guided decoding by Grammar
111 -def guided_grammar_completion(client: OpenAI, model: str):
112 - simplified_sql_grammar = """
113 - root ::= select_statement
114 -
115 - select_statement ::= "SELECT " column " from " table " where " condition
116 -
117 - column ::= "col_1 " | "col_2 "
118 -
119 - table ::= "table_1 " | "table_2 "
120 -
121 - condition ::= column "= " number
122 -
123 - number ::= "1 " | "2 "
124 - """
125 -
126 - # This may be very slow https://github.com/vllm-project/vllm/issues/12122
127 - prompt = (
128 - "Generate an SQL query to show the 'username' and 'email'"
129 - "from the 'users' table."
130 - )
131 - completion = client.chat.completions.create(
132 - model=model,
133 - messages=[
134 - {
135 - "role": "user",
136 - "content": prompt,
137 - }
138 - ],
139 - extra_body={"guided_grammar": simplified_sql_grammar},
140 - )
141 - print_completion_details(completion)
142 -
143 -
144 -def main():
145 - client: OpenAI = OpenAI(
146 - api_key=openai_api_key,
147 - base_url=openai_api_base,
148 - )
149 -
150 - models = client.models.list()
151 - model: str = models.data[0].id
152 -
153 - print("Guided Regex Completion:")
154 - guided_regex_completion(client, model)
155 -
156 - print("\nGuided JSON Completion (People):")
157 - guided_json_completion(client, model)
158 -
159 - print("\nGuided JSON Completion (CarDescription):")
160 - guided_car_json_completion(client, model)
161 -
162 - print("\nGuided Grammar Completion:")
163 - guided_grammar_completion(client, model)
164 -
165 -
166 -if __name__ == "__main__":
167 - main()
examples/online_serving/structured_outputs/README.md new
+54
@@ -0,0 +1,54 @@
1 +# Structured Outputs
2 +
3 +This script demonstrates various structured output capabilities of vLLM's OpenAI-compatible server.
4 +It can run individual constraint type or all of them.
5 +It supports both streaming responses and concurrent non-streaming requests.
6 +
7 +To use this example, you must start an vLLM server with any model of your choice.
8 +
9 +```bash
10 +vllm serve Qwen/Qwen2.5-3B-Instruct
11 +```
12 +
13 +To serve a reasoning model, you can use the following command:
14 +
15 +```bash
16 +vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --reasoning-parser deepseek_r1
17 +```
18 +
19 +If you want to run this script standalone with `uv`, you can use the following:
20 +
21 +```bash
22 +uvx --from git+https://github.com/vllm-project/vllm#subdirectory=examples/online_serving/structured_outputs structured-output
23 +```
24 +
25 +See [feature docs](../../../features/structured_outputs.md) for more information.
26 +
27 +!!! tip
28 + If vLLM is running remotely, then set `OPENAI_BASE_URL=<remote_url>` before running the script.
29 +
30 +## Usage
31 +
32 +Run all constraints, non-streaming:
33 +
34 +```bash
35 +uv run structured_outputs.py
36 +```
37 +
38 +Run all constraints, streaming:
39 +
40 +```bash
41 +uv run structured_outputs.py --stream
42 +```
43 +
44 +Run certain constraints, for example `structural_tag` and `regex`, streaming:
45 +
46 +```bash
47 +uv run structured_outputs.py --constraint structural_tag regex --stream
48 +```
49 +
50 +Run all constraints, with reasoning models and streaming:
51 +
52 +```bash
53 +uv run structured_outputs.py --reasoning --stream
54 +```
examples/online_serving/structured_outputs/pyproject.toml new
+8
@@ -0,0 +1,8 @@
1 +[project]
2 +name = "examples-online-structured-outputs"
3 +requires-python = ">=3.9, <3.13"
4 +dependencies = ["openai==1.78.1", "pydantic==2.11.4"]
5 +version = "0.0.0"
6 +
7 +[project.scripts]
8 +structured-outputs = "structured_outputs:main"
examples/online_serving/structured_outputs/structured_outputs.py new
+272
@@ -0,0 +1,272 @@
1 +# ruff: noqa: E501
2 +# SPDX-License-Identifier: Apache-2.0
3 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
4 +
5 +from __future__ import annotations
6 +
7 +import argparse
8 +import asyncio
9 +import enum
10 +import os
11 +from typing import TYPE_CHECKING, Any, Literal
12 +
13 +import openai
14 +import pydantic
15 +
16 +if TYPE_CHECKING:
17 + from openai.types.chat import ChatCompletionChunk
18 +
19 +
20 +ConstraintsFormat = Literal[
21 + "choice",
22 + "regex",
23 + "json",
24 + "grammar",
25 + "structural_tag",
26 +]
27 +
28 +
29 +async def print_stream_response(
30 + stream_response: openai.AsyncStream[ChatCompletionChunk],
31 + title: str,
32 + args: argparse.Namespace,
33 +):
34 + print(f"\n\n{title} (Streaming):")
35 +
36 + local_reasoning_header_printed = False
37 + local_content_header_printed = False
38 +
39 + async for chunk in stream_response:
40 + delta = chunk.choices[0].delta
41 +
42 + reasoning_chunk_text: str | None = getattr(delta, "reasoning_content", None)
43 + content_chunk_text = delta.content
44 +
45 + if args.reasoning:
46 + if reasoning_chunk_text:
47 + if not local_reasoning_header_printed:
48 + print(" Reasoning: ", end="")
49 + local_reasoning_header_printed = True
50 + print(reasoning_chunk_text, end="", flush=True)
51 +
52 + if content_chunk_text:
53 + if not local_content_header_printed:
54 + if local_reasoning_header_printed:
55 + print()
56 + print(" Content: ", end="")
57 + local_content_header_printed = True
58 + print(content_chunk_text, end="", flush=True)
59 + else:
60 + if content_chunk_text:
61 + if not local_content_header_printed:
62 + print(" Content: ", end="")
63 + local_content_header_printed = True
64 + print(content_chunk_text, end="", flush=True)
65 + print()
66 +
67 +
68 +class CarType(str, enum.Enum):
69 + SEDAN = "SEDAN"
70 + SUV = "SUV"
71 + TRUCK = "TRUCK"
72 + COUPE = "COUPE"
73 +
74 +
75 +class CarDescription(pydantic.BaseModel):
76 + brand: str
77 + model: str
78 + car_type: CarType
79 +
80 +
81 +PARAMS: dict[ConstraintsFormat, dict[str, Any]] = {
82 + "choice": {
83 + "messages": [
84 + {
85 + "role": "user",
86 + "content": "Classify this sentiment: vLLM is wonderful!",
87 + }
88 + ],
89 + "extra_body": {"guided_choice": ["positive", "negative"]},
90 + },
91 + "regex": {
92 + "messages": [
93 + {
94 + "role": "user",
95 + "content": "Generate an email address for Alan Turing, who works in Enigma. End in .com and new line. Example result: 'alan.turing@enigma.com\n'",
96 + }
97 + ],
98 + "extra_body": {
99 + "guided_regex": r"[a-z0-9.]{1,20}@\w{6,10}\.com\n",
100 + },
101 + },
102 + "json": {
103 + "messages": [
104 + {
105 + "role": "user",
106 + "content": "Generate a JSON with the brand, model and car_type of the most iconic car from the 90's",
107 + }
108 + ],
109 + "response_format": {
110 + "type": "json_schema",
111 + "json_schema": {
112 + "name": "car-description",
113 + "schema": CarDescription.model_json_schema(),
114 + },
115 + },
116 + },
117 + "grammar": {
118 + "messages": [
119 + {
120 + "role": "user",
121 + "content": "Generate an SQL query to show the 'username' and 'email'from the 'users' table.",
122 + }
123 + ],
124 + "extra_body": {
125 + "guided_grammar": """
126 +root ::= select_statement
127 +
128 +select_statement ::= "SELECT " column " from " table " where " condition
129 +
130 +column ::= "col_1 " | "col_2 "
131 +
132 +table ::= "table_1 " | "table_2 "
133 +
134 +condition ::= column "= " number
135 +
136 +number ::= "1 " | "2 "
137 +""",
138 + },
139 + },
140 + "structural_tag": {
141 + "messages": [
142 + {
143 + "role": "user",
144 + "content": """
145 +You have access to the following function to retrieve the weather in a city:
146 +
147 +{
148 + "name": "get_weather",
149 + "parameters": {
150 + "city": {
151 + "param_type": "string",
152 + "description": "The city to get the weather for",
153 + "required": True
154 + }
155 + }
156 +}
157 +
158 +If a you choose to call a function ONLY reply in the following format:
159 +<{start_tag}={function_name}>{parameters}{end_tag}
160 +where
161 +
162 +start_tag => `<function`
163 +parameters => a JSON dict with the function argument name as key and function
164 + argument value as value.
165 +end_tag => `</function>`
166 +
167 +Here is an example,
168 +<function=example_function_name>{"example_name": "example_value"}</function>
169 +
170 +Reminder:
171 +- Function calls MUST follow the specified format
172 +- Required parameters MUST be specified
173 +- Only call one function at a time
174 +- Put the entire function call reply on one line
175 +- Always add your sources when using search results to answer the user query
176 +
177 +You are a helpful assistant.
178 +
179 +Given the previous instructions, what is the weather in New York City, Boston,
180 +and San Francisco?""",
181 + },
182 + ],
183 + "response_format": {
184 + "type": "structural_tag",
185 + "structures": [
186 + {
187 + "begin": "<function=get_weather>",
188 + "schema": {
189 + "type": "object",
190 + "properties": {"city": {"type": "string"}},
191 + "required": ["city"],
192 + },
193 + "end": "</function>",
194 + }
195 + ],
196 + "triggers": ["<function="],
197 + },
198 + },
199 +}
200 +
201 +
202 +async def cli():
203 + parser = argparse.ArgumentParser(
204 + description="Run OpenAI Chat Completion with various structured outputs capabilities",
205 + )
206 + _ = parser.add_argument(
207 + "--constraint",
208 + type=str,
209 + nargs="+",
210 + choices=[*list(PARAMS), "*"],
211 + default=["*"],
212 + help="Specify which constraint(s) to run.",
213 + )
214 + _ = parser.add_argument(
215 + "--stream",
216 + action=argparse.BooleanOptionalAction,
217 + default=False,
218 + help="Enable streaming output",
219 + )
220 + _ = parser.add_argument(
221 + "--reasoning",
222 + action=argparse.BooleanOptionalAction,
223 + default=False,
224 + help="Enable printing of reasoning traces if available.",
225 + )
226 + args = parser.parse_args()
227 +
228 + base_url = os.getenv("OPENAI_BASE_URL", "http://localhost:8000/v1")
229 + client = openai.AsyncOpenAI(base_url=base_url, api_key="EMPTY")
230 + constraints = list(PARAMS) if "*" in args.constraint else list(set(args.constraint))
231 + model = (await client.models.list()).data[0].id
232 +
233 + if args.stream:
234 + results = await asyncio.gather(
235 + *[
236 + client.chat.completions.create(
237 + model=model,
238 + max_tokens=1024,
239 + stream=True,
240 + **PARAMS[name],
241 + )
242 + for name in constraints
243 + ]
244 + )
245 + for constraint, stream in zip(constraints, results):
246 + await print_stream_response(stream, constraint, args)
247 + else:
248 + results = await asyncio.gather(
249 + *[
250 + client.chat.completions.create(
251 + model=model,
252 + max_tokens=1024,
253 + stream=False,
254 + **PARAMS[name],
255 + )
256 + for name in constraints
257 + ]
258 + )
259 + for constraint, response in zip(constraints, results):
260 + print(f"\n\n{constraint}:")
261 + message = response.choices[0].message
262 + if args.reasoning and hasattr(message, "reasoning_content"):
263 + print(f" Reasoning: {message.reasoning_content or ''}")
264 + print(f" Content: {message.content!r}")
265 +
266 +
267 +def main():
268 + asyncio.run(cli())
269 +
270 +
271 +if __name__ == "__main__":
272 + main()