@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 74f8c2cf5f
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 07:00 UTC

Add openai.beta.chat.completions.parse example to structured_outputs.rst (#10433)

Michael Goin committed Nov 18, 2024 at 23:37 UTC 74f8c2cf5f6a34fd21cfbe6d72bcc1b2a2a6754a
1 file changed +96 -2
docs/source/models/structured_outputs.rst
+96 -2
@@ -10,7 +10,7 @@ This document shows you some examples of the different options that are availabl
10 Online Inference (OpenAI API)
11 -----------------------------
12
13 -You can generate structured outputs using the OpenAI’s `Completions <https://platform.openai.com/docs/api-reference/completions>`_ and `Chat <https://platform.openai.com/docs/api-reference/chat>`_ API.
13 +You can generate structured outputs using the OpenAI's `Completions <https://platform.openai.com/docs/api-reference/completions>`_ and `Chat <https://platform.openai.com/docs/api-reference/chat>`_ API.
14
15 The following parameters are supported, which must be added as extra parameters:
16
@@ -137,6 +137,100 @@ It works by using a context free EBNF grammar, which for example we can use to d
137
138 The complete code of the examples can be found on `examples/openai_chat_completion_structured_outputs.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_chat_completion_structured_outputs.py>`_.
139
140 +Experimental Automatic Parsing (OpenAI API)
141 +--------------------------------------------
142 +
143 +This section covers the OpenAI beta wrapper over the ``client.chat.completions.create()`` method that provides richer integrations with Python specific types.
144 +
145 +At the time of writing (``openai==1.54.4``), this is a "beta" feature in the OpenAI client library. Code reference can be found `here <https://github.com/openai/openai-python/blob/52357cff50bee57ef442e94d78a0de38b4173fc2/src/openai/resources/beta/chat/completions.py#L100-L104>`_.
146 +
147 +For the following examples, vLLM was setup using ``vllm serve meta-llama/Llama-3.1-8B-Instruct``
148 +
149 +Here is a simple example demonstrating how to get structured output using Pydantic models:
150 +
151 +.. code-block:: python
152 +
153 + from pydantic import BaseModel
154 + from openai import OpenAI
155 +
156 +
157 + class Info(BaseModel):
158 + name: str
159 + age: int
160 +
161 +
162 + client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
163 + completion = client.beta.chat.completions.parse(
164 + model="meta-llama/Llama-3.1-8B-Instruct",
165 + messages=[
166 + {"role": "system", "content": "You are a helpful assistant."},
167 + {"role": "user", "content": "My name is Cameron, I'm 28. What's my name and age?"},
168 + ],
169 + response_format=Info,
170 + extra_body=dict(guided_decoding_backend="outlines"),
171 + )
172 +
173 + message = completion.choices[0].message
174 + print(message)
175 + assert message.parsed
176 + print("Name:", message.parsed.name)
177 + print("Age:", message.parsed.age)
178 +
179 +Output:
180 +
181 +.. code-block:: console
182 +
183 + ParsedChatCompletionMessage[Testing](content='{"name": "Cameron", "age": 28}', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], parsed=Testing(name='Cameron', age=28))
184 + Name: Cameron
185 + Age: 28
186 +
187 +
188 +Here is a more complex example using nested Pydantic models to handle a step-by-step math solution:
189 +
190 +.. code-block:: python
191 +
192 + from typing import List
193 + from pydantic import BaseModel
194 + from openai import OpenAI
195 +
196 +
197 + class Step(BaseModel):
198 + explanation: str
199 + output: str
200 +
201 +
202 + class MathResponse(BaseModel):
203 + steps: List[Step]
204 + final_answer: str
205 +
206 +
207 + client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
208 + completion = client.beta.chat.completions.parse(
209 + model="meta-llama/Llama-3.1-8B-Instruct",
210 + messages=[
211 + {"role": "system", "content": "You are a helpful expert math tutor."},
212 + {"role": "user", "content": "Solve 8x + 31 = 2."},
213 + ],
214 + response_format=MathResponse,
215 + extra_body=dict(guided_decoding_backend="outlines"),
216 + )
217 +
218 + message = completion.choices[0].message
219 + print(message)
220 + assert message.parsed
221 + for i, step in enumerate(message.parsed.steps):
222 + print(f"Step #{i}:", step)
223 + print("Answer:", message.parsed.final_answer)
224 +
225 +Output:
226 +
227 +.. code-block:: console
228 +
229 + ParsedChatCompletionMessage[MathResponse](content='{ "steps": [{ "explanation": "First, let\'s isolate the term with the variable \'x\'. To do this, we\'ll subtract 31 from both sides of the equation.", "output": "8x + 31 - 31 = 2 - 31"}, { "explanation": "By subtracting 31 from both sides, we simplify the equation to 8x = -29.", "output": "8x = -29"}, { "explanation": "Next, let\'s isolate \'x\' by dividing both sides of the equation by 8.", "output": "8x / 8 = -29 / 8"}], "final_answer": "x = -29/8" }', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], parsed=MathResponse(steps=[Step(explanation="First, let's isolate the term with the variable 'x'. To do this, we'll subtract 31 from both sides of the equation.", output='8x + 31 - 31 = 2 - 31'), Step(explanation='By subtracting 31 from both sides, we simplify the equation to 8x = -29.', output='8x = -29'), Step(explanation="Next, let's isolate 'x' by dividing both sides of the equation by 8.", output='8x / 8 = -29 / 8')], final_answer='x = -29/8'))
230 + Step #0: explanation="First, let's isolate the term with the variable 'x'. To do this, we'll subtract 31 from both sides of the equation." output='8x + 31 - 31 = 2 - 31'
231 + Step #1: explanation='By subtracting 31 from both sides, we simplify the equation to 8x = -29.' output='8x = -29'
232 + Step #2: explanation="Next, let's isolate 'x' by dividing both sides of the equation by 8." output='8x / 8 = -29 / 8'
233 + Answer: x = -29/8
234
235 Offline Inference
236 -----------------
@@ -170,4 +264,4 @@ One example for the usage of the ``choices`` parameter is shown below:
264 )
265 print(outputs[0].outputs[0].text)
266
173 -A complete example with all options can be found in `examples/offline_inference_structured_outputs.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_structured_outputs.py>`_.
\ No newline at end of file
267 +A complete example with all options can be found in `examples/offline_inference_structured_outputs.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_structured_outputs.py>`_.