@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / cefb9e5a28
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 16:00 UTC

[Frontend] Implement Tool Calling with `tool_choice='required'` (#13483)

Signed-off-by: Liangfu Chen <liangfc@amazon.com> Signed-off-by: Matt, Matthias <matthias.matt@tuwien.ac.at> Co-authored-by: Liangfu Chen <liangfc@amazon.com> Co-authored-by: mgoin <michael@neuralmagic.com>

Matthias Matt committed Apr 2, 2025 at 16:45 UTC cefb9e5a2816da74c22115236e35c5a75ef59484
7 files changed +868 -93
docs/source/features/tool_calling.md
+7 -1
@@ -1,6 +1,6 @@
1 # Tool Calling
2
3 -vLLM currently supports named function calling, as well as the `auto` and `none` options for the `tool_choice` field in the chat completion API. The `tool_choice` option `required` is **not yet supported** but [on the roadmap](gh-issue:13002).
3 +vLLM currently supports named function calling, as well as the `auto`, `required` (as of `vllm>=0.8.3`) and `none` options for the `tool_choice` field in the chat completion API.
4
5 ## Quickstart
6
@@ -91,6 +91,12 @@ For best results, we recommend ensuring that the expected output format / schema
91 To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
92 specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
93
94 +## Required Function Calling
95 +
96 +vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses guided decoding, so this is enabled by default and will work with any supported model. The required guided decoding features (JSON schema with `anyOf`) are currently only supported in the V0 engine with the guided decoding backend `outlines`. However, support for alternative decoding backends are on the [roadmap](https://docs.vllm.ai/en/latest/getting_started/v1_user_guide.html#feature-model) for the V1 engine.
97 +
98 +When tool_choice='required' is set, the model is guaranteed to generate one or more tool calls based on the specified tool list in the `tools` parameter. The number of tool calls depends on the user's query. The output format strictly follows the schema defined in the `tools` parameter.
99 +
100 ## Automatic Function Calling
101
102 To enable this feature, you should set the following flags:
examples/online_serving/openai_chat_completion_client_with_tools_required.py new
+136
@@ -0,0 +1,136 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +"""
3 +To run this example, you can start the vLLM server
4 +without any specific flags:
5 +
6 +```bash
7 +VLLM_USE_V1=0 vllm serve unsloth/Llama-3.2-1B-Instruct \
8 + --guided-decoding-backend outlines
9 +```
10 +
11 +This example demonstrates how to generate chat completions
12 +using the OpenAI Python client library.
13 +"""
14 +
15 +from openai import OpenAI
16 +
17 +# Modify OpenAI's API key and API base to use vLLM's API server.
18 +openai_api_key = "EMPTY"
19 +openai_api_base = "http://localhost:8000/v1"
20 +
21 +client = OpenAI(
22 + # defaults to os.environ.get("OPENAI_API_KEY")
23 + api_key=openai_api_key,
24 + base_url=openai_api_base,
25 +)
26 +
27 +models = client.models.list()
28 +model = models.data[0].id
29 +
30 +tools = [
31 + {
32 + "type": "function",
33 + "function": {
34 + "name": "get_current_weather",
35 + "description": "Get the current weather in a given location",
36 + "parameters": {
37 + "type": "object",
38 + "properties": {
39 + "city": {
40 + "type":
41 + "string",
42 + "description":
43 + "The city to find the weather for"
44 + ", e.g. 'San Francisco'",
45 + },
46 + "state": {
47 + "type":
48 + "string",
49 + "description":
50 + "the two-letter abbreviation for the state that the "
51 + "city is in, e.g. 'CA' which would mean 'California'",
52 + },
53 + "unit": {
54 + "type": "string",
55 + "description": "The unit to fetch the temperature in",
56 + "enum": ["celsius", "fahrenheit"],
57 + },
58 + },
59 + "required": ["city", "state", "unit"],
60 + },
61 + },
62 + },
63 + {
64 + "type": "function",
65 + "function": {
66 + "name": "get_forecast",
67 + "description": "Get the weather forecast for a given location",
68 + "parameters": {
69 + "type": "object",
70 + "properties": {
71 + "city": {
72 + "type":
73 + "string",
74 + "description":
75 + "The city to get the forecast for, e.g. 'New York'",
76 + },
77 + "state": {
78 + "type":
79 + "string",
80 + "description":
81 + "The two-letter abbreviation for the state, e.g. 'NY'",
82 + },
83 + "days": {
84 + "type":
85 + "integer",
86 + "description":
87 + "Number of days to get the forecast for (1-7)",
88 + },
89 + "unit": {
90 + "type": "string",
91 + "description": "The unit to fetch the temperature in",
92 + "enum": ["celsius", "fahrenheit"],
93 + },
94 + },
95 + "required": ["city", "state", "days", "unit"],
96 + },
97 + },
98 + },
99 +]
100 +
101 +messages = [
102 + {
103 + "role": "user",
104 + "content": "Hi! How are you doing today?"
105 + },
106 + {
107 + "role": "assistant",
108 + "content": "I'm doing well! How can I help you?"
109 + },
110 + {
111 + "role":
112 + "user",
113 + "content":
114 + "Can you tell me what the current weather is in Dallas \
115 + and the forecast for the next 5 days, in fahrenheit?",
116 + },
117 +]
118 +
119 +chat_completion = client.chat.completions.create(
120 + messages=messages,
121 + model=model,
122 + tools=tools,
123 + tool_choice="required",
124 + stream=True # Enable streaming response
125 +)
126 +
127 +for chunk in chat_completion:
128 + if chunk.choices and chunk.choices[0].delta.tool_calls:
129 + print(chunk.choices[0].delta.tool_calls)
130 +
131 +chat_completion = client.chat.completions.create(messages=messages,
132 + model=model,
133 + tools=tools,
134 + tool_choice="required")
135 +
136 +print(chat_completion.choices[0].message.tool_calls)
tests/entrypoints/openai/test_chat.py
+121 -42
@@ -786,56 +786,135 @@ async def test_named_tool_use(client: openai.AsyncOpenAI, is_v1_server: bool,
786
787
788 @pytest.mark.asyncio
789 -async def test_required_tool_use_not_yet_supported(client: openai.AsyncOpenAI,
790 - sample_json_schema):
791 -
789 +@pytest.mark.parametrize("model_name", [MODEL_NAME])
790 +async def test_required_tool_use(client: openai.AsyncOpenAI,
791 + is_v1_server: bool, model_name: str):
792 if is_v1_server:
793 - pytest.skip("sample_json_schema has features unsupported on V1")
793 + pytest.skip(
794 + "tool_choice='required' requires features unsupported on V1")
795
795 - messages = [{
796 - "role": "system",
797 - "content": "you are a helpful assistant"
798 - }, {
799 - "role":
800 - "user",
801 - "content":
802 - f"Give an example JSON for an employee profile that "
803 - f"fits this schema: {sample_json_schema}"
804 - }]
796 + tools = [
797 + {
798 + "type": "function",
799 + "function": {
800 + "name": "get_current_weather",
801 + "description": "Get the current weather in a given location",
802 + "parameters": {
803 + "type": "object",
804 + "properties": {
805 + "city": {
806 + "type": "string",
807 + "description":
808 + "The city to find the weather for, e.g. 'Vienna'",
809 + "default": "Vienna",
810 + },
811 + "country": {
812 + "type":
813 + "string",
814 + "description":
815 + "The country that the city is in, e.g. 'Austria'",
816 + },
817 + "unit": {
818 + "type": "string",
819 + "description":
820 + "The unit to fetch the temperature in",
821 + "enum": ["celsius", "fahrenheit"],
822 + },
823 + },
824 + "required": ["country", "unit"],
825 + },
826 + },
827 + },
828 + {
829 + "type": "function",
830 + "function": {
831 + "name": "get_forecast",
832 + "description": "Get the weather forecast for a given location",
833 + "parameters": {
834 + "type": "object",
835 + "properties": {
836 + "city": {
837 + "type": "string",
838 + "description":
839 + "The city to get the forecast for, e.g. 'Vienna'",
840 + "default": "Vienna",
841 + },
842 + "country": {
843 + "type":
844 + "string",
845 + "description":
846 + "The country that the city is in, e.g. 'Austria'",
847 + },
848 + "days": {
849 + "type":
850 + "integer",
851 + "description":
852 + "Number of days to get the forecast for (1-7)",
853 + },
854 + "unit": {
855 + "type": "string",
856 + "description":
857 + "The unit to fetch the temperature in",
858 + "enum": ["celsius", "fahrenheit"],
859 + },
860 + },
861 + "required": ["country", "days", "unit"],
862 + },
863 + },
864 + },
865 + ]
866
806 - with pytest.raises(openai.BadRequestError):
807 - await client.chat.completions.create(
808 - model=MODEL_NAME,
809 - messages=messages,
810 - max_completion_tokens=1000,
811 - tools=[{
812 - "type": "function",
813 - "function": {
814 - "name": "dummy_function_name",
815 - "description": "This is a dummy function",
816 - "parameters": sample_json_schema
817 - }
818 - }],
819 - tool_choice="required")
867 + messages = [
868 + {
869 + "role": "user",
870 + "content": "Hi! How are you doing today?"
871 + },
872 + {
873 + "role": "assistant",
874 + "content": "I'm doing well! How can I help you?"
875 + },
876 + {
877 + "role":
878 + "user",
879 + "content":
880 + "Can you tell me what the current weather is in Berlin and the "\
881 + "forecast for the next 5 days, in fahrenheit?",
882 + },
883 + ]
884
821 - with pytest.raises(openai.BadRequestError):
822 - await client.chat.completions.create(
823 - model=MODEL_NAME,
824 - messages=messages,
825 - max_completion_tokens=1000,
826 - tools=[{
827 - "type": "function",
828 - "function": {
829 - "name": "dummy_function_name",
830 - "description": "This is a dummy function",
831 - "parameters": sample_json_schema
832 - }
833 - }],
834 - tool_choice="auto")
885 + # Non-streaming test
886 + chat_completion = await client.chat.completions.create(
887 + messages=messages,
888 + model=model_name,
889 + tools=tools,
890 + tool_choice="required",
891 + extra_body=dict(guided_decoding_backend="outlines"),
892 + )
893 +
894 + assert chat_completion.choices[0].message.tool_calls is not None
895 + assert len(chat_completion.choices[0].message.tool_calls) > 0
896 +
897 + # Streaming test
898 + stream = await client.chat.completions.create(
899 + messages=messages,
900 + model=model_name,
901 + tools=tools,
902 + tool_choice="required",
903 + extra_body=dict(guided_decoding_backend="outlines"),
904 + stream=True,
905 + )
906 +
907 + output = []
908 + async for chunk in stream:
909 + if chunk.choices and chunk.choices[0].delta.tool_calls:
910 + output.extend(chunk.choices[0].delta.tool_calls)
911 +
912 + assert len(output) > 0
913
914
915 @pytest.mark.asyncio
916 async def test_inconsistent_tool_choice_and_tools(client: openai.AsyncOpenAI,
917 + is_v1_server: bool,
918 sample_json_schema):
919
920 if is_v1_server:
tests/tool_use/test_chat_completion_request_validations.py
+4 -3
@@ -43,7 +43,8 @@ def test_chat_completion_request_with_no_tools():
43 assert request.tool_choice == 'none'
44
45
46 -def test_chat_completion_request_with_tool_choice_but_no_tools():
46 +@pytest.mark.parametrize('tool_choice', ['auto', 'required'])
47 +def test_chat_completion_request_with_tool_choice_but_no_tools(tool_choice):
48 with pytest.raises(ValueError,
49 match="When using `tool_choice`, `tools` must be set."):
50 ChatCompletionRequest.model_validate({
@@ -54,7 +55,7 @@ def test_chat_completion_request_with_tool_choice_but_no_tools():
55 'model':
56 'facebook/opt-125m',
57 'tool_choice':
57 - 'auto'
58 + tool_choice
59 })
60
61 with pytest.raises(ValueError,
@@ -67,7 +68,7 @@ def test_chat_completion_request_with_tool_choice_but_no_tools():
68 'model':
69 'facebook/opt-125m',
70 'tool_choice':
70 - 'auto',
71 + tool_choice,
72 'tools':
73 None
74 })
tests/tool_use/test_tool_choice_required.py new
+336
@@ -0,0 +1,336 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +import json
3 +import re
4 +from copy import deepcopy
5 +from unittest.mock import MagicMock
6 +
7 +import pytest
8 +from pydantic import TypeAdapter
9 +
10 +from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
11 + ChatCompletionToolsParam)
12 +from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
13 +
14 +EXAMPLE_TOOLS = [
15 + {
16 + "type": "function",
17 + "function": {
18 + "name": "get_current_weather",
19 + "description": "Get the current weather in a given location",
20 + "parameters": {
21 + "type": "object",
22 + "properties": {
23 + "city": {
24 + "type":
25 + "string",
26 + "description":
27 + "The city to find the weather for"
28 + ", e.g. 'San Francisco'",
29 + },
30 + },
31 + "required": ["city"],
32 + "additionalProperties": False
33 + },
34 + },
35 + "strict": True
36 + },
37 + {
38 + "type": "function",
39 + "function": {
40 + "name": "get_forecast",
41 + "description": "Get the weather forecast for a given location",
42 + "parameters": {
43 + "type": "object",
44 + "properties": {
45 + "city": {
46 + "type":
47 + "string",
48 + "description":
49 + "The city to get the forecast for, e.g. 'New York'",
50 + },
51 + "days": {
52 + "type":
53 + "integer",
54 + "description":
55 + "Number of days to get the forecast for (1-7)",
56 + },
57 + },
58 + "required": ["city", "days"],
59 + "additionalProperties": False
60 + },
61 + },
62 + "strict": True
63 + },
64 +]
65 +
66 +
67 +def _compile_and_check(tools: list[ChatCompletionToolsParam], sample_output,
68 + should_match: bool):
69 + self = MagicMock(tool_choice="required", tools=tools)
70 + schema = ChatCompletionRequest._get_guided_json_from_tool(self)
71 + assert isinstance(schema, dict)
72 +
73 + # use build_regex_from_schema used in JSONLogitsProcessor to create Guide
74 + from outlines_core.fsm.json_schema import build_regex_from_schema
75 + regex = build_regex_from_schema(json.dumps(schema))
76 + compiled = re.compile(regex)
77 + matches = compiled.fullmatch(json.dumps(sample_output)) is not None
78 +
79 + assert matches == should_match
80 +
81 +
82 +VALID_TOOL_OUTPUTS = [
83 + ([{
84 + "name": "get_current_weather",
85 + "parameters": {
86 + "city": "Vienna"
87 + }
88 + }], True),
89 + ([{
90 + "name": "get_current_weather",
91 + "parameters": {
92 + "city": "Vienna"
93 + }
94 + }, {
95 + "name": "get_current_weather",
96 + "parameters": {
97 + "city": "Berlin"
98 + }
99 + }], True),
100 + ([{
101 + "name": "get_forecast",
102 + "parameters": {
103 + "city": "Vienna",
104 + "days": 7
105 + }
106 + }], True),
107 + ([{
108 + "name": "get_forecast",
109 + "parameters": {
110 + "city": "Vienna",
111 + "days": 7
112 + }
113 + }, {
114 + "name": "get_current_weather",
115 + "parameters": {
116 + "city": "Vienna"
117 + }
118 + }], True),
119 + ([{
120 + "name": "get_forecast",
121 + "parameters": {
122 + "city": "Vienna",
123 + "days": 7
124 + }
125 + }, {
126 + "name": "get_current_weather",
127 + "parameters": {
128 + "city": "Vienna"
129 + }
130 + }, {
131 + "name": "get_forecast",
132 + "parameters": {
133 + "city": "Berlin",
134 + "days": 7
135 + }
136 + }, {
137 + "name": "get_current_weather",
138 + "parameters": {
139 + "city": "Berlin"
140 + }
141 + }], True),
142 +]
143 +
144 +VALID_TOOLS = [t[0] for t in VALID_TOOL_OUTPUTS]
145 +
146 +
147 +@pytest.mark.parametrize(
148 + "sample_output, should_match",
149 + VALID_TOOL_OUTPUTS + [
150 + (None, False),
151 + ([], False), # empty list cannot be generated
152 + ({}, False), # empty object cannot be generated
153 + ([{}], False), # list with empty object cannot be generated
154 + (
155 + [{ # function without required parameters cannot be generated
156 + "name": "get_current_weather"
157 + }],
158 + False),
159 + (
160 + [{ # function without required parameters cannot be generated
161 + "name": "get_current_weather",
162 + "parameters": {}
163 + }],
164 + False),
165 + (
166 + [{ # function without required parameters cannot be generated
167 + "name": "get_current_weather",
168 + "parameters": None
169 + }],
170 + False),
171 + (
172 + { # tool call without lists cannot be generated
173 + "name": "get_current_weather",
174 + "parameters": {
175 + "city": "Vienna"
176 + }
177 + },
178 + False),
179 + (
180 + [{ # tool call with extra parameters cannot be generated
181 + "name": "get_current_weather",
182 + "parameters": {
183 + "city": "Vienna",
184 + "extra": "value"
185 + }
186 + }],
187 + False),
188 + (
189 + [{ # tool call where parameters are first cannot be generated
190 + "parameters": {
191 + "city": "Vienna"
192 + },
193 + "name": "get_current_weather"
194 + }],
195 + False),
196 + (
197 + [{ # tool call without all required parameters cannot be generated
198 + "name": "get_forecast",
199 + "parameters": {
200 + "city": "Vienna"
201 + }
202 + }],
203 + False),
204 + ( # tool call with incorrect name/parameters cannot be generated
205 + [{
206 + "name": "get_weather",
207 + "parameters": {
208 + "city": "Vienna",
209 + "days": 7
210 + }
211 + }], False),
212 + ( # tool call with both valid and empty function cannot be generated
213 + [{
214 + "name": "get_current_weather",
215 + "parameters": {
216 + "city": "Vienna"
217 + }
218 + }, {}], False),
219 + ])
220 +def test_guided_json(sample_output, should_match):
221 + _compile_and_check(tools=TypeAdapter(
222 + list[ChatCompletionToolsParam]).validate_python(EXAMPLE_TOOLS),
223 + sample_output=sample_output,
224 + should_match=should_match)
225 +
226 +
227 +def update_parameters_none(
228 + tool: ChatCompletionToolsParam) -> ChatCompletionToolsParam:
229 + tool.function.parameters = None
230 + return tool
231 +
232 +
233 +def update_parameters_empty_dict(
234 + tool: ChatCompletionToolsParam) -> ChatCompletionToolsParam:
235 + tool.function.parameters = {}
236 + return tool
237 +
238 +
239 +@pytest.mark.parametrize(
240 + "sample_output, should_match",
241 + [
242 + (None, False),
243 + ([], False), # empty list cannot be generated
244 + ({}, False), # empty object cannot be generated
245 + ([{}], False), # list with empty object cannot be generated
246 + (
247 + [{ # function without required parameters cannot be generated
248 + "name": "get_current_weather"
249 + }],
250 + False),
251 + (
252 + [{ # function without required parameters cannot be generated
253 + "name": "get_current_weather",
254 + "parameters": None
255 + }],
256 + False),
257 + (
258 + [{ # function with extra parameters cannot be generated
259 + "name": "get_current_weather",
260 + "parameters": {
261 + "extra": "value"
262 + }
263 + }],
264 + False),
265 + (
266 + [{ # only function with empty parameters object is valid
267 + "name": "get_current_weather",
268 + "parameters": {}
269 + }],
270 + True),
271 + ])
272 +@pytest.mark.parametrize(
273 + "update_parameters",
274 + [update_parameters_none, update_parameters_empty_dict])
275 +def test_guided_json_without_parameters(sample_output, should_match,
276 + update_parameters):
277 + updated_tools = [deepcopy(EXAMPLE_TOOLS[0])]
278 + tools = TypeAdapter(
279 + list[ChatCompletionToolsParam]).validate_python(updated_tools)
280 + tools = list(map(update_parameters, tools))
281 + assert all([
282 + tool.function.parameters is None or tool.function.parameters == {}
283 + for tool in tools
284 + ])
285 + _compile_and_check(tools=tools,
286 + sample_output=sample_output,
287 + should_match=should_match)
288 +
289 +
290 +@pytest.mark.parametrize("output", VALID_TOOLS)
291 +@pytest.mark.parametrize("empty_params", [False, True])
292 +@pytest.mark.parametrize("delta_len", [1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
293 +def test_streaming_output_valid(output, empty_params, delta_len):
294 + self = MagicMock()
295 +
296 + output = deepcopy(output)
297 + if empty_params:
298 + output = [{"name": o["name"], "parameters": {}} for o in output]
299 + output_json = json.dumps(output)
300 +
301 + previous_text = ""
302 + function_name_returned = False
303 + messages = []
304 + for i in range(0, len(output_json), delta_len):
305 + delta_text = output_json[i:i + delta_len]
306 + current_text = previous_text + delta_text
307 +
308 + delta_message, function_name_returned = (
309 + OpenAIServingChat.extract_tool_call_required_streaming(
310 + self,
311 + previous_text=previous_text,
312 + current_text=current_text,
313 + delta_text=delta_text,
314 + function_name_returned=function_name_returned))
315 +
316 + if delta_message:
317 + messages.append(delta_message)
318 +
319 + previous_text = current_text
320 +
321 + assert len(messages) > 0
322 + combined_messages = "["
323 + for message in messages:
324 + if message.tool_calls[0].function.name:
325 + if len(combined_messages) > 1:
326 + combined_messages += "},"
327 +
328 + combined_messages += '{"name": "' + \
329 + message.tool_calls[0].function.name + \
330 + '", "parameters": ' + \
331 + message.tool_calls[0].function.arguments
332 + else:
333 + combined_messages += message.tool_calls[0].function.arguments
334 + combined_messages += "}]"
335 + assert json.loads(combined_messages) == output
336 + assert json.dumps(json.loads(combined_messages)) == output_json
vllm/entrypoints/openai/protocol.py
+99 -35
@@ -61,7 +61,7 @@ class OpenAIBaseModel(BaseModel):
61 field_names = set()
62 for field_name, field in cls.model_fields.items():
63 field_names.add(field_name)
64 - if alias := getattr(field, 'alias', None):
64 + if alias := getattr(field, "alias", None):
65 field_names.add(alias)
66 cls.field_names = field_names
67
@@ -70,7 +70,8 @@ class OpenAIBaseModel(BaseModel):
70 logger.warning(
71 "The following fields were present in the request "
72 "but ignored: %s",
73 - data.keys() - field_names)
73 + data.keys() - field_names,
74 + )
75 return result
76
77
@@ -234,8 +235,12 @@ class ChatCompletionRequest(OpenAIBaseModel):
235 temperature: Optional[float] = None
236 top_p: Optional[float] = None
237 tools: Optional[list[ChatCompletionToolsParam]] = None
237 - tool_choice: Optional[Union[Literal["none"], Literal["auto"],
238 - ChatCompletionNamedToolChoiceParam]] = "none"
238 + tool_choice: Optional[Union[
239 + Literal["none"],
240 + Literal["auto"],
241 + Literal["required"],
242 + ChatCompletionNamedToolChoiceParam,
243 + ]] = "none"
244
245 # NOTE this will be ignored by vLLM -- the model determines the behavior
246 parallel_tool_calls: Optional[bool] = False
@@ -340,24 +345,28 @@ class ChatCompletionRequest(OpenAIBaseModel):
345 description=(
346 "If specified, will override the default guided decoding backend "
347 "of the server for this specific request. If set, must be either "
343 - "'outlines' / 'lm-format-enforcer'"))
348 + "'outlines' / 'lm-format-enforcer'"),
349 + )
350 guided_whitespace_pattern: Optional[str] = Field(
351 default=None,
352 description=(
353 "If specified, will override the default whitespace pattern "
348 - "for guided json decoding."))
354 + "for guided json decoding."),
355 + )
356 priority: int = Field(
357 default=0,
358 description=(
359 "The priority of the request (lower means earlier handling; "
360 "default: 0). Any priority other than 0 will raise an error "
354 - "if the served model does not use priority scheduling."))
361 + "if the served model does not use priority scheduling."),
362 + )
363 request_id: str = Field(
364 default_factory=lambda: f"{random_uuid()}",
365 description=(
366 "The request_id related to this request. If the caller does "
367 "not set it, a random_uuid will be generated. This id is used "
360 - "through out the inference process and return in response."))
368 + "through out the inference process and return in response."),
369 + )
370 logits_processors: Optional[LogitsProcessors] = Field(
371 default=None,
372 description=(
@@ -415,13 +424,15 @@ class ChatCompletionRequest(OpenAIBaseModel):
424 ignore_eos=self.ignore_eos,
425 temperature=temperature,
426 length_penalty=self.length_penalty,
418 - include_stop_str_in_output=self.include_stop_str_in_output)
427 + include_stop_str_in_output=self.include_stop_str_in_output,
428 + )
429
430 def to_sampling_params(
421 - self,
422 - default_max_tokens: int,
423 - logits_processor_pattern: Optional[str],
424 - default_sampling_params: Optional[dict] = None) -> SamplingParams:
431 + self,
432 + default_max_tokens: int,
433 + logits_processor_pattern: Optional[str],
434 + default_sampling_params: Optional[dict] = None,
435 + ) -> SamplingParams:
436 # TODO(#9845): remove max_tokens when field is removed from OpenAI API
437 max_tokens = self.max_completion_tokens or self.max_tokens
438
@@ -475,7 +486,8 @@ class ChatCompletionRequest(OpenAIBaseModel):
486 grammar=self.guided_grammar,
487 json_object=guided_json_object,
488 backend=self.guided_decoding_backend,
478 - whitespace_pattern=self.guided_whitespace_pattern)
489 + whitespace_pattern=self.guided_whitespace_pattern,
490 + )
491
492 return SamplingParams.from_optional(
493 n=self.n,
@@ -522,6 +534,41 @@ class ChatCompletionRequest(OpenAIBaseModel):
534 tool = tools[tool_name]
535 return tool.parameters
536
537 + if self.tool_choice == "required":
538 + # Pydantic schema generation cannot be used since the JSON schema
539 + # has to be constructed for a specific instantiation of a tool list
540 + # so that parameters of a function are correctly generated
541 + # based on the chosen function name
542 + def get_tool_schema(tool: ChatCompletionToolsParam) -> dict:
543 + return {
544 + "properties": {
545 + "name": {
546 + "type": "string",
547 + "enum": [tool.function.name]
548 + },
549 + # parameters are always generated as '{}' in the final
550 + # output if they are missing from the request
551 + # (i.e. are None or '{}') so the schema is
552 + # updated to produce an empty object in that case
553 + "parameters": tool.function.parameters
554 + if tool.function.parameters else {
555 + "type": "object",
556 + "properties": {}
557 + }
558 + },
559 + "required": ["name", "parameters"]
560 + }
561 +
562 + json_schema = {
563 + "type": "array",
564 + "minItems": 1,
565 + "items": {
566 + "type": "object",
567 + "anyOf": [get_tool_schema(tool) for tool in self.tools]
568 + }
569 + }
570 + return json_schema
571 +
572 return None
573
574 @model_validator(mode="before")
@@ -572,8 +619,11 @@ class ChatCompletionRequest(OpenAIBaseModel):
619 "You can only use one kind of guided decoding "
620 "('guided_json', 'guided_regex' or 'guided_choice').")
621 # you can only either use guided decoding or tools, not both
575 - if guide_count > 1 and data.get("tool_choice",
576 - "none") not in ("none", "auto"):
622 + if guide_count > 1 and data.get("tool_choice", "none") not in (
623 + "none",
624 + "auto",
625 + "required",
626 + ):
627 raise ValueError(
628 "You can only either use guided decoding or tools, not both.")
629 return data
@@ -602,12 +652,15 @@ class ChatCompletionRequest(OpenAIBaseModel):
652 "When using `tool_choice`, `tools` must be set.")
653
654 # make sure that tool choice is either a named tool
605 - # OR that it's set to "auto"
606 - if data["tool_choice"] != "auto" and not isinstance(
607 - data["tool_choice"], dict):
608 - raise ValueError(
609 - "`tool_choice` must either be a named tool, \"auto\", "
610 - "or \"none\".")
655 + # OR that it's set to "auto" or "required"
656 + if data["tool_choice"] not in [
657 + "auto", "required"
658 + ] and not isinstance(data["tool_choice"], dict):
659 + raise NotImplementedError(
660 + f'Invalid value for `tool_choice`: {data["tool_choice"]}! '\
661 + 'Only named tools, "none", "auto" or "required" '\
662 + 'are supported.'
663 + )
664
665 # ensure that if "tool_choice" is specified as an object,
666 # it matches a valid tool
@@ -722,18 +775,21 @@ class CompletionRequest(OpenAIBaseModel):
775 description=(
776 "If specified, will override the default guided decoding backend "
777 "of the server for this specific request. If set, must be one of "
725 - "'outlines' / 'lm-format-enforcer'"))
778 + "'outlines' / 'lm-format-enforcer'"),
779 + )
780 guided_whitespace_pattern: Optional[str] = Field(
781 default=None,
782 description=(
783 "If specified, will override the default whitespace pattern "
730 - "for guided json decoding."))
784 + "for guided json decoding."),
785 + )
786 priority: int = Field(
787 default=0,
788 description=(
789 "The priority of the request (lower means earlier handling; "
790 "default: 0). Any priority other than 0 will raise an error "
736 - "if the served model does not use priority scheduling."))
791 + "if the served model does not use priority scheduling."),
792 + )
793 logits_processors: Optional[LogitsProcessors] = Field(
794 default=None,
795 description=(
@@ -745,6 +801,7 @@ class CompletionRequest(OpenAIBaseModel):
801 "arguments. For example: {'qualname': "
802 "'my_module.MyLogitsProcessor', 'args': [1, 2], 'kwargs': "
803 "{'param': 'value'}}."))
804 +
805 return_tokens_as_token_ids: Optional[bool] = Field(
806 default=None,
807 description=(
@@ -789,13 +846,15 @@ class CompletionRequest(OpenAIBaseModel):
846 ignore_eos=self.ignore_eos,
847 temperature=temperature,
848 length_penalty=self.length_penalty,
792 - include_stop_str_in_output=self.include_stop_str_in_output)
849 + include_stop_str_in_output=self.include_stop_str_in_output,
850 + )
851
852 def to_sampling_params(
795 - self,
796 - default_max_tokens: int,
797 - logits_processor_pattern: Optional[str],
798 - default_sampling_params: Optional[dict] = None) -> SamplingParams:
853 + self,
854 + default_max_tokens: int,
855 + logits_processor_pattern: Optional[str],
856 + default_sampling_params: Optional[dict] = None,
857 + ) -> SamplingParams:
858 max_tokens = self.max_tokens
859
860 if default_sampling_params is None:
@@ -844,7 +903,8 @@ class CompletionRequest(OpenAIBaseModel):
903 grammar=self.guided_grammar,
904 json_object=guided_json_object,
905 backend=self.guided_decoding_backend,
847 - whitespace_pattern=self.guided_whitespace_pattern)
906 + whitespace_pattern=self.guided_whitespace_pattern,
907 + )
908
909 return SamplingParams.from_optional(
910 n=self.n,
@@ -942,7 +1002,8 @@ class EmbeddingCompletionRequest(OpenAIBaseModel):
1002 description=(
1003 "The priority of the request (lower means earlier handling; "
1004 "default: 0). Any priority other than 0 will raise an error "
945 - "if the served model does not use priority scheduling."))
1005 + "if the served model does not use priority scheduling."),
1006 + )
1007
1008 # doc: end-embedding-extra-params
1009
@@ -995,7 +1056,8 @@ class EmbeddingChatRequest(OpenAIBaseModel):
1056 description=(
1057 "The priority of the request (lower means earlier handling; "
1058 "default: 0). Any priority other than 0 will raise an error "
998 - "if the served model does not use priority scheduling."))
1059 + "if the served model does not use priority scheduling."),
1060 + )
1061 # doc: end-chat-embedding-extra-params
1062
1063 @model_validator(mode="before")
@@ -1034,7 +1096,8 @@ class ScoreRequest(OpenAIBaseModel):
1096 description=(
1097 "The priority of the request (lower means earlier handling; "
1098 "default: 0). Any priority other than 0 will raise an error "
1037 - "if the served model does not use priority scheduling."))
1099 + "if the served model does not use priority scheduling."),
1100 + )
1101
1102 # doc: end-score-extra-params
1103
@@ -1059,7 +1122,8 @@ class RerankRequest(OpenAIBaseModel):
1122 description=(
1123 "The priority of the request (lower means earlier handling; "
1124 "default: 0). Any priority other than 0 will raise an error "
1062 - "if the served model does not use priority scheduling."))
1125 + "if the served model does not use priority scheduling."),
1126 + )
1127
1128 # doc: end-rerank-extra-params
1129
vllm/entrypoints/openai/serving_chat.py
+165 -12
@@ -2,13 +2,16 @@
2
3 import asyncio
4 import json
5 +import re
6 import time
7 from collections.abc import AsyncGenerator, AsyncIterator
8 from collections.abc import Sequence as GenericSequence
9 from typing import Callable, Final, Optional, Union
10
11 import jinja2
12 +import partial_json_parser
13 from fastapi import Request
14 +from pydantic import TypeAdapter
15
16 from vllm.config import ModelConfig
17 from vllm.engine.protocol import EngineClient
@@ -21,8 +24,8 @@ from vllm.entrypoints.openai.protocol import (
24 ChatCompletionRequest, ChatCompletionResponse,
25 ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice,
26 ChatCompletionStreamResponse, ChatMessage, DeltaFunctionCall, DeltaMessage,
24 - DeltaToolCall, ErrorResponse, FunctionCall, PromptTokenUsageInfo,
25 - RequestResponseMetadata, ToolCall, UsageInfo)
27 + DeltaToolCall, ErrorResponse, FunctionCall, FunctionDefinition,
28 + PromptTokenUsageInfo, RequestResponseMetadata, ToolCall, UsageInfo)
29 from vllm.entrypoints.openai.serving_engine import (OpenAIServing,
30 clamp_prompt_logprobs)
31 from vllm.entrypoints.openai.serving_models import OpenAIServingModels
@@ -150,12 +153,6 @@ class OpenAIServingChat(OpenAIServing):
153
154 tool_parser = self.tool_parser
155
153 - # validation for OpenAI tools
154 - # tool_choice = "required" is not supported
155 - if request.tool_choice == "required":
156 - return self.create_error_response(
157 - "tool_choice = \"required\" is not supported!")
158 -
156 if isinstance(tokenizer, MistralTokenizer):
157 # because of issues with pydantic we need to potentially
158 # re-serialize the tool_calls field of the request
@@ -277,6 +274,122 @@ class OpenAIServingChat(OpenAIServing):
274 return self.response_role
275 return request.messages[-1]["role"]
276
277 + @staticmethod
278 + def _bracket_level(s: str, opening='{', closing='}') -> int:
279 + """
280 + Calculate the current level of nested brackets in a given string.
281 + """
282 + level = 0
283 + for char in s:
284 + if char == opening:
285 + level += 1
286 + elif char == closing:
287 + level -= 1
288 + return level
289 +
290 + @staticmethod
291 + def _filter_delta_text(delta_text: str,
292 + previous_text: str) -> tuple[str, bool]:
293 + # remove last '},' of the tool definition stemming from the
294 + # "name"/"parameters" outer object or closing ']' of the tool list
295 + # count occurrences of opening and closing curly braces and
296 + # once level 0 is reached stop outputting text
297 + # if 0 is reached while parsing the delta_text we know the current
298 + # tool will finish in this current iteration
299 + bracket_level = OpenAIServingChat._bracket_level(previous_text)
300 + updated_delta, passed_zero = "", False
301 + for c in delta_text:
302 + if c == '{':
303 + bracket_level += 1
304 + passed_zero = bracket_level == 0
305 + elif c == '}':
306 + bracket_level -= 1
307 + passed_zero = bracket_level == 0
308 +
309 + if bracket_level != 0:
310 + updated_delta += c
311 + else:
312 + # if a comma is reached at level 0 we can stop
313 + if c == ',':
314 + break
315 + return updated_delta, passed_zero
316 +
317 + def extract_tool_call_required_streaming(
318 + self,
319 + previous_text: str,
320 + current_text: str,
321 + delta_text: str,
322 + function_name_returned: bool,
323 + ) -> tuple[Optional[DeltaMessage], bool]:
324 + try:
325 + obj = partial_json_parser.loads(current_text)
326 + except partial_json_parser.core.exceptions.MalformedJSON:
327 + logger.debug('not enough tokens to parse into JSON yet')
328 + obj = None
329 +
330 + # check if the current text is a valid array
331 + # containing a partial tool calling object
332 + # if not repeat
333 + if obj is None or not isinstance(obj, list) or not len(obj) > 0:
334 + function_name_returned = False
335 + delta_message = None
336 + else:
337 + _, finishes_previous_tool = OpenAIServingChat._filter_delta_text(
338 + delta_text, previous_text)
339 + # take the last tool call from the generated list
340 + current_tool_call = obj[-1]
341 +
342 + # once parameters have been generated the name is complete as well
343 + if not finishes_previous_tool and ("name" not in current_tool_call
344 + or "parameters"
345 + not in current_tool_call):
346 + function_name_returned = False
347 + delta_message = None
348 + else:
349 + if not function_name_returned:
350 + # get partly generated arguments from the latest tool call
351 + param_match = re.search(r'.*"parameters":\s*(.*)',
352 + current_text)
353 + arguments = param_match.group(1) if param_match else ""
354 + arguments, _ = OpenAIServingChat._filter_delta_text(
355 + arguments, previous_text)
356 +
357 + # if this iteration finishes a previous tool call but a
358 + # new incomplete tool is already generated, take the
359 + # previous from the list
360 + if (finishes_previous_tool
361 + and "parameters" not in current_tool_call):
362 + current_tool_call = obj[-2]
363 +
364 + function_name_returned = True
365 + delta_message = DeltaMessage(tool_calls=[
366 + DeltaToolCall(function=DeltaFunctionCall(
367 + name=current_tool_call["name"],
368 + arguments=arguments),
369 + index=len(obj) - 1,
370 + type="function")
371 + ])
372 +
373 + else:
374 + delta_text, _ = OpenAIServingChat._filter_delta_text(
375 + delta_text, previous_text)
376 +
377 + if delta_text != "":
378 + delta_message = DeltaMessage(tool_calls=[
379 + DeltaToolCall(
380 + function=DeltaFunctionCall(
381 + # OpenAI API returns None
382 + # instead of name every time
383 + name=None,
384 + arguments=delta_text),
385 + index=len(obj) - 1,
386 + type="function")
387 + ])
388 + else:
389 + delta_message = None
390 +
391 + return delta_message, function_name_returned
392 +
393 async def chat_completion_stream_generator(
394 self,
395 request: ChatCompletionRequest,
@@ -312,6 +425,7 @@ class OpenAIServingChat(OpenAIServing):
425 self._should_stream_with_reasoning_parsing(request))
426
427 all_previous_token_ids: Optional[list[list[int]]]
428 + function_name_returned: Optional[list[bool]] = None
429
430 # Only one of these will be used, thus previous_texts and
431 # all_previous_token_ids will not be used twice in the same iteration.
@@ -322,6 +436,10 @@ class OpenAIServingChat(OpenAIServing):
436 # For reasoning parser and tool call all enabled
437 added_content_delta_arr = [False] * num_choices
438 reasoning_end_arr = [False] * num_choices
439 + elif request.tool_choice == "required":
440 + previous_texts = [""] * num_choices
441 + function_name_returned = [False] * num_choices
442 + all_previous_token_ids = None
443 else:
444 previous_texts, all_previous_token_ids = None, None
445
@@ -521,6 +639,23 @@ class OpenAIServingChat(OpenAIServing):
639 index=i)
640 ])
641
642 + elif request.tool_choice == "required":
643 + assert previous_texts is not None
644 + assert function_name_returned is not None
645 + previous_text = previous_texts[i]
646 + current_text = previous_text + delta_text
647 + fn_name_returned = function_name_returned[i]
648 +
649 + delta_message, function_name_returned[i] = (
650 + self.extract_tool_call_required_streaming(
651 + previous_text=previous_text,
652 + current_text=current_text,
653 + delta_text=delta_text,
654 + function_name_returned=fn_name_returned))
655 +
656 + # update the previous values for the next iteration
657 + previous_texts[i] = current_text
658 +
659 # handle streaming deltas for tools with "auto" tool choice
660 # and reasoning parser
661 elif tool_choice_auto and self.enable_reasoning:
@@ -821,10 +956,10 @@ class OpenAIServingChat(OpenAIServing):
956
957 # if auto tools are not enabled, and a named tool choice using
958 # outlines is not being used
824 - if (not self.enable_auto_tools
825 - or not self.tool_parser) and not isinstance(
826 - request.tool_choice,
827 - ChatCompletionNamedToolChoiceParam):
959 + if (not self.enable_auto_tools or not self.tool_parser) and \
960 + (not isinstance(request.tool_choice,
961 + ChatCompletionNamedToolChoiceParam
962 + ) and request.tool_choice != "required"):
963 message = ChatMessage(role=role,
964 reasoning_content=reasoning_content,
965 content=content)
@@ -845,6 +980,24 @@ class OpenAIServingChat(OpenAIServing):
980 arguments=content))
981 ])
982
983 + elif request.tool_choice and request.tool_choice == "required":
984 + tool_call_class = MistralToolCall if isinstance(
985 + tokenizer, MistralTokenizer) else ToolCall
986 +
987 + # the fields of FunctionDefinition are a superset of the
988 + # tool call outputs and can be used for parsing
989 + tool_calls = TypeAdapter(
990 + list[FunctionDefinition]).validate_json(output.text)
991 + message = ChatMessage(
992 + role=role,
993 + content="",
994 + tool_calls=[
995 + tool_call_class(function=FunctionCall(
996 + name=tool_call.name,
997 + arguments=json.dumps(tool_call.parameters)))
998 + for tool_call in tool_calls
999 + ])
1000 +
1001 # if the request doesn't use tool choice
1002 # OR specifies to not use a tool
1003 elif not request.tool_choice or request.tool_choice == "none":