@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / e02ce498be
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 17:00 UTC

[Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649)

Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com> Co-authored-by: Kyle Mistele <kyle@constellate.ai>

Kyle Mistele committed Sep 4, 2024 at 15:18 UTC e02ce498be2e11a165803d4590588ba98f129797
26 files changed +2588 -83
.buildkite/test-pipeline.yaml
+10
@@ -92,6 +92,7 @@ steps:
92 - pytest -v -s entrypoints/openai
93 - pytest -v -s entrypoints/test_chat_utils.py
94
95 +
96 - label: Distributed Tests (4 GPUs) # 10min
97 working_dir: "/vllm-workspace/tests"
98 num_gpus: 4
@@ -271,6 +272,15 @@ steps:
272 - export VLLM_WORKER_MULTIPROC_METHOD=spawn
273 - bash ./run-tests.sh -c configs/models-small.txt -t 1
274
275 +- label: OpenAI-Compatible Tool Use # 20 min
276 + fast_check: false
277 + mirror_hardwares: [ amd ]
278 + source_file_dependencies:
279 + - vllm/
280 + - tests/tool_use
281 + commands:
282 + - pytest -v -s tool_use
283 +
284 ##### 1 GPU test #####
285 ##### multi gpus test #####
286
docs/source/serving/openai_compatible_server.md
+54 -4
@@ -110,6 +110,14 @@ directory [here](https://github.com/vllm-project/vllm/tree/main/examples/)
110 :func: create_parser_for_docs
111 :prog: vllm serve
112 ```
113 +## Tool Calling in the Chat Completion API
114 +### Named Function Calling
115 +vLLM supports only named function calling in the chat completion API by default. It does so using Outlines, so this is
116 +enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
117 +high-quality one.
118 +
119 +To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
120 +specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
121
122 ### Config file
123
@@ -140,10 +148,52 @@ The order of priorities is `command line > config file values > defaults`.
148 ## Tool calling in the chat completion API
149 vLLM supports only named function calling in the chat completion API. The `tool_choice` options `auto` and `required` are **not yet supported** but on the roadmap.
150
143 -To use a named function you need to define the function in the `tools` parameter and call it in the `tool_choice` parameter.
144 -
145 -It is the callers responsibility to prompt the model with the tool information, vLLM will not automatically manipulate the prompt. **This may change in the future.**
151 +It is the callers responsibility to prompt the model with the tool information, vLLM will not automatically manipulate the prompt.
152
153 vLLM will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
154
149 -Please refer to the OpenAI API reference documentation for more information.
155 +
156 +### Automatic Function Calling
157 +To enable this feature, you should set the following flags:
158 +* `--enable-auto-tool-choice` -- **mandatory** Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it
159 +deems appropriate.
160 +* `--tool-call-parser` -- select the tool parser to use - currently either `hermes` or `mistral`. Additional tool parsers
161 +will continue to be added in the future.
162 +* `--chat-template` -- **optional** for auto tool choice. the path to the chat template which handles `tool`-role messages and `assistant`-role messages
163 +that contain previously generated tool calls. Hermes and Mistral models have tool-compatible chat templates in their
164 +`tokenizer_config.json` files, but you can specify a custom template. This argument can be set to `tool_use` if your model has a tool use-specific chat
165 +template configured in the `tokenizer_config.json`. In this case, it will be used per the `transformers` specification. More on this [here](https://huggingface.co/docs/transformers/en/chat_templating#why-do-some-models-have-multiple-templates)
166 +from HuggingFace; and you can find an example of this in a `tokenizer_config.json` [here](https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/blob/main/tokenizer_config.json)
167 +
168 +If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
169 +
170 +#### Hermes Models
171 +All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported.
172 +* `NousResearch/Hermes-2-Pro-*`
173 +* `NousResearch/Hermes-2-Theta-*`
174 +* `NousResearch/Hermes-3-*`
175 +
176 +
177 +_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality & capabilities due to the merge
178 +step in their creation_.
179 +
180 +Flags: `--tool-call-parser hermes`
181 +
182 +#### Mistral Models
183 +Supported models:
184 +* `mistralai/Mistral-7B-Instruct-v0.3` (confirmed)
185 +* Additional mistral function-calling models are compatible as well.
186 +
187 +Known issues:
188 +1. Mistral 7B struggles to generate parallel tool calls correctly.
189 +2. Mistral's `tokenizer_config.json` chat template requires tool call IDs that are exactly 9 digits, which is
190 +much shorter than what vLLM generates. Since an exception is thrown when this condition
191 +is not met, the following additional chat templates are provided:
192 +
193 +* `examples/tool_chat_template_mistral.jinja` - this is the "official" Mistral chat template, but tweaked so that
194 +it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)
195 +* `examples/tool_chat_template_mistral_parallel.jinja` - this is a "better" version that adds a tool-use system prompt
196 +when tools are provided, that results in much better reliability when working with parallel tool calling.
197 +
198 +
199 +Recommended flags: `--tool-call-parser mistral --chat-template examples/tool_chat_template_mistral_parallel.jinja`
examples/openai_chat_completion_client_with_tools.py new
+162
@@ -0,0 +1,162 @@
1 +"""
2 +Set up this example by starting a vLLM OpenAI-compatible server with tool call
3 +options enabled. For example:
4 +
5 +IMPORTANT: for mistral, you must use one of the provided mistral tool call
6 +templates, or your own - the model default doesn't work for tool calls with vLLM
7 +See the vLLM docs on OpenAI server & tool calling for more details.
8 +
9 +vllm serve --model mistralai/Mistral-7B-Instruct-v0.3 \
10 + --chat-template examples/tool_chat_template_mistral.jinja \
11 + --enable-auto-tool-choice --tool-call-parser mistral
12 +
13 +OR
14 +vllm serve --model NousResearch/Hermes-2-Pro-Llama-3-8B \
15 + --chat-template examples/tool_chat_template_hermes.jinja \
16 + --enable-auto-tool-choice --tool-call-parser hermes
17 +"""
18 +import json
19 +
20 +from openai import OpenAI
21 +
22 +# Modify OpenAI's API key and API base to use vLLM's API server.
23 +openai_api_key = "EMPTY"
24 +openai_api_base = "http://localhost:8000/v1"
25 +
26 +client = OpenAI(
27 + # defaults to os.environ.get("OPENAI_API_KEY")
28 + api_key=openai_api_key,
29 + base_url=openai_api_base,
30 +)
31 +
32 +models = client.models.list()
33 +model = models.data[0].id
34 +
35 +tools = [{
36 + "type": "function",
37 + "function": {
38 + "name": "get_current_weather",
39 + "description": "Get the current weather in a given location",
40 + "parameters": {
41 + "type": "object",
42 + "properties": {
43 + "city": {
44 + "type":
45 + "string",
46 + "description":
47 + "The city to find the weather for, e.g. 'San Francisco'"
48 + },
49 + "state": {
50 + "type":
51 + "string",
52 + "description":
53 + "the two-letter abbreviation for the state that the city is"
54 + " in, e.g. 'CA' which would mean 'California'"
55 + },
56 + "unit": {
57 + "type": "string",
58 + "description": "The unit to fetch the temperature in",
59 + "enum": ["celsius", "fahrenheit"]
60 + }
61 + },
62 + "required": ["city", "state", "unit"]
63 + }
64 + }
65 +}]
66 +
67 +messages = [{
68 + "role": "user",
69 + "content": "Hi! How are you doing today?"
70 +}, {
71 + "role": "assistant",
72 + "content": "I'm doing well! How can I help you?"
73 +}, {
74 + "role":
75 + "user",
76 + "content":
77 + "Can you tell me what the temperate will be in Dallas, in fahrenheit?"
78 +}]
79 +
80 +chat_completion = client.chat.completions.create(messages=messages,
81 + model=model,
82 + tools=tools)
83 +
84 +print("Chat completion results:")
85 +print(chat_completion)
86 +print("\n\n")
87 +
88 +tool_calls_stream = client.chat.completions.create(messages=messages,
89 + model=model,
90 + tools=tools,
91 + stream=True)
92 +
93 +chunks = []
94 +for chunk in tool_calls_stream:
95 + chunks.append(chunk)
96 + if chunk.choices[0].delta.tool_calls:
97 + print(chunk.choices[0].delta.tool_calls[0])
98 + else:
99 + print(chunk.choices[0].delta)
100 +
101 +arguments = []
102 +tool_call_idx = -1
103 +for chunk in chunks:
104 +
105 + if chunk.choices[0].delta.tool_calls:
106 + tool_call = chunk.choices[0].delta.tool_calls[0]
107 +
108 + if tool_call.index != tool_call_idx:
109 + if tool_call_idx >= 0:
110 + print(
111 + f"streamed tool call arguments: {arguments[tool_call_idx]}"
112 + )
113 + tool_call_idx = chunk.choices[0].delta.tool_calls[0].index
114 + arguments.append("")
115 + if tool_call.id:
116 + print(f"streamed tool call id: {tool_call.id} ")
117 +
118 + if tool_call.function:
119 + if tool_call.function.name:
120 + print(f"streamed tool call name: {tool_call.function.name}")
121 +
122 + if tool_call.function.arguments:
123 + arguments[tool_call_idx] += tool_call.function.arguments
124 +
125 +if len(arguments):
126 + print(f"streamed tool call arguments: {arguments[-1]}")
127 +
128 +print("\n\n")
129 +
130 +messages.append({
131 + "role": "assistant",
132 + "tool_calls": chat_completion.choices[0].message.tool_calls
133 +})
134 +
135 +
136 +# Now, simulate a tool call
137 +def get_current_weather(city: str, state: str, unit: 'str'):
138 + return ("The weather in Dallas, Texas is 85 degrees fahrenheit. It is "
139 + "partly cloudly, with highs in the 90's.")
140 +
141 +
142 +available_tools = {"get_current_weather": get_current_weather}
143 +
144 +completion_tool_calls = chat_completion.choices[0].message.tool_calls
145 +for call in completion_tool_calls:
146 + tool_to_call = available_tools[call.function.name]
147 + args = json.loads(call.function.arguments)
148 + result = tool_to_call(**args)
149 + print(result)
150 + messages.append({
151 + "role": "tool",
152 + "content": result,
153 + "tool_call_id": call.id,
154 + "name": call.function.name
155 + })
156 +
157 +chat_completion_2 = client.chat.completions.create(messages=messages,
158 + model=model,
159 + tools=tools,
160 + stream=False)
161 +print("\n\n")
162 +print(chat_completion_2)
examples/tool_chat_template_hermes.jinja new
+129
@@ -0,0 +1,129 @@
1 +{%- macro json_to_python_type(json_spec) %}
2 + {%- set basic_type_map = {
3 + "string": "str",
4 + "number": "float",
5 + "integer": "int",
6 + "boolean": "bool"
7 +} %}
8 +
9 + {%- if basic_type_map[json_spec.type] is defined %}
10 + {{- basic_type_map[json_spec.type] }}
11 + {%- elif json_spec.type == "array" %}
12 + {{- "list[" + json_to_python_type(json_spec|items) + "]" }}
13 + {%- elif json_spec.type == "object" %}
14 + {%- if json_spec.additionalProperties is defined %}
15 + {{- "dict[str, " + json_to_python_type(json_spec.additionalProperties) + ']' }}
16 + {%- else %}
17 + {{- "dict" }}
18 + {%- endif %}
19 + {%- elif json_spec.type is iterable %}
20 + {{- "Union[" }}
21 + {%- for t in json_spec.type %}
22 + {{- json_to_python_type({"type": t}) }}
23 + {%- if not loop.last %}
24 + {{- "," }}
25 + {%- endif %}
26 + {%- endfor %}
27 + {{- "]" }}
28 + {%- else %}
29 + {{- "Any" }}
30 + {%- endif %}
31 +{%- endmacro %}
32 +
33 +
34 +{{- bos_token }}
35 +{{- "<|im_start|>system\nYou are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: <tools> " }}
36 +{%- if tools is iterable and tools | length > 0 %}
37 + {%- for tool in tools %}
38 + {%- if tool.function is defined %}
39 + {%- set tool = tool.function %}
40 + {%- endif %}
41 + {{- '{"type": "function", "function": ' }}
42 + {{- '{"name": "' + tool.name + '", ' }}
43 + {{- '"description": "' + tool.name + '(' }}
44 + {%- for param_name, param_fields in tool.parameters.properties|items %}
45 + {{- param_name + ": " + json_to_python_type(param_fields) }}
46 + {%- if not loop.last %}
47 + {{- ", " }}
48 + {%- endif %}
49 + {%- endfor %}
50 + {{- ")" }}
51 + {%- if tool.return is defined %}
52 + {{- " -> " + json_to_python_type(tool.return) }}
53 + {%- endif %}
54 + {{- " - " + tool.description + "\n\n" }}
55 + {%- for param_name, param_fields in tool.parameters.properties|items %}
56 + {%- if loop.first %}
57 + {{- " Args:\n" }}
58 + {%- endif %}
59 + {{- " " + param_name + "(" + json_to_python_type(param_fields) + "): " + param_fields.description|trim }}
60 + {%- endfor %}
61 + {%- if tool.return is defined and tool.return.description is defined %}
62 + {{- "\n Returns:\n " + tool.return.description }}
63 + {%- endif %}
64 + {{- '"' }}
65 + {{- ', "parameters": ' }}
66 + {%- if tool.parameters.properties | length == 0 %}
67 + {{- "{}" }}
68 + {%- else %}
69 + {{- tool.parameters|tojson }}
70 + {%- endif %}
71 + {{- "}" }}
72 + {%- if not loop.last %}
73 + {{- "\n" }}
74 + {%- endif %}
75 + {%- endfor %}
76 +{%- endif %}
77 +{{- " </tools>" }}
78 +{{- 'Use the following pydantic model json schema for each tool call you will make: {"properties": {"name": {"title": "Name", "type": "string"}, "arguments": {"title": "Arguments", "type": "object"}}, "required": ["name", "arguments"], "title": "FunctionCall", "type": "object"}}
79 +' }}
80 +{{- "For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
81 +" }}
82 +{{- "<tool_call>
83 +" }}
84 +{{- '{"name": <function-name>, "arguments": <args-dict>}
85 +' }}
86 +{{- '</tool_call><|im_end|>' }}
87 +{%- for message in messages %}
88 + {%- if message.role == "user" or message.role == "system" or (message.role == "assistant" and message.tool_calls is not defined) %}
89 + {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
90 + {%- elif message.role == "assistant" and message.tool_calls is defined %}
91 + {{- '<|im_start|>' + message.role }}
92 + {%- for tool_call in message.tool_calls %}
93 + {{- '\n<tool_call>\n' }}
94 + {%- if tool_call.function is defined %}
95 + {%- set tool_call = tool_call.function %}
96 + {%- endif %}
97 + {{- '{' }}
98 + {{- '"name": "' }}
99 + {{- tool_call.name }}
100 + {{- '"}' }}
101 + {{- ', ' }}
102 + {%- if tool_call.arguments is defined %}
103 + {{- '"arguments": ' }}
104 + {{- tool_call.arguments|tojson }}
105 + {%- endif %}
106 + {{- '\n</tool_call>' }}
107 + {%- endfor %}
108 + {{- '<|im_end|>\n' }}
109 + {%- elif message.role == "tool" %}
110 + {%- if loop.previtem and loop.previtem.role != "tool" %}
111 + {{- '<|im_start|>tool\n' }}
112 + {%- endif %}
113 + {{- '<tool_response>\n' }}
114 + {{- message.content }}
115 + {%- if not loop.last %}
116 + {{- '\n</tool_response>\n' }}
117 + {%- else %}
118 + {{- '\n</tool_response>' }}
119 + {%- endif %}
120 + {%- if not loop.last and loop.nextitem.role != "tool" %}
121 + {{- '<|im_end|>' }}
122 + {%- elif loop.last %}
123 + {{- '<|im_end|>' }}
124 + {%- endif %}
125 + {%- endif %}
126 +{%- endfor %}
127 +{%- if add_generation_prompt %}
128 + {{- '<|im_start|>assistant\n' }}
129 +{%- endif %}
examples/tool_chat_template_mistral.jinja new
+86
@@ -0,0 +1,86 @@
1 +{%- if messages[0]["role"] == "system" %}
2 + {%- set system_message = messages[0]["content"] %}
3 + {%- set loop_messages = messages[1:] %}
4 +{%- else %}
5 + {%- set loop_messages = messages %}
6 +{%- endif %}
7 +{%- if not tools is defined %}
8 + {%- set tools = none %}
9 +{%- endif %}
10 +{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
11 +
12 +{%- for message in loop_messages | rejectattr("role", "equalto", "tool") | rejectattr("role", "equalto", "tool_results") | selectattr("tool_calls", "undefined") %}
13 + {%- if (message["role"] == "user") != (loop.index0 % 2 == 0) %}
14 + {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
15 + {%- endif %}
16 +{%- endfor %}
17 +
18 +{{- bos_token }}
19 +{%- for message in loop_messages %}
20 + {%- if message["role"] == "user" %}
21 + {%- if tools is not none and (message == user_messages[-1]) %}
22 + {{- "[AVAILABLE_TOOLS] [" }}
23 + {%- for tool in tools %}
24 + {%- set tool = tool.function %}
25 + {{- '{"type": "function", "function": {' }}
26 + {%- for key, val in tool.items() if key != "return" %}
27 + {%- if val is string %}
28 + {{- '"' + key + '": "' + val + '"' }}
29 + {%- else %}
30 + {{- '"' + key + '": ' + val|tojson }}
31 + {%- endif %}
32 + {%- if not loop.last %}
33 + {{- ", " }}
34 + {%- endif %}
35 + {%- endfor %}
36 + {{- "}}" }}
37 + {%- if not loop.last %}
38 + {{- ", " }}
39 + {%- else %}
40 + {{- "]" }}
41 + {%- endif %}
42 + {%- endfor %}
43 + {{- "[/AVAILABLE_TOOLS]" }}
44 + {%- endif %}
45 + {%- if loop.last and system_message is defined %}
46 + {{- "[INST] " + system_message + "\n\n" + message["content"] + "[/INST]" }}
47 + {%- else %}
48 + {{- "[INST] " + message["content"] + "[/INST]" }}
49 + {%- endif %}
50 + {%- elif message["role"] == "tool_calls" or message.tool_calls is defined %}
51 + {%- if message.tool_calls is defined %}
52 + {%- set tool_calls = message.tool_calls %}
53 + {%- else %}
54 + {%- set tool_calls = message.content %}
55 + {%- endif %}
56 + {{- "[TOOL_CALLS] [" }}
57 + {%- for tool_call in tool_calls %}
58 + {%- set out = tool_call.function|tojson %}
59 + {{- out[:-1] }}
60 + {%- if not tool_call.id is defined or tool_call.id|length < 9 %}
61 + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (1)" + tool_call.id) }}
62 + {%- endif %}
63 + {{- ', "id": "' + tool_call.id[-9:] + '"}' }}
64 + {%- if not loop.last %}
65 + {{- ", " }}
66 + {%- else %}
67 + {{- "]" + eos_token }}
68 + {%- endif %}
69 + {%- endfor %}
70 + {%- elif message["role"] == "assistant" %}
71 + {{- " " + message["content"] + eos_token }}
72 + {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
73 + {%- if message.content is defined and message.content.content is defined %}
74 + {%- set content = message.content.content %}
75 + {%- else %}
76 + {%- set content = message.content %}
77 + {%- endif %}
78 + {{- '[TOOL_RESULTS] {"content": ' + content|string + ", " }}
79 + {%- if not message.tool_call_id is defined or message.tool_call_id|length < 9 %}
80 + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (2)" + message.tool_call_id) }}
81 + {%- endif %}
82 + {{- '"call_id": "' + message.tool_call_id[-9:] + '"}[/TOOL_RESULTS]' }}
83 + {%- else %}
84 + {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
85 + {%- endif %}
86 +{%- endfor %}
examples/tool_chat_template_mistral_parallel.jinja new
+94
@@ -0,0 +1,94 @@
1 +{%- if messages[0]["role"] == "system" %}
2 + {%- set system_message = messages[0]["content"] %}
3 + {%- set loop_messages = messages[1:] %}
4 +{%- else %}
5 + {%- set loop_messages = messages %}
6 +{%- endif %}
7 +{%- if not tools is defined %}
8 + {%- set tools = none %}
9 +{%- endif %}
10 +{%- if tools is defined %}
11 + {%- set parallel_tool_prompt = "You are a helpful assistant that can call tools. If you call one or more tools, format them in a single JSON array or objects, where each object is a tool call, not as separate objects outside of an array or multiple arrays. Use the format [{\"name\": tool call name, \"arguments\": tool call arguments}, additional tool calls] if you call more than one tool. If you call tools, do not attempt to interpret them or otherwise provide a response until you receive a tool call result that you can interpret for the user." %}
12 + {%- if system_message is defined %}
13 + {%- set system_message = parallel_tool_prompt + "\n\n" + system_message %}
14 + {%- else %}
15 + {%- set system_message = parallel_tool_prompt %}
16 + {%- endif %}
17 +{%- endif %}
18 +{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
19 +
20 +{%- for message in loop_messages | rejectattr("role", "equalto", "tool") | rejectattr("role", "equalto", "tool_results") | selectattr("tool_calls", "undefined") %}
21 + {%- if (message["role"] == "user") != (loop.index0 % 2 == 0) %}
22 + {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
23 + {%- endif %}
24 +{%- endfor %}
25 +
26 +{{- bos_token }}
27 +{%- for message in loop_messages %}
28 + {%- if message["role"] == "user" %}
29 + {%- if tools is not none and (message == user_messages[-1]) %}
30 + {{- "[AVAILABLE_TOOLS] [" }}
31 + {%- for tool in tools %}
32 + {%- set tool = tool.function %}
33 + {{- '{"type": "function", "function": {' }}
34 + {%- for key, val in tool.items() if key != "return" %}
35 + {%- if val is string %}
36 + {{- '"' + key + '": "' + val + '"' }}
37 + {%- else %}
38 + {{- '"' + key + '": ' + val|tojson }}
39 + {%- endif %}
40 + {%- if not loop.last %}
41 + {{- ", " }}
42 + {%- endif %}
43 + {%- endfor %}
44 + {{- "}}" }}
45 + {%- if not loop.last %}
46 + {{- ", " }}
47 + {%- else %}
48 + {{- "]" }}
49 + {%- endif %}
50 + {%- endfor %}
51 + {{- "[/AVAILABLE_TOOLS]" }}
52 + {%- endif %}
53 + {%- if loop.last and system_message is defined %}
54 + {{- "[INST] " + system_message + "\n\n" + message["content"] + "[/INST]" }}
55 + {%- else %}
56 + {{- "[INST] " + message["content"] + "[/INST]" }}
57 + {%- endif %}
58 + {%- elif message["role"] == "tool_calls" or message.tool_calls is defined %}
59 + {%- if message.tool_calls is defined %}
60 + {%- set tool_calls = message.tool_calls %}
61 + {%- else %}
62 + {%- set tool_calls = message.content %}
63 + {%- endif %}
64 + {{- "[TOOL_CALLS] [" }}
65 + {%- for tool_call in tool_calls %}
66 + {%- set out = tool_call.function|tojson %}
67 + {{- out[:-1] }}
68 + {%- if not tool_call.id is defined or tool_call.id|length < 9 %}
69 + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (1)" + tool_call.id) }}
70 + {%- endif %}
71 + {{- ', "id": "' + tool_call.id[-9:] + '"}' }}
72 + {%- if not loop.last %}
73 + {{- ", " }}
74 + {%- else %}
75 + {{- "]" + eos_token }}
76 + {%- endif %}
77 + {%- endfor %}
78 + {%- elif message["role"] == "assistant" %}
79 + {{- " " + message["content"] + eos_token }}
80 + {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
81 + {%- if message.content is defined and message.content.content is defined %}
82 + {%- set content = message.content.content %}
83 + {%- else %}
84 + {%- set content = message.content %}
85 + {%- endif %}
86 + {{- '[TOOL_RESULTS] {"content": ' + content|string + ", " }}
87 + {%- if not message.tool_call_id is defined or message.tool_call_id|length < 9 %}
88 + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (2)" + message.tool_call_id) }}
89 + {%- endif %}
90 + {{- '"call_id": "' + message.tool_call_id[-9:] + '"}[/TOOL_RESULTS]' }}
91 + {%- else %}
92 + {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
93 + {%- endif %}
94 +{%- endfor %}
requirements-common.txt
+1
@@ -20,6 +20,7 @@ lm-format-enforcer == 0.10.6
20 outlines >= 0.0.43, < 0.1 # Requires torch >= 2.1.0
21 typing_extensions >= 4.10
22 filelock >= 3.10.4 # filelock starts to support `mode` argument from 3.10.4
23 +partial-json-parser # used for parsing partial JSON outputs
24 pyzmq
25 msgspec
26 gguf == 0.9.1
tests/tool_use/__init__.py
tests/tool_use/conftest.py new
+32
@@ -0,0 +1,32 @@
1 +import pytest
2 +import pytest_asyncio
3 +from huggingface_hub import snapshot_download
4 +
5 +from tests.utils import RemoteOpenAIServer
6 +
7 +from .utils import ARGS, CONFIGS, ServerConfig
8 +
9 +
10 +# for each server config, download the model and return the config
11 +@pytest.fixture(scope="session", params=CONFIGS.keys())
12 +def server_config(request):
13 + config = CONFIGS[request.param]
14 + # download model and tokenizer using transformers
15 + snapshot_download(config["model"])
16 + yield CONFIGS[request.param]
17 +
18 +
19 +# run this for each server config
20 +@pytest.fixture(scope="session")
21 +def server(request, server_config: ServerConfig):
22 + model = server_config["model"]
23 + args_for_model = server_config["arguments"]
24 + with RemoteOpenAIServer(model, ARGS + args_for_model,
25 + max_wait_seconds=480) as server:
26 + yield server
27 +
28 +
29 +@pytest_asyncio.fixture
30 +async def client(server: RemoteOpenAIServer):
31 + async with server.get_async_client() as async_client:
32 + yield async_client
tests/tool_use/test_chat_completions.py new
+143
@@ -0,0 +1,143 @@
1 +from typing import List
2 +
3 +import openai
4 +import pytest
5 +
6 +from .utils import MESSAGES_WITHOUT_TOOLS, WEATHER_TOOL
7 +
8 +
9 +# test: make sure chat completions without tools provided work even when tools
10 +# are enabled. This makes sure tool call chat templates work, AND that the tool
11 +# parser stream processing doesn't change the output of the model.
12 +@pytest.mark.asyncio
13 +async def test_chat_completion_without_tools(client: openai.AsyncOpenAI):
14 + models = await client.models.list()
15 + model_name: str = models.data[0].id
16 + chat_completion = await client.chat.completions.create(
17 + messages=MESSAGES_WITHOUT_TOOLS,
18 + temperature=0,
19 + max_tokens=150,
20 + model=model_name,
21 + logprobs=False)
22 + choice = chat_completion.choices[0]
23 + stop_reason = chat_completion.choices[0].finish_reason
24 + output_text = chat_completion.choices[0].message.content
25 +
26 + # check to make sure we got text
27 + assert output_text is not None
28 + assert len(output_text) > 0
29 + assert stop_reason != "tool_calls"
30 +
31 + # check to make sure no tool calls were returned
32 + assert (choice.message.tool_calls is None
33 + or len(choice.message.tool_calls) == 0)
34 +
35 + # make the same request, streaming
36 + stream = await client.chat.completions.create(
37 + messages=MESSAGES_WITHOUT_TOOLS,
38 + temperature=0,
39 + max_tokens=150,
40 + model=model_name,
41 + logprobs=False,
42 + stream=True,
43 + )
44 + chunks: List[str] = []
45 + finish_reason_count = 0
46 + role_sent: bool = False
47 +
48 + # assemble streamed chunks
49 + async for chunk in stream:
50 + delta = chunk.choices[0].delta
51 +
52 + # make sure the role is assistant
53 + if delta.role:
54 + assert not role_sent
55 + assert delta.role == 'assistant'
56 + role_sent = True
57 +
58 + if delta.content:
59 + chunks.append(delta.content)
60 +
61 + if chunk.choices[0].finish_reason is not None:
62 + finish_reason_count += 1
63 + assert chunk.choices[0].finish_reason == choice.finish_reason
64 +
65 + # make sure tool call chunks aren't being streamed
66 + assert not delta.tool_calls or len(delta.tool_calls) == 0
67 +
68 + # make sure the role was sent, only 1 finish reason was sent, that chunks
69 + # were in fact sent, and that the chunks match non-streaming
70 + assert role_sent
71 + assert finish_reason_count == 1
72 + assert len(chunks)
73 + assert "".join(chunks) == output_text
74 +
75 +
76 +# test: conversation with tools enabled and provided that should not invoke
77 +# tools, to make sure we can still get normal chat completion responses
78 +# and that they won't be parsed as tools
79 +@pytest.mark.asyncio
80 +async def test_chat_completion_with_tools(client: openai.AsyncOpenAI):
81 + models = await client.models.list()
82 + model_name: str = models.data[0].id
83 + chat_completion = await client.chat.completions.create(
84 + messages=MESSAGES_WITHOUT_TOOLS,
85 + temperature=0,
86 + max_tokens=150,
87 + model=model_name,
88 + tools=[WEATHER_TOOL],
89 + logprobs=False)
90 + choice = chat_completion.choices[0]
91 + stop_reason = chat_completion.choices[0].finish_reason
92 + output_text = chat_completion.choices[0].message.content
93 +
94 + # check to make sure we got text
95 + assert output_text is not None
96 + assert stop_reason != 'tool_calls'
97 + assert len(output_text) > 0
98 +
99 + # check to make sure no tool calls were returned
100 + assert (choice.message.tool_calls is None
101 + or len(choice.message.tool_calls) == 0)
102 +
103 + # make the same request, streaming
104 + stream = await client.chat.completions.create(
105 + messages=MESSAGES_WITHOUT_TOOLS,
106 + temperature=0,
107 + max_tokens=150,
108 + model=model_name,
109 + logprobs=False,
110 + tools=[WEATHER_TOOL],
111 + stream=True,
112 + )
113 +
114 + chunks: List[str] = []
115 + finish_reason_count = 0
116 + role_sent: bool = False
117 +
118 + # assemble streamed chunks
119 + async for chunk in stream:
120 + delta = chunk.choices[0].delta
121 +
122 + # make sure the role is assistant
123 + if delta.role:
124 + assert delta.role == 'assistant'
125 + role_sent = True
126 +
127 + if delta.content:
128 + chunks.append(delta.content)
129 +
130 + if chunk.choices[0].finish_reason is not None:
131 + finish_reason_count += 1
132 +
133 + # make sure tool call chunks aren't being streamed
134 + assert not delta.tool_calls or len(delta.tool_calls) == 0
135 +
136 + # make sure the role was sent, only 1 finish reason was sent, that chunks
137 + # were in fact sent, and that the chunks match non-streaming
138 + assert role_sent
139 + assert finish_reason_count == 1
140 + assert chunk.choices[0].finish_reason == stop_reason
141 + assert chunk.choices[0].finish_reason != 'tool_calls'
142 + assert len(chunks)
143 + assert "".join(chunks) == output_text
tests/tool_use/test_parallel_tool_calls.py new
+193
@@ -0,0 +1,193 @@
1 +import json
2 +from typing import Dict, List, Optional
3 +
4 +import openai
5 +import pytest
6 +
7 +from .utils import (MESSAGES_ASKING_FOR_PARALLEL_TOOLS,
8 + MESSAGES_WITH_PARALLEL_TOOL_RESPONSE, SEARCH_TOOL,
9 + WEATHER_TOOL)
10 +
11 +
12 +# test: getting the model to generate parallel tool calls (streaming/not)
13 +# when requested. NOTE that not all models may support this, so some exclusions
14 +# may be added in the future. e.g. llama 3.1 models are not designed to support
15 +# parallel tool calls.
16 +@pytest.mark.asyncio
17 +async def test_parallel_tool_calls(client: openai.AsyncOpenAI):
18 + models = await client.models.list()
19 + model_name: str = models.data[0].id
20 + chat_completion = await client.chat.completions.create(
21 + messages=MESSAGES_ASKING_FOR_PARALLEL_TOOLS,
22 + temperature=0,
23 + max_tokens=200,
24 + model=model_name,
25 + tools=[WEATHER_TOOL, SEARCH_TOOL],
26 + logprobs=False)
27 +
28 + choice = chat_completion.choices[0]
29 + stop_reason = chat_completion.choices[0].finish_reason
30 + non_streamed_tool_calls = chat_completion.choices[0].message.tool_calls
31 +
32 + # make sure 2 tool calls are present
33 + assert choice.message.role == "assistant"
34 + assert non_streamed_tool_calls is not None
35 + assert len(non_streamed_tool_calls) == 2
36 +
37 + for tool_call in non_streamed_tool_calls:
38 + # make sure the tool includes a function and ID
39 + assert tool_call.type == "function"
40 + assert tool_call.function is not None
41 + assert isinstance(tool_call.id, str)
42 + assert len(tool_call.id) > 16
43 +
44 + # make sure the weather tool was called correctly
45 + assert tool_call.function.name == WEATHER_TOOL["function"]["name"]
46 + assert isinstance(tool_call.function.arguments, str)
47 +
48 + parsed_arguments = json.loads(tool_call.function.arguments)
49 + assert isinstance(parsed_arguments, Dict)
50 + assert isinstance(parsed_arguments.get("city"), str)
51 + assert isinstance(parsed_arguments.get("state"), str)
52 +
53 + assert stop_reason == "tool_calls"
54 +
55 + # make the same request, streaming
56 + stream = await client.chat.completions.create(
57 + model=model_name,
58 + messages=MESSAGES_ASKING_FOR_PARALLEL_TOOLS,
59 + temperature=0,
60 + max_tokens=200,
61 + tools=[WEATHER_TOOL, SEARCH_TOOL],
62 + logprobs=False,
63 + stream=True)
64 +
65 + role_name: Optional[str] = None
66 + finish_reason_count: int = 0
67 +
68 + tool_call_names: List[str] = []
69 + tool_call_args: List[str] = []
70 + tool_call_idx: int = -1
71 + tool_call_id_count: int = 0
72 +
73 + async for chunk in stream:
74 +
75 + # if there's a finish reason make sure it's tools
76 + if chunk.choices[0].finish_reason:
77 + finish_reason_count += 1
78 + assert chunk.choices[0].finish_reason == 'tool_calls'
79 +
80 + # if a role is being streamed make sure it wasn't already set to
81 + # something else
82 + if chunk.choices[0].delta.role:
83 + assert not role_name or role_name == 'assistant'
84 + role_name = 'assistant'
85 +
86 + # if a tool call is streamed make sure there's exactly one
87 + # (based on the request parameters
88 + streamed_tool_calls = chunk.choices[0].delta.tool_calls
89 +
90 + if streamed_tool_calls and len(streamed_tool_calls) > 0:
91 +
92 + # make sure only one diff is present - correct even for parallel
93 + assert len(streamed_tool_calls) == 1
94 + tool_call = streamed_tool_calls[0]
95 +
96 + # if a new tool is being called, set up empty arguments
97 + if tool_call.index != tool_call_idx:
98 + tool_call_idx = tool_call.index
99 + tool_call_args.append("")
100 +
101 + # if a tool call ID is streamed, make sure one hasn't been already
102 + if tool_call.id:
103 + tool_call_id_count += 1
104 + assert (isinstance(tool_call.id, str)
105 + and (len(tool_call.id) > 16))
106 +
107 + # if parts of the function start being streamed
108 + if tool_call.function:
109 + # if the function name is defined, set it. it should be streamed
110 + # IN ENTIRETY, exactly one time.
111 + if tool_call.function.name:
112 + assert isinstance(tool_call.function.name, str)
113 + tool_call_names.append(tool_call.function.name)
114 +
115 + if tool_call.function.arguments:
116 + # make sure they're a string and then add them to the list
117 + assert isinstance(tool_call.function.arguments, str)
118 +
119 + tool_call_args[
120 + tool_call.index] += tool_call.function.arguments
121 +
122 + assert finish_reason_count == 1
123 + assert role_name == 'assistant'
124 +
125 + assert (len(non_streamed_tool_calls) == len(tool_call_names) ==
126 + len(tool_call_args))
127 +
128 + for i in range(2):
129 + assert non_streamed_tool_calls[i].function.name == tool_call_names[i]
130 + streamed_args = json.loads(tool_call_args[i])
131 + non_streamed_args = json.loads(
132 + non_streamed_tool_calls[i].function.arguments)
133 + assert streamed_args == non_streamed_args
134 +
135 +
136 +# test: providing parallel tool calls back to the model to get a response
137 +# (streaming/not)
138 +@pytest.mark.asyncio
139 +async def test_parallel_tool_calls_with_results(client: openai.AsyncOpenAI):
140 + models = await client.models.list()
141 + model_name: str = models.data[0].id
142 + chat_completion = await client.chat.completions.create(
143 + messages=MESSAGES_WITH_PARALLEL_TOOL_RESPONSE,
144 + temperature=0,
145 + max_tokens=200,
146 + model=model_name,
147 + tools=[WEATHER_TOOL, SEARCH_TOOL],
148 + logprobs=False)
149 +
150 + choice = chat_completion.choices[0]
151 +
152 + assert choice.finish_reason != "tool_calls" # "stop" or "length"
153 + assert choice.message.role == "assistant"
154 + assert choice.message.tool_calls is None \
155 + or len(choice.message.tool_calls) == 0
156 + assert choice.message.content is not None
157 + assert "98" in choice.message.content # Dallas temp in tool response
158 + assert "78" in choice.message.content # Orlando temp in tool response
159 +
160 + stream = await client.chat.completions.create(
161 + messages=MESSAGES_WITH_PARALLEL_TOOL_RESPONSE,
162 + temperature=0,
163 + max_tokens=200,
164 + model=model_name,
165 + tools=[WEATHER_TOOL, SEARCH_TOOL],
166 + logprobs=False,
167 + stream=True)
168 +
169 + chunks: List[str] = []
170 + finish_reason_count = 0
171 + role_sent: bool = False
172 +
173 + async for chunk in stream:
174 + delta = chunk.choices[0].delta
175 +
176 + if delta.role:
177 + assert not role_sent
178 + assert delta.role == "assistant"
179 + role_sent = True
180 +
181 + if delta.content:
182 + chunks.append(delta.content)
183 +
184 + if chunk.choices[0].finish_reason is not None:
185 + finish_reason_count += 1
186 + assert chunk.choices[0].finish_reason == choice.finish_reason
187 +
188 + assert not delta.tool_calls or len(delta.tool_calls) == 0
189 +
190 + assert role_sent
191 + assert finish_reason_count == 1
192 + assert len(chunks)
193 + assert "".join(chunks) == choice.message.content
tests/tool_use/test_tool_calls.py new
+192
@@ -0,0 +1,192 @@
1 +import json
2 +from typing import Dict, List, Optional
3 +
4 +import openai
5 +import pytest
6 +
7 +from .utils import (MESSAGES_ASKING_FOR_TOOLS, MESSAGES_WITH_TOOL_RESPONSE,
8 + SEARCH_TOOL, WEATHER_TOOL)
9 +
10 +
11 +# test: request a chat completion that should return tool calls, so we know they
12 +# are parsable
13 +@pytest.mark.asyncio
14 +async def test_tool_call_and_choice(client: openai.AsyncOpenAI):
15 + models = await client.models.list()
16 + model_name: str = models.data[0].id
17 + chat_completion = await client.chat.completions.create(
18 + messages=MESSAGES_ASKING_FOR_TOOLS,
19 + temperature=0,
20 + max_tokens=100,
21 + model=model_name,
22 + tools=[WEATHER_TOOL, SEARCH_TOOL],
23 + logprobs=False)
24 +
25 + choice = chat_completion.choices[0]
26 + stop_reason = chat_completion.choices[0].finish_reason
27 + tool_calls = chat_completion.choices[0].message.tool_calls
28 +
29 + # make sure a tool call is present
30 + assert choice.message.role == 'assistant'
31 + assert tool_calls is not None
32 + assert len(tool_calls) == 1
33 + assert tool_calls[0].type == 'function'
34 + assert tool_calls[0].function is not None
35 + assert isinstance(tool_calls[0].id, str)
36 + assert len(tool_calls[0].id) > 16
37 +
38 + # make sure the weather tool was called (classic example) with arguments
39 + assert tool_calls[0].function.name == WEATHER_TOOL["function"]["name"]
40 + assert tool_calls[0].function.arguments is not None
41 + assert isinstance(tool_calls[0].function.arguments, str)
42 +
43 + # make sure the arguments parse properly
44 + parsed_arguments = json.loads(tool_calls[0].function.arguments)
45 + assert isinstance(parsed_arguments, Dict)
46 + assert isinstance(parsed_arguments.get("city"), str)
47 + assert isinstance(parsed_arguments.get("state"), str)
48 + assert parsed_arguments.get("city") == "Dallas"
49 + assert parsed_arguments.get("state") == "TX"
50 +
51 + assert stop_reason == "tool_calls"
52 +
53 + function_name: Optional[str] = None
54 + function_args_str: str = ''
55 + tool_call_id: Optional[str] = None
56 + role_name: Optional[str] = None
57 + finish_reason_count: int = 0
58 +
59 + # make the same request, streaming
60 + stream = await client.chat.completions.create(
61 + model=model_name,
62 + messages=MESSAGES_ASKING_FOR_TOOLS,
63 + temperature=0,
64 + max_tokens=100,
65 + tools=[WEATHER_TOOL, SEARCH_TOOL],
66 + logprobs=False,
67 + stream=True)
68 +
69 + async for chunk in stream:
70 + assert chunk.choices[0].index == 0
71 +
72 + if chunk.choices[0].finish_reason:
73 + finish_reason_count += 1
74 + assert chunk.choices[0].finish_reason == 'tool_calls'
75 +
76 + # if a role is being streamed make sure it wasn't already set to
77 + # something else
78 + if chunk.choices[0].delta.role:
79 + assert not role_name or role_name == 'assistant'
80 + role_name = 'assistant'
81 +
82 + # if a tool call is streamed make sure there's exactly one
83 + # (based on the request parameters
84 + streamed_tool_calls = chunk.choices[0].delta.tool_calls
85 +
86 + if streamed_tool_calls and len(streamed_tool_calls) > 0:
87 + assert len(streamed_tool_calls) == 1
88 + tool_call = streamed_tool_calls[0]
89 +
90 + # if a tool call ID is streamed, make sure one hasn't been already
91 + if tool_call.id:
92 + assert not tool_call_id
93 + tool_call_id = tool_call.id
94 +
95 + # if parts of the function start being streamed
96 + if tool_call.function:
97 + # if the function name is defined, set it. it should be streamed
98 + # IN ENTIRETY, exactly one time.
99 + if tool_call.function.name:
100 + assert function_name is None
101 + assert isinstance(tool_call.function.name, str)
102 + function_name = tool_call.function.name
103 + if tool_call.function.arguments:
104 + assert isinstance(tool_call.function.arguments, str)
105 + function_args_str += tool_call.function.arguments
106 +
107 + assert finish_reason_count == 1
108 + assert role_name == 'assistant'
109 + assert isinstance(tool_call_id, str) and (len(tool_call_id) > 16)
110 +
111 + # validate the name and arguments
112 + assert function_name == WEATHER_TOOL["function"]["name"]
113 + assert function_name == tool_calls[0].function.name
114 + assert isinstance(function_args_str, str)
115 +
116 + # validate arguments
117 + streamed_args = json.loads(function_args_str)
118 + assert isinstance(streamed_args, Dict)
119 + assert isinstance(streamed_args.get("city"), str)
120 + assert isinstance(streamed_args.get("state"), str)
121 + assert streamed_args.get("city") == "Dallas"
122 + assert streamed_args.get("state") == "TX"
123 +
124 + # make sure everything matches non-streaming except for ID
125 + assert function_name == tool_calls[0].function.name
126 + assert choice.message.role == role_name
127 + assert choice.message.tool_calls[0].function.name == function_name
128 +
129 + # compare streamed with non-streamed args Dict-wise, not string-wise
130 + # because character-to-character comparison might not work e.g. the tool
131 + # call parser adding extra spaces or something like that. we care about the
132 + # dicts matching not byte-wise match
133 + assert parsed_arguments == streamed_args
134 +
135 +
136 +# test: providing tools and results back to model to get a non-tool response
137 +# (streaming/not)
138 +@pytest.mark.asyncio
139 +async def test_tool_call_with_results(client: openai.AsyncOpenAI):
140 + models = await client.models.list()
141 + model_name: str = models.data[0].id
142 + chat_completion = await client.chat.completions.create(
143 + messages=MESSAGES_WITH_TOOL_RESPONSE,
144 + temperature=0,
145 + max_tokens=100,
146 + model=model_name,
147 + tools=[WEATHER_TOOL, SEARCH_TOOL],
148 + logprobs=False)
149 +
150 + choice = chat_completion.choices[0]
151 +
152 + assert choice.finish_reason != "tool_calls" # "stop" or "length"
153 + assert choice.message.role == "assistant"
154 + assert choice.message.tool_calls is None \
155 + or len(choice.message.tool_calls) == 0
156 + assert choice.message.content is not None
157 + assert "98" in choice.message.content # the temperature from the response
158 +
159 + stream = await client.chat.completions.create(
160 + messages=MESSAGES_WITH_TOOL_RESPONSE,
161 + temperature=0,
162 + max_tokens=100,
163 + model=model_name,
164 + tools=[WEATHER_TOOL, SEARCH_TOOL],
165 + logprobs=False,
166 + stream=True)
167 +
168 + chunks: List[str] = []
169 + finish_reason_count = 0
170 + role_sent: bool = False
171 +
172 + async for chunk in stream:
173 + delta = chunk.choices[0].delta
174 +
175 + if delta.role:
176 + assert not role_sent
177 + assert delta.role == "assistant"
178 + role_sent = True
179 +
180 + if delta.content:
181 + chunks.append(delta.content)
182 +
183 + if chunk.choices[0].finish_reason is not None:
184 + finish_reason_count += 1
185 + assert chunk.choices[0].finish_reason == choice.finish_reason
186 +
187 + assert not delta.tool_calls or len(delta.tool_calls) == 0
188 +
189 + assert role_sent
190 + assert finish_reason_count == 1
191 + assert len(chunks)
192 + assert "".join(chunks) == choice.message.content
tests/tool_use/utils.py new
+215
@@ -0,0 +1,215 @@
1 +from typing import Dict, List
2 +
3 +from openai.types.chat import (ChatCompletionMessageParam,
4 + ChatCompletionToolParam)
5 +from typing_extensions import TypedDict
6 +
7 +from tests.utils import VLLM_PATH
8 +
9 +
10 +class ServerConfig(TypedDict):
11 + model: str
12 + arguments: List[str]
13 +
14 +
15 +# universal args for all models go here. also good if you need to test locally
16 +# and change type or KV cache quantization or something.
17 +ARGS: List[str] = ["--enable-auto-tool-choice", "--max-model-len", "8096"]
18 +
19 +CONFIGS: Dict[str, ServerConfig] = {
20 + "hermes": {
21 + "model":
22 + "NousResearch/Hermes-2-Pro-Llama-3-8B",
23 + "arguments": [
24 + "--tool-call-parser", "hermes", "--chat-template",
25 + str(VLLM_PATH / "examples/tool_chat_template_hermes.jinja")
26 + ]
27 + },
28 + "mistral": {
29 + "model":
30 + "mistralai/Mistral-7B-Instruct-v0.3",
31 + "arguments": [
32 + "--tool-call-parser", "mistral", "--chat-template",
33 + str(VLLM_PATH / "examples/tool_chat_template_mistral.jinja"),
34 + "--ignore-patterns=\"consolidated.safetensors\""
35 + ]
36 + }
37 +}
38 +
39 +WEATHER_TOOL: ChatCompletionToolParam = {
40 + "type": "function",
41 + "function": {
42 + "name": "get_current_weather",
43 + "description": "Get the current weather in a given location",
44 + "parameters": {
45 + "type": "object",
46 + "properties": {
47 + "city": {
48 + "type":
49 + "string",
50 + "description":
51 + "The city to find the weather for, "
52 + "e.g. 'San Francisco'"
53 + },
54 + "state": {
55 + "type":
56 + "string",
57 + "description":
58 + "the two-letter abbreviation for the state "
59 + "that the city is in, e.g. 'CA' which would "
60 + "mean 'California'"
61 + },
62 + "unit": {
63 + "type": "string",
64 + "description": "The unit to fetch the temperature in",
65 + "enum": ["celsius", "fahrenheit"]
66 + }
67 + }
68 + }
69 + }
70 +}
71 +
72 +SEARCH_TOOL: ChatCompletionToolParam = {
73 + "type": "function",
74 + "function": {
75 + "name":
76 + "web_search",
77 + "description":
78 + "Search the internet and get a summary of the top "
79 + "10 webpages. Should only be used if you don't know "
80 + "the answer to a user query, and the results are likely"
81 + "to be able to be found with a web search",
82 + "parameters": {
83 + "type": "object",
84 + "properties": {
85 + "search_term": {
86 + "type":
87 + "string",
88 + "description":
89 + "The term to use in the search. This should"
90 + "ideally be keywords to search for, not a"
91 + "natural-language question"
92 + }
93 + },
94 + "required": ["search_term"]
95 + }
96 + }
97 +}
98 +
99 +MESSAGES_WITHOUT_TOOLS: List[ChatCompletionMessageParam] = [{
100 + "role":
101 + "system",
102 + "content":
103 + "You are a helpful assistant with access to tools. If a tool"
104 + " that you have would be helpful to answer a user query, "
105 + "call the tool. Otherwise, answer the user's query directly "
106 + "without calling a tool. DO NOT CALL A TOOL THAT IS IRRELEVANT "
107 + "to the user's question - just respond to it normally."
108 +}, {
109 + "role":
110 + "user",
111 + "content":
112 + "Hi! How are you?"
113 +}, {
114 + "role":
115 + "assistant",
116 + "content":
117 + "I'm doing great! How can I assist you?"
118 +}, {
119 + "role":
120 + "user",
121 + "content":
122 + "Can you tell me a joke please?"
123 +}]
124 +
125 +MESSAGES_ASKING_FOR_TOOLS: List[ChatCompletionMessageParam] = [{
126 + "role":
127 + "user",
128 + "content":
129 + "What is the weather in Dallas, Texas in Fahrenheit?"
130 +}]
131 +
132 +MESSAGES_WITH_TOOL_RESPONSE: List[ChatCompletionMessageParam] = [{
133 + "role":
134 + "user",
135 + "content":
136 + "What is the weather in Dallas, Texas in Fahrenheit?"
137 +}, {
138 + "role":
139 + "assistant",
140 + "tool_calls": [{
141 + "id": "chatcmpl-tool-03e6481b146e408e9523d9c956696295",
142 + "type": "function",
143 + "function": {
144 + "name":
145 + WEATHER_TOOL["function"]["name"],
146 + "arguments":
147 + '{"city": "Dallas", "state": "TX", '
148 + '"unit": "fahrenheit"}'
149 + }
150 + }]
151 +}, {
152 + "role":
153 + "tool",
154 + "tool_call_id":
155 + "chatcmpl-tool-03e6481b146e408e9523d9c956696295",
156 + "content":
157 + "The weather in Dallas is 98 degrees fahrenheit, with partly"
158 + "cloudy skies and a low chance of rain."
159 +}]
160 +
161 +MESSAGES_ASKING_FOR_PARALLEL_TOOLS: List[ChatCompletionMessageParam] = [{
162 + "role":
163 + "user",
164 + "content":
165 + "What is the weather in Dallas, Texas and Orlando, Florida in "
166 + "Fahrenheit?"
167 +}]
168 +
169 +MESSAGES_WITH_PARALLEL_TOOL_RESPONSE: List[ChatCompletionMessageParam] = [{
170 + "role":
171 + "user",
172 + "content":
173 + "What is the weather in Dallas, Texas and Orlando, Florida in "
174 + "Fahrenheit?"
175 +}, {
176 + "role":
177 + "assistant",
178 + "tool_calls": [{
179 + "id": "chatcmpl-tool-03e6481b146e408e9523d9c956696295",
180 + "type": "function",
181 + "function": {
182 + "name":
183 + WEATHER_TOOL["function"]["name"],
184 + "arguments":
185 + '{"city": "Dallas", "state": "TX", '
186 + '"unit": "fahrenheit"}'
187 + }
188 + }, {
189 + "id": "chatcmpl-tool-d027061e1bd21cda48bee7da829c1f5b",
190 + "type": "function",
191 + "function": {
192 + "name":
193 + WEATHER_TOOL["function"]["name"],
194 + "arguments":
195 + '{"city": "Orlando", "state": "Fl", '
196 + '"unit": "fahrenheit"}'
197 + }
198 + }]
199 +}, {
200 + "role":
201 + "tool",
202 + "tool_call_id":
203 + "chatcmpl-tool-03e6481b146e408e9523d9c956696295",
204 + "content":
205 + "The weather in Dallas TX is 98 degrees fahrenheit with mostly "
206 + "cloudy skies and a chance of rain in the evening."
207 +}, {
208 + "role":
209 + "tool",
210 + "tool_call_id":
211 + "chatcmpl-tool-d027061e1bd21cda48bee7da829c1f5b",
212 + "content":
213 + "The weather in Orlando FL is 78 degrees fahrenheit with clear"
214 + "skies."
215 +}]
vllm/entrypoints/chat_utils.py
+82 -19
@@ -1,23 +1,28 @@
1 import asyncio
2 import codecs
3 +import json
4 from abc import ABC, abstractmethod
5 from collections import defaultdict
5 -from functools import lru_cache
6 +from functools import lru_cache, partial
7 from pathlib import Path
8 from typing import (Any, Awaitable, Dict, Generic, Iterable, List, Literal,
8 - Mapping, Optional, Tuple, TypeVar, Union)
9 + Mapping, Optional, Tuple, TypeVar, Union, cast)
10
11 # yapf conflicts with isort for this block
12 # yapf: disable
12 -from openai.types.chat import ChatCompletionContentPartImageParam
13 +from openai.types.chat import (ChatCompletionAssistantMessageParam,
14 + ChatCompletionContentPartImageParam)
15 from openai.types.chat import (
16 ChatCompletionContentPartParam as OpenAIChatCompletionContentPartParam)
15 -from openai.types.chat import ChatCompletionContentPartTextParam
17 +from openai.types.chat import (ChatCompletionContentPartRefusalParam,
18 + ChatCompletionContentPartTextParam)
19 from openai.types.chat import (
20 ChatCompletionMessageParam as OpenAIChatCompletionMessageParam)
21 +from openai.types.chat import (ChatCompletionMessageToolCallParam,
22 + ChatCompletionToolMessageParam)
23 # yapf: enable
24 # pydantic needs the TypedDict from typing_extensions
20 -from pydantic import ConfigDict, TypeAdapter
25 +from pydantic import ConfigDict
26 from typing_extensions import Required, TypeAlias, TypedDict
27
28 from vllm.config import ModelConfig
@@ -54,7 +59,8 @@ class CustomChatCompletionContentPartParam(TypedDict, total=False):
59
60 ChatCompletionContentPartParam: TypeAlias = Union[
61 OpenAIChatCompletionContentPartParam, ChatCompletionContentPartAudioParam,
57 - CustomChatCompletionContentPartParam, ]
62 + ChatCompletionContentPartRefusalParam,
63 + CustomChatCompletionContentPartParam]
64
65
66 class CustomChatCompletionMessageParam(TypedDict, total=False):
@@ -72,15 +78,33 @@ class CustomChatCompletionMessageParam(TypedDict, total=False):
78 same role.
79 """
80
81 + tool_call_id: Optional[str]
82 + """Tool call that this message is responding to."""
83 +
84 + tool_calls: Optional[Iterable[ChatCompletionMessageToolCallParam]]
85 + """The tool calls generated by the model, such as function calls."""
86 +
87
88 ChatCompletionMessageParam = Union[OpenAIChatCompletionMessageParam,
89 CustomChatCompletionMessageParam]
90
91
92 # TODO: Make fields ReadOnly once mypy supports it
81 -class ConversationMessage(TypedDict):
82 - role: str
83 - content: str
93 +class ConversationMessage(TypedDict, total=False):
94 + role: Required[str]
95 + """The role of the message's author."""
96 +
97 + content: Optional[str]
98 + """The contents of the message"""
99 +
100 + tool_call_id: Optional[str]
101 + """Tool call that this message is responding to."""
102 +
103 + name: Optional[str]
104 + """The name of the function to call"""
105 +
106 + tool_calls: Optional[Iterable[ChatCompletionMessageToolCallParam]]
107 + """The tool calls generated by the model, such as function calls."""
108
109
110 ModalityStr = Literal["image", "audio"]
@@ -319,9 +343,11 @@ def _get_full_multimodal_text_prompt(placeholder_counts: Dict[str, int],
343 return "\n".join(missing_placeholders + [text_prompt])
344
345
322 -_TextParser = TypeAdapter(ChatCompletionContentPartTextParam)
323 -_ImageParser = TypeAdapter(ChatCompletionContentPartImageParam)
324 -_AudioParser = TypeAdapter(ChatCompletionContentPartAudioParam)
346 +# No need to validate using Pydantic again
347 +_TextParser = partial(cast, ChatCompletionContentPartTextParam)
348 +_ImageParser = partial(cast, ChatCompletionContentPartImageParam)
349 +_AudioParser = partial(cast, ChatCompletionContentPartAudioParam)
350 +_RefusalParser = partial(cast, ChatCompletionContentPartRefusalParam)
351
352
353 def _parse_chat_message_content_parts(
@@ -336,10 +362,10 @@ def _parse_chat_message_content_parts(
362 for part in parts:
363 part_type = part["type"]
364 if part_type == "text":
339 - text = _TextParser.validate_python(part)["text"]
365 + text = _TextParser(part)["text"]
366 texts.append(text)
367 elif part_type == "image_url":
342 - image_url = _ImageParser.validate_python(part)["image_url"]
368 + image_url = _ImageParser(part)["image_url"]
369
370 if image_url.get("detail", "auto") != "auto":
371 logger.warning(
@@ -348,7 +374,7 @@ def _parse_chat_message_content_parts(
374
375 mm_parser.parse_image(image_url["url"])
376 elif part_type == "audio_url":
351 - audio_url = _AudioParser.validate_python(part)["audio_url"]
377 + audio_url = _AudioParser(part)["audio_url"]
378
379 mm_parser.parse_audio(audio_url["url"])
380 else:
@@ -363,6 +389,11 @@ def _parse_chat_message_content_parts(
389 return [ConversationMessage(role=role, content=text_prompt)]
390
391
392 +# No need to validate using Pydantic again
393 +_AssistantParser = partial(cast, ChatCompletionAssistantMessageParam)
394 +_ToolParser = partial(cast, ChatCompletionToolMessageParam)
395 +
396 +
397 def _parse_chat_message_content(
398 message: ChatCompletionMessageParam,
399 mm_tracker: BaseMultiModalItemTracker,
@@ -371,16 +402,34 @@ def _parse_chat_message_content(
402 content = message.get("content")
403
404 if content is None:
374 - return []
375 - if isinstance(content, str):
376 - return [ConversationMessage(role=role, content=content)]
405 + content = []
406 + elif isinstance(content, str):
407 + content = [
408 + ChatCompletionContentPartTextParam(type="text", text=content)
409 + ]
410
378 - return _parse_chat_message_content_parts(
411 + result = _parse_chat_message_content_parts(
412 role,
413 content, # type: ignore
414 mm_tracker,
415 )
416
417 + for result_msg in result:
418 + if role == 'assistant':
419 + parsed_msg = _AssistantParser(message)
420 +
421 + if "tool_calls" in parsed_msg:
422 + result_msg["tool_calls"] = list(parsed_msg["tool_calls"])
423 + elif role == "tool":
424 + parsed_msg = _ToolParser(message)
425 + if "tool_call_id" in parsed_msg:
426 + result_msg["tool_call_id"] = parsed_msg["tool_call_id"]
427 +
428 + if "name" in message and isinstance(message["name"], str):
429 + result_msg["name"] = message["name"]
430 +
431 + return result
432 +
433
434 def parse_chat_messages(
435 messages: List[ChatCompletionMessageParam],
@@ -428,6 +477,20 @@ def apply_chat_template(
477 "allowed, so you must provide a chat template if the tokenizer "
478 "does not define one.")
479
480 + # per the Transformers docs & maintainers, tool call arguments in
481 + # assistant-role messages with tool_calls need to be dicts not JSON str -
482 + # this is how tool-use chat templates will expect them moving forwards
483 + # so, for messages that have tool_calls, parse the string (which we get
484 + # from openAI format) to dict
485 + for message in conversation:
486 + if (message["role"] == "assistant" and "tool_calls" in message
487 + and isinstance(message["tool_calls"], list)):
488 +
489 + for i in range(len(message["tool_calls"])):
490 + args: str = message["tool_calls"][i]["function"]["arguments"]
491 + parsed_args: Dict = json.loads(args)
492 + message["tool_calls"][i]["function"]["arguments"] = parsed_args
493 +
494 prompt = tokenizer.apply_chat_template(
495 conversation=conversation,
496 chat_template=chat_template,
vllm/entrypoints/openai/api_server.py
+6 -2
@@ -233,7 +233,7 @@ def mount_metrics(app: FastAPI):
233 metrics_route = Mount("/metrics", make_asgi_app())
234
235 # Workaround for 307 Redirect for /metrics
236 - metrics_route.path_regex = re.compile('^/metrics(?P<path>.*)$')
236 + metrics_route.path_regex = re.compile("^/metrics(?P<path>.*)$")
237 app.routes.append(metrics_route)
238
239
@@ -283,11 +283,14 @@ async def show_version():
283 @router.post("/v1/chat/completions")
284 async def create_chat_completion(request: ChatCompletionRequest,
285 raw_request: Request):
286 +
287 generator = await openai_serving_chat.create_chat_completion(
288 request, raw_request)
289 +
290 if isinstance(generator, ErrorResponse):
291 return JSONResponse(content=generator.model_dump(),
292 status_code=generator.code)
293 +
294 elif isinstance(generator, ChatCompletionResponse):
295 return JSONResponse(content=generator.model_dump())
296
@@ -422,7 +425,8 @@ async def init_app(
425 request_logger=request_logger,
426 chat_template=args.chat_template,
427 return_tokens_as_token_ids=args.return_tokens_as_token_ids,
425 - )
428 + enable_auto_tools=args.enable_auto_tool_choice,
429 + tool_parser=args.tool_call_parser)
430 openai_serving_completion = OpenAIServingCompletion(
431 async_engine_client,
432 model_config,
vllm/entrypoints/openai/cli_args.py
+18
@@ -163,6 +163,24 @@ def make_arg_parser(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
163 help="If specified, will run the OpenAI frontend server in the same "
164 "process as the model serving engine.")
165
166 + parser.add_argument(
167 + "--enable-auto-tool-choice",
168 + action="store_true",
169 + default=False,
170 + help=
171 + "Enable auto tool choice for supported models. Use --tool-call-parser"
172 + "to specify which parser to use")
173 +
174 + parser.add_argument(
175 + "--tool-call-parser",
176 + type=str,
177 + choices=["mistral", "hermes"],
178 + default=None,
179 + help=
180 + "Select the tool call parser depending on the model that you're using."
181 + " This is used to parse the model-generated tool call into OpenAI API "
182 + "format. Required for --enable-auto-tool-choice.")
183 +
184 parser = AsyncEngineArgs.add_cli_args(parser)
185
186 parser.add_argument('--max-log-len',
vllm/entrypoints/openai/protocol.py
+113 -12
@@ -5,8 +5,9 @@ from argparse import Namespace
5 from typing import Any, Dict, List, Literal, Optional, Union
6
7 import torch
8 +from openai.types.chat import ChatCompletionContentPartParam
9 from pydantic import BaseModel, ConfigDict, Field, model_validator
9 -from typing_extensions import Annotated
10 +from typing_extensions import Annotated, Required, TypedDict
11
12 from vllm.entrypoints.chat_utils import ChatCompletionMessageParam
13 from vllm.entrypoints.openai.logits_processors import get_logits_processors
@@ -35,6 +36,26 @@ assert _LONG_INFO.min == _MOCK_LONG_INFO.min
36 assert _LONG_INFO.max == _MOCK_LONG_INFO.max
37
38
39 +class CustomChatCompletionMessageParam(TypedDict, total=False):
40 + """Enables custom roles in the Chat Completion API."""
41 + role: Required[str]
42 + """The role of the message's author."""
43 +
44 + content: Union[str, List[ChatCompletionContentPartParam]]
45 + """The contents of the message."""
46 +
47 + name: str
48 + """An optional name for the participant.
49 +
50 + Provides the model information to differentiate between participants of the
51 + same role.
52 + """
53 +
54 + tool_call_id: Optional[str]
55 +
56 + tool_calls: Optional[List[dict]]
57 +
58 +
59 class OpenAIBaseModel(BaseModel):
60 # OpenAI API does not allow extra fields
61 model_config = ConfigDict(extra="forbid")
@@ -145,8 +166,11 @@ class ChatCompletionRequest(OpenAIBaseModel):
166 temperature: Optional[float] = 0.7
167 top_p: Optional[float] = 1.0
168 tools: Optional[List[ChatCompletionToolsParam]] = None
148 - tool_choice: Optional[Union[Literal["none"],
169 + tool_choice: Optional[Union[Literal["none"], Literal["auto"],
170 ChatCompletionNamedToolChoiceParam]] = "none"
171 +
172 + # NOTE this will be ignored by VLLM -- the model determines the behavior
173 + parallel_tool_calls: Optional[bool] = False
174 user: Optional[str] = None
175
176 # doc: begin-chat-completion-sampling-params
@@ -328,6 +352,9 @@ class ChatCompletionRequest(OpenAIBaseModel):
352 @model_validator(mode="before")
353 @classmethod
354 def check_guided_decoding_count(cls, data):
355 + if isinstance(data, ValueError):
356 + raise data
357 +
358 guide_count = sum([
359 "guided_json" in data and data["guided_json"] is not None,
360 "guided_regex" in data and data["guided_regex"] is not None,
@@ -339,21 +366,61 @@ class ChatCompletionRequest(OpenAIBaseModel):
366 "You can only use one kind of guided decoding "
367 "('guided_json', 'guided_regex' or 'guided_choice').")
368 # you can only either use guided decoding or tools, not both
342 - if guide_count > 1 and "tool_choice" in data and data[
343 - "tool_choice"] != "none":
369 + if guide_count > 1 and data.get("tool_choice",
370 + "none") not in ("none", "auto"):
371 raise ValueError(
372 "You can only either use guided decoding or tools, not both.")
373 return data
374
375 @model_validator(mode="before")
376 @classmethod
350 - def check_tool_choice(cls, data):
351 - if "tool_choice" in data and data["tool_choice"] != "none":
352 - if not isinstance(data["tool_choice"], dict):
353 - raise ValueError("Currently only named tools are supported.")
377 + def check_tool_usage(cls, data):
378 +
379 + # if "tool_choice" is not specified but tools are provided,
380 + # default to "auto" tool_choice
381 + if "tool_choice" not in data and "tools" in data:
382 + data["tool_choice"] = "auto"
383 +
384 + # if "tool_choice" is specified -- validation
385 + if "tool_choice" in data:
386 +
387 + # ensure that if "tool choice" is specified, tools are present
388 if "tools" not in data or data["tools"] is None:
389 raise ValueError(
390 "When using `tool_choice`, `tools` must be set.")
391 +
392 + # make sure that tool choice is either a named tool
393 + # OR that it's set to "auto"
394 + if data["tool_choice"] != "auto" and not isinstance(
395 + data["tool_choice"], dict):
396 + raise ValueError(
397 + "`tool_choice` must either be a named tool or \"auto\". "
398 + "`tool_choice=\"none\" is not supported.")
399 +
400 + # ensure that if "tool_choice" is specified as an object,
401 + # it matches a valid tool
402 + if isinstance(data["tool_choice"], dict):
403 + valid_tool = False
404 + specified_function = data["tool_choice"]["function"]
405 + if not specified_function:
406 + raise ValueError(
407 + "Incorrectly formatted `tool_choice`. Should be like "
408 + "`{\"type\": \"function\","
409 + " \"function\": {\"name\": \"my_function\"}}`")
410 + specified_function_name = specified_function["name"]
411 + if not specified_function_name:
412 + raise ValueError(
413 + "Incorrectly formatted `tool_choice`. Should be like "
414 + "`{\"type\": \"function\", "
415 + "\"function\": {\"name\": \"my_function\"}}`")
416 + for tool in data["tools"]:
417 + if tool["function"]["name"] == specified_function_name:
418 + valid_tool = True
419 + break
420 + if not valid_tool:
421 + raise ValueError(
422 + "The tool specified in `tool_choice` does not match any"
423 + " of the specified `tools`")
424 return data
425
426
@@ -413,7 +480,7 @@ class CompletionRequest(OpenAIBaseModel):
480 )
481 guided_json: Optional[Union[str, dict, BaseModel]] = Field(
482 default=None,
416 - description=("If specified, the output will follow the JSON schema."),
483 + description="If specified, the output will follow the JSON schema.",
484 )
485 guided_regex: Optional[str] = Field(
486 default=None,
@@ -633,9 +700,41 @@ class ToolCall(OpenAIBaseModel):
700 function: FunctionCall
701
702
703 +class DeltaFunctionCall(BaseModel):
704 + name: Optional[str] = None
705 + arguments: Optional[str] = None
706 +
707 +
708 +# a tool call delta where everything is optional
709 +class DeltaToolCall(OpenAIBaseModel):
710 + id: str = Field(default_factory=lambda: f"chatcmpl-tool-{random_uuid()}")
711 + type: Literal["function"] = "function"
712 + index: int
713 + function: Optional[DeltaFunctionCall] = None
714 +
715 +
716 +# the initial delta that gets sent once a new tool call is started;
717 +class InitialDeltaToolCall(DeltaToolCall):
718 + id: str = Field(default_factory=lambda: f"chatcmpl-tool-{random_uuid()}")
719 + type: Literal["function"] = "function"
720 + index: int
721 +
722 +
723 +class ExtractedToolCallInformation(BaseModel):
724 + # indicate if tools were called
725 + tools_called: bool
726 +
727 + # extracted tool calls
728 + tool_calls: List[ToolCall]
729 +
730 + # content - per OpenAI spec, content AND tool calls can be returned rarely
731 + # But some models will do this intentionally
732 + content: Optional[str] = None
733 +
734 +
735 class ChatMessage(OpenAIBaseModel):
736 role: str
638 - content: str
737 + content: Optional[str] = None
738 tool_calls: List[ToolCall] = Field(default_factory=list)
739
740
@@ -657,7 +756,9 @@ class ChatCompletionResponseChoice(OpenAIBaseModel):
756 index: int
757 message: ChatMessage
758 logprobs: Optional[ChatCompletionLogProbs] = None
660 - finish_reason: Optional[str] = None
759 + # per OpenAI spec this is the default
760 + finish_reason: Optional[str] = "stop"
761 + # not part of the OpenAI spec but included in vLLM for legacy reasons
762 stop_reason: Optional[Union[int, str]] = None
763
764
@@ -674,7 +775,7 @@ class ChatCompletionResponse(OpenAIBaseModel):
775 class DeltaMessage(OpenAIBaseModel):
776 role: Optional[str] = None
777 content: Optional[str] = None
677 - tool_calls: List[ToolCall] = Field(default_factory=list)
778 + tool_calls: List[DeltaToolCall] = Field(default_factory=list)
779
780
781 class ChatCompletionResponseStreamChoice(OpenAIBaseModel):
vllm/entrypoints/openai/serving_chat.py
+240 -35
@@ -1,6 +1,8 @@
1 import asyncio
2 +import json
3 import time
3 -from typing import AsyncGenerator, AsyncIterator, Dict, Final, List, Optional
4 +from typing import (AsyncGenerator, AsyncIterator, Callable, Dict, Final, List,
5 + Optional)
6 from typing import Sequence as GenericSequence
7 from typing import Union
8
@@ -18,15 +20,18 @@ from vllm.entrypoints.openai.protocol import (
20 ChatCompletionLogProbsContent, ChatCompletionNamedToolChoiceParam,
21 ChatCompletionRequest, ChatCompletionResponse,
22 ChatCompletionResponseChoice, ChatCompletionResponseStreamChoice,
21 - ChatCompletionStreamResponse, ChatMessage, DeltaMessage, ErrorResponse,
22 - FunctionCall, ToolCall, UsageInfo)
23 + ChatCompletionStreamResponse, ChatMessage, DeltaFunctionCall, DeltaMessage,
24 + DeltaToolCall, ErrorResponse, FunctionCall, ToolCall, UsageInfo)
25 from vllm.entrypoints.openai.serving_engine import (LoRAModulePath,
26 OpenAIServing,
27 PromptAdapterPath,
28 TextTokensPrompt)
29 +from vllm.entrypoints.openai.tool_parsers import (Hermes2ProToolParser,
30 + MistralToolParser,
31 + ToolParser)
32 from vllm.inputs import TokensPrompt
33 from vllm.logger import init_logger
29 -from vllm.outputs import RequestOutput
34 +from vllm.outputs import CompletionOutput, RequestOutput
35 from vllm.sequence import Logprob
36 from vllm.tracing import (contains_trace_headers, extract_trace_headers,
37 log_tracing_disabled_warning)
@@ -38,19 +43,19 @@ logger = init_logger(__name__)
43
44 class OpenAIServingChat(OpenAIServing):
45
41 - def __init__(
42 - self,
43 - async_engine_client: AsyncEngineClient,
44 - model_config: ModelConfig,
45 - served_model_names: List[str],
46 - response_role: str,
47 - *,
48 - lora_modules: Optional[List[LoRAModulePath]],
49 - prompt_adapters: Optional[List[PromptAdapterPath]],
50 - request_logger: Optional[RequestLogger],
51 - chat_template: Optional[str],
52 - return_tokens_as_token_ids: bool = False,
53 - ):
46 + def __init__(self,
47 + async_engine_client: AsyncEngineClient,
48 + model_config: ModelConfig,
49 + served_model_names: List[str],
50 + response_role: str,
51 + *,
52 + lora_modules: Optional[List[LoRAModulePath]],
53 + prompt_adapters: Optional[List[PromptAdapterPath]],
54 + request_logger: Optional[RequestLogger],
55 + chat_template: Optional[str],
56 + return_tokens_as_token_ids: bool = False,
57 + enable_auto_tools: bool = False,
58 + tool_parser: Optional[str] = None):
59 super().__init__(async_engine_client=async_engine_client,
60 model_config=model_config,
61 served_model_names=served_model_names,
@@ -60,10 +65,27 @@ class OpenAIServingChat(OpenAIServing):
65 return_tokens_as_token_ids=return_tokens_as_token_ids)
66
67 self.response_role = response_role
63 -
64 - # If this is None we use the tokenizer's default chat template
68 + self.use_tool_use_model_template = False
69 self.chat_template = load_chat_template(chat_template)
70
71 + # set up tool use
72 + self.enable_auto_tools: bool = enable_auto_tools
73 + if self.enable_auto_tools:
74 + logger.info(
75 + "\"auto\" tool choice has been enabled please note that while"
76 + " the parallel_tool_calls client option is preset for "
77 + "compatibility reasons, it will be ignored.")
78 +
79 + self.tool_parser: Optional[Callable[[AnyTokenizer], ToolParser]] = None
80 + if self.enable_auto_tools:
81 + if tool_parser == "mistral":
82 + self.tool_parser = MistralToolParser
83 + elif tool_parser == "hermes":
84 + self.tool_parser = Hermes2ProToolParser
85 + else:
86 + raise TypeError("Error: --enable-auto-tool-choice requires "
87 + "--tool-call-parser")
88 +
89 async def create_chat_completion(
90 self,
91 request: ChatCompletionRequest,
@@ -76,11 +98,10 @@ class OpenAIServingChat(OpenAIServing):
98 for the API specification. This API mimics the OpenAI
99 ChatCompletion API.
100
79 - NOTE: Currently we do not support the following feature:
80 - - function_call (Users should implement this by themselves)
101 """
102 error_check_ret = await self._check_model(request)
103 if error_check_ret is not None:
104 + logger.error("Error with model %s", error_check_ret)
105 return error_check_ret
106
107 try:
@@ -119,6 +140,20 @@ class OpenAIServingChat(OpenAIServing):
140 logger.error("Error in loading multi-modal data: %s", e)
141 return self.create_error_response(str(e))
142
143 + # validation for OpenAI tools
144 + # tool_choice = "required" is not supported
145 + if request.tool_choice == "required":
146 + return self.create_error_response(
147 + "tool_choice = \"required\" is not supported!")
148 +
149 + # "auto" tools requires --enable-auto-tool-choice
150 + # and --tool-call-parser
151 + if request.tool_choice == "auto" and not (
152 + self.enable_auto_tools and self.tool_parser is not None):
153 + return self.create_error_response(
154 + "\"auto\" tool choice requires "
155 + "--enable-auto-tool-choice and --tool-call-parser to be set")
156 +
157 request_id = f"chat-{random_uuid()}"
158 try:
159 guided_decode_logits_processor = (
@@ -187,6 +222,7 @@ class OpenAIServingChat(OpenAIServing):
222 if request.stream:
223 return self.chat_completion_stream_generator(
224 request, result_generator, request_id, conversation, tokenizer)
225 +
226 try:
227 return await self.chat_completion_full_generator(
228 request, result_generator, request_id, conversation, tokenizer)
@@ -219,6 +255,9 @@ class OpenAIServingChat(OpenAIServing):
255 previous_num_tokens = [0] * num_choices
256 finish_reason_sent = [False] * num_choices
257
258 + tool_parser: Optional[ToolParser] = self.tool_parser(
259 + tokenizer) if self.tool_parser else None
260 +
261 try:
262 async for res in result_generator:
263 # We need to do it here, because if there are exceptions in
@@ -228,6 +267,9 @@ class OpenAIServingChat(OpenAIServing):
267 # Send first response for each request.n (index) with
268 # the role
269 role = self.get_chat_request_role(request)
270 +
271 + # NOTE num_choices defaults to 1 so this usually executes
272 + # once per request
273 for i in range(num_choices):
274 choice_data = ChatCompletionResponseStreamChoice(
275 index=i,
@@ -240,14 +282,18 @@ class OpenAIServingChat(OpenAIServing):
282 created=created_time,
283 choices=[choice_data],
284 model=model_name)
285 +
286 + # if usage should be included
287 if (request.stream_options
288 and request.stream_options.include_usage):
245 - if (request.stream_options.continuous_usage_stats):
289 + # if continuous usage stats are requested, add it
290 + if request.stream_options.continuous_usage_stats:
291 prompt_tokens = len(res.prompt_token_ids)
292 usage = UsageInfo(prompt_tokens=prompt_tokens,
293 completion_tokens=0,
294 total_tokens=prompt_tokens)
295 chunk.usage = usage
296 + # otherwise don't
297 else:
298 chunk.usage = None
299
@@ -257,7 +303,7 @@ class OpenAIServingChat(OpenAIServing):
303 # Send response to echo the input portion of the
304 # last message
305 if request.echo:
260 - last_msg_content = ""
306 + last_msg_content: Optional[str] = ""
307 if conversation and conversation[-1].get(
308 "content") and conversation[-1].get(
309 "role") == role:
@@ -298,6 +344,7 @@ class OpenAIServingChat(OpenAIServing):
344 first_iteration = False
345
346 for output in res.outputs:
347 +
348 i = output.index
349
350 if finish_reason_sent[i]:
@@ -320,20 +367,50 @@ class OpenAIServingChat(OpenAIServing):
367 logprobs = None
368
369 delta_text = output.text[len(previous_texts[i]):]
323 - previous_texts[i] = output.text
324 - previous_num_tokens[i] = len(output.token_ids)
370 + delta_message: Optional[DeltaMessage] = None
371
326 - if request.tool_choice and type(
327 - request.tool_choice
328 - ) is ChatCompletionNamedToolChoiceParam:
372 + # handle streaming deltas for tools with named tool_choice
373 + if (request.tool_choice and type(request.tool_choice) is
374 + ChatCompletionNamedToolChoiceParam):
375 delta_message = DeltaMessage(tool_calls=[
330 - ToolCall(function=FunctionCall(
376 + DeltaToolCall(function=DeltaFunctionCall(
377 name=request.tool_choice.function.name,
332 - arguments=delta_text))
378 + arguments=delta_text),
379 + index=i)
380 ])
381 +
382 + # handle streaming deltas for tools with "auto" tool choice
383 + elif (self._should_stream_with_auto_tool_parsing(request)
384 + and tool_parser):
385 + delta_message = (
386 + tool_parser.extract_tool_calls_streaming(
387 + previous_text=previous_texts[i],
388 + current_text=output.text,
389 + delta_text=delta_text,
390 + previous_token_ids= \
391 + output.token_ids[
392 + :-1 * len(delta_token_ids)
393 + ],
394 + current_token_ids=output.token_ids,
395 + delta_token_ids=delta_token_ids
396 + )
397 + )
398 +
399 + # handle streaming just a content delta
400 else:
401 delta_message = DeltaMessage(content=delta_text)
402
403 + # set the previous values for the next iteration
404 + previous_texts[i] = output.text
405 + previous_num_tokens[i] = len(output.token_ids)
406 +
407 + # if the message delta is None (e.g. because it was a
408 + # "control token" for tool calls or the parser otherwise
409 + # wasn't ready to send a token, then
410 + # get the next token without streaming a chunk
411 + if delta_message is None:
412 + continue
413 +
414 if output.finish_reason is None:
415 # Send token-by-token response for each request.n
416
@@ -348,6 +425,8 @@ class OpenAIServingChat(OpenAIServing):
425 created=created_time,
426 choices=[choice_data],
427 model=model_name)
428 +
429 + # handle usage stats if requested & if continuous
430 if (request.stream_options
431 and request.stream_options.include_usage):
432 if (request.stream_options.continuous_usage_stats):
@@ -365,14 +444,55 @@ class OpenAIServingChat(OpenAIServing):
444
445 data = chunk.model_dump_json(exclude_unset=True)
446 yield f"data: {data}\n\n"
447 +
448 + # if the model is finished generating
449 else:
450 + # check to make sure we haven't "forgotten" to stream
451 + # any tokens that were generated but previously
452 + # matched by partial json parsing
453 + # only happens if we are NOT using guided decoding
454 + if tool_parser:
455 + index = len(
456 + tool_parser.prev_tool_call_arr) - 1 if len(
457 + tool_parser.prev_tool_call_arr) > 0 else 0
458 + else:
459 + index = 0
460 +
461 + if self._should_check_for_unstreamed_tool_arg_tokens(
462 + delta_message, output) and tool_parser:
463 + # get the expected call based on partial JSON
464 + # parsing which "autocompletes" the JSON
465 + expected_call = json.dumps(
466 + tool_parser.prev_tool_call_arr[index].get(
467 + "arguments", {}))
468 +
469 + # get what we've streamed so for for arguments
470 + # for the current tool
471 + actual_call = tool_parser.streamed_args_for_tool[
472 + index]
473 +
474 + # check to see if there's anything left to stream
475 + remaining_call = expected_call.replace(
476 + actual_call, "", 1)
477 +
478 + # set that as a delta message
479 + delta_message = DeltaMessage(tool_calls=[
480 + DeltaToolCall(index=index,
481 + function=DeltaFunctionCall(
482 + arguments=remaining_call).
483 + model_dump(exclude_none=True))
484 + ])
485 +
486 # Send the finish response for each request.n only once
487 prompt_tokens = len(res.prompt_token_ids)
488 choice_data = ChatCompletionResponseStreamChoice(
489 index=i,
490 delta=delta_message,
491 logprobs=logprobs,
375 - finish_reason=output.finish_reason,
492 + finish_reason=output.finish_reason
493 + if not (tool_parser
494 + and len(tool_parser.prev_tool_call_arr))
495 + else "tool_calls",
496 stop_reason=output.stop_reason)
497 chunk = ChatCompletionStreamResponse(
498 id=request_id,
@@ -398,6 +518,8 @@ class OpenAIServingChat(OpenAIServing):
518 yield f"data: {data}\n\n"
519 finish_reason_sent[i] = True
520
521 + # once the final token is handled, if stream_options.include_usage
522 + # is sent, send the usage
523 if (request.stream_options
524 and request.stream_options.include_usage):
525 final_usage = UsageInfo(
@@ -419,6 +541,7 @@ class OpenAIServingChat(OpenAIServing):
541
542 except ValueError as e:
543 # TODO: Use a vllm-specific Validation Error
544 + logger.error("error in chat completion stream generator: %s", e)
545 data = self.create_streaming_error_response(str(e))
546 yield f"data: {data}\n\n"
547 # Send the final done message after all response.n are finished
@@ -463,8 +586,21 @@ class OpenAIServingChat(OpenAIServing):
586 else:
587 logprobs = None
588
466 - if request.tool_choice and type(
589 + # by default, tools are not used.
590 + tools_called = False
591 +
592 + # if auto tools are not enabled, and a named tool choice using
593 + # outlines is not being used
594 + if not (self.enable_auto_tools
595 + or not self.tool_parser) and not isinstance(
596 + request.tool_choice,
597 + ChatCompletionNamedToolChoiceParam):
598 + message = ChatMessage(role=role, content=output.text)
599 +
600 + # if the request uses tools and specified a tool choice
601 + elif request.tool_choice and type(
602 request.tool_choice) is ChatCompletionNamedToolChoiceParam:
603 +
604 message = ChatMessage(
605 role=role,
606 content="",
@@ -473,14 +609,47 @@ class OpenAIServingChat(OpenAIServing):
609 name=request.tool_choice.function.name,
610 arguments=output.text))
611 ])
612 + tools_called = True
613 +
614 + # if the request doesn't use tool choice
615 + # OR specifies to not use a tool
616 elif not request.tool_choice or request.tool_choice == "none":
617 +
618 + message = ChatMessage(role=role, content=output.text)
619 +
620 + # handle when there are tools and tool choice is auto
621 + elif request.tools and (
622 + request.tool_choice == "auto"
623 + or request.tool_choice is None) and self.enable_auto_tools \
624 + and self.tool_parser:
625 +
626 + tool_parser = self.tool_parser(tokenizer)
627 + tool_call_info = tool_parser.extract_tool_calls(output.text)
628 + tools_called = tool_call_info.tools_called
629 + if tool_call_info.tools_called:
630 + message = ChatMessage(role=role,
631 + content=tool_call_info.content,
632 + tool_calls=tool_call_info.tool_calls)
633 +
634 + else:
635 + # FOR NOW make it a chat message; we will have to detect
636 + # the type to make it later.
637 + message = ChatMessage(role=role, content=output.text)
638 +
639 + # undetermined case that is still important to handle
640 + else:
641 + logger.error(
642 + "Error in chat_completion_full_generator - cannot determine"
643 + " if tools should be extracted. Returning a standard chat "
644 + "completion.")
645 message = ChatMessage(role=role, content=output.text)
646
647 choice_data = ChatCompletionResponseChoice(
648 index=output.index,
649 message=message,
650 logprobs=logprobs,
483 - finish_reason=output.finish_reason,
651 + finish_reason="tool_calls" if tools_called else
652 + output.finish_reason if output.finish_reason else "stop",
653 stop_reason=output.stop_reason)
654 choices.append(choice_data)
655
@@ -488,10 +657,11 @@ class OpenAIServingChat(OpenAIServing):
657 last_msg_content = ""
658 if conversation and conversation[-1].get(
659 "content") and conversation[-1].get("role") == role:
491 - last_msg_content = conversation[-1]["content"]
660 + last_msg_content = conversation[-1]["content"] or ""
661
662 for choice in choices:
494 - full_message = last_msg_content + choice.message.content
663 + full_message = last_msg_content + (choice.message.content
664 + or "")
665 choice.message.content = full_message
666
667 num_prompt_tokens = len(final_res.prompt_token_ids)
@@ -574,3 +744,38 @@ class OpenAIServingChat(OpenAIServing):
744 ))
745
746 return ChatCompletionLogProbs(content=logprobs_content)
747 +
748 + def _should_stream_with_auto_tool_parsing(self,
749 + request: ChatCompletionRequest):
750 + """
751 + Utility function to check if streamed tokens should go through the tool
752 + call parser that was configured.
753 +
754 + We only want to do this IF user-provided tools are set, a tool parser
755 + is configured, "auto" tool choice is enabled, and the request's tool
756 + choice field indicates that "auto" tool choice should be used.
757 + """
758 + return (request.tools and self.tool_parser and self.enable_auto_tools
759 + and request.tool_choice in ['auto', None])
760 +
761 + def _should_check_for_unstreamed_tool_arg_tokens(
762 + self,
763 + delta_message: Optional[DeltaMessage],
764 + output: CompletionOutput,
765 + ) -> bool:
766 + """
767 + Check to see if we should check for unstreamed tool arguments tokens.
768 + This is only applicable when auto tool parsing is enabled, the delta
769 + is a tool call with arguments.
770 + """
771 +
772 + # yapf: disable
773 + return bool(
774 + # if there is a delta message that includes tool calls which
775 + # include a function that has arguments
776 + self.enable_auto_tools and self.tool_parser and delta_message
777 + and delta_message.tool_calls and delta_message.tool_calls[0]
778 + and delta_message.tool_calls[0].function
779 + and delta_message.tool_calls[0].function.arguments is not None
780 + and output.finish_reason is not None
781 + )
vllm/entrypoints/openai/serving_tokenization.py
+5 -1
@@ -43,7 +43,11 @@ class OpenAIServingTokenization(OpenAIServing):
43 request_logger=request_logger)
44
45 # If this is None we use the tokenizer's default chat template
46 - self.chat_template = load_chat_template(chat_template)
46 + # the list of commonly-used chat template names for HF named templates
47 + hf_chat_templates: List[str] = ['default', 'tool_use']
48 + self.chat_template = chat_template \
49 + if chat_template in hf_chat_templates \
50 + else load_chat_template(chat_template)
51
52 async def create_tokenize(
53 self,
vllm/entrypoints/openai/tool_parsers/__init__.py new
+5
@@ -0,0 +1,5 @@
1 +from .abstract_tool_parser import ToolParser
2 +from .hermes_tool_parser import Hermes2ProToolParser
3 +from .mistral_tool_parser import MistralToolParser
4 +
5 +__all__ = ["ToolParser", "Hermes2ProToolParser", "MistralToolParser"]
\ No newline at end of file
vllm/entrypoints/openai/tool_parsers/abstract_tool_parser.py new
+58
@@ -0,0 +1,58 @@
1 +from typing import Dict, List, Sequence, Union
2 +
3 +from vllm.entrypoints.openai.protocol import (DeltaMessage,
4 + ExtractedToolCallInformation)
5 +from vllm.logger import init_logger
6 +from vllm.transformers_utils.tokenizer import AnyTokenizer
7 +
8 +logger = init_logger(__name__)
9 +
10 +
11 +class ToolParser:
12 + """
13 + Abstract ToolParser class that should not be used directly. Provided
14 + properties and methods should be used in
15 + derived classes.
16 + """
17 +
18 + def __init__(self, tokenizer: AnyTokenizer):
19 + self.prev_tool_call_arr: List[Dict] = []
20 + # the index of the tool call that is currently being parsed
21 + self.current_tool_id: int = -1
22 + self.current_tool_name_sent: bool = False
23 + self.current_tool_initial_sent: bool = False
24 + self.streamed_args_for_tool: List[str] = []
25 +
26 + self.model_tokenizer = tokenizer
27 +
28 + def extract_tool_calls(self,
29 + model_output: str) -> ExtractedToolCallInformation:
30 + """
31 + Static method that should be implemented for extracting tool calls from
32 + a complete model-generated string.
33 + Used for non-streaming responses where we have the entire model response
34 + available before sending to the client.
35 + Static because it's stateless.
36 + """
37 + raise NotImplementedError(
38 + "AbstractToolParser.extract_tool_calls has not been implemented!")
39 +
40 + def extract_tool_calls_streaming(
41 + self,
42 + previous_text: str,
43 + current_text: str,
44 + delta_text: str,
45 + previous_token_ids: Sequence[int],
46 + current_token_ids: Sequence[int],
47 + delta_token_ids: Sequence[int],
48 + ) -> Union[DeltaMessage, None]:
49 + """
50 + Instance method that should be implemented for extracting tool calls
51 + from an incomplete response; for use when handling tool calls and
52 + streaming. Has to be an instance method because it requires state -
53 + the current tokens/diffs, but also the information about what has
54 + previously been parsed and extracted (see constructor)
55 + """
56 + raise NotImplementedError(
57 + "AbstractToolParser.extract_tool_calls_streaming has not been "
58 + "implemented!")
vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py new
+344
@@ -0,0 +1,344 @@
1 +import json
2 +import re
3 +from typing import Dict, List, Sequence, Union
4 +
5 +import partial_json_parser
6 +from partial_json_parser.core.options import Allow
7 +
8 +from vllm.entrypoints.openai.protocol import (DeltaFunctionCall, DeltaMessage,
9 + DeltaToolCall,
10 + ExtractedToolCallInformation,
11 + FunctionCall,
12 + InitialDeltaToolCall, ToolCall)
13 +from vllm.entrypoints.openai.tool_parsers.abstract_tool_parser import (
14 + ToolParser)
15 +from vllm.entrypoints.openai.tool_parsers.utils import (
16 + extract_intermediate_diff)
17 +from vllm.logger import init_logger
18 +from vllm.transformers_utils.tokenizer import AnyTokenizer, MistralTokenizer
19 +
20 +logger = init_logger(__name__)
21 +
22 +
23 +class Hermes2ProToolParser(ToolParser):
24 +
25 + def __init__(self, tokenizer: AnyTokenizer):
26 + super().__init__(tokenizer)
27 +
28 + if isinstance(self.model_tokenizer, MistralTokenizer):
29 + logger.error(
30 + "Detected Mistral tokenizer when using a Hermes model")
31 + self.model_tokenizer = self.model_tokenizer.tokenizer
32 +
33 + self.current_tool_name_sent: bool = False
34 + self.prev_tool_call_arr: List[Dict] = []
35 + self.current_tool_id: int = -1
36 + self.current_tool_name_sent = False
37 + self.current_tool_initial_sent: bool = False
38 + self.streamed_args_for_tool: List[str] = [
39 + ] # map what has been streamed for each tool so far to a list
40 +
41 + self.tool_call_start_token: str = "<tool_call>"
42 + self.tool_call_end_token: str = "</tool_call>"
43 +
44 + self.tool_call_regex = re.compile(
45 + r"<tool_call>(.*?)</tool_call>|<tool_call>(.*)", re.DOTALL)
46 + self.scratch_pad_regex = re.compile(
47 + r"<scratch_pad>(.*?)</scratch_pad>", re.DOTALL)
48 +
49 + if not self.model_tokenizer:
50 + raise ValueError(
51 + "The model tokenizer must be passed to the ToolParser "
52 + "constructor during construction.")
53 + self.tool_call_start_token_id: int = self.model_tokenizer.vocab[
54 + self.tool_call_start_token]
55 + self.tool_call_end_token_id: int = self.model_tokenizer.vocab[
56 + self.tool_call_end_token]
57 + if not self.tool_call_start_token_id or not self.tool_call_end_token_id:
58 + raise RuntimeError(
59 + "Hermes 2 Pro Tool parser could not locate tool call start/end "
60 + "tokens in the tokenizer!")
61 +
62 + def extract_tool_calls(self,
63 + model_output: str) -> ExtractedToolCallInformation:
64 +
65 + # sanity check; avoid unnecessary processing
66 + if self.tool_call_start_token not in model_output:
67 + return ExtractedToolCallInformation(tools_called=False,
68 + tool_calls=[],
69 + content=model_output)
70 +
71 + else:
72 +
73 + try:
74 + # there are two possible captures - between tags, or between a
75 + # tag and end-of-string so the result of
76 + # findall is an array of tuples where one is a function call and
77 + # the other is None
78 + function_call_tuples = (
79 + self.tool_call_regex.findall(model_output))
80 +
81 + # load the JSON, and then use it to build the Function and
82 + # Tool Call
83 + raw_function_calls = [
84 + json.loads(match[0] if match[0] else match[1])
85 + for match in function_call_tuples
86 + ]
87 + tool_calls = [
88 + ToolCall(
89 + type="function",
90 + function=FunctionCall(
91 + name=function_call["name"],
92 + # function call args are JSON but as a string
93 + arguments=json.dumps(function_call["arguments"])))
94 + for function_call in raw_function_calls
95 + ]
96 +
97 + content = model_output[:model_output.
98 + find(self.tool_call_start_token)]
99 + return ExtractedToolCallInformation(
100 + tools_called=True,
101 + tool_calls=tool_calls,
102 + content=content if content else None)
103 +
104 + except Exception as e:
105 + logger.error("Error in extracting tool call from response %s",
106 + e)
107 + return ExtractedToolCallInformation(tools_called=False,
108 + tool_calls=[],
109 + content=model_output)
110 +
111 + def extract_tool_calls_streaming(
112 + self,
113 + previous_text: str,
114 + current_text: str,
115 + delta_text: str,
116 + previous_token_ids: Sequence[int],
117 + current_token_ids: Sequence[int],
118 + delta_token_ids: Sequence[int],
119 + ) -> Union[DeltaMessage, None]:
120 +
121 + logger.debug("delta_text: %s", delta_text)
122 + logger.debug("delta_token_ids: %s", delta_token_ids)
123 + # check to see if we should be streaming a tool call - is there a
124 + if self.tool_call_start_token_id not in current_token_ids:
125 + logger.debug("No tool call tokens found!")
126 + return DeltaMessage(content=delta_text)
127 +
128 + try:
129 +
130 + # figure out where we are in the parsing by counting tool call
131 + # start & end tags
132 + prev_tool_start_count = previous_token_ids.count(
133 + self.tool_call_start_token_id)
134 + prev_tool_end_count = previous_token_ids.count(
135 + self.tool_call_end_token_id)
136 + cur_tool_start_count = current_token_ids.count(
137 + self.tool_call_start_token_id)
138 + cur_tool_end_count = current_token_ids.count(
139 + self.tool_call_end_token_id)
140 +
141 + # case: if we're generating text, OR rounding out a tool call
142 + if (cur_tool_start_count == cur_tool_end_count
143 + and prev_tool_end_count == cur_tool_end_count):
144 + logger.debug("Generating text content! skipping tool parsing.")
145 + if delta_text != self.tool_call_end_token:
146 + return DeltaMessage(content=delta_text)
147 +
148 + # case: if tool open & close tag counts don't match, we're doing
149 + # imaginary "else" block here
150 + # something with tools with this diff.
151 + # flags for partial JSON parting. exported constants from
152 + # "Allow" are handled via BIT MASK
153 + flags = Allow.ALL if self.current_tool_name_sent \
154 + else Allow.ALL & ~Allow.STR
155 +
156 + # case -- we're starting a new tool call
157 + if (cur_tool_start_count > cur_tool_end_count
158 + and cur_tool_start_count > prev_tool_start_count):
159 + if len(delta_token_ids) > 1:
160 + tool_call_portion = current_text.split(
161 + self.tool_call_start_token)[-1]
162 + else:
163 + tool_call_portion = None
164 + delta = None
165 +
166 + text_portion = None
167 +
168 + # set cursors and state appropriately
169 + self.current_tool_id += 1
170 + self.current_tool_name_sent = False
171 + self.current_tool_initial_sent = False
172 + self.streamed_args_for_tool.append("")
173 + logger.debug("Starting on a new tool %s", self.current_tool_id)
174 +
175 + # case -- we're updating an existing tool call
176 + elif (cur_tool_start_count > cur_tool_end_count
177 + and cur_tool_start_count == prev_tool_start_count):
178 +
179 + # get the portion of the text that's the tool call
180 + tool_call_portion = current_text.split(
181 + self.tool_call_start_token)[-1]
182 + text_portion = None
183 +
184 + # case -- the current tool call is being closed.
185 + elif (cur_tool_start_count == cur_tool_end_count
186 + and cur_tool_end_count > prev_tool_end_count):
187 + diff = self.prev_tool_call_arr[self.current_tool_id].get(
188 + "arguments")
189 + if diff:
190 + diff = json.dumps(diff).replace(
191 + self.streamed_args_for_tool[self.current_tool_id], "")
192 + logger.debug(
193 + "Finishing tool and found diff that had not "
194 + "been streamed yet: %s", diff)
195 + self.streamed_args_for_tool[self.current_tool_id] \
196 + += diff
197 + return DeltaMessage(tool_calls=[
198 + DeltaToolCall(index=self.current_tool_id,
199 + function=DeltaFunctionCall(
200 + arguments=diff).model_dump(
201 + exclude_none=True))
202 + ])
203 +
204 + # case -- otherwise we're just generating text
205 + else:
206 + text = delta_text.replace(self.tool_call_start_token, "")
207 + text = text.replace(self.tool_call_end_token, "")
208 + delta = DeltaMessage(tool_calls=[], content=text)
209 + return delta
210 +
211 + try:
212 +
213 + current_tool_call = partial_json_parser.loads(
214 + tool_call_portion or "{}",
215 + flags) if tool_call_portion else None
216 + logger.debug("Parsed tool call %s", current_tool_call)
217 + except partial_json_parser.core.exceptions.MalformedJSON:
218 + logger.debug('not enough tokens to parse into JSON yet')
219 + return None
220 +
221 + # case - we haven't sent the initial delta with the tool call ID
222 + # (it will be sent)
223 + if not self.current_tool_initial_sent:
224 + self.current_tool_initial_sent = True
225 + return DeltaMessage(tool_calls=[
226 + InitialDeltaToolCall(
227 + index=self.current_tool_id).model_dump(
228 + exclude_none=True)
229 + ])
230 +
231 + # case - we haven't sent the tool name yet. If it's available, send
232 + # it. otherwise, wait until it's available.
233 + elif not self.current_tool_name_sent:
234 + function_name: Union[str, None] = current_tool_call.get("name")
235 + if function_name:
236 + self.current_tool_name_sent = True
237 + return DeltaMessage(tool_calls=[
238 + DeltaToolCall(index=self.current_tool_id,
239 + function=DeltaFunctionCall(
240 + name=function_name).model_dump(
241 + exclude_none=True))
242 + ])
243 + else:
244 + return None
245 + # case -- otherwise, send the tool call delta
246 +
247 + # if the tool call portion is None, send the delta as text
248 + if tool_call_portion is None:
249 + # if there's text but not tool calls, send that -
250 + # otherwise None to skip chunk
251 + delta = DeltaMessage(content=delta_text) \
252 + if text_portion is not None else None
253 + return delta
254 +
255 + # now, the nitty-gritty of tool calls
256 + # now we have the portion to parse as tool call.
257 +
258 + logger.debug("Trying to parse current tool call with ID %s",
259 + self.current_tool_id)
260 +
261 + # if we're starting a new tool call, push an empty object in as
262 + # a placeholder for the arguments
263 + if len(self.prev_tool_call_arr) <= self.current_tool_id:
264 + self.prev_tool_call_arr.append({})
265 +
266 + # main logic for tool parsing here - compare prev. partially-parsed
267 + # JSON to the current partially-parsed JSON
268 + prev_arguments = (
269 + self.prev_tool_call_arr[self.current_tool_id].get("arguments"))
270 + cur_arguments = current_tool_call.get("arguments")
271 +
272 + logger.debug("diffing old arguments: %s", prev_arguments)
273 + logger.debug("against new ones: %s", cur_arguments)
274 +
275 + # case -- no arguments have been created yet. skip sending a delta.
276 + if not cur_arguments and not prev_arguments:
277 + logger.debug("Skipping text %s - no arguments", delta_text)
278 + delta = None
279 +
280 + # case -- prev arguments are defined, but non are now.
281 + # probably impossible, but not a fatal error - just keep going
282 + elif not cur_arguments and prev_arguments:
283 + logger.error("should be impossible to have arguments reset "
284 + "mid-call. skipping streaming anything.")
285 + delta = None
286 +
287 + # case -- we now have the first info about arguments available from
288 + # autocompleting the JSON
289 + elif cur_arguments and not prev_arguments:
290 +
291 + cur_arguments_json = json.dumps(cur_arguments)
292 + logger.debug("finding %s in %s", delta_text,
293 + cur_arguments_json)
294 +
295 + # get the location where previous args differ from current
296 + args_delta_start_loc = cur_arguments_json.index(delta_text) \
297 + + len(delta_text)
298 +
299 + # use that to find the actual delta
300 + arguments_delta = cur_arguments_json[:args_delta_start_loc]
301 + logger.debug("First tokens in arguments received: %s",
302 + arguments_delta)
303 +
304 + delta = DeltaMessage(tool_calls=[
305 + DeltaToolCall(index=self.current_tool_id,
306 + function=DeltaFunctionCall(
307 + arguments=arguments_delta).model_dump(
308 + exclude_none=True))
309 + ])
310 + self.streamed_args_for_tool[self.current_tool_id] \
311 + += arguments_delta
312 +
313 + # last case -- we have an update to existing arguments.
314 + elif cur_arguments and prev_arguments:
315 +
316 + cur_args_json = json.dumps(cur_arguments)
317 + prev_args_json = json.dumps(prev_arguments)
318 + logger.debug("Searching for diff between\n%s", cur_args_json)
319 + logger.debug("and\n%s", prev_args_json)
320 + argument_diff = extract_intermediate_diff(
321 + cur_args_json, prev_args_json)
322 + logger.debug("got argument diff %s", argument_diff)
323 + delta = DeltaMessage(tool_calls=[
324 + DeltaToolCall(index=self.current_tool_id,
325 + function=DeltaFunctionCall(
326 + arguments=argument_diff).model_dump(
327 + exclude_none=True))
328 + ])
329 + self.streamed_args_for_tool[self.current_tool_id] \
330 + += argument_diff
331 +
332 + # handle saving the state for the current tool into
333 + # the "prev" list for use in diffing for the next iteration
334 + if self.current_tool_id == len(self.prev_tool_call_arr) - 1:
335 + self.prev_tool_call_arr[self.current_tool_id] = \
336 + current_tool_call
337 + else:
338 + self.prev_tool_call_arr.append(current_tool_call)
339 +
340 + return delta
341 +
342 + except Exception as e:
343 + logger.error("Error trying to handle streaming tool call: %s", e)
344 + return None # do not stream a delta. skip this token ID.
vllm/entrypoints/openai/tool_parsers/mistral_tool_parser.py new
+293
@@ -0,0 +1,293 @@
1 +import json
2 +import re
3 +from typing import Dict, List, Sequence, Union
4 +
5 +import partial_json_parser
6 +from partial_json_parser.core.options import Allow
7 +
8 +from vllm.entrypoints.openai.protocol import (DeltaFunctionCall, DeltaMessage,
9 + DeltaToolCall,
10 + ExtractedToolCallInformation,
11 + FunctionCall,
12 + InitialDeltaToolCall, ToolCall)
13 +from vllm.entrypoints.openai.tool_parsers.abstract_tool_parser import (
14 + ToolParser)
15 +from vllm.entrypoints.openai.tool_parsers.utils import (
16 + extract_intermediate_diff)
17 +from vllm.logger import init_logger
18 +from vllm.transformers_utils.tokenizer import AnyTokenizer, MistralTokenizer
19 +
20 +logger = init_logger(__name__)
21 +
22 +
23 +class MistralToolParser(ToolParser):
24 + """
25 + Tool call parser for Mistral 7B Instruct v0.3, intended for use with the
26 + examples/tool_chat_template_mistral.jinja template.
27 +
28 + Used when --enable-auto-tool-choice --tool-call-parser gmistral are all set
29 + """
30 +
31 + def __init__(self, tokenizer: AnyTokenizer):
32 + super().__init__(tokenizer)
33 +
34 + if isinstance(self.model_tokenizer, MistralTokenizer):
35 + self.model_tokenizer = self.model_tokenizer.tokenizer
36 + else:
37 + logger.info("Non-Mistral tokenizer detected when using a Mistral "
38 + "model...")
39 +
40 + # initialize properties used for state when parsing tool calls in
41 + # streaming mode
42 + self.prev_tool_call_arr: List[Dict] = []
43 + self.current_tool_id: int = -1
44 + self.current_tool_name_sent: bool = False
45 + self.current_tool_initial_sent: bool = False
46 + self.streamed_args_for_tool: List[str] = [
47 + ] # map what has been streamed for each tool so far to a list
48 + self.bot_token = "[TOOL_CALLS]"
49 + self.bot_token_id = self.model_tokenizer.vocab[self.bot_token]
50 + self.tool_call_regex = re.compile(r"\[{.*?}\]", re.DOTALL)
51 +
52 + def extract_tool_calls(self,
53 + model_output: str) -> ExtractedToolCallInformation:
54 + """
55 + Extract the tool calls from a complete model response. Requires
56 + find-and-replacing single quotes with double quotes for JSON parsing,
57 + make sure your tool call arguments don't ever include quotes!
58 + """
59 +
60 + # case -- if a tool call token is not present, return a text response
61 + if self.bot_token not in model_output:
62 + return ExtractedToolCallInformation(tools_called=False,
63 + tool_calls=[],
64 + content=model_output)
65 + try:
66 +
67 + # use a regex to find the tool call. remove the BOT token
68 + # and make sure to replace single quotes with double quotes
69 + raw_tool_call = self.tool_call_regex.findall(
70 + model_output.replace(self.bot_token, ""))[0]
71 +
72 + # load the JSON, and then use it to build the Function and
73 + # Tool Call
74 + function_call_arr = json.loads(raw_tool_call)
75 + tool_calls: List[ToolCall] = [
76 + ToolCall(
77 + type="function",
78 + function=FunctionCall(
79 + name=raw_function_call["name"],
80 + # function call args are JSON but as a string
81 + arguments=json.dumps(raw_function_call["arguments"])))
82 + for raw_function_call in function_call_arr
83 + ]
84 +
85 + # get any content before the tool call
86 + content = model_output.split(self.bot_token)[0]
87 + return ExtractedToolCallInformation(
88 + tools_called=True,
89 + tool_calls=tool_calls,
90 + content=content if len(content) > 0 else None)
91 +
92 + except Exception as e:
93 + logger.error("Error in extracting tool call from response: %s", e)
94 + print("ERROR", e)
95 + # return information to just treat the tool call as regular JSON
96 + return ExtractedToolCallInformation(tools_called=False,
97 + tool_calls=[],
98 + content=model_output)
99 +
100 + def extract_tool_calls_streaming(
101 + self,
102 + previous_text: str,
103 + current_text: str,
104 + delta_text: str,
105 + previous_token_ids: Sequence[int],
106 + current_token_ids: Sequence[int],
107 + delta_token_ids: Sequence[int],
108 + ) -> Union[DeltaMessage, None]:
109 +
110 + # if the tool call token is not in the tokens generated so far, append
111 + # output to contents since it's not a tool
112 + if self.bot_token_id not in current_token_ids:
113 + return DeltaMessage(content=delta_text)
114 +
115 + # if the tool call token ID IS in the tokens generated so far, that
116 + # means we're parsing as tool calls now
117 +
118 + # handle if we detected the BOT token which means the start of tool
119 + # calling
120 + if (self.bot_token_id in delta_token_ids
121 + and len(delta_token_ids) == 1):
122 + # if it's the only token, return None, so we don't send a chat
123 + # completion any don't send a control token
124 + return None
125 +
126 + # bit mask flags for partial JSON parsing. If the name hasn't been
127 + # sent yet, don't allow sending
128 + # an incomplete string since OpenAI only ever (as far as I have
129 + # seen) allows sending the entire tool/ function name at once.
130 + flags = Allow.ALL if self.current_tool_name_sent \
131 + else Allow.ALL & ~Allow.STR
132 + try:
133 +
134 + # replace BOT token with empty string, and convert single quotes
135 + # to double to allow parsing as JSON since mistral uses single
136 + # quotes instead of double for tool calls
137 + parsable_arr = current_text.split(self.bot_token)[1]
138 +
139 + # tool calls are generated in an array, so do partial JSON
140 + # parsing on the entire array
141 + try:
142 + tool_call_arr: List[Dict] = partial_json_parser.loads(
143 + parsable_arr, flags)
144 + except partial_json_parser.core.exceptions.MalformedJSON:
145 + logger.debug('not enough tokens to parse into JSON yet')
146 + return None
147 +
148 + # select as the current tool call the one we're on the state at
149 +
150 + current_tool_call: Dict = tool_call_arr[self.current_tool_id] \
151 + if len(tool_call_arr) > 0 else {}
152 +
153 + # case -- if no tokens have been streamed for the tool, e.g.
154 + # only the array brackets, stream nothing
155 + if len(tool_call_arr) == 0:
156 + return None
157 +
158 + # case: we are starting a new tool in the array
159 + # -> array has > 0 length AND length has moved past cursor
160 + elif (len(tool_call_arr) > 0
161 + and len(tool_call_arr) > self.current_tool_id + 1):
162 +
163 + # if we're moving on to a new call, first make sure we
164 + # haven't missed anything in the previous one that was
165 + # auto-generated due to JSON completions, but wasn't
166 + # streamed to the client yet.
167 + if self.current_tool_id >= 0:
168 + diff: Union[str, None] = current_tool_call.get("arguments")
169 +
170 + if diff:
171 + diff = json.dumps(diff).replace(
172 + self.streamed_args_for_tool[self.current_tool_id],
173 + "")
174 + delta = DeltaMessage(tool_calls=[
175 + DeltaToolCall(index=self.current_tool_id,
176 + function=DeltaFunctionCall(
177 + arguments=diff).model_dump(
178 + exclude_none=True))
179 + ])
180 + self.streamed_args_for_tool[
181 + self.current_tool_id] += diff
182 + else:
183 + delta = None
184 + else:
185 + delta = None
186 + # re-set stuff pertaining to progress in the current tool
187 + self.current_tool_id = len(tool_call_arr) - 1
188 + self.current_tool_name_sent = False
189 + self.current_tool_initial_sent = False
190 + self.streamed_args_for_tool.append("")
191 + logger.debug("starting on new tool %d", self.current_tool_id)
192 + return delta
193 +
194 + # case: update an existing tool - this is handled below
195 +
196 + # if the current tool initial data incl. the id, type=function
197 + # and idx not sent, send that
198 + if not self.current_tool_initial_sent:
199 + self.current_tool_initial_sent = True
200 + delta = DeltaMessage(tool_calls=[
201 + InitialDeltaToolCall(
202 + index=self.current_tool_id).model_dump(
203 + exclude_none=True)
204 + ])
205 +
206 + # if the current tool name hasn't been sent, send if available
207 + # - otherwise send nothing
208 + elif not self.current_tool_name_sent:
209 + function_name = current_tool_call.get("name")
210 + if function_name:
211 +
212 + delta = DeltaMessage(tool_calls=[
213 + DeltaToolCall(index=self.current_tool_id,
214 + function=DeltaFunctionCall(
215 + name=function_name).model_dump(
216 + exclude_none=True))
217 + ])
218 + self.current_tool_name_sent = True
219 + else:
220 + delta = None
221 +
222 + # now we know we're on the same tool call and we're streaming
223 + # arguments
224 + else:
225 +
226 + prev_arguments = self.prev_tool_call_arr[
227 + self.current_tool_id].get("arguments")
228 + cur_arguments = current_tool_call.get("arguments")
229 +
230 + new_text = delta_text.replace("\'", "\"")
231 +
232 + if not cur_arguments and not prev_arguments:
233 +
234 + delta = None
235 + elif not cur_arguments and prev_arguments:
236 + logger.error(
237 + "INVARIANT - impossible to have arguments reset "
238 + "mid-arguments")
239 + delta = None
240 + elif cur_arguments and not prev_arguments:
241 + cur_arguments_json = json.dumps(cur_arguments)
242 + logger.debug("finding %s in %s", new_text,
243 + cur_arguments_json)
244 +
245 + arguments_delta = cur_arguments_json[:cur_arguments_json.
246 + index(new_text) +
247 + len(new_text)]
248 + logger.debug("First tokens in arguments received: %s",
249 + arguments_delta)
250 + delta = DeltaMessage(tool_calls=[
251 + DeltaToolCall(index=self.current_tool_id,
252 + function=DeltaFunctionCall(
253 + arguments=arguments_delta).
254 + model_dump(exclude_none=True))
255 + ])
256 + self.streamed_args_for_tool[
257 + self.current_tool_id] += arguments_delta
258 +
259 + elif cur_arguments and prev_arguments:
260 + cur_args_json = json.dumps(cur_arguments)
261 + prev_args_json = json.dumps(prev_arguments)
262 + logger.debug("Searching for diff between \n%s\n%s",
263 + cur_args_json, prev_args_json)
264 +
265 + argument_diff = extract_intermediate_diff(
266 + cur_args_json, prev_args_json)
267 + logger.debug("got arguments diff: %s", argument_diff)
268 + delta = DeltaMessage(tool_calls=[
269 + DeltaToolCall(index=self.current_tool_id,
270 + function=DeltaFunctionCall(
271 + arguments=argument_diff).model_dump(
272 + exclude_none=True))
273 + ])
274 + self.streamed_args_for_tool[
275 + self.current_tool_id] += argument_diff
276 + else:
277 + # try parsing it with regular JSON - if it works we're
278 + # at the end, and we need to send the difference between
279 + # tokens streamed so far and the valid JSON
280 + delta = None
281 +
282 + # check to see if the name is defined and has been sent. if so,
283 + # stream the name - otherwise keep waiting
284 + # finish by setting old and returning None as base case
285 + self.prev_tool_call_arr = tool_call_arr
286 + return delta
287 +
288 + except Exception as e:
289 + logger.error("Error trying to handle streaming tool call: %s", e)
290 + logger.debug(
291 + "Skipping chunk as a result of tool streaming extraction "
292 + "error")
293 + return None
vllm/entrypoints/openai/tool_parsers/utils.py new
+87
@@ -0,0 +1,87 @@
1 +def find_common_prefix(s1: str, s2: str) -> str:
2 + """
3 + Finds a common prefix that is shared between two strings, if there is one.
4 + Order of arguments is NOT important.
5 +
6 + This function is provided as a UTILITY for extracting information from JSON
7 + generated by partial_json_parser, to help in ensuring that the right tokens
8 + are returned in streaming, so that close-quotes, close-brackets and
9 + close-braces are not returned prematurely.
10 +
11 + e.g. find_common_prefix('{"fruit": "ap"}', '{"fruit": "apple"}') ->
12 + '{"fruit": "ap'
13 + """
14 + prefix = ''
15 + min_length = min(len(s1), len(s2))
16 + for i in range(0, min_length):
17 + if s1[i] == s2[i]:
18 + prefix += s1[i]
19 + else:
20 + break
21 + return prefix
22 +
23 +
24 +def find_common_suffix(s1: str, s2: str) -> str:
25 + """
26 + Finds a common suffix shared between two strings, if there is one. Order of
27 + arguments is NOT important.
28 + Stops when the suffix ends OR it hits an alphanumeric character
29 +
30 + e.g. find_common_suffix('{"fruit": "ap"}', '{"fruit": "apple"}') -> '"}'
31 + """
32 + suffix = ''
33 + min_length = min(len(s1), len(s2))
34 + for i in range(1, min_length + 1):
35 + if s1[-i] == s2[-i] and not s1[-i].isalnum():
36 + suffix = s1[-i] + suffix
37 + else:
38 + break
39 + return suffix
40 +
41 +
42 +def extract_intermediate_diff(curr: str, old: str) -> str:
43 + """
44 + Given two strings, extract the difference in the middle between two strings
45 + that are known to have a common prefix and/or suffix.
46 +
47 + This function is provided as a UTILITY for extracting information from JSON
48 + generated by partial_json_parser, to help in ensuring that the right tokens
49 + are returned in streaming, so that close-quotes, close-brackets and
50 + close-braces are not returned prematurely. The order of arguments IS
51 + important - the new version of the partially-parsed JSON must be the first
52 + argument, and the secnod argument must be from the previous generation.
53 +
54 + What it returns, is tokens that should be streamed to the client.
55 +
56 + e.g. extract_intermediate_diff('{"fruit": "apple"}', '{"fruit": "ap"}')
57 + -> 'ple'
58 +
59 + """
60 + suffix = find_common_suffix(curr, old)
61 +
62 + old = old[::-1].replace(suffix[::-1], '', 1)[::-1]
63 + prefix = find_common_prefix(curr, old)
64 + diff = curr
65 + if len(suffix):
66 + diff = diff[::-1].replace(suffix[::-1], '', 1)[::-1]
67 +
68 + if len(prefix):
69 + # replace the prefix only once in case it's mirrored
70 + diff = diff.replace(prefix, '', 1)
71 +
72 + return diff
73 +
74 +
75 +def find_all_indices(string, substring):
76 + """
77 + Find all (starting) indices of a substring in a given string. Useful for
78 + tool call extraction
79 + """
80 + indices = []
81 + index = -1
82 + while True:
83 + index = string.find(substring, index + 1)
84 + if index == -1:
85 + break
86 + indices.append(index)
87 + return indices
vllm/model_executor/guided_decoding/__init__.py
+3 -2
@@ -59,8 +59,9 @@ def _adapt_request_for_tool_use(request: Union[CompletionRequest,
59 if type(request) is CompletionRequest:
60 return request
61
62 - # user has chosen to not use any tool
63 - if request.tool_choice == "none":
62 + # user has chosen to not use any tool,
63 + # OR is allowing the model to choose a tool.
64 + if request.tool_choice == "none" or request.tool_choice == "auto":
65 return request
66
67 # user has chosen to use a named tool
vllm/model_executor/guided_decoding/outlines_decoding.py
+23 -8
@@ -8,8 +8,9 @@ from typing import Tuple, Union
8 from pydantic import BaseModel
9 from transformers import PreTrainedTokenizerBase
10
11 -from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
12 - CompletionRequest)
11 +from vllm.entrypoints.openai.protocol import (
12 + ChatCompletionNamedToolChoiceParam, ChatCompletionRequest,
13 + CompletionRequest)
14 from vllm.model_executor.guided_decoding.guided_fields import (
15 GuidedDecodingRequest)
16 from vllm.model_executor.guided_decoding.outlines_logits_processors import (
@@ -101,16 +102,30 @@ def _get_guide_and_mode(
102 request: Union[CompletionRequest, ChatCompletionRequest,
103 GuidedDecodingRequest]
104 ) -> Union[Tuple[str, GuidedDecodingMode], Tuple[None, None]]:
105 + # if the request is a chat completion request, AND the tool choice is a
106 + # named tool choice, do guided decoding
107 + # using that tool as the JSON schema
108 + if isinstance(request, ChatCompletionRequest) and isinstance(
109 + request.tool_choice, ChatCompletionNamedToolChoiceParam):
110 + # Guided generation for tools/functions parameters
111 + if request.tool_choice.type == "function":
112 + for tool in request.tools:
113 + if (tool.type == "function" and tool.function.name
114 + == request.tool_choice.function.name):
115 + json = json_dumps(tool.function.parameters, sort_keys=True)
116 + return json, GuidedDecodingMode.JSON
117 + return None, None
118
105 - if request.guided_json:
106 - json = request.guided_json
107 - if isinstance(json, dict):
119 + elif request.guided_json:
120 + if isinstance(request.guided_json, dict):
121 # turn dict into hashable string
109 - json = json_dumps(json)
110 - elif isinstance(json, BaseModel):
122 + json = json_dumps(request.guided_json)
123 + elif isinstance(request.guided_json, BaseModel):
124 # use pydantic signature so that different model classes
125 # with the same fields will get hashed the same
113 - json = str(json.__signature__)
126 + json = str(request.guided_json.__signature__)
127 + else:
128 + json = request.guided_json
129 return json, GuidedDecodingMode.JSON
130 elif request.guided_regex:
131 return request.guided_regex, GuidedDecodingMode.REGEX