A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 07:00 UTC
[Frontend] Pythonic tool parser (#9859)
Signed-off-by: Mike Depinet <mike@fixie.ai>
Mike Depinet committed
Nov 13, 2024 at 20:14 UTC
f67ce05d0b826322f85403f1113f69ca3853aa39
10 files changed
+806
-26
docs/source/serving/openai_compatible_server.md
+52
-24
@@ -162,7 +162,7 @@ vllm serve <model> --chat-template ./path-to-chat-template.jinja
162
vLLM community provides a set of chat templates for popular models. You can find them in the examples
163
directory [here](https://github.com/vllm-project/vllm/tree/main/examples/)
164
165
-With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
165
+With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
166
both a `type` and a `text` field. An example is provided below:
167
```python
168
completion = client.chat.completions.create(
@@ -172,10 +172,10 @@ completion = client.chat.completions.create(
172
]
173
)
174
```
175
-Most chat templates for LLMs expect the `content` to be a `string` but there are some newer models like
175
+Most chat templates for LLMs expect the `content` to be a `string` but there are some newer models like
176
`meta-llama/Llama-Guard-3-1B` that expect the content to be parsed with the new OpenAI spec. In order to choose which
177
format the content needs to be parsed in by vLLM, please use the `--chat-template-text-format` argument to specify
178
-between `string` or `openai`. The default value is `string` and vLLM internally converts both spec formats to match
178
+between `string` or `openai`. The default value is `string` and vLLM internally converts both spec formats to match
179
this, unless explicitly specified.
180
181
@@ -191,8 +191,8 @@ this, unless explicitly specified.
191
### Config file
192
193
The `serve` module can also accept arguments from a config file in
194
-`yaml` format. The arguments in the yaml must be specified using the
195
-long form of the argument outlined [here](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#command-line-arguments-for-the-server):
194
+`yaml` format. The arguments in the yaml must be specified using the
195
+long form of the argument outlined [here](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#command-line-arguments-for-the-server):
196
197
For example:
198
@@ -208,7 +208,7 @@ uvicorn-log-level: "info"
208
$ vllm serve SOME_MODEL --config config.yaml
209
```
210
---
211
-**NOTE**
211
+**NOTE**
212
In case an argument is supplied simultaneously using command line and the config file, the value from the commandline will take precedence.
213
The order of priorities is `command line > config file values > defaults`.
214
@@ -222,30 +222,30 @@ Please see below for recommended configuration and chat templates to use when fu
222
223
224
### Named Function Calling
225
-vLLM supports named function calling in the chat completion API by default. It does so using Outlines, so this is
226
-enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
227
-high-quality one.
225
+vLLM supports named function calling in the chat completion API by default. It does so using Outlines, so this is
226
+enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
227
+high-quality one.
228
229
vLLM will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
230
231
-To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
232
-specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
231
+To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
232
+specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
233
234
235
### Automatic Function Calling
236
To enable this feature, you should set the following flags:
237
-* `--enable-auto-tool-choice` -- **mandatory** Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it
237
+* `--enable-auto-tool-choice` -- **mandatory** Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it
238
deems appropriate.
239
-* `--tool-call-parser` -- select the tool parser to use (listed below). Additional tool parsers
239
+* `--tool-call-parser` -- select the tool parser to use (listed below). Additional tool parsers
240
will continue to be added in the future, and also can register your own tool parsers in the `--tool-parser-plugin`.
241
* `--tool-parser-plugin` -- **optional** tool parser plugin used to register user defined tool parsers into vllm, the registered tool parser name can be specified in `--tool-call-parser`.
242
-* `--chat-template` -- **optional** for auto tool choice. the path to the chat template which handles `tool`-role messages and `assistant`-role messages
243
-that contain previously generated tool calls. Hermes, Mistral and Llama models have tool-compatible chat templates in their
244
-`tokenizer_config.json` files, but you can specify a custom template. This argument can be set to `tool_use` if your model has a tool use-specific chat
242
+* `--chat-template` -- **optional** for auto tool choice. the path to the chat template which handles `tool`-role messages and `assistant`-role messages
243
+that contain previously generated tool calls. Hermes, Mistral and Llama models have tool-compatible chat templates in their
244
+`tokenizer_config.json` files, but you can specify a custom template. This argument can be set to `tool_use` if your model has a tool use-specific chat
245
template configured in the `tokenizer_config.json`. In this case, it will be used per the `transformers` specification. More on this [here](https://huggingface.co/docs/transformers/en/chat_templating#why-do-some-models-have-multiple-templates)
246
from HuggingFace; and you can find an example of this in a `tokenizer_config.json` [here](https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/blob/main/tokenizer_config.json)
247
248
-If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
248
+If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
249
250
251
#### Hermes Models (`hermes`)
@@ -256,8 +256,8 @@ All Nous Research Hermes-series models newer than Hermes 2 Pro should be support
256
* `NousResearch/Hermes-3-*`
257
258
259
-_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality & capabilities due to the merge
260
-step in their creation_.
259
+_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality & capabilities due to the merge
260
+step in their creation_.
261
262
Flags: `--tool-call-parser hermes`
263
@@ -269,9 +269,9 @@ Supported models:
269
* Additional mistral function-calling models are compatible as well.
270
271
Known issues:
272
-1. Mistral 7B struggles to generate parallel tool calls correctly.
273
-2. Mistral's `tokenizer_config.json` chat template requires tool call IDs that are exactly 9 digits, which is
274
-much shorter than what vLLM generates. Since an exception is thrown when this condition
272
+1. Mistral 7B struggles to generate parallel tool calls correctly.
273
+2. Mistral's `tokenizer_config.json` chat template requires tool call IDs that are exactly 9 digits, which is
274
+much shorter than what vLLM generates. Since an exception is thrown when this condition
275
is not met, the following additional chat templates are provided:
276
277
* `examples/tool_chat_template_mistral.jinja` - this is the "official" Mistral chat template, but tweaked so that
@@ -291,11 +291,11 @@ Supported models:
291
* `meta-llama/Meta-Llama-3.1-405B-Instruct`
292
* `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8`
293
294
-The tool calling that is supported is the [JSON based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling).
294
+The tool calling that is supported is the [JSON based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) in Llama-3.2 models, see the `pythonic` tool parser below.
295
Other tool calling formats like the built in python tool calling or custom tool calling are not supported.
296
297
Known issues:
298
-1. Parallel tool calls are not supported.
298
+1. Parallel tool calls are not supported.
299
2. The model can generate parameters with a wrong format, such as generating
300
an array serialized as string instead of an array.
301
@@ -341,6 +341,34 @@ AI21's Jamba-1.5 models are supported.
341
Flags: `--tool-call-parser jamba`
342
343
344
+#### Models with Pythonic Tool Calls (`pythonic`)
345
+
346
+A growing number of models output a python list to represent tool calls instead of using JSON. This has the advantage of inherently supporting parallel tool calls and removing ambiguity around the JSON schema required for tool calls. The `pythonic` tool parser can support such models.
347
+
348
+As a concrete example, these models may look up the weather in San Francisco and Seattle by generating:
349
+```python
350
+[get_weather(city='San Francisco', metric='celsius'), get_weather(city='Seattle', metric='celsius')]
351
+```
352
+
353
+Limitations:
354
+* The model must not generate both text and tool calls in the same generation. This may not be hard to change for a specific model, but the community currently lacks consensus on which tokens to emit when starting and ending tool calls. (In particular, the Llama 3.2 models emit no such tokens.)
355
+* Llama's smaller models struggle to use tools effectively.
356
+
357
+Example supported models:
358
+* `meta-llama/Llama-3.2-1B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)
359
+* `meta-llama/Llama-3.2-3B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)
360
+* `Team-ACE/ToolACE-8B` (use with `examples/tool_chat_template_toolace.jinja`)
361
+* `fixie-ai/ultravox-v0_4-ToolACE-8B` (use with `examples/tool_chat_template_toolace.jinja`)
362
+
363
+Flags: `--tool-call-parser pythonic --chat-template {see_above}`
364
+
365
+---
366
+**WARNING**
367
+Llama's smaller models frequently fail to emit tool calls in the correct format. Your mileage may vary.
368
+
369
+---
370
+
371
+
372
### How to write a tool parser plugin
373
374
A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py.
examples/tool_chat_template_llama3.2_pythonic.jinja
new
+98
@@ -0,0 +1,98 @@
1
+{{- bos_token }}
2
+{%- if custom_tools is defined %}
3
+ {%- set tools = custom_tools %}
4
+{%- endif %}
5
+{%- if not tools_in_user_message is defined %}
6
+ {%- set tools_in_user_message = false %}
7
+{%- endif %}
8
+{%- if not date_string is defined %}
9
+ {%- if strftime_now is defined %}
10
+ {%- set date_string = strftime_now("%d %b %Y") %}
11
+ {%- else %}
12
+ {%- set date_string = "26 Jul 2024" %}
13
+ {%- endif %}
14
+{%- endif %}
15
+{%- if not tools is defined %}
16
+ {%- set tools = none %}
17
+{%- endif %}
18
+
19
+{#- This block extracts the system message, so we can slot it into the right place. #}
20
+{%- if messages[0]['role'] == 'system' %}
21
+ {%- set system_message = messages[0]['content']|trim %}
22
+ {%- set messages = messages[1:] %}
23
+{%- else %}
24
+ {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language. When you receive a tool call response, use the output to format an answer to the original user question." %}
25
+{%- endif %}
26
+
27
+{#- System message #}
28
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
29
+{%- if tools is not none %}
30
+ {{- "Environment: ipython\n" }}
31
+{%- endif %}
32
+{{- "Cutting Knowledge Date: December 2023\n" }}
33
+{{- "Today Date: " + date_string + "\n\n" }}
34
+{%- if tools is not none and not tools_in_user_message %}
35
+ {{- "You have access to the following functions. To call functions, please respond with a python list of the calls. " }}
36
+ {{- 'Respond in the format [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] ' }}
37
+ {{- "Do not use variables.\n\n" }}
38
+ {%- for t in tools %}
39
+ {{- t | tojson(indent=4) }}
40
+ {{- "\n\n" }}
41
+ {%- endfor %}
42
+{%- endif %}
43
+{{- system_message }}
44
+{{- "<|eot_id|>" }}
45
+
46
+{#- Custom tools are passed in a user message with some extra guidance #}
47
+{%- if tools_in_user_message and not tools is none %}
48
+ {#- Extract the first user message so we can plug it in here #}
49
+ {%- if messages | length != 0 %}
50
+ {%- set first_user_message = messages[0]['content']|trim %}
51
+ {%- set messages = messages[1:] %}
52
+ {%- else %}
53
+ {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
54
+ {%- endif %}
55
+ {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
56
+ {{- "Given the following functions, please respond with a python list for function calls " }}
57
+ {{- "with their proper arguments to best answer the given prompt.\n\n" }}
58
+ {{- 'Respond in the format [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] ' }}
59
+ {{- "Do not use variables.\n\n" }}
60
+ {%- for t in tools %}
61
+ {{- t | tojson(indent=4) }}
62
+ {{- "\n\n" }}
63
+ {%- endfor %}
64
+ {{- first_user_message + "<|eot_id|>"}}
65
+{%- endif %}
66
+
67
+{%- for message in messages %}
68
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
69
+ {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
70
+ {%- elif 'tool_calls' in message %}
71
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n[' -}}
72
+ {%- for tool_call in message.tool_calls %}
73
+ {%- if tool_call.function is defined %}
74
+ {%- set tool_call = tool_call.function %}
75
+ {%- endif %}
76
+ {{- tool_call.name + '(' -}}
77
+ {%- for param in tool_call.arguments %}
78
+ {{- param + '=' -}}
79
+ {{- "%sr" | format(tool_call.arguments[param]) -}}
80
+ {% if not loop.last %}, {% endif %}
81
+ {%- endfor %}
82
+ {{- ')' -}}
83
+ {% if not loop.last %}, {% endif %}
84
+ {%- endfor %}
85
+ {{- ']<|eot_id|>' -}}
86
+ {%- elif message.role == "tool" or message.role == "ipython" %}
87
+ {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
88
+ {%- if message.content is mapping %}
89
+ {{- message.content | tojson }}
90
+ {%- else %}
91
+ {{- { "output": message.content } | tojson }}
92
+ {%- endif %}
93
+ {{- "<|eot_id|>" }}
94
+ {%- endif %}
95
+{%- endfor %}
96
+{%- if add_generation_prompt %}
97
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
98
+{%- endif %}
examples/tool_chat_template_toolace.jinja
new
+65
@@ -0,0 +1,65 @@
1
+{{- bos_token }}
2
+
3
+{%- if custom_tools is defined %}
4
+ {%- set tools = custom_tools %}
5
+{%- endif %}
6
+{%- if not tools is defined %}
7
+ {%- set tools = none %}
8
+{%- endif %}
9
+
10
+{#- This block extracts the system message, so we can slot it into the right place. #}
11
+{%- if messages[0]['role'] == 'system' %}
12
+ {%- set system_message = messages[0]['content']|trim %}
13
+ {%- set messages = messages[1:] %}
14
+{%- else %}
15
+ {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language." %}
16
+{%- endif %}
17
+
18
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
19
+{%- if tools is not none and not tools_in_user_message %}
20
+ {{- "You are an expert in composing functions. You are given a question and a set of possible functions. Based on the question, you will need to make one or more function/tool calls to achieve the purpose.\n" }}
21
+ {{- "If none of the function can be used, point it out. If the given question lacks the parameters required by the function, also point it out.\n" }}
22
+ {{- "You should only return the function call in tools call sections.\n\n" }}
23
+ {{- "If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)]\n" }}
24
+ {{- "You SHOULD NOT include any other text in the response.\n" }}
25
+ {{- "Here is a list of functions in JSON format that you can invoke.\n" }}
26
+ {%- for t in tools %}
27
+ {{- t | tojson(indent=4) }}
28
+ {{- "\n\n" }}
29
+ {%- endfor %}
30
+ {{- "\n" }}
31
+{%- endif %}
32
+{{- system_message }}
33
+{{- "<|eot_id|>" }}
34
+
35
+{%- for message in messages %}
36
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
37
+ {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
38
+ {%- elif 'tool_calls' in message %}
39
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n[' -}}
40
+ {%- for tool_call in message.tool_calls %}
41
+ {%- if tool_call.function is defined %}
42
+ {%- set tool_call = tool_call.function %}
43
+ {%- endif %}
44
+ {{- tool_call.name + '(' -}}
45
+ {%- for param in tool_call.arguments %}
46
+ {{- param + '=' -}}
47
+ {{- "%sr" | format(tool_call.arguments[param]) -}}
48
+ {% if not loop.last %}, {% endif %}
49
+ {%- endfor %}
50
+ {{- ')' -}}
51
+ {% if not loop.last %}, {% endif %}
52
+ {%- endfor %}
53
+ {{- ']<|eot_id|>' -}}
54
+ {%- elif message.role == "tool" or message.role == "ipython" %}
55
+ {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
56
+ {%- if message.content is mapping %}
57
+ {{- message.content | tojson }}
58
+ {%- else %}
59
+ {{- { "output": message.content } | tojson }}
60
+ {%- endif %}
61
+ {{- "<|eot_id|>" }}
62
+ {%- endif %}
63
+{%- endfor %}
64
+
65
+{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
tests/entrypoints/openai/tool_parsers/__init__.py
tests/entrypoints/openai/tool_parsers/test_pythonic_tool_parser.py
new
+160
@@ -0,0 +1,160 @@
1
+from typing import List
2
+from unittest.mock import MagicMock
3
+
4
+import pytest
5
+
6
+from tests.entrypoints.openai.tool_parsers.utils import (
7
+ run_tool_extraction, run_tool_extraction_streaming)
8
+from vllm.entrypoints.openai.protocol import FunctionCall
9
+from vllm.entrypoints.openai.tool_parsers import ToolParser, ToolParserManager
10
+
11
+# https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#model-response-format-1
12
+SIMPLE_FUNCTION_OUTPUT = "get_weather(city='San Francisco', metric='celsius')"
13
+SIMPLE_FUNCTION_CALL = FunctionCall(
14
+ name="get_weather",
15
+ arguments='{"city": "San Francisco", "metric": "celsius"}',
16
+)
17
+MORE_TYPES_FUNCTION_OUTPUT = (
18
+ "register_user(name='John Doe', "
19
+ "age=37, "
20
+ "address={'city': 'San Francisco', 'state': 'CA'}, "
21
+ "role=None, "
22
+ "passed_test=True, "
23
+ "aliases=['John', 'Johnny'])")
24
+MORE_TYPES_FUNCTION_CALL = FunctionCall(
25
+ name="register_user",
26
+ arguments='{"name": "John Doe", '
27
+ '"age": 37, '
28
+ '"address": {"city": "San Francisco", "state": "CA"}, '
29
+ '"role": null, '
30
+ '"passed_test": true, '
31
+ '"aliases": ["John", "Johnny"]}',
32
+)
33
+PARAMETERLESS_FUNCTION_OUTPUT = "get_weather()"
34
+PARAMETERLESS_FUNCTION_CALL = FunctionCall(
35
+ name="get_weather",
36
+ arguments='{}',
37
+)
38
+EMPTY_DICT_FUNCTION_OUTPUT = "do_something_cool(additional_data={})"
39
+EMPTY_DICT_FUNCTION_CALL = FunctionCall(
40
+ name="do_something_cool",
41
+ arguments='{"additional_data": {}}',
42
+)
43
+EMPTY_LIST_FUNCTION_OUTPUT = "do_something_cool(steps=[])"
44
+EMPTY_LIST_FUNCTION_CALL = FunctionCall(
45
+ name="do_something_cool",
46
+ arguments='{"steps": []}',
47
+)
48
+ESCAPED_STRING_FUNCTION_OUTPUT = (
49
+ r"get_weather(city='Martha\'s Vineyard', metric='\"cool units\"')")
50
+ESCAPED_STRING_FUNCTION_CALL = FunctionCall(
51
+ name="get_weather",
52
+ arguments='{"city": "Martha\'s Vineyard", "metric": "\\"cool units\\""}',
53
+)
54
+
55
+
56
+@pytest.mark.parametrize("streaming", [True, False])
57
+def test_no_tool_call(streaming: bool):
58
+ mock_tokenizer = MagicMock()
59
+ tool_parser: ToolParser = ToolParserManager.get_tool_parser("pythonic")(
60
+ mock_tokenizer)
61
+ model_output = "How can I help you today?"
62
+
63
+ content, tool_calls = run_tool_extraction(tool_parser,
64
+ model_output,
65
+ streaming=streaming)
66
+
67
+ assert content == model_output
68
+ assert len(tool_calls) == 0
69
+
70
+
71
+TEST_CASES = [
72
+ pytest.param(True,
73
+ f"[{SIMPLE_FUNCTION_OUTPUT}]", [SIMPLE_FUNCTION_CALL],
74
+ id="simple_streaming"),
75
+ pytest.param(False,
76
+ f"[{SIMPLE_FUNCTION_OUTPUT}]", [SIMPLE_FUNCTION_CALL],
77
+ id="simple_nonstreaming"),
78
+ pytest.param(True,
79
+ f"[{MORE_TYPES_FUNCTION_OUTPUT}]", [MORE_TYPES_FUNCTION_CALL],
80
+ id="more_types_streaming"),
81
+ pytest.param(False,
82
+ f"[{MORE_TYPES_FUNCTION_OUTPUT}]", [MORE_TYPES_FUNCTION_CALL],
83
+ id="more_types_nonstreaming"),
84
+ pytest.param(True,
85
+ f"[{PARAMETERLESS_FUNCTION_OUTPUT}]",
86
+ [PARAMETERLESS_FUNCTION_CALL],
87
+ id="parameterless_streaming"),
88
+ pytest.param(False,
89
+ f"[{PARAMETERLESS_FUNCTION_OUTPUT}]",
90
+ [PARAMETERLESS_FUNCTION_CALL],
91
+ id="parameterless_nonstreaming"),
92
+ pytest.param(True,
93
+ f"[{EMPTY_DICT_FUNCTION_OUTPUT}]", [EMPTY_DICT_FUNCTION_CALL],
94
+ id="empty_dict_streaming"),
95
+ pytest.param(False,
96
+ f"[{EMPTY_DICT_FUNCTION_OUTPUT}]", [EMPTY_DICT_FUNCTION_CALL],
97
+ id="empty_dict_nonstreaming"),
98
+ pytest.param(True,
99
+ f"[{EMPTY_LIST_FUNCTION_OUTPUT}]", [EMPTY_LIST_FUNCTION_CALL],
100
+ id="empty_list_streaming"),
101
+ pytest.param(False,
102
+ f"[{EMPTY_LIST_FUNCTION_OUTPUT}]", [EMPTY_LIST_FUNCTION_CALL],
103
+ id="empty_list_nonstreaming"),
104
+ pytest.param(True,
105
+ f"[{ESCAPED_STRING_FUNCTION_OUTPUT}]",
106
+ [ESCAPED_STRING_FUNCTION_CALL],
107
+ id="escaped_string_streaming"),
108
+ pytest.param(False,
109
+ f"[{ESCAPED_STRING_FUNCTION_OUTPUT}]",
110
+ [ESCAPED_STRING_FUNCTION_CALL],
111
+ id="escaped_string_nonstreaming"),
112
+ pytest.param(True,
113
+ f"[{SIMPLE_FUNCTION_OUTPUT}, {MORE_TYPES_FUNCTION_OUTPUT}]",
114
+ [SIMPLE_FUNCTION_CALL, MORE_TYPES_FUNCTION_CALL],
115
+ id="parallel_calls_streaming"),
116
+ pytest.param(False,
117
+ f"[{SIMPLE_FUNCTION_OUTPUT}, {MORE_TYPES_FUNCTION_OUTPUT}]",
118
+ [SIMPLE_FUNCTION_CALL, MORE_TYPES_FUNCTION_CALL],
119
+ id="parallel_calls_nonstreaming"),
120
+]
121
+
122
+
123
+@pytest.mark.parametrize("streaming, model_output, expected_tool_calls",
124
+ TEST_CASES)
125
+def test_tool_call(streaming: bool, model_output: str,
126
+ expected_tool_calls: List[FunctionCall]):
127
+ mock_tokenizer = MagicMock()
128
+ tool_parser: ToolParser = ToolParserManager.get_tool_parser("pythonic")(
129
+ mock_tokenizer)
130
+
131
+ content, tool_calls = run_tool_extraction(tool_parser,
132
+ model_output,
133
+ streaming=streaming)
134
+
135
+ assert content is None
136
+ assert len(tool_calls) == len(expected_tool_calls)
137
+ for actual, expected in zip(tool_calls, expected_tool_calls):
138
+ assert actual.type == "function"
139
+ assert actual.function == expected
140
+
141
+
142
+def test_streaming_tool_call_with_large_steps():
143
+ mock_tokenizer = MagicMock()
144
+ tool_parser: ToolParser = ToolParserManager.get_tool_parser("pythonic")(
145
+ mock_tokenizer)
146
+ model_output_deltas = [
147
+ "[get_weather(city='San",
148
+ " Francisco', metric='celsius'), "
149
+ f"{PARAMETERLESS_FUNCTION_OUTPUT}, "
150
+ f"{EMPTY_LIST_FUNCTION_OUTPUT}]",
151
+ ]
152
+
153
+ reconstructor = run_tool_extraction_streaming(
154
+ tool_parser, model_output_deltas, assert_one_tool_per_delta=False)
155
+
156
+ assert reconstructor.other_content == ""
157
+ assert len(reconstructor.tool_calls) == 3
158
+ assert reconstructor.tool_calls[0].function == SIMPLE_FUNCTION_CALL
159
+ assert reconstructor.tool_calls[1].function == PARAMETERLESS_FUNCTION_CALL
160
+ assert reconstructor.tool_calls[2].function == EMPTY_LIST_FUNCTION_CALL
tests/entrypoints/openai/tool_parsers/utils.py
new
+123
@@ -0,0 +1,123 @@
1
+from typing import Iterable, List, Tuple, Union
2
+
3
+from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
4
+ DeltaMessage,
5
+ ExtractedToolCallInformation,
6
+ FunctionCall, ToolCall)
7
+from vllm.entrypoints.openai.tool_parsers import ToolParser
8
+
9
+
10
+class StreamingToolReconstructor:
11
+
12
+ def __init__(self, assert_one_tool_per_delta: bool = True):
13
+ self.tool_calls: List[ToolCall] = []
14
+ self.other_content: str = ""
15
+ self._assert_one_tool_per_delta = assert_one_tool_per_delta
16
+
17
+ def append_delta(self, delta: DeltaMessage):
18
+ if delta.content is not None:
19
+ self.other_content += delta.content
20
+ else:
21
+ assert delta.tool_calls, (
22
+ "Streaming results should have either content or tool calls "
23
+ "(or both)")
24
+ if self._assert_one_tool_per_delta:
25
+ # Note: This isn't strictly required by the API and may not be
26
+ # possible to adhere to depending on the token space and number of
27
+ # tokens per streamed response from the model, but it is required
28
+ # by tool_use tests, so we enforce it here by default also.
29
+ assert len(delta.tool_calls) < 2, (
30
+ "Streaming should include only one tool call per update.")
31
+ for call_delta in delta.tool_calls:
32
+ assert call_delta.type == "function", (
33
+ "Streaming tool calls should only emit function calls. Got "
34
+ f"{call_delta.type}")
35
+ current_tool_call = self.tool_calls[
36
+ call_delta.index] if call_delta.index < len(
37
+ self.tool_calls) else None
38
+ if current_tool_call:
39
+ assert (not call_delta.function.name), (
40
+ "Streaming tool calls should emit the full function name "
41
+ f"exactly once. Got {call_delta.function.name}")
42
+ assert (not call_delta.id), (
43
+ "Streaming tool calls must emit function id only once. Got "
44
+ f"{call_delta.id}")
45
+ assert (call_delta.index == len(self.tool_calls) - 1), (
46
+ f"Incorrect index for tool delta. Got {call_delta.index}, "
47
+ f"expected {len(self.tool_calls) - 1}")
48
+ current_tool_call.function.arguments += (
49
+ call_delta.function.arguments)
50
+ else:
51
+ assert call_delta.id is not None, (
52
+ "Streaming tool calls must have an id on first appearance")
53
+ assert call_delta.function.name is not None, (
54
+ "Streaming tool calls must have a function name on first "
55
+ "appearance")
56
+ assert call_delta.index == len(self.tool_calls), (
57
+ f"Incorrect index for tool delta. Got {call_delta.index}, "
58
+ f"expected {len(self.tool_calls)}")
59
+ self.tool_calls.append(
60
+ ToolCall(id=call_delta.id,
61
+ function=FunctionCall(
62
+ name=call_delta.function.name,
63
+ arguments=call_delta.function.arguments
64
+ or "")))
65
+
66
+
67
+def run_tool_extraction(
68
+ tool_parser: ToolParser,
69
+ model_output: str,
70
+ request: Union[ChatCompletionRequest, None] = None,
71
+ streaming: bool = False,
72
+ assert_one_tool_per_delta: bool = True,
73
+) -> Tuple[Union[str, None], List[ToolCall]]:
74
+ if streaming:
75
+ reconstructor = run_tool_extraction_streaming(
76
+ tool_parser,
77
+ model_output,
78
+ request,
79
+ assert_one_tool_per_delta=assert_one_tool_per_delta)
80
+ return reconstructor.other_content or None, reconstructor.tool_calls
81
+ else:
82
+ extracted = run_tool_extraction_nonstreaming(tool_parser, model_output,
83
+ request)
84
+ assert extracted.tools_called == bool(extracted.tool_calls)
85
+ return extracted.content, extracted.tool_calls
86
+
87
+
88
+def run_tool_extraction_nonstreaming(
89
+ tool_parser: ToolParser,
90
+ model_output: str,
91
+ request: Union[ChatCompletionRequest, None] = None
92
+) -> ExtractedToolCallInformation:
93
+ request = request or ChatCompletionRequest(messages=[], model="test-model")
94
+ return tool_parser.extract_tool_calls(model_output, request)
95
+
96
+
97
+def run_tool_extraction_streaming(
98
+ tool_parser: ToolParser,
99
+ model_deltas: Iterable[str],
100
+ request: Union[ChatCompletionRequest, None] = None,
101
+ assert_one_tool_per_delta: bool = True,
102
+) -> StreamingToolReconstructor:
103
+ request = request or ChatCompletionRequest(messages=[], model="test-model")
104
+ reconstructor = StreamingToolReconstructor(
105
+ assert_one_tool_per_delta=assert_one_tool_per_delta)
106
+ previous_text = ""
107
+ previous_tokens: List[int] = []
108
+ for delta in model_deltas:
109
+ token_delta = [
110
+ tool_parser.vocab.get(token)
111
+ for token in tool_parser.model_tokenizer.tokenize(delta)
112
+ if token in tool_parser.vocab
113
+ ]
114
+ current_text = previous_text + delta
115
+ current_tokens = previous_tokens + token_delta
116
+ delta_message = tool_parser.extract_tool_calls_streaming(
117
+ previous_text, current_text, delta, previous_tokens,
118
+ current_tokens, token_delta, request)
119
+ if delta_message is not None:
120
+ reconstructor.append_delta(delta_message)
121
+ previous_text = current_text
122
+ previous_tokens = current_tokens
123
+ return reconstructor
tests/tool_use/utils.py
+11
-1
@@ -122,7 +122,17 @@ CONFIGS: Dict[str, ServerConfig] = {
122
],
123
"supports_parallel":
124
False,
125
- }
125
+ },
126
+ "toolACE": {
127
+ "model":
128
+ "Team-ACE/ToolACE-8B",
129
+ "arguments": [
130
+ "--tool-call-parser", "pythonic", "--chat-template",
131
+ str(VLLM_PATH / "examples/tool_chat_template_toolace.jinja")
132
+ ],
133
+ "supports_parallel":
134
+ True,
135
+ },
136
}
137
138
WEATHER_TOOL: ChatCompletionToolParam = {
vllm/entrypoints/openai/serving_chat.py
+5
@@ -74,6 +74,11 @@ class OpenAIServingChat(OpenAIServing):
74
self.tool_parser: Optional[Callable[[AnyTokenizer], ToolParser]] = None
75
if self.enable_auto_tools:
76
try:
77
+ if (tool_parser == "pythonic" and
78
+ model_config.model.startswith("meta-llama/Llama-3.2")):
79
+ logger.warning(
80
+ "Llama3.2 models may struggle to emit valid pythonic"
81
+ " tool calls")
82
self.tool_parser = ToolParserManager.get_tool_parser(
83
tool_parser)
84
except Exception as e:
vllm/entrypoints/openai/tool_parsers/__init__.py
+3
-1
@@ -6,9 +6,11 @@ from .internlm2_tool_parser import Internlm2ToolParser
6
from .jamba_tool_parser import JambaToolParser
7
from .llama_tool_parser import Llama3JsonToolParser
8
from .mistral_tool_parser import MistralToolParser
9
+from .pythonic_tool_parser import PythonicToolParser
10
11
__all__ = [
12
"ToolParser", "ToolParserManager", "Granite20bFCToolParser",
13
"GraniteToolParser", "Hermes2ProToolParser", "MistralToolParser",
13
- "Internlm2ToolParser", "Llama3JsonToolParser", "JambaToolParser"
14
+ "Internlm2ToolParser", "Llama3JsonToolParser", "JambaToolParser",
15
+ "PythonicToolParser"
16
]
vllm/entrypoints/openai/tool_parsers/pythonic_tool_parser.py
new
+289
@@ -0,0 +1,289 @@
1
+import ast
2
+import json
3
+import re
4
+from typing import Any, Sequence, Tuple, Union
5
+
6
+from transformers import PreTrainedTokenizerBase
7
+
8
+from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
9
+ DeltaFunctionCall, DeltaMessage,
10
+ DeltaToolCall,
11
+ ExtractedToolCallInformation,
12
+ FunctionCall, ToolCall)
13
+from vllm.entrypoints.openai.tool_parsers.abstract_tool_parser import (
14
+ ToolParser, ToolParserManager)
15
+from vllm.logger import init_logger
16
+
17
+logger = init_logger(__name__)
18
+
19
+
20
+class _UnexpectedAstError(Exception):
21
+ pass
22
+
23
+
24
+@ToolParserManager.register_module("pythonic")
25
+class PythonicToolParser(ToolParser):
26
+ """
27
+ Tool call parser for models that produce tool calls in a pythonic style,
28
+ such as Llama 3.2 models.
29
+
30
+ Used when --enable-auto-tool-choice --tool-call-parser pythonic are all set
31
+ """
32
+ # TODO(mdepinet): Possible future improvements:
33
+ # 1. Support text + tools separated by either <|python_tag|> or \n\n
34
+ # 2. Support tools outside of a list (or separated by a semicolon).
35
+ # This depends on item 1 for consistent streaming.
36
+ # Neither of these are necessary for e.g. ToolACE, but both would help make
37
+ # Llama3.2 models more reliable.
38
+
39
+ TOOL_CALL_REGEX = re.compile(
40
+ r"\[([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s)?\),\s*)*([a-zA-Z]+\w*\(([a-zA-Z]+\w*=.*,\s*)*([a-zA-Z]+\w*=.*\s*)?\)\s*)+\]",
41
+ re.DOTALL)
42
+
43
+ def __init__(self, tokenizer: PreTrainedTokenizerBase):
44
+ super().__init__(tokenizer)
45
+
46
+ # Rename for readability. This is NOT a tool id.
47
+ @property
48
+ def current_tool_index(self) -> int:
49
+ return self.current_tool_id
50
+
51
+ @current_tool_index.setter
52
+ def current_tool_index(self, value: int) -> None:
53
+ self.current_tool_id = value
54
+
55
+ def extract_tool_calls(
56
+ self, model_output: str,
57
+ request: ChatCompletionRequest) -> ExtractedToolCallInformation:
58
+ """
59
+ Extract the tool calls from a complete model response.
60
+ """
61
+
62
+ if not (self.TOOL_CALL_REGEX.match(model_output)):
63
+ return ExtractedToolCallInformation(tools_called=False,
64
+ tool_calls=[],
65
+ content=model_output)
66
+
67
+ try:
68
+ module = ast.parse(model_output)
69
+ parsed = getattr(module.body[0], "value", None)
70
+ if isinstance(parsed, ast.List) and all(
71
+ isinstance(e, ast.Call) for e in parsed.elts):
72
+ return ExtractedToolCallInformation(
73
+ tools_called=True,
74
+ tool_calls=[
75
+ _handle_single_tool(e) # type: ignore
76
+ for e in parsed.elts
77
+ ],
78
+ content=None)
79
+ else:
80
+ raise _UnexpectedAstError(
81
+ "Tool output must be a list of function calls")
82
+ except Exception:
83
+ logger.exception("Error in extracting tool call from response.")
84
+ # Treat as regular text
85
+ return ExtractedToolCallInformation(tools_called=False,
86
+ tool_calls=[],
87
+ content=model_output)
88
+
89
+ def extract_tool_calls_streaming(
90
+ self,
91
+ previous_text: str,
92
+ current_text: str,
93
+ delta_text: str,
94
+ previous_token_ids: Sequence[int],
95
+ current_token_ids: Sequence[int],
96
+ delta_token_ids: Sequence[int],
97
+ request: ChatCompletionRequest,
98
+ ) -> Union[DeltaMessage, None]:
99
+
100
+ if not current_text.startswith("["):
101
+ return DeltaMessage(content=delta_text)
102
+
103
+ try:
104
+ valid_and_added_text = _make_valid_python(current_text)
105
+ if valid_and_added_text is None:
106
+ return None
107
+ valid_text, added_text = valid_and_added_text
108
+
109
+ module = ast.parse(valid_text)
110
+ parsed = getattr(module.body[0], "value", None)
111
+ if not isinstance(parsed, ast.List) or not all(
112
+ isinstance(e, ast.Call) for e in parsed.elts):
113
+ raise _UnexpectedAstError(
114
+ "Tool output must be a list of function calls")
115
+ tool_calls = [
116
+ _handle_single_tool(e) # type: ignore
117
+ for e in parsed.elts
118
+ ]
119
+
120
+ tool_deltas = []
121
+ for index, new_call in enumerate(tool_calls):
122
+ if index < self.current_tool_index:
123
+ continue
124
+
125
+ self.current_tool_index = index
126
+ if len(self.streamed_args_for_tool) == index:
127
+ self.streamed_args_for_tool.append("")
128
+
129
+ new_call_complete = index < len(
130
+ tool_calls) - 1 or ")]" not in added_text
131
+ if new_call_complete:
132
+ self.current_tool_index += 1
133
+
134
+ withheld_suffix = (added_text[:-2]
135
+ if not new_call_complete else "")
136
+ if not new_call_complete and added_text[-2] == ")":
137
+ # Function call is incomplete. Withhold the closing bracket.
138
+ withheld_suffix = withheld_suffix + "}"
139
+ # Strings get single quotes in the model-produced string.
140
+ # JSON requires double quotes.
141
+ withheld_suffix = withheld_suffix.replace("'", '"')
142
+ delta = _compute_tool_delta(self.streamed_args_for_tool[index],
143
+ new_call, index, withheld_suffix)
144
+
145
+ if delta is not None:
146
+ tool_deltas.append(delta)
147
+ if (delta.function is not None
148
+ and delta.function.arguments is not None):
149
+ self.streamed_args_for_tool[
150
+ index] += delta.function.arguments
151
+
152
+ # HACK: serving_chat.py inspects the internal state of tool parsers
153
+ # when determining it's final streaming delta, automatically
154
+ # adding autocompleted JSON.
155
+ # These two lines avoid that nonsense while ensuring finish_reason
156
+ # is set to tool_calls when at least one tool is called.
157
+ if tool_deltas and not self.prev_tool_call_arr:
158
+ self.prev_tool_call_arr = [{"arguments": {}}]
159
+
160
+ if tool_deltas:
161
+ return DeltaMessage(tool_calls=tool_deltas)
162
+ elif not added_text and self.current_tool_id > 0:
163
+ # Return an empty DeltaMessage once the tool calls are all done
164
+ # so that finish_reason gets set.
165
+ return DeltaMessage(content='')
166
+ else:
167
+ return None
168
+ except Exception:
169
+ logger.exception("Error trying to handle streaming tool call.")
170
+ logger.debug(
171
+ "Skipping chunk as a result of tool streaming extraction "
172
+ "error")
173
+ return None
174
+
175
+
176
+def _get_parameter_value(val: ast.expr) -> Any:
177
+ if isinstance(val, ast.Constant):
178
+ return val.value
179
+ elif isinstance(val, ast.Dict):
180
+ if not all(isinstance(k, ast.Constant) for k in val.keys):
181
+ raise _UnexpectedAstError(
182
+ "Dict tool call arguments must have literal keys")
183
+ return {
184
+ k.value: _get_parameter_value(v) # type: ignore
185
+ for k, v in zip(val.keys, val.values)
186
+ }
187
+ elif isinstance(val, ast.List):
188
+ return [_get_parameter_value(v) for v in val.elts]
189
+ else:
190
+ raise _UnexpectedAstError("Tool call arguments must be literals")
191
+
192
+
193
+def _handle_single_tool(call: ast.Call) -> ToolCall:
194
+ if not isinstance(call.func, ast.Name):
195
+ raise _UnexpectedAstError("Invalid tool call name")
196
+ function_name = call.func.id
197
+ arguments = {}
198
+ for keyword in call.keywords:
199
+ arguments[keyword.arg] = _get_parameter_value(keyword.value)
200
+ return ToolCall(type="function",
201
+ function=FunctionCall(name=function_name,
202
+ arguments=json.dumps(arguments)))
203
+
204
+
205
+def _make_valid_python(text: str) -> Union[Tuple[str, str], None]:
206
+ bracket_stack = []
207
+ for index, char in enumerate(text):
208
+ if char in {"[", "(", "{"}:
209
+ bracket_stack.append(char)
210
+ elif char == "]":
211
+ if not bracket_stack or bracket_stack.pop() != "[":
212
+ raise _UnexpectedAstError("Mismatched square brackets")
213
+ elif char == ")":
214
+ if not bracket_stack or bracket_stack.pop() != "(":
215
+ raise _UnexpectedAstError("Mismatched parentheses")
216
+ elif char == "}":
217
+ if not bracket_stack or bracket_stack.pop() != "{":
218
+ raise _UnexpectedAstError("Mismatched curly braces")
219
+ elif char in {"'", '"'}:
220
+ if bracket_stack and bracket_stack[-1] == char:
221
+ if index > 0 and text[index - 1] == "\\":
222
+ # Treat an escaped quote as a regular character
223
+ pass
224
+ else:
225
+ bracket_stack.pop()
226
+ elif bracket_stack and bracket_stack[-1] in {"'", '"'}:
227
+ # Double quote within a single quote string or vice versa.
228
+ pass
229
+ else:
230
+ bracket_stack.append(char)
231
+
232
+ text = text.rstrip()
233
+ if text.endswith("=") or text.endswith(":"):
234
+ # Since we have no type information for this property/parameter value,
235
+ # we can't fill in a valid value.
236
+ return None
237
+ if bracket_stack and bracket_stack[-1] == "{":
238
+ trailing_dict_text = text[:text.rfind("{")]
239
+ num_keys = trailing_dict_text.count(":")
240
+ num_values = trailing_dict_text.count(",")
241
+ if num_keys <= num_values:
242
+ return None # Incomplete property name within parameter value
243
+ if bracket_stack and bracket_stack[-1] == "(":
244
+ trailing_params_text = text[:text.rfind("(")]
245
+ num_full_param_names = trailing_params_text.count("=")
246
+ num_full_param_values = trailing_params_text.count(",")
247
+ if num_full_param_names <= num_full_param_values:
248
+ return None # Incomplete parameter name
249
+ if text.endswith(","):
250
+ text = text[:-1]
251
+ if bracket_stack and bracket_stack[-1] == "[" and not text.endswith(
252
+ "[") and not text.endswith(")"):
253
+ return None # Incomplete function name
254
+
255
+ added_text = ""
256
+ for char in reversed(bracket_stack):
257
+ if char == "[":
258
+ added_text += "]"
259
+ elif char == "(":
260
+ added_text += ")"
261
+ elif char == "{":
262
+ added_text += "}"
263
+ elif char == "'":
264
+ added_text += "'"
265
+ elif char == '"':
266
+ added_text += '"'
267
+
268
+ return text + added_text, added_text
269
+
270
+
271
+def _compute_tool_delta(previously_sent_args: str, new_call: ToolCall,
272
+ index: int,
273
+ withheld_suffix: str) -> Union[DeltaToolCall, None]:
274
+ new_call_args = new_call.function.arguments
275
+ if withheld_suffix:
276
+ assert new_call_args.endswith(withheld_suffix)
277
+ new_call_args = new_call_args[:-len(withheld_suffix)]
278
+ if not previously_sent_args:
279
+ return DeltaToolCall(id=new_call.id,
280
+ index=index,
281
+ function=DeltaFunctionCall(
282
+ name=new_call.function.name,
283
+ arguments=new_call_args,
284
+ ))
285
+
286
+ arg_diff = new_call_args[len(previously_sent_args):]
287
+ return DeltaToolCall(
288
+ id="", index=index, function=DeltaFunctionCall(
289
+ arguments=arg_diff)) if arg_diff else None