A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 08:15 UTC
[Frontend] Tool calling parser for Granite 3.0 models (#9027)
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Maximilien de Bayser committed
Nov 7, 2024 at 12:09 UTC
ae62fd17c0023f7ec363c1141787b8c017937c44
6 files changed
+314
-33
docs/source/serving/openai_compatible_server.md
+26
-18
@@ -160,14 +160,7 @@ this, unless explicitly specified.
160
:func: create_parser_for_docs
161
:prog: vllm serve
162
```
163
-## Tool Calling in the Chat Completion API
164
-### Named Function Calling
165
-vLLM supports only named function calling in the chat completion API by default. It does so using Outlines, so this is
166
-enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
167
-high-quality one.
163
169
-To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
170
-specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
164
165
### Config file
166
@@ -196,12 +189,22 @@ The order of priorities is `command line > config file values > defaults`.
189
---
190
191
## Tool calling in the chat completion API
199
-vLLM supports only named function calling in the chat completion API. The `tool_choice` options `auto` and `required` are **not yet supported** but on the roadmap.
192
+
193
+vLLM supports named function calling and `auto` tool choice in the chat completion API. The `tool_choice` options `required` is **not yet supported** but on the roadmap.
194
195
It is the callers responsibility to prompt the model with the tool information, vLLM will not automatically manipulate the prompt.
196
197
+
198
+### Named Function Calling
199
+vLLM supports named function calling in the chat completion API by default. It does so using Outlines, so this is
200
+enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
201
+high-quality one.
202
+
203
vLLM will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
204
205
+To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
206
+specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
207
+
208
209
### Automatic Function Calling
210
To enable this feature, you should set the following flags:
@@ -275,6 +278,21 @@ it works better with vLLM.
278
279
Recommended flags: `--tool-call-parser llama3_json --chat-template examples/tool_chat_template_llama3_json.jinja`
280
281
+#### IBM Granite
282
+
283
+Supported models:
284
+* `ibm-granite/granite-3.0-8b-instruct`
285
+
286
+Recommended flags: `--tool-call-parser granite --chat-template examples/tool_chat_template_granite.jinja`
287
+
288
+`examples/tool_chat_template_granite.jinja`: this is a modified chat template from the original on Huggingface. Parallel function calls are supported.
289
+
290
+* `ibm-granite/granite-20b-functioncalling`
291
+
292
+Recommended flags: `--tool-call-parser granite-20b-fc --chat-template examples/tool_chat_template_granite_20b_fc.jinja`
293
+
294
+`examples/tool_chat_template_granite_20b_fc.jinja`: this is a modified chat template from the original on Huggingface, which is not vLLM compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
295
+
296
297
#### InternLM Models (`internlm`)
298
@@ -297,16 +315,6 @@ AI21's Jamba-1.5 models are supported.
315
Flags: `--tool-call-parser jamba`
316
317
300
-#### IBM Granite (`granite-20b-fc`)
301
-
302
-Supported models:
303
-* `ibm-granite/granite-20b-functioncalling`
304
-
305
-Flags: `--tool-call-parser granite-20b-fc --chat-template examples/tool_chat_template_granite_20b_fc.jinja`
306
-
307
-The example chat template deviates slightly from the original on Huggingface, which is not vLLM compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
308
-
309
-
318
### How to write a tool parser plugin
319
320
A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py.
examples/tool_chat_template_granite.jinja
new
+40
@@ -0,0 +1,40 @@
1
+{%- if tools %}
2
+ {{- '<|start_of_role|>available_tools<|end_of_role|>
3
+' }}
4
+ {%- for tool in tools %}
5
+ {{- tool | tojson(indent=4) }}
6
+ {%- if not loop.last %}
7
+ {{- '
8
+
9
+' }}
10
+ {%- endif %}
11
+ {%- endfor %}
12
+ {{- '<|end_of_text|>
13
+' }}
14
+{%- endif %}
15
+
16
+{%- for message in messages %}
17
+ {%- if message['role'] == 'system' %}
18
+ {{- '<|start_of_role|>system<|end_of_role|>' + message['content'] + '<|end_of_text|>
19
+' }}
20
+ {%- elif message['role'] == 'user' %}
21
+ {{- '<|start_of_role|>user<|end_of_role|>' + message['content'] + '<|end_of_text|>
22
+' }}
23
+ {%- elif message['role'] == 'assistant_tool_call' or (message['role'] == 'assistant' and message.tool_calls is defined) %}
24
+ {{- '<|start_of_role|>assistant<|end_of_role|>' }}
25
+ {% for tc in message.tool_calls %}
26
+ {{- '<|tool_call|> ' + {'name': tc.function.name, 'arguments': tc.function.arguments}|tojson }}
27
+ {% endfor %}
28
+ {{- '<|end_of_text|>
29
+' }}
30
+ {%- elif message['role'] == 'assistant' %}
31
+ {{- '<|start_of_role|>assistant<|end_of_role|>' + message['content'] + '<|end_of_text|>
32
+' }}
33
+ {%- elif message['role'] == 'tool_response' or message['role'] == 'tool' %}
34
+ {{- '<|start_of_role|>tool_response<|end_of_role|>' + message['content'] + '<|end_of_text|>
35
+' }}
36
+ {%- endif %}
37
+ {%- if loop.last and add_generation_prompt %}
38
+ {{- '<|start_of_role|>assistant<|end_of_role|>' }}
39
+ {%- endif %}
40
+{%- endfor %}
tests/tool_use/conftest.py
+6
@@ -3,6 +3,7 @@ import pytest_asyncio
3
from huggingface_hub import snapshot_download
4
5
from tests.utils import RemoteOpenAIServer
6
+from vllm.platforms import current_platform
7
8
from .utils import ARGS, CONFIGS, ServerConfig
9
@@ -11,6 +12,11 @@ from .utils import ARGS, CONFIGS, ServerConfig
12
@pytest.fixture(scope="session", params=CONFIGS.keys())
13
def server_config(request):
14
config = CONFIGS[request.param]
15
+
16
+ if current_platform.is_rocm() and not config.get("supports_rocm", True):
17
+ pytest.skip("The {} model can't be tested on the ROCm platform".format(
18
+ config["model"]))
19
+
20
# download model and tokenizer using transformers
21
snapshot_download(config["model"])
22
yield CONFIGS[request.param]
tests/tool_use/utils.py
+24
-13
@@ -13,6 +13,7 @@ class ServerConfig(TypedDict, total=False):
13
arguments: List[str]
14
system_prompt: Optional[str]
15
supports_parallel: Optional[bool]
16
+ supports_rocm: Optional[bool]
17
18
19
def patch_system_prompt(messages: List[Dict[str, Any]],
@@ -36,7 +37,7 @@ def ensure_system_prompt(messages: List[Dict[str, Any]],
37
38
# universal args for all models go here. also good if you need to test locally
39
# and change type or KV cache quantization or something.
39
-ARGS: List[str] = ["--enable-auto-tool-choice", "--max-model-len", "8096"]
40
+ARGS: List[str] = ["--enable-auto-tool-choice", "--max-model-len", "1024"]
41
42
CONFIGS: Dict[str, ServerConfig] = {
43
"hermes": {
@@ -88,18 +89,28 @@ CONFIGS: Dict[str, ServerConfig] = {
89
"without calling a tool. DO NOT CALL A TOOL THAT IS IRRELEVANT "
90
"to the user's question - just respond to it normally."
91
},
91
- ## FIXME: temporary disabled due to lack of hardware specification
92
- ## for individual runs
93
- #"granite20b": {
94
- # "model":
95
- # "ibm-granite/granite-20b-functioncalling",
96
- # "arguments": [
97
- # "--tool-call-parser", "granite-20b-fc", "--chat-template",
98
- # str(VLLM_PATH / "examples/tool_chat_template_granite_20b_fc.jinja")
99
- # ],
100
- # "supports_parallel":
101
- # False,
102
- #},
92
+ "granite20b": {
93
+ "model":
94
+ "mbayser/granite-20b-functioncalling-FP8-KV",
95
+ "arguments": [
96
+ "--tool-call-parser", "granite-20b-fc", "--chat-template",
97
+ str(VLLM_PATH /
98
+ "examples/tool_chat_template_granite_20b_fc.jinja"),
99
+ "--max_num_seqs", "1", "--enforce-eager", "--cpu-offload-gb", "20"
100
+ ],
101
+ "supports_parallel":
102
+ False,
103
+ "supports_rocm":
104
+ False,
105
+ },
106
+ "granite8b": {
107
+ "model":
108
+ "ibm-granite/granite-3.0-8b-instruct",
109
+ "arguments": [
110
+ "--tool-call-parser", "granite", "--chat-template",
111
+ str(VLLM_PATH / "examples/tool_chat_template_granite.jinja")
112
+ ],
113
+ },
114
"internlm": {
115
"model":
116
"internlm/internlm2_5-7b-chat",
vllm/entrypoints/openai/tool_parsers/__init__.py
+3
-2
@@ -1,5 +1,6 @@
1
from .abstract_tool_parser import ToolParser, ToolParserManager
2
from .granite_20b_fc_tool_parser import Granite20bFCToolParser
3
+from .granite_tool_parser import GraniteToolParser
4
from .hermes_tool_parser import Hermes2ProToolParser
5
from .internlm2_tool_parser import Internlm2ToolParser
6
from .jamba_tool_parser import JambaToolParser
@@ -8,6 +9,6 @@ from .mistral_tool_parser import MistralToolParser
9
10
__all__ = [
11
"ToolParser", "ToolParserManager", "Granite20bFCToolParser",
11
- "Hermes2ProToolParser", "MistralToolParser", "Internlm2ToolParser",
12
- "Llama3JsonToolParser", "JambaToolParser"
12
+ "GraniteToolParser", "Hermes2ProToolParser", "MistralToolParser",
13
+ "Internlm2ToolParser", "Llama3JsonToolParser", "JambaToolParser"
14
]
vllm/entrypoints/openai/tool_parsers/granite_tool_parser.py
new
+215
@@ -0,0 +1,215 @@
1
+import json
2
+from typing import Dict, Sequence, Union
3
+
4
+import partial_json_parser
5
+from partial_json_parser.core.options import Allow
6
+
7
+from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
8
+ DeltaFunctionCall, DeltaMessage,
9
+ DeltaToolCall,
10
+ ExtractedToolCallInformation,
11
+ FunctionCall, ToolCall)
12
+from vllm.entrypoints.openai.tool_parsers.abstract_tool_parser import (
13
+ ToolParser, ToolParserManager)
14
+from vllm.entrypoints.openai.tool_parsers.utils import (consume_space,
15
+ find_common_prefix,
16
+ is_complete_json,
17
+ partial_json_loads)
18
+from vllm.logger import init_logger
19
+from vllm.transformers_utils.tokenizer import AnyTokenizer
20
+from vllm.utils import random_uuid
21
+
22
+logger = init_logger(__name__)
23
+
24
+
25
+@ToolParserManager.register_module("granite")
26
+class GraniteToolParser(ToolParser):
27
+ """
28
+ Tool call parser for the granite 3.0 models. Intended
29
+ for use with the examples/tool_chat_template_granite.jinja
30
+ template.
31
+
32
+ Used when --enable-auto-tool-choice --tool-call-parser granite
33
+ are all set
34
+ """
35
+
36
+ def __init__(self, tokenizer: AnyTokenizer):
37
+ super().__init__(tokenizer)
38
+
39
+ def extract_tool_calls(
40
+ self, model_output: str,
41
+ request: ChatCompletionRequest) -> ExtractedToolCallInformation:
42
+ stripped = model_output.strip()
43
+ if not stripped or stripped[0] != '[':
44
+ return ExtractedToolCallInformation(tools_called=False,
45
+ tool_calls=[],
46
+ content=model_output)
47
+ try:
48
+ raw_function_calls = json.loads(stripped)
49
+ if not isinstance(raw_function_calls, list):
50
+ raise Exception(
51
+ f"Expected dict or list, got {type(raw_function_calls)}")
52
+
53
+ logger.debug("Extracted %d tool calls", len(raw_function_calls))
54
+ tool_calls = [
55
+ ToolCall(
56
+ type="function",
57
+ function=FunctionCall(
58
+ name=function_call["name"],
59
+ # function call args are JSON but as a string
60
+ arguments=json.dumps(function_call["arguments"]),
61
+ ),
62
+ ) for function_call in raw_function_calls
63
+ ]
64
+
65
+ return ExtractedToolCallInformation(
66
+ tools_called=True,
67
+ tool_calls=tool_calls,
68
+ content=None,
69
+ )
70
+
71
+ except Exception as e:
72
+ logger.error("Error in extracting tool call from response %s", e)
73
+ return ExtractedToolCallInformation(tools_called=False,
74
+ tool_calls=[],
75
+ content=model_output)
76
+
77
+ def extract_tool_calls_streaming(
78
+ self,
79
+ previous_text: str,
80
+ current_text: str,
81
+ delta_text: str,
82
+ previous_token_ids: Sequence[int],
83
+ current_token_ids: Sequence[int],
84
+ delta_token_ids: Sequence[int],
85
+ request: ChatCompletionRequest,
86
+ ) -> Union[DeltaMessage, None]:
87
+
88
+ start_idx = consume_space(0, current_text)
89
+ if not current_text or current_text[start_idx] != '[':
90
+ return DeltaMessage(content=delta_text)
91
+
92
+ # bit mask flags for partial JSON parsing. If the name hasn't been
93
+ # sent yet, don't allow sending
94
+ # an incomplete string since OpenAI only ever (as far as I have
95
+ # seen) allows sending the entire tool/ function name at once.
96
+ flags = Allow.ALL if self.current_tool_name_sent \
97
+ else Allow.ALL & ~Allow.STR
98
+ try:
99
+ tool_call_arr = None
100
+ is_complete = None
101
+ try:
102
+ tool_calls, end_idx = partial_json_loads(
103
+ current_text[start_idx:], flags)
104
+ if type(tool_calls) is list:
105
+ tool_call_arr = tool_calls
106
+ else:
107
+ return DeltaMessage(content=delta_text)
108
+
109
+ is_complete = [True] * len(tool_calls)
110
+ if not is_complete_json(
111
+ current_text[start_idx:start_idx + end_idx]):
112
+ is_complete[-1] = False
113
+ except partial_json_parser.core.exceptions.MalformedJSON:
114
+ logger.debug('not enough tokens to parse into JSON yet')
115
+ return None
116
+
117
+ # case -- if no tokens have been streamed for the tool, e.g.
118
+ # only the array brackets, stream nothing
119
+ if not tool_call_arr:
120
+ return None
121
+
122
+ # select as the current tool call the one we're on the state at
123
+ current_tool_call: Dict = tool_call_arr[self.current_tool_id]
124
+
125
+ delta = None
126
+ # case: we are starting a new tool in the array
127
+ # -> array has > 0 length AND length has moved past cursor
128
+ if len(tool_call_arr) > self.current_tool_id + 1:
129
+
130
+ # if we're moving on to a new call, first make sure we
131
+ # haven't missed anything in the previous one that was
132
+ # auto-generated due to JSON completions, but wasn't
133
+ # streamed to the client yet.
134
+ if self.current_tool_id >= 0:
135
+ cur_arguments = current_tool_call.get("arguments")
136
+ if cur_arguments:
137
+ cur_args_json = json.dumps(cur_arguments)
138
+ sent = len(
139
+ self.streamed_args_for_tool[self.current_tool_id])
140
+ argument_diff = cur_args_json[sent:]
141
+
142
+ logger.debug("got arguments diff: %s", argument_diff)
143
+ delta = DeltaMessage(tool_calls=[
144
+ DeltaToolCall(index=self.current_tool_id,
145
+ function=DeltaFunctionCall(
146
+ arguments=argument_diff).
147
+ model_dump(exclude_none=True))
148
+ ])
149
+ self.streamed_args_for_tool[
150
+ self.current_tool_id] += argument_diff
151
+
152
+ # re-set stuff pertaining to progress in the current tool
153
+ self.current_tool_id = len(tool_call_arr) - 1
154
+ self.current_tool_name_sent = False
155
+ self.streamed_args_for_tool.append("")
156
+ logger.debug("starting on new tool %d", self.current_tool_id)
157
+ return delta
158
+
159
+ # if the current tool name hasn't been sent, send if available
160
+ # - otherwise send nothing
161
+ elif not self.current_tool_name_sent:
162
+ function_name = current_tool_call.get("name")
163
+ if function_name:
164
+
165
+ delta = DeltaMessage(tool_calls=[
166
+ DeltaToolCall(index=self.current_tool_id,
167
+ type="function",
168
+ id=f"chatcmpl-tool-{random_uuid()}",
169
+ function=DeltaFunctionCall(
170
+ name=function_name).model_dump(
171
+ exclude_none=True))
172
+ ])
173
+ self.current_tool_name_sent = True
174
+
175
+ # now we know we're on the same tool call and we're streaming
176
+ # arguments
177
+ else:
178
+ cur_arguments = current_tool_call.get("arguments")
179
+
180
+ if cur_arguments:
181
+ sent = len(
182
+ self.streamed_args_for_tool[self.current_tool_id])
183
+ cur_args_json = json.dumps(cur_arguments)
184
+ prev_arguments = self.prev_tool_call_arr[
185
+ self.current_tool_id].get("arguments")
186
+
187
+ argument_diff = None
188
+ if is_complete[self.current_tool_id]:
189
+ argument_diff = cur_args_json[sent:]
190
+ elif prev_arguments:
191
+ prev_args_json = json.dumps(prev_arguments)
192
+ if cur_args_json != prev_args_json:
193
+ prefix = find_common_prefix(
194
+ prev_args_json, cur_args_json)
195
+ argument_diff = prefix[sent:]
196
+
197
+ if argument_diff is not None:
198
+ delta = DeltaMessage(tool_calls=[
199
+ DeltaToolCall(index=self.current_tool_id,
200
+ function=DeltaFunctionCall(
201
+ arguments=argument_diff).
202
+ model_dump(exclude_none=True))
203
+ ])
204
+ self.streamed_args_for_tool[
205
+ self.current_tool_id] += argument_diff
206
+
207
+ self.prev_tool_call_arr = tool_call_arr
208
+ return delta
209
+
210
+ except Exception as e:
211
+ logger.error("Error trying to handle streaming tool call: %s", e)
212
+ logger.debug(
213
+ "Skipping chunk as a result of tool streaming extraction "
214
+ "error")
215
+ return None