A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 05:45 UTC
[Docs] Add dedicated tool calling page to docs (#10554)
Signed-off-by: mgoin <michael@neuralmagic.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Michael Goin committed
Dec 9, 2024 at 20:15 UTC
6d525288c1a40ee70f9cff2fe08657f23bae88dc
3 files changed
+288
-217
docs/source/index.rst
+1
@@ -102,6 +102,7 @@ Documentation
102
103
usage/lora
104
usage/multimodal_inputs
105
+ usage/tool_calling
106
usage/structured_outputs
107
usage/spec_decode
108
usage/compatibility_matrix
docs/source/serving/openai_compatible_server.md
-217
@@ -361,220 +361,3 @@ $ vllm serve SOME_MODEL --config config.yaml
361
**NOTE**
362
In case an argument is supplied simultaneously using command line and the config file, the value from the commandline will take precedence.
363
The order of priorities is `command line > config file values > defaults`.
364
-
365
----
366
-
367
-## Tool calling in the chat completion API
368
-vLLM currently supports named function calling, as well as the `auto` and `none` options for the `tool_choice` field in the chat completion API. The `tool_choice` option `required` is **not yet supported** but on the roadmap.
369
-
370
-It is the callers responsibility to prompt the model with the tool information, vLLM will not automatically manipulate the prompt.
371
-Please see below for recommended configuration and chat templates to use when function calling is to be used with the different models.
372
-
373
-
374
-### Named Function Calling
375
-vLLM supports named function calling in the chat completion API by default. It does so using Outlines, so this is
376
-enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
377
-high-quality one.
378
-
379
-vLLM will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
380
-
381
-To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
382
-specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
383
-
384
-
385
-### Automatic Function Calling
386
-To enable this feature, you should set the following flags:
387
-* `--enable-auto-tool-choice` -- **mandatory** Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it
388
-deems appropriate.
389
-* `--tool-call-parser` -- select the tool parser to use (listed below). Additional tool parsers
390
-will continue to be added in the future, and also can register your own tool parsers in the `--tool-parser-plugin`.
391
-* `--tool-parser-plugin` -- **optional** tool parser plugin used to register user defined tool parsers into vllm, the registered tool parser name can be specified in `--tool-call-parser`.
392
-* `--chat-template` -- **optional** for auto tool choice. the path to the chat template which handles `tool`-role messages and `assistant`-role messages
393
-that contain previously generated tool calls. Hermes, Mistral and Llama models have tool-compatible chat templates in their
394
-`tokenizer_config.json` files, but you can specify a custom template. This argument can be set to `tool_use` if your model has a tool use-specific chat
395
-template configured in the `tokenizer_config.json`. In this case, it will be used per the `transformers` specification. More on this [here](https://huggingface.co/docs/transformers/en/chat_templating#why-do-some-models-have-multiple-templates)
396
-from HuggingFace; and you can find an example of this in a `tokenizer_config.json` [here](https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/blob/main/tokenizer_config.json)
397
-
398
-If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
399
-
400
-
401
-#### Hermes Models (`hermes`)
402
-
403
-All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported.
404
-* `NousResearch/Hermes-2-Pro-*`
405
-* `NousResearch/Hermes-2-Theta-*`
406
-* `NousResearch/Hermes-3-*`
407
-
408
-
409
-_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality & capabilities due to the merge
410
-step in their creation_.
411
-
412
-Flags: `--tool-call-parser hermes`
413
-
414
-
415
-#### Mistral Models (`mistral`)
416
-
417
-Supported models:
418
-* `mistralai/Mistral-7B-Instruct-v0.3` (confirmed)
419
-* Additional mistral function-calling models are compatible as well.
420
-
421
-Known issues:
422
-1. Mistral 7B struggles to generate parallel tool calls correctly.
423
-2. Mistral's `tokenizer_config.json` chat template requires tool call IDs that are exactly 9 digits, which is
424
-much shorter than what vLLM generates. Since an exception is thrown when this condition
425
-is not met, the following additional chat templates are provided:
426
-
427
-* `examples/tool_chat_template_mistral.jinja` - this is the "official" Mistral chat template, but tweaked so that
428
-it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)
429
-* `examples/tool_chat_template_mistral_parallel.jinja` - this is a "better" version that adds a tool-use system prompt
430
-when tools are provided, that results in much better reliability when working with parallel tool calling.
431
-
432
-
433
-Recommended flags: `--tool-call-parser mistral --chat-template examples/tool_chat_template_mistral_parallel.jinja`
434
-
435
-
436
-#### Llama Models (`llama3_json`)
437
-
438
-Supported models:
439
-* `meta-llama/Meta-Llama-3.1-8B-Instruct`
440
-* `meta-llama/Meta-Llama-3.1-70B-Instruct`
441
-* `meta-llama/Meta-Llama-3.1-405B-Instruct`
442
-* `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8`
443
-
444
-The tool calling that is supported is the [JSON based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) in Llama-3.2 models, see the `pythonic` tool parser below.
445
-Other tool calling formats like the built in python tool calling or custom tool calling are not supported.
446
-
447
-Known issues:
448
-1. Parallel tool calls are not supported.
449
-2. The model can generate parameters with a wrong format, such as generating
450
- an array serialized as string instead of an array.
451
-
452
-The `tool_chat_template_llama3_json.jinja` file contains the "official" Llama chat template, but tweaked so that
453
-it works better with vLLM.
454
-
455
-Recommended flags: `--tool-call-parser llama3_json --chat-template examples/tool_chat_template_llama3_json.jinja`
456
-
457
-#### IBM Granite
458
-
459
-Supported models:
460
-* `ibm-granite/granite-3.0-8b-instruct`
461
-
462
-Recommended flags: `--tool-call-parser granite --chat-template examples/tool_chat_template_granite.jinja`
463
-
464
-`examples/tool_chat_template_granite.jinja`: this is a modified chat template from the original on Huggingface. Parallel function calls are supported.
465
-
466
-* `ibm-granite/granite-20b-functioncalling`
467
-
468
-Recommended flags: `--tool-call-parser granite-20b-fc --chat-template examples/tool_chat_template_granite_20b_fc.jinja`
469
-
470
-`examples/tool_chat_template_granite_20b_fc.jinja`: this is a modified chat template from the original on Huggingface, which is not vLLM compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
471
-
472
-
473
-#### InternLM Models (`internlm`)
474
-
475
-Supported models:
476
-* `internlm/internlm2_5-7b-chat` (confirmed)
477
-* Additional internlm2.5 function-calling models are compatible as well
478
-
479
-Known issues:
480
-* Although this implementation also supports InternLM2, the tool call results are not stable when testing with the `internlm/internlm2-chat-7b` model.
481
-
482
-Recommended flags: `--tool-call-parser internlm --chat-template examples/tool_chat_template_internlm2_tool.jinja`
483
-
484
-
485
-#### Jamba Models (`jamba`)
486
-AI21's Jamba-1.5 models are supported.
487
-* `ai21labs/AI21-Jamba-1.5-Mini`
488
-* `ai21labs/AI21-Jamba-1.5-Large`
489
-
490
-
491
-Flags: `--tool-call-parser jamba`
492
-
493
-
494
-#### Models with Pythonic Tool Calls (`pythonic`)
495
-
496
-A growing number of models output a python list to represent tool calls instead of using JSON. This has the advantage of inherently supporting parallel tool calls and removing ambiguity around the JSON schema required for tool calls. The `pythonic` tool parser can support such models.
497
-
498
-As a concrete example, these models may look up the weather in San Francisco and Seattle by generating:
499
-```python
500
-[get_weather(city='San Francisco', metric='celsius'), get_weather(city='Seattle', metric='celsius')]
501
-```
502
-
503
-Limitations:
504
-* The model must not generate both text and tool calls in the same generation. This may not be hard to change for a specific model, but the community currently lacks consensus on which tokens to emit when starting and ending tool calls. (In particular, the Llama 3.2 models emit no such tokens.)
505
-* Llama's smaller models struggle to use tools effectively.
506
-
507
-Example supported models:
508
-* `meta-llama/Llama-3.2-1B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)
509
-* `meta-llama/Llama-3.2-3B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)
510
-* `Team-ACE/ToolACE-8B` (use with `examples/tool_chat_template_toolace.jinja`)
511
-* `fixie-ai/ultravox-v0_4-ToolACE-8B` (use with `examples/tool_chat_template_toolace.jinja`)
512
-
513
-Flags: `--tool-call-parser pythonic --chat-template {see_above}`
514
-
515
----
516
-**WARNING**
517
-Llama's smaller models frequently fail to emit tool calls in the correct format. Your mileage may vary.
518
-
519
----
520
-
521
-
522
-### How to write a tool parser plugin
523
-
524
-A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py.
525
-
526
-Here is a summary of a plugin file:
527
-
528
-```python
529
-
530
-# import the required packages
531
-
532
-# define a tool parser and register it to vllm
533
-# the name list in register_module can be used
534
-# in --tool-call-parser. you can define as many
535
-# tool parsers as you want here.
536
-@ToolParserManager.register_module(["example"])
537
-class ExampleToolParser(ToolParser):
538
- def __init__(self, tokenizer: AnyTokenizer):
539
- super().__init__(tokenizer)
540
-
541
- # adjust request. e.g.: set skip special tokens
542
- # to False for tool call output.
543
- def adjust_request(
544
- self, request: ChatCompletionRequest) -> ChatCompletionRequest:
545
- return request
546
-
547
- # implement the tool call parse for stream call
548
- def extract_tool_calls_streaming(
549
- self,
550
- previous_text: str,
551
- current_text: str,
552
- delta_text: str,
553
- previous_token_ids: Sequence[int],
554
- current_token_ids: Sequence[int],
555
- delta_token_ids: Sequence[int],
556
- request: ChatCompletionRequest,
557
- ) -> Union[DeltaMessage, None]:
558
- return delta
559
-
560
- # implement the tool parse for non-stream call
561
- def extract_tool_calls(
562
- self,
563
- model_output: str,
564
- request: ChatCompletionRequest,
565
- ) -> ExtractedToolCallInformation:
566
- return ExtractedToolCallInformation(tools_called=False,
567
- tool_calls=[],
568
- content=text)
569
-
570
-
571
-```
572
-
573
-Then you can use this plugin in the command line like this.
574
-```
575
- --enable-auto-tool-choice \
576
- --tool-parser-plugin <absolute path of the plugin file>
577
- --tool-call-parser example \
578
- --chat-template <your chat template> \
579
-```
580
-
docs/source/usage/tool_calling.md
new
+287
@@ -0,0 +1,287 @@
1
+# Tool Calling
2
+
3
+vLLM currently supports named function calling, as well as the `auto` and `none` options for the `tool_choice` field in the chat completion API. The `tool_choice` option `required` is **not yet supported** but on the roadmap.
4
+
5
+## Quickstart
6
+
7
+Start the server with tool calling enabled. This example uses Meta's Llama 3.1 8B model, so we need to use the llama3 tool calling chat template from the vLLM examples directory:
8
+
9
+```bash
10
+vllm serve meta-llama/Llama-3.1-8B-Instruct \
11
+ --enable-auto-tool-choice \
12
+ --tool-call-parser llama3_json \
13
+ --chat-template examples/tool_chat_template_llama3_json.jinja
14
+```
15
+
16
+Next, make a request to the model that should result in it using the available tools:
17
+
18
+```python
19
+from openai import OpenAI
20
+import json
21
+
22
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
23
+
24
+def get_weather(location: str, unit: str):
25
+ return f"Getting the weather for {location} in {unit}..."
26
+tool_functions = {"get_weather": get_weather}
27
+
28
+tools = [{
29
+ "type": "function",
30
+ "function": {
31
+ "name": "get_weather",
32
+ "description": "Get the current weather in a given location",
33
+ "parameters": {
34
+ "type": "object",
35
+ "properties": {
36
+ "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
37
+ "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
38
+ },
39
+ "required": ["location", "unit"]
40
+ }
41
+ }
42
+}]
43
+
44
+response = client.chat.completions.create(
45
+ model=client.models.list().data[0].id,
46
+ messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
47
+ tools=tools,
48
+ tool_choice="auto"
49
+)
50
+
51
+tool_call = response.choices[0].message.tool_calls[0].function
52
+print(f"Function called: {tool_call.name}")
53
+print(f"Arguments: {tool_call.arguments}")
54
+print(f"Result: {get_weather(**json.loads(tool_call.arguments))}")
55
+```
56
+
57
+Example output:
58
+```
59
+Function called: get_weather
60
+Arguments: {"location": "San Francisco, CA", "unit": "fahrenheit"}
61
+Result: Getting the weather for San Francisco, CA in fahrenheit...
62
+```
63
+
64
+This example demonstrates:
65
+- Setting up the server with tool calling enabled
66
+- Defining an actual function to handle tool calls
67
+- Making a request with `tool_choice="auto"`
68
+- Handling the structured response and executing the corresponding function
69
+
70
+You can also specify a particular function using named function calling by setting `tool_choice={"type": "function", "function": {"name": "get_weather"}}`. Note that this will use the guided decoding backend - so the first time this is used, there will be several seconds of latency (or more) as the FSM is compiled for the first time before it is cached for subsequent requests.
71
+
72
+Remember that it's the callers responsibility to:
73
+1. Define appropriate tools in the request
74
+2. Include relevant context in the chat messages
75
+3. Handle the tool calls in your application logic
76
+
77
+For more advanced usage, including parallel tool calls and different model-specific parsers, see the sections below.
78
+
79
+## Named Function Calling
80
+vLLM supports named function calling in the chat completion API by default. It does so using Outlines through guided decoding, so this is
81
+enabled by default, and will work with any supported model. You are guaranteed a validly-parsable function call - not a
82
+high-quality one.
83
+
84
+vLLM will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
85
+For best results, we recommend ensuring that the expected output format / schema is specified in the prompt to ensure that the model's intended generation is aligned with the schema that it's being forced to generate by the guided decoding backend.
86
+
87
+To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
88
+specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
89
+
90
+
91
+## Automatic Function Calling
92
+
93
+To enable this feature, you should set the following flags:
94
+* `--enable-auto-tool-choice` -- **mandatory** Auto tool choice. tells vLLM that you want to enable the model to generate its own tool calls when it
95
+deems appropriate.
96
+* `--tool-call-parser` -- select the tool parser to use (listed below). Additional tool parsers
97
+will continue to be added in the future, and also can register your own tool parsers in the `--tool-parser-plugin`.
98
+* `--tool-parser-plugin` -- **optional** tool parser plugin used to register user defined tool parsers into vllm, the registered tool parser name can be specified in `--tool-call-parser`.
99
+* `--chat-template` -- **optional** for auto tool choice. the path to the chat template which handles `tool`-role messages and `assistant`-role messages
100
+that contain previously generated tool calls. Hermes, Mistral and Llama models have tool-compatible chat templates in their
101
+`tokenizer_config.json` files, but you can specify a custom template. This argument can be set to `tool_use` if your model has a tool use-specific chat
102
+template configured in the `tokenizer_config.json`. In this case, it will be used per the `transformers` specification. More on this [here](https://huggingface.co/docs/transformers/en/chat_templating#why-do-some-models-have-multiple-templates)
103
+from HuggingFace; and you can find an example of this in a `tokenizer_config.json` [here](https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/blob/main/tokenizer_config.json)
104
+
105
+If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
106
+
107
+
108
+### Hermes Models (`hermes`)
109
+
110
+All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported.
111
+* `NousResearch/Hermes-2-Pro-*`
112
+* `NousResearch/Hermes-2-Theta-*`
113
+* `NousResearch/Hermes-3-*`
114
+
115
+
116
+_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality & capabilities due to the merge
117
+step in their creation_.
118
+
119
+Flags: `--tool-call-parser hermes`
120
+
121
+
122
+### Mistral Models (`mistral`)
123
+
124
+Supported models:
125
+* `mistralai/Mistral-7B-Instruct-v0.3` (confirmed)
126
+* Additional mistral function-calling models are compatible as well.
127
+
128
+Known issues:
129
+1. Mistral 7B struggles to generate parallel tool calls correctly.
130
+2. Mistral's `tokenizer_config.json` chat template requires tool call IDs that are exactly 9 digits, which is
131
+much shorter than what vLLM generates. Since an exception is thrown when this condition
132
+is not met, the following additional chat templates are provided:
133
+
134
+* `examples/tool_chat_template_mistral.jinja` - this is the "official" Mistral chat template, but tweaked so that
135
+it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)
136
+* `examples/tool_chat_template_mistral_parallel.jinja` - this is a "better" version that adds a tool-use system prompt
137
+when tools are provided, that results in much better reliability when working with parallel tool calling.
138
+
139
+
140
+Recommended flags: `--tool-call-parser mistral --chat-template examples/tool_chat_template_mistral_parallel.jinja`
141
+
142
+
143
+### Llama Models (`llama3_json`)
144
+
145
+Supported models:
146
+* `meta-llama/Meta-Llama-3.1-8B-Instruct`
147
+* `meta-llama/Meta-Llama-3.1-70B-Instruct`
148
+* `meta-llama/Meta-Llama-3.1-405B-Instruct`
149
+* `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8`
150
+
151
+The tool calling that is supported is the [JSON based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) in Llama-3.2 models, see the `pythonic` tool parser below.
152
+Other tool calling formats like the built in python tool calling or custom tool calling are not supported.
153
+
154
+Known issues:
155
+1. Parallel tool calls are not supported.
156
+2. The model can generate parameters with a wrong format, such as generating
157
+ an array serialized as string instead of an array.
158
+
159
+The `tool_chat_template_llama3_json.jinja` file contains the "official" Llama chat template, but tweaked so that
160
+it works better with vLLM.
161
+
162
+Recommended flags: `--tool-call-parser llama3_json --chat-template examples/tool_chat_template_llama3_json.jinja`
163
+
164
+#### IBM Granite
165
+
166
+Supported models:
167
+* `ibm-granite/granite-3.0-8b-instruct`
168
+
169
+Recommended flags: `--tool-call-parser granite --chat-template examples/tool_chat_template_granite.jinja`
170
+
171
+`examples/tool_chat_template_granite.jinja`: this is a modified chat template from the original on Huggingface. Parallel function calls are supported.
172
+
173
+* `ibm-granite/granite-20b-functioncalling`
174
+
175
+Recommended flags: `--tool-call-parser granite-20b-fc --chat-template examples/tool_chat_template_granite_20b_fc.jinja`
176
+
177
+`examples/tool_chat_template_granite_20b_fc.jinja`: this is a modified chat template from the original on Huggingface, which is not vLLM compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
178
+
179
+
180
+### InternLM Models (`internlm`)
181
+
182
+Supported models:
183
+* `internlm/internlm2_5-7b-chat` (confirmed)
184
+* Additional internlm2.5 function-calling models are compatible as well
185
+
186
+Known issues:
187
+* Although this implementation also supports InternLM2, the tool call results are not stable when testing with the `internlm/internlm2-chat-7b` model.
188
+
189
+Recommended flags: `--tool-call-parser internlm --chat-template examples/tool_chat_template_internlm2_tool.jinja`
190
+
191
+
192
+### Jamba Models (`jamba`)
193
+AI21's Jamba-1.5 models are supported.
194
+* `ai21labs/AI21-Jamba-1.5-Mini`
195
+* `ai21labs/AI21-Jamba-1.5-Large`
196
+
197
+
198
+Flags: `--tool-call-parser jamba`
199
+
200
+
201
+### Models with Pythonic Tool Calls (`pythonic`)
202
+
203
+A growing number of models output a python list to represent tool calls instead of using JSON. This has the advantage of inherently supporting parallel tool calls and removing ambiguity around the JSON schema required for tool calls. The `pythonic` tool parser can support such models.
204
+
205
+As a concrete example, these models may look up the weather in San Francisco and Seattle by generating:
206
+```python
207
+[get_weather(city='San Francisco', metric='celsius'), get_weather(city='Seattle', metric='celsius')]
208
+```
209
+
210
+Limitations:
211
+* The model must not generate both text and tool calls in the same generation. This may not be hard to change for a specific model, but the community currently lacks consensus on which tokens to emit when starting and ending tool calls. (In particular, the Llama 3.2 models emit no such tokens.)
212
+* Llama's smaller models struggle to use tools effectively.
213
+
214
+Example supported models:
215
+* `meta-llama/Llama-3.2-1B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)
216
+* `meta-llama/Llama-3.2-3B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)
217
+* `Team-ACE/ToolACE-8B` (use with `examples/tool_chat_template_toolace.jinja`)
218
+* `fixie-ai/ultravox-v0_4-ToolACE-8B` (use with `examples/tool_chat_template_toolace.jinja`)
219
+
220
+Flags: `--tool-call-parser pythonic --chat-template {see_above}`
221
+
222
+---
223
+**WARNING**
224
+Llama's smaller models frequently fail to emit tool calls in the correct format. Your mileage may vary.
225
+
226
+---
227
+
228
+
229
+## How to write a tool parser plugin
230
+
231
+A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py.
232
+
233
+Here is a summary of a plugin file:
234
+
235
+```python
236
+
237
+# import the required packages
238
+
239
+# define a tool parser and register it to vllm
240
+# the name list in register_module can be used
241
+# in --tool-call-parser. you can define as many
242
+# tool parsers as you want here.
243
+@ToolParserManager.register_module(["example"])
244
+class ExampleToolParser(ToolParser):
245
+ def __init__(self, tokenizer: AnyTokenizer):
246
+ super().__init__(tokenizer)
247
+
248
+ # adjust request. e.g.: set skip special tokens
249
+ # to False for tool call output.
250
+ def adjust_request(
251
+ self, request: ChatCompletionRequest) -> ChatCompletionRequest:
252
+ return request
253
+
254
+ # implement the tool call parse for stream call
255
+ def extract_tool_calls_streaming(
256
+ self,
257
+ previous_text: str,
258
+ current_text: str,
259
+ delta_text: str,
260
+ previous_token_ids: Sequence[int],
261
+ current_token_ids: Sequence[int],
262
+ delta_token_ids: Sequence[int],
263
+ request: ChatCompletionRequest,
264
+ ) -> Union[DeltaMessage, None]:
265
+ return delta
266
+
267
+ # implement the tool parse for non-stream call
268
+ def extract_tool_calls(
269
+ self,
270
+ model_output: str,
271
+ request: ChatCompletionRequest,
272
+ ) -> ExtractedToolCallInformation:
273
+ return ExtractedToolCallInformation(tools_called=False,
274
+ tool_calls=[],
275
+ content=text)
276
+
277
+
278
+```
279
+
280
+Then you can use this plugin in the command line like this.
281
+```
282
+ --enable-auto-tool-choice \
283
+ --tool-parser-plugin <absolute path of the plugin file>
284
+ --tool-call-parser example \
285
+ --chat-template <your chat template> \
286
+```
287
+