@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 641 lines 30.5 KB
Rendered Raw
1 # Tool Calling
2
3 vLLM currently supports named function calling, as well as the `auto`, `required` (as of `vllm>=0.8.3`), and `none` options for the `tool_choice` field in the chat completion API.
4
5 ## Quickstart
6
7 Start the server with tool calling enabled. This example uses Meta's Llama 3.1 8B model, so we need to use the `llama3_json` tool calling chat template from the vLLM examples directory:
8
9 ```bash
10 vllm serve meta-llama/Llama-3.1-8B-Instruct \
11 --enable-auto-tool-choice \
12 --tool-call-parser llama3_json \
13 --chat-template examples/tool_chat_template_llama3.1_json.jinja
14 ```
15
16 Next, make a request that triggers the model to use the available tools:
17
18 ??? code
19
20 ```python
21 from openai import OpenAI
22 import json
23
24 client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
25
26 def get_weather(location: str, unit: str):
27 return f"Getting the weather for {location} in {unit}..."
28 tool_functions = {"get_weather": get_weather}
29
30 tools = [
31 {
32 "type": "function",
33 "function": {
34 "name": "get_weather",
35 "description": "Get the current weather in a given location",
36 "parameters": {
37 "type": "object",
38 "properties": {
39 "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
40 "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
41 },
42 "required": ["location", "unit"],
43 },
44 },
45 },
46 ]
47
48 response = client.chat.completions.create(
49 model=client.models.list().data[0].id,
50 messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
51 tools=tools,
52 tool_choice="auto",
53 )
54
55 tool_call = response.choices[0].message.tool_calls[0].function
56 print(f"Function called: {tool_call.name}")
57 print(f"Arguments: {tool_call.arguments}")
58 print(f"Result: {tool_functions[tool_call.name](**json.loads(tool_call.arguments))}")
59 ```
60
61 Example output:
62
63 ```text
64 Function called: get_weather
65 Arguments: {"location": "San Francisco, CA", "unit": "fahrenheit"}
66 Result: Getting the weather for San Francisco, CA in fahrenheit...
67 ```
68
69 This example demonstrates:
70
71 * Setting up the server with tool calling enabled
72 * Defining an actual function to handle tool calls
73 * Making a request with `tool_choice="auto"`
74 * Handling the structured response and executing the corresponding function
75
76 You can also specify a particular function using named function calling by setting `tool_choice={"type": "function", "function": {"name": "get_weather"}}`. Note that this will use the structured outputs backend - so the first time this is used, there will be several seconds of latency (or more) as the FSM is compiled for the first time before it is cached for subsequent requests.
77
78 Remember that it's the caller's responsibility to:
79
80 1. Define appropriate tools in the request
81 2. Include relevant context in the chat messages
82 3. Handle the tool calls in your application logic
83
84 For more advanced usage, including parallel tool calls and different model-specific parsers, see the sections below.
85
86 ## Named Function Calling
87
88 vLLM supports named function calling in the chat completion API by default. This should work with most structured outputs backends supported by vLLM. You are guaranteed a validly-parsable function call - not a
89 high-quality one.
90
91 vLLM will use structured outputs to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter.
92 For best results, we recommend ensuring that the expected output format / schema is specified in the prompt to ensure that the model's intended generation is aligned with the schema that it's being forced to generate by the structured outputs backend.
93
94 To use a named function, you need to define the functions in the `tools` parameter of the chat completion request, and
95 specify the `name` of one of the tools in the `tool_choice` parameter of the chat completion request.
96
97 ## Required Function Calling
98
99 vLLM supports the `tool_choice='required'` option in the chat completion API. Similar to the named function calling, it also uses structured outputs, so this is enabled by default and will work with any supported model. However, support for alternative decoding backends are on the [roadmap](../usage/v1_guide.md#features) for the V1 engine.
100
101 When tool_choice='required' is set, the model is guaranteed to generate one or more tool calls based on the specified tool list in the `tools` parameter. The number of tool calls depends on the user's query. The output format strictly follows the schema defined in the `tools` parameter.
102
103 ## None Function Calling
104
105 vLLM supports the `tool_choice='none'` option in the chat completion API. When this option is set, the model will not generate any tool calls and will respond with regular text content only, even if tools are defined in the request.
106
107 !!! note
108 When tools are specified in the request, vLLM includes tool definitions in the prompt by default, regardless of the `tool_choice` setting. To exclude tool definitions when `tool_choice='none'`, use the `--exclude-tools-when-tool-choice-none` option.
109
110 ## Constrained Decoding Behavior
111
112 Structural-tag parsers resolve call obligation, grammar activation, and argument-schema enforcement separately. With structural-tag enforcement enabled:
113
114 | `tool_choice` value | Call obligation | Structural-tag activation |
115 | --- | --- | --- |
116 | Named function | Call the selected function | Always |
117 | `"required"` | Produce at least one tool call | Always |
118 | `"auto"` | Tool calls are optional | When at least one tool sets `strict: true`, or `--tool-strict-level` is `function` or `parameter` |
119 | `"none"` | Tool calling is disabled | Disabled |
120
121 When a structural tag applies, each tool's declared parameter schema is enforced only when that tool sets `strict: true` or the server uses `--tool-strict-level parameter`. Tools with omitted or false `strict` receive broad argument-syntax constraints at levels `auto` and `function`, including for required and named calls.
122
123 For parsers using schema-derived JSON constraints, required and named calls continue to enforce the declared parameter schemas.
124
125 ### Strict Mode
126
127 Structural-tag constraints have two layers. The tool-call *envelope* (markup and function name) is constrained whenever a structural tag applies: always for `tool_choice="required"` and named function calling, and for `tool_choice="auto"` once at least one tool sets `strict: true` or the server raises the floor. The *argument schema* of an individual tool is pinned only when that tool sets `strict: true` (or under `--tool-strict-level parameter`); a tool that omits `strict` keeps its arguments unconstrained, even when another tool in the same request is strict. The `strict` field is supported across all three API surfaces: Chat Completion, Responses, and Anthropic Messages.
128
129 For best compatibility with strict schema enforcement, define tool parameter schemas in the OpenAI strict-schema style:
130
131 * Set `additionalProperties` to `false` for each object in `parameters`.
132 * Mark all fields in `properties` as required.
133 * Represent optional fields by allowing `null`, for example `{"type": ["string", "null"]}`.
134
135 vLLM also provides a global toggle via the `VLLM_ENFORCE_STRICT_TOOL_CALLING` environment variable (defaults to `true`). When set to `false`, vLLM does not attach structural tags for tool calling regardless of the per-tool `strict` field. This environment variable only affects structural-tag based tool calling; it does not change schema-derived structured outputs used by named function calling or `tool_choice="required"`.
136
137 ```bash
138 VLLM_ENFORCE_STRICT_TOOL_CALLING=false vllm serve ...
139 ```
140
141 ### Server-Side Strictness Floor
142
143 Most OpenAI-compatible clients and agent frameworks never set `strict` on their tools, so with `tool_choice="auto"` the model generates tool calls without any grammar and malformed markup can leak into the response. The `--tool-strict-level` option lets the server operator raise the floor for every request that carries tools, independently of what the client declares:
144
145 | Value | Behavior |
146 | --- | --- |
147 | `auto` (default) | Follow the request's tool choice and per-tool strictness. Required/named choices activate structural tags; `tool_choice="auto"` activates them when at least one tool sets `strict: true`. |
148 | `function` | Constrain the tool-call envelope (markup and the function name) for every request with tools, leaving argument contents free unless the client marked the tool `strict: true`. |
149 | `parameter` | Additionally pin argument schemas for every tool, as if every tool had `strict: true`. |
150
151 ```bash
152 vllm serve ... --tool-strict-level function
153 ```
154
155 The floor never relaxes a constraint the request would already receive: tools the client marked `strict: true` keep their schemas at every level. With `tool_choice="auto"`, the grammar does not force a tool call; a plain text response stays valid. `VLLM_ENFORCE_STRICT_TOOL_CALLING=false` disables structural tags entirely and takes precedence over this option.
156
157 ## Automatic Function Calling
158
159 To enable this feature, you should set the following flags:
160
161 * `--enable-auto-tool-choice` -- **mandatory** Auto tool choice. It tells vLLM that you want to enable the model to generate its own tool calls when it
162 deems appropriate.
163 * `--tool-call-parser` -- select the tool parser to use (listed below). Additional tool parsers
164 will continue to be added in the future. You can also register your own tool parsers in the `--tool-parser-plugin`.
165 * `--tool-parser-plugin` -- **optional** tool parser plugin used to register user defined tool parsers into vllm, the registered tool parser name can be specified in `--tool-call-parser`.
166 * `--chat-template` -- **optional** for auto tool choice. It's the path to the chat template which handles `tool`-role messages and `assistant`-role messages
167 that contain previously generated tool calls. Hermes, Mistral and Llama models have tool-compatible chat templates in their
168 `tokenizer_config.json` files, but you can specify a custom template. This argument can be set to `tool_use` if your model has a tool use-specific chat
169 template configured in the `tokenizer_config.json`. In this case, it will be used per the `transformers` specification. More on this [here](https://huggingface.co/docs/transformers/en/chat_templating#why-do-some-models-have-multiple-templates)
170 from HuggingFace; and you can find an example of this in a `tokenizer_config.json` [here](https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B/blob/main/tokenizer_config.json).
171
172 If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
173
174 ### Checkpoint Response Templates (`hf`)
175
176 Checkpoints can define a `response_template` in `tokenizer_config.json` to describe their reasoning, content, and tool-call wire format. Select the
177 `hf` parser to parse output from this metadata instead of a model-specific parser. Like `--tokenizer-mode hf`, it relies on the checkpoint's
178 Hugging Face metadata rather than model-specific code:
179
180 ```bash
181 vllm serve <model> \
182 --enable-auto-tool-choice \
183 --tool-call-parser hf \
184 --reasoning-parser hf
185 ```
186
187 Either parser can be selected on its own. Startup fails if the tokenizer has no `response_template`, or if it lacks the `thinking` or `tool_calls`
188 field the selected parser needs. The `hf` parser cannot be combined with other reasoning or tool call parsers.
189
190 When streaming, each tool call is emitted whole once its region parses. A call cut off before its closer, for example by a closer that is also a
191 stop token, is parsed from the text generated so far; a call that does not parse is dropped.
192
193 Response templates currently provide parsing, not format-specific constrained decoding. The parser therefore rejects requests for strict tools,
194 required or named tool choice, or `parallel_tool_calls=false`. A custom server or request chat template is allowed, but the parser still expects the
195 checkpoint's output format.
196
197 !!! note
198 With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
199
200 ### Hermes Models (`hermes`)
201
202 All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported.
203
204 * `NousResearch/Hermes-2-Pro-*`
205 * `NousResearch/Hermes-2-Theta-*`
206 * `NousResearch/Hermes-3-*`
207
208 *Note that the Hermes 2 **Theta** models are known to have degraded tool call quality and capabilities due to the merge
209 step in their creation*.
210
211 Flags: `--tool-call-parser hermes`
212
213 ### Mistral Models (`mistral`)
214
215 Supported models:
216
217 * `mistralai/Mistral-7B-Instruct-v0.3` (confirmed)
218 * Additional Mistral function-calling models are compatible as well.
219
220 Known issues:
221
222 1. Mistral 7B struggles to generate parallel tool calls correctly.
223 2. **For Transformers tokenization backend only**: Mistral's `tokenizer_config.json` chat template requires tool call IDs that are exactly 9 digits, which is
224 much shorter than what vLLM generates. Since an exception is thrown when this condition
225 is not met, the following additional chat templates are provided:
226
227 * [examples/tool_chat_template_mistral.jinja](../../examples/tool_chat_template_mistral.jinja) - this is the "official" Mistral chat template, but tweaked so that
228 it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)
229 * [examples/tool_chat_template_mistral_parallel.jinja](../../examples/tool_chat_template_mistral_parallel.jinja) - this is a "better" version that adds a tool-use system prompt
230 when tools are provided, that results in much better reliability when working with parallel tool calling.
231
232 Recommended flags:
233
234 1. To use the official Mistral AI's format:
235
236 `--tool-call-parser mistral`
237
238 2. To use the Transformers format when available:
239
240 `--tokenizer_mode hf --config_format hf --load_format hf --tool-call-parser mistral --chat-template examples/tool_chat_template_mistral_parallel.jinja`
241
242 !!! note
243 Models officially released by Mistral AI have two possible formats:
244
245 1. The official format that is used by default with `auto` or `mistral` arguments:
246
247 `--tokenizer_mode mistral --config_format mistral --load_format mistral`
248 This format uses [mistral-common](https://github.com/mistralai/mistral-common), the Mistral AI's tokenizer backend.
249
250 2. The Transformers format, when available, that is used with `hf` arguments:
251
252 `--tokenizer_mode hf --config_format hf --load_format hf --chat-template examples/tool_chat_template_mistral_parallel.jinja`
253
254 ### Llama Models (`llama3_json`)
255
256 Supported models:
257
258 All Llama 3.1, 3.2 and 4 models should be supported.
259
260 * `meta-llama/Llama-3.1-*`
261 * `meta-llama/Llama-3.2-*`
262 * `meta-llama/Llama-4-*`
263
264 The tool calling that is supported is the [JSON-based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) introduced by the Llama-3.2 models, see the `pythonic` tool parser below. As for Llama 4 models, it is recommended to use the `llama4_pythonic` tool parser.
265
266 Other tool calling formats like the built-in python tool calling or custom tool calling are not supported.
267
268 Known issues:
269
270 1. Parallel tool calls are not supported for Llama 3, but it is supported in Llama 4 models.
271 2. The model can generate parameters in an incorrect format, such as generating
272 an array serialized as string instead of an array.
273
274 VLLM provides two JSON-based chat templates for Llama 3.1 and 3.2:
275
276 * [examples/tool_chat_template_llama3.1_json.jinja](../../examples/tool_chat_template_llama3.1_json.jinja) - this is the "official" chat template for the Llama 3.1
277 models, but tweaked so that it works better with vLLM.
278 * [examples/tool_chat_template_llama3.2_json.jinja](../../examples/tool_chat_template_llama3.2_json.jinja) - this extends upon the Llama 3.1 chat template by adding support for
279 images.
280
281 Recommended flags: `--tool-call-parser llama3_json --chat-template {see_above}`
282
283 VLLM also provides a pythonic and JSON-based chat template for Llama 4, but pythonic tool calling is recommended:
284
285 * [examples/tool_chat_template_llama4_pythonic.jinja](../../examples/tool_chat_template_llama4_pythonic.jinja) - this is based on the [official chat template](https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/) for the Llama 4 models.
286
287 For Llama 4 model, use `--tool-call-parser llama4_pythonic --chat-template examples/tool_chat_template_llama4_pythonic.jinja`.
288
289 ### IBM Granite
290
291 Supported models:
292
293 * `ibm-granite/granite-4.0-h-small` and other Granite 4.0 models
294
295 Recommended flags: `--tool-call-parser granite4`
296
297 * `ibm-granite/granite-3.0-8b-instruct`
298
299 Recommended flags: `--tool-call-parser granite --chat-template examples/tool_chat_template_granite.jinja`
300
301 [examples/tool_chat_template_granite.jinja](../../examples/tool_chat_template_granite.jinja): this is a modified chat template from the original on Hugging Face. Parallel function calls are supported.
302
303 * `ibm-granite/granite-3.1-8b-instruct`
304
305 Recommended flags: `--tool-call-parser granite`
306
307 The chat template from Huggingface can be used directly. Parallel function calls are supported.
308
309 * `ibm-granite/granite-20b-functioncalling`
310
311 Recommended flags: `--tool-call-parser granite-20b-fc --chat-template examples/tool_chat_template_granite_20b_fc.jinja`
312
313 [examples/tool_chat_template_granite_20b_fc.jinja](../../examples/tool_chat_template_granite_20b_fc.jinja): this is a modified chat template from the original on Hugging Face, which is not vLLM-compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.
314
315 ### InternLM Models (`internlm`)
316
317 Supported models:
318
319 * `internlm/internlm2_5-7b-chat` (confirmed)
320 * Additional internlm2.5 function-calling models are compatible as well
321
322 Known issues:
323
324 * Although this implementation also supports InternLM2, the tool call results are not stable when testing with the `internlm/internlm2-chat-7b` model.
325
326 Recommended flags: `--tool-call-parser internlm --chat-template examples/tool_chat_template_internlm2_tool.jinja`
327
328 ### Jamba Models (`jamba`)
329
330 AI21's Jamba-1.5 models are supported.
331
332 * `ai21labs/AI21-Jamba-1.5-Mini`
333 * `ai21labs/AI21-Jamba-1.5-Large`
334
335 Flags: `--tool-call-parser jamba`
336
337 ### xLAM Models (`xlam`)
338
339 The xLAM tool parser is designed to support models that generate tool calls in various JSON formats. It detects function calls in several different output styles:
340
341 1. Direct JSON arrays: Output strings that are JSON arrays starting with `[` and ending with `]`
342 2. Thinking tags: Using `<think>...</think>` tags containing JSON arrays
343 3. Code blocks: JSON in code blocks (```json ...```)
344 4. Tool calls tags: Using `[TOOL_CALLS]` or `<tool_call>...</tool_call>` tags
345
346 Parallel function calls are supported, and the parser can effectively separate text content from tool calls.
347
348 Supported models:
349
350 * Salesforce Llama-xLAM models: `Salesforce/Llama-xLAM-2-8B-fc-r`, `Salesforce/Llama-xLAM-2-70B-fc-r`
351 * Qwen-xLAM models: `Salesforce/xLAM-1B-fc-r`, `Salesforce/xLAM-3B-fc-r`, `Salesforce/Qwen-xLAM-32B-fc-r`
352
353 Flags:
354
355 * For Llama-based xLAM models: `--tool-call-parser xlam --chat-template examples/tool_chat_template_xlam_llama.jinja`
356 * For Qwen-based xLAM models: `--tool-call-parser xlam --chat-template examples/tool_chat_template_xlam_qwen.jinja`
357
358 ### Qwen Models
359
360 For Qwen2.5, the chat template in tokenizer_config.json has already included support for the Hermes-style tool use. Therefore, you can use the `hermes` parser to enable tool calls for Qwen models. For more detailed information, please refer to the official [Qwen documentation](https://qwen.readthedocs.io/en/latest/framework/function_call.html#vllm)
361
362 * `Qwen/Qwen2.5-*`
363 * `Qwen/QwQ-32B`
364
365 Flags: `--tool-call-parser hermes`
366
367 ### DeepSeek-V3 Models (`deepseek_v3`)
368
369 Supported models:
370
371 * `deepseek-ai/DeepSeek-V3-0324` (use with [examples/tool_chat_template_deepseekv3.jinja](../../examples/tool_chat_template_deepseekv3.jinja))
372 * `deepseek-ai/DeepSeek-R1-0528` (use with [examples/tool_chat_template_deepseekr1.jinja](../../examples/tool_chat_template_deepseekr1.jinja))
373
374 Flags: `--tool-call-parser deepseek_v3 --chat-template {see_above}`
375
376 ### DeepSeek-V3.1 Models (`deepseek_v31`)
377
378 Supported models:
379
380 * `deepseek-ai/DeepSeek-V3.1` (use with [examples/tool_chat_template_deepseekv31.jinja](../../examples/tool_chat_template_deepseekv31.jinja))
381
382 Flags: `--tool-call-parser deepseek_v31 --chat-template {see_above}`
383
384 ### OpenAI OSS Models (`openai`)
385
386 Supported models:
387
388 * `openai/gpt-oss-20b`
389 * `openai/gpt-oss-120b`
390
391 Flags: `--tool-call-parser openai`
392
393 ### Kimi-K2 Models (`kimi_k2`)
394
395 Supported models:
396
397 * `moonshotai/Kimi-K2-Instruct`
398
399 Flags: `--tool-call-parser kimi_k2`
400
401 ### Hunyuan Models (`hunyuan_a13b`)
402
403 Supported models:
404
405 * `tencent/Hunyuan-A13B-Instruct` (The chat template is already included in the Hugging Face model files.)
406
407 Flags:
408
409 * For non-reasoning: `--tool-call-parser hunyuan_a13b`
410 * For reasoning: `--tool-call-parser hunyuan_a13b --reasoning-parser hunyuan_a13b`
411
412 ### Cohere Command (`cohere_command3` / `cohere_command4`)
413
414 Supported models:
415
416 * [`CohereLabs/command-a-reasoning-08-2025`](https://huggingface.co/CohereLabs/command-a-reasoning-08-2025) (`cohere_command3`)
417 * [`CohereLabs/command-a-plus-05-2026`](https://huggingface.co/CohereLabs/command-a-plus-05-2026-bf16) and [`CohereLabs/North-Mini-Code-1.0`](https://huggingface.co/CohereLabs/North-Mini-Code-1.0) (`cohere_command4`)
418
419 Flags: `--tool-call-parser cohere_command4 --reasoning-parser cohere_command4`.
420
421 Note: the Cohere parser requires the `cohere_melody` package, which is not installed by default. Before using this parser please install the [cohere_melody](https://pypi.org/project/cohere-melody/) package.
422
423 ### LongCat-Flash-Chat Models (`longcat`)
424
425 Supported models:
426
427 * `meituan-longcat/LongCat-Flash-Chat`
428 * `meituan-longcat/LongCat-Flash-Chat-FP8`
429
430 Flags: `--tool-call-parser longcat`
431
432 ### GLM-4.5 Models (`glm45`)
433
434 Supported models:
435
436 * `zai-org/GLM-4.5`
437 * `zai-org/GLM-4.5-Air`
438 * `zai-org/GLM-4.6`
439
440 Flags: `--tool-call-parser glm45`
441
442 ### GLM-4.7 Models (`glm47`)
443
444 Supported models:
445
446 * `zai-org/GLM-4.7`
447 * `zai-org/GLM-4.7-Flash`
448
449 Flags: `--tool-call-parser glm47`
450
451 ### FunctionGemma Models (`functiongemma`)
452
453 Google's FunctionGemma is a lightweight (270M parameter) model specifically designed for function calling.
454 It's built on Gemma 3 and optimized for edge deployment on devices like laptops and phones.
455
456 Supported models:
457
458 * `google/functiongemma-270m-it`
459
460 FunctionGemma uses a unique output format with `<start_function_call>` and `<end_function_call>` tags:
461
462 ```text
463 <start_function_call>call:get_weather{location:<escape>London<escape>}<end_function_call>
464 ```
465
466 The model is designed to be fine-tuned for specific function-calling tasks for best results.
467
468 Flags: `--tool-call-parser functiongemma --chat-template examples/tool_chat_template_functiongemma.jinja`
469
470 !!! note
471 FunctionGemma is intended to be fine-tuned for your specific function-calling task.
472 The base model provides general function calling capabilities, but best results
473 are achieved with task-specific fine-tuning. See Google's [FunctionGemma documentation](https://ai.google.dev/gemma/docs/functiongemma) for fine-tuning guides.
474
475 ### Qwen3-Coder Models (`qwen3_xml`)
476
477 Supported models:
478
479 * `Qwen/Qwen3-Coder-480B-A35B-Instruct`
480 * `Qwen/Qwen3-Coder-30B-A3B-Instruct`
481
482 Flags: `--tool-call-parser qwen3_xml`
483
484 ### MiMo-V2.6 Models (`mimo`)
485
486 Supported models:
487
488 * `XiaomiMiMo/MiMo-V2.6-Pro-RL`
489 * `XiaomiMiMo/MiMo-V2.6-Flash-RL`
490
491 Use the checkpoint's chat template with `--tool-call-parser mimo --reasoning-parser mimo --enable-auto-tool-choice`.
492 For schema-constrained tool arguments, set `strict: true` inside each tool's `function`, or use `--tool-strict-level parameter`.
493 The parser supports `auto`, `required`, named tools, and `parallel_tool_calls=False`, preserving leading/trailing whitespace in string arguments.
494
495 ### Step-3.5 and Step-3.7 Models (`step3p5`)
496
497 Supported models:
498
499 * `stepfun-ai/Step-3.5-Flash`
500 * `stepfun-ai/Step-3.5-Flash-FP8`
501 * `stepfun-ai/Step-3.7-Flash`
502
503 Flags: `--tool-call-parser step3p5 --reasoning-parser step3p5 --enable-auto-tool-choice`
504
505 The parser supports `auto`, `required`, and named tools.
506
507 ### Olmo 3 Models (`olmo3`)
508
509 Olmo 3 models output tool calls in a format that is very similar to the one expected by the `pythonic` parser (see below), with a few differences. Each tool call is a pythonic string, but the parallel tool calls are newline-delimited, and the calls are wrapped within XML tags as `<function_calls>..</function_calls>`. In addition, the parser also allows JSON boolean and null literals (`true`, `false`, and `null`) in addition to the pythonic ones (`True`, `False`, and `None`).
510
511 Supported models:
512
513 * `allenai/Olmo-3-7B-Instruct`
514 * `allenai/Olmo-3-32B-Think`
515
516 Flags: `--tool-call-parser olmo3`
517
518 ### Gigachat 3 Models (`gigachat3`)
519
520 Use chat template from the Hugging Face model files.
521
522 Supported models:
523
524 * `ai-sage/GigaChat3-702B-A36B-preview`
525 * `ai-sage/GigaChat3-702B-A36B-preview-bf16`
526 * `ai-sage/GigaChat3-10B-A1.8B`
527 * `ai-sage/GigaChat3-10B-A1.8B-bf16`
528
529 Flags: `--tool-call-parser gigachat3`
530
531 ### Apertus Models (`apertus`)
532
533 Use the chat template from the examples folder; it fixes several OpenAI compatibility issues: `--chat-template /vllm-workspace/examples/tool_chat_template_apertus.jinja`
534
535 Supported models:
536
537 * `swiss-ai/Apertus-8B-Instruct-2509`
538 * `swiss-ai/Apertus-70B-Instruct-2509`
539
540 Flags: `--tool-call-parser apertus`
541
542 ### Models with Pythonic Tool Calls (`pythonic`)
543
544 A growing number of models output a python list to represent tool calls instead of using JSON. This has the advantage of inherently supporting parallel tool calls and removing ambiguity around the JSON schema required for tool calls. The `pythonic` tool parser can support such models.
545
546 As a concrete example, these models may look up the weather in San Francisco and Seattle by generating:
547
548 ```python
549 [get_weather(city='San Francisco', metric='celsius'), get_weather(city='Seattle', metric='celsius')]
550 ```
551
552 Limitations:
553
554 * The model must not generate both text and tool calls in the same generation. This may not be hard to change for a specific model, but the community currently lacks consensus on which tokens to emit when starting and ending tool calls. (In particular, the Llama 3.2 models emit no such tokens.)
555 * Llama's smaller models struggle to use tools effectively.
556
557 Example supported models:
558
559 * `meta-llama/Llama-3.2-1B-Instruct` ⚠️ (use with [examples/tool_chat_template_llama3.2_pythonic.jinja](../../examples/tool_chat_template_llama3.2_pythonic.jinja))
560 * `meta-llama/Llama-3.2-3B-Instruct` ⚠️ (use with [examples/tool_chat_template_llama3.2_pythonic.jinja](../../examples/tool_chat_template_llama3.2_pythonic.jinja))
561 * `Team-ACE/ToolACE-8B` (use with [examples/tool_chat_template_toolace.jinja](../../examples/tool_chat_template_toolace.jinja))
562 * `fixie-ai/ultravox-v0_4-ToolACE-8B` (use with [examples/tool_chat_template_toolace.jinja](../../examples/tool_chat_template_toolace.jinja))
563 * `meta-llama/Llama-4-Scout-17B-16E-Instruct` ⚠️ (use with [examples/tool_chat_template_llama4_pythonic.jinja](../../examples/tool_chat_template_llama4_pythonic.jinja))
564 * `meta-llama/Llama-4-Maverick-17B-128E-Instruct` ⚠️ (use with [examples/tool_chat_template_llama4_pythonic.jinja](../../examples/tool_chat_template_llama4_pythonic.jinja))
565
566 Flags: `--tool-call-parser pythonic --chat-template {see_above}`
567
568 !!! warning
569 Llama's smaller models frequently fail to emit tool calls in the correct format. Results may vary depending on the model.
570
571 ## Benchmarking Tool-Calling Performance
572
573 To measure serving latency and throughput on realistic tool-calling traffic,
574 use the BFCL (Berkeley Function Calling Leaderboard) dataset with
575 `vllm bench serve`. See the [BFCL benchmark example](../benchmarking/cli.md#bfcl-tool-calling-benchmark)
576 for the full server + client commands.
577
578 ## How to Write a Tool Parser Plugin
579
580 A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in [vllm/tool_parsers/hermes_tool_parser.py](../../vllm/tool_parsers/hermes_tool_parser.py).
581
582 Here is a summary of a plugin file:
583
584 ??? code
585
586 ```python
587
588 # import the required packages
589
590 # define a tool parser and register it to vllm
591 # the name list in register_module can be used
592 # in --tool-call-parser. you can define as many
593 # tool parsers as you want here.
594 class ExampleToolParser(ToolParser):
595 def __init__(self, tokenizer: TokenizerLike):
596 super().__init__(tokenizer)
597
598 # adjust request. e.g.: set skip special tokens
599 # to False for tool call output.
600 def adjust_request(self, request: ChatCompletionRequest | ResponsesRequest) -> ChatCompletionRequest | ResponsesRequest:
601 return request
602
603 # implement the tool call parse for stream call
604 def extract_tool_calls_streaming(
605 self,
606 previous_text: str,
607 current_text: str,
608 delta_text: str,
609 previous_token_ids: Sequence[int],
610 current_token_ids: Sequence[int],
611 delta_token_ids: Sequence[int],
612 request: ChatCompletionRequest,
613 ) -> DeltaMessage | None:
614 return delta
615
616 # implement the tool parse for non-stream call
617 def extract_tool_calls(
618 self,
619 model_output: str,
620 request: ChatCompletionRequest,
621 ) -> ExtractedToolCallInformation:
622 return ExtractedToolCallInformation(tools_called=False,
623 tool_calls=[],
624 content=text)
625 # register the tool parser to ToolParserManager
626 ToolParserManager.register_lazy_module(
627 name="example",
628 module_path="vllm.tool_parsers.example",
629 class_name="ExampleToolParser",
630 )
631
632 ```
633
634 Then you can use this plugin in the command line like this.
635
636 ```bash
637 --enable-auto-tool-choice \
638 --tool-parser-plugin <absolute path of the plugin file>
639 --tool-call-parser example \
640 --chat-template <your chat template> \
641 ```