@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 0d115460a7

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 15, 11:00 UTC

[Docs] Use gh-file to add links to tool_calling.md (#17709)

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

Michael Yao committed May 6, 2025 at 23:27 UTC 0d115460a718cd35992415829884de0852c14fa3

1 file changed +14 -14

docs/source/features/tool_calling.md

+14 -14

@@ -141,9 +141,9 @@ Known issues:

141 much shorter than what vLLM generates. Since an exception is thrown when this condition

142 is not met, the following additional chat templates are provided:

143

144 -* `examples/tool_chat_template_mistral.jinja` - this is the "official" Mistral chat template, but tweaked so that

144 +* <gh-file:examples/tool_chat_template_mistral.jinja> - this is the "official" Mistral chat template, but tweaked so that

145 it works with vLLM's tool call IDs (provided `tool_call_id` fields are truncated to the last 9 digits)

146 -* `examples/tool_chat_template_mistral_parallel.jinja` - this is a "better" version that adds a tool-use system prompt

146 +* <gh-file:examples/tool_chat_template_mistral_parallel.jinja> - this is a "better" version that adds a tool-use system prompt

147 when tools are provided, that results in much better reliability when working with parallel tool calling.

148

149 Recommended flags: `--tool-call-parser mistral --chat-template examples/tool_chat_template_mistral_parallel.jinja`

@@ -170,15 +170,15 @@ Known issues:

170

171 VLLM provides two JSON based chat templates for Llama 3.1 and 3.2:

172

173 -* `examples/tool_chat_template_llama3.1_json.jinja` - this is the "official" chat template for the Llama 3.1

173 +* <gh-file:examples/tool_chat_template_llama3.1_json.jinja> - this is the "official" chat template for the Llama 3.1

174 models, but tweaked so that it works better with vLLM.

175 -* `examples/tool_chat_template_llama3.2_json.jinja` - this extends upon the Llama 3.1 chat template by adding support for

175 +* <gh-file:examples/tool_chat_template_llama3.2_json.jinja> - this extends upon the Llama 3.1 chat template by adding support for

176 images.

177

178 Recommended flags: `--tool-call-parser llama3_json --chat-template {see_above}`

179

180 VLLM also provides a JSON based chat template for Llama 4:

181 -* `examples/tool_chat_template_llama4_json.jinja` - this is based on the "official" chat template for the Llama 4

181 +* <gh-file:examples/tool_chat_template_llama4_json.jinja> - this is based on the "official" chat template for the Llama 4

182 models, but tweaked so that it works better with vLLM.

183

184 For Llama 4 use `--tool-call-parser llama4_json examples/tool_chat_template_llama4_json.jinja`.

@@ -191,7 +191,7 @@ Supported models:

191

192 Recommended flags: `--tool-call-parser granite --chat-template examples/tool_chat_template_granite.jinja`

193

194 -`examples/tool_chat_template_granite.jinja`: this is a modified chat template from the original on Huggingface. Parallel function calls are supported.

194 +<gh-file:examples/tool_chat_template_granite.jinja>: this is a modified chat template from the original on Huggingface. Parallel function calls are supported.

195

196 * `ibm-granite/granite-3.1-8b-instruct`

197

@@ -203,7 +203,7 @@ The chat template from Huggingface can be used directly. Parallel function calls

203

204 Recommended flags: `--tool-call-parser granite-20b-fc --chat-template examples/tool_chat_template_granite_20b_fc.jinja`

205

206 -`examples/tool_chat_template_granite_20b_fc.jinja`: this is a modified chat template from the original on Huggingface, which is not vLLM compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.

206 +<gh-file:examples/tool_chat_template_granite_20b_fc.jinja>: this is a modified chat template from the original on Huggingface, which is not vLLM compatible. It blends function description elements from the Hermes template and follows the same system prompt as "Response Generation" mode from [the paper](https://arxiv.org/abs/2407.00121). Parallel function calls are supported.

207

208 ### InternLM Models (`internlm`)

209

@@ -253,12 +253,12 @@ Limitations:

253

254 Example supported models:

255

256 -* `meta-llama/Llama-3.2-1B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)

257 -* `meta-llama/Llama-3.2-3B-Instruct`\* (use with `examples/tool_chat_template_llama3.2_pythonic.jinja`)

258 -* `Team-ACE/ToolACE-8B` (use with `examples/tool_chat_template_toolace.jinja`)

259 -* `fixie-ai/ultravox-v0_4-ToolACE-8B` (use with `examples/tool_chat_template_toolace.jinja`)

260 -* `meta-llama/Llama-4-Scout-17B-16E-Instruct`\* (use with `examples/tool_chat_template_llama4_pythonic.jinja`)

261 -* `meta-llama/Llama-4-Maverick-17B-128E-Instruct`\* (use with `examples/tool_chat_template_llama4_pythonic.jinja`)

256 +* `meta-llama/Llama-3.2-1B-Instruct`\* (use with <gh-file:examples/tool_chat_template_llama3.2_pythonic.jinja>)

257 +* `meta-llama/Llama-3.2-3B-Instruct`\* (use with <gh-file:examples/tool_chat_template_llama3.2_pythonic.jinja>)

258 +* `Team-ACE/ToolACE-8B` (use with <gh-file:examples/tool_chat_template_toolace.jinja>)

259 +* `fixie-ai/ultravox-v0_4-ToolACE-8B` (use with <gh-file:examples/tool_chat_template_toolace.jinja>)

260 +* `meta-llama/Llama-4-Scout-17B-16E-Instruct`\* (use with <gh-file:examples/tool_chat_template_llama4_pythonic.jinja>)

261 +* `meta-llama/Llama-4-Maverick-17B-128E-Instruct`\* (use with <gh-file:examples/tool_chat_template_llama4_pythonic.jinja>)

262

263 Flags: `--tool-call-parser pythonic --chat-template {see_above}`

264

@@ -270,7 +270,7 @@ Llama's smaller models frequently fail to emit tool calls in the correct format.

270

271 ## How to write a tool parser plugin

272

273 -A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py.

273 +A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in <gh-file:vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py>.

274

275 Here is a summary of a plugin file:

276