A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 13:30 UTC
[Doc] Updated Llama section in tool calling docs to have llama 3.2 config info (#16857)
Signed-off-by: jmho <jaylenho734@gmail.com>
Justin Ho committed
Apr 18, 2025 at 19:28 UTC
490b1698a5c789913b0a62109d9592621449ebcb
1 file changed
+13
-8
docs/source/features/tool_calling.md
+13
-8
@@ -152,12 +152,13 @@ Recommended flags: `--tool-call-parser mistral --chat-template examples/tool_cha
152
153
Supported models:
154
155
-* `meta-llama/Meta-Llama-3.1-8B-Instruct`
156
-* `meta-llama/Meta-Llama-3.1-70B-Instruct`
157
-* `meta-llama/Meta-Llama-3.1-405B-Instruct`
158
-* `meta-llama/Meta-Llama-3.1-405B-Instruct-FP8`
155
+All Llama 3.1 and 3.2 models should be supported.
156
+
157
+* `meta-llama/Llama-3.1-*`
158
+* `meta-llama/Llama-3.2-*`
159
+
160
+The tool calling that is supported is the [JSON based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) introduced by the Llama-3.2 models, see the `pythonic` tool parser below.
161
160
-The tool calling that is supported is the [JSON based tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/#json-based-tool-calling). For [pythonic tool calling](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/text_prompt_format.md#zero-shot-function-calling) in Llama-3.2 models, see the `pythonic` tool parser below.
162
Other tool calling formats like the built in python tool calling or custom tool calling are not supported.
163
164
Known issues:
@@ -166,10 +167,14 @@ Known issues:
167
2. The model can generate parameters with a wrong format, such as generating
168
an array serialized as string instead of an array.
169
169
-The `tool_chat_template_llama3_json.jinja` file contains the "official" Llama chat template, but tweaked so that
170
-it works better with vLLM.
170
+VLLM provides two JSON based chat templates for Llama 3.1 and 3.2:
171
+
172
+* `examples/tool_chat_template_llama3.1_json.jinja` - this is the "official" chat template for the Llama 3.1
173
+models, but tweaked so that it works better with vLLM.
174
+* `examples/tool_chat_template_llama3.2_json.jinja` - this extends upon the Llama 3.1 chat template by adding support for
175
+images.
176
172
-Recommended flags: `--tool-call-parser llama3_json --chat-template examples/tool_chat_template_llama3_json.jinja`
177
+Recommended flags: `--tool-call-parser llama3_json --chat-template {see_above}`
178
179
#### IBM Granite
180