A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 6, 14:30 UTC
[Doc] Clarify schema enforcement behavior for tool_choice modes (#37064)
Signed-off-by: yfama <yuchengu@gmail.com>
Yuchen Fama committed
Mar 16, 2026 at 18:27 UTC
31a458c0913e2c498da004e16ba2ac922bcebe96
1 file changed
+24
docs/features/tool_calling.md
+24
@@ -107,6 +107,27 @@ vLLM supports the `tool_choice='none'` option in the chat completion API. When t
107
!!! note
108
When tools are specified in the request, vLLM includes tool definitions in the prompt by default, regardless of the `tool_choice` setting. To exclude tool definitions when `tool_choice='none'`, use the `--exclude-tools-when-tool-choice-none` option.
109
110
+## Constrained Decoding Behavior
111
+
112
+Whether vLLM enforces the tool parameter schema during generation depends on the `tool_choice` mode:
113
+
114
+| `tool_choice` value | Schema-constrained decoding | Behavior |
115
+| --- | --- | --- |
116
+| Named function | Yes (via structured outputs backend) | Arguments are guaranteed to be valid JSON conforming to the function's parameter schema. |
117
+| `"required"` | Yes (via structured outputs backend) | Same as named function. The model must produce at least one tool call. |
118
+| `"auto"` | No | The model generates freely. A tool-call parser extracts tool calls from the raw text. Arguments may be malformed or not match the schema. |
119
+| `"none"` | N/A | No tool calls are produced. |
120
+
121
+When schema conformance matters, prefer `tool_choice="required"` or named function calling over `"auto"`.
122
+
123
+### Strict Mode (`strict` parameter)
124
+
125
+The [OpenAI API](https://platform.openai.com/docs/guides/function-calling#strict-mode) supports a `strict` field on function definitions. When set to `true`, OpenAI uses constrained decoding to guarantee that tool-call arguments match the function schema, even in `tool_choice="auto"` mode.
126
+
127
+vLLM **does not implement** `strict` mode today. The `strict` field is accepted in requests (to avoid breaking clients that set it), but it has no effect on decoding behavior. In auto mode, argument validity depends entirely on the model's output quality and the parser's extraction logic.
128
+
129
+Tracking issues: [#15526](https://github.com/vllm-project/vllm/issues/15526), [#16313](https://github.com/vllm-project/vllm/issues/16313).
130
+
131
## Automatic Function Calling
132
133
To enable this feature, you should set the following flags:
@@ -124,6 +145,9 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
145
146
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
147
148
+!!! note
149
+ With `tool_choice="auto"`, tool-call arguments are extracted from the model's raw text output by the selected parser. No schema-level constraint is applied during decoding, so arguments may occasionally be malformed or violate the function's parameter schema. See [Constrained Decoding Behavior](#constrained-decoding-behavior) for details.
150
+
151
### Hermes Models (`hermes`)
152
153
All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported.