A high-throughput and memory-efficient inference and serving engine for LLMs
[Frontend] Skip structural tags for auto tool_choice without strict mode (#45600)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Flora Feng committed
Jun 15, 2026 at 15:55 UTC
cd9078fe59111b02459320108bae8f72b1ddf569
8 files changed
+111
-24
docs/features/tool_calling.md
+5
-9
@@ -109,18 +109,18 @@ vLLM supports the `tool_choice='none'` option in the chat completion API. When t
109
110
## Constrained Decoding Behavior
111
112
-Whether vLLM enforces the tool parameter schema during generation depends on the `tool_choice` mode:
112
+Whether vLLM enforces the tool parameter schema during generation depends on the `tool_choice` mode and the per-tool `strict` field:
113
114
| `tool_choice` value | Schema-constrained decoding | Behavior |
115
| --- | --- | --- |
116
| Named function | Yes (via structured outputs backend) | Arguments are guaranteed to be valid JSON conforming to the function's parameter schema. |
117
| `"required"` | Yes (via structured outputs backend) | Same as named function. The model must produce at least one tool call. |
118
-| `"auto"` | Depends on the parser | Model-specific structural-tag parsers can constrain tool-call arguments with structured outputs. Other parsers generate freely and extract tool calls from raw text. |
118
+| `"auto"` | Only when `strict: true` is set on at least one tool | Structural-tag parsers constrain tool-call arguments when a tool opts in with `strict: true`. Without it, the model generates freely and tool calls are extracted from raw text. |
119
| `"none"` | N/A | No tool calls are produced. |
120
121
### Strict Mode
122
123
-Strict tool calling makes function-call arguments adhere to the function schema instead of relying only on best-effort parsing. vLLM implements strict tool calling for structural-tag based tool parsers by using the structured outputs backend under the hood.
123
+For `tool_choice="required"` or named function calling, structural-tag constraints are always applied regardless of the `strict` field. For `tool_choice="auto"`, setting `strict: true` on at least one tool opts in to structural-tag constraints; without it, the model generates freely and tool calls are extracted from raw text. The `strict` field is supported across all three API surfaces: Chat Completion, Responses, and Anthropic Messages.
124
125
For best compatibility with strict schema enforcement, define tool parameter schemas in the OpenAI strict-schema style:
126
@@ -128,16 +128,12 @@ For best compatibility with strict schema enforcement, define tool parameter sch
128
* Mark all fields in `properties` as required.
129
* Represent optional fields by allowing `null`, for example `{"type": ["string", "null"]}`.
130
131
-vLLM controls structural-tag strict tool calling with the `VLLM_ENFORCE_STRICT_TOOL_CALLING` environment variable. It defaults to `true`.
131
+vLLM also provides a global toggle via the `VLLM_ENFORCE_STRICT_TOOL_CALLING` environment variable (defaults to `true`). When set to `false`, vLLM does not attach structural tags for tool calling regardless of the per-tool `strict` field. This environment variable only affects structural-tag based tool calling; it does not change schema-derived structured outputs used by named function calling or `tool_choice="required"`.
132
133
```bash
134
VLLM_ENFORCE_STRICT_TOOL_CALLING=false vllm serve ...
135
```
136
137
-When this variable is `true`, structural-tag based tool parsers attach a structural tag to the request, so the structured outputs backend can constrain the model-specific tool-call format and function-call arguments. When it is `false`, vLLM does not attach structural tags for tool calling. In that case, `tool_choice="auto"` falls back to best-effort parser extraction from the raw model output, and no structural-tag constraint is applied.
138
-
139
-This environment variable only affects structural-tag based tool calling. It does not change schema-derived structured outputs used by named function calling or `tool_choice="required"`.
140
-
137
## Automatic Function Calling
138
139
To enable this feature, you should set the following flags:
@@ -156,7 +152,7 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
152
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
153
154
!!! note
159
- With `tool_choice="auto"`, schema-level constraint depends on the selected parser and `VLLM_ENFORCE_STRICT_TOOL_CALLING`. Structural-tag parsers can enforce tool-call constraints when it is `true`; when it is `false`, or when the selected parser has no structural-tag support, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
155
+ With `tool_choice="auto"`, schema-level constraint requires both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`. When these conditions are met and the selected parser supports structural tags, vLLM constrains tool-call arguments. Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
156
157
### Hermes Models (`hermes`)
158
tests/tool_parsers/test_deepseekv4_tool_parser.py
+19
-1
@@ -216,14 +216,32 @@ def test_streaming_emits_incremental_argument_chunks():
216
}
217
218
219
+def _with_strict(
220
+ tools: list[ChatCompletionToolsParam],
221
+) -> list[ChatCompletionToolsParam]:
222
+ return [
223
+ ChatCompletionToolsParam(
224
+ type=t.type,
225
+ function=FunctionDefinition(
226
+ name=t.function.name,
227
+ description=t.function.description,
228
+ parameters=t.function.parameters,
229
+ strict=True,
230
+ ),
231
+ )
232
+ for t in tools
233
+ ]
234
+
235
+
236
def test_get_vllm_registry_structural_tag_returns_structural_tag(
237
sample_tools: list[ChatCompletionToolsParam],
238
) -> None:
239
parser = make_parser()
240
+ strict_tools = _with_strict(sample_tools)
241
req = ChatCompletionRequest(
242
messages=[],
243
model="m",
226
- tools=sample_tools,
244
+ tools=strict_tools,
245
tool_choice="auto",
246
)
247
tag = parser.get_structural_tag(req)
tests/tool_parsers/test_qwen3coder_tool_parser.py
+22
-2
@@ -14,6 +14,7 @@ from vllm.entrypoints.openai.chat_completion.protocol import (
14
ChatCompletionNamedToolChoiceParam,
15
ChatCompletionRequest,
16
ChatCompletionToolsParam,
17
+ FunctionDefinition,
18
)
19
from vllm.entrypoints.openai.engine.protocol import (
20
DeltaMessage,
@@ -115,6 +116,23 @@ def sample_tools(request):
116
]
117
118
119
+def _with_strict(
120
+ tools: list[ChatCompletionToolsParam],
121
+) -> list[ChatCompletionToolsParam]:
122
+ return [
123
+ ChatCompletionToolsParam(
124
+ type=t.type,
125
+ function=FunctionDefinition(
126
+ name=t.function.name,
127
+ description=t.function.description,
128
+ parameters=t.function.parameters,
129
+ strict=True,
130
+ ),
131
+ )
132
+ for t in tools
133
+ ]
134
+
135
+
136
def _as_chat_completion_tools(
137
tools: list[ChatCompletionToolsParam | FunctionTool],
138
) -> list[ChatCompletionToolsParam]:
@@ -1323,10 +1341,11 @@ def test_get_vllm_registry_structural_tag_returns_structural_tag(
1341
sample_tools: list[ChatCompletionToolsParam],
1342
) -> None:
1343
request_tools = _as_chat_completion_tools(sample_tools)
1344
+ strict_tools = _with_strict(request_tools)
1345
req = ChatCompletionRequest(
1346
messages=[],
1347
model="m",
1329
- tools=request_tools,
1348
+ tools=strict_tools,
1349
tool_choice="auto",
1350
)
1351
tag = qwen3_tool_parser.get_structural_tag(req)
@@ -1364,10 +1383,11 @@ def test_adjust_request_auto_uses_vllm_registry_structural_tag(
1383
tool_parser_cls = Qwen3EngineToolParser
1384
1385
request_tools = _as_chat_completion_tools(sample_tools)
1386
+ strict_tools = _with_strict(request_tools)
1387
req = ChatCompletionRequest(
1388
messages=[],
1389
model="m",
1370
- tools=request_tools,
1390
+ tools=strict_tools,
1391
tool_choice="auto",
1392
include_reasoning=include_reasoning,
1393
)
tests/tool_parsers/test_structural_tag_registry.py
+46
-12
@@ -51,6 +51,24 @@ def sample_tools() -> list[ChatCompletionToolsParam]:
51
]
52
53
54
+@pytest.fixture
55
+def sample_tools_strict() -> list[ChatCompletionToolsParam]:
56
+ return [
57
+ ChatCompletionToolsParam(
58
+ type="function",
59
+ function={
60
+ "name": "get_weather",
61
+ "strict": True,
62
+ "parameters": {
63
+ "type": "object",
64
+ "properties": {"city": {"type": "string"}},
65
+ "required": ["city"],
66
+ },
67
+ },
68
+ )
69
+ ]
70
+
71
+
72
def test_supported_structural_tag_models_include_vllm_builtins():
73
assert SUPPORTED_STRUCTURAL_TAG_MODELS == (
74
XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS | VLLM_BUILTIN_STRUCTURAL_TAG_MODELS
@@ -61,11 +79,11 @@ def test_supported_structural_tag_models_include_vllm_builtins():
79
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
80
def test_get_model_structural_tag_supports_all_xgrammar_builtins(
81
model: str,
64
- sample_tools: list[ChatCompletionToolsParam],
82
+ sample_tools_strict: list[ChatCompletionToolsParam],
83
):
84
tag = get_model_structural_tag(
85
model=model,
68
- tools=sample_tools,
86
+ tools=sample_tools_strict,
87
tool_choice="auto",
88
reasoning=False,
89
)
@@ -219,7 +237,7 @@ def test_non_structural_tag_parser_uses_schema_constraints(
237
238
def test_get_structural_tag_disables_reasoning(
239
monkeypatch: pytest.MonkeyPatch,
222
- sample_tools: list[ChatCompletionToolsParam],
240
+ sample_tools_strict: list[ChatCompletionToolsParam],
241
):
242
captured: list[bool] = []
243
@@ -235,10 +253,10 @@ def test_get_structural_tag_disables_reasoning(
253
request = ChatCompletionRequest(
254
messages=[],
255
model="m",
238
- tools=sample_tools,
256
+ tools=sample_tools_strict,
257
tool_choice="auto",
258
)
241
- parser = Qwen3EngineToolParser(MagicMock(), tools=sample_tools)
259
+ parser = Qwen3EngineToolParser(MagicMock(), tools=sample_tools_strict)
260
261
parser.get_structural_tag(request)
262
@@ -247,7 +265,7 @@ def test_get_structural_tag_disables_reasoning(
265
266
def test_unified_parser_get_structural_tag_disables_reasoning(
267
monkeypatch: pytest.MonkeyPatch,
250
- sample_tools: list[ChatCompletionToolsParam],
268
+ sample_tools_strict: list[ChatCompletionToolsParam],
269
):
270
captured: list[bool] = []
271
@@ -266,10 +284,10 @@ def test_unified_parser_get_structural_tag_disables_reasoning(
284
request = ChatCompletionRequest(
285
messages=[],
286
model="m",
269
- tools=sample_tools,
287
+ tools=sample_tools_strict,
288
tool_choice="auto",
289
)
272
- parser = TestParser(MagicMock(), tools=sample_tools)
290
+ parser = TestParser(MagicMock(), tools=sample_tools_strict)
291
parser.reasoning_parser = MagicMock(adjust_request=lambda request: request)
292
293
parser.adjust_request(request)
@@ -279,7 +297,7 @@ def test_unified_parser_get_structural_tag_disables_reasoning(
297
298
def test_xgrammar_function_parameters_are_preserved(
299
monkeypatch: pytest.MonkeyPatch,
282
- sample_tools: list[ChatCompletionToolsParam],
300
+ sample_tools_strict: list[ChatCompletionToolsParam],
301
):
302
captured: list[list[dict]] = []
303
@@ -294,15 +312,31 @@ def test_xgrammar_function_parameters_are_preserved(
312
313
get_model_structural_tag(
314
model="llama",
297
- tools=sample_tools,
315
+ tools=sample_tools_strict,
316
tool_choice="auto",
317
reasoning=False,
318
)
319
320
assert (
303
- captured[0][0]["function"]["parameters"] == sample_tools[0].function.parameters
321
+ captured[0][0]["function"]["parameters"]
322
+ == sample_tools_strict[0].function.parameters
323
+ )
324
+ assert sample_tools_strict[0].function.parameters is not None
325
+
326
+
327
+@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
328
+def test_auto_tool_choice_skips_structural_tag_without_strict(
329
+ model: str,
330
+ sample_tools: list[ChatCompletionToolsParam],
331
+):
332
+ tag = get_model_structural_tag(
333
+ model=model,
334
+ tools=sample_tools,
335
+ tool_choice="auto",
336
+ reasoning=False,
337
)
305
- assert sample_tools[0].function.parameters is not None
338
+
339
+ assert tag is None
340
341
342
def test_get_function_parameters_relaxes_function_strict_false():
vllm/entrypoints/anthropic/protocol.py
+1
@@ -75,6 +75,7 @@ class AnthropicTool(BaseModel):
75
name: str
76
description: str | None = None
77
input_schema: dict[str, Any]
78
+ strict: bool | None = None
79
defer_loading: bool | None = None
80
81
@field_validator("input_schema")
vllm/entrypoints/anthropic/serving.py
+1
@@ -462,6 +462,7 @@ class AnthropicServingMessages(OpenAIServingChat):
462
"name": tool.name,
463
"description": tool.description,
464
"parameters": tool.input_schema,
465
+ "strict": tool.strict,
466
"defer_loading": tool.defer_loading,
467
},
468
}
vllm/entrypoints/openai/engine/protocol.py
+3
@@ -247,11 +247,14 @@ class FunctionDefinition(OpenAIBaseModel):
247
name: str
248
description: str | None = None
249
parameters: dict[str, Any] | None = None
250
+ strict: bool | None = None
251
defer_loading: bool | None = None
252
253
@model_serializer(mode="wrap")
254
def _serialize(self, handler):
255
data = handler(self)
256
+ if self.strict is None:
257
+ data.pop("strict", None)
258
if self.defer_loading is None:
259
data.pop("defer_loading", None)
260
return data
vllm/tool_parsers/structural_tag_registry.py
+14
@@ -84,6 +84,17 @@ def register_vllm_structural_tag(model: str):
84
return decorator
85
86
87
+def _any_tool_strict(
88
+ tools: Sequence[ChatCompletionToolsParam | ResponsesTool],
89
+) -> bool:
90
+ for tool in tools:
91
+ if isinstance(tool, FunctionTool) and tool.strict is True:
92
+ return True
93
+ if isinstance(tool, ChatCompletionToolsParam) and tool.function.strict is True:
94
+ return True
95
+ return False
96
+
97
+
98
def get_model_structural_tag(
99
model: str,
100
tools: Sequence[ChatCompletionToolsParam | ResponsesTool] | None,
@@ -95,6 +106,9 @@ def get_model_structural_tag(
106
if not tools or tool_choice == "none":
107
return None
108
109
+ if tool_choice == "auto" and not _any_tool_strict(tools):
110
+ return None
111
+
112
dumped_tools = [_dump_tool_for_xgrammar(tool) for tool in tools]
113
dumped_tool_choice = _dump_tool_choice_for_xgrammar(tool_choice)
114