A high-throughput and memory-efficient inference and serving engine for LLMs
[Frontend] Support strict mode for tool calling (#45003)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Chauncey committed
Jun 12, 2026 at 15:51 UTC
2043258decb048d0ad2cfb02c8fe1ba3a63aad94
29 files changed
+672
-1936
docs/features/tool_calling.md
+17
-7
@@ -115,18 +115,28 @@ Whether vLLM enforces the tool parameter schema during generation depends on the
115
| --- | --- | --- |
116
| Named function | Yes (via structured outputs backend) | Arguments are guaranteed to be valid JSON conforming to the function's parameter schema. |
117
| `"required"` | Yes (via structured outputs backend) | Same as named function. The model must produce at least one tool call. |
118
-| `"auto"` | No | The model generates freely. A tool-call parser extracts tool calls from the raw text. Arguments may be malformed or not match the schema. |
118
+| `"auto"` | Depends on the parser | Model-specific structural-tag parsers can constrain tool-call arguments with structured outputs. Other parsers generate freely and extract tool calls from raw text. |
119
| `"none"` | N/A | No tool calls are produced. |
120
121
-When schema conformance matters, prefer `tool_choice="required"` or named function calling over `"auto"`.
121
+### Strict Mode
122
123
-### Strict Mode (`strict` parameter)
123
+Strict tool calling makes function-call arguments adhere to the function schema instead of relying only on best-effort parsing. vLLM implements strict tool calling for structural-tag based tool parsers by using the structured outputs backend under the hood.
124
125
-The [OpenAI API](https://platform.openai.com/docs/guides/function-calling#strict-mode) supports a `strict` field on function definitions. When set to `true`, OpenAI uses constrained decoding to guarantee that tool-call arguments match the function schema, even in `tool_choice="auto"` mode.
125
+For best compatibility with strict schema enforcement, define tool parameter schemas in the OpenAI strict-schema style:
126
127
-vLLM **does not implement** `strict` mode today. The `strict` field is accepted in requests (to avoid breaking clients that set it), but it has no effect on decoding behavior. In auto mode, argument validity depends entirely on the model's output quality and the parser's extraction logic.
127
+* Set `additionalProperties` to `false` for each object in `parameters`.
128
+* Mark all fields in `properties` as required.
129
+* Represent optional fields by allowing `null`, for example `{"type": ["string", "null"]}`.
130
129
-Tracking issues: [#15526](https://github.com/vllm-project/vllm/issues/15526), [#16313](https://github.com/vllm-project/vllm/issues/16313).
131
+vLLM controls structural-tag strict tool calling with the `VLLM_ENFORCE_STRICT_TOOL_CALLING` environment variable. It defaults to `true`.
132
+
133
+```bash
134
+VLLM_ENFORCE_STRICT_TOOL_CALLING=false vllm serve ...
135
+```
136
+
137
+When this variable is `true`, structural-tag based tool parsers attach a structural tag to the request, so the structured outputs backend can constrain the model-specific tool-call format and function-call arguments. When it is `false`, vLLM does not attach structural tags for tool calling. In that case, `tool_choice="auto"` falls back to best-effort parser extraction from the raw model output, and no structural-tag constraint is applied.
138
+
139
+This environment variable only affects structural-tag based tool calling. It does not change schema-derived structured outputs used by named function calling or `tool_choice="required"`.
140
141
## Automatic Function Calling
142
@@ -146,7 +156,7 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
156
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
157
158
!!! note
149
- With `tool_choice="auto"`, tool-call arguments are extracted from the model's raw text output by the selected parser. No schema-level constraint is applied during decoding, so arguments may occasionally be malformed or violate the function's parameter schema. See [Constrained Decoding Behavior](#constrained-decoding-behavior) for details.
159
+ With `tool_choice="auto"`, schema-level constraint depends on the selected parser and `VLLM_ENFORCE_STRICT_TOOL_CALLING`. Structural-tag parsers can enforce tool-call constraints when it is `true`; when it is `false`, or when the selected parser has no structural-tag support, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
160
161
### Hermes Models (`hermes`)
162
requirements/common.txt
+1
-1
@@ -25,7 +25,7 @@ outlines_core == 0.2.14
25
# required for outlines backend disk cache
26
diskcache == 5.6.3
27
lark == 1.2.2
28
-xgrammar >= 0.2.0, < 1.0.0; platform_machine == "x86_64" or platform_machine == "aarch64" or platform_machine == "arm64" or platform_machine == "s390x" or platform_machine == "ppc64le"
28
+xgrammar >= 0.2.1, < 1.0.0; platform_machine == "x86_64" or platform_machine == "aarch64" or platform_machine == "arm64" or platform_machine == "s390x" or platform_machine == "ppc64le"
29
typing_extensions >= 4.10
30
filelock >= 3.16.1 # need to contain https://github.com/tox-dev/filelock/pull/317
31
partial-json-parser # used for parsing partial JSON outputs
requirements/test/rocm.txt
+1
-1
@@ -1367,7 +1367,7 @@ word2number==1.1
1367
# via lm-eval
1368
wrapt==2.1.2
1369
# via smart-open
1370
-xgrammar==0.2.0
1370
+xgrammar==0.2.1
1371
# via
1372
# -c requirements/common.txt
1373
# -r requirements/test/../common.txt
tests/entrypoints/openai/chat_completion/test_completion_with_function_calling.py
+2
-1
@@ -250,6 +250,7 @@ async def k2_client(k2_server):
250
251
252
@pytest.mark.asyncio
253
+@pytest.mark.skip(reason="Skipping Kimi K2 tool ID test")
254
@pytest.mark.parametrize("model_name", [MODEL_NAME])
255
@pytest.mark.parametrize("stream", [True, False])
256
@pytest.mark.parametrize("tool_choice", ["required"])
@@ -442,7 +443,7 @@ async def test_named_tool_use(
443
if delta.role:
444
assert delta.role == "assistant"
445
assert delta.content is None or len(delta.content) == 0
445
- if delta.tool_calls:
446
+ if delta.tool_calls and delta.tool_calls[0].function.arguments:
447
output.append(delta.tool_calls[0].function.arguments)
448
if chunk.choices[0].finish_reason is not None:
449
finish_reason_count += 1
tests/entrypoints/openai/responses/conftest.py
+1
@@ -390,6 +390,7 @@ def server_with_store(default_server_args):
390
env_dict={
391
"VLLM_ENABLE_RESPONSES_API_STORE": "1",
392
"VLLM_SERVER_DEV_MODE": "1",
393
+ "VLLM_ENFORCE_STRICT_TOOL_CALLING": "0",
394
},
395
) as remote_server:
396
yield remote_server
tests/parser/test_parse.py
+22
-4
@@ -2,13 +2,31 @@
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4
import json
5
+import os
6
7
import pytest
8
8
-from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
9
-from vllm.parser.abstract_parser import DelegatingParser
10
-from vllm.reasoning.basic_parsers import BaseThinkingReasoningParser
11
-from vllm.tool_parsers.hermes_tool_parser import Hermes2ProToolParser
9
+_STRICT_TOOL_CALLING_ENV = "VLLM_ENFORCE_STRICT_TOOL_CALLING"
10
+_STRICT_TOOL_CALLING_ENV_VALUE = os.environ.get(_STRICT_TOOL_CALLING_ENV)
11
+os.environ[_STRICT_TOOL_CALLING_ENV] = "0"
12
+
13
+from vllm.entrypoints.openai.chat_completion.protocol import ( # noqa: E402
14
+ ChatCompletionRequest,
15
+)
16
+from vllm.parser.abstract_parser import DelegatingParser # noqa: E402
17
+from vllm.reasoning.basic_parsers import ( # noqa: E402
18
+ BaseThinkingReasoningParser,
19
+)
20
+from vllm.tool_parsers.hermes_tool_parser import Hermes2ProToolParser # noqa: E402
21
+
22
+
23
+@pytest.fixture(scope="module", autouse=True)
24
+def restore_strict_tool_calling_env():
25
+ yield
26
+ if _STRICT_TOOL_CALLING_ENV_VALUE is None:
27
+ os.environ.pop(_STRICT_TOOL_CALLING_ENV, None)
28
+ else:
29
+ os.environ[_STRICT_TOOL_CALLING_ENV] = _STRICT_TOOL_CALLING_ENV_VALUE
30
31
32
class ThinkReasoningParser(BaseThinkingReasoningParser):
tests/tool_parsers/test_qwen3coder_tool_parser.py
+13
-190
@@ -3,6 +3,7 @@
3
4
import json
5
from collections.abc import Generator
6
+from unittest.mock import MagicMock
7
8
import pytest
9
from openai.types.responses.function_tool import FunctionTool
@@ -19,15 +20,12 @@ from vllm.entrypoints.openai.engine.protocol import (
20
FunctionCall,
21
ToolCall,
22
)
23
+from vllm.parser.abstract_parser import DelegatingParser
24
from vllm.tokenizers import TokenizerLike, get_tokenizer
25
from vllm.tokenizers.detokenizer_utils import detokenize_incrementally
26
from vllm.tool_parsers.qwen3coder_tool_parser import (
27
Qwen3CoderToolParser,
28
)
27
-from vllm.tool_parsers.qwen3xml_tool_parser import (
28
- Qwen3XMLToolParser,
29
- StreamingXMLToolCallParser,
30
-)
29
30
MODEL = "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8"
31
@@ -43,17 +41,8 @@ def qwen3_tool_parser(qwen3_tokenizer, sample_tools):
41
42
43
@pytest.fixture
46
-def qwen3_xml_tool_parser(qwen3_tokenizer, sample_tools):
47
- return Qwen3XMLToolParser(qwen3_tokenizer, tools=sample_tools)
48
-
49
-
50
-@pytest.fixture(params=["xml"])
51
-def qwen3_tool_parser_parametrized(qwen3_tool_parser, qwen3_xml_tool_parser, request):
52
- """Parameterized fixture that provides both parser types for testing"""
53
- if request.param == "original":
54
- return qwen3_tool_parser
55
- else:
56
- return qwen3_xml_tool_parser
44
+def qwen3_tool_parser_parametrized(qwen3_tool_parser):
45
+ return qwen3_tool_parser
46
47
48
WEATHER_PARAMS = {
@@ -168,47 +157,6 @@ def assert_tool_calls(
157
)
158
159
171
-def test_qwen3xml_deferred_array_parses_json_literals():
172
- parser = StreamingXMLToolCallParser()
173
- parser.set_tools(
174
- [
175
- ChatCompletionToolsParam(
176
- type="function",
177
- function={
178
- "name": "AskUserQuestion",
179
- "parameters": QUESTION_PARAMS,
180
- },
181
- )
182
- ]
183
- )
184
-
185
- delta = parser.parse_single_streaming_chunks(
186
- """<tool_call>
187
-<function=AskUserQuestion>
188
-<parameter=questions>
189
-[{"question": "Pick a color", "multiSelect": false, "answer": null}]
190
-</parameter>
191
-</function>
192
-</tool_call>"""
193
- )
194
-
195
- arguments = "".join(
196
- tool_call.function.arguments or ""
197
- for tool_call in delta.tool_calls or []
198
- if tool_call.function and tool_call.function.arguments is not None
199
- )
200
-
201
- assert json.loads(arguments) == {
202
- "questions": [
203
- {
204
- "question": "Pick a color",
205
- "multiSelect": False,
206
- "answer": None,
207
- }
208
- ]
209
- }
210
-
211
-
160
def stream_delta_message_generator(
161
qwen3_tool_parser,
162
qwen3_tokenizer: TokenizerLike,
@@ -523,7 +471,7 @@ hello world
471
</function>
472
</tool_call>"""
473
526
- parser = Qwen3XMLToolParser(qwen3_tokenizer, tools=tools)
474
+ parser = Qwen3CoderToolParser(qwen3_tokenizer, tools=tools)
475
request = ChatCompletionRequest(model=MODEL, messages=[], tools=tools)
476
extracted_tool_calls = parser.extract_tool_calls(model_output, request=request)
477
@@ -1146,125 +1094,6 @@ TX
1094
assert parsed_args["state"] == "TX"
1095
1096
1149
-def test_extract_tool_calls_complex_type_with_single_quote(
1150
- qwen3_tokenizer,
1151
-):
1152
- """Test parameter type conversion based on tool schema"""
1153
- tools = [
1154
- ChatCompletionToolsParam(
1155
- type="function",
1156
- function={
1157
- "name": "test_types",
1158
- "parameters": {
1159
- "type": "object",
1160
- "properties": {
1161
- "int_param": {"type": "integer"},
1162
- "float_param": {"type": "float"},
1163
- "bool_param": {"type": "boolean"},
1164
- "str_param": {"type": "string"},
1165
- "obj_param": {"type": "object"},
1166
- },
1167
- },
1168
- },
1169
- )
1170
- ]
1171
-
1172
- model_output = """<tool_call>
1173
-<function=test_types>
1174
-<parameter=obj_param>
1175
-{'key': 'value'}
1176
-</parameter>
1177
-</function>
1178
-</tool_call>"""
1179
-
1180
- parser = Qwen3XMLToolParser(qwen3_tokenizer, tools=tools)
1181
- request = ChatCompletionRequest(model=MODEL, messages=[], tools=tools)
1182
- extracted_tool_calls = parser.extract_tool_calls(model_output, request=request)
1183
-
1184
- args = json.loads(extracted_tool_calls.tool_calls[0].function.arguments)
1185
- assert args["obj_param"] == {"key": "value"}
1186
-
1187
-
1188
-def test_extract_tool_calls_streaming_missing_opening_tag(
1189
- qwen3_tool_parser_parametrized, qwen3_tokenizer
1190
-):
1191
- """Test streaming with missing opening <tool_call> tag
1192
-
1193
- This tests that the streaming parser correctly handles
1194
- tool calls that start directly with <function=...>
1195
- """
1196
- model_output = """I'll check the weather for you.
1197
-
1198
-<function=get_current_weather>
1199
-<parameter=city>
1200
-Dallas
1201
-</parameter>
1202
-<parameter=state>
1203
-TX
1204
-</parameter>
1205
-<parameter=unit>
1206
-fahrenheit
1207
-</parameter>
1208
-</function>
1209
-</tool_call>"""
1210
-
1211
- request = ChatCompletionRequest(model=MODEL, messages=[])
1212
-
1213
- other_content = ""
1214
- tool_states = {}
1215
-
1216
- for delta_message in stream_delta_message_generator(
1217
- qwen3_tool_parser_parametrized, qwen3_tokenizer, model_output, request
1218
- ):
1219
- if delta_message.content:
1220
- other_content += delta_message.content
1221
-
1222
- if delta_message.tool_calls:
1223
- for tool_call in delta_message.tool_calls:
1224
- idx = tool_call.index
1225
-
1226
- if idx not in tool_states:
1227
- tool_states[idx] = {
1228
- "id": None,
1229
- "name": None,
1230
- "arguments": "",
1231
- "type": None,
1232
- }
1233
-
1234
- if tool_call.id:
1235
- tool_states[idx]["id"] = tool_call.id
1236
-
1237
- if tool_call.type:
1238
- assert tool_call.type == "function"
1239
- tool_states[idx]["type"] = tool_call.type
1240
-
1241
- if tool_call.function:
1242
- if tool_call.function.name:
1243
- tool_states[idx]["name"] = tool_call.function.name
1244
-
1245
- if tool_call.function.arguments is not None:
1246
- tool_states[idx]["arguments"] += tool_call.function.arguments
1247
-
1248
- # Verify content was streamed
1249
- assert "I'll check the weather for you." in other_content
1250
-
1251
- # Verify we got the tool call
1252
- assert len(tool_states) == 1
1253
- assert len(qwen3_tool_parser_parametrized.prev_tool_call_arr) == 1
1254
-
1255
- state = tool_states[0]
1256
- assert state["id"] is not None
1257
- assert state["type"] == "function"
1258
- assert state["name"] == "get_current_weather"
1259
-
1260
- # Verify arguments were parsed correctly despite missing opening tag
1261
- assert state["arguments"] is not None
1262
- args = json.loads(state["arguments"])
1263
- assert args["city"] == "Dallas"
1264
- assert args["state"] == "TX"
1265
- assert args["unit"] == "fahrenheit"
1266
-
1267
-
1097
def test_malformed_xml_no_gt_delimiter(qwen3_tool_parser):
1098
"""Regression: malformed XML without '>' must not crash (PR #36774)."""
1099
model_output = (
@@ -1456,15 +1285,12 @@ def test_get_vllm_registry_structural_tag_returns_structural_tag(
1285
1286
@pytest.mark.parametrize("include_reasoning", [True, False])
1287
def test_adjust_request_auto_uses_vllm_registry_structural_tag(
1459
- monkeypatch: pytest.MonkeyPatch,
1460
- qwen3_tool_parser: Qwen3CoderToolParser,
1288
sample_tools: list[ChatCompletionToolsParam],
1289
include_reasoning: bool,
1290
) -> None:
1464
- monkeypatch.setattr(
1465
- "vllm.tool_parsers.abstract_tool_parser.VLLM_ENFORCE_STRICT_TOOL_CALLING",
1466
- True,
1467
- )
1291
+ class TestParser(DelegatingParser):
1292
+ tool_parser_cls = Qwen3CoderToolParser
1293
+
1294
request_tools = _as_chat_completion_tools(sample_tools)
1295
req = ChatCompletionRequest(
1296
messages=[],
@@ -1473,7 +1299,7 @@ def test_adjust_request_auto_uses_vllm_registry_structural_tag(
1299
tool_choice="auto",
1300
include_reasoning=include_reasoning,
1301
)
1476
- out = qwen3_tool_parser.adjust_request(req)
1302
+ out = TestParser(MagicMock(), tools=sample_tools).adjust_request(req)
1303
assert out.structured_outputs is not None
1304
assert out.structured_outputs.structural_tag is not None
1305
assert isinstance(out.structured_outputs.structural_tag, str)
@@ -1482,14 +1308,11 @@ def test_adjust_request_auto_uses_vllm_registry_structural_tag(
1308
1309
1310
def test_adjust_request_required_prefers_structural_tag(
1485
- monkeypatch: pytest.MonkeyPatch,
1486
- qwen3_tool_parser: Qwen3CoderToolParser,
1311
sample_tools: list[ChatCompletionToolsParam],
1312
) -> None:
1489
- monkeypatch.setattr(
1490
- "vllm.tool_parsers.abstract_tool_parser.VLLM_ENFORCE_STRICT_TOOL_CALLING",
1491
- True,
1492
- )
1313
+ class TestParser(DelegatingParser):
1314
+ tool_parser_cls = Qwen3CoderToolParser
1315
+
1316
request_tools = _as_chat_completion_tools(sample_tools)
1317
req = ChatCompletionRequest(
1318
messages=[],
@@ -1497,6 +1320,6 @@ def test_adjust_request_required_prefers_structural_tag(
1320
tools=request_tools,
1321
tool_choice="required",
1322
)
1500
- out = qwen3_tool_parser.adjust_request(req)
1323
+ out = TestParser(MagicMock(), tools=sample_tools).adjust_request(req)
1324
assert out.structured_outputs is not None
1325
assert out.structured_outputs.structural_tag is not None
tests/tool_parsers/test_qwen3xml_tool_parser.py
deleted
-72
@@ -1,72 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-
4
-
5
-import pytest
6
-
7
-from tests.tool_parsers.common_tests import (
8
- ToolParserTestConfig,
9
- ToolParserTests,
10
-)
11
-
12
-
13
-class TestQwen3xmlToolParser(ToolParserTests):
14
- @pytest.fixture
15
- def test_config(self) -> ToolParserTestConfig:
16
- return ToolParserTestConfig(
17
- parser_name="qwen3_xml",
18
- # Test data
19
- no_tool_calls_output="This is a regular response without any tool calls.",
20
- single_tool_call_output="<tool_call>\n<function=get_weather>\n<parameter=city>Tokyo</parameter>\n</function>\n</tool_call>",
21
- parallel_tool_calls_output="<tool_call>\n<function=get_weather>\n<parameter=city>Tokyo</parameter>\n</function>\n</tool_call><tool_call>\n<function=get_time>\n<parameter=timezone>Asia/Tokyo</parameter>\n</function>\n</tool_call>",
22
- various_data_types_output=(
23
- "<tool_call>\n<function=test_function>\n"
24
- "<parameter=string_field>hello</parameter>\n"
25
- "<parameter=int_field>42</parameter>\n"
26
- "<parameter=float_field>3.14</parameter>\n"
27
- "<parameter=bool_field>true</parameter>\n"
28
- "<parameter=null_field>null</parameter>\n"
29
- '<parameter=array_field>["a", "b", "c"]</parameter>\n'
30
- '<parameter=object_field>{"nested": "value"}</parameter>\n'
31
- "</function>\n</tool_call>"
32
- ),
33
- empty_arguments_output="<tool_call>\n<function=refresh>\n</function>\n</tool_call>",
34
- surrounding_text_output=(
35
- "Let me check the weather for you.\n\n"
36
- "<tool_call>\n<function=get_weather>\n"
37
- "<parameter=city>Tokyo</parameter>\n"
38
- "</function>\n</tool_call>\n\n"
39
- "I will get that information."
40
- ),
41
- escaped_strings_output=(
42
- "<tool_call>\n<function=test_function>\n"
43
- '<parameter=quoted>He said "hello"</parameter>\n'
44
- "<parameter=path>C:\\Users\\file.txt</parameter>\n"
45
- "<parameter=newline>line1\nline2</parameter>\n"
46
- "</function>\n</tool_call>"
47
- ),
48
- malformed_input_outputs=[
49
- "<tool_call><function=func>",
50
- "<tool_call><function=></function></tool_call>",
51
- ],
52
- # Expected results
53
- single_tool_call_expected_name="get_weather",
54
- single_tool_call_expected_args={"city": "Tokyo"},
55
- parallel_tool_calls_count=2,
56
- parallel_tool_calls_names=["get_weather", "get_time"],
57
- # xfail markers - Qwen3XML has systematic streaming issues
58
- xfail_streaming={
59
- "test_single_tool_call_simple_args": (
60
- "Qwen3XML streaming has systematic issues"
61
- ),
62
- "test_parallel_tool_calls": "Qwen3XML streaming has systematic issues",
63
- "test_various_data_types": "Qwen3XML streaming has systematic issues",
64
- "test_empty_arguments": "Qwen3XML streaming has systematic issues",
65
- "test_surrounding_text": "Qwen3XML streaming has systematic issues",
66
- "test_escaped_strings": "Qwen3XML streaming has systematic issues",
67
- "test_streaming_reconstruction": (
68
- "Qwen3XML streaming reconstruction has known issues"
69
- ),
70
- },
71
- supports_typed_arguments=False,
72
- )
tests/tool_parsers/test_structural_tag_registry.py
new
+314
@@ -0,0 +1,314 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+
4
+from types import SimpleNamespace
5
+from unittest.mock import MagicMock
6
+
7
+import pytest
8
+from xgrammar import StructuralTag
9
+
10
+from vllm.entrypoints.openai.chat_completion.protocol import (
11
+ ChatCompletionNamedFunction,
12
+ ChatCompletionNamedToolChoiceParam,
13
+ ChatCompletionRequest,
14
+ ChatCompletionToolsParam,
15
+)
16
+from vllm.parser.abstract_parser import DelegatingParser
17
+from vllm.tool_parsers.abstract_tool_parser import ToolParser
18
+from vllm.tool_parsers.deepseekv3_tool_parser import DeepSeekV3ToolParser
19
+from vllm.tool_parsers.deepseekv4_tool_parser import DeepSeekV4ToolParser
20
+from vllm.tool_parsers.deepseekv31_tool_parser import DeepSeekV31ToolParser
21
+from vllm.tool_parsers.deepseekv32_tool_parser import DeepSeekV32ToolParser
22
+from vllm.tool_parsers.glm47_moe_tool_parser import Glm47MoeModelToolParser
23
+from vllm.tool_parsers.hermes_tool_parser import Hermes2ProToolParser
24
+from vllm.tool_parsers.kimi_k2_tool_parser import KimiK2ToolParser
25
+from vllm.tool_parsers.llama_tool_parser import Llama3JsonToolParser
26
+from vllm.tool_parsers.minimax_m2_tool_parser import MinimaxM2ToolParser
27
+from vllm.tool_parsers.qwen3coder_tool_parser import Qwen3CoderToolParser
28
+from vllm.tool_parsers.structural_tag_registry import (
29
+ SUPPORTED_STRUCTURAL_TAG_MODELS,
30
+ VLLM_BUILTIN_STRUCTURAL_TAG_MODELS,
31
+ XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS,
32
+ _get_function_parameters,
33
+ get_model_structural_tag,
34
+)
35
+
36
+
37
+@pytest.fixture
38
+def sample_tools() -> list[ChatCompletionToolsParam]:
39
+ return [
40
+ ChatCompletionToolsParam(
41
+ type="function",
42
+ function={
43
+ "name": "get_weather",
44
+ "parameters": {
45
+ "type": "object",
46
+ "properties": {"city": {"type": "string"}},
47
+ "required": ["city"],
48
+ },
49
+ },
50
+ )
51
+ ]
52
+
53
+
54
+def test_supported_structural_tag_models_include_vllm_builtins():
55
+ assert SUPPORTED_STRUCTURAL_TAG_MODELS == (
56
+ XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS | VLLM_BUILTIN_STRUCTURAL_TAG_MODELS
57
+ )
58
+ assert "hermes" in VLLM_BUILTIN_STRUCTURAL_TAG_MODELS
59
+
60
+
61
+@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
62
+def test_get_model_structural_tag_supports_all_xgrammar_builtins(
63
+ model: str,
64
+ sample_tools: list[ChatCompletionToolsParam],
65
+):
66
+ tag = get_model_structural_tag(
67
+ model=model,
68
+ tools=sample_tools,
69
+ tool_choice="auto",
70
+ reasoning=False,
71
+ )
72
+
73
+ assert isinstance(tag, StructuralTag)
74
+
75
+
76
+def test_get_model_structural_tag_supports_vllm_hermes(
77
+ sample_tools: list[ChatCompletionToolsParam],
78
+):
79
+ tag = get_model_structural_tag(
80
+ model="hermes",
81
+ tools=sample_tools,
82
+ tool_choice="required",
83
+ reasoning=False,
84
+ )
85
+
86
+ assert isinstance(tag, StructuralTag)
87
+ assert tag.model_dump() == {
88
+ "type": "structural_tag",
89
+ "format": {
90
+ "type": "tags_with_separator",
91
+ "tags": [
92
+ {
93
+ "type": "tag",
94
+ "begin": '<tool_call>\n{"name": "get_weather", "arguments": ',
95
+ "content": {
96
+ "type": "json_schema",
97
+ "json_schema": {
98
+ "type": "object",
99
+ "properties": {"city": {"type": "string"}},
100
+ "required": ["city"],
101
+ },
102
+ "style": "json",
103
+ },
104
+ "end": "}\n</tool_call>",
105
+ },
106
+ {
107
+ "type": "tag",
108
+ "begin": '<tool_call>{"name": "get_weather", "arguments": ',
109
+ "content": {
110
+ "type": "json_schema",
111
+ "json_schema": {
112
+ "type": "object",
113
+ "properties": {"city": {"type": "string"}},
114
+ "required": ["city"],
115
+ },
116
+ "style": "json",
117
+ },
118
+ "end": "}</tool_call>",
119
+ },
120
+ ],
121
+ "separator": "",
122
+ "at_least_one": True,
123
+ "stop_after_first": False,
124
+ },
125
+ }
126
+
127
+
128
+def test_hermes_required_tool_calls_use_empty_separator():
129
+ tools = [
130
+ ChatCompletionToolsParam(
131
+ type="function",
132
+ function={
133
+ "name": "get_weather",
134
+ "parameters": {"type": "object", "properties": {}},
135
+ },
136
+ ),
137
+ ChatCompletionToolsParam(
138
+ type="function",
139
+ function={
140
+ "name": "get_time",
141
+ "parameters": {"type": "object", "properties": {}},
142
+ },
143
+ ),
144
+ ]
145
+
146
+ tag = get_model_structural_tag(
147
+ model="hermes",
148
+ tools=tools,
149
+ tool_choice="required",
150
+ reasoning=False,
151
+ )
152
+
153
+ assert tag is not None
154
+ assert tag.format.separator == ""
155
+
156
+
157
+@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
158
+def test_get_model_structural_tag_supports_named_tool_choice(
159
+ model: str,
160
+ sample_tools: list[ChatCompletionToolsParam],
161
+):
162
+ tag = get_model_structural_tag(
163
+ model=model,
164
+ tools=sample_tools,
165
+ tool_choice=ChatCompletionNamedToolChoiceParam(
166
+ function=ChatCompletionNamedFunction(name="get_weather")
167
+ ),
168
+ reasoning=False,
169
+ )
170
+
171
+ assert isinstance(tag, StructuralTag)
172
+
173
+
174
+@pytest.mark.parametrize(
175
+ ("parser_cls", "model"),
176
+ [
177
+ (DeepSeekV3ToolParser, "deepseek_r1"),
178
+ (DeepSeekV31ToolParser, "deepseek_v3_1"),
179
+ (DeepSeekV32ToolParser, "deepseek_v3_2"),
180
+ (DeepSeekV4ToolParser, "deepseek_v4"),
181
+ (Glm47MoeModelToolParser, "glm_4_7"),
182
+ (Hermes2ProToolParser, "hermes"),
183
+ (KimiK2ToolParser, "kimi"),
184
+ (Llama3JsonToolParser, "llama"),
185
+ (MinimaxM2ToolParser, "minimax"),
186
+ (Qwen3CoderToolParser, "qwen_3_coder"),
187
+ ],
188
+)
189
+def test_tool_parsers_declare_matching_xgrammar_builtin_model(parser_cls, model):
190
+ assert parser_cls.structural_tag_model == model
191
+ assert not parser_cls.supports_required_and_named
192
+
193
+
194
+def test_tool_parsers_without_structural_tag_support_required_and_named():
195
+ class NonStructuralTagToolParser(ToolParser):
196
+ pass
197
+
198
+ assert NonStructuralTagToolParser.structural_tag_model is None
199
+ assert NonStructuralTagToolParser.supports_required_and_named
200
+
201
+
202
+def test_non_structural_tag_parser_uses_schema_constraints(
203
+ sample_tools: list[ChatCompletionToolsParam],
204
+):
205
+ parser = ToolParser(MagicMock())
206
+ request = ChatCompletionRequest(
207
+ messages=[],
208
+ model="m",
209
+ tools=sample_tools,
210
+ tool_choice="required",
211
+ )
212
+
213
+ out = parser.adjust_request(request)
214
+
215
+ assert out.structured_outputs is not None
216
+ assert out.structured_outputs.json is not None
217
+ assert out.structured_outputs.structural_tag is None
218
+
219
+
220
+def test_get_structural_tag_disables_reasoning(
221
+ monkeypatch: pytest.MonkeyPatch,
222
+ sample_tools: list[ChatCompletionToolsParam],
223
+):
224
+ captured: list[bool] = []
225
+
226
+ def fake_get_model_structural_tag(*, reasoning: bool, **kwargs):
227
+ captured.append(reasoning)
228
+ return None
229
+
230
+ monkeypatch.setattr(
231
+ "vllm.tool_parsers.structural_tag_registry.get_model_structural_tag",
232
+ fake_get_model_structural_tag,
233
+ )
234
+
235
+ request = ChatCompletionRequest(
236
+ messages=[],
237
+ model="m",
238
+ tools=sample_tools,
239
+ tool_choice="auto",
240
+ )
241
+ parser = Qwen3CoderToolParser(MagicMock(), tools=sample_tools)
242
+
243
+ parser.get_structural_tag(request)
244
+
245
+ assert captured == [False]
246
+
247
+
248
+def test_unified_parser_get_structural_tag_disables_reasoning(
249
+ monkeypatch: pytest.MonkeyPatch,
250
+ sample_tools: list[ChatCompletionToolsParam],
251
+):
252
+ captured: list[bool] = []
253
+
254
+ def fake_get_model_structural_tag(*, reasoning: bool, **kwargs):
255
+ captured.append(reasoning)
256
+ return None
257
+
258
+ monkeypatch.setattr(
259
+ "vllm.tool_parsers.structural_tag_registry.get_model_structural_tag",
260
+ fake_get_model_structural_tag,
261
+ )
262
+
263
+ class TestParser(DelegatingParser):
264
+ tool_parser_cls = Qwen3CoderToolParser
265
+
266
+ request = ChatCompletionRequest(
267
+ messages=[],
268
+ model="m",
269
+ tools=sample_tools,
270
+ tool_choice="auto",
271
+ )
272
+ parser = TestParser(MagicMock(), tools=sample_tools)
273
+ parser.reasoning_parser = MagicMock(adjust_request=lambda request: request)
274
+
275
+ parser.adjust_request(request)
276
+
277
+ assert captured == [False]
278
+
279
+
280
+def test_xgrammar_function_parameters_are_preserved(
281
+ monkeypatch: pytest.MonkeyPatch,
282
+ sample_tools: list[ChatCompletionToolsParam],
283
+):
284
+ captured: list[list[dict]] = []
285
+
286
+ def fake_get_xgrammar_model_structural_tag(*, tools: list[dict], **kwargs):
287
+ captured.append(tools)
288
+ return None
289
+
290
+ monkeypatch.setattr(
291
+ "vllm.tool_parsers.structural_tag_registry.get_xgrammar_model_structural_tag",
292
+ fake_get_xgrammar_model_structural_tag,
293
+ )
294
+
295
+ get_model_structural_tag(
296
+ model="llama",
297
+ tools=sample_tools,
298
+ tool_choice="auto",
299
+ reasoning=False,
300
+ )
301
+
302
+ assert (
303
+ captured[0][0]["function"]["parameters"] == sample_tools[0].function.parameters
304
+ )
305
+ assert sample_tools[0].function.parameters is not None
306
+
307
+
308
+def test_get_function_parameters_relaxes_function_strict_false():
309
+ function = SimpleNamespace(
310
+ parameters={"type": "object", "properties": {}},
311
+ strict=False,
312
+ )
313
+
314
+ assert _get_function_parameters(function) is True
vllm/entrypoints/openai/api_server.py
-14
@@ -308,20 +308,6 @@ async def init_app_state(
308
) -> None:
309
vllm_config = engine_client.vllm_config
310
311
- # Propagate enable_in_reasoning to the API-server process. The engine core
312
- # runs in a separate process, so the contextvar that backs
313
- # `get_current_vllm_config_or_none()` is None on this stack. Tool parsers
314
- # call `get_enable_structured_outputs_in_reasoning()` during request
315
- # handling and need to see the real flag, otherwise they silently fall
316
- # back to False and mismatch the engine-side bitmask gating.
317
- from vllm.tool_parsers.structural_tag_registry import (
318
- set_enable_structured_outputs_in_reasoning,
319
- )
320
-
321
- set_enable_structured_outputs_in_reasoning(
322
- vllm_config.structured_outputs_config.enable_in_reasoning
323
- )
324
-
311
if args.tool_call_parser is not None:
312
from vllm.parser.metrics import init_parser_metrics
313
vllm/entrypoints/openai/chat_completion/batch_serving.py
+2
-2
@@ -74,7 +74,7 @@ class OpenAIServingChatBatch(OpenAIServingChat):
74
if error_check_ret is not None:
75
return error_check_ret
76
77
- tool_parser = render.tool_parser
77
+ parser = render.parser
78
tool_dicts: list[dict] | None = None
79
80
all_conversations: list[list[ConversationMessage]] = []
@@ -94,7 +94,7 @@ class OpenAIServingChatBatch(OpenAIServingChat):
94
default_template_content_format=render.chat_template_content_format,
95
default_template_kwargs=render.default_chat_template_kwargs,
96
tool_dicts=tool_dicts,
97
- tool_parser=tool_parser,
97
+ parser=parser,
98
)
99
all_conversations.append(conversation)
100
all_engine_prompts.append(engine_prompts[0])
vllm/entrypoints/openai/responses/serving.py
+5
-8
@@ -102,10 +102,9 @@ from vllm.logprobs import Logprob as SampleLogprob
102
from vllm.logprobs import SampleLogprobs
103
from vllm.lora.request import LoRARequest
104
from vllm.outputs import CompletionOutput
105
-from vllm.parser import ParserManager
105
+from vllm.parser import Parser, ParserManager
106
from vllm.sampling_params import SamplingParams, StructuredOutputsParams
107
from vllm.tokenizers import TokenizerLike
108
-from vllm.tool_parsers import ToolParser
108
from vllm.utils import random_uuid
109
from vllm.utils.collection_utils import as_list
110
@@ -613,8 +612,7 @@ class OpenAIServingResponses(OpenAIServing):
612
default_template_content_format=self.chat_template_content_format,
613
default_template_kwargs=chat_template_kwargs,
614
tool_dicts=tool_dicts,
616
- tool_parser=self.parser.tool_parser_cls if self.parser else None,
617
- reasoning_parser=self.parser.reasoning_parser_cls if self.parser else None,
615
+ parser=self.parser,
616
)
617
return messages, engine_inputs
618
@@ -623,7 +621,7 @@ class OpenAIServingResponses(OpenAIServing):
621
request: ResponsesRequest,
622
messages: list[ResponseInputOutputItem],
623
tool_dicts: list[dict[str, Any]] | None,
626
- tool_parser: type[ToolParser] | None,
624
+ parser: type[Parser] | None,
625
chat_template: str | None,
626
chat_template_content_format: ChatTemplateContentFormatOption,
627
):
@@ -638,8 +636,7 @@ class OpenAIServingResponses(OpenAIServing):
636
default_template_content_format=chat_template_content_format,
637
default_template_kwargs=chat_template_kwargs,
638
tool_dicts=tool_dicts,
641
- tool_parser=tool_parser,
642
- reasoning_parser=self.parser.reasoning_parser_cls if self.parser else None,
639
+ parser=parser,
640
)
641
return engine_inputs
642
@@ -707,7 +704,7 @@ class OpenAIServingResponses(OpenAIServing):
704
context.request,
705
context.parser.response_messages,
706
context.tool_dicts,
710
- context.parser_cls.tool_parser_cls if context.parser_cls else None,
707
+ context.parser_cls,
708
context.chat_template,
709
context.chat_template_content_format,
710
)
vllm/entrypoints/serve/render/serving.py
+24
-28
@@ -43,8 +43,7 @@ from vllm.inputs import (
43
tokens_input,
44
)
45
from vllm.logger import init_logger
46
-from vllm.parser import ParserManager
47
-from vllm.reasoning.abs_reasoning_parsers import ReasoningParser
46
+from vllm.parser import Parser, ParserManager
47
from vllm.renderers import BaseRenderer, merge_kwargs
48
from vllm.renderers.inputs.preprocess import (
49
extract_prompt_components,
@@ -52,7 +51,6 @@ from vllm.renderers.inputs.preprocess import (
51
parse_model_prompt,
52
prompt_to_seq,
53
)
55
-from vllm.tool_parsers import ToolParser
54
from vllm.utils import random_uuid
55
from vllm.utils.mistral import is_mistral_tokenizer, is_mistral_tool_parser
56
from vllm.utils.mistral import mt as _mt
@@ -89,16 +87,12 @@ class OpenAIServingRender:
87
self.trust_request_chat_template = trust_request_chat_template
88
self.enable_auto_tools = enable_auto_tools
89
self.exclude_tools_when_tool_choice_none = exclude_tools_when_tool_choice_none
92
- self.tool_parser: type[ToolParser] | None = ParserManager.get_tool_parser(
90
+ self.parser: type[Parser] | None = ParserManager.get_parser(
91
tool_parser_name=tool_parser,
92
+ reasoning_parser_name=reasoning_parser,
93
enable_auto_tools=enable_auto_tools,
94
model_name=model_config.model,
95
)
97
- self.reasoning_parser: type[ReasoningParser] | None = (
98
- ParserManager.get_reasoning_parser(
99
- reasoning_parser_name=reasoning_parser,
100
- )
101
- )
96
self.default_chat_template_kwargs: dict[str, Any] = (
97
default_chat_template_kwargs or {}
98
)
@@ -193,7 +187,7 @@ class OpenAIServingRender:
187
"""
188
tokenizer = self.renderer.tokenizer
189
196
- tool_parser = self.tool_parser
190
+ tool_parser = self.parser.tool_parser_cls if self.parser is not None else None
191
192
if is_mistral_tokenizer(tokenizer):
193
# because of issues with pydantic we need to potentially
@@ -252,9 +246,8 @@ class OpenAIServingRender:
246
default_template_content_format=self.chat_template_content_format,
247
default_template_kwargs=self.default_chat_template_kwargs,
248
tool_dicts=tool_dicts,
255
- tool_parser=tool_parser,
249
+ parser=self.parser,
250
skip_mm_cache=skip_mm_cache,
257
- reasoning_parser=self.reasoning_parser,
251
)
252
else:
253
# For GPT-OSS.
@@ -526,8 +519,7 @@ class OpenAIServingRender:
519
default_template_content_format: ChatTemplateContentFormatOption,
520
default_template_kwargs: dict[str, Any] | None,
521
tool_dicts: list[dict[str, Any]] | None = None,
529
- tool_parser: type[ToolParser] | None = None,
530
- reasoning_parser: type[ReasoningParser] | None = None,
522
+ parser: type[Parser] | None = None,
523
*,
524
skip_mm_cache: bool = False,
525
) -> tuple[list[ConversationMessage], list[EngineInput]]:
@@ -567,14 +559,6 @@ class OpenAIServingRender:
559
skip_mm_cache=skip_mm_cache,
560
)
561
570
- if reasoning_parser is not None:
571
- tokenizer = renderer.get_tokenizer()
572
- request = reasoning_parser(
573
- tokenizer,
574
- model_config=self.model_config,
575
- chat_template_kwargs=chat_params.chat_template_kwargs,
576
- ).adjust_request(request=request)
577
-
562
# tool parsing is done only if a tool_parser has been set and if
563
# tool_choice is not "none" (if tool_choice is "none" but a tool_parser
564
# is set, we want to prevent parsing a tool_call hallucinated by the LLM
@@ -582,15 +566,22 @@ class OpenAIServingRender:
566
# Exception: Mistral grammar-capable tokenizers always call
567
# adjust_request — even for tool_choice="none" — so that the grammar
568
# factory can prevent special-token leakage.
585
- if tool_parser is not None:
586
- tool_choice = getattr(request, "tool_choice", "none")
569
+ if parser is not None:
570
tokenizer = renderer.get_tokenizer()
571
+ tool_parser = parser.tool_parser_cls
572
+ tool_choice = getattr(request, "tool_choice", "none")
573
is_mistral_grammar_eligible = (
589
- is_mistral_tool_parser(tool_parser)
574
+ tool_parser is not None
575
+ and is_mistral_tool_parser(tool_parser)
576
and is_mistral_tokenizer(tokenizer)
577
and tokenizer.supports_grammar
578
)
593
- if tool_choice != "none" or is_mistral_grammar_eligible:
579
+ should_adjust_request = (
580
+ parser.reasoning_parser_cls is not None
581
+ or tool_choice != "none"
582
+ or is_mistral_grammar_eligible
583
+ )
584
+ if should_adjust_request:
585
if not isinstance(request, ChatCompletionRequest | ResponsesRequest):
586
msg = (
587
"Tool usage is only supported "
@@ -598,8 +589,13 @@ class OpenAIServingRender:
589
f"but got {type(request).__name__}"
590
)
591
raise NotImplementedError(msg)
601
- request = tool_parser(tokenizer, request.tools).adjust_request(
602
- request=request
592
+ request = parser(
593
+ tokenizer,
594
+ request.tools,
595
+ model_config=self.model_config,
596
+ chat_template_kwargs=chat_params.chat_template_kwargs,
597
+ ).adjust_request(
598
+ request=request,
599
)
600
601
return conversation, [engine_input]
vllm/envs.py
+6
-7
@@ -200,6 +200,7 @@ if TYPE_CHECKING:
200
MOONCAKE_REQUESTER_LOCAL_HOSTNAME: str | None = None
201
VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE: int = 163840
202
VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS: int = 1
203
+ VLLM_ENFORCE_STRICT_TOOL_CALLING: bool = True
204
VLLM_MQ_MAX_CHUNK_BYTES_MB: int = 16
205
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: int = 300
206
VLLM_KV_CACHE_LAYOUT: Literal["NHD", "HND"] | None = None
@@ -227,7 +228,6 @@ if TYPE_CHECKING:
228
VLLM_GPT_OSS_HARMONY_SYSTEM_INSTRUCTIONS: bool = False
229
VLLM_SYSTEM_START_DATE: str | None = None
230
VLLM_TOOL_JSON_ERROR_AUTOMATIC_RETRY: bool = False
230
- VLLM_ENFORCE_STRICT_TOOL_CALLING: bool = False
231
VLLM_CUSTOM_SCOPES_FOR_PROFILING: bool = False
232
VLLM_NVTX_SCOPES_FOR_PROFILING: bool = False
233
VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES: bool = True
@@ -1536,6 +1536,11 @@ environment_variables: dict[str, Callable[[], Any]] = {
1536
"VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS": lambda: int(
1537
os.getenv("VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS", "1")
1538
),
1539
+ # Enforce function parameter schemas in structural-tag based tool calling.
1540
+ "VLLM_ENFORCE_STRICT_TOOL_CALLING": lambda: os.getenv(
1541
+ "VLLM_ENFORCE_STRICT_TOOL_CALLING", "True"
1542
+ ).lower()
1543
+ in ("true", "1"),
1544
# Control the max chunk bytes (in MB) for the rpc message queue.
1545
# Object larger than this threshold will be broadcast to worker
1546
# processes via zmq.
@@ -1659,12 +1664,6 @@ environment_variables: dict[str, Callable[[], Any]] = {
1664
"VLLM_TOOL_JSON_ERROR_AUTOMATIC_RETRY": lambda: bool(
1665
int(os.getenv("VLLM_TOOL_JSON_ERROR_AUTOMATIC_RETRY", "0"))
1666
),
1662
- # When 1,the model structural tags will be used to enforce the model
1663
- # output conforming to the model's tool-calling format and schema.
1664
- # Default 0 (off).
1665
- "VLLM_ENFORCE_STRICT_TOOL_CALLING": lambda: bool(
1666
- int(os.getenv("VLLM_ENFORCE_STRICT_TOOL_CALLING", "0"))
1667
- ),
1667
# Add optional custom scopes for profiling, disable to avoid overheads
1668
"VLLM_CUSTOM_SCOPES_FOR_PROFILING": lambda: bool(
1669
int(os.getenv("VLLM_CUSTOM_SCOPES_FOR_PROFILING", "0"))
vllm/parser/abstract_parser.py
+36
@@ -25,6 +25,7 @@ from vllm.entrypoints.openai.responses.protocol import ResponsesRequest
25
from vllm.logger import init_logger
26
from vllm.parser.metrics import record_tool_parser_invocation
27
from vllm.reasoning.abs_reasoning_parsers import ReasoningParser
28
+from vllm.sampling_params import StructuredOutputsParams
29
from vllm.tokenizers import TokenizerLike
30
from vllm.tool_parsers.abstract_tool_parser import Tool, ToolParser
31
from vllm.tool_parsers.streaming import (
@@ -427,10 +428,45 @@ class DelegatingParser(Parser):
428
) -> ChatCompletionRequest | ResponsesRequest:
429
if self._reasoning_parser is not None:
430
request = self._reasoning_parser.adjust_request(request)
431
+ if self._tool_parser is not None:
432
+ request = self._apply_structural_tag(request)
433
if self._tool_parser is not None:
434
request = self._tool_parser.adjust_request(request)
435
return request
436
437
+ def _apply_structural_tag(
438
+ self, request: ChatCompletionRequest | ResponsesRequest
439
+ ) -> ChatCompletionRequest | ResponsesRequest:
440
+ if (
441
+ not isinstance(request, ChatCompletionRequest)
442
+ or self._tool_parser is None
443
+ or self._tool_parser.structural_tag_model is None
444
+ or not request.tools
445
+ ):
446
+ return request
447
+
448
+ need_tool_calling = (
449
+ request.tool_choice == "auto"
450
+ or request.tool_choice == "required"
451
+ or isinstance(request.tool_choice, ChatCompletionNamedToolChoiceParam)
452
+ )
453
+ if not need_tool_calling:
454
+ return request
455
+
456
+ structure_tag = self._tool_parser.get_structural_tag(
457
+ request,
458
+ reasoning=False,
459
+ )
460
+ if structure_tag is None:
461
+ return request
462
+
463
+ structural_tag = json.dumps(structure_tag.model_dump())
464
+ request.structured_outputs = StructuredOutputsParams(
465
+ structural_tag=structural_tag,
466
+ )
467
+ request.response_format = None
468
+ return request
469
+
470
def extract_reasoning_streaming(
471
self,
472
previous_text: str,
vllm/tool_parsers/__init__.py
+4
-4
@@ -119,8 +119,8 @@ _TOOL_PARSERS_TO_REGISTER = {
119
"LongcatFlashToolParser",
120
),
121
"mimo": (
122
- "qwen3xml_tool_parser",
123
- "Qwen3XMLToolParser",
122
+ "qwen3coder_tool_parser",
123
+ "Qwen3CoderToolParser",
124
),
125
"minimax_m2": (
126
"minimax_m2_tool_parser",
@@ -159,8 +159,8 @@ _TOOL_PARSERS_TO_REGISTER = {
159
"Qwen3CoderToolParser",
160
),
161
"qwen3_xml": (
162
- "qwen3xml_tool_parser",
163
- "Qwen3XMLToolParser",
162
+ "qwen3coder_tool_parser",
163
+ "Qwen3CoderToolParser",
164
),
165
"seed_oss": (
166
"seed_oss_tool_parser",
vllm/tool_parsers/abstract_tool_parser.py
+36
-28
@@ -6,6 +6,7 @@ import json
6
import os
7
from collections.abc import Callable, Sequence
8
from functools import cached_property
9
+from typing import Any
10
11
from openai.types.responses import (
12
ResponseFormatTextJSONSchemaConfig,
@@ -13,8 +14,8 @@ from openai.types.responses import (
14
)
15
from openai.types.responses.function_tool import FunctionTool
16
17
+import vllm.envs as envs
18
from vllm.entrypoints.openai.chat_completion.protocol import (
17
- ChatCompletionNamedToolChoiceParam,
19
ChatCompletionRequest,
20
ChatCompletionToolsParam,
21
)
@@ -25,7 +26,6 @@ from vllm.entrypoints.openai.engine.protocol import (
26
from vllm.entrypoints.openai.responses.protocol import (
27
ResponsesRequest,
28
)
28
-from vllm.envs import VLLM_ENFORCE_STRICT_TOOL_CALLING
29
from vllm.logger import init_logger
30
from vllm.sampling_params import (
31
StructuredOutputsParams,
@@ -57,6 +57,17 @@ class ToolParser:
57
# extract_tool_calls / extract_tool_calls_streaming methods for
58
# required/named tool_choice, treating them the same as "auto".
59
supports_required_and_named: bool = True
60
+ # xgrammar builtin structural tag model key. Subclasses set this when
61
+ # their parsed tool-call syntax matches a builtin xgrammar format.
62
+ structural_tag_model: str | None = None
63
+
64
+ def __init_subclass__(cls, **kwargs: Any) -> None:
65
+ super().__init_subclass__(**kwargs)
66
+ if (
67
+ cls.structural_tag_model is not None
68
+ and envs.VLLM_ENFORCE_STRICT_TOOL_CALLING
69
+ ):
70
+ cls.supports_required_and_named = False
71
72
def __init__(
73
self,
@@ -112,32 +123,16 @@ class ToolParser:
123
if not request.tools:
124
return request
125
115
- # Step 1 (highest priority for ChatCompletionRequest): apply
116
- # vLLM-owned structural tag support for model-specific tool formats.
126
+ # Set structured output params when tool constraints are derived from
127
+ # the tool schema. Unified parsers handle model-specific structural
128
+ # tags before calling into the tool parser.
129
+ structured_outputs = getattr(request, "structured_outputs", None)
130
if (
118
- isinstance(request, ChatCompletionRequest)
119
- and VLLM_ENFORCE_STRICT_TOOL_CALLING
131
+ structured_outputs is not None
132
+ and structured_outputs.structural_tag is not None
133
):
121
- need_tool_calling = (
122
- request.tool_choice == "auto"
123
- or request.tool_choice == "required"
124
- or isinstance(request.tool_choice, ChatCompletionNamedToolChoiceParam)
125
- )
126
- if need_tool_calling:
127
- structure_tag = self.get_structural_tag(request)
128
- if structure_tag is not None:
129
- if request.structured_outputs is None:
130
- request.structured_outputs = StructuredOutputsParams(
131
- structural_tag=json.dumps(structure_tag.model_dump()),
132
- )
133
- else:
134
- request.structured_outputs.structural_tag = json.dumps(
135
- structure_tag.model_dump()
136
- )
137
- return request
138
-
139
- # Step 2: set structured output params when tool constraints are
140
- # derived from the tool schema.
134
+ return request
135
+
136
json_schema_from_tool = get_json_schema_from_tools(
137
tool_choice=request.tool_choice, tools=request.tools
138
)
@@ -169,8 +164,21 @@ class ToolParser:
164
165
return request
166
172
- def get_structural_tag(self, request: ChatCompletionRequest):
173
- return None
167
+ def get_structural_tag(
168
+ self, request: ChatCompletionRequest, *, reasoning: bool = False
169
+ ):
170
+ if self.structural_tag_model is None:
171
+ return None
172
+ if not envs.VLLM_ENFORCE_STRICT_TOOL_CALLING:
173
+ return None
174
+ from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
175
+
176
+ return get_model_structural_tag(
177
+ model=self.structural_tag_model,
178
+ tools=request.tools,
179
+ tool_choice=request.tool_choice,
180
+ reasoning=reasoning,
181
+ )
182
183
def extract_tool_calls(
184
self, model_output: str, request: ChatCompletionRequest
vllm/tool_parsers/deepseekv31_tool_parser.py
+2
@@ -25,6 +25,8 @@ logger = init_logger(__name__)
25
26
27
class DeepSeekV31ToolParser(ToolParser):
28
+ structural_tag_model = "deepseek_v3_1"
29
+
30
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
31
super().__init__(tokenizer, tools)
32
vllm/tool_parsers/deepseekv32_tool_parser.py
+1
@@ -53,6 +53,7 @@ class DeepSeekV32ToolParser(ToolParser):
53
54
tool_call_start_token: str = "<｜DSML｜function_calls>"
55
tool_call_end_token: str = "</｜DSML｜function_calls>"
56
+ structural_tag_model = "deepseek_v3_2"
57
58
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
59
super().__init__(tokenizer, tools)
vllm/tool_parsers/deepseekv3_tool_parser.py
+2
@@ -28,6 +28,8 @@ logger = init_logger(__name__)
28
29
30
class DeepSeekV3ToolParser(ToolParser):
31
+ structural_tag_model = "deepseek_r1"
32
+
33
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
34
super().__init__(tokenizer, tools)
35
vllm/tool_parsers/deepseekv4_tool_parser.py
+1
-15
@@ -1,14 +1,7 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4
-from vllm.entrypoints.openai.chat_completion.protocol import (
5
- ChatCompletionRequest,
6
-)
4
from vllm.tool_parsers.deepseekv32_tool_parser import DeepSeekV32ToolParser
8
-from vllm.tool_parsers.structural_tag_registry import (
9
- get_enable_structured_outputs_in_reasoning,
10
- get_model_structural_tag,
11
-)
5
6
7
class DeepSeekV4ToolParser(DeepSeekV32ToolParser):
@@ -21,11 +14,4 @@ class DeepSeekV4ToolParser(DeepSeekV32ToolParser):
14
15
tool_call_start_token: str = "<｜DSML｜tool_calls>"
16
tool_call_end_token: str = "</｜DSML｜tool_calls>"
24
-
25
- def get_structural_tag(self, request: ChatCompletionRequest):
26
- return get_model_structural_tag(
27
- model="deepseek_v4",
28
- tools=request.tools,
29
- tool_choice=request.tool_choice,
30
- reasoning=get_enable_structured_outputs_in_reasoning(),
31
- )
17
+ structural_tag_model = "deepseek_v4"
vllm/tool_parsers/glm47_moe_tool_parser.py
+1
@@ -24,6 +24,7 @@ logger = init_logger(__name__)
24
25
class Glm47MoeModelToolParser(Glm4MoeModelToolParser):
26
supports_required_and_named = False
27
+ structural_tag_model = "glm_4_7"
28
29
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
30
super().__init__(tokenizer, tools)
vllm/tool_parsers/hermes_tool_parser.py
+1
@@ -32,6 +32,7 @@ logger = init_logger(__name__)
32
33
34
class Hermes2ProToolParser(ToolParser):
35
+ structural_tag_model = "hermes"
36
tool_call_start_token: str = "<tool_call>"
37
tool_call_end_token: str = "</tool_call>"
38
tool_call_regex = re.compile(
vllm/tool_parsers/kimi_k2_tool_parser.py
+2
@@ -29,6 +29,8 @@ logger = init_logger(__name__)
29
30
31
class KimiK2ToolParser(ToolParser):
32
+ structural_tag_model = "kimi"
33
+
34
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
35
super().__init__(tokenizer, tools)
36
vllm/tool_parsers/llama_tool_parser.py
+1
@@ -46,6 +46,7 @@ class Llama3JsonToolParser(ToolParser):
46
"""
47
48
bot_token: str = "<|python_tag|>"
49
+ structural_tag_model = "llama"
50
# Simple regex to find opening braces - we'll use JSON decoder for parsing
51
# This handles arbitrary nesting depth correctly
52
tool_call_start_regex: re.Pattern = re.compile(r"\{")
vllm/tool_parsers/minimax_m2_tool_parser.py
+2
@@ -34,6 +34,8 @@ logger = init_logger(__name__)
34
35
36
class MinimaxM2ToolParser(ToolParser):
37
+ structural_tag_model = "minimax"
38
+
39
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
40
super().__init__(tokenizer, tools)
41
vllm/tool_parsers/qwen3coder_tool_parser.py
+1
-14
@@ -18,17 +18,12 @@ from vllm.entrypoints.openai.engine.protocol import (
18
FunctionCall,
19
ToolCall,
20
)
21
-from vllm.envs import VLLM_ENFORCE_STRICT_TOOL_CALLING
21
from vllm.logger import init_logger
22
from vllm.tokenizers import TokenizerLike
23
from vllm.tool_parsers.abstract_tool_parser import (
24
Tool,
25
ToolParser,
26
)
28
-from vllm.tool_parsers.structural_tag_registry import (
29
- get_enable_structured_outputs_in_reasoning,
30
- get_model_structural_tag,
31
-)
27
from vllm.tool_parsers.utils import (
28
coerce_to_schema_type,
29
extract_types_from_schema,
@@ -39,7 +34,7 @@ logger = init_logger(__name__)
34
35
36
class Qwen3CoderToolParser(ToolParser):
42
- supports_required_and_named: bool = not VLLM_ENFORCE_STRICT_TOOL_CALLING
37
+ structural_tag_model = "qwen_3_coder"
38
39
def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
40
super().__init__(tokenizer, tools)
@@ -589,11 +584,3 @@ class Qwen3CoderToolParser(ToolParser):
584
return result
585
586
return None
592
-
593
- def get_structural_tag(self, request: ChatCompletionRequest):
594
- return get_model_structural_tag(
595
- model="qwen_3_5",
596
- tools=request.tools,
597
- tool_choice=request.tool_choice,
598
- reasoning=get_enable_structured_outputs_in_reasoning(),
599
- )
vllm/tool_parsers/qwen3xml_tool_parser.py
deleted
-1300
@@ -1,1300 +0,0 @@
1
-# SPDX-License-Identifier: Apache-2.0
2
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-import json
4
-from collections.abc import Sequence
5
-from typing import Any
6
-from xml.parsers.expat import ParserCreate
7
-
8
-import regex as re
9
-
10
-from vllm.entrypoints.chat_utils import make_tool_call_id
11
-from vllm.entrypoints.openai.chat_completion.protocol import (
12
- ChatCompletionRequest,
13
-)
14
-from vllm.entrypoints.openai.engine.protocol import (
15
- DeltaFunctionCall,
16
- DeltaMessage,
17
- DeltaToolCall,
18
- ExtractedToolCallInformation,
19
- FunctionCall,
20
- ToolCall,
21
-)
22
-from vllm.logger import init_logger
23
-from vllm.tokenizers import TokenizerLike
24
-from vllm.tool_parsers.abstract_tool_parser import (
25
- Tool,
26
- ToolParser,
27
-)
28
-from vllm.tool_parsers.utils import find_tool_properties, safe_literal_eval
29
-
30
-logger = init_logger(__name__)
31
-
32
-
33
-class StreamingXMLToolCallParser:
34
- """
35
- Simplified streaming XML tool call parser
36
- Supports streaming input, parsing, and output
37
- """
38
-
39
- def __init__(self):
40
- self.reset_streaming_state()
41
-
42
- # Tool configuration information
43
- self.tools: list[Tool] | None = None
44
- self.tool_call_start_token: str = "<tool_call>"
45
- self.tool_call_end_token: str = "</tool_call>"
46
- self.function_start_token: str = "<function="
47
- self.function_end_token: str = "</function>"
48
- self.parameter_start_token: str = "<parameter="
49
- self.parameter_end_token: str = "</parameter>"
50
-
51
- def reset_streaming_state(self):
52
- """Reset streaming parsing state"""
53
-
54
- self.deltas = []
55
- # state for streaming
56
- self.tool_call_index = 0
57
- self.current_call_id = None
58
- self.last_completed_call_id = None
59
- self.current_function_name = None
60
- self.current_function_open = False
61
- self.parameters = {}
62
- self.current_param_name = None
63
- self.current_param_value = ""
64
- self.current_param_value_converted = ""
65
- self.current_param_is_first = False
66
- self.should_emit_end_newline = False
67
- self.start_quote_emitted = False
68
-
69
- self.streaming_buffer = ""
70
- self.last_processed_pos = 0
71
-
72
- self.text_content_buffer = ""
73
-
74
- # state for preprocessing and deferred parsing
75
- self._pre_inside_parameter = False
76
- self._pre_param_buffer = ""
77
- self._pre_current_param_name = None
78
- self.defer_current_parameter = False
79
- self.deferred_param_raw_value = ""
80
-
81
- # recreate parser
82
- self.parser = ParserCreate()
83
- self.setup_parser()
84
-
85
- def parse_single_streaming_chunks(self, xml_chunk: str) -> DeltaMessage:
86
- """
87
- Parse single streaming XML chunk and return Delta response
88
- This is the actual streaming interface that receives chunks
89
- one by one and maintains internal state
90
-
91
- Args:
92
- xml_chunk: Single XML chunk string
93
- Returns:
94
- DeltaMessage: Contains delta information generated by this chunk,
95
- returns empty response if no complete elements
96
- """
97
- # Record delta count before processing
98
- initial_delta_count = len(self.deltas)
99
-
100
- self.streaming_buffer += xml_chunk
101
-
102
- found_elements = self._process_complete_xml_elements()
103
-
104
- if found_elements:
105
- # If complete elements found, check if end events were missed
106
- # some tags may not have been triggered
107
- try:
108
- new_deltas = self.deltas[initial_delta_count:]
109
- # If this chunk contains </function>
110
- # but didn't generate '}', then complete it
111
- if (
112
- self.current_call_id is not None
113
- and self.function_end_token in xml_chunk
114
- ):
115
- # - Added '}' (non-empty parameter ending)
116
- # - Added '{}' (empty parameter function)
117
- has_function_close = any(
118
- (
119
- td.tool_calls
120
- and any(
121
- (
122
- tc.function
123
- and tc.id == self.current_call_id
124
- and isinstance(tc.function.arguments, str)
125
- and (tc.function.arguments in ("}", "{}"))
126
- )
127
- for tc in td.tool_calls
128
- )
129
- )
130
- for td in new_deltas
131
- )
132
- if not has_function_close:
133
- # Close potentially unclosed element
134
- if self.current_param_name:
135
- self._end_element("parameter")
136
- if self.current_function_name:
137
- self._end_element("function")
138
- # If this chunk contains </tool_call>
139
- # but didn't generate final empty delta, then complete it
140
- if (
141
- self.current_call_id is not None
142
- and self.tool_call_end_token in xml_chunk
143
- ):
144
- has_toolcall_close = any(
145
- (
146
- td.tool_calls
147
- and any(
148
- (
149
- tc.type == "function"
150
- and tc.function
151
- and tc.function.arguments == ""
152
- and tc.id == self.current_call_id
153
- )
154
- for tc in td.tool_calls
155
- )
156
- )
157
- for td in new_deltas
158
- )
159
- if not has_toolcall_close:
160
- # Close potentially unclosed element
161
- if self.current_param_name:
162
- self._end_element("parameter")
163
- if self.current_function_name:
164
- self._end_element("function")
165
- self._end_element("tool_call")
166
- except Exception as e:
167
- logger.warning("Error with fallback parsing: %s", e)
168
- # Merge newly generated deltas into single response
169
- result_delta = self._merge_new_deltas_to_single_response(
170
- initial_delta_count
171
- )
172
- return result_delta
173
- else:
174
- # No complete elements, check if there's unoutput text content
175
- if self.text_content_buffer and self.tool_call_index == 0:
176
- # Has text content but no tool_call yet, output text content
177
- text_delta = DeltaMessage(content=self.text_content_buffer)
178
- self._emit_delta(text_delta)
179
- # Clear buffer to avoid duplicate output
180
- self.text_content_buffer = ""
181
- return text_delta
182
-
183
- # If this chunk contains end tags but wasn't triggered by parser,
184
- # manually complete end events
185
- # Only execute when still on the same call as when entered,
186
- # to prevent accidentally closing new calls
187
- # in multi <tool_call> scenarios
188
- if self.current_call_id is not None and (
189
- self.function_end_token in xml_chunk
190
- or self.tool_call_end_token in xml_chunk
191
- ):
192
- # Close potentially unclosed element
193
- if self.current_param_name:
194
- self._end_element("parameter")
195
- if self.function_end_token in xml_chunk and self.current_function_name:
196
- self._end_element("function")
197
- if self.tool_call_end_token in xml_chunk:
198
- self._end_element("tool_call")
199
- # Return the merged delta result generated by this fallback
200
- result_delta = self._merge_new_deltas_to_single_response(
201
- initial_delta_count
202
- )
203
- return result_delta
204
-
205
- # No complete elements, return empty response
206
- return DeltaMessage(content=None)
207
-
208
- def _escape_xml_special_chars(self, text: str) -> str:
209
- """
210
- Escape XML special characters
211
- Args:
212
- text: Original text
213
- Returns:
214
- Escaped text
215
- """
216
- xml_escapes = {
217
- "&": "&",
218
- "<": "<",
219
- ">": ">",
220
- '"': """,
221
- "'": "'",
222
- }
223
-
224
- for char, escape in xml_escapes.items():
225
- text = text.replace(char, escape)
226
-
227
- return text
228
-
229
- def _process_complete_xml_elements(self) -> bool:
230
- """
231
- Process complete XML elements in buffer
232
-
233
- Returns:
234
- bool: Whether complete elements were found and processed
235
- """
236
- found_any = False
237
-
238
- while self.last_processed_pos < len(self.streaming_buffer):
239
- # Find next complete xml element
240
- element, end_pos = self._find_next_complete_element(self.last_processed_pos)
241
- if element is None:
242
- # No complete element found, wait for more data
243
- break
244
-
245
- # Check if this element should be skipped
246
- if self._should_skip_element(element):
247
- self.last_processed_pos = end_pos
248
- continue
249
-
250
- # Found complete XML element, process it
251
- try:
252
- preprocessed_element = self._preprocess_xml_chunk(element)
253
- # Check if this is the first tool_call start
254
- if (
255
- (
256
- preprocessed_element.strip().startswith("<tool_call>")
257
- or preprocessed_element.strip().startswith("<function name=")
258
- )
259
- and self.tool_call_index == 0
260
- ) and self.text_content_buffer:
261
- # First tool_call starts,
262
- # output previously collected text content first
263
- text_delta = DeltaMessage(content=self.text_content_buffer)
264
- self._emit_delta(text_delta)
265
- # Clear buffer for potential subsequent text content
266
- self.text_content_buffer = ""
267
-
268
- # If a new tool_call starts and
269
- # there are already completed tool_calls
270
- if (
271
- preprocessed_element.strip().startswith("<tool_call>")
272
- and self.tool_call_index > 0
273
- and self.current_call_id
274
- ):
275
- # Reset parser state but preserve generated deltas
276
- if self.current_param_name:
277
- self._end_element("parameter")
278
- if self.current_function_open or self.current_function_name:
279
- self._end_element("function")
280
- # Output final tool_call tail delta
281
- final_delta = DeltaMessage(
282
- role=None,
283
- content=None,
284
- reasoning=None,
285
- tool_calls=[
286
- DeltaToolCall(
287
- index=self.tool_call_index - 1,
288
- id=self.current_call_id,
289
- type="function",
290
- function=DeltaFunctionCall(name=None, arguments=""),
291
- )
292
- ],
293
- )
294
- self._emit_delta(final_delta)
295
- # Reset XML parser and current call state
296
- self._reset_xml_parser_after_tool_call()
297
- # Parse preprocessed element
298
- self.parser.Parse(preprocessed_element, False)
299
- found_any = True
300
-
301
- except Exception as e:
302
- logger.warning("Error when parsing XML elements: %s", e)
303
-
304
- # Update processed position
305
- self.last_processed_pos = end_pos
306
-
307
- return found_any
308
-
309
- def _should_skip_element(self, element: str) -> bool:
310
- """
311
- Determine whether an element should be skipped
312
-
313
- Args:
314
- element: Element to evaluate
315
-
316
- Returns:
317
- bool: True means should skip, False means should process
318
- """
319
-
320
- # If it's a tool_call XML tag, don't skip
321
- if (
322
- element.startswith(self.tool_call_start_token)
323
- or element.startswith(self.function_start_token)
324
- or element.startswith(self.parameter_start_token)
325
- ):
326
- return False
327
-
328
- # If currently not parsing tool calls and not blank,
329
- # collect this text instead of skipping
330
- # Only process other XML elements after tool_call appears,
331
- # otherwise treat as plain text
332
- if self.current_call_id is None and element:
333
- # Collect text content to buffer
334
- self.text_content_buffer += element
335
- return True # Still skip, but content has been collected
336
-
337
- # If currently parsing tool calls,
338
- # this might be parameter value, don't skip
339
- if self.current_call_id is not None:
340
- return False
341
-
342
- # Skip blank content
343
- return not element
344
-
345
- def _find_next_complete_element(self, start_pos: int) -> tuple[str | None, int]:
346
- """
347
- Find next complete XML element from specified position
348
-
349
- Args:
350
- start_pos: Position to start searching
351
-
352
- Returns:
353
- (Complete element string, element end position),
354
- returns (None, start_pos) if no complete element found
355
- """
356
- buffer = self.streaming_buffer[start_pos:]
357
-
358
- if not buffer:
359
- return None, start_pos
360
-
361
- if buffer.startswith("<"):
362
- # Need to ensure no new < appears,
363
- # find the nearest one between < and >
364
- tag_end = buffer.find("<", 1)
365
- tag_end2 = buffer.find(">", 1)
366
- if tag_end != -1 and tag_end2 != -1:
367
- # Next nearest is <
368
- if tag_end < tag_end2:
369
- return buffer[:tag_end], start_pos + tag_end
370
- # Next nearest is >, means found XML element
371
- else:
372
- return buffer[: tag_end2 + 1], start_pos + tag_end2 + 1
373
- elif tag_end != -1:
374
- return buffer[:tag_end], start_pos + tag_end
375
- elif tag_end2 != -1:
376
- return buffer[: tag_end2 + 1], start_pos + tag_end2 + 1
377
- else:
378
- # If currently not parsing tool calls (entering a tool_call),
379
- # check if starts with <tool_call> or <function=
380
- if self.current_call_id is None:
381
- # Check if might be start of <tool_call>
382
- if buffer == "<tool_call>"[: len(buffer)]:
383
- # Might be start of <tool_call>, wait for more data
384
- return None, start_pos
385
- elif (
386
- buffer.startswith("<function=")
387
- or buffer == "<function="[: len(buffer)]
388
- ):
389
- # Might be start of <function=, wait for more data
390
- # to get the complete function tag
391
- return None, start_pos
392
- else:
393
- # Not start of <tool_call> or <function=, treat as text
394
- return buffer, start_pos + len(buffer)
395
- else:
396
- # When parsing tool calls,
397
- # wait for more data to get complete tag
398
- return None, start_pos
399
- else:
400
- # Find text content (until next < or buffer end)
401
- next_tag_pos = buffer.find("<")
402
- if next_tag_pos != -1:
403
- # Found text content
404
- text_content = buffer[:next_tag_pos]
405
- return text_content, start_pos + next_tag_pos
406
- else:
407
- # Buffer end is all text, process
408
- # (no longer wait for more data)
409
- remaining = buffer
410
- return remaining, start_pos + len(remaining)
411
-
412
- def _merge_new_deltas_to_single_response(self, initial_count: int) -> DeltaMessage:
413
- """
414
- Merge newly generated deltas from this processing
415
- into a single DeltaMessage
416
-
417
- Args:
418
- initial_count: Delta count before processing
419
-
420
- Returns:
421
- Merged DeltaMessage containing all newly generated delta information
422
- """
423
- if len(self.deltas) <= initial_count:
424
- return DeltaMessage(content=None)
425
-
426
- # Get newly generated deltas
427
- new_deltas = self.deltas[initial_count:]
428
-
429
- if len(new_deltas) == 1:
430
- # Only one new delta, return directly
431
- return new_deltas[0]
432
-
433
- # Merge multiple new deltas
434
- merged_tool_calls: list[DeltaToolCall] = []
435
- merged_content: str = ""
436
-
437
- for delta in new_deltas:
438
- if delta.content:
439
- merged_content += delta.content
440
- if delta.tool_calls:
441
- # For tool_calls, we need to intelligently merge arguments
442
- for tool_call in delta.tool_calls:
443
- # Find if there's already a tool_call with the same call_id
444
- existing_call = None
445
- for existing in merged_tool_calls:
446
- if existing.id == tool_call.id:
447
- existing_call = existing
448
- break
449
-
450
- if existing_call and existing_call.function:
451
- # Merge to existing tool_call
452
- if tool_call.function and tool_call.function.name:
453
- existing_call.function.name = tool_call.function.name
454
- if (
455
- tool_call.function
456
- and tool_call.function.arguments is not None
457
- ):
458
- if existing_call.function.arguments is None:
459
- existing_call.function.arguments = ""
460
-
461
- # For streaming JSON parameters,
462
- # simply concatenate in order
463
- new_args = tool_call.function.arguments
464
- existing_call.function.arguments += new_args
465
- if tool_call.type:
466
- existing_call.type = tool_call.type
467
- else:
468
- # Add new tool_call
469
- merged_tool_calls.append(tool_call)
470
-
471
- return DeltaMessage(
472
- content=merged_content if merged_content else None,
473
- tool_calls=merged_tool_calls,
474
- )
475
-
476
- def _preprocess_xml_chunk(self, chunk: str) -> str:
477
- """
478
- Preprocess XML chunk, handle non-standard formats,
479
- and escape special characters
480
-
481
- Args:
482
- chunk: Original XML chunk
483
-
484
- Returns:
485
- Processed XML chunk
486
- """
487
-
488
- # Check if this is a tool_call related element
489
- is_tool_call = False
490
- if chunk.startswith(self.tool_call_start_token) or chunk.startswith(
491
- self.tool_call_end_token
492
- ):
493
- is_tool_call = True
494
- if chunk.startswith(self.function_start_token) or chunk.startswith(
495
- self.function_end_token
496
- ):
497
- is_tool_call = True
498
- if chunk.startswith(self.parameter_start_token) or chunk.startswith(
499
- self.parameter_end_token
500
- ):
501
- is_tool_call = True
502
- # Handle <function=name> format -> <function name="name">
503
- processed = re.sub(r"<function=([^>]+)>", r'<function name="\1">', chunk)
504
- # Handle <parameter=name> format -> <parameter name="name">
505
- processed = re.sub(r"<parameter=([^>]+)>", r'<parameter name="\1">', processed)
506
-
507
- original_chunk = chunk
508
- # If in parameter value accumulation mode
509
- if self._pre_inside_parameter:
510
- # Parameter end: output accumulated raw text
511
- # safely then return </parameter>
512
- if processed.startswith("</parameter>"):
513
- body_text = self._pre_param_buffer
514
- # Trigger deferred parsing mode
515
- # literal_eval+json output in end_element
516
- self.defer_current_parameter = True
517
- self.deferred_param_raw_value = body_text
518
- # Clean up state
519
- self._pre_inside_parameter = False
520
- self._pre_param_buffer = ""
521
- self._pre_current_param_name = None
522
- safe_text = self._escape_xml_special_chars(body_text)
523
- return f"{safe_text}</parameter>"
524
- else:
525
- # If this is the first block of content after entering parameter
526
- # evaluate if deferred parsing is needed;
527
- # If not needed, exit accumulation mode
528
- # and pass through directly
529
- if self._pre_param_buffer == "":
530
- # Get current parameter type
531
- param_type = (
532
- self._get_param_type(self._pre_current_param_name)
533
- if self._pre_current_param_name
534
- else "string"
535
- )
536
- # Only these types need deferred parsing to
537
- # handle Python literals containing single quotes
538
- is_object_type = param_type in ["object"]
539
- is_complex_type = (
540
- param_type in ["array", "arr", "sequence"]
541
- or param_type.startswith("dict")
542
- or param_type.startswith("list")
543
- )
544
-
545
- # Only delay when contains container symbols
546
- # and has single quotes and is complex type
547
- has_container_hint = (
548
- ("[" in original_chunk)
549
- or ("{" in original_chunk)
550
- or ("(" in original_chunk)
551
- )
552
-
553
- # Determine if deferred parsing is needed
554
- need_defer = False
555
- if is_complex_type:
556
- # Complex type, always need deferred parsing
557
- need_defer = True
558
- elif (
559
- is_object_type
560
- and has_container_hint
561
- and ("'" in original_chunk)
562
- ):
563
- # Object type with container symbols
564
- # and single quotes, need deferred parsing
565
- need_defer = True
566
-
567
- if not need_defer:
568
- # No need for deferred parsing,
569
- # exit parameter mode directly
570
- self._pre_inside_parameter = False
571
- return self._escape_xml_special_chars(original_chunk)
572
- self._pre_param_buffer += original_chunk
573
- return ""
574
-
575
- # Parameter start: enable accumulation
576
- if processed.startswith("<parameter name="):
577
- m = re.match(r'<parameter name="([^"]+)">', processed)
578
- if m:
579
- self._pre_current_param_name = m.group(1)
580
- self._pre_inside_parameter = True
581
- self._pre_param_buffer = ""
582
- return processed
583
-
584
- # If processed doesn't contain special_token, escape processed
585
- # This is because XML parsing encounters special characters
586
- # and reports errors, so escaping is needed
587
- if not is_tool_call:
588
- processed = self._escape_xml_special_chars(processed)
589
- return processed
590
-
591
- def _emit_delta(self, delta: DeltaMessage):
592
- """Emit Delta response (streaming output)"""
593
- self.deltas.append(delta)
594
-
595
- def _auto_close_open_parameter_if_needed(self, incoming_tag: str | None = None):
596
- """Before starting to process new elements,
597
- if there are unclosed tags from before,
598
- automatically complete their endings to the parser.
599
- - If there are unclosed parameters,
600
- it's equivalent to feeding `</parameter>`
601
- - When about to start a new function or tool_call,
602
- if there are unclosed functions, complete `</function>`.
603
- - When about to start a new tool_call,
604
- if there are unclosed tool_calls, complete `</tool_call>`.
605
- """
606
- # First close unclosed parameters
607
- if self.current_param_name:
608
- self._end_element("parameter")
609
-
610
- # If about to start new function or tool_call,
611
- # and there are unclosed functions, close function first
612
- if incoming_tag in ("function", "tool_call") and self.current_function_name:
613
- self._end_element("function")
614
-
615
- # If about to start new tool_call,
616
- # and there are unclosed tool_calls, close tool_call first
617
- if incoming_tag == "tool_call" and self.current_call_id:
618
- self._end_element("tool_call")
619
-
620
- def _start_element(self, name: str, attrs: dict[str, str]):
621
- """Handle XML start element events"""
622
-
623
- if name == "root":
624
- return
625
-
626
- if name == "tool_call":
627
- # Before opening new tool_call,
628
- # automatically complete previous unclosed tags
629
- self._auto_close_open_parameter_if_needed("tool_call")
630
-
631
- self.parameters = {}
632
- self.current_call_id = make_tool_call_id()
633
- self.current_param_is_first = True
634
- self.tool_call_index += 1
635
- elif name.startswith("function") or (name == "function"):
636
- # If missing tool_call, manually complete
637
- if not self.current_call_id:
638
- self._start_element("tool_call", {})
639
- # Before opening new function,
640
- # automatically complete previous unclosed tags (parameter/function)
641
- self._auto_close_open_parameter_if_needed("function")
642
- function_name = self._extract_function_name(name, attrs)
643
- self.current_function_name = function_name
644
- self.current_function_open = True
645
- if function_name:
646
- delta = DeltaMessage(
647
- tool_calls=[
648
- DeltaToolCall(
649
- index=self.tool_call_index - 1,
650
- id=self.current_call_id,
651
- type="function",
652
- function=DeltaFunctionCall(
653
- name=function_name, arguments=""
654
- ),
655
- )
656
- ]
657
- )
658
- self._emit_delta(delta)
659
- elif name.startswith("parameter") or (name == "parameter"):
660
- # If previous parameter hasn't ended normally,
661
- # complete its end first, then start new parameter
662
- self._auto_close_open_parameter_if_needed("parameter")
663
- param_name = self._extract_parameter_name(name, attrs)
664
- self.current_param_name = param_name
665
- self.current_param_value = ""
666
- self.current_param_value_converted = ""
667
- self.start_quote_emitted = False # Reset start quote flag
668
-
669
- # Only output parameter name and colon,
670
- # don't output quotes
671
- # decide after parameter value type is determined
672
- if param_name:
673
- if not self.parameters:
674
- # First parameter
675
- # start JSON, only output parameter name and colon
676
- json_start = f'{{"{param_name}": '
677
- delta = DeltaMessage(
678
- tool_calls=[
679
- DeltaToolCall(
680
- index=self.tool_call_index - 1,
681
- id=self.current_call_id,
682
- type="function",
683
- function=DeltaFunctionCall(
684
- name=None, arguments=json_start
685
- ),
686
- )
687
- ]
688
- )
689
- self._emit_delta(delta)
690
- self.current_param_is_first = True
691
- else:
692
- # Subsequent parameters
693
- # add comma and parameter name, no quotes
694
- json_continue = f', "{param_name}": '
695
- delta = DeltaMessage(
696
- tool_calls=[
697
- DeltaToolCall(
698
- index=self.tool_call_index - 1,
699
- id=self.current_call_id,
700
- type="function",
701
- function=DeltaFunctionCall(
702
- name=None, arguments=json_continue
703
- ),
704
- )
705
- ]
706
- )
707
- self._emit_delta(delta)
708
- self.current_param_is_first = False
709
-
710
- def _char_data(self, data: str):
711
- """Handle XML character data events"""
712
- if data and self.current_param_name:
713
- # If preprocessing stage determines deferred parsing is needed,
714
- # only cache character data, no streaming output
715
- if self.defer_current_parameter:
716
- original_data = data
717
- if self.should_emit_end_newline:
718
- original_data = "\n" + original_data
719
- self.should_emit_end_newline = False
720
- if original_data.endswith("\n"):
721
- self.should_emit_end_newline = True
722
- original_data = original_data[:-1]
723
- self.current_param_value += original_data
724
- return
725
-
726
- param_type = self._get_param_type(self.current_param_name)
727
-
728
- # Check if this is the first time receiving data for this parameter
729
- # If this is the first packet of data and starts with \n, remove \n
730
- if not self.current_param_value and data.startswith("\n"):
731
- data = data[1:]
732
-
733
- # Output start quote for string type (if not already output)
734
- if (
735
- param_type in ["string", "str", "text", "varchar", "char", "enum"]
736
- and not self.start_quote_emitted
737
- ):
738
- quote_delta = DeltaMessage(
739
- tool_calls=[
740
- DeltaToolCall(
741
- index=self.tool_call_index - 1,
742
- id=self.current_call_id,
743
- type="function",
744
- function=DeltaFunctionCall(name=None, arguments='"'),
745
- )
746
- ]
747
- )
748
- self._emit_delta(quote_delta)
749
- self.start_quote_emitted = True
750
-
751
- if not data:
752
- return
753
-
754
- original_data = data
755
- # Delay output of trailing newline
756
- if self.should_emit_end_newline:
757
- original_data = "\n" + original_data
758
- self.should_emit_end_newline = False
759
- if original_data.endswith("\n"):
760
- self.should_emit_end_newline = True
761
- original_data = original_data[:-1]
762
- self.current_param_value += original_data
763
-
764
- # convert parameter value by param_type
765
- converted_value = self._convert_param_value(
766
- self.current_param_value, param_type
767
- )
768
- output_data = self._convert_for_json_streaming(converted_value, param_type)
769
-
770
- delta_data = output_data[len(self.current_param_value_converted) :]
771
- self.current_param_value_converted = output_data
772
-
773
- delta = DeltaMessage(
774
- tool_calls=[
775
- DeltaToolCall(
776
- index=self.tool_call_index - 1,
777
- id=self.current_call_id,
778
- type="function",
779
- function=DeltaFunctionCall(name=None, arguments=delta_data),
780
- )
781
- ]
782
- )
783
- self._emit_delta(delta)
784
-
785
- def _end_element(self, name: str):
786
- """Handle XML end element events"""
787
-
788
- if name == "root":
789
- return
790
-
791
- # If function or tool_call ends and there are still unclosed parameters,
792
- # complete parameter end first
793
- if (
794
- name.startswith("function") or name == "function" or name == "tool_call"
795
- ) and self.current_param_name:
796
- self._auto_close_open_parameter_if_needed()
797
-
798
- if (
799
- name.startswith("parameter") or name == "parameter"
800
- ) and self.current_param_name:
801
- # End current parameter
802
- param_name = self.current_param_name
803
- param_value = self.current_param_value
804
-
805
- # If in deferred parsing mode,
806
- # perform overall parsing on raw content
807
- # accumulated in preprocessing stage and output once
808
- if self.defer_current_parameter:
809
- raw_text = (
810
- self.deferred_param_raw_value
811
- if self.deferred_param_raw_value
812
- else param_value
813
- )
814
- parsed_value = None
815
- output_arguments = None
816
- try:
817
- # If previously delayed trailing newline,
818
- # add it back before parsing
819
- if self.should_emit_end_newline:
820
- raw_for_parse = raw_text + "\n"
821
- else:
822
- raw_for_parse = raw_text
823
- try:
824
- parsed_value = json.loads(raw_for_parse)
825
- except json.JSONDecodeError:
826
- parsed_value = safe_literal_eval(raw_for_parse)
827
- output_arguments = json.dumps(parsed_value, ensure_ascii=False)
828
- except Exception:
829
- # Fallback: output as string as-is
830
- output_arguments = json.dumps(raw_text, ensure_ascii=False)
831
- parsed_value = raw_text
832
-
833
- delta = DeltaMessage(
834
- tool_calls=[
835
- DeltaToolCall(
836
- index=self.tool_call_index - 1,
837
- id=self.current_call_id,
838
- type="function",
839
- function=DeltaFunctionCall(
840
- name=None, arguments=output_arguments
841
- ),
842
- )
843
- ]
844
- )
845
- self._emit_delta(delta)
846
-
847
- # Clean up and store
848
- self.should_emit_end_newline = False
849
- self.parameters[param_name] = parsed_value
850
- self.current_param_name = None
851
- self.current_param_value = ""
852
- self.current_param_value_converted = ""
853
- self.start_quote_emitted = False
854
- self.defer_current_parameter = False
855
- self.deferred_param_raw_value = ""
856
- return
857
-
858
- param_type = self._get_param_type(param_name)
859
-
860
- # convert complete parameter value by param_type
861
- converted_value = self._convert_param_value(param_value, param_type)
862
-
863
- # Decide whether to add end quote based on parameter type
864
- if param_type in ["string", "str", "text", "varchar", "char", "enum"]:
865
- # For empty string parameters, need special handling
866
- if not param_value and not self.start_quote_emitted:
867
- # No start quote output,
868
- # directly output complete empty string
869
- delta = DeltaMessage(
870
- tool_calls=[
871
- DeltaToolCall(
872
- index=self.tool_call_index - 1,
873
- id=self.current_call_id,
874
- type="function",
875
- function=DeltaFunctionCall(name=None, arguments='""'),
876
- )
877
- ]
878
- )
879
- self._emit_delta(delta)
880
- else:
881
- # Non-empty parameter value, output end quote
882
- delta = DeltaMessage(
883
- tool_calls=[
884
- DeltaToolCall(
885
- index=self.tool_call_index - 1,
886
- id=self.current_call_id,
887
- type="function",
888
- function=DeltaFunctionCall(name=None, arguments='"'),
889
- )
890
- ]
891
- )
892
- self._emit_delta(delta)
893
-
894
- self.should_emit_end_newline = False
895
- # Store converted value
896
- self.parameters[param_name] = converted_value
897
- self.current_param_name = None
898
- self.current_param_value = ""
899
- self.current_param_value_converted = ""
900
- self.start_quote_emitted = False
901
-
902
- elif name.startswith("function") or name == "function":
903
- # if there are parameters, close JSON object
904
- if self.parameters:
905
- delta = DeltaMessage(
906
- tool_calls=[
907
- DeltaToolCall(
908
- index=self.tool_call_index - 1,
909
- id=self.current_call_id,
910
- type="function",
911
- function=DeltaFunctionCall(name=None, arguments="}"),
912
- )
913
- ]
914
- )
915
- self._emit_delta(delta)
916
- # return empty object
917
- else:
918
- delta = DeltaMessage(
919
- tool_calls=[
920
- DeltaToolCall(
921
- index=self.tool_call_index - 1,
922
- id=self.current_call_id,
923
- type="function",
924
- function=DeltaFunctionCall(name=None, arguments="{}"),
925
- )
926
- ]
927
- )
928
- self._emit_delta(delta)
929
- self.current_function_open = False
930
-
931
- elif name == "tool_call":
932
- # Before ending tool_call,
933
- # ensure function is closed to complete missing right brace
934
- if self.current_function_open:
935
- # If there are still unclosed parameters, close them first
936
- if self.current_param_name:
937
- self._end_element("parameter")
938
- # Close function, ensure output '}' or '{}'
939
- self._end_element("function")
940
- # Final Delta
941
- delta = DeltaMessage(
942
- tool_calls=[
943
- DeltaToolCall(
944
- index=self.tool_call_index - 1,
945
- id=self.current_call_id,
946
- type="function",
947
- function=DeltaFunctionCall(name=None, arguments=""),
948
- )
949
- ]
950
- )
951
- self._emit_delta(delta)
952
-
953
- # Check if there's text content to output (between tool_calls)
954
- if self.text_content_buffer.strip():
955
- text_delta = DeltaMessage(content=self.text_content_buffer)
956
- self._emit_delta(text_delta)
957
-
958
- self._reset_xml_parser_after_tool_call()
959
-
960
- def setup_parser(self):
961
- """Set up XML parser event handlers"""
962
- self.parser.buffer_text = True
963
- self.parser.StartElementHandler = self._start_element
964
- self.parser.EndElementHandler = self._end_element
965
- self.parser.CharacterDataHandler = self._char_data
966
-
967
- def set_tools(self, tools: list[Tool] | None):
968
- """Set tool configuration information"""
969
- self.tools = tools
970
-
971
- def _extract_function_name(self, name: str, attrs: dict[str, str]) -> str | None:
972
- """Extract function name from various formats"""
973
- if attrs and "name" in attrs:
974
- return attrs["name"]
975
-
976
- if "=" in name:
977
- parts = name.split("=", 1)
978
- if len(parts) == 2 and parts[0] == "function":
979
- return parts[1]
980
-
981
- return None
982
-
983
- def _extract_parameter_name(self, name: str, attrs: dict[str, str]) -> str | None:
984
- """Extract parameter name from various formats"""
985
- if attrs and "name" in attrs:
986
- return attrs["name"]
987
-
988
- if "=" in name:
989
- parts = name.split("=", 1)
990
- if len(parts) == 2 and parts[0] == "parameter":
991
- return parts[1]
992
-
993
- return None
994
-
995
- def _get_param_type(self, param_name: str) -> str:
996
- """Get parameter type based on tool configuration, defaults to string
997
- Args:
998
- param_name: Parameter name
999
-
1000
- Returns:
1001
- Parameter type
1002
- """
1003
- if not self.tools or not self.current_function_name:
1004
- return "string"
1005
-
1006
- properties = find_tool_properties(self.tools, self.current_function_name)
1007
- if param_name in properties and isinstance(properties[param_name], dict):
1008
- return self.repair_param_type(
1009
- str(properties[param_name].get("type", "string"))
1010
- )
1011
- return "string"
1012
-
1013
- def repair_param_type(self, param_type: str) -> str:
1014
- """Repair unknown parameter types by treating them as string
1015
- Args:
1016
- param_type: Parameter type
1017
-
1018
- Returns:
1019
- Repaired parameter type
1020
- """
1021
- if (
1022
- param_type in ["string", "str", "text", "varchar", "char", "enum"]
1023
- or param_type.startswith("int")
1024
- or param_type.startswith("uint")
1025
- or param_type.startswith("long")
1026
- or param_type.startswith("short")
1027
- or param_type.startswith("unsigned")
1028
- or param_type.startswith("num")
1029
- or param_type.startswith("float")
1030
- or param_type in ["boolean", "bool", "binary"]
1031
- or (
1032
- param_type in ["object", "array", "arr", "sequence"]
1033
- or param_type.startswith("dict")
1034
- or param_type.startswith("list")
1035
- )
1036
- ):
1037
- return param_type
1038
- else:
1039
- return "string"
1040
-
1041
- def _convert_param_value(self, param_value: str, param_type: str) -> Any:
1042
- """Convert value based on parameter type
1043
- Args:
1044
- param_value: Parameter value
1045
- param_type: Parameter type
1046
-
1047
- Returns:
1048
- Converted value
1049
- """
1050
- if param_value.lower() == "null":
1051
- return None
1052
-
1053
- param_type = param_type.strip().lower()
1054
- if param_type in ["string", "str", "text", "varchar", "char", "enum"]:
1055
- return param_value
1056
- elif (
1057
- param_type.startswith("int")
1058
- or param_type.startswith("uint")
1059
- or param_type.startswith("long")
1060
- or param_type.startswith("short")
1061
- or param_type.startswith("unsigned")
1062
- ):
1063
- try:
1064
- return int(param_value)
1065
- except (ValueError, TypeError):
1066
- logger.warning(
1067
- "Parsed value '%s' of parameter '%s' is not an integer "
1068
- "in tool '%s', degenerating to string.",
1069
- param_value,
1070
- )
1071
- return param_value
1072
- elif param_type.startswith("num") or param_type.startswith("float"):
1073
- try:
1074
- float_param_value: float = float(param_value)
1075
- return (
1076
- float_param_value
1077
- if float_param_value - int(float_param_value) != 0
1078
- else int(float_param_value)
1079
- )
1080
- except (ValueError, TypeError):
1081
- logger.warning(
1082
- "Parsed value '%s' of parameter '%s' is not a float "
1083
- "in tool '%s', degenerating to string.",
1084
- param_value,
1085
- )
1086
- return param_value
1087
- elif param_type in ["boolean", "bool", "binary"]:
1088
- param_value = param_value.lower()
1089
- return param_value == "true"
1090
- else:
1091
- return param_value
1092
-
1093
- def _convert_for_json_streaming(self, converted_value: Any, param_type: str) -> str:
1094
- """Convert converted_value based on
1095
- whether it's empty and if type is string
1096
- Args:
1097
- converted_value: Converted value
1098
- param_type: Parameter type
1099
-
1100
- Returns:
1101
- Converted string for streaming output
1102
- """
1103
- # Check if value is empty, but exclude numeric 0
1104
- if converted_value is None or converted_value == "":
1105
- return ""
1106
-
1107
- if param_type in ["string", "str", "text", "varchar", "char", "enum"]:
1108
- # String type, remove double quotes
1109
- return json.dumps(converted_value, ensure_ascii=False)[1:-1]
1110
- else:
1111
- # Non-string type, return complete JSON string
1112
- if not isinstance(converted_value, str):
1113
- return json.dumps(converted_value, ensure_ascii=False)
1114
- else:
1115
- return converted_value
1116
-
1117
- def _reset_xml_parser_after_tool_call(self):
1118
- """
1119
- Each tool_call is treated as a separate XML document,
1120
- so we need to reset the parser after each tool_call.
1121
- """
1122
-
1123
- # recreate XML parser
1124
- self.parser = ParserCreate()
1125
- self.setup_parser()
1126
-
1127
- # Reset current tool_call state
1128
- if self.current_call_id:
1129
- self.last_completed_call_id = self.current_call_id
1130
- self.current_call_id = None
1131
- self.current_function_name = None
1132
- self.current_function_open = False
1133
- self.parameters = {}
1134
- self.current_param_name = None
1135
- self.current_param_value = ""
1136
- self.current_param_value_converted = ""
1137
- self.current_param_is_first = False
1138
- self.should_emit_end_newline = False
1139
- self.start_quote_emitted = False
1140
- self.text_content_buffer = ""
1141
-
1142
- # Reset preprocessing and deferred parsing state
1143
- self._pre_inside_parameter = False
1144
- self._pre_param_buffer = ""
1145
- self._pre_current_param_name = None
1146
- self.defer_current_parameter = False
1147
- self.deferred_param_raw_value = ""
1148
-
1149
-
1150
-class Qwen3XMLToolParser(ToolParser):
1151
- def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):
1152
- super().__init__(tokenizer, tools)
1153
- self.parser = StreamingXMLToolCallParser()
1154
-
1155
- # Add missing attributes for compatibility with serving_chat.py
1156
- self.prev_tool_call_arr: list[dict] = []
1157
- self.streamed_args_for_tool: list[str] = []
1158
-
1159
- logger.info(
1160
- "vLLM Successfully import tool parser %s !", self.__class__.__name__
1161
- )
1162
-
1163
- def extract_tool_calls(
1164
- self,
1165
- model_output: str,
1166
- request: ChatCompletionRequest,
1167
- ) -> ExtractedToolCallInformation:
1168
- self.parser.reset_streaming_state()
1169
- # Reset tool call tracking arrays for new extraction
1170
- self.prev_tool_call_arr = []
1171
- self.streamed_args_for_tool = []
1172
- self.parser.set_tools(self.tools)
1173
- result = self.parser.parse_single_streaming_chunks(model_output)
1174
- if not result.tool_calls:
1175
- return ExtractedToolCallInformation(
1176
- tool_calls=[],
1177
- tools_called=False,
1178
- content=result.content,
1179
- )
1180
- else:
1181
- tool_calls = []
1182
- for tool_call in result.tool_calls:
1183
- if tool_call.function and tool_call.function.name:
1184
- tool_calls.append(
1185
- ToolCall(
1186
- id=tool_call.id,
1187
- type=tool_call.type,
1188
- function=FunctionCall(
1189
- name=tool_call.function.name,
1190
- arguments=tool_call.function.arguments,
1191
- ),
1192
- )
1193
- )
1194
-
1195
- # Update tool call tracking arrays for compatibility
1196
- tool_index = (
1197
- tool_call.index
1198
- if tool_call.index is not None
1199
- else len(self.prev_tool_call_arr) - 1
1200
- )
1201
-
1202
- # Ensure we have enough entries in our tracking arrays
1203
- while len(self.prev_tool_call_arr) <= tool_index:
1204
- self.prev_tool_call_arr.append({"name": "", "arguments": ""})
1205
- while len(self.streamed_args_for_tool) <= tool_index:
1206
- self.streamed_args_for_tool.append("")
1207
-
1208
- # Update tool call information
1209
- self.prev_tool_call_arr[tool_index]["name"] = (
1210
- tool_call.function.name
1211
- )
1212
- self.prev_tool_call_arr[tool_index]["arguments"] = (
1213
- tool_call.function.arguments
1214
- )
1215
-
1216
- # Update streamed arguments
1217
- if tool_call.function.arguments:
1218
- self.streamed_args_for_tool[tool_index] = (
1219
- tool_call.function.arguments
1220
- )
1221
-
1222
- return ExtractedToolCallInformation(
1223
- tool_calls=tool_calls,
1224
- tools_called=len(tool_calls) > 0,
1225
- content=result.content,
1226
- )
1227
-
1228
- def extract_tool_calls_streaming(
1229
- self,
1230
- previous_text: str,
1231
- current_text: str,
1232
- delta_text: str,
1233
- previous_token_ids: Sequence[int],
1234
- current_token_ids: Sequence[int],
1235
- delta_token_ids: Sequence[int],
1236
- request: ChatCompletionRequest,
1237
- ) -> DeltaMessage | None:
1238
- if not previous_text:
1239
- self.parser.reset_streaming_state()
1240
- # Reset tool call tracking arrays for new streaming session
1241
- self.prev_tool_call_arr = []
1242
- self.streamed_args_for_tool = []
1243
- self.parser.set_tools(self.tools)
1244
-
1245
- # Model sometimes outputs separately causing delta_text to be empty.
1246
- # If there were tool_calls before and all current tool_calls have ended,
1247
- # return an empty tool_call for outer streaming output
1248
- # to correctly output tool_call field
1249
- if not delta_text and delta_token_ids:
1250
- open_calls = current_text.count(
1251
- self.parser.tool_call_start_token
1252
- ) - current_text.count(self.parser.tool_call_end_token)
1253
- if (
1254
- open_calls == 0
1255
- and self.parser.tool_call_index > 0
1256
- or not self.parser.tool_call_index
1257
- and current_text
1258
- ):
1259
- return DeltaMessage(content="")
1260
- return None
1261
-
1262
- # Parse the delta text and get the result
1263
- delta = self.parser.parse_single_streaming_chunks(delta_text)
1264
-
1265
- # Update tool call tracking arrays based on incremental parsing results
1266
- if delta and delta.tool_calls:
1267
- for tool_call in delta.tool_calls:
1268
- if tool_call.function:
1269
- tool_index = (
1270
- tool_call.index
1271
- if tool_call.index is not None
1272
- else len(self.prev_tool_call_arr) - 1
1273
- )
1274
-
1275
- # Ensure we have enough entries in our tracking arrays
1276
- while len(self.prev_tool_call_arr) <= tool_index:
1277
- self.prev_tool_call_arr.append({"name": "", "arguments": ""})
1278
- while len(self.streamed_args_for_tool) <= tool_index:
1279
- self.streamed_args_for_tool.append("")
1280
-
1281
- # Update tool name if provided
1282
- if tool_call.function.name:
1283
- self.prev_tool_call_arr[tool_index]["name"] = (
1284
- tool_call.function.name
1285
- )
1286
-
1287
- # Update arguments incrementally
1288
- if tool_call.function.arguments is not None:
1289
- # Concatenate the incremental arguments
1290
- # to the existing streamed arguments
1291
- self.prev_tool_call_arr[tool_index]["arguments"] += (
1292
- tool_call.function.arguments
1293
- )
1294
- self.streamed_args_for_tool[tool_index] += (
1295
- tool_call.function.arguments
1296
- )
1297
- if delta.content is None and not delta.tool_calls and delta.reasoning is None:
1298
- # If no content and no tool calls, return None to indicate no update
1299
- return None
1300
- return delta
vllm/tool_parsers/structural_tag_registry.py
+174
-240
@@ -1,14 +1,15 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
4
-# Model-specific structural tag builders adapted from XGrammar's
5
-# builtin structural tag implementations:
6
-# https://github.com/mlc-ai/xgrammar/blob/main/python/xgrammar/builtin_structural_tag.py
7
-
4
from collections.abc import Callable
5
from typing import Any, Literal
6
11
-from xgrammar import StructuralTag
7
+from xgrammar import StructuralTag, normalize_tool_choice
8
+from xgrammar import get_model_structural_tag as get_xgrammar_model_structural_tag
9
+from xgrammar.openai_tool_call_schema import (
10
+ BuiltinToolParam,
11
+ FunctionToolParam,
12
+)
13
from xgrammar.structural_tag import (
14
AnyTextFormat,
15
ConstStringFormat,
@@ -24,23 +25,51 @@ from vllm.entrypoints.openai.chat_completion.protocol import (
25
ChatCompletionToolsParam,
26
)
27
27
-SimplifiedToolChoice = Literal["auto", "required", "forced"]
28
ToolChoice = (
29
Literal["none", "auto", "required"] | ChatCompletionNamedToolChoiceParam | None
30
)
31
+SimplifiedToolChoice = Literal["auto", "required", "forced"]
32
StructuralTagBuilder = Callable[
32
- [list[ChatCompletionToolsParam], SimplifiedToolChoice, bool],
33
+ [
34
+ list[FunctionToolParam],
35
+ list[BuiltinToolParam],
36
+ SimplifiedToolChoice,
37
+ bool,
38
+ ],
39
StructuralTag,
40
]
41
36
-_structural_tag_registry: dict[str, StructuralTagBuilder] = {}
42
+# Keep this list in sync with xgrammar.builtin_structural_tag. It is used for
43
+# vLLM-side validation and for documenting the xgrammar builtin surface that
44
+# can be requested by tool parsers through ``structural_tag_model``.
45
+XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS = frozenset(
46
+ {
47
+ "llama",
48
+ "kimi",
49
+ "deepseek_r1",
50
+ "deepseek_v3_1",
51
+ "qwen_3_5",
52
+ "qwen_3_coder",
53
+ "qwen_3",
54
+ "harmony",
55
+ "deepseek_v3_2",
56
+ "glm_4_7",
57
+ "deepseek_v4",
58
+ }
59
+)
60
+VLLM_BUILTIN_STRUCTURAL_TAG_MODELS = frozenset({"hermes"})
61
+SUPPORTED_STRUCTURAL_TAG_MODELS = (
62
+ XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS | VLLM_BUILTIN_STRUCTURAL_TAG_MODELS
63
+)
64
+
65
+_VLLM_STRUCTURAL_TAG_REGISTRY: dict[str, StructuralTagBuilder] = {}
66
67
39
-def register_model_structural_tag(name: str):
40
- """Register a vLLM-owned model-specific structural tag builder."""
68
+def register_vllm_structural_tag(model: str):
69
+ """Register a vLLM-owned structural tag builder."""
70
71
def decorator(func: StructuralTagBuilder) -> StructuralTagBuilder:
43
- _structural_tag_registry[name] = func
72
+ _VLLM_STRUCTURAL_TAG_REGISTRY[model] = func
73
return func
74
75
return decorator
@@ -52,279 +81,184 @@ def get_model_structural_tag(
81
tool_choice: ToolChoice,
82
reasoning: bool,
83
) -> StructuralTag | None:
55
- """Build a structural tag from vLLM-owned model-specific builders."""
56
-
57
- builder = _structural_tag_registry.get(model)
58
- if builder is None:
59
- supported = list(_structural_tag_registry.keys())
60
- raise ValueError(f"Unknown format type: {model}, supported types: {supported}")
84
+ """Build a structural tag with xgrammar's builtin model templates."""
85
62
- normalized_tools, simplified_tool_choice = _normalize_tool_choice(
63
- tools=tools,
64
- tool_choice=tool_choice,
65
- )
66
- if not normalized_tools:
86
+ if not tools or tool_choice == "none":
87
return None
88
69
- return builder(normalized_tools, simplified_tool_choice, reasoning)
70
-
89
+ dumped_tools = [_model_dump(tool) for tool in tools]
90
+ dumped_tool_choice = _model_dump(tool_choice)
91
72
-def _normalize_tool_choice(
73
- tools: list[ChatCompletionToolsParam] | None,
74
- tool_choice: ToolChoice,
75
-) -> tuple[list[ChatCompletionToolsParam], SimplifiedToolChoice]:
76
- """Normalize vLLM ChatCompletion tool_choice for structural tag builders."""
77
-
78
- if not tools:
79
- return [], "auto"
80
-
81
- if tool_choice is None or tool_choice == "none":
82
- return [], "auto"
92
+ if model in _VLLM_STRUCTURAL_TAG_REGISTRY:
93
+ function_tools, builtin_tools, simplified_tool_choice = normalize_tool_choice(
94
+ dumped_tools,
95
+ dumped_tool_choice,
96
+ )
97
+ return _VLLM_STRUCTURAL_TAG_REGISTRY[model](
98
+ function_tools,
99
+ builtin_tools,
100
+ simplified_tool_choice,
101
+ reasoning,
102
+ )
103
84
- if tool_choice == "auto":
85
- return tools, "auto"
104
+ if model not in XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS:
105
+ supported = sorted(SUPPORTED_STRUCTURAL_TAG_MODELS)
106
+ raise ValueError(f"Unknown format type: {model}, supported types: {supported}")
107
87
- if tool_choice == "required":
88
- return tools, "required"
108
+ return get_xgrammar_model_structural_tag(
109
+ model=model,
110
+ tools=dumped_tools,
111
+ tool_choice=dumped_tool_choice,
112
+ reasoning=reasoning,
113
+ )
114
90
- if isinstance(tool_choice, ChatCompletionNamedToolChoiceParam):
91
- tool_name = tool_choice.function.name
92
- filtered_tools = [tool for tool in tools if tool.function.name == tool_name]
93
- if not filtered_tools:
94
- raise ValueError(
95
- f"The tool with name '{tool_name}' is not found in the tools list."
96
- )
97
- return filtered_tools, "forced"
115
99
- raise ValueError(f"Unsupported tool_choice for structural tag: {tool_choice}")
116
+def _model_dump(value: Any) -> Any:
117
+ """Convert vLLM/Pydantic request objects to xgrammar's dict protocol."""
118
119
+ if hasattr(value, "model_dump"):
120
+ return value.model_dump(exclude_none=True)
121
+ return value
122
102
-def _get_function_parameters(function: Any) -> dict[str, Any] | bool:
103
- """Return the JSON schema used for constrained tool arguments."""
123
124
+def _get_function_parameters(function) -> dict[str, Any] | bool:
125
if getattr(function, "strict", None) is False:
126
return True
107
- if function.parameters is None:
108
- return True
109
- return function.parameters
110
-
111
-
112
-_enable_structured_outputs_in_reasoning: bool = False
113
-
127
+ return function.parameters if function.parameters is not None else True
128
+
129
+
130
+def _hermes_tool_tags(tools: list[FunctionToolParam]) -> list[TagFormat]:
131
+ arguments_field_prefix = '", "arguments": '
132
+ formats = [
133
+ # <tool_call>
134
+ # {"name": "t1", "arguments": {"q": "v"}}
135
+ # </tool_call>
136
+ ('<tool_call>\n{"name": "', "}\n</tool_call>"),
137
+ # <tool_call>{"name": "t1", "arguments": {"q": "v"}}</tool_call>
138
+ ('<tool_call>{"name": "', "}</tool_call>"),
139
+ ]
140
+
141
+ return [
142
+ TagFormat(
143
+ begin=begin + tool.function.name + arguments_field_prefix,
144
+ content=JSONSchemaFormat(
145
+ json_schema=_get_function_parameters(tool.function)
146
+ ),
147
+ end=end,
148
+ )
149
+ for tool in tools
150
+ for begin, end in formats
151
+ ]
152
115
-def set_enable_structured_outputs_in_reasoning(enabled: bool) -> None:
116
- """Publish the engine's ``enable_in_reasoning`` flag to tool parsers.
153
118
- Called once during APIServer startup so request-time parsers can read
119
- it without going through the EngineCore-only contextvar.
120
- """
154
+@register_vllm_structural_tag("hermes")
155
+def get_hermes_structural_tag(
156
+ tools: list[FunctionToolParam],
157
+ builtin_tools: list[BuiltinToolParam],
158
+ tool_choice: SimplifiedToolChoice,
159
+ reasoning: bool,
160
+) -> StructuralTag:
161
+ del builtin_tools, reasoning
162
122
- global _enable_structured_outputs_in_reasoning
123
- _enable_structured_outputs_in_reasoning = bool(enabled)
163
+ tool_call_trigger = "<tool_call>"
164
165
+ if tool_choice == "auto":
166
+ tags = _hermes_tool_tags(tools)
167
+ suffix_tag = (
168
+ TriggeredTagsFormat(triggers=[tool_call_trigger], tags=tags)
169
+ if tags
170
+ else AnyTextFormat()
171
+ )
172
+ elif tool_choice == "forced":
173
+ suffix_tag = TagsWithSeparatorFormat(
174
+ tags=_hermes_tool_tags(tools),
175
+ separator="",
176
+ at_least_one=True,
177
+ stop_after_first=True,
178
+ )
179
+ else:
180
+ suffix_tag = TagsWithSeparatorFormat(
181
+ tags=_hermes_tool_tags(tools),
182
+ separator="",
183
+ at_least_one=True,
184
+ )
185
126
-def get_enable_structured_outputs_in_reasoning() -> bool:
127
- """Whether structured outputs are active during the reasoning phase.
186
+ return StructuralTag(format=suffix_tag)
187
129
- When ``True``, the structural tag will cover the reasoning part:
130
- ``<think>...</think>`` prefix (if available); when ``False`` (default), the tag only
131
- constrains the post-reasoning suffix.
132
- """
188
134
- return _enable_structured_outputs_in_reasoning
189
+def _minimax_tool_tags(tools: list[FunctionToolParam]) -> list[TagFormat]:
190
+ return [
191
+ TagFormat(
192
+ begin=f'<invoke name="{tool.function.name}">\n',
193
+ content=JSONSchemaFormat(
194
+ json_schema=_get_function_parameters(tool.function),
195
+ style="minimax_xml",
196
+ ),
197
+ end="</invoke>\n",
198
+ )
199
+ for tool in tools
200
+ ]
201
202
137
-@register_model_structural_tag("deepseek_v4")
138
-def get_deepseek_v4_structural_tag(
139
- tools: list[ChatCompletionToolsParam],
203
+@register_vllm_structural_tag("minimax")
204
+def get_minimax_structural_tag(
205
+ tools: list[FunctionToolParam],
206
+ builtin_tools: list[BuiltinToolParam],
207
tool_choice: SimplifiedToolChoice,
208
reasoning: bool,
209
) -> StructuralTag:
143
- """Build DeepSeek V4 structural tags."""
144
-
145
- invoke_begin_prefix = '<｜DSML｜invoke name="'
146
- invoke_begin_suffix = '">\n'
147
- invoke_end = "</｜DSML｜invoke>\n"
148
- tool_calls_prefix = "\n\n"
149
- function_calls_begin = "<｜DSML｜tool_calls>\n"
150
- function_calls_end = "</｜DSML｜tool_calls>"
151
- function_calls_trigger = "<｜DSML｜tool_calls>"
152
- think_tag_end = "</think>"
153
- think_exclude_tokens = ["<think>", "</think>"]
154
- xml_style = "deepseek_xml"
210
+ del builtin_tools, reasoning
211
156
- if tool_choice == "auto":
157
- tags = []
158
- for tool in tools:
159
- function = tool.function
160
- parameters = _get_function_parameters(function)
161
- tags.append(
162
- TagFormat(
163
- begin=invoke_begin_prefix + function.name + invoke_begin_suffix,
164
- content=JSONSchemaFormat(
165
- json_schema=parameters,
166
- style=xml_style,
167
- ),
168
- end=invoke_end,
169
- )
170
- )
212
+ tool_call_begin = "<minimax:tool_call>\n"
213
+ tool_call_end = "</minimax:tool_call>"
214
+ tool_call_trigger = "<minimax:tool_call>"
215
172
- if tags:
173
- function_calling_tags = TagsWithSeparatorFormat(
174
- tags=tags,
175
- separator="\n",
176
- at_least_one=True,
177
- )
178
- suffix_tag = TriggeredTagsFormat(
179
- triggers=[function_calls_trigger],
216
+ tags = _minimax_tool_tags(tools)
217
+
218
+ if tool_choice == "auto":
219
+ suffix_tag = (
220
+ TriggeredTagsFormat(
221
+ triggers=[tool_call_trigger],
222
tags=[
223
TagFormat(
182
- begin=function_calls_begin,
183
- content=function_calling_tags,
184
- end=function_calls_end,
224
+ begin=tool_call_begin,
225
+ content=TagsWithSeparatorFormat(
226
+ tags=tags,
227
+ separator="",
228
+ at_least_one=True,
229
+ ),
230
+ end=tool_call_end,
231
)
232
],
187
- excludes=think_exclude_tokens,
233
+ excludes=["<think>", "</think>"],
234
)
189
- else:
190
- suffix_tag = AnyTextFormat(excludes=think_exclude_tokens)
191
-
235
+ if tags
236
+ else AnyTextFormat(excludes=["<think>", "</think>"])
237
+ )
238
elif tool_choice == "forced":
193
- if not tools:
194
- raise ValueError("Forced tool choice must resolve to exactly one tool.")
195
- function = tools[0].function
239
suffix_tag = SequenceFormat(
240
elements=[
198
- ConstStringFormat(value=tool_calls_prefix + function_calls_begin),
199
- TagFormat(
200
- begin=invoke_begin_prefix + function.name + invoke_begin_suffix,
201
- content=JSONSchemaFormat(
202
- json_schema=_get_function_parameters(function),
203
- style=xml_style,
204
- ),
205
- end=invoke_end,
241
+ ConstStringFormat(value="\n" + tool_call_begin),
242
+ TagsWithSeparatorFormat(
243
+ tags=tags,
244
+ separator="",
245
+ at_least_one=True,
246
+ stop_after_first=True,
247
),
207
- ConstStringFormat(value=function_calls_end),
248
+ ConstStringFormat(value=tool_call_end),
249
]
250
)
210
-
211
- elif tool_choice == "required":
212
- tags = []
213
- for tool in tools:
214
- function = tool.function
215
- parameters = _get_function_parameters(function)
216
- tags.append(
217
- TagFormat(
218
- begin=invoke_begin_prefix + function.name + invoke_begin_suffix,
219
- content=JSONSchemaFormat(
220
- json_schema=parameters,
221
- style=xml_style,
222
- ),
223
- end=invoke_end,
224
- )
225
- )
226
- assert len(tags) > 0
251
+ else:
252
suffix_tag = SequenceFormat(
253
elements=[
229
- ConstStringFormat(value=tool_calls_prefix + function_calls_begin),
254
+ ConstStringFormat(value="\n" + tool_call_begin),
255
TagsWithSeparatorFormat(
256
tags=tags,
232
- separator="\n",
257
+ separator="",
258
at_least_one=True,
259
),
235
- ConstStringFormat(value=function_calls_end),
236
- ]
237
- )
238
-
239
- if not reasoning:
240
- return StructuralTag(format=suffix_tag)
241
-
242
- prefix_tag = TagFormat(begin="", content=AnyTextFormat(), end=think_tag_end)
243
- return StructuralTag(format=SequenceFormat(elements=[prefix_tag, suffix_tag]))
244
-
245
-
246
-@register_model_structural_tag("qwen_3_5")
247
-def get_qwen_3_5_structural_tag(
248
- tools: list[ChatCompletionToolsParam],
249
- tool_choice: SimplifiedToolChoice,
250
- reasoning: bool,
251
-) -> StructuralTag:
252
- """Build Qwen XML structural tags.
253
-
254
- This format is used for Qwen3-Coder/Qwen3.5/Qwen3.6 and is compatible with
255
- Qwen variants that use the same XML tool-call format.
256
- """
257
- tool_call_begin_prefix = "<tool_call>\n<function="
258
- tool_call_begin_suffix = ">\n"
259
- tool_call_end = "\n</function>\n</tool_call>"
260
- tool_call_trigger = "<tool_call>\n<function="
261
- think_tag_end = "</think>"
262
- think_suffix = "\n\n"
263
- think_exclude_tokens = ["<think>", "</think>"]
264
-
265
- if tool_choice == "auto":
266
- tags = []
267
- for tool in tools:
268
- function = tool.function
269
- parameters = _get_function_parameters(function)
270
- tags.append(
271
- TagFormat(
272
- begin=f"{tool_call_begin_prefix}{function.name}{tool_call_begin_suffix}",
273
- content=JSONSchemaFormat(json_schema=parameters, style="qwen_xml"),
274
- end=tool_call_end,
275
- )
276
- )
277
-
278
- if tags:
279
- suffix_tag = TriggeredTagsFormat(
280
- triggers=[tool_call_trigger],
281
- tags=tags,
282
- excludes=think_exclude_tokens,
283
- )
284
- else:
285
- suffix_tag = AnyTextFormat(excludes=think_exclude_tokens)
286
-
287
- elif tool_choice == "forced":
288
- if not tools:
289
- raise ValueError("Forced tool choice must resolve to exactly one tool.")
290
- function = tools[0].function
291
- suffix_tag = TagFormat(
292
- begin=f"{tool_call_begin_prefix}{function.name}{tool_call_begin_suffix}",
293
- content=JSONSchemaFormat(
294
- json_schema=_get_function_parameters(function),
295
- style="qwen_xml",
296
- ),
297
- end=tool_call_end,
298
- )
299
-
300
- elif tool_choice == "required":
301
- tags = []
302
- for tool in tools:
303
- function = tool.function
304
- parameters = _get_function_parameters(function)
305
- tags.append(
306
- TagFormat(
307
- begin=f"{tool_call_begin_prefix}{function.name}{tool_call_begin_suffix}",
308
- content=JSONSchemaFormat(json_schema=parameters, style="qwen_xml"),
309
- end=tool_call_end,
310
- )
311
- )
312
- assert len(tags) > 0
313
- suffix_tag = TagsWithSeparatorFormat(
314
- tags=tags,
315
- separator="",
316
- at_least_one=True,
317
- )
318
-
319
- if not reasoning:
320
- result = StructuralTag(format=suffix_tag)
321
- else:
322
- prefix_tag = SequenceFormat(
323
- elements=[
324
- TagFormat(begin="", content=AnyTextFormat(), end=think_tag_end),
325
- ConstStringFormat(value=think_suffix),
260
+ ConstStringFormat(value=tool_call_end),
261
]
262
)
328
- result = StructuralTag(format=SequenceFormat(elements=[prefix_tag, suffix_tag]))
263
330
- return result
264
+ return StructuralTag(format=suffix_tag)