@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 431cea3eea
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 5, 03:30 UTC

[Bugfix] Fix tool_calls Iterable consumed when debug logging is enabled (#34844)

Signed-off-by: Wojciech Wais <wojciech.wais@gmail.com> Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Xinyu Chen <xinyu1.chen@intel.com> Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com> Signed-off-by: Rishi Puri <riship@nvidia.com> Signed-off-by: Jaebok Lee <jaebok9541@naver.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: yuwei <yuwei@dev.local> Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com> Signed-off-by: Ibrahim Arshad <38925737+ibrahim1023@users.noreply.github.com> Signed-off-by: Li <chuali@amd.com> Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com> Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> Signed-off-by: Kunshang Ji <jikunshang95@gmail.com> Signed-off-by: R <Ganesh.R@amd.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: lkm2835 <lkm2835@gmail.com> Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com> Signed-off-by: vnadathur <glvikramn@gmail.com> Signed-off-by: WorldExplored <srreyansh.sethi@gmail.com> Signed-off-by: Srreyansh Sethi <107075589+WorldExplored@users.noreply.github.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Elham Harirpoush <elham.harirpoush@arm.com> Signed-off-by: Yan Ma <yan.ma@intel.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: jackcfwang <jackcfwang@tencent.com> Signed-off-by: Chendi Xue <chendi.xue@intel.com> Signed-off-by: Injae Ryou <injaeryou@gmail.com> Signed-off-by: Richard Zou <zou3519@gmail.com> Signed-off-by: milesial <milesial@users.noreply.github.com> Signed-off-by: Elvir Crncevic <elvircrn@gmail.com> Signed-off-by: whx-sjtu <2952154980@qq.com> Signed-off-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Signed-off-by: PatchouliTaisa <patchychen@tencent.com> Signed-off-by: jatseng-ai <jatseng@amd.com> Signed-off-by: jatseng-ai <janet.tseng@amd.com> Signed-off-by: Matthias Gehre <matthias.gehre@amd.com> Signed-off-by: xaguilar-amd <xaguilar@amd.com> Signed-off-by: rdondeti <ravitez.dondeti@gmail.com> Signed-off-by: Ravitez Dondeti <ravitez.dondeti@gmail.com> Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Peter Nguyen <petern0408@gmail.com> Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com> Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com> Signed-off-by: Jesus Federico <jefp@amazon.com> Signed-off-by: manu <fortin.emmanuel@gmail.com> Signed-off-by: ZhanqiuHu <zhu@redhat.com> Signed-off-by: Yifan Zong <yzong@redhat.com> Signed-off-by: Rahul-Tuli <rtuli@redhat.com> Signed-off-by: Fynn Schmitt-Ulms <fschmitt@redhat.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Michael Goin <mgoin64@gmail.com> Signed-off-by: Benjamin Chislett <bchislett@nvidia.com> Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn> Signed-off-by: leeyongjun <jqueen.astro@gmail.com> Signed-off-by: Ziying Tao <tzzying@outlook.com> Signed-off-by: jiang1.li <jiang1.li@intel.com> Signed-off-by: Vibhav Agarwal <vibhavagarwal5@gmail.com> Signed-off-by: ShubyM <shubymishra20@gmail.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Signed-off-by: Itay Etelis <itay.etelis@ibm.com> Signed-off-by: EdalatiAli <aliedalati@cohere.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com> Signed-off-by: r266-tech <r266.tech@gmail.com> Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com> Signed-off-by: Mark McLoughlin <markmc@redhat.com> Signed-off-by: Animesh Jain <anijain@umich.edu> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Signed-off-by: zhxchen17 <zhxchen17@fb.com> Signed-off-by: EricccYang <yangyang4991@gmail.com> Signed-off-by: Kaicheng Yang <53411596+EricccYang@users.noreply.github.com> Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: sihao.li <sihao.li@intel.com> Signed-off-by: sfeng33 <4florafeng@gmail.com> Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com> Signed-off-by: Tihomir Elek <tiho.elek@gmail.com> Signed-off-by: yiliu30 <yi4.liu@intel.com> Signed-off-by: yewentao256 <zhyanwentao@126.com> Signed-off-by: Santino Ramos <santinor@inferact.ai> Signed-off-by: haosdent <haosdent@gmail.com> Signed-off-by: JartX <sagformas@epdcenter.es> Signed-off-by: George-ao <yuyiao772@gmail.com> Signed-off-by: Yuyi Ao <yuyiao772@gmail.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: Mukesh Baphna <mukesh@hippocraticai.com> Signed-off-by: Pedram Razavi <pedram.razavi@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com> Co-authored-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com> Co-authored-by: Rishi Puri <riship@nvidia.com> Co-authored-by: zzaebok <44357534+zzaebok@users.noreply.github.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Yuwei An <ayw.sirius19@gmail.com> Co-authored-by: yuwei <yuwei@dev.local> Co-authored-by: Artem Perevedentsev <aperevedents@nvidia.com> Co-authored-by: Ibrahim Arshad <38925737+ibrahim1023@users.noreply.github.com> Co-authored-by: Chuan (Richard) Li <chuali@amd.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com> Co-authored-by: Ganesh R <ganesh.r@amd.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Co-authored-by: Kyungmin Lee <30465912+lkm2835@users.noreply.github.com> Co-authored-by: Ronen Schaffer <ronen.schaffer@ibm.com> Co-authored-by: Srreyansh Sethi <107075589+WorldExplored@users.noreply.github.com> Co-authored-by: vnadathur <glvikramn@gmail.com> Co-authored-by: vnadathur <236933696+vnadathur@users.noreply.github.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Elham <elham.harirpoush@arm.com> Co-authored-by: Yan Ma <yan.ma@intel.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Chaofan Wang <jackcfwang@tencent.com> Co-authored-by: Chendi.Xue <chendi.xue@intel.com> Co-authored-by: Injae Ryou <injaeryou@gmail.com> Co-authored-by: Richard Zou <zou3519@users.noreply.github.com> Co-authored-by: milesial <milesial@users.noreply.github.com> Co-authored-by: Elvir Crnčević <elvircrn@gmail.com> Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com> Co-authored-by: Hexiang Wang <56632993+whx-sjtu@users.noreply.github.com> Co-authored-by: Lalithnarayan C <Lalithnarayan.C@amd.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Co-authored-by: PatchyTIS <58251192+PatchouliTIS@users.noreply.github.com> Co-authored-by: PatchouliTaisa <patchychen@tencent.com> Co-authored-by: jatseng-ai <janet.tseng@amd.com> Co-authored-by: Matthias Gehre <matthias.gehre@amd.com> Co-authored-by: xaguilar-amd <xavier.aguilarfruto@amd.com> Co-authored-by: Ravitez Dondeti <dondetir@users.noreply.github.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by: Peter Nguyen <petern0408@gmail.com> Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io> Co-authored-by: zhrrr <43847754+izhuhaoran@users.noreply.github.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: TJian <tunjian.tan@embeddedllm.com> Co-authored-by: Jesus Federico <14651+jefp@users.noreply.github.com> Co-authored-by: Manu <efortin@users.noreply.github.com> Co-authored-by: zhanqiuhu <49648934+ZhanqiuHu@users.noreply.github.com> Co-authored-by: yzong-rh <yzong@redhat.com> Co-authored-by: Fynn Schmitt-Ulms <fschmitt@redhat.com> Co-authored-by: Rahul-Tuli <rtuli@redhat.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Benjamin Chislett <bchislett@nvidia.com> Co-authored-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn> Co-authored-by: Lee Yongjun <35302114+elwhyjay@users.noreply.github.com> Co-authored-by: z1ying <55220715+z1ying@users.noreply.github.com> Co-authored-by: Li, Jiang <jiang1.li@intel.com> Co-authored-by: Vibhav Agarwal <vibhavagarwal5@gmail.com> Co-authored-by: vibhav-agarwal <vibhav.agarwal@glance.com> Co-authored-by: ShubyM <shubymishra20@gmail.com> Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com> Co-authored-by: Itay Etelis <itay.etelis@ibm.com> Co-authored-by: EdalatiAli <aliedalati@cohere.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: r266-tech <r2668940489@gmail.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Martin Hickey <martin.hickey@ie.ibm.com> Co-authored-by: Or Ozeri <or@ozery.com> Co-authored-by: Mark McLoughlin <markmc@redhat.com> Co-authored-by: Le Yang <562593859@qq.com> Co-authored-by: Animesh Jain <anijain@umich.edu> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Zhengxu Chen <zhxchen17@fb.com> Co-authored-by: Kaicheng Yang <53411596+EricccYang@users.noreply.github.com> Co-authored-by: maobaolong <baoloongmao@tencent.com> Co-authored-by: sihao_li <165983188+1643661061leo@users.noreply.github.com> Co-authored-by: Flora Feng <4florafeng@gmail.com> Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> Co-authored-by: zofia <110436990+zufangzhu@users.noreply.github.com> Co-authored-by: Tihomir Elek <tiho.elek@gmail.com> Co-authored-by: Yi Liu <yi4.liu@intel.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: Santino Ramos <51103228+santiramos27@users.noreply.github.com> Co-authored-by: haosdent <haosdent@gmail.com> Co-authored-by: JartX <sagformas@epdcenter.es> Co-authored-by: Yuyi Ao <yuyiao772@gmail.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com> Co-authored-by: mukesh-hai <mukesh@hippocraticai.com> Co-authored-by: Pedram Razavi <pedram@sierra.ai>

Wojciech Wais committed Apr 15, 2026 at 10:32 UTC 431cea3eea16012aa60dc2cfccedbaeadd62b729
3 files changed +193 -2
tests/entrypoints/openai/test_tool_calls_serialization.py new
+150
@@ -0,0 +1,150 @@
1 +# SPDX-License-Identifier: Apache-2.0
2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3 +"""Unit tests for tool_calls Iterable → list materialisation.
4 +
5 +Regression tests for https://github.com/vllm-project/vllm/issues/34792.
6 +
7 +Setting VLLM_LOGGING_LEVEL=debug caused tool calling to break for Mistral
8 +models because:
9 + 1. The OpenAI Python SDK types tool_calls as Iterable[...] in
10 + ChatCompletionAssistantMessageParam.
11 + 2. Pydantic v2, when validating from Python objects (not from raw JSON),
12 + wraps Iterable fields in a one-shot lazy iterator.
13 + 3. Debug logging called model_dump_json() which consumed that iterator.
14 + 4. The Mistral tokenizer then saw empty tool_calls and raised
15 + "ValueError: Unexpected tool call id ...".
16 +"""
17 +
18 +import pytest
19 +
20 +from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
21 +
22 +
23 +def _make_tool_call(tc_id: str, name: str, args: str) -> dict:
24 + return {
25 + "id": tc_id,
26 + "type": "function",
27 + "function": {"name": name, "arguments": args},
28 + }
29 +
30 +
31 +def _make_request(messages: list) -> ChatCompletionRequest:
32 + return ChatCompletionRequest(
33 + model="test-model",
34 + messages=messages,
35 + )
36 +
37 +
38 +def test_tool_calls_list_preserved_after_model_dump():
39 + """tool_calls in assistant messages must be readable after model_dump_json.
40 +
41 + When the request is built from Python dicts (as in the Anthropic → OpenAI
42 + conversion path), Pydantic v2 previously wrapped the Iterable tool_calls
43 + in a one-shot iterator. model_dump_json() consumed it, leaving subsequent
44 + readers (e.g. the Mistral tokenizer) with an empty sequence.
45 + """
46 + tool_call = _make_tool_call("call_abc123", "get_weather", '{"city": "Paris"}')
47 + messages = [
48 + {"role": "user", "content": "What is the weather in Paris?"},
49 + {"role": "assistant", "content": None, "tool_calls": [tool_call]},
50 + {
51 + "role": "tool",
52 + "tool_call_id": "call_abc123",
53 + "content": '{"temperature": 20}',
54 + },
55 + ]
56 +
57 + req = _make_request(messages)
58 +
59 + # Simulate debug logging: serialize the model (this was the trigger)
60 + _ = req.model_dump_json()
61 +
62 + # The assistant message must still have accessible tool_calls afterwards
63 + assistant_msg = req.messages[1]
64 + assert isinstance(assistant_msg, dict)
65 + tool_calls = assistant_msg.get("tool_calls")
66 + assert tool_calls is not None, "tool_calls must not be None after model_dump_json"
67 + assert isinstance(tool_calls, list), "tool_calls must be a list"
68 + assert len(tool_calls) > 0, "tool_calls must not be empty after model_dump_json"
69 +
70 +
71 +def test_tool_calls_from_generator_are_materialised():
72 + """tool_calls passed as a generator must be converted to list on validation."""
73 + tool_call = _make_tool_call("call_gen1", "search", '{"query": "vllm"}')
74 +
75 + def tool_calls_gen():
76 + yield tool_call
77 +
78 + messages = [
79 + {"role": "user", "content": "Search for vllm"},
80 + {
81 + "role": "assistant",
82 + "content": None,
83 + "tool_calls": tool_calls_gen(), # one-shot generator
84 + },
85 + ]
86 +
87 + req = _make_request(messages)
88 + assistant_msg = req.messages[1]
89 + assert isinstance(assistant_msg, dict)
90 +
91 + # Iterate twice — must not raise or return empty on second pass
92 + tool_calls_first = list(assistant_msg.get("tool_calls", []))
93 + tool_calls_second = list(assistant_msg.get("tool_calls", []))
94 +
95 + assert len(tool_calls_first) == 1, "First read must return the tool call"
96 + assert len(tool_calls_second) == 1, "Second read must also return the tool call"
97 +
98 +
99 +def test_tool_calls_list_passthrough():
100 + """tool_calls already provided as a list must remain a list."""
101 + tool_call = _make_tool_call("call_list1", "calculate", '{"expr": "2+2"}')
102 + messages = [
103 + {"role": "user", "content": "Calculate 2+2"},
104 + {"role": "assistant", "content": None, "tool_calls": [tool_call]},
105 + ]
106 +
107 + req = _make_request(messages)
108 + assistant_msg = req.messages[1]
109 + assert isinstance(assistant_msg, dict)
110 + assert isinstance(assistant_msg.get("tool_calls"), list)
111 +
112 +
113 +def test_messages_without_tool_calls_unaffected():
114 + """Messages without tool_calls must be handled correctly."""
115 + messages = [
116 + {"role": "system", "content": "You are a helpful assistant."},
117 + {"role": "user", "content": "Hello!"},
118 + {"role": "assistant", "content": "Hi there!"},
119 + ]
120 +
121 + req = _make_request(messages)
122 + # None of the messages should have tool_calls injected
123 + for msg in req.messages:
124 + assert isinstance(msg, dict)
125 + assert msg.get("tool_calls") is None or msg.get("tool_calls") == []
126 +
127 +
128 +@pytest.mark.parametrize("num_tool_calls", [1, 3])
129 +def test_multiple_tool_calls_materialised(num_tool_calls: int):
130 + """Multiple tool calls in a single message are all preserved."""
131 + tool_calls = [
132 + _make_tool_call(f"call_{i}", f"func_{i}", f'{{"arg": {i}}}')
133 + for i in range(num_tool_calls)
134 + ]
135 + messages = [
136 + {"role": "user", "content": "Do things"},
137 + {"role": "assistant", "content": None, "tool_calls": iter(tool_calls)},
138 + ]
139 +
140 + req = _make_request(messages)
141 + assistant_msg = req.messages[1]
142 + assert isinstance(assistant_msg, dict)
143 +
144 + result_tool_calls = assistant_msg.get("tool_calls")
145 + assert isinstance(result_tool_calls, list)
146 + assert len(result_tool_calls) == num_tool_calls
147 +
148 + # Verify after model_dump_json too
149 + _ = req.model_dump_json()
150 + assert len(assistant_msg.get("tool_calls", [])) == num_tool_calls
vllm/entrypoints/chat_utils.py
+2 -2
@@ -290,7 +290,7 @@ class CustomChatCompletionMessageParam(TypedDict, total=False):
290 tool_call_id: str | None
291 """Tool call that this message is responding to."""
292
293 - tool_calls: Iterable[ChatCompletionMessageToolCallParam] | None
293 + tool_calls: list[ChatCompletionMessageToolCallParam] | None
294 """The tool calls generated by the model, such as function calls."""
295
296 reasoning: str | None
@@ -321,7 +321,7 @@ class ConversationMessage(TypedDict, total=False):
321 name: str | None
322 """The name of the function to call"""
323
324 - tool_calls: Iterable[ChatCompletionMessageToolCallParam] | None
324 + tool_calls: list[ChatCompletionMessageToolCallParam] | None
325 """The tool calls generated by the model, such as function calls."""
326
327 reasoning: str | None
vllm/entrypoints/openai/chat_completion/protocol.py
+41
@@ -357,6 +357,47 @@ class ChatCompletionRequest(OpenAIBaseModel):
357
358 # --8<-- [end:chat-completion-extra-params]
359
360 + @model_validator(mode="before")
361 + @classmethod
362 + def _materialize_tool_calls_before(cls, data: Any) -> Any:
363 + """Eagerly convert tool_calls generators/iterators to lists.
364 +
365 + Must run before Pydantic field validation so that one-shot
366 + generators are not consumed during union type matching of
367 + ChatCompletionAssistantMessageParam (which types tool_calls
368 + as Iterable[...]).
369 + """
370 + if not isinstance(data, dict):
371 + return data
372 + messages = data.get("messages")
373 + if not isinstance(messages, list):
374 + return data
375 + for msg in messages:
376 + if not isinstance(msg, dict):
377 + continue
378 + tool_calls = msg.get("tool_calls")
379 + if tool_calls is not None and not isinstance(tool_calls, list):
380 + msg["tool_calls"] = list(tool_calls)
381 + return data
382 +
383 + @model_validator(mode="after")
384 + def _materialize_tool_calls_after(self) -> "ChatCompletionRequest":
385 + """Convert Pydantic ValidatorIterator wrappers back to lists.
386 +
387 + Even after the "before" validator converts iterables to lists,
388 + Pydantic re-wraps them in a ValidatorIterator when validating
389 + against ChatCompletionAssistantMessageParam's Iterable[...] type.
390 + This "after" pass materialises those wrappers so downstream code
391 + (tokenizers, model_dump_json) always sees plain lists.
392 + """
393 + for msg in self.messages:
394 + if not isinstance(msg, dict):
395 + continue
396 + tool_calls = msg.get("tool_calls")
397 + if tool_calls is not None and not isinstance(tool_calls, list):
398 + msg["tool_calls"] = list(tool_calls)
399 + return self
400 +
401 def build_chat_params(
402 self,
403 default_template: str | None,