@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 530 lines 22.1 KB
Rendered Raw
1 # Reasoning Outputs
2
3 vLLM offers support for reasoning models like [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1), which are designed to generate outputs containing both reasoning steps and final conclusions.
4
5 Reasoning models return an additional `reasoning` field in their outputs, which contains the reasoning steps that led to the final conclusion. This field is not present in the outputs of other models.
6
7 !!! warning
8 `reasoning` used to be called `reasoning_content`. To migrate, directly replace `reasoning_content` with `reasoning`.
9 It is important that you also update your client code. Otherwise, your client code could silently read an empty `reasoning_content`, even when `reasoning` is populated.
10
11 ## Supported Models
12
13 vLLM currently supports the following reasoning models:
14
15 | Model Series | Parser Name | Structured Output Support | Tool Calling |
16 | ------------ | ----------- | ---------------- | ----------- |
17 | [Cohere Command A Reasoning](https://huggingface.co/CohereLabs/command-a-reasoning-08-2025) | `cohere_command3` | `json`, `regex` | ✅ |
18 | [Cohere Command A Plus](https://huggingface.co/CohereLabs/command-a-plus-05-2026-bf16) | `cohere_command4` | `json`, `regex` | ✅ |
19 | [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d) | `deepseek_r1` | `json`, `regex` | ❌ |
20 | [Gemma 4 series](https://huggingface.co/google/gemma-4-26B-A4B-it) | `gemma4` | `json`, `regex` | ✅ |
21 | [DeepSeek-V3.1](https://huggingface.co/collections/deepseek-ai/deepseek-v31-68a491bed32bd77e7fca048f) | `deepseek_v3` | `json`, `regex` | ❌ |
22 | [ERNIE-4.5-VL series](https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT) | `ernie45` | `json`, `regex` | ❌ |
23 | [ERNIE-4.5-21B-A3B-Thinking](https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-Thinking) | `ernie45` | `json`, `regex` | ✅ |
24 | [GLM-4.5 series](https://huggingface.co/collections/zai-org/glm-45-687c621d34bda8c9e4bf503b) | `glm45` | `json`, `regex` | ✅ |
25 | [Holo2 series](https://huggingface.co/collections/Hcompany/holo2) | `holo2` | `json`, `regex` | ✅ |
26 | [Hunyuan A13B series](https://huggingface.co/collections/tencent/hunyuan-a13b-685ec38e5b46321e3ea7c4be) | `hunyuan_a13b` | `json`, `regex` | ✅ |
27 | [IBM Granite 3.2 language models](https://huggingface.co/collections/ibm-granite/granite-32-language-models-67b3bc8c13508f6d064cff9a) | `granite` | ❌ | ❌ |
28 | [IBM Granite 4.2 language models](https://huggingface.co/ibm-granite/granite-4.2-30b) | `granite_thinking_parser` | `json`, `regex` | ✅ |
29 | [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) | `minimax_m2_append_think` | `json`, `regex` | ✅ |
30 | [Qwen3 series](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `qwen3` | `json`, `regex` | ✅ |
31 | [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B) | `deepseek_r1` | `json`, `regex` | ✅ |
32 | [Step-3.5 / Step-3.7 Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) | `step3p5` | `json`, `regex` | ✅ |
33
34 !!! note
35 IBM Granite 3.2 and DeepSeek-V3.1 reasoning is disabled by default; to enable it, you must also pass `thinking=True` in your `chat_template_kwargs`.
36 IBM Granite 4.2 reasoning is enabled by default. To disable it, you must pass `enable_thinking=False` in your `chat_template_kwargs`.
37 The reasoning feature for the Qwen3 series is enabled by default. To disable it, you must pass `enable_thinking=False` in your `chat_template_kwargs`.
38 Gemma 4 reasoning is disabled by default; to enable it, pass `enable_thinking=True` in your `chat_template_kwargs` or set `reasoning_effort` (which enables it automatically).
39 DeepSeek-V3.1 tool calling is supported in non-thinking mode.
40 Holo2 reasoning is enabled by default. To disable it, you must also pass `thinking=False` in your `chat_template_kwargs`.
41
42 ## Quickstart
43
44 To use reasoning models, you need to specify the `--reasoning-parser` flags when making a request to the chat completion endpoint. The `--reasoning-parser` flag specifies the reasoning parser to use for extracting reasoning content from the model output.
45
46 ```bash
47 vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
48 --reasoning-parser deepseek_r1
49 ```
50
51 Next, make a request to the model that should return the reasoning content in the response.
52
53 ??? code
54
55 ```python
56 from openai import OpenAI
57
58 # Modify OpenAI's API key and API base to use vLLM's API server.
59 openai_api_key = "EMPTY"
60 openai_api_base = "http://localhost:8000/v1"
61
62 client = OpenAI(
63 api_key=openai_api_key,
64 base_url=openai_api_base,
65 )
66
67 models = client.models.list()
68 model = models.data[0].id
69
70 # Round 1
71 messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
72 # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
73 # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
74 # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
75 response = client.chat.completions.create(model=model, messages=messages)
76
77 reasoning = response.choices[0].message.reasoning
78 content = response.choices[0].message.content
79
80 print("reasoning:", reasoning)
81 print("content:", content)
82 ```
83
84 The `reasoning` field contains the reasoning steps that led to the final conclusion, while the `content` field contains the final conclusion.
85
86 ## Streaming chat completions
87
88 Streaming chat completions are also supported for reasoning models. The `reasoning` field is available in the `delta` field in [chat completion response chunks](https://platform.openai.com/docs/api-reference/chat/streaming).
89
90 ??? console "Json"
91
92 ```json
93 {
94 "id": "chatcmpl-123",
95 "object": "chat.completion.chunk",
96 "created": 1694268190,
97 "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
98 "system_fingerprint": "fp_44709d6fcb",
99 "choices": [
100 {
101 "index": 0,
102 "delta": {
103 "role": "assistant",
104 "reasoning": "is",
105 },
106 "logprobs": null,
107 "finish_reason": null
108 }
109 ]
110 }
111 ```
112
113 OpenAI Python client library does not officially support `reasoning` attribute for streaming output. But the client supports extra attributes in the response. You can use `getattr` to check if the `reasoning` attribute is present in the response. For example:
114
115 ??? code
116
117 ```python
118 from openai import OpenAI
119
120 # Modify OpenAI's API key and API base to use vLLM's API server.
121 openai_api_key = "EMPTY"
122 openai_api_base = "http://localhost:8000/v1"
123
124 client = OpenAI(
125 api_key=openai_api_key,
126 base_url=openai_api_base,
127 )
128
129 models = client.models.list()
130 model = models.data[0].id
131
132 messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
133 # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
134 # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
135 # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
136 stream = client.chat.completions.create(
137 model=model,
138 messages=messages,
139 stream=True,
140 )
141
142 print("client: Start streaming chat completions...")
143 printed_reasoning = False
144 printed_content = False
145
146 for chunk in stream:
147 # Safely extract reasoning and content from delta,
148 # defaulting to None if attributes don't exist or are empty strings
149 reasoning = (
150 getattr(chunk.choices[0].delta, "reasoning", None) or None
151 )
152 content = getattr(chunk.choices[0].delta, "content", None) or None
153
154 if reasoning is not None:
155 if not printed_reasoning:
156 printed_reasoning = True
157 print("reasoning:", end="", flush=True)
158 print(reasoning, end="", flush=True)
159 elif content is not None:
160 if not printed_content:
161 printed_content = True
162 print("\ncontent:", end="", flush=True)
163 # Extract and print the content
164 print(content, end="", flush=True)
165 ```
166
167 Remember to check whether the `reasoning` exists in the response before accessing it. You could check out the [example](https://github.com/vllm-project/vllm/blob/main/examples/reasoning/openai_chat_completion_with_reasoning_streaming.py).
168
169 ## Tool Calling
170
171 The reasoning content is also available when both tool calling and the reasoning parser are enabled. Additionally, tool calling only parses functions from the `content` field, not from the `reasoning`.
172
173 ??? code
174
175 ```python
176 from openai import OpenAI
177
178 client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
179
180 tools = [
181 {
182 "type": "function",
183 "function": {
184 "name": "get_weather",
185 "description": "Get the current weather in a given location",
186 "parameters": {
187 "type": "object",
188 "properties": {
189 "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
190 "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
191 },
192 "required": ["location", "unit"],
193 }
194 },
195 }
196 ]
197
198 response = client.chat.completions.create(
199 model=client.models.list().data[0].id,
200 messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
201 tools=tools,
202 tool_choice="auto",
203 )
204
205 print(response)
206 tool_call = response.choices[0].message.tool_calls[0].function
207
208 print(f"reasoning: {response.choices[0].message.reasoning}")
209 print(f"Function called: {tool_call.name}")
210 print(f"Arguments: {tool_call.arguments}")
211 ```
212
213 For more examples, please refer to [examples/reasoning/openai_chat_completion_tool_calls_with_reasoning.py](../../examples/reasoning/openai_chat_completion_tool_calls_with_reasoning.py).
214
215 ## Server-Level Default Chat Template Kwargs
216
217 You can set default `chat_template_kwargs` at the server level using the `--default-chat-template-kwargs` CLI argument. This is useful for configuring reasoning behavior across all requests without requiring clients to specify it in each request.
218
219 ### Disabling Thinking Mode by Default
220
221 For models like Qwen3 where thinking is enabled by default, you can disable it server-wide:
222
223 ```bash
224 vllm serve Qwen/Qwen3-8B \
225 --reasoning-parser qwen3 \
226 --default-chat-template-kwargs '{"enable_thinking": false}'
227 ```
228
229 ### Enabling Thinking Mode by Default
230
231 For models like IBM Granite 3.2 or DeepSeek-V3.1 where thinking is disabled by default, you can enable it server-wide:
232
233 ```bash
234 vllm serve ibm-granite/granite-3.2-2b-instruct \
235 --reasoning-parser granite \
236 --default-chat-template-kwargs '{"thinking": true}'
237 ```
238
239 ### Request-Level Override
240
241 Request-level `chat_template_kwargs` always take priority over server defaults. For example, if the server is started with `enable_thinking=false`, a client can still enable it for a specific request:
242
243 ```python
244 response = client.chat.completions.create(
245 model=model,
246 messages=messages,
247 extra_body={"chat_template_kwargs": {"enable_thinking": True}} # Overrides server default
248 )
249 ```
250
251 ## Thinking Budget Control
252
253 Some models, such as [Qwen3](https://qwen.readthedocs.io/en/latest/getting_started/quickstart.html#thinking-budget), [DeepSeek](https://www.alibabacloud.com/help/en/model-studio/deep-thinking), and [Nemotron3](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16), support a thinking budget that limits the maximum number of tokens used for reasoning.
254
255 Token counting starts from `reasoning_start_str`. Once the reasoning token count reaches the configured `thinking_token_budget`, vLLM forces the model to produce `reasoning_end_str`, effectively terminating the reasoning block.
256
257 To use this feature:
258
259 - `--reasoning-parser` enables reasoning extraction.
260 - `--reasoning-config` defines the reasoning boundary tokens (e.g., `reasoning_start_str`, `reasoning_end_str`). If not set, vLLM will attempt to automatically initialize these tokens from the reasoning parser.
261 - `thinking_token_budget` (a sampling parameter) sets the per-request reasoning token limit.
262
263 If `thinking_token_budget` is not specified, no explicit reasoning limit is applied beyond normal generation constraints such as `max_tokens`.
264
265 `--reasoning-config` accepts a JSON object corresponding to
266 [ReasoningConfig][vllm.config.ReasoningConfig] with the following fields:
267
268 | Field | Type | Description |
269 |-----------------------|----------------|--------------------------------------------------|
270 | `reasoning_start_str` | `str \| null` | String that marks the start of reasoning content |
271 | `reasoning_end_str` | `str \| null` | String that marks the end of reasoning content |
272
273 !!! note
274 `reasoning_end_str` can include a transition phrase before the reasoning end token. For example, setting `reasoning_end_str` to `"I have to give the solution based on the reasoning directly now.</think>"` instructs the model to emit that phrase when the budget is exhausted, making the reasoning termination more natural.
275
276 ### Online Serving
277
278 ```bash
279 vllm serve Qwen/Qwen3-0.6B \
280 --reasoning-parser qwen3 \
281 --reasoning-config '{"reasoning_start_str": "<think>", "reasoning_end_str": "I have to give the solution based on the reasoning directly now.</think>"}'
282 ```
283
284 Then make a request with `thinking_token_budget` to limit the reasoning tokens:
285
286 ```bash
287 curl http://localhost:8000/v1/chat/completions \
288 -H "Content-Type: application/json" \
289 -d '{
290 "model": "Qwen/Qwen3-0.6B",
291 "messages": [
292 { "role": "user", "content": "9.11 and 9.8, which is greater?" }
293 ],
294 "thinking_token_budget": 10
295 }'
296 ```
297
298 ### Offline Inference
299
300 ```python
301 from vllm import LLM, SamplingParams
302 from vllm.config import ReasoningConfig
303
304 llm = LLM(
305 model="Qwen/Qwen3-0.6B",
306 reasoning_config=ReasoningConfig(
307 reasoning_start_str="<think>",
308 reasoning_end_str="I have to give the solution based on the thinking directly now.</think>",
309 ),
310 )
311
312 sampling_params = SamplingParams(thinking_token_budget=10)
313
314 messages = [
315 {"role": "user", "content": "9.11 and 9.8, which is greater?"},
316 ]
317
318 outputs = llm.chat(messages, sampling_params=sampling_params)
319
320 for output in outputs:
321 print("text:", output.outputs[0].text)
322 ```
323
324 ## Automatic `enable_thinking` Activation
325
326 Some models (such as Gemma 4, DeepSeek-V4-Pro and IBM Granite 3.2) require `enable_thinking: true` in their chat template kwargs to activate thinking mode — without it, reasoning tokens are never generated regardless of other settings.
327
328 When you set `reasoning_effort` in a Chat Completions request (or `reasoning.effort` in a Responses API request), vLLM automatically injects `enable_thinking` into the chat template kwargs:
329
330 - `reasoning_effort` = `"low"`, `"medium"`, or `"high"` → `enable_thinking = true`
331 - `reasoning_effort` = `"none"` → `enable_thinking = false`
332 - `reasoning_effort` not set → `enable_thinking` is not injected (preserves existing behavior)
333
334 This means you no longer need to manually pass `chat_template_kwargs: {"enable_thinking": true}` when using `reasoning_effort` — it is handled automatically.
335
336 !!! note
337 If you explicitly set `enable_thinking` in `chat_template_kwargs`, your value takes priority over the automatic injection. This allows you to override the behavior if needed.
338
339 For models whose templates don't declare `enable_thinking` (e.g., DeepSeek R1), the injected kwarg is harmlessly filtered out by `resolve_chat_template_kwargs`.
340
341 ### Example
342
343 ```python
344 from openai import OpenAI
345
346 client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
347
348 # reasoning_effort automatically enables thinking for models that need it
349 response = client.chat.completions.create(
350 model="google/gemma-4-26B-A4B-it",
351 messages=[{"role": "user", "content": "What is 15 * 37?"}],
352 reasoning_effort="high", # Automatically sets enable_thinking=true
353 )
354
355 print(response.choices[0].message.reasoning)
356 print(response.choices[0].message.content)
357 ```
358
359 ## Suppressing Reasoning Output
360
361 You can suppress reasoning content from API responses using the `include_reasoning` parameter. When set to `false`, reasoning tokens are still generated (so model quality is unaffected) but excluded from the response. This reduces network traffic without changing inference behavior.
362
363 The parameter is supported in both the Chat Completions API and the Responses API, for streaming and non-streaming requests.
364
365 When `include_reasoning=false`, vLLM also suppresses per-token metadata (logprobs and token IDs) to prevent leaking reasoning content through decoded token text in logprob entries or raw token IDs.
366
367 ### Chat Completions API
368
369 ```python
370 from openai import OpenAI
371
372 client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
373 model = client.models.list().data[0].id
374
375 # Reasoning is included by default (include_reasoning=True)
376 response = client.chat.completions.create(
377 model=model,
378 messages=[{"role": "user", "content": "What is 15 * 37?"}],
379 extra_body={"include_reasoning": False},
380 )
381
382 msg = response.choices[0].message
383 assert msg.content # Content is still present
384 assert not getattr(msg, "reasoning", None) # Reasoning is suppressed
385 ```
386
387 Streaming works the same way, reasoning deltas are omitted from chunks:
388
389 ```python
390 stream = client.chat.completions.create(
391 model=model,
392 messages=[{"role": "user", "content": "What is 15 * 37?"}],
393 stream=True,
394 extra_body={"include_reasoning": False},
395 )
396
397 for chunk in stream:
398 delta = chunk.choices[0].delta
399 # delta.reasoning will always be None
400 if delta.content:
401 print(delta.content, end="", flush=True)
402 ```
403
404 ### Responses API
405
406 ```python
407 from openai import OpenAI
408
409 client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
410
411 response = client.responses.create(
412 model=client.models.list().data[0].id,
413 input="What is 15 * 37?",
414 include_reasoning=False,
415 )
416
417 # No "reasoning" items in output
418 types = [item.type for item in response.output]
419 assert "reasoning" not in types
420 ```
421
422 ## Limitations
423
424 - The reasoning content is only available for online serving's chat completion endpoint (`/v1/chat/completions`), Anthropic Messages API (`/v1/messages`) and the Responses API (`/v1/responses`).
425
426 ## How to support a new reasoning model
427
428 You can add a new `ReasoningParser` similar to [vllm/reasoning/deepseek_r1_reasoning_parser.py](../../vllm/reasoning/deepseek_r1_reasoning_parser.py).
429
430 ??? code
431
432 ```python
433 # import the required packages
434
435 from vllm.reasoning import ReasoningParser, ReasoningParserManager
436 from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
437 from vllm.entrypoints.openai.engine.protocol import DeltaMessage
438
439 # define a reasoning parser and register it to vllm
440 # the name list in register_module can be used
441 # in --reasoning-parser.
442 class ExampleParser(ReasoningParser):
443 def __init__(self, tokenizer: TokenizerLike):
444 super().__init__(tokenizer)
445
446 def extract_reasoning_streaming(
447 self,
448 previous_text: str,
449 current_text: str,
450 delta_text: str,
451 previous_token_ids: Sequence[int],
452 current_token_ids: Sequence[int],
453 delta_token_ids: Sequence[int],
454 ) -> DeltaMessage | None:
455 """
456 Instance method that should be implemented for extracting reasoning
457 from an incomplete response; for use when handling reasoning calls and
458 streaming. Has to be an instance method because it requires state -
459 the current tokens/diffs, but also the information about what has
460 previously been parsed and extracted (see constructor)
461 """
462
463 def extract_reasoning(
464 self,
465 model_output: str,
466 request: ChatCompletionRequest | ResponsesRequest,
467 ) -> tuple[str | None, str | None]:
468 """
469 Extract reasoning content from a complete model-generated string.
470
471 Used for non-streaming responses where we have the entire model response
472 available before sending to the client.
473
474 Parameters:
475 model_output: str
476 The model-generated string to extract reasoning content from.
477
478 request: ChatCompletionRequest
479 The request object that was used to generate the model_output.
480
481 Returns:
482 tuple[Optional[str], Optional[str]]
483 A tuple containing the reasoning content and the content.
484 """
485 # Register the reasoning parser
486 ReasoningParserManager.register_lazy_module(
487 name="example",
488 module_path="vllm.reasoning.example_reasoning_parser",
489 class_name="ExampleParser",
490 )
491 ```
492
493 Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in [vllm/reasoning/deepseek_r1_reasoning_parser.py](../../vllm/reasoning/deepseek_r1_reasoning_parser.py).
494
495 ??? code
496
497 ```python
498 @dataclass
499 class DeepSeekReasoner(Reasoner):
500 """
501 Reasoner for DeepSeek R series models.
502 """
503 start_token_id: int
504 end_token_id: int
505
506 start_token: str = "<think>"
507 end_token: str = "</think>"
508
509 @classmethod
510 def from_tokenizer(cls, tokenizer: PythonBackend) -> Reasoner:
511 return cls(
512 start_token_id=tokenizer.encode("<think>", add_special_tokens=False)[0],
513 end_token_id=tokenizer.encode("</think>", add_special_tokens=False)[0],
514 )
515
516 def is_reasoning_end(self, input_ids: list[int]) -> bool:
517 return self.end_token_id in input_ids
518
519 def is_reasoning_end_streaming(self, input_ids: list[int], delta_ids: list[int]) -> bool:
520 return self.end_token_id in delta_token_ids
521 ...
522 ```
523
524 The structured output engine like [xgrammar](https://github.com/mlc-ai/xgrammar) will use `end_token_id` to check if the reasoning content is present in the model output and skip the structured output if it is the case.
525
526 Finally, you can enable reasoning for the model by using the `--reasoning-parser` flags.
527
528 ```bash
529 vllm serve <model_tag> --reasoning-parser example
530 ```