A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 00:45 UTC
[Frontend] Support reasoning content for deepseek r1 (#12473)
Signed-off-by: Ce Gao <cegao@tensorchord.ai> Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Michael Goin <mgoin@redhat.com>
Ce Gao committed
Jan 29, 2025 at 11:38 UTC
a7e3eba66fff82f7e12bb2354c4b26635f0f7761
16 files changed
+977
-5
docs/source/features/reasoning_outputs.md
new
+151
@@ -0,0 +1,151 @@
1
+(reasoning-outputs)=
2
+
3
+# Reasoning Outputs
4
+
5
+vLLM offers support for reasoning models like [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1), which are designed to generate outputs containing both reasoning steps and final conclusions.
6
+
7
+Reasoning models return a additional `reasoning_content` field in their outputs, which contains the reasoning steps that led to the final conclusion. This field is not present in the outputs of other models.
8
+
9
+## Supported Models
10
+
11
+vLLM currently supports the following reasoning models:
12
+
13
+- [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d) (`deepseek_r1`, which looks for `<think> ... </think>`)
14
+
15
+## Quickstart
16
+
17
+To use reasoning models, you need to specify the `--enable-reasoning` and `--reasoning-parser` flags when making a request to the chat completion endpoint. The `--reasoning-parser` flag specifies the reasoning parser to use for extracting reasoning content from the model output.
18
+
19
+```bash
20
+vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
21
+ --enable-reasoning --reasoning-parser deepseek_r1
22
+```
23
+
24
+Next, make a request to the model that should return the reasoning content in the response.
25
+
26
+```python
27
+from openai import OpenAI
28
+
29
+# Modify OpenAI's API key and API base to use vLLM's API server.
30
+openai_api_key = "EMPTY"
31
+openai_api_base = "http://localhost:8000/v1"
32
+
33
+client = OpenAI(
34
+ api_key=openai_api_key,
35
+ base_url=openai_api_base,
36
+)
37
+
38
+models = client.models.list()
39
+model = models.data[0].id
40
+
41
+# Round 1
42
+messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
43
+response = client.chat.completions.create(model=model, messages=messages)
44
+
45
+reasoning_content = response.choices[0].message.reasoning_content
46
+content = response.choices[0].message.content
47
+
48
+print("reasoning_content:", reasoning_content)
49
+print("content:", content)
50
+```
51
+
52
+The `reasoning_content` field contains the reasoning steps that led to the final conclusion, while the `content` field contains the final conclusion.
53
+
54
+## Streaming chat completions
55
+
56
+Streaming chat completions are also supported for reasoning models. The `reasoning_content` field is available in the `delta` field in [chat completion response chunks](https://platform.openai.com/docs/api-reference/chat/streaming).
57
+
58
+```json
59
+{
60
+ "id": "chatcmpl-123",
61
+ "object": "chat.completion.chunk",
62
+ "created": 1694268190,
63
+ "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
64
+ "system_fingerprint": "fp_44709d6fcb",
65
+ "choices": [
66
+ {
67
+ "index": 0,
68
+ "delta": {
69
+ "role": "assistant",
70
+ "reasoning_content": "is",
71
+ },
72
+ "logprobs": null,
73
+ "finish_reason": null
74
+ }
75
+ ]
76
+}
77
+```
78
+
79
+Please note that it is not compatible with the OpenAI Python client library. You can use the `requests` library to make streaming requests.
80
+
81
+## How to support a new reasoning model
82
+
83
+You can add a new `ReasoningParser` similar to `vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py`.
84
+
85
+```python
86
+# import the required packages
87
+
88
+from vllm.entrypoints.openai.reasoning_parsers.abs_reasoning_parsers import (
89
+ ReasoningParser, ReasoningParserManager)
90
+from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
91
+ DeltaMessage)
92
+
93
+# define a reasoning parser and register it to vllm
94
+# the name list in register_module can be used
95
+# in --reasoning-parser.
96
+@ReasoningParserManager.register_module(["example"])
97
+class ExampleParser(ReasoningParser):
98
+ def __init__(self, tokenizer: AnyTokenizer):
99
+ super().__init__(tokenizer)
100
+
101
+ def extract_reasoning_content_streaming(
102
+ self,
103
+ previous_text: str,
104
+ current_text: str,
105
+ delta_text: str,
106
+ previous_token_ids: Sequence[int],
107
+ current_token_ids: Sequence[int],
108
+ delta_token_ids: Sequence[int],
109
+ ) -> Union[DeltaMessage, None]:
110
+ """
111
+ Instance method that should be implemented for extracting reasoning
112
+ from an incomplete response; for use when handling reasoning calls and
113
+ streaming. Has to be an instance method because it requires state -
114
+ the current tokens/diffs, but also the information about what has
115
+ previously been parsed and extracted (see constructor)
116
+ """
117
+
118
+ def extract_reasoning_content(
119
+ self, model_output: str, request: ChatCompletionRequest
120
+ ) -> Tuple[Optional[str], Optional[str]]:
121
+ """
122
+ Extract reasoning content from a complete model-generated string.
123
+
124
+ Used for non-streaming responses where we have the entire model response
125
+ available before sending to the client.
126
+
127
+ Parameters:
128
+ model_output: str
129
+ The model-generated string to extract reasoning content from.
130
+
131
+ request: ChatCompletionRequest
132
+ The request object that was used to generate the model_output.
133
+
134
+ Returns:
135
+ Tuple[Optional[str], Optional[str]]
136
+ A tuple containing the reasoning content and the content.
137
+ """
138
+```
139
+
140
+After defining the reasoning parser, you can use it by specifying the `--reasoning-parser` flag when making a request to the chat completion endpoint.
141
+
142
+```bash
143
+vllm serve <model_tag> \
144
+ --enable-reasoning --reasoning-parser example
145
+```
146
+
147
+## Limitations
148
+
149
+- The reasoning content is only available for online serving's chat completion endpoint (`/v1/chat/completions`).
150
+- It is not compatible with the [`structured_outputs`](#structured_outputs) and [`tool_calling`](#tool_calling) features.
151
+- The reasoning content is not available for all models. Check the model's documentation to see if it supports reasoning.
docs/source/index.md
+1
@@ -90,6 +90,7 @@ models/extensions/index
90
features/quantization/index
91
features/lora
92
features/tool_calling
93
+features/reasoning_outputs
94
features/structured_outputs
95
features/automatic_prefix_caching
96
features/disagg_prefill
examples/online_serving/openai_chat_completion_with_reasoning.py
new
+53
@@ -0,0 +1,53 @@
1
+"""
2
+An example shows how to generate chat completions from reasoning models
3
+like DeepSeekR1.
4
+
5
+To run this example, you need to start the vLLM server with the reasoning
6
+parser:
7
+
8
+```bash
9
+vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
10
+ --enable-reasoning --reasoning-parser deepseek_r1
11
+```
12
+
13
+This example demonstrates how to generate chat completions from reasoning models
14
+using the OpenAI Python client library.
15
+"""
16
+
17
+from openai import OpenAI
18
+
19
+# Modify OpenAI's API key and API base to use vLLM's API server.
20
+openai_api_key = "EMPTY"
21
+openai_api_base = "http://localhost:8000/v1"
22
+
23
+client = OpenAI(
24
+ api_key=openai_api_key,
25
+ base_url=openai_api_base,
26
+)
27
+
28
+models = client.models.list()
29
+model = models.data[0].id
30
+
31
+# Round 1
32
+messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
33
+response = client.chat.completions.create(model=model, messages=messages)
34
+
35
+reasoning_content = response.choices[0].message.reasoning_content
36
+content = response.choices[0].message.content
37
+
38
+print("reasoning_content:", reasoning_content)
39
+print("content:", content)
40
+
41
+# Round 2
42
+messages.append({"role": "assistant", "content": content})
43
+messages.append({
44
+ "role": "user",
45
+ "content": "How many Rs are there in the word 'strawberry'?",
46
+})
47
+response = client.chat.completions.create(model=model, messages=messages)
48
+
49
+reasoning_content = response.choices[0].message.reasoning_content
50
+content = response.choices[0].message.content
51
+
52
+print("reasoning_content:", reasoning_content)
53
+print("content:", content)
examples/online_serving/openai_chat_completion_with_reasoning_streaming.py
new
+90
@@ -0,0 +1,90 @@
1
+"""
2
+An example shows how to generate chat completions from reasoning models
3
+like DeepSeekR1.
4
+
5
+To run this example, you need to start the vLLM server with the reasoning
6
+parser:
7
+
8
+```bash
9
+vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
10
+ --enable-reasoning --reasoning-parser deepseek_r1
11
+```
12
+
13
+Unlike openai_chat_completion_with_reasoning.py, this example demonstrates the
14
+streaming chat completions feature.
15
+
16
+The streaming chat completions feature allows you to receive chat completions
17
+in real-time as they are generated by the model. This is useful for scenarios
18
+where you want to display chat completions to the user as they are generated
19
+by the model.
20
+
21
+Here we do not use the OpenAI Python client library, because it does not support
22
+`reasoning_content` fields in the response.
23
+"""
24
+
25
+import json
26
+
27
+import requests
28
+
29
+# Modify OpenAI's API key and API base to use vLLM's API server.
30
+openai_api_key = "EMPTY"
31
+openai_api_base = "http://localhost:8000/v1"
32
+
33
+models = requests.get(
34
+ f"{openai_api_base}/models",
35
+ headers={
36
+ "Authorization": f"Bearer {openai_api_key}"
37
+ },
38
+).json()
39
+model = models["data"][0]["id"]
40
+
41
+# Streaming chat completions
42
+messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
43
+
44
+response = requests.post(
45
+ f"{openai_api_base}/chat/completions",
46
+ headers={"Authorization": f"Bearer {openai_api_key}"},
47
+ json={
48
+ "model": model,
49
+ "messages": messages,
50
+ "stream": True
51
+ },
52
+)
53
+
54
+print("client: Start streaming chat completions...")
55
+printed_reasoning_content = False
56
+printed_content = False
57
+# Make the streaming request
58
+if response.status_code == 200:
59
+ # Process the streaming response
60
+ for line in response.iter_lines():
61
+ if line: # Filter out keep-alive new lines
62
+ # Decode the line and parse the JSON
63
+ decoded_line = line.decode("utf-8")
64
+ if decoded_line.startswith("data:"):
65
+ data = decoded_line[5:].strip() # Remove "data:" prefix
66
+ if data == "[DONE]": # End of stream
67
+ print("\nclient: Stream completed.")
68
+ break
69
+ try:
70
+ # Parse the JSON data
71
+ chunk = json.loads(data)
72
+ reasoning_content = chunk["choices"][0]["delta"].get(
73
+ "reasoning_content", "")
74
+ content = chunk["choices"][0]["delta"].get("content", "")
75
+
76
+ if reasoning_content:
77
+ if not printed_reasoning_content:
78
+ printed_reasoning_content = True
79
+ print("reasoning_content:", end="", flush=True)
80
+ print(reasoning_content, end="", flush=True)
81
+ elif content:
82
+ if not printed_content:
83
+ printed_content = True
84
+ print("\ncontent:", end="", flush=True)
85
+ # Extract and print the content
86
+ print(content, end="", flush=True)
87
+ except json.JSONDecodeError:
88
+ print("Error decoding JSON:", decoded_line)
89
+else:
90
+ print(f"Error: {response.status_code} - {response.text}")
tests/entrypoints/openai/reasoning_parsers/__init__.py
tests/entrypoints/openai/reasoning_parsers/test_deepseekr1_reasoning_parser.py
new
+120
@@ -0,0 +1,120 @@
1
+from typing import List
2
+
3
+import pytest
4
+from transformers import AutoTokenizer
5
+
6
+from tests.entrypoints.openai.reasoning_parsers.utils import (
7
+ run_reasoning_extraction)
8
+from vllm.entrypoints.openai.reasoning_parsers import (ReasoningParser,
9
+ ReasoningParserManager)
10
+
11
+parser_name = "deepseek_r1"
12
+start_token = "<think>"
13
+end_token = "</think>"
14
+
15
+SIMPLE_REASONING = {
16
+ "output": "<think>This is a reasoning section</think>This is the rest",
17
+ "reasoning_content": "This is a reasoning section",
18
+ "content": "This is the rest",
19
+}
20
+COMPLETE_REASONING = {
21
+ "output": "<think>This is a reasoning section</think>",
22
+ "reasoning_content": "This is a reasoning section",
23
+ "content": None,
24
+}
25
+NO_REASONING = {
26
+ "output": "This is a reasoning section",
27
+ "reasoning_content": None,
28
+ "content": "This is a reasoning section",
29
+}
30
+MULTIPLE_LINES = {
31
+ "output": "<think>This\nThat</think>This is the rest\nThat",
32
+ "reasoning_content": "This\nThat",
33
+ "content": "This is the rest\nThat",
34
+}
35
+SHORTEST_REASONING_NO_STREAMING = {
36
+ "output": "<think></think>This is the rest",
37
+ "reasoning_content": "",
38
+ "content": "This is the rest",
39
+}
40
+SHORTEST_REASONING = {
41
+ "output": "<think></think>This is the rest",
42
+ "reasoning_content": None,
43
+ "content": "This is the rest",
44
+}
45
+
46
+TEST_CASES = [
47
+ pytest.param(
48
+ False,
49
+ SIMPLE_REASONING,
50
+ id="simple_streaming",
51
+ ),
52
+ pytest.param(
53
+ True,
54
+ SIMPLE_REASONING,
55
+ id="simple_streaming",
56
+ ),
57
+ pytest.param(
58
+ False,
59
+ COMPLETE_REASONING,
60
+ id="complete_streaming",
61
+ ),
62
+ pytest.param(
63
+ True,
64
+ COMPLETE_REASONING,
65
+ id="complete_streaming",
66
+ ),
67
+ pytest.param(
68
+ False,
69
+ NO_REASONING,
70
+ id="no_streaming",
71
+ ),
72
+ pytest.param(
73
+ True,
74
+ NO_REASONING,
75
+ id="no_streaming",
76
+ ),
77
+ pytest.param(
78
+ False,
79
+ MULTIPLE_LINES,
80
+ id="multiple_lines_streaming",
81
+ ),
82
+ pytest.param(
83
+ True,
84
+ MULTIPLE_LINES,
85
+ id="multiple_lines_streaming",
86
+ ),
87
+ pytest.param(
88
+ True,
89
+ SHORTEST_REASONING,
90
+ id="shortest_streaming",
91
+ ),
92
+ pytest.param(
93
+ False,
94
+ SHORTEST_REASONING_NO_STREAMING,
95
+ id="shortest_streaming",
96
+ ),
97
+]
98
+
99
+
100
+@pytest.mark.parametrize("streaming, param_dict", TEST_CASES)
101
+def test_reasoning(
102
+ streaming: bool,
103
+ param_dict: dict,
104
+):
105
+ tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
106
+ tokenizer.add_tokens([start_token, end_token])
107
+ output = tokenizer.tokenize(param_dict["output"])
108
+ # decode everything to tokens
109
+ output_tokens: List[str] = [
110
+ tokenizer.convert_tokens_to_string([token]) for token in output
111
+ ]
112
+ parser: ReasoningParser = ReasoningParserManager.get_reasoning_parser(
113
+ parser_name)(tokenizer)
114
+
115
+ reasoning, content = run_reasoning_extraction(parser,
116
+ output_tokens,
117
+ streaming=streaming)
118
+
119
+ assert reasoning == param_dict["reasoning_content"]
120
+ assert content == param_dict["content"]
tests/entrypoints/openai/reasoning_parsers/utils.py
new
+93
@@ -0,0 +1,93 @@
1
+from typing import List, Optional, Tuple, Union
2
+
3
+from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
4
+ DeltaMessage)
5
+from vllm.entrypoints.openai.reasoning_parsers import ReasoningParser
6
+
7
+
8
+class StreamingReasoningReconstructor:
9
+
10
+ def __init__(self):
11
+ self.reasoning_content = None
12
+ self.other_content = None
13
+
14
+ def append_delta(self, delta: DeltaMessage):
15
+ # content and the reasoning content should not be present
16
+ # at the same time
17
+ assert delta.content is None or delta.reasoning_content is None, (
18
+ "Both content and reasoning content are present in the "
19
+ "delta message")
20
+ if delta.content is not None:
21
+ if self.other_content is None:
22
+ self.other_content = delta.content
23
+ else:
24
+ self.other_content += delta.content
25
+ else:
26
+ if self.reasoning_content is None:
27
+ self.reasoning_content = delta.reasoning_content
28
+ else:
29
+ self.reasoning_content += delta.reasoning_content
30
+
31
+
32
+def run_reasoning_extraction(
33
+ reasoning_parser: ReasoningParser,
34
+ model_output: List[str],
35
+ request: Union[ChatCompletionRequest, None] = None,
36
+ streaming: bool = False,
37
+) -> Tuple[Optional[str], Optional[str]]:
38
+ if streaming:
39
+ reconstructor = run_reasoning_extraction_streaming(
40
+ reasoning_parser,
41
+ model_output,
42
+ request,
43
+ )
44
+ return (
45
+ reconstructor.reasoning_content,
46
+ reconstructor.other_content or None,
47
+ )
48
+ else:
49
+ reasoning, content = run_reasoning_extraction_nonstreaming(
50
+ reasoning_parser, model_output, request)
51
+ return reasoning, content
52
+
53
+
54
+def run_reasoning_extraction_nonstreaming(
55
+ reasoning_parser: ReasoningParser,
56
+ model_output: List[str],
57
+ request: Union[ChatCompletionRequest, None] = None,
58
+) -> Tuple[Optional[str], Optional[str]]:
59
+ request = request or ChatCompletionRequest(messages=[], model="test-model")
60
+ return reasoning_parser.extract_reasoning_content(
61
+ model_output=''.join(model_output), request=request)
62
+
63
+
64
+def run_reasoning_extraction_streaming(
65
+ reasoning_parser: ReasoningParser,
66
+ model_deltas: List[str],
67
+ request: Union[ChatCompletionRequest, None] = None,
68
+) -> StreamingReasoningReconstructor:
69
+ request = request or ChatCompletionRequest(messages=[], model="test-model")
70
+ reconstructor = StreamingReasoningReconstructor()
71
+ previous_text = ""
72
+ previous_tokens: List[int] = []
73
+ for delta in model_deltas:
74
+ token_delta = [
75
+ reasoning_parser.vocab.get(token)
76
+ for token in reasoning_parser.model_tokenizer.tokenize(delta)
77
+ if token in reasoning_parser.vocab
78
+ ]
79
+ current_text = previous_text + delta
80
+ current_tokens = previous_tokens + token_delta
81
+ delta_message = reasoning_parser.extract_reasoning_content_streaming(
82
+ previous_text,
83
+ current_text,
84
+ delta,
85
+ previous_tokens,
86
+ current_tokens,
87
+ token_delta,
88
+ )
89
+ if delta_message is not None:
90
+ reconstructor.append_delta(delta_message)
91
+ previous_text = current_text
92
+ previous_tokens = current_tokens
93
+ return reconstructor
tests/entrypoints/openai/test_cli_args.py
+29
@@ -116,6 +116,35 @@ def test_enable_auto_choice_passes_with_tool_call_parser(serve_parser):
116
validate_parsed_serve_args(args)
117
118
119
+def test_enable_auto_choice_fails_with_enable_reasoning(serve_parser):
120
+ """Ensure validation fails if reasoning is enabled with auto tool choice"""
121
+ args = serve_parser.parse_args(args=[
122
+ "--enable-auto-tool-choice",
123
+ "--enable-reasoning",
124
+ ])
125
+ with pytest.raises(TypeError):
126
+ validate_parsed_serve_args(args)
127
+
128
+
129
+def test_enable_reasoning_passes_with_reasoning_parser(serve_parser):
130
+ """Ensure validation passes if reasoning is enabled
131
+ with a reasoning parser"""
132
+ args = serve_parser.parse_args(args=[
133
+ "--enable-reasoning",
134
+ "--reasoning-parser",
135
+ "deepseek_r1",
136
+ ])
137
+ validate_parsed_serve_args(args)
138
+
139
+
140
+def test_enable_reasoning_fails_without_reasoning_parser(serve_parser):
141
+ """Ensure validation fails if reasoning is enabled
142
+ without a reasoning parser"""
143
+ args = serve_parser.parse_args(args=["--enable-reasoning"])
144
+ with pytest.raises(TypeError):
145
+ validate_parsed_serve_args(args)
146
+
147
+
148
def test_chat_template_validation_for_happy_paths(serve_parser):
149
"""Ensure validation passes if the chat template exists"""
150
args = serve_parser.parse_args(
vllm/entrypoints/openai/api_server.py
+10
@@ -61,6 +61,7 @@ from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
61
TokenizeRequest,
62
TokenizeResponse,
63
UnloadLoraAdapterRequest)
64
+from vllm.entrypoints.openai.reasoning_parsers import ReasoningParserManager
65
# yapf: enable
66
from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
67
from vllm.entrypoints.openai.serving_completion import OpenAIServingCompletion
@@ -771,6 +772,8 @@ async def init_app_state(
772
return_tokens_as_token_ids=args.return_tokens_as_token_ids,
773
enable_auto_tools=args.enable_auto_tool_choice,
774
tool_parser=args.tool_call_parser,
775
+ enable_reasoning=args.enable_reasoning,
776
+ reasoning_parser=args.reasoning_parser,
777
enable_prompt_tokens_details=args.enable_prompt_tokens_details,
778
) if model_config.runner_type == "generate" else None
779
state.openai_serving_completion = OpenAIServingCompletion(
@@ -844,6 +847,13 @@ async def run_server(args, **uvicorn_kwargs) -> None:
847
raise KeyError(f"invalid tool call parser: {args.tool_call_parser} "
848
f"(chose from {{ {','.join(valid_tool_parses)} }})")
849
850
+ valid_reasoning_parses = ReasoningParserManager.reasoning_parsers.keys()
851
+ if args.enable_reasoning \
852
+ and args.reasoning_parser not in valid_reasoning_parses:
853
+ raise KeyError(
854
+ f"invalid reasoning parser: {args.reasoning_parser} "
855
+ f"(chose from {{ {','.join(valid_reasoning_parses)} }})")
856
+
857
# workaround to make sure that we bind the port before the engine is set up.
858
# This avoids race conditions with ray.
859
# see https://github.com/vllm-project/vllm/issues/8204
vllm/entrypoints/openai/cli_args.py
+30
@@ -12,6 +12,7 @@ from typing import List, Optional, Sequence, Union, get_args
12
from vllm.engine.arg_utils import AsyncEngineArgs, nullable_str
13
from vllm.entrypoints.chat_utils import (ChatTemplateContentFormatOption,
14
validate_chat_template)
15
+from vllm.entrypoints.openai.reasoning_parsers import ReasoningParserManager
16
from vllm.entrypoints.openai.serving_models import (LoRAModulePath,
17
PromptAdapterPath)
18
from vllm.entrypoints.openai.tool_parsers import ToolParserManager
@@ -208,6 +209,23 @@ def make_arg_parser(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
209
default=False,
210
help="Enable auto tool choice for supported models. Use "
211
"``--tool-call-parser`` to specify which parser to use.")
212
+ parser.add_argument(
213
+ "--enable-reasoning",
214
+ action="store_true",
215
+ default=False,
216
+ help="Whether to enable reasoning_content for the model. "
217
+ "If enabled, the model will be able to generate reasoning content.")
218
+
219
+ valid_reasoning_parsers = ReasoningParserManager.reasoning_parsers.keys()
220
+ parser.add_argument(
221
+ "--reasoning-parser",
222
+ type=str,
223
+ metavar="{" + ",".join(valid_reasoning_parsers) + "}",
224
+ default=None,
225
+ help=
226
+ "Select the reasoning parser depending on the model that you're using."
227
+ " This is used to parse the reasoning content into OpenAI API "
228
+ "format. Required for ``--enable-reasoning``.")
229
230
valid_tool_parsers = ToolParserManager.tool_parsers.keys()
231
parser.add_argument(
@@ -267,6 +285,18 @@ def validate_parsed_serve_args(args: argparse.Namespace):
285
raise TypeError("Error: --enable-auto-tool-choice requires "
286
"--tool-call-parser")
287
288
+ # Enable reasoning needs a reasoning parser to be valid
289
+ if args.enable_reasoning and not args.reasoning_parser:
290
+ raise TypeError("Error: --enable-reasoning requires "
291
+ "--reasoning-parser")
292
+
293
+ # Ref https://api-docs.deepseek.com/guides/reasoning_model
294
+ # tool call and reasoning cannot be enabled at the same time.
295
+ if args.enable_auto_tool_choice and args.enable_reasoning:
296
+ raise TypeError(
297
+ "Error: --enable-auto-tool-choice and "
298
+ "--enable-reasoning cannot be enabled at the same time")
299
+
300
301
def create_parser_for_docs() -> FlexibleArgumentParser:
302
parser_for_docs = FlexibleArgumentParser(
vllm/entrypoints/openai/protocol.py
+2
@@ -1202,6 +1202,7 @@ class ExtractedToolCallInformation(BaseModel):
1202
1203
class ChatMessage(OpenAIBaseModel):
1204
role: str
1205
+ reasoning_content: Optional[str] = None
1206
content: Optional[str] = None
1207
tool_calls: List[ToolCall] = Field(default_factory=list)
1208
@@ -1243,6 +1244,7 @@ class ChatCompletionResponse(OpenAIBaseModel):
1244
class DeltaMessage(OpenAIBaseModel):
1245
role: Optional[str] = None
1246
content: Optional[str] = None
1247
+ reasoning_content: Optional[str] = None
1248
tool_calls: List[DeltaToolCall] = Field(default_factory=list)
1249
1250
vllm/entrypoints/openai/reasoning_parsers/__init__.py
new
+6
@@ -0,0 +1,6 @@
1
+from .abs_reasoning_parsers import ReasoningParser, ReasoningParserManager
2
+from .deepseek_r1_reasoning_parser import DeepSeekR1ReasoningParser
3
+
4
+__all__ = [
5
+ "ReasoningParser", "ReasoningParserManager", "DeepSeekR1ReasoningParser"
6
+]
vllm/entrypoints/openai/reasoning_parsers/abs_reasoning_parsers.py
new
+158
@@ -0,0 +1,158 @@
1
+import os
2
+from functools import cached_property
3
+from typing import Callable, Dict, List, Optional, Sequence, Tuple, Type, Union
4
+
5
+from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
6
+ DeltaMessage)
7
+from vllm.logger import init_logger
8
+from vllm.transformers_utils.tokenizer import AnyTokenizer
9
+from vllm.utils import import_from_path, is_list_of
10
+
11
+logger = init_logger(__name__)
12
+
13
+
14
+class ReasoningParser:
15
+ """
16
+ Abstract reasoning parser class that should not be used directly.
17
+ Provided and methods should be used in derived classes.
18
+
19
+ It is used to extract reasoning content from the model output.
20
+ """
21
+
22
+ def __init__(self, tokenizer: AnyTokenizer):
23
+ self.model_tokenizer = tokenizer
24
+
25
+ @cached_property
26
+ def vocab(self) -> Dict[str, int]:
27
+ # NOTE: Only PreTrainedTokenizerFast is guaranteed to have .vocab
28
+ # whereas all tokenizers have .get_vocab()
29
+ return self.model_tokenizer.get_vocab()
30
+
31
+ def extract_reasoning_content(
32
+ self, model_output: str, request: ChatCompletionRequest
33
+ ) -> Tuple[Optional[str], Optional[str]]:
34
+ """
35
+ Extract reasoning content from a complete model-generated string.
36
+
37
+ Used for non-streaming responses where we have the entire model response
38
+ available before sending to the client.
39
+
40
+ Parameters:
41
+ model_output: str
42
+ The model-generated string to extract reasoning content from.
43
+
44
+ request: ChatCompletionRequest
45
+ The request object that was used to generate the model_output.
46
+
47
+ Returns:
48
+ Tuple[Optional[str], Optional[str]]
49
+ A tuple containing the reasoning content and the content.
50
+ """
51
+
52
+ raise NotImplementedError(
53
+ "AbstractReasoningParser.extract_reasoning_calls "
54
+ "has not been implemented!")
55
+
56
+ def extract_reasoning_content_streaming(
57
+ self,
58
+ previous_text: str,
59
+ current_text: str,
60
+ delta_text: str,
61
+ previous_token_ids: Sequence[int],
62
+ current_token_ids: Sequence[int],
63
+ delta_token_ids: Sequence[int],
64
+ ) -> Union[DeltaMessage, None]:
65
+ """
66
+ Instance method that should be implemented for extracting reasoning
67
+ from an incomplete response; for use when handling reasoning calls and
68
+ streaming. Has to be an instance method because it requires state -
69
+ the current tokens/diffs, but also the information about what has
70
+ previously been parsed and extracted (see constructor)
71
+ """
72
+ raise NotImplementedError(
73
+ "AbstractReasoningParser.extract_reasoning_content_streaming "
74
+ "has not been implemented!")
75
+
76
+
77
+class ReasoningParserManager:
78
+ reasoning_parsers: Dict[str, Type] = {}
79
+
80
+ @classmethod
81
+ def get_reasoning_parser(cls, name) -> Type:
82
+ """
83
+ Get reasoning parser by name which is registered by `register_module`.
84
+
85
+ Raise a KeyError exception if the name is not registered.
86
+ """
87
+ if name in cls.reasoning_parsers:
88
+ return cls.reasoning_parsers[name]
89
+
90
+ raise KeyError(f"reasoning helper: '{name}' not found in "
91
+ "reasoning_parsers")
92
+
93
+ @classmethod
94
+ def _register_module(cls,
95
+ module: Type,
96
+ module_name: Optional[Union[str, List[str]]] = None,
97
+ force: bool = True) -> None:
98
+ if not issubclass(module, ReasoningParser):
99
+ raise TypeError("module must be subclass of ReasoningParser, "
100
+ f"but got {type(module)}")
101
+ if module_name is None:
102
+ module_name = module.__name__
103
+ if isinstance(module_name, str):
104
+ module_name = [module_name]
105
+ for name in module_name:
106
+ if not force and name in cls.reasoning_parsers:
107
+ existed_module = cls.reasoning_parsers[name]
108
+ raise KeyError(f"{name} is already registered "
109
+ f"at {existed_module.__module__}")
110
+ cls.reasoning_parsers[name] = module
111
+
112
+ @classmethod
113
+ def register_module(
114
+ cls,
115
+ name: Optional[Union[str, List[str]]] = None,
116
+ force: bool = True,
117
+ module: Union[Type, None] = None) -> Union[type, Callable]:
118
+ """
119
+ Register module with the given name or name list. it can be used as a
120
+ decoder(with module as None) or normal function(with module as not
121
+ None).
122
+ """
123
+ if not isinstance(force, bool):
124
+ raise TypeError(f"force must be a boolean, but got {type(force)}")
125
+
126
+ # raise the error ahead of time
127
+ if not (name is None or isinstance(name, str)
128
+ or is_list_of(name, str)):
129
+ raise TypeError(
130
+ "name must be None, an instance of str, or a sequence of str, "
131
+ f"but got {type(name)}")
132
+
133
+ # use it as a normal method: x.register_module(module=SomeClass)
134
+ if module is not None:
135
+ cls._register_module(module=module, module_name=name, force=force)
136
+ return module
137
+
138
+ # use it as a decorator: @x.register_module()
139
+ def _register(module):
140
+ cls._register_module(module=module, module_name=name, force=force)
141
+ return module
142
+
143
+ return _register
144
+
145
+ @classmethod
146
+ def import_reasoning_parser(cls, plugin_path: str) -> None:
147
+ """
148
+ Import a user-defined reasoning parser by the path
149
+ of the reasoning parser define file.
150
+ """
151
+ module_name = os.path.splitext(os.path.basename(plugin_path))[0]
152
+
153
+ try:
154
+ import_from_path(module_name, plugin_path)
155
+ except Exception:
156
+ logger.exception("Failed to load module '%s' from %s.",
157
+ module_name, plugin_path)
158
+ return
vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py
new
+133
@@ -0,0 +1,133 @@
1
+import re
2
+from typing import Optional, Sequence, Tuple, Union
3
+
4
+from transformers import PreTrainedTokenizerBase
5
+
6
+from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
7
+ DeltaMessage)
8
+from vllm.entrypoints.openai.reasoning_parsers.abs_reasoning_parsers import (
9
+ ReasoningParser, ReasoningParserManager)
10
+from vllm.logger import init_logger
11
+
12
+logger = init_logger(__name__)
13
+
14
+
15
+@ReasoningParserManager.register_module("deepseek_r1")
16
+class DeepSeekR1ReasoningParser(ReasoningParser):
17
+ """
18
+ Reasoning parser for DeepSeek R1 model.
19
+
20
+ The DeepSeek R1 model uses <think>...</think> tokens to denote reasoning
21
+ text. This parser extracts the reasoning content from the model output.
22
+ """
23
+
24
+ def __init__(self, tokenizer: PreTrainedTokenizerBase):
25
+ super().__init__(tokenizer)
26
+ self.think_start_token = "<think>"
27
+ self.think_end_token = "</think>"
28
+
29
+ self.reasoning_regex = re.compile(
30
+ rf"{self.think_start_token}(.*?){self.think_end_token}", re.DOTALL)
31
+
32
+ if not self.model_tokenizer:
33
+ raise ValueError(
34
+ "The model tokenizer must be passed to the ReasoningParser "
35
+ "constructor during construction.")
36
+
37
+ self.think_start_token_id = self.vocab.get(self.think_start_token)
38
+ self.think_end_token_id = self.vocab.get(self.think_end_token)
39
+ if (self.think_start_token_id is None
40
+ or self.think_end_token_id is None):
41
+ raise RuntimeError(
42
+ "DeepSeek R1 reasoning parser could not locate think start/end "
43
+ "tokens in the tokenizer!")
44
+
45
+ def extract_reasoning_content_streaming(
46
+ self,
47
+ previous_text: str,
48
+ current_text: str,
49
+ delta_text: str,
50
+ previous_token_ids: Sequence[int],
51
+ current_token_ids: Sequence[int],
52
+ delta_token_ids: Sequence[int],
53
+ ) -> Union[DeltaMessage, None]:
54
+ """
55
+ Extract reasoning content from a delta message.
56
+ Handles streaming output where previous + delta = current.
57
+ Uses token IDs for faster processing.
58
+ For text <think>abc</think>xyz:
59
+ - 'abc' goes to reasoning_content
60
+ - 'xyz' goes to content
61
+ """
62
+ # Skip single special tokens
63
+ if len(delta_token_ids) == 1 and (delta_token_ids[0] in [
64
+ self.think_start_token_id, self.think_end_token_id
65
+ ]):
66
+ return None
67
+
68
+ if self.think_start_token_id in previous_token_ids:
69
+ if self.think_end_token_id in delta_token_ids:
70
+ # <think> in previous, </think> in delta,
71
+ # extract reasoning content
72
+ end_index = delta_text.find(self.think_end_token)
73
+ reasoning_content = delta_text[:end_index]
74
+ content = delta_text[end_index + len(self.think_end_token):]
75
+ return DeltaMessage(reasoning_content=reasoning_content,
76
+ content=content if content else None)
77
+ elif self.think_end_token_id in previous_token_ids:
78
+ # <think> in previous, </think> in previous,
79
+ # reasoning content continues
80
+ return DeltaMessage(content=delta_text)
81
+ else:
82
+ # <think> in previous, no </think> in previous or delta,
83
+ # reasoning content continues
84
+ return DeltaMessage(reasoning_content=delta_text)
85
+ elif self.think_start_token_id in delta_token_ids:
86
+ logger.info(delta_text)
87
+ if self.think_end_token_id in delta_token_ids:
88
+ # <think> in delta, </think> in delta, extract reasoning content
89
+ start_index = delta_text.find(self.think_start_token)
90
+ end_index = delta_text.find(self.think_end_token)
91
+ reasoning_content = delta_text[start_index +
92
+ len(self.think_start_token
93
+ ):end_index]
94
+ content = delta_text[end_index + len(self.think_end_token):]
95
+ return DeltaMessage(reasoning_content=reasoning_content,
96
+ content=content if content else None)
97
+ else:
98
+ # <think> in delta, no </think> in delta,
99
+ # reasoning content continues
100
+ return DeltaMessage(reasoning_content=delta_text)
101
+ else:
102
+ # No <think> in previous or delta, reasoning content continues.
103
+ return DeltaMessage(content=delta_text)
104
+
105
+ def extract_reasoning_content(
106
+ self, model_output: str, request: ChatCompletionRequest
107
+ ) -> Tuple[Optional[str], Optional[str]]:
108
+
109
+ # Check if the model output contains the <think> tokens.
110
+ if (self.think_start_token not in model_output
111
+ or self.think_end_token not in model_output):
112
+ return None, model_output
113
+ else:
114
+ # Use a regex to find the reasoning content
115
+ reasoning_content = self.reasoning_regex.findall(model_output)[0]
116
+
117
+ # Remove the reasoning content from the model output
118
+ # Although deepseek's <think> token is always at the
119
+ # beginning of the line, we cannot guarantee that the
120
+ # other models will follow this convention.
121
+ # Therefore, we need to add :start_index.
122
+ start_index = model_output.find(self.think_start_token)
123
+ if start_index != -1:
124
+ end_index = start_index + len(
125
+ f"{self.think_start_token}{reasoning_content}{self.think_end_token}"
126
+ )
127
+ model_output = model_output[:start_index] + \
128
+ model_output[end_index:]
129
+
130
+ if len(model_output) == 0:
131
+ return reasoning_content, None
132
+
133
+ return reasoning_content, model_output
vllm/entrypoints/openai/serving_chat.py
+100
-5
@@ -21,6 +21,8 @@ from vllm.entrypoints.openai.protocol import (
21
ChatCompletionStreamResponse, ChatMessage, DeltaFunctionCall, DeltaMessage,
22
DeltaToolCall, ErrorResponse, FunctionCall, PromptTokenUsageInfo,
23
RequestResponseMetadata, ToolCall, UsageInfo)
24
+from vllm.entrypoints.openai.reasoning_parsers import (ReasoningParser,
25
+ ReasoningParserManager)
26
from vllm.entrypoints.openai.serving_engine import OpenAIServing
27
from vllm.entrypoints.openai.serving_models import OpenAIServingModels
28
from vllm.entrypoints.openai.tool_parsers import ToolParser, ToolParserManager
@@ -47,6 +49,8 @@ class OpenAIServingChat(OpenAIServing):
49
chat_template: Optional[str],
50
chat_template_content_format: ChatTemplateContentFormatOption,
51
return_tokens_as_token_ids: bool = False,
52
+ enable_reasoning: bool = False,
53
+ reasoning_parser: Optional[str] = None,
54
enable_auto_tools: bool = False,
55
tool_parser: Optional[str] = None,
56
enable_prompt_tokens_details: bool = False,
@@ -69,6 +73,18 @@ class OpenAIServingChat(OpenAIServing):
73
" the parallel_tool_calls client option is preset for "
74
"compatibility reasons, it will be ignored.")
75
76
+ self.enable_reasoning: bool = enable_reasoning
77
+ self.reasoning_parser: Optional[Callable[[AnyTokenizer],
78
+ ReasoningParser]] = None
79
+ if self.enable_reasoning:
80
+ try:
81
+ self.reasoning_parser = (
82
+ ReasoningParserManager.get_reasoning_parser(
83
+ reasoning_parser))
84
+ except Exception as e:
85
+ raise TypeError("Error: --enable-reasoning requires "
86
+ f"reasoning_parser:'{reasoning_parser}' "
87
+ "which has not been registered") from e
88
self.tool_parser: Optional[Callable[[AnyTokenizer], ToolParser]] = None
89
if self.enable_auto_tools:
90
try:
@@ -285,14 +301,35 @@ class OpenAIServingChat(OpenAIServing):
301
not tool_choice_function_name
302
and self._should_stream_with_auto_tool_parsing(request))
303
304
+ should_stream_with_reasoning_parsing = (
305
+ self._should_stream_with_reasoning_parsing(request))
306
+
307
all_previous_token_ids: Optional[List[List[int]]]
289
- if tool_choice_auto:
308
+
309
+ # Only one of these will be used, thus previous_texts and
310
+ # all_previous_token_ids will not be used twice in the same iteration.
311
+ if tool_choice_auto or should_stream_with_reasoning_parsing:
312
# These are only required in "auto" tool choice case
313
previous_texts = [""] * num_choices
314
all_previous_token_ids = [[]] * num_choices
315
else:
316
previous_texts, all_previous_token_ids = None, None
317
318
+ try:
319
+ # There is no need to check if the reasoning_parser is None
320
+ # because the should_stream_with_reasoning_parsing check
321
+ # already ensures that the reasoning_parser is not None.
322
+ # but the pre-commit hook requires it.
323
+ if should_stream_with_reasoning_parsing and \
324
+ self.reasoning_parser is not None:
325
+ reasoning_parser = self.reasoning_parser(tokenizer)
326
+ except RuntimeError as e:
327
+ logger.exception("Error in reasoning parser creation.")
328
+ data = self.create_streaming_error_response(str(e))
329
+ yield f"data: {data}\n\n"
330
+ yield "data: [DONE]\n\n"
331
+ return
332
+
333
# Prepare the tool parser if it's needed
334
try:
335
if tool_choice_auto and self.tool_parser:
@@ -456,6 +493,32 @@ class OpenAIServingChat(OpenAIServing):
493
# update the previous values for the next iteration
494
previous_texts[i] = current_text
495
all_previous_token_ids[i] = current_token_ids
496
+ # reasoning_content cannot be enabled with tool_choice.
497
+ # If it is, the tool_choice will be used instead.
498
+ elif self.enable_reasoning:
499
+ # handle reasoning_content delta
500
+ assert reasoning_parser is not None
501
+ assert previous_texts is not None
502
+ assert all_previous_token_ids is not None
503
+ previous_text = previous_texts[i]
504
+ previous_token_ids = all_previous_token_ids[i]
505
+ current_text = previous_text + delta_text
506
+ current_token_ids = previous_token_ids + list(
507
+ output.token_ids)
508
+
509
+ delta_message = (reasoning_parser.
510
+ extract_reasoning_content_streaming(
511
+ previous_text,
512
+ current_text,
513
+ delta_text,
514
+ previous_token_ids,
515
+ current_token_ids,
516
+ output.token_ids,
517
+ ))
518
+
519
+ # update the previous values for the next iteration
520
+ previous_texts[i] = current_text
521
+ all_previous_token_ids[i] = current_token_ids
522
523
# handle streaming just a content delta
524
else:
@@ -642,17 +705,38 @@ class OpenAIServingChat(OpenAIServing):
705
else:
706
logprobs = None
707
708
+ should_stream_with_reasoning_parsing = (
709
+ self._should_stream_with_reasoning_parsing(request))
710
+
711
# In the OpenAI API the finish_reason is "tools_called"
712
# if the tool choice is auto and the model produced a tool
713
# call. The same is not true for named function calls
714
auto_tools_called = False
715
716
+ if should_stream_with_reasoning_parsing and \
717
+ self.reasoning_parser is not None:
718
+ try:
719
+ reasoning_parser = self.reasoning_parser(tokenizer)
720
+ except RuntimeError as e:
721
+ logger.exception("Error in reasoning parser creation.")
722
+ return self.create_error_response(str(e))
723
+
724
+ reasoning_content, content = (
725
+ reasoning_parser.extract_reasoning_content(
726
+ output.text, request=request))
727
+
728
+ if reasoning_content:
729
+ message = ChatMessage(role=role,
730
+ content=content,
731
+ reasoning_content=reasoning_content)
732
+ else:
733
+ message = ChatMessage(role=role, content=output.text)
734
+
735
# if auto tools are not enabled, and a named tool choice using
736
# outlines is not being used
652
- if (not self.enable_auto_tools
653
- or not self.tool_parser) and not isinstance(
654
- request.tool_choice,
655
- ChatCompletionNamedToolChoiceParam):
737
+ elif (not self.enable_auto_tools
738
+ or not self.tool_parser) and not isinstance(
739
+ request.tool_choice, ChatCompletionNamedToolChoiceParam):
740
message = ChatMessage(role=role, content=output.text)
741
742
# if the request uses tools and specified a tool choice
@@ -835,6 +919,17 @@ class OpenAIServingChat(OpenAIServing):
919
return (request.tools and self.tool_parser and self.enable_auto_tools
920
and request.tool_choice in ['auto', None])
921
922
+ def _should_stream_with_reasoning_parsing(self,
923
+ request: ChatCompletionRequest):
924
+ """
925
+ Utility function to check if streamed tokens should go through the
926
+ reasoning parser that was configured.
927
+
928
+ We only want to do this IF reasoning is enabled and a reasoning
929
+ parser is configured.
930
+ """
931
+ return self.enable_reasoning and self.reasoning_parser is not None
932
+
933
def _should_check_for_unstreamed_tool_arg_tokens(
934
self,
935
delta_message: Optional[DeltaMessage],
vllm/scripts.py
+1
@@ -167,6 +167,7 @@ def main():
167
"Must be a YAML with the following options:"
168
"https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#cli-reference"
169
)
170
+
171
serve_parser = make_arg_parser(serve_parser)
172
serve_parser.set_defaults(dispatch_function=serve)
173