@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / a7e3eba66f
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 00:45 UTC

[Frontend] Support reasoning content for deepseek r1 (#12473)

Signed-off-by: Ce Gao <cegao@tensorchord.ai> Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Michael Goin <mgoin@redhat.com>

Ce Gao committed Jan 29, 2025 at 11:38 UTC a7e3eba66fff82f7e12bb2354c4b26635f0f7761
16 files changed +977 -5
docs/source/features/reasoning_outputs.md new
+151
@@ -0,0 +1,151 @@
1 +(reasoning-outputs)=
2 +
3 +# Reasoning Outputs
4 +
5 +vLLM offers support for reasoning models like [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1), which are designed to generate outputs containing both reasoning steps and final conclusions.
6 +
7 +Reasoning models return a additional `reasoning_content` field in their outputs, which contains the reasoning steps that led to the final conclusion. This field is not present in the outputs of other models.
8 +
9 +## Supported Models
10 +
11 +vLLM currently supports the following reasoning models:
12 +
13 +- [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d) (`deepseek_r1`, which looks for `<think> ... </think>`)
14 +
15 +## Quickstart
16 +
17 +To use reasoning models, you need to specify the `--enable-reasoning` and `--reasoning-parser` flags when making a request to the chat completion endpoint. The `--reasoning-parser` flag specifies the reasoning parser to use for extracting reasoning content from the model output.
18 +
19 +```bash
20 +vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
21 + --enable-reasoning --reasoning-parser deepseek_r1
22 +```
23 +
24 +Next, make a request to the model that should return the reasoning content in the response.
25 +
26 +```python
27 +from openai import OpenAI
28 +
29 +# Modify OpenAI's API key and API base to use vLLM's API server.
30 +openai_api_key = "EMPTY"
31 +openai_api_base = "http://localhost:8000/v1"
32 +
33 +client = OpenAI(
34 + api_key=openai_api_key,
35 + base_url=openai_api_base,
36 +)
37 +
38 +models = client.models.list()
39 +model = models.data[0].id
40 +
41 +# Round 1
42 +messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
43 +response = client.chat.completions.create(model=model, messages=messages)
44 +
45 +reasoning_content = response.choices[0].message.reasoning_content
46 +content = response.choices[0].message.content
47 +
48 +print("reasoning_content:", reasoning_content)
49 +print("content:", content)
50 +```
51 +
52 +The `reasoning_content` field contains the reasoning steps that led to the final conclusion, while the `content` field contains the final conclusion.
53 +
54 +## Streaming chat completions
55 +
56 +Streaming chat completions are also supported for reasoning models. The `reasoning_content` field is available in the `delta` field in [chat completion response chunks](https://platform.openai.com/docs/api-reference/chat/streaming).
57 +
58 +```json
59 +{
60 + "id": "chatcmpl-123",
61 + "object": "chat.completion.chunk",
62 + "created": 1694268190,
63 + "model": "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
64 + "system_fingerprint": "fp_44709d6fcb",
65 + "choices": [
66 + {
67 + "index": 0,
68 + "delta": {
69 + "role": "assistant",
70 + "reasoning_content": "is",
71 + },
72 + "logprobs": null,
73 + "finish_reason": null
74 + }
75 + ]
76 +}
77 +```
78 +
79 +Please note that it is not compatible with the OpenAI Python client library. You can use the `requests` library to make streaming requests.
80 +
81 +## How to support a new reasoning model
82 +
83 +You can add a new `ReasoningParser` similar to `vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py`.
84 +
85 +```python
86 +# import the required packages
87 +
88 +from vllm.entrypoints.openai.reasoning_parsers.abs_reasoning_parsers import (
89 + ReasoningParser, ReasoningParserManager)
90 +from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
91 + DeltaMessage)
92 +
93 +# define a reasoning parser and register it to vllm
94 +# the name list in register_module can be used
95 +# in --reasoning-parser.
96 +@ReasoningParserManager.register_module(["example"])
97 +class ExampleParser(ReasoningParser):
98 + def __init__(self, tokenizer: AnyTokenizer):
99 + super().__init__(tokenizer)
100 +
101 + def extract_reasoning_content_streaming(
102 + self,
103 + previous_text: str,
104 + current_text: str,
105 + delta_text: str,
106 + previous_token_ids: Sequence[int],
107 + current_token_ids: Sequence[int],
108 + delta_token_ids: Sequence[int],
109 + ) -> Union[DeltaMessage, None]:
110 + """
111 + Instance method that should be implemented for extracting reasoning
112 + from an incomplete response; for use when handling reasoning calls and
113 + streaming. Has to be an instance method because it requires state -
114 + the current tokens/diffs, but also the information about what has
115 + previously been parsed and extracted (see constructor)
116 + """
117 +
118 + def extract_reasoning_content(
119 + self, model_output: str, request: ChatCompletionRequest
120 + ) -> Tuple[Optional[str], Optional[str]]:
121 + """
122 + Extract reasoning content from a complete model-generated string.
123 +
124 + Used for non-streaming responses where we have the entire model response
125 + available before sending to the client.
126 +
127 + Parameters:
128 + model_output: str
129 + The model-generated string to extract reasoning content from.
130 +
131 + request: ChatCompletionRequest
132 + The request object that was used to generate the model_output.
133 +
134 + Returns:
135 + Tuple[Optional[str], Optional[str]]
136 + A tuple containing the reasoning content and the content.
137 + """
138 +```
139 +
140 +After defining the reasoning parser, you can use it by specifying the `--reasoning-parser` flag when making a request to the chat completion endpoint.
141 +
142 +```bash
143 +vllm serve <model_tag> \
144 + --enable-reasoning --reasoning-parser example
145 +```
146 +
147 +## Limitations
148 +
149 +- The reasoning content is only available for online serving's chat completion endpoint (`/v1/chat/completions`).
150 +- It is not compatible with the [`structured_outputs`](#structured_outputs) and [`tool_calling`](#tool_calling) features.
151 +- The reasoning content is not available for all models. Check the model's documentation to see if it supports reasoning.
docs/source/index.md
+1
@@ -90,6 +90,7 @@ models/extensions/index
90 features/quantization/index
91 features/lora
92 features/tool_calling
93 +features/reasoning_outputs
94 features/structured_outputs
95 features/automatic_prefix_caching
96 features/disagg_prefill
examples/online_serving/openai_chat_completion_with_reasoning.py new
+53
@@ -0,0 +1,53 @@
1 +"""
2 +An example shows how to generate chat completions from reasoning models
3 +like DeepSeekR1.
4 +
5 +To run this example, you need to start the vLLM server with the reasoning
6 +parser:
7 +
8 +```bash
9 +vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
10 + --enable-reasoning --reasoning-parser deepseek_r1
11 +```
12 +
13 +This example demonstrates how to generate chat completions from reasoning models
14 +using the OpenAI Python client library.
15 +"""
16 +
17 +from openai import OpenAI
18 +
19 +# Modify OpenAI's API key and API base to use vLLM's API server.
20 +openai_api_key = "EMPTY"
21 +openai_api_base = "http://localhost:8000/v1"
22 +
23 +client = OpenAI(
24 + api_key=openai_api_key,
25 + base_url=openai_api_base,
26 +)
27 +
28 +models = client.models.list()
29 +model = models.data[0].id
30 +
31 +# Round 1
32 +messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
33 +response = client.chat.completions.create(model=model, messages=messages)
34 +
35 +reasoning_content = response.choices[0].message.reasoning_content
36 +content = response.choices[0].message.content
37 +
38 +print("reasoning_content:", reasoning_content)
39 +print("content:", content)
40 +
41 +# Round 2
42 +messages.append({"role": "assistant", "content": content})
43 +messages.append({
44 + "role": "user",
45 + "content": "How many Rs are there in the word 'strawberry'?",
46 +})
47 +response = client.chat.completions.create(model=model, messages=messages)
48 +
49 +reasoning_content = response.choices[0].message.reasoning_content
50 +content = response.choices[0].message.content
51 +
52 +print("reasoning_content:", reasoning_content)
53 +print("content:", content)
examples/online_serving/openai_chat_completion_with_reasoning_streaming.py new
+90
@@ -0,0 +1,90 @@
1 +"""
2 +An example shows how to generate chat completions from reasoning models
3 +like DeepSeekR1.
4 +
5 +To run this example, you need to start the vLLM server with the reasoning
6 +parser:
7 +
8 +```bash
9 +vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
10 + --enable-reasoning --reasoning-parser deepseek_r1
11 +```
12 +
13 +Unlike openai_chat_completion_with_reasoning.py, this example demonstrates the
14 +streaming chat completions feature.
15 +
16 +The streaming chat completions feature allows you to receive chat completions
17 +in real-time as they are generated by the model. This is useful for scenarios
18 +where you want to display chat completions to the user as they are generated
19 +by the model.
20 +
21 +Here we do not use the OpenAI Python client library, because it does not support
22 +`reasoning_content` fields in the response.
23 +"""
24 +
25 +import json
26 +
27 +import requests
28 +
29 +# Modify OpenAI's API key and API base to use vLLM's API server.
30 +openai_api_key = "EMPTY"
31 +openai_api_base = "http://localhost:8000/v1"
32 +
33 +models = requests.get(
34 + f"{openai_api_base}/models",
35 + headers={
36 + "Authorization": f"Bearer {openai_api_key}"
37 + },
38 +).json()
39 +model = models["data"][0]["id"]
40 +
41 +# Streaming chat completions
42 +messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
43 +
44 +response = requests.post(
45 + f"{openai_api_base}/chat/completions",
46 + headers={"Authorization": f"Bearer {openai_api_key}"},
47 + json={
48 + "model": model,
49 + "messages": messages,
50 + "stream": True
51 + },
52 +)
53 +
54 +print("client: Start streaming chat completions...")
55 +printed_reasoning_content = False
56 +printed_content = False
57 +# Make the streaming request
58 +if response.status_code == 200:
59 + # Process the streaming response
60 + for line in response.iter_lines():
61 + if line: # Filter out keep-alive new lines
62 + # Decode the line and parse the JSON
63 + decoded_line = line.decode("utf-8")
64 + if decoded_line.startswith("data:"):
65 + data = decoded_line[5:].strip() # Remove "data:" prefix
66 + if data == "[DONE]": # End of stream
67 + print("\nclient: Stream completed.")
68 + break
69 + try:
70 + # Parse the JSON data
71 + chunk = json.loads(data)
72 + reasoning_content = chunk["choices"][0]["delta"].get(
73 + "reasoning_content", "")
74 + content = chunk["choices"][0]["delta"].get("content", "")
75 +
76 + if reasoning_content:
77 + if not printed_reasoning_content:
78 + printed_reasoning_content = True
79 + print("reasoning_content:", end="", flush=True)
80 + print(reasoning_content, end="", flush=True)
81 + elif content:
82 + if not printed_content:
83 + printed_content = True
84 + print("\ncontent:", end="", flush=True)
85 + # Extract and print the content
86 + print(content, end="", flush=True)
87 + except json.JSONDecodeError:
88 + print("Error decoding JSON:", decoded_line)
89 +else:
90 + print(f"Error: {response.status_code} - {response.text}")
tests/entrypoints/openai/reasoning_parsers/__init__.py
tests/entrypoints/openai/reasoning_parsers/test_deepseekr1_reasoning_parser.py new
+120
@@ -0,0 +1,120 @@
1 +from typing import List
2 +
3 +import pytest
4 +from transformers import AutoTokenizer
5 +
6 +from tests.entrypoints.openai.reasoning_parsers.utils import (
7 + run_reasoning_extraction)
8 +from vllm.entrypoints.openai.reasoning_parsers import (ReasoningParser,
9 + ReasoningParserManager)
10 +
11 +parser_name = "deepseek_r1"
12 +start_token = "<think>"
13 +end_token = "</think>"
14 +
15 +SIMPLE_REASONING = {
16 + "output": "<think>This is a reasoning section</think>This is the rest",
17 + "reasoning_content": "This is a reasoning section",
18 + "content": "This is the rest",
19 +}
20 +COMPLETE_REASONING = {
21 + "output": "<think>This is a reasoning section</think>",
22 + "reasoning_content": "This is a reasoning section",
23 + "content": None,
24 +}
25 +NO_REASONING = {
26 + "output": "This is a reasoning section",
27 + "reasoning_content": None,
28 + "content": "This is a reasoning section",
29 +}
30 +MULTIPLE_LINES = {
31 + "output": "<think>This\nThat</think>This is the rest\nThat",
32 + "reasoning_content": "This\nThat",
33 + "content": "This is the rest\nThat",
34 +}
35 +SHORTEST_REASONING_NO_STREAMING = {
36 + "output": "<think></think>This is the rest",
37 + "reasoning_content": "",
38 + "content": "This is the rest",
39 +}
40 +SHORTEST_REASONING = {
41 + "output": "<think></think>This is the rest",
42 + "reasoning_content": None,
43 + "content": "This is the rest",
44 +}
45 +
46 +TEST_CASES = [
47 + pytest.param(
48 + False,
49 + SIMPLE_REASONING,
50 + id="simple_streaming",
51 + ),
52 + pytest.param(
53 + True,
54 + SIMPLE_REASONING,
55 + id="simple_streaming",
56 + ),
57 + pytest.param(
58 + False,
59 + COMPLETE_REASONING,
60 + id="complete_streaming",
61 + ),
62 + pytest.param(
63 + True,
64 + COMPLETE_REASONING,
65 + id="complete_streaming",
66 + ),
67 + pytest.param(
68 + False,
69 + NO_REASONING,
70 + id="no_streaming",
71 + ),
72 + pytest.param(
73 + True,
74 + NO_REASONING,
75 + id="no_streaming",
76 + ),
77 + pytest.param(
78 + False,
79 + MULTIPLE_LINES,
80 + id="multiple_lines_streaming",
81 + ),
82 + pytest.param(
83 + True,
84 + MULTIPLE_LINES,
85 + id="multiple_lines_streaming",
86 + ),
87 + pytest.param(
88 + True,
89 + SHORTEST_REASONING,
90 + id="shortest_streaming",
91 + ),
92 + pytest.param(
93 + False,
94 + SHORTEST_REASONING_NO_STREAMING,
95 + id="shortest_streaming",
96 + ),
97 +]
98 +
99 +
100 +@pytest.mark.parametrize("streaming, param_dict", TEST_CASES)
101 +def test_reasoning(
102 + streaming: bool,
103 + param_dict: dict,
104 +):
105 + tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
106 + tokenizer.add_tokens([start_token, end_token])
107 + output = tokenizer.tokenize(param_dict["output"])
108 + # decode everything to tokens
109 + output_tokens: List[str] = [
110 + tokenizer.convert_tokens_to_string([token]) for token in output
111 + ]
112 + parser: ReasoningParser = ReasoningParserManager.get_reasoning_parser(
113 + parser_name)(tokenizer)
114 +
115 + reasoning, content = run_reasoning_extraction(parser,
116 + output_tokens,
117 + streaming=streaming)
118 +
119 + assert reasoning == param_dict["reasoning_content"]
120 + assert content == param_dict["content"]
tests/entrypoints/openai/reasoning_parsers/utils.py new
+93
@@ -0,0 +1,93 @@
1 +from typing import List, Optional, Tuple, Union
2 +
3 +from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
4 + DeltaMessage)
5 +from vllm.entrypoints.openai.reasoning_parsers import ReasoningParser
6 +
7 +
8 +class StreamingReasoningReconstructor:
9 +
10 + def __init__(self):
11 + self.reasoning_content = None
12 + self.other_content = None
13 +
14 + def append_delta(self, delta: DeltaMessage):
15 + # content and the reasoning content should not be present
16 + # at the same time
17 + assert delta.content is None or delta.reasoning_content is None, (
18 + "Both content and reasoning content are present in the "
19 + "delta message")
20 + if delta.content is not None:
21 + if self.other_content is None:
22 + self.other_content = delta.content
23 + else:
24 + self.other_content += delta.content
25 + else:
26 + if self.reasoning_content is None:
27 + self.reasoning_content = delta.reasoning_content
28 + else:
29 + self.reasoning_content += delta.reasoning_content
30 +
31 +
32 +def run_reasoning_extraction(
33 + reasoning_parser: ReasoningParser,
34 + model_output: List[str],
35 + request: Union[ChatCompletionRequest, None] = None,
36 + streaming: bool = False,
37 +) -> Tuple[Optional[str], Optional[str]]:
38 + if streaming:
39 + reconstructor = run_reasoning_extraction_streaming(
40 + reasoning_parser,
41 + model_output,
42 + request,
43 + )
44 + return (
45 + reconstructor.reasoning_content,
46 + reconstructor.other_content or None,
47 + )
48 + else:
49 + reasoning, content = run_reasoning_extraction_nonstreaming(
50 + reasoning_parser, model_output, request)
51 + return reasoning, content
52 +
53 +
54 +def run_reasoning_extraction_nonstreaming(
55 + reasoning_parser: ReasoningParser,
56 + model_output: List[str],
57 + request: Union[ChatCompletionRequest, None] = None,
58 +) -> Tuple[Optional[str], Optional[str]]:
59 + request = request or ChatCompletionRequest(messages=[], model="test-model")
60 + return reasoning_parser.extract_reasoning_content(
61 + model_output=''.join(model_output), request=request)
62 +
63 +
64 +def run_reasoning_extraction_streaming(
65 + reasoning_parser: ReasoningParser,
66 + model_deltas: List[str],
67 + request: Union[ChatCompletionRequest, None] = None,
68 +) -> StreamingReasoningReconstructor:
69 + request = request or ChatCompletionRequest(messages=[], model="test-model")
70 + reconstructor = StreamingReasoningReconstructor()
71 + previous_text = ""
72 + previous_tokens: List[int] = []
73 + for delta in model_deltas:
74 + token_delta = [
75 + reasoning_parser.vocab.get(token)
76 + for token in reasoning_parser.model_tokenizer.tokenize(delta)
77 + if token in reasoning_parser.vocab
78 + ]
79 + current_text = previous_text + delta
80 + current_tokens = previous_tokens + token_delta
81 + delta_message = reasoning_parser.extract_reasoning_content_streaming(
82 + previous_text,
83 + current_text,
84 + delta,
85 + previous_tokens,
86 + current_tokens,
87 + token_delta,
88 + )
89 + if delta_message is not None:
90 + reconstructor.append_delta(delta_message)
91 + previous_text = current_text
92 + previous_tokens = current_tokens
93 + return reconstructor
tests/entrypoints/openai/test_cli_args.py
+29
@@ -116,6 +116,35 @@ def test_enable_auto_choice_passes_with_tool_call_parser(serve_parser):
116 validate_parsed_serve_args(args)
117
118
119 +def test_enable_auto_choice_fails_with_enable_reasoning(serve_parser):
120 + """Ensure validation fails if reasoning is enabled with auto tool choice"""
121 + args = serve_parser.parse_args(args=[
122 + "--enable-auto-tool-choice",
123 + "--enable-reasoning",
124 + ])
125 + with pytest.raises(TypeError):
126 + validate_parsed_serve_args(args)
127 +
128 +
129 +def test_enable_reasoning_passes_with_reasoning_parser(serve_parser):
130 + """Ensure validation passes if reasoning is enabled
131 + with a reasoning parser"""
132 + args = serve_parser.parse_args(args=[
133 + "--enable-reasoning",
134 + "--reasoning-parser",
135 + "deepseek_r1",
136 + ])
137 + validate_parsed_serve_args(args)
138 +
139 +
140 +def test_enable_reasoning_fails_without_reasoning_parser(serve_parser):
141 + """Ensure validation fails if reasoning is enabled
142 + without a reasoning parser"""
143 + args = serve_parser.parse_args(args=["--enable-reasoning"])
144 + with pytest.raises(TypeError):
145 + validate_parsed_serve_args(args)
146 +
147 +
148 def test_chat_template_validation_for_happy_paths(serve_parser):
149 """Ensure validation passes if the chat template exists"""
150 args = serve_parser.parse_args(
vllm/entrypoints/openai/api_server.py
+10
@@ -61,6 +61,7 @@ from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
61 TokenizeRequest,
62 TokenizeResponse,
63 UnloadLoraAdapterRequest)
64 +from vllm.entrypoints.openai.reasoning_parsers import ReasoningParserManager
65 # yapf: enable
66 from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
67 from vllm.entrypoints.openai.serving_completion import OpenAIServingCompletion
@@ -771,6 +772,8 @@ async def init_app_state(
772 return_tokens_as_token_ids=args.return_tokens_as_token_ids,
773 enable_auto_tools=args.enable_auto_tool_choice,
774 tool_parser=args.tool_call_parser,
775 + enable_reasoning=args.enable_reasoning,
776 + reasoning_parser=args.reasoning_parser,
777 enable_prompt_tokens_details=args.enable_prompt_tokens_details,
778 ) if model_config.runner_type == "generate" else None
779 state.openai_serving_completion = OpenAIServingCompletion(
@@ -844,6 +847,13 @@ async def run_server(args, **uvicorn_kwargs) -> None:
847 raise KeyError(f"invalid tool call parser: {args.tool_call_parser} "
848 f"(chose from {{ {','.join(valid_tool_parses)} }})")
849
850 + valid_reasoning_parses = ReasoningParserManager.reasoning_parsers.keys()
851 + if args.enable_reasoning \
852 + and args.reasoning_parser not in valid_reasoning_parses:
853 + raise KeyError(
854 + f"invalid reasoning parser: {args.reasoning_parser} "
855 + f"(chose from {{ {','.join(valid_reasoning_parses)} }})")
856 +
857 # workaround to make sure that we bind the port before the engine is set up.
858 # This avoids race conditions with ray.
859 # see https://github.com/vllm-project/vllm/issues/8204
vllm/entrypoints/openai/cli_args.py
+30
@@ -12,6 +12,7 @@ from typing import List, Optional, Sequence, Union, get_args
12 from vllm.engine.arg_utils import AsyncEngineArgs, nullable_str
13 from vllm.entrypoints.chat_utils import (ChatTemplateContentFormatOption,
14 validate_chat_template)
15 +from vllm.entrypoints.openai.reasoning_parsers import ReasoningParserManager
16 from vllm.entrypoints.openai.serving_models import (LoRAModulePath,
17 PromptAdapterPath)
18 from vllm.entrypoints.openai.tool_parsers import ToolParserManager
@@ -208,6 +209,23 @@ def make_arg_parser(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
209 default=False,
210 help="Enable auto tool choice for supported models. Use "
211 "``--tool-call-parser`` to specify which parser to use.")
212 + parser.add_argument(
213 + "--enable-reasoning",
214 + action="store_true",
215 + default=False,
216 + help="Whether to enable reasoning_content for the model. "
217 + "If enabled, the model will be able to generate reasoning content.")
218 +
219 + valid_reasoning_parsers = ReasoningParserManager.reasoning_parsers.keys()
220 + parser.add_argument(
221 + "--reasoning-parser",
222 + type=str,
223 + metavar="{" + ",".join(valid_reasoning_parsers) + "}",
224 + default=None,
225 + help=
226 + "Select the reasoning parser depending on the model that you're using."
227 + " This is used to parse the reasoning content into OpenAI API "
228 + "format. Required for ``--enable-reasoning``.")
229
230 valid_tool_parsers = ToolParserManager.tool_parsers.keys()
231 parser.add_argument(
@@ -267,6 +285,18 @@ def validate_parsed_serve_args(args: argparse.Namespace):
285 raise TypeError("Error: --enable-auto-tool-choice requires "
286 "--tool-call-parser")
287
288 + # Enable reasoning needs a reasoning parser to be valid
289 + if args.enable_reasoning and not args.reasoning_parser:
290 + raise TypeError("Error: --enable-reasoning requires "
291 + "--reasoning-parser")
292 +
293 + # Ref https://api-docs.deepseek.com/guides/reasoning_model
294 + # tool call and reasoning cannot be enabled at the same time.
295 + if args.enable_auto_tool_choice and args.enable_reasoning:
296 + raise TypeError(
297 + "Error: --enable-auto-tool-choice and "
298 + "--enable-reasoning cannot be enabled at the same time")
299 +
300
301 def create_parser_for_docs() -> FlexibleArgumentParser:
302 parser_for_docs = FlexibleArgumentParser(
vllm/entrypoints/openai/protocol.py
+2
@@ -1202,6 +1202,7 @@ class ExtractedToolCallInformation(BaseModel):
1202
1203 class ChatMessage(OpenAIBaseModel):
1204 role: str
1205 + reasoning_content: Optional[str] = None
1206 content: Optional[str] = None
1207 tool_calls: List[ToolCall] = Field(default_factory=list)
1208
@@ -1243,6 +1244,7 @@ class ChatCompletionResponse(OpenAIBaseModel):
1244 class DeltaMessage(OpenAIBaseModel):
1245 role: Optional[str] = None
1246 content: Optional[str] = None
1247 + reasoning_content: Optional[str] = None
1248 tool_calls: List[DeltaToolCall] = Field(default_factory=list)
1249
1250
vllm/entrypoints/openai/reasoning_parsers/__init__.py new
+6
@@ -0,0 +1,6 @@
1 +from .abs_reasoning_parsers import ReasoningParser, ReasoningParserManager
2 +from .deepseek_r1_reasoning_parser import DeepSeekR1ReasoningParser
3 +
4 +__all__ = [
5 + "ReasoningParser", "ReasoningParserManager", "DeepSeekR1ReasoningParser"
6 +]
vllm/entrypoints/openai/reasoning_parsers/abs_reasoning_parsers.py new
+158
@@ -0,0 +1,158 @@
1 +import os
2 +from functools import cached_property
3 +from typing import Callable, Dict, List, Optional, Sequence, Tuple, Type, Union
4 +
5 +from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
6 + DeltaMessage)
7 +from vllm.logger import init_logger
8 +from vllm.transformers_utils.tokenizer import AnyTokenizer
9 +from vllm.utils import import_from_path, is_list_of
10 +
11 +logger = init_logger(__name__)
12 +
13 +
14 +class ReasoningParser:
15 + """
16 + Abstract reasoning parser class that should not be used directly.
17 + Provided and methods should be used in derived classes.
18 +
19 + It is used to extract reasoning content from the model output.
20 + """
21 +
22 + def __init__(self, tokenizer: AnyTokenizer):
23 + self.model_tokenizer = tokenizer
24 +
25 + @cached_property
26 + def vocab(self) -> Dict[str, int]:
27 + # NOTE: Only PreTrainedTokenizerFast is guaranteed to have .vocab
28 + # whereas all tokenizers have .get_vocab()
29 + return self.model_tokenizer.get_vocab()
30 +
31 + def extract_reasoning_content(
32 + self, model_output: str, request: ChatCompletionRequest
33 + ) -> Tuple[Optional[str], Optional[str]]:
34 + """
35 + Extract reasoning content from a complete model-generated string.
36 +
37 + Used for non-streaming responses where we have the entire model response
38 + available before sending to the client.
39 +
40 + Parameters:
41 + model_output: str
42 + The model-generated string to extract reasoning content from.
43 +
44 + request: ChatCompletionRequest
45 + The request object that was used to generate the model_output.
46 +
47 + Returns:
48 + Tuple[Optional[str], Optional[str]]
49 + A tuple containing the reasoning content and the content.
50 + """
51 +
52 + raise NotImplementedError(
53 + "AbstractReasoningParser.extract_reasoning_calls "
54 + "has not been implemented!")
55 +
56 + def extract_reasoning_content_streaming(
57 + self,
58 + previous_text: str,
59 + current_text: str,
60 + delta_text: str,
61 + previous_token_ids: Sequence[int],
62 + current_token_ids: Sequence[int],
63 + delta_token_ids: Sequence[int],
64 + ) -> Union[DeltaMessage, None]:
65 + """
66 + Instance method that should be implemented for extracting reasoning
67 + from an incomplete response; for use when handling reasoning calls and
68 + streaming. Has to be an instance method because it requires state -
69 + the current tokens/diffs, but also the information about what has
70 + previously been parsed and extracted (see constructor)
71 + """
72 + raise NotImplementedError(
73 + "AbstractReasoningParser.extract_reasoning_content_streaming "
74 + "has not been implemented!")
75 +
76 +
77 +class ReasoningParserManager:
78 + reasoning_parsers: Dict[str, Type] = {}
79 +
80 + @classmethod
81 + def get_reasoning_parser(cls, name) -> Type:
82 + """
83 + Get reasoning parser by name which is registered by `register_module`.
84 +
85 + Raise a KeyError exception if the name is not registered.
86 + """
87 + if name in cls.reasoning_parsers:
88 + return cls.reasoning_parsers[name]
89 +
90 + raise KeyError(f"reasoning helper: '{name}' not found in "
91 + "reasoning_parsers")
92 +
93 + @classmethod
94 + def _register_module(cls,
95 + module: Type,
96 + module_name: Optional[Union[str, List[str]]] = None,
97 + force: bool = True) -> None:
98 + if not issubclass(module, ReasoningParser):
99 + raise TypeError("module must be subclass of ReasoningParser, "
100 + f"but got {type(module)}")
101 + if module_name is None:
102 + module_name = module.__name__
103 + if isinstance(module_name, str):
104 + module_name = [module_name]
105 + for name in module_name:
106 + if not force and name in cls.reasoning_parsers:
107 + existed_module = cls.reasoning_parsers[name]
108 + raise KeyError(f"{name} is already registered "
109 + f"at {existed_module.__module__}")
110 + cls.reasoning_parsers[name] = module
111 +
112 + @classmethod
113 + def register_module(
114 + cls,
115 + name: Optional[Union[str, List[str]]] = None,
116 + force: bool = True,
117 + module: Union[Type, None] = None) -> Union[type, Callable]:
118 + """
119 + Register module with the given name or name list. it can be used as a
120 + decoder(with module as None) or normal function(with module as not
121 + None).
122 + """
123 + if not isinstance(force, bool):
124 + raise TypeError(f"force must be a boolean, but got {type(force)}")
125 +
126 + # raise the error ahead of time
127 + if not (name is None or isinstance(name, str)
128 + or is_list_of(name, str)):
129 + raise TypeError(
130 + "name must be None, an instance of str, or a sequence of str, "
131 + f"but got {type(name)}")
132 +
133 + # use it as a normal method: x.register_module(module=SomeClass)
134 + if module is not None:
135 + cls._register_module(module=module, module_name=name, force=force)
136 + return module
137 +
138 + # use it as a decorator: @x.register_module()
139 + def _register(module):
140 + cls._register_module(module=module, module_name=name, force=force)
141 + return module
142 +
143 + return _register
144 +
145 + @classmethod
146 + def import_reasoning_parser(cls, plugin_path: str) -> None:
147 + """
148 + Import a user-defined reasoning parser by the path
149 + of the reasoning parser define file.
150 + """
151 + module_name = os.path.splitext(os.path.basename(plugin_path))[0]
152 +
153 + try:
154 + import_from_path(module_name, plugin_path)
155 + except Exception:
156 + logger.exception("Failed to load module '%s' from %s.",
157 + module_name, plugin_path)
158 + return
vllm/entrypoints/openai/reasoning_parsers/deepseek_r1_reasoning_parser.py new
+133
@@ -0,0 +1,133 @@
1 +import re
2 +from typing import Optional, Sequence, Tuple, Union
3 +
4 +from transformers import PreTrainedTokenizerBase
5 +
6 +from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
7 + DeltaMessage)
8 +from vllm.entrypoints.openai.reasoning_parsers.abs_reasoning_parsers import (
9 + ReasoningParser, ReasoningParserManager)
10 +from vllm.logger import init_logger
11 +
12 +logger = init_logger(__name__)
13 +
14 +
15 +@ReasoningParserManager.register_module("deepseek_r1")
16 +class DeepSeekR1ReasoningParser(ReasoningParser):
17 + """
18 + Reasoning parser for DeepSeek R1 model.
19 +
20 + The DeepSeek R1 model uses <think>...</think> tokens to denote reasoning
21 + text. This parser extracts the reasoning content from the model output.
22 + """
23 +
24 + def __init__(self, tokenizer: PreTrainedTokenizerBase):
25 + super().__init__(tokenizer)
26 + self.think_start_token = "<think>"
27 + self.think_end_token = "</think>"
28 +
29 + self.reasoning_regex = re.compile(
30 + rf"{self.think_start_token}(.*?){self.think_end_token}", re.DOTALL)
31 +
32 + if not self.model_tokenizer:
33 + raise ValueError(
34 + "The model tokenizer must be passed to the ReasoningParser "
35 + "constructor during construction.")
36 +
37 + self.think_start_token_id = self.vocab.get(self.think_start_token)
38 + self.think_end_token_id = self.vocab.get(self.think_end_token)
39 + if (self.think_start_token_id is None
40 + or self.think_end_token_id is None):
41 + raise RuntimeError(
42 + "DeepSeek R1 reasoning parser could not locate think start/end "
43 + "tokens in the tokenizer!")
44 +
45 + def extract_reasoning_content_streaming(
46 + self,
47 + previous_text: str,
48 + current_text: str,
49 + delta_text: str,
50 + previous_token_ids: Sequence[int],
51 + current_token_ids: Sequence[int],
52 + delta_token_ids: Sequence[int],
53 + ) -> Union[DeltaMessage, None]:
54 + """
55 + Extract reasoning content from a delta message.
56 + Handles streaming output where previous + delta = current.
57 + Uses token IDs for faster processing.
58 + For text <think>abc</think>xyz:
59 + - 'abc' goes to reasoning_content
60 + - 'xyz' goes to content
61 + """
62 + # Skip single special tokens
63 + if len(delta_token_ids) == 1 and (delta_token_ids[0] in [
64 + self.think_start_token_id, self.think_end_token_id
65 + ]):
66 + return None
67 +
68 + if self.think_start_token_id in previous_token_ids:
69 + if self.think_end_token_id in delta_token_ids:
70 + # <think> in previous, </think> in delta,
71 + # extract reasoning content
72 + end_index = delta_text.find(self.think_end_token)
73 + reasoning_content = delta_text[:end_index]
74 + content = delta_text[end_index + len(self.think_end_token):]
75 + return DeltaMessage(reasoning_content=reasoning_content,
76 + content=content if content else None)
77 + elif self.think_end_token_id in previous_token_ids:
78 + # <think> in previous, </think> in previous,
79 + # reasoning content continues
80 + return DeltaMessage(content=delta_text)
81 + else:
82 + # <think> in previous, no </think> in previous or delta,
83 + # reasoning content continues
84 + return DeltaMessage(reasoning_content=delta_text)
85 + elif self.think_start_token_id in delta_token_ids:
86 + logger.info(delta_text)
87 + if self.think_end_token_id in delta_token_ids:
88 + # <think> in delta, </think> in delta, extract reasoning content
89 + start_index = delta_text.find(self.think_start_token)
90 + end_index = delta_text.find(self.think_end_token)
91 + reasoning_content = delta_text[start_index +
92 + len(self.think_start_token
93 + ):end_index]
94 + content = delta_text[end_index + len(self.think_end_token):]
95 + return DeltaMessage(reasoning_content=reasoning_content,
96 + content=content if content else None)
97 + else:
98 + # <think> in delta, no </think> in delta,
99 + # reasoning content continues
100 + return DeltaMessage(reasoning_content=delta_text)
101 + else:
102 + # No <think> in previous or delta, reasoning content continues.
103 + return DeltaMessage(content=delta_text)
104 +
105 + def extract_reasoning_content(
106 + self, model_output: str, request: ChatCompletionRequest
107 + ) -> Tuple[Optional[str], Optional[str]]:
108 +
109 + # Check if the model output contains the <think> tokens.
110 + if (self.think_start_token not in model_output
111 + or self.think_end_token not in model_output):
112 + return None, model_output
113 + else:
114 + # Use a regex to find the reasoning content
115 + reasoning_content = self.reasoning_regex.findall(model_output)[0]
116 +
117 + # Remove the reasoning content from the model output
118 + # Although deepseek's <think> token is always at the
119 + # beginning of the line, we cannot guarantee that the
120 + # other models will follow this convention.
121 + # Therefore, we need to add :start_index.
122 + start_index = model_output.find(self.think_start_token)
123 + if start_index != -1:
124 + end_index = start_index + len(
125 + f"{self.think_start_token}{reasoning_content}{self.think_end_token}"
126 + )
127 + model_output = model_output[:start_index] + \
128 + model_output[end_index:]
129 +
130 + if len(model_output) == 0:
131 + return reasoning_content, None
132 +
133 + return reasoning_content, model_output
vllm/entrypoints/openai/serving_chat.py
+100 -5
@@ -21,6 +21,8 @@ from vllm.entrypoints.openai.protocol import (
21 ChatCompletionStreamResponse, ChatMessage, DeltaFunctionCall, DeltaMessage,
22 DeltaToolCall, ErrorResponse, FunctionCall, PromptTokenUsageInfo,
23 RequestResponseMetadata, ToolCall, UsageInfo)
24 +from vllm.entrypoints.openai.reasoning_parsers import (ReasoningParser,
25 + ReasoningParserManager)
26 from vllm.entrypoints.openai.serving_engine import OpenAIServing
27 from vllm.entrypoints.openai.serving_models import OpenAIServingModels
28 from vllm.entrypoints.openai.tool_parsers import ToolParser, ToolParserManager
@@ -47,6 +49,8 @@ class OpenAIServingChat(OpenAIServing):
49 chat_template: Optional[str],
50 chat_template_content_format: ChatTemplateContentFormatOption,
51 return_tokens_as_token_ids: bool = False,
52 + enable_reasoning: bool = False,
53 + reasoning_parser: Optional[str] = None,
54 enable_auto_tools: bool = False,
55 tool_parser: Optional[str] = None,
56 enable_prompt_tokens_details: bool = False,
@@ -69,6 +73,18 @@ class OpenAIServingChat(OpenAIServing):
73 " the parallel_tool_calls client option is preset for "
74 "compatibility reasons, it will be ignored.")
75
76 + self.enable_reasoning: bool = enable_reasoning
77 + self.reasoning_parser: Optional[Callable[[AnyTokenizer],
78 + ReasoningParser]] = None
79 + if self.enable_reasoning:
80 + try:
81 + self.reasoning_parser = (
82 + ReasoningParserManager.get_reasoning_parser(
83 + reasoning_parser))
84 + except Exception as e:
85 + raise TypeError("Error: --enable-reasoning requires "
86 + f"reasoning_parser:'{reasoning_parser}' "
87 + "which has not been registered") from e
88 self.tool_parser: Optional[Callable[[AnyTokenizer], ToolParser]] = None
89 if self.enable_auto_tools:
90 try:
@@ -285,14 +301,35 @@ class OpenAIServingChat(OpenAIServing):
301 not tool_choice_function_name
302 and self._should_stream_with_auto_tool_parsing(request))
303
304 + should_stream_with_reasoning_parsing = (
305 + self._should_stream_with_reasoning_parsing(request))
306 +
307 all_previous_token_ids: Optional[List[List[int]]]
289 - if tool_choice_auto:
308 +
309 + # Only one of these will be used, thus previous_texts and
310 + # all_previous_token_ids will not be used twice in the same iteration.
311 + if tool_choice_auto or should_stream_with_reasoning_parsing:
312 # These are only required in "auto" tool choice case
313 previous_texts = [""] * num_choices
314 all_previous_token_ids = [[]] * num_choices
315 else:
316 previous_texts, all_previous_token_ids = None, None
317
318 + try:
319 + # There is no need to check if the reasoning_parser is None
320 + # because the should_stream_with_reasoning_parsing check
321 + # already ensures that the reasoning_parser is not None.
322 + # but the pre-commit hook requires it.
323 + if should_stream_with_reasoning_parsing and \
324 + self.reasoning_parser is not None:
325 + reasoning_parser = self.reasoning_parser(tokenizer)
326 + except RuntimeError as e:
327 + logger.exception("Error in reasoning parser creation.")
328 + data = self.create_streaming_error_response(str(e))
329 + yield f"data: {data}\n\n"
330 + yield "data: [DONE]\n\n"
331 + return
332 +
333 # Prepare the tool parser if it's needed
334 try:
335 if tool_choice_auto and self.tool_parser:
@@ -456,6 +493,32 @@ class OpenAIServingChat(OpenAIServing):
493 # update the previous values for the next iteration
494 previous_texts[i] = current_text
495 all_previous_token_ids[i] = current_token_ids
496 + # reasoning_content cannot be enabled with tool_choice.
497 + # If it is, the tool_choice will be used instead.
498 + elif self.enable_reasoning:
499 + # handle reasoning_content delta
500 + assert reasoning_parser is not None
501 + assert previous_texts is not None
502 + assert all_previous_token_ids is not None
503 + previous_text = previous_texts[i]
504 + previous_token_ids = all_previous_token_ids[i]
505 + current_text = previous_text + delta_text
506 + current_token_ids = previous_token_ids + list(
507 + output.token_ids)
508 +
509 + delta_message = (reasoning_parser.
510 + extract_reasoning_content_streaming(
511 + previous_text,
512 + current_text,
513 + delta_text,
514 + previous_token_ids,
515 + current_token_ids,
516 + output.token_ids,
517 + ))
518 +
519 + # update the previous values for the next iteration
520 + previous_texts[i] = current_text
521 + all_previous_token_ids[i] = current_token_ids
522
523 # handle streaming just a content delta
524 else:
@@ -642,17 +705,38 @@ class OpenAIServingChat(OpenAIServing):
705 else:
706 logprobs = None
707
708 + should_stream_with_reasoning_parsing = (
709 + self._should_stream_with_reasoning_parsing(request))
710 +
711 # In the OpenAI API the finish_reason is "tools_called"
712 # if the tool choice is auto and the model produced a tool
713 # call. The same is not true for named function calls
714 auto_tools_called = False
715
716 + if should_stream_with_reasoning_parsing and \
717 + self.reasoning_parser is not None:
718 + try:
719 + reasoning_parser = self.reasoning_parser(tokenizer)
720 + except RuntimeError as e:
721 + logger.exception("Error in reasoning parser creation.")
722 + return self.create_error_response(str(e))
723 +
724 + reasoning_content, content = (
725 + reasoning_parser.extract_reasoning_content(
726 + output.text, request=request))
727 +
728 + if reasoning_content:
729 + message = ChatMessage(role=role,
730 + content=content,
731 + reasoning_content=reasoning_content)
732 + else:
733 + message = ChatMessage(role=role, content=output.text)
734 +
735 # if auto tools are not enabled, and a named tool choice using
736 # outlines is not being used
652 - if (not self.enable_auto_tools
653 - or not self.tool_parser) and not isinstance(
654 - request.tool_choice,
655 - ChatCompletionNamedToolChoiceParam):
737 + elif (not self.enable_auto_tools
738 + or not self.tool_parser) and not isinstance(
739 + request.tool_choice, ChatCompletionNamedToolChoiceParam):
740 message = ChatMessage(role=role, content=output.text)
741
742 # if the request uses tools and specified a tool choice
@@ -835,6 +919,17 @@ class OpenAIServingChat(OpenAIServing):
919 return (request.tools and self.tool_parser and self.enable_auto_tools
920 and request.tool_choice in ['auto', None])
921
922 + def _should_stream_with_reasoning_parsing(self,
923 + request: ChatCompletionRequest):
924 + """
925 + Utility function to check if streamed tokens should go through the
926 + reasoning parser that was configured.
927 +
928 + We only want to do this IF reasoning is enabled and a reasoning
929 + parser is configured.
930 + """
931 + return self.enable_reasoning and self.reasoning_parser is not None
932 +
933 def _should_check_for_unstreamed_tool_arg_tokens(
934 self,
935 delta_message: Optional[DeltaMessage],
vllm/scripts.py
+1
@@ -167,6 +167,7 @@ def main():
167 "Must be a YAML with the following options:"
168 "https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#cli-reference"
169 )
170 +
171 serve_parser = make_arg_parser(serve_parser)
172 serve_parser.set_defaults(dispatch_function=serve)
173