@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 66785cc05c
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 06:45 UTC

Support chat template and `echo` for chat API (#1756)

Adam Brusselback committed Nov 30, 2023 at 19:43 UTC 66785cc05c05c7f19f319533c23d1998b9d80bf9
7 files changed +439 -180
docs/source/getting_started/quickstart.rst
+53 -1
@@ -107,6 +107,7 @@ OpenAI-Compatible Server
107 ------------------------
108
109 vLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API.
110 +By default, it starts the server at ``http://localhost:8000``. You can specify the address with ``--host`` and ``--port`` arguments. The server currently hosts one model at a time (OPT-125M in the above command) and implements `list models <https://platform.openai.com/docs/api-reference/models/list>`_, `create chat completion <https://platform.openai.com/docs/api-reference/chat/completions/create>`_, and `create completion <https://platform.openai.com/docs/api-reference/completions/create>`_ endpoints. We are actively adding support for more endpoints.
111
112 Start the server:
113
@@ -122,7 +123,13 @@ Use model from www.modelscope.cn
123 $ VLLM_USE_MODELSCOPE=True python -m vllm.entrypoints.openai.api_server \
124 $ --model="qwen/Qwen-7B-Chat" --revision="v1.1.8" --trust-remote-code
125
125 -By default, it starts the server at ``http://localhost:8000``. You can specify the address with ``--host`` and ``--port`` arguments. The server currently hosts one model at a time (OPT-125M in the above command) and implements `list models <https://platform.openai.com/docs/api-reference/models/list>`_ and `create completion <https://platform.openai.com/docs/api-reference/completions/create>`_ endpoints. We are actively adding support for more endpoints.
126 +By default, the server uses a predefined chat template stored in the tokenizer. You can override this template by using the ``--chat-template`` argument:
127 +
128 +.. code-block:: console
129 +
130 + $ python -m vllm.entrypoints.openai.api_server \
131 + $ --model facebook/opt-125m \
132 + $ --chat-template ./examples/template_chatml.json
133
134 This server can be queried in the same format as OpenAI API. For example, list the models:
135
@@ -130,6 +137,9 @@ This server can be queried in the same format as OpenAI API. For example, list t
137
138 $ curl http://localhost:8000/v1/models
139
140 +Using OpenAI Completions API with vLLM
141 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
142 +
143 Query the model with input prompts:
144
145 .. code-block:: console
@@ -156,3 +166,45 @@ Since this server is compatible with OpenAI API, you can use it as a drop-in rep
166 print("Completion result:", completion)
167
168 For a more detailed client example, refer to `examples/openai_completion_client.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py>`_.
169 +
170 +Using OpenAI Chat API with vLLM
171 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
172 +
173 +The vLLM server is designed to support the OpenAI Chat API, allowing you to engage in dynamic conversations with the model. The chat interface is a more interactive way to communicate with the model, allowing back-and-forth exchanges that can be stored in the chat history. This is useful for tasks that require context or more detailed explanations.
174 +
175 +Querying the model using OpenAI Chat API:
176 +
177 +You can use the `create chat completion <https://platform.openai.com/docs/api-reference/chat/completions/create>`_ endpoint to communicate with the model in a chat-like interface:
178 +
179 +.. code-block:: console
180 +
181 + $ curl http://localhost:8000/v1/chat/completions \
182 + $ -H "Content-Type: application/json" \
183 + $ -d '{
184 + $ "model": "facebook/opt-125m",
185 + $ "messages": [
186 + $ {"role": "system", "content": "You are a helpful assistant."},
187 + $ {"role": "user", "content": "Who won the world series in 2020?"}
188 + $ ]
189 + $ }'
190 +
191 +Python Client Example:
192 +
193 +Using the `openai` python package, you can also communicate with the model in a chat-like manner:
194 +
195 +.. code-block:: python
196 +
197 + import openai
198 + # Set OpenAI's API key and API base to use vLLM's API server.
199 + openai.api_key = "EMPTY"
200 + openai.api_base = "http://localhost:8000/v1"
201 + chat_response = openai.ChatCompletion.create(
202 + model="facebook/opt-125m",
203 + messages=[
204 + {"role": "system", "content": "You are a helpful assistant."},
205 + {"role": "user", "content": "Tell me a joke."},
206 + ]
207 + )
208 + print("Chat response:", chat_response)
209 +
210 +For more in-depth examples and advanced features of the chat API, you can refer to the official OpenAI documentation.
examples/template_alpaca.jinja new
+29
@@ -0,0 +1,29 @@
1 +{{ (messages|selectattr('role', 'equalto', 'system')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'system')|list) else '' }}
2 +
3 +{% for message in messages %}
4 +{% if message['role'] == 'user' %}
5 +### Instruction:
6 +{{ message['content']|trim -}}
7 +{% if not loop.last %}
8 +
9 +
10 +{% endif %}
11 +{% elif message['role'] == 'assistant' %}
12 +### Response:
13 +{{ message['content']|trim -}}
14 +{% if not loop.last %}
15 +
16 +
17 +{% endif %}
18 +{% elif message['role'] == 'user_context' %}
19 +### Input:
20 +{{ message['content']|trim -}}
21 +{% if not loop.last %}
22 +
23 +
24 +{% endif %}
25 +{% endif %}
26 +{% endfor %}
27 +{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}
28 +### Response:
29 +{% endif %}
\ No newline at end of file
examples/template_chatml.jinja new
+2
@@ -0,0 +1,2 @@
1 +{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
2 +{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\n' }}{% endif %}
\ No newline at end of file
examples/template_inkbot.jinja new
+30
@@ -0,0 +1,30 @@
1 +<#meta#>
2 +- Date: {{ (messages|selectattr('role', 'equalto', 'meta-current_date')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'meta-current_date')|list) else '' }}
3 +- Task: {{ (messages|selectattr('role', 'equalto', 'meta-task_name')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'meta-task_name')|list) else '' }}
4 +<#system#>
5 +{{ (messages|selectattr('role', 'equalto', 'system')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'system')|list) else '' }}
6 +<#chat#>
7 +{% for message in messages %}
8 +{% if message['role'] == 'user' %}
9 +<#user#>
10 +{{ message['content']|trim -}}
11 +{% if not loop.last %}
12 +
13 +{% endif %}
14 +{% elif message['role'] == 'assistant' %}
15 +<#bot#>
16 +{{ message['content']|trim -}}
17 +{% if not loop.last %}
18 +
19 +{% endif %}
20 +{% elif message['role'] == 'user_context' %}
21 +<#user_context#>
22 +{{ message['content']|trim -}}
23 +{% if not loop.last %}
24 +
25 +{% endif %}
26 +{% endif %}
27 +{% endfor %}
28 +{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}
29 +<#bot#>
30 +{% endif %}
\ No newline at end of file
tests/async_engine/test_openai_server.py new
+119
@@ -0,0 +1,119 @@
1 +from argparse import Namespace
2 +from dataclasses import dataclass
3 +
4 +import pytest
5 +from fastapi.testclient import TestClient
6 +
7 +from vllm.entrypoints.openai.api_server import *
8 +
9 +# Define models, templates, and their corresponding expected outputs
10 +MODEL_TEMPLATE_GENERATON_OUTPUT = [
11 + ("facebook/opt-125m", None, True,
12 + "Hello</s>Hi there!</s>What is the capital of</s>"),
13 + ("facebook/opt-125m", None, False,
14 + "Hello</s>Hi there!</s>What is the capital of</s>"),
15 + ("facebook/opt-125m", "../../examples/template_chatml.jinja", True,
16 + """<|im_start|>user
17 +Hello<|im_end|>
18 +<|im_start|>assistant
19 +Hi there!<|im_end|>
20 +<|im_start|>user
21 +What is the capital of<|im_end|>
22 +<|im_start|>assistant
23 +"""),
24 + ("facebook/opt-125m", "../../examples/template_chatml.jinja", False,
25 + """<|im_start|>user
26 +Hello<|im_end|>
27 +<|im_start|>assistant
28 +Hi there!<|im_end|>
29 +<|im_start|>user
30 +What is the capital of""")
31 +]
32 +
33 +TEST_MESSAGES = [
34 + {
35 + 'role': 'user',
36 + 'content': 'Hello'
37 + },
38 + {
39 + 'role': 'assistant',
40 + 'content': 'Hi there!'
41 + },
42 + {
43 + 'role': 'user',
44 + 'content': 'What is the capital of'
45 + },
46 +]
47 +client = TestClient(app)
48 +
49 +
50 +@dataclass
51 +class MockTokenizer:
52 + chat_template = None
53 +
54 +
55 +def test_load_chat_template():
56 + # Testing chatml template
57 + template = "../../examples/template_chatml.jinja"
58 + mock_args = Namespace(chat_template=template)
59 + tokenizer = MockTokenizer()
60 +
61 + # Call the function with the mocked args
62 + load_chat_template(mock_args, tokenizer)
63 +
64 + template_content = tokenizer.chat_template
65 +
66 + # Test assertions
67 + assert template_content is not None
68 + # Hard coded value for template_chatml.jinja
69 + assert template_content == """{% for message in messages %}{{'<|im_start|>' + message['role'] + '\\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\\n'}}{% endif %}{% endfor %}
70 +{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\\n' }}{% endif %}"""
71 +
72 +
73 +def test_no_load_chat_template():
74 + # Testing chatml template
75 + template = "../../examples/does_not_exist"
76 + mock_args = Namespace(chat_template=template)
77 + tokenizer = MockTokenizer()
78 +
79 + # Call the function with the mocked args
80 + load_chat_template(mock_args, tokenizer=tokenizer)
81 + template_content = tokenizer.chat_template
82 +
83 + # Test assertions
84 + assert template_content is not None
85 + # Hard coded value for template_chatml.jinja
86 + assert template_content == """../../examples/does_not_exist"""
87 +
88 +
89 +@pytest.mark.asyncio
90 +@pytest.mark.parametrize(
91 + "model,template,add_generation_prompt,expected_output",
92 + MODEL_TEMPLATE_GENERATON_OUTPUT)
93 +async def test_get_gen_prompt(model, template, add_generation_prompt,
94 + expected_output):
95 + # Initialize the tokenizer
96 + tokenizer = get_tokenizer(tokenizer_name=model)
97 +
98 + mock_args = Namespace(chat_template=template)
99 + load_chat_template(mock_args, tokenizer)
100 +
101 + # Create a mock request object using keyword arguments
102 + mock_request = ChatCompletionRequest(
103 + model=model,
104 + messages=TEST_MESSAGES,
105 + add_generation_prompt=add_generation_prompt)
106 +
107 + # Call the function and get the result
108 + result = tokenizer.apply_chat_template(
109 + conversation=mock_request.messages,
110 + tokenize=False,
111 + add_generation_prompt=mock_request.add_generation_prompt)
112 +
113 + # Test assertion
114 + assert result == expected_output, f"The generated prompt does not match the expected output for model {model} and template {template}"
115 +
116 +
117 +def test_health_endpoint():
118 + response = client.get("/health")
119 + assert response.status_code == 200
vllm/entrypoints/openai/api_server.py
+204 -179
@@ -3,6 +3,7 @@
3
4 import argparse
5 import asyncio
6 +import codecs
7 import json
8 import time
9 from http import HTTPStatus
@@ -14,7 +15,6 @@ from fastapi import Request
15 from fastapi.exceptions import RequestValidationError
16 from fastapi.middleware.cors import CORSMiddleware
17 from fastapi.responses import JSONResponse, StreamingResponse, Response
17 -from packaging import version
18
19 from vllm.engine.arg_utils import AsyncEngineArgs
20 from vllm.engine.async_llm_engine import AsyncLLMEngine
@@ -31,20 +31,55 @@ from vllm.sampling_params import SamplingParams
31 from vllm.transformers_utils.tokenizer import get_tokenizer
32 from vllm.utils import random_uuid
33
34 -try:
35 - import fastchat
36 - from fastchat.conversation import Conversation, SeparatorStyle
37 - from fastchat.model.model_adapter import get_conversation_template
38 - _fastchat_available = True
39 -except ImportError:
40 - _fastchat_available = False
41 -
34 TIMEOUT_KEEP_ALIVE = 5 # seconds
35
36 logger = init_logger(__name__)
37 served_model = None
38 app = fastapi.FastAPI()
39 engine = None
40 +response_role = None
41 +
42 +
43 +def parse_args():
44 + parser = argparse.ArgumentParser(
45 + description="vLLM OpenAI-Compatible RESTful API server.")
46 + parser.add_argument("--host", type=str, default=None, help="host name")
47 + parser.add_argument("--port", type=int, default=8000, help="port number")
48 + parser.add_argument("--allow-credentials",
49 + action="store_true",
50 + help="allow credentials")
51 + parser.add_argument("--allowed-origins",
52 + type=json.loads,
53 + default=["*"],
54 + help="allowed origins")
55 + parser.add_argument("--allowed-methods",
56 + type=json.loads,
57 + default=["*"],
58 + help="allowed methods")
59 + parser.add_argument("--allowed-headers",
60 + type=json.loads,
61 + default=["*"],
62 + help="allowed headers")
63 + parser.add_argument("--served-model-name",
64 + type=str,
65 + default=None,
66 + help="The model name used in the API. If not "
67 + "specified, the model name will be the same as "
68 + "the huggingface name.")
69 + parser.add_argument("--chat-template",
70 + type=str,
71 + default=None,
72 + help="The file path to the chat template, "
73 + "or the template in single-line form "
74 + "for the specified model")
75 + parser.add_argument("--response-role",
76 + type=str,
77 + default="assistant",
78 + help="The role name to return if "
79 + "`request.add_generation_prompt=true`.")
80 +
81 + parser = AsyncEngineArgs.add_cli_args(parser)
82 + return parser.parse_args()
83
84
85 def create_error_response(status_code: HTTPStatus,
@@ -54,6 +89,25 @@ def create_error_response(status_code: HTTPStatus,
89 status_code=status_code.value)
90
91
92 +def load_chat_template(args, tokenizer):
93 + if args.chat_template is not None:
94 + try:
95 + with open(args.chat_template, "r") as f:
96 + chat_template = f.read()
97 + except OSError:
98 + # If opening a file fails, set chat template to be args to
99 + # ensure we decode so our escape are interpreted correctly
100 + chat_template = codecs.decode(args.chat_template, "unicode_escape")
101 +
102 + tokenizer.chat_template = chat_template
103 + logger.info(
104 + f"Using supplied chat template:\n{tokenizer.chat_template}")
105 + elif tokenizer.chat_template is not None:
106 + logger.info(f"Using default chat template:\n{tokenizer.chat_template}")
107 + else:
108 + logger.warning("No chat template provided. Chat API will not work.")
109 +
110 +
111 @app.exception_handler(RequestValidationError)
112 async def validation_exception_handler(_, exc):
113 return create_error_response(HTTPStatus.BAD_REQUEST, str(exc))
@@ -69,53 +123,6 @@ async def check_model(request) -> Optional[JSONResponse]:
123 return ret
124
125
72 -async def get_gen_prompt(request) -> str:
73 - if not _fastchat_available:
74 - raise ModuleNotFoundError(
75 - "fastchat is not installed. Please install fastchat to use "
76 - "the chat completion and conversation APIs: `$ pip install fschat`"
77 - )
78 - if version.parse(fastchat.__version__) < version.parse("0.2.23"):
79 - raise ImportError(
80 - f"fastchat version is low. Current version: {fastchat.__version__} "
81 - "Please upgrade fastchat to use: `$ pip install -U fschat`")
82 -
83 - conv = get_conversation_template(request.model)
84 - conv = Conversation(
85 - name=conv.name,
86 - system_template=conv.system_template,
87 - system_message=conv.system_message,
88 - roles=conv.roles,
89 - messages=list(conv.messages), # prevent in-place modification
90 - offset=conv.offset,
91 - sep_style=SeparatorStyle(conv.sep_style),
92 - sep=conv.sep,
93 - sep2=conv.sep2,
94 - stop_str=conv.stop_str,
95 - stop_token_ids=conv.stop_token_ids,
96 - )
97 -
98 - if isinstance(request.messages, str):
99 - prompt = request.messages
100 - else:
101 - for message in request.messages:
102 - msg_role = message["role"]
103 - if msg_role == "system":
104 - conv.system_message = message["content"]
105 - elif msg_role == "user":
106 - conv.append_message(conv.roles[0], message["content"])
107 - elif msg_role == "assistant":
108 - conv.append_message(conv.roles[1], message["content"])
109 - else:
110 - raise ValueError(f"Unknown role: {msg_role}")
111 -
112 - # Add a blank message for the assistant.
113 - conv.append_message(conv.roles[1], None)
114 - prompt = conv.get_prompt()
115 -
116 - return prompt
117 -
118 -
126 async def check_length(
127 request: Union[ChatCompletionRequest, CompletionRequest],
128 prompt: Optional[str] = None,
@@ -207,7 +214,6 @@ async def create_chat_completion(request: ChatCompletionRequest,
214 - function_call (Users should implement this by themselves)
215 - logit_bias (to be supported by vLLM engine)
216 """
210 -
217 error_check_ret = await check_model(request)
218 if error_check_ret is not None:
219 return error_check_ret
@@ -217,7 +223,15 @@ async def create_chat_completion(request: ChatCompletionRequest,
223 return create_error_response(HTTPStatus.BAD_REQUEST,
224 "logit_bias is not currently supported")
225
220 - prompt = await get_gen_prompt(request)
226 + try:
227 + prompt = tokenizer.apply_chat_template(
228 + conversation=request.messages,
229 + tokenize=False,
230 + add_generation_prompt=request.add_generation_prompt)
231 + except Exception as e:
232 + logger.error(f"Error in applying chat template from request: {str(e)}")
233 + return create_error_response(HTTPStatus.BAD_REQUEST, str(e))
234 +
235 token_ids, error_check_ret = await check_length(request, prompt=prompt)
236 if error_check_ret is not None:
237 return error_check_ret
@@ -225,6 +239,7 @@ async def create_chat_completion(request: ChatCompletionRequest,
239 model_name = request.model
240 request_id = f"cmpl-{random_uuid()}"
241 created_time = int(time.monotonic())
242 + chunk_object_type = "chat.completion.chunk"
243 try:
244 spaces_between_special_tokens = request.spaces_between_special_tokens
245 sampling_params = SamplingParams(
@@ -249,128 +264,162 @@ async def create_chat_completion(request: ChatCompletionRequest,
264 result_generator = engine.generate(prompt, sampling_params, request_id,
265 token_ids)
266
252 - def create_stream_response_json(
253 - index: int,
254 - text: str,
255 - finish_reason: Optional[str] = None,
256 - usage: Optional[UsageInfo] = None,
257 - ) -> str:
258 - choice_data = ChatCompletionResponseStreamChoice(
259 - index=index,
260 - delta=DeltaMessage(content=text),
261 - finish_reason=finish_reason,
262 - )
263 - response = ChatCompletionStreamResponse(
264 - id=request_id,
265 - created=created_time,
266 - model=model_name,
267 - choices=[choice_data],
268 - )
269 - if usage is not None:
270 - response.usage = usage
271 - # exclude unset to leave details out of each sse
272 - response_json = response.json(exclude_unset=True, ensure_ascii=False)
273 -
274 - return response_json
267 + def get_role() -> str:
268 + if request.add_generation_prompt:
269 + return response_role
270 + else:
271 + return request.messages[-1]["role"]
272
273 async def completion_stream_generator() -> AsyncGenerator[str, None]:
277 - # First chunk with role
274 + # Send first response for each request.n (index) with the role
275 + role = get_role()
276 for i in range(request.n):
277 choice_data = ChatCompletionResponseStreamChoice(
280 - index=i,
281 - delta=DeltaMessage(role="assistant"),
282 - finish_reason=None,
283 - )
278 + index=i, delta=DeltaMessage(role=role), finish_reason=None)
279 chunk = ChatCompletionStreamResponse(id=request_id,
280 + object=chunk_object_type,
281 + created=created_time,
282 choices=[choice_data],
283 model=model_name)
284 data = chunk.json(exclude_unset=True, ensure_ascii=False)
285 yield f"data: {data}\n\n"
286
287 + # Send response to echo the input portion of the last message
288 + if request.echo:
289 + last_msg_content = ""
290 + if request.messages and isinstance(
291 + request.messages, list) and request.messages[-1].get(
292 + "content") and request.messages[-1].get(
293 + "role") == role:
294 + last_msg_content = request.messages[-1]["content"]
295 + if last_msg_content:
296 + for i in range(request.n):
297 + choice_data = ChatCompletionResponseStreamChoice(
298 + index=i,
299 + delta=DeltaMessage(content=last_msg_content),
300 + finish_reason=None)
301 + chunk = ChatCompletionStreamResponse(
302 + id=request_id,
303 + object=chunk_object_type,
304 + created=created_time,
305 + choices=[choice_data],
306 + model=model_name)
307 + data = chunk.json(exclude_unset=True, ensure_ascii=False)
308 + yield f"data: {data}\n\n"
309 +
310 + # Send response for each token for each request.n (index)
311 previous_texts = [""] * request.n
312 previous_num_tokens = [0] * request.n
313 + finish_reason_sent = [False] * request.n
314 async for res in result_generator:
315 res: RequestOutput
316 for output in res.outputs:
317 i = output.index
296 - delta_text = output.text[len(previous_texts[i]):]
297 - previous_texts[i] = output.text
298 - completion_tokens = len(output.token_ids)
299 - previous_num_tokens[i] = completion_tokens
300 - response_json = create_stream_response_json(
301 - index=i,
302 - text=delta_text,
303 - )
304 - yield f"data: {response_json}\n\n"
305 - if output.finish_reason is not None:
318 +
319 + if finish_reason_sent[i]:
320 + continue
321 +
322 + if output.finish_reason is None:
323 + # Send token-by-token response for each request.n
324 + delta_text = output.text[len(previous_texts[i]):]
325 + previous_texts[i] = output.text
326 + completion_tokens = len(output.token_ids)
327 + previous_num_tokens[i] = completion_tokens
328 + choice_data = ChatCompletionResponseStreamChoice(
329 + index=i,
330 + delta=DeltaMessage(content=delta_text),
331 + finish_reason=None)
332 + chunk = ChatCompletionStreamResponse(
333 + id=request_id,
334 + object=chunk_object_type,
335 + created=created_time,
336 + choices=[choice_data],
337 + model=model_name)
338 + data = chunk.json(exclude_unset=True, ensure_ascii=False)
339 + yield f"data: {data}\n\n"
340 + else:
341 + # Send the finish response for each request.n only once
342 prompt_tokens = len(res.prompt_token_ids)
343 final_usage = UsageInfo(
344 prompt_tokens=prompt_tokens,
345 completion_tokens=completion_tokens,
346 total_tokens=prompt_tokens + completion_tokens,
347 )
312 - response_json = create_stream_response_json(
313 - index=i,
314 - text="",
315 - finish_reason=output.finish_reason,
316 - usage=final_usage,
317 - )
318 - yield f"data: {response_json}\n\n"
348 + choice_data = ChatCompletionResponseStreamChoice(
349 + index=i, delta=[], finish_reason=output.finish_reason)
350 + chunk = ChatCompletionStreamResponse(
351 + id=request_id,
352 + object=chunk_object_type,
353 + created=created_time,
354 + choices=[choice_data],
355 + model=model_name)
356 + if final_usage is not None:
357 + chunk.usage = final_usage
358 + data = chunk.json(exclude_unset=True,
359 + exclude_none=True,
360 + ensure_ascii=False)
361 + yield f"data: {data}\n\n"
362 + finish_reason_sent[i] = True
363 + # Send the final done message after all response.n are finished
364 yield "data: [DONE]\n\n"
365
321 - # Streaming response
322 - if request.stream:
323 - return StreamingResponse(completion_stream_generator(),
324 - media_type="text/event-stream")
325 -
326 - # Non-streaming response
327 - final_res: RequestOutput = None
328 - async for res in result_generator:
329 - if await raw_request.is_disconnected():
330 - # Abort the request if the client disconnects.
331 - await engine.abort(request_id)
332 - return create_error_response(HTTPStatus.BAD_REQUEST,
333 - "Client disconnected")
334 - final_res = res
335 - assert final_res is not None
336 - choices = []
337 - for output in final_res.outputs:
338 - choice_data = ChatCompletionResponseChoice(
339 - index=output.index,
340 - message=ChatMessage(role="assistant", content=output.text),
341 - finish_reason=output.finish_reason,
366 + async def completion_full_generator():
367 + final_res: RequestOutput = None
368 + async for res in result_generator:
369 + if await raw_request.is_disconnected():
370 + # Abort the request if the client disconnects.
371 + await engine.abort(request_id)
372 + return create_error_response(HTTPStatus.BAD_REQUEST,
373 + "Client disconnected")
374 + final_res = res
375 + assert final_res is not None
376 +
377 + choices = []
378 + role = get_role()
379 + for output in final_res.outputs:
380 + choice_data = ChatCompletionResponseChoice(
381 + index=output.index,
382 + message=ChatMessage(role=role, content=output.text),
383 + finish_reason=output.finish_reason,
384 + )
385 + choices.append(choice_data)
386 +
387 + if request.echo:
388 + last_msg_content = ""
389 + if request.messages and isinstance(
390 + request.messages, list) and request.messages[-1].get(
391 + "content") and request.messages[-1].get(
392 + "role") == role:
393 + last_msg_content = request.messages[-1]["content"]
394 +
395 + for choice in choices:
396 + full_message = last_msg_content + choice.message.content
397 + choice.message.content = full_message
398 +
399 + num_prompt_tokens = len(final_res.prompt_token_ids)
400 + num_generated_tokens = sum(
401 + len(output.token_ids) for output in final_res.outputs)
402 + usage = UsageInfo(
403 + prompt_tokens=num_prompt_tokens,
404 + completion_tokens=num_generated_tokens,
405 + total_tokens=num_prompt_tokens + num_generated_tokens,
406 + )
407 + response = ChatCompletionResponse(
408 + id=request_id,
409 + created=created_time,
410 + model=model_name,
411 + choices=choices,
412 + usage=usage,
413 )
343 - choices.append(choice_data)
414
345 - num_prompt_tokens = len(final_res.prompt_token_ids)
346 - num_generated_tokens = sum(
347 - len(output.token_ids) for output in final_res.outputs)
348 - usage = UsageInfo(
349 - prompt_tokens=num_prompt_tokens,
350 - completion_tokens=num_generated_tokens,
351 - total_tokens=num_prompt_tokens + num_generated_tokens,
352 - )
353 - response = ChatCompletionResponse(
354 - id=request_id,
355 - created=created_time,
356 - model=model_name,
357 - choices=choices,
358 - usage=usage,
359 - )
415 + return response
416
417 + # Streaming response
418 if request.stream:
362 - # When user requests streaming but we don't stream, we still need to
363 - # return a streaming response with a single event.
364 - response_json = response.json(ensure_ascii=False)
365 -
366 - async def fake_stream_generator() -> AsyncGenerator[str, None]:
367 - yield f"data: {response_json}\n\n"
368 - yield "data: [DONE]\n\n"
369 -
370 - return StreamingResponse(fake_stream_generator(),
419 + return StreamingResponse(completion_stream_generator(),
420 media_type="text/event-stream")
372 -
373 - return response
421 + else:
422 + return await completion_full_generator()
423
424
425 @app.post("/v1/completions")
@@ -642,34 +691,7 @@ async def create_completion(request: CompletionRequest, raw_request: Request):
691
692
693 if __name__ == "__main__":
645 - parser = argparse.ArgumentParser(
646 - description="vLLM OpenAI-Compatible RESTful API server.")
647 - parser.add_argument("--host", type=str, default=None, help="host name")
648 - parser.add_argument("--port", type=int, default=8000, help="port number")
649 - parser.add_argument("--allow-credentials",
650 - action="store_true",
651 - help="allow credentials")
652 - parser.add_argument("--allowed-origins",
653 - type=json.loads,
654 - default=["*"],
655 - help="allowed origins")
656 - parser.add_argument("--allowed-methods",
657 - type=json.loads,
658 - default=["*"],
659 - help="allowed methods")
660 - parser.add_argument("--allowed-headers",
661 - type=json.loads,
662 - default=["*"],
663 - help="allowed headers")
664 - parser.add_argument("--served-model-name",
665 - type=str,
666 - default=None,
667 - help="The model name used in the API. If not "
668 - "specified, the model name will be the same as "
669 - "the huggingface name.")
670 -
671 - parser = AsyncEngineArgs.add_cli_args(parser)
672 - args = parser.parse_args()
694 + args = parse_args()
695
696 app.add_middleware(
697 CORSMiddleware,
@@ -686,6 +708,8 @@ if __name__ == "__main__":
708 else:
709 served_model = args.model
710
711 + response_role = args.response_role
712 +
713 engine_args = AsyncEngineArgs.from_cli_args(args)
714 engine = AsyncLLMEngine.from_engine_args(engine_args)
715 engine_model_config = asyncio.run(engine.get_model_config())
@@ -696,6 +720,7 @@ if __name__ == "__main__":
720 engine_model_config.tokenizer,
721 tokenizer_mode=engine_model_config.tokenizer_mode,
722 trust_remote_code=engine_model_config.trust_remote_code)
723 + load_chat_template(args, tokenizer)
724
725 uvicorn.run(app,
726 host=args.host,
vllm/entrypoints/openai/protocol.py
+2
@@ -73,6 +73,8 @@ class ChatCompletionRequest(BaseModel):
73 stop_token_ids: Optional[List[int]] = Field(default_factory=list)
74 skip_special_tokens: Optional[bool] = True
75 spaces_between_special_tokens: Optional[bool] = True
76 + add_generation_prompt: Optional[bool] = True
77 + echo: Optional[bool] = False
78
79
80 class CompletionRequest(BaseModel):