A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 06:45 UTC
Support chat template and `echo` for chat API (#1756)
Adam Brusselback committed
Nov 30, 2023 at 19:43 UTC
66785cc05c05c7f19f319533c23d1998b9d80bf9
7 files changed
+439
-180
docs/source/getting_started/quickstart.rst
+53
-1
@@ -107,6 +107,7 @@ OpenAI-Compatible Server
107
------------------------
108
109
vLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API.
110
+By default, it starts the server at ``http://localhost:8000``. You can specify the address with ``--host`` and ``--port`` arguments. The server currently hosts one model at a time (OPT-125M in the above command) and implements `list models <https://platform.openai.com/docs/api-reference/models/list>`_, `create chat completion <https://platform.openai.com/docs/api-reference/chat/completions/create>`_, and `create completion <https://platform.openai.com/docs/api-reference/completions/create>`_ endpoints. We are actively adding support for more endpoints.
111
112
Start the server:
113
@@ -122,7 +123,13 @@ Use model from www.modelscope.cn
123
$ VLLM_USE_MODELSCOPE=True python -m vllm.entrypoints.openai.api_server \
124
$ --model="qwen/Qwen-7B-Chat" --revision="v1.1.8" --trust-remote-code
125
125
-By default, it starts the server at ``http://localhost:8000``. You can specify the address with ``--host`` and ``--port`` arguments. The server currently hosts one model at a time (OPT-125M in the above command) and implements `list models <https://platform.openai.com/docs/api-reference/models/list>`_ and `create completion <https://platform.openai.com/docs/api-reference/completions/create>`_ endpoints. We are actively adding support for more endpoints.
126
+By default, the server uses a predefined chat template stored in the tokenizer. You can override this template by using the ``--chat-template`` argument:
127
+
128
+.. code-block:: console
129
+
130
+ $ python -m vllm.entrypoints.openai.api_server \
131
+ $ --model facebook/opt-125m \
132
+ $ --chat-template ./examples/template_chatml.json
133
134
This server can be queried in the same format as OpenAI API. For example, list the models:
135
@@ -130,6 +137,9 @@ This server can be queried in the same format as OpenAI API. For example, list t
137
138
$ curl http://localhost:8000/v1/models
139
140
+Using OpenAI Completions API with vLLM
141
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
142
+
143
Query the model with input prompts:
144
145
.. code-block:: console
@@ -156,3 +166,45 @@ Since this server is compatible with OpenAI API, you can use it as a drop-in rep
166
print("Completion result:", completion)
167
168
For a more detailed client example, refer to `examples/openai_completion_client.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py>`_.
169
+
170
+Using OpenAI Chat API with vLLM
171
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
172
+
173
+The vLLM server is designed to support the OpenAI Chat API, allowing you to engage in dynamic conversations with the model. The chat interface is a more interactive way to communicate with the model, allowing back-and-forth exchanges that can be stored in the chat history. This is useful for tasks that require context or more detailed explanations.
174
+
175
+Querying the model using OpenAI Chat API:
176
+
177
+You can use the `create chat completion <https://platform.openai.com/docs/api-reference/chat/completions/create>`_ endpoint to communicate with the model in a chat-like interface:
178
+
179
+.. code-block:: console
180
+
181
+ $ curl http://localhost:8000/v1/chat/completions \
182
+ $ -H "Content-Type: application/json" \
183
+ $ -d '{
184
+ $ "model": "facebook/opt-125m",
185
+ $ "messages": [
186
+ $ {"role": "system", "content": "You are a helpful assistant."},
187
+ $ {"role": "user", "content": "Who won the world series in 2020?"}
188
+ $ ]
189
+ $ }'
190
+
191
+Python Client Example:
192
+
193
+Using the `openai` python package, you can also communicate with the model in a chat-like manner:
194
+
195
+.. code-block:: python
196
+
197
+ import openai
198
+ # Set OpenAI's API key and API base to use vLLM's API server.
199
+ openai.api_key = "EMPTY"
200
+ openai.api_base = "http://localhost:8000/v1"
201
+ chat_response = openai.ChatCompletion.create(
202
+ model="facebook/opt-125m",
203
+ messages=[
204
+ {"role": "system", "content": "You are a helpful assistant."},
205
+ {"role": "user", "content": "Tell me a joke."},
206
+ ]
207
+ )
208
+ print("Chat response:", chat_response)
209
+
210
+For more in-depth examples and advanced features of the chat API, you can refer to the official OpenAI documentation.
examples/template_alpaca.jinja
new
+29
@@ -0,0 +1,29 @@
1
+{{ (messages|selectattr('role', 'equalto', 'system')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'system')|list) else '' }}
2
+
3
+{% for message in messages %}
4
+{% if message['role'] == 'user' %}
5
+### Instruction:
6
+{{ message['content']|trim -}}
7
+{% if not loop.last %}
8
+
9
+
10
+{% endif %}
11
+{% elif message['role'] == 'assistant' %}
12
+### Response:
13
+{{ message['content']|trim -}}
14
+{% if not loop.last %}
15
+
16
+
17
+{% endif %}
18
+{% elif message['role'] == 'user_context' %}
19
+### Input:
20
+{{ message['content']|trim -}}
21
+{% if not loop.last %}
22
+
23
+
24
+{% endif %}
25
+{% endif %}
26
+{% endfor %}
27
+{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}
28
+### Response:
29
+{% endif %}
\ No newline at end of file
examples/template_chatml.jinja
new
+2
@@ -0,0 +1,2 @@
1
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
2
+{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\n' }}{% endif %}
\ No newline at end of file
examples/template_inkbot.jinja
new
+30
@@ -0,0 +1,30 @@
1
+<#meta#>
2
+- Date: {{ (messages|selectattr('role', 'equalto', 'meta-current_date')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'meta-current_date')|list) else '' }}
3
+- Task: {{ (messages|selectattr('role', 'equalto', 'meta-task_name')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'meta-task_name')|list) else '' }}
4
+<#system#>
5
+{{ (messages|selectattr('role', 'equalto', 'system')|list|last).content|trim if (messages|selectattr('role', 'equalto', 'system')|list) else '' }}
6
+<#chat#>
7
+{% for message in messages %}
8
+{% if message['role'] == 'user' %}
9
+<#user#>
10
+{{ message['content']|trim -}}
11
+{% if not loop.last %}
12
+
13
+{% endif %}
14
+{% elif message['role'] == 'assistant' %}
15
+<#bot#>
16
+{{ message['content']|trim -}}
17
+{% if not loop.last %}
18
+
19
+{% endif %}
20
+{% elif message['role'] == 'user_context' %}
21
+<#user_context#>
22
+{{ message['content']|trim -}}
23
+{% if not loop.last %}
24
+
25
+{% endif %}
26
+{% endif %}
27
+{% endfor %}
28
+{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}
29
+<#bot#>
30
+{% endif %}
\ No newline at end of file
tests/async_engine/test_openai_server.py
new
+119
@@ -0,0 +1,119 @@
1
+from argparse import Namespace
2
+from dataclasses import dataclass
3
+
4
+import pytest
5
+from fastapi.testclient import TestClient
6
+
7
+from vllm.entrypoints.openai.api_server import *
8
+
9
+# Define models, templates, and their corresponding expected outputs
10
+MODEL_TEMPLATE_GENERATON_OUTPUT = [
11
+ ("facebook/opt-125m", None, True,
12
+ "Hello</s>Hi there!</s>What is the capital of</s>"),
13
+ ("facebook/opt-125m", None, False,
14
+ "Hello</s>Hi there!</s>What is the capital of</s>"),
15
+ ("facebook/opt-125m", "../../examples/template_chatml.jinja", True,
16
+ """<|im_start|>user
17
+Hello<|im_end|>
18
+<|im_start|>assistant
19
+Hi there!<|im_end|>
20
+<|im_start|>user
21
+What is the capital of<|im_end|>
22
+<|im_start|>assistant
23
+"""),
24
+ ("facebook/opt-125m", "../../examples/template_chatml.jinja", False,
25
+ """<|im_start|>user
26
+Hello<|im_end|>
27
+<|im_start|>assistant
28
+Hi there!<|im_end|>
29
+<|im_start|>user
30
+What is the capital of""")
31
+]
32
+
33
+TEST_MESSAGES = [
34
+ {
35
+ 'role': 'user',
36
+ 'content': 'Hello'
37
+ },
38
+ {
39
+ 'role': 'assistant',
40
+ 'content': 'Hi there!'
41
+ },
42
+ {
43
+ 'role': 'user',
44
+ 'content': 'What is the capital of'
45
+ },
46
+]
47
+client = TestClient(app)
48
+
49
+
50
+@dataclass
51
+class MockTokenizer:
52
+ chat_template = None
53
+
54
+
55
+def test_load_chat_template():
56
+ # Testing chatml template
57
+ template = "../../examples/template_chatml.jinja"
58
+ mock_args = Namespace(chat_template=template)
59
+ tokenizer = MockTokenizer()
60
+
61
+ # Call the function with the mocked args
62
+ load_chat_template(mock_args, tokenizer)
63
+
64
+ template_content = tokenizer.chat_template
65
+
66
+ # Test assertions
67
+ assert template_content is not None
68
+ # Hard coded value for template_chatml.jinja
69
+ assert template_content == """{% for message in messages %}{{'<|im_start|>' + message['role'] + '\\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\\n'}}{% endif %}{% endfor %}
70
+{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\\n' }}{% endif %}"""
71
+
72
+
73
+def test_no_load_chat_template():
74
+ # Testing chatml template
75
+ template = "../../examples/does_not_exist"
76
+ mock_args = Namespace(chat_template=template)
77
+ tokenizer = MockTokenizer()
78
+
79
+ # Call the function with the mocked args
80
+ load_chat_template(mock_args, tokenizer=tokenizer)
81
+ template_content = tokenizer.chat_template
82
+
83
+ # Test assertions
84
+ assert template_content is not None
85
+ # Hard coded value for template_chatml.jinja
86
+ assert template_content == """../../examples/does_not_exist"""
87
+
88
+
89
+@pytest.mark.asyncio
90
+@pytest.mark.parametrize(
91
+ "model,template,add_generation_prompt,expected_output",
92
+ MODEL_TEMPLATE_GENERATON_OUTPUT)
93
+async def test_get_gen_prompt(model, template, add_generation_prompt,
94
+ expected_output):
95
+ # Initialize the tokenizer
96
+ tokenizer = get_tokenizer(tokenizer_name=model)
97
+
98
+ mock_args = Namespace(chat_template=template)
99
+ load_chat_template(mock_args, tokenizer)
100
+
101
+ # Create a mock request object using keyword arguments
102
+ mock_request = ChatCompletionRequest(
103
+ model=model,
104
+ messages=TEST_MESSAGES,
105
+ add_generation_prompt=add_generation_prompt)
106
+
107
+ # Call the function and get the result
108
+ result = tokenizer.apply_chat_template(
109
+ conversation=mock_request.messages,
110
+ tokenize=False,
111
+ add_generation_prompt=mock_request.add_generation_prompt)
112
+
113
+ # Test assertion
114
+ assert result == expected_output, f"The generated prompt does not match the expected output for model {model} and template {template}"
115
+
116
+
117
+def test_health_endpoint():
118
+ response = client.get("/health")
119
+ assert response.status_code == 200
vllm/entrypoints/openai/api_server.py
+204
-179
@@ -3,6 +3,7 @@
3
4
import argparse
5
import asyncio
6
+import codecs
7
import json
8
import time
9
from http import HTTPStatus
@@ -14,7 +15,6 @@ from fastapi import Request
15
from fastapi.exceptions import RequestValidationError
16
from fastapi.middleware.cors import CORSMiddleware
17
from fastapi.responses import JSONResponse, StreamingResponse, Response
17
-from packaging import version
18
19
from vllm.engine.arg_utils import AsyncEngineArgs
20
from vllm.engine.async_llm_engine import AsyncLLMEngine
@@ -31,20 +31,55 @@ from vllm.sampling_params import SamplingParams
31
from vllm.transformers_utils.tokenizer import get_tokenizer
32
from vllm.utils import random_uuid
33
34
-try:
35
- import fastchat
36
- from fastchat.conversation import Conversation, SeparatorStyle
37
- from fastchat.model.model_adapter import get_conversation_template
38
- _fastchat_available = True
39
-except ImportError:
40
- _fastchat_available = False
41
-
34
TIMEOUT_KEEP_ALIVE = 5 # seconds
35
36
logger = init_logger(__name__)
37
served_model = None
38
app = fastapi.FastAPI()
39
engine = None
40
+response_role = None
41
+
42
+
43
+def parse_args():
44
+ parser = argparse.ArgumentParser(
45
+ description="vLLM OpenAI-Compatible RESTful API server.")
46
+ parser.add_argument("--host", type=str, default=None, help="host name")
47
+ parser.add_argument("--port", type=int, default=8000, help="port number")
48
+ parser.add_argument("--allow-credentials",
49
+ action="store_true",
50
+ help="allow credentials")
51
+ parser.add_argument("--allowed-origins",
52
+ type=json.loads,
53
+ default=["*"],
54
+ help="allowed origins")
55
+ parser.add_argument("--allowed-methods",
56
+ type=json.loads,
57
+ default=["*"],
58
+ help="allowed methods")
59
+ parser.add_argument("--allowed-headers",
60
+ type=json.loads,
61
+ default=["*"],
62
+ help="allowed headers")
63
+ parser.add_argument("--served-model-name",
64
+ type=str,
65
+ default=None,
66
+ help="The model name used in the API. If not "
67
+ "specified, the model name will be the same as "
68
+ "the huggingface name.")
69
+ parser.add_argument("--chat-template",
70
+ type=str,
71
+ default=None,
72
+ help="The file path to the chat template, "
73
+ "or the template in single-line form "
74
+ "for the specified model")
75
+ parser.add_argument("--response-role",
76
+ type=str,
77
+ default="assistant",
78
+ help="The role name to return if "
79
+ "`request.add_generation_prompt=true`.")
80
+
81
+ parser = AsyncEngineArgs.add_cli_args(parser)
82
+ return parser.parse_args()
83
84
85
def create_error_response(status_code: HTTPStatus,
@@ -54,6 +89,25 @@ def create_error_response(status_code: HTTPStatus,
89
status_code=status_code.value)
90
91
92
+def load_chat_template(args, tokenizer):
93
+ if args.chat_template is not None:
94
+ try:
95
+ with open(args.chat_template, "r") as f:
96
+ chat_template = f.read()
97
+ except OSError:
98
+ # If opening a file fails, set chat template to be args to
99
+ # ensure we decode so our escape are interpreted correctly
100
+ chat_template = codecs.decode(args.chat_template, "unicode_escape")
101
+
102
+ tokenizer.chat_template = chat_template
103
+ logger.info(
104
+ f"Using supplied chat template:\n{tokenizer.chat_template}")
105
+ elif tokenizer.chat_template is not None:
106
+ logger.info(f"Using default chat template:\n{tokenizer.chat_template}")
107
+ else:
108
+ logger.warning("No chat template provided. Chat API will not work.")
109
+
110
+
111
@app.exception_handler(RequestValidationError)
112
async def validation_exception_handler(_, exc):
113
return create_error_response(HTTPStatus.BAD_REQUEST, str(exc))
@@ -69,53 +123,6 @@ async def check_model(request) -> Optional[JSONResponse]:
123
return ret
124
125
72
-async def get_gen_prompt(request) -> str:
73
- if not _fastchat_available:
74
- raise ModuleNotFoundError(
75
- "fastchat is not installed. Please install fastchat to use "
76
- "the chat completion and conversation APIs: `$ pip install fschat`"
77
- )
78
- if version.parse(fastchat.__version__) < version.parse("0.2.23"):
79
- raise ImportError(
80
- f"fastchat version is low. Current version: {fastchat.__version__} "
81
- "Please upgrade fastchat to use: `$ pip install -U fschat`")
82
-
83
- conv = get_conversation_template(request.model)
84
- conv = Conversation(
85
- name=conv.name,
86
- system_template=conv.system_template,
87
- system_message=conv.system_message,
88
- roles=conv.roles,
89
- messages=list(conv.messages), # prevent in-place modification
90
- offset=conv.offset,
91
- sep_style=SeparatorStyle(conv.sep_style),
92
- sep=conv.sep,
93
- sep2=conv.sep2,
94
- stop_str=conv.stop_str,
95
- stop_token_ids=conv.stop_token_ids,
96
- )
97
-
98
- if isinstance(request.messages, str):
99
- prompt = request.messages
100
- else:
101
- for message in request.messages:
102
- msg_role = message["role"]
103
- if msg_role == "system":
104
- conv.system_message = message["content"]
105
- elif msg_role == "user":
106
- conv.append_message(conv.roles[0], message["content"])
107
- elif msg_role == "assistant":
108
- conv.append_message(conv.roles[1], message["content"])
109
- else:
110
- raise ValueError(f"Unknown role: {msg_role}")
111
-
112
- # Add a blank message for the assistant.
113
- conv.append_message(conv.roles[1], None)
114
- prompt = conv.get_prompt()
115
-
116
- return prompt
117
-
118
-
126
async def check_length(
127
request: Union[ChatCompletionRequest, CompletionRequest],
128
prompt: Optional[str] = None,
@@ -207,7 +214,6 @@ async def create_chat_completion(request: ChatCompletionRequest,
214
- function_call (Users should implement this by themselves)
215
- logit_bias (to be supported by vLLM engine)
216
"""
210
-
217
error_check_ret = await check_model(request)
218
if error_check_ret is not None:
219
return error_check_ret
@@ -217,7 +223,15 @@ async def create_chat_completion(request: ChatCompletionRequest,
223
return create_error_response(HTTPStatus.BAD_REQUEST,
224
"logit_bias is not currently supported")
225
220
- prompt = await get_gen_prompt(request)
226
+ try:
227
+ prompt = tokenizer.apply_chat_template(
228
+ conversation=request.messages,
229
+ tokenize=False,
230
+ add_generation_prompt=request.add_generation_prompt)
231
+ except Exception as e:
232
+ logger.error(f"Error in applying chat template from request: {str(e)}")
233
+ return create_error_response(HTTPStatus.BAD_REQUEST, str(e))
234
+
235
token_ids, error_check_ret = await check_length(request, prompt=prompt)
236
if error_check_ret is not None:
237
return error_check_ret
@@ -225,6 +239,7 @@ async def create_chat_completion(request: ChatCompletionRequest,
239
model_name = request.model
240
request_id = f"cmpl-{random_uuid()}"
241
created_time = int(time.monotonic())
242
+ chunk_object_type = "chat.completion.chunk"
243
try:
244
spaces_between_special_tokens = request.spaces_between_special_tokens
245
sampling_params = SamplingParams(
@@ -249,128 +264,162 @@ async def create_chat_completion(request: ChatCompletionRequest,
264
result_generator = engine.generate(prompt, sampling_params, request_id,
265
token_ids)
266
252
- def create_stream_response_json(
253
- index: int,
254
- text: str,
255
- finish_reason: Optional[str] = None,
256
- usage: Optional[UsageInfo] = None,
257
- ) -> str:
258
- choice_data = ChatCompletionResponseStreamChoice(
259
- index=index,
260
- delta=DeltaMessage(content=text),
261
- finish_reason=finish_reason,
262
- )
263
- response = ChatCompletionStreamResponse(
264
- id=request_id,
265
- created=created_time,
266
- model=model_name,
267
- choices=[choice_data],
268
- )
269
- if usage is not None:
270
- response.usage = usage
271
- # exclude unset to leave details out of each sse
272
- response_json = response.json(exclude_unset=True, ensure_ascii=False)
273
-
274
- return response_json
267
+ def get_role() -> str:
268
+ if request.add_generation_prompt:
269
+ return response_role
270
+ else:
271
+ return request.messages[-1]["role"]
272
273
async def completion_stream_generator() -> AsyncGenerator[str, None]:
277
- # First chunk with role
274
+ # Send first response for each request.n (index) with the role
275
+ role = get_role()
276
for i in range(request.n):
277
choice_data = ChatCompletionResponseStreamChoice(
280
- index=i,
281
- delta=DeltaMessage(role="assistant"),
282
- finish_reason=None,
283
- )
278
+ index=i, delta=DeltaMessage(role=role), finish_reason=None)
279
chunk = ChatCompletionStreamResponse(id=request_id,
280
+ object=chunk_object_type,
281
+ created=created_time,
282
choices=[choice_data],
283
model=model_name)
284
data = chunk.json(exclude_unset=True, ensure_ascii=False)
285
yield f"data: {data}\n\n"
286
287
+ # Send response to echo the input portion of the last message
288
+ if request.echo:
289
+ last_msg_content = ""
290
+ if request.messages and isinstance(
291
+ request.messages, list) and request.messages[-1].get(
292
+ "content") and request.messages[-1].get(
293
+ "role") == role:
294
+ last_msg_content = request.messages[-1]["content"]
295
+ if last_msg_content:
296
+ for i in range(request.n):
297
+ choice_data = ChatCompletionResponseStreamChoice(
298
+ index=i,
299
+ delta=DeltaMessage(content=last_msg_content),
300
+ finish_reason=None)
301
+ chunk = ChatCompletionStreamResponse(
302
+ id=request_id,
303
+ object=chunk_object_type,
304
+ created=created_time,
305
+ choices=[choice_data],
306
+ model=model_name)
307
+ data = chunk.json(exclude_unset=True, ensure_ascii=False)
308
+ yield f"data: {data}\n\n"
309
+
310
+ # Send response for each token for each request.n (index)
311
previous_texts = [""] * request.n
312
previous_num_tokens = [0] * request.n
313
+ finish_reason_sent = [False] * request.n
314
async for res in result_generator:
315
res: RequestOutput
316
for output in res.outputs:
317
i = output.index
296
- delta_text = output.text[len(previous_texts[i]):]
297
- previous_texts[i] = output.text
298
- completion_tokens = len(output.token_ids)
299
- previous_num_tokens[i] = completion_tokens
300
- response_json = create_stream_response_json(
301
- index=i,
302
- text=delta_text,
303
- )
304
- yield f"data: {response_json}\n\n"
305
- if output.finish_reason is not None:
318
+
319
+ if finish_reason_sent[i]:
320
+ continue
321
+
322
+ if output.finish_reason is None:
323
+ # Send token-by-token response for each request.n
324
+ delta_text = output.text[len(previous_texts[i]):]
325
+ previous_texts[i] = output.text
326
+ completion_tokens = len(output.token_ids)
327
+ previous_num_tokens[i] = completion_tokens
328
+ choice_data = ChatCompletionResponseStreamChoice(
329
+ index=i,
330
+ delta=DeltaMessage(content=delta_text),
331
+ finish_reason=None)
332
+ chunk = ChatCompletionStreamResponse(
333
+ id=request_id,
334
+ object=chunk_object_type,
335
+ created=created_time,
336
+ choices=[choice_data],
337
+ model=model_name)
338
+ data = chunk.json(exclude_unset=True, ensure_ascii=False)
339
+ yield f"data: {data}\n\n"
340
+ else:
341
+ # Send the finish response for each request.n only once
342
prompt_tokens = len(res.prompt_token_ids)
343
final_usage = UsageInfo(
344
prompt_tokens=prompt_tokens,
345
completion_tokens=completion_tokens,
346
total_tokens=prompt_tokens + completion_tokens,
347
)
312
- response_json = create_stream_response_json(
313
- index=i,
314
- text="",
315
- finish_reason=output.finish_reason,
316
- usage=final_usage,
317
- )
318
- yield f"data: {response_json}\n\n"
348
+ choice_data = ChatCompletionResponseStreamChoice(
349
+ index=i, delta=[], finish_reason=output.finish_reason)
350
+ chunk = ChatCompletionStreamResponse(
351
+ id=request_id,
352
+ object=chunk_object_type,
353
+ created=created_time,
354
+ choices=[choice_data],
355
+ model=model_name)
356
+ if final_usage is not None:
357
+ chunk.usage = final_usage
358
+ data = chunk.json(exclude_unset=True,
359
+ exclude_none=True,
360
+ ensure_ascii=False)
361
+ yield f"data: {data}\n\n"
362
+ finish_reason_sent[i] = True
363
+ # Send the final done message after all response.n are finished
364
yield "data: [DONE]\n\n"
365
321
- # Streaming response
322
- if request.stream:
323
- return StreamingResponse(completion_stream_generator(),
324
- media_type="text/event-stream")
325
-
326
- # Non-streaming response
327
- final_res: RequestOutput = None
328
- async for res in result_generator:
329
- if await raw_request.is_disconnected():
330
- # Abort the request if the client disconnects.
331
- await engine.abort(request_id)
332
- return create_error_response(HTTPStatus.BAD_REQUEST,
333
- "Client disconnected")
334
- final_res = res
335
- assert final_res is not None
336
- choices = []
337
- for output in final_res.outputs:
338
- choice_data = ChatCompletionResponseChoice(
339
- index=output.index,
340
- message=ChatMessage(role="assistant", content=output.text),
341
- finish_reason=output.finish_reason,
366
+ async def completion_full_generator():
367
+ final_res: RequestOutput = None
368
+ async for res in result_generator:
369
+ if await raw_request.is_disconnected():
370
+ # Abort the request if the client disconnects.
371
+ await engine.abort(request_id)
372
+ return create_error_response(HTTPStatus.BAD_REQUEST,
373
+ "Client disconnected")
374
+ final_res = res
375
+ assert final_res is not None
376
+
377
+ choices = []
378
+ role = get_role()
379
+ for output in final_res.outputs:
380
+ choice_data = ChatCompletionResponseChoice(
381
+ index=output.index,
382
+ message=ChatMessage(role=role, content=output.text),
383
+ finish_reason=output.finish_reason,
384
+ )
385
+ choices.append(choice_data)
386
+
387
+ if request.echo:
388
+ last_msg_content = ""
389
+ if request.messages and isinstance(
390
+ request.messages, list) and request.messages[-1].get(
391
+ "content") and request.messages[-1].get(
392
+ "role") == role:
393
+ last_msg_content = request.messages[-1]["content"]
394
+
395
+ for choice in choices:
396
+ full_message = last_msg_content + choice.message.content
397
+ choice.message.content = full_message
398
+
399
+ num_prompt_tokens = len(final_res.prompt_token_ids)
400
+ num_generated_tokens = sum(
401
+ len(output.token_ids) for output in final_res.outputs)
402
+ usage = UsageInfo(
403
+ prompt_tokens=num_prompt_tokens,
404
+ completion_tokens=num_generated_tokens,
405
+ total_tokens=num_prompt_tokens + num_generated_tokens,
406
+ )
407
+ response = ChatCompletionResponse(
408
+ id=request_id,
409
+ created=created_time,
410
+ model=model_name,
411
+ choices=choices,
412
+ usage=usage,
413
)
343
- choices.append(choice_data)
414
345
- num_prompt_tokens = len(final_res.prompt_token_ids)
346
- num_generated_tokens = sum(
347
- len(output.token_ids) for output in final_res.outputs)
348
- usage = UsageInfo(
349
- prompt_tokens=num_prompt_tokens,
350
- completion_tokens=num_generated_tokens,
351
- total_tokens=num_prompt_tokens + num_generated_tokens,
352
- )
353
- response = ChatCompletionResponse(
354
- id=request_id,
355
- created=created_time,
356
- model=model_name,
357
- choices=choices,
358
- usage=usage,
359
- )
415
+ return response
416
417
+ # Streaming response
418
if request.stream:
362
- # When user requests streaming but we don't stream, we still need to
363
- # return a streaming response with a single event.
364
- response_json = response.json(ensure_ascii=False)
365
-
366
- async def fake_stream_generator() -> AsyncGenerator[str, None]:
367
- yield f"data: {response_json}\n\n"
368
- yield "data: [DONE]\n\n"
369
-
370
- return StreamingResponse(fake_stream_generator(),
419
+ return StreamingResponse(completion_stream_generator(),
420
media_type="text/event-stream")
372
-
373
- return response
421
+ else:
422
+ return await completion_full_generator()
423
424
425
@app.post("/v1/completions")
@@ -642,34 +691,7 @@ async def create_completion(request: CompletionRequest, raw_request: Request):
691
692
693
if __name__ == "__main__":
645
- parser = argparse.ArgumentParser(
646
- description="vLLM OpenAI-Compatible RESTful API server.")
647
- parser.add_argument("--host", type=str, default=None, help="host name")
648
- parser.add_argument("--port", type=int, default=8000, help="port number")
649
- parser.add_argument("--allow-credentials",
650
- action="store_true",
651
- help="allow credentials")
652
- parser.add_argument("--allowed-origins",
653
- type=json.loads,
654
- default=["*"],
655
- help="allowed origins")
656
- parser.add_argument("--allowed-methods",
657
- type=json.loads,
658
- default=["*"],
659
- help="allowed methods")
660
- parser.add_argument("--allowed-headers",
661
- type=json.loads,
662
- default=["*"],
663
- help="allowed headers")
664
- parser.add_argument("--served-model-name",
665
- type=str,
666
- default=None,
667
- help="The model name used in the API. If not "
668
- "specified, the model name will be the same as "
669
- "the huggingface name.")
670
-
671
- parser = AsyncEngineArgs.add_cli_args(parser)
672
- args = parser.parse_args()
694
+ args = parse_args()
695
696
app.add_middleware(
697
CORSMiddleware,
@@ -686,6 +708,8 @@ if __name__ == "__main__":
708
else:
709
served_model = args.model
710
711
+ response_role = args.response_role
712
+
713
engine_args = AsyncEngineArgs.from_cli_args(args)
714
engine = AsyncLLMEngine.from_engine_args(engine_args)
715
engine_model_config = asyncio.run(engine.get_model_config())
@@ -696,6 +720,7 @@ if __name__ == "__main__":
720
engine_model_config.tokenizer,
721
tokenizer_mode=engine_model_config.tokenizer_mode,
722
trust_remote_code=engine_model_config.trust_remote_code)
723
+ load_chat_template(args, tokenizer)
724
725
uvicorn.run(app,
726
host=args.host,
vllm/entrypoints/openai/protocol.py
+2
@@ -73,6 +73,8 @@ class ChatCompletionRequest(BaseModel):
73
stop_token_ids: Optional[List[int]] = Field(default_factory=list)
74
skip_special_tokens: Optional[bool] = True
75
spaces_between_special_tokens: Optional[bool] = True
76
+ add_generation_prompt: Optional[bool] = True
77
+ echo: Optional[bool] = False
78
79
80
class CompletionRequest(BaseModel):