A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
[Doc][CI/Build] Update docs and tests to use `vllm serve` (#6431)
Cyrus Leung committed
Jul 17, 2024 at 15:43 UTC
5bf35a91e465278b51b0c98fa1b13beb8b04d431
23 files changed
+155
-175
docs/source/getting_started/quickstart.rst
+2
-5
@@ -73,16 +73,13 @@ Start the server:
73
74
.. code-block:: console
75
76
- $ python -m vllm.entrypoints.openai.api_server \
77
- $ --model facebook/opt-125m
76
+ $ vllm serve facebook/opt-125m
77
78
By default, the server uses a predefined chat template stored in the tokenizer. You can override this template by using the ``--chat-template`` argument:
79
80
.. code-block:: console
81
83
- $ python -m vllm.entrypoints.openai.api_server \
84
- $ --model facebook/opt-125m \
85
- $ --chat-template ./examples/template_chatml.jinja
82
+ $ vllm serve facebook/opt-125m --chat-template ./examples/template_chatml.jinja
83
84
This server can be queried in the same format as OpenAI API. For example, list the models:
85
docs/source/models/adding_model.rst
+2
-2
@@ -114,7 +114,7 @@ Just add the following lines in your code:
114
from your_code import YourModelForCausalLM
115
ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
116
117
-If you are running api server with `python -m vllm.entrypoints.openai.api_server args`, you can wrap the entrypoint with the following code:
117
+If you are running api server with :code:`vllm serve <args>`, you can wrap the entrypoint with the following code:
118
119
.. code-block:: python
120
@@ -124,4 +124,4 @@ If you are running api server with `python -m vllm.entrypoints.openai.api_server
124
import runpy
125
runpy.run_module('vllm.entrypoints.openai.api_server', run_name='__main__')
126
127
-Save the above code in a file and run it with `python your_file.py args`.
127
+Save the above code in a file and run it with :code:`python your_file.py <args>`.
docs/source/models/engine_args.rst
+2
-2
@@ -8,7 +8,7 @@ Below, you can find an explanation of every engine argument for vLLM:
8
.. argparse::
9
:module: vllm.engine.arg_utils
10
:func: _engine_args_parser
11
- :prog: -m vllm.entrypoints.openai.api_server
11
+ :prog: vllm serve
12
:nodefaultconst:
13
14
Async Engine Arguments
@@ -19,5 +19,5 @@ Below are the additional arguments related to the asynchronous engine:
19
.. argparse::
20
:module: vllm.engine.arg_utils
21
:func: _async_engine_args_parser
22
- :prog: -m vllm.entrypoints.openai.api_server
22
+ :prog: vllm serve
23
:nodefaultconst:
\ No newline at end of file
docs/source/models/lora.rst
+1
-2
@@ -61,8 +61,7 @@ LoRA adapted models can also be served with the Open-AI compatible vLLM server.
61
62
.. code-block:: bash
63
64
- python -m vllm.entrypoints.openai.api_server \
65
- --model meta-llama/Llama-2-7b-hf \
64
+ vllm serve meta-llama/Llama-2-7b-hf \
65
--enable-lora \
66
--lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/
67
docs/source/models/vlm.rst
+1
-3
@@ -94,9 +94,7 @@ Below is an example on how to launch the same ``llava-hf/llava-1.5-7b-hf`` with
94
95
.. code-block:: bash
96
97
- python -m vllm.entrypoints.openai.api_server \
98
- --model llava-hf/llava-1.5-7b-hf \
99
- --chat-template template_llava.jinja
97
+ vllm serve llava-hf/llava-1.5-7b-hf --chat-template template_llava.jinja
98
99
.. important::
100
We have removed all vision language related CLI args in the ``0.5.1`` release. **This is a breaking change**, so please update your code to follow
docs/source/serving/deploying_with_dstack.rst
+1
-1
@@ -40,7 +40,7 @@ Next, to provision a VM instance with LLM of your choice(`NousResearch/Llama-2-7
40
gpu: 24GB
41
commands:
42
- pip install vllm
43
- - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000
43
+ - vllm serve $MODEL --port 8000
44
model:
45
format: openai
46
type: chat
docs/source/serving/distributed_serving.rst
+2
-4
@@ -35,16 +35,14 @@ To run multi-GPU serving, pass in the :code:`--tensor-parallel-size` argument wh
35
36
.. code-block:: console
37
38
- $ python -m vllm.entrypoints.openai.api_server \
39
- $ --model facebook/opt-13b \
38
+ $ vllm serve facebook/opt-13b \
39
$ --tensor-parallel-size 4
40
41
You can also additionally specify :code:`--pipeline-parallel-size` to enable pipeline parallelism. For example, to run API server on 8 GPUs with pipeline parallelism and tensor parallelism:
42
43
.. code-block:: console
44
46
- $ python -m vllm.entrypoints.openai.api_server \
47
- $ --model gpt2 \
45
+ $ vllm serve gpt2 \
46
$ --tensor-parallel-size 4 \
47
$ --pipeline-parallel-size 2 \
48
$ --distributed-executor-backend ray
docs/source/serving/openai_compatible_server.md
+3
-5
@@ -4,7 +4,7 @@ vLLM provides an HTTP server that implements OpenAI's [Completions](https://plat
4
5
You can start the server using Python, or using [Docker](deploying_with_docker.rst):
6
```bash
7
-python -m vllm.entrypoints.openai.api_server --model NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
7
+vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
8
```
9
10
To call the server, you can use the official OpenAI Python client library, or any other HTTP client.
@@ -97,9 +97,7 @@ template, or the template in string form. Without a chat template, the server wi
97
and all chat requests will error.
98
99
```bash
100
-python -m vllm.entrypoints.openai.api_server \
101
- --model ... \
102
- --chat-template ./path-to-chat-template.jinja
100
+vllm serve <model> --chat-template ./path-to-chat-template.jinja
101
```
102
103
vLLM community provides a set of chat templates for popular models. You can find them in the examples
@@ -110,7 +108,7 @@ directory [here](https://github.com/vllm-project/vllm/tree/main/examples/)
108
```{argparse}
109
:module: vllm.entrypoints.openai.cli_args
110
:func: create_parser_for_docs
113
-:prog: -m vllm.entrypoints.openai.api_server
111
+:prog: vllm serve
112
```
113
114
## Tool calling in the chat completion API
examples/api_client.py
+2
-3
@@ -1,8 +1,7 @@
1
-"""Example Python client for vllm.entrypoints.api_server
1
+"""Example Python client for `vllm.entrypoints.api_server`
2
NOTE: The API server is used only for demonstration and simple performance
3
benchmarks. It is not intended for production use.
4
-For production use, we recommend vllm.entrypoints.openai.api_server
5
-and the OpenAI client API
4
+For production use, we recommend `vllm serve` and the OpenAI client API.
5
"""
6
7
import argparse
examples/logging_configuration.md
+3
-9
@@ -95,9 +95,7 @@ to the path of the custom logging configuration JSON file:
95
96
```bash
97
VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
98
- python3 -m vllm.entrypoints.openai.api_server \
99
- --max-model-len 2048 \
100
- --model mistralai/Mistral-7B-v0.1
98
+ vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048
99
```
100
101
@@ -152,9 +150,7 @@ to the path of the custom logging configuration JSON file:
150
151
```bash
152
VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
155
- python3 -m vllm.entrypoints.openai.api_server \
156
- --max-model-len 2048 \
157
- --model mistralai/Mistral-7B-v0.1
153
+ vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048
154
```
155
156
@@ -167,9 +163,7 @@ loggers.
163
164
```bash
165
VLLM_CONFIGURE_LOGGING=0 \
170
- python3 -m vllm.entrypoints.openai.api_server \
171
- --max-model-len 2048 \
172
- --model mistralai/Mistral-7B-v0.1
166
+ vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048
167
```
168
169
examples/openai_vision_api_client.py
+1
-3
@@ -1,9 +1,7 @@
1
"""An example showing how to use vLLM to serve VLMs.
2
3
Launch the vLLM server with the following command:
4
-python -m vllm.entrypoints.openai.api_server \
5
- --model llava-hf/llava-1.5-7b-hf \
6
- --chat-template template_llava.jinja
4
+vllm serve llava-hf/llava-1.5-7b-hf --chat-template template_llava.jinja
5
"""
6
import base64
7
examples/production_monitoring/Otel.md
+3
-3
@@ -36,7 +36,7 @@
36
```
37
export OTEL_SERVICE_NAME="vllm-server"
38
export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
39
- python -m vllm.entrypoints.openai.api_server --model="facebook/opt-125m" --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
39
+ vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
40
```
41
42
1. In a new shell, send requests with trace context from a dummy client
@@ -62,7 +62,7 @@ By default, `grpc` is used. To set `http/protobuf` as the protocol, configure th
62
```
63
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
64
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
65
-python -m vllm.entrypoints.openai.api_server --model="facebook/opt-125m" --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
65
+vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
66
```
67
68
## Instrumentation of FastAPI
@@ -74,7 +74,7 @@ OpenTelemetry allows automatic instrumentation of FastAPI.
74
75
1. Run vLLM with `opentelemetry-instrument`
76
```
77
- opentelemetry-instrument python -m vllm.entrypoints.openai.api_server --model="facebook/opt-125m"
77
+ opentelemetry-instrument vllm serve facebook/opt-125m
78
```
79
80
1. Send a request to vLLM and find its trace in Jaeger. It should contain spans from FastAPI.
examples/production_monitoring/README.md
+1
-2
@@ -10,8 +10,7 @@ Install:
10
11
Prometheus metric logging is enabled by default in the OpenAI-compatible server. Launch via the entrypoint:
12
```bash
13
-python3 -m vllm.entrypoints.openai.api_server \
14
- --model mistralai/Mistral-7B-v0.1 \
13
+vllm serve mistralai/Mistral-7B-v0.1 \
14
--max-model-len 2048 \
15
--disable-log-requests
16
```
tests/async_engine/test_openapi_server_ray.py
+11
-11
@@ -9,17 +9,17 @@ MODEL_NAME = "facebook/opt-125m"
9
10
@pytest.fixture(scope="module")
11
def server():
12
- with RemoteOpenAIServer([
13
- "--model",
14
- MODEL_NAME,
15
- # use half precision for speed and memory savings in CI environment
16
- "--dtype",
17
- "float16",
18
- "--max-model-len",
19
- "2048",
20
- "--enforce-eager",
21
- "--engine-use-ray"
22
- ]) as remote_server:
12
+ args = [
13
+ # use half precision for speed and memory savings in CI environment
14
+ "--dtype",
15
+ "float16",
16
+ "--max-model-len",
17
+ "2048",
18
+ "--enforce-eager",
19
+ "--engine-use-ray"
20
+ ]
21
+
22
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
23
yield remote_server
24
25
tests/distributed/test_pipeline_parallel.py
+1
-5
@@ -15,8 +15,6 @@ from ..utils import RemoteOpenAIServer
15
])
16
def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL, MODEL_NAME):
17
pp_args = [
18
- "--model",
19
- MODEL_NAME,
18
# use half precision for speed and memory savings in CI environment
19
"--dtype",
20
"bfloat16",
@@ -34,8 +32,6 @@ def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL, MODEL_NAME):
32
# schedule all workers in a node other than the head node,
33
# which can cause the test to fail.
34
tp_args = [
37
- "--model",
38
- MODEL_NAME,
35
# use half precision for speed and memory savings in CI environment
36
"--dtype",
37
"bfloat16",
@@ -53,7 +49,7 @@ def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL, MODEL_NAME):
49
50
results = []
51
for args in [pp_args, tp_args]:
56
- with RemoteOpenAIServer(args) as server:
52
+ with RemoteOpenAIServer(MODEL_NAME, args) as server:
53
client = server.get_client()
54
55
# test models list
tests/entrypoints/openai/test_chat.py
+21
-21
@@ -27,27 +27,27 @@ def zephyr_lora_files():
27
28
@pytest.fixture(scope="module")
29
def server(zephyr_lora_files):
30
- with RemoteOpenAIServer([
31
- "--model",
32
- MODEL_NAME,
33
- # use half precision for speed and memory savings in CI environment
34
- "--dtype",
35
- "bfloat16",
36
- "--max-model-len",
37
- "8192",
38
- "--enforce-eager",
39
- # lora config below
40
- "--enable-lora",
41
- "--lora-modules",
42
- f"zephyr-lora={zephyr_lora_files}",
43
- f"zephyr-lora2={zephyr_lora_files}",
44
- "--max-lora-rank",
45
- "64",
46
- "--max-cpu-loras",
47
- "2",
48
- "--max-num-seqs",
49
- "128",
50
- ]) as remote_server:
30
+ args = [
31
+ # use half precision for speed and memory savings in CI environment
32
+ "--dtype",
33
+ "bfloat16",
34
+ "--max-model-len",
35
+ "8192",
36
+ "--enforce-eager",
37
+ # lora config below
38
+ "--enable-lora",
39
+ "--lora-modules",
40
+ f"zephyr-lora={zephyr_lora_files}",
41
+ f"zephyr-lora2={zephyr_lora_files}",
42
+ "--max-lora-rank",
43
+ "64",
44
+ "--max-cpu-loras",
45
+ "2",
46
+ "--max-num-seqs",
47
+ "128",
48
+ ]
49
+
50
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
51
yield remote_server
52
53
tests/entrypoints/openai/test_completion.py
+30
-30
@@ -37,36 +37,36 @@ def zephyr_pa_files():
37
38
@pytest.fixture(scope="module")
39
def server(zephyr_lora_files, zephyr_pa_files):
40
- with RemoteOpenAIServer([
41
- "--model",
42
- MODEL_NAME,
43
- # use half precision for speed and memory savings in CI environment
44
- "--dtype",
45
- "bfloat16",
46
- "--max-model-len",
47
- "8192",
48
- "--max-num-seqs",
49
- "128",
50
- "--enforce-eager",
51
- # lora config
52
- "--enable-lora",
53
- "--lora-modules",
54
- f"zephyr-lora={zephyr_lora_files}",
55
- f"zephyr-lora2={zephyr_lora_files}",
56
- "--max-lora-rank",
57
- "64",
58
- "--max-cpu-loras",
59
- "2",
60
- # pa config
61
- "--enable-prompt-adapter",
62
- "--prompt-adapters",
63
- f"zephyr-pa={zephyr_pa_files}",
64
- f"zephyr-pa2={zephyr_pa_files}",
65
- "--max-prompt-adapters",
66
- "2",
67
- "--max-prompt-adapter-token",
68
- "128",
69
- ]) as remote_server:
40
+ args = [
41
+ # use half precision for speed and memory savings in CI environment
42
+ "--dtype",
43
+ "bfloat16",
44
+ "--max-model-len",
45
+ "8192",
46
+ "--max-num-seqs",
47
+ "128",
48
+ "--enforce-eager",
49
+ # lora config
50
+ "--enable-lora",
51
+ "--lora-modules",
52
+ f"zephyr-lora={zephyr_lora_files}",
53
+ f"zephyr-lora2={zephyr_lora_files}",
54
+ "--max-lora-rank",
55
+ "64",
56
+ "--max-cpu-loras",
57
+ "2",
58
+ # pa config
59
+ "--enable-prompt-adapter",
60
+ "--prompt-adapters",
61
+ f"zephyr-pa={zephyr_pa_files}",
62
+ f"zephyr-pa2={zephyr_pa_files}",
63
+ "--max-prompt-adapters",
64
+ "2",
65
+ "--max-prompt-adapter-token",
66
+ "128",
67
+ ]
68
+
69
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
70
yield remote_server
71
72
tests/entrypoints/openai/test_embedding.py
+11
-11
@@ -11,17 +11,17 @@ EMBEDDING_MODEL_NAME = "intfloat/e5-mistral-7b-instruct"
11
12
@pytest.fixture(scope="module")
13
def embedding_server():
14
- with RemoteOpenAIServer([
15
- "--model",
16
- EMBEDDING_MODEL_NAME,
17
- # use half precision for speed and memory savings in CI environment
18
- "--dtype",
19
- "bfloat16",
20
- "--enforce-eager",
21
- "--max-model-len",
22
- "8192",
23
- "--enforce-eager",
24
- ]) as remote_server:
14
+ args = [
15
+ # use half precision for speed and memory savings in CI environment
16
+ "--dtype",
17
+ "bfloat16",
18
+ "--enforce-eager",
19
+ "--max-model-len",
20
+ "8192",
21
+ "--enforce-eager",
22
+ ]
23
+
24
+ with RemoteOpenAIServer(EMBEDDING_MODEL_NAME, args) as remote_server:
25
yield remote_server
26
27
tests/entrypoints/openai/test_models.py
+21
-21
@@ -19,27 +19,27 @@ def zephyr_lora_files():
19
20
@pytest.fixture(scope="module")
21
def server(zephyr_lora_files):
22
- with RemoteOpenAIServer([
23
- "--model",
24
- MODEL_NAME,
25
- # use half precision for speed and memory savings in CI environment
26
- "--dtype",
27
- "bfloat16",
28
- "--max-model-len",
29
- "8192",
30
- "--enforce-eager",
31
- # lora config below
32
- "--enable-lora",
33
- "--lora-modules",
34
- f"zephyr-lora={zephyr_lora_files}",
35
- f"zephyr-lora2={zephyr_lora_files}",
36
- "--max-lora-rank",
37
- "64",
38
- "--max-cpu-loras",
39
- "2",
40
- "--max-num-seqs",
41
- "128",
42
- ]) as remote_server:
22
+ args = [
23
+ # use half precision for speed and memory savings in CI environment
24
+ "--dtype",
25
+ "bfloat16",
26
+ "--max-model-len",
27
+ "8192",
28
+ "--enforce-eager",
29
+ # lora config below
30
+ "--enable-lora",
31
+ "--lora-modules",
32
+ f"zephyr-lora={zephyr_lora_files}",
33
+ f"zephyr-lora2={zephyr_lora_files}",
34
+ "--max-lora-rank",
35
+ "64",
36
+ "--max-cpu-loras",
37
+ "2",
38
+ "--max-num-seqs",
39
+ "128",
40
+ ]
41
+
42
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
43
yield remote_server
44
45
tests/entrypoints/openai/test_tokenization.py
+12
-12
@@ -12,18 +12,18 @@ MODEL_NAME = "HuggingFaceH4/zephyr-7b-beta"
12
13
@pytest.fixture(scope="module")
14
def server():
15
- with RemoteOpenAIServer([
16
- "--model",
17
- MODEL_NAME,
18
- # use half precision for speed and memory savings in CI environment
19
- "--dtype",
20
- "bfloat16",
21
- "--max-model-len",
22
- "8192",
23
- "--enforce-eager",
24
- "--max-num-seqs",
25
- "128",
26
- ]) as remote_server:
15
+ args = [
16
+ # use half precision for speed and memory savings in CI environment
17
+ "--dtype",
18
+ "bfloat16",
19
+ "--max-model-len",
20
+ "8192",
21
+ "--enforce-eager",
22
+ "--max-num-seqs",
23
+ "128",
24
+ ]
25
+
26
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
27
yield remote_server
28
29
tests/entrypoints/openai/test_vision.py
+11
-11
@@ -23,17 +23,17 @@ TEST_IMAGE_URLS = [
23
24
@pytest.fixture(scope="module")
25
def server():
26
- with RemoteOpenAIServer([
27
- "--model",
28
- MODEL_NAME,
29
- "--dtype",
30
- "bfloat16",
31
- "--max-model-len",
32
- "4096",
33
- "--enforce-eager",
34
- "--chat-template",
35
- str(LLAVA_CHAT_TEMPLATE),
36
- ]) as remote_server:
26
+ args = [
27
+ "--dtype",
28
+ "bfloat16",
29
+ "--max-model-len",
30
+ "4096",
31
+ "--enforce-eager",
32
+ "--chat-template",
33
+ str(LLAVA_CHAT_TEMPLATE),
34
+ ]
35
+
36
+ with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
37
yield remote_server
38
39
tests/tensorizer_loader/test_tensorizer.py
+2
-2
@@ -214,12 +214,12 @@ def test_openai_apiserver_with_tensorizer(vllm_runner, tmp_path):
214
215
## Start OpenAI API server
216
openai_args = [
217
- "--model", model_ref, "--dtype", "float16", "--load-format",
217
+ "--dtype", "float16", "--load-format",
218
"tensorizer", "--model-loader-extra-config",
219
json.dumps(model_loader_extra_config),
220
]
221
222
- with RemoteOpenAIServer(openai_args) as server:
222
+ with RemoteOpenAIServer(model_ref, openai_args) as server:
223
print("Server ready.")
224
225
client = server.get_client()
tests/utils.py
+11
-7
@@ -49,7 +49,13 @@ class RemoteOpenAIServer:
49
DUMMY_API_KEY = "token-abc123" # vLLM's OpenAI server does not need API key
50
MAX_SERVER_START_WAIT_S = 600 # wait for server to start for 60 seconds
51
52
- def __init__(self, cli_args: List[str], *, auto_port: bool = True) -> None:
52
+ def __init__(
53
+ self,
54
+ model: str,
55
+ cli_args: List[str],
56
+ *,
57
+ auto_port: bool = True,
58
+ ) -> None:
59
if auto_port:
60
if "-p" in cli_args or "--port" in cli_args:
61
raise ValueError("You have manually specified the port"
@@ -68,12 +74,10 @@ class RemoteOpenAIServer:
74
# the current process might initialize cuda,
75
# to be safe, we should use spawn method
76
env['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn'
71
- self.proc = subprocess.Popen(
72
- [sys.executable, "-m", "vllm.entrypoints.openai.api_server"] +
73
- cli_args,
74
- env=env,
75
- stdout=sys.stdout,
76
- stderr=sys.stderr)
77
+ self.proc = subprocess.Popen(["vllm", "serve"] + [model] + cli_args,
78
+ env=env,
79
+ stdout=sys.stdout,
80
+ stderr=sys.stderr)
81
self._wait_for_server(url=self.url_for("health"),
82
timeout=self.MAX_SERVER_START_WAIT_S)
83