@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 5bf35a91e4
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 20:45 UTC

[Doc][CI/Build] Update docs and tests to use `vllm serve` (#6431)

Cyrus Leung committed Jul 17, 2024 at 15:43 UTC 5bf35a91e465278b51b0c98fa1b13beb8b04d431
23 files changed +155 -175
docs/source/getting_started/quickstart.rst
+2 -5
@@ -73,16 +73,13 @@ Start the server:
73
74 .. code-block:: console
75
76 - $ python -m vllm.entrypoints.openai.api_server \
77 - $ --model facebook/opt-125m
76 + $ vllm serve facebook/opt-125m
77
78 By default, the server uses a predefined chat template stored in the tokenizer. You can override this template by using the ``--chat-template`` argument:
79
80 .. code-block:: console
81
83 - $ python -m vllm.entrypoints.openai.api_server \
84 - $ --model facebook/opt-125m \
85 - $ --chat-template ./examples/template_chatml.jinja
82 + $ vllm serve facebook/opt-125m --chat-template ./examples/template_chatml.jinja
83
84 This server can be queried in the same format as OpenAI API. For example, list the models:
85
docs/source/models/adding_model.rst
+2 -2
@@ -114,7 +114,7 @@ Just add the following lines in your code:
114 from your_code import YourModelForCausalLM
115 ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
116
117 -If you are running api server with `python -m vllm.entrypoints.openai.api_server args`, you can wrap the entrypoint with the following code:
117 +If you are running api server with :code:`vllm serve <args>`, you can wrap the entrypoint with the following code:
118
119 .. code-block:: python
120
@@ -124,4 +124,4 @@ If you are running api server with `python -m vllm.entrypoints.openai.api_server
124 import runpy
125 runpy.run_module('vllm.entrypoints.openai.api_server', run_name='__main__')
126
127 -Save the above code in a file and run it with `python your_file.py args`.
127 +Save the above code in a file and run it with :code:`python your_file.py <args>`.
docs/source/models/engine_args.rst
+2 -2
@@ -8,7 +8,7 @@ Below, you can find an explanation of every engine argument for vLLM:
8 .. argparse::
9 :module: vllm.engine.arg_utils
10 :func: _engine_args_parser
11 - :prog: -m vllm.entrypoints.openai.api_server
11 + :prog: vllm serve
12 :nodefaultconst:
13
14 Async Engine Arguments
@@ -19,5 +19,5 @@ Below are the additional arguments related to the asynchronous engine:
19 .. argparse::
20 :module: vllm.engine.arg_utils
21 :func: _async_engine_args_parser
22 - :prog: -m vllm.entrypoints.openai.api_server
22 + :prog: vllm serve
23 :nodefaultconst:
\ No newline at end of file
docs/source/models/lora.rst
+1 -2
@@ -61,8 +61,7 @@ LoRA adapted models can also be served with the Open-AI compatible vLLM server.
61
62 .. code-block:: bash
63
64 - python -m vllm.entrypoints.openai.api_server \
65 - --model meta-llama/Llama-2-7b-hf \
64 + vllm serve meta-llama/Llama-2-7b-hf \
65 --enable-lora \
66 --lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/
67
docs/source/models/vlm.rst
+1 -3
@@ -94,9 +94,7 @@ Below is an example on how to launch the same ``llava-hf/llava-1.5-7b-hf`` with
94
95 .. code-block:: bash
96
97 - python -m vllm.entrypoints.openai.api_server \
98 - --model llava-hf/llava-1.5-7b-hf \
99 - --chat-template template_llava.jinja
97 + vllm serve llava-hf/llava-1.5-7b-hf --chat-template template_llava.jinja
98
99 .. important::
100 We have removed all vision language related CLI args in the ``0.5.1`` release. **This is a breaking change**, so please update your code to follow
docs/source/serving/deploying_with_dstack.rst
+1 -1
@@ -40,7 +40,7 @@ Next, to provision a VM instance with LLM of your choice(`NousResearch/Llama-2-7
40 gpu: 24GB
41 commands:
42 - pip install vllm
43 - - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000
43 + - vllm serve $MODEL --port 8000
44 model:
45 format: openai
46 type: chat
docs/source/serving/distributed_serving.rst
+2 -4
@@ -35,16 +35,14 @@ To run multi-GPU serving, pass in the :code:`--tensor-parallel-size` argument wh
35
36 .. code-block:: console
37
38 - $ python -m vllm.entrypoints.openai.api_server \
39 - $ --model facebook/opt-13b \
38 + $ vllm serve facebook/opt-13b \
39 $ --tensor-parallel-size 4
40
41 You can also additionally specify :code:`--pipeline-parallel-size` to enable pipeline parallelism. For example, to run API server on 8 GPUs with pipeline parallelism and tensor parallelism:
42
43 .. code-block:: console
44
46 - $ python -m vllm.entrypoints.openai.api_server \
47 - $ --model gpt2 \
45 + $ vllm serve gpt2 \
46 $ --tensor-parallel-size 4 \
47 $ --pipeline-parallel-size 2 \
48 $ --distributed-executor-backend ray
docs/source/serving/openai_compatible_server.md
+3 -5
@@ -4,7 +4,7 @@ vLLM provides an HTTP server that implements OpenAI's [Completions](https://plat
4
5 You can start the server using Python, or using [Docker](deploying_with_docker.rst):
6 ```bash
7 -python -m vllm.entrypoints.openai.api_server --model NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
7 +vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
8 ```
9
10 To call the server, you can use the official OpenAI Python client library, or any other HTTP client.
@@ -97,9 +97,7 @@ template, or the template in string form. Without a chat template, the server wi
97 and all chat requests will error.
98
99 ```bash
100 -python -m vllm.entrypoints.openai.api_server \
101 - --model ... \
102 - --chat-template ./path-to-chat-template.jinja
100 +vllm serve <model> --chat-template ./path-to-chat-template.jinja
101 ```
102
103 vLLM community provides a set of chat templates for popular models. You can find them in the examples
@@ -110,7 +108,7 @@ directory [here](https://github.com/vllm-project/vllm/tree/main/examples/)
108 ```{argparse}
109 :module: vllm.entrypoints.openai.cli_args
110 :func: create_parser_for_docs
113 -:prog: -m vllm.entrypoints.openai.api_server
111 +:prog: vllm serve
112 ```
113
114 ## Tool calling in the chat completion API
examples/api_client.py
+2 -3
@@ -1,8 +1,7 @@
1 -"""Example Python client for vllm.entrypoints.api_server
1 +"""Example Python client for `vllm.entrypoints.api_server`
2 NOTE: The API server is used only for demonstration and simple performance
3 benchmarks. It is not intended for production use.
4 -For production use, we recommend vllm.entrypoints.openai.api_server
5 -and the OpenAI client API
4 +For production use, we recommend `vllm serve` and the OpenAI client API.
5 """
6
7 import argparse
examples/logging_configuration.md
+3 -9
@@ -95,9 +95,7 @@ to the path of the custom logging configuration JSON file:
95
96 ```bash
97 VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
98 - python3 -m vllm.entrypoints.openai.api_server \
99 - --max-model-len 2048 \
100 - --model mistralai/Mistral-7B-v0.1
98 + vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048
99 ```
100
101
@@ -152,9 +150,7 @@ to the path of the custom logging configuration JSON file:
150
151 ```bash
152 VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \
155 - python3 -m vllm.entrypoints.openai.api_server \
156 - --max-model-len 2048 \
157 - --model mistralai/Mistral-7B-v0.1
153 + vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048
154 ```
155
156
@@ -167,9 +163,7 @@ loggers.
163
164 ```bash
165 VLLM_CONFIGURE_LOGGING=0 \
170 - python3 -m vllm.entrypoints.openai.api_server \
171 - --max-model-len 2048 \
172 - --model mistralai/Mistral-7B-v0.1
166 + vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048
167 ```
168
169
examples/openai_vision_api_client.py
+1 -3
@@ -1,9 +1,7 @@
1 """An example showing how to use vLLM to serve VLMs.
2
3 Launch the vLLM server with the following command:
4 -python -m vllm.entrypoints.openai.api_server \
5 - --model llava-hf/llava-1.5-7b-hf \
6 - --chat-template template_llava.jinja
4 +vllm serve llava-hf/llava-1.5-7b-hf --chat-template template_llava.jinja
5 """
6 import base64
7
examples/production_monitoring/Otel.md
+3 -3
@@ -36,7 +36,7 @@
36 ```
37 export OTEL_SERVICE_NAME="vllm-server"
38 export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
39 - python -m vllm.entrypoints.openai.api_server --model="facebook/opt-125m" --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
39 + vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
40 ```
41
42 1. In a new shell, send requests with trace context from a dummy client
@@ -62,7 +62,7 @@ By default, `grpc` is used. To set `http/protobuf` as the protocol, configure th
62 ```
63 export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
64 export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
65 -python -m vllm.entrypoints.openai.api_server --model="facebook/opt-125m" --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
65 +vllm serve facebook/opt-125m --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
66 ```
67
68 ## Instrumentation of FastAPI
@@ -74,7 +74,7 @@ OpenTelemetry allows automatic instrumentation of FastAPI.
74
75 1. Run vLLM with `opentelemetry-instrument`
76 ```
77 - opentelemetry-instrument python -m vllm.entrypoints.openai.api_server --model="facebook/opt-125m"
77 + opentelemetry-instrument vllm serve facebook/opt-125m
78 ```
79
80 1. Send a request to vLLM and find its trace in Jaeger. It should contain spans from FastAPI.
examples/production_monitoring/README.md
+1 -2
@@ -10,8 +10,7 @@ Install:
10
11 Prometheus metric logging is enabled by default in the OpenAI-compatible server. Launch via the entrypoint:
12 ```bash
13 -python3 -m vllm.entrypoints.openai.api_server \
14 - --model mistralai/Mistral-7B-v0.1 \
13 +vllm serve mistralai/Mistral-7B-v0.1 \
14 --max-model-len 2048 \
15 --disable-log-requests
16 ```
tests/async_engine/test_openapi_server_ray.py
+11 -11
@@ -9,17 +9,17 @@ MODEL_NAME = "facebook/opt-125m"
9
10 @pytest.fixture(scope="module")
11 def server():
12 - with RemoteOpenAIServer([
13 - "--model",
14 - MODEL_NAME,
15 - # use half precision for speed and memory savings in CI environment
16 - "--dtype",
17 - "float16",
18 - "--max-model-len",
19 - "2048",
20 - "--enforce-eager",
21 - "--engine-use-ray"
22 - ]) as remote_server:
12 + args = [
13 + # use half precision for speed and memory savings in CI environment
14 + "--dtype",
15 + "float16",
16 + "--max-model-len",
17 + "2048",
18 + "--enforce-eager",
19 + "--engine-use-ray"
20 + ]
21 +
22 + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
23 yield remote_server
24
25
tests/distributed/test_pipeline_parallel.py
+1 -5
@@ -15,8 +15,6 @@ from ..utils import RemoteOpenAIServer
15 ])
16 def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL, MODEL_NAME):
17 pp_args = [
18 - "--model",
19 - MODEL_NAME,
18 # use half precision for speed and memory savings in CI environment
19 "--dtype",
20 "bfloat16",
@@ -34,8 +32,6 @@ def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL, MODEL_NAME):
32 # schedule all workers in a node other than the head node,
33 # which can cause the test to fail.
34 tp_args = [
37 - "--model",
38 - MODEL_NAME,
35 # use half precision for speed and memory savings in CI environment
36 "--dtype",
37 "bfloat16",
@@ -53,7 +49,7 @@ def test_compare_tp(TP_SIZE, PP_SIZE, EAGER_MODE, CHUNKED_PREFILL, MODEL_NAME):
49
50 results = []
51 for args in [pp_args, tp_args]:
56 - with RemoteOpenAIServer(args) as server:
52 + with RemoteOpenAIServer(MODEL_NAME, args) as server:
53 client = server.get_client()
54
55 # test models list
tests/entrypoints/openai/test_chat.py
+21 -21
@@ -27,27 +27,27 @@ def zephyr_lora_files():
27
28 @pytest.fixture(scope="module")
29 def server(zephyr_lora_files):
30 - with RemoteOpenAIServer([
31 - "--model",
32 - MODEL_NAME,
33 - # use half precision for speed and memory savings in CI environment
34 - "--dtype",
35 - "bfloat16",
36 - "--max-model-len",
37 - "8192",
38 - "--enforce-eager",
39 - # lora config below
40 - "--enable-lora",
41 - "--lora-modules",
42 - f"zephyr-lora={zephyr_lora_files}",
43 - f"zephyr-lora2={zephyr_lora_files}",
44 - "--max-lora-rank",
45 - "64",
46 - "--max-cpu-loras",
47 - "2",
48 - "--max-num-seqs",
49 - "128",
50 - ]) as remote_server:
30 + args = [
31 + # use half precision for speed and memory savings in CI environment
32 + "--dtype",
33 + "bfloat16",
34 + "--max-model-len",
35 + "8192",
36 + "--enforce-eager",
37 + # lora config below
38 + "--enable-lora",
39 + "--lora-modules",
40 + f"zephyr-lora={zephyr_lora_files}",
41 + f"zephyr-lora2={zephyr_lora_files}",
42 + "--max-lora-rank",
43 + "64",
44 + "--max-cpu-loras",
45 + "2",
46 + "--max-num-seqs",
47 + "128",
48 + ]
49 +
50 + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
51 yield remote_server
52
53
tests/entrypoints/openai/test_completion.py
+30 -30
@@ -37,36 +37,36 @@ def zephyr_pa_files():
37
38 @pytest.fixture(scope="module")
39 def server(zephyr_lora_files, zephyr_pa_files):
40 - with RemoteOpenAIServer([
41 - "--model",
42 - MODEL_NAME,
43 - # use half precision for speed and memory savings in CI environment
44 - "--dtype",
45 - "bfloat16",
46 - "--max-model-len",
47 - "8192",
48 - "--max-num-seqs",
49 - "128",
50 - "--enforce-eager",
51 - # lora config
52 - "--enable-lora",
53 - "--lora-modules",
54 - f"zephyr-lora={zephyr_lora_files}",
55 - f"zephyr-lora2={zephyr_lora_files}",
56 - "--max-lora-rank",
57 - "64",
58 - "--max-cpu-loras",
59 - "2",
60 - # pa config
61 - "--enable-prompt-adapter",
62 - "--prompt-adapters",
63 - f"zephyr-pa={zephyr_pa_files}",
64 - f"zephyr-pa2={zephyr_pa_files}",
65 - "--max-prompt-adapters",
66 - "2",
67 - "--max-prompt-adapter-token",
68 - "128",
69 - ]) as remote_server:
40 + args = [
41 + # use half precision for speed and memory savings in CI environment
42 + "--dtype",
43 + "bfloat16",
44 + "--max-model-len",
45 + "8192",
46 + "--max-num-seqs",
47 + "128",
48 + "--enforce-eager",
49 + # lora config
50 + "--enable-lora",
51 + "--lora-modules",
52 + f"zephyr-lora={zephyr_lora_files}",
53 + f"zephyr-lora2={zephyr_lora_files}",
54 + "--max-lora-rank",
55 + "64",
56 + "--max-cpu-loras",
57 + "2",
58 + # pa config
59 + "--enable-prompt-adapter",
60 + "--prompt-adapters",
61 + f"zephyr-pa={zephyr_pa_files}",
62 + f"zephyr-pa2={zephyr_pa_files}",
63 + "--max-prompt-adapters",
64 + "2",
65 + "--max-prompt-adapter-token",
66 + "128",
67 + ]
68 +
69 + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
70 yield remote_server
71
72
tests/entrypoints/openai/test_embedding.py
+11 -11
@@ -11,17 +11,17 @@ EMBEDDING_MODEL_NAME = "intfloat/e5-mistral-7b-instruct"
11
12 @pytest.fixture(scope="module")
13 def embedding_server():
14 - with RemoteOpenAIServer([
15 - "--model",
16 - EMBEDDING_MODEL_NAME,
17 - # use half precision for speed and memory savings in CI environment
18 - "--dtype",
19 - "bfloat16",
20 - "--enforce-eager",
21 - "--max-model-len",
22 - "8192",
23 - "--enforce-eager",
24 - ]) as remote_server:
14 + args = [
15 + # use half precision for speed and memory savings in CI environment
16 + "--dtype",
17 + "bfloat16",
18 + "--enforce-eager",
19 + "--max-model-len",
20 + "8192",
21 + "--enforce-eager",
22 + ]
23 +
24 + with RemoteOpenAIServer(EMBEDDING_MODEL_NAME, args) as remote_server:
25 yield remote_server
26
27
tests/entrypoints/openai/test_models.py
+21 -21
@@ -19,27 +19,27 @@ def zephyr_lora_files():
19
20 @pytest.fixture(scope="module")
21 def server(zephyr_lora_files):
22 - with RemoteOpenAIServer([
23 - "--model",
24 - MODEL_NAME,
25 - # use half precision for speed and memory savings in CI environment
26 - "--dtype",
27 - "bfloat16",
28 - "--max-model-len",
29 - "8192",
30 - "--enforce-eager",
31 - # lora config below
32 - "--enable-lora",
33 - "--lora-modules",
34 - f"zephyr-lora={zephyr_lora_files}",
35 - f"zephyr-lora2={zephyr_lora_files}",
36 - "--max-lora-rank",
37 - "64",
38 - "--max-cpu-loras",
39 - "2",
40 - "--max-num-seqs",
41 - "128",
42 - ]) as remote_server:
22 + args = [
23 + # use half precision for speed and memory savings in CI environment
24 + "--dtype",
25 + "bfloat16",
26 + "--max-model-len",
27 + "8192",
28 + "--enforce-eager",
29 + # lora config below
30 + "--enable-lora",
31 + "--lora-modules",
32 + f"zephyr-lora={zephyr_lora_files}",
33 + f"zephyr-lora2={zephyr_lora_files}",
34 + "--max-lora-rank",
35 + "64",
36 + "--max-cpu-loras",
37 + "2",
38 + "--max-num-seqs",
39 + "128",
40 + ]
41 +
42 + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
43 yield remote_server
44
45
tests/entrypoints/openai/test_tokenization.py
+12 -12
@@ -12,18 +12,18 @@ MODEL_NAME = "HuggingFaceH4/zephyr-7b-beta"
12
13 @pytest.fixture(scope="module")
14 def server():
15 - with RemoteOpenAIServer([
16 - "--model",
17 - MODEL_NAME,
18 - # use half precision for speed and memory savings in CI environment
19 - "--dtype",
20 - "bfloat16",
21 - "--max-model-len",
22 - "8192",
23 - "--enforce-eager",
24 - "--max-num-seqs",
25 - "128",
26 - ]) as remote_server:
15 + args = [
16 + # use half precision for speed and memory savings in CI environment
17 + "--dtype",
18 + "bfloat16",
19 + "--max-model-len",
20 + "8192",
21 + "--enforce-eager",
22 + "--max-num-seqs",
23 + "128",
24 + ]
25 +
26 + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
27 yield remote_server
28
29
tests/entrypoints/openai/test_vision.py
+11 -11
@@ -23,17 +23,17 @@ TEST_IMAGE_URLS = [
23
24 @pytest.fixture(scope="module")
25 def server():
26 - with RemoteOpenAIServer([
27 - "--model",
28 - MODEL_NAME,
29 - "--dtype",
30 - "bfloat16",
31 - "--max-model-len",
32 - "4096",
33 - "--enforce-eager",
34 - "--chat-template",
35 - str(LLAVA_CHAT_TEMPLATE),
36 - ]) as remote_server:
26 + args = [
27 + "--dtype",
28 + "bfloat16",
29 + "--max-model-len",
30 + "4096",
31 + "--enforce-eager",
32 + "--chat-template",
33 + str(LLAVA_CHAT_TEMPLATE),
34 + ]
35 +
36 + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
37 yield remote_server
38
39
tests/tensorizer_loader/test_tensorizer.py
+2 -2
@@ -214,12 +214,12 @@ def test_openai_apiserver_with_tensorizer(vllm_runner, tmp_path):
214
215 ## Start OpenAI API server
216 openai_args = [
217 - "--model", model_ref, "--dtype", "float16", "--load-format",
217 + "--dtype", "float16", "--load-format",
218 "tensorizer", "--model-loader-extra-config",
219 json.dumps(model_loader_extra_config),
220 ]
221
222 - with RemoteOpenAIServer(openai_args) as server:
222 + with RemoteOpenAIServer(model_ref, openai_args) as server:
223 print("Server ready.")
224
225 client = server.get_client()
tests/utils.py
+11 -7
@@ -49,7 +49,13 @@ class RemoteOpenAIServer:
49 DUMMY_API_KEY = "token-abc123" # vLLM's OpenAI server does not need API key
50 MAX_SERVER_START_WAIT_S = 600 # wait for server to start for 60 seconds
51
52 - def __init__(self, cli_args: List[str], *, auto_port: bool = True) -> None:
52 + def __init__(
53 + self,
54 + model: str,
55 + cli_args: List[str],
56 + *,
57 + auto_port: bool = True,
58 + ) -> None:
59 if auto_port:
60 if "-p" in cli_args or "--port" in cli_args:
61 raise ValueError("You have manually specified the port"
@@ -68,12 +74,10 @@ class RemoteOpenAIServer:
74 # the current process might initialize cuda,
75 # to be safe, we should use spawn method
76 env['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn'
71 - self.proc = subprocess.Popen(
72 - [sys.executable, "-m", "vllm.entrypoints.openai.api_server"] +
73 - cli_args,
74 - env=env,
75 - stdout=sys.stdout,
76 - stderr=sys.stderr)
77 + self.proc = subprocess.Popen(["vllm", "serve"] + [model] + cli_args,
78 + env=env,
79 + stdout=sys.stdout,
80 + stderr=sys.stderr)
81 self._wait_for_server(url=self.url_for("health"),
82 timeout=self.MAX_SERVER_START_WAIT_S)
83