A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
[Minor] Fix the format in quick start guide related to Model Scope (#2425)
Zhuohan Li committed
Jan 11, 2024 at 19:44 UTC
f745847ef7401744f25ca68681d6e0bc939a8a67
1 file changed
+8
-27
docs/source/getting_started/quickstart.rst
+8
-27
@@ -11,6 +11,14 @@ This guide shows how to use vLLM to:
11
12
Be sure to complete the :ref:`installation instructions <installation>` before continuing with this guide.
13
14
+.. note::
15
+
16
+ By default, vLLM downloads model from `HuggingFace <https://huggingface.co/>`_. If you would like to use models from `ModelScope <https://www.modelscope.cn>`_ in the following examples, please set the environment variable:
17
+
18
+ .. code-block:: shell
19
+
20
+ export VLLM_USE_MODELSCOPE=True
21
+
22
Offline Batched Inference
23
-------------------------
24
@@ -40,16 +48,6 @@ Initialize vLLM's engine for offline inference with the ``LLM`` class and the `O
48
49
llm = LLM(model="facebook/opt-125m")
50
43
-Use model from www.modelscope.cn
44
-
45
-.. code-block:: shell
46
-
47
- export VLLM_USE_MODELSCOPE=True
48
-
49
-.. code-block:: python
50
-
51
- llm = LLM(model="qwen/Qwen-7B-Chat", revision="v1.1.8", trust_remote_code=True)
52
-
51
Call ``llm.generate`` to generate the outputs. It adds the input prompts to vLLM engine's waiting queue and executes the vLLM engine to generate the outputs with high throughput. The outputs are returned as a list of ``RequestOutput`` objects, which include all the output tokens.
52
53
.. code-block:: python
@@ -77,16 +75,6 @@ Start the server:
75
76
$ python -m vllm.entrypoints.api_server
77
80
-Use model from www.modelscope.cn
81
-
82
-.. code-block:: console
83
-
84
- $ VLLM_USE_MODELSCOPE=True python -m vllm.entrypoints.api_server \
85
- $ --model="qwen/Qwen-7B-Chat" \
86
- $ --revision="v1.1.8" \
87
- $ --trust-remote-code
88
-
89
-
78
By default, this command starts the server at ``http://localhost:8000`` with the OPT-125M model.
79
80
Query the model in shell:
@@ -116,13 +104,6 @@ Start the server:
104
$ python -m vllm.entrypoints.openai.api_server \
105
$ --model facebook/opt-125m
106
119
-Use model from www.modelscope.cn
120
-
121
-.. code-block:: console
122
-
123
- $ VLLM_USE_MODELSCOPE=True python -m vllm.entrypoints.openai.api_server \
124
- $ --model="qwen/Qwen-7B-Chat" --revision="v1.1.8" --trust-remote-code
125
-
107
By default, the server uses a predefined chat template stored in the tokenizer. You can override this template by using the ``--chat-template`` argument:
108
109
.. code-block:: console