@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / f745847ef7
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 05:30 UTC

[Minor] Fix the format in quick start guide related to Model Scope (#2425)

Zhuohan Li committed Jan 11, 2024 at 19:44 UTC f745847ef7401744f25ca68681d6e0bc939a8a67
1 file changed +8 -27
docs/source/getting_started/quickstart.rst
+8 -27
@@ -11,6 +11,14 @@ This guide shows how to use vLLM to:
11
12 Be sure to complete the :ref:`installation instructions <installation>` before continuing with this guide.
13
14 +.. note::
15 +
16 + By default, vLLM downloads model from `HuggingFace <https://huggingface.co/>`_. If you would like to use models from `ModelScope <https://www.modelscope.cn>`_ in the following examples, please set the environment variable:
17 +
18 + .. code-block:: shell
19 +
20 + export VLLM_USE_MODELSCOPE=True
21 +
22 Offline Batched Inference
23 -------------------------
24
@@ -40,16 +48,6 @@ Initialize vLLM's engine for offline inference with the ``LLM`` class and the `O
48
49 llm = LLM(model="facebook/opt-125m")
50
43 -Use model from www.modelscope.cn
44 -
45 -.. code-block:: shell
46 -
47 - export VLLM_USE_MODELSCOPE=True
48 -
49 -.. code-block:: python
50 -
51 - llm = LLM(model="qwen/Qwen-7B-Chat", revision="v1.1.8", trust_remote_code=True)
52 -
51 Call ``llm.generate`` to generate the outputs. It adds the input prompts to vLLM engine's waiting queue and executes the vLLM engine to generate the outputs with high throughput. The outputs are returned as a list of ``RequestOutput`` objects, which include all the output tokens.
52
53 .. code-block:: python
@@ -77,16 +75,6 @@ Start the server:
75
76 $ python -m vllm.entrypoints.api_server
77
80 -Use model from www.modelscope.cn
81 -
82 -.. code-block:: console
83 -
84 - $ VLLM_USE_MODELSCOPE=True python -m vllm.entrypoints.api_server \
85 - $ --model="qwen/Qwen-7B-Chat" \
86 - $ --revision="v1.1.8" \
87 - $ --trust-remote-code
88 -
89 -
78 By default, this command starts the server at ``http://localhost:8000`` with the OPT-125M model.
79
80 Query the model in shell:
@@ -116,13 +104,6 @@ Start the server:
104 $ python -m vllm.entrypoints.openai.api_server \
105 $ --model facebook/opt-125m
106
119 -Use model from www.modelscope.cn
120 -
121 -.. code-block:: console
122 -
123 - $ VLLM_USE_MODELSCOPE=True python -m vllm.entrypoints.openai.api_server \
124 - $ --model="qwen/Qwen-7B-Chat" --revision="v1.1.8" --trust-remote-code
125 -
107 By default, the server uses a predefined chat template stored in the tokenizer. You can override this template by using the ``--chat-template`` argument:
108
109 .. code-block:: console