@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 228cfbd03f
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 09:30 UTC

[Doc] Improve quickstart documentation (#9256)

Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>

Rafael Vasquez committed Oct 25, 2024 at 17:32 UTC 228cfbd03fd1ad9b26001817a6d414cc9f2c22ae
1 file changed +52 -46
docs/source/getting_started/quickstart.rst
+52 -46
@@ -1,38 +1,50 @@
1 .. _quickstart:
2
3 +==========
4 Quickstart
5 ==========
6
6 -This guide shows how to use vLLM to:
7 +This guide will help you quickly get started with vLLM to:
8
8 -* run offline batched inference on a dataset;
9 -* build an API server for a large language model;
10 -* start an OpenAI-compatible API server.
9 +* :ref:`Run offline batched inference <offline_batched_inference>`
10 +* :ref:`Run OpenAI-compatible inference <openai_compatible_server>`
11
12 -Be sure to complete the :ref:`installation instructions <installation>` before continuing with this guide.
12 +Prerequisites
13 +--------------
14 +- OS: Linux
15 +- Python: 3.8 - 3.12
16 +- GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
17
14 -.. note::
18 +Installation
19 +--------------
20 +
21 +You can install vLLM using pip. It's recommended to use `conda <https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html>`_ to create and manage Python environments.
22 +
23 +.. code-block:: console
24
16 - By default, vLLM downloads model from `HuggingFace <https://huggingface.co/>`_. If you would like to use models from `ModelScope <https://www.modelscope.cn>`_ in the following examples, please set the environment variable:
25 + $ conda create -n myenv python=3.10 -y
26 + $ conda activate myenv
27 + $ pip install vllm
28
18 - .. code-block:: shell
29 +Please refer to the :ref:`installation documentation <installation>` for more details on installing vLLM.
30
20 - export VLLM_USE_MODELSCOPE=True
31 +.. _offline_batched_inference:
32
33 Offline Batched Inference
34 -------------------------
35
25 -We first show an example of using vLLM for offline batched inference on a dataset. In other words, we use vLLM to generate texts for a list of input prompts.
36 +With vLLM installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing). The example script for this section can be found `here <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference.py>`__.
37 +
38 +The first line of this example imports the classes :class:`~vllm.LLM` and :class:`~vllm.SamplingParams`:
39
27 -Import :class:`~vllm.LLM` and :class:`~vllm.SamplingParams` from vLLM.
28 -The :class:`~vllm.LLM` class is the main class for running offline inference with vLLM engine.
29 -The :class:`~vllm.SamplingParams` class specifies the parameters for the sampling process.
40 +- :class:`~vllm.LLM` is the main class for running offline inference with vLLM engine.
41 +- :class:`~vllm.SamplingParams` specifies the parameters for the sampling process.
42
43 .. code-block:: python
44
45 from vllm import LLM, SamplingParams
46
35 -Define the list of input prompts and the sampling parameters for generation. The sampling temperature is set to 0.8 and the nucleus sampling probability is set to 0.95. For more information about the sampling parameters, refer to the `class definition <https://github.com/vllm-project/vllm/blob/main/vllm/sampling_params.py>`_.
47 +The next section defines a list of input prompts and sampling parameters for text generation. The `sampling temperature <https://arxiv.org/html/2402.05201v1>`_ is set to ``0.8`` and the `nucleus sampling probability <https://en.wikipedia.org/wiki/Top-p_sampling>`_ is set to ``0.95``. You can find more information about the sampling parameters `here <https://docs.vllm.ai/en/stable/dev/sampling_params.html>`__.
48
49 .. code-block:: python
50
@@ -44,46 +56,46 @@ Define the list of input prompts and the sampling parameters for generation. The
56 ]
57 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
58
47 -Initialize vLLM's engine for offline inference with the :class:`~vllm.LLM` class and the `OPT-125M model <https://arxiv.org/abs/2205.01068>`_. The list of supported models can be found at :ref:`supported models <supported_models>`.
59 +The :class:`~vllm.LLM` class initializes vLLM's engine and the `OPT-125M model <https://arxiv.org/abs/2205.01068>`_ for offline inference. The list of supported models can be found :ref:`here <supported_models>`.
60
61 .. code-block:: python
62
63 llm = LLM(model="facebook/opt-125m")
64
53 -Call ``llm.generate`` to generate the outputs. It adds the input prompts to vLLM engine's waiting queue and executes the vLLM engine to generate the outputs with high throughput. The outputs are returned as a list of ``RequestOutput`` objects, which include all the output tokens.
65 +.. note::
66 +
67 + By default, vLLM downloads models from `HuggingFace <https://huggingface.co/>`_. If you would like to use models from `ModelScope <https://www.modelscope.cn>`_, set the environment variable ``VLLM_USE_MODELSCOPE`` before initializing the engine.
68 +
69 +Now, the fun part! The outputs are generated using ``llm.generate``. It adds the input prompts to the vLLM engine's waiting queue and executes the vLLM engine to generate the outputs with high throughput. The outputs are returned as a list of ``RequestOutput`` objects, which include all of the output tokens.
70
71 .. code-block:: python
72
73 outputs = llm.generate(prompts, sampling_params)
74
59 - # Print the outputs.
75 for output in outputs:
76 prompt = output.prompt
77 generated_text = output.outputs[0].text
78 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
79
65 -
66 -The code example can also be found in `examples/offline_inference.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference.py>`_.
80 +.. _openai_compatible_server:
81
82 OpenAI-Compatible Server
83 ------------------------
84
85 vLLM can be deployed as a server that implements the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API.
72 -By default, it starts the server at ``http://localhost:8000``. You can specify the address with ``--host`` and ``--port`` arguments. The server currently hosts one model at a time (OPT-125M in the command below) and implements `list models <https://platform.openai.com/docs/api-reference/models/list>`_, `create chat completion <https://platform.openai.com/docs/api-reference/chat/completions/create>`_, and `create completion <https://platform.openai.com/docs/api-reference/completions/create>`_ endpoints. We are actively adding support for more endpoints.
86 +By default, it starts the server at ``http://localhost:8000``. You can specify the address with ``--host`` and ``--port`` arguments. The server currently hosts one model at a time and implements endpoints such as `list models <https://platform.openai.com/docs/api-reference/models/list>`_, `create chat completion <https://platform.openai.com/docs/api-reference/chat/completions/create>`_, and `create completion <https://platform.openai.com/docs/api-reference/completions/create>`_ endpoints.
87
74 -Start the server:
88 +Run the following command to start the vLLM server with the `Qwen2.5-1.5B-Instruct <https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct>`_ model:
89
90 .. code-block:: console
91
78 - $ vllm serve facebook/opt-125m
92 + $ vllm serve Qwen/Qwen2.5-1.5B-Instruct
93
80 -By default, the server uses a predefined chat template stored in the tokenizer. You can override this template by using the ``--chat-template`` argument:
81 -
82 -.. code-block:: console
94 +.. note::
95
84 - $ vllm serve facebook/opt-125m --chat-template ./examples/template_chatml.jinja
96 + By default, the server uses a predefined chat template stored in the tokenizer. You can learn about overriding it `here <https://github.com/vllm-project/vllm/blob/main/docs/source/serving/openai_compatible_server.md#chat-template>`__.
97
86 -This server can be queried in the same format as OpenAI API. For example, list the models:
98 +This server can be queried in the same format as OpenAI API. For example, to list the models:
99
100 .. code-block:: console
101
@@ -91,17 +103,17 @@ This server can be queried in the same format as OpenAI API. For example, list t
103
104 You can pass in the argument ``--api-key`` or environment variable ``VLLM_API_KEY`` to enable the server to check for API key in the header.
105
94 -Using OpenAI Completions API with vLLM
95 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
106 +OpenAI Completions API with vLLM
107 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
108
97 -Query the model with input prompts:
109 +Once your server is started, you can query the model with input prompts:
110
111 .. code-block:: console
112
113 $ curl http://localhost:8000/v1/completions \
114 $ -H "Content-Type: application/json" \
115 $ -d '{
104 - $ "model": "facebook/opt-125m",
116 + $ "model": "Qwen/Qwen2.5-1.5B-Instruct",
117 $ "prompt": "San Francisco is a",
118 $ "max_tokens": 7,
119 $ "temperature": 0
@@ -120,36 +132,32 @@ Since this server is compatible with OpenAI API, you can use it as a drop-in rep
132 api_key=openai_api_key,
133 base_url=openai_api_base,
134 )
123 - completion = client.completions.create(model="facebook/opt-125m",
135 + completion = client.completions.create(model="Qwen/Qwen2.5-1.5B-Instruct",
136 prompt="San Francisco is a")
137 print("Completion result:", completion)
138
127 -For a more detailed client example, refer to `examples/openai_completion_client.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py>`_.
128 -
129 -Using OpenAI Chat API with vLLM
130 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
139 +A more detailed client example can be found `here <https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py>`__.
140
132 -The vLLM server is designed to support the OpenAI Chat API, allowing you to engage in dynamic conversations with the model. The chat interface is a more interactive way to communicate with the model, allowing back-and-forth exchanges that can be stored in the chat history. This is useful for tasks that require context or more detailed explanations.
141 +OpenAI Chat API with vLLM
142 +~~~~~~~~~~~~~~~~~~~~~~~~~~
143
134 -Querying the model using OpenAI Chat API:
144 +vLLM is designed to also support the OpenAI Chat API. The chat interface is a more dynamic, interactive way to communicate with the model, allowing back-and-forth exchanges that can be stored in the chat history. This is useful for tasks that require context or more detailed explanations.
145
136 -You can use the `create chat completion <https://platform.openai.com/docs/api-reference/chat/completions/create>`_ endpoint to communicate with the model in a chat-like interface:
146 +You can use the `create chat completion <https://platform.openai.com/docs/api-reference/chat/completions/create>`_ endpoint to interact with the model:
147
148 .. code-block:: console
149
150 $ curl http://localhost:8000/v1/chat/completions \
151 $ -H "Content-Type: application/json" \
152 $ -d '{
143 - $ "model": "facebook/opt-125m",
153 + $ "model": "Qwen/Qwen2.5-1.5B-Instruct",
154 $ "messages": [
155 $ {"role": "system", "content": "You are a helpful assistant."},
156 $ {"role": "user", "content": "Who won the world series in 2020?"}
157 $ ]
158 $ }'
159
150 -Python Client Example:
151 -
152 -Using the `openai` python package, you can also communicate with the model in a chat-like manner:
160 +Alternatively, you can use the `openai` python package:
161
162 .. code-block:: python
163
@@ -164,12 +172,10 @@ Using the `openai` python package, you can also communicate with the model in a
172 )
173
174 chat_response = client.chat.completions.create(
167 - model="facebook/opt-125m",
175 + model="Qwen/Qwen2.5-1.5B-Instruct",
176 messages=[
177 {"role": "system", "content": "You are a helpful assistant."},
178 {"role": "user", "content": "Tell me a joke."},
179 ]
180 )
181 print("Chat response:", chat_response)
174 -
175 -For more in-depth examples and advanced features of the chat API, you can refer to the official OpenAI documentation.