@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / bec7b2dc26
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 09:15 UTC

Add quickstart guide (#148)

Zhuohan Li committed Jun 18, 2023 at 01:26 UTC bec7b2dc26c332563e9c0e53474a47c6c69b2a96
4 files changed +129 -9
docs/source/getting_started/installation.rst
+2
@@ -1,3 +1,5 @@
1 +.. _installation:
2 +
3 Installation
4 ============
5
docs/source/getting_started/quickstart.rst
+105 -8
@@ -1,30 +1,49 @@
1 +.. _quickstart:
2 +
3 Quickstart
4 ==========
5
4 -LLM
5 ----
6 +This guide shows how to use vLLM to:
7 +
8 +* run offline batched inference on a dataset;
9 +* build an API server for a large language model;
10 +* start an OpenAI-compatible API server.
11 +
12 +Be sure to complete the :ref:`installation instructions <installation>` before continuing with this guide.
13
7 -Placeholder.
14 +Offline Batched Inference
15 +-------------------------
16 +
17 +We first show an example of using vLLM for offline batched inference on a dataset. In other words, we use vLLM to generate texts for a list of input prompts.
18 +
19 +Import ``LLM`` and ``SamplingParams`` from vLLM. The ``LLM`` class is the main class for running offline inference with vLLM engine. The ``SamplingParams`` class specifies the parameters for the sampling process.
20
21 .. code-block:: python
22
23 from vllm import LLM, SamplingParams
24
13 - # Sample prompts.
25 +Define the list of input prompts and the sampling parameters for generation. The sampling temperature is set to 0.8 and the nucleus sampling probability is set to 0.95. For more information about the sampling parameters, refer to the `class definition <https://github.com/WoosukKwon/vllm/blob/main/vllm/sampling_params.py>`_.
26 +
27 +.. code-block:: python
28 +
29 prompts = [
30 "Hello, my name is",
31 "The president of the United States is",
32 "The capital of France is",
33 "The future of AI is",
34 ]
20 - # Create a sampling params object.
35 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
36
23 - # Create an LLM.
37 +Initialize vLLM's engine for offline inference with the ``LLM`` class and the `OPT-125M model <https://arxiv.org/abs/2205.01068>`_. The list of supported models can be found at :ref:`supported models <supported_models>`.
38 +
39 +.. code-block:: python
40 +
41 llm = LLM(model="facebook/opt-125m")
42
26 - # Generate texts from the prompts. The output is a list of RequestOutput objects
27 - # that contain the prompt, generated text, and other information.
43 +Call ``llm.generate`` to generate the outputs. It adds the input prompts to vLLM engine's waiting queue and executes the vLLM engine to generate the outputs with high throughput. The outputs are returned as a list of ``RequestOutput`` objects, which include all the output tokens.
44 +
45 +.. code-block:: python
46 +
47 outputs = llm.generate(prompts, sampling_params)
48
49 # Print the outputs.
@@ -32,3 +51,81 @@ Placeholder.
51 prompt = output.prompt
52 generated_text = output.outputs[0].text
53 print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
54 +
55 +
56 +The code example can also be found in `examples/offline_inference.py <https://github.com/WoosukKwon/vllm/blob/main/examples/offline_inference.py>`_.
57 +
58 +
59 +API Server
60 +----------
61 +
62 +vLLM can be deployed as an LLM service. We provide an example `FastAPI <https://fastapi.tiangolo.com/>`_ server. Check `vllm/entrypoints/api_server.py <https://github.com/WoosukKwon/vllm/blob/main/vllm/entrypoints/api_server.py>`_ for the server implementation. The server uses ``AsyncLLMEngine`` class to support asynchronous processing of incoming requests.
63 +
64 +Start the server:
65 +
66 +.. code-block:: console
67 +
68 + $ python -m vllm.entrypoints.api_server
69 +
70 +By default, this command starts the server at ``http://localhost:8000`` with the OPT-125M model.
71 +
72 +Query the model in shell:
73 +
74 +.. code-block:: console
75 +
76 + $ curl http://localhost:8000/generate \
77 + $ -d '{
78 + $ "prompt": "San Francisco is a",
79 + $ "use_beam_search": true,
80 + $ "n": 4,
81 + $ "temperature": 0
82 + $ }'
83 +
84 +See `examples/api_client.py <https://github.com/WoosukKwon/vllm/blob/main/examples/api_client.py>`_ for a more detailed client example.
85 +
86 +OpenAI-Compatible Server
87 +------------------------
88 +
89 +vLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API.
90 +
91 +Start the server:
92 +
93 +.. code-block:: console
94 +
95 + $ python -m vllm.entrypoints.openai.api_server \
96 + $ --model facebook/opt-125m
97 +
98 +By default, it starts the server at ``http://localhost:8000``. You can specify the address with ``--host`` and ``--port`` arguments. The server currently hosts one model at a time (OPT-125M in the above command) and implements `list models <https://platform.openai.com/docs/api-reference/models/list>`_ and `create completion <https://platform.openai.com/docs/api-reference/completions/create>`_ endpoints. We are actively adding support for more endpoints.
99 +
100 +This server can be queried in the same format as OpenAI API. For example, list the models:
101 +
102 +.. code-block:: console
103 +
104 + $ curl http://localhost:8000/v1/models
105 +
106 +Query the model with input prompts:
107 +
108 +.. code-block:: console
109 +
110 + $ curl http://localhost:8000/v1/completions \
111 + $ -H "Content-Type: application/json" \
112 + $ -d '{
113 + $ "model": "facebook/opt-125m",
114 + $ "prompt": "San Francisco is a",
115 + $ "max_tokens": 7,
116 + $ "temperature": 0
117 + $ }'
118 +
119 +Since this server is compatible with OpenAI API, you can use it as a drop-in replacement for any applications using OpenAI API. For example, another way to query the server is via the ``openai`` python package:
120 +
121 +.. code-block:: python
122 +
123 + import openai
124 + # Modify OpenAI's API key and API base to use vLLM's API server.
125 + openai.api_key = "EMPTY"
126 + openai.api_base = "http://localhost:8000/v1"
127 + completion = openai.Completion.create(model="facebook/opt-125m",
128 + prompt="San Francisco is a")
129 + print("Completion result:", completion)
130 +
131 +For a more detailed client example, refer to `examples/openai_client.py <https://github.com/WoosukKwon/vllm/blob/main/examples/openai_client.py>`_.
docs/source/index.rst
+2
@@ -1,6 +1,8 @@
1 Welcome to vLLM!
2 ================
3
4 +vLLM is a high-throughput and memory-efficient inference and serving engine for large language models (LLM).
5 +
6 Documentation
7 -------------
8
vllm/outputs.py
+20 -1
@@ -4,6 +4,18 @@ from vllm.sequence import SequenceGroup, SequenceStatus
4
5
6 class CompletionOutput:
7 + """The output data of one completion output of a request.
8 +
9 + Args:
10 + index: The index of the output in the request.
11 + text: The generated output text.
12 + token_ids: The token IDs of the generated output text.
13 + cumulative_logprob: The cumulative log probability of the generated
14 + output text.
15 + logprobs: The log probabilities of the top probability words at each
16 + position if the logprobs are requested.
17 + finish_reason: The reason why the sequence is finished.
18 + """
19
20 def __init__(
21 self,
@@ -11,7 +23,7 @@ class CompletionOutput:
23 text: str,
24 token_ids: List[int],
25 cumulative_logprob: float,
14 - logprobs: List[Dict[int, float]],
26 + logprobs: Optional[List[Dict[int, float]]],
27 finish_reason: Optional[str] = None,
28 ) -> None:
29 self.index = index
@@ -34,7 +46,14 @@ class CompletionOutput:
46
47
48 class RequestOutput:
49 + """The output data of a request to the LLM.
50
51 + Args:
52 + request_id: The unique ID of the request.
53 + prompt: The prompt string of the request.
54 + prompt_token_ids: The token IDs of the prompt.
55 + outputs: The output sequences of the request.
56 + """
57 def __init__(
58 self,
59 request_id: str,