A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 08:00 UTC
Fix repo & documentation URLs (#163)
Woosuk Kwon committed
Jun 19, 2023 at 20:03 UTC
b7e62d3454afc807ddcc2fe3a56ce075461f1bc4
7 files changed
+22
-22
README.md
+6
-6
@@ -10,7 +10,7 @@ Easy, fast, and cheap LLM serving for everyone
10
</h3>
11
12
<p align="center">
13
-| <a href="https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/"><b>Documentation</b></a> | <a href=""><b>Blog</b></a> |
13
+| <a href="https://vllm.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href=""><b>Blog</b></a> |
14
15
</p>
16
@@ -46,7 +46,7 @@ vLLM seamlessly supports many Huggingface models, including the following archit
46
- LLaMA (e.g., `lmsys/vicuna-13b-v1.3`, `young-geng/koala`, `openlm-research/open_llama_13b`, etc.)
47
- OPT (e.g., `facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.)
48
49
-Install vLLM with pip or [from source](https://llm-serving-cacheflow.readthedocs-hosted.com/en/latest/getting_started/installation.html#build-from-source):
49
+Install vLLM with pip or [from source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source):
50
51
```bash
52
pip install vllm
@@ -54,10 +54,10 @@ pip install vllm
54
55
## Getting Started
56
57
-Visit our [documentation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) to get started.
58
-- [Installation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/installation.html)
59
-- [Quickstart](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/quickstart.html)
60
-- [Supported Models](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/models/supported_models.html)
57
+Visit our [documentation](https://vllm.readthedocs.io/en/latest/) to get started.
58
+- [Installation](https://vllm.readthedocs.io/en/latest/getting_started/installation.html)
59
+- [Quickstart](https://vllm.readthedocs.io/en/latest/getting_started/quickstart.html)
60
+- [Supported Models](https://vllm.readthedocs.io/en/latest/models/supported_models.html)
61
62
## Performance
63
docs/source/conf.py
+1
-1
@@ -57,7 +57,7 @@ html_logo = 'assets/logos/vllm-logo-text-light.png'
57
html_theme_options = {
58
'logo_only': True,
59
'path_to_docs': 'docs/source',
60
- 'repository_url': 'https://github.com/WoosukKwon/vllm',
60
+ 'repository_url': 'https://github.com/vllm-project/vllm',
61
'use_repository_button': True,
62
}
63
docs/source/getting_started/installation.rst
+1
-1
@@ -52,6 +52,6 @@ You can also build and install vLLM from source:
52
53
.. code-block:: console
54
55
- $ git clone https://github.com/WoosukKwon/vllm.git
55
+ $ git clone https://github.com/vllm-project/vllm.git
56
$ cd vllm
57
$ pip install -e . # This may take 5-10 minutes.
docs/source/getting_started/quickstart.rst
+5
-5
@@ -22,7 +22,7 @@ Import ``LLM`` and ``SamplingParams`` from vLLM. The ``LLM`` class is the main c
22
23
from vllm import LLM, SamplingParams
24
25
-Define the list of input prompts and the sampling parameters for generation. The sampling temperature is set to 0.8 and the nucleus sampling probability is set to 0.95. For more information about the sampling parameters, refer to the `class definition <https://github.com/WoosukKwon/vllm/blob/main/vllm/sampling_params.py>`_.
25
+Define the list of input prompts and the sampling parameters for generation. The sampling temperature is set to 0.8 and the nucleus sampling probability is set to 0.95. For more information about the sampling parameters, refer to the `class definition <https://github.com/vllm-project/vllm/blob/main/vllm/sampling_params.py>`_.
26
27
.. code-block:: python
28
@@ -53,13 +53,13 @@ Call ``llm.generate`` to generate the outputs. It adds the input prompts to vLLM
53
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
54
55
56
-The code example can also be found in `examples/offline_inference.py <https://github.com/WoosukKwon/vllm/blob/main/examples/offline_inference.py>`_.
56
+The code example can also be found in `examples/offline_inference.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference.py>`_.
57
58
59
API Server
60
----------
61
62
-vLLM can be deployed as an LLM service. We provide an example `FastAPI <https://fastapi.tiangolo.com/>`_ server. Check `vllm/entrypoints/api_server.py <https://github.com/WoosukKwon/vllm/blob/main/vllm/entrypoints/api_server.py>`_ for the server implementation. The server uses ``AsyncLLMEngine`` class to support asynchronous processing of incoming requests.
62
+vLLM can be deployed as an LLM service. We provide an example `FastAPI <https://fastapi.tiangolo.com/>`_ server. Check `vllm/entrypoints/api_server.py <https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/api_server.py>`_ for the server implementation. The server uses ``AsyncLLMEngine`` class to support asynchronous processing of incoming requests.
63
64
Start the server:
65
@@ -81,7 +81,7 @@ Query the model in shell:
81
$ "temperature": 0
82
$ }'
83
84
-See `examples/api_client.py <https://github.com/WoosukKwon/vllm/blob/main/examples/api_client.py>`_ for a more detailed client example.
84
+See `examples/api_client.py <https://github.com/vllm-project/vllm/blob/main/examples/api_client.py>`_ for a more detailed client example.
85
86
OpenAI-Compatible Server
87
------------------------
@@ -128,4 +128,4 @@ Since this server is compatible with OpenAI API, you can use it as a drop-in rep
128
prompt="San Francisco is a")
129
print("Completion result:", completion)
130
131
-For a more detailed client example, refer to `examples/openai_client.py <https://github.com/WoosukKwon/vllm/blob/main/examples/openai_client.py>`_.
131
+For a more detailed client example, refer to `examples/openai_client.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_client.py>`_.
docs/source/index.rst
+3
-3
@@ -15,9 +15,9 @@ Welcome to vLLM!
15
</p>
16
17
<p style="text-align:center">
18
- <a class="github-button" href="https://github.com/WoosukKwon/vllm" data-show-count="true" data-size="large" aria-label="Star skypilot-org/skypilot on GitHub">Star</a>
19
- <a class="github-button" href="https://github.com/WoosukKwon/vllm/subscription" data-icon="octicon-eye" data-size="large" aria-label="Watch skypilot-org/skypilot on GitHub">Watch</a>
20
- <a class="github-button" href="https://github.com/WoosukKwon/vllm/fork" data-icon="octicon-repo-forked" data-size="large" aria-label="Fork skypilot-org/skypilot on GitHub">Fork</a>
18
+ <a class="github-button" href="https://github.com/vllm-project/vllm" data-show-count="true" data-size="large" aria-label="Star skypilot-org/skypilot on GitHub">Star</a>
19
+ <a class="github-button" href="https://github.com/vllm-project/vllm/subscription" data-icon="octicon-eye" data-size="large" aria-label="Watch skypilot-org/skypilot on GitHub">Watch</a>
20
+ <a class="github-button" href="https://github.com/vllm-project/vllm/fork" data-icon="octicon-repo-forked" data-size="large" aria-label="Fork skypilot-org/skypilot on GitHub">Fork</a>
21
</p>
22
23
docs/source/models/adding_model.rst
+5
-5
@@ -11,22 +11,22 @@ This document provides a high-level guide on integrating a `HuggingFace Transfor
11
However, for models that include new operators (e.g., a new attention mechanism), the process can be a bit more complex.
12
13
.. tip::
14
- If you are encountering issues while integrating your model into vLLM, feel free to open an issue on our `GitHub <https://github.com/WoosukKwon/vllm/issues>`_ repository.
14
+ If you are encountering issues while integrating your model into vLLM, feel free to open an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ repository.
15
We will be happy to help you out!
16
17
18
0. Fork the vLLM repository
19
--------------------------------
20
21
-Start by forking our `GitHub <https://github.com/WoosukKwon/vllm/issues>`_ repository and then :ref:`build it from source <build_from_source>`.
21
+Start by forking our `GitHub <https://github.com/vllm-project/vllm/issues>`_ repository and then :ref:`build it from source <build_from_source>`.
22
This gives you the ability to modify the codebase and test your model.
23
24
25
1. Bring your model code
26
------------------------
27
28
-Clone the PyTorch model code from the HuggingFace Transformers repository and put it into the `vllm/model_executor/models <https://github.com/WoosukKwon/vllm/tree/main/vllm/model_executor/models>`_ directory.
29
-For instance, vLLM's `OPT model <https://github.com/WoosukKwon/vllm/blob/main/vllm/model_executor/models/opt.py>`_ was adpated from the HuggingFace's `modeling_opt.py <https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py>`_ file.
28
+Clone the PyTorch model code from the HuggingFace Transformers repository and put it into the `vllm/model_executor/models <https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models>`_ directory.
29
+For instance, vLLM's `OPT model <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/opt.py>`_ was adpated from the HuggingFace's `modeling_opt.py <https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py>`_ file.
30
31
.. warning::
32
When copying the model code, make sure to review and adhere to the code's copyright and licensing terms.
@@ -91,4 +91,4 @@ While the process is straightforward for most layers, the tensor-parallel layers
91
5. Register your model
92
----------------------
93
94
-Finally, include your :code:`*ForCausalLM` class in `vllm/model_executor/models/__init__.py <https://github.com/WoosukKwon/vllm/blob/main/vllm/model_executor/models/__init__.py>`_ and register it to the :code:`_MODEL_REGISTRY` in `vllm/model_executor/model_loader.py <https://github.com/WoosukKwon/vllm/blob/main/vllm/model_executor/model_loader.py>`_.
94
+Finally, include your :code:`*ForCausalLM` class in `vllm/model_executor/models/__init__.py <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/__init__.py>`_ and register it to the :code:`_MODEL_REGISTRY` in `vllm/model_executor/model_loader.py <https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/model_loader.py>`_.
docs/source/models/supported_models.rst
+1
-1
@@ -29,7 +29,7 @@ Alongside each architecture, we include some popular models that use it.
29
30
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
31
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.
32
-Alternatively, you can raise an issue on our `GitHub <https://github.com/WoosukKwon/vllm/issues>`_ project.
32
+Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
33
34
.. tip::
35
The easiest way to check if your model is supported is to run the program below: