A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 19:30 UTC
[Docs] Update readme (#7316)
Simon Mo committed
Aug 11, 2024 at 17:13 UTC
f020a6297e7539fdc5688fad366309fed5c0456a
2 files changed
+18
-14
README.md
+11
-8
@@ -10,7 +10,7 @@ Easy, fast, and cheap LLM serving for everyone
10
</h3>
11
12
<p align="center">
13
-| <a href="https://docs.vllm.ai"><b>Documentation</b></a> | <a href="https://vllm.ai"><b>Blog</b></a> | <a href="https://arxiv.org/abs/2309.06180"><b>Paper</b></a> | <a href="https://discord.gg/jz7wjKhh6g"><b>Discord</b></a> |
13
+| <a href="https://docs.vllm.ai"><b>Documentation</b></a> | <a href="https://vllm.ai"><b>Blog</b></a> | <a href="https://arxiv.org/abs/2309.06180"><b>Paper</b></a> | <a href="https://discord.gg/jz7wjKhh6g"><b>Discord</b></a> | <a href="https://x.com/vllm_project"><b>Twitter/X</b></a> |
14
15
</p>
16
@@ -36,10 +36,12 @@ vLLM is fast with:
36
- Efficient management of attention key and value memory with **PagedAttention**
37
- Continuous batching of incoming requests
38
- Fast model execution with CUDA/HIP graph
39
-- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629), FP8 KV Cache
40
-- Optimized CUDA kernels
39
+- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), INT4, INT8, and FP8.
40
+- Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
41
+- Speculative decoding
42
+- Chunked prefill
43
42
-**Performance benchmark**: We include a [performance benchmark](https://buildkite.com/vllm/performance-benchmark/builds/4068) that compares the performance of vllm against other LLM serving engines ([TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [text-generation-inference](https://github.com/huggingface/text-generation-inference) and [lmdeploy](https://github.com/InternLM/lmdeploy)).
44
+**Performance benchmark**: We include a [performance benchmark](https://buildkite.com/vllm/performance-benchmark/builds/4068) that compares the performance of vLLM against other LLM serving engines ([TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [text-generation-inference](https://github.com/huggingface/text-generation-inference) and [lmdeploy](https://github.com/InternLM/lmdeploy)).
45
46
vLLM is flexible and easy to use with:
47
@@ -48,20 +50,21 @@ vLLM is flexible and easy to use with:
50
- Tensor parallelism and pipeline parallelism support for distributed inference
51
- Streaming outputs
52
- OpenAI-compatible API server
51
-- Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs
52
-- (Experimental) Prefix caching support
53
-- (Experimental) Multi-lora support
53
+- Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
54
+- Prefix caching support
55
+- Multi-lora support
56
57
vLLM seamlessly supports most popular open-source models on HuggingFace, including:
58
- Transformer-like LLMs (e.g., Llama)
59
- Mixture-of-Expert LLMs (e.g., Mixtral)
60
+- Embedding Models (e.g. E5-Mistral)
61
- Multi-modal LLMs (e.g., LLaVA)
62
63
Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
64
65
## Getting Started
66
64
-Install vLLM with pip or [from source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source):
67
+Install vLLM with `pip` or [from source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source):
68
69
```bash
70
pip install vllm
docs/source/index.rst
+7
-6
@@ -31,8 +31,10 @@ vLLM is fast with:
31
* Efficient management of attention key and value memory with **PagedAttention**
32
* Continuous batching of incoming requests
33
* Fast model execution with CUDA/HIP graph
34
-* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_, FP8 KV Cache
35
-* Optimized CUDA kernels
34
+* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, INT4, INT8, and FP8
35
+* Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
36
+* Speculative decoding
37
+* Chunked prefill
38
39
vLLM is flexible and easy to use with:
40
@@ -41,9 +43,9 @@ vLLM is flexible and easy to use with:
43
* Tensor parallelism and pipeline parallelism support for distributed inference
44
* Streaming outputs
45
* OpenAI-compatible API server
44
-* Support NVIDIA GPUs and AMD GPUs
45
-* (Experimental) Prefix caching support
46
-* (Experimental) Multi-lora support
46
+* Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
47
+* Prefix caching support
48
+* Multi-lora support
49
50
For more information, check out the following:
51
@@ -53,7 +55,6 @@ For more information, check out the following:
55
* :ref:`vLLM Meetups <meetups>`.
56
57
56
-
58
Documentation
59
-------------
60