@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / f020a6297e
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 19:30 UTC

[Docs] Update readme (#7316)

Simon Mo committed Aug 11, 2024 at 17:13 UTC f020a6297e7539fdc5688fad366309fed5c0456a
2 files changed +18 -14
README.md
+11 -8
@@ -10,7 +10,7 @@ Easy, fast, and cheap LLM serving for everyone
10 </h3>
11
12 <p align="center">
13 -| <a href="https://docs.vllm.ai"><b>Documentation</b></a> | <a href="https://vllm.ai"><b>Blog</b></a> | <a href="https://arxiv.org/abs/2309.06180"><b>Paper</b></a> | <a href="https://discord.gg/jz7wjKhh6g"><b>Discord</b></a> |
13 +| <a href="https://docs.vllm.ai"><b>Documentation</b></a> | <a href="https://vllm.ai"><b>Blog</b></a> | <a href="https://arxiv.org/abs/2309.06180"><b>Paper</b></a> | <a href="https://discord.gg/jz7wjKhh6g"><b>Discord</b></a> | <a href="https://x.com/vllm_project"><b>Twitter/X</b></a> |
14
15 </p>
16
@@ -36,10 +36,12 @@ vLLM is fast with:
36 - Efficient management of attention key and value memory with **PagedAttention**
37 - Continuous batching of incoming requests
38 - Fast model execution with CUDA/HIP graph
39 -- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629), FP8 KV Cache
40 -- Optimized CUDA kernels
39 +- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), INT4, INT8, and FP8.
40 +- Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
41 +- Speculative decoding
42 +- Chunked prefill
43
42 -**Performance benchmark**: We include a [performance benchmark](https://buildkite.com/vllm/performance-benchmark/builds/4068) that compares the performance of vllm against other LLM serving engines ([TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [text-generation-inference](https://github.com/huggingface/text-generation-inference) and [lmdeploy](https://github.com/InternLM/lmdeploy)).
44 +**Performance benchmark**: We include a [performance benchmark](https://buildkite.com/vllm/performance-benchmark/builds/4068) that compares the performance of vLLM against other LLM serving engines ([TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [text-generation-inference](https://github.com/huggingface/text-generation-inference) and [lmdeploy](https://github.com/InternLM/lmdeploy)).
45
46 vLLM is flexible and easy to use with:
47
@@ -48,20 +50,21 @@ vLLM is flexible and easy to use with:
50 - Tensor parallelism and pipeline parallelism support for distributed inference
51 - Streaming outputs
52 - OpenAI-compatible API server
51 -- Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs
52 -- (Experimental) Prefix caching support
53 -- (Experimental) Multi-lora support
53 +- Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
54 +- Prefix caching support
55 +- Multi-lora support
56
57 vLLM seamlessly supports most popular open-source models on HuggingFace, including:
58 - Transformer-like LLMs (e.g., Llama)
59 - Mixture-of-Expert LLMs (e.g., Mixtral)
60 +- Embedding Models (e.g. E5-Mistral)
61 - Multi-modal LLMs (e.g., LLaVA)
62
63 Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
64
65 ## Getting Started
66
64 -Install vLLM with pip or [from source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source):
67 +Install vLLM with `pip` or [from source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source):
68
69 ```bash
70 pip install vllm
docs/source/index.rst
+7 -6
@@ -31,8 +31,10 @@ vLLM is fast with:
31 * Efficient management of attention key and value memory with **PagedAttention**
32 * Continuous batching of incoming requests
33 * Fast model execution with CUDA/HIP graph
34 -* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_, FP8 KV Cache
35 -* Optimized CUDA kernels
34 +* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, INT4, INT8, and FP8
35 +* Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
36 +* Speculative decoding
37 +* Chunked prefill
38
39 vLLM is flexible and easy to use with:
40
@@ -41,9 +43,9 @@ vLLM is flexible and easy to use with:
43 * Tensor parallelism and pipeline parallelism support for distributed inference
44 * Streaming outputs
45 * OpenAI-compatible API server
44 -* Support NVIDIA GPUs and AMD GPUs
45 -* (Experimental) Prefix caching support
46 -* (Experimental) Multi-lora support
46 +* Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
47 +* Prefix caching support
48 +* Multi-lora support
49
50 For more information, check out the following:
51
@@ -53,7 +55,6 @@ For more information, check out the following:
55 * :ref:`vLLM Meetups <meetups>`.
56
57
56 -
58 Documentation
59 -------------
60