A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
[Docs] Add CUDA graph support to docs (#2148)
Woosuk Kwon committed
Dec 17, 2023 at 01:49 UTC
26c52a5ea6e9523b1c8bb8365b214ca29056c266
2 files changed
+4
-2
README.md
+2
-1
@@ -35,6 +35,7 @@ vLLM is fast with:
35
- State-of-the-art serving throughput
36
- Efficient management of attention key and value memory with **PagedAttention**
37
- Continuous batching of incoming requests
38
+- Fast model execution with CUDA/HIP graph
39
- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629)
40
- Optimized CUDA kernels
41
@@ -45,7 +46,7 @@ vLLM is flexible and easy to use with:
46
- Tensor parallelism support for distributed inference
47
- Streaming outputs
48
- OpenAI-compatible API server
48
-- Support NVIDIA GPUs and AMD GPUs.
49
+- Support NVIDIA GPUs and AMD GPUs
50
51
vLLM seamlessly supports many Hugging Face models, including the following architectures:
52
docs/source/index.rst
+2
-1
@@ -30,6 +30,7 @@ vLLM is fast with:
30
* State-of-the-art serving throughput
31
* Efficient management of attention key and value memory with **PagedAttention**
32
* Continuous batching of incoming requests
33
+* Fast model execution with CUDA/HIP graph
34
* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_
35
* Optimized CUDA kernels
36
@@ -40,7 +41,7 @@ vLLM is flexible and easy to use with:
41
* Tensor parallelism support for distributed inference
42
* Streaming outputs
43
* OpenAI-compatible API server
43
-* Support NVIDIA GPUs and AMD GPUs.
44
+* Support NVIDIA GPUs and AMD GPUs
45
46
For more information, check out the following:
47