A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
[Docs] Add supported quantization methods to docs (#2135)
Woosuk Kwon committed
Dec 15, 2023 at 13:29 UTC
b81a6a6bb342d7b9166a1c7a6b69507fb53ff33e
2 files changed
+4
-2
README.md
+2
-1
@@ -35,6 +35,7 @@ vLLM is fast with:
35
- State-of-the-art serving throughput
36
- Efficient management of attention key and value memory with **PagedAttention**
37
- Continuous batching of incoming requests
38
+- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629)
39
- Optimized CUDA kernels
40
41
vLLM is flexible and easy to use with:
@@ -44,7 +45,7 @@ vLLM is flexible and easy to use with:
45
- Tensor parallelism support for distributed inference
46
- Streaming outputs
47
- OpenAI-compatible API server
47
-- Support NVIDIA CUDA and AMD ROCm.
48
+- Support NVIDIA GPUs and AMD GPUs.
49
50
vLLM seamlessly supports many Hugging Face models, including the following architectures:
51
docs/source/index.rst
+2
-1
@@ -30,6 +30,7 @@ vLLM is fast with:
30
* State-of-the-art serving throughput
31
* Efficient management of attention key and value memory with **PagedAttention**
32
* Continuous batching of incoming requests
33
+* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_
34
* Optimized CUDA kernels
35
36
vLLM is flexible and easy to use with:
@@ -39,7 +40,7 @@ vLLM is flexible and easy to use with:
40
* Tensor parallelism support for distributed inference
41
* Streaming outputs
42
* OpenAI-compatible API server
42
-* Support NVIDIA CUDA and AMD ROCm.
43
+* Support NVIDIA GPUs and AMD GPUs.
44
45
For more information, check out the following:
46