@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / b81a6a6bb3
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 05:30 UTC

[Docs] Add supported quantization methods to docs (#2135)

Woosuk Kwon committed Dec 15, 2023 at 13:29 UTC b81a6a6bb342d7b9166a1c7a6b69507fb53ff33e
2 files changed +4 -2
README.md
+2 -1
@@ -35,6 +35,7 @@ vLLM is fast with:
35 - State-of-the-art serving throughput
36 - Efficient management of attention key and value memory with **PagedAttention**
37 - Continuous batching of incoming requests
38 +- Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629)
39 - Optimized CUDA kernels
40
41 vLLM is flexible and easy to use with:
@@ -44,7 +45,7 @@ vLLM is flexible and easy to use with:
45 - Tensor parallelism support for distributed inference
46 - Streaming outputs
47 - OpenAI-compatible API server
47 -- Support NVIDIA CUDA and AMD ROCm.
48 +- Support NVIDIA GPUs and AMD GPUs.
49
50 vLLM seamlessly supports many Hugging Face models, including the following architectures:
51
docs/source/index.rst
+2 -1
@@ -30,6 +30,7 @@ vLLM is fast with:
30 * State-of-the-art serving throughput
31 * Efficient management of attention key and value memory with **PagedAttention**
32 * Continuous batching of incoming requests
33 +* Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_
34 * Optimized CUDA kernels
35
36 vLLM is flexible and easy to use with:
@@ -39,7 +40,7 @@ vLLM is flexible and easy to use with:
40 * Tensor parallelism support for distributed inference
41 * Streaming outputs
42 * OpenAI-compatible API server
42 -* Support NVIDIA CUDA and AMD ROCm.
43 +* Support NVIDIA GPUs and AMD GPUs.
44
45 For more information, check out the following:
46