@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 26c52a5ea6
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 05:30 UTC

[Docs] Add CUDA graph support to docs (#2148)

Woosuk Kwon committed Dec 17, 2023 at 01:49 UTC 26c52a5ea6e9523b1c8bb8365b214ca29056c266
2 files changed +4 -2
README.md
+2 -1
@@ -35,6 +35,7 @@ vLLM is fast with:
35 - State-of-the-art serving throughput
36 - Efficient management of attention key and value memory with **PagedAttention**
37 - Continuous batching of incoming requests
38 +- Fast model execution with CUDA/HIP graph
39 - Quantization: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [SqueezeLLM](https://arxiv.org/abs/2306.07629)
40 - Optimized CUDA kernels
41
@@ -45,7 +46,7 @@ vLLM is flexible and easy to use with:
46 - Tensor parallelism support for distributed inference
47 - Streaming outputs
48 - OpenAI-compatible API server
48 -- Support NVIDIA GPUs and AMD GPUs.
49 +- Support NVIDIA GPUs and AMD GPUs
50
51 vLLM seamlessly supports many Hugging Face models, including the following architectures:
52
docs/source/index.rst
+2 -1
@@ -30,6 +30,7 @@ vLLM is fast with:
30 * State-of-the-art serving throughput
31 * Efficient management of attention key and value memory with **PagedAttention**
32 * Continuous batching of incoming requests
33 +* Fast model execution with CUDA/HIP graph
34 * Quantization: `GPTQ <https://arxiv.org/abs/2210.17323>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, `SqueezeLLM <https://arxiv.org/abs/2306.07629>`_
35 * Optimized CUDA kernels
36
@@ -40,7 +41,7 @@ vLLM is flexible and easy to use with:
41 * Tensor parallelism support for distributed inference
42 * Streaming outputs
43 * OpenAI-compatible API server
43 -* Support NVIDIA GPUs and AMD GPUs.
44 +* Support NVIDIA GPUs and AMD GPUs
45
46 For more information, check out the following:
47