A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 07:15 UTC
[Doc] Readme standardization (#18695)
Co-authored-by: Soren Dreano <soren@numind.ai>
SorenDreano committed
Jun 3, 2025 at 20:50 UTC
19bdaf32b139656627c8b311361a0fa38ae98f4b
1 file changed
+5
-5
README.md
+5
-5
@@ -58,8 +58,8 @@ vLLM is fast with:
58
- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
59
- Continuous batching of incoming requests
60
- Fast model execution with CUDA/HIP graph
61
-- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [AutoRound](https://arxiv.org/abs/2309.05516),INT4, INT8, and FP8.
62
-- Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
61
+- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [AutoRound](https://arxiv.org/abs/2309.05516), INT4, INT8, and FP8
62
+- Optimized CUDA kernels, including integration with FlashAttention and FlashInfer
63
- Speculative decoding
64
- Chunked prefill
65
@@ -72,14 +72,14 @@ vLLM is flexible and easy to use with:
72
- Tensor parallelism and pipeline parallelism support for distributed inference
73
- Streaming outputs
74
- OpenAI-compatible API server
75
-- Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
75
+- Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron
76
- Prefix caching support
77
- Multi-LoRA support
78
79
vLLM seamlessly supports most popular open-source models on HuggingFace, including:
80
- Transformer-like LLMs (e.g., Llama)
81
- Mixture-of-Expert LLMs (e.g., Mixtral, Deepseek-V2 and V3)
82
-- Embedding Models (e.g. E5-Mistral)
82
+- Embedding Models (e.g., E5-Mistral)
83
- Multi-modal LLMs (e.g., LLaVA)
84
85
Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
@@ -162,4 +162,4 @@ If you use vLLM for your research, please cite our [paper](https://arxiv.org/abs
162
163
## Media Kit
164
165
-- If you wish to use vLLM's logo, please refer to [our media kit repo](https://github.com/vllm-project/media-kit).
165
+- If you wish to use vLLM's logo, please refer to [our media kit repo](https://github.com/vllm-project/media-kit)