A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 5, 11:00 UTC
[Docs] Update README (#39251)
Signed-off-by: mgoin <mgoin64@gmail.com>
Michael Goin committed
Apr 8, 2026 at 05:34 UTC
2488d1dca2df05059fcfbad0a1612ef2a5202b47
1 file changed
+26
-19
README.md
+26
-19
@@ -23,47 +23,54 @@ For events, please visit [vllm.ai/events](https://vllm.ai/events) to join us.
23
24
vLLM is a fast and easy-to-use library for LLM inference and serving.
25
26
-Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.
26
+Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.
27
28
vLLM is fast with:
29
30
- State-of-the-art serving throughput
31
- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
32
-- Continuous batching of incoming requests
33
-- Fast model execution with CUDA/HIP graph
34
-- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [AutoRound](https://arxiv.org/abs/2309.05516), INT4, INT8, and FP8
35
-- Optimized CUDA kernels, including integration with FlashAttention and FlashInfer
36
-- Speculative decoding
37
-- Chunked prefill
32
+- Continuous batching of incoming requests, chunked prefill, prefix caching
33
+- Fast and flexible model execution with piecewise and full CUDA/HIP graphs
34
+- Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and [more](https://docs.vllm.ai/en/latest/features/quantization/index.html)
35
+- Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton
36
+- Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL
37
+- Speculative decoding including n-gram, suffix, EAGLE, DFlash
38
+- Automatic kernel generation and graph-level transformations using torch.compile
39
+- Disaggregated prefill, decode, and encode
40
41
vLLM is flexible and easy to use with:
42
43
- Seamless integration with popular Hugging Face models
44
- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
43
-- Tensor, pipeline, data and expert parallelism support for distributed inference
45
+- Tensor, pipeline, data, expert, and context parallelism for distributed inference
46
- Streaming outputs
45
-- OpenAI-compatible API server
46
-- Support for NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, Arm CPUs, and TPU. Additionally, support for diverse hardware plugins such as Intel Gaudi, IBM Spyre and Huawei Ascend.
47
-- Prefix caching support
48
-- Multi-LoRA support
47
+- Generation of structured outputs using xgrammar or guidance
48
+- Tool calling and reasoning parsers
49
+- OpenAI-compatible API server, plus Anthropic Messages API and gRPC support
50
+- Efficient multi-LoRA support for dense and MoE layers
51
+- Support for NVIDIA GPUs, AMD GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more.
52
50
-vLLM seamlessly supports most popular open-source models on HuggingFace, including:
53
+vLLM seamlessly supports 200+ model architectures on HuggingFace, including:
54
52
-- Transformer-like LLMs (e.g., Llama)
53
-- Mixture-of-Expert LLMs (e.g., Mixtral, Deepseek-V2 and V3)
54
-- Embedding Models (e.g., E5-Mistral)
55
-- Multi-modal LLMs (e.g., LLaVA)
55
+- Decoder-only LLMs (e.g., Llama, Qwen, Gemma)
56
+- Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)
57
+- Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)
58
+- Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral)
59
+- Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT)
60
+- Reward and classification models (e.g., Qwen-Math)
61
62
Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
63
64
## Getting Started
65
61
-Install vLLM with `pip` or [from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source):
66
+Install vLLM with [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`:
67
68
```bash
64
-pip install vllm
69
+uv pip install vllm
70
```
71
72
+Or [build from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for development.
73
+
74
Visit our [documentation](https://docs.vllm.ai/en/latest/) to learn more.
75
76
- [Installation](https://docs.vllm.ai/en/latest/getting_started/installation.html)