@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 110 lines 5.57 KB
Rendered Raw
1 <!-- markdownlint-disable MD001 MD041 -->
2 <p align="center">
3 <picture>
4 <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/assets/logos/vllm-logo-text-dark.png">
5 <img alt="vLLM" src="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/assets/logos/vllm-logo-text-light.png" width=55%>
6 </picture>
7 </p>
8
9 <h3 align="center">
10 Easy, fast, and cheap LLM serving for everyone
11 </h3>
12
13 <p align="center">
14 | <a href="https://docs.vllm.ai"><b>Documentation</b></a> | <a href="https://blog.vllm.ai/"><b>Blog</b></a> | <a href="https://arxiv.org/abs/2309.06180"><b>Paper</b></a> | <a href="https://x.com/vllm_project"><b>Twitter/X</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> |
15 </p>
16
17 🔥 We have built a vLLM website to help you get started with vLLM. Please visit [vllm.ai](https://vllm.ai) to learn more.
18 For events, please visit [vllm.ai/events](https://vllm.ai/events) to join us.
19
20 ---
21
22 ## About
23
24 vLLM is a fast and easy-to-use library for LLM inference and serving.
25
26 Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.
27
28 vLLM is fast with:
29
30 - State-of-the-art serving throughput
31 - Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
32 - Continuous batching of incoming requests, chunked prefill, prefix caching
33 - Fast and flexible model execution with piecewise and full CUDA/HIP graphs
34 - Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and [more](https://docs.vllm.ai/en/latest/features/quantization/index.html)
35 - Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton
36 - Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL
37 - Speculative decoding including n-gram, suffix, EAGLE, DFlash
38 - Automatic kernel generation and graph-level transformations using torch.compile
39 - Disaggregated prefill, decode, and encode
40
41 vLLM is flexible and easy to use with:
42
43 - Seamless integration with popular Hugging Face models
44 - High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
45 - Tensor, pipeline, data, expert, and context parallelism for distributed inference
46 - Streaming outputs
47 - Generation of structured outputs using xgrammar or guidance
48 - Tool calling and reasoning parsers
49 - OpenAI-compatible API server, plus Anthropic Messages API and gRPC support
50 - Efficient multi-LoRA support for dense and MoE layers
51 - Support for NVIDIA GPUs, AMD GPUs, Intel GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more.
52
53 vLLM seamlessly supports 200+ model architectures on Hugging Face, including:
54
55 - Decoder-only LLMs (e.g., Llama, Qwen, Gemma)
56 - Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)
57 - Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)
58 - Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral)
59 - Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT)
60 - Reward and classification models (e.g., Qwen-Math)
61
62 Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
63
64 ## Getting Started
65
66 Install vLLM with [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`:
67
68 ```bash
69 uv pip install vllm
70 ```
71
72 Or [build from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for development.
73
74 Visit our [documentation](https://docs.vllm.ai/en/latest/) to learn more.
75
76 - [Installation](https://docs.vllm.ai/en/latest/getting_started/installation.html)
77 - [Quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart.html)
78 - [List of Supported Models](https://docs.vllm.ai/en/latest/models/supported_models.html)
79
80 ## Contributing
81
82 We welcome and value any contributions and collaborations.
83 Please check out [Contributing to vLLM](https://docs.vllm.ai/en/latest/contributing/index.html) for how to get involved.
84
85 ## Citation
86
87 If you use vLLM for your research, please cite our [paper](https://arxiv.org/abs/2309.06180):
88
89 ```bibtex
90 @inproceedings{kwon2023efficient,
91 title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
92 author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
93 booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
94 year={2023}
95 }
96 ```
97
98 ## Contact Us
99
100 <!-- --8<-- [start:contact-us] -->
101 - For technical questions and feature requests, please use GitHub [Issues](https://github.com/vllm-project/vllm/issues)
102 - For discussing with fellow users, please use the [vLLM Forum](https://discuss.vllm.ai)
103 - For coordinating contributions and development, please use [Slack](https://slack.vllm.ai)
104 - For security disclosures, please use GitHub's [Security Advisories](https://github.com/vllm-project/vllm/security/advisories) feature
105 - For collaborations and partnerships, please contact us at [collaboration@vllm.ai](mailto:collaboration@vllm.ai)
106 <!-- --8<-- [end:contact-us] -->
107
108 ## Media Kit
109
110 - If you wish to use vLLM's logo, please refer to [our media kit repo](https://github.com/vllm-project/media-kit)