A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 17:00 UTC
main
[Kernel] changing fused moe kernel chunk size default to 32k (#7995)
Avshalom Manevich committed about 2 years ago
[TPU] Async output processing for TPU (#8011)
Woosuk Kwon committed about 2 years ago
[Core] Logprobs support in Multi-step (#7652)
afeldman-nm committed about 2 years ago
[VLM] Disallow overflowing `max_model_len` for multimodal models (#7998)
Cyrus Leung committed about 2 years ago
Add more percentiles and latencies (#7759)
Wei-Sheng Chin committed about 2 years ago
support bitsandbytes 8-bit and FP4 quantized models (#7445)
chenqianfzh committed about 2 years ago
[Neuron] Adding support for context-lenght, token-gen buckets. (#7885)
Harsha vardhan manoj Bikki committed about 2 years ago
[misc] update tpu int8 to use new vLLM Parameters (#7973)
Dipika Sikka committed about 2 years ago
[Bugfix] Fix incorrect vocal embedding shards for GGUF model in tensor parallelism (#7954)
Isotr0py committed about 2 years ago
extend cuda graph size for H200 (#7894)
kushanam committed about 2 years ago
Pavani Majety committed about 2 years ago
[Core] Combine async postprocessor and multi-step (#7921)
Alexander Matveev committed about 2 years ago
[Bugfix] Unify rank computation across regular decoding and speculative decoding (#7899)
Jonas M. Kübler committed about 2 years ago
Revert "[Core][Kernels] Use FlashInfer backend for FP8 KV Cache when available." (#7982)
youkaichao committed about 2 years ago
[VLM][Core] Fix exceptions on ragged NestedTensors (#7974)
Peter Salas committed about 2 years ago
[torch.compile] remove reset (#7975)
youkaichao committed about 2 years ago
[Frontend] Minor optimizations to zmq decoupled front-end (#7957)
Nick Hill committed about 2 years ago
Remove faulty Meta-Llama-3-8B-Instruct-FP8.yaml lm-eval test (#7961)
Michael Goin committed about 2 years ago
[torch.compile] avoid Dynamo guard evaluation overhead (#7898)
youkaichao committed about 2 years ago
[Bugfix] Make torch registration of punica ops optional (#7970)
bnellnm committed about 2 years ago