A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 11:00 UTC
main
Cheng Kuan Yong Jason committed over 1 year ago
[misc] add instructions on how to install nvshmem/pplx/deepep (#17964)
youkaichao committed over 1 year ago
[Model] Broadcast Ovis2 implementation to fit Ovis1.6 (#17861)
Isotr0py committed over 1 year ago
[BUG] [ROCm] [MLA] Fix variable name bug due to change in variable name in PR #17483 (#17961)
TJian committed over 1 year ago
[BugFix] [ROCm]: Bugfix and handle addition case of input for `rocm_aiter_rms_norm` (#17857)
TJian committed over 1 year ago
[Misc] not show --model in vllm serve --help (#16691)
Reid committed over 1 year ago
[New Model]: nomic-embed-text-v2-moe (#17785)
wang.yuqi committed over 1 year ago
[FP8][ROCm][Attention] Enable FP8 KV cache on ROCm for V1 (#17870)
Gregory Shtrasberg committed over 1 year ago
[Misc] Add compressed-tensors NVFP4A16 emulation support (#17914)
Dipika Sikka committed over 1 year ago
[Frontend] Add /classify endpoint (#17032)
Frieda Huang committed over 1 year ago
[doc] update lora doc (#17936)
Reid committed over 1 year ago
[Bugfix]: v1 engine - consider lora adapters in allowed_token_ids (#17855)
Ben Browning committed over 1 year ago
fix amd triton mla path (#17871)
Shiyan Deng committed over 1 year ago
[Perf] Use small max_num_batched_tokens for A100 (#17885)
Kuntai Du committed over 1 year ago
[Bugfix] Add revision to `transformers.Auto*.from_pretrained` processors (#17948)
xinli-centml committed over 1 year ago
[Kernel] fp4 marlin kernel (#17687)
Jinzhen Lin committed over 1 year ago
[v1] Rename specialized_manager.py to single_type_kv_cache_manager.py (#17946)
Chen Zhang committed over 1 year ago
[v1] Pass BlockTable and KVCacheSpec to AttentionMetadataBuilders (#17483)
Chen Zhang committed over 1 year ago
[Misc] remove --model from vllm serve usage (#17944)
Reid committed over 1 year ago
Don't default construct `ModelConfig` when default constructing `VllmConfig` (#17943)
Harry Mellor committed over 1 year ago