A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 04:45 UTC
main
[TPU] Fix tpu model runner test (#19995)
Chenyaaang committed over 1 year ago
[Bugfix][Benchmark] Fix Marlin benchmark (#19929)
22quinn committed over 1 year ago
Feat Dynamic Quantization for MoE Layers in GPTQ Marlin Backend (#19395)
Jun-Howie committed over 1 year ago
[CI/Build] Push latest tag for cpu and neuron docker image (#19897)
22quinn committed over 1 year ago
[Feature] Support sequence parallelism for static fp8 quantization (#19181)
cascade committed over 1 year ago
[Misc] Add type alias `ReqId` and `EngineId` for better readability (#19880)
lkchen committed over 1 year ago
[Bugfix][v1] Fix step pooler implementation and step pooling usage in v1 (#19956)
Isotr0py committed over 1 year ago
[EP+DP] Optimize the little operations in the DeepGEMM + DeepEP low latency case (#19885)
Tyler Michael Smith committed over 1 year ago
[Docs] Fix syntax highlighting of shell commands (#19870)
Lukas Geiger committed over 1 year ago
[doc] improve readability for long commands (#19920)
Reid committed over 1 year ago
[Bugfix] Fix CI bitsandbytes failure (#19969)
Jee Jee Li committed over 1 year ago
[doc] use MkDocs collapsible blocks - supplement (#19973)
Reid committed over 1 year ago
[Doc] Update V1 status for decoder-only embedding models (#19952)
Isotr0py committed over 1 year ago
lkchen committed over 1 year ago
[P/D][NixlConnector] Support `tp_size > num_kv_heads` deployments (#19691)
Nicolò Lucchesi committed over 1 year ago
[doc] Fold long code blocks to improve readability (#19926)
Reid committed over 1 year ago
Fix(models/siglip): Add compatibility for Gemma models quantized by llm-compressor (#19643)
Vensen committed over 1 year ago
[Misc] Configurable timeout for execute_model RPC calls via env var (#19544)
jinqinn committed over 1 year ago
[Core] feat: Implement Priority Scheduling in V1 Engine (#19057)
amit committed over 1 year ago
[Perf][CLI] Improve overall startup time (#19941)
Aaron Pham committed over 1 year ago