A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 20:45 UTC
main
[Bugfix] Miscalculated latency lead to time_to_first_token_seconds inaccurate. (#6686)
Allen.Dou committed over 2 years ago
[Bugfix] Fix speculative decode seeded test (#6743)
Nick Hill committed over 2 years ago
Adding f-string to validation error which is missing (#6748)
LF Marques committed over 2 years ago
[Bugfix]fix modelscope compatible issue (#6730)
liuyhwangyh committed over 2 years ago
[Build/CI] Update run-amd-test.sh. Enable Docker Hub login. (#6711)
Alexei-V-Ivanov-AMD committed over 2 years ago
[Docs][ROCm] Detailed instructions to build from source (#6680)
Woosuk Kwon committed over 2 years ago
[Bugfix] Fix token padding for chameleon (#6724)
Roger Wang committed over 2 years ago
[SpecDecoding] Update MLPSpeculator CI tests to use smaller model (#6714)
Nick Hill committed over 2 years ago
[Bugfix] fix flashinfer cudagraph capture for PP (#6708)
William Lin committed over 2 years ago
[bitsandbytes]: support read bnb pre-quantized model (#5753)
dongmao zhang committed over 2 years ago
[Bugfix] StatLoggers: cache spec decode metrics when they get collected. (#6645)
Thomas Parnell committed over 2 years ago
[CI] Add smoke test for non-uniform AutoFP8 quantization (#6702)
Michael Goin committed over 2 years ago
[build] relax wheel size limit (#6704)
youkaichao committed over 2 years ago
Bump `transformers` version for Llama 3.1 hotfix and patch Chameleon (#6690)
Roger Wang committed over 2 years ago
[Model] Pipeline Parallel Support for DeepSeek v2 (#6519)
Travis Johnson committed over 2 years ago
[Frontend] Add Usage data in each chunk for chat_serving. #6540 (#6652)
Yehoshua Cohen committed over 2 years ago
[Misc] Add ignored layers for `fp8` quantization (#6657)
Michael Goin committed over 2 years ago
Bump version to 0.5.3.post1 (#6696)
Simon Mo committed over 2 years ago
[BugFix] Fix RoPE error in Llama 3.1 (#6693)
Woosuk Kwon committed over 2 years ago
[Bugfix] Fix a log error in chunked prefill (#6694)
Woosuk Kwon committed over 2 years ago