A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 05:30 UTC
main
[Speculative decoding 1/9] Optimized rejection sampler (#2336)
Cade Daniel committed almost 3 years ago
Add baichuan chat template jinjia file (#2390)
KKY committed almost 3 years ago
[Minor] Remove unused code in attention (#2384)
Woosuk Kwon committed almost 3 years ago
Fix eager mode performance (#2377)
Woosuk Kwon committed almost 3 years ago
Changed scheduler to use deques instead of lists (#2290)
Nadav Shmayovits committed almost 3 years ago
Ensure metrics are logged regardless of requests (#2347)
Iskren Ivov Chernev committed almost 3 years ago
Build docker image with shared objects from "build" step (#2237)
Alexandre Payot committed almost 3 years ago
Miner fix of type hint (#2340)
ljss committed almost 3 years ago
Woosuk Kwon committed almost 3 years ago
Revert the changes in test_cache (#2335)
Woosuk Kwon committed almost 3 years ago
Remove unused const TIMEOUT_TO_PREVENT_DEADLOCK (#2321)
Ronen Schaffer committed almost 3 years ago
Use NCCL instead of ray for control-plane communication to remove serialization overhead (#2221)
Zhuohan Li committed almost 3 years ago
Remove deprecated parameter: concurrency_count (#2315)
Ronen Schaffer committed almost 3 years ago
Enable CUDA graph for GPTQ & SqueezeLLM (#2318)
Woosuk Kwon committed almost 3 years ago
[Minor] Fix typo and remove unused code (#2305)
Roy committed almost 3 years ago
[FIX] Support non-zero CUDA devices in custom kernels (#1959)
Jee Li committed almost 3 years ago
Support GPT-NeoX Models without attention biases (#2301)
Jong-hun Shin committed almost 3 years ago
[BUGFIX] Fix communication test (#2285)
Zhuohan Li committed almost 3 years ago
[BUGFIX] Fix the path of test prompts (#2273)
Zhuohan Li committed almost 3 years ago
[BUGFIX] Fix API server test (#2270)
Zhuohan Li committed almost 3 years ago