A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 22:00 UTC
main
[VLM] Remove `image_input_type` from VLM config (#5852)
xwjiang2010 committed over 2 years ago
[Frontend] Add template related params to request (#5709)
danieljannai21 committed over 2 years ago
[Core] Optimize block_manager_v2 vs block_manager_v1 (to make V2 default) (#5602)
Alexander Matveev committed over 2 years ago
[Model] Changes to MLPSpeculator to support tie_weights and input_scale (#5965)
Thomas Parnell committed over 2 years ago
[Frontend] Relax api url assertion for openai benchmarking (#6046)
James Whedbee committed over 2 years ago
[BugFix] Ensure worker model loop is always stopped at the right time (#5987)
Nick Hill committed over 2 years ago
[Bugfix] Add explicit `end_forward` calls to flashinfer (#6044)
Antoni Baum committed over 2 years ago
[Bugfix][Doc] Fix Doc Formatting (#6048)
Roger Wang committed over 2 years ago
add FAQ doc under 'serving' (#5946)
ning.zhang committed over 2 years ago
[Bugfix] adding chunking mechanism to fused_moe to handle large inputs (#6029)
Avshalom Manevich committed over 2 years ago
[Bugfix] Use RayActorError for older versions of Ray in RayTokenizerGroupPool (#6039)
Antoni Baum committed over 2 years ago
[doc][misc] further lower visibility of simple api server (#6041)
youkaichao committed over 2 years ago
[Misc] update benchmark backend for scalellm (#6018)
zhyncs committed over 2 years ago
[doc][misc] remove deprecated api server in doc (#6037)
youkaichao committed over 2 years ago
[ CI ] Re-enable Large Model LM Eval (#6031)
Robert Shaw committed over 2 years ago
[Speculative Decoding 2/2 ] Integrate typical acceptance sampler into Spec Decode Worker (#5348)
sroy745 committed over 2 years ago
[misc][cuda] use nvml to avoid accidentally cuda initialization (#6007)
youkaichao committed over 2 years ago
[ Misc ] Refactor w8a8 to use `process_weights_after_load` (Simplify Weight Loading) (#5940)
Robert Shaw committed over 2 years ago
[Misc] Fix `get_min_capability` (#5971)
Dipika Sikka committed over 2 years ago
[ CI ] Temporarily Disable Large LM-Eval Tests (#6005)
Robert Shaw committed over 2 years ago