A high-throughput and memory-efficient inference and serving engine for LLMs
main
[CI] Reuse loaded config for cached tokenizer (#49509)
Andreas Karatzas committed 3 months ago
Add `sm_107` for Rubin (#49387)
Tyler Michael Smith committed 3 months ago
[CI][AMD] Deprecate DinD for MI355 tests (#49257)
Andreas Karatzas committed 3 months ago
[Kernel] ReplaySSM: cache SSM inputs for faster Mamba2 standard decode (#48018)
Johnny-Liou committed 3 months ago
[Bugfix] Accept RFC 2397 parameters in base64 data URLs (#48973)
Thomas Fahrner committed 3 months ago
[UX] Improve data-parallel launch validation (#49124)
Taneem Ibrahim committed 3 months ago
[UX] Reject incompatible nested runtime overrides (#49247)
Taneem Ibrahim committed 3 months ago
[ROCM] Fix AITER Fused AllReduce RMSNorm for Transformers Backend (#49673)
BadrBasowid committed 3 months ago
[Bugfix][Benchmarks] Restore --skip-tokenizer-init with custom dataset (#49180)
mgazz committed 3 months ago
Encoder cache extension hooks (#48218)
hotTea committed 3 months ago
[Bugfix] Skip linear bias in layerwise reload to avoid corruption (#49586)
li-jinpeng committed 3 months ago
[PD][NixlPush][Bugfix] Fix blocking handshake call on writer thread (#49221)
Nicolò Lucchesi committed 3 months ago
Remove Quantization test parallelism (#49693)
Kevin H. Luu committed 3 months ago
[Perf][GLM-5.2] Blackwell decode optimizations (#48597)
xiaozhoupy committed 3 months ago
Bump Transformers version to 5.14.1 (#49223)
Harry Mellor committed 3 months ago
[CI] Use explicit devices in IR tests (#49513)
Andreas Karatzas committed 3 months ago
[Bugfix] Fix humming kernel crash when layer.has_bias is None (#48769)
Kyle Sayers committed 3 months ago
[CompressedTensors] DeepSeek4 CT Quantization Support (#41276)
Kyle Sayers committed 3 months ago
[Bugfix] Detect mixed precision in packed KV cache specs (#49623)
Michael Goin committed 3 months ago
[Model] Support llm-compressor Inkling NVFP4 weights (#49258)
Michael Goin committed 3 months ago