A high-throughput and memory-efficient inference and serving engine for LLMs
main
[Bugfix][Multimodal] Scope cache hash kwargs by modality (#54918)
waizuichougou committed about 2 months ago
[CI] Add Kimi-K3-pruned75-DSpark-TP4 gsm8k eval (#54817)
Misha Goin committed about 2 months ago
[Bugfix][KV Offload] Scale UniformTypeKVCacheSpecs groups by DCP (#50883)
drakosha committed about 2 months ago
[Bugfix][KV Offload] Ignore stale async lookup results (#54872)
AlexHuang committed about 2 months ago
[Hardware][XPU] Register matmul and linear batch-invariant kernels for XPU (#49209)
Tomasz Zielinski committed about 2 months ago
[Online quantization] Add targeted online quantization configuration based on user patterns (#51285)
fxmarty-amd committed about 2 months ago
[Bugfix][KV Offload] Ensure tracker progress for oversized offers (#54759)
Cheng Rui committed about 2 months ago
[Bugfix] `adjust_dcp_kv_cache_interleave_size` for NixlConnector only (#54803)
Nicolò Lucchesi committed about 2 months ago
Include chat template fallbacks in package_data (#53762)
Tarun Kumar committed about 2 months ago
[CI] Shard CPU jobs above the 24h P90 threshold (#54751)
Kevin H. Luu committed about 2 months ago
[Bugfix][ROCm][Build] fix profiler hang due to queue interposition bug (#54171)
Simon Danielsson committed about 2 months ago
[Bugfix] Raise for unavailable piecewise CUDA graphs (#54782)
Isotr0py committed about 2 months ago
[Model] Fix GLM-OCR MTP weight loading (#49869)
jackLei committed about 2 months ago
[CI][Spec Decode] Add MTP placeholder-token regression coverage (#54893)
Andreas Karatzas committed about 2 months ago
[CI][ROCm] Prefetch safetensors weights in AMD CI (#54898)
Andreas Karatzas committed about 2 months ago
[ROCm][CI] Add MTP and other spec-decode acceptance coverage (#53399)
Andreas Karatzas committed about 2 months ago
[Bugfix] Fix cross-batch buffer race corrupting DiskBackend loads (#51667)
Fabian Joswig committed about 2 months ago
[CI][AMD] Preserve diagnostics for unwritable checkouts (#53437)
Andreas Karatzas committed about 2 months ago
[CI] Shard LoRA TP distributed tests (#52350)
Kevin H. Luu committed about 2 months ago
[Bugfix] Fix launch render hanging on shutdown (#54913)
Chauncey committed about 2 months ago