A high-throughput and memory-efficient inference and serving engine for LLMs
main
[XPU][TEST] Add entrypoints test in Intel GPU CI (#53980)
xiangdong committed about 2 months ago
[Bugfix][Security] Bound embedding densification before to_dense() (#54632)
lzhan011 committed about 2 months ago
[Bugfix][Frontend] Preserve token offset origins after left text pre-trimming (#54692)
waizuichougou committed about 2 months ago
[CI][ROCm] Expand weight loading test coverage on AMD and cap its KV cache (#54037)
stefankoncarevic committed about 2 months ago
fxmarty-amd committed about 2 months ago
[Distributed] Support pre-shared ncclUniqueId rendezvous for weight transfer (#53784)
dharak-cohere committed about 2 months ago
[Bugfix][Security] Bound the validation-error response body (#54684)
lzhan011 committed about 2 months ago
[Core] Release NCCL communicator memory in sleep mode (#51485)
aoshen02 committed about 2 months ago
[Feature][MM_UUIDs] Allow empty video URLs when using multi-modal UUIDs (#54220)
Jan Kwieciński committed about 2 months ago
[Bugfix][Frontend] Truncate the assistant tokens mask with the prompt (#54539)
Hotragn Pettugani committed about 2 months ago
[DecodeBenchConnector] Fix HMA cache-group mapping (#54647)
Jeff (Junze) Ma committed about 2 months ago
[ROCm][CI] Stabilize the sqrt-softplus top-k tie oracle (#54403)
Andreas Karatzas committed about 2 months ago
[Attention][DSA] Enable W4A16 DSA (#51724)
sychen52 committed about 2 months ago
[ROCm] Keep GLM-5.2 on MRV1 and disable default breakable cudagraph (#53155)
Rohan Potdar committed about 2 months ago
[ROCm][MoE] Split AITER CK and Triton MXFP4 W4A16 into separate backends (#50622)
afriedri committed about 2 months ago
[Distributed] Add opt-in FlashInfer PCIe IPC all-reduce backend (#53576)
Gabriel Wu committed about 2 months ago
[1/N][warmup][DSv4] Migrate generic MLA metadata and indexing kernels (#50175)
Roberto L. Castro committed about 2 months ago
[Bugfix][EC Connector] Fall back when MADV_POPULATE_WRITE is unsupported (#53190)
xiaolinchen committed about 2 months ago
[Feat] Add request-level preemption count histogram metric (#49984)
Amy (Yu-Hsuan) Lin committed about 2 months ago
[Kernel] add Flashinfer cutedsl w4a16 linear (#53014)
Lain committed about 2 months ago