@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / e691918e3b
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 23:15 UTC

[misc] Do not allow to use lora with chunked prefill. (#5538)

Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

SangBin Cho committed Jun 15, 2024 at 23:59 UTC e691918e3bd75a05bc473c77577c494aa6442640
1 file changed +2
vllm/config.py
+2
@@ -1092,6 +1092,8 @@ class LoRAConfig:
1092 "Due to limitations of the custom LoRA CUDA kernel, "
1093 "max_num_batched_tokens must be <= 65528 when "
1094 "LoRA is enabled.")
1095 + if scheduler_config.chunked_prefill_enabled:
1096 + raise ValueError("LoRA is not supported with chunked prefill yet.")
1097
1098
1099 @dataclass