@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 91f4522cbf
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 18:15 UTC

[multi-step] Raise error if not using async engine (#7703)

William Lin committed Aug 21, 2024 at 11:49 UTC 91f4522cbf85df0a65f619b25f6751edf2d5f0d6
1 file changed +5
vllm/engine/llm_engine.py
+5
@@ -1302,6 +1302,11 @@ class LLMEngine:
1302 raise NotImplementedError(
1303 "Pipeline parallelism is only supported through AsyncLLMEngine "
1304 "as performance will be severely degraded otherwise.")
1305 +
1306 + if self.scheduler_config.num_scheduler_steps > 1:
1307 + raise NotImplementedError(
1308 + "Multiple scheduler steps (multi-step) are only supported "
1309 + "through AsyncLLMEngine. ")
1310 seq_group_metadata_list, scheduler_outputs = self.scheduler[
1311 0].schedule()
1312