A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 18:30 UTC
[FrontEnd][Perf] `merge_async_iterators` fast-path for single-prompt requests (#15150)
Signed-off-by: Nick Hill <nhill@redhat.com>
Nick Hill committed
Mar 19, 2025 at 14:04 UTC
22d33baca2c0c639cfd45c48e99803e56c3efa74
1 file changed
+5
vllm/utils.py
+5
@@ -411,6 +411,11 @@ async def merge_async_iterators(
411
When it yields, it yields a tuple (i, item) where i is the index of the
412
iterator that yields the item.
413
"""
414
+ if len(iterators) == 1:
415
+ # Fast-path single iterator case.
416
+ async for item in iterators[0]:
417
+ yield 0, item
418
+ return
419
420
loop = asyncio.get_running_loop()
421