A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Core] Run garbage collector after CUDA graph capture to fix throughput regression (#24128)
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com> Co-authored-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Micah Williamson committed
Sep 9, 2025 at 09:38 UTC
1c63a16b653d1f3f4260e862f01eaffed283c4c8
1 file changed
+1
vllm/v1/worker/gpu_model_runner.py
+1
@@ -2885,6 +2885,7 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin):
2885
finally:
2886
if should_freeze:
2887
gc.unfreeze()
2888
+ gc.collect()
2889
2890
# Trigger CUDA graph capture for specific shapes.
2891
# Capture the large shapes first so that the smaller shapes