A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 05:45 UTC
[torch.compile][misc] fix comments (#10993)
Signed-off-by: youkaichao <youkaichao@gmail.com>
youkaichao committed
Dec 8, 2024 at 11:18 UTC
43b05fa314e90e551d87211e8bdde2e2bb5a0bdc
1 file changed
+2
-2
vllm/config.py
+2
-2
@@ -2177,8 +2177,8 @@ class CompilationConfig(BaseModel):
2177
TODO: move outside cudagraph logic into compilation.
2178
torch.compile will handle cudagraph capture logic in the future.
2179
- cudagraph_capture_sizes: sizes to capture cudagraph.
2180
- - None: capture sizes are inferred from compilation context.
2181
- - List[int]: capture sizes are specified.
2180
+ - None (default): capture sizes are inferred from vllm config.
2181
+ - List[int]: capture sizes are specified as given.
2182
- cudagraph_num_of_warmups: number of warmup runs for cudagraph.
2183
It means the first several runs will be treated as warmup runs.
2184
Only after that, the execution will be recorded, and the recorded