[Doc] Surface the --kv-cache-memory suggestion at INFO and document fast-startup knobs (#47374)
Signed-off-by: Nils Matteson <nils@thaw.sh> Signed-off-by: Nils Matteson <nilsmatteson@icloud.com> Co-authored-by: Nils Matteson <nils@thaw.sh> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Nils Matteson committed
Jul 7, 2026 at 16:05 UTC
dd0d74cd921ed8b3651fc5bbc5ce2b1296842fcd
2 files changed
+9
-1
docs/configuration/optimization.md
+8
@@ -16,6 +16,14 @@ vLLM provides 4 optimization levels (`-O0`, `-O1`, `-O2`, `-O3`) that allow user
16
17
For more information, see the [optimization level documentation](../design/optimization_levels.md).
18
19
+## Faster Startup
20
+
21
+Beyond the optimization levels, three mechanisms reduce time-to-first-token on repeated boots of the same (model, config, hardware) combination:
22
+
23
+- **Reuse the compile cache.** vLLM persists `torch.compile` artifacts under `VLLM_CACHE_ROOT` (default `~/.cache/vllm`), and the cache directory can be copied between machines or baked into a container image; see the [torch.compile design doc](../design/torch_compile.md). Set `VLLM_FORCE_AOT_LOAD=1` to fail loudly instead of silently recompiling when the cache misses (any change to the model, config, relevant `VLLM_*` environment variables, torch build, or GPU model invalidates it).
24
+- **Skip memory profiling with `--kv-cache-memory`.** On startup, vLLM logs the exact `--kv-cache-memory` value that reproduces the current allocation. Passing it back on the next boot skips the memory-profiling measurement and the CUDA-graph memory estimation pass. Note that this has performance implications: the KV cache is sized to exactly the given value instead of being measured, so a conservative value caps batch concurrency (and therefore throughput), while an optimistic one fails at allocation time. The value is only valid on the same GPU with the same initial free memory; if a boot OOMs after hardware or co-tenant changes, remove the flag to re-profile.
25
+- **Serve without CUDA graphs using `--enforce-eager`.** Skips both compilation and CUDA-graph capture for the fastest possible startup, at the cost of steady-state decode performance. Useful for development loops and for measuring how much of a boot is compile/capture.
26
+
27
## Preemption
28
29
Due to the autoregressive nature of transformer architecture, there are times when KV cache space is insufficient to handle all batched requests.
vllm/v1/worker/gpu_worker.py
+1
-1
@@ -832,7 +832,7 @@ class Worker(WorkerBase):
832
f"{format_gib(self.available_kv_cache_memory_bytes)} GiB."
833
)
834
835
- logger.debug(msg)
835
+ logger.info(msg)
836
837
if self.use_v2_model_runner:
838
# V2: Run full execute_model + sample_tokens to JIT compile triton kernels.