| 1 | # JIT Kernel Warmup |
| 2 | |
| 3 | vLLM uses JIT-generated kernels from Triton, CuTeDSL, TileLang, and other backends. This contract makes their required specializations available during startup, before the first request, by warming the kernel's **compile-key space** without dummy runtime launches or real tensor allocation. |
| 4 | |
| 5 | Use it when adding a warmable JIT kernel or migrating an existing warmup path. |
| 6 | |
| 7 | ## In This Guide |
| 8 | |
| 9 | - [1. Quickstart](#1-quickstart): for contributors adding or migrating a warmable kernel. |
| 10 | - [2. Search-Space Reference](#2-search-space-reference): additional details regarding warmup input expansion and traced dispatch rules. |
| 11 | |
| 12 | ## 1. Quickstart |
| 13 | |
| 14 | Each warmable kernel defines its compile-key mapping and compile-only entry point beside its normal runtime implementation. The startup registry then warms only the wrappers selected by the current engine configuration. |
| 15 | |
| 16 | ### Define the Kernel Wrapper |
| 17 | |
| 18 | Here, a **kernel wrapper** (or just **wrapper**) is an instance of a concrete `VllmJitKernel` subclass. |
| 19 | |
| 20 | Expose one wrapper near the kernel's normal runtime entry point. Prefer this backend-agnostic shape: |
| 21 | |
| 22 | ```python |
| 23 | class MyKernel(VllmJitKernel["MyKernel.CompileKey"]): |
| 24 | |
| 25 | @dataclass(frozen=True) |
| 26 | class CompileKey: |
| 27 | ... |
| 28 | |
| 29 | @staticmethod |
| 30 | def kernel(...): |
| 31 | ... |
| 32 | |
| 33 | def dispatch(self, ...) -> CompileKey: |
| 34 | return self.CompileKey(...) |
| 35 | |
| 36 | def get_warmup_keys(self, ...) -> list[CompileKey]: |
| 37 | return self._trace_dispatch(self.dispatch)(...) |
| 38 | |
| 39 | def compile(self, compile_key: CompileKey) -> None: |
| 40 | ... |
| 41 | |
| 42 | def __call__(self, ...): |
| 43 | return self.kernel(...) |
| 44 | |
| 45 | |
| 46 | _MY_KERNEL = MyKernel() |
| 47 | ``` |
| 48 | |
| 49 | `CompileKey`, `dispatch(...)`, and `get_warmup_keys(...)` are backend-agnostic. Backend-specific behavior belongs in `kernel(...)`, `compile(...)`, and `__call__(...)`. |
| 50 | |
| 51 | The module-level singleton should be used by warmup and by the runtime call path. This keeps dispatch behavior shared instead of duplicated. |
| 52 | |
| 53 | `VllmJitKernel.warmup(...)` materializes every key returned by `get_warmup_keys(...)`; wrappers should not reimplement it. |
| 54 | |
| 55 | ### Choose Compile-Key Fields |
| 56 | |
| 57 | `CompileKey` must be frozen and hashable. Include only fields on which the backend specializes, such as tile sizes, head dimensions, dtypes, pointer alignment classes, or backend selectors; exclude runtime-only values. When unsure, inspect the backend cache key, specialization arguments, or verbose JIT-monitor output. |
| 58 | |
| 59 | ### Generate Warmup Keys |
| 60 | |
| 61 | Use `_trace_dispatch(self.dispatch)` to describe representative inputs. The tracer maps them through the same specialization logic and deduplicates equal keys: |
| 62 | |
| 63 | ```python |
| 64 | def get_warmup_keys(self, vllm_config: VllmConfig) -> list[CompileKey]: |
| 65 | max_tokens = vllm_config.scheduler_config.max_num_batched_tokens |
| 66 | return self._trace_dispatch(self.dispatch)( |
| 67 | num_tokens=WarmupIntRange(1, max_tokens + 1), |
| 68 | ) |
| 69 | ``` |
| 70 | |
| 71 | Use independent ranges or alternatives for cartesian products, `zip_inputs(...)` for coupled rows, and `_when` for validity constraints. The complete syntax is documented in [Search-Space Reference](#2-search-space-reference). |
| 72 | |
| 73 | ### Materialize Without Launching |
| 74 | |
| 75 | The base contract uses `compile(compile_key)` internally to mean "make this specialization available". Depending on the backend, that may compile from source, call a compile-only API, load an already-built artifact, or compile on cache miss. Contributors only implement this hook when their backend helper does not provide it. |
| 76 | |
| 77 | For Triton, implement `warmup_inputs(...)`. `@kernel_launcher` binds those inputs to `__call__(...)`, whose body returns the runtime grid and derived launch arguments as a `LaunchSpec`. The shared helper then routes that specification to the normal kernel launch or to `kernel.warmup(...)` during startup. |
| 78 | |
| 79 | Materialization should not launch a real inference workload or allocate real tensors. `TritonWarmupTensor(..., strides=None)` represents compact row-major storage; provide explicit representative strides for padded, transposed, or otherwise strided runtime inputs. |
| 80 | |
| 81 | ### Register the Selected Wrapper |
| 82 | |
| 83 | Register the wrapper where the runtime implementation is selected: |
| 84 | |
| 85 | ```python |
| 86 | _MY_KERNEL.register_warmup() |
| 87 | ``` |
| 88 | |
| 89 | Registration records metadata only. It does not compile or launch the kernel. Repeated registrations from equivalent layers are allowed and deduplicated later. |
| 90 | |
| 91 | ### Review Checklist |
| 92 | |
| 93 | - Warm actual compile keys rather than representative non-key inputs. |
| 94 | - Keep specialization mapping in `dispatch(...)` instead of duplicating it in warmup code. |
| 95 | - Use fake tensors or backend compile-only descriptors; never perform a dummy runtime launch. |
| 96 | - Keep registration metadata-only so model construction remains cheap and side-effect free. |
| 97 | - Materialize registered kernels only through `kernel_warmup()`. |
| 98 | - Keep runtime execution and startup compilation separate and easy to review. |
| 99 | - Use one module-level wrapper instance for registration and runtime calls. |
| 100 | |
| 101 | ## 2. Search-Space Reference |
| 102 | |
| 103 | ### How Tracing Works |
| 104 | |
| 105 | `_trace_dispatch(...)` expands the inputs declared by `get_warmup_keys(...)`. Each concrete combination becomes a `dispatch_values` mapping from input names to selected values. `_when` may reject that mapping; otherwise the tracer evaluates `dispatch(...)` to construct one `CompileKey`. Equal keys are deduplicated after all combinations are evaluated. |
| 106 | |
| 107 | One call to `dispatch(...)` returns one key, but many input points may map to the same key. Prefer this traced mapping over manually reconstructing keys in warmup code; `dispatch(...)` should express the same specialization logic used by the runtime path. |
| 108 | |
| 109 | ### Define Input Spaces |
| 110 | |
| 111 | Use ranges and alternatives for independent axes, `zip_inputs(...)` for coupled rows, and `_when` for validity constraints. |
| 112 | |
| 113 | #### Integer Ranges |
| 114 | |
| 115 | Use `WarmupIntRange` for integer ranges: |
| 116 | |
| 117 | ```python |
| 118 | return self._trace_dispatch(self.dispatch)( |
| 119 | num_prefills=WarmupIntRange(1, max_prefills + 1), |
| 120 | ) |
| 121 | ``` |
| 122 | |
| 123 | `WarmupIntRange(start, stop, step)` follows Python `range(...)` semantics: `start` is inclusive, `stop` is exclusive, and `step` defaults to 1. |
| 124 | |
| 125 | For non-linear integer sequences, use `advance` to provide the action that computes each next value: |
| 126 | |
| 127 | ```python |
| 128 | return self._trace_dispatch(self.dispatch)( |
| 129 | num_tokens=WarmupIntRange( |
| 130 | 1, |
| 131 | max_tokens + 1, |
| 132 | advance=lambda value: next_power_of_2(value) + 1, |
| 133 | ), |
| 134 | ) |
| 135 | ``` |
| 136 | |
| 137 | This is useful for traversing specialization boundaries without enumerating every integer. `advance` cannot be combined with a non-default `step`, and it must return a value greater than its input so expansion always makes forward progress. |
| 138 | |
| 139 | #### Independent Alternatives |
| 140 | |
| 141 | Use tuples or lists for independent alternatives. Multiple expanded inputs form a cartesian product: |
| 142 | |
| 143 | ```python |
| 144 | return self._trace_dispatch(self.dispatch)( |
| 145 | query_slice_start=WarmupIntRange(0, 2), |
| 146 | query_slice_stop=(1, 2 * max_tokens - 1, 2 * max_tokens), |
| 147 | COMPRESS_RATIO=list(compress_ratios), |
| 148 | ) |
| 149 | ``` |
| 150 | |
| 151 | #### Coupled Inputs |
| 152 | |
| 153 | Use `zip_inputs(...)` when values must vary together row-by-row: |
| 154 | |
| 155 | ```python |
| 156 | WARMUP_INPUTS = zip_inputs( |
| 157 | dict(compress_ratio=1, topk=0, topk_width=512), |
| 158 | dict(compress_ratio=4, topk=512, topk_width=512), |
| 159 | ) |
| 160 | |
| 161 | |
| 162 | return self._trace_dispatch(self.dispatch)( |
| 163 | WARMUP_INPUTS, |
| 164 | WINDOW_SIZE=window_size, |
| 165 | ) |
| 166 | ``` |
| 167 | |
| 168 | Multiple `zip_inputs(...)` groups may be passed as positional arguments. The tracer forms the cartesian product across groups while preserving row-wise coupling inside each group. |
| 169 | |
| 170 | Every row in a `zip_inputs(...)` group must use the same string keys. A `zip_inputs(...)` group cannot specify a field that is also specified as a keyword input to `_trace_dispatch(...)`. |
| 171 | |
| 172 | #### Conditional Filtering |
| 173 | |
| 174 | Use `_when=...` to filter generated input points before they are passed to `dispatch(...)`. This is useful when independent ranges contain invalid combinations, but the validity rule belongs with the kernel warmup definition. |
| 175 | |
| 176 | ```python |
| 177 | def _is_valid_warmup_input( |
| 178 | self, |
| 179 | *, |
| 180 | query_len: int, |
| 181 | num_reqs: int, |
| 182 | max_num_batched_tokens: int, |
| 183 | ) -> bool: |
| 184 | return query_len + num_reqs - 1 <= max_num_batched_tokens |
| 185 | |
| 186 | |
| 187 | return self._trace_dispatch(self.dispatch)( |
| 188 | query_len=WarmupIntRange(1, max_tokens + 1), |
| 189 | num_reqs=WarmupIntRange(1, max_reqs + 1), |
| 190 | max_num_batched_tokens=max_tokens, |
| 191 | _when=self._is_valid_warmup_input, |
| 192 | ) |
| 193 | ``` |
| 194 | |
| 195 | `_when` accepts a function, bound method, or lambda and supports the same AST subset as `dispatch(...)`, including local assignments in function predicates. |
| 196 | |
| 197 | The predicate is evaluated on the expanded warmup inputs. If it returns `False`, that input point is skipped and no `CompileKey` is produced for it. |
| 198 | |
| 199 | ### Write Dispatch Rules |
| 200 | |
| 201 | #### Local Assignments |
| 202 | |
| 203 | The traced body may contain local assignments, optionally annotated, followed by one `return self.CompileKey(...)` call. Local assignments let a kernel name intermediate specialization choices once and reuse them across fields: |
| 204 | |
| 205 | ```python |
| 206 | def dispatch( |
| 207 | self, |
| 208 | *, |
| 209 | num_tokens: int, |
| 210 | vectorized: bool, |
| 211 | ) -> CompileKey: |
| 212 | block_size = next_power_of_2(num_tokens) |
| 213 | return self.CompileKey( |
| 214 | BLOCK_SIZE=block_size, |
| 215 | VECTOR_WIDTH=4 if vectorized and block_size >= 4 else 1, |
| 216 | ) |
| 217 | ``` |
| 218 | |
| 219 | #### Supported Expressions |
| 220 | |
| 221 | The evaluator supports these expressions inside local assignments and `CompileKey(...)` fields: |
| 222 | |
| 223 | | Feature | What It Allows | |
| 224 | | --- | --- | |
| 225 | | Names | Read dispatch inputs, local assignments, defaults, and module globals. | |
| 226 | | Constants | Use literals such as integers, strings, booleans, and `None`. | |
| 227 | | Attributes | Read structured values such as `cfg.block_size` or `mla_dims.v_head_dim`. | |
| 228 | | Subscriptions | Read sequence positions or mapping values such as `config[0]` and `config["block_size"]`. | |
| 229 | | Tuple/list literals | Build shapes, strides, and other small structured fields. | |
| 230 | | Conditional expressions | Select a field with `x if condition else y`. | |
| 231 | | Boolean expressions | Combine predicates with `and`, `or`, and `not`. | |
| 232 | | Comparisons | Use `==`, `!=`, `<`, `<=`, `>`, `>=`, `in`, `not in`, `is`, and `is not`. | |
| 233 | | Arithmetic | Use `+`, `-`, `*`, `//`, `%`, and `**`. | |
| 234 | | Unary minus | Build negative sentinel values or signed descriptors. | |
| 235 | | Helper calls | Call helpers with positional and explicit keyword arguments. | |
| 236 | |
| 237 | Python builtins such as `min(...)`, `max(...)`, and `len(...)` are resolved unless the name is overridden locally or globally. |
| 238 | |
| 239 | #### Helper Calls |
| 240 | |
| 241 | Helpers are useful for small specialization rules: |
| 242 | |
| 243 | ```python |
| 244 | def dispatch(self, *, num_tokens: int, block_size: int) -> CompileKey: |
| 245 | return self.CompileKey( |
| 246 | PADDED_TOKENS=round_up(num_tokens, multiple=block_size), |
| 247 | ) |
| 248 | ``` |
| 249 | |
| 250 | `_trace_dispatch(...)` does not inspect helper bodies. It evaluates the call arguments and invokes the helper as ordinary Python, so control flow inside that helper is outside the AST interpreter's scope. Keep helpers deterministic and side-effect free. |
| 251 | |
| 252 | #### Direct Keyword Forwarding |
| 253 | |
| 254 | For many direct pass-through fields, the dispatch `**kwargs` parameter may be unpacked into `CompileKey(...)`: |
| 255 | |
| 256 | ```python |
| 257 | def dispatch( |
| 258 | self, |
| 259 | *, |
| 260 | num_tokens: int, |
| 261 | **compile_key_fields: int, |
| 262 | ) -> CompileKey: |
| 263 | return self.CompileKey( |
| 264 | **compile_key_fields, |
| 265 | block_size=next_power_of_2(num_tokens), |
| 266 | ) |
| 267 | ``` |
| 268 | |
| 269 | Unmatched dispatch arguments become compile-key fields and warmup inputs. Keep transformed inputs named and explicit. The unpacking must use the dispatch method's own `**kwargs` parameter directly and exactly once; arbitrary mappings, repeated unpacking, and helper-call `**kwargs` are rejected. The fully explicit form remains supported and is often clearer for non-trivial mappings. |
| 270 | |
| 271 | #### Unsupported Syntax |
| 272 | |
| 273 | Conditional expressions (`x if condition else y`) are supported, but statement-level `if` blocks are not supported directly inside traced `dispatch(...)` or `_when` bodies. The tracer expects a straight-line sequence of local assignments followed by one return expression. Small, pure helpers called by traced expressions execute as normal Python with concrete values and may use ordinary control flow, including `if` blocks. Do not put loops, mutation, side effects, or backend imports directly inside traced functions. Put environment and model gating in `get_warmup_keys(...)` or the outer warmup entry point. |
| 274 | |
| 275 | ### Compile-Key Deduplication |
| 276 | |
| 277 | `_trace_dispatch(...)` deduplicates the resulting keys while preserving order. This is important when many runtime-like inputs map to the same static bucket. |
| 278 | |
| 279 | For example, this warmup range expands every token count, but the compile key only depends on the power-of-two bucket: |
| 280 | |
| 281 | ```python |
| 282 | def dispatch( |
| 283 | self, |
| 284 | *, |
| 285 | num_tokens: int, |
| 286 | ) -> CompileKey: |
| 287 | return self.CompileKey( |
| 288 | BLOCK_SIZE=next_power_of_2(num_tokens), |
| 289 | ) |
| 290 | |
| 291 | |
| 292 | def get_warmup_keys(self, vllm_config: VllmConfig) -> list[CompileKey]: |
| 293 | max_tokens = vllm_config.scheduler_config.max_num_batched_tokens |
| 294 | return self._trace_dispatch(self.dispatch)( |
| 295 | num_tokens=WarmupIntRange(1, max_tokens + 1), |
| 296 | ) |
| 297 | ``` |
| 298 | |
| 299 | For `max_tokens == 8`, the expanded inputs are `1, 2, 3, 4, 5, 6, 7, 8`, but the returned keys are: |
| 300 | |
| 301 | ```python |
| 302 | [ |
| 303 | CompileKey(BLOCK_SIZE=1), |
| 304 | CompileKey(BLOCK_SIZE=2), |
| 305 | CompileKey(BLOCK_SIZE=4), |
| 306 | CompileKey(BLOCK_SIZE=8), |
| 307 | ] |
| 308 | ``` |
| 309 | |
| 310 | Deduplication happens after `dispatch(...)` is evaluated, so the warmup system removes duplicate compile keys, not duplicate input values. `CompileKey` must be hashable for this to work; using `@dataclass(frozen=True)` is the standard pattern. |