@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 474 lines 23.6 KB
Rendered Raw
1 # Optimization and Tuning
2
3 This guide covers optimization strategies and performance tuning for vLLM V1.
4
5 !!! tip
6 Running out of memory? Consult [this guide](./conserving_memory.md) on how to conserve memory.
7
8 ## Optimization Levels
9
10 vLLM provides 4 optimization levels (`-O0`, `-O1`, `-O2`, `-O3`) that allow users to trade off startup time for performance:
11
12 - `-O0`: No optimizations. Fastest startup time, but lowest performance.
13 - `-O1`: Fast optimization. Simple compilation and fast fusions, and PIECEWISE cudagraphs.
14 - `-O2`: Default optimization. Additional compilation ranges, additional fusions, FULL_AND_PIECEWISE cudagraphs.
15 - `-O3`: Aggressive optimization. Currently equal to `-O2`, but may include additional time-consuming or experimental optimizations in the future.
16
17 For more information, see the [optimization level documentation](../design/optimization_levels.md).
18
19 ## Faster Startup
20
21 Beyond the optimization levels, three mechanisms reduce time-to-first-token on repeated boots of the same (model, config, hardware) combination:
22
23 - **Reuse the compile cache.** vLLM persists `torch.compile` artifacts under `VLLM_CACHE_ROOT` (default `~/.cache/vllm`), and the cache directory can be copied between machines or baked into a container image; see the [torch.compile design doc](../design/torch_compile.md). Set `VLLM_FORCE_AOT_LOAD=1` to fail loudly instead of silently recompiling when the cache misses (any change to the model, config, relevant `VLLM_*` environment variables, torch build, or GPU model invalidates it).
24 - **Skip memory profiling with `--kv-cache-memory`.** On startup, vLLM logs the exact `--kv-cache-memory` value that reproduces the current allocation. Passing it back on the next boot skips the memory-profiling measurement and the CUDA-graph memory estimation pass. Note that this has performance implications: the KV cache is sized to exactly the given value instead of being measured, so a conservative value caps batch concurrency (and therefore throughput), while an optimistic one fails at allocation time. The value is only valid on the same GPU with the same initial free memory; if a boot OOMs after hardware or co-tenant changes, remove the flag to re-profile.
25 - **Serve without CUDA graphs using `--enforce-eager`.** Skips both compilation and CUDA-graph capture for the fastest possible startup, at the cost of steady-state decode performance. Useful for development loops and for measuring how much of a boot is compile/capture.
26
27 ## Preemption
28
29 Due to the autoregressive nature of transformer architecture, there are times when KV cache space is insufficient to handle all batched requests.
30 In such cases, vLLM can preempt requests to free up KV cache space for other requests. Preempted requests are recomputed when sufficient KV cache space becomes
31 available again. When this occurs, you may see the following warning:
32
33 ```text
34 WARNING 05-09 00:49:33 scheduler.py:1057 Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory. total_cumulative_preemption_cnt=1
35 ```
36
37 While this mechanism ensures system robustness, preemption and recomputation can adversely affect end-to-end latency.
38 If you frequently encounter preemptions, consider the following actions:
39
40 - Increase `gpu_memory_utilization`. vLLM pre-allocates GPU cache using this percentage of memory. By increasing utilization, you can provide more KV cache space.
41 - Decrease `max_num_seqs` or `max_num_batched_tokens`. This reduces the number of concurrent requests in a batch, thereby requiring less KV cache space.
42 - Increase `tensor_parallel_size`. This shards model weights across GPUs, allowing each GPU to have more memory available for KV cache. However, increasing this value may cause excessive synchronization overhead.
43 - Increase `pipeline_parallel_size`. This distributes model layers across GPUs, reducing the memory needed for model weights on each GPU, indirectly leaving more memory available for KV cache. However, increasing this value may cause latency penalties.
44
45 You can monitor the number of preemption requests through Prometheus metrics exposed by vLLM. Additionally, you can log the cumulative number of preemption requests by setting `disable_log_stats=False`.
46
47 In vLLM V1, the default preemption mode is `RECOMPUTE` rather than `SWAP`, as recomputation has lower overhead in the V1 architecture.
48
49 ## Chunked Prefill
50
51 Chunked prefill allows vLLM to process large prefills in smaller chunks and batch them together with decode requests. This feature helps improve both throughput and latency by better balancing compute-bound (prefill) and memory-bound (decode) operations.
52
53 In V1, **chunked prefill is enabled by default whenever possible**. With chunked prefill enabled, the scheduling policy prioritizes decode requests. It batches all pending decode requests before scheduling any prefill operations. When there are available tokens in the `max_num_batched_tokens` budget, it schedules pending prefills. If a pending prefill request cannot fit into `max_num_batched_tokens`, it automatically chunks it.
54
55 This policy has two benefits:
56
57 - It improves inter-token latency (ITL) and generation decode because decode requests are prioritized.
58 - It helps achieve better GPU utilization by locating compute-bound (prefill) and memory-bound (decode) requests to the same batch.
59
60 ### Performance Tuning with Chunked Prefill
61
62 You can tune the performance by adjusting `max_num_batched_tokens`:
63
64 - Smaller values (e.g., 2048) achieve better ITL because there are fewer prefills slowing down decodes.
65 - Higher values achieve better time to first token (TTFT) as you can process more prefill tokens in a batch.
66 - For optimal throughput, we recommend setting `max_num_batched_tokens > 8192` especially for smaller models on large GPUs.
67 - If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the V0 default scheduling policy (except that it still prioritizes decodes).
68
69 !!! warning
70 When chunked prefill is disabled, `max_num_batched_tokens` must be greater than `max_model_len`.
71 In that case, if `max_num_batched_tokens < max_model_len`, vLLM may crash at server start‑up.
72
73 ```python
74 from vllm import LLM
75
76 # Set max_num_batched_tokens to tune performance
77 llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", max_num_batched_tokens=16384)
78 ```
79
80 See related papers for more details (<https://arxiv.org/pdf/2401.08671> or <https://arxiv.org/pdf/2308.16369>).
81
82 ## Parallelism Strategies
83
84 vLLM supports multiple parallelism strategies that can be combined to optimize performance across different hardware configurations.
85
86 ### Tensor Parallelism (TP)
87
88 Tensor parallelism shards model parameters across multiple GPUs within each model layer. This is the most common strategy for large model inference within a single node.
89
90 **When to use:**
91
92 - When the model is too large to fit on a single GPU
93 - When you need to reduce memory pressure per GPU to allow more KV cache space for higher throughput
94
95 ```python
96 from vllm import LLM
97
98 # Split model across 4 GPUs
99 llm = LLM(model="meta-llama/Llama-3.3-70B-Instruct", tensor_parallel_size=4)
100 ```
101
102 For models that are too large to fit on a single GPU (like 70B parameter models), tensor parallelism is essential.
103
104 ### Pipeline Parallelism (PP)
105
106 Pipeline parallelism distributes model layers across multiple GPUs. Each GPU processes different parts of the model in sequence.
107
108 **When to use:**
109
110 - When you've already maxed out efficient tensor parallelism but need to distribute the model further, or across nodes
111 - For very deep and narrow models where layer distribution is more efficient than tensor sharding
112
113 Pipeline parallelism can be combined with tensor parallelism for very large models:
114
115 ```python
116 from vllm import LLM
117
118 # Combine pipeline and tensor parallelism
119 llm = LLM(
120 model="meta-llama/Llama-3.3-70B-Instruct",
121 tensor_parallel_size=4,
122 pipeline_parallel_size=2,
123 )
124 ```
125
126 ### Expert Parallelism (EP)
127
128 Expert parallelism is a specialized form of parallelism for Mixture of Experts (MoE) models, where different expert networks are distributed across GPUs.
129
130 **When to use:**
131
132 - Specifically for MoE models (like DeepSeekV3, Qwen3MoE, Llama-4)
133 - When you want to balance the expert computation load across GPUs
134
135 Expert parallelism is enabled by setting `enable_expert_parallel=True`, which will use expert parallelism instead of tensor parallelism for MoE layers.
136 It will use the same degree of parallelism as what you have set for tensor parallelism.
137
138 ### Data Parallelism (DP)
139
140 Data parallelism replicates the entire model across multiple GPU sets and processes different batches of requests in parallel.
141
142 **When to use:**
143
144 - When you have enough GPUs to replicate the entire model
145 - When you need to scale throughput rather than model size
146 - In multi-user environments where isolation between request batches is beneficial
147
148 Data parallelism can be combined with the other parallelism strategies and is set by `data_parallel_size=N`.
149 Note that MoE layers will be sharded according to the product of the tensor parallel size and data parallel size.
150
151 ### NUMA Binding for Multi-Socket GPU Nodes
152
153 On multi-socket GPU servers, GPU worker processes can lose performance if their
154 CPU execution and memory allocation drift away from the NUMA node nearest to the
155 GPU. vLLM can pin each worker with `numactl` before the Python subprocess starts,
156 so the interpreter, imports, and early allocator state are created with the
157 desired NUMA policy from the beginning.
158
159 Use `--numa-bind` to enable the feature. By default, vLLM auto-detects the
160 GPU-to-NUMA mapping and uses `--cpunodebind=<node> --membind=<node>` for each
161 worker. When you need a custom CPU policy, add `--numa-bind-cpus` and vLLM will
162 switch to `--physcpubind=<cpu-list> --membind=<node>`.
163
164 These `--numa-bind*` options only apply to GPU execution processes. They do not
165 configure the CPU backend's separate thread-affinity controls. Automatic
166 GPU-to-NUMA detection is currently implemented for CUDA/NVML-based as well as
167 ROCM-based platforms; other GPU backends must provide explicit binding lists if
168 they use these options.
169
170 `--numa-bind-nodes` takes one non-negative NUMA node index per visible GPU, in
171 the same order as the GPU indices.
172 `--numa-bind-cpus` takes one `numactl` CPU list per visible GPU, in the same
173 order as the GPU indices. Each CPU list must use
174 `numactl --physcpubind` syntax such as `0-3`, `0,2,4-7`, or `16-31,48-63`.
175
176 ```bash
177 # Auto-detect NUMA nodes for visible GPUs
178 vllm serve meta-llama/Llama-3.1-8B-Instruct \
179 --tensor-parallel-size 4 \
180 --numa-bind
181
182 # Explicit NUMA-node mapping
183 vllm serve meta-llama/Llama-3.1-8B-Instruct \
184 --tensor-parallel-size 4 \
185 --numa-bind \
186 --numa-bind-nodes 0 0 1 1
187
188 # Explicit CPU pinning, useful for PCT or other high-frequency core layouts
189 vllm serve meta-llama/Llama-3.1-8B-Instruct \
190 --tensor-parallel-size 4 \
191 --numa-bind \
192 --numa-bind-nodes 0 0 1 1 \
193 --numa-bind-cpus 0-3 4-7 48-51 52-55
194 ```
195
196 Notes:
197
198 - CLI usage forces multiprocessing to use the `spawn` method automatically. If you enable NUMA binding through the Python API, also set `VLLM_WORKER_MULTIPROC_METHOD=spawn`.
199 - Automatic detection relies on NVML and NUMA support from the host. If it cannot determine the mapping reliably, pass `--numa-bind-nodes` explicitly.
200 - Explicit `--numa-bind-nodes` and `--numa-bind-cpus` values must be valid `numactl` inputs. vLLM does a small amount of validation, but the effective binding semantics are still determined by `numactl`.
201 - The current implementation binds GPU execution processes such as `EngineCore` and multiprocessing workers. It does not apply NUMA binding to frontend API server processes or the DP coordinator.
202 - In containerized environments, NUMA policy syscalls may require extra permissions, such as `--cap-add SYS_NICE` when running via `docker run`.
203
204 ### CPU Backend Thread Affinity
205
206 The CPU backend uses a different mechanism from `--numa-bind`. CPU execution is
207 configured through CPU-specific environment variables such as
208 `VLLM_CPU_OMP_THREADS_BIND`, `VLLM_CPU_NUM_OF_RESERVED_CPU`, and
209 `CPU_VISIBLE_MEMORY_NODES`, rather than the GPU-oriented `--numa-bind*` CLI
210 options.
211
212 By default, `VLLM_CPU_OMP_THREADS_BIND=auto` derives OpenMP placement from the
213 available CPU and NUMA topology for each CPU worker. To override the automatic
214 policy, set `VLLM_CPU_OMP_THREADS_BIND` explicitly using the CPU list format
215 documented for the CPU backend, or use `nobind` to disable this behavior.
216
217 For the current CPU backend setup and tuning guidance, see:
218
219 - [Related runtime environment variables](../getting_started/installation/cpu.md#related-runtime-environment-variables)
220 - [How to decide `VLLM_CPU_OMP_THREADS_BIND`](../getting_started/installation/cpu.md#how-to-decide-vllm_cpu_omp_threads_bind)
221
222 The GPU-only `--numa-bind`, `--numa-bind-nodes`, and `--numa-bind-cpus` options
223 do not configure CPU worker affinity.
224
225 ### Batch-level DP for Multi-Modal Encoders
226
227 By default, TP is used to shard the weights of multi-modal encoders just like for language decoders,
228 in order to reduce the memory and compute load on each GPU.
229
230 However, since the size of multi-modal encoders is very small compared to language decoders,
231 there is relatively little gain from TP. On the other hand, TP incurs significant communication
232 overhead because of all-reduce being performed after every layer.
233
234 Given this, it may be advantageous to instead shard the batched input data using TP, essentially
235 performing batch-level DP. This has been shown to improve the throughput and TTFT by around 10% for
236 `tensor_parallel_size=8`. For vision encoders that use hardware-unoptimized Conv3D operations,
237 batch-level DP can provide another 40% improvement compared to regular TP.
238
239 Nevertheless, since the weights of the multi-modal encoder are replicated across each TP rank,
240 there will be a minor increase in memory consumption and may cause OOM if you can barely fit the model already.
241
242 You can enable batch-level DP by setting `mm_encoder_tp_mode="data"`, for example:
243
244 ```python
245 from vllm import LLM
246
247 llm = LLM(
248 model="Qwen/Qwen2.5-VL-72B-Instruct",
249 tensor_parallel_size=4,
250 # When mm_encoder_tp_mode="data",
251 # the vision encoder uses TP=4 (not DP=1) to shard the input data,
252 # so the TP size becomes the effective DP size.
253 # Note that this is independent of the DP size for language decoder which is used in expert parallel setting.
254 mm_encoder_tp_mode="data",
255 # The language decoder uses TP=4 to shard the weights regardless
256 # of the setting of mm_encoder_tp_mode
257 )
258 ```
259
260 !!! important
261 Batch-level DP is not to be confused with API request-level DP
262 (which is instead controlled by `data_parallel_size`).
263
264 Batch-level DP needs to be implemented on a per-model basis,
265 and enabled by setting `supports_encoder_tp_data = True` in the model class.
266 Regardless, you need to set `mm_encoder_tp_mode="data"` in engine arguments to use this feature.
267
268 Known supported models (with corresponding benchmarks):
269
270 - dots_ocr (<https://github.com/vllm-project/vllm/pull/25466>)
271 - GLM-4.1V or above (<https://github.com/vllm-project/vllm/pull/23168>)
272 - InternVL (<https://github.com/vllm-project/vllm/pull/23909>)
273 - Kimi-VL (<https://github.com/vllm-project/vllm/pull/23817>)
274 - Llama4 (<https://github.com/vllm-project/vllm/pull/18368>)
275 - MiniCPM-V-2.5 or above (<https://github.com/vllm-project/vllm/pull/23327>, <https://github.com/vllm-project/vllm/pull/23948>)
276 - Qwen2-VL or above (<https://github.com/vllm-project/vllm/pull/22742>, <https://github.com/vllm-project/vllm/pull/24955>, <https://github.com/vllm-project/vllm/pull/25445>)
277 - Step3 (<https://github.com/vllm-project/vllm/pull/22697>)
278
279 ## Input Processing
280
281 ### fastokens Backend
282
283 By default vLLM uses the standard Hugging Face `tokenizers` library to power
284 the fast tokenizer. For BPE tokenizers (Qwen, Llama, DeepSeek, GPT-OSS, etc.)
285 you can switch to the [fastokens](https://github.com/crusoecloud/fastokens)
286 Rust backend, a drop-in replacement that's substantially faster on
287 encode/decode and on streaming detokenization. `VLLM_USE_FASTOKENS` is
288 available in vLLM v0.23.0 and later. If your installed vLLM version does not
289 recognize the environment variable, upgrade vLLM before enabling the override:
290
291 ```console
292 VLLM_USE_FASTOKENS=1 vllm serve Qwen/Qwen3-8B
293 ```
294
295 Equivalent in the offline API:
296
297 ```python
298 import os
299 os.environ["VLLM_USE_FASTOKENS"] = "1"
300
301 from vllm import LLM
302 llm = LLM(model="Qwen/Qwen3-8B")
303 ```
304
305 The `fastokens` Python package (>= 0.2.0) must be installed; if it isn't,
306 vLLM raises a clear `ImportError` at tokenizer load. The override applies to
307 any `--tokenizer-mode` that ends up loading an HF fast tokenizer (`hf`,
308 `deepseek_v32`, `deepseek_v4`, …). Models that don't use the HF
309 fast tokenizer (`mistral`, `kimi_audio`) ignore the flag.
310
311 Tokenizer-bound workloads — long shared prefixes, bursty short prompts,
312 batch detokenization — see the largest wins. If your bottleneck is GPU
313 prefill/decode, the tokenizer change is unlikely to be visible end-to-end.
314
315 ### Parallel Processing
316
317 You can run input processing in parallel via [API server scale-out](../serving/data_parallel_deployment.md#internal-load-balancing).
318 This is useful when input processing (which is run inside the API server)
319 becomes a bottleneck compared to model execution (which is run inside engine core)
320 and you have excess CPU capacity.
321
322 ```console
323 # Run 4 API processes and 1 engine core process
324 vllm serve Qwen/Qwen2.5-VL-3B-Instruct --api-server-count 4
325
326 # Run 4 API processes and 2 engine core processes
327 vllm serve Qwen/Qwen2.5-VL-3B-Instruct --api-server-count 4 -dp 2
328 ```
329
330 !!! note
331 API server scale-out is only available for online inference.
332
333 !!! warning
334 By default, 8 CPU threads are used in each API server to load media items (e.g. images)
335 from request data.
336
337 If you apply API server scale-out, consider adjusting `VLLM_MEDIA_LOADING_THREAD_COUNT`
338 to avoid CPU resource exhaustion.
339
340 !!! note
341 API server scale-out disables [multi-modal IPC caching](#ipc-caching)
342 because it requires a one-to-one correspondence between API and engine core processes.
343
344 This does not impact [multi-modal processor caching](#processor-caching).
345
346 ## Multi-Modal Caching
347
348 Multi-modal caching avoids repeated transfer or processing of the same multi-modal data,
349 which commonly occurs in multi-turn conversations.
350
351 ### Processor Caching
352
353 Multi-modal processor caching is automatically enabled
354 to avoid repeatedly processing the same multi-modal inputs in `BaseMultiModalProcessor`.
355
356 ### IPC Caching
357
358 Multi-modal IPC caching is automatically enabled when
359 there is a one-to-one correspondence between API (`P0`) and engine core (`P1`) processes,
360 to avoid repeatedly transferring the same multi-modal inputs between them.
361
362 #### Key-Replicated Cache
363
364 By default, IPC caching uses a **key-replicated cache**, where cache keys exist
365 in both the API (`P0`) and engine core (`P1`) processes, but the actual cache
366 data resides only in `P1`.
367
368 #### Shared Memory Cache
369
370 When multiple worker processes are involved (e.g., when TP > 1), a
371 **shared-memory cache** is more efficient. This can be enabled by setting
372 `mm_processor_cache_type="shm"`. In this mode, cache keys are stored
373 on `P0`, while the cache data itself lives in shared memory accessible by all
374 processes.
375
376 ### Configuration
377
378 You can adjust the size of the cache by setting the value of `mm_processor_cache_gb` (default 4 GiB).
379 A processed item larger than this budget is served uncached (with a warning)
380 instead of failing engine startup; raise `mm_processor_cache_gb` if you want
381 those items cached.
382
383 If you do not benefit much from the cache, you can disable both IPC
384 and processor caching completely via `mm_processor_cache_gb=0`.
385
386 Examples:
387
388 ```python
389 # Use a larger cache
390 llm = LLM(
391 model="Qwen/Qwen2.5-VL-3B-Instruct",
392 mm_processor_cache_gb=8,
393 )
394
395 # Use a shared-memory based IPC cache
396 llm = LLM(
397 model="Qwen/Qwen2.5-VL-3B-Instruct",
398 tensor_parallel_size=2,
399 mm_processor_cache_type="shm",
400 mm_processor_cache_gb=8,
401 )
402
403 # Disable the cache
404 llm = LLM(
405 model="Qwen/Qwen2.5-VL-3B-Instruct",
406 mm_processor_cache_gb=0,
407 )
408 ```
409
410 ### Cache Placement
411
412 Based on the configuration, the content of the multi-modal caches on `P0` and `P1` are as follows:
413
414 | mm_processor_cache_type | Cache Type | `P0` Cache | `P1` Engine Cache | `P1` Worker Cache | Max. Memory |
415 | ----------------- | ----------- | ---------- | ---------- | ----------- | ----------- |
416 | lru | Processor Caching | K + V | N/A | N/A | `mm_processor_cache_gb * data_parallel_size` |
417 | lru | Key-Replicated Caching | K | K + V | N/A | `mm_processor_cache_gb * api_server_count` |
418 | shm | Shared Memory Caching | K | N/A | V | `mm_processor_cache_gb * api_server_count` |
419 | N/A | Disabled | N/A | N/A | N/A | `0` |
420
421 K: Stores the hashes of multi-modal items
422 V: Stores the processed tensor data of multi-modal items
423
424 ## CPU Resources for GPU Deployments
425
426 vLLM V1 uses a multi-process architecture (see [V1 Process Architecture](../design/arch_overview.md#v1-process-architecture)) where each process requires CPU resources. Underprovisioning CPU cores is a common source of performance degradation, especially in virtualized environments.
427
428 ### Minimum CPU Requirements
429
430 For a deployment with `N` GPUs, there are at minimum:
431
432 - **1 API server process** -- handles HTTP requests, tokenization, and input processing
433 - **1 engine core process** -- runs the scheduler and coordinates GPU workers
434 - **N GPU worker processes** -- one per GPU, executes model forward passes
435
436 This means there are always at least **`2 + N` processes** competing for CPU time.
437
438 !!! warning
439 Using fewer physical CPU cores than processes will cause contention and significantly degrade throughput and latency. The engine core process runs a busy loop and is particularly sensitive to CPU starvation.
440
441 The minimum is `2 + N` physical cores (1 for the API server, 1 for the engine core, and 1 per GPU worker). In practice, allocating more cores improves performance because the OS, PyTorch background threads, and other system processes also need CPU time.
442
443 !!! important
444 Please note we are referring to **physical CPU cores** here. If your system has hyperthreading enabled, then 1 vCPU = 1 hyperthread = 1/2 physical CPU core, so you need `2 x (2 + N)` minimum vCPUs.
445
446 ### Data Parallel and Multi-API Server Deployments
447
448 When using data parallelism or multiple API servers, the CPU requirements increase:
449
450 ```console
451 Minimum physical cores = A + DP + N + (1 if DP > 1 else 0)
452 ```
453
454 where `A` is the API server count (defaults to `DP`), `DP` is the data parallel size, and `N` is the total number of GPUs. For example, with `DP=4, TP=2` on 8 GPUs:
455
456 ```console
457 4 API servers + 4 engine cores + 8 GPU workers + 1 DP coordinator = 17 processes
458 ```
459
460 ### Performance Impact
461
462 CPU underprovisioning particularly impacts:
463
464 - **Input processing throughput** -- tokenization, chat template rendering, and multi-modal data loading all run on CPU
465 - **Scheduling latency** -- the engine core scheduler runs on CPU and directly affects how quickly new tokens are dispatched to the GPU workers
466 - **Output processing** -- detokenization, networking, and especially streaming token responses use CPU cycles
467
468 If you observe that GPU utilization is lower than expected, CPU contention may be the bottleneck. Increasing the number of available CPU cores and even the clock speed can significantly improve end-to-end performance.
469
470 ## Attention Backend Selection
471
472 vLLM supports multiple attention backends optimized for different hardware and use cases. The backend is automatically selected based on your GPU architecture, model type, and configuration, but you can also manually specify one for optimal performance.
473
474 For detailed information on available backends, their feature support, and how to configure them, see the [Attention Backend Feature Support](../design/attention_backends.md) documentation.