@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 54 lines 2.5 KB
Rendered Raw
1 # IndexCache
2
3 IndexCache reduces redundant top-k computation in DeepSeek-V3.2 (DSA) models by caching and reusing top-k indices across layers.
4
5 ## Background
6
7 DeepSeek-V3.2 uses a DeepSeek Sparse Attention (DSA) mechanism where top-k token selection is computed per layer. For deep models with many layers, this computation can be expensive. IndexCache allows skipping redundant top-k computations by reusing indices from previous layers.
8
9 See: [IndexCache Paper](https://arxiv.org/abs/2603.12201)
10
11 ## Usage
12
13 ### CLI
14
15 ```bash
16 vllm serve deepseek-ai/DeepSeek-V3.2 \
17 --hf-overrides '{"use_index_cache": true, "index_topk_freq": 4}' ...
18 ```
19
20 ### Configuration Reference
21
22 | Parameter | Type | Default | Description |
23 |----------------------|------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------|
24 | `use_index_cache` | bool | false | Enable IndexCache. Must be set to true to use this feature |
25 | `index_topk_freq` | int | 1 | Frequency (in layers) at which top-k is computed. 1 = compute on every layer (disabled), 4 = compute on 1/4 of layers |
26 | `index_topk_pattern` | str | null | Per-layer F/S pattern. Overrides index_topk_freq if set. Each character maps to one DSA layer: F = Full, S = Shared |
27
28 ### Configuration Examples
29
30 **Using `index_topk_freq`** (compute every N layers):
31
32 ```bash
33 vllm serve deepseek-ai/DeepSeek-V3.2 \
34 --hf-overrides '{"use_index_cache": true, "index_topk_freq": 4}' ...
35 ```
36
37 **Using `index_topk_pattern`** (explicit per-layer control):
38
39 ```bash
40 # custom pattern for 61 layers: F = compute, S = reuse
41 vllm serve deepseek-ai/DeepSeek-V3.2 \
42 --hf-overrides '{"use_index_cache": true, "index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSF"}'
43 ```
44
45 ## How It Works
46
47 1. When IndexCache is enabled, layers marked with `"F"` (Full) calculate and store top-k indices
48 2. Subsequent layers marked with `"S"` (Shared) receive the cached indices from the previous layer instead of recomputing
49 3. The cached indices are passed through the layer stack, reducing total computation
50
51 ## Requirements
52
53 - DeepSeek-V3.2 or compatible DSA model
54 - `use_index_cache: true` via `--hf-overrides`