@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 109 lines 5.1 KB
Rendered Raw
1 # Cross-Encoder Output Reuse
2
3 Cross-Encoder output reuse lets an Encoder load image embeddings computed by
4 another Encoder from a shared Mooncake Store. Local cache misses check the Store
5 before encoding; newly computed outputs are published asynchronously. Both paths
6 use the existing `ECMooncakeConnector` P2P delivery to Prefill.
7
8 ## Requirements
9
10 - A working [disaggregated Encoder](disagg_encoder.md) deployment using
11 `ECMooncakeConnector`. See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
12 for the base E/PD configuration.
13 - Model Runner V2 (`VLLM_USE_V2_MODEL_RUNNER=1`), Encoder TP=1, and no dynamic LoRA.
14 - `--mm-processor-cache-gb` greater than zero to preserve content identifiers.
15 - Mooncake 0.3.12 or later with a running RAM Store. Disk offloading
16 (`enable_offload: true`) is not supported.
17
18 Shared reuse supports images with contiguous 2D FP16, BF16 or FP32 encoder outputs.
19 Other modalities use normal encoding; non-contiguous outputs are not published.
20
21 ## Usage
22
23 On each Encoder, add `cross_encoder_cache: true` to the existing producer
24 `--ec-transfer-config`, preserving its P2P settings:
25
26 ```json
27 {
28 "ec_connector": "ECMooncakeConnector",
29 "ec_role": "ec_producer",
30 "ec_connector_extra_config": {
31 "cross_encoder_cache": true
32 }
33 }
34 ```
35
36 Prefill keeps its existing `ECMooncakeConnector` consumer configuration and does
37 not need a Store client.
38
39 Create a JSON file pointing to an existing Store. For an independently managed
40 RAM pool, use `standalone-store` with `global_segment_size` set to zero:
41
42 ```json
43 {
44 "metadata_server": "http://STORE_HOST:2379/metadata",
45 "master_server_address": "STORE_HOST:50051",
46 "protocol": "tcp",
47 "device_name": "",
48 "mode": "standalone-store",
49 "global_segment_size": 0,
50 "local_buffer_size": "4GB"
51 }
52 ```
53
54 Replace the addresses with your Store endpoints and set the path on each Encoder
55 before starting vLLM:
56
57 ```bash
58 export MOONCAKE_CONFIG_PATH=/path/to/mooncake_config.json
59 ```
60
61 This connects to an existing Store; it does not start the storage services.
62 For Store setup and tenant configuration, see the
63 [Mooncake Store guide](mooncake_store_connector_usage.md#prerequisites).
64
65 ## Configuration
66
67 These options belong in the producer's `ec_connector_extra_config`:
68
69 | Option | Default | Description |
70 | --- | --- | --- |
71 | `cross_encoder_cache` | `false` | Enable shared output reuse. |
72 | `embedding_cache_prefix` | `""` | Namespace prefix for shared embeddings. |
73 | `embedding_model_identity` | Configured model path | Override the model field in Store keys. Matching multimodal identifiers are still required. |
74 | `store_max_pending_items` | `32` | Maximum pending publications per Encoder. Must be positive. |
75 | `store_max_pending_bytes` | `2147483648` (2 GiB) | Maximum retained tensor storage for pending publications per Encoder. Must be positive. |
76 | `store_read_buffer_bytes` | `134217728` (128 MiB) | Maximum reusable CPU staging buffer per Encoder; pinned for CUDA outputs. Must be positive. |
77
78 Encoders sharing outputs must use the same immutable weights, compatible
79 preprocessing and matching multimodal identifiers. Use the same configured model
80 path for automatic identifiers: `embedding_model_identity` alone does not enable
81 reuse across different paths. Change `embedding_cache_prefix` when replacing
82 weights in place or changing output-affecting settings. Caller-provided UUIDs
83 must consistently identify the same input; see [cached inputs](multimodal_inputs.md#cached-inputs).
84
85 ## Limitations
86
87 - Store reads are synchronous. Reuse skips Encoder computation, but not
88 preprocessing, P2P delivery or the scheduler's Encoder budget reservation.
89 - Hits are read in batches through one lazily allocated, registered CPU staging
90 buffer. Each chunk is copied to independent output tensors before the buffer
91 is reused. Reads are split by `store_read_buffer_bytes`; an individual object
92 larger than this capacity (including its 24-byte header) falls back to encoding.
93 Staging memory is additional to the Store's `local_buffer_size` and is released
94 on healthy shutdown. The first load includes allocation and registration costs.
95 - Publication is best-effort. A request can finish before its outputs reach the
96 Store, so concurrent cold requests may still encode the same image.
97 - Publication budgets count each retained backing storage once across pending
98 views. Its charge is released after the last view is safely reclaimed.
99 Exceeding a budget skips the write. Store client buffers and the P2P pool
100 consume additional memory.
101 - Cache misses and recoverable read errors fall back to encoding. Recoverable
102 publication errors skip the write. Unexpected native errors or unconfirmed
103 I/O completion or buffer release can fail the worker.
104 - Incompatible objects are rejected without replacement; they can trigger
105 repeated fallback until evicted.
106 - An independent Store can retain embeddings across Encoder restarts. Capacity,
107 eviction and resilience to Store failures depend on the Store deployment.
108 - The `protocol:v3` namespace isolates the compact embedding format from older
109 cache objects. Encoders using different format versions do not share hits.