| 1 | # EPD Correctness Test |
| 2 | |
| 3 | This test verifies that EPD (Encoder-Prefill-Decode) disaggregation produces identical outputs to a baseline single instance. |
| 4 | |
| 5 | ## What It Tests |
| 6 | |
| 7 | - **Baseline**: Single vLLM instance serving a multimodal model |
| 8 | - **EPD (1E+1PD)**: 1 Encoder + 1 Prefill-Decode instance |
| 9 | - **Baseline (1P+1D)**: 1 Prefill + 1 Decode instance |
| 10 | - **EPD (1E+1P+1D)**: 1 Encoder + 1 Prefill + 1 Decode instance |
| 11 | |
| 12 | The test ensures that disaggregated encoding produces **identical** outputs to the baseline. |
| 13 | |
| 14 | Note that currently PD disaggregation set up may give slightly different results from a single instance. Therefore, we need the result from 1P+1D as the baseline for 1E+1P+1D |
| 15 | |
| 16 | Please refer to [Disaggregated Encoder Feature](../../../../docs/features/disagg_encoder.md) for the detailed explanation for the EPD features. |
| 17 | |
| 18 | ## Files |
| 19 | |
| 20 | - `run_epd_correctness_test.sh` - Main test script (starts all instances and runs tests) |
| 21 | - `test_epd_correctness.py` - Python test script (compares outputs) |
| 22 | |
| 23 | ## Usage |
| 24 | |
| 25 | ### Multimodal Prompts (Default) |
| 26 | |
| 27 | ```bash |
| 28 | cd vllm |
| 29 | ./tests/v1/ec_connector/integration/run_epd_correctness_test.sh |
| 30 | ``` |
| 31 | |
| 32 | This runs the test with actual multimodal (image) prompts. |
| 33 | |
| 34 | ### Text-Only Prompts |
| 35 | |
| 36 | ```bash |
| 37 | cd vllm |
| 38 | USE_MM_PROMPTS=0 ./tests/v1/ec_connector/integration/run_epd_correctness_test.sh |
| 39 | ``` |
| 40 | |
| 41 | This runs a quick test with text-only prompts to verify the setup works. |
| 42 | |
| 43 | ### Custom Configuration |
| 44 | |
| 45 | ```bash |
| 46 | # Use specific GPUs |
| 47 | GPU_E=0 GPU_PD=1 GPU_P=1 GPU_D=2 bash ./tests/v1/ec_connector/integration/run_epd_correctness_test.sh |
| 48 | |
| 49 | # Use specific ports |
| 50 | ENDPOINT_PORT=10001 bash ./tests/v1/ec_connector/integration/run_epd_correctness_test.sh |
| 51 | |
| 52 | # Use specific model |
| 53 | MODEL="Qwen/Qwen2.5-VL-3B-Instruct" bash ./tests/v1/ec_connector/integration/run_epd_correctness_test.sh |
| 54 | |
| 55 | # Use specific storage path |
| 56 | EC_SHARED_STORAGE_PATH="/tmp/my_ec_cache" bash ./tests/v1/ec_connector/integration/run_epd_correctness_test.sh |
| 57 | ``` |
| 58 | |
| 59 | ## How It Works |
| 60 | |
| 61 | ### NIXL EC failure isolation (1E + 1PD) |
| 62 | |
| 63 | ```bash |
| 64 | PATH="$PWD/.venv/bin:$PATH" CUDA_VISIBLE_DEVICES=0,1 \ |
| 65 | .venv/bin/python -m pytest \ |
| 66 | tests/v1/ec_connector/integration/test_nixl_failure.py -v -s |
| 67 | ``` |
| 68 | |
| 69 | Requires two CUDA GPUs and NIXL. `MODEL` overrides the default |
| 70 | `Qwen/Qwen2.5-VL-3B-Instruct`. The test starts real E/PD servers with CUDA |
| 71 | graphs enabled and uses the proxy's rewrite helper. Only the failing request's |
| 72 | control endpoint is replaced: a ZMQ peer returns `NACK_MISSING` and verifies |
| 73 | that the consumer actually requested the encoding. |
| 74 | |
| 75 | Metadata-only requests must return a request-level error; requests retaining |
| 76 | the image must succeed through local fallback. After each case, a fresh |
| 77 | metadata-only image request must succeed through the real NIXL transfer path. |
| 78 | No server restart, proxy retry, cache hit, or timeout race can mask the result. |
| 79 | This is a correctness test, not a throughput benchmark. |
| 80 | |
| 81 | ### Mooncake EC (1E + 1PD) |
| 82 | |
| 83 | ```bash |
| 84 | PYTHON_BIN="$PWD/.venv/bin/python" \ |
| 85 | bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh |
| 86 | ``` |
| 87 | |
| 88 | Requires two GPUs and Mooncake TransferEngine. TCP is the default transport; |
| 89 | no RDMA-capable network hardware is required. |
| 90 | The script sets `MC_FORCE_TCP=1` in TCP mode so Mooncake cannot auto-select RDMA. |
| 91 | The baseline runs first on GPU 0, followed by E on GPU 0 and PD on GPU 1. |
| 92 | `MOONCAKE_EC_PROTOCOL=rdma` selects RDMA instead; host-specific transport |
| 93 | environment variables should be set by the caller. |
| 94 | |
| 95 | Three black-box cases check fixed short answers and compare with the baseline: |
| 96 | |
| 97 | - One image: read the STOP sign. |
| 98 | - Two different images (including a local file): identify flowers and birds. |
| 99 | - The same image twice: read both STOP signs. |
| 100 | |
| 101 | By default, all three requests run concurrently for two rounds, exercising |
| 102 | shared hashes across requests and reuse after completion. Every response is |
| 103 | compared, not just the final round. Set `CONCURRENCY` and `REPEAT` to override. |
| 104 | Prefix caching is disabled and CUDA graphs remain enabled. This is a small |
| 105 | correctness suite, not a performance benchmark or failure-injection suite. |
| 106 | |
| 107 | `LOG_PATH` and `BASELINE_FILE` select the log directory and reference output. |
| 108 | `SKIP_BASELINE=1` reuses a reference generated with the same model and test |
| 109 | configuration. `USE_MM_PROMPTS=0` only checks text routing, not EC transfer. |
| 110 | |
| 111 | Buildkite runs this script as `mooncake-ec-tcp-e2e-2-gpus` on two L4 GPUs. |
| 112 | The job is defined in `.buildkite/test_areas/disaggregated_mooncake.yaml` and |
| 113 | selected for changes to EC, multimodal processing, scheduler/model-runner |
| 114 | integration, the proxy, or these tests (subject to normal PR CI approval). |
| 115 | It reuses the CI image's Python packages through a system-site-packages venv |
| 116 | and installs the CUDA-compatible Mooncake wheel. It uses loopback networking |
| 117 | and TCP only, without RDMA devices or peer-memory setup. |
| 118 | |
| 119 | The job fails on startup errors, request errors, or answer mismatches. On |
| 120 | failure, the script prints the last 100 lines of each server/proxy log to |
| 121 | the CI job log; full files remain under `LOG_PATH` while the container exists. |
| 122 | |
| 123 | ### Step 1: Baseline |
| 124 | |
| 125 | 1. Start single vLLM instance on GPU |
| 126 | 2. Run test prompts (multimodal or text-only) |
| 127 | 3. Save outputs to `.vllm_epd_baseline.txt` |
| 128 | 4. Shutdown instance |
| 129 | |
| 130 | ### Step 2: EPD (1E + 1PD) |
| 131 | |
| 132 | 1. Clear encoder cache storage |
| 133 | 2. Start instances and proxy |
| 134 | 3. Run same test prompts |
| 135 | 4. Assert outputs match baseline exactly |
| 136 | 5. Shutdown instances |
| 137 | |
| 138 | ### Step 3: EPD (1E + 1P + 1D) |
| 139 | |
| 140 | 1. Clear encoder cache storage |
| 141 | 2. Start instances and proxy |
| 142 | 3. Run same test prompts |
| 143 | 4. Assert outputs match baseline exactly |
| 144 | 5. Shutdown instances |
| 145 | |
| 146 | ## Test Scenarios |
| 147 | |
| 148 | ### Multimodal Prompts (--use_mm_prompts) |
| 149 | |
| 150 | Tests encoder cache transfer: |
| 151 | |
| 152 | - Single image query |
| 153 | - Multiple images in one request |
| 154 | - Mixed image and text |
| 155 | - Image with detailed questions |
| 156 | |
| 157 | ### Text-Only Prompts (default) |
| 158 | |
| 159 | Quick sanity check: |
| 160 | |
| 161 | - Simple text queries |
| 162 | - Text-only explanations |
| 163 | - Verifies proxy routing works |
| 164 | |
| 165 | ## Expected Behavior |
| 166 | |
| 167 | ### ✅ Test Passes When |
| 168 | |
| 169 | - All disagg outputs match baseline outputs exactly |
| 170 | - No errors during instance startup |
| 171 | - Encoder cache is properly saved and loaded |
| 172 | - Proxy correctly routes requests |
| 173 | |
| 174 | ### ❌ Test Fails When |
| 175 | |
| 176 | - Outputs differ between baseline and disagg |
| 177 | - Server startup fails |
| 178 | - Encoder cache not found (should fall back to local execution) |
| 179 | - Proxy routing errors |
| 180 | |
| 181 | ## Notes |
| 182 | |
| 183 | - The test uses deterministic generation (`temperature=0.0`, `seed=42`) |
| 184 | - Encoder cache should enable exact output reproduction |
| 185 | - Test cleans up all instances and cache files after completion |
| 186 | - Safe to run multiple times (idempotent) |
| 187 | - We setup the PD disagg part with NixlConnector. Please read details about EPD in `examples/disaggregated/disaggregated_encoder/README.md` |
| 188 | |
| 189 | ## Requirements |
| 190 | |
| 191 | - Multiple GPUs (3 for 1E+1P+1D, 2 for 1E+1PD, 1 for baseline) |
| 192 | - 1E+1P+1D is runnable with 2 GPU by assign E and P on the same GPU now. |
| 193 | - Multimodal model (e.g., Qwen2.5-VL-3B-Instruct) |
| 194 | - Internet access (for accessing vllm test images) |
| 195 | |
| 196 | ## Debugging |
| 197 | |
| 198 | ### Check Logs |
| 199 | |
| 200 | Logs and baseline output are saved in `/tmp/` by default. |
| 201 | Can be customized by changing the environment variables. |
| 202 | |
| 203 | ### Check Encoder Cache |
| 204 | |
| 205 | ```bash |
| 206 | # Verify cache files are created |
| 207 | ls -la $EC_SHARED_STORAGE_PATH/ |
| 208 | |
| 209 | # Should see directories with mm_hash names |
| 210 | # Each containing encoder_cache.safetensors |
| 211 | ``` |
| 212 | |
| 213 | ### Manual Testing |
| 214 | |
| 215 | Run individual components: |
| 216 | |
| 217 | ```bash |
| 218 | # Baseline only |
| 219 | python test_epd_correctness.py \ |
| 220 | --service_url http://localhost:8000 \ |
| 221 | --model_name Qwen/Qwen2.5-VL-3B-Instruct \ |
| 222 | --mode baseline \ |
| 223 | --baseline_file test_output.txt \ |
| 224 | --use_mm_prompts |
| 225 | |
| 226 | # Disagg only (requires baseline output file!) |
| 227 | python test_epd_correctness.py \ |
| 228 | --service_url http://localhost:8000 \ |
| 229 | --model_name Qwen/Qwen2.5-VL-3B-Instruct \ |
| 230 | --mode disagg \ |
| 231 | --baseline_file test_output.txt \ |
| 232 | --use_mm_prompts |
| 233 | ``` |