| 1 | # Scale-out EC connector e2e tests |
| 2 | |
| 3 | End-to-end tests for the disaggregated multimodal (encoder-cache) flow over |
| 4 | the scale-out endpoints, in the bash-script style of |
| 5 | `tests/v1/kv_connector/nixl_integration`. |
| 6 | |
| 7 | ## Topology |
| 8 | |
| 9 | ```text |
| 10 | render (GPU-less) encode (EC producer) |
| 11 | /v1/chat/completions/render /inference/v1/generate (kwargs_data) |
| 12 | /v1/chat/completions/derender | |
| 13 | prefill (EC consumer) |
| 14 | /inference/v1/generate (mm_metadata + ec_transfer_params) |
| 15 | ``` |
| 16 | |
| 17 | - The **render** server (`vllm launch render`) preprocesses the chat request |
| 18 | into token ids and multimodal features (`kwargs_data` + `mm_metadata`). |
| 19 | - The **encode** instance (`ec_role: ec_producer`, encode-only) runs the |
| 20 | vision encoder on `kwargs_data` and publishes embeddings through the |
| 21 | `ECExampleConnector` shared storage. Its response carries |
| 22 | `ec_transfer_params`. |
| 23 | - The **prefill** instance (`ec_role: ec_consumer`) receives metadata-only |
| 24 | features (`mm_metadata`, no `kwargs_data`) plus `ec_transfer_params`, loads |
| 25 | the embeddings through the EC connector, and generates output token ids. |
| 26 | - The **derender** endpoint turns the output token ids back into a chat |
| 27 | completion response. |
| 28 | |
| 29 | ## Running |
| 30 | |
| 31 | ```bash |
| 32 | bash tests/entrypoints/scale_out/ec_integration/run_scale_out_ec_e2e_test.sh |
| 33 | ``` |
| 34 | |
| 35 | The script first collects baseline outputs from a single `vllm serve` |
| 36 | instance, then runs the scale-out topology and compares the derendered |
| 37 | outputs for exact equality. It also checks that metadata-only features are |
| 38 | rejected when `ec_transfer_params` is missing. |
| 39 | |
| 40 | Defaults: `Qwen/Qwen3-VL-2B-Instruct`, encode on GPU 0, prefill on GPU 1. |
| 41 | Override via environment variables (`MODEL`, `GPU_E`, `GPU_PD`, |
| 42 | `EC_SHARED_STORAGE_PATH`, `PYTHON=.venv/bin/python`, ...); see the header of |
| 43 | the script for the full list. |