| 1 | # Renderer APIs |
| 2 | |
| 3 | Our renderer API is designed to disaggregate the render phase(preprocessing) and enable a token-in / token-out API server. |
| 4 | |
| 5 | - GPU-less deployment of frontend: Allow preprocessing (tokenization, MM input processing) and postprocessing (detokenization, tool call parsing, reasoning parsing) to run without GPU. |
| 6 | - Disaggregated tokenization: Support use cases such as llm-d, Dynamo, and custom frontends that need to leverage vLLM's preprocessing logic without running the full inference engine. |
| 7 | - Tokens-in / tokens-out engine: Make the engine a pure token-in / token-out service, decoupled from request preprocessing. |
| 8 | |
| 9 | The dedicated `vllm launch render` server always exposes the `/render` and |
| 10 | `/derender` endpoints. |
| 11 | |
| 12 | Scale-out endpoints, including `/render`, `/derender`, and |
| 13 | `/inference/v1/generate`, are disabled by default on a standard inference |
| 14 | server. To expose them with `vllm serve`, opt in explicitly: |
| 15 | |
| 16 | ```bash |
| 17 | vllm serve <model> --enable-scale-out |
| 18 | ``` |
| 19 | |
| 20 | ## API Reference |
| 21 | |
| 22 | - [Completions Render API](renderer.md) (`/v1/completions/render`) |
| 23 | - Render completion requests |
| 24 | - [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`) |
| 25 | - Render chat completions |
| 26 | - [Responses Render API](renderer.md) (`/v1/responses/render`) |
| 27 | - Render a self-contained Responses request |
| 28 | |
| 29 | ## Get Responses prompt token IDs |
| 30 | |
| 31 | Use `/v1/responses/render` to get prompt token IDs before choosing a model |
| 32 | replica. Rendering applies prompt construction and preprocessing without running |
| 33 | inference. |
| 34 | |
| 35 | The Responses render endpoint uses the same prompt construction as |
| 36 | `/v1/responses` and returns one token-in `GenerateRequest`. It is stateless: |
| 37 | inline history is supported, but `previous_response_id` is not. Callers must |
| 38 | resolve stored response state and include the resulting history in the request |
| 39 | before rendering. |
| 40 | |
| 41 | Configure the renderer and generation workers with the same model, tokenizer, |
| 42 | chat template, and preprocessing options. Render the full request, including |
| 43 | instructions, history, tools, and any template or truncation options, so the |
| 44 | returned IDs reflect the prompt the model will receive. |
| 45 | |
| 46 | For multimodal requests, the `GenerateRequest` contains the model-processed |
| 47 | multimodal payload, which can be substantially larger than the source image or |
| 48 | video. The caller must forward that payload unchanged to the generation service |
| 49 | and provision transport limits and memory accordingly. |
| 50 | |
| 51 | For example, start a standard inference server with scale-out endpoints enabled: |
| 52 | |
| 53 | ```bash |
| 54 | vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-scale-out |
| 55 | ``` |
| 56 | |
| 57 | Send a Responses request and use `jq` to extract the `token_ids` field: |
| 58 | |
| 59 | ```bash |
| 60 | curl --fail --silent --show-error http://localhost:8000/v1/responses/render \ |
| 61 | -H "Content-Type: application/json" \ |
| 62 | -d '{ |
| 63 | "model": "meta-llama/Llama-3.1-8B-Instruct", |
| 64 | "input": "Explain prefix caching in one sentence.", |
| 65 | "max_output_tokens": 32 |
| 66 | }' | jq '.token_ids' |
| 67 | ``` |
| 68 | |
| 69 | To count the rendered prompt tokens for this text request, replace the `jq` |
| 70 | filter with `'.token_ids | length'`. The endpoint returns the full |
| 71 | `GenerateRequest`; `jq` filters the response on the client. Keep the complete |
| 72 | response if you will forward it to `/inference/v1/generate`, including any |
| 73 | multimodal features. |
| 74 | |
| 75 | If the server has `--api-key` or `VLLM_API_KEY` configured, add |
| 76 | `-H "Authorization: Bearer <api-key>"` to the request. See |
| 77 | [API key authentication limitations](../../usage/security.md#api-key-authentication-limitations) |
| 78 | for the existing authentication boundaries. |
| 79 | |
| 80 | For the post processing counterpart that turns generated token IDs back into OpenAI compatible responses, see the [Derenderer APIs](derenderer.md). |
| 81 | |
| 82 | ## Multimodal Render Features |
| 83 | |
| 84 | Multimodal render responses include a `features` object with per-modality |
| 85 | hashes, placeholder ranges, and serialized processor data. When the model |
| 86 | exposes placeholder-metadata or `keep_on_cpu` fields (for example |
| 87 | `image_grid_thw`), the response also includes `mm_metadata`. Each |
| 88 | `mm_metadata` entry is a base64-encoded `MultiModalKwargsItem` containing |
| 89 | only those fields, not encoder inputs such as `pixel_values`. |
| 90 | |
| 91 | The arrays in `mm_hashes`, `mm_placeholders`, `kwargs_data`, and |
| 92 | `mm_metadata` use the same per-modality item order. Downstream workers |
| 93 | should split those fields: |
| 94 | |
| 95 | - Encode requests keep `kwargs_data`. |
| 96 | - Prefill requests may omit `kwargs_data` and send `mm_metadata` only when |
| 97 | `ec_transfer_params` is also set, so embeddings are loaded by the EC |
| 98 | connector. Omitting `kwargs_data` without `ec_transfer_params` is rejected. |
| 99 | - Legacy clients that ignore `mm_metadata` and keep sending `kwargs_data` |
| 100 | continue to work. |
| 101 | |
| 102 | ## Example |
| 103 | |
| 104 | The example below shows how a disaggregated encode / prefill coordinator can |
| 105 | split a multimodal render response. The render step returns both |
| 106 | `kwargs_data` (encoder tensors plus metadata) and `mm_metadata` (metadata |
| 107 | only). Encode keeps the full payload; prefill drops `kwargs_data` after the |
| 108 | EC connector has published embeddings. |
| 109 | |
| 110 | ```python |
| 111 | import httpx |
| 112 | |
| 113 | MODEL = "Qwen/Qwen3-VL-2B-Instruct" |
| 114 | RENDER = "http://localhost:8100" # vllm launch render ... |
| 115 | ENCODE = "http://localhost:8200" # encode worker |
| 116 | PREFILL = "http://localhost:8300" # prefill worker |
| 117 | |
| 118 | chat_request = { |
| 119 | "model": MODEL, |
| 120 | "messages": [ |
| 121 | { |
| 122 | "role": "user", |
| 123 | "content": [ |
| 124 | {"type": "image_url", "image_url": {"url": "<data-url>"}}, |
| 125 | {"type": "text", "text": "Describe this image."}, |
| 126 | ], |
| 127 | } |
| 128 | ], |
| 129 | } |
| 130 | |
| 131 | with httpx.Client(timeout=120.0) as client: |
| 132 | # 1. Render: preprocess into token IDs and multimodal features. |
| 133 | render_response = client.post( |
| 134 | f"{RENDER}/v1/chat/completions/render", json=chat_request |
| 135 | ).json() |
| 136 | |
| 137 | features = render_response["features"] |
| 138 | # features["kwargs_data"]["image"][0] -> pixel_values + image_grid_thw |
| 139 | # features["mm_metadata"]["image"][0] -> image_grid_thw only |
| 140 | |
| 141 | # 2. Encode: send full kwargs_data so the encoder can run vision towers. |
| 142 | encode_response = client.post( |
| 143 | f"{ENCODE}/inference/v1/generate", |
| 144 | json={ |
| 145 | "token_ids": render_response["token_ids"], |
| 146 | "features": { |
| 147 | "mm_hashes": features["mm_hashes"], |
| 148 | "mm_placeholders": features["mm_placeholders"], |
| 149 | "kwargs_data": features["kwargs_data"], |
| 150 | }, |
| 151 | "sampling_params": {"max_tokens": 1}, |
| 152 | }, |
| 153 | ).json() |
| 154 | ec_transfer_params = encode_response["ec_transfer_params"] |
| 155 | |
| 156 | # 3. Prefill: omit kwargs_data; load embeddings via EC connector. |
| 157 | prefill_response = client.post( |
| 158 | f"{PREFILL}/inference/v1/generate", |
| 159 | json={ |
| 160 | "token_ids": render_response["token_ids"], |
| 161 | "features": { |
| 162 | "mm_hashes": features["mm_hashes"], |
| 163 | "mm_placeholders": features["mm_placeholders"], |
| 164 | "mm_metadata": features["mm_metadata"], |
| 165 | }, |
| 166 | "ec_transfer_params": ec_transfer_params, |
| 167 | "sampling_params": {"max_tokens": 64}, |
| 168 | }, |
| 169 | ).json() |
| 170 | |
| 171 | print(prefill_response["choices"][0]["token_ids"]) |
| 172 | ``` |
| 173 | |
| 174 | Single-process clients can keep passing the full render response to |
| 175 | `/inference/v1/generate` unchanged; `mm_metadata` is optional and ignored |
| 176 | when `kwargs_data` is present. |
| 177 | |
| 178 | ### Payload shape |
| 179 | |
| 180 | #### Render response |
| 181 | |
| 182 | `/v1/chat/completions/render` returns both `kwargs_data` and `mm_metadata`. |
| 183 | The arrays share the same per-modality item order. Base64 blobs are truncated |
| 184 | below for readability. |
| 185 | |
| 186 | ```json |
| 187 | { |
| 188 | "token_ids": [151644, 872], |
| 189 | "features": { |
| 190 | "mm_hashes": {"image": ["abc123..."]}, |
| 191 | "mm_placeholders": {"image": [{"offset": 0, "length": 256}]}, |
| 192 | "kwargs_data": { |
| 193 | "image": ["<base64 MultiModalKwargsItem: pixel_values + image_grid_thw>"] |
| 194 | }, |
| 195 | "mm_metadata": { |
| 196 | "image": ["<base64 MultiModalKwargsItem: image_grid_thw only>"] |
| 197 | } |
| 198 | } |
| 199 | } |
| 200 | ``` |
| 201 | |
| 202 | Forward `kwargs_data` to the encode worker. Keep `mm_metadata` for prefill. |
| 203 | |
| 204 | #### Prefill request |
| 205 | |
| 206 | Prefill omits `kwargs_data` and sends `mm_metadata` with |
| 207 | `ec_transfer_params` from the encode response: |
| 208 | |
| 209 | ```json |
| 210 | { |
| 211 | "token_ids": [151644, 872], |
| 212 | "features": { |
| 213 | "mm_hashes": {"image": ["abc123..."]}, |
| 214 | "mm_placeholders": {"image": [{"offset": 0, "length": 256}]}, |
| 215 | "mm_metadata": { |
| 216 | "image": ["<base64 MultiModalKwargsItem: image_grid_thw only>"] |
| 217 | } |
| 218 | }, |
| 219 | "ec_transfer_params": { |
| 220 | "ec_items": [{"mm_hash": "abc123...", "peer_host": "10.0.0.1"}] |
| 221 | }, |
| 222 | "sampling_params": {"max_tokens": 64} |
| 223 | } |
| 224 | ``` |