[Render] Add round trip parity test and docs for derender (#48617)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Martin Hickey committed
Jul 17, 2026 at 06:02 UTC
4d4e04f45221709079316599f01b33a5233deaf2
7 files changed
+397
-2
docs/serving/online_serving/README.md
+6
-2
@@ -137,8 +137,12 @@ For further details on renderer APIs, please refer to [this page](renderer.md).
137
138
### Derenderer APIs
139
140
-- `/v1/completions/derender` - Derenderer completion requests
141
-- `/v1/chat/completions/derender` - Derenderer chat completion requests
140
+For further details on derenderer APIs, please refer to [this page](derenderer.md).
141
+
142
+- [Chat Completions Derender API](derenderer.md) (`/v1/chat/completions/derender`)
143
+ - Derender chat completion requests
144
+- [Completions Derender API](derenderer.md) (`/v1/completions/derender`)
145
+ - Derender completion requests
146
147
## Tokenize APIs
148
docs/serving/online_serving/derenderer.md
new
+98
@@ -0,0 +1,98 @@
1
+# Derenderer APIs
2
+
3
+The derenderer API is the post processing counterpart to the [Renderer APIs](renderer.md). Where `/render` turns a request into token ID (preprocessing), `/derender` turns generated token IDs back into a fully formed OpenAI compatible response (detokenization, reasoning parsing, tool call parsing), all without a GPU.
4
+
5
+This closes the loop for a token-in / token-out engine in disaggregated serving:
6
+
7
+- **GPU less post processing**: Detokenization, reasoning parsing, and tool call parsing run on the same GPU less frontend that hosts `/render`
8
+- **Parser parity**: The derenderer reuses vLLM's tool and reasoning parsers, so a disaggregated deployment produces the same `content`/`reasoning`/ `tool_calls` split as a standard `vllm serve` server
9
+- **Non-streaming**: The endpoints expect a complete `GenerateResponse` with all token IDs present and perform one-shot parsing. Streaming derender would require a separate endpoint design and is not currently supported but is in the pipeline
10
+
11
+Both endpoints are hosted by the GPU less rendering server started with [`vllm launch render`](../../cli/launch/render.md), alongside the `/render`
12
+endpoints.
13
+
14
+## Pipeline
15
+
16
+```text
17
+ render generate derender
18
+ request ───────────────▶ token_ids ─────────▶ token_ids ──────────▶ response
19
+ (chat / (GPU less) (token-in / (GPU less) (OpenAI
20
+ completion) │ token-out engine) ▲ compatible)
21
+ └─────────────── request + prompt_tokens ──┘
22
+```
23
+
24
+The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need.
25
+
26
+## API Reference
27
+
28
+- Chat Completions Derender API (`/v1/chat/completions/derender`)
29
+ - Post process a single `GenerateResponse` into a `ChatCompletionResponse`
30
+- Completions Derender API (`/v1/completions/derender`)
31
+ - Post process a list of `GenerateResponse` objects (one per prompt) into a `CompletionResponse`
32
+
33
+## Request format
34
+
35
+Each request wraps the engine's `GenerateResponse`(s) together with the caller metadata needed to reconstruct the final response without a GPU.
36
+
37
+`/v1/chat/completions/derender`:
38
+
39
+??? code
40
+
41
+ ```python
42
+ --8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-chat-request"
43
+ ```
44
+
45
+`/v1/completions/derender`:
46
+
47
+??? code
48
+
49
+ ```python
50
+ --8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-completion-request"
51
+ ```
52
+
53
+Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or parser runs.
54
+
55
+## Example
56
+
57
+The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
58
+
59
+```python
60
+import httpx
61
+
62
+MODEL = "meta-llama/Llama-3.2-1B-Instruct"
63
+RENDER = "http://localhost:8100" # vllm launch render ...
64
+ENGINE = "http://localhost:8200" # token-in / token-out engine
65
+
66
+chat_request = {
67
+ "model": MODEL,
68
+ "messages": [{"role": "user", "content": "What is 2+2?"}],
69
+ "max_tokens": 32,
70
+}
71
+
72
+with httpx.Client(timeout=60.0) as client:
73
+ # 1. Render: request -> token IDs (GPU less)
74
+ generate_request = client.post(
75
+ f"{RENDER}/v1/chat/completions/render", json=chat_request
76
+ ).json()
77
+ prompt_tokens = len(generate_request["token_ids"])
78
+
79
+ # 2. Generate: token IDs -> token IDs (token-in / token-out engine)
80
+ generate_response = client.post(
81
+ f"{ENGINE}/inference/v1/generate", json=generate_request
82
+ ).json()
83
+
84
+ # 3. Derender: token IDs -> ChatCompletionResponse (GPU less)
85
+ response = client.post(
86
+ f"{RENDER}/v1/chat/completions/derender",
87
+ json={
88
+ "model": MODEL,
89
+ "generate_response": generate_response,
90
+ "prompt_tokens": prompt_tokens,
91
+ "chat_request": chat_request,
92
+ },
93
+ ).json()
94
+
95
+print(response["choices"][0]["message"]["content"])
96
+```
97
+
98
+Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. Omit `chat_request` for plain detokenization only.
docs/serving/online_serving/renderer.md
+2
@@ -12,3 +12,5 @@ Our renderer API is designed to disaggregate the render phase(preprocessing) and
12
- Render completion requests
13
- [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`)
14
- Render chat completions
15
+
16
+For the post processing counterpart that turns generated token IDs back into OpenAI compatible responses, see the [Derenderer APIs](derenderer.md).
docs/usage/security.md
+2
@@ -155,8 +155,10 @@ When `--api-key` is configured, the following `/v1` endpoints require Bearer tok
155
- `/v1/chat/completions` - Chat completions
156
- `/v1/chat/completions/batch` - Batch chat completions
157
- `/v1/chat/completions/render` - Render chat completion requests
158
+- `/v1/chat/completions/derender` - Derender chat completion requests
159
- `/v1/completions` - Text completions
160
- `/v1/completions/render` - Render completion requests
161
+- `/v1/completions/derender` - Derender completion requests
162
- `/v1/embeddings` - Generate embeddings
163
- `/v1/audio/transcriptions` - Audio transcription
164
- `/v1/audio/translations` - Audio translation
tests/entrypoints/scale_out/derender/test_derender_parity.py
new
+268
@@ -0,0 +1,268 @@
1
+# SPDX-License-Identifier: Apache-2.0
2
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+
4
+"""Round trip parity CI: render -> generate -> derender -> render == render.
5
+
6
+Pins the coupled parsing path (``/v1/chat/completions`` on a normal GPU
7
+server which parses generated tokens incrementally as they stream out)
8
+against the disaggregated path (``OnlineDerenderer.derender_chat`` via
9
+``/v1/chat/completions/derender`` which parses the same tokens all at once,
10
+out of process). A standard ``vllm serve`` GPU server mounts both, so one
11
+real generation lets both parsers run on identical input.
12
+
13
+Both paths consume the same generated token IDs (extracted from the coupled
14
+response's ``token_ids`` via ``return_token_ids=True``), so generation
15
+nondeterminism is irrelevant. The only variable under test is whether the
16
+two parsing code paths agree. Parity is asserted unconditionally as only the
17
+stronger per case assertions (e.g. "a tool call was produced") are gated
18
+behind the marker actually having been emitted since a 1.5B model is not
19
+guaranteed to emit ``<think>`` / ``<tool_call>``.
20
+"""
21
+
22
+import json
23
+
24
+import httpx
25
+import pytest
26
+import pytest_asyncio
27
+
28
+from tests.utils import RemoteOpenAIServer
29
+
30
+MODEL = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"
31
+ARGS = [
32
+ "--enable-auto-tool-choice",
33
+ "--tool-call-parser",
34
+ "hermes",
35
+ "--reasoning-parser",
36
+ "deepseek_r1",
37
+]
38
+
39
+TOOLS = [
40
+ {
41
+ "type": "function",
42
+ "function": {
43
+ "name": "get_weather",
44
+ "description": "Get weather for a city",
45
+ "parameters": {
46
+ "type": "object",
47
+ "properties": {"city": {"type": "string"}},
48
+ },
49
+ },
50
+ }
51
+]
52
+FORCE_WEATHER_TOOL = {"type": "function", "function": {"name": "get_weather"}}
53
+
54
+
55
+@pytest.fixture(scope="module")
56
+def server():
57
+ with RemoteOpenAIServer(MODEL, ARGS) as remote_server:
58
+ yield remote_server
59
+
60
+
61
+@pytest_asyncio.fixture
62
+async def client(server):
63
+ async with httpx.AsyncClient(
64
+ base_url=server.url_for(""), timeout=60.0
65
+ ) as http_client:
66
+ yield http_client
67
+
68
+
69
+# ---------------------------------------------------------------------------
70
+# Helpers
71
+# ---------------------------------------------------------------------------
72
+
73
+
74
+async def _coupled(client: httpx.AsyncClient, messages: list[dict], **extra) -> dict:
75
+ resp = await client.post(
76
+ "/v1/chat/completions",
77
+ json={
78
+ "model": MODEL,
79
+ "messages": messages,
80
+ "temperature": 0,
81
+ "max_tokens": 128,
82
+ "return_token_ids": True,
83
+ **extra,
84
+ },
85
+ )
86
+ assert resp.status_code == 200, resp.text
87
+ return resp.json()
88
+
89
+
90
+async def _disagg(
91
+ client: httpx.AsyncClient,
92
+ output_ids: list[int],
93
+ prompt_tokens: int,
94
+ finish_reason: str,
95
+ chat_request: dict,
96
+ logprobs: dict | None = None,
97
+) -> dict:
98
+ resp = await client.post(
99
+ "/v1/chat/completions/derender",
100
+ json={
101
+ "model": MODEL,
102
+ "generate_response": {
103
+ "request_id": "parity",
104
+ "choices": [
105
+ {
106
+ "index": 0,
107
+ "token_ids": output_ids,
108
+ "finish_reason": finish_reason,
109
+ "logprobs": logprobs,
110
+ }
111
+ ],
112
+ },
113
+ "prompt_tokens": prompt_tokens,
114
+ "chat_request": chat_request,
115
+ },
116
+ )
117
+ assert resp.status_code == 200, resp.text
118
+ return resp.json()
119
+
120
+
121
+def _tool_sig(response_choice: dict) -> list[tuple[str, dict]]:
122
+ """[(name, json normalized args)] so key ordering / whitespace don't
123
+ cause false negatives."""
124
+ return [
125
+ (tc["function"]["name"], json.loads(tc["function"]["arguments"]))
126
+ for tc in (response_choice["message"].get("tool_calls") or [])
127
+ ]
128
+
129
+
130
+def _assert_parity(coupled: dict, disagg: dict) -> None:
131
+ """Both paths saw the same tokens, so they must agree unconditionally."""
132
+ c, d = coupled["choices"][0], disagg["choices"][0]
133
+ assert d["message"]["content"] == c["message"]["content"]
134
+ assert d["message"].get("reasoning") == c["message"].get("reasoning")
135
+ assert _tool_sig(d) == _tool_sig(c)
136
+ assert d["finish_reason"] == c["finish_reason"]
137
+ assert disagg["usage"]["prompt_tokens"] == coupled["usage"]["prompt_tokens"]
138
+ assert disagg["usage"]["completion_tokens"] == len(c["token_ids"])
139
+
140
+
141
+async def _run_parity_case(
142
+ client: httpx.AsyncClient, messages: list[dict], **extra
143
+) -> tuple[dict, dict]:
144
+ """Run the coupled request then feed its generated tokens into the
145
+ disaggregated derender endpoint. Returns (coupled, disagg)."""
146
+ coupled = await _coupled(client, messages, **extra)
147
+ ch = coupled["choices"][0]
148
+ chat_request = {"model": MODEL, "messages": messages, **extra}
149
+ disagg = await _disagg(
150
+ client,
151
+ ch["token_ids"],
152
+ coupled["usage"]["prompt_tokens"],
153
+ ch["finish_reason"],
154
+ chat_request,
155
+ )
156
+ return coupled, disagg
157
+
158
+
159
+# ---------------------------------------------------------------------------
160
+# Parity cases
161
+# ---------------------------------------------------------------------------
162
+
163
+
164
+@pytest.mark.asyncio
165
+async def test_parity_plain(client):
166
+ """Plain detokenization parity. No reasoning/tool markers involved."""
167
+ messages = [
168
+ {"role": "user", "content": "What is 2+2? Answer in one short sentence."}
169
+ ]
170
+ coupled, disagg = await _run_parity_case(client, messages)
171
+ _assert_parity(coupled, disagg)
172
+
173
+
174
+@pytest.mark.asyncio
175
+async def test_parity_reasoning(client):
176
+ """Reasoning/content split parity for <think>...</think> outputs."""
177
+ messages = [{"role": "user", "content": "What is 17 times 23? Think it through."}]
178
+ coupled, disagg = await _run_parity_case(
179
+ client, messages, include_reasoning=True, max_tokens=256
180
+ )
181
+ _assert_parity(coupled, disagg)
182
+
183
+ if not coupled["choices"][0]["message"].get("reasoning"):
184
+ pytest.skip("Model did not emit a <think> block")
185
+ assert disagg["choices"][0]["message"]["reasoning"]
186
+
187
+
188
+@pytest.mark.asyncio
189
+async def test_parity_tool_call(client):
190
+ """Tool call name+args parity."""
191
+ messages = [{"role": "user", "content": "What's the weather in Paris?"}]
192
+ coupled, disagg = await _run_parity_case(
193
+ client, messages, tools=TOOLS, tool_choice=FORCE_WEATHER_TOOL, max_tokens=1024
194
+ )
195
+ _assert_parity(coupled, disagg)
196
+
197
+ if not _tool_sig(coupled["choices"][0]):
198
+ pytest.skip("Model did not emit a tool call")
199
+ assert _tool_sig(disagg["choices"][0])
200
+
201
+
202
+@pytest.mark.asyncio
203
+async def test_parity_reasoning_and_tool_call(client):
204
+ """Combined reasoning + tool call parity means the highest drift risk
205
+ since it exercises both parser branches on the same output."""
206
+ messages = [{"role": "user", "content": "What's the weather in Paris?"}]
207
+ coupled, disagg = await _run_parity_case(
208
+ client,
209
+ messages,
210
+ tools=TOOLS,
211
+ tool_choice=FORCE_WEATHER_TOOL,
212
+ include_reasoning=True,
213
+ max_tokens=1024,
214
+ )
215
+ _assert_parity(coupled, disagg)
216
+
217
+ c_msg = coupled["choices"][0]["message"]
218
+ if not (c_msg.get("reasoning") and _tool_sig(coupled["choices"][0])):
219
+ pytest.skip("Model did not emit both a <think> block and a tool call")
220
+ d_msg = disagg["choices"][0]["message"]
221
+ assert d_msg["reasoning"]
222
+ assert _tool_sig(disagg["choices"][0])
223
+
224
+
225
+@pytest.mark.asyncio
226
+async def test_parity_logprobs(client):
227
+ """token_id:N resolution parity vs. the coupled server's real strings.
228
+
229
+ A real disaggregated worker only has token IDs so it emits logprobs
230
+ with ``token_id:N`` placeholders (``return_tokens_as_token_ids=True``
231
+ reproduces that shape here). ``/derender`` must resolve those
232
+ placeholders to the same token strings/bytes the coupled server
233
+ resolves them to directly.
234
+ """
235
+ messages = [{"role": "user", "content": "What is 2+2?"}]
236
+ extra = {"logprobs": True, "top_logprobs": 3}
237
+
238
+ # What a real GPU less worker would hand to /derender is token IDs plus
239
+ # logprobs still in token_id:N placeholder form
240
+ placeholder = await _coupled(
241
+ client, messages, return_tokens_as_token_ids=True, **extra
242
+ )
243
+ ch = placeholder["choices"][0]
244
+ chat_request = {"model": MODEL, "messages": messages, **extra}
245
+ disagg = await _disagg(
246
+ client,
247
+ ch["token_ids"],
248
+ placeholder["usage"]["prompt_tokens"],
249
+ ch["finish_reason"],
250
+ chat_request,
251
+ logprobs=ch["logprobs"],
252
+ )
253
+
254
+ # The coupled server resolving the same greedy generation
255
+ # to real token strings itself.
256
+ resolved = await _coupled(client, messages, **extra)
257
+ assert resolved["choices"][0]["token_ids"] == ch["token_ids"], (
258
+ "greedy (temperature=0) generation was expected to be deterministic "
259
+ "across the two coupled calls used to build this test's fixtures"
260
+ )
261
+ _assert_parity(resolved, disagg)
262
+
263
+ r_content = resolved["choices"][0]["logprobs"]["content"]
264
+ d_content = disagg["choices"][0]["logprobs"]["content"]
265
+ assert len(d_content) == len(r_content)
266
+ for d_entry, r_entry in zip(d_content, r_content):
267
+ assert d_entry["token"] == r_entry["token"]
268
+ assert d_entry["bytes"] == r_entry["bytes"]
vllm/entrypoints/scale_out/token_in_token_out/protocol.py
+13
@@ -266,8 +266,13 @@ class DerenderChatRequest(BaseModel):
266
and ``parser.parse_delta()`` instead of ``parser.parse()``.
267
"""
268
269
+ # --8<-- [start:derender-chat-request]
270
model: str
271
+ """Served model name."""
272
+
273
generate_response: GenerateResponse
274
+ """The complete token-in / token-out engine response to derender."""
275
+
276
prompt_tokens: int | None = None
277
"""Prompt token count for usage; defaults to 0 if omitted.
278
@@ -282,6 +287,7 @@ class DerenderChatRequest(BaseModel):
287
request context they expect (request.tools, request.tool_choice,
288
request._grammar_from_tool_parser, etc.).
289
"""
290
+ # --8<-- [end:derender-chat-request]
291
292
293
class DerenderCompletionRequest(BaseModel):
@@ -292,8 +298,14 @@ class DerenderCompletionRequest(BaseModel):
298
returned by /v1/completions/render.
299
"""
300
301
+ # --8<-- [start:derender-completion-request]
302
model: str
303
+ """Served model name."""
304
+
305
generate_responses: list[GenerateResponse]
306
+ """One response per prompt, parallel to the list[GenerateRequest]
307
+ returned by /v1/completions/render."""
308
+
309
prompt_tokens: list[int] | None = None
310
"""One prompt token count per response; each defaults to 0 if omitted.
311
@@ -306,6 +318,7 @@ class DerenderCompletionRequest(BaseModel):
318
Mirrors chat_request on DerenderChatRequest. Required by the parsing
319
so parsers receive the full request context.
320
"""
321
+ # --8<-- [end:derender-completion-request]
322
323
@model_validator(mode="after")
324
def _validate_prompt_tokens_length(self) -> "DerenderCompletionRequest":
vllm/renderers/online_derenderer.py
+8
@@ -7,6 +7,7 @@ from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption
7
from vllm.entrypoints.generate.base.serving import resolve_token_id_placeholder
8
from vllm.entrypoints.openai.chat_completion.protocol import (
9
ChatCompletionLogProbs,
10
+ ChatCompletionNamedToolChoiceParam,
11
ChatCompletionRequest,
12
ChatCompletionResponseChoice,
13
ChatMessage,
@@ -136,6 +137,13 @@ class OnlineDerenderer:
137
else []
138
)
139
140
+ is_named_tool_choice = (
141
+ type(chat_request.tool_choice) is ChatCompletionNamedToolChoiceParam
142
+ )
143
+ is_required_tool_choice = chat_request.tool_choice == "required"
144
+ if is_named_tool_choice or is_required_tool_choice:
145
+ content = content or ""
146
+
147
message = ChatMessage(
148
role="assistant",
149
reasoning=reasoning,