@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 244 lines 16.8 KB
Rendered Raw
1 # Derenderer APIs
2
3 The derenderer API is the post processing counterpart to the [Renderer APIs](renderer.md). Where `/render` turns a request into token ID (preprocessing), `/derender` turns generated token IDs back into a fully formed OpenAI compatible response (detokenization, reasoning parsing, tool call parsing), all without a GPU.
4
5 This closes the loop for a token-in / token-out engine in disaggregated serving:
6
7 - **GPU less post processing**: Detokenization, reasoning parsing, and tool call parsing run on the same GPU less frontend that hosts `/render`
8 - **Parser parity**: The derenderer reuses vLLM's tool and reasoning parsers, so a disaggregated deployment produces the same `content`/`reasoning`/ `tool_calls` split as a standard `vllm serve` server
9 - **Streaming and one-shot parsing**: Non-streaming calls send a complete `GenerateResponse` with all token IDs present and get one-shot parsing. Both endpoints also accept `stream: true`, taking one `GenerateStreamResponse` delta plus a client carried `stream_state` and returning `{chunk, stream_state}`. The chat endpoint's streaming path supports reasoning and tool call parsing while emitting the same `reasoning`/`content`/`tool_calls` deltas the generate streaming path would for the same token IDs (see [Streaming](#streaming))
10
11 Both endpoints are hosted by the GPU less rendering server started with [`vllm launch render`](../../cli/launch/render.md), alongside the `/render`
12 endpoints.
13
14 ## Pipeline
15
16 ```text
17 render generate derender
18 request ───────────────▶ token_ids ─────────▶ token_ids ──────────▶ response
19 (chat / (GPU less) (token-in / (GPU less) (OpenAI
20 completion) │ token-out engine) ▲ compatible)
21 └─────────────── request + prompt_tokens ──┘
22 ```
23
24 The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. The chat streaming path additionally needs `prompt_token_ids` (see [Streaming cost](#streaming-cost)) when a parser is configured.
25
26 All derender endpoints also accept an optional `prompt_token_ids` (the `token_ids` of the `GenerateRequest` returned by `/render`). It's used to seed detokenization from the end of the prompt, the same way the engine does. Without it, the first output token is decoded as if it started a new sequence. On SentencePiece tokenizers (Llama-2, Phi-3) that drops its leading space compared to `/v1/chat/completions` and `/v1/completions`. If it's omitted, derender falls back to the generate response's `prompt_token_ids` (set when the `GenerateRequest` has `return_token_ids`), then to decoding without prompt context.
27
28 Seeding only reads the last few IDs (7 today), so a suffix of the prompt is enough and a long prompt doesn't need to travel back to derender. The exception is streaming chat with a tool or reasoning parser configured. There the parser also reads `prompt_token_ids` to decide whether the prompt left reasoning open and it can look back to any point in the prompt. In this case, send the full list (see [Streaming cost](#streaming-cost)).
29
30 ## API Reference
31
32 - Chat Completions Derender API (`/v1/chat/completions/derender`)
33 - Post process a single `GenerateResponse` into a `ChatCompletionResponse`
34 - With `stream: true`, post process one `GenerateStreamResponse` chunk into a `ChatCompletionStreamResponse` chunk
35 - Completions Derender API (`/v1/completions/derender`)
36 - Post process a list of `GenerateResponse` objects (one per prompt) into a `CompletionResponse`
37 - With `stream: true`, post process one `GenerateStreamResponse` chunk into a `CompletionStreamResponse` chunk
38
39 ## Request format
40
41 Each request wraps the engine's `GenerateResponse`(s) together with the caller metadata needed to reconstruct the final response without a GPU.
42
43 `/v1/chat/completions/derender`:
44
45 ??? code
46
47 ```python
48 --8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-chat-request"
49 ```
50
51 `/v1/completions/derender`:
52
53 ??? code
54
55 ```python
56 --8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-completion-request"
57 ```
58
59 Streaming requests set `stream: true` and carry one generate chunk (`generate_chunk`) plus `stream_state` instead of a complete response.
60
61 `/v1/chat/completions/derender` with `stream: true`:
62
63 ??? code
64
65 ```python
66 --8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-chat-stream-request"
67 ```
68
69 `/v1/completions/derender` with `stream: true`:
70
71 ??? code
72
73 ```python
74 --8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-completion-stream-request"
75 ```
76
77 Both return `{"chunk": ..., "stream_state": ...}`. `chunk` is a `ChatCompletionStreamResponse` or `CompletionStreamResponse` and `stream_state` goes with the next call.
78
79 Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or parser runs.
80
81 ## Parser configuration
82
83 The derenderer builds its tool and reasoning parsers from its own server flags plus the `chat_request` sent with each call. It can't see how the request was rendered or served anywhere else, so for its output to match a `vllm serve` server:
84
85 - Start the render server with the same `--tool-call-parser`, `--reasoning-parser`, `--enable-auto-tool-choice`, `--chat-template` and `--default-chat-template-kwargs` you'd give `vllm serve` for this model. If `/render` and `/derender` run on different servers, give both the same values.
86 - Send the full `chat_request` that went to `/render`, not just `messages` and `tools`. Fields like `chat_template_kwargs`, `reasoning_effort`, `tool_choice` and `include_reasoning` change how the output is parsed.
87
88 A mismatch doesn't fail. The parser just splits `reasoning`, `content` and `tool_calls` differently from what `vllm serve` would return.
89
90 `/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. It only reads `skip_special_tokens` from `completion_request`.
91
92 ## Streaming
93
94 Streaming derender mirrors `/inference/v1/generate`. Set `stream: true` in the body on the same path and the endpoint takes one generate stream chunk instead of a complete response. It answers with JSON, not SSE. Each call turns one generate chunk into one derendered chunk and the client forwards that to its own caller.
95
96 The server keeps no state between calls. Everything the next call needs is in `stream_state` which the client carries:
97
98 - **Echo `stream_state` back.** Leave it out (or send `null`) on the first call, then send the `stream_state` from each response with the next request, unchanged. State that doesn't add up (e.g. `output_chunk_lens` that don't sum to the number of output tokens) is rejected with a `400`.
99 - **One choice per chunk.** `/inference/v1/generate` sends one choice per SSE event and each derender call accepts at most one. More than one is rejected with a `400`.
100 - **`n > 1` is N streams.** Generate interleaves chunks for different choices and tags each with its `index`. Keep one `stream_state` per index and send each chunk with the state for its index. Each index gets its own `role` delta on its first chunk.
101 - **Send every chunk through, including the last ones.** The finish reason usually arrives with the final tokens. A chunk with a `finish_reason` and no tokens is accepted too and flushes any buffered tool call arguments. With `stream_options: {"include_usage": true}`, generate ends with a usage only chunk (`choices: []`). Send it through derender like any other chunk to get the usage chunk for the response. `usage.prompt_tokens` is the request's `prompt_tokens` if set, otherwise the generate chunk's.
102 - **Send the same context on every call.** `chat_request` and `prompt_token_ids` aren't kept between calls, so they go with every chunk including the usage chunk. Both are required when a tool or reasoning parser is configured. `prompt_token_ids` is the `token_ids` of the `GenerateRequest` returned by `/render`.
103 - **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
104 - **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
105
106 Streaming derender resolves `logprobs` on the plain detokenization path, including `token_id:N` placeholders, same as the non-streaming endpoints (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
107
108 ## Streaming cost
109
110 Streaming derender threads a client carried `stream_state` across per-chunk calls instead of keeping session state on the server.
111
112 Plain detokenization (no parser configured) carries only a small, bounded incremental decode window in `stream_state` independent of generation length.
113
114 When a tool or reasoning parser is configured, parser internal state (buffered markup, reasoning/tool phase) can't be serialized. `stream_state` therefore instead carries the full `output_token_ids` seen so far plus `output_chunk_lens`, the token count of each chunk they arrived in. Each chunk rebuilds a fresh parser and replays that history through `parse_delta`, one call per original chunk, before processing the new tokens for real. For the parser path only, this means:
115
116 - **Transport**: `output_token_ids` and `output_chunk_lens` round-trip in full in both directions on every call. `output_chunk_lens` has one entry per chunk, which is one per token without speculative decoding. This means O(n) bytes per chunk, O(n²) bytes over a full generation. Bounded by `max_model_len`. `prompt_token_ids` is sent in full on every call too and it isn't trimmed as `output_token_ids` grows. This means that for most of a stream it dominates the per chunk payload. A 100k token prompt with 1k tokens of output means `prompt_token_ids` is ~99% of the request body on every chunk.
117 - **Compute**: replay is O(n) `parse_delta` calls per chunk (O(n²) per generation). `parse_delta` itself is O(n) for parsers that re-scan accumulated text (e.g. Hermes tool-call JSON, DeepSeek-R1 reasoning). The per-generation cost is O(n³) character work, not O(n²). This is a deliberately minimal first implementation with no caching layer.
118 - The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk. Send the full list and not a suffix because the last reasoning marker can be anywhere in the prompt.
119 - Replay runs off the event loop on the renderer's executor (`renderer_num_workers`, default `1`). Size it for the expected number of concurrent parser configured streams.
120
121 `output_token_ids` and `prompt_token_ids` are both bounded by `max_model_len` but callers streaming long reasoning traces through a parser configured model should expect materially more state transport and CPU cost than the plain detokenization path.
122
123 ## Streaming state and logprobs
124
125 Streaming derender is stateless on the server side: all mutable state lives in the client-carried `stream_state` (`DerenderStreamState`), passed back on every per-chunk call. Besides the bounded incremental detokenization window (`prev_tokens`, `prefix_offset`, `read_offset`), `role_sent`, and the parser path's replay fields (see [Streaming cost](#streaming-cost)), the state carries two fields for logprob handling:
126
127 - `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
128 - `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
129
130 When a streamed `GenerateResponseStreamChoice` carries `logprobs`, the `token_id:N` placeholders are resolved per chunk and the resolved logprobs are attached to the corresponding streamed choice — for chat as `ChatCompletionLogProbs`, for completions converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
131
132 ## Example
133
134 The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
135
136 Launch the two servers first:
137
138 ```bash
139 vllm launch render meta-llama/Llama-3.2-1B-Instruct --port 8100
140 vllm serve meta-llama/Llama-3.2-1B-Instruct --tokens-only --port 8200
141 ```
142
143 ```python
144 import httpx
145
146 MODEL = "meta-llama/Llama-3.2-1B-Instruct"
147 RENDER = "http://localhost:8100" # vllm launch render ...
148 ENGINE = "http://localhost:8200" # token-in / token-out engine
149
150 chat_request = {
151 "model": MODEL,
152 "messages": [{"role": "user", "content": "What is 2+2?"}],
153 "max_tokens": 32,
154 }
155
156 with httpx.Client(timeout=60.0) as client:
157 # 1. Render: request -> token IDs (GPU less)
158 generate_request = client.post(
159 f"{RENDER}/v1/chat/completions/render", json=chat_request
160 ).json()
161 prompt_token_ids = generate_request["token_ids"]
162
163 # 2. Generate: token IDs -> token IDs (token-in / token-out engine)
164 generate_response = client.post(
165 f"{ENGINE}/inference/v1/generate", json=generate_request
166 ).json()
167
168 # 3. Derender: token IDs -> ChatCompletionResponse (GPU less)
169 response = client.post(
170 f"{RENDER}/v1/chat/completions/derender",
171 json={
172 "model": MODEL,
173 "generate_response": generate_response,
174 "prompt_tokens": len(prompt_token_ids),
175 "prompt_token_ids": prompt_token_ids,
176 "chat_request": chat_request,
177 },
178 ).json()
179
180 print(response["choices"][0]["message"]["content"])
181 ```
182
183 Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. `chat_request` can only be omitted for a model with no tool or reasoning parser configured. A parser configured model rejects a missing `chat_request` with a 400 rather than silently falling back to plain detokenization.
184
185 ### Streaming example
186
187 The same round trip, streamed. `/render` keeps `stream` and `stream_options` on the `GenerateRequest` it returns, so the generate call streams too. Each generate chunk goes through derender with the `stream_state` from the previous call (see [Streaming](#streaming)).
188
189 ```python
190 import json
191
192 import httpx
193
194 MODEL = "meta-llama/Llama-3.2-1B-Instruct"
195 RENDER = "http://localhost:8100" # vllm launch render ...
196 ENGINE = "http://localhost:8200" # token-in / token-out engine
197
198 chat_request = {
199 "model": MODEL,
200 "messages": [{"role": "user", "content": "What is 2+2?"}],
201 "max_tokens": 32,
202 "stream": True,
203 "stream_options": {"include_usage": True},
204 }
205
206 with httpx.Client(timeout=60.0) as client:
207 # 1. Render: request -> token IDs (GPU less)
208 generate_request = client.post(
209 f"{RENDER}/v1/chat/completions/render", json=chat_request
210 ).json()
211 prompt_token_ids = generate_request["token_ids"]
212
213 # 2. Generate: stream token IDs (token-in / token-out engine)
214 stream_state = None # one per choice index when n > 1
215 with client.stream(
216 "POST", f"{ENGINE}/inference/v1/generate", json=generate_request
217 ) as generate_stream:
218 for line in generate_stream.iter_lines():
219 if not line.startswith("data: ") or line == "data: [DONE]":
220 continue
221
222 # 3. Derender: one generate chunk -> one chat completion chunk
223 derendered = client.post(
224 f"{RENDER}/v1/chat/completions/derender",
225 json={
226 "stream": True,
227 "model": MODEL,
228 "generate_chunk": json.loads(line[len("data: ") :]),
229 "stream_state": stream_state,
230 "prompt_tokens": len(prompt_token_ids),
231 "prompt_token_ids": prompt_token_ids,
232 "chat_request": chat_request,
233 },
234 ).json()
235 stream_state = derendered["stream_state"]
236
237 chunk = derendered["chunk"]
238 for choice in chunk["choices"]:
239 print(choice["delta"].get("content") or "", end="", flush=True)
240 if chunk.get("usage"):
241 print(f"\n{chunk['usage']}")
242 ```
243
244 With a tool or reasoning parser configured, `delta` also carries `reasoning` and `tool_calls`, the same deltas `/v1/chat/completions` streams for those tokens.