@tduncan / Nvidia-Optimization-Stack-VLLM / commits / fcfacc3499

[Bugfix][Frontend] Seed derender detokenization from the prompt (#59046)

Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>

Martin Hickey committed Oct 2, 2026 at 04:17 UTC fcfacc3499b3f45978fc92a1b300f817cd17bb77

9 files changed +760 -163

docs/serving/online_serving/derenderer.md

+9 -3

@@ -23,6 +23,10 @@ endpoints.

24 The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. The chat streaming path additionally needs `prompt_token_ids` (see [Streaming cost](#streaming-cost)) when a parser is configured.

26 +All derender endpoints also accept an optional `prompt_token_ids` (the `token_ids` of the `GenerateRequest` returned by `/render`). It's used to seed detokenization from the end of the prompt, the same way the engine does. Without it, the first output token is decoded as if it started a new sequence. On SentencePiece tokenizers (Llama-2, Phi-3) that drops its leading space compared to `/v1/chat/completions` and `/v1/completions`. If it's omitted, derender falls back to the generate response's `prompt_token_ids` (set when the `GenerateRequest` has `return_token_ids`), then to decoding without prompt context.

27 +

28 +Seeding only reads the last few IDs (7 today), so a suffix of the prompt is enough and a long prompt doesn't need to travel back to derender. The exception is streaming chat with a tool or reasoning parser configured. There the parser also reads `prompt_token_ids` to decide whether the prompt left reasoning open and it can look back to any point in the prompt. In this case, send the full list (see [Streaming cost](#streaming-cost)).

29 +

30 ## API Reference

32 - Chat Completions Derender API (`/v1/chat/completions/derender`)

@@ -96,6 +100,7 @@ The server keeps no state between calls. Everything the next call needs is in `s

100 - **`n > 1` is N streams.** Generate interleaves chunks for different choices and tags each with its `index`. Keep one `stream_state` per index and send each chunk with the state for its index. Each index gets its own `role` delta on its first chunk.

101 - **Send every chunk through, including the last ones.** The finish reason usually arrives with the final tokens. A chunk with a `finish_reason` and no tokens is accepted too and flushes any buffered tool call arguments. With `stream_options: {"include_usage": true}`, generate ends with a usage only chunk (`choices: []`). Send it through derender like any other chunk to get the usage chunk for the response. `usage.prompt_tokens` is the request's `prompt_tokens` if set, otherwise the generate chunk's.

102 - **Send the same context on every call.** `chat_request` and `prompt_token_ids` aren't kept between calls, so they go with every chunk including the usage chunk. Both are required when a tool or reasoning parser is configured. `prompt_token_ids` is the `token_ids` of the `GenerateRequest` returned by `/render`.

103 +- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.

104 - **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.

105

106 Streaming derender resolves `logprobs` on the plain detokenization path, including `token_id:N` placeholders, same as the non-streaming endpoints (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.

@@ -110,7 +115,7 @@ When a tool or reasoning parser is configured, parser internal state (buffered m

115

116 - **Transport**: `output_token_ids` and `output_chunk_lens` round-trip in full in both directions on every call. `output_chunk_lens` has one entry per chunk, which is one per token without speculative decoding. This means O(n) bytes per chunk, O(n²) bytes over a full generation. Bounded by `max_model_len`. `prompt_token_ids` is sent in full on every call too and it isn't trimmed as `output_token_ids` grows. This means that for most of a stream it dominates the per chunk payload. A 100k token prompt with 1k tokens of output means `prompt_token_ids` is ~99% of the request body on every chunk.

117 - **Compute**: replay is O(n) `parse_delta` calls per chunk (O(n²) per generation). `parse_delta` itself is O(n) for parsers that re-scan accumulated text (e.g. Hermes tool-call JSON, DeepSeek-R1 reasoning). The per-generation cost is O(n³) character work, not O(n²). This is a deliberately minimal first implementation with no caching layer.

113 -- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk.

118 +- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk. Send the full list and not a suffix because the last reasoning marker can be anywhere in the prompt.

119 - Replay runs off the event loop on the renderer's executor (`renderer_num_workers`, default `1`). Size it for the expected number of concurrent parser configured streams.

120

121 `output_token_ids` and `prompt_token_ids` are both bounded by `max_model_len` but callers streaming long reasoning traces through a parser configured model should expect materially more state transport and CPU cost than the plain detokenization path.

@@ -153,7 +158,7 @@ with httpx.Client(timeout=60.0) as client:

158 generate_request = client.post(

159 f"{RENDER}/v1/chat/completions/render", json=chat_request

160 ).json()

156 - prompt_tokens = len(generate_request["token_ids"])

161 + prompt_token_ids = generate_request["token_ids"]

162

163 # 2. Generate: token IDs -> token IDs (token-in / token-out engine)

164 generate_response = client.post(

@@ -166,7 +171,8 @@ with httpx.Client(timeout=60.0) as client:

171 json={

172 "model": MODEL,

173 "generate_response": generate_response,

169 - "prompt_tokens": prompt_tokens,

174 + "prompt_tokens": len(prompt_token_ids),

175 + "prompt_token_ids": prompt_token_ids,

176 "chat_request": chat_request,

177 },

178 ).json()

requirements/common.txt

+1 -1

@@ -9,7 +9,7 @@ blake3

9 py-cpuinfo

10 transformers >= 5.16.1

11 huggingface_hub >= 1.31.0 # Required for the `huggingface_hub.utils.httpx` re-export

12 -tokenizers >= 0.21.1 # Required for fast incremental detokenization.

12 +tokenizers >= 0.22.0 # Required for fast incremental detokenization.

13 safetensors >= 0.6.2 # MXFP4/MXFP6 dtype support (F8_E8M0, F4) added in 0.6.0: https://github.com/huggingface/safetensors/pull/611

14 protobuf >= 5.29.6, !=6.30.*, !=6.31.*, !=6.32.*, !=6.33.0.*, !=6.33.1.*, !=6.33.2.*, !=6.33.3.*, !=6.33.4.*; platform_machine != "s390x"

15 protobuf == 7.36.1; platform_machine == "s390x"

tests/entrypoints/scale_out/derender/test_derender.py

+135

@@ -27,6 +27,11 @@ async def client(server):

27 yield http_client

30 +@pytest.fixture(scope="module")

31 +def tokenizer():

32 + return get_tokenizer(MODEL_NAME)

33 +

34 +

35 # ---------------------------------------------------------------------------

36 # Helpers

37 # ---------------------------------------------------------------------------

@@ -372,6 +377,58 @@ async def test_derender_chat_model_omitted_resolves_served_name(client):

377 assert response.json()["model"] == MODEL_NAME

378

379

380 +@pytest.mark.asyncio

381 +async def test_derender_chat_leading_space_seeded_from_prompt(client, tokenizer):

382 + """prompt_token_ids keeps the first token's leading space on this

383 + Metaspace tokenizer, matching the coupled path."""

384 + gen_req = await _render_chat(client)

385 + prompt_token_ids = gen_req["token_ids"]

386 + output_ids = tokenizer.encode("Hello there, output", add_special_tokens=False)

387 +

388 + unseeded = await client.post(

389 + "/v1/chat/completions/derender",

390 + json={

391 + "model": MODEL_NAME,

392 + "generate_response": _make_generate_response(output_ids),

393 + },

394 + )

395 + seeded = await client.post(

396 + "/v1/chat/completions/derender",

397 + json={

398 + "model": MODEL_NAME,

399 + "generate_response": _make_generate_response(output_ids),

400 + "prompt_token_ids": prompt_token_ids,

401 + },

402 + )

403 + assert unseeded.status_code == 200

404 + assert seeded.status_code == 200

405 + unseeded_content = unseeded.json()["choices"][0]["message"]["content"]

406 + seeded_content = seeded.json()["choices"][0]["message"]["content"]

407 +

408 + assert not unseeded_content.startswith(" ")

409 + assert seeded_content.startswith(" ")

410 + assert seeded_content.lstrip(" ") == unseeded_content

411 +

412 +

413 +@pytest.mark.asyncio

414 +async def test_derender_chat_oversized_prompt_token_ids_rejected(client):

415 + """prompt_token_ids longer than max_model_len returns 400."""

416 + gen_req = await _render_chat(client)

417 + synthetic_ids = gen_req["token_ids"][:3]

418 + oversized_prompt_ids = [42] * 1_000_000

419 +

420 + response = await client.post(

421 + "/v1/chat/completions/derender",

422 + json={

423 + "model": MODEL_NAME,

424 + "generate_response": _make_generate_response(synthetic_ids),

425 + "prompt_token_ids": oversized_prompt_ids,

426 + },

427 + )

428 + assert response.status_code == 400

429 + assert "max_model_len" in response.json()["error"]["message"]

430 +

431 +

432 # ---------------------------------------------------------------------------

433 # Completion derender tests

434 # ---------------------------------------------------------------------------

@@ -486,6 +543,84 @@ async def test_derender_completion_prompt_tokens_length_mismatch(client):

543 assert response.status_code == 400

544

545

546 +@pytest.mark.asyncio

547 +async def test_derender_completion_prompt_token_ids_length_mismatch(client):

548 + """len(prompt_token_ids) != len(generate_responses) returns 400, the

549 + same as the existing prompt_tokens length check."""

550 + gr1 = await _render_completion(client, "Hello")

551 + ids1 = gr1["token_ids"][:3]

552 +

553 + response = await client.post(

554 + "/v1/completions/derender",

555 + json={

556 + "model": MODEL_NAME,

557 + "generate_responses": [

558 + _make_completion_generate_response(ids1, gr1["request_id"]),

559 + ],

560 + "prompt_token_ids": [[1, 2], [3, 4]],

561 + },

562 + )

563 + assert response.status_code == 400

564 +

565 +

566 +@pytest.mark.asyncio

567 +async def test_derender_completion_leading_space_seeded_from_prompt(client, tokenizer):

568 + """Completions counterpart of

569 + test_derender_chat_leading_space_seeded_from_prompt."""

570 + gr1 = await _render_completion(client, "Hello world")

571 + prompt_token_ids = gr1["token_ids"]

572 + output_ids = tokenizer.encode("Hello there, output", add_special_tokens=False)

573 +

574 + unseeded = await client.post(

575 + "/v1/completions/derender",

576 + json={

577 + "model": MODEL_NAME,

578 + "generate_responses": [

579 + _make_completion_generate_response(output_ids, gr1["request_id"]),

580 + ],

581 + },

582 + )

583 + seeded = await client.post(

584 + "/v1/completions/derender",

585 + json={

586 + "model": MODEL_NAME,

587 + "generate_responses": [

588 + _make_completion_generate_response(output_ids, gr1["request_id"]),

589 + ],

590 + "prompt_token_ids": [prompt_token_ids],

591 + },

592 + )

593 + assert unseeded.status_code == 200

594 + assert seeded.status_code == 200

595 + unseeded_text = unseeded.json()["choices"][0]["text"]

596 + seeded_text = seeded.json()["choices"][0]["text"]

597 +

598 + assert not unseeded_text.startswith(" ")

599 + assert seeded_text.startswith(" ")

600 + assert seeded_text.lstrip(" ") == unseeded_text

601 +

602 +

603 +@pytest.mark.asyncio

604 +async def test_derender_completion_oversized_prompt_token_ids_rejected(client):

605 + """A prompt_token_ids entry longer than max_model_len returns 400."""

606 + gr1 = await _render_completion(client, "Hello")

607 + ids1 = gr1["token_ids"][:3]

608 + oversized_prompt_ids = [42] * 1_000_000

609 +

610 + response = await client.post(

611 + "/v1/completions/derender",

612 + json={

613 + "model": MODEL_NAME,

614 + "generate_responses": [

615 + _make_completion_generate_response(ids1, gr1["request_id"]),

616 + ],

617 + "prompt_token_ids": [oversized_prompt_ids],

618 + },

619 + )

620 + assert response.status_code == 400

621 + assert "max_model_len" in response.json()["error"]["message"]

622 +

623 +

624 @pytest.mark.asyncio

625 async def test_derender_completion_empty_generate_responses(client):

626 """Empty generate_responses list returns 400."""

tests/entrypoints/scale_out/derender/test_derender_stream.py

+341 -88

@@ -20,11 +20,17 @@ Tests are split into two layers:

20 import json

21 import threading

22 from concurrent.futures import ThreadPoolExecutor

23 +from unittest.mock import AsyncMock, MagicMock

25 +import httpx

26 import pytest

27 import pytest_asyncio

28 +from pydantic import ValidationError

29 +from tokenizers.decoders import DecodeStream

31 +import vllm.renderers.online_derenderer as online_derenderer_module

32 from tests.entrypoints.scale_out.derender.utils import stream_chat_derender

33 +from tests.utils import RemoteLaunchRenderServer

34 from vllm.entrypoints.generate.base.protocol import (

35 DeltaFunctionCall,

36 DeltaMessage,

@@ -32,7 +38,17 @@ from vllm.entrypoints.generate.base.protocol import (

38 PerRequestMetrics,

39 SpeculativeDecodingMetrics,

40 )

41 +from vllm.entrypoints.openai.chat_completion.protocol import (

42 + ChatCompletionRequest,

43 + ChatCompletionStreamResponse,

44 +)

45 +from vllm.entrypoints.openai.completion.protocol import CompletionRequest

46 +from vllm.entrypoints.scale_out.derender.serving import ServingDerender

47 from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

48 + DerenderChatRequest,

49 + DerenderChatStreamRequest,

50 + DerenderCompletionRequest,

51 + DerenderCompletionStreamRequest,

52 DerenderStreamState,

53 GenerateResponse,

54 GenerateResponseChoice,

@@ -41,6 +57,12 @@ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

57 )

58 from vllm.entrypoints.serve.engine.protocol import ErrorResponse, UsageInfo

59 from vllm.parser import Parser

60 +from vllm.renderers.online_derenderer import (

61 + OnlineDerenderer,

62 + _decode_params,

63 + _seed_stream_state,

64 +)

65 +from vllm.tokenizers import get_tokenizer

66 from vllm.utils import random_uuid

68 MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM"

@@ -170,6 +192,7 @@ def _make_stream_chunk(

192 usage: dict | None = None,

193 metrics: PerRequestMetrics | None = None,

194 logprobs: dict | None = None,

195 + prompt_token_ids: list[int] | None = None,

196 ) -> GenerateStreamResponse:

197 """Build a GenerateStreamResponse SSE chunk."""

198 return GenerateStreamResponse(

@@ -184,6 +207,7 @@ def _make_stream_chunk(

207 ],

208 usage=UsageInfo(**usage) if usage else None,

209 metrics=metrics,

210 + prompt_token_ids=prompt_token_ids,

211 )

212

213

@@ -228,18 +252,12 @@ def _make_usage_chunk(

252 @pytest.fixture(scope="module")

253 def tokenizer():

254 """Load the tiny tokenizer used across unit tests."""

231 - from vllm.tokenizers import get_tokenizer

232 -

255 return get_tokenizer(MODEL_NAME)

256

257

258 @pytest.fixture(scope="module")

259 def derenderer(tokenizer, request):

260 """Construct a minimal OnlineDerenderer backed by a stub renderer."""

239 - from unittest.mock import MagicMock

240 -

241 - from vllm.renderers.online_derenderer import OnlineDerenderer

242 -

261 renderer = MagicMock()

262 renderer.get_tokenizer.return_value = tokenizer

263

@@ -273,10 +291,6 @@ def parsed_derenderer(tokenizer):

291 `renderer._executor` must be a real `ThreadPoolExecutor`.

292 `loop.run_in_executor` cannot submit work to a `MagicMock`.

293 """

276 - from unittest.mock import MagicMock

277 -

278 - from vllm.renderers.online_derenderer import OnlineDerenderer

279 -

294 renderer = MagicMock()

295 renderer.get_tokenizer.return_value = tokenizer

296 renderer._executor = ThreadPoolExecutor(max_workers=2)

@@ -433,6 +447,57 @@ class TestDetokenizeDelta:

447 # Bounded by a small constant independent of len(token_ids)

448 assert max_window <= 32

449

450 + def test_truncated_multibyte_char_is_held_back(self, derenderer, tokenizer):

451 + """Output ending mid-character emits no U+FFFD: batch == stream == engine."""

452 + emoji_ids = tokenizer.encode("🌈", add_special_tokens=False)

453 + byte_ids = emoji_ids[-4:]

454 + assert len(byte_ids) == 4

455 + output_ids = byte_ids[:-1]

456 +

457 + batch, _ = derenderer._detokenize_delta(

458 + tokenizer, output_ids, DerenderStreamState()

459 + )

460 + streamed = self._chunked(derenderer, tokenizer, [[t] for t in output_ids])

461 + engine = _decode_stream_expected(tokenizer, [], output_ids)

462 +

463 + assert "�" not in batch

464 + assert batch == streamed == engine

465 +

466 + full, _ = derenderer._detokenize_delta(

467 + tokenizer, byte_ids, DerenderStreamState()

468 + )

469 + assert full.endswith("🌈")

470 +

471 + @pytest.mark.parametrize(

472 + ("skip", "spaces", "preserve", "fast", "expected"),

473 + [

474 + (True, True, False, True, (True, True)),

475 + (True, False, False, True, (True, True)),

476 + (True, False, False, False, (True, False)),

477 + (False, False, False, True, (False, False)),

478 + (False, False, True, True, (False, False)),

479 + (True, False, True, True, (False, False)),

480 + (True, True, True, True, (False, True)),

481 + ],

482 + )

483 + def test_decode_params_mirror_engine(

484 + self, monkeypatch, skip, spaces, preserve, fast, expected

485 + ):

486 + """Flags match the engine's fast or slow detokenizer for the tokenizer."""

487 + monkeypatch.setattr(

488 + online_derenderer_module, "uses_fast_detokenizer", lambda _: fast

489 + )

490 + request = CompletionRequest(

491 + model=MODEL_NAME,

492 + prompt="x",

493 + skip_special_tokens=skip,

494 + spaces_between_special_tokens=spaces,

495 + )

496 + assert _decode_params(None, request, preserve_special=preserve) == expected

497 +

498 + def test_decode_params_default_without_request(self, tokenizer):

499 + assert _decode_params(tokenizer, None) == (True, True)

500 +

501 def test_n_independent_streams_same_result(self, derenderer, tokenizer):

502 """N parallel streams with the same token sequence give the same text."""

503 token_ids = tokenizer.encode("parallel streams")[:8]

@@ -451,6 +516,242 @@ class TestDetokenizeDelta:

516 assert results[0] == self._one_shot(tokenizer, token_ids)

517

518

519 +# ---------------------------------------------------------------------------

520 +# Prompt seeded detokenization (matches the engine's primed detokenizer)

521 +# ---------------------------------------------------------------------------

522 +

523 +

524 +def _decode_stream_expected(

525 + tokenizer, prompt_ids: list[int], output_ids: list[int]

526 +) -> str:

527 + """Engine-equivalent decode via `DecodeStream`, primed by stepping the

528 + prompt (`DecodeStream(ids=...)` needs tokenizers >= 0.22)."""

529 + stream = DecodeStream(skip_special_tokens=True)

530 + for tid in prompt_ids:

531 + stream.step(tokenizer.backend_tokenizer, tid)

532 + return "".join(

533 + s

534 + for tid in output_ids

535 + if (s := stream.step(tokenizer.backend_tokenizer, tid)) is not None

536 + )

537 +

538 +

539 +@pytest.fixture(scope="module")

540 +def leading_space_ids(tokenizer):

541 + """(prompt_ids, output_ids, expected) where seeded and unseeded decode

542 + differ by the leading space."""

543 + prompt_ids = tokenizer.encode("[INST] Hi [/INST]", add_special_tokens=False)

544 + output_ids = tokenizer.encode("Hello there, output", add_special_tokens=False)

545 + expected = _decode_stream_expected(tokenizer, prompt_ids, output_ids)

546 + unseeded = tokenizer.decode(output_ids, skip_special_tokens=True)

547 + assert expected.startswith(" ") and not unseeded.startswith(" "), (

548 + "fixture no longer reproduces the leading-space divergence; "

549 + "pick a different prompt/output pair"

550 + )

551 + return prompt_ids, output_ids, expected

552 +

553 +

554 +class TestSeedStreamState:

555 + """`_seed_stream_state`: pure helper building the initial decode state."""

556 +

557 + def test_seeds_prev_tokens_from_prompt_tail(self, tokenizer, leading_space_ids):

558 + prompt_ids, _, _ = leading_space_ids

559 + state = _seed_stream_state(tokenizer, prompt_ids, skip_special_tokens=True)

560 + assert state.prev_tokens

561 + assert state.read_offset == len(state.prev_tokens)

562 +

563 + def test_none_prompt_token_ids_returns_empty_state(self, tokenizer):

564 + assert (

565 + _seed_stream_state(tokenizer, None, skip_special_tokens=True)

566 + == DerenderStreamState()

567 + )

568 +

569 + def test_empty_prompt_token_ids_returns_empty_state(self, tokenizer):

570 + assert (

571 + _seed_stream_state(tokenizer, [], skip_special_tokens=True)

572 + == DerenderStreamState()

573 + )

574 +

575 +

576 +class TestPromptSeededLeadingSpace:

577 + """Every decode entry point keeps the first token's leading space when

578 + given prompt_token_ids."""

579 +

580 + @pytest.mark.asyncio

581 + async def test_chat_batch_seeded_via_explicit_prompt_token_ids(

582 + self, derenderer, leading_space_ids

583 + ):

584 + prompt_ids, output_ids, expected = leading_space_ids

585 + choices = await derenderer.derender_chat(

586 + GenerateResponse(

587 + request_id="t",

588 + choices=[

589 + GenerateResponseChoice(

590 + index=0, token_ids=output_ids, finish_reason="stop"

591 + )

592 + ],

593 + ),

594 + prompt_token_ids=prompt_ids,

595 + )

596 + assert choices[0].message.content == expected

597 +

598 + @pytest.mark.asyncio

599 + async def test_chat_batch_seeded_via_generate_response_fallback(

600 + self, derenderer, leading_space_ids

601 + ):

602 + prompt_ids, output_ids, expected = leading_space_ids

603 + choices = await derenderer.derender_chat(

604 + GenerateResponse(

605 + request_id="t",

606 + choices=[

607 + GenerateResponseChoice(

608 + index=0, token_ids=output_ids, finish_reason="stop"

609 + )

610 + ],

611 + prompt_token_ids=prompt_ids,

612 + ),

613 + )

614 + assert choices[0].message.content == expected

615 +

616 + @pytest.mark.asyncio

617 + async def test_chat_batch_explicit_takes_precedence_over_fallback(

618 + self, derenderer, tokenizer, leading_space_ids

619 + ):

620 + prompt_ids, output_ids, expected = leading_space_ids

621 + wrong_ids = tokenizer.encode("a different prompt", add_special_tokens=False)

622 + choices = await derenderer.derender_chat(

623 + GenerateResponse(

624 + request_id="t",

625 + choices=[

626 + GenerateResponseChoice(

627 + index=0, token_ids=output_ids, finish_reason="stop"

628 + )

629 + ],

630 + prompt_token_ids=wrong_ids,

631 + ),

632 + prompt_token_ids=prompt_ids,

633 + )

634 + assert choices[0].message.content == expected

635 +

636 + @pytest.mark.asyncio

637 + async def test_chat_batch_without_prompt_context_stays_unseeded(

638 + self, derenderer, leading_space_ids

639 + ):

640 + _, output_ids, _ = leading_space_ids

641 + choices = await derenderer.derender_chat(

642 + GenerateResponse(

643 + request_id="t",

644 + choices=[

645 + GenerateResponseChoice(

646 + index=0, token_ids=output_ids, finish_reason="stop"

647 + )

648 + ],

649 + ),

650 + )

651 + assert not choices[0].message.content.startswith(" ")

652 +

653 + @pytest.mark.asyncio

654 + async def test_completion_batch_seeded(self, derenderer, leading_space_ids):

655 + prompt_ids, output_ids, expected = leading_space_ids

656 + choices, _, _ = await derenderer.derender_completion(

657 + [

658 + GenerateResponse(

659 + request_id="t",

660 + choices=[

661 + GenerateResponseChoice(

662 + index=0, token_ids=output_ids, finish_reason="stop"

663 + )

664 + ],

665 + )

666 + ],

667 + prompt_token_ids=[prompt_ids],

668 + )

669 + assert choices[0].text == expected

670 +

671 + @pytest.mark.asyncio

672 + @pytest.mark.parametrize("from_generate_chunk", [False, True])

673 + async def test_chat_stream_first_chunk_seeded(

674 + self, derenderer, leading_space_ids, from_generate_chunk

675 + ):

676 + prompt_ids, output_ids, expected = leading_space_ids

677 + mid = len(output_ids) // 2

678 +

679 + chunk1, state = await derenderer.derender_chat_stream(

680 + model=MODEL_NAME,

681 + generate_chunk=_make_stream_chunk(

682 + output_ids[:mid],

683 + prompt_token_ids=prompt_ids if from_generate_chunk else None,

684 + ),

685 + prompt_token_ids=None if from_generate_chunk else prompt_ids,

686 + )

687 + # Only needed on the first chunk; the carried state already holds

688 + # the seeded window from here on.

689 + chunk2, _ = await derenderer.derender_chat_stream(

690 + model=MODEL_NAME,

691 + generate_chunk=_make_stream_chunk(output_ids[mid:], finish_reason="stop"),

692 + state=state,

693 + )

694 +

695 + streamed = (chunk1.choices[0].delta.content or "") + (

696 + chunk2.choices[0].delta.content or ""

697 + )

698 + assert streamed == expected

699 +

700 + @pytest.mark.asyncio

701 + @pytest.mark.parametrize("from_generate_chunk", [False, True])

702 + async def test_completion_stream_first_chunk_seeded(

703 + self, derenderer, leading_space_ids, from_generate_chunk

704 + ):

705 + prompt_ids, output_ids, expected = leading_space_ids

706 + mid = len(output_ids) // 2

707 +

708 + chunk1, state = await derenderer.derender_completion_stream(

709 + model=MODEL_NAME,

710 + generate_chunk=_make_stream_chunk(

711 + output_ids[:mid],

712 + prompt_token_ids=prompt_ids if from_generate_chunk else None,

713 + ),

714 + prompt_token_ids=None if from_generate_chunk else prompt_ids,

715 + )

716 + chunk2, _ = await derenderer.derender_completion_stream(

717 + model=MODEL_NAME,

718 + generate_chunk=_make_stream_chunk(output_ids[mid:], finish_reason="stop"),

719 + state=state,

720 + )

721 +

722 + assert chunk1.choices[0].text + chunk2.choices[0].text == expected

723 +

724 + @pytest.mark.asyncio

725 + async def test_parser_path_ephemeral_state_seeded(

726 + self, parsed_derenderer, monkeypatch, leading_space_ids

727 + ):

728 + """The parser path's per-call detok window is seeded too, so

729 + `parse_delta` sees the same text as coupled serving."""

730 +

731 + class _EchoParser(Parser):

732 + def parse_delta(

733 + self,

734 + delta_text,

735 + delta_token_ids,

736 + request,

737 + prompt_token_ids=None,

738 + *,

739 + finished,

740 + ):

741 + return DeltaMessage(content=delta_text) if delta_text else None

742 +

743 + monkeypatch.setattr(parsed_derenderer, "parser", _EchoParser)

744 + prompt_ids, output_ids, expected = leading_space_ids

745 +

746 + chunk, _ = await parsed_derenderer.derender_chat_stream(

747 + model=MODEL_NAME,

748 + generate_chunk=_make_stream_chunk(output_ids, finish_reason="stop"),

749 + chat_request=_chat_request(),

750 + prompt_token_ids=prompt_ids,

751 + )

752 + assert chunk.choices[0].delta.content == expected

753 +

754 +

755 @pytest.mark.asyncio

756 @pytest.mark.parametrize(

757 "stream_method",

@@ -460,8 +761,6 @@ async def test_streaming_detokenization_runs_off_event_loop(

761 derenderer, monkeypatch, stream_method

762 ):

763 """Streaming detokenization runs on the renderer executor."""

463 - import vllm.renderers.online_derenderer as online_derenderer_module

464 -

764 event_loop_thread_id = threading.get_ident()

765 executor_thread_id = derenderer.renderer._executor.submit(

766 threading.get_ident

@@ -561,8 +860,6 @@ class TestDerenderCompletionStream:

860 @pytest.mark.asyncio

861 async def test_skip_special_tokens_threaded(self, derenderer, tokenizer):

862 """completion_request.skip_special_tokens is honored (not hardcoded True)."""

564 - from vllm.entrypoints.openai.completion.protocol import CompletionRequest

565 -

863 eos = tokenizer.eos_token_id

864 if eos is None:

865 pytest.skip("tokenizer has no eos token to exercise special stripping")

@@ -789,8 +1086,6 @@ class TestDerenderChatStream:

1086

1087

1088 def _chat_request(**kwargs):

792 - from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest

793 -

1089 kwargs.setdefault("messages", [{"role": "user", "content": "hi"}])

1090 kwargs.setdefault("model", MODEL_NAME)

1091 return ChatCompletionRequest(**kwargs)

@@ -1072,8 +1367,6 @@ def harmony_encode():

1367 @pytest.fixture(scope="module")

1368 def harmony_tokenizer():

1369 pytest.importorskip("openai_harmony")

1075 - from vllm.tokenizers import get_tokenizer

1076 -

1370 return get_tokenizer(HARMONY_MODEL, trust_remote_code=True)

1371

1372

@@ -1085,10 +1378,7 @@ def harmony_derenderer(harmony_tokenizer):

1378 with a real tokenizer and parser, so the replay path is exercised

1379 against Harmony's actual channel grammar rather than a stub.

1380 """

1088 - from unittest.mock import MagicMock

1089 -

1381 from vllm.parser.harmony import HarmonyParser

1091 - from vllm.renderers.online_derenderer import OnlineDerenderer

1382

1383 renderer = MagicMock()

1384 renderer.get_tokenizer.return_value = harmony_tokenizer

@@ -1224,20 +1514,14 @@ class TestDerenderStreamStateValidation:

1514 """DerenderStreamState rejects malformed caller supplied offsets/lengths."""

1515

1516 def test_negative_prefix_offset_rejected(self):

1227 - from pydantic import ValidationError

1228 -

1517 with pytest.raises(ValidationError):

1518 DerenderStreamState(prefix_offset=-1)

1519

1520 def test_negative_read_offset_rejected(self):

1233 - from pydantic import ValidationError

1234 -

1521 with pytest.raises(ValidationError):

1522 DerenderStreamState(read_offset=-1)

1523

1524 def test_prev_tokens_over_cap_rejected(self):

1239 - from pydantic import ValidationError

1240 -

1525 with pytest.raises(ValidationError):

1526 DerenderStreamState(prev_tokens=["a"] * 1025)

1527

@@ -1246,14 +1530,10 @@ class TestDerenderStreamStateValidation:

1530 assert len(state.prev_tokens) == 1024

1531

1532 def test_output_chunk_lens_mismatch_rejected(self):

1249 - from pydantic import ValidationError

1250 -

1533 with pytest.raises(ValidationError):

1534 DerenderStreamState(output_token_ids=[1, 2, 3], output_chunk_lens=[1, 1])

1535

1536 def test_output_chunk_lens_zero_entry_rejected(self):

1255 - from pydantic import ValidationError

1256 -

1537 with pytest.raises(ValidationError):

1538 DerenderStreamState(output_token_ids=[1, 2], output_chunk_lens=[0, 2])

1539

@@ -1264,14 +1544,40 @@ class TestDerenderStreamStateValidation:

1544 assert state.output_chunk_lens == [1, 2]

1545

1546

1547 +_GEN = {"request_id": "t", "choices": []}

1548 +_GEN_BAD = {**_GEN, "prompt_token_ids": [-1]}

1549 +

1550 +

1551 +@pytest.mark.parametrize(

1552 + "model_cls,payload",

1553 + [

1554 + (DerenderChatRequest, {"generate_response": _GEN, "prompt_token_ids": [-1]}),

1555 + (DerenderChatRequest, {"generate_response": _GEN_BAD}),

1556 + (

1557 + DerenderCompletionRequest,

1558 + {"generate_responses": [_GEN], "prompt_token_ids": [[-1]]},

1559 + ),

1560 + (

1561 + DerenderChatStreamRequest,

1562 + {"stream": True, "generate_chunk": _GEN, "prompt_token_ids": [-1]},

1563 + ),

1564 + (

1565 + DerenderCompletionStreamRequest,

1566 + {"stream": True, "generate_chunk": _GEN_BAD},

1567 + ),

1568 + ],

1569 +)

1570 +def test_negative_prompt_token_ids_rejected(model_cls, payload):

1571 + """Negative prompt ids would reach convert_ids_to_tokens, so they get a

1572 + 400 at parse time like `GenerateRequest.token_ids` does."""

1573 + with pytest.raises(ValidationError, match="prompt_token_ids"):

1574 + model_cls.model_validate(payload)

1575 +

1576 +

1577 class TestServingDerenderStreamErrorHandling:

1578 """Malformed stream_state must surface as 400 and not an unhandled 500."""

1579

1580 def _make_serving(self, side_effect: Exception):

1271 - from unittest.mock import AsyncMock, MagicMock

1272 -

1273 - from vllm.entrypoints.scale_out.derender.serving import ServingDerender

1274 -

1581 models = MagicMock()

1582 models.is_base_model.return_value = True

1583 models.model_config = MagicMock()

@@ -1289,10 +1595,6 @@ class TestServingDerenderStreamErrorHandling:

1595 @pytest.mark.asyncio

1596 @pytest.mark.parametrize("exc", [KeyError("bad byte"), IndexError("oob")])

1597 async def test_completion_stream_bad_state_returns_400(self, exc):

1292 - from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

1293 - DerenderCompletionStreamRequest,

1294 - )

1295 -

1598 serving = self._make_serving(exc)

1599 request = DerenderCompletionStreamRequest(

1600 stream=True,

@@ -1307,10 +1609,6 @@ class TestServingDerenderStreamErrorHandling:

1609 @pytest.mark.asyncio

1610 @pytest.mark.parametrize("exc", [KeyError("bad byte"), IndexError("oob")])

1611 async def test_chat_stream_bad_state_returns_400(self, exc):

1310 - from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

1311 - DerenderChatStreamRequest,

1312 - )

1313 -

1612 serving = self._make_serving(exc)

1613 request = DerenderChatStreamRequest(

1614 stream=True,

@@ -1329,13 +1627,6 @@ class TestServingDerenderStreamValidation:

1627 before touching the tokenizer)."""

1628

1629 def _make_serving(self, *, parser_configured: bool, max_model_len: int = 100_000):

1332 - from unittest.mock import AsyncMock, MagicMock

1333 -

1334 - from vllm.entrypoints.openai.chat_completion.protocol import (

1335 - ChatCompletionStreamResponse,

1336 - )

1337 - from vllm.entrypoints.scale_out.derender.serving import ServingDerender

1338 -

1630 models = MagicMock()

1631 models.is_base_model.return_value = True

1632 models.model_config = MagicMock()

@@ -1353,10 +1644,6 @@ class TestServingDerenderStreamValidation:

1644

1645 @pytest.mark.asyncio

1646 async def test_missing_chat_request_with_parser_rejected(self):

1356 - from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

1357 - DerenderChatStreamRequest,

1358 - )

1359 -

1647 serving = self._make_serving(parser_configured=True)

1648 request = DerenderChatStreamRequest(

1649 stream=True,

@@ -1371,10 +1658,6 @@ class TestServingDerenderStreamValidation:

1658

1659 @pytest.mark.asyncio

1660 async def test_missing_chat_request_without_parser_ok(self):

1374 - from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

1375 - DerenderChatStreamRequest,

1376 - )

1377 -

1661 serving = self._make_serving(parser_configured=False)

1662 request = DerenderChatStreamRequest(

1663 stream=True,

@@ -1390,10 +1673,6 @@ class TestServingDerenderStreamValidation:

1673 the same way it rejects a missing chat_request. Without it,

1674 parse_delta cannot tell whether the prompt left reasoning open and

1675 would misclassify reasoning content as plain content."""

1393 - from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

1394 - DerenderChatStreamRequest,

1395 - )

1396 -

1676 serving = self._make_serving(parser_configured=True)

1677 request = DerenderChatStreamRequest(

1678 stream=True,

@@ -1409,10 +1688,6 @@ class TestServingDerenderStreamValidation:

1688

1689 @pytest.mark.asyncio

1690 async def test_prompt_token_ids_present_with_parser_ok(self):

1412 - from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

1413 - DerenderChatStreamRequest,

1414 - )

1415 -

1691 serving = self._make_serving(parser_configured=True)

1692 request = DerenderChatStreamRequest(

1693 stream=True,

@@ -1426,10 +1701,6 @@ class TestServingDerenderStreamValidation:

1701

1702 @pytest.mark.asyncio

1703 async def test_oversized_output_token_ids_rejected(self):

1429 - from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

1430 - DerenderChatStreamRequest,

1431 - )

1432 -

1704 serving = self._make_serving(parser_configured=False, max_model_len=4)

1705 request = DerenderChatStreamRequest(

1706 stream=True,

@@ -1451,10 +1722,6 @@ class TestServingDerenderStreamValidation:

1722 parser configured deployment rescans it in full on every chunk

1723 (is_reasoning_end / adjust_initial_state_from_prompt), so it must be

1724 bounded the same way output_token_ids is."""

1454 - from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

1455 - DerenderChatStreamRequest,

1456 - )

1457 -

1725 serving = self._make_serving(parser_configured=True, max_model_len=4)

1726 request = DerenderChatStreamRequest(

1727 stream=True,

@@ -1475,10 +1742,6 @@ class TestServingDerenderStreamValidation:

1742 DerenderStreamState is threaded through every choice). The check

1743 could never fire since derender_chat_stream itself rejects anything

1744 above 1 first."""

1478 - from vllm.entrypoints.scale_out.token_in_token_out.protocol import (

1479 - DerenderChatStreamRequest,

1480 - )

1481 -

1745 serving = self._make_serving(parser_configured=False)

1746 two_choices = GenerateStreamResponse(

1747 request_id="t",

@@ -1503,16 +1766,12 @@ class TestServingDerenderStreamValidation:

1766

1767 @pytest.fixture(scope="module")

1768 def server():

1506 - from tests.utils import RemoteLaunchRenderServer

1507 -

1769 with RemoteLaunchRenderServer(MODEL_NAME, []) as remote_server:

1770 yield remote_server

1771

1772

1773 @pytest_asyncio.fixture

1774 async def client(server):

1514 - import httpx

1515 -

1775 async with httpx.AsyncClient(

1776 base_url=server.url_for(""), timeout=30.0

1777 ) as http_client:

@@ -1776,8 +2035,6 @@ _PARSER_TOOLS = [

2035

2036 @pytest.fixture(scope="module")

2037 def parser_server():

1779 - from tests.utils import RemoteLaunchRenderServer

1780 -

2038 args = [

2039 "--enable-auto-tool-choice",

2040 "--tool-call-parser",

@@ -1791,8 +2048,6 @@ def parser_server():

2048

2049 @pytest_asyncio.fixture

2050 async def parser_client(parser_server):

1794 - import httpx

1795 -

2051 async with httpx.AsyncClient(

2052 base_url=parser_server.url_for(""), timeout=60.0

2053 ) as http_client:

@@ -1801,8 +2056,6 @@ async def parser_client(parser_server):

2056

2057 @pytest.fixture(scope="module")

2058 def parser_tokenizer():

1804 - from vllm.tokenizers import get_tokenizer

1805 -

2059 return get_tokenizer(PARSER_MODEL)

2060

2061

vllm/entrypoints/scale_out/derender/serving.py

+32 -1

@@ -117,6 +117,20 @@ class ServingDerender(BaseServing):

117

118 return None

119

120 + def _validate_prompt_token_ids(

121 + self,

122 + prompt_token_ids: list[list[int] | None],

123 + ) -> ErrorResponse | None:

124 + """Reject caller-supplied prompt_token_ids longer than max_model_len."""

125 + max_model_len = self.model_config.max_model_len

126 + for ids in prompt_token_ids:

127 + if ids is not None and len(ids) > max_model_len:

128 + return self.create_error_response(

129 + f"prompt_token_ids length ({len(ids)}) exceeds "

130 + f"max_model_len ({max_model_len})."

131 + )

132 + return None

133 +

134 async def derender_chat_response(

135 self,

136 request: DerenderChatRequest,

@@ -138,6 +152,10 @@ class ServingDerender(BaseServing):

152 if bounds_error is not None:

153 return bounds_error

154

155 + prompt_ids_error = self._validate_prompt_token_ids([request.prompt_token_ids])

156 + if prompt_ids_error is not None:

157 + return prompt_ids_error

158 +

159 if self.online_derenderer.parser is not None and request.chat_request is None:

160 return self.create_error_response(

161 "chat_request is required when a tool or reasoning parser is "

@@ -147,7 +165,9 @@ class ServingDerender(BaseServing):

165

166 try:

167 choices = await self.online_derenderer.derender_chat(

150 - request.generate_response, request.chat_request

168 + request.generate_response,

169 + request.chat_request,

170 + request.prompt_token_ids,

171 )

172 except ValueError as exc:

173 return self.create_error_response(str(exc))

@@ -203,6 +223,11 @@ class ServingDerender(BaseServing):

223 if bounds_error is not None:

224 return bounds_error

225

226 + if request.prompt_token_ids is not None:

227 + prompt_ids_error = self._validate_prompt_token_ids(request.prompt_token_ids)

228 + if prompt_ids_error is not None:

229 + return prompt_ids_error

230 +

231 (

232 choices,

233 total_prompt_tokens,

@@ -211,6 +236,7 @@ class ServingDerender(BaseServing):

236 request.generate_responses,

237 request.prompt_tokens,

238 completion_request=request.completion_request,

239 + prompt_token_ids=request.prompt_token_ids,

240 )

241

242 first = request.generate_responses[0]

@@ -367,6 +393,10 @@ class ServingDerender(BaseServing):

393 if error_check_ret is not None:

394 return error_check_ret

395

396 + prompt_ids_error = self._validate_prompt_token_ids([request.prompt_token_ids])

397 + if prompt_ids_error is not None:

398 + return prompt_ids_error

399 +

400 model_name = request.model or self.models.model_name()

401 try:

402 (

@@ -378,6 +408,7 @@ class ServingDerender(BaseServing):

408 state=request.stream_state,

409 prompt_tokens=request.prompt_tokens,

410 completion_request=request.completion_request,

411 + prompt_token_ids=request.prompt_token_ids,

412 )

413 except ValueError as exc:

414 return self.create_error_response(str(exc))

vllm/entrypoints/scale_out/token_in_token_out/protocol.py

+43 -5

@@ -5,6 +5,7 @@ from typing import Annotated, Any, Literal, TypeAlias

5 from pydantic import (

6 BaseModel,

7 Field,

8 + NonNegativeInt,

9 PrivateAttr,

10 field_validator,

11 model_validator,

@@ -362,7 +363,7 @@ class GenerateStreamResponse(BaseModel):

363 )

364 choices: list[GenerateResponseStreamChoice]

365 usage: UsageInfo | None = Field(default=None)

365 - prompt_token_ids: list[int] | None = None

366 + prompt_token_ids: list[NonNegativeInt] | None = None

367 mm_placeholders: dict[str, list[PlaceholderRangeInfo]] | None = None

368 metrics: PerRequestMetrics | None = None

369

@@ -382,7 +383,7 @@ class GenerateResponse(BaseModel):

383 usage: UsageInfo | None = Field(default=None)

384 prompt_logprobs: list[dict[int, Logprob] | None] | None = None

385 prompt_token_id_logprobs: str | None = None

385 - prompt_token_ids: list[int] | None = None

386 + prompt_token_ids: list[NonNegativeInt] | None = None

387 mm_placeholders: dict[str, list[PlaceholderRangeInfo]] | None = None

388 metrics: PerRequestMetrics | None = None

389

@@ -421,6 +422,15 @@ class DerenderChatRequest(BaseModel):

422 len(GenerateRequest.token_ids) from the render step.

423 """

424

425 + prompt_token_ids: list[NonNegativeInt] | None = None

426 + """Prompt token IDs (`GenerateRequest.token_ids` from /render). Seeds

427 + detokenization from the prompt tail so the first output token keeps its

428 + leading space on SentencePiece tokenizers. Falls back to

429 + `generate_response.prompt_token_ids`, then to unseeded decoding.

430 +

431 + Only the last few IDs are read, so a suffix of the prompt is enough.

432 + """

433 +

434 chat_request: ChatCompletionRequest | None = None

435 """The original (post-adjust_request) ChatCompletionRequest from /render.

436

@@ -455,6 +465,14 @@ class DerenderCompletionRequest(BaseModel):

465 If provided, len(prompt_tokens) must equal len(generate_responses).

466 """

467

468 + prompt_token_ids: list[list[NonNegativeInt] | None] | None = None

469 + """One prompt token ID list per response, used to seed detokenization.

470 + See `DerenderChatRequest.prompt_token_ids`.

471 +

472 + If provided, len(prompt_token_ids) must equal len(generate_responses).

473 + A `None` entry falls back to `generate_responses[i].prompt_token_ids`.

474 + """

475 +

476 completion_request: CompletionRequest | None = None

477 """The original (post-adjust_request) CompletionRequest from /render.

478

@@ -472,6 +490,13 @@ class DerenderCompletionRequest(BaseModel):

490 f"prompt_tokens length ({len(self.prompt_tokens)}) must equal "

491 f"generate_responses length ({len(self.generate_responses)})"

492 )

493 + if self.prompt_token_ids is not None and len(self.prompt_token_ids) != len(

494 + self.generate_responses

495 + ):

496 + raise ValueError(

497 + f"prompt_token_ids length ({len(self.prompt_token_ids)}) must "

498 + f"equal generate_responses length ({len(self.generate_responses)})"

499 + )

500 return self

501

502

@@ -627,7 +652,7 @@ class DerenderChatStreamRequest(BaseModel):

652 prompt_tokens: int | None = None

653 """Prompt token count for usage. Forwarded from the render step."""

654

630 - prompt_token_ids: list[int] | None = None

655 + prompt_token_ids: list[NonNegativeInt] | None = None

656 """Prompt token IDs. Required by the parser path's `parse_delta` to

657 settle its initial reasoning state (e.g. chat templates that pre-open

658 ``<think>``). `prompt_tokens` is a usage count and cannot serve this

@@ -636,8 +661,15 @@ class DerenderChatStreamRequest(BaseModel):

661 Rejected with a 400 (by `ServingDerender`) when a tool or reasoning

662 parser is configured and this is omitted. Without it, `parse_delta`

663 cannot tell whether the prompt left reasoning open and would silently

639 - misclassify reasoning content as plain content. Unused on the plain

640 - detokenization path.

664 + misclassify reasoning content as plain content. On all paths it also

665 + seeds detokenization on the first chunk, falling back to

666 + `generate_chunk.prompt_token_ids` when omitted (see

667 + `DerenderChatRequest.prompt_token_ids`).

668 +

669 + With a parser configured, send the full list. Parsers look back to the

670 + last reasoning marker which can be anywhere in the prompt, so a suffix

671 + can flip the initial reasoning state. Without a parser, a suffix is

672 + enough.

673 """

674

675 chat_request: ChatCompletionRequest | None = None

@@ -666,6 +698,12 @@ class DerenderCompletionStreamRequest(BaseModel):

698 prompt_tokens: int | None = None

699 """Prompt token count for usage."""

700

701 + prompt_token_ids: list[NonNegativeInt] | None = None

702 + """Prompt token IDs, used on the first chunk to seed detokenization.

703 + Falls back to `generate_chunk.prompt_token_ids`. See

704 + `DerenderChatRequest.prompt_token_ids`.

705 + """

706 +

707 completion_request: CompletionRequest | None = None

708 """The original (post adjust_request) CompletionRequest from /render."""

709 # --8<-- [end:derender-completion-stream-request]

vllm/renderers/online_derenderer.py

+191 -56

@@ -40,9 +40,14 @@ from vllm.logger import init_logger

40 from vllm.parser import Parser, ParserManager

41 from vllm.renderers import BaseRenderer

42 from vllm.tokenizers import TokenizerLike

43 -from vllm.tokenizers.detokenizer_utils import detokenize_incrementally

43 +from vllm.tokenizers.detokenizer_utils import (

44 + convert_prompt_ids_to_tokens,

45 + detokenize_incrementally,

46 + get_leading_space_marker,

47 +)

48 from vllm.utils import random_uuid

49 from vllm.utils.async_utils import make_async

50 +from vllm.v1.engine.detokenizer import uses_fast_detokenizer

52 logger = init_logger(__name__)

@@ -115,17 +120,37 @@ class OnlineDerenderer:

120 self,

121 generate_response: GenerateResponse,

122 chat_request: ChatCompletionRequest | None = None,

123 + prompt_token_ids: list[int] | None = None,

124 ) -> list[ChatCompletionResponseChoice]:

119 - return await self._derender_chat_async(generate_response, chat_request)

125 + return await self._derender_chat_async(

126 + generate_response, chat_request, prompt_token_ids

127 + )

128

129 def _derender_chat(

130 self,

131 generate_response: GenerateResponse,

132 chat_request: ChatCompletionRequest | None = None,

133 + prompt_token_ids: list[int] | None = None,

134 ) -> list[ChatCompletionResponseChoice]:

135 tokenizer = self.renderer.get_tokenizer()

136 choices: list[ChatCompletionResponseChoice] = []

137

138 + has_parser = self.parser is not None and chat_request is not None

139 + skip_special, spaces_between = _decode_params(

140 + tokenizer, chat_request, preserve_special=has_parser

141 + )

142 + seed_ids = (

143 + prompt_token_ids

144 + if prompt_token_ids is not None

145 + else generate_response.prompt_token_ids

146 + )

147 + # Choices continue the same prompt, so they share one seeded window.

148 + seed_state = _seed_stream_state(

149 + tokenizer,

150 + seed_ids,

151 + skip_special_tokens=skip_special,

152 + )

153 +

154 for choice in generate_response.choices:

155 if not choice.token_ids:

156 raise ValueError(f"choice {choice.index} has empty or null token_ids")

@@ -136,12 +161,17 @@ class OnlineDerenderer:

161 else None

162 )

163

139 - if self.parser is not None and chat_request is not None:

164 + if has_parser:

165 + assert self.parser is not None and chat_request is not None

166 # Parser path: decode with special tokens preserved

167 # so the parser can see markers like </think>,

168 # <tool_call>, or Harmony channel tokens.

143 - decoded_text = tokenizer.decode(

144 - choice.token_ids, skip_special_tokens=False

169 + decoded_text, _ = self._detokenize_delta(

170 + tokenizer,

171 + choice.token_ids,

172 + seed_state,

173 + skip_special_tokens=skip_special,

174 + spaces_between_special_tokens=spaces_between,

175 )

176

177 chat_template_kwargs: dict[str, Any] = {}

@@ -201,13 +231,12 @@ class OnlineDerenderer:

231 else:

232 # No parser: plain detokenization honouring the request's

233 # skip_special_tokens (default True when no request was given).

204 - skip_special = (

205 - chat_request.skip_special_tokens

206 - if chat_request is not None

207 - else True

208 - )

209 - decoded_text = tokenizer.decode(

210 - choice.token_ids, skip_special_tokens=skip_special

234 + decoded_text, _ = self._detokenize_delta(

235 + tokenizer,

236 + choice.token_ids,

237 + seed_state,

238 + skip_special_tokens=skip_special,

239 + spaces_between_special_tokens=spaces_between,

240 )

241 message = ChatMessage(role="assistant", content=decoded_text)

242

@@ -239,7 +268,7 @@ class OnlineDerenderer:

268 (tracked by ``read_offset``). The delta tokens are fed straight onto it.

269

270 The window is bounded. ``detokenize_incrementally`` never reads before

242 - ``prefix_offset``, so after processing we trim ``prev_tokens`` to that

271 + ``prefix_offset``, so after each token we trim ``prev_tokens`` to that

272 tail and rebase the offsets to it. State transport therefore stays

273 O(window) per chunk instead of re-sending the full token history.

274

@@ -273,18 +302,20 @@ class OnlineDerenderer:

302 skip_special_tokens=skip_special_tokens,

303 spaces_between_special_tokens=spaces_between_special_tokens,

304 )

276 - prev_tokens = prev_tokens + new_toks

305 + # Trim to the tail still readable by detokenize_incrementally

306 + # (everything before prefix_offset is dead) and rebase the

307 + # offsets, so the window stays bounded within a long batch

308 + # decode as well as across chunks.

309 + prev_tokens = (prev_tokens + new_toks)[prefix_offset:]

310 + read_offset -= prefix_offset

311 + prefix_offset = 0

312 text_parts.append(text)

313

279 - # Trim to the tail still readable by detokenize_incrementally

280 - # (everything before prefix_offset is dead) and rebase the offsets so

281 - # the carried window stays bounded regardless of generation length.

282 - trimmed = prev_tokens[prefix_offset:]

314 updated_state = state.model_copy(

315 update={

285 - "prev_tokens": trimmed,

286 - "prefix_offset": 0,

287 - "read_offset": read_offset - prefix_offset,

316 + "prev_tokens": prev_tokens,

317 + "prefix_offset": prefix_offset,

318 + "read_offset": read_offset,

319 }

320 )

321 return "".join(text_parts), updated_state

@@ -315,20 +346,16 @@ class OnlineDerenderer:

346 (validated by the caller — see `ServingDerender`) because

347 plain detokenization would leak raw parser markup into `content`.

348 prompt_tokens: Prompt token count for the usage chunk.

318 - prompt_token_ids: Prompt token IDs. Parser path only. Lets

319 - `parse_delta` settle its initial reasoning state. Required

320 - when a reasoning or tool parser is configured (validated by

321 - the caller — see ``ServingDerender``). Without it reasoning

322 - left open by the prompt would be misclassified as content.

349 + prompt_token_ids: Prompt token IDs. Required on the parser path

350 + (validated by the caller) and optional otherwise, falling

351 + back to ``generate_chunk.prompt_token_ids``. See

352 + `DerenderChatStreamRequest.prompt_token_ids`.

353

354 Returns:

355 (chunk, updated_state) — the derendered SSE chunk and the state

356 the client must pass to the next call.

357

358 """

329 - if state is None:

330 - state = DerenderStreamState()

331 -

359 # A single DerenderStreamState is threaded through every choice in

360 # this chunk. Correct only when there is at most one choice per SSE

361 # event (n=1, one call per index), as the streaming derender

@@ -353,23 +380,34 @@ class OnlineDerenderer:

380 parser_cls,

381 model,

382 generate_chunk,

356 - state,

383 + state if state is not None else DerenderStreamState(),

384 chat_request,

385 prompt_tokens,

386 prompt_token_ids,

387 )

388

389 tokenizer = self.renderer.get_tokenizer()

363 - skip_special = (

364 - chat_request.skip_special_tokens if chat_request is not None else True

365 - )

390 + skip_special, spaces_between = _decode_params(tokenizer, chat_request)

391 + # Seed on the first chunk only. A carried state already has the prompt.

392 + if state is None:

393 + state = _seed_stream_state(

394 + tokenizer,

395 + prompt_token_ids

396 + if prompt_token_ids is not None

397 + else generate_chunk.prompt_token_ids,

398 + skip_special_tokens=skip_special,

399 + )

400 stream_choices: list[ChatCompletionResponseStreamChoice] = []

401 updated_state = state

402

403 for choice in generate_chunk.choices:

404 delta_tids = choice.token_ids or []

405 new_text, updated_state = await self._detokenize_delta_async(

372 - tokenizer, delta_tids, updated_state, skip_special_tokens=skip_special

406 + tokenizer,

407 + delta_tids,

408 + updated_state,

409 + skip_special_tokens=skip_special,

410 + spaces_between_special_tokens=spaces_between,

411 )

412

413 # NOTE: parser-configured servers dispatch to

@@ -454,7 +492,8 @@ class OnlineDerenderer:

492 `state.output_chunk_lens`, so the rebuilt parser state matches

493 standard serving, which calls `parse_delta` once per engine step.

494 Text comes from a fresh incremental detokenizer with special tokens

457 - preserved (``skip_special_tokens=False``).

495 + preserved (``skip_special_tokens=False``), seeded from the prompt

496 + tail on every call since replay starts from scratch.

497 """

498 tokenizer = self.renderer.get_tokenizer()

499

@@ -480,7 +519,12 @@ class OnlineDerenderer:

519 # across both the replay and current chunk phases (via `nonlocal`)

520 # so multi-byte characters split across that boundary still decode

521 # correctly. Discarded once the call returns.

483 - detok_state = DerenderStreamState()

522 + skip_special, spaces_between = _decode_params(

523 + tokenizer, chat_request, preserve_special=True

524 + )

525 + detok_state = _seed_stream_state(

526 + tokenizer, prompt_token_ids, skip_special_tokens=skip_special

527 + )

528

529 def _replay(token_ids: list[int], chunk_lens: list[int]) -> None:

530 """Replay prior chunks through `parse_delta` to rebuild parser

@@ -493,7 +537,11 @@ class OnlineDerenderer:

537 chunk = token_ids[start : start + chunk_len]

538 start += chunk_len

539 text, detok_state = self._detokenize_delta(

496 - tokenizer, chunk, detok_state, skip_special_tokens=False

540 + tokenizer,

541 + chunk,

542 + detok_state,

543 + skip_special_tokens=skip_special,

544 + spaces_between_special_tokens=spaces_between,

545 )

546 parser.parse_delta(

547 text,

@@ -530,7 +578,11 @@ class OnlineDerenderer:

578 # granularity), not one per token. See the granularity note

579 # in this method's docstring.

580 text, detok_state = self._detokenize_delta(

533 - tokenizer, delta_tids, detok_state, skip_special_tokens=False

581 + tokenizer,

582 + delta_tids,

583 + detok_state,

584 + skip_special_tokens=skip_special,

585 + spaces_between_special_tokens=spaces_between,

586 )

587 delta_message = parser.parse_delta(

588 text,

@@ -633,9 +685,10 @@ class OnlineDerenderer:

685 generate_responses: list[GenerateResponse],

686 prompt_tokens: list[int] | None = None,

687 completion_request: CompletionRequest | None = None,

688 + prompt_token_ids: list[list[int] | None] | None = None,

689 ) -> tuple[list[CompletionResponseChoice], int, int]:

690 return await self._derender_completion_async(

638 - generate_responses, prompt_tokens, completion_request

691 + generate_responses, prompt_tokens, completion_request, prompt_token_ids

692 )

693

694 def _derender_completion(

@@ -643,24 +696,34 @@ class OnlineDerenderer:

696 generate_responses: list[GenerateResponse],

697 prompt_tokens: list[int] | None = None,

698 completion_request: CompletionRequest | None = None,

699 + prompt_token_ids: list[list[int] | None] | None = None,

700 ) -> tuple[list[CompletionResponseChoice], int, int]:

701 n = len(generate_responses)

702 prompt_tokens_list: list[int] = (

703 prompt_tokens if prompt_tokens is not None else [0] * n

704 )

651 -

652 - skip_special = (

653 - completion_request.skip_special_tokens

654 - if completion_request is not None

655 - else True

705 + prompt_token_ids_list: list[list[int] | None] = (

706 + prompt_token_ids if prompt_token_ids is not None else [None] * n

707 )

708 +

709 tokenizer = self.renderer.get_tokenizer()

710 + skip_special, spaces_between = _decode_params(tokenizer, completion_request)

711 choices: list[CompletionResponseChoice] = []

712 total_prompt_tokens = 0

713 total_completion_tokens = 0

714 index = 0

715

663 - for gen, pt in zip(generate_responses, prompt_tokens_list):

716 + for gen, pt, seed_ids_override in zip(

717 + generate_responses, prompt_tokens_list, prompt_token_ids_list

718 + ):

719 + seed_ids = (

720 + seed_ids_override

721 + if seed_ids_override is not None

722 + else gen.prompt_token_ids

723 + )

724 + seed_state = _seed_stream_state(

725 + tokenizer, seed_ids, skip_special_tokens=skip_special

726 + )

727 for choice in gen.choices:

728 if not choice.token_ids:

729 raise ValueError(

@@ -668,8 +731,12 @@ class OnlineDerenderer:

731 "has empty or null token_ids"

732 )

733

671 - decoded_text = tokenizer.decode(

672 - choice.token_ids, skip_special_tokens=skip_special

734 + decoded_text, _ = self._detokenize_delta(

735 + tokenizer,

736 + choice.token_ids,

737 + seed_state,

738 + skip_special_tokens=skip_special,

739 + spaces_between_special_tokens=spaces_between,

740 )

741 completion_logprobs = None

742 if choice.logprobs is not None:

@@ -698,6 +765,7 @@ class OnlineDerenderer:

765 state: DerenderStreamState | None = None,

766 prompt_tokens: int | None = None,

767 completion_request: CompletionRequest | None = None,

768 + prompt_token_ids: list[int] | None = None,

769 ) -> tuple[CompletionStreamResponse, DerenderStreamState]:

770 """Process one GenerateStreamResponse chunk for streaming completions.

771

@@ -716,14 +784,13 @@ class OnlineDerenderer:

784 prompt_tokens: Prompt token count for usage (from the render step).

785 completion_request: Original CompletionRequest from ``/render``;

786 supplies ``skip_special_tokens``.

787 + prompt_token_ids: Seeds the first chunk's decode. Falls back to

788 + ``generate_chunk.prompt_token_ids``.

789

790 Returns:

791 (chunk, updated_state) — the derendered chunk and updated state.

792

793 """

724 - if state is None:

725 - state = DerenderStreamState()

726 -

794 # See the equivalent check in derender_chat_stream: a single

795 # DerenderStreamState is threaded through every choice in this

796 # chunk, so more than one choice per chunk would corrupt the

@@ -734,18 +801,27 @@ class OnlineDerenderer:

801 )

802

803 tokenizer = self.renderer.get_tokenizer()

737 - skip_special = (

738 - completion_request.skip_special_tokens

739 - if completion_request is not None

740 - else True

741 - )

804 + skip_special, spaces_between = _decode_params(tokenizer, completion_request)

805 + # Seed on the first chunk only. A carried state already has the prompt.

806 + if state is None:

807 + state = _seed_stream_state(

808 + tokenizer,

809 + prompt_token_ids

810 + if prompt_token_ids is not None

811 + else generate_chunk.prompt_token_ids,

812 + skip_special_tokens=skip_special,

813 + )

814 stream_choices: list[CompletionResponseStreamChoice] = []

815 updated_state = state

816

817 for choice in generate_chunk.choices:

818 delta_tids = choice.token_ids or []

819 new_text, updated_state = await self._detokenize_delta_async(

748 - tokenizer, delta_tids, updated_state, skip_special_tokens=skip_special

820 + tokenizer,

821 + delta_tids,

822 + updated_state,

823 + skip_special_tokens=skip_special,

824 + spaces_between_special_tokens=spaces_between,

825 )

826

827 completion_logprobs = None

@@ -810,6 +886,65 @@ def _logprob_context_tail(

886 return (list(context_token_ids) + list(delta_token_ids))[-_LOGPROB_CONTEXT_WINDOW:]

887

888

889 +def _decode_params(

890 + tokenizer: TokenizerLike,

891 + request: ChatCompletionRequest | CompletionRequest | None,

892 + preserve_special: bool = False,

893 +) -> tuple[bool, bool]:

894 + """Derive `(skip_special_tokens, spaces_between_special_tokens)` the way

895 + the engine does (`IncrementalDetokenizer`).

896 +

897 + Args:

898 + tokenizer: Picks the engine's fast or slow detokenizer rules.

899 + request: The original request, if the caller supplied one.

900 + preserve_special: Keep special tokens so a parser can see markers.

901 + The serving side does this via `adjust_request`.

902 +

903 + """

904 + skip_special = (

905 + False

906 + if preserve_special

907 + else (request.skip_special_tokens if request is not None else True)

908 + )

909 + spaces_between = (

910 + request.spaces_between_special_tokens if request is not None else True

911 + )

912 + # Only the fast detokenizer forces spaces on when skipping special tokens.

913 + if skip_special and uses_fast_detokenizer(tokenizer):

914 + spaces_between = True

915 + return skip_special, spaces_between

916 +

917 +

918 +def _seed_stream_state(

919 + tokenizer: TokenizerLike,

920 + prompt_token_ids: list[int] | None,

921 + skip_special_tokens: bool,

922 +) -> DerenderStreamState:

923 + """Build the initial decode state from the prompt tail, the same way the

924 + engine primes its incremental detokenizer. Without it, Metaspace

925 + tokenizers drop the first output token's leading space.

926 +

927 + Returns an empty state when `prompt_token_ids` is empty or omitted.

928 + """

929 + if not prompt_token_ids:

930 + if prompt_token_ids is None and get_leading_space_marker(tokenizer) is not None:

931 + logger.warning_once(

932 + "derender got no prompt_token_ids, so the first output token "

933 + "may lose its leading space compared to the coupled endpoint. "

934 + "Pass prompt_token_ids from the render step to avoid this."

935 + )

936 + return DerenderStreamState()

937 +

938 + prev_tokens, prefix_offset, read_offset = convert_prompt_ids_to_tokens(

939 + tokenizer, prompt_token_ids, skip_special_tokens=skip_special_tokens

940 + )

941 + return DerenderStreamState(

942 + prev_tokens=prev_tokens,

943 + prefix_offset=prefix_offset,

944 + read_offset=read_offset,

945 + )

946 +

947 +

948 def _parse_token_id_placeholder(token: str) -> int | None:

949 """Extract token ID from a 'token_id:N' placeholder string."""

950 if not token.startswith("token_id:"):

vllm/tokenizers/detokenizer_utils.py

+2 -2

@@ -63,7 +63,7 @@ _CACHED_MARKER_KEY = "_vllm_space_marker_cache"

63 _NOT_CACHED = "__not_computed__"

66 -def _get_leading_space_marker(tokenizer: TokenizerLike) -> str | None:

66 +def get_leading_space_marker(tokenizer: TokenizerLike) -> str | None:

67 """Read the space marker from the tokenizer's pre_tokenizer config.

69 Only Metaspace pre_tokenizers (used by SentencePiece-based models like

@@ -160,7 +160,7 @@ def convert_ids_list_to_tokens(

160 """

161 if not token_ids:

162 return []

163 - marker = _get_leading_space_marker(tokenizer)

163 + marker = get_leading_space_marker(tokenizer)

164 if marker is None:

165 return [tokenizer.decode([tid]) or "" for tid in token_ids]

166 raw_tokens = tokenizer.convert_ids_to_tokens(token_ids)

vllm/v1/engine/detokenizer.py

+6 -7

@@ -3,9 +3,7 @@

3 import sys

4 from abc import ABC, abstractmethod

6 -import tokenizers

6 import tokenizers.decoders

8 -from packaging import version

7 from tokenizers import Tokenizer

8 from transformers import TokenizersBackend

@@ -20,14 +18,15 @@ from vllm.v1.engine import EngineCoreRequest

19 logger = init_logger(__name__)

23 -# Only tokenizers >= 0.22.0 supports DecodeStream with native prefill

24 -# (ids parameter) used for FastIncrementalDetokenizer.

25 -USE_FAST_DETOKENIZER = version.parse(tokenizers.__version__) >= version.parse("0.22.0")

26 -

21 # Error string from https://github.com/huggingface/tokenizers/blob/909fdde2a4ffedd9295206f705eb612be2a91b12/tokenizers/src/tokenizer/mod.rs#L1042

22 INVALID_PREFIX_ERR_MSG = "Invalid prefix encountered"

25 +def uses_fast_detokenizer(tokenizer: TokenizerLike) -> bool:

26 + """Whether the engine detokenizes with `FastIncrementalDetokenizer`."""

27 + return isinstance(tokenizer, TokenizersBackend)

28 +

29 +

30 class IncrementalDetokenizer:

31 def __init__(self):

32 self.token_ids: list[int] = []

@@ -58,7 +57,7 @@ class IncrementalDetokenizer:

57 # No tokenizer => skipping detokenization.

58 return IncrementalDetokenizer()

61 - if USE_FAST_DETOKENIZER and isinstance(tokenizer, TokenizersBackend):

60 + if uses_fast_detokenizer(tokenizer):

61 # Fast tokenizer => use tokenizers library DecodeStream.

62 return FastIncrementalDetokenizer(tokenizer, request)