@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 2a16ece2d3

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 4, 03:45 UTC

tokenizer: Add fastokens support (#41741)

Signed-off-by: AlonKejzman <alonkeizman@gmail.com> Co-authored-by: Roger Wang <hey@rogerw.io>

AlonKejzman committed May 7, 2026 at 17:49 UTC 2a16ece2d342c0c154a4949ad317b521f8c04ec4

6 files changed +93 -5

docs/configuration/optimization.md

+30

@@ -270,6 +270,36 @@ Known supported models (with corresponding benchmarks):

270

271 ## Input Processing

272

273 +### fastokens Tokenizer Mode

274 +

275 +By default vLLM uses the standard Hugging Face `tokenizers` library to power

276 +the fast tokenizer (`--tokenizer-mode hf`). For BPE tokenizers (Qwen, Llama,

277 +DeepSeek, GPT-OSS, etc.) you can switch to the

278 +[fastokens](https://github.com/crusoecloud/fastokens) Rust backend, a drop-in

279 +replacement that's substantially faster on encode/decode and on streaming

280 +detokenization:

281 +

282 +```console

283 +vllm serve Qwen/Qwen3-8B --tokenizer-mode fastokens

284 +```

285 +

286 +Equivalent in the offline API:

287 +

288 +```python

289 +from vllm import LLM

290 +llm = LLM(model="Qwen/Qwen3-8B", tokenizer_mode="fastokens")

291 +```

292 +

293 +The `fastokens` Python package must be installed; if it isn't, vLLM raises

294 +a clear `ImportError` at tokenizer load. `fastokens` loads a Hugging Face

295 +fast tokenizer with its inner Rust tokenizer replaced by the fastokens shim,

296 +so it is mutually exclusive with non-HF modes such as `mistral` or

297 +`deepseek_v32`.

298 +

299 +Tokenizer-bound workloads — long shared prefixes, bursty short prompts,

300 +batch detokenization — see the largest wins. If your bottleneck is GPU

301 +prefill/decode, the tokenizer change is unlikely to be visible end-to-end.

302 +

303 ### Parallel Processing

304

305 You can run input processing in parallel via [API server scale-out](../serving/data_parallel_deployment.md#internal-load-balancing).

docs/design/huggingface_integration.md

+1 -1

@@ -21,7 +21,7 @@ Let's say we want to serve the popular Qwen model by running `vllm serve Qwen/Qw

22 Beyond that, there are two more things vLLM depends on Hugging Face for.

24 -1. **Tokenizer**: vLLM uses the tokenizer from Hugging Face to tokenize the input text. The tokenizer is loaded using [AutoTokenizer.from_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) with the `model` argument as the model name and the `--revision` argument as the revision. It is also possible to use a tokenizer from another model by specifying the `--tokenizer` argument in the `vllm serve` command. Other relevant arguments are `--tokenizer-revision` and `--tokenizer-mode`. Please check Hugging Face's documentation for the meaning of these arguments. This part of the logic can be found in the [get_tokenizer](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/tokenizer.py#L87) function. After obtaining the tokenizer, notably, vLLM will cache some expensive attributes of the tokenizer in [vllm.tokenizers.hf.get_cached_tokenizer][].

24 +1. **Tokenizer**: vLLM uses the tokenizer from Hugging Face to tokenize the input text. The tokenizer is loaded using [AutoTokenizer.from_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) with the `model` argument as the model name and the `--revision` argument as the revision. It is also possible to use a tokenizer from another model by specifying the `--tokenizer` argument in the `vllm serve` command. Other relevant arguments are `--tokenizer-revision` and `--tokenizer-mode`. Setting `--tokenizer-mode fastokens` swaps in a drop-in Rust BPE backend for the HF fast tokenizer (see [fastokens Tokenizer Mode](../configuration/optimization.md#fastokens-tokenizer-mode)). Please check Hugging Face's documentation for the meaning of these arguments. This part of the logic can be found in the [get_tokenizer](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/tokenizer.py#L87) function. After obtaining the tokenizer, notably, vLLM will cache some expensive attributes of the tokenizer in [vllm.tokenizers.hf.get_cached_tokenizer][].

26 2. **Model weight**: vLLM downloads the model weight from the Hugging Face model hub using the `model` argument as the model name and the `--revision` argument as the revision. vLLM provides the argument `--load-format` to control what files to download from the model hub. By default, it will try to load the weights in the safetensors format and fall back to the PyTorch bin format if the safetensors format is not available. We can also pass `--load-format dummy` to skip downloading the weights.

27 - It is recommended to use the safetensors format, as it is efficient for loading in distributed inference and also safe from arbitrary code execution. See the [documentation](https://huggingface.co/docs/safetensors/en/index) for more information on the safetensors format. This part of the logic can be found [here](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/model_executor/model_loader/loader.py#L385). Please note that:

vllm/config/model.py

+6 -1

@@ -83,7 +83,9 @@ logger = init_logger(__name__)

83 RunnerOption = Literal["auto", RunnerType]

84 ConvertType = Literal["none", "embed", "classify"]

85 ConvertOption = Literal["auto", ConvertType]

86 -TokenizerMode = Literal["auto", "hf", "slow", "mistral", "deepseek_v32", "deepseek_v4"]

86 +TokenizerMode = Literal[

87 + "auto", "hf", "slow", "mistral", "deepseek_v32", "deepseek_v4", "fastokens"

88 +]

89 ModelDType = Literal["auto", "half", "float16", "bfloat16", "float", "float32"]

90 LogprobsMode = Literal[

91 "raw_logits", "raw_logprobs", "processed_logits", "processed_logprobs"

@@ -136,6 +138,9 @@ class ModelConfig:

138 - "deepseek_v32" will always use the tokenizer from `deepseek_v32`.

139 - "deepseek_v4" will always use the tokenizer from `deepseek_v4`.

140 - "qwen_vl" will always use the tokenizer from `qwen_vl`.

141 + - "fastokens" loads a Hugging Face fast tokenizer powered by the

142 + [fastokens](https://github.com/crusoecloud/fastokens) Rust BPE backend

143 + (requires the `fastokens` package to be installed).

144 - Other custom values can be supported via plugins."""

145 trust_remote_code: bool = False

146 """Trust remote code (e.g., from HuggingFace) when downloading the model

vllm/tokenizers/fastokens.py new

+47

@@ -0,0 +1,47 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +"""``fastokens`` tokenizer mode.

4 +

5 +Loads a Hugging Face fast tokenizer whose internal Rust tokenizer is replaced

6 +by the fastokens shim. fastokens also rebinds

7 +``tokenizers.decoders.DecodeStream`` so the streaming detokenizer accepts the

8 +shim. Both patches are installed for the lifetime of the process —

9 +``patch_transformers()`` is idempotent.

10 +"""

11 +

12 +from pathlib import Path

13 +

14 +from .hf import CachedHfTokenizer, HfTokenizer

15 +from .protocol import TokenizerLike

16 +

17 +

18 +def _apply_fastokens_patch() -> None:

19 + try:

20 + import fastokens

21 + except ImportError as e:

22 + raise ImportError(

23 + "The 'fastokens' package is required for tokenizer_mode='fastokens'."

24 + ) from e

25 + fastokens.patch_transformers()

26 +

27 +

28 +class FastokensTokenizer(TokenizerLike):

29 + @classmethod

30 + def from_pretrained(

31 + cls,

32 + path_or_repo_id: str | Path,

33 + *args,

34 + trust_remote_code: bool = False,

35 + revision: str | None = None,

36 + download_dir: str | None = None,

37 + **kwargs,

38 + ) -> HfTokenizer:

39 + _apply_fastokens_patch()

40 + return CachedHfTokenizer.from_pretrained(

41 + path_or_repo_id,

42 + *args,

43 + trust_remote_code=trust_remote_code,

44 + revision=revision,

45 + download_dir=download_dir,

46 + **kwargs,

47 + )

vllm/tokenizers/registry.py

@@ -43,6 +43,7 @@ _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS: set[str] = {"step3_vl"}

43 _VLLM_TOKENIZERS = {

44 "deepseek_v32": ("deepseek_v32", "DeepseekV32Tokenizer"),

45 "deepseek_v4": ("deepseek_v4", "DeepseekV4Tokenizer"),

46 + "fastokens": ("fastokens", "FastokensTokenizer"),

47 "grok2": ("grok2", "Grok2Tokenizer"),

48 "hf": ("hf", "CachedHfTokenizer"),

49 "kimi_audio": ("kimi_audio", "KimiAudioTokenizer"),

vllm/v1/engine/detokenizer.py

+8 -3

@@ -3,9 +3,9 @@

3 from abc import ABC, abstractmethod

5 import tokenizers

6 +import tokenizers.decoders

7 from packaging import version

8 from tokenizers import Tokenizer

8 -from tokenizers.decoders import DecodeStream

9 from transformers import PreTrainedTokenizerFast

11 from vllm.logger import init_logger

@@ -177,7 +177,10 @@ class FastIncrementalDetokenizer(BaseIncrementalDetokenizer):

177 self.tokenizer: Tokenizer = tokenizer._tokenizer

178

179 # Use native prefill to prime the decode stream with prompt tokens.

180 - self.stream = DecodeStream(

180 + # Look up DecodeStream on the module so backend patches (e.g. the

181 + # fastokens shim that replaces ``tokenizers.decoders.DecodeStream``)

182 + # are honored regardless of import order.

183 + self.stream = tokenizers.decoders.DecodeStream(

184 ids=request.prompt_token_ids,

185 skip_special_tokens=self.skip_special_tokens,

186 )

@@ -237,7 +240,9 @@ class FastIncrementalDetokenizer(BaseIncrementalDetokenizer):

240 " for request %s, resetting decode stream.",

241 self.request_id,

242 )

240 - self.stream = DecodeStream(skip_special_tokens=self.skip_special_tokens)

243 + self.stream = tokenizers.decoders.DecodeStream(

244 + skip_special_tokens=self.skip_special_tokens

245 + )

246 token = self.stream.step(self.tokenizer, next_token_id)

247 return token

248