A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 3, 03:00 UTC
[Frontend] Rework fastokens integration (#43168)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Nick Hill committed
May 21, 2026 at 15:36 UTC
0f66623b0d739dc94afddb67863c37d6f5816579
7 files changed
+63
-55
docs/configuration/optimization.md
+16
-13
@@ -270,31 +270,34 @@ Known supported models (with corresponding benchmarks):
270
271
## Input Processing
272
273
-### fastokens Tokenizer Mode
273
+### fastokens Backend
274
275
By default vLLM uses the standard Hugging Face `tokenizers` library to power
276
-the fast tokenizer (`--tokenizer-mode hf`). For BPE tokenizers (Qwen, Llama,
277
-DeepSeek, GPT-OSS, etc.) you can switch to the
278
-[fastokens](https://github.com/crusoecloud/fastokens) Rust backend, a drop-in
279
-replacement that's substantially faster on encode/decode and on streaming
280
-detokenization:
276
+the fast tokenizer. For BPE tokenizers (Qwen, Llama, DeepSeek, GPT-OSS, etc.)
277
+you can switch to the [fastokens](https://github.com/crusoecloud/fastokens)
278
+Rust backend, a drop-in replacement that's substantially faster on
279
+encode/decode and on streaming detokenization. Enable it by setting
280
+`VLLM_USE_FASTOKENS=1`:
281
282
```console
283
-vllm serve Qwen/Qwen3-8B --tokenizer-mode fastokens
283
+VLLM_USE_FASTOKENS=1 vllm serve Qwen/Qwen3-8B
284
```
285
286
Equivalent in the offline API:
287
288
```python
289
+import os
290
+os.environ["VLLM_USE_FASTOKENS"] = "1"
291
+
292
from vllm import LLM
290
-llm = LLM(model="Qwen/Qwen3-8B", tokenizer_mode="fastokens")
293
+llm = LLM(model="Qwen/Qwen3-8B")
294
```
295
293
-The `fastokens` Python package must be installed; if it isn't, vLLM raises
294
-a clear `ImportError` at tokenizer load. `fastokens` loads a Hugging Face
295
-fast tokenizer with its inner Rust tokenizer replaced by the fastokens shim,
296
-so it is mutually exclusive with non-HF modes such as `mistral` or
297
-`deepseek_v32`.
296
+The `fastokens` Python package (>= 0.2.0) must be installed; if it isn't,
297
+vLLM raises a clear `ImportError` at tokenizer load. The override applies to
298
+any `--tokenizer-mode` that ends up loading an HF fast tokenizer (`hf`,
299
+`deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Modes that don't use the HF
300
+fast tokenizer (`mistral`, `grok2`, `kimi_audio`) ignore the flag.
301
302
Tokenizer-bound workloads — long shared prefixes, bursty short prompts,
303
batch detokenization — see the largest wins. If your bottleneck is GPU
docs/design/huggingface_integration.md
+1
-1
@@ -21,7 +21,7 @@ Let's say we want to serve the popular Qwen model by running `vllm serve Qwen/Qw
21
22
Beyond that, there are two more things vLLM depends on Hugging Face for.
23
24
-1. **Tokenizer**: vLLM uses the tokenizer from Hugging Face to tokenize the input text. The tokenizer is loaded using [AutoTokenizer.from_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) with the `model` argument as the model name and the `--revision` argument as the revision. It is also possible to use a tokenizer from another model by specifying the `--tokenizer` argument in the `vllm serve` command. Other relevant arguments are `--tokenizer-revision` and `--tokenizer-mode`. Setting `--tokenizer-mode fastokens` swaps in a drop-in Rust BPE backend for the HF fast tokenizer (see [fastokens Tokenizer Mode](../configuration/optimization.md#fastokens-tokenizer-mode)). Please check Hugging Face's documentation for the meaning of these arguments. This part of the logic can be found in the [get_tokenizer](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/tokenizer.py#L87) function. After obtaining the tokenizer, notably, vLLM will cache some expensive attributes of the tokenizer in [vllm.tokenizers.hf.get_cached_tokenizer][].
24
+1. **Tokenizer**: vLLM uses the tokenizer from Hugging Face to tokenize the input text. The tokenizer is loaded using [AutoTokenizer.from_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) with the `model` argument as the model name and the `--revision` argument as the revision. It is also possible to use a tokenizer from another model by specifying the `--tokenizer` argument in the `vllm serve` command. Other relevant arguments are `--tokenizer-revision` and `--tokenizer-mode`. Setting `VLLM_USE_FASTOKENS=1` swaps in a drop-in Rust BPE backend for any HF fast tokenizer loaded by vLLM (see [fastokens Backend](../configuration/optimization.md#fastokens-backend)). Please check Hugging Face's documentation for the meaning of these arguments. This part of the logic can be found in the [get_tokenizer](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/tokenizer.py#L87) function. After obtaining the tokenizer, notably, vLLM will cache some expensive attributes of the tokenizer in [vllm.tokenizers.hf.get_cached_tokenizer][].
25
26
2. **Model weight**: vLLM downloads the model weight from the Hugging Face model hub using the `model` argument as the model name and the `--revision` argument as the revision. vLLM provides the argument `--load-format` to control what files to download from the model hub. By default, it will try to load the weights in the safetensors format and fall back to the PyTorch bin format if the safetensors format is not available. We can also pass `--load-format dummy` to skip downloading the weights.
27
- It is recommended to use the safetensors format, as it is efficient for loading in distributed inference and also safe from arbitrary code execution. See the [documentation](https://huggingface.co/docs/safetensors/en/index) for more information on the safetensors format. This part of the logic can be found [here](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/model_executor/model_loader/loader.py#L385). Please note that:
vllm/config/model.py
+8
-7
@@ -93,9 +93,7 @@ def is_cumem_allocator_available() -> bool:
93
RunnerOption = Literal["auto", RunnerType]
94
ConvertType = Literal["none", "embed", "classify"]
95
ConvertOption = Literal["auto", ConvertType]
96
-TokenizerMode = Literal[
97
- "auto", "hf", "slow", "mistral", "deepseek_v32", "deepseek_v4", "fastokens"
98
-]
96
+TokenizerMode = Literal["auto", "hf", "slow", "mistral", "deepseek_v32", "deepseek_v4"]
97
ModelDType = Literal["auto", "half", "float16", "bfloat16", "float", "float32"]
98
LogprobsMode = Literal[
99
"raw_logits", "raw_logprobs", "processed_logits", "processed_logprobs"
@@ -148,10 +146,13 @@ class ModelConfig:
146
- "deepseek_v32" will always use the tokenizer from `deepseek_v32`.
147
- "deepseek_v4" will always use the tokenizer from `deepseek_v4`.
148
- "qwen_vl" will always use the tokenizer from `qwen_vl`.
151
- - "fastokens" loads a Hugging Face fast tokenizer powered by the
152
- [fastokens](https://github.com/crusoecloud/fastokens) Rust BPE backend
153
- (requires the `fastokens` package to be installed).
154
- - Other custom values can be supported via plugins."""
149
+ - Other custom values can be supported via plugins.
150
+
151
+ To swap the Rust BPE backend that powers HF fast tokenizers for the
152
+ [fastokens](https://github.com/crusoecloud/fastokens) implementation, set
153
+ `VLLM_USE_FASTOKENS=1` instead — that override applies to any mode that
154
+ loads an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`,
155
+ `qwen_vl`, …)."""
156
trust_remote_code: bool = False
157
"""Trust remote code (e.g., from HuggingFace) when downloading the model
158
and tokenizer."""
vllm/envs.py
+7
@@ -17,6 +17,7 @@ if TYPE_CHECKING:
17
VLLM_PORT: int | None = None
18
VLLM_RPC_BASE_PATH: str = tempfile.gettempdir()
19
VLLM_USE_MODELSCOPE: bool = False
20
+ VLLM_USE_FASTOKENS: bool = False
21
VLLM_RINGBUFFER_WARNING_INTERVAL: int = 60
22
VLLM_NCCL_SO_PATH: str | None = None
23
LD_LIBRARY_PATH: str | None = None
@@ -676,6 +677,12 @@ environment_variables: dict[str, Callable[[], Any]] = {
677
"VLLM_USE_MODELSCOPE": lambda: (
678
os.environ.get("VLLM_USE_MODELSCOPE", "False").lower() == "true"
679
),
680
+ # If true, replace the Rust BPE backend that powers HF fast tokenizers
681
+ # with the `fastokens` (https://github.com/crusoecloud/fastokens) shim.
682
+ # Applies to any tokenizer mode that loads an HF fast tokenizer
683
+ # (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). The `fastokens`
684
+ # Python package must be installed.
685
+ "VLLM_USE_FASTOKENS": lambda: bool(int(os.getenv("VLLM_USE_FASTOKENS", "0"))),
686
# Interval in seconds to log a warning message when the ring buffer is full
687
"VLLM_RINGBUFFER_WARNING_INTERVAL": lambda: int(
688
os.environ.get("VLLM_RINGBUFFER_WARNING_INTERVAL", "60")
vllm/renderers/registry.py
-1
@@ -22,7 +22,6 @@ logger = init_logger(__name__)
22
_VLLM_RENDERERS = {
23
"deepseek_v32": ("deepseek_v32", "DeepseekV32Renderer"),
24
"deepseek_v4": ("deepseek_v4", "DeepseekV4Renderer"),
25
- "fastokens": ("hf", "HfRenderer"),
25
"grok2": ("grok2", "Grok2Renderer"),
26
"hf": ("hf", "HfRenderer"),
27
"kimi_audio": ("hf", "HfRenderer"),
vllm/tokenizers/fastokens.py
+24
-32
@@ -1,47 +1,39 @@
1
# SPDX-License-Identifier: Apache-2.0
2
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
-"""``fastokens`` tokenizer mode.
3
+"""``fastokens`` backend patch.
4
5
-Loads a Hugging Face fast tokenizer whose internal Rust tokenizer is replaced
6
-by the fastokens shim. fastokens also rebinds
7
-``tokenizers.decoders.DecodeStream`` so the streaming detokenizer accepts the
8
-shim. Both patches are installed for the lifetime of the process —
9
-``patch_transformers()`` is idempotent.
5
+When ``VLLM_USE_FASTOKENS=1`` is set, ``fastokens.patch_transformers()`` swaps
6
+the inner Rust tokenizer of every HF fast tokenizer loaded afterwards with the
7
+fastokens shim and rebinds ``tokenizers.decoders.DecodeStream`` so the
8
+streaming detokenizer accepts the shim. The patch is process-global and
9
+idempotent, so it applies to any tokenizer mode that ends up loading an HF
10
+fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …).
11
"""
12
12
-from pathlib import Path
13
+from importlib.metadata import PackageNotFoundError, version
14
14
-from .hf import CachedHfTokenizer, HfTokenizer
15
-from .protocol import TokenizerLike
15
+from packaging.version import Version
16
17
+_MIN_FASTOKENS_VERSION = "0.2.0"
18
18
-def _apply_fastokens_patch() -> None:
19
+
20
+def apply_fastokens_patch() -> None:
21
try:
22
import fastokens
23
except ImportError as e:
24
raise ImportError(
23
- "The 'fastokens' package is required for tokenizer_mode='fastokens'."
25
+ f"The 'fastokens' package (>= {_MIN_FASTOKENS_VERSION}) is required "
26
+ "when VLLM_USE_FASTOKENS=1."
27
) from e
25
- fastokens.patch_transformers()
28
27
-
28
-class FastokensTokenizer(TokenizerLike):
29
- @classmethod
30
- def from_pretrained(
31
- cls,
32
- path_or_repo_id: str | Path,
33
- *args,
34
- trust_remote_code: bool = False,
35
- revision: str | None = None,
36
- download_dir: str | None = None,
37
- **kwargs,
38
- ) -> HfTokenizer:
39
- _apply_fastokens_patch()
40
- return CachedHfTokenizer.from_pretrained(
41
- path_or_repo_id,
42
- *args,
43
- trust_remote_code=trust_remote_code,
44
- revision=revision,
45
- download_dir=download_dir,
46
- **kwargs,
29
+ try:
30
+ installed = version("fastokens")
31
+ except PackageNotFoundError:
32
+ installed = None
33
+ if installed is None or Version(installed) < Version(_MIN_FASTOKENS_VERSION):
34
+ raise ImportError(
35
+ f"fastokens >= {_MIN_FASTOKENS_VERSION} is required when "
36
+ f"VLLM_USE_FASTOKENS=1 (found {installed or 'unknown'})."
37
)
38
+
39
+ fastokens.patch_transformers()
vllm/tokenizers/registry.py
+7
-1
@@ -43,7 +43,6 @@ _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS: set[str] = {"step3_vl"}
43
_VLLM_TOKENIZERS = {
44
"deepseek_v32": ("deepseek_v32", "DeepseekV32Tokenizer"),
45
"deepseek_v4": ("deepseek_v4", "DeepseekV4Tokenizer"),
46
- "fastokens": ("fastokens", "FastokensTokenizer"),
46
"grok2": ("grok2", "Grok2Tokenizer"),
47
"hf": ("hf", "CachedHfTokenizer"),
48
"kimi_audio": ("kimi_audio", "KimiAudioTokenizer"),
@@ -204,6 +203,13 @@ def get_tokenizer(
203
**kwargs,
204
) -> _T:
205
"""Gets a tokenizer for the given model name via HuggingFace or ModelScope."""
206
+ if envs.VLLM_USE_FASTOKENS:
207
+ # Process-global, idempotent patch that swaps the Rust BPE backend
208
+ # of any HF fast tokenizer loaded afterwards. No-op for non-HF modes.
209
+ from .fastokens import apply_fastokens_patch
210
+
211
+ apply_fastokens_patch()
212
+
213
tokenizer_mode, tokenizer_name, args, kwargs = cached_resolve_tokenizer_args(
214
tokenizer_name,
215
*args,