@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 116 lines 5.48 KB
Rendered Raw
1 # Token Classification Usages
2
3 ## Summary
4
5 - Model Usage: token classification
6 - Pooling Tasks: `token_classify`
7 - Offline APIs:
8 - `LLM.encode(..., pooling_task="token_classify")`
9 - Online APIs:
10 - Pooling API (`/pooling`)
11
12 The key distinction between (sequence) classification and token classification lies in their output granularity: (sequence) classification produces a single result for an entire input sequence, whereas token classification yields a result for each individual token within the sequence.
13
14 Many classification models support both (sequence) classification and token classification. For further details on (sequence) classification, please refer to [this page](classify.md).
15
16 !!! note
17
18 Pooling multitask support has been removed since v0.21. When the default pooling task (classify) is not
19 what you want, you need to manually specify it via `PoolerConfig(task="token_classify")` offline or
20 `--pooler-config.task token_classify` online.
21
22 ## Typical Use Cases
23
24 ### Named Entity Recognition (NER)
25
26 For implementation examples, see:
27
28 Offline: [examples/pooling/token_classify/ner_offline.py](../../../examples/pooling/token_classify/ner_offline.py)
29
30 Online: [examples/pooling/token_classify/ner_online.py](../../../examples/pooling/token_classify/ner_online.py)
31
32 ### Forced Alignment
33
34 Forced alignment takes audio and reference text as input and produces word-level timestamps.
35
36 Offline: [examples/pooling/token_classify/forced_alignment_offline.py](../../../examples/pooling/token_classify/forced_alignment_offline.py)
37
38 ### Sparse retrieval (lexical matching)
39
40 The BAAI/bge-m3 model leverages token classification for sparse retrieval. For more information, see [this page](specific_models.md#baaibge-m3).
41
42 ## Supported Models
43
44 | Architecture | Models | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
45 | ------------ | ------ | ----------------- | --------------------------- | --------------------------------------- |
46 | `BertForTokenClassification` | bert-based | `boltuix/NeuroBERT-NER` (see note), etc. | | |
47 | `ModernBertForTokenClassification` | ModernBERT-based | `disham993/electrical-ner-ModernBERT-base` | | |
48 | `OpenAIPrivacyFilterForTokenClassification` | gpt-oss-based encoder | `openai/privacy-filter` | | |
49 | `Qwen3ForTokenClassification`<sup>C</sup> | Qwen3-based | `bd2lcco/Qwen3-0.6B-finetuned` | | |
50 | `RobertaForTokenClassification` | RoBERTa-based | `Jean-Baptiste/roberta-large-ner-english` | | |
51 | `XLMRobertaForTokenClassification` | XLM-RoBERTa-based | `Davlan/xlm-roberta-base-ner-hrl` | | |
52 | `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* |
53
54 <sup>C</sup> Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))
55 \* Feature support is the same as that of the original model.
56
57 If your model is not in the above list, we will try to automatically convert the model using
58 [as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model]. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.
59
60 ### Multimodal Models
61
62 !!! note
63 For more information about multimodal models inputs, see [this page](../supported_models.md#list-of-multimodal-language-models).
64
65 | Architecture | Models | Inputs | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
66 | --------------------------------------------- | ------------------- | ----------------- | ------------------------------------------ | ------------------------------ | ------------------------------------------ |
67 | `Qwen3ASRForcedAlignerForTokenClassification` | Qwen3-ForcedAligner | T + A<sup>+</sup> | `Qwen/Qwen3-ForcedAligner-0.6B` (see note) | | ✅︎ |
68
69 !!! note
70 Forced alignment usage requires `--hf-overrides '{"architectures": ["Qwen3ASRForcedAlignerForTokenClassification"]}'`.
71 Please refer to [examples/pooling/token_classify/forced_alignment_offline.py](../../../examples/pooling/token_classify/forced_alignment_offline.py).
72
73 ### Reward Models
74
75 Using token classification models as reward models. For details on reward models, see [Reward Models](reward.md).
76
77 --8<-- "docs/models/pooling_models/reward.md:supported-token-reward-models"
78
79 ## Offline Inference
80
81 ### Pooling Parameters
82
83 The following [pooling parameters][vllm.PoolingParams] are supported.
84
85 ```python
86 --8<-- "vllm/pooling_params.py:common-pooling-params"
87 --8<-- "vllm/pooling_params.py:classify-pooling-params"
88 ```
89
90 ### `LLM.encode`
91
92 The [encode][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.encode] method is available to all pooling models in vLLM.
93
94 Set `pooling_task="token_classify"` when using `LLM.encode` for token classification Models:
95
96 ```python
97 from vllm import LLM
98
99 llm = LLM(model="boltuix/NeuroBERT-NER", runner="pooling")
100 (output,) = llm.encode("Hello, my name is", pooling_task="token_classify")
101
102 data = output.outputs.data
103 print(f"Data: {data!r}")
104 ```
105
106 ## Online Serving
107
108 Please refer to the [Pooling API](README.md#pooling-api) and use `"task":"token_classify"`.
109
110 ## More examples
111
112 More examples can be found here: [examples/pooling/token_classify](../../../examples/pooling/token_classify)
113
114 ## Supported Features
115
116 Token classification features should be consistent with (sequence) classification. For more information, see [this page](classify.md#supported-features).