A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 14, 05:30 UTC
[Doc] Update pooling model docs (#22186)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Aug 4, 2025 at 18:37 UTC
1539ced93a1ac3a78bef57d362cb9707c52f2a29
2 files changed
+43
-28
docs/models/pooling_models.md
+1
-1
@@ -120,7 +120,7 @@ A code example can be found here: <gh-file:examples/offline_inference/basic/clas
120
### `LLM.score`
121
122
The [score][vllm.LLM.score] method outputs similarity scores between sentence pairs.
123
-It is designed for embedding models and cross encoder models. Embedding models use cosine similarity, and [cross-encoder models](https://www.sbert.net/examples/applications/cross-encoder/README.html) serve as rerankers between candidate query-document pairs in RAG systems.
123
+It is designed for embedding models and cross-encoder models. Embedding models use cosine similarity, and [cross-encoder models](https://www.sbert.net/examples/applications/cross-encoder/README.html) serve as rerankers between candidate query-document pairs in RAG systems.
124
125
!!! note
126
vLLM can only perform the model inference component (e.g. embedding, reranking) of RAG.
docs/models/supported_models.md
+42
-27
@@ -311,6 +311,8 @@ See [this page](generative_models.md) for more information on how to use generat
311
312
#### Text Generation
313
314
+These models primarily accept the [`LLM.generate`](./generative_models.md#llmgenerate) API. Chat/Instruct models additionally support the [`LLM.chat`](./generative_models.md#llmchat) API.
315
+
316
<style>
317
th {
318
white-space: nowrap;
@@ -419,7 +421,9 @@ See [this page](./pooling_models.md) for more information on how to use pooling
421
Since some model architectures support both generative and pooling tasks,
422
you should explicitly specify `--runner pooling` to ensure that the model is used in pooling mode instead of generative mode.
423
422
-#### Text Embedding
424
+#### Embedding
425
+
426
+These models primarily support the [`LLM.embed`](./pooling_models.md#llmembed) API.
427
428
| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/distributed_serving.md) | [V1](gh-issue:8779) |
429
|--------------|--------|-------------------|----------------------|---------------------------|---------------------|
@@ -457,28 +461,10 @@ If your model is not in the above list, we will try to automatically convert the
461
[as_embedding_model][vllm.model_executor.models.adapters.as_embedding_model]. By default, the embeddings
462
of the whole prompt are extracted from the normalized hidden state corresponding to the last token.
463
460
-#### Reward Modeling
461
-
462
-| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/distributed_serving.md) | [V1](gh-issue:8779) |
463
-|--------------|--------|-------------------|----------------------|---------------------------|---------------------|
464
-| `InternLM2ForRewardModel` | InternLM2-based | `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc. | ✅︎ | ✅︎ | ✅︎ |
465
-| `LlamaForCausalLM`<sup>C</sup> | Llama-based | `peiyi9979/math-shepherd-mistral-7b-prm`, etc. | ✅︎ | ✅︎ | ✅︎ |
466
-| `Qwen2ForRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-RM-72B`, etc. | ✅︎ | ✅︎ | ✅︎ |
467
-| `Qwen2ForProcessRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-PRM-7B`, etc. | ✅︎ | ✅︎ | ✅︎ |
468
-| `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* | \* |
469
-
470
-<sup>C</sup> Automatically converted into a reward model via `--convert reward`. ([details](./pooling_models.md#model-conversion))
471
-\* Feature support is the same as that of the original model.
472
-
473
-If your model is not in the above list, we will try to automatically convert the model using
474
-[as_reward_model][vllm.model_executor.models.adapters.as_reward_model]. By default, we return the hidden states of each token directly.
475
-
476
-!!! important
477
- For process-supervised reward models such as `peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
478
- e.g.: `--override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.
479
-
464
#### Classification
465
466
+These models primarily support the [`LLM.classify`](./pooling_models.md#llmclassify) API.
467
+
468
| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/distributed_serving.md) | [V1](gh-issue:8779) |
469
|--------------|--------|-------------------|----------------------|---------------------------|---------------------|
470
| `JambaForSequenceClassification` | Jamba | `ai21labs/Jamba-tiny-reward-dev`, etc. | ✅︎ | ✅︎ | |
@@ -491,7 +477,10 @@ If your model is not in the above list, we will try to automatically convert the
477
If your model is not in the above list, we will try to automatically convert the model using
478
[as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model]. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.
479
494
-#### Sentence Pair Scoring
480
+#### Cross-encoder / Reranker
481
+
482
+Cross-encoder and reranker models are a subset of classification models that accept two prompts as input.
483
+These models primarily support the [`LLM.score`](./pooling_models.md#llmscore) API.
484
485
| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/distributed_serving.md) | [V1](gh-issue:8779) |
486
|--------------|--------|-------------------|----------------------|---------------------------|---------------------|
@@ -501,6 +490,7 @@ If your model is not in the above list, we will try to automatically convert the
490
| `Qwen3ForSequenceClassification` | Qwen3-based | `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `Qwen/Qwen3-Reranker-0.6B` (see note), etc. | ✅︎ | ✅︎ | ✅︎ |
491
| `RobertaForSequenceClassification` | RoBERTa-based | `cross-encoder/quora-roberta-base`, etc. | | | |
492
| `XLMRobertaForSequenceClassification` | XLM-RoBERTa-based | `BAAI/bge-reranker-v2-m3`, etc. | | | |
493
+| `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* | \* |
494
495
<sup>C</sup> Automatically converted into a classification model via `--convert classify`. ([details](./pooling_models.md#model-conversion))
496
\* Feature support is the same as that of the original model.
@@ -526,6 +516,28 @@ If your model is not in the above list, we will try to automatically convert the
516
vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'
517
```
518
519
+#### Reward Modeling
520
+
521
+These models primarily support the [`LLM.reward`](./pooling_models.md#llmreward) API.
522
+
523
+| Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/distributed_serving.md) | [V1](gh-issue:8779) |
524
+|--------------|--------|-------------------|----------------------|---------------------------|---------------------|
525
+| `InternLM2ForRewardModel` | InternLM2-based | `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc. | ✅︎ | ✅︎ | ✅︎ |
526
+| `LlamaForCausalLM`<sup>C</sup> | Llama-based | `peiyi9979/math-shepherd-mistral-7b-prm`, etc. | ✅︎ | ✅︎ | ✅︎ |
527
+| `Qwen2ForRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-RM-72B`, etc. | ✅︎ | ✅︎ | ✅︎ |
528
+| `Qwen2ForProcessRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-PRM-7B`, etc. | ✅︎ | ✅︎ | ✅︎ |
529
+| `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* | \* |
530
+
531
+<sup>C</sup> Automatically converted into a reward model via `--convert reward`. ([details](./pooling_models.md#model-conversion))
532
+\* Feature support is the same as that of the original model.
533
+
534
+If your model is not in the above list, we will try to automatically convert the model using
535
+[as_reward_model][vllm.model_executor.models.adapters.as_reward_model]. By default, we return the hidden states of each token directly.
536
+
537
+!!! important
538
+ For process-supervised reward models such as `peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
539
+ e.g.: `--override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.
540
+
541
[](){ #supported-mm-models }
542
543
## List of Multimodal Language Models
@@ -579,6 +591,8 @@ See [this page](generative_models.md) for more information on how to use generat
591
592
#### Text Generation
593
594
+These models primarily accept the [`LLM.generate`](./generative_models.md#llmgenerate) API. Chat/Instruct models additionally support the [`LLM.chat`](./generative_models.md#llmchat) API.
595
+
596
| Architecture | Models | Inputs | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/distributed_serving.md) | [V1](gh-issue:8779) |
597
|--------------|--------|--------|-------------------|----------------------|---------------------------|---------------------|
598
| `AriaForConditionalGeneration` | Aria | T + I<sup>+</sup> | `rhymes-ai/Aria` | | | ✅︎ |
@@ -720,11 +734,9 @@ Speech2Text models trained specifically for Automatic Speech Recognition.
734
735
See [this page](./pooling_models.md) for more information on how to use pooling models.
736
723
-!!! important
724
- Since some model architectures support both generative and pooling tasks,
725
- you should explicitly specify `--runner pooling` to ensure that the model is used in pooling mode instead of generative mode.
737
+#### Embedding
738
727
-#### Text Embedding
739
+These models primarily support the [`LLM.embed`](./pooling_models.md#llmembed) API.
740
741
!!! note
742
To get the best results, you should use pooling models that are specifically trained as such.
@@ -742,7 +754,10 @@ The following table lists those that are tested in vLLM.
754
755
---
756
745
-#### Scoring
757
+#### Cross-encoder / Reranker
758
+
759
+Cross-encoder and reranker models are a subset of classification models that accept two prompts as input.
760
+These models primarily support the [`LLM.score`](./pooling_models.md#llmscore) API.
761
762
| Architecture | Models | Inputs | Example HF Models | [LoRA][lora-adapter] | [PP][distributed-serving] | [V1](gh-issue:8779) |
763
|-------------------------------------|--------------------|----------|--------------------------|------------------------|-----------------------------|-----------------------|