@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 86ae693f20

Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 14, 23:45 UTC

[Deprecation][2/N] Replace `--task` with `--runner` and `--convert` (#21470)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Cyrus Leung committed Jul 28, 2025 at 10:42 UTC 86ae693f207fc9433f0b6d2c4331ef021dad50fe

94 files changed +1111 -1077

docs/features/multimodal_inputs.md

+2 -2

@@ -343,7 +343,7 @@ Here is a simple example using Phi-3.5-Vision.

343 First, launch the OpenAI-compatible server:

344

345 ```bash

346 -vllm serve microsoft/Phi-3.5-vision-instruct --task generate \

346 +vllm serve microsoft/Phi-3.5-vision-instruct --runner generate \

347 --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt '{"image":2}'

348 ```

349

@@ -422,7 +422,7 @@ Instead of `image_url`, you can pass a video file via `video_url`. Here is a sim

422 First, launch the OpenAI-compatible server:

423

424 ```bash

425 -vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --task generate --max-model-len 8192

425 +vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --runner generate --max-model-len 8192

426 ```

427

428 Then, you can use the OpenAI client as follows:

docs/features/prompt_embeds.md

+1 -1

@@ -34,7 +34,7 @@ Prompt embeddings are passed in as base64 encoded torch tensors.

34 First, launch the OpenAI-compatible server:

36 ```bash

37 -vllm serve meta-llama/Llama-3.2-1B-Instruct --task generate \

37 +vllm serve meta-llama/Llama-3.2-1B-Instruct --runner generate \

38 --max-model-len 4096 --enable-prompt-embeds

39 ```

docs/models/generative_models.md

+10 -3

@@ -2,12 +2,19 @@

3 vLLM provides first-class support for generative models, which covers most of LLMs.

5 -In vLLM, generative models implement the [VllmModelForTextGeneration][vllm.model_executor.models.VllmModelForTextGeneration] interface.

5 +In vLLM, generative models implement the[VllmModelForTextGeneration][vllm.model_executor.models.VllmModelForTextGeneration] interface.

6 Based on the final hidden states of the input, these models output log probabilities of the tokens to generate,

7 which are then passed through [Sampler][vllm.model_executor.layers.Sampler] to obtain the final text.

9 -For generative models, the only supported `--task` option is `"generate"`.

10 -Usually, this is automatically inferred so you don't have to specify it.

9 +## Configuration

10 +

11 +### Model Runner (`--runner`)

12 +

13 +Run a model in generation mode via the option `--runner generate`.

14 +

15 +!!! tip

16 + There is no need to set this option in the vast majority of cases as vLLM can automatically

17 + detect the model runner to use via `--runner auto`.

19 ## Offline Inference

docs/models/pooling_models.md

+49 -28

@@ -1,9 +1,9 @@

1 # Pooling Models

3 -vLLM also supports pooling models, including embedding, reranking and reward models.

3 +vLLM also supports pooling models, such as embedding, classification and reward models.

5 In vLLM, pooling models implement the [VllmModelForPooling][vllm.model_executor.models.VllmModelForPooling] interface.

6 -These models use a [Pooler][vllm.model_executor.layers.Pooler] to extract the final hidden states of the input

6 +These models use a [Pooler][vllm.model_executor.layers.pooler.Pooler] to extract the final hidden states of the input

7 before returning them.

9 !!! note

@@ -11,18 +11,39 @@ before returning them.

11 As shown in the [Compatibility Matrix](../features/compatibility_matrix.md), most vLLM features are not applicable to

12 pooling models as they only work on the generation or decode stage, so performance may not improve as much.

14 -If the model doesn't implement this interface, you can set `--task` which tells vLLM

15 -to convert the model into a pooling model.

14 +## Configuration

17 -| `--task` | Model type | Supported pooling tasks |

18 -|------------|----------------------|-------------------------------|

19 -| `embed` | Embedding model | `encode`, `embed` |

20 -| `classify` | Classification model | `encode`, `classify`, `score` |

21 -| `reward` | Reward model | `encode` |

16 +### Model Runner

23 -## Pooling Tasks

18 +Run a model in pooling mode via the option `--runner pooling`.

25 -In vLLM, we define the following pooling tasks and corresponding APIs:

20 +!!! tip

21 + There is no need to set this option in the vast majority of cases as vLLM can automatically

22 + detect the model runner to use via `--runner auto`.

23 +

24 +### Model Conversion

25 +

26 +vLLM can adapt models for various pooling tasks via the option `--convert <type>`.

27 +

28 +If `--runner pooling` has been set (manually or automatically) but the model does not implement the

29 +[VllmModelForPooling][vllm.model_executor.models.VllmModelForPooling] interface,

30 +vLLM will attempt to automatically convert the model according to the architecture names

31 +shown in the table below.

32 +

33 +| Architecture | `--convert` | Supported pooling tasks |

34 +|-------------------------------------------------|-------------|-------------------------------|

35 +| `*ForTextEncoding`, `*EmbeddingModel`, `*Model` | `embed` | `encode`, `embed` |

36 +| `*For*Classification`, `*ClassificationModel` | `classify` | `encode`, `classify`, `score` |

37 +| `*ForRewardModeling`, `*RewardModel` | `reward` | `encode` |

38 +

39 +!!! tip

40 + You can explicitly set `--convert <type>` to specify how to convert the model.

41 +

42 +### Pooling Tasks

43 +

44 +Each pooling model in vLLM supports one or more of these tasks according to

45 +[Pooler.get_supported_tasks][vllm.model_executor.layers.pooler.Pooler.get_supported_tasks],

46 +enabling the corresponding APIs:

48 | Task | APIs |

49 |------------|--------------------|

@@ -31,11 +52,19 @@ In vLLM, we define the following pooling tasks and corresponding APIs:

52 | `classify` | `classify` |

53 | `score` | `score` |

34 -\*The `score` API falls back to `embed` task if the model does not support `score` task.

55 +\* The `score` API falls back to `embed` task if the model does not support `score` task.

36 -Each pooling model in vLLM supports one or more of these tasks according to [Pooler.get_supported_tasks][vllm.model_executor.layers.Pooler.get_supported_tasks].

57 +### Pooler Configuration

38 -By default, the pooler assigned to each task has the following attributes:

59 +#### Predefined models

60 +

61 +If the [Pooler][vllm.model_executor.layers.pooler.Pooler] defined by the model accepts `pooler_config`,

62 +you can override some of its attributes via the `--override-pooler-config` option.

63 +

64 +#### Converted models

65 +

66 +If the model has been converted via `--convert` (see above),

67 +the pooler assigned to each task has the following attributes by default:

70 |------------|----------------|---------------|---------|

@@ -43,20 +72,12 @@ By default, the pooler assigned to each task has the following attributes:

72 | `embed` | `LAST` | ✅︎ | ❌ |

73 | `classify` | `LAST` | ❌ | ✅︎ |

46 -These defaults may be overridden by the model's implementation in vLLM.

47 -

75 When loading [Sentence Transformers](https://huggingface.co/sentence-transformers) models,

49 -we attempt to override the defaults based on its Sentence Transformers configuration file (`modules.json`),

50 -which takes priority over the model's defaults.

76 +its Sentence Transformers configuration file (`modules.json`) takes priority over the model's defaults.

78 You can further customize this via the `--override-pooler-config` option,

79 which takes priority over both the model's and Sentence Transformers's defaults.

55 -!!! note

56 -

57 - The above configuration may be disregarded if the model's implementation in vLLM defines its own pooler

58 - that is not based on [PoolerConfig][vllm.config.PoolerConfig].

59 -

81 ## Offline Inference

83 The [LLM][vllm.LLM] class provides various methods for offline inference.

@@ -70,7 +91,7 @@ It returns the extracted hidden states directly, which is useful for reward mode

91 ```python

92 from vllm import LLM

73 -llm = LLM(model="Qwen/Qwen2.5-Math-RM-72B", task="reward")

94 +llm = LLM(model="Qwen/Qwen2.5-Math-RM-72B", runner="pooling")

95 (output,) = llm.encode("Hello, my name is")

97 data = output.outputs.data

@@ -85,7 +106,7 @@ It is primarily designed for embedding models.

106 ```python

107 from vllm import LLM

108

88 -llm = LLM(model="intfloat/e5-mistral-7b-instruct", task="embed")

109 +llm = LLM(model="intfloat/e5-mistral-7b-instruct", runner="pooling")

110 (output,) = llm.embed("Hello, my name is")

111

112 embeds = output.outputs.embedding

@@ -102,7 +123,7 @@ It is primarily designed for classification models.

123 ```python

124 from vllm import LLM

125

105 -llm = LLM(model="jason9693/Qwen2.5-1.5B-apeach", task="classify")

126 +llm = LLM(model="jason9693/Qwen2.5-1.5B-apeach", runner="pooling")

127 (output,) = llm.classify("Hello, my name is")

128

129 probs = output.outputs.probs

@@ -123,7 +144,7 @@ It is designed for embedding models and cross encoder models. Embedding models u

144 ```python

145 from vllm import LLM

146

126 -llm = LLM(model="BAAI/bge-reranker-v2-m3", task="score")

147 +llm = LLM(model="BAAI/bge-reranker-v2-m3", runner="pooling")

148 (output,) = llm.score("What is the capital of France?",

149 "The capital of Brazil is Brasilia.")

150

@@ -175,7 +196,7 @@ You can change the output dimensions of embedding models that support Matryoshka

196 from vllm import LLM, PoolingParams

197

198 llm = LLM(model="jinaai/jina-embeddings-v3",

178 - task="embed",

199 + runner="pooling",

200 trust_remote_code=True)

201 outputs = llm.embed(["Follow the white rabbit."],

202 pooling_params=PoolingParams(dimensions=32))

docs/models/supported_models.md

+51 -50

@@ -1,7 +1,6 @@

1 # Supported Models

3 vLLM supports [generative](./generative_models.md) and [pooling](./pooling_models.md) models across various tasks.

4 -If a model supports more than one task, you can set the task via the `--task` argument.

5 For each task, we list the model architectures that have been implemented in vLLM.

6 Alongside each architecture, we include some popular models that use it.

@@ -24,7 +23,7 @@ To check if the modeling backend is Transformers, you can simply do this:

24 ```python

25 from vllm import LLM

27 -llm = LLM(model=..., task="generate") # Name or path of your model

26 +llm = LLM(model=...) # Name or path of your model

27 llm.apply_model(lambda model: print(type(model)))

28 ```

@@ -158,13 +157,13 @@ The [Transformers backend][transformers-backend] enables you to run models direc

157 ```python

158 from vllm import LLM

159

161 - # For generative models (task=generate) only

162 - llm = LLM(model=..., task="generate") # Name or path of your model

160 + # For generative models (runner=generate) only

161 + llm = LLM(model=..., runner="generate") # Name or path of your model

162 output = llm.generate("Hello, my name is")

163 print(output)

164

166 - # For pooling models (task={embed,classify,reward,score}) only

167 - llm = LLM(model=..., task="embed") # Name or path of your model

165 + # For pooling models (runner=pooling) only

166 + llm = LLM(model=..., runner="pooling") # Name or path of your model

167 output = llm.encode("Hello, my name is")

168 print(output)

169 ```

@@ -281,13 +280,13 @@ And use with `trust_remote_code=True`.

280 ```python

281 from vllm import LLM

282

284 -llm = LLM(model=..., revision=..., task=..., trust_remote_code=True)

283 +llm = LLM(model=..., revision=..., runner=..., trust_remote_code=True)

284

286 -# For generative models (task=generate) only

285 +# For generative models (runner=generate) only

286 output = llm.generate("Hello, my name is")

287 print(output)

288

290 -# For pooling models (task={embed,classify,reward,score}) only

289 +# For pooling models (runner=pooling) only

290 output = llm.encode("Hello, my name is")

291 print(output)

292 ```

@@ -312,8 +311,6 @@ See [this page](generative_models.md) for more information on how to use generat

311

312 #### Text Generation

313

315 -Specified using `--task generate`.

316 -

314 <style>

315 th {

316 white-space: nowrap;

@@ -420,25 +417,27 @@ See [this page](./pooling_models.md) for more information on how to use pooling

417

418 !!! important

419 Since some model architectures support both generative and pooling tasks,

423 - you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.

420 + you should explicitly specify `--runner pooling` to ensure that the model is used in pooling mode instead of generative mode.

421

422 #### Text Embedding

423

427 -Specified using `--task embed`.

428 -

425 |--------------|--------|-------------------|----------------------|---------------------------|---------------------|

432 -| `Gemma2Model` | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | | ✅︎ |

427 +| `Gemma2Model`C | Gemma 2-based | `BAAI/bge-multilingual-gemma2`, etc. | ✅︎ | | ✅︎ |

428 | `GritLM` | GritLM | `parasail-ai/GritLM-7B-vllm`. | ✅︎ | ✅︎ | |

438 -| `LlamaModel`, `LlamaForCausalLM`, `MistralModel`, etc. | Llama-based | `intfloat/e5-mistral-7b-instruct`, etc. | ✅︎ | ✅︎ | ✅︎ |

439 -| `Qwen2Model`, `Qwen2ForCausalLM` | Qwen2-based | `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc. | ✅︎ | ✅︎ | ✅︎ |

440 -| `Qwen3Model`, `Qwen3ForCausalLM` | Qwen3-based | `Qwen/Qwen3-Embedding-0.6B`, etc. | ✅︎ | ✅︎ | ✅︎ |

433 +| `LlamaModel`C, `LlamaForCausalLM`C, `MistralModel`C, etc. | Llama-based | `intfloat/e5-mistral-7b-instruct`, etc. | ✅︎ | ✅︎ | ✅︎ |

434 +| `Qwen2Model`C, `Qwen2ForCausalLM`C | Qwen2-based | `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc. | ✅︎ | ✅︎ | ✅︎ |

435 +| `Qwen3Model`C, `Qwen3ForCausalLM`C | Qwen3-based | `Qwen/Qwen3-Embedding-0.6B`, etc. | ✅︎ | ✅︎ | ✅︎ |

437 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* | \* |

438 +

439 +C Automatically converted into an embedding model via `--convert embed`. ([details](./pooling_models.md#model-conversion))

440 +\* Feature support is the same as that of the original model.

441

442 !!! note

443 `ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.

@@ -460,14 +459,16 @@ of the whole prompt are extracted from the normalized hidden state corresponding

459

460 #### Reward Modeling

461

463 -Specified using `--task reward`.

464 -

463 |--------------|--------|-------------------|----------------------|---------------------------|---------------------|

464 | `InternLM2ForRewardModel` | InternLM2-based | `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc. | ✅︎ | ✅︎ | ✅︎ |

468 -| `LlamaForCausalLM` | Llama-based | `peiyi9979/math-shepherd-mistral-7b-prm`, etc. | ✅︎ | ✅︎ | ✅︎ |

465 +| `LlamaForCausalLM`C | Llama-based | `peiyi9979/math-shepherd-mistral-7b-prm`, etc. | ✅︎ | ✅︎ | ✅︎ |

466 | `Qwen2ForRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-RM-72B`, etc. | ✅︎ | ✅︎ | ✅︎ |

467 | `Qwen2ForProcessRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-PRM-7B`, etc. | ✅︎ | ✅︎ | ✅︎ |

468 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* | \* |

469 +

470 +C Automatically converted into a reward model via `--convert reward`. ([details](./pooling_models.md#model-conversion))

471 +\* Feature support is the same as that of the original model.

472

473 If your model is not in the above list, we will try to automatically convert the model using

474 [as_reward_model][vllm.model_executor.models.adapters.as_reward_model]. By default, we return the hidden states of each token directly.

@@ -478,28 +479,31 @@ If your model is not in the above list, we will try to automatically convert the

479

480 #### Classification

481

481 -Specified using `--task classify`.

482 -

483 |--------------|--------|-------------------|----------------------|---------------------------|---------------------|

484 | `JambaForSequenceClassification` | Jamba | `ai21labs/Jamba-tiny-reward-dev`, etc. | ✅︎ | ✅︎ | |

485 | `GPT2ForSequenceClassification` | GPT2 | `nie3e/sentiment-polish-gpt2-small` | | | ✅︎ |

486 +| `*Model`C, `*ForCausalLM`C, etc. | Generative models | N/A | \* | \* | \* |

487 +

488 +C Automatically converted into a classification model via `--convert classify`. ([details](./pooling_models.md#model-conversion))

489 +\* Feature support is the same as that of the original model.

490

491 If your model is not in the above list, we will try to automatically convert the model using

492 [as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model]. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.

493

494 #### Sentence Pair Scoring

495

493 -Specified using `--task score`.

497 +|--------------|--------|-------------------|----------------------|---------------------------|---------------------|

499 +| `GemmaForSequenceClassification` | Gemma-based | `BAAI/bge-reranker-v2-gemma` (see note), etc. | ✅︎ | ✅︎ | ✅︎ |

500 +| `Qwen2ForSequenceClassification` | Qwen2-based | `mixedbread-ai/mxbai-rerank-base-v2` (see note), etc. | ✅︎ | ✅︎ | ✅︎ |

501 +| `Qwen3ForSequenceClassification` | Qwen3-based | `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `Qwen/Qwen3-Reranker-0.6B` (see note), etc. | ✅︎ | ✅︎ | ✅︎ |

504

496 -|--------------|--------|-------------------|---------------------|

505 +C Automatically converted into a classification model via `--convert classify`. ([details](./pooling_models.md#model-conversion))

506 +\* Feature support is the same as that of the original model.

507

508 !!! note

509 Load the official original `BAAI/bge-reranker-v2-gemma` by using the following command.

@@ -575,8 +579,6 @@ See [this page](generative_models.md) for more information on how to use generat

579

580 #### Text Generation

581

578 -Specified using `--task generate`.

579 -

583 |--------------|--------|--------|-------------------|----------------------|---------------------------|---------------------|

584 | `AriaForConditionalGeneration` | Aria | T + I+ | `rhymes-ai/Aria` | | | ✅︎ |

@@ -705,8 +707,6 @@ Some models are supported only via the [Transformers backend](#transformers). Th

707

708 #### Transcription

709

708 -Specified using `--task transcription`.

709 -

710 Speech2Text models trained specifically for Automatic Speech Recognition.

711

@@ -719,14 +719,10 @@ See [this page](./pooling_models.md) for more information on how to use pooling

719

720 !!! important

721 Since some model architectures support both generative and pooling tasks,

722 - you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.

722 + you should explicitly specify `--runner pooling` to ensure that the model is used in pooling mode instead of generative mode.

723

724 #### Text Embedding

725

726 -Specified using `--task embed`.

727 -

728 -Any text generation model can be converted into an embedding model by passing `--task embed`.

729 -

726 !!! note

727 To get the best results, you should use pooling models that are specifically trained as such.

728

@@ -734,19 +730,24 @@ The following table lists those that are tested in vLLM.

730

732 |--------------|--------|--------|-------------------|----------------------|---------------------------|---------------------|

738 -| `Phi3VForCausalLM` | Phi-3-Vision-based | T + I | `TIGER-Lab/VLM2Vec-Full` | 🚧 | ✅︎ | |

734 +| `Phi3VForCausalLM`C | Phi-3-Vision-based | T + I | `TIGER-Lab/VLM2Vec-Full` | 🚧 | ✅︎ | |

735 +| `*ForConditionalGeneration`C, `*ForCausalLM`C, etc. | Generative models | \* | N/A | \* | \* | \* |

736 +

737 +C Automatically converted into an embedding model via `--convert embed`. ([details](./pooling_models.md#model-conversion))

738 +\* Feature support is the same as that of the original model.

739

740 ---

741

742 #### Scoring

743

744 -Specified using `--task score`.

745 -

745 |-------------------------------------|--------------------|----------|--------------------------|------------------------|-----------------------------|-----------------------|

746 | `JinaVLForSequenceClassification` | JinaVL-based | T + IE+ | `jinaai/jina-reranker-m0`, etc. | | | ✅︎ |

747

748 +C Automatically converted into a classification model via `--convert classify`. ([details](./pooling_models.md#model-conversion))

749 +\* Feature support is the same as that of the original model.

750 +

751 ## Model Support Policy

752

753 At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:

docs/serving/openai_compatible_server.md

+12 -12

@@ -45,17 +45,17 @@ To call the server, in your preferred text editor, create a script that uses an

45 We currently support the following OpenAI APIs:

47 - [Completions API][completions-api] (`/v1/completions`)

48 - - Only applicable to [text generation models](../models/generative_models.md) (`--task generate`).

48 + - Only applicable to [text generation models](../models/generative_models.md).

49 - *Note: `suffix` parameter is not supported.*

50 - [Chat Completions API][chat-api] (`/v1/chat/completions`)

51 - - Only applicable to [text generation models](../models/generative_models.md) (`--task generate`) with a [chat template][chat-template].

51 + - Only applicable to [text generation models](../models/generative_models.md) with a [chat template][chat-template].

52 - *Note: `parallel_tool_calls` and `user` parameters are ignored.*

53 - [Embeddings API][embeddings-api] (`/v1/embeddings`)

54 - - Only applicable to [embedding models](../models/pooling_models.md) (`--task embed`).

54 + - Only applicable to [embedding models](../models/pooling_models.md).

55 - [Transcriptions API][transcriptions-api] (`/v1/audio/transcriptions`)

56 - - Only applicable to Automatic Speech Recognition (ASR) models (OpenAI Whisper) (`--task generate`).

56 + - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).

57 - [Translation API][translations-api] (`/v1/audio/translations`)

58 - - Only applicable to Automatic Speech Recognition (ASR) models (OpenAI Whisper) (`--task generate`).

58 + - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).

60 In addition, we have the following custom APIs:

@@ -64,14 +64,14 @@ In addition, we have the following custom APIs:

64 - [Pooling API][pooling-api] (`/pooling`)

65 - Applicable to all [pooling models](../models/pooling_models.md).

66 - [Classification API][classification-api] (`/classify`)

67 - - Only applicable to [classification models](../models/pooling_models.md) (`--task classify`).

67 + - Only applicable to [classification models](../models/pooling_models.md).

68 - [Score API][score-api] (`/score`)

69 - - Applicable to embedding models and [cross-encoder models](../models/pooling_models.md) (`--task score`).

69 + - Applicable to [embedding models and cross-encoder models](../models/pooling_models.md).

70 - [Re-rank API][rerank-api] (`/rerank`, `/v1/rerank`, `/v2/rerank`)

71 - Implements [Jina AI's v1 re-rank API](https://jina.ai/reranker/)

72 - Also compatible with [Cohere's v1 & v2 re-rank APIs](https://docs.cohere.com/v2/reference/rerank)

73 - Jina and Cohere's APIs are very similar; Jina's includes extra information in the rerank endpoint's response.

74 - - Only applicable to [cross-encoder models](../models/pooling_models.md) (`--task score`).

74 + - Only applicable to [cross-encoder models](../models/pooling_models.md).

76 [](){ #chat-template }

@@ -250,14 +250,14 @@ and passing a list of `messages` in the request. Refer to the examples below for

250 To serve the model:

251

252 ```bash

253 - vllm serve TIGER-Lab/VLM2Vec-Full --task embed \

253 + vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \

254 --trust-remote-code \

255 --max-model-len 4096 \

256 --chat-template examples/template_vlm2vec.jinja

257 ```

258

259 !!! important

260 - Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--task embed`

260 + Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--runner pooling`

261 to run this model in embedding mode instead of text generation mode.

262

263 The custom chat template is completely different from the original one for this model,

@@ -296,14 +296,14 @@ and passing a list of `messages` in the request. Refer to the examples below for

296 To serve the model:

297

298 ```bash

299 - vllm serve MrLight/dse-qwen2-2b-mrl-v1 --task embed \

299 + vllm serve MrLight/dse-qwen2-2b-mrl-v1 --runner pooling \

300 --trust-remote-code \

301 --max-model-len 8192 \

302 --chat-template examples/template_dse_qwen2_vl.jinja

303 ```

304

305 !!! important

306 - Like with VLM2Vec, we have to explicitly pass `--task embed`.

306 + Like with VLM2Vec, we have to explicitly pass `--runner pooling`.

307

308 Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled

309 by a custom chat template: <gh-file:examples/template_dse_qwen2_vl.jinja>

examples/offline_inference/basic/classify.py

+4 -2

@@ -12,7 +12,9 @@ def parse_args():

12 parser = EngineArgs.add_cli_args(parser)

13 # Set example specific arguments

14 parser.set_defaults(

15 - model="jason9693/Qwen2.5-1.5B-apeach", task="classify", enforce_eager=True

15 + model="jason9693/Qwen2.5-1.5B-apeach",

16 + runner="pooling",

17 + enforce_eager=True,

18 )

19 return parser.parse_args()

@@ -27,7 +29,7 @@ def main(args: Namespace):

29 ]

31 # Create an LLM.

30 - # You should pass task="classify" for classification models

32 + # You should pass runner="pooling" for classification models

33 llm = LLM(**vars(args))

35 # Generate logits. The output is a list of ClassificationRequestOutputs.

examples/offline_inference/basic/embed.py

+2 -2

@@ -13,7 +13,7 @@ def parse_args():

13 # Set example specific arguments

14 parser.set_defaults(

15 model="intfloat/e5-mistral-7b-instruct",

16 - task="embed",

16 + runner="pooling",

17 enforce_eager=True,

18 max_model_len=1024,

19 )

@@ -30,7 +30,7 @@ def main(args: Namespace):

30 ]

32 # Create an LLM.

33 - # You should pass task="embed" for embedding models

33 + # You should pass runner="pooling" for embedding models

34 llm = LLM(**vars(args))

36 # Generate embedding. The output is a list of EmbeddingRequestOutputs.

examples/offline_inference/basic/score.py

+4 -2

@@ -12,7 +12,9 @@ def parse_args():

12 parser = EngineArgs.add_cli_args(parser)

13 # Set example specific arguments

14 parser.set_defaults(

15 - model="BAAI/bge-reranker-v2-m3", task="score", enforce_eager=True

15 + model="BAAI/bge-reranker-v2-m3",

16 + runner="pooling",

17 + enforce_eager=True,

18 )

19 return parser.parse_args()

@@ -26,7 +28,7 @@ def main(args: Namespace):

28 ]

30 # Create an LLM.

29 - # You should pass task="score" for cross-encoder models

31 + # You should pass runner="pooling" for cross-encoder models

32 llm = LLM(**vars(args))

34 # Generate scores. The output is a list of ScoringRequestOutputs.

examples/offline_inference/embed_jina_embeddings_v3.py

+4 -2

@@ -12,7 +12,9 @@ def parse_args():

12 parser = EngineArgs.add_cli_args(parser)

13 # Set example specific arguments

14 parser.set_defaults(

15 - model="jinaai/jina-embeddings-v3", task="embed", trust_remote_code=True

15 + model="jinaai/jina-embeddings-v3",

16 + runner="pooling",

17 + trust_remote_code=True,

18 )

19 return parser.parse_args()

@@ -29,7 +31,7 @@ def main(args: Namespace):

31 ]

33 # Create an LLM.

32 - # You should pass task="embed" for embedding models

34 + # You should pass runner="pooling" for embedding models

35 llm = LLM(**vars(args))

37 # Generate embedding. The output is a list of EmbeddingRequestOutputs.

examples/offline_inference/embed_matryoshka_fy.py

+4 -2

@@ -12,7 +12,9 @@ def parse_args():

12 parser = EngineArgs.add_cli_args(parser)

13 # Set example specific arguments

14 parser.set_defaults(

15 - model="jinaai/jina-embeddings-v3", task="embed", trust_remote_code=True

15 + model="jinaai/jina-embeddings-v3",

16 + runner="pooling",

17 + trust_remote_code=True,

18 )

19 return parser.parse_args()

@@ -29,7 +31,7 @@ def main(args: Namespace):

31 ]

33 # Create an LLM.

32 - # You should pass task="embed" for embedding models

34 + # You should pass runner="pooling" for embedding models

35 llm = LLM(**vars(args))

37 # Generate embedding. The output is a list of EmbeddingRequestOutputs.

examples/offline_inference/qwen3_reranker.py

+2 -2

@@ -17,7 +17,7 @@ model_name = "Qwen/Qwen3-Reranker-0.6B"

17 # Models converted offline using this method can not only be more efficient

18 # and support the vllm score API, but also make the init parameters more

19 # concise, for example.

20 -# llm = LLM(model="tomaarsen/Qwen3-Reranker-0.6B-seq-cls", task="score")

20 +# llm = LLM(model="tomaarsen/Qwen3-Reranker-0.6B-seq-cls", runner="pooling")

22 # If you want to load the official original version, the init parameters are

23 # as follows.

@@ -27,7 +27,7 @@ def get_llm() -> LLM:

27 """Initializes and returns the LLM model for Qwen3-Reranker."""

28 return LLM(

29 model=model_name,

30 - task="score",

30 + runner="pooling",

31 hf_overrides={

32 "architectures": ["Qwen3ForSequenceClassification"],

33 "classifier_from_token": ["no", "yes"],

examples/offline_inference/vision_language_pooling.py

+3 -3

@@ -70,7 +70,7 @@ def run_e5_v(query: Query) -> ModelRequestData:

71 engine_args = EngineArgs(

72 model="royokong/e5-v",

73 - task="embed",

73 + runner="pooling",

74 max_model_len=4096,

75 limit_mm_per_prompt={"image": 1},

76 )

@@ -102,7 +102,7 @@ def run_vlm2vec(query: Query) -> ModelRequestData:

102

103 engine_args = EngineArgs(

104 model="TIGER-Lab/VLM2Vec-Full",

105 - task="embed",

105 + runner="pooling",

106 max_model_len=4096,

107 trust_remote_code=True,

108 mm_processor_kwargs={"num_crops": 4},

@@ -122,7 +122,7 @@ def run_jinavl_reranker(query: Query) -> ModelRequestData:

122

123 engine_args = EngineArgs(

124 model="jinaai/jina-reranker-m0",

125 - task="score",

125 + runner="pooling",

126 max_model_len=32768,

127 trust_remote_code=True,

128 mm_processor_kwargs={

examples/online_serving/openai_chat_completion_client_for_multimodal.py

+1 -1

@@ -9,7 +9,7 @@ Launch the vLLM server with the following command:

9 vllm serve llava-hf/llava-1.5-7b-hf

11 (multi-image inference with Phi-3.5-vision-instruct)

12 -vllm serve microsoft/Phi-3.5-vision-instruct --task generate \

12 +vllm serve microsoft/Phi-3.5-vision-instruct --runner generate \

13 --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt '{"image":2}'

15 (audio inference with Ultravox)

examples/online_serving/openai_chat_embedding_client_for_multimodal.py

+1 -1

@@ -92,7 +92,7 @@ def dse_qwen2_vl(inp: dict):

92 def parse_args():

93 parser = argparse.ArgumentParser(

94 "Script to call a specified VLM through the API. Make sure to serve "

95 - "the model with --task embed before running this."

95 + "the model with `--runner pooling` before running this."

96 )

97 parser.add_argument(

98 "--model",

examples/online_serving/openai_cross_encoder_score.py

+1 -1

@@ -3,7 +3,7 @@

3 """

4 Example online usage of Score API.

6 -Run `vllm serve <model> --task score` to start up the server in vLLM.

6 +Run `vllm serve <model> --runner pooling` to start up the server in vLLM.

7 """

9 import argparse

examples/online_serving/openai_cross_encoder_score_for_multimodal.py

+1 -1

@@ -3,7 +3,7 @@

3 """

4 Example online usage of Score API.

6 -Run `vllm serve <model> --task score` to start up the server in vLLM.

6 +Run `vllm serve <model> --runner pooling` to start up the server in vLLM.

7 """

9 import argparse

examples/online_serving/openai_pooling_client.py

+1 -1

@@ -3,7 +3,7 @@

3 """

4 Example online usage of Pooling API.

6 -Run `vllm serve <model> --task <embed|classify|reward|score>`

6 +Run `vllm serve <model> --runner pooling`

7 to start up the server in vLLM.

8 """

examples/online_serving/prompt_embed_inference_with_openai_client.py

+1 -1

@@ -10,7 +10,7 @@ This script demonstrates how to:

11 Run the vLLM server first:

12 vllm serve meta-llama/Llama-3.2-1B-Instruct \

13 - --task generate \

13 + --runner generate \

14 --max-model-len 4096 \

15 --enable-prompt-embeds

tests/compile/test_async_tp.py

-3

@@ -148,9 +148,6 @@ def async_tp_pass_on_test_model(local_rank: int, world_size: int,

148 # in the vllm_config, it's not really used.

149 model_name = "nm-testing/TinyLlama-1.1B-Chat-v1.0-FP8-e2e"

150 vllm_config.model_config = ModelConfig(model=model_name,

151 - task="auto",

152 - tokenizer=model_name,

153 - tokenizer_mode="auto",

151 trust_remote_code=True,

152 dtype=dtype,

153 seed=42)

tests/compile/test_basic_correctness.py

+3 -3

@@ -62,8 +62,8 @@ class TestSetting:

62 TestSetting(

63 model="BAAI/bge-multilingual-gemma2",

64 model_args=[

65 - "--task", "embed", "--dtype", "bfloat16", "--max-model-len",

66 - "2048"

65 + "--runner", "pooling", "--dtype", "bfloat16",

66 + "--max-model-len", "2048"

67 ],

68 pp_size=1,

69 tp_size=1,

@@ -75,7 +75,7 @@ class TestSetting:

75 # # encoder-based embedding model (BERT)

76 # TestSetting(

77 # model="BAAI/bge-base-en-v1.5",

78 - # model_args=["--task", "embed"],

78 + # model_args=["--runner", "pooling"],

79 # pp_size=1,

80 # tp_size=1,

81 # attn_backend="XFORMERS",

tests/compile/test_fusion_all_reduce.py

-3

@@ -125,9 +125,6 @@ def all_reduce_fusion_pass_on_test_model(local_rank: int, world_size: int,

125 # in the vllm_config, it's not really used.

126 model_name = "nm-testing/TinyLlama-1.1B-Chat-v1.0-FP8-e2e"

127 vllm_config.model_config = ModelConfig(model=model_name,

128 - task="auto",

129 - tokenizer=model_name,

130 - tokenizer_mode="auto",

128 trust_remote_code=True,

129 dtype=dtype,

130 seed=42)

tests/compile/test_sequence_parallelism.py

-3

@@ -250,9 +250,6 @@ def sequence_parallelism_pass_on_test_model(

250 # in the vllm_config, it's not really used.

251 model_name = "nm-testing/TinyLlama-1.1B-Chat-v1.0-FP8-e2e"

252 vllm_config.model_config = ModelConfig(model=model_name,

253 - task="auto",

254 - tokenizer=model_name,

255 - tokenizer_mode="auto",

253 trust_remote_code=True,

254 dtype=dtype,

255 seed=42)

tests/conftest.py

+5 -3

@@ -23,7 +23,7 @@ from vllm import LLM, SamplingParams

23 from vllm.assets.audio import AudioAsset

24 from vllm.assets.image import ImageAsset

25 from vllm.assets.video import VideoAsset

26 -from vllm.config import TaskOption, _get_and_verify_dtype

26 +from vllm.config import ConvertOption, RunnerOption, _get_and_verify_dtype

27 from vllm.connections import global_http_connection

28 from vllm.distributed import (cleanup_dist_env_and_memory,

29 init_distributed_environment,

@@ -769,7 +769,8 @@ class VllmRunner:

769 def __init__(

770 self,

771 model_name: str,

772 - task: TaskOption = "auto",

772 + runner: RunnerOption = "auto",

773 + convert: ConvertOption = "auto",

774 tokenizer_name: Optional[str] = None,

775 tokenizer_mode: str = "auto",

776 trust_remote_code: bool = True,

@@ -786,7 +787,8 @@ class VllmRunner:

787 ) -> None:

788 self.llm = LLM(

789 model=model_name,

789 - task=task,

790 + runner=runner,

791 + convert=convert,

792 tokenizer=tokenizer_name,

793 tokenizer_mode=tokenizer_mode,

794 trust_remote_code=trust_remote_code,

tests/distributed/test_expert_parallel.py

+13 -13

@@ -6,7 +6,7 @@ from typing import Literal, NamedTuple, Optional

7 import pytest

9 -from vllm.config import TaskOption

9 +from vllm.config import RunnerOption

10 from vllm.logger import init_logger

12 from ..utils import compare_two_settings, create_new_process_for_each_test

@@ -31,14 +31,14 @@ class EPTestOptions(NamedTuple):

31 class EPTestSettings:

32 parallel_setups: list[ParallelSetup]

33 distributed_backends: list[str]

34 - task: TaskOption

34 + runner: RunnerOption

35 test_options: EPTestOptions

37 @staticmethod

38 def detailed(

39 *,

40 tp_base: int = 2,

41 - task: TaskOption = "auto",

41 + runner: RunnerOption = "auto",

42 trust_remote_code: bool = False,

43 tokenizer_mode: Optional[str] = None,

44 load_format: Optional[str] = None,

@@ -63,7 +63,7 @@ class EPTestSettings:

63 chunked_prefill=False),

64 ],

65 distributed_backends=["mp", "ray"],

66 - task=task,

66 + runner=runner,

67 test_options=EPTestOptions(trust_remote_code=trust_remote_code,

68 tokenizer_mode=tokenizer_mode,

69 load_format=load_format,

@@ -74,7 +74,7 @@ class EPTestSettings:

74 def fast(

75 *,

76 tp_base: int = 2,

77 - task: TaskOption = "auto",

77 + runner: RunnerOption = "auto",

78 trust_remote_code: bool = False,

79 tokenizer_mode: Optional[str] = None,

80 load_format: Optional[str] = None,

@@ -87,7 +87,7 @@ class EPTestSettings:

87 chunked_prefill=False),

88 ],

89 distributed_backends=["mp"],

90 - task=task,

90 + runner=runner,

91 test_options=EPTestOptions(trust_remote_code=trust_remote_code,

92 tokenizer_mode=tokenizer_mode,

93 load_format=load_format,

@@ -100,7 +100,7 @@ class EPTestSettings:

100 for parallel_setup in self.parallel_setups:

101 for distributed_backend in self.distributed_backends:

102 yield (model_name, parallel_setup, distributed_backend,

103 - self.task, opts)

103 + self.runner, opts)

104

105

106 # NOTE: You can adjust tp_base locally to fit the model in GPU

@@ -118,7 +118,7 @@ def _compare_tp(

118 model_name: str,

119 parallel_setup: ParallelSetup,

120 distributed_backend: str,

121 - task: TaskOption,

121 + runner: RunnerOption,

122 test_options: EPTestOptions,

123 num_gpus_available: int,

124 *,

@@ -154,8 +154,8 @@ def _compare_tp(

154 common_args.append("--enable-chunked-prefill")

155 if eager_mode:

156 common_args.append("--enforce-eager")

157 - if task != "auto":

158 - common_args.extend(["--task", task])

157 + if runner != "auto":

158 + common_args.extend(["--runner", runner])

159 if trust_remote_code:

160 common_args.append("--trust-remote-code")

161 if tokenizer_mode:

@@ -203,7 +203,7 @@ def _compare_tp(

203

204

205 @pytest.mark.parametrize(

206 - ("model_name", "parallel_setup", "distributed_backend", "task",

206 + ("model_name", "parallel_setup", "distributed_backend", "runner",

207 "test_options"),

208 [

209 params for model_name, settings in TEST_MODELS.items()

@@ -215,14 +215,14 @@ def test_ep(

215 model_name: str,

216 parallel_setup: ParallelSetup,

217 distributed_backend: str,

218 - task: TaskOption,

218 + runner: RunnerOption,

219 test_options: EPTestOptions,

220 num_gpus_available,

221 ):

222 _compare_tp(model_name,

223 parallel_setup,

224 distributed_backend,

225 - task,

225 + runner,

226 test_options,

227 num_gpus_available,

228 method="generate")

tests/distributed/test_pipeline_parallel.py

+22 -22

@@ -14,7 +14,7 @@ from typing import Literal, NamedTuple, Optional

15 import pytest

17 -from vllm.config import _FLOAT16_NOT_SUPPORTED_MODELS, TaskOption

17 +from vllm.config import _FLOAT16_NOT_SUPPORTED_MODELS, RunnerOption

18 from vllm.logger import init_logger

19 from vllm.transformers_utils.config import get_config

@@ -60,7 +60,7 @@ class PPTestSettings:

60 distributed_backends: list[str]

61 # vllm major version: "0" for V0, "1" for V1

62 vllm_major_versions: list[str]

63 - task: TaskOption

63 + runner: RunnerOption

64 test_options: PPTestOptions

66 def __post_init__(self):

@@ -76,7 +76,7 @@ class PPTestSettings:

76 tp_base: int = 1,

77 pp_base: int = 2,

78 multi_node_only: bool = False,

79 - task: TaskOption = "auto",

79 + runner: RunnerOption = "auto",

80 load_format: Optional[str] = None,

81 ):

82 return PPTestSettings(

@@ -104,7 +104,7 @@ class PPTestSettings:

104 ],

105 distributed_backends=["mp", "mp", "ray", "ray"],

106 vllm_major_versions=["0", "1", "0", "1"],

107 - task=task,

107 + runner=runner,

108 test_options=PPTestOptions(multi_node_only=multi_node_only,

109 load_format=load_format),

110 )

@@ -114,7 +114,7 @@ class PPTestSettings:

114 *,

115 tp_base: int = 1,

116 pp_base: int = 2,

117 - task: TaskOption = "auto",

117 + runner: RunnerOption = "auto",

118 multi_node_only: bool = False,

119 load_format: Optional[str] = None,

120 ):

@@ -127,7 +127,7 @@ class PPTestSettings:

127 ],

128 distributed_backends=["mp"],

129 vllm_major_versions=["0"],

130 - task=task,

130 + runner=runner,

131 test_options=PPTestOptions(multi_node_only=multi_node_only,

132 load_format=load_format),

133 )

@@ -139,7 +139,7 @@ class PPTestSettings:

139 for backend, vllm_major_version in zip(self.distributed_backends,

140 self.vllm_major_versions):

141 yield (model_id, parallel_setup, backend, vllm_major_version,

142 - self.task, opts)

142 + self.runner, opts)

143

144

145 # NOTE: You can adjust tp_base and/or pp_base locally to fit the model in GPU

@@ -211,10 +211,10 @@ TEXT_GENERATION_MODELS = {

211

212 EMBEDDING_MODELS = { # type: ignore[var-annotated]

213 # [Text-only]

214 - "intfloat/e5-mistral-7b-instruct": PPTestSettings.fast(task="embed"),

215 - "BAAI/bge-multilingual-gemma2": PPTestSettings.fast(task="embed"),

214 + "intfloat/e5-mistral-7b-instruct": PPTestSettings.fast(runner="pooling"),

215 + "BAAI/bge-multilingual-gemma2": PPTestSettings.fast(runner="pooling"),

216 "Qwen/Qwen2.5-Math-RM-72B": PPTestSettings.fast(

217 - load_format="dummy", task="embed"

217 + load_format="dummy", runner="pooling"

218 ),

219 }

220

@@ -269,7 +269,7 @@ def _compare_tp(

269 parallel_setup: ParallelSetup,

270 distributed_backend: str,

271 vllm_major_version: str,

272 - task: TaskOption,

272 + runner: RunnerOption,

273 test_options: PPTestOptions,

274 num_gpus_available: int,

275 *,

@@ -335,8 +335,8 @@ def _compare_tp(

335 common_args.append("--enable-chunked-prefill")

336 if eager_mode:

337 common_args.append("--enforce-eager")

338 - if task != "auto":

339 - common_args.extend(["--task", task])

338 + if runner != "auto":

339 + common_args.extend(["--runner", runner])

340 if trust_remote_code:

341 common_args.append("--trust-remote-code")

342 if tokenizer_mode:

@@ -415,7 +415,7 @@ def _compare_tp(

415

416 @pytest.mark.parametrize(

417 ("model_id", "parallel_setup", "distributed_backend", "vllm_major_version",

418 - "task", "test_options"),

418 + "runner", "test_options"),

419 [

420 params for model_id, settings in TEXT_GENERATION_MODELS.items()

421 for params in settings.iter_params(model_id) if model_id in TEST_MODELS

@@ -427,7 +427,7 @@ def test_tp_language_generation(

427 parallel_setup: ParallelSetup,

428 distributed_backend: str,

429 vllm_major_version: str,

430 - task: TaskOption,

430 + runner: RunnerOption,

431 test_options: PPTestOptions,

432 num_gpus_available,

433 ):

@@ -435,7 +435,7 @@ def test_tp_language_generation(

435 parallel_setup,

436 distributed_backend,

437 vllm_major_version,

438 - task,

438 + runner,

439 test_options,

440 num_gpus_available,

441 method="generate",

@@ -444,7 +444,7 @@ def test_tp_language_generation(

444

445 @pytest.mark.parametrize(

446 ("model_id", "parallel_setup", "distributed_backend", "vllm_major_version",

447 - "task", "test_options"),

447 + "runner", "test_options"),

448 [

449 params for model_id, settings in EMBEDDING_MODELS.items()

450 for params in settings.iter_params(model_id) if model_id in TEST_MODELS

@@ -456,7 +456,7 @@ def test_tp_language_embedding(

456 parallel_setup: ParallelSetup,

457 distributed_backend: str,

458 vllm_major_version: str,

459 - task: TaskOption,

459 + runner: RunnerOption,

460 test_options: PPTestOptions,

461 num_gpus_available,

462 ):

@@ -464,7 +464,7 @@ def test_tp_language_embedding(

464 parallel_setup,

465 distributed_backend,

466 vllm_major_version,

467 - task,

467 + runner,

468 test_options,

469 num_gpus_available,

470 method="encode",

@@ -473,7 +473,7 @@ def test_tp_language_embedding(

473

474 @pytest.mark.parametrize(

475 ("model_id", "parallel_setup", "distributed_backend", "vllm_major_version",

476 - "task", "test_options"),

476 + "runner", "test_options"),

477 [

478 params for model_id, settings in MULTIMODAL_MODELS.items()

479 for params in settings.iter_params(model_id) if model_id in TEST_MODELS

@@ -485,7 +485,7 @@ def test_tp_multimodal_generation(

485 parallel_setup: ParallelSetup,

486 distributed_backend: str,

487 vllm_major_version: str,

488 - task: TaskOption,

488 + runner: RunnerOption,

489 test_options: PPTestOptions,

490 num_gpus_available,

491 ):

@@ -493,7 +493,7 @@ def test_tp_multimodal_generation(

493 parallel_setup,

494 distributed_backend,

495 vllm_major_version,

496 - task,

496 + runner,

497 test_options,

498 num_gpus_available,

499 method="generate",

tests/distributed/test_sequence_parallel.py

+15 -15

@@ -14,7 +14,7 @@ from typing import Literal, NamedTuple, Optional

15 import pytest

17 -from vllm.config import TaskOption

17 +from vllm.config import RunnerOption

18 from vllm.logger import init_logger

20 from ..models.registry import HF_EXAMPLE_MODELS

@@ -48,7 +48,7 @@ class SPTestSettings:

48 distributed_backends: list[str]

49 # vllm major version: "0" for V0, "1" for V1

50 vllm_major_versions: list[str]

51 - task: TaskOption

51 + runner: RunnerOption

52 test_options: SPTestOptions

54 def __post_init__(self):

@@ -64,7 +64,7 @@ class SPTestSettings:

64 tp_base: int = 2,

65 pp_base: int = 1,

66 multi_node_only: bool = False,

67 - task: TaskOption = "auto",

67 + runner: RunnerOption = "auto",

68 load_format: Optional[str] = None,

69 ):

70 parallel_setups = []

@@ -81,7 +81,7 @@ class SPTestSettings:

81 parallel_setups=parallel_setups,

82 distributed_backends=["mp", "ray"],

83 vllm_major_versions=["1", "1"],

84 - task=task,

84 + runner=runner,

85 test_options=SPTestOptions(multi_node_only=multi_node_only,

86 load_format=load_format),

87 )

@@ -91,7 +91,7 @@ class SPTestSettings:

91 *,

92 tp_base: int = 2,

93 pp_base: int = 1,

94 - task: TaskOption = "auto",

94 + runner: RunnerOption = "auto",

95 multi_node_only: bool = False,

96 load_format: Optional[str] = None,

97 ):

@@ -109,7 +109,7 @@ class SPTestSettings:

109 parallel_setups=parallel_setups,

110 distributed_backends=["mp", "ray"],

111 vllm_major_versions=["1", "1"],

112 - task=task,

112 + runner=runner,

113 test_options=SPTestOptions(multi_node_only=multi_node_only,

114 load_format=load_format),

115 )

@@ -119,7 +119,7 @@ class SPTestSettings:

119 *,

120 tp_base: int = 2,

121 pp_base: int = 1,

122 - task: TaskOption = "auto",

122 + runner: RunnerOption = "auto",

123 multi_node_only: bool = False,

124 load_format: Optional[str] = None,

125 ):

@@ -135,7 +135,7 @@ class SPTestSettings:

135 parallel_setups=parallel_setups,

136 distributed_backends=["mp", "ray"],

137 vllm_major_versions=["1", "1"],

138 - task=task,

138 + runner=runner,

139 test_options=SPTestOptions(multi_node_only=multi_node_only,

140 load_format=load_format),

141 )

@@ -147,7 +147,7 @@ class SPTestSettings:

147 for backend, vllm_major_version in zip(self.distributed_backends,

148 self.vllm_major_versions):

149 yield (model_id, parallel_setup, backend, vllm_major_version,

150 - self.task, opts)

150 + self.runner, opts)

151

152

153 def _compare_sp(

@@ -155,7 +155,7 @@ def _compare_sp(

155 parallel_setup: ParallelSetup,

156 distributed_backend: str,

157 vllm_major_version: str,

158 - task: TaskOption,

158 + runner: RunnerOption,

159 test_options: SPTestOptions,

160 num_gpus_available: int,

161 *,

@@ -217,8 +217,8 @@ def _compare_sp(

217 common_args.append("--enable-chunked-prefill")

218 if eager_mode:

219 common_args.append("--enforce-eager")

220 - if task != "auto":

221 - common_args.extend(["--task", task])

220 + if runner != "auto":

221 + common_args.extend(["--runner", runner])

222 if trust_remote_code:

223 common_args.append("--trust-remote-code")

224 if tokenizer_mode:

@@ -298,7 +298,7 @@ SP_TEST_MODELS = [

298

299 @pytest.mark.parametrize(

300 ("model_id", "parallel_setup", "distributed_backend", "vllm_major_version",

301 - "task", "test_options"),

301 + "runner", "test_options"),

302 [

303 params for model_id, settings in SP_TEXT_GENERATION_MODELS.items()

304 for params in settings.iter_params(model_id)

@@ -311,7 +311,7 @@ def test_tp_sp_generation(

311 parallel_setup: ParallelSetup,

312 distributed_backend: str,

313 vllm_major_version: str,

314 - task: TaskOption,

314 + runner: RunnerOption,

315 test_options: SPTestOptions,

316 num_gpus_available,

317 ):

@@ -319,7 +319,7 @@ def test_tp_sp_generation(

319 parallel_setup,

320 distributed_backend,

321 vllm_major_version,

322 - task,

322 + runner,

323 test_options,

324 num_gpus_available,

325 method="generate",

tests/entrypoints/openai/correctness/test_mteb_embed.py

+2 -1

@@ -19,7 +19,8 @@ MAIN_SCORE = 0.7422994752439667

19 @pytest.fixture(scope="module")

20 def server():

21 args = [

22 - "--task", "embed", "--enforce-eager", "--disable-uvicorn-access-log"

22 + "--runner", "pooling", "--enforce-eager",

23 + "--disable-uvicorn-access-log"

24 ]

26 with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:

tests/entrypoints/openai/correctness/test_mteb_score.py

+2 -1

@@ -21,7 +21,8 @@ MODEL_NAME = "cross-encoder/ms-marco-MiniLM-L-6-v2"

21 @pytest.fixture(scope="module")

22 def server():

23 args = [

24 - "--task", "score", "--enforce-eager", "--disable-uvicorn-access-log"

24 + "--runner", "pooling", "--enforce-eager",

25 + "--disable-uvicorn-access-log"

26 ]

28 with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:

tests/entrypoints/openai/test_chat_logit_bias_validation.py

-4

@@ -15,10 +15,6 @@ MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct"

15 def get_vocab_size(model_name):

16 config = ModelConfig(

17 model=model_name,

18 - task="auto",

19 - tokenizer=model_name,

20 - tokenizer_mode="auto",

21 - trust_remote_code=False,

18 seed=0,

19 dtype="bfloat16",

20 )

tests/entrypoints/openai/test_chat_template.py

@@ -102,6 +102,7 @@ def test_get_gen_prompt(model, template, add_generation_prompt,

102 tokenizer=model_info.tokenizer or model,

103 tokenizer_mode=model_info.tokenizer_mode,

104 trust_remote_code=model_info.trust_remote_code,

105 + revision=model_info.revision,

106 hf_overrides=model_info.hf_overrides,

107 )

108

tests/entrypoints/openai/test_embedding.py

+2 -2

@@ -33,8 +33,8 @@ def v1(run_with_both_engines):

33 @pytest.fixture(scope="module")

34 def server():

35 args = [

36 - "--task",

37 - "embed",

36 + "--runner",

37 + "pooling",

38 # use half precision for speed and memory savings in CI environment

39 "--dtype",

40 DTYPE,

tests/entrypoints/openai/test_embedding_dimensions.py

+2 -2

@@ -42,8 +42,8 @@ def dtype(request):

42 @pytest.fixture(scope="module")

43 def server(model_info, dtype: str):

44 args = [

45 - "--task",

46 - "embed",

45 + "--runner",

46 + "pooling",

47 # use half precision for speed and memory savings in CI environment

48 "--dtype",

49 dtype,

tests/entrypoints/openai/test_openai_schema.py

+1 -1

@@ -21,7 +21,7 @@ LONG_TIMEOUT_SECONDS: Final[int] = 60

21 @pytest.fixture(scope="module")

22 def server():

23 args = [

24 - "--task",

24 + "--runner",

25 "generate",

26 "--max-model-len",

27 "2048",

tests/entrypoints/openai/test_optional_middleware.py

+2 -2

@@ -27,8 +27,8 @@ def server(request: pytest.FixtureRequest):

27 passed_params = [passed_params]

29 args = [

30 - "--task",

31 - "embed",

30 + "--runner",

31 + "pooling",

32 # use half precision for speed and memory savings in CI environment

33 "--dtype",

34 "float16",

tests/entrypoints/openai/test_pooling.py

+2 -2

@@ -20,8 +20,8 @@ DUMMY_CHAT_TEMPLATE = """{% for message in messages %}{{message['role'] + ': ' +

20 @pytest.fixture(scope="module")

21 def server():

22 args = [

23 - "--task",

24 - "reward",

23 + "--runner",

24 + "pooling",

25 # use half precision for speed and memory savings in CI environment

26 "--dtype",

27 "bfloat16",

tests/entrypoints/openai/test_skip_tokenizer.py

+2 -2

@@ -26,8 +26,8 @@ def v1(run_with_both_engines):

26 @pytest.fixture(scope="module")

27 def server():

28 args = [

29 - "--task",

30 - "embed",

29 + "--runner",

30 + "pooling",

31 # use half precision for speed and memory savings in CI environment

32 "--dtype",

33 DTYPE,

tests/entrypoints/openai/test_truncation.py

+2 -2

@@ -29,8 +29,8 @@ input = """Immerse yourself in the enchanting chronicle of calculus, a

29 @pytest.fixture(scope="module")

30 def server():

31 args = [

32 - "--task",

33 - "embed",

32 + "--runner",

33 + "pooling",

34 "--dtype",

35 "bfloat16",

36 "--enforce-eager",

tests/entrypoints/openai/test_video.py

+1 -1

@@ -25,7 +25,7 @@ TEST_VIDEO_URLS = [

25 @pytest.fixture(scope="module")

26 def server():

27 args = [

28 - "--task",

28 + "--runner",

29 "generate",

30 "--max-model-len",

31 "32768",

tests/entrypoints/openai/test_vision.py

+1 -1

@@ -48,7 +48,7 @@ EXPECTED_MM_BEAM_SEARCH_RES = [

48 @pytest.fixture(scope="module")

49 def server():

50 args = [

51 - "--task",

51 + "--runner",

52 "generate",

53 "--max-model-len",

54 "2048",

tests/entrypoints/openai/test_vision_embedding.py

+2 -2

@@ -31,8 +31,8 @@ TEST_IMAGE_URLS = [

31 @pytest.fixture(scope="module")

32 def server():

33 args = [

34 - "--task",

35 - "embed",

34 + "--runner",

35 + "pooling",

36 "--max-model-len",

37 "2048",

38 "--max-num-seqs",

tests/entrypoints/test_chat_utils.py

+9 -33

@@ -47,12 +47,8 @@ MISTRAL_MODEL_ID = "mistralai/Mistral-Small-3.1-24B-Instruct-2503"

47 @pytest.fixture(scope="function")

48 def phi3v_model_config():

49 return ModelConfig(PHI3V_MODEL_ID,

50 - task="generate",

51 - tokenizer=PHI3V_MODEL_ID,

52 - tokenizer_mode="auto",

50 + runner="generate",

51 trust_remote_code=True,

54 - dtype="auto",

55 - seed=0,

52 limit_mm_per_prompt={

53 "image": 2,

54 })

@@ -61,12 +57,8 @@ def phi3v_model_config():

57 @pytest.fixture(scope="function")

58 def phi3v_model_config_mm_interleaved():

59 return ModelConfig(PHI3V_MODEL_ID,

64 - task="generate",

65 - tokenizer=PHI3V_MODEL_ID,

66 - tokenizer_mode="auto",

60 + runner="generate",

61 trust_remote_code=True,

68 - dtype="auto",

69 - seed=0,

62 interleave_mm_strings=True,

63 limit_mm_per_prompt={

64 "image": 2,

@@ -86,11 +78,7 @@ def phi3v_tokenizer():

78 @pytest.fixture(scope="function")

79 def qwen25omni_model_config_mm_interleaved():

80 return ModelConfig(QWEN25OMNI_MODEL_ID,

89 - task="generate",

90 - tokenizer=QWEN25OMNI_MODEL_ID,

91 - tokenizer_mode="auto",

92 - dtype="auto",

93 - seed=0,

81 + runner="generate",

82 interleave_mm_strings=True,

83 limit_mm_per_prompt={

84 "image": 2,

@@ -112,12 +100,7 @@ def qwen25omni_tokenizer():

100 @pytest.fixture(scope="module")

101 def mllama_model_config():

102 return ModelConfig(MLLAMA_MODEL_ID,

115 - task="generate",

116 - tokenizer=MLLAMA_MODEL_ID,

117 - tokenizer_mode="auto",

118 - trust_remote_code=True,

119 - dtype="auto",

120 - seed=0,

103 + runner="generate",

104 limit_mm_per_prompt={

105 "image": 2,

106 })

@@ -136,12 +119,7 @@ def mllama_tokenizer():

119 @pytest.fixture(scope="function")

120 def mistral_model_config():

121 return ModelConfig(MISTRAL_MODEL_ID,

139 - task="generate",

140 - tokenizer=MISTRAL_MODEL_ID,

141 - tokenizer_mode="auto",

142 - trust_remote_code=True,

143 - dtype="auto",

144 - seed=0,

122 + runner="generate",

123 limit_mm_per_prompt={

124 "image": 2,

125 })

@@ -1105,12 +1083,7 @@ def test_multimodal_image_parsing_matches_hf(model, image_url):

1083

1084 # Build a config for the model

1085 model_config = ModelConfig(model,

1108 - task="generate",

1109 - tokenizer=model,

1110 - tokenizer_mode="auto",

1111 - trust_remote_code=True,

1112 - dtype="auto",

1113 - seed=0,

1086 + runner="generate",

1087 limit_mm_per_prompt={

1088 "image": 2,

1089 })

@@ -1170,6 +1143,7 @@ def test_resolve_hf_chat_template(sample_json_schema, model, use_tools):

1143 model,

1144 tokenizer=model_info.tokenizer or model,

1145 tokenizer_mode=model_info.tokenizer_mode,

1146 + revision=model_info.revision,

1147 trust_remote_code=model_info.trust_remote_code,

1148 hf_overrides=model_info.hf_overrides,

1149 )

@@ -1225,6 +1199,7 @@ def test_resolve_content_format_hf_defined(model, expected_format):

1199 model,

1200 tokenizer=model_info.tokenizer or model,

1201 tokenizer_mode=model_info.tokenizer_mode,

1202 + revision=model_info.revision,

1203 trust_remote_code=model_info.trust_remote_code,

1204 hf_overrides=model_info.hf_overrides,

1205 )

@@ -1284,6 +1259,7 @@ def test_resolve_content_format_fallbacks(model, expected_format):

1259 model,

1260 tokenizer=model_info.tokenizer or model,

1261 tokenizer_mode=model_info.tokenizer_mode,

1262 + revision=model_info.revision,

1263 trust_remote_code=model_info.trust_remote_code,

1264 hf_overrides=model_info.hf_overrides,

1265 )

tests/lora/test_worker.py

-5

@@ -38,13 +38,8 @@ def test_worker_apply_lora(sql_lora_files):

38 vllm_config = VllmConfig(

39 model_config=ModelConfig(

40 "meta-llama/Llama-2-7b-hf",

41 - task="auto",

42 - tokenizer="meta-llama/Llama-2-7b-hf",

43 - tokenizer_mode="auto",

44 - trust_remote_code=False,

41 seed=0,

42 dtype="float16",

47 - revision=None,

43 enforce_eager=True,

44 ),

45 load_config=LoadConfig(

tests/model_executor/test_guided_processors.py

+2 -8

@@ -69,10 +69,7 @@ async def test_guided_logits_processor_black_box(backend: str, is_local: bool,

70 config = ModelConfig(

71 MODEL_NAME,

72 - task="generate",

73 - tokenizer=MODEL_NAME,

74 - tokenizer_mode="auto",

75 - trust_remote_code=False,

72 + runner="generate",

73 seed=0,

74 dtype="bfloat16",

75 )

@@ -113,10 +110,7 @@ async def test_guided_logits_processor_with_reasoning(

110

111 config = ModelConfig(

112 REASONING_MODEL_NAME,

116 - task="generate",

117 - tokenizer=REASONING_MODEL_NAME,

118 - tokenizer_mode="auto",

119 - trust_remote_code=False,

113 + runner="generate",

114 seed=0,

115 dtype="bfloat16",

116 )

tests/model_executor/test_model_load_with_params.py

-2

@@ -57,7 +57,6 @@ def test_model_loading_with_params(vllm_runner, monkeypatch):

58 vllm_model.apply_model(check_model)

60 - # assert output

60 assert output

@@ -99,7 +98,6 @@ def test_roberta_model_loading_with_params(vllm_runner, monkeypatch):

99 vllm_model.apply_model(check_model)

100

102 - # assert output

101 assert output

102

103

tests/models/language/pooling/embed_utils.py

+1 -1

@@ -52,7 +52,7 @@ def correctness_test_embed_models(hf_runner,

52 vllm_extra_kwargs["dtype"] = model_info.dtype

54 with vllm_runner(model_info.name,

55 - task="embed",

55 + runner="pooling",

56 max_model_len=None,

57 **vllm_extra_kwargs) as vllm_model:

58 vllm_outputs = vllm_model.embed(example_prompts)

tests/models/language/pooling/mteb_utils.py

+2 -5

@@ -172,7 +172,7 @@ def mteb_test_embed_models(hf_runner,

172 vllm_extra_kwargs["dtype"] = model_info.dtype

173

174 with vllm_runner(model_info.name,

175 - task="embed",

175 + runner="pooling",

176 max_model_len=None,

177 **vllm_extra_kwargs) as vllm_model:

178

@@ -279,15 +279,12 @@ def mteb_test_rerank_models(hf_runner,

279 vllm_extra_kwargs["dtype"] = model_info.dtype

280

281 with vllm_runner(model_info.name,

282 - task="score",

282 + runner="pooling",

283 max_model_len=None,

284 max_num_seqs=8,

285 **vllm_extra_kwargs) as vllm_model:

286

287 model_config = vllm_model.llm.llm_engine.model_config

288 -

289 - if model_info.architecture:

290 - assert (model_info.architecture in model_config.architectures)

288 assert model_config.hf_config.num_labels == 1

289

290 vllm_main_score = run_mteb_rerank(vllm_mteb_encoder(vllm_model),

tests/models/language/pooling/test_embedding.py

+1 -1

@@ -85,7 +85,7 @@ def test_models(

85 hf_outputs = hf_model.encode(example_prompts)

87 with vllm_runner(model,

88 - task="embed",

88 + runner="pooling",

89 max_model_len=max_model_len,

90 **vllm_extra_kwargs) as vllm_model:

91 vllm_outputs = vllm_model.embed(example_prompts)

tests/models/language/pooling/test_gritlm.py

+5 -8

@@ -28,10 +28,7 @@ def test_find_array():

29 model_config = ModelConfig(

30 MODEL_NAME,

31 - task="embed",

32 - tokenizer=MODEL_NAME,

33 - tokenizer_mode="auto",

34 - trust_remote_code=False,

31 + runner="pooling",

32 dtype="bfloat16",

33 seed=0,

34 )

@@ -117,7 +114,7 @@ def test_gritlm_offline_embedding(vllm_runner):

114

115 with vllm_runner(

116 MODEL_NAME,

120 - task="embed",

117 + runner="pooling",

118 max_model_len=MAX_MODEL_LEN,

119 ) as vllm_model:

120 llm = vllm_model.llm

@@ -140,7 +137,7 @@ def test_gritlm_offline_embedding(vllm_runner):

137 async def test_gritlm_api_server_embedding():

138 queries, q_instruction, documents, d_instruction = get_test_data()

139

143 - args = ["--task", "embed", "--max_model_len", str(MAX_MODEL_LEN)]

140 + args = ["--runner", "pooling", "--max_model_len", str(MAX_MODEL_LEN)]

141

142 with RemoteOpenAIServer(MODEL_NAME, args) as server:

143 client_embedding = server.get_async_client()

@@ -164,7 +161,7 @@ def test_gritlm_offline_generate(monkeypatch: pytest.MonkeyPatch, vllm_runner):

161

162 with vllm_runner(

163 MODEL_NAME,

167 - task="generate",

164 + runner="generate",

165 max_model_len=MAX_MODEL_LEN,

166 ) as vllm_model:

167 llm = vllm_model.llm

@@ -179,7 +176,7 @@ def test_gritlm_offline_generate(monkeypatch: pytest.MonkeyPatch, vllm_runner):

176 async def test_gritlm_api_server_generate():

177 input = "<|user|>\nWhat is the capital of France?\n<|assistant|>\n"

178

182 - args = ["--task", "generate", "--max_model_len", str(MAX_MODEL_LEN)]

179 + args = ["--runner", "generate", "--max_model_len", str(MAX_MODEL_LEN)]

180

181 with RemoteOpenAIServer(MODEL_NAME, args) as server:

182 client_generate = server.get_async_client()

tests/models/language/pooling/test_jina.py

+6 -1

@@ -4,6 +4,7 @@ from functools import partial

5 import pytest

7 +import vllm.envs as envs

8 from vllm import PoolingParams

10 from ...utils import EmbedModelInfo, RerankModelInfo

@@ -62,6 +63,10 @@ def test_embed_models_correctness(hf_runner, vllm_runner,

63 @pytest.mark.parametrize("model_info", RERANK_MODELS)

64 def test_rerank_models_mteb(hf_runner, vllm_runner,

65 model_info: RerankModelInfo) -> None:

66 + if (model_info.architecture == "XLMRobertaForSequenceClassification"

67 + and envs.VLLM_USE_V1):

68 + pytest.skip("Not supported yet")

69 +

70 mteb_test_rerank_models(hf_runner, vllm_runner, model_info)

@@ -92,7 +97,7 @@ def test_matryoshka(

97 hf_outputs = matryoshka_fy(hf_outputs, dimensions)

99 with vllm_runner(model_info.name,

95 - task="embed",

100 + runner="pooling",

101 dtype=dtype,

102 max_model_len=None) as vllm_model:

103 assert vllm_model.llm.llm_engine.model_config.is_matryoshka

tests/models/language/pooling/test_nomic_max_model_len.py

+11 -9

@@ -21,7 +21,7 @@ max_model_len = int(original_max_position_embeddings * factor)

22 @pytest.mark.parametrize("model_info", MODELS)

23 def test_default(model_info, vllm_runner):

24 - with vllm_runner(model_info.name, task="embed",

24 + with vllm_runner(model_info.name, runner="pooling",

25 max_model_len=None) as vllm_model:

26 model_config = vllm_model.llm.llm_engine.model_config

27 if model_info.name == "nomic-ai/nomic-embed-text-v2-moe":

@@ -36,7 +36,7 @@ def test_default(model_info, vllm_runner):

36 @pytest.mark.parametrize("model_info", MODELS)

37 def test_set_max_model_len_legal(model_info, vllm_runner):

38 # set max_model_len <= 512

39 - with vllm_runner(model_info.name, task="embed",

39 + with vllm_runner(model_info.name, runner="pooling",

40 max_model_len=256) as vllm_model:

41 model_config = vllm_model.llm.llm_engine.model_config

42 assert model_config.max_model_len == 256

@@ -46,11 +46,12 @@ def test_set_max_model_len_legal(model_info, vllm_runner):

46 # For nomic-embed-text-v2-moe the length is set to 512

47 # by sentence_bert_config.json.

48 with pytest.raises(ValueError):

49 - with vllm_runner(model_info.name, task="embed",

49 + with vllm_runner(model_info.name,

50 + runner="pooling",

51 max_model_len=1024):

52 pass

53 else:

53 - with vllm_runner(model_info.name, task="embed",

54 + with vllm_runner(model_info.name, runner="pooling",

55 max_model_len=1024) as vllm_model:

56 model_config = vllm_model.llm.llm_engine.model_config

57 assert model_config.max_model_len == 1024

@@ -60,14 +61,15 @@ def test_set_max_model_len_legal(model_info, vllm_runner):

61 def test_set_max_model_len_illegal(model_info, vllm_runner):

62 # set max_model_len > 2048

63 with pytest.raises(ValueError):

63 - with vllm_runner(model_info.name, task="embed", max_model_len=4096):

64 + with vllm_runner(model_info.name, runner="pooling",

65 + max_model_len=4096):

66 pass

68 # set max_model_len > 2048 by hf_overrides

69 hf_overrides = {"max_model_len": 4096}

70 with pytest.raises(ValueError):

71 with vllm_runner(model_info.name,

70 - task="embed",

72 + runner="pooling",

73 max_model_len=None,

74 hf_overrides=hf_overrides):

75 pass

@@ -87,7 +89,7 @@ def test_use_rope_scaling_legal(model_info, vllm_runner):

89 }

91 with vllm_runner(model_info.name,

90 - task="embed",

92 + runner="pooling",

93 max_model_len=None,

94 hf_overrides=hf_overrides):

95 pass

@@ -107,7 +109,7 @@ def test_use_rope_scaling_illegal(model_info, vllm_runner):

109 # illegal max_model_len

110 with pytest.raises(ValueError):

111 with vllm_runner(model_info.name,

110 - task="embed",

112 + runner="pooling",

113 max_model_len=max_model_len + 1,

114 hf_overrides=hf_overrides):

115 pass

@@ -125,7 +127,7 @@ def test_use_rope_scaling_illegal(model_info, vllm_runner):

127 # illegal max_model_len by hf_overrides

128 with pytest.raises(ValueError):

129 with vllm_runner(model_info.name,

128 - task="embed",

130 + runner="pooling",

131 max_model_len=None,

132 hf_overrides=hf_overrides):

133 pass

tests/models/language/pooling/test_scoring.py

+12 -6

@@ -37,7 +37,9 @@ def test_cross_encoder_1_to_1(vllm_runner, hf_runner, model_name):

37 with hf_runner(model_name, dtype=DTYPE, is_cross_encoder=True) as hf_model:

38 hf_outputs = hf_model.predict([text_pair]).tolist()

40 - with vllm_runner(model_name, task="score", dtype=DTYPE,

40 + with vllm_runner(model_name,

41 + runner="pooling",

42 + dtype=DTYPE,

43 max_model_len=None) as vllm_model:

44 vllm_outputs = vllm_model.score(text_pair[0], text_pair[1])

@@ -56,7 +58,9 @@ def test_cross_encoder_1_to_N(vllm_runner, hf_runner, model_name):

58 with hf_runner(model_name, dtype=DTYPE, is_cross_encoder=True) as hf_model:

59 hf_outputs = hf_model.predict(text_pairs).tolist()

59 - with vllm_runner(model_name, task="score", dtype=DTYPE,

61 + with vllm_runner(model_name,

62 + runner="pooling",

63 + dtype=DTYPE,

64 max_model_len=None) as vllm_model:

65 vllm_outputs = vllm_model.score(TEXTS_1[0], TEXTS_2)

@@ -76,7 +80,9 @@ def test_cross_encoder_N_to_N(vllm_runner, hf_runner, model_name):

80 with hf_runner(model_name, dtype=DTYPE, is_cross_encoder=True) as hf_model:

81 hf_outputs = hf_model.predict(text_pairs).tolist()

79 - with vllm_runner(model_name, task="score", dtype=DTYPE,

83 + with vllm_runner(model_name,

84 + runner="pooling",

85 + dtype=DTYPE,

86 max_model_len=None) as vllm_model:

87 vllm_outputs = vllm_model.score(TEXTS_1, TEXTS_2)

@@ -103,7 +109,7 @@ def test_embedding_1_to_1(vllm_runner, hf_runner, emb_model_name):

109 ]

110

111 with vllm_runner(emb_model_name,

106 - task="embed",

112 + runner="pooling",

113 dtype=DTYPE,

114 max_model_len=None) as vllm_model:

115 vllm_outputs = vllm_model.score(text_pair[0], text_pair[1])

@@ -131,7 +137,7 @@ def test_embedding_1_to_N(vllm_runner, hf_runner, emb_model_name):

137 ]

138

139 with vllm_runner(emb_model_name,

134 - task="embed",

140 + runner="pooling",

141 dtype=DTYPE,

142 max_model_len=None) as vllm_model:

143 vllm_outputs = vllm_model.score(TEXTS_1[0], TEXTS_2)

@@ -160,7 +166,7 @@ def test_embedding_N_to_N(vllm_runner, hf_runner, emb_model_name):

166 ]

167

168 with vllm_runner(emb_model_name,

163 - task="embed",

169 + runner="pooling",

170 dtype=DTYPE,

171 max_model_len=None) as vllm_model:

172 vllm_outputs = vllm_model.score(TEXTS_1, TEXTS_2)

tests/models/language/pooling/test_truncation_control.py

+3 -3

@@ -26,7 +26,7 @@ def test_smaller_truncation_size(vllm_runner,

27 truncate_prompt_tokens = 10

29 - with vllm_runner(model_name, task="embed",

29 + with vllm_runner(model_name, runner="pooling",

30 max_model_len=max_model_len) as vllm_model:

31 vllm_output = vllm_model.llm.encode(

32 input_str, truncate_prompt_tokens=truncate_prompt_tokens)

@@ -41,7 +41,7 @@ def test_max_truncation_size(vllm_runner,

41 input_str=input_str):

42 truncate_prompt_tokens = -1

44 - with vllm_runner(model_name, task="embed",

44 + with vllm_runner(model_name, runner="pooling",

45 max_model_len=max_model_len) as vllm_model:

46 vllm_output = vllm_model.llm.encode(

47 input_str, truncate_prompt_tokens=truncate_prompt_tokens)

@@ -58,7 +58,7 @@ def test_bigger_truncation_size(vllm_runner,

58 truncate_prompt_tokens = max_model_len + 1

60 with pytest.raises(ValueError), vllm_runner(

61 - model_name, task="embed",

61 + model_name, runner="pooling",

62 max_model_len=max_model_len) as vllm_model:

64 llm_output = vllm_model.llm.encode(

tests/models/multimodal/generation/test_common.py

+2 -3

@@ -222,7 +222,6 @@ VLM_TEST_SETTINGS = {

222 },

223 marks=[large_gpu_mark(min_gb=32)],

224 ),

225 - # Check "auto" with fallback to transformers

225 "internvl-transformers": VLMTestInfo(

226 models=["OpenGVLab/InternVL3-1B-hf"],

227 test_type=(VLMTestType.IMAGE, VLMTestType.MULTI_IMAGE),

@@ -232,7 +231,7 @@ VLM_TEST_SETTINGS = {

231 use_tokenizer_eos=True,

232 image_size_factors=[(0.25, 0.5, 1.0)],

233 vllm_runner_kwargs={

235 - "model_impl": "auto",

234 + "model_impl": "transformers",

235 },

236 auto_cls=AutoModelForImageTextToText,

237 marks=[pytest.mark.core_model],

@@ -638,7 +637,7 @@ VLM_TEST_SETTINGS = {

637 img_idx_to_prompt=lambda idx: f"<|image_{idx}|>\n",

638 max_model_len=4096,

639 max_num_seqs=2,

641 - task="generate",

640 + runner="generate",

641 # use sdpa mode for hf runner since phi3v didn't work with flash_attn

642 hf_model_kwargs={"_attn_implementation": "sdpa"},

643 use_tokenizer_eos=True,

tests/models/multimodal/generation/test_granite_speech.py

+1 -1

@@ -65,7 +65,7 @@ def run_test(

65 # max_model_len should be greater than image_feature_size

66 with vllm_runner(

67 model,

68 - task="generate",

68 + runner="generate",

69 max_model_len=max_model_len,

70 max_num_seqs=1,

71 dtype=dtype,

tests/models/multimodal/generation/test_interleaved.py

+1 -1

@@ -48,7 +48,7 @@ def test_models(vllm_runner, model, dtype: str, max_tokens: int) -> None:

48 ]

50 with vllm_runner(model,

51 - task="generate",

51 + runner="generate",

52 dtype=dtype,

53 limit_mm_per_prompt={"image": 2},

54 max_model_len=32768,

tests/models/multimodal/generation/test_phi4mm.py

+1 -1

@@ -99,7 +99,7 @@ def run_test(

99 # max_model_len should be greater than image_feature_size

100 with vllm_runner(

101 model,

102 - task="generate",

102 + runner="generate",

103 max_model_len=max_model_len,

104 max_num_seqs=2,

105 dtype=dtype,

tests/models/multimodal/generation/test_qwen2_vl.py

+1 -1

@@ -267,7 +267,7 @@ def run_embedding_input_test(

267

268 # max_model_len should be greater than image_feature_size

269 with vllm_runner(model,

270 - task="generate",

270 + runner="generate",

271 max_model_len=4000,

272 max_num_seqs=3,

273 dtype=dtype,

tests/models/multimodal/generation/vlm_utils/core.py

+3 -3

@@ -6,7 +6,7 @@ from typing import Any, Callable, Optional

6 import torch

7 from transformers.models.auto.auto_factory import _BaseAutoModelClass

9 -from vllm.config import TaskOption

9 +from vllm.config import RunnerOption

10 from vllm.transformers_utils.tokenizer import AnyTokenizer

12 from .....conftest import HfRunner, VllmRunner

@@ -37,7 +37,7 @@ def run_test(

37 vllm_runner_kwargs: Optional[dict[str, Any]],

38 hf_model_kwargs: Optional[dict[str, Any]],

39 patch_hf_runner: Optional[Callable[[HfRunner], HfRunner]],

40 - task: TaskOption = "auto",

40 + runner: RunnerOption = "auto",

41 distributed_executor_backend: Optional[str] = None,

42 tensor_parallel_size: int = 1,

43 vllm_embeddings: Optional[torch.Tensor] = None,

@@ -83,7 +83,7 @@ def run_test(

83 tensor_parallel_size=tensor_parallel_size,

84 distributed_executor_backend=distributed_executor_backend,

85 enforce_eager=enforce_eager,

86 - task=task,

86 + runner=runner,

87 **vllm_runner_kwargs_) as vllm_model:

88 tokenizer = vllm_model.llm.get_tokenizer()

tests/models/multimodal/generation/vlm_utils/types.py

+3 -3

@@ -11,7 +11,7 @@ from pytest import MarkDecorator

11 from transformers import AutoModelForCausalLM

12 from transformers.models.auto.auto_factory import _BaseAutoModelClass

14 -from vllm.config import TaskOption

14 +from vllm.config import RunnerOption

15 from vllm.sequence import SampleLogprobs

16 from vllm.transformers_utils.tokenizer import AnyTokenizer

@@ -109,7 +109,7 @@ class VLMTestInfo(NamedTuple):

109 enforce_eager: bool = True

110 max_model_len: int = 1024

111 max_num_seqs: int = 256

112 - task: TaskOption = "auto"

112 + runner: RunnerOption = "auto"

113 tensor_parallel_size: int = 1

114 vllm_runner_kwargs: Optional[dict[str, Any]] = None

115

@@ -173,7 +173,7 @@ class VLMTestInfo(NamedTuple):

173 "enforce_eager": self.enforce_eager,

174 "max_model_len": self.max_model_len,

175 "max_num_seqs": self.max_num_seqs,

176 - "task": self.task,

176 + "runner": self.runner,

177 "tensor_parallel_size": self.tensor_parallel_size,

178 "vllm_runner_kwargs": self.vllm_runner_kwargs,

179 "hf_output_post_proc": self.hf_output_post_proc,

tests/models/multimodal/pooling/test_dse_qwen2_vl.py

+1 -1

@@ -92,7 +92,7 @@ def _run_test(

92 # if we run HF first, the cuda initialization will be done and it

93 # will hurt multiprocessing backend with fork method (the default method).

94 with vllm_runner(model,

95 - task="embed",

95 + runner="pooling",

96 dtype=dtype,

97 enforce_eager=True,

98 max_model_len=8192) as vllm_model:

tests/models/multimodal/pooling/test_jinavl_reranker.py

+1 -1

@@ -49,7 +49,7 @@ def vllm_reranker(

50 with vllm_runner(

51 model_name,

52 - task="score",

52 + runner="pooling",

53 dtype=dtype,

54 max_num_seqs=2,

55 max_model_len=2048,

tests/models/multimodal/pooling/test_llava_next.py

+1 -1

@@ -64,7 +64,7 @@ def _run_test(

64 # if we run HF first, the cuda initialization will be done and it

65 # will hurt multiprocessing backend with fork method (the default method).

66 with vllm_runner(model,

67 - task="embed",

67 + runner="pooling",

68 dtype=dtype,

69 max_model_len=4096,

70 enforce_eager=True) as vllm_model:

tests/models/multimodal/pooling/test_phi3v.py

+1 -1

@@ -44,7 +44,7 @@ def _run_test(

44 # vLLM needs a fresh new process without cuda initialization.

45 # if we run HF first, the cuda initialization will be done and it

46 # will hurt multiprocessing backend with fork method (the default method).

47 - with vllm_runner(model, task="embed", dtype=dtype,

47 + with vllm_runner(model, runner="pooling", dtype=dtype,

48 enforce_eager=True) as vllm_model:

49 vllm_outputs = vllm_model.embed(input_texts, images=input_images)

tests/models/multimodal/pooling/test_prithvi_mae.py

+1 -1

@@ -34,7 +34,7 @@ def _run_test(

34 set_default_torch_num_threads(1),

35 vllm_runner(

36 model,

37 - task="embed",

37 + runner="pooling",

38 dtype=torch.float16,

39 enforce_eager=True,

40 skip_tokenizer_init=True,

tests/models/multimodal/processing/test_common.py

+1 -4

@@ -58,13 +58,10 @@ def _test_processing_correctness(

59 model_config = ModelConfig(

60 model_id,

61 - task="auto",

61 tokenizer=model_info.tokenizer or model_id,

62 tokenizer_mode=model_info.tokenizer_mode,

64 - trust_remote_code=model_info.trust_remote_code,

65 - seed=0,

66 - dtype="auto",

63 revision=model_info.revision,

64 + trust_remote_code=model_info.trust_remote_code,

65 hf_overrides=model_info.hf_overrides,

66 )

tests/models/multimodal/test_mapping.py

+1 -4

@@ -54,13 +54,10 @@ def test_hf_model_weights_mapper(model_arch: str):

55 model_config = ModelConfig(

56 model_id,

57 - task="auto",

57 tokenizer=model_info.tokenizer or model_id,

58 tokenizer_mode=model_info.tokenizer_mode,

59 + revision=model_info.revision,

60 trust_remote_code=model_info.trust_remote_code,

61 - seed=0,

62 - dtype="auto",

63 - revision=None,

61 hf_overrides=model_info.hf_overrides,

62 )

63 model_cls = MULTIMODAL_REGISTRY._get_model_cls(model_config)

tests/models/quantization/test_bitsandbytes.py

+1 -1

@@ -172,7 +172,7 @@ def test_4bit_bnb_embedding_model(

172

173 # Inflight 4bit quantization

174 with vllm_runner(model_name,

175 - task="embed",

175 + runner="pooling",

176 dtype=dtype,

177 gpu_memory_utilization=0.5,

178 quantization="bitsandbytes") as vllm_model:

tests/models/test_initialization.py

+5 -1

@@ -7,13 +7,15 @@ import pytest

7 from transformers import PretrainedConfig

9 from vllm import LLM

10 +from vllm.config import ModelImpl

11 from vllm.engine.llm_engine import LLMEngine as V0LLMEngine

12 from vllm.utils import GiB_bytes

13 from vllm.v1.core.kv_cache_utils import get_kv_cache_config

14 from vllm.v1.engine.core import EngineCore as V1EngineCore

16 from ..utils import create_new_process_for_each_test

16 -from .registry import AUTO_EXAMPLE_MODELS, HF_EXAMPLE_MODELS, HfExampleModels

17 +from .registry import (_TRANSFORMERS_BACKEND_MODELS, AUTO_EXAMPLE_MODELS,

18 + HF_EXAMPLE_MODELS, HfExampleModels)

21 @create_new_process_for_each_test()

@@ -126,6 +128,8 @@ def can_initialize(model_arch: str, monkeypatch: pytest.MonkeyPatch,

128 # these tests seem to produce leftover memory

129 gpu_memory_utilization=0.80,

130 load_format="dummy",

131 + model_impl=ModelImpl.TRANSFORMERS

132 + if model_arch in _TRANSFORMERS_BACKEND_MODELS else ModelImpl.VLLM,

133 hf_overrides=hf_overrides,

134 )

135

tests/models/test_registry.py

+12 -9

@@ -24,11 +24,9 @@ from .registry import HF_EXAMPLE_MODELS

25 @pytest.mark.parametrize("model_arch", ModelRegistry.get_supported_archs())

26 def test_registry_imports(model_arch):

27 - model_info = HF_EXAMPLE_MODELS.get_hf_info(model_arch)

28 - model_info.check_transformers_version(on_fail="skip")

29 -

27 # Ensure all model classes can be imported successfully

31 - model_cls, _ = ModelRegistry.resolve_model_cls(model_arch)

28 + model_cls = ModelRegistry._try_load_model_cls(model_arch)

29 + assert model_cls is not None

31 if model_arch in _SPECULATIVE_DECODING_MODELS:

32 return # Ignore these models which do not have a unified format

@@ -56,14 +54,16 @@ def test_registry_imports(model_arch):

54 ("XLMRobertaForSequenceClassification", False, False, True),

55 ])

56 def test_registry_model_property(model_arch, is_mm, init_cuda, is_ce):

59 - assert ModelRegistry.is_multimodal_model(model_arch) is is_mm

57 + model_info = ModelRegistry._try_inspect_model_cls(model_arch)

58 + assert model_info is not None

61 - assert ModelRegistry.is_cross_encoder_model(model_arch) is is_ce

60 + assert model_info.supports_multimodal is is_mm

61 + assert model_info.supports_cross_encoding is is_ce

63 if init_cuda and current_platform.is_cuda_alike():

64 assert not torch.cuda.is_initialized()

66 - ModelRegistry.resolve_model_cls(model_arch)

66 + ModelRegistry._try_load_model_cls(model_arch)

67 if not torch.cuda.is_initialized():

68 warnings.warn(

69 "This model no longer initializes CUDA on import. "

@@ -82,12 +82,15 @@ def test_registry_model_property(model_arch, is_mm, init_cuda, is_ce):

82 ("Qwen2VLForConditionalGeneration", True, True),

83 ])

84 def test_registry_is_pp(model_arch, is_pp, init_cuda):

85 - assert ModelRegistry.is_pp_supported_model(model_arch) is is_pp

85 + model_info = ModelRegistry._try_inspect_model_cls(model_arch)

86 + assert model_info is not None

87 +

88 + assert model_info.supports_pp is is_pp

90 if init_cuda and current_platform.is_cuda_alike():

91 assert not torch.cuda.is_initialized()

90 - ModelRegistry.resolve_model_cls(model_arch)

93 + ModelRegistry._try_load_model_cls(model_arch)

94 if not torch.cuda.is_initialized():

95 warnings.warn(

96 "This model no longer initializes CUDA on import. "

tests/models/test_transformers.py

+13 -1

@@ -33,6 +33,10 @@ def check_implementation(

33 args = (example_prompts, max_tokens, num_logprobs)

35 with runner_test(model, **kwargs_test, **kwargs) as model_test:

36 + model_config = model_test.llm.llm_engine.model_config

37 + assert model_config.architecture == (

38 + model_config._get_transformers_backend_cls())

39 +

40 outputs_test = model_test.generate_greedy_logprobs(*args)

42 with runner_ref(model, **kwargs_ref) as model_ref:

@@ -130,8 +134,13 @@ def test_quantization(

134 model_impl="transformers",

135 enforce_eager=True,

136 **quantization_kwargs) as vllm_model: # type: ignore[arg-type]

137 + model_config = vllm_model.llm.llm_engine.model_config

138 + assert model_config.architecture == (

139 + model_config._get_transformers_backend_cls())

140 +

141 transformers_outputs = vllm_model.generate_greedy_logprobs(

142 example_prompts, max_tokens=max_tokens, num_logprobs=num_logprobs)

143 +

144 check_logprobs_close(

145 outputs_0_lst=transformers_outputs,

146 outputs_1_lst=vllm_outputs,

@@ -151,7 +160,6 @@ def test_classify(

160 example_prompts,

161 model: str,

162 dtype: str,

154 - monkeypatch,

163 ) -> None:

164 import torch

165 from transformers import AutoModelForSequenceClassification

@@ -160,6 +168,10 @@ def test_classify(

168 max_model_len=512,

169 dtype=dtype,

170 model_impl="transformers") as vllm_model:

171 + model_config = vllm_model.llm.llm_engine.model_config

172 + assert model_config.architecture == (

173 + model_config._get_transformers_backend_cls())

174 +

175 vllm_outputs = vllm_model.classify(example_prompts)

176

177 with hf_runner(model,

tests/models/utils.py

+4 -3

@@ -8,7 +8,7 @@ from typing import Any, NamedTuple, Optional, Union

8 import torch

9 import torch.nn.functional as F

11 -from vllm.config import ModelConfig, TaskOption

11 +from vllm.config import ModelConfig, RunnerOption

12 from vllm.inputs import InputContext

13 from vllm.sequence import Logprob, PromptLogprobs, SampleLogprobs

@@ -255,7 +255,7 @@ def check_logprobs_close(

255

256 def build_model_context(

257 model_id: str,

258 - task: TaskOption = "auto",

258 + runner: RunnerOption = "auto",

259 dtype: Union[str, torch.dtype] = "auto",

260 model_config_kwargs: Optional[dict[str, Any]] = None,

261 mm_processor_kwargs: Optional[dict[str, Any]] = None,

@@ -280,9 +280,10 @@ def build_model_context(

280 model_config_kwargs = model_config_kwargs or {}

281 model_config = ModelConfig(

282 model_id,

283 - task=task,

283 + runner=runner,

284 tokenizer=model_info.tokenizer or model_id,

285 tokenizer_mode=model_info.tokenizer_mode,

286 + revision=model_info.revision,

287 trust_remote_code=model_info.trust_remote_code,

288 dtype=dtype,

289 seed=0,

tests/multimodal/test_processing.py

+1 -24

@@ -954,13 +954,6 @@ def test_limit_mm_per_prompt_dummy(model_id, limit, num_supported, is_valid):

954

955 model_config = ModelConfig(

956 model=model_id,

957 - task="auto",

958 - tokenizer=model_id,

959 - tokenizer_mode="auto",

960 - trust_remote_code=False,

961 - seed=0,

962 - dtype="auto",

963 - revision=None,

957 limit_mm_per_prompt=limit_mm_per_prompt,

958 )

959

@@ -993,13 +986,6 @@ def test_limit_mm_per_prompt_apply(model_id, num_images, limit, is_valid):

986

987 model_config = ModelConfig(

988 model=model_id,

996 - task="auto",

997 - tokenizer=model_id,

998 - tokenizer_mode="auto",

999 - trust_remote_code=False,

1000 - seed=0,

1001 - dtype="auto",

1002 - revision=None,

989 limit_mm_per_prompt=limit_mm_per_prompt,

990 )

991

@@ -1061,16 +1047,7 @@ class _ProcessorProxy:

1047 )

1048 # yapf: enable

1049 def test_hf_processor_kwargs(model_id, call_kwargs, expected_kwargs):

1064 - model_config = ModelConfig(

1065 - model=model_id,

1066 - task="auto",

1067 - tokenizer=model_id,

1068 - tokenizer_mode="auto",

1069 - trust_remote_code=False,

1070 - seed=0,

1071 - dtype="auto",

1072 - revision=None,

1073 - )

1050 + model_config = ModelConfig(model_id)

1051

1052 processor = MULTIMODAL_REGISTRY.create_processor(model_config)

1053 orig_get_hf_processor = processor.info.get_hf_processor

tests/quantization/test_configs.py

+1 -9

@@ -57,15 +57,7 @@ def test_auto_gptq(model_arg_exptype: tuple[str, None, str]) -> None:

57 model_path, quantization_arg, expected_type = model_arg_exptype

59 try:

60 - model_config = ModelConfig(model_path,

61 - task="auto",

62 - tokenizer=model_path,

63 - tokenizer_mode="auto",

64 - trust_remote_code=False,

65 - seed=0,

66 - dtype="float16",

67 - revision=None,

68 - quantization=quantization_arg)

60 + model_config = ModelConfig(model_path, quantization=quantization_arg)

61 found_quantization_type = model_config.quantization

62 except ValueError:

63 found_quantization_type = "ERROR"

tests/test_config.py

+95 -225

@@ -74,115 +74,116 @@ def test_update_config():

74 new_config3 = update_config(config3, {"a": "new_value"})

77 +# Can remove once --task option is fully deprecated

78 @pytest.mark.parametrize(

78 - ("model_id", "expected_runner_type", "expected_task"),

79 + ("model_id", "expected_runner_type", "expected_convert_type",

80 + "expected_task"),

81 [

80 - ("distilbert/distilgpt2", "generate", "generate"),

81 - ("intfloat/multilingual-e5-small", "pooling", "embed"),

82 - ("jason9693/Qwen2.5-1.5B-apeach", "pooling", "classify"),

83 - ("cross-encoder/ms-marco-MiniLM-L-6-v2", "pooling", "classify"),

84 - ("Qwen/Qwen2.5-Math-RM-72B", "pooling", "reward"),

85 - ("openai/whisper-small", "generate", "transcription"),

82 + ("distilbert/distilgpt2", "generate", "none", "generate"),

83 + ("intfloat/multilingual-e5-small", "pooling", "none", "embed"),

84 + ("jason9693/Qwen2.5-1.5B-apeach", "pooling", "classify", "classify"),

85 + ("cross-encoder/ms-marco-MiniLM-L-6-v2", "pooling", "none",

86 + "classify"),

87 + ("Qwen/Qwen2.5-Math-RM-72B", "pooling", "none", "reward"),

88 + ("openai/whisper-small", "generate", "none", "transcription"),

89 ],

90 )

88 -def test_auto_task(model_id, expected_runner_type, expected_task):

89 - config = ModelConfig(

90 - model_id,

91 - task="auto",

92 - tokenizer=model_id,

93 - tokenizer_mode="auto",

94 - trust_remote_code=False,

95 - seed=0,

96 - dtype="float16",

97 - )

91 +def test_auto_task(model_id, expected_runner_type, expected_convert_type,

92 + expected_task):

93 + config = ModelConfig(model_id, task="auto")

95 assert config.runner_type == expected_runner_type

96 + assert config.convert_type == expected_convert_type

97 + assert expected_task in config.supported_tasks

101 - if config.runner_type == "pooling":

102 - assert config.task == expected_task

103 - else:

104 - assert expected_task in config.supported_tasks

100 +# Can remove once --task option is fully deprecated

101 +@pytest.mark.parametrize(

102 + ("model_id", "expected_runner_type", "expected_convert_type",

103 + "expected_task"),

104 + [

105 + ("distilbert/distilgpt2", "pooling", "embed", "embed"),

106 + ("intfloat/multilingual-e5-small", "pooling", "embed", "embed"),

107 + ("jason9693/Qwen2.5-1.5B-apeach", "pooling", "classify", "classify"),

108 + ("cross-encoder/ms-marco-MiniLM-L-6-v2", "pooling", "classify",

109 + "classify"),

110 + ("Qwen/Qwen2.5-Math-RM-72B", "pooling", "embed", "embed"),

111 + ("openai/whisper-small", "pooling", "embed", "embed"),

112 + ],

113 +)

114 +def test_score_task(model_id, expected_runner_type, expected_convert_type,

115 + expected_task):

116 + config = ModelConfig(model_id, task="score")

117

118 + assert config.runner_type == expected_runner_type

119 + assert config.convert_type == expected_convert_type

120 + assert expected_task in config.supported_tasks

121 +

122 +

123 +# Can remove once --task option is fully deprecated

124 @pytest.mark.parametrize(

108 - ("model_id", "expected_runner_type", "expected_task"),

125 + ("model_id", "expected_runner_type", "expected_convert_type",

126 + "expected_task"),

127 [

110 - ("distilbert/distilgpt2", "pooling", "embed"),

111 - ("intfloat/multilingual-e5-small", "pooling", "embed"),

112 - ("jason9693/Qwen2.5-1.5B-apeach", "pooling", "classify"),

113 - ("cross-encoder/ms-marco-MiniLM-L-6-v2", "pooling", "classify"),

114 - ("Qwen/Qwen2.5-Math-RM-72B", "pooling", "embed"),

115 - ("openai/whisper-small", "pooling", "embed"),

128 + ("openai/whisper-small", "generate", "none", "transcription"),

129 ],

130 )

118 -def test_score_task(model_id, expected_runner_type, expected_task):

119 - config = ModelConfig(

120 - model_id,

121 - task="score",

122 - tokenizer=model_id,

123 - tokenizer_mode="auto",

124 - trust_remote_code=False,

125 - seed=0,

126 - dtype="float16",

127 - )

131 +def test_transcription_task(model_id, expected_runner_type,

132 + expected_convert_type, expected_task):

133 + config = ModelConfig(model_id, task="transcription")

134

135 assert config.runner_type == expected_runner_type

130 - assert config.task == expected_task

136 + assert config.convert_type == expected_convert_type

137 + assert expected_task in config.supported_tasks

138

139

133 -@pytest.mark.parametrize(("model_id", "expected_runner_type", "expected_task"),

134 - [

135 - ("Qwen/Qwen2.5-1.5B-Instruct", "draft", "auto"),

136 - ])

137 -def test_draft_task(model_id, expected_runner_type, expected_task):

138 - config = ModelConfig(

139 - model_id,

140 - runner="draft",

141 - tokenizer=model_id,

142 - seed=0,

143 - dtype="float16",

144 - )

140 +@pytest.mark.parametrize(

141 + ("model_id", "expected_runner_type", "expected_convert_type"),

142 + [

143 + ("distilbert/distilgpt2", "generate", "none"),

144 + ("intfloat/multilingual-e5-small", "pooling", "none"),

145 + ("jason9693/Qwen2.5-1.5B-apeach", "pooling", "classify"),

146 + ("cross-encoder/ms-marco-MiniLM-L-6-v2", "pooling", "none"),

147 + ("Qwen/Qwen2.5-Math-RM-72B", "pooling", "none"),

148 + ("openai/whisper-small", "generate", "none"),

149 + ],

150 +)

151 +def test_auto_runner(model_id, expected_runner_type, expected_convert_type):

152 + config = ModelConfig(model_id, runner="auto")

153

154 assert config.runner_type == expected_runner_type

147 - assert config.task == expected_task

155 + assert config.convert_type == expected_convert_type

156

157

158 @pytest.mark.parametrize(

151 - ("model_id", "expected_runner_type", "expected_task"),

159 + ("model_id", "expected_runner_type", "expected_convert_type"),

160 [

153 - ("openai/whisper-small", "generate", "transcription"),

161 + ("distilbert/distilgpt2", "pooling", "embed"),

162 + ("intfloat/multilingual-e5-small", "pooling", "none"),

163 + ("jason9693/Qwen2.5-1.5B-apeach", "pooling", "classify"),

164 + ("cross-encoder/ms-marco-MiniLM-L-6-v2", "pooling", "none"),

165 + ("Qwen/Qwen2.5-Math-RM-72B", "pooling", "none"),

166 + ("openai/whisper-small", "pooling", "embed"),

167 ],

168 )

156 -def test_transcription_task(model_id, expected_runner_type, expected_task):

157 - config = ModelConfig(

158 - model_id,

159 - task="transcription",

160 - tokenizer=model_id,

161 - tokenizer_mode="auto",

162 - trust_remote_code=False,

163 - seed=0,

164 - dtype="float16",

165 - )

169 +def test_pooling_runner(model_id, expected_runner_type, expected_convert_type):

170 + config = ModelConfig(model_id, runner="pooling")

171

172 assert config.runner_type == expected_runner_type

168 - assert config.task == expected_task

173 + assert config.convert_type == expected_convert_type

174

175

171 -@pytest.mark.parametrize(("model_id", "bad_task"), [

172 - ("Qwen/Qwen2.5-Math-RM-72B", "generate"),

173 - ("Qwen/Qwen3-0.6B", "transcription"),

174 -])

175 -def test_incorrect_task(model_id, bad_task):

176 - with pytest.raises(ValueError, match=r"does not support task=.*"):

177 - ModelConfig(

178 - model_id,

179 - task=bad_task,

180 - tokenizer=model_id,

181 - tokenizer_mode="auto",

182 - trust_remote_code=False,

183 - seed=0,

184 - dtype="float16",

185 - )

176 +@pytest.mark.parametrize(

177 + ("model_id", "expected_runner_type", "expected_convert_type"),

178 + [

179 + ("Qwen/Qwen2.5-1.5B-Instruct", "draft", "none"),

180 + ],

181 +)

182 +def test_draft_runner(model_id, expected_runner_type, expected_convert_type):

183 + config = ModelConfig(model_id, runner="draft")

184 +

185 + assert config.runner_type == expected_runner_type

186 + assert config.convert_type == expected_convert_type

187

188

189 MODEL_IDS_EXPECTED = [

@@ -195,17 +196,7 @@ MODEL_IDS_EXPECTED = [

196 @pytest.mark.parametrize("model_id_expected", MODEL_IDS_EXPECTED)

197 def test_disable_sliding_window(model_id_expected):

198 model_id, expected = model_id_expected

198 - model_config = ModelConfig(

199 - model_id,

200 - task="auto",

201 - tokenizer=model_id,

202 - tokenizer_mode="auto",

203 - trust_remote_code=False,

204 - seed=0,

205 - dtype="float16",

206 - revision=None,

207 - disable_sliding_window=True,

208 - )

199 + model_config = ModelConfig(model_id, disable_sliding_window=True)

200 assert model_config.max_model_len == expected

201

202

@@ -214,16 +205,7 @@ def test_get_sliding_window():

205 # Test that the sliding window is correctly computed.

206 # For Qwen1.5/Qwen2, get_sliding_window() should be None

207 # when use_sliding_window is False.

217 - qwen2_model_config = ModelConfig(

218 - "Qwen/Qwen1.5-7B",

219 - task="auto",

220 - tokenizer="Qwen/Qwen1.5-7B",

221 - tokenizer_mode="auto",

222 - trust_remote_code=False,

223 - seed=0,

224 - dtype="float16",

225 - revision=None,

226 - )

208 + qwen2_model_config = ModelConfig("Qwen/Qwen1.5-7B")

209

210 qwen2_model_config.hf_config.use_sliding_window = False

211 qwen2_model_config.hf_config.sliding_window = TEST_SLIDING_WINDOW

@@ -232,16 +214,7 @@ def test_get_sliding_window():

214 qwen2_model_config.hf_config.use_sliding_window = True

215 assert qwen2_model_config.get_sliding_window() == TEST_SLIDING_WINDOW

216

235 - mistral_model_config = ModelConfig(

236 - "mistralai/Mistral-7B-v0.1",

237 - task="auto",

238 - tokenizer="mistralai/Mistral-7B-v0.1",

239 - tokenizer_mode="auto",

240 - trust_remote_code=False,

241 - seed=0,

242 - dtype="float16",

243 - revision=None,

244 - )

217 + mistral_model_config = ModelConfig("mistralai/Mistral-7B-v0.1")

218 mistral_model_config.hf_config.sliding_window = None

219 assert mistral_model_config.get_sliding_window() is None

220

@@ -253,16 +226,7 @@ def test_get_sliding_window():

226 reason="Xformers backend is not supported on ROCm.")

227 def test_get_pooling_config():

228 model_id = "sentence-transformers/all-MiniLM-L12-v2"

256 - model_config = ModelConfig(

257 - model_id,

258 - task="auto",

259 - tokenizer=model_id,

260 - tokenizer_mode="auto",

261 - trust_remote_code=False,

262 - seed=0,

263 - dtype="float16",

264 - revision=None,

265 - )

229 + model_config = ModelConfig(model_id)

230

231 pooling_config = model_config._init_pooler_config()

232 assert pooling_config is not None

@@ -275,14 +239,7 @@ def test_get_pooling_config():

239 reason="Xformers backend is not supported on ROCm.")

240 def test_get_pooling_config_from_args():

241 model_id = "sentence-transformers/all-MiniLM-L12-v2"

278 - model_config = ModelConfig(model_id,

279 - task="auto",

280 - tokenizer=model_id,

281 - tokenizer_mode="auto",

282 - trust_remote_code=False,

283 - seed=0,

284 - dtype="float16",

285 - revision=None)

242 + model_config = ModelConfig(model_id)

243

244 override_pooler_config = PoolerConfig(pooling_type='CLS', normalize=True)

245 model_config.override_pooler_config = override_pooler_config

@@ -295,16 +252,8 @@ def test_get_pooling_config_from_args():

252 @pytest.mark.skipif(current_platform.is_rocm(),

253 reason="Xformers backend is not supported on ROCm.")

254 def test_get_bert_tokenization_sentence_transformer_config():

298 - bge_model_config = ModelConfig(

299 - model="BAAI/bge-base-en-v1.5",

300 - task="auto",

301 - tokenizer="BAAI/bge-base-en-v1.5",

302 - tokenizer_mode="auto",

303 - trust_remote_code=False,

304 - seed=0,

305 - dtype="float16",

306 - revision=None,

307 - )

255 + model_id = "BAAI/bge-base-en-v1.5"

256 + bge_model_config = ModelConfig(model_id)

257

258 bert_bge_model_config = bge_model_config._get_encoder_config()

259

@@ -317,27 +266,13 @@ def test_rope_customization():

266 TEST_ROPE_THETA = 16_000_000.0

267 LONGCHAT_ROPE_SCALING = {"rope_type": "linear", "factor": 8.0}

268

320 - llama_model_config = ModelConfig(

321 - "meta-llama/Meta-Llama-3-8B-Instruct",

322 - task="auto",

323 - tokenizer="meta-llama/Meta-Llama-3-8B-Instruct",

324 - tokenizer_mode="auto",

325 - trust_remote_code=False,

326 - dtype="float16",

327 - seed=0,

328 - )

269 + llama_model_config = ModelConfig("meta-llama/Meta-Llama-3-8B-Instruct")

270 assert getattr(llama_model_config.hf_config, "rope_scaling", None) is None

271 assert getattr(llama_model_config.hf_config, "rope_theta", None) == 500_000

272 assert llama_model_config.max_model_len == 8192

273

274 llama_model_config = ModelConfig(

275 "meta-llama/Meta-Llama-3-8B-Instruct",

335 - task="auto",

336 - tokenizer="meta-llama/Meta-Llama-3-8B-Instruct",

337 - tokenizer_mode="auto",

338 - trust_remote_code=False,

339 - dtype="float16",

340 - seed=0,

276 hf_overrides={

277 "rope_scaling": TEST_ROPE_SCALING,

278 "rope_theta": TEST_ROPE_THETA,

@@ -349,15 +284,7 @@ def test_rope_customization():

284 None) == TEST_ROPE_THETA

285 assert llama_model_config.max_model_len == 16384

286

352 - longchat_model_config = ModelConfig(

353 - "lmsys/longchat-13b-16k",

354 - task="auto",

355 - tokenizer="lmsys/longchat-13b-16k",

356 - tokenizer_mode="auto",

357 - trust_remote_code=False,

358 - dtype="float16",

359 - seed=0,

360 - )

287 + longchat_model_config = ModelConfig("lmsys/longchat-13b-16k")

288 # Check if LONGCHAT_ROPE_SCALING entries are in longchat_model_config

289 assert all(

290 longchat_model_config.hf_config.rope_scaling.get(key) == value

@@ -366,12 +293,6 @@ def test_rope_customization():

293

294 longchat_model_config = ModelConfig(

295 "lmsys/longchat-13b-16k",

369 - task="auto",

370 - tokenizer="lmsys/longchat-13b-16k",

371 - tokenizer_mode="auto",

372 - trust_remote_code=False,

373 - dtype="float16",

374 - seed=0,

296 hf_overrides={

297 "rope_scaling": TEST_ROPE_SCALING,

298 },

@@ -390,15 +311,7 @@ def test_rope_customization():

311 ("meta-llama/Llama-3.2-11B-Vision", True),

312 ])

313 def test_is_encoder_decoder(model_id, is_encoder_decoder):

393 - config = ModelConfig(

394 - model_id,

395 - task="auto",

396 - tokenizer=model_id,

397 - tokenizer_mode="auto",

398 - trust_remote_code=False,

399 - dtype="float16",

400 - seed=0,

401 - )

314 + config = ModelConfig(model_id)

315

316 assert config.is_encoder_decoder == is_encoder_decoder

317

@@ -408,15 +321,7 @@ def test_is_encoder_decoder(model_id, is_encoder_decoder):

321 ("Qwen/Qwen2-VL-2B-Instruct", True),

322 ])

323 def test_uses_mrope(model_id, uses_mrope):

411 - config = ModelConfig(

412 - model_id,

413 - task="auto",

414 - tokenizer=model_id,

415 - tokenizer_mode="auto",

416 - trust_remote_code=False,

417 - dtype="float16",

418 - seed=0,

419 - )

324 + config = ModelConfig(model_id)

325

326 assert config.uses_mrope == uses_mrope

327

@@ -426,26 +331,12 @@ def test_generation_config_loading():

331

332 # When set generation_config to "vllm", the default generation config

333 # will not be loaded.

429 - model_config = ModelConfig(model_id,

430 - task="auto",

431 - tokenizer=model_id,

432 - tokenizer_mode="auto",

433 - trust_remote_code=False,

434 - seed=0,

435 - dtype="float16",

436 - generation_config="vllm")

334 + model_config = ModelConfig(model_id, generation_config="vllm")

335 assert model_config.get_diff_sampling_param() == {}

336

337 # When set generation_config to "auto", the default generation config

338 # should be loaded.

441 - model_config = ModelConfig(model_id,

442 - task="auto",

443 - tokenizer=model_id,

444 - tokenizer_mode="auto",

445 - trust_remote_code=False,

446 - seed=0,

447 - dtype="float16",

448 - generation_config="auto")

339 + model_config = ModelConfig(model_id, generation_config="auto")

340

341 correct_generation_config = {

342 "repetition_penalty": 1.1,

@@ -461,12 +352,6 @@ def test_generation_config_loading():

352

353 model_config = ModelConfig(

354 model_id,

464 - task="auto",

465 - tokenizer=model_id,

466 - tokenizer_mode="auto",

467 - trust_remote_code=False,

468 - seed=0,

469 - dtype="float16",

355 generation_config="auto",

356 override_generation_config=override_generation_config)

357

@@ -479,12 +364,6 @@ def test_generation_config_loading():

364 # is set, the override_generation_config should be used directly.

365 model_config = ModelConfig(

366 model_id,

482 - task="auto",

483 - tokenizer=model_id,

484 - tokenizer_mode="auto",

485 - trust_remote_code=False,

486 - seed=0,

487 - dtype="float16",

367 generation_config="vllm",

368 override_generation_config=override_generation_config)

369

@@ -515,16 +394,7 @@ def test_load_config_pt_load_map_location(pt_load_map_location):

394 def test_get_and_verify_max_len(model_id, max_model_len, expected_max_len,

395 should_raise):

396 """Test get_and_verify_max_len with different configurations."""

518 - model_config = ModelConfig(

519 - model_id,

520 - task="auto",

521 - tokenizer=model_id,

522 - tokenizer_mode="auto",

523 - trust_remote_code=False,

524 - seed=0,

525 - dtype="float16",

526 - revision=None,

527 - )

397 + model_config = ModelConfig(model_id)

398

399 if should_raise:

400 with pytest.raises(ValueError):

tests/test_sampling_params.py

-5

@@ -21,13 +21,8 @@ def test_max_tokens_none():

21 def model_config():

22 return ModelConfig(

23 MODEL_NAME,

24 - task="auto",

25 - tokenizer=MODEL_NAME,

26 - tokenizer_mode="auto",

27 - trust_remote_code=False,

24 seed=0,

25 dtype="float16",

30 - revision=None,

26 )

tests/v1/core/test_kv_cache_utils.py

+2 -10

@@ -695,11 +695,7 @@ def test_estimate_max_model_len(model_id, max_model_len,

695 # Create a VllmConfig

696 model_config = ModelConfig(

697 model_id,

698 - task="generate",

699 - tokenizer=model_id,

700 - tokenizer_mode="auto",

701 - trust_remote_code=False,

702 - seed=0,

698 + runner="generate",

699 dtype="float16",

700 max_model_len=max_model_len,

701 )

@@ -733,11 +729,7 @@ def test_get_max_concurrency_for_kv_cache_config():

729 max_model_len = 16384

730 model_config = ModelConfig(

731 model_id,

736 - task="generate",

737 - tokenizer=model_id,

738 - tokenizer_mode="auto",

739 - trust_remote_code=False,

740 - seed=0,

732 + runner="generate",

733 dtype="float16",

734 max_model_len=max_model_len,

735 )

tests/v1/core/test_scheduler.py

-3

@@ -1248,9 +1248,6 @@ def create_scheduler_with_priority(

1248 )

1249 model_config = ModelConfig(

1250 model=model,

1251 - task="auto",

1252 - tokenizer=model,

1253 - tokenizer_mode="auto",

1251 trust_remote_code=True,

1252 dtype="float16",

1253 seed=42,

tests/v1/core/utils.py

-3

@@ -59,9 +59,6 @@ def create_scheduler(

59 )

60 model_config = ModelConfig(

61 model=model,

62 - task="auto",

63 - tokenizer=model,

64 - tokenizer_mode="auto",

62 trust_remote_code=True,

63 dtype="float16",

64 seed=42,

tests/v1/kv_connector/unit/utils.py

-3

@@ -68,9 +68,6 @@ def create_vllm_config(

68 )

69 model_config = ModelConfig(

70 model=model,

71 - task="auto",

72 - tokenizer=model,

73 - tokenizer_mode="auto",

71 trust_remote_code=True,

72 dtype="float16",

73 seed=42,

tests/v1/spec_decode/test_eagle.py

+2 -7

@@ -24,13 +24,8 @@ eagle3_dir = "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B"

25 def _create_proposer(method: str, k: int) -> EagleProposer:

26 model_config = ModelConfig(model=model_dir,

27 - task="generate",

28 - max_model_len=100,

29 - tokenizer=model_dir,

30 - tokenizer_mode="auto",

31 - dtype="auto",

32 - seed=None,

33 - trust_remote_code=False)

27 + runner="generate",

28 + max_model_len=100)

30 # Choose model directory based on method

31 draft_model_dir = eagle_dir if method == "eagle" else eagle3_dir

tests/v1/spec_decode/test_ngram.py

+1 -8

@@ -44,14 +44,7 @@ def test_ngram_proposer():

45 def ngram_proposer(min_n: int, max_n: int, k: int) -> NgramProposer:

46 # Dummy model config. Just to set max_model_len.

47 - model_config = ModelConfig(model="facebook/opt-125m",

48 - task="generate",

49 - max_model_len=100,

50 - tokenizer="facebook/opt-125m",

51 - tokenizer_mode="auto",

52 - dtype="auto",

53 - seed=None,

54 - trust_remote_code=False)

47 + model_config = ModelConfig(model="facebook/opt-125m")

48 return NgramProposer(

49 vllm_config=VllmConfig(model_config=model_config,

50 speculative_config=SpeculativeConfig.

tests/v1/tpu/worker/test_tpu_model_runner.py

-4

@@ -26,10 +26,6 @@ def get_vllm_config():

26 )

27 model_config = ModelConfig(

28 model="facebook/opt-125m",

29 - task="generate",

30 - tokenizer="facebook/opt-125m",

31 - tokenizer_mode="auto",

32 - trust_remote_code=True,

29 dtype="bfloat16", # TPUs typically use bfloat16

30 seed=42,

31 )

tests/v1/worker/test_gpu_model_runner.py

-4

@@ -76,10 +76,6 @@ def get_vllm_config():

76 )

77 model_config = ModelConfig(

78 model="facebook/opt-125m",

79 - task="generate",

80 - tokenizer="facebook/opt-125m",

81 - tokenizer_mode="auto",

82 - trust_remote_code=True,

79 dtype="float16",

80 seed=42,

81 )

vllm/config.py

+321 -205

@@ -26,7 +26,7 @@ from pydantic import (ConfigDict, SkipValidation, TypeAdapter, field_validator,

26 from pydantic.dataclasses import dataclass

27 from safetensors.torch import _TYPES as _SAFETENSORS_TO_TORCH_DTYPE

28 from torch.distributed import ProcessGroup, ReduceOp

29 -from typing_extensions import Self, runtime_checkable

29 +from typing_extensions import Self, assert_never, runtime_checkable

31 import vllm.envs as envs

32 from vllm import version

@@ -102,12 +102,63 @@ RunnerOption = Literal["auto", "generate", "pooling", "draft"]

102

103 RunnerType = Literal["generate", "pooling", "draft"]

104

105 -_RUNNER_TASKS: dict[RunnerType, list[_ResolvedTask]] = {

105 +ConvertOption = Literal["auto", "none", "embed", "classify", "reward"]

106 +

107 +ConvertType = Literal["none", "embed", "classify", "reward"]

108 +

109 +_RUNNER_TASKS: dict[RunnerType, list[TaskOption]] = {

110 "generate": ["generate", "transcription"],

107 - "pooling": ["encode", "embed", "classify", "reward"],

111 + "pooling": ["embedding", "embed", "classify", "score", "reward"],

112 + "draft": ["draft"],

113 +}

114 +

115 +_RUNNER_CONVERTS: dict[RunnerType, list[ConvertType]] = {

116 + "generate": [],

117 + "pooling": ["embed", "classify", "reward"],

118 "draft": [],

119 }

120

121 +# Some model suffixes are based on auto classes from Transformers:

122 +# https://huggingface.co/docs/transformers/en/model_doc/auto

123 +# NOTE: Items higher on this list priority over lower ones

124 +_SUFFIX_TO_DEFAULTS: list[tuple[str, tuple[RunnerType, ConvertType]]] = [

125 + ("ForCausalLM", ("generate", "none")),

126 + ("ForConditionalGeneration", ("generate", "none")),

127 + ("ChatModel", ("generate", "none")),

128 + ("LMHeadModel", ("generate", "none")),

129 + ("ForTextEncoding", ("pooling", "embed")),

130 + ("EmbeddingModel", ("pooling", "embed")),

131 + ("ForSequenceClassification", ("pooling", "classify")),

132 + ("ForAudioClassification", ("pooling", "classify")),

133 + ("ForImageClassification", ("pooling", "classify")),

134 + ("ForVideoClassification", ("pooling", "classify")),

135 + ("ClassificationModel", ("pooling", "classify")),

136 + ("ForRewardModeling", ("pooling", "reward")),

137 + ("RewardModel", ("pooling", "reward")),

138 + # Let other `*Model`s take priority

139 + ("Model", ("pooling", "embed")),

140 +]

141 +

142 +

143 +def iter_architecture_defaults():

144 + yield from _SUFFIX_TO_DEFAULTS

145 +

146 +

147 +def try_match_architecture_defaults(

148 + architecture: str,

149 + *,

150 + runner_type: Optional[RunnerType] = None,

151 + convert_type: Optional[ConvertType] = None,

152 +) -> Optional[tuple[str, tuple[RunnerType, ConvertType]]]:

153 + for suffix, (default_runner_type,

154 + default_convert_type) in iter_architecture_defaults():

155 + if ((runner_type is None or runner_type == default_runner_type) and

156 + (convert_type is None or convert_type == default_convert_type)

157 + and architecture.endswith(suffix)):

158 + return suffix, (default_runner_type, default_convert_type)

159 +

160 + return None

161 +

162

163 @runtime_checkable

164 class SupportsHash(Protocol):

@@ -236,11 +287,16 @@ class ModelConfig:

287 runner: RunnerOption = "auto"

288 """The type of model runner to use. Each vLLM instance only supports one

289 model runner, even if the same model can be used for multiple types."""

239 - task: TaskOption = "auto"

240 - """The task to use the model for. If the model supports more than one

241 - model runner, this is used to select which model runner to run.

242 -

243 - Note that the model may support other tasks using the same model runner."""

290 + convert: ConvertOption = "auto"

291 + """Convert the model using adapters defined in

292 + [vllm.model_executor.models.adapters][]. The most common use case is to

293 + adapt a text generation model to be used for pooling tasks."""

294 + task: Optional[TaskOption] = None

295 + """[DEPRECATED] The task to use the model for. If the model supports more

296 + than one model runner, this is used to select which model runner to run.

297 +

298 + Note that the model may support other tasks using the same model runner.

299 + """

300 tokenizer: SkipValidation[str] = None # type: ignore

301 """Name or path of the Hugging Face tokenizer to use. If unspecified, model

302 name or path will be used."""

@@ -558,48 +614,103 @@ class ModelConfig:

614 self.hf_image_processor_config = get_hf_image_processor_config(

615 self.model, hf_token=self.hf_token, revision=self.revision)

616

561 - # For pooling models, self.task is used to indicate the

562 - # user-selected task

563 - if self.task == "score":

564 - if self._is_classify_task(self.architectures):

565 - self.task = "classify"

617 + architectures = self.architectures

618 + registry = self.registry

619 + is_generative_model = registry.is_text_generation_model(

620 + architectures, self)

621 + is_pooling_model = registry.is_pooling_model(architectures, self)

622 +

623 + def _task_to_convert(task: TaskOption) -> ConvertType:

624 + if task == "embedding" or task == "embed":

625 + return "embed"

626 + if task == "classify":

627 + return "classify"

628 + if task == "reward":

629 + return "reward"

630 + if task == "score":

631 + new_task = self._get_default_pooling_task(architectures)

632 + return "classify" if new_task == "classify" else "embed"

633 +

634 + return "none"

635 +

636 + if self.task is not None:

637 + runner: RunnerOption = "auto"

638 + convert: ConvertOption = "auto"

639 + msg_prefix = ("The 'task' option has been deprecated and will be "

640 + "removed in v0.13.0 or v1.0, whichever comes first.")

641 + msg_hint = "Please remove this option."

642 +

643 + is_generative_task = self.task in _RUNNER_TASKS["generate"]

644 + is_pooling_task = self.task in _RUNNER_TASKS["pooling"]

645 +

646 + if is_generative_model and is_pooling_model:

647 + if is_generative_task:

648 + runner = "generate"

649 + convert = "auto"

650 + msg_hint = ("Please replace this option with `--runner "

651 + "generate` to continue using this model "

652 + "as a generative model.")

653 + elif is_pooling_task:

654 + runner = "pooling"

655 + convert = "auto"

656 + msg_hint = ("Please replace this option with `--runner "

657 + "pooling` to continue using this model "

658 + "as a pooling model.")

659 + else: # task == "auto"

660 + pass

661 + elif is_generative_model or is_pooling_model:

662 + if is_generative_task:

663 + runner = "generate"

664 + convert = "auto"

665 + msg_hint = "Please remove this option"

666 + elif is_pooling_task:

667 + runner = "pooling"

668 + convert = _task_to_convert(self.task)

669 + msg_hint = ("Please replace this option with `--convert "

670 + f"{convert}` to continue using this model "

671 + "as a pooling model.")

672 + else: # task == "auto"

673 + pass

674 else:

567 - self.task = "embed"

568 - elif self.task == "embedding":

569 - msg = ("The 'embedding' task has been renamed to 'embed', please "

570 - "use the new name. The old name will be removed in v1.0.")

675 + raise AssertionError("The model should be a generative or "

676 + "pooling model when task is set to "

677 + f"{self.task!r}.")

678 +

679 + self.runner = runner

680 + self.convert = convert

681 +

682 + msg = f"{msg_prefix} {msg_hint}"

683 warnings.warn(msg, DeprecationWarning, stacklevel=2)

684

573 - self.task = "embed"

685 + self.runner_type = self._get_runner_type(architectures, self.runner)

686 + self.convert_type = self._get_convert_type(architectures,

687 + self.runner_type,

688 + self.convert)

689 +

690 + if self.runner_type == "generate" and not is_generative_model:

691 + generate_converts = _RUNNER_CONVERTS["generate"]

692 + if self.convert_type not in generate_converts:

693 + # Currently we don't have any converters for generative models

694 + raise ValueError(

695 + "This model does not support `--runner generate`.")

696 + if self.runner_type == "pooling" and not is_pooling_model:

697 + pooling_converts = _RUNNER_CONVERTS["pooling"]

698 + if self.convert_type not in pooling_converts:

699 + convert_option = "<" + "|".join(pooling_converts) + ">"

700 + raise ValueError(

701 + "This model does not support `--runner pooling`. "

702 + f"You can pass `--convert {convert_option} to adapt "

703 + "it into a pooling model.")

704

575 - model_info, arch = self.registry.inspect_model_cls(self.architectures)

705 + self.supported_tasks = self._get_supported_tasks(

706 + architectures, self.runner_type, self.convert_type)

707 +

708 + # Note: Initialize these attributes early because transformers fallback

709 + # may fail to load dynamic modules in child processes

710 + model_info, arch = registry.inspect_model_cls(architectures, self)

711 self._model_info = model_info

712 self._architecture = arch

578 -

579 - all_supported_tasks = self._get_supported_tasks(self.task)

580 - logger.debug("Tasks supported by runner type: %s", all_supported_tasks)

581 - supported_runner_types = self._get_supported_runner_types(

582 - all_supported_tasks)

583 - runner_type = self._resolve_runner(self.runner, self.task,

584 - supported_runner_types,

585 - all_supported_tasks)

586 -

587 - logger.debug("Selected runner type: %s", runner_type)

588 - # For pooling models, self.task is used to indicate the

589 - # user-selected task

590 - if runner_type == "pooling" and self.task == "auto":

591 - selected_task = all_supported_tasks[runner_type][-1]

592 - assert selected_task != "encode"

593 - self.task = selected_task

594 - self.supported_runner_types = supported_runner_types

595 - self.runner_type = runner_type

596 - self.supported_tasks = all_supported_tasks[runner_type]

597 -

598 - if self.runner_type in ("draft",

599 - "generate") and self.task != "transcription":

600 - self.truncation_side = "left"

601 - else:

602 - self.truncation_side = "right"

713 + logger.info("Resolved architecture: %s", arch)

714

715 self.pooler_config = self._init_pooler_config()

716

@@ -652,16 +763,10 @@ class ModelConfig:

763 self.original_max_model_len = self.max_model_len

764 self.max_model_len = self.get_and_verify_max_len(self.max_model_len)

765 self.multimodal_config = self._init_multimodal_config()

655 - self.model_supports_multimodal_raw_input = (

656 - self.registry.supports_multimodal_raw_input(self.architectures))

766 +

767 if not self.skip_tokenizer_init:

768 self._verify_tokenizer_mode()

769

660 - self.is_attention_free = self._init_attention_free()

661 - self.is_hybrid = self._init_is_hybrid()

662 - self.has_noops = self._init_has_noops()

663 - self.has_inner_state = self._init_has_inner_state()

664 -

770 if (not current_platform.is_neuron() and self.override_neuron_config):

771 raise ValueError(

772 "`override_neuron_config` is only supported on Neuron.")

@@ -702,30 +807,13 @@ class ModelConfig:

807

808 @property

809 def architectures(self) -> list[str]:

705 - # architectures in the model config.

706 - architectures = getattr(self.hf_config, "architectures", [])

707 - # The registry assumes that it can always inspect the vLLM model class

708 - # for a given architecture. This assumption breaks down for the

709 - # Transformers backend, which may use a different class depending on

710 - # the model type. To work around this, we add the correct Transformers

711 - # backend class to the architectures list. We must do this here because

712 - # we need access to the `hf_config` to determine the backend class.

713 - transformers_backend_cls = self._get_transformers_backend_cls()

714 - if (self.model_impl != ModelImpl.VLLM.value

715 - and all(arch != transformers_backend_cls

716 - for arch in architectures)):

717 - architectures.append(transformers_backend_cls)

718 - return architectures

810 + return getattr(self.hf_config, "architectures", [])

811

812 @property

813 def architecture(self) -> str:

722 - # The architecture vllm actually used.

814 + """The architecture vllm actually used."""

815 return self._architecture

816

725 - @property

726 - def model_info(self):

727 - return self._model_info

728 -

817 def maybe_pull_model_tokenizer_for_s3(self, model: str,

818 tokenizer: str) -> None:

819 """Pull model/tokenizer from S3 to temporary directory when needed.

@@ -763,7 +851,7 @@ class ModelConfig:

851 self.tokenizer = s3_tokenizer.dir

852

853 def _init_multimodal_config(self) -> Optional["MultiModalConfig"]:

766 - if self.registry.is_multimodal_model(self.architectures):

854 + if self.registry.is_multimodal_model(self.architectures, self):

855 return MultiModalConfig(

856 limit_per_prompt=self.limit_mm_per_prompt,

857 media_io_kwargs=self.media_io_kwargs,

@@ -819,19 +907,6 @@ class ModelConfig:

907

908 return None

909

822 - def _init_attention_free(self) -> bool:

823 - return self.registry.is_attention_free_model(self.architectures)

824 -

825 - def _init_is_hybrid(self) -> bool:

826 - return self.registry.is_hybrid_model(self.architectures)

827 -

828 - def _init_has_noops(self) -> bool:

829 - architectures = getattr(self.hf_config, "architectures", [])

830 - return self.registry.is_noops_model(architectures)

831 -

832 - def _init_has_inner_state(self) -> bool:

833 - return self.registry.model_has_inner_state(self.architectures)

834 -

910 def _verify_tokenizer_mode(self) -> None:

911 tokenizer_mode = cast(TokenizerMode, self.tokenizer_mode.lower())

912 if tokenizer_mode not in get_args(TokenizerMode):

@@ -840,155 +915,168 @@ class ModelConfig:

915 f"one of {get_args(TokenizerMode)}.")

916 self.tokenizer_mode = tokenizer_mode

917

843 - def _is_classify_task(self, architectures: list[str]):

918 + def _get_default_runner_type(

919 + self,

920 + architectures: list[str],

921 + ) -> RunnerType:

922 + registry = self.registry

923 +

924 + # Some Sentence Transformers models use *ForCausalLM archs

925 + if get_pooling_config(self.model, self.revision):

926 + return "pooling"

927 +

928 for arch in architectures:

845 - if arch.endswith("ForSequenceClassification"):

846 - return True

847 - return self.registry.is_cross_encoder_model(architectures)

929 + if arch in registry.get_supported_archs():

930 + if registry.is_pooling_model(architectures, self):

931 + return "pooling"

932 + if registry.is_text_generation_model(architectures, self):

933 + return "generate"

934 +

935 + match = try_match_architecture_defaults(arch)

936 + if match:

937 + _, (runner_type, _) = match

938 + return runner_type

939 +

940 + return "generate"

941

849 - def _get_preferred_pooling_task(

942 + def _get_runner_type(

943 self,

944 architectures: list[str],

852 - ) -> _ResolvedTask:

853 - model_id = self.model

854 - if get_pooling_config(model_id, self.revision):

945 + runner: RunnerOption,

946 + ) -> RunnerType:

947 + if runner != "auto":

948 + return runner

949 +

950 + runner_type = self._get_default_runner_type(architectures)

951 +

952 + logger.info(

953 + "Resolved `--runner auto` to `--runner %s`. "

954 + "Pass the value explicitly to silence this message.", runner_type)

955 +

956 + return runner_type

957 +

958 + def _get_default_convert_type(

959 + self,

960 + architectures: list[str],

961 + runner_type: RunnerType,

962 + ) -> ConvertType:

963 + registry = self.registry

964 +

965 + for arch in architectures:

966 + if arch in registry.get_supported_archs():

967 + if (runner_type == "generate"

968 + and registry.is_text_generation_model(

969 + architectures, self)):

970 + return "none"

971 + if (runner_type == "pooling"

972 + and registry.is_pooling_model(architectures, self)):

973 + return "none"

974 +

975 + match = try_match_architecture_defaults(arch,

976 + runner_type=runner_type)

977 + if match:

978 + _, (_, convert_type) = match

979 + return convert_type

980 +

981 + # This is to handle Sentence Transformers models that use *ForCausalLM

982 + # and also multi-modal pooling models which are not defined as

983 + # Sentence Transformers models

984 + if runner_type == "pooling":

985 return "embed"

856 - if self.registry.is_transcription_model(architectures):

857 - return "transcription"

986

859 - suffix_to_preferred_task: list[tuple[str, _ResolvedTask]] = [

860 - # Other models follow this pattern

861 - ("EmbeddingModel", "embed"),

862 - ("RewardModel", "reward"),

863 - ]

987 + return "none"

988 +

989 + def _get_convert_type(

990 + self,

991 + architectures: list[str],

992 + runner_type: RunnerType,

993 + convert: ConvertOption,

994 + ) -> ConvertType:

995 + if convert != "auto":

996 + return convert

997

865 - for suffix, pref_task in suffix_to_preferred_task:

866 - if self.architecture.endswith(suffix):

867 - return pref_task

998 + convert_type = self._get_default_convert_type(architectures,

999 + runner_type)

1000

869 - return "embed"

1001 + logger.info(

1002 + "Resolved `--convert auto` to `--convert %s`. "

1003 + "Pass the value explicitly to silence this message.", convert_type)

1004 +

1005 + return convert_type

1006

1007 def _get_supported_generation_tasks(

1008 self,

873 - task_option: TaskOption,

1009 + architectures: list[str],

1010 + convert_type: ConvertType,

1011 ) -> list[_ResolvedTask]:

1012 registry = self.registry

876 - architectures = self.architectures

1013

878 - if registry.is_transcription_only_model(architectures):

1014 + if registry.is_transcription_only_model(architectures, self):

1015 return ["transcription"]

1016

1017 + # TODO: Use get_supported_generation_tasks once V0 is removed

1018 supported_tasks = list[_ResolvedTask]()

882 - if registry.is_text_generation_model(architectures):

1019 + if (registry.is_text_generation_model(architectures, self)

1020 + or convert_type in _RUNNER_CONVERTS["generate"]):

1021 supported_tasks.append("generate")

1022

885 - if registry.is_transcription_model(architectures):

886 - supported_tasks.append("transcription")

1023 + if registry.is_transcription_model(architectures, self):

1024 + supported_tasks.append("transcription")

1025

1026 return supported_tasks

1027

1028 + def _get_default_pooling_task(

1029 + self,

1030 + architectures: list[str],

1031 + ) -> Literal["embed", "classify", "reward"]:

1032 + if self.registry.is_cross_encoder_model(architectures, self):

1033 + return "classify"

1034 +

1035 + for arch in architectures:

1036 + match = try_match_architecture_defaults(arch,

1037 + runner_type="pooling")

1038 + if match:

1039 + _, (_, convert_type) = match

1040 + assert convert_type != "none"

1041 + return convert_type

1042 +

1043 + return "embed"

1044 +

1045 def _get_supported_pooling_tasks(

1046 self,

892 - task_option: TaskOption,

1047 + architectures: list[str],

1048 + convert_type: ConvertType,

1049 ) -> list[_ResolvedTask]:

1050 registry = self.registry

895 - architectures = self.architectures

1051

1052 + # TODO: Use get_supported_pooling_tasks once V0 is removed

1053 supported_tasks = list[_ResolvedTask]()

898 - if registry.is_pooling_model(architectures):

1054 + if (registry.is_pooling_model(architectures, self)

1055 + or convert_type in _RUNNER_CONVERTS["pooling"]):

1056 supported_tasks.append("encode")

1057

901 - # For now, users must specify the task (other than "pooling")

902 - # to use for pooling models

903 - if task_option == "auto":

904 - preferred_task = self._get_preferred_pooling_task(

905 - architectures)

906 -

907 - supported_tasks.append(preferred_task)

908 - elif task_option in _RUNNER_TASKS["pooling"]:

909 - supported_tasks.append(cast(_ResolvedTask, task_option))

1058 + extra_task = (self._get_default_pooling_task(architectures)

1059 + if convert_type == "none" else convert_type)

1060 + supported_tasks.append(extra_task)

1061

1062 return supported_tasks

1063

1064 def _get_supported_tasks(

1065 self,

915 - task_option: TaskOption,

916 - ) -> dict[RunnerType, list[_ResolvedTask]]:

917 - if self._is_classify_task(self.architectures):

918 - return {"generate": [], "pooling": ["classify"], "draft": []}

919 - else:

920 - return {

921 - "generate": self._get_supported_generation_tasks(task_option),

922 - "pooling": self._get_supported_pooling_tasks(task_option),

923 - "draft": ["draft"]

924 - }

925 -

926 - def _get_supported_runner_types(

927 - self,

928 - supported_tasks: dict[RunnerType, list[_ResolvedTask]],

929 - ) -> set[RunnerType]:

930 - return {

931 - runner

932 - for runner, runner_tasks in supported_tasks.items()

933 - if len(runner_tasks) > 0

934 - }

935 -

936 - def _resolve_runner(

937 - self,

938 - runner_option: RunnerOption,

939 - task_option: TaskOption,

940 - supported_runner_types: set[RunnerType],

941 - supported_tasks: dict[RunnerType, list[_ResolvedTask]],

942 - ) -> RunnerType:

943 - if not supported_runner_types:

944 - raise ValueError("This model does not support any model runners!")

945 -

946 - if runner_option != "auto":

947 - if runner_option not in supported_runner_types:

948 - raise ValueError(

949 - f"This model does not support runner={runner_option!r}. "

950 - f"Available runners: {supported_runner_types}")

951 -

952 - return runner_option

953 -

954 - if task_option != "auto":

955 - for runner, runner_tasks in supported_tasks.items():

956 - if task_option in runner_tasks:

957 - return runner

958 - else:

959 - task_runner: RunnerType = next(

960 - runner for runner, tasks in _RUNNER_TASKS.items()

961 - if task_option in tasks)

962 - raise ValueError(

963 - f"This model does not support task={task_option!r}. "

964 - f"Available tasks for runner={task_runner!r}: "

965 - f"{supported_tasks[task_runner]}")

966 -

967 - if "classify" in supported_tasks.get("pooling", []):

968 - # When multiple pooling tasks are present, default to

969 - # pooling (eg cross-encoder) for non-standard architectures.

970 - return "pooling"

971 -

972 - suffix_to_preferred_runner: list[tuple[str, RunnerType]] = [

973 - ("ForCausalLM", "generate"),

974 - ("ForConditionalGeneration", "generate"),

975 - ("ChatModel", "generate"),

976 - ("LMHeadModel", "generate"),

977 - ("EmbeddingModel", "pooling"),

978 - ("RewardModel", "pooling"),

979 - ]

980 -

981 - for suffix, pref_runner in suffix_to_preferred_runner:

982 - if self.architecture.endswith(

983 - suffix) and pref_runner in supported_runner_types:

984 - return pref_runner

985 -

986 - if "generate" in supported_runner_types:

987 - return "generate"

988 - if "pooling" in supported_runner_types:

989 - return "pooling"

1066 + architectures: list[str],

1067 + runner_type: RunnerType,

1068 + convert_type: ConvertType,

1069 + ) -> list[_ResolvedTask]:

1070 + if runner_type == "generate":

1071 + return self._get_supported_generation_tasks(

1072 + architectures, convert_type)

1073 + if runner_type == "pooling":

1074 + return self._get_supported_pooling_tasks(architectures,

1075 + convert_type)

1076 + if runner_type == "draft":

1077 + return ["draft"]

1078

991 - raise AssertionError("This line should not be reached")

1079 + assert_never(runner_type)

1080

1081 def _parse_quant_hf_config(self):

1082 quant_cfg = getattr(self.hf_config, "quantization_config", None)

@@ -1216,7 +1304,8 @@ class ModelConfig:

1304

1305 pipeline_parallel_size = parallel_config.pipeline_parallel_size

1306 if pipeline_parallel_size > 1:

1219 - if not self.registry.is_pp_supported_model(self.architectures):

1307 + if not self.registry.is_pp_supported_model(self.architectures,

1308 + self):

1309 raise NotImplementedError(

1310 "Pipeline parallelism is not supported for this model. "

1311 "Supported models implement the `SupportsPP` interface.")

@@ -1558,16 +1647,40 @@ class ModelConfig:

1647

1648 @property

1649 def is_cross_encoder(self) -> bool:

1561 - return self.task == "classify"

1650 + return (self._model_info.supports_cross_encoding

1651 + or self.convert_type == "classify")

1652

1653 @property

1564 - def use_mla(self) -> bool:

1565 - return self.is_deepseek_mla and not envs.VLLM_MLA_DISABLE

1654 + def is_pp_supported(self) -> bool:

1655 + return self._model_info.supports_pp

1656 +

1657 + @property

1658 + def is_multimodal_raw_input_supported(self) -> bool:

1659 + return self._model_info.supports_multimodal_raw_input

1660 +

1661 + @property

1662 + def is_attention_free(self) -> bool:

1663 + return self._model_info.is_attention_free

1664 +

1665 + @property

1666 + def is_hybrid(self) -> bool:

1667 + return self._model_info.is_hybrid

1668 +

1669 + @property

1670 + def has_noops(self) -> bool:

1671 + return self._model_info.has_noops

1672 +

1673 + @property

1674 + def has_inner_state(self):

1675 + return self._model_info.has_inner_state

1676

1677 @property

1678 def is_v1_compatible(self) -> bool:

1569 - architectures = getattr(self.hf_config, "architectures", [])

1570 - return me_models.ModelRegistry.is_v1_compatible(architectures)

1679 + return not self._model_info.supports_v0_only

1680 +

1681 + @property

1682 + def use_mla(self) -> bool:

1683 + return self.is_deepseek_mla and not envs.VLLM_MLA_DISABLE

1684

1685 @property

1686 def is_matryoshka(self) -> bool:

@@ -4769,7 +4882,10 @@ class VllmConfig:

4882 self.scheduler_config.max_model_len = max_model_len

4883

4884 def try_verify_and_update_config(self):

4772 - architecture = getattr(self.model_config, "architecture", None)

4885 + if self.model_config is None:

4886 + return

4887 +

4888 + architecture = self.model_config.architecture

4889 if architecture is None:

4890 return

4891

@@ -4782,7 +4898,7 @@ class VllmConfig:

4898 if self.model_config.is_hybrid:

4899 HybridAttentionMambaModelConfig.verify_and_update_config(self)

4900

4785 - if self.model_config.task == "classify":

4901 + if self.model_config.convert_type == "classify":

4902 # Maybe convert ForCausalLM into ForSequenceClassification model.

4903 from vllm.model_executor.models.adapters import (

4904 SequenceClassificationConfig)

vllm/engine/arg_utils.py

+19 -10

@@ -22,14 +22,15 @@ from typing_extensions import TypeIs

23 import vllm.envs as envs

24 from vllm.config import (BlockSize, CacheConfig, CacheDType, CompilationConfig,

25 - ConfigFormat, ConfigType, DecodingConfig,

26 - DetailedTraceModules, Device, DeviceConfig,

27 - DistributedExecutorBackend, GuidedDecodingBackend,

28 - GuidedDecodingBackendV1, HfOverrides, KVEventsConfig,

29 - KVTransferConfig, LoadConfig, LogprobsMode,

30 - LoRAConfig, ModelConfig, ModelDType, ModelImpl,

31 - MultiModalConfig, ObservabilityConfig, ParallelConfig,

32 - PoolerConfig, PrefixCachingHashAlgo, SchedulerConfig,

25 + ConfigFormat, ConfigType, ConvertOption,

26 + DecodingConfig, DetailedTraceModules, Device,

27 + DeviceConfig, DistributedExecutorBackend,

28 + GuidedDecodingBackend, GuidedDecodingBackendV1,

29 + HfOverrides, KVEventsConfig, KVTransferConfig,

30 + LoadConfig, LogprobsMode, LoRAConfig, ModelConfig,

31 + ModelDType, ModelImpl, MultiModalConfig,

32 + ObservabilityConfig, ParallelConfig, PoolerConfig,

33 + PrefixCachingHashAlgo, RunnerOption, SchedulerConfig,

34 SchedulerPolicy, SpeculativeConfig, TaskOption,

35 TokenizerMode, VllmConfig, get_attr_docs, get_field)

36 from vllm.logger import init_logger

@@ -270,7 +271,9 @@ class EngineArgs:

271 str, List[str]]] = ModelConfig.served_model_name

272 tokenizer: Optional[str] = ModelConfig.tokenizer

273 hf_config_path: Optional[str] = ModelConfig.hf_config_path

273 - task: TaskOption = ModelConfig.task

274 + runner: RunnerOption = ModelConfig.runner

275 + convert: ConvertOption = ModelConfig.convert

276 + task: Optional[TaskOption] = ModelConfig.task

277 skip_tokenizer_init: bool = ModelConfig.skip_tokenizer_init

278 enable_prompt_embeds: bool = ModelConfig.enable_prompt_embeds

279 tokenizer_mode: TokenizerMode = ModelConfig.tokenizer_mode

@@ -461,7 +464,11 @@ class EngineArgs:

464 )

465 if not ('serve' in sys.argv[1:] and '--help' in sys.argv[1:]):

466 model_group.add_argument("--model", **model_kwargs["model"])

464 - model_group.add_argument("--task", **model_kwargs["task"])

467 + model_group.add_argument("--runner", **model_kwargs["runner"])

468 + model_group.add_argument("--convert", **model_kwargs["convert"])

469 + model_group.add_argument("--task",

470 + **model_kwargs["task"],

471 + deprecated=True)

472 model_group.add_argument("--tokenizer", **model_kwargs["tokenizer"])

473 model_group.add_argument("--tokenizer-mode",

474 **model_kwargs["tokenizer_mode"])

@@ -870,6 +877,8 @@ class EngineArgs:

877 return ModelConfig(

878 model=self.model,

879 hf_config_path=self.hf_config_path,

880 + runner=self.runner,

881 + convert=self.convert,

882 task=self.task,

883 tokenizer=self.tokenizer,

884 tokenizer_mode=self.tokenizer_mode,

vllm/entrypoints/llm.py

+37 -56

@@ -20,8 +20,8 @@ from vllm.beam_search import (BeamSearchInstance, BeamSearchOutput,

20 create_sort_beams_key_function)

21 from vllm.config import (CompilationConfig, ModelDType, TokenizerMode,

22 is_init_field)

23 -from vllm.engine.arg_utils import (EngineArgs, HfOverrides, PoolerConfig,

24 - TaskOption)

23 +from vllm.engine.arg_utils import (ConvertOption, EngineArgs, HfOverrides,

24 + PoolerConfig, RunnerOption)

25 from vllm.engine.llm_engine import LLMEngine

26 from vllm.entrypoints.chat_utils import (ChatCompletionMessageParam,

27 ChatTemplateContentFormatOption,

@@ -170,7 +170,8 @@ class LLM:

170 self,

171 model: str,

172 *,

173 - task: TaskOption = "auto",

173 + runner: RunnerOption = "auto",

174 + convert: ConvertOption = "auto",

175 tokenizer: Optional[str] = None,

176 tokenizer_mode: TokenizerMode = "auto",

177 skip_tokenizer_init: bool = False,

@@ -244,7 +245,8 @@ class LLM:

245

246 engine_args = EngineArgs(

247 model=model,

247 - task=task,

248 + runner=runner,

249 + convert=convert,

250 tokenizer=tokenizer,

251 tokenizer_mode=tokenizer_mode,

252 skip_tokenizer_init=skip_tokenizer_init,

@@ -459,18 +461,10 @@ class LLM:

461 model_config = self.llm_engine.model_config

462 runner_type = model_config.runner_type

463 if runner_type != "generate":

462 - messages = [

463 - "LLM.generate() is only supported for generative models."

464 - ]

465 -

466 - if "generate" in model_config.supported_runner_types:

467 - messages.append(

468 - "Your model supports the 'generate' runner, but is "

469 - f"currently initialized for the '{runner_type}' runner. "

470 - "Please initialize vLLM using `--task generate` or "

471 - "`--task transcription`.")

472 -

473 - raise ValueError(" ".join(messages))

464 + raise ValueError(

465 + "LLM.generate() is only supported for generative models. "

466 + "Try passing `--runner generate` to use the model as a "

467 + "generative model.")

468

469 if prompt_token_ids is not None:

470 parsed_prompts = self._convert_v1_inputs(

@@ -497,7 +491,8 @@ class LLM:

491 truncate_prompt_tokens = None

492 if isinstance(sampling_params, SamplingParams):

493 truncate_prompt_tokens = sampling_params.truncate_prompt_tokens

500 - _validate_truncation_size(self.llm_engine.model_config.max_model_len,

494 +

495 + _validate_truncation_size(model_config.max_model_len,

496 truncate_prompt_tokens, tokenization_kwargs)

497

498 # Add any modality specific loras to the corresponding prompts

@@ -1100,16 +1095,10 @@ class LLM:

1095 model_config = self.llm_engine.model_config

1096 runner_type = model_config.runner_type

1097 if runner_type != "pooling":

1103 - messages = ["LLM.encode() is only supported for pooling models."]

1104 -

1105 - if "pooling" in model_config.supported_runner_types:

1106 - messages.append(

1107 - "Your model supports the 'pooling' runner, but is "

1108 - f"currently initialized for the '{runner_type}' runner. "

1109 - "Please initialize vLLM using `--task embed`, "

1110 - "`--task classify`, `--task score` etc.")

1111 -

1112 - raise ValueError(" ".join(messages))

1098 + raise ValueError(

1099 + "LLM.encode() is only supported for pooling models. "

1100 + "Try passing `--runner pooling` to use the model as a "

1101 + "pooling model.")

1102

1103 if prompt_token_ids is not None:

1104 parsed_prompts = self._convert_v1_inputs(

@@ -1183,8 +1172,9 @@ class LLM:

1172 embedding vectors in the same order as the input prompts.

1173 """

1174 if "embed" not in self.supported_tasks:

1186 - raise ValueError("Embedding API is not supported by this model. "

1187 - "Please set `--task embed`.")

1175 + raise ValueError(

1176 + "Embedding API is not supported by this model. "

1177 + "Try converting the model using `--convert embed`.")

1178

1179 items = self.encode(

1180 prompts,

@@ -1229,7 +1219,7 @@ class LLM:

1219 if "classify" not in self.supported_tasks:

1220 raise ValueError(

1221 "Classification API is not supported by this model. "

1232 - "Please set `--task classify`.")

1222 + "Try converting the model using `--convert classify`.")

1223

1224 items = self.encode(

1225 prompts,

@@ -1283,27 +1273,26 @@ class LLM:

1273 use_tqdm: Union[bool, Callable[..., tqdm]] = True,

1274 lora_request: Optional[Union[list[LoRARequest], LoRARequest]] = None,

1275 ) -> list[ScoringRequestOutput]:

1276 + model_config = self.llm_engine.model_config

1277

1278 if isinstance(tokenizer, MistralTokenizer):

1279 raise ValueError(

1289 - "Score API is only enabled for `--task embed or score`")

1280 + "Score API is not supported for Mistral tokenizer")

1281

1282 if len(data_1) == 1:

1283 data_1 = data_1 * len(data_2)

1284

1285 pooling_params = PoolingParams(task="score")

1286 tokenization_kwargs: dict[str, Any] = {}

1296 - _validate_truncation_size(self.llm_engine.model_config.max_model_len,

1287 +

1288 + _validate_truncation_size(model_config.max_model_len,

1289 truncate_prompt_tokens, tokenization_kwargs)

1290

1291 parsed_prompts = []

1292

1293 input_pairs = [(t1, t2) for t1, t2 in zip(data_1, data_2)]

1294

1303 - if self.llm_engine.model_config.is_multimodal_model:

1304 -

1305 - model_config = self.llm_engine.model_config

1306 -

1295 + if model_config.is_multimodal_model:

1296 for q, d in input_pairs:

1297 _, engine_prompt = get_score_prompt(

1298 model_config=model_config,

@@ -1314,11 +1303,9 @@ class LLM:

1303 )

1304

1305 parsed_prompts.append(engine_prompt)

1317 -

1306 else:

1319 -

1307 for q, t in input_pairs:

1321 - if self.llm_engine.model_config.use_pad_token:

1308 + if model_config.use_pad_token:

1309 # cross_encoder models defaults to using pad_token.

1310 prompt_inputs = tokenizer(

1311 text=q, # type: ignore[arg-type]

@@ -1396,23 +1383,18 @@ class LLM:

1383 model_config = self.llm_engine.model_config

1384 runner_type = model_config.runner_type

1385 if runner_type != "pooling":

1399 - messages = ["LLM.score() is only supported for pooling models."]

1400 -

1401 - if "pooling" in model_config.supported_runner_types:

1402 - messages.append(

1403 - "Your model supports the 'pooling' runner, but is "

1404 - f"currently initialized for the '{runner_type}' runner. "

1405 - "Please initialize vLLM using `--task embed`, "

1406 - "`--task classify`, `--task score` etc.")

1407 -

1408 - raise ValueError(" ".join(messages))

1386 + raise ValueError(

1387 + "LLM.score() is only supported for pooling models. "

1388 + "Try passing `--runner pooling` to use the model as a "

1389 + "pooling model.")

1390

1391 supported_tasks = self.supported_tasks

1392 if all(t not in supported_tasks for t in ("embed", "classify")):

1393 raise ValueError("Score API is not supported by this model. "

1413 - "Please set `--task embed` or `--task classify`.")

1394 + "Try converting the model using "

1395 + "`--convert embed` or `--convert classify`.")

1396

1415 - if (model_config.task == "classify"

1397 + if (model_config.is_cross_encoder

1398 and getattr(model_config.hf_config, "num_labels", 0) != 1):

1399 raise ValueError("Score API is only enabled for num_labels == 1.")

1400

@@ -1421,15 +1403,14 @@ class LLM:

1403 # lists of tokens to the `text` and `text_pair` kwargs

1404 tokenizer = self.get_tokenizer()

1405

1424 - if not self.llm_engine.model_config.is_multimodal_model:

1406 + if not model_config.is_multimodal_model:

1407

1408 def check_data_type(data: Union[SingletonPrompt,

1409 Sequence[SingletonPrompt],

1410 ScoreMultiModalParam]):

1411 if isinstance(data, dict) and "content" in data:

1430 - raise ValueError(

1431 - f"ScoreMultiModalParam is not supported for {self.llm_engine.model_config.architecture}", # noqa: E501

1432 - )

1412 + raise ValueError("ScoreMultiModalParam is not supported "

1413 + f"for {model_config.architecture}")

1414

1415 check_data_type(data_1)

1416 check_data_type(data_2)

@@ -1471,7 +1452,7 @@ class LLM:

1452

1453 _validate_score_input_lens(data_1, data_2) # type: ignore[arg-type]

1454

1474 - if self.llm_engine.model_config.is_cross_encoder:

1455 + if model_config.is_cross_encoder:

1456 return self._cross_encoding_score(

1457 tokenizer,

1458 data_1, # type: ignore[arg-type]

vllm/entrypoints/openai/api_server.py

-1

@@ -1734,7 +1734,6 @@ async def init_app_state(

1734 state.openai_serving_models,

1735 request_logger=request_logger,

1736 ) if "transcription" in supported_tasks else None

1737 - state.task = model_config.task

1737

1738 state.enable_server_load_tracking = args.enable_server_load_tracking

1739 state.server_load_metrics = 0

vllm/model_executor/model_loader/utils.py

+27 -104

@@ -9,9 +9,8 @@ from dataclasses import dataclass, field

9 from typing import Optional

11 import torch

12 -import transformers

12 from torch import nn

14 -from transformers.dynamic_module_utils import get_class_from_dynamic_module

13 +from typing_extensions import assert_never

15 from vllm.attention import Attention

16 from vllm.config import (ModelConfig, ModelImpl, VllmConfig,

@@ -20,13 +19,10 @@ from vllm.logger import init_logger

19 from vllm.model_executor.layers.linear import QKVCrossParallelLinear

20 from vllm.model_executor.layers.quantization.base_config import (

21 QuantizationConfig, QuantizeMethodBase)

23 -from vllm.model_executor.models import ModelRegistry

22 from vllm.model_executor.models.adapters import (as_embedding_model,

23 as_reward_model,

24 as_seq_cls_model)

25 from vllm.model_executor.models.interfaces import SupportsQuant

28 -from vllm.model_executor.models.registry import (_PREVIOUSLY_SUPPORTED_MODELS,

29 - _TRANSFORMERS_BACKEND_MODELS)

26 from vllm.utils import is_pin_memory_available

28 logger = init_logger(__name__)

@@ -169,61 +165,6 @@ def device_loading_context(module: torch.nn.Module,

165 # New parameters or parameters already on target device are untouched

166

167

172 -def resolve_transformers_arch(model_config: ModelConfig,

173 - architectures: list[str]):

174 - if model_config.model_impl == ModelImpl.VLLM:

175 - raise ValueError(

176 - "Attempting to resolve architecture from the Transformers library "

177 - "but the model implementation is set to vLLM. This should never "

178 - "happen.")

179 -

180 - for i, arch in enumerate(architectures):

181 - if arch in _TRANSFORMERS_BACKEND_MODELS:

182 - continue

183 -

184 - if model_config.model_impl == ModelImpl.AUTO:

185 - logger.warning(

186 - "%s has no vLLM implementation, falling back to Transformers "

187 - "implementation. Some features may not be supported and "

188 - "performance may not be optimal.", arch)

189 -

190 - auto_map: dict[str, str] = getattr(model_config.hf_config, "auto_map",

191 - None) or dict()

192 - # Make sure that config class is always initialized before model class,

193 - # otherwise the model class won't be able to access the config class,

194 - # the expected auto_map should have correct order like:

195 - # "auto_map": {

196 - # "AutoConfig": "<your-repo-name>--<config-name>",

197 - # "AutoModel": "<your-repo-name>--<config-name>",

198 - # "AutoModelFor<Task>": "<your-repo-name>--<config-name>",

199 - # },

200 - auto_modules = {

201 - name:

202 - get_class_from_dynamic_module(module,

203 - model_config.model,

204 - revision=model_config.revision)

205 - for name, module in sorted(auto_map.items(), key=lambda x: x[0])

206 - }

207 - model_module = getattr(transformers, arch, None)

208 - if model_module is None:

209 - if "AutoModel" not in auto_map:

210 - raise ValueError(

211 - f"Cannot find model module. '{arch}' is not a registered "

212 - "model in the Transformers library (only relevant if the "

213 - "model is meant to be in Transformers) and 'AutoModel' is "

214 - "not present in the model config's 'auto_map' (relevant "

215 - "if the model is custom).")

216 - model_module = auto_modules["AutoModel"]

217 -

218 - if not model_module.is_backend_compatible():

219 - raise ValueError(

220 - f"The Transformers implementation of '{arch}' is not "

221 - "compatible with vLLM.")

222 -

223 - architectures[i] = model_config._get_transformers_backend_cls()

224 - return architectures

225 -

226 -

168 def get_model_architecture(

169 model_config: ModelConfig) -> tuple[type[nn.Module], str]:

170 architectures = getattr(model_config.hf_config, "architectures", [])

@@ -239,56 +180,38 @@ def get_model_architecture(

180 "bitsandbytes",

181 ]

182

242 - vllm_supported_archs = ModelRegistry.get_supported_archs()

243 - is_supported = lambda arch: (arch in vllm_supported_archs and arch not in

244 - _TRANSFORMERS_BACKEND_MODELS)

245 - vllm_not_supported = not any(is_supported(arch) for arch in architectures)

246 -

247 - if vllm_not_supported:

248 - # try automatic conversion in adapters.py

249 - for arch in architectures:

250 - if not arch.endswith("ForSequenceClassification"):

251 - continue

183 + if (model_config.quantization is not None

184 + and model_config.quantization not in mixtral_supported

185 + and "MixtralForCausalLM" in architectures):

186 + architectures = ["QuantMixtralForCausalLM"]

187

253 - assert model_config.task == "classify"

254 - causal_lm_arch = arch.replace("ForSequenceClassification",

255 - "ForCausalLM")

256 - causal_lm_arch_vllm_supported = (causal_lm_arch

257 - in vllm_supported_archs)

258 - if not causal_lm_arch_vllm_supported:

259 - continue

188 + model_cls, arch = model_config.registry.resolve_model_cls(

189 + architectures,

190 + model_config=model_config,

191 + )

192

261 - architectures = [causal_lm_arch]

262 - vllm_not_supported = False

263 - break

264 -

265 - if any(arch in _PREVIOUSLY_SUPPORTED_MODELS for arch in architectures):

266 - previous_version = _PREVIOUSLY_SUPPORTED_MODELS[architectures[0]]

267 - raise ValueError(

268 - f"Model architecture {architectures[0]} was supported"

269 - f" in vLLM until version {previous_version}, and is "

270 - "not supported anymore. Please use an older version"

271 - " of vLLM if you want to use this model architecture.")

272 -

273 - if (model_config.model_impl == ModelImpl.TRANSFORMERS or

274 - model_config.model_impl == ModelImpl.AUTO and vllm_not_supported):

275 - architectures = resolve_transformers_arch(model_config, architectures)

276 - logger.debug_once("Resolve transformers arch %s", str(architectures))

277 - elif (model_config.quantization is not None

278 - and model_config.quantization not in mixtral_supported

279 - and "MixtralForCausalLM" in architectures):

280 - architectures = ["QuantMixtralForCausalLM"]

193 + if arch == model_config._get_transformers_backend_cls():

194 + assert model_config.model_impl != ModelImpl.VLLM

195 + if model_config.model_impl == ModelImpl.AUTO:

196 + logger.warning_once(

197 + "%s has no vLLM implementation, falling back to Transformers "

198 + "implementation. Some features may not be supported and "

199 + "performance may not be optimal.", arch)

200

282 - model_cls, arch = ModelRegistry.resolve_model_cls(architectures)

283 - if model_config.task == "embed":

284 - logger.debug_once("Automatic conversion using `as_embedding_model`.")

201 + convert_type = model_config.convert_type

202 + if convert_type == "none":

203 + pass

204 + elif convert_type == "embed":

205 + logger.debug_once("Converting to embedding model.")

206 model_cls = as_embedding_model(model_cls)

286 - elif model_config.task == "classify":

287 - logger.debug_once("Automatic conversion using `as_seq_cls_model`.")

207 + elif convert_type == "classify":

208 + logger.debug_once("Converting to sequence classification model.")

209 model_cls = as_seq_cls_model(model_cls)

289 - elif model_config.task == "reward":

290 - logger.debug_once("Automatic conversion using `as_reward_model`.")

210 + elif convert_type == "reward":

211 + logger.debug_once("Converting to reward model.")

212 model_cls = as_reward_model(model_cls)

213 + else:

214 + assert_never(convert_type)

215

216 return model_cls, arch

217

vllm/model_executor/models/config.py

+4 -2

@@ -253,8 +253,10 @@ class HybridAttentionMambaModelConfig(VerifyAndUpdateConfig):

253 dtype=kv_cache_dtype,

254 use_mla=model_config.use_mla).page_size_bytes

255

256 - model_cls = ModelRegistry.resolve_model_cls(

257 - model_config._model_info.architecture)[0]

256 + model_cls, _ = ModelRegistry.resolve_model_cls(

257 + model_config.architecture,

258 + model_config=model_config,

259 + )

260

261 # get mamba page size

262 mamba_page_size = MambaSpec(

vllm/model_executor/models/registry.py

+187 -62

@@ -12,19 +12,24 @@ import sys

12 import tempfile

13 from abc import ABC, abstractmethod

14 from collections.abc import Set

15 -from dataclasses import asdict, dataclass, field

15 +from dataclasses import dataclass, field

16 from functools import lru_cache

17 from typing import Callable, Optional, TypeVar, Union

19 import torch.nn as nn

20 +import transformers

22 +from vllm.config import (ModelConfig, ModelImpl, iter_architecture_defaults,

23 + try_match_architecture_defaults)

24 from vllm.logger import init_logger

25 +from vllm.transformers_utils.dynamic_module import (

26 + try_get_class_from_dynamic_module)

28 from .interfaces import (has_inner_state, has_noops, is_attention_free,

29 is_hybrid, supports_cross_encoding,

30 supports_multimodal, supports_multimodal_raw_input,

31 supports_pp, supports_transcription, supports_v0_only)

27 -from .interfaces_base import is_text_generation_model

32 +from .interfaces_base import is_pooling_model, is_text_generation_model

34 logger = init_logger(__name__)

@@ -311,7 +316,7 @@ class _ModelInfo:

316 return _ModelInfo(

317 architecture=model.__name__,

318 is_text_generation_model=is_text_generation_model(model),

314 - is_pooling_model=True, # Can convert any model into a pooling model

319 + is_pooling_model=is_pooling_model(model),

320 supports_cross_encoding=supports_cross_encoding(model),

321 supports_multimodal=supports_multimodal(model),

322 supports_multimodal_raw_input=supports_multimodal_raw_input(model),

@@ -465,6 +470,16 @@ class _ModelRegistry:

470 f"Model architectures {architectures} failed "

471 "to be inspected. Please check the logs for more details.")

472

473 + for arch in architectures:

474 + if arch in _PREVIOUSLY_SUPPORTED_MODELS:

475 + previous_version = _PREVIOUSLY_SUPPORTED_MODELS[arch]

476 +

477 + raise ValueError(

478 + f"Model architecture {arch} was supported in vLLM until "

479 + f"v{previous_version}, and is not supported anymore. "

480 + "Please use an older version of vLLM if you want to "

481 + "use this model architecture.")

482 +

483 raise ValueError(

484 f"Model architectures {architectures} are not supported for now. "

485 f"Supported architectures: {all_supported_archs}")

@@ -477,174 +492,284 @@ class _ModelRegistry:

492 return _try_load_model_cls(model_arch, self.models[model_arch])

493

494 def _try_inspect_model_cls(self, model_arch: str) -> Optional[_ModelInfo]:

480 - if model_arch in self.models:

481 - return _try_inspect_model_cls(model_arch, self.models[model_arch])

495 + if model_arch not in self.models:

496 + return None

497

483 - if model_arch.endswith("ForSequenceClassification"):

484 - causal_lm_arch = model_arch.replace("ForSequenceClassification",

485 - "ForCausalLM")

486 - if causal_lm_arch not in self.models:

498 + return _try_inspect_model_cls(model_arch, self.models[model_arch])

499 +

500 + def _try_resolve_transformers(

501 + self,

502 + architecture: str,

503 + model_config: ModelConfig,

504 + ) -> Optional[str]:

505 + if architecture in _TRANSFORMERS_BACKEND_MODELS:

506 + return architecture

507 +

508 + auto_map: dict[str, str] = getattr(model_config.hf_config, "auto_map",

509 + None) or dict()

510 +

511 + # Make sure that config class is always initialized before model class,

512 + # otherwise the model class won't be able to access the config class,

513 + # the expected auto_map should have correct order like:

514 + # "auto_map": {

515 + # "AutoConfig": "<your-repo-name>--<config-name>",

516 + # "AutoModel": "<your-repo-name>--<config-name>",

517 + # "AutoModelFor<Task>": "<your-repo-name>--<config-name>",

518 + # },

519 + for prefix in ("AutoConfig", "AutoModel"):

520 + for name, module in auto_map.items():

521 + if name.startswith(prefix):

522 + try_get_class_from_dynamic_module(

523 + module,

524 + model_config.model,

525 + revision=model_config.revision,

526 + warn_on_fail=False,

527 + )

528 +

529 + model_module = getattr(transformers, architecture, None)

530 +

531 + if model_module is None:

532 + for name, module in auto_map.items():

533 + if name.startswith("AutoModel"):

534 + model_module = try_get_class_from_dynamic_module(

535 + module,

536 + model_config.model,

537 + revision=model_config.revision,

538 + warn_on_fail=True,

539 + )

540 + if model_module is not None:

541 + break

542 + else:

543 + if model_config.model_impl != ModelImpl.TRANSFORMERS:

544 + return None

545 +

546 + raise ValueError(

547 + f"Cannot find model module. {architecture!r} is not a "

548 + "registered model in the Transformers library (only "

549 + "relevant if the model is meant to be in Transformers) "

550 + "and 'AutoModel' is not present in the model config's "

551 + "'auto_map' (relevant if the model is custom).")

552 +

553 + if not model_module.is_backend_compatible():

554 + if model_config.model_impl != ModelImpl.TRANSFORMERS:

555 return None

556

489 - info = _try_inspect_model_cls(causal_lm_arch,

490 - self.models[causal_lm_arch])

557 + raise ValueError(

558 + f"The Transformers implementation of {architecture!r} "

559 + "is not compatible with vLLM.")

560

492 - info = _ModelInfo(**dict(

493 - asdict(info), **{

494 - "architecture": model_arch,

495 - "supports_cross_encoding": True

496 - }))

497 - return info

561 + return model_config._get_transformers_backend_cls()

562

499 - return None

563 + def _normalize_arch(

564 + self,

565 + architecture: str,

566 + model_config: ModelConfig,

567 + ) -> str:

568 + if architecture in self.models:

569 + return architecture

570 +

571 + # This may be called in order to resolve runner_type and convert_type

572 + # in the first place, in which case we consider the default match

573 + match = try_match_architecture_defaults(

574 + architecture,

575 + runner_type=getattr(model_config, "runner_type", None),

576 + convert_type=getattr(model_config, "convert_type", None),

577 + )

578 + if match:

579 + suffix, _ = match

580 +

581 + # Get the name of the base model to convert

582 + for repl_suffix, _ in iter_architecture_defaults():

583 + base_arch = architecture.replace(suffix, repl_suffix)

584 + if base_arch in self.models:

585 + return base_arch

586 +

587 + return architecture

588

589 def _normalize_archs(

590 self,

503 - architectures: Union[str, list[str]],

591 + architectures: list[str],

592 + model_config: ModelConfig,

593 ) -> list[str]:

505 - if isinstance(architectures, str):

506 - architectures = [architectures]

594 if not architectures:

595 logger.warning("No model architectures are specified")

596

510 - # filter out support architectures

511 - normalized_arch = list(

512 - filter(lambda model: model in self.models, architectures))

513 -

514 - # try automatic conversion in adapters.py

515 - for arch in architectures:

516 - if not arch.endswith("ForSequenceClassification"):

517 - continue

518 - causal_lm_arch = arch.replace("ForSequenceClassification",

519 - "ForCausalLM")

520 - if causal_lm_arch in self.models:

521 - normalized_arch.append(arch)

522 -

523 - # NOTE(Isotr0py): Be careful of architectures' order!

524 - # Make sure Transformers backend architecture is at the end of the

525 - # list, otherwise pooling models automatic conversion will fail!

526 - for arch in normalized_arch:

527 - if arch.startswith("TransformersFor"):

528 - normalized_arch.remove(arch)

529 - normalized_arch.append(arch)

530 -

531 - return normalized_arch

597 + return [

598 + self._normalize_arch(arch, model_config) for arch in architectures

599 + ]

600

601 def inspect_model_cls(

602 self,

603 architectures: Union[str, list[str]],

604 + model_config: ModelConfig,

605 ) -> tuple[_ModelInfo, str]:

537 - architectures = self._normalize_archs(architectures)

606 + if isinstance(architectures, str):

607 + architectures = [architectures]

608

539 - for arch in architectures:

540 - model_info = self._try_inspect_model_cls(arch)

609 + normalized_archs = self._normalize_archs(architectures, model_config)

610 +

611 + # Require transformers impl

612 + if model_config.model_impl == ModelImpl.TRANSFORMERS:

613 + arch = self._try_resolve_transformers(architectures[0],

614 + model_config)

615 + if arch is not None:

616 + model_info = self._try_inspect_model_cls(arch)

617 + if model_info is not None:

618 + return (model_info, arch)

619 +

620 + for arch, normalized_arch in zip(architectures, normalized_archs):

621 + model_info = self._try_inspect_model_cls(normalized_arch)

622 if model_info is not None:

623 return (model_info, arch)

624

625 + # Fallback to transformers impl

626 + if model_config.model_impl in (ModelImpl.AUTO, ModelImpl.TRANSFORMERS):

627 + arch = self._try_resolve_transformers(architectures[0],

628 + model_config)

629 + if arch is not None:

630 + model_info = self._try_inspect_model_cls(arch)

631 + if model_info is not None:

632 + return (model_info, arch)

633 +

634 return self._raise_for_unsupported(architectures)

635

636 def resolve_model_cls(

637 self,

638 architectures: Union[str, list[str]],

639 + model_config: ModelConfig,

640 ) -> tuple[type[nn.Module], str]:

550 - architectures = self._normalize_archs(architectures)

641 + if isinstance(architectures, str):

642 + architectures = [architectures]

643

552 - for arch in architectures:

553 - model_cls = self._try_load_model_cls(arch)

644 + normalized_archs = self._normalize_archs(architectures, model_config)

645 +

646 + # Require transformers impl

647 + if model_config.model_impl == ModelImpl.TRANSFORMERS:

648 + arch = self._try_resolve_transformers(architectures[0],

649 + model_config)

650 + if arch is not None:

651 + model_cls = self._try_load_model_cls(arch)

652 + if model_cls is not None:

653 + return (model_cls, arch)

654 +

655 + for arch, normalized_arch in zip(architectures, normalized_archs):

656 + model_cls = self._try_load_model_cls(normalized_arch)

657 if model_cls is not None:

658 return (model_cls, arch)

659

660 + # Fallback to transformers impl

661 + if model_config.model_impl in (ModelImpl.AUTO, ModelImpl.TRANSFORMERS):

662 + arch = self._try_resolve_transformers(architectures[0],

663 + model_config)

664 + if arch is not None:

665 + model_cls = self._try_load_model_cls(arch)

666 + if model_cls is not None:

667 + return (model_cls, arch)

668 +

669 return self._raise_for_unsupported(architectures)

670

671 def is_text_generation_model(

672 self,

673 architectures: Union[str, list[str]],

674 + model_config: ModelConfig,

675 ) -> bool:

563 - model_cls, _ = self.inspect_model_cls(architectures)

676 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

677 return model_cls.is_text_generation_model

678

679 def is_pooling_model(

680 self,

681 architectures: Union[str, list[str]],

682 + model_config: ModelConfig,

683 ) -> bool:

570 - model_cls, _ = self.inspect_model_cls(architectures)

684 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

685 return model_cls.is_pooling_model

686

687 def is_cross_encoder_model(

688 self,

689 architectures: Union[str, list[str]],

690 + model_config: ModelConfig,

691 ) -> bool:

577 - model_cls, _ = self.inspect_model_cls(architectures)

692 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

693 return model_cls.supports_cross_encoding

694

695 def is_multimodal_model(

696 self,

697 architectures: Union[str, list[str]],

698 + model_config: ModelConfig,

699 ) -> bool:

584 - model_cls, _ = self.inspect_model_cls(architectures)

700 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

701 return model_cls.supports_multimodal

702

703 def supports_multimodal_raw_input(

704 self,

705 architectures: Union[str, list[str]],

706 + model_config: ModelConfig,

707 ) -> bool:

591 - model_cls, _ = self.inspect_model_cls(architectures)

708 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

709 return model_cls.supports_multimodal_raw_input

710

711 def is_pp_supported_model(

712 self,

713 architectures: Union[str, list[str]],

714 + model_config: ModelConfig,

715 ) -> bool:

598 - model_cls, _ = self.inspect_model_cls(architectures)

716 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

717 return model_cls.supports_pp

718

719 def model_has_inner_state(

720 self,

721 architectures: Union[str, list[str]],

722 + model_config: ModelConfig,

723 ) -> bool:

605 - model_cls, _ = self.inspect_model_cls(architectures)

724 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

725 return model_cls.has_inner_state

726

727 def is_attention_free_model(

728 self,

729 architectures: Union[str, list[str]],

730 + model_config: ModelConfig,

731 ) -> bool:

612 - model_cls, _ = self.inspect_model_cls(architectures)

732 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

733 return model_cls.is_attention_free

734

735 def is_hybrid_model(

736 self,

737 architectures: Union[str, list[str]],

738 + model_config: ModelConfig,

739 ) -> bool:

619 - model_cls, _ = self.inspect_model_cls(architectures)

740 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

741 return model_cls.is_hybrid

742

743 def is_noops_model(

744 self,

745 architectures: Union[str, list[str]],

746 + model_config: ModelConfig,

747 ) -> bool:

626 - model_cls, _ = self.inspect_model_cls(architectures)

748 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

749 return model_cls.has_noops

750

751 def is_transcription_model(

752 self,

753 architectures: Union[str, list[str]],

754 + model_config: ModelConfig,

755 ) -> bool:

633 - model_cls, _ = self.inspect_model_cls(architectures)

756 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

757 return model_cls.supports_transcription

758

759 def is_transcription_only_model(

760 self,

761 architectures: Union[str, list[str]],

762 + model_config: ModelConfig,

763 ) -> bool:

640 - model_cls, _ = self.inspect_model_cls(architectures)

764 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

765 return model_cls.supports_transcription_only

766

767 def is_v1_compatible(

768 self,

769 architectures: Union[str, list[str]],

770 + model_config: ModelConfig,

771 ) -> bool:

647 - model_cls, _ = self.inspect_model_cls(architectures)

772 + model_cls, _ = self.inspect_model_cls(architectures, model_config)

773 return not model_cls.supports_v0_only

774

775

vllm/transformers_utils/dynamic_module.py new

+60

@@ -0,0 +1,60 @@

1 +# SPDX-License-Identifier: Apache-2.0

2 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

3 +import os

4 +from typing import Optional, Union

5 +

6 +from transformers.dynamic_module_utils import get_class_from_dynamic_module

7 +

8 +import vllm.envs as envs

9 +from vllm.logger import init_logger

10 +

11 +logger = init_logger(__name__)

12 +

13 +

14 +def try_get_class_from_dynamic_module(

15 + class_reference: str,

16 + pretrained_model_name_or_path: str,

17 + cache_dir: Optional[Union[str, os.PathLike]] = None,

18 + force_download: bool = False,

19 + resume_download: Optional[bool] = None,

20 + proxies: Optional[dict[str, str]] = None,

21 + token: Optional[Union[bool, str]] = None,

22 + revision: Optional[str] = None,

23 + local_files_only: bool = False,

24 + repo_type: Optional[str] = None,

25 + code_revision: Optional[str] = None,

26 + warn_on_fail: bool = True,

27 + **kwargs,

28 +) -> Optional[type]:

29 + """

30 + As [transformers.dynamic_module_utils.get_class_from_dynamic_module][],

31 + but ignoring any errors.

32 + """

33 + try:

34 + return get_class_from_dynamic_module(

35 + class_reference,

36 + pretrained_model_name_or_path,

37 + cache_dir=cache_dir,

38 + force_download=force_download,

39 + resume_download=resume_download,

40 + proxies=proxies,

41 + token=token,

42 + revision=revision,

43 + local_files_only=local_files_only,

44 + repo_type=repo_type,

45 + code_revision=code_revision,

46 + **kwargs,

47 + )

48 + except Exception:

49 + location = "ModelScope" if envs.VLLM_USE_MODELSCOPE else "HF Hub"

50 +

51 + if warn_on_fail:

52 + logger.warning(

53 + "Unable to load %s from %s on %s.",

54 + class_reference,

55 + pretrained_model_name_or_path,

56 + location,

57 + exc_info=True,

58 + )

59 +

60 + return None

vllm/transformers_utils/tokenizer_group.py

+11 -1

@@ -3,6 +3,8 @@

4 from typing import Optional

6 +from typing_extensions import assert_never

7 +

8 from vllm.config import LoRAConfig, ModelConfig, SchedulerConfig

9 from vllm.lora.request import LoRARequest

10 from vllm.transformers_utils.tokenizer import (AnyTokenizer, encode_tokens,

@@ -108,6 +110,14 @@ class TokenizerGroup:

110 def init_tokenizer_from_configs(model_config: ModelConfig,

111 scheduler_config: SchedulerConfig,

112 lora_config: Optional[LoRAConfig]):

113 + runner_type = model_config.runner_type

114 + if runner_type == "generate" or runner_type == "draft":

115 + truncation_side = "left"

116 + elif runner_type == "pooling":

117 + truncation_side = "right"

118 + else:

119 + assert_never(runner_type)

120 +

121 return TokenizerGroup(

122 tokenizer_id=model_config.tokenizer,

123 enable_lora=bool(lora_config),

@@ -117,4 +127,4 @@ def init_tokenizer_from_configs(model_config: ModelConfig,

127 tokenizer_mode=model_config.tokenizer_mode,

128 trust_remote_code=model_config.trust_remote_code,

129 revision=model_config.tokenizer_revision,

120 - truncation_side=model_config.truncation_side)

130 + truncation_side=truncation_side)

vllm/v1/worker/gpu_model_runner.py

+3 -3

@@ -127,8 +127,8 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin):

127 self.is_multimodal_model = model_config.is_multimodal_model

128 self.is_pooling_model = model_config.pooler_config is not None

129 self.is_encoder_only_model = False

130 - self.model_supports_multimodal_raw_input = (

131 - model_config.model_supports_multimodal_raw_input)

130 + self.is_multimodal_raw_input_supported = (

131 + model_config.is_multimodal_raw_input_supported)

132 self.max_model_len = model_config.max_model_len

133 self.max_num_tokens = scheduler_config.max_num_batched_tokens

134 self.max_num_reqs = scheduler_config.max_num_seqs

@@ -583,7 +583,7 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin):

583 ) -> dict[str, Any]:

584

585 model_kwargs: dict[str, Any] = {}

586 - if self.model_supports_multimodal_raw_input:

586 + if self.is_multimodal_raw_input_supported:

587 # This model requires the raw multimodal data in input.

588 if scheduler_output:

589 multi_modal_kwargs_list = []