A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Doc] Show default pooling method in a table (#11904)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Jan 10, 2025 at 11:25 UTC
3de2b1eafb12e420c563cb7153d4d2f0e8451ca9
2 files changed
+45
-22
docs/source/models/generative_models.md
+4
-4
@@ -8,14 +8,14 @@ In vLLM, generative models implement the {class}`~vllm.model_executor.models.Vll
8
Based on the final hidden states of the input, these models output log probabilities of the tokens to generate,
9
which are then passed through {class}`~vllm.model_executor.layers.Sampler` to obtain the final text.
10
11
+For generative models, the only supported `--task` option is `"generate"`.
12
+Usually, this is automatically inferred so you don't have to specify it.
13
+
14
## Offline Inference
15
16
The {class}`~vllm.LLM` class provides various methods for offline inference.
17
See [Engine Arguments](#engine-args) for a list of options when initializing the model.
18
16
-For generative models, the only supported {code}`task` option is {code}`"generate"`.
17
-Usually, this is automatically inferred so you don't have to specify it.
18
-
19
### `LLM.generate`
20
21
The {class}`~vllm.LLM.generate` method is available to all generative models in vLLM.
@@ -33,7 +33,7 @@ for output in outputs:
33
```
34
35
You can optionally control the language generation by passing {class}`~vllm.SamplingParams`.
36
-For example, you can use greedy sampling by setting {code}`temperature=0`:
36
+For example, you can use greedy sampling by setting `temperature=0`:
37
38
```python
39
llm = LLM(model="facebook/opt-125m")
docs/source/models/pooling_models.md
+41
-18
@@ -14,30 +14,53 @@ As shown in the [Compatibility Matrix](#compatibility-matrix), most vLLM feature
14
pooling models as they only work on the generation or decode stage, so performance may not improve as much.
15
```
16
17
-## Offline Inference
18
-
19
-The {class}`~vllm.LLM` class provides various methods for offline inference.
20
-See [Engine Arguments](#engine-args) for a list of options when initializing the model.
21
-
22
-For pooling models, we support the following {code}`task` options:
23
-
24
-- Embedding ({code}`"embed"` / {code}`"embedding"`)
25
-- Classification ({code}`"classify"`)
26
-- Sentence Pair Scoring ({code}`"score"`)
27
-- Reward Modeling ({code}`"reward"`)
17
+For pooling models, we support the following `--task` options.
18
+The selected option sets the default pooler used to extract the final hidden states:
19
+
20
+```{list-table}
21
+:widths: 50 25 25 25
22
+:header-rows: 1
23
+
24
+* - Task
25
+ - Pooling Type
26
+ - Normalization
27
+ - Softmax
28
+* - Embedding (`embed`)
29
+ - `LAST`
30
+ - ✅︎
31
+ - ✗
32
+* - Classification (`classify`)
33
+ - `LAST`
34
+ - ✗
35
+ - ✅︎
36
+* - Sentence Pair Scoring (`score`)
37
+ - \*
38
+ - \*
39
+ - \*
40
+* - Reward Modeling (`reward`)
41
+ - `ALL`
42
+ - ✗
43
+ - ✗
44
+```
45
29
-The selected task determines the default {class}`~vllm.model_executor.layers.Pooler` that is used:
46
+\*The default pooler is always defined by the model.
47
31
-- Embedding: Extract only the hidden states corresponding to the last token, and apply normalization.
32
-- Classification: Extract only the hidden states corresponding to the last token, and apply softmax.
33
-- Sentence Pair Scoring: Extract only the hidden states corresponding to the last token, and apply softmax.
34
-- Reward Modeling: Extract all of the hidden states and return them directly.
48
+```{note}
49
+If the model's implementation in vLLM defines its own pooler, the default pooler is set to that instead of the one specified in this table.
50
+```
51
52
When loading [Sentence Transformers](https://huggingface.co/sentence-transformers) models,
37
-we attempt to override the default pooler based on its Sentence Transformers configuration file ({code}`modules.json`).
53
+we attempt to override the default pooler based on its Sentence Transformers configuration file (`modules.json`).
54
39
-You can customize the model's pooling method via the {code}`override_pooler_config` option,
55
+```{tip}
56
+You can customize the model's pooling method via the `--override-pooler-config` option,
57
which takes priority over both the model's and Sentence Transformers's defaults.
58
+```
59
+
60
+## Offline Inference
61
+
62
+The {class}`~vllm.LLM` class provides various methods for offline inference.
63
+See [Engine Arguments](#engine-args) for a list of options when initializing the model.
64
65
### `LLM.encode`
66