@tduncan / Nvidia-Optimization-Stack-VLLM
Code Commits Issues Pulls Cloud Agent CI/CD
main
main
md 146 lines 6.09 KB
Rendered Raw
1 # Reward Usages
2
3 A reward model (RM) is designed to evaluate and score the quality of outputs generated by a language model, acting as a proxy for human preferences.
4
5 ## Summary
6
7 - Model Usage: reward
8 - Pooling Task:
9
10 | Model Types | Pooling Tasks |
11 |------------------------------------|----------------|
12 | (sequence) (outcome) reward models | classify |
13 | token (outcome) reward models | token_classify |
14 | process reward models | token_classify |
15
16 - Offline APIs:
17 - `LLM.encode(..., pooling_task="...")`
18 - Online APIs:
19 - Pooling API (`/pooling`)
20
21 ## Supported Models
22
23 ### Reward Models
24
25 Using sequence classification models as (sequence) (outcome) reward models, the usage and supported features are the same as for normal [classification models](classify.md).
26
27 --8<-- [start:supported-sequence-reward-models]
28
29 | Architecture | Models | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
30 | ------------ | ------ | ----------------- | -------------------- | ------------------------- |
31 | `JambaForSequenceClassification` | Jamba | `ai21labs/Jamba-tiny-reward-dev`, etc. | ✅︎ | ✅︎ |
32 | `Qwen3ForSequenceClassification`<sup>C</sup> | Qwen3-based | `Skywork/Skywork-Reward-V2-Qwen3-0.6B`, etc. | ✅︎ | ✅︎ |
33 | `LlamaForSequenceClassification`<sup>C</sup> | Llama-based | `Skywork/Skywork-Reward-V2-Llama-3.2-1B`, etc. | ✅︎ | ✅︎ |
34 | `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* |
35
36 <sup>C</sup> Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))
37
38 If your model is not in the above list, we will try to automatically convert the model using
39 [as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model]. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.
40
41 --8<-- [end:supported-sequence-reward-models]
42
43 ### Token Reward Models
44
45 The key distinction between (sequence) classification and token classification lies in their output granularity: (sequence) classification produces a single result for an entire input sequence, whereas token classification yields a result for each individual token within the sequence.
46
47 Using token classification models as token (outcome) reward models, the usage and supported features are the same as for normal [token classification models](token_classify.md).
48
49 --8<-- [start:supported-token-reward-models]
50
51 | Architecture | Models | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
52 | ------------ | ------ | ----------------- | -------------------- | ------------------------- |
53 | `InternLM2ForRewardModel` | InternLM2-based | `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc. | ✅︎ | ✅︎ |
54 | `Qwen2ForRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-RM-72B`, etc. | ✅︎ | ✅︎ |
55 | `*Model`<sup>C</sup>, `*ForCausalLM`<sup>C</sup>, etc. | Generative models | N/A | \* | \* |
56
57 <sup>C</sup> Automatically converted into a classification model via `--convert classify`. ([details](./README.md#model-conversion))
58
59 If your model is not in the above list, we will try to automatically convert the model using
60 [as_seq_cls_model][vllm.model_executor.models.adapters.as_seq_cls_model].
61
62 --8<-- [end:supported-token-reward-models]
63
64 ### Process Reward Models
65
66 The process reward models used for evaluating intermediate steps are crucial to achieving the desired outcome.
67
68 | Architecture | Models | Example HF Models | [LoRA](../../features/lora.md) | [PP](../../serving/parallelism_scaling.md) |
69 | ------------ | ------ | ----------------- | -------------------- | ------------------------- |
70 | `LlamaForCausalLM` | Llama-based | `peiyi9979/math-shepherd-mistral-7b-prm`, etc. | ✅︎ | ✅︎ |
71 | `Qwen2ForProcessRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-PRM-7B`, etc. | ✅︎ | ✅︎ |
72
73 !!! important
74 For process-supervised reward models such as `peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
75 e.g.: `--pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.
76
77 ## Offline Inference
78
79 ### Pooling Parameters
80
81 The following [pooling parameters][vllm.PoolingParams] are supported.
82
83 ```python
84 --8<-- "vllm/pooling_params.py:common-pooling-params"
85 --8<-- "vllm/pooling_params.py:classify-pooling-params"
86 ```
87
88 ### `LLM.encode`
89
90 The [encode][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.encode] method is available to all pooling models in vLLM.
91
92 - Reward Models
93
94 Set `pooling_task="classify"` when using `LLM.encode` for (sequence) (outcome) reward models:
95
96 ```python
97 from vllm import LLM
98
99 llm = LLM(model="Skywork/Skywork-Reward-V2-Qwen3-0.6B", runner="pooling")
100 (output,) = llm.encode("Hello, my name is", pooling_task="classify")
101
102 data = output.outputs.data
103 print(f"Data: {data!r}")
104 ```
105
106 - Token Reward Models
107
108 Set `pooling_task="token_classify"` when using `LLM.encode` for token (outcome) reward models:
109
110 ```python
111 from vllm import LLM
112
113 llm = LLM(model="internlm/internlm2-1_8b-reward", runner="pooling", trust_remote_code=True)
114 (output,) = llm.encode("Hello, my name is", pooling_task="token_classify")
115
116 data = output.outputs.data
117 print(f"Data: {data!r}")
118 ```
119
120 - Process Reward Models
121
122 Set `pooling_task="token_classify"` when using `LLM.encode` for token (outcome) reward models:
123
124 ```python
125 from vllm import LLM
126
127 llm = LLM(model="Qwen/Qwen2.5-Math-PRM-7B", runner="pooling")
128 (output,) = llm.encode("Hello, my name is<extra_0><extra_0><extra_0>", pooling_task="token_classify")
129
130 data = output.outputs.data
131 print(f"Data: {data!r}")
132 ```
133
134 ## Online Serving
135
136 Please refer to the [Pooling API](README.md#pooling-api). Pooling task corresponding to reward model types refer to the [table above](#summary).
137
138 ## More examples
139
140 More examples can be found here: [examples/pooling/reward](../../../examples/pooling/reward)
141
142 ## Deprecated Features
143
144 ### `LLM.reward`
145
146 `llm.reward` API is deprecated and was removed in v0.24. Please use `LLM.encode` with `pooling_task="classify"` or `pooling_task="token_classify"` instead.