A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Doc] Convert list tables to MyST (#11594)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Cyrus Leung committed
Dec 29, 2024 at 15:56 UTC
32b4c63f02b2ab28a49a040b1d170a903a5cd9dc
6 files changed
+961
-975
docs/source/getting_started/debugging.md
+1
-1
@@ -197,4 +197,4 @@ if __name__ == '__main__':
197
## Known Issues
198
199
- In `v0.5.2`, `v0.5.3`, and `v0.5.3.post1`, there is a bug caused by [zmq](https://github.com/zeromq/pyzmq/issues/2000) , which can occasionally cause vLLM to hang depending on the machine configuration. The solution is to upgrade to the latest version of `vllm` to include the [fix](gh-pr:6759).
200
-- To circumvent a NCCL [bug](https://github.com/NVIDIA/nccl/issues/1234) , all vLLM processes will set an environment variable ``NCCL_CUMEM_ENABLE=0`` to disable NCCL's ``cuMem`` allocator. It does not affect performance but only gives memory benefits. When external processes want to set up a NCCL connection with vLLM's processes, they should also set this environment variable, otherwise, inconsistent environment setup will cause NCCL to hang or crash, as observed in the [RLHF integration](https://github.com/OpenRLHF/OpenRLHF/pull/604) and the [discussion](gh-issue:5723#issuecomment-2554389656) .
200
+- To circumvent a NCCL [bug](https://github.com/NVIDIA/nccl/issues/1234) , all vLLM processes will set an environment variable `NCCL_CUMEM_ENABLE=0` to disable NCCL's `cuMem` allocator. It does not affect performance but only gives memory benefits. When external processes want to set up a NCCL connection with vLLM's processes, they should also set this environment variable, otherwise, inconsistent environment setup will cause NCCL to hang or crash, as observed in the [RLHF integration](https://github.com/OpenRLHF/OpenRLHF/pull/604) and the [discussion](gh-issue:5723#issuecomment-2554389656) .
docs/source/getting_started/gaudi-installation.md
+19
-20
@@ -141,26 +141,25 @@ Gaudi2 devices. Configurations that are not listed may or may not work.
141
142
Currently in vLLM for HPU we support four execution modes, depending on selected HPU PyTorch Bridge backend (via `PT_HPU_LAZY_MODE` environment variable), and `--enforce-eager` flag.
143
144
-```{eval-rst}
145
-.. list-table:: vLLM execution modes
146
- :widths: 25 25 50
147
- :header-rows: 1
148
-
149
- * - ``PT_HPU_LAZY_MODE``
150
- - ``enforce_eager``
151
- - execution mode
152
- * - 0
153
- - 0
154
- - torch.compile
155
- * - 0
156
- - 1
157
- - PyTorch eager mode
158
- * - 1
159
- - 0
160
- - HPU Graphs
161
- * - 1
162
- - 1
163
- - PyTorch lazy mode
144
+```{list-table} vLLM execution modes
145
+:widths: 25 25 50
146
+:header-rows: 1
147
+
148
+* - `PT_HPU_LAZY_MODE`
149
+ - `enforce_eager`
150
+ - execution mode
151
+* - 0
152
+ - 0
153
+ - torch.compile
154
+* - 0
155
+ - 1
156
+ - PyTorch eager mode
157
+* - 1
158
+ - 0
159
+ - HPU Graphs
160
+* - 1
161
+ - 1
162
+ - PyTorch lazy mode
163
```
164
165
```{warning}
docs/source/getting_started/tpu-installation.md
+26
-27
@@ -68,33 +68,32 @@ gcloud alpha compute tpus queued-resources create QUEUED_RESOURCE_ID \
68
--service-account SERVICE_ACCOUNT
69
```
70
71
-```{eval-rst}
72
-.. list-table:: Parameter descriptions
73
- :header-rows: 1
74
-
75
- * - Parameter name
76
- - Description
77
- * - QUEUED_RESOURCE_ID
78
- - The user-assigned ID of the queued resource request.
79
- * - TPU_NAME
80
- - The user-assigned name of the TPU which is created when the queued
81
- resource request is allocated.
82
- * - PROJECT_ID
83
- - Your Google Cloud project
84
- * - ZONE
85
- - The GCP zone where you want to create your Cloud TPU. The value you use
86
- depends on the version of TPUs you are using. For more information, see
87
- `TPU regions and zones <https://cloud.google.com/tpu/docs/regions-zones>`_
88
- * - ACCELERATOR_TYPE
89
- - The TPU version you want to use. Specify the TPU version, for example
90
- `v5litepod-4` specifies a v5e TPU with 4 cores. For more information,
91
- see `TPU versions <https://cloud.devsite.corp.google.com/tpu/docs/system-architecture-tpu-vm#versions>`_.
92
- * - RUNTIME_VERSION
93
- - The TPU VM runtime version to use. For more information see `TPU VM images <https://cloud.google.com/tpu/docs/runtimes>`_.
94
- * - SERVICE_ACCOUNT
95
- - The email address for your service account. You can find it in the IAM
96
- Cloud Console under *Service Accounts*. For example:
97
- `tpu-service-account@<your_project_ID>.iam.gserviceaccount.com`
71
+```{list-table} Parameter descriptions
72
+:header-rows: 1
73
+
74
+* - Parameter name
75
+ - Description
76
+* - QUEUED_RESOURCE_ID
77
+ - The user-assigned ID of the queued resource request.
78
+* - TPU_NAME
79
+ - The user-assigned name of the TPU which is created when the queued
80
+ resource request is allocated.
81
+* - PROJECT_ID
82
+ - Your Google Cloud project
83
+* - ZONE
84
+ - The GCP zone where you want to create your Cloud TPU. The value you use
85
+ depends on the version of TPUs you are using. For more information, see
86
+ `TPU regions and zones <https://cloud.google.com/tpu/docs/regions-zones>`_
87
+* - ACCELERATOR_TYPE
88
+ - The TPU version you want to use. Specify the TPU version, for example
89
+ `v5litepod-4` specifies a v5e TPU with 4 cores. For more information,
90
+ see `TPU versions <https://cloud.devsite.corp.google.com/tpu/docs/system-architecture-tpu-vm#versions>`_.
91
+* - RUNTIME_VERSION
92
+ - The TPU VM runtime version to use. For more information see `TPU VM images <https://cloud.google.com/tpu/docs/runtimes>`_.
93
+* - SERVICE_ACCOUNT
94
+ - The email address for your service account. You can find it in the IAM
95
+ Cloud Console under *Service Accounts*. For example:
96
+ `tpu-service-account@<your_project_ID>.iam.gserviceaccount.com`
97
```
98
99
Connect to your TPU using SSH:
docs/source/models/supported_models.md
+598
-608
@@ -72,291 +72,290 @@ See [this page](#generative-models) for more information on how to use generativ
72
73
#### Text Generation (`--task generate`)
74
75
-```{eval-rst}
76
-.. list-table::
77
- :widths: 25 25 50 5 5
78
- :header-rows: 1
79
-
80
- * - Architecture
81
- - Models
82
- - Example HF Models
83
- - :ref:`LoRA <lora-adapter>`
84
- - :ref:`PP <distributed-serving>`
85
- * - :code:`AquilaForCausalLM`
86
- - Aquila, Aquila2
87
- - :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
88
- - ✅︎
89
- - ✅︎
90
- * - :code:`ArcticForCausalLM`
91
- - Arctic
92
- - :code:`Snowflake/snowflake-arctic-base`, :code:`Snowflake/snowflake-arctic-instruct`, etc.
93
- -
94
- - ✅︎
95
- * - :code:`BaiChuanForCausalLM`
96
- - Baichuan2, Baichuan
97
- - :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
98
- - ✅︎
99
- - ✅︎
100
- * - :code:`BloomForCausalLM`
101
- - BLOOM, BLOOMZ, BLOOMChat
102
- - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
103
- -
104
- - ✅︎
105
- * - :code:`BartForConditionalGeneration`
106
- - BART
107
- - :code:`facebook/bart-base`, :code:`facebook/bart-large-cnn`, etc.
108
- -
109
- -
110
- * - :code:`ChatGLMModel`
111
- - ChatGLM
112
- - :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
113
- - ✅︎
114
- - ✅︎
115
- * - :code:`CohereForCausalLM`, :code:`Cohere2ForCausalLM`
116
- - Command-R
117
- - :code:`CohereForAI/c4ai-command-r-v01`, :code:`CohereForAI/c4ai-command-r7b-12-2024`, etc.
118
- - ✅︎
119
- - ✅︎
120
- * - :code:`DbrxForCausalLM`
121
- - DBRX
122
- - :code:`databricks/dbrx-base`, :code:`databricks/dbrx-instruct`, etc.
123
- -
124
- - ✅︎
125
- * - :code:`DeciLMForCausalLM`
126
- - DeciLM
127
- - :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
128
- -
129
- - ✅︎
130
- * - :code:`DeepseekForCausalLM`
131
- - DeepSeek
132
- - :code:`deepseek-ai/deepseek-llm-67b-base`, :code:`deepseek-ai/deepseek-llm-7b-chat` etc.
133
- -
134
- - ✅︎
135
- * - :code:`DeepseekV2ForCausalLM`
136
- - DeepSeek-V2
137
- - :code:`deepseek-ai/DeepSeek-V2`, :code:`deepseek-ai/DeepSeek-V2-Chat` etc.
138
- -
139
- - ✅︎
140
- * - :code:`DeepseekV3ForCausalLM`
141
- - DeepSeek-V3
142
- - :code:`deepseek-ai/DeepSeek-V3-Base`, :code:`deepseek-ai/DeepSeek-V3` etc.
143
- -
144
- - ✅︎
145
- * - :code:`ExaoneForCausalLM`
146
- - EXAONE-3
147
- - :code:`LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
148
- - ✅︎
149
- - ✅︎
150
- * - :code:`FalconForCausalLM`
151
- - Falcon
152
- - :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
153
- -
154
- - ✅︎
155
- * - :code:`FalconMambaForCausalLM`
156
- - FalconMamba
157
- - :code:`tiiuae/falcon-mamba-7b`, :code:`tiiuae/falcon-mamba-7b-instruct`, etc.
158
- - ✅︎
159
- - ✅︎
160
- * - :code:`GemmaForCausalLM`
161
- - Gemma
162
- - :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
163
- - ✅︎
164
- - ✅︎
165
- * - :code:`Gemma2ForCausalLM`
166
- - Gemma2
167
- - :code:`google/gemma-2-9b`, :code:`google/gemma-2-27b`, etc.
168
- - ✅︎
169
- - ✅︎
170
- * - :code:`GlmForCausalLM`
171
- - GLM-4
172
- - :code:`THUDM/glm-4-9b-chat-hf`, etc.
173
- - ✅︎
174
- - ✅︎
175
- * - :code:`GPT2LMHeadModel`
176
- - GPT-2
177
- - :code:`gpt2`, :code:`gpt2-xl`, etc.
178
- -
179
- - ✅︎
180
- * - :code:`GPTBigCodeForCausalLM`
181
- - StarCoder, SantaCoder, WizardCoder
182
- - :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
183
- - ✅︎
184
- - ✅︎
185
- * - :code:`GPTJForCausalLM`
186
- - GPT-J
187
- - :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
188
- -
189
- - ✅︎
190
- * - :code:`GPTNeoXForCausalLM`
191
- - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
192
- - :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
193
- -
194
- - ✅︎
195
- * - :code:`GraniteForCausalLM`
196
- - Granite 3.0, Granite 3.1, PowerLM
197
- - :code:`ibm-granite/granite-3.0-2b-base`, :code:`ibm-granite/granite-3.1-8b-instruct`, :code:`ibm/PowerLM-3b`, etc.
198
- - ✅︎
199
- - ✅︎
200
- * - :code:`GraniteMoeForCausalLM`
201
- - Granite 3.0 MoE, PowerMoE
202
- - :code:`ibm-granite/granite-3.0-1b-a400m-base`, :code:`ibm-granite/granite-3.0-3b-a800m-instruct`, :code:`ibm/PowerMoE-3b`, etc.
203
- - ✅︎
204
- - ✅︎
205
- * - :code:`GritLM`
206
- - GritLM
207
- - :code:`parasail-ai/GritLM-7B-vllm`.
208
- - ✅︎
209
- - ✅︎
210
- * - :code:`InternLMForCausalLM`
211
- - InternLM
212
- - :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
213
- - ✅︎
214
- - ✅︎
215
- * - :code:`InternLM2ForCausalLM`
216
- - InternLM2
217
- - :code:`internlm/internlm2-7b`, :code:`internlm/internlm2-chat-7b`, etc.
218
- - ✅︎
219
- - ✅︎
220
- * - :code:`JAISLMHeadModel`
221
- - Jais
222
- - :code:`inceptionai/jais-13b`, :code:`inceptionai/jais-13b-chat`, :code:`inceptionai/jais-30b-v3`, :code:`inceptionai/jais-30b-chat-v3`, etc.
223
- -
224
- - ✅︎
225
- * - :code:`JambaForCausalLM`
226
- - Jamba
227
- - :code:`ai21labs/AI21-Jamba-1.5-Large`, :code:`ai21labs/AI21-Jamba-1.5-Mini`, :code:`ai21labs/Jamba-v0.1`, etc.
228
- - ✅︎
229
- - ✅︎
230
- * - :code:`LlamaForCausalLM`
231
- - Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
232
- - :code:`meta-llama/Meta-Llama-3.1-405B-Instruct`, :code:`meta-llama/Meta-Llama-3.1-70B`, :code:`meta-llama/Meta-Llama-3-70B-Instruct`, :code:`meta-llama/Llama-2-70b-hf`, :code:`01-ai/Yi-34B`, etc.
233
- - ✅︎
234
- - ✅︎
235
- * - :code:`MambaForCausalLM`
236
- - Mamba
237
- - :code:`state-spaces/mamba-130m-hf`, :code:`state-spaces/mamba-790m-hf`, :code:`state-spaces/mamba-2.8b-hf`, etc.
238
- -
239
- - ✅︎
240
- * - :code:`MiniCPMForCausalLM`
241
- - MiniCPM
242
- - :code:`openbmb/MiniCPM-2B-sft-bf16`, :code:`openbmb/MiniCPM-2B-dpo-bf16`, :code:`openbmb/MiniCPM-S-1B-sft`, etc.
243
- - ✅︎
244
- - ✅︎
245
- * - :code:`MiniCPM3ForCausalLM`
246
- - MiniCPM3
247
- - :code:`openbmb/MiniCPM3-4B`, etc.
248
- - ✅︎
249
- - ✅︎
250
- * - :code:`MistralForCausalLM`
251
- - Mistral, Mistral-Instruct
252
- - :code:`mistralai/Mistral-7B-v0.1`, :code:`mistralai/Mistral-7B-Instruct-v0.1`, etc.
253
- - ✅︎
254
- - ✅︎
255
- * - :code:`MixtralForCausalLM`
256
- - Mixtral-8x7B, Mixtral-8x7B-Instruct
257
- - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, :code:`mistral-community/Mixtral-8x22B-v0.1`, etc.
258
- - ✅︎
259
- - ✅︎
260
- * - :code:`MPTForCausalLM`
261
- - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
262
- - :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
263
- -
264
- - ✅︎
265
- * - :code:`NemotronForCausalLM`
266
- - Nemotron-3, Nemotron-4, Minitron
267
- - :code:`nvidia/Minitron-8B-Base`, :code:`mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
268
- - ✅︎
269
- - ✅︎
270
- * - :code:`OLMoForCausalLM`
271
- - OLMo
272
- - :code:`allenai/OLMo-1B-hf`, :code:`allenai/OLMo-7B-hf`, etc.
273
- -
274
- - ✅︎
275
- * - :code:`OLMo2ForCausalLM`
276
- - OLMo2
277
- - :code:`allenai/OLMo2-7B-1124`, etc.
278
- -
279
- - ✅︎
280
- * - :code:`OLMoEForCausalLM`
281
- - OLMoE
282
- - :code:`allenai/OLMoE-1B-7B-0924`, :code:`allenai/OLMoE-1B-7B-0924-Instruct`, etc.
283
- - ✅︎
284
- - ✅︎
285
- * - :code:`OPTForCausalLM`
286
- - OPT, OPT-IML
287
- - :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
288
- -
289
- - ✅︎
290
- * - :code:`OrionForCausalLM`
291
- - Orion
292
- - :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
293
- -
294
- - ✅︎
295
- * - :code:`PhiForCausalLM`
296
- - Phi
297
- - :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
298
- - ✅︎
299
- - ✅︎
300
- * - :code:`Phi3ForCausalLM`
301
- - Phi-3
302
- - :code:`microsoft/Phi-3-mini-4k-instruct`, :code:`microsoft/Phi-3-mini-128k-instruct`, :code:`microsoft/Phi-3-medium-128k-instruct`, etc.
303
- - ✅︎
304
- - ✅︎
305
- * - :code:`Phi3SmallForCausalLM`
306
- - Phi-3-Small
307
- - :code:`microsoft/Phi-3-small-8k-instruct`, :code:`microsoft/Phi-3-small-128k-instruct`, etc.
308
- -
309
- - ✅︎
310
- * - :code:`PhiMoEForCausalLM`
311
- - Phi-3.5-MoE
312
- - :code:`microsoft/Phi-3.5-MoE-instruct`, etc.
313
- - ✅︎
314
- - ✅︎
315
- * - :code:`PersimmonForCausalLM`
316
- - Persimmon
317
- - :code:`adept/persimmon-8b-base`, :code:`adept/persimmon-8b-chat`, etc.
318
- -
319
- - ✅︎
320
- * - :code:`QWenLMHeadModel`
321
- - Qwen
322
- - :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
323
- - ✅︎
324
- - ✅︎
325
- * - :code:`Qwen2ForCausalLM`
326
- - Qwen2
327
- - :code:`Qwen/QwQ-32B-Preview`, :code:`Qwen/Qwen2-7B-Instruct`, :code:`Qwen/Qwen2-7B`, etc.
328
- - ✅︎
329
- - ✅︎
330
- * - :code:`Qwen2MoeForCausalLM`
331
- - Qwen2MoE
332
- - :code:`Qwen/Qwen1.5-MoE-A2.7B`, :code:`Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
333
- -
334
- - ✅︎
335
- * - :code:`StableLmForCausalLM`
336
- - StableLM
337
- - :code:`stabilityai/stablelm-3b-4e1t`, :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
338
- -
339
- - ✅︎
340
- * - :code:`Starcoder2ForCausalLM`
341
- - Starcoder2
342
- - :code:`bigcode/starcoder2-3b`, :code:`bigcode/starcoder2-7b`, :code:`bigcode/starcoder2-15b`, etc.
343
- -
344
- - ✅︎
345
- * - :code:`SolarForCausalLM`
346
- - Solar Pro
347
- - :code:`upstage/solar-pro-preview-instruct`, etc.
348
- - ✅︎
349
- - ✅︎
350
- * - :code:`TeleChat2ForCausalLM`
351
- - TeleChat2
352
- - :code:`TeleAI/TeleChat2-3B`, :code:`TeleAI/TeleChat2-7B`, :code:`TeleAI/TeleChat2-35B`, etc.
353
- - ✅︎
354
- - ✅︎
355
- * - :code:`XverseForCausalLM`
356
- - XVERSE
357
- - :code:`xverse/XVERSE-7B-Chat`, :code:`xverse/XVERSE-13B-Chat`, :code:`xverse/XVERSE-65B-Chat`, etc.
358
- - ✅︎
359
- - ✅︎
75
+```{list-table}
76
+:widths: 25 25 50 5 5
77
+:header-rows: 1
78
+
79
+* - Architecture
80
+ - Models
81
+ - Example HF Models
82
+ - [LoRA](#lora-adapter)
83
+ - [PP](#distributed-serving)
84
+* - `AquilaForCausalLM`
85
+ - Aquila, Aquila2
86
+ - `BAAI/Aquila-7B`, `BAAI/AquilaChat-7B`, etc.
87
+ - ✅︎
88
+ - ✅︎
89
+* - `ArcticForCausalLM`
90
+ - Arctic
91
+ - `Snowflake/snowflake-arctic-base`, `Snowflake/snowflake-arctic-instruct`, etc.
92
+ -
93
+ - ✅︎
94
+* - `BaiChuanForCausalLM`
95
+ - Baichuan2, Baichuan
96
+ - `baichuan-inc/Baichuan2-13B-Chat`, `baichuan-inc/Baichuan-7B`, etc.
97
+ - ✅︎
98
+ - ✅︎
99
+* - `BloomForCausalLM`
100
+ - BLOOM, BLOOMZ, BLOOMChat
101
+ - `bigscience/bloom`, `bigscience/bloomz`, etc.
102
+ -
103
+ - ✅︎
104
+* - `BartForConditionalGeneration`
105
+ - BART
106
+ - `facebook/bart-base`, `facebook/bart-large-cnn`, etc.
107
+ -
108
+ -
109
+* - `ChatGLMModel`
110
+ - ChatGLM
111
+ - `THUDM/chatglm2-6b`, `THUDM/chatglm3-6b`, etc.
112
+ - ✅︎
113
+ - ✅︎
114
+* - `CohereForCausalLM`, `Cohere2ForCausalLM`
115
+ - Command-R
116
+ - `CohereForAI/c4ai-command-r-v01`, `CohereForAI/c4ai-command-r7b-12-2024`, etc.
117
+ - ✅︎
118
+ - ✅︎
119
+* - `DbrxForCausalLM`
120
+ - DBRX
121
+ - `databricks/dbrx-base`, `databricks/dbrx-instruct`, etc.
122
+ -
123
+ - ✅︎
124
+* - `DeciLMForCausalLM`
125
+ - DeciLM
126
+ - `Deci/DeciLM-7B`, `Deci/DeciLM-7B-instruct`, etc.
127
+ -
128
+ - ✅︎
129
+* - `DeepseekForCausalLM`
130
+ - DeepSeek
131
+ - `deepseek-ai/deepseek-llm-67b-base`, `deepseek-ai/deepseek-llm-7b-chat` etc.
132
+ -
133
+ - ✅︎
134
+* - `DeepseekV2ForCausalLM`
135
+ - DeepSeek-V2
136
+ - `deepseek-ai/DeepSeek-V2`, `deepseek-ai/DeepSeek-V2-Chat` etc.
137
+ -
138
+ - ✅︎
139
+* - `DeepseekV3ForCausalLM`
140
+ - DeepSeek-V3
141
+ - `deepseek-ai/DeepSeek-V3-Base`, `deepseek-ai/DeepSeek-V3` etc.
142
+ -
143
+ - ✅︎
144
+* - `ExaoneForCausalLM`
145
+ - EXAONE-3
146
+ - `LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
147
+ - ✅︎
148
+ - ✅︎
149
+* - `FalconForCausalLM`
150
+ - Falcon
151
+ - `tiiuae/falcon-7b`, `tiiuae/falcon-40b`, `tiiuae/falcon-rw-7b`, etc.
152
+ -
153
+ - ✅︎
154
+* - `FalconMambaForCausalLM`
155
+ - FalconMamba
156
+ - `tiiuae/falcon-mamba-7b`, `tiiuae/falcon-mamba-7b-instruct`, etc.
157
+ - ✅︎
158
+ - ✅︎
159
+* - `GemmaForCausalLM`
160
+ - Gemma
161
+ - `google/gemma-2b`, `google/gemma-7b`, etc.
162
+ - ✅︎
163
+ - ✅︎
164
+* - `Gemma2ForCausalLM`
165
+ - Gemma2
166
+ - `google/gemma-2-9b`, `google/gemma-2-27b`, etc.
167
+ - ✅︎
168
+ - ✅︎
169
+* - `GlmForCausalLM`
170
+ - GLM-4
171
+ - `THUDM/glm-4-9b-chat-hf`, etc.
172
+ - ✅︎
173
+ - ✅︎
174
+* - `GPT2LMHeadModel`
175
+ - GPT-2
176
+ - `gpt2`, `gpt2-xl`, etc.
177
+ -
178
+ - ✅︎
179
+* - `GPTBigCodeForCausalLM`
180
+ - StarCoder, SantaCoder, WizardCoder
181
+ - `bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, `WizardLM/WizardCoder-15B-V1.0`, etc.
182
+ - ✅︎
183
+ - ✅︎
184
+* - `GPTJForCausalLM`
185
+ - GPT-J
186
+ - `EleutherAI/gpt-j-6b`, `nomic-ai/gpt4all-j`, etc.
187
+ -
188
+ - ✅︎
189
+* - `GPTNeoXForCausalLM`
190
+ - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
191
+ - `EleutherAI/gpt-neox-20b`, `EleutherAI/pythia-12b`, `OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, `databricks/dolly-v2-12b`, `stabilityai/stablelm-tuned-alpha-7b`, etc.
192
+ -
193
+ - ✅︎
194
+* - `GraniteForCausalLM`
195
+ - Granite 3.0, Granite 3.1, PowerLM
196
+ - `ibm-granite/granite-3.0-2b-base`, `ibm-granite/granite-3.1-8b-instruct`, `ibm/PowerLM-3b`, etc.
197
+ - ✅︎
198
+ - ✅︎
199
+* - `GraniteMoeForCausalLM`
200
+ - Granite 3.0 MoE, PowerMoE
201
+ - `ibm-granite/granite-3.0-1b-a400m-base`, `ibm-granite/granite-3.0-3b-a800m-instruct`, `ibm/PowerMoE-3b`, etc.
202
+ - ✅︎
203
+ - ✅︎
204
+* - `GritLM`
205
+ - GritLM
206
+ - `parasail-ai/GritLM-7B-vllm`.
207
+ - ✅︎
208
+ - ✅︎
209
+* - `InternLMForCausalLM`
210
+ - InternLM
211
+ - `internlm/internlm-7b`, `internlm/internlm-chat-7b`, etc.
212
+ - ✅︎
213
+ - ✅︎
214
+* - `InternLM2ForCausalLM`
215
+ - InternLM2
216
+ - `internlm/internlm2-7b`, `internlm/internlm2-chat-7b`, etc.
217
+ - ✅︎
218
+ - ✅︎
219
+* - `JAISLMHeadModel`
220
+ - Jais
221
+ - `inceptionai/jais-13b`, `inceptionai/jais-13b-chat`, `inceptionai/jais-30b-v3`, `inceptionai/jais-30b-chat-v3`, etc.
222
+ -
223
+ - ✅︎
224
+* - `JambaForCausalLM`
225
+ - Jamba
226
+ - `ai21labs/AI21-Jamba-1.5-Large`, `ai21labs/AI21-Jamba-1.5-Mini`, `ai21labs/Jamba-v0.1`, etc.
227
+ - ✅︎
228
+ - ✅︎
229
+* - `LlamaForCausalLM`
230
+ - Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
231
+ - `meta-llama/Meta-Llama-3.1-405B-Instruct`, `meta-llama/Meta-Llama-3.1-70B`, `meta-llama/Meta-Llama-3-70B-Instruct`, `meta-llama/Llama-2-70b-hf`, `01-ai/Yi-34B`, etc.
232
+ - ✅︎
233
+ - ✅︎
234
+* - `MambaForCausalLM`
235
+ - Mamba
236
+ - `state-spaces/mamba-130m-hf`, `state-spaces/mamba-790m-hf`, `state-spaces/mamba-2.8b-hf`, etc.
237
+ -
238
+ - ✅︎
239
+* - `MiniCPMForCausalLM`
240
+ - MiniCPM
241
+ - `openbmb/MiniCPM-2B-sft-bf16`, `openbmb/MiniCPM-2B-dpo-bf16`, `openbmb/MiniCPM-S-1B-sft`, etc.
242
+ - ✅︎
243
+ - ✅︎
244
+* - `MiniCPM3ForCausalLM`
245
+ - MiniCPM3
246
+ - `openbmb/MiniCPM3-4B`, etc.
247
+ - ✅︎
248
+ - ✅︎
249
+* - `MistralForCausalLM`
250
+ - Mistral, Mistral-Instruct
251
+ - `mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc.
252
+ - ✅︎
253
+ - ✅︎
254
+* - `MixtralForCausalLM`
255
+ - Mixtral-8x7B, Mixtral-8x7B-Instruct
256
+ - `mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc.
257
+ - ✅︎
258
+ - ✅︎
259
+* - `MPTForCausalLM`
260
+ - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
261
+ - `mosaicml/mpt-7b`, `mosaicml/mpt-7b-storywriter`, `mosaicml/mpt-30b`, etc.
262
+ -
263
+ - ✅︎
264
+* - `NemotronForCausalLM`
265
+ - Nemotron-3, Nemotron-4, Minitron
266
+ - `nvidia/Minitron-8B-Base`, `mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
267
+ - ✅︎
268
+ - ✅︎
269
+* - `OLMoForCausalLM`
270
+ - OLMo
271
+ - `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc.
272
+ -
273
+ - ✅︎
274
+* - `OLMo2ForCausalLM`
275
+ - OLMo2
276
+ - `allenai/OLMo2-7B-1124`, etc.
277
+ -
278
+ - ✅︎
279
+* - `OLMoEForCausalLM`
280
+ - OLMoE
281
+ - `allenai/OLMoE-1B-7B-0924`, `allenai/OLMoE-1B-7B-0924-Instruct`, etc.
282
+ - ✅︎
283
+ - ✅︎
284
+* - `OPTForCausalLM`
285
+ - OPT, OPT-IML
286
+ - `facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.
287
+ -
288
+ - ✅︎
289
+* - `OrionForCausalLM`
290
+ - Orion
291
+ - `OrionStarAI/Orion-14B-Base`, `OrionStarAI/Orion-14B-Chat`, etc.
292
+ -
293
+ - ✅︎
294
+* - `PhiForCausalLM`
295
+ - Phi
296
+ - `microsoft/phi-1_5`, `microsoft/phi-2`, etc.
297
+ - ✅︎
298
+ - ✅︎
299
+* - `Phi3ForCausalLM`
300
+ - Phi-3
301
+ - `microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-mini-128k-instruct`, `microsoft/Phi-3-medium-128k-instruct`, etc.
302
+ - ✅︎
303
+ - ✅︎
304
+* - `Phi3SmallForCausalLM`
305
+ - Phi-3-Small
306
+ - `microsoft/Phi-3-small-8k-instruct`, `microsoft/Phi-3-small-128k-instruct`, etc.
307
+ -
308
+ - ✅︎
309
+* - `PhiMoEForCausalLM`
310
+ - Phi-3.5-MoE
311
+ - `microsoft/Phi-3.5-MoE-instruct`, etc.
312
+ - ✅︎
313
+ - ✅︎
314
+* - `PersimmonForCausalLM`
315
+ - Persimmon
316
+ - `adept/persimmon-8b-base`, `adept/persimmon-8b-chat`, etc.
317
+ -
318
+ - ✅︎
319
+* - `QWenLMHeadModel`
320
+ - Qwen
321
+ - `Qwen/Qwen-7B`, `Qwen/Qwen-7B-Chat`, etc.
322
+ - ✅︎
323
+ - ✅︎
324
+* - `Qwen2ForCausalLM`
325
+ - Qwen2
326
+ - `Qwen/QwQ-32B-Preview`, `Qwen/Qwen2-7B-Instruct`, `Qwen/Qwen2-7B`, etc.
327
+ - ✅︎
328
+ - ✅︎
329
+* - `Qwen2MoeForCausalLM`
330
+ - Qwen2MoE
331
+ - `Qwen/Qwen1.5-MoE-A2.7B`, `Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
332
+ -
333
+ - ✅︎
334
+* - `StableLmForCausalLM`
335
+ - StableLM
336
+ - `stabilityai/stablelm-3b-4e1t`, `stabilityai/stablelm-base-alpha-7b-v2`, etc.
337
+ -
338
+ - ✅︎
339
+* - `Starcoder2ForCausalLM`
340
+ - Starcoder2
341
+ - `bigcode/starcoder2-3b`, `bigcode/starcoder2-7b`, `bigcode/starcoder2-15b`, etc.
342
+ -
343
+ - ✅︎
344
+* - `SolarForCausalLM`
345
+ - Solar Pro
346
+ - `upstage/solar-pro-preview-instruct`, etc.
347
+ - ✅︎
348
+ - ✅︎
349
+* - `TeleChat2ForCausalLM`
350
+ - TeleChat2
351
+ - `TeleAI/TeleChat2-3B`, `TeleAI/TeleChat2-7B`, `TeleAI/TeleChat2-35B`, etc.
352
+ - ✅︎
353
+ - ✅︎
354
+* - `XverseForCausalLM`
355
+ - XVERSE
356
+ - `xverse/XVERSE-7B-Chat`, `xverse/XVERSE-13B-Chat`, `xverse/XVERSE-65B-Chat`, etc.
357
+ - ✅︎
358
+ - ✅︎
359
```
360
361
```{note}
@@ -374,51 +373,50 @@ you should explicitly specify the task type to ensure that the model is used in
373
374
#### Text Embedding (`--task embed`)
375
377
-```{eval-rst}
378
-.. list-table::
379
- :widths: 25 25 50 5 5
380
- :header-rows: 1
381
-
382
- * - Architecture
383
- - Models
384
- - Example HF Models
385
- - :ref:`LoRA <lora-adapter>`
386
- - :ref:`PP <distributed-serving>`
387
- * - :code:`BertModel`
388
- - BERT-based
389
- - :code:`BAAI/bge-base-en-v1.5`, etc.
390
- -
391
- -
392
- * - :code:`Gemma2Model`
393
- - Gemma2-based
394
- - :code:`BAAI/bge-multilingual-gemma2`, etc.
395
- -
396
- - ✅︎
397
- * - :code:`GritLM`
398
- - GritLM
399
- - :code:`parasail-ai/GritLM-7B-vllm`.
400
- - ✅︎
401
- - ✅︎
402
- * - :code:`LlamaModel`, :code:`LlamaForCausalLM`, :code:`MistralModel`, etc.
403
- - Llama-based
404
- - :code:`intfloat/e5-mistral-7b-instruct`, etc.
405
- - ✅︎
406
- - ✅︎
407
- * - :code:`Qwen2Model`, :code:`Qwen2ForCausalLM`
408
- - Qwen2-based
409
- - :code:`ssmits/Qwen2-7B-Instruct-embed-base` (see note), :code:`Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc.
410
- - ✅︎
411
- - ✅︎
412
- * - :code:`RobertaModel`, :code:`RobertaForMaskedLM`
413
- - RoBERTa-based
414
- - :code:`sentence-transformers/all-roberta-large-v1`, :code:`sentence-transformers/all-roberta-large-v1`, etc.
415
- -
416
- -
417
- * - :code:`XLMRobertaModel`
418
- - XLM-RoBERTa-based
419
- - :code:`intfloat/multilingual-e5-large`, etc.
420
- -
421
- -
376
+```{list-table}
377
+:widths: 25 25 50 5 5
378
+:header-rows: 1
379
+
380
+* - Architecture
381
+ - Models
382
+ - Example HF Models
383
+ - [LoRA](#lora-adapter)
384
+ - [PP](#distributed-serving)
385
+* - `BertModel`
386
+ - BERT-based
387
+ - `BAAI/bge-base-en-v1.5`, etc.
388
+ -
389
+ -
390
+* - `Gemma2Model`
391
+ - Gemma2-based
392
+ - `BAAI/bge-multilingual-gemma2`, etc.
393
+ -
394
+ - ✅︎
395
+* - `GritLM`
396
+ - GritLM
397
+ - `parasail-ai/GritLM-7B-vllm`.
398
+ - ✅︎
399
+ - ✅︎
400
+* - `LlamaModel`, `LlamaForCausalLM`, `MistralModel`, etc.
401
+ - Llama-based
402
+ - `intfloat/e5-mistral-7b-instruct`, etc.
403
+ - ✅︎
404
+ - ✅︎
405
+* - `Qwen2Model`, `Qwen2ForCausalLM`
406
+ - Qwen2-based
407
+ - `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc.
408
+ - ✅︎
409
+ - ✅︎
410
+* - `RobertaModel`, `RobertaForMaskedLM`
411
+ - RoBERTa-based
412
+ - `sentence-transformers/all-roberta-large-v1`, `sentence-transformers/all-roberta-large-v1`, etc.
413
+ -
414
+ -
415
+* - `XLMRobertaModel`
416
+ - XLM-RoBERTa-based
417
+ - `intfloat/multilingual-e5-large`, etc.
418
+ -
419
+ -
420
```
421
422
```{note}
@@ -440,31 +438,30 @@ of the whole prompt are extracted from the normalized hidden state corresponding
438
439
#### Reward Modeling (`--task reward`)
440
443
-```{eval-rst}
444
-.. list-table::
445
- :widths: 25 25 50 5 5
446
- :header-rows: 1
447
-
448
- * - Architecture
449
- - Models
450
- - Example HF Models
451
- - :ref:`LoRA <lora-adapter>`
452
- - :ref:`PP <distributed-serving>`
453
- * - :code:`InternLM2ForRewardModel`
454
- - InternLM2-based
455
- - :code:`internlm/internlm2-1_8b-reward`, :code:`internlm/internlm2-7b-reward`, etc.
456
- - ✅︎
457
- - ✅︎
458
- * - :code:`LlamaForCausalLM`
459
- - Llama-based
460
- - :code:`peiyi9979/math-shepherd-mistral-7b-prm`, etc.
461
- - ✅︎
462
- - ✅︎
463
- * - :code:`Qwen2ForRewardModel`
464
- - Qwen2-based
465
- - :code:`Qwen/Qwen2.5-Math-RM-72B`, etc.
466
- - ✅︎
467
- - ✅︎
441
+```{list-table}
442
+:widths: 25 25 50 5 5
443
+:header-rows: 1
444
+
445
+* - Architecture
446
+ - Models
447
+ - Example HF Models
448
+ - [LoRA](#lora-adapter)
449
+ - [PP](#distributed-serving)
450
+* - `InternLM2ForRewardModel`
451
+ - InternLM2-based
452
+ - `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc.
453
+ - ✅︎
454
+ - ✅︎
455
+* - `LlamaForCausalLM`
456
+ - Llama-based
457
+ - `peiyi9979/math-shepherd-mistral-7b-prm`, etc.
458
+ - ✅︎
459
+ - ✅︎
460
+* - `Qwen2ForRewardModel`
461
+ - Qwen2-based
462
+ - `Qwen/Qwen2.5-Math-RM-72B`, etc.
463
+ - ✅︎
464
+ - ✅︎
465
```
466
467
If your model is not in the above list, we will try to automatically convert the model using
@@ -477,26 +474,25 @@ e.g.: {code}`--override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 1
474
475
#### Classification (`--task classify`)
476
480
-```{eval-rst}
481
-.. list-table::
482
- :widths: 25 25 50 5 5
483
- :header-rows: 1
484
-
485
- * - Architecture
486
- - Models
487
- - Example HF Models
488
- - :ref:`LoRA <lora-adapter>`
489
- - :ref:`PP <distributed-serving>`
490
- * - :code:`JambaForSequenceClassification`
491
- - Jamba
492
- - :code:`ai21labs/Jamba-tiny-reward-dev`, etc.
493
- - ✅︎
494
- - ✅︎
495
- * - :code:`Qwen2ForSequenceClassification`
496
- - Qwen2-based
497
- - :code:`jason9693/Qwen2.5-1.5B-apeach`, etc.
498
- - ✅︎
499
- - ✅︎
477
+```{list-table}
478
+:widths: 25 25 50 5 5
479
+:header-rows: 1
480
+
481
+* - Architecture
482
+ - Models
483
+ - Example HF Models
484
+ - [LoRA](#lora-adapter)
485
+ - [PP](#distributed-serving)
486
+* - `JambaForSequenceClassification`
487
+ - Jamba
488
+ - `ai21labs/Jamba-tiny-reward-dev`, etc.
489
+ - ✅︎
490
+ - ✅︎
491
+* - `Qwen2ForSequenceClassification`
492
+ - Qwen2-based
493
+ - `jason9693/Qwen2.5-1.5B-apeach`, etc.
494
+ - ✅︎
495
+ - ✅︎
496
```
497
498
If your model is not in the above list, we will try to automatically convert the model using
@@ -504,31 +500,30 @@ If your model is not in the above list, we will try to automatically convert the
500
501
#### Sentence Pair Scoring (`--task score`)
502
507
-```{eval-rst}
508
-.. list-table::
509
- :widths: 25 25 50 5 5
510
- :header-rows: 1
511
-
512
- * - Architecture
513
- - Models
514
- - Example HF Models
515
- - :ref:`LoRA <lora-adapter>`
516
- - :ref:`PP <distributed-serving>`
517
- * - :code:`BertForSequenceClassification`
518
- - BERT-based
519
- - :code:`cross-encoder/ms-marco-MiniLM-L-6-v2`, etc.
520
- -
521
- -
522
- * - :code:`RobertaForSequenceClassification`
523
- - RoBERTa-based
524
- - :code:`cross-encoder/quora-roberta-base`, etc.
525
- -
526
- -
527
- * - :code:`XLMRobertaForSequenceClassification`
528
- - XLM-RoBERTa-based
529
- - :code:`BAAI/bge-reranker-v2-m3`, etc.
530
- -
531
- -
503
+```{list-table}
504
+:widths: 25 25 50 5 5
505
+:header-rows: 1
506
+
507
+* - Architecture
508
+ - Models
509
+ - Example HF Models
510
+ - [LoRA](#lora-adapter)
511
+ - [PP](#distributed-serving)
512
+* - `BertForSequenceClassification`
513
+ - BERT-based
514
+ - `cross-encoder/ms-marco-MiniLM-L-6-v2`, etc.
515
+ -
516
+ -
517
+* - `RobertaForSequenceClassification`
518
+ - RoBERTa-based
519
+ - `cross-encoder/quora-roberta-base`, etc.
520
+ -
521
+ -
522
+* - `XLMRobertaForSequenceClassification`
523
+ - XLM-RoBERTa-based
524
+ - `BAAI/bge-reranker-v2-m3`, etc.
525
+ -
526
+ -
527
```
528
529
(supported-mm-models)=
@@ -558,186 +553,182 @@ See [this page](#generative-models) for more information on how to use generativ
553
554
#### Text Generation (`--task generate`)
555
561
-```{eval-rst}
562
-.. list-table::
563
- :widths: 25 25 15 20 5 5 5
564
- :header-rows: 1
565
-
566
- * - Architecture
567
- - Models
568
- - Inputs
569
- - Example HF Models
570
- - :ref:`LoRA <lora-adapter>`
571
- - :ref:`PP <distributed-serving>`
572
- - V1
573
- * - :code:`AriaForConditionalGeneration`
574
- - Aria
575
- - T + I
576
- - :code:`rhymes-ai/Aria`
577
- -
578
- - ✅︎
579
- -
580
- * - :code:`Blip2ForConditionalGeneration`
581
- - BLIP-2
582
- - T + I\ :sup:`E`
583
- - :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
584
- -
585
- - ✅︎
586
- -
587
- * - :code:`ChameleonForConditionalGeneration`
588
- - Chameleon
589
- - T + I
590
- - :code:`facebook/chameleon-7b` etc.
591
- -
592
- - ✅︎
593
- -
594
- * - :code:`FuyuForCausalLM`
595
- - Fuyu
596
- - T + I
597
- - :code:`adept/fuyu-8b` etc.
598
- -
599
- - ✅︎
600
- -
601
- * - :code:`ChatGLMModel`
602
- - GLM-4V
603
- - T + I
604
- - :code:`THUDM/glm-4v-9b` etc.
605
- - ✅︎
606
- - ✅︎
607
- -
608
- * - :code:`H2OVLChatModel`
609
- - H2OVL
610
- - T + I\ :sup:`E+`
611
- - :code:`h2oai/h2ovl-mississippi-800m`, :code:`h2oai/h2ovl-mississippi-2b`, etc.
612
- -
613
- - ✅︎
614
- -
615
- * - :code:`Idefics3ForConditionalGeneration`
616
- - Idefics3
617
- - T + I
618
- - :code:`HuggingFaceM4/Idefics3-8B-Llama3` etc.
619
- - ✅︎
620
- -
621
- -
622
- * - :code:`InternVLChatModel`
623
- - InternVL 2.5, Mono-InternVL, InternVL 2.0
624
- - T + I\ :sup:`E+`
625
- - :code:`OpenGVLab/InternVL2_5-4B`, :code:`OpenGVLab/Mono-InternVL-2B`, :code:`OpenGVLab/InternVL2-4B`, etc.
626
- -
627
- - ✅︎
628
- - ✅︎
629
- * - :code:`LlavaForConditionalGeneration`
630
- - LLaVA-1.5
631
- - T + I\ :sup:`E+`
632
- - :code:`llava-hf/llava-1.5-7b-hf`, :code:`TIGER-Lab/Mantis-8B-siglip-llama3` (see note), etc.
633
- -
634
- - ✅︎
635
- - ✅︎
636
- * - :code:`LlavaNextForConditionalGeneration`
637
- - LLaVA-NeXT
638
- - T + I\ :sup:`E+`
639
- - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
640
- -
641
- - ✅︎
642
- -
643
- * - :code:`LlavaNextVideoForConditionalGeneration`
644
- - LLaVA-NeXT-Video
645
- - T + V
646
- - :code:`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
647
- -
648
- - ✅︎
649
- -
650
- * - :code:`LlavaOnevisionForConditionalGeneration`
651
- - LLaVA-Onevision
652
- - T + I\ :sup:`+` + V\ :sup:`+`
653
- - :code:`llava-hf/llava-onevision-qwen2-7b-ov-hf`, :code:`llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
654
- -
655
- - ✅︎
656
- -
657
- * - :code:`MiniCPMV`
658
- - MiniCPM-V
659
- - T + I\ :sup:`E+`
660
- - :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
661
- - ✅︎
662
- - ✅︎
663
- -
664
- * - :code:`MllamaForConditionalGeneration`
665
- - Llama 3.2
666
- - T + I\ :sup:`+`
667
- - :code:`meta-llama/Llama-3.2-90B-Vision-Instruct`, :code:`meta-llama/Llama-3.2-11B-Vision`, etc.
668
- -
669
- -
670
- -
671
- * - :code:`MolmoForCausalLM`
672
- - Molmo
673
- - T + I
674
- - :code:`allenai/Molmo-7B-D-0924`, :code:`allenai/Molmo-72B-0924`, etc.
675
- -
676
- - ✅︎
677
- - ✅︎
678
- * - :code:`NVLM_D_Model`
679
- - NVLM-D 1.0
680
- - T + I\ :sup:`E+`
681
- - :code:`nvidia/NVLM-D-72B`, etc.
682
- -
683
- - ✅︎
684
- - ✅︎
685
- * - :code:`PaliGemmaForConditionalGeneration`
686
- - PaliGemma, PaliGemma 2
687
- - T + I\ :sup:`E`
688
- - :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, :code:`google/paligemma2-3b-ft-docci-448`, etc.
689
- -
690
- - ✅︎
691
- -
692
- * - :code:`Phi3VForCausalLM`
693
- - Phi-3-Vision, Phi-3.5-Vision
694
- - T + I\ :sup:`E+`
695
- - :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
696
- -
697
- - ✅︎
698
- - ✅︎
699
- * - :code:`PixtralForConditionalGeneration`
700
- - Pixtral
701
- - T + I\ :sup:`+`
702
- - :code:`mistralai/Pixtral-12B-2409`, :code:`mistral-community/pixtral-12b` etc.
703
- -
704
- - ✅︎
705
- - ✅︎
706
- * - :code:`QWenLMHeadModel`
707
- - Qwen-VL
708
- - T + I\ :sup:`E+`
709
- - :code:`Qwen/Qwen-VL`, :code:`Qwen/Qwen-VL-Chat`, etc.
710
- - ✅︎
711
- - ✅︎
712
- -
713
- * - :code:`Qwen2AudioForConditionalGeneration`
714
- - Qwen2-Audio
715
- - T + A\ :sup:`+`
716
- - :code:`Qwen/Qwen2-Audio-7B-Instruct`
717
- -
718
- - ✅︎
719
- -
720
- * - :code:`Qwen2VLForConditionalGeneration`
721
- - Qwen2-VL
722
- - T + I\ :sup:`E+` + V\ :sup:`E+`
723
- - :code:`Qwen/QVQ-72B-Preview`, :code:`Qwen/Qwen2-VL-7B-Instruct`, :code:`Qwen/Qwen2-VL-72B-Instruct`, etc.
724
- - ✅︎
725
- - ✅︎
726
- -
727
- * - :code:`UltravoxModel`
728
- - Ultravox
729
- - T + A\ :sup:`E+`
730
- - :code:`fixie-ai/ultravox-v0_3`
731
- -
732
- - ✅︎
733
- -
556
+```{list-table}
557
+:widths: 25 25 15 20 5 5 5
558
+:header-rows: 1
559
+
560
+* - Architecture
561
+ - Models
562
+ - Inputs
563
+ - Example HF Models
564
+ - [LoRA](#lora-adapter)
565
+ - [PP](#distributed-serving)
566
+ - [V1](gh-issue:8779)
567
+* - `AriaForConditionalGeneration`
568
+ - Aria
569
+ - T + I
570
+ - `rhymes-ai/Aria`
571
+ -
572
+ - ✅︎
573
+ -
574
+* - `Blip2ForConditionalGeneration`
575
+ - BLIP-2
576
+ - T + I<sup>E</sup>
577
+ - `Salesforce/blip2-opt-2.7b`, `Salesforce/blip2-opt-6.7b`, etc.
578
+ -
579
+ - ✅︎
580
+ -
581
+* - `ChameleonForConditionalGeneration`
582
+ - Chameleon
583
+ - T + I
584
+ - `facebook/chameleon-7b` etc.
585
+ -
586
+ - ✅︎
587
+ -
588
+* - `FuyuForCausalLM`
589
+ - Fuyu
590
+ - T + I
591
+ - `adept/fuyu-8b` etc.
592
+ -
593
+ - ✅︎
594
+ -
595
+* - `ChatGLMModel`
596
+ - GLM-4V
597
+ - T + I
598
+ - `THUDM/glm-4v-9b` etc.
599
+ - ✅︎
600
+ - ✅︎
601
+ -
602
+* - `H2OVLChatModel`
603
+ - H2OVL
604
+ - T + I<sup>E+</sup>
605
+ - `h2oai/h2ovl-mississippi-800m`, `h2oai/h2ovl-mississippi-2b`, etc.
606
+ -
607
+ - ✅︎
608
+ -
609
+* - `Idefics3ForConditionalGeneration`
610
+ - Idefics3
611
+ - T + I
612
+ - `HuggingFaceM4/Idefics3-8B-Llama3` etc.
613
+ - ✅︎
614
+ -
615
+ -
616
+* - `InternVLChatModel`
617
+ - InternVL 2.5, Mono-InternVL, InternVL 2.0
618
+ - T + I<sup>E+</sup>
619
+ - `OpenGVLab/InternVL2_5-4B`, `OpenGVLab/Mono-InternVL-2B`, `OpenGVLab/InternVL2-4B`, etc.
620
+ -
621
+ - ✅︎
622
+ - ✅︎
623
+* - `LlavaForConditionalGeneration`
624
+ - LLaVA-1.5
625
+ - T + I<sup>E+</sup>
626
+ - `llava-hf/llava-1.5-7b-hf`, `TIGER-Lab/Mantis-8B-siglip-llama3` (see note), etc.
627
+ -
628
+ - ✅︎
629
+ - ✅︎
630
+* - `LlavaNextForConditionalGeneration`
631
+ - LLaVA-NeXT
632
+ - T + I<sup>E+</sup>
633
+ - `llava-hf/llava-v1.6-mistral-7b-hf`, `llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
634
+ -
635
+ - ✅︎
636
+ -
637
+* - `LlavaNextVideoForConditionalGeneration`
638
+ - LLaVA-NeXT-Video
639
+ - T + V
640
+ - `llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
641
+ -
642
+ - ✅︎
643
+ -
644
+* - `LlavaOnevisionForConditionalGeneration`
645
+ - LLaVA-Onevision
646
+ - T + I<sup>+</sup> + V<sup>+</sup>
647
+ - `llava-hf/llava-onevision-qwen2-7b-ov-hf`, `llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
648
+ -
649
+ - ✅︎
650
+ -
651
+* - `MiniCPMV`
652
+ - MiniCPM-V
653
+ - T + I<sup>E+</sup>
654
+ - `openbmb/MiniCPM-V-2` (see note), `openbmb/MiniCPM-Llama3-V-2_5`, `openbmb/MiniCPM-V-2_6`, etc.
655
+ - ✅︎
656
+ - ✅︎
657
+ -
658
+* - `MllamaForConditionalGeneration`
659
+ - Llama 3.2
660
+ - T + I<sup>+</sup>
661
+ - `meta-llama/Llama-3.2-90B-Vision-Instruct`, `meta-llama/Llama-3.2-11B-Vision`, etc.
662
+ -
663
+ -
664
+ -
665
+* - `MolmoForCausalLM`
666
+ - Molmo
667
+ - T + I
668
+ - `allenai/Molmo-7B-D-0924`, `allenai/Molmo-72B-0924`, etc.
669
+ -
670
+ - ✅︎
671
+ - ✅︎
672
+* - `NVLM_D_Model`
673
+ - NVLM-D 1.0
674
+ - T + I<sup>E+</sup>
675
+ - `nvidia/NVLM-D-72B`, etc.
676
+ -
677
+ - ✅︎
678
+ - ✅︎
679
+* - `PaliGemmaForConditionalGeneration`
680
+ - PaliGemma, PaliGemma 2
681
+ - T + I<sup>E</sup>
682
+ - `google/paligemma-3b-pt-224`, `google/paligemma-3b-mix-224`, `google/paligemma2-3b-ft-docci-448`, etc.
683
+ -
684
+ - ✅︎
685
+ -
686
+* - `Phi3VForCausalLM`
687
+ - Phi-3-Vision, Phi-3.5-Vision
688
+ - T + I<sup>E+</sup>
689
+ - `microsoft/Phi-3-vision-128k-instruct`, `microsoft/Phi-3.5-vision-instruct` etc.
690
+ -
691
+ - ✅︎
692
+ - ✅︎
693
+* - `PixtralForConditionalGeneration`
694
+ - Pixtral
695
+ - T + I<sup>+</sup>
696
+ - `mistralai/Pixtral-12B-2409`, `mistral-community/pixtral-12b` etc.
697
+ -
698
+ - ✅︎
699
+ - ✅︎
700
+* - `QWenLMHeadModel`
701
+ - Qwen-VL
702
+ - T + I<sup>E+</sup>
703
+ - `Qwen/Qwen-VL`, `Qwen/Qwen-VL-Chat`, etc.
704
+ - ✅︎
705
+ - ✅︎
706
+ -
707
+* - `Qwen2AudioForConditionalGeneration`
708
+ - Qwen2-Audio
709
+ - T + A<sup>+</sup>
710
+ - `Qwen/Qwen2-Audio-7B-Instruct`
711
+ -
712
+ - ✅︎
713
+ -
714
+* - `Qwen2VLForConditionalGeneration`
715
+ - Qwen2-VL
716
+ - T + I<sup>E+</sup> + V<sup>E+</sup>
717
+ - `Qwen/QVQ-72B-Preview`, `Qwen/Qwen2-VL-7B-Instruct`, `Qwen/Qwen2-VL-72B-Instruct`, etc.
718
+ - ✅︎
719
+ - ✅︎
720
+ -
721
+* - `UltravoxModel`
722
+ - Ultravox
723
+ - T + A<sup>E+</sup>
724
+ - `fixie-ai/ultravox-v0_3`
725
+ -
726
+ - ✅︎
727
+ -
728
```
729
736
-```{eval-rst}
737
-:sup:`E` Pre-computed embeddings can be inputted for this modality.
738
-
739
-:sup:`+` Multiple items can be inputted per text prompt for this modality.
740
-```
730
+<sup>E</sup> Pre-computed embeddings can be inputted for this modality.
731
+<sup>+</sup> Multiple items can be inputted per text prompt for this modality.
732
733
````{important}
734
To enable multiple multi-modal items per text prompt, you have to set {code}`limit_mm_per_prompt` (offline inference)
@@ -787,38 +778,37 @@ To get the best results, you should use pooling models that are specifically tra
778
779
The following table lists those that are tested in vLLM.
780
790
-```{eval-rst}
791
-.. list-table::
792
- :widths: 25 25 15 25 5 5
793
- :header-rows: 1
794
-
795
- * - Architecture
796
- - Models
797
- - Inputs
798
- - Example HF Models
799
- - :ref:`LoRA <lora-adapter>`
800
- - :ref:`PP <distributed-serving>`
801
- * - :code:`LlavaNextForConditionalGeneration`
802
- - LLaVA-NeXT-based
803
- - T / I
804
- - :code:`royokong/e5-v`
805
- -
806
- - ✅︎
807
- * - :code:`Phi3VForCausalLM`
808
- - Phi-3-Vision-based
809
- - T + I
810
- - :code:`TIGER-Lab/VLM2Vec-Full`
811
- - 🚧
812
- - ✅︎
813
- * - :code:`Qwen2VLForConditionalGeneration`
814
- - Qwen2-VL-based
815
- - T + I
816
- - :code:`MrLight/dse-qwen2-2b-mrl-v1`
817
- -
818
- - ✅︎
781
+```{list-table}
782
+:widths: 25 25 15 25 5 5
783
+:header-rows: 1
784
+
785
+* - Architecture
786
+ - Models
787
+ - Inputs
788
+ - Example HF Models
789
+ - [LoRA](#lora-adapter)
790
+ - [PP](#distributed-serving)
791
+* - `LlavaNextForConditionalGeneration`
792
+ - LLaVA-NeXT-based
793
+ - T / I
794
+ - `royokong/e5-v`
795
+ -
796
+ - ✅︎
797
+* - `Phi3VForCausalLM`
798
+ - Phi-3-Vision-based
799
+ - T + I
800
+ - `TIGER-Lab/VLM2Vec-Full`
801
+ - 🚧
802
+ - ✅︎
803
+* - `Qwen2VLForConditionalGeneration`
804
+ - Qwen2-VL-based
805
+ - T + I
806
+ - `MrLight/dse-qwen2-2b-mrl-v1`
807
+ -
808
+ - ✅︎
809
```
810
821
-______________________________________________________________________
811
+_________________
812
813
# Model Support Policy
814
docs/source/quantization/supported_hardware.md
+113
-114
@@ -4,121 +4,120 @@
4
5
The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:
6
7
-```{eval-rst}
8
-.. list-table::
9
- :header-rows: 1
10
- :widths: 20 8 8 8 8 8 8 8 8 8 8
7
+```{list-table}
8
+:header-rows: 1
9
+:widths: 20 8 8 8 8 8 8 8 8 8 8
10
12
- * - Implementation
13
- - Volta
14
- - Turing
15
- - Ampere
16
- - Ada
17
- - Hopper
18
- - AMD GPU
19
- - Intel GPU
20
- - x86 CPU
21
- - AWS Inferentia
22
- - Google TPU
23
- * - AWQ
24
- - ✗
25
- - ✅︎
26
- - ✅︎
27
- - ✅︎
28
- - ✅︎
29
- - ✗
30
- - ✅︎
31
- - ✅︎
32
- - ✗
33
- - ✗
34
- * - GPTQ
35
- - ✅︎
36
- - ✅︎
37
- - ✅︎
38
- - ✅︎
39
- - ✅︎
40
- - ✗
41
- - ✅︎
42
- - ✅︎
43
- - ✗
44
- - ✗
45
- * - Marlin (GPTQ/AWQ/FP8)
46
- - ✗
47
- - ✗
48
- - ✅︎
49
- - ✅︎
50
- - ✅︎
51
- - ✗
52
- - ✗
53
- - ✗
54
- - ✗
55
- - ✗
56
- * - INT8 (W8A8)
57
- - ✗
58
- - ✅︎
59
- - ✅︎
60
- - ✅︎
61
- - ✅︎
62
- - ✗
63
- - ✗
64
- - ✅︎
65
- - ✗
66
- - ✗
67
- * - FP8 (W8A8)
68
- - ✗
69
- - ✗
70
- - ✗
71
- - ✅︎
72
- - ✅︎
73
- - ✅︎
74
- - ✗
75
- - ✗
76
- - ✗
77
- - ✗
78
- * - AQLM
79
- - ✅︎
80
- - ✅︎
81
- - ✅︎
82
- - ✅︎
83
- - ✅︎
84
- - ✗
85
- - ✗
86
- - ✗
87
- - ✗
88
- - ✗
89
- * - bitsandbytes
90
- - ✅︎
91
- - ✅︎
92
- - ✅︎
93
- - ✅︎
94
- - ✅︎
95
- - ✗
96
- - ✗
97
- - ✗
98
- - ✗
99
- - ✗
100
- * - DeepSpeedFP
101
- - ✅︎
102
- - ✅︎
103
- - ✅︎
104
- - ✅︎
105
- - ✅︎
106
- - ✗
107
- - ✗
108
- - ✗
109
- - ✗
110
- - ✗
111
- * - GGUF
112
- - ✅︎
113
- - ✅︎
114
- - ✅︎
115
- - ✅︎
116
- - ✅︎
117
- - ✗
118
- - ✗
119
- - ✗
120
- - ✗
121
- - ✗
11
+* - Implementation
12
+ - Volta
13
+ - Turing
14
+ - Ampere
15
+ - Ada
16
+ - Hopper
17
+ - AMD GPU
18
+ - Intel GPU
19
+ - x86 CPU
20
+ - AWS Inferentia
21
+ - Google TPU
22
+* - AWQ
23
+ - ✗
24
+ - ✅︎
25
+ - ✅︎
26
+ - ✅︎
27
+ - ✅︎
28
+ - ✗
29
+ - ✅︎
30
+ - ✅︎
31
+ - ✗
32
+ - ✗
33
+* - GPTQ
34
+ - ✅︎
35
+ - ✅︎
36
+ - ✅︎
37
+ - ✅︎
38
+ - ✅︎
39
+ - ✗
40
+ - ✅︎
41
+ - ✅︎
42
+ - ✗
43
+ - ✗
44
+* - Marlin (GPTQ/AWQ/FP8)
45
+ - ✗
46
+ - ✗
47
+ - ✅︎
48
+ - ✅︎
49
+ - ✅︎
50
+ - ✗
51
+ - ✗
52
+ - ✗
53
+ - ✗
54
+ - ✗
55
+* - INT8 (W8A8)
56
+ - ✗
57
+ - ✅︎
58
+ - ✅︎
59
+ - ✅︎
60
+ - ✅︎
61
+ - ✗
62
+ - ✗
63
+ - ✅︎
64
+ - ✗
65
+ - ✗
66
+* - FP8 (W8A8)
67
+ - ✗
68
+ - ✗
69
+ - ✗
70
+ - ✅︎
71
+ - ✅︎
72
+ - ✅︎
73
+ - ✗
74
+ - ✗
75
+ - ✗
76
+ - ✗
77
+* - AQLM
78
+ - ✅︎
79
+ - ✅︎
80
+ - ✅︎
81
+ - ✅︎
82
+ - ✅︎
83
+ - ✗
84
+ - ✗
85
+ - ✗
86
+ - ✗
87
+ - ✗
88
+* - bitsandbytes
89
+ - ✅︎
90
+ - ✅︎
91
+ - ✅︎
92
+ - ✅︎
93
+ - ✅︎
94
+ - ✗
95
+ - ✗
96
+ - ✗
97
+ - ✗
98
+ - ✗
99
+* - DeepSpeedFP
100
+ - ✅︎
101
+ - ✅︎
102
+ - ✅︎
103
+ - ✅︎
104
+ - ✅︎
105
+ - ✗
106
+ - ✗
107
+ - ✗
108
+ - ✗
109
+ - ✗
110
+* - GGUF
111
+ - ✅︎
112
+ - ✅︎
113
+ - ✅︎
114
+ - ✅︎
115
+ - ✅︎
116
+ - ✗
117
+ - ✗
118
+ - ✗
119
+ - ✗
120
+ - ✗
121
```
122
123
## Notes:
docs/source/serving/deploying_with_helm.md
+204
-205
@@ -43,209 +43,208 @@ chart **including persistent volumes** and deletes the release.
43
44
## Values
45
46
-```{eval-rst}
47
-.. list-table:: Values
48
- :widths: 25 25 25 25
49
- :header-rows: 1
50
-
51
- * - Key
52
- - Type
53
- - Default
54
- - Description
55
- * - autoscaling
56
- - object
57
- - {"enabled":false,"maxReplicas":100,"minReplicas":1,"targetCPUUtilizationPercentage":80}
58
- - Autoscaling configuration
59
- * - autoscaling.enabled
60
- - bool
61
- - false
62
- - Enable autoscaling
63
- * - autoscaling.maxReplicas
64
- - int
65
- - 100
66
- - Maximum replicas
67
- * - autoscaling.minReplicas
68
- - int
69
- - 1
70
- - Minimum replicas
71
- * - autoscaling.targetCPUUtilizationPercentage
72
- - int
73
- - 80
74
- - Target CPU utilization for autoscaling
75
- * - configs
76
- - object
77
- - {}
78
- - Configmap
79
- * - containerPort
80
- - int
81
- - 8000
82
- - Container port
83
- * - customObjects
84
- - list
85
- - []
86
- - Custom Objects configuration
87
- * - deploymentStrategy
88
- - object
89
- - {}
90
- - Deployment strategy configuration
91
- * - externalConfigs
92
- - list
93
- - []
94
- - External configuration
95
- * - extraContainers
96
- - list
97
- - []
98
- - Additional containers configuration
99
- * - extraInit
100
- - object
101
- - {"pvcStorage":"1Gi","s3modelpath":"relative_s3_model_path/opt-125m", "awsEc2MetadataDisabled": true}
102
- - Additional configuration for the init container
103
- * - extraInit.pvcStorage
104
- - string
105
- - "50Gi"
106
- - Storage size of the s3
107
- * - extraInit.s3modelpath
108
- - string
109
- - "relative_s3_model_path/opt-125m"
110
- - Path of the model on the s3 which hosts model weights and config files
111
- * - extraInit.awsEc2MetadataDisabled
112
- - boolean
113
- - true
114
- - Disables the use of the Amazon EC2 instance metadata service
115
- * - extraPorts
116
- - list
117
- - []
118
- - Additional ports configuration
119
- * - gpuModels
120
- - list
121
- - ["TYPE_GPU_USED"]
122
- - Type of gpu used
123
- * - image
124
- - object
125
- - {"command":["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"],"repository":"vllm/vllm-openai","tag":"latest"}
126
- - Image configuration
127
- * - image.command
128
- - list
129
- - ["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"]
130
- - Container launch command
131
- * - image.repository
132
- - string
133
- - "vllm/vllm-openai"
134
- - Image repository
135
- * - image.tag
136
- - string
137
- - "latest"
138
- - Image tag
139
- * - livenessProbe
140
- - object
141
- - {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":15,"periodSeconds":10}
142
- - Liveness probe configuration
143
- * - livenessProbe.failureThreshold
144
- - int
145
- - 3
146
- - Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not alive
147
- * - livenessProbe.httpGet
148
- - object
149
- - {"path":"/health","port":8000}
150
- - Configuration of the Kubelet http request on the server
151
- * - livenessProbe.httpGet.path
152
- - string
153
- - "/health"
154
- - Path to access on the HTTP server
155
- * - livenessProbe.httpGet.port
156
- - int
157
- - 8000
158
- - Name or number of the port to access on the container, on which the server is listening
159
- * - livenessProbe.initialDelaySeconds
160
- - int
161
- - 15
162
- - Number of seconds after the container has started before liveness probe is initiated
163
- * - livenessProbe.periodSeconds
164
- - int
165
- - 10
166
- - How often (in seconds) to perform the liveness probe
167
- * - maxUnavailablePodDisruptionBudget
168
- - string
169
- - ""
170
- - Disruption Budget Configuration
171
- * - readinessProbe
172
- - object
173
- - {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":5,"periodSeconds":5}
174
- - Readiness probe configuration
175
- * - readinessProbe.failureThreshold
176
- - int
177
- - 3
178
- - Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not ready
179
- * - readinessProbe.httpGet
180
- - object
181
- - {"path":"/health","port":8000}
182
- - Configuration of the Kubelet http request on the server
183
- * - readinessProbe.httpGet.path
184
- - string
185
- - "/health"
186
- - Path to access on the HTTP server
187
- * - readinessProbe.httpGet.port
188
- - int
189
- - 8000
190
- - Name or number of the port to access on the container, on which the server is listening
191
- * - readinessProbe.initialDelaySeconds
192
- - int
193
- - 5
194
- - Number of seconds after the container has started before readiness probe is initiated
195
- * - readinessProbe.periodSeconds
196
- - int
197
- - 5
198
- - How often (in seconds) to perform the readiness probe
199
- * - replicaCount
200
- - int
201
- - 1
202
- - Number of replicas
203
- * - resources
204
- - object
205
- - {"limits":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1},"requests":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1}}
206
- - Resource configuration
207
- * - resources.limits."nvidia.com/gpu"
208
- - int
209
- - 1
210
- - Number of gpus used
211
- * - resources.limits.cpu
212
- - int
213
- - 4
214
- - Number of CPUs
215
- * - resources.limits.memory
216
- - string
217
- - "16Gi"
218
- - CPU memory configuration
219
- * - resources.requests."nvidia.com/gpu"
220
- - int
221
- - 1
222
- - Number of gpus used
223
- * - resources.requests.cpu
224
- - int
225
- - 4
226
- - Number of CPUs
227
- * - resources.requests.memory
228
- - string
229
- - "16Gi"
230
- - CPU memory configuration
231
- * - secrets
232
- - object
233
- - {}
234
- - Secrets configuration
235
- * - serviceName
236
- - string
237
- -
238
- - Service name
239
- * - servicePort
240
- - int
241
- - 80
242
- - Service port
243
- * - labels.environment
244
- - string
245
- - test
246
- - Environment name
247
- * - labels.release
248
- - string
249
- - test
250
- - Release name
46
+```{list-table}
47
+:widths: 25 25 25 25
48
+:header-rows: 1
49
+
50
+* - Key
51
+ - Type
52
+ - Default
53
+ - Description
54
+* - autoscaling
55
+ - object
56
+ - {"enabled":false,"maxReplicas":100,"minReplicas":1,"targetCPUUtilizationPercentage":80}
57
+ - Autoscaling configuration
58
+* - autoscaling.enabled
59
+ - bool
60
+ - false
61
+ - Enable autoscaling
62
+* - autoscaling.maxReplicas
63
+ - int
64
+ - 100
65
+ - Maximum replicas
66
+* - autoscaling.minReplicas
67
+ - int
68
+ - 1
69
+ - Minimum replicas
70
+* - autoscaling.targetCPUUtilizationPercentage
71
+ - int
72
+ - 80
73
+ - Target CPU utilization for autoscaling
74
+* - configs
75
+ - object
76
+ - {}
77
+ - Configmap
78
+* - containerPort
79
+ - int
80
+ - 8000
81
+ - Container port
82
+* - customObjects
83
+ - list
84
+ - []
85
+ - Custom Objects configuration
86
+* - deploymentStrategy
87
+ - object
88
+ - {}
89
+ - Deployment strategy configuration
90
+* - externalConfigs
91
+ - list
92
+ - []
93
+ - External configuration
94
+* - extraContainers
95
+ - list
96
+ - []
97
+ - Additional containers configuration
98
+* - extraInit
99
+ - object
100
+ - {"pvcStorage":"1Gi","s3modelpath":"relative_s3_model_path/opt-125m", "awsEc2MetadataDisabled": true}
101
+ - Additional configuration for the init container
102
+* - extraInit.pvcStorage
103
+ - string
104
+ - "50Gi"
105
+ - Storage size of the s3
106
+* - extraInit.s3modelpath
107
+ - string
108
+ - "relative_s3_model_path/opt-125m"
109
+ - Path of the model on the s3 which hosts model weights and config files
110
+* - extraInit.awsEc2MetadataDisabled
111
+ - boolean
112
+ - true
113
+ - Disables the use of the Amazon EC2 instance metadata service
114
+* - extraPorts
115
+ - list
116
+ - []
117
+ - Additional ports configuration
118
+* - gpuModels
119
+ - list
120
+ - ["TYPE_GPU_USED"]
121
+ - Type of gpu used
122
+* - image
123
+ - object
124
+ - {"command":["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"],"repository":"vllm/vllm-openai","tag":"latest"}
125
+ - Image configuration
126
+* - image.command
127
+ - list
128
+ - ["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"]
129
+ - Container launch command
130
+* - image.repository
131
+ - string
132
+ - "vllm/vllm-openai"
133
+ - Image repository
134
+* - image.tag
135
+ - string
136
+ - "latest"
137
+ - Image tag
138
+* - livenessProbe
139
+ - object
140
+ - {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":15,"periodSeconds":10}
141
+ - Liveness probe configuration
142
+* - livenessProbe.failureThreshold
143
+ - int
144
+ - 3
145
+ - Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not alive
146
+* - livenessProbe.httpGet
147
+ - object
148
+ - {"path":"/health","port":8000}
149
+ - Configuration of the Kubelet http request on the server
150
+* - livenessProbe.httpGet.path
151
+ - string
152
+ - "/health"
153
+ - Path to access on the HTTP server
154
+* - livenessProbe.httpGet.port
155
+ - int
156
+ - 8000
157
+ - Name or number of the port to access on the container, on which the server is listening
158
+* - livenessProbe.initialDelaySeconds
159
+ - int
160
+ - 15
161
+ - Number of seconds after the container has started before liveness probe is initiated
162
+* - livenessProbe.periodSeconds
163
+ - int
164
+ - 10
165
+ - How often (in seconds) to perform the liveness probe
166
+* - maxUnavailablePodDisruptionBudget
167
+ - string
168
+ - ""
169
+ - Disruption Budget Configuration
170
+* - readinessProbe
171
+ - object
172
+ - {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":5,"periodSeconds":5}
173
+ - Readiness probe configuration
174
+* - readinessProbe.failureThreshold
175
+ - int
176
+ - 3
177
+ - Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not ready
178
+* - readinessProbe.httpGet
179
+ - object
180
+ - {"path":"/health","port":8000}
181
+ - Configuration of the Kubelet http request on the server
182
+* - readinessProbe.httpGet.path
183
+ - string
184
+ - "/health"
185
+ - Path to access on the HTTP server
186
+* - readinessProbe.httpGet.port
187
+ - int
188
+ - 8000
189
+ - Name or number of the port to access on the container, on which the server is listening
190
+* - readinessProbe.initialDelaySeconds
191
+ - int
192
+ - 5
193
+ - Number of seconds after the container has started before readiness probe is initiated
194
+* - readinessProbe.periodSeconds
195
+ - int
196
+ - 5
197
+ - How often (in seconds) to perform the readiness probe
198
+* - replicaCount
199
+ - int
200
+ - 1
201
+ - Number of replicas
202
+* - resources
203
+ - object
204
+ - {"limits":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1},"requests":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1}}
205
+ - Resource configuration
206
+* - resources.limits."nvidia.com/gpu"
207
+ - int
208
+ - 1
209
+ - Number of gpus used
210
+* - resources.limits.cpu
211
+ - int
212
+ - 4
213
+ - Number of CPUs
214
+* - resources.limits.memory
215
+ - string
216
+ - "16Gi"
217
+ - CPU memory configuration
218
+* - resources.requests."nvidia.com/gpu"
219
+ - int
220
+ - 1
221
+ - Number of gpus used
222
+* - resources.requests.cpu
223
+ - int
224
+ - 4
225
+ - Number of CPUs
226
+* - resources.requests.memory
227
+ - string
228
+ - "16Gi"
229
+ - CPU memory configuration
230
+* - secrets
231
+ - object
232
+ - {}
233
+ - Secrets configuration
234
+* - serviceName
235
+ - string
236
+ -
237
+ - Service name
238
+* - servicePort
239
+ - int
240
+ - 80
241
+ - Service port
242
+* - labels.environment
243
+ - string
244
+ - test
245
+ - Environment name
246
+* - labels.release
247
+ - string
248
+ - test
249
+ - Release name
250
```