A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 23:30 UTC
[Bugfix][Docs] Fix offline Whisper (#13274)
Nicolò Lucchesi committed
Feb 15, 2025 at 06:32 UTC
579d7a63b2d87d62957a7e2dfe136f0674251657
2 files changed
+21
-1
docs/source/models/supported_models.md
+20
@@ -938,6 +938,26 @@ The following table lists those that are tested in vLLM.
938
* ✅︎
939
:::
940
941
+#### Transcription (`--task transcription`)
942
+
943
+Speech2Text models trained specifically for Automatic Speech Recognition.
944
+
945
+:::{list-table}
946
+:widths: 25 25 25 5 5
947
+:header-rows: 1
948
+
949
+- * Architecture
950
+ * Models
951
+ * Example HF Models
952
+ * [LoRA](#lora-adapter)
953
+ * [PP](#distributed-serving)
954
+- * `Whisper`
955
+ * Whisper-based
956
+ * `openai/whisper-large-v3-turbo`
957
+ * 🚧
958
+ * 🚧
959
+:::
960
+
961
_________________
962
963
## Model Support Policy
vllm/entrypoints/llm.py
+1
-1
@@ -421,7 +421,7 @@ class LLM:
421
instead pass them via the ``inputs`` parameter.
422
"""
423
runner_type = self.llm_engine.model_config.runner_type
424
- if runner_type != "generate":
424
+ if runner_type not in ["generate", "transcription"]:
425
messages = [
426
"LLM.generate() is only supported for (conditional) generation "
427
"models (XForCausalLM, XForConditionalGeneration).",