A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 16, 03:15 UTC
[Docs] Add Modal to deployment frameworks (#11907)
Charles Frye committed
Jan 9, 2025 at 15:26 UTC
36f5303578397d122693a19007be38ba2f02bcbc
3 files changed
+9
-1
docs/source/deployment/frameworks/bentoml.md
+1
-1
@@ -2,6 +2,6 @@
2
3
# BentoML
4
5
-[BentoML](https://github.com/bentoml/BentoML) allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-complicant image and deploy it on Kubernetes.
5
+[BentoML](https://github.com/bentoml/BentoML) allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-compliant image and deploy it on Kubernetes.
6
7
For details, see the tutorial [vLLM inference in the BentoML documentation](https://docs.bentoml.com/en/latest/use-cases/large-language-models/vllm.html).
docs/source/deployment/frameworks/index.md
+1
@@ -8,6 +8,7 @@ cerebrium
8
dstack
9
helm
10
lws
11
+modal
12
skypilot
13
triton
14
```
docs/source/deployment/frameworks/modal.md
new
+7
@@ -0,0 +1,7 @@
1
+(deployment-modal)=
2
+
3
+# Modal
4
+
5
+vLLM can be run on cloud GPUs with [Modal](https://modal.com), a serverless computing platform designed for fast auto-scaling.
6
+
7
+For details on how to deploy vLLM on Modal, see [this tutorial in the Modal documentation](https://modal.com/docs/examples/vllm_inference).