@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 36f5303578
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 03:15 UTC

[Docs] Add Modal to deployment frameworks (#11907)

Charles Frye committed Jan 9, 2025 at 15:26 UTC 36f5303578397d122693a19007be38ba2f02bcbc
3 files changed +9 -1
docs/source/deployment/frameworks/bentoml.md
+1 -1
@@ -2,6 +2,6 @@
2
3 # BentoML
4
5 -[BentoML](https://github.com/bentoml/BentoML) allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-complicant image and deploy it on Kubernetes.
5 +[BentoML](https://github.com/bentoml/BentoML) allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-compliant image and deploy it on Kubernetes.
6
7 For details, see the tutorial [vLLM inference in the BentoML documentation](https://docs.bentoml.com/en/latest/use-cases/large-language-models/vllm.html).
docs/source/deployment/frameworks/index.md
+1
@@ -8,6 +8,7 @@ cerebrium
8 dstack
9 helm
10 lws
11 +modal
12 skypilot
13 triton
14 ```
docs/source/deployment/frameworks/modal.md new
+7
@@ -0,0 +1,7 @@
1 +(deployment-modal)=
2 +
3 +# Modal
4 +
5 +vLLM can be run on cloud GPUs with [Modal](https://modal.com), a serverless computing platform designed for fast auto-scaling.
6 +
7 +For details on how to deploy vLLM on Modal, see [this tutorial in the Modal documentation](https://modal.com/docs/examples/vllm_inference).