@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / b0925b3878
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

docs: Add BentoML deployment doc (#3336)

Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>

Sherlock Xu committed Mar 13, 2024 at 01:34 UTC b0925b38789bb3b20dcc39e229fcfe12a311e487
2 files changed +9
docs/source/index.rst
+1
@@ -73,6 +73,7 @@ Documentation
73 serving/run_on_sky
74 serving/deploying_with_kserve
75 serving/deploying_with_triton
76 + serving/deploying_with_bentoml
77 serving/deploying_with_docker
78 serving/serving_with_langchain
79 serving/metrics
docs/source/serving/deploying_with_bentoml.rst new
+8
@@ -0,0 +1,8 @@
1 +.. _deploying_with_bentoml:
2 +
3 +Deploying with BentoML
4 +======================
5 +
6 +`BentoML <https://github.com/bentoml/BentoML>`_ allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-complicant image and deploy it on Kubernetes.
7 +
8 +For details, see the tutorial `vLLM inference in the BentoML documentation <https://docs.bentoml.com/en/latest/use-cases/large-language-models/vllm.html>`_.
\ No newline at end of file