@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 49d849b3ab
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

docs: Add tutorial on deploying vLLM model with KServe (#2586)

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

Yuan Tang committed Mar 1, 2024 at 14:04 UTC 49d849b3ab7aa6ae493ccde1d85d226833f73fbb
2 files changed +9
docs/source/index.rst
+1
@@ -70,6 +70,7 @@ Documentation
70
71 serving/distributed_serving
72 serving/run_on_sky
73 + serving/deploying_with_kserve
74 serving/deploying_with_triton
75 serving/deploying_with_docker
76 serving/serving_with_langchain
docs/source/serving/deploying_with_kserve.rst new
+8
@@ -0,0 +1,8 @@
1 +.. _deploying_with_kserve:
2 +
3 +Deploying with KServe
4 +============================
5 +
6 +vLLM can be deployed with `KServe <https://github.com/kserve/kserve>`_ on Kubernetes for highly scalable distributed model serving.
7 +
8 +Please see `this guide <https://kserve.github.io/website/latest/modelserving/v1beta1/llm/vllm/>`_ for more details on using vLLM with KServe.