A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 04:15 UTC
docs: Add tutorial on deploying vLLM model with KServe (#2586)
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
Yuan Tang committed
Mar 1, 2024 at 14:04 UTC
49d849b3ab7aa6ae493ccde1d85d226833f73fbb
2 files changed
+9
docs/source/index.rst
+1
@@ -70,6 +70,7 @@ Documentation
70
71
serving/distributed_serving
72
serving/run_on_sky
73
+ serving/deploying_with_kserve
74
serving/deploying_with_triton
75
serving/deploying_with_docker
76
serving/serving_with_langchain
docs/source/serving/deploying_with_kserve.rst
new
+8
@@ -0,0 +1,8 @@
1
+.. _deploying_with_kserve:
2
+
3
+Deploying with KServe
4
+============================
5
+
6
+vLLM can be deployed with `KServe <https://github.com/kserve/kserve>`_ on Kubernetes for highly scalable distributed model serving.
7
+
8
+Please see `this guide <https://kserve.github.io/website/latest/modelserving/v1beta1/llm/vllm/>`_ for more details on using vLLM with KServe.