A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 01:45 UTC
Support to serve vLLM on Kubernetes with LWS (#4829)
Signed-off-by: kerthcet <kerthcet@gmail.com>
Kante Yin committed
May 17, 2024 at 07:37 UTC
8e7fb5d43ae74e0a75a7da940a63c7891208d268
2 files changed
+13
docs/source/serving/deploying_with_lws.rst
new
+12
@@ -0,0 +1,12 @@
1
+.. _deploying_with_lws:
2
+
3
+Deploying with LWS
4
+============================
5
+
6
+LeaderWorkerSet (LWS) is a Kubernetes API that aims to address common deployment patterns of AI/ML inference workloads.
7
+A major use case is for multi-host/multi-node distributed inference.
8
+
9
+vLLM can be deployed with `LWS <https://github.com/kubernetes-sigs/lws>`_ on Kubernetes for distributed model serving.
10
+
11
+Please see `this guide <https://github.com/kubernetes-sigs/lws/tree/main/docs/examples/vllm>`_ for more details on
12
+deploying vLLM on Kubernetes using LWS.
docs/source/serving/integrations.rst
+1
@@ -8,4 +8,5 @@ Integrations
8
deploying_with_kserve
9
deploying_with_triton
10
deploying_with_bentoml
11
+ deploying_with_lws
12
serving_with_langchain