@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 8e7fb5d43a
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 01:45 UTC

Support to serve vLLM on Kubernetes with LWS (#4829)

Signed-off-by: kerthcet <kerthcet@gmail.com>

Kante Yin committed May 17, 2024 at 07:37 UTC 8e7fb5d43ae74e0a75a7da940a63c7891208d268
2 files changed +13
docs/source/serving/deploying_with_lws.rst new
+12
@@ -0,0 +1,12 @@
1 +.. _deploying_with_lws:
2 +
3 +Deploying with LWS
4 +============================
5 +
6 +LeaderWorkerSet (LWS) is a Kubernetes API that aims to address common deployment patterns of AI/ML inference workloads.
7 +A major use case is for multi-host/multi-node distributed inference.
8 +
9 +vLLM can be deployed with `LWS <https://github.com/kubernetes-sigs/lws>`_ on Kubernetes for distributed model serving.
10 +
11 +Please see `this guide <https://github.com/kubernetes-sigs/lws/tree/main/docs/examples/vllm>`_ for more details on
12 +deploying vLLM on Kubernetes using LWS.
docs/source/serving/integrations.rst
+1
@@ -8,4 +8,5 @@ Integrations
8 deploying_with_kserve
9 deploying_with_triton
10 deploying_with_bentoml
11 + deploying_with_lws
12 serving_with_langchain