@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 7406274041
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 16, 05:45 UTC

[Doc] add KubeAI to serving integrations (#10837)

Signed-off-by: Sam Stoelinga <sammiestoel@gmail.com>

Sam Stoelinga committed Dec 6, 2024 at 09:03 UTC 74062740416db8572627dda1f87925268ba2f1d3
2 files changed +18
docs/source/serving/deploying_with_kubeai.rst new
+17
@@ -0,0 +1,17 @@
1 +.. _deploying_with_kubeai:
2 +
3 +Deploying with KubeAI
4 +=====================
5 +
6 +`KubeAI <https://github.com/substratusai/kubeai>`_ is a Kubernetes operator that enables you to deploy and manage AI models on Kubernetes. It provides a simple and scalable way to deploy vLLM in production. Functionality such as scale-from-zero, load based autoscaling, model caching, and much more is provided out of the box with zero external dependencies.
7 +
8 +
9 +Please see the Installation Guides for environment specific instructions:
10 +
11 +* `Any Kubernetes Cluster <https://www.kubeai.org/installation/any/>`_
12 +* `EKS <https://www.kubeai.org/installation/eks/>`_
13 +* `GKE <https://www.kubeai.org/installation/gke/>`_
14 +
15 +Once you have KubeAI installed, you can
16 +`configure text generation models <https://www.kubeai.org/how-to/configure-text-generation-models/>`_
17 +using vLLM.
\ No newline at end of file
docs/source/serving/integrations.rst
+1
@@ -6,6 +6,7 @@ Integrations
6
7 run_on_sky
8 deploying_with_kserve
9 + deploying_with_kubeai
10 deploying_with_triton
11 deploying_with_bentoml
12 deploying_with_cerebrium