A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 02:15 UTC
[Docs] Add Kuberay to deployment integrations (#20592)
Signed-off-by: Ricardo Decal <rdecal@anyscale.com>
Ricardo Decal committed
Jul 14, 2025 at 23:13 UTC
054c8657e30518af0aab10366f66f03287e45eff
2 files changed
+21
docs/deployment/integrations/kuberay.md
new
+20
@@ -0,0 +1,20 @@
1
+# KubeRay
2
+
3
+[KubeRay](https://github.com/ray-project/kuberay) provides a Kubernetes-native way to run vLLM workloads on Ray clusters.
4
+A Ray cluster can be declared in YAML, and the operator then handles pod scheduling, networking configuration, restarts, and blue-green deployments — all while preserving the familiar Kubernetes experience.
5
+
6
+## Why KubeRay instead of manual scripts?
7
+
8
+| Feature | Manual scripts | KubeRay |
9
+|---------|-----------------------------------------------------------|---------|
10
+| Cluster bootstrap | Manually SSH into every node and run a script | One command to create or update the whole cluster: `kubectl apply -f cluster.yaml` |
11
+| Autoscaling | Manual | Automatically patches CRDs for adjusting cluster size |
12
+| Upgrades | Tear down & re-create manually | Blue/green deployment updates supported |
13
+| Declarative config | Bash flags & environment variables | Git-ops-friendly YAML CRDs (RayCluster/RayService) |
14
+
15
+Using KubeRay reduces the operational burden and simplifies integration of Ray + vLLM with existing Kubernetes workflows (CI/CD, secrets, storage classes, etc.).
16
+
17
+## Learn more
18
+
19
+* ["Serve a Large Language Model using Ray Serve LLM on Kubernetes"](https://docs.ray.io/en/master/cluster/kubernetes/examples/rayserve-llm-example.html) - An end-to-end example of how to serve a model using vLLM, KubeRay, and Ray Serve.
20
+* [KubeRay documentation](https://docs.ray.io/en/latest/cluster/kubernetes/index.html)
docs/deployment/k8s.md
+1
@@ -13,6 +13,7 @@ Alternatively, you can deploy vLLM to Kubernetes using any of the following:
13
- [Helm](frameworks/helm.md)
14
- [InftyAI/llmaz](integrations/llmaz.md)
15
- [KServe](integrations/kserve.md)
16
+- [KubeRay](integrations/kuberay.md)
17
- [kubernetes-sigs/lws](frameworks/lws.md)
18
- [meta-llama/llama-stack](integrations/llamastack.md)
19
- [substratusai/kubeai](integrations/kubeai.md)