A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 03:30 UTC
[Docs] Add Ray Serve LLM section to openai compatible server guide (#20595)
Signed-off-by: Ricardo Decal <rdecal@anyscale.com>
Ricardo Decal committed
Jul 7, 2025 at 20:08 UTC
90a2769f2030bd11299f83871d7bc1c06db88cfb
1 file changed
+14
docs/serving/openai_compatible_server.md
+14
@@ -775,3 +775,17 @@ The following extra parameters are supported:
775
```python
776
--8<-- "vllm/entrypoints/openai/protocol.py:rerank-extra-params"
777
```
778
+
779
+## Ray Serve LLM
780
+
781
+Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure.
782
+
783
+Key capabilities:
784
+
785
+- Exposes an OpenAI-compatible HTTP API as well as a Pythonic API.
786
+- Scales from a single GPU to a multi-node cluster without code changes.
787
+- Provides observability and autoscaling policies through Ray dashboards and metrics.
788
+
789
+The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: <gh-file:examples/online_serving/ray_serve_deepseek.py>.
790
+
791
+Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/serving-llms.html).