A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 15, 03:30 UTC
[Docs] Improve docstring for ray data llm example (#20597)
Signed-off-by: Ricardo Decal <rdecal@anyscale.com>
Ricardo Decal committed
Jul 7, 2025 at 20:06 UTC
e60d422f19f1f103307a759e4e5399ad93340cbe
1 file changed
+11
-9
examples/offline_inference/batch_llm_inference.py
+11
-9
@@ -3,17 +3,19 @@
3
"""
4
This example shows how to use Ray Data for data parallel batch inference.
5
6
-Ray Data is a data processing framework that can handle large datasets
7
-and integrates tightly with vLLM for data-parallel inference.
8
-
9
-As of Ray 2.44, Ray Data has a native integration with
10
-vLLM (under ray.data.llm).
6
+Ray Data is a data processing framework that can process very large datasets
7
+with first-class support for vLLM.
8
9
Ray Data provides functionality for:
13
-* Reading and writing to cloud storage (S3, GCS, etc.)
14
-* Automatic sharding and load-balancing across a cluster
15
-* Optimized configuration of vLLM using continuous batching
16
-* Compatible with tensor/pipeline parallel inference as well.
10
+* Reading and writing to most popular file formats and cloud object storage.
11
+* Streaming execution, so you can run inference on datasets that far exceed
12
+ the aggregate RAM of the cluster.
13
+* Scale up the workload without code changes.
14
+* Automatic sharding, load-balancing, and autoscaling across a Ray cluster,
15
+ with built-in fault-tolerance and retry semantics.
16
+* Continuous batching that keeps vLLM replicas saturated and maximizes GPU
17
+ utilization.
18
+* Compatible with tensor/pipeline parallel inference.
19
20
Learn more about Ray Data's LLM integration:
21
https://docs.ray.io/en/latest/data/working-with-llms.html