51c1ee9b7c · tduncan/Nvidia-Optimization-Stack-VLLM

.github/mergify.yml

+1 -3

@@ -477,9 +477,7 @@ pull_request_rules:

477 conditions:

478 - label != stale

479 - or:

480 - - files~=^examples/online_serving/disaggregated[^/]*/.*

481 - - files~=^examples/offline_inference/disaggregated[^/]*/.*

482 - - files~=^examples/others/lmcache/

480 + - files~=^examples/disaggregated/

481 - files~=^tests/v1/kv_connector/

482 - files~=^vllm/distributed/kv_transfer/

483 - title~=(?i)\bP/?D\b

docs/deployment/frameworks/anyscale.md

+1 -1

@@ -3,7 +3,7 @@

3 [Anyscale](https://www.anyscale.com) is a managed, multi-cloud platform developed by the creators of Ray.

4

5 Anyscale automates the entire lifecycle of Ray clusters in your AWS, GCP, or Azure account, delivering the flexibility of open-source Ray

6 -without the operational overhead of maintaining Kubernetes control planes, configuring autoscalers, managing observability stacks, or manually managing head and worker nodes with helper scripts like [examples/online_serving/run_cluster.sh](../../../examples/online_serving/run_cluster.sh).

6 +without the operational overhead of maintaining Kubernetes control planes, configuring autoscalers, managing observability stacks, or manually managing head and worker nodes with helper scripts like [examples/ray_serving/run_cluster.sh](../../../examples/ray_serving/run_cluster.sh).

7

8 When serving large language models with vLLM, Anyscale can rapidly provision [production-ready HTTPS endpoints](https://docs.anyscale.com/examples/deploy-ray-serve-llms) or [fault-tolerant batch inference jobs](https://docs.anyscale.com/examples/ray-data-llm).

9

docs/design/p2p_nccl_connector.md

+2 -2

@@ -88,7 +88,7 @@ pip install "vllm>=0.9.2"

88 #### Proxy (e.g. 10.0.1.1)

89

90 ```shell

91 -cd {your vllm directory}/examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/

91 +cd {your vllm directory}/examples/disaggregated/p2p_nccl_xpyd/

92 python3 disagg_proxy_p2p_nccl_xpyd.py &

93 ```

94

@@ -181,7 +181,7 @@ python3 disagg_proxy_p2p_nccl_xpyd.py &

181 #### Proxy (e.g. 10.0.1.1)

182

183 ```shell

184 -cd {your vllm directory}/examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/

184 +cd {your vllm directory}/examples/disaggregated/p2p_nccl_xpyd/

185 python3 disagg_proxy_p2p_nccl_xpyd.py &

186 ```

187

docs/features/disagg_encoder.md

+2 -2

@@ -36,10 +36,10 @@ The current reference pathway is **ExampleConnector**.

36 Below ready-to-run scripts shows the workflow:

37

38 1 Encoder instance + 1 PD instance:

39 -`examples/online_serving/disaggregated_encoder/disagg_1e1pd_example.sh`

39 +`examples/disaggregated/disaggregated_encoder/disagg_1e1pd_example.sh`

40

41 1 Encoder instance + 1 Prefill instance + 1 Decode instance:

42 -`examples/online_serving/disaggregated_encoder/disagg_1e1p1d_example.sh`

42 +`examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`

43

44 ---

45

docs/features/disagg_prefill.md

+6 -6

@@ -17,15 +17,15 @@ Two main reasons:

17

18 ## Usage example

19

20 -Please refer to [examples/online_serving/disaggregated_prefill.sh](../../examples/online_serving/disaggregated_prefill.sh) for the example usage of disaggregated prefilling.

20 +Please refer to [examples/disaggregated/disaggregated_prefill.sh](../../examples/disaggregated/disaggregated_prefill.sh) for the example usage of disaggregated prefilling.

21

22 Now supports 6 types of connectors:

23

24 -- **ExampleConnector**: refer to [examples/offline_inference/disaggregated-prefill-v1/run.sh](../../examples/offline_inference/disaggregated-prefill-v1/run.sh) for the example usage of ExampleConnector disaggregated prefilling.

25 -- **LMCacheConnectorV1**: refer to [examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission.

24 +- **ExampleConnector**: refer to [examples/disaggregated/example_connector/run.sh](../../examples/disaggregated/example_connector/run.sh) for the example usage of ExampleConnector disaggregated prefilling.

25 +- **LMCacheConnectorV1**: refer to [examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission.

26 - **NixlConnector**: refer to [tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md). For feature compatibility details, see [NixlConnector Compatibility Matrix](nixl_connector_compatibility.md).

27 -- **P2pNcclConnector**: refer to [examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh](../../examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh) for the example usage of P2pNcclConnector disaggregated prefilling.

28 -- **MooncakeConnector**: refer to [examples/online_serving/disaggregated_serving/mooncake_connector/run_mooncake_connector.sh](../../examples/online_serving/disaggregated_serving/mooncake_connector/run_mooncake_connector.sh) for the example usage of ExampleConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).

27 +- **P2pNcclConnector**: refer to [examples/disaggregated/p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh](../../examples/disaggregated/p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh) for the example usage of P2pNcclConnector disaggregated prefilling.

28 +- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).

29 - **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:

30

31 ```bash

@@ -44,7 +44,7 @@ For NixlConnector, you may also specify one or multiple NIXL_Backend. Such as:

44 --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"block_size": 64, "cpu_bytes_to_use": 1000000000}}'

45 ```

46

47 -- **FlexKVConnectorV1**: refer to [examples/offline_inference/prefix_caching_flexkv.py](../../examples/offline_inference/prefix_caching_flexkv.py) for the example usage of FlexKVConnectorV1. FlexKV is a distributed KV Store and multi-level cache management system for ultra-large-scale LLM inference.

47 +- **FlexKVConnectorV1**: refer to [examples/disaggregated/flexkv_connector/prefix_caching_flexkv.py](../../examples/disaggregated/flexkv_connector/prefix_caching_flexkv.py) for the example usage of FlexKVConnectorV1. FlexKV is a distributed KV Store and multi-level cache management system for ultra-large-scale LLM inference.

48

49 ```bash

50 --kv-transfer-config '{"kv_connector":"FlexKVConnectorV1","kv_role":"kv_both"}'

docs/features/mooncake_connector_usage.md

+3 -3

@@ -31,7 +31,7 @@ vllm serve Qwen/Qwen2.5-7B-Instruct --port 8020 --kv-transfer-config '{"kv_conne

31 ### Proxy

32

33 ```bash

34 -python examples/online_serving/disaggregated_serving/mooncake_connector/mooncake_connector_proxy.py --prefill http://192.168.0.2:8010 --decode http://192.168.0.3:8020

34 +python examples/disaggregated/disaggregated_serving/mooncake_connector/mooncake_connector_proxy.py --prefill http://192.168.0.2:8010 --decode http://192.168.0.3:8020

35 ```

36

37 Now you can send requests to the proxy server through port 8000.

@@ -65,5 +65,5 @@ Now you can send requests to the proxy server through port 8000.

65

66 Refer to these example scripts in the vLLM repository:

67

68 -- [run_mooncake_connector.sh](../../examples/online_serving/disaggregated_serving/mooncake_connector/run_mooncake_connector.sh)

69 -- [mooncake_connector_proxy.py](../../examples/online_serving/disaggregated_serving/mooncake_connector/mooncake_connector_proxy.py)

68 +- [run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh)

69 +- [mooncake_connector_proxy.py](../../examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py)

docs/serving/distributed_troubleshooting.md

+2 -2

@@ -4,11 +4,11 @@ For general troubleshooting, see [Troubleshooting](../usage/troubleshooting.md).

4

5 ## Verify inter-node GPU communication

6

7 -After you start the Ray cluster, verify GPU-to-GPU communication across nodes. Proper configuration can be non-trivial. For more information, see [troubleshooting script](../usage/troubleshooting.md#incorrect-hardwaredriver). If you need additional environment variables for communication configuration, append them to [examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh), for example `-e NCCL_SOCKET_IFNAME=eth0`. Setting environment variables during cluster creation is recommended because the variables propagate to all nodes. In contrast, setting environment variables in the shell affects only the local node. For more information, see <https://github.com/vllm-project/vllm/issues/6803>.

7 +After you start the Ray cluster, verify GPU-to-GPU communication across nodes. Proper configuration can be non-trivial. For more information, see [troubleshooting script](../usage/troubleshooting.md#incorrect-hardwaredriver). If you need additional environment variables for communication configuration, append them to [examples/ray_serving/run_cluster.sh](../../examples/ray_serving/run_cluster.sh), for example `-e NCCL_SOCKET_IFNAME=eth0`. Setting environment variables during cluster creation is recommended because the variables propagate to all nodes. In contrast, setting environment variables in the shell affects only the local node. For more information, see <https://github.com/vllm-project/vllm/issues/6803>.

8

9 ## No available node types can fulfill resource request

10

11 -The error message `Error: No available node types can fulfill resource request` can appear even when the cluster has enough GPUs. The issue often occurs when nodes have multiple IP addresses and vLLM can't select the correct one. Ensure that vLLM and Ray use the same IP address by setting `VLLM_HOST_IP` in [examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh) (with a different value on each node). Use `ray status` and `ray list nodes` to verify the chosen IP address. For more information, see <https://github.com/vllm-project/vllm/issues/7815>.

11 +The error message `Error: No available node types can fulfill resource request` can appear even when the cluster has enough GPUs. The issue often occurs when nodes have multiple IP addresses and vLLM can't select the correct one. Ensure that vLLM and Ray use the same IP address by setting `VLLM_HOST_IP` in [examples/ray_serving/run_cluster.sh](../../examples/ray_serving/run_cluster.sh) (with a different value on each node). Use `ray status` and `ray list nodes` to verify the chosen IP address. For more information, see <https://github.com/vllm-project/vllm/issues/7815>.

12

13 ## Ray observability

14

docs/serving/openai_compatible_server.md

+1 -1

@@ -542,6 +542,6 @@ Key capabilities:

542 - Scales from a single GPU to a multi-node cluster without code changes.

543 - Provides observability and autoscaling policies through Ray dashboards and metrics.

544

545 -The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: [examples/online_serving/ray_serve_deepseek.py](../../examples/online_serving/ray_serve_deepseek.py).

545 +The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: [examples/ray_serving/ray_serve_deepseek.py](../../examples/ray_serving/ray_serve_deepseek.py).

546

547 Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/index.html).

docs/serving/parallelism_scaling.md

+2 -2

@@ -78,7 +78,7 @@ For details, see the [Ray documentation](https://docs.ray.io/en/latest/index.htm

78

79 ### Ray cluster setup with containers

80

81 -The helper script [examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh) starts containers across nodes and initializes Ray. By default, the script runs Docker without administrative privileges, which prevents access to the GPU performance counters when profiling or tracing. To enable admin privileges, add the `--cap-add=CAP_SYS_ADMIN` flag to the Docker command.

81 +The helper script [examples/ray_serving/run_cluster.sh](../../examples/ray_serving/run_cluster.sh) starts containers across nodes and initializes Ray. By default, the script runs Docker without administrative privileges, which prevents access to the GPU performance counters when profiling or tracing. To enable admin privileges, add the `--cap-add=CAP_SYS_ADMIN` flag to the Docker command.

82

83 Choose one node as the head node and run:

84

@@ -162,7 +162,7 @@ vllm serve /path/to/the/model/in/the/container \

162

163 Efficient tensor parallelism requires fast internode communication, preferably through high-speed network adapters such as InfiniBand.

164 To set up the cluster to use InfiniBand, append additional arguments like `--privileged -e NCCL_IB_HCA=mlx5` to the

165 -[examples/online_serving/run_cluster.sh](../../examples/online_serving/run_cluster.sh) helper script.

165 +[examples/ray_serving/run_cluster.sh](../../examples/ray_serving/run_cluster.sh) helper script.

166 Contact your system administrator for more information about the required flags.

167

168 ## Enabling GPUDirect RDMA

examples/disaggregated/disaggregated_encoder/README.md renamed

examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh renamed

examples/disaggregated/disaggregated_encoder/disagg_1e1pd_example.sh renamed

examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py renamed

examples/disaggregated/disaggregated_prefill.py renamed

examples/disaggregated/disaggregated_prefill.sh renamed

examples/disaggregated/disaggregated_serving/README.md renamed

examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py renamed

+1 -1

@@ -5,7 +5,7 @@ This file provides a disaggregated prefilling proxy demo to demonstrate an

5 example usage of XpYd disaggregated prefilling.

6 We can launch multiple vllm instances (2 for prefill and 2 for decode), and

7 launch this proxy demo through:

8 - python3 examples/online_serving/disaggregated_serving/disagg_proxy_demo.py \

8 + python3 examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py \

9 --model $model_name \

10 --prefill localhost:8100 localhost:8101 \

11 --decode localhost:8200 localhost:8201 \

examples/disaggregated/disaggregated_serving/example_mm_serve.py renamed

examples/disaggregated/disaggregated_serving/kv_events.sh renamed

examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py renamed

examples/disaggregated/ec_both_encoder/ec_both_encoder.sh renamed

examples/disaggregated/example_connector/README.md renamed

+1 -1

@@ -5,6 +5,6 @@ This example contains scripts that demonstrate disaggregated prefill in the offl

5 ## Files

6

7 - `run.sh` - A helper script that will run `prefill_example.py` and `decode_example.py` sequentially.

8 - - Make sure you are in the `examples/offline_inference/disaggregated-prefill-v1` directory before running `run.sh`.

8 + - Make sure you are in the `examples/disaggregated/example_connector` directory before running `run.sh`.

9 - `prefill_example.py` - A script which performs prefill only, saving the KV state to the `local_storage` directory and the prompts to `output.txt`.

10 - `decode_example.py` - A script which performs decode only, loading the KV state from the `local_storage` directory and the prompts from `output.txt`.

examples/disaggregated/example_connector/decode_example.py renamed

examples/disaggregated/example_connector/prefill_example.py renamed

examples/disaggregated/example_connector/run.sh renamed

examples/disaggregated/flexkv_connector/prefix_caching_flexkv.py renamed

+1 -1

@@ -14,7 +14,7 @@ Requirements:

14

15 Usage:

16 1. Run this script:

17 - python examples/offline_inference/prefix_caching_flexkv.py \

17 + python examples/disaggregated/flexkv_connector/prefix_caching_flexkv.py \

18 --model /path/to/your/model

19

20 2. Arguments:

examples/disaggregated/kv_load_failure_recovery_offline/README.md renamed

+2 -2

@@ -1,12 +1,12 @@

1 # KV Load Failure Recovery Test

2

3 -This example builds upon the `disaggregated-prefill-v1` example in `examples/offline_inference`.

3 +This example builds upon the `example_connector` example in `examples/disaggregated`.

4

5 It demonstrates vLLM's ability to recover from KV load failures in both synchronous and asynchronous loading modes. The goal is to verify that vLLM correctly identifies invalid KV blocks, reschedules the affected requests, and ensures successful and consistent output.

6

7 ## Files

8

9 -- `prefill_example.py` – performs the prefill stage and saves KV data (same as in `disaggregated-prefill-v1`).

9 +- `prefill_example.py` – performs the prefill stage and saves KV data (same as in `example_connector`).

10 - `decode_example.py` – performs the decode stage. Accepts:

11 - `--simulate-failure`: simulates KV load failure using a custom connector.

12 - `--async-load`: enables asynchronous KV loading mode.

examples/disaggregated/kv_load_failure_recovery_offline/decode_example.py renamed

examples/disaggregated/kv_load_failure_recovery_offline/load_recovery_example_connector.py renamed

examples/disaggregated/kv_load_failure_recovery_offline/prefill_example.py renamed

examples/disaggregated/kv_load_failure_recovery_offline/run.sh renamed

examples/disaggregated/lmcache/README.md renamed

examples/disaggregated/lmcache/cpu_offload_lmcache.py renamed

examples/disaggregated/lmcache/disagg_prefill_lmcache_v0.py renamed

examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-decoder-config.yaml renamed

examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml renamed

examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh renamed

examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_proxy_server.py renamed

examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh renamed

examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py renamed

examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py renamed

examples/disaggregated/mooncake_connector/run_mooncake_connector.sh renamed

examples/disaggregated/p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh renamed

examples/disaggregated/p2p_nccl_xpyd/disagg_proxy_p2p_nccl_xpyd.py renamed

examples/ray_serving/batch_llm_inference.py renamed

examples/ray_serving/elastic_ep/bench.sh renamed

examples/ray_serving/elastic_ep/scale.py renamed

examples/ray_serving/elastic_ep/serve_deepseek_v2.sh renamed

examples/ray_serving/multi-node-serving.sh renamed

examples/ray_serving/ray_serve_deepseek.py renamed

examples/ray_serving/run_cluster.sh renamed

tests/v1/ec_connector/integration/README.md

+1 -1

@@ -122,7 +122,7 @@ Quick sanity check:

122 - Encoder cache should enable exact output reproduction

123 - Test cleans up all instances and cache files after completion

124 - Safe to run multiple times (idempotent)

125 -- We setup the PD disagg part with NixlConnector. Please read details about EPD in `examples/online_serving/disaggregated_encoder/README.md`

125 +- We setup the PD disagg part with NixlConnector. Please read details about EPD in `examples/disaggregated/disaggregated_encoder/README.md`

126

127 ## Requirements

128

tests/v1/ec_connector/integration/run_epd_correctness_test.sh

+2 -2

@@ -185,7 +185,7 @@ run_epd_1e_1pd() {

185

186 # Start proxy

187 echo "Starting EPD proxy on port $PROXY_PORT"

188 - python "${GIT_ROOT}/examples/online_serving/disaggregated_encoder/disagg_epd_proxy.py" \

188 + python "${GIT_ROOT}/examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py" \

189 --host "0.0.0.0" \

190 --port "$PROXY_PORT" \

191 --encode-servers-urls "http://localhost:$ENCODE_PORT" \

@@ -411,7 +411,7 @@ run_epd_1e_1p_1d() {

411

412 # Start proxy

413 echo "Starting EPD proxy on port $PROXY_PORT"

414 - python "${GIT_ROOT}/examples/online_serving/disaggregated_encoder/disagg_epd_proxy.py" \

414 + python "${GIT_ROOT}/examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py" \

415 --host "0.0.0.0" \

416 --port "$PROXY_PORT" \

417 --encode-servers-urls "http://localhost:$ENCODE_PORT" \

vllm/distributed/kv_transfer/README.md

+1 -1

@@ -22,7 +22,7 @@ NOTE: If you want to not only transfer KV caches, but adjust the model execution

22

23 ## Disaggregated prefilling

24

25 -The example usage is in [this file](../../../examples/online_serving/disaggregated_prefill.sh).

25 +The example usage is in [this file](../../../examples/disaggregated/disaggregated_prefill.sh).

26

27 Here is the diagram of how we run disaggregated prefilling.

28

[Examples] Resettle Disaggregated examples. (#40759)